2024-03-11T20:58:52.994 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T20:58:52.994 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789 -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/orig.config.yaml 2024-03-11T20:58:53.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T20:58:53.120 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789 branch: main description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591789' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEL3NFitolg0GcBmA1aZq6ZUy9YNJX5q4shU5wJozfPBMjoHBuRLq5bz0GvpfWeyo2O18kiF7TALmAS+k76myWY= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBo4QJ51si+hcxMlkyJvTAtHEimko/01zzJxKlVxhkgPF+6wxegdI8U6w+5hqjNiOLxDGSAQOtD0rmEnMba0cAU= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T20:58:53.121 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-11T20:58:53.121 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-11T20:58:53.122 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T20:58:53.123 INFO:teuthology.task.internal:Checking packages... 2024-03-11T20:58:53.146 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-11T20:58:53.146 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T20:58:53.146 INFO:teuthology.packaging:ref: None 2024-03-11T20:58:53.146 INFO:teuthology.packaging:tag: None 2024-03-11T20:58:53.147 INFO:teuthology.packaging:branch: main 2024-03-11T20:58:53.147 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:58:53.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-03-11T20:58:53.507 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:58:53.509 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T20:58:53.515 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T20:58:53.515 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T20:58:53.549 INFO:teuthology.task.internal:Saving configuration 2024-03-11T20:58:53.560 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T20:58:53.588 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:58:53.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:52:51.095129', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAHM47WkfN21p+ucyE18vp7B/e+eupg8iVHtwQXd9hfl'} 2024-03-11T20:58:53.636 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:52:51.096233', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/OO2eU/hykjvUJRMTRCG6dGKzhabaC99xv/FlmalSM'} 2024-03-11T20:58:53.636 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T20:58:53.643 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T20:58:53.643 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T20:58:53.643 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T20:58:53.733 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7ad48f3a0>, signals=[15]) 2024-03-11T20:58:53.733 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T20:58:53.739 INFO:teuthology.task.internal:Opening connections... 2024-03-11T20:58:53.739 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-03-11T20:58:53.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:58:53.821 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-11T20:58:53.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:58:53.902 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T20:58:53.910 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-03-11T20:58:53.916 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-03-11T20:58:53.916 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="22.04" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=jammy 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-11T20:58:53.961 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=jammy 2024-03-11T20:58:53.962 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-03-11T20:58:53.981 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-11T20:58:53.986 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-11T20:58:53.986 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Ubuntu" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="22.04" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_CODENAME=jammy 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:ID=ubuntu 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE=debian 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-11T20:58:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:UBUNTU_CODENAME=jammy 2024-03-11T20:58:54.032 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-11T20:58:54.055 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T20:58:54.066 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T20:58:54.081 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T20:58:54.081 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:58:54.084 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:58:54.087 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T20:58:54.107 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T20:58:54.107 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:58:54.131 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:58:54.149 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T20:58:54.202 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T20:58:54.275 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T20:58:54.275 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T20:58:54.275 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:58:54.276 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:58:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.280 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-11T20:58:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.281 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-11T20:58:54.327 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-56-generic 2024-03-11T20:58:54.328 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.15.0-56-generic 2024-03-11T20:58:54.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-03-11T20:58:54.329 INFO:teuthology.orchestra.run.smithi143.stdout:5.15.0-56-generic 2024-03-11T20:58:54.329 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.15.0-56-generic 2024-03-11T20:58:54.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-03-11T20:58:54.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-03-11T20:58:54.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-03-11T20:58:54.631 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-11T20:58:54.632 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-11T20:58:54.744 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:58:54.744 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:58:54.825 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-11T20:58:54.826 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-11T20:58:54.838 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-11T20:58:54.862 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-11T20:58:54.919 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,241 kB] 2024-03-11T20:58:54.936 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,241 kB] 2024-03-11T20:58:54.960 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [223 kB] 2024-03-11T20:58:54.962 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-11T20:58:54.962 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,526 kB] 2024-03-11T20:58:54.978 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [223 kB] 2024-03-11T20:58:54.981 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-11T20:58:54.982 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,526 kB] 2024-03-11T20:58:54.984 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-11T20:58:54.985 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-11T20:58:54.989 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:58:54.989 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-11T20:58:55.001 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-11T20:58:55.008 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-11T20:58:55.010 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-11T20:58:55.016 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-11T20:58:55.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-11T20:58:55.020 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-11T20:58:55.020 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-11T20:58:55.020 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-11T20:58:55.021 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:58:55.021 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-11T20:58:55.029 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-11T20:58:55.038 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-11T20:58:55.039 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-11T20:58:55.040 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-11T20:58:55.040 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-11T20:58:55.043 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-11T20:58:55.073 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-11T20:58:55.074 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-11T20:58:55.187 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-11T20:58:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-11T20:58:55.326 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-11T20:58:55.327 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,460 kB] 2024-03-11T20:58:55.521 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,460 kB] 2024-03-11T20:58:55.626 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-11T20:58:55.666 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-11T20:58:55.667 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-11T20:58:55.681 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-11T20:58:55.681 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-11T20:58:55.682 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-11T20:58:55.682 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-11T20:58:55.683 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-11T20:58:55.741 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-11T20:58:55.741 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-11T20:58:55.742 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-11T20:58:55.748 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:58:55.749 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:58:55.912 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-11T20:58:55.913 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-11T20:58:56.540 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-11T20:58:56.578 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-11T20:58:56.582 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-11T20:58:56.590 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-11T20:58:56.592 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-11T20:58:56.592 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-11T20:58:56.594 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-11T20:58:56.595 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-11T20:58:56.595 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-11T20:58:56.596 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-11T20:58:56.622 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-11T20:58:56.637 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-11T20:58:56.656 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-11T20:58:56.658 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-11T20:58:56.658 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-11T20:58:56.663 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-11T20:58:56.664 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-11T20:58:56.664 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-11T20:58:56.667 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-11T20:58:56.670 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-11T20:58:56.787 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-11T20:58:56.836 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-11T20:58:56.870 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-11T20:58:56.875 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-11T20:58:56.918 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-11T20:58:56.994 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-11T20:58:57.014 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-11T20:58:57.042 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-11T20:58:57.046 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-11T20:58:57.094 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-11T20:59:01.460 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 12.1 MB in 3s (4,734 kB/s) 2024-03-11T20:59:01.480 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 12.1 MB in 3s (4,831 kB/s) 2024-03-11T20:59:02.775 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T20:59:02.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:59:02.854 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T20:59:02.859 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T20:59:02.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:59:02.929 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T20:59:03.049 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-03-11T20:59:03.049 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-03-11T20:59:03.100 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-03-11T20:59:03.101 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-03-11T20:59:03.221 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:59:03.221 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:59:03.221 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:59:03.222 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:59:03.222 INFO:teuthology.orchestra.run.smithi203.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-11T20:59:03.222 INFO:teuthology.orchestra.run.smithi203.stdout: python2.7-minimal 2024-03-11T20:59:03.222 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:59:03.223 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-03-11T20:59:03.223 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:59:03.223 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-11T20:59:03.224 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-11T20:59:03.225 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-03-11T20:59:03.225 INFO:teuthology.orchestra.run.smithi203.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-11T20:59:03.225 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-03-11T20:59:03.225 INFO:teuthology.orchestra.run.smithi203.stdout: thermald 2024-03-11T20:59:03.249 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout: python2.7-minimal 2024-03-11T20:59:03.250 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:59:03.251 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-03-11T20:59:03.251 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:59:03.251 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-11T20:59:03.252 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-11T20:59:03.253 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-03-11T20:59:03.253 INFO:teuthology.orchestra.run.smithi143.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-11T20:59:03.253 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-03-11T20:59:03.253 INFO:teuthology.orchestra.run.smithi143.stdout: thermald 2024-03-11T20:59:03.278 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-03-11T20:59:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:59:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-11T20:59:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-11T20:59:03.279 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-03-11T20:59:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-11T20:59:03.302 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-03-11T20:59:03.303 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:59:03.303 INFO:teuthology.orchestra.run.smithi143.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-11T20:59:03.303 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-11T20:59:03.304 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2024-03-11T20:59:03.304 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-11T20:59:03.459 INFO:teuthology.orchestra.run.smithi203.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:59:03.459 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 113 MB of archives. 2024-03-11T20:59:03.459 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-11T20:59:03.459 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-11T20:59:03.483 INFO:teuthology.orchestra.run.smithi143.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:59:03.484 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 113 MB of archives. 2024-03-11T20:59:03.484 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-11T20:59:03.484 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-11T20:59:04.879 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-11T20:59:04.897 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-11T20:59:05.232 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-11T20:59:05.282 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-11T20:59:07.443 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-11T20:59:07.443 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-11T20:59:07.443 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-11T20:59:07.567 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-11T20:59:07.567 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-11T20:59:07.568 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-11T20:59:07.989 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-11T20:59:08.085 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-11T20:59:08.165 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-11T20:59:08.318 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-11T20:59:08.485 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 113 MB in 5s (23.7 MB/s) 2024-03-11T20:59:08.682 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-11T20:59:08.729 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 113 MB in 5s (22.5 MB/s) 2024-03-11T20:59:09.005 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-11T20:59:10.724 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-11T20:59:10.729 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:10.861 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:11.025 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-11T20:59:11.029 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:11.093 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:13.327 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-11T20:59:13.349 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:13.436 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:13.453 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-11T20:59:13.474 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:13.578 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:13.905 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-11T20:59:13.926 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:13.955 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:14.056 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-11T20:59:14.076 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:14.114 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:20.455 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:20.580 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:20.600 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:20.719 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:20.888 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:20.992 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:21.060 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:21.172 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-11T20:59:21.318 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-11T20:59:21.362 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:59:21.483 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-11T20:59:21.494 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-11T20:59:21.533 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:59:27.161 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-11T20:59:27.198 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:27.236 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:27.452 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-11T20:59:27.489 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:59:27.526 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:29.798 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:29.928 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:31.054 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:59:31.085 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:59:31.176 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:31.207 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:59:31.302 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-11T20:59:31.440 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:31.582 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:59:31.712 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:31.864 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-11T20:59:31.993 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:32.772 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:32.772 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:32.772 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:32.772 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:32.926 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:33.344 INFO:teuthology.orchestra.run.smithi143.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:33.344 INFO:teuthology.orchestra.run.smithi143.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:33.344 INFO:teuthology.orchestra.run.smithi143.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:33.344 INFO:teuthology.orchestra.run.smithi143.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:33.503 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:34.352 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:34.980 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:35.729 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-11T20:59:35.830 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-11T20:59:36.699 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:36.762 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-11T20:59:36.763 INFO:teuthology.orchestra.run.smithi203.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:36.765 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-11T20:59:36.899 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-11T20:59:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:59:37.096 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-11T20:59:37.096 INFO:teuthology.orchestra.run.smithi143.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:50.684 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-11T20:59:50.685 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub' 2024-03-11T20:59:50.689 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:59:50.716 INFO:teuthology.orchestra.run.smithi203.stdout:Generating grub configuration file ... 2024-03-11T20:59:50.989 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:51.004 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:51.333 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:51.340 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:51.370 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-11T20:59:51.370 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub' 2024-03-11T20:59:51.393 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:59:51.415 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:59:51.415 INFO:teuthology.orchestra.run.smithi143.stdout:Generating grub configuration file ... 2024-03-11T20:59:51.421 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:59:51.594 INFO:teuthology.orchestra.run.smithi203.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:59:51.594 INFO:teuthology.orchestra.run.smithi203.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:59:51.594 INFO:teuthology.orchestra.run.smithi203.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:59:51.635 INFO:teuthology.orchestra.run.smithi203.stdout:done 2024-03-11T20:59:51.684 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:51.699 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:52.037 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:52.043 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:52.098 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-03-11T20:59:52.113 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:59:52.117 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.100.97 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-03-11T20:59:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:59:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-03-11T20:59:52.119 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:59:52.120 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-11T20:59:52.120 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:59:52.120 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:59:52.120 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T20:59:52.120 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:59:52.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-03-11T20:59:52.293 INFO:teuthology.orchestra.run.smithi143.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:59:52.294 INFO:teuthology.orchestra.run.smithi143.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:59:52.294 INFO:teuthology.orchestra.run.smithi143.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:59:52.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-03-11T20:59:52.339 INFO:teuthology.orchestra.run.smithi143.stdout:done 2024-03-11T20:59:52.476 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:59:52.476 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:59:52.480 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:59:52.500 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:59:52.843 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-03-11T20:59:52.913 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 21 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.15.0.100.97 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:59:52.914 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:59:52.915 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-03-11T20:59:52.915 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-03-11T20:59:52.915 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:59:52.915 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-03-11T20:59:52.915 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-11T20:59:52.915 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:59:52.915 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:59:52.915 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T20:59:52.915 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:59:52.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-03-11T20:59:53.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-03-11T20:59:53.254 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:59:53.337 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:59:53.419 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:59:53.500 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:59:53.789 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T20:59:53.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:59:53.870 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T20:59:54.092 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-03-11T20:59:54.092 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-03-11T20:59:54.275 INFO:teuthology.orchestra.run.smithi203.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-11T20:59:54.275 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:59:54.275 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:59:54.275 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:59:54.276 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:59:54.276 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:59:54.276 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:59:54.276 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:59:54.317 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:59:54.319 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.100.97 2024-03-11T20:59:54.335 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:59:54.336 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:59:54.336 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-03-11T20:59:54.336 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-03-11T20:59:54.336 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:59:54.336 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-03-11T20:59:54.336 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-11T20:59:54.382 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.VUJrtO54I0 2024-03-11T20:59:54.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VUJrtO54I0 2024-03-11T20:59:54.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.VUJrtO54I0 2024-03-11T20:59:54.571 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.VUJrtO54I0 is 10KB 2024-03-11T20:59:54.585 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.VUJrtO54I0 2024-03-11T20:59:54.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-11T20:59:54.645 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T20:59:54.647 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T20:59:54.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-11T20:59:54.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-11T20:59:54.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:59:54.713 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-11T20:59:54.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo update-grub 2024-03-11T20:59:54.722 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T20:59:54.916 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-03-11T20:59:54.917 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-03-11T20:59:55.088 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-11T20:59:55.088 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:59:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:59:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:59:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:59:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:59:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:59:55.089 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:59:55.133 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:59:55.135 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 21 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-03-11T20:59:55.190 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.15.0.100.97 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:59:55.191 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-03-11T20:59:55.191 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-11T20:59:55.236 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.lXXMfYGxFF 2024-03-11T20:59:55.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub/grub.cfg /tmp/tmp.lXXMfYGxFF 2024-03-11T20:59:55.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.lXXMfYGxFF 2024-03-11T20:59:55.407 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.lXXMfYGxFF is 10KB 2024-03-11T20:59:55.420 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.lXXMfYGxFF 2024-03-11T20:59:55.426 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-11T20:59:55.479 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T20:59:55.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-11T20:59:55.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-11T20:59:55.558 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-11T20:59:55.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo update-grub 2024-03-11T20:59:55.961 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub' 2024-03-11T20:59:55.962 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:59:55.964 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-03-11T20:59:56.134 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:56.140 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:56.356 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:56.362 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:56.400 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:59:56.404 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:59:56.521 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:59:56.521 INFO:teuthology.orchestra.run.smithi203.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:59:56.521 INFO:teuthology.orchestra.run.smithi203.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:59:56.530 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-03-11T20:59:56.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-03-11T20:59:56.879 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub' 2024-03-11T20:59:56.880 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:59:56.893 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-03-11T20:59:57.081 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:59:57.086 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:59:57.305 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:59:57.310 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:59:57.353 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:59:57.359 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:59:57.509 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:59:57.510 INFO:teuthology.orchestra.run.smithi143.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:59:57.510 INFO:teuthology.orchestra.run.smithi143.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:59:57.533 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-03-11T20:59:57.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-03-11T21:00:26.578 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:00:26.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:00:26.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:00:27.546 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:00:27.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:00:27.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:00:44.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-11T21:00:45.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-03-11T21:00:53.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:00:53.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:00:54.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:00:54.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:00:57.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-11T21:00:58.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-03-11T21:01:09.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:01:09.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:01:10.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:01:10.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:01:28.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-03-11T21:01:43.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:01:43.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:01:44.055 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-11T21:01:44.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:01:44.899 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-100-generic"... 2024-03-11T21:01:44.899 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-11T21:01:44.945 INFO:teuthology.orchestra.run.smithi143.stdout:5.15.0-100-generic 2024-03-11T21:01:44.945 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-11T21:01:44.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:01:44.945 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-11T21:01:45.947 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:01:45.947 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:01:46.123 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-03-11T21:01:46.150 DEBUG:teuthology.parallel:result is None 2024-03-11T21:02:09.038 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:02:24.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:02:24.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:02:24.393 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-11T21:02:24.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:02:24.899 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-100-generic"... 2024-03-11T21:02:24.899 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-11T21:02:24.945 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-100-generic 2024-03-11T21:02:24.945 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-11T21:02:24.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:02:24.946 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-11T21:02:25.946 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:02:25.946 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:02:26.308 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-03-11T21:02:26.338 DEBUG:teuthology.parallel:result is None 2024-03-11T21:02:26.339 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:02:26.349 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:02:26.349 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:02:26.352 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:02:26.357 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:02:26.369 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:02:26.378 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:02:26.378 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:02:26.401 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:02:26.445 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:02:26.453 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:02:26.454 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:02:26.461 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:02:26.482 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:02:26.489 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:02:26.506 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:02:26.514 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:02:26.516 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:02:26.524 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:02:26.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:02:26.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:02:26.582 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:02:26.591 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:02:26.592 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:02:26.597 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:02:26.628 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:02:26.648 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:02:26.696 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:02:26.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:02:26.760 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:02:26.766 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:02:26.812 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:02:26.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:02:26.885 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-03-11T21:02:26.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-11T21:02:26.958 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:02:26.967 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:02:26.967 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:02:26.983 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:02:26.994 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2024-03-11T21:02:26.994 DEBUG:teuthology.task.selinux:Excluding smithi203: OS 'ubuntu' does not support SELinux 2024-03-11T21:02:26.995 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:02:26.995 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-11T21:02:26.995 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:02:26.995 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:02:27.008 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:02:27.026 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:02:27.027 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi143.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:09:26.608 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-11T21:09:26.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:09:26.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:26.692 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-11T21:09:26.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-11T21:09:26.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:09:26.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:26.849 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-11T21:09:26.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-11T21:09:26.927 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:09:26.937 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:09:26.937 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:09:26.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:09:26.940 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:09:26.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Command line: ntpd -gq 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: ---------------------------------------------------- 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: ntp-4 is maintained by Network Time Foundation, 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: corporation. Support and training for ntp-4 are 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: available at https://www.nwtime.org/support 2024-03-11T21:09:26.966 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: ---------------------------------------------------- 2024-03-11T21:09:26.967 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: proto: precision = 0.062 usec (-24) 2024-03-11T21:09:26.967 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: basedate set to 2022-02-04 2024-03-11T21:09:26.967 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: gps base set to 2022-02-06 (week 2196) 2024-03-11T21:09:26.967 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T21:09:26.968 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T21:09:26.968 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: restrict ::: KOD does nothing without LIMITED. 2024-03-11T21:09:26.968 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen and drop on 0 v6wildcard [::]:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen normally on 2 lo 127.0.0.1:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen normally on 4 lo [::1]:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2024-03-11T21:09:26.969 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:26 ntpd[17186]: Listening on routing socket on fd #22 for interface updates 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Command line: ntpd -gq 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: ---------------------------------------------------- 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: ntp-4 is maintained by Network Time Foundation, 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: corporation. Support and training for ntp-4 are 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: available at https://www.nwtime.org/support 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: ---------------------------------------------------- 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: proto: precision = 0.044 usec (-24) 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: basedate set to 2022-02-04 2024-03-11T21:09:26.999 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: gps base set to 2022-02-06 (week 2196) 2024-03-11T21:09:27.000 INFO:teuthology.orchestra.run.smithi203.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T21:09:27.000 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T21:09:27.000 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: restrict ::: KOD does nothing without LIMITED. 2024-03-11T21:09:27.001 INFO:teuthology.orchestra.run.smithi203.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-11T21:09:27.001 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen and drop on 0 v6wildcard [::]:123 2024-03-11T21:09:27.001 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-11T21:09:27.002 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen normally on 2 lo 127.0.0.1:123 2024-03-11T21:09:27.002 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen normally on 3 enp3s0f1 172.21.15.203:123 2024-03-11T21:09:27.002 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen normally on 4 lo [::1]:123 2024-03-11T21:09:27.002 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4a23%5]:123 2024-03-11T21:09:27.002 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:27 ntpd[17167]: Listening on routing socket on fd #22 for interface updates 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:27.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:27 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:28 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:28.966 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:28.966 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:28.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:28.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:28.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:28.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:28 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:29.966 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:29.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:29.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:29.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:29.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:29.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:29 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:29.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:30 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:30.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:30.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:30.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:30 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:31.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:31 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:31.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:31.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:31.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:31.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:31.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:31.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:31 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:31.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:31.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:31.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:32.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:32 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:32.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:32 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:32.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:33 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:33.966 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:33.966 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stdout:11 Mar 21:09:33 ntpd[17186]: ntpd: time slew +0.000715 s 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew +0.000715s 2024-03-11T21:09:33.967 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-11T21:09:33.968 INFO:teuthology.orchestra.run.smithi143.stderr:11 Mar 21:09:33 ntpd[17186]: can't open /var/log/ntpstats/loopstats.20240311: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T21:09:33.999 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T21:09:34.000 INFO:teuthology.orchestra.run.smithi203.stdout:11 Mar 21:09:34 ntpd[17167]: ntpd: time slew -0.001326 s 2024-03-11T21:09:34.000 INFO:teuthology.orchestra.run.smithi203.stdout:ntpd: time slew -0.001326s 2024-03-11T21:09:34.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-11T21:09:34.000 INFO:teuthology.orchestra.run.smithi203.stderr:11 Mar 21:09:34 ntpd[17167]: can't open /var/log/ntpstats/loopstats.20240311: Permission denied 2024-03-11T21:09:34.017 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:09:34.017 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-03-11T21:09:34.017 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.017 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.018 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.018 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.050 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:09:34.051 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-03-11T21:09:34.051 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.051 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.051 INFO:teuthology.orchestra.run.smithi203.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.051 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T21:09:34.051 INFO:teuthology.run_tasks:Running task install... 2024-03-11T21:09:34.060 DEBUG:teuthology.task.install:project ceph 2024-03-11T21:09:34.060 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-11T21:09:34.060 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-11T21:09:34.060 INFO:teuthology.task.install:Using flavor: default 2024-03-11T21:09:34.066 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T21:09:34.067 INFO:teuthology.task.install:extra packages: [] 2024-03-11T21:09:34.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2024-03-11T21:09:34.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-key list | grep Ceph 2024-03-11T21:09:34.129 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-11T21:09:34.156 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-11T21:09:34.158 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-11T21:09:34.158 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2024-03-11T21:09:34.159 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-11T21:09:34.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:09:34.183 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-11T21:09:34.183 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph.com (release key) 2024-03-11T21:09:34.184 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-11T21:09:34.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:09:34.304 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/ 2024-03-11T21:09:34.304 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:09:34.326 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/ 2024-03-11T21:09:34.326 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:09:34.419 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:09:34.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-11T21:09:34.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-03-11T21:09:34.762 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T21:09:34.780 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:09:34.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-11T21:09:34.788 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T21:09:34.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-03-11T21:09:34.870 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T21:09:34.889 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-11T21:09:34.952 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T21:09:34.968 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-11T21:09:35.015 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-11T21:09:35.035 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-11T21:09:35.040 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T21:09:35.040 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-11T21:09:35.062 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-11T21:09:35.073 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T21:09:35.081 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T21:09:35.084 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-03-11T21:09:35.089 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-11T21:09:35.112 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T21:09:35.132 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-11T21:09:35.244 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-03-11T21:09:35.329 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 28.9 kB in 1s (41.0 kB/s) 2024-03-11T21:09:35.388 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 28.9 kB in 0s (69.1 kB/s) 2024-03-11T21:09:36.281 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T21:09:36.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g2b0101eb-1jammy cephadm=19.0.0-1979-g2b0101eb-1jammy ceph-mds=19.0.0-1979-g2b0101eb-1jammy ceph-mgr=19.0.0-1979-g2b0101eb-1jammy ceph-common=19.0.0-1979-g2b0101eb-1jammy ceph-fuse=19.0.0-1979-g2b0101eb-1jammy ceph-test=19.0.0-1979-g2b0101eb-1jammy ceph-volume=19.0.0-1979-g2b0101eb-1jammy radosgw=19.0.0-1979-g2b0101eb-1jammy python3-rados=19.0.0-1979-g2b0101eb-1jammy python3-rgw=19.0.0-1979-g2b0101eb-1jammy python3-cephfs=19.0.0-1979-g2b0101eb-1jammy python3-rbd=19.0.0-1979-g2b0101eb-1jammy libcephfs2=19.0.0-1979-g2b0101eb-1jammy libcephfs-dev=19.0.0-1979-g2b0101eb-1jammy librados2=19.0.0-1979-g2b0101eb-1jammy librbd1=19.0.0-1979-g2b0101eb-1jammy rbd-fuse=19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:09:36.315 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T21:09:36.331 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g2b0101eb-1jammy cephadm=19.0.0-1979-g2b0101eb-1jammy ceph-mds=19.0.0-1979-g2b0101eb-1jammy ceph-mgr=19.0.0-1979-g2b0101eb-1jammy ceph-common=19.0.0-1979-g2b0101eb-1jammy ceph-fuse=19.0.0-1979-g2b0101eb-1jammy ceph-test=19.0.0-1979-g2b0101eb-1jammy ceph-volume=19.0.0-1979-g2b0101eb-1jammy radosgw=19.0.0-1979-g2b0101eb-1jammy python3-rados=19.0.0-1979-g2b0101eb-1jammy python3-rgw=19.0.0-1979-g2b0101eb-1jammy python3-cephfs=19.0.0-1979-g2b0101eb-1jammy python3-rbd=19.0.0-1979-g2b0101eb-1jammy libcephfs2=19.0.0-1979-g2b0101eb-1jammy libcephfs-dev=19.0.0-1979-g2b0101eb-1jammy librados2=19.0.0-1979-g2b0101eb-1jammy librbd1=19.0.0-1979-g2b0101eb-1jammy rbd-fuse=19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:09:36.347 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-03-11T21:09:36.384 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-03-11T21:09:36.518 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-03-11T21:09:36.519 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-03-11T21:09:36.555 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-03-11T21:09:36.556 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-03-11T21:09:36.674 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T21:09:36.675 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T21:09:36.676 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-03-11T21:09:36.676 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T21:09:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet zip 2024-03-11T21:09:36.678 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-03-11T21:09:36.679 INFO:teuthology.orchestra.run.smithi203.stdout: btrfs-tools 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-11T21:09:36.716 INFO:teuthology.orchestra.run.smithi143.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-11T21:09:36.717 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet zip 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-03-11T21:09:36.718 INFO:teuthology.orchestra.run.smithi143.stdout: btrfs-tools 2024-03-11T21:09:36.779 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-03-11T21:09:36.779 INFO:teuthology.orchestra.run.smithi203.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-11T21:09:36.779 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-11T21:09:36.779 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-11T21:09:36.779 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T21:09:36.780 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-11T21:09:36.781 INFO:teuthology.orchestra.run.smithi203.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-11T21:09:36.814 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-03-11T21:09:36.814 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-11T21:09:36.814 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-11T21:09:36.814 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-11T21:09:36.815 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-11T21:09:36.815 INFO:teuthology.orchestra.run.smithi143.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-11T21:09:36.815 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T21:09:36.816 INFO:teuthology.orchestra.run.smithi143.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-11T21:09:36.817 INFO:teuthology.orchestra.run.smithi143.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-11T21:09:36.923 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-11T21:09:36.923 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 176 MB of archives. 2024-03-11T21:09:36.924 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 731 MB of additional disk space will be used. 2024-03-11T21:09:36.924 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g2b0101eb-1jammy [3,546 kB] 2024-03-11T21:09:36.967 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-11T21:09:37.002 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-11T21:09:37.002 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 176 MB of archives. 2024-03-11T21:09:37.003 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 731 MB of additional disk space will be used. 2024-03-11T21:09:37.003 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-11T21:09:37.059 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g2b0101eb-1jammy [3,546 kB] 2024-03-11T21:09:37.388 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-11T21:09:37.400 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-11T21:09:37.435 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-11T21:09:37.447 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-11T21:09:37.482 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-11T21:09:37.486 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g2b0101eb-1jammy [3,220 kB] 2024-03-11T21:09:37.529 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-11T21:09:37.694 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g2b0101eb-1jammy [711 kB] 2024-03-11T21:09:37.714 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-11T21:09:37.721 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g2b0101eb-1jammy [356 kB] 2024-03-11T21:09:37.725 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-11T21:09:37.742 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g2b0101eb-1jammy [32.8 kB] 2024-03-11T21:09:37.742 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g2b0101eb-1jammy [180 kB] 2024-03-11T21:09:37.755 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g2b0101eb-1jammy [64.6 kB] 2024-03-11T21:09:37.755 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-11T21:09:37.758 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g2b0101eb-1jammy [330 kB] 2024-03-11T21:09:37.760 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-11T21:09:37.764 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-11T21:09:37.764 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g2b0101eb-1jammy [6,613 kB] 2024-03-11T21:09:37.765 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-11T21:09:37.766 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-11T21:09:37.771 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-11T21:09:37.785 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-11T21:09:37.801 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-11T21:09:37.811 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-11T21:09:37.811 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-11T21:09:37.813 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-11T21:09:37.831 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-11T21:09:37.953 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-11T21:09:37.999 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-11T21:09:38.005 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g2b0101eb-1jammy [3,220 kB] 2024-03-11T21:09:38.027 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g2b0101eb-1jammy [112 kB] 2024-03-11T21:09:38.030 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g2b0101eb-1jammy [467 kB] 2024-03-11T21:09:38.053 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g2b0101eb-1jammy [24.2 MB] 2024-03-11T21:09:38.355 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-11T21:09:38.390 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-11T21:09:38.392 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-11T21:09:38.408 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-11T21:09:38.442 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-11T21:09:38.444 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-11T21:09:38.471 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-11T21:09:38.523 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-11T21:09:38.524 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-11T21:09:38.544 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-11T21:09:38.551 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-11T21:09:38.554 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-11T21:09:38.567 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-11T21:09:38.568 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-11T21:09:38.575 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-11T21:09:38.597 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-11T21:09:38.604 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-11T21:09:38.604 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-11T21:09:38.611 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g2b0101eb-1jammy [711 kB] 2024-03-11T21:09:38.616 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-11T21:09:38.617 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-11T21:09:38.733 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-11T21:09:38.749 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g2b0101eb-1jammy [356 kB] 2024-03-11T21:09:38.783 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-11T21:09:38.809 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g2b0101eb-1jammy [32.8 kB] 2024-03-11T21:09:38.817 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-11T21:09:38.823 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g2b0101eb-1jammy [180 kB] 2024-03-11T21:09:38.846 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-11T21:09:38.866 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g2b0101eb-1jammy [64.6 kB] 2024-03-11T21:09:38.866 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-11T21:09:38.867 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-11T21:09:38.877 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g2b0101eb-1jammy [330 kB] 2024-03-11T21:09:38.894 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-11T21:09:38.907 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-11T21:09:38.911 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-11T21:09:38.958 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-11T21:09:38.962 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g2b0101eb-1jammy [6,613 kB] 2024-03-11T21:09:39.078 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g2b0101eb-1jammy [5,081 kB] 2024-03-11T21:09:39.119 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-11T21:09:39.135 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-11T21:09:39.142 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-11T21:09:39.147 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-11T21:09:39.170 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-11T21:09:39.170 INFO:teuthology.orchestra.run.smithi203.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-11T21:09:39.172 INFO:teuthology.orchestra.run.smithi203.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-11T21:09:39.184 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-11T21:09:39.192 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-11T21:09:39.197 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-11T21:09:39.218 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-11T21:09:39.222 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-11T21:09:39.275 INFO:teuthology.orchestra.run.smithi203.stdout:Get:47 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g2b0101eb-1jammy [243 kB] 2024-03-11T21:09:39.279 INFO:teuthology.orchestra.run.smithi203.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g2b0101eb-1jammy [124 kB] 2024-03-11T21:09:39.282 INFO:teuthology.orchestra.run.smithi203.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g2b0101eb-1jammy [1,522 kB] 2024-03-11T21:09:39.314 INFO:teuthology.orchestra.run.smithi203.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g2b0101eb-1jammy [6,247 kB] 2024-03-11T21:09:39.338 INFO:teuthology.orchestra.run.smithi203.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-11T21:09:39.387 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-11T21:09:39.538 INFO:teuthology.orchestra.run.smithi203.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g2b0101eb-1jammy [22.8 MB] 2024-03-11T21:09:39.655 INFO:teuthology.orchestra.run.smithi203.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-11T21:09:39.693 INFO:teuthology.orchestra.run.smithi203.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-11T21:09:39.704 INFO:teuthology.orchestra.run.smithi203.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-11T21:09:39.717 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-11T21:09:39.757 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-11T21:09:39.770 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-11T21:09:39.909 INFO:teuthology.orchestra.run.smithi203.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-11T21:09:39.910 INFO:teuthology.orchestra.run.smithi203.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-11T21:09:39.916 INFO:teuthology.orchestra.run.smithi203.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-11T21:09:39.921 INFO:teuthology.orchestra.run.smithi203.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-11T21:09:39.936 INFO:teuthology.orchestra.run.smithi203.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-11T21:09:39.958 INFO:teuthology.orchestra.run.smithi203.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-11T21:09:39.963 INFO:teuthology.orchestra.run.smithi203.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-11T21:09:39.980 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-11T21:09:39.980 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-11T21:09:39.986 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-11T21:09:39.992 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-11T21:09:40.006 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-11T21:09:40.027 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-11T21:09:40.033 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-11T21:09:40.082 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g2b0101eb-1jammy [112 kB] 2024-03-11T21:09:40.101 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g2b0101eb-1jammy [467 kB] 2024-03-11T21:09:40.139 INFO:teuthology.orchestra.run.smithi203.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-11T21:09:40.175 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g2b0101eb-1jammy [24.2 MB] 2024-03-11T21:09:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-11T21:09:40.404 INFO:teuthology.orchestra.run.smithi203.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-11T21:09:40.462 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-11T21:09:40.559 INFO:teuthology.orchestra.run.smithi203.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g2b0101eb-1jammy [14.0 kB] 2024-03-11T21:09:40.560 INFO:teuthology.orchestra.run.smithi203.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [867 kB] 2024-03-11T21:09:40.578 INFO:teuthology.orchestra.run.smithi203.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g2b0101eb-1jammy [2,192 kB] 2024-03-11T21:09:40.582 INFO:teuthology.orchestra.run.smithi203.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-11T21:09:40.597 INFO:teuthology.orchestra.run.smithi203.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-11T21:09:40.616 INFO:teuthology.orchestra.run.smithi203.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-11T21:09:40.617 INFO:teuthology.orchestra.run.smithi203.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g2b0101eb-1jammy [777 kB] 2024-03-11T21:09:40.629 INFO:teuthology.orchestra.run.smithi203.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g2b0101eb-1jammy [145 kB] 2024-03-11T21:09:40.639 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-11T21:09:40.639 INFO:teuthology.orchestra.run.smithi203.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g2b0101eb-1jammy [1,934 kB] 2024-03-11T21:09:40.652 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-11T21:09:40.673 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-11T21:09:40.681 INFO:teuthology.orchestra.run.smithi203.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-11T21:09:40.718 INFO:teuthology.orchestra.run.smithi203.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g2b0101eb-1jammy [8,625 kB] 2024-03-11T21:09:40.737 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-11T21:09:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-11T21:09:40.978 INFO:teuthology.orchestra.run.smithi203.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-11T21:09:40.981 INFO:teuthology.orchestra.run.smithi203.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-11T21:09:40.981 INFO:teuthology.orchestra.run.smithi203.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-11T21:09:40.984 INFO:teuthology.orchestra.run.smithi203.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-11T21:09:41.031 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-11T21:09:41.032 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-11T21:09:41.035 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-11T21:09:41.035 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-11T21:09:41.038 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-11T21:09:41.073 INFO:teuthology.orchestra.run.smithi203.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g2b0101eb-1jammy [14.2 kB] 2024-03-11T21:09:41.074 INFO:teuthology.orchestra.run.smithi203.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g2b0101eb-1jammy [50.0 MB] 2024-03-11T21:09:41.211 INFO:teuthology.orchestra.run.smithi203.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-11T21:09:41.254 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-11T21:09:41.641 INFO:teuthology.orchestra.run.smithi203.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-11T21:09:41.646 INFO:teuthology.orchestra.run.smithi203.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-11T21:09:41.693 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-11T21:09:41.699 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-11T21:09:41.936 INFO:teuthology.orchestra.run.smithi203.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-11T21:09:41.937 INFO:teuthology.orchestra.run.smithi203.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-11T21:09:41.940 INFO:teuthology.orchestra.run.smithi203.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-11T21:09:41.941 INFO:teuthology.orchestra.run.smithi203.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-11T21:09:41.943 INFO:teuthology.orchestra.run.smithi203.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-11T21:09:41.945 INFO:teuthology.orchestra.run.smithi203.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-11T21:09:41.946 INFO:teuthology.orchestra.run.smithi203.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-11T21:09:41.948 INFO:teuthology.orchestra.run.smithi203.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-11T21:09:41.990 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-11T21:09:41.991 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-11T21:09:41.993 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-11T21:09:41.995 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-11T21:09:41.996 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-11T21:09:41.998 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-11T21:09:42.000 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-11T21:09:42.001 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-11T21:09:42.128 INFO:teuthology.orchestra.run.smithi203.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-11T21:09:42.182 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-11T21:09:42.542 INFO:teuthology.orchestra.run.smithi203.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-11T21:09:42.587 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-11T21:09:42.587 INFO:teuthology.orchestra.run.smithi203.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-11T21:09:42.623 INFO:teuthology.orchestra.run.smithi203.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-11T21:09:42.633 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-11T21:09:42.652 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-11T21:09:42.719 INFO:teuthology.orchestra.run.smithi203.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-11T21:09:42.759 INFO:teuthology.orchestra.run.smithi203.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-11T21:09:42.764 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-11T21:09:42.771 INFO:teuthology.orchestra.run.smithi203.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-11T21:09:42.776 INFO:teuthology.orchestra.run.smithi203.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-11T21:09:42.804 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-11T21:09:42.816 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-11T21:09:42.821 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-11T21:09:42.823 INFO:teuthology.orchestra.run.smithi203.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-11T21:09:42.826 INFO:teuthology.orchestra.run.smithi203.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-11T21:09:42.849 INFO:teuthology.orchestra.run.smithi203.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-11T21:09:42.868 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-11T21:09:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-11T21:09:42.893 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-11T21:09:43.012 INFO:teuthology.orchestra.run.smithi203.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-11T21:09:43.057 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-11T21:09:43.269 INFO:teuthology.orchestra.run.smithi203.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g2b0101eb-1jammy [121 kB] 2024-03-11T21:09:43.272 INFO:teuthology.orchestra.run.smithi203.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g2b0101eb-1jammy [38.4 kB] 2024-03-11T21:09:43.274 INFO:teuthology.orchestra.run.smithi203.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g2b0101eb-1jammy [13.1 MB] 2024-03-11T21:09:43.331 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-11T21:09:43.348 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-11T21:09:43.352 INFO:teuthology.orchestra.run.smithi203.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-11T21:09:43.477 INFO:teuthology.orchestra.run.smithi203.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-11T21:09:43.539 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-11T21:09:43.568 INFO:teuthology.orchestra.run.smithi203.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-11T21:09:43.641 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-11T21:09:43.642 INFO:teuthology.orchestra.run.smithi203.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [91.7 kB] 2024-03-11T21:09:43.701 INFO:teuthology.orchestra.run.smithi203.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-11T21:09:43.822 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-11T21:09:43.825 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-11T21:09:43.827 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-11T21:09:43.840 INFO:teuthology.orchestra.run.smithi203.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-11T21:09:43.844 INFO:teuthology.orchestra.run.smithi203.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-11T21:09:43.853 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-11T21:09:43.853 INFO:teuthology.orchestra.run.smithi203.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-11T21:09:43.858 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-11T21:09:43.882 INFO:teuthology.orchestra.run.smithi203.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-11T21:09:43.888 INFO:teuthology.orchestra.run.smithi203.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-11T21:09:44.174 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g2b0101eb-1jammy [5,081 kB] 2024-03-11T21:09:44.581 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 176 MB in 7s (24.4 MB/s) 2024-03-11T21:09:44.584 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g2b0101eb-1jammy [243 kB] 2024-03-11T21:09:44.592 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g2b0101eb-1jammy [124 kB] 2024-03-11T21:09:44.616 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g2b0101eb-1jammy [1,522 kB] 2024-03-11T21:09:44.734 INFO:teuthology.orchestra.run.smithi143.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g2b0101eb-1jammy [6,247 kB] 2024-03-11T21:09:44.749 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-11T21:09:44.850 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-11T21:09:44.855 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-11T21:09:44.940 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T21:09:45.251 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-11T21:09:45.270 INFO:teuthology.orchestra.run.smithi143.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g2b0101eb-1jammy [22.8 MB] 2024-03-11T21:09:45.278 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-11T21:09:45.318 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T21:09:45.646 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-11T21:09:45.673 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-11T21:09:45.712 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T21:09:46.149 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-11T21:09:46.176 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:46.224 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:46.603 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-11T21:09:46.629 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:46.669 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:47.005 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-11T21:09:47.032 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:47.072 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:47.425 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-11T21:09:47.451 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-11T21:09:47.491 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T21:09:47.579 INFO:teuthology.orchestra.run.smithi143.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g2b0101eb-1jammy [14.0 kB] 2024-03-11T21:09:47.580 INFO:teuthology.orchestra.run.smithi143.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [867 kB] 2024-03-11T21:09:47.596 INFO:teuthology.orchestra.run.smithi143.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g2b0101eb-1jammy [2,192 kB] 2024-03-11T21:09:47.769 INFO:teuthology.orchestra.run.smithi143.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g2b0101eb-1jammy [777 kB] 2024-03-11T21:09:47.786 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librados2. 2024-03-11T21:09:47.801 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:47.838 INFO:teuthology.orchestra.run.smithi143.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g2b0101eb-1jammy [145 kB] 2024-03-11T21:09:47.841 INFO:teuthology.orchestra.run.smithi143.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g2b0101eb-1jammy [1,934 kB] 2024-03-11T21:09:47.844 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:48.033 INFO:teuthology.orchestra.run.smithi143.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g2b0101eb-1jammy [8,625 kB] 2024-03-11T21:09:48.306 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librbd1. 2024-03-11T21:09:48.332 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:48.372 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:48.816 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs2. 2024-03-11T21:09:48.842 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:48.883 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:48.949 INFO:teuthology.orchestra.run.smithi143.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g2b0101eb-1jammy [14.2 kB] 2024-03-11T21:09:48.950 INFO:teuthology.orchestra.run.smithi143.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g2b0101eb-1jammy [50.0 MB] 2024-03-11T21:09:49.186 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rados. 2024-03-11T21:09:49.212 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:49.252 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:49.705 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-11T21:09:49.731 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:09:49.771 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:50.024 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cephfs. 2024-03-11T21:09:50.038 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:50.082 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:50.351 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-11T21:09:50.378 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:09:50.418 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:50.763 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-11T21:09:50.790 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-11T21:09:50.829 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T21:09:51.149 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-prettytable. 2024-03-11T21:09:51.176 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-11T21:09:51.215 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-11T21:09:51.443 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rbd. 2024-03-11T21:09:51.469 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:51.509 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:51.829 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-11T21:09:51.856 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-11T21:09:51.896 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T21:09:52.140 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-11T21:09:52.166 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-11T21:09:52.206 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T21:09:52.484 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-11T21:09:52.500 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-11T21:09:52.534 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T21:09:52.872 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua5.1. 2024-03-11T21:09:52.897 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-11T21:09:52.938 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-11T21:09:53.249 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-any. 2024-03-11T21:09:53.278 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-11T21:09:53.315 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-11T21:09:53.542 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package zip. 2024-03-11T21:09:53.558 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-11T21:09:53.601 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking zip (3.0-12build2) ... 2024-03-11T21:09:53.810 INFO:teuthology.orchestra.run.smithi143.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g2b0101eb-1jammy [121 kB] 2024-03-11T21:09:53.888 INFO:teuthology.orchestra.run.smithi143.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g2b0101eb-1jammy [38.4 kB] 2024-03-11T21:09:53.889 INFO:teuthology.orchestra.run.smithi143.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g2b0101eb-1jammy [13.1 MB] 2024-03-11T21:09:53.946 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package unzip. 2024-03-11T21:09:53.973 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-11T21:09:54.012 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-11T21:09:54.408 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package luarocks. 2024-03-11T21:09:54.435 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-11T21:09:54.474 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-11T21:09:54.787 INFO:teuthology.orchestra.run.smithi143.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [91.7 kB] 2024-03-11T21:09:54.860 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librgw2. 2024-03-11T21:09:54.887 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:54.926 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:55.348 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 176 MB in 18s (9,814 kB/s) 2024-03-11T21:09:55.487 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rgw. 2024-03-11T21:09:55.506 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-11T21:09:55.514 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:55.562 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:55.601 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-11T21:09:55.606 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-11T21:09:55.696 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T21:09:55.866 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-11T21:09:55.893 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-11T21:09:55.924 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T21:09:56.041 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-11T21:09:56.068 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-11T21:09:56.108 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T21:09:56.235 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libradosstriper1. 2024-03-11T21:09:56.262 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:56.302 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:56.435 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-11T21:09:56.462 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-11T21:09:56.502 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T21:09:56.730 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-common. 2024-03-11T21:09:56.758 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:56.797 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:56.913 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-11T21:09:56.940 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:56.980 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:57.375 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-11T21:09:57.401 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:57.453 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-11T21:09:57.838 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T21:09:57.877 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:09:58.069 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-base. 2024-03-11T21:09:58.096 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:58.145 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:58.238 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-11T21:09:58.265 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-11T21:09:58.297 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T21:09:58.616 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librados2. 2024-03-11T21:09:58.643 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:58.682 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:58.699 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-11T21:09:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-11T21:09:58.790 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-11T21:09:59.093 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cheroot. 2024-03-11T21:09:59.121 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-11T21:09:59.135 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librbd1. 2024-03-11T21:09:59.160 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T21:09:59.162 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:59.201 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:09:59.480 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-11T21:09:59.507 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-11T21:09:59.547 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-11T21:09:59.671 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2024-03-11T21:09:59.698 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:09:59.737 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:00.033 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-11T21:10:00.060 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-11T21:10:00.100 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-11T21:10:00.123 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2024-03-11T21:10:00.150 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:00.240 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:00.394 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-11T21:10:00.421 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-11T21:10:00.461 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-11T21:10:00.542 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-11T21:10:00.568 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:00.609 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:00.773 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempora. 2024-03-11T21:10:00.800 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-11T21:10:00.842 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-11T21:10:00.861 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2024-03-11T21:10:00.887 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:00.927 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-portend. 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-11T21:10:01.251 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-11T21:10:01.273 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:01.313 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:01.470 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-11T21:10:01.497 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-11T21:10:01.528 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-11T21:10:01.619 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-11T21:10:01.643 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-11T21:10:01.683 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T21:10:01.832 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-11T21:10:01.859 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-11T21:10:01.898 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-11T21:10:01.977 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2024-03-11T21:10:02.003 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-11T21:10:02.043 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-11T21:10:02.252 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-natsort. 2024-03-11T21:10:02.279 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-11T21:10:02.298 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2024-03-11T21:10:02.319 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-11T21:10:02.323 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:02.363 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:02.639 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-logutils. 2024-03-11T21:10:02.666 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-11T21:10:02.705 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-11T21:10:02.715 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-11T21:10:02.742 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-11T21:10:02.782 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T21:10:03.000 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-11T21:10:03.027 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-11T21:10:03.067 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-11T21:10:03.086 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-11T21:10:03.111 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-11T21:10:03.151 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T21:10:03.420 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-mako. 2024-03-11T21:10:03.447 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-11T21:10:03.471 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-11T21:10:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T21:10:03.497 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-11T21:10:03.537 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T21:10:03.840 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-11T21:10:03.867 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-11T21:10:03.907 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-11T21:10:03.923 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package lua5.1. 2024-03-11T21:10:03.950 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-11T21:10:03.998 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-11T21:10:04.177 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-11T21:10:04.204 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-11T21:10:04.249 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-11T21:10:04.326 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package lua-any. 2024-03-11T21:10:04.342 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-11T21:10:04.385 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-11T21:10:04.529 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webob. 2024-03-11T21:10:04.556 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-11T21:10:04.595 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-11T21:10:04.612 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package zip. 2024-03-11T21:10:04.638 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-11T21:10:04.680 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking zip (3.0-12build2) ... 2024-03-11T21:10:04.958 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-waitress. 2024-03-11T21:10:04.984 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-11T21:10:05.015 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package unzip. 2024-03-11T21:10:05.028 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T21:10:05.041 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-11T21:10:05.346 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempita. 2024-03-11T21:10:05.370 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-11T21:10:05.410 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T21:10:05.610 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package luarocks. 2024-03-11T21:10:05.636 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-11T21:10:05.680 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-paste. 2024-03-11T21:10:05.684 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-11T21:10:05.707 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-11T21:10:05.747 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T21:10:06.050 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-11T21:10:06.066 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-11T21:10:06.084 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2024-03-11T21:10:06.100 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T21:10:06.110 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:06.150 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:06.378 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-11T21:10:06.405 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-11T21:10:06.445 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-11T21:10:06.686 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2024-03-11T21:10:06.713 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:06.748 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webtest. 2024-03-11T21:10:06.752 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:06.775 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-11T21:10:06.815 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-11T21:10:07.126 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pecan. 2024-03-11T21:10:07.138 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-11T21:10:07.153 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-11T21:10:07.166 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-11T21:10:07.193 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T21:10:07.230 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T21:10:07.573 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-certifi. 2024-03-11T21:10:07.602 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-11T21:10:07.608 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2024-03-11T21:10:07.634 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:07.639 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-11T21:10:07.682 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:07.934 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-idna. 2024-03-11T21:10:07.961 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-11T21:10:08.001 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-11T21:10:08.102 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2024-03-11T21:10:08.129 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:08.169 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:08.312 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-urllib3. 2024-03-11T21:10:08.339 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-11T21:10:08.379 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T21:10:08.724 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests. 2024-03-11T21:10:08.751 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-11T21:10:08.790 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T21:10:09.102 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-11T21:10:09.129 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-11T21:10:09.169 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T21:10:09.465 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2024-03-11T21:10:09.472 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-11T21:10:09.480 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:09.499 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:09.539 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:09.541 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:09.927 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-11T21:10:09.954 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:09.993 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:11.020 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-11T21:10:11.047 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-11T21:10:11.095 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-11T21:10:11.247 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-11T21:10:11.274 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-11T21:10:11.314 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-11T21:10:11.399 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cheroot. 2024-03-11T21:10:11.424 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-11T21:10:11.464 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T21:10:11.592 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-openssl. 2024-03-11T21:10:11.618 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-11T21:10:11.658 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-11T21:10:11.783 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-11T21:10:11.800 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-11T21:10:11.833 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-11T21:10:11.995 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr. 2024-03-11T21:10:12.022 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:12.061 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:12.111 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-11T21:10:12.137 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-11T21:10:12.177 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-11T21:10:12.457 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mon. 2024-03-11T21:10:12.471 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-11T21:10:12.484 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:12.498 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-11T21:10:12.523 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:12.538 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-11T21:10:12.857 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempora. 2024-03-11T21:10:12.884 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-11T21:10:12.925 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-11T21:10:13.135 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-osd. 2024-03-11T21:10:13.162 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:13.203 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:13.251 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-portend. 2024-03-11T21:10:13.278 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-11T21:10:13.326 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-11T21:10:13.712 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-11T21:10:13.739 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-11T21:10:13.787 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-11T21:10:14.148 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-11T21:10:14.174 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-11T21:10:14.215 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-11T21:10:14.281 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph. 2024-03-11T21:10:14.296 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:14.339 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:14.527 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-natsort. 2024-03-11T21:10:14.553 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-11T21:10:14.585 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-11T21:10:14.660 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-fuse. 2024-03-11T21:10:14.685 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:14.725 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:14.913 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2024-03-11T21:10:14.940 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-11T21:10:14.980 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-11T21:10:15.078 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mds. 2024-03-11T21:10:15.105 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:15.145 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:15.282 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-11T21:10:15.309 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-11T21:10:15.349 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-11T21:10:15.548 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package cephadm. 2024-03-11T21:10:15.575 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:15.615 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:15.651 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2024-03-11T21:10:15.678 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-11T21:10:15.717 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T21:10:15.934 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-11T21:10:15.961 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-11T21:10:16.001 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-11T21:10:16.071 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-11T21:10:16.098 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-11T21:10:16.146 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-11T21:10:16.331 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-11T21:10:16.355 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T21:10:16.382 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-11T21:10:16.395 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:16.397 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-11T21:10:16.432 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-11T21:10:16.726 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2024-03-11T21:10:16.753 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-11T21:10:16.792 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-11T21:10:17.170 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2024-03-11T21:10:17.197 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-11T21:10:17.240 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T21:10:17.426 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-babel. 2024-03-11T21:10:17.453 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T21:10:17.492 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:17.548 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2024-03-11T21:10:17.575 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-11T21:10:17.614 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T21:10:17.795 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jinja2. 2024-03-11T21:10:17.811 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-11T21:10:17.845 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T21:10:17.875 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2024-03-11T21:10:17.902 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-11T21:10:17.942 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T21:10:18.224 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-11T21:10:18.253 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:18.261 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-11T21:10:18.288 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-11T21:10:18.290 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:18.319 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T21:10:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-11T21:10:18.627 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-11T21:10:18.633 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-11T21:10:18.654 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-11T21:10:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-11T21:10:18.693 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-11T21:10:18.975 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2024-03-11T21:10:18.997 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-routes. 2024-03-11T21:10:19.002 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-11T21:10:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-11T21:10:19.042 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-11T21:10:19.063 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T21:10:19.369 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2024-03-11T21:10:19.396 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-11T21:10:19.400 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-11T21:10:19.427 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:19.436 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T21:10:19.466 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:19.799 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-certifi. 2024-03-11T21:10:19.826 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-11T21:10:19.865 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-11T21:10:19.895 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-11T21:10:19.910 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-11T21:10:19.953 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T21:10:20.193 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-idna. 2024-03-11T21:10:20.220 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-11T21:10:20.276 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-11T21:10:20.432 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-joblib. 2024-03-11T21:10:20.459 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-11T21:10:20.498 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T21:10:20.587 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-urllib3. 2024-03-11T21:10:20.614 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-11T21:10:20.645 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T21:10:20.923 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests. 2024-03-11T21:10:20.939 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-11T21:10:20.973 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T21:10:21.027 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-11T21:10:21.054 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-11T21:10:21.094 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-11T21:10:21.281 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-11T21:10:21.308 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-11T21:10:21.380 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn. 2024-03-11T21:10:21.407 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-11T21:10:21.425 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T21:10:21.438 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T21:10:21.737 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-11T21:10:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:21.812 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:22.123 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-11T21:10:22.150 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:22.175 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-11T21:10:22.202 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:22.249 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:22.266 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:22.569 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-11T21:10:22.596 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-11T21:10:22.635 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-11T21:10:22.938 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2024-03-11T21:10:22.962 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-11T21:10:22.996 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-11T21:10:23.278 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cachetools. 2024-03-11T21:10:23.306 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-11T21:10:23.316 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2024-03-11T21:10:23.343 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:23.345 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-11T21:10:23.382 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:23.640 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-11T21:10:23.667 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-11T21:10:23.706 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-11T21:10:23.785 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2024-03-11T21:10:23.812 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:23.851 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:23.985 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-11T21:10:24.001 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-11T21:10:24.044 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T21:10:24.330 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rsa. 2024-03-11T21:10:24.357 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-11T21:10:24.396 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-11T21:10:24.451 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2024-03-11T21:10:24.476 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:24.507 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:24.726 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-google-auth. 2024-03-11T21:10:24.753 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-11T21:10:24.792 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-11T21:10:25.120 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-11T21:10:25.149 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-11T21:10:25.187 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T21:10:25.500 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-websocket. 2024-03-11T21:10:25.527 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-11T21:10:25.561 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2024-03-11T21:10:25.565 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-11T21:10:25.588 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:25.627 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:25.860 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-11T21:10:25.888 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-11T21:10:25.927 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T21:10:25.946 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2024-03-11T21:10:25.974 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:26.013 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:26.367 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2024-03-11T21:10:26.394 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:26.425 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:26.658 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-11T21:10:26.686 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:26.724 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:26.803 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2024-03-11T21:10:26.830 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:26.870 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:27.044 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-11T21:10:27.069 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-11T21:10:27.111 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T21:10:27.198 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-11T21:10:27.225 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-11T21:10:27.264 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-11T21:10:27.406 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-11T21:10:27.433 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T21:10:27.481 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T21:10:27.575 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-11T21:10:27.591 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T21:10:27.634 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:27.733 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package jq. 2024-03-11T21:10:27.749 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T21:10:27.792 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-11T21:10:28.069 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package socat. 2024-03-11T21:10:28.085 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-11T21:10:28.128 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-11T21:10:28.423 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package xmlstarlet. 2024-03-11T21:10:28.439 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-11T21:10:28.473 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-11T21:10:28.656 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-babel. 2024-03-11T21:10:28.683 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T21:10:28.731 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:28.819 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-test. 2024-03-11T21:10:28.847 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:28.886 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:29.059 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2024-03-11T21:10:29.086 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-11T21:10:29.125 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T21:10:29.437 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-11T21:10:29.464 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:29.495 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:29.807 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-11T21:10:29.834 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-11T21:10:29.865 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-11T21:10:30.160 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2024-03-11T21:10:30.187 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-11T21:10:30.226 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T21:10:30.513 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-11T21:10:30.529 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:30.571 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:31.025 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-11T21:10:31.041 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-11T21:10:31.083 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T21:10:31.432 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-volume. 2024-03-11T21:10:31.460 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:31.490 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:31.778 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-11T21:10:31.793 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:31.827 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:31.912 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2024-03-11T21:10:31.940 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-11T21:10:31.970 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T21:10:32.139 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-11T21:10:32.166 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-11T21:10:32.207 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T21:10:32.366 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-11T21:10:32.393 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-11T21:10:32.432 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-11T21:10:32.512 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-11T21:10:32.540 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-11T21:10:32.579 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-11T21:10:32.743 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2024-03-11T21:10:32.770 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-11T21:10:32.810 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T21:10:32.942 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package nvme-cli. 2024-03-11T21:10:32.969 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-11T21:10:33.008 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T21:10:33.405 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package pkg-config. 2024-03-11T21:10:33.434 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-11T21:10:33.471 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T21:10:33.553 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-11T21:10:33.581 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:33.713 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:33.733 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-11T21:10:33.761 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-11T21:10:33.800 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-11T21:10:34.237 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-11T21:10:34.264 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-11T21:10:34.303 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-11T21:10:34.606 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastescript. 2024-03-11T21:10:34.633 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-11T21:10:34.651 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cachetools. 2024-03-11T21:10:34.666 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-11T21:10:34.679 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-11T21:10:34.717 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-11T21:10:34.960 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pluggy. 2024-03-11T21:10:34.987 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-11T21:10:35.026 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-11T21:10:35.028 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-11T21:10:35.056 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-11T21:10:35.095 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-11T21:10:35.338 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-psutil. 2024-03-11T21:10:35.365 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-11T21:10:35.404 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-11T21:10:35.424 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-11T21:10:35.451 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-11T21:10:35.490 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T21:10:35.715 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-py. 2024-03-11T21:10:35.743 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-11T21:10:35.782 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-11T21:10:35.801 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2024-03-11T21:10:35.830 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-11T21:10:35.868 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-11T21:10:36.119 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pygments. 2024-03-11T21:10:36.147 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-11T21:10:36.172 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-google-auth. 2024-03-11T21:10:36.187 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T21:10:36.200 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-11T21:10:36.238 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-11T21:10:36.559 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-11T21:10:36.587 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-11T21:10:36.657 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-11T21:10:36.685 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-11T21:10:36.723 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-11T21:10:36.729 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T21:10:37.037 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2024-03-11T21:10:37.043 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-toml. 2024-03-11T21:10:37.054 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-11T21:10:37.071 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-11T21:10:37.095 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-11T21:10:37.109 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-11T21:10:37.391 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-11T21:10:37.412 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pytest. 2024-03-11T21:10:37.418 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-11T21:10:37.440 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-11T21:10:37.457 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T21:10:37.479 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T21:10:37.808 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplejson. 2024-03-11T21:10:37.824 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-11T21:10:37.865 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-11T21:10:38.143 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-11T21:10:38.160 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-11T21:10:38.162 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-11T21:10:38.179 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:38.202 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-11T21:10:38.221 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:38.559 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-11T21:10:38.585 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-11T21:10:38.624 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T21:10:38.898 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package radosgw. 2024-03-11T21:10:38.911 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-11T21:10:38.926 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:38.938 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T21:10:38.964 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:38.969 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T21:10:39.205 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2024-03-11T21:10:39.221 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T21:10:39.263 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-11T21:10:39.516 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2024-03-11T21:10:39.544 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-11T21:10:39.586 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-11T21:10:39.793 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package rbd-fuse. 2024-03-11T21:10:39.821 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:39.859 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:39.894 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2024-03-11T21:10:39.910 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-11T21:10:39.953 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-11T21:10:40.138 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-11T21:10:40.324 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2024-03-11T21:10:40.352 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:40.391 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:40.397 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T21:10:40.524 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T21:10:40.784 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-11T21:10:41.116 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-11T21:10:41.524 INFO:teuthology.orchestra.run.smithi203.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-11T21:10:41.801 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:42.309 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user cephadm....done 2024-03-11T21:10:42.457 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T21:10:42.759 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-11T21:10:42.770 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-volume. 2024-03-11T21:10:42.798 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T21:10:42.837 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:43.018 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-11T21:10:43.133 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-11T21:10:43.136 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-11T21:10:43.149 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:43.191 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:43.388 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-11T21:10:43.544 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-11T21:10:43.572 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-11T21:10:43.619 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T21:10:43.647 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T21:10:43.773 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-11T21:10:44.007 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-11T21:10:44.035 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-11T21:10:44.074 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-11T21:10:44.083 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T21:10:44.436 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-11T21:10:44.445 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2024-03-11T21:10:44.472 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-11T21:10:44.511 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T21:10:44.695 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-11T21:10:44.934 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package pkg-config. 2024-03-11T21:10:44.961 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-11T21:10:44.971 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-11T21:10:45.000 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T21:10:45.262 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-11T21:10:45.282 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-11T21:10:45.289 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-11T21:10:45.328 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-11T21:10:45.532 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-11T21:10:45.748 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-11T21:10:45.775 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-11T21:10:45.809 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-11T21:10:45.814 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-11T21:10:46.093 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2024-03-11T21:10:46.120 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-11T21:10:46.146 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:46.159 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-11T21:10:46.386 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T21:10:46.504 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T21:10:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2024-03-11T21:10:46.532 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-11T21:10:46.571 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-11T21:10:46.638 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T21:10:46.764 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-11T21:10:46.882 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-psutil. 2024-03-11T21:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-11T21:10:46.949 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-11T21:10:47.024 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-11T21:10:47.285 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2024-03-11T21:10:47.313 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-11T21:10:47.352 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-11T21:10:47.467 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-11T21:10:47.531 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-11T21:10:47.586 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T21:10:47.680 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pygments. 2024-03-11T21:10:47.695 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-11T21:10:47.709 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-11T21:10:47.747 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T21:10:48.055 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:48.181 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-11T21:10:48.194 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-11T21:10:48.221 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-11T21:10:48.260 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-11T21:10:48.449 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T21:10:48.571 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-toml. 2024-03-11T21:10:48.601 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-11T21:10:48.637 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-11T21:10:48.717 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-11T21:10:48.949 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2024-03-11T21:10:48.977 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-11T21:10:49.001 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up zip (3.0-12build2) ... 2024-03-11T21:10:49.015 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T21:10:49.119 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T21:10:49.361 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2024-03-11T21:10:49.389 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-11T21:10:49.427 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-11T21:10:49.705 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-11T21:10:49.732 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-11T21:10:49.746 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T21:10:49.771 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-11T21:10:49.997 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T21:10:50.124 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-11T21:10:50.250 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T21:10:50.501 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2024-03-11T21:10:50.529 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:50.534 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-11T21:10:50.567 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:50.911 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-11T21:10:51.162 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T21:10:51.354 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2024-03-11T21:10:51.382 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T21:10:51.429 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T21:10:51.758 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-11T21:10:51.831 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T21:10:52.018 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T21:10:52.135 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T21:10:52.166 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-11T21:10:52.377 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-11T21:10:52.401 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-11T21:10:52.502 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:52.738 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-11T21:10:52.787 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-idna (3.3-1) ... 2024-03-11T21:10:53.104 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T21:10:53.175 INFO:teuthology.orchestra.run.smithi143.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-11T21:10:53.254 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:53.796 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2024-03-11T21:10:53.927 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T21:10:54.178 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-11T21:10:54.311 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T21:10:54.429 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-11T21:10:54.554 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:10:54.555 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-11T21:10:54.688 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-11T21:10:54.824 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-11T21:10:54.948 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T21:10:55.082 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-11T21:10:55.100 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T21:10:55.192 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T21:10:55.209 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-11T21:10:55.468 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-11T21:10:55.495 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T21:10:55.720 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-11T21:10:55.838 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-11T21:10:55.971 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:10:56.097 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-11T21:10:56.106 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-11T21:10:56.374 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-11T21:10:56.415 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-11T21:10:56.651 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-11T21:10:56.683 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-11T21:10:56.885 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-11T21:10:57.128 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-11T21:10:57.184 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-11T21:10:57.379 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:10:57.444 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-11T21:10:57.678 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-11T21:10:57.825 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T21:10:57.954 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T21:10:58.080 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-11T21:10:58.224 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T21:10:58.350 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T21:10:58.365 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T21:10:58.467 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-11T21:10:58.491 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T21:10:58.718 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-11T21:10:58.776 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-11T21:10:59.052 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-11T21:10:59.081 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-11T21:10:59.125 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-11T21:10:59.189 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T21:10:59.303 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T21:10:59.323 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-11T21:10:59.429 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-11T21:10:59.555 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:59.644 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-11T21:10:59.708 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T21:10:59.834 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-11T21:10:59.874 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T21:11:00.008 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T21:11:00.102 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T21:11:00.344 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-11T21:11:00.410 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-11T21:11:00.646 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up zip (3.0-12build2) ... 2024-03-11T21:11:00.653 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-11T21:11:00.772 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T21:11:00.779 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-11T21:11:01.047 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:11:01.173 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T21:11:01.441 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T21:11:01.499 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T21:11:01.700 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T21:11:01.809 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T21:11:01.827 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-11T21:11:01.953 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T21:11:02.068 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-11T21:11:02.186 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-11T21:11:02.287 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-11T21:11:02.462 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-11T21:11:02.680 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-11T21:11:02.738 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T21:11:02.940 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T21:11:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-11T21:11:03.342 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T21:11:03.444 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-11T21:11:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T21:11:03.737 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T21:11:03.994 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-11T21:11:04.073 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T21:11:04.199 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:04.262 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-11T21:11:04.317 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:04.388 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:04.418 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-11T21:11:04.682 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-idna (3.3-1) ... 2024-03-11T21:11:04.746 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:04.872 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:04.991 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T21:11:04.998 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-11T21:11:05.284 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:05.410 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:05.536 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:05.750 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T21:11:05.750 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T21:11:06.155 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T21:11:06.213 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:06.338 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:06.426 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:11:06.465 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:06.558 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-11T21:11:06.582 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T21:11:06.809 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T21:11:06.936 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-11T21:11:07.062 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T21:11:07.355 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-11T21:11:07.606 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-11T21:11:07.745 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:07.857 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:11:07.871 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:07.983 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-11T21:11:07.997 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:08.123 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:08.241 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:08.276 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-11T21:11:08.528 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-11T21:11:08.700 INFO:teuthology.orchestra.run.smithi203.stdout:Adding group ceph....done 2024-03-11T21:11:08.787 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-11T21:11:09.046 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-11T21:11:09.110 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user ceph....done 2024-03-11T21:11:09.280 INFO:teuthology.orchestra.run.smithi203.stdout:Setting system user ceph properties....done 2024-03-11T21:11:09.291 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-11T21:11:09.295 INFO:teuthology.orchestra.run.smithi203.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-11T21:11:09.434 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-11T21:11:09.551 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T21:11:09.652 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-11T21:11:09.762 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-11T21:11:09.937 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T21:11:10.063 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T21:11:10.201 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:10.327 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:10.364 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-11T21:11:10.682 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-11T21:11:10.768 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:11:10.768 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:11:10.925 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T21:11:11.051 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-11T21:11:11.177 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T21:11:11.213 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:11.266 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-11T21:11:11.498 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T21:11:11.564 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-11T21:11:11.621 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T21:11:11.998 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-11T21:11:12.015 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:12.232 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T21:11:12.233 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T21:11:12.249 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-11T21:11:12.375 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-11T21:11:12.627 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T21:11:12.717 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:12.753 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T21:11:12.932 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T21:11:12.932 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T21:11:13.095 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T21:11:13.405 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T21:11:13.409 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:13.686 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T21:11:13.686 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T21:11:13.723 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-11T21:11:13.849 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-11T21:11:14.125 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-11T21:11:14.136 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:14.262 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:14.393 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T21:11:14.414 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:14.624 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T21:11:14.624 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T21:11:14.730 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-11T21:11:15.098 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-11T21:11:15.106 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.274 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.376 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.383 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T21:11:15.502 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.703 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T21:11:15.829 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.863 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-11T21:11:15.955 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:15.980 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-11T21:11:16.081 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-11T21:11:16.200 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-11T21:11:16.418 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:16.519 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:16.637 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-11T21:11:16.930 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:17.031 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:17.141 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:17.310 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-11T21:11:17.352 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T21:11:17.353 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T21:11:17.876 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:17.976 INFO:teuthology.orchestra.run.smithi203.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-11T21:11:17.982 DEBUG:teuthology.parallel:result is None 2024-03-11T21:11:18.002 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:18.470 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:18.596 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T21:11:19.734 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:19.835 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:19.936 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:20.062 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:20.188 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:20.773 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2024-03-11T21:11:21.108 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2024-03-11T21:11:21.294 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2024-03-11T21:11:21.307 INFO:teuthology.orchestra.run.smithi143.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-11T21:11:21.447 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-11T21:11:21.741 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-11T21:11:22.199 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:22.325 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:22.890 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:11:22.890 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:11:23.335 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:23.687 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-11T21:11:24.192 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:24.425 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T21:11:24.426 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T21:11:24.885 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:25.098 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T21:11:25.098 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T21:11:25.570 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:25.831 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T21:11:25.831 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T21:11:26.305 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:26.431 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:26.582 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:26.802 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T21:11:26.802 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T21:11:27.251 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:27.402 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:27.528 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:27.670 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T21:11:28.007 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-11T21:11:28.108 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-11T21:11:28.260 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-11T21:11:29.354 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-11T21:11:30.006 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-11T21:11:30.012 DEBUG:teuthology.parallel:result is None 2024-03-11T21:11:30.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:11:30.155 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2024-03-11T21:11:30.173 INFO:teuthology.orchestra.run.smithi143.stdout:19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:11:30.174 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:11:30.174 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g2b0101eb-1jammy is installed. 2024-03-11T21:11:30.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:11:30.494 DEBUG:teuthology.orchestra.run.smithi203:> dpkg-query -W -f '${Version}' ceph 2024-03-11T21:11:30.513 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:11:30.514 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T21:11:30.514 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g2b0101eb-1jammy is installed. 2024-03-11T21:11:30.516 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T21:11:30.516 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:11:30.516 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T21:11:30.530 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:11:30.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T21:11:30.569 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T21:11:30.570 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:11:30.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T21:11:30.583 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T21:11:30.638 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:11:30.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T21:11:30.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T21:11:30.708 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T21:11:30.708 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:11:30.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T21:11:30.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T21:11:30.775 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:11:30.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T21:11:30.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T21:11:30.844 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T21:11:30.844 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:11:30.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T21:11:30.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T21:11:30.912 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:11:30.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T21:11:30.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T21:11:30.980 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:11:31.080 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'root'} 2024-03-11T21:11:31.081 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:11:31.081 INFO:tasks.cephadm:Cluster fsid is eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:11:31.081 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:11:31.081 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.143', 'mon.c': '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]', 'mon.b': '172.21.15.203'} 2024-03-11T21:11:31.081 INFO:tasks.cephadm:First mon is mon.a on smithi143 2024-03-11T21:11:31.082 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:11:31.082 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:11:31.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-03-11T21:11:31.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-03-11T21:11:31.111 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:11:31.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:11:31.393 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.16+braggi16', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979-g2b0101eb-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-03-11 04:01:31.529734', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-03-11T21:11:31.675 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:11:31.677 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-03-11T21:11:31.677 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.421 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774557 Mar 11 21:11 /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.422 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.921 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774557 Mar 11 21:11 /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.922 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.930 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:11:32.949 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-11T21:11:32.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:11:32.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:11:33.141 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:11:33.146 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:12:28.485 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:12:28.486 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:12:28.486 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:12:28.486 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-03-11T21:12:28.487 INFO:teuthology.orchestra.run.smithi143.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:12:28.487 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:12:28.487 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:12:34.975 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:12:35.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-03-11T21:12:35.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-03-11T21:12:35.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-03-11T21:12:35.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-03-11T21:12:35.098 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:12:35.099 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:12:35.099 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:12:35.099 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-11T21:12:35.100 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:12:35.101 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-11T21:12:35.102 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:12:35.102 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:12:35.130 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eebe2e36-dfeb-11ee-95c8-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-11T21:12:35.130 DEBUG:teuthology.orchestra.run.smithi143:mon.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:12:35.175 DEBUG:teuthology.orchestra.run.smithi143:mgr.y> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:12:35.219 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:12:35.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.143 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:12:35.424 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:12:35.424 INFO:teuthology.orchestra.run.smithi143.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', 'eebe2e36-dfeb-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.143', '--skip-admin-label'] 2024-03-11T21:12:35.424 INFO:teuthology.orchestra.run.smithi143.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:12:35.425 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying podman|docker is present... 2024-03-11T21:12:35.425 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying lvm2 is present... 2024-03-11T21:12:35.425 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying time synchronization is in place... 2024-03-11T21:12:35.430 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:12:35.430 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:12:35.434 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:12:35.434 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.438 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-03-11T21:12:35.439 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-03-11T21:12:35.442 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-03-11T21:12:35.442 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.446 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-03-11T21:12:35.446 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout masked 2024-03-11T21:12:35.449 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-03-11T21:12:35.450 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.453 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-03-11T21:12:35.453 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-03-11T21:12:35.456 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-03-11T21:12:35.457 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.462 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout enabled 2024-03-11T21:12:35.464 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout active 2024-03-11T21:12:35.464 INFO:teuthology.orchestra.run.smithi143.stdout:Unit ntp.service is enabled and running 2024-03-11T21:12:35.464 INFO:teuthology.orchestra.run.smithi143.stdout:Repeating the final host check... 2024-03-11T21:12:35.464 INFO:teuthology.orchestra.run.smithi143.stdout:docker (/usr/bin/docker) is present 2024-03-11T21:12:35.465 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl is present 2024-03-11T21:12:35.465 INFO:teuthology.orchestra.run.smithi143.stdout:lvcreate is present 2024-03-11T21:12:35.467 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:12:35.468 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:12:35.471 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:12:35.471 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.475 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-03-11T21:12:35.475 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-03-11T21:12:35.478 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-03-11T21:12:35.478 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.481 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-03-11T21:12:35.481 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout masked 2024-03-11T21:12:35.484 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-03-11T21:12:35.485 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.488 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-03-11T21:12:35.488 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-03-11T21:12:35.491 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-03-11T21:12:35.491 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout inactive 2024-03-11T21:12:35.495 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout enabled 2024-03-11T21:12:35.499 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stdout active 2024-03-11T21:12:35.499 INFO:teuthology.orchestra.run.smithi143.stdout:Unit ntp.service is enabled and running 2024-03-11T21:12:35.499 INFO:teuthology.orchestra.run.smithi143.stdout:Host looks OK 2024-03-11T21:12:35.499 INFO:teuthology.orchestra.run.smithi143.stdout:Cluster fsid: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:35.499 INFO:teuthology.orchestra.run.smithi143.stdout:Acquiring lock 140679187936016 on /run/cephadm/eebe2e36-dfeb-11ee-95c8-87774f69a715.lock 2024-03-11T21:12:35.500 INFO:teuthology.orchestra.run.smithi143.stdout:Lock 140679187936016 acquired on /run/cephadm/eebe2e36-dfeb-11ee-95c8-87774f69a715.lock 2024-03-11T21:12:35.500 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying IP 172.21.15.143 port 3300 ... 2024-03-11T21:12:35.500 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying IP 172.21.15.143 port 6789 ... 2024-03-11T21:12:35.500 INFO:teuthology.orchestra.run.smithi143.stdout:Base mon IP(s) is [172.21.15.143:3300, 172.21.15.143:6789], mon addrv is [v2:172.21.15.143:3300,v1:172.21.15.143:6789] 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.143 metric 100 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 158.69.67.47 via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.143 metric 100 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.143 metric 100 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 172.21.0.1 dev enp3s0f1 proto dhcp scope link src 172.21.15.143 metric 100 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 172.21.0.2 dev enp3s0f1 proto dhcp scope link src 172.21.15.143 metric 100 2024-03-11T21:12:35.503 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.143 metric 100 2024-03-11T21:12:35.505 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:12:35.505 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T21:12:35.505 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1761sec pref medium 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7783/64 scope link 2024-03-11T21:12:35.507 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:12:35.508 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.0/20` 2024-03-11T21:12:35.508 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.0/20` 2024-03-11T21:12:35.508 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.1/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.1/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.2/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.0.2/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.15.254/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Mon IP `172.21.15.143` is in CIDR network `172.21.15.254/32` 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.0.1/32', '172.21.0.1/32', '172.21.0.2/32', '172.21.0.2/32', '172.21.15.254/32', '172.21.15.254/32'] 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:12:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:12:35.765 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/docker: stdout 2b0101eb74d60610d544cb8a473e23a35ac1cc0b: Pulling from ceph-ci/ceph 2024-03-11T21:12:35.766 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/docker: stdout Digest: sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:12:35.766 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:12:35.766 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:12:37.304 INFO:teuthology.orchestra.run.smithi143.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:12:37.304 INFO:teuthology.orchestra.run.smithi143.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:12:37.304 INFO:teuthology.orchestra.run.smithi143.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:12:38.524 INFO:teuthology.orchestra.run.smithi143.stdout:stat: stdout 167 167 2024-03-11T21:12:38.525 INFO:teuthology.orchestra.run.smithi143.stdout:Creating initial keys... 2024-03-11T21:12:39.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-authtool: stdout AQDHc+9lv+3eEBAAj3O9zcBhLTM1wZGWIbAshw== 2024-03-11T21:12:41.008 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-authtool: stdout AQDIc+9lmOzpHxAAM00DxAiSuSxncH1uAjtipQ== 2024-03-11T21:12:42.361 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-authtool: stdout AQDJc+9lKfSzLhAAl/TO4hFRbGKhYbzVZaG1YA== 2024-03-11T21:12:42.362 INFO:teuthology.orchestra.run.smithi143.stdout:Creating initial monmap... 2024-03-11T21:12:43.616 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool for a [v2:172.21.15.143:3300,v1:172.21.15.143:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:setting min_mon_release = quincy 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: set fsid to eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:12:43.617 INFO:teuthology.orchestra.run.smithi143.stdout:Creating mon... 2024-03-11T21:12:44.865 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 1 imported monmap: 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-03-11T21:12:43.118202+0000 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr created 2024-03-11T21:12:43.118202+0000 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.a 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 0 /usr/bin/ceph-mon: set fsid to eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.866 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Git sha 0 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: DB SUMMARY 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: DB Session ID: SY5WHKD33YH0LVIZWHPT 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.create_if_missing: 1 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:12:44.867 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.env: 0x55beaf1c9b20 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.info_log: 0x55beb0394b40 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.statistics: (nil) 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.use_fsync: 0 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T21:12:44.868 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.db_log_dir: 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.wal_dir: 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:12:44.869 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.write_buffer_manager: 0x55beb0369360 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.unordered_write: 0 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:12:44.870 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.row_cache: None 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.wal_filter: None 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.wal_compression: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:12:44.871 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:12:44.872 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_open_files: -1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Compression algorithms supported: 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kZSTD supported: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T21:12:44.873 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.266+0000 7f6c0a5bac80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.merge_operator: 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_filter: None 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:12:44.874 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55beb0394c60) 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x55beb039b090 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-03-11T21:12:44.875 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-03-11T21:12:44.876 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-03-11T21:12:44.877 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression: NoCompression 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.num_levels: 7 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:12:44.878 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:12:44.879 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:12:44.880 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:12:44.881 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T21:12:44.882 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.ttl: 2592000 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:12:44.883 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 95550424-d66e-44fd-b3c6-d24c71cda1ce 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55beb0468000 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: DB pointer 0x55beb0458000 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6bfa72b700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6bfa72b700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:12:44.884 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:12:44.886 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x55beb039b090#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.270+0000 7f6c0a5bac80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.274+0000 7f6c0a5bac80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T21:12:44.274+0000 7f6c0a5bac80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-03-11T21:12:44.887 INFO:teuthology.orchestra.run.smithi143.stdout:create mon.a on 2024-03-11T21:12:45.104 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-11T21:12:45.317 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-03-11T21:12:45.575 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target -> /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target. 2024-03-11T21:12:45.575 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target -> /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target. 2024-03-11T21:12:45.868 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:12:45.868 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:12:46.079 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target.wants/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service -> /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service. 2024-03-11T21:12:46.088 INFO:teuthology.orchestra.run.smithi143.stdout:firewalld does not appear to be present 2024-03-11T21:12:46.089 INFO:teuthology.orchestra.run.smithi143.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:12:46.089 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mon to start... 2024-03-11T21:12:46.091 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mon... 2024-03-11T21:12:47.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:47 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.380422+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:12:48.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout id: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.197312s) 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:48.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 Processor -- start 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- start start 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9554108f10 con 0x7f9554108520 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9559e42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9559e42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48570/0 (socket says 172.21.15.143:48570) 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9559e42700 1 -- 172.21.15.143:0/2227202480 learned_addr learned my addr 172.21.15.143:0/2227202480 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9559e42700 1 -- 172.21.15.143:0/2227202480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554109720 con 0x7f9554108520 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9559e42700 1 --2- 172.21.15.143:0/2227202480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95500098a0 tx=0x7f9550009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db32ad484b6e81b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9558e40700 1 -- 172.21.15.143:0/2227202480 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9550004160 con 0x7f9554108520 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9558e40700 1 -- 172.21.15.143:0/2227202480 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550004300 con 0x7f9554108520 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f9558e40700 1 -- 172.21.15.143:0/2227202480 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95500036a0 con 0x7f9554108520 2024-03-11T21:12:48.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- 172.21.15.143:0/2227202480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 msgr2=0x7f9554108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 --2- 172.21.15.143:0/2227202480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95500098a0 tx=0x7f9550009bb0 comp rx=0 tx=0).stop 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- 172.21.15.143:0/2227202480 shutdown_connections 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 --2- 172.21.15.143:0/2227202480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f9554108940 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- 172.21.15.143:0/2227202480 >> 172.21.15.143:0/2227202480 conn(0x7f955407b8f0 msgr2=0x7f9554106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- 172.21.15.143:0/2227202480 shutdown_connections 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- 172.21.15.143:0/2227202480 wait complete. 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 librados: starting msgr at 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 librados: starting objecter 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 Processor -- start 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 -- start start 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 librados: setting wanted keys 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.446+0000 7f955c0a6700 1 librados: calling monclient init 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9554109c50 con 0x7f9554108520 2024-03-11T21:12:48.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f9559e42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f9559e42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48580/0 (socket says 172.21.15.143:48580) 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f9559e42700 1 -- 172.21.15.143:0/2635308368 learned_addr learned my addr 172.21.15.143:0/2635308368 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f9559e42700 1 -- 172.21.15.143:0/2635308368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955419c1a0 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f9559e42700 1 --2- 172.21.15.143:0/2635308368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9550000c00 tx=0x7f9550004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 10 client.?.objecter ms_handle_connect 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9550003db0 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550011630 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955419c430 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9550011950 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955419f120 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 librados: init done 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f955001a460 con 0x7f9554108520 2024-03-11T21:12:48.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f955001b780 con 0x7f9554108520 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 10 client.4098.objecter ms_dispatch 0x7f9554001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f954affd700 20 client.4098.objecter dump_active .. 0 homeless 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.450+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9554063410 con 0x7f9554108520 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.454+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9550031020 con 0x7f9554108520 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f955402d830 con 0x7f9554108520 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f954affd700 1 -- 172.21.15.143:0/2635308368 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f955001b980 con 0x7f9554108520 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 10 librados: watch_flush enter 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 10 librados: watch_flush exit 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 20 client.4098.objecter shutdown clearing up homeless session... 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 10 client.4098.objecter successfully canceled tick 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 msgr2=0x7f955419bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 --2- 172.21.15.143:0/2635308368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9550000c00 tx=0x7f9550004750 comp rx=0 tx=0).stop 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 shutdown_connections 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 --2- 172.21.15.143:0/2635308368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9554108520 0x7f955419bc60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 >> 172.21.15.143:0/2635308368 conn(0x7f955407b8f0 msgr2=0x7f95541928d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 shutdown_connections 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 -- 172.21.15.143:0/2635308368 wait complete. 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:47.578+0000 7f955c0a6700 1 librados: shutdown 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:mon is available 2024-03-11T21:12:48.110 INFO:teuthology.orchestra.run.smithi143.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.383969+0000 mon.a (mon.0) 2 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.384150+0000 mon.a (mon.0) 3 : cluster 0 monmap e1: 1 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.386094+0000 mon.a (mon.0) 4 : cluster 0 fsmap 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.388373+0000 mon.a (mon.0) 5 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: cluster 2024-03-11T21:12:47.388903+0000 mon.a (mon.0) 6 : cluster 0 mgrmap e1: no daemons active 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:48 smithi143 bash[21028]: audit 2024-03-11T21:12:47.581242+0000 mon.a (mon.0) 7 : audit 0 from='client.? 172.21.15.143:0/2635308368' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T21:12:49.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:49 smithi143 bash[21028]: audit 2024-03-11T21:12:49.163514+0000 mon.a (mon.0) 8 : audit 1 from='client.? 172.21.15.143:0/3030914453' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:12:49.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:49 smithi143 bash[21028]: audit 2024-03-11T21:12:49.167395+0000 mon.a (mon.0) 9 : audit 1 from='client.? 172.21.15.143:0/3030914453' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:12:49.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:49.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:12:49.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout fsid = eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:49.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.143:3300,v1:172.21.15.143:6789] 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:12:49.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 Processor -- start 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- start start 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a28106c50 con 0x7f6a28106260 2024-03-11T21:12:49.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48590/0 (socket says 172.21.15.143:48590) 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 -- 172.21.15.143:0/1348097909 learned_addr learned my addr 172.21.15.143:0/1348097909 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 -- 172.21.15.143:0/1348097909 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28107460 con 0x7f6a28106260 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 --2- 172.21.15.143:0/1348097909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6a240098a0 tx=0x7f6a24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d962e04e713786 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2c99c700 1 -- 172.21.15.143:0/1348097909 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a24004160 con 0x7f6a28106260 2024-03-11T21:12:49.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2c99c700 1 -- 172.21.15.143:0/1348097909 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a24004300 con 0x7f6a28106260 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2c99c700 1 -- 172.21.15.143:0/1348097909 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a240036a0 con 0x7f6a28106260 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- 172.21.15.143:0/1348097909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 msgr2=0x7f6a28106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 --2- 172.21.15.143:0/1348097909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6a240098a0 tx=0x7f6a24009bb0 comp rx=0 tx=0).stop 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- 172.21.15.143:0/1348097909 shutdown_connections 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 --2- 172.21.15.143:0/1348097909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a28106680 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- 172.21.15.143:0/1348097909 >> 172.21.15.143:0/1348097909 conn(0x7f6a28101800 msgr2=0x7f6a28103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- 172.21.15.143:0/1348097909 shutdown_connections 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- 172.21.15.143:0/1348097909 wait complete. 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 librados: starting msgr at 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 librados: starting objecter 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 Processor -- start 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- start start 2024-03-11T21:12:49.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 librados: setting wanted keys 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 librados: calling monclient init 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2fc02700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a28107990 con 0x7f6a28106260 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48594/0 (socket says 172.21.15.143:48594) 2024-03-11T21:12:49.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 -- 172.21.15.143:0/3030914453 learned_addr learned my addr 172.21.15.143:0/3030914453 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:49.838 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.034+0000 7f6a2d99e700 1 -- 172.21.15.143:0/3030914453 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2807c070 con 0x7f6a28106260 2024-03-11T21:12:49.838 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2d99e700 1 --2- 172.21.15.143:0/3030914453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6a24000c00 tx=0x7f6a24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:49.838 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 10 client.?.objecter ms_handle_connect 0x7f6a28106260 2024-03-11T21:12:49.838 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a24003db0 con 0x7f6a28106260 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a24011630 con 0x7f6a28106260 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2807a260 con 0x7f6a28106260 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a24011950 con 0x7f6a28106260 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2fc02700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2807a7a0 con 0x7f6a28106260 2024-03-11T21:12:49.839 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2fc02700 1 librados: init done 2024-03-11T21:12:49.840 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6a24011af0 con 0x7f6a28106260 2024-03-11T21:12:49.840 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6a2401c450 con 0x7f6a28106260 2024-03-11T21:12:49.840 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 10 client.4100.objecter ms_dispatch 0x7f6a28001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:12:49.840 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:12:49.840 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 20 client.4100.objecter dump_active .. 0 homeless 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a28063410 con 0x7f6a28106260 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.038+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6a24032020 con 0x7f6a28106260 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.158+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6a28192fa0 con 0x7f6a28106260 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.162+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+340 (secure 0 0 0) 0x7f6a24004260 con 0x7f6a28106260 2024-03-11T21:12:49.841 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a1affd700 1 -- 172.21.15.143:0/3030914453 <== mon.0 v2:172.21.15.143:3300/0 8 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f6a2401c6a0 con 0x7f6a28106260 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 10 librados: watch_flush enter 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 10 librados: watch_flush exit 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 20 client.4100.objecter shutdown clearing up homeless session... 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 10 client.4100.objecter successfully canceled tick 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 msgr2=0x7f6a2807bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:49.842 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 --2- 172.21.15.143:0/3030914453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6a24000c00 tx=0x7f6a24004750 comp rx=0 tx=0).stop 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 shutdown_connections 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 --2- 172.21.15.143:0/3030914453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a28106260 0x7f6a2807bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 >> 172.21.15.143:0/3030914453 conn(0x7f6a28101800 msgr2=0x7f6a281905b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 shutdown_connections 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 -- 172.21.15.143:0/3030914453 wait complete. 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:49.166+0000 7f6a2fc02700 1 librados: shutdown 2024-03-11T21:12:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:Generating new minimal ceph.conf... 2024-03-11T21:12:51.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:50 smithi143 bash[21028]: audit 2024-03-11T21:12:50.834563+0000 mon.a (mon.0) 10 : audit 0 from='client.? 172.21.15.143:0/70454828' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:51.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.698+0000 7fdb0d99e700 1 Processor -- start 2024-03-11T21:12:51.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.698+0000 7fdb0d99e700 1 -- start start 2024-03-11T21:12:51.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.698+0000 7fdb0d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:51.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.698+0000 7fdb0d99e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0807a870 con 0x7fdb0807be40 2024-03-11T21:12:51.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:51.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48598/0 (socket says 172.21.15.143:48598) 2024-03-11T21:12:51.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 -- 172.21.15.143:0/2373230361 learned_addr learned my addr 172.21.15.143:0/2373230361 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:51.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 -- 172.21.15.143:0/2373230361 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0807a9f0 con 0x7fdb0807be40 2024-03-11T21:12:51.362 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- 172.21.15.143:0/2373230361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098a0 tx=0x7fdaf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd0fa9022a3d557 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:51.362 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb05ffb700 1 -- 172.21.15.143:0/2373230361 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaf8004160 con 0x7fdb0807be40 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb05ffb700 1 -- 172.21.15.143:0/2373230361 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fdaf8004300 con 0x7fdb0807be40 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb05ffb700 1 -- 172.21.15.143:0/2373230361 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaf8011960 con 0x7fdb0807be40 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/2373230361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 msgr2=0x7fdb0807a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 --2- 172.21.15.143:0/2373230361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098a0 tx=0x7fdaf8009bb0 comp rx=0 tx=0).stop 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/2373230361 shutdown_connections 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 --2- 172.21.15.143:0/2373230361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0807a2a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/2373230361 >> 172.21.15.143:0/2373230361 conn(0x7fdb08101800 msgr2=0x7fdb08103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/2373230361 shutdown_connections 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/2373230361 wait complete. 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 librados: starting msgr at 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 librados: starting objecter 2024-03-11T21:12:51.363 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 Processor -- start 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- start start 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 librados: setting wanted keys 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 librados: calling monclient init 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0810c7d0 con 0x7fdb0807be40 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48610/0 (socket says 172.21.15.143:48610) 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 -- 172.21.15.143:0/70454828 learned_addr learned my addr 172.21.15.143:0/70454828 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 -- 172.21.15.143:0/70454828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0819c850 con 0x7fdb0807be40 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb06ffd700 1 --2- 172.21.15.143:0/70454828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8011670 tx=0x7fdaf80117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0c99c700 10 client.?.objecter ms_handle_connect 0x7fdb0807be40 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0c99c700 10 client.?.objecter resend_mon_ops 2024-03-11T21:12:51.364 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaf801a430 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fdaf8020070 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0819cae0 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaf8019660 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0819f7d0 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.702+0000 7fdb0d99e700 1 librados: init done 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdaf8019cb0 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdaf800b720 con 0x7fdb0807be40 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 10 client.4102.objecter ms_dispatch 0x7fdb08001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 20 client.4102.objecter dump_active .. 0 homeless 2024-03-11T21:12:51.365 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf4005150 con 0x7fdb0807be40 2024-03-11T21:12:51.366 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.706+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdaf80042a0 con 0x7fdb0807be40 2024-03-11T21:12:51.366 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.830+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdaf4005710 con 0x7fdb0807be40 2024-03-11T21:12:51.367 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.830+0000 7fdb0c99c700 1 -- 172.21.15.143:0/70454828 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fdaf801abc0 con 0x7fdb0807be40 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 10 librados: watch_flush enter 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 10 librados: watch_flush exit 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 10 client.4102.objecter successfully canceled tick 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 msgr2=0x7fdb0819c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 --2- 172.21.15.143:0/70454828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8011670 tx=0x7fdaf80117f0 comp rx=0 tx=0).stop 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 shutdown_connections 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 --2- 172.21.15.143:0/70454828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0807be40 0x7fdb0819c310 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 >> 172.21.15.143:0/70454828 conn(0x7fdb08101800 msgr2=0x7fdb08102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:51.368 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 shutdown_connections 2024-03-11T21:12:51.369 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 -- 172.21.15.143:0/70454828 wait complete. 2024-03-11T21:12:51.369 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:50.834+0000 7fdb0d99e700 1 librados: shutdown 2024-03-11T21:12:51.369 INFO:teuthology.orchestra.run.smithi143.stdout:Restarting the monitor... 2024-03-11T21:12:51.644 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:51 smithi143 systemd[1]: Stopping Ceph mon.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:12:51.644 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:51 smithi143 bash[21028]: debug 2024-03-11T21:12:51.502+0000 7f1daacb2700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:12:51.644 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:51 smithi143 bash[21028]: debug 2024-03-11T21:12:51.502+0000 7f1daacb2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:12:52.205 INFO:teuthology.orchestra.run.smithi143.stdout:Setting public_network to 172.21.0.2/32,172.21.0.0/20,172.21.0.1/32,172.21.15.254/32 in mon config section 2024-03-11T21:12:52.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:52 smithi143 bash[21411]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:12:52.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:52 smithi143 bash[21473]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:12:52.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:52 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:12:52.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:52 smithi143 systemd[1]: Stopped Ceph mon.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:12:52.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:52 smithi143 systemd[1]: Started Ceph mon.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:12:53.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.346+0000 7f6fc3231c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.346+0000 7f6fc3231c80 0 ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.346+0000 7f6fc3231c80 0 pidfile_write: ignore empty --pid-file 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 0 load: jerasure load: lrc 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Git sha 0 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: DB SUMMARY 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: DB Session ID: A264FF22XJ32UIV8V2QI 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: CURRENT file: CURRENT 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92581 ; 2024-03-11T21:12:53.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.create_if_missing: 0 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.env: 0x559695e7eb20 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.info_log: 0x559696f2f920 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.statistics: (nil) 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.use_fsync: 0 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T21:12:53.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.db_log_dir: 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.wal_dir: 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:12:53.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.write_buffer_manager: 0x559696f03400 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:12:53.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.unordered_write: 0 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.row_cache: None 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.wal_filter: None 2024-03-11T21:12:53.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.wal_compression: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:12:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_open_files: -1 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:12:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Compression algorithms supported: 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kZSTD supported: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:12:53.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.merge_operator: 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_filter: None 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559696f2fa20) 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cache_index_and_filter_blocks: 1 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: pin_top_level_index_and_filter: 1 2024-03-11T21:12:53.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: index_type: 0 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: data_block_index_type: 0 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: index_shortening: 1 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: checksum: 4 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: no_block_cache: 0 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_cache: 0x559696f33090 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_cache_name: BinnedLRUCache 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_cache_options: 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: capacity : 536870912 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: num_shard_bits : 4 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: strict_capacity_limit : 0 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: high_pri_pool_ratio: 0.000 2024-03-11T21:12:53.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_cache_compressed: (nil) 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: persistent_cache: (nil) 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_size: 4096 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_size_deviation: 10 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_restart_interval: 16 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: index_block_restart_interval: 1 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: metadata_block_size: 4096 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: partition_filters: 0 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: use_delta_encoding: 1 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: filter_policy: bloomfilter 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: whole_key_filtering: 1 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: verify_compression: 0 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: read_amp_bytes_per_bit: 0 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: format_version: 5 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: enable_index_compression: 1 2024-03-11T21:12:53.456 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: block_align: 0 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: max_auto_readahead_size: 262144 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: prepopulate_block_cache: 0 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: initial_auto_readahead_size: 8192 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: num_file_reads_for_auto_readahead: 2 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression: NoCompression 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.num_levels: 7 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:12:53.457 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:12:53.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:12:53.459 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:12:53.460 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T21:12:53.461 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.ttl: 2592000 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:12:53.462 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.350+0000 7f6fc3231c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 95550424-d66e-44fd-b3c6-d24c71cda1ce 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191573357959, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191573359148, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88334, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 86422, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12032, "raw_average_key_size": 49, "raw_value_size": 79867, "raw_average_value_size": 325, "num_data_blocks": 11, "num_entries": 245, "num_filter_entries": 245, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191573, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "95550424-d66e-44fd-b3c6-d24c71cda1ce", "db_session_id": "A264FF22XJ32UIV8V2QI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191573359258, "job": 1, "event": "recovery_finished"} 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.354+0000 7f6fc3231c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55969702a000 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 4 rocksdb: DB pointer 0x559697014000 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fb199a700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fb199a700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:12:53.463 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: ** DB Stats ** 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: ** Compaction Stats [default] ** 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: L0 2/0 88.10 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Sum 2/0 88.10 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:12:53.464 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: ** Compaction Stats [default] ** 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Cumulative compaction: 0.00 GB write, 14.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Interval compaction: 0.00 GB write, 14.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:12:53.465 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Block cache BinnedLRUCache@0x559696f33090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] at bind addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 1 mon.a@-1(???) e1 preinit fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).mds e1 new map 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).mds e1 print_map 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: e1 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: legacy client fscid: -1 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: No filesystems configured 2024-03-11T21:12:53.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: debug 2024-03-11T21:12:53.358+0000 7f6fc3231c80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cluster 2024-03-11T21:12:53.384419+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cluster 2024-03-11T21:12:53.384508+0000 mon.a (mon.0) 2 : cluster 0 monmap e1: 1 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cluster 2024-03-11T21:12:53.404364+0000 mon.a (mon.0) 3 : cluster 0 fsmap 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cluster 2024-03-11T21:12:53.404402+0000 mon.a (mon.0) 4 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:12:53.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:53 smithi143 bash[21512]: cluster 2024-03-11T21:12:53.404848+0000 mon.a (mon.0) 5 : cluster 0 mgrmap e1: no daemons active 2024-03-11T21:12:54.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 Processor -- start 2024-03-11T21:12:54.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- start start 2024-03-11T21:12:54.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd944074fa0 con 0x7fd944106d00 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49340/0 (socket says 172.21.15.143:49340) 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94259c700 1 -- 172.21.15.143:0/906850095 learned_addr learned my addr 172.21.15.143:0/906850095 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94259c700 1 -- 172.21.15.143:0/906850095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd944109630 con 0x7fd944106d00 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94259c700 1 --2- 172.21.15.143:0/906850095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009e80 tx=0x7fd93c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1c6c62e1af61ac server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94159a700 1 -- 172.21.15.143:0/906850095 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93c017070 con 0x7fd944106d00 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94159a700 1 -- 172.21.15.143:0/906850095 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fd93c00b8c0 con 0x7fd944106d00 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd94159a700 1 -- 172.21.15.143:0/906850095 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93c004340 con 0x7fd944106d00 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- 172.21.15.143:0/906850095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 msgr2=0x7fd9441090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 --2- 172.21.15.143:0/906850095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009e80 tx=0x7fd93c00b330 comp rx=0 tx=0).stop 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- 172.21.15.143:0/906850095 shutdown_connections 2024-03-11T21:12:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 --2- 172.21.15.143:0/906850095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd9441090f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- 172.21.15.143:0/906850095 >> 172.21.15.143:0/906850095 conn(0x7fd944100730 msgr2=0x7fd944102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.406+0000 7fd948cf3700 1 -- 172.21.15.143:0/906850095 shutdown_connections 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- 172.21.15.143:0/906850095 wait complete. 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 librados: starting msgr at 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 librados: starting objecter 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 Processor -- start 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- start start 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 librados: setting wanted keys 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 librados: calling monclient init 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94410a5a0 con 0x7fd944106d00 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd94259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd94259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49344/0 (socket says 172.21.15.143:49344) 2024-03-11T21:12:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd94259c700 1 -- 172.21.15.143:0/562235916 learned_addr learned my addr 172.21.15.143:0/562235916 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd94259c700 1 -- 172.21.15.143:0/562235916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd944106ac0 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd94259c700 1 --2- 172.21.15.143:0/562235916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009530 tx=0x7fd93c004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 10 client.?.objecter ms_handle_connect 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93c021020 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fd93c003bd0 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93c017040 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd944104cb0 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9441051f0 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 librados: init done 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd93c024020 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd93c00f050 con 0x7fd944106d00 2024-03-11T21:12:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 10 client.14098.objecter ms_dispatch 0x7fd944001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd9337fe700 20 client.14098.objecter dump_active .. 0 homeless 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.410+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944063410 con 0x7fd944106d00 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.414+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd93c02b020 con 0x7fd944106d00 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.542+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd9441053c0 con 0x7fd944106d00 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd9337fe700 1 -- 172.21.15.143:0/562235916 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd93c00b860 con 0x7fd944106d00 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 10 librados: watch_flush enter 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 10 librados: watch_flush exit 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 20 client.14098.objecter shutdown clearing up homeless session... 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 10 client.14098.objecter successfully canceled tick 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 msgr2=0x7fd944106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 --2- 172.21.15.143:0/562235916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009530 tx=0x7fd93c004490 comp rx=0 tx=0).stop 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 shutdown_connections 2024-03-11T21:12:54.109 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 --2- 172.21.15.143:0/562235916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd944106d00 0x7fd944106580 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:54.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 >> 172.21.15.143:0/562235916 conn(0x7fd944100730 msgr2=0x7fd944101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:54.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 shutdown_connections 2024-03-11T21:12:54.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 -- 172.21.15.143:0/562235916 wait complete. 2024-03-11T21:12:54.110 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:53.550+0000 7fd948cf3700 1 librados: shutdown 2024-03-11T21:12:54.110 INFO:teuthology.orchestra.run.smithi143.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:12:54.125 INFO:teuthology.orchestra.run.smithi143.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:12:54.125 INFO:teuthology.orchestra.run.smithi143.stdout:Creating mgr... 2024-03-11T21:12:54.125 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:12:54.125 INFO:teuthology.orchestra.run.smithi143.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:12:54.393 INFO:teuthology.orchestra.run.smithi143.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:12:54.393 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:12:54.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:54 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:12:54.609 INFO:teuthology.orchestra.run.smithi143.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715.target.wants/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y.service -> /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service. 2024-03-11T21:12:54.618 INFO:teuthology.orchestra.run.smithi143.stdout:firewalld does not appear to be present 2024-03-11T21:12:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:12:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:firewalld does not appear to be present 2024-03-11T21:12:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:12:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mgr to start... 2024-03-11T21:12:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mgr... 2024-03-11T21:12:54.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:54 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:12:54.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:54 smithi143 bash[21512]: audit 2024-03-11T21:12:53.552509+0000 mon.a (mon.0) 6 : audit 1 from='client.? 172.21.15.143:0/562235916' entity='client.admin' 2024-03-11T21:12:56.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:56 smithi143 bash[21512]: audit 2024-03-11T21:12:56.023439+0000 mon.a (mon.0) 7 : audit 0 from='client.? 172.21.15.143:0/2664124205' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:12:56.445 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:55 smithi143 bash[21768]: debug 2024-03-11T21:12:55.961+0000 7f385fe14200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:12:56.601 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:12:56.602 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:12:56.602 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:12:56.602 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:12:56.603 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:12:56.604 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.605 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:12:47.385218+0000", 2024-03-11T21:12:56.606 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 Processor -- start 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- start start 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64106c50 con 0x7f4a64106260 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49370/0 (socket says 172.21.15.143:49370) 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 -- 172.21.15.143:0/1211044572 learned_addr learned my addr 172.21.15.143:0/1211044572 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 -- 172.21.15.143:0/1211044572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64107460 con 0x7f4a64106260 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 --2- 172.21.15.143:0/1211044572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098a0 tx=0x7f4a54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb8dcc246686b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a61ffb700 1 -- 172.21.15.143:0/1211044572 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a54004160 con 0x7f4a64106260 2024-03-11T21:12:56.607 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a61ffb700 1 -- 172.21.15.143:0/1211044572 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4a54004300 con 0x7f4a64106260 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a61ffb700 1 -- 172.21.15.143:0/1211044572 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a54011960 con 0x7f4a64106260 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- 172.21.15.143:0/1211044572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 msgr2=0x7f4a64106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 --2- 172.21.15.143:0/1211044572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098a0 tx=0x7f4a54009bb0 comp rx=0 tx=0).stop 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- 172.21.15.143:0/1211044572 shutdown_connections 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 --2- 172.21.15.143:0/1211044572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a64106680 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- 172.21.15.143:0/1211044572 >> 172.21.15.143:0/1211044572 conn(0x7f4a64101820 msgr2=0x7f4a64103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- 172.21.15.143:0/1211044572 shutdown_connections 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- 172.21.15.143:0/1211044572 wait complete. 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 librados: starting msgr at 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 librados: starting objecter 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 Processor -- start 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- start start 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 librados: setting wanted keys 2024-03-11T21:12:56.608 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 librados: calling monclient init 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a6961e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6410a690 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49378/0 (socket says 172.21.15.143:49378) 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 -- 172.21.15.143:0/2664124205 learned_addr learned my addr 172.21.15.143:0/2664124205 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.881+0000 7f4a62ffd700 1 -- 172.21.15.143:0/2664124205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6407c070 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a62ffd700 1 --2- 172.21.15.143:0/2664124205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4a54003ea0 tx=0x7f4a540117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 10 client.?.objecter ms_handle_connect 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a540038b0 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4a5401a890 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6407a260 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a540247b0 con 0x7f4a64106260 2024-03-11T21:12:56.609 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a6961e700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6407a7a0 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a6961e700 1 librados: init done 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4a54022020 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4a5402dd00 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 10 client.14102.objecter ms_dispatch 0x7f4a64001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 20 client.14102.objecter dump_active .. 0 homeless 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64063410 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:55.885+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4a5401aa40 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.017+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4a6407aa20 con 0x7f4a64106260 2024-03-11T21:12:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.017+0000 7f4a4bfff700 1 -- 172.21.15.143:0/2664124205 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4a5401a430 con 0x7f4a64106260 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 10 librados: watch_flush enter 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 10 librados: watch_flush exit 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 20 client.14102.objecter shutdown clearing up homeless session... 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 10 client.14102.objecter successfully canceled tick 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 msgr2=0x7f4a6407bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 --2- 172.21.15.143:0/2664124205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4a54003ea0 tx=0x7f4a540117f0 comp rx=0 tx=0).stop 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 shutdown_connections 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 --2- 172.21.15.143:0/2664124205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64106260 0x7f4a6407bb30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 >> 172.21.15.143:0/2664124205 conn(0x7f4a64101820 msgr2=0x7f4a64102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:12:56.611 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 shutdown_connections 2024-03-11T21:12:56.612 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 -- 172.21.15.143:0/2664124205 wait complete. 2024-03-11T21:12:56.612 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:56.021+0000 7f4a6961e700 1 librados: shutdown 2024-03-11T21:12:56.612 INFO:teuthology.orchestra.run.smithi143.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:12:57.872 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:57 smithi143 bash[21768]: debug 2024-03-11T21:12:57.541+0000 7f385fe14200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:12:57.873 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:57 smithi143 bash[21768]: debug 2024-03-11T21:12:57.637+0000 7f385fe14200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:12:57.873 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:57 smithi143 bash[21768]: debug 2024-03-11T21:12:57.885+0000 7f385fe14200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:12:58.879 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:58 smithi143 bash[21768]: debug 2024-03-11T21:12:58.677+0000 7f385fe14200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:12:58.880 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:58 smithi143 bash[21768]: debug 2024-03-11T21:12:58.789+0000 7f385fe14200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:12:59.688 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:58 smithi143 bash[21768]: debug 2024-03-11T21:12:58.893+0000 7f385fe14200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:12:59.688 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:59 smithi143 bash[21768]: debug 2024-03-11T21:12:59.249+0000 7f385fe14200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:12:59.688 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:59 smithi143 bash[21768]: debug 2024-03-11T21:12:59.361+0000 7f385fe14200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:12:59.688 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:59 smithi143 bash[21768]: debug 2024-03-11T21:12:59.481+0000 7f385fe14200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:12:59.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:12:59 smithi143 bash[21512]: audit 2024-03-11T21:12:59.692130+0000 mon.a (mon.0) 8 : audit 0 from='client.? 172.21.15.143:0/2412168068' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:12:59.945 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:12:59 smithi143 bash[21768]: debug 2024-03-11T21:12:59.841+0000 7f385fe14200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:13:00.260 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:00.260 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:00.261 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:13:00.262 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:13:00.263 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:13:00.264 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:12:47.385218+0000", 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:00.265 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:13:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.545+0000 7f88d2880700 1 Processor -- start 2024-03-11T21:13:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.545+0000 7f88d2880700 1 -- start start 2024-03-11T21:13:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc074fa0 con 0x7f88cc104ae0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49388/0 (socket says 172.21.15.143:49388) 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 -- 172.21.15.143:0/4029427333 learned_addr learned my addr 172.21.15.143:0/4029427333 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 -- 172.21.15.143:0/4029427333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc105440 con 0x7f88cc104ae0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- 172.21.15.143:0/4029427333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0098a0 tx=0x7f88bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4887da137bde5522 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88caffd700 1 -- 172.21.15.143:0/4029427333 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88bc004160 con 0x7f88cc104ae0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88caffd700 1 -- 172.21.15.143:0/4029427333 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f88bc004300 con 0x7f88cc104ae0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88caffd700 1 -- 172.21.15.143:0/4029427333 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88bc011960 con 0x7f88cc104ae0 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/4029427333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 msgr2=0x7f88cc104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 --2- 172.21.15.143:0/4029427333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0098a0 tx=0x7f88bc009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/4029427333 shutdown_connections 2024-03-11T21:13:00.267 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 --2- 172.21.15.143:0/4029427333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc104f00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/4029427333 >> 172.21.15.143:0/4029427333 conn(0x7f88cc100760 msgr2=0x7f88cc102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/4029427333 shutdown_connections 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/4029427333 wait complete. 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 librados: starting msgr at 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 librados: starting objecter 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 Processor -- start 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- start start 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 librados: setting wanted keys 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 librados: calling monclient init 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc106410 con 0x7f88cc104ae0 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49404/0 (socket says 172.21.15.143:49404) 2024-03-11T21:13:00.268 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 -- 172.21.15.143:0/2412168068 learned_addr learned my addr 172.21.15.143:0/2412168068 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 -- 172.21.15.143:0/2412168068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc19c8b0 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88cbfff700 1 --2- 172.21.15.143:0/2412168068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011670 tx=0x7f88bc0117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 10 client.?.objecter ms_handle_connect 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88bc01a430 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f88bc020070 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc19cb40 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88bc019660 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc19f830 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f88bc019c60 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88d2880700 1 librados: init done 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f88bc00bac0 con 0x7f88cc104ae0 2024-03-11T21:13:00.269 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 10 client.14104.objecter ms_dispatch 0x7f88cc001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.549+0000 7f88c97fa700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.553+0000 7f88c97fa700 20 client.14104.objecter dump_active .. 0 homeless 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.553+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88cc063410 con 0x7f88cc104ae0 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.553+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f88bc0042a0 con 0x7f88cc104ae0 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.685+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f88cc02d830 con 0x7f88cc104ae0 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88c97fa700 1 -- 172.21.15.143:0/2412168068 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f88bc01abc0 con 0x7f88cc104ae0 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 10 librados: watch_flush enter 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 10 librados: watch_flush exit 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 20 client.14104.objecter shutdown clearing up homeless session... 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 10 client.14104.objecter successfully canceled tick 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 msgr2=0x7f88cc19c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 --2- 172.21.15.143:0/2412168068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011670 tx=0x7f88bc0117f0 comp rx=0 tx=0).stop 2024-03-11T21:13:00.270 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 shutdown_connections 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 --2- 172.21.15.143:0/2412168068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88cc104ae0 0x7f88cc19c370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 >> 172.21.15.143:0/2412168068 conn(0x7f88cc100760 msgr2=0x7f88cc192f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 shutdown_connections 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 -- 172.21.15.143:0/2412168068 wait complete. 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:12:59.689+0000 7f88d2880700 1 librados: shutdown 2024-03-11T21:13:00.271 INFO:teuthology.orchestra.run.smithi143.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:13:00.332 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:00 smithi143 bash[21768]: debug 2024-03-11T21:13:00.084+0000 7f385fe14200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:13:00.696 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:00 smithi143 bash[21768]: debug 2024-03-11T21:13:00.344+0000 7f385fe14200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:13:00.696 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:00 smithi143 bash[21768]: debug 2024-03-11T21:13:00.456+0000 7f385fe14200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:13:01.195 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:00 smithi143 bash[21768]: debug 2024-03-11T21:13:00.732+0000 7f385fe14200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:13:01.195 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:00 smithi143 bash[21768]: debug 2024-03-11T21:13:00.972+0000 7f385fe14200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:01.695 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:01 smithi143 bash[21768]: debug 2024-03-11T21:13:01.404+0000 7f385fe14200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:13:01.696 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:01 smithi143 bash[21768]: debug 2024-03-11T21:13:01.520+0000 7f385fe14200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:13:02.196 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:01 smithi143 bash[21768]: debug 2024-03-11T21:13:01.764+0000 7f385fe14200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:03.180 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:02 smithi143 bash[21768]: debug 2024-03-11T21:13:02.776+0000 7f385fe14200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:13:03.180 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:02 smithi143 bash[21768]: debug 2024-03-11T21:13:02.888+0000 7f385fe14200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:13:03.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:03 smithi143 bash[21512]: audit 2024-03-11T21:13:03.267737+0000 mon.a (mon.0) 9 : audit 0 from='client.? 172.21.15.143:0/1525886837' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:13:03.431 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:03 smithi143 bash[21768]: debug 2024-03-11T21:13:03.192+0000 7f385fe14200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:13:03.431 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:03 smithi143 bash[21768]: debug 2024-03-11T21:13:03.304+0000 7f385fe14200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:13:03.810 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:13:03.811 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:13:03.812 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:13:03.821 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:13:03.821 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:13:03.821 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:13:03.821 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.821 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:13:03.822 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:12:47.385218+0000", 2024-03-11T21:13:03.823 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79de1cf700 1 Processor -- start 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79de1cf700 1 -- start start 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79de1cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79de1cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8079e70 con 0x7f79d8104b80 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79d77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79d77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49410/0 (socket says 172.21.15.143:49410) 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79d77fe700 1 -- 172.21.15.143:0/4052356758 learned_addr learned my addr 172.21.15.143:0/4052356758 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.124+0000 7f79d77fe700 1 -- 172.21.15.143:0/4052356758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d81074f0 con 0x7f79d8104b80 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 --2- 172.21.15.143:0/4052356758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8af11f2f8e4684 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d67fc700 1 -- 172.21.15.143:0/4052356758 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c8004160 con 0x7f79d8104b80 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d67fc700 1 -- 172.21.15.143:0/4052356758 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f79c8004300 con 0x7f79d8104b80 2024-03-11T21:13:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d67fc700 1 -- 172.21.15.143:0/4052356758 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c8011960 con 0x7f79d8104b80 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/4052356758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 msgr2=0x7f79d8106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 --2- 172.21.15.143:0/4052356758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/4052356758 shutdown_connections 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 --2- 172.21.15.143:0/4052356758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d8106fb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/4052356758 >> 172.21.15.143:0/4052356758 conn(0x7f79d81007c0 msgr2=0x7f79d8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/4052356758 shutdown_connections 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/4052356758 wait complete. 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 librados: starting msgr at 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 librados: starting objecter 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 Processor -- start 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- start start 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 librados: setting wanted keys 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 librados: calling monclient init 2024-03-11T21:13:03.825 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d81084c0 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49424/0 (socket says 172.21.15.143:49424) 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 -- 172.21.15.143:0/1525886837 learned_addr learned my addr 172.21.15.143:0/1525886837 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 -- 172.21.15.143:0/1525886837 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d81a0d30 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d77fe700 1 --2- 172.21.15.143:0/1525886837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79c8011740 tx=0x7f79c8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 10 client.?.objecter ms_handle_connect 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c80038b0 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f79c80248a0 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d81a0fc0 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c801a6b0 con 0x7f79d8104b80 2024-03-11T21:13:03.826 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 10 client.14106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d81a3cb0 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79de1cf700 1 librados: init done 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f79c801acf0 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f79c800b840 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 10 client.14106.objecter ms_dispatch 0x7f79d8001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 3 client.14106.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 3 client.14106.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.128+0000 7f79d4ff9700 20 client.14106.objecter dump_active .. 0 homeless 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.132+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d8063410 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.132+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f79c8043dd0 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.260+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f79d802d830 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79d4ff9700 1 -- 172.21.15.143:0/1525886837 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f79c8032020 con 0x7f79d8104b80 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 10 librados: watch_flush enter 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 10 librados: watch_flush exit 2024-03-11T21:13:03.827 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 20 client.14106.objecter shutdown clearing up homeless session... 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 10 client.14106.objecter successfully canceled tick 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 msgr2=0x7f79d81a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 --2- 172.21.15.143:0/1525886837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79c8011740 tx=0x7f79c8003dc0 comp rx=0 tx=0).stop 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 shutdown_connections 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 --2- 172.21.15.143:0/1525886837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d8104b80 0x7f79d81a07f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 >> 172.21.15.143:0/1525886837 conn(0x7f79d81007c0 msgr2=0x7f79d8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 shutdown_connections 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 -- 172.21.15.143:0/1525886837 wait complete. 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:03.264+0000 7f79de1cf700 1 librados: shutdown 2024-03-11T21:13:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:mgr not available, waiting (3/15)... 2024-03-11T21:13:04.196 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:03 smithi143 bash[21768]: debug 2024-03-11T21:13:03.760+0000 7f385fe14200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:13:04.196 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:03 smithi143 bash[21768]: debug 2024-03-11T21:13:03.880+0000 7f385fe14200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:13:04.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: cluster 2024-03-11T21:13:03.906132+0000 mon.a (mon.0) 10 : cluster 1 Activating manager daemon y 2024-03-11T21:13:04.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: cluster 2024-03-11T21:13:03.913520+0000 mon.a (mon.0) 11 : cluster 0 mgrmap e2: y(active, starting, since 0.00750196s) 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.914442+0000 mon.a (mon.0) 12 : audit 0 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.914666+0000 mon.a (mon.0) 13 : audit 0 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.914840+0000 mon.a (mon.0) 14 : audit 0 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.914965+0000 mon.a (mon.0) 15 : audit 0 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.916683+0000 mon.a (mon.0) 16 : audit 0 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: cluster 2024-03-11T21:13:03.942002+0000 mon.a (mon.0) 17 : cluster 1 Manager daemon y is now available 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.954709+0000 mon.a (mon.0) 18 : audit 1 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.957315+0000 mon.a (mon.0) 19 : audit 1 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.961025+0000 mon.a (mon.0) 20 : audit 1 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.966387+0000 mon.a (mon.0) 21 : audit 1 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' 2024-03-11T21:13:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:04 smithi143 bash[21512]: audit 2024-03-11T21:13:03.971732+0000 mon.a (mon.0) 22 : audit 1 from='mgr.14100 172.21.15.143:0/68991059' entity='mgr.y' 2024-03-11T21:13:06.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:05 smithi143 bash[21512]: cluster 2024-03-11T21:13:04.920414+0000 mon.a (mon.0) 23 : cluster 0 mgrmap e3: y(active, since 1.0144s) 2024-03-11T21:13:07.756 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:07.757 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:13:07.758 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.759 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:13:07.760 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:07.760 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:13:07.760 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:13:07.760 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:13:07.770 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:13:07.771 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:12:47.385218+0000", 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe95543f700 1 Processor -- start 2024-03-11T21:13:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe95543f700 1 -- start start 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe95543f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe95543f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe950108f10 con 0x7fe950108520 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe94effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe94effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48604/0 (socket says 172.21.15.143:48604) 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.747+0000 7fe94effd700 1 -- 172.21.15.143:0/3055305936 learned_addr learned my addr 172.21.15.143:0/3055305936 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 -- 172.21.15.143:0/3055305936 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950109720 con 0x7fe950108520 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 --2- 172.21.15.143:0/3055305936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9440098a0 tx=0x7fe944009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ac9739f749293b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94dffb700 1 -- 172.21.15.143:0/3055305936 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe944004160 con 0x7fe950108520 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94dffb700 1 -- 172.21.15.143:0/3055305936 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fe944004300 con 0x7fe950108520 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94dffb700 1 -- 172.21.15.143:0/3055305936 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe944011960 con 0x7fe950108520 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/3055305936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 msgr2=0x7fe950108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 --2- 172.21.15.143:0/3055305936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9440098a0 tx=0x7fe944009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/3055305936 shutdown_connections 2024-03-11T21:13:07.773 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 --2- 172.21.15.143:0/3055305936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe950108940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/3055305936 >> 172.21.15.143:0/3055305936 conn(0x7fe95007b8f0 msgr2=0x7fe950106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/3055305936 shutdown_connections 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/3055305936 wait complete. 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 librados: starting msgr at 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 librados: starting objecter 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 Processor -- start 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- start start 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 librados: setting wanted keys 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 librados: calling monclient init 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95010c950 con 0x7fe950108520 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48616/0 (socket says 172.21.15.143:48616) 2024-03-11T21:13:07.774 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 -- 172.21.15.143:0/2667928970 learned_addr learned my addr 172.21.15.143:0/2667928970 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 -- 172.21.15.143:0/2667928970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95019c9a0 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe94effd700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe944006af0 tx=0x7fe944003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 10 client.?.objecter ms_handle_connect 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe944011950 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7fe94401a890 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95019cc30 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9440196e0 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95019f920 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe95543f700 1 librados: init done 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.751+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fe9440198d0 con 0x7fe950108520 2024-03-11T21:13:07.775 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 0x7fe93803cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe950063410 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe94404d320 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 10 client.14114.objecter ms_dispatch 0x7fe950001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe94e7fc700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 0x7fe93803cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 20 client.14114.objecter dump_active .. 0 homeless 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe937fff700 10 client.14114.objecter ms_handle_connect 0x7fe93803a750 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.755+0000 7fe94e7fc700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 0x7fe93803cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe940006fd0 tx=0x7fe940006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.759+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe9440042c0 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:06.915+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fe94402d960 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.267+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe95019fcd0 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.267+0000 7fe937fff700 1 -- 172.21.15.143:0/2667928970 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fe944022340 con 0x7fe950108520 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 10 librados: watch_flush enter 2024-03-11T21:13:07.776 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 10 librados: watch_flush exit 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 20 client.14114.objecter shutdown clearing up homeless session... 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 10 client.14114.objecter successfully canceled tick 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 msgr2=0x7fe93803cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 0x7fe93803cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe940006fd0 tx=0x7fe940006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 msgr2=0x7fe95019c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe944006af0 tx=0x7fe944003980 comp rx=0 tx=0).stop 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 shutdown_connections 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7fe93803a750 0x7fe93803cc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 --2- 172.21.15.143:0/2667928970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe950108520 0x7fe95019c460 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 >> 172.21.15.143:0/2667928970 conn(0x7fe95007b8f0 msgr2=0x7fe950105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 shutdown_connections 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 -- 172.21.15.143:0/2667928970 wait complete. 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:07.271+0000 7fe95543f700 1 librados: shutdown 2024-03-11T21:13:07.777 INFO:teuthology.orchestra.run.smithi143.stdout:mgr is available 2024-03-11T21:13:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:07 smithi143 bash[21512]: cluster 2024-03-11T21:13:06.917417+0000 mon.a (mon.0) 24 : cluster 0 mgrmap e4: y(active, since 3s) 2024-03-11T21:13:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:07 smithi143 bash[21512]: audit 2024-03-11T21:13:07.269307+0000 mon.a (mon.0) 25 : audit 0 from='client.? 172.21.15.143:0/2667928970' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:13:09.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:09 smithi143 bash[21512]: audit 2024-03-11T21:13:09.007651+0000 mon.a (mon.0) 26 : audit 1 from='client.? 172.21.15.143:0/4248282793' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout fsid = eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:13:09.670 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.627+0000 7f4b559d1700 1 Processor -- start 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- start start 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50106c50 con 0x7f4b50106260 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48628/0 (socket says 172.21.15.143:48628) 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 -- 172.21.15.143:0/517801147 learned_addr learned my addr 172.21.15.143:0/517801147 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 -- 172.21.15.143:0/517801147 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50107460 con 0x7f4b50106260 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- 172.21.15.143:0/517801147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098a0 tx=0x7f4b44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edf07f70d8d5bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4dffb700 1 -- 172.21.15.143:0/517801147 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b44004160 con 0x7f4b50106260 2024-03-11T21:13:09.671 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4dffb700 1 -- 172.21.15.143:0/517801147 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4b44004300 con 0x7f4b50106260 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4dffb700 1 -- 172.21.15.143:0/517801147 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b44011960 con 0x7f4b50106260 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- 172.21.15.143:0/517801147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 msgr2=0x7f4b50106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 --2- 172.21.15.143:0/517801147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098a0 tx=0x7f4b44009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- 172.21.15.143:0/517801147 shutdown_connections 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 --2- 172.21.15.143:0/517801147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b50106680 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- 172.21.15.143:0/517801147 >> 172.21.15.143:0/517801147 conn(0x7f4b50101800 msgr2=0x7f4b50103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- 172.21.15.143:0/517801147 shutdown_connections 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- 172.21.15.143:0/517801147 wait complete. 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 librados: starting msgr at 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 librados: starting objecter 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 Processor -- start 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- start start 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 librados: setting wanted keys 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 librados: calling monclient init 2024-03-11T21:13:09.672 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b559d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5010a690 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48630/0 (socket says 172.21.15.143:48630) 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 -- 172.21.15.143:0/4248282793 learned_addr learned my addr 172.21.15.143:0/4248282793 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 -- 172.21.15.143:0/4248282793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5007c070 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b4effd700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006af0 tx=0x7f4b44003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b549cf700 10 client.?.objecter ms_handle_connect 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b549cf700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.631+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b44011950 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5007a260 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4b4401a890 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b44019700 con 0x7f4b50106260 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b559d1700 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:09.673 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5007a7a0 con 0x7f4b50106260 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b559d1700 1 librados: init done 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f4b4401aa40 con 0x7f4b50106260 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50063410 con 0x7f4b50106260 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 0x7f4b3803cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b4404d2d0 con 0x7f4b50106260 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 10 client.14116.objecter ms_dispatch 0x7f4b50001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 20 client.14116.objecter dump_active .. 0 homeless 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b4e7fc700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 0x7f4b3803cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b4e7fc700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 0x7f4b3803cce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b40006fd0 tx=0x7f4b40006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.635+0000 7f4b549cf700 10 client.14116.objecter ms_handle_connect 0x7f4b3803a820 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:08.639+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4b4404ccf0 con 0x7f4b50106260 2024-03-11T21:13:09.674 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.003+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4b5007ade0 con 0x7f4b50106260 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b549cf700 1 -- 172.21.15.143:0/4248282793 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f4b44019980 con 0x7f4b50106260 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 10 librados: watch_flush enter 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 10 librados: watch_flush exit 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 20 client.14116.objecter shutdown clearing up homeless session... 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 10 client.14116.objecter successfully canceled tick 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 msgr2=0x7f4b3803cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 0x7f4b3803cce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b40006fd0 tx=0x7f4b40006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 msgr2=0x7f4b5007bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006af0 tx=0x7f4b44003750 comp rx=0 tx=0).stop 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 shutdown_connections 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4b3803a820 0x7f4b3803cce0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 --2- 172.21.15.143:0/4248282793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b50106260 0x7f4b5007bb30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 >> 172.21.15.143:0/4248282793 conn(0x7f4b50101800 msgr2=0x7f4b501022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 shutdown_connections 2024-03-11T21:13:09.675 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 -- 172.21.15.143:0/4248282793 wait complete. 2024-03-11T21:13:09.676 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:09.007+0000 7f4b559d1700 1 librados: shutdown 2024-03-11T21:13:09.676 INFO:teuthology.orchestra.run.smithi143.stdout:Enabling cephadm module... 2024-03-11T21:13:11.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:11 smithi143 bash[21512]: audit 2024-03-11T21:13:11.029933+0000 mon.a (mon.0) 27 : audit 1 from='client.? 172.21.15.143:0/806297297' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:13:11.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:11 smithi143 bash[21768]: ignoring --setuser ceph since I am not root 2024-03-11T21:13:11.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:11 smithi143 bash[21768]: ignoring --setgroup ceph since I am not root 2024-03-11T21:13:11.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:11 smithi143 bash[21768]: debug 2024-03-11T21:13:11.267+0000 7f914cff4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.551+0000 7faabd3ce700 1 Processor -- start 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.551+0000 7faabd3ce700 1 -- start start 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.551+0000 7faabd3ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.551+0000 7faabd3ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8108f00 con 0x7faab8108510 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48638/0 (socket says 172.21.15.143:48638) 2024-03-11T21:13:11.704 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 -- 172.21.15.143:0/2886563275 learned_addr learned my addr 172.21.15.143:0/2886563275 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 -- 172.21.15.143:0/2886563275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8109710 con 0x7faab8108510 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- 172.21.15.143:0/2886563275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d57a3eed43a166e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab5ffb700 1 -- 172.21.15.143:0/2886563275 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faaa8004160 con 0x7faab8108510 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab5ffb700 1 -- 172.21.15.143:0/2886563275 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7faaa8004300 con 0x7faab8108510 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab5ffb700 1 -- 172.21.15.143:0/2886563275 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faaa8011960 con 0x7faab8108510 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/2886563275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 msgr2=0x7faab8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 --2- 172.21.15.143:0/2886563275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa8009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/2886563275 shutdown_connections 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 --2- 172.21.15.143:0/2886563275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab8108930 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/2886563275 >> 172.21.15.143:0/2886563275 conn(0x7faab807b8e0 msgr2=0x7faab8106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/2886563275 shutdown_connections 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/2886563275 wait complete. 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 librados: starting msgr at 2024-03-11T21:13:11.705 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 librados: starting objecter 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 Processor -- start 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- start start 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 librados: setting wanted keys 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 librados: calling monclient init 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab810c940 con 0x7faab8108510 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48646/0 (socket says 172.21.15.143:48646) 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 -- 172.21.15.143:0/806297297 learned_addr learned my addr 172.21.15.143:0/806297297 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 -- 172.21.15.143:0/806297297 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab819c9b0 con 0x7faab8108510 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faab6ffd700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faaa8000c00 tx=0x7faaa8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faaaffff700 10 client.?.objecter ms_handle_connect 0x7faab8108510 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faaaffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:11.706 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faaa80038b0 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7faaa80248a0 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faaa801a6a0 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab819cc40 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab819f930 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.555+0000 7faabd3ce700 1 librados: init done 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7faaa8024440 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8063410 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 0x7faaa403cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faaa804d290 con 0x7faab8108510 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 10 client.14118.objecter ms_dispatch 0x7faab8001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 20 client.14118.objecter dump_active .. 0 homeless 2024-03-11T21:13:11.707 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faab67fc700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 0x7faaa403cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faab67fc700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 0x7faaa403cd90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faaa000ad30 tx=0x7faaa00093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.559+0000 7faaaffff700 10 client.14118.objecter ms_handle_connect 0x7faaa403a8d0 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:10.563+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7faaa804ccf0 con 0x7faab8108510 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.027+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7faab819fcd0 con 0x7faab8108510 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.079+0000 7faaaffff700 1 -- 172.21.15.143:0/806297297 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7faaa8004300 con 0x7faab8108510 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 10 librados: watch_flush enter 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 10 librados: watch_flush exit 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 20 client.14118.objecter shutdown clearing up homeless session... 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 10 client.14118.objecter successfully canceled tick 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 msgr2=0x7faaa403cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 0x7faaa403cd90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faaa000ad30 tx=0x7faaa00093f0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 msgr2=0x7faab819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faaa8000c00 tx=0x7faaa8003dc0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.708 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 shutdown_connections 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7faaa403a8d0 0x7faaa403cd90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 --2- 172.21.15.143:0/806297297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faab8108510 0x7faab819c470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 >> 172.21.15.143:0/806297297 conn(0x7faab807b8e0 msgr2=0x7faab8105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 shutdown_connections 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 -- 172.21.15.143:0/806297297 wait complete. 2024-03-11T21:13:11.709 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:11.083+0000 7faabd3ce700 1 librados: shutdown 2024-03-11T21:13:12.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:12 smithi143 bash[21512]: audit 2024-03-11T21:13:11.081400+0000 mon.a (mon.0) 28 : audit 1 from='client.? 172.21.15.143:0/806297297' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:13:12.328 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:12 smithi143 bash[21512]: cluster 2024-03-11T21:13:11.086650+0000 mon.a (mon.0) 29 : cluster 0 mgrmap e5: y(active, since 7s) 2024-03-11T21:13:13.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:13 smithi143 bash[21512]: audit 2024-03-11T21:13:12.998885+0000 mon.a (mon.0) 30 : audit 0 from='client.? 172.21.15.143:0/681089800' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:13:13.179 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:12 smithi143 bash[21768]: debug 2024-03-11T21:13:12.866+0000 7f914cff4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:13:13.179 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:12 smithi143 bash[21768]: debug 2024-03-11T21:13:12.966+0000 7f914cff4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:13:13.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:13 smithi143 bash[21768]: debug 2024-03-11T21:13:13.218+0000 7f914cff4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:13:13.580 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 Processor -- start 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- start start 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d50106c50 con 0x7f4d50106260 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48666/0 (socket says 172.21.15.143:48666) 2024-03-11T21:13:13.581 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 -- 172.21.15.143:0/2442360032 learned_addr learned my addr 172.21.15.143:0/2442360032 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 -- 172.21.15.143:0/2442360032 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d50107460 con 0x7f4d50106260 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- 172.21.15.143:0/2442360032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098a0 tx=0x7f4d40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbe935e94840bda server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4e7fc700 1 -- 172.21.15.143:0/2442360032 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d40004160 con 0x7f4d50106260 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4e7fc700 1 -- 172.21.15.143:0/2442360032 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4d40004300 con 0x7f4d50106260 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4e7fc700 1 -- 172.21.15.143:0/2442360032 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d40011960 con 0x7f4d50106260 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- 172.21.15.143:0/2442360032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 msgr2=0x7f4d50106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/2442360032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098a0 tx=0x7f4d40009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- 172.21.15.143:0/2442360032 shutdown_connections 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/2442360032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d50106680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- 172.21.15.143:0/2442360032 >> 172.21.15.143:0/2442360032 conn(0x7f4d50101800 msgr2=0x7f4d50103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- 172.21.15.143:0/2442360032 shutdown_connections 2024-03-11T21:13:13.582 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- 172.21.15.143:0/2442360032 wait complete. 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 librados: starting msgr at 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 librados: starting objecter 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 Processor -- start 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- start start 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 librados: setting wanted keys 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 librados: calling monclient init 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d55c0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5010a690 con 0x7f4d50106260 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:13.583 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48674/0 (socket says 172.21.15.143:48674) 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 -- 172.21.15.143:0/681089800 learned_addr learned my addr 172.21.15.143:0/681089800 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 -- 172.21.15.143:0/681089800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5019c8b0 con 0x7f4d50106260 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4f7fe700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006af0 tx=0x7f4d40003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4cff9700 10 client.?.objecter ms_handle_connect 0x7f4d50106260 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.498+0000 7f4d4cff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d40011950 con 0x7f4d50106260 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4d4001a890 con 0x7f4d50106260 2024-03-11T21:13:13.584 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d40019700 con 0x7f4d50106260 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5019cb40 con 0x7f4d50106260 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d55c0c700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5019f830 con 0x7f4d50106260 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d55c0c700 1 librados: init done 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4d40019930 con 0x7f4d50106260 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 0x7f4d3803cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4d4004d320 con 0x7f4d50106260 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 10 client.14122.objecter ms_dispatch 0x7f4d50001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:13.585 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4effd700 1 -- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 msgr2=0x7f4d3803cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4cff9700 20 client.14122.objecter dump_active .. 0 homeless 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d4effd700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 0x7f4d3803cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.502+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d50063410 con 0x7f4d50106260 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.506+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4d4004cd40 con 0x7f4d50106260 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.706+0000 7f4d4effd700 1 -- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 msgr2=0x7f4d3803cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.706+0000 7f4d4effd700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 0x7f4d3803cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.994+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4d5019fb40 con 0x7f4d50106260 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.994+0000 7f4d4cff9700 1 -- 172.21.15.143:0/681089800 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f4d4001aa40 con 0x7f4d50106260 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 10 librados: watch_flush enter 2024-03-11T21:13:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 10 librados: watch_flush exit 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 20 client.14122.objecter shutdown clearing up homeless session... 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 10 client.14122.objecter successfully canceled tick 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 msgr2=0x7f4d3803cd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 0x7f4d3803cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 msgr2=0x7f4d5019c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006af0 tx=0x7f4d40003750 comp rx=0 tx=0).stop 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 shutdown_connections 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f4d3803a8d0 0x7f4d3803cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 --2- 172.21.15.143:0/681089800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d50106260 0x7f4d5019c370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 >> 172.21.15.143:0/681089800 conn(0x7f4d50101800 msgr2=0x7f4d501022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 shutdown_connections 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 -- 172.21.15.143:0/681089800 wait complete. 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:12.998+0000 7f4d55c0c700 1 librados: shutdown 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for the mgr to restart... 2024-03-11T21:13:13.587 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mgr epoch 5... 2024-03-11T21:13:14.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.018+0000 7f914cff4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:13:14.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.126+0000 7f914cff4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:13:14.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.234+0000 7f914cff4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:13:14.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.594+0000 7f914cff4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:13:14.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.710+0000 7f914cff4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:13:14.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:14 smithi143 bash[21768]: debug 2024-03-11T21:13:14.810+0000 7f914cff4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:13:15.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:15 smithi143 bash[21768]: debug 2024-03-11T21:13:15.166+0000 7f914cff4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:13:15.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:15 smithi143 bash[21768]: debug 2024-03-11T21:13:15.406+0000 7f914cff4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:13:15.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:15 smithi143 bash[21768]: debug 2024-03-11T21:13:15.666+0000 7f914cff4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:13:15.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:15 smithi143 bash[21768]: debug 2024-03-11T21:13:15.778+0000 7f914cff4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:13:16.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:16 smithi143 bash[21768]: debug 2024-03-11T21:13:16.050+0000 7f914cff4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:13:16.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:16 smithi143 bash[21768]: debug 2024-03-11T21:13:16.290+0000 7f914cff4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:16.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:16 smithi143 bash[21768]: debug 2024-03-11T21:13:16.702+0000 7f914cff4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:13:16.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:16 smithi143 bash[21768]: debug 2024-03-11T21:13:16.818+0000 7f914cff4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:13:17.450 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:17 smithi143 bash[21768]: debug 2024-03-11T21:13:17.062+0000 7f914cff4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:18.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:18 smithi143 bash[21768]: debug 2024-03-11T21:13:18.062+0000 7f914cff4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:13:18.448 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:18 smithi143 bash[21768]: debug 2024-03-11T21:13:18.174+0000 7f914cff4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:13:18.947 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:18 smithi143 bash[21768]: debug 2024-03-11T21:13:18.466+0000 7f914cff4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:13:18.948 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:18 smithi143 bash[21768]: debug 2024-03-11T21:13:18.578+0000 7f914cff4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:13:19.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21768]: debug 2024-03-11T21:13:19.026+0000 7f914cff4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21768]: debug 2024-03-11T21:13:19.146+0000 7f914cff4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: cluster 2024-03-11T21:13:19.169823+0000 mon.a (mon.0) 31 : cluster 1 Active manager daemon y restarted 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: cluster 2024-03-11T21:13:19.170505+0000 mon.a (mon.0) 32 : cluster 1 Activating manager daemon y 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: cluster 2024-03-11T21:13:19.180988+0000 mon.a (mon.0) 33 : cluster 0 osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: cluster 2024-03-11T21:13:19.181238+0000 mon.a (mon.0) 34 : cluster 0 mgrmap e6: y(active, starting, since 0.0110273s) 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: audit 2024-03-11T21:13:19.183810+0000 mon.a (mon.0) 35 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: audit 2024-03-11T21:13:19.186078+0000 mon.a (mon.0) 36 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:13:19.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: audit 2024-03-11T21:13:19.187149+0000 mon.a (mon.0) 37 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:13:19.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: audit 2024-03-11T21:13:19.187371+0000 mon.a (mon.0) 38 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:13:19.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: audit 2024-03-11T21:13:19.187558+0000 mon.a (mon.0) 39 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:13:19.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:19 smithi143 bash[21512]: cluster 2024-03-11T21:13:19.213823+0000 mon.a (mon.0) 40 : cluster 1 Manager daemon y is now available 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: cephadm 2024-03-11T21:13:19.232972+0000 mgr.y (mgr.14120) 1 : cephadm 1 Found migration_current of "None". Setting to last migration. 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.237321+0000 mon.a (mon.0) 41 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.243349+0000 mon.a (mon.0) 42 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.248196+0000 mon.a (mon.0) 43 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.250383+0000 mon.a (mon.0) 44 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.254387+0000 mon.a (mon.0) 45 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.262762+0000 mon.a (mon.0) 46 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: audit 2024-03-11T21:13:19.265955+0000 mon.a (mon.0) 47 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:13:20.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:20 smithi143 bash[21512]: cluster 2024-03-11T21:13:20.182710+0000 mon.a (mon.0) 48 : cluster 0 mgrmap e7: y(active, since 1.01249s) 2024-03-11T21:13:20.800 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 Processor -- start 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- start start 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376007a850 con 0x7f376007be20 2024-03-11T21:13:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f37652e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f37652e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52338/0 (socket says 172.21.15.143:52338) 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f37652e2700 1 -- 172.21.15.143:0/832975024 learned_addr learned my addr 172.21.15.143:0/832975024 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f37652e2700 1 -- 172.21.15.143:0/832975024 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376007a9d0 con 0x7f376007be20 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f37652e2700 1 --2- 172.21.15.143:0/832975024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37500098a0 tx=0x7f3750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293c7c82df33b477 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3757fff700 1 -- 172.21.15.143:0/832975024 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3750004160 con 0x7f376007be20 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3757fff700 1 -- 172.21.15.143:0/832975024 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f3750004300 con 0x7f376007be20 2024-03-11T21:13:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3757fff700 1 -- 172.21.15.143:0/832975024 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3750011960 con 0x7f376007be20 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- 172.21.15.143:0/832975024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 msgr2=0x7f376007a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 --2- 172.21.15.143:0/832975024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37500098a0 tx=0x7f3750009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- 172.21.15.143:0/832975024 shutdown_connections 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 --2- 172.21.15.143:0/832975024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376007a280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- 172.21.15.143:0/832975024 >> 172.21.15.143:0/832975024 conn(0x7f3760101800 msgr2=0x7f3760103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- 172.21.15.143:0/832975024 shutdown_connections 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 -- 172.21.15.143:0/832975024 wait complete. 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 librados: starting msgr at 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.402+0000 7f3767546700 1 librados: starting objecter 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 Processor -- start 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 -- start start 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 librados: setting wanted keys 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 librados: calling monclient init 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376010c790 con 0x7f376007be20 2024-03-11T21:13:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37652e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:20.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37652e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52344/0 (socket says 172.21.15.143:52344) 2024-03-11T21:13:20.832 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37652e2700 1 -- 172.21.15.143:0/1742599282 learned_addr learned my addr 172.21.15.143:0/1742599282 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37652e2700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376019c840 con 0x7f376007be20 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37652e2700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3750011670 tx=0x7f37500117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 10 client.?.objecter ms_handle_connect 0x7f376007be20 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f375001a430 con 0x7f376007be20 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f3750020070 con 0x7f376007be20 2024-03-11T21:13:20.833 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376019cad0 con 0x7f376007be20 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3750019660 con 0x7f376007be20 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376019f7c0 con 0x7f376007be20 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 librados: init done 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3750019920 con 0x7f376007be20 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3760063410 con 0x7f374c03a8d0 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f375004d570 con 0x7f376007be20 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 10 client.14124.objecter ms_dispatch 0x7f3760001550 osd_map(1..1 src has 1..1) v4 2024-03-11T21:13:20.834 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 3 client.14124.objecter handle_osd_map got epochs [1,1] > 0 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3764ae1700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 3 client.14124.objecter handle_osd_map decoding full epoch 1 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.406+0000 7f37567fc700 20 client.14124.objecter dump_active .. 0 homeless 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.610+0000 7f3764ae1700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:14.610+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:15.014+0000 7f3764ae1700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:15.014+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:15.818+0000 7f3764ae1700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:15.818+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:17.422+0000 7f3764ae1700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1839244747 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:17.422+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T21:13:20.835 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:19.174+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f375001abc0 con 0x7f376007be20 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:19.174+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 msgr2=0x7f374c03cd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:19.174+0000 7f37567fc700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:19.410+0000 7f3766d45700 10 client.14124.objecter tick 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.174+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f375004ebc0 con 0x7f376007be20 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.174+0000 7f37567fc700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 0x7f374c0403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.174+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3750012ea0 con 0x7f374c03dfb0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.174+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 0x7f374c0403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.178+0000 7f3764ae1700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 0x7f374c0403a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f375c003a10 tx=0x7f375c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.178+0000 7f37567fc700 10 client.14124.objecter ms_handle_connect 0x7f374c03dfb0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.178+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f3750012ea0 con 0x7f374c03dfb0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3760102a10 con 0x7f374c03dfb0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f37567fc700 1 -- 172.21.15.143:0/1742599282 <== mgr.14120 v2:172.21.15.143:6800/3787195463 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3760102a10 con 0x7f374c03dfb0 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 10 librados: watch_flush enter 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 10 librados: watch_flush exit 2024-03-11T21:13:20.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 20 client.14124.objecter shutdown clearing up homeless session... 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 10 client.14124.objecter successfully canceled tick 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 msgr2=0x7f374c0403a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 0x7f374c0403a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f375c003a10 tx=0x7f375c009290 comp rx=0 tx=0).stop 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 msgr2=0x7f376019c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3750011670 tx=0x7f37500117f0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 shutdown_connections 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f374c03dfb0 0x7f374c0403a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376007be20 0x7f376019c300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 --2- 172.21.15.143:0/1742599282 >> [v2:172.21.15.143:6800/1839244747,v1:172.21.15.143:6801/1839244747] conn(0x7f374c03a8d0 0x7f374c03cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 >> 172.21.15.143:0/1742599282 conn(0x7f3760101800 msgr2=0x7f3760102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 shutdown_connections 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 -- 172.21.15.143:0/1742599282 wait complete. 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:20.190+0000 7f3767546700 1 librados: shutdown 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:mgr epoch 5 is available 2024-03-11T21:13:20.837 INFO:teuthology.orchestra.run.smithi143.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: audit 2024-03-11T21:13:20.181752+0000 mgr.y (mgr.14120) 2 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: audit 2024-03-11T21:13:20.193850+0000 mgr.y (mgr.14120) 3 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: audit 2024-03-11T21:13:20.559564+0000 mon.a (mon.0) 49 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: audit 2024-03-11T21:13:20.565358+0000 mon.a (mon.0) 50 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: cephadm 2024-03-11T21:13:20.861640+0000 mgr.y (mgr.14120) 4 : cephadm 1 [11/Mar/2024:21:13:20] ENGINE Bus STARTING 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: cephadm 2024-03-11T21:13:20.976989+0000 mgr.y (mgr.14120) 5 : cephadm 1 [11/Mar/2024:21:13:20] ENGINE Serving on https://172.21.15.143:7150 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: cephadm 2024-03-11T21:13:21.078173+0000 mgr.y (mgr.14120) 6 : cephadm 1 [11/Mar/2024:21:13:21] ENGINE Serving on http://172.21.15.143:8765 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: cephadm 2024-03-11T21:13:21.078333+0000 mgr.y (mgr.14120) 7 : cephadm 1 [11/Mar/2024:21:13:21] ENGINE Bus STARTED 2024-03-11T21:13:21.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:21 smithi143 bash[21512]: audit 2024-03-11T21:13:21.126455+0000 mon.a (mon.0) 51 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:22.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:22 smithi143 bash[21512]: cluster 2024-03-11T21:13:21.566245+0000 mon.a (mon.0) 52 : cluster 0 mgrmap e8: y(active, since 2s) 2024-03-11T21:13:22.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:22 smithi143 bash[21512]: audit 2024-03-11T21:13:22.058370+0000 mgr.y (mgr.14120) 8 : audit 0 from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:22.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:22 smithi143 bash[21512]: audit 2024-03-11T21:13:22.067361+0000 mon.a (mon.0) 53 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:22.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:22 smithi143 bash[21512]: audit 2024-03-11T21:13:22.096089+0000 mon.a (mon.0) 54 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.649+0000 7f20d65a0700 1 Processor -- start 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- start start 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d010e8a0 con 0x7f20d010deb0 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52412/0 (socket says 172.21.15.143:52412) 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 -- 172.21.15.143:0/2516527072 learned_addr learned my addr 172.21.15.143:0/2516527072 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:22.637 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 -- 172.21.15.143:0/2516527072 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d010ea20 con 0x7f20d010deb0 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- 172.21.15.143:0/2516527072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20c00098a0 tx=0x7f20c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d486ab28d886df57 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cf7fe700 1 -- 172.21.15.143:0/2516527072 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20c0004160 con 0x7f20d010deb0 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cf7fe700 1 -- 172.21.15.143:0/2516527072 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f20c0004300 con 0x7f20d010deb0 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cf7fe700 1 -- 172.21.15.143:0/2516527072 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20c0011960 con 0x7f20d010deb0 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/2516527072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 msgr2=0x7f20d010e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 --2- 172.21.15.143:0/2516527072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20c00098a0 tx=0x7f20c0009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/2516527072 shutdown_connections 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 --2- 172.21.15.143:0/2516527072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d010e2d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/2516527072 >> 172.21.15.143:0/2516527072 conn(0x7f20d007b8e0 msgr2=0x7f20d007dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/2516527072 shutdown_connections 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/2516527072 wait complete. 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 librados: starting msgr at 2024-03-11T21:13:22.638 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 librados: starting objecter 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 Processor -- start 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- start start 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 librados: setting wanted keys 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 librados: calling monclient init 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d010da20 con 0x7f20d010deb0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52422/0 (socket says 172.21.15.143:52422) 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 -- 172.21.15.143:0/432750733 learned_addr learned my addr 172.21.15.143:0/432750733 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 -- 172.21.15.143:0/432750733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d01a0d70 con 0x7f20d010deb0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cffff700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20c0006af0 tx=0x7f20c0003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cdffb700 10 client.?.objecter ms_handle_connect 0x7f20d010deb0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cdffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20c0011950 con 0x7f20d010deb0 2024-03-11T21:13:22.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f20c001a890 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d01a1000 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20c00196e0 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d01a3cf0 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.653+0000 7f20d65a0700 1 librados: init done 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f20c00198d0 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d0110ed0 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 0x7f20bc03cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f20c004d470 con 0x7f20d010deb0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 10 client.14132.objecter ms_dispatch 0x7f20d0001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.657+0000 7f20cdffb700 20 client.14132.objecter dump_active .. 0 homeless 2024-03-11T21:13:22.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.661+0000 7f20c77fe700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 0x7f20bc03cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.661+0000 7f20c77fe700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 0x7f20bc03cd90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006fd0 tx=0x7f20b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.661+0000 7f20cdffb700 10 client.14132.objecter ms_handle_connect 0x7f20bc03a8d0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:21.661+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f20c00042c0 con 0x7f20d010deb0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.053+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d007cb10 con 0x7f20bc03a8d0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20cdffb700 1 -- 172.21.15.143:0/432750733 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f20d007cb10 con 0x7f20bc03a8d0 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 10 librados: watch_flush enter 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 10 librados: watch_flush exit 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 20 client.14132.objecter shutdown clearing up homeless session... 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 10 client.14132.objecter successfully canceled tick 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 msgr2=0x7f20bc03cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 0x7f20bc03cd90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006fd0 tx=0x7f20b8006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 msgr2=0x7f20d01a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20c0006af0 tx=0x7f20c0003980 comp rx=0 tx=0).stop 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 shutdown_connections 2024-03-11T21:13:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f20bc03a8d0 0x7f20bc03cd90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:22.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 --2- 172.21.15.143:0/432750733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20d010deb0 0x7f20d01a0830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:22.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 >> 172.21.15.143:0/432750733 conn(0x7f20d007b8e0 msgr2=0x7f20d007c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:22.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 shutdown_connections 2024-03-11T21:13:22.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 -- 172.21.15.143:0/432750733 wait complete. 2024-03-11T21:13:22.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:22.065+0000 7f20d65a0700 1 librados: shutdown 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f139a5f2700 1 Processor -- start 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f139a5f2700 1 -- start start 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f139a5f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f139a5f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394106c50 con 0x7f1394106260 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f1393fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f1393fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43150/0 (socket says 172.21.15.143:43150) 2024-03-11T21:13:24.447 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.545+0000 7f1393fff700 1 -- 172.21.15.143:0/2557334828 learned_addr learned my addr 172.21.15.143:0/2557334828 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 -- 172.21.15.143:0/2557334828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1394107460 con 0x7f1394106260 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 --2- 172.21.15.143:0/2557334828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1384009e80 tx=0x7f138400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81704413dd4bf873 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1392ffd700 1 -- 172.21.15.143:0/2557334828 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1384017070 con 0x7f1394106260 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1392ffd700 1 -- 172.21.15.143:0/2557334828 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f138400b8c0 con 0x7f1394106260 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1392ffd700 1 -- 172.21.15.143:0/2557334828 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1384004340 con 0x7f1394106260 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/2557334828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 msgr2=0x7f1394106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 --2- 172.21.15.143:0/2557334828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1384009e80 tx=0x7f138400b330 comp rx=0 tx=0).stop 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/2557334828 shutdown_connections 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 --2- 172.21.15.143:0/2557334828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f1394106680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/2557334828 >> 172.21.15.143:0/2557334828 conn(0x7f1394101800 msgr2=0x7f1394103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:24.448 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/2557334828 shutdown_connections 2024-03-11T21:13:24.449 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/2557334828 wait complete. 2024-03-11T21:13:24.450 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 librados: starting msgr at 2024-03-11T21:13:24.450 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 librados: starting objecter 2024-03-11T21:13:24.450 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 Processor -- start 2024-03-11T21:13:24.450 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- start start 2024-03-11T21:13:24.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:24 smithi143 bash[21512]: audit 2024-03-11T21:13:23.918836+0000 mgr.y (mgr.14120) 9 : audit 0 from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 librados: setting wanted keys 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 librados: calling monclient init 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139410a690 con 0x7f1394106260 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43162/0 (socket says 172.21.15.143:43162) 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 -- 172.21.15.143:0/3493262922 learned_addr learned my addr 172.21.15.143:0/3493262922 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 -- 172.21.15.143:0/3493262922 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139419c7c0 con 0x7f1394106260 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f1393fff700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f138400b7b0 tx=0x7f138400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 10 client.?.objecter ms_handle_connect 0x7f1394106260 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:24.456 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1384021020 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f1384003b50 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139419ca50 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1384017040 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139419f740 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f139a5f2700 1 librados: init done 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.549+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1384024020 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1394063410 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 0x7f137c03cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f138400f050 con 0x7f1394106260 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 10 client.14134.objecter ms_dispatch 0x7f1394001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 20 client.14134.objecter dump_active .. 0 homeless 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13937fe700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 0x7f137c03cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13917fa700 10 client.14134.objecter ms_handle_connect 0x7f137c03a900 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.553+0000 7f13937fe700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 0x7f137c03cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1388006fd0 tx=0x7f1388006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.557+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f138402b020 con 0x7f1394106260 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.913+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f139407a820 con 0x7f137c03a900 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.913+0000 7f13917fa700 1 -- 172.21.15.143:0/3493262922 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f139407a820 con 0x7f137c03a900 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 10 librados: watch_flush enter 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 10 librados: watch_flush exit 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 20 client.14134.objecter shutdown clearing up homeless session... 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 10 client.14134.objecter successfully canceled tick 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 msgr2=0x7f137c03cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 0x7f137c03cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1388006fd0 tx=0x7f1388006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:24.458 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 msgr2=0x7f139419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f138400b7b0 tx=0x7f138400bf10 comp rx=0 tx=0).stop 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 shutdown_connections 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f137c03a900 0x7f137c03cdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 --2- 172.21.15.143:0/3493262922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1394106260 0x7f139419c280 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 >> 172.21.15.143:0/3493262922 conn(0x7f1394101800 msgr2=0x7f139407a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 shutdown_connections 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 -- 172.21.15.143:0/3493262922 wait complete. 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:23.917+0000 7f139a5f2700 1 librados: shutdown 2024-03-11T21:13:24.459 INFO:teuthology.orchestra.run.smithi143.stdout:Generating ssh key... 2024-03-11T21:13:26.450 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: Generating public/private rsa key pair. 2024-03-11T21:13:26.450 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: Your identification has been saved in /tmp/tmpqn1g6cff/key. 2024-03-11T21:13:26.450 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: Your public key has been saved in /tmp/tmpqn1g6cff/key.pub. 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: The key fingerprint is: 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: SHA256:hP6Z+OdSSremK1uyWhC6S3mZbCBaYt7O638Ig2JQurE ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: The key's randomart image is: 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: +---[RSA 3072]----+ 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: | | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: | . . | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: | o . . . | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: |+ . o . | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: |+== . . S | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: |E*.B + o.oo | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: |+.+.O =.=+ . | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: | .o+ o.*o + | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: | o=+o+o+B. | 2024-03-11T21:13:26.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:26 smithi143 bash[21768]: +----[SHA256]-----+ 2024-03-11T21:13:26.639 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.325+0000 7f90504cc700 1 Processor -- start 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.325+0000 7f90504cc700 1 -- start start 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.325+0000 7f90504cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.325+0000 7f90504cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9048106c50 con 0x7f9048106260 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43164/0 (socket says 172.21.15.143:43164) 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 -- 172.21.15.143:0/3900063477 learned_addr learned my addr 172.21.15.143:0/3900063477 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:26.640 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 -- 172.21.15.143:0/3900063477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048107460 con 0x7f9048106260 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- 172.21.15.143:0/3900063477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f90380098a0 tx=0x7f9038009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc704be08f6f4de server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904d266700 1 -- 172.21.15.143:0/3900063477 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9038004160 con 0x7f9048106260 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904d266700 1 -- 172.21.15.143:0/3900063477 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f9038004300 con 0x7f9048106260 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904d266700 1 -- 172.21.15.143:0/3900063477 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9038011960 con 0x7f9048106260 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/3900063477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 msgr2=0x7f9048106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 --2- 172.21.15.143:0/3900063477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f90380098a0 tx=0x7f9038009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/3900063477 shutdown_connections 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 --2- 172.21.15.143:0/3900063477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f9048106680 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:26.641 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/3900063477 >> 172.21.15.143:0/3900063477 conn(0x7f9048101800 msgr2=0x7f9048103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/3900063477 shutdown_connections 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/3900063477 wait complete. 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 librados: starting msgr at 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 librados: starting objecter 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 Processor -- start 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- start start 2024-03-11T21:13:26.642 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 librados: setting wanted keys 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 librados: calling monclient init 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904810a690 con 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43174/0 (socket says 172.21.15.143:43174) 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 -- 172.21.15.143:0/1101550804 learned_addr learned my addr 172.21.15.143:0/1101550804 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 -- 172.21.15.143:0/1101550804 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904819c8b0 con 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f904e268700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9038003ea0 tx=0x7f90380117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f903f7fe700 10 client.?.objecter ms_handle_connect 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f903f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90380038b0 con 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f903801a890 con 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90380247b0 con 0x7f9048106260 2024-03-11T21:13:26.643 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904819cb40 con 0x7f9048106260 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904819f830 con 0x7f9048106260 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.329+0000 7f90504cc700 1 librados: init done 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9038022020 con 0x7f9048106260 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9048063410 con 0x7f9048106260 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 0x7f903403cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f903804d220 con 0x7f9048106260 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 10 client.14136.objecter ms_dispatch 0x7f9048001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 20 client.14136.objecter dump_active .. 0 homeless 2024-03-11T21:13:26.644 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f904da67700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 0x7f903403cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f904da67700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 0x7f903403cde0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.333+0000 7f903f7fe700 10 client.14136.objecter ms_handle_connect 0x7f903403a920 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.337+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f903800b3e0 con 0x7f9048106260 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:25.689+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f90481029b0 con 0x7f903403a920 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.057+0000 7f903f7fe700 1 -- 172.21.15.143:0/1101550804 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f90481029b0 con 0x7f903403a920 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 10 librados: watch_flush enter 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 10 librados: watch_flush exit 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 20 client.14136.objecter shutdown clearing up homeless session... 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 10 client.14136.objecter successfully canceled tick 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 msgr2=0x7f903403cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 0x7f903403cde0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 msgr2=0x7f904819c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9038003ea0 tx=0x7f90380117f0 comp rx=0 tx=0).stop 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 shutdown_connections 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f903403a920 0x7f903403cde0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 --2- 172.21.15.143:0/1101550804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9048106260 0x7f904819c370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 >> 172.21.15.143:0/1101550804 conn(0x7f9048101800 msgr2=0x7f90481022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 shutdown_connections 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 -- 172.21.15.143:0/1101550804 wait complete. 2024-03-11T21:13:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:26.061+0000 7f90504cc700 1 librados: shutdown 2024-03-11T21:13:27.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:27 smithi143 bash[21512]: audit 2024-03-11T21:13:25.694822+0000 mgr.y (mgr.14120) 10 : audit 0 from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:27.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:27 smithi143 bash[21512]: cephadm 2024-03-11T21:13:25.695056+0000 mgr.y (mgr.14120) 11 : cephadm 1 Generating ssh key... 2024-03-11T21:13:27.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:27 smithi143 bash[21512]: audit 2024-03-11T21:13:26.055545+0000 mon.a (mon.0) 55 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:27.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:27 smithi143 bash[21512]: audit 2024-03-11T21:13:26.061133+0000 mon.a (mon.0) 56 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:28.352 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2rHwTO1Q7RJEqDeyO2sJ2Gr9K79tP6UEF+o4b4G5XAJb10vunWzLueHGxNWb0sYuhi4LwFRcy1beuq++uCcTPz7Ska6V4IWfPUlrAGxp6KPIgHM8XXkSBbFJQpS52LIffRNcIkbsUbdsdSObT8F1MtfDaJtINo+V8/XUGjdU5b20LxY6e8GJR8asBPKoZux+DmFkHWAShPqjwBFQ/40UBUytpcP5UthZGpL6SqjyQ6MHvuaqlVeQuWGDe69Ycg7cOsukFMvb7xMTDG7CJrqpGnuDBIoP81taINR5AaCwFQSGMY1HLg6KWTLFcT8BWkAvy0mgX8kBmr2EbANf4W+owLbLhvbyq3mbfNOJiywQwPnfXLc8Pf5U0fY/KQ8WlypJJAiiBcc4/7dG4IKMc5YGAb6b1qePTLKt/s/Lmen6B2xdCXXF47ItyUscO+y9tn778Hj8fRmv+1QRdoOEhyG1ENMpNANa9ZLtxwfEv4wLwQBXrP9bwRirj+jyPEA6phGk= ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:13:28.352 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.437+0000 7ff5fe4c5700 1 Processor -- start 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.437+0000 7ff5fe4c5700 1 -- start start 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.437+0000 7ff5fe4c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.437+0000 7ff5fe4c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8106c50 con 0x7ff5f8106260 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.437+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43190/0 (socket says 172.21.15.143:43190) 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 -- 172.21.15.143:0/2420456872 learned_addr learned my addr 172.21.15.143:0/2420456872 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 -- 172.21.15.143:0/2420456872 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8107460 con 0x7ff5f8106260 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 --2- 172.21.15.143:0/2420456872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098a0 tx=0x7ff5e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4736bdab9920b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f6ffd700 1 -- 172.21.15.143:0/2420456872 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5e8004160 con 0x7ff5f8106260 2024-03-11T21:13:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f6ffd700 1 -- 172.21.15.143:0/2420456872 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7ff5e8004300 con 0x7ff5f8106260 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f6ffd700 1 -- 172.21.15.143:0/2420456872 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5e8011960 con 0x7ff5f8106260 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2420456872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 msgr2=0x7ff5f8106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2420456872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098a0 tx=0x7ff5e8009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2420456872 shutdown_connections 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2420456872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f8106680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2420456872 >> 172.21.15.143:0/2420456872 conn(0x7ff5f8101800 msgr2=0x7ff5f8103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2420456872 shutdown_connections 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2420456872 wait complete. 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 librados: starting msgr at 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 librados: starting objecter 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 Processor -- start 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- start start 2024-03-11T21:13:28.354 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 librados: setting wanted keys 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 librados: calling monclient init 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f810a690 con 0x7ff5f8106260 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43200/0 (socket says 172.21.15.143:43200) 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 -- 172.21.15.143:0/2824120869 learned_addr learned my addr 172.21.15.143:0/2824120869 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 -- 172.21.15.143:0/2824120869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f807c070 con 0x7ff5f8106260 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f7fff700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8006af0 tx=0x7ff5e8003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f57fa700 10 client.?.objecter ms_handle_connect 0x7ff5f8106260 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f57fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5e8011950 con 0x7ff5f8106260 2024-03-11T21:13:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7ff5e801a890 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f807a260 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5e8019700 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f807a7a0 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.441+0000 7ff5fe4c5700 1 librados: init done 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff5e8019980 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8063410 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 0x7ff5e403cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff5e804d450 con 0x7ff5f8106260 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 10 client.14138.objecter ms_dispatch 0x7ff5f8001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:28.356 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 20 client.14138.objecter dump_active .. 0 homeless 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f77fe700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 0x7ff5e403cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f57fa700 10 client.14138.objecter ms_handle_connect 0x7ff5e403a870 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.445+0000 7ff5f77fe700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 0x7ff5e403cd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0006fd0 tx=0x7ff5e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.449+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff5e8019d70 con 0x7ff5f8106260 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5f807ae50 con 0x7ff5e403a870 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5f57fa700 1 -- 172.21.15.143:0/2824120869 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff5f807ae50 con 0x7ff5e403a870 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 10 librados: watch_flush enter 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 10 librados: watch_flush exit 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 20 client.14138.objecter shutdown clearing up homeless session... 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 10 client.14138.objecter successfully canceled tick 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 msgr2=0x7ff5e403cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 0x7ff5e403cd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0006fd0 tx=0x7ff5e0006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 msgr2=0x7ff5f807bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8006af0 tx=0x7ff5e8003750 comp rx=0 tx=0).stop 2024-03-11T21:13:28.357 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 shutdown_connections 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7ff5e403a870 0x7ff5e403cd30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 --2- 172.21.15.143:0/2824120869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f8106260 0x7ff5f807bb30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.817+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 >> 172.21.15.143:0/2824120869 conn(0x7ff5f8101800 msgr2=0x7ff5f81022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.821+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 shutdown_connections 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.821+0000 7ff5fe4c5700 1 -- 172.21.15.143:0/2824120869 wait complete. 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:27.821+0000 7ff5fe4c5700 1 librados: shutdown 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:13:28.358 INFO:teuthology.orchestra.run.smithi143.stdout:Adding host smithi143... 2024-03-11T21:13:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:28 smithi143 bash[21512]: audit 2024-03-11T21:13:27.820866+0000 mgr.y (mgr.14120) 12 : audit 0 from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:30.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:30 smithi143 bash[21512]: audit 2024-03-11T21:13:29.659827+0000 mgr.y (mgr.14120) 13 : audit 0 from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "addr": "172.21.15.143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:31.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:31 smithi143 bash[21512]: cephadm 2024-03-11T21:13:30.249703+0000 mgr.y (mgr.14120) 14 : cephadm 1 Deploying cephadm binary to smithi143 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout Added host 'smithi143' with addr '172.21.15.143' 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 Processor -- start 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- start start 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa407a8d0 con 0x7f4fa407bea0 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:35.733 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43216/0 (socket says 172.21.15.143:43216) 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 -- 172.21.15.143:0/1732557462 learned_addr learned my addr 172.21.15.143:0/1732557462 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 -- 172.21.15.143:0/1732557462 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa407aa50 con 0x7f4fa407bea0 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 --2- 172.21.15.143:0/1732557462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f940098a0 tx=0x7f4f94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59c02cb748a4c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa37fe700 1 -- 172.21.15.143:0/1732557462 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f94004160 con 0x7f4fa407bea0 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa37fe700 1 -- 172.21.15.143:0/1732557462 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4f94004300 con 0x7f4fa407bea0 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa37fe700 1 -- 172.21.15.143:0/1732557462 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f94011960 con 0x7f4fa407bea0 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- 172.21.15.143:0/1732557462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 msgr2=0x7f4fa407a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1732557462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f940098a0 tx=0x7f4f94009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- 172.21.15.143:0/1732557462 shutdown_connections 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1732557462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa407a300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- 172.21.15.143:0/1732557462 >> 172.21.15.143:0/1732557462 conn(0x7f4fa41017e0 msgr2=0x7f4fa4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- 172.21.15.143:0/1732557462 shutdown_connections 2024-03-11T21:13:35.734 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- 172.21.15.143:0/1732557462 wait complete. 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 librados: starting msgr at 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 librados: starting objecter 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 Processor -- start 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- start start 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 librados: setting wanted keys 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 librados: calling monclient init 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4faaebe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa41083b0 con 0x7f4fa407bea0 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43224/0 (socket says 172.21.15.143:43224) 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.289+0000 7f4fa8c5a700 1 -- 172.21.15.143:0/1064016171 learned_addr learned my addr 172.21.15.143:0/1064016171 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa8c5a700 1 -- 172.21.15.143:0/1064016171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa41a0c50 con 0x7f4fa407bea0 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa8c5a700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f94003ea0 tx=0x7f4f940117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:35.735 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 10 client.?.objecter ms_handle_connect 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f940038b0 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f4f9401a890 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa41a0ee0 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4faaebe700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f940247b0 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa41a3bd0 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4faaebe700 1 librados: init done 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4f94022020 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa4063410 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 0x7f4f8c03cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4f9404d440 con 0x7f4fa407bea0 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 10 client.14140.objecter ms_dispatch 0x7f4fa4001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:35.736 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 20 client.14140.objecter dump_active .. 0 homeless 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa3fff700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 0x7f4f8c03cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa3fff700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 0x7f4f8c03cde0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f98006fd0 tx=0x7f4f98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.293+0000 7f4fa1ffb700 10 client.14140.objecter ms_handle_connect 0x7f4f8c03a920 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.297+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4f9404cd90 con 0x7f4fa407bea0 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:29.653+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "addr": "172.21.15.143", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fa402d970 con 0x7f4f8c03a920 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.296+0000 7f4faa6bd700 10 client.14140.objecter tick 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.896+0000 7f4fa1ffb700 1 -- 172.21.15.143:0/1064016171 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4fa402d970 con 0x7f4f8c03a920 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 10 librados: watch_flush enter 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 10 librados: watch_flush exit 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 20 client.14140.objecter shutdown clearing up homeless session... 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 10 client.14140.objecter successfully canceled tick 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 msgr2=0x7f4f8c03cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:35.737 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 0x7f4f8c03cde0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f98006fd0 tx=0x7f4f98006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 msgr2=0x7f4fa41a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f94003ea0 tx=0x7f4f940117f0 comp rx=0 tx=0).stop 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 shutdown_connections 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f4f8c03a920 0x7f4f8c03cde0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 --2- 172.21.15.143:0/1064016171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4fa407bea0 0x7f4fa41a0710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 >> 172.21.15.143:0/1064016171 conn(0x7f4fa41017e0 msgr2=0x7f4fa4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 shutdown_connections 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 -- 172.21.15.143:0/1064016171 wait complete. 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:34.900+0000 7f4faaebe700 1 librados: shutdown 2024-03-11T21:13:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:Deploying unmanaged mon service... 2024-03-11T21:13:36.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:35 smithi143 bash[21512]: audit 2024-03-11T21:13:34.902023+0000 mon.a (mon.0) 57 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:36.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:35 smithi143 bash[21512]: cephadm 2024-03-11T21:13:34.902443+0000 mgr.y (mgr.14120) 15 : cephadm 1 Added host smithi143 2024-03-11T21:13:36.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:35 smithi143 bash[21512]: audit 2024-03-11T21:13:34.950005+0000 mon.a (mon.0) 58 : audit 0 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.232+0000 7f40b0344700 1 Processor -- start 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.232+0000 7f40b0344700 1 -- start start 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8106c50 con 0x7f40a8106260 2024-03-11T21:13:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45838/0 (socket says 172.21.15.143:45838) 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 -- 172.21.15.143:0/3511160356 learned_addr learned my addr 172.21.15.143:0/3511160356 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 -- 172.21.15.143:0/3511160356 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a8107460 con 0x7f40a8106260 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- 172.21.15.143:0/3511160356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f40a40098a0 tx=0x7f40a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bd368afb9d39e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ad0de700 1 -- 172.21.15.143:0/3511160356 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a4004160 con 0x7f40a8106260 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ad0de700 1 -- 172.21.15.143:0/3511160356 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f40a4004300 con 0x7f40a8106260 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ad0de700 1 -- 172.21.15.143:0/3511160356 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a4011960 con 0x7f40a8106260 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- 172.21.15.143:0/3511160356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 msgr2=0x7f40a8106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 --2- 172.21.15.143:0/3511160356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f40a40098a0 tx=0x7f40a4009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- 172.21.15.143:0/3511160356 shutdown_connections 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 --2- 172.21.15.143:0/3511160356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a8106680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- 172.21.15.143:0/3511160356 >> 172.21.15.143:0/3511160356 conn(0x7f40a81017e0 msgr2=0x7f40a8103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- 172.21.15.143:0/3511160356 shutdown_connections 2024-03-11T21:13:38.302 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- 172.21.15.143:0/3511160356 wait complete. 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 librados: starting msgr at 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 librados: starting objecter 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 Processor -- start 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- start start 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 librados: setting wanted keys 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 librados: calling monclient init 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40b0344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a810a690 con 0x7f40a8106260 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45854/0 (socket says 172.21.15.143:45854) 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 -- 172.21.15.143:0/3736036868 learned_addr learned my addr 172.21.15.143:0/3736036868 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 -- 172.21.15.143:0/3736036868 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a819a690 con 0x7f40a8106260 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f40ae0e0700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40a4003ea0 tx=0x7f40a40117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:38.303 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f409f7fe700 10 client.?.objecter ms_handle_connect 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.236+0000 7f409f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a40038b0 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1024+0+0 (secure 0 0 0) 0x7f40a401a890 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40b0344700 1 -- 172.21.15.143:0/3736036868 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a819a920 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a40247b0 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40b0344700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40b0344700 1 -- 172.21.15.143:0/3736036868 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a819d610 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40b0344700 1 librados: init done 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f40a4022020 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4090005120 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 0x7f409403cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f40a404d610 con 0x7f40a8106260 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 10 client.14142.objecter ms_dispatch 0x7f40a8001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:38.304 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 20 client.14142.objecter dump_active .. 0 homeless 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40ad8df700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 0x7f409403cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f409f7fe700 10 client.14142.objecter ms_handle_connect 0x7f409403a920 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.240+0000 7f40ad8df700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 0x7f409403cde0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4098006fd0 tx=0x7f4098006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.244+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f40a4019c70 con 0x7f40a8106260 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.636+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4090000bc0 con 0x7f409403a920 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.644+0000 7f409f7fe700 1 -- 172.21.15.143:0/3736036868 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4090000bc0 con 0x7f409403a920 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 10 librados: watch_flush enter 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 10 librados: watch_flush exit 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 20 client.14142.objecter shutdown clearing up homeless session... 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 10 client.14142.objecter successfully canceled tick 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 msgr2=0x7f409403cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:38.305 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 0x7f409403cde0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4098006fd0 tx=0x7f4098006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 msgr2=0x7f40a819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40a4003ea0 tx=0x7f40a40117f0 comp rx=0 tx=0).stop 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 shutdown_connections 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f409403a920 0x7f409403cde0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 --2- 172.21.15.143:0/3736036868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a8106260 0x7f40a819a150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 >> 172.21.15.143:0/3736036868 conn(0x7f40a81017e0 msgr2=0x7f40a8102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 shutdown_connections 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 -- 172.21.15.143:0/3736036868 wait complete. 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:37.648+0000 7f409d7fa700 1 librados: shutdown 2024-03-11T21:13:38.306 INFO:teuthology.orchestra.run.smithi143.stdout:Deploying unmanaged mgr service... 2024-03-11T21:13:38.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:38 smithi143 bash[21512]: audit 2024-03-11T21:13:37.643644+0000 mgr.y (mgr.14120) 16 : audit 0 from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:38.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:38 smithi143 bash[21512]: cephadm 2024-03-11T21:13:37.645600+0000 mgr.y (mgr.14120) 17 : cephadm 1 Saving service mon spec with placement count:5 2024-03-11T21:13:38.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:38 smithi143 bash[21512]: audit 2024-03-11T21:13:37.649009+0000 mon.a (mon.0) 59 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:38.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:38 smithi143 bash[21512]: audit 2024-03-11T21:13:38.026010+0000 mon.a (mon.0) 60 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:38.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:38 smithi143 bash[21512]: audit 2024-03-11T21:13:38.562060+0000 mon.a (mon.0) 61 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02f536c700 1 Processor -- start 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02f536c700 1 -- start start 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02f536c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02f536c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f007d210 con 0x7f02f007c820 2024-03-11T21:13:40.207 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45856/0 (socket says 172.21.15.143:45856) 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.248+0000 7f02eeffd700 1 -- 172.21.15.143:0/3698393365 learned_addr learned my addr 172.21.15.143:0/3698393365 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 -- 172.21.15.143:0/3698393365 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f007da70 con 0x7f02f007c820 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 --2- 172.21.15.143:0/3698393365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f02e00098a0 tx=0x7f02e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac6fb45301aa6b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02edffb700 1 -- 172.21.15.143:0/3698393365 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02e0004160 con 0x7f02f007c820 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02edffb700 1 -- 172.21.15.143:0/3698393365 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f02e0004300 con 0x7f02f007c820 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02edffb700 1 -- 172.21.15.143:0/3698393365 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02e0011930 con 0x7f02f007c820 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/3698393365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 msgr2=0x7f02f007cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 --2- 172.21.15.143:0/3698393365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f02e00098a0 tx=0x7f02e0009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:40.208 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/3698393365 shutdown_connections 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 --2- 172.21.15.143:0/3698393365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f007cc40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/3698393365 >> 172.21.15.143:0/3698393365 conn(0x7f02f007b8e0 msgr2=0x7f02f0106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/3698393365 shutdown_connections 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/3698393365 wait complete. 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 librados: starting msgr at 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 librados: starting objecter 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 Processor -- start 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- start start 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 librados: setting wanted keys 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 librados: calling monclient init 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f007ecb0 con 0x7f02f007c820 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:40.209 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45870/0 (socket says 172.21.15.143:45870) 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 -- 172.21.15.143:0/4027736304 learned_addr learned my addr 172.21.15.143:0/4027736304 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 -- 172.21.15.143:0/4027736304 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f01a0d90 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02eeffd700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006af0 tx=0x7f02e0011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 10 client.?.objecter ms_handle_connect 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02e00036a0 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f02e0024430 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f01a1020 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02e001a5c0 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f01a3d10 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02f536c700 1 librados: init done 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f02e001a8c0 con 0x7f02f007c820 2024-03-11T21:13:40.210 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.252+0000 7f02e7fff700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 0x7f02dc03cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f02e004d1d0 con 0x7f02f007c820 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 10 client.14144.objecter ms_dispatch 0x7f02f0001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f0063410 con 0x7f02f007c820 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 20 client.14144.objecter dump_active .. 0 homeless 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02ee7fc700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 0x7f02dc03cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02ee7fc700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 0x7f02dc03cd90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f02d8006fd0 tx=0x7f02d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:40.211 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.256+0000 7f02e7fff700 10 client.14144.objecter ms_handle_connect 0x7f02dc03a8d0 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.260+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f02e0019c70 con 0x7f02f007c820 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.636+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f02f0062220 con 0x7f02dc03a8d0 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.644+0000 7f02e7fff700 1 -- 172.21.15.143:0/4027736304 <== mgr.14120 v2:172.21.15.143:6800/3787195463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f02f0062220 con 0x7f02dc03a8d0 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 10 librados: watch_flush enter 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 10 librados: watch_flush exit 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 20 client.14144.objecter shutdown clearing up homeless session... 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 10 client.14144.objecter successfully canceled tick 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 msgr2=0x7f02dc03cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 0x7f02dc03cd90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f02d8006fd0 tx=0x7f02d8006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 msgr2=0x7f02f01a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006af0 tx=0x7f02e0011ec0 comp rx=0 tx=0).stop 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 shutdown_connections 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f02dc03a8d0 0x7f02dc03cd90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 --2- 172.21.15.143:0/4027736304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f007c820 0x7f02f01a0850 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 >> 172.21.15.143:0/4027736304 conn(0x7f02f007b8e0 msgr2=0x7f02f0197550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:40.213 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 shutdown_connections 2024-03-11T21:13:40.213 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 -- 172.21.15.143:0/4027736304 wait complete. 2024-03-11T21:13:40.213 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:39.648+0000 7f02f536c700 1 librados: shutdown 2024-03-11T21:13:40.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:40 smithi143 bash[21512]: cluster 2024-03-11T21:13:39.188572+0000 mgr.y (mgr.14120) 18 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:13:40.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:40 smithi143 bash[21512]: audit 2024-03-11T21:13:39.644619+0000 mgr.y (mgr.14120) 19 : audit 0 from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:40.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:40 smithi143 bash[21512]: cephadm 2024-03-11T21:13:39.646098+0000 mgr.y (mgr.14120) 20 : cephadm 1 Saving service mgr spec with placement count:2 2024-03-11T21:13:40.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:40 smithi143 bash[21512]: audit 2024-03-11T21:13:39.649548+0000 mon.a (mon.0) 62 : audit 1 from='mgr.14120 172.21.15.143:0/1152347828' entity='mgr.y' 2024-03-11T21:13:42.090 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f48595700 1 Processor -- start 2024-03-11T21:13:42.090 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f48595700 1 -- start start 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f48595700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f48595700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40108f10 con 0x7f7f40108520 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f46331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f46331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45886/0 (socket says 172.21.15.143:45886) 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f46331700 1 -- 172.21.15.143:0/15272250 learned_addr learned my addr 172.21.15.143:0/15272250 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f46331700 1 -- 172.21.15.143:0/15272250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40109720 con 0x7f7f40108520 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f46331700 1 --2- 172.21.15.143:0/15272250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f300098a0 tx=0x7f7f30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50704c9eb7d8745 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f4532f700 1 -- 172.21.15.143:0/15272250 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f30004160 con 0x7f7f40108520 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f4532f700 1 -- 172.21.15.143:0/15272250 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7f30004300 con 0x7f7f40108520 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.132+0000 7f7f4532f700 1 -- 172.21.15.143:0/15272250 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f30011930 con 0x7f7f40108520 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/15272250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 msgr2=0x7f7f40108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 --2- 172.21.15.143:0/15272250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f300098a0 tx=0x7f7f30009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.091 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/15272250 shutdown_connections 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 --2- 172.21.15.143:0/15272250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f40108940 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/15272250 >> 172.21.15.143:0/15272250 conn(0x7f7f4007b8f0 msgr2=0x7f7f40106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/15272250 shutdown_connections 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/15272250 wait complete. 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 librados: starting msgr at 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 librados: starting objecter 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 Processor -- start 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- start start 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 librados: setting wanted keys 2024-03-11T21:13:42.097 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 librados: calling monclient init 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4010c950 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f46331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f46331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45892/0 (socket says 172.21.15.143:45892) 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f46331700 1 -- 172.21.15.143:0/2392072655 learned_addr learned my addr 172.21.15.143:0/2392072655 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f46331700 1 -- 172.21.15.143:0/2392072655 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4019c9c0 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f46331700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f30006af0 tx=0x7f7f30011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 10 client.?.objecter ms_handle_connect 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f300039e0 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7f30003b80 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4019cc50 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f30019620 con 0x7f7f40108520 2024-03-11T21:13:42.098 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4019f940 con 0x7f7f40108520 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f48595700 1 librados: init done 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7f300198d0 con 0x7f7f40108520 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 0x7f7f2803cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7f3001a8b0 con 0x7f7f40108520 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 10 client.14146.objecter ms_dispatch 0x7f7f40001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f377fe700 20 client.14146.objecter dump_active .. 0 homeless 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.136+0000 7f7f45b30700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 0x7f7f2803cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.140+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4010cad0 con 0x7f7f40108520 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.140+0000 7f7f377fe700 10 client.14146.objecter ms_handle_connect 0x7f7f2803a8d0 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.140+0000 7f7f45b30700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 0x7f7f2803cd90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00ad30 tx=0x7f7f3c0093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.144+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7f3000b9d0 con 0x7f7f40108520 2024-03-11T21:13:42.099 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.504+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f7f4019fb10 con 0x7f7f40108520 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.504+0000 7f7f377fe700 1 -- 172.21.15.143:0/2392072655 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7f3000bc30 con 0x7f7f40108520 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 10 librados: watch_flush enter 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 10 librados: watch_flush exit 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 20 client.14146.objecter shutdown clearing up homeless session... 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 10 client.14146.objecter successfully canceled tick 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 msgr2=0x7f7f2803cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 0x7f7f2803cd90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00ad30 tx=0x7f7f3c0093f0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 msgr2=0x7f7f4019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f30006af0 tx=0x7f7f30011af0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 shutdown_connections 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f7f2803a8d0 0x7f7f2803cd90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 --2- 172.21.15.143:0/2392072655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f40108520 0x7f7f4019c480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:42.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 >> 172.21.15.143:0/2392072655 conn(0x7f7f4007b8f0 msgr2=0x7f7f40105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:42.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 shutdown_connections 2024-03-11T21:13:42.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 -- 172.21.15.143:0/2392072655 wait complete. 2024-03-11T21:13:42.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:41.508+0000 7f7f48595700 1 librados: shutdown 2024-03-11T21:13:42.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:42 smithi143 bash[21512]: cluster 2024-03-11T21:13:41.188847+0000 mgr.y (mgr.14120) 21 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:13:42.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:42 smithi143 bash[21512]: audit 2024-03-11T21:13:41.510881+0000 mon.a (mon.0) 63 : audit 1 from='client.? 172.21.15.143:0/2392072655' entity='client.admin' 2024-03-11T21:13:43.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 Processor -- start 2024-03-11T21:13:43.836 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- start start 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061007a870 con 0x7f061007be40 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45908/0 (socket says 172.21.15.143:45908) 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060ffff700 1 -- 172.21.15.143:0/3468990313 learned_addr learned my addr 172.21.15.143:0/3468990313 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060ffff700 1 -- 172.21.15.143:0/3468990313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061007a9f0 con 0x7f061007be40 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060ffff700 1 --2- 172.21.15.143:0/3468990313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f06000098a0 tx=0x7f0600009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411b0a40a8114186 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060effd700 1 -- 172.21.15.143:0/3468990313 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0600004160 con 0x7f061007be40 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060effd700 1 -- 172.21.15.143:0/3468990313 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0600004300 con 0x7f061007be40 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f060effd700 1 -- 172.21.15.143:0/3468990313 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0600011930 con 0x7f061007be40 2024-03-11T21:13:43.837 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- 172.21.15.143:0/3468990313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 msgr2=0x7f061007a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 --2- 172.21.15.143:0/3468990313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f06000098a0 tx=0x7f0600009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- 172.21.15.143:0/3468990313 shutdown_connections 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 --2- 172.21.15.143:0/3468990313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061007a2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- 172.21.15.143:0/3468990313 >> 172.21.15.143:0/3468990313 conn(0x7f06101017e0 msgr2=0x7f0610103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- 172.21.15.143:0/3468990313 shutdown_connections 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- 172.21.15.143:0/3468990313 wait complete. 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 librados: starting msgr at 2024-03-11T21:13:43.844 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 librados: starting objecter 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 Processor -- start 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 -- start start 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 librados: setting wanted keys 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.896+0000 7f06169ee700 1 librados: calling monclient init 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061010c820 con 0x7f061007be40 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45920/0 (socket says 172.21.15.143:45920) 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060ffff700 1 -- 172.21.15.143:0/2268019773 learned_addr learned my addr 172.21.15.143:0/2268019773 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060ffff700 1 -- 172.21.15.143:0/2268019773 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061019c860 con 0x7f061007be40 2024-03-11T21:13:43.845 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060ffff700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0600006af0 tx=0x7f0600011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 10 client.?.objecter ms_handle_connect 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06000036a0 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0600024440 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061019caf0 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f060001a430 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061019f7e0 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 librados: init done 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f06000248d0 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610063410 con 0x7f061007be40 2024-03-11T21:13:43.846 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 0x7f05fc03cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f060004cee0 con 0x7f061007be40 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 10 client.14148.objecter ms_dispatch 0x7f0610001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 20 client.14148.objecter dump_active .. 0 homeless 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060f7fe700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 0x7f05fc03cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060f7fe700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 0x7f05fc03cd90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05f800ad30 tx=0x7f05f80093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.900+0000 7f060d7fa700 10 client.14148.objecter ms_handle_connect 0x7f05fc03a8d0 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:42.904+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f06000172b0 con 0x7f061007be40 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.260+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f061019fc30 con 0x7f061007be40 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.264+0000 7f060d7fa700 1 -- 172.21.15.143:0/2268019773 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0600004300 con 0x7f061007be40 2024-03-11T21:13:43.847 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 10 librados: watch_flush enter 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 10 librados: watch_flush exit 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 20 client.14148.objecter shutdown clearing up homeless session... 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 10 client.14148.objecter successfully canceled tick 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 msgr2=0x7f05fc03cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 0x7f05fc03cd90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05f800ad30 tx=0x7f05f80093f0 comp rx=0 tx=0).stop 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 msgr2=0x7f061019c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0600006af0 tx=0x7f0600011f30 comp rx=0 tx=0).stop 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 shutdown_connections 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f05fc03a8d0 0x7f05fc03cd90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 --2- 172.21.15.143:0/2268019773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f061007be40 0x7f061019c320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 >> 172.21.15.143:0/2268019773 conn(0x7f06101017e0 msgr2=0x7f0610102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:43.848 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 shutdown_connections 2024-03-11T21:13:43.849 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 -- 172.21.15.143:0/2268019773 wait complete. 2024-03-11T21:13:43.849 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:43.268+0000 7f06169ee700 1 librados: shutdown 2024-03-11T21:13:43.849 INFO:teuthology.orchestra.run.smithi143.stdout:Enabling the dashboard module... 2024-03-11T21:13:44.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:44 smithi143 bash[21512]: cluster 2024-03-11T21:13:43.189196+0000 mgr.y (mgr.14120) 22 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:13:44.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:44 smithi143 bash[21512]: audit 2024-03-11T21:13:43.270040+0000 mon.a (mon.0) 64 : audit 1 from='client.? 172.21.15.143:0/2268019773' entity='client.admin' 2024-03-11T21:13:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:45 smithi143 bash[21512]: audit 2024-03-11T21:13:45.220312+0000 mon.a (mon.0) 65 : audit 1 from='client.? 172.21.15.143:0/1819885215' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:13:45.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:45 smithi143 bash[21768]: ignoring --setuser ceph since I am not root 2024-03-11T21:13:45.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:45 smithi143 bash[21768]: ignoring --setgroup ceph since I am not root 2024-03-11T21:13:45.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:45 smithi143 bash[21768]: debug 2024-03-11T21:13:45.468+0000 7f9a40622200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:13:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe6138cb700 1 Processor -- start 2024-03-11T21:13:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe6138cb700 1 -- start start 2024-03-11T21:13:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe6138cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe6138cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c07a870 con 0x7fe60c07be40 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe611667700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe611667700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40566/0 (socket says 172.21.15.143:40566) 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe611667700 1 -- 172.21.15.143:0/3439362875 learned_addr learned my addr 172.21.15.143:0/3439362875 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.732+0000 7fe611667700 1 -- 172.21.15.143:0/3439362875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c07a9f0 con 0x7fe60c07be40 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 --2- 172.21.15.143:0/3439362875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009e80 tx=0x7fe5fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4815dd5c0a0a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60bfff700 1 -- 172.21.15.143:0/3439362875 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc017070 con 0x7fe60c07be40 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60bfff700 1 -- 172.21.15.143:0/3439362875 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe5fc00b8c0 con 0x7fe60c07be40 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60bfff700 1 -- 172.21.15.143:0/3439362875 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc004400 con 0x7fe60c07be40 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/3439362875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 msgr2=0x7fe60c07a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 --2- 172.21.15.143:0/3439362875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009e80 tx=0x7fe5fc00b330 comp rx=0 tx=0).stop 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/3439362875 shutdown_connections 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 --2- 172.21.15.143:0/3439362875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c07a2a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/3439362875 >> 172.21.15.143:0/3439362875 conn(0x7fe60c1017e0 msgr2=0x7fe60c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:45.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/3439362875 shutdown_connections 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/3439362875 wait complete. 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 librados: starting msgr at 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 librados: starting objecter 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 Processor -- start 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- start start 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 librados: setting wanted keys 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 librados: calling monclient init 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c10c820 con 0x7fe60c07be40 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40578/0 (socket says 172.21.15.143:40578) 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 -- 172.21.15.143:0/1819885215 learned_addr learned my addr 172.21.15.143:0/1819885215 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:45.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 -- 172.21.15.143:0/1819885215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c19c810 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe611667700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc004000 tx=0x7fe5fc0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 10 client.?.objecter ms_handle_connect 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc020020 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe5fc003d00 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc017040 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c19caa0 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c19f790 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe6138cb700 1 librados: init done 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.736+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe5fc023020 con 0x7fe60c07be40 2024-03-11T21:13:45.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c063410 con 0x7fe60c07be40 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 0x7fe5f403cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe5fc00f050 con 0x7fe60c07be40 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 10 client.14150.objecter ms_dispatch 0x7fe60c001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 20 client.14150.objecter dump_active .. 0 homeless 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe610e66700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 0x7fe5f403cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe610e66700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 0x7fe5f403cde0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.740+0000 7fe60a7fc700 10 client.14150.objecter ms_handle_connect 0x7fe5f403a920 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:44.744+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe5fc02a020 con 0x7fe60c07be40 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.212+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe60c19fb70 con 0x7fe60c07be40 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.272+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe5fc02c300 con 0x7fe60c07be40 2024-03-11T21:13:45.952 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 10 librados: watch_flush enter 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 10 librados: watch_flush exit 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 20 client.14150.objecter shutdown clearing up homeless session... 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 10 client.14150.objecter successfully canceled tick 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 msgr2=0x7fe5f403cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 0x7fe5f403cde0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.276+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 msgr2=0x7fe60c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc004000 tx=0x7fe5fc0049c0 comp rx=0 tx=0).stop 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe60a7fc700 1 -- 172.21.15.143:0/1819885215 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe5fc012430 con 0x7fe60c07be40 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 shutdown_connections 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7fe5f403a920 0x7fe5f403cde0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 --2- 172.21.15.143:0/1819885215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe60c07be40 0x7fe60c19c2d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 >> 172.21.15.143:0/1819885215 conn(0x7fe60c1017e0 msgr2=0x7fe60c1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 shutdown_connections 2024-03-11T21:13:45.953 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 -- 172.21.15.143:0/1819885215 wait complete. 2024-03-11T21:13:45.954 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:45.280+0000 7fe6138cb700 1 librados: shutdown 2024-03-11T21:13:46.597 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:46 smithi143 bash[21512]: audit 2024-03-11T21:13:45.279566+0000 mon.a (mon.0) 66 : audit 1 from='client.? 172.21.15.143:0/1819885215' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:13:46.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:46 smithi143 bash[21512]: cluster 2024-03-11T21:13:45.285601+0000 mon.a (mon.0) 67 : cluster 0 mgrmap e9: y(active, since 26s) 2024-03-11T21:13:47.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:47 smithi143 bash[21512]: audit 2024-03-11T21:13:47.274931+0000 mon.a (mon.0) 68 : audit 0 from='client.? 172.21.15.143:0/912585349' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:13:47.404 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:47 smithi143 bash[21768]: debug 2024-03-11T21:13:47.060+0000 7f9a40622200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:13:47.404 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:47 smithi143 bash[21768]: debug 2024-03-11T21:13:47.155+0000 7f9a40622200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:13:47.698 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:47 smithi143 bash[21768]: debug 2024-03-11T21:13:47.411+0000 7f9a40622200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:13:47.895 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:47.895 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 Processor -- start 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- start start 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c106c50 con 0x7f5b2c106260 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b30a24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b30a24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40616/0 (socket says 172.21.15.143:40616) 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b30a24700 1 -- 172.21.15.143:0/2110357867 learned_addr learned my addr 172.21.15.143:0/2110357867 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b30a24700 1 -- 172.21.15.143:0/2110357867 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c107460 con 0x7f5b2c106260 2024-03-11T21:13:47.896 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b30a24700 1 --2- 172.21.15.143:0/2110357867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b180098a0 tx=0x7f5b18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4aa43eaba5247 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b2b7fe700 1 -- 172.21.15.143:0/2110357867 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b18004160 con 0x7f5b2c106260 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b2b7fe700 1 -- 172.21.15.143:0/2110357867 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5b18004300 con 0x7f5b2c106260 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b2b7fe700 1 -- 172.21.15.143:0/2110357867 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b18011930 con 0x7f5b2c106260 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- 172.21.15.143:0/2110357867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 msgr2=0x7f5b2c106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 --2- 172.21.15.143:0/2110357867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b180098a0 tx=0x7f5b18009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- 172.21.15.143:0/2110357867 shutdown_connections 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 --2- 172.21.15.143:0/2110357867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c106680 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- 172.21.15.143:0/2110357867 >> 172.21.15.143:0/2110357867 conn(0x7f5b2c101820 msgr2=0x7f5b2c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- 172.21.15.143:0/2110357867 shutdown_connections 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- 172.21.15.143:0/2110357867 wait complete. 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 librados: starting msgr at 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 librados: starting objecter 2024-03-11T21:13:47.897 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 Processor -- start 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 -- start start 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 librados: setting wanted keys 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.784+0000 7f5b32c88700 1 librados: calling monclient init 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c10a690 con 0x7f5b2c106260 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b30a24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b30a24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40628/0 (socket says 172.21.15.143:40628) 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b30a24700 1 -- 172.21.15.143:0/912585349 learned_addr learned my addr 172.21.15.143:0/912585349 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b30a24700 1 -- 172.21.15.143:0/912585349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c07c070 con 0x7f5b2c106260 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b30a24700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b18006af0 tx=0x7f5b18011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 10 client.?.objecter ms_handle_connect 0x7f5b2c106260 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:47.898 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b180039f0 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c07a260 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5b18003b90 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b1801a6f0 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c07a7a0 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 librados: init done 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5b1801a9a0 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c10a810 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 0x7f5b1c03cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5b1804d3d0 con 0x7f5b2c106260 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 10 client.14154.objecter ms_dispatch 0x7f5b2c001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b2bfff700 1 -- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 msgr2=0x7f5b1c03cde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b2bfff700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 0x7f5b1c03cde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.788+0000 7f5b29ffb700 20 client.14154.objecter dump_active .. 0 homeless 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.792+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5b1800f2e0 con 0x7f5b2c106260 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.992+0000 7f5b2bfff700 1 -- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 msgr2=0x7f5b1c03cde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:46.992+0000 7f5b2bfff700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 0x7f5b1c03cde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.267+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5b2c063410 con 0x7f5b2c106260 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.267+0000 7f5b29ffb700 1 -- 172.21.15.143:0/912585349 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f5b18004300 con 0x7f5b2c106260 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 10 librados: watch_flush enter 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 10 librados: watch_flush exit 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 20 client.14154.objecter shutdown clearing up homeless session... 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 10 client.14154.objecter successfully canceled tick 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 msgr2=0x7f5b1c03cde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:13:47.900 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 0x7f5b1c03cde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 msgr2=0x7f5b2c07bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b18006af0 tx=0x7f5b18011a20 comp rx=0 tx=0).stop 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 shutdown_connections 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f5b1c03a920 0x7f5b1c03cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 --2- 172.21.15.143:0/912585349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2c106260 0x7f5b2c07bb30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 >> 172.21.15.143:0/912585349 conn(0x7f5b2c101820 msgr2=0x7f5b2c102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 shutdown_connections 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 -- 172.21.15.143:0/912585349 wait complete. 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:47.271+0000 7f5b32c88700 1 librados: shutdown 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for the mgr to restart... 2024-03-11T21:13:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:Waiting for mgr epoch 9... 2024-03-11T21:13:48.557 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:48 smithi143 bash[21768]: debug 2024-03-11T21:13:48.211+0000 7f9a40622200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:13:48.558 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:48 smithi143 bash[21768]: debug 2024-03-11T21:13:48.323+0000 7f9a40622200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:13:48.558 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:48 smithi143 bash[21768]: debug 2024-03-11T21:13:48.431+0000 7f9a40622200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:13:48.899 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:48 smithi143 bash[21768]: debug 2024-03-11T21:13:48.799+0000 7f9a40622200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:13:49.198 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:48 smithi143 bash[21768]: debug 2024-03-11T21:13:48.911+0000 7f9a40622200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:13:49.199 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:49 smithi143 bash[21768]: debug 2024-03-11T21:13:49.015+0000 7f9a40622200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:13:49.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:49 smithi143 bash[21768]: debug 2024-03-11T21:13:49.367+0000 7f9a40622200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:13:49.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:49 smithi143 bash[21768]: debug 2024-03-11T21:13:49.607+0000 7f9a40622200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:13:50.199 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:49 smithi143 bash[21768]: debug 2024-03-11T21:13:49.867+0000 7f9a40622200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:13:50.199 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:49 smithi143 bash[21768]: debug 2024-03-11T21:13:49.979+0000 7f9a40622200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:13:50.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:50 smithi143 bash[21768]: debug 2024-03-11T21:13:50.255+0000 7f9a40622200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:13:50.699 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:50 smithi143 bash[21768]: debug 2024-03-11T21:13:50.495+0000 7f9a40622200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:51.198 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:50 smithi143 bash[21768]: debug 2024-03-11T21:13:50.903+0000 7f9a40622200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:13:51.199 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:51 smithi143 bash[21768]: debug 2024-03-11T21:13:51.023+0000 7f9a40622200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:13:51.698 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:51 smithi143 bash[21768]: debug 2024-03-11T21:13:51.271+0000 7f9a40622200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:13:52.677 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:52 smithi143 bash[21768]: debug 2024-03-11T21:13:52.279+0000 7f9a40622200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:13:52.677 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:52 smithi143 bash[21768]: debug 2024-03-11T21:13:52.395+0000 7f9a40622200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:13:52.949 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:52 smithi143 bash[21768]: debug 2024-03-11T21:13:52.687+0000 7f9a40622200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:13:52.949 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:52 smithi143 bash[21768]: debug 2024-03-11T21:13:52.803+0000 7f9a40622200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:13:53.495 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21768]: debug 2024-03-11T21:13:53.255+0000 7f9a40622200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21768]: debug 2024-03-11T21:13:53.379+0000 7f9a40622200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: cluster 2024-03-11T21:13:53.401541+0000 mon.a (mon.0) 69 : cluster 1 Active manager daemon y restarted 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: cluster 2024-03-11T21:13:53.402033+0000 mon.a (mon.0) 70 : cluster 1 Activating manager daemon y 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: cluster 2024-03-11T21:13:53.408254+0000 mon.a (mon.0) 71 : cluster 0 osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: cluster 2024-03-11T21:13:53.408426+0000 mon.a (mon.0) 72 : cluster 0 mgrmap e10: y(active, starting, since 0.00658684s) 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: audit 2024-03-11T21:13:53.409478+0000 mon.a (mon.0) 73 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: audit 2024-03-11T21:13:53.410479+0000 mon.a (mon.0) 74 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: audit 2024-03-11T21:13:53.411409+0000 mon.a (mon.0) 75 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: audit 2024-03-11T21:13:53.411615+0000 mon.a (mon.0) 76 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:13:53.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: audit 2024-03-11T21:13:53.411816+0000 mon.a (mon.0) 77 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:13:53.497 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:53 smithi143 bash[21512]: cluster 2024-03-11T21:13:53.438019+0000 mon.a (mon.0) 78 : cluster 1 Manager daemon y is now available 2024-03-11T21:13:54.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: audit 2024-03-11T21:13:53.462605+0000 mon.a (mon.0) 79 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:54.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: audit 2024-03-11T21:13:53.463859+0000 mon.a (mon.0) 80 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:54.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: audit 2024-03-11T21:13:53.482759+0000 mon.a (mon.0) 81 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: audit 2024-03-11T21:13:53.486182+0000 mon.a (mon.0) 82 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: cephadm 2024-03-11T21:13:53.955860+0000 mgr.y (mgr.14152) 1 : cephadm 1 [11/Mar/2024:21:13:53] ENGINE Bus STARTING 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: cephadm 2024-03-11T21:13:54.070481+0000 mgr.y (mgr.14152) 2 : cephadm 1 [11/Mar/2024:21:13:54] ENGINE Serving on https://172.21.15.143:7150 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: cephadm 2024-03-11T21:13:54.171454+0000 mgr.y (mgr.14152) 3 : cephadm 1 [11/Mar/2024:21:13:54] ENGINE Serving on http://172.21.15.143:8765 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: cephadm 2024-03-11T21:13:54.171607+0000 mgr.y (mgr.14152) 4 : cephadm 1 [11/Mar/2024:21:13:54] ENGINE Bus STARTED 2024-03-11T21:13:54.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:54 smithi143 bash[21512]: cluster 2024-03-11T21:13:54.414320+0000 mon.a (mon.0) 83 : cluster 0 mgrmap e11: y(active, since 1.01248s) 2024-03-11T21:13:55.143 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:13:55.143 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:13:55.143 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.743+0000 7f6c398de700 1 Processor -- start 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- start start 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3407a890 con 0x7f6c3407be60 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40636/0 (socket says 172.21.15.143:40636) 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 -- 172.21.15.143:0/1809716484 learned_addr learned my addr 172.21.15.143:0/1809716484 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 -- 172.21.15.143:0/1809716484 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3407aa10 con 0x7f6c3407be60 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- 172.21.15.143:0/1809716484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6c240098a0 tx=0x7f6c24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ab9b9352398939 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c31ffb700 1 -- 172.21.15.143:0/1809716484 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c24004160 con 0x7f6c3407be60 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c31ffb700 1 -- 172.21.15.143:0/1809716484 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6c24004300 con 0x7f6c3407be60 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c31ffb700 1 -- 172.21.15.143:0/1809716484 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c24011930 con 0x7f6c3407be60 2024-03-11T21:13:55.144 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/1809716484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 msgr2=0x7f6c3407a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 --2- 172.21.15.143:0/1809716484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6c240098a0 tx=0x7f6c24009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/1809716484 shutdown_connections 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 --2- 172.21.15.143:0/1809716484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3407a2c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/1809716484 >> 172.21.15.143:0/1809716484 conn(0x7f6c34101820 msgr2=0x7f6c34103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/1809716484 shutdown_connections 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/1809716484 wait complete. 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 librados: starting msgr at 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 librados: starting objecter 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 Processor -- start 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- start start 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 librados: setting wanted keys 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 librados: calling monclient init 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:55.145 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3410c800 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40644/0 (socket says 172.21.15.143:40644) 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 -- 172.21.15.143:0/946719802 learned_addr learned my addr 172.21.15.143:0/946719802 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3419c890 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c32ffd700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009fb0 tx=0x7f6c24011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c388dc700 10 client.?.objecter ms_handle_connect 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c388dc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c240119a0 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6c24003b20 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3419cb20 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c2401ad50 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3419f810 con 0x7f6c3407be60 2024-03-11T21:13:55.146 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.747+0000 7f6c398de700 1 librados: init done 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f6c2401a6a0 con 0x7f6c3407be60 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c1c03d540 con 0x7f6c1c03a970 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6c2404d020 con 0x7f6c3407be60 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 10 client.14156.objecter ms_dispatch 0x7f6c34001550 osd_map(2..2 src has 1..2) v4 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 3 client.14156.objecter handle_osd_map got epochs [2,2] > 0 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 3 client.14156.objecter handle_osd_map decoding full epoch 2 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c327fc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.751+0000 7f6c388dc700 20 client.14156.objecter dump_active .. 0 homeless 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.955+0000 7f6c327fc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:48.955+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:49.359+0000 7f6c327fc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:55.147 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:49.359+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:50.163+0000 7f6c327fc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:50.163+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:51.767+0000 7f6c327fc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3787195463 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:51.767+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:53.403+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f6c2402d440 con 0x7f6c3407be60 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:53.403+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 msgr2=0x7f6c1c03ce30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:53.403+0000 7f6c388dc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:53.751+0000 7f6c390dd700 10 client.14156.objecter tick 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.407+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f6c2404eef0 con 0x7f6c3407be60 2024-03-11T21:13:55.148 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.407+0000 7f6c388dc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 0x7f6c1c040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.407+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c1c03d540 con 0x7f6c1c03e420 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.407+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 0x7f6c1c040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.411+0000 7f6c327fc700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 0x7f6c1c040810 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6c28003a10 tx=0x7f6c28009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.411+0000 7f6c388dc700 10 client.14156.objecter ms_handle_connect 0x7f6c1c03e420 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.411+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f6c1c03d540 con 0x7f6c1c03e420 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6c34063410 con 0x7f6c1c03e420 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c388dc700 1 -- 172.21.15.143:0/946719802 <== mgr.14152 v2:172.21.15.143:6800/3638739149 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6c34063410 con 0x7f6c1c03e420 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 10 librados: watch_flush enter 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 10 librados: watch_flush exit 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 20 client.14156.objecter shutdown clearing up homeless session... 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 10 client.14156.objecter successfully canceled tick 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 msgr2=0x7f6c1c040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 0x7f6c1c040810 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6c28003a10 tx=0x7f6c28009270 comp rx=0 tx=0).stop 2024-03-11T21:13:55.149 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 msgr2=0x7f6c3419c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009fb0 tx=0x7f6c24011ea0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 shutdown_connections 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6c1c03e420 0x7f6c1c040810 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3407be60 0x7f6c3419c350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 --2- 172.21.15.143:0/946719802 >> [v2:172.21.15.143:6800/3787195463,v1:172.21.15.143:6801/3787195463] conn(0x7f6c1c03a970 0x7f6c1c03ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 >> 172.21.15.143:0/946719802 conn(0x7f6c34101820 msgr2=0x7f6c34102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 shutdown_connections 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 -- 172.21.15.143:0/946719802 wait complete. 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:54.427+0000 7f6c398de700 1 librados: shutdown 2024-03-11T21:13:55.150 INFO:teuthology.orchestra.run.smithi143.stdout:mgr epoch 9 is available 2024-03-11T21:13:55.151 INFO:teuthology.orchestra.run.smithi143.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:13:55.749 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:55 smithi143 bash[21512]: audit 2024-03-11T21:13:54.413225+0000 mgr.y (mgr.14152) 5 : audit 0 from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:13:55.749 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:55 smithi143 bash[21512]: audit 2024-03-11T21:13:54.428863+0000 mgr.y (mgr.14152) 6 : audit 0 from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:13:56.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:56 smithi143 bash[21512]: cluster 2024-03-11T21:13:55.462107+0000 mon.a (mon.0) 84 : cluster 0 mgrmap e12: y(active, since 2s) 2024-03-11T21:13:56.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:56 smithi143 bash[21512]: audit 2024-03-11T21:13:56.312235+0000 mgr.y (mgr.14152) 7 : audit 0 from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:56.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:56 smithi143 bash[21512]: audit 2024-03-11T21:13:56.378056+0000 mon.a (mon.0) 85 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:13:56.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:56 smithi143 bash[21512]: audit 2024-03-11T21:13:56.383639+0000 mon.a (mon.0) 86 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:13:56.946 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:13:56.946 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.935+0000 7f8bfb345700 1 Processor -- start 2024-03-11T21:13:56.946 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.935+0000 7f8bfb345700 1 -- start start 2024-03-11T21:13:56.946 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.935+0000 7f8bfb345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.935+0000 7f8bfb345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4108f10 con 0x7f8bf4108520 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.935+0000 7f8bf90e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58322/0 (socket says 172.21.15.143:58322) 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 -- 172.21.15.143:0/3733982490 learned_addr learned my addr 172.21.15.143:0/3733982490 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 -- 172.21.15.143:0/3733982490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4109720 con 0x7f8bf4108520 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 --2- 172.21.15.143:0/3733982490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098a0 tx=0x7f8be4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f128869d763b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bebfff700 1 -- 172.21.15.143:0/3733982490 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4004160 con 0x7f8bf4108520 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bebfff700 1 -- 172.21.15.143:0/3733982490 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8be4004300 con 0x7f8bf4108520 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bebfff700 1 -- 172.21.15.143:0/3733982490 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4011930 con 0x7f8bf4108520 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/3733982490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 msgr2=0x7f8bf4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 --2- 172.21.15.143:0/3733982490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098a0 tx=0x7f8be4009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/3733982490 shutdown_connections 2024-03-11T21:13:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 --2- 172.21.15.143:0/3733982490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf4108940 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/3733982490 >> 172.21.15.143:0/3733982490 conn(0x7f8bf407b8f0 msgr2=0x7f8bf4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/3733982490 shutdown_connections 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/3733982490 wait complete. 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 librados: starting msgr at 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 librados: starting objecter 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 Processor -- start 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- start start 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 librados: setting wanted keys 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 librados: calling monclient init 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf410c950 con 0x7f8bf4108520 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58326/0 (socket says 172.21.15.143:58326) 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 -- 172.21.15.143:0/2491610329 learned_addr learned my addr 172.21.15.143:0/2491610329 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 -- 172.21.15.143:0/2491610329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf419c9c0 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bf90e1700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006af0 tx=0x7f8be4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bea7fc700 10 client.?.objecter ms_handle_connect 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bea7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be40039e0 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8be4003b80 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4019620 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf419cc50 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf419f940 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.939+0000 7f8bfb345700 1 librados: init done 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f8be401a430 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf410cad0 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 0x7f8be003cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8be404d6a0 con 0x7f8bf4108520 2024-03-11T21:13:56.949 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 10 client.14164.objecter ms_dispatch 0x7f8bf4001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 20 client.14164.objecter dump_active .. 0 homeless 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bf88e0700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 0x7f8be003cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bf88e0700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 0x7f8be003cd90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006fd0 tx=0x7f8bf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.943+0000 7f8bea7fc700 10 client.14164.objecter ms_handle_connect 0x7f8be003a8d0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:55.947+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8be40042a0 con 0x7f8bf4108520 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.307+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bf41062e0 con 0x7f8be003a8d0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.383+0000 7f8bea7fc700 1 -- 172.21.15.143:0/2491610329 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8bf41062e0 con 0x7f8be003a8d0 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 10 librados: watch_flush enter 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 10 librados: watch_flush exit 2024-03-11T21:13:56.950 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 20 client.14164.objecter shutdown clearing up homeless session... 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 10 client.14164.objecter successfully canceled tick 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 msgr2=0x7f8be003cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 0x7f8be003cd90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006fd0 tx=0x7f8bf0006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 msgr2=0x7f8bf419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006af0 tx=0x7f8be4011af0 comp rx=0 tx=0).stop 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 shutdown_connections 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8be003a8d0 0x7f8be003cd90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 --2- 172.21.15.143:0/2491610329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf4108520 0x7f8bf419c480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 >> 172.21.15.143:0/2491610329 conn(0x7f8bf407b8f0 msgr2=0x7f8bf4105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 shutdown_connections 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 -- 172.21.15.143:0/2491610329 wait complete. 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:56.387+0000 7f8bfb345700 1 librados: shutdown 2024-03-11T21:13:56.951 INFO:teuthology.orchestra.run.smithi143.stdout:Creating initial admin user... 2024-03-11T21:13:58.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:58 smithi143 bash[21512]: audit 2024-03-11T21:13:58.175877+0000 mgr.y (mgr.14152) 8 : audit 0 from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:13:59.100 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$vH2wOp6fklD81HCtOimNqO23TWFi8iIsItuxthOZaaPef0/pYEosu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710191638, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.763+0000 7f7bb7a34700 1 Processor -- start 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.763+0000 7f7bb7a34700 1 -- start start 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0108f10 con 0x7f7bb0108520 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58330/0 (socket says 172.21.15.143:58330) 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 -- 172.21.15.143:0/126347313 learned_addr learned my addr 172.21.15.143:0/126347313 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 -- 172.21.15.143:0/126347313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0109720 con 0x7f7bb0108520 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- 172.21.15.143:0/126347313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00098a0 tx=0x7f7ba0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4439d59dd108a49b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba7fff700 1 -- 172.21.15.143:0/126347313 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba0004160 con 0x7f7bb0108520 2024-03-11T21:13:59.101 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba7fff700 1 -- 172.21.15.143:0/126347313 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7ba0004300 con 0x7f7bb0108520 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba7fff700 1 -- 172.21.15.143:0/126347313 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba0011930 con 0x7f7bb0108520 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/126347313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 msgr2=0x7f7bb0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/126347313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00098a0 tx=0x7f7ba0009bb0 comp rx=0 tx=0).stop 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/126347313 shutdown_connections 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/126347313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb0108940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/126347313 >> 172.21.15.143:0/126347313 conn(0x7f7bb007b8f0 msgr2=0x7f7bb0106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/126347313 shutdown_connections 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/126347313 wait complete. 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 librados: starting msgr at 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 librados: starting objecter 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 Processor -- start 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- start start 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 librados: setting wanted keys 2024-03-11T21:13:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 librados: calling monclient init 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb010c950 con 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58334/0 (socket says 172.21.15.143:58334) 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 -- 172.21.15.143:0/3514638457 learned_addr learned my addr 172.21.15.143:0/3514638457 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 -- 172.21.15.143:0/3514638457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb019c9c0 con 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb57d0700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0003fa0 tx=0x7f7ba0011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba67fc700 10 client.?.objecter ms_handle_connect 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba67fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba00036a0 con 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7ba001a430 con 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb019cc50 con 0x7f7bb0108520 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:13:59.103 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba00193c0 con 0x7f7bb0108520 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb019f940 con 0x7f7bb0108520 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.767+0000 7f7bb7a34700 1 librados: init done 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7ba001a930 con 0x7f7bb0108520 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb010cad0 con 0x7f7bb0108520 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 0x7f7b9c03ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ba004d210 con 0x7f7bb0108520 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 10 client.14166.objecter ms_dispatch 0x7f7bb0001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 20 client.14166.objecter dump_active .. 0 homeless 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7bb4fcf700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 0x7f7b9c03ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7bb4fcf700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 0x7f7b9c03ce30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7bac006fd0 tx=0x7f7bac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.771+0000 7f7ba67fc700 10 client.14166.objecter ms_handle_connect 0x7f7b9c03a970 2024-03-11T21:13:59.104 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:57.775+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7ba002cc80 con 0x7f7bb0108520 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.171+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb0063410 con 0x7f7b9c03a970 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.435+0000 7f7ba67fc700 1 -- 172.21.15.143:0/3514638457 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7bb0063410 con 0x7f7b9c03a970 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 10 librados: watch_flush enter 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 10 librados: watch_flush exit 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 20 client.14166.objecter shutdown clearing up homeless session... 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 10 client.14166.objecter successfully canceled tick 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 msgr2=0x7f7b9c03ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 0x7f7b9c03ce30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7bac006fd0 tx=0x7f7bac006e40 comp rx=0 tx=0).stop 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 msgr2=0x7f7bb019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0003fa0 tx=0x7f7ba0011f30 comp rx=0 tx=0).stop 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 shutdown_connections 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7b9c03a970 0x7f7b9c03ce30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 --2- 172.21.15.143:0/3514638457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb0108520 0x7f7bb019c480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 >> 172.21.15.143:0/3514638457 conn(0x7f7bb007b8f0 msgr2=0x7f7bb0105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:13:59.105 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 shutdown_connections 2024-03-11T21:13:59.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 -- 172.21.15.143:0/3514638457 wait complete. 2024-03-11T21:13:59.106 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:58.439+0000 7f7bb7a34700 1 librados: shutdown 2024-03-11T21:13:59.106 INFO:teuthology.orchestra.run.smithi143.stdout:Fetching dashboard port number... 2024-03-11T21:13:59.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:13:59 smithi143 bash[21512]: audit 2024-03-11T21:13:58.437608+0000 mon.a (mon.0) 87 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:00.915 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.931+0000 7fb71fa27700 1 Processor -- start 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.931+0000 7fb71fa27700 1 -- start start 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.931+0000 7fb71fa27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.931+0000 7fb71fa27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71807d210 con 0x7fb71807c820 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58350/0 (socket says 172.21.15.143:58350) 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 -- 172.21.15.143:0/356936970 learned_addr learned my addr 172.21.15.143:0/356936970 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 -- 172.21.15.143:0/356936970 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71807da70 con 0x7fb71807c820 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- 172.21.15.143:0/356936970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb7000098a0 tx=0x7fb700009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f059e54b7a65994 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb717fff700 1 -- 172.21.15.143:0/356936970 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb700004160 con 0x7fb71807c820 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb717fff700 1 -- 172.21.15.143:0/356936970 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb700004300 con 0x7fb71807c820 2024-03-11T21:14:00.916 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb717fff700 1 -- 172.21.15.143:0/356936970 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb700011930 con 0x7fb71807c820 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/356936970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 msgr2=0x7fb71807cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 --2- 172.21.15.143:0/356936970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb7000098a0 tx=0x7fb700009bb0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/356936970 shutdown_connections 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 --2- 172.21.15.143:0/356936970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb71807cc40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/356936970 >> 172.21.15.143:0/356936970 conn(0x7fb71807b8e0 msgr2=0x7fb718106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/356936970 shutdown_connections 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/356936970 wait complete. 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 librados: starting msgr at 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 librados: starting objecter 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 Processor -- start 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- start start 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 librados: setting wanted keys 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 librados: calling monclient init 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:00.917 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71807ecb0 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58358/0 (socket says 172.21.15.143:58358) 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 -- 172.21.15.143:0/949493368 learned_addr learned my addr 172.21.15.143:0/949493368 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 -- 172.21.15.143:0/949493368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7181a0d90 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71d7c3700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb700009870 tx=0x7fb700011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb7167fc700 10 client.?.objecter ms_handle_connect 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb7167fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7000039a0 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb700003b40 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7181a1020 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb700019640 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7181a3d10 con 0x7fb71807c820 2024-03-11T21:14:00.918 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.935+0000 7fb71fa27700 1 librados: init done 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb7000198f0 con 0x7fb71807c820 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 0x7fb70403cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb70004d6e0 con 0x7fb71807c820 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 10 client.14168.objecter ms_dispatch 0x7fb718001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 20 client.14168.objecter dump_active .. 0 homeless 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb718063410 con 0x7fb71807c820 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb71cfc2700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 0x7fb70403cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb7167fc700 10 client.14168.objecter ms_handle_connect 0x7fb70403a920 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.939+0000 7fb71cfc2700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 0x7fb70403cde0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb70800ad30 tx=0x7fb7080093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:13:59.943+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb700018d00 con 0x7fb71807c820 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.323+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb71802d830 con 0x7fb71807c820 2024-03-11T21:14:00.919 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.323+0000 7fb7167fc700 1 -- 172.21.15.143:0/949493368 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb700018350 con 0x7fb71807c820 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 10 librados: watch_flush enter 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 10 librados: watch_flush exit 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 20 client.14168.objecter shutdown clearing up homeless session... 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 10 client.14168.objecter successfully canceled tick 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 msgr2=0x7fb70403cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 0x7fb70403cde0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb70800ad30 tx=0x7fb7080093f0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 msgr2=0x7fb7181a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb700009870 tx=0x7fb700011af0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 shutdown_connections 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fb70403a920 0x7fb70403cde0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 --2- 172.21.15.143:0/949493368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71807c820 0x7fb7181a0850 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb700009870 tx=0x7fb700011af0 comp rx=0 tx=0).stop 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 >> 172.21.15.143:0/949493368 conn(0x7fb71807b8e0 msgr2=0x7fb718197550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 shutdown_connections 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 -- 172.21.15.143:0/949493368 wait complete. 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:00.327+0000 7fb71fa27700 1 librados: shutdown 2024-03-11T21:14:00.920 INFO:teuthology.orchestra.run.smithi143.stdout:firewalld does not appear to be present 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout:Ceph Dashboard is now available at: 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout: URL: https://smithi143.front.sepia.ceph.com:8443/ 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout: User: admin 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout: Password: vqlibce8jf 2024-03-11T21:14:00.921 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:00.922 INFO:teuthology.orchestra.run.smithi143.stdout:Saving cluster configuration to /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config directory 2024-03-11T21:14:00.922 INFO:teuthology.orchestra.run.smithi143.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:14:01.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:01 smithi143 bash[21512]: audit 2024-03-11T21:14:00.221821+0000 mon.a (mon.0) 88 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:01.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:01 smithi143 bash[21512]: audit 2024-03-11T21:14:00.326614+0000 mon.a (mon.0) 89 : audit 0 from='client.? 172.21.15.143:0/949493368' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:14:01.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:01 smithi143 bash[21512]: cluster 2024-03-11T21:14:00.439252+0000 mon.a (mon.0) 90 : cluster 0 mgrmap e13: y(active, since 7s) 2024-03-11T21:14:02.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:02 smithi143 bash[21512]: audit 2024-03-11T21:14:01.362048+0000 mon.a (mon.0) 91 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:02.794 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 Processor -- start 2024-03-11T21:14:02.794 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- start start 2024-03-11T21:14:02.794 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d28108f00 con 0x7f8d28108510 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d27fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d27fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58364/0 (socket says 172.21.15.143:58364) 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d27fff700 1 -- 172.21.15.143:0/2681118626 learned_addr learned my addr 172.21.15.143:0/2681118626 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d27fff700 1 -- 172.21.15.143:0/2681118626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28109710 con 0x7f8d28108510 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d27fff700 1 --2- 172.21.15.143:0/2681118626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d180098a0 tx=0x7f8d18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf5383844aaae68 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d26ffd700 1 -- 172.21.15.143:0/2681118626 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d18004160 con 0x7f8d28108510 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d26ffd700 1 -- 172.21.15.143:0/2681118626 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8d18004300 con 0x7f8d28108510 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d26ffd700 1 -- 172.21.15.143:0/2681118626 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d18011930 con 0x7f8d28108510 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- 172.21.15.143:0/2681118626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 msgr2=0x7f8d28108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 --2- 172.21.15.143:0/2681118626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d180098a0 tx=0x7f8d18009bb0 comp rx=0 tx=0).stop 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- 172.21.15.143:0/2681118626 shutdown_connections 2024-03-11T21:14:02.795 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 --2- 172.21.15.143:0/2681118626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d28108930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- 172.21.15.143:0/2681118626 >> 172.21.15.143:0/2681118626 conn(0x7f8d2807b8e0 msgr2=0x7f8d28106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- 172.21.15.143:0/2681118626 shutdown_connections 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.855+0000 7f8d2e344700 1 -- 172.21.15.143:0/2681118626 wait complete. 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 librados: starting msgr at 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 librados: starting objecter 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 Processor -- start 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 -- start start 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 librados: setting wanted keys 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 librados: calling monclient init 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2810c940 con 0x7f8d28108510 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d27fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d27fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58372/0 (socket says 172.21.15.143:58372) 2024-03-11T21:14:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d27fff700 1 -- 172.21.15.143:0/333126447 learned_addr learned my addr 172.21.15.143:0/333126447 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d27fff700 1 -- 172.21.15.143:0/333126447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2819c9d0 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d27fff700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d18006af0 tx=0x7f8d18011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 10 client.?.objecter ms_handle_connect 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d180039e0 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8d18003b80 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2819cc60 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d18019620 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2819f950 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 librados: init done 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f8d18019940 con 0x7f8d28108510 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 0x7f8d1003ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8d1804d7e0 con 0x7f8d28108510 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 10 client.14170.objecter ms_dispatch 0x7f8d28001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 20 client.14170.objecter dump_active .. 0 homeless 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2810cac0 con 0x7f8d28108510 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d277fe700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 0x7f8d1003ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d277fe700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 0x7f8d1003ce30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006fd0 tx=0x7f8d1c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.859+0000 7f8d257fa700 10 client.14170.objecter ms_handle_connect 0x7f8d1003a970 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:01.863+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8d1801ab00 con 0x7f8d28108510 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.215+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f8d2819fb90 con 0x7f8d28108510 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.215+0000 7f8d257fa700 1 -- 172.21.15.143:0/333126447 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f8d18032030 con 0x7f8d28108510 2024-03-11T21:14:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 10 librados: watch_flush enter 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 10 librados: watch_flush exit 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 20 client.14170.objecter shutdown clearing up homeless session... 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 10 client.14170.objecter successfully canceled tick 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 msgr2=0x7f8d1003ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 0x7f8d1003ce30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006fd0 tx=0x7f8d1c006e40 comp rx=0 tx=0).stop 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 msgr2=0x7f8d2819c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d18006af0 tx=0x7f8d18011af0 comp rx=0 tx=0).stop 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 shutdown_connections 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f8d1003a970 0x7f8d1003ce30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 --2- 172.21.15.143:0/333126447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d28108510 0x7f8d2819c490 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 >> 172.21.15.143:0/333126447 conn(0x7f8d2807b8e0 msgr2=0x7f8d28105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 shutdown_connections 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 -- 172.21.15.143:0/333126447 wait complete. 2024-03-11T21:14:02.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:02.219+0000 7f8d2e344700 1 librados: shutdown 2024-03-11T21:14:04.789 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f7324d81700 1 Processor -- start 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f7324d81700 1 -- start start 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f7324d81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f7324d81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7320106c50 con 0x7f7320106260 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51864/0 (socket says 172.21.15.143:51864) 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731e59c700 1 -- 172.21.15.143:0/1428896912 learned_addr learned my addr 172.21.15.143:0/1428896912 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731e59c700 1 -- 172.21.15.143:0/1428896912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320107460 con 0x7f7320106260 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731e59c700 1 --2- 172.21.15.143:0/1428896912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73100098a0 tx=0x7f7310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba691bf3abd8b0f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731d59a700 1 -- 172.21.15.143:0/1428896912 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7310004160 con 0x7f7320106260 2024-03-11T21:14:04.790 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731d59a700 1 -- 172.21.15.143:0/1428896912 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7310004300 con 0x7f7320106260 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.630+0000 7f731d59a700 1 -- 172.21.15.143:0/1428896912 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7310011930 con 0x7f7320106260 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/1428896912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 msgr2=0x7f7320106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 --2- 172.21.15.143:0/1428896912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73100098a0 tx=0x7f7310009bb0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/1428896912 shutdown_connections 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 --2- 172.21.15.143:0/1428896912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f7320106680 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/1428896912 >> 172.21.15.143:0/1428896912 conn(0x7f73201017e0 msgr2=0x7f7320103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/1428896912 shutdown_connections 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/1428896912 wait complete. 2024-03-11T21:14:04.791 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 librados: starting msgr at 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 librados: starting objecter 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 Processor -- start 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- start start 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 librados: setting wanted keys 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 librados: calling monclient init 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732010a690 con 0x7f7320106260 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51878/0 (socket says 172.21.15.143:51878) 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731e59c700 1 -- 172.21.15.143:0/2285002917 learned_addr learned my addr 172.21.15.143:0/2285002917 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731e59c700 1 -- 172.21.15.143:0/2285002917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732019a690 con 0x7f7320106260 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731e59c700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7310009fb0 tx=0x7f7310011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 10 client.?.objecter ms_handle_connect 0x7f7320106260 2024-03-11T21:14:04.792 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:04.793 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73100119a0 con 0x7f7320106260 2024-03-11T21:14:04.793 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7310003b20 con 0x7f7320106260 2024-03-11T21:14:04.793 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732019a920 con 0x7f7320106260 2024-03-11T21:14:04.793 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f731001ad50 con 0x7f7320106260 2024-03-11T21:14:04.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:04.796 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732019d610 con 0x7f7320106260 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 librados: init done 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f731002d440 con 0x7f7320106260 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732010a810 con 0x7f7320106260 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 0x7f730c03ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f731004d3e0 con 0x7f7320106260 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 10 client.14172.objecter ms_dispatch 0x7f7320001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f73177fe700 20 client.14172.objecter dump_active .. 0 homeless 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.634+0000 7f731dd9b700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 0x7f730c03ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.638+0000 7f73177fe700 10 client.14172.objecter ms_handle_connect 0x7f730c03aa10 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.638+0000 7f731dd9b700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 0x7f730c03ced0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f730800ad30 tx=0x7f73080093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:03.642+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f731002b340 con 0x7f7320106260 2024-03-11T21:14:04.797 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.198+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f732019d9c0 con 0x7f7320106260 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.202+0000 7f73177fe700 1 -- 172.21.15.143:0/2285002917 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) v1 ==== 153+0+0 (secure 0 0 0) 0x7f731000b7b0 con 0x7f7320106260 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 10 librados: watch_flush enter 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 10 librados: watch_flush exit 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 20 client.14172.objecter shutdown clearing up homeless session... 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 10 client.14172.objecter successfully canceled tick 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 msgr2=0x7f730c03ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 0x7f730c03ced0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f730800ad30 tx=0x7f73080093f0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 msgr2=0x7f732019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7310009fb0 tx=0x7f7310011ea0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 shutdown_connections 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f730c03aa10 0x7f730c03ced0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 --2- 172.21.15.143:0/2285002917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7320106260 0x7f732019a150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 >> 172.21.15.143:0/2285002917 conn(0x7f73201017e0 msgr2=0x7f7320102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 shutdown_connections 2024-03-11T21:14:04.798 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 -- 172.21.15.143:0/2285002917 wait complete. 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr 2024-03-11T21:14:04.206+0000 7f7324d81700 1 librados: shutdown 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: ceph telemetry on 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout:For more information see: 2024-03-11T21:14:04.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.800 INFO:teuthology.orchestra.run.smithi143.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:14:04.800 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:04.800 INFO:teuthology.orchestra.run.smithi143.stdout:Bootstrap complete. 2024-03-11T21:14:04.851 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:14:04.851 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:14:04.851 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:14:04.899 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:14:04.899 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:14:04.899 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:14:04.947 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:14:04.947 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:14:04.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:14:05.007 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:14:05.007 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:14:05.007 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:14:05.055 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:14:05.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2rHwTO1Q7RJEqDeyO2sJ2Gr9K79tP6UEF+o4b4G5XAJb10vunWzLueHGxNWb0sYuhi4LwFRcy1beuq++uCcTPz7Ska6V4IWfPUlrAGxp6KPIgHM8XXkSBbFJQpS52LIffRNcIkbsUbdsdSObT8F1MtfDaJtINo+V8/XUGjdU5b20LxY6e8GJR8asBPKoZux+DmFkHWAShPqjwBFQ/40UBUytpcP5UthZGpL6SqjyQ6MHvuaqlVeQuWGDe69Ycg7cOsukFMvb7xMTDG7CJrqpGnuDBIoP81taINR5AaCwFQSGMY1HLg6KWTLFcT8BWkAvy0mgX8kBmr2EbANf4W+owLbLhvbyq3mbfNOJiywQwPnfXLc8Pf5U0fY/KQ8WlypJJAiiBcc4/7dG4IKMc5YGAb6b1qePTLKt/s/Lmen6B2xdCXXF47ItyUscO+y9tn778Hj8fRmv+1QRdoOEhyG1ENMpNANa9ZLtxwfEv4wLwQBXrP9bwRirj+jyPEA6phGk= ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:14:05.120 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2rHwTO1Q7RJEqDeyO2sJ2Gr9K79tP6UEF+o4b4G5XAJb10vunWzLueHGxNWb0sYuhi4LwFRcy1beuq++uCcTPz7Ska6V4IWfPUlrAGxp6KPIgHM8XXkSBbFJQpS52LIffRNcIkbsUbdsdSObT8F1MtfDaJtINo+V8/XUGjdU5b20LxY6e8GJR8asBPKoZux+DmFkHWAShPqjwBFQ/40UBUytpcP5UthZGpL6SqjyQ6MHvuaqlVeQuWGDe69Ycg7cOsukFMvb7xMTDG7CJrqpGnuDBIoP81taINR5AaCwFQSGMY1HLg6KWTLFcT8BWkAvy0mgX8kBmr2EbANf4W+owLbLhvbyq3mbfNOJiywQwPnfXLc8Pf5U0fY/KQ8WlypJJAiiBcc4/7dG4IKMc5YGAb6b1qePTLKt/s/Lmen6B2xdCXXF47ItyUscO+y9tn778Hj8fRmv+1QRdoOEhyG1ENMpNANa9ZLtxwfEv4wLwQBXrP9bwRirj+jyPEA6phGk= ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:14:05.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2rHwTO1Q7RJEqDeyO2sJ2Gr9K79tP6UEF+o4b4G5XAJb10vunWzLueHGxNWb0sYuhi4LwFRcy1beuq++uCcTPz7Ska6V4IWfPUlrAGxp6KPIgHM8XXkSBbFJQpS52LIffRNcIkbsUbdsdSObT8F1MtfDaJtINo+V8/XUGjdU5b20LxY6e8GJR8asBPKoZux+DmFkHWAShPqjwBFQ/40UBUytpcP5UthZGpL6SqjyQ6MHvuaqlVeQuWGDe69Ycg7cOsukFMvb7xMTDG7CJrqpGnuDBIoP81taINR5AaCwFQSGMY1HLg6KWTLFcT8BWkAvy0mgX8kBmr2EbANf4W+owLbLhvbyq3mbfNOJiywQwPnfXLc8Pf5U0fY/KQ8WlypJJAiiBcc4/7dG4IKMc5YGAb6b1qePTLKt/s/Lmen6B2xdCXXF47ItyUscO+y9tn778Hj8fRmv+1QRdoOEhyG1ENMpNANa9ZLtxwfEv4wLwQBXrP9bwRirj+jyPEA6phGk= ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:14:05.155 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2rHwTO1Q7RJEqDeyO2sJ2Gr9K79tP6UEF+o4b4G5XAJb10vunWzLueHGxNWb0sYuhi4LwFRcy1beuq++uCcTPz7Ska6V4IWfPUlrAGxp6KPIgHM8XXkSBbFJQpS52LIffRNcIkbsUbdsdSObT8F1MtfDaJtINo+V8/XUGjdU5b20LxY6e8GJR8asBPKoZux+DmFkHWAShPqjwBFQ/40UBUytpcP5UthZGpL6SqjyQ6MHvuaqlVeQuWGDe69Ycg7cOsukFMvb7xMTDG7CJrqpGnuDBIoP81taINR5AaCwFQSGMY1HLg6KWTLFcT8BWkAvy0mgX8kBmr2EbANf4W+owLbLhvbyq3mbfNOJiywQwPnfXLc8Pf5U0fY/KQ8WlypJJAiiBcc4/7dG4IKMc5YGAb6b1qePTLKt/s/Lmen6B2xdCXXF47ItyUscO+y9tn778Hj8fRmv+1QRdoOEhyG1ENMpNANa9ZLtxwfEv4wLwQBXrP9bwRirj+jyPEA6phGk= ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:14:05.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:14:05.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:05 smithi143 bash[21512]: audit 2024-03-11T21:14:04.206484+0000 mon.a (mon.0) 92 : audit 1 from='client.? 172.21.15.143:0/2285002917' entity='client.admin' 2024-03-11T21:14:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/854810488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 msgr2=0x7fc164103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/854810488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc164103e50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc154009910 tx=0x7fc154009c20 comp rx=0 tx=0).stop 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/854810488 shutdown_connections 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/854810488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc164103e50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/854810488 >> 172.21.15.143:0/854810488 conn(0x7fc1640fefd0 msgr2=0x7fc164101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/854810488 shutdown_connections 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/854810488 wait complete. 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 librados: starting msgr at 2024-03-11T21:14:12.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 librados: starting objecter 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 Processor -- start 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- start start 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 librados: setting wanted keys 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 librados: calling monclient init 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164108f90 con 0x7fc164103a30 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc163fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc163fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51898/0 (socket says 172.21.15.143:51898) 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc163fff700 1 -- 172.21.15.143:0/2006002318 learned_addr learned my addr 172.21.15.143:0/2006002318 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc163fff700 1 -- 172.21.15.143:0/2006002318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164198d00 con 0x7fc164103a30 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc163fff700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc154005b40 tx=0x7fc1540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 10 client.?.objecter ms_handle_connect 0x7fc164103a30 2024-03-11T21:14:12.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:12.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc154027070 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc15400f460 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc164198f90 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc154017440 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16419bc80 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc16a3d8700 1 librados: init done 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.478+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc15402a070 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 0x7fc15003ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc164109710 con 0x7fc164103a30 2024-03-11T21:14:12.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc15404d1f0 con 0x7fc164103a30 2024-03-11T21:14:12.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 10 client.14174.objecter ms_dispatch 0x7fc164001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:12.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:12.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:12.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.482+0000 7fc1617fa700 20 client.14174.objecter dump_active .. 0 homeless 2024-03-11T21:14:12.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.486+0000 7fc1637fe700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 0x7fc15003ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:12.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.486+0000 7fc1617fa700 10 client.14174.objecter ms_handle_connect 0x7fc15003a970 2024-03-11T21:14:12.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.486+0000 7fc1637fe700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 0x7fc15003ce30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc14c006fd0 tx=0x7fc14c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:12.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.486+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc154031070 con 0x7fc164103a30 2024-03-11T21:14:12.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.694+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc16402d830 con 0x7fc164103a30 2024-03-11T21:14:12.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.698+0000 7fc1617fa700 1 -- 172.21.15.143:0/2006002318 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc154013070 con 0x7fc164103a30 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 10 librados: watch_flush enter 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 10 librados: watch_flush exit 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 20 client.14174.objecter shutdown clearing up homeless session... 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 10 client.14174.objecter successfully canceled tick 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 msgr2=0x7fc15003ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 0x7fc15003ce30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc14c006fd0 tx=0x7fc14c006e40 comp rx=0 tx=0).stop 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 msgr2=0x7fc1641987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:12.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc154005b40 tx=0x7fc1540057f0 comp rx=0 tx=0).stop 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 shutdown_connections 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fc15003a970 0x7fc15003ce30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 --2- 172.21.15.143:0/2006002318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc164103a30 0x7fc1641987c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 >> 172.21.15.143:0/2006002318 conn(0x7fc1640fefd0 msgr2=0x7fc1640ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 shutdown_connections 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 -- 172.21.15.143:0/2006002318 wait complete. 2024-03-11T21:14:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:12.702+0000 7fc16a3d8700 1 librados: shutdown 2024-03-11T21:14:13.396 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.158770+0000 mon.a (mon.0) 93 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.165157+0000 mon.a (mon.0) 94 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.166787+0000 mon.a (mon.0) 95 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.174571+0000 mon.a (mon.0) 96 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.210431+0000 mon.a (mon.0) 97 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.218543+0000 mon.a (mon.0) 98 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.703644+0000 mon.a (mon.0) 99 : audit 1 from='client.? 172.21.15.143:0/2006002318' entity='client.admin' 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.767513+0000 mon.a (mon.0) 100 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:13.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:13 smithi143 bash[21512]: audit 2024-03-11T21:14:12.771746+0000 mon.a (mon.0) 101 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:13.398 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:14:13.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:14:15.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:15 smithi143 bash[21512]: cluster 2024-03-11T21:14:13.412986+0000 mgr.y (mgr.14152) 9 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:17.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:17 smithi143 bash[21512]: cluster 2024-03-11T21:14:15.413434+0000 mgr.y (mgr.14152) 10 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:18.107 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:19.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:19 smithi143 bash[21512]: cluster 2024-03-11T21:14:17.413944+0000 mgr.y (mgr.14152) 11 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/2793288752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 msgr2=0x7f430c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 --2- 172.21.15.143:0/2793288752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c100b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4308009910 tx=0x7f4308009c20 comp rx=0 tx=0).stop 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/2793288752 shutdown_connections 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 --2- 172.21.15.143:0/2793288752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c100b20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/2793288752 >> 172.21.15.143:0/2793288752 conn(0x7f430c0fbce0 msgr2=0x7f430c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/2793288752 shutdown_connections 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/2793288752 wait complete. 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 librados: starting msgr at 2024-03-11T21:14:20.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 librados: starting objecter 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 Processor -- start 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- start start 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 librados: setting wanted keys 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 librados: calling monclient init 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:20.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c105c60 con 0x7f430c100700 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43123f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43123f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40252/0 (socket says 172.21.15.143:40252) 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43123f7700 1 -- 172.21.15.143:0/3663644291 learned_addr learned my addr 172.21.15.143:0/3663644291 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43123f7700 1 -- 172.21.15.143:0/3663644291 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c1959d0 con 0x7f430c100700 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43123f7700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4308005b40 tx=0x7f43080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 10 client.?.objecter ms_handle_connect 0x7f430c100700 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4308027070 con 0x7f430c100700 2024-03-11T21:14:20.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f430800f460 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c195c60 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4308017440 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c198950 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 librados: init done 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f430802a070 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c1063e0 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 0x7f42f803ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f430804d160 con 0x7f430c100700 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 10 client.14176.objecter ms_dispatch 0x7f430c001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f43037fe700 20 client.14176.objecter dump_active .. 0 homeless 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.530+0000 7f4311bf6700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 0x7f42f803ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:20.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.534+0000 7f4311bf6700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 0x7f42f803ce80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006fd0 tx=0x7f42fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:20.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.534+0000 7f43037fe700 10 client.14176.objecter ms_handle_connect 0x7f42f803a9c0 2024-03-11T21:14:20.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.538+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4308013070 con 0x7f430c100700 2024-03-11T21:14:20.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.746+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f430c0fd850 con 0x7f42f803a9c0 2024-03-11T21:14:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f43037fe700 1 -- 172.21.15.143:0/3663644291 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f430c0fd850 con 0x7f42f803a9c0 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 10 librados: watch_flush enter 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 10 librados: watch_flush exit 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 20 client.14176.objecter shutdown clearing up homeless session... 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 10 client.14176.objecter successfully canceled tick 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 msgr2=0x7f42f803ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 0x7f42f803ce80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006fd0 tx=0x7f42fc006e40 comp rx=0 tx=0).stop 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 msgr2=0x7f430c195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4308005b40 tx=0x7f43080057f0 comp rx=0 tx=0).stop 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 shutdown_connections 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f42f803a9c0 0x7f42f803ce80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:20.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 --2- 172.21.15.143:0/3663644291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c100700 0x7f430c195490 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:20.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.750+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 >> 172.21.15.143:0/3663644291 conn(0x7f430c0fbce0 msgr2=0x7f430c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:20.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.754+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 shutdown_connections 2024-03-11T21:14:20.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.754+0000 7f431465b700 1 -- 172.21.15.143:0/3663644291 wait complete. 2024-03-11T21:14:20.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:20.754+0000 7f431465b700 1 librados: shutdown 2024-03-11T21:14:21.404 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:21 smithi143 bash[21512]: cluster 2024-03-11T21:14:19.414497+0000 mgr.y (mgr.14152) 12 : cluster 0 pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:21.404 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:21 smithi143 bash[21512]: audit 2024-03-11T21:14:20.752823+0000 mon.a (mon.0) 102 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:21.404 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:21 smithi143 bash[21512]: audit 2024-03-11T21:14:20.771512+0000 mon.a (mon.0) 103 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:21.404 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:21 smithi143 bash[21512]: audit 2024-03-11T21:14:20.773436+0000 mon.a (mon.0) 104 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:21.404 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:21 smithi143 bash[21512]: audit 2024-03-11T21:14:20.774130+0000 mon.a (mon.0) 105 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:21.405 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-03-11T21:14:21.405 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:14:21.405 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:14:21.411 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:14:21.411 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:14:21.460 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-03-11T21:14:21.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch host add smithi203 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: audit 2024-03-11T21:14:20.749534+0000 mgr.y (mgr.14152) 13 : audit 0 from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: cephadm 2024-03-11T21:14:20.775157+0000 mgr.y (mgr.14152) 14 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: cephadm 2024-03-11T21:14:20.889355+0000 mgr.y (mgr.14152) 15 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: cephadm 2024-03-11T21:14:21.003663+0000 mgr.y (mgr.14152) 16 : cephadm 1 Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: cephadm 2024-03-11T21:14:21.117903+0000 mgr.y (mgr.14152) 17 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: audit 2024-03-11T21:14:21.236413+0000 mon.a (mon.0) 106 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: audit 2024-03-11T21:14:21.240370+0000 mon.a (mon.0) 107 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:22.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:22 smithi143 bash[21512]: audit 2024-03-11T21:14:21.244931+0000 mon.a (mon.0) 108 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:23.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:23 smithi143 bash[21512]: cluster 2024-03-11T21:14:21.414910+0000 mgr.y (mgr.14152) 18 : cluster 0 pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:25.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:25 smithi143 bash[21512]: cluster 2024-03-11T21:14:23.415323+0000 mgr.y (mgr.14152) 19 : cluster 0 pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:26.168 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:27.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:27 smithi143 bash[21512]: cluster 2024-03-11T21:14:25.415736+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:28.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/3727487211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 msgr2=0x7f85fc109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 --2- 172.21.15.143:0/3727487211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc109db0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f85f400b0a0 tx=0x7f85f400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/3727487211 shutdown_connections 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 --2- 172.21.15.143:0/3727487211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc109db0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/3727487211 >> 172.21.15.143:0/3727487211 conn(0x7f85fc0ff4a0 msgr2=0x7f85fc1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/3727487211 shutdown_connections 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/3727487211 wait complete. 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 librados: starting msgr at 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 librados: starting objecter 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 Processor -- start 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- start start 2024-03-11T21:14:28.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 librados: setting wanted keys 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 librados: calling monclient init 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc10d4d0 con 0x7f85fc1099d0 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8600c23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8600c23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51706/0 (socket says 172.21.15.143:51706) 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8600c23700 1 -- 172.21.15.143:0/1624106551 learned_addr learned my addr 172.21.15.143:0/1624106551 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8600c23700 1 -- 172.21.15.143:0/1624106551 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc19ecf0 con 0x7f85fc1099d0 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8600c23700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85f4015040 tx=0x7f85f400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 10 client.?.objecter ms_handle_connect 0x7f85fc1099d0 2024-03-11T21:14:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f4017040 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f85f4022070 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc19ef80 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f40047e0 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc198970 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 librados: init done 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f85f4027460 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc105110 con 0x7f85fc1099d0 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 0x7f85e403ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:28.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f85f400f050 con 0x7f85fc1099d0 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 10 client.14178.objecter ms_dispatch 0x7f85fc001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.513+0000 7f85f9ffb700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.517+0000 7f85f9ffb700 20 client.14178.objecter dump_active .. 0 homeless 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.517+0000 7f85fbfff700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 0x7f85e403ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.517+0000 7f85f9ffb700 10 client.14178.objecter ms_handle_connect 0x7f85e403a9c0 2024-03-11T21:14:28.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.517+0000 7f85fbfff700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 0x7f85e403ce80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85f000ad30 tx=0x7f85f00093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:28.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.521+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f85f4025360 con 0x7f85fc1099d0 2024-03-11T21:14:28.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:28.729+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7f85fc199ea0 con 0x7f85e403a9c0 2024-03-11T21:14:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:29 smithi143 bash[21512]: cluster 2024-03-11T21:14:27.416222+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:30.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:30 smithi143 bash[21512]: audit 2024-03-11T21:14:28.735569+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:14:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:31 smithi143 bash[21512]: cluster 2024-03-11T21:14:29.416658+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:31 smithi143 bash[21512]: cephadm 2024-03-11T21:14:29.632543+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi203 2024-03-11T21:14:33.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:33.517+0000 7f8602686700 10 client.14178.objecter tick 2024-03-11T21:14:33.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:33 smithi143 bash[21512]: cluster 2024-03-11T21:14:31.417090+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:34.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.709+0000 7f85f9ffb700 1 -- 172.21.15.143:0/1624106551 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f85fc199ea0 con 0x7f85e403a9c0 2024-03-11T21:14:34.701 INFO:teuthology.orchestra.run.smithi143.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-03-11T21:14:34.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 10 librados: watch_flush enter 2024-03-11T21:14:34.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 10 librados: watch_flush exit 2024-03-11T21:14:34.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 20 client.14178.objecter shutdown clearing up homeless session... 2024-03-11T21:14:34.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 10 client.14178.objecter successfully canceled tick 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 msgr2=0x7f85e403ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 0x7f85e403ce80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85f000ad30 tx=0x7f85f00093f0 comp rx=0 tx=0).stop 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 msgr2=0x7f85fc19e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85f4015040 tx=0x7f85f400bd40 comp rx=0 tx=0).stop 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 shutdown_connections 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f85e403a9c0 0x7f85e403ce80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 --2- 172.21.15.143:0/1624106551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85fc1099d0 0x7f85fc19e7b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 >> 172.21.15.143:0/1624106551 conn(0x7f85fc0ff4a0 msgr2=0x7f85fc10a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 shutdown_connections 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 -- 172.21.15.143:0/1624106551 wait complete. 2024-03-11T21:14:34.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:34.713+0000 7f8602e87700 1 librados: shutdown 2024-03-11T21:14:35.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:14:35.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:35 smithi143 bash[21512]: cluster 2024-03-11T21:14:33.417470+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:35.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:35 smithi143 bash[21512]: audit 2024-03-11T21:14:34.712546+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:35.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:35 smithi143 bash[21512]: audit 2024-03-11T21:14:34.720254+0000 mon.a (mon.0) 110 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:35.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:35 smithi143 bash[21512]: audit 2024-03-11T21:14:35.193069+0000 mon.a (mon.0) 111 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:36.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:36 smithi143 bash[21512]: cephadm 2024-03-11T21:14:34.713413+0000 mgr.y (mgr.14152) 27 : cephadm 1 Added host smithi203 2024-03-11T21:14:37.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:37 smithi143 bash[21512]: cluster 2024-03-11T21:14:35.417936+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:37.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:37 smithi143 bash[21512]: audit 2024-03-11T21:14:36.719431+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:39.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:38 smithi143 bash[21512]: cluster 2024-03-11T21:14:37.418366+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:39.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:38 smithi143 bash[21512]: audit 2024-03-11T21:14:37.915643+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:40.030 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:41.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:40 smithi143 bash[21512]: cluster 2024-03-11T21:14:39.418752+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- 172.21.15.143:0/159135071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 msgr2=0x7f271c108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 --2- 172.21.15.143:0/159135071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c108420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2718009910 tx=0x7f2718009c20 comp rx=0 tx=0).stop 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- 172.21.15.143:0/159135071 shutdown_connections 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 --2- 172.21.15.143:0/159135071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c108420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- 172.21.15.143:0/159135071 >> 172.21.15.143:0/159135071 conn(0x7f271c077610 msgr2=0x7f271c079a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- 172.21.15.143:0/159135071 shutdown_connections 2024-03-11T21:14:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- 172.21.15.143:0/159135071 wait complete. 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 librados: starting msgr at 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 librados: starting objecter 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 Processor -- start 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- start start 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 librados: setting wanted keys 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 librados: calling monclient init 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2723468700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c1118d0 con 0x7f271c108040 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2721204700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2721204700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39354/0 (socket says 172.21.15.143:39354) 2024-03-11T21:14:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.473+0000 7f2721204700 1 -- 172.21.15.143:0/2328460355 learned_addr learned my addr 172.21.15.143:0/2328460355 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2721204700 1 -- 172.21.15.143:0/2328460355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c19d4f0 con 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2721204700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2718004f80 tx=0x7f2718005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 10 client.?.objecter ms_handle_connect 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2718004390 con 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2718004530 con 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2718003aa0 con 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c19d780 con 0x7f271c108040 2024-03-11T21:14:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2723468700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1a1880 con 0x7f271c108040 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2723468700 1 librados: init done 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2718003d10 con 0x7f271c108040 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c109510 con 0x7f271c108040 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 0x7f270803ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f271801e070 con 0x7f271c108040 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 10 client.14180.objecter ms_dispatch 0x7f271c001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 20 client.14180.objecter dump_active .. 0 homeless 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2720a03700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 0x7f270803ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f27127fc700 10 client.14180.objecter ms_handle_connect 0x7f270803a9c0 2024-03-11T21:14:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.477+0000 7f2720a03700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 0x7f270803ce80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f270c006fd0 tx=0x7f270c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:42.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.481+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2718013070 con 0x7f271c108040 2024-03-11T21:14:42.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.693+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f271c063410 con 0x7f270803a9c0 2024-03-11T21:14:42.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f27127fc700 1 -- 172.21.15.143:0/2328460355 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f271c063410 con 0x7f270803a9c0 2024-03-11T21:14:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:14:42.687 INFO:teuthology.orchestra.run.smithi143.stdout:[{"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 10 librados: watch_flush enter 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 10 librados: watch_flush exit 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 20 client.14180.objecter shutdown clearing up homeless session... 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 10 client.14180.objecter successfully canceled tick 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 msgr2=0x7f270803ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 0x7f270803ce80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f270c006fd0 tx=0x7f270c006e40 comp rx=0 tx=0).stop 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 msgr2=0x7f271c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2718004f80 tx=0x7f2718005060 comp rx=0 tx=0).stop 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 shutdown_connections 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f270803a9c0 0x7f270803ce80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 --2- 172.21.15.143:0/2328460355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f271c108040 0x7f271c19cfb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 >> 172.21.15.143:0/2328460355 conn(0x7f271c077610 msgr2=0x7f271c10ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 shutdown_connections 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 -- 172.21.15.143:0/2328460355 wait complete. 2024-03-11T21:14:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:42.697+0000 7f2723468700 1 librados: shutdown 2024-03-11T21:14:43.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:42 smithi143 bash[21512]: cluster 2024-03-11T21:14:41.419122+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:43.405 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-03-11T21:14:43.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd crush tunables jewel 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:42.699681+0000 mgr.y (mgr.14152) 32 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.198698+0000 mon.a (mon.0) 114 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.203477+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.209063+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.212903+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.214359+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.216164+0000 mon.a (mon.0) 119 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.217642+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: cephadm 2024-03-11T21:14:43.220880+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.835311+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.840911+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:44.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:44 smithi143 bash[21512]: audit 2024-03-11T21:14:43.846775+0000 mon.a (mon.0) 123 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:45 smithi143 bash[21512]: cluster 2024-03-11T21:14:43.419474+0000 mgr.y (mgr.14152) 34 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:45 smithi143 bash[21512]: cephadm 2024-03-11T21:14:43.436359+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:14:45.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:45 smithi143 bash[21512]: cephadm 2024-03-11T21:14:43.600952+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:14:45.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:45 smithi143 bash[21512]: cephadm 2024-03-11T21:14:43.714909+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:14:47.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:47 smithi143 bash[21512]: cluster 2024-03-11T21:14:45.419844+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:48.116 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:49.631 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:49 smithi143 bash[21512]: cluster 2024-03-11T21:14:47.420218+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:50.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/495727071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 msgr2=0x7fd0dc106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:50.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/495727071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc106ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009910 tx=0x7fd0cc009c20 comp rx=0 tx=0).stop 2024-03-11T21:14:50.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/495727071 shutdown_connections 2024-03-11T21:14:50.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/495727071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc106ac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:50.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/495727071 >> 172.21.15.143:0/495727071 conn(0x7fd0dc0fc210 msgr2=0x7fd0dc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/495727071 shutdown_connections 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/495727071 wait complete. 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 librados: starting msgr at 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 librados: starting objecter 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 Processor -- start 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- start start 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 librados: setting wanted keys 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 librados: calling monclient init 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc10a1e0 con 0x7fd0dc1066e0 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:55510/0 (socket says 172.21.15.143:55510) 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0dbfff700 1 -- 172.21.15.143:0/4063246455 learned_addr learned my addr 172.21.15.143:0/4063246455 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0dbfff700 1 -- 172.21.15.143:0/4063246455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc19cba0 con 0x7fd0dc1066e0 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0dbfff700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0098e0 tx=0x7fd0cc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0d97fa700 10 client.?.objecter ms_handle_connect 0x7fd0dc1066e0 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0d97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:50.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0cc027070 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd0cc00bde0 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0cc022c10 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc19ce90 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1967e0 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.517+0000 7fd0e26d3700 1 librados: init done 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fd0cc022470 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc101e20 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 0x7fd0c803ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd0cc01e030 con 0x7fd0dc1066e0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 10 client.14182.objecter ms_dispatch 0x7fd0dc001550 osd_map(3..3 src has 1..3) v4 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 3 client.14182.objecter handle_osd_map got epochs [3,3] > 0 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 3 client.14182.objecter handle_osd_map decoding full epoch 3 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 20 client.14182.objecter dump_active .. 0 homeless 2024-03-11T21:14:50.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0db7fe700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 0x7fd0c803ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:50.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0db7fe700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 0x7fd0c803ce80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400ad30 tx=0x7fd0c40093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:50.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.521+0000 7fd0d97fa700 10 client.14182.objecter ms_handle_connect 0x7fd0c803a9c0 2024-03-11T21:14:50.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.525+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd0cc013070 con 0x7fd0dc1066e0 2024-03-11T21:14:50.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:50.741+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7fd0dc063410 con 0x7fd0dc1066e0 2024-03-11T21:14:51.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.221+0000 7fd0d97fa700 1 -- 172.21.15.143:0/4063246455 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd0cc030070 con 0x7fd0dc1066e0 2024-03-11T21:14:51.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 10 librados: watch_flush enter 2024-03-11T21:14:51.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 10 librados: watch_flush exit 2024-03-11T21:14:51.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 20 client.14182.objecter shutdown clearing up homeless session... 2024-03-11T21:14:51.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 10 client.14182.objecter successfully canceled tick 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 msgr2=0x7fd0c803ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 0x7fd0c803ce80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400ad30 tx=0x7fd0c40093f0 comp rx=0 tx=0).stop 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 msgr2=0x7fd0dc19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0098e0 tx=0x7fd0cc005030 comp rx=0 tx=0).stop 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 shutdown_connections 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd0c803a9c0 0x7fd0c803ce80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 --2- 172.21.15.143:0/4063246455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0dc1066e0 0x7fd0dc19c660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 >> 172.21.15.143:0/4063246455 conn(0x7fd0dc0fc210 msgr2=0x7fd0dc06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 shutdown_connections 2024-03-11T21:14:51.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 -- 172.21.15.143:0/4063246455 wait complete. 2024-03-11T21:14:51.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:14:51.225+0000 7fd0e26d3700 1 librados: shutdown 2024-03-11T21:14:51.219 INFO:teuthology.orchestra.run.smithi143.stderr:adjusted tunables profile to jewel 2024-03-11T21:14:51.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:51 smithi143 bash[21512]: cluster 2024-03-11T21:14:49.420610+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:51.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:51 smithi143 bash[21512]: audit 2024-03-11T21:14:50.746903+0000 mon.a (mon.0) 124 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-11T21:14:51.886 INFO:tasks.cephadm:Adding mon.a on smithi143 2024-03-11T21:14:51.886 INFO:tasks.cephadm:Adding mon.c on smithi143 2024-03-11T21:14:51.886 INFO:tasks.cephadm:Adding mon.b on smithi203 2024-03-11T21:14:51.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b' 2024-03-11T21:14:52.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:52 smithi143 bash[21512]: audit 2024-03-11T21:14:51.224592+0000 mon.a (mon.0) 125 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-11T21:14:52.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:52 smithi143 bash[21512]: cluster 2024-03-11T21:14:51.229997+0000 mon.a (mon.0) 126 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:14:53.094 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:14:53.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:53 smithi143 bash[21512]: cluster 2024-03-11T21:14:51.420975+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:55.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/2660437595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 msgr2=0x7fd49c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:55.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/2660437595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c107ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009910 tx=0x7fd48c009c20 comp rx=0 tx=0).stop 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/2660437595 shutdown_connections 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/2660437595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c107ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/2660437595 >> 172.21.15.203:0/2660437595 conn(0x7fd49c0fd2d0 msgr2=0x7fd49c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/2660437595 shutdown_connections 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/2660437595 wait complete. 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 librados: starting msgr at 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 librados: starting objecter 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 Processor -- start 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- start start 2024-03-11T21:14:55.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 librados: setting wanted keys 2024-03-11T21:14:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 librados: calling monclient init 2024-03-11T21:14:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c10b2c0 con 0x7fd49c1077c0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a092c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a092c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.203:49288/0 (socket says 172.21.15.203:49288) 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a092c700 1 -- 172.21.15.203:0/4109631371 learned_addr learned my addr 172.21.15.203:0/4109631371 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a092c700 1 -- 172.21.15.203:0/4109631371 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c075f80 con 0x7fd49c1077c0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a092c700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd48c004f80 tx=0x7fd48c005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd499ffb700 10 client.?.objecter ms_handle_connect 0x7fd49c1077c0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd499ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd48c004390 con 0x7fd49c1077c0 2024-03-11T21:14:55.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c074620 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c074ae0 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd4a2b90700 1 librados: init done 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd48c004530 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.336+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd48c003990 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488005120 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fd48c0046e0 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 0x7fd48403ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd48c01e070 con 0x7fd49c1077c0 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 10 client.14184.objecter ms_dispatch 0x7fd49c001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:14:55.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:14:55.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:14:55.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 20 client.14184.objecter dump_active .. 0 homeless 2024-03-11T21:14:55.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd49bfff700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 0x7fd48403ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:14:55.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd499ffb700 10 client.14184.objecter ms_handle_connect 0x7fd48403a9c0 2024-03-11T21:14:55.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.340+0000 7fd49bfff700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 0x7fd48403ce80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd490006fd0 tx=0x7fd490006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:14:55.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.352+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd48c013070 con 0x7fd49c1077c0 2024-03-11T21:14:55.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.580+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fd488000ca0 con 0x7fd48403a9c0 2024-03-11T21:14:55.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd499ffb700 1 -- 172.21.15.203:0/4109631371 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd488000ca0 con 0x7fd48403a9c0 2024-03-11T21:14:55.584 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-03-11T21:14:55.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 10 librados: watch_flush enter 2024-03-11T21:14:55.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 10 librados: watch_flush exit 2024-03-11T21:14:55.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 20 client.14184.objecter shutdown clearing up homeless session... 2024-03-11T21:14:55.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 10 client.14184.objecter successfully canceled tick 2024-03-11T21:14:55.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 msgr2=0x7fd48403ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:55.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 0x7fd48403ce80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd490006fd0 tx=0x7fd490006e40 comp rx=0 tx=0).stop 2024-03-11T21:14:55.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 msgr2=0x7fd49c0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd48c004f80 tx=0x7fd48c005060 comp rx=0 tx=0).stop 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 shutdown_connections 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd48403a9c0 0x7fd48403ce80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 --2- 172.21.15.203:0/4109631371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd49c1077c0 0x7fd49c0740e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 >> 172.21.15.203:0/4109631371 conn(0x7fd49c0fd2d0 msgr2=0x7fd49c1097d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 shutdown_connections 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 -- 172.21.15.203:0/4109631371 wait complete. 2024-03-11T21:14:55.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:14:55.596+0000 7fd4a2b90700 1 librados: shutdown 2024-03-11T21:14:55.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:55 smithi143 bash[21512]: cluster 2024-03-11T21:14:53.421468+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:56.311 DEBUG:teuthology.orchestra.run.smithi143:mon.c> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:14:56.315 DEBUG:teuthology.orchestra.run.smithi203:mon.b> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:14:56.318 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:14:56.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:14:56.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: cluster 2024-03-11T21:14:55.421914+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.582072+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: cephadm 2024-03-11T21:14:55.588123+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b;count:3 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.596035+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.608003+0000 mon.a (mon.0) 128 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.611625+0000 mon.a (mon.0) 129 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.613198+0000 mon.a (mon.0) 130 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.623911+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.628473+0000 mon.a (mon.0) 132 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: audit 2024-03-11T21:14:55.630207+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:56.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:56 smithi143 bash[21512]: cephadm 2024-03-11T21:14:55.632345+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi203 2024-03-11T21:14:57.968 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:14:58.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:14:58 smithi143 bash[21512]: cluster 2024-03-11T21:14:57.422308+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:00.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:00 smithi203 systemd[1]: Started Ceph mon.b for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:15:00.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- 172.21.15.203:0/3470419812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 msgr2=0x7f35c00745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:00.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 --2- 172.21.15.203:0/3470419812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c00745f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009910 tx=0x7f35b4009c20 comp rx=0 tx=0).stop 2024-03-11T21:15:00.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- 172.21.15.203:0/3470419812 shutdown_connections 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 --2- 172.21.15.203:0/3470419812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c00745f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- 172.21.15.203:0/3470419812 >> 172.21.15.203:0/3470419812 conn(0x7f35c00fd2b0 msgr2=0x7f35c00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- 172.21.15.203:0/3470419812 shutdown_connections 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- 172.21.15.203:0/3470419812 wait complete. 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 librados: starting msgr at 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 librados: starting objecter 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 Processor -- start 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- start start 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 librados: setting wanted keys 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 librados: calling monclient init 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:00.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35c4c99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c010d510 con 0x7f35c0074210 2024-03-11T21:15:00.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.203:50856/0 (socket says 172.21.15.203:50856) 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.652+0000 7f35be59c700 1 -- 172.21.15.203:0/1475484477 learned_addr learned my addr 172.21.15.203:0/1475484477 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35be59c700 1 -- 172.21.15.203:0/1475484477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c019ed40 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35be59c700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35b4004f80 tx=0x7f35b4005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 10 client.?.objecter ms_handle_connect 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35b4026070 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c019efd0 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f35b4004370 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35b40038f0 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35c4c99700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c01989c0 con 0x7f35c0074210 2024-03-11T21:15:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35c4c99700 1 librados: init done 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c00758c0 con 0x7f35c0074210 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f35b4029070 con 0x7f35c0074210 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 0x7f35a803ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35b401e070 con 0x7f35c0074210 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 10 client.14186.objecter ms_dispatch 0x7f35c0001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 20 client.14186.objecter dump_active .. 0 homeless 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35bdd9b700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 0x7f35a803ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35af7fe700 10 client.14186.objecter ms_handle_connect 0x7f35a803a970 2024-03-11T21:15:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.656+0000 7f35bdd9b700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 0x7f35a803ce30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35b0006fd0 tx=0x7f35b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:00.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.672+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f35b4013070 con 0x7f35c0074210 2024-03-11T21:15:00.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.952+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35c0063410 con 0x7f35c0074210 2024-03-11T21:15:00.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.952+0000 7f35af7fe700 1 -- 172.21.15.203:0/1475484477 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f35b4030030 con 0x7f35c0074210 2024-03-11T21:15:00.940 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-11T21:15:00.941 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","modified":"2024-03-11T21:12:43.118202Z","created":"2024-03-11T21:12:43.118202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 10 librados: watch_flush enter 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 10 librados: watch_flush exit 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 20 client.14186.objecter shutdown clearing up homeless session... 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 10 client.14186.objecter successfully canceled tick 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 msgr2=0x7f35a803ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 0x7f35a803ce30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35b0006fd0 tx=0x7f35b0006e40 comp rx=0 tx=0).stop 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 msgr2=0x7f35c019e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:00.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35b4004f80 tx=0x7f35b4005060 comp rx=0 tx=0).stop 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 shutdown_connections 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f35a803a970 0x7f35a803ce30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 --2- 172.21.15.203:0/1475484477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c0074210 0x7f35c019e800 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 >> 172.21.15.203:0/1475484477 conn(0x7f35c00fd2b0 msgr2=0x7f35c00fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 shutdown_connections 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 -- 172.21.15.203:0/1475484477 wait complete. 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:00.956+0000 7f35c4c99700 1 librados: shutdown 2024-03-11T21:15:00.944 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-03-11T21:15:00.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:00 smithi143 bash[21512]: cluster 2024-03-11T21:14:59.422771+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:00.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:00 smithi143 bash[21512]: audit 2024-03-11T21:15:00.563833+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:00.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:00 smithi143 bash[21512]: audit 2024-03-11T21:15:00.570067+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:01.871 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:01 smithi143 bash[21512]: audit 2024-03-11T21:15:00.614023+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:01.871 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:01 smithi143 bash[21512]: audit 2024-03-11T21:15:00.615434+0000 mon.a (mon.0) 137 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:01.871 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:01 smithi143 bash[21512]: audit 2024-03-11T21:15:00.617139+0000 mon.a (mon.0) 138 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:01.871 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:01 smithi143 bash[21512]: cephadm 2024-03-11T21:15:00.619229+0000 mgr.y (mgr.14152) 49 : cephadm 1 Deploying daemon mon.c on smithi143 2024-03-11T21:15:01.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:01 smithi143 bash[21512]: audit 2024-03-11T21:15:00.953623+0000 mon.a (mon.0) 139 : audit 0 from='client.? 172.21.15.203:0/1475484477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.768+0000 7f8ac30bcc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.768+0000 7f8ac30bcc80 0 ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.768+0000 7f8ac30bcc80 0 pidfile_write: ignore empty --pid-file 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.772+0000 7f8ac30bcc80 0 load: jerasure load: lrc 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Git sha 0 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: DB SUMMARY 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: DB Session ID: ONO4V4T27TVRI6WR65BY 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: CURRENT file: CURRENT 2024-03-11T21:15:02.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.create_if_missing: 0 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.env: 0x5605fa70db20 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.info_log: 0x5605fcd97920 2024-03-11T21:15:02.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:15:02.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.statistics: (nil) 2024-03-11T21:15:02.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.use_fsync: 0 2024-03-11T21:15:02.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T21:15:02.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:15:02.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.db_log_dir: 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.wal_dir: 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:15:02.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.write_buffer_manager: 0x5605fcd6b400 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:15:02.035 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.unordered_write: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.row_cache: None 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.wal_filter: None 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.wal_compression: 0 2024-03-11T21:15:02.036 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T21:15:02.037 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_open_files: -1 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:15:02.038 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Compression algorithms supported: 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kZSTD supported: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:15:02.039 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T21:15:02.040 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:15:02.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:15:02.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.merge_operator: 2024-03-11T21:15:02.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_filter: None 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5605fcd97a20) 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cache_index_and_filter_blocks: 1 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: pin_top_level_index_and_filter: 1 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: index_type: 0 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: data_block_index_type: 0 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: index_shortening: 1 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:15:02.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: checksum: 4 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: no_block_cache: 0 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_cache: 0x5605fcd9b090 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_cache_name: BinnedLRUCache 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_cache_options: 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: capacity : 536870912 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: num_shard_bits : 4 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: strict_capacity_limit : 0 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: high_pri_pool_ratio: 0.000 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_cache_compressed: (nil) 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: persistent_cache: (nil) 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_size: 4096 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_size_deviation: 10 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_restart_interval: 16 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: index_block_restart_interval: 1 2024-03-11T21:15:02.043 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: metadata_block_size: 4096 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: partition_filters: 0 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: use_delta_encoding: 1 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: filter_policy: bloomfilter 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: whole_key_filtering: 1 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: verify_compression: 0 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: read_amp_bytes_per_bit: 0 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: format_version: 5 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: enable_index_compression: 1 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: block_align: 0 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: max_auto_readahead_size: 262144 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: prepopulate_block_cache: 0 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: initial_auto_readahead_size: 8192 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: num_file_reads_for_auto_readahead: 2 2024-03-11T21:15:02.044 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression: NoCompression 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.num_levels: 7 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:02.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:15:02.046 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:15:02.047 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:15:02.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.ttl: 2592000 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:15:02.049 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:15:02.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9f98c7a0-4350-4b61-825f-c52b9b7322fa 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191701778558, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191701779185, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191701, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9f98c7a0-4350-4b61-825f-c52b9b7322fa", "db_session_id": "ONO4V4T27TVRI6WR65BY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191701779308, "job": 1, "event": "recovery_finished"} 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.776+0000 7f8ac30bcc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5605fce92000 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 4 rocksdb: DB pointer 0x5605fce7c000 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ab1825700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ab1825700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: ** DB Stats ** 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:02.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: ** Compaction Stats [default] ** 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: ** Compaction Stats [default] ** 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:15:02.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Cumulative compaction: 0.00 GB write, 0.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Interval compaction: 0.00 GB write, 0.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Block cache BinnedLRUCache@0x5605fcd9b090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 0 using public_addr v2:172.21.15.203:0/0 -> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-03-11T21:15:02.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 0 starting mon.b rank -1 at public addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] at bind addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.780+0000 7f8ac30bcc80 1 mon.b@-1(???) e0 preinit fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).mds e1 new map 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: e1 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: legacy client fscid: -1 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: No filesystems configured 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab0022700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab9034700 4 rocksdb: (Original Log Time 2024/03/11-21:15:01.807184) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab9034700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab9034700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191701807235, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1119, "num_deletes": 25, "total_data_size": 2250590, "memory_usage": 2271312, "flush_reason": "Manual Compaction"} 2024-03-11T21:15:02.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab9034700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:25.415736+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:27.416222+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:28.735569+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:02.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:29.416658+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:29.632543+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi203 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:31.417090+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:33.417470+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:34.712546+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:34.720254+0000 mon.a (mon.0) 110 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:35.193069+0000 mon.a (mon.0) 111 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:34.713413+0000 mgr.y (mgr.14152) 27 : cephadm 1 Added host smithi203 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:35.417936+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:36.719431+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:37.418366+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:37.915643+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:39.418752+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:41.419122+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:42.699681+0000 mgr.y (mgr.14152) 32 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:15:02.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.198698+0000 mon.a (mon.0) 114 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.203477+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.209063+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.212903+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.214359+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.216164+0000 mon.a (mon.0) 119 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.217642+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:43.220880+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.835311+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.840911+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:43.846775+0000 mon.a (mon.0) 123 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:43.419474+0000 mgr.y (mgr.14152) 34 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:43.436359+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:43.600952+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:15:02.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:43.714909+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:45.419844+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:47.420218+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:49.420610+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:50.746903+0000 mon.a (mon.0) 124 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:51.224592+0000 mon.a (mon.0) 125 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:51.229997+0000 mon.a (mon.0) 126 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:51.420975+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:53.421468+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:55.421914+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.582072+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:55.588123+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b;count:3 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.596035+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.608003+0000 mon.a (mon.0) 128 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:02.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.611625+0000 mon.a (mon.0) 129 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.613198+0000 mon.a (mon.0) 130 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.623911+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.628473+0000 mon.a (mon.0) 132 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:14:55.630207+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:14:55.632345+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi203 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:57.422308+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cluster 2024-03-11T21:14:59.422771+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.563833+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.570067+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.614023+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.615434+0000 mon.a (mon.0) 137 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.617139+0000 mon.a (mon.0) 138 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: cephadm 2024-03-11T21:15:00.619229+0000 mgr.y (mgr.14152) 49 : cephadm 1 Deploying daemon mon.c on smithi143 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: audit 2024-03-11T21:15:00.953623+0000 mon.a (mon.0) 139 : audit 0 from='client.? 172.21.15.203:0/1475484477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:02.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:01 smithi203 bash[23654]: debug 2024-03-11T21:15:01.804+0000 7f8ab482b700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T21:15:02.703 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:15:02.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:15:04.365 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:15:04 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:04.365 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:04 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:04.653 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:04 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:04.889 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:15:04 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:04.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:04 smithi143 systemd[1]: Started Ceph mon.c for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.585+0000 7f94830eac80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.585+0000 7f94830eac80 0 ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.585+0000 7f94830eac80 0 pidfile_write: ignore empty --pid-file 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 0 load: jerasure load: lrc 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T21:15:05.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Git sha 0 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: DB SUMMARY 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: DB Session ID: VPW01JHWFZYRR3DHZSGB 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: CURRENT file: CURRENT 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.create_if_missing: 0 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:15:05.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.env: 0x55c816801b20 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.info_log: 0x55c817f0d920 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.statistics: (nil) 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.use_fsync: 0 2024-03-11T21:15:05.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:15:05.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.db_log_dir: 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.wal_dir: 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.write_buffer_manager: 0x55c817ee1400 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:15:05.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.unordered_write: 0 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:15:05.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.row_cache: None 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.wal_filter: None 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.wal_compression: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:15:05.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:15:05.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_open_files: -1 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Compression algorithms supported: 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kZSTD supported: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T21:15:05.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T21:15:05.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.merge_operator: 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_filter: None 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c817f0da20) 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cache_index_and_filter_blocks: 1 2024-03-11T21:15:05.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: pin_top_level_index_and_filter: 1 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: index_type: 0 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: data_block_index_type: 0 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: index_shortening: 1 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: checksum: 4 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: no_block_cache: 0 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_cache: 0x55c817f11090 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_cache_name: BinnedLRUCache 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_cache_options: 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: capacity : 536870912 2024-03-11T21:15:05.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: num_shard_bits : 4 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: strict_capacity_limit : 0 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: high_pri_pool_ratio: 0.000 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_cache_compressed: (nil) 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: persistent_cache: (nil) 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_size: 4096 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_size_deviation: 10 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_restart_interval: 16 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: index_block_restart_interval: 1 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: metadata_block_size: 4096 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: partition_filters: 0 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: use_delta_encoding: 1 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: filter_policy: bloomfilter 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: whole_key_filtering: 1 2024-03-11T21:15:05.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: verify_compression: 0 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: read_amp_bytes_per_bit: 0 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: format_version: 5 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: enable_index_compression: 1 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: block_align: 0 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: max_auto_readahead_size: 262144 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: prepopulate_block_cache: 0 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: initial_auto_readahead_size: 8192 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: num_file_reads_for_auto_readahead: 2 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression: NoCompression 2024-03-11T21:15:05.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.num_levels: 7 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:15:05.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:05.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:15:05.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:15:05.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T21:15:05.969 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.ttl: 2592000 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:15:05.970 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 11265c8b-7d26-416a-83a9-ae4528f7b2fe 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191705595655, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.589+0000 7f94830eac80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191705596169, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191705, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "11265c8b-7d26-416a-83a9-ae4528f7b2fe", "db_session_id": "VPW01JHWFZYRR3DHZSGB", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191705596270, "job": 1, "event": "recovery_finished"} 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:15:05.971 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c818008000 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 4 rocksdb: DB pointer 0x55c817ff2000 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f9471853700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f9471853700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: ** DB Stats ** 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: ** Compaction Stats [default] ** 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:05.972 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: ** Compaction Stats [default] ** 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:15:05.973 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Block cache BinnedLRUCache@0x55c817f11090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 0 using public_addrv [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.593+0000 7f94830eac80 0 starting mon.c rank -1 at public addrs [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] at bind addrs [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.597+0000 7f94830eac80 1 mon.c@-1(???) e0 preinit fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 0 mon.c@-1(synchronizing).mds e1 new map 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T21:15:05.974 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: e1 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: legacy client fscid: -1 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: No filesystems configured 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9470050700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9479062700 4 rocksdb: (Original Log Time 2024/03/11-21:15:05.615355) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9479062700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9479062700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710191705615433, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1123, "num_deletes": 25, "total_data_size": 2251463, "memory_usage": 2272064, "flush_reason": "Manual Compaction"} 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9479062700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:15:05.975 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.609+0000 7f9474859700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.613+0000 7f9474859700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.613+0000 7f9474859700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.613+0000 7f9474859700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.613+0000 7f9474859700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:25.415736+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:27.416222+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:28.735569+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:29.416658+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:29.632543+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi203 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:31.417090+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.976 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:33.417470+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:34.712546+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:34.720254+0000 mon.a (mon.0) 110 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:35.193069+0000 mon.a (mon.0) 111 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:34.713413+0000 mgr.y (mgr.14152) 27 : cephadm 1 Added host smithi203 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:35.417936+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:36.719431+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:37.418366+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:37.915643+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:39.418752+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:41.419122+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:42.699681+0000 mgr.y (mgr.14152) 32 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.198698+0000 mon.a (mon.0) 114 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.977 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.203477+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.209063+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.212903+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.214359+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.216164+0000 mon.a (mon.0) 119 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.217642+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:43.220880+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.835311+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.840911+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:43.846775+0000 mon.a (mon.0) 123 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:43.419474+0000 mgr.y (mgr.14152) 34 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:43.436359+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:43.600952+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:43.714909+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:15:05.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:45.419844+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:47.420218+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:49.420610+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:50.746903+0000 mon.a (mon.0) 124 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:51.224592+0000 mon.a (mon.0) 125 : audit 1 from='client.? 172.21.15.143:0/4063246455' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:51.229997+0000 mon.a (mon.0) 126 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:51.420975+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:53.421468+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:55.421914+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.582072+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:55.588123+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi143:172.21.15.143=a;smithi143:[v2:172.21.15.143:3301,v1:172.21.15.143:6790]=c;smithi203:172.21.15.203=b;count:3 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.596035+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.608003+0000 mon.a (mon.0) 128 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.611625+0000 mon.a (mon.0) 129 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.613198+0000 mon.a (mon.0) 130 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.623911+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.628473+0000 mon.a (mon.0) 132 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:14:55.630207+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:14:55.632345+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi203 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:57.422308+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cluster 2024-03-11T21:14:59.422771+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.563833+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.570067+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.614023+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.615434+0000 mon.a (mon.0) 137 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.617139+0000 mon.a (mon.0) 138 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: cephadm 2024-03-11T21:15:00.619229+0000 mgr.y (mgr.14152) 49 : cephadm 1 Deploying daemon mon.c on smithi143 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: audit 2024-03-11T21:15:00.953623+0000 mon.a (mon.0) 139 : audit 0 from='client.? 172.21.15.203:0/1475484477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:05.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:05 smithi143 bash[29284]: debug 2024-03-11T21:15:05.613+0000 7f9474859700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T21:15:06.406 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:15:07.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:01.423137+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:01.813786+0000 mon.a (mon.0) 140 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:01.818529+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:01.818882+0000 mon.a (mon.0) 142 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:01.841156+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:02.813425+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:03.423562+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:03.813963+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:03.816316+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:04.814158+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:05.424020+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:05.620621+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:05.814128+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:06.621421+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:06.814132+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.861950+0000 mon.a (mon.0) 151 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.867999+0000 mon.a (mon.0) 152 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.888828+0000 mon.a (mon.0) 153 : cluster 0 fsmap 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.888918+0000 mon.a (mon.0) 154 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.889208+0000 mon.a (mon.0) 155 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: cluster 2024-03-11T21:15:06.889456+0000 mon.a (mon.0) 156 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:06.896483+0000 mon.a (mon.0) 157 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:07.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:06 smithi143 bash[21512]: audit 2024-03-11T21:15:06.904529+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:01.423137+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:01.813786+0000 mon.a (mon.0) 140 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:01.818529+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:01.818882+0000 mon.a (mon.0) 142 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:01.841156+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:02.813425+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:03.423562+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:03.813963+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:03.816316+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:04.814158+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:05.424020+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:05.620621+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:05.814128+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:06.621421+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:06.814132+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.861950+0000 mon.a (mon.0) 151 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.867999+0000 mon.a (mon.0) 152 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.888828+0000 mon.a (mon.0) 153 : cluster 0 fsmap 2024-03-11T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.888918+0000 mon.a (mon.0) 154 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:07.246 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.889208+0000 mon.a (mon.0) 155 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T21:15:07.246 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: cluster 2024-03-11T21:15:06.889456+0000 mon.a (mon.0) 156 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:07.246 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:06.896483+0000 mon.a (mon.0) 157 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:07.246 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:06 smithi203 bash[23654]: audit 2024-03-11T21:15:06.904529+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:08.199 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:15:07 smithi143 bash[21768]: debug 2024-03-11T21:15:07.809+0000 7f99fed91700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-11T21:15:12.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:07.424353+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:07.628925+0000 mon.a (mon.0) 163 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:07.629287+0000 mon.a (mon.0) 164 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:07.630827+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:07.648877+0000 mon.a (mon.0) 165 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:07.649121+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:08.621875+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:09.424703+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:09.621913+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:09.624645+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:10.622343+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:11.425139+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:11.622250+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.622359+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.633392+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.674201+0000 mon.a (mon.0) 173 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.681199+0000 mon.a (mon.0) 174 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0],c=[v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.682208+0000 mon.a (mon.0) 175 : cluster 0 fsmap 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.682269+0000 mon.a (mon.0) 176 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.682544+0000 mon.a (mon.0) 177 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: cluster 2024-03-11T21:15:12.682784+0000 mon.a (mon.0) 178 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.688535+0000 mon.a (mon.0) 179 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:01.423137+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:01.813786+0000 mon.a (mon.0) 140 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:01.818529+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:01.818882+0000 mon.a (mon.0) 142 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:01.841156+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:02.813425+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:03.423562+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:03.813963+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:03.816316+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:04.814158+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:05.424020+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:05.620621+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:05.814128+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:06.621421+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:06.814132+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.861950+0000 mon.a (mon.0) 151 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.867999+0000 mon.a (mon.0) 152 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.888828+0000 mon.a (mon.0) 153 : cluster 0 fsmap 2024-03-11T21:15:12.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.888918+0000 mon.a (mon.0) 154 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.889208+0000 mon.a (mon.0) 155 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:06.889456+0000 mon.a (mon.0) 156 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:06.896483+0000 mon.a (mon.0) 157 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.693449+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.698862+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.704388+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[21512]: audit 2024-03-11T21:15:12.709364+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:06.904529+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:07.424353+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:07.628925+0000 mon.a (mon.0) 163 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:07.629287+0000 mon.a (mon.0) 164 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:07.630827+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:07.648877+0000 mon.a (mon.0) 165 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:07.649121+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:08.621875+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:09.424703+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:09.621913+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:09.624645+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:10.622343+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:11.425139+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:12.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:11.622250+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.622359+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.633392+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.674201+0000 mon.a (mon.0) 173 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.681199+0000 mon.a (mon.0) 174 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0],c=[v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.682208+0000 mon.a (mon.0) 175 : cluster 0 fsmap 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.682269+0000 mon.a (mon.0) 176 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.682544+0000 mon.a (mon.0) 177 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: cluster 2024-03-11T21:15:12.682784+0000 mon.a (mon.0) 178 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.688535+0000 mon.a (mon.0) 179 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.693449+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.698862+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.704388+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:12.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:12 smithi143 bash[29284]: audit 2024-03-11T21:15:12.709364+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:07.424353+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:07.628925+0000 mon.a (mon.0) 163 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:07.629287+0000 mon.a (mon.0) 164 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:07.630827+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:07.648877+0000 mon.a (mon.0) 165 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:07.649121+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:08.621875+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:09.424703+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:09.621913+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:09.624645+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:10.622343+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:11.425139+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:11.622250+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.622359+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.633392+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.674201+0000 mon.a (mon.0) 173 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.681199+0000 mon.a (mon.0) 174 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0],b=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0],c=[v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.682208+0000 mon.a (mon.0) 175 : cluster 0 fsmap 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.682269+0000 mon.a (mon.0) 176 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.682544+0000 mon.a (mon.0) 177 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: cluster 2024-03-11T21:15:12.682784+0000 mon.a (mon.0) 178 : cluster 1 overall HEALTH_OK 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.688535+0000 mon.a (mon.0) 179 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.693449+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.698862+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.704388+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:12 smithi203 bash[23654]: audit 2024-03-11T21:15:12.709364+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:13.150 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.159+0000 7f3437897700 1 -- 172.21.15.203:0/989918945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0035c0 msgr2=0x7f341c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.159+0000 7f3437897700 1 --2- 172.21.15.203:0/989918945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0035c0 0x7f341c005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f343006a0a0 tx=0x7f342000fe60 comp rx=0 tx=0).stop 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.159+0000 7f3437897700 1 -- 172.21.15.203:0/989918945 shutdown_connections 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.159+0000 7f3437897700 1 --2- 172.21.15.203:0/989918945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0035c0 0x7f341c005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.159+0000 7f3437897700 1 -- 172.21.15.203:0/989918945 >> 172.21.15.203:0/989918945 conn(0x7f34300fc2f0 msgr2=0x7f34300fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- 172.21.15.203:0/989918945 shutdown_connections 2024-03-11T21:15:13.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- 172.21.15.203:0/989918945 wait complete. 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 librados: starting msgr at 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 librados: starting objecter 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 Processor -- start 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- start start 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 librados: setting wanted keys 2024-03-11T21:15:13.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 librados: calling monclient init 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 0x7f3430196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3430108e60 con 0x7f343019ce80 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3430108ce0 con 0x7f341c0035c0 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3430108fe0 con 0x7f34301970e0 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 0x7f3430196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:60638/0 (socket says 172.21.15.203:60638) 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 -- 172.21.15.203:0/1825490912 learned_addr learned my addr 172.21.15.203:0/1825490912 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:15:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 msgr2=0x7f343019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 msgr2=0x7f343019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435e34700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435633700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435e34700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 msgr2=0x7f3430197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435e34700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 msgr2=0x7f3430197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435e34700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3435e34700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 msgr2=0x7f3430197590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 msgr2=0x7f343019c940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 -- 172.21.15.203:0/1825490912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34301a3810 con 0x7f343019ce80 2024-03-11T21:15:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3434e32700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 0x7f3430196ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f342c00b550 tx=0x7f342c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f34267fc700 10 client.?.objecter ms_handle_connect 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f34267fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c004230 con 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f342c0043d0 con 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34301a3a40 con 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011ad0 con 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34301a3f80 con 0x7f343019ce80 2024-03-11T21:15:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.163+0000 7f3437897700 1 librados: init done 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f342c011c70 con 0x7f343019ce80 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3430063410 con 0x7f343019ce80 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 0x7f341403d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f342c04bd50 con 0x7f343019ce80 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 10 client.14190.objecter ms_dispatch 0x7f3430001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:15:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:15:13.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 20 client.14190.objecter dump_active .. 0 homeless 2024-03-11T21:15:13.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f3435633700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 0x7f341403d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:13.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f3435633700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 0x7f341403d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f342000fa10 tx=0x7f3420005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:13.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.167+0000 7f34267fc700 10 client.14190.objecter ms_handle_connect 0x7f341403ae10 2024-03-11T21:15:13.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.171+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f342c0202a0 con 0x7f343019ce80 2024-03-11T21:15:13.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.479+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34301a4210 con 0x7f343019ce80 2024-03-11T21:15:13.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.479+0000 7f34267fc700 1 -- 172.21.15.203:0/1825490912 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f342c018350 con 0x7f343019ce80 2024-03-11T21:15:13.470 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-11T21:15:13.470 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","modified":"2024-03-11T21:15:07.622625Z","created":"2024-03-11T21:12:43.118202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3301","nonce":0},{"type":"v1","addr":"172.21.15.143:6790","nonce":0}]},"addr":"172.21.15.143:6790/0","public_addr":"172.21.15.143:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 10 librados: watch_flush enter 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 10 librados: watch_flush exit 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 20 client.14190.objecter shutdown clearing up homeless session... 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 10 client.14190.objecter successfully canceled tick 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 msgr2=0x7f341403d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 0x7f341403d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f342000fa10 tx=0x7f3420005de0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 msgr2=0x7f3430196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 0x7f3430196ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f342c00b550 tx=0x7f342c00b910 comp rx=0 tx=0).stop 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 shutdown_connections 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f341403ae10 0x7f341403d2d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34301970e0 0x7f3430197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f341c0035c0 0x7f343019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 --2- 172.21.15.203:0/1825490912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343019ce80 0x7f3430196ba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 >> 172.21.15.203:0/1825490912 conn(0x7f34300fc2f0 msgr2=0x7f3430103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 shutdown_connections 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 -- 172.21.15.203:0/1825490912 wait complete. 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:13.483+0000 7f3437897700 1 librados: shutdown 2024-03-11T21:15:13.474 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-03-11T21:15:14.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.333706+0000 mon.a (mon.0) 184 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:14.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.335694+0000 mon.a (mon.0) 185 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:14.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.338233+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:15:14.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.338689+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:15:14.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.483728+0000 mon.a (mon.0) 186 : audit 0 from='client.? 172.21.15.203:0/1825490912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.621598+0000 mon.a (mon.0) 187 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.672584+0000 mon.a (mon.0) 188 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.677389+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.681834+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.686244+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.690802+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.728329+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.732961+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.737036+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.740974+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.741969+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.743061+0000 mon.a (mon.0) 198 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:14.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:13 smithi203 bash[23654]: audit 2024-03-11T21:15:13.744101+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:14.135 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:15:14.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:15:14.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.333706+0000 mon.a (mon.0) 184 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.335694+0000 mon.a (mon.0) 185 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.338233+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.338689+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.483728+0000 mon.a (mon.0) 186 : audit 0 from='client.? 172.21.15.203:0/1825490912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.621598+0000 mon.a (mon.0) 187 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.672584+0000 mon.a (mon.0) 188 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.677389+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.681834+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.686244+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.690802+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.728329+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.732961+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.737036+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.740974+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.741969+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.743061+0000 mon.a (mon.0) 198 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[21512]: audit 2024-03-11T21:15:13.744101+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.333706+0000 mon.a (mon.0) 184 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.335694+0000 mon.a (mon.0) 185 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.338233+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.338689+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.483728+0000 mon.a (mon.0) 186 : audit 0 from='client.? 172.21.15.203:0/1825490912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.621598+0000 mon.a (mon.0) 187 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.672584+0000 mon.a (mon.0) 188 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.677389+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.681834+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.686244+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.690802+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.728329+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.732961+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.737036+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.740974+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.741969+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.743061+0000 mon.a (mon.0) 198 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:14.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:13 smithi143 bash[29284]: audit 2024-03-11T21:15:13.744101+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:15.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:14 smithi203 bash[23654]: cluster 2024-03-11T21:15:13.425464+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:15.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:14 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.502949+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:14 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.504907+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:14 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.741596+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:15:15.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:14 smithi203 bash[23654]: cephadm 2024-03-11T21:15:13.745310+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi143 2024-03-11T21:15:15.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[21512]: cluster 2024-03-11T21:15:13.425464+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:15.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.502949+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.504907+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.741596+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:15:15.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[21512]: cephadm 2024-03-11T21:15:13.745310+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi143 2024-03-11T21:15:15.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[29284]: cluster 2024-03-11T21:15:13.425464+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:15.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.502949+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.504907+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:15:15.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.741596+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:15:15.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:14 smithi143 bash[29284]: cephadm 2024-03-11T21:15:13.745310+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi143 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: cluster 2024-03-11T21:15:15.425856+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: audit 2024-03-11T21:15:15.894831+0000 mon.a (mon.0) 200 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: audit 2024-03-11T21:15:15.902567+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: cephadm 2024-03-11T21:15:15.914365+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: audit 2024-03-11T21:15:15.915036+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: audit 2024-03-11T21:15:15.916616+0000 mon.a (mon.0) 203 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:17.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: audit 2024-03-11T21:15:15.918115+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[21512]: cephadm 2024-03-11T21:15:15.920148+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi143 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: cluster 2024-03-11T21:15:15.425856+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: audit 2024-03-11T21:15:15.894831+0000 mon.a (mon.0) 200 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: audit 2024-03-11T21:15:15.902567+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: cephadm 2024-03-11T21:15:15.914365+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: audit 2024-03-11T21:15:15.915036+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: audit 2024-03-11T21:15:15.916616+0000 mon.a (mon.0) 203 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: audit 2024-03-11T21:15:15.918115+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:17.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:16 smithi143 bash[29284]: cephadm 2024-03-11T21:15:15.920148+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi143 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: cluster 2024-03-11T21:15:15.425856+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: audit 2024-03-11T21:15:15.894831+0000 mon.a (mon.0) 200 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: audit 2024-03-11T21:15:15.902567+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: cephadm 2024-03-11T21:15:15.914365+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: audit 2024-03-11T21:15:15.915036+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: audit 2024-03-11T21:15:15.916616+0000 mon.a (mon.0) 203 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: audit 2024-03-11T21:15:15.918115+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:17.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:16 smithi203 bash[23654]: cephadm 2024-03-11T21:15:15.920148+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi143 2024-03-11T21:15:18.839 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:18.915 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: cluster 2024-03-11T21:15:17.426312+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: audit 2024-03-11T21:15:17.657405+0000 mon.a (mon.0) 205 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: audit 2024-03-11T21:15:17.669833+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: cephadm 2024-03-11T21:15:17.687885+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: audit 2024-03-11T21:15:17.688553+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: audit 2024-03-11T21:15:17.690275+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: audit 2024-03-11T21:15:17.691862+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:18.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:18 smithi203 bash[23654]: cephadm 2024-03-11T21:15:17.693815+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi203 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: cluster 2024-03-11T21:15:17.426312+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: audit 2024-03-11T21:15:17.657405+0000 mon.a (mon.0) 205 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: audit 2024-03-11T21:15:17.669833+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: cephadm 2024-03-11T21:15:17.687885+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: audit 2024-03-11T21:15:17.688553+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: audit 2024-03-11T21:15:17.690275+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: audit 2024-03-11T21:15:17.691862+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:18.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[21512]: cephadm 2024-03-11T21:15:17.693815+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi203 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: cluster 2024-03-11T21:15:17.426312+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: audit 2024-03-11T21:15:17.657405+0000 mon.a (mon.0) 205 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: audit 2024-03-11T21:15:17.669833+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: cephadm 2024-03-11T21:15:17.687885+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: audit 2024-03-11T21:15:17.688553+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: audit 2024-03-11T21:15:17.690275+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: audit 2024-03-11T21:15:17.691862+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:18.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:18 smithi143 bash[29284]: cephadm 2024-03-11T21:15:17.693815+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi203 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: cluster 2024-03-11T21:15:19.426831+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.428277+0000 mon.a (mon.0) 210 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.435581+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.462125+0000 mon.a (mon.0) 212 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.465914+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.467678+0000 mon.a (mon.0) 214 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[21512]: audit 2024-03-11T21:15:19.476977+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: cluster 2024-03-11T21:15:19.426831+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.428277+0000 mon.a (mon.0) 210 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.435581+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.462125+0000 mon.a (mon.0) 212 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.465914+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.467678+0000 mon.a (mon.0) 214 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:20.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:20 smithi143 bash[29284]: audit 2024-03-11T21:15:19.476977+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: cluster 2024-03-11T21:15:19.426831+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.428277+0000 mon.a (mon.0) 210 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.435581+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.462125+0000 mon.a (mon.0) 212 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.465914+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.467678+0000 mon.a (mon.0) 214 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:20.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:20 smithi203 bash[23654]: audit 2024-03-11T21:15:19.476977+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- 172.21.15.143:0/902953471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8109360 msgr2=0x7f07c8110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/902953471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8109360 0x7f07c8110f30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f07c000b0d0 tx=0x7f07c000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- 172.21.15.143:0/902953471 shutdown_connections 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/902953471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8109360 0x7f07c8110f30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/902953471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8104280 0x7f07c8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/902953471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c8108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- 172.21.15.143:0/902953471 >> 172.21.15.143:0/902953471 conn(0x7f07c80ffb10 msgr2=0x7f07c8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- 172.21.15.143:0/902953471 shutdown_connections 2024-03-11T21:15:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- 172.21.15.143:0/902953471 wait complete. 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 librados: starting msgr at 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 librados: starting objecter 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 Processor -- start 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- start start 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 librados: setting wanted keys 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 librados: calling monclient init 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 0x7f07c8199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:21.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 0x7f07c819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c81138b0 con 0x7f07c8104280 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8113730 con 0x7f07c8109360 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cf5aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8113a30 con 0x7f07c8104c30 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07ccb45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07cd346700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 0x7f07c8199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07ccb45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c819a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49330/0 (socket says 172.21.15.143:49330) 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.232+0000 7f07ccb45700 1 -- 172.21.15.143:0/2346858904 learned_addr learned my addr 172.21.15.143:0/2346858904 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:15:21.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cdb47700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 0x7f07c819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 -- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 msgr2=0x7f07c819a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c819a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 -- 172.21.15.143:0/2346858904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 msgr2=0x7f07c819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 0x7f07c819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 -- 172.21.15.143:0/2346858904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c819ee00 con 0x7f07c8104280 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cdb47700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 0x7f07c819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cd346700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 0x7f07c8199df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c970 tx=0x7f07bc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 10 client.?.objecter ms_handle_connect 0x7f07c8104280 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc014070 con 0x7f07c8104280 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f07bc004440 con 0x7f07c8104280 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c819f0f0 con 0x7f07c8104280 2024-03-11T21:15:21.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc0039f0 con 0x7f07c8104280 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cf5aa700 10 client.14196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c81a6970 con 0x7f07c8104280 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cf5aa700 1 librados: init done 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f07bc017070 con 0x7f07c8104280 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c804faa0 con 0x7f07c8104280 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 0x7f07b403d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07bc04bb20 con 0x7f07c8104280 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 10 client.14196.objecter ms_dispatch 0x7f07c8001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 3 client.14196.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 3 client.14196.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 20 client.14196.objecter dump_active .. 0 homeless 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ccb45700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 0x7f07b403d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ccb45700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 0x7f07b403d040 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07c8101ce0 tx=0x7f07c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.236+0000 7f07ba7fc700 10 client.14196.objecter ms_handle_connect 0x7f07b403ab80 2024-03-11T21:15:21.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.240+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f07bc01e030 con 0x7f07c8104280 2024-03-11T21:15:21.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f07c819b150 con 0x7f07c8104280 2024-03-11T21:15:21.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07ba7fc700 1 -- 172.21.15.143:0/2346858904 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f07bc00d340 con 0x7f07c8104280 2024-03-11T21:15:21.439 INFO:teuthology.orchestra.run.smithi143.stdout:# minimal ceph.conf for eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:21.439 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-03-11T21:15:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:15:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: mon_host = [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 10 librados: watch_flush enter 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 10 librados: watch_flush exit 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 20 client.14196.objecter shutdown clearing up homeless session... 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 10 client.14196.objecter successfully canceled tick 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 msgr2=0x7f07b403d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 0x7f07b403d040 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07c8101ce0 tx=0x7f07c4009260 comp rx=0 tx=0).stop 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 msgr2=0x7f07c8199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 0x7f07c8199df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c970 tx=0x7f07bc00cc80 comp rx=0 tx=0).stop 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 shutdown_connections 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f07b403ab80 0x7f07b403d040 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c8104280 0x7f07c8199df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07c8109360 0x7f07c819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 --2- 172.21.15.143:0/2346858904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07c8104c30 0x7f07c819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 >> 172.21.15.143:0/2346858904 conn(0x7f07c80ffb10 msgr2=0x7f07c8107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 shutdown_connections 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 -- 172.21.15.143:0/2346858904 wait complete. 2024-03-11T21:15:21.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:21.448+0000 7f07cf5aa700 1 librados: shutdown 2024-03-11T21:15:22.068 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:15:22.068 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:15:22.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:15:22.123 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:15:22.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:15:22.182 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:15:22.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:15:22.196 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:15:22.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:15:22.251 INFO:tasks.cephadm:Adding mgr.y on smithi143 2024-03-11T21:15:22.251 INFO:tasks.cephadm:Adding mgr.x on smithi203 2024-03-11T21:15:22.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi143=y;smithi203=x' 2024-03-11T21:15:22.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:22 smithi203 bash[23654]: cluster 2024-03-11T21:15:21.427253+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:22.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:22 smithi203 bash[23654]: audit 2024-03-11T21:15:21.452366+0000 mon.a (mon.0) 216 : audit 0 from='client.? 172.21.15.143:0/2346858904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:22.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:22 smithi143 bash[21512]: cluster 2024-03-11T21:15:21.427253+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:22.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:22 smithi143 bash[21512]: audit 2024-03-11T21:15:21.452366+0000 mon.a (mon.0) 216 : audit 0 from='client.? 172.21.15.143:0/2346858904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:22.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:22 smithi143 bash[29284]: cluster 2024-03-11T21:15:21.427253+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:22.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:22 smithi143 bash[29284]: audit 2024-03-11T21:15:21.452366+0000 mon.a (mon.0) 216 : audit 0 from='client.? 172.21.15.143:0/2346858904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:24.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:24 smithi143 bash[21512]: cluster 2024-03-11T21:15:23.427612+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:24.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:24 smithi143 bash[29284]: cluster 2024-03-11T21:15:23.427612+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:24.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:24 smithi203 bash[23654]: cluster 2024-03-11T21:15:23.427612+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:25.987 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:15:26.684 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:26 smithi203 bash[23654]: cluster 2024-03-11T21:15:25.428034+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:26.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:26 smithi143 bash[21512]: cluster 2024-03-11T21:15:25.428034+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:26.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:26 smithi143 bash[29284]: cluster 2024-03-11T21:15:25.428034+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:28.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/260660606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401052d0 msgr2=0x7f79401076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:28.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- 172.21.15.203:0/260660606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401052d0 0x7f79401076c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7928009990 tx=0x7f7928009ca0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/260660606 shutdown_connections 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- 172.21.15.203:0/260660606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79401029a0 0x7f7940104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- 172.21.15.203:0/260660606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7940107c00 0x7f7940109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- 172.21.15.203:0/260660606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401052d0 0x7f79401076c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/260660606 >> 172.21.15.203:0/260660606 conn(0x7f79400fc330 msgr2=0x7f79400fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/260660606 shutdown_connections 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/260660606 wait complete. 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 librados: starting msgr at 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 librados: starting objecter 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 Processor -- start 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- start start 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 librados: setting wanted keys 2024-03-11T21:15:28.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 librados: calling monclient init 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79401052d0 0x7f7940100950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7940107c00 0x7f7940100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794010cf40 con 0x7f7940107c00 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794010cdc0 con 0x7f79401029a0 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794010d0c0 con 0x7f79401052d0 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:43184/0 (socket says 172.21.15.203:43184) 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 -- 172.21.15.203:0/863623443 learned_addr learned my addr 172.21.15.203:0/863623443 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793effd700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79401052d0 0x7f7940100950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 -- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79401052d0 msgr2=0x7f7940100950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79401052d0 0x7f7940100950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 -- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7940107c00 msgr2=0x7f7940100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7940107c00 0x7f7940100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 -- 172.21.15.203:0/863623443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940101740 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793f7fe700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7930007f10 tx=0x7f793000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 10 client.?.objecter ms_handle_connect 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930003a00 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7930003ba0 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79401a1820 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930015400 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 10 client.24103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79401a1d30 con 0x7f79401029a0 2024-03-11T21:15:28.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 librados: init done 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794004faa0 con 0x7f79401029a0 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7930004000 con 0x7f79401029a0 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 0x7f792c03d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f793004bf30 con 0x7f79401029a0 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 10 client.24103.objecter ms_dispatch 0x7f7940001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 3 client.24103.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 3 client.24103.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793cff9700 20 client.24103.objecter dump_active .. 0 homeless 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.063+0000 7f793effd700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 0x7f792c03d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.067+0000 7f793cff9700 10 client.24103.objecter ms_handle_connect 0x7f792c03ab80 2024-03-11T21:15:28.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.067+0000 7f793effd700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 0x7f792c03d040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f792800b5c0 tx=0x7f7928003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:28.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.071+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7930010070 con 0x7f79401029a0 2024-03-11T21:15:28.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.275+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi143=y;smithi203=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f7940062220 con 0x7f792c03ab80 2024-03-11T21:15:28.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.291+0000 7f793cff9700 1 -- 172.21.15.203:0/863623443 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7940062220 con 0x7f792c03ab80 2024-03-11T21:15:28.282 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-03-11T21:15:28.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 10 librados: watch_flush enter 2024-03-11T21:15:28.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 10 librados: watch_flush exit 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 20 client.24103.objecter shutdown clearing up homeless session... 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 10 client.24103.objecter successfully canceled tick 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 msgr2=0x7f792c03d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 0x7f792c03d040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f792800b5c0 tx=0x7f7928003680 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 msgr2=0x7f7940102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7930007f10 tx=0x7f793000d3b0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 shutdown_connections 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f792c03ab80 0x7f792c03d040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7940107c00 0x7f7940100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79401029a0 0x7f7940102300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 --2- 172.21.15.203:0/863623443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79401052d0 0x7f7940100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 >> 172.21.15.203:0/863623443 conn(0x7f79400fc330 msgr2=0x7f7940103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:28.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 shutdown_connections 2024-03-11T21:15:28.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 -- 172.21.15.203:0/863623443 wait complete. 2024-03-11T21:15:28.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:15:28.295+0000 7f794619c700 1 librados: shutdown 2024-03-11T21:15:28.819 DEBUG:teuthology.orchestra.run.smithi203:mgr.x> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:15:28.821 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:15:28.822 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:15:28.822 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:15:28.828 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:15:28.829 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-11 21:06:09.504289193 +0000 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-11 21:06:09.236291588 +0000 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-11 21:06:09.236291588 +0000 2024-03-11T21:15:28.875 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-03-11T21:15:28.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:15:28.933 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-11T21:15:28.934 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-11T21:15:28.934 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000357489 s, 1.4 MB/s 2024-03-11T21:15:28.935 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:15:28.983 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-03-11T21:15:29.030 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-11 21:06:10.016284621 +0000 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-11 21:06:09.744287050 +0000 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-11 21:06:09.744287050 +0000 2024-03-11T21:15:29.031 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-03-11T21:15:29.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:15:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-11T21:15:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-11T21:15:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000419774 s, 1.2 MB/s 2024-03-11T21:15:29.086 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:15:29.136 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-11 21:06:10.484280439 +0000 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-11 21:06:10.228282727 +0000 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-11 21:06:10.228282727 +0000 2024-03-11T21:15:29.182 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-03-11T21:15:29.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:15:29.240 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-11T21:15:29.241 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-11T21:15:29.241 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000417673 s, 1.2 MB/s 2024-03-11T21:15:29.242 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:15:29.292 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-11 21:06:10.948276296 +0000 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-11 21:06:10.696278547 +0000 2024-03-11T21:15:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-11 21:06:10.696278547 +0000 2024-03-11T21:15:29.339 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-03-11T21:15:29.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:15:29.393 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-11T21:15:29.393 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-11T21:15:29.393 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000422673 s, 1.2 MB/s 2024-03-11T21:15:29.395 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:15:29.444 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:15:29.444 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:15:29.451 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:15:29.451 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-11T21:15:29.499 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-11 21:06:09.353323041 +0000 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-11 21:06:09.085325430 +0000 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-11 21:06:09.085325430 +0000 2024-03-11T21:15:29.500 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-03-11T21:15:29.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:15:29.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: cluster 2024-03-11T21:15:27.428399+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:29.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.282929+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi143=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:29.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: cephadm 2024-03-11T21:15:28.286640+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi143=y;smithi203=x;count:2 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.294993+0000 mon.a (mon.0) 217 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.337641+0000 mon.a (mon.0) 218 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.341358+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.343139+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.353741+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.357151+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.359550+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.366392+0000 mon.a (mon.0) 224 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:29.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:29 smithi203 bash[23654]: audit 2024-03-11T21:15:28.368284+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.543 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-11T21:15:29.543 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-11T21:15:29.543 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000378838 s, 1.4 MB/s 2024-03-11T21:15:29.544 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:15:29.593 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-11 21:06:09.849318616 +0000 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-11 21:06:09.589320935 +0000 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-11 21:06:09.589320935 +0000 2024-03-11T21:15:29.639 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-03-11T21:15:29.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:15:29.694 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-11T21:15:29.694 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-11T21:15:29.694 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000369713 s, 1.4 MB/s 2024-03-11T21:15:29.695 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:15:29.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: cluster 2024-03-11T21:15:27.428399+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.282929+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi143=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: cephadm 2024-03-11T21:15:28.286640+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi143=y;smithi203=x;count:2 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.294993+0000 mon.a (mon.0) 217 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.337641+0000 mon.a (mon.0) 218 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.341358+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.343139+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.353741+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.357151+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.359550+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.366392+0000 mon.a (mon.0) 224 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[21512]: audit 2024-03-11T21:15:28.368284+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: cluster 2024-03-11T21:15:27.428399+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:29.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.282929+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi143=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: cephadm 2024-03-11T21:15:28.286640+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi143=y;smithi203=x;count:2 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.294993+0000 mon.a (mon.0) 217 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.337641+0000 mon.a (mon.0) 218 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.341358+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.343139+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.353741+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.357151+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.359550+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.366392+0000 mon.a (mon.0) 224 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:29.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:29 smithi143 bash[29284]: audit 2024-03-11T21:15:28.368284+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:29.744 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-11 21:06:10.097316404 +0000 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-11 21:06:10.093316440 +0000 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-11 21:06:10.093316440 +0000 2024-03-11T21:15:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-03-11T21:15:29.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:15:29.845 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-11T21:15:29.845 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-11T21:15:29.845 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000419402 s, 1.2 MB/s 2024-03-11T21:15:29.847 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:15:29.899 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-11 21:06:10.837309801 +0000 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-11 21:06:10.605311872 +0000 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-11 21:06:10.605311872 +0000 2024-03-11T21:15:29.951 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-03-11T21:15:29.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:15:30.005 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-11T21:15:30.005 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-11T21:15:30.005 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000376971 s, 1.4 MB/s 2024-03-11T21:15:30.006 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:15:30.061 INFO:tasks.cephadm:Deploying osd.0 on smithi143 with /dev/vg_nvme/lv_4... 2024-03-11T21:15:30.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T21:15:30.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:30 smithi143 bash[21512]: cephadm 2024-03-11T21:15:28.370242+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi203 2024-03-11T21:15:30.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:30 smithi143 bash[29284]: cephadm 2024-03-11T21:15:28.370242+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi203 2024-03-11T21:15:30.458 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:30 smithi203 bash[23654]: cephadm 2024-03-11T21:15:28.370242+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi203 2024-03-11T21:15:30.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:30.999 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:31.000 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:31.000 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:31.000 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:31.000 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:15:31.000 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:30 smithi203 systemd[1]: Started Ceph mgr.x for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:15:31.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: cluster 2024-03-11T21:15:29.428874+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:31.665 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: audit 2024-03-11T21:15:31.018798+0000 mon.a (mon.0) 226 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.665 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: audit 2024-03-11T21:15:31.026231+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.665 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: audit 2024-03-11T21:15:31.047104+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.665 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: audit 2024-03-11T21:15:31.054490+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.665 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[23654]: audit 2024-03-11T21:15:31.123192+0000 mon.a (mon.0) 230 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: cluster 2024-03-11T21:15:29.428874+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: audit 2024-03-11T21:15:31.018798+0000 mon.a (mon.0) 226 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: audit 2024-03-11T21:15:31.026231+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: audit 2024-03-11T21:15:31.047104+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: audit 2024-03-11T21:15:31.054490+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[21512]: audit 2024-03-11T21:15:31.123192+0000 mon.a (mon.0) 230 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: cluster 2024-03-11T21:15:29.428874+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: audit 2024-03-11T21:15:31.018798+0000 mon.a (mon.0) 226 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: audit 2024-03-11T21:15:31.026231+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: audit 2024-03-11T21:15:31.047104+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: audit 2024-03-11T21:15:31.054490+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:31.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:31 smithi143 bash[29284]: audit 2024-03-11T21:15:31.123192+0000 mon.a (mon.0) 230 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.916 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[24601]: debug 2024-03-11T21:15:31.783+0000 7f35e083d700 1 -- 172.21.15.203:0/1909116710 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6868885a0 con 0x55f68698e400 2024-03-11T21:15:33.574 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:31 smithi203 bash[24601]: debug 2024-03-11T21:15:31.927+0000 7f35eaa3f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:15:33.574 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:33 smithi203 bash[24601]: debug 2024-03-11T21:15:33.487+0000 7f35eaa3f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:15:33.575 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:33 smithi203 bash[23654]: cluster 2024-03-11T21:15:31.429286+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:33.575 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:33 smithi203 bash[23654]: audit 2024-03-11T21:15:32.696043+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:33.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:33 smithi143 bash[21512]: cluster 2024-03-11T21:15:31.429286+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:33.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:33 smithi143 bash[21512]: audit 2024-03-11T21:15:32.696043+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:33.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:33 smithi143 bash[29284]: cluster 2024-03-11T21:15:31.429286+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:33.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:33 smithi143 bash[29284]: audit 2024-03-11T21:15:32.696043+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:34.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:33 smithi203 bash[24601]: debug 2024-03-11T21:15:33.583+0000 7f35eaa3f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:15:34.030 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:33 smithi203 bash[24601]: debug 2024-03-11T21:15:33.831+0000 7f35eaa3f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:15:34.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:34 smithi143 bash[21512]: cluster 2024-03-11T21:15:33.429684+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:34.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:34 smithi143 bash[29284]: cluster 2024-03-11T21:15:33.429684+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:34.717 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:34 smithi203 bash[24601]: debug 2024-03-11T21:15:34.619+0000 7f35eaa3f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:34.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:34 smithi203 bash[23654]: cluster 2024-03-11T21:15:33.429684+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:34.784 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:35.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:34 smithi203 bash[24601]: debug 2024-03-11T21:15:34.727+0000 7f35eaa3f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:15:35.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:34 smithi203 bash[24601]: debug 2024-03-11T21:15:34.827+0000 7f35eaa3f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:15:35.529 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:35 smithi203 bash[24601]: debug 2024-03-11T21:15:35.179+0000 7f35eaa3f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:15:35.529 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:35 smithi203 bash[24601]: debug 2024-03-11T21:15:35.291+0000 7f35eaa3f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:15:35.530 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:35 smithi203 bash[24601]: debug 2024-03-11T21:15:35.391+0000 7f35eaa3f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:15:36.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:35 smithi203 bash[24601]: debug 2024-03-11T21:15:35.743+0000 7f35eaa3f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:15:36.030 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:35 smithi203 bash[24601]: debug 2024-03-11T21:15:35.979+0000 7f35eaa3f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:15:36.529 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:36 smithi203 bash[24601]: debug 2024-03-11T21:15:36.246+0000 7f35eaa3f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:15:36.530 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:36 smithi203 bash[24601]: debug 2024-03-11T21:15:36.362+0000 7f35eaa3f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:15:37.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:36 smithi203 bash[24601]: debug 2024-03-11T21:15:36.626+0000 7f35eaa3f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:15:37.030 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:36 smithi203 bash[24601]: debug 2024-03-11T21:15:36.870+0000 7f35eaa3f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:37.529 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[24601]: debug 2024-03-11T21:15:37.274+0000 7f35eaa3f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[24601]: debug 2024-03-11T21:15:37.390+0000 7f35eaa3f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: cluster 2024-03-11T21:15:35.430109+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.314466+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.322594+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.324424+0000 mon.a (mon.0) 234 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.326294+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.335399+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.405490+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.406593+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:37.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[23654]: audit 2024-03-11T21:15:36.407343+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:37.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: cluster 2024-03-11T21:15:35.430109+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:37.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.314466+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.322594+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.324424+0000 mon.a (mon.0) 234 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.326294+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.335399+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.405490+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.406593+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[21512]: audit 2024-03-11T21:15:36.407343+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: cluster 2024-03-11T21:15:35.430109+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.314466+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.322594+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.324424+0000 mon.a (mon.0) 234 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:37.581 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.326294+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:37.582 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.335399+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:37.582 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.405490+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:15:37.582 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.406593+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:15:37.582 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:37 smithi143 bash[29284]: audit 2024-03-11T21:15:36.407343+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:38.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:37 smithi203 bash[24601]: debug 2024-03-11T21:15:37.630+0000 7f35eaa3f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:38.046 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T21:15:38.046 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T21:15:38.047 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-11T21:15:38.047 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-11T21:15:38.047 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148752 s, 705 MB/s 2024-03-11T21:15:38.047 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-11T21:15:38.618 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:38 smithi203 bash[23654]: cephadm 2024-03-11T21:15:36.405095+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:15:38.619 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:38 smithi203 bash[23654]: cephadm 2024-03-11T21:15:36.408278+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi143 2024-03-11T21:15:38.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:38 smithi143 bash[21512]: cephadm 2024-03-11T21:15:36.405095+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:15:38.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:38 smithi143 bash[21512]: cephadm 2024-03-11T21:15:36.408278+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi143 2024-03-11T21:15:38.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:38 smithi143 bash[29284]: cephadm 2024-03-11T21:15:36.405095+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:15:38.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:38 smithi143 bash[29284]: cephadm 2024-03-11T21:15:36.408278+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi143 2024-03-11T21:15:38.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_4 2024-03-11T21:15:39.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:38 smithi203 bash[24601]: debug 2024-03-11T21:15:38.626+0000 7f35eaa3f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:15:39.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:38 smithi203 bash[24601]: debug 2024-03-11T21:15:38.742+0000 7f35eaa3f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:15:39.310 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[24601]: debug 2024-03-11T21:15:39.038+0000 7f35eaa3f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:15:39.311 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[24601]: debug 2024-03-11T21:15:39.150+0000 7f35eaa3f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: cluster 2024-03-11T21:15:37.430412+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:38.519232+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:38.526611+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:38.577308+0000 mon.a (mon.0) 242 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:39.204670+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:39.206385+0000 mon.a (mon.0) 244 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:39.590 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[23654]: audit 2024-03-11T21:15:39.214580+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: cluster 2024-03-11T21:15:37.430412+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:39.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:38.519232+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:38.526611+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:38.577308+0000 mon.a (mon.0) 242 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:39.204670+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:39.206385+0000 mon.a (mon.0) 244 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[21512]: audit 2024-03-11T21:15:39.214580+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: cluster 2024-03-11T21:15:37.430412+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:38.519232+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:38.526611+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:38.577308+0000 mon.a (mon.0) 242 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:39.204670+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:39.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:39.206385+0000 mon.a (mon.0) 244 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:39.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:39 smithi143 bash[29284]: audit 2024-03-11T21:15:39.214580+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:15:40.029 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[24601]: debug 2024-03-11T21:15:39.598+0000 7f35eaa3f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:15:40.030 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:15:39 smithi203 bash[24601]: debug 2024-03-11T21:15:39.718+0000 7f35eaa3f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:15:40.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[21512]: cluster 2024-03-11T21:15:39.744714+0000 mon.a (mon.0) 246 : cluster 0 Standby manager daemon x started 2024-03-11T21:15:40.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[21512]: audit 2024-03-11T21:15:39.746926+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:15:40.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[21512]: audit 2024-03-11T21:15:39.747515+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:15:40.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[21512]: audit 2024-03-11T21:15:39.748784+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[21512]: audit 2024-03-11T21:15:39.749367+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[29284]: cluster 2024-03-11T21:15:39.744714+0000 mon.a (mon.0) 246 : cluster 0 Standby manager daemon x started 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[29284]: audit 2024-03-11T21:15:39.746926+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[29284]: audit 2024-03-11T21:15:39.747515+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[29284]: audit 2024-03-11T21:15:39.748784+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:15:40.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:40 smithi143 bash[29284]: audit 2024-03-11T21:15:39.749367+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:15:40.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:40 smithi203 bash[23654]: cluster 2024-03-11T21:15:39.744714+0000 mon.a (mon.0) 246 : cluster 0 Standby manager daemon x started 2024-03-11T21:15:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:40 smithi203 bash[23654]: audit 2024-03-11T21:15:39.746926+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:15:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:40 smithi203 bash[23654]: audit 2024-03-11T21:15:39.747515+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:15:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:40 smithi203 bash[23654]: audit 2024-03-11T21:15:39.748784+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:15:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:40 smithi203 bash[23654]: audit 2024-03-11T21:15:39.749367+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.203:0/502167320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:15:41.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[21512]: cluster 2024-03-11T21:15:39.430835+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:41.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[21512]: cluster 2024-03-11T21:15:40.344631+0000 mon.a (mon.0) 247 : cluster 0 mgrmap e14: y(active, since 106s), standbys: x 2024-03-11T21:15:41.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[21512]: audit 2024-03-11T21:15:40.344924+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:15:41.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[29284]: cluster 2024-03-11T21:15:39.430835+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:41.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[29284]: cluster 2024-03-11T21:15:40.344631+0000 mon.a (mon.0) 247 : cluster 0 mgrmap e14: y(active, since 106s), standbys: x 2024-03-11T21:15:41.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:41 smithi143 bash[29284]: audit 2024-03-11T21:15:40.344924+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:15:41.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:41 smithi203 bash[23654]: cluster 2024-03-11T21:15:39.430835+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:41.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:41 smithi203 bash[23654]: cluster 2024-03-11T21:15:40.344631+0000 mon.a (mon.0) 247 : cluster 0 mgrmap e14: y(active, since 106s), standbys: x 2024-03-11T21:15:41.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:41 smithi203 bash[23654]: audit 2024-03-11T21:15:40.344924+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:15:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:43.636 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:43 smithi143 bash[21512]: cluster 2024-03-11T21:15:41.431336+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:43.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:43 smithi143 bash[29284]: cluster 2024-03-11T21:15:41.431336+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:43.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:43 smithi203 bash[23654]: cluster 2024-03-11T21:15:41.431336+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:45.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:45 smithi143 bash[21512]: cluster 2024-03-11T21:15:43.431778+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:45.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:45 smithi143 bash[29284]: cluster 2024-03-11T21:15:43.431778+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:45.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:45 smithi203 bash[23654]: cluster 2024-03-11T21:15:43.431778+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 -- 172.21.15.143:0/530494015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 msgr2=0x7f3e94104d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 --2- 172.21.15.143:0/530494015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94104d50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e840098f0 tx=0x7f3e84009c00 comp rx=0 tx=0).stop 2024-03-11T21:15:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 -- 172.21.15.143:0/530494015 shutdown_connections 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 --2- 172.21.15.143:0/530494015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94104d50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 --2- 172.21.15.143:0/530494015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e94109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 --2- 172.21.15.143:0/530494015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 -- 172.21.15.143:0/530494015 >> 172.21.15.143:0/530494015 conn(0x7f3e940fc310 msgr2=0x7f3e940fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 -- 172.21.15.143:0/530494015 shutdown_connections 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 -- 172.21.15.143:0/530494015 wait complete. 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 librados: starting msgr at 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 librados: starting objecter 2024-03-11T21:15:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.040+0000 7f3e9b138700 1 Processor -- start 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- start start 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 librados: setting wanted keys 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 librados: calling monclient init 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e9419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9410cf50 con 0x7f3e94102960 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9410cdd0 con 0x7f3e94107bc0 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9410d0d0 con 0x7f3e94105290 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41966/0 (socket says 172.21.15.143:41966) 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 -- 172.21.15.143:0/2896064622 learned_addr learned my addr 172.21.15.143:0/2896064622 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e93fff700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e996d5700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e9419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 -- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 msgr2=0x7f3e94197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 -- 172.21.15.143:0/2896064622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 msgr2=0x7f3e9419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e9419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 -- 172.21.15.143:0/2896064622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9419f3b0 con 0x7f3e94102960 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e93fff700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:15:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e996d5700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e9419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e98ed4700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e840094f0 tx=0x7f3e8400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 10 client.?.objecter ms_handle_connect 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400ba40 con 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3e840054c0 con 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9419f640 con 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84004de0 con 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 10 client.14208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9419faf0 con 0x7f3e94102960 2024-03-11T21:15:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 librados: init done 2024-03-11T21:15:46.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9404faa0 con 0x7f3e94102960 2024-03-11T21:15:46.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f3e84005040 con 0x7f3e94102960 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 0x7f3e7c071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e8408d9a0 con 0x7f3e94102960 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 10 client.14208.objecter ms_dispatch 0x7f3e94001550 osd_map(4..4 src has 1..4) v4 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 3 client.14208.objecter handle_osd_map got epochs [4,4] > 0 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 3 client.14208.objecter handle_osd_map decoding full epoch 4 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e93fff700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 0x7f3e7c071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 20 client.14208.objecter dump_active .. 0 homeless 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e91ffb700 10 client.14208.objecter ms_handle_connect 0x7f3e7c06f690 2024-03-11T21:15:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.044+0000 7f3e93fff700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 0x7f3e7c071b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009750 tx=0x7f3e88009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:15:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.052+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3e84013030 con 0x7f3e94102960 2024-03-11T21:15:46.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:46.268+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e9402d9a0 con 0x7f3e7c06f690 2024-03-11T21:15:46.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[21512]: audit 2024-03-11T21:15:46.279404+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:15:46.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[21512]: audit 2024-03-11T21:15:46.285467+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:15:46.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[21512]: audit 2024-03-11T21:15:46.286763+0000 mon.a (mon.0) 251 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:46.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[29284]: audit 2024-03-11T21:15:46.279404+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:15:46.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[29284]: audit 2024-03-11T21:15:46.285467+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:15:46.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:46 smithi143 bash[29284]: audit 2024-03-11T21:15:46.286763+0000 mon.a (mon.0) 251 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:46.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:46 smithi203 bash[23654]: audit 2024-03-11T21:15:46.279404+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:15:46.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:46 smithi203 bash[23654]: audit 2024-03-11T21:15:46.285467+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:15:46.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:46 smithi203 bash[23654]: audit 2024-03-11T21:15:46.286763+0000 mon.a (mon.0) 251 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:47.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:47 smithi143 bash[29284]: cluster 2024-03-11T21:15:45.432206+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:47.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:47 smithi143 bash[29284]: audit 2024-03-11T21:15:46.275042+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:47.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:47 smithi143 bash[21512]: cluster 2024-03-11T21:15:45.432206+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:47.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:47 smithi143 bash[21512]: audit 2024-03-11T21:15:46.275042+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:47 smithi203 bash[23654]: cluster 2024-03-11T21:15:45.432206+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:47 smithi203 bash[23654]: audit 2024-03-11T21:15:46.275042+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:49.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:49 smithi143 bash[21512]: cluster 2024-03-11T21:15:47.432582+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:49.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:49 smithi143 bash[29284]: cluster 2024-03-11T21:15:47.432582+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:49.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:49 smithi203 bash[23654]: cluster 2024-03-11T21:15:47.432582+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:51.048+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:15:51.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:51 smithi143 bash[21512]: cluster 2024-03-11T21:15:49.433063+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:51.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:51 smithi143 bash[29284]: cluster 2024-03-11T21:15:49.433063+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:51.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:51 smithi203 bash[23654]: cluster 2024-03-11T21:15:49.433063+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:53.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:53 smithi143 bash[21512]: cluster 2024-03-11T21:15:51.433459+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:53.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:53 smithi143 bash[29284]: cluster 2024-03-11T21:15:51.433459+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:53.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:53 smithi203 bash[23654]: cluster 2024-03-11T21:15:51.433459+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: cluster 2024-03-11T21:15:53.433820+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: audit 2024-03-11T21:15:54.428123+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.143:0/1447915181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: audit 2024-03-11T21:15:54.428866+0000 mon.a (mon.0) 252 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: audit 2024-03-11T21:15:54.433468+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]': finished 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: cluster 2024-03-11T21:15:54.438810+0000 mon.a (mon.0) 254 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: audit 2024-03-11T21:15:54.439317+0000 mon.a (mon.0) 255 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:15:55.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[21512]: audit 2024-03-11T21:15:55.168991+0000 mon.a (mon.0) 256 : audit 0 from='client.? 172.21.15.143:0/3919436952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: cluster 2024-03-11T21:15:53.433820+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: audit 2024-03-11T21:15:54.428123+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.143:0/1447915181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: audit 2024-03-11T21:15:54.428866+0000 mon.a (mon.0) 252 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: audit 2024-03-11T21:15:54.433468+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]': finished 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: cluster 2024-03-11T21:15:54.438810+0000 mon.a (mon.0) 254 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: audit 2024-03-11T21:15:54.439317+0000 mon.a (mon.0) 255 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:15:55.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:55 smithi143 bash[29284]: audit 2024-03-11T21:15:55.168991+0000 mon.a (mon.0) 256 : audit 0 from='client.? 172.21.15.143:0/3919436952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:15:55.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: cluster 2024-03-11T21:15:53.433820+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: audit 2024-03-11T21:15:54.428123+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.143:0/1447915181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: audit 2024-03-11T21:15:54.428866+0000 mon.a (mon.0) 252 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]: dispatch 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: audit 2024-03-11T21:15:54.433468+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d"}]': finished 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: cluster 2024-03-11T21:15:54.438810+0000 mon.a (mon.0) 254 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: audit 2024-03-11T21:15:54.439317+0000 mon.a (mon.0) 255 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:15:55.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:55 smithi203 bash[23654]: audit 2024-03-11T21:15:55.168991+0000 mon.a (mon.0) 256 : audit 0 from='client.? 172.21.15.143:0/3919436952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:15:56.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:15:56.052+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:15:56.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:56 smithi143 bash[21512]: cluster 2024-03-11T21:15:55.434211+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:56.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:56 smithi143 bash[29284]: cluster 2024-03-11T21:15:55.434211+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:56 smithi203 bash[23654]: cluster 2024-03-11T21:15:55.434211+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:58.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:15:58 smithi203 bash[23654]: cluster 2024-03-11T21:15:57.434555+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:58.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:15:58 smithi143 bash[21512]: cluster 2024-03-11T21:15:57.434555+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:15:58.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:15:58 smithi143 bash[29284]: cluster 2024-03-11T21:15:57.434555+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:00.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:00 smithi203 bash[23654]: cluster 2024-03-11T21:15:59.434944+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:00.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:00 smithi143 bash[21512]: cluster 2024-03-11T21:15:59.434944+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:00.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:00 smithi143 bash[29284]: cluster 2024-03-11T21:15:59.434944+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:01.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:01.056+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:16:02.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:02 smithi203 bash[23654]: cluster 2024-03-11T21:16:01.435376+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:02.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:02 smithi143 bash[21512]: cluster 2024-03-11T21:16:01.435376+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:02.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:02 smithi143 bash[29284]: cluster 2024-03-11T21:16:01.435376+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:04.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:04 smithi143 bash[21512]: cluster 2024-03-11T21:16:03.435787+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:04.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:04 smithi143 bash[29284]: cluster 2024-03-11T21:16:03.435787+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:04.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:04 smithi203 bash[23654]: cluster 2024-03-11T21:16:03.435787+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:06.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:06.060+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:16:06.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:06 smithi203 bash[23654]: cluster 2024-03-11T21:16:05.436196+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:06.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:06 smithi143 bash[21512]: cluster 2024-03-11T21:16:05.436196+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:06.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:06 smithi143 bash[29284]: cluster 2024-03-11T21:16:05.436196+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:07.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:07 smithi203 bash[23654]: audit 2024-03-11T21:16:06.834353+0000 mon.a (mon.0) 257 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:16:07.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:07 smithi203 bash[23654]: audit 2024-03-11T21:16:06.836101+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:07 smithi203 bash[23654]: cephadm 2024-03-11T21:16:06.837536+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi143 2024-03-11T21:16:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[21512]: audit 2024-03-11T21:16:06.834353+0000 mon.a (mon.0) 257 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:16:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[21512]: audit 2024-03-11T21:16:06.836101+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[21512]: cephadm 2024-03-11T21:16:06.837536+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi143 2024-03-11T21:16:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[29284]: audit 2024-03-11T21:16:06.834353+0000 mon.a (mon.0) 257 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:16:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[29284]: audit 2024-03-11T21:16:06.836101+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:07 smithi143 bash[29284]: cephadm 2024-03-11T21:16:06.837536+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi143 2024-03-11T21:16:08.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:08 smithi203 bash[23654]: cluster 2024-03-11T21:16:07.436620+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:08.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:08 smithi143 bash[21512]: cluster 2024-03-11T21:16:07.436620+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:08.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:08 smithi143 bash[29284]: cluster 2024-03-11T21:16:07.436620+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:10 smithi203 bash[23654]: cluster 2024-03-11T21:16:09.437082+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:10.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:10 smithi143 bash[29284]: cluster 2024-03-11T21:16:09.437082+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:10.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:10 smithi143 bash[21512]: cluster 2024-03-11T21:16:09.437082+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:11.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:11.064+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:16:11.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:11.104 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:11.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:11.375 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:11.376 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:11.376 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:16:11 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:16:12.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[29284]: audit 2024-03-11T21:16:11.383527+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[29284]: audit 2024-03-11T21:16:11.388611+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[29284]: audit 2024-03-11T21:16:11.426539+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:12.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[29284]: cluster 2024-03-11T21:16:11.437345+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:12.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[21512]: audit 2024-03-11T21:16:11.383527+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[21512]: audit 2024-03-11T21:16:11.388611+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[21512]: audit 2024-03-11T21:16:11.426539+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:12.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:12 smithi143 bash[21512]: cluster 2024-03-11T21:16:11.437345+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:12.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:12 smithi203 bash[23654]: audit 2024-03-11T21:16:11.383527+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:12 smithi203 bash[23654]: audit 2024-03-11T21:16:11.388611+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:12.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:12 smithi203 bash[23654]: audit 2024-03-11T21:16:11.426539+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:12.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:12 smithi203 bash[23654]: cluster 2024-03-11T21:16:11.437345+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:14.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:14 smithi143 bash[21512]: cluster 2024-03-11T21:16:13.437756+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:14.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:14 smithi143 bash[29284]: cluster 2024-03-11T21:16:13.437756+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:14.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:14 smithi203 bash[23654]: cluster 2024-03-11T21:16:13.437756+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:16.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:16.068+0000 7f3e9a937700 10 client.14208.objecter tick 2024-03-11T21:16:16.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:16 smithi143 bash[29284]: cluster 2024-03-11T21:16:15.438219+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:16.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:16 smithi143 bash[21512]: cluster 2024-03-11T21:16:15.438219+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:16.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:16 smithi203 bash[23654]: cluster 2024-03-11T21:16:15.438219+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:17.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[21512]: audit 2024-03-11T21:16:16.660866+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:17.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[21512]: audit 2024-03-11T21:16:16.665615+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[21512]: audit 2024-03-11T21:16:17.234586+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[21512]: audit 2024-03-11T21:16:17.235378+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[21512]: audit 2024-03-11T21:16:17.242797+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[29284]: audit 2024-03-11T21:16:16.660866+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[29284]: audit 2024-03-11T21:16:16.665615+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[29284]: audit 2024-03-11T21:16:17.234586+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[29284]: audit 2024-03-11T21:16:17.235378+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:17.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:17 smithi143 bash[29284]: audit 2024-03-11T21:16:17.242797+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:18.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:17 smithi203 bash[23654]: audit 2024-03-11T21:16:16.660866+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:18.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:17 smithi203 bash[23654]: audit 2024-03-11T21:16:16.665615+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:18.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:17 smithi203 bash[23654]: audit 2024-03-11T21:16:17.234586+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:18.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:17 smithi203 bash[23654]: audit 2024-03-11T21:16:17.235378+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:18.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:17 smithi203 bash[23654]: audit 2024-03-11T21:16:17.242797+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:18.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:18 smithi143 bash[21512]: cluster 2024-03-11T21:16:17.438592+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:18.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:18 smithi143 bash[21512]: audit 2024-03-11T21:16:17.780421+0000 mon.a (mon.0) 267 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:16:18.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:18 smithi143 bash[29284]: cluster 2024-03-11T21:16:17.438592+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:18.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:18 smithi143 bash[29284]: audit 2024-03-11T21:16:17.780421+0000 mon.a (mon.0) 267 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:16:19.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:18 smithi203 bash[23654]: cluster 2024-03-11T21:16:17.438592+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:19.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:18 smithi203 bash[23654]: audit 2024-03-11T21:16:17.780421+0000 mon.a (mon.0) 267 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:16:19.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[21512]: audit 2024-03-11T21:16:18.671321+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:16:19.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[21512]: cluster 2024-03-11T21:16:18.676250+0000 mon.a (mon.0) 269 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[21512]: audit 2024-03-11T21:16:18.677362+0000 mon.a (mon.0) 270 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[21512]: audit 2024-03-11T21:16:18.677906+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[29284]: audit 2024-03-11T21:16:18.671321+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[29284]: cluster 2024-03-11T21:16:18.676250+0000 mon.a (mon.0) 269 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[29284]: audit 2024-03-11T21:16:18.677362+0000 mon.a (mon.0) 270 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:16:19.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:19 smithi143 bash[29284]: audit 2024-03-11T21:16:18.677906+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:19 smithi203 bash[23654]: audit 2024-03-11T21:16:18.671321+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:16:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:19 smithi203 bash[23654]: cluster 2024-03-11T21:16:18.676250+0000 mon.a (mon.0) 269 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:16:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:19 smithi203 bash[23654]: audit 2024-03-11T21:16:18.677362+0000 mon.a (mon.0) 270 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:16:20.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:19 smithi203 bash[23654]: audit 2024-03-11T21:16:18.677906+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.276+0000 7f3e91ffb700 1 -- 172.21.15.143:0/2896064622 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3e9402d9a0 con 0x7f3e7c06f690 2024-03-11T21:16:20.266 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 0 on host 'smithi143' 2024-03-11T21:16:20.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 10 librados: watch_flush enter 2024-03-11T21:16:20.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 10 librados: watch_flush exit 2024-03-11T21:16:20.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 20 client.14208.objecter shutdown clearing up homeless session... 2024-03-11T21:16:20.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 10 client.14208.objecter successfully canceled tick 2024-03-11T21:16:20.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 msgr2=0x7f3e7c071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:16:20.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 0x7f3e7c071b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009750 tx=0x7f3e88009310 comp rx=0 tx=0).stop 2024-03-11T21:16:20.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 msgr2=0x7f3e94197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e840094f0 tx=0x7f3e8400b560 comp rx=0 tx=0).stop 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 shutdown_connections 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f3e7c06f690 0x7f3e7c071b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e94102960 0x7f3e94197750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e94107bc0 0x7f3e9419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 --2- 172.21.15.143:0/2896064622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e94105290 0x7f3e94197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 >> 172.21.15.143:0/2896064622 conn(0x7f3e940fc310 msgr2=0x7f3e941035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 shutdown_connections 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 -- 172.21.15.143:0/2896064622 wait complete. 2024-03-11T21:16:20.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:20.280+0000 7f3e9b138700 1 librados: shutdown 2024-03-11T21:16:20.921 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: cluster 2024-03-11T21:16:19.438867+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:19.677318+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: cluster 2024-03-11T21:16:19.681863+0000 mon.a (mon.0) 273 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:19.682705+0000 mon.a (mon.0) 274 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:19.686564+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:20.199993+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:20.237885+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:20.243221+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[21512]: audit 2024-03-11T21:16:20.266226+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:20.922 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: cluster 2024-03-11T21:16:19.438867+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:19.677318+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: cluster 2024-03-11T21:16:19.681863+0000 mon.a (mon.0) 273 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:19.682705+0000 mon.a (mon.0) 274 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:19.686564+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:20.199993+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:20.237885+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:20.243221+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:20.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:20 smithi143 bash[29284]: audit 2024-03-11T21:16:20.266226+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:20.924 DEBUG:teuthology.orchestra.run.smithi143:osd.0> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:16:20.927 INFO:tasks.cephadm:Deploying osd.1 on smithi143 with /dev/vg_nvme/lv_3... 2024-03-11T21:16:20.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T21:16:21.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: cluster 2024-03-11T21:16:19.438867+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:21.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:19.677318+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: cluster 2024-03-11T21:16:19.681863+0000 mon.a (mon.0) 273 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:19.682705+0000 mon.a (mon.0) 274 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:19.686564+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:20.199993+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023]' entity='osd.0' 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:20.237885+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:20.243221+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:21.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:20 smithi203 bash[23654]: audit 2024-03-11T21:16:20.266226+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:21.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: cluster 2024-03-11T21:16:18.754990+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: cluster 2024-03-11T21:16:18.755122+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: audit 2024-03-11T21:16:20.685603+0000 mon.a (mon.0) 280 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: cluster 2024-03-11T21:16:21.249737+0000 mon.a (mon.0) 281 : cluster 1 osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] boot 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: cluster 2024-03-11T21:16:21.249795+0000 mon.a (mon.0) 282 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[21512]: audit 2024-03-11T21:16:21.250143+0000 mon.a (mon.0) 283 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: cluster 2024-03-11T21:16:18.754990+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: cluster 2024-03-11T21:16:18.755122+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:21.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: audit 2024-03-11T21:16:20.685603+0000 mon.a (mon.0) 280 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:21.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: cluster 2024-03-11T21:16:21.249737+0000 mon.a (mon.0) 281 : cluster 1 osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] boot 2024-03-11T21:16:21.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: cluster 2024-03-11T21:16:21.249795+0000 mon.a (mon.0) 282 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:16:21.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:21 smithi143 bash[29284]: audit 2024-03-11T21:16:21.250143+0000 mon.a (mon.0) 283 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: cluster 2024-03-11T21:16:18.754990+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: cluster 2024-03-11T21:16:18.755122+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: audit 2024-03-11T21:16:20.685603+0000 mon.a (mon.0) 280 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: cluster 2024-03-11T21:16:21.249737+0000 mon.a (mon.0) 281 : cluster 1 osd.0 [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] boot 2024-03-11T21:16:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: cluster 2024-03-11T21:16:21.249795+0000 mon.a (mon.0) 282 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:16:22.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:21 smithi203 bash[23654]: audit 2024-03-11T21:16:21.250143+0000 mon.a (mon.0) 283 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:16:23.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:23 smithi203 bash[23654]: cluster 2024-03-11T21:16:21.439227+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:23.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:23 smithi203 bash[23654]: cluster 2024-03-11T21:16:22.260328+0000 mon.a (mon.0) 284 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:16:23.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:23 smithi143 bash[21512]: cluster 2024-03-11T21:16:21.439227+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:23.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:23 smithi143 bash[21512]: cluster 2024-03-11T21:16:22.260328+0000 mon.a (mon.0) 284 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:16:23.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:23 smithi143 bash[29284]: cluster 2024-03-11T21:16:21.439227+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:23.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:23 smithi143 bash[29284]: cluster 2024-03-11T21:16:22.260328+0000 mon.a (mon.0) 284 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:16:25.185 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:25.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:25 smithi203 bash[23654]: cluster 2024-03-11T21:16:23.439703+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:25.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:25 smithi143 bash[21512]: cluster 2024-03-11T21:16:23.439703+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:25.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:25 smithi143 bash[29284]: cluster 2024-03-11T21:16:23.439703+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:27.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:27 smithi203 bash[23654]: cluster 2024-03-11T21:16:25.440157+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:27.590 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:27 smithi143 bash[29284]: cluster 2024-03-11T21:16:25.440157+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:27.591 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:27 smithi143 bash[21512]: cluster 2024-03-11T21:16:25.440157+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:29.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:29 smithi143 bash[21512]: cluster 2024-03-11T21:16:27.440663+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:29.502 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:29 smithi143 bash[29284]: cluster 2024-03-11T21:16:27.440663+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:29.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:29 smithi203 bash[23654]: cluster 2024-03-11T21:16:27.440663+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013373 s, 784 MB/s 2024-03-11T21:16:30.124 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-11T21:16:30.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_3 2024-03-11T21:16:31.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:31 smithi203 bash[23654]: cluster 2024-03-11T21:16:29.441020+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:31.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:31 smithi143 bash[21512]: cluster 2024-03-11T21:16:29.441020+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:31.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:31 smithi143 bash[29284]: cluster 2024-03-11T21:16:29.441020+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: cephadm 2024-03-11T21:16:31.362713+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.372395+0000 mon.a (mon.0) 285 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.381667+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.383597+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.393036+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.397356+0000 mon.a (mon.0) 289 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:32.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.399032+0000 mon.a (mon.0) 290 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[21512]: audit 2024-03-11T21:16:31.408571+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: cephadm 2024-03-11T21:16:31.362713+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.372395+0000 mon.a (mon.0) 285 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.381667+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.383597+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.393036+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.397356+0000 mon.a (mon.0) 289 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.399032+0000 mon.a (mon.0) 290 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:32.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:32 smithi143 bash[29284]: audit 2024-03-11T21:16:31.408571+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: cephadm 2024-03-11T21:16:31.362713+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:16:32.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.372395+0000 mon.a (mon.0) 285 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.381667+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.383597+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.393036+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.397356+0000 mon.a (mon.0) 289 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.399032+0000 mon.a (mon.0) 290 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:32.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:32 smithi203 bash[23654]: audit 2024-03-11T21:16:31.408571+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:16:33.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:33 smithi143 bash[29284]: cephadm 2024-03-11T21:16:31.384951+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi143 to 16241M 2024-03-11T21:16:33.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:33 smithi143 bash[29284]: cluster 2024-03-11T21:16:31.441414+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:33.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:33 smithi143 bash[21512]: cephadm 2024-03-11T21:16:31.384951+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi143 to 16241M 2024-03-11T21:16:33.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:33 smithi143 bash[21512]: cluster 2024-03-11T21:16:31.441414+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:33.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:33 smithi203 bash[23654]: cephadm 2024-03-11T21:16:31.384951+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi143 to 16241M 2024-03-11T21:16:33.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:33 smithi203 bash[23654]: cluster 2024-03-11T21:16:31.441414+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:35.675 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:35.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:35 smithi143 bash[21512]: cluster 2024-03-11T21:16:33.441865+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:35.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:35 smithi143 bash[29284]: cluster 2024-03-11T21:16:33.441865+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:35.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:35 smithi203 bash[23654]: cluster 2024-03-11T21:16:33.441865+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:36.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:36 smithi143 bash[21512]: cluster 2024-03-11T21:16:35.442287+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:36.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:36 smithi143 bash[29284]: cluster 2024-03-11T21:16:35.442287+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:36.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:36 smithi203 bash[23654]: cluster 2024-03-11T21:16:35.442287+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:38.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/665776080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 msgr2=0x7fde58100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:16:38.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/665776080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58100b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde48009990 tx=0x7fde48009ca0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/665776080 shutdown_connections 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/665776080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 0x7fde58101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/665776080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58100b10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/665776080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde581014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/665776080 >> 172.21.15.143:0/665776080 conn(0x7fde580fc310 msgr2=0x7fde580fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:16:38.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/665776080 shutdown_connections 2024-03-11T21:16:38.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/665776080 wait complete. 2024-03-11T21:16:38.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 librados: starting msgr at 2024-03-11T21:16:38.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 librados: starting objecter 2024-03-11T21:16:38.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 Processor -- start 2024-03-11T21:16:38.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- start start 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 librados: setting wanted keys 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 librados: calling monclient init 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde58199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 0x7fde581a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:16:38.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5810f150 con 0x7fde58101a10 2024-03-11T21:16:38.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5810efd0 con 0x7fde58101050 2024-03-11T21:16:38.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5810f2d0 con 0x7fde581006f0 2024-03-11T21:16:38.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:16:38.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5ce68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 0x7fde581a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:16:38.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44986/0 (socket says 172.21.15.143:44986) 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 -- 172.21.15.143:0/3516930370 learned_addr learned my addr 172.21.15.143:0/3516930370 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde577fe700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde58199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 -- 172.21.15.143:0/3516930370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 msgr2=0x7fde58199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde58199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 -- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 msgr2=0x7fde581a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:16:38.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 0x7fde581a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:16:38.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 -- 172.21.15.143:0/3516930370 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde581a14d0 con 0x7fde581006f0 2024-03-11T21:16:38.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde577fe700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde58199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:16:38.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde57fff700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde48000c00 tx=0x7fde4800f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:16:38.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde557fa700 10 client.?.objecter ms_handle_connect 0x7fde581006f0 2024-03-11T21:16:38.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde557fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48027070 con 0x7fde581006f0 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde581a1760 con 0x7fde581006f0 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fde4800fe50 con 0x7fde581006f0 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48017650 con 0x7fde581006f0 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 10 client.24143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde581a1c10 con 0x7fde581006f0 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.120+0000 7fde5e8cb700 1 librados: init done 2024-03-11T21:16:38.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5804faa0 con 0x7fde581006f0 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7fde4802a070 con 0x7fde581006f0 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 0x7fde44071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fde4808db20 con 0x7fde581006f0 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde577fe700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 0x7fde44071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 10 client.24143.objecter ms_dispatch 0x7fde58001550 osd_map(9..9 src has 1..9) v4 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 3 client.24143.objecter handle_osd_map got epochs [9,9] > 0 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 3 client.24143.objecter handle_osd_map decoding full epoch 9 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 20 client.24143.objecter dump_active .. 0 homeless 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde557fa700 10 client.24143.objecter ms_handle_connect 0x7fde4406f690 2024-03-11T21:16:38.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.124+0000 7fde577fe700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 0x7fde44071b50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde40005d90 tx=0x7fde40005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:16:38.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.128+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fde48013070 con 0x7fde581006f0 2024-03-11T21:16:38.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:38.344+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fde58062220 con 0x7fde4406f690 2024-03-11T21:16:38.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:38 smithi203 bash[23654]: cluster 2024-03-11T21:16:37.442869+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:38.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:38 smithi203 bash[23654]: audit 2024-03-11T21:16:38.351107+0000 mgr.y (mgr.14152) 118 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:38.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:38 smithi203 bash[23654]: audit 2024-03-11T21:16:38.354732+0000 mon.a (mon.0) 292 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:16:38.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:38 smithi203 bash[23654]: audit 2024-03-11T21:16:38.360001+0000 mon.a (mon.0) 293 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:16:38.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:38 smithi203 bash[23654]: audit 2024-03-11T21:16:38.360953+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[21512]: cluster 2024-03-11T21:16:37.442869+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[21512]: audit 2024-03-11T21:16:38.351107+0000 mgr.y (mgr.14152) 118 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[21512]: audit 2024-03-11T21:16:38.354732+0000 mon.a (mon.0) 292 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[21512]: audit 2024-03-11T21:16:38.360001+0000 mon.a (mon.0) 293 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[21512]: audit 2024-03-11T21:16:38.360953+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[29284]: cluster 2024-03-11T21:16:37.442869+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[29284]: audit 2024-03-11T21:16:38.351107+0000 mgr.y (mgr.14152) 118 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[29284]: audit 2024-03-11T21:16:38.354732+0000 mon.a (mon.0) 292 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[29284]: audit 2024-03-11T21:16:38.360001+0000 mon.a (mon.0) 293 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:16:38.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:38 smithi143 bash[29284]: audit 2024-03-11T21:16:38.360953+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:40.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:40 smithi203 bash[23654]: cluster 2024-03-11T21:16:39.443373+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:40.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:40 smithi143 bash[21512]: cluster 2024-03-11T21:16:39.443373+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:40.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:40 smithi143 bash[29284]: cluster 2024-03-11T21:16:39.443373+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:42.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:42 smithi203 bash[23654]: cluster 2024-03-11T21:16:41.443870+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:42.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:42 smithi143 bash[21512]: cluster 2024-03-11T21:16:41.443870+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:42.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:42 smithi143 bash[29284]: cluster 2024-03-11T21:16:41.443870+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:43.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:43.124+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:16:44.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:44 smithi203 bash[23654]: cluster 2024-03-11T21:16:43.444301+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:44.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:44 smithi143 bash[21512]: cluster 2024-03-11T21:16:43.444301+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:44.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:44 smithi143 bash[29284]: cluster 2024-03-11T21:16:43.444301+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:46.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:46 smithi203 bash[23654]: cluster 2024-03-11T21:16:45.444772+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:46.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:46 smithi143 bash[29284]: cluster 2024-03-11T21:16:45.444772+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:46.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:46 smithi143 bash[21512]: cluster 2024-03-11T21:16:45.444772+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: audit 2024-03-11T21:16:46.646862+0000 mon.c (mon.2) 3 : audit 1 from='client.? 172.21.15.143:0/1803644088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: audit 2024-03-11T21:16:46.647554+0000 mon.a (mon.0) 295 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: audit 2024-03-11T21:16:46.650453+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]': finished 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: cluster 2024-03-11T21:16:46.655802+0000 mon.a (mon.0) 297 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: audit 2024-03-11T21:16:46.656324+0000 mon.a (mon.0) 298 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:16:47.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:47 smithi203 bash[23654]: audit 2024-03-11T21:16:47.419870+0000 mon.c (mon.2) 4 : audit 0 from='client.? 172.21.15.143:0/1845775440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:16:47.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: audit 2024-03-11T21:16:46.646862+0000 mon.c (mon.2) 3 : audit 1 from='client.? 172.21.15.143:0/1803644088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: audit 2024-03-11T21:16:46.647554+0000 mon.a (mon.0) 295 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: audit 2024-03-11T21:16:46.650453+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]': finished 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: cluster 2024-03-11T21:16:46.655802+0000 mon.a (mon.0) 297 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: audit 2024-03-11T21:16:46.656324+0000 mon.a (mon.0) 298 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[21512]: audit 2024-03-11T21:16:47.419870+0000 mon.c (mon.2) 4 : audit 0 from='client.? 172.21.15.143:0/1845775440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: audit 2024-03-11T21:16:46.646862+0000 mon.c (mon.2) 3 : audit 1 from='client.? 172.21.15.143:0/1803644088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: audit 2024-03-11T21:16:46.647554+0000 mon.a (mon.0) 295 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: audit 2024-03-11T21:16:46.650453+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02"}]': finished 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: cluster 2024-03-11T21:16:46.655802+0000 mon.a (mon.0) 297 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: audit 2024-03-11T21:16:46.656324+0000 mon.a (mon.0) 298 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:16:47.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:47 smithi143 bash[29284]: audit 2024-03-11T21:16:47.419870+0000 mon.c (mon.2) 4 : audit 0 from='client.? 172.21.15.143:0/1845775440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:16:48.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:48.128+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:16:48.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:48 smithi143 bash[21512]: cluster 2024-03-11T21:16:47.445086+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:48.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:48 smithi143 bash[29284]: cluster 2024-03-11T21:16:47.445086+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:48 smithi203 bash[23654]: cluster 2024-03-11T21:16:47.445086+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:50.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:50 smithi143 bash[21512]: cluster 2024-03-11T21:16:49.445507+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:50.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:50 smithi143 bash[29284]: cluster 2024-03-11T21:16:49.445507+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:51.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:50 smithi203 bash[23654]: cluster 2024-03-11T21:16:49.445507+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:52.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:52 smithi143 bash[21512]: cluster 2024-03-11T21:16:51.446058+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:52.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:52 smithi143 bash[29284]: cluster 2024-03-11T21:16:51.446058+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:53.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:52 smithi203 bash[23654]: cluster 2024-03-11T21:16:51.446058+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:53.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:53.132+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:16:54.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:54 smithi143 bash[21512]: cluster 2024-03-11T21:16:53.446598+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:54 smithi143 bash[29284]: cluster 2024-03-11T21:16:53.446598+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:55.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:54 smithi203 bash[23654]: cluster 2024-03-11T21:16:53.446598+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:56.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:56 smithi143 bash[21512]: cluster 2024-03-11T21:16:55.447060+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:56.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:56 smithi143 bash[29284]: cluster 2024-03-11T21:16:55.447060+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:57.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:56 smithi203 bash[23654]: cluster 2024-03-11T21:16:55.447060+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:58.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:16:58.136+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:16:58.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:58 smithi143 bash[21512]: cluster 2024-03-11T21:16:57.447543+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:58.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:58 smithi143 bash[29284]: cluster 2024-03-11T21:16:57.447543+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:59.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:58 smithi203 bash[23654]: cluster 2024-03-11T21:16:57.447543+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:16:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[21512]: audit 2024-03-11T21:16:59.094419+0000 mon.a (mon.0) 299 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:16:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[21512]: audit 2024-03-11T21:16:59.096272+0000 mon.a (mon.0) 300 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[21512]: cephadm 2024-03-11T21:16:59.098016+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi143 2024-03-11T21:16:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[29284]: audit 2024-03-11T21:16:59.094419+0000 mon.a (mon.0) 299 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:16:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[29284]: audit 2024-03-11T21:16:59.096272+0000 mon.a (mon.0) 300 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:16:59 smithi143 bash[29284]: cephadm 2024-03-11T21:16:59.098016+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi143 2024-03-11T21:17:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:59 smithi203 bash[23654]: audit 2024-03-11T21:16:59.094419+0000 mon.a (mon.0) 299 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:17:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:59 smithi203 bash[23654]: audit 2024-03-11T21:16:59.096272+0000 mon.a (mon.0) 300 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:16:59 smithi203 bash[23654]: cephadm 2024-03-11T21:16:59.098016+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi143 2024-03-11T21:17:00.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:00 smithi143 bash[21512]: cluster 2024-03-11T21:16:59.448010+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:00 smithi143 bash[29284]: cluster 2024-03-11T21:16:59.448010+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:01.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:00 smithi203 bash[23654]: cluster 2024-03-11T21:16:59.448010+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:02.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:02 smithi143 bash[21512]: cluster 2024-03-11T21:17:01.448417+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:02.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:02 smithi143 bash[29284]: cluster 2024-03-11T21:17:01.448417+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:03.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:02 smithi203 bash[23654]: cluster 2024-03-11T21:17:01.448417+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:03.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:03.140+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:17:03.664 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.664 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.665 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.665 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.946 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.946 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:03.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:03 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[21512]: cluster 2024-03-11T21:17:03.448751+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[21512]: audit 2024-03-11T21:17:03.825025+0000 mon.a (mon.0) 301 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[21512]: audit 2024-03-11T21:17:03.830055+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[21512]: audit 2024-03-11T21:17:03.838713+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[29284]: cluster 2024-03-11T21:17:03.448751+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:04.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[29284]: audit 2024-03-11T21:17:03.825025+0000 mon.a (mon.0) 301 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[29284]: audit 2024-03-11T21:17:03.830055+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:04 smithi143 bash[29284]: audit 2024-03-11T21:17:03.838713+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:05.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:04 smithi203 bash[23654]: cluster 2024-03-11T21:17:03.448751+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:05.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:04 smithi203 bash[23654]: audit 2024-03-11T21:17:03.825025+0000 mon.a (mon.0) 301 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:05.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:04 smithi203 bash[23654]: audit 2024-03-11T21:17:03.830055+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:05.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:04 smithi203 bash[23654]: audit 2024-03-11T21:17:03.838713+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:06.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:06 smithi143 bash[21512]: cluster 2024-03-11T21:17:05.449256+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:06.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:06 smithi143 bash[29284]: cluster 2024-03-11T21:17:05.449256+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:07.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:06 smithi203 bash[23654]: cluster 2024-03-11T21:17:05.449256+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:08.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:08.144+0000 7fde5e0ca700 10 client.24143.objecter tick 2024-03-11T21:17:08.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:08 smithi143 bash[21512]: cluster 2024-03-11T21:17:07.449784+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:08.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:08 smithi143 bash[29284]: cluster 2024-03-11T21:17:07.449784+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:09.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:08 smithi203 bash[23654]: cluster 2024-03-11T21:17:07.449784+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[29284]: audit 2024-03-11T21:17:09.117045+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[29284]: audit 2024-03-11T21:17:09.121617+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[29284]: audit 2024-03-11T21:17:09.122406+0000 mon.a (mon.0) 306 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[29284]: audit 2024-03-11T21:17:09.123065+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[29284]: audit 2024-03-11T21:17:09.127432+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[21512]: audit 2024-03-11T21:17:09.117045+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[21512]: audit 2024-03-11T21:17:09.121617+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[21512]: audit 2024-03-11T21:17:09.122406+0000 mon.a (mon.0) 306 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:10.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[21512]: audit 2024-03-11T21:17:09.123065+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:10.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:10 smithi143 bash[21512]: audit 2024-03-11T21:17:09.127432+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:10 smithi203 bash[23654]: audit 2024-03-11T21:17:09.117045+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:10 smithi203 bash[23654]: audit 2024-03-11T21:17:09.121617+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:10.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:10 smithi203 bash[23654]: audit 2024-03-11T21:17:09.122406+0000 mon.a (mon.0) 306 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:10.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:10 smithi203 bash[23654]: audit 2024-03-11T21:17:09.123065+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:10.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:10 smithi203 bash[23654]: audit 2024-03-11T21:17:09.127432+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:11.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[21512]: cluster 2024-03-11T21:17:09.450142+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:11.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[21512]: audit 2024-03-11T21:17:10.293798+0000 mon.c (mon.2) 5 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:11.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[21512]: audit 2024-03-11T21:17:10.294152+0000 mon.a (mon.0) 309 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:11.392 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[29284]: cluster 2024-03-11T21:17:09.450142+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:11.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[29284]: audit 2024-03-11T21:17:10.293798+0000 mon.c (mon.2) 5 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:11.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:11 smithi143 bash[29284]: audit 2024-03-11T21:17:10.294152+0000 mon.a (mon.0) 309 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:11.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:11 smithi203 bash[23654]: cluster 2024-03-11T21:17:09.450142+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:11.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:11 smithi203 bash[23654]: audit 2024-03-11T21:17:10.293798+0000 mon.c (mon.2) 5 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:11.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:11 smithi203 bash[23654]: audit 2024-03-11T21:17:10.294152+0000 mon.a (mon.0) 309 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[21512]: audit 2024-03-11T21:17:11.138369+0000 mon.a (mon.0) 310 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[21512]: cluster 2024-03-11T21:17:11.143372+0000 mon.a (mon.0) 311 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[21512]: audit 2024-03-11T21:17:11.144050+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[21512]: audit 2024-03-11T21:17:11.144677+0000 mon.c (mon.2) 6 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[21512]: audit 2024-03-11T21:17:11.145415+0000 mon.a (mon.0) 313 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[29284]: audit 2024-03-11T21:17:11.138369+0000 mon.a (mon.0) 310 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:17:12.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[29284]: cluster 2024-03-11T21:17:11.143372+0000 mon.a (mon.0) 311 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:17:12.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[29284]: audit 2024-03-11T21:17:11.144050+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:12.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[29284]: audit 2024-03-11T21:17:11.144677+0000 mon.c (mon.2) 6 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:12 smithi143 bash[29284]: audit 2024-03-11T21:17:11.145415+0000 mon.a (mon.0) 313 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:12 smithi203 bash[23654]: audit 2024-03-11T21:17:11.138369+0000 mon.a (mon.0) 310 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:17:12.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:12 smithi203 bash[23654]: cluster 2024-03-11T21:17:11.143372+0000 mon.a (mon.0) 311 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:17:12.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:12 smithi203 bash[23654]: audit 2024-03-11T21:17:11.144050+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:12.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:12 smithi203 bash[23654]: audit 2024-03-11T21:17:11.144677+0000 mon.c (mon.2) 6 : audit 1 from='osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:12 smithi203 bash[23654]: audit 2024-03-11T21:17:11.145415+0000 mon.a (mon.0) 313 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:17:12.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.804+0000 7fde557fa700 1 -- 172.21.15.143:0/3516930370 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde58062220 con 0x7fde4406f690 2024-03-11T21:17:12.792 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 1 on host 'smithi143' 2024-03-11T21:17:12.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 10 librados: watch_flush enter 2024-03-11T21:17:12.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 10 librados: watch_flush exit 2024-03-11T21:17:12.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 20 client.24143.objecter shutdown clearing up homeless session... 2024-03-11T21:17:12.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 10 client.24143.objecter successfully canceled tick 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 msgr2=0x7fde44071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 0x7fde44071b50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde40005d90 tx=0x7fde40005d20 comp rx=0 tx=0).stop 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 msgr2=0x7fde58199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde48000c00 tx=0x7fde4800f740 comp rx=0 tx=0).stop 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 shutdown_connections 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fde4406f690 0x7fde44071b50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58101a10 0x7fde581a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:12.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde581006f0 0x7fde58199870 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:12.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 --2- 172.21.15.143:0/3516930370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58101050 0x7fde58199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:12.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 >> 172.21.15.143:0/3516930370 conn(0x7fde580fc310 msgr2=0x7fde580fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:17:12.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 shutdown_connections 2024-03-11T21:17:12.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 -- 172.21.15.143:0/3516930370 wait complete. 2024-03-11T21:17:12.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:12.808+0000 7fde5e8cb700 1 librados: shutdown 2024-03-11T21:17:13.146 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: cluster 2024-03-11T21:17:11.450498+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:13.146 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.140233+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: cluster 2024-03-11T21:17:12.143238+0000 mon.a (mon.0) 315 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.144005+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.153787+0000 mon.a (mon.0) 317 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.790294+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.799089+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[21512]: audit 2024-03-11T21:17:12.800338+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: cluster 2024-03-11T21:17:11.450498+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.140233+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: cluster 2024-03-11T21:17:12.143238+0000 mon.a (mon.0) 315 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.144005+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.153787+0000 mon.a (mon.0) 317 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.790294+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.799089+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:13 smithi143 bash[29284]: audit 2024-03-11T21:17:12.800338+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:13.478 DEBUG:teuthology.orchestra.run.smithi143:osd.1> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:17:13.481 INFO:tasks.cephadm:Deploying osd.2 on smithi143 with /dev/vg_nvme/lv_2... 2024-03-11T21:17:13.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T21:17:13.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: cluster 2024-03-11T21:17:11.450498+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.140233+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: cluster 2024-03-11T21:17:12.143238+0000 mon.a (mon.0) 315 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.144005+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.153787+0000 mon.a (mon.0) 317 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.790294+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.799089+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:13 smithi203 bash[23654]: audit 2024-03-11T21:17:12.800338+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[29284]: cluster 2024-03-11T21:17:11.310896+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[29284]: cluster 2024-03-11T21:17:11.310966+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[29284]: cluster 2024-03-11T21:17:13.156171+0000 mon.a (mon.0) 321 : cluster 1 osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] boot 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[29284]: cluster 2024-03-11T21:17:13.156271+0000 mon.a (mon.0) 322 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[29284]: audit 2024-03-11T21:17:13.156891+0000 mon.a (mon.0) 323 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[21512]: cluster 2024-03-11T21:17:11.310896+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:14.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[21512]: cluster 2024-03-11T21:17:11.310966+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:14.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[21512]: cluster 2024-03-11T21:17:13.156171+0000 mon.a (mon.0) 321 : cluster 1 osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] boot 2024-03-11T21:17:14.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[21512]: cluster 2024-03-11T21:17:13.156271+0000 mon.a (mon.0) 322 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:17:14.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:14 smithi143 bash[21512]: audit 2024-03-11T21:17:13.156891+0000 mon.a (mon.0) 323 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:14 smithi203 bash[23654]: cluster 2024-03-11T21:17:11.310896+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:14 smithi203 bash[23654]: cluster 2024-03-11T21:17:11.310966+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:14 smithi203 bash[23654]: cluster 2024-03-11T21:17:13.156171+0000 mon.a (mon.0) 321 : cluster 1 osd.1 [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] boot 2024-03-11T21:17:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:14 smithi203 bash[23654]: cluster 2024-03-11T21:17:13.156271+0000 mon.a (mon.0) 322 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:17:14.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:14 smithi203 bash[23654]: audit 2024-03-11T21:17:13.156891+0000 mon.a (mon.0) 323 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:15.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:15 smithi143 bash[29284]: cluster 2024-03-11T21:17:13.450959+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:15.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:15 smithi143 bash[29284]: cluster 2024-03-11T21:17:14.176972+0000 mon.a (mon.0) 324 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:17:15.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:15 smithi143 bash[21512]: cluster 2024-03-11T21:17:13.450959+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:15.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:15 smithi143 bash[21512]: cluster 2024-03-11T21:17:14.176972+0000 mon.a (mon.0) 324 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:17:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:15 smithi203 bash[23654]: cluster 2024-03-11T21:17:13.450959+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:15.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:15 smithi203 bash[23654]: cluster 2024-03-11T21:17:14.176972+0000 mon.a (mon.0) 324 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:17:17.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:17 smithi143 bash[21512]: cluster 2024-03-11T21:17:15.451395+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:17.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:17 smithi143 bash[29284]: cluster 2024-03-11T21:17:15.451395+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:17.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:17 smithi203 bash[23654]: cluster 2024-03-11T21:17:15.451395+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:17.737 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:19.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:19 smithi143 bash[21512]: cluster 2024-03-11T21:17:17.451897+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:19 smithi143 bash[29284]: cluster 2024-03-11T21:17:17.451897+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:19.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:19 smithi203 bash[23654]: cluster 2024-03-11T21:17:17.451897+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:21 smithi143 bash[21512]: cluster 2024-03-11T21:17:19.452378+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:21.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:21 smithi143 bash[29284]: cluster 2024-03-11T21:17:19.452378+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:21.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:21 smithi203 bash[23654]: cluster 2024-03-11T21:17:19.452378+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:22.419 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T21:17:22.419 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T21:17:22.419 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-11T21:17:22.419 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-11T21:17:22.420 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0173221 s, 605 MB/s 2024-03-11T21:17:22.420 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-11T21:17:23.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_2 2024-03-11T21:17:23.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:23 smithi203 bash[23654]: cluster 2024-03-11T21:17:21.452869+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:23.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:23 smithi143 bash[21512]: cluster 2024-03-11T21:17:21.452869+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:23.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:23 smithi143 bash[29284]: cluster 2024-03-11T21:17:21.452869+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: cluster 2024-03-11T21:17:23.453244+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: cephadm 2024-03-11T21:17:23.847374+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.855702+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.865027+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.867176+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.868992+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: cephadm 2024-03-11T21:17:23.870315+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi143 to 8120M 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.876485+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.880789+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.882387+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[21512]: audit 2024-03-11T21:17:23.891844+0000 mon.a (mon.0) 332 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: cluster 2024-03-11T21:17:23.453244+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: cephadm 2024-03-11T21:17:23.847374+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.855702+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.865027+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.867176+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.868992+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: cephadm 2024-03-11T21:17:23.870315+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi143 to 8120M 2024-03-11T21:17:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.876485+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.880789+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:25.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.882387+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:25.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:24 smithi143 bash[29284]: audit 2024-03-11T21:17:23.891844+0000 mon.a (mon.0) 332 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: cluster 2024-03-11T21:17:23.453244+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: cephadm 2024-03-11T21:17:23.847374+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.855702+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.865027+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.867176+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.868992+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: cephadm 2024-03-11T21:17:23.870315+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi143 to 8120M 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.876485+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.880789+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:25.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.882387+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:25.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:24 smithi203 bash[23654]: audit 2024-03-11T21:17:23.891844+0000 mon.a (mon.0) 332 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:27.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:26 smithi143 bash[21512]: cluster 2024-03-11T21:17:25.453773+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:27.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:26 smithi143 bash[29284]: cluster 2024-03-11T21:17:25.453773+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:27.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:26 smithi203 bash[23654]: cluster 2024-03-11T21:17:25.453773+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:28.158 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:29.099 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:28 smithi143 bash[29284]: cluster 2024-03-11T21:17:27.454227+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:29.100 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:28 smithi143 bash[21512]: cluster 2024-03-11T21:17:27.454227+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:29.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:28 smithi203 bash[23654]: cluster 2024-03-11T21:17:27.454227+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:30.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- 172.21.15.143:0/3276474044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 msgr2=0x7fe9441023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- 172.21.15.143:0/3276474044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe9441023c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098f0 tx=0x7fe92c009c00 comp rx=0 tx=0).stop 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- 172.21.15.143:0/3276474044 shutdown_connections 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- 172.21.15.143:0/3276474044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944102900 0x7fe9441091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- 172.21.15.143:0/3276474044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944100d60 0x7fe944101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- 172.21.15.143:0/3276474044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe9441023c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- 172.21.15.143:0/3276474044 >> 172.21.15.143:0/3276474044 conn(0x7fe9440fc300 msgr2=0x7fe9440fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- 172.21.15.143:0/3276474044 shutdown_connections 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- 172.21.15.143:0/3276474044 wait complete. 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 librados: starting msgr at 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 librados: starting objecter 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 Processor -- start 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- start start 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 librados: setting wanted keys 2024-03-11T21:17:30.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 librados: calling monclient init 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe944103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 0x7fe944103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94410be60 con 0x7fe944100d60 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94410bce0 con 0x7fe944102900 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe94ad94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94410bfe0 con 0x7fe944101f60 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54712/0 (socket says 172.21.15.143:54712) 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 -- 172.21.15.143:0/752225695 learned_addr learned my addr 172.21.15.143:0/752225695 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe943fff700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe944103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe949331700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 0x7fe944103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:17:30.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 -- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 msgr2=0x7fe944103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe944103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 -- 172.21.15.143:0/752225695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 msgr2=0x7fe944103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 0x7fe944103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 -- 172.21.15.143:0/752225695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944104130 con 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe949331700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 0x7fe944103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe948b30700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe93c007f10 tx=0x7fe93c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe941ffb700 10 client.?.objecter ms_handle_connect 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.795+0000 7fe941ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00ddd0 con 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe93c018070 con 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0125b0 con 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9441a1910 con 0x7fe944100d60 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe94ad94700 10 client.14262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:17:30.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9441a1dd0 con 0x7fe944100d60 2024-03-11T21:17:30.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe94ad94700 1 librados: init done 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94410c620 con 0x7fe944100d60 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7fe93c0133f0 con 0x7fe944100d60 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 0x7fe930071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe943fff700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 0x7fe930071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7fe93c08c9a0 con 0x7fe944100d60 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 10 client.14262.objecter ms_dispatch 0x7fe944001550 osd_map(14..14 src has 1..14) v4 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 3 client.14262.objecter handle_osd_map got epochs [14,14] > 0 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 3 client.14262.objecter handle_osd_map decoding full epoch 14 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 20 client.14262.objecter dump_active .. 0 homeless 2024-03-11T21:17:30.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe941ffb700 10 client.14262.objecter ms_handle_connect 0x7fe93006f690 2024-03-11T21:17:30.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.799+0000 7fe943fff700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 0x7fe930071b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098c0 tx=0x7fe92c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:17:30.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:30.803+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe93c010070 con 0x7fe944100d60 2024-03-11T21:17:31.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:31.019+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe94402db60 con 0x7fe93006f690 2024-03-11T21:17:31.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:30 smithi143 bash[21512]: cluster 2024-03-11T21:17:29.454717+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:31.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:30 smithi143 bash[29284]: cluster 2024-03-11T21:17:29.454717+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:31.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:30 smithi203 bash[23654]: cluster 2024-03-11T21:17:29.454717+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:32.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[21512]: audit 2024-03-11T21:17:31.024912+0000 mgr.y (mgr.14152) 148 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:32.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[21512]: audit 2024-03-11T21:17:31.031692+0000 mon.a (mon.0) 333 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:32.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[21512]: audit 2024-03-11T21:17:31.037251+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:32.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[21512]: audit 2024-03-11T21:17:31.038358+0000 mon.a (mon.0) 335 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:32.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[29284]: audit 2024-03-11T21:17:31.024912+0000 mgr.y (mgr.14152) 148 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:32.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[29284]: audit 2024-03-11T21:17:31.031692+0000 mon.a (mon.0) 333 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:32.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[29284]: audit 2024-03-11T21:17:31.037251+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:32.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:31 smithi143 bash[29284]: audit 2024-03-11T21:17:31.038358+0000 mon.a (mon.0) 335 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:32.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:31 smithi203 bash[23654]: audit 2024-03-11T21:17:31.024912+0000 mgr.y (mgr.14152) 148 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:32.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:31 smithi203 bash[23654]: audit 2024-03-11T21:17:31.031692+0000 mon.a (mon.0) 333 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:32.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:31 smithi203 bash[23654]: audit 2024-03-11T21:17:31.037251+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:32.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:31 smithi203 bash[23654]: audit 2024-03-11T21:17:31.038358+0000 mon.a (mon.0) 335 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:32 smithi143 bash[21512]: cluster 2024-03-11T21:17:31.455178+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:32 smithi143 bash[29284]: cluster 2024-03-11T21:17:31.455178+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:33.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:32 smithi203 bash[23654]: cluster 2024-03-11T21:17:31.455178+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:35.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:34 smithi143 bash[21512]: cluster 2024-03-11T21:17:33.455717+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:35.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:34 smithi143 bash[29284]: cluster 2024-03-11T21:17:33.455717+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:35.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:34 smithi203 bash[23654]: cluster 2024-03-11T21:17:33.455717+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:35.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:35.803+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:17:37.159 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:36 smithi143 bash[29284]: cluster 2024-03-11T21:17:35.456311+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:37.160 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:36 smithi143 bash[21512]: cluster 2024-03-11T21:17:35.456311+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:37.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:36 smithi203 bash[23654]: cluster 2024-03-11T21:17:35.456311+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:39.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:38 smithi143 bash[21512]: cluster 2024-03-11T21:17:37.456804+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:39.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:38 smithi143 bash[29284]: cluster 2024-03-11T21:17:37.456804+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:39.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:38 smithi203 bash[23654]: cluster 2024-03-11T21:17:37.456804+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:40.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[29284]: audit 2024-03-11T21:17:39.416913+0000 mon.a (mon.0) 336 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]: dispatch 2024-03-11T21:17:40.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[29284]: audit 2024-03-11T21:17:39.419716+0000 mon.a (mon.0) 337 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]': finished 2024-03-11T21:17:40.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[29284]: cluster 2024-03-11T21:17:39.423078+0000 mon.a (mon.0) 338 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:17:40.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[29284]: audit 2024-03-11T21:17:39.423369+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:40.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[21512]: audit 2024-03-11T21:17:39.416913+0000 mon.a (mon.0) 336 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]: dispatch 2024-03-11T21:17:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[21512]: audit 2024-03-11T21:17:39.419716+0000 mon.a (mon.0) 337 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]': finished 2024-03-11T21:17:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[21512]: cluster 2024-03-11T21:17:39.423078+0000 mon.a (mon.0) 338 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:17:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:39 smithi143 bash[21512]: audit 2024-03-11T21:17:39.423369+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:40.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:39 smithi203 bash[23654]: audit 2024-03-11T21:17:39.416913+0000 mon.a (mon.0) 336 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]: dispatch 2024-03-11T21:17:40.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:39 smithi203 bash[23654]: audit 2024-03-11T21:17:39.419716+0000 mon.a (mon.0) 337 : audit 1 from='client.? 172.21.15.143:0/2086960836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d"}]': finished 2024-03-11T21:17:40.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:39 smithi203 bash[23654]: cluster 2024-03-11T21:17:39.423078+0000 mon.a (mon.0) 338 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:17:40.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:39 smithi203 bash[23654]: audit 2024-03-11T21:17:39.423369+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:40.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:40.807+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:17:41.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:40 smithi143 bash[21512]: cluster 2024-03-11T21:17:39.457227+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:41.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:40 smithi143 bash[21512]: audit 2024-03-11T21:17:40.150236+0000 mon.c (mon.2) 7 : audit 0 from='client.? 172.21.15.143:0/535655355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:41.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:40 smithi143 bash[29284]: cluster 2024-03-11T21:17:39.457227+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:41.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:40 smithi143 bash[29284]: audit 2024-03-11T21:17:40.150236+0000 mon.c (mon.2) 7 : audit 0 from='client.? 172.21.15.143:0/535655355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:41.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:40 smithi203 bash[23654]: cluster 2024-03-11T21:17:39.457227+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:41.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:40 smithi203 bash[23654]: audit 2024-03-11T21:17:40.150236+0000 mon.c (mon.2) 7 : audit 0 from='client.? 172.21.15.143:0/535655355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:43.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:42 smithi143 bash[21512]: cluster 2024-03-11T21:17:41.457648+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:43.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:42 smithi143 bash[29284]: cluster 2024-03-11T21:17:41.457648+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:43.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:42 smithi203 bash[23654]: cluster 2024-03-11T21:17:41.457648+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:45.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:44 smithi143 bash[21512]: cluster 2024-03-11T21:17:43.458011+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:45.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:44 smithi143 bash[29284]: cluster 2024-03-11T21:17:43.458011+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:45.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:44 smithi203 bash[23654]: cluster 2024-03-11T21:17:43.458011+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:45.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:45.811+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[21512]: cluster 2024-03-11T21:17:45.458469+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[21512]: cluster 2024-03-11T21:17:46.481315+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[21512]: cluster 2024-03-11T21:17:46.481417+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[29284]: cluster 2024-03-11T21:17:45.458469+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[29284]: cluster 2024-03-11T21:17:46.481315+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:47.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:46 smithi143 bash[29284]: cluster 2024-03-11T21:17:46.481417+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:47.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:46 smithi203 bash[23654]: cluster 2024-03-11T21:17:45.458469+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:47.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:46 smithi203 bash[23654]: cluster 2024-03-11T21:17:46.481315+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:47.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:46 smithi203 bash[23654]: cluster 2024-03-11T21:17:46.481417+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:48.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:47 smithi143 bash[21512]: cluster 2024-03-11T21:17:46.916669+0000 mon.a (mon.0) 340 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:17:48.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:47 smithi143 bash[21512]: audit 2024-03-11T21:17:46.916993+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:48.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:47 smithi143 bash[29284]: cluster 2024-03-11T21:17:46.916669+0000 mon.a (mon.0) 340 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:17:48.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:47 smithi143 bash[29284]: audit 2024-03-11T21:17:46.916993+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:48.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:47 smithi203 bash[23654]: cluster 2024-03-11T21:17:46.916669+0000 mon.a (mon.0) 340 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:17:48.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:47 smithi203 bash[23654]: audit 2024-03-11T21:17:46.916993+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:17:49.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:48 smithi143 bash[21512]: cluster 2024-03-11T21:17:47.458908+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:49.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:48 smithi143 bash[29284]: cluster 2024-03-11T21:17:47.458908+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:49.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:48 smithi203 bash[23654]: cluster 2024-03-11T21:17:47.458908+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:50.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:50.815+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:17:51.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:50 smithi143 bash[21512]: cluster 2024-03-11T21:17:49.459317+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:51.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:50 smithi143 bash[29284]: cluster 2024-03-11T21:17:49.459317+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:51.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:50 smithi203 bash[23654]: cluster 2024-03-11T21:17:49.459317+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[29284]: cluster 2024-03-11T21:17:51.459572+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[29284]: audit 2024-03-11T21:17:52.096201+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[29284]: audit 2024-03-11T21:17:52.097658+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[29284]: cephadm 2024-03-11T21:17:52.099041+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi143 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[21512]: cluster 2024-03-11T21:17:51.459572+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[21512]: audit 2024-03-11T21:17:52.096201+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:17:53.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[21512]: audit 2024-03-11T21:17:52.097658+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:53.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:52 smithi143 bash[21512]: cephadm 2024-03-11T21:17:52.099041+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi143 2024-03-11T21:17:53.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:52 smithi203 bash[23654]: cluster 2024-03-11T21:17:51.459572+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:53.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:52 smithi203 bash[23654]: audit 2024-03-11T21:17:52.096201+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:17:53.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:52 smithi203 bash[23654]: audit 2024-03-11T21:17:52.097658+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:53.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:52 smithi203 bash[23654]: cephadm 2024-03-11T21:17:52.099041+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi143 2024-03-11T21:17:55.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:54 smithi143 bash[21512]: cluster 2024-03-11T21:17:53.459979+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:55.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:54 smithi143 bash[29284]: cluster 2024-03-11T21:17:53.459979+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:55.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:54 smithi203 bash[23654]: cluster 2024-03-11T21:17:53.459979+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:55.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:17:55.819+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:17:56.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.686 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.687 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.945 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.945 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.946 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:56 smithi143 bash[21512]: cluster 2024-03-11T21:17:55.460473+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:56.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:56 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:17:56.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:56 smithi143 bash[29284]: cluster 2024-03-11T21:17:55.460473+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:57.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:56 smithi203 bash[23654]: cluster 2024-03-11T21:17:55.460473+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:58.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[21512]: audit 2024-03-11T21:17:56.980662+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[21512]: audit 2024-03-11T21:17:56.987256+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[21512]: audit 2024-03-11T21:17:57.013389+0000 mon.a (mon.0) 346 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:58.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[29284]: audit 2024-03-11T21:17:56.980662+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[29284]: audit 2024-03-11T21:17:56.987256+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:57 smithi143 bash[29284]: audit 2024-03-11T21:17:57.013389+0000 mon.a (mon.0) 346 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:58.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:57 smithi203 bash[23654]: audit 2024-03-11T21:17:56.980662+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:57 smithi203 bash[23654]: audit 2024-03-11T21:17:56.987256+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:17:58.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:57 smithi203 bash[23654]: audit 2024-03-11T21:17:57.013389+0000 mon.a (mon.0) 346 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:59.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:17:58 smithi203 bash[23654]: cluster 2024-03-11T21:17:57.460889+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:59.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:17:58 smithi143 bash[29284]: cluster 2024-03-11T21:17:57.460889+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:17:59.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:17:58 smithi143 bash[21512]: cluster 2024-03-11T21:17:57.460889+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:00.823+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:18:01.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:00 smithi203 bash[23654]: cluster 2024-03-11T21:17:59.461378+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:01.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:00 smithi143 bash[21512]: cluster 2024-03-11T21:17:59.461378+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:01.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:00 smithi143 bash[29284]: cluster 2024-03-11T21:17:59.461378+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:03.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: cluster 2024-03-11T21:18:01.461757+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: audit 2024-03-11T21:18:02.295251+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: audit 2024-03-11T21:18:02.301770+0000 mon.a (mon.0) 348 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: audit 2024-03-11T21:18:02.872914+0000 mon.a (mon.0) 349 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: audit 2024-03-11T21:18:02.874830+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[21512]: audit 2024-03-11T21:18:02.883721+0000 mon.a (mon.0) 351 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: cluster 2024-03-11T21:18:01.461757+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: audit 2024-03-11T21:18:02.295251+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: audit 2024-03-11T21:18:02.301770+0000 mon.a (mon.0) 348 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: audit 2024-03-11T21:18:02.872914+0000 mon.a (mon.0) 349 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: audit 2024-03-11T21:18:02.874830+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:03.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:03 smithi143 bash[29284]: audit 2024-03-11T21:18:02.883721+0000 mon.a (mon.0) 351 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: cluster 2024-03-11T21:18:01.461757+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: audit 2024-03-11T21:18:02.295251+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: audit 2024-03-11T21:18:02.301770+0000 mon.a (mon.0) 348 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: audit 2024-03-11T21:18:02.872914+0000 mon.a (mon.0) 349 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: audit 2024-03-11T21:18:02.874830+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:03.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:03 smithi203 bash[23654]: audit 2024-03-11T21:18:02.883721+0000 mon.a (mon.0) 351 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:04.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:04 smithi143 bash[21512]: audit 2024-03-11T21:18:03.586560+0000 mon.a (mon.0) 352 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:04.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:04 smithi143 bash[29284]: audit 2024-03-11T21:18:03.586560+0000 mon.a (mon.0) 352 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:04.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:04 smithi203 bash[23654]: audit 2024-03-11T21:18:03.586560+0000 mon.a (mon.0) 352 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[29284]: cluster 2024-03-11T21:18:03.462193+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[29284]: audit 2024-03-11T21:18:04.306518+0000 mon.a (mon.0) 353 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[29284]: cluster 2024-03-11T21:18:04.311087+0000 mon.a (mon.0) 354 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[29284]: audit 2024-03-11T21:18:04.311867+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[29284]: audit 2024-03-11T21:18:04.312747+0000 mon.a (mon.0) 356 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:05.698 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[21512]: cluster 2024-03-11T21:18:03.462193+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:05.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[21512]: audit 2024-03-11T21:18:04.306518+0000 mon.a (mon.0) 353 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:05.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[21512]: cluster 2024-03-11T21:18:04.311087+0000 mon.a (mon.0) 354 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:05.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[21512]: audit 2024-03-11T21:18:04.311867+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:18:05.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:05 smithi143 bash[21512]: audit 2024-03-11T21:18:04.312747+0000 mon.a (mon.0) 356 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:05.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:05 smithi203 bash[23654]: cluster 2024-03-11T21:18:03.462193+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:05.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:05 smithi203 bash[23654]: audit 2024-03-11T21:18:04.306518+0000 mon.a (mon.0) 353 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:05.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:05 smithi203 bash[23654]: cluster 2024-03-11T21:18:04.311087+0000 mon.a (mon.0) 354 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:05.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:05 smithi203 bash[23654]: audit 2024-03-11T21:18:04.311867+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:18:05.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:05 smithi203 bash[23654]: audit 2024-03-11T21:18:04.312747+0000 mon.a (mon.0) 356 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:05.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:05.827+0000 7fe94a593700 10 client.14262.objecter tick 2024-03-11T21:18:06.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.087+0000 7fe941ffb700 1 -- 172.21.15.143:0/752225695 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe94402db60 con 0x7fe93006f690 2024-03-11T21:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 2 on host 'smithi143' 2024-03-11T21:18:06.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 10 librados: watch_flush enter 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 10 librados: watch_flush exit 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 20 client.14262.objecter shutdown clearing up homeless session... 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 10 client.14262.objecter successfully canceled tick 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 msgr2=0x7fe930071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 0x7fe930071b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098c0 tx=0x7fe92c019040 comp rx=0 tx=0).stop 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 msgr2=0x7fe944104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe93c007f10 tx=0x7fe93c00d3b0 comp rx=0 tx=0).stop 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 shutdown_connections 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fe93006f690 0x7fe930071b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:06.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944100d60 0x7fe944104cf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe944102900 0x7fe944103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 --2- 172.21.15.143:0/752225695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe944101f60 0x7fe944103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 >> 172.21.15.143:0/752225695 conn(0x7fe9440fc300 msgr2=0x7fe94406a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 shutdown_connections 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 -- 172.21.15.143:0/752225695 wait complete. 2024-03-11T21:18:06.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:06.091+0000 7fe94ad94700 1 librados: shutdown 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:05.308149+0000 mon.a (mon.0) 357 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: cluster 2024-03-11T21:18:05.310933+0000 mon.a (mon.0) 358 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:05.311734+0000 mon.a (mon.0) 359 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:05.315810+0000 mon.a (mon.0) 360 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:05.799311+0000 mon.a (mon.0) 361 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:06.043720+0000 mon.a (mon.0) 362 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:06.050735+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:06.073092+0000 mon.a (mon.0) 364 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[21512]: audit 2024-03-11T21:18:06.315618+0000 mon.a (mon.0) 365 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:05.308149+0000 mon.a (mon.0) 357 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: cluster 2024-03-11T21:18:05.310933+0000 mon.a (mon.0) 358 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:05.311734+0000 mon.a (mon.0) 359 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:05.315810+0000 mon.a (mon.0) 360 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:05.799311+0000 mon.a (mon.0) 361 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:06.043720+0000 mon.a (mon.0) 362 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:06.050735+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:06.073092+0000 mon.a (mon.0) 364 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:06.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:06 smithi143 bash[29284]: audit 2024-03-11T21:18:06.315618+0000 mon.a (mon.0) 365 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:05.308149+0000 mon.a (mon.0) 357 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: cluster 2024-03-11T21:18:05.310933+0000 mon.a (mon.0) 358 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:05.311734+0000 mon.a (mon.0) 359 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:05.315810+0000 mon.a (mon.0) 360 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:05.799311+0000 mon.a (mon.0) 361 : audit 1 from='osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625]' entity='osd.2' 2024-03-11T21:18:06.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:06.043720+0000 mon.a (mon.0) 362 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:06.050735+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:06.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:06.073092+0000 mon.a (mon.0) 364 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:06.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:06 smithi203 bash[23654]: audit 2024-03-11T21:18:06.315618+0000 mon.a (mon.0) 365 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:06.908 DEBUG:teuthology.orchestra.run.smithi143:osd.2> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:18:06.911 INFO:tasks.cephadm:Deploying osd.3 on smithi143 with /dev/vg_nvme/lv_1... 2024-03-11T21:18:06.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T21:18:07.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: cluster 2024-03-11T21:18:04.587848+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: cluster 2024-03-11T21:18:04.587941+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: cluster 2024-03-11T21:18:05.462740+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: cluster 2024-03-11T21:18:06.804033+0000 mon.a (mon.0) 366 : cluster 1 osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] boot 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: cluster 2024-03-11T21:18:06.804082+0000 mon.a (mon.0) 367 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[21512]: audit 2024-03-11T21:18:06.804545+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: cluster 2024-03-11T21:18:04.587848+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:07.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: cluster 2024-03-11T21:18:04.587941+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:07.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: cluster 2024-03-11T21:18:05.462740+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:07.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: cluster 2024-03-11T21:18:06.804033+0000 mon.a (mon.0) 366 : cluster 1 osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] boot 2024-03-11T21:18:07.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: cluster 2024-03-11T21:18:06.804082+0000 mon.a (mon.0) 367 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:07.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:07 smithi143 bash[29284]: audit 2024-03-11T21:18:06.804545+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: cluster 2024-03-11T21:18:04.587848+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: cluster 2024-03-11T21:18:04.587941+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: cluster 2024-03-11T21:18:05.462740+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: cluster 2024-03-11T21:18:06.804033+0000 mon.a (mon.0) 366 : cluster 1 osd.2 [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] boot 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: cluster 2024-03-11T21:18:06.804082+0000 mon.a (mon.0) 367 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:07 smithi203 bash[23654]: audit 2024-03-11T21:18:06.804545+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:08.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:08 smithi143 bash[21512]: audit 2024-03-11T21:18:07.598971+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:08.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:08 smithi143 bash[29284]: audit 2024-03-11T21:18:07.598971+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:08.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:08 smithi203 bash[23654]: audit 2024-03-11T21:18:07.598971+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:09.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[21512]: cluster 2024-03-11T21:18:07.463294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:09.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[21512]: audit 2024-03-11T21:18:08.355137+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[21512]: cluster 2024-03-11T21:18:08.361678+0000 mon.a (mon.0) 371 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[21512]: audit 2024-03-11T21:18:08.364043+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[29284]: cluster 2024-03-11T21:18:07.463294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[29284]: audit 2024-03-11T21:18:08.355137+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[29284]: cluster 2024-03-11T21:18:08.361678+0000 mon.a (mon.0) 371 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:09.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:09 smithi143 bash[29284]: audit 2024-03-11T21:18:08.364043+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:09.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:09 smithi203 bash[23654]: cluster 2024-03-11T21:18:07.463294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:09.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:09 smithi203 bash[23654]: audit 2024-03-11T21:18:08.355137+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:09.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:09 smithi203 bash[23654]: cluster 2024-03-11T21:18:08.361678+0000 mon.a (mon.0) 371 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:09.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:09 smithi203 bash[23654]: audit 2024-03-11T21:18:08.364043+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:10.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:10 smithi143 bash[29284]: audit 2024-03-11T21:18:09.357775+0000 mon.a (mon.0) 373 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:10.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:10 smithi143 bash[29284]: cluster 2024-03-11T21:18:09.364542+0000 mon.a (mon.0) 374 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:10.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:10 smithi143 bash[21512]: audit 2024-03-11T21:18:09.357775+0000 mon.a (mon.0) 373 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:10.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:10 smithi143 bash[21512]: cluster 2024-03-11T21:18:09.364542+0000 mon.a (mon.0) 374 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:10.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:10 smithi203 bash[23654]: audit 2024-03-11T21:18:09.357775+0000 mon.a (mon.0) 373 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:10.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:10 smithi203 bash[23654]: cluster 2024-03-11T21:18:09.364542+0000 mon.a (mon.0) 374 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:11.034 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:11.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[21512]: cluster 2024-03-11T21:18:09.463779+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:11.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[21512]: cluster 2024-03-11T21:18:10.389573+0000 mon.a (mon.0) 375 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:11.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[21512]: audit 2024-03-11T21:18:10.697966+0000 mon.a (mon.0) 376 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:11.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[29284]: cluster 2024-03-11T21:18:09.463779+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:11.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[29284]: cluster 2024-03-11T21:18:10.389573+0000 mon.a (mon.0) 375 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:11.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:11 smithi143 bash[29284]: audit 2024-03-11T21:18:10.697966+0000 mon.a (mon.0) 376 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:11.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:11 smithi203 bash[23654]: cluster 2024-03-11T21:18:09.463779+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:11.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:11 smithi203 bash[23654]: cluster 2024-03-11T21:18:10.389573+0000 mon.a (mon.0) 375 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:11.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:11 smithi203 bash[23654]: audit 2024-03-11T21:18:10.697966+0000 mon.a (mon.0) 376 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:14.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: cluster 2024-03-11T21:18:11.464230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:14.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: cluster 2024-03-11T21:18:12.405442+0000 mon.a (mon.0) 377 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T21:18:14.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:12.457830+0000 mon.a (mon.0) 378 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.349505+0000 mon.a (mon.0) 379 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.350314+0000 mon.a (mon.0) 380 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.350844+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.351078+0000 mon.a (mon.0) 382 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.354403+0000 mon.a (mon.0) 383 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.354617+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:14 smithi203 bash[23654]: audit 2024-03-11T21:18:13.354910+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:14.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: cluster 2024-03-11T21:18:11.464230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: cluster 2024-03-11T21:18:12.405442+0000 mon.a (mon.0) 377 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:12.457830+0000 mon.a (mon.0) 378 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.349505+0000 mon.a (mon.0) 379 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.350314+0000 mon.a (mon.0) 380 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.350844+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.351078+0000 mon.a (mon.0) 382 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.354403+0000 mon.a (mon.0) 383 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.354617+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[21512]: audit 2024-03-11T21:18:13.354910+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:14.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: cluster 2024-03-11T21:18:11.464230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: cluster 2024-03-11T21:18:12.405442+0000 mon.a (mon.0) 377 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:12.457830+0000 mon.a (mon.0) 378 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.349505+0000 mon.a (mon.0) 379 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.350314+0000 mon.a (mon.0) 380 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.350844+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.351078+0000 mon.a (mon.0) 382 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.354403+0000 mon.a (mon.0) 383 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.354617+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:14.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:14 smithi143 bash[29284]: audit 2024-03-11T21:18:13.354910+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:15.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:13.352300+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: cluster 2024-03-11T21:18:13.464703+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: cluster 2024-03-11T21:18:13.538140+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: cluster 2024-03-11T21:18:13.538275+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.273086+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.330348+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.333166+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.334025+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.334350+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:14.334720+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:15.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[21512]: audit 2024-03-11T21:18:15.220712+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:13.352300+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: cluster 2024-03-11T21:18:13.464703+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: cluster 2024-03-11T21:18:13.538140+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: cluster 2024-03-11T21:18:13.538275+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.273086+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.330348+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.333166+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.334025+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.334350+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:14.334720+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:15.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:15 smithi143 bash[29284]: audit 2024-03-11T21:18:15.220712+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:13.352300+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: cluster 2024-03-11T21:18:13.464703+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: cluster 2024-03-11T21:18:13.538140+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: cluster 2024-03-11T21:18:13.538275+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.273086+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.330348+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.333166+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.334025+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.334350+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:14.334720+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:15.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:15 smithi203 bash[23654]: audit 2024-03-11T21:18:15.220712+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:16.208 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T21:18:16.209 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T21:18:16.209 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-11T21:18:16.209 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-11T21:18:16.209 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157679 s, 665 MB/s 2024-03-11T21:18:16.209 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-11T21:18:16.498 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:16 smithi143 bash[21512]: cluster 2024-03-11T21:18:15.291928+0000 mon.a (mon.0) 390 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:16.499 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:16 smithi143 bash[29284]: cluster 2024-03-11T21:18:15.291928+0000 mon.a (mon.0) 390 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:16.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:16 smithi203 bash[23654]: cluster 2024-03-11T21:18:15.291928+0000 mon.a (mon.0) 390 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:17.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:17 smithi143 bash[21512]: cluster 2024-03-11T21:18:15.465404+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:17.444 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:17 smithi143 bash[29284]: cluster 2024-03-11T21:18:15.465404+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:17.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_1 2024-03-11T21:18:17.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:17 smithi203 bash[23654]: cluster 2024-03-11T21:18:15.465404+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: cluster 2024-03-11T21:18:17.465980+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: cephadm 2024-03-11T21:18:17.667180+0000 mgr.y (mgr.14152) 174 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.675628+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.684015+0000 mon.a (mon.0) 392 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.686143+0000 mon.a (mon.0) 393 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.687565+0000 mon.a (mon.0) 394 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.688925+0000 mon.a (mon.0) 395 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: cephadm 2024-03-11T21:18:17.689866+0000 mgr.y (mgr.14152) 175 : cephadm 1 Adjusting osd_memory_target on smithi143 to 5413M 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.695112+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.698812+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:18.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.699859+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[21512]: audit 2024-03-11T21:18:17.706430+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: cluster 2024-03-11T21:18:17.465980+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: cephadm 2024-03-11T21:18:17.667180+0000 mgr.y (mgr.14152) 174 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.675628+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.684015+0000 mon.a (mon.0) 392 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.686143+0000 mon.a (mon.0) 393 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.687565+0000 mon.a (mon.0) 394 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.688925+0000 mon.a (mon.0) 395 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: cephadm 2024-03-11T21:18:17.689866+0000 mgr.y (mgr.14152) 175 : cephadm 1 Adjusting osd_memory_target on smithi143 to 5413M 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.695112+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:18.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.698812+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:18.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.699859+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:18.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:18 smithi143 bash[29284]: audit 2024-03-11T21:18:17.706430+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: cluster 2024-03-11T21:18:17.465980+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: cephadm 2024-03-11T21:18:17.667180+0000 mgr.y (mgr.14152) 174 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.675628+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.684015+0000 mon.a (mon.0) 392 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.686143+0000 mon.a (mon.0) 393 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.687565+0000 mon.a (mon.0) 394 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.688925+0000 mon.a (mon.0) 395 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: cephadm 2024-03-11T21:18:17.689866+0000 mgr.y (mgr.14152) 175 : cephadm 1 Adjusting osd_memory_target on smithi143 to 5413M 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.695112+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:19.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.698812+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:19.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.699859+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:19.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:18 smithi203 bash[23654]: audit 2024-03-11T21:18:17.706430+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:20.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:20 smithi143 bash[21512]: cluster 2024-03-11T21:18:19.466576+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 188 KiB/s wr, 13 op/s 2024-03-11T21:18:20.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:20 smithi143 bash[29284]: cluster 2024-03-11T21:18:19.466576+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 188 KiB/s wr, 13 op/s 2024-03-11T21:18:21.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:20 smithi203 bash[23654]: cluster 2024-03-11T21:18:19.466576+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 188 KiB/s wr, 13 op/s 2024-03-11T21:18:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:22 smithi143 bash[21512]: cluster 2024-03-11T21:18:21.467162+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 9 op/s 2024-03-11T21:18:22.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:22 smithi143 bash[29284]: cluster 2024-03-11T21:18:21.467162+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 9 op/s 2024-03-11T21:18:23.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:22 smithi203 bash[23654]: cluster 2024-03-11T21:18:21.467162+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 9 op/s 2024-03-11T21:18:24.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:24 smithi143 bash[21512]: cluster 2024-03-11T21:18:23.467713+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-03-11T21:18:24.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:24 smithi143 bash[29284]: cluster 2024-03-11T21:18:23.467713+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-03-11T21:18:25.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:24 smithi203 bash[23654]: cluster 2024-03-11T21:18:23.467713+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 -- 172.21.15.143:0/1311590939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 msgr2=0x7f751c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 --2- 172.21.15.143:0/1311590939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c1056f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7504009990 tx=0x7f7504009ca0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 -- 172.21.15.143:0/1311590939 shutdown_connections 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 --2- 172.21.15.143:0/1311590939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c105c30 0x7f751c10c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 --2- 172.21.15.143:0/1311590939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c104090 0x7f751c1044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 --2- 172.21.15.143:0/1311590939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c1056f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 -- 172.21.15.143:0/1311590939 >> 172.21.15.143:0/1311590939 conn(0x7f751c0ff630 msgr2=0x7f751c101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:18:25.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 -- 172.21.15.143:0/1311590939 shutdown_connections 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 -- 172.21.15.143:0/1311590939 wait complete. 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 librados: starting msgr at 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 librados: starting objecter 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.351+0000 7f75228c1700 1 Processor -- start 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- start start 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 librados: setting wanted keys 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 librados: calling monclient init 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 0x7f751c1a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c10f220 con 0x7f751c104090 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:18:25.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c10f0a0 con 0x7f751c105c30 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c10f3a0 con 0x7f751c105290 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34836/0 (socket says 172.21.15.143:34836) 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 -- 172.21.15.143:0/1588112755 learned_addr learned my addr 172.21.15.143:0/1588112755 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48730/0 (socket says 172.21.15.143:48730) 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f7520e5e700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 0x7f751c1a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 -- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 msgr2=0x7f751c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 -- 172.21.15.143:0/1588112755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 msgr2=0x7f751c1a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 0x7f751c1a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 -- 172.21.15.143:0/1588112755 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c1a1540 con 0x7f751c104090 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f7520e5e700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 0x7f751c1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:18:25.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751bfff700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7514009f20 tx=0x7f751400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:18:25.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 10 client.?.objecter ms_handle_connect 0x7f751c104090 2024-03-11T21:18:25.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514003a00 con 0x7f751c104090 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7514003ba0 con 0x7f751c104090 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c1a1830 con 0x7f751c104090 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 10 client.14298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140046d0 con 0x7f751c104090 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c1a1d70 con 0x7f751c104090 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 librados: init done 2024-03-11T21:18:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c04faa0 con 0x7f751c104090 2024-03-11T21:18:25.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f7514012050 con 0x7f751c104090 2024-03-11T21:18:25.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 0x7f7508071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3017+0+0 (secure 0 0 0) 0x7f751408c9b0 con 0x7f751c104090 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 10 client.14298.objecter ms_dispatch 0x7f751c001550 osd_map(23..23 src has 1..23) v4 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 3 client.14298.objecter handle_osd_map got epochs [23,23] > 0 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 3 client.14298.objecter handle_osd_map decoding full epoch 23 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f751b7fe700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 0x7f7508071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.355+0000 7f75197fa700 20 client.14298.objecter dump_active .. 0 homeless 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.359+0000 7f751b7fe700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 0x7f7508071ac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f750400b860 tx=0x7f750400b680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:18:25.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.359+0000 7f75197fa700 10 client.14298.objecter ms_handle_connect 0x7f750806f600 2024-03-11T21:18:25.346 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.363+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7514010040 con 0x7f751c104090 2024-03-11T21:18:25.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:25.575+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f751c02dca0 con 0x7f750806f600 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[29284]: audit 2024-03-11T21:18:25.587827+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[29284]: audit 2024-03-11T21:18:25.591107+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[29284]: audit 2024-03-11T21:18:25.591790+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[21512]: audit 2024-03-11T21:18:25.587827+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[21512]: audit 2024-03-11T21:18:25.591107+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:25.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:25 smithi143 bash[21512]: audit 2024-03-11T21:18:25.591790+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:26.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:25 smithi203 bash[23654]: audit 2024-03-11T21:18:25.587827+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:26.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:25 smithi203 bash[23654]: audit 2024-03-11T21:18:25.591107+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:26.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:25 smithi203 bash[23654]: audit 2024-03-11T21:18:25.591790+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:26.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:26 smithi143 bash[21512]: cluster 2024-03-11T21:18:25.468279+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-03-11T21:18:26.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:26 smithi143 bash[21512]: audit 2024-03-11T21:18:25.581083+0000 mgr.y (mgr.14152) 180 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:26.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:26 smithi143 bash[29284]: cluster 2024-03-11T21:18:25.468279+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-03-11T21:18:26.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:26 smithi143 bash[29284]: audit 2024-03-11T21:18:25.581083+0000 mgr.y (mgr.14152) 180 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:27.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:26 smithi203 bash[23654]: cluster 2024-03-11T21:18:25.468279+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-03-11T21:18:27.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:26 smithi203 bash[23654]: audit 2024-03-11T21:18:25.581083+0000 mgr.y (mgr.14152) 180 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:28.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:28 smithi143 bash[21512]: cluster 2024-03-11T21:18:27.468813+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:28.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:28 smithi143 bash[29284]: cluster 2024-03-11T21:18:27.468813+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:29.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:28 smithi203 bash[23654]: cluster 2024-03-11T21:18:27.468813+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:30.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:30.359+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:30.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:30 smithi143 bash[21512]: cluster 2024-03-11T21:18:29.469413+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:30 smithi143 bash[29284]: cluster 2024-03-11T21:18:29.469413+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:31.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:30 smithi203 bash[23654]: cluster 2024-03-11T21:18:29.469413+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:18:32.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:32 smithi143 bash[21512]: cluster 2024-03-11T21:18:31.469946+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:32.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:32 smithi143 bash[29284]: cluster 2024-03-11T21:18:31.469946+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:33.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:32 smithi203 bash[23654]: cluster 2024-03-11T21:18:31.469946+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: cluster 2024-03-11T21:18:33.470360+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: audit 2024-03-11T21:18:34.312682+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.143:0/2597331954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: audit 2024-03-11T21:18:34.313215+0000 mon.a (mon.0) 403 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: audit 2024-03-11T21:18:34.315457+0000 mon.a (mon.0) 404 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]': finished 2024-03-11T21:18:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: cluster 2024-03-11T21:18:34.318731+0000 mon.a (mon.0) 405 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[21512]: audit 2024-03-11T21:18:34.318989+0000 mon.a (mon.0) 406 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: cluster 2024-03-11T21:18:33.470360+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: audit 2024-03-11T21:18:34.312682+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.143:0/2597331954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: audit 2024-03-11T21:18:34.313215+0000 mon.a (mon.0) 403 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: audit 2024-03-11T21:18:34.315457+0000 mon.a (mon.0) 404 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]': finished 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: cluster 2024-03-11T21:18:34.318731+0000 mon.a (mon.0) 405 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:18:34.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:34 smithi143 bash[29284]: audit 2024-03-11T21:18:34.318989+0000 mon.a (mon.0) 406 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:18:35.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: cluster 2024-03-11T21:18:33.470360+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:35.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: audit 2024-03-11T21:18:34.312682+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.143:0/2597331954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:35.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: audit 2024-03-11T21:18:34.313215+0000 mon.a (mon.0) 403 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]: dispatch 2024-03-11T21:18:35.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: audit 2024-03-11T21:18:34.315457+0000 mon.a (mon.0) 404 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc"}]': finished 2024-03-11T21:18:35.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: cluster 2024-03-11T21:18:34.318731+0000 mon.a (mon.0) 405 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:18:35.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:34 smithi203 bash[23654]: audit 2024-03-11T21:18:34.318989+0000 mon.a (mon.0) 406 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:18:35.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:35.363+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:36.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:35 smithi203 bash[23654]: audit 2024-03-11T21:18:35.066889+0000 mon.a (mon.0) 407 : audit 0 from='client.? 172.21.15.143:0/740417009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:36.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:35 smithi143 bash[21512]: audit 2024-03-11T21:18:35.066889+0000 mon.a (mon.0) 407 : audit 0 from='client.? 172.21.15.143:0/740417009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:36.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:35 smithi143 bash[29284]: audit 2024-03-11T21:18:35.066889+0000 mon.a (mon.0) 407 : audit 0 from='client.? 172.21.15.143:0/740417009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:37.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:36 smithi203 bash[23654]: cluster 2024-03-11T21:18:35.470856+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:37.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:36 smithi143 bash[21512]: cluster 2024-03-11T21:18:35.470856+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:37.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:36 smithi143 bash[29284]: cluster 2024-03-11T21:18:35.470856+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:38 smithi203 bash[23654]: cluster 2024-03-11T21:18:37.471354+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:38 smithi143 bash[21512]: cluster 2024-03-11T21:18:37.471354+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:38 smithi143 bash[29284]: cluster 2024-03-11T21:18:37.471354+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:40.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:40.367+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:41.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:40 smithi203 bash[23654]: cluster 2024-03-11T21:18:39.471895+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:41.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:40 smithi143 bash[21512]: cluster 2024-03-11T21:18:39.471895+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:41.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:40 smithi143 bash[29284]: cluster 2024-03-11T21:18:39.471895+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:43.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:42 smithi203 bash[23654]: cluster 2024-03-11T21:18:41.472361+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:43.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:42 smithi143 bash[21512]: cluster 2024-03-11T21:18:41.472361+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:43.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:42 smithi143 bash[29284]: cluster 2024-03-11T21:18:41.472361+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:44.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:44 smithi143 bash[29284]: cluster 2024-03-11T21:18:43.472743+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:44.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:44 smithi143 bash[21512]: cluster 2024-03-11T21:18:43.472743+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:45.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:44 smithi203 bash[23654]: cluster 2024-03-11T21:18:43.472743+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:45.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:45.371+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:46.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:46 smithi143 bash[29284]: cluster 2024-03-11T21:18:45.473283+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:46.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:46 smithi143 bash[21512]: cluster 2024-03-11T21:18:45.473283+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:47.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:46 smithi203 bash[23654]: cluster 2024-03-11T21:18:45.473283+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:48.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:47 smithi203 bash[23654]: audit 2024-03-11T21:18:47.403524+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:18:48.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:47 smithi203 bash[23654]: audit 2024-03-11T21:18:47.405345+0000 mon.a (mon.0) 409 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:48.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:47 smithi143 bash[21512]: audit 2024-03-11T21:18:47.403524+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:18:48.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:47 smithi143 bash[21512]: audit 2024-03-11T21:18:47.405345+0000 mon.a (mon.0) 409 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:48.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:47 smithi143 bash[29284]: audit 2024-03-11T21:18:47.403524+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:18:48.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:47 smithi143 bash[29284]: audit 2024-03-11T21:18:47.405345+0000 mon.a (mon.0) 409 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:49.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:48 smithi203 bash[23654]: cephadm 2024-03-11T21:18:47.406999+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi143 2024-03-11T21:18:49.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:48 smithi203 bash[23654]: cluster 2024-03-11T21:18:47.473828+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:49.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:48 smithi143 bash[21512]: cephadm 2024-03-11T21:18:47.406999+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi143 2024-03-11T21:18:49.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:48 smithi143 bash[21512]: cluster 2024-03-11T21:18:47.473828+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:49.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:48 smithi143 bash[29284]: cephadm 2024-03-11T21:18:47.406999+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi143 2024-03-11T21:18:49.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:48 smithi143 bash[29284]: cluster 2024-03-11T21:18:47.473828+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:50.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:50.375+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:50.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:50 smithi143 bash[21512]: cluster 2024-03-11T21:18:49.474382+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:50.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:50 smithi143 bash[29284]: cluster 2024-03-11T21:18:49.474382+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:51.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:50 smithi203 bash[23654]: cluster 2024-03-11T21:18:49.474382+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:52.138 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.138 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.139 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.139 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.139 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.413 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.414 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.415 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.415 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:52.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:18:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:52 smithi203 bash[23654]: cluster 2024-03-11T21:18:51.474906+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:52 smithi203 bash[23654]: audit 2024-03-11T21:18:52.430463+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:52 smithi203 bash[23654]: audit 2024-03-11T21:18:52.438927+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:52 smithi203 bash[23654]: audit 2024-03-11T21:18:52.474234+0000 mon.a (mon.0) 412 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:53.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[21512]: cluster 2024-03-11T21:18:51.474906+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[21512]: audit 2024-03-11T21:18:52.430463+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[21512]: audit 2024-03-11T21:18:52.438927+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[21512]: audit 2024-03-11T21:18:52.474234+0000 mon.a (mon.0) 412 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[29284]: cluster 2024-03-11T21:18:51.474906+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[29284]: audit 2024-03-11T21:18:52.430463+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[29284]: audit 2024-03-11T21:18:52.438927+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:53.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:52 smithi143 bash[29284]: audit 2024-03-11T21:18:52.474234+0000 mon.a (mon.0) 412 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:55.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:54 smithi203 bash[23654]: cluster 2024-03-11T21:18:53.475246+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:55.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:54 smithi143 bash[21512]: cluster 2024-03-11T21:18:53.475246+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:55.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:54 smithi143 bash[29284]: cluster 2024-03-11T21:18:53.475246+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:55.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:18:55.379+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:18:57.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:56 smithi203 bash[23654]: cluster 2024-03-11T21:18:55.475816+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:57.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:56 smithi143 bash[21512]: cluster 2024-03-11T21:18:55.475816+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:56 smithi143 bash[29284]: cluster 2024-03-11T21:18:55.475816+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:59.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: cluster 2024-03-11T21:18:57.476415+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:59.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: audit 2024-03-11T21:18:57.808610+0000 mon.a (mon.0) 413 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: audit 2024-03-11T21:18:57.813680+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: audit 2024-03-11T21:18:57.814686+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: audit 2024-03-11T21:18:57.815510+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[21512]: audit 2024-03-11T21:18:57.820133+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: cluster 2024-03-11T21:18:57.476415+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: audit 2024-03-11T21:18:57.808610+0000 mon.a (mon.0) 413 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: audit 2024-03-11T21:18:57.813680+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: audit 2024-03-11T21:18:57.814686+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: audit 2024-03-11T21:18:57.815510+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:59.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:58 smithi143 bash[29284]: audit 2024-03-11T21:18:57.820133+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: cluster 2024-03-11T21:18:57.476415+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: audit 2024-03-11T21:18:57.808610+0000 mon.a (mon.0) 413 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: audit 2024-03-11T21:18:57.813680+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: audit 2024-03-11T21:18:57.814686+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: audit 2024-03-11T21:18:57.815510+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:59.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:58 smithi203 bash[23654]: audit 2024-03-11T21:18:57.820133+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:00.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:18:59 smithi143 bash[21512]: audit 2024-03-11T21:18:59.279719+0000 mon.a (mon.0) 418 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:19:00.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:18:59 smithi143 bash[29284]: audit 2024-03-11T21:18:59.279719+0000 mon.a (mon.0) 418 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:19:00.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:18:59 smithi203 bash[23654]: audit 2024-03-11T21:18:59.279719+0000 mon.a (mon.0) 418 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:19:00.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:00.383+0000 7f75220c0700 10 client.14298.objecter tick 2024-03-11T21:19:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[21512]: cluster 2024-03-11T21:18:59.476914+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[21512]: audit 2024-03-11T21:18:59.820021+0000 mon.a (mon.0) 419 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:19:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[21512]: cluster 2024-03-11T21:18:59.825420+0000 mon.a (mon.0) 420 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:19:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[21512]: audit 2024-03-11T21:18:59.825776+0000 mon.a (mon.0) 421 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:19:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[21512]: audit 2024-03-11T21:18:59.826164+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:01.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[29284]: cluster 2024-03-11T21:18:59.476914+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:01.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[29284]: audit 2024-03-11T21:18:59.820021+0000 mon.a (mon.0) 419 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:19:01.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[29284]: cluster 2024-03-11T21:18:59.825420+0000 mon.a (mon.0) 420 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:19:01.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[29284]: audit 2024-03-11T21:18:59.825776+0000 mon.a (mon.0) 421 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:19:01.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:00 smithi143 bash[29284]: audit 2024-03-11T21:18:59.826164+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:00 smithi203 bash[23654]: cluster 2024-03-11T21:18:59.476914+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:00 smithi203 bash[23654]: audit 2024-03-11T21:18:59.820021+0000 mon.a (mon.0) 419 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:19:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:00 smithi203 bash[23654]: cluster 2024-03-11T21:18:59.825420+0000 mon.a (mon.0) 420 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:19:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:00 smithi203 bash[23654]: audit 2024-03-11T21:18:59.825776+0000 mon.a (mon.0) 421 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:19:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:00 smithi203 bash[23654]: audit 2024-03-11T21:18:59.826164+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.843+0000 7f75197fa700 1 -- 172.21.15.143:0/1588112755 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f751c02dca0 con 0x7f750806f600 2024-03-11T21:19:01.827 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 3 on host 'smithi143' 2024-03-11T21:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 10 librados: watch_flush enter 2024-03-11T21:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 10 librados: watch_flush exit 2024-03-11T21:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 20 client.14298.objecter shutdown clearing up homeless session... 2024-03-11T21:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 10 client.14298.objecter successfully canceled tick 2024-03-11T21:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 msgr2=0x7f7508071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 0x7f7508071ac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f750400b860 tx=0x7f750400b680 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 msgr2=0x7f751c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7514009f20 tx=0x7f751400c3a0 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 shutdown_connections 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f750806f600 0x7f7508071ac0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f751c104090 0x7f751c199930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c105c30 0x7f751c1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 --2- 172.21.15.143:0/1588112755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f751c105290 0x7f751c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 >> 172.21.15.143:0/1588112755 conn(0x7f751c0ff630 msgr2=0x7f751c108700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 shutdown_connections 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 -- 172.21.15.143:0/1588112755 wait complete. 2024-03-11T21:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:19:01.847+0000 7f75228c1700 1 librados: shutdown 2024-03-11T21:19:02.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: audit 2024-03-11T21:19:00.821383+0000 mon.a (mon.0) 423 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:19:02.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: cluster 2024-03-11T21:19:00.825096+0000 mon.a (mon.0) 424 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T21:19:02.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: audit 2024-03-11T21:19:00.826123+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: audit 2024-03-11T21:19:00.832893+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: audit 2024-03-11T21:19:01.812766+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:02.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[21512]: audit 2024-03-11T21:19:01.818523+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:02.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: audit 2024-03-11T21:19:00.821383+0000 mon.a (mon.0) 423 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:19:02.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: cluster 2024-03-11T21:19:00.825096+0000 mon.a (mon.0) 424 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T21:19:02.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: audit 2024-03-11T21:19:00.826123+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: audit 2024-03-11T21:19:00.832893+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: audit 2024-03-11T21:19:01.812766+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:02.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:01 smithi143 bash[29284]: audit 2024-03-11T21:19:01.818523+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: audit 2024-03-11T21:19:00.821383+0000 mon.a (mon.0) 423 : audit 1 from='osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: cluster 2024-03-11T21:19:00.825096+0000 mon.a (mon.0) 424 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: audit 2024-03-11T21:19:00.826123+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: audit 2024-03-11T21:19:00.832893+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: audit 2024-03-11T21:19:01.812766+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:01 smithi203 bash[23654]: audit 2024-03-11T21:19:01.818523+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:02.635 DEBUG:teuthology.orchestra.run.smithi143:osd.3> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T21:19:02.638 INFO:tasks.cephadm:Deploying osd.4 on smithi203 with /dev/vg_nvme/lv_4... 2024-03-11T21:19:02.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T21:19:02.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: cluster 2024-03-11T21:19:00.255896+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:02.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: cluster 2024-03-11T21:19:00.256021+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:02.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: cluster 2024-03-11T21:19:01.477432+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: audit 2024-03-11T21:19:01.829307+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: audit 2024-03-11T21:19:01.831919+0000 mon.a (mon.0) 430 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: cluster 2024-03-11T21:19:01.840469+0000 mon.a (mon.0) 431 : cluster 1 osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] boot 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: cluster 2024-03-11T21:19:01.840585+0000 mon.a (mon.0) 432 : cluster 0 osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[29284]: audit 2024-03-11T21:19:01.841553+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: cluster 2024-03-11T21:19:00.255896+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: cluster 2024-03-11T21:19:00.256021+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: cluster 2024-03-11T21:19:01.477432+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: audit 2024-03-11T21:19:01.829307+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: audit 2024-03-11T21:19:01.831919+0000 mon.a (mon.0) 430 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: cluster 2024-03-11T21:19:01.840469+0000 mon.a (mon.0) 431 : cluster 1 osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] boot 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: cluster 2024-03-11T21:19:01.840585+0000 mon.a (mon.0) 432 : cluster 0 osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:19:02.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:02 smithi143 bash[21512]: audit 2024-03-11T21:19:01.841553+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: cluster 2024-03-11T21:19:00.255896+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: cluster 2024-03-11T21:19:00.256021+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: cluster 2024-03-11T21:19:01.477432+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: audit 2024-03-11T21:19:01.829307+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: audit 2024-03-11T21:19:01.831919+0000 mon.a (mon.0) 430 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: cluster 2024-03-11T21:19:01.840469+0000 mon.a (mon.0) 431 : cluster 1 osd.3 [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] boot 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: cluster 2024-03-11T21:19:01.840585+0000 mon.a (mon.0) 432 : cluster 0 osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:19:03.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:02 smithi203 bash[23654]: audit 2024-03-11T21:19:01.841553+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:19:04.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:03 smithi143 bash[21512]: cluster 2024-03-11T21:19:02.850877+0000 mon.a (mon.0) 434 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T21:19:04.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:03 smithi143 bash[29284]: cluster 2024-03-11T21:19:02.850877+0000 mon.a (mon.0) 434 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T21:19:04.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:03 smithi203 bash[23654]: cluster 2024-03-11T21:19:02.850877+0000 mon.a (mon.0) 434 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T21:19:05.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:04 smithi143 bash[21512]: cluster 2024-03-11T21:19:03.477957+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:05.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:04 smithi143 bash[29284]: cluster 2024-03-11T21:19:03.477957+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:05.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:04 smithi203 bash[23654]: cluster 2024-03-11T21:19:03.477957+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:06.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:05 smithi143 bash[29284]: cluster 2024-03-11T21:19:05.685445+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:06.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:05 smithi143 bash[29284]: cluster 2024-03-11T21:19:05.685562+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:06.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:05 smithi143 bash[21512]: cluster 2024-03-11T21:19:05.685445+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:06.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:05 smithi143 bash[21512]: cluster 2024-03-11T21:19:05.685562+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:06.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:05 smithi203 bash[23654]: cluster 2024-03-11T21:19:05.685445+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:06.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:05 smithi203 bash[23654]: cluster 2024-03-11T21:19:05.685562+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:07.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:06 smithi143 bash[21512]: cluster 2024-03-11T21:19:05.478521+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:07.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:06 smithi143 bash[21512]: cluster 2024-03-11T21:19:05.880463+0000 mon.a (mon.0) 435 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T21:19:07.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:06 smithi143 bash[29284]: cluster 2024-03-11T21:19:05.478521+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:07.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:06 smithi143 bash[29284]: cluster 2024-03-11T21:19:05.880463+0000 mon.a (mon.0) 435 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T21:19:07.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:06 smithi203 bash[23654]: cluster 2024-03-11T21:19:05.478521+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:07.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:06 smithi203 bash[23654]: cluster 2024-03-11T21:19:05.880463+0000 mon.a (mon.0) 435 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T21:19:07.349 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:19:09.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:08 smithi143 bash[29284]: cluster 2024-03-11T21:19:07.479180+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:09.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:08 smithi143 bash[21512]: cluster 2024-03-11T21:19:07.479180+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:09.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:08 smithi203 bash[23654]: cluster 2024-03-11T21:19:07.479180+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137879 s, 761 MB/s 2024-03-11T21:19:10.168 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-11T21:19:10.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-03-11T21:19:11.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:10 smithi143 bash[21512]: cluster 2024-03-11T21:19:09.479763+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:11.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:10 smithi143 bash[29284]: cluster 2024-03-11T21:19:09.479763+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:11.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:10 smithi203 bash[23654]: cluster 2024-03-11T21:19:09.479763+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:13.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: cluster 2024-03-11T21:19:11.480351+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:13.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: cephadm 2024-03-11T21:19:11.860354+0000 mgr.y (mgr.14152) 205 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.870341+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.882476+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.886043+0000 mon.a (mon.0) 438 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.894174+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.896047+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.897710+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: cephadm 2024-03-11T21:19:11.899115+0000 mgr.y (mgr.14152) 206 : cephadm 1 Adjusting osd_memory_target on smithi143 to 4060M 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.907481+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.913786+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.915473+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:13.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[29284]: audit 2024-03-11T21:19:11.925432+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: cluster 2024-03-11T21:19:11.480351+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: cephadm 2024-03-11T21:19:11.860354+0000 mgr.y (mgr.14152) 205 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.870341+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.882476+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.886043+0000 mon.a (mon.0) 438 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.894174+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.896047+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.897710+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: cephadm 2024-03-11T21:19:11.899115+0000 mgr.y (mgr.14152) 206 : cephadm 1 Adjusting osd_memory_target on smithi143 to 4060M 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.907481+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.913786+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.915473+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:12 smithi143 bash[21512]: audit 2024-03-11T21:19:11.925432+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: cluster 2024-03-11T21:19:11.480351+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: cephadm 2024-03-11T21:19:11.860354+0000 mgr.y (mgr.14152) 205 : cephadm 1 Detected new or changed devices on smithi143 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.870341+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.882476+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.886043+0000 mon.a (mon.0) 438 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.894174+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.896047+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.897710+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: cephadm 2024-03-11T21:19:11.899115+0000 mgr.y (mgr.14152) 206 : cephadm 1 Adjusting osd_memory_target on smithi143 to 4060M 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.907481+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.913786+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:13.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.915473+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:13.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:12 smithi203 bash[23654]: audit 2024-03-11T21:19:11.925432+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:14.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[21512]: cluster 2024-03-11T21:19:12.815408+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:14.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[21512]: cluster 2024-03-11T21:19:12.815524+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:14.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[21512]: cluster 2024-03-11T21:19:12.931911+0000 mon.a (mon.0) 446 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T21:19:14.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[29284]: cluster 2024-03-11T21:19:12.815408+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:14.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[29284]: cluster 2024-03-11T21:19:12.815524+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:14.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:13 smithi143 bash[29284]: cluster 2024-03-11T21:19:12.931911+0000 mon.a (mon.0) 446 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T21:19:14.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:13 smithi203 bash[23654]: cluster 2024-03-11T21:19:12.815408+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:14.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:13 smithi203 bash[23654]: cluster 2024-03-11T21:19:12.815524+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:14.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:13 smithi203 bash[23654]: cluster 2024-03-11T21:19:12.931911+0000 mon.a (mon.0) 446 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T21:19:15.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:14 smithi143 bash[21512]: cluster 2024-03-11T21:19:13.480993+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:15.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:14 smithi143 bash[29284]: cluster 2024-03-11T21:19:13.480993+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:15.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:14 smithi203 bash[23654]: cluster 2024-03-11T21:19:13.480993+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:15.517 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:19:17.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:16 smithi143 bash[21512]: cluster 2024-03-11T21:19:15.481656+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:17.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:16 smithi143 bash[29284]: cluster 2024-03-11T21:19:15.481656+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:17.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:16 smithi203 bash[23654]: cluster 2024-03-11T21:19:15.481656+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:17.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- 172.21.15.203:0/769675578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c101f80 msgr2=0x7f331c1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- 172.21.15.203:0/769675578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c101f80 0x7f331c1023e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33100099c0 tx=0x7f3310009cd0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- 172.21.15.203:0/769675578 shutdown_connections 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- 172.21.15.203:0/769675578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c102920 0x7f331c1091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- 172.21.15.203:0/769675578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 0x7f331c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- 172.21.15.203:0/769675578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c101f80 0x7f331c1023e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- 172.21.15.203:0/769675578 >> 172.21.15.203:0/769675578 conn(0x7f331c0fc340 msgr2=0x7f331c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- 172.21.15.203:0/769675578 shutdown_connections 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- 172.21.15.203:0/769675578 wait complete. 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 librados: starting msgr at 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 librados: starting objecter 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 Processor -- start 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- start start 2024-03-11T21:19:17.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 librados: setting wanted keys 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 librados: calling monclient init 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 0x7f331c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 0x7f331c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c10bea0 con 0x7f331c101f80 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c10bd20 con 0x7f331c102920 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3322376700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c10c020 con 0x7f331c100d80 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.819+0000 7f3320913700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:53028/0 (socket says 172.21.15.203:53028) 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 -- 172.21.15.203:0/3294744497 learned_addr learned my addr 172.21.15.203:0/3294744497 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f331bfff700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 0x7f331c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f331b7fe700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 0x7f331c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 -- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 msgr2=0x7f331c199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 0x7f331c199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 -- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 msgr2=0x7f331c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 0x7f331c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 -- 172.21.15.203:0/3294744497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c1a1590 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f331b7fe700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 0x7f331c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3320913700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f331400ada0 tx=0x7f331400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 10 client.?.objecter ms_handle_connect 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314017020 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f331400ce00 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c1a1880 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33140053a0 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3322376700 10 client.24187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c1a1e10 con 0x7f331c102920 2024-03-11T21:19:17.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3322376700 1 librados: init done 2024-03-11T21:19:17.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c04faa0 con 0x7f331c102920 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f331400c980 con 0x7f331c102920 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 0x7f3304071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3449+0+0 (secure 0 0 0) 0x7f331408c9b0 con 0x7f331c102920 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 10 client.24187.objecter ms_dispatch 0x7f331c001550 osd_map(30..30 src has 1..30) v4 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 3 client.24187.objecter handle_osd_map got epochs [30,30] > 0 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 3 client.24187.objecter handle_osd_map decoding full epoch 30 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f33197fa700 20 client.24187.objecter dump_active .. 0 homeless 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.823+0000 7f331bfff700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 0x7f3304071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.827+0000 7f33197fa700 10 client.24187.objecter ms_handle_connect 0x7f330406f730 2024-03-11T21:19:17.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.827+0000 7f331bfff700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 0x7f3304071bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f330c006fd0 tx=0x7f330c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:19:17.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:17.831+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3314021020 con 0x7f331c102920 2024-03-11T21:19:18.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:18.035+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f331c062220 con 0x7f330406f730 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: cluster 2024-03-11T21:19:17.379782+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: cluster 2024-03-11T21:19:17.383752+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: cluster 2024-03-11T21:19:17.482235+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: audit 2024-03-11T21:19:18.042973+0000 mgr.y (mgr.14152) 210 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: audit 2024-03-11T21:19:18.050509+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:19:19.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: audit 2024-03-11T21:19:18.057045+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[21512]: audit 2024-03-11T21:19:18.058799+0000 mon.a (mon.0) 449 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: cluster 2024-03-11T21:19:17.379782+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: cluster 2024-03-11T21:19:17.383752+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: cluster 2024-03-11T21:19:17.482235+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: audit 2024-03-11T21:19:18.042973+0000 mgr.y (mgr.14152) 210 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: audit 2024-03-11T21:19:18.050509+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: audit 2024-03-11T21:19:18.057045+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:19:19.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:18 smithi143 bash[29284]: audit 2024-03-11T21:19:18.058799+0000 mon.a (mon.0) 449 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:19.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: cluster 2024-03-11T21:19:17.379782+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:19:19.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: cluster 2024-03-11T21:19:17.383752+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:19:19.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: cluster 2024-03-11T21:19:17.482235+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:19.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: audit 2024-03-11T21:19:18.042973+0000 mgr.y (mgr.14152) 210 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:19.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: audit 2024-03-11T21:19:18.050509+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:19:19.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: audit 2024-03-11T21:19:18.057045+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:19:19.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:18 smithi203 bash[23654]: audit 2024-03-11T21:19:18.058799+0000 mon.a (mon.0) 449 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:21.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:20 smithi143 bash[21512]: cluster 2024-03-11T21:19:19.482827+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:21.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:20 smithi143 bash[29284]: cluster 2024-03-11T21:19:19.482827+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:21.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:20 smithi203 bash[23654]: cluster 2024-03-11T21:19:19.482827+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:22.827+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:23.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:22 smithi143 bash[21512]: cluster 2024-03-11T21:19:21.483365+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:23.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:22 smithi143 bash[29284]: cluster 2024-03-11T21:19:21.483365+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:23.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:22 smithi203 bash[23654]: cluster 2024-03-11T21:19:21.483365+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:25.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:24 smithi143 bash[21512]: cluster 2024-03-11T21:19:23.484011+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:25.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:24 smithi143 bash[29284]: cluster 2024-03-11T21:19:23.484011+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:25.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:24 smithi203 bash[23654]: cluster 2024-03-11T21:19:23.484011+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: cluster 2024-03-11T21:19:25.484556+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: audit 2024-03-11T21:19:25.972042+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.203:0/3132084150' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: audit 2024-03-11T21:19:25.975111+0000 mon.a (mon.0) 450 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: audit 2024-03-11T21:19:25.978455+0000 mon.a (mon.0) 451 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]': finished 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: cluster 2024-03-11T21:19:25.985666+0000 mon.a (mon.0) 452 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: audit 2024-03-11T21:19:25.986367+0000 mon.a (mon.0) 453 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[21512]: audit 2024-03-11T21:19:26.726080+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.203:0/1207762757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: cluster 2024-03-11T21:19:25.484556+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: audit 2024-03-11T21:19:25.972042+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.203:0/3132084150' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: audit 2024-03-11T21:19:25.975111+0000 mon.a (mon.0) 450 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: audit 2024-03-11T21:19:25.978455+0000 mon.a (mon.0) 451 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]': finished 2024-03-11T21:19:27.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: cluster 2024-03-11T21:19:25.985666+0000 mon.a (mon.0) 452 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:19:27.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: audit 2024-03-11T21:19:25.986367+0000 mon.a (mon.0) 453 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:27.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:26 smithi143 bash[29284]: audit 2024-03-11T21:19:26.726080+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.203:0/1207762757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:19:27.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: cluster 2024-03-11T21:19:25.484556+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:27.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: audit 2024-03-11T21:19:25.972042+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.203:0/3132084150' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: audit 2024-03-11T21:19:25.975111+0000 mon.a (mon.0) 450 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]: dispatch 2024-03-11T21:19:27.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: audit 2024-03-11T21:19:25.978455+0000 mon.a (mon.0) 451 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b9e088-d377-4335-8863-11d581828f7e"}]': finished 2024-03-11T21:19:27.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: cluster 2024-03-11T21:19:25.985666+0000 mon.a (mon.0) 452 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:19:27.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: audit 2024-03-11T21:19:25.986367+0000 mon.a (mon.0) 453 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:27.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:26 smithi203 bash[23654]: audit 2024-03-11T21:19:26.726080+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.203:0/1207762757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:19:27.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:27.831+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:29.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:28 smithi203 bash[23654]: cluster 2024-03-11T21:19:27.485102+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:29.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:28 smithi143 bash[21512]: cluster 2024-03-11T21:19:27.485102+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:29.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:28 smithi143 bash[29284]: cluster 2024-03-11T21:19:27.485102+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:31.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:30 smithi203 bash[23654]: cluster 2024-03-11T21:19:29.485704+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:31.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:30 smithi143 bash[21512]: cluster 2024-03-11T21:19:29.485704+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:31.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:30 smithi143 bash[29284]: cluster 2024-03-11T21:19:29.485704+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:32.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:32.835+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:33.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:32 smithi203 bash[23654]: cluster 2024-03-11T21:19:31.486272+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:33.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:32 smithi143 bash[21512]: cluster 2024-03-11T21:19:31.486272+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:33.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:32 smithi143 bash[29284]: cluster 2024-03-11T21:19:31.486272+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:35.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:34 smithi203 bash[23654]: cluster 2024-03-11T21:19:33.486833+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:35.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:34 smithi143 bash[21512]: cluster 2024-03-11T21:19:33.486833+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:35.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:34 smithi143 bash[29284]: cluster 2024-03-11T21:19:33.486833+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:37.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:36 smithi203 bash[23654]: cluster 2024-03-11T21:19:35.487418+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:37.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:36 smithi143 bash[21512]: cluster 2024-03-11T21:19:35.487418+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:37.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:36 smithi143 bash[29284]: cluster 2024-03-11T21:19:35.487418+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:37.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:37.839+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:38.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:37 smithi203 bash[23654]: audit 2024-03-11T21:19:37.771945+0000 mon.a (mon.0) 454 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:19:38.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:37 smithi203 bash[23654]: audit 2024-03-11T21:19:37.773815+0000 mon.a (mon.0) 455 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:38.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:37 smithi143 bash[21512]: audit 2024-03-11T21:19:37.771945+0000 mon.a (mon.0) 454 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:19:38.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:37 smithi143 bash[21512]: audit 2024-03-11T21:19:37.773815+0000 mon.a (mon.0) 455 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:38.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:37 smithi143 bash[29284]: audit 2024-03-11T21:19:37.771945+0000 mon.a (mon.0) 454 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:19:38.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:37 smithi143 bash[29284]: audit 2024-03-11T21:19:37.773815+0000 mon.a (mon.0) 455 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:39.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:38 smithi203 bash[23654]: cluster 2024-03-11T21:19:37.487979+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:39.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:38 smithi203 bash[23654]: cephadm 2024-03-11T21:19:37.775487+0000 mgr.y (mgr.14152) 221 : cephadm 1 Deploying daemon osd.4 on smithi203 2024-03-11T21:19:39.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:38 smithi143 bash[21512]: cluster 2024-03-11T21:19:37.487979+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:39.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:38 smithi143 bash[21512]: cephadm 2024-03-11T21:19:37.775487+0000 mgr.y (mgr.14152) 221 : cephadm 1 Deploying daemon osd.4 on smithi203 2024-03-11T21:19:39.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:38 smithi143 bash[29284]: cluster 2024-03-11T21:19:37.487979+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:39.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:38 smithi143 bash[29284]: cephadm 2024-03-11T21:19:37.775487+0000 mgr.y (mgr.14152) 221 : cephadm 1 Deploying daemon osd.4 on smithi203 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:19:41 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:40 smithi203 bash[23654]: cluster 2024-03-11T21:19:39.488478+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:40 smithi203 bash[23654]: cluster 2024-03-11T21:19:39.983844+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:40 smithi203 bash[23654]: cluster 2024-03-11T21:19:39.983935+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:40 smithi203 bash[23654]: cluster 2024-03-11T21:19:40.007784+0000 mon.a (mon.0) 456 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:40 smithi203 bash[23654]: audit 2024-03-11T21:19:40.008130+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:41.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:41 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:19:41.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[21512]: cluster 2024-03-11T21:19:39.488478+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[21512]: cluster 2024-03-11T21:19:39.983844+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[21512]: cluster 2024-03-11T21:19:39.983935+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[21512]: cluster 2024-03-11T21:19:40.007784+0000 mon.a (mon.0) 456 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[21512]: audit 2024-03-11T21:19:40.008130+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[29284]: cluster 2024-03-11T21:19:39.488478+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[29284]: cluster 2024-03-11T21:19:39.983844+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[29284]: cluster 2024-03-11T21:19:39.983935+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[29284]: cluster 2024-03-11T21:19:40.007784+0000 mon.a (mon.0) 456 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T21:19:41.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:41 smithi143 bash[29284]: audit 2024-03-11T21:19:40.008130+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:41.564 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:41 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:19:41.564 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:19:41 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:19:42.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:42.843+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:42.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:42 smithi203 bash[23654]: cluster 2024-03-11T21:19:41.489018+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:42.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:42 smithi203 bash[23654]: audit 2024-03-11T21:19:41.631297+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:42 smithi203 bash[23654]: audit 2024-03-11T21:19:41.642078+0000 mon.a (mon.0) 459 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:42 smithi203 bash[23654]: audit 2024-03-11T21:19:41.643143+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:42.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[21512]: cluster 2024-03-11T21:19:41.489018+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:42.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[21512]: audit 2024-03-11T21:19:41.631297+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[21512]: audit 2024-03-11T21:19:41.642078+0000 mon.a (mon.0) 459 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[21512]: audit 2024-03-11T21:19:41.643143+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[29284]: cluster 2024-03-11T21:19:41.489018+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[29284]: audit 2024-03-11T21:19:41.631297+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[29284]: audit 2024-03-11T21:19:41.642078+0000 mon.a (mon.0) 459 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:42.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:42 smithi143 bash[29284]: audit 2024-03-11T21:19:41.643143+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:44.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:44 smithi143 bash[21512]: cluster 2024-03-11T21:19:43.489562+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:44.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:44 smithi143 bash[29284]: cluster 2024-03-11T21:19:43.489562+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:45.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:44 smithi203 bash[23654]: cluster 2024-03-11T21:19:43.489562+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:46.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:46 smithi143 bash[21512]: cluster 2024-03-11T21:19:45.490075+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:46.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:46 smithi143 bash[29284]: cluster 2024-03-11T21:19:45.490075+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:47.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:46 smithi203 bash[23654]: cluster 2024-03-11T21:19:45.490075+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:47.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:47.847+0000 7f3321b75700 10 client.24187.objecter tick 2024-03-11T21:19:47.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:47 smithi143 bash[21512]: audit 2024-03-11T21:19:47.552060+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:47.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:47 smithi143 bash[21512]: audit 2024-03-11T21:19:47.554993+0000 mon.a (mon.0) 461 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:47.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:47 smithi143 bash[29284]: audit 2024-03-11T21:19:47.552060+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:47.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:47 smithi143 bash[29284]: audit 2024-03-11T21:19:47.554993+0000 mon.a (mon.0) 461 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:47.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:47 smithi203 bash[23654]: audit 2024-03-11T21:19:47.552060+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:47.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:47 smithi203 bash[23654]: audit 2024-03-11T21:19:47.554993+0000 mon.a (mon.0) 461 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:19:48.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: cluster 2024-03-11T21:19:47.490622+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:48.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:47.644832+0000 mon.a (mon.0) 462 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:47.653096+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: cluster 2024-03-11T21:19:47.653116+0000 mon.a (mon.0) 463 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:47.653806+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:47.656136+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:48.249079+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[21512]: audit 2024-03-11T21:19:48.257969+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: cluster 2024-03-11T21:19:47.490622+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:47.644832+0000 mon.a (mon.0) 462 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:47.653096+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: cluster 2024-03-11T21:19:47.653116+0000 mon.a (mon.0) 463 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:47.653806+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:48.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:47.656136+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:48.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:48.249079+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:48.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:48 smithi143 bash[29284]: audit 2024-03-11T21:19:48.257969+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: cluster 2024-03-11T21:19:47.490622+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:47.644832+0000 mon.a (mon.0) 462 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:47.653096+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: cluster 2024-03-11T21:19:47.653116+0000 mon.a (mon.0) 463 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:47.653806+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:47.656136+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:48.249079+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:48 smithi203 bash[23654]: audit 2024-03-11T21:19:48.257969+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: audit 2024-03-11T21:19:48.647569+0000 mon.a (mon.0) 468 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: cluster 2024-03-11T21:19:48.657141+0000 mon.a (mon.0) 469 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: audit 2024-03-11T21:19:48.658614+0000 mon.a (mon.0) 470 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: audit 2024-03-11T21:19:48.889974+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: audit 2024-03-11T21:19:48.891888+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: audit 2024-03-11T21:19:48.901935+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: cluster 2024-03-11T21:19:49.657957+0000 mon.a (mon.0) 474 : cluster 1 osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] boot 2024-03-11T21:19:49.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:49 smithi203 bash[23654]: cluster 2024-03-11T21:19:49.658061+0000 mon.a (mon.0) 475 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:19:49.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: audit 2024-03-11T21:19:48.647569+0000 mon.a (mon.0) 468 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:19:49.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: cluster 2024-03-11T21:19:48.657141+0000 mon.a (mon.0) 469 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T21:19:49.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: audit 2024-03-11T21:19:48.658614+0000 mon.a (mon.0) 470 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: audit 2024-03-11T21:19:48.889974+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: audit 2024-03-11T21:19:48.891888+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: audit 2024-03-11T21:19:48.901935+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: cluster 2024-03-11T21:19:49.657957+0000 mon.a (mon.0) 474 : cluster 1 osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] boot 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[29284]: cluster 2024-03-11T21:19:49.658061+0000 mon.a (mon.0) 475 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: audit 2024-03-11T21:19:48.647569+0000 mon.a (mon.0) 468 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: cluster 2024-03-11T21:19:48.657141+0000 mon.a (mon.0) 469 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: audit 2024-03-11T21:19:48.658614+0000 mon.a (mon.0) 470 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: audit 2024-03-11T21:19:48.889974+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: audit 2024-03-11T21:19:48.891888+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: audit 2024-03-11T21:19:48.901935+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:49.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: cluster 2024-03-11T21:19:49.657957+0000 mon.a (mon.0) 474 : cluster 1 osd.4 [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] boot 2024-03-11T21:19:49.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:49 smithi143 bash[21512]: cluster 2024-03-11T21:19:49.658061+0000 mon.a (mon.0) 475 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:19:50.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:50 smithi203 bash[23654]: cluster 2024-03-11T21:19:48.517161+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:50.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:50 smithi203 bash[23654]: cluster 2024-03-11T21:19:48.517218+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:50.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:50 smithi203 bash[23654]: cluster 2024-03-11T21:19:49.491249+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v202: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:50.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:50 smithi203 bash[23654]: audit 2024-03-11T21:19:49.660847+0000 mon.a (mon.0) 476 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:50.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:50 smithi203 bash[23654]: cluster 2024-03-11T21:19:50.658965+0000 mon.a (mon.0) 477 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:19:50.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[21512]: cluster 2024-03-11T21:19:48.517161+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:50.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[21512]: cluster 2024-03-11T21:19:48.517218+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:50.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[21512]: cluster 2024-03-11T21:19:49.491249+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v202: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[21512]: audit 2024-03-11T21:19:49.660847+0000 mon.a (mon.0) 476 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[21512]: cluster 2024-03-11T21:19:50.658965+0000 mon.a (mon.0) 477 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[29284]: cluster 2024-03-11T21:19:48.517161+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[29284]: cluster 2024-03-11T21:19:48.517218+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[29284]: cluster 2024-03-11T21:19:49.491249+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v202: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[29284]: audit 2024-03-11T21:19:49.660847+0000 mon.a (mon.0) 476 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:19:50.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:50 smithi143 bash[29284]: cluster 2024-03-11T21:19:50.658965+0000 mon.a (mon.0) 477 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:19:51.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.075+0000 7f33197fa700 1 -- 172.21.15.203:0/3294744497 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f331c062220 con 0x7f330406f730 2024-03-11T21:19:51.060 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 4 on host 'smithi203' 2024-03-11T21:19:51.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 10 librados: watch_flush enter 2024-03-11T21:19:51.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 10 librados: watch_flush exit 2024-03-11T21:19:51.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 20 client.24187.objecter shutdown clearing up homeless session... 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 10 client.24187.objecter successfully canceled tick 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 msgr2=0x7f3304071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 0x7f3304071bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f330c006fd0 tx=0x7f330c008040 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 msgr2=0x7f331c1a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f331400ada0 tx=0x7f331400c330 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 shutdown_connections 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f330406f730 0x7f3304071bf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f331c102920 0x7f331c1a0ef0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331c100d80 0x7f331c199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 --2- 172.21.15.203:0/3294744497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f331c101f80 0x7f331c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:19:51.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 >> 172.21.15.203:0/3294744497 conn(0x7f331c0fc340 msgr2=0x7f331c0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:19:51.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 shutdown_connections 2024-03-11T21:19:51.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 -- 172.21.15.203:0/3294744497 wait complete. 2024-03-11T21:19:51.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:19:51.079+0000 7f3322376700 1 librados: shutdown 2024-03-11T21:19:51.630 DEBUG:teuthology.orchestra.run.smithi203:osd.4> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T21:19:51.633 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-03-11T21:19:51.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T21:19:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[21512]: audit 2024-03-11T21:19:51.067711+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[21512]: audit 2024-03-11T21:19:51.077036+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[21512]: audit 2024-03-11T21:19:51.092742+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[21512]: cluster 2024-03-11T21:19:51.660529+0000 mon.a (mon.0) 481 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T21:19:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[29284]: audit 2024-03-11T21:19:51.067711+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[29284]: audit 2024-03-11T21:19:51.077036+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[29284]: audit 2024-03-11T21:19:51.092742+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:52 smithi143 bash[29284]: cluster 2024-03-11T21:19:51.660529+0000 mon.a (mon.0) 481 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T21:19:52.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:52 smithi203 bash[23654]: audit 2024-03-11T21:19:51.067711+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:52 smithi203 bash[23654]: audit 2024-03-11T21:19:51.077036+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:19:52.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:52 smithi203 bash[23654]: audit 2024-03-11T21:19:51.092742+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:52.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:52 smithi203 bash[23654]: cluster 2024-03-11T21:19:51.660529+0000 mon.a (mon.0) 481 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T21:19:53.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:53 smithi143 bash[21512]: cluster 2024-03-11T21:19:51.491783+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:53.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:53 smithi143 bash[21512]: cluster 2024-03-11T21:19:52.663676+0000 mon.a (mon.0) 482 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T21:19:53.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:53 smithi143 bash[29284]: cluster 2024-03-11T21:19:51.491783+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:53.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:53 smithi143 bash[29284]: cluster 2024-03-11T21:19:52.663676+0000 mon.a (mon.0) 482 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T21:19:53.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:53 smithi203 bash[23654]: cluster 2024-03-11T21:19:51.491783+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:53.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:53 smithi203 bash[23654]: cluster 2024-03-11T21:19:52.663676+0000 mon.a (mon.0) 482 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T21:19:54.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:54 smithi143 bash[21512]: cluster 2024-03-11T21:19:53.492268+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:54.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:54 smithi143 bash[21512]: cluster 2024-03-11T21:19:53.664798+0000 mon.a (mon.0) 483 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T21:19:54.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:54 smithi143 bash[29284]: cluster 2024-03-11T21:19:53.492268+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:54.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:54 smithi143 bash[29284]: cluster 2024-03-11T21:19:53.664798+0000 mon.a (mon.0) 483 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T21:19:55.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:54 smithi203 bash[23654]: cluster 2024-03-11T21:19:53.492268+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:19:55.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:54 smithi203 bash[23654]: cluster 2024-03-11T21:19:53.664798+0000 mon.a (mon.0) 483 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T21:19:56.028 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:19:56.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:56 smithi143 bash[21512]: cluster 2024-03-11T21:19:55.492845+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-11T21:19:56.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:56 smithi143 bash[29284]: cluster 2024-03-11T21:19:55.492845+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-11T21:19:56.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:56 smithi203 bash[23654]: cluster 2024-03-11T21:19:55.492845+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-11T21:19:58.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:19:58 smithi143 bash[21512]: cluster 2024-03-11T21:19:57.493425+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:19:58.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:19:58 smithi143 bash[29284]: cluster 2024-03-11T21:19:57.493425+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:19:58.955 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:19:58 smithi203 bash[23654]: cluster 2024-03-11T21:19:57.493425+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128989 s, 813 MB/s 2024-03-11T21:19:59.637 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-11T21:20:00.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-03-11T21:20:00.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:00 smithi143 bash[21512]: cluster 2024-03-11T21:19:59.494097+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:20:00.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:00 smithi143 bash[21512]: cluster 2024-03-11T21:20:00.000175+0000 mon.a (mon.0) 484 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:00.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:00 smithi143 bash[29284]: cluster 2024-03-11T21:19:59.494097+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:20:00.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:00 smithi143 bash[29284]: cluster 2024-03-11T21:20:00.000175+0000 mon.a (mon.0) 484 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:01.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:00 smithi203 bash[23654]: cluster 2024-03-11T21:19:59.494097+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:20:01.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:00 smithi203 bash[23654]: cluster 2024-03-11T21:20:00.000175+0000 mon.a (mon.0) 484 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: cephadm 2024-03-11T21:20:00.688233+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.698848+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.710037+0000 mon.a (mon.0) 486 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.712346+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: cephadm 2024-03-11T21:20:00.713935+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi203 to 17265M 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.720205+0000 mon.a (mon.0) 488 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.724753+0000 mon.a (mon.0) 489 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.726400+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[21512]: audit 2024-03-11T21:20:00.738618+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: cephadm 2024-03-11T21:20:00.688233+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.698848+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.710037+0000 mon.a (mon.0) 486 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.712346+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: cephadm 2024-03-11T21:20:00.713935+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi203 to 17265M 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.720205+0000 mon.a (mon.0) 488 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.724753+0000 mon.a (mon.0) 489 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.726400+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:01 smithi143 bash[29284]: audit 2024-03-11T21:20:00.738618+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: cephadm 2024-03-11T21:20:00.688233+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.698848+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.710037+0000 mon.a (mon.0) 486 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.712346+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: cephadm 2024-03-11T21:20:00.713935+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi203 to 17265M 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.720205+0000 mon.a (mon.0) 488 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.724753+0000 mon.a (mon.0) 489 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.726400+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:02.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:01 smithi203 bash[23654]: audit 2024-03-11T21:20:00.738618+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:03.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:02 smithi203 bash[23654]: cluster 2024-03-11T21:20:01.494782+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T21:20:03.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:02 smithi143 bash[21512]: cluster 2024-03-11T21:20:01.494782+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T21:20:03.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:02 smithi143 bash[29284]: cluster 2024-03-11T21:20:01.494782+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T21:20:04.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:04 smithi203 bash[23654]: cluster 2024-03-11T21:20:03.495442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:04.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:04 smithi143 bash[21512]: cluster 2024-03-11T21:20:03.495442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:04.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:04 smithi143 bash[29284]: cluster 2024-03-11T21:20:03.495442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:05.036 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:20:06.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:06 smithi203 bash[23654]: cluster 2024-03-11T21:20:05.378970+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:06.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:06 smithi203 bash[23654]: cluster 2024-03-11T21:20:05.379071+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:06.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:06 smithi203 bash[23654]: cluster 2024-03-11T21:20:05.482931+0000 mon.a (mon.0) 492 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T21:20:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:06 smithi203 bash[23654]: cluster 2024-03-11T21:20:05.496009+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[21512]: cluster 2024-03-11T21:20:05.378970+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[21512]: cluster 2024-03-11T21:20:05.379071+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[21512]: cluster 2024-03-11T21:20:05.482931+0000 mon.a (mon.0) 492 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[21512]: cluster 2024-03-11T21:20:05.496009+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[29284]: cluster 2024-03-11T21:20:05.378970+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[29284]: cluster 2024-03-11T21:20:05.379071+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[29284]: cluster 2024-03-11T21:20:05.482931+0000 mon.a (mon.0) 492 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T21:20:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:06 smithi143 bash[29284]: cluster 2024-03-11T21:20:05.496009+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:07.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- 172.21.15.203:0/3340626471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 msgr2=0x7f29d81108d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:07.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- 172.21.15.203:0/3340626471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81108d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f29d000b0d0 tx=0x7f29d000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- 172.21.15.203:0/3340626471 shutdown_connections 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- 172.21.15.203:0/3340626471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81108d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- 172.21.15.203:0/3340626471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8108480 0x7f29d81088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- 172.21.15.203:0/3340626471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8109680 0x7f29d8109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- 172.21.15.203:0/3340626471 >> 172.21.15.203:0/3340626471 conn(0x7f29d8077470 msgr2=0x7f29d8079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- 172.21.15.203:0/3340626471 shutdown_connections 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- 172.21.15.203:0/3340626471 wait complete. 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 librados: starting msgr at 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 librados: starting objecter 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 Processor -- start 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- start start 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 librados: setting wanted keys 2024-03-11T21:20:07.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 librados: calling monclient init 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8108480 0x7f29d819dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 0x7f29d819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d81135a0 con 0x7f29d8108480 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8113420 con 0x7f29d810a020 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29e012c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8113720 con 0x7f29d8109680 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44272/0 (socket says 172.21.15.203:44272) 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 -- 172.21.15.203:0/3000375260 learned_addr learned my addr 172.21.15.203:0/3000375260 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29ddec8700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8108480 0x7f29d819dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29dd6c7700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 0x7f29d819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 -- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 msgr2=0x7f29d819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 0x7f29d819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 -- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8108480 msgr2=0x7f29d819dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8108480 0x7f29d819dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 -- 172.21.15.203:0/3000375260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d81a58a0 con 0x7f29d810a020 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29dd6c7700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 0x7f29d819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29de6c9700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29d0007cf0 tx=0x7f29d0007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29ceffd700 10 client.?.objecter ms_handle_connect 0x7f29d810a020 2024-03-11T21:20:07.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.307+0000 7f29ceffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:20:07.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0021070 con 0x7f29d810a020 2024-03-11T21:20:07.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d81a5b30 con 0x7f29d810a020 2024-03-11T21:20:07.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f29d001bcc0 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29e012c700 10 client.24214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0012c90 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d81a5fe0 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29e012c700 1 librados: init done 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d804faa0 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f29d0024070 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 0x7f29c4071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3991+0+0 (secure 0 0 0) 0x7f29d000f020 con 0x7f29d810a020 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 10 client.24214.objecter ms_dispatch 0x7f29d8001550 osd_map(40..40 src has 1..40) v4 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 3 client.24214.objecter handle_osd_map got epochs [40,40] > 0 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 3 client.24214.objecter handle_osd_map decoding full epoch 40 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 20 client.24214.objecter dump_active .. 0 homeless 2024-03-11T21:20:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ddec8700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 0x7f29c4071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:07.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ceffd700 10 client.24214.objecter ms_handle_connect 0x7f29c406f600 2024-03-11T21:20:07.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.311+0000 7f29ddec8700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 0x7f29c4071ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29c8005950 tx=0x7f29c800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:20:07.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.315+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f29d002b020 con 0x7f29d810a020 2024-03-11T21:20:07.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:07.523+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f29d802dc70 con 0x7f29c406f600 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[21512]: audit 2024-03-11T21:20:07.539323+0000 mon.a (mon.0) 493 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[21512]: audit 2024-03-11T21:20:07.546488+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[21512]: audit 2024-03-11T21:20:07.548353+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[29284]: audit 2024-03-11T21:20:07.539323+0000 mon.a (mon.0) 493 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[29284]: audit 2024-03-11T21:20:07.546488+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:07.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:07 smithi143 bash[29284]: audit 2024-03-11T21:20:07.548353+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:07 smithi203 bash[23654]: audit 2024-03-11T21:20:07.539323+0000 mon.a (mon.0) 493 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:07 smithi203 bash[23654]: audit 2024-03-11T21:20:07.546488+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:07 smithi203 bash[23654]: audit 2024-03-11T21:20:07.548353+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:08.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:08 smithi143 bash[29284]: cluster 2024-03-11T21:20:07.496509+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:08.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:08 smithi143 bash[29284]: audit 2024-03-11T21:20:07.532261+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:08.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:08 smithi143 bash[21512]: cluster 2024-03-11T21:20:07.496509+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:08.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:08 smithi143 bash[21512]: audit 2024-03-11T21:20:07.532261+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:09.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:08 smithi203 bash[23654]: cluster 2024-03-11T21:20:07.496509+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:09.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:08 smithi203 bash[23654]: audit 2024-03-11T21:20:07.532261+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:10.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:10 smithi143 bash[21512]: cluster 2024-03-11T21:20:09.497076+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:10.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:10 smithi143 bash[29284]: cluster 2024-03-11T21:20:09.497076+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:11.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:10 smithi203 bash[23654]: cluster 2024-03-11T21:20:09.497076+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:12.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:12.314+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:12.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:12 smithi203 bash[23654]: cluster 2024-03-11T21:20:11.497608+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:12.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:12 smithi143 bash[21512]: cluster 2024-03-11T21:20:11.497608+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:12.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:12 smithi143 bash[29284]: cluster 2024-03-11T21:20:11.497608+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:14.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:14 smithi203 bash[23654]: cluster 2024-03-11T21:20:13.498178+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:14.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:14 smithi143 bash[21512]: cluster 2024-03-11T21:20:13.498178+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:14.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:14 smithi143 bash[29284]: cluster 2024-03-11T21:20:13.498178+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[29284]: audit 2024-03-11T21:20:15.357280+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.203:0/3346514071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[29284]: audit 2024-03-11T21:20:15.360359+0000 mon.a (mon.0) 496 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[29284]: audit 2024-03-11T21:20:15.363748+0000 mon.a (mon.0) 497 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]': finished 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[29284]: cluster 2024-03-11T21:20:15.369857+0000 mon.a (mon.0) 498 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[29284]: audit 2024-03-11T21:20:15.370480+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[21512]: audit 2024-03-11T21:20:15.357280+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.203:0/3346514071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[21512]: audit 2024-03-11T21:20:15.360359+0000 mon.a (mon.0) 496 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[21512]: audit 2024-03-11T21:20:15.363748+0000 mon.a (mon.0) 497 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]': finished 2024-03-11T21:20:15.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[21512]: cluster 2024-03-11T21:20:15.369857+0000 mon.a (mon.0) 498 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T21:20:15.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:15 smithi143 bash[21512]: audit 2024-03-11T21:20:15.370480+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:16.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:15 smithi203 bash[23654]: audit 2024-03-11T21:20:15.357280+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.203:0/3346514071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:16.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:15 smithi203 bash[23654]: audit 2024-03-11T21:20:15.360359+0000 mon.a (mon.0) 496 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]: dispatch 2024-03-11T21:20:16.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:15 smithi203 bash[23654]: audit 2024-03-11T21:20:15.363748+0000 mon.a (mon.0) 497 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01"}]': finished 2024-03-11T21:20:16.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:15 smithi203 bash[23654]: cluster 2024-03-11T21:20:15.369857+0000 mon.a (mon.0) 498 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T21:20:16.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:15 smithi203 bash[23654]: audit 2024-03-11T21:20:15.370480+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:16.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:16 smithi143 bash[21512]: cluster 2024-03-11T21:20:15.498701+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:16.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:16 smithi143 bash[21512]: audit 2024-03-11T21:20:16.105938+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.203:0/1923632259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:20:16.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:16 smithi143 bash[29284]: cluster 2024-03-11T21:20:15.498701+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:16.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:16 smithi143 bash[29284]: audit 2024-03-11T21:20:16.105938+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.203:0/1923632259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:20:17.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:16 smithi203 bash[23654]: cluster 2024-03-11T21:20:15.498701+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:17.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:16 smithi203 bash[23654]: audit 2024-03-11T21:20:16.105938+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.203:0/1923632259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:20:17.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:17.318+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:18.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:18 smithi143 bash[21512]: cluster 2024-03-11T21:20:17.499243+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:18.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:18 smithi143 bash[29284]: cluster 2024-03-11T21:20:17.499243+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:19.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:18 smithi203 bash[23654]: cluster 2024-03-11T21:20:17.499243+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:20.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:20 smithi143 bash[21512]: cluster 2024-03-11T21:20:19.499801+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:20.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:20 smithi143 bash[29284]: cluster 2024-03-11T21:20:19.499801+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:21.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:20 smithi203 bash[23654]: cluster 2024-03-11T21:20:19.499801+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:22.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:22.322+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:22.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[21512]: cluster 2024-03-11T21:20:21.500473+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:22.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[21512]: cluster 2024-03-11T21:20:21.802417+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:22.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[21512]: cluster 2024-03-11T21:20:21.802519+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:22.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[29284]: cluster 2024-03-11T21:20:21.500473+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:22.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[29284]: cluster 2024-03-11T21:20:21.802417+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:22.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:22 smithi143 bash[29284]: cluster 2024-03-11T21:20:21.802519+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:22 smithi203 bash[23654]: cluster 2024-03-11T21:20:21.500473+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:22 smithi203 bash[23654]: cluster 2024-03-11T21:20:21.802417+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:22 smithi203 bash[23654]: cluster 2024-03-11T21:20:21.802519+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:23.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:23 smithi143 bash[21512]: cluster 2024-03-11T21:20:22.629071+0000 mon.a (mon.0) 500 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T21:20:23.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:23 smithi143 bash[21512]: audit 2024-03-11T21:20:22.629720+0000 mon.a (mon.0) 501 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:23.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:23 smithi143 bash[29284]: cluster 2024-03-11T21:20:22.629071+0000 mon.a (mon.0) 500 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T21:20:23.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:23 smithi143 bash[29284]: audit 2024-03-11T21:20:22.629720+0000 mon.a (mon.0) 501 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:24.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:23 smithi203 bash[23654]: cluster 2024-03-11T21:20:22.629071+0000 mon.a (mon.0) 500 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T21:20:24.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:23 smithi203 bash[23654]: audit 2024-03-11T21:20:22.629720+0000 mon.a (mon.0) 501 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:24.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:24 smithi203 bash[23654]: cluster 2024-03-11T21:20:23.501058+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:24.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:24 smithi143 bash[21512]: cluster 2024-03-11T21:20:23.501058+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:24.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:24 smithi143 bash[29284]: cluster 2024-03-11T21:20:23.501058+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:26.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[21512]: cluster 2024-03-11T21:20:25.501609+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:26.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[21512]: cluster 2024-03-11T21:20:26.175706+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:26.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[21512]: cluster 2024-03-11T21:20:26.175817+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:26.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[29284]: cluster 2024-03-11T21:20:25.501609+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:26.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[29284]: cluster 2024-03-11T21:20:26.175706+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:26.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:26 smithi143 bash[29284]: cluster 2024-03-11T21:20:26.175817+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:27.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:26 smithi203 bash[23654]: cluster 2024-03-11T21:20:25.501609+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:27.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:26 smithi203 bash[23654]: cluster 2024-03-11T21:20:26.175706+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:27.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:26 smithi203 bash[23654]: cluster 2024-03-11T21:20:26.175817+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:27.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:27.326+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:27.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:27 smithi203 bash[23654]: cluster 2024-03-11T21:20:26.660143+0000 mon.a (mon.0) 502 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T21:20:27.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:27 smithi203 bash[23654]: audit 2024-03-11T21:20:26.660434+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:27.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:27 smithi203 bash[23654]: audit 2024-03-11T21:20:27.102370+0000 mon.a (mon.0) 504 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:20:27.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:27 smithi203 bash[23654]: audit 2024-03-11T21:20:27.104180+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:27.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:27 smithi203 bash[23654]: cephadm 2024-03-11T21:20:27.105985+0000 mgr.y (mgr.14152) 249 : cephadm 1 Deploying daemon osd.5 on smithi203 2024-03-11T21:20:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[21512]: cluster 2024-03-11T21:20:26.660143+0000 mon.a (mon.0) 502 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[21512]: audit 2024-03-11T21:20:26.660434+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[21512]: audit 2024-03-11T21:20:27.102370+0000 mon.a (mon.0) 504 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[21512]: audit 2024-03-11T21:20:27.104180+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[21512]: cephadm 2024-03-11T21:20:27.105985+0000 mgr.y (mgr.14152) 249 : cephadm 1 Deploying daemon osd.5 on smithi203 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[29284]: cluster 2024-03-11T21:20:26.660143+0000 mon.a (mon.0) 502 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[29284]: audit 2024-03-11T21:20:26.660434+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[29284]: audit 2024-03-11T21:20:27.102370+0000 mon.a (mon.0) 504 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[29284]: audit 2024-03-11T21:20:27.104180+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:27 smithi143 bash[29284]: cephadm 2024-03-11T21:20:27.105985+0000 mgr.y (mgr.14152) 249 : cephadm 1 Deploying daemon osd.5 on smithi203 2024-03-11T21:20:28.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:28 smithi143 bash[21512]: cluster 2024-03-11T21:20:27.502086+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:28.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:28 smithi143 bash[29284]: cluster 2024-03-11T21:20:27.502086+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:28.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:28 smithi203 bash[23654]: cluster 2024-03-11T21:20:27.502086+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:30.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:30 smithi203 bash[23654]: cluster 2024-03-11T21:20:29.502624+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:30.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:30.919 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:20:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:30.920 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:20:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:30.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:30 smithi143 bash[21512]: cluster 2024-03-11T21:20:29.502624+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:30.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:30 smithi143 bash[29284]: cluster 2024-03-11T21:20:29.502624+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:31.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:31 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:31.172 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:20:31 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:31.172 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:20:31 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:20:32.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:32.330+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:32.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:32 smithi203 bash[23654]: audit 2024-03-11T21:20:31.241516+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:32 smithi203 bash[23654]: audit 2024-03-11T21:20:31.252787+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:32 smithi203 bash[23654]: audit 2024-03-11T21:20:31.266297+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:32.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[21512]: audit 2024-03-11T21:20:31.241516+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[21512]: audit 2024-03-11T21:20:31.252787+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[21512]: audit 2024-03-11T21:20:31.266297+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:32.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[29284]: audit 2024-03-11T21:20:31.241516+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[29284]: audit 2024-03-11T21:20:31.252787+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:32 smithi143 bash[29284]: audit 2024-03-11T21:20:31.266297+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:33.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:33 smithi203 bash[23654]: cluster 2024-03-11T21:20:31.503165+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:33.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:33 smithi143 bash[21512]: cluster 2024-03-11T21:20:31.503165+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:33.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:33 smithi143 bash[29284]: cluster 2024-03-11T21:20:31.503165+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:35.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:35 smithi203 bash[23654]: cluster 2024-03-11T21:20:33.503733+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:35.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:35 smithi143 bash[21512]: cluster 2024-03-11T21:20:33.503733+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:35.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:35 smithi143 bash[29284]: cluster 2024-03-11T21:20:33.503733+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:37.317 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:37.334+0000 7f29df92b700 10 client.24214.objecter tick 2024-03-11T21:20:37.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:37 smithi203 bash[23654]: cluster 2024-03-11T21:20:35.504267+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:37.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:37 smithi143 bash[29284]: cluster 2024-03-11T21:20:35.504267+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:37.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:37 smithi143 bash[21512]: cluster 2024-03-11T21:20:35.504267+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:37.305105+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:37.308053+0000 mon.a (mon.0) 509 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:38.013748+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:38.025306+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:38.027402+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:38.029227+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:38.499 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:38 smithi203 bash[23654]: audit 2024-03-11T21:20:38.040975+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:37.305105+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:37.308053+0000 mon.a (mon.0) 509 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:38.013748+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:38.025306+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:38.027402+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:38.029227+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[21512]: audit 2024-03-11T21:20:38.040975+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:37.305105+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:37.308053+0000 mon.a (mon.0) 509 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:38.013748+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:38.025306+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:38.027402+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:38.029227+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:38.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:38 smithi143 bash[29284]: audit 2024-03-11T21:20:38.040975+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: cluster 2024-03-11T21:20:37.504761+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: audit 2024-03-11T21:20:38.278664+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: audit 2024-03-11T21:20:38.282540+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: cluster 2024-03-11T21:20:38.284051+0000 mon.a (mon.0) 516 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: audit 2024-03-11T21:20:38.284624+0000 mon.a (mon.0) 517 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:39.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:39 smithi203 bash[23654]: audit 2024-03-11T21:20:38.285740+0000 mon.a (mon.0) 518 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: cluster 2024-03-11T21:20:37.504761+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: audit 2024-03-11T21:20:38.278664+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: audit 2024-03-11T21:20:38.282540+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: cluster 2024-03-11T21:20:38.284051+0000 mon.a (mon.0) 516 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: audit 2024-03-11T21:20:38.284624+0000 mon.a (mon.0) 517 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:39.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[21512]: audit 2024-03-11T21:20:38.285740+0000 mon.a (mon.0) 518 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: cluster 2024-03-11T21:20:37.504761+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: audit 2024-03-11T21:20:38.278664+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: audit 2024-03-11T21:20:38.282540+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: cluster 2024-03-11T21:20:38.284051+0000 mon.a (mon.0) 516 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: audit 2024-03-11T21:20:38.284624+0000 mon.a (mon.0) 517 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:39.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:39 smithi143 bash[29284]: audit 2024-03-11T21:20:38.285740+0000 mon.a (mon.0) 518 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:20:40.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:40 smithi203 bash[23654]: audit 2024-03-11T21:20:39.281040+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:20:40.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:40 smithi203 bash[23654]: cluster 2024-03-11T21:20:39.287559+0000 mon.a (mon.0) 520 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-03-11T21:20:40.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:40 smithi203 bash[23654]: audit 2024-03-11T21:20:39.288480+0000 mon.a (mon.0) 521 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:40 smithi203 bash[23654]: audit 2024-03-11T21:20:39.298800+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:40 smithi203 bash[23654]: audit 2024-03-11T21:20:39.755440+0000 mon.a (mon.0) 523 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[21512]: audit 2024-03-11T21:20:39.281040+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[21512]: cluster 2024-03-11T21:20:39.287559+0000 mon.a (mon.0) 520 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[21512]: audit 2024-03-11T21:20:39.288480+0000 mon.a (mon.0) 521 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[21512]: audit 2024-03-11T21:20:39.298800+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[21512]: audit 2024-03-11T21:20:39.755440+0000 mon.a (mon.0) 523 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[29284]: audit 2024-03-11T21:20:39.281040+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[29284]: cluster 2024-03-11T21:20:39.287559+0000 mon.a (mon.0) 520 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-03-11T21:20:40.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[29284]: audit 2024-03-11T21:20:39.288480+0000 mon.a (mon.0) 521 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[29284]: audit 2024-03-11T21:20:39.298800+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:40.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:40 smithi143 bash[29284]: audit 2024-03-11T21:20:39.755440+0000 mon.a (mon.0) 523 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T21:20:40.876 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.890+0000 7f29ceffd700 1 -- 172.21.15.203:0/3000375260 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f29d802dc70 con 0x7f29c406f600 2024-03-11T21:20:40.876 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-03-11T21:20:40.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 10 librados: watch_flush enter 2024-03-11T21:20:40.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 10 librados: watch_flush exit 2024-03-11T21:20:40.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 20 client.24214.objecter shutdown clearing up homeless session... 2024-03-11T21:20:40.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 10 client.24214.objecter successfully canceled tick 2024-03-11T21:20:40.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 msgr2=0x7f29c4071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:40.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 0x7f29c4071ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29c8005950 tx=0x7f29c800a300 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 msgr2=0x7f29d81a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29d0007cf0 tx=0x7f29d0007d20 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 shutdown_connections 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f29c406f600 0x7f29c4071ac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8108480 0x7f29d819dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29d810a020 0x7f29d81a51d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 --2- 172.21.15.203:0/3000375260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29d8109680 0x7f29d819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 >> 172.21.15.203:0/3000375260 conn(0x7f29d8077470 msgr2=0x7f29d8078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 shutdown_connections 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 -- 172.21.15.203:0/3000375260 wait complete. 2024-03-11T21:20:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:40.894+0000 7f29e012c700 1 librados: shutdown 2024-03-11T21:20:41.452 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T21:20:41.455 INFO:tasks.cephadm:Deploying osd.6 on smithi203 with /dev/vg_nvme/lv_2... 2024-03-11T21:20:41.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:38.300306+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:38.300441+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:39.505301+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:40.291322+0000 mon.a (mon.0) 524 : cluster 1 osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] boot 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:40.291413+0000 mon.a (mon.0) 525 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T21:20:41.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: audit 2024-03-11T21:20:40.291759+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: audit 2024-03-11T21:20:40.852464+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: audit 2024-03-11T21:20:40.861711+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: audit 2024-03-11T21:20:40.884104+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[21512]: cluster 2024-03-11T21:20:41.291908+0000 mon.a (mon.0) 530 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:38.300306+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:38.300441+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:39.505301+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:40.291322+0000 mon.a (mon.0) 524 : cluster 1 osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] boot 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:40.291413+0000 mon.a (mon.0) 525 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: audit 2024-03-11T21:20:40.291759+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: audit 2024-03-11T21:20:40.852464+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: audit 2024-03-11T21:20:40.861711+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: audit 2024-03-11T21:20:40.884104+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:41.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:41 smithi143 bash[29284]: cluster 2024-03-11T21:20:41.291908+0000 mon.a (mon.0) 530 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:38.300306+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:38.300441+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:39.505301+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:40.291322+0000 mon.a (mon.0) 524 : cluster 1 osd.5 [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] boot 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:40.291413+0000 mon.a (mon.0) 525 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: audit 2024-03-11T21:20:40.291759+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: audit 2024-03-11T21:20:40.852464+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: audit 2024-03-11T21:20:40.861711+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:41.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: audit 2024-03-11T21:20:40.884104+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:41.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:41 smithi203 bash[23654]: cluster 2024-03-11T21:20:41.291908+0000 mon.a (mon.0) 530 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T21:20:43.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:43 smithi203 bash[23654]: cluster 2024-03-11T21:20:41.505850+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:43.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:43 smithi203 bash[23654]: cluster 2024-03-11T21:20:41.921241+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:43.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:43 smithi203 bash[23654]: cluster 2024-03-11T21:20:41.921335+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:43.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:43 smithi203 bash[23654]: cluster 2024-03-11T21:20:42.293718+0000 mon.a (mon.0) 531 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[21512]: cluster 2024-03-11T21:20:41.505850+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[21512]: cluster 2024-03-11T21:20:41.921241+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[21512]: cluster 2024-03-11T21:20:41.921335+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[21512]: cluster 2024-03-11T21:20:42.293718+0000 mon.a (mon.0) 531 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[29284]: cluster 2024-03-11T21:20:41.505850+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[29284]: cluster 2024-03-11T21:20:41.921241+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[29284]: cluster 2024-03-11T21:20:41.921335+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:43.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:43 smithi143 bash[29284]: cluster 2024-03-11T21:20:42.293718+0000 mon.a (mon.0) 531 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T21:20:44.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:44 smithi143 bash[21512]: cluster 2024-03-11T21:20:43.315143+0000 mon.a (mon.0) 532 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T21:20:44.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:44 smithi143 bash[29284]: cluster 2024-03-11T21:20:43.315143+0000 mon.a (mon.0) 532 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T21:20:44.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:44 smithi203 bash[23654]: cluster 2024-03-11T21:20:43.315143+0000 mon.a (mon.0) 532 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[21512]: cluster 2024-03-11T21:20:43.506445+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[21512]: cluster 2024-03-11T21:20:44.308028+0000 mon.a (mon.0) 533 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[21512]: cluster 2024-03-11T21:20:44.324323+0000 mon.a (mon.0) 534 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[29284]: cluster 2024-03-11T21:20:43.506445+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[29284]: cluster 2024-03-11T21:20:44.308028+0000 mon.a (mon.0) 533 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:20:45.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:45 smithi143 bash[29284]: cluster 2024-03-11T21:20:44.324323+0000 mon.a (mon.0) 534 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T21:20:45.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:45 smithi203 bash[23654]: cluster 2024-03-11T21:20:43.506445+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 222 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:45.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:45 smithi203 bash[23654]: cluster 2024-03-11T21:20:44.308028+0000 mon.a (mon.0) 533 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:20:45.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:45 smithi203 bash[23654]: cluster 2024-03-11T21:20:44.324323+0000 mon.a (mon.0) 534 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T21:20:45.828 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:20:46.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:46 smithi143 bash[21512]: cluster 2024-03-11T21:20:45.353148+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:20:46.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:46 smithi143 bash[21512]: cluster 2024-03-11T21:20:45.355527+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:20:46.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:46 smithi143 bash[29284]: cluster 2024-03-11T21:20:45.353148+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:20:46.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:46 smithi143 bash[29284]: cluster 2024-03-11T21:20:45.355527+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:20:46.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:46 smithi203 bash[23654]: cluster 2024-03-11T21:20:45.353148+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:20:46.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:46 smithi203 bash[23654]: cluster 2024-03-11T21:20:45.355527+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:47 smithi203 bash[23654]: cluster 2024-03-11T21:20:45.507039+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:47 smithi203 bash[23654]: cluster 2024-03-11T21:20:46.318938+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:20:47.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:47 smithi203 bash[23654]: cluster 2024-03-11T21:20:46.318996+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[21512]: cluster 2024-03-11T21:20:45.507039+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[21512]: cluster 2024-03-11T21:20:46.318938+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[21512]: cluster 2024-03-11T21:20:46.318996+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[29284]: cluster 2024-03-11T21:20:45.507039+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[29284]: cluster 2024-03-11T21:20:46.318938+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:20:47.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:47 smithi143 bash[29284]: cluster 2024-03-11T21:20:46.318996+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2024-03-11T21:20:49.493 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T21:20:49.493 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T21:20:49.493 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-11T21:20:49.493 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-11T21:20:49.493 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134585 s, 779 MB/s 2024-03-11T21:20:49.494 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-11T21:20:49.566 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:49 smithi203 bash[23654]: cluster 2024-03-11T21:20:47.507632+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:49.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:49 smithi143 bash[21512]: cluster 2024-03-11T21:20:47.507632+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:49.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:49 smithi143 bash[29284]: cluster 2024-03-11T21:20:47.507632+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:50.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-03-11T21:20:51.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: cluster 2024-03-11T21:20:49.508248+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:20:51.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.690271+0000 mon.a (mon.0) 537 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.701609+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.703914+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.705701+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.716155+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.721034+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.722811+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[21512]: audit 2024-03-11T21:20:50.735105+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: cluster 2024-03-11T21:20:49.508248+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.690271+0000 mon.a (mon.0) 537 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.701609+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.703914+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.705701+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.716155+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.721034+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:51.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.722811+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:51.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:51 smithi143 bash[29284]: audit 2024-03-11T21:20:50.735105+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: cluster 2024-03-11T21:20:49.508248+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.690271+0000 mon.a (mon.0) 537 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.701609+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.703914+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.705701+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.716155+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.721034+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.722811+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:51.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:51 smithi203 bash[23654]: audit 2024-03-11T21:20:50.735105+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:20:52.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:52 smithi143 bash[21512]: cephadm 2024-03-11T21:20:50.679431+0000 mgr.y (mgr.14152) 262 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:52.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:52 smithi143 bash[21512]: cephadm 2024-03-11T21:20:50.707135+0000 mgr.y (mgr.14152) 263 : cephadm 1 Adjusting osd_memory_target on smithi203 to 8632M 2024-03-11T21:20:52.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:52 smithi143 bash[29284]: cephadm 2024-03-11T21:20:50.679431+0000 mgr.y (mgr.14152) 262 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:52.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:52 smithi143 bash[29284]: cephadm 2024-03-11T21:20:50.707135+0000 mgr.y (mgr.14152) 263 : cephadm 1 Adjusting osd_memory_target on smithi203 to 8632M 2024-03-11T21:20:52.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:52 smithi203 bash[23654]: cephadm 2024-03-11T21:20:50.679431+0000 mgr.y (mgr.14152) 262 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:20:52.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:52 smithi203 bash[23654]: cephadm 2024-03-11T21:20:50.707135+0000 mgr.y (mgr.14152) 263 : cephadm 1 Adjusting osd_memory_target on smithi203 to 8632M 2024-03-11T21:20:53.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:53 smithi143 bash[29284]: cluster 2024-03-11T21:20:51.508804+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-11T21:20:53.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:53 smithi143 bash[21512]: cluster 2024-03-11T21:20:51.508804+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-11T21:20:53.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:53 smithi203 bash[23654]: cluster 2024-03-11T21:20:51.508804+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-11T21:20:55.009 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:20:55.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:55 smithi143 bash[21512]: cluster 2024-03-11T21:20:53.509377+0000 mgr.y (mgr.14152) 265 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:55 smithi143 bash[29284]: cluster 2024-03-11T21:20:53.509377+0000 mgr.y (mgr.14152) 265 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:55.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:55 smithi203 bash[23654]: cluster 2024-03-11T21:20:53.509377+0000 mgr.y (mgr.14152) 265 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- 172.21.15.203:0/239859941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 msgr2=0x7f82e8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- 172.21.15.203:0/239859941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8100b10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b0a0 tx=0x7f82e000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- 172.21.15.203:0/239859941 shutdown_connections 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- 172.21.15.203:0/239859941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101a10 0x7f82e8101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- 172.21.15.203:0/239859941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8100b10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- 172.21.15.203:0/239859941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101050 0x7f82e81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- 172.21.15.203:0/239859941 >> 172.21.15.203:0/239859941 conn(0x7f82e80fc310 msgr2=0x7f82e80fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- 172.21.15.203:0/239859941 shutdown_connections 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- 172.21.15.203:0/239859941 wait complete. 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 librados: starting msgr at 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 librados: starting objecter 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 Processor -- start 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- start start 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 librados: setting wanted keys 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 librados: calling monclient init 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 0x7f82e8199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101a10 0x7f82e81a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:57.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e810f1b0 con 0x7f82e8101050 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e810f030 con 0x7f82e81006f0 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ef45d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e810f330 con 0x7f82e8101a10 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45182/0 (socket says 172.21.15.203:45182) 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 -- 172.21.15.203:0/1287735711 learned_addr learned my addr 172.21.15.203:0/1287735711 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed9fa700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101a10 0x7f82e81a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:57.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ec9f8700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 0x7f82e8199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 -- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101a10 msgr2=0x7f82e81a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101a10 0x7f82e81a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 -- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 msgr2=0x7f82e8199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 0x7f82e8199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 -- 172.21.15.203:0/1287735711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e81a1530 con 0x7f82e81006f0 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ec9f8700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 0x7f82e8199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82ed1f9700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009610 tx=0x7f82e0004e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82de7fc700 10 client.?.objecter ms_handle_connect 0x7f82e81006f0 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82de7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.202+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0021070 con 0x7f82e81006f0 2024-03-11T21:20:57.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e81a17c0 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f82e0007c80 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0005340 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ef45d700 10 client.24241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e81a1d40 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ef45d700 1 librados: init done 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e804faa0 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f82e0024070 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 0x7f82d4071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4423+0+0 (secure 0 0 0) 0x7f82e000f020 con 0x7f82e81006f0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 10 client.24241.objecter ms_dispatch 0x7f82e8001550 osd_map(50..50 src has 1..50) v4 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 3 client.24241.objecter handle_osd_map got epochs [50,50] > 0 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 3 client.24241.objecter handle_osd_map decoding full epoch 50 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 20 client.24241.objecter dump_active .. 0 homeless 2024-03-11T21:20:57.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ec9f8700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 0x7f82d4071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:20:57.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82de7fc700 10 client.24241.objecter ms_handle_connect 0x7f82d406f730 2024-03-11T21:20:57.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.206+0000 7f82ec9f8700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 0x7f82d4071bf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007900 tx=0x7f82d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:20:57.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.210+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f82e002b020 con 0x7f82e81006f0 2024-03-11T21:20:57.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:20:57.430+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f82e8062220 con 0x7f82d406f730 2024-03-11T21:20:57.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:57 smithi143 bash[21512]: cluster 2024-03-11T21:20:55.509980+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:20:57.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:57 smithi143 bash[29284]: cluster 2024-03-11T21:20:55.509980+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:20:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:57 smithi203 bash[23654]: cluster 2024-03-11T21:20:55.509980+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:20:58.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[21512]: audit 2024-03-11T21:20:57.447390+0000 mon.a (mon.0) 545 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:58.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[21512]: audit 2024-03-11T21:20:57.453875+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:58.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[21512]: audit 2024-03-11T21:20:57.455597+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[29284]: audit 2024-03-11T21:20:57.447390+0000 mon.a (mon.0) 545 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[29284]: audit 2024-03-11T21:20:57.453875+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:58 smithi143 bash[29284]: audit 2024-03-11T21:20:57.455597+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:58 smithi203 bash[23654]: audit 2024-03-11T21:20:57.447390+0000 mon.a (mon.0) 545 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:20:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:58 smithi203 bash[23654]: audit 2024-03-11T21:20:57.453875+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:20:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:58 smithi203 bash[23654]: audit 2024-03-11T21:20:57.455597+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:59.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:59 smithi143 bash[21512]: audit 2024-03-11T21:20:57.440178+0000 mgr.y (mgr.14152) 267 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:59.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:20:59 smithi143 bash[21512]: cluster 2024-03-11T21:20:57.510618+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:59.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:59 smithi143 bash[29284]: audit 2024-03-11T21:20:57.440178+0000 mgr.y (mgr.14152) 267 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:59.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:20:59 smithi143 bash[29284]: cluster 2024-03-11T21:20:57.510618+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:20:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:59 smithi203 bash[23654]: audit 2024-03-11T21:20:57.440178+0000 mgr.y (mgr.14152) 267 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:20:59 smithi203 bash[23654]: cluster 2024-03-11T21:20:57.510618+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:01.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:01 smithi143 bash[21512]: cluster 2024-03-11T21:20:59.511239+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:01.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:01 smithi143 bash[29284]: cluster 2024-03-11T21:20:59.511239+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:01.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:01 smithi203 bash[23654]: cluster 2024-03-11T21:20:59.511239+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:02.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:02.210+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:03.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:03 smithi143 bash[29284]: cluster 2024-03-11T21:21:01.511808+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:03.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:03 smithi143 bash[21512]: cluster 2024-03-11T21:21:01.511808+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:03.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:03 smithi203 bash[23654]: cluster 2024-03-11T21:21:01.511808+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:05.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: cluster 2024-03-11T21:21:03.512375+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: audit 2024-03-11T21:21:05.351812+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.203:0/1483580193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: audit 2024-03-11T21:21:05.354909+0000 mon.a (mon.0) 548 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: audit 2024-03-11T21:21:05.358571+0000 mon.a (mon.0) 549 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]': finished 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: cluster 2024-03-11T21:21:05.364821+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 7 total, 6 up, 7 in 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[21512]: audit 2024-03-11T21:21:05.365469+0000 mon.a (mon.0) 551 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: cluster 2024-03-11T21:21:03.512375+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: audit 2024-03-11T21:21:05.351812+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.203:0/1483580193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: audit 2024-03-11T21:21:05.354909+0000 mon.a (mon.0) 548 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: audit 2024-03-11T21:21:05.358571+0000 mon.a (mon.0) 549 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]': finished 2024-03-11T21:21:05.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: cluster 2024-03-11T21:21:05.364821+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 7 total, 6 up, 7 in 2024-03-11T21:21:05.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:05 smithi143 bash[29284]: audit 2024-03-11T21:21:05.365469+0000 mon.a (mon.0) 551 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: cluster 2024-03-11T21:21:03.512375+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: audit 2024-03-11T21:21:05.351812+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.203:0/1483580193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: audit 2024-03-11T21:21:05.354909+0000 mon.a (mon.0) 548 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]: dispatch 2024-03-11T21:21:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: audit 2024-03-11T21:21:05.358571+0000 mon.a (mon.0) 549 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241"}]': finished 2024-03-11T21:21:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: cluster 2024-03-11T21:21:05.364821+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 7 total, 6 up, 7 in 2024-03-11T21:21:05.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:05 smithi203 bash[23654]: audit 2024-03-11T21:21:05.365469+0000 mon.a (mon.0) 551 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:06.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:06 smithi143 bash[21512]: audit 2024-03-11T21:21:06.072810+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.203:0/237240326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:06.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:06 smithi143 bash[29284]: audit 2024-03-11T21:21:06.072810+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.203:0/237240326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:06 smithi203 bash[23654]: audit 2024-03-11T21:21:06.072810+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.203:0/237240326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:07.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:07.214+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:07.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:07 smithi143 bash[21512]: cluster 2024-03-11T21:21:05.512882+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:07.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:07 smithi143 bash[29284]: cluster 2024-03-11T21:21:05.512882+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:07.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:07 smithi203 bash[23654]: cluster 2024-03-11T21:21:05.512882+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:09.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:09 smithi143 bash[21512]: cluster 2024-03-11T21:21:07.513472+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:09.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:09 smithi143 bash[29284]: cluster 2024-03-11T21:21:07.513472+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:09.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:09 smithi203 bash[23654]: cluster 2024-03-11T21:21:07.513472+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:11.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:11 smithi143 bash[21512]: cluster 2024-03-11T21:21:09.514137+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:11.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:11 smithi143 bash[29284]: cluster 2024-03-11T21:21:09.514137+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:11.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:11 smithi203 bash[23654]: cluster 2024-03-11T21:21:09.514137+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:12.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:12.218+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:13.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:13 smithi143 bash[21512]: cluster 2024-03-11T21:21:11.514746+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:13.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:13 smithi143 bash[29284]: cluster 2024-03-11T21:21:11.514746+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:13.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:13 smithi203 bash[23654]: cluster 2024-03-11T21:21:11.514746+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:14 smithi203 bash[23654]: cluster 2024-03-11T21:21:13.498481+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:14 smithi203 bash[23654]: cluster 2024-03-11T21:21:13.498595+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:14.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:14 smithi143 bash[21512]: cluster 2024-03-11T21:21:13.498481+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:14.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:14 smithi143 bash[21512]: cluster 2024-03-11T21:21:13.498595+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:14.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:14 smithi143 bash[29284]: cluster 2024-03-11T21:21:13.498481+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:14.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:14 smithi143 bash[29284]: cluster 2024-03-11T21:21:13.498595+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:15.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:15 smithi203 bash[23654]: cluster 2024-03-11T21:21:13.515319+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:15.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:15 smithi203 bash[23654]: cluster 2024-03-11T21:21:13.922864+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:15.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:15 smithi203 bash[23654]: cluster 2024-03-11T21:21:13.923018+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:15.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:15 smithi203 bash[23654]: cluster 2024-03-11T21:21:14.424880+0000 mon.a (mon.0) 552 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T21:21:15.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:15 smithi203 bash[23654]: audit 2024-03-11T21:21:14.425187+0000 mon.a (mon.0) 553 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:15.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[29284]: cluster 2024-03-11T21:21:13.515319+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:15.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[29284]: cluster 2024-03-11T21:21:13.922864+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[29284]: cluster 2024-03-11T21:21:13.923018+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[29284]: cluster 2024-03-11T21:21:14.424880+0000 mon.a (mon.0) 552 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[29284]: audit 2024-03-11T21:21:14.425187+0000 mon.a (mon.0) 553 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[21512]: cluster 2024-03-11T21:21:13.515319+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[21512]: cluster 2024-03-11T21:21:13.922864+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[21512]: cluster 2024-03-11T21:21:13.923018+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[21512]: cluster 2024-03-11T21:21:14.424880+0000 mon.a (mon.0) 552 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T21:21:15.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:15 smithi143 bash[21512]: audit 2024-03-11T21:21:14.425187+0000 mon.a (mon.0) 553 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:17.222+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:17.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[21512]: cluster 2024-03-11T21:21:15.515912+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:17.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[21512]: audit 2024-03-11T21:21:17.096568+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:21:17.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[21512]: audit 2024-03-11T21:21:17.098574+0000 mon.a (mon.0) 555 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:17.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[29284]: cluster 2024-03-11T21:21:15.515912+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:17.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[29284]: audit 2024-03-11T21:21:17.096568+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:21:17.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:17 smithi143 bash[29284]: audit 2024-03-11T21:21:17.098574+0000 mon.a (mon.0) 555 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:17.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:17 smithi203 bash[23654]: cluster 2024-03-11T21:21:15.515912+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:17.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:17 smithi203 bash[23654]: audit 2024-03-11T21:21:17.096568+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:21:17.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:17 smithi203 bash[23654]: audit 2024-03-11T21:21:17.098574+0000 mon.a (mon.0) 555 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:18.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:18 smithi143 bash[21512]: cephadm 2024-03-11T21:21:17.100317+0000 mgr.y (mgr.14152) 278 : cephadm 1 Deploying daemon osd.6 on smithi203 2024-03-11T21:21:18.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:18 smithi143 bash[29284]: cephadm 2024-03-11T21:21:17.100317+0000 mgr.y (mgr.14152) 278 : cephadm 1 Deploying daemon osd.6 on smithi203 2024-03-11T21:21:18.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:18 smithi203 bash[23654]: cephadm 2024-03-11T21:21:17.100317+0000 mgr.y (mgr.14152) 278 : cephadm 1 Deploying daemon osd.6 on smithi203 2024-03-11T21:21:19.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:19 smithi143 bash[21512]: cluster 2024-03-11T21:21:17.516470+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:19.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:19 smithi143 bash[29284]: cluster 2024-03-11T21:21:17.516470+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:19 smithi203 bash[23654]: cluster 2024-03-11T21:21:17.516470+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:20.681 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:20 smithi203 bash[23654]: cluster 2024-03-11T21:21:19.516967+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:20.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:20 smithi143 bash[21512]: cluster 2024-03-11T21:21:19.516967+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:20.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:20 smithi143 bash[29284]: cluster 2024-03-11T21:21:19.516967+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:21.024 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.024 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.024 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.024 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.025 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:21.025 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:21:20 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:21:22.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:22.226+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:22.346 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:22 smithi203 bash[23654]: audit 2024-03-11T21:21:21.090059+0000 mon.a (mon.0) 556 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.346 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:22 smithi203 bash[23654]: audit 2024-03-11T21:21:21.100832+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.346 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:22 smithi203 bash[23654]: audit 2024-03-11T21:21:21.104656+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[21512]: audit 2024-03-11T21:21:21.090059+0000 mon.a (mon.0) 556 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[21512]: audit 2024-03-11T21:21:21.100832+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[21512]: audit 2024-03-11T21:21:21.104656+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:22.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[29284]: audit 2024-03-11T21:21:21.090059+0000 mon.a (mon.0) 556 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[29284]: audit 2024-03-11T21:21:21.100832+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:22.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:22 smithi143 bash[29284]: audit 2024-03-11T21:21:21.104656+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:23.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:23 smithi203 bash[23654]: cluster 2024-03-11T21:21:21.517530+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:23.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:23 smithi143 bash[21512]: cluster 2024-03-11T21:21:21.517530+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:23.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:23 smithi143 bash[29284]: cluster 2024-03-11T21:21:21.517530+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:25.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:25 smithi143 bash[21512]: cluster 2024-03-11T21:21:23.518094+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:25.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:25 smithi143 bash[29284]: cluster 2024-03-11T21:21:23.518094+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:25.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:25 smithi203 bash[23654]: cluster 2024-03-11T21:21:23.518094+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:27.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:27.230+0000 7f82eec5c700 10 client.24241.objecter tick 2024-03-11T21:21:27.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:27 smithi203 bash[23654]: cluster 2024-03-11T21:21:25.518680+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:27.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:27 smithi143 bash[21512]: cluster 2024-03-11T21:21:25.518680+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:27.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:27 smithi143 bash[29284]: cluster 2024-03-11T21:21:25.518680+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: cluster 2024-03-11T21:21:26.933158+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: cluster 2024-03-11T21:21:26.933246+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: cluster 2024-03-11T21:21:27.115798+0000 mon.a (mon.0) 559 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: audit 2024-03-11T21:21:27.116459+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: audit 2024-03-11T21:21:27.127522+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: audit 2024-03-11T21:21:27.130570+0000 mon.a (mon.0) 561 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: audit 2024-03-11T21:21:27.807385+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:28 smithi203 bash[23654]: audit 2024-03-11T21:21:27.819789+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: cluster 2024-03-11T21:21:26.933158+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: cluster 2024-03-11T21:21:26.933246+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: cluster 2024-03-11T21:21:27.115798+0000 mon.a (mon.0) 559 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: audit 2024-03-11T21:21:27.116459+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: audit 2024-03-11T21:21:27.127522+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: audit 2024-03-11T21:21:27.130570+0000 mon.a (mon.0) 561 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: audit 2024-03-11T21:21:27.807385+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[21512]: audit 2024-03-11T21:21:27.819789+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:28.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: cluster 2024-03-11T21:21:26.933158+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: cluster 2024-03-11T21:21:26.933246+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: cluster 2024-03-11T21:21:27.115798+0000 mon.a (mon.0) 559 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: audit 2024-03-11T21:21:27.116459+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: audit 2024-03-11T21:21:27.127522+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: audit 2024-03-11T21:21:27.130570+0000 mon.a (mon.0) 561 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: audit 2024-03-11T21:21:27.807385+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:28.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:28 smithi143 bash[29284]: audit 2024-03-11T21:21:27.819789+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: cluster 2024-03-11T21:21:27.519233+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.130394+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.135060+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: cluster 2024-03-11T21:21:28.141849+0000 mon.a (mon.0) 565 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.142769+0000 mon.a (mon.0) 566 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.143443+0000 mon.a (mon.0) 567 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.498036+0000 mon.a (mon.0) 568 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.500158+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[21512]: audit 2024-03-11T21:21:28.515516+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: cluster 2024-03-11T21:21:27.519233+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.130394+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.135060+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: cluster 2024-03-11T21:21:28.141849+0000 mon.a (mon.0) 565 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.142769+0000 mon.a (mon.0) 566 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.143443+0000 mon.a (mon.0) 567 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.498036+0000 mon.a (mon.0) 568 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:29.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.500158+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:29.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:29 smithi143 bash[29284]: audit 2024-03-11T21:21:28.515516+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: cluster 2024-03-11T21:21:27.519233+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.130394+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.135060+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: cluster 2024-03-11T21:21:28.141849+0000 mon.a (mon.0) 565 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.142769+0000 mon.a (mon.0) 566 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.143443+0000 mon.a (mon.0) 567 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.498036+0000 mon.a (mon.0) 568 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.500158+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:29.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:29 smithi203 bash[23654]: audit 2024-03-11T21:21:28.515516+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:30.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: cluster 2024-03-11T21:21:28.095985+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:30.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: cluster 2024-03-11T21:21:28.096079+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:30.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: audit 2024-03-11T21:21:29.132790+0000 mon.a (mon.0) 571 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: cluster 2024-03-11T21:21:29.139748+0000 mon.a (mon.0) 572 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: audit 2024-03-11T21:21:29.140915+0000 mon.a (mon.0) 573 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[21512]: audit 2024-03-11T21:21:29.150271+0000 mon.a (mon.0) 574 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: cluster 2024-03-11T21:21:28.095985+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: cluster 2024-03-11T21:21:28.096079+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: audit 2024-03-11T21:21:29.132790+0000 mon.a (mon.0) 571 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: cluster 2024-03-11T21:21:29.139748+0000 mon.a (mon.0) 572 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T21:21:30.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: audit 2024-03-11T21:21:29.140915+0000 mon.a (mon.0) 573 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:30 smithi143 bash[29284]: audit 2024-03-11T21:21:29.150271+0000 mon.a (mon.0) 574 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: cluster 2024-03-11T21:21:28.095985+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: cluster 2024-03-11T21:21:28.096079+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: audit 2024-03-11T21:21:29.132790+0000 mon.a (mon.0) 571 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: cluster 2024-03-11T21:21:29.139748+0000 mon.a (mon.0) 572 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: audit 2024-03-11T21:21:29.140915+0000 mon.a (mon.0) 573 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:30 smithi203 bash[23654]: audit 2024-03-11T21:21:29.150271+0000 mon.a (mon.0) 574 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:30.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82de7fc700 1 -- 172.21.15.203:0/1287735711 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f82e8062220 con 0x7f82d406f730 2024-03-11T21:21:30.697 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 6 on host 'smithi203' 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 10 librados: watch_flush enter 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 10 librados: watch_flush exit 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 20 client.24241.objecter shutdown clearing up homeless session... 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 10 client.24241.objecter successfully canceled tick 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 msgr2=0x7f82d4071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 0x7f82d4071bf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007900 tx=0x7f82d8008040 comp rx=0 tx=0).stop 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 msgr2=0x7f82e8199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009610 tx=0x7f82e0004e20 comp rx=0 tx=0).stop 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 shutdown_connections 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f82d406f730 0x7f82d4071bf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:30.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e8101a10 0x7f82e81a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e81006f0 0x7f82e8199920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 --2- 172.21.15.203:0/1287735711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e8101050 0x7f82e8199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 >> 172.21.15.203:0/1287735711 conn(0x7f82e80fc310 msgr2=0x7f82e80fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.714+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 shutdown_connections 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.718+0000 7f82ef45d700 1 -- 172.21.15.203:0/1287735711 wait complete. 2024-03-11T21:21:30.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:30.718+0000 7f82ef45d700 1 librados: shutdown 2024-03-11T21:21:31.278 DEBUG:teuthology.orchestra.run.smithi203:osd.6> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T21:21:31.281 INFO:tasks.cephadm:Deploying osd.7 on smithi203 with /dev/vg_nvme/lv_1... 2024-03-11T21:21:31.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T21:21:31.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: cluster 2024-03-11T21:21:29.519789+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:31.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: cluster 2024-03-11T21:21:30.142472+0000 mon.a (mon.0) 575 : cluster 1 osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] boot 2024-03-11T21:21:31.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: cluster 2024-03-11T21:21:30.143018+0000 mon.a (mon.0) 576 : cluster 0 osdmap e56: 7 total, 7 up, 7 in 2024-03-11T21:21:31.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: audit 2024-03-11T21:21:30.143435+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: audit 2024-03-11T21:21:30.697925+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: audit 2024-03-11T21:21:30.707898+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[21512]: audit 2024-03-11T21:21:30.708984+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: cluster 2024-03-11T21:21:29.519789+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: cluster 2024-03-11T21:21:30.142472+0000 mon.a (mon.0) 575 : cluster 1 osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] boot 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: cluster 2024-03-11T21:21:30.143018+0000 mon.a (mon.0) 576 : cluster 0 osdmap e56: 7 total, 7 up, 7 in 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: audit 2024-03-11T21:21:30.143435+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: audit 2024-03-11T21:21:30.697925+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: audit 2024-03-11T21:21:30.707898+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:31 smithi143 bash[29284]: audit 2024-03-11T21:21:30.708984+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:31.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: cluster 2024-03-11T21:21:29.519789+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:21:31.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: cluster 2024-03-11T21:21:30.142472+0000 mon.a (mon.0) 575 : cluster 1 osd.6 [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] boot 2024-03-11T21:21:31.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: cluster 2024-03-11T21:21:30.143018+0000 mon.a (mon.0) 576 : cluster 0 osdmap e56: 7 total, 7 up, 7 in 2024-03-11T21:21:31.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: audit 2024-03-11T21:21:30.143435+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:21:31.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: audit 2024-03-11T21:21:30.697925+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: audit 2024-03-11T21:21:30.707898+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:31.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:31 smithi203 bash[23654]: audit 2024-03-11T21:21:30.708984+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:32.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:32 smithi143 bash[21512]: cluster 2024-03-11T21:21:31.154983+0000 mon.a (mon.0) 581 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T21:21:32.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:32 smithi143 bash[21512]: cluster 2024-03-11T21:21:32.157023+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T21:21:32.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:32 smithi143 bash[29284]: cluster 2024-03-11T21:21:31.154983+0000 mon.a (mon.0) 581 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T21:21:32.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:32 smithi143 bash[29284]: cluster 2024-03-11T21:21:32.157023+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T21:21:32.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:32 smithi203 bash[23654]: cluster 2024-03-11T21:21:31.154983+0000 mon.a (mon.0) 581 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T21:21:32.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:32 smithi203 bash[23654]: cluster 2024-03-11T21:21:32.157023+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T21:21:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:33 smithi143 bash[21512]: cluster 2024-03-11T21:21:31.520333+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:33.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:33 smithi143 bash[29284]: cluster 2024-03-11T21:21:31.520333+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:33.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:33 smithi203 bash[23654]: cluster 2024-03-11T21:21:31.520333+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:34.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:34 smithi143 bash[21512]: cluster 2024-03-11T21:21:33.184472+0000 mon.a (mon.0) 583 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T21:21:34.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:34 smithi143 bash[29284]: cluster 2024-03-11T21:21:33.184472+0000 mon.a (mon.0) 583 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T21:21:34.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:34 smithi203 bash[23654]: cluster 2024-03-11T21:21:33.184472+0000 mon.a (mon.0) 583 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T21:21:35.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:35 smithi143 bash[21512]: cluster 2024-03-11T21:21:33.520889+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:35.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:35 smithi143 bash[21512]: cluster 2024-03-11T21:21:34.197728+0000 mon.a (mon.0) 584 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T21:21:35.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:35 smithi143 bash[29284]: cluster 2024-03-11T21:21:33.520889+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:35.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:35 smithi143 bash[29284]: cluster 2024-03-11T21:21:34.197728+0000 mon.a (mon.0) 584 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T21:21:35.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:35 smithi203 bash[23654]: cluster 2024-03-11T21:21:33.520889+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:35.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:35 smithi203 bash[23654]: cluster 2024-03-11T21:21:34.197728+0000 mon.a (mon.0) 584 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T21:21:35.649 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:21:37.265 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:37 smithi203 bash[23654]: cluster 2024-03-11T21:21:35.521454+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:21:37.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:37 smithi143 bash[21512]: cluster 2024-03-11T21:21:35.521454+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:21:37.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:37 smithi143 bash[29284]: cluster 2024-03-11T21:21:35.521454+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:21:39.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:39 smithi143 bash[21512]: cluster 2024-03-11T21:21:37.522018+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:21:39.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:39 smithi143 bash[29284]: cluster 2024-03-11T21:21:37.522018+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:21:39.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:39 smithi203 bash[23654]: cluster 2024-03-11T21:21:37.522018+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:21:39.544 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T21:21:39.544 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T21:21:39.544 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-11T21:21:39.544 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-11T21:21:39.545 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.017019 s, 616 MB/s 2024-03-11T21:21:39.545 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-11T21:21:40.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: cluster 2024-03-11T21:21:39.522569+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.613982+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.624635+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.627171+0000 mon.a (mon.0) 587 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.628828+0000 mon.a (mon.0) 588 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.630661+0000 mon.a (mon.0) 589 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.638458+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.643057+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.644911+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[21512]: audit 2024-03-11T21:21:40.654971+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: cluster 2024-03-11T21:21:39.522569+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.613982+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.624635+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.627171+0000 mon.a (mon.0) 587 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.628828+0000 mon.a (mon.0) 588 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.630661+0000 mon.a (mon.0) 589 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.638458+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.643057+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:41.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.644911+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:41.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:41 smithi143 bash[29284]: audit 2024-03-11T21:21:40.654971+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: cluster 2024-03-11T21:21:39.522569+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.613982+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.624635+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.627171+0000 mon.a (mon.0) 587 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.628828+0000 mon.a (mon.0) 588 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.630661+0000 mon.a (mon.0) 589 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.638458+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.643057+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.644911+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:41.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:41 smithi203 bash[23654]: audit 2024-03-11T21:21:40.654971+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:21:42.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:42 smithi203 bash[23654]: cephadm 2024-03-11T21:21:40.605109+0000 mgr.y (mgr.14152) 291 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:21:42.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:42 smithi203 bash[23654]: cephadm 2024-03-11T21:21:40.631990+0000 mgr.y (mgr.14152) 292 : cephadm 1 Adjusting osd_memory_target on smithi203 to 5755M 2024-03-11T21:21:42.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:42 smithi143 bash[21512]: cephadm 2024-03-11T21:21:40.605109+0000 mgr.y (mgr.14152) 291 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:21:42.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:42 smithi143 bash[21512]: cephadm 2024-03-11T21:21:40.631990+0000 mgr.y (mgr.14152) 292 : cephadm 1 Adjusting osd_memory_target on smithi203 to 5755M 2024-03-11T21:21:42.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:42 smithi143 bash[29284]: cephadm 2024-03-11T21:21:40.605109+0000 mgr.y (mgr.14152) 291 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:21:42.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:42 smithi143 bash[29284]: cephadm 2024-03-11T21:21:40.631990+0000 mgr.y (mgr.14152) 292 : cephadm 1 Adjusting osd_memory_target on smithi203 to 5755M 2024-03-11T21:21:43.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:43 smithi203 bash[23654]: cluster 2024-03-11T21:21:41.523207+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T21:21:43.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:43 smithi143 bash[21512]: cluster 2024-03-11T21:21:41.523207+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T21:21:43.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:43 smithi143 bash[29284]: cluster 2024-03-11T21:21:41.523207+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T21:21:44.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:44 smithi203 bash[23654]: cluster 2024-03-11T21:21:42.719712+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:44 smithi203 bash[23654]: cluster 2024-03-11T21:21:42.719832+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:44 smithi203 bash[23654]: cluster 2024-03-11T21:21:43.040062+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:44 smithi203 bash[23654]: cluster 2024-03-11T21:21:43.040159+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:44 smithi203 bash[23654]: cluster 2024-03-11T21:21:43.227416+0000 mon.a (mon.0) 594 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T21:21:44.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[21512]: cluster 2024-03-11T21:21:42.719712+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[21512]: cluster 2024-03-11T21:21:42.719832+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[21512]: cluster 2024-03-11T21:21:43.040062+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[21512]: cluster 2024-03-11T21:21:43.040159+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[21512]: cluster 2024-03-11T21:21:43.227416+0000 mon.a (mon.0) 594 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[29284]: cluster 2024-03-11T21:21:42.719712+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[29284]: cluster 2024-03-11T21:21:42.719832+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[29284]: cluster 2024-03-11T21:21:43.040062+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:44.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[29284]: cluster 2024-03-11T21:21:43.040159+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:44.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:44 smithi143 bash[29284]: cluster 2024-03-11T21:21:43.227416+0000 mon.a (mon.0) 594 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T21:21:45.203 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:21:45.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:45 smithi203 bash[23654]: cluster 2024-03-11T21:21:43.523582+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-11T21:21:45.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:45 smithi143 bash[29284]: cluster 2024-03-11T21:21:43.523582+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-11T21:21:45.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:45 smithi143 bash[21512]: cluster 2024-03-11T21:21:43.523582+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-11T21:21:47.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/2419849361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 msgr2=0x7f51141044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:21:47.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- 172.21.15.203:0/2419849361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f51141044a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50fc0099c0 tx=0x7f50fc009cd0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/2419849361 shutdown_connections 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- 172.21.15.203:0/2419849361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105c20 0x7f511410c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- 172.21.15.203:0/2419849361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f51141044a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- 172.21.15.203:0/2419849361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105280 0x7f51141056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/2419849361 >> 172.21.15.203:0/2419849361 conn(0x7f51140ff640 msgr2=0x7f5114101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/2419849361 shutdown_connections 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/2419849361 wait complete. 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 librados: starting msgr at 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 librados: starting objecter 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 Processor -- start 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- start start 2024-03-11T21:21:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 librados: setting wanted keys 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 librados: calling monclient init 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105280 0x7f5114199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 0x7f51141a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511410f250 con 0x7f5114105c20 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511410f0d0 con 0x7f5114104080 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511410f3d0 con 0x7f5114105280 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:42070/0 (socket says 172.21.15.203:42070) 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 -- 172.21.15.203:0/3753308448 learned_addr learned my addr 172.21.15.203:0/3753308448 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5111d9b700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105280 0x7f5114199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5112d9d700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 0x7f51141a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:21:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 -- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105280 msgr2=0x7f5114199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105280 0x7f5114199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 -- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 msgr2=0x7f51141a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 0x7f51141a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 -- 172.21.15.203:0/3753308448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51141a1530 con 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5112d9d700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 0x7f51141a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f511259c700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009f50 tx=0x7f50fc009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f510b7fe700 10 client.?.objecter ms_handle_connect 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f510b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc027070 con 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51141a17c0 con 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f50fc00bda0 con 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00f460 con 0x7f5114104080 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 10 client.24268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:21:47.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141a1c10 con 0x7f5114104080 2024-03-11T21:21:47.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 librados: init done 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.277+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f4005150 con 0x7f5114104080 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f50fc02a070 con 0x7f5114104080 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 0x7f5100071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(61..61 src has 11..61) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f50fc08e5e0 con 0x7f5114104080 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 10 client.24268.objecter ms_dispatch 0x7f5114001550 osd_map(61..61 src has 11..61) v4 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 3 client.24268.objecter handle_osd_map got epochs [61,61] > 0 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 3 client.24268.objecter handle_osd_map decoding full epoch 61 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 20 client.24268.objecter dump_active .. 0 homeless 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f5111d9b700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 0x7f5100071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f510b7fe700 10 client.24268.objecter ms_handle_connect 0x7f510006f730 2024-03-11T21:21:47.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.281+0000 7f5111d9b700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 0x7f5100071bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5104005950 tx=0x7f51040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:21:47.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.285+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f50fc013070 con 0x7f5114104080 2024-03-11T21:21:47.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:47.497+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f50f4000bf0 con 0x7f510006f730 2024-03-11T21:21:47.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:47 smithi203 bash[23654]: cluster 2024-03-11T21:21:45.524211+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:47.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:47 smithi143 bash[21512]: cluster 2024-03-11T21:21:45.524211+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:47.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:47 smithi143 bash[29284]: cluster 2024-03-11T21:21:45.524211+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:48.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:48 smithi203 bash[23654]: audit 2024-03-11T21:21:47.516513+0000 mon.a (mon.0) 595 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:21:48.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:48 smithi203 bash[23654]: audit 2024-03-11T21:21:47.524173+0000 mon.a (mon.0) 596 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:21:48.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:48 smithi203 bash[23654]: audit 2024-03-11T21:21:47.525923+0000 mon.a (mon.0) 597 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:48.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[21512]: audit 2024-03-11T21:21:47.516513+0000 mon.a (mon.0) 595 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:21:48.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[21512]: audit 2024-03-11T21:21:47.524173+0000 mon.a (mon.0) 596 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:21:48.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[21512]: audit 2024-03-11T21:21:47.525923+0000 mon.a (mon.0) 597 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:48.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[29284]: audit 2024-03-11T21:21:47.516513+0000 mon.a (mon.0) 595 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:21:48.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[29284]: audit 2024-03-11T21:21:47.524173+0000 mon.a (mon.0) 596 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:21:48.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:48 smithi143 bash[29284]: audit 2024-03-11T21:21:47.525923+0000 mon.a (mon.0) 597 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:49.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:49 smithi203 bash[23654]: audit 2024-03-11T21:21:47.509309+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:49.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:49 smithi203 bash[23654]: cluster 2024-03-11T21:21:47.524778+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:49.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:49 smithi143 bash[21512]: audit 2024-03-11T21:21:47.509309+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:49.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:49 smithi143 bash[21512]: cluster 2024-03-11T21:21:47.524778+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:49.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:49 smithi143 bash[29284]: audit 2024-03-11T21:21:47.509309+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:49.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:49 smithi143 bash[29284]: cluster 2024-03-11T21:21:47.524778+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:51 smithi203 bash[23654]: cluster 2024-03-11T21:21:49.525306+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:51 smithi203 bash[23654]: cluster 2024-03-11T21:21:50.154131+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:51 smithi203 bash[23654]: cluster 2024-03-11T21:21:50.154218+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:51 smithi203 bash[23654]: cluster 2024-03-11T21:21:50.256808+0000 mon.a (mon.0) 598 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T21:21:51.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[21512]: cluster 2024-03-11T21:21:49.525306+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[21512]: cluster 2024-03-11T21:21:50.154131+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[21512]: cluster 2024-03-11T21:21:50.154218+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[21512]: cluster 2024-03-11T21:21:50.256808+0000 mon.a (mon.0) 598 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[29284]: cluster 2024-03-11T21:21:49.525306+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[29284]: cluster 2024-03-11T21:21:50.154131+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[29284]: cluster 2024-03-11T21:21:50.154218+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:21:51.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:51 smithi143 bash[29284]: cluster 2024-03-11T21:21:50.256808+0000 mon.a (mon.0) 598 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T21:21:52.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:52.281+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:21:53.500 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:53 smithi203 bash[23654]: cluster 2024-03-11T21:21:51.525866+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:53.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:53 smithi143 bash[21512]: cluster 2024-03-11T21:21:51.525866+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:53.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:53 smithi143 bash[29284]: cluster 2024-03-11T21:21:51.525866+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:55.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:55 smithi203 bash[23654]: cluster 2024-03-11T21:21:53.526413+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:55.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:55 smithi143 bash[21512]: cluster 2024-03-11T21:21:53.526413+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:55.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:55 smithi143 bash[29284]: cluster 2024-03-11T21:21:53.526413+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:56.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: audit 2024-03-11T21:21:55.451412+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.203:0/2585593423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: audit 2024-03-11T21:21:55.455138+0000 mon.a (mon.0) 599 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: audit 2024-03-11T21:21:55.458649+0000 mon.a (mon.0) 600 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]': finished 2024-03-11T21:21:56.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: cluster 2024-03-11T21:21:55.464322+0000 mon.a (mon.0) 601 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T21:21:56.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: audit 2024-03-11T21:21:55.464877+0000 mon.a (mon.0) 602 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:21:56.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:56 smithi203 bash[23654]: audit 2024-03-11T21:21:56.233769+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.203:0/3982991854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:56.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: audit 2024-03-11T21:21:55.451412+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.203:0/2585593423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: audit 2024-03-11T21:21:55.455138+0000 mon.a (mon.0) 599 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: audit 2024-03-11T21:21:55.458649+0000 mon.a (mon.0) 600 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]': finished 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: cluster 2024-03-11T21:21:55.464322+0000 mon.a (mon.0) 601 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: audit 2024-03-11T21:21:55.464877+0000 mon.a (mon.0) 602 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[21512]: audit 2024-03-11T21:21:56.233769+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.203:0/3982991854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: audit 2024-03-11T21:21:55.451412+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.203:0/2585593423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: audit 2024-03-11T21:21:55.455138+0000 mon.a (mon.0) 599 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]: dispatch 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: audit 2024-03-11T21:21:55.458649+0000 mon.a (mon.0) 600 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88b53171-69d2-4922-a787-e63d44d01c60"}]': finished 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: cluster 2024-03-11T21:21:55.464322+0000 mon.a (mon.0) 601 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: audit 2024-03-11T21:21:55.464877+0000 mon.a (mon.0) 602 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:21:56.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:56 smithi143 bash[29284]: audit 2024-03-11T21:21:56.233769+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.203:0/3982991854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:21:57.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:21:57.285+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:21:57.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:57 smithi203 bash[23654]: cluster 2024-03-11T21:21:55.526900+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:57.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:57 smithi143 bash[21512]: cluster 2024-03-11T21:21:55.526900+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:57.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:57 smithi143 bash[29284]: cluster 2024-03-11T21:21:55.526900+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:59.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:21:59 smithi203 bash[23654]: cluster 2024-03-11T21:21:57.527455+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:59.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:21:59 smithi143 bash[21512]: cluster 2024-03-11T21:21:57.527455+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:21:59.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:21:59 smithi143 bash[29284]: cluster 2024-03-11T21:21:57.527455+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:01.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:01 smithi203 bash[23654]: cluster 2024-03-11T21:21:59.527975+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:01.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:01 smithi143 bash[21512]: cluster 2024-03-11T21:21:59.527975+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:01.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:01 smithi143 bash[29284]: cluster 2024-03-11T21:21:59.527975+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:02.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:02.289+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:22:03.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:03 smithi203 bash[23654]: cluster 2024-03-11T21:22:01.528513+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:03.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:03 smithi143 bash[21512]: cluster 2024-03-11T21:22:01.528513+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:03.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:03 smithi143 bash[29284]: cluster 2024-03-11T21:22:01.528513+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:05.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:05 smithi203 bash[23654]: cluster 2024-03-11T21:22:03.529071+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:05 smithi143 bash[21512]: cluster 2024-03-11T21:22:03.529071+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:05 smithi143 bash[29284]: cluster 2024-03-11T21:22:03.529071+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:07.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:07.293+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:22:07.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:07 smithi203 bash[23654]: cluster 2024-03-11T21:22:05.529645+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:07.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:07 smithi143 bash[29284]: cluster 2024-03-11T21:22:05.529645+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:07.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:07 smithi143 bash[21512]: cluster 2024-03-11T21:22:05.529645+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:08 smithi203 bash[23654]: audit 2024-03-11T21:22:07.708249+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:22:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:08 smithi203 bash[23654]: audit 2024-03-11T21:22:07.710166+0000 mon.a (mon.0) 604 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:08.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:08 smithi143 bash[21512]: audit 2024-03-11T21:22:07.708249+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:22:08.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:08 smithi143 bash[21512]: audit 2024-03-11T21:22:07.710166+0000 mon.a (mon.0) 604 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:08.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:08 smithi143 bash[29284]: audit 2024-03-11T21:22:07.708249+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:22:08.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:08 smithi143 bash[29284]: audit 2024-03-11T21:22:07.710166+0000 mon.a (mon.0) 604 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:09.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:09 smithi203 bash[23654]: cluster 2024-03-11T21:22:07.530193+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:09.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:09 smithi203 bash[23654]: cephadm 2024-03-11T21:22:07.711687+0000 mgr.y (mgr.14152) 308 : cephadm 1 Deploying daemon osd.7 on smithi203 2024-03-11T21:22:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:09 smithi143 bash[29284]: cluster 2024-03-11T21:22:07.530193+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:09 smithi143 bash[29284]: cephadm 2024-03-11T21:22:07.711687+0000 mgr.y (mgr.14152) 308 : cephadm 1 Deploying daemon osd.7 on smithi203 2024-03-11T21:22:09.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:09 smithi143 bash[21512]: cluster 2024-03-11T21:22:07.530193+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:09.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:09 smithi143 bash[21512]: cephadm 2024-03-11T21:22:07.711687+0000 mgr.y (mgr.14152) 308 : cephadm 1 Deploying daemon osd.7 on smithi203 2024-03-11T21:22:11.607 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:11 smithi203 bash[23654]: cluster 2024-03-11T21:22:09.530760+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:11.607 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:11 smithi203 bash[23654]: cluster 2024-03-11T21:22:10.164281+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:22:11.607 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:11 smithi203 bash[23654]: cluster 2024-03-11T21:22:10.167687+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[29284]: cluster 2024-03-11T21:22:09.530760+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[29284]: cluster 2024-03-11T21:22:10.164281+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[29284]: cluster 2024-03-11T21:22:10.167687+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[21512]: cluster 2024-03-11T21:22:09.530760+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[21512]: cluster 2024-03-11T21:22:10.164281+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-03-11T21:22:11.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:11 smithi143 bash[21512]: cluster 2024-03-11T21:22:10.167687+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-03-11T21:22:11.858 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:11.859 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:11.859 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:11.859 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:11.860 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.274 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.274 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.274 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.275 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:22:11 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:12.280 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:12.297+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:22:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:13 smithi203 bash[23654]: cluster 2024-03-11T21:22:11.531373+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:13 smithi203 bash[23654]: audit 2024-03-11T21:22:12.039896+0000 mon.a (mon.0) 605 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:13 smithi203 bash[23654]: audit 2024-03-11T21:22:12.052517+0000 mon.a (mon.0) 606 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:13 smithi203 bash[23654]: audit 2024-03-11T21:22:12.056227+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:13.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[21512]: cluster 2024-03-11T21:22:11.531373+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:13.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[21512]: audit 2024-03-11T21:22:12.039896+0000 mon.a (mon.0) 605 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[21512]: audit 2024-03-11T21:22:12.052517+0000 mon.a (mon.0) 606 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[21512]: audit 2024-03-11T21:22:12.056227+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:13.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[29284]: cluster 2024-03-11T21:22:11.531373+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:13.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[29284]: audit 2024-03-11T21:22:12.039896+0000 mon.a (mon.0) 605 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[29284]: audit 2024-03-11T21:22:12.052517+0000 mon.a (mon.0) 606 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:13.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:13 smithi143 bash[29284]: audit 2024-03-11T21:22:12.056227+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:15.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:15 smithi203 bash[23654]: cluster 2024-03-11T21:22:13.531926+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:15.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:15 smithi143 bash[21512]: cluster 2024-03-11T21:22:13.531926+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:15.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:15 smithi143 bash[29284]: cluster 2024-03-11T21:22:13.531926+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:17.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:17 smithi203 bash[23654]: cluster 2024-03-11T21:22:15.532472+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:17.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:17.301+0000 7f5113fff700 10 client.24268.objecter tick 2024-03-11T21:22:17.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:17 smithi143 bash[21512]: cluster 2024-03-11T21:22:15.532472+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:17.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:17 smithi143 bash[29284]: cluster 2024-03-11T21:22:15.532472+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:19.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: cluster 2024-03-11T21:22:17.533062+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:19.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.121188+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.125248+0000 mon.a (mon.0) 608 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.906869+0000 mon.a (mon.0) 609 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.918417+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.920839+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.922771+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:19.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:19 smithi203 bash[23654]: audit 2024-03-11T21:22:18.933740+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: cluster 2024-03-11T21:22:17.533062+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.121188+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.125248+0000 mon.a (mon.0) 608 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.906869+0000 mon.a (mon.0) 609 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.918417+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.920839+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.922771+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[21512]: audit 2024-03-11T21:22:18.933740+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: cluster 2024-03-11T21:22:17.533062+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.121188+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.125248+0000 mon.a (mon.0) 608 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.906869+0000 mon.a (mon.0) 609 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.918417+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.920839+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.922771+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:19 smithi143 bash[29284]: audit 2024-03-11T21:22:18.933740+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:20 smithi203 bash[23654]: audit 2024-03-11T21:22:19.060841+0000 mon.a (mon.0) 614 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:22:20.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:20 smithi203 bash[23654]: audit 2024-03-11T21:22:19.063740+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:20.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:20 smithi203 bash[23654]: cluster 2024-03-11T21:22:19.066498+0000 mon.a (mon.0) 615 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T21:22:20.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:20 smithi203 bash[23654]: audit 2024-03-11T21:22:19.067062+0000 mon.a (mon.0) 616 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:20.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:20 smithi203 bash[23654]: audit 2024-03-11T21:22:19.067866+0000 mon.a (mon.0) 617 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[29284]: audit 2024-03-11T21:22:19.060841+0000 mon.a (mon.0) 614 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[29284]: audit 2024-03-11T21:22:19.063740+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[29284]: cluster 2024-03-11T21:22:19.066498+0000 mon.a (mon.0) 615 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[29284]: audit 2024-03-11T21:22:19.067062+0000 mon.a (mon.0) 616 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[29284]: audit 2024-03-11T21:22:19.067866+0000 mon.a (mon.0) 617 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[21512]: audit 2024-03-11T21:22:19.060841+0000 mon.a (mon.0) 614 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[21512]: audit 2024-03-11T21:22:19.063740+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[21512]: cluster 2024-03-11T21:22:19.066498+0000 mon.a (mon.0) 615 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T21:22:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[21512]: audit 2024-03-11T21:22:19.067062+0000 mon.a (mon.0) 616 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:20.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:20 smithi143 bash[21512]: audit 2024-03-11T21:22:19.067866+0000 mon.a (mon.0) 617 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-11T21:22:21.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: cluster 2024-03-11T21:22:19.533630+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:21.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: audit 2024-03-11T21:22:20.063653+0000 mon.a (mon.0) 618 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:22:21.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: cluster 2024-03-11T21:22:20.069617+0000 mon.a (mon.0) 619 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T21:22:21.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: audit 2024-03-11T21:22:20.071569+0000 mon.a (mon.0) 620 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: audit 2024-03-11T21:22:20.079713+0000 mon.a (mon.0) 621 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.367 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:21 smithi203 bash[23654]: audit 2024-03-11T21:22:20.608198+0000 mon.a (mon.0) 622 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: cluster 2024-03-11T21:22:19.533630+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: audit 2024-03-11T21:22:20.063653+0000 mon.a (mon.0) 618 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: cluster 2024-03-11T21:22:20.069617+0000 mon.a (mon.0) 619 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: audit 2024-03-11T21:22:20.071569+0000 mon.a (mon.0) 620 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: audit 2024-03-11T21:22:20.079713+0000 mon.a (mon.0) 621 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[29284]: audit 2024-03-11T21:22:20.608198+0000 mon.a (mon.0) 622 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: cluster 2024-03-11T21:22:19.533630+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: audit 2024-03-11T21:22:20.063653+0000 mon.a (mon.0) 618 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: cluster 2024-03-11T21:22:20.069617+0000 mon.a (mon.0) 619 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: audit 2024-03-11T21:22:20.071569+0000 mon.a (mon.0) 620 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: audit 2024-03-11T21:22:20.079713+0000 mon.a (mon.0) 621 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:21.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:21 smithi143 bash[21512]: audit 2024-03-11T21:22:20.608198+0000 mon.a (mon.0) 622 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T21:22:21.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.868+0000 7f510b7fe700 1 -- 172.21.15.203:0/3753308448 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f50f4000bf0 con 0x7f510006f730 2024-03-11T21:22:21.855 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 7 on host 'smithi203' 2024-03-11T21:22:21.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 10 librados: watch_flush enter 2024-03-11T21:22:21.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 10 librados: watch_flush exit 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 20 client.24268.objecter shutdown clearing up homeless session... 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 10 client.24268.objecter successfully canceled tick 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 msgr2=0x7f5100071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 0x7f5100071bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5104005950 tx=0x7f51040058e0 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 msgr2=0x7f5114199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009f50 tx=0x7f50fc009580 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 shutdown_connections 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f510006f730 0x7f5100071bf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5114105c20 0x7f51141a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5114104080 0x7f5114199920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 --2- 172.21.15.203:0/3753308448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5114105280 0x7f5114199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 >> 172.21.15.203:0/3753308448 conn(0x7f51140ff640 msgr2=0x7f51141086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 shutdown_connections 2024-03-11T21:22:21.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 -- 172.21.15.203:0/3753308448 wait complete. 2024-03-11T21:22:21.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:21.872+0000 7f5118871700 1 librados: shutdown 2024-03-11T21:22:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: cluster 2024-03-11T21:22:19.077700+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: cluster 2024-03-11T21:22:19.077797+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: cluster 2024-03-11T21:22:21.074093+0000 mon.a (mon.0) 623 : cluster 1 osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] boot 2024-03-11T21:22:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: cluster 2024-03-11T21:22:21.074212+0000 mon.a (mon.0) 624 : cluster 0 osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:22:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: audit 2024-03-11T21:22:21.074526+0000 mon.a (mon.0) 625 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: audit 2024-03-11T21:22:21.831072+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: audit 2024-03-11T21:22:21.831989+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: audit 2024-03-11T21:22:21.843300+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:22 smithi203 bash[23654]: cluster 2024-03-11T21:22:22.076220+0000 mon.a (mon.0) 629 : cluster 0 osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:22:22.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: cluster 2024-03-11T21:22:19.077700+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: cluster 2024-03-11T21:22:19.077797+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: cluster 2024-03-11T21:22:21.074093+0000 mon.a (mon.0) 623 : cluster 1 osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] boot 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: cluster 2024-03-11T21:22:21.074212+0000 mon.a (mon.0) 624 : cluster 0 osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: audit 2024-03-11T21:22:21.074526+0000 mon.a (mon.0) 625 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: audit 2024-03-11T21:22:21.831072+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: audit 2024-03-11T21:22:21.831989+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: audit 2024-03-11T21:22:21.843300+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[29284]: cluster 2024-03-11T21:22:22.076220+0000 mon.a (mon.0) 629 : cluster 0 osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: cluster 2024-03-11T21:22:19.077700+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:22.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: cluster 2024-03-11T21:22:19.077797+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: cluster 2024-03-11T21:22:21.074093+0000 mon.a (mon.0) 623 : cluster 1 osd.7 [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] boot 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: cluster 2024-03-11T21:22:21.074212+0000 mon.a (mon.0) 624 : cluster 0 osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: audit 2024-03-11T21:22:21.074526+0000 mon.a (mon.0) 625 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: audit 2024-03-11T21:22:21.831072+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: audit 2024-03-11T21:22:21.831989+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: audit 2024-03-11T21:22:21.843300+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:22.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:22 smithi143 bash[21512]: cluster 2024-03-11T21:22:22.076220+0000 mon.a (mon.0) 629 : cluster 0 osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:22:22.567 DEBUG:teuthology.orchestra.run.smithi203:osd.7> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T21:22:22.570 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T21:22:22.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:22:23.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:23 smithi143 bash[21512]: cluster 2024-03-11T21:22:21.534188+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:23.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:23 smithi143 bash[21512]: cluster 2024-03-11T21:22:23.076068+0000 mon.a (mon.0) 630 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:22:23.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:23 smithi143 bash[29284]: cluster 2024-03-11T21:22:21.534188+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:23.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:23 smithi143 bash[29284]: cluster 2024-03-11T21:22:23.076068+0000 mon.a (mon.0) 630 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:22:23.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:23 smithi203 bash[23654]: cluster 2024-03-11T21:22:21.534188+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:22:23.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:23 smithi203 bash[23654]: cluster 2024-03-11T21:22:23.076068+0000 mon.a (mon.0) 630 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:22:24.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:24 smithi203 bash[23654]: cluster 2024-03-11T21:22:23.501333+0000 mon.a (mon.0) 631 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:22:24.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:24 smithi203 bash[23654]: cluster 2024-03-11T21:22:23.534718+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:24.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:24 smithi143 bash[21512]: cluster 2024-03-11T21:22:23.501333+0000 mon.a (mon.0) 631 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:22:24.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:24 smithi143 bash[21512]: cluster 2024-03-11T21:22:23.534718+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:24.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:24 smithi143 bash[29284]: cluster 2024-03-11T21:22:23.501333+0000 mon.a (mon.0) 631 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:22:24.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:24 smithi143 bash[29284]: cluster 2024-03-11T21:22:23.534718+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:25.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:25 smithi203 bash[23654]: cluster 2024-03-11T21:22:24.517178+0000 mon.a (mon.0) 632 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:22:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:25 smithi143 bash[21512]: cluster 2024-03-11T21:22:24.517178+0000 mon.a (mon.0) 632 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:22:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:25 smithi143 bash[29284]: cluster 2024-03-11T21:22:24.517178+0000 mon.a (mon.0) 632 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:22:26.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:26 smithi203 bash[23654]: cluster 2024-03-11T21:22:25.535323+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v316: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:26 smithi143 bash[29284]: cluster 2024-03-11T21:22:25.535323+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v316: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:26.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:26 smithi143 bash[21512]: cluster 2024-03-11T21:22:25.535323+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v316: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:27.322 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:28.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:28 smithi143 bash[21512]: cluster 2024-03-11T21:22:27.535943+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:22:28.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:28 smithi143 bash[29284]: cluster 2024-03-11T21:22:27.535943+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:22:28.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:28 smithi203 bash[23654]: cluster 2024-03-11T21:22:27.535943+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1818885009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410107b60 msgr2=0x7f5410109fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- 172.21.15.143:0/1818885009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410107b60 0x7f5410109fd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f540800b0a0 tx=0x7f540800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1818885009 shutdown_connections 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- 172.21.15.143:0/1818885009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410107b60 0x7f5410109fd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- 172.21.15.143:0/1818885009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410102900 0x7f5410104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- 172.21.15.143:0/1818885009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410105230 0x7f5410107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1818885009 >> 172.21.15.143:0/1818885009 conn(0x7f54100fc310 msgr2=0x7f54100fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:30.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1818885009 shutdown_connections 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1818885009 wait complete. 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 librados: starting msgr at 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 librados: starting objecter 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 Processor -- start 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- start start 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 librados: setting wanted keys 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 librados: calling monclient init 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 0x7f5410197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 0x7f5410197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 0x7f541019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541010cf00 con 0x7f5410107b60 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541010cd80 con 0x7f5410105230 2024-03-11T21:22:30.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541010d080 con 0x7f5410102900 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416598700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 0x7f5410197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 0x7f541019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416598700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 0x7f5410197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:57736/0 (socket says 172.21.15.143:57736) 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416598700 1 -- 172.21.15.143:0/1514207247 learned_addr learned my addr 172.21.15.143:0/1514207247 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5415d97700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 0x7f5410197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 -- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 msgr2=0x7f5410197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 0x7f5410197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 -- 172.21.15.143:0/1514207247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 msgr2=0x7f5410197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 0x7f5410197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 -- 172.21.15.143:0/1514207247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541019f270 con 0x7f5410107b60 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5415d97700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 0x7f5410197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f5416d99700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 0x7f541019ebd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f541006b600 tx=0x7f5408004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:30.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54077fe700 10 client.?.objecter ms_handle_connect 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54077fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408021070 con 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5408007b80 con 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541019f500 con 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540801b690 con 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 10 client.14391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541019f950 con 0x7f5410107b60 2024-03-11T21:22:30.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.393+0000 7f54187fc700 1 librados: init done 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541004faa0 con 0x7f5410107b60 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f5408024070 con 0x7f5410107b60 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 0x7f53fc071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(70..70 src has 11..70) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f540800f020 con 0x7f5410107b60 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 10 client.14391.objecter ms_dispatch 0x7f5410001550 osd_map(70..70 src has 11..70) v4 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 3 client.14391.objecter handle_osd_map got epochs [70,70] > 0 2024-03-11T21:22:30.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 3 client.14391.objecter handle_osd_map decoding full epoch 70 2024-03-11T21:22:30.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f5416598700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 0x7f53fc071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:30.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 20 client.14391.objecter dump_active .. 0 homeless 2024-03-11T21:22:30.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f54077fe700 10 client.14391.objecter ms_handle_connect 0x7f53fc06f730 2024-03-11T21:22:30.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.397+0000 7f5416598700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 0x7f53fc071bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f540000bd30 tx=0x7f540000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:30.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.401+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f540802b020 con 0x7f5410107b60 2024-03-11T21:22:30.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.613+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f54100629f0 con 0x7f5410107b60 2024-03-11T21:22:30.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54077fe700 1 -- 172.21.15.143:0/1514207247 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) v1 ==== 74+0+130 (secure 0 0 0) 0x7f540805c770 con 0x7f5410107b60 2024-03-11T21:22:30.596 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:22:30.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 10 librados: watch_flush enter 2024-03-11T21:22:30.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 10 librados: watch_flush exit 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 20 client.14391.objecter shutdown clearing up homeless session... 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 10 client.14391.objecter successfully canceled tick 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 msgr2=0x7f53fc071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 0x7f53fc071bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f540000bd30 tx=0x7f540000b480 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 msgr2=0x7f541019ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 0x7f541019ebd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f541006b600 tx=0x7f5408004d00 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 shutdown_connections 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f53fc06f730 0x7f53fc071bf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5410107b60 0x7f541019ebd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5410102900 0x7f5410197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 --2- 172.21.15.143:0/1514207247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5410105230 0x7f5410197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 >> 172.21.15.143:0/1514207247 conn(0x7f54100fc310 msgr2=0x7f5410103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 shutdown_connections 2024-03-11T21:22:30.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 -- 172.21.15.143:0/1514207247 wait complete. 2024-03-11T21:22:30.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:30.617+0000 7f54187fc700 1 librados: shutdown 2024-03-11T21:22:30.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:30 smithi143 bash[29284]: cluster 2024-03-11T21:22:29.536507+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:22:30.843 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:30 smithi143 bash[21512]: cluster 2024-03-11T21:22:29.536507+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:22:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:30 smithi203 bash[23654]: cluster 2024-03-11T21:22:29.536507+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:22:31.370 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192141,"num_in_osds":8,"osd_in_since":1710192115,"num_remapped_pgs":0} 2024-03-11T21:22:31.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.620166+0000 mon.a (mon.0) 633 : audit 0 from='client.? 172.21.15.143:0/1514207247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: cephadm 2024-03-11T21:22:30.833577+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.841240+0000 mon.a (mon.0) 634 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.849875+0000 mon.a (mon.0) 635 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.852429+0000 mon.a (mon.0) 636 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.854113+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.855645+0000 mon.a (mon.0) 638 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.857145+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: cephadm 2024-03-11T21:22:30.858327+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi203 to 4316M 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.864095+0000 mon.a (mon.0) 640 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.869370+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.870723+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[29284]: audit 2024-03-11T21:22:30.882101+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.620166+0000 mon.a (mon.0) 633 : audit 0 from='client.? 172.21.15.143:0/1514207247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: cephadm 2024-03-11T21:22:30.833577+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.841240+0000 mon.a (mon.0) 634 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.849875+0000 mon.a (mon.0) 635 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.852429+0000 mon.a (mon.0) 636 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.854113+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.855645+0000 mon.a (mon.0) 638 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.857145+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: cephadm 2024-03-11T21:22:30.858327+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi203 to 4316M 2024-03-11T21:22:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.864095+0000 mon.a (mon.0) 640 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:31.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.869370+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:31.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.870723+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:31.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:31 smithi143 bash[21512]: audit 2024-03-11T21:22:30.882101+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.620166+0000 mon.a (mon.0) 633 : audit 0 from='client.? 172.21.15.143:0/1514207247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: cephadm 2024-03-11T21:22:30.833577+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi203 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.841240+0000 mon.a (mon.0) 634 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.849875+0000 mon.a (mon.0) 635 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.852429+0000 mon.a (mon.0) 636 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.854113+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.855645+0000 mon.a (mon.0) 638 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.857145+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: cephadm 2024-03-11T21:22:30.858327+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi203 to 4316M 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.864095+0000 mon.a (mon.0) 640 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.869370+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.870723+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:31 smithi203 bash[23654]: audit 2024-03-11T21:22:30.882101+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:33.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:32 smithi143 bash[21512]: cluster 2024-03-11T21:22:31.536929+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:22:33.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:32 smithi143 bash[29284]: cluster 2024-03-11T21:22:31.536929+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:22:33.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:32 smithi203 bash[23654]: cluster 2024-03-11T21:22:31.536929+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:22:35.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:34 smithi143 bash[21512]: cluster 2024-03-11T21:22:33.537504+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:22:35.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:34 smithi143 bash[29284]: cluster 2024-03-11T21:22:33.537504+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:22:35.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:34 smithi203 bash[23654]: cluster 2024-03-11T21:22:33.537504+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T21:22:36.113 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:37.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[29284]: cluster 2024-03-11T21:22:35.357247+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:37.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[29284]: cluster 2024-03-11T21:22:35.357348+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:37.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[29284]: cluster 2024-03-11T21:22:35.538097+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:22:37.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[29284]: cluster 2024-03-11T21:22:35.903272+0000 mon.a (mon.0) 644 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:22:37.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[21512]: cluster 2024-03-11T21:22:35.357247+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:37.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[21512]: cluster 2024-03-11T21:22:35.357348+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:37.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[21512]: cluster 2024-03-11T21:22:35.538097+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:22:37.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:36 smithi143 bash[21512]: cluster 2024-03-11T21:22:35.903272+0000 mon.a (mon.0) 644 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:22:37.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:36 smithi203 bash[23654]: cluster 2024-03-11T21:22:35.357247+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:37.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:36 smithi203 bash[23654]: cluster 2024-03-11T21:22:35.357348+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:37.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:36 smithi203 bash[23654]: cluster 2024-03-11T21:22:35.538097+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T21:22:37.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:36 smithi203 bash[23654]: cluster 2024-03-11T21:22:35.903272+0000 mon.a (mon.0) 644 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:22:39.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/2305687396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 msgr2=0x7f1ca01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:39.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/2305687396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01011f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009990 tx=0x7f1c94009ca0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/2305687396 shutdown_connections 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/2305687396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/2305687396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 0x7f1ca010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/2305687396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01011f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/2305687396 >> 172.21.15.143:0/2305687396 conn(0x7f1ca0076dc0 msgr2=0x7f1ca00771d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/2305687396 shutdown_connections 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/2305687396 wait complete. 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 librados: starting msgr at 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 librados: starting objecter 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 Processor -- start 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- start start 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 librados: setting wanted keys 2024-03-11T21:22:39.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 librados: calling monclient init 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 0x7f1ca0102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0110550 con 0x7f1ca0106d80 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01103d0 con 0x7f1ca0101730 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01106d0 con 0x7f1ca0100d70 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58594/0 (socket says 172.21.15.143:58594) 2024-03-11T21:22:39.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 -- 172.21.15.143:0/910422738 learned_addr learned my addr 172.21.15.143:0/910422738 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56938/0 (socket says 172.21.15.143:56938) 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9effd700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 0x7f1ca0102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 -- 172.21.15.143:0/910422738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 msgr2=0x7f1ca0102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 0x7f1ca0102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 -- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 msgr2=0x7f1ca0103470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 -- 172.21.15.143:0/910422738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca01a6110 con 0x7f1ca0100d70 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9ffff700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9f7fe700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000e9a0 tx=0x7f1c9000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9cff9700 10 client.?.objecter ms_handle_connect 0x7f1ca0100d70 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9cff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000cbd0 con 0x7f1ca0100d70 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1c9000cd70 con 0x7f1ca0100d70 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca01a6400 con 0x7f1ca0100d70 2024-03-11T21:22:39.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90010e60 con 0x7f1ca0100d70 2024-03-11T21:22:39.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 10 client.24296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:22:39.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca01a6990 con 0x7f1ca0100d70 2024-03-11T21:22:39.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.137+0000 7f1ca60e3700 1 librados: init done 2024-03-11T21:22:39.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca004faa0 con 0x7f1ca0100d70 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f1c90010430 con 0x7f1ca0100d70 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 0x7f1c88071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(71..71 src has 21..71) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f1c9008d9c0 con 0x7f1ca0100d70 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 10 client.24296.objecter ms_dispatch 0x7f1ca0001550 osd_map(71..71 src has 21..71) v4 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 3 client.24296.objecter handle_osd_map got epochs [71,71] > 0 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 3 client.24296.objecter handle_osd_map decoding full epoch 71 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9effd700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 0x7f1c88071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 20 client.24296.objecter dump_active .. 0 homeless 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9cff9700 10 client.24296.objecter ms_handle_connect 0x7f1c8806f730 2024-03-11T21:22:39.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.141+0000 7f1c9effd700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 0x7f1c88071bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400b2e0 tx=0x7f1c9401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:39.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.145+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1c9008fe20 con 0x7f1ca0100d70 2024-03-11T21:22:39.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:38 smithi143 bash[21512]: cluster 2024-03-11T21:22:37.538704+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:38 smithi143 bash[29284]: cluster 2024-03-11T21:22:37.538704+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:39.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:38 smithi203 bash[23654]: cluster 2024-03-11T21:22:37.538704+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:39.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.349+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca01a6bf0 con 0x7f1ca0100d70 2024-03-11T21:22:39.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.349+0000 7f1c9cff9700 1 -- 172.21.15.143:0/910422738 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+13612 (secure 0 0 0) 0x7f1c9005b7d0 con 0x7f1ca0100d70 2024-03-11T21:22:39.330 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:22:39.330 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":71,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","created":"2024-03-11T21:12:47.384719+0000","modified":"2024-03-11T21:22:35.885559+0000","last_up_change":"2024-03-11T21:22:21.063566+0000","last_in_change":"2024-03-11T21:21:55.456354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:08.036690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6803","nonce":2903432023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6805","nonce":2903432023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6809","nonce":2903432023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6807","nonce":2903432023}]},"public_addr":"172.21.15.143:6803/2903432023","cluster_addr":"172.21.15.143:6805/2903432023","heartbeat_back_addr":"172.21.15.143:6809/2903432023","heartbeat_front_addr":"172.21.15.143:6807/2903432023","state":["exists","up"]},{"osd":1,"uuid":"2948dc44-68d3-4b3d-92bf-2b0c46374f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6811","nonce":397942214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6813","nonce":397942214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6817","nonce":397942214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6815","nonce":397942214}]},"public_addr":"172.21.15.143:6811/397942214","cluster_addr":"172.21.15.143:6813/397942214","heartbeat_back_addr":"172.21.15.143:6817/397942214","heartbeat_front_addr":"172.21.15.143:6815/397942214","state":["exists","up"]},{"osd":2,"uuid":"7217589a-5835-407b-91a7-f5c055a48f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6819","nonce":2592131625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6821","nonce":2592131625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6825","nonce":2592131625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6823","nonce":2592131625}]},"public_addr":"172.21.15.143:6819/2592131625","cluster_addr":"172.21.15.143:6821/2592131625","heartbeat_back_addr":"172.21.15.143:6825/2592131625","heartbeat_front_addr":"172.21.15.143:6823/2592131625","state":["exists","up"]},{"osd":3,"uuid":"a3e7177f-a5a0-44ad-a65d-9aeac3a680cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6827","nonce":2146571980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6829","nonce":2146571980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6833","nonce":2146571980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6831","nonce":2146571980}]},"public_addr":"172.21.15.143:6827/2146571980","cluster_addr":"172.21.15.143:6829/2146571980","heartbeat_back_addr":"172.21.15.143:6833/2146571980","heartbeat_front_addr":"172.21.15.143:6831/2146571980","state":["exists","up"]},{"osd":4,"uuid":"d7b9e088-d377-4335-8863-11d581828f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6801","nonce":2090835691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6803","nonce":2090835691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6807","nonce":2090835691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6805","nonce":2090835691}]},"public_addr":"172.21.15.203:6801/2090835691","cluster_addr":"172.21.15.203:6803/2090835691","heartbeat_back_addr":"172.21.15.203:6807/2090835691","heartbeat_front_addr":"172.21.15.203:6805/2090835691","state":["exists","up"]},{"osd":5,"uuid":"b3b2a75c-68a5-4e00-8068-622e1ebbaa01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6809","nonce":2432038345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6811","nonce":2432038345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6815","nonce":2432038345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6813","nonce":2432038345}]},"public_addr":"172.21.15.203:6809/2432038345","cluster_addr":"172.21.15.203:6811/2432038345","heartbeat_back_addr":"172.21.15.203:6815/2432038345","heartbeat_front_addr":"172.21.15.203:6813/2432038345","state":["exists","up"]},{"osd":6,"uuid":"ceffb234-db0c-4ffa-a142-9b201df9b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6817","nonce":513748834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6819","nonce":513748834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6823","nonce":513748834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6821","nonce":513748834}]},"public_addr":"172.21.15.203:6817/513748834","cluster_addr":"172.21.15.203:6819/513748834","heartbeat_back_addr":"172.21.15.203:6823/513748834","heartbeat_front_addr":"172.21.15.203:6821/513748834","state":["exists","up"]},{"osd":7,"uuid":"88b53171-69d2-4922-a787-e63d44d01c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6825","nonce":1251497952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6827","nonce":1251497952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6831","nonce":1251497952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6829","nonce":1251497952}]},"public_addr":"172.21.15.203:6825/1251497952","cluster_addr":"172.21.15.203:6827/1251497952","heartbeat_back_addr":"172.21.15.203:6831/1251497952","heartbeat_front_addr":"172.21.15.203:6829/1251497952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:26.933251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:50.154223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:43.040165+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:22:35.357354+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:13.923023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:42.719840+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:21:28.096083+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:22:19.077812+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/143435387":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2913593434":"2024-03-12T21:13:53.401579+0000","172.21.15.143:6801/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:6800/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2791822486":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/906834629":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6801/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/3217951344":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/4274199357":"2024-03-12T21:13:53.401579+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:22:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 10 librados: watch_flush enter 2024-03-11T21:22:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 10 librados: watch_flush exit 2024-03-11T21:22:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 20 client.24296.objecter shutdown clearing up homeless session... 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 10 client.24296.objecter successfully canceled tick 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 msgr2=0x7f1c88071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 0x7f1c88071bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400b2e0 tx=0x7f1c9401a040 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 msgr2=0x7f1ca01029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000e9a0 tx=0x7f1c9000ecb0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 shutdown_connections 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f1c8806f730 0x7f1c88071bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca0106d80 0x7f1ca0103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca0100d70 0x7f1ca01029f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 --2- 172.21.15.143:0/910422738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ca0101730 0x7f1ca0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 >> 172.21.15.143:0/910422738 conn(0x7f1ca0076dc0 msgr2=0x7f1ca006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 shutdown_connections 2024-03-11T21:22:39.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 -- 172.21.15.143:0/910422738 wait complete. 2024-03-11T21:22:39.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:39.353+0000 7f1ca60e3700 1 librados: shutdown 2024-03-11T21:22:40.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:39 smithi143 bash[21512]: audit 2024-03-11T21:22:39.353274+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.143:0/910422738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:22:40.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:39 smithi143 bash[29284]: audit 2024-03-11T21:22:39.353274+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.143:0/910422738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:22:40.159 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:18:08.036690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:22:40.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:22:40.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:39 smithi203 bash[23654]: audit 2024-03-11T21:22:39.353274+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.143:0/910422738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:22:41.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:40 smithi143 bash[21512]: cluster 2024-03-11T21:22:39.539232+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:40 smithi143 bash[29284]: cluster 2024-03-11T21:22:39.539232+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:40 smithi203 bash[23654]: cluster 2024-03-11T21:22:39.539232+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:42.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:41 smithi143 bash[21512]: cluster 2024-03-11T21:22:41.027611+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:42.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:41 smithi143 bash[21512]: cluster 2024-03-11T21:22:41.027723+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:42.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:41 smithi143 bash[29284]: cluster 2024-03-11T21:22:41.027611+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:42.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:41 smithi143 bash[29284]: cluster 2024-03-11T21:22:41.027723+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:42.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:41 smithi203 bash[23654]: cluster 2024-03-11T21:22:41.027611+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:42.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:41 smithi203 bash[23654]: cluster 2024-03-11T21:22:41.027723+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[21512]: cluster 2024-03-11T21:22:41.539799+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[21512]: cluster 2024-03-11T21:22:41.926759+0000 mon.a (mon.0) 645 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:22:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[21512]: cluster 2024-03-11T21:22:42.208130+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[21512]: cluster 2024-03-11T21:22:42.208225+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:43.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[29284]: cluster 2024-03-11T21:22:41.539799+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:43.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[29284]: cluster 2024-03-11T21:22:41.926759+0000 mon.a (mon.0) 645 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:22:43.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[29284]: cluster 2024-03-11T21:22:42.208130+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:43.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:42 smithi143 bash[29284]: cluster 2024-03-11T21:22:42.208225+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:42 smithi203 bash[23654]: cluster 2024-03-11T21:22:41.539799+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:43.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:42 smithi203 bash[23654]: cluster 2024-03-11T21:22:41.926759+0000 mon.a (mon.0) 645 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:22:43.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:42 smithi203 bash[23654]: cluster 2024-03-11T21:22:42.208130+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:43.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:42 smithi203 bash[23654]: cluster 2024-03-11T21:22:42.208225+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:44.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:43 smithi143 bash[21512]: cluster 2024-03-11T21:22:42.932626+0000 mon.a (mon.0) 646 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:22:44.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:43 smithi143 bash[29284]: cluster 2024-03-11T21:22:42.932626+0000 mon.a (mon.0) 646 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:22:44.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:43 smithi203 bash[23654]: cluster 2024-03-11T21:22:42.932626+0000 mon.a (mon.0) 646 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:22:44.916 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:44 smithi143 bash[21512]: cluster 2024-03-11T21:22:43.540377+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:45.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:44 smithi143 bash[29284]: cluster 2024-03-11T21:22:43.540377+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:45.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:44 smithi203 bash[23654]: cluster 2024-03-11T21:22:43.540377+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:47.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:46 smithi143 bash[29284]: cluster 2024-03-11T21:22:45.540898+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:47.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:46 smithi143 bash[21512]: cluster 2024-03-11T21:22:45.540898+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:47.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:46 smithi203 bash[23654]: cluster 2024-03-11T21:22:45.540898+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:47.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- 172.21.15.143:0/1119399579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 msgr2=0x7f7884102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 --2- 172.21.15.143:0/1119399579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884102430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f787400b0d0 tx=0x7f787400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- 172.21.15.143:0/1119399579 shutdown_connections 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 --2- 172.21.15.143:0/1119399579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884102430 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 --2- 172.21.15.143:0/1119399579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884107130 0x7f788410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 --2- 172.21.15.143:0/1119399579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884102a00 0x7f7884106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- 172.21.15.143:0/1119399579 >> 172.21.15.143:0/1119399579 conn(0x7f78840791b0 msgr2=0x7f78840795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- 172.21.15.143:0/1119399579 shutdown_connections 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- 172.21.15.143:0/1119399579 wait complete. 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 librados: starting msgr at 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 librados: starting objecter 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 Processor -- start 2024-03-11T21:22:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 -- start start 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 librados: setting wanted keys 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.961+0000 7f78897c4700 1 librados: calling monclient init 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 0x7f788419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884107130 0x7f788419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7884111630 con 0x7f7884102050 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78841114b0 con 0x7f7884107130 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78841117b0 con 0x7f7884102a00 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78827fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 0x7f788419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50208/0 (socket says 172.21.15.143:50208) 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 -- 172.21.15.143:0/1729656241 learned_addr learned my addr 172.21.15.143:0/1729656241 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78837fe700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884107130 0x7f788419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 -- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 msgr2=0x7f788419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 0x7f788419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 -- 172.21.15.143:0/1729656241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884107130 msgr2=0x7f788419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884107130 0x7f788419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 -- 172.21.15.143:0/1729656241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788419ef10 con 0x7f7884102050 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78827fc700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 0x7f788419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:22:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f7882ffd700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7874007a20 tx=0x7f7874007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 10 client.?.objecter ms_handle_connect 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787401f070 con 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7874003c00 con 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788419f1a0 con 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787401b440 con 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 10 client.14403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78841a6a80 con 0x7f7884102050 2024-03-11T21:22:47.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 librados: init done 2024-03-11T21:22:47.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788404faa0 con 0x7f7884102050 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f7874022070 con 0x7f7884102050 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 0x7f7870071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(73..73 src has 21..73) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f787400f020 con 0x7f7884102050 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 10 client.14403.objecter ms_dispatch 0x7f7884001550 osd_map(73..73 src has 21..73) v4 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78827fc700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 0x7f7870071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 3 client.14403.objecter handle_osd_map got epochs [73,73] > 0 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 3 client.14403.objecter handle_osd_map decoding full epoch 73 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 20 client.14403.objecter dump_active .. 0 homeless 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f787bfff700 10 client.14403.objecter ms_handle_connect 0x7f787006f730 2024-03-11T21:22:47.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.965+0000 7f78827fc700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 0x7f7870071bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f788410f0c0 tx=0x7f786c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:47.973+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7874029020 con 0x7f7884102050 2024-03-11T21:22:48.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.177+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78841a6c50 con 0x7f7884102050 2024-03-11T21:22:48.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.177+0000 7f787bfff700 1 -- 172.21.15.143:0/1729656241 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78740233b0 con 0x7f7884102050 2024-03-11T21:22:48.158 INFO:teuthology.orchestra.run.smithi143.stdout:pg_num: 1 2024-03-11T21:22:48.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 10 librados: watch_flush enter 2024-03-11T21:22:48.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 10 librados: watch_flush exit 2024-03-11T21:22:48.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 20 client.14403.objecter shutdown clearing up homeless session... 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 10 client.14403.objecter successfully canceled tick 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 msgr2=0x7f7870071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 0x7f7870071bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f788410f0c0 tx=0x7f786c00a300 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 msgr2=0x7f7884199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7874007a20 tx=0x7f7874007a50 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 shutdown_connections 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f787006f730 0x7f7870071bf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7884102050 0x7f7884199f50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7884107130 0x7f788419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 --2- 172.21.15.143:0/1729656241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7884102a00 0x7f788419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 >> 172.21.15.143:0/1729656241 conn(0x7f78840791b0 msgr2=0x7f78840ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 shutdown_connections 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 -- 172.21.15.143:0/1729656241 wait complete. 2024-03-11T21:22:48.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:22:48.181+0000 7f78897c4700 1 librados: shutdown 2024-03-11T21:22:48.873 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi143 2024-03-11T21:22:48.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi143=foo.a' 2024-03-11T21:22:49.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:48 smithi143 bash[21512]: cluster 2024-03-11T21:22:47.541506+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:49.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:48 smithi143 bash[21512]: audit 2024-03-11T21:22:48.181980+0000 mon.a (mon.0) 647 : audit 0 from='client.? 172.21.15.143:0/1729656241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:22:49.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:48 smithi143 bash[29284]: cluster 2024-03-11T21:22:47.541506+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:49.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:48 smithi143 bash[29284]: audit 2024-03-11T21:22:48.181980+0000 mon.a (mon.0) 647 : audit 0 from='client.? 172.21.15.143:0/1729656241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:22:49.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:48 smithi203 bash[23654]: cluster 2024-03-11T21:22:47.541506+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:49.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:48 smithi203 bash[23654]: audit 2024-03-11T21:22:48.181980+0000 mon.a (mon.0) 647 : audit 0 from='client.? 172.21.15.143:0/1729656241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:22:51.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:50 smithi143 bash[21512]: cluster 2024-03-11T21:22:49.542091+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:50 smithi143 bash[29284]: cluster 2024-03-11T21:22:49.542091+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:51.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:50 smithi203 bash[23654]: cluster 2024-03-11T21:22:49.542091+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:53.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:52 smithi143 bash[21512]: cluster 2024-03-11T21:22:51.542669+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:52 smithi143 bash[29284]: cluster 2024-03-11T21:22:51.542669+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:53.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:52 smithi203 bash[23654]: cluster 2024-03-11T21:22:51.542669+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:53.621 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:22:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:53 smithi203 bash[23654]: cluster 2024-03-11T21:22:53.131884+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:53 smithi203 bash[23654]: cluster 2024-03-11T21:22:53.132016+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:54.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:53 smithi143 bash[21512]: cluster 2024-03-11T21:22:53.131884+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:54.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:53 smithi143 bash[21512]: cluster 2024-03-11T21:22:53.132016+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:54.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:53 smithi143 bash[29284]: cluster 2024-03-11T21:22:53.131884+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:54.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:53 smithi143 bash[29284]: cluster 2024-03-11T21:22:53.132016+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:55.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:54 smithi203 bash[23654]: cluster 2024-03-11T21:22:53.543259+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:55.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:54 smithi203 bash[23654]: cluster 2024-03-11T21:22:53.971425+0000 mon.a (mon.0) 648 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:22:55.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:54 smithi143 bash[21512]: cluster 2024-03-11T21:22:53.543259+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:55.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:54 smithi143 bash[21512]: cluster 2024-03-11T21:22:53.971425+0000 mon.a (mon.0) 648 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:22:55.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:54 smithi143 bash[29284]: cluster 2024-03-11T21:22:53.543259+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:55.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:54 smithi143 bash[29284]: cluster 2024-03-11T21:22:53.971425+0000 mon.a (mon.0) 648 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/1730848622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c103980 msgr2=0x7f0b0c10a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- 172.21.15.203:0/1730848622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c103980 0x7f0b0c10a2a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009990 tx=0x7f0afc009ca0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/1730848622 shutdown_connections 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- 172.21.15.203:0/1730848622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c103980 0x7f0b0c10a2a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- 172.21.15.203:0/1730848622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c101e60 0x7f0b0c102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- 172.21.15.203:0/1730848622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c102fc0 0x7f0b0c103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/1730848622 >> 172.21.15.203:0/1730848622 conn(0x7f0b0c0fd400 msgr2=0x7f0b0c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/1730848622 shutdown_connections 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/1730848622 wait complete. 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 librados: starting msgr at 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 librados: starting objecter 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 Processor -- start 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- start start 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 librados: setting wanted keys 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 librados: calling monclient init 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 0x7f0b0c199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c103980 0x7f0b0c1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c10cef0 con 0x7f0b0c103980 2024-03-11T21:22:55.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c10cd70 con 0x7f0b0c101e60 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c10d070 con 0x7f0b0c102fc0 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45792/0 (socket says 172.21.15.203:45792) 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 -- 172.21.15.203:0/4024964368 learned_addr learned my addr 172.21.15.203:0/4024964368 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0affd700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 0x7f0b0c199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0bfff700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c103980 0x7f0b0c1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 -- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 msgr2=0x7f0b0c199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 0x7f0b0c199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 -- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c103980 msgr2=0x7f0b0c1a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c103980 0x7f0b0c1a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 -- 172.21.15.203:0/4024964368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c1a13e0 con 0x7f0b0c101e60 2024-03-11T21:22:55.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0affd700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 0x7f0b0c199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b0b7fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b04007e20 tx=0x7f0b0400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b08ff9700 10 client.?.objecter ms_handle_connect 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b08ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04016020 con 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0b0400ae00 con 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c1a16d0 con 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04011680 con 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 10 client.24307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c1a1c10 con 0x7f0b0c101e60 2024-03-11T21:22:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.911+0000 7f0b120fe700 1 librados: init done 2024-03-11T21:22:55.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c04faa0 con 0x7f0b0c101e60 2024-03-11T21:22:55.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f0b04019020 con 0x7f0b0c101e60 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 0x7f0af4071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(74..74 src has 21..74) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f0b0408d330 con 0x7f0b0c101e60 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 10 client.24307.objecter ms_dispatch 0x7f0b0c001550 osd_map(74..74 src has 21..74) v4 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 3 client.24307.objecter handle_osd_map got epochs [74,74] > 0 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 3 client.24307.objecter handle_osd_map decoding full epoch 74 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 20 client.24307.objecter dump_active .. 0 homeless 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b0affd700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 0x7f0af4071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b08ff9700 10 client.24307.objecter ms_handle_connect 0x7f0af406f730 2024-03-11T21:22:55.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.915+0000 7f0b0affd700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 0x7f0af4071bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0b00005fd0 tx=0x7f0b00005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:22:55.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:55.919+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0b04020070 con 0x7f0b0c101e60 2024-03-11T21:22:56.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.143+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi143=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b0c075020 con 0x7f0af406f730 2024-03-11T21:22:56.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b08ff9700 1 -- 172.21.15.203:0/4024964368 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f0b0c075020 con 0x7f0af406f730 2024-03-11T21:22:56.141 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:22:56.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 10 librados: watch_flush enter 2024-03-11T21:22:56.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 10 librados: watch_flush exit 2024-03-11T21:22:56.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 20 client.24307.objecter shutdown clearing up homeless session... 2024-03-11T21:22:56.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 10 client.24307.objecter successfully canceled tick 2024-03-11T21:22:56.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 msgr2=0x7f0af4071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:56.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 0x7f0af4071bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0b00005fd0 tx=0x7f0b00005ee0 comp rx=0 tx=0).stop 2024-03-11T21:22:56.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 msgr2=0x7f0b0c199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:22:56.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.159+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b04007e20 tx=0x7f0b0400a340 comp rx=0 tx=0).stop 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 shutdown_connections 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f0af406f730 0x7f0af4071bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c103980 0x7f0b0c1a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b0c101e60 0x7f0b0c199780 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 --2- 172.21.15.203:0/4024964368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c102fc0 0x7f0b0c199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 >> 172.21.15.203:0/4024964368 conn(0x7f0b0c0fd400 msgr2=0x7f0b0c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 shutdown_connections 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 -- 172.21.15.203:0/4024964368 wait complete. 2024-03-11T21:22:56.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:22:56.163+0000 7f0b120fe700 1 librados: shutdown 2024-03-11T21:22:56.692 DEBUG:teuthology.orchestra.run.smithi143:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:22:56.695 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi203 2024-03-11T21:22:56.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T21:22:57.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: cluster 2024-03-11T21:22:55.543840+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:57.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.152216+0000 mgr.y (mgr.14152) 335 : audit 0 from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi143=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: cephadm 2024-03-11T21:22:56.156232+0000 mgr.y (mgr.14152) 336 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.164828+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.183209+0000 mon.a (mon.0) 650 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.809746+0000 mon.a (mon.0) 651 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.811376+0000 mon.a (mon.0) 652 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.820858+0000 mon.a (mon.0) 653 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.825644+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.830135+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.842989+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[21512]: audit 2024-03-11T21:22:56.846946+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:57.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: cluster 2024-03-11T21:22:55.543840+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.152216+0000 mgr.y (mgr.14152) 335 : audit 0 from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi143=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: cephadm 2024-03-11T21:22:56.156232+0000 mgr.y (mgr.14152) 336 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.164828+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.183209+0000 mon.a (mon.0) 650 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.809746+0000 mon.a (mon.0) 651 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.811376+0000 mon.a (mon.0) 652 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.820858+0000 mon.a (mon.0) 653 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.825644+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.830135+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.842989+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:57 smithi143 bash[29284]: audit 2024-03-11T21:22:56.846946+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:57.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: cluster 2024-03-11T21:22:55.543840+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.152216+0000 mgr.y (mgr.14152) 335 : audit 0 from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi143=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: cephadm 2024-03-11T21:22:56.156232+0000 mgr.y (mgr.14152) 336 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.164828+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.183209+0000 mon.a (mon.0) 650 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.809746+0000 mon.a (mon.0) 651 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.811376+0000 mon.a (mon.0) 652 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.820858+0000 mon.a (mon.0) 653 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.825644+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:22:57.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.830135+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:22:57.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.842989+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:22:57.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:57 smithi203 bash[23654]: audit 2024-03-11T21:22:56.846946+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:58.401 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:58 smithi143 bash[21512]: cephadm 2024-03-11T21:22:56.848912+0000 mgr.y (mgr.14152) 337 : cephadm 1 Deploying daemon rgw.foo.a on smithi143 2024-03-11T21:22:58.401 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:58 smithi143 bash[29284]: cephadm 2024-03-11T21:22:56.848912+0000 mgr.y (mgr.14152) 337 : cephadm 1 Deploying daemon rgw.foo.a on smithi143 2024-03-11T21:22:58.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:58 smithi203 bash[23654]: cephadm 2024-03-11T21:22:56.848912+0000 mgr.y (mgr.14152) 337 : cephadm 1 Deploying daemon rgw.foo.a on smithi143 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[21512]: cluster 2024-03-11T21:22:57.544447+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[21512]: cluster 2024-03-11T21:22:57.703682+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[21512]: cluster 2024-03-11T21:22:57.703777+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[21512]: cluster 2024-03-11T21:22:58.179403+0000 mon.a (mon.0) 658 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[29284]: cluster 2024-03-11T21:22:57.544447+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[29284]: cluster 2024-03-11T21:22:57.703682+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[29284]: cluster 2024-03-11T21:22:57.703777+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:59.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 bash[29284]: cluster 2024-03-11T21:22:58.179403+0000 mon.a (mon.0) 658 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:22:59.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:59 smithi203 bash[23654]: cluster 2024-03-11T21:22:57.544447+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:22:59.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:59 smithi203 bash[23654]: cluster 2024-03-11T21:22:57.703682+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:22:59.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:59 smithi203 bash[23654]: cluster 2024-03-11T21:22:57.703777+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:22:59.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:22:59 smithi203 bash[23654]: cluster 2024-03-11T21:22:58.179403+0000 mon.a (mon.0) 658 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:22:59.713 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.714 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.714 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.715 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.715 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.716 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:59.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.188 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.189 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.189 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.190 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:00.190 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:22:59 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: cluster 2024-03-11T21:22:59.544946+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.894809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.900568+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.924206+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: cephadm 2024-03-11T21:22:59.924870+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.930336+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.936292+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[21512]: audit 2024-03-11T21:22:59.984960+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: cluster 2024-03-11T21:22:59.544946+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.894809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.900568+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.924206+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: cephadm 2024-03-11T21:22:59.924870+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.930336+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.936292+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:00 smithi143 bash[29284]: audit 2024-03-11T21:22:59.984960+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: cluster 2024-03-11T21:22:59.544946+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.894809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.900568+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.924206+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: cephadm 2024-03-11T21:22:59.924870+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi143=foo.a;count:1 2024-03-11T21:23:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.930336+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.936292+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:01.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:00 smithi203 bash[23654]: audit 2024-03-11T21:22:59.984960+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:01.422 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:03.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[21512]: cluster 2024-03-11T21:23:01.545469+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:03.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[21512]: cluster 2024-03-11T21:23:01.915433+0000 mon.a (mon.0) 665 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:23:03.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[21512]: audit 2024-03-11T21:23:01.920004+0000 mon.a (mon.0) 666 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:23:03.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[21512]: audit 2024-03-11T21:23:02.719910+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:03.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[29284]: cluster 2024-03-11T21:23:01.545469+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:03.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[29284]: cluster 2024-03-11T21:23:01.915433+0000 mon.a (mon.0) 665 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:23:03.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[29284]: audit 2024-03-11T21:23:01.920004+0000 mon.a (mon.0) 666 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:23:03.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:02 smithi143 bash[29284]: audit 2024-03-11T21:23:02.719910+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:02 smithi203 bash[23654]: cluster 2024-03-11T21:23:01.545469+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:02 smithi203 bash[23654]: cluster 2024-03-11T21:23:01.915433+0000 mon.a (mon.0) 665 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:23:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:02 smithi203 bash[23654]: audit 2024-03-11T21:23:01.920004+0000 mon.a (mon.0) 666 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:23:03.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:02 smithi203 bash[23654]: audit 2024-03-11T21:23:02.719910+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/1415097952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524105c30 msgr2=0x7fd52410c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- 172.21.15.203:0/1415097952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524105c30 0x7fd52410c4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00b0d0 tx=0x7fd51c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/1415097952 shutdown_connections 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- 172.21.15.203:0/1415097952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524104090 0x7fd5241044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- 172.21.15.203:0/1415097952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524105c30 0x7fd52410c4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- 172.21.15.203:0/1415097952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd5241056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/1415097952 >> 172.21.15.203:0/1415097952 conn(0x7fd5240ff670 msgr2=0x7fd524101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/1415097952 shutdown_connections 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/1415097952 wait complete. 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 librados: starting msgr at 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 librados: starting objecter 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 Processor -- start 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- start start 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 librados: setting wanted keys 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 librados: calling monclient init 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd524199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524105c30 0x7fd5241a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52410f260 con 0x7fd524105c30 2024-03-11T21:23:03.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52410f0e0 con 0x7fd524104090 2024-03-11T21:23:03.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52410f3e0 con 0x7fd524105290 2024-03-11T21:23:03.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:03.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:47408/0 (socket says 172.21.15.203:47408) 2024-03-11T21:23:03.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 -- 172.21.15.203:0/912379331 learned_addr learned my addr 172.21.15.203:0/912379331 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:23:03.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd522d9d700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524105c30 0x7fd5241a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd521d9b700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd524199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 -- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 msgr2=0x7fd524199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd524199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 -- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524105c30 msgr2=0x7fd5241a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524105c30 0x7fd5241a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 -- 172.21.15.203:0/912379331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5241a1480 con 0x7fd524104090 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd521d9b700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd524199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd52259c700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd51400c970 tx=0x7fd51400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 10 client.?.objecter ms_handle_connect 0x7fd524104090 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514004220 con 0x7fd524104090 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd514007d20 con 0x7fd524104090 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5241a1770 con 0x7fd524104090 2024-03-11T21:23:03.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 10 client.24322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514003ca0 con 0x7fd524104090 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5241a1d00 con 0x7fd524104090 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 librados: init done 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52404faa0 con 0x7fd524104090 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.763+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7fd5140078a0 con 0x7fd524104090 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 0x7fd510071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(77..77 src has 21..77) v4 ==== 5646+0+0 (secure 0 0 0) 0x7fd51408d140 con 0x7fd524104090 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 10 client.24322.objecter ms_dispatch 0x7fd524001550 osd_map(77..77 src has 21..77) v4 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 3 client.24322.objecter handle_osd_map got epochs [77,77] > 0 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 3 client.24322.objecter handle_osd_map decoding full epoch 77 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 20 client.24322.objecter dump_active .. 0 homeless 2024-03-11T21:23:03.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd521d9b700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 0x7fd510071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:03.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd51b7fe700 10 client.24322.objecter ms_handle_connect 0x7fd51006f600 2024-03-11T21:23:03.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.767+0000 7fd521d9b700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 0x7fd510071ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd50c005fd0 tx=0x7fd50c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:03.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.771+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd514014030 con 0x7fd524104090 2024-03-11T21:23:03.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:03.987+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fd52402d830 con 0x7fd524104090 2024-03-11T21:23:04.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:03 smithi143 bash[21512]: audit 2024-03-11T21:23:02.909495+0000 mon.a (mon.0) 668 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:23:04.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:03 smithi143 bash[21512]: cluster 2024-03-11T21:23:02.917490+0000 mon.a (mon.0) 669 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:23:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:03 smithi143 bash[29284]: audit 2024-03-11T21:23:02.909495+0000 mon.a (mon.0) 668 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:23:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:03 smithi143 bash[29284]: cluster 2024-03-11T21:23:02.917490+0000 mon.a (mon.0) 669 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:23:04.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:03 smithi203 bash[23654]: audit 2024-03-11T21:23:02.909495+0000 mon.a (mon.0) 668 : audit 1 from='client.? 172.21.15.143:0/3462036358' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:23:04.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:03 smithi203 bash[23654]: cluster 2024-03-11T21:23:02.917490+0000 mon.a (mon.0) 669 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:23:04.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.919+0000 7fd51b7fe700 1 -- 172.21.15.203:0/912379331 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v79) v1 ==== 160+0+0 (secure 0 0 0) 0x7fd51405aed0 con 0x7fd524104090 2024-03-11T21:23:04.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 10 librados: watch_flush enter 2024-03-11T21:23:04.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 10 librados: watch_flush exit 2024-03-11T21:23:04.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 20 client.24322.objecter shutdown clearing up homeless session... 2024-03-11T21:23:04.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 10 client.24322.objecter successfully canceled tick 2024-03-11T21:23:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 msgr2=0x7fd510071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 0x7fd510071ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd50c005fd0 tx=0x7fd50c005dc0 comp rx=0 tx=0).stop 2024-03-11T21:23:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 msgr2=0x7fd524199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.923+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd51400c970 tx=0x7fd51400cd30 comp rx=0 tx=0).stop 2024-03-11T21:23:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 shutdown_connections 2024-03-11T21:23:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7fd51006f600 0x7fd510071ac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524105c30 0x7fd5241a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd524104090 0x7fd524199820 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 --2- 172.21.15.203:0/912379331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd524105290 0x7fd524199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 >> 172.21.15.203:0/912379331 conn(0x7fd5240ff670 msgr2=0x7fd524108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 shutdown_connections 2024-03-11T21:23:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 -- 172.21.15.203:0/912379331 wait complete. 2024-03-11T21:23:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:04.927+0000 7fd528c04700 1 librados: shutdown 2024-03-11T21:23:04.918 INFO:teuthology.orchestra.run.smithi203.stderr:pool 'datapool' created 2024-03-11T21:23:05.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: cluster 2024-03-11T21:23:03.546023+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v342: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: cluster 2024-03-11T21:23:03.924590+0000 mon.a (mon.0) 670 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: audit 2024-03-11T21:23:03.930380+0000 mon.a (mon.0) 671 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: audit 2024-03-11T21:23:03.991915+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.203:0/912379331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: audit 2024-03-11T21:23:03.996694+0000 mon.a (mon.0) 672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: audit 2024-03-11T21:23:04.915916+0000 mon.a (mon.0) 673 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: audit 2024-03-11T21:23:04.916224+0000 mon.a (mon.0) 674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[21512]: cluster 2024-03-11T21:23:04.925157+0000 mon.a (mon.0) 675 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: cluster 2024-03-11T21:23:03.546023+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v342: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: cluster 2024-03-11T21:23:03.924590+0000 mon.a (mon.0) 670 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: audit 2024-03-11T21:23:03.930380+0000 mon.a (mon.0) 671 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: audit 2024-03-11T21:23:03.991915+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.203:0/912379331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: audit 2024-03-11T21:23:03.996694+0000 mon.a (mon.0) 672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: audit 2024-03-11T21:23:04.915916+0000 mon.a (mon.0) 673 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:23:05.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: audit 2024-03-11T21:23:04.916224+0000 mon.a (mon.0) 674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:23:05.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:05 smithi143 bash[29284]: cluster 2024-03-11T21:23:04.925157+0000 mon.a (mon.0) 675 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:23:05.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: cluster 2024-03-11T21:23:03.546023+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v342: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: cluster 2024-03-11T21:23:03.924590+0000 mon.a (mon.0) 670 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: audit 2024-03-11T21:23:03.930380+0000 mon.a (mon.0) 671 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: audit 2024-03-11T21:23:03.991915+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.203:0/912379331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: audit 2024-03-11T21:23:03.996694+0000 mon.a (mon.0) 672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: audit 2024-03-11T21:23:04.915916+0000 mon.a (mon.0) 673 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: audit 2024-03-11T21:23:04.916224+0000 mon.a (mon.0) 674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:23:05.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:05 smithi203 bash[23654]: cluster 2024-03-11T21:23:04.925157+0000 mon.a (mon.0) 675 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:23:06.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:05.403275+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:05.411581+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:05.413652+0000 mon.a (mon.0) 678 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:06.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:05.414595+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: cephadm 2024-03-11T21:23:05.419133+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: cluster 2024-03-11T21:23:06.053522+0000 mon.a (mon.0) 680 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:06.054949+0000 mon.c (mon.2) 12 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:06.059702+0000 mon.a (mon.0) 681 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[21512]: audit 2024-03-11T21:23:06.061261+0000 mon.a (mon.0) 682 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:05.403275+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:05.411581+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:05.413652+0000 mon.a (mon.0) 678 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:05.414595+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: cephadm 2024-03-11T21:23:05.419133+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: cluster 2024-03-11T21:23:06.053522+0000 mon.a (mon.0) 680 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:23:06.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:06.054949+0000 mon.c (mon.2) 12 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:06.059702+0000 mon.a (mon.0) 681 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:06 smithi143 bash[29284]: audit 2024-03-11T21:23:06.061261+0000 mon.a (mon.0) 682 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:05.403275+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:05.411581+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:05.413652+0000 mon.a (mon.0) 678 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:05.414595+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: cephadm 2024-03-11T21:23:05.419133+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: cluster 2024-03-11T21:23:06.053522+0000 mon.a (mon.0) 680 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:06.054949+0000 mon.c (mon.2) 12 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:06.059702+0000 mon.a (mon.0) 681 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:06.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:06 smithi203 bash[23654]: audit 2024-03-11T21:23:06.061261+0000 mon.a (mon.0) 682 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[21512]: cluster 2024-03-11T21:23:05.546715+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v345: 68 pgs: 16 creating+peering, 28 unknown, 24 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[21512]: cluster 2024-03-11T21:23:06.410289+0000 mon.a (mon.0) 683 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[21512]: audit 2024-03-11T21:23:07.047032+0000 mon.a (mon.0) 684 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[21512]: audit 2024-03-11T21:23:07.047259+0000 mon.a (mon.0) 685 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[21512]: cluster 2024-03-11T21:23:07.057920+0000 mon.a (mon.0) 686 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:23:08.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[29284]: cluster 2024-03-11T21:23:05.546715+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v345: 68 pgs: 16 creating+peering, 28 unknown, 24 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2024-03-11T21:23:08.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[29284]: cluster 2024-03-11T21:23:06.410289+0000 mon.a (mon.0) 683 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:23:08.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[29284]: audit 2024-03-11T21:23:07.047032+0000 mon.a (mon.0) 684 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[29284]: audit 2024-03-11T21:23:07.047259+0000 mon.a (mon.0) 685 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:08 smithi143 bash[29284]: cluster 2024-03-11T21:23:07.057920+0000 mon.a (mon.0) 686 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:23:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:08 smithi203 bash[23654]: cluster 2024-03-11T21:23:05.546715+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v345: 68 pgs: 16 creating+peering, 28 unknown, 24 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2024-03-11T21:23:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:08 smithi203 bash[23654]: cluster 2024-03-11T21:23:06.410289+0000 mon.a (mon.0) 683 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:23:08.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:08 smithi203 bash[23654]: audit 2024-03-11T21:23:07.047032+0000 mon.a (mon.0) 684 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:08 smithi203 bash[23654]: audit 2024-03-11T21:23:07.047259+0000 mon.a (mon.0) 685 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:23:08.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:08 smithi203 bash[23654]: cluster 2024-03-11T21:23:07.057920+0000 mon.a (mon.0) 686 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:23:09.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[21512]: cluster 2024-03-11T21:23:07.547540+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v348: 100 pgs: 4 creating+activating, 10 creating+peering, 28 unknown, 58 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:23:09.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[21512]: cluster 2024-03-11T21:23:08.113594+0000 mon.a (mon.0) 687 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:23:09.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[21512]: audit 2024-03-11T21:23:08.116076+0000 mon.c (mon.2) 13 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[21512]: audit 2024-03-11T21:23:08.119591+0000 mon.a (mon.0) 688 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[21512]: audit 2024-03-11T21:23:08.123355+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[29284]: cluster 2024-03-11T21:23:07.547540+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v348: 100 pgs: 4 creating+activating, 10 creating+peering, 28 unknown, 58 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:23:09.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[29284]: cluster 2024-03-11T21:23:08.113594+0000 mon.a (mon.0) 687 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:23:09.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[29284]: audit 2024-03-11T21:23:08.116076+0000 mon.c (mon.2) 13 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[29284]: audit 2024-03-11T21:23:08.119591+0000 mon.a (mon.0) 688 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:09 smithi143 bash[29284]: audit 2024-03-11T21:23:08.123355+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:09 smithi203 bash[23654]: cluster 2024-03-11T21:23:07.547540+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v348: 100 pgs: 4 creating+activating, 10 creating+peering, 28 unknown, 58 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:23:09.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:09 smithi203 bash[23654]: cluster 2024-03-11T21:23:08.113594+0000 mon.a (mon.0) 687 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:23:09.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:09 smithi203 bash[23654]: audit 2024-03-11T21:23:08.116076+0000 mon.c (mon.2) 13 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:09 smithi203 bash[23654]: audit 2024-03-11T21:23:08.119591+0000 mon.a (mon.0) 688 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:09.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:09 smithi203 bash[23654]: audit 2024-03-11T21:23:08.123355+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:23:10.281 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: audit 2024-03-11T21:23:09.108808+0000 mon.a (mon.0) 690 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: audit 2024-03-11T21:23:09.108958+0000 mon.a (mon.0) 691 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: cluster 2024-03-11T21:23:09.118193+0000 mon.a (mon.0) 692 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: audit 2024-03-11T21:23:09.145679+0000 mon.c (mon.2) 14 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: audit 2024-03-11T21:23:09.147040+0000 mon.a (mon.0) 693 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: audit 2024-03-11T21:23:09.149213+0000 mon.a (mon.0) 694 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: cluster 2024-03-11T21:23:09.518945+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[21512]: cluster 2024-03-11T21:23:09.519036+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: audit 2024-03-11T21:23:09.108808+0000 mon.a (mon.0) 690 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: audit 2024-03-11T21:23:09.108958+0000 mon.a (mon.0) 691 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: cluster 2024-03-11T21:23:09.118193+0000 mon.a (mon.0) 692 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: audit 2024-03-11T21:23:09.145679+0000 mon.c (mon.2) 14 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: audit 2024-03-11T21:23:09.147040+0000 mon.a (mon.0) 693 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: audit 2024-03-11T21:23:09.149213+0000 mon.a (mon.0) 694 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: cluster 2024-03-11T21:23:09.518945+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:10.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:10 smithi143 bash[29284]: cluster 2024-03-11T21:23:09.519036+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:10.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: audit 2024-03-11T21:23:09.108808+0000 mon.a (mon.0) 690 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: audit 2024-03-11T21:23:09.108958+0000 mon.a (mon.0) 691 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:23:10.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: cluster 2024-03-11T21:23:09.118193+0000 mon.a (mon.0) 692 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:23:10.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: audit 2024-03-11T21:23:09.145679+0000 mon.c (mon.2) 14 : audit 1 from='client.? 172.21.15.143:0/3339680007' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: audit 2024-03-11T21:23:09.147040+0000 mon.a (mon.0) 693 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: audit 2024-03-11T21:23:09.149213+0000 mon.a (mon.0) 694 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:23:10.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: cluster 2024-03-11T21:23:09.518945+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:10.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:10 smithi203 bash[23654]: cluster 2024-03-11T21:23:09.519036+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: cluster 2024-03-11T21:23:09.548461+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v351: 132 pgs: 4 creating+activating, 4 creating+peering, 41 unknown, 83 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.141550+0000 mon.a (mon.0) 695 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.141716+0000 mon.a (mon.0) 696 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: cluster 2024-03-11T21:23:10.152508+0000 mon.a (mon.0) 697 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.449671+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.377 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.455163+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.468343+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.530669+0000 mon.a (mon.0) 701 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:11.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.532840+0000 mon.a (mon.0) 702 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:11.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.533822+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:11.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:11 smithi203 bash[23654]: audit 2024-03-11T21:23:10.810583+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: cluster 2024-03-11T21:23:09.548461+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v351: 132 pgs: 4 creating+activating, 4 creating+peering, 41 unknown, 83 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:23:11.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.141550+0000 mon.a (mon.0) 695 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.141716+0000 mon.a (mon.0) 696 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: cluster 2024-03-11T21:23:10.152508+0000 mon.a (mon.0) 697 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:23:11.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.449671+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.455163+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.468343+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.530669+0000 mon.a (mon.0) 701 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.532840+0000 mon.a (mon.0) 702 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.533822+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[21512]: audit 2024-03-11T21:23:10.810583+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: cluster 2024-03-11T21:23:09.548461+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v351: 132 pgs: 4 creating+activating, 4 creating+peering, 41 unknown, 83 active+clean; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.141550+0000 mon.a (mon.0) 695 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.141716+0000 mon.a (mon.0) 696 : audit 1 from='client.? 172.21.15.143:0/3842484601' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:23:11.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: cluster 2024-03-11T21:23:10.152508+0000 mon.a (mon.0) 697 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.449671+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.455163+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.468343+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.530669+0000 mon.a (mon.0) 701 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.532840+0000 mon.a (mon.0) 702 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.533822+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:11.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:11 smithi143 bash[29284]: audit 2024-03-11T21:23:10.810583+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:12.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:12 smithi143 bash[21512]: cephadm 2024-03-11T21:23:10.539187+0000 mgr.y (mgr.14152) 347 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:12.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:12 smithi143 bash[29284]: cephadm 2024-03-11T21:23:10.539187+0000 mgr.y (mgr.14152) 347 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:12.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:12 smithi203 bash[23654]: cephadm 2024-03-11T21:23:10.539187+0000 mgr.y (mgr.14152) 347 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[21512]: cluster 2024-03-11T21:23:11.550053+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 4 creating+activating, 3 creating+peering, 20 unknown, 105 active+clean; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.9 KiB/s wr, 48 op/s 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[21512]: audit 2024-03-11T21:23:12.957285+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.203:0/3848484854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[21512]: audit 2024-03-11T21:23:12.962030+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[29284]: cluster 2024-03-11T21:23:11.550053+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 4 creating+activating, 3 creating+peering, 20 unknown, 105 active+clean; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.9 KiB/s wr, 48 op/s 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[29284]: audit 2024-03-11T21:23:12.957285+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.203:0/3848484854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:13.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:13 smithi143 bash[29284]: audit 2024-03-11T21:23:12.962030+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:13.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:13 smithi203 bash[23654]: cluster 2024-03-11T21:23:11.550053+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 4 creating+activating, 3 creating+peering, 20 unknown, 105 active+clean; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.9 KiB/s wr, 48 op/s 2024-03-11T21:23:13.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:13 smithi203 bash[23654]: audit 2024-03-11T21:23:12.957285+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.203:0/3848484854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:13.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:13 smithi203 bash[23654]: audit 2024-03-11T21:23:12.962030+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:23:14.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:14 smithi143 bash[21512]: audit 2024-03-11T21:23:13.161001+0000 mon.a (mon.0) 706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:23:14.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:14 smithi143 bash[21512]: cluster 2024-03-11T21:23:13.168614+0000 mon.a (mon.0) 707 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:23:14.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:14 smithi143 bash[29284]: audit 2024-03-11T21:23:13.161001+0000 mon.a (mon.0) 706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:23:14.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:14 smithi143 bash[29284]: cluster 2024-03-11T21:23:13.168614+0000 mon.a (mon.0) 707 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:23:14.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:14 smithi203 bash[23654]: audit 2024-03-11T21:23:13.161001+0000 mon.a (mon.0) 706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:23:14.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:14 smithi203 bash[23654]: cluster 2024-03-11T21:23:13.168614+0000 mon.a (mon.0) 707 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:23:15.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:13.550835+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.3 KiB/s wr, 144 op/s 2024-03-11T21:23:15.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:14.172597+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:23:15.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:14.172654+0000 mon.a (mon.0) 709 : cluster 1 Cluster is now healthy 2024-03-11T21:23:15.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:14.190841+0000 mon.a (mon.0) 710 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:23:15.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:14.192687+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:15.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[21512]: cluster 2024-03-11T21:23:14.192811+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:15.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:13.550835+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.3 KiB/s wr, 144 op/s 2024-03-11T21:23:15.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:14.172597+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:23:15.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:14.172654+0000 mon.a (mon.0) 709 : cluster 1 Cluster is now healthy 2024-03-11T21:23:15.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:14.190841+0000 mon.a (mon.0) 710 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:23:15.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:14.192687+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:15.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:15 smithi143 bash[29284]: cluster 2024-03-11T21:23:14.192811+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:15.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:13.550835+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.3 KiB/s wr, 144 op/s 2024-03-11T21:23:15.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:14.172597+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:23:15.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:14.172654+0000 mon.a (mon.0) 709 : cluster 1 Cluster is now healthy 2024-03-11T21:23:15.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:14.190841+0000 mon.a (mon.0) 710 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:23:15.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:14.192687+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:15.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:15 smithi203 bash[23654]: cluster 2024-03-11T21:23:14.192811+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:15.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.203 --placement '1;smithi203=iscsi.a' 2024-03-11T21:23:16.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:16 smithi143 bash[21512]: cluster 2024-03-11T21:23:15.190368+0000 mon.a (mon.0) 711 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:23:16.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:16 smithi143 bash[29284]: cluster 2024-03-11T21:23:15.190368+0000 mon.a (mon.0) 711 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:23:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:16 smithi203 bash[23654]: cluster 2024-03-11T21:23:15.190368+0000 mon.a (mon.0) 711 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:23:17.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:17 smithi203 bash[23654]: cluster 2024-03-11T21:23:15.551756+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 6.7 KiB/s wr, 210 op/s 2024-03-11T21:23:17.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:17 smithi143 bash[21512]: cluster 2024-03-11T21:23:15.551756+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 6.7 KiB/s wr, 210 op/s 2024-03-11T21:23:17.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:17 smithi143 bash[29284]: cluster 2024-03-11T21:23:15.551756+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 6.7 KiB/s wr, 210 op/s 2024-03-11T21:23:19.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:19 smithi203 bash[23654]: cluster 2024-03-11T21:23:17.552938+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.2 KiB/s wr, 155 op/s 2024-03-11T21:23:19.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:19 smithi203 bash[23654]: cluster 2024-03-11T21:23:18.520203+0000 mon.a (mon.0) 712 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:23:19.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:19 smithi143 bash[29284]: cluster 2024-03-11T21:23:17.552938+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.2 KiB/s wr, 155 op/s 2024-03-11T21:23:19.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:19 smithi143 bash[29284]: cluster 2024-03-11T21:23:18.520203+0000 mon.a (mon.0) 712 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:23:19.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:19 smithi143 bash[21512]: cluster 2024-03-11T21:23:17.552938+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.2 KiB/s wr, 155 op/s 2024-03-11T21:23:19.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:19 smithi143 bash[21512]: cluster 2024-03-11T21:23:18.520203+0000 mon.a (mon.0) 712 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:23:20.559 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:21.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:21 smithi203 bash[23654]: cluster 2024-03-11T21:23:19.553785+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.5 KiB/s wr, 60 op/s 2024-03-11T21:23:21.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:21 smithi143 bash[29284]: cluster 2024-03-11T21:23:19.553785+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.5 KiB/s wr, 60 op/s 2024-03-11T21:23:21.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:21 smithi143 bash[21512]: cluster 2024-03-11T21:23:19.553785+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.5 KiB/s wr, 60 op/s 2024-03-11T21:23:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/3141827121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 msgr2=0x7f6e101014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- 172.21.15.203:0/3141827121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e101014f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009990 tx=0x7f6e00009ca0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/3141827121 shutdown_connections 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- 172.21.15.203:0/3141827121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10100710 0x7f6e10100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- 172.21.15.203:0/3141827121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10101a30 0x7f6e10101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- 172.21.15.203:0/3141827121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e101014f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/3141827121 >> 172.21.15.203:0/3141827121 conn(0x7f6e100fc330 msgr2=0x7f6e100fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/3141827121 shutdown_connections 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/3141827121 wait complete. 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 librados: starting msgr at 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 librados: starting objecter 2024-03-11T21:23:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 Processor -- start 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- start start 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 librados: setting wanted keys 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 librados: calling monclient init 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 0x7f6e101998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10101a30 0x7f6e101a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1010f190 con 0x7f6e10101a30 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1010f010 con 0x7f6e10101070 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1010f310 con 0x7f6e10100710 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:60418/0 (socket says 172.21.15.203:60418) 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 -- 172.21.15.203:0/4293799218 learned_addr learned my addr 172.21.15.203:0/4293799218 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e15790700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 0x7f6e101998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:22.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e15f91700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10101a30 0x7f6e101a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 -- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 msgr2=0x7f6e101998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 0x7f6e101998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 -- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10101a30 msgr2=0x7f6e101a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10101a30 0x7f6e101a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 -- 172.21.15.203:0/4293799218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e101a1520 con 0x7f6e10101070 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e15790700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 0x7f6e101998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e14f8f700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006010 tx=0x7f6e000047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e067fc700 10 client.?.objecter ms_handle_connect 0x7f6e10101070 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e067fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00027070 con 0x7f6e10101070 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e101a17b0 con 0x7f6e10101070 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 10 client.24376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e101a1c60 con 0x7f6e10101070 2024-03-11T21:23:22.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e179f4700 1 librados: init done 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6e00004140 con 0x7f6e10101070 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.823+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00022d60 con 0x7f6e10101070 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df4005120 con 0x7f6e10101070 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f6e0002a070 con 0x7f6e10101070 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 0x7f6dfc071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 21..88) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f6e0008ee10 con 0x7f6e10101070 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 10 client.24376.objecter ms_dispatch 0x7f6e10001550 osd_map(88..88 src has 21..88) v4 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 3 client.24376.objecter handle_osd_map got epochs [88,88] > 0 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 3 client.24376.objecter handle_osd_map decoding full epoch 88 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 20 client.24376.objecter dump_active .. 0 homeless 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e15790700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 0x7f6dfc071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e067fc700 10 client.24376.objecter ms_handle_connect 0x7f6dfc06f730 2024-03-11T21:23:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.827+0000 7f6e15790700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 0x7f6dfc071bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009dd0 tx=0x7f6e0c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:22.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:22.831+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6e00013070 con 0x7f6e10101070 2024-03-11T21:23:23.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.051+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f6df4000c90 con 0x7f6dfc06f730 2024-03-11T21:23:23.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e067fc700 1 -- 172.21.15.203:0/4293799218 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f6df4000c90 con 0x7f6dfc06f730 2024-03-11T21:23:23.053 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled iscsi.datapool update... 2024-03-11T21:23:23.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 10 librados: watch_flush enter 2024-03-11T21:23:23.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 10 librados: watch_flush exit 2024-03-11T21:23:23.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 20 client.24376.objecter shutdown clearing up homeless session... 2024-03-11T21:23:23.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 10 client.24376.objecter successfully canceled tick 2024-03-11T21:23:23.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 msgr2=0x7f6dfc071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 0x7f6dfc071bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009dd0 tx=0x7f6e0c009040 comp rx=0 tx=0).stop 2024-03-11T21:23:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 msgr2=0x7f6e10199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006010 tx=0x7f6e000047b0 comp rx=0 tx=0).stop 2024-03-11T21:23:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.071+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 shutdown_connections 2024-03-11T21:23:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f6dfc06f730 0x7f6dfc071bf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e10101a30 0x7f6e101a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e10100710 0x7f6e101998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 --2- 172.21.15.203:0/4293799218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e10101070 0x7f6e10199e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 >> 172.21.15.203:0/4293799218 conn(0x7f6e100fc330 msgr2=0x7f6e100fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 shutdown_connections 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 -- 172.21.15.203:0/4293799218 wait complete. 2024-03-11T21:23:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:23.075+0000 7f6e179f4700 1 librados: shutdown 2024-03-11T21:23:23.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:23 smithi203 bash[23654]: cluster 2024-03-11T21:23:21.554865+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.2 KiB/s wr, 48 op/s 2024-03-11T21:23:23.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:23 smithi203 bash[23654]: audit 2024-03-11T21:23:23.076609+0000 mon.a (mon.0) 713 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:23.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:23 smithi203 bash[23654]: audit 2024-03-11T21:23:23.108349+0000 mon.a (mon.0) 714 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:23.610 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T21:23:23.611 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:23:23.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:23:23.619 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[29284]: cluster 2024-03-11T21:23:21.554865+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.2 KiB/s wr, 48 op/s 2024-03-11T21:23:23.619 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[29284]: audit 2024-03-11T21:23:23.076609+0000 mon.a (mon.0) 713 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:23.620 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[29284]: audit 2024-03-11T21:23:23.108349+0000 mon.a (mon.0) 714 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:23.620 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[21512]: cluster 2024-03-11T21:23:21.554865+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.2 KiB/s wr, 48 op/s 2024-03-11T21:23:23.620 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[21512]: audit 2024-03-11T21:23:23.076609+0000 mon.a (mon.0) 713 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:23.620 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:23 smithi143 bash[21512]: audit 2024-03-11T21:23:23.108349+0000 mon.a (mon.0) 714 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:23.623 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:23:23.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:23:23.636 DEBUG:teuthology.orchestra.run.smithi203:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T21:23:23.682 INFO:tasks.cephadm:Adding prometheus.a on smithi203 2024-03-11T21:23:23.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi203=a' 2024-03-11T21:23:24.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.061167+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: cephadm 2024-03-11T21:23:23.067876+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: cluster 2024-03-11T21:23:23.518545+0000 mon.a (mon.0) 715 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.735369+0000 mon.a (mon.0) 716 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.736972+0000 mon.a (mon.0) 717 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.747669+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.755375+0000 mon.a (mon.0) 719 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.758275+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:23:24.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:24 smithi203 bash[23654]: audit 2024-03-11T21:23:23.789939+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.061167+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: cephadm 2024-03-11T21:23:23.067876+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: cluster 2024-03-11T21:23:23.518545+0000 mon.a (mon.0) 715 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.735369+0000 mon.a (mon.0) 716 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.736972+0000 mon.a (mon.0) 717 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.747669+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.755375+0000 mon.a (mon.0) 719 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:23:24.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.758275+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[21512]: audit 2024-03-11T21:23:23.789939+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.061167+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: cephadm 2024-03-11T21:23:23.067876+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: cluster 2024-03-11T21:23:23.518545+0000 mon.a (mon.0) 715 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.735369+0000 mon.a (mon.0) 716 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.736972+0000 mon.a (mon.0) 717 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.747669+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.755375+0000 mon.a (mon.0) 719 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.758275+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:23:24.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:24 smithi143 bash[29284]: audit 2024-03-11T21:23:23.789939+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:25.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:25 smithi203 bash[23654]: cluster 2024-03-11T21:23:23.555424+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-03-11T21:23:25.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:25 smithi203 bash[23654]: cephadm 2024-03-11T21:23:23.792209+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi203 2024-03-11T21:23:25.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:25 smithi143 bash[21512]: cluster 2024-03-11T21:23:23.555424+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-03-11T21:23:25.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:25 smithi143 bash[21512]: cephadm 2024-03-11T21:23:23.792209+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi203 2024-03-11T21:23:25.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:25 smithi143 bash[29284]: cluster 2024-03-11T21:23:23.555424+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-03-11T21:23:25.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:25 smithi143 bash[29284]: cephadm 2024-03-11T21:23:23.792209+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi203 2024-03-11T21:23:26.064 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.065 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.065 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.065 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.066 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.066 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:23:25 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.335 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.335 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.336 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.336 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.336 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:26.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:26 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:27.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: cluster 2024-03-11T21:23:25.556456+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-11T21:23:27.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: cluster 2024-03-11T21:23:26.290305+0000 osd.7 (osd.7) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:27.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: cluster 2024-03-11T21:23:26.290372+0000 osd.7 (osd.7) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:27.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: audit 2024-03-11T21:23:26.417031+0000 mon.a (mon.0) 722 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: audit 2024-03-11T21:23:26.429110+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: audit 2024-03-11T21:23:26.472487+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: audit 2024-03-11T21:23:26.490081+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: audit 2024-03-11T21:23:26.555071+0000 mon.a (mon.0) 726 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:27.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[23654]: cluster 2024-03-11T21:23:27.244065+0000 mon.a (mon.0) 727 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:23:27.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: cluster 2024-03-11T21:23:25.556456+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-11T21:23:27.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: cluster 2024-03-11T21:23:26.290305+0000 osd.7 (osd.7) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: cluster 2024-03-11T21:23:26.290372+0000 osd.7 (osd.7) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: audit 2024-03-11T21:23:26.417031+0000 mon.a (mon.0) 722 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: audit 2024-03-11T21:23:26.429110+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: audit 2024-03-11T21:23:26.472487+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: audit 2024-03-11T21:23:26.490081+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: audit 2024-03-11T21:23:26.555071+0000 mon.a (mon.0) 726 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[21512]: cluster 2024-03-11T21:23:27.244065+0000 mon.a (mon.0) 727 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: cluster 2024-03-11T21:23:25.556456+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-11T21:23:27.834 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: cluster 2024-03-11T21:23:26.290305+0000 osd.7 (osd.7) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: cluster 2024-03-11T21:23:26.290372+0000 osd.7 (osd.7) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: audit 2024-03-11T21:23:26.417031+0000 mon.a (mon.0) 722 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: audit 2024-03-11T21:23:26.429110+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: audit 2024-03-11T21:23:26.472487+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: audit 2024-03-11T21:23:26.490081+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: audit 2024-03-11T21:23:26.555071+0000 mon.a (mon.0) 726 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:27.835 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:27 smithi143 bash[29284]: cluster 2024-03-11T21:23:27.244065+0000 mon.a (mon.0) 727 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:23:27.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:27 smithi203 bash[48389]: debug Processing osd blocklist entries for this node 2024-03-11T21:23:27.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[48389]: debug Reading the configuration object to update local LIO configuration 2024-03-11T21:23:27.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[48389]: debug Configuration does not have an entry for this host(smithi203.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T21:23:28.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[48389]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[21512]: cephadm 2024-03-11T21:23:26.474205+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[21512]: cluster 2024-03-11T21:23:27.509429+0000 mon.a (mon.0) 728 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[21512]: audit 2024-03-11T21:23:27.729375+0000 mon.a (mon.0) 729 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[21512]: audit 2024-03-11T21:23:27.987292+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.203:0/250111458' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[29284]: cephadm 2024-03-11T21:23:26.474205+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[29284]: cluster 2024-03-11T21:23:27.509429+0000 mon.a (mon.0) 728 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[29284]: audit 2024-03-11T21:23:27.729375+0000 mon.a (mon.0) 729 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:28.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:28 smithi143 bash[29284]: audit 2024-03-11T21:23:27.987292+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.203:0/250111458' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:23:28.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[23654]: cephadm 2024-03-11T21:23:26.474205+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:23:28.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[23654]: cluster 2024-03-11T21:23:27.509429+0000 mon.a (mon.0) 728 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T21:23:28.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[23654]: audit 2024-03-11T21:23:27.729375+0000 mon.a (mon.0) 729 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:28.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:28 smithi203 bash[23654]: audit 2024-03-11T21:23:27.987292+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.203:0/250111458' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:23:29.467 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:29.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:29 smithi143 bash[21512]: cluster 2024-03-11T21:23:27.557362+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:29.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:29 smithi143 bash[21512]: cluster 2024-03-11T21:23:28.522680+0000 mon.a (mon.0) 730 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:23:29.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:29 smithi143 bash[29284]: cluster 2024-03-11T21:23:27.557362+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:29.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:29 smithi143 bash[29284]: cluster 2024-03-11T21:23:28.522680+0000 mon.a (mon.0) 730 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:23:29.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:29 smithi203 bash[23654]: cluster 2024-03-11T21:23:27.557362+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:29.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:29 smithi203 bash[23654]: cluster 2024-03-11T21:23:28.522680+0000 mon.a (mon.0) 730 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:23:31.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:31 smithi143 bash[21512]: cluster 2024-03-11T21:23:29.558251+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 169 B/s rd, 169 B/s wr, 1 op/s 2024-03-11T21:23:31.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:31 smithi143 bash[29284]: cluster 2024-03-11T21:23:29.558251+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 169 B/s rd, 169 B/s wr, 1 op/s 2024-03-11T21:23:31.742 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:31 smithi203 bash[23654]: cluster 2024-03-11T21:23:29.558251+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 169 B/s rd, 169 B/s wr, 1 op/s 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- 172.21.15.203:0/80916183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a801039b0 msgr2=0x7f7a8010a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/80916183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a801039b0 0x7f7a8010a2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a70009990 tx=0x7f7a70009ca0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- 172.21.15.203:0/80916183 shutdown_connections 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/80916183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a801039b0 0x7f7a8010a2d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/80916183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 0x7f7a801022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/80916183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a80102ff0 0x7f7a80103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- 172.21.15.203:0/80916183 >> 172.21.15.203:0/80916183 conn(0x7f7a800fd430 msgr2=0x7f7a800ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- 172.21.15.203:0/80916183 shutdown_connections 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- 172.21.15.203:0/80916183 wait complete. 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 librados: starting msgr at 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 librados: starting objecter 2024-03-11T21:23:31.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 Processor -- start 2024-03-11T21:23:31.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- start start 2024-03-11T21:23:31.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 librados: setting wanted keys 2024-03-11T21:23:31.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 librados: calling monclient init 2024-03-11T21:23:31.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 0x7f7a801976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:31.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:31.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a801039b0 0x7f7a8019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8010cfa0 con 0x7f7a801039b0 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8010ce20 con 0x7f7a80102ff0 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a84eaf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8010d120 con 0x7f7a80101e90 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45794/0 (socket says 172.21.15.203:45794) 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 -- 172.21.15.203:0/1655499337 learned_addr learned my addr 172.21.15.203:0/1655499337 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7ed9d700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a801039b0 0x7f7a8019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7e59c700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 0x7f7a801976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 -- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 msgr2=0x7f7a801976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 0x7f7a801976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 -- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a801039b0 msgr2=0x7f7a8019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:31.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a801039b0 0x7f7a8019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.899+0000 7f7a7dd9b700 1 -- 172.21.15.203:0/1655499337 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8019f310 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a7dd9b700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c920 tx=0x7f7a6800cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 10 client.?.objecter ms_handle_connect 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68007b00 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7a68004cb0 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68005550 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8019f600 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a84eaf700 10 client.24412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8019fb70 con 0x7f7a80102ff0 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a84eaf700 1 librados: init done 2024-03-11T21:23:31.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8004faa0 con 0x7f7a80102ff0 2024-03-11T21:23:31.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f7a68004750 con 0x7f7a80102ff0 2024-03-11T21:23:31.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 0x7f7a6c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:31.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 21..91) v4 ==== 7120+0+0 (secure 0 0 0) 0x7f7a6808dfb0 con 0x7f7a80102ff0 2024-03-11T21:23:31.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 10 client.24412.objecter ms_dispatch 0x7f7a80001550 osd_map(91..91 src has 21..91) v4 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 3 client.24412.objecter handle_osd_map got epochs [91,91] > 0 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 3 client.24412.objecter handle_osd_map decoding full epoch 91 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a777fe700 20 client.24412.objecter dump_active .. 0 homeless 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.903+0000 7f7a7e59c700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 0x7f7a6c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.907+0000 7f7a777fe700 10 client.24412.objecter ms_handle_connect 0x7f7a6c06f7b0 2024-03-11T21:23:31.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.907+0000 7f7a7e59c700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 0x7f7a6c071c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009f20 tx=0x7f7a780094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:31.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:31.915+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7a6801d020 con 0x7f7a80102ff0 2024-03-11T21:23:32.111 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.127+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a80062220 con 0x7f7a6c06f7b0 2024-03-11T21:23:32.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.143+0000 7f7a777fe700 1 -- 172.21.15.203:0/1655499337 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f7a80062220 con 0x7f7a6c06f7b0 2024-03-11T21:23:32.128 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled prometheus update... 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 10 librados: watch_flush enter 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 10 librados: watch_flush exit 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 20 client.24412.objecter shutdown clearing up homeless session... 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 10 client.24412.objecter successfully canceled tick 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 msgr2=0x7f7a6c071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 0x7f7a6c071c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009f20 tx=0x7f7a780094f0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 msgr2=0x7f7a80197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c920 tx=0x7f7a6800cce0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 shutdown_connections 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f7a6c06f7b0 0x7f7a6c071c70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a801039b0 0x7f7a8019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a80101e90 0x7f7a801976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 --2- 172.21.15.203:0/1655499337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a80102ff0 0x7f7a80197bf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 >> 172.21.15.203:0/1655499337 conn(0x7f7a800fd430 msgr2=0x7f7a800fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 shutdown_connections 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 -- 172.21.15.203:0/1655499337 wait complete. 2024-03-11T21:23:32.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:32.147+0000 7f7a84eaf700 1 librados: shutdown 2024-03-11T21:23:32.757 DEBUG:teuthology.orchestra.run.smithi203:prometheus.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:23:32.760 INFO:tasks.cephadm:Adding node-exporter.a on smithi143 2024-03-11T21:23:32.760 INFO:tasks.cephadm:Adding node-exporter.b on smithi203 2024-03-11T21:23:32.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi143=a;smithi203=b' 2024-03-11T21:23:33.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:32 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: cluster 2024-03-11T21:23:31.559327+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.099098+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.110039+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.112514+0000 mon.a (mon.0) 733 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.114736+0000 mon.a (mon.0) 734 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.126124+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.138811+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:33.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: cephadm 2024-03-11T21:23:32.142386+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi203=a;count:1 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.151128+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.210653+0000 mon.a (mon.0) 737 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.212032+0000 mgr.y (mgr.14152) 365 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: cephadm 2024-03-11T21:23:32.214406+0000 mgr.y (mgr.14152) 366 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.203:5000 to Dashboard 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.214937+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.215576+0000 mgr.y (mgr.14152) 367 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.224527+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.238177+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.239019+0000 mgr.y (mgr.14152) 368 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.248646+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.253521+0000 mon.a (mon.0) 742 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.257347+0000 mon.a (mon.0) 743 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.259311+0000 mon.a (mon.0) 744 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:33 smithi203 bash[23654]: audit 2024-03-11T21:23:32.271255+0000 mon.a (mon.0) 745 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: cluster 2024-03-11T21:23:31.559327+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.099098+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.110039+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.112514+0000 mon.a (mon.0) 733 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.114736+0000 mon.a (mon.0) 734 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.126124+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.138811+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: cephadm 2024-03-11T21:23:32.142386+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi203=a;count:1 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.151128+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.210653+0000 mon.a (mon.0) 737 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.212032+0000 mgr.y (mgr.14152) 365 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: cephadm 2024-03-11T21:23:32.214406+0000 mgr.y (mgr.14152) 366 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.203:5000 to Dashboard 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.214937+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.215576+0000 mgr.y (mgr.14152) 367 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.224527+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.238177+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.239019+0000 mgr.y (mgr.14152) 368 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.248646+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.253521+0000 mon.a (mon.0) 742 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.257347+0000 mon.a (mon.0) 743 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.259311+0000 mon.a (mon.0) 744 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[21512]: audit 2024-03-11T21:23:32.271255+0000 mon.a (mon.0) 745 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: cluster 2024-03-11T21:23:31.559327+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.099098+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.110039+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.112514+0000 mon.a (mon.0) 733 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.114736+0000 mon.a (mon.0) 734 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.126124+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.138811+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: cephadm 2024-03-11T21:23:32.142386+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi203=a;count:1 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.151128+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.210653+0000 mon.a (mon.0) 737 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.212032+0000 mgr.y (mgr.14152) 365 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: cephadm 2024-03-11T21:23:32.214406+0000 mgr.y (mgr.14152) 366 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.203:5000 to Dashboard 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.214937+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.215576+0000 mgr.y (mgr.14152) 367 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.224527+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.238177+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.239019+0000 mgr.y (mgr.14152) 368 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi203"}]: dispatch 2024-03-11T21:23:33.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.248646+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:33.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.253521+0000 mon.a (mon.0) 742 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:33.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.257347+0000 mon.a (mon.0) 743 : audit 0 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:33.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.259311+0000 mon.a (mon.0) 744 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:33.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:33 smithi143 bash[29284]: audit 2024-03-11T21:23:32.271255+0000 mon.a (mon.0) 745 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:34.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:34 smithi143 bash[29284]: cephadm 2024-03-11T21:23:32.548396+0000 mgr.y (mgr.14152) 369 : cephadm 1 Deploying daemon prometheus.a on smithi203 2024-03-11T21:23:34.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:34 smithi143 bash[29284]: cluster 2024-03-11T21:23:33.526771+0000 mon.a (mon.0) 746 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:23:34.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:34 smithi143 bash[21512]: cephadm 2024-03-11T21:23:32.548396+0000 mgr.y (mgr.14152) 369 : cephadm 1 Deploying daemon prometheus.a on smithi203 2024-03-11T21:23:34.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:34 smithi143 bash[21512]: cluster 2024-03-11T21:23:33.526771+0000 mon.a (mon.0) 746 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:23:34.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:34 smithi203 bash[23654]: cephadm 2024-03-11T21:23:32.548396+0000 mgr.y (mgr.14152) 369 : cephadm 1 Deploying daemon prometheus.a on smithi203 2024-03-11T21:23:34.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:34 smithi203 bash[23654]: cluster 2024-03-11T21:23:33.526771+0000 mon.a (mon.0) 746 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:23:35.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:35 smithi143 bash[21512]: cluster 2024-03-11T21:23:33.560051+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 161 B/s wr, 2 op/s 2024-03-11T21:23:35.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:35 smithi143 bash[29284]: cluster 2024-03-11T21:23:33.560051+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 161 B/s wr, 2 op/s 2024-03-11T21:23:35.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:35 smithi203 bash[23654]: cluster 2024-03-11T21:23:33.560051+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 161 B/s wr, 2 op/s 2024-03-11T21:23:37.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[21512]: cluster 2024-03-11T21:23:35.561027+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:37.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[21512]: cluster 2024-03-11T21:23:36.147451+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:37.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[21512]: cluster 2024-03-11T21:23:36.147566+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:37.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[29284]: cluster 2024-03-11T21:23:35.561027+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:37.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[29284]: cluster 2024-03-11T21:23:36.147451+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:37.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:37 smithi143 bash[29284]: cluster 2024-03-11T21:23:36.147566+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:37.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:37 smithi203 bash[23654]: cluster 2024-03-11T21:23:35.561027+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T21:23:37.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:37 smithi203 bash[23654]: cluster 2024-03-11T21:23:36.147451+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:37.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:37 smithi203 bash[23654]: cluster 2024-03-11T21:23:36.147566+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:37.564 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:38.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:37 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:23:38.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:38 smithi143 bash[21512]: cluster 2024-03-11T21:23:37.119467+0000 mon.a (mon.0) 747 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:23:38.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:38 smithi143 bash[29284]: cluster 2024-03-11T21:23:37.119467+0000 mon.a (mon.0) 747 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:23:38.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:38 smithi203 bash[23654]: cluster 2024-03-11T21:23:37.119467+0000 mon.a (mon.0) 747 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[21512]: cluster 2024-03-11T21:23:37.209460+0000 osd.7 (osd.7) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[21512]: cluster 2024-03-11T21:23:37.212448+0000 osd.7 (osd.7) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[21512]: cluster 2024-03-11T21:23:37.561822+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[21512]: audit 2024-03-11T21:23:37.681648+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[21512]: cluster 2024-03-11T21:23:38.528716+0000 mon.a (mon.0) 748 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[29284]: cluster 2024-03-11T21:23:37.209460+0000 osd.7 (osd.7) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[29284]: cluster 2024-03-11T21:23:37.212448+0000 osd.7 (osd.7) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[29284]: cluster 2024-03-11T21:23:37.561822+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:39.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[29284]: audit 2024-03-11T21:23:37.681648+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:39.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:39 smithi143 bash[29284]: cluster 2024-03-11T21:23:38.528716+0000 mon.a (mon.0) 748 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:23:39.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:39 smithi203 bash[23654]: cluster 2024-03-11T21:23:37.209460+0000 osd.7 (osd.7) 5 : cluster 0 1.0 scrub starts 2024-03-11T21:23:39.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:39 smithi203 bash[23654]: cluster 2024-03-11T21:23:37.212448+0000 osd.7 (osd.7) 6 : cluster 0 1.0 scrub ok 2024-03-11T21:23:39.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:39 smithi203 bash[23654]: cluster 2024-03-11T21:23:37.561822+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:39.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:39 smithi203 bash[23654]: audit 2024-03-11T21:23:37.681648+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:39.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:39 smithi203 bash[23654]: cluster 2024-03-11T21:23:38.528716+0000 mon.a (mon.0) 748 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:23:41.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:41 smithi143 bash[21512]: cluster 2024-03-11T21:23:39.562698+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:23:41.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:41 smithi143 bash[29284]: cluster 2024-03-11T21:23:39.562698+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:23:41.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:41 smithi203 bash[23654]: cluster 2024-03-11T21:23:39.562698+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/1685685438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105c30 msgr2=0x7f19c410c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- 172.21.15.203:0/1685685438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105c30 0x7f19c410c4e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00b0d0 tx=0x7f19bc00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/1685685438 shutdown_connections 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- 172.21.15.203:0/1685685438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105c30 0x7f19c410c4e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- 172.21.15.203:0/1685685438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 0x7f19c41044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- 172.21.15.203:0/1685685438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105290 0x7f19c41056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/1685685438 >> 172.21.15.203:0/1685685438 conn(0x7f19c40ff670 msgr2=0x7f19c4101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/1685685438 shutdown_connections 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/1685685438 wait complete. 2024-03-11T21:23:43.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 librados: starting msgr at 2024-03-11T21:23:43.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 librados: starting objecter 2024-03-11T21:23:43.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 Processor -- start 2024-03-11T21:23:43.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- start start 2024-03-11T21:23:43.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 librados: setting wanted keys 2024-03-11T21:23:43.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 librados: calling monclient init 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 0x7f19c4199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 0x7f19c41a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410f260 con 0x7f19c4105c30 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410f0e0 con 0x7f19c4105290 2024-03-11T21:23:43.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410f3e0 con 0x7f19c4104090 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:60008/0 (socket says 172.21.15.203:60008) 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 -- 172.21.15.203:0/4173446535 learned_addr learned my addr 172.21.15.203:0/4173446535 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c8fc3700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 0x7f19c41a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c3fff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 0x7f19c4199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 -- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 msgr2=0x7f19c4199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 0x7f19c4199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 -- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 msgr2=0x7f19c41a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 0x7f19c41a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 -- 172.21.15.203:0/4173446535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c41a1480 con 0x7f19c4105290 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c8fc3700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 0x7f19c41a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19bbfff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d950 tx=0x7f19ac00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c1ffb700 10 client.?.objecter ms_handle_connect 0x7f19c4105290 2024-03-11T21:23:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c1ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac016070 con 0x7f19c4105290 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f19ac010460 con 0x7f19c4105290 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c41a1770 con 0x7f19c4105290 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00f620 con 0x7f19c4105290 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 10 client.24418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c41a1d00 con 0x7f19c4105290 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 librados: init done 2024-03-11T21:23:43.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.398+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c404faa0 con 0x7f19c4105290 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f19ac019070 con 0x7f19c4105290 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 0x7f19b0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(94..94 src has 21..94) v4 ==== 7120+0+0 (secure 0 0 0) 0x7f19ac08dbe0 con 0x7f19c4105290 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 10 client.24418.objecter ms_dispatch 0x7f19c4001550 osd_map(94..94 src has 21..94) v4 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 3 client.24418.objecter handle_osd_map got epochs [94,94] > 0 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 3 client.24418.objecter handle_osd_map decoding full epoch 94 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 20 client.24418.objecter dump_active .. 0 homeless 2024-03-11T21:23:43.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c3fff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 0x7f19b0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:43.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c1ffb700 10 client.24418.objecter ms_handle_connect 0x7f19b006f7b0 2024-03-11T21:23:43.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.402+0000 7f19c3fff700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 0x7f19b0071c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19b4005950 tx=0x7f19b40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:43.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.406+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f19ac093020 con 0x7f19c4105290 2024-03-11T21:23:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:43 smithi143 bash[21512]: cluster 2024-03-11T21:23:41.563738+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:43.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:43 smithi143 bash[29284]: cluster 2024-03-11T21:23:41.563738+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:43.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:43 smithi203 bash[23654]: cluster 2024-03-11T21:23:41.563738+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:23:43.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.630+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi143=a;smithi203=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f19c4062220 con 0x7f19b006f7b0 2024-03-11T21:23:43.631 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled node-exporter update... 2024-03-11T21:23:43.631 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.646+0000 7f19c1ffb700 1 -- 172.21.15.203:0/4173446535 <== mgr.14152 v2:172.21.15.143:6800/3638739149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f19c4062220 con 0x7f19b006f7b0 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 10 librados: watch_flush enter 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 10 librados: watch_flush exit 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 20 client.24418.objecter shutdown clearing up homeless session... 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 10 client.24418.objecter successfully canceled tick 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 msgr2=0x7f19b0071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 0x7f19b0071c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19b4005950 tx=0x7f19b40058e0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.633 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 msgr2=0x7f19c4199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d950 tx=0x7f19ac00dd10 comp rx=0 tx=0).stop 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 shutdown_connections 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f19b006f7b0 0x7f19b0071c70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c4105c30 0x7f19c41a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4104090 0x7f19c4199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 --2- 172.21.15.203:0/4173446535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c4105290 0x7f19c4199d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 >> 172.21.15.203:0/4173446535 conn(0x7f19c40ff670 msgr2=0x7f19c4108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 shutdown_connections 2024-03-11T21:23:43.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 -- 172.21.15.203:0/4173446535 wait complete. 2024-03-11T21:23:43.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:43.650+0000 7f19caa26700 1 librados: shutdown 2024-03-11T21:23:44.735 DEBUG:teuthology.orchestra.run.smithi143:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:23:44.738 DEBUG:teuthology.orchestra.run.smithi203:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:23:44.740 INFO:tasks.cephadm:Adding alertmanager.a on smithi143 2024-03-11T21:23:44.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi143=a' 2024-03-11T21:23:44.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:44 smithi203 bash[23654]: cluster 2024-03-11T21:23:43.564873+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:44 smithi203 bash[23654]: audit 2024-03-11T21:23:43.641676+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi143=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:44.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:44 smithi203 bash[23654]: cephadm 2024-03-11T21:23:43.645392+0000 mgr.y (mgr.14152) 378 : cephadm 1 Saving service node-exporter spec with placement smithi143=a;smithi203=b;count:2 2024-03-11T21:23:44.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:44 smithi203 bash[23654]: audit 2024-03-11T21:23:43.654969+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:44.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[21512]: cluster 2024-03-11T21:23:43.564873+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[21512]: audit 2024-03-11T21:23:43.641676+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi143=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[21512]: cephadm 2024-03-11T21:23:43.645392+0000 mgr.y (mgr.14152) 378 : cephadm 1 Saving service node-exporter spec with placement smithi143=a;smithi203=b;count:2 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[21512]: audit 2024-03-11T21:23:43.654969+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[29284]: cluster 2024-03-11T21:23:43.564873+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[29284]: audit 2024-03-11T21:23:43.641676+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi143=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[29284]: cephadm 2024-03-11T21:23:43.645392+0000 mgr.y (mgr.14152) 378 : cephadm 1 Saving service node-exporter spec with placement smithi143=a;smithi203=b;count:2 2024-03-11T21:23:44.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:44 smithi143 bash[29284]: audit 2024-03-11T21:23:43.654969+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:46 smithi203 bash[23654]: cluster 2024-03-11T21:23:45.565972+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:46 smithi143 bash[21512]: cluster 2024-03-11T21:23:45.565972+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:46.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:46 smithi143 bash[29284]: cluster 2024-03-11T21:23:45.565972+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:47.494 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.494 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.495 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.495 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.496 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.755 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.755 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.756 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:47 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:23:47.756 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.757 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:47 smithi203 bash[23654]: cluster 2024-03-11T21:23:46.580777+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:47.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:47 smithi203 bash[23654]: cluster 2024-03-11T21:23:46.580913+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:47.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:47 smithi203 bash[23654]: cluster 2024-03-11T21:23:46.671073+0000 mon.a (mon.0) 750 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:23:47.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.757 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.757 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.758 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.758 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:23:47.758 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:47 smithi203 systemd[1]: Started Ceph prometheus.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:23:47.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[21512]: cluster 2024-03-11T21:23:46.580777+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:47.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[21512]: cluster 2024-03-11T21:23:46.580913+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:47.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[21512]: cluster 2024-03-11T21:23:46.671073+0000 mon.a (mon.0) 750 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:23:47.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[29284]: cluster 2024-03-11T21:23:46.580777+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T21:23:47.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[29284]: cluster 2024-03-11T21:23:46.580913+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T21:23:47.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:47 smithi143 bash[29284]: cluster 2024-03-11T21:23:46.671073+0000 mon.a (mon.0) 750 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:23:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: cluster 2024-03-11T21:23:47.567049+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:23:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: audit 2024-03-11T21:23:47.689698+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: audit 2024-03-11T21:23:47.825736+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: audit 2024-03-11T21:23:47.836551+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: audit 2024-03-11T21:23:47.874400+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: audit 2024-03-11T21:23:47.883357+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[23654]: cluster 2024-03-11T21:23:48.535686+0000 mon.a (mon.0) 755 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[24601]: ignoring --setuser ceph since I am not root 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[24601]: ignoring --setgroup ceph since I am not root 2024-03-11T21:23:49.024 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[24601]: debug 2024-03-11T21:23:48.938+0000 7f069cebd700 1 -- 172.21.15.203:0/588539864 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562fdd0785a0 con 0x562fdddf8400 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.782Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.782Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.782Z caller=main.go:570 level=info host_details="(Linux 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 smithi203 (none))" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.782Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.782Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.784Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.784Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.786Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.786Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.788Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.788Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.448µs 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.788Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:23:49.025 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.788Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.788Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.176µs wal_replay_duration=219.823µs wbl_replay_duration=189ns total_replay_duration=295.326µs 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.789Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.789Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.789Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.814Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.288736ms db_storage=757ns remote_storage=1.165µs web_handler=338ns query_engine=530ns scrape=248.159µs scrape_sd=103.774µs notify=995ns notify_sd=2.201µs rules=23.606174ms tracing=6.955µs 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.814Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.814Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.910Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=ceph-exporter\": EOF" 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.910Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=mgr-prometheus\": read tcp 172.21.15.203:36742->172.21.15.143:8765: read: connection reset by peer" 2024-03-11T21:23:49.026 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:23:48 smithi203 bash[49511]: ts=2024-03-11T21:23:48.911Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=nvmeof\": read tcp 172.21.15.203:36738->172.21.15.143:8765: read: connection reset by peer" 2024-03-11T21:23:49.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: cluster 2024-03-11T21:23:47.567049+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:23:49.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: audit 2024-03-11T21:23:47.689698+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:49.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: audit 2024-03-11T21:23:47.825736+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: audit 2024-03-11T21:23:47.836551+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: audit 2024-03-11T21:23:47.874400+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: audit 2024-03-11T21:23:47.883357+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21512]: cluster 2024-03-11T21:23:48.535686+0000 mon.a (mon.0) 755 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21768]: ignoring --setuser ceph since I am not root 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[21768]: ignoring --setgroup ceph since I am not root 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:49 smithi143 bash[21768]: debug 2024-03-11T21:23:49.083+0000 7f3bda5e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:23:49.063 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: cluster 2024-03-11T21:23:47.567049+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: audit 2024-03-11T21:23:47.689698+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: audit 2024-03-11T21:23:47.825736+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: audit 2024-03-11T21:23:47.836551+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: audit 2024-03-11T21:23:47.874400+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: audit 2024-03-11T21:23:47.883357+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:23:49.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:48 smithi143 bash[29284]: cluster 2024-03-11T21:23:48.535686+0000 mon.a (mon.0) 755 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:23:49.523 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:49 smithi203 bash[24601]: debug 2024-03-11T21:23:49.086+0000 7f06a6ca1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:23:50.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:49 smithi143 bash[21512]: audit 2024-03-11T21:23:48.874300+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:23:50.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:49 smithi143 bash[21512]: cluster 2024-03-11T21:23:48.885426+0000 mon.a (mon.0) 757 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-03-11T21:23:50.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:49 smithi143 bash[29284]: audit 2024-03-11T21:23:48.874300+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:23:50.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:49 smithi143 bash[29284]: cluster 2024-03-11T21:23:48.885426+0000 mon.a (mon.0) 757 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-03-11T21:23:50.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:49 smithi203 bash[23654]: audit 2024-03-11T21:23:48.874300+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.143:0/2147494222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:23:50.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:49 smithi203 bash[23654]: cluster 2024-03-11T21:23:48.885426+0000 mon.a (mon.0) 757 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-03-11T21:23:50.556 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:50.937 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:50 smithi143 bash[21768]: debug 2024-03-11T21:23:50.651+0000 7f3bda5e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:23:50.937 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:50 smithi143 bash[21768]: debug 2024-03-11T21:23:50.747+0000 7f3bda5e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:23:51.015 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:50 smithi203 bash[24601]: debug 2024-03-11T21:23:50.682+0000 7f06a6ca1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:23:51.015 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:50 smithi203 bash[24601]: debug 2024-03-11T21:23:50.778+0000 7f06a6ca1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:23:51.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:51 smithi203 bash[24601]: debug 2024-03-11T21:23:51.030+0000 7f06a6ca1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:23:51.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:51 smithi143 bash[21768]: debug 2024-03-11T21:23:51.003+0000 7f3bda5e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:23:52.023 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:51 smithi203 bash[24601]: debug 2024-03-11T21:23:51.838+0000 7f06a6ca1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:23:52.023 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:51 smithi203 bash[24601]: debug 2024-03-11T21:23:51.950+0000 7f06a6ca1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:23:52.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:51 smithi143 bash[21768]: debug 2024-03-11T21:23:51.795+0000 7f3bda5e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:23:52.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:51 smithi143 bash[21768]: debug 2024-03-11T21:23:51.907+0000 7f3bda5e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:23:52.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:52 smithi143 bash[21768]: debug 2024-03-11T21:23:52.015+0000 7f3bda5e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:23:52.400 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:52 smithi203 bash[24601]: debug 2024-03-11T21:23:52.058+0000 7f06a6ca1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:23:52.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:52 smithi143 bash[21768]: debug 2024-03-11T21:23:52.371+0000 7f3bda5e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:23:52.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:52 smithi143 bash[21768]: debug 2024-03-11T21:23:52.483+0000 7f3bda5e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:23:52.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:52 smithi143 bash[21768]: debug 2024-03-11T21:23:52.591+0000 7f3bda5e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:23:52.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:52 smithi203 bash[24601]: debug 2024-03-11T21:23:52.418+0000 7f06a6ca1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:23:52.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:52 smithi203 bash[24601]: debug 2024-03-11T21:23:52.530+0000 7f06a6ca1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:23:52.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:52 smithi203 bash[24601]: debug 2024-03-11T21:23:52.634+0000 7f06a6ca1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/3474326570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 msgr2=0x7f9cb41023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- 172.21.15.203:0/3474326570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb41023c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80099c0 tx=0x7f9ca8009cd0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/3474326570 shutdown_connections 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- 172.21.15.203:0/3474326570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 0x7f9cb4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- 172.21.15.203:0/3474326570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4102900 0x7f9cb41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- 172.21.15.203:0/3474326570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb41023c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/3474326570 >> 172.21.15.203:0/3474326570 conn(0x7f9cb40fc300 msgr2=0x7f9cb40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/3474326570 shutdown_connections 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/3474326570 wait complete. 2024-03-11T21:23:53.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 librados: starting msgr at 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 librados: starting objecter 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 Processor -- start 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- start start 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 librados: setting wanted keys 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 librados: calling monclient init 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 0x7f9cb4104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb4103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4102900 0x7f9cb4103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb410be30 con 0x7f9cb4100d60 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb410bcb0 con 0x7f9cb4101f60 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb410bfb0 con 0x7f9cb4102900 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb4103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:53.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb4103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:54506/0 (socket says 172.21.15.203:54506) 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 -- 172.21.15.203:0/1625946111 learned_addr learned my addr 172.21.15.203:0/1625946111 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 0x7f9cb4104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb37fe700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4102900 0x7f9cb4103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4102900 msgr2=0x7f9cb4103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4102900 0x7f9cb4103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 msgr2=0x7f9cb4104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 0x7f9cb4104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:53.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4104130 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb4100d60 0x7f9cb4104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb27fc700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb4103340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8000c00 tx=0x7f9ca8005ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9c9bfff700 10 client.?.objecter ms_handle_connect 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9c9bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8027070 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb41a17f0 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9ca8004180 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca800f460 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 10 client.24433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb41a1bf0 con 0x7f9cb4101f60 2024-03-11T21:23:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 librados: init done 2024-03-11T21:23:53.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.102+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb404faa0 con 0x7f9cb4101f60 2024-03-11T21:23:53.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 91820+0+0 (secure 0 0 0) 0x7f9ca802a070 con 0x7f9cb4101f60 2024-03-11T21:23:53.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9cb2ffd700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3638739149 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(96..96 src has 21..96) v4 ==== 7120+0+0 (secure 0 0 0) 0x7f9ca808f090 con 0x7f9cb4101f60 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 10 client.24433.objecter ms_dispatch 0x7f9cb4001550 osd_map(96..96 src has 21..96) v4 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 3 client.24433.objecter handle_osd_map got epochs [96,96] > 0 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 3 client.24433.objecter handle_osd_map decoding full epoch 96 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T21:23:53.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.106+0000 7f9c9bfff700 20 client.24433.objecter dump_active .. 0 homeless 2024-03-11T21:23:53.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.110+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9ca8013070 con 0x7f9cb4101f60 2024-03-11T21:23:53.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:52 smithi143 bash[21768]: debug 2024-03-11T21:23:52.947+0000 7f3bda5e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:23:53.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:53 smithi143 bash[21768]: debug 2024-03-11T21:23:53.195+0000 7f3bda5e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:23:53.236 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:53 smithi203 bash[24601]: debug 2024-03-11T21:23:53.002+0000 7f06a6ca1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:23:53.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.310+0000 7f9cb2ffd700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3638739149 2024-03-11T21:23:53.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.310+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T21:23:53.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.334+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi143=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cb4062220 con 0x7f9c9c06f800 2024-03-11T21:23:53.498 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:53 smithi203 bash[24601]: debug 2024-03-11T21:23:53.254+0000 7f06a6ca1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:23:53.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:53 smithi143 bash[21768]: debug 2024-03-11T21:23:53.459+0000 7f3bda5e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:23:53.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:53 smithi143 bash[21768]: debug 2024-03-11T21:23:53.571+0000 7f3bda5e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:23:53.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.714+0000 7f9cb2ffd700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3638739149 2024-03-11T21:23:53.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:53.714+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T21:23:53.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:53 smithi203 bash[24601]: debug 2024-03-11T21:23:53.514+0000 7f06a6ca1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:23:53.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:53 smithi203 bash[24601]: debug 2024-03-11T21:23:53.626+0000 7f06a6ca1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:23:54.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:53 smithi143 bash[21768]: debug 2024-03-11T21:23:53.847+0000 7f3bda5e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:23:54.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:54 smithi143 bash[21768]: debug 2024-03-11T21:23:54.091+0000 7f3bda5e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:23:54.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:53 smithi203 bash[24601]: debug 2024-03-11T21:23:53.898+0000 7f06a6ca1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:23:54.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:54 smithi203 bash[24601]: debug 2024-03-11T21:23:54.142+0000 7f06a6ca1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:23:54.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:54.518+0000 7f9cb2ffd700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3638739149 2024-03-11T21:23:54.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:54.518+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T21:23:54.852 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:54 smithi143 bash[21768]: debug 2024-03-11T21:23:54.503+0000 7f3bda5e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:23:54.852 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:54 smithi143 bash[21768]: debug 2024-03-11T21:23:54.623+0000 7f3bda5e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:23:54.898 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:54 smithi203 bash[24601]: debug 2024-03-11T21:23:54.550+0000 7f06a6ca1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:23:54.898 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:54 smithi203 bash[24601]: debug 2024-03-11T21:23:54.666+0000 7f06a6ca1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:23:55.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:54 smithi143 bash[21768]: debug 2024-03-11T21:23:54.875+0000 7f3bda5e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:23:55.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:54 smithi203 bash[24601]: debug 2024-03-11T21:23:54.914+0000 7f06a6ca1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:23:56.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:56.122+0000 7f9cb2ffd700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/3638739149 2024-03-11T21:23:56.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:56.122+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T21:23:56.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:55 smithi143 bash[21768]: debug 2024-03-11T21:23:55.895+0000 7f3bda5e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:23:56.187 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:56 smithi143 bash[21768]: debug 2024-03-11T21:23:56.011+0000 7f3bda5e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:23:56.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:55 smithi203 bash[24601]: debug 2024-03-11T21:23:55.922+0000 7f06a6ca1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:23:56.273 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:56 smithi203 bash[24601]: debug 2024-03-11T21:23:56.034+0000 7f06a6ca1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:23:56.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:56 smithi143 bash[21768]: debug 2024-03-11T21:23:56.307+0000 7f3bda5e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:23:56.687 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:56 smithi143 bash[21768]: debug 2024-03-11T21:23:56.423+0000 7f3bda5e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:23:56.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:56 smithi203 bash[24601]: debug 2024-03-11T21:23:56.326+0000 7f06a6ca1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:23:56.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:56 smithi203 bash[24601]: debug 2024-03-11T21:23:56.442+0000 7f06a6ca1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:23:57.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:57.026+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mgrmap(e 18) v1 ==== 91436+0+0 (secure 0 0 0) 0x7f9ca8022790 con 0x7f9cb4101f60 2024-03-11T21:23:57.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:57.026+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 msgr2=0x7f9c9c071cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T21:23:57.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:57.026+0000 7f9c9bfff700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/3638739149,v1:172.21.15.143:6801/3638739149] conn(0x7f9c9c06f800 0x7f9c9c071cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:23:57.127 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:56 smithi143 bash[21768]: debug 2024-03-11T21:23:56.875+0000 7f3bda5e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:23:57.127 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: debug 2024-03-11T21:23:56.999+0000 7f3bda5e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:23:57.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.022147+0000 mon.a (mon.0) 758 : cluster 1 Active manager daemon y restarted 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.022756+0000 mon.a (mon.0) 759 : cluster 1 Activating manager daemon y 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.037442+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.038548+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.040892+0000 mon.a (mon.0) 760 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.041866+0000 mon.a (mon.0) 761 : cluster 0 mgrmap e18: y(active, starting, since 0.0192926s), standbys: x 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.042201+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.042370+0000 mon.a (mon.0) 762 : cluster 0 Standby manager daemon x restarted 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.042466+0000 mon.a (mon.0) 763 : cluster 0 Standby manager daemon x started 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.042891+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.046348+0000 mon.c (mon.2) 15 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.046523+0000 mon.c (mon.2) 16 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.046692+0000 mon.c (mon.2) 17 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:57.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.048245+0000 mon.c (mon.2) 18 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.048549+0000 mon.c (mon.2) 19 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.048859+0000 mon.c (mon.2) 20 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.049177+0000 mon.c (mon.2) 21 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.049477+0000 mon.c (mon.2) 22 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.049909+0000 mon.c (mon.2) 23 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.050224+0000 mon.c (mon.2) 24 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.050587+0000 mon.c (mon.2) 25 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.050905+0000 mon.c (mon.2) 26 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.051230+0000 mon.c (mon.2) 27 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.051537+0000 mon.c (mon.2) 28 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.051812+0000 mon.c (mon.2) 29 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:23:57.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21512]: audit 2024-03-11T21:23:57.052510+0000 mon.c (mon.2) 30 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.022147+0000 mon.a (mon.0) 758 : cluster 1 Active manager daemon y restarted 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.022756+0000 mon.a (mon.0) 759 : cluster 1 Activating manager daemon y 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.037442+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.038548+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.040892+0000 mon.a (mon.0) 760 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.041866+0000 mon.a (mon.0) 761 : cluster 0 mgrmap e18: y(active, starting, since 0.0192926s), standbys: x 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.042201+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.042370+0000 mon.a (mon.0) 762 : cluster 0 Standby manager daemon x restarted 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.042466+0000 mon.a (mon.0) 763 : cluster 0 Standby manager daemon x started 2024-03-11T21:23:57.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.042891+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.046348+0000 mon.c (mon.2) 15 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.046523+0000 mon.c (mon.2) 16 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.046692+0000 mon.c (mon.2) 17 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.048245+0000 mon.c (mon.2) 18 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.048549+0000 mon.c (mon.2) 19 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.048859+0000 mon.c (mon.2) 20 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.049177+0000 mon.c (mon.2) 21 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.049477+0000 mon.c (mon.2) 22 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.049909+0000 mon.c (mon.2) 23 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.050224+0000 mon.c (mon.2) 24 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.050587+0000 mon.c (mon.2) 25 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.050905+0000 mon.c (mon.2) 26 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.051230+0000 mon.c (mon.2) 27 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:23:57.131 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.051537+0000 mon.c (mon.2) 28 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:23:57.132 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.051812+0000 mon.c (mon.2) 29 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:23:57.132 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[29284]: audit 2024-03-11T21:23:57.052510+0000 mon.c (mon.2) 30 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:23:57.172 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:56 smithi203 bash[24601]: debug 2024-03-11T21:23:56.890+0000 7f06a6ca1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:23:57.172 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: debug 2024-03-11T21:23:57.010+0000 7f06a6ca1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:23:57.173 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: [11/Mar/2024:21:23:57] ENGINE Bus STARTING 2024-03-11T21:23:57.173 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: CherryPy Checker: 2024-03-11T21:23:57.173 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: The Application mounted at '' has an empty config. 2024-03-11T21:23:57.173 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: [11/Mar/2024:21:23:57] ENGINE Serving on http://:::9283 2024-03-11T21:23:57.173 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[24601]: [11/Mar/2024:21:23:57] ENGINE Bus STARTED 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.022147+0000 mon.a (mon.0) 758 : cluster 1 Active manager daemon y restarted 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.022756+0000 mon.a (mon.0) 759 : cluster 1 Activating manager daemon y 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.037442+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.038548+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.040892+0000 mon.a (mon.0) 760 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.041866+0000 mon.a (mon.0) 761 : cluster 0 mgrmap e18: y(active, starting, since 0.0192926s), standbys: x 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.042201+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.042370+0000 mon.a (mon.0) 762 : cluster 0 Standby manager daemon x restarted 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.042466+0000 mon.a (mon.0) 763 : cluster 0 Standby manager daemon x started 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.042891+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.203:0/109801692' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:23:57.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.046348+0000 mon.c (mon.2) 15 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.046523+0000 mon.c (mon.2) 16 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.046692+0000 mon.c (mon.2) 17 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.048245+0000 mon.c (mon.2) 18 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.048549+0000 mon.c (mon.2) 19 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.048859+0000 mon.c (mon.2) 20 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.049177+0000 mon.c (mon.2) 21 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.049477+0000 mon.c (mon.2) 22 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.049909+0000 mon.c (mon.2) 23 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.050224+0000 mon.c (mon.2) 24 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.050587+0000 mon.c (mon.2) 25 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:23:57.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.050905+0000 mon.c (mon.2) 26 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:23:57.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.051230+0000 mon.c (mon.2) 27 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:23:57.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.051537+0000 mon.c (mon.2) 28 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:23:57.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.051812+0000 mon.c (mon.2) 29 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:23:57.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:57 smithi203 bash[23654]: audit 2024-03-11T21:23:57.052510+0000 mon.c (mon.2) 30 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:23:57.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: [11/Mar/2024:21:23:57] ENGINE Bus STARTING 2024-03-11T21:23:57.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: CherryPy Checker: 2024-03-11T21:23:57.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: The Application mounted at '' has an empty config. 2024-03-11T21:23:57.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: [11/Mar/2024:21:23:57] ENGINE Serving on http://:::9283 2024-03-11T21:23:57.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:23:57 smithi143 bash[21768]: [11/Mar/2024:21:23:57] ENGINE Bus STARTED 2024-03-11T21:23:58.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.034+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mon.1 v2:172.21.15.203:3300/0 8 ==== mgrmap(e 19) v1 ==== 91563+0+0 (secure 0 0 0) 0x7f9ca8058590 con 0x7f9cb4101f60 2024-03-11T21:23:58.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.038+0000 7f9c9bfff700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c9c078f80 0x7f9c9c07b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:23:58.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.038+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi143=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ca8063f80 con 0x7f9c9c078f80 2024-03-11T21:23:58.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.038+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c9c078f80 0x7f9c9c07b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:23:58.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.038+0000 7f9cb2ffd700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c9c078f80 0x7f9c9c07b370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4101dc0 tx=0x7f9cac009040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:23:58.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.038+0000 7f9c9bfff700 10 client.24433.objecter ms_handle_connect 0x7f9c9c078f80 2024-03-11T21:23:58.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.050+0000 7f9c9bfff700 1 -- 172.21.15.203:0/1625946111 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f9ca8063f80 con 0x7f9c9c078f80 2024-03-11T21:23:58.034 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled alertmanager update... 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 10 librados: watch_flush enter 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 10 librados: watch_flush exit 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 20 client.24433.objecter shutdown clearing up homeless session... 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 10 client.24433.objecter successfully canceled tick 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c9c078f80 msgr2=0x7f9c9c07b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c9c078f80 0x7f9c9c07b370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4101dc0 tx=0x7f9cac009040 comp rx=0 tx=0).stop 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 msgr2=0x7f9cb4103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 --2- 172.21.15.203:0/1625946111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb4101f60 0x7f9cb4103340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8000c00 tx=0x7f9ca8005ee0 comp rx=0 tx=0).stop 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb37fe700 1 -- 172.21.15.203:0/1625946111 reap_dead start 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 shutdown_connections 2024-03-11T21:23:58.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 >> 172.21.15.203:0/1625946111 conn(0x7f9cb40fc300 msgr2=0x7f9cb406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:23:58.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 shutdown_connections 2024-03-11T21:23:58.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 -- 172.21.15.203:0/1625946111 wait complete. 2024-03-11T21:23:58.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:23:58.054+0000 7f9cb9458700 1 librados: shutdown 2024-03-11T21:23:58.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:23:58.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: cluster 2024-03-11T21:23:57.081410+0000 mon.a (mon.0) 764 : cluster 1 Manager daemon y is now available 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.117220+0000 mon.c (mon.2) 31 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.120022+0000 mon.c (mon.2) 32 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.146190+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.150523+0000 mon.c (mon.2) 33 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.160317+0000 mon.c (mon.2) 34 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.160875+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.184866+0000 mon.c (mon.2) 35 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.185284+0000 mon.a (mon.0) 767 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: cephadm 2024-03-11T21:23:57.845974+0000 mgr.y (mgr.14550) 1 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Bus STARTING 2024-03-11T21:23:58.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: cephadm 2024-03-11T21:23:57.948094+0000 mgr.y (mgr.14550) 2 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Serving on http://172.21.15.143:8765 2024-03-11T21:23:58.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.971978+0000 mon.a (mon.0) 768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:57.977966+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: cluster 2024-03-11T21:23:58.037452+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e19: y(active, since 1.01489s), standbys: x 2024-03-11T21:23:58.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:58 smithi203 bash[23654]: audit 2024-03-11T21:23:58.054739+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: cluster 2024-03-11T21:23:57.081410+0000 mon.a (mon.0) 764 : cluster 1 Manager daemon y is now available 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.117220+0000 mon.c (mon.2) 31 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.120022+0000 mon.c (mon.2) 32 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.146190+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.150523+0000 mon.c (mon.2) 33 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.160317+0000 mon.c (mon.2) 34 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.160875+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.184866+0000 mon.c (mon.2) 35 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.185284+0000 mon.a (mon.0) 767 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: cephadm 2024-03-11T21:23:57.845974+0000 mgr.y (mgr.14550) 1 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Bus STARTING 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: cephadm 2024-03-11T21:23:57.948094+0000 mgr.y (mgr.14550) 2 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Serving on http://172.21.15.143:8765 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.971978+0000 mon.a (mon.0) 768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:57.977966+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: cluster 2024-03-11T21:23:58.037452+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e19: y(active, since 1.01489s), standbys: x 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[21512]: audit 2024-03-11T21:23:58.054739+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: cluster 2024-03-11T21:23:57.081410+0000 mon.a (mon.0) 764 : cluster 1 Manager daemon y is now available 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.117220+0000 mon.c (mon.2) 31 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.120022+0000 mon.c (mon.2) 32 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.146190+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.150523+0000 mon.c (mon.2) 33 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:58.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.160317+0000 mon.c (mon.2) 34 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.160875+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.184866+0000 mon.c (mon.2) 35 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.185284+0000 mon.a (mon.0) 767 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: cephadm 2024-03-11T21:23:57.845974+0000 mgr.y (mgr.14550) 1 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Bus STARTING 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: cephadm 2024-03-11T21:23:57.948094+0000 mgr.y (mgr.14550) 2 : cephadm 1 [11/Mar/2024:21:23:57] ENGINE Serving on http://172.21.15.143:8765 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.971978+0000 mon.a (mon.0) 768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:57.977966+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: cluster 2024-03-11T21:23:58.037452+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e19: y(active, since 1.01489s), standbys: x 2024-03-11T21:23:58.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:58 smithi143 bash[29284]: audit 2024-03-11T21:23:58.054739+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:23:58.761 DEBUG:teuthology.orchestra.run.smithi143:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:23:58.765 INFO:tasks.cephadm:Adding grafana.a on smithi203 2024-03-11T21:23:58.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi203=a' 2024-03-11T21:23:59.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: audit 2024-03-11T21:23:58.046935+0000 mgr.y (mgr.14550) 3 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:59.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: audit 2024-03-11T21:23:58.048193+0000 mgr.y (mgr.14550) 4 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi143=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: cephadm 2024-03-11T21:23:58.050750+0000 mgr.y (mgr.14550) 5 : cephadm 1 Saving service alertmanager spec with placement smithi143=a;count:1 2024-03-11T21:23:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: cluster 2024-03-11T21:23:58.051798+0000 mgr.y (mgr.14550) 6 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: cephadm 2024-03-11T21:23:58.062632+0000 mgr.y (mgr.14550) 7 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Serving on https://172.21.15.143:7150 2024-03-11T21:23:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: cephadm 2024-03-11T21:23:58.062801+0000 mgr.y (mgr.14550) 8 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Bus STARTED 2024-03-11T21:23:59.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:23:59 smithi203 bash[23654]: cluster 2024-03-11T21:23:58.535916+0000 mon.a (mon.0) 772 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: audit 2024-03-11T21:23:58.046935+0000 mgr.y (mgr.14550) 3 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: audit 2024-03-11T21:23:58.048193+0000 mgr.y (mgr.14550) 4 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi143=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: cephadm 2024-03-11T21:23:58.050750+0000 mgr.y (mgr.14550) 5 : cephadm 1 Saving service alertmanager spec with placement smithi143=a;count:1 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: cluster 2024-03-11T21:23:58.051798+0000 mgr.y (mgr.14550) 6 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: cephadm 2024-03-11T21:23:58.062632+0000 mgr.y (mgr.14550) 7 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Serving on https://172.21.15.143:7150 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: cephadm 2024-03-11T21:23:58.062801+0000 mgr.y (mgr.14550) 8 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Bus STARTED 2024-03-11T21:23:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[29284]: cluster 2024-03-11T21:23:58.535916+0000 mon.a (mon.0) 772 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: audit 2024-03-11T21:23:58.046935+0000 mgr.y (mgr.14550) 3 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: audit 2024-03-11T21:23:58.048193+0000 mgr.y (mgr.14550) 4 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi143=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: cephadm 2024-03-11T21:23:58.050750+0000 mgr.y (mgr.14550) 5 : cephadm 1 Saving service alertmanager spec with placement smithi143=a;count:1 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: cluster 2024-03-11T21:23:58.051798+0000 mgr.y (mgr.14550) 6 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: cephadm 2024-03-11T21:23:58.062632+0000 mgr.y (mgr.14550) 7 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Serving on https://172.21.15.143:7150 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: cephadm 2024-03-11T21:23:58.062801+0000 mgr.y (mgr.14550) 8 : cephadm 1 [11/Mar/2024:21:23:58] ENGINE Bus STARTED 2024-03-11T21:23:59.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:23:59 smithi143 bash[21512]: cluster 2024-03-11T21:23:58.535916+0000 mon.a (mon.0) 772 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:24:00.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:00 smithi203 bash[23654]: cluster 2024-03-11T21:23:59.050213+0000 mgr.y (mgr.14550) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:00.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:00 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.025582+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:00.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:00 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.025698+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:00.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:00 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.057200+0000 mon.a (mon.0) 773 : cluster 0 mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:24:00.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[21512]: cluster 2024-03-11T21:23:59.050213+0000 mgr.y (mgr.14550) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:00.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.025582+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.025698+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.057200+0000 mon.a (mon.0) 773 : cluster 0 mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[29284]: cluster 2024-03-11T21:23:59.050213+0000 mgr.y (mgr.14550) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.025582+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.025698+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:00.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:00 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.057200+0000 mon.a (mon.0) 773 : cluster 0 mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:24:01.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.189935+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:01.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.190088+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:01.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[21512]: cluster 2024-03-11T21:24:00.541853+0000 mon.a (mon.0) 774 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:24:01.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.189935+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:01.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.190088+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:01.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:01 smithi143 bash[29284]: cluster 2024-03-11T21:24:00.541853+0000 mon.a (mon.0) 774 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:24:02.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:01 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.189935+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:02.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:01 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.190088+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:02.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:01 smithi203 bash[23654]: cluster 2024-03-11T21:24:00.541853+0000 mon.a (mon.0) 774 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:24:02.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[21512]: cluster 2024-03-11T21:24:01.050906+0000 mgr.y (mgr.14550) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:02.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[21512]: cluster 2024-03-11T21:24:01.568573+0000 mon.a (mon.0) 775 : cluster 0 mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:24:02.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[21512]: cluster 2024-03-11T21:24:02.181440+0000 osd.1 (osd.1) 15 : cluster 0 2.9 scrub starts 2024-03-11T21:24:02.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[21512]: cluster 2024-03-11T21:24:02.183760+0000 osd.1 (osd.1) 16 : cluster 0 2.9 scrub ok 2024-03-11T21:24:02.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[29284]: cluster 2024-03-11T21:24:01.050906+0000 mgr.y (mgr.14550) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:02.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[29284]: cluster 2024-03-11T21:24:01.568573+0000 mon.a (mon.0) 775 : cluster 0 mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:24:02.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[29284]: cluster 2024-03-11T21:24:02.181440+0000 osd.1 (osd.1) 15 : cluster 0 2.9 scrub starts 2024-03-11T21:24:02.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:02 smithi143 bash[29284]: cluster 2024-03-11T21:24:02.183760+0000 osd.1 (osd.1) 16 : cluster 0 2.9 scrub ok 2024-03-11T21:24:03.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:02 smithi203 bash[23654]: cluster 2024-03-11T21:24:01.050906+0000 mgr.y (mgr.14550) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:24:03.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:02 smithi203 bash[23654]: cluster 2024-03-11T21:24:01.568573+0000 mon.a (mon.0) 775 : cluster 0 mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:24:03.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:02 smithi203 bash[23654]: cluster 2024-03-11T21:24:02.181440+0000 osd.1 (osd.1) 15 : cluster 0 2.9 scrub starts 2024-03-11T21:24:03.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:02 smithi203 bash[23654]: cluster 2024-03-11T21:24:02.183760+0000 osd.1 (osd.1) 16 : cluster 0 2.9 scrub ok 2024-03-11T21:24:03.506 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: cluster 2024-03-11T21:24:03.051578+0000 mgr.y (mgr.14550) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-11T21:24:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: cluster 2024-03-11T21:24:03.168157+0000 osd.7 (osd.7) 7 : cluster 0 2.b scrub starts 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: cluster 2024-03-11T21:24:03.169104+0000 osd.7 (osd.7) 8 : cluster 0 2.b scrub ok 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: audit 2024-03-11T21:24:03.200525+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: audit 2024-03-11T21:24:03.208004+0000 mon.a (mon.0) 777 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: audit 2024-03-11T21:24:03.368180+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: audit 2024-03-11T21:24:03.377384+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[29284]: cluster 2024-03-11T21:24:03.542626+0000 mon.a (mon.0) 780 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: cluster 2024-03-11T21:24:03.051578+0000 mgr.y (mgr.14550) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: cluster 2024-03-11T21:24:03.168157+0000 osd.7 (osd.7) 7 : cluster 0 2.b scrub starts 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: cluster 2024-03-11T21:24:03.169104+0000 osd.7 (osd.7) 8 : cluster 0 2.b scrub ok 2024-03-11T21:24:04.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: audit 2024-03-11T21:24:03.200525+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: audit 2024-03-11T21:24:03.208004+0000 mon.a (mon.0) 777 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: audit 2024-03-11T21:24:03.368180+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: audit 2024-03-11T21:24:03.377384+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:04 smithi143 bash[21512]: cluster 2024-03-11T21:24:03.542626+0000 mon.a (mon.0) 780 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:24:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: cluster 2024-03-11T21:24:03.051578+0000 mgr.y (mgr.14550) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-11T21:24:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: cluster 2024-03-11T21:24:03.168157+0000 osd.7 (osd.7) 7 : cluster 0 2.b scrub starts 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: cluster 2024-03-11T21:24:03.169104+0000 osd.7 (osd.7) 8 : cluster 0 2.b scrub ok 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: audit 2024-03-11T21:24:03.200525+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: audit 2024-03-11T21:24:03.208004+0000 mon.a (mon.0) 777 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: audit 2024-03-11T21:24:03.368180+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: audit 2024-03-11T21:24:03.377384+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:04.497 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:04 smithi203 bash[23654]: cluster 2024-03-11T21:24:03.542626+0000 mon.a (mon.0) 780 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:24:05.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.374039+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.381234+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.383299+0000 mon.c (mon.2) 36 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.383629+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.384610+0000 mon.c (mon.2) 37 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.384903+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.385764+0000 mon.c (mon.2) 38 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.386058+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.386902+0000 mon.c (mon.2) 39 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.387193+0000 mon.a (mon.0) 786 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.387909+0000 mgr.y (mgr.14550) 12 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3804M 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.391793+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.428280+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.442406+0000 mon.a (mon.0) 789 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.446631+0000 mon.c (mon.2) 40 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.447305+0000 mon.a (mon.0) 790 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.449330+0000 mon.c (mon.2) 41 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.449938+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.451928+0000 mon.c (mon.2) 42 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.452575+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.454423+0000 mon.c (mon.2) 43 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.455013+0000 mon.a (mon.0) 793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.456494+0000 mgr.y (mgr.14550) 13 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3804M 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.463668+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.469951+0000 mon.c (mon.2) 44 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:04.471487+0000 mon.c (mon.2) 45 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.473923+0000 mgr.y (mgr.14550) 14 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.474292+0000 mgr.y (mgr.14550) 15 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.690383+0000 mgr.y (mgr.14550) 16 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.690590+0000 mgr.y (mgr.14550) 17 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.854981+0000 mgr.y (mgr.14550) 18 : cephadm 1 Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cephadm 2024-03-11T21:24:04.905276+0000 mgr.y (mgr.14550) 19 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cluster 2024-03-11T21:24:05.144829+0000 osd.1 (osd.1) 17 : cluster 0 2.4 scrub starts 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: cluster 2024-03-11T21:24:05.146436+0000 osd.1 (osd.1) 18 : cluster 0 2.4 scrub ok 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:05.192611+0000 mon.a (mon.0) 795 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:05.201526+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:05.245430+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:05.260133+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:05 smithi203 bash[23654]: audit 2024-03-11T21:24:05.272102+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.661 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.374039+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.381234+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.383299+0000 mon.c (mon.2) 36 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.383629+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.384610+0000 mon.c (mon.2) 37 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.384903+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.385764+0000 mon.c (mon.2) 38 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.386058+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.386902+0000 mon.c (mon.2) 39 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.387193+0000 mon.a (mon.0) 786 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.387909+0000 mgr.y (mgr.14550) 12 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3804M 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.391793+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.428280+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.442406+0000 mon.a (mon.0) 789 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.446631+0000 mon.c (mon.2) 40 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.447305+0000 mon.a (mon.0) 790 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.449330+0000 mon.c (mon.2) 41 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.449938+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.451928+0000 mon.c (mon.2) 42 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.374039+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.381234+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.383299+0000 mon.c (mon.2) 36 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.663 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.383629+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.384610+0000 mon.c (mon.2) 37 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.384903+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.385764+0000 mon.c (mon.2) 38 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.386058+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.386902+0000 mon.c (mon.2) 39 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.387193+0000 mon.a (mon.0) 786 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.387909+0000 mgr.y (mgr.14550) 12 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3804M 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.391793+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.428280+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.442406+0000 mon.a (mon.0) 789 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.446631+0000 mon.c (mon.2) 40 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.664 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.447305+0000 mon.a (mon.0) 790 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.449330+0000 mon.c (mon.2) 41 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.449938+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.451928+0000 mon.c (mon.2) 42 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.452575+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.454423+0000 mon.c (mon.2) 43 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.455013+0000 mon.a (mon.0) 793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.456494+0000 mgr.y (mgr.14550) 13 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3804M 2024-03-11T21:24:05.665 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.463668+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.469951+0000 mon.c (mon.2) 44 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:04.471487+0000 mon.c (mon.2) 45 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.473923+0000 mgr.y (mgr.14550) 14 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.474292+0000 mgr.y (mgr.14550) 15 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.690383+0000 mgr.y (mgr.14550) 16 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.690590+0000 mgr.y (mgr.14550) 17 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.854981+0000 mgr.y (mgr.14550) 18 : cephadm 1 Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cephadm 2024-03-11T21:24:04.905276+0000 mgr.y (mgr.14550) 19 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cluster 2024-03-11T21:24:05.144829+0000 osd.1 (osd.1) 17 : cluster 0 2.4 scrub starts 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: cluster 2024-03-11T21:24:05.146436+0000 osd.1 (osd.1) 18 : cluster 0 2.4 scrub ok 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:05.192611+0000 mon.a (mon.0) 795 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:05.201526+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:05.245430+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:05.260133+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[21512]: audit 2024-03-11T21:24:05.272102+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.452575+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.454423+0000 mon.c (mon.2) 43 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.455013+0000 mon.a (mon.0) 793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.456494+0000 mgr.y (mgr.14550) 13 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3804M 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.463668+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.469951+0000 mon.c (mon.2) 44 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:04.471487+0000 mon.c (mon.2) 45 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.473923+0000 mgr.y (mgr.14550) 14 : cephadm 1 Updating smithi143:/etc/ceph/ceph.conf 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.474292+0000 mgr.y (mgr.14550) 15 : cephadm 1 Updating smithi203:/etc/ceph/ceph.conf 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.690383+0000 mgr.y (mgr.14550) 16 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.690590+0000 mgr.y (mgr.14550) 17 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:24:05.667 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.854981+0000 mgr.y (mgr.14550) 18 : cephadm 1 Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cephadm 2024-03-11T21:24:04.905276+0000 mgr.y (mgr.14550) 19 : cephadm 1 Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cluster 2024-03-11T21:24:05.144829+0000 osd.1 (osd.1) 17 : cluster 0 2.4 scrub starts 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: cluster 2024-03-11T21:24:05.146436+0000 osd.1 (osd.1) 18 : cluster 0 2.4 scrub ok 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:05.192611+0000 mon.a (mon.0) 795 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:05.201526+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:05.245430+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:05.260133+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 bash[29284]: audit 2024-03-11T21:24:05.272102+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2496197501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b4406a700 msgr2=0x7f8b4406aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2496197501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b4406a700 0x7f8b4406aae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b40009990 tx=0x7f8b40009ca0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2496197501 shutdown_connections 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2496197501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b441071b0 0x7f8b4410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2496197501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b4406a700 0x7f8b4406aae0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2496197501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 0x7f8b44106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2496197501 >> 172.21.15.203:0/2496197501 conn(0x7f8b44077a40 msgr2=0x7f8b44077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2496197501 shutdown_connections 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2496197501 wait complete. 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 librados: starting msgr at 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 librados: starting objecter 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 Processor -- start 2024-03-11T21:24:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- start start 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 librados: setting wanted keys 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 librados: calling monclient init 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b4406a700 0x7f8b441010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 0x7f8b44101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44111650 con 0x7f8b4406a700 2024-03-11T21:24:06.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441114d0 con 0x7f8b441071b0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441117d0 con 0x7f8b4406b0b0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:33936/0 (socket says 172.21.15.203:33936) 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 -- 172.21.15.203:0/2528626456 learned_addr learned my addr 172.21.15.203:0/2528626456 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b492b4700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b4406a700 0x7f8b441010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b48ab3700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 0x7f8b44101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 -- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 msgr2=0x7f8b44101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 0x7f8b44101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 -- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b4406a700 msgr2=0x7f8b441010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b4406a700 0x7f8b441010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 -- 172.21.15.203:0/2528626456 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b441a6140 con 0x7f8b441071b0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b49ab5700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00ef30 tx=0x7f8b3c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:06.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b3a7fc700 10 client.?.objecter ms_handle_connect 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b3a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0092e0 con 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b441a63d0 con 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8b3c004bb0 con 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c004020 con 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 10 client.24457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b441a6910 con 0x7f8b441071b0 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.090+0000 7f8b4b518700 1 librados: init done 2024-03-11T21:24:06.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4404faa0 con 0x7f8b441071b0 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8b3c01b030 con 0x7f8b441071b0 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 0x7f8b30071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(100..100 src has 21..100) v4 ==== 7421+0+0 (secure 0 0 0) 0x7f8b3c08dca0 con 0x7f8b441071b0 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 10 client.24457.objecter ms_dispatch 0x7f8b44001550 osd_map(100..100 src has 21..100) v4 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 3 client.24457.objecter handle_osd_map got epochs [100,100] > 0 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 3 client.24457.objecter handle_osd_map decoding full epoch 100 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b492b4700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 0x7f8b30071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:06.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 20 client.24457.objecter dump_active .. 0 homeless 2024-03-11T21:24:06.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b3a7fc700 10 client.24457.objecter ms_handle_connect 0x7f8b3006f820 2024-03-11T21:24:06.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.094+0000 7f8b492b4700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 0x7f8b30071ce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000b2e0 tx=0x7f8b40005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:06.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.098+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8b3c022020 con 0x7f8b441071b0 2024-03-11T21:24:06.173 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.173 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.173 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.174 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.174 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.174 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:05 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.314+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b440ff580 con 0x7f8b3006f820 2024-03-11T21:24:06.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.322+0000 7f8b3a7fc700 1 -- 172.21.15.203:0/2528626456 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8b440ff580 con 0x7f8b3006f820 2024-03-11T21:24:06.306 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled grafana update... 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 10 librados: watch_flush enter 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 10 librados: watch_flush exit 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 20 client.24457.objecter shutdown clearing up homeless session... 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 10 client.24457.objecter successfully canceled tick 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 msgr2=0x7f8b30071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 0x7f8b30071ce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000b2e0 tx=0x7f8b40005960 comp rx=0 tx=0).stop 2024-03-11T21:24:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 msgr2=0x7f8b44101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00ef30 tx=0x7f8b3c00c3a0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 shutdown_connections 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8b3006f820 0x7f8b30071ce0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b4406a700 0x7f8b441010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b441071b0 0x7f8b44101b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 --2- 172.21.15.203:0/2528626456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b4406b0b0 0x7f8b44101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 >> 172.21.15.203:0/2528626456 conn(0x7f8b44077a40 msgr2=0x7f8b440fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 shutdown_connections 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 -- 172.21.15.203:0/2528626456 wait complete. 2024-03-11T21:24:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:06.326+0000 7f8b4b518700 1 librados: shutdown 2024-03-11T21:24:06.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.438 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.438 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.438 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.439 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:06.439 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:06 smithi143 systemd[1]: Started Ceph node-exporter.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:24:06.439 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[58001]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-03-11T21:24:06.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: cephadm 2024-03-11T21:24:05.020560+0000 mgr.y (mgr.14550) 20 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: cluster 2024-03-11T21:24:05.051925+0000 mgr.y (mgr.14550) 21 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:24:06.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: cephadm 2024-03-11T21:24:05.071233+0000 mgr.y (mgr.14550) 22 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: cephadm 2024-03-11T21:24:05.280167+0000 mgr.y (mgr.14550) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi143 2024-03-11T21:24:06.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: audit 2024-03-11T21:24:06.330253+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: audit 2024-03-11T21:24:06.350547+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: audit 2024-03-11T21:24:06.356874+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:06 smithi203 bash[23654]: audit 2024-03-11T21:24:06.399708+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: cephadm 2024-03-11T21:24:05.020560+0000 mgr.y (mgr.14550) 20 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: cluster 2024-03-11T21:24:05.051925+0000 mgr.y (mgr.14550) 21 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: cephadm 2024-03-11T21:24:05.071233+0000 mgr.y (mgr.14550) 22 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: cephadm 2024-03-11T21:24:05.280167+0000 mgr.y (mgr.14550) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi143 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: audit 2024-03-11T21:24:06.330253+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: audit 2024-03-11T21:24:06.350547+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: audit 2024-03-11T21:24:06.356874+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[21512]: audit 2024-03-11T21:24:06.399708+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: cephadm 2024-03-11T21:24:05.020560+0000 mgr.y (mgr.14550) 20 : cephadm 1 Updating smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: cluster 2024-03-11T21:24:05.051925+0000 mgr.y (mgr.14550) 21 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:24:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: cephadm 2024-03-11T21:24:05.071233+0000 mgr.y (mgr.14550) 22 : cephadm 1 Updating smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:24:06.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: cephadm 2024-03-11T21:24:05.280167+0000 mgr.y (mgr.14550) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi143 2024-03-11T21:24:06.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: audit 2024-03-11T21:24:06.330253+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: audit 2024-03-11T21:24:06.350547+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: audit 2024-03-11T21:24:06.356874+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:06.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:06 smithi143 bash[29284]: audit 2024-03-11T21:24:06.399708+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:07.021 DEBUG:teuthology.orchestra.run.smithi203:grafana.a> sudo journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:24:07.023 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:24:07.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:24:07.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.273 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.273 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.274 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.274 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.274 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.275 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[21512]: audit 2024-03-11T21:24:06.323844+0000 mgr.y (mgr.14550) 24 : audit 0 from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[21512]: cephadm 2024-03-11T21:24:06.325623+0000 mgr.y (mgr.14550) 25 : cephadm 1 Saving service grafana spec with placement smithi203=a;count:1 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[21512]: cephadm 2024-03-11T21:24:06.400661+0000 mgr.y (mgr.14550) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi203 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[21512]: cluster 2024-03-11T21:24:06.844233+0000 osd.3 (osd.3) 11 : cluster 0 2.1b scrub starts 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[21512]: cluster 2024-03-11T21:24:06.846217+0000 osd.3 (osd.3) 12 : cluster 0 2.1b scrub ok 2024-03-11T21:24:07.769 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[29284]: audit 2024-03-11T21:24:06.323844+0000 mgr.y (mgr.14550) 24 : audit 0 from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:07.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[29284]: cephadm 2024-03-11T21:24:06.325623+0000 mgr.y (mgr.14550) 25 : cephadm 1 Saving service grafana spec with placement smithi203=a;count:1 2024-03-11T21:24:07.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[29284]: cephadm 2024-03-11T21:24:06.400661+0000 mgr.y (mgr.14550) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi203 2024-03-11T21:24:07.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[29284]: cluster 2024-03-11T21:24:06.844233+0000 osd.3 (osd.3) 11 : cluster 0 2.1b scrub starts 2024-03-11T21:24:07.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[29284]: cluster 2024-03-11T21:24:06.846217+0000 osd.3 (osd.3) 12 : cluster 0 2.1b scrub ok 2024-03-11T21:24:07.773 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.773 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.773 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.774 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.774 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.775 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.778 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: Started Ceph node-exporter.b for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:24:07.778 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[50519]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[23654]: audit 2024-03-11T21:24:06.323844+0000 mgr.y (mgr.14550) 24 : audit 0 from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[23654]: cephadm 2024-03-11T21:24:06.325623+0000 mgr.y (mgr.14550) 25 : cephadm 1 Saving service grafana spec with placement smithi203=a;count:1 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[23654]: cephadm 2024-03-11T21:24:06.400661+0000 mgr.y (mgr.14550) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi203 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[23654]: cluster 2024-03-11T21:24:06.844233+0000 osd.3 (osd.3) 11 : cluster 0 2.1b scrub starts 2024-03-11T21:24:07.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:07 smithi203 bash[23654]: cluster 2024-03-11T21:24:06.846217+0000 osd.3 (osd.3) 12 : cluster 0 2.1b scrub ok 2024-03-11T21:24:08.043 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:07 smithi143 bash[58001]: v1.5.0: Pulling from prometheus/node-exporter 2024-03-11T21:24:08.437 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 22b70bddd3ac: Pulling fs layer 2024-03-11T21:24:08.437 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 5c12815fee55: Pulling fs layer 2024-03-11T21:24:08.438 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: c0e87333d380: Pulling fs layer 2024-03-11T21:24:08.438 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:08 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:08.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:08.773 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[50519]: v1.5.0: Pulling from prometheus/node-exporter 2024-03-11T21:24:08.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cluster 2024-03-11T21:24:07.052689+0000 mgr.y (mgr.14550) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:24:08.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cluster 2024-03-11T21:24:07.180178+0000 osd.2 (osd.2) 11 : cluster 0 5.b scrub starts 2024-03-11T21:24:08.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cluster 2024-03-11T21:24:07.182139+0000 osd.2 (osd.2) 12 : cluster 0 5.b scrub ok 2024-03-11T21:24:08.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: audit 2024-03-11T21:24:07.532245+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: audit 2024-03-11T21:24:07.547392+0000 mon.a (mon.0) 805 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: audit 2024-03-11T21:24:07.575805+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: audit 2024-03-11T21:24:07.585415+0000 mon.a (mon.0) 807 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: audit 2024-03-11T21:24:07.602818+0000 mon.a (mon.0) 808 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cephadm 2024-03-11T21:24:07.641185+0000 mgr.y (mgr.14550) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi143 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cluster 2024-03-11T21:24:07.860120+0000 osd.3 (osd.3) 13 : cluster 0 3.18 scrub starts 2024-03-11T21:24:08.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[23654]: cluster 2024-03-11T21:24:07.864599+0000 osd.3 (osd.3) 14 : cluster 0 3.18 scrub ok 2024-03-11T21:24:08.931 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 5c12815fee55: Download complete 2024-03-11T21:24:08.931 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 22b70bddd3ac: Verifying Checksum 2024-03-11T21:24:08.931 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 22b70bddd3ac: Download complete 2024-03-11T21:24:08.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cluster 2024-03-11T21:24:07.052689+0000 mgr.y (mgr.14550) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:24:08.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cluster 2024-03-11T21:24:07.180178+0000 osd.2 (osd.2) 11 : cluster 0 5.b scrub starts 2024-03-11T21:24:08.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cluster 2024-03-11T21:24:07.182139+0000 osd.2 (osd.2) 12 : cluster 0 5.b scrub ok 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: audit 2024-03-11T21:24:07.532245+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: audit 2024-03-11T21:24:07.547392+0000 mon.a (mon.0) 805 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: audit 2024-03-11T21:24:07.575805+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: audit 2024-03-11T21:24:07.585415+0000 mon.a (mon.0) 807 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: audit 2024-03-11T21:24:07.602818+0000 mon.a (mon.0) 808 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cephadm 2024-03-11T21:24:07.641185+0000 mgr.y (mgr.14550) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi143 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cluster 2024-03-11T21:24:07.860120+0000 osd.3 (osd.3) 13 : cluster 0 3.18 scrub starts 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[29284]: cluster 2024-03-11T21:24:07.864599+0000 osd.3 (osd.3) 14 : cluster 0 3.18 scrub ok 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cluster 2024-03-11T21:24:07.052689+0000 mgr.y (mgr.14550) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cluster 2024-03-11T21:24:07.180178+0000 osd.2 (osd.2) 11 : cluster 0 5.b scrub starts 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cluster 2024-03-11T21:24:07.182139+0000 osd.2 (osd.2) 12 : cluster 0 5.b scrub ok 2024-03-11T21:24:08.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: audit 2024-03-11T21:24:07.532245+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: audit 2024-03-11T21:24:07.547392+0000 mon.a (mon.0) 805 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: audit 2024-03-11T21:24:07.575805+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: audit 2024-03-11T21:24:07.585415+0000 mon.a (mon.0) 807 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: audit 2024-03-11T21:24:07.602818+0000 mon.a (mon.0) 808 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cephadm 2024-03-11T21:24:07.641185+0000 mgr.y (mgr.14550) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi143 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cluster 2024-03-11T21:24:07.860120+0000 osd.3 (osd.3) 13 : cluster 0 3.18 scrub starts 2024-03-11T21:24:08.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[21512]: cluster 2024-03-11T21:24:07.864599+0000 osd.3 (osd.3) 14 : cluster 0 3.18 scrub ok 2024-03-11T21:24:09.187 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:08 smithi143 bash[58001]: 22b70bddd3ac: Pull complete 2024-03-11T21:24:09.187 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[58001]: c0e87333d380: Verifying Checksum 2024-03-11T21:24:09.187 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[58001]: c0e87333d380: Download complete 2024-03-11T21:24:09.273 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[50519]: 22b70bddd3ac: Pulling fs layer 2024-03-11T21:24:09.273 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[50519]: 5c12815fee55: Pulling fs layer 2024-03-11T21:24:09.273 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:08 smithi203 bash[50519]: c0e87333d380: Pulling fs layer 2024-03-11T21:24:09.663 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: 22b70bddd3ac: Verifying Checksum 2024-03-11T21:24:09.663 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: 22b70bddd3ac: Download complete 2024-03-11T21:24:09.663 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: 22b70bddd3ac: Pull complete 2024-03-11T21:24:09.663 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: 5c12815fee55: Verifying Checksum 2024-03-11T21:24:09.663 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: 5c12815fee55: Download complete 2024-03-11T21:24:09.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[23654]: audit 2024-03-11T21:24:08.057828+0000 mgr.y (mgr.14550) 29 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:09.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[23654]: cluster 2024-03-11T21:24:08.574379+0000 osd.5 (osd.5) 9 : cluster 0 3.1c scrub starts 2024-03-11T21:24:09.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[23654]: cluster 2024-03-11T21:24:08.576863+0000 osd.5 (osd.5) 10 : cluster 0 3.1c scrub ok 2024-03-11T21:24:09.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[23654]: cluster 2024-03-11T21:24:09.213247+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:24:09.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[23654]: cluster 2024-03-11T21:24:09.214852+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:24:09.687 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[58001]: 5c12815fee55: Pull complete 2024-03-11T21:24:09.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[21512]: audit 2024-03-11T21:24:08.057828+0000 mgr.y (mgr.14550) 29 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:09.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[21512]: cluster 2024-03-11T21:24:08.574379+0000 osd.5 (osd.5) 9 : cluster 0 3.1c scrub starts 2024-03-11T21:24:09.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[21512]: cluster 2024-03-11T21:24:08.576863+0000 osd.5 (osd.5) 10 : cluster 0 3.1c scrub ok 2024-03-11T21:24:09.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[21512]: cluster 2024-03-11T21:24:09.213247+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:24:09.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[21512]: cluster 2024-03-11T21:24:09.214852+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:24:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[29284]: audit 2024-03-11T21:24:08.057828+0000 mgr.y (mgr.14550) 29 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[29284]: cluster 2024-03-11T21:24:08.574379+0000 osd.5 (osd.5) 9 : cluster 0 3.1c scrub starts 2024-03-11T21:24:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[29284]: cluster 2024-03-11T21:24:08.576863+0000 osd.5 (osd.5) 10 : cluster 0 3.1c scrub ok 2024-03-11T21:24:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[29284]: cluster 2024-03-11T21:24:09.213247+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:24:09.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[29284]: cluster 2024-03-11T21:24:09.214852+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:24:10.003 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: c0e87333d380: Verifying Checksum 2024-03-11T21:24:10.003 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:09 smithi203 bash[50519]: c0e87333d380: Download complete 2024-03-11T21:24:10.034 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[58001]: c0e87333d380: Pull complete 2024-03-11T21:24:10.035 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:09 smithi143 bash[58001]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-03-11T21:24:10.035 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[58001]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:24:10.273 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[50519]: 5c12815fee55: Pull complete 2024-03-11T21:24:10.773 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[50519]: c0e87333d380: Pull complete 2024-03-11T21:24:10.773 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[50519]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-03-11T21:24:10.773 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[50519]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:24:10.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[23654]: cluster 2024-03-11T21:24:09.053744+0000 mgr.y (mgr.14550) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T21:24:10.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[23654]: cluster 2024-03-11T21:24:10.041496+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-03-11T21:24:10.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:10 smithi203 bash[23654]: cluster 2024-03-11T21:24:10.043692+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-03-11T21:24:10.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[29284]: cluster 2024-03-11T21:24:09.053744+0000 mgr.y (mgr.14550) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T21:24:10.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[29284]: cluster 2024-03-11T21:24:10.041496+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-03-11T21:24:10.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[29284]: cluster 2024-03-11T21:24:10.043692+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-03-11T21:24:10.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[21512]: cluster 2024-03-11T21:24:09.053744+0000 mgr.y (mgr.14550) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T21:24:10.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[21512]: cluster 2024-03-11T21:24:10.041496+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-03-11T21:24:10.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:10 smithi143 bash[21512]: cluster 2024-03-11T21:24:10.043692+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-03-11T21:24:11.780 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:24:11.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:11 smithi143 bash[21512]: cluster 2024-03-11T21:24:10.199296+0000 osd.2 (osd.2) 13 : cluster 0 2.e scrub starts 2024-03-11T21:24:11.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:11 smithi143 bash[21512]: cluster 2024-03-11T21:24:10.200741+0000 osd.2 (osd.2) 14 : cluster 0 2.e scrub ok 2024-03-11T21:24:11.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:11 smithi143 bash[29284]: cluster 2024-03-11T21:24:10.199296+0000 osd.2 (osd.2) 13 : cluster 0 2.e scrub starts 2024-03-11T21:24:11.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:11 smithi143 bash[29284]: cluster 2024-03-11T21:24:10.200741+0000 osd.2 (osd.2) 14 : cluster 0 2.e scrub ok 2024-03-11T21:24:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[23654]: cluster 2024-03-11T21:24:10.199296+0000 osd.2 (osd.2) 13 : cluster 0 2.e scrub starts 2024-03-11T21:24:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[23654]: cluster 2024-03-11T21:24:10.200741+0000 osd.2 (osd.2) 14 : cluster 0 2.e scrub ok 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.731Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.731Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.732Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.732Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.732Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.732Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:24:12.024 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:24:12.025 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:24:12.026 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:24:12.027 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.733Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.734Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:24:12.028 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:24:11 smithi203 bash[50519]: ts=2024-03-11T21:24:11.734Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:24:12.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: cluster 2024-03-11T21:24:11.054390+0000 mgr.y (mgr.14550) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:24:12.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: cluster 2024-03-11T21:24:11.161871+0000 osd.2 (osd.2) 15 : cluster 0 5.d deep-scrub starts 2024-03-11T21:24:12.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: cluster 2024-03-11T21:24:11.167205+0000 osd.2 (osd.2) 16 : cluster 0 5.d deep-scrub ok 2024-03-11T21:24:12.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: cluster 2024-03-11T21:24:11.861528+0000 osd.3 (osd.3) 15 : cluster 0 2.19 scrub starts 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: cluster 2024-03-11T21:24:11.863918+0000 osd.3 (osd.3) 16 : cluster 0 2.19 scrub ok 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: audit 2024-03-11T21:24:12.148119+0000 mon.a (mon.0) 809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[21512]: audit 2024-03-11T21:24:12.182403+0000 mon.c (mon.2) 46 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: cluster 2024-03-11T21:24:11.054390+0000 mgr.y (mgr.14550) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: cluster 2024-03-11T21:24:11.161871+0000 osd.2 (osd.2) 15 : cluster 0 5.d deep-scrub starts 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: cluster 2024-03-11T21:24:11.167205+0000 osd.2 (osd.2) 16 : cluster 0 5.d deep-scrub ok 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: cluster 2024-03-11T21:24:11.861528+0000 osd.3 (osd.3) 15 : cluster 0 2.19 scrub starts 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: cluster 2024-03-11T21:24:11.863918+0000 osd.3 (osd.3) 16 : cluster 0 2.19 scrub ok 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: audit 2024-03-11T21:24:12.148119+0000 mon.a (mon.0) 809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:12.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:12 smithi143 bash[29284]: audit 2024-03-11T21:24:12.182403+0000 mon.c (mon.2) 46 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:13.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: cluster 2024-03-11T21:24:11.054390+0000 mgr.y (mgr.14550) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:24:13.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: cluster 2024-03-11T21:24:11.161871+0000 osd.2 (osd.2) 15 : cluster 0 5.d deep-scrub starts 2024-03-11T21:24:13.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: cluster 2024-03-11T21:24:11.167205+0000 osd.2 (osd.2) 16 : cluster 0 5.d deep-scrub ok 2024-03-11T21:24:13.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: cluster 2024-03-11T21:24:11.861528+0000 osd.3 (osd.3) 15 : cluster 0 2.19 scrub starts 2024-03-11T21:24:13.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: cluster 2024-03-11T21:24:11.863918+0000 osd.3 (osd.3) 16 : cluster 0 2.19 scrub ok 2024-03-11T21:24:13.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: audit 2024-03-11T21:24:12.148119+0000 mon.a (mon.0) 809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:13.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:12 smithi203 bash[23654]: audit 2024-03-11T21:24:12.182403+0000 mon.c (mon.2) 46 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:13.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[29284]: cluster 2024-03-11T21:24:12.883588+0000 osd.3 (osd.3) 17 : cluster 0 3.1e scrub starts 2024-03-11T21:24:13.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[29284]: cluster 2024-03-11T21:24:12.887774+0000 osd.3 (osd.3) 18 : cluster 0 3.1e scrub ok 2024-03-11T21:24:13.721 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[21512]: cluster 2024-03-11T21:24:12.883588+0000 osd.3 (osd.3) 17 : cluster 0 3.1e scrub starts 2024-03-11T21:24:13.721 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[21512]: cluster 2024-03-11T21:24:12.887774+0000 osd.3 (osd.3) 18 : cluster 0 3.1e scrub ok 2024-03-11T21:24:13.721 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.550Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:24:13.721 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.550Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:24:13.721 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.551Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:24:13.721 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.551Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.551Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.551Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:24:13.722 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:24:13.723 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:24:13.724 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:24:13.725 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:13 smithi143 bash[58001]: ts=2024-03-11T21:24:13.552Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:24:14.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:13 smithi203 bash[23654]: cluster 2024-03-11T21:24:12.883588+0000 osd.3 (osd.3) 17 : cluster 0 3.1e scrub starts 2024-03-11T21:24:14.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:13 smithi203 bash[23654]: cluster 2024-03-11T21:24:12.887774+0000 osd.3 (osd.3) 18 : cluster 0 3.1e scrub ok 2024-03-11T21:24:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[29284]: cluster 2024-03-11T21:24:13.055771+0000 mgr.y (mgr.14550) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[29284]: cluster 2024-03-11T21:24:13.612818+0000 osd.5 (osd.5) 11 : cluster 0 2.12 scrub starts 2024-03-11T21:24:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[29284]: cluster 2024-03-11T21:24:13.615085+0000 osd.5 (osd.5) 12 : cluster 0 2.12 scrub ok 2024-03-11T21:24:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[29284]: cluster 2024-03-11T21:24:13.928919+0000 osd.3 (osd.3) 19 : cluster 0 3.b scrub starts 2024-03-11T21:24:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[29284]: cluster 2024-03-11T21:24:13.931369+0000 osd.3 (osd.3) 20 : cluster 0 3.b scrub ok 2024-03-11T21:24:14.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[21512]: cluster 2024-03-11T21:24:13.055771+0000 mgr.y (mgr.14550) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:14.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[21512]: cluster 2024-03-11T21:24:13.612818+0000 osd.5 (osd.5) 11 : cluster 0 2.12 scrub starts 2024-03-11T21:24:14.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[21512]: cluster 2024-03-11T21:24:13.615085+0000 osd.5 (osd.5) 12 : cluster 0 2.12 scrub ok 2024-03-11T21:24:14.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[21512]: cluster 2024-03-11T21:24:13.928919+0000 osd.3 (osd.3) 19 : cluster 0 3.b scrub starts 2024-03-11T21:24:14.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:14 smithi143 bash[21512]: cluster 2024-03-11T21:24:13.931369+0000 osd.3 (osd.3) 20 : cluster 0 3.b scrub ok 2024-03-11T21:24:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:14 smithi203 bash[23654]: cluster 2024-03-11T21:24:13.055771+0000 mgr.y (mgr.14550) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:14 smithi203 bash[23654]: cluster 2024-03-11T21:24:13.612818+0000 osd.5 (osd.5) 11 : cluster 0 2.12 scrub starts 2024-03-11T21:24:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:14 smithi203 bash[23654]: cluster 2024-03-11T21:24:13.615085+0000 osd.5 (osd.5) 12 : cluster 0 2.12 scrub ok 2024-03-11T21:24:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:14 smithi203 bash[23654]: cluster 2024-03-11T21:24:13.928919+0000 osd.3 (osd.3) 19 : cluster 0 3.b scrub starts 2024-03-11T21:24:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:14 smithi203 bash[23654]: cluster 2024-03-11T21:24:13.931369+0000 osd.3 (osd.3) 20 : cluster 0 3.b scrub ok 2024-03-11T21:24:16.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:15.056540+0000 mgr.y (mgr.14550) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-11T21:24:16.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:15.432464+0000 osd.4 (osd.4) 9 : cluster 0 3.14 scrub starts 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:15.436920+0000 osd.4 (osd.4) 10 : cluster 0 3.14 scrub ok 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:15.617673+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:15.622814+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.079224+0000 osd.7 (osd.7) 11 : cluster 0 2.1d scrub starts 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.081609+0000 osd.7 (osd.7) 12 : cluster 0 2.1d scrub ok 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:15.056540+0000 mgr.y (mgr.14550) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:15.432464+0000 osd.4 (osd.4) 9 : cluster 0 3.14 scrub starts 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:15.436920+0000 osd.4 (osd.4) 10 : cluster 0 3.14 scrub ok 2024-03-11T21:24:16.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:15.617673+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-03-11T21:24:16.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:15.622814+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-03-11T21:24:16.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.079224+0000 osd.7 (osd.7) 11 : cluster 0 2.1d scrub starts 2024-03-11T21:24:16.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:16 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.081609+0000 osd.7 (osd.7) 12 : cluster 0 2.1d scrub ok 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:15.056540+0000 mgr.y (mgr.14550) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:15.432464+0000 osd.4 (osd.4) 9 : cluster 0 3.14 scrub starts 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:15.436920+0000 osd.4 (osd.4) 10 : cluster 0 3.14 scrub ok 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:15.617673+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:15.622814+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.079224+0000 osd.7 (osd.7) 11 : cluster 0 2.1d scrub starts 2024-03-11T21:24:17.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:16 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.081609+0000 osd.7 (osd.7) 12 : cluster 0 2.1d scrub ok 2024-03-11T21:24:17.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/4240883062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a810e650 msgr2=0x7f97a8110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- 172.21.15.143:0/4240883062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a810e650 0x7f97a8110ac0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97a00098f0 tx=0x7f97a0009c00 comp rx=0 tx=0).stop 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/4240883062 shutdown_connections 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- 172.21.15.143:0/4240883062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a8074210 0x7f97a80745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- 172.21.15.143:0/4240883062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a810e650 0x7f97a8110ac0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- 172.21.15.143:0/4240883062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 0x7f97a810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/4240883062 >> 172.21.15.143:0/4240883062 conn(0x7f97a80fd8e0 msgr2=0x7f97a80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/4240883062 shutdown_connections 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/4240883062 wait complete. 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 librados: starting msgr at 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 librados: starting objecter 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 Processor -- start 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- start start 2024-03-11T21:24:17.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 librados: setting wanted keys 2024-03-11T21:24:17.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 librados: calling monclient init 2024-03-11T21:24:17.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 0x7f97a8199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:17.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 0x7f97a819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:17.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:17.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8113860 con 0x7f97a810e650 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81136e0 con 0x7f97a8074b30 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81139e0 con 0x7f97a8074210 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36956/0 (socket says 172.21.15.143:36956) 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 -- 172.21.15.143:0/353758899 learned_addr learned my addr 172.21.15.143:0/353758899 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a6ffd700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 0x7f97a8199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a67fc700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 0x7f97a819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 -- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 msgr2=0x7f97a8199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 0x7f97a8199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 -- 172.21.15.143:0/353758899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 msgr2=0x7f97a819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 0x7f97a819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 -- 172.21.15.143:0/353758899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a819ee40 con 0x7f97a810e650 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a6ffd700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 0x7f97a8199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:17.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97a77fe700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009230 tx=0x7f97a000fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f979ffff700 10 client.?.objecter ms_handle_connect 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f979ffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0027070 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f97a000bb40 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a819f0d0 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0004930 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.711+0000 7f97ad392700 10 client.14595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a81a69b0 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f97ad392700 1 librados: init done 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a804faa0 con 0x7f97a810e650 2024-03-11T21:24:17.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f97a002a070 con 0x7f97a810e650 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 0x7f9794071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 21..100) v4 ==== 7421+0+0 (secure 0 0 0) 0x7f97a008ec40 con 0x7f97a810e650 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 10 client.14595.objecter ms_dispatch 0x7f97a8001550 osd_map(100..100 src has 21..100) v4 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 3 client.14595.objecter handle_osd_map got epochs [100,100] > 0 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 3 client.14595.objecter handle_osd_map decoding full epoch 100 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f97a6ffd700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 0x7f9794071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:17.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 20 client.14595.objecter dump_active .. 0 homeless 2024-03-11T21:24:17.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f97a6ffd700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 0x7f9794071d30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9790009750 tx=0x7f9790009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:17.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.715+0000 7f979ffff700 10 client.14595.objecter ms_handle_connect 0x7f979406f870 2024-03-11T21:24:17.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:17.719+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f97a0013030 con 0x7f97a810e650 2024-03-11T21:24:17.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.285056+0000 osd.2 (osd.2) 17 : cluster 0 2.c scrub starts 2024-03-11T21:24:17.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.287570+0000 osd.2 (osd.2) 18 : cluster 0 2.c scrub ok 2024-03-11T21:24:17.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.420221+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-03-11T21:24:17.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[29284]: cluster 2024-03-11T21:24:16.422480+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-03-11T21:24:17.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.285056+0000 osd.2 (osd.2) 17 : cluster 0 2.c scrub starts 2024-03-11T21:24:17.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.287570+0000 osd.2 (osd.2) 18 : cluster 0 2.c scrub ok 2024-03-11T21:24:17.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.420221+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-03-11T21:24:17.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:17 smithi143 bash[21512]: cluster 2024-03-11T21:24:16.422480+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-03-11T21:24:17.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.011+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f97a81a6b80 con 0x7f97a810e650 2024-03-11T21:24:17.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f979ffff700 1 -- 172.21.15.143:0/353758899 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f97a005c2e0 con 0x7f97a810e650 2024-03-11T21:24:17.992 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2024-03-11T21:24:17.992 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCCdu9lcNTRABAA960MUUk9xNdS4xo9uZFhZw== 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 10 librados: watch_flush enter 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 10 librados: watch_flush exit 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 20 client.14595.objecter shutdown clearing up homeless session... 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 10 client.14595.objecter successfully canceled tick 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 msgr2=0x7f9794071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 0x7f9794071d30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9790009750 tx=0x7f9790009310 comp rx=0 tx=0).stop 2024-03-11T21:24:17.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 msgr2=0x7f97a819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.015+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009230 tx=0x7f97a000fb40 comp rx=0 tx=0).stop 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 shutdown_connections 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f979406f870 0x7f9794071d30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a810e650 0x7f97a819e700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a8074210 0x7f97a8199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 --2- 172.21.15.143:0/353758899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074b30 0x7f97a819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:17.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 >> 172.21.15.143:0/353758899 conn(0x7f97a80fd8e0 msgr2=0x7f97a81060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:17.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 shutdown_connections 2024-03-11T21:24:17.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 -- 172.21.15.143:0/353758899 wait complete. 2024-03-11T21:24:17.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:18.019+0000 7f97ad392700 1 librados: shutdown 2024-03-11T21:24:18.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:17 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.285056+0000 osd.2 (osd.2) 17 : cluster 0 2.c scrub starts 2024-03-11T21:24:18.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:17 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.287570+0000 osd.2 (osd.2) 18 : cluster 0 2.c scrub ok 2024-03-11T21:24:18.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:17 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.420221+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-03-11T21:24:18.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:17 smithi203 bash[23654]: cluster 2024-03-11T21:24:16.422480+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-03-11T21:24:18.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:18.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: cluster 2024-03-11T21:24:17.057266+0000 mgr.y (mgr.14550) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:18.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: cluster 2024-03-11T21:24:17.412920+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-03-11T21:24:18.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: cluster 2024-03-11T21:24:17.418519+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-03-11T21:24:18.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: cluster 2024-03-11T21:24:17.619368+0000 osd.5 (osd.5) 15 : cluster 0 6.19 scrub starts 2024-03-11T21:24:18.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: cluster 2024-03-11T21:24:17.621811+0000 osd.5 (osd.5) 16 : cluster 0 6.19 scrub ok 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: audit 2024-03-11T21:24:18.013446+0000 mon.a (mon.0) 810 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[21512]: audit 2024-03-11T21:24:18.017168+0000 mon.a (mon.0) 811 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: cluster 2024-03-11T21:24:17.057266+0000 mgr.y (mgr.14550) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: cluster 2024-03-11T21:24:17.412920+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: cluster 2024-03-11T21:24:17.418519+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: cluster 2024-03-11T21:24:17.619368+0000 osd.5 (osd.5) 15 : cluster 0 6.19 scrub starts 2024-03-11T21:24:18.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: cluster 2024-03-11T21:24:17.621811+0000 osd.5 (osd.5) 16 : cluster 0 6.19 scrub ok 2024-03-11T21:24:18.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: audit 2024-03-11T21:24:18.013446+0000 mon.a (mon.0) 810 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:18.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:18 smithi143 bash[29284]: audit 2024-03-11T21:24:18.017168+0000 mon.a (mon.0) 811 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:19.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: cluster 2024-03-11T21:24:17.057266+0000 mgr.y (mgr.14550) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:19.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: cluster 2024-03-11T21:24:17.412920+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-03-11T21:24:19.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: cluster 2024-03-11T21:24:17.418519+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-03-11T21:24:19.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: cluster 2024-03-11T21:24:17.619368+0000 osd.5 (osd.5) 15 : cluster 0 6.19 scrub starts 2024-03-11T21:24:19.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: cluster 2024-03-11T21:24:17.621811+0000 osd.5 (osd.5) 16 : cluster 0 6.19 scrub ok 2024-03-11T21:24:19.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: audit 2024-03-11T21:24:18.013446+0000 mon.a (mon.0) 810 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:19.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:18 smithi203 bash[23654]: audit 2024-03-11T21:24:18.017168+0000 mon.a (mon.0) 811 : audit 1 from='client.? 172.21.15.143:0/353758899' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:19.443 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:24:19.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:24:19.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:24:19.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:24:19.704 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: audit 2024-03-11T21:24:18.065813+0000 mgr.y (mgr.14550) 35 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:19.704 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.364393+0000 osd.0 (osd.0) 15 : cluster 0 3.6 scrub starts 2024-03-11T21:24:19.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.368545+0000 osd.0 (osd.0) 16 : cluster 0 3.6 scrub ok 2024-03-11T21:24:19.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.461707+0000 osd.6 (osd.6) 11 : cluster 0 5.11 scrub starts 2024-03-11T21:24:19.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.464020+0000 osd.6 (osd.6) 12 : cluster 0 5.11 scrub ok 2024-03-11T21:24:19.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.490935+0000 osd.4 (osd.4) 11 : cluster 0 2.f scrub starts 2024-03-11T21:24:19.706 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[21512]: cluster 2024-03-11T21:24:18.494088+0000 osd.4 (osd.4) 12 : cluster 0 2.f scrub ok 2024-03-11T21:24:19.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: audit 2024-03-11T21:24:18.065813+0000 mgr.y (mgr.14550) 35 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:19.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.364393+0000 osd.0 (osd.0) 15 : cluster 0 3.6 scrub starts 2024-03-11T21:24:19.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.368545+0000 osd.0 (osd.0) 16 : cluster 0 3.6 scrub ok 2024-03-11T21:24:19.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.461707+0000 osd.6 (osd.6) 11 : cluster 0 5.11 scrub starts 2024-03-11T21:24:19.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.464020+0000 osd.6 (osd.6) 12 : cluster 0 5.11 scrub ok 2024-03-11T21:24:19.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.490935+0000 osd.4 (osd.4) 11 : cluster 0 2.f scrub starts 2024-03-11T21:24:19.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:19 smithi143 bash[29284]: cluster 2024-03-11T21:24:18.494088+0000 osd.4 (osd.4) 12 : cluster 0 2.f scrub ok 2024-03-11T21:24:19.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: audit 2024-03-11T21:24:18.065813+0000 mgr.y (mgr.14550) 35 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:19.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.364393+0000 osd.0 (osd.0) 15 : cluster 0 3.6 scrub starts 2024-03-11T21:24:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.368545+0000 osd.0 (osd.0) 16 : cluster 0 3.6 scrub ok 2024-03-11T21:24:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.461707+0000 osd.6 (osd.6) 11 : cluster 0 5.11 scrub starts 2024-03-11T21:24:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.464020+0000 osd.6 (osd.6) 12 : cluster 0 5.11 scrub ok 2024-03-11T21:24:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.490935+0000 osd.4 (osd.4) 11 : cluster 0 2.f scrub starts 2024-03-11T21:24:19.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:19 smithi203 bash[23654]: cluster 2024-03-11T21:24:18.494088+0000 osd.4 (osd.4) 12 : cluster 0 2.f scrub ok 2024-03-11T21:24:20.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.058752+0000 mgr.y (mgr.14550) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.139870+0000 osd.7 (osd.7) 13 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.148234+0000 osd.7 (osd.7) 14 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.306755+0000 osd.2 (osd.2) 19 : cluster 0 2.10 scrub starts 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.308850+0000 osd.2 (osd.2) 20 : cluster 0 2.10 scrub ok 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.546693+0000 osd.5 (osd.5) 17 : cluster 0 2.18 scrub starts 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.548387+0000 osd.5 (osd.5) 18 : cluster 0 2.18 scrub ok 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.942020+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[29284]: cluster 2024-03-11T21:24:19.944682+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.058752+0000 mgr.y (mgr.14550) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.139870+0000 osd.7 (osd.7) 13 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.148234+0000 osd.7 (osd.7) 14 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:24:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.306755+0000 osd.2 (osd.2) 19 : cluster 0 2.10 scrub starts 2024-03-11T21:24:20.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.308850+0000 osd.2 (osd.2) 20 : cluster 0 2.10 scrub ok 2024-03-11T21:24:20.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.546693+0000 osd.5 (osd.5) 17 : cluster 0 2.18 scrub starts 2024-03-11T21:24:20.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.548387+0000 osd.5 (osd.5) 18 : cluster 0 2.18 scrub ok 2024-03-11T21:24:20.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.942020+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T21:24:20.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:20 smithi143 bash[21512]: cluster 2024-03-11T21:24:19.944682+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T21:24:21.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.058752+0000 mgr.y (mgr.14550) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:21.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.139870+0000 osd.7 (osd.7) 13 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:24:21.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.148234+0000 osd.7 (osd.7) 14 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:24:21.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.306755+0000 osd.2 (osd.2) 19 : cluster 0 2.10 scrub starts 2024-03-11T21:24:21.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.308850+0000 osd.2 (osd.2) 20 : cluster 0 2.10 scrub ok 2024-03-11T21:24:21.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.546693+0000 osd.5 (osd.5) 17 : cluster 0 2.18 scrub starts 2024-03-11T21:24:21.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.548387+0000 osd.5 (osd.5) 18 : cluster 0 2.18 scrub ok 2024-03-11T21:24:21.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.942020+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T21:24:21.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:20 smithi203 bash[23654]: cluster 2024-03-11T21:24:19.944682+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T21:24:21.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:21 smithi143 bash[21512]: cluster 2024-03-11T21:24:20.320612+0000 osd.2 (osd.2) 21 : cluster 0 6.14 scrub starts 2024-03-11T21:24:21.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:21 smithi143 bash[21512]: cluster 2024-03-11T21:24:20.323390+0000 osd.2 (osd.2) 22 : cluster 0 6.14 scrub ok 2024-03-11T21:24:21.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:21 smithi143 bash[29284]: cluster 2024-03-11T21:24:20.320612+0000 osd.2 (osd.2) 21 : cluster 0 6.14 scrub starts 2024-03-11T21:24:21.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:21 smithi143 bash[29284]: cluster 2024-03-11T21:24:20.323390+0000 osd.2 (osd.2) 22 : cluster 0 6.14 scrub ok 2024-03-11T21:24:22.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:21 smithi203 bash[23654]: cluster 2024-03-11T21:24:20.320612+0000 osd.2 (osd.2) 21 : cluster 0 6.14 scrub starts 2024-03-11T21:24:22.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:21 smithi203 bash[23654]: cluster 2024-03-11T21:24:20.323390+0000 osd.2 (osd.2) 22 : cluster 0 6.14 scrub ok 2024-03-11T21:24:22.896 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.897 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.897 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.897 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.898 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.898 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.059458+0000 mgr.y (mgr.14550) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.430794+0000 osd.1 (osd.1) 23 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.433261+0000 osd.1 (osd.1) 24 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.553882+0000 osd.5 (osd.5) 19 : cluster 0 6.d scrub starts 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.556928+0000 osd.5 (osd.5) 20 : cluster 0 6.d scrub ok 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.881968+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.882108+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.988785+0000 osd.3 (osd.3) 23 : cluster 0 3.8 scrub starts 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[21512]: cluster 2024-03-11T21:24:21.993382+0000 osd.3 (osd.3) 24 : cluster 0 3.8 scrub ok 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:22.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.059458+0000 mgr.y (mgr.14550) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.430794+0000 osd.1 (osd.1) 23 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.433261+0000 osd.1 (osd.1) 24 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.553882+0000 osd.5 (osd.5) 19 : cluster 0 6.d scrub starts 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.556928+0000 osd.5 (osd.5) 20 : cluster 0 6.d scrub ok 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.881968+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.882108+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.988785+0000 osd.3 (osd.3) 23 : cluster 0 3.8 scrub starts 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 bash[29284]: cluster 2024-03-11T21:24:21.993382+0000 osd.3 (osd.3) 24 : cluster 0 3.8 scrub ok 2024-03-11T21:24:22.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.059458+0000 mgr.y (mgr.14550) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.430794+0000 osd.1 (osd.1) 23 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.433261+0000 osd.1 (osd.1) 24 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.553882+0000 osd.5 (osd.5) 19 : cluster 0 6.d scrub starts 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.556928+0000 osd.5 (osd.5) 20 : cluster 0 6.d scrub ok 2024-03-11T21:24:23.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.881968+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:23.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.882108+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:23.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.988785+0000 osd.3 (osd.3) 23 : cluster 0 3.8 scrub starts 2024-03-11T21:24:23.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:22 smithi203 bash[23654]: cluster 2024-03-11T21:24:21.993382+0000 osd.3 (osd.3) 24 : cluster 0 3.8 scrub ok 2024-03-11T21:24:23.187 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.187 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.187 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.187 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.187 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:22 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.187 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.188 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: Started Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:24:23.188 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.188 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.188 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.189 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:23.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:22.567743+0000 osd.5 (osd.5) 23 : cluster 0 3.16 scrub starts 2024-03-11T21:24:23.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:22.571629+0000 osd.5 (osd.5) 24 : cluster 0 3.16 scrub ok 2024-03-11T21:24:23.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:22.623901+0000 mon.a (mon.0) 812 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:24:23.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.079149+0000 osd.7 (osd.7) 15 : cluster 0 6.3 scrub starts 2024-03-11T21:24:23.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.082257+0000 osd.7 (osd.7) 16 : cluster 0 6.3 scrub ok 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: audit 2024-03-11T21:24:23.212657+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: audit 2024-03-11T21:24:23.220793+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: audit 2024-03-11T21:24:23.254893+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: audit 2024-03-11T21:24:23.259800+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: audit 2024-03-11T21:24:23.308485+0000 mon.c (mon.2) 47 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.544800+0000 mon.a (mon.0) 817 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:22.567743+0000 osd.5 (osd.5) 23 : cluster 0 3.16 scrub starts 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:22.571629+0000 osd.5 (osd.5) 24 : cluster 0 3.16 scrub ok 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:22.623901+0000 mon.a (mon.0) 812 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.079149+0000 osd.7 (osd.7) 15 : cluster 0 6.3 scrub starts 2024-03-11T21:24:23.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.082257+0000 osd.7 (osd.7) 16 : cluster 0 6.3 scrub ok 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: audit 2024-03-11T21:24:23.212657+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: audit 2024-03-11T21:24:23.220793+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: audit 2024-03-11T21:24:23.254893+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: audit 2024-03-11T21:24:23.259800+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: audit 2024-03-11T21:24:23.308485+0000 mon.c (mon.2) 47 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:23.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:23 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.544800+0000 mon.a (mon.0) 817 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:24:24.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:22.567743+0000 osd.5 (osd.5) 23 : cluster 0 3.16 scrub starts 2024-03-11T21:24:24.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:22.571629+0000 osd.5 (osd.5) 24 : cluster 0 3.16 scrub ok 2024-03-11T21:24:24.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:22.623901+0000 mon.a (mon.0) 812 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:24:24.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.079149+0000 osd.7 (osd.7) 15 : cluster 0 6.3 scrub starts 2024-03-11T21:24:24.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.082257+0000 osd.7 (osd.7) 16 : cluster 0 6.3 scrub ok 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: audit 2024-03-11T21:24:23.212657+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: audit 2024-03-11T21:24:23.220793+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: audit 2024-03-11T21:24:23.254893+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: audit 2024-03-11T21:24:23.259800+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: audit 2024-03-11T21:24:23.308485+0000 mon.c (mon.2) 47 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:24.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:23 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.544800+0000 mon.a (mon.0) 817 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:24:24.245 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:24.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.060133+0000 mgr.y (mgr.14550) 38 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:24.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.471866+0000 osd.1 (osd.1) 25 : cluster 0 2.a deep-scrub starts 2024-03-11T21:24:24.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.473309+0000 osd.1 (osd.1) 26 : cluster 0 2.a deep-scrub ok 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.576447+0000 osd.5 (osd.5) 25 : cluster 0 3.c scrub starts 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:23.584808+0000 osd.5 (osd.5) 26 : cluster 0 3.c scrub ok 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.040555+0000 osd.3 (osd.3) 25 : cluster 0 4.0 scrub starts 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.043642+0000 osd.3 (osd.3) 26 : cluster 0 4.0 scrub ok 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.071281+0000 osd.7 (osd.7) 17 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.073656+0000 osd.7 (osd.7) 18 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.401890+0000 osd.4 (osd.4) 13 : cluster 0 5.18 scrub starts 2024-03-11T21:24:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.404078+0000 osd.4 (osd.4) 14 : cluster 0 5.18 scrub ok 2024-03-11T21:24:24.938 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.660Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.660Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.662Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.143 port=9094 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.663Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.693Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.694Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.697Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:24:24.939 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[58551]: ts=2024-03-11T21:24:24.697Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:24:24.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.060133+0000 mgr.y (mgr.14550) 38 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:24.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.471866+0000 osd.1 (osd.1) 25 : cluster 0 2.a deep-scrub starts 2024-03-11T21:24:24.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.473309+0000 osd.1 (osd.1) 26 : cluster 0 2.a deep-scrub ok 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.576447+0000 osd.5 (osd.5) 25 : cluster 0 3.c scrub starts 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:23.584808+0000 osd.5 (osd.5) 26 : cluster 0 3.c scrub ok 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.040555+0000 osd.3 (osd.3) 25 : cluster 0 4.0 scrub starts 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.043642+0000 osd.3 (osd.3) 26 : cluster 0 4.0 scrub ok 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.071281+0000 osd.7 (osd.7) 17 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.073656+0000 osd.7 (osd.7) 18 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.401890+0000 osd.4 (osd.4) 13 : cluster 0 5.18 scrub starts 2024-03-11T21:24:24.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:24 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.404078+0000 osd.4 (osd.4) 14 : cluster 0 5.18 scrub ok 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.060133+0000 mgr.y (mgr.14550) 38 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.471866+0000 osd.1 (osd.1) 25 : cluster 0 2.a deep-scrub starts 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.473309+0000 osd.1 (osd.1) 26 : cluster 0 2.a deep-scrub ok 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.576447+0000 osd.5 (osd.5) 25 : cluster 0 3.c scrub starts 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:23.584808+0000 osd.5 (osd.5) 26 : cluster 0 3.c scrub ok 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.040555+0000 osd.3 (osd.3) 25 : cluster 0 4.0 scrub starts 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.043642+0000 osd.3 (osd.3) 26 : cluster 0 4.0 scrub ok 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.071281+0000 osd.7 (osd.7) 17 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:24:25.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.073656+0000 osd.7 (osd.7) 18 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:24:25.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.401890+0000 osd.4 (osd.4) 13 : cluster 0 5.18 scrub starts 2024-03-11T21:24:25.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:24 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.404078+0000 osd.4 (osd.4) 14 : cluster 0 5.18 scrub ok 2024-03-11T21:24:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.208338+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-03-11T21:24:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.210050+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-03-11T21:24:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.457367+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.457462+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.482585+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.486290+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.490027+0000 osd.1 (osd.1) 29 : cluster 0 4.2 scrub starts 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.493338+0000 osd.1 (osd.1) 30 : cluster 0 4.2 scrub ok 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:24.630072+0000 mon.a (mon.0) 818 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.030645+0000 osd.3 (osd.3) 27 : cluster 0 5.14 scrub starts 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.034236+0000 osd.3 (osd.3) 28 : cluster 0 5.14 scrub ok 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.208338+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.210050+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.457367+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.457462+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.482585+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.486290+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.490027+0000 osd.1 (osd.1) 29 : cluster 0 4.2 scrub starts 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.493338+0000 osd.1 (osd.1) 30 : cluster 0 4.2 scrub ok 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:24.630072+0000 mon.a (mon.0) 818 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.030645+0000 osd.3 (osd.3) 27 : cluster 0 5.14 scrub starts 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:25 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.034236+0000 osd.3 (osd.3) 28 : cluster 0 5.14 scrub ok 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.208338+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.210050+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.457367+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:25.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.457462+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.482585+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.486290+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.490027+0000 osd.1 (osd.1) 29 : cluster 0 4.2 scrub starts 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.493338+0000 osd.1 (osd.1) 30 : cluster 0 4.2 scrub ok 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:24.630072+0000 mon.a (mon.0) 818 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.030645+0000 osd.3 (osd.3) 27 : cluster 0 5.14 scrub starts 2024-03-11T21:24:25.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:25 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.034236+0000 osd.3 (osd.3) 28 : cluster 0 5.14 scrub ok 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4023371332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0106050 msgr2=0x7f26b010dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4023371332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0106050 0x7f26b010dc20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f26a800b0a0 tx=0x7f26a800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4023371332 shutdown_connections 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4023371332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0106050 0x7f26b010dc20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4023371332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0100f80 0x7f26b0101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4023371332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0101930 0x7f26b0105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4023371332 >> 172.21.15.203:0/4023371332 conn(0x7f26b0077e30 msgr2=0x7f26b0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4023371332 shutdown_connections 2024-03-11T21:24:26.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4023371332 wait complete. 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 librados: starting msgr at 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.809+0000 7f26b5e1b700 1 librados: starting objecter 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 Processor -- start 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- start start 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 librados: setting wanted keys 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 librados: calling monclient init 2024-03-11T21:24:26.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0101930 0x7f26b0109800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0106050 0x7f26b0109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b010f490 con 0x7f26b0106050 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b010f310 con 0x7f26b0100f80 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b010f610 con 0x7f26b0101930 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:42388/0 (socket says 172.21.15.203:42388) 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 -- 172.21.15.203:0/4087925932 learned_addr learned my addr 172.21.15.203:0/4087925932 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26a6ffd700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0101930 0x7f26b0109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26affff700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0106050 0x7f26b0109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 -- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0101930 msgr2=0x7f26b0109800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0101930 0x7f26b0109800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 -- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0106050 msgr2=0x7f26b0109d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0106050 0x7f26b0109d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 -- 172.21.15.203:0/4087925932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b010a300 con 0x7f26b0100f80 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26af7fe700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26a000c9b0 tx=0x7f26a000cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 10 client.?.objecter ms_handle_connect 0x7f26b0100f80 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0007b00 con 0x7f26b0100f80 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b010a5f0 con 0x7f26b0100f80 2024-03-11T21:24:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f26a0007ca0 con 0x7f26b0100f80 2024-03-11T21:24:26.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0018660 con 0x7f26b0100f80 2024-03-11T21:24:26.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 10 client.24466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:26.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b01aac70 con 0x7f26b0100f80 2024-03-11T21:24:26.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 librados: init done 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b0102450 con 0x7f26b0100f80 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.813+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f26a000f450 con 0x7f26b0100f80 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 0x7f269c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(103..103 src has 21..103) v4 ==== 7421+0+0 (secure 0 0 0) 0x7f26a008e270 con 0x7f26b0100f80 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 10 client.24466.objecter ms_dispatch 0x7f26b0001550 osd_map(103..103 src has 21..103) v4 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 3 client.24466.objecter handle_osd_map got epochs [103,103] > 0 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 3 client.24466.objecter handle_osd_map decoding full epoch 103 2024-03-11T21:24:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 20 client.24466.objecter dump_active .. 0 homeless 2024-03-11T21:24:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26a6ffd700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 0x7f269c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26ad7fa700 10 client.24466.objecter ms_handle_connect 0x7f269c06f820 2024-03-11T21:24:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.817+0000 7f26a6ffd700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 0x7f269c071ce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2698005fd0 tx=0x7f2698005db0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:26.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:26.821+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f26a0014030 con 0x7f26b0100f80 2024-03-11T21:24:26.937 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[58551]: ts=2024-03-11T21:24:26.664Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000647573s 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.060750+0000 mgr.y (mgr.14550) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.171786+0000 osd.2 (osd.2) 25 : cluster 0 6.f scrub starts 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.174104+0000 osd.2 (osd.2) 26 : cluster 0 6.f scrub ok 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.458156+0000 osd.1 (osd.1) 31 : cluster 0 3.19 scrub starts 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.462320+0000 osd.1 (osd.1) 32 : cluster 0 3.19 scrub ok 2024-03-11T21:24:26.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.473997+0000 osd.0 (osd.0) 19 : cluster 0 6.0 scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:25.476450+0000 osd.0 (osd.0) 20 : cluster 0 6.0 scrub ok 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:26.023089+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[21512]: cluster 2024-03-11T21:24:26.024914+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.060750+0000 mgr.y (mgr.14550) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.171786+0000 osd.2 (osd.2) 25 : cluster 0 6.f scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.174104+0000 osd.2 (osd.2) 26 : cluster 0 6.f scrub ok 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.458156+0000 osd.1 (osd.1) 31 : cluster 0 3.19 scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.462320+0000 osd.1 (osd.1) 32 : cluster 0 3.19 scrub ok 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.473997+0000 osd.0 (osd.0) 19 : cluster 0 6.0 scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:25.476450+0000 osd.0 (osd.0) 20 : cluster 0 6.0 scrub ok 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:26.023089+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-03-11T21:24:26.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:26 smithi143 bash[29284]: cluster 2024-03-11T21:24:26.024914+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.060750+0000 mgr.y (mgr.14550) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.171786+0000 osd.2 (osd.2) 25 : cluster 0 6.f scrub starts 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.174104+0000 osd.2 (osd.2) 26 : cluster 0 6.f scrub ok 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.458156+0000 osd.1 (osd.1) 31 : cluster 0 3.19 scrub starts 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.462320+0000 osd.1 (osd.1) 32 : cluster 0 3.19 scrub ok 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.473997+0000 osd.0 (osd.0) 19 : cluster 0 6.0 scrub starts 2024-03-11T21:24:27.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:25.476450+0000 osd.0 (osd.0) 20 : cluster 0 6.0 scrub ok 2024-03-11T21:24:27.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:26.023089+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-03-11T21:24:27.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:26 smithi203 bash[23654]: cluster 2024-03-11T21:24:26.024914+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-03-11T21:24:27.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.117+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f26b010ace0 con 0x7f26b0100f80 2024-03-11T21:24:27.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26ad7fa700 1 -- 172.21.15.203:0/4087925932 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (secure 0 0 0) 0x7f26a005b7b0 con 0x7f26b0100f80 2024-03-11T21:24:27.112 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-03-11T21:24:27.112 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCLdu9lLre5BxAA8ZgeludqkSvvSBeAt5my2Q== 2024-03-11T21:24:27.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 10 librados: watch_flush enter 2024-03-11T21:24:27.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 10 librados: watch_flush exit 2024-03-11T21:24:27.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 20 client.24466.objecter shutdown clearing up homeless session... 2024-03-11T21:24:27.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 10 client.24466.objecter successfully canceled tick 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 msgr2=0x7f269c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 0x7f269c071ce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2698005fd0 tx=0x7f2698005db0 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 msgr2=0x7f26b010d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26a000c9b0 tx=0x7f26a000cd70 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 shutdown_connections 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f269c06f820 0x7f269c071ce0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b0106050 0x7f26b0109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26b0100f80 0x7f26b010d710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 --2- 172.21.15.203:0/4087925932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26b0101930 0x7f26b0109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 >> 172.21.15.203:0/4087925932 conn(0x7f26b0077e30 msgr2=0x7f26b00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 shutdown_connections 2024-03-11T21:24:27.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 -- 172.21.15.203:0/4087925932 wait complete. 2024-03-11T21:24:27.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-11T21:24:27.129+0000 7f26b5e1b700 1 librados: shutdown 2024-03-11T21:24:27.744 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-11T21:24:27.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:24:27.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:24:27.767 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:24:27.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:24:27.767 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: cluster 2024-03-11T21:24:26.569581+0000 osd.5 (osd.5) 27 : cluster 0 2.16 scrub starts 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: cluster 2024-03-11T21:24:26.571772+0000 osd.5 (osd.5) 28 : cluster 0 2.16 scrub ok 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: audit 2024-03-11T21:24:27.123929+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.203:0/4087925932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: audit 2024-03-11T21:24:27.129237+0000 mon.a (mon.0) 819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: audit 2024-03-11T21:24:27.134248+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: audit 2024-03-11T21:24:27.164334+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[21512]: audit 2024-03-11T21:24:27.484852+0000 mon.c (mon.2) 48 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: cluster 2024-03-11T21:24:26.569581+0000 osd.5 (osd.5) 27 : cluster 0 2.16 scrub starts 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: cluster 2024-03-11T21:24:26.571772+0000 osd.5 (osd.5) 28 : cluster 0 2.16 scrub ok 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: audit 2024-03-11T21:24:27.123929+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.203:0/4087925932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: audit 2024-03-11T21:24:27.129237+0000 mon.a (mon.0) 819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: audit 2024-03-11T21:24:27.134248+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: audit 2024-03-11T21:24:27.164334+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:27 smithi143 bash[29284]: audit 2024-03-11T21:24:27.484852+0000 mon.c (mon.2) 48 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:28.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: cluster 2024-03-11T21:24:26.569581+0000 osd.5 (osd.5) 27 : cluster 0 2.16 scrub starts 2024-03-11T21:24:28.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: cluster 2024-03-11T21:24:26.571772+0000 osd.5 (osd.5) 28 : cluster 0 2.16 scrub ok 2024-03-11T21:24:28.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: audit 2024-03-11T21:24:27.123929+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.203:0/4087925932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:28.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: audit 2024-03-11T21:24:27.129237+0000 mon.a (mon.0) 819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:24:28.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: audit 2024-03-11T21:24:27.134248+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:24:28.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: audit 2024-03-11T21:24:27.164334+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:28.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:27 smithi203 bash[23654]: audit 2024-03-11T21:24:27.484852+0000 mon.c (mon.2) 48 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:28.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:27.061394+0000 mgr.y (mgr.14550) 40 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:27.409863+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:27.412261+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:27.568236+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:27.570217+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.089741+0000 osd.7 (osd.7) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.089851+0000 osd.7 (osd.7) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.158102+0000 osd.2 (osd.2) 27 : cluster 0 5.6 scrub starts 2024-03-11T21:24:28.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.160848+0000 osd.2 (osd.2) 28 : cluster 0 5.6 scrub ok 2024-03-11T21:24:28.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.543490+0000 mon.a (mon.0) 822 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:24:28.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:27.061394+0000 mgr.y (mgr.14550) 40 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:28.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:27.409863+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T21:24:28.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:27.412261+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:27.568236+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:27.570217+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.089741+0000 osd.7 (osd.7) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.089851+0000 osd.7 (osd.7) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.158102+0000 osd.2 (osd.2) 27 : cluster 0 5.6 scrub starts 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.160848+0000 osd.2 (osd.2) 28 : cluster 0 5.6 scrub ok 2024-03-11T21:24:28.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:28 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.543490+0000 mon.a (mon.0) 822 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:27.061394+0000 mgr.y (mgr.14550) 40 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:27.409863+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:27.412261+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:27.568236+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:27.570217+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-03-11T21:24:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.089741+0000 osd.7 (osd.7) 21 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:29.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.089851+0000 osd.7 (osd.7) 22 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:29.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.158102+0000 osd.2 (osd.2) 27 : cluster 0 5.6 scrub starts 2024-03-11T21:24:29.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.160848+0000 osd.2 (osd.2) 28 : cluster 0 5.6 scrub ok 2024-03-11T21:24:29.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:28 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.543490+0000 mon.a (mon.0) 822 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE Bus STOPPING 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE Bus STOPPED 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE Bus STARTING 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE Serving on http://:::9283 2024-03-11T21:24:29.437 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21768]: [11/Mar/2024:21:24:29] ENGINE Bus STARTED 2024-03-11T21:24:29.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:24:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.076313+0000 mgr.y (mgr.14550) 41 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.524775+0000 osd.0 (osd.0) 21 : cluster 0 6.10 scrub starts 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.526368+0000 osd.0 (osd.0) 22 : cluster 0 6.10 scrub ok 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.545251+0000 osd.5 (osd.5) 29 : cluster 0 2.3 scrub starts 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:28.549100+0000 osd.5 (osd.5) 30 : cluster 0 2.3 scrub ok 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.948743+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.957693+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.971162+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.979524+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.982148+0000 mon.c (mon.2) 49 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.983257+0000 mon.c (mon.2) 50 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:28.993832+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.048855+0000 osd.7 (osd.7) 23 : cluster 0 6.15 scrub starts 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.051408+0000 osd.7 (osd.7) 24 : cluster 0 6.15 scrub ok 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:29.068864+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:29.080032+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:29.087157+0000 mon.c (mon.2) 51 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: audit 2024-03-11T21:24:29.099660+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.188140+0000 osd.2 (osd.2) 29 : cluster 0 2.1 scrub starts 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.189422+0000 osd.2 (osd.2) 30 : cluster 0 2.1 scrub ok 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.471278+0000 osd.4 (osd.4) 15 : cluster 0 3.3 scrub starts 2024-03-11T21:24:29.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:29 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.475274+0000 osd.4 (osd.4) 16 : cluster 0 3.3 scrub ok 2024-03-11T21:24:29.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.076313+0000 mgr.y (mgr.14550) 41 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.524775+0000 osd.0 (osd.0) 21 : cluster 0 6.10 scrub starts 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.526368+0000 osd.0 (osd.0) 22 : cluster 0 6.10 scrub ok 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.545251+0000 osd.5 (osd.5) 29 : cluster 0 2.3 scrub starts 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:28.549100+0000 osd.5 (osd.5) 30 : cluster 0 2.3 scrub ok 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.948743+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.957693+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.971162+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.979524+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.982148+0000 mon.c (mon.2) 49 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.983257+0000 mon.c (mon.2) 50 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:28.993832+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.048855+0000 osd.7 (osd.7) 23 : cluster 0 6.15 scrub starts 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.051408+0000 osd.7 (osd.7) 24 : cluster 0 6.15 scrub ok 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:29.068864+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:29.080032+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:29.087157+0000 mon.c (mon.2) 51 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: audit 2024-03-11T21:24:29.099660+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.188140+0000 osd.2 (osd.2) 29 : cluster 0 2.1 scrub starts 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.189422+0000 osd.2 (osd.2) 30 : cluster 0 2.1 scrub ok 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.471278+0000 osd.4 (osd.4) 15 : cluster 0 3.3 scrub starts 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.475274+0000 osd.4 (osd.4) 16 : cluster 0 3.3 scrub ok 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.076313+0000 mgr.y (mgr.14550) 41 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.524775+0000 osd.0 (osd.0) 21 : cluster 0 6.10 scrub starts 2024-03-11T21:24:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.526368+0000 osd.0 (osd.0) 22 : cluster 0 6.10 scrub ok 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.545251+0000 osd.5 (osd.5) 29 : cluster 0 2.3 scrub starts 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:28.549100+0000 osd.5 (osd.5) 30 : cluster 0 2.3 scrub ok 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.948743+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.957693+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.971162+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.979524+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.982148+0000 mon.c (mon.2) 49 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.983257+0000 mon.c (mon.2) 50 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:28.993832+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.048855+0000 osd.7 (osd.7) 23 : cluster 0 6.15 scrub starts 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.051408+0000 osd.7 (osd.7) 24 : cluster 0 6.15 scrub ok 2024-03-11T21:24:29.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:29.068864+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:29.080032+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:29.087157+0000 mon.c (mon.2) 51 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: audit 2024-03-11T21:24:29.099660+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.188140+0000 osd.2 (osd.2) 29 : cluster 0 2.1 scrub starts 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.189422+0000 osd.2 (osd.2) 30 : cluster 0 2.1 scrub ok 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.471278+0000 osd.4 (osd.4) 15 : cluster 0 3.3 scrub starts 2024-03-11T21:24:29.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:29 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.475274+0000 osd.4 (osd.4) 16 : cluster 0 3.3 scrub ok 2024-03-11T21:24:30.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cephadm 2024-03-11T21:24:29.009987+0000 mgr.y (mgr.14550) 42 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:24:30.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.061875+0000 mgr.y (mgr.14550) 43 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: audit 2024-03-11T21:24:29.088572+0000 mgr.y (mgr.14550) 44 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cephadm 2024-03-11T21:24:29.114900+0000 mgr.y (mgr.14550) 45 : cephadm 1 Deploying daemon grafana.a on smithi203 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.502830+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:29.506560+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.100310+0000 osd.3 (osd.3) 29 : cluster 0 3.2 scrub starts 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.104984+0000 osd.3 (osd.3) 30 : cluster 0 3.2 scrub ok 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.442524+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.445416+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cephadm 2024-03-11T21:24:29.009987+0000 mgr.y (mgr.14550) 42 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:24:30.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.061875+0000 mgr.y (mgr.14550) 43 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: audit 2024-03-11T21:24:29.088572+0000 mgr.y (mgr.14550) 44 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cephadm 2024-03-11T21:24:29.114900+0000 mgr.y (mgr.14550) 45 : cephadm 1 Deploying daemon grafana.a on smithi203 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.502830+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:29.506560+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.100310+0000 osd.3 (osd.3) 29 : cluster 0 3.2 scrub starts 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.104984+0000 osd.3 (osd.3) 30 : cluster 0 3.2 scrub ok 2024-03-11T21:24:30.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.442524+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-03-11T21:24:30.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:30 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.445416+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-03-11T21:24:31.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cephadm 2024-03-11T21:24:29.009987+0000 mgr.y (mgr.14550) 42 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:24:31.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.061875+0000 mgr.y (mgr.14550) 43 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:31.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: audit 2024-03-11T21:24:29.088572+0000 mgr.y (mgr.14550) 44 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:24:31.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cephadm 2024-03-11T21:24:29.114900+0000 mgr.y (mgr.14550) 45 : cephadm 1 Deploying daemon grafana.a on smithi203 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.502830+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:29.506560+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.100310+0000 osd.3 (osd.3) 29 : cluster 0 3.2 scrub starts 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.104984+0000 osd.3 (osd.3) 30 : cluster 0 3.2 scrub ok 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.442524+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-03-11T21:24:31.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:30 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.445416+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-03-11T21:24:31.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.468159+0000 osd.6 (osd.6) 15 : cluster 0 3.a deep-scrub starts 2024-03-11T21:24:31.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.484218+0000 osd.6 (osd.6) 16 : cluster 0 3.a deep-scrub ok 2024-03-11T21:24:31.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.546685+0000 osd.0 (osd.0) 23 : cluster 0 6.16 scrub starts 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:30.549165+0000 osd.0 (osd.0) 24 : cluster 0 6.16 scrub ok 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:31.217362+0000 osd.2 (osd.2) 31 : cluster 0 5.8 scrub starts 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[29284]: cluster 2024-03-11T21:24:31.219868+0000 osd.2 (osd.2) 32 : cluster 0 5.8 scrub ok 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.468159+0000 osd.6 (osd.6) 15 : cluster 0 3.a deep-scrub starts 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.484218+0000 osd.6 (osd.6) 16 : cluster 0 3.a deep-scrub ok 2024-03-11T21:24:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.546685+0000 osd.0 (osd.0) 23 : cluster 0 6.16 scrub starts 2024-03-11T21:24:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:30.549165+0000 osd.0 (osd.0) 24 : cluster 0 6.16 scrub ok 2024-03-11T21:24:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:31.217362+0000 osd.2 (osd.2) 31 : cluster 0 5.8 scrub starts 2024-03-11T21:24:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:31 smithi143 bash[21512]: cluster 2024-03-11T21:24:31.219868+0000 osd.2 (osd.2) 32 : cluster 0 5.8 scrub ok 2024-03-11T21:24:32.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.468159+0000 osd.6 (osd.6) 15 : cluster 0 3.a deep-scrub starts 2024-03-11T21:24:32.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.484218+0000 osd.6 (osd.6) 16 : cluster 0 3.a deep-scrub ok 2024-03-11T21:24:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.546685+0000 osd.0 (osd.0) 23 : cluster 0 6.16 scrub starts 2024-03-11T21:24:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:30.549165+0000 osd.0 (osd.0) 24 : cluster 0 6.16 scrub ok 2024-03-11T21:24:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:31.217362+0000 osd.2 (osd.2) 31 : cluster 0 5.8 scrub starts 2024-03-11T21:24:32.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:31 smithi203 bash[23654]: cluster 2024-03-11T21:24:31.219868+0000 osd.2 (osd.2) 32 : cluster 0 5.8 scrub ok 2024-03-11T21:24:32.366 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[21512]: cluster 2024-03-11T21:24:31.062555+0000 mgr.y (mgr.14550) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.024918+0000 osd.7 (osd.7) 25 : cluster 0 3.13 scrub starts 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.028750+0000 osd.7 (osd.7) 26 : cluster 0 3.13 scrub ok 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.439605+0000 osd.4 (osd.4) 19 : cluster 0 6.1e scrub starts 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.442043+0000 osd.4 (osd.4) 20 : cluster 0 6.1e scrub ok 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[29284]: cluster 2024-03-11T21:24:31.062555+0000 mgr.y (mgr.14550) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.024918+0000 osd.7 (osd.7) 25 : cluster 0 3.13 scrub starts 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.028750+0000 osd.7 (osd.7) 26 : cluster 0 3.13 scrub ok 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.439605+0000 osd.4 (osd.4) 19 : cluster 0 6.1e scrub starts 2024-03-11T21:24:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:32 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.442043+0000 osd.4 (osd.4) 20 : cluster 0 6.1e scrub ok 2024-03-11T21:24:33.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:32 smithi203 bash[23654]: cluster 2024-03-11T21:24:31.062555+0000 mgr.y (mgr.14550) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-03-11T21:24:33.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:32 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.024918+0000 osd.7 (osd.7) 25 : cluster 0 3.13 scrub starts 2024-03-11T21:24:33.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:32 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.028750+0000 osd.7 (osd.7) 26 : cluster 0 3.13 scrub ok 2024-03-11T21:24:33.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:32 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.439605+0000 osd.4 (osd.4) 19 : cluster 0 6.1e scrub starts 2024-03-11T21:24:33.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:32 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.442043+0000 osd.4 (osd.4) 20 : cluster 0 6.1e scrub ok 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.476817+0000 osd.6 (osd.6) 17 : cluster 0 2.14 scrub starts 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.479132+0000 osd.6 (osd.6) 18 : cluster 0 2.14 scrub ok 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.546983+0000 osd.0 (osd.0) 25 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:32.552320+0000 osd.0 (osd.0) 26 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.068645+0000 osd.7 (osd.7) 27 : cluster 0 3.e deep-scrub starts 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.079611+0000 osd.7 (osd.7) 28 : cluster 0 3.e deep-scrub ok 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.139088+0000 osd.3 (osd.3) 31 : cluster 0 5.13 scrub starts 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.141530+0000 osd.3 (osd.3) 32 : cluster 0 5.13 scrub ok 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.468581+0000 osd.4 (osd.4) 21 : cluster 0 6.2 scrub starts 2024-03-11T21:24:33.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.471007+0000 osd.4 (osd.4) 22 : cluster 0 6.2 scrub ok 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.476817+0000 osd.6 (osd.6) 17 : cluster 0 2.14 scrub starts 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.479132+0000 osd.6 (osd.6) 18 : cluster 0 2.14 scrub ok 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.546983+0000 osd.0 (osd.0) 25 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:32.552320+0000 osd.0 (osd.0) 26 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.068645+0000 osd.7 (osd.7) 27 : cluster 0 3.e deep-scrub starts 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.079611+0000 osd.7 (osd.7) 28 : cluster 0 3.e deep-scrub ok 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.139088+0000 osd.3 (osd.3) 31 : cluster 0 5.13 scrub starts 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.141530+0000 osd.3 (osd.3) 32 : cluster 0 5.13 scrub ok 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.468581+0000 osd.4 (osd.4) 21 : cluster 0 6.2 scrub starts 2024-03-11T21:24:33.928 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:33 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.471007+0000 osd.4 (osd.4) 22 : cluster 0 6.2 scrub ok 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.476817+0000 osd.6 (osd.6) 17 : cluster 0 2.14 scrub starts 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.479132+0000 osd.6 (osd.6) 18 : cluster 0 2.14 scrub ok 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.546983+0000 osd.0 (osd.0) 25 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:32.552320+0000 osd.0 (osd.0) 26 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.068645+0000 osd.7 (osd.7) 27 : cluster 0 3.e deep-scrub starts 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.079611+0000 osd.7 (osd.7) 28 : cluster 0 3.e deep-scrub ok 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.139088+0000 osd.3 (osd.3) 31 : cluster 0 5.13 scrub starts 2024-03-11T21:24:34.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.141530+0000 osd.3 (osd.3) 32 : cluster 0 5.13 scrub ok 2024-03-11T21:24:34.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.468581+0000 osd.4 (osd.4) 21 : cluster 0 6.2 scrub starts 2024-03-11T21:24:34.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:33 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.471007+0000 osd.4 (osd.4) 22 : cluster 0 6.2 scrub ok 2024-03-11T21:24:34.937 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[58551]: ts=2024-03-11T21:24:34.665Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002095605s 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.063917+0000 mgr.y (mgr.14550) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.491486+0000 osd.5 (osd.5) 33 : cluster 0 5.1b scrub starts 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:33.494076+0000 osd.5 (osd.5) 34 : cluster 0 5.1b scrub ok 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:34.052510+0000 osd.7 (osd.7) 29 : cluster 0 2.5 scrub starts 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:34.054847+0000 osd.7 (osd.7) 30 : cluster 0 2.5 scrub ok 2024-03-11T21:24:34.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:34.481175+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[21512]: cluster 2024-03-11T21:24:34.483294+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.063917+0000 mgr.y (mgr.14550) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.491486+0000 osd.5 (osd.5) 33 : cluster 0 5.1b scrub starts 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:33.494076+0000 osd.5 (osd.5) 34 : cluster 0 5.1b scrub ok 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:34.052510+0000 osd.7 (osd.7) 29 : cluster 0 2.5 scrub starts 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:34.054847+0000 osd.7 (osd.7) 30 : cluster 0 2.5 scrub ok 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:34.481175+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T21:24:34.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:34 smithi143 bash[29284]: cluster 2024-03-11T21:24:34.483294+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.063917+0000 mgr.y (mgr.14550) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.491486+0000 osd.5 (osd.5) 33 : cluster 0 5.1b scrub starts 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:33.494076+0000 osd.5 (osd.5) 34 : cluster 0 5.1b scrub ok 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:34.052510+0000 osd.7 (osd.7) 29 : cluster 0 2.5 scrub starts 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:34.054847+0000 osd.7 (osd.7) 30 : cluster 0 2.5 scrub ok 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:34.481175+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T21:24:35.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:34 smithi203 bash[23654]: cluster 2024-03-11T21:24:34.483294+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.073254+0000 osd.7 (osd.7) 31 : cluster 0 2.0 scrub starts 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.076068+0000 osd.7 (osd.7) 32 : cluster 0 2.0 scrub ok 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.350176+0000 osd.1 (osd.1) 35 : cluster 0 2.6 scrub starts 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.353152+0000 osd.1 (osd.1) 36 : cluster 0 2.6 scrub ok 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.488783+0000 osd.4 (osd.4) 25 : cluster 0 4.1 scrub starts 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.491318+0000 osd.4 (osd.4) 26 : cluster 0 4.1 scrub ok 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.073254+0000 osd.7 (osd.7) 31 : cluster 0 2.0 scrub starts 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.076068+0000 osd.7 (osd.7) 32 : cluster 0 2.0 scrub ok 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.350176+0000 osd.1 (osd.1) 35 : cluster 0 2.6 scrub starts 2024-03-11T21:24:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.353152+0000 osd.1 (osd.1) 36 : cluster 0 2.6 scrub ok 2024-03-11T21:24:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.488783+0000 osd.4 (osd.4) 25 : cluster 0 4.1 scrub starts 2024-03-11T21:24:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:35 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.491318+0000 osd.4 (osd.4) 26 : cluster 0 4.1 scrub ok 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.073254+0000 osd.7 (osd.7) 31 : cluster 0 2.0 scrub starts 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.076068+0000 osd.7 (osd.7) 32 : cluster 0 2.0 scrub ok 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.350176+0000 osd.1 (osd.1) 35 : cluster 0 2.6 scrub starts 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.353152+0000 osd.1 (osd.1) 36 : cluster 0 2.6 scrub ok 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.488783+0000 osd.4 (osd.4) 25 : cluster 0 4.1 scrub starts 2024-03-11T21:24:36.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:35 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.491318+0000 osd.4 (osd.4) 26 : cluster 0 4.1 scrub ok 2024-03-11T21:24:36.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/2357533472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 msgr2=0x7fe7ac10ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:36.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/2357533472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac10ed00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009fc0 tx=0x7fe7a400b330 comp rx=0 tx=0).stop 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/2357533472 shutdown_connections 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/2357533472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac10ed00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/2357533472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/2357533472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 0x7fe7ac106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/2357533472 >> 172.21.15.143:0/2357533472 conn(0x7fe7ac0791b0 msgr2=0x7fe7ac0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/2357533472 shutdown_connections 2024-03-11T21:24:36.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/2357533472 wait complete. 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 librados: starting msgr at 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 librados: starting objecter 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 Processor -- start 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- start start 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 librados: setting wanted keys 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 librados: calling monclient init 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 0x7fe7ac074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac111650 con 0x7fe7ac107130 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1114d0 con 0x7fe7ac102a00 2024-03-11T21:24:36.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1117d0 con 0x7fe7ac102050 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47908/0 (socket says 172.21.15.143:47908) 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 -- 172.21.15.143:0/825657652 learned_addr learned my addr 172.21.15.143:0/825657652 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b2410700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b140e700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 0x7fe7ac074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 -- 172.21.15.143:0/825657652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 msgr2=0x7fe7ac074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 0x7fe7ac074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 -- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 msgr2=0x7fe7ac074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 -- 172.21.15.143:0/825657652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac1a60c0 con 0x7fe7ac102050 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b2410700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b1c0f700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800ec10 tx=0x7fe7a800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:36.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe79effd700 10 client.?.objecter ms_handle_connect 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe79effd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800cd40 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac1a63b0 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe7a80044a0 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8010430 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.067+0000 7fe7b3e73700 10 client.24476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac1a68f0 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe7b3e73700 1 librados: init done 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac04faa0 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe7a8003fb0 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 0x7fe798071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(104..104 src has 21..104) v4 ==== 7421+0+0 (secure 0 0 0) 0x7fe7a808e620 con 0x7fe7ac102050 2024-03-11T21:24:36.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe7b140e700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 0x7fe798071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 10 client.24476.objecter ms_dispatch 0x7fe7ac001550 osd_map(104..104 src has 21..104) v4 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 3 client.24476.objecter handle_osd_map got epochs [104,104] > 0 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 3 client.24476.objecter handle_osd_map decoding full epoch 104 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 20 client.24476.objecter dump_active .. 0 homeless 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe79effd700 10 client.24476.objecter ms_handle_connect 0x7fe79806f820 2024-03-11T21:24:36.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.071+0000 7fe7b140e700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 0x7fe798071ce0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0005950 tx=0x7fe7a00058e0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:36.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.075+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe7a8016330 con 0x7fe7ac102050 2024-03-11T21:24:36.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.347+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe7ac1a6bb0 con 0x7fe7ac102050 2024-03-11T21:24:36.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.355+0000 7fe79effd700 1 -- 172.21.15.143:0/825657652 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) v1 ==== 74+0+175616 (secure 0 0 0) 0x7fe7a805bb60 con 0x7fe7ac102050 2024-03-11T21:24:36.335 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 10 librados: watch_flush enter 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 10 librados: watch_flush exit 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 20 client.24476.objecter shutdown clearing up homeless session... 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 10 client.24476.objecter successfully canceled tick 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 msgr2=0x7fe798071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 0x7fe798071ce0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0005950 tx=0x7fe7a00058e0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 msgr2=0x7fe7ac0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800ec10 tx=0x7fe7a800ef20 comp rx=0 tx=0).stop 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 shutdown_connections 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe79806f820 0x7fe798071ce0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7ac107130 0x7fe7ac074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7ac102050 0x7fe7ac0740e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 --2- 172.21.15.143:0/825657652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7ac102a00 0x7fe7ac074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:36.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 >> 172.21.15.143:0/825657652 conn(0x7fe7ac0791b0 msgr2=0x7fe7ac0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:36.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 shutdown_connections 2024-03-11T21:24:36.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 -- 172.21.15.143:0/825657652 wait complete. 2024-03-11T21:24:36.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:36.359+0000 7fe7b3e73700 1 librados: shutdown 2024-03-11T21:24:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.064605+0000 mgr.y (mgr.14550) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.541415+0000 osd.6 (osd.6) 19 : cluster 0 2.11 scrub starts 2024-03-11T21:24:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.543570+0000 osd.6 (osd.6) 20 : cluster 0 2.11 scrub ok 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.733238+0000 osd.2 (osd.2) 33 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:35.733350+0000 osd.2 (osd.2) 34 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:36.085380+0000 osd.3 (osd.3) 33 : cluster 0 3.7 scrub starts 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:36.087702+0000 osd.3 (osd.3) 34 : cluster 0 3.7 scrub ok 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:36.324355+0000 osd.1 (osd.1) 37 : cluster 0 5.c scrub starts 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: cluster 2024-03-11T21:24:36.326554+0000 osd.1 (osd.1) 38 : cluster 0 5.c scrub ok 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[21512]: audit 2024-03-11T21:24:36.350730+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.143:0/825657652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.064605+0000 mgr.y (mgr.14550) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.541415+0000 osd.6 (osd.6) 19 : cluster 0 2.11 scrub starts 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.543570+0000 osd.6 (osd.6) 20 : cluster 0 2.11 scrub ok 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.733238+0000 osd.2 (osd.2) 33 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:35.733350+0000 osd.2 (osd.2) 34 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:36.085380+0000 osd.3 (osd.3) 33 : cluster 0 3.7 scrub starts 2024-03-11T21:24:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:36.087702+0000 osd.3 (osd.3) 34 : cluster 0 3.7 scrub ok 2024-03-11T21:24:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:36.324355+0000 osd.1 (osd.1) 37 : cluster 0 5.c scrub starts 2024-03-11T21:24:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: cluster 2024-03-11T21:24:36.326554+0000 osd.1 (osd.1) 38 : cluster 0 5.c scrub ok 2024-03-11T21:24:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:36 smithi143 bash[29284]: audit 2024-03-11T21:24:36.350730+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.143:0/825657652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.064605+0000 mgr.y (mgr.14550) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.541415+0000 osd.6 (osd.6) 19 : cluster 0 2.11 scrub starts 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.543570+0000 osd.6 (osd.6) 20 : cluster 0 2.11 scrub ok 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.733238+0000 osd.2 (osd.2) 33 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:35.733350+0000 osd.2 (osd.2) 34 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:36.085380+0000 osd.3 (osd.3) 33 : cluster 0 3.7 scrub starts 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:36.087702+0000 osd.3 (osd.3) 34 : cluster 0 3.7 scrub ok 2024-03-11T21:24:37.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:36.324355+0000 osd.1 (osd.1) 37 : cluster 0 5.c scrub starts 2024-03-11T21:24:37.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: cluster 2024-03-11T21:24:36.326554+0000 osd.1 (osd.1) 38 : cluster 0 5.c scrub ok 2024-03-11T21:24:37.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:36 smithi203 bash[23654]: audit 2024-03-11T21:24:36.350730+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.143:0/825657652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:24:37.338 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":21,"flags":0,"active_gid":14550,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":584821010},{"type":"v1","addr":"172.21.15.143:6801","nonce":584821010}]},"active_addr":"172.21.15.143:6801/584821010","active_change":"2024-03-11T21:23:57.022550+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24427,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.143:8443/","prometheus":"http://172.21.15.143:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":97,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2354513289}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":629066818}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":4155584643}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3203852881}]}]} 2024-03-11T21:24:37.342 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:24:37.342 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:24:37.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:24:37.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:37 smithi143 bash[29284]: cluster 2024-03-11T21:24:36.715659+0000 mon.a (mon.0) 831 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:24:37.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:37 smithi143 bash[21512]: cluster 2024-03-11T21:24:36.715659+0000 mon.a (mon.0) 831 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:24:38.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:37 smithi203 bash[23654]: cluster 2024-03-11T21:24:36.715659+0000 mon.a (mon.0) 831 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:24:38.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:39.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:37.065271+0000 mgr.y (mgr.14550) 49 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:39.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:37.441839+0000 osd.5 (osd.5) 35 : cluster 0 5.16 scrub starts 2024-03-11T21:24:39.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:37.443709+0000 osd.5 (osd.5) 36 : cluster 0 5.16 scrub ok 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:37.609734+0000 osd.6 (osd.6) 21 : cluster 0 2.7 scrub starts 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:37.613094+0000 osd.6 (osd.6) 22 : cluster 0 2.7 scrub ok 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.142920+0000 osd.3 (osd.3) 35 : cluster 0 6.c scrub starts 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.145397+0000 osd.3 (osd.3) 36 : cluster 0 6.c scrub ok 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.334823+0000 osd.1 (osd.1) 39 : cluster 0 3.0 scrub starts 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.340025+0000 osd.1 (osd.1) 40 : cluster 0 3.0 scrub ok 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.424606+0000 osd.4 (osd.4) 27 : cluster 0 6.17 scrub starts 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.426956+0000 osd.4 (osd.4) 28 : cluster 0 6.17 scrub ok 2024-03-11T21:24:39.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:38 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.548424+0000 mon.a (mon.0) 832 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:24:39.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:37.065271+0000 mgr.y (mgr.14550) 49 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:39.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:37.441839+0000 osd.5 (osd.5) 35 : cluster 0 5.16 scrub starts 2024-03-11T21:24:39.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:37.443709+0000 osd.5 (osd.5) 36 : cluster 0 5.16 scrub ok 2024-03-11T21:24:39.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:37.609734+0000 osd.6 (osd.6) 21 : cluster 0 2.7 scrub starts 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:37.613094+0000 osd.6 (osd.6) 22 : cluster 0 2.7 scrub ok 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.142920+0000 osd.3 (osd.3) 35 : cluster 0 6.c scrub starts 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.145397+0000 osd.3 (osd.3) 36 : cluster 0 6.c scrub ok 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.334823+0000 osd.1 (osd.1) 39 : cluster 0 3.0 scrub starts 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.340025+0000 osd.1 (osd.1) 40 : cluster 0 3.0 scrub ok 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.424606+0000 osd.4 (osd.4) 27 : cluster 0 6.17 scrub starts 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.426956+0000 osd.4 (osd.4) 28 : cluster 0 6.17 scrub ok 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.548424+0000 mon.a (mon.0) 832 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:24:39.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:37.065271+0000 mgr.y (mgr.14550) 49 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:37.441839+0000 osd.5 (osd.5) 35 : cluster 0 5.16 scrub starts 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:37.443709+0000 osd.5 (osd.5) 36 : cluster 0 5.16 scrub ok 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:37.609734+0000 osd.6 (osd.6) 21 : cluster 0 2.7 scrub starts 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:37.613094+0000 osd.6 (osd.6) 22 : cluster 0 2.7 scrub ok 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.142920+0000 osd.3 (osd.3) 35 : cluster 0 6.c scrub starts 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.145397+0000 osd.3 (osd.3) 36 : cluster 0 6.c scrub ok 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.334823+0000 osd.1 (osd.1) 39 : cluster 0 3.0 scrub starts 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.340025+0000 osd.1 (osd.1) 40 : cluster 0 3.0 scrub ok 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.424606+0000 osd.4 (osd.4) 27 : cluster 0 6.17 scrub starts 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.426956+0000 osd.4 (osd.4) 28 : cluster 0 6.17 scrub ok 2024-03-11T21:24:39.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:38 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.548424+0000 mon.a (mon.0) 832 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:24:40.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: audit 2024-03-11T21:24:38.086129+0000 mgr.y (mgr.14550) 50 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:40.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.448265+0000 osd.5 (osd.5) 37 : cluster 0 5.7 scrub starts 2024-03-11T21:24:40.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:38.450950+0000 osd.5 (osd.5) 38 : cluster 0 5.7 scrub ok 2024-03-11T21:24:40.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.096351+0000 osd.3 (osd.3) 37 : cluster 0 5.0 scrub starts 2024-03-11T21:24:40.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.099896+0000 osd.3 (osd.3) 38 : cluster 0 5.0 scrub ok 2024-03-11T21:24:40.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.109609+0000 osd.7 (osd.7) 33 : cluster 0 6.5 scrub starts 2024-03-11T21:24:40.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:39 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.113939+0000 osd.7 (osd.7) 34 : cluster 0 6.5 scrub ok 2024-03-11T21:24:40.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: audit 2024-03-11T21:24:38.086129+0000 mgr.y (mgr.14550) 50 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:40.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.448265+0000 osd.5 (osd.5) 37 : cluster 0 5.7 scrub starts 2024-03-11T21:24:40.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:38.450950+0000 osd.5 (osd.5) 38 : cluster 0 5.7 scrub ok 2024-03-11T21:24:40.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.096351+0000 osd.3 (osd.3) 37 : cluster 0 5.0 scrub starts 2024-03-11T21:24:40.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.099896+0000 osd.3 (osd.3) 38 : cluster 0 5.0 scrub ok 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.109609+0000 osd.7 (osd.7) 33 : cluster 0 6.5 scrub starts 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.113939+0000 osd.7 (osd.7) 34 : cluster 0 6.5 scrub ok 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: audit 2024-03-11T21:24:38.086129+0000 mgr.y (mgr.14550) 50 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.448265+0000 osd.5 (osd.5) 37 : cluster 0 5.7 scrub starts 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:38.450950+0000 osd.5 (osd.5) 38 : cluster 0 5.7 scrub ok 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.096351+0000 osd.3 (osd.3) 37 : cluster 0 5.0 scrub starts 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.099896+0000 osd.3 (osd.3) 38 : cluster 0 5.0 scrub ok 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.109609+0000 osd.7 (osd.7) 33 : cluster 0 6.5 scrub starts 2024-03-11T21:24:40.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:39 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.113939+0000 osd.7 (osd.7) 34 : cluster 0 6.5 scrub ok 2024-03-11T21:24:41.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:40 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.065968+0000 mgr.y (mgr.14550) 51 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:41.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:40 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.435410+0000 osd.0 (osd.0) 27 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:24:41.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:40 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.439019+0000 osd.0 (osd.0) 28 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:24:41.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:40 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.489430+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-03-11T21:24:41.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:40 smithi203 bash[23654]: cluster 2024-03-11T21:24:39.491832+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-03-11T21:24:41.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.065968+0000 mgr.y (mgr.14550) 51 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:41.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.435410+0000 osd.0 (osd.0) 27 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:24:41.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.439019+0000 osd.0 (osd.0) 28 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:24:41.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.489430+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-03-11T21:24:41.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[21512]: cluster 2024-03-11T21:24:39.491832+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-03-11T21:24:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.065968+0000 mgr.y (mgr.14550) 51 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.435410+0000 osd.0 (osd.0) 27 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:24:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.439019+0000 osd.0 (osd.0) 28 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:24:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.489430+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-03-11T21:24:41.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:40 smithi143 bash[29284]: cluster 2024-03-11T21:24:39.491832+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.445453+0000 osd.0 (osd.0) 29 : cluster 0 3.1b scrub starts 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.449249+0000 osd.0 (osd.0) 30 : cluster 0 3.1b scrub ok 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.505076+0000 osd.5 (osd.5) 41 : cluster 0 6.a scrub starts 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.507388+0000 osd.5 (osd.5) 42 : cluster 0 6.a scrub ok 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.578584+0000 osd.6 (osd.6) 23 : cluster 0 5.2 scrub starts 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:40.580956+0000 osd.6 (osd.6) 24 : cluster 0 5.2 scrub ok 2024-03-11T21:24:42.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.337620+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:24:42.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:41 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.339885+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:24:42.118 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:24:42.137 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.445453+0000 osd.0 (osd.0) 29 : cluster 0 3.1b scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.449249+0000 osd.0 (osd.0) 30 : cluster 0 3.1b scrub ok 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.505076+0000 osd.5 (osd.5) 41 : cluster 0 6.a scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.507388+0000 osd.5 (osd.5) 42 : cluster 0 6.a scrub ok 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.578584+0000 osd.6 (osd.6) 23 : cluster 0 5.2 scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:40.580956+0000 osd.6 (osd.6) 24 : cluster 0 5.2 scrub ok 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.337620+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.339885+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.445453+0000 osd.0 (osd.0) 29 : cluster 0 3.1b scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.449249+0000 osd.0 (osd.0) 30 : cluster 0 3.1b scrub ok 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.505076+0000 osd.5 (osd.5) 41 : cluster 0 6.a scrub starts 2024-03-11T21:24:42.138 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.507388+0000 osd.5 (osd.5) 42 : cluster 0 6.a scrub ok 2024-03-11T21:24:42.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.578584+0000 osd.6 (osd.6) 23 : cluster 0 5.2 scrub starts 2024-03-11T21:24:42.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:40.580956+0000 osd.6 (osd.6) 24 : cluster 0 5.2 scrub ok 2024-03-11T21:24:42.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.337620+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:24:42.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:41 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.339885+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:24:43.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.066699+0000 mgr.y (mgr.14550) 52 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:24:43.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.154740+0000 osd.7 (osd.7) 35 : cluster 0 3.d deep-scrub starts 2024-03-11T21:24:43.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.172675+0000 osd.7 (osd.7) 36 : cluster 0 3.d deep-scrub ok 2024-03-11T21:24:43.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.534511+0000 osd.5 (osd.5) 43 : cluster 0 5.f scrub starts 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:41.537374+0000 osd.5 (osd.5) 44 : cluster 0 5.f scrub ok 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.034523+0000 osd.3 (osd.3) 39 : cluster 0 2.1e scrub starts 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.036809+0000 osd.3 (osd.3) 40 : cluster 0 2.1e scrub ok 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.333711+0000 osd.1 (osd.1) 43 : cluster 0 5.19 scrub starts 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.336263+0000 osd.1 (osd.1) 44 : cluster 0 5.19 scrub ok 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.387884+0000 osd.4 (osd.4) 29 : cluster 0 3.1a scrub starts 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: cluster 2024-03-11T21:24:42.391814+0000 osd.4 (osd.4) 30 : cluster 0 3.1a scrub ok 2024-03-11T21:24:43.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:42 smithi203 bash[23654]: audit 2024-03-11T21:24:42.514206+0000 mon.c (mon.2) 53 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.066699+0000 mgr.y (mgr.14550) 52 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.154740+0000 osd.7 (osd.7) 35 : cluster 0 3.d deep-scrub starts 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.172675+0000 osd.7 (osd.7) 36 : cluster 0 3.d deep-scrub ok 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.534511+0000 osd.5 (osd.5) 43 : cluster 0 5.f scrub starts 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:41.537374+0000 osd.5 (osd.5) 44 : cluster 0 5.f scrub ok 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.034523+0000 osd.3 (osd.3) 39 : cluster 0 2.1e scrub starts 2024-03-11T21:24:43.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.036809+0000 osd.3 (osd.3) 40 : cluster 0 2.1e scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.333711+0000 osd.1 (osd.1) 43 : cluster 0 5.19 scrub starts 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.336263+0000 osd.1 (osd.1) 44 : cluster 0 5.19 scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.387884+0000 osd.4 (osd.4) 29 : cluster 0 3.1a scrub starts 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: cluster 2024-03-11T21:24:42.391814+0000 osd.4 (osd.4) 30 : cluster 0 3.1a scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[29284]: audit 2024-03-11T21:24:42.514206+0000 mon.c (mon.2) 53 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.066699+0000 mgr.y (mgr.14550) 52 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.154740+0000 osd.7 (osd.7) 35 : cluster 0 3.d deep-scrub starts 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.172675+0000 osd.7 (osd.7) 36 : cluster 0 3.d deep-scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.534511+0000 osd.5 (osd.5) 43 : cluster 0 5.f scrub starts 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:41.537374+0000 osd.5 (osd.5) 44 : cluster 0 5.f scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.034523+0000 osd.3 (osd.3) 39 : cluster 0 2.1e scrub starts 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.036809+0000 osd.3 (osd.3) 40 : cluster 0 2.1e scrub ok 2024-03-11T21:24:43.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.333711+0000 osd.1 (osd.1) 43 : cluster 0 5.19 scrub starts 2024-03-11T21:24:43.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.336263+0000 osd.1 (osd.1) 44 : cluster 0 5.19 scrub ok 2024-03-11T21:24:43.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.387884+0000 osd.4 (osd.4) 29 : cluster 0 3.1a scrub starts 2024-03-11T21:24:43.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: cluster 2024-03-11T21:24:42.391814+0000 osd.4 (osd.4) 30 : cluster 0 3.1a scrub ok 2024-03-11T21:24:43.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:42 smithi143 bash[21512]: audit 2024-03-11T21:24:42.514206+0000 mon.c (mon.2) 53 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.022685+0000 osd.3 (osd.3) 41 : cluster 0 6.13 scrub starts 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.024850+0000 osd.3 (osd.3) 42 : cluster 0 6.13 scrub ok 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.137216+0000 osd.7 (osd.7) 37 : cluster 0 6.8 scrub starts 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.139890+0000 osd.7 (osd.7) 38 : cluster 0 6.8 scrub ok 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.022685+0000 osd.3 (osd.3) 41 : cluster 0 6.13 scrub starts 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.024850+0000 osd.3 (osd.3) 42 : cluster 0 6.13 scrub ok 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.137216+0000 osd.7 (osd.7) 37 : cluster 0 6.8 scrub starts 2024-03-11T21:24:43.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:43 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.139890+0000 osd.7 (osd.7) 38 : cluster 0 6.8 scrub ok 2024-03-11T21:24:44.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:43 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.022685+0000 osd.3 (osd.3) 41 : cluster 0 6.13 scrub starts 2024-03-11T21:24:44.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:43 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.024850+0000 osd.3 (osd.3) 42 : cluster 0 6.13 scrub ok 2024-03-11T21:24:44.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:43 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.137216+0000 osd.7 (osd.7) 37 : cluster 0 6.8 scrub starts 2024-03-11T21:24:44.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:43 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.139890+0000 osd.7 (osd.7) 38 : cluster 0 6.8 scrub ok 2024-03-11T21:24:45.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.068125+0000 mgr.y (mgr.14550) 53 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:45.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.362940+0000 osd.1 (osd.1) 45 : cluster 0 5.12 scrub starts 2024-03-11T21:24:45.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.364471+0000 osd.1 (osd.1) 46 : cluster 0 5.12 scrub ok 2024-03-11T21:24:45.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.416543+0000 osd.3 (osd.3) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.416656+0000 osd.3 (osd.3) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.444069+0000 osd.0 (osd.0) 31 : cluster 0 3.12 scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.446539+0000 osd.0 (osd.0) 32 : cluster 0 3.12 scrub ok 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.573633+0000 osd.5 (osd.5) 45 : cluster 0 5.15 scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.576131+0000 osd.5 (osd.5) 46 : cluster 0 5.15 scrub ok 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:43.751798+0000 mon.a (mon.0) 833 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.056730+0000 osd.3 (osd.3) 45 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.059186+0000 osd.3 (osd.3) 46 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.319093+0000 osd.1 (osd.1) 47 : cluster 0 6.1d scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.321973+0000 osd.1 (osd.1) 48 : cluster 0 6.1d scrub ok 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.375562+0000 osd.4 (osd.4) 31 : cluster 0 5.1c scrub starts 2024-03-11T21:24:45.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:44 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.377639+0000 osd.4 (osd.4) 32 : cluster 0 5.1c scrub ok 2024-03-11T21:24:45.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.068125+0000 mgr.y (mgr.14550) 53 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:45.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.362940+0000 osd.1 (osd.1) 45 : cluster 0 5.12 scrub starts 2024-03-11T21:24:45.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.364471+0000 osd.1 (osd.1) 46 : cluster 0 5.12 scrub ok 2024-03-11T21:24:45.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.416543+0000 osd.3 (osd.3) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:45.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.416656+0000 osd.3 (osd.3) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.444069+0000 osd.0 (osd.0) 31 : cluster 0 3.12 scrub starts 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.446539+0000 osd.0 (osd.0) 32 : cluster 0 3.12 scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.573633+0000 osd.5 (osd.5) 45 : cluster 0 5.15 scrub starts 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.576131+0000 osd.5 (osd.5) 46 : cluster 0 5.15 scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:43.751798+0000 mon.a (mon.0) 833 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.056730+0000 osd.3 (osd.3) 45 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.059186+0000 osd.3 (osd.3) 46 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.319093+0000 osd.1 (osd.1) 47 : cluster 0 6.1d scrub starts 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.321973+0000 osd.1 (osd.1) 48 : cluster 0 6.1d scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.375562+0000 osd.4 (osd.4) 31 : cluster 0 5.1c scrub starts 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.377639+0000 osd.4 (osd.4) 32 : cluster 0 5.1c scrub ok 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.068125+0000 mgr.y (mgr.14550) 53 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:45.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.362940+0000 osd.1 (osd.1) 45 : cluster 0 5.12 scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.364471+0000 osd.1 (osd.1) 46 : cluster 0 5.12 scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.416543+0000 osd.3 (osd.3) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.416656+0000 osd.3 (osd.3) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.444069+0000 osd.0 (osd.0) 31 : cluster 0 3.12 scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.446539+0000 osd.0 (osd.0) 32 : cluster 0 3.12 scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.573633+0000 osd.5 (osd.5) 45 : cluster 0 5.15 scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.576131+0000 osd.5 (osd.5) 46 : cluster 0 5.15 scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:43.751798+0000 mon.a (mon.0) 833 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.056730+0000 osd.3 (osd.3) 45 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.059186+0000 osd.3 (osd.3) 46 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.319093+0000 osd.1 (osd.1) 47 : cluster 0 6.1d scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.321973+0000 osd.1 (osd.1) 48 : cluster 0 6.1d scrub ok 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.375562+0000 osd.4 (osd.4) 31 : cluster 0 5.1c scrub starts 2024-03-11T21:24:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:44 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.377639+0000 osd.4 (osd.4) 32 : cluster 0 5.1c scrub ok 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/125089703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074930 msgr2=0x7fb39010e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- 172.21.15.143:0/125089703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074930 0x7fb39010e330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb380009990 tx=0x7fb380009ca0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/125089703 shutdown_connections 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- 172.21.15.143:0/125089703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb39010e870 0x7fb390110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- 172.21.15.143:0/125089703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb390074010 0x7fb3900743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- 172.21.15.143:0/125089703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074930 0x7fb39010e330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/125089703 >> 172.21.15.143:0/125089703 conn(0x7fb390078eb0 msgr2=0x7fb3900792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/125089703 shutdown_connections 2024-03-11T21:24:45.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/125089703 wait complete. 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 librados: starting msgr at 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 librados: starting objecter 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 Processor -- start 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- start start 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 librados: setting wanted keys 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 librados: calling monclient init 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 0x7fb390199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39010e870 0x7fb39019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb390113800 con 0x7fb390074010 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb390113680 con 0x7fb39010e870 2024-03-11T21:24:45.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb390113980 con 0x7fb390074930 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 0x7fb390199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:59274/0 (socket says 172.21.15.143:59274) 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 -- 172.21.15.143:0/4216050500 learned_addr learned my addr 172.21.15.143:0/4216050500 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb3949cd700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39010e870 0x7fb39019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 -- 172.21.15.143:0/4216050500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39010e870 msgr2=0x7fb39019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39010e870 0x7fb39019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 -- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 msgr2=0x7fb390199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 0x7fb390199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 -- 172.21.15.143:0/4216050500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39019edf0 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38ffff700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 0x7fb390199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38d7fa700 10 client.?.objecter ms_handle_connect 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38f7fe700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb380000c00 tx=0x7fb3800047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380027070 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb380022470 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39019f080 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380017650 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 10 client.24482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3901a6960 con 0x7fb390074930 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.450+0000 7fb396430700 1 librados: init done 2024-03-11T21:24:45.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39004faa0 con 0x7fb390074930 2024-03-11T21:24:45.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb38002a070 con 0x7fb390074930 2024-03-11T21:24:45.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 0x7fb37c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(107..107 src has 56..107) v4 ==== 7421+0+0 (secure 0 0 0) 0x7fb38008f0c0 con 0x7fb390074930 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38ffff700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 0x7fb37c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 10 client.24482.objecter ms_dispatch 0x7fb390001550 osd_map(107..107 src has 56..107) v4 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 3 client.24482.objecter handle_osd_map got epochs [107,107] > 0 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 3 client.24482.objecter handle_osd_map decoding full epoch 107 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 20 client.24482.objecter dump_active .. 0 homeless 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38ffff700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 0x7fb37c071ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb378005950 tx=0x7fb37800a300 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:45.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.454+0000 7fb38d7fa700 10 client.24482.objecter ms_handle_connect 0x7fb37c06f820 2024-03-11T21:24:45.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.458+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb380013070 con 0x7fb390074930 2024-03-11T21:24:45.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.674+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb39002d830 con 0x7fb390074930 2024-03-11T21:24:45.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb38d7fa700 1 -- 172.21.15.143:0/4216050500 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+23089 (secure 0 0 0) 0x7fb38005c680 con 0x7fb390074930 2024-03-11T21:24:45.655 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:24:45.655 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":107,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","created":"2024-03-11T21:12:47.384719+0000","modified":"2024-03-11T21:24:43.739700+0000","last_up_change":"2024-03-11T21:22:21.063566+0000","last_in_change":"2024-03-11T21:21:55.456354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:08.036690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:23:01.432928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:23:03.441476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-03-11T21:23:04.441432+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:23:05.388543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:23:07.642696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6803","nonce":2903432023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6805","nonce":2903432023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6809","nonce":2903432023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6807","nonce":2903432023}]},"public_addr":"172.21.15.143:6803/2903432023","cluster_addr":"172.21.15.143:6805/2903432023","heartbeat_back_addr":"172.21.15.143:6809/2903432023","heartbeat_front_addr":"172.21.15.143:6807/2903432023","state":["exists","up"]},{"osd":1,"uuid":"2948dc44-68d3-4b3d-92bf-2b0c46374f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6811","nonce":397942214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6813","nonce":397942214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6817","nonce":397942214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6815","nonce":397942214}]},"public_addr":"172.21.15.143:6811/397942214","cluster_addr":"172.21.15.143:6813/397942214","heartbeat_back_addr":"172.21.15.143:6817/397942214","heartbeat_front_addr":"172.21.15.143:6815/397942214","state":["exists","up"]},{"osd":2,"uuid":"7217589a-5835-407b-91a7-f5c055a48f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6819","nonce":2592131625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6821","nonce":2592131625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6825","nonce":2592131625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6823","nonce":2592131625}]},"public_addr":"172.21.15.143:6819/2592131625","cluster_addr":"172.21.15.143:6821/2592131625","heartbeat_back_addr":"172.21.15.143:6825/2592131625","heartbeat_front_addr":"172.21.15.143:6823/2592131625","state":["exists","up"]},{"osd":3,"uuid":"a3e7177f-a5a0-44ad-a65d-9aeac3a680cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6827","nonce":2146571980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6829","nonce":2146571980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6833","nonce":2146571980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6831","nonce":2146571980}]},"public_addr":"172.21.15.143:6827/2146571980","cluster_addr":"172.21.15.143:6829/2146571980","heartbeat_back_addr":"172.21.15.143:6833/2146571980","heartbeat_front_addr":"172.21.15.143:6831/2146571980","state":["exists","up"]},{"osd":4,"uuid":"d7b9e088-d377-4335-8863-11d581828f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6801","nonce":2090835691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6803","nonce":2090835691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6807","nonce":2090835691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6805","nonce":2090835691}]},"public_addr":"172.21.15.203:6801/2090835691","cluster_addr":"172.21.15.203:6803/2090835691","heartbeat_back_addr":"172.21.15.203:6807/2090835691","heartbeat_front_addr":"172.21.15.203:6805/2090835691","state":["exists","up"]},{"osd":5,"uuid":"b3b2a75c-68a5-4e00-8068-622e1ebbaa01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6809","nonce":2432038345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6811","nonce":2432038345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6815","nonce":2432038345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6813","nonce":2432038345}]},"public_addr":"172.21.15.203:6809/2432038345","cluster_addr":"172.21.15.203:6811/2432038345","heartbeat_back_addr":"172.21.15.203:6815/2432038345","heartbeat_front_addr":"172.21.15.203:6813/2432038345","state":["exists","up"]},{"osd":6,"uuid":"ceffb234-db0c-4ffa-a142-9b201df9b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6817","nonce":513748834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6819","nonce":513748834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6823","nonce":513748834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6821","nonce":513748834}]},"public_addr":"172.21.15.203:6817/513748834","cluster_addr":"172.21.15.203:6819/513748834","heartbeat_back_addr":"172.21.15.203:6823/513748834","heartbeat_front_addr":"172.21.15.203:6821/513748834","state":["exists","up"]},{"osd":7,"uuid":"88b53171-69d2-4922-a787-e63d44d01c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6825","nonce":1251497952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6827","nonce":1251497952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6831","nonce":1251497952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6829","nonce":1251497952}]},"public_addr":"172.21.15.203:6825/1251497952","cluster_addr":"172.21.15.203:6827/1251497952","heartbeat_back_addr":"172.21.15.203:6831/1251497952","heartbeat_front_addr":"172.21.15.203:6829/1251497952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:23:46.580919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:24.457469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:35.733355+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:43.416661+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:00.190125+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:21.882113+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:00.025703+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:28.089858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6801/3638739149":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4274199357":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/3217951344":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/3912416566":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/906834629":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2791822486":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2190104670":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4171894415":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2845155916":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2913593434":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/143435387":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2930358289":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6800/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3638739149":"2024-03-12T21:23:57.022184+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 10 librados: watch_flush enter 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 10 librados: watch_flush exit 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 20 client.24482.objecter shutdown clearing up homeless session... 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 10 client.24482.objecter successfully canceled tick 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 msgr2=0x7fb37c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 0x7fb37c071ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb378005950 tx=0x7fb37800a300 comp rx=0 tx=0).stop 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 msgr2=0x7fb39019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.678+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb380000c00 tx=0x7fb3800047b0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 shutdown_connections 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb37c06f820 0x7fb37c071ce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb390074010 0x7fb390199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39010e870 0x7fb39019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 --2- 172.21.15.143:0/4216050500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb390074930 0x7fb39019a320 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 >> 172.21.15.143:0/4216050500 conn(0x7fb390078eb0 msgr2=0x7fb3901062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 shutdown_connections 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 -- 172.21.15.143:0/4216050500 wait complete. 2024-03-11T21:24:45.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:45.682+0000 7fb396430700 1 librados: shutdown 2024-03-11T21:24:45.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.459393+0000 osd.0 (osd.0) 33 : cluster 0 3.1f scrub starts 2024-03-11T21:24:45.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.463369+0000 osd.0 (osd.0) 34 : cluster 0 3.1f scrub ok 2024-03-11T21:24:45.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.525102+0000 osd.5 (osd.5) 47 : cluster 0 6.7 scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:44.527228+0000 osd.5 (osd.5) 48 : cluster 0 6.7 scrub ok 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.127723+0000 osd.7 (osd.7) 39 : cluster 0 5.1a scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.130071+0000 osd.7 (osd.7) 40 : cluster 0 5.1a scrub ok 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.309485+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.310949+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.356065+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.358430+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[29284]: audit 2024-03-11T21:24:45.676689+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.143:0/4216050500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.459393+0000 osd.0 (osd.0) 33 : cluster 0 3.1f scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.463369+0000 osd.0 (osd.0) 34 : cluster 0 3.1f scrub ok 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.525102+0000 osd.5 (osd.5) 47 : cluster 0 6.7 scrub starts 2024-03-11T21:24:45.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:44.527228+0000 osd.5 (osd.5) 48 : cluster 0 6.7 scrub ok 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.127723+0000 osd.7 (osd.7) 39 : cluster 0 5.1a scrub starts 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.130071+0000 osd.7 (osd.7) 40 : cluster 0 5.1a scrub ok 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.309485+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.310949+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.356065+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.358430+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:24:45.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:45 smithi143 bash[21512]: audit 2024-03-11T21:24:45.676689+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.143:0/4216050500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.459393+0000 osd.0 (osd.0) 33 : cluster 0 3.1f scrub starts 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.463369+0000 osd.0 (osd.0) 34 : cluster 0 3.1f scrub ok 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.525102+0000 osd.5 (osd.5) 47 : cluster 0 6.7 scrub starts 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:44.527228+0000 osd.5 (osd.5) 48 : cluster 0 6.7 scrub ok 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.127723+0000 osd.7 (osd.7) 39 : cluster 0 5.1a scrub starts 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.130071+0000 osd.7 (osd.7) 40 : cluster 0 5.1a scrub ok 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.309485+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-03-11T21:24:46.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.310949+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-03-11T21:24:46.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.356065+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:24:46.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.358430+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:24:46.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:45 smithi203 bash[23654]: audit 2024-03-11T21:24:45.676689+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.143:0/4216050500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:46.469 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:24:46.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:24:47.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:46 smithi203 bash[23654]: cluster 2024-03-11T21:24:45.068814+0000 mgr.y (mgr.14550) 54 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:47.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:46 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.338387+0000 osd.4 (osd.4) 35 : cluster 0 6.e scrub starts 2024-03-11T21:24:47.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:46 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.341242+0000 osd.4 (osd.4) 36 : cluster 0 6.e scrub ok 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[21512]: cluster 2024-03-11T21:24:45.068814+0000 mgr.y (mgr.14550) 54 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.338387+0000 osd.4 (osd.4) 35 : cluster 0 6.e scrub starts 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.341242+0000 osd.4 (osd.4) 36 : cluster 0 6.e scrub ok 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[29284]: cluster 2024-03-11T21:24:45.068814+0000 mgr.y (mgr.14550) 54 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.338387+0000 osd.4 (osd.4) 35 : cluster 0 6.e scrub starts 2024-03-11T21:24:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:46 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.341242+0000 osd.4 (osd.4) 36 : cluster 0 6.e scrub ok 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.438791+0000 osd.0 (osd.0) 35 : cluster 0 5.5 scrub starts 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.441271+0000 osd.0 (osd.0) 36 : cluster 0 5.5 scrub ok 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.550313+0000 osd.6 (osd.6) 25 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:46.552633+0000 osd.6 (osd.6) 26 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:47.090207+0000 osd.3 (osd.3) 47 : cluster 0 6.6 scrub starts 2024-03-11T21:24:48.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:47 smithi203 bash[23654]: cluster 2024-03-11T21:24:47.092649+0000 osd.3 (osd.3) 48 : cluster 0 6.6 scrub ok 2024-03-11T21:24:48.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.438791+0000 osd.0 (osd.0) 35 : cluster 0 5.5 scrub starts 2024-03-11T21:24:48.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.441271+0000 osd.0 (osd.0) 36 : cluster 0 5.5 scrub ok 2024-03-11T21:24:48.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.550313+0000 osd.6 (osd.6) 25 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:24:48.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:46.552633+0000 osd.6 (osd.6) 26 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:24:48.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:47.090207+0000 osd.3 (osd.3) 47 : cluster 0 6.6 scrub starts 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[21512]: cluster 2024-03-11T21:24:47.092649+0000 osd.3 (osd.3) 48 : cluster 0 6.6 scrub ok 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.438791+0000 osd.0 (osd.0) 35 : cluster 0 5.5 scrub starts 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.441271+0000 osd.0 (osd.0) 36 : cluster 0 5.5 scrub ok 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.550313+0000 osd.6 (osd.6) 25 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:46.552633+0000 osd.6 (osd.6) 26 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:47.090207+0000 osd.3 (osd.3) 47 : cluster 0 6.6 scrub starts 2024-03-11T21:24:48.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:47 smithi143 bash[29284]: cluster 2024-03-11T21:24:47.092649+0000 osd.3 (osd.3) 48 : cluster 0 6.6 scrub ok 2024-03-11T21:24:48.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:48 smithi203 bash[23654]: cluster 2024-03-11T21:24:47.069634+0000 mgr.y (mgr.14550) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:48 smithi203 bash[23654]: cluster 2024-03-11T21:24:47.435358+0000 osd.0 (osd.0) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:24:49.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:48 smithi203 bash[23654]: cluster 2024-03-11T21:24:47.440303+0000 osd.0 (osd.0) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[29284]: cluster 2024-03-11T21:24:47.069634+0000 mgr.y (mgr.14550) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[29284]: cluster 2024-03-11T21:24:47.435358+0000 osd.0 (osd.0) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[29284]: cluster 2024-03-11T21:24:47.440303+0000 osd.0 (osd.0) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[21512]: cluster 2024-03-11T21:24:47.069634+0000 mgr.y (mgr.14550) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[21512]: cluster 2024-03-11T21:24:47.435358+0000 osd.0 (osd.0) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:24:49.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:48 smithi143 bash[21512]: cluster 2024-03-11T21:24:47.440303+0000 osd.0 (osd.0) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:24:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:49 smithi203 bash[23654]: audit 2024-03-11T21:24:48.094214+0000 mgr.y (mgr.14550) 56 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:49 smithi203 bash[23654]: cluster 2024-03-11T21:24:48.995410+0000 osd.3 (osd.3) 49 : cluster 0 6.11 scrub starts 2024-03-11T21:24:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:49 smithi203 bash[23654]: cluster 2024-03-11T21:24:48.997885+0000 osd.3 (osd.3) 50 : cluster 0 6.11 scrub ok 2024-03-11T21:24:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:49 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.267984+0000 osd.4 (osd.4) 37 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:24:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:49 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.278150+0000 osd.4 (osd.4) 38 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[29284]: audit 2024-03-11T21:24:48.094214+0000 mgr.y (mgr.14550) 56 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[29284]: cluster 2024-03-11T21:24:48.995410+0000 osd.3 (osd.3) 49 : cluster 0 6.11 scrub starts 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[29284]: cluster 2024-03-11T21:24:48.997885+0000 osd.3 (osd.3) 50 : cluster 0 6.11 scrub ok 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.267984+0000 osd.4 (osd.4) 37 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.278150+0000 osd.4 (osd.4) 38 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:24:50.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[21512]: audit 2024-03-11T21:24:48.094214+0000 mgr.y (mgr.14550) 56 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:50.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[21512]: cluster 2024-03-11T21:24:48.995410+0000 osd.3 (osd.3) 49 : cluster 0 6.11 scrub starts 2024-03-11T21:24:50.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[21512]: cluster 2024-03-11T21:24:48.997885+0000 osd.3 (osd.3) 50 : cluster 0 6.11 scrub ok 2024-03-11T21:24:50.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.267984+0000 osd.4 (osd.4) 37 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:24:50.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:49 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.278150+0000 osd.4 (osd.4) 38 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.071190+0000 mgr.y (mgr.14550) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.480207+0000 osd.0 (osd.0) 39 : cluster 0 2.13 scrub starts 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.482583+0000 osd.0 (osd.0) 40 : cluster 0 2.13 scrub ok 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.490099+0000 osd.6 (osd.6) 27 : cluster 0 5.1f scrub starts 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.493026+0000 osd.6 (osd.6) 28 : cluster 0 5.1f scrub ok 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.958068+0000 osd.3 (osd.3) 51 : cluster 0 6.1b scrub starts 2024-03-11T21:24:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:50 smithi203 bash[23654]: cluster 2024-03-11T21:24:49.960545+0000 osd.3 (osd.3) 52 : cluster 0 6.1b scrub ok 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.071190+0000 mgr.y (mgr.14550) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.480207+0000 osd.0 (osd.0) 39 : cluster 0 2.13 scrub starts 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.482583+0000 osd.0 (osd.0) 40 : cluster 0 2.13 scrub ok 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.490099+0000 osd.6 (osd.6) 27 : cluster 0 5.1f scrub starts 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.493026+0000 osd.6 (osd.6) 28 : cluster 0 5.1f scrub ok 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.958068+0000 osd.3 (osd.3) 51 : cluster 0 6.1b scrub starts 2024-03-11T21:24:51.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[21512]: cluster 2024-03-11T21:24:49.960545+0000 osd.3 (osd.3) 52 : cluster 0 6.1b scrub ok 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.071190+0000 mgr.y (mgr.14550) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.480207+0000 osd.0 (osd.0) 39 : cluster 0 2.13 scrub starts 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.482583+0000 osd.0 (osd.0) 40 : cluster 0 2.13 scrub ok 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.490099+0000 osd.6 (osd.6) 27 : cluster 0 5.1f scrub starts 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.493026+0000 osd.6 (osd.6) 28 : cluster 0 5.1f scrub ok 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.958068+0000 osd.3 (osd.3) 51 : cluster 0 6.1b scrub starts 2024-03-11T21:24:51.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:50 smithi143 bash[29284]: cluster 2024-03-11T21:24:49.960545+0000 osd.3 (osd.3) 52 : cluster 0 6.1b scrub ok 2024-03-11T21:24:51.243 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[21512]: cluster 2024-03-11T21:24:50.516418+0000 osd.0 (osd.0) 41 : cluster 0 5.3 scrub starts 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[21512]: cluster 2024-03-11T21:24:50.519642+0000 osd.0 (osd.0) 42 : cluster 0 5.3 scrub ok 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[21512]: cluster 2024-03-11T21:24:51.109967+0000 osd.7 (osd.7) 41 : cluster 0 3.11 scrub starts 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[21512]: cluster 2024-03-11T21:24:51.114162+0000 osd.7 (osd.7) 42 : cluster 0 3.11 scrub ok 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[29284]: cluster 2024-03-11T21:24:50.516418+0000 osd.0 (osd.0) 41 : cluster 0 5.3 scrub starts 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[29284]: cluster 2024-03-11T21:24:50.519642+0000 osd.0 (osd.0) 42 : cluster 0 5.3 scrub ok 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[29284]: cluster 2024-03-11T21:24:51.109967+0000 osd.7 (osd.7) 41 : cluster 0 3.11 scrub starts 2024-03-11T21:24:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:51 smithi143 bash[29284]: cluster 2024-03-11T21:24:51.114162+0000 osd.7 (osd.7) 42 : cluster 0 3.11 scrub ok 2024-03-11T21:24:52.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:51 smithi203 bash[23654]: cluster 2024-03-11T21:24:50.516418+0000 osd.0 (osd.0) 41 : cluster 0 5.3 scrub starts 2024-03-11T21:24:52.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:51 smithi203 bash[23654]: cluster 2024-03-11T21:24:50.519642+0000 osd.0 (osd.0) 42 : cluster 0 5.3 scrub ok 2024-03-11T21:24:52.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:51 smithi203 bash[23654]: cluster 2024-03-11T21:24:51.109967+0000 osd.7 (osd.7) 41 : cluster 0 3.11 scrub starts 2024-03-11T21:24:52.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:51 smithi203 bash[23654]: cluster 2024-03-11T21:24:51.114162+0000 osd.7 (osd.7) 42 : cluster 0 3.11 scrub ok 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[21512]: cluster 2024-03-11T21:24:51.071930+0000 mgr.y (mgr.14550) 58 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[21512]: cluster 2024-03-11T21:24:52.005592+0000 osd.3 (osd.3) 53 : cluster 0 5.17 scrub starts 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[21512]: cluster 2024-03-11T21:24:52.007345+0000 osd.3 (osd.3) 54 : cluster 0 5.17 scrub ok 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[29284]: cluster 2024-03-11T21:24:51.071930+0000 mgr.y (mgr.14550) 58 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[29284]: cluster 2024-03-11T21:24:52.005592+0000 osd.3 (osd.3) 53 : cluster 0 5.17 scrub starts 2024-03-11T21:24:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:52 smithi143 bash[29284]: cluster 2024-03-11T21:24:52.007345+0000 osd.3 (osd.3) 54 : cluster 0 5.17 scrub ok 2024-03-11T21:24:53.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:52 smithi203 bash[23654]: cluster 2024-03-11T21:24:51.071930+0000 mgr.y (mgr.14550) 58 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:53.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:52 smithi203 bash[23654]: cluster 2024-03-11T21:24:52.005592+0000 osd.3 (osd.3) 53 : cluster 0 5.17 scrub starts 2024-03-11T21:24:53.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:52 smithi203 bash[23654]: cluster 2024-03-11T21:24:52.007345+0000 osd.3 (osd.3) 54 : cluster 0 5.17 scrub ok 2024-03-11T21:24:54.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/2080774096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84109360 msgr2=0x7efd84110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:54.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- 172.21.15.143:0/2080774096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84109360 0x7efd84110f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0099c0 tx=0x7efd7c009cd0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/2080774096 shutdown_connections 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- 172.21.15.143:0/2080774096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84109360 0x7efd84110f30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- 172.21.15.143:0/2080774096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104280 0x7efd84104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- 172.21.15.143:0/2080774096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84104c30 0x7efd84108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/2080774096 >> 172.21.15.143:0/2080774096 conn(0x7efd840ffb10 msgr2=0x7efd84101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/2080774096 shutdown_connections 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/2080774096 wait complete. 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 librados: starting msgr at 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 librados: starting objecter 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 Processor -- start 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- start start 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 librados: setting wanted keys 2024-03-11T21:24:54.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 librados: calling monclient init 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104c30 0x7efd8419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd84113860 con 0x7efd84104280 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd841136e0 con 0x7efd84104c30 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd841139e0 con 0x7efd84109360 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd83fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd83fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52656/0 (socket says 172.21.15.143:52656) 2024-03-11T21:24:54.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:52310/0 (socket says 172.21.15.143:52310) 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd83fff700 1 -- 172.21.15.143:0/3676381108 learned_addr learned my addr 172.21.15.143:0/3676381108 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd837fe700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104c30 0x7efd8419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 -- 172.21.15.143:0/3676381108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104c30 msgr2=0x7efd8419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104c30 0x7efd8419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 -- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 msgr2=0x7efd84199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 -- 172.21.15.143:0/3676381108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8419ed30 con 0x7efd84109360 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd83fff700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd88a34700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098e0 tx=0x7efd7c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd817fa700 10 client.?.objecter ms_handle_connect 0x7efd84109360 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd817fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c009e60 con 0x7efd84109360 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8419efc0 con 0x7efd84109360 2024-03-11T21:24:54.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7efd7c005550 con 0x7efd84109360 2024-03-11T21:24:54.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c003a60 con 0x7efd84109360 2024-03-11T21:24:54.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 10 client.24488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:24:54.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841a68a0 con 0x7efd84109360 2024-03-11T21:24:54.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.466+0000 7efd8a497700 1 librados: init done 2024-03-11T21:24:54.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8404faa0 con 0x7efd84109360 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7efd7c0050d0 con 0x7efd84109360 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 0x7efd70071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(107..107 src has 56..107) v4 ==== 7421+0+0 (secure 0 0 0) 0x7efd7c08f740 con 0x7efd84109360 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 10 client.24488.objecter ms_dispatch 0x7efd84001550 osd_map(107..107 src has 56..107) v4 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 3 client.24488.objecter handle_osd_map got epochs [107,107] > 0 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 3 client.24488.objecter handle_osd_map decoding full epoch 107 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd83fff700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 0x7efd70071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 20 client.24488.objecter dump_active .. 0 homeless 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd83fff700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 0x7efd70071ce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd74007fd0 tx=0x7efd74009040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:54.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.470+0000 7efd817fa700 10 client.24488.objecter ms_handle_connect 0x7efd7006f820 2024-03-11T21:24:54.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.474+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7efd7c013070 con 0x7efd84109360 2024-03-11T21:24:54.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.682+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd841a6a70 con 0x7efd84109360 2024-03-11T21:24:54.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.686+0000 7efd817fa700 1 -- 172.21.15.143:0/3676381108 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+23089 (secure 0 0 0) 0x7efd7c05cde0 con 0x7efd84109360 2024-03-11T21:24:54.665 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:24:54.666 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":107,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","created":"2024-03-11T21:12:47.384719+0000","modified":"2024-03-11T21:24:43.739700+0000","last_up_change":"2024-03-11T21:22:21.063566+0000","last_in_change":"2024-03-11T21:21:55.456354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:08.036690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:23:01.432928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:23:03.441476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-03-11T21:23:04.441432+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:23:05.388543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:23:07.642696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6803","nonce":2903432023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6805","nonce":2903432023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6809","nonce":2903432023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6807","nonce":2903432023}]},"public_addr":"172.21.15.143:6803/2903432023","cluster_addr":"172.21.15.143:6805/2903432023","heartbeat_back_addr":"172.21.15.143:6809/2903432023","heartbeat_front_addr":"172.21.15.143:6807/2903432023","state":["exists","up"]},{"osd":1,"uuid":"2948dc44-68d3-4b3d-92bf-2b0c46374f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6811","nonce":397942214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6813","nonce":397942214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6817","nonce":397942214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":397942214},{"type":"v1","addr":"172.21.15.143:6815","nonce":397942214}]},"public_addr":"172.21.15.143:6811/397942214","cluster_addr":"172.21.15.143:6813/397942214","heartbeat_back_addr":"172.21.15.143:6817/397942214","heartbeat_front_addr":"172.21.15.143:6815/397942214","state":["exists","up"]},{"osd":2,"uuid":"7217589a-5835-407b-91a7-f5c055a48f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6819","nonce":2592131625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6821","nonce":2592131625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6825","nonce":2592131625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6823","nonce":2592131625}]},"public_addr":"172.21.15.143:6819/2592131625","cluster_addr":"172.21.15.143:6821/2592131625","heartbeat_back_addr":"172.21.15.143:6825/2592131625","heartbeat_front_addr":"172.21.15.143:6823/2592131625","state":["exists","up"]},{"osd":3,"uuid":"a3e7177f-a5a0-44ad-a65d-9aeac3a680cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6827","nonce":2146571980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6829","nonce":2146571980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6833","nonce":2146571980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6831","nonce":2146571980}]},"public_addr":"172.21.15.143:6827/2146571980","cluster_addr":"172.21.15.143:6829/2146571980","heartbeat_back_addr":"172.21.15.143:6833/2146571980","heartbeat_front_addr":"172.21.15.143:6831/2146571980","state":["exists","up"]},{"osd":4,"uuid":"d7b9e088-d377-4335-8863-11d581828f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6801","nonce":2090835691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6803","nonce":2090835691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6807","nonce":2090835691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6805","nonce":2090835691}]},"public_addr":"172.21.15.203:6801/2090835691","cluster_addr":"172.21.15.203:6803/2090835691","heartbeat_back_addr":"172.21.15.203:6807/2090835691","heartbeat_front_addr":"172.21.15.203:6805/2090835691","state":["exists","up"]},{"osd":5,"uuid":"b3b2a75c-68a5-4e00-8068-622e1ebbaa01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6809","nonce":2432038345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6811","nonce":2432038345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6815","nonce":2432038345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6813","nonce":2432038345}]},"public_addr":"172.21.15.203:6809/2432038345","cluster_addr":"172.21.15.203:6811/2432038345","heartbeat_back_addr":"172.21.15.203:6815/2432038345","heartbeat_front_addr":"172.21.15.203:6813/2432038345","state":["exists","up"]},{"osd":6,"uuid":"ceffb234-db0c-4ffa-a142-9b201df9b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6817","nonce":513748834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6819","nonce":513748834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6823","nonce":513748834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6821","nonce":513748834}]},"public_addr":"172.21.15.203:6817/513748834","cluster_addr":"172.21.15.203:6819/513748834","heartbeat_back_addr":"172.21.15.203:6823/513748834","heartbeat_front_addr":"172.21.15.203:6821/513748834","state":["exists","up"]},{"osd":7,"uuid":"88b53171-69d2-4922-a787-e63d44d01c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6825","nonce":1251497952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6827","nonce":1251497952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6831","nonce":1251497952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6829","nonce":1251497952}]},"public_addr":"172.21.15.203:6825/1251497952","cluster_addr":"172.21.15.203:6827/1251497952","heartbeat_back_addr":"172.21.15.203:6831/1251497952","heartbeat_front_addr":"172.21.15.203:6829/1251497952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:23:46.580919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:24.457469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:35.733355+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:43.416661+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:00.190125+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:21.882113+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:00.025703+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:28.089858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6801/3638739149":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4274199357":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/3217951344":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/3912416566":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/906834629":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2791822486":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2190104670":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4171894415":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2845155916":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2913593434":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/143435387":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2930358289":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6800/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3638739149":"2024-03-12T21:23:57.022184+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 10 librados: watch_flush enter 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 10 librados: watch_flush exit 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 20 client.24488.objecter shutdown clearing up homeless session... 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 10 client.24488.objecter successfully canceled tick 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 msgr2=0x7efd70071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 0x7efd70071ce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd74007fd0 tx=0x7efd74009040 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 msgr2=0x7efd8419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098e0 tx=0x7efd7c00fa00 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 shutdown_connections 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd7006f820 0x7efd70071ce0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd84104280 0x7efd84199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd84109360 0x7efd8419e5f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 --2- 172.21.15.143:0/3676381108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd84104c30 0x7efd8419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 >> 172.21.15.143:0/3676381108 conn(0x7efd840ffb10 msgr2=0x7efd84100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 shutdown_connections 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 -- 172.21.15.143:0/3676381108 wait complete. 2024-03-11T21:24:54.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:24:54.690+0000 7efd8a497700 1 librados: shutdown 2024-03-11T21:24:54.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[21512]: cluster 2024-03-11T21:24:53.073311+0000 mgr.y (mgr.14550) 59 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[21512]: cluster 2024-03-11T21:24:53.518922+0000 osd.0 (osd.0) 43 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[21512]: cluster 2024-03-11T21:24:53.521313+0000 osd.0 (osd.0) 44 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[21512]: audit 2024-03-11T21:24:54.687141+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.143:0/3676381108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[29284]: cluster 2024-03-11T21:24:53.073311+0000 mgr.y (mgr.14550) 59 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[29284]: cluster 2024-03-11T21:24:53.518922+0000 osd.0 (osd.0) 43 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[29284]: cluster 2024-03-11T21:24:53.521313+0000 osd.0 (osd.0) 44 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:24:54.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:54 smithi143 bash[29284]: audit 2024-03-11T21:24:54.687141+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.143:0/3676381108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:55.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:54 smithi203 bash[23654]: cluster 2024-03-11T21:24:53.073311+0000 mgr.y (mgr.14550) 59 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:55.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:54 smithi203 bash[23654]: cluster 2024-03-11T21:24:53.518922+0000 osd.0 (osd.0) 43 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:24:55.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:54 smithi203 bash[23654]: cluster 2024-03-11T21:24:53.521313+0000 osd.0 (osd.0) 44 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:24:55.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:54 smithi203 bash[23654]: audit 2024-03-11T21:24:54.687141+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.143:0/3676381108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:24:55.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:24:55.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:24:55.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:24:55.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:24:55.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:24:55.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:24:55.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:24:55.520 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:24:57.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[21512]: cluster 2024-03-11T21:24:55.074091+0000 mgr.y (mgr.14550) 60 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-11T21:24:57.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[21512]: cluster 2024-03-11T21:24:55.483799+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-03-11T21:24:57.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[21512]: cluster 2024-03-11T21:24:55.485962+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-03-11T21:24:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[29284]: cluster 2024-03-11T21:24:55.074091+0000 mgr.y (mgr.14550) 60 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-11T21:24:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[29284]: cluster 2024-03-11T21:24:55.483799+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-03-11T21:24:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:56 smithi143 bash[29284]: cluster 2024-03-11T21:24:55.485962+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-03-11T21:24:57.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:56 smithi203 bash[23654]: cluster 2024-03-11T21:24:55.074091+0000 mgr.y (mgr.14550) 60 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-11T21:24:57.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:56 smithi203 bash[23654]: cluster 2024-03-11T21:24:55.483799+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-03-11T21:24:57.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:56 smithi203 bash[23654]: cluster 2024-03-11T21:24:55.485962+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-03-11T21:24:58.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:57 smithi203 bash[23654]: audit 2024-03-11T21:24:57.096129+0000 mon.c (mon.2) 56 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:57 smithi203 bash[23654]: audit 2024-03-11T21:24:57.096903+0000 mon.a (mon.0) 834 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:57 smithi203 bash[23654]: audit 2024-03-11T21:24:57.543939+0000 mon.c (mon.2) 57 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:58.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[29284]: audit 2024-03-11T21:24:57.096129+0000 mon.c (mon.2) 56 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[29284]: audit 2024-03-11T21:24:57.096903+0000 mon.a (mon.0) 834 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[29284]: audit 2024-03-11T21:24:57.543939+0000 mon.c (mon.2) 57 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:58.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[21512]: audit 2024-03-11T21:24:57.096129+0000 mon.c (mon.2) 56 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[21512]: audit 2024-03-11T21:24:57.096903+0000 mon.a (mon.0) 834 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:24:58.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:57 smithi143 bash[21512]: audit 2024-03-11T21:24:57.543939+0000 mon.c (mon.2) 57 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:58.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:24:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[29284]: cluster 2024-03-11T21:24:57.074785+0000 mgr.y (mgr.14550) 61 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[29284]: audit 2024-03-11T21:24:57.821555+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[29284]: cluster 2024-03-11T21:24:57.829194+0000 mon.a (mon.0) 836 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[21512]: cluster 2024-03-11T21:24:57.074785+0000 mgr.y (mgr.14550) 61 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[21512]: audit 2024-03-11T21:24:57.821555+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:24:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:58 smithi143 bash[21512]: cluster 2024-03-11T21:24:57.829194+0000 mon.a (mon.0) 836 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:24:59.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:58 smithi203 bash[23654]: cluster 2024-03-11T21:24:57.074785+0000 mgr.y (mgr.14550) 61 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:59.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:58 smithi203 bash[23654]: audit 2024-03-11T21:24:57.821555+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:24:59.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:58 smithi203 bash[23654]: cluster 2024-03-11T21:24:57.829194+0000 mon.a (mon.0) 836 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:25:00.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:59 smithi143 bash[21512]: audit 2024-03-11T21:24:58.104051+0000 mgr.y (mgr.14550) 62 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:00.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:24:59 smithi143 bash[21512]: cluster 2024-03-11T21:24:58.845605+0000 mon.a (mon.0) 837 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:25:00.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:59 smithi143 bash[29284]: audit 2024-03-11T21:24:58.104051+0000 mgr.y (mgr.14550) 62 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:00.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:24:59 smithi143 bash[29284]: cluster 2024-03-11T21:24:58.845605+0000 mon.a (mon.0) 837 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:25:00.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:59 smithi203 bash[23654]: audit 2024-03-11T21:24:58.104051+0000 mgr.y (mgr.14550) 62 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:00.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:24:59 smithi203 bash[23654]: cluster 2024-03-11T21:24:58.845605+0000 mon.a (mon.0) 837 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:25:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.426 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.426 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.427 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.428 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.429 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:00.429 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:01.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:00 smithi143 bash[21512]: cluster 2024-03-11T21:24:59.075546+0000 mgr.y (mgr.14550) 63 : cluster 0 pgmap v46: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:01.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:00 smithi143 bash[29284]: cluster 2024-03-11T21:24:59.075546+0000 mgr.y (mgr.14550) 63 : cluster 0 pgmap v46: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:00 smithi203 bash[23654]: cluster 2024-03-11T21:24:59.075546+0000 mgr.y (mgr.14550) 63 : cluster 0 pgmap v46: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:02.436 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[21512]: cluster 2024-03-11T21:25:01.076201+0000 mgr.y (mgr.14550) 64 : cluster 0 pgmap v47: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[21512]: cluster 2024-03-11T21:25:01.548922+0000 osd.0 (osd.0) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[21512]: cluster 2024-03-11T21:25:01.549042+0000 osd.0 (osd.0) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[21512]: cluster 2024-03-11T21:25:01.858094+0000 mon.a (mon.0) 838 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[29284]: cluster 2024-03-11T21:25:01.076201+0000 mgr.y (mgr.14550) 64 : cluster 0 pgmap v47: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:03.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[29284]: cluster 2024-03-11T21:25:01.548922+0000 osd.0 (osd.0) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:03.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[29284]: cluster 2024-03-11T21:25:01.549042+0000 osd.0 (osd.0) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:03.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:02 smithi143 bash[29284]: cluster 2024-03-11T21:25:01.858094+0000 mon.a (mon.0) 838 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:25:03.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:02 smithi203 bash[23654]: cluster 2024-03-11T21:25:01.076201+0000 mgr.y (mgr.14550) 64 : cluster 0 pgmap v47: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:02 smithi203 bash[23654]: cluster 2024-03-11T21:25:01.548922+0000 osd.0 (osd.0) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:02 smithi203 bash[23654]: cluster 2024-03-11T21:25:01.549042+0000 osd.0 (osd.0) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:03.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:02 smithi203 bash[23654]: cluster 2024-03-11T21:25:01.858094+0000 mon.a (mon.0) 838 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:25:04.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:03 smithi143 bash[21512]: cluster 2024-03-11T21:25:02.190945+0000 osd.7 (osd.7) 43 : cluster 0 5.4 scrub starts 2024-03-11T21:25:04.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:03 smithi143 bash[21512]: cluster 2024-03-11T21:25:02.193670+0000 osd.7 (osd.7) 44 : cluster 0 5.4 scrub ok 2024-03-11T21:25:04.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:03 smithi143 bash[29284]: cluster 2024-03-11T21:25:02.190945+0000 osd.7 (osd.7) 43 : cluster 0 5.4 scrub starts 2024-03-11T21:25:04.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:03 smithi143 bash[29284]: cluster 2024-03-11T21:25:02.193670+0000 osd.7 (osd.7) 44 : cluster 0 5.4 scrub ok 2024-03-11T21:25:04.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:03 smithi203 bash[23654]: cluster 2024-03-11T21:25:02.190945+0000 osd.7 (osd.7) 43 : cluster 0 5.4 scrub starts 2024-03-11T21:25:04.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:03 smithi203 bash[23654]: cluster 2024-03-11T21:25:02.193670+0000 osd.7 (osd.7) 44 : cluster 0 5.4 scrub ok 2024-03-11T21:25:05.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[21512]: cluster 2024-03-11T21:25:03.077185+0000 mgr.y (mgr.14550) 65 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:25:05.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[21512]: cluster 2024-03-11T21:25:03.178561+0000 osd.7 (osd.7) 45 : cluster 0 5.10 scrub starts 2024-03-11T21:25:05.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[21512]: cluster 2024-03-11T21:25:03.181030+0000 osd.7 (osd.7) 46 : cluster 0 5.10 scrub ok 2024-03-11T21:25:05.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[29284]: cluster 2024-03-11T21:25:03.077185+0000 mgr.y (mgr.14550) 65 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:25:05.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[29284]: cluster 2024-03-11T21:25:03.178561+0000 osd.7 (osd.7) 45 : cluster 0 5.10 scrub starts 2024-03-11T21:25:05.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:04 smithi143 bash[29284]: cluster 2024-03-11T21:25:03.181030+0000 osd.7 (osd.7) 46 : cluster 0 5.10 scrub ok 2024-03-11T21:25:05.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:04 smithi203 bash[23654]: cluster 2024-03-11T21:25:03.077185+0000 mgr.y (mgr.14550) 65 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:25:05.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:04 smithi203 bash[23654]: cluster 2024-03-11T21:25:03.178561+0000 osd.7 (osd.7) 45 : cluster 0 5.10 scrub starts 2024-03-11T21:25:05.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:04 smithi203 bash[23654]: cluster 2024-03-11T21:25:03.181030+0000 osd.7 (osd.7) 46 : cluster 0 5.10 scrub ok 2024-03-11T21:25:07.115 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:06 smithi143 bash[21512]: cluster 2024-03-11T21:25:05.077824+0000 mgr.y (mgr.14550) 66 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:25:07.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:06 smithi143 bash[29284]: cluster 2024-03-11T21:25:05.077824+0000 mgr.y (mgr.14550) 66 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:25:07.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:06 smithi203 bash[23654]: cluster 2024-03-11T21:25:05.077824+0000 mgr.y (mgr.14550) 66 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:25:08.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:09.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[21512]: cluster 2024-03-11T21:25:07.078287+0000 mgr.y (mgr.14550) 67 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 621 B/s rd, 0 op/s 2024-03-11T21:25:09.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[21512]: cluster 2024-03-11T21:25:08.468099+0000 osd.4 (osd.4) 39 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:09.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[21512]: cluster 2024-03-11T21:25:08.468192+0000 osd.4 (osd.4) 40 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:09.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[29284]: cluster 2024-03-11T21:25:07.078287+0000 mgr.y (mgr.14550) 67 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 621 B/s rd, 0 op/s 2024-03-11T21:25:09.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[29284]: cluster 2024-03-11T21:25:08.468099+0000 osd.4 (osd.4) 39 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:09.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:08 smithi143 bash[29284]: cluster 2024-03-11T21:25:08.468192+0000 osd.4 (osd.4) 40 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:09.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:08 smithi203 bash[23654]: cluster 2024-03-11T21:25:07.078287+0000 mgr.y (mgr.14550) 67 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 621 B/s rd, 0 op/s 2024-03-11T21:25:09.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:08 smithi203 bash[23654]: cluster 2024-03-11T21:25:08.468099+0000 osd.4 (osd.4) 39 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:09.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:08 smithi203 bash[23654]: cluster 2024-03-11T21:25:08.468192+0000 osd.4 (osd.4) 40 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:09 smithi143 bash[21512]: audit 2024-03-11T21:25:08.114118+0000 mgr.y (mgr.14550) 68 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:09 smithi143 bash[21512]: cluster 2024-03-11T21:25:08.891868+0000 mon.a (mon.0) 839 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:25:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:09 smithi143 bash[29284]: audit 2024-03-11T21:25:08.114118+0000 mgr.y (mgr.14550) 68 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:10.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:09 smithi143 bash[29284]: cluster 2024-03-11T21:25:08.891868+0000 mon.a (mon.0) 839 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:25:10.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:09 smithi203 bash[23654]: audit 2024-03-11T21:25:08.114118+0000 mgr.y (mgr.14550) 68 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:10.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:09 smithi203 bash[23654]: cluster 2024-03-11T21:25:08.891868+0000 mon.a (mon.0) 839 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:25:11.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:10 smithi143 bash[21512]: cluster 2024-03-11T21:25:09.078897+0000 mgr.y (mgr.14550) 69 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:11.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:10 smithi203 bash[23654]: cluster 2024-03-11T21:25:09.078897+0000 mgr.y (mgr.14550) 69 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:11.390 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:11 smithi143 bash[29284]: cluster 2024-03-11T21:25:09.078897+0000 mgr.y (mgr.14550) 69 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 -- 172.21.15.143:0/729415205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b54102d50 msgr2=0x7f2b5406a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/729415205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b54102d50 0x7f2b5406a460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009990 tx=0x7f2b50009ca0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 -- 172.21.15.143:0/729415205 shutdown_connections 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/729415205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b54102d50 0x7f2b5406a460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/729415205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b5410e730 0x7f2b54110b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/729415205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b5406a9a0 0x7f2b5406ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 -- 172.21.15.143:0/729415205 >> 172.21.15.143:0/729415205 conn(0x7f2b54077c90 msgr2=0x7f2b540780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 -- 172.21.15.143:0/729415205 shutdown_connections 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 -- 172.21.15.143:0/729415205 wait complete. 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 librados: starting msgr at 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.566+0000 7f2b5b3df700 1 librados: starting objecter 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 Processor -- start 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- start start 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 librados: setting wanted keys 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 librados: calling monclient init 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b5406a9a0 0x7f2b541033a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 0x7f2b541038e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b5410e730 0x7f2b54103e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54113830 con 0x7f2b5410e730 2024-03-11T21:25:11.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541136b0 con 0x7f2b5406a9a0 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541139b0 con 0x7f2b54102d50 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 0x7f2b541038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5997c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b5410e730 0x7f2b54103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 0x7f2b541038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:50322/0 (socket says 172.21.15.143:50322) 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 -- 172.21.15.143:0/2054785751 learned_addr learned my addr 172.21.15.143:0/2054785751 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5917b700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b5406a9a0 0x7f2b541033a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 -- 172.21.15.143:0/2054785751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b5406a9a0 msgr2=0x7f2b541033a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b5406a9a0 0x7f2b541033a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 -- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b5410e730 msgr2=0x7f2b54103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b5410e730 0x7f2b54103e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b541aa470 con 0x7f2b54102d50 2024-03-11T21:25:11.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5997c700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b5410e730 0x7f2b54103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5897a700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 0x7f2b541038e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400e9d0 tx=0x7f2b4400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 10 client.?.objecter ms_handle_connect 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4400cbd0 con 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b541aa760 con 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2b4400ee70 con 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44018420 con 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 10 client.24494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b541aaca0 con 0x7f2b54102d50 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 librados: init done 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 10 client.24494.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:11.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 15 client.24494.objecter _session_command_op_assign -1 1 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 20 client.24494.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 10 client.24494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b54069d10 con 0x7f2b54102d50 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2b44010430 con 0x7f2b54102d50 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b4006f820 0x7f2b40071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(111..111 src has 56..111) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f2b4408e4a0 con 0x7f2b54102d50 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 10 client.24494.objecter ms_dispatch 0x7f2b54001550 osd_map(111..111 src has 56..111) v4 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 3 client.24494.objecter handle_osd_map got epochs [111,111] > 0 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 3 client.24494.objecter handle_osd_map decoding full epoch 111 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5917b700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b4006f820 0x7f2b40071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 10 client.24494.objecter checking command 1 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2b40079650 0x7f2b4007bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter _get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 15 client.24494.objecter _session_command_op_remove -1 1 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter _get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 15 client.24494.objecter _session_command_op_assign 2 1 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 4 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 10 client.24494.objecter _send_command 1 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b44012ec0 con 0x7f2b40079650 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 20 client.24494.objecter dump_active .. 0 homeless 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=111) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b44095050 con 0x7f2b54102d50 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.570+0000 7f2b5997c700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2b40079650 0x7f2b4007bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b5917b700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b4006f820 0x7f2b40071ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000b7f0 tx=0x7f2b50005fd0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 10 client.24494.objecter ms_handle_connect 0x7f2b4006f820 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b5997c700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2b40079650 0x7f2b4007bab0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 10 client.24494.objecter ms_handle_connect 0x7f2b40079650 2024-03-11T21:25:11.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2b44012ec0 con 0x7f2b40079650 2024-03-11T21:25:11.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 10 client.24494.objecter ms_dispatch 0x7f2b54001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:11.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 10 client.24494.objecter _finish_command 1 = system:0 2024-03-11T21:25:11.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 4 2024-03-11T21:25:11.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.574+0000 7f2b4a7fc700 15 client.24494.objecter _session_command_op_remove 2 1 2024-03-11T21:25:11.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 client.24494.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:11.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 15 client.24494.objecter _session_command_op_assign -1 2 2024-03-11T21:25:11.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter _get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter _get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 15 client.24494.objecter _session_command_op_remove -1 2 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter get_session s=0x7f2b40078f80 osd=2 3 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 15 client.24494.objecter _session_command_op_assign 2 2 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 4 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 client.24494.objecter _send_command 2 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b54101350 con 0x7f2b40079650 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b4a7fc700 1 -- 172.21.15.143:0/2054785751 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b54101350 con 0x7f2b40079650 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b4a7fc700 10 client.24494.objecter ms_dispatch 0x7f2b54001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b4a7fc700 10 client.24494.objecter _finish_command 2 = system:0 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b4a7fc700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 4 2024-03-11T21:25:11.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b4a7fc700 15 client.24494.objecter _session_command_op_remove 2 2 2024-03-11T21:25:11.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 librados: watch_flush enter 2024-03-11T21:25:11.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 librados: watch_flush exit 2024-03-11T21:25:11.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 client.24494.objecter close_session for osd.2 2024-03-11T21:25:11.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2b40079650 msgr2=0x7f2b4007bab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2b40079650 0x7f2b4007bab0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter put_session s=0x7f2b40078f80 osd=2 1 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 20 client.24494.objecter shutdown clearing up homeless session... 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 10 client.24494.objecter successfully canceled tick 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b4006f820 msgr2=0x7f2b40071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b4006f820 0x7f2b40071ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000b7f0 tx=0x7f2b50005fd0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 msgr2=0x7f2b541038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 --2- 172.21.15.143:0/2054785751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b54102d50 0x7f2b541038e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400e9d0 tx=0x7f2b4400ece0 comp rx=0 tx=0).stop 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5997c700 1 -- 172.21.15.143:0/2054785751 reap_dead start 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 shutdown_connections 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 >> 172.21.15.143:0/2054785751 conn(0x7f2b54077c90 msgr2=0x7f2b54110230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 shutdown_connections 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 -- 172.21.15.143:0/2054785751 wait complete. 2024-03-11T21:25:11.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:11.590+0000 7f2b5b3df700 1 librados: shutdown 2024-03-11T21:25:12.436 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:13.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: cluster 2024-03-11T21:25:11.079584+0000 mgr.y (mgr.14550) 70 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: cluster 2024-03-11T21:25:11.267548+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: cluster 2024-03-11T21:25:11.267675+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: cluster 2024-03-11T21:25:11.920651+0000 mon.a (mon.0) 840 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: audit 2024-03-11T21:25:12.192980+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[21512]: audit 2024-03-11T21:25:12.575313+0000 mon.c (mon.2) 58 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: cluster 2024-03-11T21:25:11.079584+0000 mgr.y (mgr.14550) 70 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: cluster 2024-03-11T21:25:11.267548+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: cluster 2024-03-11T21:25:11.267675+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: cluster 2024-03-11T21:25:11.920651+0000 mon.a (mon.0) 840 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:25:13.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: audit 2024-03-11T21:25:12.192980+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:13.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:12 smithi143 bash[29284]: audit 2024-03-11T21:25:12.575313+0000 mon.c (mon.2) 58 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:13.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: cluster 2024-03-11T21:25:11.079584+0000 mgr.y (mgr.14550) 70 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: cluster 2024-03-11T21:25:11.267548+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: cluster 2024-03-11T21:25:11.267675+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: cluster 2024-03-11T21:25:11.920651+0000 mon.a (mon.0) 840 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:25:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: audit 2024-03-11T21:25:12.192980+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:13.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:12 smithi203 bash[23654]: audit 2024-03-11T21:25:12.575313+0000 mon.c (mon.2) 58 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:14.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:13 smithi143 bash[21512]: cluster 2024-03-11T21:25:12.163276+0000 osd.7 (osd.7) 47 : cluster 0 5.9 scrub starts 2024-03-11T21:25:14.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:13 smithi143 bash[21512]: cluster 2024-03-11T21:25:12.165702+0000 osd.7 (osd.7) 48 : cluster 0 5.9 scrub ok 2024-03-11T21:25:14.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:13 smithi143 bash[29284]: cluster 2024-03-11T21:25:12.163276+0000 osd.7 (osd.7) 47 : cluster 0 5.9 scrub starts 2024-03-11T21:25:14.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:13 smithi143 bash[29284]: cluster 2024-03-11T21:25:12.165702+0000 osd.7 (osd.7) 48 : cluster 0 5.9 scrub ok 2024-03-11T21:25:14.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:13 smithi203 bash[23654]: cluster 2024-03-11T21:25:12.163276+0000 osd.7 (osd.7) 47 : cluster 0 5.9 scrub starts 2024-03-11T21:25:14.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:13 smithi203 bash[23654]: cluster 2024-03-11T21:25:12.165702+0000 osd.7 (osd.7) 48 : cluster 0 5.9 scrub ok 2024-03-11T21:25:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:14 smithi143 bash[29284]: cluster 2024-03-11T21:25:13.080966+0000 mgr.y (mgr.14550) 71 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:14 smithi143 bash[21512]: cluster 2024-03-11T21:25:13.080966+0000 mgr.y (mgr.14550) 71 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:15.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:14 smithi203 bash[23654]: cluster 2024-03-11T21:25:13.080966+0000 mgr.y (mgr.14550) 71 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:17.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:16 smithi143 bash[21512]: cluster 2024-03-11T21:25:15.081678+0000 mgr.y (mgr.14550) 72 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:17.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:16 smithi143 bash[29284]: cluster 2024-03-11T21:25:15.081678+0000 mgr.y (mgr.14550) 72 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:17.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:16 smithi203 bash[23654]: cluster 2024-03-11T21:25:15.081678+0000 mgr.y (mgr.14550) 72 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:17.434 INFO:teuthology.orchestra.run.smithi143.stdout:81604378711 2024-03-11T21:25:17.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/279080188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14101700 msgr2=0x7f5e1410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- 172.21.15.143:0/279080188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14101700 0x7f5e1410db70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009f90 tx=0x7f5e0c00b330 comp rx=0 tx=0).stop 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/279080188 shutdown_connections 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- 172.21.15.143:0/279080188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14101700 0x7f5e1410db70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- 172.21.15.143:0/279080188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e14106dc0 0x7f5e141071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- 172.21.15.143:0/279080188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14100d40 0x7f5e141011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/279080188 >> 172.21.15.143:0/279080188 conn(0x7f5e14076d80 msgr2=0x7f5e14077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/279080188 shutdown_connections 2024-03-11T21:25:18.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/279080188 wait complete. 2024-03-11T21:25:18.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 librados: starting msgr at 2024-03-11T21:25:18.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 librados: starting objecter 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 Processor -- start 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- start start 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 librados: setting wanted keys 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 librados: calling monclient init 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 0x7f5e1419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 0x7f5e1419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e14106dc0 0x7f5e14199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e14110570 con 0x7f5e14100d40 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e141103f0 con 0x7f5e14106dc0 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e141106f0 con 0x7f5e14101700 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e11d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 0x7f5e1419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 0x7f5e1419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 0x7f5e1419f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42034/0 (socket says 172.21.15.143:42034) 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e11d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 0x7f5e1419fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43224/0 (socket says 172.21.15.143:43224) 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 -- 172.21.15.143:0/3374861926 learned_addr learned my addr 172.21.15.143:0/3374861926 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e12d9d700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e14106dc0 0x7f5e14199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 -- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 msgr2=0x7f5e1419fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 0x7f5e1419fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 -- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e14106dc0 msgr2=0x7f5e14199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e14106dc0 0x7f5e14199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1419a220 con 0x7f5e14100d40 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e11d9b700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e14101700 0x7f5e1419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e1259c700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 0x7f5e1419f900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400d950 tx=0x7f5e0400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e037fe700 10 client.?.objecter ms_handle_connect 0x7f5e14100d40 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e037fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04016070 con 0x7f5e14100d40 2024-03-11T21:25:18.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5e04009dc0 con 0x7f5e14100d40 2024-03-11T21:25:18.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1419a510 con 0x7f5e14100d40 2024-03-11T21:25:18.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0400f3c0 con 0x7f5e14100d40 2024-03-11T21:25:18.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.022+0000 7f5e18883700 10 client.14628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e141a67d0 con 0x7f5e14100d40 2024-03-11T21:25:18.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 1 librados: init done 2024-03-11T21:25:18.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 10 client.14628.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 15 client.14628.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 20 client.14628.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 10 client.14628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e14102400 con 0x7f5e14100d40 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5e04019070 con 0x7f5e14100d40 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5dfc06f820 0x7f5dfc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f5e0408dd50 con 0x7f5e14100d40 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 10 client.14628.objecter ms_dispatch 0x7f5e14001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 3 client.14628.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 3 client.14628.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e11d9b700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5dfc06f820 0x7f5dfc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 10 client.14628.objecter checking command 1 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5dfc0796d0 0x7f5dfc07bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter _get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 15 client.14628.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter _get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 15 client.14628.objecter _session_command_op_assign 4 1 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 4 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 10 client.14628.objecter _send_command 1 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5dfc07c200 con 0x7f5dfc0796d0 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 20 client.14628.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e04090450 con 0x7f5e14100d40 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 10 client.14628.objecter ms_handle_connect 0x7f5dfc06f820 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e11d9b700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5dfc06f820 0x7f5dfc071ce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009750 tx=0x7f5e08009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e12d9d700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5dfc0796d0 0x7f5dfc07bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e12d9d700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5dfc0796d0 0x7f5dfc07bb30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.026+0000 7f5e037fe700 10 client.14628.objecter ms_handle_connect 0x7f5dfc0796d0 2024-03-11T21:25:18.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.030+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5dfc07c200 con 0x7f5dfc0796d0 2024-03-11T21:25:18.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.030+0000 7f5e037fe700 10 client.14628.objecter ms_dispatch 0x7f5e14001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.030+0000 7f5e037fe700 10 client.14628.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.030+0000 7f5e037fe700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 4 2024-03-11T21:25:18.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.030+0000 7f5e037fe700 15 client.14628.objecter _session_command_op_remove 4 1 2024-03-11T21:25:18.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 10 client.14628.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 15 client.14628.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 20 client.14628.objecter _get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 20 client.14628.objecter _get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 15 client.14628.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 20 client.14628.objecter get_session s=0x7f5dfc079000 osd=4 3 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 15 client.14628.objecter _session_command_op_assign 4 2 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 4 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 10 client.14628.objecter _send_command 2 2024-03-11T21:25:18.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.046+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e1404faa0 con 0x7f5dfc0796d0 2024-03-11T21:25:18.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e037fe700 1 -- 172.21.15.143:0/3374861926 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e1404faa0 con 0x7f5dfc0796d0 2024-03-11T21:25:18.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e037fe700 10 client.14628.objecter ms_dispatch 0x7f5e14001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e037fe700 10 client.14628.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e037fe700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 4 2024-03-11T21:25:18.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e037fe700 15 client.14628.objecter _session_command_op_remove 4 2 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 10 librados: watch_flush enter 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 10 librados: watch_flush exit 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 10 client.14628.objecter close_session for osd.4 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5dfc0796d0 msgr2=0x7f5dfc07bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5dfc0796d0 0x7f5dfc07bb30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 20 client.14628.objecter put_session s=0x7f5dfc079000 osd=4 1 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 20 client.14628.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 10 client.14628.objecter successfully canceled tick 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5dfc06f820 msgr2=0x7f5dfc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5dfc06f820 0x7f5dfc071ce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009750 tx=0x7f5e08009310 comp rx=0 tx=0).stop 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 msgr2=0x7f5e1419f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 --2- 172.21.15.143:0/3374861926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e14100d40 0x7f5e1419f900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400d950 tx=0x7f5e0400dc60 comp rx=0 tx=0).stop 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e12d9d700 1 -- 172.21.15.143:0/3374861926 reap_dead start 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 shutdown_connections 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 >> 172.21.15.143:0/3374861926 conn(0x7f5e14076d80 msgr2=0x7f5e14069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 shutdown_connections 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 -- 172.21.15.143:0/3374861926 wait complete. 2024-03-11T21:25:18.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.050+0000 7f5e18883700 1 librados: shutdown 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- 172.21.15.143:0/2247396437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c100d10 msgr2=0x7fde1c101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- 172.21.15.143:0/2247396437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c100d10 0x7fde1c101190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fde100098f0 tx=0x7fde10009c00 comp rx=0 tx=0).stop 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- 172.21.15.143:0/2247396437 shutdown_connections 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- 172.21.15.143:0/2247396437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde1c1016d0 0x7fde1c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- 172.21.15.143:0/2247396437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c106d80 0x7fde1c107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- 172.21.15.143:0/2247396437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c100d10 0x7fde1c101190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- 172.21.15.143:0/2247396437 >> 172.21.15.143:0/2247396437 conn(0x7fde1c077cb0 msgr2=0x7fde1c0780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- 172.21.15.143:0/2247396437 shutdown_connections 2024-03-11T21:25:18.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- 172.21.15.143:0/2247396437 wait complete. 2024-03-11T21:25:18.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 librados: starting msgr at 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 librados: starting objecter 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 Processor -- start 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- start start 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 librados: setting wanted keys 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 librados: calling monclient init 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 0x7fde1c1029f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c1016d0 0x7fde1c102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde1c106d80 0x7fde1c103470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c110550 con 0x7fde1c1016d0 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1103d0 con 0x7fde1c106d80 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde22899700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1106d0 con 0x7fde1c100d10 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 0x7fde1c1029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 0x7fde1c1029f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43234/0 (socket says 172.21.15.143:43234) 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 -- 172.21.15.143:0/646135283 learned_addr learned my addr 172.21.15.143:0/646135283 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1b7fe700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c1016d0 0x7fde1c102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde20e36700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde1c106d80 0x7fde1c103470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 -- 172.21.15.143:0/646135283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde1c106d80 msgr2=0x7fde1c103470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde1c106d80 0x7fde1c103470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 -- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c1016d0 msgr2=0x7fde1c102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c1016d0 0x7fde1c102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c1a6140 con 0x7fde1c100d10 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1b7fe700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1c1016d0 0x7fde1c102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde1bfff700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 0x7fde1c1029f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00ec10 tx=0x7fde0c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde197fa700 10 client.?.objecter ms_handle_connect 0x7fde1c100d10 2024-03-11T21:25:18.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde197fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.070+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c00cd40 con 0x7fde1c100d10 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fde0c0044a0 con 0x7fde1c100d10 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c010430 con 0x7fde1c100d10 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c1a6430 con 0x7fde1c100d10 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 10 client.24506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c1a68f0 con 0x7fde1c100d10 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 1 librados: init done 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 10 client.24506.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 15 client.24506.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 20 client.24506.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 10 client.24506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde1c069cf0 con 0x7fde1c100d10 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fde0c004000 con 0x7fde1c100d10 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde0406f820 0x7fde04071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fde0c08e820 con 0x7fde1c100d10 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 10 client.24506.objecter ms_dispatch 0x7fde1c001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 3 client.24506.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 3 client.24506.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde1b7fe700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde0406f820 0x7fde04071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 10 client.24506.objecter checking command 1 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fde040796d0 0x7fde0407bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter _get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 15 client.24506.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter _get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 15 client.24506.objecter _session_command_op_assign 0 1 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 4 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 10 client.24506.objecter _send_command 1 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde0c012ec0 con 0x7fde040796d0 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 20 client.24506.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde0c08ecb0 con 0x7fde1c100d10 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde20e36700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fde040796d0 0x7fde0407bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde197fa700 10 client.24506.objecter ms_handle_connect 0x7fde0406f820 2024-03-11T21:25:18.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.074+0000 7fde1b7fe700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde0406f820 0x7fde04071ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde10009fd0 tx=0x7fde10019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 10 client.24506.objecter ms_handle_connect 0x7fde040796d0 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde20e36700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fde040796d0 0x7fde0407bb30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fde0c012ec0 con 0x7fde040796d0 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 10 client.24506.objecter ms_dispatch 0x7fde1c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 10 client.24506.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 4 2024-03-11T21:25:18.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.078+0000 7fde197fa700 15 client.24506.objecter _session_command_op_remove 0 1 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 client.24506.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 15 client.24506.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter _get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter _get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 15 client.24506.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter get_session s=0x7fde04079000 osd=0 3 2024-03-11T21:25:18.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 15 client.24506.objecter _session_command_op_assign 0 2 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 4 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 client.24506.objecter _send_command 2 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde1c102430 con 0x7fde040796d0 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde197fa700 1 -- 172.21.15.143:0/646135283 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde1c102430 con 0x7fde040796d0 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde197fa700 10 client.24506.objecter ms_dispatch 0x7fde1c001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde197fa700 10 client.24506.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde197fa700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 4 2024-03-11T21:25:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde197fa700 15 client.24506.objecter _session_command_op_remove 0 2 2024-03-11T21:25:18.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 librados: watch_flush enter 2024-03-11T21:25:18.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 librados: watch_flush exit 2024-03-11T21:25:18.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 client.24506.objecter close_session for osd.0 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fde040796d0 msgr2=0x7fde0407bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fde040796d0 0x7fde0407bb30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter put_session s=0x7fde04079000 osd=0 1 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 20 client.24506.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 10 client.24506.objecter successfully canceled tick 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde0406f820 msgr2=0x7fde04071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde0406f820 0x7fde04071ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde10009fd0 tx=0x7fde10019040 comp rx=0 tx=0).stop 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 msgr2=0x7fde1c1029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 --2- 172.21.15.143:0/646135283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1c100d10 0x7fde1c1029f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00ec10 tx=0x7fde0c00ef20 comp rx=0 tx=0).stop 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde20e36700 1 -- 172.21.15.143:0/646135283 reap_dead start 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 shutdown_connections 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 >> 172.21.15.143:0/646135283 conn(0x7fde1c077cb0 msgr2=0x7fde1c06a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 shutdown_connections 2024-03-11T21:25:18.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 -- 172.21.15.143:0/646135283 wait complete. 2024-03-11T21:25:18.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.094+0000 7fde22899700 1 librados: shutdown 2024-03-11T21:25:18.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:17 smithi143 bash[21512]: cluster 2024-03-11T21:25:16.223815+0000 osd.7 (osd.7) 49 : cluster 0 6.1c scrub starts 2024-03-11T21:25:18.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:17 smithi143 bash[21512]: cluster 2024-03-11T21:25:16.227103+0000 osd.7 (osd.7) 50 : cluster 0 6.1c scrub ok 2024-03-11T21:25:18.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:17 smithi143 bash[29284]: cluster 2024-03-11T21:25:16.223815+0000 osd.7 (osd.7) 49 : cluster 0 6.1c scrub starts 2024-03-11T21:25:18.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:17 smithi143 bash[29284]: cluster 2024-03-11T21:25:16.227103+0000 osd.7 (osd.7) 50 : cluster 0 6.1c scrub ok 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 -- 172.21.15.143:0/4073032751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 msgr2=0x7faf5406aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 --2- 172.21.15.143:0/4073032751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf5406aa80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf500098f0 tx=0x7faf50009c00 comp rx=0 tx=0).stop 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 -- 172.21.15.143:0/4073032751 shutdown_connections 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 --2- 172.21.15.143:0/4073032751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf54107220 0x7faf5410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 --2- 172.21.15.143:0/4073032751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf5406aa80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 --2- 172.21.15.143:0/4073032751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf5406b050 0x7faf54106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 -- 172.21.15.143:0/4073032751 >> 172.21.15.143:0/4073032751 conn(0x7faf54077de0 msgr2=0x7faf540781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 -- 172.21.15.143:0/4073032751 shutdown_connections 2024-03-11T21:25:18.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.242+0000 7faf5bee5700 1 -- 172.21.15.143:0/4073032751 wait complete. 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 librados: starting msgr at 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 librados: starting objecter 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 Processor -- start 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- start start 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 librados: setting wanted keys 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 librados: calling monclient init 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf54100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf5406b050 0x7faf541012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf54107220 0x7faf54101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf54111650 con 0x7faf5406b050 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541114d0 con 0x7faf54107220 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf541117d0 con 0x7faf5406a6a0 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf54100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf54100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43262/0 (socket says 172.21.15.143:43262) 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 -- 172.21.15.143:0/1757970921 learned_addr learned my addr 172.21.15.143:0/1757970921 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5a482700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf54107220 0x7faf54101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 -- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf54107220 msgr2=0x7faf54101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf54107220 0x7faf54101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 -- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf5406b050 msgr2=0x7faf541012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59480700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf5406b050 0x7faf541012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf5406b050 0x7faf541012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf541a61e0 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59480700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf5406b050 0x7faf541012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf59c81700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf54100d90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf500059d0 tx=0x7faf5000fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf4affd700 10 client.?.objecter ms_handle_connect 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf4affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50027070 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7faf5000bb60 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50004930 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf541a6470 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 10 client.24512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf541a6870 con 0x7faf5406a6a0 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 1 librados: init done 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 10 client.24512.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 15 client.24512.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.246+0000 7faf5bee5700 20 client.24512.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf5bee5700 10 client.24512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf541035b0 con 0x7faf5406a6a0 2024-03-11T21:25:18.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7faf5002a070 con 0x7faf5406a6a0 2024-03-11T21:25:18.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf4006f820 0x7faf40071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7faf5008f760 con 0x7faf5406a6a0 2024-03-11T21:25:18.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 10 client.24512.objecter ms_dispatch 0x7faf54001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 3 client.24512.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 3 client.24512.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf59480700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf4006f820 0x7faf40071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 10 client.24512.objecter checking command 1 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf400796d0 0x7faf4007bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter _get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 15 client.24512.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter _get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 15 client.24512.objecter _session_command_op_assign 5 1 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 4 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 10 client.24512.objecter _send_command 1 2024-03-11T21:25:18.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf50019e90 con 0x7faf400796d0 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 20 client.24512.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf5008fbf0 con 0x7faf5406a6a0 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf59480700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf4006f820 0x7faf40071ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf44005fd0 tx=0x7faf44005ee0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 10 client.24512.objecter ms_handle_connect 0x7faf4006f820 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf5a482700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf400796d0 0x7faf4007bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf4affd700 10 client.24512.objecter ms_handle_connect 0x7faf400796d0 2024-03-11T21:25:18.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.250+0000 7faf5a482700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf400796d0 0x7faf4007bb30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.254+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf50019e90 con 0x7faf400796d0 2024-03-11T21:25:18.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.254+0000 7faf4affd700 10 client.24512.objecter ms_dispatch 0x7faf54001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.254+0000 7faf4affd700 10 client.24512.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.254+0000 7faf4affd700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 4 2024-03-11T21:25:18.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.254+0000 7faf4affd700 15 client.24512.objecter _session_command_op_remove 5 1 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 client.24512.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 15 client.24512.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter _get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter _get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 15 client.24512.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter get_session s=0x7faf40079000 osd=5 3 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 15 client.24512.objecter _session_command_op_assign 5 2 2024-03-11T21:25:18.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 4 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 client.24512.objecter _send_command 2 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf5404faa0 con 0x7faf400796d0 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 -- 172.21.15.143:0/3808037150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce8102a70 msgr2=0x7f4ce8106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 --2- 172.21.15.143:0/3808037150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce8102a70 0x7f4ce8106a60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0099c0 tx=0x7f4cdc009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 -- 172.21.15.143:0/3808037150 shutdown_connections 2024-03-11T21:25:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 --2- 172.21.15.143:0/3808037150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 --2- 172.21.15.143:0/3808037150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce81020c0 0x7f4ce81024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 --2- 172.21.15.143:0/3808037150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce8102a70 0x7f4ce8106a60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 -- 172.21.15.143:0/3808037150 >> 172.21.15.143:0/3808037150 conn(0x7f4ce8078f00 msgr2=0x7f4ce8079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 -- 172.21.15.143:0/3808037150 shutdown_connections 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf4affd700 1 -- 172.21.15.143:0/1757970921 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7faf5404faa0 con 0x7faf400796d0 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf4affd700 10 client.24512.objecter ms_dispatch 0x7faf54001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf4affd700 10 client.24512.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf4affd700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 4 2024-03-11T21:25:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf4affd700 15 client.24512.objecter _session_command_op_remove 5 2 2024-03-11T21:25:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 librados: watch_flush enter 2024-03-11T21:25:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7f4cee168700 1 -- 172.21.15.143:0/3808037150 wait complete. 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 librados: watch_flush exit 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 client.24512.objecter close_session for osd.5 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf400796d0 msgr2=0x7faf4007bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf400796d0 0x7faf4007bb30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter put_session s=0x7faf40079000 osd=5 1 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 20 client.24512.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.274+0000 7faf5bee5700 10 client.24512.objecter successfully canceled tick 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf4006f820 msgr2=0x7faf40071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf4006f820 0x7faf40071ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf44005fd0 tx=0x7faf44005ee0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 msgr2=0x7faf54100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 --2- 172.21.15.143:0/1757970921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf5406a6a0 0x7faf54100d90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf500059d0 tx=0x7faf5000fa70 comp rx=0 tx=0).stop 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5a482700 1 -- 172.21.15.143:0/1757970921 reap_dead start 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 shutdown_connections 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 >> 172.21.15.143:0/1757970921 conn(0x7faf54077de0 msgr2=0x7faf540feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 shutdown_connections 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 -- 172.21.15.143:0/1757970921 wait complete. 2024-03-11T21:25:18.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7faf5bee5700 1 librados: shutdown 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 librados: starting msgr at 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 librados: starting objecter 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 Processor -- start 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 -- start start 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 librados: setting wanted keys 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 librados: calling monclient init 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce81020c0 0x7f4ce819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce8102a70 0x7f4ce819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce8111640 con 0x7f4ce8107140 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81114c0 con 0x7f4ce8102a70 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81117c0 con 0x7f4ce81020c0 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce81020c0 0x7f4ce819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce8197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42086/0 (socket says 172.21.15.143:42086) 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 -- 172.21.15.143:0/228218154 learned_addr learned my addr 172.21.15.143:0/228218154 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce77fe700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce81020c0 0x7f4ce819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43276/0 (socket says 172.21.15.143:43276) 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce6ffd700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce8102a70 0x7f4ce819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 -- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce81020c0 msgr2=0x7f4ce819d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce81020c0 0x7f4ce819d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 -- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce8102a70 msgr2=0x7f4ce819ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce8102a70 0x7f4ce819ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce81981b0 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce6ffd700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ce8102a70 0x7f4ce819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce7fff700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce8197a50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000c370 tx=0x7f4ce000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce4ff9700 10 client.?.objecter ms_handle_connect 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce4ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000ce20 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce8198440 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4ce00090d0 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.278+0000 7f4cee168700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8074010 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce00046b0 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 1 librados: init done 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4ce001a020 con 0x7f4ce8107140 2024-03-11T21:25:18.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 10 client.14643.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 15 client.14643.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 20 client.14643.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ce8103540 con 0x7f4ce8107140 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4cd006f6f0 0x7f4cd0071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce77fe700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4cd006f6f0 0x7f4cd0071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f4ce008d400 con 0x7f4ce8107140 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 10 client.14643.objecter ms_dispatch 0x7f4ce8001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 3 client.14643.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 3 client.14643.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 10 client.14643.objecter checking command 1 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4cd00794a0 0x7f4cd007b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce77fe700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4cd006f6f0 0x7f4cd0071bb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8198b90 tx=0x7f4cd8009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter _get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 15 client.14643.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter _get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 15 client.14643.objecter _session_command_op_assign 6 1 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 4 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 10 client.14643.objecter _send_command 1 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ce00090d0 con 0x7f4cd00794a0 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 20 client.14643.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 10 client.14643.objecter ms_handle_connect 0x7f4cd006f6f0 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ce0092060 con 0x7f4ce8107140 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce6ffd700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4cd00794a0 0x7f4cd007b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce6ffd700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4cd00794a0 0x7f4cd007b900 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.282+0000 7f4ce4ff9700 10 client.14643.objecter ms_handle_connect 0x7f4cd00794a0 2024-03-11T21:25:18.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.286+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4ce00090d0 con 0x7f4cd00794a0 2024-03-11T21:25:18.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.286+0000 7f4ce4ff9700 10 client.14643.objecter ms_dispatch 0x7f4ce8001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.286+0000 7f4ce4ff9700 10 client.14643.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.286+0000 7f4ce4ff9700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 4 2024-03-11T21:25:18.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.286+0000 7f4ce4ff9700 15 client.14643.objecter _session_command_op_remove 6 1 2024-03-11T21:25:18.272 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:18.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:17 smithi203 bash[23654]: cluster 2024-03-11T21:25:16.223815+0000 osd.7 (osd.7) 49 : cluster 0 6.1c scrub starts 2024-03-11T21:25:18.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:17 smithi203 bash[23654]: cluster 2024-03-11T21:25:16.227103+0000 osd.7 (osd.7) 50 : cluster 0 6.1c scrub ok 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 10 client.14643.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 15 client.14643.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 20 client.14643.objecter _get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 20 client.14643.objecter _get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 15 client.14643.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 20 client.14643.objecter get_session s=0x7f4cd0078dd0 osd=6 3 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 15 client.14643.objecter _session_command_op_assign 6 2 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 4 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 10 client.14643.objecter _send_command 2 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.302+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ce804faa0 con 0x7f4cd00794a0 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce4ff9700 1 -- 172.21.15.143:0/228218154 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4ce804faa0 con 0x7f4cd00794a0 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce4ff9700 10 client.14643.objecter ms_dispatch 0x7f4ce8001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce4ff9700 10 client.14643.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce4ff9700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 4 2024-03-11T21:25:18.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce4ff9700 15 client.14643.objecter _session_command_op_remove 6 2 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 10 librados: watch_flush enter 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 10 librados: watch_flush exit 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 10 client.14643.objecter close_session for osd.6 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4cd00794a0 msgr2=0x7f4cd007b900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4cd00794a0 0x7f4cd007b900 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 20 client.14643.objecter put_session s=0x7f4cd0078dd0 osd=6 1 2024-03-11T21:25:18.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 20 client.14643.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 10 client.14643.objecter successfully canceled tick 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4cd006f6f0 msgr2=0x7f4cd0071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4cd006f6f0 0x7f4cd0071bb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8198b90 tx=0x7f4cd8009190 comp rx=0 tx=0).stop 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 msgr2=0x7f4ce8197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 --2- 172.21.15.143:0/228218154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce8107140 0x7f4ce8197a50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000c370 tx=0x7f4ce000c680 comp rx=0 tx=0).stop 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4ce7fff700 1 -- 172.21.15.143:0/228218154 reap_dead start 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 shutdown_connections 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 >> 172.21.15.143:0/228218154 conn(0x7f4ce8078f00 msgr2=0x7f4ce80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 shutdown_connections 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 -- 172.21.15.143:0/228218154 wait complete. 2024-03-11T21:25:18.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.306+0000 7f4cee168700 1 librados: shutdown 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/3002969067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 msgr2=0x7fa768102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- 172.21.15.143:0/3002969067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa768102500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099c0 tx=0x7fa758009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/3002969067 shutdown_connections 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- 172.21.15.143:0/3002969067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa768107150 0x7fa76810ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- 172.21.15.143:0/3002969067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa768102500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- 172.21.15.143:0/3002969067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/3002969067 >> 172.21.15.143:0/3002969067 conn(0x7fa768078f40 msgr2=0x7fa768079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/3002969067 shutdown_connections 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/3002969067 wait complete. 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 librados: starting msgr at 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 librados: starting objecter 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 Processor -- start 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- start start 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 librados: setting wanted keys 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 librados: calling monclient init 2024-03-11T21:25:18.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa7680740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa768107150 0x7fa768074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa768111670 con 0x7fa768102ad0 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681114f0 con 0x7fa768107150 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681117f0 con 0x7fa768102120 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa7680740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa7667fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa7667fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42104/0 (socket says 172.21.15.143:42104) 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa7667fc700 1 -- 172.21.15.143:0/1192538975 learned_addr learned my addr 172.21.15.143:0/1192538975 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa7680740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43296/0 (socket says 172.21.15.143:43296) 2024-03-11T21:25:18.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa7677fe700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa768107150 0x7fa768074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 -- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa768107150 msgr2=0x7fa768074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa768107150 0x7fa768074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 -- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 msgr2=0x7fa768074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7681a61a0 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa7667fc700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa768102ad0 0x7fa768074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa766ffd700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa7680740e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099c0 tx=0x7fa75800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76c88d700 10 client.?.objecter ms_handle_connect 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76c88d700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758009e60 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa75800fce0 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800bea0 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7681a6430 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 10 client.24524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7681a6830 con 0x7fa768102120 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.374+0000 7fa76d88f700 1 librados: init done 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76d88f700 10 client.24524.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76d88f700 15 client.24524.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76d88f700 20 client.24524.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76d88f700 10 client.24524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa768103550 con 0x7fa768102120 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa7580050d0 con 0x7fa768102120 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa75006f820 0x7fa750071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fa75808f2c0 con 0x7fa768102120 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 10 client.24524.objecter ms_dispatch 0x7fa768001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 3 client.24524.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 3 client.24524.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa7667fc700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa75006f820 0x7fa750071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 10 client.24524.objecter checking command 1 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa7500796d0 0x7fa75007bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter _get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 15 client.24524.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter _get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 15 client.24524.objecter _session_command_op_assign 7 1 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 4 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 10 client.24524.objecter _send_command 1 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa758024ec0 con 0x7fa7500796d0 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 20 client.24524.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa75808f750 con 0x7fa768102120 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 10 client.24524.objecter ms_handle_connect 0x7fa75006f820 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa7667fc700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa75006f820 0x7fa750071ce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7681085b0 tx=0x7fa75c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa7677fe700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa7500796d0 0x7fa75007bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa76c88d700 10 client.24524.objecter ms_handle_connect 0x7fa7500796d0 2024-03-11T21:25:18.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.378+0000 7fa7677fe700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa7500796d0 0x7fa75007bb30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.382+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa758024ec0 con 0x7fa7500796d0 2024-03-11T21:25:18.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.382+0000 7fa76c88d700 10 client.24524.objecter ms_dispatch 0x7fa768001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.382+0000 7fa76c88d700 10 client.24524.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.382+0000 7fa76c88d700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 4 2024-03-11T21:25:18.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.382+0000 7fa76c88d700 15 client.24524.objecter _session_command_op_remove 7 1 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 10 client.24524.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 15 client.24524.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 20 client.24524.objecter _get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 20 client.24524.objecter _get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 15 client.24524.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 20 client.24524.objecter get_session s=0x7fa750079000 osd=7 3 2024-03-11T21:25:18.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 15 client.24524.objecter _session_command_op_assign 7 2 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 4 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 10 client.24524.objecter _send_command 2 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.394+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa76804faa0 con 0x7fa7500796d0 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76c88d700 1 -- 172.21.15.143:0/1192538975 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa76804faa0 con 0x7fa7500796d0 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76c88d700 10 client.24524.objecter ms_dispatch 0x7fa768001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76c88d700 10 client.24524.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76c88d700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 4 2024-03-11T21:25:18.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76c88d700 15 client.24524.objecter _session_command_op_remove 7 2 2024-03-11T21:25:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 10 librados: watch_flush enter 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 10 librados: watch_flush exit 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 10 client.24524.objecter close_session for osd.7 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa7500796d0 msgr2=0x7fa75007bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa7500796d0 0x7fa75007bb30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 20 client.24524.objecter put_session s=0x7fa750079000 osd=7 1 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 20 client.24524.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 10 client.24524.objecter successfully canceled tick 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa75006f820 msgr2=0x7fa750071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa75006f820 0x7fa750071ce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7681085b0 tx=0x7fa75c009260 comp rx=0 tx=0).stop 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 msgr2=0x7fa7680740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 --2- 172.21.15.143:0/1192538975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa768102120 0x7fa7680740e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099c0 tx=0x7fa75800fa00 comp rx=0 tx=0).stop 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa7677fe700 1 -- 172.21.15.143:0/1192538975 reap_dead start 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 shutdown_connections 2024-03-11T21:25:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 >> 172.21.15.143:0/1192538975 conn(0x7fa768078f40 msgr2=0x7fa7680ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 shutdown_connections 2024-03-11T21:25:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 -- 172.21.15.143:0/1192538975 wait complete. 2024-03-11T21:25:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.398+0000 7fa76d88f700 1 librados: shutdown 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 -- 172.21.15.143:0/677234747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258101ec0 msgr2=0x7f8258102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 --2- 172.21.15.143:0/677234747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258101ec0 0x7f8258102340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f82540098f0 tx=0x7f8254009c00 comp rx=0 tx=0).stop 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 -- 172.21.15.143:0/677234747 shutdown_connections 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 --2- 172.21.15.143:0/677234747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258107f30 0x7f8258108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 --2- 172.21.15.143:0/677234747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258102880 0x7f825810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 --2- 172.21.15.143:0/677234747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258101ec0 0x7f8258102340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 -- 172.21.15.143:0/677234747 >> 172.21.15.143:0/677234747 conn(0x7f8258078eb0 msgr2=0x7f82580792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 -- 172.21.15.143:0/677234747 shutdown_connections 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 -- 172.21.15.143:0/677234747 wait complete. 2024-03-11T21:25:18.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 librados: starting msgr at 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.678+0000 7f825f423700 1 librados: starting objecter 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 Processor -- start 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- start start 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 librados: setting wanted keys 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 librados: calling monclient init 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258101ec0 0x7f8258199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 0x7f825819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258107f30 0x7f825819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82581116f0 con 0x7f8258102880 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8258111570 con 0x7f8258101ec0 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8258111870 con 0x7f8258107f30 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 0x7f825819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d9c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258107f30 0x7f825819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 0x7f825819a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42132/0 (socket says 172.21.15.143:42132) 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 -- 172.21.15.143:0/4121814748 learned_addr learned my addr 172.21.15.143:0/4121814748 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d1bf700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258101ec0 0x7f8258199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 -- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258107f30 msgr2=0x7f825819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258107f30 0x7f825819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 -- 172.21.15.143:0/4121814748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258101ec0 msgr2=0x7f8258199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258101ec0 0x7f8258199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825819ee90 con 0x7f8258102880 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d9c0700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8258107f30 0x7f825819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:18.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d1bf700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258101ec0 0x7f8258199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825c9be700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 0x7f825819a3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f82540059d0 tx=0x7f825400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 10 client.?.objecter ms_handle_connect 0x7f8258102880 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254027070 con 0x7f8258102880 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f825400bb60 con 0x7f8258102880 2024-03-11T21:25:18.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825819f120 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254004930 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 10 client.14655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82581a6a00 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 librados: init done 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 10 client.14655.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 15 client.14655.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 20 client.14655.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 10 client.14655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8258103550 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f825402a070 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f824406f820 0x7f8244071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f825408edf0 con 0x7f8258102880 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 10 client.14655.objecter ms_dispatch 0x7f8258001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 3 client.14655.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 3 client.14655.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d1bf700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f824406f820 0x7f8244071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 10 client.14655.objecter checking command 1 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f8244079650 0x7f824407bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter _get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 15 client.14655.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter _get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 15 client.14655.objecter _session_command_op_assign 1 1 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 4 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 10 client.14655.objecter _send_command 1 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f825400bd80 con 0x7f8244079650 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 20 client.14655.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82540914f0 con 0x7f8258102880 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.682+0000 7f825d9c0700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f8244079650 0x7f824407bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f825d1bf700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f824406f820 0x7f8244071ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8248005fd0 tx=0x7f8248009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 10 client.14655.objecter ms_handle_connect 0x7f824406f820 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 10 client.14655.objecter ms_handle_connect 0x7f8244079650 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f825d9c0700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f8244079650 0x7f824407bab0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f825400bd80 con 0x7f8244079650 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 10 client.14655.objecter ms_dispatch 0x7f8258001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 10 client.14655.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 4 2024-03-11T21:25:18.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.686+0000 7f824e7fc700 15 client.14655.objecter _session_command_op_remove 1 1 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 client.14655.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 15 client.14655.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter _get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter _get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 15 client.14655.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter get_session s=0x7f8244078f80 osd=1 3 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 15 client.14655.objecter _session_command_op_assign 1 2 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 4 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 client.14655.objecter _send_command 2 2024-03-11T21:25:18.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f825804faa0 con 0x7f8244079650 2024-03-11T21:25:18.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f824e7fc700 1 -- 172.21.15.143:0/4121814748 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f825804faa0 con 0x7f8244079650 2024-03-11T21:25:18.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f824e7fc700 10 client.14655.objecter ms_dispatch 0x7f8258001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f824e7fc700 10 client.14655.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f824e7fc700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 4 2024-03-11T21:25:18.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f824e7fc700 15 client.14655.objecter _session_command_op_remove 1 2 2024-03-11T21:25:18.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 librados: watch_flush enter 2024-03-11T21:25:18.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 librados: watch_flush exit 2024-03-11T21:25:18.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 client.14655.objecter close_session for osd.1 2024-03-11T21:25:18.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f8244079650 msgr2=0x7f824407bab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f8244079650 0x7f824407bab0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter put_session s=0x7f8244078f80 osd=1 1 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 20 client.14655.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 10 client.14655.objecter successfully canceled tick 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f824406f820 msgr2=0x7f8244071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f824406f820 0x7f8244071ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8248005fd0 tx=0x7f8248009310 comp rx=0 tx=0).stop 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 msgr2=0x7f825819a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 --2- 172.21.15.143:0/4121814748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258102880 0x7f825819a3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f82540059d0 tx=0x7f825400fa70 comp rx=0 tx=0).stop 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825d9c0700 1 -- 172.21.15.143:0/4121814748 reap_dead start 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 shutdown_connections 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 >> 172.21.15.143:0/4121814748 conn(0x7f8258078eb0 msgr2=0x7f8258074010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 shutdown_connections 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 -- 172.21.15.143:0/4121814748 wait complete. 2024-03-11T21:25:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.702+0000 7f825f423700 1 librados: shutdown 2024-03-11T21:25:18.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- 172.21.15.143:0/4125900864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c106dc0 msgr2=0x7fd16c1071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- 172.21.15.143:0/4125900864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c106dc0 0x7fd16c1071a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd168009990 tx=0x7fd168009ca0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- 172.21.15.143:0/4125900864 shutdown_connections 2024-03-11T21:25:18.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- 172.21.15.143:0/4125900864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c101700 0x7fd16c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- 172.21.15.143:0/4125900864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c106dc0 0x7fd16c1071a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- 172.21.15.143:0/4125900864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 0x7fd16c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- 172.21.15.143:0/4125900864 >> 172.21.15.143:0/4125900864 conn(0x7fd16c076d80 msgr2=0x7fd16c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- 172.21.15.143:0/4125900864 shutdown_connections 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- 172.21.15.143:0/4125900864 wait complete. 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 librados: starting msgr at 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 librados: starting objecter 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 Processor -- start 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- start start 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 librados: setting wanted keys 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 librados: calling monclient init 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 0x7fd16c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c101700 0x7fd16c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 0x7fd16c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c110520 con 0x7fd16c106dc0 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1103a0 con 0x7fd16c100d40 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd173d25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1106a0 con 0x7fd16c101700 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1722c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 0x7fd16c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1722c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 0x7fd16c19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42146/0 (socket says 172.21.15.143:42146) 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1722c2700 1 -- 172.21.15.143:0/1331916824 learned_addr learned my addr 172.21.15.143:0/1331916824 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c101700 0x7fd16c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd171ac1700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 0x7fd16c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 -- 172.21.15.143:0/1331916824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 msgr2=0x7fd16c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 0x7fd16c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 -- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 msgr2=0x7fd16c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 0x7fd16c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.878+0000 7fd1712c0700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c19ed70 con 0x7fd16c101700 2024-03-11T21:25:18.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd1722c2700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16c106dc0 0x7fd16c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:18.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd171ac1700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd16c100d40 0x7fd16c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:25:18.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd1712c0700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c101700 0x7fd16c19a2a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c970 tx=0x7fd15c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.?.objecter ms_handle_connect 0x7fd16c101700 2024-03-11T21:25:18.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:18.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00ce60 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd15c004830 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0053a0 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c19f060 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 10 client.24536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c1a68e0 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 1 librados: init done 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 10 client.24536.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 15 client.24536.objecter _session_command_op_assign -1 1 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 20 client.24536.objecter _get_session osd=-1 returning homeless 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 10 client.24536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd16c102400 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd15c005540 con 0x7fd16c101700 2024-03-11T21:25:18.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15806f820 0x7fd158071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(112..112 src has 56..112) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fd15c08e5b0 con 0x7fd16c101700 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.24536.objecter ms_dispatch 0x7fd16c001550 osd_map(112..112 src has 56..112) v4 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 3 client.24536.objecter handle_osd_map got epochs [112,112] > 0 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 3 client.24536.objecter handle_osd_map decoding full epoch 112 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd171ac1700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15806f820 0x7fd158071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.24536.objecter checking command 1 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1580796d0 0x7fd15807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter _get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 15 client.24536.objecter _session_command_op_remove -1 1 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter _get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 15 client.24536.objecter _session_command_op_assign 3 1 2024-03-11T21:25:18.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 4 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.24536.objecter _send_command 1 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd15c004520 con 0x7fd1580796d0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 20 client.24536.objecter dump_active .. 0 homeless 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd15c08ea90 con 0x7fd16c101700 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd1722c2700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1580796d0 0x7fd15807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd162ffd700 10 client.24536.objecter ms_handle_connect 0x7fd15806f820 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.882+0000 7fd171ac1700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15806f820 0x7fd158071ce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd16800b2e0 tx=0x7fd16801a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd1722c2700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1580796d0 0x7fd15807bb30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 10 client.24536.objecter ms_handle_connect 0x7fd1580796d0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd15c004520 con 0x7fd1580796d0 2024-03-11T21:25:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 10 client.24536.objecter ms_dispatch 0x7fd16c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:25:18.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 10 client.24536.objecter _finish_command 1 = system:0 2024-03-11T21:25:18.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 4 2024-03-11T21:25:18.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.886+0000 7fd162ffd700 15 client.24536.objecter _session_command_op_remove 3 1 2024-03-11T21:25:18.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 client.24536.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 15 client.24536.objecter _session_command_op_assign -1 2 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter _get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter _get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 15 client.24536.objecter _session_command_op_remove -1 2 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter get_session s=0x7fd158079000 osd=3 3 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 15 client.24536.objecter _session_command_op_assign 3 2 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 4 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 client.24536.objecter _send_command 2 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd16c04faa0 con 0x7fd1580796d0 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd162ffd700 1 -- 172.21.15.143:0/1331916824 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd16c04faa0 con 0x7fd1580796d0 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd162ffd700 10 client.24536.objecter ms_dispatch 0x7fd16c001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:25:18.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd162ffd700 10 client.24536.objecter _finish_command 2 = system:0 2024-03-11T21:25:18.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd162ffd700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 4 2024-03-11T21:25:18.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd162ffd700 15 client.24536.objecter _session_command_op_remove 3 2 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 librados: watch_flush enter 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 librados: watch_flush exit 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 client.24536.objecter close_session for osd.3 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1580796d0 msgr2=0x7fd15807bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1580796d0 0x7fd15807bb30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter put_session s=0x7fd158079000 osd=3 1 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 20 client.24536.objecter shutdown clearing up homeless session... 2024-03-11T21:25:18.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 10 client.24536.objecter successfully canceled tick 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15806f820 msgr2=0x7fd158071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15806f820 0x7fd158071ce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd16800b2e0 tx=0x7fd16801a040 comp rx=0 tx=0).stop 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c101700 msgr2=0x7fd16c19a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 --2- 172.21.15.143:0/1331916824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd16c101700 0x7fd16c19a2a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c970 tx=0x7fd15c00cc80 comp rx=0 tx=0).stop 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd1722c2700 1 -- 172.21.15.143:0/1331916824 reap_dead start 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 shutdown_connections 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 >> 172.21.15.143:0/1331916824 conn(0x7fd16c076d80 msgr2=0x7fd16c0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 shutdown_connections 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 -- 172.21.15.143:0/1331916824 wait complete. 2024-03-11T21:25:18.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:18.902+0000 7fd173d25700 1 librados: shutdown 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[21512]: cluster 2024-03-11T21:25:17.082403+0000 mgr.y (mgr.14550) 73 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[21512]: cluster 2024-03-11T21:25:17.423501+0000 osd.5 (osd.5) 49 : cluster 0 2.12 scrub starts 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[21512]: cluster 2024-03-11T21:25:17.425931+0000 osd.5 (osd.5) 50 : cluster 0 2.12 scrub ok 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[29284]: cluster 2024-03-11T21:25:17.082403+0000 mgr.y (mgr.14550) 73 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[29284]: cluster 2024-03-11T21:25:17.423501+0000 osd.5 (osd.5) 49 : cluster 0 2.12 scrub starts 2024-03-11T21:25:18.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:18 smithi143 bash[29284]: cluster 2024-03-11T21:25:17.425931+0000 osd.5 (osd.5) 50 : cluster 0 2.12 scrub ok 2024-03-11T21:25:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:18 smithi203 bash[23654]: cluster 2024-03-11T21:25:17.082403+0000 mgr.y (mgr.14550) 73 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-03-11T21:25:19.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:18 smithi203 bash[23654]: cluster 2024-03-11T21:25:17.423501+0000 osd.5 (osd.5) 49 : cluster 0 2.12 scrub starts 2024-03-11T21:25:19.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:18 smithi203 bash[23654]: cluster 2024-03-11T21:25:17.425931+0000 osd.5 (osd.5) 50 : cluster 0 2.12 scrub ok 2024-03-11T21:25:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:19 smithi203 bash[23654]: audit 2024-03-11T21:25:18.125245+0000 mgr.y (mgr.14550) 74 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:19 smithi203 bash[23654]: cluster 2024-03-11T21:25:18.225415+0000 osd.7 (osd.7) 51 : cluster 0 3.f scrub starts 2024-03-11T21:25:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:19 smithi203 bash[23654]: cluster 2024-03-11T21:25:18.230025+0000 osd.7 (osd.7) 52 : cluster 0 3.f scrub ok 2024-03-11T21:25:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:19 smithi203 bash[23654]: cluster 2024-03-11T21:25:18.885361+0000 osd.3 (osd.3) 55 : cluster 0 2.1b scrub starts 2024-03-11T21:25:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:19 smithi203 bash[23654]: cluster 2024-03-11T21:25:18.886906+0000 osd.3 (osd.3) 56 : cluster 0 2.1b scrub ok 2024-03-11T21:25:20.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: audit 2024-03-11T21:25:18.125245+0000 mgr.y (mgr.14550) 74 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:18.225415+0000 osd.7 (osd.7) 51 : cluster 0 3.f scrub starts 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:18.230025+0000 osd.7 (osd.7) 52 : cluster 0 3.f scrub ok 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:18.885361+0000 osd.3 (osd.3) 55 : cluster 0 2.1b scrub starts 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:18.886906+0000 osd.3 (osd.3) 56 : cluster 0 2.1b scrub ok 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: audit 2024-03-11T21:25:18.125245+0000 mgr.y (mgr.14550) 74 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:18.225415+0000 osd.7 (osd.7) 51 : cluster 0 3.f scrub starts 2024-03-11T21:25:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:18.230025+0000 osd.7 (osd.7) 52 : cluster 0 3.f scrub ok 2024-03-11T21:25:20.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:18.885361+0000 osd.3 (osd.3) 55 : cluster 0 2.1b scrub starts 2024-03-11T21:25:20.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:18.886906+0000 osd.3 (osd.3) 56 : cluster 0 2.1b scrub ok 2024-03-11T21:25:20.958 INFO:teuthology.orchestra.run.smithi143.stdout:197568495673 2024-03-11T21:25:20.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:25:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:34359738478 2024-03-11T21:25:20.996 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:25:20.997 INFO:teuthology.orchestra.run.smithi143.stdout:150323855426 2024-03-11T21:25:20.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:25:21.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:20 smithi203 bash[23654]: cluster 2024-03-11T21:25:19.083796+0000 mgr.y (mgr.14550) 75 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:21.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:20 smithi203 bash[23654]: cluster 2024-03-11T21:25:19.437814+0000 osd.5 (osd.5) 51 : cluster 0 3.1c scrub starts 2024-03-11T21:25:21.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:20 smithi203 bash[23654]: cluster 2024-03-11T21:25:19.442336+0000 osd.5 (osd.5) 52 : cluster 0 3.1c scrub ok 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:19.083796+0000 mgr.y (mgr.14550) 75 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:19.437814+0000 osd.5 (osd.5) 51 : cluster 0 3.1c scrub starts 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[29284]: cluster 2024-03-11T21:25:19.442336+0000 osd.5 (osd.5) 52 : cluster 0 3.1c scrub ok 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:19.083796+0000 mgr.y (mgr.14550) 75 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:19.437814+0000 osd.5 (osd.5) 51 : cluster 0 3.1c scrub starts 2024-03-11T21:25:21.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:20 smithi143 bash[21512]: cluster 2024-03-11T21:25:19.442336+0000 osd.5 (osd.5) 52 : cluster 0 3.1c scrub ok 2024-03-11T21:25:21.719 INFO:teuthology.orchestra.run.smithi143.stdout:55834574947 2024-03-11T21:25:21.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:25:22.436 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:22.754 INFO:teuthology.orchestra.run.smithi143.stdout:240518168623 2024-03-11T21:25:22.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:25:23.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:21.084325+0000 mgr.y (mgr.14550) 76 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:23.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:21.285477+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:21.288608+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:21.385618+0000 osd.5 (osd.5) 53 : cluster 0 3.5 scrub starts 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:21.392617+0000 osd.5 (osd.5) 54 : cluster 0 3.5 scrub ok 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:22.243726+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:22.245927+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:22.356263+0000 osd.0 (osd.0) 49 : cluster 0 3.6 scrub starts 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[21512]: cluster 2024-03-11T21:25:22.360870+0000 osd.0 (osd.0) 50 : cluster 0 3.6 scrub ok 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:21.084325+0000 mgr.y (mgr.14550) 76 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:21.285477+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:21.288608+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-03-11T21:25:23.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:21.385618+0000 osd.5 (osd.5) 53 : cluster 0 3.5 scrub starts 2024-03-11T21:25:23.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:21.392617+0000 osd.5 (osd.5) 54 : cluster 0 3.5 scrub ok 2024-03-11T21:25:23.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:22.243726+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-03-11T21:25:23.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:22.245927+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-03-11T21:25:23.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:22.356263+0000 osd.0 (osd.0) 49 : cluster 0 3.6 scrub starts 2024-03-11T21:25:23.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:22 smithi143 bash[29284]: cluster 2024-03-11T21:25:22.360870+0000 osd.0 (osd.0) 50 : cluster 0 3.6 scrub ok 2024-03-11T21:25:23.048 INFO:teuthology.orchestra.run.smithi143.stdout:115964117069 2024-03-11T21:25:23.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:25:23.067 INFO:teuthology.orchestra.run.smithi143.stdout:283467841574 2024-03-11T21:25:23.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:25:23.148 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.148 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.149 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.149 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.149 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.149 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.150 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.150 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.150 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:21.084325+0000 mgr.y (mgr.14550) 76 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:23.150 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:21.285477+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:21.288608+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:21.385618+0000 osd.5 (osd.5) 53 : cluster 0 3.5 scrub starts 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:21.392617+0000 osd.5 (osd.5) 54 : cluster 0 3.5 scrub ok 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:22.243726+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:22.245927+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:22.356263+0000 osd.0 (osd.0) 49 : cluster 0 3.6 scrub starts 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:22 smithi203 bash[23654]: cluster 2024-03-11T21:25:22.360870+0000 osd.0 (osd.0) 50 : cluster 0 3.6 scrub ok 2024-03-11T21:25:23.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.402 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.402 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.402 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.403 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.404 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.404 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.404 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.404 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.404 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.405 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.405 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:25:23.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:23 smithi203 systemd[1]: Started Ceph grafana.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:25:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: cluster 2024-03-11T21:25:23.085155+0000 mgr.y (mgr.14550) 77 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:23.478529+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:23.505939+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:23.528815+0000 mon.a (mon.0) 844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:23.538783+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:23.606861+0000 mon.c (mon.2) 59 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:24.136363+0000 mon.a (mon.0) 846 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[23654]: audit 2024-03-11T21:25:24.151199+0000 mon.a (mon.0) 847 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.470909161Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.471776488Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.471869473Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.471931762Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472075165Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472132864Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472166213Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.47220222Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.47224958Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472276729Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472304856Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472327046Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472358503Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T21:25:24.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=settings t=2024-03-11T21:25:24.472386795Z level=info msg="App mode production" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=sqlstore t=2024-03-11T21:25:24.472537534Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=sqlstore t=2024-03-11T21:25:24.472606098Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.474474284Z level=info msg="Starting DB migrations" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.47467634Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.537393948Z level=info msg="Executing migration" id="create user table" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.579051391Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.604214794Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.629688462Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.654918076Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.680276437Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T21:25:24.712 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.73058006Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: cluster 2024-03-11T21:25:23.085155+0000 mgr.y (mgr.14550) 77 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:23.478529+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:23.505939+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:23.528815+0000 mon.a (mon.0) 844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:23.538783+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:23.606861+0000 mon.c (mon.2) 59 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:24.136363+0000 mon.a (mon.0) 846 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[21512]: audit 2024-03-11T21:25:24.151199+0000 mon.a (mon.0) 847 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: cluster 2024-03-11T21:25:23.085155+0000 mgr.y (mgr.14550) 77 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:23.478529+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:23.505939+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:23.528815+0000 mon.a (mon.0) 844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:23.538783+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:23.606861+0000 mon.c (mon.2) 59 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:24.136363+0000 mon.a (mon.0) 846 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:24 smithi143 bash[29284]: audit 2024-03-11T21:25:24.151199+0000 mon.a (mon.0) 847 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:24.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.755759828Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.79769178Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.823055238Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.848243642Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.89021191Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.915488956Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.94067083Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T21:25:24.962 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:24 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:24.982660911Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T21:25:25.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.007935695Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T21:25:25.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.033204319Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.058428409Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.08370148Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.10895606Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.159202376Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.201425023Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T21:25:25.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.243409683Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.285380619Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.310690424Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.352581093Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.394658807Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.428441245Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T21:25:25.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.453732935Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T21:25:25.744 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.49573727Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T21:25:25.744 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.512716025Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.529496223Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.546501893Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.571750874Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.596943675Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.622256515Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.647859612Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.672802318Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.71479726Z level=info msg="Executing migration" id="create star table" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.74006147Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T21:25:25.745 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.765396983Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T21:25:25.996 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.790585981Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.832643709Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.874656357Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.916713502Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.941971183Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.967270785Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:25 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:25.99250069Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T21:25:25.997 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.017702545Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T21:25:26.249 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.043022578Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.068303872Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.101890055Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.127185912Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.169217288Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.211184536Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.253105802Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T21:25:26.250 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.270394623Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T21:25:26.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[23654]: cluster 2024-03-11T21:25:25.085593+0000 mgr.y (mgr.14550) 78 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:25:26.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[23654]: cluster 2024-03-11T21:25:25.218385+0000 osd.7 (osd.7) 55 : cluster 0 2.1d scrub starts 2024-03-11T21:25:26.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[23654]: cluster 2024-03-11T21:25:25.220980+0000 osd.7 (osd.7) 56 : cluster 0 2.1d scrub ok 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.304036896Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.320962258Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.362917069Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.40487065Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.430135131Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.488705627Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.514127192Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T21:25:26.519 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.539371047Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T21:25:26.572 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.564619068Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.589877426Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.61515938Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.640339731Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.6654833Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.707430224Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.732850412Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T21:25:26.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.774874078Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T21:25:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[21512]: cluster 2024-03-11T21:25:25.085593+0000 mgr.y (mgr.14550) 78 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:25:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[21512]: cluster 2024-03-11T21:25:25.218385+0000 osd.7 (osd.7) 55 : cluster 0 2.1d scrub starts 2024-03-11T21:25:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[21512]: cluster 2024-03-11T21:25:25.220980+0000 osd.7 (osd.7) 56 : cluster 0 2.1d scrub ok 2024-03-11T21:25:26.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[29284]: cluster 2024-03-11T21:25:25.085593+0000 mgr.y (mgr.14550) 78 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:25:26.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[29284]: cluster 2024-03-11T21:25:25.218385+0000 osd.7 (osd.7) 55 : cluster 0 2.1d scrub starts 2024-03-11T21:25:26.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:26 smithi143 bash[29284]: cluster 2024-03-11T21:25:25.220980+0000 osd.7 (osd.7) 56 : cluster 0 2.1d scrub ok 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.816815967Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.833720116Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.850653481Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.892537061Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.917924626Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.959878284Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:26 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:26.985292912Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.010548717Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T21:25:27.057 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.052511872Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.077698266Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.103007273Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.128217119Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.170345046Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.195425981Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.237637845Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.279581377Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.304894761Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T21:25:27.326 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.346812655Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T21:25:27.594 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.594 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.595 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.595 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.595 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.595 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.388750144Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.422408167Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.447650569Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.472956277Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.514918073Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.540269249Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.565401133Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.590724518Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T21:25:27.596 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.616015655Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T21:25:27.600 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.641176105Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.666496623Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.72512233Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.75034862Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.817247946Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.842559094Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T21:25:27.847 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.867735134Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.909704226Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.935027363Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:27 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:27.977082431Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.019090036Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.036055035Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.069544976Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.086754057Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T21:25:28.110 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.131092447Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T21:25:28.371 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:28.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.022603+0000 osd.3 (osd.3) 57 : cluster 0 2.19 scrub starts 2024-03-11T21:25:28.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.025900+0000 osd.3 (osd.3) 58 : cluster 0 2.19 scrub ok 2024-03-11T21:25:28.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.086250+0000 mgr.y (mgr.14550) 79 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:28.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: audit 2024-03-11T21:25:27.207194+0000 mon.a (mon.0) 848 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:28.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: audit 2024-03-11T21:25:27.605363+0000 mon.c (mon.2) 60 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:28.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.902321+0000 osd.6 (osd.6) 31 : cluster 0 5.11 scrub starts 2024-03-11T21:25:28.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.904773+0000 osd.6 (osd.6) 32 : cluster 0 5.11 scrub ok 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.173089766Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.198458651Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.24068873Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.282772487Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.308068503Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T21:25:28.372 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.350066516Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.022603+0000 osd.3 (osd.3) 57 : cluster 0 2.19 scrub starts 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.025900+0000 osd.3 (osd.3) 58 : cluster 0 2.19 scrub ok 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.086250+0000 mgr.y (mgr.14550) 79 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: audit 2024-03-11T21:25:27.207194+0000 mon.a (mon.0) 848 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: audit 2024-03-11T21:25:27.605363+0000 mon.c (mon.2) 60 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.902321+0000 osd.6 (osd.6) 31 : cluster 0 5.11 scrub starts 2024-03-11T21:25:28.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.904773+0000 osd.6 (osd.6) 32 : cluster 0 5.11 scrub ok 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.022603+0000 osd.3 (osd.3) 57 : cluster 0 2.19 scrub starts 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.025900+0000 osd.3 (osd.3) 58 : cluster 0 2.19 scrub ok 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.086250+0000 mgr.y (mgr.14550) 79 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: audit 2024-03-11T21:25:27.207194+0000 mon.a (mon.0) 848 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: audit 2024-03-11T21:25:27.605363+0000 mon.c (mon.2) 60 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.902321+0000 osd.6 (osd.6) 31 : cluster 0 5.11 scrub starts 2024-03-11T21:25:28.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:28 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.904773+0000 osd.6 (osd.6) 32 : cluster 0 5.11 scrub ok 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.392112161Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.4173895Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.459473504Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.484808093Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.526726127Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.552196119Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.594158466Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.619477727Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T21:25:28.624 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.644694688Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T21:25:28.876 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.6701284Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.712061338Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.737353183Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.762665085Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.80472548Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.829882888Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.855342033Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.872197515Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T21:25:28.877 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.897790198Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T21:25:29.154 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.931469101Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T21:25:29.154 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.973503321Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T21:25:29.154 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:28 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:28.998798376Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T21:25:29.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.040820442Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T21:25:29.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.108092013Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T21:25:29.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.133327307Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T21:25:29.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.175426086Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T21:25:29.406 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.200623975Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.22582184Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.268011873Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.317096836Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.360847655Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.402779928Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.428109415Z level=info msg="Executing migration" id="create session table" 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.502544+0000 osd.5 (osd.5) 55 : cluster 0 3.16 scrub starts 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.506447+0000 osd.5 (osd.5) 56 : cluster 0 3.16 scrub ok 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.982142+0000 osd.3 (osd.3) 59 : cluster 0 6.b scrub starts 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:27.983530+0000 osd.3 (osd.3) 60 : cluster 0 6.b scrub ok 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: audit 2024-03-11T21:25:28.136511+0000 mgr.y (mgr.14550) 80 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:29.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:28.164528+0000 osd.7 (osd.7) 57 : cluster 0 2.b scrub starts 2024-03-11T21:25:29.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: cluster 2024-03-11T21:25:28.167025+0000 osd.7 (osd.7) 58 : cluster 0 2.b scrub ok 2024-03-11T21:25:29.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: audit 2024-03-11T21:25:28.243761+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[23654]: audit 2024-03-11T21:25:28.250707+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.502544+0000 osd.5 (osd.5) 55 : cluster 0 3.16 scrub starts 2024-03-11T21:25:29.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.506447+0000 osd.5 (osd.5) 56 : cluster 0 3.16 scrub ok 2024-03-11T21:25:29.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.982142+0000 osd.3 (osd.3) 59 : cluster 0 6.b scrub starts 2024-03-11T21:25:29.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:27.983530+0000 osd.3 (osd.3) 60 : cluster 0 6.b scrub ok 2024-03-11T21:25:29.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: audit 2024-03-11T21:25:28.136511+0000 mgr.y (mgr.14550) 80 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:28.164528+0000 osd.7 (osd.7) 57 : cluster 0 2.b scrub starts 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: cluster 2024-03-11T21:25:28.167025+0000 osd.7 (osd.7) 58 : cluster 0 2.b scrub ok 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: audit 2024-03-11T21:25:28.243761+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[29284]: audit 2024-03-11T21:25:28.250707+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.502544+0000 osd.5 (osd.5) 55 : cluster 0 3.16 scrub starts 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.506447+0000 osd.5 (osd.5) 56 : cluster 0 3.16 scrub ok 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.982142+0000 osd.3 (osd.3) 59 : cluster 0 6.b scrub starts 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:27.983530+0000 osd.3 (osd.3) 60 : cluster 0 6.b scrub ok 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: audit 2024-03-11T21:25:28.136511+0000 mgr.y (mgr.14550) 80 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:28.164528+0000 osd.7 (osd.7) 57 : cluster 0 2.b scrub starts 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: cluster 2024-03-11T21:25:28.167025+0000 osd.7 (osd.7) 58 : cluster 0 2.b scrub ok 2024-03-11T21:25:29.624 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: audit 2024-03-11T21:25:28.243761+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:29 smithi143 bash[21512]: audit 2024-03-11T21:25:28.250707+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.47013847Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.495527111Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.520813202Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.562866166Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.605079719Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.630329656Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T21:25:29.660 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.65586317Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T21:25:29.921 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.6811393Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T21:25:29.921 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.723260258Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T21:25:29.921 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.765263928Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T21:25:29.921 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.807325687Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T21:25:29.921 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.83267543Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T21:25:29.922 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.857988103Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T21:25:29.922 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.900090969Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T21:25:29.922 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.942264788Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T21:25:30.190 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:29 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:29.984365412Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.026763716Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.052133483Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.094173864Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.136143694Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.161412878Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T21:25:30.191 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.211906215Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T21:25:30.452 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.020963+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-03-11T21:25:30.452 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.024579+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-03-11T21:25:30.452 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.087411+0000 mgr.y (mgr.14550) 81 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:30.452 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.101432+0000 osd.2 (osd.2) 37 : cluster 0 2.10 scrub starts 2024-03-11T21:25:30.452 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.103766+0000 osd.2 (osd.2) 38 : cluster 0 2.10 scrub ok 2024-03-11T21:25:30.452 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.237202705Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T21:25:30.452 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.279252709Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T21:25:30.453 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.321355863Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T21:25:30.453 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.363343711Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T21:25:30.453 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.405448935Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T21:25:30.453 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.447761787Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T21:25:30.453 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.473035188Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.020963+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.024579+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.087411+0000 mgr.y (mgr.14550) 81 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.101432+0000 osd.2 (osd.2) 37 : cluster 0 2.10 scrub starts 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.103766+0000 osd.2 (osd.2) 38 : cluster 0 2.10 scrub ok 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.020963+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.024579+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-03-11T21:25:30.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.087411+0000 mgr.y (mgr.14550) 81 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:30.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.101432+0000 osd.2 (osd.2) 37 : cluster 0 2.10 scrub starts 2024-03-11T21:25:30.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:30 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.103766+0000 osd.2 (osd.2) 38 : cluster 0 2.10 scrub ok 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.498310491Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.523544973Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.548857709Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.591163697Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.616399062Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.641656512Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.666741232Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.708748883Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T21:25:30.713 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.73419899Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T21:25:30.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.776162778Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T21:25:30.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.801382085Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.82672736Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.868859966Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.894243655Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.919399674Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:30 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:30.961417746Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T21:25:30.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.003561616Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T21:25:31.233 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.507576+0000 osd.1 (osd.1) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:31.233 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: cluster 2024-03-11T21:25:29.507688+0000 osd.1 (osd.1) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:31.233 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.127896+0000 osd.2 (osd.2) 39 : cluster 0 5.b deep-scrub starts 2024-03-11T21:25:31.233 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.130040+0000 osd.2 (osd.2) 40 : cluster 0 5.b deep-scrub ok 2024-03-11T21:25:31.233 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.262980+0000 mon.a (mon.0) 851 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:25:31.234 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: audit 2024-03-11T21:25:30.274925+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.234 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: audit 2024-03-11T21:25:30.291111+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.234 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: audit 2024-03-11T21:25:31.017237+0000 mon.c (mon.2) 61 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.234 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: audit 2024-03-11T21:25:31.019200+0000 mon.c (mon.2) 62 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.045694294Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.062647885Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.09615955Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.130072009Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.163875076Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.180984356Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T21:25:31.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.231574866Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T21:25:31.493 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.507576+0000 osd.1 (osd.1) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:31.493 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: cluster 2024-03-11T21:25:29.507688+0000 osd.1 (osd.1) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:31.493 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.127896+0000 osd.2 (osd.2) 39 : cluster 0 5.b deep-scrub starts 2024-03-11T21:25:31.493 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.130040+0000 osd.2 (osd.2) 40 : cluster 0 5.b deep-scrub ok 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.262980+0000 mon.a (mon.0) 851 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: audit 2024-03-11T21:25:30.274925+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: audit 2024-03-11T21:25:30.291111+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: audit 2024-03-11T21:25:31.017237+0000 mon.c (mon.2) 61 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: audit 2024-03-11T21:25:31.019200+0000 mon.c (mon.2) 62 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[29284]: audit 2024-03-11T21:25:31.030611+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.507576+0000 osd.1 (osd.1) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: cluster 2024-03-11T21:25:29.507688+0000 osd.1 (osd.1) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.127896+0000 osd.2 (osd.2) 39 : cluster 0 5.b deep-scrub starts 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.130040+0000 osd.2 (osd.2) 40 : cluster 0 5.b deep-scrub ok 2024-03-11T21:25:31.494 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.262980+0000 mon.a (mon.0) 851 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:25:31.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: audit 2024-03-11T21:25:30.274925+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: audit 2024-03-11T21:25:30.291111+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: audit 2024-03-11T21:25:31.017237+0000 mon.c (mon.2) 61 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: audit 2024-03-11T21:25:31.019200+0000 mon.c (mon.2) 62 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:31 smithi143 bash[21512]: audit 2024-03-11T21:25:31.030611+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.505 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[23654]: audit 2024-03-11T21:25:31.030611+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.273768295Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.299292362Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.324598089Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.36654291Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.391916046Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.433881707Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T21:25:31.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.475807535Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T21:25:31.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.50113387Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T21:25:31.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.526376765Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T21:25:31.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.551628559Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T21:25:31.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.576951983Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T21:25:31.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.618913951Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T21:25:31.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.644331633Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T21:25:31.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.677907511Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T21:25:31.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.719906566Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T21:25:31.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.761879932Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T21:25:32.053 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.803920669Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T21:25:32.053 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.829239849Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T21:25:32.053 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.854295847Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.896358576Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.938411219Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:31 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:31.980597166Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.006113433Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.048129314Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T21:25:32.054 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.073452015Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T21:25:32.189 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.115423788Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.140816354Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.182789149Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.22481443Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.250140394Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.29228293Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T21:25:32.339 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.317485693Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.695375+0000 osd.1 (osd.1) 53 : cluster 0 2.d scrub starts 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.696733+0000 osd.1 (osd.1) 54 : cluster 0 2.d scrub ok 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.944524+0000 osd.3 (osd.3) 63 : cluster 0 3.18 scrub starts 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.949058+0000 osd.3 (osd.3) 64 : cluster 0 3.18 scrub ok 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.960539+0000 osd.4 (osd.4) 41 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:30.974530+0000 osd.4 (osd.4) 42 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:31.087923+0000 mgr.y (mgr.14550) 82 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cephadm 2024-03-11T21:25:31.108299+0000 mgr.y (mgr.14550) 83 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cephadm 2024-03-11T21:25:31.117577+0000 mgr.y (mgr.14550) 84 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi143 2024-03-11T21:25:32.340 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:31.135828+0000 osd.2 (osd.2) 41 : cluster 0 5.a scrub starts 2024-03-11T21:25:32.341 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[23654]: cluster 2024-03-11T21:25:31.137029+0000 osd.2 (osd.2) 42 : cluster 0 5.a scrub ok 2024-03-11T21:25:32.599 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.359495461Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.401459353Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.44354645Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.468975742Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.49415202Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.519394573Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.544708511Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.569972263Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T21:25:32.600 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.595135354Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T21:25:32.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.695375+0000 osd.1 (osd.1) 53 : cluster 0 2.d scrub starts 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.696733+0000 osd.1 (osd.1) 54 : cluster 0 2.d scrub ok 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.944524+0000 osd.3 (osd.3) 63 : cluster 0 3.18 scrub starts 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.949058+0000 osd.3 (osd.3) 64 : cluster 0 3.18 scrub ok 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.960539+0000 osd.4 (osd.4) 41 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:30.974530+0000 osd.4 (osd.4) 42 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:31.087923+0000 mgr.y (mgr.14550) 82 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cephadm 2024-03-11T21:25:31.108299+0000 mgr.y (mgr.14550) 83 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cephadm 2024-03-11T21:25:31.117577+0000 mgr.y (mgr.14550) 84 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi143 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:31.135828+0000 osd.2 (osd.2) 41 : cluster 0 5.a scrub starts 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[21512]: cluster 2024-03-11T21:25:31.137029+0000 osd.2 (osd.2) 42 : cluster 0 5.a scrub ok 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.695375+0000 osd.1 (osd.1) 53 : cluster 0 2.d scrub starts 2024-03-11T21:25:32.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.696733+0000 osd.1 (osd.1) 54 : cluster 0 2.d scrub ok 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.944524+0000 osd.3 (osd.3) 63 : cluster 0 3.18 scrub starts 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.949058+0000 osd.3 (osd.3) 64 : cluster 0 3.18 scrub ok 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.960539+0000 osd.4 (osd.4) 41 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:30.974530+0000 osd.4 (osd.4) 42 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:31.087923+0000 mgr.y (mgr.14550) 82 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cephadm 2024-03-11T21:25:31.108299+0000 mgr.y (mgr.14550) 83 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cephadm 2024-03-11T21:25:31.117577+0000 mgr.y (mgr.14550) 84 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi143 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:31.135828+0000 osd.2 (osd.2) 41 : cluster 0 5.a scrub starts 2024-03-11T21:25:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:32 smithi143 bash[29284]: cluster 2024-03-11T21:25:31.137029+0000 osd.2 (osd.2) 42 : cluster 0 5.a scrub ok 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.620588336Z level=info msg="Executing migration" id="create team table" 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.695958665Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.721304618Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.763190788Z level=info msg="Executing migration" id="create team member table" 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.788476347Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T21:25:32.894 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.81369875Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T21:25:32.895 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.839048448Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T21:25:32.895 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.864332579Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T21:25:32.895 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.914918933Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.956904721Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:32 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:32.998902Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.024236441Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.049511734Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.091616128Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.133589854Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T21:25:33.155 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.175875839Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T21:25:33.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[23654]: cluster 2024-03-11T21:25:31.914104+0000 osd.3 (osd.3) 65 : cluster 0 3.1e scrub starts 2024-03-11T21:25:33.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[23654]: cluster 2024-03-11T21:25:31.918654+0000 osd.3 (osd.3) 66 : cluster 0 3.1e scrub ok 2024-03-11T21:25:33.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[23654]: cluster 2024-03-11T21:25:32.116350+0000 osd.2 (osd.2) 43 : cluster 0 6.f scrub starts 2024-03-11T21:25:33.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[23654]: cluster 2024-03-11T21:25:32.119305+0000 osd.2 (osd.2) 44 : cluster 0 6.f scrub ok 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.201224247Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.226470104Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.251707641Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.29386345Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.319050455Z level=info msg="Executing migration" id="create tag table" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.361053578Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.403164115Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T21:25:33.408 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.428468931Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T21:25:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[21512]: cluster 2024-03-11T21:25:31.914104+0000 osd.3 (osd.3) 65 : cluster 0 3.1e scrub starts 2024-03-11T21:25:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[21512]: cluster 2024-03-11T21:25:31.918654+0000 osd.3 (osd.3) 66 : cluster 0 3.1e scrub ok 2024-03-11T21:25:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[21512]: cluster 2024-03-11T21:25:32.116350+0000 osd.2 (osd.2) 43 : cluster 0 6.f scrub starts 2024-03-11T21:25:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[21512]: cluster 2024-03-11T21:25:32.119305+0000 osd.2 (osd.2) 44 : cluster 0 6.f scrub ok 2024-03-11T21:25:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[29284]: cluster 2024-03-11T21:25:31.914104+0000 osd.3 (osd.3) 65 : cluster 0 3.1e scrub starts 2024-03-11T21:25:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[29284]: cluster 2024-03-11T21:25:31.918654+0000 osd.3 (osd.3) 66 : cluster 0 3.1e scrub ok 2024-03-11T21:25:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[29284]: cluster 2024-03-11T21:25:32.116350+0000 osd.2 (osd.2) 43 : cluster 0 6.f scrub starts 2024-03-11T21:25:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:33 smithi143 bash[29284]: cluster 2024-03-11T21:25:32.119305+0000 osd.2 (osd.2) 44 : cluster 0 6.f scrub ok 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.453631806Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.478963744Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.512479126Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.537824578Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.56314061Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.588354634Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.613571684Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.63888082Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.655836372Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.672732142Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T21:25:33.694 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.714932031Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T21:25:33.946 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.740199578Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.782274029Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.807684356Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.832940759Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.858188136Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.883453172Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.925413442Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T21:25:33.947 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:33 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:33.967499307Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T21:25:34.244 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.00948221Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T21:25:34.244 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.034771427Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T21:25:34.244 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.076738498Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T21:25:34.244 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.102019435Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T21:25:34.244 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.144049965Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T21:25:34.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[21512]: cluster 2024-03-11T21:25:32.392785+0000 osd.5 (osd.5) 57 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:25:34.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[21512]: cluster 2024-03-11T21:25:32.395306+0000 osd.5 (osd.5) 58 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:25:34.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[21512]: cluster 2024-03-11T21:25:33.089250+0000 mgr.y (mgr.14550) 85 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:34.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[21512]: cluster 2024-03-11T21:25:34.053821+0000 osd.7 (osd.7) 59 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:25:34.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[21512]: cluster 2024-03-11T21:25:34.058803+0000 osd.7 (osd.7) 60 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:25:34.407 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[29284]: cluster 2024-03-11T21:25:32.392785+0000 osd.5 (osd.5) 57 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:25:34.407 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[29284]: cluster 2024-03-11T21:25:32.395306+0000 osd.5 (osd.5) 58 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:25:34.407 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[29284]: cluster 2024-03-11T21:25:33.089250+0000 mgr.y (mgr.14550) 85 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:34.407 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[29284]: cluster 2024-03-11T21:25:34.053821+0000 osd.7 (osd.7) 59 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:25:34.407 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:34 smithi143 bash[29284]: cluster 2024-03-11T21:25:34.058803+0000 osd.7 (osd.7) 60 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:25:34.508 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.271241618Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T21:25:34.508 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.320109606Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T21:25:34.508 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.395378833Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T21:25:34.508 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.481390751Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T21:25:34.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[23654]: cluster 2024-03-11T21:25:32.392785+0000 osd.5 (osd.5) 57 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:25:34.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[23654]: cluster 2024-03-11T21:25:32.395306+0000 osd.5 (osd.5) 58 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:25:34.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[23654]: cluster 2024-03-11T21:25:33.089250+0000 mgr.y (mgr.14550) 85 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:34.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[23654]: cluster 2024-03-11T21:25:34.053821+0000 osd.7 (osd.7) 59 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:25:34.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[23654]: cluster 2024-03-11T21:25:34.058803+0000 osd.7 (osd.7) 60 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.529500403Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.554787107Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.580038824Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.613508317Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.638860037Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.66411474Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T21:25:34.761 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.70617049Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T21:25:34.762 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.756795339Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T21:25:34.762 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.782095096Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.807310921Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.832702723Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.891296795Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.908314244Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.950490671Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:34 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:34.975847397Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.001060462Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T21:25:35.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.043176454Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T21:25:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[23654]: cluster 2024-03-11T21:25:33.755199+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-03-11T21:25:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[23654]: cluster 2024-03-11T21:25:33.757429+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.085181877Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.127180945Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.169154344Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.194517776Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.219771695Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T21:25:35.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.245100401Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T21:25:35.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:35 smithi143 bash[21512]: cluster 2024-03-11T21:25:33.755199+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-03-11T21:25:35.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:35 smithi143 bash[21512]: cluster 2024-03-11T21:25:33.757429+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-03-11T21:25:35.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:35 smithi143 bash[29284]: cluster 2024-03-11T21:25:33.755199+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-03-11T21:25:35.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:35 smithi143 bash[29284]: cluster 2024-03-11T21:25:33.757429+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-03-11T21:25:35.552 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.304008491Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T21:25:35.552 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.362672465Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T21:25:35.552 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.404693543Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.429929568Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.472069731Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.497335206Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.522630746Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.547986769Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T21:25:35.553 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.573306883Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.607098548Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.632449725Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.657696343Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.699774163Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.733408848Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.758724188Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.800809009Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T21:25:35.805 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.826095116Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.851501912Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.876723927Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.901998631Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.927331205Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.969344976Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:35 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:35.986376302Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.028439643Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.070401927Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T21:25:36.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.095756314Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T21:25:36.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[23654]: cluster 2024-03-11T21:25:34.707554+0000 osd.1 (osd.1) 57 : cluster 0 2.9 scrub starts 2024-03-11T21:25:36.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[23654]: cluster 2024-03-11T21:25:34.709950+0000 osd.1 (osd.1) 58 : cluster 0 2.9 scrub ok 2024-03-11T21:25:36.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[23654]: cluster 2024-03-11T21:25:35.089902+0000 mgr.y (mgr.14550) 86 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:36.336 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.121139511Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T21:25:36.336 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.146376766Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T21:25:36.336 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.171661668Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.213551857Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.23892883Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.264218695Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.30630382Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.331565929Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T21:25:36.337 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.357195611Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T21:25:36.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[29284]: cluster 2024-03-11T21:25:34.707554+0000 osd.1 (osd.1) 57 : cluster 0 2.9 scrub starts 2024-03-11T21:25:36.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[29284]: cluster 2024-03-11T21:25:34.709950+0000 osd.1 (osd.1) 58 : cluster 0 2.9 scrub ok 2024-03-11T21:25:36.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[29284]: cluster 2024-03-11T21:25:35.089902+0000 mgr.y (mgr.14550) 86 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:36.509 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[21512]: cluster 2024-03-11T21:25:34.707554+0000 osd.1 (osd.1) 57 : cluster 0 2.9 scrub starts 2024-03-11T21:25:36.510 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[21512]: cluster 2024-03-11T21:25:34.709950+0000 osd.1 (osd.1) 58 : cluster 0 2.9 scrub ok 2024-03-11T21:25:36.510 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:36 smithi143 bash[21512]: cluster 2024-03-11T21:25:35.089902+0000 mgr.y (mgr.14550) 86 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:36.605 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.399188334Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.424448753Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.466515038Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.491842809Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.517086852Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.559088671Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.601214372Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T21:25:36.606 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.626452395Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.651707117Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.676995158Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.718977851Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.760918063Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.76124151Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.78623189Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.811411984Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.836791077Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.837291945Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.837650088Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.853555711Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T21:25:36.875 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.895706109Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.921022542Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.963049025Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:36 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:36.988435879Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.03040071Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.072589747Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.097747253Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T21:25:37.127 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.123116662Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T21:25:37.128 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.14836281Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T21:25:37.396 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[23654]: cluster 2024-03-11T21:25:36.942365+0000 osd.6 (osd.6) 33 : cluster 0 2.11 scrub starts 2024-03-11T21:25:37.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[23654]: cluster 2024-03-11T21:25:36.945572+0000 osd.6 (osd.6) 34 : cluster 0 2.11 scrub ok 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.207447139Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.232781434Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.258010928Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.31698414Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.375556358Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T21:25:37.397 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.417778879Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T21:25:37.674 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.443104904Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.485245952Z level=info msg="Executing migration" id="create permission table" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.552356849Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.577556746Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.603002659Z level=info msg="Executing migration" id="create role table" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.628248286Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T21:25:37.675 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.65341815Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T21:25:37.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:37 smithi143 bash[21512]: cluster 2024-03-11T21:25:36.942365+0000 osd.6 (osd.6) 33 : cluster 0 2.11 scrub starts 2024-03-11T21:25:37.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:37 smithi143 bash[21512]: cluster 2024-03-11T21:25:36.945572+0000 osd.6 (osd.6) 34 : cluster 0 2.11 scrub ok 2024-03-11T21:25:37.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:37 smithi143 bash[29284]: cluster 2024-03-11T21:25:36.942365+0000 osd.6 (osd.6) 33 : cluster 0 2.11 scrub starts 2024-03-11T21:25:37.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:37 smithi143 bash[29284]: cluster 2024-03-11T21:25:36.945572+0000 osd.6 (osd.6) 34 : cluster 0 2.11 scrub ok 2024-03-11T21:25:37.902 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.695640374Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T21:25:37.902 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.721038577Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.754648508Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.771483821Z level=info msg="Executing migration" id="create team role table" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.796939141Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.822141774Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.847409598Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.872666287Z level=info msg="Executing migration" id="create user role table" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.89793362Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T21:25:37.903 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.923298937Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T21:25:38.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:38.197 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.948585259Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T21:25:38.197 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:37 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:37.990897082Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T21:25:38.197 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.01625642Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T21:25:38.197 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.041427416Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T21:25:38.198 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.083635814Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T21:25:38.198 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.108920135Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T21:25:38.198 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.150987595Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T21:25:38.198 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.176365207Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T21:25:38.198 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.218341386Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.035667+0000 osd.5 (osd.5) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.035771+0000 osd.5 (osd.5) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.090658+0000 mgr.y (mgr.14550) 87 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.298831+0000 mon.a (mon.0) 855 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.330357+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.333845+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.956423+0000 osd.7 (osd.7) 61 : cluster 0 3.15 scrub starts 2024-03-11T21:25:38.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.960046+0000 osd.7 (osd.7) 62 : cluster 0 3.15 scrub ok 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.243665312Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.269007328Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.311162484Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.353164886Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.395295469Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.437366324Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.462569831Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T21:25:38.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.487902401Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.035667+0000 osd.5 (osd.5) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.035771+0000 osd.5 (osd.5) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.090658+0000 mgr.y (mgr.14550) 87 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.298831+0000 mon.a (mon.0) 855 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.330357+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.333845+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.956423+0000 osd.7 (osd.7) 61 : cluster 0 3.15 scrub starts 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.960046+0000 osd.7 (osd.7) 62 : cluster 0 3.15 scrub ok 2024-03-11T21:25:38.541 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.035667+0000 osd.5 (osd.5) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.035771+0000 osd.5 (osd.5) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.090658+0000 mgr.y (mgr.14550) 87 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.298831+0000 mon.a (mon.0) 855 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.330357+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.333845+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.956423+0000 osd.7 (osd.7) 61 : cluster 0 3.15 scrub starts 2024-03-11T21:25:38.542 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:38 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.960046+0000 osd.7 (osd.7) 62 : cluster 0 3.15 scrub ok 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.513158188Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.538394315Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.580472903Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.622541808Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.647849957Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.664774209Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T21:25:38.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.681580556Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T21:25:38.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.715347649Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T21:25:38.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.749305773Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T21:25:39.014 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.774535672Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T21:25:39.014 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.799806415Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.825023529Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.850344872Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.875642698Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.876003938Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.900886432Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.901230689Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.926119239Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.951417439Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:38 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:38.993381398Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:25:39.015 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.035496029Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:25:39.268 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.077594201Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.119670437Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.14502197Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.187093099Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.212149726Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.245909975Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:25:39.269 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.271205054Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T21:25:39.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.664272+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.669201+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.977861+0000 osd.4 (osd.4) 43 : cluster 0 2.f scrub starts 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:37.981006+0000 osd.4 (osd.4) 44 : cluster 0 2.f scrub ok 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: audit 2024-03-11T21:25:38.147849+0000 mgr.y (mgr.14550) 88 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:39.005862+0000 osd.7 (osd.7) 63 : cluster 0 3.13 scrub starts 2024-03-11T21:25:39.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[23654]: cluster 2024-03-11T21:25:39.009624+0000 osd.7 (osd.7) 64 : cluster 0 3.13 scrub ok 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.321736999Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.355398327Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.397323543Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.422758283Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.464687832Z level=info msg="Executing migration" id="add share column" 2024-03-11T21:25:39.520 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.490015975Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T21:25:39.521 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.515303352Z level=info msg="Executing migration" id="create file table" 2024-03-11T21:25:39.521 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.540617646Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T21:25:39.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.664272+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.669201+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.977861+0000 osd.4 (osd.4) 43 : cluster 0 2.f scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:37.981006+0000 osd.4 (osd.4) 44 : cluster 0 2.f scrub ok 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: audit 2024-03-11T21:25:38.147849+0000 mgr.y (mgr.14550) 88 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:39.005862+0000 osd.7 (osd.7) 63 : cluster 0 3.13 scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[21512]: cluster 2024-03-11T21:25:39.009624+0000 osd.7 (osd.7) 64 : cluster 0 3.13 scrub ok 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.664272+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.669201+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.977861+0000 osd.4 (osd.4) 43 : cluster 0 2.f scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:37.981006+0000 osd.4 (osd.4) 44 : cluster 0 2.f scrub ok 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: audit 2024-03-11T21:25:38.147849+0000 mgr.y (mgr.14550) 88 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:39.005862+0000 osd.7 (osd.7) 63 : cluster 0 3.13 scrub starts 2024-03-11T21:25:39.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:39 smithi143 bash[29284]: cluster 2024-03-11T21:25:39.009624+0000 osd.7 (osd.7) 64 : cluster 0 3.13 scrub ok 2024-03-11T21:25:39.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.591219363Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T21:25:39.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.633346434Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T21:25:39.772 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.658577749Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T21:25:39.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.68384654Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T21:25:39.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.725825551Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T21:25:39.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.751191593Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T21:25:39.773 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.776439758Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.843496849Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.868927291Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.894157045Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.936144529Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:39 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:39.961401487Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.003385691Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.045402602Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T21:25:40.075 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.095835537Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T21:25:40.343 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.121235474Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.146445306Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.213363482Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.238728029Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.263922242Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.322884839Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.364729067Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T21:25:40.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[23654]: cluster 2024-03-11T21:25:38.969615+0000 osd.3 (osd.3) 67 : cluster 0 5.14 scrub starts 2024-03-11T21:25:40.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[23654]: cluster 2024-03-11T21:25:38.972789+0000 osd.3 (osd.3) 68 : cluster 0 5.14 scrub ok 2024-03-11T21:25:40.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[23654]: cluster 2024-03-11T21:25:38.979863+0000 osd.4 (osd.4) 45 : cluster 0 3.14 scrub starts 2024-03-11T21:25:40.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[23654]: cluster 2024-03-11T21:25:38.983848+0000 osd.4 (osd.4) 46 : cluster 0 3.14 scrub ok 2024-03-11T21:25:40.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[23654]: cluster 2024-03-11T21:25:39.092366+0000 mgr.y (mgr.14550) 89 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.398511764Z level=info msg="Executing migration" id="create folder table" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.432197366Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.457745586Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.483009354Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.508251766Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=migrator t=2024-03-11T21:25:40.533430134Z level=info msg="migrations completed" performed=485 skipped=0 duration=16.058812374s 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=sqlstore t=2024-03-11T21:25:40.534788805Z level=info msg="Created default organization" 2024-03-11T21:25:40.661 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=plugin.loader t=2024-03-11T21:25:40.681996324Z level=info msg="Plugin registered" pluginID=input 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[21512]: cluster 2024-03-11T21:25:38.969615+0000 osd.3 (osd.3) 67 : cluster 0 5.14 scrub starts 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[21512]: cluster 2024-03-11T21:25:38.972789+0000 osd.3 (osd.3) 68 : cluster 0 5.14 scrub ok 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[21512]: cluster 2024-03-11T21:25:38.979863+0000 osd.4 (osd.4) 45 : cluster 0 3.14 scrub starts 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[21512]: cluster 2024-03-11T21:25:38.983848+0000 osd.4 (osd.4) 46 : cluster 0 3.14 scrub ok 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[21512]: cluster 2024-03-11T21:25:39.092366+0000 mgr.y (mgr.14550) 89 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[29284]: cluster 2024-03-11T21:25:38.969615+0000 osd.3 (osd.3) 67 : cluster 0 5.14 scrub starts 2024-03-11T21:25:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[29284]: cluster 2024-03-11T21:25:38.972789+0000 osd.3 (osd.3) 68 : cluster 0 5.14 scrub ok 2024-03-11T21:25:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[29284]: cluster 2024-03-11T21:25:38.979863+0000 osd.4 (osd.4) 45 : cluster 0 3.14 scrub starts 2024-03-11T21:25:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[29284]: cluster 2024-03-11T21:25:38.983848+0000 osd.4 (osd.4) 46 : cluster 0 3.14 scrub ok 2024-03-11T21:25:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:40 smithi143 bash[29284]: cluster 2024-03-11T21:25:39.092366+0000 mgr.y (mgr.14550) 89 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:25:41.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=plugin.loader t=2024-03-11T21:25:40.712300865Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T21:25:41.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=plugin.loader t=2024-03-11T21:25:40.7123282Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T21:25:41.022 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=secrets t=2024-03-11T21:25:40.712375058Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T21:25:41.023 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=query_data t=2024-03-11T21:25:40.717952683Z level=info msg="Query Service initialization" 2024-03-11T21:25:41.023 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:40 smithi203 bash[51372]: logger=live.push_http t=2024-03-11T21:25:40.72582844Z level=info msg="Live Push Gateway initialization" 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:39.971852+0000 osd.3 (osd.3) 69 : cluster 0 5.13 scrub starts 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:39.973968+0000 osd.3 (osd.3) 70 : cluster 0 5.13 scrub ok 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.276929+0000 osd.0 (osd.0) 53 : cluster 0 5.1e scrub starts 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.279698+0000 osd.0 (osd.0) 54 : cluster 0 5.1e scrub ok 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.943748+0000 osd.6 (osd.6) 35 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:25:41.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.946293+0000 osd.6 (osd.6) 36 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:39.971852+0000 osd.3 (osd.3) 69 : cluster 0 5.13 scrub starts 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:39.973968+0000 osd.3 (osd.3) 70 : cluster 0 5.13 scrub ok 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.276929+0000 osd.0 (osd.0) 53 : cluster 0 5.1e scrub starts 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.279698+0000 osd.0 (osd.0) 54 : cluster 0 5.1e scrub ok 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.943748+0000 osd.6 (osd.6) 35 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:25:41.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:41 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.946293+0000 osd.6 (osd.6) 36 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:25:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:39.971852+0000 osd.3 (osd.3) 69 : cluster 0 5.13 scrub starts 2024-03-11T21:25:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:39.973968+0000 osd.3 (osd.3) 70 : cluster 0 5.13 scrub ok 2024-03-11T21:25:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.276929+0000 osd.0 (osd.0) 53 : cluster 0 5.1e scrub starts 2024-03-11T21:25:41.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.279698+0000 osd.0 (osd.0) 54 : cluster 0 5.1e scrub ok 2024-03-11T21:25:41.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.943748+0000 osd.6 (osd.6) 35 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:25:41.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.946293+0000 osd.6 (osd.6) 36 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:25:42.198 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:42.272 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[51372]: logger=infra.usagestats.collector t=2024-03-11T21:25:41.910650779Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[51372]: logger=provisioning.datasources t=2024-03-11T21:25:41.911932776Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:41 smithi203 bash[51372]: logger=provisioning.datasources t=2024-03-11T21:25:41.951218125Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=provisioning.alerting t=2024-03-11T21:25:42.002213935Z level=info msg="starting to provision alerting" 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=provisioning.alerting t=2024-03-11T21:25:42.002257109Z level=info msg="finished to provision alerting" 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=ngalert.state.manager t=2024-03-11T21:25:42.002540101Z level=info msg="Warming state cache for startup" 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=grafanaStorageLogger t=2024-03-11T21:25:42.002998797Z level=info msg="storage starting" 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=http.server t=2024-03-11T21:25:42.007492146Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=ngalert.state.manager t=2024-03-11T21:25:42.126313863Z level=info msg="State cache has been initialized" states=0 duration=123.771662ms 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=ngalert.multiorg.alertmanager t=2024-03-11T21:25:42.126428531Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=ticker t=2024-03-11T21:25:42.12649277Z level=info msg=starting first_tick=2024-03-11T21:25:50Z 2024-03-11T21:25:42.273 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[51372]: logger=sqlstore.transactions t=2024-03-11T21:25:42.138836474Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- 172.21.15.143:0/3601095834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a18069d70 msgr2=0x7f6a1806a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/3601095834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a18069d70 0x7f6a1806a150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000b0d0 tx=0x7f6a1000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- 172.21.15.143:0/3601095834 shutdown_connections 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/3601095834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a181071c0 0x7f6a1810ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/3601095834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a18069d70 0x7f6a1806a150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/3601095834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 0x7f6a18106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- 172.21.15.143:0/3601095834 >> 172.21.15.143:0/3601095834 conn(0x7f6a18076d80 msgr2=0x7f6a18077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- 172.21.15.143:0/3601095834 shutdown_connections 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- 172.21.15.143:0/3601095834 wait complete. 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 librados: starting msgr at 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 librados: starting objecter 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 Processor -- start 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- start start 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 librados: setting wanted keys 2024-03-11T21:25:42.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 librados: calling monclient init 2024-03-11T21:25:42.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a18069d70 0x7f6a18197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 0x7f6a18198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18111670 con 0x7f6a18069d70 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a181114f0 con 0x7f6a1806a720 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1ff78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a181117f0 con 0x7f6a181071c0 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1e515700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1e515700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51554/0 (socket says 172.21.15.143:51554) 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1e515700 1 -- 172.21.15.143:0/152722699 learned_addr learned my addr 172.21.15.143:0/152722699 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.485+0000 7f6a1dd14700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a18069d70 0x7f6a18197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1d513700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 0x7f6a18198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 -- 172.21.15.143:0/152722699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 msgr2=0x7f6a18198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 0x7f6a18198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 -- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a18069d70 msgr2=0x7f6a18197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a18069d70 0x7f6a18197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 -- 172.21.15.143:0/152722699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1819cd20 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1e515700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400d980 tx=0x7f6a1400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 10 client.?.objecter ms_handle_connect 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14017070 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6a14005020 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14009990 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1819d010 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1ff78700 10 client.24542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a181a4890 con 0x7f6a181071c0 2024-03-11T21:25:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1ff78700 1 librados: init done 2024-03-11T21:25:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1804faa0 con 0x7f6a181071c0 2024-03-11T21:25:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f6a14004a30 con 0x7f6a181071c0 2024-03-11T21:25:42.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 0x7f6a04071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(114..114 src has 56..114) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f6a1408dd10 con 0x7f6a181071c0 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 10 client.24542.objecter ms_dispatch 0x7f6a18001550 osd_map(114..114 src has 56..114) v4 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 3 client.24542.objecter handle_osd_map got epochs [114,114] > 0 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 3 client.24542.objecter handle_osd_map decoding full epoch 114 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a1dd14700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 0x7f6a04071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.489+0000 7f6a0effd700 20 client.24542.objecter dump_active .. 0 homeless 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.493+0000 7f6a1dd14700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 0x7f6a04071ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6a10000f80 tx=0x7f6a1000b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:42.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.493+0000 7f6a0effd700 10 client.24542.objecter ms_handle_connect 0x7f6a0406f820 2024-03-11T21:25:42.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.497+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6a1405b390 con 0x7f6a181071c0 2024-03-11T21:25:42.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.705+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6a181a4a60 con 0x7f6a181071c0 2024-03-11T21:25:42.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.705+0000 7f6a0effd700 1 -- 172.21.15.143:0/152722699 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a1405ede0 con 0x7f6a181071c0 2024-03-11T21:25:42.686 INFO:teuthology.orchestra.run.smithi143.stdout:81604378717 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.946654+0000 osd.4 (osd.4) 47 : cluster 0 6.17 scrub starts 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.948800+0000 osd.3 (osd.3) 71 : cluster 0 3.8 scrub starts 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.948948+0000 osd.4 (osd.4) 48 : cluster 0 6.17 scrub ok 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.953958+0000 osd.3 (osd.3) 72 : cluster 0 3.8 scrub ok 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.984351+0000 osd.2 (osd.2) 45 : cluster 0 5.d scrub starts 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:40.987383+0000 osd.2 (osd.2) 46 : cluster 0 5.d scrub ok 2024-03-11T21:25:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[29284]: cluster 2024-03-11T21:25:41.092801+0000 mgr.y (mgr.14550) 90 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.946654+0000 osd.4 (osd.4) 47 : cluster 0 6.17 scrub starts 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.948800+0000 osd.3 (osd.3) 71 : cluster 0 3.8 scrub starts 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.948948+0000 osd.4 (osd.4) 48 : cluster 0 6.17 scrub ok 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.953958+0000 osd.3 (osd.3) 72 : cluster 0 3.8 scrub ok 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.984351+0000 osd.2 (osd.2) 45 : cluster 0 5.d scrub starts 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:40.987383+0000 osd.2 (osd.2) 46 : cluster 0 5.d scrub ok 2024-03-11T21:25:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:42 smithi143 bash[21512]: cluster 2024-03-11T21:25:41.092801+0000 mgr.y (mgr.14550) 90 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 10 librados: watch_flush enter 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 10 librados: watch_flush exit 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 20 client.24542.objecter shutdown clearing up homeless session... 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 10 client.24542.objecter successfully canceled tick 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 msgr2=0x7f6a04071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 0x7f6a04071ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6a10000f80 tx=0x7f6a1000b040 comp rx=0 tx=0).stop 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 msgr2=0x7f6a1819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400d980 tx=0x7f6a1400dc90 comp rx=0 tx=0).stop 2024-03-11T21:25:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 shutdown_connections 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6a0406f820 0x7f6a04071ce0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a18069d70 0x7f6a18197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a181071c0 0x7f6a1819c5e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 --2- 172.21.15.143:0/152722699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a1806a720 0x7f6a18198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 >> 172.21.15.143:0/152722699 conn(0x7f6a18076d80 msgr2=0x7f6a180febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 shutdown_connections 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 -- 172.21.15.143:0/152722699 wait complete. 2024-03-11T21:25:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:42.709+0000 7f6a1ff78700 1 librados: shutdown 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.946654+0000 osd.4 (osd.4) 47 : cluster 0 6.17 scrub starts 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.948800+0000 osd.3 (osd.3) 71 : cluster 0 3.8 scrub starts 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.948948+0000 osd.4 (osd.4) 48 : cluster 0 6.17 scrub ok 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.953958+0000 osd.3 (osd.3) 72 : cluster 0 3.8 scrub ok 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.984351+0000 osd.2 (osd.2) 45 : cluster 0 5.d scrub starts 2024-03-11T21:25:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:40.987383+0000 osd.2 (osd.2) 46 : cluster 0 5.d scrub ok 2024-03-11T21:25:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:42 smithi203 bash[23654]: cluster 2024-03-11T21:25:41.092801+0000 mgr.y (mgr.14550) 90 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: cluster 2024-03-11T21:25:41.356749+0000 osd.5 (osd.5) 61 : cluster 0 6.d deep-scrub starts 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: cluster 2024-03-11T21:25:41.358666+0000 osd.5 (osd.5) 62 : cluster 0 6.d deep-scrub ok 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: audit 2024-03-11T21:25:42.635691+0000 mon.c (mon.2) 63 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: audit 2024-03-11T21:25:42.711643+0000 mon.c (mon.2) 64 : audit 0 from='client.? 172.21.15.143:0/152722699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.017975+0000 osd.7 (osd.7) 65 : cluster 0 2.0 scrub starts 2024-03-11T21:25:43.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.020940+0000 osd.7 (osd.7) 66 : cluster 0 2.0 scrub ok 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: cluster 2024-03-11T21:25:41.356749+0000 osd.5 (osd.5) 61 : cluster 0 6.d deep-scrub starts 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: cluster 2024-03-11T21:25:41.358666+0000 osd.5 (osd.5) 62 : cluster 0 6.d deep-scrub ok 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: audit 2024-03-11T21:25:42.635691+0000 mon.c (mon.2) 63 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: audit 2024-03-11T21:25:42.711643+0000 mon.c (mon.2) 64 : audit 0 from='client.? 172.21.15.143:0/152722699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.017975+0000 osd.7 (osd.7) 65 : cluster 0 2.0 scrub starts 2024-03-11T21:25:43.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:43 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.020940+0000 osd.7 (osd.7) 66 : cluster 0 2.0 scrub ok 2024-03-11T21:25:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: cluster 2024-03-11T21:25:41.356749+0000 osd.5 (osd.5) 61 : cluster 0 6.d deep-scrub starts 2024-03-11T21:25:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: cluster 2024-03-11T21:25:41.358666+0000 osd.5 (osd.5) 62 : cluster 0 6.d deep-scrub ok 2024-03-11T21:25:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: audit 2024-03-11T21:25:42.635691+0000 mon.c (mon.2) 63 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: audit 2024-03-11T21:25:42.711643+0000 mon.c (mon.2) 64 : audit 0 from='client.? 172.21.15.143:0/152722699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:25:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.017975+0000 osd.7 (osd.7) 65 : cluster 0 2.0 scrub starts 2024-03-11T21:25:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:43 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.020940+0000 osd.7 (osd.7) 66 : cluster 0 2.0 scrub ok 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- 172.21.15.143:0/1582032173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf006a7f0 msgr2=0x7facf006abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 --2- 172.21.15.143:0/1582032173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf006a7f0 0x7facf006abd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7face800b0a0 tx=0x7face800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- 172.21.15.143:0/1582032173 shutdown_connections 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 --2- 172.21.15.143:0/1582032173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf01044c0 0x7facf010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 --2- 172.21.15.143:0/1582032173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf006a7f0 0x7facf006abd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 --2- 172.21.15.143:0/1582032173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf0103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- 172.21.15.143:0/1582032173 >> 172.21.15.143:0/1582032173 conn(0x7facf0076d80 msgr2=0x7facf0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- 172.21.15.143:0/1582032173 shutdown_connections 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- 172.21.15.143:0/1582032173 wait complete. 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 librados: starting msgr at 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 librados: starting objecter 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 Processor -- start 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 -- start start 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 librados: setting wanted keys 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.425+0000 7facf6b85700 1 librados: calling monclient init 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf006a7f0 0x7facf0100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf01044c0 0x7facf0101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf01117e0 con 0x7facf0103b00 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0111660 con 0x7facf006a7f0 2024-03-11T21:25:44.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf0111960 con 0x7facf01044c0 2024-03-11T21:25:44.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf5122700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf01044c0 0x7facf0101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36468/0 (socket says 172.21.15.143:36468) 2024-03-11T21:25:44.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 -- 172.21.15.143:0/3201715596 learned_addr learned my addr 172.21.15.143:0/3201715596 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 -- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf01044c0 msgr2=0x7facf0101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf4921700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf006a7f0 0x7facf0100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf01044c0 0x7facf0101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 -- 172.21.15.143:0/3201715596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf006a7f0 msgr2=0x7facf0100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf006a7f0 0x7facf0100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 -- 172.21.15.143:0/3201715596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf01a6310 con 0x7facf0103b00 2024-03-11T21:25:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7faceffff700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facd800eb30 tx=0x7facd800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 10 client.?.objecter ms_handle_connect 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd800cc90 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7facd800ce30 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf01a6600 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8003df0 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 10 client.14673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf01a6b40 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 librados: init done 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf004faa0 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7facd8003f90 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 0x7facdc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(114..114 src has 56..114) v4 ==== 7450+0+0 (secure 0 0 0) 0x7facd808e4b0 con 0x7facf0103b00 2024-03-11T21:25:44.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 10 client.14673.objecter ms_dispatch 0x7facf0001550 osd_map(114..114 src has 56..114) v4 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf4921700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 0x7facdc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 3 client.14673.objecter handle_osd_map got epochs [114,114] > 0 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 3 client.14673.objecter handle_osd_map decoding full epoch 114 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 20 client.14673.objecter dump_active .. 0 homeless 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facf4921700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 0x7facdc071ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7face800b070 tx=0x7face800b700 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:44.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.429+0000 7facedffb700 10 client.14673.objecter ms_handle_connect 0x7facdc06f820 2024-03-11T21:25:44.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.437+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7facd805b9d0 con 0x7facf0103b00 2024-03-11T21:25:44.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.094097+0000 mgr.y (mgr.14550) 91 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:44.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.243583+0000 osd.0 (osd.0) 55 : cluster 0 6.16 scrub starts 2024-03-11T21:25:44.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.245992+0000 osd.0 (osd.0) 56 : cluster 0 6.16 scrub ok 2024-03-11T21:25:44.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.864418+0000 osd.6 (osd.6) 37 : cluster 0 2.7 scrub starts 2024-03-11T21:25:44.515 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.867363+0000 osd.6 (osd.6) 38 : cluster 0 2.7 scrub ok 2024-03-11T21:25:44.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.094097+0000 mgr.y (mgr.14550) 91 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:44.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.243583+0000 osd.0 (osd.0) 55 : cluster 0 6.16 scrub starts 2024-03-11T21:25:44.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.245992+0000 osd.0 (osd.0) 56 : cluster 0 6.16 scrub ok 2024-03-11T21:25:44.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.864418+0000 osd.6 (osd.6) 37 : cluster 0 2.7 scrub starts 2024-03-11T21:25:44.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:44 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.867363+0000 osd.6 (osd.6) 38 : cluster 0 2.7 scrub ok 2024-03-11T21:25:44.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7facf01a6e00 con 0x7facf0103b00 2024-03-11T21:25:44.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facedffb700 1 -- 172.21.15.143:0/3201715596 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7facd805f420 con 0x7facf0103b00 2024-03-11T21:25:44.630 INFO:teuthology.orchestra.run.smithi143.stdout:240518168628 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 10 librados: watch_flush enter 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 10 librados: watch_flush exit 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 20 client.14673.objecter shutdown clearing up homeless session... 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 10 client.14673.objecter successfully canceled tick 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 msgr2=0x7facdc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 0x7facdc071ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7face800b070 tx=0x7face800b700 comp rx=0 tx=0).stop 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 msgr2=0x7facf01013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facd800eb30 tx=0x7facd800ee40 comp rx=0 tx=0).stop 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 shutdown_connections 2024-03-11T21:25:44.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7facdc06f820 0x7facdc071ce0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf01044c0 0x7facf0101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf006a7f0 0x7facf0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 --2- 172.21.15.143:0/3201715596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf0103b00 0x7facf01013f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 >> 172.21.15.143:0/3201715596 conn(0x7facf0076d80 msgr2=0x7facf00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 shutdown_connections 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 -- 172.21.15.143:0/3201715596 wait complete. 2024-03-11T21:25:44.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:44.653+0000 7facf6b85700 1 librados: shutdown 2024-03-11T21:25:44.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:44 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.094097+0000 mgr.y (mgr.14550) 91 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:44.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:44 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.243583+0000 osd.0 (osd.0) 55 : cluster 0 6.16 scrub starts 2024-03-11T21:25:44.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:44 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.245992+0000 osd.0 (osd.0) 56 : cluster 0 6.16 scrub ok 2024-03-11T21:25:44.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:44 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.864418+0000 osd.6 (osd.6) 37 : cluster 0 2.7 scrub starts 2024-03-11T21:25:44.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:44 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.867363+0000 osd.6 (osd.6) 38 : cluster 0 2.7 scrub ok 2024-03-11T21:25:45.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 -- 172.21.15.143:0/1630289459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104250 msgr2=0x7f6d54104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 --2- 172.21.15.143:0/1630289459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104250 0x7f6d54104630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0099c0 tx=0x7f6d4c009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 -- 172.21.15.143:0/1630289459 shutdown_connections 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 --2- 172.21.15.143:0/1630289459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d54110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 --2- 172.21.15.143:0/1630289459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104250 0x7f6d54104630 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 --2- 172.21.15.143:0/1630289459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104c00 0x7f6d54108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 -- 172.21.15.143:0/1630289459 >> 172.21.15.143:0/1630289459 conn(0x7f6d540ffac0 msgr2=0x7f6d54101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 -- 172.21.15.143:0/1630289459 shutdown_connections 2024-03-11T21:25:45.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.125+0000 7f6d59191700 1 -- 172.21.15.143:0/1630289459 wait complete. 2024-03-11T21:25:45.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 librados: starting msgr at 2024-03-11T21:25:45.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 librados: starting objecter 2024-03-11T21:25:45.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 Processor -- start 2024-03-11T21:25:45.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- start start 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 librados: setting wanted keys 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 librados: calling monclient init 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104250 0x7f6d54199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d541137e0 con 0x7f6d54109330 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54113660 con 0x7f6d54104250 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54113960 con 0x7f6d54104c00 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:45.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46486/0 (socket says 172.21.15.143:46486) 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 -- 172.21.15.143:0/297240757 learned_addr learned my addr 172.21.15.143:0/297240757 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36478/0 (socket says 172.21.15.143:36478) 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d52d9d700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104250 0x7f6d54199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 -- 172.21.15.143:0/297240757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104250 msgr2=0x7f6d54199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104250 0x7f6d54199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 -- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 msgr2=0x7f6d5419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 -- 172.21.15.143:0/297240757 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5419edd0 con 0x7f6d54104c00 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5359e700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d5259c700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00c8f0 tx=0x7f6d3c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 10 client.?.objecter ms_handle_connect 0x7f6d54104c00 2024-03-11T21:25:45.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c007b00 con 0x7f6d54104c00 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6d3c00ce30 con 0x7f6d54104c00 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5419f0c0 con 0x7f6d54104c00 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 10 client.24554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c018400 con 0x7f6d54104c00 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d541a6940 con 0x7f6d54104c00 2024-03-11T21:25:45.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 librados: init done 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5404faa0 con 0x7f6d54104c00 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f6d3c00f450 con 0x7f6d54104c00 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 0x7f6d40071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(114..114 src has 56..114) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f6d3c08e030 con 0x7f6d54104c00 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 10 client.24554.objecter ms_dispatch 0x7f6d54001550 osd_map(114..114 src has 56..114) v4 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 3 client.24554.objecter handle_osd_map got epochs [114,114] > 0 2024-03-11T21:25:45.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.129+0000 7f6d4bfff700 3 client.24554.objecter handle_osd_map decoding full epoch 114 2024-03-11T21:25:45.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.133+0000 7f6d52d9d700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 0x7f6d40071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:45.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.133+0000 7f6d4bfff700 20 client.24554.objecter dump_active .. 0 homeless 2024-03-11T21:25:45.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.133+0000 7f6d4bfff700 10 client.24554.objecter ms_handle_connect 0x7f6d4006f820 2024-03-11T21:25:45.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.133+0000 7f6d52d9d700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 0x7f6d40071ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0059d0 tx=0x7f6d4c005940 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:45.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.137+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6d3c014030 con 0x7f6d54104c00 2024-03-11T21:25:45.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.349+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6d541a6c00 con 0x7f6d54104c00 2024-03-11T21:25:45.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.349+0000 7f6d4bfff700 1 -- 172.21.15.143:0/297240757 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6d3c05b5d0 con 0x7f6d54104c00 2024-03-11T21:25:45.328 INFO:teuthology.orchestra.run.smithi143.stdout:115964117074 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 10 librados: watch_flush enter 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 10 librados: watch_flush exit 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 20 client.24554.objecter shutdown clearing up homeless session... 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 10 client.24554.objecter successfully canceled tick 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 msgr2=0x7f6d40071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 0x7f6d40071ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0059d0 tx=0x7f6d4c005940 comp rx=0 tx=0).stop 2024-03-11T21:25:45.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 msgr2=0x7f6d5419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00c8f0 tx=0x7f6d3c00ccb0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 shutdown_connections 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6d4006f820 0x7f6d40071ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d54109330 0x7f6d5419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104250 0x7f6d54199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 --2- 172.21.15.143:0/297240757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d54104c00 0x7f6d5419a300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 >> 172.21.15.143:0/297240757 conn(0x7f6d540ffac0 msgr2=0x7f6d541001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 shutdown_connections 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 -- 172.21.15.143:0/297240757 wait complete. 2024-03-11T21:25:45.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:45.353+0000 7f6d59191700 1 librados: shutdown 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.873127+0000 osd.4 (osd.4) 49 : cluster 0 3.1a scrub starts 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:43.877212+0000 osd.4 (osd.4) 50 : cluster 0 3.1a scrub ok 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:44.012014+0000 osd.2 (osd.2) 47 : cluster 0 2.c scrub starts 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:44.013741+0000 osd.2 (osd.2) 48 : cluster 0 2.c scrub ok 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:44.630708+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: cluster 2024-03-11T21:25:44.630772+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[29284]: audit 2024-03-11T21:25:44.656615+0000 mon.a (mon.0) 856 : audit 0 from='client.? 172.21.15.143:0/3201715596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.873127+0000 osd.4 (osd.4) 49 : cluster 0 3.1a scrub starts 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:43.877212+0000 osd.4 (osd.4) 50 : cluster 0 3.1a scrub ok 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:44.012014+0000 osd.2 (osd.2) 47 : cluster 0 2.c scrub starts 2024-03-11T21:25:45.632 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:44.013741+0000 osd.2 (osd.2) 48 : cluster 0 2.c scrub ok 2024-03-11T21:25:45.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:44.630708+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:45.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: cluster 2024-03-11T21:25:44.630772+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:45.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:45 smithi143 bash[21512]: audit 2024-03-11T21:25:44.656615+0000 mon.a (mon.0) 856 : audit 0 from='client.? 172.21.15.143:0/3201715596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:25:45.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.873127+0000 osd.4 (osd.4) 49 : cluster 0 3.1a scrub starts 2024-03-11T21:25:45.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:43.877212+0000 osd.4 (osd.4) 50 : cluster 0 3.1a scrub ok 2024-03-11T21:25:45.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:44.012014+0000 osd.2 (osd.2) 47 : cluster 0 2.c scrub starts 2024-03-11T21:25:45.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:44.013741+0000 osd.2 (osd.2) 48 : cluster 0 2.c scrub ok 2024-03-11T21:25:45.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:44.630708+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:45.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: cluster 2024-03-11T21:25:44.630772+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:45.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:45 smithi203 bash[23654]: audit 2024-03-11T21:25:44.656615+0000 mon.a (mon.0) 856 : audit 0 from='client.? 172.21.15.143:0/3201715596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1700803716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 msgr2=0x7fa10c1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- 172.21.15.143:0/1700803716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1072d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa1080099c0 tx=0x7fa108009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1700803716 shutdown_connections 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- 172.21.15.143:0/1700803716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1072d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- 172.21.15.143:0/1700803716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 0x7fa10c10db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- 172.21.15.143:0/1700803716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1700803716 >> 172.21.15.143:0/1700803716 conn(0x7fa10c077a40 msgr2=0x7fa10c077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1700803716 shutdown_connections 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1700803716 wait complete. 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 librados: starting msgr at 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 librados: starting objecter 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 Processor -- start 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- start start 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 librados: setting wanted keys 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 librados: calling monclient init 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 0x7fa10c102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c1105d0 con 0x7fa10c106ef0 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c110450 con 0x7fa10c1019c0 2024-03-11T21:25:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c110750 con 0x7fa10c101000 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46500/0 (socket says 172.21.15.143:46500) 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 -- 172.21.15.143:0/1602193619 learned_addr learned my addr 172.21.15.143:0/1602193619 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111a19700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa110a17700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 0x7fa10c102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 -- 172.21.15.143:0/1602193619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 msgr2=0x7fa10c102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 0x7fa10c102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 -- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 msgr2=0x7fa10c1034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 -- 172.21.15.143:0/1602193619 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c1a60f0 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111a19700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa111218700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa108005910 tx=0x7fa10800fab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa1027fc700 10 client.?.objecter ms_handle_connect 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa1027fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108027070 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa10800fd40 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10c1a6380 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108022d90 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10c1a67e0 con 0x7fa10c101000 2024-03-11T21:25:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.169+0000 7fa11347c700 1 librados: init done 2024-03-11T21:25:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10c04faa0 con 0x7fa10c101000 2024-03-11T21:25:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa10802a070 con 0x7fa10c101000 2024-03-11T21:25:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 0x7fa0f8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa110a17700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 0x7fa0f8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(115..115 src has 56..115) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fa10808ef80 con 0x7fa10c101000 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 10 client.24560.objecter ms_dispatch 0x7fa10c001550 osd_map(115..115 src has 56..115) v4 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 3 client.24560.objecter handle_osd_map got epochs [115,115] > 0 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 3 client.24560.objecter handle_osd_map decoding full epoch 115 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 20 client.24560.objecter dump_active .. 0 homeless 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa1027fc700 10 client.24560.objecter ms_handle_connect 0x7fa0f806f820 2024-03-11T21:25:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.173+0000 7fa110a17700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 0x7fa0f8071ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa10c109b50 tx=0x7fa0fc009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.177+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa108013070 con 0x7fa10c101000 2024-03-11T21:25:46.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa10c1a6a40 con 0x7fa10c101000 2024-03-11T21:25:46.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa1027fc700 1 -- 172.21.15.143:0/1602193619 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa10805c520 con 0x7fa10c101000 2024-03-11T21:25:46.366 INFO:teuthology.orchestra.run.smithi143.stdout:55834574952 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 10 librados: watch_flush enter 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 10 librados: watch_flush exit 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 20 client.24560.objecter shutdown clearing up homeless session... 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 10 client.24560.objecter successfully canceled tick 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 msgr2=0x7fa0f8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 0x7fa0f8071ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa10c109b50 tx=0x7fa0fc009260 comp rx=0 tx=0).stop 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 msgr2=0x7fa10c102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa108005910 tx=0x7fa10800fab0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 shutdown_connections 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa0f806f820 0x7fa0f8071ce0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c106ef0 0x7fa10c1034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10c101000 0x7fa10c102a30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 --2- 172.21.15.143:0/1602193619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa10c1019c0 0x7fa10c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.389+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 >> 172.21.15.143:0/1602193619 conn(0x7fa10c077a40 msgr2=0x7fa10c06a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:46.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.393+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 shutdown_connections 2024-03-11T21:25:46.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.393+0000 7fa11347c700 1 -- 172.21.15.143:0/1602193619 wait complete. 2024-03-11T21:25:46.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:46.393+0000 7fa11347c700 1 librados: shutdown 2024-03-11T21:25:46.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.094810+0000 mgr.y (mgr.14550) 92 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:46.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.328162+0000 mon.a (mon.0) 857 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:25:46.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[21512]: audit 2024-03-11T21:25:45.354834+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.143:0/297240757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:25:46.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.887402+0000 osd.6 (osd.6) 39 : cluster 0 2.14 scrub starts 2024-03-11T21:25:46.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.889824+0000 osd.6 (osd.6) 40 : cluster 0 2.14 scrub ok 2024-03-11T21:25:46.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.094810+0000 mgr.y (mgr.14550) 92 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:46.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.328162+0000 mon.a (mon.0) 857 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:25:46.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[29284]: audit 2024-03-11T21:25:45.354834+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.143:0/297240757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:25:46.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.887402+0000 osd.6 (osd.6) 39 : cluster 0 2.14 scrub starts 2024-03-11T21:25:46.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:46 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.889824+0000 osd.6 (osd.6) 40 : cluster 0 2.14 scrub ok 2024-03-11T21:25:46.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:46 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.094810+0000 mgr.y (mgr.14550) 92 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:46.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:46 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.328162+0000 mon.a (mon.0) 857 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:25:46.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:46 smithi203 bash[23654]: audit 2024-03-11T21:25:45.354834+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.143:0/297240757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:25:46.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:46 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.887402+0000 osd.6 (osd.6) 39 : cluster 0 2.14 scrub starts 2024-03-11T21:25:46.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:46 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.889824+0000 osd.6 (osd.6) 40 : cluster 0 2.14 scrub ok 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.372222+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.376387+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.980749+0000 osd.3 (osd.3) 73 : cluster 0 5.0 scrub starts 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:45.984088+0000 osd.3 (osd.3) 74 : cluster 0 5.0 scrub ok 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.283697+0000 osd.0 (osd.0) 57 : cluster 0 3.1f scrub starts 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.288008+0000 osd.0 (osd.0) 58 : cluster 0 3.1f scrub ok 2024-03-11T21:25:47.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: audit 2024-03-11T21:25:46.393092+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.143:0/1602193619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.792455+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.792563+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.915425+0000 osd.6 (osd.6) 41 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[29284]: cluster 2024-03-11T21:25:46.921392+0000 osd.6 (osd.6) 42 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.372222+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.376387+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.980749+0000 osd.3 (osd.3) 73 : cluster 0 5.0 scrub starts 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:45.984088+0000 osd.3 (osd.3) 74 : cluster 0 5.0 scrub ok 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.283697+0000 osd.0 (osd.0) 57 : cluster 0 3.1f scrub starts 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.288008+0000 osd.0 (osd.0) 58 : cluster 0 3.1f scrub ok 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: audit 2024-03-11T21:25:46.393092+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.143:0/1602193619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:25:47.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.792455+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:47.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.792563+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:47.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.915425+0000 osd.6 (osd.6) 41 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:25:47.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:47 smithi143 bash[21512]: cluster 2024-03-11T21:25:46.921392+0000 osd.6 (osd.6) 42 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:25:47.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.372222+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T21:25:47.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.376387+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.980749+0000 osd.3 (osd.3) 73 : cluster 0 5.0 scrub starts 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:45.984088+0000 osd.3 (osd.3) 74 : cluster 0 5.0 scrub ok 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.283697+0000 osd.0 (osd.0) 57 : cluster 0 3.1f scrub starts 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.288008+0000 osd.0 (osd.0) 58 : cluster 0 3.1f scrub ok 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: audit 2024-03-11T21:25:46.393092+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.143:0/1602193619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.792455+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.792563+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.915425+0000 osd.6 (osd.6) 41 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:25:47.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:47 smithi203 bash[23654]: cluster 2024-03-11T21:25:46.921392+0000 osd.6 (osd.6) 42 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:25:47.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378711 got 81604378717 for osd.2 2024-03-11T21:25:47.963 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:47.981 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168623 got 240518168628 for osd.6 2024-03-11T21:25:47.981 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:48.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:48.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:48 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.096145+0000 mgr.y (mgr.14550) 93 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:48.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:48 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.343399+0000 mon.a (mon.0) 858 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:25:48.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:48 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.046314+0000 osd.7 (osd.7) 69 : cluster 0 2.1c scrub starts 2024-03-11T21:25:48.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:48 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.050639+0000 osd.7 (osd.7) 70 : cluster 0 2.1c scrub ok 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.096145+0000 mgr.y (mgr.14550) 93 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.343399+0000 mon.a (mon.0) 858 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.046314+0000 osd.7 (osd.7) 69 : cluster 0 2.1c scrub starts 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.050639+0000 osd.7 (osd.7) 70 : cluster 0 2.1c scrub ok 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.096145+0000 mgr.y (mgr.14550) 93 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.343399+0000 mon.a (mon.0) 858 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:25:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.046314+0000 osd.7 (osd.7) 69 : cluster 0 2.1c scrub starts 2024-03-11T21:25:48.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:48 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.050639+0000 osd.7 (osd.7) 70 : cluster 0 2.1c scrub ok 2024-03-11T21:25:49.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117069 got 115964117074 for osd.3 2024-03-11T21:25:49.547 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.895727+0000 osd.4 (osd.4) 51 : cluster 0 4.1 scrub starts 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.898311+0000 osd.4 (osd.4) 52 : cluster 0 4.1 scrub ok 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.991655+0000 osd.2 (osd.2) 51 : cluster 0 2.1 scrub starts 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:47.993990+0000 osd.2 (osd.2) 52 : cluster 0 2.1 scrub ok 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: audit 2024-03-11T21:25:48.150938+0000 mgr.y (mgr.14550) 94 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.261521+0000 osd.0 (osd.0) 59 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.263508+0000 osd.0 (osd.0) 60 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:25:49.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.371852+0000 osd.5 (osd.5) 65 : cluster 0 2.16 scrub starts 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.374146+0000 osd.5 (osd.5) 66 : cluster 0 2.16 scrub ok 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:49.000498+0000 osd.7 (osd.7) 71 : cluster 0 6.15 scrub starts 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[21512]: cluster 2024-03-11T21:25:49.003000+0000 osd.7 (osd.7) 72 : cluster 0 6.15 scrub ok 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.895727+0000 osd.4 (osd.4) 51 : cluster 0 4.1 scrub starts 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.898311+0000 osd.4 (osd.4) 52 : cluster 0 4.1 scrub ok 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.991655+0000 osd.2 (osd.2) 51 : cluster 0 2.1 scrub starts 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:47.993990+0000 osd.2 (osd.2) 52 : cluster 0 2.1 scrub ok 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: audit 2024-03-11T21:25:48.150938+0000 mgr.y (mgr.14550) 94 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:49.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.261521+0000 osd.0 (osd.0) 59 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:25:49.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.263508+0000 osd.0 (osd.0) 60 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:25:49.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.371852+0000 osd.5 (osd.5) 65 : cluster 0 2.16 scrub starts 2024-03-11T21:25:49.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.374146+0000 osd.5 (osd.5) 66 : cluster 0 2.16 scrub ok 2024-03-11T21:25:49.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:49.000498+0000 osd.7 (osd.7) 71 : cluster 0 6.15 scrub starts 2024-03-11T21:25:49.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:49 smithi143 bash[29284]: cluster 2024-03-11T21:25:49.003000+0000 osd.7 (osd.7) 72 : cluster 0 6.15 scrub ok 2024-03-11T21:25:49.819 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574947 got 55834574952 for osd.1 2024-03-11T21:25:49.819 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- 172.21.15.143:0/3881672542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 msgr2=0x7f1aa4108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- 172.21.15.143:0/3881672542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa4108c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1a940099c0 tx=0x7f1a94009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- 172.21.15.143:0/3881672542 shutdown_connections 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- 172.21.15.143:0/3881672542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- 172.21.15.143:0/3881672542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa4110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- 172.21.15.143:0/3881672542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa4108c70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- 172.21.15.143:0/3881672542 >> 172.21.15.143:0/3881672542 conn(0x7f1aa40ffb00 msgr2=0x7f1aa4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- 172.21.15.143:0/3881672542 shutdown_connections 2024-03-11T21:25:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- 172.21.15.143:0/3881672542 wait complete. 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 librados: starting msgr at 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 librados: starting objecter 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 Processor -- start 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- start start 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 librados: setting wanted keys 2024-03-11T21:25:49.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 librados: calling monclient init 2024-03-11T21:25:49.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa41138b0 con 0x7f1aa4104270 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4113730 con 0x7f1aa4109350 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aac544700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4113a30 con 0x7f1aa4104c20 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aa9adf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aaa2e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aa9adf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46518/0 (socket says 172.21.15.143:46518) 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.969+0000 7f1aa9adf700 1 -- 172.21.15.143:0/3173711335 learned_addr learned my addr 172.21.15.143:0/3173711335 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 -- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 msgr2=0x7f1aa419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaaae1700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 -- 172.21.15.143:0/3173711335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 msgr2=0x7f1aa419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 -- 172.21.15.143:0/3173711335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa419ee40 con 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aa9adf700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaaae1700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aaa2e0700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4199e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000e9f0 tx=0x7f1aa000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000cbd0 con 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1aa000cd70 con 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa419f130 con 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0018680 con 0x7f1aa4104270 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aac544700 10 client.14691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa41a69b0 con 0x7f1aa4104270 2024-03-11T21:25:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aac544700 1 librados: init done 2024-03-11T21:25:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa404faa0 con 0x7f1aa4104270 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1aa0010430 con 0x7f1aa4104270 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 0x7f1a90071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(116..116 src has 66..116) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f1aa008e300 con 0x7f1aa4104270 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 10 client.14691.objecter ms_dispatch 0x7f1aa4001550 osd_map(116..116 src has 66..116) v4 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 3 client.14691.objecter handle_osd_map got epochs [116,116] > 0 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 3 client.14691.objecter handle_osd_map decoding full epoch 116 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aa9adf700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 0x7f1a90071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 20 client.14691.objecter dump_active .. 0 homeless 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1a9b7fe700 10 client.14691.objecter ms_handle_connect 0x7f1a9006f820 2024-03-11T21:25:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.973+0000 7f1aa9adf700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 0x7f1a90071ce0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a940037a0 tx=0x7f1a94019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:49.977+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1aa005b8a0 con 0x7f1aa4104270 2024-03-11T21:25:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.895727+0000 osd.4 (osd.4) 51 : cluster 0 4.1 scrub starts 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.898311+0000 osd.4 (osd.4) 52 : cluster 0 4.1 scrub ok 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.991655+0000 osd.2 (osd.2) 51 : cluster 0 2.1 scrub starts 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:47.993990+0000 osd.2 (osd.2) 52 : cluster 0 2.1 scrub ok 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: audit 2024-03-11T21:25:48.150938+0000 mgr.y (mgr.14550) 94 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.261521+0000 osd.0 (osd.0) 59 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.263508+0000 osd.0 (osd.0) 60 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.371852+0000 osd.5 (osd.5) 65 : cluster 0 2.16 scrub starts 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.374146+0000 osd.5 (osd.5) 66 : cluster 0 2.16 scrub ok 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:49.000498+0000 osd.7 (osd.7) 71 : cluster 0 6.15 scrub starts 2024-03-11T21:25:50.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:49 smithi203 bash[23654]: cluster 2024-03-11T21:25:49.003000+0000 osd.7 (osd.7) 72 : cluster 0 6.15 scrub ok 2024-03-11T21:25:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.197+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1aa41a6ca0 con 0x7f1aa4104270 2024-03-11T21:25:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.197+0000 7f1a9b7fe700 1 -- 172.21.15.143:0/3173711335 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1aa005f2f0 con 0x7f1aa4104270 2024-03-11T21:25:50.175 INFO:teuthology.orchestra.run.smithi143.stdout:283467841580 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 10 librados: watch_flush enter 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 10 librados: watch_flush exit 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 20 client.14691.objecter shutdown clearing up homeless session... 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 10 client.14691.objecter successfully canceled tick 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 msgr2=0x7f1a90071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 0x7f1a90071ce0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a940037a0 tx=0x7f1a94019040 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 msgr2=0x7f1aa4199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4199e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000e9f0 tx=0x7f1aa000ed00 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 shutdown_connections 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a9006f820 0x7f1a90071ce0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa4104270 0x7f1aa4199e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aa4109350 0x7f1aa419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 --2- 172.21.15.143:0/3173711335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa4104c20 0x7f1aa419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 >> 172.21.15.143:0/3173711335 conn(0x7f1aa40ffb00 msgr2=0x7f1aa4100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 shutdown_connections 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 -- 172.21.15.143:0/3173711335 wait complete. 2024-03-11T21:25:50.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.201+0000 7f1aac544700 1 librados: shutdown 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- 172.21.15.143:0/1528664296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 msgr2=0x7f611c108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 --2- 172.21.15.143:0/1528664296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c108c50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f61040099c0 tx=0x7f6104009cd0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- 172.21.15.143:0/1528664296 shutdown_connections 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 --2- 172.21.15.143:0/1528664296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 --2- 172.21.15.143:0/1528664296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 --2- 172.21.15.143:0/1528664296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c108c50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- 172.21.15.143:0/1528664296 >> 172.21.15.143:0/1528664296 conn(0x7f611c0ffac0 msgr2=0x7f611c101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- 172.21.15.143:0/1528664296 shutdown_connections 2024-03-11T21:25:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- 172.21.15.143:0/1528664296 wait complete. 2024-03-11T21:25:50.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 librados: starting msgr at 2024-03-11T21:25:50.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 librados: starting objecter 2024-03-11T21:25:50.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 Processor -- start 2024-03-11T21:25:50.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 -- start start 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 librados: setting wanted keys 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.357+0000 7f6121947700 1 librados: calling monclient init 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c113890 con 0x7f611c104c00 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c113710 con 0x7f611c109330 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c113a10 con 0x7f611c104250 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46530/0 (socket says 172.21.15.143:46530) 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 -- 172.21.15.143:0/180935073 learned_addr learned my addr 172.21.15.143:0/180935073 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6113fff700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611b7fe700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 -- 172.21.15.143:0/180935073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 msgr2=0x7f611c19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 -- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 msgr2=0x7f611c19a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c19a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 -- 172.21.15.143:0/180935073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c19ee30 con 0x7f611c104250 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611b7fe700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6113fff700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f611affd700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f610c00eb30 tx=0x7f610c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 10 client.?.objecter ms_handle_connect 0x7f611c104250 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c00cc90 con 0x7f611c104250 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f610c00ce30 con 0x7f611c104250 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c19f120 con 0x7f611c104250 2024-03-11T21:25:50.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c003df0 con 0x7f611c104250 2024-03-11T21:25:50.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 10 client.24572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:50.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c1a69a0 con 0x7f611c104250 2024-03-11T21:25:50.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 librados: init done 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c04faa0 con 0x7f611c104250 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f610c004000 con 0x7f611c104250 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 0x7f6108071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(116..116 src has 66..116) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f610c08e450 con 0x7f611c104250 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 10 client.24572.objecter ms_dispatch 0x7f611c001550 osd_map(116..116 src has 66..116) v4 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 3 client.24572.objecter handle_osd_map got epochs [116,116] > 0 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 3 client.24572.objecter handle_osd_map decoding full epoch 116 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6113fff700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 0x7f6108071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 20 client.24572.objecter dump_active .. 0 homeless 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6113fff700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 0x7f6108071ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6104000c00 tx=0x7f6104005fb0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:50.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.361+0000 7f6120945700 10 client.24572.objecter ms_handle_connect 0x7f610806f820 2024-03-11T21:25:50.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.365+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f610c05b9f0 con 0x7f611c104250 2024-03-11T21:25:50.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f611c1a6c30 con 0x7f611c104250 2024-03-11T21:25:50.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6120945700 1 -- 172.21.15.143:0/180935073 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f610c05f440 con 0x7f611c104250 2024-03-11T21:25:50.554 INFO:teuthology.orchestra.run.smithi143.stdout:197568495679 2024-03-11T21:25:50.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 10 librados: watch_flush enter 2024-03-11T21:25:50.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 10 librados: watch_flush exit 2024-03-11T21:25:50.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 20 client.24572.objecter shutdown clearing up homeless session... 2024-03-11T21:25:50.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 10 client.24572.objecter successfully canceled tick 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 msgr2=0x7f6108071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 0x7f6108071ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6104000c00 tx=0x7f6104005fb0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 msgr2=0x7f611c199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f610c00eb30 tx=0x7f610c00ee40 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 shutdown_connections 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f610806f820 0x7f6108071ce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f611c109330 0x7f611c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f611c104250 0x7f611c199e20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 --2- 172.21.15.143:0/180935073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f611c104c00 0x7f611c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 >> 172.21.15.143:0/180935073 conn(0x7f611c0ffac0 msgr2=0x7f611c100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 shutdown_connections 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 -- 172.21.15.143:0/180935073 wait complete. 2024-03-11T21:25:50.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.577+0000 7f6121947700 1 librados: shutdown 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.917816+0000 osd.4 (osd.4) 53 : cluster 0 6.1a scrub starts 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:48.919918+0000 osd.4 (osd.4) 54 : cluster 0 6.1a scrub ok 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:49.097613+0000 mgr.y (mgr.14550) 95 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:49.634992+0000 osd.1 (osd.1) 61 : cluster 0 2.6 scrub starts 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:49.638046+0000 osd.1 (osd.1) 62 : cluster 0 2.6 scrub ok 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.004627+0000 osd.3 (osd.3) 75 : cluster 0 4.0 scrub starts 2024-03-11T21:25:50.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.006694+0000 osd.3 (osd.3) 76 : cluster 0 4.0 scrub ok 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: audit 2024-03-11T21:25:50.201888+0000 mon.a (mon.0) 859 : audit 0 from='client.? 172.21.15.143:0/3173711335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[21512]: audit 2024-03-11T21:25:50.580567+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.143:0/180935073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.917816+0000 osd.4 (osd.4) 53 : cluster 0 6.1a scrub starts 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:48.919918+0000 osd.4 (osd.4) 54 : cluster 0 6.1a scrub ok 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:49.097613+0000 mgr.y (mgr.14550) 95 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:49.634992+0000 osd.1 (osd.1) 61 : cluster 0 2.6 scrub starts 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:49.638046+0000 osd.1 (osd.1) 62 : cluster 0 2.6 scrub ok 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.004627+0000 osd.3 (osd.3) 75 : cluster 0 4.0 scrub starts 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.006694+0000 osd.3 (osd.3) 76 : cluster 0 4.0 scrub ok 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: audit 2024-03-11T21:25:50.201888+0000 mon.a (mon.0) 859 : audit 0 from='client.? 172.21.15.143:0/3173711335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:25:50.823 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:50 smithi143 bash[29284]: audit 2024-03-11T21:25:50.580567+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.143:0/180935073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- 172.21.15.143:0/2462226204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd30069d70 msgr2=0x7efd3006a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- 172.21.15.143:0/2462226204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd30069d70 0x7efd3006a150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efd20009920 tx=0x7efd20009c30 comp rx=0 tx=0).stop 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- 172.21.15.143:0/2462226204 shutdown_connections 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- 172.21.15.143:0/2462226204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd30069d70 0x7efd3006a150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- 172.21.15.143:0/2462226204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd301071c0 0x7efd3010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- 172.21.15.143:0/2462226204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- 172.21.15.143:0/2462226204 >> 172.21.15.143:0/2462226204 conn(0x7efd30076d80 msgr2=0x7efd30077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- 172.21.15.143:0/2462226204 shutdown_connections 2024-03-11T21:25:50.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- 172.21.15.143:0/2462226204 wait complete. 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 librados: starting msgr at 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 librados: starting objecter 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 Processor -- start 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- start start 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 librados: setting wanted keys 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 librados: calling monclient init 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd30069d70 0x7efd30197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 0x7efd3019c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30111690 con 0x7efd301071c0 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30111510 con 0x7efd30069d70 2024-03-11T21:25:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd363f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd30111810 con 0x7efd3006a720 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 0x7efd3019c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46544/0 (socket says 172.21.15.143:46544) 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2f7fe700 1 -- 172.21.15.143:0/1458706123 learned_addr learned my addr 172.21.15.143:0/1458706123 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2ffff700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd30069d70 0x7efd30197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 -- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 msgr2=0x7efd30198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 -- 172.21.15.143:0/1458706123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd30069d70 msgr2=0x7efd30197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd30069d70 0x7efd30197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 -- 172.21.15.143:0/1458706123 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3019cd20 con 0x7efd301071c0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd3498f700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 0x7efd3019c5e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efd2800be20 tx=0x7efd28009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2d7fa700 10 client.?.objecter ms_handle_connect 0x7efd301071c0 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.993+0000 7efd2d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:50.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28019070 con 0x7efd301071c0 2024-03-11T21:25:50.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3019d010 con 0x7efd301071c0 2024-03-11T21:25:50.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7efd2800cc10 con 0x7efd301071c0 2024-03-11T21:25:50.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd280125a0 con 0x7efd301071c0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd363f2700 10 client.14703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd301a4890 con 0x7efd301071c0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd363f2700 1 librados: init done 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3004faa0 con 0x7efd301071c0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7efd2801c070 con 0x7efd301071c0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 0x7efd18071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(116..116 src has 66..116) v4 ==== 7450+0+0 (secure 0 0 0) 0x7efd2808dac0 con 0x7efd301071c0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 10 client.14703.objecter ms_dispatch 0x7efd30001550 osd_map(116..116 src has 66..116) v4 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 3 client.14703.objecter handle_osd_map got epochs [116,116] > 0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2ffff700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 0x7efd18071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 3 client.14703.objecter handle_osd_map decoding full epoch 116 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 20 client.14703.objecter dump_active .. 0 homeless 2024-03-11T21:25:50.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2ffff700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 0x7efd18071ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd20000c00 tx=0x7efd2000f520 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:50.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:50.997+0000 7efd2d7fa700 10 client.14703.objecter ms_handle_connect 0x7efd1806f820 2024-03-11T21:25:50.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.001+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7efd2805afe0 con 0x7efd301071c0 2024-03-11T21:25:51.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- 172.21.15.143:0/587296177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f01071c0 msgr2=0x7f31f010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- 172.21.15.143:0/587296177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f01071c0 0x7f31f010ed40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f31e800b0d0 tx=0x7f31e800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- 172.21.15.143:0/587296177 shutdown_connections 2024-03-11T21:25:51.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- 172.21.15.143:0/587296177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f01071c0 0x7f31f010ed40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- 172.21.15.143:0/587296177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f0069d70 0x7f31f006a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- 172.21.15.143:0/587296177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006a720 0x7f31f0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- 172.21.15.143:0/587296177 >> 172.21.15.143:0/587296177 conn(0x7f31f0076d80 msgr2=0x7f31f0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- 172.21.15.143:0/587296177 shutdown_connections 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- 172.21.15.143:0/587296177 wait complete. 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 librados: starting msgr at 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 librados: starting objecter 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 Processor -- start 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- start start 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 librados: setting wanted keys 2024-03-11T21:25:51.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 librados: calling monclient init 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f01071c0 0x7f31f019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0111680 con 0x7f31f0069d70 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0111500 con 0x7f31f01071c0 2024-03-11T21:25:51.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f66f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0111800 con 0x7f31f006a720 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46570/0 (socket says 172.21.15.143:46570) 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36600/0 (socket says 172.21.15.143:36600) 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ef7fe700 1 -- 172.21.15.143:0/3295927020 learned_addr learned my addr 172.21.15.143:0/3295927020 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31f4c90700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f01071c0 0x7f31f019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 -- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 msgr2=0x7f31f019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 -- 172.21.15.143:0/3295927020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f01071c0 msgr2=0x7f31f019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f01071c0 0x7f31f019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 -- 172.21.15.143:0/3295927020 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f019ee70 con 0x7f31f0069d70 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ef7fe700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31effff700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31e000c970 tx=0x7f31e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:51.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ed7fa700 10 client.?.objecter ms_handle_connect 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.033+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00078f0 con 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f019f160 con 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31f66f3700 10 client.14709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f31e000f450 con 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f01a69e0 con 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0018660 con 0x7f31f0069d70 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31f66f3700 1 librados: init done 2024-03-11T21:25:51.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f004faa0 con 0x7f31f0069d70 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f31e0007a90 con 0x7f31f0069d70 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 0x7f31d8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(116..116 src has 66..116) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f31e008e190 con 0x7f31f0069d70 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 10 client.14709.objecter ms_dispatch 0x7f31f0001550 osd_map(116..116 src has 66..116) v4 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 3 client.14709.objecter handle_osd_map got epochs [116,116] > 0 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 3 client.14709.objecter handle_osd_map decoding full epoch 116 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ef7fe700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 0x7f31d8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 20 client.14709.objecter dump_active .. 0 homeless 2024-03-11T21:25:51.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ed7fa700 10 client.14709.objecter ms_handle_connect 0x7f31d806f820 2024-03-11T21:25:51.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.037+0000 7f31ef7fe700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 0x7f31d8071ce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009680 tx=0x7f31e4009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.917816+0000 osd.4 (osd.4) 53 : cluster 0 6.1a scrub starts 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:48.919918+0000 osd.4 (osd.4) 54 : cluster 0 6.1a scrub ok 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:49.097613+0000 mgr.y (mgr.14550) 95 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:49.634992+0000 osd.1 (osd.1) 61 : cluster 0 2.6 scrub starts 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:49.638046+0000 osd.1 (osd.1) 62 : cluster 0 2.6 scrub ok 2024-03-11T21:25:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.004627+0000 osd.3 (osd.3) 75 : cluster 0 4.0 scrub starts 2024-03-11T21:25:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.006694+0000 osd.3 (osd.3) 76 : cluster 0 4.0 scrub ok 2024-03-11T21:25:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: audit 2024-03-11T21:25:50.201888+0000 mon.a (mon.0) 859 : audit 0 from='client.? 172.21.15.143:0/3173711335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:25:51.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:50 smithi203 bash[23654]: audit 2024-03-11T21:25:50.580567+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.143:0/180935073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:25:51.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.045+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f31e0014030 con 0x7f31f0069d70 2024-03-11T21:25:51.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.213+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efd301a4a60 con 0x7efd301071c0 2024-03-11T21:25:51.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.213+0000 7efd2d7fa700 1 -- 172.21.15.143:0/1458706123 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efd2805ea30 con 0x7efd301071c0 2024-03-11T21:25:51.192 INFO:teuthology.orchestra.run.smithi143.stdout:150323855433 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 10 librados: watch_flush enter 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 10 librados: watch_flush exit 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 20 client.14703.objecter shutdown clearing up homeless session... 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 10 client.14703.objecter successfully canceled tick 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 msgr2=0x7efd18071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 0x7efd18071ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd20000c00 tx=0x7efd2000f520 comp rx=0 tx=0).stop 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 msgr2=0x7efd3019c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 0x7efd3019c5e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efd2800be20 tx=0x7efd28009ed0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 shutdown_connections 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7efd1806f820 0x7efd18071ce0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd301071c0 0x7efd3019c5e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd30069d70 0x7efd30197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 --2- 172.21.15.143:0/1458706123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd3006a720 0x7efd30198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 >> 172.21.15.143:0/1458706123 conn(0x7efd30076d80 msgr2=0x7efd300febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:51.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 shutdown_connections 2024-03-11T21:25:51.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 -- 172.21.15.143:0/1458706123 wait complete. 2024-03-11T21:25:51.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.217+0000 7efd363f2700 1 librados: shutdown 2024-03-11T21:25:51.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.253+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f31f01a6c10 con 0x7f31f0069d70 2024-03-11T21:25:51.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.253+0000 7f31ed7fa700 1 -- 172.21.15.143:0/3295927020 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f31e005f060 con 0x7f31f0069d70 2024-03-11T21:25:51.233 INFO:teuthology.orchestra.run.smithi143.stdout:34359738484 2024-03-11T21:25:51.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 10 librados: watch_flush enter 2024-03-11T21:25:51.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 10 librados: watch_flush exit 2024-03-11T21:25:51.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 20 client.14709.objecter shutdown clearing up homeless session... 2024-03-11T21:25:51.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 10 client.14709.objecter successfully canceled tick 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 msgr2=0x7f31d8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 0x7f31d8071ce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009680 tx=0x7f31e4009260 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 msgr2=0x7f31f0199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31e000c970 tx=0x7f31e000cc80 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 shutdown_connections 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f31d806f820 0x7f31d8071ce0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f0069d70 0x7f31f0199e60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31f01071c0 0x7f31f019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 --2- 172.21.15.143:0/3295927020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f006a720 0x7f31f019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 >> 172.21.15.143:0/3295927020 conn(0x7f31f0076d80 msgr2=0x7f31f010d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 shutdown_connections 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 -- 172.21.15.143:0/3295927020 wait complete. 2024-03-11T21:25:51.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:25:51.257+0000 7f31f66f3700 1 librados: shutdown 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.629408+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.631540+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[29284]: audit 2024-03-11T21:25:51.219047+0000 mon.a (mon.0) 860 : audit 0 from='client.? 172.21.15.143:0/1458706123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[29284]: audit 2024-03-11T21:25:51.259046+0000 mon.a (mon.0) 861 : audit 0 from='client.? 172.21.15.143:0/3295927020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.629408+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.631540+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[21512]: audit 2024-03-11T21:25:51.219047+0000 mon.a (mon.0) 860 : audit 0 from='client.? 172.21.15.143:0/1458706123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:25:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:51 smithi143 bash[21512]: audit 2024-03-11T21:25:51.259046+0000 mon.a (mon.0) 861 : audit 0 from='client.? 172.21.15.143:0/3295927020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:25:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:51 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.629408+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-03-11T21:25:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:51 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.631540+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-03-11T21:25:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:51 smithi203 bash[23654]: audit 2024-03-11T21:25:51.219047+0000 mon.a (mon.0) 860 : audit 0 from='client.? 172.21.15.143:0/1458706123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:25:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:51 smithi203 bash[23654]: audit 2024-03-11T21:25:51.259046+0000 mon.a (mon.0) 861 : audit 0 from='client.? 172.21.15.143:0/3295927020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:25:52.438 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:25:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.972431+0000 osd.2 (osd.2) 53 : cluster 0 6.14 scrub starts 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:50.974029+0000 osd.2 (osd.2) 54 : cluster 0 6.14 scrub ok 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:51.098118+0000 mgr.y (mgr.14550) 96 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:51.430223+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:51.434471+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:52.030091+0000 osd.7 (osd.7) 73 : cluster 0 2.8 scrub starts 2024-03-11T21:25:52.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[29284]: cluster 2024-03-11T21:25:52.032596+0000 osd.7 (osd.7) 74 : cluster 0 2.8 scrub ok 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.972431+0000 osd.2 (osd.2) 53 : cluster 0 6.14 scrub starts 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:50.974029+0000 osd.2 (osd.2) 54 : cluster 0 6.14 scrub ok 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:51.098118+0000 mgr.y (mgr.14550) 96 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:51.430223+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:51.434471+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:52.030091+0000 osd.7 (osd.7) 73 : cluster 0 2.8 scrub starts 2024-03-11T21:25:52.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:52 smithi143 bash[21512]: cluster 2024-03-11T21:25:52.032596+0000 osd.7 (osd.7) 74 : cluster 0 2.8 scrub ok 2024-03-11T21:25:53.022 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841574 got 283467841580 for osd.7 2024-03-11T21:25:53.023 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:53.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.972431+0000 osd.2 (osd.2) 53 : cluster 0 6.14 scrub starts 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:50.974029+0000 osd.2 (osd.2) 54 : cluster 0 6.14 scrub ok 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:51.098118+0000 mgr.y (mgr.14550) 96 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:51.430223+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:51.434471+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:52.030091+0000 osd.7 (osd.7) 73 : cluster 0 2.8 scrub starts 2024-03-11T21:25:53.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:52 smithi203 bash[23654]: cluster 2024-03-11T21:25:52.032596+0000 osd.7 (osd.7) 74 : cluster 0 2.8 scrub ok 2024-03-11T21:25:53.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495673 got 197568495679 for osd.5 2024-03-11T21:25:53.725 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[29284]: cluster 2024-03-11T21:25:52.413023+0000 osd.5 (osd.5) 69 : cluster 0 5.7 scrub starts 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[29284]: cluster 2024-03-11T21:25:52.415458+0000 osd.5 (osd.5) 70 : cluster 0 5.7 scrub ok 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[29284]: cluster 2024-03-11T21:25:53.049204+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[29284]: cluster 2024-03-11T21:25:53.051412+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[21512]: cluster 2024-03-11T21:25:52.413023+0000 osd.5 (osd.5) 69 : cluster 0 5.7 scrub starts 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[21512]: cluster 2024-03-11T21:25:52.415458+0000 osd.5 (osd.5) 70 : cluster 0 5.7 scrub ok 2024-03-11T21:25:53.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[21512]: cluster 2024-03-11T21:25:53.049204+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-03-11T21:25:53.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:53 smithi143 bash[21512]: cluster 2024-03-11T21:25:53.051412+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-03-11T21:25:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:53 smithi203 bash[23654]: cluster 2024-03-11T21:25:52.413023+0000 osd.5 (osd.5) 69 : cluster 0 5.7 scrub starts 2024-03-11T21:25:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:53 smithi203 bash[23654]: cluster 2024-03-11T21:25:52.415458+0000 osd.5 (osd.5) 70 : cluster 0 5.7 scrub ok 2024-03-11T21:25:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:53 smithi203 bash[23654]: cluster 2024-03-11T21:25:53.049204+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-03-11T21:25:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:53 smithi203 bash[23654]: cluster 2024-03-11T21:25:53.051412+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-03-11T21:25:54.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738478 got 34359738484 for osd.0 2024-03-11T21:25:54.301 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:54.628 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855426 got 150323855433 for osd.4 2024-03-11T21:25:54.628 DEBUG:teuthology.parallel:result is None 2024-03-11T21:25:54.628 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:25:54.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[29284]: cluster 2024-03-11T21:25:53.099716+0000 mgr.y (mgr.14550) 97 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[29284]: cluster 2024-03-11T21:25:53.704168+0000 osd.1 (osd.1) 65 : cluster 0 2.15 deep-scrub starts 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[29284]: cluster 2024-03-11T21:25:53.706216+0000 osd.1 (osd.1) 66 : cluster 0 2.15 deep-scrub ok 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[21512]: cluster 2024-03-11T21:25:53.099716+0000 mgr.y (mgr.14550) 97 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[21512]: cluster 2024-03-11T21:25:53.704168+0000 osd.1 (osd.1) 65 : cluster 0 2.15 deep-scrub starts 2024-03-11T21:25:54.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:54 smithi143 bash[21512]: cluster 2024-03-11T21:25:53.706216+0000 osd.1 (osd.1) 66 : cluster 0 2.15 deep-scrub ok 2024-03-11T21:25:55.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:54 smithi203 bash[23654]: cluster 2024-03-11T21:25:53.099716+0000 mgr.y (mgr.14550) 97 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:55.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:54 smithi203 bash[23654]: cluster 2024-03-11T21:25:53.704168+0000 osd.1 (osd.1) 65 : cluster 0 2.15 deep-scrub starts 2024-03-11T21:25:55.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:54 smithi203 bash[23654]: cluster 2024-03-11T21:25:53.706216+0000 osd.1 (osd.1) 66 : cluster 0 2.15 deep-scrub ok 2024-03-11T21:25:55.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:55 smithi143 systemd[1]: Stopping Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:25:55.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[58551]: ts=2024-03-11T21:25:55.734Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:25:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[21512]: cluster 2024-03-11T21:25:54.340255+0000 osd.0 (osd.0) 61 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:25:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[21512]: cluster 2024-03-11T21:25:54.342580+0000 osd.0 (osd.0) 62 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:25:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[21512]: cluster 2024-03-11T21:25:54.358584+0000 osd.5 (osd.5) 71 : cluster 0 5.16 scrub starts 2024-03-11T21:25:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[21512]: cluster 2024-03-11T21:25:54.360637+0000 osd.5 (osd.5) 72 : cluster 0 5.16 scrub ok 2024-03-11T21:25:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[29284]: cluster 2024-03-11T21:25:54.340255+0000 osd.0 (osd.0) 61 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:25:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[29284]: cluster 2024-03-11T21:25:54.342580+0000 osd.0 (osd.0) 62 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:25:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[29284]: cluster 2024-03-11T21:25:54.358584+0000 osd.5 (osd.5) 71 : cluster 0 5.16 scrub starts 2024-03-11T21:25:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:55 smithi143 bash[29284]: cluster 2024-03-11T21:25:54.360637+0000 osd.5 (osd.5) 72 : cluster 0 5.16 scrub ok 2024-03-11T21:25:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:55 smithi203 bash[23654]: cluster 2024-03-11T21:25:54.340255+0000 osd.0 (osd.0) 61 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:25:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:55 smithi203 bash[23654]: cluster 2024-03-11T21:25:54.342580+0000 osd.0 (osd.0) 62 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:25:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:55 smithi203 bash[23654]: cluster 2024-03-11T21:25:54.358584+0000 osd.5 (osd.5) 71 : cluster 0 5.16 scrub starts 2024-03-11T21:25:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:55 smithi203 bash[23654]: cluster 2024-03-11T21:25:54.360637+0000 osd.5 (osd.5) 72 : cluster 0 5.16 scrub ok 2024-03-11T21:25:56.597 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[63300]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:25:56.597 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[63360]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:25:56.597 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:56 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:25:56.597 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:56 smithi143 systemd[1]: Stopped Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:25:56.597 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:56 smithi143 systemd[1]: Started Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:25:56.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[21512]: cluster 2024-03-11T21:25:55.100423+0000 mgr.y (mgr.14550) 98 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:56.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.139166+0000 osd.3 (osd.3) 77 : cluster 0 6.11 scrub starts 2024-03-11T21:25:56.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.141548+0000 osd.3 (osd.3) 78 : cluster 0 6.11 scrub ok 2024-03-11T21:25:56.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[29284]: cluster 2024-03-11T21:25:55.100423+0000 mgr.y (mgr.14550) 98 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:56.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.139166+0000 osd.3 (osd.3) 77 : cluster 0 6.11 scrub starts 2024-03-11T21:25:56.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:56 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.141548+0000 osd.3 (osd.3) 78 : cluster 0 6.11 scrub ok 2024-03-11T21:25:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:56 smithi203 bash[23654]: cluster 2024-03-11T21:25:55.100423+0000 mgr.y (mgr.14550) 98 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:56 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.139166+0000 osd.3 (osd.3) 77 : cluster 0 6.11 scrub starts 2024-03-11T21:25:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:56 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.141548+0000 osd.3 (osd.3) 78 : cluster 0 6.11 scrub ok 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.744Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.744Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.746Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.143 port=9094 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.747Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.778Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.778Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.781Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:25:57.936 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[63388]: ts=2024-03-11T21:25:57.781Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.341659+0000 osd.0 (osd.0) 63 : cluster 0 6.10 scrub starts 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.343942+0000 osd.0 (osd.0) 64 : cluster 0 6.10 scrub ok 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: audit 2024-03-11T21:25:56.632572+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: audit 2024-03-11T21:25:56.641457+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cephadm 2024-03-11T21:25:56.646409+0000 mgr.y (mgr.14550) 99 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.732254+0000 osd.1 (osd.1) 67 : cluster 0 3.0 scrub starts 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:56.736708+0000 osd.1 (osd.1) 68 : cluster 0 3.0 scrub ok 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cephadm 2024-03-11T21:25:56.888739+0000 mgr.y (mgr.14550) 100 : cephadm 1 Reconfiguring daemon prometheus.a on smithi203 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.000525+0000 osd.4 (osd.4) 55 : cluster 0 5.e scrub starts 2024-03-11T21:25:57.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.003421+0000 osd.4 (osd.4) 56 : cluster 0 5.e scrub ok 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.341659+0000 osd.0 (osd.0) 63 : cluster 0 6.10 scrub starts 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.343942+0000 osd.0 (osd.0) 64 : cluster 0 6.10 scrub ok 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: audit 2024-03-11T21:25:56.632572+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: audit 2024-03-11T21:25:56.641457+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cephadm 2024-03-11T21:25:56.646409+0000 mgr.y (mgr.14550) 99 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.732254+0000 osd.1 (osd.1) 67 : cluster 0 3.0 scrub starts 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:56.736708+0000 osd.1 (osd.1) 68 : cluster 0 3.0 scrub ok 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cephadm 2024-03-11T21:25:56.888739+0000 mgr.y (mgr.14550) 100 : cephadm 1 Reconfiguring daemon prometheus.a on smithi203 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.000525+0000 osd.4 (osd.4) 55 : cluster 0 5.e scrub starts 2024-03-11T21:25:57.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:57 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.003421+0000 osd.4 (osd.4) 56 : cluster 0 5.e scrub ok 2024-03-11T21:25:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.341659+0000 osd.0 (osd.0) 63 : cluster 0 6.10 scrub starts 2024-03-11T21:25:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.343942+0000 osd.0 (osd.0) 64 : cluster 0 6.10 scrub ok 2024-03-11T21:25:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: audit 2024-03-11T21:25:56.632572+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: audit 2024-03-11T21:25:56.641457+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cephadm 2024-03-11T21:25:56.646409+0000 mgr.y (mgr.14550) 99 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.732254+0000 osd.1 (osd.1) 67 : cluster 0 3.0 scrub starts 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:56.736708+0000 osd.1 (osd.1) 68 : cluster 0 3.0 scrub ok 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cephadm 2024-03-11T21:25:56.888739+0000 mgr.y (mgr.14550) 100 : cephadm 1 Reconfiguring daemon prometheus.a on smithi203 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.000525+0000 osd.4 (osd.4) 55 : cluster 0 5.e scrub starts 2024-03-11T21:25:58.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:57 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.003421+0000 osd.4 (osd.4) 56 : cluster 0 5.e scrub ok 2024-03-11T21:25:58.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.101057+0000 mgr.y (mgr.14550) 101 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.395426+0000 osd.5 (osd.5) 73 : cluster 0 6.19 scrub starts 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.398029+0000 osd.5 (osd.5) 74 : cluster 0 6.19 scrub ok 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: audit 2024-03-11T21:25:57.664539+0000 mon.c (mon.2) 68 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.989444+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[21512]: cluster 2024-03-11T21:25:57.991789+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-03-11T21:25:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.101057+0000 mgr.y (mgr.14550) 101 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.395426+0000 osd.5 (osd.5) 73 : cluster 0 6.19 scrub starts 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.398029+0000 osd.5 (osd.5) 74 : cluster 0 6.19 scrub ok 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: audit 2024-03-11T21:25:57.664539+0000 mon.c (mon.2) 68 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.989444+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:58 smithi143 bash[29284]: cluster 2024-03-11T21:25:57.991789+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.101057+0000 mgr.y (mgr.14550) 101 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.395426+0000 osd.5 (osd.5) 73 : cluster 0 6.19 scrub starts 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.398029+0000 osd.5 (osd.5) 74 : cluster 0 6.19 scrub ok 2024-03-11T21:25:58.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: audit 2024-03-11T21:25:57.664539+0000 mon.c (mon.2) 68 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:58.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.989444+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-03-11T21:25:58.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:58 smithi203 bash[23654]: cluster 2024-03-11T21:25:57.991789+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-03-11T21:25:59.935 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[63388]: ts=2024-03-11T21:25:59.748Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000626201s 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.020126+0000 osd.2 (osd.2) 55 : cluster 0 5.6 scrub starts 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.022723+0000 osd.2 (osd.2) 56 : cluster 0 5.6 scrub ok 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: audit 2024-03-11T21:25:58.158449+0000 mgr.y (mgr.14550) 102 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.330017+0000 osd.0 (osd.0) 65 : cluster 0 5.3 scrub starts 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.332805+0000 osd.0 (osd.0) 66 : cluster 0 5.3 scrub ok 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.433625+0000 osd.5 (osd.5) 75 : cluster 0 5.f scrub starts 2024-03-11T21:25:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.434775+0000 osd.5 (osd.5) 76 : cluster 0 5.f scrub ok 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.942458+0000 osd.4 (osd.4) 59 : cluster 0 5.18 scrub starts 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[21512]: cluster 2024-03-11T21:25:58.944356+0000 osd.4 (osd.4) 60 : cluster 0 5.18 scrub ok 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.020126+0000 osd.2 (osd.2) 55 : cluster 0 5.6 scrub starts 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.022723+0000 osd.2 (osd.2) 56 : cluster 0 5.6 scrub ok 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: audit 2024-03-11T21:25:58.158449+0000 mgr.y (mgr.14550) 102 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.330017+0000 osd.0 (osd.0) 65 : cluster 0 5.3 scrub starts 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.332805+0000 osd.0 (osd.0) 66 : cluster 0 5.3 scrub ok 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.433625+0000 osd.5 (osd.5) 75 : cluster 0 5.f scrub starts 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.434775+0000 osd.5 (osd.5) 76 : cluster 0 5.f scrub ok 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.942458+0000 osd.4 (osd.4) 59 : cluster 0 5.18 scrub starts 2024-03-11T21:25:59.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:25:59 smithi143 bash[29284]: cluster 2024-03-11T21:25:58.944356+0000 osd.4 (osd.4) 60 : cluster 0 5.18 scrub ok 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.020126+0000 osd.2 (osd.2) 55 : cluster 0 5.6 scrub starts 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.022723+0000 osd.2 (osd.2) 56 : cluster 0 5.6 scrub ok 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: audit 2024-03-11T21:25:58.158449+0000 mgr.y (mgr.14550) 102 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.330017+0000 osd.0 (osd.0) 65 : cluster 0 5.3 scrub starts 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.332805+0000 osd.0 (osd.0) 66 : cluster 0 5.3 scrub ok 2024-03-11T21:25:59.938 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.433625+0000 osd.5 (osd.5) 75 : cluster 0 5.f scrub starts 2024-03-11T21:25:59.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.434775+0000 osd.5 (osd.5) 76 : cluster 0 5.f scrub ok 2024-03-11T21:25:59.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.942458+0000 osd.4 (osd.4) 59 : cluster 0 5.18 scrub starts 2024-03-11T21:25:59.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:25:59 smithi203 bash[23654]: cluster 2024-03-11T21:25:58.944356+0000 osd.4 (osd.4) 60 : cluster 0 5.18 scrub ok 2024-03-11T21:26:00.106 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:00.628 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 systemd[1]: Stopping Ceph prometheus.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:26:00.628 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:26:00.628 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:26:00.628 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:26:00.628 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.495Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.496Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.496Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:26:00.629 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[49511]: ts=2024-03-11T21:26:00.496Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:26:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.102571+0000 mgr.y (mgr.14550) 103 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.197924+0000 osd.7 (osd.7) 77 : cluster 0 2.5 scrub starts 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.200261+0000 osd.7 (osd.7) 78 : cluster 0 2.5 scrub ok 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.438990+0000 osd.5 (osd.5) 77 : cluster 0 5.15 scrub starts 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.442004+0000 osd.5 (osd.5) 78 : cluster 0 5.15 scrub ok 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.986190+0000 osd.4 (osd.4) 61 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[21512]: cluster 2024-03-11T21:25:59.988511+0000 osd.4 (osd.4) 62 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.102571+0000 mgr.y (mgr.14550) 103 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.197924+0000 osd.7 (osd.7) 77 : cluster 0 2.5 scrub starts 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.200261+0000 osd.7 (osd.7) 78 : cluster 0 2.5 scrub ok 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.438990+0000 osd.5 (osd.5) 77 : cluster 0 5.15 scrub starts 2024-03-11T21:26:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.442004+0000 osd.5 (osd.5) 78 : cluster 0 5.15 scrub ok 2024-03-11T21:26:00.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.986190+0000 osd.4 (osd.4) 61 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:26:00.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:00 smithi143 bash[29284]: cluster 2024-03-11T21:25:59.988511+0000 osd.4 (osd.4) 62 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:26:01.022 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[52037]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:26:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.102571+0000 mgr.y (mgr.14550) 103 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.197924+0000 osd.7 (osd.7) 77 : cluster 0 2.5 scrub starts 2024-03-11T21:26:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.200261+0000 osd.7 (osd.7) 78 : cluster 0 2.5 scrub ok 2024-03-11T21:26:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.438990+0000 osd.5 (osd.5) 77 : cluster 0 5.15 scrub starts 2024-03-11T21:26:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.442004+0000 osd.5 (osd.5) 78 : cluster 0 5.15 scrub ok 2024-03-11T21:26:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.986190+0000 osd.4 (osd.4) 61 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:26:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:00 smithi203 bash[23654]: cluster 2024-03-11T21:25:59.988511+0000 osd.4 (osd.4) 62 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:26:01.457 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPING 2024-03-11T21:26:01.457 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:26:01.457 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPED 2024-03-11T21:26:01.457 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTING 2024-03-11T21:26:01.457 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Serving on http://:::9283 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTED 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPING 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPED 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTING 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Serving on http://:::9283 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTED 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPING 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STOPPED 2024-03-11T21:26:01.458 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTING 2024-03-11T21:26:01.522 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52096]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:26:01.522 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T21:26:01.522 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 systemd[1]: Stopped Ceph prometheus.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:26:01.522 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 systemd[1]: Started Ceph prometheus.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:26:01.936 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Serving on http://:::9283 2024-03-11T21:26:01.936 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:01 smithi143 bash[21768]: [11/Mar/2024:21:26:01] ENGINE Bus STARTED 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.885Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.885Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.885Z caller=main.go:570 level=info host_details="(Linux 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 smithi203 (none))" 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.885Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.885Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.887Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.887Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.888Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:26:02.023 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.888Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.890Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.890Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.029µs 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.890Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.891Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.892Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.892Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=43.26µs wal_replay_duration=1.806958ms wbl_replay_duration=116ns total_replay_duration=1.870063ms 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.893Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.893Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.893Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.917Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.173474ms db_storage=759ns remote_storage=1.12µs web_handler=372ns query_engine=568ns scrape=317.293µs scrape_sd=114.226µs notify=17.092µs notify_sd=17.869µs rules=23.352184ms tracing=5.775µs 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.917Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:26:02.024 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:26:01 smithi203 bash[52126]: ts=2024-03-11T21:26:01.917Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: cluster 2024-03-11T21:26:00.762820+0000 osd.1 (osd.1) 69 : cluster 0 5.1d scrub starts 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: cluster 2024-03-11T21:26:00.765261+0000 osd.1 (osd.1) 70 : cluster 0 5.1d scrub ok 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: cluster 2024-03-11T21:26:01.103156+0000 mgr.y (mgr.14550) 104 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.123667+0000 mon.a (mon.0) 864 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.141910+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.180826+0000 mon.c (mon.2) 69 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.181930+0000 mgr.y (mgr.14550) 105 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.184178+0000 mon.c (mon.2) 70 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.185127+0000 mgr.y (mgr.14550) 106 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.194902+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.225383+0000 mon.c (mon.2) 71 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.226247+0000 mgr.y (mgr.14550) 107 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.230115+0000 mon.c (mon.2) 72 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.231300+0000 mgr.y (mgr.14550) 108 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.238506+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.254732+0000 mon.c (mon.2) 73 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.255409+0000 mgr.y (mgr.14550) 109 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.260372+0000 mon.c (mon.2) 74 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.260907+0000 mgr.y (mgr.14550) 110 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.267609+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: cluster 2024-03-11T21:26:00.762820+0000 osd.1 (osd.1) 69 : cluster 0 5.1d scrub starts 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: cluster 2024-03-11T21:26:00.765261+0000 osd.1 (osd.1) 70 : cluster 0 5.1d scrub ok 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: cluster 2024-03-11T21:26:01.103156+0000 mgr.y (mgr.14550) 104 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.123667+0000 mon.a (mon.0) 864 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.141910+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.180826+0000 mon.c (mon.2) 69 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.181930+0000 mgr.y (mgr.14550) 105 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.184178+0000 mon.c (mon.2) 70 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.185127+0000 mgr.y (mgr.14550) 106 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.194902+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.225383+0000 mon.c (mon.2) 71 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.226247+0000 mgr.y (mgr.14550) 107 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.230115+0000 mon.c (mon.2) 72 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.231300+0000 mgr.y (mgr.14550) 108 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.238506+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.254732+0000 mon.c (mon.2) 73 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.255409+0000 mgr.y (mgr.14550) 109 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.260372+0000 mon.c (mon.2) 74 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.260907+0000 mgr.y (mgr.14550) 110 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.267609+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: audit 2024-03-11T21:26:01.293832+0000 mon.c (mon.2) 75 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: cluster 2024-03-11T21:26:01.308445+0000 osd.0 (osd.0) 67 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[29284]: cluster 2024-03-11T21:26:01.314608+0000 osd.0 (osd.0) 68 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: audit 2024-03-11T21:26:01.293832+0000 mon.c (mon.2) 75 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:02.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: cluster 2024-03-11T21:26:01.308445+0000 osd.0 (osd.0) 67 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:26:02.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:02 smithi143 bash[21512]: cluster 2024-03-11T21:26:01.314608+0000 osd.0 (osd.0) 68 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: cluster 2024-03-11T21:26:00.762820+0000 osd.1 (osd.1) 69 : cluster 0 5.1d scrub starts 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: cluster 2024-03-11T21:26:00.765261+0000 osd.1 (osd.1) 70 : cluster 0 5.1d scrub ok 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: cluster 2024-03-11T21:26:01.103156+0000 mgr.y (mgr.14550) 104 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.123667+0000 mon.a (mon.0) 864 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.141910+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.180826+0000 mon.c (mon.2) 69 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.181930+0000 mgr.y (mgr.14550) 105 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.184178+0000 mon.c (mon.2) 70 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.185127+0000 mgr.y (mgr.14550) 106 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi143.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.194902+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.225383+0000 mon.c (mon.2) 71 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.226247+0000 mgr.y (mgr.14550) 107 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.230115+0000 mon.c (mon.2) 72 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.231300+0000 mgr.y (mgr.14550) 108 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.238506+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.254732+0000 mon.c (mon.2) 73 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.255409+0000 mgr.y (mgr.14550) 109 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.260372+0000 mon.c (mon.2) 74 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.260907+0000 mgr.y (mgr.14550) 110 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:26:02.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.267609+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:02.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: audit 2024-03-11T21:26:01.293832+0000 mon.c (mon.2) 75 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:02.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: cluster 2024-03-11T21:26:01.308445+0000 osd.0 (osd.0) 67 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:26:02.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:02 smithi203 bash[23654]: cluster 2024-03-11T21:26:01.314608+0000 osd.0 (osd.0) 68 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.034574+0000 osd.4 (osd.4) 63 : cluster 0 6.2 scrub starts 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.037037+0000 osd.4 (osd.4) 64 : cluster 0 6.2 scrub ok 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.210196+0000 osd.3 (osd.3) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.210374+0000 osd.3 (osd.3) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.034574+0000 osd.4 (osd.4) 63 : cluster 0 6.2 scrub starts 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.037037+0000 osd.4 (osd.4) 64 : cluster 0 6.2 scrub ok 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.210196+0000 osd.3 (osd.3) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:03.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:03 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.210374+0000 osd.3 (osd.3) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:03.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- 172.21.15.143:0/1506722872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108410 msgr2=0x7f2ee8108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:03.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/1506722872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108410 0x7f2ee8108890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80098f0 tx=0x7f2ed8009c00 comp rx=0 tx=0).stop 2024-03-11T21:26:03.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- 172.21.15.143:0/1506722872 shutdown_connections 2024-03-11T21:26:03.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/1506722872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee8108dd0 0x7f2ee81152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/1506722872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee810e4a0 0x7f2ee810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/1506722872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108410 0x7f2ee8108890 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- 172.21.15.143:0/1506722872 >> 172.21.15.143:0/1506722872 conn(0x7f2ee80755b0 msgr2=0x7f2ee80779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:03.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- 172.21.15.143:0/1506722872 shutdown_connections 2024-03-11T21:26:03.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- 172.21.15.143:0/1506722872 wait complete. 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 librados: starting msgr at 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 librados: starting objecter 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 Processor -- start 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- start start 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 librados: setting wanted keys 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 librados: calling monclient init 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8108410 0x7f2ee819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 0x7f2ee81a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8117bf0 con 0x7f2ee810e4a0 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8117a70 con 0x7f2ee8108410 2024-03-11T21:26:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eeeaed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8117d70 con 0x7f2ee8108dd0 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eed08a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 0x7f2ee81a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46294/0 (socket says 172.21.15.143:46294) 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 -- 172.21.15.143:0/4057390845 learned_addr learned my addr 172.21.15.143:0/4057390845 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eec889700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8108410 0x7f2ee819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:03.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 -- 172.21.15.143:0/4057390845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8108410 msgr2=0x7f2ee819e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:03.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8108410 0x7f2ee819e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 -- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 msgr2=0x7f2ee81a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 0x7f2ee81a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 -- 172.21.15.143:0/4057390845 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee81a3190 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2eed08a700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 0x7f2ee81a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee7fff700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80059d0 tx=0x7f2ed800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee5ffb700 10 client.?.objecter ms_handle_connect 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee5ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8027070 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2ed800bb60 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.485+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8004930 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee81a3420 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eeeaed700 10 client.24590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee81aad00 con 0x7f2ee8108dd0 2024-03-11T21:26:03.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eeeaed700 1 librados: init done 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee8109b50 con 0x7f2ee8108dd0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2ed802a070 con 0x7f2ee8108dd0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 0x7f2ec8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(117..117 src has 66..117) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f2ed808ed00 con 0x7f2ee8108dd0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 10 client.24590.objecter ms_dispatch 0x7f2ee8001550 osd_map(117..117 src has 66..117) v4 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 3 client.24590.objecter handle_osd_map got epochs [117,117] > 0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 3 client.24590.objecter handle_osd_map decoding full epoch 117 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eec889700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 0x7f2ec8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 20 client.24590.objecter dump_active .. 0 homeless 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2ee5ffb700 10 client.24590.objecter ms_handle_connect 0x7f2ec806f820 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.489+0000 7f2eec889700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 0x7f2ec8071ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0005fd0 tx=0x7f2ed000af20 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:03.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.493+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2ed8013030 con 0x7f2ee8108dd0 2024-03-11T21:26:03.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:03 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.034574+0000 osd.4 (osd.4) 63 : cluster 0 6.2 scrub starts 2024-03-11T21:26:03.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:03 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.037037+0000 osd.4 (osd.4) 64 : cluster 0 6.2 scrub ok 2024-03-11T21:26:03.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:03 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.210196+0000 osd.3 (osd.3) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:03.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:03 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.210374+0000 osd.3 (osd.3) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:03.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.705+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ee80760a0 con 0x7f2ec806f820 2024-03-11T21:26:03.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.741+0000 7f2ee5ffb700 1 -- 172.21.15.143:0/4057390845 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346847 (secure 0 0 0) 0x7f2ee80760a0 con 0x7f2ec806f820 2024-03-11T21:26:03.721 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:26:03.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 10 librados: watch_flush enter 2024-03-11T21:26:03.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 10 librados: watch_flush exit 2024-03-11T21:26:03.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 20 client.24590.objecter shutdown clearing up homeless session... 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 10 client.24590.objecter successfully canceled tick 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 msgr2=0x7f2ec8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 0x7f2ec8071ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0005fd0 tx=0x7f2ed000af20 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 msgr2=0x7f2ee819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80059d0 tx=0x7f2ed800fa70 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 shutdown_connections 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ec806f820 0x7f2ec8071ce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee810e4a0 0x7f2ee81a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8108410 0x7f2ee819e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 --2- 172.21.15.143:0/4057390845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee8108dd0 0x7f2ee819e6c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 >> 172.21.15.143:0/4057390845 conn(0x7f2ee80755b0 msgr2=0x7f2ee8075990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 shutdown_connections 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 -- 172.21.15.143:0/4057390845 wait complete. 2024-03-11T21:26:03.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:03.745+0000 7f2eeeaed700 1 librados: shutdown 2024-03-11T21:26:03.729 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.468412+0000 osd.5 (osd.5) 79 : cluster 0 2.18 scrub starts 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.470783+0000 osd.5 (osd.5) 80 : cluster 0 2.18 scrub ok 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.939703+0000 osd.6 (osd.6) 43 : cluster 0 3.a scrub starts 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.941737+0000 osd.2 (osd.2) 57 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.944032+0000 osd.2 (osd.2) 58 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:02.944772+0000 osd.6 (osd.6) 44 : cluster 0 3.a scrub ok 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.104514+0000 mgr.y (mgr.14550) 111 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:04.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.140924+0000 mon.a (mon.0) 869 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.334653+0000 osd.0 (osd.0) 69 : cluster 0 2.1f scrub starts 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.337005+0000 osd.0 (osd.0) 70 : cluster 0 2.1f scrub ok 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[21512]: audit 2024-03-11T21:26:03.710776+0000 mgr.y (mgr.14550) 112 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.468412+0000 osd.5 (osd.5) 79 : cluster 0 2.18 scrub starts 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.470783+0000 osd.5 (osd.5) 80 : cluster 0 2.18 scrub ok 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.939703+0000 osd.6 (osd.6) 43 : cluster 0 3.a scrub starts 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.941737+0000 osd.2 (osd.2) 57 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.944032+0000 osd.2 (osd.2) 58 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:02.944772+0000 osd.6 (osd.6) 44 : cluster 0 3.a scrub ok 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.104514+0000 mgr.y (mgr.14550) 111 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:04.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.140924+0000 mon.a (mon.0) 869 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:26:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.334653+0000 osd.0 (osd.0) 69 : cluster 0 2.1f scrub starts 2024-03-11T21:26:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.337005+0000 osd.0 (osd.0) 70 : cluster 0 2.1f scrub ok 2024-03-11T21:26:04.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:04 smithi143 bash[29284]: audit 2024-03-11T21:26:03.710776+0000 mgr.y (mgr.14550) 112 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.468412+0000 osd.5 (osd.5) 79 : cluster 0 2.18 scrub starts 2024-03-11T21:26:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.470783+0000 osd.5 (osd.5) 80 : cluster 0 2.18 scrub ok 2024-03-11T21:26:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.939703+0000 osd.6 (osd.6) 43 : cluster 0 3.a scrub starts 2024-03-11T21:26:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.941737+0000 osd.2 (osd.2) 57 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.944032+0000 osd.2 (osd.2) 58 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:02.944772+0000 osd.6 (osd.6) 44 : cluster 0 3.a scrub ok 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.104514+0000 mgr.y (mgr.14550) 111 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.140924+0000 mon.a (mon.0) 869 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.334653+0000 osd.0 (osd.0) 69 : cluster 0 2.1f scrub starts 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.337005+0000 osd.0 (osd.0) 70 : cluster 0 2.1f scrub ok 2024-03-11T21:26:04.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:04 smithi203 bash[23654]: audit 2024-03-11T21:26:03.710776+0000 mgr.y (mgr.14550) 112 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:04.608 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-03-11T21:26:03.134889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1111,"num_read_kb":957,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":674,"ondisk_log_size":674,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249276,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711940,"statfs":{"total":767960285184,"available":767705026560,"internally_reserved":0,"allocated":8384512,"data_stored":5136424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.036524"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990197+0000","last_change":"2024-03-11T21:24:49.960853+0000","last_active":"2024-03-11T21:25:47.990197+0000","last_peered":"2024-03-11T21:25:47.990197+0000","last_clean":"2024-03-11T21:25:47.990197+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:25:47.990197+0000","last_undegraded":"2024-03-11T21:25:47.990197+0000","last_fullsized":"2024-03-11T21:25:47.990197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:49.960558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:49.960558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:06.409818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:58.944464+0000","last_change":"2024-03-11T21:25:58.944512+0000","last_active":"2024-03-11T21:25:58.944464+0000","last_peered":"2024-03-11T21:25:58.944464+0000","last_clean":"2024-03-11T21:25:58.944464+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:25:58.944464+0000","last_undegraded":"2024-03-11T21:25:58.944464+0000","last_fullsized":"2024-03-11T21:25:58.944464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.944364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.944364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.570898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.988437+0000","last_change":"2024-03-11T21:24:32.552575+0000","last_active":"2024-03-11T21:25:47.988437+0000","last_peered":"2024-03-11T21:25:47.988437+0000","last_clean":"2024-03-11T21:25:47.988437+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:25:47.988437+0000","last_undegraded":"2024-03-11T21:25:47.988437+0000","last_fullsized":"2024-03-11T21:25:47.988437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:24:32.552332+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:00.895943+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.991519+0000","last_change":"2024-03-11T21:25:31.918871+0000","last_active":"2024-03-11T21:25:47.991519+0000","last_peered":"2024-03-11T21:25:47.991519+0000","last_clean":"2024-03-11T21:25:47.991519+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:25:47.991519+0000","last_undegraded":"2024-03-11T21:25:47.991519+0000","last_fullsized":"2024-03-11T21:25:47.991519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:31.918660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.918660+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.711285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.920104+0000","last_change":"2024-03-11T21:25:48.920160+0000","last_active":"2024-03-11T21:25:48.920104+0000","last_peered":"2024-03-11T21:25:48.920104+0000","last_clean":"2024-03-11T21:25:48.920104+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:25:48.920104+0000","last_undegraded":"2024-03-11T21:25:48.920104+0000","last_fullsized":"2024-03-11T21:25:48.920104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.919930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.919930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.064632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.342205+0000","last_change":"2024-03-11T21:24:42.336554+0000","last_active":"2024-03-11T21:25:47.342205+0000","last_peered":"2024-03-11T21:25:47.342205+0000","last_clean":"2024-03-11T21:25:47.342205+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:25:47.342205+0000","last_undegraded":"2024-03-11T21:25:47.342205+0000","last_fullsized":"2024-03-11T21:25:47.342205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:42.336276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:42.336276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:07.379877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.991759+0000","last_change":"2024-03-11T21:24:42.037060+0000","last_active":"2024-03-11T21:25:47.991759+0000","last_peered":"2024-03-11T21:25:47.991759+0000","last_clean":"2024-03-11T21:25:47.991759+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:25:47.991759+0000","last_undegraded":"2024-03-11T21:25:47.991759+0000","last_fullsized":"2024-03-11T21:25:47.991759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:42.036821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:24:42.036821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:58.322946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.988297+0000","last_change":"2024-03-11T21:25:46.288260+0000","last_active":"2024-03-11T21:25:47.988297+0000","last_peered":"2024-03-11T21:25:47.988297+0000","last_clean":"2024-03-11T21:25:47.988297+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:25:47.988297+0000","last_undegraded":"2024-03-11T21:25:47.988297+0000","last_fullsized":"2024-03-11T21:25:47.988297+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:46.288021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.288021+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.040825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:57.398191+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:25:57.398191+0000","last_peered":"2024-03-11T21:25:57.398191+0000","last_clean":"2024-03-11T21:25:57.398191+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:25:57.398191+0000","last_undegraded":"2024-03-11T21:25:57.398191+0000","last_fullsized":"2024-03-11T21:25:57.398191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.062649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047829+0000","last_change":"2024-03-11T21:24:45.130299+0000","last_active":"2024-03-11T21:25:48.047829+0000","last_peered":"2024-03-11T21:25:48.047829+0000","last_clean":"2024-03-11T21:25:48.047829+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:25:48.047829+0000","last_undegraded":"2024-03-11T21:25:48.047829+0000","last_fullsized":"2024-03-11T21:25:48.047829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:45.130084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:45.130084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:55.211095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048488+0000","last_change":"2024-03-11T21:25:25.221221+0000","last_active":"2024-03-11T21:25:48.048488+0000","last_peered":"2024-03-11T21:25:48.048488+0000","last_clean":"2024-03-11T21:25:48.048488+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:25:48.048488+0000","last_undegraded":"2024-03-11T21:25:48.048488+0000","last_fullsized":"2024-03-11T21:25:48.048488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:25.220993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:25.220993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:32.679280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.891212+0000","last_change":"2024-03-11T21:25:19.442614+0000","last_active":"2024-03-11T21:25:47.891212+0000","last_peered":"2024-03-11T21:25:47.891212+0000","last_clean":"2024-03-11T21:25:47.891212+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:25:47.891212+0000","last_undegraded":"2024-03-11T21:25:47.891212+0000","last_fullsized":"2024-03-11T21:25:47.891212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:19.442347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:19.442347+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:31.033058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.984693+0000","last_change":"2024-03-11T21:24:55.486190+0000","last_active":"2024-03-11T21:25:47.984693+0000","last_peered":"2024-03-11T21:25:47.984693+0000","last_clean":"2024-03-11T21:25:47.984693+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:25:47.984693+0000","last_undegraded":"2024-03-11T21:25:47.984693+0000","last_fullsized":"2024-03-11T21:25:47.984693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:55.485972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:55.485972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:02.474929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890371+0000","last_change":"2024-03-11T21:24:33.494373+0000","last_active":"2024-03-11T21:25:47.890371+0000","last_peered":"2024-03-11T21:25:47.890371+0000","last_clean":"2024-03-11T21:25:47.890371+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:25:47.890371+0000","last_undegraded":"2024-03-11T21:25:47.890371+0000","last_fullsized":"2024-03-11T21:25:47.890371+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:33.494087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:33.494087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:56.778630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.050804+0000","last_change":"2024-03-11T21:25:48.050860+0000","last_active":"2024-03-11T21:25:48.050804+0000","last_peered":"2024-03-11T21:25:48.050804+0000","last_clean":"2024-03-11T21:25:48.050804+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:25:48.050804+0000","last_undegraded":"2024-03-11T21:25:48.050804+0000","last_fullsized":"2024-03-11T21:25:48.050804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.050648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.050648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.250408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":125,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:51.434722+0000","last_change":"2024-03-11T21:25:51.434868+0000","last_active":"2024-03-11T21:25:51.434722+0000","last_peered":"2024-03-11T21:25:51.434722+0000","last_clean":"2024-03-11T21:25:51.434722+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:25:51.434722+0000","last_undegraded":"2024-03-11T21:25:51.434722+0000","last_fullsized":"2024-03-11T21:25:51.434722+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:51.434497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:51.434497+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.747829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.991811+0000","last_change":"2024-03-11T21:24:44.059553+0000","last_active":"2024-03-11T21:25:47.991811+0000","last_peered":"2024-03-11T21:25:47.991811+0000","last_clean":"2024-03-11T21:25:47.991811+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:25:47.991811+0000","last_undegraded":"2024-03-11T21:25:47.991811+0000","last_fullsized":"2024-03-11T21:25:47.991811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:44.059198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:44.059198+0000","last_clean_scrub_stamp":"2024-03-11T21:24:44.059198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:04.426641+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.336327+0000","last_change":"2024-03-11T21:24:44.377845+0000","last_active":"2024-03-11T21:25:47.336327+0000","last_peered":"2024-03-11T21:25:47.336327+0000","last_clean":"2024-03-11T21:25:47.336327+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:25:47.336327+0000","last_undegraded":"2024-03-11T21:25:47.336327+0000","last_fullsized":"2024-03-11T21:25:47.336327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:44.377648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:44.377648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:04.645892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990401+0000","last_change":"2024-03-11T21:25:18.887001+0000","last_active":"2024-03-11T21:25:47.990401+0000","last_peered":"2024-03-11T21:25:47.990401+0000","last_clean":"2024-03-11T21:25:47.990401+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:25:47.990401+0000","last_undegraded":"2024-03-11T21:25:47.990401+0000","last_fullsized":"2024-03-11T21:25:47.990401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:18.886911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:18.886911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:36.216660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.336061+0000","last_change":"2024-03-11T21:25:43.877458+0000","last_active":"2024-03-11T21:25:47.336061+0000","last_peered":"2024-03-11T21:25:47.336061+0000","last_clean":"2024-03-11T21:25:47.336061+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:25:47.336061+0000","last_undegraded":"2024-03-11T21:25:47.336061+0000","last_fullsized":"2024-03-11T21:25:47.336061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:43.877225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.877225+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.780065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:59.988697+0000","last_change":"2024-03-11T21:25:59.988760+0000","last_active":"2024-03-11T21:25:59.988697+0000","last_peered":"2024-03-11T21:25:59.988697+0000","last_clean":"2024-03-11T21:25:59.988697+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:25:59.988697+0000","last_undegraded":"2024-03-11T21:25:59.988697+0000","last_fullsized":"2024-03-11T21:25:59.988697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.988522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.034265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:00.765469+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:26:00.765469+0000","last_peered":"2024-03-11T21:26:00.765469+0000","last_clean":"2024-03-11T21:26:00.765469+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:26:00.765469+0000","last_undegraded":"2024-03-11T21:26:00.765469+0000","last_fullsized":"2024-03-11T21:26:00.765469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.655474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.894844+0000","last_change":"2024-03-11T21:24:46.552880+0000","last_active":"2024-03-11T21:25:47.894844+0000","last_peered":"2024-03-11T21:25:47.894844+0000","last_clean":"2024-03-11T21:25:47.894844+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:25:47.894844+0000","last_undegraded":"2024-03-11T21:25:47.894844+0000","last_fullsized":"2024-03-11T21:25:47.894844+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:24:46.552643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:52.011626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.986226+0000","last_change":"2024-03-11T21:24:40.449514+0000","last_active":"2024-03-11T21:25:47.986226+0000","last_peered":"2024-03-11T21:25:47.986226+0000","last_clean":"2024-03-11T21:25:47.986226+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:25:47.986226+0000","last_undegraded":"2024-03-11T21:25:47.986226+0000","last_fullsized":"2024-03-11T21:25:47.986226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T21:24:40.449263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:40.449263+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:09.501009+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.343867+0000","last_change":"2024-03-11T21:24:44.322363+0000","last_active":"2024-03-11T21:25:47.343867+0000","last_peered":"2024-03-11T21:25:47.343867+0000","last_clean":"2024-03-11T21:25:47.343867+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:25:47.343867+0000","last_undegraded":"2024-03-11T21:25:47.343867+0000","last_fullsized":"2024-03-11T21:25:47.343867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:44.321991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:44.321991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:54.186664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.987230+0000","last_change":"2024-03-11T21:25:40.280094+0000","last_active":"2024-03-11T21:25:47.987230+0000","last_peered":"2024-03-11T21:25:47.987230+0000","last_clean":"2024-03-11T21:25:47.987230+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:25:47.987230+0000","last_undegraded":"2024-03-11T21:25:47.987230+0000","last_fullsized":"2024-03-11T21:25:47.987230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.279715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.279715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.096032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.989972+0000","last_change":"2024-03-11T21:25:27.026231+0000","last_active":"2024-03-11T21:25:47.989972+0000","last_peered":"2024-03-11T21:25:47.989972+0000","last_clean":"2024-03-11T21:25:47.989972+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:25:47.989972+0000","last_undegraded":"2024-03-11T21:25:47.989972+0000","last_fullsized":"2024-03-11T21:25:47.989972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:27.025913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.025913+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:37.624911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990212+0000","last_change":"2024-03-11T21:25:30.949367+0000","last_active":"2024-03-11T21:25:47.990212+0000","last_peered":"2024-03-11T21:25:47.990212+0000","last_clean":"2024-03-11T21:25:47.990212+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:25:47.990212+0000","last_undegraded":"2024-03-11T21:25:47.990212+0000","last_fullsized":"2024-03-11T21:25:47.990212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:30.949072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.949072+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:33.375300+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048885+0000","last_change":"2024-03-11T21:25:16.227364+0000","last_active":"2024-03-11T21:25:48.048885+0000","last_peered":"2024-03-11T21:25:48.048885+0000","last_clean":"2024-03-11T21:25:48.048885+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:25:48.048885+0000","last_undegraded":"2024-03-11T21:25:48.048885+0000","last_fullsized":"2024-03-11T21:25:48.048885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:25:16.227113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:16.227113+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:28.899079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":139,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.124634+0000","last_change":"2024-03-11T21:24:49.493264+0000","last_active":"2024-03-11T21:25:53.124634+0000","last_peered":"2024-03-11T21:25:53.124634+0000","last_clean":"2024-03-11T21:25:53.124634+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:25:53.124634+0000","last_undegraded":"2024-03-11T21:25:53.124634+0000","last_fullsized":"2024-03-11T21:25:53.124634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:24:49.493039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:49.493039+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:02.486040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":112,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:02.470946+0000","last_change":"2024-03-11T21:26:02.471008+0000","last_active":"2024-03-11T21:26:02.470946+0000","last_peered":"2024-03-11T21:26:02.470946+0000","last_clean":"2024-03-11T21:26:02.470946+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:26:02.470946+0000","last_undegraded":"2024-03-11T21:26:02.470946+0000","last_fullsized":"2024-03-11T21:26:02.470946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.470792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.470792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.529057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.342519+0000","last_change":"2024-03-11T21:24:25.462572+0000","last_active":"2024-03-11T21:25:47.342519+0000","last_peered":"2024-03-11T21:25:47.342519+0000","last_clean":"2024-03-11T21:25:47.342519+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:25:47.342519+0000","last_undegraded":"2024-03-11T21:25:47.342519+0000","last_fullsized":"2024-03-11T21:25:47.342519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:24:25.462333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:25.462333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:48.623001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990718+0000","last_change":"2024-03-11T21:24:43.025094+0000","last_active":"2024-03-11T21:25:47.990718+0000","last_peered":"2024-03-11T21:25:47.990718+0000","last_clean":"2024-03-11T21:25:47.990718+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:25:47.990718+0000","last_undegraded":"2024-03-11T21:25:47.990718+0000","last_fullsized":"2024-03-11T21:25:47.990718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:43.024865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:43.024865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:09.711568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047771+0000","last_change":"2024-03-11T21:25:03.181247+0000","last_active":"2024-03-11T21:25:48.047771+0000","last_peered":"2024-03-11T21:25:48.047771+0000","last_clean":"2024-03-11T21:25:48.047771+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:25:48.047771+0000","last_undegraded":"2024-03-11T21:25:48.047771+0000","last_fullsized":"2024-03-11T21:25:48.047771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:03.181042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:03.181042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:11.194423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.895546+0000","last_change":"2024-03-11T21:25:40.946608+0000","last_active":"2024-03-11T21:25:47.895546+0000","last_peered":"2024-03-11T21:25:47.895546+0000","last_clean":"2024-03-11T21:25:47.895546+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:25:47.895546+0000","last_undegraded":"2024-03-11T21:25:47.895546+0000","last_fullsized":"2024-03-11T21:25:47.895546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.946307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890624+0000","last_change":"2024-03-11T21:25:27.506732+0000","last_active":"2024-03-11T21:25:47.890624+0000","last_peered":"2024-03-11T21:25:47.890624+0000","last_clean":"2024-03-11T21:25:47.890624+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:25:47.890624+0000","last_undegraded":"2024-03-11T21:25:47.890624+0000","last_fullsized":"2024-03-11T21:25:47.890624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:27.506461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.506461+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.042628+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.051576+0000","last_change":"2024-03-11T21:25:53.051632+0000","last_active":"2024-03-11T21:25:53.051576+0000","last_peered":"2024-03-11T21:25:53.051576+0000","last_clean":"2024-03-11T21:25:53.051576+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:25:53.051576+0000","last_undegraded":"2024-03-11T21:25:53.051576+0000","last_fullsized":"2024-03-11T21:25:53.051576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.051422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.051422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.051498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.894491+0000","last_change":"2024-03-11T21:25:27.905116+0000","last_active":"2024-03-11T21:25:47.894491+0000","last_peered":"2024-03-11T21:25:47.894491+0000","last_clean":"2024-03-11T21:25:47.894491+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:25:47.894491+0000","last_undegraded":"2024-03-11T21:25:47.894491+0000","last_fullsized":"2024-03-11T21:25:47.894491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.904786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.904786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:28.273395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.374314+0000","last_change":"2024-03-11T21:25:48.374372+0000","last_active":"2024-03-11T21:25:48.374314+0000","last_peered":"2024-03-11T21:25:48.374314+0000","last_clean":"2024-03-11T21:25:48.374314+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:25:48.374314+0000","last_undegraded":"2024-03-11T21:25:48.374314+0000","last_fullsized":"2024-03-11T21:25:48.374314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.374158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.374158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.742286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.988039+0000","last_change":"2024-03-11T21:25:37.334081+0000","last_active":"2024-03-11T21:25:47.988039+0000","last_peered":"2024-03-11T21:25:47.988039+0000","last_clean":"2024-03-11T21:25:47.988039+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:25:47.988039+0000","last_undegraded":"2024-03-11T21:25:47.988039+0000","last_fullsized":"2024-03-11T21:25:47.988039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:25:37.333856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.333856+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.493865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:56.141735+0000","last_change":"2024-03-11T21:25:56.141805+0000","last_active":"2024-03-11T21:25:56.141735+0000","last_peered":"2024-03-11T21:25:56.141735+0000","last_clean":"2024-03-11T21:25:56.141735+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:25:56.141735+0000","last_undegraded":"2024-03-11T21:25:56.141735+0000","last_fullsized":"2024-03-11T21:25:56.141735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.141559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.141559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.340511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.345470+0000","last_change":"2024-03-11T21:24:43.364595+0000","last_active":"2024-03-11T21:25:47.345470+0000","last_peered":"2024-03-11T21:25:47.345470+0000","last_clean":"2024-03-11T21:25:47.345470+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:25:47.345470+0000","last_undegraded":"2024-03-11T21:25:47.345470+0000","last_fullsized":"2024-03-11T21:25:47.345470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:43.364477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:43.364477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:10.386272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.706373+0000","last_change":"2024-03-11T21:25:53.706422+0000","last_active":"2024-03-11T21:25:53.706373+0000","last_peered":"2024-03-11T21:25:53.706373+0000","last_clean":"2024-03-11T21:25:53.706373+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:25:53.706373+0000","last_undegraded":"2024-03-11T21:25:53.706373+0000","last_fullsized":"2024-03-11T21:25:53.706373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.706230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.384575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.336700+0000","last_change":"2024-03-11T21:25:38.984101+0000","last_active":"2024-03-11T21:25:47.336700+0000","last_peered":"2024-03-11T21:25:47.336700+0000","last_clean":"2024-03-11T21:25:47.336700+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:25:47.336700+0000","last_undegraded":"2024-03-11T21:25:47.336700+0000","last_fullsized":"2024-03-11T21:25:47.336700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:38.983858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.983858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.131782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:56.344194+0000","last_change":"2024-03-11T21:25:56.344253+0000","last_active":"2024-03-11T21:25:56.344194+0000","last_peered":"2024-03-11T21:25:56.344194+0000","last_clean":"2024-03-11T21:25:56.344194+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:25:56.344194+0000","last_undegraded":"2024-03-11T21:25:56.344194+0000","last_fullsized":"2024-03-11T21:25:56.344194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.343979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.343979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.628275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990652+0000","last_change":"2024-03-11T21:25:39.974174+0000","last_active":"2024-03-11T21:25:47.990652+0000","last_peered":"2024-03-11T21:25:47.990652+0000","last_clean":"2024-03-11T21:25:47.990652+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:25:47.990652+0000","last_undegraded":"2024-03-11T21:25:47.990652+0000","last_fullsized":"2024-03-11T21:25:47.990652+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.973978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.973978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.092042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.894688+0000","last_change":"2024-03-11T21:25:45.890051+0000","last_active":"2024-03-11T21:25:47.894688+0000","last_peered":"2024-03-11T21:25:47.894688+0000","last_clean":"2024-03-11T21:25:47.894688+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:25:47.894688+0000","last_undegraded":"2024-03-11T21:25:47.894688+0000","last_fullsized":"2024-03-11T21:25:47.894688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:45.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.889834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.143004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.049045+0000","last_change":"2024-03-11T21:25:37.960424+0000","last_active":"2024-03-11T21:25:48.049045+0000","last_peered":"2024-03-11T21:25:48.049045+0000","last_clean":"2024-03-11T21:25:48.049045+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:25:48.049045+0000","last_undegraded":"2024-03-11T21:25:48.049045+0000","last_fullsized":"2024-03-11T21:25:48.049045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:25:37.960057+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.960057+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.161700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.338015+0000","last_change":"2024-03-11T21:25:40.949182+0000","last_active":"2024-03-11T21:25:47.338015+0000","last_peered":"2024-03-11T21:25:47.338015+0000","last_clean":"2024-03-11T21:25:47.338015+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:25:47.338015+0000","last_undegraded":"2024-03-11T21:25:47.338015+0000","last_fullsized":"2024-03-11T21:25:47.338015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.948963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.948963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.061308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'11","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.129602+0000","last_change":"2024-03-11T21:25:38.973030+0000","last_active":"2024-03-11T21:25:53.129602+0000","last_peered":"2024-03-11T21:25:53.129602+0000","last_clean":"2024-03-11T21:25:53.129602+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:25:53.129602+0000","last_undegraded":"2024-03-11T21:25:53.129602+0000","last_fullsized":"2024-03-11T21:25:53.129602+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:38.972802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.972802+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.415779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:54.342789+0000","last_change":"2024-03-11T21:25:54.342862+0000","last_active":"2024-03-11T21:25:54.342789+0000","last_peered":"2024-03-11T21:25:54.342789+0000","last_clean":"2024-03-11T21:25:54.342789+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:25:54.342789+0000","last_undegraded":"2024-03-11T21:25:54.342789+0000","last_fullsized":"2024-03-11T21:25:54.342789+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.342588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.028053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:01.314702+0000","last_change":"2024-03-11T21:26:01.314774+0000","last_active":"2024-03-11T21:26:01.314702+0000","last_peered":"2024-03-11T21:26:01.314702+0000","last_clean":"2024-03-11T21:26:01.314702+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:26:01.314702+0000","last_undegraded":"2024-03-11T21:26:01.314702+0000","last_fullsized":"2024-03-11T21:26:01.314702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:26:01.314615+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.665529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.987226+0000","last_change":"2024-03-11T21:25:43.246214+0000","last_active":"2024-03-11T21:25:47.987226+0000","last_peered":"2024-03-11T21:25:47.987226+0000","last_clean":"2024-03-11T21:25:47.987226+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:25:47.987226+0000","last_undegraded":"2024-03-11T21:25:47.987226+0000","last_fullsized":"2024-03-11T21:25:47.987226+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.246002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.246002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.409171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:59.442179+0000","last_change":"2024-03-11T21:25:59.442248+0000","last_active":"2024-03-11T21:25:59.442179+0000","last_peered":"2024-03-11T21:25:59.442179+0000","last_clean":"2024-03-11T21:25:59.442179+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:25:59.442179+0000","last_undegraded":"2024-03-11T21:25:59.442179+0000","last_fullsized":"2024-03-11T21:25:59.442179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:59.442015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.442015+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.052362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.889924+0000","last_change":"2024-03-11T21:25:17.426143+0000","last_active":"2024-03-11T21:25:47.889924+0000","last_peered":"2024-03-11T21:25:47.889924+0000","last_clean":"2024-03-11T21:25:47.889924+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:25:47.889924+0000","last_undegraded":"2024-03-11T21:25:47.889924+0000","last_fullsized":"2024-03-11T21:25:47.889924+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:17.425944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:17.425944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:23.643326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048190+0000","last_change":"2024-03-11T21:25:39.009861+0000","last_active":"2024-03-11T21:25:48.048190+0000","last_peered":"2024-03-11T21:25:48.048190+0000","last_clean":"2024-03-11T21:25:48.048190+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:25:48.048190+0000","last_undegraded":"2024-03-11T21:25:48.048190+0000","last_fullsized":"2024-03-11T21:25:48.048190+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:39.009635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.009635+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:46.254494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:49.003214+0000","last_change":"2024-03-11T21:25:49.003277+0000","last_active":"2024-03-11T21:25:49.003214+0000","last_peered":"2024-03-11T21:25:49.003214+0000","last_clean":"2024-03-11T21:25:49.003214+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:25:49.003214+0000","last_undegraded":"2024-03-11T21:25:49.003214+0000","last_fullsized":"2024-03-11T21:25:49.003214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:49.003014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.003014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.285739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:54.360857+0000","last_change":"2024-03-11T21:25:54.360923+0000","last_active":"2024-03-11T21:25:54.360857+0000","last_peered":"2024-03-11T21:25:54.360857+0000","last_clean":"2024-03-11T21:25:54.360857+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:25:54.360857+0000","last_undegraded":"2024-03-11T21:25:54.360857+0000","last_fullsized":"2024-03-11T21:25:54.360857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.360647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.360647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.912837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.894345+0000","last_change":"2024-03-11T21:25:36.945869+0000","last_active":"2024-03-11T21:25:47.894345+0000","last_peered":"2024-03-11T21:25:47.894345+0000","last_clean":"2024-03-11T21:25:47.894345+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:25:47.894345+0000","last_undegraded":"2024-03-11T21:25:47.894345+0000","last_fullsized":"2024-03-11T21:25:47.894345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:36.945583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:36.945583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.141176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.894479+0000","last_change":"2024-03-11T21:25:46.921649+0000","last_active":"2024-03-11T21:25:47.894479+0000","last_peered":"2024-03-11T21:25:47.894479+0000","last_clean":"2024-03-11T21:25:47.894479+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:25:47.894479+0000","last_undegraded":"2024-03-11T21:25:47.894479+0000","last_fullsized":"2024-03-11T21:25:47.894479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.921404+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.755306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:50.974094+0000","last_change":"2024-03-11T21:25:50.974118+0000","last_active":"2024-03-11T21:25:50.974094+0000","last_peered":"2024-03-11T21:25:50.974094+0000","last_clean":"2024-03-11T21:25:50.974094+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:25:50.974094+0000","last_undegraded":"2024-03-11T21:25:50.974094+0000","last_fullsized":"2024-03-11T21:25:50.974094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:50.974032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.974032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.283434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.991898+0000","last_change":"2024-03-11T21:24:52.007473+0000","last_active":"2024-03-11T21:25:47.991898+0000","last_peered":"2024-03-11T21:25:47.991898+0000","last_clean":"2024-03-11T21:25:47.991898+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:25:47.991898+0000","last_undegraded":"2024-03-11T21:25:47.991898+0000","last_fullsized":"2024-03-11T21:25:47.991898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:52.007351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:52.007351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:56.600550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.339771+0000","last_change":"2024-03-11T21:25:29.104018+0000","last_active":"2024-03-11T21:25:47.339771+0000","last_peered":"2024-03-11T21:25:47.339771+0000","last_clean":"2024-03-11T21:25:47.339771+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:25:47.339771+0000","last_undegraded":"2024-03-11T21:25:47.339771+0000","last_fullsized":"2024-03-11T21:25:47.339771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:29.103775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.103775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:44.777028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":113,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047041+0000","last_change":"2024-03-11T21:24:51.114429+0000","last_active":"2024-03-11T21:25:48.047041+0000","last_peered":"2024-03-11T21:25:48.047041+0000","last_clean":"2024-03-11T21:25:48.047041+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:25:48.047041+0000","last_undegraded":"2024-03-11T21:25:48.047041+0000","last_fullsized":"2024-03-11T21:25:48.047041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:24:51.114174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:51.114174+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:54.149101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990632+0000","last_change":"2024-03-11T21:25:27.983629+0000","last_active":"2024-03-11T21:25:47.990632+0000","last_peered":"2024-03-11T21:25:47.990632+0000","last_clean":"2024-03-11T21:25:47.990632+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:25:47.990632+0000","last_undegraded":"2024-03-11T21:25:47.990632+0000","last_fullsized":"2024-03-11T21:25:47.990632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.983536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.983536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.044284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.337455+0000","last_change":"2024-03-11T21:25:37.981238+0000","last_active":"2024-03-11T21:25:47.337455+0000","last_peered":"2024-03-11T21:25:47.337455+0000","last_clean":"2024-03-11T21:25:47.337455+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:25:47.337455+0000","last_undegraded":"2024-03-11T21:25:47.337455+0000","last_fullsized":"2024-03-11T21:25:47.337455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:25:37.981019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.981019+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.806629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048239+0000","last_change":"2024-03-11T21:24:33.079892+0000","last_active":"2024-03-11T21:25:48.048239+0000","last_peered":"2024-03-11T21:25:48.048239+0000","last_clean":"2024-03-11T21:25:48.048239+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:25:48.048239+0000","last_undegraded":"2024-03-11T21:25:48.048239+0000","last_fullsized":"2024-03-11T21:25:48.048239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:24:33.079623+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:24:33.079623+0000","last_clean_scrub_stamp":"2024-03-11T21:24:33.079623+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:52.673115+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:02.944229+0000","last_change":"2024-03-11T21:26:02.944301+0000","last_active":"2024-03-11T21:26:02.944229+0000","last_peered":"2024-03-11T21:26:02.944229+0000","last_clean":"2024-03-11T21:26:02.944229+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:26:02.944229+0000","last_undegraded":"2024-03-11T21:26:02.944229+0000","last_fullsized":"2024-03-11T21:26:02.944229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:21.682420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.263645+0000","last_change":"2024-03-11T21:25:48.263693+0000","last_active":"2024-03-11T21:25:48.263645+0000","last_peered":"2024-03-11T21:25:48.263645+0000","last_clean":"2024-03-11T21:25:48.263645+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:25:48.263645+0000","last_undegraded":"2024-03-11T21:25:48.263645+0000","last_fullsized":"2024-03-11T21:25:48.263645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.263515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.384625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.344154+0000","last_change":"2024-03-11T21:25:33.757745+0000","last_active":"2024-03-11T21:25:47.344154+0000","last_peered":"2024-03-11T21:25:47.344154+0000","last_clean":"2024-03-11T21:25:47.344154+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:25:47.344154+0000","last_undegraded":"2024-03-11T21:25:47.344154+0000","last_fullsized":"2024-03-11T21:25:47.344154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:33.757442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:33.757442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.904053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":160,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.124368+0000","last_change":"2024-03-11T21:25:21.393014+0000","last_active":"2024-03-11T21:25:53.124368+0000","last_peered":"2024-03-11T21:25:53.124368+0000","last_clean":"2024-03-11T21:25:53.124368+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:25:53.124368+0000","last_undegraded":"2024-03-11T21:25:53.124368+0000","last_fullsized":"2024-03-11T21:25:53.124368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:25:21.392637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:21.392637+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:43.705727+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:50.631606+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:25:50.631606+0000","last_peered":"2024-03-11T21:25:50.631606+0000","last_clean":"2024-03-11T21:25:50.631606+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:25:50.631606+0000","last_undegraded":"2024-03-11T21:25:50.631606+0000","last_fullsized":"2024-03-11T21:25:50.631606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.556814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"84'11","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:58.332980+0000","last_change":"2024-03-11T21:25:58.333052+0000","last_active":"2024-03-11T21:25:58.332980+0000","last_peered":"2024-03-11T21:25:58.332980+0000","last_clean":"2024-03-11T21:25:58.332980+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:25:58.332980+0000","last_undegraded":"2024-03-11T21:25:58.332980+0000","last_fullsized":"2024-03-11T21:25:58.332980+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:58.332816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.332816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.183920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.991693+0000","last_change":"2024-03-11T21:24:47.092913+0000","last_active":"2024-03-11T21:25:47.991693+0000","last_peered":"2024-03-11T21:25:47.991693+0000","last_clean":"2024-03-11T21:25:47.991693+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:25:47.991693+0000","last_undegraded":"2024-03-11T21:25:47.991693+0000","last_fullsized":"2024-03-11T21:25:47.991693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:47.092662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:47.092662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:16.057958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890084+0000","last_change":"2024-03-11T21:24:39.492065+0000","last_active":"2024-03-11T21:25:47.890084+0000","last_peered":"2024-03-11T21:25:47.890084+0000","last_clean":"2024-03-11T21:25:47.890084+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:25:47.890084+0000","last_undegraded":"2024-03-11T21:25:47.890084+0000","last_fullsized":"2024-03-11T21:25:47.890084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:39.491841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:24:39.491841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:06.714919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.336226+0000","last_change":"2024-03-11T21:25:30.974808+0000","last_active":"2024-03-11T21:25:47.336226+0000","last_peered":"2024-03-11T21:25:47.336226+0000","last_clean":"2024-03-11T21:25:47.336226+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:25:47.336226+0000","last_undegraded":"2024-03-11T21:25:47.336226+0000","last_fullsized":"2024-03-11T21:25:47.336226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.974544+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.198159+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.987581+0000","last_change":"2024-03-11T21:24:46.441508+0000","last_active":"2024-03-11T21:25:47.987581+0000","last_peered":"2024-03-11T21:25:47.987581+0000","last_clean":"2024-03-11T21:25:47.987581+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:25:47.987581+0000","last_undegraded":"2024-03-11T21:25:47.987581+0000","last_fullsized":"2024-03-11T21:25:47.987581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:46.441282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:46.441282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:16.384824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047906+0000","last_change":"2024-03-11T21:24:39.114177+0000","last_active":"2024-03-11T21:25:48.047906+0000","last_peered":"2024-03-11T21:25:48.047906+0000","last_clean":"2024-03-11T21:25:48.047906+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:25:48.047906+0000","last_undegraded":"2024-03-11T21:25:48.047906+0000","last_fullsized":"2024-03-11T21:25:48.047906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:39.113951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:39.113951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:51.928905+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":134,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:56.736868+0000","last_change":"2024-03-11T21:25:56.736951+0000","last_active":"2024-03-11T21:25:56.736868+0000","last_peered":"2024-03-11T21:25:56.736868+0000","last_clean":"2024-03-11T21:25:56.736868+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:25:56.736868+0000","last_undegraded":"2024-03-11T21:25:56.736868+0000","last_fullsized":"2024-03-11T21:25:56.736868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:25:56.736721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.736721+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.841976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.994240+0000","last_change":"2024-03-11T21:25:47.994296+0000","last_active":"2024-03-11T21:25:47.994240+0000","last_peered":"2024-03-11T21:25:47.994240+0000","last_clean":"2024-03-11T21:25:47.994240+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:25:47.994240+0000","last_undegraded":"2024-03-11T21:25:47.994240+0000","last_fullsized":"2024-03-11T21:25:47.994240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.994008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.994008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.859169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:58.022888+0000","last_change":"2024-03-11T21:25:58.022948+0000","last_active":"2024-03-11T21:25:58.022888+0000","last_peered":"2024-03-11T21:25:58.022888+0000","last_clean":"2024-03-11T21:25:58.022888+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:25:58.022888+0000","last_undegraded":"2024-03-11T21:25:58.022888+0000","last_fullsized":"2024-03-11T21:25:58.022888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.022737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.022737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.306838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.892006+0000","last_change":"2024-03-11T21:24:44.527532+0000","last_active":"2024-03-11T21:25:47.892006+0000","last_peered":"2024-03-11T21:25:47.892006+0000","last_clean":"2024-03-11T21:25:47.892006+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:25:47.892006+0000","last_undegraded":"2024-03-11T21:25:47.892006+0000","last_fullsized":"2024-03-11T21:25:47.892006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:44.527238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:44.527238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:48.813132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990270+0000","last_change":"2024-03-11T21:24:30.105246+0000","last_active":"2024-03-11T21:25:47.990270+0000","last_peered":"2024-03-11T21:25:47.990270+0000","last_clean":"2024-03-11T21:25:47.990270+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:25:47.990270+0000","last_undegraded":"2024-03-11T21:25:47.990270+0000","last_fullsized":"2024-03-11T21:25:47.990270+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:24:30.104996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:30.104996+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:56.746023+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890202+0000","last_change":"2024-03-11T21:25:32.395539+0000","last_active":"2024-03-11T21:25:47.890202+0000","last_peered":"2024-03-11T21:25:47.890202+0000","last_clean":"2024-03-11T21:25:47.890202+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:25:47.890202+0000","last_undegraded":"2024-03-11T21:25:47.890202+0000","last_fullsized":"2024-03-11T21:25:47.890202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.395316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.615969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":359,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047072+0000","last_change":"2024-03-11T21:25:21.288916+0000","last_active":"2024-03-11T21:25:48.047072+0000","last_peered":"2024-03-11T21:25:48.047072+0000","last_clean":"2024-03-11T21:25:48.047072+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:25:48.047072+0000","last_undegraded":"2024-03-11T21:25:48.047072+0000","last_fullsized":"2024-03-11T21:25:48.047072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:25:21.288622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:25:21.288622+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:33.339389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048563+0000","last_change":"2024-03-11T21:25:02.193912+0000","last_active":"2024-03-11T21:25:48.048563+0000","last_peered":"2024-03-11T21:25:48.048563+0000","last_clean":"2024-03-11T21:25:48.048563+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:25:48.048563+0000","last_undegraded":"2024-03-11T21:25:48.048563+0000","last_fullsized":"2024-03-11T21:25:48.048563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:02.193681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:02.193681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:31.088279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.342996+0000","last_change":"2024-03-11T21:24:45.311069+0000","last_active":"2024-03-11T21:25:47.342996+0000","last_peered":"2024-03-11T21:25:47.342996+0000","last_clean":"2024-03-11T21:25:47.342996+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:25:47.342996+0000","last_undegraded":"2024-03-11T21:25:47.342996+0000","last_fullsized":"2024-03-11T21:25:47.342996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:45.310954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:45.310954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:58.434964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":115,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.984860+0000","last_change":"2024-03-11T21:24:47.440572+0000","last_active":"2024-03-11T21:25:47.984860+0000","last_peered":"2024-03-11T21:25:47.984860+0000","last_clean":"2024-03-11T21:25:47.984860+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:25:47.984860+0000","last_undegraded":"2024-03-11T21:25:47.984860+0000","last_fullsized":"2024-03-11T21:25:47.984860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:24:47.440317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:47.440317+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:05.236011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.046931+0000","last_change":"2024-03-11T21:25:43.021158+0000","last_active":"2024-03-11T21:25:48.046931+0000","last_peered":"2024-03-11T21:25:48.046931+0000","last_clean":"2024-03-11T21:25:48.046931+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:25:48.046931+0000","last_undegraded":"2024-03-11T21:25:48.046931+0000","last_fullsized":"2024-03-11T21:25:48.046931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.020950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.020950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.379154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:52.415624+0000","last_change":"2024-03-11T21:25:52.415680+0000","last_active":"2024-03-11T21:25:52.415624+0000","last_peered":"2024-03-11T21:25:52.415624+0000","last_clean":"2024-03-11T21:25:52.415624+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:25:52.415624+0000","last_undegraded":"2024-03-11T21:25:52.415624+0000","last_fullsized":"2024-03-11T21:25:52.415624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.415468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.415468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.982658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.342017+0000","last_change":"2024-03-11T21:24:41.340246+0000","last_active":"2024-03-11T21:25:47.342017+0000","last_peered":"2024-03-11T21:25:47.342017+0000","last_clean":"2024-03-11T21:25:47.342017+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:25:47.342017+0000","last_undegraded":"2024-03-11T21:25:47.342017+0000","last_fullsized":"2024-03-11T21:25:47.342017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:41.339898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:41.339898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:51.001661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:59.200463+0000","last_change":"2024-03-11T21:25:59.200564+0000","last_active":"2024-03-11T21:25:59.200463+0000","last_peered":"2024-03-11T21:25:59.200463+0000","last_clean":"2024-03-11T21:25:59.200463+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:25:59.200463+0000","last_undegraded":"2024-03-11T21:25:59.200463+0000","last_fullsized":"2024-03-11T21:25:59.200463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.200270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.200270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.037981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":188,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.128827+0000","last_change":"2024-03-11T21:25:37.669639+0000","last_active":"2024-03-11T21:25:53.128827+0000","last_peered":"2024-03-11T21:25:53.128827+0000","last_clean":"2024-03-11T21:25:53.128827+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:25:53.128827+0000","last_undegraded":"2024-03-11T21:25:53.128827+0000","last_fullsized":"2024-03-11T21:25:53.128827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:25:37.669221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.669221+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.744954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.895132+0000","last_change":"2024-03-11T21:24:40.581179+0000","last_active":"2024-03-11T21:25:47.895132+0000","last_peered":"2024-03-11T21:25:47.895132+0000","last_clean":"2024-03-11T21:25:47.895132+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:25:47.895132+0000","last_undegraded":"2024-03-11T21:25:47.895132+0000","last_fullsized":"2024-03-11T21:25:47.895132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:40.580975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:40.580975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:04.622589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:02.037230+0000","last_change":"2024-03-11T21:26:02.037288+0000","last_active":"2024-03-11T21:26:02.037230+0000","last_peered":"2024-03-11T21:26:02.037230+0000","last_clean":"2024-03-11T21:26:02.037230+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:26:02.037230+0000","last_undegraded":"2024-03-11T21:26:02.037230+0000","last_fullsized":"2024-03-11T21:26:02.037230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.939102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:49.638247+0000","last_change":"2024-03-11T21:25:49.638326+0000","last_active":"2024-03-11T21:25:49.638247+0000","last_peered":"2024-03-11T21:25:49.638247+0000","last_clean":"2024-03-11T21:25:49.638247+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:25:49.638247+0000","last_undegraded":"2024-03-11T21:25:49.638247+0000","last_fullsized":"2024-03-11T21:25:49.638247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:49.638061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.638061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.499420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.992149+0000","last_change":"2024-03-11T21:24:36.087831+0000","last_active":"2024-03-11T21:25:47.992149+0000","last_peered":"2024-03-11T21:25:47.992149+0000","last_clean":"2024-03-11T21:25:47.992149+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:25:47.992149+0000","last_undegraded":"2024-03-11T21:25:47.992149+0000","last_fullsized":"2024-03-11T21:25:47.992149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:24:36.087707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:36.087707+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:05.693429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":271,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:02.009530+0000","last_change":"2024-03-11T21:25:50.006793+0000","last_active":"2024-03-11T21:26:02.009530+0000","last_peered":"2024-03-11T21:26:02.009530+0000","last_clean":"2024-03-11T21:26:02.009530+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:26:02.009530+0000","last_undegraded":"2024-03-11T21:26:02.009530+0000","last_fullsized":"2024-03-11T21:26:02.009530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:25:50.006698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.006698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.269418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:57.991997+0000","last_change":"2024-03-11T21:25:57.992113+0000","last_active":"2024-03-11T21:25:57.991997+0000","last_peered":"2024-03-11T21:25:57.991997+0000","last_clean":"2024-03-11T21:25:57.991997+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:25:57.991997+0000","last_undegraded":"2024-03-11T21:25:57.991997+0000","last_fullsized":"2024-03-11T21:25:57.991997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.991803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.991803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.426825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047664+0000","last_change":"2024-03-11T21:25:34.059053+0000","last_active":"2024-03-11T21:25:48.047664+0000","last_peered":"2024-03-11T21:25:48.047664+0000","last_clean":"2024-03-11T21:25:48.047664+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:25:48.047664+0000","last_undegraded":"2024-03-11T21:25:48.047664+0000","last_fullsized":"2024-03-11T21:25:48.047664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.058815+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.431730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"77'1","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.895354+0000","last_change":"2024-03-11T21:25:43.867607+0000","last_active":"2024-03-11T21:25:47.895354+0000","last_peered":"2024-03-11T21:25:47.895354+0000","last_clean":"2024-03-11T21:25:47.895354+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:25:47.895354+0000","last_undegraded":"2024-03-11T21:25:47.895354+0000","last_fullsized":"2024-03-11T21:25:47.895354+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:43.867377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.867377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.362412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":120,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.987818+0000","last_change":"2024-03-11T21:25:22.361282+0000","last_active":"2024-03-11T21:25:47.987818+0000","last_peered":"2024-03-11T21:25:47.987818+0000","last_clean":"2024-03-11T21:25:47.987818+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:25:47.987818+0000","last_undegraded":"2024-03-11T21:25:47.987818+0000","last_fullsized":"2024-03-11T21:25:47.987818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:22.360881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.360881+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:27.226640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.898475+0000","last_change":"2024-03-11T21:25:47.898533+0000","last_active":"2024-03-11T21:25:47.898475+0000","last_peered":"2024-03-11T21:25:47.898475+0000","last_clean":"2024-03-11T21:25:47.898475+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:25:47.898475+0000","last_undegraded":"2024-03-11T21:25:47.898475+0000","last_fullsized":"2024-03-11T21:25:47.898475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.898322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.898322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.986939+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.128822+0000","last_change":"2024-03-11T21:25:45.984353+0000","last_active":"2024-03-11T21:25:53.128822+0000","last_peered":"2024-03-11T21:25:53.128822+0000","last_clean":"2024-03-11T21:25:53.128822+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:25:53.128822+0000","last_undegraded":"2024-03-11T21:25:53.128822+0000","last_fullsized":"2024-03-11T21:25:53.128822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:45.984098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.984098+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.447890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990735+0000","last_change":"2024-03-11T21:24:38.145631+0000","last_active":"2024-03-11T21:25:47.990735+0000","last_peered":"2024-03-11T21:25:47.990735+0000","last_clean":"2024-03-11T21:25:47.990735+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:25:47.990735+0000","last_undegraded":"2024-03-11T21:25:47.990735+0000","last_fullsized":"2024-03-11T21:25:47.990735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:38.145410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:38.145410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:59.652560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:58.434873+0000","last_change":"2024-03-11T21:25:58.434898+0000","last_active":"2024-03-11T21:25:58.434873+0000","last_peered":"2024-03-11T21:25:58.434873+0000","last_clean":"2024-03-11T21:25:58.434873+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:25:58.434873+0000","last_undegraded":"2024-03-11T21:25:58.434873+0000","last_fullsized":"2024-03-11T21:25:58.434873+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.434794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.434794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:52.032760+0000","last_change":"2024-03-11T21:25:52.032818+0000","last_active":"2024-03-11T21:25:52.032760+0000","last_peered":"2024-03-11T21:25:52.032760+0000","last_clean":"2024-03-11T21:25:52.032760+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:25:52.032760+0000","last_undegraded":"2024-03-11T21:25:52.032760+0000","last_fullsized":"2024-03-11T21:25:52.032760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.032606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.032606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.435492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.335742+0000","last_change":"2024-03-11T21:24:49.278410+0000","last_active":"2024-03-11T21:25:47.335742+0000","last_peered":"2024-03-11T21:25:47.335742+0000","last_clean":"2024-03-11T21:25:47.335742+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:25:47.335742+0000","last_undegraded":"2024-03-11T21:25:47.335742+0000","last_fullsized":"2024-03-11T21:25:47.335742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:24:49.278162+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:13.051996+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890615+0000","last_change":"2024-03-11T21:25:41.358918+0000","last_active":"2024-03-11T21:25:47.890615+0000","last_peered":"2024-03-11T21:25:47.890615+0000","last_clean":"2024-03-11T21:25:47.890615+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:25:47.890615+0000","last_undegraded":"2024-03-11T21:25:47.890615+0000","last_fullsized":"2024-03-11T21:25:47.890615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.358679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.152690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:57.003681+0000","last_change":"2024-03-11T21:25:57.003681+0000","last_active":"2024-03-11T21:25:57.003681+0000","last_peered":"2024-03-11T21:25:57.003681+0000","last_clean":"2024-03-11T21:25:57.003681+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:25:57.003681+0000","last_undegraded":"2024-03-11T21:25:57.003681+0000","last_fullsized":"2024-03-11T21:25:57.003681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:57.003432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.003432+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.840351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.342282+0000","last_change":"2024-03-11T21:25:34.710182+0000","last_active":"2024-03-11T21:25:47.342282+0000","last_peered":"2024-03-11T21:25:47.342282+0000","last_clean":"2024-03-11T21:25:47.342282+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:25:47.342282+0000","last_undegraded":"2024-03-11T21:25:47.342282+0000","last_fullsized":"2024-03-11T21:25:47.342282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:34.709964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.709964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.742096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990050+0000","last_change":"2024-03-11T21:25:40.954211+0000","last_active":"2024-03-11T21:25:47.990050+0000","last_peered":"2024-03-11T21:25:47.990050+0000","last_clean":"2024-03-11T21:25:47.990050+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:25:47.990050+0000","last_undegraded":"2024-03-11T21:25:47.990050+0000","last_fullsized":"2024-03-11T21:25:47.990050+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:40.953969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.953969+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.832583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.337721+0000","last_change":"2024-03-11T21:24:46.341529+0000","last_active":"2024-03-11T21:25:47.337721+0000","last_peered":"2024-03-11T21:25:47.337721+0000","last_clean":"2024-03-11T21:25:47.337721+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:25:47.337721+0000","last_undegraded":"2024-03-11T21:25:47.337721+0000","last_fullsized":"2024-03-11T21:25:47.337721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:46.341251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:46.341251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:01.721114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":145,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.129262+0000","last_change":"2024-03-11T21:25:40.987597+0000","last_active":"2024-03-11T21:25:53.129262+0000","last_peered":"2024-03-11T21:25:53.129262+0000","last_clean":"2024-03-11T21:25:53.129262+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:25:53.129262+0000","last_undegraded":"2024-03-11T21:25:53.129262+0000","last_fullsized":"2024-03-11T21:25:53.129262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:25:40.987394+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.987394+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.827259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.344234+0000","last_change":"2024-03-11T21:24:23.473441+0000","last_active":"2024-03-11T21:25:47.344234+0000","last_peered":"2024-03-11T21:25:47.344234+0000","last_clean":"2024-03-11T21:25:47.344234+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:25:47.344234+0000","last_undegraded":"2024-03-11T21:25:47.344234+0000","last_fullsized":"2024-03-11T21:25:47.344234+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:24:23.473313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:51.592716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.990501+0000","last_change":"2024-03-11T21:25:29.024987+0000","last_active":"2024-03-11T21:25:47.990501+0000","last_peered":"2024-03-11T21:25:47.990501+0000","last_clean":"2024-03-11T21:25:47.990501+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:25:47.990501+0000","last_undegraded":"2024-03-11T21:25:47.990501+0000","last_fullsized":"2024-03-11T21:25:47.990501+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:29.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.024617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.368987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.341236+0000","last_change":"2024-03-11T21:25:32.119545+0000","last_active":"2024-03-11T21:25:47.341236+0000","last_peered":"2024-03-11T21:25:47.341236+0000","last_clean":"2024-03-11T21:25:47.341236+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:25:47.341236+0000","last_undegraded":"2024-03-11T21:25:47.341236+0000","last_fullsized":"2024-03-11T21:25:47.341236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.119316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.119316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.928370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.343289+0000","last_change":"2024-03-11T21:24:36.326754+0000","last_active":"2024-03-11T21:25:47.343289+0000","last_peered":"2024-03-11T21:25:47.343289+0000","last_clean":"2024-03-11T21:25:47.343289+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:25:47.343289+0000","last_undegraded":"2024-03-11T21:25:47.343289+0000","last_fullsized":"2024-03-11T21:25:47.343289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:36.326567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:36.326567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:03.815834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.047832+0000","last_change":"2024-03-11T21:25:28.167250+0000","last_active":"2024-03-11T21:25:48.047832+0000","last_peered":"2024-03-11T21:25:48.047832+0000","last_clean":"2024-03-11T21:25:48.047832+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:25:48.047832+0000","last_undegraded":"2024-03-11T21:25:48.047832+0000","last_fullsized":"2024-03-11T21:25:48.047832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:28.167038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:28.167038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:41.505025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":142,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:26:02.944979+0000","last_change":"2024-03-11T21:26:02.945068+0000","last_active":"2024-03-11T21:26:02.944979+0000","last_peered":"2024-03-11T21:26:02.944979+0000","last_clean":"2024-03-11T21:26:02.944979+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:26:02.944979+0000","last_undegraded":"2024-03-11T21:26:02.944979+0000","last_fullsized":"2024-03-11T21:26:02.944979+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:26:02.944783+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944783+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.686877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.046757+0000","last_change":"2024-03-11T21:24:43.140258+0000","last_active":"2024-03-11T21:25:48.046757+0000","last_peered":"2024-03-11T21:25:48.046757+0000","last_clean":"2024-03-11T21:25:48.046757+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:25:48.046757+0000","last_undegraded":"2024-03-11T21:25:48.046757+0000","last_fullsized":"2024-03-11T21:25:48.046757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:43.139909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:43.139909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:08.547429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.340902+0000","last_change":"2024-03-11T21:25:30.130283+0000","last_active":"2024-03-11T21:25:47.340902+0000","last_peered":"2024-03-11T21:25:47.340902+0000","last_clean":"2024-03-11T21:25:47.340902+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:25:47.340902+0000","last_undegraded":"2024-03-11T21:25:47.340902+0000","last_fullsized":"2024-03-11T21:25:47.340902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.130050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.236058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.340814+0000","last_change":"2024-03-11T21:25:44.013839+0000","last_active":"2024-03-11T21:25:47.340814+0000","last_peered":"2024-03-11T21:25:47.340814+0000","last_clean":"2024-03-11T21:25:47.340814+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:25:47.340814+0000","last_undegraded":"2024-03-11T21:25:47.340814+0000","last_fullsized":"2024-03-11T21:25:47.340814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:44.013746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:44.013746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.618233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.048767+0000","last_change":"2024-03-11T21:24:41.173004+0000","last_active":"2024-03-11T21:25:48.048767+0000","last_peered":"2024-03-11T21:25:48.048767+0000","last_clean":"2024-03-11T21:25:48.048767+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:25:48.048767+0000","last_undegraded":"2024-03-11T21:25:48.048767+0000","last_fullsized":"2024-03-11T21:25:48.048767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:24:41.172690+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:11.021661+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.984628+0000","last_change":"2024-03-11T21:24:53.521553+0000","last_active":"2024-03-11T21:25:47.984628+0000","last_peered":"2024-03-11T21:25:47.984628+0000","last_clean":"2024-03-11T21:25:47.984628+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:25:47.984628+0000","last_undegraded":"2024-03-11T21:25:47.984628+0000","last_fullsized":"2024-03-11T21:25:47.984628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:24:53.521326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:07.522501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.340441+0000","last_change":"2024-03-11T21:25:31.137161+0000","last_active":"2024-03-11T21:25:47.340441+0000","last_peered":"2024-03-11T21:25:47.340441+0000","last_clean":"2024-03-11T21:25:47.340441+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:25:47.340441+0000","last_undegraded":"2024-03-11T21:25:47.340441+0000","last_fullsized":"2024-03-11T21:25:47.340441+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:31.137035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.137035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.158341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.343458+0000","last_change":"2024-03-11T21:25:30.696848+0000","last_active":"2024-03-11T21:25:47.343458+0000","last_peered":"2024-03-11T21:25:47.343458+0000","last_clean":"2024-03-11T21:25:47.343458+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:25:47.343458+0000","last_undegraded":"2024-03-11T21:25:47.343458+0000","last_fullsized":"2024-03-11T21:25:47.343458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.696738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.696738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.966425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.890728+0000","last_change":"2024-03-11T21:25:45.376793+0000","last_active":"2024-03-11T21:25:47.890728+0000","last_peered":"2024-03-11T21:25:47.890728+0000","last_clean":"2024-03-11T21:25:47.890728+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:25:47.890728+0000","last_undegraded":"2024-03-11T21:25:47.890728+0000","last_fullsized":"2024-03-11T21:25:47.890728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:45.376408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.376408+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.150185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.889763+0000","last_change":"2024-03-11T21:24:40.507624+0000","last_active":"2024-03-11T21:25:47.889763+0000","last_peered":"2024-03-11T21:25:47.889763+0000","last_clean":"2024-03-11T21:25:47.889763+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:25:47.889763+0000","last_undegraded":"2024-03-11T21:25:47.889763+0000","last_fullsized":"2024-03-11T21:25:47.889763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:40.507401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:40.507401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:25:54.284529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:47.341497+0000","last_change":"2024-03-11T21:25:22.246229+0000","last_active":"2024-03-11T21:25:47.341497+0000","last_peered":"2024-03-11T21:25:47.341497+0000","last_clean":"2024-03-11T21:25:47.341497+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:25:47.341497+0000","last_undegraded":"2024-03-11T21:25:47.341497+0000","last_fullsized":"2024-03-11T21:25:47.341497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:22.245940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.245940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.310931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:48.049072+0000","last_change":"2024-03-11T21:25:18.230459+0000","last_active":"2024-03-11T21:25:48.049072+0000","last_peered":"2024-03-11T21:25:48.049072+0000","last_clean":"2024-03-11T21:25:48.049072+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:25:48.049072+0000","last_undegraded":"2024-03-11T21:25:48.049072+0000","last_fullsized":"2024-03-11T21:25:48.049072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:18.230036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:18.230036+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:30.431832+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":136,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T21:25:53.124896+0000","last_change":"2024-03-11T21:25:12.165879+0000","last_active":"2024-03-11T21:25:53.124896+0000","last_peered":"2024-03-11T21:25:53.124896+0000","last_clean":"2024-03-11T21:25:53.124896+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:25:53.124896+0000","last_undegraded":"2024-03-11T21:25:53.124896+0000","last_fullsized":"2024-03-11T21:25:53.124896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:12.165710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:12.165710+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:24.012554+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841583,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36744,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708408,"statfs":{"total":95995035648,"available":95957409792,"internally_reserved":0,"allocated":1441792,"data_stored":1012635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168632,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1409024,"data_stored":1010415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":419929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36724,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708428,"statfs":{"total":95995035648,"available":95957430272,"internally_reserved":0,"allocated":1421312,"data_stored":1011532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378722,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":819200,"data_stored":420507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":806912,"data_stored":419985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855436,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":421454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495682,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":419967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:26:04.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.427830+0000 osd.5 (osd.5) 81 : cluster 0 6.7 scrub starts 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.429933+0000 osd.5 (osd.5) 82 : cluster 0 6.7 scrub ok 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.725366+0000 osd.1 (osd.1) 71 : cluster 0 3.19 scrub starts 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:03.728761+0000 osd.1 (osd.1) 72 : cluster 0 3.19 scrub ok 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.125566+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.128019+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.175668+0000 osd.3 (osd.3) 81 : cluster 0 5.17 scrub starts 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.178135+0000 osd.3 (osd.3) 82 : cluster 0 5.17 scrub ok 2024-03-11T21:26:05.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.382812+0000 osd.0 (osd.0) 71 : cluster 0 6.18 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.385058+0000 osd.0 (osd.0) 72 : cluster 0 6.18 scrub ok 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.427830+0000 osd.5 (osd.5) 81 : cluster 0 6.7 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.429933+0000 osd.5 (osd.5) 82 : cluster 0 6.7 scrub ok 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.725366+0000 osd.1 (osd.1) 71 : cluster 0 3.19 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:03.728761+0000 osd.1 (osd.1) 72 : cluster 0 3.19 scrub ok 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.125566+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.128019+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.175668+0000 osd.3 (osd.3) 81 : cluster 0 5.17 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.178135+0000 osd.3 (osd.3) 82 : cluster 0 5.17 scrub ok 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.382812+0000 osd.0 (osd.0) 71 : cluster 0 6.18 scrub starts 2024-03-11T21:26:05.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:05 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.385058+0000 osd.0 (osd.0) 72 : cluster 0 6.18 scrub ok 2024-03-11T21:26:05.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.427830+0000 osd.5 (osd.5) 81 : cluster 0 6.7 scrub starts 2024-03-11T21:26:05.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.429933+0000 osd.5 (osd.5) 82 : cluster 0 6.7 scrub ok 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.725366+0000 osd.1 (osd.1) 71 : cluster 0 3.19 scrub starts 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:03.728761+0000 osd.1 (osd.1) 72 : cluster 0 3.19 scrub ok 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.125566+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.128019+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.175668+0000 osd.3 (osd.3) 81 : cluster 0 5.17 scrub starts 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.178135+0000 osd.3 (osd.3) 82 : cluster 0 5.17 scrub ok 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.382812+0000 osd.0 (osd.0) 71 : cluster 0 6.18 scrub starts 2024-03-11T21:26:05.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:05 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.385058+0000 osd.0 (osd.0) 72 : cluster 0 6.18 scrub ok 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.758104+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.760252+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.953834+0000 osd.6 (osd.6) 45 : cluster 0 2.1a scrub starts 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:04.956340+0000 osd.6 (osd.6) 46 : cluster 0 2.1a scrub ok 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.070990+0000 osd.4 (osd.4) 65 : cluster 0 6.e scrub starts 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.072663+0000 osd.4 (osd.4) 66 : cluster 0 6.e scrub ok 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.105013+0000 mgr.y (mgr.14550) 113 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.758104+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-03-11T21:26:06.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.760252+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-03-11T21:26:06.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.953834+0000 osd.6 (osd.6) 45 : cluster 0 2.1a scrub starts 2024-03-11T21:26:06.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:04.956340+0000 osd.6 (osd.6) 46 : cluster 0 2.1a scrub ok 2024-03-11T21:26:06.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.070990+0000 osd.4 (osd.4) 65 : cluster 0 6.e scrub starts 2024-03-11T21:26:06.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.072663+0000 osd.4 (osd.4) 66 : cluster 0 6.e scrub ok 2024-03-11T21:26:06.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:06 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.105013+0000 mgr.y (mgr.14550) 113 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.758104+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.760252+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.953834+0000 osd.6 (osd.6) 45 : cluster 0 2.1a scrub starts 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:04.956340+0000 osd.6 (osd.6) 46 : cluster 0 2.1a scrub ok 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.070990+0000 osd.4 (osd.4) 65 : cluster 0 6.e scrub starts 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.072663+0000 osd.4 (osd.4) 66 : cluster 0 6.e scrub ok 2024-03-11T21:26:06.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:06 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.105013+0000 mgr.y (mgr.14550) 113 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:07.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.388410+0000 osd.5 (osd.5) 83 : cluster 0 6.a scrub starts 2024-03-11T21:26:07.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.390839+0000 osd.5 (osd.5) 84 : cluster 0 6.a scrub ok 2024-03-11T21:26:07.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.975222+0000 osd.6 (osd.6) 47 : cluster 0 5.1f scrub starts 2024-03-11T21:26:07.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[21512]: cluster 2024-03-11T21:26:05.978307+0000 osd.6 (osd.6) 48 : cluster 0 5.1f scrub ok 2024-03-11T21:26:07.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.388410+0000 osd.5 (osd.5) 83 : cluster 0 6.a scrub starts 2024-03-11T21:26:07.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.390839+0000 osd.5 (osd.5) 84 : cluster 0 6.a scrub ok 2024-03-11T21:26:07.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.975222+0000 osd.6 (osd.6) 47 : cluster 0 5.1f scrub starts 2024-03-11T21:26:07.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[29284]: cluster 2024-03-11T21:26:05.978307+0000 osd.6 (osd.6) 48 : cluster 0 5.1f scrub ok 2024-03-11T21:26:07.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:07 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.388410+0000 osd.5 (osd.5) 83 : cluster 0 6.a scrub starts 2024-03-11T21:26:07.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:07 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.390839+0000 osd.5 (osd.5) 84 : cluster 0 6.a scrub ok 2024-03-11T21:26:07.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:07 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.975222+0000 osd.6 (osd.6) 47 : cluster 0 5.1f scrub starts 2024-03-11T21:26:07.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:07 smithi203 bash[23654]: cluster 2024-03-11T21:26:05.978307+0000 osd.6 (osd.6) 48 : cluster 0 5.1f scrub ok 2024-03-11T21:26:08.185 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:26:07 smithi143 bash[63388]: ts=2024-03-11T21:26:07.750Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003425576s 2024-03-11T21:26:08.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:08.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: cluster 2024-03-11T21:26:07.105675+0000 mgr.y (mgr.14550) 114 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:08.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.393734+0000 mon.a (mon.0) 870 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.408007+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: cluster 2024-03-11T21:26:07.647583+0000 osd.1 (osd.1) 75 : cluster 0 2.a scrub starts 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: cluster 2024-03-11T21:26:07.649128+0000 osd.1 (osd.1) 76 : cluster 0 2.a scrub ok 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.986955+0000 mon.a (mon.0) 872 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.994502+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.996171+0000 mon.c (mon.2) 76 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:07.996944+0000 mon.c (mon.2) 77 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: audit 2024-03-11T21:26:08.002944+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.144635+0000 osd.3 (osd.3) 83 : cluster 0 3.2 scrub starts 2024-03-11T21:26:08.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:08 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.148903+0000 osd.3 (osd.3) 84 : cluster 0 3.2 scrub ok 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: cluster 2024-03-11T21:26:07.105675+0000 mgr.y (mgr.14550) 114 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.393734+0000 mon.a (mon.0) 870 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.408007+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: cluster 2024-03-11T21:26:07.647583+0000 osd.1 (osd.1) 75 : cluster 0 2.a scrub starts 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: cluster 2024-03-11T21:26:07.649128+0000 osd.1 (osd.1) 76 : cluster 0 2.a scrub ok 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.986955+0000 mon.a (mon.0) 872 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.994502+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.996171+0000 mon.c (mon.2) 76 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:07.996944+0000 mon.c (mon.2) 77 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:08.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: audit 2024-03-11T21:26:08.002944+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.144635+0000 osd.3 (osd.3) 83 : cluster 0 3.2 scrub starts 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.148903+0000 osd.3 (osd.3) 84 : cluster 0 3.2 scrub ok 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: cluster 2024-03-11T21:26:07.105675+0000 mgr.y (mgr.14550) 114 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.393734+0000 mon.a (mon.0) 870 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.408007+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: cluster 2024-03-11T21:26:07.647583+0000 osd.1 (osd.1) 75 : cluster 0 2.a scrub starts 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: cluster 2024-03-11T21:26:07.649128+0000 osd.1 (osd.1) 76 : cluster 0 2.a scrub ok 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.986955+0000 mon.a (mon.0) 872 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.994502+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.996171+0000 mon.c (mon.2) 76 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:08.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:07.996944+0000 mon.c (mon.2) 77 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:08.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: audit 2024-03-11T21:26:08.002944+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:08.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.144635+0000 osd.3 (osd.3) 83 : cluster 0 3.2 scrub starts 2024-03-11T21:26:08.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:08 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.148903+0000 osd.3 (osd.3) 84 : cluster 0 3.2 scrub ok 2024-03-11T21:26:09.407 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[21512]: audit 2024-03-11T21:26:08.169879+0000 mgr.y (mgr.14550) 115 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.196603+0000 osd.7 (osd.7) 81 : cluster 0 3.e scrub starts 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.200521+0000 osd.7 (osd.7) 82 : cluster 0 3.e scrub ok 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.928300+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[21512]: cluster 2024-03-11T21:26:08.930403+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-03-11T21:26:09.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[29284]: audit 2024-03-11T21:26:08.169879+0000 mgr.y (mgr.14550) 115 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:09.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.196603+0000 osd.7 (osd.7) 81 : cluster 0 3.e scrub starts 2024-03-11T21:26:09.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.200521+0000 osd.7 (osd.7) 82 : cluster 0 3.e scrub ok 2024-03-11T21:26:09.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.928300+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-03-11T21:26:09.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:09 smithi143 bash[29284]: cluster 2024-03-11T21:26:08.930403+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-03-11T21:26:09.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:09 smithi203 bash[23654]: audit 2024-03-11T21:26:08.169879+0000 mgr.y (mgr.14550) 115 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:09.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:09 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.196603+0000 osd.7 (osd.7) 81 : cluster 0 3.e scrub starts 2024-03-11T21:26:09.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:09 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.200521+0000 osd.7 (osd.7) 82 : cluster 0 3.e scrub ok 2024-03-11T21:26:09.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:09 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.928300+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-03-11T21:26:09.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:09 smithi203 bash[23654]: cluster 2024-03-11T21:26:08.930403+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-03-11T21:26:10.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[21512]: cluster 2024-03-11T21:26:09.107167+0000 mgr.y (mgr.14550) 116 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[21512]: cluster 2024-03-11T21:26:09.572088+0000 osd.1 (osd.1) 77 : cluster 0 6.1d scrub starts 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[21512]: cluster 2024-03-11T21:26:09.574786+0000 osd.1 (osd.1) 78 : cluster 0 6.1d scrub ok 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[21512]: cluster 2024-03-11T21:26:09.764022+0000 osd.0 (osd.0) 73 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[21512]: cluster 2024-03-11T21:26:09.764130+0000 osd.0 (osd.0) 74 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[29284]: cluster 2024-03-11T21:26:09.107167+0000 mgr.y (mgr.14550) 116 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[29284]: cluster 2024-03-11T21:26:09.572088+0000 osd.1 (osd.1) 77 : cluster 0 6.1d scrub starts 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[29284]: cluster 2024-03-11T21:26:09.574786+0000 osd.1 (osd.1) 78 : cluster 0 6.1d scrub ok 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[29284]: cluster 2024-03-11T21:26:09.764022+0000 osd.0 (osd.0) 73 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:10.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:10 smithi143 bash[29284]: cluster 2024-03-11T21:26:09.764130+0000 osd.0 (osd.0) 74 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:10 smithi203 bash[23654]: cluster 2024-03-11T21:26:09.107167+0000 mgr.y (mgr.14550) 116 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:10 smithi203 bash[23654]: cluster 2024-03-11T21:26:09.572088+0000 osd.1 (osd.1) 77 : cluster 0 6.1d scrub starts 2024-03-11T21:26:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:10 smithi203 bash[23654]: cluster 2024-03-11T21:26:09.574786+0000 osd.1 (osd.1) 78 : cluster 0 6.1d scrub ok 2024-03-11T21:26:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:10 smithi203 bash[23654]: cluster 2024-03-11T21:26:09.764022+0000 osd.0 (osd.0) 73 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:10 smithi203 bash[23654]: cluster 2024-03-11T21:26:09.764130+0000 osd.0 (osd.0) 74 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.002592+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.004872+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.158342+0000 osd.7 (osd.7) 83 : cluster 0 3.11 scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.162645+0000 osd.7 (osd.7) 84 : cluster 0 3.11 scrub ok 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.417922+0000 mon.a (mon.0) 875 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.600070+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:10.602216+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:11.143518+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[29284]: cluster 2024-03-11T21:26:11.145846+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.002592+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.004872+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.158342+0000 osd.7 (osd.7) 83 : cluster 0 3.11 scrub starts 2024-03-11T21:26:11.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.162645+0000 osd.7 (osd.7) 84 : cluster 0 3.11 scrub ok 2024-03-11T21:26:11.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.417922+0000 mon.a (mon.0) 875 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:26:11.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.600070+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-03-11T21:26:11.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:10.602216+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-03-11T21:26:11.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:11.143518+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-03-11T21:26:11.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:11 smithi143 bash[21512]: cluster 2024-03-11T21:26:11.145846+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.002592+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.004872+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.158342+0000 osd.7 (osd.7) 83 : cluster 0 3.11 scrub starts 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.162645+0000 osd.7 (osd.7) 84 : cluster 0 3.11 scrub ok 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.417922+0000 mon.a (mon.0) 875 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.600070+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-03-11T21:26:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:10.602216+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-03-11T21:26:11.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:11.143518+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-03-11T21:26:11.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:11 smithi203 bash[23654]: cluster 2024-03-11T21:26:11.145846+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-03-11T21:26:12.396 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:26:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:11.107800+0000 mgr.y (mgr.14550) 117 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:11.454564+0000 osd.5 (osd.5) 85 : cluster 0 5.1b scrub starts 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:11.456885+0000 osd.5 (osd.5) 86 : cluster 0 5.1b scrub ok 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.154359+0000 osd.7 (osd.7) 87 : cluster 0 6.8 scrub starts 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.184884+0000 osd.3 (osd.3) 85 : cluster 0 2.1e scrub starts 2024-03-11T21:26:12.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.187168+0000 osd.3 (osd.3) 86 : cluster 0 2.1e scrub ok 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:11.107800+0000 mgr.y (mgr.14550) 117 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:11.454564+0000 osd.5 (osd.5) 85 : cluster 0 5.1b scrub starts 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:11.456885+0000 osd.5 (osd.5) 86 : cluster 0 5.1b scrub ok 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.154359+0000 osd.7 (osd.7) 87 : cluster 0 6.8 scrub starts 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.184884+0000 osd.3 (osd.3) 85 : cluster 0 2.1e scrub starts 2024-03-11T21:26:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:12 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.187168+0000 osd.3 (osd.3) 86 : cluster 0 2.1e scrub ok 2024-03-11T21:26:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:11.107800+0000 mgr.y (mgr.14550) 117 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:12.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:11.454564+0000 osd.5 (osd.5) 85 : cluster 0 5.1b scrub starts 2024-03-11T21:26:12.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:11.456885+0000 osd.5 (osd.5) 86 : cluster 0 5.1b scrub ok 2024-03-11T21:26:12.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.154359+0000 osd.7 (osd.7) 87 : cluster 0 6.8 scrub starts 2024-03-11T21:26:12.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.184884+0000 osd.3 (osd.3) 85 : cluster 0 2.1e scrub starts 2024-03-11T21:26:12.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:12 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.187168+0000 osd.3 (osd.3) 86 : cluster 0 2.1e scrub ok 2024-03-11T21:26:12.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/4237463564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074010 msgr2=0x7f81580743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:12.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- 172.21.15.143:0/4237463564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074010 0x7f81580743f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8150009990 tx=0x7f8150009ca0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/4237463564 shutdown_connections 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- 172.21.15.143:0/4237463564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 0x7f8158110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- 172.21.15.143:0/4237463564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074010 0x7f81580743f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- 172.21.15.143:0/4237463564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074930 0x7f815810e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/4237463564 >> 172.21.15.143:0/4237463564 conn(0x7f8158078eb0 msgr2=0x7f81580792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/4237463564 shutdown_connections 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/4237463564 wait complete. 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 librados: starting msgr at 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 librados: starting objecter 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 Processor -- start 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- start start 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 librados: setting wanted keys 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 librados: calling monclient init 2024-03-11T21:26:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 0x7f815819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 0x7f815819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8158113850 con 0x7f8158074010 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81581136d0 con 0x7f815810e870 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81581139d0 con 0x7f8158074930 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f8157fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 0x7f815819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49788/0 (socket says 172.21.15.143:49788) 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 -- 172.21.15.143:0/1022736196 learned_addr learned my addr 172.21.15.143:0/1022736196 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:12.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f8157fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 0x7f815819a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46314/0 (socket says 172.21.15.143:46314) 2024-03-11T21:26:12.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 -- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 msgr2=0x7f815819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 0x7f815819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 -- 172.21.15.143:0/1022736196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 msgr2=0x7f815819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815d02a700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 0x7f815819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 0x7f815819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 -- 172.21.15.143:0/1022736196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815819ee40 con 0x7f8158074010 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815c829700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f815000b2e0 tx=0x7f81500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f8155ffb700 10 client.?.objecter ms_handle_connect 0x7f8158074010 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f8155ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8150026070 con 0x7f8158074010 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.925+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815819f0d0 con 0x7f8158074010 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f815ea8d700 10 client.14721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:12.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81581a69b0 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f815ea8d700 1 librados: init done 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8150021410 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8150017650 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8138005120 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8150029070 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 0x7f8144071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(118..118 src has 66..118) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f815008f190 con 0x7f8158074010 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 10 client.14721.objecter ms_dispatch 0x7f8158001550 osd_map(118..118 src has 66..118) v4 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 3 client.14721.objecter handle_osd_map got epochs [118,118] > 0 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 3 client.14721.objecter handle_osd_map decoding full epoch 118 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8157fff700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 0x7f8144071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:12.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 20 client.14721.objecter dump_active .. 0 homeless 2024-03-11T21:26:12.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8157fff700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 0x7f8144071ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8140009680 tx=0x7f8140009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:12.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.929+0000 7f8155ffb700 10 client.14721.objecter ms_handle_connect 0x7f814406f820 2024-03-11T21:26:12.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:12.933+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8150021e20 con 0x7f8158074010 2024-03-11T21:26:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.137+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8138000bc0 con 0x7f814406f820 2024-03-11T21:26:13.134 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.157+0000 7f8155ffb700 1 -- 172.21.15.143:0/1022736196 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346405 (secure 0 0 0) 0x7f8138000bc0 con 0x7f814406f820 2024-03-11T21:26:13.138 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 10 librados: watch_flush enter 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 10 librados: watch_flush exit 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 20 client.14721.objecter shutdown clearing up homeless session... 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 10 client.14721.objecter successfully canceled tick 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 msgr2=0x7f8144071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 0x7f8144071ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8140009680 tx=0x7f8140009260 comp rx=0 tx=0).stop 2024-03-11T21:26:13.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 msgr2=0x7f8158199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f815000b2e0 tx=0x7f81500047b0 comp rx=0 tx=0).stop 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 shutdown_connections 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f814406f820 0x7f8144071ce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8158074010 0x7f8158199e80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f815810e870 0x7f815819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 --2- 172.21.15.143:0/1022736196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8158074930 0x7f815819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 >> 172.21.15.143:0/1022736196 conn(0x7f8158078eb0 msgr2=0x7f81581062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 shutdown_connections 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 -- 172.21.15.143:0/1022736196 wait complete. 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:13.161+0000 7f815ea8d700 1 librados: shutdown 2024-03-11T21:26:13.143 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:26:13.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.157148+0000 osd.7 (osd.7) 88 : cluster 0 6.8 scrub ok 2024-03-11T21:26:13.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.570942+0000 osd.1 (osd.1) 81 : cluster 0 5.c scrub starts 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[21512]: cluster 2024-03-11T21:26:12.573279+0000 osd.1 (osd.1) 82 : cluster 0 5.c scrub ok 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[21512]: audit 2024-03-11T21:26:12.679864+0000 mon.c (mon.2) 78 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.157148+0000 osd.7 (osd.7) 88 : cluster 0 6.8 scrub ok 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.570942+0000 osd.1 (osd.1) 81 : cluster 0 5.c scrub starts 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[29284]: cluster 2024-03-11T21:26:12.573279+0000 osd.1 (osd.1) 82 : cluster 0 5.c scrub ok 2024-03-11T21:26:13.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:13 smithi143 bash[29284]: audit 2024-03-11T21:26:12.679864+0000 mon.c (mon.2) 78 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:13.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:13 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.157148+0000 osd.7 (osd.7) 88 : cluster 0 6.8 scrub ok 2024-03-11T21:26:13.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:13 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.570942+0000 osd.1 (osd.1) 81 : cluster 0 5.c scrub starts 2024-03-11T21:26:13.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:13 smithi203 bash[23654]: cluster 2024-03-11T21:26:12.573279+0000 osd.1 (osd.1) 82 : cluster 0 5.c scrub ok 2024-03-11T21:26:13.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:13 smithi203 bash[23654]: audit 2024-03-11T21:26:12.679864+0000 mon.c (mon.2) 78 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:14.048 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-03-11T21:26:13.108018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1121,"num_read_kb":967,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":674,"ondisk_log_size":674,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248532,"kb_used_data":7420,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712684,"statfs":{"total":767960285184,"available":767705788416,"internally_reserved":0,"allocated":7598080,"data_stored":4830232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.973129"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593592+0000","last_change":"2024-03-11T21:24:49.960853+0000","last_active":"2024-03-11T21:26:10.593592+0000","last_peered":"2024-03-11T21:26:10.593592+0000","last_clean":"2024-03-11T21:26:10.593592+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:26:10.593592+0000","last_undegraded":"2024-03-11T21:26:10.593592+0000","last_fullsized":"2024-03-11T21:26:10.593592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:49.960558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:49.960558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410167+0000","last_change":"2024-03-11T21:25:58.944512+0000","last_active":"2024-03-11T21:26:10.410167+0000","last_peered":"2024-03-11T21:26:10.410167+0000","last_clean":"2024-03-11T21:26:10.410167+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:26:10.410167+0000","last_undegraded":"2024-03-11T21:26:10.410167+0000","last_fullsized":"2024-03-11T21:26:10.410167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.944364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.944364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.570898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":119,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420851+0000","last_change":"2024-03-11T21:26:03.337169+0000","last_active":"2024-03-11T21:26:10.420851+0000","last_peered":"2024-03-11T21:26:10.420851+0000","last_clean":"2024-03-11T21:26:10.420851+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:26:10.420851+0000","last_undegraded":"2024-03-11T21:26:10.420851+0000","last_fullsized":"2024-03-11T21:26:10.420851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:26:03.337010+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.337010+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:29.657159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594613+0000","last_change":"2024-03-11T21:25:31.918871+0000","last_active":"2024-03-11T21:26:10.594613+0000","last_peered":"2024-03-11T21:26:10.594613+0000","last_clean":"2024-03-11T21:26:10.594613+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:26:10.594613+0000","last_undegraded":"2024-03-11T21:26:10.594613+0000","last_fullsized":"2024-03-11T21:26:10.594613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:31.918660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.918660+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.711285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410391+0000","last_change":"2024-03-11T21:25:48.920160+0000","last_active":"2024-03-11T21:26:10.410391+0000","last_peered":"2024-03-11T21:26:10.410391+0000","last_clean":"2024-03-11T21:26:10.410391+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:26:10.410391+0000","last_undegraded":"2024-03-11T21:26:10.410391+0000","last_fullsized":"2024-03-11T21:26:10.410391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.919930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.919930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.064632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.418884+0000","last_change":"2024-03-11T21:24:42.336554+0000","last_active":"2024-03-11T21:26:10.418884+0000","last_peered":"2024-03-11T21:26:10.418884+0000","last_clean":"2024-03-11T21:26:10.418884+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:26:10.418884+0000","last_undegraded":"2024-03-11T21:26:10.418884+0000","last_fullsized":"2024-03-11T21:26:10.418884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:42.336276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:42.336276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:12.187339+0000","last_change":"2024-03-11T21:26:12.187395+0000","last_active":"2024-03-11T21:26:12.187339+0000","last_peered":"2024-03-11T21:26:12.187339+0000","last_clean":"2024-03-11T21:26:12.187339+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:26:12.187339+0000","last_undegraded":"2024-03-11T21:26:12.187339+0000","last_fullsized":"2024-03-11T21:26:12.187339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.187178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.187178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.244264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420694+0000","last_change":"2024-03-11T21:25:46.288260+0000","last_active":"2024-03-11T21:26:10.420694+0000","last_peered":"2024-03-11T21:26:10.420694+0000","last_clean":"2024-03-11T21:26:10.420694+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:26:10.420694+0000","last_undegraded":"2024-03-11T21:26:10.420694+0000","last_fullsized":"2024-03-11T21:26:10.420694+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:46.288021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.288021+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.040825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593236+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:26:10.593236+0000","last_peered":"2024-03-11T21:26:10.593236+0000","last_clean":"2024-03-11T21:26:10.593236+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:26:10.593236+0000","last_undegraded":"2024-03-11T21:26:10.593236+0000","last_fullsized":"2024-03-11T21:26:10.593236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.062649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:11.146010+0000","last_change":"2024-03-11T21:26:11.146066+0000","last_active":"2024-03-11T21:26:11.146010+0000","last_peered":"2024-03-11T21:26:11.146010+0000","last_clean":"2024-03-11T21:26:11.146010+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:26:11.146010+0000","last_undegraded":"2024-03-11T21:26:11.146010+0000","last_fullsized":"2024-03-11T21:26:11.146010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.145856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.145856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.221066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677508+0000","last_change":"2024-03-11T21:25:25.221221+0000","last_active":"2024-03-11T21:26:10.677508+0000","last_peered":"2024-03-11T21:26:10.677508+0000","last_clean":"2024-03-11T21:26:10.677508+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:26:10.677508+0000","last_undegraded":"2024-03-11T21:26:10.677508+0000","last_fullsized":"2024-03-11T21:26:10.677508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:25.220993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:25.220993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:32.679280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":131,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594781+0000","last_change":"2024-03-11T21:25:19.442614+0000","last_active":"2024-03-11T21:26:10.594781+0000","last_peered":"2024-03-11T21:26:10.594781+0000","last_clean":"2024-03-11T21:26:10.594781+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:26:10.594781+0000","last_undegraded":"2024-03-11T21:26:10.594781+0000","last_fullsized":"2024-03-11T21:26:10.594781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:19.442347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:19.442347+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:31.033058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.414853+0000","last_change":"2024-03-11T21:26:04.385285+0000","last_active":"2024-03-11T21:26:10.414853+0000","last_peered":"2024-03-11T21:26:10.414853+0000","last_clean":"2024-03-11T21:26:10.414853+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:26:10.414853+0000","last_undegraded":"2024-03-11T21:26:10.414853+0000","last_fullsized":"2024-03-11T21:26:10.414853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.385068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.385068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.394697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:11.457054+0000","last_change":"2024-03-11T21:26:11.457112+0000","last_active":"2024-03-11T21:26:11.457054+0000","last_peered":"2024-03-11T21:26:11.457054+0000","last_clean":"2024-03-11T21:26:11.457054+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:26:11.457054+0000","last_undegraded":"2024-03-11T21:26:11.457054+0000","last_fullsized":"2024-03-11T21:26:11.457054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.456900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.456900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:19.697223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676677+0000","last_change":"2024-03-11T21:25:48.050860+0000","last_active":"2024-03-11T21:26:10.676677+0000","last_peered":"2024-03-11T21:26:10.676677+0000","last_clean":"2024-03-11T21:26:10.676677+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:26:10.676677+0000","last_undegraded":"2024-03-11T21:26:10.676677+0000","last_fullsized":"2024-03-11T21:26:10.676677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.050648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.050648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.250408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":129,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594741+0000","last_change":"2024-03-11T21:25:51.434868+0000","last_active":"2024-03-11T21:26:10.594741+0000","last_peered":"2024-03-11T21:26:10.594741+0000","last_clean":"2024-03-11T21:26:10.594741+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:26:10.594741+0000","last_undegraded":"2024-03-11T21:26:10.594741+0000","last_fullsized":"2024-03-11T21:26:10.594741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:51.434497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:51.434497+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.747829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594902+0000","last_change":"2024-03-11T21:24:44.059553+0000","last_active":"2024-03-11T21:26:10.594902+0000","last_peered":"2024-03-11T21:26:10.594902+0000","last_clean":"2024-03-11T21:26:10.594902+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:26:10.594902+0000","last_undegraded":"2024-03-11T21:26:10.594902+0000","last_fullsized":"2024-03-11T21:26:10.594902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:44.059198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:44.059198+0000","last_clean_scrub_stamp":"2024-03-11T21:24:44.059198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.411168+0000","last_change":"2024-03-11T21:26:10.005094+0000","last_active":"2024-03-11T21:26:10.411168+0000","last_peered":"2024-03-11T21:26:10.411168+0000","last_clean":"2024-03-11T21:26:10.411168+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:26:10.411168+0000","last_undegraded":"2024-03-11T21:26:10.411168+0000","last_fullsized":"2024-03-11T21:26:10.411168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.004882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.004882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:30.146208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593864+0000","last_change":"2024-03-11T21:25:18.887001+0000","last_active":"2024-03-11T21:26:10.593864+0000","last_peered":"2024-03-11T21:26:10.593864+0000","last_clean":"2024-03-11T21:26:10.593864+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:26:10.593864+0000","last_undegraded":"2024-03-11T21:26:10.593864+0000","last_fullsized":"2024-03-11T21:26:10.593864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:18.886911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:18.886911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:36.216660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410806+0000","last_change":"2024-03-11T21:25:43.877458+0000","last_active":"2024-03-11T21:26:10.410806+0000","last_peered":"2024-03-11T21:26:10.410806+0000","last_clean":"2024-03-11T21:26:10.410806+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:26:10.410806+0000","last_undegraded":"2024-03-11T21:26:10.410806+0000","last_fullsized":"2024-03-11T21:26:10.410806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:43.877225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.877225+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.780065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.412196+0000","last_change":"2024-03-11T21:25:59.988760+0000","last_active":"2024-03-11T21:26:10.412196+0000","last_peered":"2024-03-11T21:26:10.412196+0000","last_clean":"2024-03-11T21:26:10.412196+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:26:10.412196+0000","last_undegraded":"2024-03-11T21:26:10.412196+0000","last_fullsized":"2024-03-11T21:26:10.412196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.988522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.034265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420714+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:26:10.420714+0000","last_peered":"2024-03-11T21:26:10.420714+0000","last_clean":"2024-03-11T21:26:10.420714+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:26:10.420714+0000","last_undegraded":"2024-03-11T21:26:10.420714+0000","last_fullsized":"2024-03-11T21:26:10.420714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.655474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.984908+0000","last_change":"2024-03-11T21:26:04.956591+0000","last_active":"2024-03-11T21:26:10.984908+0000","last_peered":"2024-03-11T21:26:10.984908+0000","last_clean":"2024-03-11T21:26:10.984908+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:26:10.984908+0000","last_undegraded":"2024-03-11T21:26:10.984908+0000","last_fullsized":"2024-03-11T21:26:10.984908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.956350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.956350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.718796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":108,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.416698+0000","last_change":"2024-03-11T21:24:40.449514+0000","last_active":"2024-03-11T21:26:10.416698+0000","last_peered":"2024-03-11T21:26:10.416698+0000","last_clean":"2024-03-11T21:26:10.416698+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:26:10.416698+0000","last_undegraded":"2024-03-11T21:26:10.416698+0000","last_fullsized":"2024-03-11T21:26:10.416698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T21:24:40.449263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:40.449263+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420927+0000","last_change":"2024-03-11T21:26:09.575095+0000","last_active":"2024-03-11T21:26:10.420927+0000","last_peered":"2024-03-11T21:26:10.420927+0000","last_clean":"2024-03-11T21:26:10.420927+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:26:10.420927+0000","last_undegraded":"2024-03-11T21:26:10.420927+0000","last_fullsized":"2024-03-11T21:26:10.420927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:09.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:09.574796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.590692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.415710+0000","last_change":"2024-03-11T21:25:40.280094+0000","last_active":"2024-03-11T21:26:10.415710+0000","last_peered":"2024-03-11T21:26:10.415710+0000","last_clean":"2024-03-11T21:26:10.415710+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:26:10.415710+0000","last_undegraded":"2024-03-11T21:26:10.415710+0000","last_fullsized":"2024-03-11T21:26:10.415710+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.279715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.279715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.096032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593466+0000","last_change":"2024-03-11T21:25:27.026231+0000","last_active":"2024-03-11T21:26:10.593466+0000","last_peered":"2024-03-11T21:26:10.593466+0000","last_clean":"2024-03-11T21:26:10.593466+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:26:10.593466+0000","last_undegraded":"2024-03-11T21:26:10.593466+0000","last_fullsized":"2024-03-11T21:26:10.593466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:27.025913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.025913+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:37.624911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594722+0000","last_change":"2024-03-11T21:25:30.949367+0000","last_active":"2024-03-11T21:26:10.594722+0000","last_peered":"2024-03-11T21:26:10.594722+0000","last_clean":"2024-03-11T21:26:10.594722+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:26:10.594722+0000","last_undegraded":"2024-03-11T21:26:10.594722+0000","last_fullsized":"2024-03-11T21:26:10.594722+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:30.949072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.949072+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:33.375300+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677348+0000","last_change":"2024-03-11T21:25:16.227364+0000","last_active":"2024-03-11T21:26:10.677348+0000","last_peered":"2024-03-11T21:26:10.677348+0000","last_clean":"2024-03-11T21:26:10.677348+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:26:10.677348+0000","last_undegraded":"2024-03-11T21:26:10.677348+0000","last_fullsized":"2024-03-11T21:26:10.677348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:25:16.227113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:16.227113+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:28.899079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":153,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.985844+0000","last_change":"2024-03-11T21:26:05.978542+0000","last_active":"2024-03-11T21:26:10.985844+0000","last_peered":"2024-03-11T21:26:10.985844+0000","last_clean":"2024-03-11T21:26:10.985844+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:26:10.985844+0000","last_undegraded":"2024-03-11T21:26:10.985844+0000","last_fullsized":"2024-03-11T21:26:10.985844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:05.978319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.978319+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.180257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":116,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593179+0000","last_change":"2024-03-11T21:26:02.471008+0000","last_active":"2024-03-11T21:26:10.593179+0000","last_peered":"2024-03-11T21:26:10.593179+0000","last_clean":"2024-03-11T21:26:10.593179+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:26:10.593179+0000","last_undegraded":"2024-03-11T21:26:10.593179+0000","last_fullsized":"2024-03-11T21:26:10.593179+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.470792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.470792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.529057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":131,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.419405+0000","last_change":"2024-03-11T21:26:03.728864+0000","last_active":"2024-03-11T21:26:10.419405+0000","last_peered":"2024-03-11T21:26:10.419405+0000","last_clean":"2024-03-11T21:26:10.419405+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:26:10.419405+0000","last_undegraded":"2024-03-11T21:26:10.419405+0000","last_fullsized":"2024-03-11T21:26:10.419405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:03.728766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.728766+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.097957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594389+0000","last_change":"2024-03-11T21:24:43.025094+0000","last_active":"2024-03-11T21:26:10.594389+0000","last_peered":"2024-03-11T21:26:10.594389+0000","last_clean":"2024-03-11T21:26:10.594389+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:26:10.594389+0000","last_undegraded":"2024-03-11T21:26:10.594389+0000","last_fullsized":"2024-03-11T21:26:10.594389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:43.024865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:43.024865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676375+0000","last_change":"2024-03-11T21:25:03.181247+0000","last_active":"2024-03-11T21:26:10.676375+0000","last_peered":"2024-03-11T21:26:10.676375+0000","last_clean":"2024-03-11T21:26:10.676375+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:26:10.676375+0000","last_undegraded":"2024-03-11T21:26:10.676375+0000","last_fullsized":"2024-03-11T21:26:10.676375+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:03.181042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:03.181042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:11.194423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.986051+0000","last_change":"2024-03-11T21:25:40.946608+0000","last_active":"2024-03-11T21:26:10.986051+0000","last_peered":"2024-03-11T21:26:10.986051+0000","last_clean":"2024-03-11T21:26:10.986051+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:26:10.986051+0000","last_undegraded":"2024-03-11T21:26:10.986051+0000","last_fullsized":"2024-03-11T21:26:10.986051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.946307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594734+0000","last_change":"2024-03-11T21:25:27.506732+0000","last_active":"2024-03-11T21:26:10.594734+0000","last_peered":"2024-03-11T21:26:10.594734+0000","last_clean":"2024-03-11T21:26:10.594734+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:26:10.594734+0000","last_undegraded":"2024-03-11T21:26:10.594734+0000","last_fullsized":"2024-03-11T21:26:10.594734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:27.506461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.506461+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.042628+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.675904+0000","last_change":"2024-03-11T21:25:53.051632+0000","last_active":"2024-03-11T21:26:10.675904+0000","last_peered":"2024-03-11T21:26:10.675904+0000","last_clean":"2024-03-11T21:26:10.675904+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:26:10.675904+0000","last_undegraded":"2024-03-11T21:26:10.675904+0000","last_fullsized":"2024-03-11T21:26:10.675904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.051422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.051422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.051498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.984660+0000","last_change":"2024-03-11T21:25:27.905116+0000","last_active":"2024-03-11T21:26:10.984660+0000","last_peered":"2024-03-11T21:26:10.984660+0000","last_clean":"2024-03-11T21:26:10.984660+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:26:10.984660+0000","last_undegraded":"2024-03-11T21:26:10.984660+0000","last_fullsized":"2024-03-11T21:26:10.984660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.904786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.904786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:28.273395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595025+0000","last_change":"2024-03-11T21:25:48.374372+0000","last_active":"2024-03-11T21:26:10.595025+0000","last_peered":"2024-03-11T21:26:10.595025+0000","last_clean":"2024-03-11T21:26:10.595025+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:26:10.595025+0000","last_undegraded":"2024-03-11T21:26:10.595025+0000","last_fullsized":"2024-03-11T21:26:10.595025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.374158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.374158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.742286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":115,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.421061+0000","last_change":"2024-03-11T21:25:37.334081+0000","last_active":"2024-03-11T21:26:10.421061+0000","last_peered":"2024-03-11T21:26:10.421061+0000","last_clean":"2024-03-11T21:26:10.421061+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:26:10.421061+0000","last_undegraded":"2024-03-11T21:26:10.421061+0000","last_fullsized":"2024-03-11T21:26:10.421061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:25:37.333856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.333856+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.493865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593172+0000","last_change":"2024-03-11T21:25:56.141805+0000","last_active":"2024-03-11T21:26:10.593172+0000","last_peered":"2024-03-11T21:26:10.593172+0000","last_clean":"2024-03-11T21:26:10.593172+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:26:10.593172+0000","last_undegraded":"2024-03-11T21:26:10.593172+0000","last_fullsized":"2024-03-11T21:26:10.593172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.141559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.141559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.340511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420435+0000","last_change":"2024-03-11T21:24:43.364595+0000","last_active":"2024-03-11T21:26:10.420435+0000","last_peered":"2024-03-11T21:26:10.420435+0000","last_clean":"2024-03-11T21:26:10.420435+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:26:10.420435+0000","last_undegraded":"2024-03-11T21:26:10.420435+0000","last_fullsized":"2024-03-11T21:26:10.420435+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:43.364477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:43.364477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.419745+0000","last_change":"2024-03-11T21:25:53.706422+0000","last_active":"2024-03-11T21:26:10.419745+0000","last_peered":"2024-03-11T21:26:10.419745+0000","last_clean":"2024-03-11T21:26:10.419745+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:26:10.419745+0000","last_undegraded":"2024-03-11T21:26:10.419745+0000","last_fullsized":"2024-03-11T21:26:10.419745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.706230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.384575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.411615+0000","last_change":"2024-03-11T21:25:38.984101+0000","last_active":"2024-03-11T21:26:10.411615+0000","last_peered":"2024-03-11T21:26:10.411615+0000","last_clean":"2024-03-11T21:26:10.411615+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:26:10.411615+0000","last_undegraded":"2024-03-11T21:26:10.411615+0000","last_fullsized":"2024-03-11T21:26:10.411615+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:38.983858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.983858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.131782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.414550+0000","last_change":"2024-03-11T21:25:56.344253+0000","last_active":"2024-03-11T21:26:10.414550+0000","last_peered":"2024-03-11T21:26:10.414550+0000","last_clean":"2024-03-11T21:26:10.414550+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:26:10.414550+0000","last_undegraded":"2024-03-11T21:26:10.414550+0000","last_fullsized":"2024-03-11T21:26:10.414550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.343979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.343979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.628275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594094+0000","last_change":"2024-03-11T21:25:39.974174+0000","last_active":"2024-03-11T21:26:10.594094+0000","last_peered":"2024-03-11T21:26:10.594094+0000","last_clean":"2024-03-11T21:26:10.594094+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:26:10.594094+0000","last_undegraded":"2024-03-11T21:26:10.594094+0000","last_fullsized":"2024-03-11T21:26:10.594094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.973978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.973978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.092042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.985422+0000","last_change":"2024-03-11T21:25:45.890051+0000","last_active":"2024-03-11T21:26:10.985422+0000","last_peered":"2024-03-11T21:26:10.985422+0000","last_clean":"2024-03-11T21:26:10.985422+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:26:10.985422+0000","last_undegraded":"2024-03-11T21:26:10.985422+0000","last_fullsized":"2024-03-11T21:26:10.985422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:45.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.889834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.143004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677800+0000","last_change":"2024-03-11T21:25:37.960424+0000","last_active":"2024-03-11T21:26:10.677800+0000","last_peered":"2024-03-11T21:26:10.677800+0000","last_clean":"2024-03-11T21:26:10.677800+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:26:10.677800+0000","last_undegraded":"2024-03-11T21:26:10.677800+0000","last_fullsized":"2024-03-11T21:26:10.677800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:25:37.960057+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.960057+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.161700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.412597+0000","last_change":"2024-03-11T21:25:40.949182+0000","last_active":"2024-03-11T21:26:10.412597+0000","last_peered":"2024-03-11T21:26:10.412597+0000","last_clean":"2024-03-11T21:26:10.412597+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:26:10.412597+0000","last_undegraded":"2024-03-11T21:26:10.412597+0000","last_fullsized":"2024-03-11T21:26:10.412597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.948963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.948963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.061308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'11","reported_seq":150,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594270+0000","last_change":"2024-03-11T21:25:38.973030+0000","last_active":"2024-03-11T21:26:10.594270+0000","last_peered":"2024-03-11T21:26:10.594270+0000","last_clean":"2024-03-11T21:26:10.594270+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:26:10.594270+0000","last_undegraded":"2024-03-11T21:26:10.594270+0000","last_fullsized":"2024-03-11T21:26:10.594270+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:38.972802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.972802+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.415779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.416449+0000","last_change":"2024-03-11T21:25:54.342862+0000","last_active":"2024-03-11T21:26:10.416449+0000","last_peered":"2024-03-11T21:26:10.416449+0000","last_clean":"2024-03-11T21:26:10.416449+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:26:10.416449+0000","last_undegraded":"2024-03-11T21:26:10.416449+0000","last_fullsized":"2024-03-11T21:26:10.416449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.342588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.028053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.415322+0000","last_change":"2024-03-11T21:26:01.314774+0000","last_active":"2024-03-11T21:26:10.415322+0000","last_peered":"2024-03-11T21:26:10.415322+0000","last_clean":"2024-03-11T21:26:10.415322+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:26:10.415322+0000","last_undegraded":"2024-03-11T21:26:10.415322+0000","last_fullsized":"2024-03-11T21:26:10.415322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:26:01.314615+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.665529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.415524+0000","last_change":"2024-03-11T21:25:43.246214+0000","last_active":"2024-03-11T21:26:10.415524+0000","last_peered":"2024-03-11T21:26:10.415524+0000","last_clean":"2024-03-11T21:26:10.415524+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:26:10.415524+0000","last_undegraded":"2024-03-11T21:26:10.415524+0000","last_fullsized":"2024-03-11T21:26:10.415524+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.246002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.246002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.409171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":150,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595017+0000","last_change":"2024-03-11T21:25:59.442248+0000","last_active":"2024-03-11T21:26:10.595017+0000","last_peered":"2024-03-11T21:26:10.595017+0000","last_clean":"2024-03-11T21:26:10.595017+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:26:10.595017+0000","last_undegraded":"2024-03-11T21:26:10.595017+0000","last_fullsized":"2024-03-11T21:26:10.595017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:59.442015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.442015+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.052362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593433+0000","last_change":"2024-03-11T21:25:17.426143+0000","last_active":"2024-03-11T21:26:10.593433+0000","last_peered":"2024-03-11T21:26:10.593433+0000","last_clean":"2024-03-11T21:26:10.593433+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:26:10.593433+0000","last_undegraded":"2024-03-11T21:26:10.593433+0000","last_fullsized":"2024-03-11T21:26:10.593433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:17.425944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:17.425944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:23.643326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676610+0000","last_change":"2024-03-11T21:25:39.009861+0000","last_active":"2024-03-11T21:26:10.676610+0000","last_peered":"2024-03-11T21:26:10.676610+0000","last_clean":"2024-03-11T21:26:10.676610+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:26:10.676610+0000","last_undegraded":"2024-03-11T21:26:10.676610+0000","last_fullsized":"2024-03-11T21:26:10.676610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:39.009635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.009635+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:46.254494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676953+0000","last_change":"2024-03-11T21:25:49.003277+0000","last_active":"2024-03-11T21:26:10.676953+0000","last_peered":"2024-03-11T21:26:10.676953+0000","last_clean":"2024-03-11T21:26:10.676953+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:26:10.676953+0000","last_undegraded":"2024-03-11T21:26:10.676953+0000","last_fullsized":"2024-03-11T21:26:10.676953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:49.003014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.003014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.285739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594471+0000","last_change":"2024-03-11T21:25:54.360923+0000","last_active":"2024-03-11T21:26:10.594471+0000","last_peered":"2024-03-11T21:26:10.594471+0000","last_clean":"2024-03-11T21:26:10.594471+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:26:10.594471+0000","last_undegraded":"2024-03-11T21:26:10.594471+0000","last_fullsized":"2024-03-11T21:26:10.594471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.360647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.360647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.912837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.984681+0000","last_change":"2024-03-11T21:25:36.945869+0000","last_active":"2024-03-11T21:26:10.984681+0000","last_peered":"2024-03-11T21:26:10.984681+0000","last_clean":"2024-03-11T21:26:10.984681+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:26:10.984681+0000","last_undegraded":"2024-03-11T21:26:10.984681+0000","last_fullsized":"2024-03-11T21:26:10.984681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:36.945583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:36.945583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.141176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.984990+0000","last_change":"2024-03-11T21:25:46.921649+0000","last_active":"2024-03-11T21:26:10.984990+0000","last_peered":"2024-03-11T21:26:10.984990+0000","last_clean":"2024-03-11T21:26:10.984990+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:26:10.984990+0000","last_undegraded":"2024-03-11T21:26:10.984990+0000","last_fullsized":"2024-03-11T21:26:10.984990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.921404+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.755306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934504+0000","last_change":"2024-03-11T21:25:50.974118+0000","last_active":"2024-03-11T21:26:10.934504+0000","last_peered":"2024-03-11T21:26:10.934504+0000","last_clean":"2024-03-11T21:26:10.934504+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:26:10.934504+0000","last_undegraded":"2024-03-11T21:26:10.934504+0000","last_fullsized":"2024-03-11T21:26:10.934504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:50.974032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.974032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.283434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594982+0000","last_change":"2024-03-11T21:26:04.178398+0000","last_active":"2024-03-11T21:26:10.594982+0000","last_peered":"2024-03-11T21:26:10.594982+0000","last_clean":"2024-03-11T21:26:10.594982+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:26:10.594982+0000","last_undegraded":"2024-03-11T21:26:10.594982+0000","last_fullsized":"2024-03-11T21:26:10.594982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.178145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.178145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.425359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.933897+0000","last_change":"2024-03-11T21:25:29.104018+0000","last_active":"2024-03-11T21:26:10.933897+0000","last_peered":"2024-03-11T21:26:10.933897+0000","last_clean":"2024-03-11T21:26:10.933897+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:26:10.933897+0000","last_undegraded":"2024-03-11T21:26:10.933897+0000","last_fullsized":"2024-03-11T21:26:10.933897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:29.103775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.103775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:44.777028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.675851+0000","last_change":"2024-03-11T21:26:10.162900+0000","last_active":"2024-03-11T21:26:10.675851+0000","last_peered":"2024-03-11T21:26:10.675851+0000","last_clean":"2024-03-11T21:26:10.675851+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:26:10.675851+0000","last_undegraded":"2024-03-11T21:26:10.675851+0000","last_fullsized":"2024-03-11T21:26:10.675851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:10.162661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.162661+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:28.988891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594302+0000","last_change":"2024-03-11T21:25:27.983629+0000","last_active":"2024-03-11T21:26:10.594302+0000","last_peered":"2024-03-11T21:26:10.594302+0000","last_clean":"2024-03-11T21:26:10.594302+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:26:10.594302+0000","last_undegraded":"2024-03-11T21:26:10.594302+0000","last_fullsized":"2024-03-11T21:26:10.594302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.983536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.983536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.044284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.412137+0000","last_change":"2024-03-11T21:25:37.981238+0000","last_active":"2024-03-11T21:26:10.412137+0000","last_peered":"2024-03-11T21:26:10.412137+0000","last_clean":"2024-03-11T21:26:10.412137+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:26:10.412137+0000","last_undegraded":"2024-03-11T21:26:10.412137+0000","last_fullsized":"2024-03-11T21:26:10.412137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:25:37.981019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.981019+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.806629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":130,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677409+0000","last_change":"2024-03-11T21:26:08.200786+0000","last_active":"2024-03-11T21:26:10.677409+0000","last_peered":"2024-03-11T21:26:10.677409+0000","last_clean":"2024-03-11T21:26:10.677409+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:26:10.677409+0000","last_undegraded":"2024-03-11T21:26:10.677409+0000","last_fullsized":"2024-03-11T21:26:10.677409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:26:08.200532+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:24:33.079623+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.200532+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.719717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.933826+0000","last_change":"2024-03-11T21:26:02.944301+0000","last_active":"2024-03-11T21:26:10.933826+0000","last_peered":"2024-03-11T21:26:10.933826+0000","last_clean":"2024-03-11T21:26:10.933826+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:26:10.933826+0000","last_undegraded":"2024-03-11T21:26:10.933826+0000","last_fullsized":"2024-03-11T21:26:10.933826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:21.682420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.414596+0000","last_change":"2024-03-11T21:25:48.263693+0000","last_active":"2024-03-11T21:26:10.414596+0000","last_peered":"2024-03-11T21:26:10.414596+0000","last_clean":"2024-03-11T21:26:10.414596+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:26:10.414596+0000","last_undegraded":"2024-03-11T21:26:10.414596+0000","last_fullsized":"2024-03-11T21:26:10.414596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.263515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.384625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420024+0000","last_change":"2024-03-11T21:25:33.757745+0000","last_active":"2024-03-11T21:26:10.420024+0000","last_peered":"2024-03-11T21:26:10.420024+0000","last_clean":"2024-03-11T21:26:10.420024+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:26:10.420024+0000","last_undegraded":"2024-03-11T21:26:10.420024+0000","last_fullsized":"2024-03-11T21:26:10.420024+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:33.757442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:33.757442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.904053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":166,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594280+0000","last_change":"2024-03-11T21:25:21.393014+0000","last_active":"2024-03-11T21:26:10.594280+0000","last_peered":"2024-03-11T21:26:10.594280+0000","last_clean":"2024-03-11T21:26:10.594280+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:26:10.594280+0000","last_undegraded":"2024-03-11T21:26:10.594280+0000","last_fullsized":"2024-03-11T21:26:10.594280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:25:21.392637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:21.392637+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:43.705727+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420183+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:26:10.420183+0000","last_peered":"2024-03-11T21:26:10.420183+0000","last_clean":"2024-03-11T21:26:10.420183+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:26:10.420183+0000","last_undegraded":"2024-03-11T21:26:10.420183+0000","last_fullsized":"2024-03-11T21:26:10.420183+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.556814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"84'11","reported_seq":153,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.417118+0000","last_change":"2024-03-11T21:25:58.333052+0000","last_active":"2024-03-11T21:26:10.417118+0000","last_peered":"2024-03-11T21:26:10.417118+0000","last_clean":"2024-03-11T21:26:10.417118+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:26:10.417118+0000","last_undegraded":"2024-03-11T21:26:10.417118+0000","last_fullsized":"2024-03-11T21:26:10.417118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:58.332816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.332816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.183920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594734+0000","last_change":"2024-03-11T21:24:47.092913+0000","last_active":"2024-03-11T21:26:10.594734+0000","last_peered":"2024-03-11T21:26:10.594734+0000","last_clean":"2024-03-11T21:26:10.594734+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:26:10.594734+0000","last_undegraded":"2024-03-11T21:26:10.594734+0000","last_fullsized":"2024-03-11T21:26:10.594734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:47.092662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:47.092662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:16.057958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593589+0000","last_change":"2024-03-11T21:24:39.492065+0000","last_active":"2024-03-11T21:26:10.593589+0000","last_peered":"2024-03-11T21:26:10.593589+0000","last_clean":"2024-03-11T21:26:10.593589+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:26:10.593589+0000","last_undegraded":"2024-03-11T21:26:10.593589+0000","last_fullsized":"2024-03-11T21:26:10.593589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:39.491841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:24:39.491841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":150,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410587+0000","last_change":"2024-03-11T21:25:30.974808+0000","last_active":"2024-03-11T21:26:10.410587+0000","last_peered":"2024-03-11T21:26:10.410587+0000","last_clean":"2024-03-11T21:26:10.410587+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:26:10.410587+0000","last_undegraded":"2024-03-11T21:26:10.410587+0000","last_fullsized":"2024-03-11T21:26:10.410587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.974544+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.198159+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.418663+0000","last_change":"2024-03-11T21:24:46.441508+0000","last_active":"2024-03-11T21:26:10.418663+0000","last_peered":"2024-03-11T21:26:10.418663+0000","last_clean":"2024-03-11T21:26:10.418663+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:26:10.418663+0000","last_undegraded":"2024-03-11T21:26:10.418663+0000","last_fullsized":"2024-03-11T21:26:10.418663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:46.441282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:24:46.441282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:16.384824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676961+0000","last_change":"2024-03-11T21:26:04.128265+0000","last_active":"2024-03-11T21:26:10.676961+0000","last_peered":"2024-03-11T21:26:10.676961+0000","last_clean":"2024-03-11T21:26:10.676961+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:26:10.676961+0000","last_undegraded":"2024-03-11T21:26:10.676961+0000","last_fullsized":"2024-03-11T21:26:10.676961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.128031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.128031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.667989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.418939+0000","last_change":"2024-03-11T21:25:56.736951+0000","last_active":"2024-03-11T21:26:10.418939+0000","last_peered":"2024-03-11T21:26:10.418939+0000","last_clean":"2024-03-11T21:26:10.418939+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:26:10.418939+0000","last_undegraded":"2024-03-11T21:26:10.418939+0000","last_fullsized":"2024-03-11T21:26:10.418939+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:25:56.736721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.736721+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.841976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.933800+0000","last_change":"2024-03-11T21:25:47.994296+0000","last_active":"2024-03-11T21:26:10.933800+0000","last_peered":"2024-03-11T21:26:10.933800+0000","last_clean":"2024-03-11T21:26:10.933800+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:26:10.933800+0000","last_undegraded":"2024-03-11T21:26:10.933800+0000","last_fullsized":"2024-03-11T21:26:10.933800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.994008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.994008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.859169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934118+0000","last_change":"2024-03-11T21:25:58.022948+0000","last_active":"2024-03-11T21:26:10.934118+0000","last_peered":"2024-03-11T21:26:10.934118+0000","last_clean":"2024-03-11T21:26:10.934118+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:26:10.934118+0000","last_undegraded":"2024-03-11T21:26:10.934118+0000","last_fullsized":"2024-03-11T21:26:10.934118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.022737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.022737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.306838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595277+0000","last_change":"2024-03-11T21:26:03.430155+0000","last_active":"2024-03-11T21:26:10.595277+0000","last_peered":"2024-03-11T21:26:10.595277+0000","last_clean":"2024-03-11T21:26:10.595277+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:26:10.595277+0000","last_undegraded":"2024-03-11T21:26:10.595277+0000","last_fullsized":"2024-03-11T21:26:10.595277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:03.429945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.429945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.219815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594258+0000","last_change":"2024-03-11T21:26:08.149166+0000","last_active":"2024-03-11T21:26:10.594258+0000","last_peered":"2024-03-11T21:26:10.594258+0000","last_clean":"2024-03-11T21:26:10.594258+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:26:10.594258+0000","last_undegraded":"2024-03-11T21:26:10.594258+0000","last_fullsized":"2024-03-11T21:26:10.594258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:08.148915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.148915+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.413377+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593760+0000","last_change":"2024-03-11T21:25:32.395539+0000","last_active":"2024-03-11T21:26:10.593760+0000","last_peered":"2024-03-11T21:26:10.593760+0000","last_clean":"2024-03-11T21:26:10.593760+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:26:10.593760+0000","last_undegraded":"2024-03-11T21:26:10.593760+0000","last_fullsized":"2024-03-11T21:26:10.593760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.395316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.615969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":363,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.675781+0000","last_change":"2024-03-11T21:25:21.288916+0000","last_active":"2024-03-11T21:26:10.675781+0000","last_peered":"2024-03-11T21:26:10.675781+0000","last_clean":"2024-03-11T21:26:10.675781+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:26:10.675781+0000","last_undegraded":"2024-03-11T21:26:10.675781+0000","last_fullsized":"2024-03-11T21:26:10.675781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:25:21.288622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:25:21.288622+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:33.339389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677091+0000","last_change":"2024-03-11T21:25:02.193912+0000","last_active":"2024-03-11T21:26:10.677091+0000","last_peered":"2024-03-11T21:26:10.677091+0000","last_clean":"2024-03-11T21:26:10.677091+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:26:10.677091+0000","last_undegraded":"2024-03-11T21:26:10.677091+0000","last_fullsized":"2024-03-11T21:26:10.677091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:02.193681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:02.193681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:31.088279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.602417+0000","last_change":"2024-03-11T21:26:10.602526+0000","last_active":"2024-03-11T21:26:10.602417+0000","last_peered":"2024-03-11T21:26:10.602417+0000","last_clean":"2024-03-11T21:26:10.602417+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:26:10.602417+0000","last_undegraded":"2024-03-11T21:26:10.602417+0000","last_fullsized":"2024-03-11T21:26:10.602417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.602230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.602230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:36.316832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":119,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.415036+0000","last_change":"2024-03-11T21:24:47.440572+0000","last_active":"2024-03-11T21:26:10.415036+0000","last_peered":"2024-03-11T21:26:10.415036+0000","last_clean":"2024-03-11T21:26:10.415036+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:26:10.415036+0000","last_undegraded":"2024-03-11T21:26:10.415036+0000","last_fullsized":"2024-03-11T21:26:10.415036+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:24:47.440317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:47.440317+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.675599+0000","last_change":"2024-03-11T21:25:43.021158+0000","last_active":"2024-03-11T21:26:10.675599+0000","last_peered":"2024-03-11T21:26:10.675599+0000","last_clean":"2024-03-11T21:26:10.675599+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:26:10.675599+0000","last_undegraded":"2024-03-11T21:26:10.675599+0000","last_fullsized":"2024-03-11T21:26:10.675599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.020950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.020950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.379154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595008+0000","last_change":"2024-03-11T21:25:52.415680+0000","last_active":"2024-03-11T21:26:10.595008+0000","last_peered":"2024-03-11T21:26:10.595008+0000","last_clean":"2024-03-11T21:26:10.595008+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:26:10.595008+0000","last_undegraded":"2024-03-11T21:26:10.595008+0000","last_fullsized":"2024-03-11T21:26:10.595008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.415468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.415468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.982658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.418335+0000","last_change":"2024-03-11T21:26:04.760536+0000","last_active":"2024-03-11T21:26:10.418335+0000","last_peered":"2024-03-11T21:26:10.418335+0000","last_clean":"2024-03-11T21:26:10.418335+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:26:10.418335+0000","last_undegraded":"2024-03-11T21:26:10.418335+0000","last_fullsized":"2024-03-11T21:26:10.418335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.760262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.760262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.318865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677255+0000","last_change":"2024-03-11T21:25:59.200564+0000","last_active":"2024-03-11T21:26:10.677255+0000","last_peered":"2024-03-11T21:26:10.677255+0000","last_clean":"2024-03-11T21:26:10.677255+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:26:10.677255+0000","last_undegraded":"2024-03-11T21:26:10.677255+0000","last_fullsized":"2024-03-11T21:26:10.677255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.200270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.200270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.037981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":194,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420118+0000","last_change":"2024-03-11T21:25:37.669639+0000","last_active":"2024-03-11T21:26:10.420118+0000","last_peered":"2024-03-11T21:26:10.420118+0000","last_clean":"2024-03-11T21:26:10.420118+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:26:10.420118+0000","last_undegraded":"2024-03-11T21:26:10.420118+0000","last_fullsized":"2024-03-11T21:26:10.420118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:25:37.669221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.669221+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.744954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.985168+0000","last_change":"2024-03-11T21:26:08.930659+0000","last_active":"2024-03-11T21:26:10.985168+0000","last_peered":"2024-03-11T21:26:10.985168+0000","last_clean":"2024-03-11T21:26:10.985168+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:26:10.985168+0000","last_undegraded":"2024-03-11T21:26:10.985168+0000","last_fullsized":"2024-03-11T21:26:10.985168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:08.930415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.930415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.664164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410529+0000","last_change":"2024-03-11T21:26:02.037288+0000","last_active":"2024-03-11T21:26:10.410529+0000","last_peered":"2024-03-11T21:26:10.410529+0000","last_clean":"2024-03-11T21:26:10.410529+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:26:10.410529+0000","last_undegraded":"2024-03-11T21:26:10.410529+0000","last_fullsized":"2024-03-11T21:26:10.410529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.939102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420550+0000","last_change":"2024-03-11T21:25:49.638326+0000","last_active":"2024-03-11T21:26:10.420550+0000","last_peered":"2024-03-11T21:26:10.420550+0000","last_clean":"2024-03-11T21:26:10.420550+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:26:10.420550+0000","last_undegraded":"2024-03-11T21:26:10.420550+0000","last_fullsized":"2024-03-11T21:26:10.420550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:49.638061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.638061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.499420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595220+0000","last_change":"2024-03-11T21:24:36.087831+0000","last_active":"2024-03-11T21:26:10.595220+0000","last_peered":"2024-03-11T21:26:10.595220+0000","last_clean":"2024-03-11T21:26:10.595220+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:26:10.595220+0000","last_undegraded":"2024-03-11T21:26:10.595220+0000","last_fullsized":"2024-03-11T21:26:10.595220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:24:36.087707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:24:36.087707+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":285,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:12.033015+0000","last_change":"2024-03-11T21:25:50.006793+0000","last_active":"2024-03-11T21:26:12.033015+0000","last_peered":"2024-03-11T21:26:12.033015+0000","last_clean":"2024-03-11T21:26:12.033015+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:26:12.033015+0000","last_undegraded":"2024-03-11T21:26:12.033015+0000","last_fullsized":"2024-03-11T21:26:12.033015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:25:50.006698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.006698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.269418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":171,"num_read_kb":166,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410196+0000","last_change":"2024-03-11T21:25:57.992113+0000","last_active":"2024-03-11T21:26:10.410196+0000","last_peered":"2024-03-11T21:26:10.410196+0000","last_clean":"2024-03-11T21:26:10.410196+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:26:10.410196+0000","last_undegraded":"2024-03-11T21:26:10.410196+0000","last_fullsized":"2024-03-11T21:26:10.410196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.991803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.991803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.426825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676253+0000","last_change":"2024-03-11T21:25:34.059053+0000","last_active":"2024-03-11T21:26:10.676253+0000","last_peered":"2024-03-11T21:26:10.676253+0000","last_clean":"2024-03-11T21:26:10.676253+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:26:10.676253+0000","last_undegraded":"2024-03-11T21:26:10.676253+0000","last_fullsized":"2024-03-11T21:26:10.676253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.058815+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.431730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"77'1","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.985927+0000","last_change":"2024-03-11T21:25:43.867607+0000","last_active":"2024-03-11T21:26:10.985927+0000","last_peered":"2024-03-11T21:26:10.985927+0000","last_clean":"2024-03-11T21:26:10.985927+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:26:10.985927+0000","last_undegraded":"2024-03-11T21:26:10.985927+0000","last_fullsized":"2024-03-11T21:26:10.985927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:43.867377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.867377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.362412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.416092+0000","last_change":"2024-03-11T21:25:22.361282+0000","last_active":"2024-03-11T21:26:10.416092+0000","last_peered":"2024-03-11T21:26:10.416092+0000","last_clean":"2024-03-11T21:26:10.416092+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:26:10.416092+0000","last_undegraded":"2024-03-11T21:26:10.416092+0000","last_fullsized":"2024-03-11T21:26:10.416092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:22.360881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.360881+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:27.226640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410481+0000","last_change":"2024-03-11T21:25:47.898533+0000","last_active":"2024-03-11T21:26:10.410481+0000","last_peered":"2024-03-11T21:26:10.410481+0000","last_clean":"2024-03-11T21:26:10.410481+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:26:10.410481+0000","last_undegraded":"2024-03-11T21:26:10.410481+0000","last_fullsized":"2024-03-11T21:26:10.410481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.898322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.898322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.986939+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":150,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.592992+0000","last_change":"2024-03-11T21:25:45.984353+0000","last_active":"2024-03-11T21:26:10.592992+0000","last_peered":"2024-03-11T21:26:10.592992+0000","last_clean":"2024-03-11T21:26:10.592992+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:26:10.592992+0000","last_undegraded":"2024-03-11T21:26:10.592992+0000","last_fullsized":"2024-03-11T21:26:10.592992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:45.984098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.984098+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.447890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593985+0000","last_change":"2024-03-11T21:24:38.145631+0000","last_active":"2024-03-11T21:26:10.593985+0000","last_peered":"2024-03-11T21:26:10.593985+0000","last_clean":"2024-03-11T21:26:10.593985+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:26:10.593985+0000","last_undegraded":"2024-03-11T21:26:10.593985+0000","last_fullsized":"2024-03-11T21:26:10.593985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:38.145410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:24:38.145410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595121+0000","last_change":"2024-03-11T21:25:58.434898+0000","last_active":"2024-03-11T21:26:10.595121+0000","last_peered":"2024-03-11T21:26:10.595121+0000","last_clean":"2024-03-11T21:26:10.595121+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:26:10.595121+0000","last_undegraded":"2024-03-11T21:26:10.595121+0000","last_fullsized":"2024-03-11T21:26:10.595121+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.434794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.434794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.675782+0000","last_change":"2024-03-11T21:25:52.032818+0000","last_active":"2024-03-11T21:26:10.675782+0000","last_peered":"2024-03-11T21:26:10.675782+0000","last_clean":"2024-03-11T21:26:10.675782+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:26:10.675782+0000","last_undegraded":"2024-03-11T21:26:10.675782+0000","last_fullsized":"2024-03-11T21:26:10.675782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.032606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.032606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.435492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.410217+0000","last_change":"2024-03-11T21:24:49.278410+0000","last_active":"2024-03-11T21:26:10.410217+0000","last_peered":"2024-03-11T21:26:10.410217+0000","last_clean":"2024-03-11T21:26:10.410217+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:26:10.410217+0000","last_undegraded":"2024-03-11T21:26:10.410217+0000","last_fullsized":"2024-03-11T21:26:10.410217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:24:49.278162+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:13.051996+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.595118+0000","last_change":"2024-03-11T21:25:41.358918+0000","last_active":"2024-03-11T21:26:10.595118+0000","last_peered":"2024-03-11T21:26:10.595118+0000","last_clean":"2024-03-11T21:26:10.595118+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:26:10.595118+0000","last_undegraded":"2024-03-11T21:26:10.595118+0000","last_fullsized":"2024-03-11T21:26:10.595118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.358679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.152690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":150,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.411771+0000","last_change":"2024-03-11T21:25:57.003681+0000","last_active":"2024-03-11T21:26:10.411771+0000","last_peered":"2024-03-11T21:26:10.411771+0000","last_clean":"2024-03-11T21:26:10.411771+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:26:10.411771+0000","last_undegraded":"2024-03-11T21:26:10.411771+0000","last_fullsized":"2024-03-11T21:26:10.411771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:57.003432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.003432+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.840351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.419141+0000","last_change":"2024-03-11T21:25:34.710182+0000","last_active":"2024-03-11T21:26:10.419141+0000","last_peered":"2024-03-11T21:26:10.419141+0000","last_clean":"2024-03-11T21:26:10.419141+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:26:10.419141+0000","last_undegraded":"2024-03-11T21:26:10.419141+0000","last_fullsized":"2024-03-11T21:26:10.419141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:34.709964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.709964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.742096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":131,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593319+0000","last_change":"2024-03-11T21:25:40.954211+0000","last_active":"2024-03-11T21:26:10.593319+0000","last_peered":"2024-03-11T21:26:10.593319+0000","last_clean":"2024-03-11T21:26:10.593319+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:26:10.593319+0000","last_undegraded":"2024-03-11T21:26:10.593319+0000","last_fullsized":"2024-03-11T21:26:10.593319+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:40.953969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.953969+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.832583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.412148+0000","last_change":"2024-03-11T21:26:05.072908+0000","last_active":"2024-03-11T21:26:10.412148+0000","last_peered":"2024-03-11T21:26:10.412148+0000","last_clean":"2024-03-11T21:26:10.412148+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:26:10.412148+0000","last_undegraded":"2024-03-11T21:26:10.412148+0000","last_fullsized":"2024-03-11T21:26:10.412148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.072676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.072676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.590231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":151,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934091+0000","last_change":"2024-03-11T21:25:40.987597+0000","last_active":"2024-03-11T21:26:10.934091+0000","last_peered":"2024-03-11T21:26:10.934091+0000","last_clean":"2024-03-11T21:26:10.934091+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:26:10.934091+0000","last_undegraded":"2024-03-11T21:26:10.934091+0000","last_fullsized":"2024-03-11T21:26:10.934091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:25:40.987394+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.987394+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.827259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.419565+0000","last_change":"2024-03-11T21:26:07.649240+0000","last_active":"2024-03-11T21:26:10.419565+0000","last_peered":"2024-03-11T21:26:10.419565+0000","last_clean":"2024-03-11T21:26:10.419565+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:26:10.419565+0000","last_undegraded":"2024-03-11T21:26:10.419565+0000","last_fullsized":"2024-03-11T21:26:10.419565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:07.649133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:26:07.649133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.066187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594014+0000","last_change":"2024-03-11T21:25:29.024987+0000","last_active":"2024-03-11T21:26:10.594014+0000","last_peered":"2024-03-11T21:26:10.594014+0000","last_clean":"2024-03-11T21:26:10.594014+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:26:10.594014+0000","last_undegraded":"2024-03-11T21:26:10.594014+0000","last_fullsized":"2024-03-11T21:26:10.594014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:29.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.024617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.368987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934247+0000","last_change":"2024-03-11T21:25:32.119545+0000","last_active":"2024-03-11T21:26:10.934247+0000","last_peered":"2024-03-11T21:26:10.934247+0000","last_clean":"2024-03-11T21:26:10.934247+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:26:10.934247+0000","last_undegraded":"2024-03-11T21:26:10.934247+0000","last_fullsized":"2024-03-11T21:26:10.934247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.119316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.119316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.928370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:12.573439+0000","last_change":"2024-03-11T21:26:12.573500+0000","last_active":"2024-03-11T21:26:12.573439+0000","last_peered":"2024-03-11T21:26:12.573439+0000","last_clean":"2024-03-11T21:26:12.573439+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:26:12.573439+0000","last_undegraded":"2024-03-11T21:26:12.573439+0000","last_fullsized":"2024-03-11T21:26:12.573439+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.573292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.573292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.647582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676701+0000","last_change":"2024-03-11T21:25:28.167250+0000","last_active":"2024-03-11T21:26:10.676701+0000","last_peered":"2024-03-11T21:26:10.676701+0000","last_clean":"2024-03-11T21:26:10.676701+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:26:10.676701+0000","last_undegraded":"2024-03-11T21:26:10.676701+0000","last_fullsized":"2024-03-11T21:26:10.676701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:28.167038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:28.167038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:41.505025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.984849+0000","last_change":"2024-03-11T21:26:02.945068+0000","last_active":"2024-03-11T21:26:10.984849+0000","last_peered":"2024-03-11T21:26:10.984849+0000","last_clean":"2024-03-11T21:26:10.984849+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:26:10.984849+0000","last_undegraded":"2024-03-11T21:26:10.984849+0000","last_fullsized":"2024-03-11T21:26:10.984849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:26:02.944783+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944783+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.686877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:12.157324+0000","last_change":"2024-03-11T21:26:12.157386+0000","last_active":"2024-03-11T21:26:12.157324+0000","last_peered":"2024-03-11T21:26:12.157324+0000","last_clean":"2024-03-11T21:26:12.157324+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:26:12.157324+0000","last_undegraded":"2024-03-11T21:26:12.157324+0000","last_fullsized":"2024-03-11T21:26:12.157324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.157160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.157160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.057339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.933955+0000","last_change":"2024-03-11T21:25:30.130283+0000","last_active":"2024-03-11T21:26:10.933955+0000","last_peered":"2024-03-11T21:26:10.933955+0000","last_clean":"2024-03-11T21:26:10.933955+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:26:10.933955+0000","last_undegraded":"2024-03-11T21:26:10.933955+0000","last_fullsized":"2024-03-11T21:26:10.933955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.130050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.236058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934348+0000","last_change":"2024-03-11T21:25:44.013839+0000","last_active":"2024-03-11T21:26:10.934348+0000","last_peered":"2024-03-11T21:26:10.934348+0000","last_clean":"2024-03-11T21:26:10.934348+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:26:10.934348+0000","last_undegraded":"2024-03-11T21:26:10.934348+0000","last_fullsized":"2024-03-11T21:26:10.934348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:44.013746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:44.013746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.618233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677689+0000","last_change":"2024-03-11T21:24:41.173004+0000","last_active":"2024-03-11T21:26:10.677689+0000","last_peered":"2024-03-11T21:26:10.677689+0000","last_clean":"2024-03-11T21:26:10.677689+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:26:10.677689+0000","last_undegraded":"2024-03-11T21:26:10.677689+0000","last_fullsized":"2024-03-11T21:26:10.677689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:24:41.172690+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:11.021661+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.414782+0000","last_change":"2024-03-11T21:24:53.521553+0000","last_active":"2024-03-11T21:26:10.414782+0000","last_peered":"2024-03-11T21:26:10.414782+0000","last_clean":"2024-03-11T21:26:10.414782+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:26:10.414782+0000","last_undegraded":"2024-03-11T21:26:10.414782+0000","last_fullsized":"2024-03-11T21:26:10.414782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:24:53.521326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.933894+0000","last_change":"2024-03-11T21:25:31.137161+0000","last_active":"2024-03-11T21:26:10.933894+0000","last_peered":"2024-03-11T21:26:10.933894+0000","last_clean":"2024-03-11T21:26:10.933894+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:26:10.933894+0000","last_undegraded":"2024-03-11T21:26:10.933894+0000","last_fullsized":"2024-03-11T21:26:10.933894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:31.137035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.137035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.158341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.420480+0000","last_change":"2024-03-11T21:25:30.696848+0000","last_active":"2024-03-11T21:26:10.420480+0000","last_peered":"2024-03-11T21:26:10.420480+0000","last_clean":"2024-03-11T21:26:10.420480+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:26:10.420480+0000","last_undegraded":"2024-03-11T21:26:10.420480+0000","last_fullsized":"2024-03-11T21:26:10.420480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.696738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.696738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.966425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.594455+0000","last_change":"2024-03-11T21:25:45.376793+0000","last_active":"2024-03-11T21:26:10.594455+0000","last_peered":"2024-03-11T21:26:10.594455+0000","last_clean":"2024-03-11T21:26:10.594455+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:26:10.594455+0000","last_undegraded":"2024-03-11T21:26:10.594455+0000","last_fullsized":"2024-03-11T21:26:10.594455+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:45.376408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.376408+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.150185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.593286+0000","last_change":"2024-03-11T21:26:05.391058+0000","last_active":"2024-03-11T21:26:10.593286+0000","last_peered":"2024-03-11T21:26:10.593286+0000","last_clean":"2024-03-11T21:26:10.593286+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:26:10.593286+0000","last_undegraded":"2024-03-11T21:26:10.593286+0000","last_fullsized":"2024-03-11T21:26:10.593286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.390850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.390850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.856109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.934257+0000","last_change":"2024-03-11T21:25:22.246229+0000","last_active":"2024-03-11T21:26:10.934257+0000","last_peered":"2024-03-11T21:26:10.934257+0000","last_clean":"2024-03-11T21:26:10.934257+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:26:10.934257+0000","last_undegraded":"2024-03-11T21:26:10.934257+0000","last_fullsized":"2024-03-11T21:26:10.934257+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:22.245940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.245940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.310931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.677739+0000","last_change":"2024-03-11T21:25:18.230459+0000","last_active":"2024-03-11T21:26:10.677739+0000","last_peered":"2024-03-11T21:26:10.677739+0000","last_clean":"2024-03-11T21:26:10.677739+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:26:10.677739+0000","last_undegraded":"2024-03-11T21:26:10.677739+0000","last_fullsized":"2024-03-11T21:26:10.677739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:18.230036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:18.230036+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:30.431832+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":142,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-11T21:26:10.676186+0000","last_change":"2024-03-11T21:25:12.165879+0000","last_active":"2024-03-11T21:26:10.676186+0000","last_peered":"2024-03-11T21:26:10.676186+0000","last_clean":"2024-03-11T21:26:10.676186+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:26:10.676186+0000","last_undegraded":"2024-03-11T21:26:10.676186+0000","last_fullsized":"2024-03-11T21:26:10.676186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:12.165710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:12.165710+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:24.012554+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":171,"num_read_kb":166,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841585,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36652,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708500,"statfs":{"total":95995035648,"available":95957504000,"internally_reserved":0,"allocated":1343488,"data_stored":974361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1310720,"data_stored":972141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":724992,"data_stored":381655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36628,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708524,"statfs":{"total":95995035648,"available":95957528576,"internally_reserved":0,"allocated":1323008,"data_stored":973258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378724,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31884,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713268,"statfs":{"total":95995035648,"available":95962386432,"internally_reserved":0,"allocated":720896,"data_stored":382233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":381711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855438,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27756,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717396,"statfs":{"total":95995035648,"available":95966613504,"internally_reserved":0,"allocated":753664,"data_stored":383180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495684,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31872,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713280,"statfs":{"total":95995035648,"available":95962398720,"internally_reserved":0,"allocated":712704,"data_stored":381693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:26:14.054 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:26:14.054 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:26:14.054 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:26:14.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.108632+0000 mgr.y (mgr.14550) 118 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: audit 2024-03-11T21:26:13.141801+0000 mgr.y (mgr.14550) 119 : audit 0 from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.332694+0000 osd.0 (osd.0) 75 : cluster 0 3.1 scrub starts 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.337078+0000 osd.0 (osd.0) 76 : cluster 0 3.1 scrub ok 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.485222+0000 osd.5 (osd.5) 87 : cluster 0 2.2 scrub starts 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.487348+0000 osd.5 (osd.5) 88 : cluster 0 2.2 scrub ok 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.567334+0000 osd.1 (osd.1) 83 : cluster 0 5.19 scrub starts 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:13.569884+0000 osd.1 (osd.1) 84 : cluster 0 5.19 scrub ok 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.122205+0000 osd.7 (osd.7) 89 : cluster 0 3.d scrub starts 2024-03-11T21:26:14.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.127393+0000 osd.7 (osd.7) 90 : cluster 0 3.d scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.195170+0000 osd.3 (osd.3) 87 : cluster 0 6.c scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.197757+0000 osd.3 (osd.3) 88 : cluster 0 6.c scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.108632+0000 mgr.y (mgr.14550) 118 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: audit 2024-03-11T21:26:13.141801+0000 mgr.y (mgr.14550) 119 : audit 0 from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.332694+0000 osd.0 (osd.0) 75 : cluster 0 3.1 scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.337078+0000 osd.0 (osd.0) 76 : cluster 0 3.1 scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.485222+0000 osd.5 (osd.5) 87 : cluster 0 2.2 scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.487348+0000 osd.5 (osd.5) 88 : cluster 0 2.2 scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.567334+0000 osd.1 (osd.1) 83 : cluster 0 5.19 scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:13.569884+0000 osd.1 (osd.1) 84 : cluster 0 5.19 scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.122205+0000 osd.7 (osd.7) 89 : cluster 0 3.d scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.127393+0000 osd.7 (osd.7) 90 : cluster 0 3.d scrub ok 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.195170+0000 osd.3 (osd.3) 87 : cluster 0 6.c scrub starts 2024-03-11T21:26:14.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:14 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.197757+0000 osd.3 (osd.3) 88 : cluster 0 6.c scrub ok 2024-03-11T21:26:15.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.108632+0000 mgr.y (mgr.14550) 118 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:26:15.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: audit 2024-03-11T21:26:13.141801+0000 mgr.y (mgr.14550) 119 : audit 0 from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:15.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.332694+0000 osd.0 (osd.0) 75 : cluster 0 3.1 scrub starts 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.337078+0000 osd.0 (osd.0) 76 : cluster 0 3.1 scrub ok 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.485222+0000 osd.5 (osd.5) 87 : cluster 0 2.2 scrub starts 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.487348+0000 osd.5 (osd.5) 88 : cluster 0 2.2 scrub ok 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.567334+0000 osd.1 (osd.1) 83 : cluster 0 5.19 scrub starts 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:13.569884+0000 osd.1 (osd.1) 84 : cluster 0 5.19 scrub ok 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.122205+0000 osd.7 (osd.7) 89 : cluster 0 3.d scrub starts 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.127393+0000 osd.7 (osd.7) 90 : cluster 0 3.d scrub ok 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.195170+0000 osd.3 (osd.3) 87 : cluster 0 6.c scrub starts 2024-03-11T21:26:15.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:14 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.197757+0000 osd.3 (osd.3) 88 : cluster 0 6.c scrub ok 2024-03-11T21:26:15.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:15 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.590952+0000 osd.1 (osd.1) 85 : cluster 0 5.12 scrub starts 2024-03-11T21:26:15.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:15 smithi143 bash[21512]: cluster 2024-03-11T21:26:14.593248+0000 osd.1 (osd.1) 86 : cluster 0 5.12 scrub ok 2024-03-11T21:26:15.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:15 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.590952+0000 osd.1 (osd.1) 85 : cluster 0 5.12 scrub starts 2024-03-11T21:26:15.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:15 smithi143 bash[29284]: cluster 2024-03-11T21:26:14.593248+0000 osd.1 (osd.1) 86 : cluster 0 5.12 scrub ok 2024-03-11T21:26:16.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:15 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.590952+0000 osd.1 (osd.1) 85 : cluster 0 5.12 scrub starts 2024-03-11T21:26:16.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:15 smithi203 bash[23654]: cluster 2024-03-11T21:26:14.593248+0000 osd.1 (osd.1) 86 : cluster 0 5.12 scrub ok 2024-03-11T21:26:16.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[21512]: cluster 2024-03-11T21:26:15.109291+0000 mgr.y (mgr.14550) 120 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:16.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[21512]: cluster 2024-03-11T21:26:15.352359+0000 osd.0 (osd.0) 77 : cluster 0 6.9 scrub starts 2024-03-11T21:26:16.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[21512]: cluster 2024-03-11T21:26:15.354666+0000 osd.0 (osd.0) 78 : cluster 0 6.9 scrub ok 2024-03-11T21:26:16.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[29284]: cluster 2024-03-11T21:26:15.109291+0000 mgr.y (mgr.14550) 120 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:16.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[29284]: cluster 2024-03-11T21:26:15.352359+0000 osd.0 (osd.0) 77 : cluster 0 6.9 scrub starts 2024-03-11T21:26:16.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:16 smithi143 bash[29284]: cluster 2024-03-11T21:26:15.354666+0000 osd.0 (osd.0) 78 : cluster 0 6.9 scrub ok 2024-03-11T21:26:17.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:16 smithi203 bash[23654]: cluster 2024-03-11T21:26:15.109291+0000 mgr.y (mgr.14550) 120 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:17.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:16 smithi203 bash[23654]: cluster 2024-03-11T21:26:15.352359+0000 osd.0 (osd.0) 77 : cluster 0 6.9 scrub starts 2024-03-11T21:26:17.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:16 smithi203 bash[23654]: cluster 2024-03-11T21:26:15.354666+0000 osd.0 (osd.0) 78 : cluster 0 6.9 scrub ok 2024-03-11T21:26:18.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:18.837 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:18.859 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:18 smithi143 bash[21512]: cluster 2024-03-11T21:26:17.109968+0000 mgr.y (mgr.14550) 121 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:18.860 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:18 smithi143 bash[29284]: cluster 2024-03-11T21:26:17.109968+0000 mgr.y (mgr.14550) 121 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:19.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:18 smithi203 bash[23654]: cluster 2024-03-11T21:26:17.109968+0000 mgr.y (mgr.14550) 121 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:19.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[29284]: cluster 2024-03-11T21:26:17.922835+0000 osd.4 (osd.4) 69 : cluster 0 3.9 scrub starts 2024-03-11T21:26:19.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[29284]: cluster 2024-03-11T21:26:17.926850+0000 osd.4 (osd.4) 70 : cluster 0 3.9 scrub ok 2024-03-11T21:26:19.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[29284]: audit 2024-03-11T21:26:18.179660+0000 mgr.y (mgr.14550) 122 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:19.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[21512]: cluster 2024-03-11T21:26:17.922835+0000 osd.4 (osd.4) 69 : cluster 0 3.9 scrub starts 2024-03-11T21:26:19.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[21512]: cluster 2024-03-11T21:26:17.926850+0000 osd.4 (osd.4) 70 : cluster 0 3.9 scrub ok 2024-03-11T21:26:19.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:19 smithi143 bash[21512]: audit 2024-03-11T21:26:18.179660+0000 mgr.y (mgr.14550) 122 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:19 smithi203 bash[23654]: cluster 2024-03-11T21:26:17.922835+0000 osd.4 (osd.4) 69 : cluster 0 3.9 scrub starts 2024-03-11T21:26:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:19 smithi203 bash[23654]: cluster 2024-03-11T21:26:17.926850+0000 osd.4 (osd.4) 70 : cluster 0 3.9 scrub ok 2024-03-11T21:26:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:19 smithi203 bash[23654]: audit 2024-03-11T21:26:18.179660+0000 mgr.y (mgr.14550) 122 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:20.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[29284]: cluster 2024-03-11T21:26:19.111427+0000 mgr.y (mgr.14550) 123 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:20.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[29284]: cluster 2024-03-11T21:26:19.923693+0000 osd.6 (osd.6) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[29284]: cluster 2024-03-11T21:26:19.923800+0000 osd.6 (osd.6) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.275257+0000 osd.3 (osd.3) 89 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.277134+0000 osd.3 (osd.3) 90 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[21512]: cluster 2024-03-11T21:26:19.111427+0000 mgr.y (mgr.14550) 123 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[21512]: cluster 2024-03-11T21:26:19.923693+0000 osd.6 (osd.6) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[21512]: cluster 2024-03-11T21:26:19.923800+0000 osd.6 (osd.6) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.275257+0000 osd.3 (osd.3) 89 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:26:20.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:20 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.277134+0000 osd.3 (osd.3) 90 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:26:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:20 smithi203 bash[23654]: cluster 2024-03-11T21:26:19.111427+0000 mgr.y (mgr.14550) 123 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:20 smithi203 bash[23654]: cluster 2024-03-11T21:26:19.923693+0000 osd.6 (osd.6) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:20 smithi203 bash[23654]: cluster 2024-03-11T21:26:19.923800+0000 osd.6 (osd.6) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:20 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.275257+0000 osd.3 (osd.3) 89 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:26:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:20 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.277134+0000 osd.3 (osd.3) 90 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.448189+0000 osd.0 (osd.0) 79 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.454733+0000 osd.0 (osd.0) 80 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.620671+0000 mon.a (mon.0) 876 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.984010+0000 osd.7 (osd.7) 91 : cluster 0 5.10 scrub starts 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[29284]: cluster 2024-03-11T21:26:20.986523+0000 osd.7 (osd.7) 92 : cluster 0 5.10 scrub ok 2024-03-11T21:26:21.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.448189+0000 osd.0 (osd.0) 79 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:26:21.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.454733+0000 osd.0 (osd.0) 80 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:26:21.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.620671+0000 mon.a (mon.0) 876 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:26:21.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.984010+0000 osd.7 (osd.7) 91 : cluster 0 5.10 scrub starts 2024-03-11T21:26:21.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:21 smithi143 bash[21512]: cluster 2024-03-11T21:26:20.986523+0000 osd.7 (osd.7) 92 : cluster 0 5.10 scrub ok 2024-03-11T21:26:22.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/3209859880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100cf0 msgr2=0x7f6ec010ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:22.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/3209859880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100cf0 0x7f6ec010ecc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b0d0 tx=0x7f6eb800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/3209859880 shutdown_connections 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/3209859880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100cf0 0x7f6ec010ecc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/3209859880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0106e00 0x7f6ec01071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/3209859880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0100330 0x7f6ec01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/3209859880 >> 172.21.15.143:0/3209859880 conn(0x7f6ec00772e0 msgr2=0x7f6ec00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:22.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/3209859880 shutdown_connections 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/3209859880 wait complete. 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 librados: starting msgr at 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 librados: starting objecter 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 Processor -- start 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- start start 2024-03-11T21:26:22.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 librados: setting wanted keys 2024-03-11T21:26:22.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 librados: calling monclient init 2024-03-11T21:26:22.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 0x7f6ec0101510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:22.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0100cf0 0x7f6ec0101a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:22.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:22.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec0111720 con 0x7f6ec0100330 2024-03-11T21:26:22.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01115a0 con 0x7f6ec0100cf0 2024-03-11T21:26:22.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01118a0 con 0x7f6ec0106e00 2024-03-11T21:26:22.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:22.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:53200/0 (socket says 172.21.15.143:53200) 2024-03-11T21:26:22.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec4e4f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 0x7f6ec0101510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 -- 172.21.15.143:0/536519126 learned_addr learned my addr 172.21.15.143:0/536519126 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebffff700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0100cf0 0x7f6ec0101a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 -- 172.21.15.143:0/536519126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0100cf0 msgr2=0x7f6ec0101a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0100cf0 0x7f6ec0101a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 -- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 msgr2=0x7f6ec0101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:22.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 0x7f6ec0101510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 -- 172.21.15.143:0/536519126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec01a6120 con 0x7f6ec0106e00 2024-03-11T21:26:22.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec4e4f700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 0x7f6ec0101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:22.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec5650700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8006070 tx=0x7f6eb8007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:22.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebdffb700 10 client.?.objecter ms_handle_connect 0x7f6ec0106e00 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebdffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8021020 con 0x7f6ec0106e00 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6eb8004c50 con 0x7f6ec0106e00 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8005050 con 0x7f6ec0106e00 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec01a63b0 con 0x7f6ec0106e00 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 10 client.24602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:22.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec01a68b0 con 0x7f6ec0106e00 2024-03-11T21:26:22.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 librados: init done 2024-03-11T21:26:22.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.025+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec01029f0 con 0x7f6ec0106e00 2024-03-11T21:26:22.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f6eb8024020 con 0x7f6ec0106e00 2024-03-11T21:26:22.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 0x7f6ea8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(119..119 src has 66..119) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f6eb800f020 con 0x7f6ec0106e00 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 10 client.24602.objecter ms_dispatch 0x7f6ec0001550 osd_map(119..119 src has 66..119) v4 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 3 client.24602.objecter handle_osd_map got epochs [119,119] > 0 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 3 client.24602.objecter handle_osd_map decoding full epoch 119 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ec4e4f700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 0x7f6ea8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 20 client.24602.objecter dump_active .. 0 homeless 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ebdffb700 10 client.24602.objecter ms_handle_connect 0x7f6ea806f820 2024-03-11T21:26:22.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.029+0000 7f6ec4e4f700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 0x7f6ea8071ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009de0 tx=0x7f6eb0009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:22.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.033+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6eb805c840 con 0x7f6ec0106e00 2024-03-11T21:26:22.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:21 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.448189+0000 osd.0 (osd.0) 79 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:26:22.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:21 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.454733+0000 osd.0 (osd.0) 80 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:26:22.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:21 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.620671+0000 mon.a (mon.0) 876 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:26:22.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:21 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.984010+0000 osd.7 (osd.7) 91 : cluster 0 5.10 scrub starts 2024-03-11T21:26:22.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:21 smithi203 bash[23654]: cluster 2024-03-11T21:26:20.986523+0000 osd.7 (osd.7) 92 : cluster 0 5.10 scrub ok 2024-03-11T21:26:22.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:26:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:22.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.285+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ec01a6bd0 con 0x7f6ec0106e00 2024-03-11T21:26:22.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.285+0000 7f6ebdffb700 1 -- 172.21.15.143:0/536519126 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6eb8062080 con 0x7f6ec0106e00 2024-03-11T21:26:22.263 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:26:22.263 INFO:teuthology.orchestra.run.smithi143.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:26:22.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.285+0000 7f6ec70b3700 10 librados: watch_flush enter 2024-03-11T21:26:22.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 10 librados: watch_flush exit 2024-03-11T21:26:22.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 20 client.24602.objecter shutdown clearing up homeless session... 2024-03-11T21:26:22.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 10 client.24602.objecter successfully canceled tick 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 msgr2=0x7f6ea8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 0x7f6ea8071ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009de0 tx=0x7f6eb0009190 comp rx=0 tx=0).stop 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 msgr2=0x7f6ec0101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8006070 tx=0x7f6eb8007a20 comp rx=0 tx=0).stop 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 shutdown_connections 2024-03-11T21:26:22.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f6ea806f820 0x7f6ea8071ce0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ec0100330 0x7f6ec0101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ec0106e00 0x7f6ec0101f90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 --2- 172.21.15.143:0/536519126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec0100cf0 0x7f6ec0101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 >> 172.21.15.143:0/536519126 conn(0x7f6ec00772e0 msgr2=0x7f6ec01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 shutdown_connections 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 -- 172.21.15.143:0/536519126 wait complete. 2024-03-11T21:26:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:22.289+0000 7f6ec70b3700 1 librados: shutdown 2024-03-11T21:26:22.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:22 smithi143 bash[29284]: cluster 2024-03-11T21:26:21.112144+0000 mgr.y (mgr.14550) 124 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:22.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:22 smithi143 bash[29284]: audit 2024-03-11T21:26:22.289695+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.143:0/536519126' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:26:22.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:22 smithi143 bash[21512]: cluster 2024-03-11T21:26:21.112144+0000 mgr.y (mgr.14550) 124 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:22.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:22 smithi143 bash[21512]: audit 2024-03-11T21:26:22.289695+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.143:0/536519126' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:26:23.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:22 smithi203 bash[23654]: cluster 2024-03-11T21:26:21.112144+0000 mgr.y (mgr.14550) 124 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:23.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:22 smithi203 bash[23654]: audit 2024-03-11T21:26:22.289695+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.143:0/536519126' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:26:23.059 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:26:23.060 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:26:23.060 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-11T21:26:23.076 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T21:26:23.076 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}}, 'workunit': {'branch': 'main', 'sha1': '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'}} 2024-03-11T21:26:23.076 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T21:26:23.077 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-11T21:26:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:24 smithi143 bash[21512]: cluster 2024-03-11T21:26:23.113031+0000 mgr.y (mgr.14550) 125 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:24.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:24 smithi143 bash[29284]: cluster 2024-03-11T21:26:23.113031+0000 mgr.y (mgr.14550) 125 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:25.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:24 smithi203 bash[23654]: cluster 2024-03-11T21:26:23.113031+0000 mgr.y (mgr.14550) 125 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:25.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:25 smithi143 bash[29284]: cluster 2024-03-11T21:26:24.347973+0000 osd.3 (osd.3) 91 : cluster 0 3.7 scrub starts 2024-03-11T21:26:25.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:25 smithi143 bash[29284]: cluster 2024-03-11T21:26:24.352397+0000 osd.3 (osd.3) 92 : cluster 0 3.7 scrub ok 2024-03-11T21:26:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:25 smithi143 bash[21512]: cluster 2024-03-11T21:26:24.347973+0000 osd.3 (osd.3) 91 : cluster 0 3.7 scrub starts 2024-03-11T21:26:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:25 smithi143 bash[21512]: cluster 2024-03-11T21:26:24.352397+0000 osd.3 (osd.3) 92 : cluster 0 3.7 scrub ok 2024-03-11T21:26:26.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:25 smithi203 bash[23654]: cluster 2024-03-11T21:26:24.347973+0000 osd.3 (osd.3) 91 : cluster 0 3.7 scrub starts 2024-03-11T21:26:26.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:25 smithi203 bash[23654]: cluster 2024-03-11T21:26:24.352397+0000 osd.3 (osd.3) 92 : cluster 0 3.7 scrub ok 2024-03-11T21:26:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[29284]: cluster 2024-03-11T21:26:25.113741+0000 mgr.y (mgr.14550) 126 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[29284]: cluster 2024-03-11T21:26:25.424327+0000 osd.0 (osd.0) 81 : cluster 0 5.5 scrub starts 2024-03-11T21:26:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[29284]: cluster 2024-03-11T21:26:25.426611+0000 osd.0 (osd.0) 82 : cluster 0 5.5 scrub ok 2024-03-11T21:26:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[21512]: cluster 2024-03-11T21:26:25.113741+0000 mgr.y (mgr.14550) 126 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[21512]: cluster 2024-03-11T21:26:25.424327+0000 osd.0 (osd.0) 81 : cluster 0 5.5 scrub starts 2024-03-11T21:26:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:26 smithi143 bash[21512]: cluster 2024-03-11T21:26:25.426611+0000 osd.0 (osd.0) 82 : cluster 0 5.5 scrub ok 2024-03-11T21:26:27.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:26 smithi203 bash[23654]: cluster 2024-03-11T21:26:25.113741+0000 mgr.y (mgr.14550) 126 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:27.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:26 smithi203 bash[23654]: cluster 2024-03-11T21:26:25.424327+0000 osd.0 (osd.0) 81 : cluster 0 5.5 scrub starts 2024-03-11T21:26:27.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:26 smithi203 bash[23654]: cluster 2024-03-11T21:26:25.426611+0000 osd.0 (osd.0) 82 : cluster 0 5.5 scrub ok 2024-03-11T21:26:27.852 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:27.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:27 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.285646+0000 osd.4 (osd.4) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:27.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:27 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.285755+0000 osd.4 (osd.4) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:27.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:27 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.285646+0000 osd.4 (osd.4) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:27.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:27 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.285755+0000 osd.4 (osd.4) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:28.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:27 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.285646+0000 osd.4 (osd.4) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:28.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:27 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.285755+0000 osd.4 (osd.4) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:28.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.114420+0000 mgr.y (mgr.14550) 127 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.352273+0000 osd.3 (osd.3) 93 : cluster 0 6.1b scrub starts 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.355084+0000 osd.3 (osd.3) 94 : cluster 0 6.1b scrub ok 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.657681+0000 mon.a (mon.0) 877 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: audit 2024-03-11T21:26:27.711225+0000 mon.c (mon.2) 80 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.749700+0000 osd.5 (osd.5) 89 : cluster 0 2.12 scrub starts 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:27.752188+0000 osd.5 (osd.5) 90 : cluster 0 2.12 scrub ok 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:28.311022+0000 osd.3 (osd.3) 95 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:26:28.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[21512]: cluster 2024-03-11T21:26:28.313475+0000 osd.3 (osd.3) 96 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.114420+0000 mgr.y (mgr.14550) 127 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.352273+0000 osd.3 (osd.3) 93 : cluster 0 6.1b scrub starts 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.355084+0000 osd.3 (osd.3) 94 : cluster 0 6.1b scrub ok 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.657681+0000 mon.a (mon.0) 877 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: audit 2024-03-11T21:26:27.711225+0000 mon.c (mon.2) 80 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.749700+0000 osd.5 (osd.5) 89 : cluster 0 2.12 scrub starts 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:27.752188+0000 osd.5 (osd.5) 90 : cluster 0 2.12 scrub ok 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:28.311022+0000 osd.3 (osd.3) 95 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:26:28.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:28 smithi143 bash[29284]: cluster 2024-03-11T21:26:28.313475+0000 osd.3 (osd.3) 96 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.114420+0000 mgr.y (mgr.14550) 127 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.352273+0000 osd.3 (osd.3) 93 : cluster 0 6.1b scrub starts 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.355084+0000 osd.3 (osd.3) 94 : cluster 0 6.1b scrub ok 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.657681+0000 mon.a (mon.0) 877 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: audit 2024-03-11T21:26:27.711225+0000 mon.c (mon.2) 80 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.749700+0000 osd.5 (osd.5) 89 : cluster 0 2.12 scrub starts 2024-03-11T21:26:29.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:27.752188+0000 osd.5 (osd.5) 90 : cluster 0 2.12 scrub ok 2024-03-11T21:26:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:28.311022+0000 osd.3 (osd.3) 95 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:26:29.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:28 smithi203 bash[23654]: cluster 2024-03-11T21:26:28.313475+0000 osd.3 (osd.3) 96 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[29284]: audit 2024-03-11T21:26:28.190645+0000 mgr.y (mgr.14550) 128 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.006524+0000 osd.7 (osd.7) 93 : cluster 0 5.9 scrub starts 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.009706+0000 osd.7 (osd.7) 94 : cluster 0 5.9 scrub ok 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[21512]: audit 2024-03-11T21:26:28.190645+0000 mgr.y (mgr.14550) 128 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.006524+0000 osd.7 (osd.7) 93 : cluster 0 5.9 scrub starts 2024-03-11T21:26:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:29 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.009706+0000 osd.7 (osd.7) 94 : cluster 0 5.9 scrub ok 2024-03-11T21:26:30.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:29 smithi203 bash[23654]: audit 2024-03-11T21:26:28.190645+0000 mgr.y (mgr.14550) 128 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:30.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:29 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.006524+0000 osd.7 (osd.7) 93 : cluster 0 5.9 scrub starts 2024-03-11T21:26:30.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:29 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.009706+0000 osd.7 (osd.7) 94 : cluster 0 5.9 scrub ok 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.115261+0000 mgr.y (mgr.14550) 129 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.964422+0000 osd.7 (osd.7) 95 : cluster 0 6.1c scrub starts 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.967302+0000 osd.7 (osd.7) 96 : cluster 0 6.1c scrub ok 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.970357+0000 osd.6 (osd.6) 53 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:29.972673+0000 osd.6 (osd.6) 54 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:30.312032+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-03-11T21:26:30.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[21512]: cluster 2024-03-11T21:26:30.314825+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.115261+0000 mgr.y (mgr.14550) 129 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.964422+0000 osd.7 (osd.7) 95 : cluster 0 6.1c scrub starts 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.967302+0000 osd.7 (osd.7) 96 : cluster 0 6.1c scrub ok 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.970357+0000 osd.6 (osd.6) 53 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:29.972673+0000 osd.6 (osd.6) 54 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:30.312032+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-03-11T21:26:30.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:30 smithi143 bash[29284]: cluster 2024-03-11T21:26:30.314825+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.115261+0000 mgr.y (mgr.14550) 129 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.964422+0000 osd.7 (osd.7) 95 : cluster 0 6.1c scrub starts 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.967302+0000 osd.7 (osd.7) 96 : cluster 0 6.1c scrub ok 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.970357+0000 osd.6 (osd.6) 53 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:29.972673+0000 osd.6 (osd.6) 54 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:30.312032+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-03-11T21:26:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:30 smithi203 bash[23654]: cluster 2024-03-11T21:26:30.314825+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- 172.21.15.143:0/3103185697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8106e00 msgr2=0x7f53b81071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- 172.21.15.143:0/3103185697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8106e00 0x7f53b81071e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0099c0 tx=0x7f53ac009cd0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- 172.21.15.143:0/3103185697 shutdown_connections 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- 172.21.15.143:0/3103185697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100cf0 0x7f53b810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- 172.21.15.143:0/3103185697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8106e00 0x7f53b81071e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- 172.21.15.143:0/3103185697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8100330 0x7f53b81007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- 172.21.15.143:0/3103185697 >> 172.21.15.143:0/3103185697 conn(0x7f53b80772e0 msgr2=0x7f53b80fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- 172.21.15.143:0/3103185697 shutdown_connections 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- 172.21.15.143:0/3103185697 wait complete. 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 librados: starting msgr at 2024-03-11T21:26:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 librados: starting objecter 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 Processor -- start 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- start start 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 librados: setting wanted keys 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 librados: calling monclient init 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100330 0x7f53b819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8106e00 0x7f53b8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8111720 con 0x7f53b8106e00 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b81115a0 con 0x7f53b8100330 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bf103700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b81118a0 con 0x7f53b8100cf0 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47980/0 (socket says 172.21.15.143:47980) 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53b7fff700 1 -- 172.21.15.143:0/1839643026 learned_addr learned my addr 172.21.15.143:0/1839643026 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:31.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bd6a0700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8106e00 0x7f53b8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:31.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.089+0000 7f53bce9f700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100330 0x7f53b819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:31.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 -- 172.21.15.143:0/1839643026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100330 msgr2=0x7f53b819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:31.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100330 0x7f53b819d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 -- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8106e00 msgr2=0x7f53b8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:31.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8106e00 0x7f53b8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 -- 172.21.15.143:0/1839643026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b81981f0 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b7fff700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53a400eb30 tx=0x7f53a400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 10 client.?.objecter ms_handle_connect 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a400cc90 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f53a400ce30 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b81984e0 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bf103700 10 client.24608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b81a47a0 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bf103700 1 librados: init done 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4003df0 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b804faa0 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f53a4021440 con 0x7f53b8100cf0 2024-03-11T21:26:31.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 0x7f53a8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bce9f700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 0x7f53a8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(120..120 src has 66..120) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f53a408e490 con 0x7f53b8100cf0 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 10 client.24608.objecter ms_dispatch 0x7f53b8001550 osd_map(120..120 src has 66..120) v4 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 3 client.24608.objecter handle_osd_map got epochs [120,120] > 0 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 3 client.24608.objecter handle_osd_map decoding full epoch 120 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 20 client.24608.objecter dump_active .. 0 homeless 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53b5ffb700 10 client.24608.objecter ms_handle_connect 0x7f53a806f820 2024-03-11T21:26:31.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.093+0000 7f53bce9f700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 0x7f53a8071ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac005ea0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:31.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.101+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f53a405c790 con 0x7f53b8100cf0 2024-03-11T21:26:31.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.309+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f53b81a4a80 con 0x7f53b8100cf0 2024-03-11T21:26:31.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53b5ffb700 1 -- 172.21.15.143:0/1839643026 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v32)=0 v32) v1 ==== 136+0+0 (secure 0 0 0) 0x7f53a40601e0 con 0x7f53b8100cf0 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 10 librados: watch_flush enter 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 10 librados: watch_flush exit 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 20 client.24608.objecter shutdown clearing up homeless session... 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 10 client.24608.objecter successfully canceled tick 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 msgr2=0x7f53a8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 0x7f53a8071ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac005ea0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 msgr2=0x7f53b819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53a400eb30 tx=0x7f53a400ee40 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 shutdown_connections 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f53a806f820 0x7f53a8071ce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b8106e00 0x7f53b8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53b8100330 0x7f53b819d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 --2- 172.21.15.143:0/1839643026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b8100cf0 0x7f53b819de10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:31.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 >> 172.21.15.143:0/1839643026 conn(0x7f53b80772e0 msgr2=0x7f53b81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:31.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 shutdown_connections 2024-03-11T21:26:31.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 -- 172.21.15.143:0/1839643026 wait complete. 2024-03-11T21:26:31.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:31.317+0000 7f53bf103700 1 librados: shutdown 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE Bus STOPPING 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE Bus STOPPED 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE Bus STARTING 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE Serving on http://:::9283 2024-03-11T21:26:31.685 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:31 smithi143 bash[21768]: [11/Mar/2024:21:26:31] ENGINE Bus STARTED 2024-03-11T21:26:32.083 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-11T21:26:32.084 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:26:32.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:26:32.435 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:26:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: cluster 2024-03-11T21:26:31.115600+0000 mgr.y (mgr.14550) 130 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: audit 2024-03-11T21:26:31.317958+0000 mon.a (mon.0) 878 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: audit 2024-03-11T21:26:31.349212+0000 mon.c (mon.2) 81 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: cluster 2024-03-11T21:26:31.871033+0000 osd.7 (osd.7) 97 : cluster 0 3.f scrub starts 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: cluster 2024-03-11T21:26:31.873660+0000 osd.7 (osd.7) 98 : cluster 0 3.f scrub ok 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: audit 2024-03-11T21:26:32.043099+0000 mon.c (mon.2) 82 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: audit 2024-03-11T21:26:32.044803+0000 mon.c (mon.2) 83 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[21512]: audit 2024-03-11T21:26:32.055159+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: cluster 2024-03-11T21:26:31.115600+0000 mgr.y (mgr.14550) 130 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: audit 2024-03-11T21:26:31.317958+0000 mon.a (mon.0) 878 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: audit 2024-03-11T21:26:31.349212+0000 mon.c (mon.2) 81 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: cluster 2024-03-11T21:26:31.871033+0000 osd.7 (osd.7) 97 : cluster 0 3.f scrub starts 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: cluster 2024-03-11T21:26:31.873660+0000 osd.7 (osd.7) 98 : cluster 0 3.f scrub ok 2024-03-11T21:26:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: audit 2024-03-11T21:26:32.043099+0000 mon.c (mon.2) 82 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: audit 2024-03-11T21:26:32.044803+0000 mon.c (mon.2) 83 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:32 smithi143 bash[29284]: audit 2024-03-11T21:26:32.055159+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: cluster 2024-03-11T21:26:31.115600+0000 mgr.y (mgr.14550) 130 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: audit 2024-03-11T21:26:31.317958+0000 mon.a (mon.0) 878 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: audit 2024-03-11T21:26:31.349212+0000 mon.c (mon.2) 81 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: cluster 2024-03-11T21:26:31.871033+0000 osd.7 (osd.7) 97 : cluster 0 3.f scrub starts 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: cluster 2024-03-11T21:26:31.873660+0000 osd.7 (osd.7) 98 : cluster 0 3.f scrub ok 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: audit 2024-03-11T21:26:32.043099+0000 mon.c (mon.2) 82 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: audit 2024-03-11T21:26:32.044803+0000 mon.c (mon.2) 83 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:32 smithi203 bash[23654]: audit 2024-03-11T21:26:32.055159+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:26:33.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[29284]: cluster 2024-03-11T21:26:31.758038+0000 osd.5 (osd.5) 91 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:26:33.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[29284]: cluster 2024-03-11T21:26:31.763322+0000 osd.5 (osd.5) 92 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:26:33.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[29284]: cluster 2024-03-11T21:26:32.524046+0000 osd.0 (osd.0) 83 : cluster 0 3.6 scrub starts 2024-03-11T21:26:33.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[29284]: cluster 2024-03-11T21:26:32.528626+0000 osd.0 (osd.0) 84 : cluster 0 3.6 scrub ok 2024-03-11T21:26:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[21512]: cluster 2024-03-11T21:26:31.758038+0000 osd.5 (osd.5) 91 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:26:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[21512]: cluster 2024-03-11T21:26:31.763322+0000 osd.5 (osd.5) 92 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:26:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[21512]: cluster 2024-03-11T21:26:32.524046+0000 osd.0 (osd.0) 83 : cluster 0 3.6 scrub starts 2024-03-11T21:26:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:33 smithi143 bash[21512]: cluster 2024-03-11T21:26:32.528626+0000 osd.0 (osd.0) 84 : cluster 0 3.6 scrub ok 2024-03-11T21:26:33.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:33 smithi203 bash[23654]: cluster 2024-03-11T21:26:31.758038+0000 osd.5 (osd.5) 91 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:26:33.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:33 smithi203 bash[23654]: cluster 2024-03-11T21:26:31.763322+0000 osd.5 (osd.5) 92 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:26:33.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:33 smithi203 bash[23654]: cluster 2024-03-11T21:26:32.524046+0000 osd.0 (osd.0) 83 : cluster 0 3.6 scrub starts 2024-03-11T21:26:33.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:33 smithi203 bash[23654]: cluster 2024-03-11T21:26:32.528626+0000 osd.0 (osd.0) 84 : cluster 0 3.6 scrub ok 2024-03-11T21:26:34.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:34 smithi143 bash[29284]: cluster 2024-03-11T21:26:33.117007+0000 mgr.y (mgr.14550) 131 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:34 smithi143 bash[21512]: cluster 2024-03-11T21:26:33.117007+0000 mgr.y (mgr.14550) 131 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:34.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:34 smithi203 bash[23654]: cluster 2024-03-11T21:26:33.117007+0000 mgr.y (mgr.14550) 131 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:35.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:35 smithi143 bash[29284]: cluster 2024-03-11T21:26:34.804608+0000 osd.7 (osd.7) 99 : cluster 0 5.4 scrub starts 2024-03-11T21:26:35.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:35 smithi143 bash[29284]: cluster 2024-03-11T21:26:34.807049+0000 osd.7 (osd.7) 100 : cluster 0 5.4 scrub ok 2024-03-11T21:26:35.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:35 smithi143 bash[21512]: cluster 2024-03-11T21:26:34.804608+0000 osd.7 (osd.7) 99 : cluster 0 5.4 scrub starts 2024-03-11T21:26:35.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:35 smithi143 bash[21512]: cluster 2024-03-11T21:26:34.807049+0000 osd.7 (osd.7) 100 : cluster 0 5.4 scrub ok 2024-03-11T21:26:35.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:35 smithi203 bash[23654]: cluster 2024-03-11T21:26:34.804608+0000 osd.7 (osd.7) 99 : cluster 0 5.4 scrub starts 2024-03-11T21:26:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:35 smithi203 bash[23654]: cluster 2024-03-11T21:26:34.807049+0000 osd.7 (osd.7) 100 : cluster 0 5.4 scrub ok 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[21512]: cluster 2024-03-11T21:26:35.117649+0000 mgr.y (mgr.14550) 132 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[21512]: cluster 2024-03-11T21:26:35.763200+0000 osd.7 (osd.7) 101 : cluster 0 2.1d scrub starts 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[21512]: cluster 2024-03-11T21:26:35.765472+0000 osd.7 (osd.7) 102 : cluster 0 2.1d scrub ok 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[29284]: cluster 2024-03-11T21:26:35.117649+0000 mgr.y (mgr.14550) 132 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[29284]: cluster 2024-03-11T21:26:35.763200+0000 osd.7 (osd.7) 101 : cluster 0 2.1d scrub starts 2024-03-11T21:26:36.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:36 smithi143 bash[29284]: cluster 2024-03-11T21:26:35.765472+0000 osd.7 (osd.7) 102 : cluster 0 2.1d scrub ok 2024-03-11T21:26:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:36 smithi203 bash[23654]: cluster 2024-03-11T21:26:35.117649+0000 mgr.y (mgr.14550) 132 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:36 smithi203 bash[23654]: cluster 2024-03-11T21:26:35.763200+0000 osd.7 (osd.7) 101 : cluster 0 2.1d scrub starts 2024-03-11T21:26:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:36 smithi203 bash[23654]: cluster 2024-03-11T21:26:35.765472+0000 osd.7 (osd.7) 102 : cluster 0 2.1d scrub ok 2024-03-11T21:26:36.852 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:37.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:37 smithi143 bash[21512]: cluster 2024-03-11T21:26:36.754052+0000 osd.7 (osd.7) 103 : cluster 0 1.0 scrub starts 2024-03-11T21:26:37.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:37 smithi143 bash[21512]: cluster 2024-03-11T21:26:36.758077+0000 osd.7 (osd.7) 104 : cluster 0 1.0 scrub ok 2024-03-11T21:26:37.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:37 smithi143 bash[29284]: cluster 2024-03-11T21:26:36.754052+0000 osd.7 (osd.7) 103 : cluster 0 1.0 scrub starts 2024-03-11T21:26:37.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:37 smithi143 bash[29284]: cluster 2024-03-11T21:26:36.758077+0000 osd.7 (osd.7) 104 : cluster 0 1.0 scrub ok 2024-03-11T21:26:37.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:37 smithi203 bash[23654]: cluster 2024-03-11T21:26:36.754052+0000 osd.7 (osd.7) 103 : cluster 0 1.0 scrub starts 2024-03-11T21:26:37.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:37 smithi203 bash[23654]: cluster 2024-03-11T21:26:36.758077+0000 osd.7 (osd.7) 104 : cluster 0 1.0 scrub ok 2024-03-11T21:26:38.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:38.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:38 smithi203 bash[23654]: cluster 2024-03-11T21:26:37.118317+0000 mgr.y (mgr.14550) 133 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:38.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:38 smithi203 bash[23654]: cluster 2024-03-11T21:26:38.188971+0000 osd.3 (osd.3) 99 : cluster 0 3.18 scrub starts 2024-03-11T21:26:38.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:38 smithi203 bash[23654]: cluster 2024-03-11T21:26:38.192542+0000 osd.3 (osd.3) 100 : cluster 0 3.18 scrub ok 2024-03-11T21:26:38.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[21512]: cluster 2024-03-11T21:26:37.118317+0000 mgr.y (mgr.14550) 133 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:38.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[21512]: cluster 2024-03-11T21:26:38.188971+0000 osd.3 (osd.3) 99 : cluster 0 3.18 scrub starts 2024-03-11T21:26:38.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[21512]: cluster 2024-03-11T21:26:38.192542+0000 osd.3 (osd.3) 100 : cluster 0 3.18 scrub ok 2024-03-11T21:26:38.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[29284]: cluster 2024-03-11T21:26:37.118317+0000 mgr.y (mgr.14550) 133 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:38.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[29284]: cluster 2024-03-11T21:26:38.188971+0000 osd.3 (osd.3) 99 : cluster 0 3.18 scrub starts 2024-03-11T21:26:38.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:38 smithi143 bash[29284]: cluster 2024-03-11T21:26:38.192542+0000 osd.3 (osd.3) 100 : cluster 0 3.18 scrub ok 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- 172.21.15.143:0/188694937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 msgr2=0x7ff01010db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- 172.21.15.143:0/188694937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff01010db80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff00800b0a0 tx=0x7ff00800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- 172.21.15.143:0/188694937 shutdown_connections 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- 172.21.15.143:0/188694937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010106dd0 0x7ff0101071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- 172.21.15.143:0/188694937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff01010db80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- 172.21.15.143:0/188694937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010100d20 0x7ff0101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- 172.21.15.143:0/188694937 >> 172.21.15.143:0/188694937 conn(0x7ff010077d30 msgr2=0x7ff010078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:39.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- 172.21.15.143:0/188694937 shutdown_connections 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- 172.21.15.143:0/188694937 wait complete. 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 librados: starting msgr at 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 librados: starting objecter 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 Processor -- start 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- start start 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 librados: setting wanted keys 2024-03-11T21:26:39.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 librados: calling monclient init 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff010102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010106dd0 0x7ff010103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0101105d0 con 0x7ff010100d20 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff010110450 con 0x7ff010106dd0 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff017564700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff010110750 con 0x7ff0101016e0 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff014aff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff010102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:38994/0 (socket says 172.21.15.143:38994) 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 -- 172.21.15.143:0/1864034277 learned_addr learned my addr 172.21.15.143:0/1864034277 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:39.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015b01700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010106dd0 0x7ff010103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 -- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 msgr2=0x7ff010102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff010102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 -- 172.21.15.143:0/1864034277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010106dd0 msgr2=0x7ff010103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010106dd0 0x7ff010103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff015300700 1 -- 172.21.15.143:0/1864034277 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0101a60d0 con 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.953+0000 7ff014aff700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff010102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff015300700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00eb30 tx=0x7ff00c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 10 client.?.objecter ms_handle_connect 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00cc90 con 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0101a63c0 con 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff00c00ce30 con 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c003df0 con 0x7ff010100d20 2024-03-11T21:26:39.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff017564700 10 client.14739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:39.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0101a6900 con 0x7ff010100d20 2024-03-11T21:26:39.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff017564700 1 librados: init done 2024-03-11T21:26:39.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01004faa0 con 0x7ff010100d20 2024-03-11T21:26:39.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff00c003f90 con 0x7ff010100d20 2024-03-11T21:26:39.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 0x7feffc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:39.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[21512]: audit 2024-03-11T21:26:38.202096+0000 mgr.y (mgr.14550) 134 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:39.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[21512]: cluster 2024-03-11T21:26:39.216319+0000 osd.3 (osd.3) 101 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:26:39.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[21512]: cluster 2024-03-11T21:26:39.218956+0000 osd.3 (osd.3) 102 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:26:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[29284]: audit 2024-03-11T21:26:38.202096+0000 mgr.y (mgr.14550) 134 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[29284]: cluster 2024-03-11T21:26:39.216319+0000 osd.3 (osd.3) 101 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:26:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:39 smithi143 bash[29284]: cluster 2024-03-11T21:26:39.218956+0000 osd.3 (osd.3) 102 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(120..120 src has 66..120) v4 ==== 7450+0+0 (secure 0 0 0) 0x7ff00c08e4b0 con 0x7ff010100d20 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 10 client.14739.objecter ms_dispatch 0x7ff010001550 osd_map(120..120 src has 66..120) v4 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 3 client.14739.objecter handle_osd_map got epochs [120,120] > 0 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 3 client.14739.objecter handle_osd_map decoding full epoch 120 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff014aff700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 0x7feffc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 20 client.14739.objecter dump_active .. 0 homeless 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff014aff700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 0x7feffc071ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff010078470 tx=0x7ff000009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:39.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.957+0000 7ff0067fc700 10 client.14739.objecter ms_handle_connect 0x7feffc06f820 2024-03-11T21:26:39.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:39.961+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff00c05b9d0 con 0x7ff010100d20 2024-03-11T21:26:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:39 smithi203 bash[23654]: audit 2024-03-11T21:26:38.202096+0000 mgr.y (mgr.14550) 134 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:39 smithi203 bash[23654]: cluster 2024-03-11T21:26:39.216319+0000 osd.3 (osd.3) 101 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:26:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:39 smithi203 bash[23654]: cluster 2024-03-11T21:26:39.218956+0000 osd.3 (osd.3) 102 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:26:40.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.165+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff01006abb0 con 0x7feffc06f820 2024-03-11T21:26:40.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.181+0000 7ff0067fc700 1 -- 172.21.15.143:0/1864034277 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346909 (secure 0 0 0) 0x7ff01006abb0 con 0x7feffc06f820 2024-03-11T21:26:40.164 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 10 librados: watch_flush enter 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 10 librados: watch_flush exit 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 20 client.14739.objecter shutdown clearing up homeless session... 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 10 client.14739.objecter successfully canceled tick 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 msgr2=0x7feffc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 0x7feffc071ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff010078470 tx=0x7ff000009190 comp rx=0 tx=0).stop 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 msgr2=0x7ff0101029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00eb30 tx=0x7ff00c00ee40 comp rx=0 tx=0).stop 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 shutdown_connections 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feffc06f820 0x7feffc071ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff010100d20 0x7ff0101029d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff010106dd0 0x7ff010103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:40.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 --2- 172.21.15.143:0/1864034277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0101016e0 0x7ff010102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:40.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 >> 172.21.15.143:0/1864034277 conn(0x7ff010077d30 msgr2=0x7ff01006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:40.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 shutdown_connections 2024-03-11T21:26:40.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 -- 172.21.15.143:0/1864034277 wait complete. 2024-03-11T21:26:40.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:40.185+0000 7ff017564700 1 librados: shutdown 2024-03-11T21:26:40.168 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:26:40.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[21512]: cluster 2024-03-11T21:26:38.933706+0000 osd.2 (osd.2) 59 : cluster 0 5.b scrub starts 2024-03-11T21:26:40.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[21512]: cluster 2024-03-11T21:26:38.936560+0000 osd.2 (osd.2) 60 : cluster 0 5.b scrub ok 2024-03-11T21:26:40.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[21512]: cluster 2024-03-11T21:26:39.119136+0000 mgr.y (mgr.14550) 135 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:40.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[29284]: cluster 2024-03-11T21:26:38.933706+0000 osd.2 (osd.2) 59 : cluster 0 5.b scrub starts 2024-03-11T21:26:40.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[29284]: cluster 2024-03-11T21:26:38.936560+0000 osd.2 (osd.2) 60 : cluster 0 5.b scrub ok 2024-03-11T21:26:40.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:40 smithi143 bash[29284]: cluster 2024-03-11T21:26:39.119136+0000 mgr.y (mgr.14550) 135 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:41.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:40 smithi203 bash[23654]: cluster 2024-03-11T21:26:38.933706+0000 osd.2 (osd.2) 59 : cluster 0 5.b scrub starts 2024-03-11T21:26:41.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:40 smithi203 bash[23654]: cluster 2024-03-11T21:26:38.936560+0000 osd.2 (osd.2) 60 : cluster 0 5.b scrub ok 2024-03-11T21:26:41.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:40 smithi203 bash[23654]: cluster 2024-03-11T21:26:39.119136+0000 mgr.y (mgr.14550) 135 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:41.039 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-11T21:26:39.118557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1145,"num_read_kb":991,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248704,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712512,"statfs":{"total":767960285184,"available":767705612288,"internally_reserved":0,"allocated":7794688,"data_stored":4953432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.466376"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159922+0000","last_change":"2024-03-11T21:26:27.355463+0000","last_active":"2024-03-11T21:26:28.159922+0000","last_peered":"2024-03-11T21:26:28.159922+0000","last_clean":"2024-03-11T21:26:28.159922+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:26:28.159922+0000","last_undegraded":"2024-03-11T21:26:28.159922+0000","last_fullsized":"2024-03-11T21:26:28.159922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.355103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.355103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.390277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649713+0000","last_change":"2024-03-11T21:25:58.944512+0000","last_active":"2024-03-11T21:26:27.649713+0000","last_peered":"2024-03-11T21:26:27.649713+0000","last_clean":"2024-03-11T21:26:27.649713+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:26:27.649713+0000","last_undegraded":"2024-03-11T21:26:27.649713+0000","last_fullsized":"2024-03-11T21:26:27.649713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.944364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.944364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.570898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162490+0000","last_change":"2024-03-11T21:26:03.337169+0000","last_active":"2024-03-11T21:26:28.162490+0000","last_peered":"2024-03-11T21:26:28.162490+0000","last_clean":"2024-03-11T21:26:28.162490+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:26:28.162490+0000","last_undegraded":"2024-03-11T21:26:28.162490+0000","last_fullsized":"2024-03-11T21:26:28.162490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:26:03.337010+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.337010+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:29.657159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161802+0000","last_change":"2024-03-11T21:25:31.918871+0000","last_active":"2024-03-11T21:26:28.161802+0000","last_peered":"2024-03-11T21:26:28.161802+0000","last_clean":"2024-03-11T21:26:28.161802+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:26:28.161802+0000","last_undegraded":"2024-03-11T21:26:28.161802+0000","last_fullsized":"2024-03-11T21:26:28.161802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:31.918660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.918660+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.711285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649777+0000","last_change":"2024-03-11T21:25:48.920160+0000","last_active":"2024-03-11T21:26:27.649777+0000","last_peered":"2024-03-11T21:26:27.649777+0000","last_clean":"2024-03-11T21:26:27.649777+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:26:27.649777+0000","last_undegraded":"2024-03-11T21:26:27.649777+0000","last_fullsized":"2024-03-11T21:26:27.649777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.919930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.919930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.064632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.654717+0000","last_change":"2024-03-11T21:26:13.570113+0000","last_active":"2024-03-11T21:26:27.654717+0000","last_peered":"2024-03-11T21:26:27.654717+0000","last_clean":"2024-03-11T21:26:27.654717+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:26:27.654717+0000","last_undegraded":"2024-03-11T21:26:27.654717+0000","last_fullsized":"2024-03-11T21:26:27.654717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.569895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.569895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.192341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162348+0000","last_change":"2024-03-11T21:26:12.187395+0000","last_active":"2024-03-11T21:26:28.162348+0000","last_peered":"2024-03-11T21:26:28.162348+0000","last_clean":"2024-03-11T21:26:28.162348+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:26:28.162348+0000","last_undegraded":"2024-03-11T21:26:28.162348+0000","last_fullsized":"2024-03-11T21:26:28.162348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.187178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.187178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.244264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162210+0000","last_change":"2024-03-11T21:25:46.288260+0000","last_active":"2024-03-11T21:26:28.162210+0000","last_peered":"2024-03-11T21:26:28.162210+0000","last_clean":"2024-03-11T21:26:28.162210+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:26:28.162210+0000","last_undegraded":"2024-03-11T21:26:28.162210+0000","last_fullsized":"2024-03-11T21:26:28.162210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:46.288021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.288021+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.040825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155205+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:26:28.155205+0000","last_peered":"2024-03-11T21:26:28.155205+0000","last_clean":"2024-03-11T21:26:28.155205+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:26:28.155205+0000","last_undegraded":"2024-03-11T21:26:28.155205+0000","last_fullsized":"2024-03-11T21:26:28.155205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.062649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158340+0000","last_change":"2024-03-11T21:26:11.146066+0000","last_active":"2024-03-11T21:26:28.158340+0000","last_peered":"2024-03-11T21:26:28.158340+0000","last_clean":"2024-03-11T21:26:28.158340+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:26:28.158340+0000","last_undegraded":"2024-03-11T21:26:28.158340+0000","last_fullsized":"2024-03-11T21:26:28.158340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.145856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.145856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.221066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:35.765638+0000","last_change":"2024-03-11T21:26:35.765693+0000","last_active":"2024-03-11T21:26:35.765638+0000","last_peered":"2024-03-11T21:26:35.765638+0000","last_clean":"2024-03-11T21:26:35.765638+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:26:35.765638+0000","last_undegraded":"2024-03-11T21:26:35.765638+0000","last_fullsized":"2024-03-11T21:26:35.765638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:35.765486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:35.765486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:44.892300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":150,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:31.763426+0000","last_change":"2024-03-11T21:26:31.763473+0000","last_active":"2024-03-11T21:26:31.763426+0000","last_peered":"2024-03-11T21:26:31.763426+0000","last_clean":"2024-03-11T21:26:31.763426+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:26:31.763426+0000","last_undegraded":"2024-03-11T21:26:31.763426+0000","last_fullsized":"2024-03-11T21:26:31.763426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.763327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:46.643188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159633+0000","last_change":"2024-03-11T21:26:04.385285+0000","last_active":"2024-03-11T21:26:28.159633+0000","last_peered":"2024-03-11T21:26:28.159633+0000","last_clean":"2024-03-11T21:26:28.159633+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:26:28.159633+0000","last_undegraded":"2024-03-11T21:26:28.159633+0000","last_fullsized":"2024-03-11T21:26:28.159633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.385068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.385068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.394697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156016+0000","last_change":"2024-03-11T21:26:11.457112+0000","last_active":"2024-03-11T21:26:28.156016+0000","last_peered":"2024-03-11T21:26:28.156016+0000","last_clean":"2024-03-11T21:26:28.156016+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:26:28.156016+0000","last_undegraded":"2024-03-11T21:26:28.156016+0000","last_fullsized":"2024-03-11T21:26:28.156016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.456900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.456900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:19.697223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160805+0000","last_change":"2024-03-11T21:25:48.050860+0000","last_active":"2024-03-11T21:26:28.160805+0000","last_peered":"2024-03-11T21:26:28.160805+0000","last_clean":"2024-03-11T21:26:28.160805+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:26:28.160805+0000","last_undegraded":"2024-03-11T21:26:28.160805+0000","last_fullsized":"2024-03-11T21:26:28.160805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.050648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.050648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.250408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":133,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159722+0000","last_change":"2024-03-11T21:25:51.434868+0000","last_active":"2024-03-11T21:26:28.159722+0000","last_peered":"2024-03-11T21:26:28.159722+0000","last_clean":"2024-03-11T21:26:28.159722+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:26:28.159722+0000","last_undegraded":"2024-03-11T21:26:28.159722+0000","last_fullsized":"2024-03-11T21:26:28.159722+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:51.434497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:51.434497+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.747829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163041+0000","last_change":"2024-03-11T21:26:20.277298+0000","last_active":"2024-03-11T21:26:28.163041+0000","last_peered":"2024-03-11T21:26:28.163041+0000","last_clean":"2024-03-11T21:26:28.163041+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:26:28.163041+0000","last_undegraded":"2024-03-11T21:26:28.163041+0000","last_fullsized":"2024-03-11T21:26:28.163041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.277140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.790443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650750+0000","last_change":"2024-03-11T21:26:10.005094+0000","last_active":"2024-03-11T21:26:27.650750+0000","last_peered":"2024-03-11T21:26:27.650750+0000","last_clean":"2024-03-11T21:26:27.650750+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:26:27.650750+0000","last_undegraded":"2024-03-11T21:26:27.650750+0000","last_fullsized":"2024-03-11T21:26:27.650750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.004882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.004882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:30.146208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160511+0000","last_change":"2024-03-11T21:25:18.887001+0000","last_active":"2024-03-11T21:26:28.160511+0000","last_peered":"2024-03-11T21:26:28.160511+0000","last_clean":"2024-03-11T21:26:28.160511+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:26:28.160511+0000","last_undegraded":"2024-03-11T21:26:28.160511+0000","last_fullsized":"2024-03-11T21:26:28.160511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:18.886911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:18.886911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:36.216660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650102+0000","last_change":"2024-03-11T21:25:43.877458+0000","last_active":"2024-03-11T21:26:27.650102+0000","last_peered":"2024-03-11T21:26:27.650102+0000","last_clean":"2024-03-11T21:26:27.650102+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:26:27.650102+0000","last_undegraded":"2024-03-11T21:26:27.650102+0000","last_fullsized":"2024-03-11T21:26:27.650102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:43.877225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.877225+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.780065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651606+0000","last_change":"2024-03-11T21:25:59.988760+0000","last_active":"2024-03-11T21:26:27.651606+0000","last_peered":"2024-03-11T21:26:27.651606+0000","last_clean":"2024-03-11T21:26:27.651606+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:26:27.651606+0000","last_undegraded":"2024-03-11T21:26:27.651606+0000","last_fullsized":"2024-03-11T21:26:27.651606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.988522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.034265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.659105+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:26:27.659105+0000","last_peered":"2024-03-11T21:26:27.659105+0000","last_clean":"2024-03-11T21:26:27.659105+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:26:27.659105+0000","last_undegraded":"2024-03-11T21:26:27.659105+0000","last_fullsized":"2024-03-11T21:26:27.659105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.655474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157019+0000","last_change":"2024-03-11T21:26:04.956591+0000","last_active":"2024-03-11T21:26:28.157019+0000","last_peered":"2024-03-11T21:26:28.157019+0000","last_clean":"2024-03-11T21:26:28.157019+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:26:28.157019+0000","last_undegraded":"2024-03-11T21:26:28.157019+0000","last_fullsized":"2024-03-11T21:26:28.157019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.956350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.956350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.718796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163838+0000","last_change":"2024-03-11T21:26:20.455065+0000","last_active":"2024-03-11T21:26:28.163838+0000","last_peered":"2024-03-11T21:26:28.163838+0000","last_clean":"2024-03-11T21:26:28.163838+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:26:28.163838+0000","last_undegraded":"2024-03-11T21:26:28.163838+0000","last_fullsized":"2024-03-11T21:26:28.163838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.454747+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.108877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.659377+0000","last_change":"2024-03-11T21:26:09.575095+0000","last_active":"2024-03-11T21:26:27.659377+0000","last_peered":"2024-03-11T21:26:27.659377+0000","last_clean":"2024-03-11T21:26:27.659377+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:26:27.659377+0000","last_undegraded":"2024-03-11T21:26:27.659377+0000","last_fullsized":"2024-03-11T21:26:27.659377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:09.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:09.574796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.590692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160946+0000","last_change":"2024-03-11T21:25:40.280094+0000","last_active":"2024-03-11T21:26:28.160946+0000","last_peered":"2024-03-11T21:26:28.160946+0000","last_clean":"2024-03-11T21:26:28.160946+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:26:28.160946+0000","last_undegraded":"2024-03-11T21:26:28.160946+0000","last_fullsized":"2024-03-11T21:26:28.160946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.279715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.279715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.096032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":115,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163021+0000","last_change":"2024-03-11T21:25:27.026231+0000","last_active":"2024-03-11T21:26:28.163021+0000","last_peered":"2024-03-11T21:26:28.163021+0000","last_clean":"2024-03-11T21:26:28.163021+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:26:28.163021+0000","last_undegraded":"2024-03-11T21:26:28.163021+0000","last_fullsized":"2024-03-11T21:26:28.163021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:27.025913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.025913+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:37.624911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160761+0000","last_change":"2024-03-11T21:25:30.949367+0000","last_active":"2024-03-11T21:26:28.160761+0000","last_peered":"2024-03-11T21:26:28.160761+0000","last_clean":"2024-03-11T21:26:28.160761+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:26:28.160761+0000","last_undegraded":"2024-03-11T21:26:28.160761+0000","last_fullsized":"2024-03-11T21:26:28.160761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:30.949072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.949072+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:33.375300+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:29.967476+0000","last_change":"2024-03-11T21:26:29.967551+0000","last_active":"2024-03-11T21:26:29.967476+0000","last_peered":"2024-03-11T21:26:29.967476+0000","last_clean":"2024-03-11T21:26:29.967476+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:26:29.967476+0000","last_undegraded":"2024-03-11T21:26:29.967476+0000","last_fullsized":"2024-03-11T21:26:29.967476+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:26:29.967316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.967316+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.913576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":162,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161127+0000","last_change":"2024-03-11T21:26:05.978542+0000","last_active":"2024-03-11T21:26:28.161127+0000","last_peered":"2024-03-11T21:26:28.161127+0000","last_clean":"2024-03-11T21:26:28.161127+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:26:28.161127+0000","last_undegraded":"2024-03-11T21:26:28.161127+0000","last_fullsized":"2024-03-11T21:26:28.161127+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:05.978319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.978319+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.180257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.154939+0000","last_change":"2024-03-11T21:26:02.471008+0000","last_active":"2024-03-11T21:26:28.154939+0000","last_peered":"2024-03-11T21:26:28.154939+0000","last_clean":"2024-03-11T21:26:28.154939+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:26:28.154939+0000","last_undegraded":"2024-03-11T21:26:28.154939+0000","last_fullsized":"2024-03-11T21:26:28.154939+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.470792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.470792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.529057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":135,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655293+0000","last_change":"2024-03-11T21:26:03.728864+0000","last_active":"2024-03-11T21:26:27.655293+0000","last_peered":"2024-03-11T21:26:27.655293+0000","last_clean":"2024-03-11T21:26:27.655293+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:26:27.655293+0000","last_undegraded":"2024-03-11T21:26:27.655293+0000","last_fullsized":"2024-03-11T21:26:27.655293+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:03.728766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.728766+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.097957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.313657+0000","last_change":"2024-03-11T21:26:28.313721+0000","last_active":"2024-03-11T21:26:28.313657+0000","last_peered":"2024-03-11T21:26:28.313657+0000","last_clean":"2024-03-11T21:26:28.313657+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:26:28.313657+0000","last_undegraded":"2024-03-11T21:26:28.313657+0000","last_fullsized":"2024-03-11T21:26:28.313657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:26:28.313489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.975519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157122+0000","last_change":"2024-03-11T21:26:20.986742+0000","last_active":"2024-03-11T21:26:28.157122+0000","last_peered":"2024-03-11T21:26:28.157122+0000","last_clean":"2024-03-11T21:26:28.157122+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:26:28.157122+0000","last_undegraded":"2024-03-11T21:26:28.157122+0000","last_fullsized":"2024-03-11T21:26:28.157122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.986534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.986534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.313433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159785+0000","last_change":"2024-03-11T21:25:40.946608+0000","last_active":"2024-03-11T21:26:28.159785+0000","last_peered":"2024-03-11T21:26:28.159785+0000","last_clean":"2024-03-11T21:26:28.159785+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:26:28.159785+0000","last_undegraded":"2024-03-11T21:26:28.159785+0000","last_fullsized":"2024-03-11T21:26:28.159785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.946307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158034+0000","last_change":"2024-03-11T21:25:27.506732+0000","last_active":"2024-03-11T21:26:28.158034+0000","last_peered":"2024-03-11T21:26:28.158034+0000","last_clean":"2024-03-11T21:26:28.158034+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:26:28.158034+0000","last_undegraded":"2024-03-11T21:26:28.158034+0000","last_fullsized":"2024-03-11T21:26:28.158034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:27.506461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.506461+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.042628+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157904+0000","last_change":"2024-03-11T21:25:53.051632+0000","last_active":"2024-03-11T21:26:28.157904+0000","last_peered":"2024-03-11T21:26:28.157904+0000","last_clean":"2024-03-11T21:26:28.157904+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:26:28.157904+0000","last_undegraded":"2024-03-11T21:26:28.157904+0000","last_fullsized":"2024-03-11T21:26:28.157904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.051422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.051422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.051498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:29.972866+0000","last_change":"2024-03-11T21:26:29.972923+0000","last_active":"2024-03-11T21:26:29.972866+0000","last_peered":"2024-03-11T21:26:29.972866+0000","last_clean":"2024-03-11T21:26:29.972866+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:26:29.972866+0000","last_undegraded":"2024-03-11T21:26:29.972866+0000","last_fullsized":"2024-03-11T21:26:29.972866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.972683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.497791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158883+0000","last_change":"2024-03-11T21:25:48.374372+0000","last_active":"2024-03-11T21:26:28.158883+0000","last_peered":"2024-03-11T21:26:28.158883+0000","last_clean":"2024-03-11T21:26:28.158883+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:26:28.158883+0000","last_undegraded":"2024-03-11T21:26:28.158883+0000","last_fullsized":"2024-03-11T21:26:28.158883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.374158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.374158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.742286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":119,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161755+0000","last_change":"2024-03-11T21:25:37.334081+0000","last_active":"2024-03-11T21:26:28.161755+0000","last_peered":"2024-03-11T21:26:28.161755+0000","last_clean":"2024-03-11T21:26:28.161755+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:26:28.161755+0000","last_undegraded":"2024-03-11T21:26:28.161755+0000","last_fullsized":"2024-03-11T21:26:28.161755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:25:37.333856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.333856+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.493865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162837+0000","last_change":"2024-03-11T21:25:56.141805+0000","last_active":"2024-03-11T21:26:28.162837+0000","last_peered":"2024-03-11T21:26:28.162837+0000","last_clean":"2024-03-11T21:26:28.162837+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:26:28.162837+0000","last_undegraded":"2024-03-11T21:26:28.162837+0000","last_fullsized":"2024-03-11T21:26:28.162837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.141559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.141559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.340511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.657776+0000","last_change":"2024-03-11T21:26:14.593548+0000","last_active":"2024-03-11T21:26:27.657776+0000","last_peered":"2024-03-11T21:26:27.657776+0000","last_clean":"2024-03-11T21:26:27.657776+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:26:27.657776+0000","last_undegraded":"2024-03-11T21:26:27.657776+0000","last_fullsized":"2024-03-11T21:26:27.657776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.593259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.593259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.052839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.658748+0000","last_change":"2024-03-11T21:25:53.706422+0000","last_active":"2024-03-11T21:26:27.658748+0000","last_peered":"2024-03-11T21:26:27.658748+0000","last_clean":"2024-03-11T21:26:27.658748+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:26:27.658748+0000","last_undegraded":"2024-03-11T21:26:27.658748+0000","last_fullsized":"2024-03-11T21:26:27.658748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.706230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.384575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651087+0000","last_change":"2024-03-11T21:25:38.984101+0000","last_active":"2024-03-11T21:26:27.651087+0000","last_peered":"2024-03-11T21:26:27.651087+0000","last_clean":"2024-03-11T21:26:27.651087+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:26:27.651087+0000","last_undegraded":"2024-03-11T21:26:27.651087+0000","last_fullsized":"2024-03-11T21:26:27.651087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:38.983858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.983858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.131782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159287+0000","last_change":"2024-03-11T21:25:56.344253+0000","last_active":"2024-03-11T21:26:28.159287+0000","last_peered":"2024-03-11T21:26:28.159287+0000","last_clean":"2024-03-11T21:26:28.159287+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:26:28.159287+0000","last_undegraded":"2024-03-11T21:26:28.159287+0000","last_fullsized":"2024-03-11T21:26:28.159287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.343979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.343979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.628275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161092+0000","last_change":"2024-03-11T21:25:39.974174+0000","last_active":"2024-03-11T21:26:28.161092+0000","last_peered":"2024-03-11T21:26:28.161092+0000","last_clean":"2024-03-11T21:26:28.161092+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:26:28.161092+0000","last_undegraded":"2024-03-11T21:26:28.161092+0000","last_fullsized":"2024-03-11T21:26:28.161092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.973978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.973978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.092042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158481+0000","last_change":"2024-03-11T21:25:45.890051+0000","last_active":"2024-03-11T21:26:28.158481+0000","last_peered":"2024-03-11T21:26:28.158481+0000","last_clean":"2024-03-11T21:26:28.158481+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:26:28.158481+0000","last_undegraded":"2024-03-11T21:26:28.158481+0000","last_fullsized":"2024-03-11T21:26:28.158481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:45.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.889834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.143004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160798+0000","last_change":"2024-03-11T21:25:37.960424+0000","last_active":"2024-03-11T21:26:28.160798+0000","last_peered":"2024-03-11T21:26:28.160798+0000","last_clean":"2024-03-11T21:26:28.160798+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:26:28.160798+0000","last_undegraded":"2024-03-11T21:26:28.160798+0000","last_fullsized":"2024-03-11T21:26:28.160798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:25:37.960057+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.960057+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.161700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.652218+0000","last_change":"2024-03-11T21:25:40.949182+0000","last_active":"2024-03-11T21:26:27.652218+0000","last_peered":"2024-03-11T21:26:27.652218+0000","last_clean":"2024-03-11T21:26:27.652218+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:26:27.652218+0000","last_undegraded":"2024-03-11T21:26:27.652218+0000","last_fullsized":"2024-03-11T21:26:27.652218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.948963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.948963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.061308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'11","reported_seq":159,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162470+0000","last_change":"2024-03-11T21:25:38.973030+0000","last_active":"2024-03-11T21:26:28.162470+0000","last_peered":"2024-03-11T21:26:28.162470+0000","last_clean":"2024-03-11T21:26:28.162470+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:26:28.162470+0000","last_undegraded":"2024-03-11T21:26:28.162470+0000","last_fullsized":"2024-03-11T21:26:28.162470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:38.972802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.972802+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.415779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163679+0000","last_change":"2024-03-11T21:25:54.342862+0000","last_active":"2024-03-11T21:26:28.163679+0000","last_peered":"2024-03-11T21:26:28.163679+0000","last_clean":"2024-03-11T21:26:28.163679+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:26:28.163679+0000","last_undegraded":"2024-03-11T21:26:28.163679+0000","last_fullsized":"2024-03-11T21:26:28.163679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.342588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.028053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.164154+0000","last_change":"2024-03-11T21:26:01.314774+0000","last_active":"2024-03-11T21:26:28.164154+0000","last_peered":"2024-03-11T21:26:28.164154+0000","last_clean":"2024-03-11T21:26:28.164154+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:26:28.164154+0000","last_undegraded":"2024-03-11T21:26:28.164154+0000","last_fullsized":"2024-03-11T21:26:28.164154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:26:01.314615+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.665529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160665+0000","last_change":"2024-03-11T21:25:43.246214+0000","last_active":"2024-03-11T21:26:28.160665+0000","last_peered":"2024-03-11T21:26:28.160665+0000","last_clean":"2024-03-11T21:26:28.160665+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:26:28.160665+0000","last_undegraded":"2024-03-11T21:26:28.160665+0000","last_fullsized":"2024-03-11T21:26:28.160665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.246002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.246002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.409171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":159,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160588+0000","last_change":"2024-03-11T21:25:59.442248+0000","last_active":"2024-03-11T21:26:28.160588+0000","last_peered":"2024-03-11T21:26:28.160588+0000","last_clean":"2024-03-11T21:26:28.160588+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:26:28.160588+0000","last_undegraded":"2024-03-11T21:26:28.160588+0000","last_fullsized":"2024-03-11T21:26:28.160588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:59.442015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.442015+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.052362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155914+0000","last_change":"2024-03-11T21:26:27.752411+0000","last_active":"2024-03-11T21:26:28.155914+0000","last_peered":"2024-03-11T21:26:28.155914+0000","last_clean":"2024-03-11T21:26:28.155914+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:26:28.155914+0000","last_undegraded":"2024-03-11T21:26:28.155914+0000","last_fullsized":"2024-03-11T21:26:28.155914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.752198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.752198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:42.079868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159660+0000","last_change":"2024-03-11T21:25:39.009861+0000","last_active":"2024-03-11T21:26:28.159660+0000","last_peered":"2024-03-11T21:26:28.159660+0000","last_clean":"2024-03-11T21:26:28.159660+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:26:28.159660+0000","last_undegraded":"2024-03-11T21:26:28.159660+0000","last_fullsized":"2024-03-11T21:26:28.159660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:39.009635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.009635+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:46.254494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159414+0000","last_change":"2024-03-11T21:25:49.003277+0000","last_active":"2024-03-11T21:26:28.159414+0000","last_peered":"2024-03-11T21:26:28.159414+0000","last_clean":"2024-03-11T21:26:28.159414+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:26:28.159414+0000","last_undegraded":"2024-03-11T21:26:28.159414+0000","last_fullsized":"2024-03-11T21:26:28.159414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:49.003014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.003014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.285739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158320+0000","last_change":"2024-03-11T21:25:54.360923+0000","last_active":"2024-03-11T21:26:28.158320+0000","last_peered":"2024-03-11T21:26:28.158320+0000","last_clean":"2024-03-11T21:26:28.158320+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:26:28.158320+0000","last_undegraded":"2024-03-11T21:26:28.158320+0000","last_fullsized":"2024-03-11T21:26:28.158320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.360647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.360647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.912837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155177+0000","last_change":"2024-03-11T21:25:36.945869+0000","last_active":"2024-03-11T21:26:28.155177+0000","last_peered":"2024-03-11T21:26:28.155177+0000","last_clean":"2024-03-11T21:26:28.155177+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:26:28.155177+0000","last_undegraded":"2024-03-11T21:26:28.155177+0000","last_fullsized":"2024-03-11T21:26:28.155177+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:36.945583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:36.945583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.141176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":118,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.154812+0000","last_change":"2024-03-11T21:25:46.921649+0000","last_active":"2024-03-11T21:26:28.154812+0000","last_peered":"2024-03-11T21:26:28.154812+0000","last_clean":"2024-03-11T21:26:28.154812+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:26:28.154812+0000","last_undegraded":"2024-03-11T21:26:28.154812+0000","last_fullsized":"2024-03-11T21:26:28.154812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.921404+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.755306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.656138+0000","last_change":"2024-03-11T21:25:50.974118+0000","last_active":"2024-03-11T21:26:27.656138+0000","last_peered":"2024-03-11T21:26:27.656138+0000","last_clean":"2024-03-11T21:26:27.656138+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:26:27.656138+0000","last_undegraded":"2024-03-11T21:26:27.656138+0000","last_fullsized":"2024-03-11T21:26:27.656138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:50.974032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.974032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.283434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163192+0000","last_change":"2024-03-11T21:26:04.178398+0000","last_active":"2024-03-11T21:26:28.163192+0000","last_peered":"2024-03-11T21:26:28.163192+0000","last_clean":"2024-03-11T21:26:28.163192+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:26:28.163192+0000","last_undegraded":"2024-03-11T21:26:28.163192+0000","last_fullsized":"2024-03-11T21:26:28.163192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.178145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.178145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.425359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.653084+0000","last_change":"2024-03-11T21:25:29.104018+0000","last_active":"2024-03-11T21:26:27.653084+0000","last_peered":"2024-03-11T21:26:27.653084+0000","last_clean":"2024-03-11T21:26:27.653084+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:26:27.653084+0000","last_undegraded":"2024-03-11T21:26:27.653084+0000","last_fullsized":"2024-03-11T21:26:27.653084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:29.103775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.103775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:44.777028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157314+0000","last_change":"2024-03-11T21:26:10.162900+0000","last_active":"2024-03-11T21:26:28.157314+0000","last_peered":"2024-03-11T21:26:28.157314+0000","last_clean":"2024-03-11T21:26:28.157314+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:26:28.157314+0000","last_undegraded":"2024-03-11T21:26:28.157314+0000","last_fullsized":"2024-03-11T21:26:28.157314+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:10.162661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.162661+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:28.988891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161364+0000","last_change":"2024-03-11T21:25:27.983629+0000","last_active":"2024-03-11T21:26:28.161364+0000","last_peered":"2024-03-11T21:26:28.161364+0000","last_clean":"2024-03-11T21:26:28.161364+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:26:28.161364+0000","last_undegraded":"2024-03-11T21:26:28.161364+0000","last_fullsized":"2024-03-11T21:26:28.161364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.983536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.983536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.044284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651535+0000","last_change":"2024-03-11T21:25:37.981238+0000","last_active":"2024-03-11T21:26:27.651535+0000","last_peered":"2024-03-11T21:26:27.651535+0000","last_clean":"2024-03-11T21:26:27.651535+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:26:27.651535+0000","last_undegraded":"2024-03-11T21:26:27.651535+0000","last_fullsized":"2024-03-11T21:26:27.651535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:25:37.981019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.981019+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.806629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":134,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161099+0000","last_change":"2024-03-11T21:26:08.200786+0000","last_active":"2024-03-11T21:26:28.161099+0000","last_peered":"2024-03-11T21:26:28.161099+0000","last_clean":"2024-03-11T21:26:28.161099+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:26:28.161099+0000","last_undegraded":"2024-03-11T21:26:28.161099+0000","last_fullsized":"2024-03-11T21:26:28.161099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:26:08.200532+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:24:33.079623+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.200532+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.719717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.652919+0000","last_change":"2024-03-11T21:26:02.944301+0000","last_active":"2024-03-11T21:26:27.652919+0000","last_peered":"2024-03-11T21:26:27.652919+0000","last_clean":"2024-03-11T21:26:27.652919+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:26:27.652919+0000","last_undegraded":"2024-03-11T21:26:27.652919+0000","last_fullsized":"2024-03-11T21:26:27.652919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:21.682420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159314+0000","last_change":"2024-03-11T21:25:48.263693+0000","last_active":"2024-03-11T21:26:28.159314+0000","last_peered":"2024-03-11T21:26:28.159314+0000","last_clean":"2024-03-11T21:26:28.159314+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:26:28.159314+0000","last_undegraded":"2024-03-11T21:26:28.159314+0000","last_fullsized":"2024-03-11T21:26:28.159314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.263515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.384625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.659099+0000","last_change":"2024-03-11T21:25:33.757745+0000","last_active":"2024-03-11T21:26:27.659099+0000","last_peered":"2024-03-11T21:26:27.659099+0000","last_clean":"2024-03-11T21:26:27.659099+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:26:27.659099+0000","last_undegraded":"2024-03-11T21:26:27.659099+0000","last_fullsized":"2024-03-11T21:26:27.659099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:33.757442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:33.757442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.904053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":175,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160779+0000","last_change":"2024-03-11T21:25:21.393014+0000","last_active":"2024-03-11T21:26:28.160779+0000","last_peered":"2024-03-11T21:26:28.160779+0000","last_clean":"2024-03-11T21:26:28.160779+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:26:28.160779+0000","last_undegraded":"2024-03-11T21:26:28.160779+0000","last_fullsized":"2024-03-11T21:26:28.160779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:25:21.392637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:21.392637+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:43.705727+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":117,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.656677+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:26:27.656677+0000","last_peered":"2024-03-11T21:26:27.656677+0000","last_clean":"2024-03-11T21:26:27.656677+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:26:27.656677+0000","last_undegraded":"2024-03-11T21:26:27.656677+0000","last_fullsized":"2024-03-11T21:26:27.656677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.556814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"84'11","reported_seq":162,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.164743+0000","last_change":"2024-03-11T21:25:58.333052+0000","last_active":"2024-03-11T21:26:28.164743+0000","last_peered":"2024-03-11T21:26:28.164743+0000","last_clean":"2024-03-11T21:26:28.164743+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:26:28.164743+0000","last_undegraded":"2024-03-11T21:26:28.164743+0000","last_fullsized":"2024-03-11T21:26:28.164743+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:58.332816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.332816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.183920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:30.314997+0000","last_change":"2024-03-11T21:26:30.315059+0000","last_active":"2024-03-11T21:26:30.314997+0000","last_peered":"2024-03-11T21:26:30.314997+0000","last_clean":"2024-03-11T21:26:30.314997+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:26:30.314997+0000","last_undegraded":"2024-03-11T21:26:30.314997+0000","last_fullsized":"2024-03-11T21:26:30.314997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:30.314840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:30.314840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:39.460296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156206+0000","last_change":"2024-03-11T21:26:13.487586+0000","last_active":"2024-03-11T21:26:28.156206+0000","last_peered":"2024-03-11T21:26:28.156206+0000","last_clean":"2024-03-11T21:26:28.156206+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:26:28.156206+0000","last_undegraded":"2024-03-11T21:26:28.156206+0000","last_fullsized":"2024-03-11T21:26:28.156206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.487359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.487359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.520405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":154,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650125+0000","last_change":"2024-03-11T21:25:30.974808+0000","last_active":"2024-03-11T21:26:27.650125+0000","last_peered":"2024-03-11T21:26:27.650125+0000","last_clean":"2024-03-11T21:26:27.650125+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:26:27.650125+0000","last_undegraded":"2024-03-11T21:26:27.650125+0000","last_fullsized":"2024-03-11T21:26:27.650125+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.974544+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.198159+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.164000+0000","last_change":"2024-03-11T21:26:25.426916+0000","last_active":"2024-03-11T21:26:28.164000+0000","last_peered":"2024-03-11T21:26:28.164000+0000","last_clean":"2024-03-11T21:26:28.164000+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:26:28.164000+0000","last_undegraded":"2024-03-11T21:26:28.164000+0000","last_fullsized":"2024-03-11T21:26:28.164000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:25.426623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:25.426623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.790210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159705+0000","last_change":"2024-03-11T21:26:04.128265+0000","last_active":"2024-03-11T21:26:28.159705+0000","last_peered":"2024-03-11T21:26:28.159705+0000","last_clean":"2024-03-11T21:26:28.159705+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:26:28.159705+0000","last_undegraded":"2024-03-11T21:26:28.159705+0000","last_fullsized":"2024-03-11T21:26:28.159705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.128031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.128031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.667989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":142,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.657456+0000","last_change":"2024-03-11T21:25:56.736951+0000","last_active":"2024-03-11T21:26:27.657456+0000","last_peered":"2024-03-11T21:26:27.657456+0000","last_clean":"2024-03-11T21:26:27.657456+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:26:27.657456+0000","last_undegraded":"2024-03-11T21:26:27.657456+0000","last_fullsized":"2024-03-11T21:26:27.657456+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:25:56.736721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.736721+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.841976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.653540+0000","last_change":"2024-03-11T21:25:47.994296+0000","last_active":"2024-03-11T21:26:27.653540+0000","last_peered":"2024-03-11T21:26:27.653540+0000","last_clean":"2024-03-11T21:26:27.653540+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:26:27.653540+0000","last_undegraded":"2024-03-11T21:26:27.653540+0000","last_fullsized":"2024-03-11T21:26:27.653540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.994008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.994008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.859169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655908+0000","last_change":"2024-03-11T21:25:58.022948+0000","last_active":"2024-03-11T21:26:27.655908+0000","last_peered":"2024-03-11T21:26:27.655908+0000","last_clean":"2024-03-11T21:26:27.655908+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:26:27.655908+0000","last_undegraded":"2024-03-11T21:26:27.655908+0000","last_fullsized":"2024-03-11T21:26:27.655908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.022737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.022737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.306838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162053+0000","last_change":"2024-03-11T21:26:03.430155+0000","last_active":"2024-03-11T21:26:28.162053+0000","last_peered":"2024-03-11T21:26:28.162053+0000","last_clean":"2024-03-11T21:26:28.162053+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:26:28.162053+0000","last_undegraded":"2024-03-11T21:26:28.162053+0000","last_fullsized":"2024-03-11T21:26:28.162053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:03.429945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.429945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.219815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160720+0000","last_change":"2024-03-11T21:26:08.149166+0000","last_active":"2024-03-11T21:26:28.160720+0000","last_peered":"2024-03-11T21:26:28.160720+0000","last_clean":"2024-03-11T21:26:28.160720+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:26:28.160720+0000","last_undegraded":"2024-03-11T21:26:28.160720+0000","last_fullsized":"2024-03-11T21:26:28.160720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:08.148915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.148915+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.413377+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155903+0000","last_change":"2024-03-11T21:25:32.395539+0000","last_active":"2024-03-11T21:26:28.155903+0000","last_peered":"2024-03-11T21:26:28.155903+0000","last_clean":"2024-03-11T21:26:28.155903+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:26:28.155903+0000","last_undegraded":"2024-03-11T21:26:28.155903+0000","last_fullsized":"2024-03-11T21:26:28.155903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.395316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.615969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":376,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:36.758248+0000","last_change":"2024-03-11T21:26:36.758336+0000","last_active":"2024-03-11T21:26:36.758248+0000","last_peered":"2024-03-11T21:26:36.758248+0000","last_clean":"2024-03-11T21:26:36.758248+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:26:36.758248+0000","last_undegraded":"2024-03-11T21:26:36.758248+0000","last_fullsized":"2024-03-11T21:26:36.758248+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:26:36.758088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:26:36.758088+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:04.601201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:34.807245+0000","last_change":"2024-03-11T21:26:34.807309+0000","last_active":"2024-03-11T21:26:34.807245+0000","last_peered":"2024-03-11T21:26:34.807245+0000","last_clean":"2024-03-11T21:26:34.807245+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:26:34.807245+0000","last_undegraded":"2024-03-11T21:26:34.807245+0000","last_fullsized":"2024-03-11T21:26:34.807245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:34.807061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:34.807061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:56.122316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.658387+0000","last_change":"2024-03-11T21:26:10.602526+0000","last_active":"2024-03-11T21:26:27.658387+0000","last_peered":"2024-03-11T21:26:27.658387+0000","last_clean":"2024-03-11T21:26:27.658387+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:26:27.658387+0000","last_undegraded":"2024-03-11T21:26:27.658387+0000","last_fullsized":"2024-03-11T21:26:27.658387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.602230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.602230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:36.316832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":131,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159743+0000","last_change":"2024-03-11T21:26:13.337394+0000","last_active":"2024-03-11T21:26:28.159743+0000","last_peered":"2024-03-11T21:26:28.159743+0000","last_clean":"2024-03-11T21:26:28.159743+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:26:28.159743+0000","last_undegraded":"2024-03-11T21:26:28.159743+0000","last_fullsized":"2024-03-11T21:26:28.159743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:26:13.337090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.337090+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.002783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155894+0000","last_change":"2024-03-11T21:25:43.021158+0000","last_active":"2024-03-11T21:26:28.155894+0000","last_peered":"2024-03-11T21:26:28.155894+0000","last_clean":"2024-03-11T21:26:28.155894+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:26:28.155894+0000","last_undegraded":"2024-03-11T21:26:28.155894+0000","last_fullsized":"2024-03-11T21:26:28.155894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.020950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.020950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.379154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161540+0000","last_change":"2024-03-11T21:25:52.415680+0000","last_active":"2024-03-11T21:26:28.161540+0000","last_peered":"2024-03-11T21:26:28.161540+0000","last_clean":"2024-03-11T21:26:28.161540+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:26:28.161540+0000","last_undegraded":"2024-03-11T21:26:28.161540+0000","last_fullsized":"2024-03-11T21:26:28.161540+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.415468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.415468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.982658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.654399+0000","last_change":"2024-03-11T21:26:04.760536+0000","last_active":"2024-03-11T21:26:27.654399+0000","last_peered":"2024-03-11T21:26:27.654399+0000","last_clean":"2024-03-11T21:26:27.654399+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:26:27.654399+0000","last_undegraded":"2024-03-11T21:26:27.654399+0000","last_fullsized":"2024-03-11T21:26:27.654399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.760262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.760262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.318865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159971+0000","last_change":"2024-03-11T21:25:59.200564+0000","last_active":"2024-03-11T21:26:28.159971+0000","last_peered":"2024-03-11T21:26:28.159971+0000","last_clean":"2024-03-11T21:26:28.159971+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:26:28.159971+0000","last_undegraded":"2024-03-11T21:26:28.159971+0000","last_fullsized":"2024-03-11T21:26:28.159971+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.200270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.200270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.037981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":203,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156983+0000","last_change":"2024-03-11T21:25:37.669639+0000","last_active":"2024-03-11T21:26:28.156983+0000","last_peered":"2024-03-11T21:26:28.156983+0000","last_clean":"2024-03-11T21:26:28.156983+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:26:28.156983+0000","last_undegraded":"2024-03-11T21:26:28.156983+0000","last_fullsized":"2024-03-11T21:26:28.156983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:25:37.669221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.669221+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.744954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157283+0000","last_change":"2024-03-11T21:26:08.930659+0000","last_active":"2024-03-11T21:26:28.157283+0000","last_peered":"2024-03-11T21:26:28.157283+0000","last_clean":"2024-03-11T21:26:28.157283+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:26:28.157283+0000","last_undegraded":"2024-03-11T21:26:28.157283+0000","last_fullsized":"2024-03-11T21:26:28.157283+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:08.930415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.930415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.664164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649827+0000","last_change":"2024-03-11T21:26:02.037288+0000","last_active":"2024-03-11T21:26:27.649827+0000","last_peered":"2024-03-11T21:26:27.649827+0000","last_clean":"2024-03-11T21:26:27.649827+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:26:27.649827+0000","last_undegraded":"2024-03-11T21:26:27.649827+0000","last_fullsized":"2024-03-11T21:26:27.649827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.939102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":115,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.659582+0000","last_change":"2024-03-11T21:25:49.638326+0000","last_active":"2024-03-11T21:26:27.659582+0000","last_peered":"2024-03-11T21:26:27.659582+0000","last_clean":"2024-03-11T21:26:27.659582+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:26:27.659582+0000","last_undegraded":"2024-03-11T21:26:27.659582+0000","last_fullsized":"2024-03-11T21:26:27.659582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:49.638061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.638061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.499420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":137,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163387+0000","last_change":"2024-03-11T21:26:24.352686+0000","last_active":"2024-03-11T21:26:28.163387+0000","last_peered":"2024-03-11T21:26:28.163387+0000","last_clean":"2024-03-11T21:26:28.163387+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:26:28.163387+0000","last_undegraded":"2024-03-11T21:26:28.163387+0000","last_fullsized":"2024-03-11T21:26:28.163387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:26:24.352407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:24.352407+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.744324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":313,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:36.088471+0000","last_change":"2024-03-11T21:25:50.006793+0000","last_active":"2024-03-11T21:26:36.088471+0000","last_peered":"2024-03-11T21:26:36.088471+0000","last_clean":"2024-03-11T21:26:36.088471+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:26:36.088471+0000","last_undegraded":"2024-03-11T21:26:36.088471+0000","last_fullsized":"2024-03-11T21:26:36.088471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:25:50.006698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.006698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.269418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649776+0000","last_change":"2024-03-11T21:25:57.992113+0000","last_active":"2024-03-11T21:26:27.649776+0000","last_peered":"2024-03-11T21:26:27.649776+0000","last_clean":"2024-03-11T21:26:27.649776+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:26:27.649776+0000","last_undegraded":"2024-03-11T21:26:27.649776+0000","last_fullsized":"2024-03-11T21:26:27.649776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.991803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.991803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.426825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159117+0000","last_change":"2024-03-11T21:25:34.059053+0000","last_active":"2024-03-11T21:26:28.159117+0000","last_peered":"2024-03-11T21:26:28.159117+0000","last_clean":"2024-03-11T21:26:28.159117+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:26:28.159117+0000","last_undegraded":"2024-03-11T21:26:28.159117+0000","last_fullsized":"2024-03-11T21:26:28.159117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.058815+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.431730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"77'1","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159517+0000","last_change":"2024-03-11T21:25:43.867607+0000","last_active":"2024-03-11T21:26:28.159517+0000","last_peered":"2024-03-11T21:26:28.159517+0000","last_clean":"2024-03-11T21:26:28.159517+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:26:28.159517+0000","last_undegraded":"2024-03-11T21:26:28.159517+0000","last_fullsized":"2024-03-11T21:26:28.159517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:43.867377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.867377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.362412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":136,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:32.528865+0000","last_change":"2024-03-11T21:26:32.529005+0000","last_active":"2024-03-11T21:26:32.528865+0000","last_peered":"2024-03-11T21:26:32.528865+0000","last_clean":"2024-03-11T21:26:32.528865+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:26:32.528865+0000","last_undegraded":"2024-03-11T21:26:32.528865+0000","last_fullsized":"2024-03-11T21:26:32.528865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:32.528639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:32.528639+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.761031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":115,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650328+0000","last_change":"2024-03-11T21:25:47.898533+0000","last_active":"2024-03-11T21:26:27.650328+0000","last_peered":"2024-03-11T21:26:27.650328+0000","last_clean":"2024-03-11T21:26:27.650328+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:26:27.650328+0000","last_undegraded":"2024-03-11T21:26:27.650328+0000","last_fullsized":"2024-03-11T21:26:27.650328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.898322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.898322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.986939+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":159,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161517+0000","last_change":"2024-03-11T21:25:45.984353+0000","last_active":"2024-03-11T21:26:28.161517+0000","last_peered":"2024-03-11T21:26:28.161517+0000","last_clean":"2024-03-11T21:26:28.161517+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:26:28.161517+0000","last_undegraded":"2024-03-11T21:26:28.161517+0000","last_fullsized":"2024-03-11T21:26:28.161517+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:45.984098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.984098+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.447890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161026+0000","last_change":"2024-03-11T21:26:14.197969+0000","last_active":"2024-03-11T21:26:28.161026+0000","last_peered":"2024-03-11T21:26:28.161026+0000","last_clean":"2024-03-11T21:26:28.161026+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:26:28.161026+0000","last_undegraded":"2024-03-11T21:26:28.161026+0000","last_fullsized":"2024-03-11T21:26:28.161026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.197767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.197767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:40.681392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161809+0000","last_change":"2024-03-11T21:25:58.434898+0000","last_active":"2024-03-11T21:26:28.161809+0000","last_peered":"2024-03-11T21:26:28.161809+0000","last_clean":"2024-03-11T21:26:28.161809+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:26:28.161809+0000","last_undegraded":"2024-03-11T21:26:28.161809+0000","last_fullsized":"2024-03-11T21:26:28.161809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.434794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.434794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156707+0000","last_change":"2024-03-11T21:25:52.032818+0000","last_active":"2024-03-11T21:26:28.156707+0000","last_peered":"2024-03-11T21:26:28.156707+0000","last_clean":"2024-03-11T21:26:28.156707+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:26:28.156707+0000","last_undegraded":"2024-03-11T21:26:28.156707+0000","last_fullsized":"2024-03-11T21:26:28.156707+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.032606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.032606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.435492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":138,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650094+0000","last_change":"2024-03-11T21:26:17.927107+0000","last_active":"2024-03-11T21:26:27.650094+0000","last_peered":"2024-03-11T21:26:27.650094+0000","last_clean":"2024-03-11T21:26:27.650094+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:26:27.650094+0000","last_undegraded":"2024-03-11T21:26:27.650094+0000","last_fullsized":"2024-03-11T21:26:27.650094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:26:17.926863+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:26:17.926863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.586229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159293+0000","last_change":"2024-03-11T21:25:41.358918+0000","last_active":"2024-03-11T21:26:28.159293+0000","last_peered":"2024-03-11T21:26:28.159293+0000","last_clean":"2024-03-11T21:26:28.159293+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:26:28.159293+0000","last_undegraded":"2024-03-11T21:26:28.159293+0000","last_fullsized":"2024-03-11T21:26:28.159293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.358679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.152690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":159,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.152082+0000","last_change":"2024-03-11T21:25:57.003681+0000","last_active":"2024-03-11T21:26:28.152082+0000","last_peered":"2024-03-11T21:26:28.152082+0000","last_clean":"2024-03-11T21:26:28.152082+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:26:28.152082+0000","last_undegraded":"2024-03-11T21:26:28.152082+0000","last_fullsized":"2024-03-11T21:26:28.152082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:57.003432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.003432+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.840351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655032+0000","last_change":"2024-03-11T21:25:34.710182+0000","last_active":"2024-03-11T21:26:27.655032+0000","last_peered":"2024-03-11T21:26:27.655032+0000","last_clean":"2024-03-11T21:26:27.655032+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:26:27.655032+0000","last_undegraded":"2024-03-11T21:26:27.655032+0000","last_fullsized":"2024-03-11T21:26:27.655032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:34.709964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.709964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.742096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":135,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160291+0000","last_change":"2024-03-11T21:25:40.954211+0000","last_active":"2024-03-11T21:26:28.160291+0000","last_peered":"2024-03-11T21:26:28.160291+0000","last_clean":"2024-03-11T21:26:28.160291+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:26:28.160291+0000","last_undegraded":"2024-03-11T21:26:28.160291+0000","last_fullsized":"2024-03-11T21:26:28.160291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:40.953969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.953969+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.832583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651601+0000","last_change":"2024-03-11T21:26:05.072908+0000","last_active":"2024-03-11T21:26:27.651601+0000","last_peered":"2024-03-11T21:26:27.651601+0000","last_clean":"2024-03-11T21:26:27.651601+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:26:27.651601+0000","last_undegraded":"2024-03-11T21:26:27.651601+0000","last_fullsized":"2024-03-11T21:26:27.651601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.072676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.072676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.590231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":160,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157239+0000","last_change":"2024-03-11T21:25:40.987597+0000","last_active":"2024-03-11T21:26:28.157239+0000","last_peered":"2024-03-11T21:26:28.157239+0000","last_clean":"2024-03-11T21:26:28.157239+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:26:28.157239+0000","last_undegraded":"2024-03-11T21:26:28.157239+0000","last_fullsized":"2024-03-11T21:26:28.157239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:25:40.987394+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.987394+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.827259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.656037+0000","last_change":"2024-03-11T21:26:07.649240+0000","last_active":"2024-03-11T21:26:27.656037+0000","last_peered":"2024-03-11T21:26:27.656037+0000","last_clean":"2024-03-11T21:26:27.656037+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:26:27.656037+0000","last_undegraded":"2024-03-11T21:26:27.656037+0000","last_fullsized":"2024-03-11T21:26:27.656037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:07.649133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:26:07.649133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.066187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160791+0000","last_change":"2024-03-11T21:25:29.024987+0000","last_active":"2024-03-11T21:26:28.160791+0000","last_peered":"2024-03-11T21:26:28.160791+0000","last_clean":"2024-03-11T21:26:28.160791+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:26:28.160791+0000","last_undegraded":"2024-03-11T21:26:28.160791+0000","last_fullsized":"2024-03-11T21:26:28.160791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:29.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.024617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.368987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655919+0000","last_change":"2024-03-11T21:25:32.119545+0000","last_active":"2024-03-11T21:26:27.655919+0000","last_peered":"2024-03-11T21:26:27.655919+0000","last_clean":"2024-03-11T21:26:27.655919+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:26:27.655919+0000","last_undegraded":"2024-03-11T21:26:27.655919+0000","last_fullsized":"2024-03-11T21:26:27.655919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.119316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.119316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.928370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.658417+0000","last_change":"2024-03-11T21:26:12.573500+0000","last_active":"2024-03-11T21:26:27.658417+0000","last_peered":"2024-03-11T21:26:27.658417+0000","last_clean":"2024-03-11T21:26:27.658417+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:26:27.658417+0000","last_undegraded":"2024-03-11T21:26:27.658417+0000","last_fullsized":"2024-03-11T21:26:27.658417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.573292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.573292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.647582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159401+0000","last_change":"2024-03-11T21:25:28.167250+0000","last_active":"2024-03-11T21:26:28.159401+0000","last_peered":"2024-03-11T21:26:28.159401+0000","last_clean":"2024-03-11T21:26:28.159401+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:26:28.159401+0000","last_undegraded":"2024-03-11T21:26:28.159401+0000","last_fullsized":"2024-03-11T21:26:28.159401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:28.167038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:28.167038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:41.505025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":150,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155514+0000","last_change":"2024-03-11T21:26:02.945068+0000","last_active":"2024-03-11T21:26:28.155514+0000","last_peered":"2024-03-11T21:26:28.155514+0000","last_clean":"2024-03-11T21:26:28.155514+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:26:28.155514+0000","last_undegraded":"2024-03-11T21:26:28.155514+0000","last_fullsized":"2024-03-11T21:26:28.155514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:26:02.944783+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944783+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.686877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156162+0000","last_change":"2024-03-11T21:26:12.157386+0000","last_active":"2024-03-11T21:26:28.156162+0000","last_peered":"2024-03-11T21:26:28.156162+0000","last_clean":"2024-03-11T21:26:28.156162+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:26:28.156162+0000","last_undegraded":"2024-03-11T21:26:28.156162+0000","last_fullsized":"2024-03-11T21:26:28.156162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.157160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.157160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.057339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655026+0000","last_change":"2024-03-11T21:25:30.130283+0000","last_active":"2024-03-11T21:26:27.655026+0000","last_peered":"2024-03-11T21:26:27.655026+0000","last_clean":"2024-03-11T21:26:27.655026+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:26:27.655026+0000","last_undegraded":"2024-03-11T21:26:27.655026+0000","last_fullsized":"2024-03-11T21:26:27.655026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.130050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:38.236058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.655349+0000","last_change":"2024-03-11T21:25:44.013839+0000","last_active":"2024-03-11T21:26:27.655349+0000","last_peered":"2024-03-11T21:26:27.655349+0000","last_clean":"2024-03-11T21:26:27.655349+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:26:27.655349+0000","last_undegraded":"2024-03-11T21:26:27.655349+0000","last_fullsized":"2024-03-11T21:26:27.655349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:44.013746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:44.013746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.618233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":146,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160504+0000","last_change":"2024-03-11T21:26:14.127753+0000","last_active":"2024-03-11T21:26:28.160504+0000","last_peered":"2024-03-11T21:26:28.160504+0000","last_clean":"2024-03-11T21:26:28.160504+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:26:28.160504+0000","last_undegraded":"2024-03-11T21:26:28.160504+0000","last_fullsized":"2024-03-11T21:26:28.160504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:26:14.127405+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.127405+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:31.505662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159533+0000","last_change":"2024-03-11T21:26:15.354895+0000","last_active":"2024-03-11T21:26:28.159533+0000","last_peered":"2024-03-11T21:26:28.159533+0000","last_clean":"2024-03-11T21:26:28.159533+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:26:28.159533+0000","last_undegraded":"2024-03-11T21:26:28.159533+0000","last_fullsized":"2024-03-11T21:26:28.159533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:15.354676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:26:15.354676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.022398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.654509+0000","last_change":"2024-03-11T21:25:31.137161+0000","last_active":"2024-03-11T21:26:27.654509+0000","last_peered":"2024-03-11T21:26:27.654509+0000","last_clean":"2024-03-11T21:26:27.654509+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:26:27.654509+0000","last_undegraded":"2024-03-11T21:26:27.654509+0000","last_fullsized":"2024-03-11T21:26:27.654509+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:31.137035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.137035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:40.158341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.659028+0000","last_change":"2024-03-11T21:25:30.696848+0000","last_active":"2024-03-11T21:26:27.659028+0000","last_peered":"2024-03-11T21:26:27.659028+0000","last_clean":"2024-03-11T21:26:27.659028+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:26:27.659028+0000","last_undegraded":"2024-03-11T21:26:27.659028+0000","last_fullsized":"2024-03-11T21:26:27.659028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.696738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.696738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.966425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159545+0000","last_change":"2024-03-11T21:25:45.376793+0000","last_active":"2024-03-11T21:26:28.159545+0000","last_peered":"2024-03-11T21:26:28.159545+0000","last_clean":"2024-03-11T21:26:28.159545+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:26:28.159545+0000","last_undegraded":"2024-03-11T21:26:28.159545+0000","last_fullsized":"2024-03-11T21:26:28.159545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:45.376408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.376408+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.150185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155781+0000","last_change":"2024-03-11T21:26:05.391058+0000","last_active":"2024-03-11T21:26:28.155781+0000","last_peered":"2024-03-11T21:26:28.155781+0000","last_clean":"2024-03-11T21:26:28.155781+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:26:28.155781+0000","last_undegraded":"2024-03-11T21:26:28.155781+0000","last_fullsized":"2024-03-11T21:26:28.155781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.390850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.390850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.856109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.656527+0000","last_change":"2024-03-11T21:25:22.246229+0000","last_active":"2024-03-11T21:26:27.656527+0000","last_peered":"2024-03-11T21:26:27.656527+0000","last_clean":"2024-03-11T21:26:27.656527+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:26:27.656527+0000","last_undegraded":"2024-03-11T21:26:27.656527+0000","last_fullsized":"2024-03-11T21:26:27.656527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:22.245940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.245940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.310931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":135,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:31.873739+0000","last_change":"2024-03-11T21:26:31.873778+0000","last_active":"2024-03-11T21:26:31.873739+0000","last_peered":"2024-03-11T21:26:31.873739+0000","last_clean":"2024-03-11T21:26:31.873739+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:26:31.873739+0000","last_undegraded":"2024-03-11T21:26:31.873739+0000","last_fullsized":"2024-03-11T21:26:31.873739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:31.873666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.873666+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.702452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":159,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:29.009973+0000","last_change":"2024-03-11T21:26:29.009973+0000","last_active":"2024-03-11T21:26:29.009973+0000","last_peered":"2024-03-11T21:26:29.009973+0000","last_clean":"2024-03-11T21:26:29.009973+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:26:29.009973+0000","last_undegraded":"2024-03-11T21:26:29.009973+0000","last_fullsized":"2024-03-11T21:26:29.009973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:29.009718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.009718+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.739503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841590,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36672,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708480,"statfs":{"total":95995035648,"available":95957483520,"internally_reserved":0,"allocated":1368064,"data_stored":989761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168639,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28320,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716832,"statfs":{"total":95995035648,"available":95966035968,"internally_reserved":0,"allocated":1335296,"data_stored":987541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":749568,"data_stored":397055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36652,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708500,"statfs":{"total":95995035648,"available":95957504000,"internally_reserved":0,"allocated":1347584,"data_stored":988658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378729,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":745472,"data_stored":397633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":397111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855443,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":778240,"data_stored":398580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495689,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31896,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":737280,"data_stored":397093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:26:41.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:26:41.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:40.097908+0000 osd.4 (osd.4) 73 : cluster 0 3.3 scrub starts 2024-03-11T21:26:41.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:40.102751+0000 osd.4 (osd.4) 74 : cluster 0 3.3 scrub ok 2024-03-11T21:26:41.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: audit 2024-03-11T21:26:40.171615+0000 mgr.y (mgr.14550) 136 : audit 0 from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:40.719391+0000 osd.1 (osd.1) 87 : cluster 0 3.4 scrub starts 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:40.724965+0000 osd.1 (osd.1) 88 : cluster 0 3.4 scrub ok 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.248396+0000 osd.3 (osd.3) 103 : cluster 0 2.19 scrub starts 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.250130+0000 osd.3 (osd.3) 104 : cluster 0 2.19 scrub ok 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:40.097908+0000 osd.4 (osd.4) 73 : cluster 0 3.3 scrub starts 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:40.102751+0000 osd.4 (osd.4) 74 : cluster 0 3.3 scrub ok 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: audit 2024-03-11T21:26:40.171615+0000 mgr.y (mgr.14550) 136 : audit 0 from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:40.719391+0000 osd.1 (osd.1) 87 : cluster 0 3.4 scrub starts 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:40.724965+0000 osd.1 (osd.1) 88 : cluster 0 3.4 scrub ok 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.248396+0000 osd.3 (osd.3) 103 : cluster 0 2.19 scrub starts 2024-03-11T21:26:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:41 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.250130+0000 osd.3 (osd.3) 104 : cluster 0 2.19 scrub ok 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:40.097908+0000 osd.4 (osd.4) 73 : cluster 0 3.3 scrub starts 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:40.102751+0000 osd.4 (osd.4) 74 : cluster 0 3.3 scrub ok 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: audit 2024-03-11T21:26:40.171615+0000 mgr.y (mgr.14550) 136 : audit 0 from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:40.719391+0000 osd.1 (osd.1) 87 : cluster 0 3.4 scrub starts 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:40.724965+0000 osd.1 (osd.1) 88 : cluster 0 3.4 scrub ok 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.248396+0000 osd.3 (osd.3) 103 : cluster 0 2.19 scrub starts 2024-03-11T21:26:42.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:41 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.250130+0000 osd.3 (osd.3) 104 : cluster 0 2.19 scrub ok 2024-03-11T21:26:42.435 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:26:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:42.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.119577+0000 mgr.y (mgr.14550) 137 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:42.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.715727+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T21:26:42.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.718061+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T21:26:42.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.119577+0000 mgr.y (mgr.14550) 137 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:42.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.715727+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T21:26:42.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:42 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.718061+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T21:26:43.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:42 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.119577+0000 mgr.y (mgr.14550) 137 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:43.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:42 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.715727+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T21:26:43.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:42 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.718061+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T21:26:43.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.904702+0000 osd.2 (osd.2) 61 : cluster 0 5.a scrub starts 2024-03-11T21:26:43.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[21512]: cluster 2024-03-11T21:26:41.906670+0000 osd.2 (osd.2) 62 : cluster 0 5.a scrub ok 2024-03-11T21:26:43.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[21512]: audit 2024-03-11T21:26:42.743292+0000 mon.c (mon.2) 84 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.904702+0000 osd.2 (osd.2) 61 : cluster 0 5.a scrub starts 2024-03-11T21:26:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[29284]: cluster 2024-03-11T21:26:41.906670+0000 osd.2 (osd.2) 62 : cluster 0 5.a scrub ok 2024-03-11T21:26:43.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:43 smithi143 bash[29284]: audit 2024-03-11T21:26:42.743292+0000 mon.c (mon.2) 84 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:44.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:43 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.904702+0000 osd.2 (osd.2) 61 : cluster 0 5.a scrub starts 2024-03-11T21:26:44.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:43 smithi203 bash[23654]: cluster 2024-03-11T21:26:41.906670+0000 osd.2 (osd.2) 62 : cluster 0 5.a scrub ok 2024-03-11T21:26:44.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:43 smithi203 bash[23654]: audit 2024-03-11T21:26:42.743292+0000 mon.c (mon.2) 84 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[21512]: cluster 2024-03-11T21:26:43.121013+0000 mgr.y (mgr.14550) 138 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[21512]: cluster 2024-03-11T21:26:43.868816+0000 osd.6 (osd.6) 55 : cluster 0 2.11 scrub starts 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[21512]: cluster 2024-03-11T21:26:43.870992+0000 osd.6 (osd.6) 56 : cluster 0 2.11 scrub ok 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[29284]: cluster 2024-03-11T21:26:43.121013+0000 mgr.y (mgr.14550) 138 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[29284]: cluster 2024-03-11T21:26:43.868816+0000 osd.6 (osd.6) 55 : cluster 0 2.11 scrub starts 2024-03-11T21:26:44.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:44 smithi143 bash[29284]: cluster 2024-03-11T21:26:43.870992+0000 osd.6 (osd.6) 56 : cluster 0 2.11 scrub ok 2024-03-11T21:26:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:44 smithi203 bash[23654]: cluster 2024-03-11T21:26:43.121013+0000 mgr.y (mgr.14550) 138 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:45.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:44 smithi203 bash[23654]: cluster 2024-03-11T21:26:43.868816+0000 osd.6 (osd.6) 55 : cluster 0 2.11 scrub starts 2024-03-11T21:26:45.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:44 smithi203 bash[23654]: cluster 2024-03-11T21:26:43.870992+0000 osd.6 (osd.6) 56 : cluster 0 2.11 scrub ok 2024-03-11T21:26:45.818 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[29284]: cluster 2024-03-11T21:26:45.121692+0000 mgr.y (mgr.14550) 139 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[29284]: cluster 2024-03-11T21:26:45.682402+0000 osd.5 (osd.5) 93 : cluster 0 3.5 scrub starts 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[29284]: cluster 2024-03-11T21:26:45.687234+0000 osd.5 (osd.5) 94 : cluster 0 3.5 scrub ok 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[21512]: cluster 2024-03-11T21:26:45.121692+0000 mgr.y (mgr.14550) 139 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[21512]: cluster 2024-03-11T21:26:45.682402+0000 osd.5 (osd.5) 93 : cluster 0 3.5 scrub starts 2024-03-11T21:26:46.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:46 smithi143 bash[21512]: cluster 2024-03-11T21:26:45.687234+0000 osd.5 (osd.5) 94 : cluster 0 3.5 scrub ok 2024-03-11T21:26:47.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:46 smithi203 bash[23654]: cluster 2024-03-11T21:26:45.121692+0000 mgr.y (mgr.14550) 139 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:47.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:46 smithi203 bash[23654]: cluster 2024-03-11T21:26:45.682402+0000 osd.5 (osd.5) 93 : cluster 0 3.5 scrub starts 2024-03-11T21:26:47.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:46 smithi203 bash[23654]: cluster 2024-03-11T21:26:45.687234+0000 osd.5 (osd.5) 94 : cluster 0 3.5 scrub ok 2024-03-11T21:26:47.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:45.972616+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:45.975110+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:46.778158+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:46.780372+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:46.932581+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[29284]: cluster 2024-03-11T21:26:46.932692+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:45.972616+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:26:47.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:45.975110+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:26:47.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:46.778158+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-03-11T21:26:47.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:46.780372+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-03-11T21:26:47.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:46.932581+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:47.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:47 smithi143 bash[21512]: cluster 2024-03-11T21:26:46.932692+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:45.972616+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:26:48.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:45.975110+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:26:48.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:46.778158+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-03-11T21:26:48.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:46.780372+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-03-11T21:26:48.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:46.932581+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:48.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:47 smithi203 bash[23654]: cluster 2024-03-11T21:26:46.932692+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:48.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:48.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[21512]: cluster 2024-03-11T21:26:47.122251+0000 mgr.y (mgr.14550) 140 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:48.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[21512]: cluster 2024-03-11T21:26:47.651865+0000 mon.a (mon.0) 880 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:26:48.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[21512]: cluster 2024-03-11T21:26:48.274114+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-03-11T21:26:48.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[21512]: cluster 2024-03-11T21:26:48.277899+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-03-11T21:26:48.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[29284]: cluster 2024-03-11T21:26:47.122251+0000 mgr.y (mgr.14550) 140 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:48.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[29284]: cluster 2024-03-11T21:26:47.651865+0000 mon.a (mon.0) 880 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:26:48.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[29284]: cluster 2024-03-11T21:26:48.274114+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-03-11T21:26:48.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:48 smithi143 bash[29284]: cluster 2024-03-11T21:26:48.277899+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-03-11T21:26:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:48 smithi203 bash[23654]: cluster 2024-03-11T21:26:47.122251+0000 mgr.y (mgr.14550) 140 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:48 smithi203 bash[23654]: cluster 2024-03-11T21:26:47.651865+0000 mon.a (mon.0) 880 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:26:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:48 smithi203 bash[23654]: cluster 2024-03-11T21:26:48.274114+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-03-11T21:26:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:48 smithi203 bash[23654]: cluster 2024-03-11T21:26:48.277899+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-03-11T21:26:49.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3847220880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5541011f0 msgr2=0x7fa554110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:49.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3847220880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5541011f0 0x7fa554110fc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00b0a0 tx=0x7fa54c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3847220880 shutdown_connections 2024-03-11T21:26:49.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3847220880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5541011f0 0x7fa554110fc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3847220880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa554102b70 0x7fa554102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3847220880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554100830 0x7fa554100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3847220880 >> 172.21.15.143:0/3847220880 conn(0x7fa554075f80 msgr2=0x7fa554076390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3847220880 shutdown_connections 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3847220880 wait complete. 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 librados: starting msgr at 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 librados: starting objecter 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 Processor -- start 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- start start 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 librados: setting wanted keys 2024-03-11T21:26:49.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 librados: calling monclient init 2024-03-11T21:26:49.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa554100830 0x7fa554199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5541011f0 0x7fa55419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa554113900 con 0x7fa554100830 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa554113780 con 0x7fa5541011f0 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.192+0000 7fa55ae8c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa554113a80 con 0x7fa554102b70 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa558c28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa554100830 0x7fa554199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:50586/0 (socket says 172.21.15.143:50586) 2024-03-11T21:26:49.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 -- 172.21.15.143:0/3313603103 learned_addr learned my addr 172.21.15.143:0/3313603103 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa553fff700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5541011f0 0x7fa55419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 -- 172.21.15.143:0/3313603103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5541011f0 msgr2=0x7fa55419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5541011f0 0x7fa55419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 -- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa554100830 msgr2=0x7fa554199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa554100830 0x7fa554199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 -- 172.21.15.143:0/3313603103 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55419edf0 con 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa559429700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00b070 tx=0x7fa54c007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 10 client.?.objecter ms_handle_connect 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c021020 con 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa54c004c50 con 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0052b0 con 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55419f080 con 0x7fa554102b70 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa55ae8c700 10 client.24620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:49.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5541a6960 con 0x7fa554102b70 2024-03-11T21:26:49.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa55ae8c700 1 librados: init done 2024-03-11T21:26:49.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55406a000 con 0x7fa554102b70 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa54c024020 con 0x7fa554102b70 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 0x7fa53c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa558c28700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 0x7fa53c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(121..121 src has 66..121) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fa54c00f070 con 0x7fa554102b70 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 10 client.24620.objecter ms_dispatch 0x7fa554001550 osd_map(121..121 src has 66..121) v4 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 3 client.24620.objecter handle_osd_map got epochs [121,121] > 0 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 3 client.24620.objecter handle_osd_map decoding full epoch 121 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.196+0000 7fa551ffb700 20 client.24620.objecter dump_active .. 0 homeless 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.200+0000 7fa558c28700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 0x7fa53c071ce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa54400abd0 tx=0x7fa54400a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:49.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.200+0000 7fa551ffb700 10 client.24620.objecter ms_handle_connect 0x7fa53c06f820 2024-03-11T21:26:49.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.204+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa54c02b020 con 0x7fa554102b70 2024-03-11T21:26:49.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.408+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa55419b140 con 0x7fa53c06f820 2024-03-11T21:26:49.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.440+0000 7fa551ffb700 1 -- 172.21.15.143:0/3313603103 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346824 (secure 0 0 0) 0x7fa55419b140 con 0x7fa53c06f820 2024-03-11T21:26:49.423 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 10 librados: watch_flush enter 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 10 librados: watch_flush exit 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 20 client.24620.objecter shutdown clearing up homeless session... 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 10 client.24620.objecter successfully canceled tick 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 msgr2=0x7fa53c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 0x7fa53c071ce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa54400abd0 tx=0x7fa54400a380 comp rx=0 tx=0).stop 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 msgr2=0x7fa55419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00b070 tx=0x7fa54c007a20 comp rx=0 tx=0).stop 2024-03-11T21:26:49.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 shutdown_connections 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa53c06f820 0x7fa53c071ce0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa554100830 0x7fa554199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa554102b70 0x7fa55419e6b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 --2- 172.21.15.143:0/3313603103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5541011f0 0x7fa55419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 >> 172.21.15.143:0/3313603103 conn(0x7fa554075f80 msgr2=0x7fa5540fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 shutdown_connections 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 -- 172.21.15.143:0/3313603103 wait complete. 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:49.444+0000 7fa55ae8c700 1 librados: shutdown 2024-03-11T21:26:49.427 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: audit 2024-03-11T21:26:48.213551+0000 mgr.y (mgr.14550) 141 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:48.983140+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:48.996614+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.034839+0000 osd.4 (osd.4) 75 : cluster 0 2.f deep-scrub starts 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.041007+0000 osd.4 (osd.4) 76 : cluster 0 2.f deep-scrub ok 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.244344+0000 osd.3 (osd.3) 107 : cluster 0 3.1e scrub starts 2024-03-11T21:26:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.246837+0000 osd.3 (osd.3) 108 : cluster 0 3.1e scrub ok 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: audit 2024-03-11T21:26:48.213551+0000 mgr.y (mgr.14550) 141 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:48.983140+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:48.996614+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.034839+0000 osd.4 (osd.4) 75 : cluster 0 2.f deep-scrub starts 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.041007+0000 osd.4 (osd.4) 76 : cluster 0 2.f deep-scrub ok 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.244344+0000 osd.3 (osd.3) 107 : cluster 0 3.1e scrub starts 2024-03-11T21:26:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:49 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.246837+0000 osd.3 (osd.3) 108 : cluster 0 3.1e scrub ok 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: audit 2024-03-11T21:26:48.213551+0000 mgr.y (mgr.14550) 141 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:48.983140+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:48.996614+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.034839+0000 osd.4 (osd.4) 75 : cluster 0 2.f deep-scrub starts 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.041007+0000 osd.4 (osd.4) 76 : cluster 0 2.f deep-scrub ok 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.244344+0000 osd.3 (osd.3) 107 : cluster 0 3.1e scrub starts 2024-03-11T21:26:50.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:49 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.246837+0000 osd.3 (osd.3) 108 : cluster 0 3.1e scrub ok 2024-03-11T21:26:50.239 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-03-11T21:26:49.122479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1155,"num_read_kb":1001,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":252892,"kb_used_data":7636,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708324,"statfs":{"total":767960285184,"available":767701323776,"internally_reserved":0,"allocated":7819264,"data_stored":4968832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003921"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159922+0000","last_change":"2024-03-11T21:26:27.355463+0000","last_active":"2024-03-11T21:26:28.159922+0000","last_peered":"2024-03-11T21:26:28.159922+0000","last_clean":"2024-03-11T21:26:28.159922+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:26:28.159922+0000","last_undegraded":"2024-03-11T21:26:28.159922+0000","last_fullsized":"2024-03-11T21:26:28.159922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.355103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.355103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.390277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649713+0000","last_change":"2024-03-11T21:25:58.944512+0000","last_active":"2024-03-11T21:26:27.649713+0000","last_peered":"2024-03-11T21:26:27.649713+0000","last_clean":"2024-03-11T21:26:27.649713+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:26:27.649713+0000","last_undegraded":"2024-03-11T21:26:27.649713+0000","last_fullsized":"2024-03-11T21:26:27.649713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.944364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.944364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.570898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":125,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.653116+0000","last_change":"2024-03-11T21:26:03.337169+0000","last_active":"2024-03-11T21:26:47.653116+0000","last_peered":"2024-03-11T21:26:47.653116+0000","last_clean":"2024-03-11T21:26:47.653116+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:26:47.653116+0000","last_undegraded":"2024-03-11T21:26:47.653116+0000","last_fullsized":"2024-03-11T21:26:47.653116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:26:03.337010+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.337010+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:29.657159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161802+0000","last_change":"2024-03-11T21:25:31.918871+0000","last_active":"2024-03-11T21:26:28.161802+0000","last_peered":"2024-03-11T21:26:28.161802+0000","last_clean":"2024-03-11T21:26:28.161802+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:26:28.161802+0000","last_undegraded":"2024-03-11T21:26:28.161802+0000","last_fullsized":"2024-03-11T21:26:28.161802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:31.918660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:31.918660+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.711285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649777+0000","last_change":"2024-03-11T21:25:48.920160+0000","last_active":"2024-03-11T21:26:27.649777+0000","last_peered":"2024-03-11T21:26:27.649777+0000","last_clean":"2024-03-11T21:26:27.649777+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:26:27.649777+0000","last_undegraded":"2024-03-11T21:26:27.649777+0000","last_fullsized":"2024-03-11T21:26:27.649777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.919930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.919930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.064632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646385+0000","last_change":"2024-03-11T21:26:13.570113+0000","last_active":"2024-03-11T21:26:47.646385+0000","last_peered":"2024-03-11T21:26:47.646385+0000","last_clean":"2024-03-11T21:26:47.646385+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:26:47.646385+0000","last_undegraded":"2024-03-11T21:26:47.646385+0000","last_fullsized":"2024-03-11T21:26:47.646385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.569895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.569895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.192341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162348+0000","last_change":"2024-03-11T21:26:12.187395+0000","last_active":"2024-03-11T21:26:28.162348+0000","last_peered":"2024-03-11T21:26:28.162348+0000","last_clean":"2024-03-11T21:26:28.162348+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:26:28.162348+0000","last_undegraded":"2024-03-11T21:26:28.162348+0000","last_fullsized":"2024-03-11T21:26:28.162348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.187178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.187178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.244264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.652957+0000","last_change":"2024-03-11T21:25:46.288260+0000","last_active":"2024-03-11T21:26:47.652957+0000","last_peered":"2024-03-11T21:26:47.652957+0000","last_clean":"2024-03-11T21:26:47.652957+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:26:47.652957+0000","last_undegraded":"2024-03-11T21:26:47.652957+0000","last_fullsized":"2024-03-11T21:26:47.652957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:46.288021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.288021+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.040825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155205+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:26:28.155205+0000","last_peered":"2024-03-11T21:26:28.155205+0000","last_clean":"2024-03-11T21:26:28.155205+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:26:28.155205+0000","last_undegraded":"2024-03-11T21:26:28.155205+0000","last_fullsized":"2024-03-11T21:26:28.155205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.062649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641413+0000","last_change":"2024-03-11T21:26:11.146066+0000","last_active":"2024-03-11T21:26:47.641413+0000","last_peered":"2024-03-11T21:26:47.641413+0000","last_clean":"2024-03-11T21:26:47.641413+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:26:47.641413+0000","last_undegraded":"2024-03-11T21:26:47.641413+0000","last_fullsized":"2024-03-11T21:26:47.641413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.145856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.145856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.221066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":124,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642198+0000","last_change":"2024-03-11T21:26:35.765693+0000","last_active":"2024-03-11T21:26:47.642198+0000","last_peered":"2024-03-11T21:26:47.642198+0000","last_clean":"2024-03-11T21:26:47.642198+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:26:47.642198+0000","last_undegraded":"2024-03-11T21:26:47.642198+0000","last_fullsized":"2024-03-11T21:26:47.642198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:35.765486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:35.765486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:44.892300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":150,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:31.763426+0000","last_change":"2024-03-11T21:26:31.763473+0000","last_active":"2024-03-11T21:26:31.763426+0000","last_peered":"2024-03-11T21:26:31.763426+0000","last_clean":"2024-03-11T21:26:31.763426+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:26:31.763426+0000","last_undegraded":"2024-03-11T21:26:31.763426+0000","last_fullsized":"2024-03-11T21:26:31.763426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.763327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:46.643188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.648627+0000","last_change":"2024-03-11T21:26:04.385285+0000","last_active":"2024-03-11T21:26:47.648627+0000","last_peered":"2024-03-11T21:26:47.648627+0000","last_clean":"2024-03-11T21:26:47.648627+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:26:47.648627+0000","last_undegraded":"2024-03-11T21:26:47.648627+0000","last_fullsized":"2024-03-11T21:26:47.648627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.385068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.385068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.394697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156016+0000","last_change":"2024-03-11T21:26:11.457112+0000","last_active":"2024-03-11T21:26:28.156016+0000","last_peered":"2024-03-11T21:26:28.156016+0000","last_clean":"2024-03-11T21:26:28.156016+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:26:28.156016+0000","last_undegraded":"2024-03-11T21:26:28.156016+0000","last_fullsized":"2024-03-11T21:26:28.156016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.456900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.456900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:19.697223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641430+0000","last_change":"2024-03-11T21:25:48.050860+0000","last_active":"2024-03-11T21:26:47.641430+0000","last_peered":"2024-03-11T21:26:47.641430+0000","last_clean":"2024-03-11T21:26:47.641430+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:26:47.641430+0000","last_undegraded":"2024-03-11T21:26:47.641430+0000","last_fullsized":"2024-03-11T21:26:47.641430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.050648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.050648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.250408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":133,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159722+0000","last_change":"2024-03-11T21:25:51.434868+0000","last_active":"2024-03-11T21:26:28.159722+0000","last_peered":"2024-03-11T21:26:28.159722+0000","last_clean":"2024-03-11T21:26:28.159722+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:26:28.159722+0000","last_undegraded":"2024-03-11T21:26:28.159722+0000","last_fullsized":"2024-03-11T21:26:28.159722+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:25:51.434497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:51.434497+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.747829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163041+0000","last_change":"2024-03-11T21:26:20.277298+0000","last_active":"2024-03-11T21:26:28.163041+0000","last_peered":"2024-03-11T21:26:28.163041+0000","last_clean":"2024-03-11T21:26:28.163041+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:26:28.163041+0000","last_undegraded":"2024-03-11T21:26:28.163041+0000","last_fullsized":"2024-03-11T21:26:28.163041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.277140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.790443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650750+0000","last_change":"2024-03-11T21:26:10.005094+0000","last_active":"2024-03-11T21:26:27.650750+0000","last_peered":"2024-03-11T21:26:27.650750+0000","last_clean":"2024-03-11T21:26:27.650750+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:26:27.650750+0000","last_undegraded":"2024-03-11T21:26:27.650750+0000","last_fullsized":"2024-03-11T21:26:27.650750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.004882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.004882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:30.146208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:39.219189+0000","last_change":"2024-03-11T21:26:39.219283+0000","last_active":"2024-03-11T21:26:39.219189+0000","last_peered":"2024-03-11T21:26:39.219189+0000","last_clean":"2024-03-11T21:26:39.219189+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:26:39.219189+0000","last_undegraded":"2024-03-11T21:26:39.219189+0000","last_fullsized":"2024-03-11T21:26:39.219189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:26:39.218970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:01.129287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650102+0000","last_change":"2024-03-11T21:25:43.877458+0000","last_active":"2024-03-11T21:26:27.650102+0000","last_peered":"2024-03-11T21:26:27.650102+0000","last_clean":"2024-03-11T21:26:27.650102+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:26:27.650102+0000","last_undegraded":"2024-03-11T21:26:27.650102+0000","last_fullsized":"2024-03-11T21:26:27.650102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:43.877225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.877225+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.780065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651606+0000","last_change":"2024-03-11T21:25:59.988760+0000","last_active":"2024-03-11T21:26:27.651606+0000","last_peered":"2024-03-11T21:26:27.651606+0000","last_clean":"2024-03-11T21:26:27.651606+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:26:27.651606+0000","last_undegraded":"2024-03-11T21:26:27.651606+0000","last_fullsized":"2024-03-11T21:26:27.651606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.988522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.034265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646297+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:26:47.646297+0000","last_peered":"2024-03-11T21:26:47.646297+0000","last_clean":"2024-03-11T21:26:47.646297+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:26:47.646297+0000","last_undegraded":"2024-03-11T21:26:47.646297+0000","last_fullsized":"2024-03-11T21:26:47.646297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.655474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157019+0000","last_change":"2024-03-11T21:26:04.956591+0000","last_active":"2024-03-11T21:26:28.157019+0000","last_peered":"2024-03-11T21:26:28.157019+0000","last_clean":"2024-03-11T21:26:28.157019+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:26:28.157019+0000","last_undegraded":"2024-03-11T21:26:28.157019+0000","last_fullsized":"2024-03-11T21:26:28.157019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.956350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.956350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.718796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":124,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.651753+0000","last_change":"2024-03-11T21:26:20.455065+0000","last_active":"2024-03-11T21:26:47.651753+0000","last_peered":"2024-03-11T21:26:47.651753+0000","last_clean":"2024-03-11T21:26:47.651753+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:26:47.651753+0000","last_undegraded":"2024-03-11T21:26:47.651753+0000","last_fullsized":"2024-03-11T21:26:47.651753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.454747+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.108877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646693+0000","last_change":"2024-03-11T21:26:09.575095+0000","last_active":"2024-03-11T21:26:47.646693+0000","last_peered":"2024-03-11T21:26:47.646693+0000","last_clean":"2024-03-11T21:26:47.646693+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:26:47.646693+0000","last_undegraded":"2024-03-11T21:26:47.646693+0000","last_fullsized":"2024-03-11T21:26:47.646693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:09.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:09.574796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.590692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.652606+0000","last_change":"2024-03-11T21:25:40.280094+0000","last_active":"2024-03-11T21:26:47.652606+0000","last_peered":"2024-03-11T21:26:47.652606+0000","last_clean":"2024-03-11T21:26:47.652606+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:26:47.652606+0000","last_undegraded":"2024-03-11T21:26:47.652606+0000","last_fullsized":"2024-03-11T21:26:47.652606+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.279715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.279715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.096032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:41.250211+0000","last_change":"2024-03-11T21:26:41.250247+0000","last_active":"2024-03-11T21:26:41.250211+0000","last_peered":"2024-03-11T21:26:41.250211+0000","last_clean":"2024-03-11T21:26:41.250211+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:26:41.250211+0000","last_undegraded":"2024-03-11T21:26:41.250211+0000","last_fullsized":"2024-03-11T21:26:41.250211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:26:41.250134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.250134+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.792042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":134,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.192764+0000","last_change":"2024-03-11T21:26:38.192925+0000","last_active":"2024-03-11T21:26:38.192764+0000","last_peered":"2024-03-11T21:26:38.192764+0000","last_clean":"2024-03-11T21:26:38.192764+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:26:38.192764+0000","last_undegraded":"2024-03-11T21:26:38.192764+0000","last_fullsized":"2024-03-11T21:26:38.192764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:38.192555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.192555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.368750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642517+0000","last_change":"2024-03-11T21:26:29.967551+0000","last_active":"2024-03-11T21:26:47.642517+0000","last_peered":"2024-03-11T21:26:47.642517+0000","last_clean":"2024-03-11T21:26:47.642517+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:26:47.642517+0000","last_undegraded":"2024-03-11T21:26:47.642517+0000","last_fullsized":"2024-03-11T21:26:47.642517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:26:29.967316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.967316+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.913576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":164,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.159929+0000","last_change":"2024-03-11T21:26:05.978542+0000","last_active":"2024-03-11T21:26:38.159929+0000","last_peered":"2024-03-11T21:26:38.159929+0000","last_clean":"2024-03-11T21:26:38.159929+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:26:38.159929+0000","last_undegraded":"2024-03-11T21:26:38.159929+0000","last_fullsized":"2024-03-11T21:26:38.159929+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:05.978319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.978319+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.180257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.154939+0000","last_change":"2024-03-11T21:26:02.471008+0000","last_active":"2024-03-11T21:26:28.154939+0000","last_peered":"2024-03-11T21:26:28.154939+0000","last_clean":"2024-03-11T21:26:28.154939+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:26:28.154939+0000","last_undegraded":"2024-03-11T21:26:28.154939+0000","last_fullsized":"2024-03-11T21:26:28.154939+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.470792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.470792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.529057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646922+0000","last_change":"2024-03-11T21:26:03.728864+0000","last_active":"2024-03-11T21:26:47.646922+0000","last_peered":"2024-03-11T21:26:47.646922+0000","last_clean":"2024-03-11T21:26:47.646922+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:26:47.646922+0000","last_undegraded":"2024-03-11T21:26:47.646922+0000","last_fullsized":"2024-03-11T21:26:47.646922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:03.728766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.728766+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.097957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.313657+0000","last_change":"2024-03-11T21:26:28.313721+0000","last_active":"2024-03-11T21:26:28.313657+0000","last_peered":"2024-03-11T21:26:28.313657+0000","last_clean":"2024-03-11T21:26:28.313657+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:26:28.313657+0000","last_undegraded":"2024-03-11T21:26:28.313657+0000","last_fullsized":"2024-03-11T21:26:28.313657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:26:28.313489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.975519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641727+0000","last_change":"2024-03-11T21:26:20.986742+0000","last_active":"2024-03-11T21:26:47.641727+0000","last_peered":"2024-03-11T21:26:47.641727+0000","last_clean":"2024-03-11T21:26:47.641727+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:26:47.641727+0000","last_undegraded":"2024-03-11T21:26:47.641727+0000","last_fullsized":"2024-03-11T21:26:47.641727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.986534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.986534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.313433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159785+0000","last_change":"2024-03-11T21:25:40.946608+0000","last_active":"2024-03-11T21:26:28.159785+0000","last_peered":"2024-03-11T21:26:28.159785+0000","last_clean":"2024-03-11T21:26:28.159785+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:26:28.159785+0000","last_undegraded":"2024-03-11T21:26:28.159785+0000","last_fullsized":"2024-03-11T21:26:28.159785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.946307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:45.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158034+0000","last_change":"2024-03-11T21:25:27.506732+0000","last_active":"2024-03-11T21:26:28.158034+0000","last_peered":"2024-03-11T21:26:28.158034+0000","last_clean":"2024-03-11T21:26:28.158034+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:26:28.158034+0000","last_undegraded":"2024-03-11T21:26:28.158034+0000","last_fullsized":"2024-03-11T21:26:28.158034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:27.506461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.506461+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.042628+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641180+0000","last_change":"2024-03-11T21:25:53.051632+0000","last_active":"2024-03-11T21:26:47.641180+0000","last_peered":"2024-03-11T21:26:47.641180+0000","last_clean":"2024-03-11T21:26:47.641180+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:26:47.641180+0000","last_undegraded":"2024-03-11T21:26:47.641180+0000","last_fullsized":"2024-03-11T21:26:47.641180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.051422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.051422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.051498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:29.972866+0000","last_change":"2024-03-11T21:26:29.972923+0000","last_active":"2024-03-11T21:26:29.972866+0000","last_peered":"2024-03-11T21:26:29.972866+0000","last_clean":"2024-03-11T21:26:29.972866+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:26:29.972866+0000","last_undegraded":"2024-03-11T21:26:29.972866+0000","last_fullsized":"2024-03-11T21:26:29.972866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.972683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.497791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158883+0000","last_change":"2024-03-11T21:25:48.374372+0000","last_active":"2024-03-11T21:26:28.158883+0000","last_peered":"2024-03-11T21:26:28.158883+0000","last_clean":"2024-03-11T21:26:28.158883+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:26:28.158883+0000","last_undegraded":"2024-03-11T21:26:28.158883+0000","last_fullsized":"2024-03-11T21:26:28.158883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.374158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.374158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.742286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.652751+0000","last_change":"2024-03-11T21:25:37.334081+0000","last_active":"2024-03-11T21:26:47.652751+0000","last_peered":"2024-03-11T21:26:47.652751+0000","last_clean":"2024-03-11T21:26:47.652751+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:26:47.652751+0000","last_undegraded":"2024-03-11T21:26:47.652751+0000","last_fullsized":"2024-03-11T21:26:47.652751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:25:37.333856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.333856+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.493865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162837+0000","last_change":"2024-03-11T21:25:56.141805+0000","last_active":"2024-03-11T21:26:28.162837+0000","last_peered":"2024-03-11T21:26:28.162837+0000","last_clean":"2024-03-11T21:26:28.162837+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:26:28.162837+0000","last_undegraded":"2024-03-11T21:26:28.162837+0000","last_fullsized":"2024-03-11T21:26:28.162837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.141559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.141559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.340511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645655+0000","last_change":"2024-03-11T21:26:14.593548+0000","last_active":"2024-03-11T21:26:47.645655+0000","last_peered":"2024-03-11T21:26:47.645655+0000","last_clean":"2024-03-11T21:26:47.645655+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:26:47.645655+0000","last_undegraded":"2024-03-11T21:26:47.645655+0000","last_fullsized":"2024-03-11T21:26:47.645655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.593259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.593259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.052839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645606+0000","last_change":"2024-03-11T21:25:53.706422+0000","last_active":"2024-03-11T21:26:47.645606+0000","last_peered":"2024-03-11T21:26:47.645606+0000","last_clean":"2024-03-11T21:26:47.645606+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:26:47.645606+0000","last_undegraded":"2024-03-11T21:26:47.645606+0000","last_fullsized":"2024-03-11T21:26:47.645606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:25:53.706230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.384575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651087+0000","last_change":"2024-03-11T21:25:38.984101+0000","last_active":"2024-03-11T21:26:27.651087+0000","last_peered":"2024-03-11T21:26:27.651087+0000","last_clean":"2024-03-11T21:26:27.651087+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:26:27.651087+0000","last_undegraded":"2024-03-11T21:26:27.651087+0000","last_fullsized":"2024-03-11T21:26:27.651087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:38.983858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.983858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.131782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.648289+0000","last_change":"2024-03-11T21:25:56.344253+0000","last_active":"2024-03-11T21:26:47.648289+0000","last_peered":"2024-03-11T21:26:47.648289+0000","last_clean":"2024-03-11T21:26:47.648289+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:26:47.648289+0000","last_undegraded":"2024-03-11T21:26:47.648289+0000","last_fullsized":"2024-03-11T21:26:47.648289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:56.343979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.343979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.628275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161092+0000","last_change":"2024-03-11T21:25:39.974174+0000","last_active":"2024-03-11T21:26:28.161092+0000","last_peered":"2024-03-11T21:26:28.161092+0000","last_clean":"2024-03-11T21:26:28.161092+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:26:28.161092+0000","last_undegraded":"2024-03-11T21:26:28.161092+0000","last_fullsized":"2024-03-11T21:26:28.161092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.973978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.973978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.092042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158481+0000","last_change":"2024-03-11T21:25:45.890051+0000","last_active":"2024-03-11T21:26:28.158481+0000","last_peered":"2024-03-11T21:26:28.158481+0000","last_clean":"2024-03-11T21:26:28.158481+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:26:28.158481+0000","last_undegraded":"2024-03-11T21:26:28.158481+0000","last_fullsized":"2024-03-11T21:26:28.158481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:45.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.889834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.143004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642564+0000","last_change":"2024-03-11T21:25:37.960424+0000","last_active":"2024-03-11T21:26:47.642564+0000","last_peered":"2024-03-11T21:26:47.642564+0000","last_clean":"2024-03-11T21:26:47.642564+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:26:47.642564+0000","last_undegraded":"2024-03-11T21:26:47.642564+0000","last_fullsized":"2024-03-11T21:26:47.642564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:25:37.960057+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.960057+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.161700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.652218+0000","last_change":"2024-03-11T21:25:40.949182+0000","last_active":"2024-03-11T21:26:27.652218+0000","last_peered":"2024-03-11T21:26:27.652218+0000","last_clean":"2024-03-11T21:26:27.652218+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:26:27.652218+0000","last_undegraded":"2024-03-11T21:26:27.652218+0000","last_fullsized":"2024-03-11T21:26:27.652218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:40.948963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.948963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.061308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'11","reported_seq":161,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.164809+0000","last_change":"2024-03-11T21:25:38.973030+0000","last_active":"2024-03-11T21:26:38.164809+0000","last_peered":"2024-03-11T21:26:38.164809+0000","last_clean":"2024-03-11T21:26:38.164809+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:26:38.164809+0000","last_undegraded":"2024-03-11T21:26:38.164809+0000","last_fullsized":"2024-03-11T21:26:38.164809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:38.972802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:38.972802+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.415779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.651467+0000","last_change":"2024-03-11T21:25:54.342862+0000","last_active":"2024-03-11T21:26:47.651467+0000","last_peered":"2024-03-11T21:26:47.651467+0000","last_clean":"2024-03-11T21:26:47.651467+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:26:47.651467+0000","last_undegraded":"2024-03-11T21:26:47.651467+0000","last_fullsized":"2024-03-11T21:26:47.651467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.342588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.028053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.647065+0000","last_change":"2024-03-11T21:26:01.314774+0000","last_active":"2024-03-11T21:26:47.647065+0000","last_peered":"2024-03-11T21:26:47.647065+0000","last_clean":"2024-03-11T21:26:47.647065+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:26:47.647065+0000","last_undegraded":"2024-03-11T21:26:47.647065+0000","last_fullsized":"2024-03-11T21:26:47.647065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:26:01.314615+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.665529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645723+0000","last_change":"2024-03-11T21:25:43.246214+0000","last_active":"2024-03-11T21:26:47.645723+0000","last_peered":"2024-03-11T21:26:47.645723+0000","last_clean":"2024-03-11T21:26:47.645723+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:26:47.645723+0000","last_undegraded":"2024-03-11T21:26:47.645723+0000","last_fullsized":"2024-03-11T21:26:47.645723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.246002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.246002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.409171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":161,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.159880+0000","last_change":"2024-03-11T21:25:59.442248+0000","last_active":"2024-03-11T21:26:38.159880+0000","last_peered":"2024-03-11T21:26:38.159880+0000","last_clean":"2024-03-11T21:26:38.159880+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:26:38.159880+0000","last_undegraded":"2024-03-11T21:26:38.159880+0000","last_fullsized":"2024-03-11T21:26:38.159880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:59.442015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.442015+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.052362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155914+0000","last_change":"2024-03-11T21:26:27.752411+0000","last_active":"2024-03-11T21:26:28.155914+0000","last_peered":"2024-03-11T21:26:28.155914+0000","last_clean":"2024-03-11T21:26:28.155914+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:26:28.155914+0000","last_undegraded":"2024-03-11T21:26:28.155914+0000","last_fullsized":"2024-03-11T21:26:28.155914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.752198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.752198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:42.079868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642294+0000","last_change":"2024-03-11T21:25:39.009861+0000","last_active":"2024-03-11T21:26:47.642294+0000","last_peered":"2024-03-11T21:26:47.642294+0000","last_clean":"2024-03-11T21:26:47.642294+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:26:47.642294+0000","last_undegraded":"2024-03-11T21:26:47.642294+0000","last_fullsized":"2024-03-11T21:26:47.642294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:39.009635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.009635+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641408+0000","last_change":"2024-03-11T21:25:49.003277+0000","last_active":"2024-03-11T21:26:47.641408+0000","last_peered":"2024-03-11T21:26:47.641408+0000","last_clean":"2024-03-11T21:26:47.641408+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:26:47.641408+0000","last_undegraded":"2024-03-11T21:26:47.641408+0000","last_fullsized":"2024-03-11T21:26:47.641408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:49.003014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.003014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.285739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.158320+0000","last_change":"2024-03-11T21:25:54.360923+0000","last_active":"2024-03-11T21:26:28.158320+0000","last_peered":"2024-03-11T21:26:28.158320+0000","last_clean":"2024-03-11T21:26:28.158320+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:26:28.158320+0000","last_undegraded":"2024-03-11T21:26:28.158320+0000","last_fullsized":"2024-03-11T21:26:28.158320+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:54.360647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:54.360647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.912837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.871166+0000","last_change":"2024-03-11T21:26:43.871224+0000","last_active":"2024-03-11T21:26:43.871166+0000","last_peered":"2024-03-11T21:26:43.871166+0000","last_clean":"2024-03-11T21:26:43.871166+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:26:43.871166+0000","last_undegraded":"2024-03-11T21:26:43.871166+0000","last_fullsized":"2024-03-11T21:26:43.871166+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:43.871005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:43.871005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.711985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":118,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.154812+0000","last_change":"2024-03-11T21:25:46.921649+0000","last_active":"2024-03-11T21:26:28.154812+0000","last_peered":"2024-03-11T21:26:28.154812+0000","last_clean":"2024-03-11T21:26:28.154812+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:26:28.154812+0000","last_undegraded":"2024-03-11T21:26:28.154812+0000","last_fullsized":"2024-03-11T21:26:28.154812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:25:46.921404+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.755306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994964+0000","last_change":"2024-03-11T21:25:50.974118+0000","last_active":"2024-03-11T21:26:47.994964+0000","last_peered":"2024-03-11T21:26:47.994964+0000","last_clean":"2024-03-11T21:26:47.994964+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:26:47.994964+0000","last_undegraded":"2024-03-11T21:26:47.994964+0000","last_fullsized":"2024-03-11T21:26:47.994964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:50.974032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.974032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.283434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163192+0000","last_change":"2024-03-11T21:26:04.178398+0000","last_active":"2024-03-11T21:26:28.163192+0000","last_peered":"2024-03-11T21:26:28.163192+0000","last_clean":"2024-03-11T21:26:28.163192+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:26:28.163192+0000","last_undegraded":"2024-03-11T21:26:28.163192+0000","last_fullsized":"2024-03-11T21:26:28.163192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.178145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.178145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.425359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":124,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.993736+0000","last_change":"2024-03-11T21:26:45.975343+0000","last_active":"2024-03-11T21:26:47.993736+0000","last_peered":"2024-03-11T21:26:47.993736+0000","last_clean":"2024-03-11T21:26:47.993736+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:26:47.993736+0000","last_undegraded":"2024-03-11T21:26:47.993736+0000","last_fullsized":"2024-03-11T21:26:47.993736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.975124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.185402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641058+0000","last_change":"2024-03-11T21:26:10.162900+0000","last_active":"2024-03-11T21:26:47.641058+0000","last_peered":"2024-03-11T21:26:47.641058+0000","last_clean":"2024-03-11T21:26:47.641058+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:26:47.641058+0000","last_undegraded":"2024-03-11T21:26:47.641058+0000","last_fullsized":"2024-03-11T21:26:47.641058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:10.162661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.162661+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:28.988891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161364+0000","last_change":"2024-03-11T21:25:27.983629+0000","last_active":"2024-03-11T21:26:28.161364+0000","last_peered":"2024-03-11T21:26:28.161364+0000","last_clean":"2024-03-11T21:26:28.161364+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:26:28.161364+0000","last_undegraded":"2024-03-11T21:26:28.161364+0000","last_fullsized":"2024-03-11T21:26:28.161364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:27.983536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:27.983536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.044284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651535+0000","last_change":"2024-03-11T21:25:37.981238+0000","last_active":"2024-03-11T21:26:27.651535+0000","last_peered":"2024-03-11T21:26:27.651535+0000","last_clean":"2024-03-11T21:26:27.651535+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:26:27.651535+0000","last_undegraded":"2024-03-11T21:26:27.651535+0000","last_fullsized":"2024-03-11T21:26:27.651535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:25:37.981019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.981019+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.806629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":136,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.643021+0000","last_change":"2024-03-11T21:26:08.200786+0000","last_active":"2024-03-11T21:26:47.643021+0000","last_peered":"2024-03-11T21:26:47.643021+0000","last_clean":"2024-03-11T21:26:47.643021+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:26:47.643021+0000","last_undegraded":"2024-03-11T21:26:47.643021+0000","last_fullsized":"2024-03-11T21:26:47.643021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:26:08.200532+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:24:33.079623+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.200532+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.719717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.993640+0000","last_change":"2024-03-11T21:26:02.944301+0000","last_active":"2024-03-11T21:26:47.993640+0000","last_peered":"2024-03-11T21:26:47.993640+0000","last_clean":"2024-03-11T21:26:47.993640+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:26:47.993640+0000","last_undegraded":"2024-03-11T21:26:47.993640+0000","last_fullsized":"2024-03-11T21:26:47.993640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:21.682420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.648474+0000","last_change":"2024-03-11T21:25:48.263693+0000","last_active":"2024-03-11T21:26:47.648474+0000","last_peered":"2024-03-11T21:26:47.648474+0000","last_clean":"2024-03-11T21:26:47.648474+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:26:47.648474+0000","last_undegraded":"2024-03-11T21:26:47.648474+0000","last_fullsized":"2024-03-11T21:26:47.648474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:25:48.263515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.384625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.647602+0000","last_change":"2024-03-11T21:25:33.757745+0000","last_active":"2024-03-11T21:26:47.647602+0000","last_peered":"2024-03-11T21:26:47.647602+0000","last_clean":"2024-03-11T21:26:47.647602+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:26:47.647602+0000","last_undegraded":"2024-03-11T21:26:47.647602+0000","last_fullsized":"2024-03-11T21:26:47.647602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:33.757442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:33.757442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.904053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":185,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.687427+0000","last_change":"2024-03-11T21:26:45.687511+0000","last_active":"2024-03-11T21:26:45.687427+0000","last_peered":"2024-03-11T21:26:45.687427+0000","last_clean":"2024-03-11T21:26:45.687427+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:26:45.687427+0000","last_undegraded":"2024-03-11T21:26:45.687427+0000","last_fullsized":"2024-03-11T21:26:45.687427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:26:45.687244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.687244+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.865274+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":119,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645567+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:26:47.645567+0000","last_peered":"2024-03-11T21:26:47.645567+0000","last_clean":"2024-03-11T21:26:47.645567+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:26:47.645567+0000","last_undegraded":"2024-03-11T21:26:47.645567+0000","last_fullsized":"2024-03-11T21:26:47.645567+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.556814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"84'11","reported_seq":166,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.652236+0000","last_change":"2024-03-11T21:25:58.333052+0000","last_active":"2024-03-11T21:26:47.652236+0000","last_peered":"2024-03-11T21:26:47.652236+0000","last_clean":"2024-03-11T21:26:47.652236+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:26:47.652236+0000","last_undegraded":"2024-03-11T21:26:47.652236+0000","last_fullsized":"2024-03-11T21:26:47.652236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:58.332816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.332816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.183920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:30.314997+0000","last_change":"2024-03-11T21:26:30.315059+0000","last_active":"2024-03-11T21:26:30.314997+0000","last_peered":"2024-03-11T21:26:30.314997+0000","last_clean":"2024-03-11T21:26:30.314997+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:26:30.314997+0000","last_undegraded":"2024-03-11T21:26:30.314997+0000","last_fullsized":"2024-03-11T21:26:30.314997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:30.314840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:30.314840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:39.460296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.156206+0000","last_change":"2024-03-11T21:26:13.487586+0000","last_active":"2024-03-11T21:26:28.156206+0000","last_peered":"2024-03-11T21:26:28.156206+0000","last_clean":"2024-03-11T21:26:28.156206+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:26:28.156206+0000","last_undegraded":"2024-03-11T21:26:28.156206+0000","last_fullsized":"2024-03-11T21:26:28.156206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.487359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.487359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.520405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":162,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:40.102926+0000","last_change":"2024-03-11T21:26:40.103016+0000","last_active":"2024-03-11T21:26:40.102926+0000","last_peered":"2024-03-11T21:26:40.102926+0000","last_clean":"2024-03-11T21:26:40.102926+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:26:40.102926+0000","last_undegraded":"2024-03-11T21:26:40.102926+0000","last_fullsized":"2024-03-11T21:26:40.102926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:26:40.102764+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.102764+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.411337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.648893+0000","last_change":"2024-03-11T21:26:25.426916+0000","last_active":"2024-03-11T21:26:47.648893+0000","last_peered":"2024-03-11T21:26:47.648893+0000","last_clean":"2024-03-11T21:26:47.648893+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:26:47.648893+0000","last_undegraded":"2024-03-11T21:26:47.648893+0000","last_fullsized":"2024-03-11T21:26:47.648893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:25.426623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:25.426623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.790210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641687+0000","last_change":"2024-03-11T21:26:04.128265+0000","last_active":"2024-03-11T21:26:47.641687+0000","last_peered":"2024-03-11T21:26:47.641687+0000","last_clean":"2024-03-11T21:26:47.641687+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:26:47.641687+0000","last_undegraded":"2024-03-11T21:26:47.641687+0000","last_fullsized":"2024-03-11T21:26:47.641687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.128031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.128031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.667989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":144,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.647696+0000","last_change":"2024-03-11T21:25:56.736951+0000","last_active":"2024-03-11T21:26:47.647696+0000","last_peered":"2024-03-11T21:26:47.647696+0000","last_clean":"2024-03-11T21:26:47.647696+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:26:47.647696+0000","last_undegraded":"2024-03-11T21:26:47.647696+0000","last_fullsized":"2024-03-11T21:26:47.647696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:25:56.736721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:56.736721+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.841976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.993784+0000","last_change":"2024-03-11T21:25:47.994296+0000","last_active":"2024-03-11T21:26:47.993784+0000","last_peered":"2024-03-11T21:26:47.993784+0000","last_clean":"2024-03-11T21:26:47.993784+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:26:47.993784+0000","last_undegraded":"2024-03-11T21:26:47.993784+0000","last_fullsized":"2024-03-11T21:26:47.993784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.994008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.994008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.859169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994406+0000","last_change":"2024-03-11T21:25:58.022948+0000","last_active":"2024-03-11T21:26:47.994406+0000","last_peered":"2024-03-11T21:26:47.994406+0000","last_clean":"2024-03-11T21:26:47.994406+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:26:47.994406+0000","last_undegraded":"2024-03-11T21:26:47.994406+0000","last_fullsized":"2024-03-11T21:26:47.994406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.022737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.022737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.306838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.162053+0000","last_change":"2024-03-11T21:26:03.430155+0000","last_active":"2024-03-11T21:26:28.162053+0000","last_peered":"2024-03-11T21:26:28.162053+0000","last_clean":"2024-03-11T21:26:28.162053+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:26:28.162053+0000","last_undegraded":"2024-03-11T21:26:28.162053+0000","last_fullsized":"2024-03-11T21:26:28.162053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:03.429945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.429945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.219815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160720+0000","last_change":"2024-03-11T21:26:08.149166+0000","last_active":"2024-03-11T21:26:28.160720+0000","last_peered":"2024-03-11T21:26:28.160720+0000","last_clean":"2024-03-11T21:26:28.160720+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:26:28.160720+0000","last_undegraded":"2024-03-11T21:26:28.160720+0000","last_fullsized":"2024-03-11T21:26:28.160720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:08.148915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.148915+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.413377+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155903+0000","last_change":"2024-03-11T21:25:32.395539+0000","last_active":"2024-03-11T21:26:28.155903+0000","last_peered":"2024-03-11T21:26:28.155903+0000","last_clean":"2024-03-11T21:26:28.155903+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:26:28.155903+0000","last_undegraded":"2024-03-11T21:26:28.155903+0000","last_fullsized":"2024-03-11T21:26:28.155903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.395316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.615969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":378,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.640987+0000","last_change":"2024-03-11T21:26:36.758336+0000","last_active":"2024-03-11T21:26:47.640987+0000","last_peered":"2024-03-11T21:26:47.640987+0000","last_clean":"2024-03-11T21:26:47.640987+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:26:47.640987+0000","last_undegraded":"2024-03-11T21:26:47.640987+0000","last_fullsized":"2024-03-11T21:26:47.640987+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:26:36.758088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:26:36.758088+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:04.601201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642219+0000","last_change":"2024-03-11T21:26:34.807309+0000","last_active":"2024-03-11T21:26:47.642219+0000","last_peered":"2024-03-11T21:26:47.642219+0000","last_clean":"2024-03-11T21:26:47.642219+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:26:47.642219+0000","last_undegraded":"2024-03-11T21:26:47.642219+0000","last_fullsized":"2024-03-11T21:26:47.642219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:34.807061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:34.807061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:56.122316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646971+0000","last_change":"2024-03-11T21:26:10.602526+0000","last_active":"2024-03-11T21:26:47.646971+0000","last_peered":"2024-03-11T21:26:47.646971+0000","last_clean":"2024-03-11T21:26:47.646971+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:26:47.646971+0000","last_undegraded":"2024-03-11T21:26:47.646971+0000","last_fullsized":"2024-03-11T21:26:47.646971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.602230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.602230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:36.316832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":133,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646237+0000","last_change":"2024-03-11T21:26:13.337394+0000","last_active":"2024-03-11T21:26:47.646237+0000","last_peered":"2024-03-11T21:26:47.646237+0000","last_clean":"2024-03-11T21:26:47.646237+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:26:47.646237+0000","last_undegraded":"2024-03-11T21:26:47.646237+0000","last_fullsized":"2024-03-11T21:26:47.646237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:26:13.337090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.337090+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.002783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.640768+0000","last_change":"2024-03-11T21:25:43.021158+0000","last_active":"2024-03-11T21:26:47.640768+0000","last_peered":"2024-03-11T21:26:47.640768+0000","last_clean":"2024-03-11T21:26:47.640768+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:26:47.640768+0000","last_undegraded":"2024-03-11T21:26:47.640768+0000","last_fullsized":"2024-03-11T21:26:47.640768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.020950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.020950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.379154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161540+0000","last_change":"2024-03-11T21:25:52.415680+0000","last_active":"2024-03-11T21:26:28.161540+0000","last_peered":"2024-03-11T21:26:28.161540+0000","last_clean":"2024-03-11T21:26:28.161540+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:26:28.161540+0000","last_undegraded":"2024-03-11T21:26:28.161540+0000","last_fullsized":"2024-03-11T21:26:28.161540+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.415468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.415468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.982658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646018+0000","last_change":"2024-03-11T21:26:04.760536+0000","last_active":"2024-03-11T21:26:47.646018+0000","last_peered":"2024-03-11T21:26:47.646018+0000","last_clean":"2024-03-11T21:26:47.646018+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:26:47.646018+0000","last_undegraded":"2024-03-11T21:26:47.646018+0000","last_fullsized":"2024-03-11T21:26:47.646018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.760262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.760262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.318865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642220+0000","last_change":"2024-03-11T21:25:59.200564+0000","last_active":"2024-03-11T21:26:47.642220+0000","last_peered":"2024-03-11T21:26:47.642220+0000","last_clean":"2024-03-11T21:26:47.642220+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:26:47.642220+0000","last_undegraded":"2024-03-11T21:26:47.642220+0000","last_fullsized":"2024-03-11T21:26:47.642220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:59.200270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:59.200270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.037981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":215,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.647849+0000","last_change":"2024-03-11T21:26:40.725235+0000","last_active":"2024-03-11T21:26:47.647849+0000","last_peered":"2024-03-11T21:26:47.647849+0000","last_clean":"2024-03-11T21:26:47.647849+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:26:47.647849+0000","last_undegraded":"2024-03-11T21:26:47.647849+0000","last_fullsized":"2024-03-11T21:26:47.647849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:26:40.724980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.724980+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.435897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.157283+0000","last_change":"2024-03-11T21:26:08.930659+0000","last_active":"2024-03-11T21:26:28.157283+0000","last_peered":"2024-03-11T21:26:28.157283+0000","last_clean":"2024-03-11T21:26:28.157283+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:26:28.157283+0000","last_undegraded":"2024-03-11T21:26:28.157283+0000","last_fullsized":"2024-03-11T21:26:28.157283+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:08.930415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.930415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.664164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649827+0000","last_change":"2024-03-11T21:26:02.037288+0000","last_active":"2024-03-11T21:26:27.649827+0000","last_peered":"2024-03-11T21:26:27.649827+0000","last_clean":"2024-03-11T21:26:27.649827+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:26:27.649827+0000","last_undegraded":"2024-03-11T21:26:27.649827+0000","last_fullsized":"2024-03-11T21:26:27.649827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.939102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":117,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.649958+0000","last_change":"2024-03-11T21:25:49.638326+0000","last_active":"2024-03-11T21:26:47.649958+0000","last_peered":"2024-03-11T21:26:47.649958+0000","last_clean":"2024-03-11T21:26:47.649958+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:26:47.649958+0000","last_undegraded":"2024-03-11T21:26:47.649958+0000","last_fullsized":"2024-03-11T21:26:47.649958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:49.638061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:49.638061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.499420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":137,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.163387+0000","last_change":"2024-03-11T21:26:24.352686+0000","last_active":"2024-03-11T21:26:28.163387+0000","last_peered":"2024-03-11T21:26:28.163387+0000","last_clean":"2024-03-11T21:26:28.163387+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:26:28.163387+0000","last_undegraded":"2024-03-11T21:26:28.163387+0000","last_fullsized":"2024-03-11T21:26:28.163387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:26:24.352407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:24.352407+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.744324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":323,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:46.112358+0000","last_change":"2024-03-11T21:25:50.006793+0000","last_active":"2024-03-11T21:26:46.112358+0000","last_peered":"2024-03-11T21:26:46.112358+0000","last_clean":"2024-03-11T21:26:46.112358+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:26:46.112358+0000","last_undegraded":"2024-03-11T21:26:46.112358+0000","last_fullsized":"2024-03-11T21:26:46.112358+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:25:50.006698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.006698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.269418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.649776+0000","last_change":"2024-03-11T21:25:57.992113+0000","last_active":"2024-03-11T21:26:27.649776+0000","last_peered":"2024-03-11T21:26:27.649776+0000","last_clean":"2024-03-11T21:26:27.649776+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:26:27.649776+0000","last_undegraded":"2024-03-11T21:26:27.649776+0000","last_fullsized":"2024-03-11T21:26:27.649776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.991803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.991803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.426825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641983+0000","last_change":"2024-03-11T21:25:34.059053+0000","last_active":"2024-03-11T21:26:47.641983+0000","last_peered":"2024-03-11T21:26:47.641983+0000","last_clean":"2024-03-11T21:26:47.641983+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:26:47.641983+0000","last_undegraded":"2024-03-11T21:26:47.641983+0000","last_fullsized":"2024-03-11T21:26:47.641983+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:25:34.058815+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.431730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"77'1","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159517+0000","last_change":"2024-03-11T21:25:43.867607+0000","last_active":"2024-03-11T21:26:28.159517+0000","last_peered":"2024-03-11T21:26:28.159517+0000","last_clean":"2024-03-11T21:26:28.159517+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:26:28.159517+0000","last_undegraded":"2024-03-11T21:26:28.159517+0000","last_fullsized":"2024-03-11T21:26:28.159517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:25:43.867377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.867377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.362412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":138,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.652895+0000","last_change":"2024-03-11T21:26:32.529005+0000","last_active":"2024-03-11T21:26:47.652895+0000","last_peered":"2024-03-11T21:26:47.652895+0000","last_clean":"2024-03-11T21:26:47.652895+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:26:47.652895+0000","last_undegraded":"2024-03-11T21:26:47.652895+0000","last_fullsized":"2024-03-11T21:26:47.652895+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:32.528639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:32.528639+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.761031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":115,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650328+0000","last_change":"2024-03-11T21:25:47.898533+0000","last_active":"2024-03-11T21:26:27.650328+0000","last_peered":"2024-03-11T21:26:27.650328+0000","last_clean":"2024-03-11T21:26:27.650328+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:26:27.650328+0000","last_undegraded":"2024-03-11T21:26:27.650328+0000","last_fullsized":"2024-03-11T21:26:27.650328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:47.898322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:47.898322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:14.986939+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":161,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.164765+0000","last_change":"2024-03-11T21:25:45.984353+0000","last_active":"2024-03-11T21:26:38.164765+0000","last_peered":"2024-03-11T21:26:38.164765+0000","last_clean":"2024-03-11T21:26:38.164765+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:26:38.164765+0000","last_undegraded":"2024-03-11T21:26:38.164765+0000","last_fullsized":"2024-03-11T21:26:38.164765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:45.984098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.984098+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.447890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161026+0000","last_change":"2024-03-11T21:26:14.197969+0000","last_active":"2024-03-11T21:26:28.161026+0000","last_peered":"2024-03-11T21:26:28.161026+0000","last_clean":"2024-03-11T21:26:28.161026+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:26:28.161026+0000","last_undegraded":"2024-03-11T21:26:28.161026+0000","last_fullsized":"2024-03-11T21:26:28.161026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.197767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.197767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:40.681392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.161809+0000","last_change":"2024-03-11T21:25:58.434898+0000","last_active":"2024-03-11T21:26:28.161809+0000","last_peered":"2024-03-11T21:26:28.161809+0000","last_clean":"2024-03-11T21:26:28.161809+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:26:28.161809+0000","last_undegraded":"2024-03-11T21:26:28.161809+0000","last_fullsized":"2024-03-11T21:26:28.161809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:58.434794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:58.434794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641084+0000","last_change":"2024-03-11T21:25:52.032818+0000","last_active":"2024-03-11T21:26:47.641084+0000","last_peered":"2024-03-11T21:26:47.641084+0000","last_clean":"2024-03-11T21:26:47.641084+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:26:47.641084+0000","last_undegraded":"2024-03-11T21:26:47.641084+0000","last_fullsized":"2024-03-11T21:26:47.641084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:52.032606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:52.032606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:16.435492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":138,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.650094+0000","last_change":"2024-03-11T21:26:17.927107+0000","last_active":"2024-03-11T21:26:27.650094+0000","last_peered":"2024-03-11T21:26:27.650094+0000","last_clean":"2024-03-11T21:26:27.650094+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:26:27.650094+0000","last_undegraded":"2024-03-11T21:26:27.650094+0000","last_fullsized":"2024-03-11T21:26:27.650094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:26:17.926863+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:26:17.926863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.586229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159293+0000","last_change":"2024-03-11T21:25:41.358918+0000","last_active":"2024-03-11T21:26:28.159293+0000","last_peered":"2024-03-11T21:26:28.159293+0000","last_clean":"2024-03-11T21:26:28.159293+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:26:28.159293+0000","last_undegraded":"2024-03-11T21:26:28.159293+0000","last_fullsized":"2024-03-11T21:26:28.159293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.358679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.152690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":161,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:38.159786+0000","last_change":"2024-03-11T21:25:57.003681+0000","last_active":"2024-03-11T21:26:38.159786+0000","last_peered":"2024-03-11T21:26:38.159786+0000","last_clean":"2024-03-11T21:26:38.159786+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:26:38.159786+0000","last_undegraded":"2024-03-11T21:26:38.159786+0000","last_fullsized":"2024-03-11T21:26:38.159786+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:25:57.003432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.003432+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:15.840351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":124,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646670+0000","last_change":"2024-03-11T21:26:41.718271+0000","last_active":"2024-03-11T21:26:47.646670+0000","last_peered":"2024-03-11T21:26:47.646670+0000","last_clean":"2024-03-11T21:26:47.646670+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:26:47.646670+0000","last_undegraded":"2024-03-11T21:26:47.646670+0000","last_fullsized":"2024-03-11T21:26:47.646670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.718071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.718071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:44.881942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":135,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160291+0000","last_change":"2024-03-11T21:25:40.954211+0000","last_active":"2024-03-11T21:26:28.160291+0000","last_peered":"2024-03-11T21:26:28.160291+0000","last_clean":"2024-03-11T21:26:28.160291+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:26:28.160291+0000","last_undegraded":"2024-03-11T21:26:28.160291+0000","last_fullsized":"2024-03-11T21:26:28.160291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:25:40.953969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.953969+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.832583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:27.651601+0000","last_change":"2024-03-11T21:26:05.072908+0000","last_active":"2024-03-11T21:26:27.651601+0000","last_peered":"2024-03-11T21:26:27.651601+0000","last_clean":"2024-03-11T21:26:27.651601+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:26:27.651601+0000","last_undegraded":"2024-03-11T21:26:27.651601+0000","last_fullsized":"2024-03-11T21:26:27.651601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.072676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.072676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.590231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":164,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994388+0000","last_change":"2024-03-11T21:25:40.987597+0000","last_active":"2024-03-11T21:26:47.994388+0000","last_peered":"2024-03-11T21:26:47.994388+0000","last_clean":"2024-03-11T21:26:47.994388+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:26:47.994388+0000","last_undegraded":"2024-03-11T21:26:47.994388+0000","last_fullsized":"2024-03-11T21:26:47.994388+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:25:40.987394+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:25:40.987394+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645322+0000","last_change":"2024-03-11T21:26:07.649240+0000","last_active":"2024-03-11T21:26:47.645322+0000","last_peered":"2024-03-11T21:26:47.645322+0000","last_clean":"2024-03-11T21:26:47.645322+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:26:47.645322+0000","last_undegraded":"2024-03-11T21:26:47.645322+0000","last_fullsized":"2024-03-11T21:26:47.645322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:07.649133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:26:07.649133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.066187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.160791+0000","last_change":"2024-03-11T21:25:29.024987+0000","last_active":"2024-03-11T21:26:28.160791+0000","last_peered":"2024-03-11T21:26:28.160791+0000","last_clean":"2024-03-11T21:26:28.160791+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:26:28.160791+0000","last_undegraded":"2024-03-11T21:26:28.160791+0000","last_fullsized":"2024-03-11T21:26:28.160791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:25:29.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:29.024617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:39.368987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994897+0000","last_change":"2024-03-11T21:25:32.119545+0000","last_active":"2024-03-11T21:26:47.994897+0000","last_peered":"2024-03-11T21:26:47.994897+0000","last_clean":"2024-03-11T21:26:47.994897+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:26:47.994897+0000","last_undegraded":"2024-03-11T21:26:47.994897+0000","last_fullsized":"2024-03-11T21:26:47.994897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:32.119316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:32.119316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.928370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.647207+0000","last_change":"2024-03-11T21:26:12.573500+0000","last_active":"2024-03-11T21:26:47.647207+0000","last_peered":"2024-03-11T21:26:47.647207+0000","last_clean":"2024-03-11T21:26:47.647207+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:26:47.647207+0000","last_undegraded":"2024-03-11T21:26:47.647207+0000","last_fullsized":"2024-03-11T21:26:47.647207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.573292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.573292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:13.647582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":124,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642003+0000","last_change":"2024-03-11T21:26:46.780598+0000","last_active":"2024-03-11T21:26:47.642003+0000","last_peered":"2024-03-11T21:26:47.642003+0000","last_clean":"2024-03-11T21:26:47.642003+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:26:47.642003+0000","last_undegraded":"2024-03-11T21:26:47.642003+0000","last_fullsized":"2024-03-11T21:26:47.642003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.780383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.780383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.322054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":150,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155514+0000","last_change":"2024-03-11T21:26:02.945068+0000","last_active":"2024-03-11T21:26:28.155514+0000","last_peered":"2024-03-11T21:26:28.155514+0000","last_clean":"2024-03-11T21:26:28.155514+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:26:28.155514+0000","last_undegraded":"2024-03-11T21:26:28.155514+0000","last_fullsized":"2024-03-11T21:26:28.155514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:26:02.944783+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.944783+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.686877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.640632+0000","last_change":"2024-03-11T21:26:12.157386+0000","last_active":"2024-03-11T21:26:47.640632+0000","last_peered":"2024-03-11T21:26:47.640632+0000","last_clean":"2024-03-11T21:26:47.640632+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:26:47.640632+0000","last_undegraded":"2024-03-11T21:26:47.640632+0000","last_fullsized":"2024-03-11T21:26:47.640632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.157160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.157160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.057339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994066+0000","last_change":"2024-03-11T21:26:38.936837+0000","last_active":"2024-03-11T21:26:47.994066+0000","last_peered":"2024-03-11T21:26:47.994066+0000","last_clean":"2024-03-11T21:26:47.994066+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:26:47.994066+0000","last_undegraded":"2024-03-11T21:26:47.994066+0000","last_fullsized":"2024-03-11T21:26:47.994066+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.936573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.936573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.223059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994353+0000","last_change":"2024-03-11T21:25:44.013839+0000","last_active":"2024-03-11T21:26:47.994353+0000","last_peered":"2024-03-11T21:26:47.994353+0000","last_clean":"2024-03-11T21:26:47.994353+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:26:47.994353+0000","last_undegraded":"2024-03-11T21:26:47.994353+0000","last_fullsized":"2024-03-11T21:26:47.994353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:44.013746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:44.013746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.618233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.642490+0000","last_change":"2024-03-11T21:26:14.127753+0000","last_active":"2024-03-11T21:26:47.642490+0000","last_peered":"2024-03-11T21:26:47.642490+0000","last_clean":"2024-03-11T21:26:47.642490+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:26:47.642490+0000","last_undegraded":"2024-03-11T21:26:47.642490+0000","last_fullsized":"2024-03-11T21:26:47.642490+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:26:14.127405+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.127405+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:31.505662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.645777+0000","last_change":"2024-03-11T21:26:15.354895+0000","last_active":"2024-03-11T21:26:47.645777+0000","last_peered":"2024-03-11T21:26:47.645777+0000","last_clean":"2024-03-11T21:26:47.645777+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:26:47.645777+0000","last_undegraded":"2024-03-11T21:26:47.645777+0000","last_fullsized":"2024-03-11T21:26:47.645777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:15.354676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:26:15.354676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.022398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994142+0000","last_change":"2024-03-11T21:26:41.906852+0000","last_active":"2024-03-11T21:26:47.994142+0000","last_peered":"2024-03-11T21:26:47.994142+0000","last_clean":"2024-03-11T21:26:47.994142+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:26:47.994142+0000","last_undegraded":"2024-03-11T21:26:47.994142+0000","last_fullsized":"2024-03-11T21:26:47.994142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.906678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.906678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.797651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.646045+0000","last_change":"2024-03-11T21:25:30.696848+0000","last_active":"2024-03-11T21:26:47.646045+0000","last_peered":"2024-03-11T21:26:47.646045+0000","last_clean":"2024-03-11T21:26:47.646045+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:26:47.646045+0000","last_undegraded":"2024-03-11T21:26:47.646045+0000","last_fullsized":"2024-03-11T21:26:47.646045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:30.696738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:30.696738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.966425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":125,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.159545+0000","last_change":"2024-03-11T21:25:45.376793+0000","last_active":"2024-03-11T21:26:28.159545+0000","last_peered":"2024-03-11T21:26:28.159545+0000","last_clean":"2024-03-11T21:26:28.159545+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:26:28.159545+0000","last_undegraded":"2024-03-11T21:26:28.159545+0000","last_fullsized":"2024-03-11T21:26:28.159545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:25:45.376408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:25:45.376408+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.150185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-11T21:26:28.155781+0000","last_change":"2024-03-11T21:26:05.391058+0000","last_active":"2024-03-11T21:26:28.155781+0000","last_peered":"2024-03-11T21:26:28.155781+0000","last_clean":"2024-03-11T21:26:28.155781+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:26:28.155781+0000","last_undegraded":"2024-03-11T21:26:28.155781+0000","last_fullsized":"2024-03-11T21:26:28.155781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.390850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.390850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.856109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.994836+0000","last_change":"2024-03-11T21:25:22.246229+0000","last_active":"2024-03-11T21:26:47.994836+0000","last_peered":"2024-03-11T21:26:47.994836+0000","last_clean":"2024-03-11T21:26:47.994836+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:26:47.994836+0000","last_undegraded":"2024-03-11T21:26:47.994836+0000","last_fullsized":"2024-03-11T21:26:47.994836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:22.245940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:25:22.245940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.310931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.643256+0000","last_change":"2024-03-11T21:26:31.873778+0000","last_active":"2024-03-11T21:26:47.643256+0000","last_peered":"2024-03-11T21:26:47.643256+0000","last_clean":"2024-03-11T21:26:47.643256+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:26:47.643256+0000","last_undegraded":"2024-03-11T21:26:47.643256+0000","last_fullsized":"2024-03-11T21:26:47.643256+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:31.873666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.873666+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.702452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":163,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.641437+0000","last_change":"2024-03-11T21:26:29.009973+0000","last_active":"2024-03-11T21:26:47.641437+0000","last_peered":"2024-03-11T21:26:47.641437+0000","last_clean":"2024-03-11T21:26:47.641437+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:26:47.641437+0000","last_undegraded":"2024-03-11T21:26:47.641437+0000","last_fullsized":"2024-03-11T21:26:47.641437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:29.009718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.009718+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.739503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841592,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36688,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708464,"statfs":{"total":95995035648,"available":95957467136,"internally_reserved":0,"allocated":1380352,"data_stored":997461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168641,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28320,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716832,"statfs":{"total":95995035648,"available":95966035968,"internally_reserved":0,"allocated":1335296,"data_stored":987541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":761856,"data_stored":404755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36652,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708500,"statfs":{"total":95995035648,"available":95957504000,"internally_reserved":0,"allocated":1347584,"data_stored":988658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378731,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":745472,"data_stored":397633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117087,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":733184,"data_stored":397111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":778240,"data_stored":398580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495691,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31896,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":737280,"data_stored":397093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:26:50.245 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:26:50.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump 2024-03-11T21:26:50.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:50 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.123017+0000 mgr.y (mgr.14550) 142 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:50.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:50 smithi143 bash[21512]: audit 2024-03-11T21:26:49.414674+0000 mgr.y (mgr.14550) 143 : audit 0 from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:50.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:50 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.123017+0000 mgr.y (mgr.14550) 142 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:50.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:50 smithi143 bash[29284]: audit 2024-03-11T21:26:49.414674+0000 mgr.y (mgr.14550) 143 : audit 0 from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:50 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.123017+0000 mgr.y (mgr.14550) 142 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:50 smithi203 bash[23654]: audit 2024-03-11T21:26:49.414674+0000 mgr.y (mgr.14550) 143 : audit 0 from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.908307+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:49.911408+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:50.710049+0000 osd.7 (osd.7) 107 : cluster 0 3.13 scrub starts 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:50.713713+0000 osd.7 (osd.7) 108 : cluster 0 3.13 scrub ok 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:50.745757+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[21512]: cluster 2024-03-11T21:26:50.749687+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-03-11T21:26:51.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.908307+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-03-11T21:26:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:49.911408+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-03-11T21:26:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:50.710049+0000 osd.7 (osd.7) 107 : cluster 0 3.13 scrub starts 2024-03-11T21:26:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:50.713713+0000 osd.7 (osd.7) 108 : cluster 0 3.13 scrub ok 2024-03-11T21:26:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:50.745757+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-03-11T21:26:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:51 smithi143 bash[29284]: cluster 2024-03-11T21:26:50.749687+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.908307+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:49.911408+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:50.710049+0000 osd.7 (osd.7) 107 : cluster 0 3.13 scrub starts 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:50.713713+0000 osd.7 (osd.7) 108 : cluster 0 3.13 scrub ok 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:50.745757+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-03-11T21:26:52.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:51 smithi203 bash[23654]: cluster 2024-03-11T21:26:50.749687+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-03-11T21:26:52.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:26:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:51.123638+0000 mgr.y (mgr.14550) 144 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:51.596449+0000 osd.0 (osd.0) 85 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:51.608209+0000 osd.0 (osd.0) 86 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:51.715672+0000 osd.5 (osd.5) 97 : cluster 0 2.16 scrub starts 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:51.717998+0000 osd.5 (osd.5) 98 : cluster 0 2.16 scrub ok 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.281534+0000 osd.3 (osd.3) 109 : cluster 0 5.0 scrub starts 2024-03-11T21:26:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.285131+0000 osd.3 (osd.3) 110 : cluster 0 5.0 scrub ok 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:51.123638+0000 mgr.y (mgr.14550) 144 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:51.596449+0000 osd.0 (osd.0) 85 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:51.608209+0000 osd.0 (osd.0) 86 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:51.715672+0000 osd.5 (osd.5) 97 : cluster 0 2.16 scrub starts 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:51.717998+0000 osd.5 (osd.5) 98 : cluster 0 2.16 scrub ok 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.281534+0000 osd.3 (osd.3) 109 : cluster 0 5.0 scrub starts 2024-03-11T21:26:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:52 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.285131+0000 osd.3 (osd.3) 110 : cluster 0 5.0 scrub ok 2024-03-11T21:26:53.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:51.123638+0000 mgr.y (mgr.14550) 144 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:51.596449+0000 osd.0 (osd.0) 85 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:51.608209+0000 osd.0 (osd.0) 86 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:51.715672+0000 osd.5 (osd.5) 97 : cluster 0 2.16 scrub starts 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:51.717998+0000 osd.5 (osd.5) 98 : cluster 0 2.16 scrub ok 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.281534+0000 osd.3 (osd.3) 109 : cluster 0 5.0 scrub starts 2024-03-11T21:26:53.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:52 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.285131+0000 osd.3 (osd.3) 110 : cluster 0 5.0 scrub ok 2024-03-11T21:26:53.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.588514+0000 osd.0 (osd.0) 87 : cluster 0 3.17 scrub starts 2024-03-11T21:26:53.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.592262+0000 osd.0 (osd.0) 88 : cluster 0 3.17 scrub ok 2024-03-11T21:26:53.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.730351+0000 osd.5 (osd.5) 99 : cluster 0 2.3 scrub starts 2024-03-11T21:26:53.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[21512]: cluster 2024-03-11T21:26:52.732666+0000 osd.5 (osd.5) 100 : cluster 0 2.3 scrub ok 2024-03-11T21:26:53.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.588514+0000 osd.0 (osd.0) 87 : cluster 0 3.17 scrub starts 2024-03-11T21:26:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.592262+0000 osd.0 (osd.0) 88 : cluster 0 3.17 scrub ok 2024-03-11T21:26:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.730351+0000 osd.5 (osd.5) 99 : cluster 0 2.3 scrub starts 2024-03-11T21:26:53.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:53 smithi143 bash[29284]: cluster 2024-03-11T21:26:52.732666+0000 osd.5 (osd.5) 100 : cluster 0 2.3 scrub ok 2024-03-11T21:26:54.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:53 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.588514+0000 osd.0 (osd.0) 87 : cluster 0 3.17 scrub starts 2024-03-11T21:26:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:53 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.592262+0000 osd.0 (osd.0) 88 : cluster 0 3.17 scrub ok 2024-03-11T21:26:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:53 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.730351+0000 osd.5 (osd.5) 99 : cluster 0 2.3 scrub starts 2024-03-11T21:26:54.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:53 smithi203 bash[23654]: cluster 2024-03-11T21:26:52.732666+0000 osd.5 (osd.5) 100 : cluster 0 2.3 scrub ok 2024-03-11T21:26:54.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:54 smithi143 bash[29284]: cluster 2024-03-11T21:26:53.125536+0000 mgr.y (mgr.14550) 145 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:54 smithi143 bash[21512]: cluster 2024-03-11T21:26:53.125536+0000 mgr.y (mgr.14550) 145 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:55.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:54 smithi203 bash[23654]: cluster 2024-03-11T21:26:53.125536+0000 mgr.y (mgr.14550) 145 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:55.029 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:55.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.614790+0000 osd.7 (osd.7) 109 : cluster 0 3.15 scrub starts 2024-03-11T21:26:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.618676+0000 osd.7 (osd.7) 110 : cluster 0 3.15 scrub ok 2024-03-11T21:26:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.926286+0000 osd.6 (osd.6) 59 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:26:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.928730+0000 osd.6 (osd.6) 60 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:26:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.975170+0000 osd.4 (osd.4) 77 : cluster 0 3.14 scrub starts 2024-03-11T21:26:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[21512]: cluster 2024-03-11T21:26:54.979239+0000 osd.4 (osd.4) 78 : cluster 0 3.14 scrub ok 2024-03-11T21:26:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.614790+0000 osd.7 (osd.7) 109 : cluster 0 3.15 scrub starts 2024-03-11T21:26:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.618676+0000 osd.7 (osd.7) 110 : cluster 0 3.15 scrub ok 2024-03-11T21:26:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.926286+0000 osd.6 (osd.6) 59 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:26:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.928730+0000 osd.6 (osd.6) 60 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:26:55.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.975170+0000 osd.4 (osd.4) 77 : cluster 0 3.14 scrub starts 2024-03-11T21:26:55.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:55 smithi143 bash[29284]: cluster 2024-03-11T21:26:54.979239+0000 osd.4 (osd.4) 78 : cluster 0 3.14 scrub ok 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.614790+0000 osd.7 (osd.7) 109 : cluster 0 3.15 scrub starts 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.618676+0000 osd.7 (osd.7) 110 : cluster 0 3.15 scrub ok 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.926286+0000 osd.6 (osd.6) 59 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.928730+0000 osd.6 (osd.6) 60 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.975170+0000 osd.4 (osd.4) 77 : cluster 0 3.14 scrub starts 2024-03-11T21:26:56.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:55 smithi203 bash[23654]: cluster 2024-03-11T21:26:54.979239+0000 osd.4 (osd.4) 78 : cluster 0 3.14 scrub ok 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.126233+0000 mgr.y (mgr.14550) 146 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.481739+0000 osd.5 (osd.5) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.481885+0000 osd.5 (osd.5) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.652854+0000 osd.0 (osd.0) 89 : cluster 0 2.13 scrub starts 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.654458+0000 osd.0 (osd.0) 90 : cluster 0 2.13 scrub ok 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.655597+0000 osd.7 (osd.7) 111 : cluster 0 6.3 scrub starts 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.658455+0000 osd.7 (osd.7) 112 : cluster 0 6.3 scrub ok 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:55.702292+0000 mon.a (mon.0) 881 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:26:56.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:56.248821+0000 osd.3 (osd.3) 111 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[21512]: cluster 2024-03-11T21:26:56.252409+0000 osd.3 (osd.3) 112 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.126233+0000 mgr.y (mgr.14550) 146 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.481739+0000 osd.5 (osd.5) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.481885+0000 osd.5 (osd.5) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.652854+0000 osd.0 (osd.0) 89 : cluster 0 2.13 scrub starts 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.654458+0000 osd.0 (osd.0) 90 : cluster 0 2.13 scrub ok 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.655597+0000 osd.7 (osd.7) 111 : cluster 0 6.3 scrub starts 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.658455+0000 osd.7 (osd.7) 112 : cluster 0 6.3 scrub ok 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:55.702292+0000 mon.a (mon.0) 881 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:56.248821+0000 osd.3 (osd.3) 111 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:26:56.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:56 smithi143 bash[29284]: cluster 2024-03-11T21:26:56.252409+0000 osd.3 (osd.3) 112 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.126233+0000 mgr.y (mgr.14550) 146 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.481739+0000 osd.5 (osd.5) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.481885+0000 osd.5 (osd.5) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.652854+0000 osd.0 (osd.0) 89 : cluster 0 2.13 scrub starts 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.654458+0000 osd.0 (osd.0) 90 : cluster 0 2.13 scrub ok 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.655597+0000 osd.7 (osd.7) 111 : cluster 0 6.3 scrub starts 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.658455+0000 osd.7 (osd.7) 112 : cluster 0 6.3 scrub ok 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:55.702292+0000 mon.a (mon.0) 881 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:56.248821+0000 osd.3 (osd.3) 111 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:26:57.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:56 smithi203 bash[23654]: cluster 2024-03-11T21:26:56.252409+0000 osd.3 (osd.3) 112 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[21512]: cluster 2024-03-11T21:26:56.918400+0000 osd.6 (osd.6) 61 : cluster 0 3.10 scrub starts 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[21512]: cluster 2024-03-11T21:26:56.922344+0000 osd.6 (osd.6) 62 : cluster 0 3.10 scrub ok 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.220752+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.223034+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[29284]: cluster 2024-03-11T21:26:56.918400+0000 osd.6 (osd.6) 61 : cluster 0 3.10 scrub starts 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[29284]: cluster 2024-03-11T21:26:56.922344+0000 osd.6 (osd.6) 62 : cluster 0 3.10 scrub ok 2024-03-11T21:26:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.220752+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T21:26:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:57 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.223034+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T21:26:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:57 smithi203 bash[23654]: cluster 2024-03-11T21:26:56.918400+0000 osd.6 (osd.6) 61 : cluster 0 3.10 scrub starts 2024-03-11T21:26:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:57 smithi203 bash[23654]: cluster 2024-03-11T21:26:56.922344+0000 osd.6 (osd.6) 62 : cluster 0 3.10 scrub ok 2024-03-11T21:26:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:57 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.220752+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T21:26:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:57 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.223034+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T21:26:58.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/260963771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 msgr2=0x7fa88c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- 172.21.15.143:0/260963771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c10ee70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa88400b0a0 tx=0x7fa88400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/260963771 shutdown_connections 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- 172.21.15.143:0/260963771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c10ee70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- 172.21.15.143:0/260963771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c06abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- 172.21.15.143:0/260963771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/260963771 >> 172.21.15.143:0/260963771 conn(0x7fa88c076d80 msgr2=0x7fa88c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/260963771 shutdown_connections 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/260963771 wait complete. 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 librados: starting msgr at 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 librados: starting objecter 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 Processor -- start 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- start start 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 librados: setting wanted keys 2024-03-11T21:26:58.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 librados: calling monclient init 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c1117e0 con 0x7fa88c1044c0 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c111660 con 0x7fa88c06a7f0 2024-03-11T21:26:58.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c111960 con 0x7fa88c103b00 2024-03-11T21:26:58.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:54932/0 (socket says 172.21.15.143:54932) 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa89232a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 -- 172.21.15.143:0/955236556 learned_addr learned my addr 172.21.15.143:0/955236556 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891b29700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 -- 172.21.15.143:0/955236556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 msgr2=0x7fa88c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 -- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 msgr2=0x7fa88c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 -- 172.21.15.143:0/955236556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c198410 con 0x7fa88c103b00 2024-03-11T21:26:58.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891b29700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa89232a700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa891328700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa88000e9c0 tx=0x7fa88000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa87effd700 10 client.?.objecter ms_handle_connect 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa87effd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88000cbd0 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa88000cd70 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c198700 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880005610 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 10 client.24626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c1a49e0 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 librados: init done 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.372+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c103610 con 0x7fa88c103b00 2024-03-11T21:26:58.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa880004750 con 0x7fa88c103b00 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 0x7fa878071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(122..122 src has 66..122) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fa88008e490 con 0x7fa88c103b00 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 10 client.24626.objecter ms_dispatch 0x7fa88c001550 osd_map(122..122 src has 66..122) v4 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 3 client.24626.objecter handle_osd_map got epochs [122,122] > 0 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 3 client.24626.objecter handle_osd_map decoding full epoch 122 2024-03-11T21:26:58.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa891b29700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 0x7fa878071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:58.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 20 client.24626.objecter dump_active .. 0 homeless 2024-03-11T21:26:58.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa87effd700 10 client.24626.objecter ms_handle_connect 0x7fa87806f820 2024-03-11T21:26:58.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.376+0000 7fa891b29700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 0x7fa878071ce0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa888005fd0 tx=0x7fa888005dc0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:58.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.380+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa88005b9b0 con 0x7fa88c103b00 2024-03-11T21:26:58.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:26:58.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.604+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fa88c04faa0 con 0x7fa88c103b00 2024-03-11T21:26:58.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.604+0000 7fa87effd700 1 -- 172.21.15.143:0/955236556 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v122) v1 ==== 56+0+5196 (secure 0 0 0) 0x7fa880055e20 con 0x7fa88c103b00 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:epoch 122 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:created 2024-03-11T21:12:47.384719+0000 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:modified 2024-03-11T21:26:55.679713+0000 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:crush_version 18 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:full_ratio 0.95 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:backfillfull_ratio 0.9 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:nearfull_ratio 0.85 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:require_min_compat_client luminous 2024-03-11T21:26:58.583 INFO:teuthology.orchestra.run.smithi143.stdout:min_compat_client luminous 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:require_osd_release squid 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:stretch_mode_enabled false 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 4 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 87 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:max_osd 8 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] [v2:172.21.15.143:6804/2903432023,v1:172.21.15.143:6805/2903432023] exists,up 62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1 up in weight 1 up_from 13 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] [v2:172.21.15.143:6812/397942214,v1:172.21.15.143:6813/397942214] exists,up 2948dc44-68d3-4b3d-92bf-2b0c46374f02 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2 up in weight 1 up_from 19 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] [v2:172.21.15.143:6820/2592131625,v1:172.21.15.143:6821/2592131625] exists,up 7217589a-5835-407b-91a7-f5c055a48f3d 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3 up in weight 1 up_from 27 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] [v2:172.21.15.143:6828/2146571980,v1:172.21.15.143:6829/2146571980] exists,up a3e7177f-a5a0-44ad-a65d-9aeac3a680cc 2024-03-11T21:26:58.584 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4 up in weight 1 up_from 35 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] [v2:172.21.15.203:6802/2090835691,v1:172.21.15.203:6803/2090835691] exists,up d7b9e088-d377-4335-8863-11d581828f7e 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5 up in weight 1 up_from 46 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] [v2:172.21.15.203:6810/2432038345,v1:172.21.15.203:6811/2432038345] exists,up b3b2a75c-68a5-4e00-8068-622e1ebbaa01 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6 up in weight 1 up_from 56 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] [v2:172.21.15.203:6818/513748834,v1:172.21.15.203:6819/513748834] exists,up ceffb234-db0c-4ffa-a142-9b201df9b241 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7 up in weight 1 up_from 66 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] [v2:172.21.15.203:6826/1251497952,v1:172.21.15.203:6827/1251497952] exists,up 88b53171-69d2-4922-a787-e63d44d01c60 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:pg_upmap_items 2.18 [7,2] 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6801/3638739149 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/4274199357 expires 2024-03-12T21:13:53.401579+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/3217951344 expires 2024-03-12T21:13:19.169870+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/3912416566 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/906834629 expires 2024-03-12T21:13:19.169870+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/2791822486 expires 2024-03-12T21:13:19.169870+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6800/3787195463 expires 2024-03-12T21:13:53.401579+0000 2024-03-11T21:26:58.585 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/2190104670 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/4171894415 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6801/3787195463 expires 2024-03-12T21:13:53.401579+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/2845155916 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6801/1839244747 expires 2024-03-12T21:13:19.169870+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/2913593434 expires 2024-03-12T21:13:53.401579+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/143435387 expires 2024-03-12T21:13:53.401579+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:0/2930358289 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6800/1839244747 expires 2024-03-12T21:13:19.169870+0000 2024-03-11T21:26:58.586 INFO:teuthology.orchestra.run.smithi143.stdout:blocklist 172.21.15.143:6800/3638739149 expires 2024-03-12T21:23:57.022184+0000 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 10 librados: watch_flush enter 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 10 librados: watch_flush exit 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 20 client.24626.objecter shutdown clearing up homeless session... 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 10 client.24626.objecter successfully canceled tick 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 msgr2=0x7fa878071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 0x7fa878071ce0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa888005fd0 tx=0x7fa888005dc0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 msgr2=0x7fa88c19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:58.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa88000e9c0 tx=0x7fa88000ecd0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 shutdown_connections 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa87806f820 0x7fa878071ce0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa88c1044c0 0x7fa88c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa88c06a7f0 0x7fa88c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 --2- 172.21.15.143:0/955236556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa88c103b00 0x7fa88c19dfa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 >> 172.21.15.143:0/955236556 conn(0x7fa88c076d80 msgr2=0x7fa88c106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 shutdown_connections 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 -- 172.21.15.143:0/955236556 wait complete. 2024-03-11T21:26:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:26:58.608+0000 7fa893d8d700 1 librados: shutdown 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.126833+0000 mgr.y (mgr.14550) 147 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.673909+0000 osd.5 (osd.5) 103 : cluster 0 5.16 scrub starts 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.676452+0000 osd.5 (osd.5) 104 : cluster 0 5.16 scrub ok 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: audit 2024-03-11T21:26:57.774651+0000 mon.c (mon.2) 85 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.833406+0000 osd.2 (osd.2) 67 : cluster 0 2.e scrub starts 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: cluster 2024-03-11T21:26:57.835712+0000 osd.2 (osd.2) 68 : cluster 0 2.e scrub ok 2024-03-11T21:26:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[21512]: audit 2024-03-11T21:26:58.609310+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.143:0/955236556' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.126833+0000 mgr.y (mgr.14550) 147 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.673909+0000 osd.5 (osd.5) 103 : cluster 0 5.16 scrub starts 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.676452+0000 osd.5 (osd.5) 104 : cluster 0 5.16 scrub ok 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: audit 2024-03-11T21:26:57.774651+0000 mon.c (mon.2) 85 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.833406+0000 osd.2 (osd.2) 67 : cluster 0 2.e scrub starts 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: cluster 2024-03-11T21:26:57.835712+0000 osd.2 (osd.2) 68 : cluster 0 2.e scrub ok 2024-03-11T21:26:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:58 smithi143 bash[29284]: audit 2024-03-11T21:26:58.609310+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.143:0/955236556' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.126833+0000 mgr.y (mgr.14550) 147 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.673909+0000 osd.5 (osd.5) 103 : cluster 0 5.16 scrub starts 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.676452+0000 osd.5 (osd.5) 104 : cluster 0 5.16 scrub ok 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: audit 2024-03-11T21:26:57.774651+0000 mon.c (mon.2) 85 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.833406+0000 osd.2 (osd.2) 67 : cluster 0 2.e scrub starts 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: cluster 2024-03-11T21:26:57.835712+0000 osd.2 (osd.2) 68 : cluster 0 2.e scrub ok 2024-03-11T21:26:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:58 smithi203 bash[23654]: audit 2024-03-11T21:26:58.609310+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.143:0/955236556' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:26:59.438 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] [v2:172.21.15.143:6804/2903432023,v1:172.21.15.143:6805/2903432023] exists,up 62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d', 'osd.1 up in weight 1 up_from 13 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] [v2:172.21.15.143:6812/397942214,v1:172.21.15.143:6813/397942214] exists,up 2948dc44-68d3-4b3d-92bf-2b0c46374f02', 'osd.2 up in weight 1 up_from 19 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] [v2:172.21.15.143:6820/2592131625,v1:172.21.15.143:6821/2592131625] exists,up 7217589a-5835-407b-91a7-f5c055a48f3d', 'osd.3 up in weight 1 up_from 27 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] [v2:172.21.15.143:6828/2146571980,v1:172.21.15.143:6829/2146571980] exists,up a3e7177f-a5a0-44ad-a65d-9aeac3a680cc', 'osd.4 up in weight 1 up_from 35 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] [v2:172.21.15.203:6802/2090835691,v1:172.21.15.203:6803/2090835691] exists,up d7b9e088-d377-4335-8863-11d581828f7e', 'osd.5 up in weight 1 up_from 46 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] [v2:172.21.15.203:6810/2432038345,v1:172.21.15.203:6811/2432038345] exists,up b3b2a75c-68a5-4e00-8068-622e1ebbaa01', 'osd.6 up in weight 1 up_from 56 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] [v2:172.21.15.203:6818/513748834,v1:172.21.15.203:6819/513748834] exists,up ceffb234-db0c-4ffa-a142-9b201df9b241', 'osd.7 up in weight 1 up_from 66 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] [v2:172.21.15.203:6826/1251497952,v1:172.21.15.203:6827/1251497952] exists,up 88b53171-69d2-4922-a787-e63d44d01c60'] 2024-03-11T21:26:59.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon mon.a config show 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: audit 2024-03-11T21:26:58.222706+0000 mgr.y (mgr.14550) 148 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: cluster 2024-03-11T21:26:58.679925+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: cluster 2024-03-11T21:26:58.680046+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: cluster 2024-03-11T21:26:58.901151+0000 osd.4 (osd.4) 79 : cluster 0 6.17 scrub starts 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: cluster 2024-03-11T21:26:58.903601+0000 osd.4 (osd.4) 80 : cluster 0 6.17 scrub ok 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.596799+0000 mon.a (mon.0) 882 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:26:59.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: audit 2024-03-11T21:26:58.222706+0000 mgr.y (mgr.14550) 148 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: cluster 2024-03-11T21:26:58.679925+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T21:26:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: cluster 2024-03-11T21:26:58.680046+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T21:26:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: cluster 2024-03-11T21:26:58.901151+0000 osd.4 (osd.4) 79 : cluster 0 6.17 scrub starts 2024-03-11T21:26:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: cluster 2024-03-11T21:26:58.903601+0000 osd.4 (osd.4) 80 : cluster 0 6.17 scrub ok 2024-03-11T21:26:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:26:59 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.596799+0000 mon.a (mon.0) 882 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:27:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: audit 2024-03-11T21:26:58.222706+0000 mgr.y (mgr.14550) 148 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:00.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: cluster 2024-03-11T21:26:58.679925+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:00.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: cluster 2024-03-11T21:26:58.680046+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:00.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: cluster 2024-03-11T21:26:58.901151+0000 osd.4 (osd.4) 79 : cluster 0 6.17 scrub starts 2024-03-11T21:27:00.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: cluster 2024-03-11T21:26:58.903601+0000 osd.4 (osd.4) 80 : cluster 0 6.17 scrub ok 2024-03-11T21:27:00.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:26:59 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.596799+0000 mon.a (mon.0) 882 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.128363+0000 mgr.y (mgr.14550) 149 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.719490+0000 osd.5 (osd.5) 105 : cluster 0 3.1d scrub starts 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.725593+0000 osd.5 (osd.5) 106 : cluster 0 3.1d scrub ok 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.819435+0000 osd.2 (osd.2) 71 : cluster 0 2.c scrub starts 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.821986+0000 osd.2 (osd.2) 72 : cluster 0 2.c scrub ok 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.855246+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-03-11T21:27:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.857601+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.949088+0000 osd.4 (osd.4) 81 : cluster 0 5.1 scrub starts 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[29284]: cluster 2024-03-11T21:26:59.951229+0000 osd.4 (osd.4) 82 : cluster 0 5.1 scrub ok 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.128363+0000 mgr.y (mgr.14550) 149 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.719490+0000 osd.5 (osd.5) 105 : cluster 0 3.1d scrub starts 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.725593+0000 osd.5 (osd.5) 106 : cluster 0 3.1d scrub ok 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.819435+0000 osd.2 (osd.2) 71 : cluster 0 2.c scrub starts 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.821986+0000 osd.2 (osd.2) 72 : cluster 0 2.c scrub ok 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.855246+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.857601+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.949088+0000 osd.4 (osd.4) 81 : cluster 0 5.1 scrub starts 2024-03-11T21:27:00.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:00 smithi143 bash[21512]: cluster 2024-03-11T21:26:59.951229+0000 osd.4 (osd.4) 82 : cluster 0 5.1 scrub ok 2024-03-11T21:27:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.128363+0000 mgr.y (mgr.14550) 149 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.719490+0000 osd.5 (osd.5) 105 : cluster 0 3.1d scrub starts 2024-03-11T21:27:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.725593+0000 osd.5 (osd.5) 106 : cluster 0 3.1d scrub ok 2024-03-11T21:27:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.819435+0000 osd.2 (osd.2) 71 : cluster 0 2.c scrub starts 2024-03-11T21:27:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.821986+0000 osd.2 (osd.2) 72 : cluster 0 2.c scrub ok 2024-03-11T21:27:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.855246+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-03-11T21:27:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.857601+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-03-11T21:27:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.949088+0000 osd.4 (osd.4) 81 : cluster 0 5.1 scrub starts 2024-03-11T21:27:01.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:00 smithi203 bash[23654]: cluster 2024-03-11T21:26:59.951229+0000 osd.4 (osd.4) 82 : cluster 0 5.1 scrub ok 2024-03-11T21:27:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:01 smithi203 bash[23654]: cluster 2024-03-11T21:27:00.894807+0000 osd.1 (osd.1) 95 : cluster 0 2.d scrub starts 2024-03-11T21:27:02.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:01 smithi203 bash[23654]: cluster 2024-03-11T21:27:00.897224+0000 osd.1 (osd.1) 96 : cluster 0 2.d scrub ok 2024-03-11T21:27:02.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:01 smithi143 bash[29284]: cluster 2024-03-11T21:27:00.894807+0000 osd.1 (osd.1) 95 : cluster 0 2.d scrub starts 2024-03-11T21:27:02.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:01 smithi143 bash[29284]: cluster 2024-03-11T21:27:00.897224+0000 osd.1 (osd.1) 96 : cluster 0 2.d scrub ok 2024-03-11T21:27:02.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:01 smithi143 bash[21512]: cluster 2024-03-11T21:27:00.894807+0000 osd.1 (osd.1) 95 : cluster 0 2.d scrub starts 2024-03-11T21:27:02.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:01 smithi143 bash[21512]: cluster 2024-03-11T21:27:00.897224+0000 osd.1 (osd.1) 96 : cluster 0 2.d scrub ok 2024-03-11T21:27:02.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:02 smithi203 bash[23654]: cluster 2024-03-11T21:27:01.129106+0000 mgr.y (mgr.14550) 150 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:02 smithi143 bash[21512]: cluster 2024-03-11T21:27:01.129106+0000 mgr.y (mgr.14550) 150 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:03.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:02 smithi143 bash[29284]: cluster 2024-03-11T21:27:01.129106+0000 mgr.y (mgr.14550) 150 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:03 smithi203 bash[23654]: cluster 2024-03-11T21:27:02.845723+0000 osd.2 (osd.2) 73 : cluster 0 6.14 scrub starts 2024-03-11T21:27:04.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:03 smithi203 bash[23654]: cluster 2024-03-11T21:27:02.848541+0000 osd.2 (osd.2) 74 : cluster 0 6.14 scrub ok 2024-03-11T21:27:04.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:03 smithi203 bash[23654]: cluster 2024-03-11T21:27:03.212715+0000 osd.3 (osd.3) 115 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:27:04.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:03 smithi203 bash[23654]: cluster 2024-03-11T21:27:03.215026+0000 osd.3 (osd.3) 116 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[21512]: cluster 2024-03-11T21:27:02.845723+0000 osd.2 (osd.2) 73 : cluster 0 6.14 scrub starts 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[21512]: cluster 2024-03-11T21:27:02.848541+0000 osd.2 (osd.2) 74 : cluster 0 6.14 scrub ok 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[21512]: cluster 2024-03-11T21:27:03.212715+0000 osd.3 (osd.3) 115 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[21512]: cluster 2024-03-11T21:27:03.215026+0000 osd.3 (osd.3) 116 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[29284]: cluster 2024-03-11T21:27:02.845723+0000 osd.2 (osd.2) 73 : cluster 0 6.14 scrub starts 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[29284]: cluster 2024-03-11T21:27:02.848541+0000 osd.2 (osd.2) 74 : cluster 0 6.14 scrub ok 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[29284]: cluster 2024-03-11T21:27:03.212715+0000 osd.3 (osd.3) 115 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:27:04.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:03 smithi143 bash[29284]: cluster 2024-03-11T21:27:03.215026+0000 osd.3 (osd.3) 116 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:27:04.222 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:04 smithi203 bash[23654]: cluster 2024-03-11T21:27:03.130509+0000 mgr.y (mgr.14550) 151 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:04 smithi203 bash[23654]: cluster 2024-03-11T21:27:03.706147+0000 osd.5 (osd.5) 107 : cluster 0 5.f scrub starts 2024-03-11T21:27:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:04 smithi203 bash[23654]: cluster 2024-03-11T21:27:03.708584+0000 osd.5 (osd.5) 108 : cluster 0 5.f scrub ok 2024-03-11T21:27:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:04 smithi203 bash[23654]: cluster 2024-03-11T21:27:04.007521+0000 osd.4 (osd.4) 83 : cluster 0 3.1a scrub starts 2024-03-11T21:27:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:04 smithi203 bash[23654]: cluster 2024-03-11T21:27:04.011624+0000 osd.4 (osd.4) 84 : cluster 0 3.1a scrub ok 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[21512]: cluster 2024-03-11T21:27:03.130509+0000 mgr.y (mgr.14550) 151 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[21512]: cluster 2024-03-11T21:27:03.706147+0000 osd.5 (osd.5) 107 : cluster 0 5.f scrub starts 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[21512]: cluster 2024-03-11T21:27:03.708584+0000 osd.5 (osd.5) 108 : cluster 0 5.f scrub ok 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[21512]: cluster 2024-03-11T21:27:04.007521+0000 osd.4 (osd.4) 83 : cluster 0 3.1a scrub starts 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[21512]: cluster 2024-03-11T21:27:04.011624+0000 osd.4 (osd.4) 84 : cluster 0 3.1a scrub ok 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[29284]: cluster 2024-03-11T21:27:03.130509+0000 mgr.y (mgr.14550) 151 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[29284]: cluster 2024-03-11T21:27:03.706147+0000 osd.5 (osd.5) 107 : cluster 0 5.f scrub starts 2024-03-11T21:27:05.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[29284]: cluster 2024-03-11T21:27:03.708584+0000 osd.5 (osd.5) 108 : cluster 0 5.f scrub ok 2024-03-11T21:27:05.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[29284]: cluster 2024-03-11T21:27:04.007521+0000 osd.4 (osd.4) 83 : cluster 0 3.1a scrub starts 2024-03-11T21:27:05.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:04 smithi143 bash[29284]: cluster 2024-03-11T21:27:04.011624+0000 osd.4 (osd.4) 84 : cluster 0 3.1a scrub ok 2024-03-11T21:27:05.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:05 smithi143 bash[21512]: cluster 2024-03-11T21:27:04.803428+0000 osd.2 (osd.2) 75 : cluster 0 2.1 scrub starts 2024-03-11T21:27:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:05 smithi143 bash[21512]: cluster 2024-03-11T21:27:04.805090+0000 osd.2 (osd.2) 76 : cluster 0 2.1 scrub ok 2024-03-11T21:27:05.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:05 smithi143 bash[29284]: cluster 2024-03-11T21:27:04.803428+0000 osd.2 (osd.2) 75 : cluster 0 2.1 scrub starts 2024-03-11T21:27:05.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:05 smithi143 bash[29284]: cluster 2024-03-11T21:27:04.805090+0000 osd.2 (osd.2) 76 : cluster 0 2.1 scrub ok 2024-03-11T21:27:06.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:05 smithi203 bash[23654]: cluster 2024-03-11T21:27:04.803428+0000 osd.2 (osd.2) 75 : cluster 0 2.1 scrub starts 2024-03-11T21:27:06.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:05 smithi203 bash[23654]: cluster 2024-03-11T21:27:04.805090+0000 osd.2 (osd.2) 76 : cluster 0 2.1 scrub ok 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:05.131224+0000 mgr.y (mgr.14550) 152 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:05.683786+0000 osd.0 (osd.0) 91 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:05.686424+0000 osd.0 (osd.0) 92 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:05.954838+0000 osd.1 (osd.1) 97 : cluster 0 2.15 scrub starts 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:05.957234+0000 osd.1 (osd.1) 98 : cluster 0 2.15 scrub ok 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.104673+0000 osd.3 (osd.3) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.104781+0000 osd.3 (osd.3) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.211780+0000 osd.3 (osd.3) 119 : cluster 0 4.0 scrub starts 2024-03-11T21:27:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:06 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.214804+0000 osd.3 (osd.3) 120 : cluster 0 4.0 scrub ok 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:05.131224+0000 mgr.y (mgr.14550) 152 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:05.683786+0000 osd.0 (osd.0) 91 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:05.686424+0000 osd.0 (osd.0) 92 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:05.954838+0000 osd.1 (osd.1) 97 : cluster 0 2.15 scrub starts 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:05.957234+0000 osd.1 (osd.1) 98 : cluster 0 2.15 scrub ok 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.104673+0000 osd.3 (osd.3) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.104781+0000 osd.3 (osd.3) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.211780+0000 osd.3 (osd.3) 119 : cluster 0 4.0 scrub starts 2024-03-11T21:27:07.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.214804+0000 osd.3 (osd.3) 120 : cluster 0 4.0 scrub ok 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:05.131224+0000 mgr.y (mgr.14550) 152 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:05.683786+0000 osd.0 (osd.0) 91 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:05.686424+0000 osd.0 (osd.0) 92 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:05.954838+0000 osd.1 (osd.1) 97 : cluster 0 2.15 scrub starts 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:05.957234+0000 osd.1 (osd.1) 98 : cluster 0 2.15 scrub ok 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.104673+0000 osd.3 (osd.3) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.104781+0000 osd.3 (osd.3) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.211780+0000 osd.3 (osd.3) 119 : cluster 0 4.0 scrub starts 2024-03-11T21:27:07.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:06 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.214804+0000 osd.3 (osd.3) 120 : cluster 0 4.0 scrub ok 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout: "name": "mon.a", 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster": "ceph", 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout: "admin_socket_mode": "", 2024-03-11T21:27:07.455 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_client_required": "cephx, none", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_cluster_required": "cephx", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_debug": "true", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_service_required": "cephx", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "auth_supported": "", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_aio": "true", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_aio_poll_ms": "250", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_aio_reap_max": "16", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_async_discard": "false", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_block_size": "4096", 2024-03-11T21:27:07.456 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_debug_aio": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_debug_inflight_ios": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_enable_discard": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_flock_retry": "3", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_inject_crash": "0", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_ioring": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_ioring_hipri": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-11T21:27:07.457 INFO:teuthology.orchestra.run.smithi143.stdout: "bdev_type": "", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_alloc_size": "1048576", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_allocator": "hybrid", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_buffered_io": "true", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_check_for_zeros": "false", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_compact_log_sync": "false", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-11T21:27:07.458 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_max_log_runway": "4194304", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_max_prefetch": "1048576", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_min_flush_size": "524288", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_min_log_runway": "1048576", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_replay_recovery": "false", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_sync_write": "false", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_allocation_from_file": "true", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_allocator": "hybrid", 2024-03-11T21:27:07.459 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_create": "true", 2024-03-11T21:27:07.460 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_db_create": "false", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_db_path": "", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_db_size": "0", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_path": "", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_preallocate_file": "false", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_size": "107374182400", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_wal_create": "false", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_wal_path": "", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_block_wal_size": "100663296", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bluefs": "true", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-11T21:27:07.461 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_autotune": "true", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_size": "0", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-11T21:27:07.462 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_type": "2q", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_clone_cow": "true", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-11T21:27:07.463 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_mode": "none", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_csum_type": "crc32c", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_freelist": "false", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-11T21:27:07.464 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_small_allocations": "0", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_default_buffered_read": "true", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_default_buffered_write": "false", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-11T21:27:07.465 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fail_eio": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_mount": "false", 2024-03-11T21:27:07.466 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_umount": "false", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_ignore_data_csum": "false", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_log_op_age": "5.000000", 2024-03-11T21:27:07.467 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_alloc_size": "0", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_blob_size": "0", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_max_deferred_txc": "32", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_min_alloc_size": "0", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_nid_prealloc": "1024", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-11T21:27:07.468 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_retry_disk_reads": "3", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_rocksdb_cf": "true", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_spdk_mem": "512", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-11T21:27:07.469 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_tracing": "false", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-11T21:27:07.470 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_zero_block_detection": "false", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-11T21:27:07.471 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_cluster_require_signatures": "false", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_cluster_require_version": "2", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_require_signatures": "false", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_require_version": "2", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_service_require_signatures": "false", 2024-03-11T21:27:07.472 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_service_require_version": "2", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "cephx_sign_messages": "true", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "chdir": "", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_acl_type": "", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_asio_thread_count": "2", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_cache_mid": "0.750000", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_cache_size": "16384", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_caps_release_delay": "5", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_check_pool_perm": "true", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_debug_force_sync_read": "false", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_debug_getattr_caps": "false", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_debug_inject_tick_delay": "0", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-11T21:27:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: "client_die_on_failed_remount": "false", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_dirsize_rbytes": "true", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_force_lazyio": "false", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_fs": "", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_inject_release_failure": "false", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_max_inline_size": "4096", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-11T21:27:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: "client_mds_namespace": "", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_metadata": "", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_mount_gid": "-1", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_mount_timeout": "300", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_mount_uid": "-1", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_mountpoint": "/", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_notify_timeout": "10", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc": "true", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc_max_dirty": "104857600", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc_max_objects": "1000", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc_size": "209715200", 2024-03-11T21:27:07.475 INFO:teuthology.orchestra.run.smithi143.stdout: "client_oc_target_dirty": "8388608", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_permissions": "true", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_quota": "true", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_quota_df": "true", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_readahead_max_bytes": "0", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_readahead_max_periods": "4", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_readahead_min": "131072", 2024-03-11T21:27:07.476 INFO:teuthology.orchestra.run.smithi143.stdout: "client_reconnect_stale": "false", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_shutdown_timeout": "30", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_snapdir": ".snap", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_tick_interval": "1", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_trace": "", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_try_dentry_invalidate": "false", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_use_faked_inos": "false", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_use_random_mds": "false", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_graylog": "false", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-11T21:27:07.477 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_graylog_port": "12201", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_monitors": "default=true", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_syslog": "false", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "clog_to_syslog_level": "info", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "-", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_network": "", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_network_interface": "", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "compressor_zlib_isal": "false", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "compressor_zlib_level": "5", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "compressor_zlib_winsize": "-15", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "compressor_zstd_level": "1", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-11T21:27:07.478 INFO:teuthology.orchestra.run.smithi143.stdout: "crimson_alien_op_num_threads": "6", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crimson_seastar_cpu_cores": "", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_location": "", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_location_hook": "", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_location_hook_timeout": "10", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "daemonize": "false", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "daos_pool": "tank", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_alienstore": "0/5", 2024-03-11T21:27:07.479 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_allow_any_pool_priority": "false", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_asok": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_asok_assert_abort": "false", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_asserts_on_shutdown": "true", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_auth": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_bdev": "1/3", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_bluefs": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_bluestore": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_buffer": "0/1", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_ceph_exporter": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_cephfs_mirror": "0/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_cephsqlite": "0/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_client": "0/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_compressor": "1/5", 2024-03-11T21:27:07.480 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_context": "0/1", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_crimson_interrupt": "0/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_crush": "1/1", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_crypto": "1/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_cyanstore": "0/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_deliberately_leak_memory": "false", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_disable_randomized_ping": "false", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_dpdk": "1/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_eventtrace": "1/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_filer": "0/1", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_filestore": "1/3", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_finisher": "1/1", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_fuse": "1/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_heartbeat_testing_span": "0", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_heartbeatmap": "1/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-11T21:27:07.481 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_javaclient": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_journal": "1/3", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_journaler": "0/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_kstore": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_lockdep": "0/1", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mclock": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds_balancer": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds_locker": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds_log": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds_log_expire": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mds_migrator": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_memstore": "1/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mgr": "2/5", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mgrc": "20/20", 2024-03-11T21:27:07.482 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_mon": "20/20", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_monc": "0/10", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_ms": "1/1", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_none": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_objclass": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_objectcacher": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_objecter": "0/1", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_optracker": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_osd": "1/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_paxos": "20/20", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_perfcounter": "1/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_prioritycache": "1/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rados": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rbd": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rbd_mirror": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rbd_pwl": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rbd_replay": "0/5", 2024-03-11T21:27:07.483 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_refs": "0/0", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_reserver": "1/1", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_access": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_datacache": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_dbstore": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_flight": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rgw_sync": "1/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_rocksdb": "4/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore": "0/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_backref": "0/5", 2024-03-11T21:27:07.484 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_cache": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_cleaner": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_device": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_epm": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_journal": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_lba": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_odata": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_omap": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_onode": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_t": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_seastore_tm": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_striper": "0/1", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_test": "0/5", 2024-03-11T21:27:07.485 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_throttle": "1/1", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_timer": "0/1", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_tp": "0/5", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "debug_trace": "1/5", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "device_failure_prediction_mode": "none", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "err_to_graylog": "false", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "err_to_journald": "false", 2024-03-11T21:27:07.486 INFO:teuthology.orchestra.run.smithi143.stdout: "err_to_stderr": "true", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "err_to_syslog": "false", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event_tracing": "false", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_addr": "0.0.0.0", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_http_port": "9926", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_prio_limit": "5", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_sort_metrics": "true", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "exporter_stats_period": "5", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "fake_statfs_for_testing": "0", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "fatal_signal_handlers": "true", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "filer_max_purge_ops": "10", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "filer_max_truncate_ops": "128", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_apply_finisher_threads": "1", 2024-03-11T21:27:07.487 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_blackhole": "false", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_btrfs_clone_range": "true", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_btrfs_snap": "true", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_caller_concurrency": "10", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_collect_device_partition_information": "true", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_commit_timeout": "600.000000", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_debug_inject_read_err": "false", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_debug_omap_check": "false", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_debug_verify_split": "false", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_dump_file": "", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fadvise": "true", 2024-03-11T21:27:07.488 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fail_eio": "true", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fd_cache_shards": "16", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fd_cache_size": "128", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fiemap": "false", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fiemap_threshold": "4096", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_inject_stall": "0", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_journal_parallel": "false", 2024-03-11T21:27:07.489 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_journal_trailing": "false", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_journal_writeahead": "false", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_kill_at": "0", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattrs": "0", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-11T21:27:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_xattr_value_size": "0", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_merge_threshold": "-10", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_odsync_write": "false", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_omap_backend": "rocksdb", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_omap_backend_path": "", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_op_thread_timeout": "60", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_op_threads": "2", 2024-03-11T21:27:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_punch_hole": "false", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_queue_max_ops": "50", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_seek_data_hole": "false", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_sloppy_crc": "false", 2024-03-11T21:27:07.492 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_splice": "false", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_split_multiple": "2", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_split_rand_factor": "20", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_update_to": "1000", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_enable": "true", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-11T21:27:07.493 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_xfs_extsize": "false", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "filestore_zfs_snap": "false", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fio_dir": "/tmp/fio", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_allow_other": "true", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_atomic_o_trunc": "true", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_big_writes": "true", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_debug": "false", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_default_permissions": "false", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_disable_pagecache": "false", 2024-03-11T21:27:07.494 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_max_write": "0", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_multithreaded": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_require_active_mds": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_set_user_groups": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_splice_move": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_splice_read": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_splice_write": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "fuse_use_invalidate_cb": "true", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "gss_target_name": "ceph", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_file": "", 2024-03-11T21:27:07.495 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_inject_failure": "0", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_interval": "5", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "host": "smithi143", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-11T21:27:07.496 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "inject_early_sigterm": "false", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "jaeger_agent_port": "6799", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "jaeger_tracing_enable": "false", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_aio": "true", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_align_min_size": "65536", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_block_align": "true", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_block_size": "4096", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_dio": "true", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_discard": "false", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_force_aio": "false", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_ignore_corruption": "false", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_max_write_bytes": "10485760", 2024-03-11T21:27:07.497 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_max_write_entries": "100", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_replay_from": "0", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_write_header_frequency": "0", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journal_zero_on_create": "false", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journaler_prefetch_periods": "10", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journaler_prezero_periods": "5", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "journaler_write_head_interval": "15", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "key": "", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "keyfile": "", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-11T21:27:07.498 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_backend": "rocksdb", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_default_stripe_size": "65536", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_fsck_on_mount": "false", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_max_bytes": "67108864", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_max_ops": "512", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_nid_prealloc": "1024", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_onode_map_size": "1024", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_sync_submit_transaction": "false", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "kstore_sync_transaction": "false", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "librados_thread_count": "2", 2024-03-11T21:27:07.499 INFO:teuthology.orchestra.run.smithi143.stdout: "lockdep": "false", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "lockdep_force_backtrace": "false", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_coarse_timestamps": "true", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_flush_on_exit": "false", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_graylog_host": "127.0.0.1", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_graylog_port": "12201", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_max_new": "1000", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_max_recent": "10000", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_stderr_prefix": "debug ", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_stop_at_utilization": "0.970000", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_to_file": "true", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_to_graylog": "false", 2024-03-11T21:27:07.500 INFO:teuthology.orchestra.run.smithi143.stdout: "log_to_journald": "false", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "log_to_stderr": "false", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "log_to_syslog": "false", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "max_rotating_auth_attempts": "10", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_action_on_write_error": "1", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_alternate_name_max": "8192", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_asio_thread_count": "2", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_export_pin": "true", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_fragment_dirs": "true", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_fragment_interval": "5", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_interval": "10", 2024-03-11T21:27:07.501 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_max": "-1", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_max_until": "-1", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_merge_size": "50", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_midchunk": "0.300000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_min_start": "0.200000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_minchunk": "0.001000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_mode": "0", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_need_max": "1.200000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_need_min": "0.800000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_overload_epochs": "2", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-11T21:27:07.502 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_split_bits": "3", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_split_size": "10000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_target_decay": "10.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_beacon_grace": "15.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_beacon_interval": "4.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_mid": "0.700000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_release_free_interval": "10", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_reservation": "0.050000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_trim_interval": "1", 2024-03-11T21:27:07.503 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache_trim_threshold": "262144", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_client_prealloc_inos": "1000", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_connect_bootstrapping": "false", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_damage_table_max_entries": "10000", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_debug_auth_pins": "false", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_debug_frag": "false", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_debug_scatterstat": "false", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_debug_subtrees": "false", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_decay_halflife": "5.000000", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_default_dir_hash": "2", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_defer_session_stale": "true", 2024-03-11T21:27:07.504 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_deny_all_reconnect": "false", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dir_keys_per_op": "16384", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dir_max_commit_size": "10", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dir_max_entries": "0", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dir_prefetch": "true", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dump_cache_on_map": "false", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-11T21:27:07.505 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_early_reply": "true", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_enable_op_tracker": "true", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_enforce_unique_name": "true", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_export_ephemeral_random": "true", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-11T21:27:07.506 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_health_summarize_threshold": "10", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_health_dummy": "false", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_migrator_session_race": "false", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_join_fs": "", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_journal_format": "1", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_create_at": "0", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_export_at": "0", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_import_at": "0", 2024-03-11T21:27:07.507 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_journal_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_journal_expire_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_journal_replay_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_link_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_mdstable_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_openc_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_rename_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_shutdown_at": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_event_large_threshold": "524288", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_events_per_segment": "1024", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_max_events": "-1", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_max_segments": "128", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_pause": "false", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_segment_size": "0", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-11T21:27:07.508 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_trim_threshold": "128", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log_warn_factor": "2.000000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_caps_per_client": "1048576", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_completed_flushes": "100000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_completed_requests": "100000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_export_size": "20971520", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_file_recover": "32", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_purge_files": "64", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_purge_ops": "8192", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_max_snaps_per_dir": "100", 2024-03-11T21:27:07.509 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_metrics_update_interval": "2", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_min_caps_per_client": "100", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_min_caps_working_set": "10000", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_numa_node": "-1", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_complaint_time": "30.000000", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_history_duration": "600", 2024-03-11T21:27:07.510 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_history_size": "20", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_history_slow_op_size": "20", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_op_log_threshold": "5", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_ping_grace": "15", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_ping_interval": "5", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_max_caps": "30000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_recall_warning_threshold": "262144", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_replay_interval": "1.000000", 2024-03-11T21:27:07.511 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_root_ino_gid": "0", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_root_ino_uid": "0", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_shutdown_check": "0", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_skip_ino": "0", 2024-03-11T21:27:07.512 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_snap_max_uid": "4294967294", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_snap_min_uid": "0", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_snap_rstat": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_standby_replay_damaged": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_symlink_recovery": "true", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_thrash_exports": "0", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_thrash_fragments": "0", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_tick_interval": "5.000000", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_valgrind_exit": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_verify_backtrace": "1", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_verify_scatter": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_wipe_sessions": "false", 2024-03-11T21:27:07.513 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool_debug": "false", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "memstore_device_bytes": "1073741824", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "memstore_page_set": "false", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "memstore_page_size": "65536", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_client_bytes": "134217728", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_client_messages": "512", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_disabled_modules": "", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-11T21:27:07.514 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_max_pg_num_change": "128", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_mds_bytes": "134217728", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_mds_messages": "128", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_mon_bytes": "134217728", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_mon_messages": "128", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_osd_bytes": "536870912", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_osd_messages": "8192", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_pool": "true", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_standby_modules": "true", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_stats_period": "5", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_stats_threshold": "5", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_tick_period": "2", 2024-03-11T21:27:07.515 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_allow_pool_delete": "true", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_allow_pool_size_one": "false", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_auth_validate_all_caps": "true", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_bytes": "104857600", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_directed_command_retry": "2", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-11T21:27:07.516 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_hunt_parallel": "3", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_log_interval": "1.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_ping_interval": "10.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_client_target_rank": "-1", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_file_level": "debug", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_file": "true", 2024-03-11T21:27:07.517 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_journald": "false", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_stderr": "true", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_compact_on_bootstrap": "false", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_compact_on_start": "false", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_compact_on_trim": "true", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-11T21:27:07.518 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_cpu_threads": "4", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_crush_min_required_version": "hammer", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_daemon_bytes": "419430400", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_data_avail_crit": "5", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_data_avail_warn": "5", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_data_size_warn": "16106127360", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_dump_json": "false", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_dump_transactions": "false", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_extra_checks": "false", 2024-03-11T21:27:07.519 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_no_require_reef": "false", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_no_require_squid": "false", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_down_mkfs_grace": "60", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_election_default_strategy": "1", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_election_timeout": "5.000000", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_elector_ping_divisor": "2", 2024-03-11T21:27:07.520 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_enable_op_tracker": "true", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_fake_pool_delete": "false", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_force_quorum_join": "false", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_globalid_prealloc": "10000", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_detail_to_clog": "true", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_log_update_period": "5", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_max_detail": "50", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_to_clog": "true", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_to_clog_interval": "600", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_host": "[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_host_override": "", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_initial_members": "", 2024-03-11T21:27:07.521 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_lease": "5.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_log_full_interval": "50", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_log_max": "10000", 2024-03-11T21:27:07.522 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_log_max_summary": "50", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_log_epochs": "500", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_osd": "10000", 2024-03-11T21:27:07.523 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_pg_per_osd": "10000", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_pool_pg_num": "65536", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mds_force_trim_to": "0", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mds_skip_sanity": "false", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_memory_autotune": "true", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_memory_target": "2147483648", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_beacon_grace": "30", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_digest_period": "5", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_inactive_grace": "60", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_min_osdmap_epochs": "50", 2024-03-11T21:27:07.524 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_complaint_time": "30", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_history_duration": "600", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_history_size": "20", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_history_slow_op_size": "20", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_op_log_threshold": "5", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_auto_mark_in": "false", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_cache_size": "500", 2024-03-11T21:27:07.525 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_down_out_interval": "600", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_force_trim_to": "0", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-11T21:27:07.526 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_min_down_reporters": "2", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_report_timeout": "900", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-11T21:27:07.527 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_stuck_threshold": "60", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_probe_timeout": "2.000000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_reweight_max_change": "0.050000", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_reweight_max_osds": "4", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-11T21:27:07.528 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_scrub_interval": "86400", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_scrub_max_keys": "100", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_scrub_timeout": "300", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_session_timeout": "300", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_smart_report_timeout": "5", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_stat_smooth_intervals": "6", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_stretch_pool_min_size": "2", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_stretch_pool_size": "4", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_debug": "false", 2024-03-11T21:27:07.529 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_provider_kill_at": "0", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_requester_kill_at": "0", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_sync_timeout": "60.000000", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_target_pg_per_osd": "100", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_tick_interval": "5", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_timecheck_interval": "300.000000", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_use_min_delay_socket": "false", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_older_version_delay": "604800", 2024-03-11T21:27:07.530 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_misplaced": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_older_version": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_pool_no_app": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-11T21:27:07.531 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "monmap": "", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "motr_tracing_enabled": "false", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_op_threads": "3", 2024-03-11T21:27:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_cm": "false", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_device_name": "", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_dscp": "96", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_local_gid": "", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_port_num": "1", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-11T21:27:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_sl": "3", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_support_srq": "true", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_rdma_type": "ib", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_async_reap_threshold": "5", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_before_connect": "false", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_ipv4": "true", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_ipv6": "false", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_msgr1": "true", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_msgr2": "true", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_port_max": "7568", 2024-03-11T21:27:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_port_min": "6800", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_retry_count": "3", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_bind_retry_delay": "5", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_blackhole_client": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_blackhole_mds": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_blackhole_mgr": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_blackhole_mon": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_blackhole_osd": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_client_mode": "crc secure", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_cluster_mode": "crc secure", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_cluster_type": "", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_compress_secure": "false", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_connection_idle_timeout": "900", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_connection_ready_timeout": "10", 2024-03-11T21:27:07.535 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_crc_data": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_crc_header": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_die_on_bad_msg": "false", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_die_on_bug": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_die_on_old_message": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_die_on_skipped_message": "false", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_coremask": "0xF", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_enable_tso": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_hugepages": "", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-11T21:27:07.536 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_lro": "true", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_memory_channel": "4", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_pmd": "", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_port_id": "0", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_dump_on_send": "false", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_initial_backoff": "0.200000", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_delay_max": "1.000000", 2024-03-11T21:27:07.537 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_delay_type": "", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_network_congestion": "0", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_inject_socket_failures": "0", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_learn_addr_from_peer": "true", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_max_accept_failures": "4", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_max_backoff": "15.000000", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_mon_client_mode": "secure crc", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_mon_service_mode": "secure crc", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_osd_compress_min_size": "1024", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_osd_compress_mode": "none", 2024-03-11T21:27:07.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_pq_min_cost": "65536", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_public_type": "", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_service_mode": "crc secure", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_tcp_listen_backlog": "512", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_tcp_nodelay": "true", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_tcp_rcvbuf": "0", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "ms_type": "async+posix", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "no_config_file": "false", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_completion_locks_per_session": "32", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_inflight_ops": "1024", 2024-03-11T21:27:07.539 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_tick_interval": "5.000000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter_timeout": "10.000000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "objectstore_blackhole": "false", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "openssl_engine_opts": "", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_delay_time": "5.000000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_hist_halflife": "1000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_max_low_ops": "2", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_max_ops": "4", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_agent_slop": "0.020000", 2024-03-11T21:27:07.540 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_asio_thread_count": "2", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_async_recovery_min_cost": "100", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backfill_scan_max": "512", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backfill_scan_min": "64", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backoff_on_degraded": "false", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backoff_on_peering": "false", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_backoff_on_unfound": "true", 2024-03-11T21:27:07.541 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_beacon_report_interval": "300", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bench_duration": "30", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bench_max_block_size": "67108864", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_blkin_trace_all": "false", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_check_for_log_corruption": "false", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-11T21:27:07.542 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_class_update_on_start": "true", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_client_message_cap": "256", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_client_message_size_cap": "524288000", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_client_op_priority": "63", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_client_watch_timeout": "30", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_command_max_records": "256", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_compact_on_start": "false", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-11T21:27:07.543 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_crush_update_on_start": "true", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_crush_update_weight_set": "true", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_feed_pullee": "-1", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_misdirected_ops": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_no_acting_change": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_no_purge_strays": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_op_order": "false", 2024-03-11T21:27:07.544 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_shutdown": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_trim_objects": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_verify_snaps": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_keys": "1024", 2024-03-11T21:27:07.545 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_stride": "524288", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_default_notify_timeout": "30", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_delete_sleep": "0.000000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_discard_disconnected_ops": "true", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_enable_op_tracker": "true", 2024-03-11T21:27:07.546 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_fast_info": "true", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_fast_shutdown": "true", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_function_tracing": "false", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_grace": "20", 2024-03-11T21:27:07.547 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_interval": "6", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_min_peers": "10", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_min_size": "2000", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_stale": "600", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_hit_set_max_size": "100000", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_hit_set_min_size": "1000", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-11T21:27:07.548 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_journal_size": "5120", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_kill_backfill_at": "0", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_map_cache_size": "50", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_map_dedup": "true", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_map_message_max": "40", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_map_share_max_epochs": "40", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_attr_name_len": "100", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_attr_size": "0", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_backfills": "1", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_markdown_count": "5", 2024-03-11T21:27:07.549 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_markdown_period": "600", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_object_name_len": "2048", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_object_namespace_len": "256", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_object_size": "134217728", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_pg_blocked_by": "16", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_pg_log_entries": "10000", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_pgls": "1024", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_push_cost": "8388608", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_push_objects": "10", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_scrubs": "3", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-11T21:27:07.550 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_trimming_pgs": "2", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_write_op_reply_len": "64", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_max_write_size": "90", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-11T21:27:07.551 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_profile": "balanced", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-11T21:27:07.552 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_base": "805306368", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_cache_min": "134217728", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_target": "4294967296", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_target_autotune": "false", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_min_pg_log_entries": "250", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_min_recovery_priority": "0", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-11T21:27:07.553 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mon_report_interval": "5", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_num_cache_shards": "32", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_num_op_tracker_shard": "32", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_numa_auto_affinity": "true", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_numa_node": "-1", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_numa_prefer_iface": "true", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_objecter_finishers": "1", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_objectstore": "bluestore", 2024-03-11T21:27:07.554 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_objectstore_fuse": "false", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_objectstore_tracing": "false", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_complaint_time": "30.000000", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_history_duration": "600", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_history_size": "20", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_history_slow_op_size": "20", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_log_threshold": "5", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_shards": "0", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_shards_hdd": "5", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_shards_ssd": "8", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-11T21:27:07.555 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_pq_min_cost": "65536", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_queue_cut_off": "high", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_op_thread_timeout": "15", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_open_classes_on_start": "true", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_os_flags": "0", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_peering_op_priority": "255", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_delete_cost": "1048576", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_delete_priority": "5", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_log_trim_max": "10000", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_log_trim_min": "100", 2024-03-11T21:27:07.556 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_crimson": "false", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-11T21:27:07.557 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_flags": "0", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_min_size": "0", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-11T21:27:07.558 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_pg_num": "32", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_pgp_num": "0", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_read_ratio": "70", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_size": "3", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_default_type": "replicated", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_push_per_object_cost": "1000", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recover_clone_overlap": "true", 2024-03-11T21:27:07.559 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_cost": "20971520", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_active": "0", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_max_single_start": "1", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_op_priority": "3", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_priority": "5", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_sleep": "0.000000", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-11T21:27:07.560 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_repair_during_recovery": "false", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_requested_scrub_priority": "120", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_auto_repair": "false", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_begin_hour": "0", 2024-03-11T21:27:07.561 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_begin_week_day": "0", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_chunk_max": "25", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_chunk_min": "5", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_cost": "52428800", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_during_recovery": "false", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_end_hour": "0", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_end_week_day": "0", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_event_cost": "4096", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-11T21:27:07.562 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_invalid_stats": "true", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_max_preemptions": "5", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_priority": "5", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_sleep": "0.000000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_skip_data_digest": "false", 2024-03-11T21:27:07.563 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_smart_report_timeout": "5", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_cost": "1048576", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_priority": "5", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_target_transaction_size": "30", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-11T21:27:07.564 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_tracing": "false", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_use_stale_snap": "false", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "osdc_blkin_trace_all": "false", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_kill_at": "0", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_max_join_drift": "10", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_min": "500", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_min_wait": "0.050000", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_propose_interval": "1.000000", 2024-03-11T21:27:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_service_trim_max": "500", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_service_trim_min": "10", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_stash_full_interval": "25", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_trim_max": "500", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "paxos_trim_min": "250", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "perf": "true", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pid_file": "", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "-", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrv": "", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "public_bind_addr": "-", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "public_network": "", 2024-03-11T21:27:07.566 INFO:teuthology.orchestra.run.smithi143.stdout: "public_network_interface": "", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "qat_compressor_enabled": "false", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "qat_compressor_session_max_number": "256", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rados_mon_op_timeout": "0", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rados_osd_op_timeout": "0", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rados_tracing": "false", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_atime_update_interval": "60", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_balance_parent_reads": "false", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_balance_snap_reads": "false", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_blkin_trace_all": "false", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache": "true", 2024-03-11T21:27:07.567 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_policy": "writearound", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_size": "33554432", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_clone_copy_on_read": "false", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_compression_hint": "none", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_concurrent_management_ops": "10", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_clone_format": "auto", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_data_pool": "", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_features": "61", 2024-03-11T21:27:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_format": "2", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_map_options": "", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_order": "22", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_pool": "rbd", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_stripe_count": "0", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_default_stripe_unit": "0", 2024-03-11T21:27:07.569 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_enable_alloc_hint": "true", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_io_scheduler": "simple", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-11T21:27:07.570 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_order": "24", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_pool": "", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_journal_splay_width": "4", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_localize_parent_reads": "false", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_localize_snap_reads": "false", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-11T21:27:07.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-11T21:27:07.572 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-11T21:27:07.573 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_mtime_update_interval": "60", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_non_blocking_aio": "true", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_op_thread_timeout": "60", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_op_threads": "1", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_parent_cache_enabled": "false", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_plugins": "", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_bps_burst": "0", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_bps_limit": "0", 2024-03-11T21:27:07.574 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_exclude_ops": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_iops_burst": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_iops_limit": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-11T21:27:07.575 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_read_from_replica_policy": "default", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_skip_partial_discard": "true", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_tracing": "false", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_validate_names": "true", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd_validate_pool": "true", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "restapi_base_url": "", 2024-03-11T21:27:07.576 INFO:teuthology.orchestra.run.smithi143.stdout: "restapi_log_level": "", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_acl_grants_max_num": "100", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_admin_entry": "admin", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_backend_store": "rados", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_barbican_url": "", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_beast_enable_async": "true", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_counters_cache": "false", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-11T21:27:07.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_cache_enabled": "true", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_cache_expiry_interval": "900", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_cache_lru_size": "10000", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_config_store": "rados", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_content_length_compat": "false", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_copy_obj_progress": "true", 2024-03-11T21:27:07.578 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_cors_rules_max_num": "100", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_cross_domain_policy": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_addr": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_password": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_kmip_username": "", 2024-03-11T21:27:07.579 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_require_ssl": "true", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-11T21:27:07.580 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_addr": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_auth": "token", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_namespace": "", 2024-03-11T21:27:07.581 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_prefix": "", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_token_file": "", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_curl_buffersize": "524288", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_curl_low_speed_time": "300", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-11T21:27:07.582 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_d4n_port": "6379", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_log_changes_size": "1000", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_log_num_shards": "128", 2024-03-11T21:27:07.583 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_log_window": "30", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-11T21:27:07.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dns_name": "", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dns_s3website_name": "", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_dynamic_resharding": "true", 2024-03-11T21:27:07.585 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_gc_threads": "true", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_lc_threads": "true", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_ops_log": "false", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_quota_threads": "true", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_static_website": "false", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enable_usage_log": "false", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_enforce_swift_acls": "true", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_exit_timeout_secs": "120", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_expose_bucket": "false", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_extended_http_attrs": "", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_filter": "none", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_frontends": "beast port=7480", 2024-03-11T21:27:07.586 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_deferred": "50", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_objs": "32", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_gc_processor_period": "3600", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_init_timeout": "300", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-11T21:27:07.587 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_kafka_connection_idle": "30", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_domain": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_password": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_password_path": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_project": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_tenant": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_token": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_token_path": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_admin_user": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_api_version": "2", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_barbican_domain": "", 2024-03-11T21:27:07.588 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_barbican_password": "", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_barbican_project": "", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_barbican_user": "", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_url": "", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_debug_interval": "-1", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_lock_max_time": "90", 2024-03-11T21:27:07.589 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_max_objs": "32", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_max_rules": "1000", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_max_worker": "3", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lc_thread_delay": "0", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_dnattr": "uid", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_searchfilter": "", 2024-03-11T21:27:07.590 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_log_http_headers": "", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-11T21:27:07.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_log_object_name_utc": "false", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_attr_name_len": "0", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_attr_size": "0", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-11T21:27:07.592 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_chunk_size": "4194304", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_listing_results": "1000", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_notify_retries": "10", 2024-03-11T21:27:07.593 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_put_param_size": "1048576", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_put_size": "5368709120", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_max_slo_entries": "1000", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_md_log_max_shards": "64", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-11T21:27:07.594 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_mp_lock_max_time": "600", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-11T21:27:07.595 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_max_gc": "300", 2024-03-11T21:27:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-11T21:27:07.597 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_num_async_rados_threads": "32", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_num_control_oids": "8", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_numa_node": "-1", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_objexp_chunk_size": "100", 2024-03-11T21:27:07.598 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_objexp_gc_interval": "600", 2024-03-11T21:27:07.599 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-11T21:27:07.599 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-11T21:27:07.599 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-11T21:27:07.599 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_op_thread_timeout": "600", 2024-03-11T21:27:07.599 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_op_tracing": "false", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_opa_token": "", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_opa_url": "", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_opa_verify_ssl": "true", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ops_log_rados": "false", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_ops_log_socket_path": "", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-11T21:27:07.600 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_period_push_interval": "2.000000", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-11T21:27:07.601 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_cache_lanes": "3", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_cache_partitions": "3", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-11T21:27:07.602 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_print_continue": "true", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-11T21:27:07.603 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_rados_tracing": "false", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_realm": "", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_realm_id": "", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_region": "", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-11T21:27:07.604 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-11T21:27:07.605 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-11T21:27:07.605 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-11T21:27:07.605 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_request_uri": "", 2024-03-11T21:27:07.605 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_reshard_batch_size": "64", 2024-03-11T21:27:07.605 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_reshard_max_aio": "128", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_reshard_num_logs": "16", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_reshard_thread_interval": "600", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_resolve_cname": "false", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-11T21:27:07.606 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_run_sync_thread": "true", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-11T21:27:07.607 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_scheduler_type": "throttler", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_script_uri": "", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_service_provider_name": "", 2024-03-11T21:27:07.608 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_client_id": "", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_client_secret": "", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_entry": "sts", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_key": "sts", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-11T21:27:07.609 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_min_session_duration": "900", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sts_token_introspection_url": "", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_account_in_url": "false", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_auth_entry": "auth", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_auth_url": "", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_custom_header": "", 2024-03-11T21:27:07.610 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_need_stats": "true", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_tenant_name": "", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_token_expiration": "86400", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_url": "", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_url_prefix": "swift", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-11T21:27:07.611 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_lease_period": "120", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-11T21:27:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_thread_pool_size": "512", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-11T21:27:07.613 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_comment": "", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_createby": "", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_encoding": "", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_flag": "false", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-11T21:27:07.614 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_origin": "", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_torrent_tracker": "", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_trust_forwarded_https": "false", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_usage_max_shards": "32", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_usage_max_user_shards": "1", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_use_opa_authz": "false", 2024-03-11T21:27:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_counters_cache": "false", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_max_buckets": "1000", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_policies_max_num": "100", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-11T21:27:07.616 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_user_unique_email": "true", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_verify_ssl": "true", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zone": "", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zone_id": "", 2024-03-11T21:27:07.617 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zonegroup": "", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zonegroup_id": "", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rgwlc_auto_session_clear": "true", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_block_size": "4096", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_size": "536870912", 2024-03-11T21:27:07.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_index_type": "binary_search", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-11T21:27:07.619 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_partition_filters": "false", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_perf": "false", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "run_dir": "/var/run/ceph", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_block_create": "true", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_cache_lru_size": "67108864", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_default_max_object_size": "16777216", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_device_size": "53687091200", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_journal_batch_capacity": "16", 2024-03-11T21:27:07.620 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_main_device_type": "SSD", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "seastore_segment_size": "67108864", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "set_keepcaps": "false", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "setgroup": "ceph", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "setuser": "ceph", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "setuser_match_path": "", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-11T21:27:07.621 INFO:teuthology.orchestra.run.smithi143.stdout: "thp": "false", 2024-03-11T21:27:07.622 INFO:teuthology.orchestra.run.smithi143.stdout: "threadpool_default_timeout": "60", 2024-03-11T21:27:07.622 INFO:teuthology.orchestra.run.smithi143.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-11T21:27:07.622 INFO:teuthology.orchestra.run.smithi143.stdout: "throttler_perf_counter": "true" 2024-03-11T21:27:07.622 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:27:07.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.615349+0000 osd.7 (osd.7) 113 : cluster 0 2.0 scrub starts 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.617810+0000 osd.7 (osd.7) 114 : cluster 0 2.0 scrub ok 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.765895+0000 mon.a (mon.0) 883 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.790587+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[29284]: cluster 2024-03-11T21:27:06.797038+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.615349+0000 osd.7 (osd.7) 113 : cluster 0 2.0 scrub starts 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.617810+0000 osd.7 (osd.7) 114 : cluster 0 2.0 scrub ok 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.765895+0000 mon.a (mon.0) 883 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:27:07.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.790587+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T21:27:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:07 smithi143 bash[21512]: cluster 2024-03-11T21:27:06.797038+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T21:27:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:07 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.615349+0000 osd.7 (osd.7) 113 : cluster 0 2.0 scrub starts 2024-03-11T21:27:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:07 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.617810+0000 osd.7 (osd.7) 114 : cluster 0 2.0 scrub ok 2024-03-11T21:27:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:07 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.765895+0000 mon.a (mon.0) 883 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:27:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:07 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.790587+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T21:27:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:07 smithi203 bash[23654]: cluster 2024-03-11T21:27:06.797038+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T21:27:08.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-11T21:27:08.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:09.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.132103+0000 mgr.y (mgr.14550) 153 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.635514+0000 osd.7 (osd.7) 115 : cluster 0 2.5 scrub starts 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.637901+0000 osd.7 (osd.7) 116 : cluster 0 2.5 scrub ok 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.658129+0000 osd.0 (osd.0) 93 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.660929+0000 osd.0 (osd.0) 94 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.814338+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T21:27:09.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:08 smithi203 bash[23654]: cluster 2024-03-11T21:27:07.816747+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.132103+0000 mgr.y (mgr.14550) 153 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.635514+0000 osd.7 (osd.7) 115 : cluster 0 2.5 scrub starts 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.637901+0000 osd.7 (osd.7) 116 : cluster 0 2.5 scrub ok 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.658129+0000 osd.0 (osd.0) 93 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.660929+0000 osd.0 (osd.0) 94 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.814338+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[21512]: cluster 2024-03-11T21:27:07.816747+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T21:27:09.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.132103+0000 mgr.y (mgr.14550) 153 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.635514+0000 osd.7 (osd.7) 115 : cluster 0 2.5 scrub starts 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.637901+0000 osd.7 (osd.7) 116 : cluster 0 2.5 scrub ok 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.658129+0000 osd.0 (osd.0) 93 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.660929+0000 osd.0 (osd.0) 94 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.814338+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T21:27:09.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:08 smithi143 bash[29284]: cluster 2024-03-11T21:27:07.816747+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: audit 2024-03-11T21:27:08.230936+0000 mgr.y (mgr.14550) 154 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.628605+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.630140+0000 osd.7 (osd.7) 117 : cluster 0 2.1c scrub starts 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.630976+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.632371+0000 osd.7 (osd.7) 118 : cluster 0 2.1c scrub ok 2024-03-11T21:27:10.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.789200+0000 osd.5 (osd.5) 109 : cluster 0 6.d scrub starts 2024-03-11T21:27:10.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.791876+0000 osd.5 (osd.5) 110 : cluster 0 6.d scrub ok 2024-03-11T21:27:10.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.852974+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-03-11T21:27:10.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:08.855887+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-03-11T21:27:10.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:09.019716+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:27:10.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:09 smithi203 bash[23654]: cluster 2024-03-11T21:27:09.022570+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: audit 2024-03-11T21:27:08.230936+0000 mgr.y (mgr.14550) 154 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.628605+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.630140+0000 osd.7 (osd.7) 117 : cluster 0 2.1c scrub starts 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.630976+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.632371+0000 osd.7 (osd.7) 118 : cluster 0 2.1c scrub ok 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.789200+0000 osd.5 (osd.5) 109 : cluster 0 6.d scrub starts 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.791876+0000 osd.5 (osd.5) 110 : cluster 0 6.d scrub ok 2024-03-11T21:27:10.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.852974+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:08.855887+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:09.019716+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[21512]: cluster 2024-03-11T21:27:09.022570+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: audit 2024-03-11T21:27:08.230936+0000 mgr.y (mgr.14550) 154 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.628605+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.630140+0000 osd.7 (osd.7) 117 : cluster 0 2.1c scrub starts 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.630976+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.632371+0000 osd.7 (osd.7) 118 : cluster 0 2.1c scrub ok 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.789200+0000 osd.5 (osd.5) 109 : cluster 0 6.d scrub starts 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.791876+0000 osd.5 (osd.5) 110 : cluster 0 6.d scrub ok 2024-03-11T21:27:10.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.852974+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-03-11T21:27:10.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:08.855887+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-03-11T21:27:10.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:09.019716+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:27:10.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:09 smithi143 bash[29284]: cluster 2024-03-11T21:27:09.022570+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:27:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:10 smithi203 bash[23654]: cluster 2024-03-11T21:27:09.133594+0000 mgr.y (mgr.14550) 155 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:27:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:10 smithi203 bash[23654]: cluster 2024-03-11T21:27:10.024779+0000 osd.4 (osd.4) 85 : cluster 0 6.1e scrub starts 2024-03-11T21:27:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:10 smithi203 bash[23654]: cluster 2024-03-11T21:27:10.026729+0000 osd.4 (osd.4) 86 : cluster 0 6.1e scrub ok 2024-03-11T21:27:11.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[29284]: cluster 2024-03-11T21:27:09.133594+0000 mgr.y (mgr.14550) 155 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:27:11.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[29284]: cluster 2024-03-11T21:27:10.024779+0000 osd.4 (osd.4) 85 : cluster 0 6.1e scrub starts 2024-03-11T21:27:11.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[29284]: cluster 2024-03-11T21:27:10.026729+0000 osd.4 (osd.4) 86 : cluster 0 6.1e scrub ok 2024-03-11T21:27:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[21512]: cluster 2024-03-11T21:27:09.133594+0000 mgr.y (mgr.14550) 155 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:27:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[21512]: cluster 2024-03-11T21:27:10.024779+0000 osd.4 (osd.4) 85 : cluster 0 6.1e scrub starts 2024-03-11T21:27:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:10 smithi143 bash[21512]: cluster 2024-03-11T21:27:10.026729+0000 osd.4 (osd.4) 86 : cluster 0 6.1e scrub ok 2024-03-11T21:27:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:11 smithi203 bash[23654]: cluster 2024-03-11T21:27:11.014390+0000 osd.4 (osd.4) 87 : cluster 0 5.18 scrub starts 2024-03-11T21:27:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:11 smithi203 bash[23654]: cluster 2024-03-11T21:27:11.016812+0000 osd.4 (osd.4) 88 : cluster 0 5.18 scrub ok 2024-03-11T21:27:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:11 smithi143 bash[21512]: cluster 2024-03-11T21:27:11.014390+0000 osd.4 (osd.4) 87 : cluster 0 5.18 scrub starts 2024-03-11T21:27:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:11 smithi143 bash[21512]: cluster 2024-03-11T21:27:11.016812+0000 osd.4 (osd.4) 88 : cluster 0 5.18 scrub ok 2024-03-11T21:27:12.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:11 smithi143 bash[29284]: cluster 2024-03-11T21:27:11.014390+0000 osd.4 (osd.4) 87 : cluster 0 5.18 scrub starts 2024-03-11T21:27:12.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:11 smithi143 bash[29284]: cluster 2024-03-11T21:27:11.016812+0000 osd.4 (osd.4) 88 : cluster 0 5.18 scrub ok 2024-03-11T21:27:12.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:12 smithi203 bash[23654]: cluster 2024-03-11T21:27:11.134153+0000 mgr.y (mgr.14550) 156 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:12 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.158392+0000 osd.3 (osd.3) 121 : cluster 0 3.8 scrub starts 2024-03-11T21:27:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:12 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.163084+0000 osd.3 (osd.3) 122 : cluster 0 3.8 scrub ok 2024-03-11T21:27:13.115 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:13.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[21512]: cluster 2024-03-11T21:27:11.134153+0000 mgr.y (mgr.14550) 156 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:13.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.158392+0000 osd.3 (osd.3) 121 : cluster 0 3.8 scrub starts 2024-03-11T21:27:13.136 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.163084+0000 osd.3 (osd.3) 122 : cluster 0 3.8 scrub ok 2024-03-11T21:27:13.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[29284]: cluster 2024-03-11T21:27:11.134153+0000 mgr.y (mgr.14550) 156 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:13.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.158392+0000 osd.3 (osd.3) 121 : cluster 0 3.8 scrub starts 2024-03-11T21:27:13.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:12 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.163084+0000 osd.3 (osd.3) 122 : cluster 0 3.8 scrub ok 2024-03-11T21:27:14.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.660258+0000 osd.0 (osd.0) 97 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.664349+0000 osd.0 (osd.0) 98 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.753249+0000 osd.2 (osd.2) 79 : cluster 0 5.6 scrub starts 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.755913+0000 osd.2 (osd.2) 80 : cluster 0 5.6 scrub ok 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: audit 2024-03-11T21:27:12.805954+0000 mon.c (mon.2) 87 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.888630+0000 osd.6 (osd.6) 67 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[29284]: cluster 2024-03-11T21:27:12.893769+0000 osd.6 (osd.6) 68 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.660258+0000 osd.0 (osd.0) 97 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.664349+0000 osd.0 (osd.0) 98 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.753249+0000 osd.2 (osd.2) 79 : cluster 0 5.6 scrub starts 2024-03-11T21:27:14.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.755913+0000 osd.2 (osd.2) 80 : cluster 0 5.6 scrub ok 2024-03-11T21:27:14.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: audit 2024-03-11T21:27:12.805954+0000 mon.c (mon.2) 87 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:14.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.888630+0000 osd.6 (osd.6) 67 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:27:14.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:13 smithi143 bash[21512]: cluster 2024-03-11T21:27:12.893769+0000 osd.6 (osd.6) 68 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:27:14.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.660258+0000 osd.0 (osd.0) 97 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:27:14.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.664349+0000 osd.0 (osd.0) 98 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:27:14.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.753249+0000 osd.2 (osd.2) 79 : cluster 0 5.6 scrub starts 2024-03-11T21:27:14.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.755913+0000 osd.2 (osd.2) 80 : cluster 0 5.6 scrub ok 2024-03-11T21:27:14.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: audit 2024-03-11T21:27:12.805954+0000 mon.c (mon.2) 87 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:14.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.888630+0000 osd.6 (osd.6) 67 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:27:14.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:13 smithi203 bash[23654]: cluster 2024-03-11T21:27:12.893769+0000 osd.6 (osd.6) 68 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:27:14.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[21512]: cluster 2024-03-11T21:27:13.135541+0000 mgr.y (mgr.14550) 157 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:14.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[21512]: cluster 2024-03-11T21:27:13.603324+0000 osd.7 (osd.7) 119 : cluster 0 6.15 scrub starts 2024-03-11T21:27:14.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[21512]: cluster 2024-03-11T21:27:13.605752+0000 osd.7 (osd.7) 120 : cluster 0 6.15 scrub ok 2024-03-11T21:27:14.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[29284]: cluster 2024-03-11T21:27:13.135541+0000 mgr.y (mgr.14550) 157 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:14.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[29284]: cluster 2024-03-11T21:27:13.603324+0000 osd.7 (osd.7) 119 : cluster 0 6.15 scrub starts 2024-03-11T21:27:14.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:14 smithi143 bash[29284]: cluster 2024-03-11T21:27:13.605752+0000 osd.7 (osd.7) 120 : cluster 0 6.15 scrub ok 2024-03-11T21:27:15.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:14 smithi203 bash[23654]: cluster 2024-03-11T21:27:13.135541+0000 mgr.y (mgr.14550) 157 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:15.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:14 smithi203 bash[23654]: cluster 2024-03-11T21:27:13.603324+0000 osd.7 (osd.7) 119 : cluster 0 6.15 scrub starts 2024-03-11T21:27:15.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:14 smithi203 bash[23654]: cluster 2024-03-11T21:27:13.605752+0000 osd.7 (osd.7) 120 : cluster 0 6.15 scrub ok 2024-03-11T21:27:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:15 smithi143 bash[21512]: cluster 2024-03-11T21:27:14.794424+0000 osd.5 (osd.5) 111 : cluster 0 6.7 scrub starts 2024-03-11T21:27:16.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:15 smithi143 bash[21512]: cluster 2024-03-11T21:27:14.796592+0000 osd.5 (osd.5) 112 : cluster 0 6.7 scrub ok 2024-03-11T21:27:16.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:15 smithi143 bash[29284]: cluster 2024-03-11T21:27:14.794424+0000 osd.5 (osd.5) 111 : cluster 0 6.7 scrub starts 2024-03-11T21:27:16.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:15 smithi143 bash[29284]: cluster 2024-03-11T21:27:14.796592+0000 osd.5 (osd.5) 112 : cluster 0 6.7 scrub ok 2024-03-11T21:27:16.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:15 smithi203 bash[23654]: cluster 2024-03-11T21:27:14.794424+0000 osd.5 (osd.5) 111 : cluster 0 6.7 scrub starts 2024-03-11T21:27:16.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:15 smithi203 bash[23654]: cluster 2024-03-11T21:27:14.796592+0000 osd.5 (osd.5) 112 : cluster 0 6.7 scrub ok 2024-03-11T21:27:16.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/145249325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8107e70 msgr2=0x7f7fc8108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- 172.21.15.143:0/145249325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8107e70 0x7f7fc8108250 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40098f0 tx=0x7f7fc4009c00 comp rx=0 tx=0).stop 2024-03-11T21:27:16.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/145249325 shutdown_connections 2024-03-11T21:27:16.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- 172.21.15.143:0/145249325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8107e70 0x7f7fc8108250 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- 172.21.15.143:0/145249325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc806a4d0 0x7f7fc810ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- 172.21.15.143:0/145249325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc8069b10 0x7f7fc8069f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/145249325 >> 172.21.15.143:0/145249325 conn(0x7f7fc8076d80 msgr2=0x7f7fc8077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/145249325 shutdown_connections 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/145249325 wait complete. 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 librados: starting msgr at 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 librados: starting objecter 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 Processor -- start 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- start start 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 librados: setting wanted keys 2024-03-11T21:27:16.427 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 librados: calling monclient init 2024-03-11T21:27:16.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc8069b10 0x7f7fc8100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.428 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 0x7f7fc8101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8107e70 0x7f7fc8101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8111610 con 0x7f7fc806a4d0 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8111490 con 0x7f7fc8069b10 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8111790 con 0x7f7fc8107e70 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8107e70 0x7f7fc8101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcc80d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 0x7f7fc8101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcc80d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 0x7f7fc8101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46160/0 (socket says 172.21.15.143:46160) 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 learned_addr learned my addr 172.21.15.143:0/461254146 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd00e700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc8069b10 0x7f7fc8100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc8069b10 msgr2=0x7f7fc8100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc8069b10 0x7f7fc8100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 msgr2=0x7f7fc8101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 0x7f7fc8101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc81a6180 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc806a4d0 0x7f7fc8101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcd80f700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8107e70 0x7f7fc8101890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009cd0 tx=0x7f7fc000c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fbe7fc700 10 client.?.objecter ms_handle_connect 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fbe7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc000cdc0 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc81a6410 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7fc0003a00 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.448+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc00044f0 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 10 client.24632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc81a6900 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 1 librados: init done 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fc804faa0 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7fc001a020 con 0x7f7fc8107e70 2024-03-11T21:27:16.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7fb406f820 0x7f7fb4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(124..124 src has 66..124) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f7fc008db80 con 0x7f7fc8107e70 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 10 client.24632.objecter ms_dispatch 0x7f7fc8001550 osd_map(124..124 src has 66..124) v4 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 3 client.24632.objecter handle_osd_map got epochs [124,124] > 0 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 3 client.24632.objecter handle_osd_map decoding full epoch 124 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcd00e700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7fb406f820 0x7f7fb4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 20 client.24632.objecter dump_active .. 0 homeless 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f7fc0010040 con 0x7f7fc8107e70 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 0x7f7fc8108f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcd00e700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7fb406f820 0x7f7fb4071ce0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009fd0 tx=0x7f7fc4019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fb406f820 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81094d0 con 0x7f7fc8108bb0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 0x7f7fc8108f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fc8109680 con 0x7f7fc8108bb0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 0x7f7fc8108f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb800d620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc8108bb0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb801d070 con 0x7f7fc8108bb0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.0 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f7fb8018640 con 0x7f7fc8108bb0 2024-03-11T21:27:16.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 msgr2=0x7f7fc8108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 0x7f7fc8108f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb800d620 comp rx=0 tx=0).stop 2024-03-11T21:27:16.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 msgr2=0x7f7fc8108f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.452+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8108bb0 0x7f7fc8108f90 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb800d620 comp rx=0 tx=0).stop 2024-03-11T21:27:16.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 0x7f7fc81aab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81ab0c0 con 0x7f7fc8109890 2024-03-11T21:27:16.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 0x7f7fc81aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f7fc81ab2c0 con 0x7f7fc8109890 2024-03-11T21:27:16.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 0x7f7fc81aab80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.460+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc8109890 2024-03-11T21:27:16.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb801d070 con 0x7f7fc8109890 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.0 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f7fb801e040 con 0x7f7fc8109890 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 msgr2=0x7f7fc81aab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 0x7f7fc81aab80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb8004750 comp rx=0 tx=0).stop 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 msgr2=0x7f7fc81aab80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc8109890 0x7f7fc81aab80 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81028a0 tx=0x7f7fb8004750 comp rx=0 tx=0).stop 2024-03-11T21:27:16.441 INFO:teuthology.orchestra.run.smithi143.stdout:mon.a: {} 2024-03-11T21:27:16.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 0x7f7fc802cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc802d150 con 0x7f7fc802c800 2024-03-11T21:27:16.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 0x7f7fc802cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fc802d350 con 0x7f7fc802c800 2024-03-11T21:27:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 0x7f7fc802cc10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000c00 tx=0x7f7fb800a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc802c800 2024-03-11T21:27:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.464+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8036070 con 0x7f7fc802c800 2024-03-11T21:27:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f7fb8005730 con 0x7f7fc802c800 2024-03-11T21:27:16.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 msgr2=0x7f7fc802cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 0x7f7fc802cc10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000c00 tx=0x7f7fb800a400 comp rx=0 tx=0).stop 2024-03-11T21:27:16.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 msgr2=0x7f7fc802cc10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802c800 0x7f7fc802cc10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:16.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.468+0000 7f7fcd80f700 1 -- 172.21.15.143:0/461254146 reap_dead start 2024-03-11T21:27:16.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.472+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 0x7f7fc8100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.472+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc802dac0 con 0x7f7fc802d6e0 2024-03-11T21:27:16.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 0x7f7fc8100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f7fc802dc40 con 0x7f7fc802d6e0 2024-03-11T21:27:16.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 0x7f7fc8100e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d660 tx=0x7f7fb8027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc802d6e0 2024-03-11T21:27:16.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.476+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb801f480 con 0x7f7fc802d6e0 2024-03-11T21:27:16.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f7fb803b9b0 con 0x7f7fc802d6e0 2024-03-11T21:27:16.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 msgr2=0x7f7fc8100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 0x7f7fc8100e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d660 tx=0x7f7fb8027000 comp rx=0 tx=0).stop 2024-03-11T21:27:16.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 msgr2=0x7f7fc8100e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fc802d6e0 0x7f7fc8100e10 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d660 tx=0x7f7fb8027000 comp rx=0 tx=0).stop 2024-03-11T21:27:16.457 INFO:teuthology.orchestra.run.smithi143.stdout:mon.b: {} 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 0x7f7fc81ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81b0370 con 0x7f7fc8101470 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 0x7f7fc81ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fc81b04f0 con 0x7f7fc8101470 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc8101470 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 0x7f7fc81ad510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81a6ad0 tx=0x7f7fb800a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8052070 con 0x7f7fc8101470 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f7fb8032930 con 0x7f7fc8101470 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 msgr2=0x7f7fc81ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 0x7f7fc81ad510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81a6ad0 tx=0x7f7fb800a400 comp rx=0 tx=0).stop 2024-03-11T21:27:16.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 msgr2=0x7f7fc81ad510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.480+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8101470 0x7f7fc81ad510 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fc81a6ad0 tx=0x7f7fb800a400 comp rx=0 tx=0).stop 2024-03-11T21:27:16.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 0x7f7fc81ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:16.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81af730 con 0x7f7fc81b06a0 2024-03-11T21:27:16.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 0x7f7fc81ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:16.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fcc80d700 1 -- 172.21.15.143:0/461254146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f7fc81af930 con 0x7f7fc81b06a0 2024-03-11T21:27:16.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fbe7fc700 10 client.24632.objecter ms_handle_connect 0x7f7fc81b06a0 2024-03-11T21:27:16.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fbe7fc700 10 client.24632.objecter resend_mon_ops 2024-03-11T21:27:16.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fcc80d700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 0x7f7fc81ada50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800e010 tx=0x7f7fb8043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:16.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.488+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8036050 con 0x7f7fc81b06a0 2024-03-11T21:27:16.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 <== mon.2 v2:172.21.15.143:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f7fb8054830 con 0x7f7fc81b06a0 2024-03-11T21:27:16.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 msgr2=0x7f7fc81ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 0x7f7fc81ada50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800e010 tx=0x7f7fb8043000 comp rx=0 tx=0).stop 2024-03-11T21:27:16.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fbe7fc700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 msgr2=0x7f7fc81ada50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:27:16.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fbe7fc700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc81b06a0 0x7f7fc81ada50 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800e010 tx=0x7f7fb8043000 comp rx=0 tx=0).stop 2024-03-11T21:27:16.468 INFO:teuthology.orchestra.run.smithi143.stdout:mon.c: {} 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 10 librados: watch_flush enter 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 10 librados: watch_flush exit 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 20 client.24632.objecter shutdown clearing up homeless session... 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 10 client.24632.objecter successfully canceled tick 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7fb406f820 msgr2=0x7f7fb4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7fb406f820 0x7f7fb4071ce0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009fd0 tx=0x7f7fc4019040 comp rx=0 tx=0).stop 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8107e70 msgr2=0x7f7fc8101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 --2- 172.21.15.143:0/461254146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fc8107e70 0x7f7fc8101890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009cd0 tx=0x7f7fc000c620 comp rx=0 tx=0).stop 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcd80f700 1 -- 172.21.15.143:0/461254146 reap_dead start 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 shutdown_connections 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 >> 172.21.15.143:0/461254146 conn(0x7f7fc8076d80 msgr2=0x7f7fc80ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 shutdown_connections 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 -- 172.21.15.143:0/461254146 wait complete. 2024-03-11T21:27:16.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:16.492+0000 7f7fcf272700 1 librados: shutdown 2024-03-11T21:27:16.470 INFO:teuthology.orchestra.run.smithi143.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:16.470 INFO:teuthology.orchestra.run.smithi143.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:16.470 INFO:teuthology.orchestra.run.smithi143.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:17.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:16 smithi143 bash[21512]: cluster 2024-03-11T21:27:15.136244+0000 mgr.y (mgr.14550) 158 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:17.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:16 smithi143 bash[21512]: audit 2024-03-11T21:27:16.457807+0000 mon.c (mon.2) 88 : audit 0 from='client.? 172.21.15.143:0/461254146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:17.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:16 smithi143 bash[29284]: cluster 2024-03-11T21:27:15.136244+0000 mgr.y (mgr.14550) 158 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:17.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:16 smithi143 bash[29284]: audit 2024-03-11T21:27:16.457807+0000 mon.c (mon.2) 88 : audit 0 from='client.? 172.21.15.143:0/461254146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:17.270 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T21:27:17.284 INFO:tasks.workunit:Pulling workunits from ref 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-11T21:27:17.284 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T21:27:17.284 DEBUG:teuthology.orchestra.run.smithi143:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T21:27:17.285 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-11T21:27:17.285 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:27:17.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:27:17.285 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:27:17.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:27:17.286 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-11T21:27:17.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:17.286 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-11T21:27:17.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:27:17.287 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-11T21:27:17.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:27:17.287 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-11T21:27:17.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:27:17.289 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:16 smithi203 bash[23654]: cluster 2024-03-11T21:27:15.136244+0000 mgr.y (mgr.14550) 158 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:17.289 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:16 smithi203 bash[23654]: audit 2024-03-11T21:27:16.457807+0000 mon.c (mon.2) 88 : audit 0 from='client.? 172.21.15.143:0/461254146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.297 INFO:teuthology.orchestra.run.smithi143.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-11T21:27:17.297 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-11T21:27:17.317 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:17.322 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-11T21:27:17.322 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-11T21:27:17.368 INFO:tasks.workunit:timeout=3h 2024-03-11T21:27:17.369 INFO:tasks.workunit:cleanup=True 2024-03-11T21:27:17.369 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-11T21:27:17.417 INFO:tasks.workunit.client.0.smithi143.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T21:27:17.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:17.448 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:17.551 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:17.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:17.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:17.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:17.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:17.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[21512]: cluster 2024-03-11T21:27:16.223727+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:17.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[21512]: cluster 2024-03-11T21:27:16.223867+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:17.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[21512]: cluster 2024-03-11T21:27:16.821762+0000 mon.a (mon.0) 884 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:27:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:17.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:17.965 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:17.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:18.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:18.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:17 smithi203 bash[23654]: cluster 2024-03-11T21:27:16.223727+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:18.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:17 smithi203 bash[23654]: cluster 2024-03-11T21:27:16.223867+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:18.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:17 smithi203 bash[23654]: cluster 2024-03-11T21:27:16.821762+0000 mon.a (mon.0) 884 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:27:18.095 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:18.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[29284]: cluster 2024-03-11T21:27:16.223727+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:18.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[29284]: cluster 2024-03-11T21:27:16.223867+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:18.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:17 smithi143 bash[29284]: cluster 2024-03-11T21:27:16.821762+0000 mon.a (mon.0) 884 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:27:18.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:18.330 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:18.358 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:18.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:18.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:18.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:18.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:18.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:18.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:18.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:19.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:19.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[23654]: cluster 2024-03-11T21:27:17.136885+0000 mgr.y (mgr.14550) 159 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[23654]: cluster 2024-03-11T21:27:17.622034+0000 osd.0 (osd.0) 99 : cluster 0 6.0 scrub starts 2024-03-11T21:27:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[23654]: cluster 2024-03-11T21:27:17.623627+0000 osd.0 (osd.0) 100 : cluster 0 6.0 scrub ok 2024-03-11T21:27:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[23654]: cluster 2024-03-11T21:27:17.707898+0000 osd.7 (osd.7) 123 : cluster 0 6.5 scrub starts 2024-03-11T21:27:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:18 smithi203 bash[23654]: cluster 2024-03-11T21:27:17.710421+0000 osd.7 (osd.7) 124 : cluster 0 6.5 scrub ok 2024-03-11T21:27:19.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[21512]: cluster 2024-03-11T21:27:17.136885+0000 mgr.y (mgr.14550) 159 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[21512]: cluster 2024-03-11T21:27:17.622034+0000 osd.0 (osd.0) 99 : cluster 0 6.0 scrub starts 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[21512]: cluster 2024-03-11T21:27:17.623627+0000 osd.0 (osd.0) 100 : cluster 0 6.0 scrub ok 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[21512]: cluster 2024-03-11T21:27:17.707898+0000 osd.7 (osd.7) 123 : cluster 0 6.5 scrub starts 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[21512]: cluster 2024-03-11T21:27:17.710421+0000 osd.7 (osd.7) 124 : cluster 0 6.5 scrub ok 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[29284]: cluster 2024-03-11T21:27:17.136885+0000 mgr.y (mgr.14550) 159 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[29284]: cluster 2024-03-11T21:27:17.622034+0000 osd.0 (osd.0) 99 : cluster 0 6.0 scrub starts 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[29284]: cluster 2024-03-11T21:27:17.623627+0000 osd.0 (osd.0) 100 : cluster 0 6.0 scrub ok 2024-03-11T21:27:19.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[29284]: cluster 2024-03-11T21:27:17.707898+0000 osd.7 (osd.7) 123 : cluster 0 6.5 scrub starts 2024-03-11T21:27:19.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:18 smithi143 bash[29284]: cluster 2024-03-11T21:27:17.710421+0000 osd.7 (osd.7) 124 : cluster 0 6.5 scrub ok 2024-03-11T21:27:19.138 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:19.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:19.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:19.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:19.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.530 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:19.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.660 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:19.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:19.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:19.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:19.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:19.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:19 smithi203 bash[23654]: audit 2024-03-11T21:27:18.242433+0000 mgr.y (mgr.14550) 160 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:19 smithi203 bash[23654]: cluster 2024-03-11T21:27:18.706859+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-03-11T21:27:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:19 smithi203 bash[23654]: cluster 2024-03-11T21:27:18.708446+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-03-11T21:27:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:19 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.228528+0000 osd.3 (osd.3) 123 : cluster 0 5.13 scrub starts 2024-03-11T21:27:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:19 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.229910+0000 osd.3 (osd.3) 124 : cluster 0 5.13 scrub ok 2024-03-11T21:27:20.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:20.059 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:20.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:20.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[29284]: audit 2024-03-11T21:27:18.242433+0000 mgr.y (mgr.14550) 160 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:20.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[29284]: cluster 2024-03-11T21:27:18.706859+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[29284]: cluster 2024-03-11T21:27:18.708446+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.228528+0000 osd.3 (osd.3) 123 : cluster 0 5.13 scrub starts 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.229910+0000 osd.3 (osd.3) 124 : cluster 0 5.13 scrub ok 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[21512]: audit 2024-03-11T21:27:18.242433+0000 mgr.y (mgr.14550) 160 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[21512]: cluster 2024-03-11T21:27:18.706859+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[21512]: cluster 2024-03-11T21:27:18.708446+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.228528+0000 osd.3 (osd.3) 123 : cluster 0 5.13 scrub starts 2024-03-11T21:27:20.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:19 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.229910+0000 osd.3 (osd.3) 124 : cluster 0 5.13 scrub ok 2024-03-11T21:27:20.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:20.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:20.323 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:20.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:20.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:20.585 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:20.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:20.715 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:20.817 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:20.848 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:20.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.138330+0000 mgr.y (mgr.14550) 161 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.697059+0000 osd.7 (osd.7) 127 : cluster 0 3.e deep-scrub starts 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.700656+0000 osd.7 (osd.7) 128 : cluster 0 3.e deep-scrub ok 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.911522+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.913022+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.921760+0000 osd.5 (osd.5) 113 : cluster 0 5.7 scrub starts 2024-03-11T21:27:20.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:19.924161+0000 osd.5 (osd.5) 114 : cluster 0 5.7 scrub ok 2024-03-11T21:27:20.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.198013+0000 osd.3 (osd.3) 125 : cluster 0 5.17 scrub starts 2024-03-11T21:27:20.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:20 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.200186+0000 osd.3 (osd.3) 126 : cluster 0 5.17 scrub ok 2024-03-11T21:27:20.978 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:20.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:21.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.138330+0000 mgr.y (mgr.14550) 161 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.697059+0000 osd.7 (osd.7) 127 : cluster 0 3.e deep-scrub starts 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.700656+0000 osd.7 (osd.7) 128 : cluster 0 3.e deep-scrub ok 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.911522+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.913022+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.921760+0000 osd.5 (osd.5) 113 : cluster 0 5.7 scrub starts 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:19.924161+0000 osd.5 (osd.5) 114 : cluster 0 5.7 scrub ok 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.198013+0000 osd.3 (osd.3) 125 : cluster 0 5.17 scrub starts 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.200186+0000 osd.3 (osd.3) 126 : cluster 0 5.17 scrub ok 2024-03-11T21:27:21.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.138330+0000 mgr.y (mgr.14550) 161 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.697059+0000 osd.7 (osd.7) 127 : cluster 0 3.e deep-scrub starts 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.700656+0000 osd.7 (osd.7) 128 : cluster 0 3.e deep-scrub ok 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.911522+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.913022+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.921760+0000 osd.5 (osd.5) 113 : cluster 0 5.7 scrub starts 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:19.924161+0000 osd.5 (osd.5) 114 : cluster 0 5.7 scrub ok 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.198013+0000 osd.3 (osd.3) 125 : cluster 0 5.17 scrub starts 2024-03-11T21:27:21.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:20 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.200186+0000 osd.3 (osd.3) 126 : cluster 0 5.17 scrub ok 2024-03-11T21:27:21.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:21.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:21.239 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:21.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:21.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:21.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:21.496 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:21.599 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:21.624 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:21.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:21.753 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:21.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:21.886 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:21.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:21.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.658099+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.660033+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.908490+0000 osd.5 (osd.5) 115 : cluster 0 3.c scrub starts 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:20.912364+0000 osd.5 (osd.5) 116 : cluster 0 3.c scrub ok 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.115775+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.120775+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.658099+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.660033+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.908490+0000 osd.5 (osd.5) 115 : cluster 0 3.c scrub starts 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:20.912364+0000 osd.5 (osd.5) 116 : cluster 0 3.c scrub ok 2024-03-11T21:27:21.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.115775+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-03-11T21:27:21.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:21 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.120775+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-03-11T21:27:21.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:22.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.658099+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-03-11T21:27:22.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.660033+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-03-11T21:27:22.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.908490+0000 osd.5 (osd.5) 115 : cluster 0 3.c scrub starts 2024-03-11T21:27:22.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:20.912364+0000 osd.5 (osd.5) 116 : cluster 0 3.c scrub ok 2024-03-11T21:27:22.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.115775+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-03-11T21:27:22.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:21 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.120775+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-03-11T21:27:22.016 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:22.084 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:22.084 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:22.085 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:22.085 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:22.105 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:22.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:22.147 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:22.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:22.279 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:22.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:22.408 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:22.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:22.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:22.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:22.668 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:22.772 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:22.799 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:22.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:22.930 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:22.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.138886+0000 mgr.y (mgr.14550) 162 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.741098+0000 osd.2 (osd.2) 81 : cluster 0 5.8 scrub starts 2024-03-11T21:27:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.742353+0000 osd.2 (osd.2) 82 : cluster 0 5.8 scrub ok 2024-03-11T21:27:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.838172+0000 osd.4 (osd.4) 91 : cluster 0 5.e scrub starts 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[21512]: cluster 2024-03-11T21:27:21.839966+0000 osd.4 (osd.4) 92 : cluster 0 5.e scrub ok 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.138886+0000 mgr.y (mgr.14550) 162 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.741098+0000 osd.2 (osd.2) 81 : cluster 0 5.8 scrub starts 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.742353+0000 osd.2 (osd.2) 82 : cluster 0 5.8 scrub ok 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.838172+0000 osd.4 (osd.4) 91 : cluster 0 5.e scrub starts 2024-03-11T21:27:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:22 smithi143 bash[29284]: cluster 2024-03-11T21:27:21.839966+0000 osd.4 (osd.4) 92 : cluster 0 5.e scrub ok 2024-03-11T21:27:23.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.061 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:23.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:22 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.138886+0000 mgr.y (mgr.14550) 162 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:23.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:22 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.741098+0000 osd.2 (osd.2) 81 : cluster 0 5.8 scrub starts 2024-03-11T21:27:23.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:22 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.742353+0000 osd.2 (osd.2) 82 : cluster 0 5.8 scrub ok 2024-03-11T21:27:23.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:22 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.838172+0000 osd.4 (osd.4) 91 : cluster 0 5.e scrub starts 2024-03-11T21:27:23.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:22 smithi203 bash[23654]: cluster 2024-03-11T21:27:21.839966+0000 osd.4 (osd.4) 92 : cluster 0 5.e scrub ok 2024-03-11T21:27:23.191 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:23.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:23.320 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:23.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:23.450 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:23.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:23.580 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:23.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:23.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:23.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.842 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:23.945 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:23 smithi203 bash[23654]: cluster 2024-03-11T21:27:22.538857+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T21:27:23.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:23 smithi203 bash[23654]: cluster 2024-03-11T21:27:22.541104+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T21:27:23.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:23 smithi203 bash[23654]: cluster 2024-03-11T21:27:22.790012+0000 osd.4 (osd.4) 93 : cluster 0 6.1a scrub starts 2024-03-11T21:27:23.957 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:23 smithi203 bash[23654]: cluster 2024-03-11T21:27:22.792350+0000 osd.4 (osd.4) 94 : cluster 0 6.1a scrub ok 2024-03-11T21:27:23.970 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:23.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:24.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[29284]: cluster 2024-03-11T21:27:22.538857+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T21:27:24.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[29284]: cluster 2024-03-11T21:27:22.541104+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T21:27:24.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[29284]: cluster 2024-03-11T21:27:22.790012+0000 osd.4 (osd.4) 93 : cluster 0 6.1a scrub starts 2024-03-11T21:27:24.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[29284]: cluster 2024-03-11T21:27:22.792350+0000 osd.4 (osd.4) 94 : cluster 0 6.1a scrub ok 2024-03-11T21:27:24.068 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[21512]: cluster 2024-03-11T21:27:22.538857+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T21:27:24.068 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[21512]: cluster 2024-03-11T21:27:22.541104+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T21:27:24.068 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[21512]: cluster 2024-03-11T21:27:22.790012+0000 osd.4 (osd.4) 93 : cluster 0 6.1a scrub starts 2024-03-11T21:27:24.068 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:23 smithi143 bash[21512]: cluster 2024-03-11T21:27:22.792350+0000 osd.4 (osd.4) 94 : cluster 0 6.1a scrub ok 2024-03-11T21:27:24.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:24.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:24.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:24.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:24.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:24.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:24.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:24.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:24.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.743 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:24.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:24.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[21512]: cluster 2024-03-11T21:27:23.140360+0000 mgr.y (mgr.14550) 163 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:24.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[21512]: cluster 2024-03-11T21:27:23.770972+0000 osd.4 (osd.4) 95 : cluster 0 6.2 scrub starts 2024-03-11T21:27:24.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[21512]: cluster 2024-03-11T21:27:23.772569+0000 osd.4 (osd.4) 96 : cluster 0 6.2 scrub ok 2024-03-11T21:27:24.875 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:24.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:24.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:24.989 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:24 smithi203 bash[23654]: cluster 2024-03-11T21:27:23.140360+0000 mgr.y (mgr.14550) 163 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:24.989 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:24 smithi203 bash[23654]: cluster 2024-03-11T21:27:23.770972+0000 osd.4 (osd.4) 95 : cluster 0 6.2 scrub starts 2024-03-11T21:27:24.989 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:24 smithi203 bash[23654]: cluster 2024-03-11T21:27:23.772569+0000 osd.4 (osd.4) 96 : cluster 0 6.2 scrub ok 2024-03-11T21:27:25.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:25.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:25.136 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:25.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[29284]: cluster 2024-03-11T21:27:23.140360+0000 mgr.y (mgr.14550) 163 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:25.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[29284]: cluster 2024-03-11T21:27:23.770972+0000 osd.4 (osd.4) 95 : cluster 0 6.2 scrub starts 2024-03-11T21:27:25.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:24 smithi143 bash[29284]: cluster 2024-03-11T21:27:23.772569+0000 osd.4 (osd.4) 96 : cluster 0 6.2 scrub ok 2024-03-11T21:27:25.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:25.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:25.392 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:25.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:25.523 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:25.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:25.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:25.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:25.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:25.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:25.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.004725+0000 osd.5 (osd.5) 117 : cluster 0 5.15 scrub starts 2024-03-11T21:27:25.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.007416+0000 osd.5 (osd.5) 118 : cluster 0 5.15 scrub ok 2024-03-11T21:27:25.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.261084+0000 osd.3 (osd.3) 127 : cluster 0 3.2 scrub starts 2024-03-11T21:27:25.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.263670+0000 osd.3 (osd.3) 128 : cluster 0 3.2 scrub ok 2024-03-11T21:27:25.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.004725+0000 osd.5 (osd.5) 117 : cluster 0 5.15 scrub starts 2024-03-11T21:27:25.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.007416+0000 osd.5 (osd.5) 118 : cluster 0 5.15 scrub ok 2024-03-11T21:27:25.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.261084+0000 osd.3 (osd.3) 127 : cluster 0 3.2 scrub starts 2024-03-11T21:27:25.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:25 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.263670+0000 osd.3 (osd.3) 128 : cluster 0 3.2 scrub ok 2024-03-11T21:27:25.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:25.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:26.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:26.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:26.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:25 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.004725+0000 osd.5 (osd.5) 117 : cluster 0 5.15 scrub starts 2024-03-11T21:27:26.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:25 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.007416+0000 osd.5 (osd.5) 118 : cluster 0 5.15 scrub ok 2024-03-11T21:27:26.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:25 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.261084+0000 osd.3 (osd.3) 127 : cluster 0 3.2 scrub starts 2024-03-11T21:27:26.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:25 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.263670+0000 osd.3 (osd.3) 128 : cluster 0 3.2 scrub ok 2024-03-11T21:27:26.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:26.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.300 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:26.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:26.428 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:26.532 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:26.569 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:26.672 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:26.801 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:26.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:26.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:26.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:27.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:27.086 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:27.102 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:26 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.141008+0000 mgr.y (mgr.14550) 164 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:27.103 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:26 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.631600+0000 osd.7 (osd.7) 131 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:27:27.103 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:26 smithi203 bash[23654]: cluster 2024-03-11T21:27:25.633903+0000 osd.7 (osd.7) 132 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:27:27.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:27.299 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.141008+0000 mgr.y (mgr.14550) 164 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:27.300 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.631600+0000 osd.7 (osd.7) 131 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:27:27.300 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[21512]: cluster 2024-03-11T21:27:25.633903+0000 osd.7 (osd.7) 132 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:27:27.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.141008+0000 mgr.y (mgr.14550) 164 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:27.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.631600+0000 osd.7 (osd.7) 131 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:27:27.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:26 smithi143 bash[29284]: cluster 2024-03-11T21:27:25.633903+0000 osd.7 (osd.7) 132 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:27:27.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:27.333 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:27.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:27.464 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:27.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:27.595 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:27.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:27.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:27.857 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:27.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:27.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:27 smithi203 bash[23654]: cluster 2024-03-11T21:27:26.596040+0000 osd.7 (osd.7) 133 : cluster 0 6.8 scrub starts 2024-03-11T21:27:27.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:27 smithi203 bash[23654]: cluster 2024-03-11T21:27:26.598045+0000 osd.7 (osd.7) 134 : cluster 0 6.8 scrub ok 2024-03-11T21:27:27.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:27 smithi203 bash[23654]: audit 2024-03-11T21:27:27.837120+0000 mon.c (mon.2) 89 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:27.985 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:27.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:28.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:28.114 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[29284]: cluster 2024-03-11T21:27:26.596040+0000 osd.7 (osd.7) 133 : cluster 0 6.8 scrub starts 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[29284]: cluster 2024-03-11T21:27:26.598045+0000 osd.7 (osd.7) 134 : cluster 0 6.8 scrub ok 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[29284]: audit 2024-03-11T21:27:27.837120+0000 mon.c (mon.2) 89 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[21512]: cluster 2024-03-11T21:27:26.596040+0000 osd.7 (osd.7) 133 : cluster 0 6.8 scrub starts 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[21512]: cluster 2024-03-11T21:27:26.598045+0000 osd.7 (osd.7) 134 : cluster 0 6.8 scrub ok 2024-03-11T21:27:28.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:27 smithi143 bash[21512]: audit 2024-03-11T21:27:27.837120+0000 mon.c (mon.2) 89 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:28.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:28.243 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:28.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:28.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:28.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:28.507 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:28.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:28.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:28.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:28.767 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:28.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:28.898 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:29.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:29.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:27.141447+0000 mgr.y (mgr.14550) 165 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:27.823766+0000 osd.4 (osd.4) 97 : cluster 0 6.e scrub starts 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:27.825836+0000 osd.4 (osd.4) 98 : cluster 0 6.e scrub ok 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:27.979429+0000 osd.0 (osd.0) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:27.979543+0000 osd.0 (osd.0) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:28.068781+0000 osd.1 (osd.1) 103 : cluster 0 3.19 scrub starts 2024-03-11T21:27:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:28 smithi203 bash[23654]: cluster 2024-03-11T21:27:28.074632+0000 osd.1 (osd.1) 104 : cluster 0 3.19 scrub ok 2024-03-11T21:27:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:29.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:29.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:29.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:27.141447+0000 mgr.y (mgr.14550) 165 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:27:29.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:27.823766+0000 osd.4 (osd.4) 97 : cluster 0 6.e scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:27.825836+0000 osd.4 (osd.4) 98 : cluster 0 6.e scrub ok 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:27.979429+0000 osd.0 (osd.0) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:27.979543+0000 osd.0 (osd.0) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:28.068781+0000 osd.1 (osd.1) 103 : cluster 0 3.19 scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[21512]: cluster 2024-03-11T21:27:28.074632+0000 osd.1 (osd.1) 104 : cluster 0 3.19 scrub ok 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:27.141447+0000 mgr.y (mgr.14550) 165 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:27.823766+0000 osd.4 (osd.4) 97 : cluster 0 6.e scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:27.825836+0000 osd.4 (osd.4) 98 : cluster 0 6.e scrub ok 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:27.979429+0000 osd.0 (osd.0) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:27.979543+0000 osd.0 (osd.0) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:28.068781+0000 osd.1 (osd.1) 103 : cluster 0 3.19 scrub starts 2024-03-11T21:27:29.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:28 smithi143 bash[29284]: cluster 2024-03-11T21:27:28.074632+0000 osd.1 (osd.1) 104 : cluster 0 3.19 scrub ok 2024-03-11T21:27:29.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:29.284 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:29.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:29.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:29.544 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:29.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.673 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:29.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:29.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:29.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:29 smithi143 bash[21512]: audit 2024-03-11T21:27:28.250887+0000 mgr.y (mgr.14550) 166 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:29.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:29 smithi143 bash[21512]: cluster 2024-03-11T21:27:28.896828+0000 mon.a (mon.0) 885 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:27:29.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:29 smithi143 bash[29284]: audit 2024-03-11T21:27:28.250887+0000 mgr.y (mgr.14550) 166 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:29.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:29 smithi143 bash[29284]: cluster 2024-03-11T21:27:28.896828+0000 mon.a (mon.0) 885 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:27:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:30.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:30.059 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:30.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:30.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:29 smithi203 bash[23654]: audit 2024-03-11T21:27:28.250887+0000 mgr.y (mgr.14550) 166 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:30.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:29 smithi203 bash[23654]: cluster 2024-03-11T21:27:28.896828+0000 mon.a (mon.0) 885 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:27:30.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:30.320 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:30.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:30.452 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:30.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:30.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:30.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:30.712 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:30.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:30.843 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:30.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:30.975 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:31.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:30 smithi203 bash[23654]: cluster 2024-03-11T21:27:29.142056+0000 mgr.y (mgr.14550) 167 : cluster 0 pgmap v138: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:30 smithi203 bash[23654]: cluster 2024-03-11T21:27:29.964111+0000 osd.5 (osd.5) 119 : cluster 0 2.18 scrub starts 2024-03-11T21:27:31.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:30 smithi203 bash[23654]: cluster 2024-03-11T21:27:29.967190+0000 osd.5 (osd.5) 120 : cluster 0 2.18 scrub ok 2024-03-11T21:27:31.106 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:31.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[21512]: cluster 2024-03-11T21:27:29.142056+0000 mgr.y (mgr.14550) 167 : cluster 0 pgmap v138: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[21512]: cluster 2024-03-11T21:27:29.964111+0000 osd.5 (osd.5) 119 : cluster 0 2.18 scrub starts 2024-03-11T21:27:31.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[21512]: cluster 2024-03-11T21:27:29.967190+0000 osd.5 (osd.5) 120 : cluster 0 2.18 scrub ok 2024-03-11T21:27:31.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[29284]: cluster 2024-03-11T21:27:29.142056+0000 mgr.y (mgr.14550) 167 : cluster 0 pgmap v138: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[29284]: cluster 2024-03-11T21:27:29.964111+0000 osd.5 (osd.5) 119 : cluster 0 2.18 scrub starts 2024-03-11T21:27:31.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:30 smithi143 bash[29284]: cluster 2024-03-11T21:27:29.967190+0000 osd.5 (osd.5) 120 : cluster 0 2.18 scrub ok 2024-03-11T21:27:31.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:31.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:31.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:31.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:31.497 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:31.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:31.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:31.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:31.759 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:31.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:31.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 -- 172.21.15.143:0/404843556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c10a600 msgr2=0x7ff04c110f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 --2- 172.21.15.143:0/404843556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c10a600 0x7ff04c110f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff04400b0a0 tx=0x7ff04400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 -- 172.21.15.143:0/404843556 shutdown_connections 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 --2- 172.21.15.143:0/404843556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c10a600 0x7ff04c110f80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 --2- 172.21.15.143:0/404843556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff04c1009e0 0x7ff04c100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 --2- 172.21.15.143:0/404843556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 0x7ff04c109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 -- 172.21.15.143:0/404843556 >> 172.21.15.143:0/404843556 conn(0x7ff04c076040 msgr2=0x7ff04c076450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 -- 172.21.15.143:0/404843556 shutdown_connections 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 -- 172.21.15.143:0/404843556 wait complete. 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 librados: starting msgr at 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.884+0000 7ff0518bd700 1 librados: starting objecter 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 Processor -- start 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- start start 2024-03-11T21:27:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 librados: setting wanted keys 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 librados: calling monclient init 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 0x7ff04c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff04c10a600 0x7ff04c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c113890 con 0x7ff04c1009e0 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c113710 con 0x7ff04c1013b0 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c113a10 con 0x7ff04c10a600 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff04c10a600 0x7ff04c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43598/0 (socket says 172.21.15.143:43598) 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 -- 172.21.15.143:0/2225503984 learned_addr learned my addr 172.21.15.143:0/2225503984 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 0x7ff04c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff04c10a600 msgr2=0x7ff04c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff04c10a600 0x7ff04c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 msgr2=0x7ff04c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 0x7ff04c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c19ed80 con 0x7ff04c1009e0 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff04c1013b0 0x7ff04c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff03400ea50 tx=0x7ff03400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 10 client.?.objecter ms_handle_connect 0x7ff04c1009e0 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03400cbd0 con 0x7ff04c1009e0 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff03400cd70 con 0x7ff04c1009e0 2024-03-11T21:27:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034018680 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c19f070 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 10 client.14769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c1a68f0 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 librados: init done 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff04c06a010 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.888+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff034010430 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 0x7ff038071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(126..126 src has 66..126) v4 ==== 7450+0+0 (secure 0 0 0) 0x7ff03408e5d0 con 0x7ff04c1009e0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 osd_map(126..126 src has 66..126) v4 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 3 client.14769.objecter handle_osd_map got epochs [126,126] > 0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 3 client.14769.objecter handle_osd_map decoding full epoch 126 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 0x7ff038071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 20 client.14769.objecter dump_active .. 0 homeless 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v126) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff034094170 con 0x7ff04c1009e0 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 10 client.14769.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 1 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff04c0629f0 0x7ff04c062df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 1 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 0 1 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 4 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 10 client.14769.objecter _send_command 1 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c19b0d0 con 0x7ff04c0629f0 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff04c0629f0 0x7ff04c062df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff03806f820 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 0x7ff038071ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0768b0 tx=0x7ff03c005db0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c0629f0 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff04c0629f0 0x7ff04c062df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c19b0d0 con 0x7ff04c0629f0 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 10 client.14769.objecter _finish_command 1 = system:0 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 4 2024-03-11T21:27:31.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.892+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 0 1 2024-03-11T21:27:31.880 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:27:31.880 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:27:31.880 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:27:31.880 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:27:31.884 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 10 client.14769.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 2 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 2 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c04faa0 osd=0 3 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 0 2 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 4 2024-03-11T21:27:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 10 client.14769.objecter _send_command 2 2024-03-11T21:27:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.916+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c062470 con 0x7ff04c0629f0 2024-03-11T21:27:31.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c062470 con 0x7ff04c0629f0 2024-03-11T21:27:31.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:31.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 10 client.14769.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:31.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 4 2024-03-11T21:27:31.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 0 2 2024-03-11T21:27:31.897 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:27:31.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 10 client.14769.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 3 2024-03-11T21:27:31.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7ff04c10bbc0 0x7ff04c1aab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 1 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 4 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 10 client.14769.objecter _send_command 3 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c02d9a0 con 0x7ff04c10bbc0 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7ff04c10bbc0 0x7ff04c1aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7ff04c10bbc0 0x7ff04c1aab40 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c10bbc0 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c02d9a0 con 0x7ff04c10bbc0 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:27:31.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 10 client.14769.objecter _finish_command 3 = system:0 2024-03-11T21:27:31.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 4 2024-03-11T21:27:31.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.920+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 1 3 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 10 client.14769.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 4 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 4 2024-03-11T21:27:31.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c02d830 osd=1 3 2024-03-11T21:27:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 1 4 2024-03-11T21:27:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 4 2024-03-11T21:27:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 10 client.14769.objecter _send_command 4 2024-03-11T21:27:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1ab080 con 0x7ff04c10bbc0 2024-03-11T21:27:31.917 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:27:31.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1ab080 con 0x7ff04c10bbc0 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 10 client.14769.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 4 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 1 4 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 10 client.14769.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 5 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff04c10b2a0 0x7ff04c1ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 5 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 2 5 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 4 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 10 client.14769.objecter _send_command 5 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c02d470 con 0x7ff04c10b2a0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff04c10b2a0 0x7ff04c1ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff04c10b2a0 0x7ff04c1ae420 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c10b2a0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c02d470 con 0x7ff04c10b2a0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 10 client.14769.objecter _finish_command 5 = system:0 2024-03-11T21:27:31.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 4 2024-03-11T21:27:31.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.940+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 2 5 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.011867+0000 osd.6 (osd.6) 69 : cluster 0 2.1a scrub starts 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.013008+0000 osd.6 (osd.6) 70 : cluster 0 2.1a scrub ok 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.504140+0000 osd.0 (osd.0) 105 : cluster 0 6.9 scrub starts 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.506449+0000 osd.0 (osd.0) 106 : cluster 0 6.9 scrub ok 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[21512]: audit 2024-03-11T21:27:31.896128+0000 mon.a (mon.0) 886 : audit 0 from='client.? 172.21.15.143:0/2225503984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.011867+0000 osd.6 (osd.6) 69 : cluster 0 2.1a scrub starts 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.013008+0000 osd.6 (osd.6) 70 : cluster 0 2.1a scrub ok 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.504140+0000 osd.0 (osd.0) 105 : cluster 0 6.9 scrub starts 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.506449+0000 osd.0 (osd.0) 106 : cluster 0 6.9 scrub ok 2024-03-11T21:27:31.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:31 smithi143 bash[29284]: audit 2024-03-11T21:27:31.896128+0000 mon.a (mon.0) 886 : audit 0 from='client.? 172.21.15.143:0/2225503984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:31.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 10 client.14769.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:31.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 6 2024-03-11T21:27:31.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 6 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c10b130 osd=2 3 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 2 6 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 4 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 10 client.14769.objecter _send_command 6 2024-03-11T21:27:31.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c02c900 con 0x7ff04c10b2a0 2024-03-11T21:27:31.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c02c900 con 0x7ff04c10b2a0 2024-03-11T21:27:31.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:31.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 10 client.14769.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:31.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 4 2024-03-11T21:27:31.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 2 6 2024-03-11T21:27:31.938 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:27:31.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 10 client.14769.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 7 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff04c1abf10 0x7ff04c1b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 7 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 3 7 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 4 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 10 client.14769.objecter _send_command 7 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c1b4a80 con 0x7ff04c1abf10 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff04c1abf10 0x7ff04c1b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c1abf10 2024-03-11T21:27:31.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff04c1abf10 0x7ff04c1b1af0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c1b4a80 con 0x7ff04c1abf10 2024-03-11T21:27:31.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:27:31.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 10 client.14769.objecter _finish_command 7 = system:0 2024-03-11T21:27:31.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 4 2024-03-11T21:27:31.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.960+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 3 7 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 10 client.14769.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 8 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 8 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c02cd00 osd=3 3 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 3 8 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 4 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 10 client.14769.objecter _send_command 8 2024-03-11T21:27:31.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.980+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1ab480 con 0x7ff04c1abf10 2024-03-11T21:27:31.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1ab480 con 0x7ff04c1abf10 2024-03-11T21:27:31.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:31.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 10 client.14769.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 4 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 3 8 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 10 client.14769.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 9 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff04c1af640 0x7ff04c1b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 9 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 4 9 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 4 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 10 client.14769.objecter _send_command 9 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c1ab3a0 con 0x7ff04c1af640 2024-03-11T21:27:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff04c1af640 0x7ff04c1b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c1af640 2024-03-11T21:27:31.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.984+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff04c1af640 0x7ff04c1b3250 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.988+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c1ab3a0 con 0x7ff04c1af640 2024-03-11T21:27:31.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.988+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:27:31.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.988+0000 7ff0508bb700 10 client.14769.objecter _finish_command 9 = system:0 2024-03-11T21:27:31.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.988+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 4 2024-03-11T21:27:31.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:31.988+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 4 9 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 10 client.14769.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 10 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 10 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1ab230 osd=4 3 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 4 10 2024-03-11T21:27:31.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 4 2024-03-11T21:27:31.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 10 client.14769.objecter _send_command 10 2024-03-11T21:27:31.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.004+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1b3790 con 0x7ff04c1af640 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1b3790 con 0x7ff04c1af640 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0508bb700 10 client.14769.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 4 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 4 10 2024-03-11T21:27:31.986 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 10 client.14769.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 11 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff04c1b3eb0 0x7ff04c1b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 11 2024-03-11T21:27:31.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 5 11 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 4 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 10 client.14769.objecter _send_command 11 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c1bb600 con 0x7ff04c1b3eb0 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.008+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff04c1b3eb0 0x7ff04c1b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c1b3eb0 2024-03-11T21:27:31.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff043fff700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff04c1b3eb0 0x7ff04c1b4310 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:31.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c1bb600 con 0x7ff04c1b3eb0 2024-03-11T21:27:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:27:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 10 client.14769.objecter _finish_command 11 = system:0 2024-03-11T21:27:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 4 2024-03-11T21:27:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.012+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 5 11 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 10 client.14769.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 12 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 12 2024-03-11T21:27:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1b3790 osd=5 3 2024-03-11T21:27:32.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 5 12 2024-03-11T21:27:32.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 4 2024-03-11T21:27:32.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 10 client.14769.objecter _send_command 12 2024-03-11T21:27:32.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.028+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1b8ce0 con 0x7ff04c1b3eb0 2024-03-11T21:27:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1b8ce0 con 0x7ff04c1b3eb0 2024-03-11T21:27:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 10 client.14769.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 4 2024-03-11T21:27:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 5 12 2024-03-11T21:27:32.011 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:27:32.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 10 client.14769.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:32.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 13 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff04c1b9360 0x7ff04c1bb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 13 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 6 13 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 4 2024-03-11T21:27:32.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 10 client.14769.objecter _send_command 13 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c1be660 con 0x7ff04c1b9360 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff04c1b9360 0x7ff04c1bb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c1b9360 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.032+0000 7ff04b7fe700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff04c1b9360 0x7ff04c1bb750 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.036+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c1be660 con 0x7ff04c1b9360 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.036+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.036+0000 7ff0508bb700 10 client.14769.objecter _finish_command 13 = system:0 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.036+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 4 2024-03-11T21:27:32.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.036+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 6 13 2024-03-11T21:27:32.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:32.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 10 client.14769.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:32.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 14 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 14 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1b8ce0 osd=6 3 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 6 14 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 4 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 10 client.14769.objecter _send_command 14 2024-03-11T21:27:32.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.052+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1bbc90 con 0x7ff04c1b9360 2024-03-11T21:27:32.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1bbc90 con 0x7ff04c1b9360 2024-03-11T21:27:32.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:32.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 10 client.14769.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:32.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 4 2024-03-11T21:27:32.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 6 14 2024-03-11T21:27:32.033 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:27:32.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 10 client.14769.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:27:32.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 15 2024-03-11T21:27:32.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff04c1bc3b0 0x7ff04c1bc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 15 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 7 15 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 4 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 10 client.14769.objecter _send_command 15 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff04c1c1a30 con 0x7ff04c1bc3b0 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff04c1bc3b0 0x7ff04c1bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff04affd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff04c1bc3b0 0x7ff04c1bc810 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 10 client.14769.objecter ms_handle_connect 0x7ff04c1bc3b0 2024-03-11T21:27:32.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff04c1c1a30 con 0x7ff04c1bc3b0 2024-03-11T21:27:32.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:27:32.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 10 client.14769.objecter _finish_command 15 = system:0 2024-03-11T21:27:32.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 4 2024-03-11T21:27:32.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.056+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 7 15 2024-03-11T21:27:32.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:27:32.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign -1 16 2024-03-11T21:27:32.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter _get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_remove -1 16 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter get_session s=0x7ff04c1bbc90 osd=7 3 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 15 client.14769.objecter _session_command_op_assign 7 16 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 4 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter _send_command 16 2024-03-11T21:27:32.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff04c1c08b0 con 0x7ff04c1bc3b0 2024-03-11T21:27:32.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0508bb700 1 -- 172.21.15.143:0/2225503984 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff04c1c08b0 con 0x7ff04c1bc3b0 2024-03-11T21:27:32.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0508bb700 10 client.14769.objecter ms_dispatch 0x7ff04c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:27:32.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0508bb700 10 client.14769.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:27:32.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0508bb700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 4 2024-03-11T21:27:32.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0508bb700 15 client.14769.objecter _session_command_op_remove 7 16 2024-03-11T21:27:32.054 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:27:32.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 librados: watch_flush enter 2024-03-11T21:27:32.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 librados: watch_flush exit 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.0 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff04c0629f0 msgr2=0x7ff04c062df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff04c0629f0 0x7ff04c062df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c04faa0 osd=0 1 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.1 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7ff04c10bbc0 msgr2=0x7ff04c1aab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7ff04c10bbc0 0x7ff04c1aab40 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02d830 osd=1 1 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.2 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff04c10b2a0 msgr2=0x7ff04c1ae420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff04c10b2a0 0x7ff04c1ae420 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c10b130 osd=2 1 2024-03-11T21:27:32.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.3 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff04c1abf10 msgr2=0x7ff04c1b1af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff04c1abf10 0x7ff04c1b1af0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff04b7fe700 1 -- 172.21.15.143:0/2225503984 reap_dead start 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c02cd00 osd=3 1 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.4 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff04c1af640 msgr2=0x7ff04c1b3250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff04c1af640 0x7ff04c1b3250 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1ab230 osd=4 1 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.5 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff04c1b3eb0 msgr2=0x7ff04c1b4310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff04c1b3eb0 0x7ff04c1b4310 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b3790 osd=5 1 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.6 2024-03-11T21:27:32.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff04c1b9360 msgr2=0x7ff04c1bb750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff04c1b9360 0x7ff04c1bb750 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1b8ce0 osd=6 1 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter close_session for osd.7 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff04c1bc3b0 msgr2=0x7ff04c1bc810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff04c1bc3b0 0x7ff04c1bc810 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter put_session s=0x7ff04c1bbc90 osd=7 1 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 20 client.14769.objecter shutdown clearing up homeless session... 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 10 client.14769.objecter successfully canceled tick 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff04b7fe700 1 -- 172.21.15.143:0/2225503984 reap_dead start 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 msgr2=0x7ff038071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 0x7ff038071ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0768b0 tx=0x7ff03c005db0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 msgr2=0x7ff04c199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff03400ea50 tx=0x7ff03400ed60 comp rx=0 tx=0).stop 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 shutdown_connections 2024-03-11T21:27:32.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff04c1009e0 0x7ff04c199d70 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff03400ea50 tx=0x7ff03400ed60 comp rx=0 tx=0).stop 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 --2- 172.21.15.143:0/2225503984 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff03806f820 0x7ff038071ce0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 >> 172.21.15.143:0/2225503984 conn(0x7ff04c076040 msgr2=0x7ff04c0ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 shutdown_connections 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 -- 172.21.15.143:0/2225503984 wait complete. 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.076+0000 7ff0518bd700 1 librados: shutdown 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:27:32.059 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:27:32.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:32.126 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:32.138 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:32.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:32.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:32.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:31 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.011867+0000 osd.6 (osd.6) 69 : cluster 0 2.1a scrub starts 2024-03-11T21:27:32.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:31 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.013008+0000 osd.6 (osd.6) 70 : cluster 0 2.1a scrub ok 2024-03-11T21:27:32.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:31 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.504140+0000 osd.0 (osd.0) 105 : cluster 0 6.9 scrub starts 2024-03-11T21:27:32.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:31 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.506449+0000 osd.0 (osd.0) 106 : cluster 0 6.9 scrub ok 2024-03-11T21:27:32.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:31 smithi203 bash[23654]: audit 2024-03-11T21:27:31.896128+0000 mon.a (mon.0) 886 : audit 0 from='client.? 172.21.15.143:0/2225503984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:32.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- 172.21.15.143:0/2655566490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640100d40 msgr2=0x7f36401011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 --2- 172.21.15.143:0/2655566490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640100d40 0x7f36401011c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3634009950 tx=0x7f3634009c60 comp rx=0 tx=0).stop 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- 172.21.15.143:0/2655566490 shutdown_connections 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 --2- 172.21.15.143:0/2655566490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f36401071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 --2- 172.21.15.143:0/2655566490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640101700 0x7f364010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 --2- 172.21.15.143:0/2655566490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640100d40 0x7f36401011c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- 172.21.15.143:0/2655566490 >> 172.21.15.143:0/2655566490 conn(0x7f3640076d80 msgr2=0x7f3640077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- 172.21.15.143:0/2655566490 shutdown_connections 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- 172.21.15.143:0/2655566490 wait complete. 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 librados: starting msgr at 2024-03-11T21:27:32.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 librados: starting objecter 2024-03-11T21:27:32.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 Processor -- start 2024-03-11T21:27:32.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 -- start start 2024-03-11T21:27:32.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 librados: setting wanted keys 2024-03-11T21:27:32.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.412+0000 7f3646a44700 1 librados: calling monclient init 2024-03-11T21:27:32.392 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640100d40 0x7f3640199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f364019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364010f460 con 0x7f3640106dc0 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364010f2e0 con 0x7f3640100d40 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364010f5e0 con 0x7f3640101700 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:54124/0 (socket says 172.21.15.143:54124) 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 -- 172.21.15.143:0/2190363571 learned_addr learned my addr 172.21.15.143:0/2190363571 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3644fe1700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f364019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363ffff700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640100d40 0x7f3640199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 -- 172.21.15.143:0/2190363571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640100d40 msgr2=0x7f3640199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640100d40 0x7f3640199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 -- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 msgr2=0x7f364019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f364019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 -- 172.21.15.143:0/2190363571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364019ee20 con 0x7f3640101700 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3644fe1700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f364019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363f7fe700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3634000c00 tx=0x7f3634005430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 10 client.?.objecter ms_handle_connect 0x7f3640101700 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36340056e0 con 0x7f3640101700 2024-03-11T21:27:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364019f0b0 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3634004c10 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634016c00 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 10 client.24650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36401a6990 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 librados: init done 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3640102400 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3634027020 con 0x7f3640101700 2024-03-11T21:27:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 0x7f3628071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(126..126 src has 66..126) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f363408f3b0 con 0x7f3640101700 2024-03-11T21:27:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 10 client.24650.objecter ms_dispatch 0x7f3640001550 osd_map(126..126 src has 66..126) v4 2024-03-11T21:27:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 3 client.24650.objecter handle_osd_map got epochs [126,126] > 0 2024-03-11T21:27:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 3 client.24650.objecter handle_osd_map decoding full epoch 126 2024-03-11T21:27:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363ffff700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 0x7f3628071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 20 client.24650.objecter dump_active .. 0 homeless 2024-03-11T21:27:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363d7fa700 10 client.24650.objecter ms_handle_connect 0x7f362806f820 2024-03-11T21:27:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.416+0000 7f363ffff700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 0x7f3628071ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3630009750 tx=0x7f3630009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:32.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.424+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3634004790 con 0x7f3640101700 2024-03-11T21:27:32.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:32.521 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:32.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.644+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f364019b170 con 0x7f3640101700 2024-03-11T21:27:32.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:32.645 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:32.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:32.766 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 -- 172.21.15.143:0/2206486520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3406a6a0 msgr2=0x7f2f3406aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 --2- 172.21.15.143:0/2206486520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3406a6a0 0x7f2f3406aa80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098f0 tx=0x7f2f24009c00 comp rx=0 tx=0).stop 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 -- 172.21.15.143:0/2206486520 shutdown_connections 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 --2- 172.21.15.143:0/2206486520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3406a6a0 0x7f2f3406aa80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 --2- 172.21.15.143:0/2206486520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f34107220 0x7f2f3410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 --2- 172.21.15.143:0/2206486520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406b050 0x7f2f34106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.812+0000 7f2f392fa700 1 -- 172.21.15.143:0/2206486520 >> 172.21.15.143:0/2206486520 conn(0x7f2f34077de0 msgr2=0x7f2f340781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- 172.21.15.143:0/2206486520 shutdown_connections 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- 172.21.15.143:0/2206486520 wait complete. 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 librados: starting msgr at 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 librados: starting objecter 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 Processor -- start 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- start start 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 librados: setting wanted keys 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 librados: calling monclient init 2024-03-11T21:27:32.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406a6a0 0x7f2f34199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 0x7f2f3419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34111630 con 0x7f2f34107220 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f341114b0 con 0x7f2f3406a6a0 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f341117b0 con 0x7f2f3406b050 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 0x7f2f3419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43642/0 (socket says 172.21.15.143:43642) 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f337fe700 1 -- 172.21.15.143:0/1281578867 learned_addr learned my addr 172.21.15.143:0/1281578867 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f32ffd700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406a6a0 0x7f2f34199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 -- 172.21.15.143:0/1281578867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406a6a0 msgr2=0x7f2f34199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406a6a0 0x7f2f34199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 -- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 msgr2=0x7f2f3419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 -- 172.21.15.143:0/1281578867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3419eef0 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f337fe700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f327fc700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 0x7f2f3419a420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f3406b6e0 tx=0x7f2f2800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 10 client.?.objecter ms_handle_connect 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28016070 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2f2800fd10 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3419f1e0 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28010e50 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 10 client.24656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f341a6a60 con 0x7f2f3406b050 2024-03-11T21:27:32.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 librados: init done 2024-03-11T21:27:32.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3404faa0 con 0x7f2f3406b050 2024-03-11T21:27:32.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2f28019070 con 0x7f2f3406b050 2024-03-11T21:27:32.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 0x7f2f1c071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(126..126 src has 66..126) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f2f2808dbc0 con 0x7f2f3406b050 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 10 client.24656.objecter ms_dispatch 0x7f2f34001550 osd_map(126..126 src has 66..126) v4 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 3 client.24656.objecter handle_osd_map got epochs [126,126] > 0 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f1bfff700 3 client.24656.objecter handle_osd_map decoding full epoch 126 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.816+0000 7f2f32ffd700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 0x7f2f1c071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.820+0000 7f2f1bfff700 20 client.24656.objecter dump_active .. 0 homeless 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.820+0000 7f2f1bfff700 10 client.24656.objecter ms_handle_connect 0x7f2f1c06f9c0 2024-03-11T21:27:32.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.820+0000 7f2f32ffd700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 0x7f2f1c071e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f24019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:32.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.824+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2f2805b0e0 con 0x7f2f3406b050 2024-03-11T21:27:32.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:32.896 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:32.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f363d7fa700 1 -- 172.21.15.143:0/2190363571 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v127) v1 ==== 87+0+0 (secure 0 0 0) 0x7f3634005b40 con 0x7f3640101700 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 10 librados: watch_flush enter 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 10 librados: watch_flush exit 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 20 client.24650.objecter shutdown clearing up homeless session... 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 10 client.24650.objecter successfully canceled tick 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 msgr2=0x7f3628071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 0x7f3628071ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3630009750 tx=0x7f3630009310 comp rx=0 tx=0).stop 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 msgr2=0x7f364019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3634000c00 tx=0x7f3634005430 comp rx=0 tx=0).stop 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 shutdown_connections 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f362806f820 0x7f3628071ce0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3640106dc0 0x7f364019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3640100d40 0x7f3640199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 --2- 172.21.15.143:0/2190363571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3640101700 0x7f364019a350 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 >> 172.21.15.143:0/2190363571 conn(0x7f3640076d80 msgr2=0x7f36400ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 shutdown_connections 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 -- 172.21.15.143:0/2190363571 wait complete. 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:32.940+0000 7f3646a44700 1 librados: shutdown 2024-03-11T21:27:32.920 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:27:32.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.142824+0000 mgr.y (mgr.14550) 168 : cluster 0 pgmap v139: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:32.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.664184+0000 osd.7 (osd.7) 135 : cluster 0 3.11 scrub starts 2024-03-11T21:27:32.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.666512+0000 osd.7 (osd.7) 136 : cluster 0 3.11 scrub ok 2024-03-11T21:27:32.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.987847+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:31.989912+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.094146+0000 osd.1 (osd.1) 105 : cluster 0 5.c scrub starts 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.095373+0000 osd.1 (osd.1) 106 : cluster 0 5.c scrub ok 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.138207+0000 mon.c (mon.2) 90 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.236959+0000 osd.3 (osd.3) 129 : cluster 0 2.1e scrub starts 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.238289+0000 osd.3 (osd.3) 130 : cluster 0 2.1e scrub ok 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.549142+0000 osd.0 (osd.0) 107 : cluster 0 3.12 scrub starts 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.551175+0000 osd.0 (osd.0) 108 : cluster 0 3.12 scrub ok 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.650926+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.143:0/2190363571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.651334+0000 mon.a (mon.0) 887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.756602+0000 mon.c (mon.2) 92 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.757353+0000 mon.c (mon.2) 93 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:27:32.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[29284]: audit 2024-03-11T21:27:32.762424+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.142824+0000 mgr.y (mgr.14550) 168 : cluster 0 pgmap v139: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.664184+0000 osd.7 (osd.7) 135 : cluster 0 3.11 scrub starts 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.666512+0000 osd.7 (osd.7) 136 : cluster 0 3.11 scrub ok 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.987847+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:31.989912+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.094146+0000 osd.1 (osd.1) 105 : cluster 0 5.c scrub starts 2024-03-11T21:27:32.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.095373+0000 osd.1 (osd.1) 106 : cluster 0 5.c scrub ok 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.138207+0000 mon.c (mon.2) 90 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.236959+0000 osd.3 (osd.3) 129 : cluster 0 2.1e scrub starts 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.238289+0000 osd.3 (osd.3) 130 : cluster 0 2.1e scrub ok 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.549142+0000 osd.0 (osd.0) 107 : cluster 0 3.12 scrub starts 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.551175+0000 osd.0 (osd.0) 108 : cluster 0 3.12 scrub ok 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.650926+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.143:0/2190363571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.651334+0000 mon.a (mon.0) 887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.756602+0000 mon.c (mon.2) 92 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.757353+0000 mon.c (mon.2) 93 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:27:32.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:32 smithi143 bash[21512]: audit 2024-03-11T21:27:32.762424+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:33.000 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:33.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.028+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f3410d7f0 con 0x7f2f1c06f9c0 2024-03-11T21:27:33.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.040+0000 7f2f1bfff700 1 -- 172.21.15.143:0/1281578867 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346776 (secure 0 0 0) 0x7f2f2407bab0 con 0x7f2f1c06f9c0 2024-03-11T21:27:33.020 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:27:33.021 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 10 librados: watch_flush enter 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 10 librados: watch_flush exit 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 20 client.24656.objecter shutdown clearing up homeless session... 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 10 client.24656.objecter successfully canceled tick 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 msgr2=0x7f2f1c071e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 0x7f2f1c071e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f24019040 comp rx=0 tx=0).stop 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 msgr2=0x7f2f3419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:33.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 0x7f2f3419a420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f3406b6e0 tx=0x7f2f2800dc30 comp rx=0 tx=0).stop 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 shutdown_connections 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2f1c06f9c0 0x7f2f1c071e80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f34107220 0x7f2f3419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3406a6a0 0x7f2f34199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 --2- 172.21.15.143:0/1281578867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3406b050 0x7f2f3419a420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 >> 172.21.15.143:0/1281578867 conn(0x7f2f34077de0 msgr2=0x7f2f3410d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 shutdown_connections 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 -- 172.21.15.143:0/1281578867 wait complete. 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:33.044+0000 7f2f392fa700 1 librados: shutdown 2024-03-11T21:27:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:27:33.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:33.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.142824+0000 mgr.y (mgr.14550) 168 : cluster 0 pgmap v139: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.664184+0000 osd.7 (osd.7) 135 : cluster 0 3.11 scrub starts 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.666512+0000 osd.7 (osd.7) 136 : cluster 0 3.11 scrub ok 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.987847+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:31.989912+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.094146+0000 osd.1 (osd.1) 105 : cluster 0 5.c scrub starts 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.095373+0000 osd.1 (osd.1) 106 : cluster 0 5.c scrub ok 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.138207+0000 mon.c (mon.2) 90 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.236959+0000 osd.3 (osd.3) 129 : cluster 0 2.1e scrub starts 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.238289+0000 osd.3 (osd.3) 130 : cluster 0 2.1e scrub ok 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.549142+0000 osd.0 (osd.0) 107 : cluster 0 3.12 scrub starts 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.551175+0000 osd.0 (osd.0) 108 : cluster 0 3.12 scrub ok 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.650926+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.143:0/2190363571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:33.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.651334+0000 mon.a (mon.0) 887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:33.141 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.756602+0000 mon.c (mon.2) 92 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:27:33.141 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.757353+0000 mon.c (mon.2) 93 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:27:33.141 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:32 smithi203 bash[23654]: audit 2024-03-11T21:27:32.762424+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:33.154 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:33.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:33.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:33.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.544 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:33.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:33.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:33.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.805 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:33.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:33.930 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.694941+0000 osd.7 (osd.7) 137 : cluster 0 3.d scrub starts 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.698811+0000 osd.7 (osd.7) 138 : cluster 0 3.d scrub ok 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.752007+0000 osd.4 (osd.4) 99 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.753951+0000 osd.4 (osd.4) 100 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.919661+0000 mon.a (mon.0) 889 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: audit 2024-03-11T21:27:32.933970+0000 mon.a (mon.0) 890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:34.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:33 smithi203 bash[23654]: cluster 2024-03-11T21:27:32.940845+0000 mon.a (mon.0) 891 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:27:34.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:34.044 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.694941+0000 osd.7 (osd.7) 137 : cluster 0 3.d scrub starts 2024-03-11T21:27:34.044 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.698811+0000 osd.7 (osd.7) 138 : cluster 0 3.d scrub ok 2024-03-11T21:27:34.044 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.752007+0000 osd.4 (osd.4) 99 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.753951+0000 osd.4 (osd.4) 100 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.919661+0000 mon.a (mon.0) 889 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: audit 2024-03-11T21:27:32.933970+0000 mon.a (mon.0) 890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[29284]: cluster 2024-03-11T21:27:32.940845+0000 mon.a (mon.0) 891 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.694941+0000 osd.7 (osd.7) 137 : cluster 0 3.d scrub starts 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.698811+0000 osd.7 (osd.7) 138 : cluster 0 3.d scrub ok 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.752007+0000 osd.4 (osd.4) 99 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.753951+0000 osd.4 (osd.4) 100 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.919661+0000 mon.a (mon.0) 889 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: audit 2024-03-11T21:27:32.933970+0000 mon.a (mon.0) 890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:34.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:33 smithi143 bash[21512]: cluster 2024-03-11T21:27:32.940845+0000 mon.a (mon.0) 891 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:27:34.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:34.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:34.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:34.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:34.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:34.455 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:34.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:34.584 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:34.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:27:34.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:34.767 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:34.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:34.891 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:34.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:34 smithi203 bash[23654]: audit 2024-03-11T21:27:33.033884+0000 mgr.y (mgr.14550) 169 : audit 0 from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:35.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:34 smithi203 bash[23654]: cluster 2024-03-11T21:27:33.143539+0000 mgr.y (mgr.14550) 170 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:35.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:35.137 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:34 smithi143 bash[21512]: audit 2024-03-11T21:27:33.033884+0000 mgr.y (mgr.14550) 169 : audit 0 from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:35.137 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:34 smithi143 bash[21512]: cluster 2024-03-11T21:27:33.143539+0000 mgr.y (mgr.14550) 170 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:35.137 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:34 smithi143 bash[29284]: audit 2024-03-11T21:27:33.033884+0000 mgr.y (mgr.14550) 169 : audit 0 from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:35.137 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:34 smithi143 bash[29284]: cluster 2024-03-11T21:27:33.143539+0000 mgr.y (mgr.14550) 170 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:35.153 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:35.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.289 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.348 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-03-11T21:27:32.935371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1195,"num_read_kb":1041,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253340,"kb_used_data":8092,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707876,"statfs":{"total":767960285184,"available":767700865024,"internally_reserved":0,"allocated":8286208,"data_stored":5261432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.796247"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621920+0000","last_change":"2024-03-11T21:26:27.355463+0000","last_active":"2024-03-11T21:27:17.621920+0000","last_peered":"2024-03-11T21:27:17.621920+0000","last_clean":"2024-03-11T21:27:17.621920+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:27:17.621920+0000","last_undegraded":"2024-03-11T21:27:17.621920+0000","last_fullsized":"2024-03-11T21:27:17.621920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.355103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.355103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.390277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.814463+0000","last_change":"2024-03-11T21:27:11.017133+0000","last_active":"2024-03-11T21:27:16.814463+0000","last_peered":"2024-03-11T21:27:16.814463+0000","last_clean":"2024-03-11T21:27:16.814463+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:27:16.814463+0000","last_undegraded":"2024-03-11T21:27:16.814463+0000","last_fullsized":"2024-03-11T21:27:16.814463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.016822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.016822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.129662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208289+0000","last_change":"2024-03-11T21:26:03.337169+0000","last_active":"2024-03-11T21:27:17.208289+0000","last_peered":"2024-03-11T21:27:17.208289+0000","last_clean":"2024-03-11T21:27:17.208289+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:27:17.208289+0000","last_undegraded":"2024-03-11T21:27:17.208289+0000","last_fullsized":"2024-03-11T21:27:17.208289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:26:03.337010+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.337010+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:29.657159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622377+0000","last_change":"2024-03-11T21:26:49.246965+0000","last_active":"2024-03-11T21:27:17.622377+0000","last_peered":"2024-03-11T21:27:17.622377+0000","last_clean":"2024-03-11T21:27:17.622377+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:27:17.622377+0000","last_undegraded":"2024-03-11T21:27:17.622377+0000","last_fullsized":"2024-03-11T21:27:17.622377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:49.246842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.246842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.231482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:22.792509+0000","last_change":"2024-03-11T21:27:22.792568+0000","last_active":"2024-03-11T21:27:22.792509+0000","last_peered":"2024-03-11T21:27:22.792509+0000","last_clean":"2024-03-11T21:27:22.792509+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:27:22.792509+0000","last_undegraded":"2024-03-11T21:27:22.792509+0000","last_fullsized":"2024-03-11T21:27:22.792509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.792362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.792362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.770903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819366+0000","last_change":"2024-03-11T21:26:13.570113+0000","last_active":"2024-03-11T21:27:16.819366+0000","last_peered":"2024-03-11T21:27:16.819366+0000","last_clean":"2024-03-11T21:27:16.819366+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:27:16.819366+0000","last_undegraded":"2024-03-11T21:27:16.819366+0000","last_fullsized":"2024-03-11T21:27:16.819366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.569895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.569895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.192341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622496+0000","last_change":"2024-03-11T21:26:12.187395+0000","last_active":"2024-03-11T21:27:17.622496+0000","last_peered":"2024-03-11T21:27:17.622496+0000","last_clean":"2024-03-11T21:27:17.622496+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:27:17.622496+0000","last_undegraded":"2024-03-11T21:27:17.622496+0000","last_fullsized":"2024-03-11T21:27:17.622496+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.187178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.187178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:27.244264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"121'16","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208223+0000","last_change":"2024-03-11T21:26:51.608487+0000","last_active":"2024-03-11T21:27:17.208223+0000","last_peered":"2024-03-11T21:27:17.208223+0000","last_clean":"2024-03-11T21:27:17.208223+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:27:17.208223+0000","last_undegraded":"2024-03-11T21:27:17.208223+0000","last_fullsized":"2024-03-11T21:27:17.208223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.608221+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.211682+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.742729+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:27:17.742729+0000","last_peered":"2024-03-11T21:27:17.742729+0000","last_clean":"2024-03-11T21:27:17.742729+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:27:17.742729+0000","last_undegraded":"2024-03-11T21:27:17.742729+0000","last_fullsized":"2024-03-11T21:27:17.742729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.062649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.708522+0000","last_change":"2024-03-11T21:27:18.708546+0000","last_active":"2024-03-11T21:27:18.708522+0000","last_peered":"2024-03-11T21:27:18.708522+0000","last_clean":"2024-03-11T21:27:18.708522+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:27:18.708522+0000","last_undegraded":"2024-03-11T21:27:18.708522+0000","last_fullsized":"2024-03-11T21:27:18.708522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.708450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.708450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:43.764001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.592414+0000","last_change":"2024-03-11T21:26:35.765693+0000","last_active":"2024-03-11T21:27:17.592414+0000","last_peered":"2024-03-11T21:27:17.592414+0000","last_clean":"2024-03-11T21:27:17.592414+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:27:17.592414+0000","last_undegraded":"2024-03-11T21:27:17.592414+0000","last_fullsized":"2024-03-11T21:27:17.592414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:35.765486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:35.765486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:44.892300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744947+0000","last_change":"2024-03-11T21:26:31.763473+0000","last_active":"2024-03-11T21:27:17.744947+0000","last_peered":"2024-03-11T21:27:17.744947+0000","last_clean":"2024-03-11T21:27:17.744947+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:27:17.744947+0000","last_undegraded":"2024-03-11T21:27:17.744947+0000","last_fullsized":"2024-03-11T21:27:17.744947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.763327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:46.643188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:22.541270+0000","last_change":"2024-03-11T21:27:22.541333+0000","last_active":"2024-03-11T21:27:22.541270+0000","last_peered":"2024-03-11T21:27:22.541270+0000","last_clean":"2024-03-11T21:27:22.541270+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:27:22.541270+0000","last_undegraded":"2024-03-11T21:27:22.541270+0000","last_fullsized":"2024-03-11T21:27:22.541270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.541117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.541117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.740379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744071+0000","last_change":"2024-03-11T21:26:11.457112+0000","last_active":"2024-03-11T21:27:17.744071+0000","last_peered":"2024-03-11T21:27:17.744071+0000","last_clean":"2024-03-11T21:27:17.744071+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:27:17.744071+0000","last_undegraded":"2024-03-11T21:27:17.744071+0000","last_fullsized":"2024-03-11T21:27:17.744071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.456900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.456900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:19.697223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591855+0000","last_change":"2024-03-11T21:27:08.632675+0000","last_active":"2024-03-11T21:27:17.591855+0000","last_peered":"2024-03-11T21:27:17.591855+0000","last_clean":"2024-03-11T21:27:17.591855+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:27:17.591855+0000","last_undegraded":"2024-03-11T21:27:17.591855+0000","last_fullsized":"2024-03-11T21:27:17.591855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.632381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.632381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.000117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.745161+0000","last_change":"2024-03-11T21:26:59.726007+0000","last_active":"2024-03-11T21:27:17.745161+0000","last_peered":"2024-03-11T21:27:17.745161+0000","last_clean":"2024-03-11T21:27:17.745161+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:27:17.745161+0000","last_undegraded":"2024-03-11T21:27:17.745161+0000","last_fullsized":"2024-03-11T21:27:17.745161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:59.725612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.725612+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.676623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":113,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622514+0000","last_change":"2024-03-11T21:26:20.277298+0000","last_active":"2024-03-11T21:27:17.622514+0000","last_peered":"2024-03-11T21:27:17.622514+0000","last_clean":"2024-03-11T21:27:17.622514+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:27:17.622514+0000","last_undegraded":"2024-03-11T21:27:17.622514+0000","last_fullsized":"2024-03-11T21:27:17.622514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.277140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.790443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.815818+0000","last_change":"2024-03-11T21:26:10.005094+0000","last_active":"2024-03-11T21:27:16.815818+0000","last_peered":"2024-03-11T21:27:16.815818+0000","last_clean":"2024-03-11T21:27:16.815818+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:27:16.815818+0000","last_undegraded":"2024-03-11T21:27:16.815818+0000","last_fullsized":"2024-03-11T21:27:16.815818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.004882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.004882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:30.146208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621990+0000","last_change":"2024-03-11T21:26:39.219283+0000","last_active":"2024-03-11T21:27:17.621990+0000","last_peered":"2024-03-11T21:27:17.621990+0000","last_clean":"2024-03-11T21:27:17.621990+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:27:17.621990+0000","last_undegraded":"2024-03-11T21:27:17.621990+0000","last_fullsized":"2024-03-11T21:27:17.621990+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:26:39.218970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:01.129287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.815177+0000","last_change":"2024-03-11T21:27:04.011882+0000","last_active":"2024-03-11T21:27:16.815177+0000","last_peered":"2024-03-11T21:27:16.815177+0000","last_clean":"2024-03-11T21:27:16.815177+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:27:16.815177+0000","last_undegraded":"2024-03-11T21:27:16.815177+0000","last_fullsized":"2024-03-11T21:27:16.815177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:27:04.011634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.011634+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.925801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.817141+0000","last_change":"2024-03-11T21:27:10.026903+0000","last_active":"2024-03-11T21:27:16.817141+0000","last_peered":"2024-03-11T21:27:16.817141+0000","last_clean":"2024-03-11T21:27:16.817141+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:27:16.817141+0000","last_undegraded":"2024-03-11T21:27:16.817141+0000","last_fullsized":"2024-03-11T21:27:16.817141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.026738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.026738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.995934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.821231+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:27:16.821231+0000","last_peered":"2024-03-11T21:27:16.821231+0000","last_clean":"2024-03-11T21:27:16.821231+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:27:16.821231+0000","last_undegraded":"2024-03-11T21:27:16.821231+0000","last_fullsized":"2024-03-11T21:27:16.821231+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.655474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706207+0000","last_change":"2024-03-11T21:26:04.956591+0000","last_active":"2024-03-11T21:27:17.706207+0000","last_peered":"2024-03-11T21:27:17.706207+0000","last_clean":"2024-03-11T21:27:17.706207+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:27:17.706207+0000","last_undegraded":"2024-03-11T21:27:17.706207+0000","last_fullsized":"2024-03-11T21:27:17.706207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.956350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.956350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:26.718796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208205+0000","last_change":"2024-03-11T21:26:20.455065+0000","last_active":"2024-03-11T21:27:17.208205+0000","last_peered":"2024-03-11T21:27:17.208205+0000","last_clean":"2024-03-11T21:27:17.208205+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:27:17.208205+0000","last_undegraded":"2024-03-11T21:27:17.208205+0000","last_fullsized":"2024-03-11T21:27:17.208205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.454747+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.108877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.821487+0000","last_change":"2024-03-11T21:26:09.575095+0000","last_active":"2024-03-11T21:27:16.821487+0000","last_peered":"2024-03-11T21:27:16.821487+0000","last_clean":"2024-03-11T21:27:16.821487+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:27:16.821487+0000","last_undegraded":"2024-03-11T21:27:16.821487+0000","last_fullsized":"2024-03-11T21:27:16.821487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:09.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:09.574796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.590692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208026+0000","last_change":"2024-03-11T21:27:08.631242+0000","last_active":"2024-03-11T21:27:17.208026+0000","last_peered":"2024-03-11T21:27:17.208026+0000","last_clean":"2024-03-11T21:27:17.208026+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:27:17.208026+0000","last_undegraded":"2024-03-11T21:27:17.208026+0000","last_fullsized":"2024-03-11T21:27:17.208026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.631010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.631010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:38.452241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621541+0000","last_change":"2024-03-11T21:26:41.250247+0000","last_active":"2024-03-11T21:27:17.621541+0000","last_peered":"2024-03-11T21:27:17.621541+0000","last_clean":"2024-03-11T21:27:17.621541+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:27:17.621541+0000","last_undegraded":"2024-03-11T21:27:17.621541+0000","last_fullsized":"2024-03-11T21:27:17.621541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:26:41.250134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.250134+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.792042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621691+0000","last_change":"2024-03-11T21:26:38.192925+0000","last_active":"2024-03-11T21:27:17.621691+0000","last_peered":"2024-03-11T21:27:17.621691+0000","last_clean":"2024-03-11T21:27:17.621691+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:27:17.621691+0000","last_undegraded":"2024-03-11T21:27:17.621691+0000","last_fullsized":"2024-03-11T21:27:17.621691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:38.192555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.192555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.368750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.590468+0000","last_change":"2024-03-11T21:26:29.967551+0000","last_active":"2024-03-11T21:27:17.590468+0000","last_peered":"2024-03-11T21:27:17.590468+0000","last_clean":"2024-03-11T21:27:17.590468+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:27:17.590468+0000","last_undegraded":"2024-03-11T21:27:17.590468+0000","last_fullsized":"2024-03-11T21:27:17.590468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:26:29.967316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.967316+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.913576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":191,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.192521+0000","last_change":"2024-03-11T21:27:08.856139+0000","last_active":"2024-03-11T21:27:18.192521+0000","last_peered":"2024-03-11T21:27:18.192521+0000","last_clean":"2024-03-11T21:27:18.192521+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:27:18.192521+0000","last_undegraded":"2024-03-11T21:27:18.192521+0000","last_fullsized":"2024-03-11T21:27:18.192521+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:08.855901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.855901+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.699794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":130,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.743365+0000","last_change":"2024-03-11T21:26:02.471008+0000","last_active":"2024-03-11T21:27:17.743365+0000","last_peered":"2024-03-11T21:27:17.743365+0000","last_clean":"2024-03-11T21:27:17.743365+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:27:17.743365+0000","last_undegraded":"2024-03-11T21:27:17.743365+0000","last_fullsized":"2024-03-11T21:27:17.743365+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:02.470792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:02.470792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":150,"reported_epoch":125,"state":"active+clean+scrubbing","last_fresh":"2024-03-11T21:27:28.072011+0000","last_change":"2024-03-11T21:27:28.066213+0000","last_active":"2024-03-11T21:27:28.072011+0000","last_peered":"2024-03-11T21:27:28.072011+0000","last_clean":"2024-03-11T21:27:28.072011+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:27:28.072011+0000","last_undegraded":"2024-03-11T21:27:28.072011+0000","last_fullsized":"2024-03-11T21:27:28.072011+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:03.728766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.728766+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622181+0000","last_change":"2024-03-11T21:26:28.313721+0000","last_active":"2024-03-11T21:27:17.622181+0000","last_peered":"2024-03-11T21:27:17.622181+0000","last_clean":"2024-03-11T21:27:17.622181+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:27:17.622181+0000","last_undegraded":"2024-03-11T21:27:17.622181+0000","last_fullsized":"2024-03-11T21:27:17.622181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:26:28.313489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.975519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591669+0000","last_change":"2024-03-11T21:26:20.986742+0000","last_active":"2024-03-11T21:27:17.591669+0000","last_peered":"2024-03-11T21:27:17.591669+0000","last_clean":"2024-03-11T21:27:17.591669+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:27:17.591669+0000","last_undegraded":"2024-03-11T21:27:17.591669+0000","last_fullsized":"2024-03-11T21:27:17.591669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.986534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.986534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.313433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.707843+0000","last_change":"2024-03-11T21:26:48.997049+0000","last_active":"2024-03-11T21:27:17.707843+0000","last_peered":"2024-03-11T21:27:17.707843+0000","last_clean":"2024-03-11T21:27:17.707843+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:27:17.707843+0000","last_undegraded":"2024-03-11T21:27:17.707843+0000","last_fullsized":"2024-03-11T21:27:17.707843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.996626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.996626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.276693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744653+0000","last_change":"2024-03-11T21:26:50.749938+0000","last_active":"2024-03-11T21:27:17.744653+0000","last_peered":"2024-03-11T21:27:17.744653+0000","last_clean":"2024-03-11T21:27:17.744653+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:27:17.744653+0000","last_undegraded":"2024-03-11T21:27:17.744653+0000","last_fullsized":"2024-03-11T21:27:17.744653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:50.749697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.749697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.985992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:20.660126+0000","last_change":"2024-03-11T21:27:20.660153+0000","last_active":"2024-03-11T21:27:20.660126+0000","last_peered":"2024-03-11T21:27:20.660126+0000","last_clean":"2024-03-11T21:27:20.660126+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:27:20.660126+0000","last_undegraded":"2024-03-11T21:27:20.660126+0000","last_fullsized":"2024-03-11T21:27:20.660126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.660038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.660038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.672421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706018+0000","last_change":"2024-03-11T21:26:29.972923+0000","last_active":"2024-03-11T21:27:17.706018+0000","last_peered":"2024-03-11T21:27:17.706018+0000","last_clean":"2024-03-11T21:27:17.706018+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:27:17.706018+0000","last_undegraded":"2024-03-11T21:27:17.706018+0000","last_fullsized":"2024-03-11T21:27:17.706018+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.972683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.497791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.745237+0000","last_change":"2024-03-11T21:26:51.718223+0000","last_active":"2024-03-11T21:27:17.745237+0000","last_peered":"2024-03-11T21:27:17.745237+0000","last_clean":"2024-03-11T21:27:17.745237+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:27:17.745237+0000","last_undegraded":"2024-03-11T21:27:17.745237+0000","last_fullsized":"2024-03-11T21:27:17.745237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.718008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.718008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.911443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.207908+0000","last_change":"2024-03-11T21:26:52.592522+0000","last_active":"2024-03-11T21:27:17.207908+0000","last_peered":"2024-03-11T21:27:17.207908+0000","last_clean":"2024-03-11T21:27:17.207908+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:27:17.207908+0000","last_undegraded":"2024-03-11T21:27:17.207908+0000","last_fullsized":"2024-03-11T21:27:17.207908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:26:52.592275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.592275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:57.848346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621483+0000","last_change":"2024-03-11T21:27:03.215324+0000","last_active":"2024-03-11T21:27:17.621483+0000","last_peered":"2024-03-11T21:27:17.621483+0000","last_clean":"2024-03-11T21:27:17.621483+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:27:17.621483+0000","last_undegraded":"2024-03-11T21:27:17.621483+0000","last_fullsized":"2024-03-11T21:27:17.621483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.215038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.969404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820096+0000","last_change":"2024-03-11T21:26:14.593548+0000","last_active":"2024-03-11T21:27:16.820096+0000","last_peered":"2024-03-11T21:27:16.820096+0000","last_clean":"2024-03-11T21:27:16.820096+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:27:16.820096+0000","last_undegraded":"2024-03-11T21:27:16.820096+0000","last_fullsized":"2024-03-11T21:27:16.820096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.593259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.593259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:43.052839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820852+0000","last_change":"2024-03-11T21:27:05.957467+0000","last_active":"2024-03-11T21:27:16.820852+0000","last_peered":"2024-03-11T21:27:16.820852+0000","last_clean":"2024-03-11T21:27:16.820852+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:27:16.820852+0000","last_undegraded":"2024-03-11T21:27:16.820852+0000","last_fullsized":"2024-03-11T21:27:16.820852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.957246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.957246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.013602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":143,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.816150+0000","last_change":"2024-03-11T21:26:54.979484+0000","last_active":"2024-03-11T21:27:16.816150+0000","last_peered":"2024-03-11T21:27:16.816150+0000","last_clean":"2024-03-11T21:27:16.816150+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:27:16.816150+0000","last_undegraded":"2024-03-11T21:27:16.816150+0000","last_fullsized":"2024-03-11T21:27:16.816150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:54.979250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.979250+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.137828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.206650+0000","last_change":"2024-03-11T21:27:07.661169+0000","last_active":"2024-03-11T21:27:17.206650+0000","last_peered":"2024-03-11T21:27:17.206650+0000","last_clean":"2024-03-11T21:27:17.206650+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:27:17.206650+0000","last_undegraded":"2024-03-11T21:27:17.206650+0000","last_fullsized":"2024-03-11T21:27:17.206650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.660940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.825652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:19.230006+0000","last_change":"2024-03-11T21:27:19.230032+0000","last_active":"2024-03-11T21:27:19.230006+0000","last_peered":"2024-03-11T21:27:19.230006+0000","last_clean":"2024-03-11T21:27:19.230006+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:27:19.230006+0000","last_undegraded":"2024-03-11T21:27:19.230006+0000","last_fullsized":"2024-03-11T21:27:19.230006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.229916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.229916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.157420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706731+0000","last_change":"2024-03-11T21:26:54.928990+0000","last_active":"2024-03-11T21:27:17.706731+0000","last_peered":"2024-03-11T21:27:17.706731+0000","last_clean":"2024-03-11T21:27:17.706731+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:27:17.706731+0000","last_undegraded":"2024-03-11T21:27:17.706731+0000","last_fullsized":"2024-03-11T21:27:17.706731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.928741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.296625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.592682+0000","last_change":"2024-03-11T21:26:54.618935+0000","last_active":"2024-03-11T21:27:17.592682+0000","last_peered":"2024-03-11T21:27:17.592682+0000","last_clean":"2024-03-11T21:27:17.592682+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:27:17.592682+0000","last_undegraded":"2024-03-11T21:27:17.592682+0000","last_fullsized":"2024-03-11T21:27:17.592682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:26:54.618687+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.618687+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.713156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.817375+0000","last_change":"2024-03-11T21:26:58.903854+0000","last_active":"2024-03-11T21:27:16.817375+0000","last_peered":"2024-03-11T21:27:16.817375+0000","last_clean":"2024-03-11T21:27:16.817375+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:27:16.817375+0000","last_undegraded":"2024-03-11T21:27:16.817375+0000","last_fullsized":"2024-03-11T21:27:16.817375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.903611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.903611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.500162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.197463+0000","last_change":"2024-03-11T21:26:56.252517+0000","last_active":"2024-03-11T21:27:18.197463+0000","last_peered":"2024-03-11T21:27:18.197463+0000","last_clean":"2024-03-11T21:27:18.197463+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:27:18.197463+0000","last_undegraded":"2024-03-11T21:27:18.197463+0000","last_fullsized":"2024-03-11T21:27:18.197463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.252414+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.142273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208082+0000","last_change":"2024-03-11T21:26:55.654571+0000","last_active":"2024-03-11T21:27:17.208082+0000","last_peered":"2024-03-11T21:27:17.208082+0000","last_clean":"2024-03-11T21:27:17.208082+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:27:17.208082+0000","last_undegraded":"2024-03-11T21:27:17.208082+0000","last_fullsized":"2024-03-11T21:27:17.208082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.654463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.654463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.184726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.207115+0000","last_change":"2024-03-11T21:26:01.314774+0000","last_active":"2024-03-11T21:27:17.207115+0000","last_peered":"2024-03-11T21:27:17.207115+0000","last_clean":"2024-03-11T21:27:17.207115+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:27:17.207115+0000","last_undegraded":"2024-03-11T21:27:17.207115+0000","last_fullsized":"2024-03-11T21:27:17.207115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:26:01.314615+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:23.665529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.207694+0000","last_change":"2024-03-11T21:27:05.686646+0000","last_active":"2024-03-11T21:27:17.207694+0000","last_peered":"2024-03-11T21:27:17.207694+0000","last_clean":"2024-03-11T21:27:17.207694+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:27:17.207694+0000","last_undegraded":"2024-03-11T21:27:17.207694+0000","last_fullsized":"2024-03-11T21:27:17.207694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.686435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:05.686435+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.686435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.382738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":187,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:25.007591+0000","last_change":"2024-03-11T21:27:25.007663+0000","last_active":"2024-03-11T21:27:25.007591+0000","last_peered":"2024-03-11T21:27:25.007591+0000","last_clean":"2024-03-11T21:27:25.007591+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:27:25.007591+0000","last_undegraded":"2024-03-11T21:27:25.007591+0000","last_fullsized":"2024-03-11T21:27:25.007591+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:25.007427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.007427+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.826870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.743924+0000","last_change":"2024-03-11T21:26:27.752411+0000","last_active":"2024-03-11T21:27:17.743924+0000","last_peered":"2024-03-11T21:27:17.743924+0000","last_clean":"2024-03-11T21:27:17.743924+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:27:17.743924+0000","last_undegraded":"2024-03-11T21:27:17.743924+0000","last_fullsized":"2024-03-11T21:27:17.743924+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.752198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.752198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:42.079868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":147,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591975+0000","last_change":"2024-03-11T21:26:50.713963+0000","last_active":"2024-03-11T21:27:17.591975+0000","last_peered":"2024-03-11T21:27:17.591975+0000","last_clean":"2024-03-11T21:27:17.591975+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:27:17.591975+0000","last_undegraded":"2024-03-11T21:27:17.591975+0000","last_fullsized":"2024-03-11T21:27:17.591975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:50.713726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.713726+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.311122+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.590498+0000","last_change":"2024-03-11T21:27:13.605988+0000","last_active":"2024-03-11T21:27:17.590498+0000","last_peered":"2024-03-11T21:27:17.590498+0000","last_clean":"2024-03-11T21:27:17.590498+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:27:17.590498+0000","last_undegraded":"2024-03-11T21:27:17.590498+0000","last_fullsized":"2024-03-11T21:27:17.590498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:13.605762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:13.605762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.100312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744924+0000","last_change":"2024-03-11T21:26:57.676747+0000","last_active":"2024-03-11T21:27:17.744924+0000","last_peered":"2024-03-11T21:27:17.744924+0000","last_clean":"2024-03-11T21:27:17.744924+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:27:17.744924+0000","last_undegraded":"2024-03-11T21:27:17.744924+0000","last_fullsized":"2024-03-11T21:27:17.744924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.676465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.676465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.099668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706320+0000","last_change":"2024-03-11T21:26:43.871224+0000","last_active":"2024-03-11T21:27:17.706320+0000","last_peered":"2024-03-11T21:27:17.706320+0000","last_clean":"2024-03-11T21:27:17.706320+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:27:17.706320+0000","last_undegraded":"2024-03-11T21:27:17.706320+0000","last_fullsized":"2024-03-11T21:27:17.706320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:43.871005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:43.871005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.711985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.705916+0000","last_change":"2024-03-11T21:26:56.922599+0000","last_active":"2024-03-11T21:27:17.705916+0000","last_peered":"2024-03-11T21:27:17.705916+0000","last_clean":"2024-03-11T21:27:17.705916+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:27:17.705916+0000","last_undegraded":"2024-03-11T21:27:17.705916+0000","last_fullsized":"2024-03-11T21:27:17.705916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:26:56.922358+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.922358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.954469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819046+0000","last_change":"2024-03-11T21:27:02.848853+0000","last_active":"2024-03-11T21:27:16.819046+0000","last_peered":"2024-03-11T21:27:16.819046+0000","last_clean":"2024-03-11T21:27:16.819046+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:27:16.819046+0000","last_undegraded":"2024-03-11T21:27:16.819046+0000","last_fullsized":"2024-03-11T21:27:16.819046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.848551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.848551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.623625+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:20.200350+0000","last_change":"2024-03-11T21:27:20.200413+0000","last_active":"2024-03-11T21:27:20.200350+0000","last_peered":"2024-03-11T21:27:20.200350+0000","last_clean":"2024-03-11T21:27:20.200350+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:27:20.200350+0000","last_undegraded":"2024-03-11T21:27:20.200350+0000","last_fullsized":"2024-03-11T21:27:20.200350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.200196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.200196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.303367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.817622+0000","last_change":"2024-03-11T21:26:45.975343+0000","last_active":"2024-03-11T21:27:16.817622+0000","last_peered":"2024-03-11T21:27:16.817622+0000","last_clean":"2024-03-11T21:27:16.817622+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:27:16.817622+0000","last_undegraded":"2024-03-11T21:27:16.817622+0000","last_fullsized":"2024-03-11T21:27:16.817622+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.975124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.185402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":139,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591051+0000","last_change":"2024-03-11T21:26:10.162900+0000","last_active":"2024-03-11T21:27:17.591051+0000","last_peered":"2024-03-11T21:27:17.591051+0000","last_clean":"2024-03-11T21:27:17.591051+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:27:17.591051+0000","last_undegraded":"2024-03-11T21:27:17.591051+0000","last_fullsized":"2024-03-11T21:27:17.591051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:10.162661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.162661+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:28.988891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621247+0000","last_change":"2024-03-11T21:26:57.223304+0000","last_active":"2024-03-11T21:27:17.621247+0000","last_peered":"2024-03-11T21:27:17.621247+0000","last_clean":"2024-03-11T21:27:17.621247+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:27:17.621247+0000","last_undegraded":"2024-03-11T21:27:17.621247+0000","last_fullsized":"2024-03-11T21:27:17.621247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.223046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.223046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:57.942237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":150,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.816768+0000","last_change":"2024-03-11T21:26:49.041325+0000","last_active":"2024-03-11T21:27:16.816768+0000","last_peered":"2024-03-11T21:27:16.816768+0000","last_clean":"2024-03-11T21:27:16.816768+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:27:16.816768+0000","last_undegraded":"2024-03-11T21:27:16.816768+0000","last_fullsized":"2024-03-11T21:27:16.816768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.041020+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.587235+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:19.700774+0000","last_change":"2024-03-11T21:27:19.700859+0000","last_active":"2024-03-11T21:27:19.700774+0000","last_peered":"2024-03-11T21:27:19.700774+0000","last_clean":"2024-03-11T21:27:19.700774+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:27:19.700774+0000","last_undegraded":"2024-03-11T21:27:19.700774+0000","last_fullsized":"2024-03-11T21:27:19.700774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:27:19.700661+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:27:19.700661+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.700661+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:19.769280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:21.742470+0000","last_change":"2024-03-11T21:27:21.742470+0000","last_active":"2024-03-11T21:27:21.742470+0000","last_peered":"2024-03-11T21:27:21.742470+0000","last_clean":"2024-03-11T21:27:21.742470+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:27:21.742470+0000","last_undegraded":"2024-03-11T21:27:21.742470+0000","last_fullsized":"2024-03-11T21:27:21.742470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.742357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.742357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.709018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.623699+0000","last_change":"2024-03-11T21:27:17.623720+0000","last_active":"2024-03-11T21:27:17.623699+0000","last_peered":"2024-03-11T21:27:17.623699+0000","last_clean":"2024-03-11T21:27:17.623699+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:27:17.623699+0000","last_undegraded":"2024-03-11T21:27:17.623699+0000","last_fullsized":"2024-03-11T21:27:17.623699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.623631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.623631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:47.137727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820896+0000","last_change":"2024-03-11T21:26:59.857825+0000","last_active":"2024-03-11T21:27:16.820896+0000","last_peered":"2024-03-11T21:27:16.820896+0000","last_clean":"2024-03-11T21:27:16.820896+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:27:16.820896+0000","last_undegraded":"2024-03-11T21:27:16.820896+0000","last_fullsized":"2024-03-11T21:27:16.820896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.857611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.857611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.753621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.192529+0000","last_change":"2024-03-11T21:26:45.687511+0000","last_active":"2024-03-11T21:27:18.192529+0000","last_peered":"2024-03-11T21:27:18.192529+0000","last_clean":"2024-03-11T21:27:18.192529+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:27:18.192529+0000","last_undegraded":"2024-03-11T21:27:18.192529+0000","last_fullsized":"2024-03-11T21:27:18.192529+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:26:45.687244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.687244+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.865274+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.822008+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:27:16.822008+0000","last_peered":"2024-03-11T21:27:16.822008+0000","last_clean":"2024-03-11T21:27:16.822008+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:27:16.822008+0000","last_undegraded":"2024-03-11T21:27:16.822008+0000","last_fullsized":"2024-03-11T21:27:16.822008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:20.556814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":192,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.197110+0000","last_change":"2024-03-11T21:27:12.664471+0000","last_active":"2024-03-11T21:27:18.197110+0000","last_peered":"2024-03-11T21:27:18.197110+0000","last_clean":"2024-03-11T21:27:18.197110+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:27:18.197110+0000","last_undegraded":"2024-03-11T21:27:18.197110+0000","last_fullsized":"2024-03-11T21:27:18.197110+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.664354+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.849172+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":113,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621322+0000","last_change":"2024-03-11T21:26:30.315059+0000","last_active":"2024-03-11T21:27:17.621322+0000","last_peered":"2024-03-11T21:27:17.621322+0000","last_clean":"2024-03-11T21:27:17.621322+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:27:17.621322+0000","last_undegraded":"2024-03-11T21:27:17.621322+0000","last_fullsized":"2024-03-11T21:27:17.621322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:30.314840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:30.314840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:39.460296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744142+0000","last_change":"2024-03-11T21:26:13.487586+0000","last_active":"2024-03-11T21:27:17.744142+0000","last_peered":"2024-03-11T21:27:17.744142+0000","last_clean":"2024-03-11T21:27:17.744142+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:27:17.744142+0000","last_undegraded":"2024-03-11T21:27:17.744142+0000","last_fullsized":"2024-03-11T21:27:17.744142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.487359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.487359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:38.520405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.814926+0000","last_change":"2024-03-11T21:26:40.103016+0000","last_active":"2024-03-11T21:27:16.814926+0000","last_peered":"2024-03-11T21:27:16.814926+0000","last_clean":"2024-03-11T21:27:16.814926+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:27:16.814926+0000","last_undegraded":"2024-03-11T21:27:16.814926+0000","last_fullsized":"2024-03-11T21:27:16.814926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:26:40.102764+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.102764+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.411337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":117,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.209449+0000","last_change":"2024-03-11T21:26:25.426916+0000","last_active":"2024-03-11T21:27:17.209449+0000","last_peered":"2024-03-11T21:27:17.209449+0000","last_clean":"2024-03-11T21:27:17.209449+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:27:17.209449+0000","last_undegraded":"2024-03-11T21:27:17.209449+0000","last_fullsized":"2024-03-11T21:27:17.209449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:25.426623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:25.426623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.790210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.710638+0000","last_change":"2024-03-11T21:27:17.710638+0000","last_active":"2024-03-11T21:27:17.710638+0000","last_peered":"2024-03-11T21:27:17.710638+0000","last_clean":"2024-03-11T21:27:17.710638+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:27:17.710638+0000","last_undegraded":"2024-03-11T21:27:17.710638+0000","last_fullsized":"2024-03-11T21:27:17.710638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.710431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.710431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.182795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":161,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:21.120957+0000","last_change":"2024-03-11T21:27:21.121066+0000","last_active":"2024-03-11T21:27:21.120957+0000","last_peered":"2024-03-11T21:27:21.120957+0000","last_clean":"2024-03-11T21:27:21.120957+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:27:21.120957+0000","last_undegraded":"2024-03-11T21:27:21.120957+0000","last_fullsized":"2024-03-11T21:27:21.120957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:27:21.120787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.120787+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.943211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.817951+0000","last_change":"2024-03-11T21:27:04.805372+0000","last_active":"2024-03-11T21:27:16.817951+0000","last_peered":"2024-03-11T21:27:16.817951+0000","last_clean":"2024-03-11T21:27:16.817951+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:27:16.817951+0000","last_undegraded":"2024-03-11T21:27:16.817951+0000","last_fullsized":"2024-03-11T21:27:16.817951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.805100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.805100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.216891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819204+0000","last_change":"2024-03-11T21:27:12.756161+0000","last_active":"2024-03-11T21:27:16.819204+0000","last_peered":"2024-03-11T21:27:16.819204+0000","last_clean":"2024-03-11T21:27:16.819204+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:27:16.819204+0000","last_undegraded":"2024-03-11T21:27:16.819204+0000","last_fullsized":"2024-03-11T21:27:16.819204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.755925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.755925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.683631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.743075+0000","last_change":"2024-03-11T21:27:14.796892+0000","last_active":"2024-03-11T21:27:17.743075+0000","last_peered":"2024-03-11T21:27:17.743075+0000","last_clean":"2024-03-11T21:27:17.743075+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:27:17.743075+0000","last_undegraded":"2024-03-11T21:27:17.743075+0000","last_fullsized":"2024-03-11T21:27:17.743075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:14.796603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.796603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.446263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:25.263770+0000","last_change":"2024-03-11T21:27:25.263826+0000","last_active":"2024-03-11T21:27:25.263770+0000","last_peered":"2024-03-11T21:27:25.263770+0000","last_clean":"2024-03-11T21:27:25.263770+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:27:25.263770+0000","last_undegraded":"2024-03-11T21:27:25.263770+0000","last_fullsized":"2024-03-11T21:27:25.263770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:25.263676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.263676+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.297940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.744204+0000","last_change":"2024-03-11T21:26:52.732913+0000","last_active":"2024-03-11T21:27:17.744204+0000","last_peered":"2024-03-11T21:27:17.744204+0000","last_clean":"2024-03-11T21:27:17.744204+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:27:17.744204+0000","last_undegraded":"2024-03-11T21:27:17.744204+0000","last_fullsized":"2024-03-11T21:27:17.744204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.732676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.732676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:19.391370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":386,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591058+0000","last_change":"2024-03-11T21:26:36.758336+0000","last_active":"2024-03-11T21:27:17.591058+0000","last_peered":"2024-03-11T21:27:17.591058+0000","last_clean":"2024-03-11T21:27:17.591058+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:27:17.591058+0000","last_undegraded":"2024-03-11T21:27:17.591058+0000","last_fullsized":"2024-03-11T21:27:17.591058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:26:36.758088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:26:36.758088+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:04.601201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":117,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.592268+0000","last_change":"2024-03-11T21:26:34.807309+0000","last_active":"2024-03-11T21:27:17.592268+0000","last_peered":"2024-03-11T21:27:17.592268+0000","last_clean":"2024-03-11T21:27:17.592268+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:27:17.592268+0000","last_undegraded":"2024-03-11T21:27:17.592268+0000","last_fullsized":"2024-03-11T21:27:17.592268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:34.807061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:34.807061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:56.122316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":113,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820206+0000","last_change":"2024-03-11T21:26:10.602526+0000","last_active":"2024-03-11T21:27:16.820206+0000","last_peered":"2024-03-11T21:27:16.820206+0000","last_clean":"2024-03-11T21:27:16.820206+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:27:16.820206+0000","last_undegraded":"2024-03-11T21:27:16.820206+0000","last_fullsized":"2024-03-11T21:27:16.820206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.602230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.602230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:36.316832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":142,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.206911+0000","last_change":"2024-03-11T21:26:13.337394+0000","last_active":"2024-03-11T21:27:17.206911+0000","last_peered":"2024-03-11T21:27:17.206911+0000","last_clean":"2024-03-11T21:27:17.206911+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:27:17.206911+0000","last_undegraded":"2024-03-11T21:27:17.206911+0000","last_fullsized":"2024-03-11T21:27:17.206911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:26:13.337090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.337090+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.002783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591053+0000","last_change":"2024-03-11T21:27:06.618092+0000","last_active":"2024-03-11T21:27:17.591053+0000","last_peered":"2024-03-11T21:27:17.591053+0000","last_clean":"2024-03-11T21:27:17.591053+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:27:17.591053+0000","last_undegraded":"2024-03-11T21:27:17.591053+0000","last_fullsized":"2024-03-11T21:27:17.591053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.617820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.617820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.715234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:19.924377+0000","last_change":"2024-03-11T21:27:19.924377+0000","last_active":"2024-03-11T21:27:19.924377+0000","last_peered":"2024-03-11T21:27:19.924377+0000","last_clean":"2024-03-11T21:27:19.924377+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:27:19.924377+0000","last_undegraded":"2024-03-11T21:27:19.924377+0000","last_fullsized":"2024-03-11T21:27:19.924377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.924170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.924170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.337440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":113,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819366+0000","last_change":"2024-03-11T21:26:04.760536+0000","last_active":"2024-03-11T21:27:16.819366+0000","last_peered":"2024-03-11T21:27:16.819366+0000","last_clean":"2024-03-11T21:27:16.819366+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:27:16.819366+0000","last_undegraded":"2024-03-11T21:27:16.819366+0000","last_fullsized":"2024-03-11T21:27:16.819366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.760262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.760262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.318865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.592143+0000","last_change":"2024-03-11T21:27:07.638125+0000","last_active":"2024-03-11T21:27:17.592143+0000","last_peered":"2024-03-11T21:27:17.592143+0000","last_clean":"2024-03-11T21:27:17.592143+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:27:17.592143+0000","last_undegraded":"2024-03-11T21:27:17.592143+0000","last_fullsized":"2024-03-11T21:27:17.592143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.637915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.637915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.738623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":232,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.197185+0000","last_change":"2024-03-11T21:26:40.725235+0000","last_active":"2024-03-11T21:27:18.197185+0000","last_peered":"2024-03-11T21:27:18.197185+0000","last_clean":"2024-03-11T21:27:18.197185+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:27:18.197185+0000","last_undegraded":"2024-03-11T21:27:18.197185+0000","last_fullsized":"2024-03-11T21:27:18.197185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:26:40.724980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.724980+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.435897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":117,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706541+0000","last_change":"2024-03-11T21:26:08.930659+0000","last_active":"2024-03-11T21:27:17.706541+0000","last_peered":"2024-03-11T21:27:17.706541+0000","last_clean":"2024-03-11T21:27:17.706541+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:27:17.706541+0000","last_undegraded":"2024-03-11T21:27:17.706541+0000","last_fullsized":"2024-03-11T21:27:17.706541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:08.930415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.930415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:33.664164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:23.772678+0000","last_change":"2024-03-11T21:27:23.772704+0000","last_active":"2024-03-11T21:27:23.772678+0000","last_peered":"2024-03-11T21:27:23.772678+0000","last_clean":"2024-03-11T21:27:23.772678+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:27:23.772678+0000","last_undegraded":"2024-03-11T21:27:23.772678+0000","last_fullsized":"2024-03-11T21:27:23.772678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.772573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.772573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.153346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820394+0000","last_change":"2024-03-11T21:27:09.022813+0000","last_active":"2024-03-11T21:27:16.820394+0000","last_peered":"2024-03-11T21:27:16.820394+0000","last_clean":"2024-03-11T21:27:16.820394+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:27:16.820394+0000","last_undegraded":"2024-03-11T21:27:16.820394+0000","last_fullsized":"2024-03-11T21:27:16.820394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:27:09.022585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.022585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.995050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622688+0000","last_change":"2024-03-11T21:26:24.352686+0000","last_active":"2024-03-11T21:27:17.622688+0000","last_peered":"2024-03-11T21:27:17.622688+0000","last_clean":"2024-03-11T21:27:17.622688+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:27:17.622688+0000","last_undegraded":"2024-03-11T21:27:17.622688+0000","last_fullsized":"2024-03-11T21:27:17.622688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:26:24.352407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:24.352407+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.744324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":382,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:26.201421+0000","last_change":"2024-03-11T21:27:06.215128+0000","last_active":"2024-03-11T21:27:26.201421+0000","last_peered":"2024-03-11T21:27:26.201421+0000","last_clean":"2024-03-11T21:27:26.201421+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:27:26.201421+0000","last_undegraded":"2024-03-11T21:27:26.201421+0000","last_fullsized":"2024-03-11T21:27:26.201421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:27:06.214816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.214816+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.841750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.814725+0000","last_change":"2024-03-11T21:26:59.951453+0000","last_active":"2024-03-11T21:27:16.814725+0000","last_peered":"2024-03-11T21:27:16.814725+0000","last_clean":"2024-03-11T21:27:16.814725+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:27:16.814725+0000","last_undegraded":"2024-03-11T21:27:16.814725+0000","last_fullsized":"2024-03-11T21:27:16.814725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.951239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.951239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.079618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":123,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.590588+0000","last_change":"2024-03-11T21:26:55.658685+0000","last_active":"2024-03-11T21:27:17.590588+0000","last_peered":"2024-03-11T21:27:17.590588+0000","last_clean":"2024-03-11T21:27:17.590588+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:27:17.590588+0000","last_undegraded":"2024-03-11T21:27:17.590588+0000","last_fullsized":"2024-03-11T21:27:17.590588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:26:55.658468+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.658468+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.668238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":142,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.707705+0000","last_change":"2024-03-11T21:27:12.894010+0000","last_active":"2024-03-11T21:27:17.707705+0000","last_peered":"2024-03-11T21:27:17.707705+0000","last_clean":"2024-03-11T21:27:17.707705+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:27:17.707705+0000","last_undegraded":"2024-03-11T21:27:17.707705+0000","last_fullsized":"2024-03-11T21:27:17.707705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.893784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.231554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":147,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.208805+0000","last_change":"2024-03-11T21:26:32.529005+0000","last_active":"2024-03-11T21:27:17.208805+0000","last_peered":"2024-03-11T21:27:17.208805+0000","last_clean":"2024-03-11T21:27:17.208805+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:27:17.208805+0000","last_undegraded":"2024-03-11T21:27:17.208805+0000","last_fullsized":"2024-03-11T21:27:17.208805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:32.528639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:32.528639+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:34.761031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:19.913153+0000","last_change":"2024-03-11T21:27:19.913202+0000","last_active":"2024-03-11T21:27:19.913153+0000","last_peered":"2024-03-11T21:27:19.913153+0000","last_clean":"2024-03-11T21:27:19.913153+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:27:19.913153+0000","last_undegraded":"2024-03-11T21:27:19.913153+0000","last_fullsized":"2024-03-11T21:27:19.913153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.913032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.913032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.839406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.197331+0000","last_change":"2024-03-11T21:26:52.285556+0000","last_active":"2024-03-11T21:27:18.197331+0000","last_peered":"2024-03-11T21:27:18.197331+0000","last_clean":"2024-03-11T21:27:18.197331+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:27:18.197331+0000","last_undegraded":"2024-03-11T21:27:18.197331+0000","last_fullsized":"2024-03-11T21:27:18.197331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:52.285152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.285152+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.725074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622060+0000","last_change":"2024-03-11T21:26:14.197969+0000","last_active":"2024-03-11T21:27:17.622060+0000","last_peered":"2024-03-11T21:27:17.622060+0000","last_clean":"2024-03-11T21:27:17.622060+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:27:17.622060+0000","last_undegraded":"2024-03-11T21:27:17.622060+0000","last_fullsized":"2024-03-11T21:27:17.622060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.197767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.197767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:40.681392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.745410+0000","last_change":"2024-03-11T21:27:03.708920+0000","last_active":"2024-03-11T21:27:17.745410+0000","last_peered":"2024-03-11T21:27:17.745410+0000","last_clean":"2024-03-11T21:27:17.745410+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:27:17.745410+0000","last_undegraded":"2024-03-11T21:27:17.745410+0000","last_fullsized":"2024-03-11T21:27:17.745410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.708595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.708595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.018137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:25.634019+0000","last_change":"2024-03-11T21:27:25.634056+0000","last_active":"2024-03-11T21:27:25.634019+0000","last_peered":"2024-03-11T21:27:25.634019+0000","last_clean":"2024-03-11T21:27:25.634019+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:27:25.634019+0000","last_undegraded":"2024-03-11T21:27:25.634019+0000","last_fullsized":"2024-03-11T21:27:25.634019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.633909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.849575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.814809+0000","last_change":"2024-03-11T21:26:17.927107+0000","last_active":"2024-03-11T21:27:16.814809+0000","last_peered":"2024-03-11T21:27:16.814809+0000","last_clean":"2024-03-11T21:27:16.814809+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:27:16.814809+0000","last_undegraded":"2024-03-11T21:27:16.814809+0000","last_fullsized":"2024-03-11T21:27:16.814809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:26:17.926863+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:26:17.926863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.586229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.742889+0000","last_change":"2024-03-11T21:27:08.792122+0000","last_active":"2024-03-11T21:27:17.742889+0000","last_peered":"2024-03-11T21:27:17.742889+0000","last_clean":"2024-03-11T21:27:17.742889+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:27:17.742889+0000","last_undegraded":"2024-03-11T21:27:17.742889+0000","last_fullsized":"2024-03-11T21:27:17.742889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.791885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.791885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.094147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":187,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:21.840085+0000","last_change":"2024-03-11T21:27:21.840085+0000","last_active":"2024-03-11T21:27:21.840085+0000","last_peered":"2024-03-11T21:27:21.840085+0000","last_clean":"2024-03-11T21:27:21.840085+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:27:21.840085+0000","last_undegraded":"2024-03-11T21:27:21.840085+0000","last_fullsized":"2024-03-11T21:27:21.840085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:21.839970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.839970+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.941185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819682+0000","last_change":"2024-03-11T21:26:41.718271+0000","last_active":"2024-03-11T21:27:16.819682+0000","last_peered":"2024-03-11T21:27:16.819682+0000","last_clean":"2024-03-11T21:27:16.819682+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:27:16.819682+0000","last_undegraded":"2024-03-11T21:27:16.819682+0000","last_fullsized":"2024-03-11T21:27:16.819682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.718071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.718071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:44.881942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":154,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.621620+0000","last_change":"2024-03-11T21:27:12.163374+0000","last_active":"2024-03-11T21:27:17.621620+0000","last_peered":"2024-03-11T21:27:17.621620+0000","last_clean":"2024-03-11T21:27:17.621620+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:27:17.621620+0000","last_undegraded":"2024-03-11T21:27:17.621620+0000","last_fullsized":"2024-03-11T21:27:17.621620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:27:12.163097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.163097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.333821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:27.825954+0000","last_change":"2024-03-11T21:27:27.825989+0000","last_active":"2024-03-11T21:27:27.825954+0000","last_peered":"2024-03-11T21:27:27.825954+0000","last_clean":"2024-03-11T21:27:27.825954+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:27:27.825954+0000","last_undegraded":"2024-03-11T21:27:27.825954+0000","last_fullsized":"2024-03-11T21:27:27.825954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:27.825842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.825842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.417136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:18.197217+0000","last_change":"2024-03-11T21:26:49.911665+0000","last_active":"2024-03-11T21:27:18.197217+0000","last_peered":"2024-03-11T21:27:18.197217+0000","last_clean":"2024-03-11T21:27:18.197217+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:27:18.197217+0000","last_undegraded":"2024-03-11T21:27:18.197217+0000","last_fullsized":"2024-03-11T21:27:18.197217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:26:49.911420+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.911420+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.431100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819715+0000","last_change":"2024-03-11T21:26:07.649240+0000","last_active":"2024-03-11T21:27:16.819715+0000","last_peered":"2024-03-11T21:27:16.819715+0000","last_clean":"2024-03-11T21:27:16.819715+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:27:16.819715+0000","last_undegraded":"2024-03-11T21:27:16.819715+0000","last_fullsized":"2024-03-11T21:27:16.819715+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:07.649133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:26:07.649133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.622040+0000","last_change":"2024-03-11T21:26:48.278251+0000","last_active":"2024-03-11T21:27:17.622040+0000","last_peered":"2024-03-11T21:27:17.622040+0000","last_clean":"2024-03-11T21:27:17.622040+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:27:17.622040+0000","last_undegraded":"2024-03-11T21:27:17.622040+0000","last_fullsized":"2024-03-11T21:27:17.622040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:48.277915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.277915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.573108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820369+0000","last_change":"2024-03-11T21:27:07.817089+0000","last_active":"2024-03-11T21:27:16.820369+0000","last_peered":"2024-03-11T21:27:16.820369+0000","last_clean":"2024-03-11T21:27:16.820369+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:27:16.820369+0000","last_undegraded":"2024-03-11T21:27:16.820369+0000","last_fullsized":"2024-03-11T21:27:16.820369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.816760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.816760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.238678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.820364+0000","last_change":"2024-03-11T21:26:12.573500+0000","last_active":"2024-03-11T21:27:16.820364+0000","last_peered":"2024-03-11T21:27:16.820364+0000","last_clean":"2024-03-11T21:27:16.820364+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:27:16.820364+0000","last_undegraded":"2024-03-11T21:27:16.820364+0000","last_fullsized":"2024-03-11T21:27:16.820364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:12.573292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:12.573292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.591915+0000","last_change":"2024-03-11T21:26:46.780598+0000","last_active":"2024-03-11T21:27:17.591915+0000","last_peered":"2024-03-11T21:27:17.591915+0000","last_clean":"2024-03-11T21:27:17.591915+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:27:17.591915+0000","last_undegraded":"2024-03-11T21:27:17.591915+0000","last_fullsized":"2024-03-11T21:27:17.591915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.780383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.780383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.322054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":169,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.706261+0000","last_change":"2024-03-11T21:27:06.797303+0000","last_active":"2024-03-11T21:27:17.706261+0000","last_peered":"2024-03-11T21:27:17.706261+0000","last_clean":"2024-03-11T21:27:17.706261+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:27:17.706261+0000","last_undegraded":"2024-03-11T21:27:17.706261+0000","last_fullsized":"2024-03-11T21:27:17.706261+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:27:06.797049+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.797049+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.836292+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:26.598208+0000","last_change":"2024-03-11T21:27:26.598269+0000","last_active":"2024-03-11T21:27:26.598208+0000","last_peered":"2024-03-11T21:27:26.598208+0000","last_clean":"2024-03-11T21:27:26.598208+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:27:26.598208+0000","last_undegraded":"2024-03-11T21:27:26.598208+0000","last_fullsized":"2024-03-11T21:27:26.598208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:26.598056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:26.598056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:47.232807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.818448+0000","last_change":"2024-03-11T21:26:38.936837+0000","last_active":"2024-03-11T21:27:16.818448+0000","last_peered":"2024-03-11T21:27:16.818448+0000","last_clean":"2024-03-11T21:27:16.818448+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:27:16.818448+0000","last_undegraded":"2024-03-11T21:27:16.818448+0000","last_fullsized":"2024-03-11T21:27:16.818448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.936573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.936573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.223059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.818628+0000","last_change":"2024-03-11T21:26:59.822204+0000","last_active":"2024-03-11T21:27:16.818628+0000","last_peered":"2024-03-11T21:27:16.818628+0000","last_clean":"2024-03-11T21:27:16.818628+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:27:16.818628+0000","last_undegraded":"2024-03-11T21:27:16.818628+0000","last_fullsized":"2024-03-11T21:27:16.818628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.821996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.821996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.831368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.592670+0000","last_change":"2024-03-11T21:26:14.127753+0000","last_active":"2024-03-11T21:27:17.592670+0000","last_peered":"2024-03-11T21:27:17.592670+0000","last_clean":"2024-03-11T21:27:17.592670+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:27:17.592670+0000","last_undegraded":"2024-03-11T21:27:17.592670+0000","last_fullsized":"2024-03-11T21:27:17.592670+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:26:14.127405+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.127405+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:31.505662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.206449+0000","last_change":"2024-03-11T21:26:15.354895+0000","last_active":"2024-03-11T21:27:17.206449+0000","last_peered":"2024-03-11T21:27:17.206449+0000","last_clean":"2024-03-11T21:27:17.206449+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:27:17.206449+0000","last_undegraded":"2024-03-11T21:27:17.206449+0000","last_fullsized":"2024-03-11T21:27:17.206449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:15.354676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:26:15.354676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:22.022398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":125,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.818665+0000","last_change":"2024-03-11T21:26:41.906852+0000","last_active":"2024-03-11T21:27:16.818665+0000","last_peered":"2024-03-11T21:27:16.818665+0000","last_clean":"2024-03-11T21:27:16.818665+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:27:16.818665+0000","last_undegraded":"2024-03-11T21:27:16.818665+0000","last_fullsized":"2024-03-11T21:27:16.818665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.906678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.906678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.797651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.821095+0000","last_change":"2024-03-11T21:27:00.897454+0000","last_active":"2024-03-11T21:27:16.821095+0000","last_peered":"2024-03-11T21:27:16.821095+0000","last_clean":"2024-03-11T21:27:16.821095+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:27:16.821095+0000","last_undegraded":"2024-03-11T21:27:16.821095+0000","last_fullsized":"2024-03-11T21:27:16.821095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.897238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.897238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.879305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":143,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:20.912602+0000","last_change":"2024-03-11T21:27:20.912735+0000","last_active":"2024-03-11T21:27:20.912602+0000","last_peered":"2024-03-11T21:27:20.912602+0000","last_clean":"2024-03-11T21:27:20.912602+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:27:20.912602+0000","last_undegraded":"2024-03-11T21:27:20.912602+0000","last_fullsized":"2024-03-11T21:27:20.912602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:20.912377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.912377+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.495526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.743849+0000","last_change":"2024-03-11T21:26:05.391058+0000","last_active":"2024-03-11T21:27:17.743849+0000","last_peered":"2024-03-11T21:27:17.743849+0000","last_clean":"2024-03-11T21:27:17.743849+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:27:17.743849+0000","last_undegraded":"2024-03-11T21:27:17.743849+0000","last_fullsized":"2024-03-11T21:27:17.743849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:05.390850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:05.390850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:17.856109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:16.819681+0000","last_change":"2024-03-11T21:26:57.835977+0000","last_active":"2024-03-11T21:27:16.819681+0000","last_peered":"2024-03-11T21:27:16.819681+0000","last_clean":"2024-03-11T21:27:16.819681+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:27:16.819681+0000","last_undegraded":"2024-03-11T21:27:16.819681+0000","last_fullsized":"2024-03-11T21:27:16.819681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.835726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.835726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.590001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:17.593175+0000","last_change":"2024-03-11T21:26:31.873778+0000","last_active":"2024-03-11T21:27:17.593175+0000","last_peered":"2024-03-11T21:27:17.593175+0000","last_clean":"2024-03-11T21:27:17.593175+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:27:17.593175+0000","last_undegraded":"2024-03-11T21:27:17.593175+0000","last_fullsized":"2024-03-11T21:27:17.593175+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:31.873666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.873666+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:35.702452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":180,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-11T21:27:23.196721+0000","last_change":"2024-03-11T21:26:29.009973+0000","last_active":"2024-03-11T21:27:23.196721+0000","last_peered":"2024-03-11T21:27:23.196721+0000","last_clean":"2024-03-11T21:27:23.196721+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:27:23.196721+0000","last_undegraded":"2024-03-11T21:27:23.196721+0000","last_fullsized":"2024-03-11T21:27:23.196721+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:29.009718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.009718+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.739503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841600,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":1028261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168649,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1396736,"data_stored":1026041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":435555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738504,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36712,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708440,"statfs":{"total":95995035648,"available":95957442560,"internally_reserved":0,"allocated":1409024,"data_stored":1027158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378739,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":436133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":794624,"data_stored":435611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855453,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":437080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495699,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":435593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:35.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-11T21:27:35.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:35.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:35.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.782 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:35.809 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:35.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:36.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:36.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:35 smithi143 bash[21512]: cluster 2024-03-11T21:27:35.493685+0000 osd.4 (osd.4) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:36.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:35 smithi143 bash[21512]: cluster 2024-03-11T21:27:35.493765+0000 osd.4 (osd.4) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:35 smithi143 bash[29284]: cluster 2024-03-11T21:27:35.493685+0000 osd.4 (osd.4) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:35 smithi143 bash[29284]: cluster 2024-03-11T21:27:35.493765+0000 osd.4 (osd.4) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:36.069 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:36.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:36.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:35 smithi203 bash[23654]: cluster 2024-03-11T21:27:35.493685+0000 osd.4 (osd.4) 101 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:36.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:35 smithi203 bash[23654]: cluster 2024-03-11T21:27:35.493765+0000 osd.4 (osd.4) 102 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:36.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:36.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:36.329 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:36.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:36.459 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:36.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:36.590 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:36.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:27:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:36.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:36.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:36.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:36.853 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:36.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:36.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:37.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:37.117 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:37.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:36 smithi143 bash[29284]: cluster 2024-03-11T21:27:35.144205+0000 mgr.y (mgr.14550) 171 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:37.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:36 smithi143 bash[29284]: cluster 2024-03-11T21:27:35.958043+0000 mon.a (mon.0) 892 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:27:37.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:36 smithi143 bash[21512]: cluster 2024-03-11T21:27:35.144205+0000 mgr.y (mgr.14550) 171 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:37.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:36 smithi143 bash[21512]: cluster 2024-03-11T21:27:35.958043+0000 mon.a (mon.0) 892 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:27:37.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:37.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:36 smithi203 bash[23654]: cluster 2024-03-11T21:27:35.144205+0000 mgr.y (mgr.14550) 171 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:37.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:36 smithi203 bash[23654]: cluster 2024-03-11T21:27:35.958043+0000 mon.a (mon.0) 892 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:27:37.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:27:37.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.406 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:37.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:37.537 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:37.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:37.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:37.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:37.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:37.927 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:38.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:38.058 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:38.161 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:38.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:38.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.321 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:38.424 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:38.445 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:38.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:38.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:38.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:38.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:38.842 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:38.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:38.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:38 smithi143 bash[21512]: cluster 2024-03-11T21:27:37.144757+0000 mgr.y (mgr.14550) 172 : cluster 0 pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:38.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:38 smithi143 bash[29284]: cluster 2024-03-11T21:27:37.144757+0000 mgr.y (mgr.14550) 172 : cluster 0 pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:38.972 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:39.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:38 smithi203 bash[23654]: cluster 2024-03-11T21:27:37.144757+0000 mgr.y (mgr.14550) 172 : cluster 0 pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:39.103 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:39.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.233 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.364 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:39.390 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:39.390 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:39.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.489 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.619 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:39.722 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.750 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:39.880 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:39.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:39 smithi203 bash[23654]: audit 2024-03-11T21:27:38.255909+0000 mgr.y (mgr.14550) 173 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:39.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:39.993 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:39 smithi143 bash[21512]: audit 2024-03-11T21:27:38.255909+0000 mgr.y (mgr.14550) 173 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:39.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:39 smithi143 bash[29284]: audit 2024-03-11T21:27:38.255909+0000 mgr.y (mgr.14550) 173 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:40.008 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:40.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:40.137 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:40.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:40.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:40.397 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:40.401 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:40.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:40.527 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:40.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:40.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:40.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:40.919 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:41.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:41.049 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:41.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:41.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:41.242 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:41 smithi143 bash[29284]: cluster 2024-03-11T21:27:39.146268+0000 mgr.y (mgr.14550) 174 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:41.242 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:41 smithi143 bash[21512]: cluster 2024-03-11T21:27:39.146268+0000 mgr.y (mgr.14550) 174 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:40 smithi203 bash[23654]: cluster 2024-03-11T21:27:39.146268+0000 mgr.y (mgr.14550) 174 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:41.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:41.306 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:41.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:41.420 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:41.434 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:41.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:41.566 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:41.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:41.697 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:41.800 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:41.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:41.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:41.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:42.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:42.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:42.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.213 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:42.239 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:42.316 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:42.345 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:42.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:42.475 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:42.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.605 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:42.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.731 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:42.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:42.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:42.991 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:43.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:43.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:43.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.232 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:42 smithi203 bash[23654]: cluster 2024-03-11T21:27:41.146989+0000 mgr.y (mgr.14550) 175 : cluster 0 pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.232 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:42 smithi203 bash[23654]: audit 2024-03-11T21:27:42.874727+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:43.232 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:42 smithi203 bash[23654]: audit 2024-03-11T21:27:42.876283+0000 mon.c (mon.2) 94 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:43.248 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:43.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.380 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:43.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[21512]: cluster 2024-03-11T21:27:41.146989+0000 mgr.y (mgr.14550) 175 : cluster 0 pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[21512]: audit 2024-03-11T21:27:42.874727+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:43.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[21512]: audit 2024-03-11T21:27:42.876283+0000 mon.c (mon.2) 94 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:43.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[29284]: cluster 2024-03-11T21:27:41.146989+0000 mgr.y (mgr.14550) 175 : cluster 0 pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[29284]: audit 2024-03-11T21:27:42.874727+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:27:43.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:42 smithi143 bash[29284]: audit 2024-03-11T21:27:42.876283+0000 mon.c (mon.2) 94 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:43.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:43.510 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:43.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:43.635 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:43.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:43.766 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:43.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:43.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:43.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:43 smithi203 bash[23654]: cluster 2024-03-11T21:27:42.800190+0000 osd.6 (osd.6) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:44.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:43 smithi203 bash[23654]: cluster 2024-03-11T21:27:42.800298+0000 osd.6 (osd.6) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:44.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:43 smithi203 bash[23654]: cluster 2024-03-11T21:27:42.983668+0000 mon.a (mon.0) 894 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:27:44.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:44.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:44.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:44.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:44.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:44 smithi143 bash[29284]: cluster 2024-03-11T21:27:42.800190+0000 osd.6 (osd.6) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:44.266 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:44 smithi143 bash[29284]: cluster 2024-03-11T21:27:42.800298+0000 osd.6 (osd.6) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:44.266 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:44 smithi143 bash[29284]: cluster 2024-03-11T21:27:42.983668+0000 mon.a (mon.0) 894 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:27:44.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:43 smithi143 bash[21512]: cluster 2024-03-11T21:27:42.800190+0000 osd.6 (osd.6) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T21:27:44.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:43 smithi143 bash[21512]: cluster 2024-03-11T21:27:42.800298+0000 osd.6 (osd.6) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T21:27:44.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:43 smithi143 bash[21512]: cluster 2024-03-11T21:27:42.983668+0000 mon.a (mon.0) 894 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:27:44.278 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:44.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:44.407 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:44.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:44.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:44.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:44.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:44.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:44.790 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:44.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.921 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:45.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:45.050 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:45 smithi143 bash[21512]: cluster 2024-03-11T21:27:43.147897+0000 mgr.y (mgr.14550) 176 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:45.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:45.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:45 smithi143 bash[29284]: cluster 2024-03-11T21:27:43.147897+0000 mgr.y (mgr.14550) 176 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:45.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:45.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:44 smithi203 bash[23654]: cluster 2024-03-11T21:27:43.147897+0000 mgr.y (mgr.14550) 176 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:45.178 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:45.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:45.308 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:45.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:45.439 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:45.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:45.561 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:45.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:45.692 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:45.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:45.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:45.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:45.944 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:46.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:46.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:46.183 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:46 smithi143 bash[29284]: cluster 2024-03-11T21:27:45.148495+0000 mgr.y (mgr.14550) 177 : cluster 0 pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:46.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:46 smithi143 bash[21512]: cluster 2024-03-11T21:27:45.148495+0000 mgr.y (mgr.14550) 177 : cluster 0 pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:46.199 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:46.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:46 smithi203 bash[23654]: cluster 2024-03-11T21:27:45.148495+0000 mgr.y (mgr.14550) 177 : cluster 0 pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:46.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.333 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:46.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:46.456 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:46.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:46.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:46.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:46.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.839 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:46.943 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.969 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:47.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:47.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:47.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.345 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:47.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:47.475 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:47.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:47.601 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:47.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:47.732 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:47.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:47.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.982 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:48.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:48.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:48.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:48.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:48.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.364 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:48.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.477 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:48 smithi143 bash[29284]: cluster 2024-03-11T21:27:47.148910+0000 mgr.y (mgr.14550) 178 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:48.477 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:48 smithi143 bash[21512]: cluster 2024-03-11T21:27:47.148910+0000 mgr.y (mgr.14550) 178 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:48.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:48.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:48 smithi203 bash[23654]: cluster 2024-03-11T21:27:47.148910+0000 mgr.y (mgr.14550) 178 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:48.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:48.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:48.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:48.743 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:48.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:48.865 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:48.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:48.994 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:49.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:49.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.243 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:49.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:49.374 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:49.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:49.504 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:49.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:49.618 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:49 smithi203 bash[23654]: audit 2024-03-11T21:27:48.261815+0000 mgr.y (mgr.14550) 179 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:49.634 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:49.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.747 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:49 smithi143 bash[29284]: audit 2024-03-11T21:27:48.261815+0000 mgr.y (mgr.14550) 179 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:49.748 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:49 smithi143 bash[21512]: audit 2024-03-11T21:27:48.261815+0000 mgr.y (mgr.14550) 179 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:49.760 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:49.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:49.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:50.019 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:50.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:50.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:50.375 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:50.397 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:50.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:50.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:50.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:50 smithi143 bash[21512]: cluster 2024-03-11T21:27:49.149577+0000 mgr.y (mgr.14550) 180 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:50.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:50 smithi143 bash[29284]: cluster 2024-03-11T21:27:49.149577+0000 mgr.y (mgr.14550) 180 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:50.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:50.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:50.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:50.893 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:50 smithi203 bash[23654]: cluster 2024-03-11T21:27:49.149577+0000 mgr.y (mgr.14550) 180 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:50.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:51.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:51.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:51.160 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:51.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:51.291 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:51.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.423 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:51.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:51.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:51.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.683 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:51.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.809 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:51.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:51.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:52.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:52.067 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:52.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.183 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:27:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:27:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:52.198 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:52.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:52.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:52.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:52.561 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:52.585 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:52.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:52.715 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:52.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:52.829 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:52 smithi143 bash[21512]: cluster 2024-03-11T21:27:51.150005+0000 mgr.y (mgr.14550) 181 : cluster 0 pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:52.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:52 smithi143 bash[29284]: cluster 2024-03-11T21:27:51.150005+0000 mgr.y (mgr.14550) 181 : cluster 0 pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:52.843 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:52.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:52.971 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:52 smithi203 bash[23654]: cluster 2024-03-11T21:27:51.150005+0000 mgr.y (mgr.14550) 181 : cluster 0 pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:53.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:53.102 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:53.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:53.232 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:53.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:53.354 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:53.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:53.485 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:53.588 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:53.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:53.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:53.742 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:53.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:53.866 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:53.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:53.996 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:54.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:54.126 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:54.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:54.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:54.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.381 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:54.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.512 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:54.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.638 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:54.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:54 smithi143 bash[29284]: cluster 2024-03-11T21:27:53.151165+0000 mgr.y (mgr.14550) 182 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:27:54.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:54 smithi143 bash[21512]: cluster 2024-03-11T21:27:53.151165+0000 mgr.y (mgr.14550) 182 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:27:54.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:54.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:54 smithi203 bash[23654]: cluster 2024-03-11T21:27:53.151165+0000 mgr.y (mgr.14550) 182 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:27:54.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:54.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.019 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:55.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:55.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:55.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:55.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:55.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:55.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:55.530 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:55.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:55.655 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:55.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:55.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:55.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:56.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:56.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:56.079 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:27:56.079 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:27:56.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:56.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:56.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.287 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:56.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:56.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:56.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:56.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:56.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 -- 172.21.15.143:0/2045044641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508100e60 msgr2=0x7fd50810d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:56.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/2045044641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508100e60 0x7fd50810d320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd50000b0a0 tx=0x7fd50000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 -- 172.21.15.143:0/2045044641 shutdown_connections 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/2045044641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508100e60 0x7fd50810d320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/2045044641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd508106550 0x7fd508106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/2045044641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5081004a0 0x7fd508100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 -- 172.21.15.143:0/2045044641 >> 172.21.15.143:0/2045044641 conn(0x7fd5080772e0 msgr2=0x7fd5080fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 -- 172.21.15.143:0/2045044641 shutdown_connections 2024-03-11T21:27:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.560+0000 7fd50f1d5700 1 -- 172.21.15.143:0/2045044641 wait complete. 2024-03-11T21:27:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:56.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 librados: starting msgr at 2024-03-11T21:27:56.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 librados: starting objecter 2024-03-11T21:27:56.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 Processor -- start 2024-03-11T21:27:56.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- start start 2024-03-11T21:27:56.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 librados: setting wanted keys 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 librados: calling monclient init 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5081004a0 0x7fd508102110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 0x7fd508102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081105c0 con 0x7fd508100e60 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508110440 con 0x7fd5081004a0 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd508110740 con 0x7fd508106550 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50d772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 0x7fd508102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50d772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60890/0 (socket says 172.21.15.143:60890) 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50d772700 1 -- 172.21.15.143:0/1286804625 learned_addr learned my addr 172.21.15.143:0/1286804625 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 -- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 msgr2=0x7fd508102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 -- 172.21.15.143:0/1286804625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5081004a0 msgr2=0x7fd508102110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50cf71700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5081004a0 0x7fd508102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5081004a0 0x7fd508102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 -- 172.21.15.143:0/1286804625 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5081a5fb0 con 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50d772700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd507fff700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 0x7fd508102650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400e9f0 tx=0x7fd4f400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 10 client.?.objecter ms_handle_connect 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400cbd0 con 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5081a62a0 con 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd4f400cd70 con 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4018680 con 0x7fd508100e60 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:27:56.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5081a6830 con 0x7fd508100e60 2024-03-11T21:27:56.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 librados: init done 2024-03-11T21:27:56.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50804faa0 con 0x7fd508100e60 2024-03-11T21:27:56.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.564+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd4f4010430 con 0x7fd508100e60 2024-03-11T21:27:56.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 0x7fd4f8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:56.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(129..129 src has 77..129) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fd4f408e5c0 con 0x7fd508100e60 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 10 client.14787.objecter ms_dispatch 0x7fd508001550 osd_map(129..129 src has 77..129) v4 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 3 client.14787.objecter handle_osd_map got epochs [129,129] > 0 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 3 client.14787.objecter handle_osd_map decoding full epoch 129 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd50cf71700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 0x7fd4f8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 20 client.14787.objecter dump_active .. 0 homeless 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd50cf71700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 0x7fd4f8071ce0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc005950 tx=0x7fd4fc005c70 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:56.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.568+0000 7fd505ffb700 10 client.14787.objecter ms_handle_connect 0x7fd4f806f820 2024-03-11T21:27:56.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.572+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd4f405bc40 con 0x7fd508100e60 2024-03-11T21:27:56.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:56.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:56.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:56 smithi143 bash[29284]: cluster 2024-03-11T21:27:55.151814+0000 mgr.y (mgr.14550) 183 : cluster 0 pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:56.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:56 smithi143 bash[21512]: cluster 2024-03-11T21:27:55.151814+0000 mgr.y (mgr.14550) 183 : cluster 0 pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:56.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:56.780+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7fd508103640 con 0x7fd508100e60 2024-03-11T21:27:56.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:56.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:56.923 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:57.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:56 smithi203 bash[23654]: cluster 2024-03-11T21:27:55.151814+0000 mgr.y (mgr.14550) 183 : cluster 0 pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:57.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:57.049 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:57.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:57.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:57.294 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:57.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:57.418 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:57.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.540 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:57.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.627+0000 7fd505ffb700 1 -- 172.21.15.143:0/1286804625 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v130) v1 ==== 129+0+0 (secure 0 0 0) 0x7fd508103640 con 0x7fd508100e60 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 10 librados: watch_flush enter 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 10 librados: watch_flush exit 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 20 client.14787.objecter shutdown clearing up homeless session... 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 10 client.14787.objecter successfully canceled tick 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 msgr2=0x7fd4f8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 0x7fd4f8071ce0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc005950 tx=0x7fd4fc005c70 comp rx=0 tx=0).stop 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 msgr2=0x7fd508102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 0x7fd508102650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400e9f0 tx=0x7fd4f400ed00 comp rx=0 tx=0).stop 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 shutdown_connections 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd4f806f820 0x7fd4f8071ce0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:57.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd508106550 0x7fd508102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5081004a0 0x7fd508102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 --2- 172.21.15.143:0/1286804625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd508100e60 0x7fd508102650 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.631+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 >> 172.21.15.143:0/1286804625 conn(0x7fd5080772e0 msgr2=0x7fd5080ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.635+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 shutdown_connections 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.635+0000 7fd50f1d5700 1 -- 172.21.15.143:0/1286804625 wait complete. 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:27:57.635+0000 7fd50f1d5700 1 librados: shutdown 2024-03-11T21:27:57.612 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pgp_num to 1 2024-03-11T21:27:57.643 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:57.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:57.790 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:57.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:57 smithi143 bash[29284]: audit 2024-03-11T21:27:56.787671+0000 mon.a (mon.0) 895 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:27:57.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:57 smithi143 bash[21512]: audit 2024-03-11T21:27:56.787671+0000 mon.a (mon.0) 895 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:27:57.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:58.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:57 smithi203 bash[23654]: audit 2024-03-11T21:27:56.787671+0000 mon.a (mon.0) 895 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:27:58.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:58.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:58.168 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:58.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:58.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:58.417 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:58.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:58.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:27:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:27:58.547 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:58.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[29284]: cluster 2024-03-11T21:27:57.152248+0000 mgr.y (mgr.14550) 184 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[29284]: audit 2024-03-11T21:27:57.632368+0000 mon.a (mon.0) 896 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[29284]: cluster 2024-03-11T21:27:57.636722+0000 mon.a (mon.0) 897 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[29284]: audit 2024-03-11T21:27:57.891490+0000 mon.c (mon.2) 95 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[21512]: cluster 2024-03-11T21:27:57.152248+0000 mgr.y (mgr.14550) 184 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[21512]: audit 2024-03-11T21:27:57.632368+0000 mon.a (mon.0) 896 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[21512]: cluster 2024-03-11T21:27:57.636722+0000 mon.a (mon.0) 897 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:27:58.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:58 smithi143 bash[21512]: audit 2024-03-11T21:27:57.891490+0000 mon.c (mon.2) 95 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:58.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:58.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:58.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:58.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:58.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:58 smithi203 bash[23654]: cluster 2024-03-11T21:27:57.152248+0000 mgr.y (mgr.14550) 184 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:58.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:58 smithi203 bash[23654]: audit 2024-03-11T21:27:57.632368+0000 mon.a (mon.0) 896 : audit 1 from='client.? 172.21.15.143:0/1286804625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:27:58.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:58 smithi203 bash[23654]: cluster 2024-03-11T21:27:57.636722+0000 mon.a (mon.0) 897 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:27:58.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:58 smithi203 bash[23654]: audit 2024-03-11T21:27:57.891490+0000 mon.c (mon.2) 95 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:58.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:59.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:59.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.183 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:59.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.304 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:59.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.435 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:59.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:59.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:59.610 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:27:59 smithi143 bash[21512]: audit 2024-03-11T21:27:58.270857+0000 mgr.y (mgr.14550) 185 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:59.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:59.691 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:27:59 smithi203 bash[23654]: audit 2024-03-11T21:27:58.270857+0000 mgr.y (mgr.14550) 185 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:59.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:59.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:27:59 smithi143 bash[29284]: audit 2024-03-11T21:27:58.270857+0000 mgr.y (mgr.14550) 185 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:59.951 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:00.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:00.082 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:00.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:28:00.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.211 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:00.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:00.340 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:00.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:00.464 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:00.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:00.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 -- 172.21.15.143:0/1930445808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30100fc0 msgr2=0x7f9c301013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 --2- 172.21.15.143:0/1930445808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30100fc0 0x7f9c301013a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0099c0 tx=0x7f9c2c009cd0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 -- 172.21.15.143:0/1930445808 shutdown_connections 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 --2- 172.21.15.143:0/1930445808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30100fc0 0x7f9c301013a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 --2- 172.21.15.143:0/1930445808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 0x7f9c3010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 --2- 172.21.15.143:0/1930445808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c30101970 0x7f9c301059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 -- 172.21.15.143:0/1930445808 >> 172.21.15.143:0/1930445808 conn(0x7f9c30077de0 msgr2=0x7f9c300781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 -- 172.21.15.143:0/1930445808 shutdown_connections 2024-03-11T21:28:00.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.707+0000 7f9c3757e700 1 -- 172.21.15.143:0/1930445808 wait complete. 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 librados: starting msgr at 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 librados: starting objecter 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 Processor -- start 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- start start 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 librados: setting wanted keys 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 librados: calling monclient init 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c30100fc0 0x7f9c3019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:55528/0 (socket says 172.21.15.143:55528) 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 -- 172.21.15.143:0/906737842 learned_addr learned my addr 172.21.15.143:0/906737842 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 0x7f9c301969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30110590 con 0x7f9c30101970 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30110410 con 0x7f9c30100fc0 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30110710 con 0x7f9c301060a0 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c30100fc0 0x7f9c3019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 0x7f9c301969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 msgr2=0x7f9c301969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 0x7f9c301969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c30100fc0 msgr2=0x7f9c3019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c30100fc0 0x7f9c3019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30197150 con 0x7f9c30101970 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c301060a0 0x7f9c301969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9c20007ee0 tx=0x7f9c20009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c267fc700 10 client.?.objecter ms_handle_connect 0x7f9c30101970 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c267fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20013030 con 0x7f9c30101970 2024-03-11T21:28:00.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9c20004cb0 con 0x7f9c30101970 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c30197440 con 0x7f9c30101970 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 10 client.14793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c200131f0 con 0x7f9c30101970 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3006a3a0 con 0x7f9c30101970 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.711+0000 7f9c3757e700 1 librados: init done 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c3757e700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9c301024a0 con 0x7f9c30101970 2024-03-11T21:28:00.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9c2000c070 con 0x7f9c30101970 2024-03-11T21:28:00.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 0x7f9c1c071c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(130..130 src has 77..130) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f9c2008d220 con 0x7f9c30101970 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 osd_map(130..130 src has 77..130) v4 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 3 client.14793.objecter handle_osd_map got epochs [130,130] > 0 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 3 client.14793.objecter handle_osd_map decoding full epoch 130 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 0x7f9c1c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 20 client.14793.objecter dump_active .. 0 homeless 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v130) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9c2005a8a0 con 0x7f9c30101970 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 1 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c10002ff0 0x7f9c100054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 1 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 0 1 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 4 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 10 client.14793.objecter _send_command 1 2024-03-11T21:28:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.715+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c10008530 con 0x7f9c10002ff0 2024-03-11T21:28:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c10002ff0 0x7f9c100054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c10002ff0 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c10002ff0 0x7f9c100054b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 0x7f9c1c071c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c000c00 tx=0x7f9c2c005fb0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c1c06f740 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c10008530 con 0x7f9c10002ff0 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 10 client.14793.objecter _finish_command 1 = system:0 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 4 2024-03-11T21:28:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.719+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 0 1 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 2 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 2 2024-03-11T21:28:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c100028e0 osd=0 3 2024-03-11T21:28:00.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 0 2 2024-03-11T21:28:00.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 4 2024-03-11T21:28:00.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 10 client.14793.objecter _send_command 2 2024-03-11T21:28:00.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.735+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c10001240 con 0x7f9c10002ff0 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c10001240 con 0x7f9c10002ff0 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 10 client.14793.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 4 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 0 2 2024-03-11T21:28:00.716 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 3 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c10006cb0 0x7f9c10007160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 3 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 1 3 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 4 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 10 client.14793.objecter _send_command 3 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c100076a0 con 0x7f9c10006cb0 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c10006cb0 0x7f9c10007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c10006cb0 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c10006cb0 0x7f9c10007160 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c100076a0 con 0x7f9c10006cb0 2024-03-11T21:28:00.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:28:00.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 10 client.14793.objecter _finish_command 3 = system:0 2024-03-11T21:28:00.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 4 2024-03-11T21:28:00.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.739+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 1 3 2024-03-11T21:28:00.722 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 4 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 4 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10001f80 osd=1 3 2024-03-11T21:28:00.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 1 4 2024-03-11T21:28:00.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 4 2024-03-11T21:28:00.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 10 client.14793.objecter _send_command 4 2024-03-11T21:28:00.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.771+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c100080c0 con 0x7f9c10006cb0 2024-03-11T21:28:00.754 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:28:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c100080c0 con 0x7f9c10006cb0 2024-03-11T21:28:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 10 client.14793.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 4 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 1 4 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 5 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c1000acf0 0x7f9c1000d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 5 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 2 5 2024-03-11T21:28:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 4 2024-03-11T21:28:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 10 client.14793.objecter _send_command 5 2024-03-11T21:28:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1000ffd0 con 0x7f9c1000acf0 2024-03-11T21:28:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c1000acf0 0x7f9c1000d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c1000acf0 0x7f9c1000d0e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.775+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c1000acf0 2024-03-11T21:28:00.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.779+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c1000ffd0 con 0x7f9c1000acf0 2024-03-11T21:28:00.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.779+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:28:00.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.779+0000 7f9c267fc700 10 client.14793.objecter _finish_command 5 = system:0 2024-03-11T21:28:00.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.779+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 4 2024-03-11T21:28:00.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.779+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 2 5 2024-03-11T21:28:00.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 6 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 6 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10001d30 osd=2 3 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 2 6 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 4 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 10 client.14793.objecter _send_command 6 2024-03-11T21:28:00.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c10007b00 con 0x7f9c1000acf0 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c10007b00 con 0x7f9c1000acf0 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c267fc700 10 client.14793.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 4 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 2 6 2024-03-11T21:28:00.786 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 7 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c1000e7a0 0x7f9c1000ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 7 2024-03-11T21:28:00.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 3 7 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 4 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 10 client.14793.objecter _send_command 7 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1000f140 con 0x7f9c1000e7a0 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.807+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c1000e7a0 0x7f9c1000ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c1000e7a0 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c1000e7a0 0x7f9c1000ec00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c1000f140 con 0x7f9c1000e7a0 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 10 client.14793.objecter _finish_command 7 = system:0 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 4 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 3 7 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7fde605fc700 1 -- 172.21.15.143:0/3191711798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102120 msgr2=0x7fde58102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.811+0000 7fde605fc700 1 --2- 172.21.15.143:0/3191711798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102120 0x7fde58102500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fde540099c0 tx=0x7fde54009cd0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/3191711798 shutdown_connections 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- 172.21.15.143:0/3191711798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde5810ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- 172.21.15.143:0/3191711798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102120 0x7fde58102500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- 172.21.15.143:0/3191711798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102ad0 0x7fde58106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/3191711798 >> 172.21.15.143:0/3191711798 conn(0x7fde58078f40 msgr2=0x7fde58079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/3191711798 shutdown_connections 2024-03-11T21:28:00.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/3191711798 wait complete. 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 librados: starting msgr at 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 librados: starting objecter 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 Processor -- start 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- start start 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 librados: setting wanted keys 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 librados: calling monclient init 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102120 0x7fde580740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde58074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60940/0 (socket says 172.21.15.143:60940) 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5eb99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde58074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5eb99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde58074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:55554/0 (socket says 172.21.15.143:55554) 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 -- 172.21.15.143:0/937669175 learned_addr learned my addr 172.21.15.143:0/937669175 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde581116f0 con 0x7fde58107150 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde58111570 con 0x7fde58102120 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde58111870 con 0x7fde58102ad0 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5e398700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102120 0x7fde580740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 -- 172.21.15.143:0/937669175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102120 msgr2=0x7fde580740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102120 0x7fde580740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 -- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 msgr2=0x7fde58074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde58074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde581a61e0 con 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde5db97700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b760 tx=0x7fde4800ba70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde4f7fe700 10 client.?.objecter ms_handle_connect 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde4f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48015020 con 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fde48011b00 con 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48010af0 con 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde581a64d0 con 0x7fde58102ad0 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 10 client.24674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:00.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde581a69c0 con 0x7fde58102ad0 2024-03-11T21:28:00.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.815+0000 7fde605fc700 1 librados: init done 2024-03-11T21:28:00.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde40005120 con 0x7fde58102ad0 2024-03-11T21:28:00.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fde48018020 con 0x7fde58102ad0 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 0x7fde44071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(130..130 src has 77..130) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fde4808df70 con 0x7fde58102ad0 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 10 client.24674.objecter ms_dispatch 0x7fde58001550 osd_map(130..130 src has 77..130) v4 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 3 client.24674.objecter handle_osd_map got epochs [130,130] > 0 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 3 client.24674.objecter handle_osd_map decoding full epoch 130 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde5e398700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 0x7fde44071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 20 client.24674.objecter dump_active .. 0 homeless 2024-03-11T21:28:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde5e398700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 0x7fde44071ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde54009990 tx=0x7fde5400b810 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.819+0000 7fde4f7fe700 10 client.24674.objecter ms_handle_connect 0x7fde4406f820 2024-03-11T21:28:00.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.823+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fde4801f020 con 0x7fde58102ad0 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 8 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 8 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10006300 osd=3 3 2024-03-11T21:28:00.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 3 8 2024-03-11T21:28:00.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 4 2024-03-11T21:28:00.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 10 client.14793.objecter _send_command 8 2024-03-11T21:28:00.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c1000f140 con 0x7f9c1000e7a0 2024-03-11T21:28:00.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c1000f140 con 0x7f9c1000e7a0 2024-03-11T21:28:00.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c267fc700 10 client.14793.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 4 2024-03-11T21:28:00.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 3 8 2024-03-11T21:28:00.818 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 9 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c10012240 0x7f9c10014630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 9 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 4 9 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 4 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 10 client.14793.objecter _send_command 9 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1000f940 con 0x7f9c10012240 2024-03-11T21:28:00.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.839+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c10012240 0x7f9c10014630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c10012240 2024-03-11T21:28:00.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c10012240 0x7f9c10014630 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c1000f940 con 0x7f9c10012240 2024-03-11T21:28:00.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:28:00.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 10 client.14793.objecter _finish_command 9 = system:0 2024-03-11T21:28:00.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 4 2024-03-11T21:28:00.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.843+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 4 9 2024-03-11T21:28:00.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 10 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 10 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1000f140 osd=4 3 2024-03-11T21:28:00.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 4 10 2024-03-11T21:28:00.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 4 2024-03-11T21:28:00.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 10 client.14793.objecter _send_command 10 2024-03-11T21:28:00.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.875+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c10014b70 con 0x7f9c10012240 2024-03-11T21:28:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:00.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c10014b70 con 0x7f9c10012240 2024-03-11T21:28:00.855 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 10 client.14793.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 4 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 4 10 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 11 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c100151f0 0x7f9c100175e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 11 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 5 11 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 4 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 10 client.14793.objecter _send_command 11 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1001a580 con 0x7f9c100151f0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c100151f0 0x7f9c100175e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c34b19700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c100151f0 0x7f9c100175e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c100151f0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c1001a580 con 0x7f9c100151f0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 10 client.14793.objecter _finish_command 11 = system:0 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 4 2024-03-11T21:28:00.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.879+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 5 11 2024-03-11T21:28:00.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 12 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 12 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c10014b70 osd=5 3 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 5 12 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 4 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 10 client.14793.objecter _send_command 12 2024-03-11T21:28:00.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.895+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c1001a5e0 con 0x7f9c100151f0 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c1001a5e0 con 0x7f9c100151f0 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 10 client.14793.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 4 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 5 12 2024-03-11T21:28:00.876 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 13 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c10019930 0x7f9c10019d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 13 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 6 13 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 4 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 10 client.14793.objecter _send_command 13 2024-03-11T21:28:00.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1001d950 con 0x7f9c10019930 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c10019930 0x7f9c10019d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c10019930 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c35b1b700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c10019930 0x7f9c10019d70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c1001d950 con 0x7f9c10019930 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:28:00.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 10 client.14793.objecter _finish_command 13 = system:0 2024-03-11T21:28:00.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 4 2024-03-11T21:28:00.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.899+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 6 13 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 14 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 14 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1001a5e0 osd=6 3 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 6 14 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 4 2024-03-11T21:28:00.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 10 client.14793.objecter _send_command 14 2024-03-11T21:28:00.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.919+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c1001c7d0 con 0x7f9c10019930 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c1001c7d0 con 0x7f9c10019930 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 10 client.14793.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 4 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 6 14 2024-03-11T21:28:00.899 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 15 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c1001cef0 0x7f9c1001d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 15 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 7 15 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 4 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 10 client.14793.objecter _send_command 15 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c10020cc0 con 0x7f9c1001cef0 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c1001cef0 0x7f9c1001d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c3531a700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c1001cef0 0x7f9c1001d350 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 10 client.14793.objecter ms_handle_connect 0x7f9c1001cef0 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c10020cc0 con 0x7f9c1001cef0 2024-03-11T21:28:00.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:28:00.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 10 client.14793.objecter _finish_command 15 = system:0 2024-03-11T21:28:00.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 4 2024-03-11T21:28:00.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.923+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 7 15 2024-03-11T21:28:00.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 10 client.14793.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign -1 16 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 20 client.14793.objecter _get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_remove -1 16 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 20 client.14793.objecter get_session s=0x7f9c1001c7d0 osd=7 3 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 15 client.14793.objecter _session_command_op_assign 7 16 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 4 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 10 client.14793.objecter _send_command 16 2024-03-11T21:28:00.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c10017ba0 con 0x7f9c1001cef0 2024-03-11T21:28:00.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:00 smithi143 bash[29284]: cluster 2024-03-11T21:27:59.153062+0000 mgr.y (mgr.14550) 186 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:00.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:00 smithi143 bash[21512]: cluster 2024-03-11T21:27:59.153062+0000 mgr.y (mgr.14550) 186 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:00.935 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:28:00.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c267fc700 1 -- 172.21.15.143:0/906737842 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c10017ba0 con 0x7f9c1001cef0 2024-03-11T21:28:00.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c267fc700 10 client.14793.objecter ms_dispatch 0x7f9c30001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.955+0000 7f9c267fc700 10 client.14793.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c267fc700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 4 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c267fc700 15 client.14793.objecter _session_command_op_remove 7 16 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 librados: watch_flush enter 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 librados: watch_flush exit 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.0 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c10002ff0 msgr2=0x7f9c100054b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c10002ff0 0x7f9c100054b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c100028e0 osd=0 1 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.1 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c10006cb0 msgr2=0x7f9c10007160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c10006cb0 0x7f9c10007160 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001f80 osd=1 1 2024-03-11T21:28:00.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.2 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c1000acf0 msgr2=0x7f9c1000d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c1000acf0 0x7f9c1000d0e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10001d30 osd=2 1 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.3 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c1000e7a0 msgr2=0x7f9c1000ec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c1000e7a0 0x7f9c1000ec00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10006300 osd=3 1 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.4 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c10012240 msgr2=0x7f9c10014630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c10012240 0x7f9c10014630 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1000f140 osd=4 1 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.5 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c100151f0 msgr2=0x7f9c100175e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c100151f0 0x7f9c100175e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c10014b70 osd=5 1 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.6 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c10019930 msgr2=0x7f9c10019d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c10019930 0x7f9c10019d70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001a5e0 osd=6 1 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter close_session for osd.7 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c1001cef0 msgr2=0x7f9c1001d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c1001cef0 0x7f9c1001d350 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter put_session s=0x7f9c1001c7d0 osd=7 1 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 20 client.14793.objecter shutdown clearing up homeless session... 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c1bfff700 10 client.14793.objecter successfully canceled tick 2024-03-11T21:28:00.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c35b1b700 1 -- 172.21.15.143:0/906737842 reap_dead start 2024-03-11T21:28:00.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c35b1b700 1 -- 172.21.15.143:0/906737842 reap_dead start 2024-03-11T21:28:00.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c35b1b700 1 -- 172.21.15.143:0/906737842 reap_dead start 2024-03-11T21:28:00.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c35b1b700 1 -- 172.21.15.143:0/906737842 reap_dead start 2024-03-11T21:28:00.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.959+0000 7f9c35b1b700 1 -- 172.21.15.143:0/906737842 reap_dead start 2024-03-11T21:28:00.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 msgr2=0x7f9c1c071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 0x7f9c1c071c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c000c00 tx=0x7f9c2c005fb0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 msgr2=0x7f9c3019cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9c20007ee0 tx=0x7f9c20009520 comp rx=0 tx=0).stop 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 shutdown_connections 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c30101970 0x7f9c3019cd20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/906737842 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f740 0x7f9c1c071c00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 >> 172.21.15.143:0/906737842 conn(0x7f9c30077de0 msgr2=0x7f9c300feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 shutdown_connections 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 -- 172.21.15.143:0/906737842 wait complete. 2024-03-11T21:28:00.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:00.963+0000 7f9c1bfff700 1 librados: shutdown 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:00.941 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:00.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:00.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:00 smithi203 bash[23654]: cluster 2024-03-11T21:27:59.153062+0000 mgr.y (mgr.14550) 186 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:00.982 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:01.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:01.114 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:01.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.151+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fde400056e0 con 0x7fde58102ad0 2024-03-11T21:28:01.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:01.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:01.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.376 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:01.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.507 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:01.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.655+0000 7fde4f7fe700 1 -- 172.21.15.143:0/937669175 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v131) v1 ==== 97+0+0 (secure 0 0 0) 0x7fde4800d040 con 0x7fde58102ad0 2024-03-11T21:28:01.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 10 librados: watch_flush enter 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 10 librados: watch_flush exit 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 20 client.24674.objecter shutdown clearing up homeless session... 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 10 client.24674.objecter successfully canceled tick 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 msgr2=0x7fde44071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 0x7fde44071ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde54009990 tx=0x7fde5400b810 comp rx=0 tx=0).stop 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 msgr2=0x7fde58074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b760 tx=0x7fde4800ba70 comp rx=0 tx=0).stop 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 shutdown_connections 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde4406f820 0x7fde44071ce0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde58107150 0x7fde58074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde58102120 0x7fde580740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:01.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 --2- 172.21.15.143:0/937669175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde58102ad0 0x7fde58074620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:01.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 >> 172.21.15.143:0/937669175 conn(0x7fde58078f40 msgr2=0x7fde580ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:01.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 shutdown_connections 2024-03-11T21:28:01.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 -- 172.21.15.143:0/937669175 wait complete. 2024-03-11T21:28:01.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:01.659+0000 7fde4d7fa700 1 librados: shutdown 2024-03-11T21:28:01.635 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:28:01.638 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:01.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:01.760 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:01 smithi203 bash[23654]: audit 2024-03-11T21:28:00.722213+0000 mon.a (mon.0) 898 : audit 0 from='client.? 172.21.15.143:0/906737842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:01 smithi203 bash[23654]: audit 2024-03-11T21:28:01.157887+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.143:0/937669175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:01 smithi203 bash[23654]: audit 2024-03-11T21:28:01.158152+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.889 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:01.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[21512]: audit 2024-03-11T21:28:00.722213+0000 mon.a (mon.0) 898 : audit 0 from='client.? 172.21.15.143:0/906737842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:01.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[21512]: audit 2024-03-11T21:28:01.157887+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.143:0/937669175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[21512]: audit 2024-03-11T21:28:01.158152+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[29284]: audit 2024-03-11T21:28:00.722213+0000 mon.a (mon.0) 898 : audit 0 from='client.? 172.21.15.143:0/906737842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:01.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[29284]: audit 2024-03-11T21:28:01.157887+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.143:0/937669175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:01 smithi143 bash[29284]: audit 2024-03-11T21:28:01.158152+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:01.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.013 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:02.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.134 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:02.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:02.265 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:02.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.377 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:02.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:02.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.512 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:02.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.636 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:02.637 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[21512]: cluster 2024-03-11T21:28:01.153405+0000 mgr.y (mgr.14550) 187 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[21512]: cluster 2024-03-11T21:28:01.641156+0000 mon.a (mon.0) 900 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[21512]: audit 2024-03-11T21:28:01.648143+0000 mon.a (mon.0) 901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[21512]: cluster 2024-03-11T21:28:01.659000+0000 mon.a (mon.0) 902 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[29284]: cluster 2024-03-11T21:28:01.153405+0000 mgr.y (mgr.14550) 187 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[29284]: cluster 2024-03-11T21:28:01.641156+0000 mon.a (mon.0) 900 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[29284]: audit 2024-03-11T21:28:01.648143+0000 mon.a (mon.0) 901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:02.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:02 smithi143 bash[29284]: cluster 2024-03-11T21:28:01.659000+0000 mon.a (mon.0) 902 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:28:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:02.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:02.767 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:02.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:02.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:03.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:02 smithi203 bash[23654]: cluster 2024-03-11T21:28:01.153405+0000 mgr.y (mgr.14550) 187 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:03.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:02 smithi203 bash[23654]: cluster 2024-03-11T21:28:01.641156+0000 mon.a (mon.0) 900 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:03.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:02 smithi203 bash[23654]: audit 2024-03-11T21:28:01.648143+0000 mon.a (mon.0) 901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:03.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:02 smithi203 bash[23654]: cluster 2024-03-11T21:28:01.659000+0000 mon.a (mon.0) 902 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:28:03.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:03.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.157 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.260 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:03.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:03.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:03.546 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:03.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:03.675 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:03.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:03.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:03.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:03.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:04.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:04.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:04.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:28:04.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:04.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:04.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:04.324 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:04.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:04.455 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:04.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:04.583 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:04.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:04.712 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:04.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:28:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:04.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:04 smithi203 bash[23654]: cluster 2024-03-11T21:28:03.154171+0000 mgr.y (mgr.14550) 188 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:04.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:04.826 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:04 smithi143 bash[29284]: cluster 2024-03-11T21:28:03.154171+0000 mgr.y (mgr.14550) 188 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:04.826 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:04 smithi143 bash[21512]: cluster 2024-03-11T21:28:03.154171+0000 mgr.y (mgr.14550) 188 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:04.841 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:04.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:04.971 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:05.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:05.102 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:05.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:28:05.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:28:05.149 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:28:05.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:28:05.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:05.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:05.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:05.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:05.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:05.489 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:05.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:05.619 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:05.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:05.750 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:05.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:05.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:05.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:06.013 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:06.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:06.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:06.227 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:06.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:06.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.402 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:06.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:06.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:06.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:06.664 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:06.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:06 smithi143 bash[21512]: cluster 2024-03-11T21:28:05.154580+0000 mgr.y (mgr.14550) 189 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:06.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:06 smithi143 bash[29284]: cluster 2024-03-11T21:28:05.154580+0000 mgr.y (mgr.14550) 189 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:06.767 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:06.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:06 smithi203 bash[23654]: cluster 2024-03-11T21:28:05.154580+0000 mgr.y (mgr.14550) 189 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:06.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:06.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:07.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:07.055 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:07.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:07.184 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:07.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:07.314 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:07.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:07.442 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:07.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:07.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:07.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:07.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:06.790569+0000 osd.1 (osd.1) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:07.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:06.790650+0000 osd.1 (osd.1) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:07.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:06.790569+0000 osd.1 (osd.1) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:07.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:06.790650+0000 osd.1 (osd.1) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:07.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:07.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:07.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:07.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:07.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:06.790569+0000 osd.1 (osd.1) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:06.790650+0000 osd.1 (osd.1) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:08.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:08.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.216 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:08.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:08.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.473 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:08.520 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:08.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:08.603 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:08.706 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:08 smithi143 bash[29284]: cluster 2024-03-11T21:28:07.155002+0000 mgr.y (mgr.14550) 190 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:28:08.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:08 smithi143 bash[29284]: cluster 2024-03-11T21:28:07.668626+0000 mon.a (mon.0) 903 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:28:08.716 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:08 smithi143 bash[21512]: cluster 2024-03-11T21:28:07.155002+0000 mgr.y (mgr.14550) 190 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:28:08.716 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:08 smithi143 bash[21512]: cluster 2024-03-11T21:28:07.668626+0000 mon.a (mon.0) 903 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:28:08.731 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:08.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:08.846 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:08 smithi203 bash[23654]: cluster 2024-03-11T21:28:07.155002+0000 mgr.y (mgr.14550) 190 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:28:08.846 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:08 smithi203 bash[23654]: cluster 2024-03-11T21:28:07.668626+0000 mon.a (mon.0) 903 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:28:08.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:08.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:09.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:09.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:09.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:09.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:09.269 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:09.269 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:09.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:09.378 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:09.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:09.510 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:09.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:09.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:09.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:09.772 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:09.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:09.903 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:09.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:09 smithi203 bash[23654]: audit 2024-03-11T21:28:08.282044+0000 mgr.y (mgr.14550) 191 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:09.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:09 smithi143 bash[29284]: audit 2024-03-11T21:28:08.282044+0000 mgr.y (mgr.14550) 191 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:09.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:09 smithi143 bash[21512]: audit 2024-03-11T21:28:08.282044+0000 mgr.y (mgr.14550) 191 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:10.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:10.031 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:10.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:10.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:10.266 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:10.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:10.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:10.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:10.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:10 smithi203 bash[23654]: cluster 2024-03-11T21:28:09.155804+0000 mgr.y (mgr.14550) 192 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:10.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:10.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:10.808 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:10.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:10.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:10 smithi143 bash[21512]: cluster 2024-03-11T21:28:09.155804+0000 mgr.y (mgr.14550) 192 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:10.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:10 smithi143 bash[29284]: cluster 2024-03-11T21:28:09.155804+0000 mgr.y (mgr.14550) 192 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:10.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:11.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:11.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:11.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:11.201 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:11.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.331 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.384 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:28:11.384 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:28:11.384 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:56 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:07.116436+0000", 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 304.29357704400002, 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0023207060000000001, 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 56 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.385 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116436+0000", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116436+0000", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116439+0000", 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7520000000000001e-06 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116447+0000", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9400000000000002e-06 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116479+0000", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1906999999999998e-05 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116499+0000", 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9894e-05 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.387 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116624+0000", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012563900000000001 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116805+0000", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018100899999999999 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116972+0000", 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000166379 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.388 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118438+0000", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001466008 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118517+0000", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8814999999999997e-05 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118588+0000", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1131999999999996e-05 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118664+0000", 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6252000000000001e-05 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118757+0000", 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2979000000000005e-05 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24350.0:55 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T21:28:11.390 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:07.116984+0000", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 304.29302968299999, 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024060290000000001, 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24350", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3339680007", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 55 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116984+0000", 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.391 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116984+0000", 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116985+0000", 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.872e-06 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.392 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.116995+0000", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.8120000000000002e-06 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.117015+0000", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9388000000000001e-05 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.117030+0000", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.502e-05 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.393 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.117388+0000", 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000358115 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.117589+0000", 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020133299999999999 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.117933+0000", 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00034412800000000002 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.394 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118783+0000", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00084958600000000005 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119081+0000", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029764100000000002 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119262+0000", 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018146300000000001 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.395 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119318+0000", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5593000000000001e-05 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119390+0000", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2077999999999997e-05 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:07.118989+0000", 2024-03-11T21:28:11.396 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 304.29102444199998, 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0023744259999999998, 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 57 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118989+0000", 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118989+0000", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118990+0000", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0720000000000001e-06 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.118996+0000", 2024-03-11T21:28:11.398 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.0229999999999998e-06 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119016+0000", 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9808000000000001e-05 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119028+0000", 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2069000000000001e-05 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.399 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119130+0000", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000102193 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119212+0000", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.2076000000000005e-05 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119435+0000", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00022305400000000001 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119544+0000", 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000108507 2024-03-11T21:28:11.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119744+0000", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020081499999999999 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.120393+0000", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000648563 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.120902+0000", 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000509385 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121162+0000", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025917099999999999 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121232+0000", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0650999999999998e-05 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121363+0000", 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000131039 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24350.0:56 5.3 5:c609908c:::notify.5:head [watch watch cookie 94899662041984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:07.119732+0000", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 304.29028145799998, 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0036703809999999999, 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24350", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3339680007", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 56 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119732+0000", 2024-03-11T21:28:11.403 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119732+0000", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119734+0000", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1090000000000001e-06 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119752+0000", 2024-03-11T21:28:11.404 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8270999999999998e-05 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119779+0000", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6882000000000001e-05 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.119827+0000", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7753000000000002e-05 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.120081+0000", 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025454399999999999 2024-03-11T21:28:11.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.120153+0000", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2008999999999994e-05 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121408+0000", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0012547470000000001 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121495+0000", 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6898999999999996e-05 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.121674+0000", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017891200000000001 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.122186+0000", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00051155299999999999 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.122658+0000", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00047253099999999999 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.123237+0000", 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00057905800000000002 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.123316+0000", 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8712000000000001e-05 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:07.123402+0000", 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6401000000000002e-05 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.408 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:214 3.12 3:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:10.189416+0000", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 301.22059732600002, 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024093999999999999, 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 214 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189416+0000", 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189416+0000", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189418+0000", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1299999999999999e-06 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189426+0000", 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9719999999999992e-06 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189444+0000", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7805999999999999e-05 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189457+0000", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2765000000000001e-05 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189531+0000", 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4683000000000003e-05 2024-03-11T21:28:11.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189693+0000", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016143099999999999 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.190016+0000", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00032305099999999998 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.190577+0000", 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00056112200000000005 2024-03-11T21:28:11.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.190855+0000", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027839999999999999 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.191763+0000", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00090736500000000004 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.191796+0000", 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3018000000000001e-05 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.191825+0000", 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9657000000000001e-05 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24350.0:202 5.3 5:c609908c:::notify.5:head [watch unwatch cookie 94899662041984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:10.189698+0000", 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 301.22031573300001, 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00251571, 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.414 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24350", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3339680007", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 202 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189698+0000", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189698+0000", 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189699+0000", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.443e-06 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189705+0000", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.1659999999999999e-06 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189725+0000", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9496000000000001e-05 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189731+0000", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7440000000000001e-06 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189831+0000", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9703000000000004e-05 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.189969+0000", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013752 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.190271+0000", 2024-03-11T21:28:11.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00030262300000000002 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.190985+0000", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00071325900000000005 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.191757+0000", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00077216600000000002 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.191853+0000", 2024-03-11T21:28:11.418 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6119000000000006e-05 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.192033+0000", 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000180197 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.192213+0000", 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018027399999999999 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:232 3.1f 3:fde4dd55:gc::gc.26:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-11T21:28:11.419 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:10.237240+0000", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 301.17277354700002, 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002466644, 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 232 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237240+0000", 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.420 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237240+0000", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237241+0000", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1689999999999999e-06 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237248+0000", 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7129999999999999e-06 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.421 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237261+0000", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3716e-05 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237268+0000", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.511e-06 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237354+0000", 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.5890000000000006e-05 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.422 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237568+0000", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00021424899999999999 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.237996+0000", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00042751699999999998 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.238681+0000", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00068551599999999995 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.238887+0000", 2024-03-11T21:28:11.423 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020561200000000001 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.239549+0000", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00066238799999999997 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.239631+0000", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1808e-05 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.239706+0000", 2024-03-11T21:28:11.424 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5555000000000004e-05 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:610 3.1b 3:db9d130d:gc::gc.24:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:10.446904+0000", 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 300.96310887200002, 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0033410559999999998, 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 610 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.425 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446904+0000", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446904+0000", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446905+0000", 2024-03-11T21:28:11.426 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0109999999999999e-06 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446913+0000", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4120000000000004e-06 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446928+0000", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4978e-05 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446934+0000", 2024-03-11T21:28:11.427 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2539999999999998e-06 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.446995+0000", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.0686999999999997e-05 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.447068+0000", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3689999999999994e-05 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.447185+0000", 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000116463 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.428 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.447526+0000", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00034057100000000002 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.450074+0000", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0025483559999999999 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.450130+0000", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6514000000000001e-05 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.429 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.450193+0000", 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2593000000000002e-05 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.450245+0000", 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2527e-05 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:872 3.1 3:86cc8a85:::obj_delete_at_hint.0000000118:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:23:10.701580+0000", 2024-03-11T21:28:11.430 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 300.70843373700001, 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0027186559999999999, 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 872 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701580+0000", 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.431 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701580+0000", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701580+0000", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.23e-07 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701584+0000", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0419999999999996e-06 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.432 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701596+0000", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1569999999999999e-05 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701605+0000", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.3190000000000001e-06 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701650+0000", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4294e-05 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701733+0000", 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3042999999999998e-05 2024-03-11T21:28:11.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.701901+0000", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000168796 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.702189+0000", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00028757499999999998 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.702416+0000", 2024-03-11T21:28:11.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00022707599999999999 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.704232+0000", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0018157270000000001 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.704265+0000", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2827999999999998e-05 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:23:10.704298+0000", 2024-03-11T21:28:11.435 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3562999999999997e-05 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(4.0,QREQUEST,e:102,from:3,reservation_nonce:2)", 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:24:24.031088+0000", 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 227.37892495899999, 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.003405188, 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031088+0000", 2024-03-11T21:28:11.436 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031088+0000", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031091+0000", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5170000000000002e-06 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031106+0000", 2024-03-11T21:28:11.437 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5102999999999999e-05 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031137+0000", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1226000000000002e-05 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031155+0000", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7609999999999999e-05 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031281+0000", 2024-03-11T21:28:11.438 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012570499999999999 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.031293+0000", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2326e-05 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.034303+0000", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0030104110000000002 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.034358+0000", 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4432999999999999e-05 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:24.034494+0000", 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000135857 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:1110 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:24:58.084409+0000", 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 193.32560453100001, 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0063000119999999998, 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.440 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1110 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084409+0000", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084409+0000", 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084412+0000", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2160000000000002e-06 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084426+0000", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3696000000000001e-05 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084463+0000", 2024-03-11T21:28:11.442 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7379999999999998e-05 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084499+0000", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6155999999999997e-05 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084719+0000", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00021964700000000001 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.084738+0000", 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8972999999999999e-05 2024-03-11T21:28:11.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.090404+0000", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056664020000000001 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.090532+0000", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012792500000000001 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:24:58.090709+0000", 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017661700000000001 2024-03-11T21:28:11.444 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:116,from:2,reservation_nonce:4)", 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:25:47.981822+0000", 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 143.42819168299999, 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0033752859999999999, 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981822+0000", 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.445 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981822+0000", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981824+0000", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8700000000000001e-06 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981842+0000", 2024-03-11T21:28:11.446 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7090000000000001e-05 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981883+0000", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0942999999999997e-05 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.981907+0000", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4355999999999999e-05 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.982032+0000", 2024-03-11T21:28:11.447 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000125315 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.982041+0000", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9709999999999993e-06 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.985058+0000", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0030166540000000001 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.985094+0000", 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6445000000000001e-05 2024-03-11T21:28:11.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:25:47.985197+0000", 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000102642 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:1242 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:03.137172+0000", 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 128.27284115399999, 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0049095290000000001, 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.449 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1242 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137172+0000", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.450 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137172+0000", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137175+0000", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8100000000000002e-06 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137187+0000", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1993e-05 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.451 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137221+0000", 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3726000000000002e-05 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137247+0000", 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6783000000000001e-05 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137426+0000", 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017878800000000001 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.137445+0000", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8618999999999999e-05 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.141506+0000", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.004061238 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.141723+0000", 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00021693 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.453 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:03.142082+0000", 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000358642 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:1293 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:28.156211+0000", 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 103.253802365, 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0085972830000000007, 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.454 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1293 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156211+0000", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156211+0000", 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156214+0000", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1599999999999998e-06 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156235+0000", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0579000000000001e-05 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156280+0000", 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5809999999999997e-05 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156309+0000", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8796e-05 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156543+0000", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00023405400000000001 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.156561+0000", 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7963000000000001e-05 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.457 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.164354+0000", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0077923439999999997 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.164610+0000", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025596400000000002 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:28.164808+0000", 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019861299999999999 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.458 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop_reply(osd.0.0:76 3.1f e121/78 ondisk, result = 0)", 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:51.603680+0000", 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 79.806333711999997, 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002430415, 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603680+0000", 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603680+0000", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603681+0000", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.217e-06 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603689+0000", 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4659999999999999e-06 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.460 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603711+0000", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1586999999999998e-05 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.603727+0000", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6096e-05 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605719+0000", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001991961 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.461 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605747+0000", 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8339000000000001e-05 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606110+0000", 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00036274899999999999 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop_reply(osd.0.0:77 3.1f e121/78 ondisk, result = 0)", 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:51.604117+0000", 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 79.805896227000005, 2024-03-11T21:28:11.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024166639999999998, 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604117+0000", 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604117+0000", 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604118+0000", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5300000000000003e-07 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604124+0000", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8699999999999997e-06 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604146+0000", 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2506000000000001e-05 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604160+0000", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3956e-05 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606259+0000", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0020988529999999999 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606280+0000", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0777000000000001e-05 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.465 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606534+0000", 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000253949 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop_reply(osd.0.0:78 3.1f e121/78 ondisk, result = 0)", 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:51.604277+0000", 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 79.80573613, 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0023495259999999998, 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.466 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604277+0000", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604277+0000", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604278+0000", 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.4000000000000001e-07 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604282+0000", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3200000000000001e-06 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604296+0000", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3823e-05 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604304+0000", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4630000000000001e-06 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606595+0000", 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0022904969999999998 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606604+0000", 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9970000000000004e-06 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606627+0000", 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2785999999999998e-05 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop_reply(osd.0.0:78 3.1f e121/78 ondisk, result = 0)", 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:51.604669+0000", 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 79.805344517999998, 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002319163, 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604669+0000", 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.470 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604669+0000", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604670+0000", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1299999999999998e-07 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604677+0000", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6299999999999998e-06 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604695+0000", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7283e-05 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.604704+0000", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.8300000000000008e-06 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606722+0000", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0020174770000000002 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606743+0000", 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0783000000000001e-05 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.606988+0000", 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00024524699999999998 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop_reply(osd.0.0:79 3.1f e121/78 ondisk, result = 0)", 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:26:51.605158+0000", 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 79.804854997999996, 2024-03-11T21:28:11.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0023081930000000001, 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605158+0000", 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605158+0000", 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605159+0000", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.097e-06 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605167+0000", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5900000000000002e-06 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605188+0000", 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1172999999999999e-05 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.605204+0000", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5613000000000001e-05 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.607188+0000", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001984694 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.607220+0000", 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1723000000000001e-05 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:26:51.607467+0000", 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00024630300000000002 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:1486 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:03.228539+0000", 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 8.1814741570000002, 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0028734250000000002, 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:28:11.477 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1486 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228539+0000", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228539+0000", 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:28:11.478 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228541+0000", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9929999999999998e-06 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228553+0000", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1891e-05 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228585+0000", 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2196000000000001e-05 2024-03-11T21:28:11.479 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.228596+0000", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0754000000000001e-05 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.229059+0000", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00046328299999999999 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.229066+0000", 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.691e-06 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.231227+0000", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0021612580000000001 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.231309+0000", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.2223000000000002e-05 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:03.231413+0000", 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000103136 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:28:11.481 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:28:11.482 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:28:11.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:11.503 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:11.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.634 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.766 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:11.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:12.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:12.138 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:12.154 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:12.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:12.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:12.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:12.407 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:12.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:12.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:12.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:12.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:12.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:12 smithi143 bash[29284]: cluster 2024-03-11T21:28:11.156224+0000 mgr.y (mgr.14550) 193 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:12.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:12 smithi143 bash[21512]: cluster 2024-03-11T21:28:11.156224+0000 mgr.y (mgr.14550) 193 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:12.796 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:12.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:12 smithi203 bash[23654]: cluster 2024-03-11T21:28:11.156224+0000 mgr.y (mgr.14550) 193 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:13.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:13.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:13.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:13.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:13.324 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:13.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:13.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:13.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.583 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:13.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:13.714 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:13.817 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:13 smithi143 bash[21512]: audit 2024-03-11T21:28:12.906549+0000 mon.c (mon.2) 97 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:13.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:13 smithi143 bash[29284]: audit 2024-03-11T21:28:12.906549+0000 mon.c (mon.2) 97 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:13.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:13.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:28:13.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:13.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:13 smithi203 bash[23654]: audit 2024-03-11T21:28:12.906549+0000 mon.c (mon.2) 97 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:14.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:14.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:14.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:14.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:14.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:14.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:14.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:14.499 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:14.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:14.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:14.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:14.742 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:14 smithi203 bash[23654]: cluster 2024-03-11T21:28:13.157003+0000 mgr.y (mgr.14550) 194 : cluster 0 pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:14.759 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:14.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:14.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:14.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:14 smithi143 bash[29284]: cluster 2024-03-11T21:28:13.157003+0000 mgr.y (mgr.14550) 194 : cluster 0 pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:14.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:14 smithi143 bash[21512]: cluster 2024-03-11T21:28:13.157003+0000 mgr.y (mgr.14550) 194 : cluster 0 pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:14.991 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.017 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:15.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:15.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:15.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:15.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:15.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:15.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:15.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:15.669 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:15.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:15.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:15.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.959 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:16.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:16.089 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:16.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:16.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:16.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:16.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:16.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:16.479 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:16.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:16.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:16.662 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:16 smithi143 bash[21512]: cluster 2024-03-11T21:28:15.157415+0000 mgr.y (mgr.14550) 195 : cluster 0 pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:16.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:16.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:16.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:16.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:16 smithi203 bash[23654]: cluster 2024-03-11T21:28:15.157415+0000 mgr.y (mgr.14550) 195 : cluster 0 pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:16.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:16.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:16 smithi143 bash[29284]: cluster 2024-03-11T21:28:15.157415+0000 mgr.y (mgr.14550) 195 : cluster 0 pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:16.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:16.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:17.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.128 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:17.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:17.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:17.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.388 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:17.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:17.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.647 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:17.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:17.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:17.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:18.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.038 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:18.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:18.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:18.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:18.303 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:18.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:18.417 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:18.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.459+0000 7f9b47f06700 1 -- 172.21.15.143:0/2672831979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401060a0 msgr2=0x7f9b4010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.459+0000 7f9b47f06700 1 --2- 172.21.15.143:0/2672831979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401060a0 0x7f9b4010dc70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b0d0 tx=0x7f9b3800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.459+0000 7f9b47f06700 1 -- 172.21.15.143:0/2672831979 shutdown_connections 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.459+0000 7f9b47f06700 1 --2- 172.21.15.143:0/2672831979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 0x7f9b401013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 --2- 172.21.15.143:0/2672831979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401060a0 0x7f9b4010dc70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 --2- 172.21.15.143:0/2672831979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b40101970 0x7f9b401059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- 172.21.15.143:0/2672831979 >> 172.21.15.143:0/2672831979 conn(0x7f9b40077de0 msgr2=0x7f9b400781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:18.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- 172.21.15.143:0/2672831979 shutdown_connections 2024-03-11T21:28:18.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- 172.21.15.143:0/2672831979 wait complete. 2024-03-11T21:28:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 librados: starting msgr at 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 librados: starting objecter 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 Processor -- start 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- start start 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 librados: setting wanted keys 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 librados: calling monclient init 2024-03-11T21:28:18.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 0x7f9b4019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b401060a0 0x7f9b40196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40110650 con 0x7f9b40100fc0 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401104d0 con 0x7f9b401060a0 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401107d0 con 0x7f9b40101970 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:36656/0 (socket says 172.21.15.143:36656) 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 -- 172.21.15.143:0/3003615208 learned_addr learned my addr 172.21.15.143:0/3003615208 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 0x7f9b4019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b401060a0 0x7f9b40196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b401060a0 msgr2=0x7f9b40196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b401060a0 0x7f9b40196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 msgr2=0x7f9b4019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 0x7f9b4019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40197170 con 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b40100fc0 0x7f9b4019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000b500 tx=0x7f9b3000b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b36ffd700 10 client.?.objecter ms_handle_connect 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b36ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30015020 con 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9b30011cb0 con 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b40197460 con 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 10 client.24680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4006a3a0 con 0x7f9b40101970 2024-03-11T21:28:18.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.463+0000 7f9b47f06700 1 librados: init done 2024-03-11T21:28:18.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30010cf0 con 0x7f9b40101970 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9b24005100 con 0x7f9b40101970 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9b30018020 con 0x7f9b40101970 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 0x7f9b2c071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(132..132 src has 77..132) v4 ==== 7450+0+0 (secure 0 0 0) 0x7f9b3008d870 con 0x7f9b40101970 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 osd_map(132..132 src has 77..132) v4 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 3 client.24680.objecter handle_osd_map got epochs [132,132] > 0 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 3 client.24680.objecter handle_osd_map decoding full epoch 132 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 0x7f9b2c071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 20 client.24680.objecter dump_active .. 0 homeless 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v132) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9b3001f020 con 0x7f9b40101970 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 10 client.24680.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 1 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9b24004730 0x7f9b24004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b2c06f6f0 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 0x7f9b2c071bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9b40197b50 tx=0x7f9b3c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9b24004730 0x7f9b24004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b24004730 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9b24004730 0x7f9b24004b70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 1 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 0 1 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 4 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 10 client.24680.objecter _send_command 1 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.467+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b24008b70 con 0x7f9b24004730 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.471+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b24008b70 con 0x7f9b24004730 2024-03-11T21:28:18.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.471+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:28:18.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.471+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 1 = system:0 2024-03-11T21:28:18.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.471+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 4 2024-03-11T21:28:18.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.471+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 0 1 2024-03-11T21:28:18.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 10 client.24680.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 2 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 2 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b240053e0 osd=0 3 2024-03-11T21:28:18.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 0 2 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 4 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 10 client.24680.objecter _send_command 2 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.503+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b240042d0 con 0x7f9b24004730 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b240042d0 con 0x7f9b24004730 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 4 2024-03-11T21:28:18.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 0 2 2024-03-11T21:28:18.480 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 10 client.24680.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9b24002750 0x7f9b24002bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 1 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 4 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 10 client.24680.objecter _send_command 3 2024-03-11T21:28:18.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b240030f0 con 0x7f9b24002750 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9b24002750 0x7f9b24002bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9b24002750 0x7f9b24002bb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b24002750 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b240030f0 con 0x7f9b24002750 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 3 = system:0 2024-03-11T21:28:18.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 4 2024-03-11T21:28:18.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.507+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 1 3 2024-03-11T21:28:18.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 10 client.24680.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 4 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 4 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24004560 osd=1 3 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 1 4 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 4 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 10 client.24680.objecter _send_command 4 2024-03-11T21:28:18.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b24003b00 con 0x7f9b24002750 2024-03-11T21:28:18.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b24003b00 con 0x7f9b24002750 2024-03-11T21:28:18.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 4 2024-03-11T21:28:18.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 1 4 2024-03-11T21:28:18.518 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:28:18.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 10 client.24680.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 5 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9b2400b240 0x7f9b2400d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 5 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 2 5 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 4 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 10 client.24680.objecter _send_command 5 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b24010520 con 0x7f9b2400b240 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9b2400b240 0x7f9b2400d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b2400b240 2024-03-11T21:28:18.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.543+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9b2400b240 0x7f9b2400d630 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.547+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b24010520 con 0x7f9b2400b240 2024-03-11T21:28:18.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.547+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:28:18.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.547+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 5 = system:0 2024-03-11T21:28:18.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.547+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 4 2024-03-11T21:28:18.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.547+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 2 5 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 10 client.24680.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 6 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 6 2024-03-11T21:28:18.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24008140 osd=2 3 2024-03-11T21:28:18.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 2 6 2024-03-11T21:28:18.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 4 2024-03-11T21:28:18.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 10 client.24680.objecter _send_command 6 2024-03-11T21:28:18.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b240035e0 con 0x7f9b2400b240 2024-03-11T21:28:18.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b240035e0 con 0x7f9b2400b240 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 4 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.563+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 2 6 2024-03-11T21:28:18.540 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 10 client.24680.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 7 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9b2400ecf0 0x7f9b2400f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 7 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 3 7 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 4 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 10 client.24680.objecter _send_command 7 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b2400f650 con 0x7f9b2400ecf0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9b2400ecf0 0x7f9b2400f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b2400ecf0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9b2400ecf0 0x7f9b2400f110 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b2400f650 con 0x7f9b2400ecf0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 7 = system:0 2024-03-11T21:28:18.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 4 2024-03-11T21:28:18.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.567+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 3 7 2024-03-11T21:28:18.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 10 client.24680.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 8 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 8 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24001b80 osd=3 3 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 3 8 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 4 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 10 client.24680.objecter _send_command 8 2024-03-11T21:28:18.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.583+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b2400f650 con 0x7f9b2400ecf0 2024-03-11T21:28:18.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.587+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b2400f650 con 0x7f9b2400ecf0 2024-03-11T21:28:18.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.587+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.587+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.587+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 4 2024-03-11T21:28:18.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.587+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 3 8 2024-03-11T21:28:18.563 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:28:18.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 10 client.24680.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 9 2024-03-11T21:28:18.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9b24010060 0x7f9b24014750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 9 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 4 9 2024-03-11T21:28:18.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 4 2024-03-11T21:28:18.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 10 client.24680.objecter _send_command 9 2024-03-11T21:28:18.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b24017660 con 0x7f9b24010060 2024-03-11T21:28:18.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9b24010060 0x7f9b24014750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b24010060 2024-03-11T21:28:18.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9b24010060 0x7f9b24014750 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b24017660 con 0x7f9b24010060 2024-03-11T21:28:18.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:28:18.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 9 = system:0 2024-03-11T21:28:18.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 4 2024-03-11T21:28:18.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.591+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 4 9 2024-03-11T21:28:18.571 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.627+0000 7f9b47f06700 10 client.24680.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 10 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 10 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b2400f650 osd=4 3 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 4 10 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 4 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 10 client.24680.objecter _send_command 10 2024-03-11T21:28:18.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b24014c90 con 0x7f9b24010060 2024-03-11T21:28:18.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b24014c90 con 0x7f9b24010060 2024-03-11T21:28:18.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 4 2024-03-11T21:28:18.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.631+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 4 10 2024-03-11T21:28:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:28:18.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 10 client.24680.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 11 2024-03-11T21:28:18.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9b240153b0 0x7f9b24015810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 11 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 5 11 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 4 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 10 client.24680.objecter _send_command 11 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b2401aa80 con 0x7f9b240153b0 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9b240153b0 0x7f9b24015810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b240153b0 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b454a1700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9b240153b0 0x7f9b24015810 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b2401aa80 con 0x7f9b240153b0 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 11 = system:0 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 4 2024-03-11T21:28:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.635+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 5 11 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 10 client.24680.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 12 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 12 2024-03-11T21:28:18.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24014c90 osd=5 3 2024-03-11T21:28:18.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 5 12 2024-03-11T21:28:18.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 4 2024-03-11T21:28:18.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 10 client.24680.objecter _send_command 12 2024-03-11T21:28:18.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b2401aa80 con 0x7f9b240153b0 2024-03-11T21:28:18.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b2401aa80 con 0x7f9b240153b0 2024-03-11T21:28:18.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 4 2024-03-11T21:28:18.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.671+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 5 12 2024-03-11T21:28:18.647 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:28:18.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 10 client.24680.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 13 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9b24019dc0 0x7f9b2401a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 13 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 6 13 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 4 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 10 client.24680.objecter _send_command 13 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b2401de70 con 0x7f9b24019dc0 2024-03-11T21:28:18.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9b24019dc0 0x7f9b2401a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b24019dc0 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b464a3700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9b24019dc0 0x7f9b2401a200 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b2401de70 con 0x7f9b24019dc0 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 13 = system:0 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 4 2024-03-11T21:28:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.675+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 6 13 2024-03-11T21:28:18.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:18.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 10 client.24680.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 14 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 14 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b2401aa80 osd=6 3 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 6 14 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 4 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 10 client.24680.objecter _send_command 14 2024-03-11T21:28:18.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.707+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b24015da0 con 0x7f9b24019dc0 2024-03-11T21:28:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:28:18.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.715+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b24015da0 con 0x7f9b24019dc0 2024-03-11T21:28:18.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.715+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.715+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.715+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 4 2024-03-11T21:28:18.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.715+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 6 14 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 10 client.24680.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 15 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9b240164c0 0x7f9b24016900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 15 2024-03-11T21:28:18.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 7 15 2024-03-11T21:28:18.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 4 2024-03-11T21:28:18.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 10 client.24680.objecter _send_command 15 2024-03-11T21:28:18.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b24016fe0 con 0x7f9b240164c0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9b240164c0 0x7f9b24016900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b45ca2700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9b240164c0 0x7f9b24016900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.719+0000 7f9b36ffd700 10 client.24680.objecter ms_handle_connect 0x7f9b240164c0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.723+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b24016fe0 con 0x7f9b240164c0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.723+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.723+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 15 = system:0 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.723+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 4 2024-03-11T21:28:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.723+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 7 15 2024-03-11T21:28:18.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:18.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 10 client.24680.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:28:18.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign -1 16 2024-03-11T21:28:18.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 20 client.24680.objecter _get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_remove -1 16 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 20 client.24680.objecter get_session s=0x7f9b24015da0 osd=7 3 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 15 client.24680.objecter _session_command_op_assign 7 16 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 4 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 10 client.24680.objecter _send_command 16 2024-03-11T21:28:18.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.743+0000 7f9b47f06700 1 -- 172.21.15.143:0/3003615208 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b24016fe0 con 0x7f9b240164c0 2024-03-11T21:28:18.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b36ffd700 1 -- 172.21.15.143:0/3003615208 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b24016fe0 con 0x7f9b240164c0 2024-03-11T21:28:18.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b36ffd700 10 client.24680.objecter ms_dispatch 0x7f9b40001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:18.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b36ffd700 10 client.24680.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:18.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b36ffd700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 4 2024-03-11T21:28:18.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b36ffd700 15 client.24680.objecter _session_command_op_remove 7 16 2024-03-11T21:28:18.722 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:28:18.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.747+0000 7f9b34ff9700 10 librados: watch_flush enter 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 librados: watch_flush exit 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.0 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9b24004730 msgr2=0x7f9b24004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9b24004730 0x7f9b24004b70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b240053e0 osd=0 1 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.1 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9b24002750 msgr2=0x7f9b24002bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9b24002750 0x7f9b24002bb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b24004560 osd=1 1 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.2 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9b2400b240 msgr2=0x7f9b2400d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9b2400b240 0x7f9b2400d630 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b464a3700 1 -- 172.21.15.143:0/3003615208 reap_dead start 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b24008140 osd=2 1 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.3 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9b2400ecf0 msgr2=0x7f9b2400f110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9b2400ecf0 0x7f9b2400f110 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b24001b80 osd=3 1 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.4 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9b24010060 msgr2=0x7f9b24014750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9b24010060 0x7f9b24014750 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b2400f650 osd=4 1 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.5 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9b240153b0 msgr2=0x7f9b24015810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9b240153b0 0x7f9b24015810 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b24014c90 osd=5 1 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.6 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9b24019dc0 msgr2=0x7f9b2401a200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9b24019dc0 0x7f9b2401a200 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b2401aa80 osd=6 1 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter close_session for osd.7 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9b240164c0 msgr2=0x7f9b24016900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9b240164c0 0x7f9b24016900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b464a3700 1 -- 172.21.15.143:0/3003615208 reap_dead start 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter put_session s=0x7f9b24015da0 osd=7 1 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 20 client.24680.objecter shutdown clearing up homeless session... 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 10 client.24680.objecter successfully canceled tick 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 msgr2=0x7f9b2c071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 0x7f9b2c071bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9b40197b50 tx=0x7f9b3c009260 comp rx=0 tx=0).stop 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 msgr2=0x7f9b4019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000b500 tx=0x7f9b3000b810 comp rx=0 tx=0).stop 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 shutdown_connections 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b40101970 0x7f9b4019cd90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 --2- 172.21.15.143:0/3003615208 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9b2c06f6f0 0x7f9b2c071bb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 >> 172.21.15.143:0/3003615208 conn(0x7f9b40077de0 msgr2=0x7f9b400feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 shutdown_connections 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 -- 172.21.15.143:0/3003615208 wait complete. 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:18.751+0000 7f9b34ff9700 1 librados: shutdown 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:18.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:18.835 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:18.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[21512]: cluster 2024-03-11T21:28:17.157839+0000 mgr.y (mgr.14550) 196 : cluster 0 pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[21512]: cluster 2024-03-11T21:28:18.149867+0000 osd.7 (osd.7) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[21512]: cluster 2024-03-11T21:28:18.149974+0000 osd.7 (osd.7) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[21512]: audit 2024-03-11T21:28:18.469279+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.143:0/3003615208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[29284]: cluster 2024-03-11T21:28:17.157839+0000 mgr.y (mgr.14550) 196 : cluster 0 pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[29284]: cluster 2024-03-11T21:28:18.149867+0000 osd.7 (osd.7) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[29284]: cluster 2024-03-11T21:28:18.149974+0000 osd.7 (osd.7) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:18.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:18 smithi143 bash[29284]: audit 2024-03-11T21:28:18.469279+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.143:0/3003615208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:18.970 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:18.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- 172.21.15.143:0/1453965109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 msgr2=0x7fdc68106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/1453965109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68106930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00aaa0 tx=0x7fdc5c00adb0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- 172.21.15.143:0/1453965109 shutdown_connections 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/1453965109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc68100e60 0x7fdc6810d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/1453965109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68106930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/1453965109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc681004a0 0x7fdc68100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- 172.21.15.143:0/1453965109 >> 172.21.15.143:0/1453965109 conn(0x7fdc680772e0 msgr2=0x7fdc680fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:18.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- 172.21.15.143:0/1453965109 shutdown_connections 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- 172.21.15.143:0/1453965109 wait complete. 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 librados: starting msgr at 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 librados: starting objecter 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 Processor -- start 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- start start 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 librados: setting wanted keys 2024-03-11T21:28:19.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 librados: calling monclient init 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 0x7fdc6819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc68100e60 0x7fdc6819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6810f530 con 0x7fdc681004a0 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6810f3b0 con 0x7fdc68100e60 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6ef05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6810f6b0 con 0x7fdc68106550 2024-03-11T21:28:19.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6d4a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6d4a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:36686/0 (socket says 172.21.15.143:36686) 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6d4a2700 1 -- 172.21.15.143:0/814896263 learned_addr learned my addr 172.21.15.143:0/814896263 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.023+0000 7fdc6cca1700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 0x7fdc6819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc67fff700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc68100e60 0x7fdc6819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 -- 172.21.15.143:0/814896263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc68100e60 msgr2=0x7fdc6819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc68100e60 0x7fdc6819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 -- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 msgr2=0x7fdc6819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 0x7fdc6819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 -- 172.21.15.143:0/814896263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68196050 con 0x7fdc68106550 2024-03-11T21:28:19.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6cca1700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 0x7fdc6819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6d4a2700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000b970 tx=0x7fdc6000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 10 client.?.objecter ms_handle_connect 0x7fdc68106550 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60019070 con 0x7fdc68106550 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc68196340 con 0x7fdc68106550 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6ef05700 10 client.24686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:19.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc68199410 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6ef05700 1 librados: init done 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fdc6000cdb0 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60014e40 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc68101bc0 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.027+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fdc6001c070 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 0x7fdc58071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc6cca1700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 0x7fdc58071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(133..133 src has 77..133) v4 ==== 7450+0+0 (secure 0 0 0) 0x7fdc6008d6b0 con 0x7fdc68106550 2024-03-11T21:28:19.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 10 client.24686.objecter ms_dispatch 0x7fdc68001550 osd_map(133..133 src has 77..133) v4 2024-03-11T21:28:19.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 3 client.24686.objecter handle_osd_map got epochs [133,133] > 0 2024-03-11T21:28:19.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 3 client.24686.objecter handle_osd_map decoding full epoch 133 2024-03-11T21:28:19.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 20 client.24686.objecter dump_active .. 0 homeless 2024-03-11T21:28:19.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc6cca1700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 0x7fdc58071cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00a860 tx=0x7fdc5c00c650 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:19.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.031+0000 7fdc65ffb700 10 client.24686.objecter ms_handle_connect 0x7fdc5806f810 2024-03-11T21:28:19.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.039+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fdc60062ca0 con 0x7fdc68106550 2024-03-11T21:28:19.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:18 smithi203 bash[23654]: cluster 2024-03-11T21:28:17.157839+0000 mgr.y (mgr.14550) 196 : cluster 0 pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:18 smithi203 bash[23654]: cluster 2024-03-11T21:28:18.149867+0000 osd.7 (osd.7) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:18 smithi203 bash[23654]: cluster 2024-03-11T21:28:18.149974+0000 osd.7 (osd.7) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:18 smithi203 bash[23654]: audit 2024-03-11T21:28:18.469279+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.143:0/3003615208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:19.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- 172.21.15.143:0/412232540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c1060a0 msgr2=0x7ffa2c10dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- 172.21.15.143:0/412232540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c1060a0 0x7ffa2c10dc70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b0d0 tx=0x7ffa2400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- 172.21.15.143:0/412232540 shutdown_connections 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- 172.21.15.143:0/412232540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c1060a0 0x7ffa2c10dc70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- 172.21.15.143:0/412232540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c100fc0 0x7ffa2c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- 172.21.15.143:0/412232540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c1059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- 172.21.15.143:0/412232540 >> 172.21.15.143:0/412232540 conn(0x7ffa2c077de0 msgr2=0x7ffa2c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- 172.21.15.143:0/412232540 shutdown_connections 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- 172.21.15.143:0/412232540 wait complete. 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 librados: starting msgr at 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 librados: starting objecter 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 Processor -- start 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 -- start start 2024-03-11T21:28:19.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 librados: setting wanted keys 2024-03-11T21:28:19.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 librados: calling monclient init 2024-03-11T21:28:19.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.063+0000 7ffa3182e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:60108/0 (socket says 172.21.15.143:60108) 2024-03-11T21:28:19.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2a7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2a7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.143:51840/0 (socket says 172.21.15.143:51840) 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:36708/0 (socket says 172.21.15.143:36708) 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1105e0 con 0x7ffa2c100fc0 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 -- 172.21.15.143:0/870866960 learned_addr learned my addr 172.21.15.143:0/870866960 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c110460 con 0x7ffa2c101970 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c110760 con 0x7ffa2c1060a0 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 -- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 msgr2=0x7ffa2c197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 -- 172.21.15.143:0/870866960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 msgr2=0x7ffa2c196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c19ccb0 con 0x7ffa2c100fc0 2024-03-11T21:28:19.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2b7fe700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:28:19.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2a7fc700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:28:19.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa2affd700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00eb30 tx=0x7ffa1c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3082c700 10 client.?.objecter ms_handle_connect 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3082c700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00cbd0 con 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c19ff80 con 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 10 client.14811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c1a0510 con 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3182e700 1 librados: init done 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ffa1c0044a0 con 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.067+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003ce0 con 0x7ffa2c100fc0 2024-03-11T21:28:19.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa18005120 con 0x7ffa2c100fc0 2024-03-11T21:28:19.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ffa1c003e80 con 0x7ffa2c100fc0 2024-03-11T21:28:19.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 0x7ffa14071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(133..133 src has 77..133) v4 ==== 7450+0+0 (secure 0 0 0) 0x7ffa1c08e530 con 0x7ffa2c100fc0 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 10 client.14811.objecter ms_dispatch 0x7ffa2c001550 osd_map(133..133 src has 77..133) v4 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 3 client.14811.objecter handle_osd_map got epochs [133,133] > 0 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 3 client.14811.objecter handle_osd_map decoding full epoch 133 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa2a7fc700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 0x7ffa14071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 20 client.14811.objecter dump_active .. 0 homeless 2024-03-11T21:28:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa2a7fc700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 0x7ffa14071ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c1074e0 tx=0x7ffa200073d0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:19.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.071+0000 7ffa3082c700 10 client.14811.objecter ms_handle_connect 0x7ffa1406f820 2024-03-11T21:28:19.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.083+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ffa1c05ba30 con 0x7ffa2c100fc0 2024-03-11T21:28:19.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:19.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:19.231 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:19.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.323+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fdc68196eb0 con 0x7fdc68106550 2024-03-11T21:28:19.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:19.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.483+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa18000bc0 con 0x7ffa1406f820 2024-03-11T21:28:19.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.495+0000 7ffa3082c700 1 -- 172.21.15.143:0/870866960 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345593 (secure 0 0 0) 0x7ffa18000bc0 con 0x7ffa1406f820 2024-03-11T21:28:19.473 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:28:19.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 10 librados: watch_flush enter 2024-03-11T21:28:19.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 10 librados: watch_flush exit 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 20 client.14811.objecter shutdown clearing up homeless session... 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 10 client.14811.objecter successfully canceled tick 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 msgr2=0x7ffa14071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 0x7ffa14071ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c1074e0 tx=0x7ffa200073d0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 msgr2=0x7ffa2c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.499+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00eb30 tx=0x7ffa1c00ee40 comp rx=0 tx=0).stop 2024-03-11T21:28:19.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 shutdown_connections 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ffa1406f820 0x7ffa14071ce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa2c100fc0 0x7ffa2c1968d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa2c1060a0 0x7ffa2c197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 --2- 172.21.15.143:0/870866960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa2c101970 0x7ffa2c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 >> 172.21.15.143:0/870866960 conn(0x7ffa2c077de0 msgr2=0x7ffa2c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:19.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.503+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 shutdown_connections 2024-03-11T21:28:19.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.511+0000 7ffa3182e700 1 -- 172.21.15.143:0/870866960 wait complete. 2024-03-11T21:28:19.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.511+0000 7ffa3182e700 1 librados: shutdown 2024-03-11T21:28:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:28:19.498 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:19.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.626 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:19.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.715+0000 7fdc65ffb700 1 -- 172.21.15.143:0/814896263 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v134) v1 ==== 91+0+0 (secure 0 0 0) 0x7fdc600593b0 con 0x7fdc68106550 2024-03-11T21:28:19.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.719+0000 7fdc6ef05700 10 librados: watch_flush enter 2024-03-11T21:28:19.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.723+0000 7fdc6ef05700 10 librados: watch_flush exit 2024-03-11T21:28:19.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.723+0000 7fdc6ef05700 20 client.24686.objecter shutdown clearing up homeless session... 2024-03-11T21:28:19.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.723+0000 7fdc6ef05700 10 client.24686.objecter successfully canceled tick 2024-03-11T21:28:19.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 msgr2=0x7fdc58071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 0x7fdc58071cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00a860 tx=0x7fdc5c00c650 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 msgr2=0x7fdc68195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000b970 tx=0x7fdc6000bc80 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 shutdown_connections 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc5806f810 0x7fdc58071cd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc681004a0 0x7fdc6819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc68106550 0x7fdc68195980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 --2- 172.21.15.143:0/814896263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc68100e60 0x7fdc6819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:19.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.727+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 >> 172.21.15.143:0/814896263 conn(0x7fdc680772e0 msgr2=0x7fdc68107260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:19.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.731+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 shutdown_connections 2024-03-11T21:28:19.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.735+0000 7fdc6ef05700 1 -- 172.21.15.143:0/814896263 wait complete. 2024-03-11T21:28:19.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:19.735+0000 7fdc6ef05700 1 librados: shutdown 2024-03-11T21:28:19.708 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:28:19.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:19.758 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:19.861 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[29284]: audit 2024-03-11T21:28:18.293048+0000 mgr.y (mgr.14550) 197 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[29284]: cluster 2024-03-11T21:28:18.707862+0000 mon.a (mon.0) 904 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[29284]: audit 2024-03-11T21:28:19.326092+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.143:0/814896263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[29284]: audit 2024-03-11T21:28:19.326576+0000 mon.a (mon.0) 905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[21512]: audit 2024-03-11T21:28:18.293048+0000 mgr.y (mgr.14550) 197 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:19.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[21512]: cluster 2024-03-11T21:28:18.707862+0000 mon.a (mon.0) 904 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:28:19.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[21512]: audit 2024-03-11T21:28:19.326092+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.143:0/814896263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:19.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:19 smithi143 bash[21512]: audit 2024-03-11T21:28:19.326576+0000 mon.a (mon.0) 905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:19.886 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:19.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:20.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:20.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:19 smithi203 bash[23654]: audit 2024-03-11T21:28:18.293048+0000 mgr.y (mgr.14550) 197 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:20.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:19 smithi203 bash[23654]: cluster 2024-03-11T21:28:18.707862+0000 mon.a (mon.0) 904 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:28:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:19 smithi203 bash[23654]: audit 2024-03-11T21:28:19.326092+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.143:0/814896263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:19 smithi203 bash[23654]: audit 2024-03-11T21:28:19.326576+0000 mon.a (mon.0) 905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:20.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.147 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:20.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:20.278 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:20.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.410 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:20.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:20.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:20.672 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:20.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:20.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[21512]: cluster 2024-03-11T21:28:19.158306+0000 mgr.y (mgr.14550) 198 : cluster 0 pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[21512]: audit 2024-03-11T21:28:19.486396+0000 mgr.y (mgr.14550) 199 : audit 0 from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[21512]: cluster 2024-03-11T21:28:19.713255+0000 mon.a (mon.0) 906 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[21512]: audit 2024-03-11T21:28:19.714964+0000 mon.a (mon.0) 907 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[21512]: cluster 2024-03-11T21:28:19.718251+0000 mon.a (mon.0) 908 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[29284]: cluster 2024-03-11T21:28:19.158306+0000 mgr.y (mgr.14550) 198 : cluster 0 pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[29284]: audit 2024-03-11T21:28:19.486396+0000 mgr.y (mgr.14550) 199 : audit 0 from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[29284]: cluster 2024-03-11T21:28:19.713255+0000 mon.a (mon.0) 906 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:20.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[29284]: audit 2024-03-11T21:28:19.714964+0000 mon.a (mon.0) 907 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:20.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:20 smithi143 bash[29284]: cluster 2024-03-11T21:28:19.718251+0000 mon.a (mon.0) 908 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:28:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.936 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:20.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:21.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:20 smithi203 bash[23654]: cluster 2024-03-11T21:28:19.158306+0000 mgr.y (mgr.14550) 198 : cluster 0 pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:21.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:20 smithi203 bash[23654]: audit 2024-03-11T21:28:19.486396+0000 mgr.y (mgr.14550) 199 : audit 0 from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:21.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:20 smithi203 bash[23654]: cluster 2024-03-11T21:28:19.713255+0000 mon.a (mon.0) 906 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:20 smithi203 bash[23654]: audit 2024-03-11T21:28:19.714964+0000 mon.a (mon.0) 907 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:20 smithi203 bash[23654]: cluster 2024-03-11T21:28:19.718251+0000 mon.a (mon.0) 908 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:28:21.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:21.060 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:21.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:21.182 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:21.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:21.304 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-03-11T21:28:19.158027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1245,"num_read_kb":1091,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":257148,"kb_used_data":7708,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":749704068,"statfs":{"total":767960285184,"available":767696965632,"internally_reserved":0,"allocated":7892992,"data_stored":5198864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002841"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788845+0000","last_change":"2024-03-11T21:26:27.355463+0000","last_active":"2024-03-11T21:28:07.788845+0000","last_peered":"2024-03-11T21:28:07.788845+0000","last_clean":"2024-03-11T21:28:07.788845+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:28:07.788845+0000","last_undegraded":"2024-03-11T21:28:07.788845+0000","last_fullsized":"2024-03-11T21:28:07.788845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.355103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.355103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.213198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663179+0000","last_change":"2024-03-11T21:27:11.017133+0000","last_active":"2024-03-11T21:28:07.663179+0000","last_peered":"2024-03-11T21:28:07.663179+0000","last_clean":"2024-03-11T21:28:07.663179+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:28:07.663179+0000","last_undegraded":"2024-03-11T21:28:07.663179+0000","last_fullsized":"2024-03-11T21:28:07.663179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.016822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.016822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.129662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.811282+0000","last_change":"2024-03-11T21:26:03.337169+0000","last_active":"2024-03-11T21:28:07.811282+0000","last_peered":"2024-03-11T21:28:07.811282+0000","last_clean":"2024-03-11T21:28:07.811282+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:28:07.811282+0000","last_undegraded":"2024-03-11T21:28:07.811282+0000","last_fullsized":"2024-03-11T21:28:07.811282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:26:03.337010+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:26:03.337010+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790078+0000","last_change":"2024-03-11T21:26:49.246965+0000","last_active":"2024-03-11T21:28:07.790078+0000","last_peered":"2024-03-11T21:28:07.790078+0000","last_clean":"2024-03-11T21:28:07.790078+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:28:07.790078+0000","last_undegraded":"2024-03-11T21:28:07.790078+0000","last_fullsized":"2024-03-11T21:28:07.790078+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:49.246842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.246842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.231482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663524+0000","last_change":"2024-03-11T21:27:22.792568+0000","last_active":"2024-03-11T21:28:07.663524+0000","last_peered":"2024-03-11T21:28:07.663524+0000","last_clean":"2024-03-11T21:28:07.663524+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:28:07.663524+0000","last_undegraded":"2024-03-11T21:28:07.663524+0000","last_fullsized":"2024-03-11T21:28:07.663524+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.792362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.792362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.770903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668056+0000","last_change":"2024-03-11T21:26:13.570113+0000","last_active":"2024-03-11T21:28:07.668056+0000","last_peered":"2024-03-11T21:28:07.668056+0000","last_clean":"2024-03-11T21:28:07.668056+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:28:07.668056+0000","last_undegraded":"2024-03-11T21:28:07.668056+0000","last_fullsized":"2024-03-11T21:28:07.668056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.569895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.569895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790250+0000","last_change":"2024-03-11T21:27:32.238387+0000","last_active":"2024-03-11T21:28:07.790250+0000","last_peered":"2024-03-11T21:28:07.790250+0000","last_clean":"2024-03-11T21:28:07.790250+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:28:07.790250+0000","last_undegraded":"2024-03-11T21:28:07.790250+0000","last_fullsized":"2024-03-11T21:28:07.790250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.238293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.238293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:39.004649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"121'16","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.811206+0000","last_change":"2024-03-11T21:26:51.608487+0000","last_active":"2024-03-11T21:28:07.811206+0000","last_peered":"2024-03-11T21:28:07.811206+0000","last_clean":"2024-03-11T21:28:07.811206+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:28:07.811206+0000","last_undegraded":"2024-03-11T21:28:07.811206+0000","last_fullsized":"2024-03-11T21:28:07.811206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.608221+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.211682+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806039+0000","last_change":"2024-03-11T21:25:57.398253+0000","last_active":"2024-03-11T21:28:07.806039+0000","last_peered":"2024-03-11T21:28:07.806039+0000","last_clean":"2024-03-11T21:28:07.806039+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:28:07.806039+0000","last_undegraded":"2024-03-11T21:28:07.806039+0000","last_fullsized":"2024-03-11T21:28:07.806039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:25:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.184520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806364+0000","last_change":"2024-03-11T21:27:18.708546+0000","last_active":"2024-03-11T21:28:07.806364+0000","last_peered":"2024-03-11T21:28:07.806364+0000","last_clean":"2024-03-11T21:28:07.806364+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:28:07.806364+0000","last_undegraded":"2024-03-11T21:28:07.806364+0000","last_fullsized":"2024-03-11T21:28:07.806364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.708450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.708450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:43.764001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807653+0000","last_change":"2024-03-11T21:26:35.765693+0000","last_active":"2024-03-11T21:28:07.807653+0000","last_peered":"2024-03-11T21:28:07.807653+0000","last_clean":"2024-03-11T21:28:07.807653+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:28:07.807653+0000","last_undegraded":"2024-03-11T21:28:07.807653+0000","last_fullsized":"2024-03-11T21:28:07.807653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:35.765486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:35.765486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.027848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808957+0000","last_change":"2024-03-11T21:26:31.763473+0000","last_active":"2024-03-11T21:28:07.808957+0000","last_peered":"2024-03-11T21:28:07.808957+0000","last_clean":"2024-03-11T21:28:07.808957+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:28:07.808957+0000","last_undegraded":"2024-03-11T21:28:07.808957+0000","last_fullsized":"2024-03-11T21:28:07.808957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.763327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810373+0000","last_change":"2024-03-11T21:27:22.541333+0000","last_active":"2024-03-11T21:28:07.810373+0000","last_peered":"2024-03-11T21:28:07.810373+0000","last_clean":"2024-03-11T21:28:07.810373+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:28:07.810373+0000","last_undegraded":"2024-03-11T21:28:07.810373+0000","last_fullsized":"2024-03-11T21:28:07.810373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.541117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.541117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.740379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807956+0000","last_change":"2024-03-11T21:26:11.457112+0000","last_active":"2024-03-11T21:28:07.807956+0000","last_peered":"2024-03-11T21:28:07.807956+0000","last_clean":"2024-03-11T21:28:07.807956+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:28:07.807956+0000","last_undegraded":"2024-03-11T21:28:07.807956+0000","last_fullsized":"2024-03-11T21:28:07.807956+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:11.456900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.456900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.184420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806199+0000","last_change":"2024-03-11T21:27:08.632675+0000","last_active":"2024-03-11T21:28:07.806199+0000","last_peered":"2024-03-11T21:28:07.806199+0000","last_clean":"2024-03-11T21:28:07.806199+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:28:07.806199+0000","last_undegraded":"2024-03-11T21:28:07.806199+0000","last_fullsized":"2024-03-11T21:28:07.806199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.632381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.632381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.000117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808755+0000","last_change":"2024-03-11T21:26:59.726007+0000","last_active":"2024-03-11T21:28:07.808755+0000","last_peered":"2024-03-11T21:28:07.808755+0000","last_clean":"2024-03-11T21:28:07.808755+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:28:07.808755+0000","last_undegraded":"2024-03-11T21:28:07.808755+0000","last_fullsized":"2024-03-11T21:28:07.808755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:59.725612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.725612+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.676623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790537+0000","last_change":"2024-03-11T21:26:20.277298+0000","last_active":"2024-03-11T21:28:07.790537+0000","last_peered":"2024-03-11T21:28:07.790537+0000","last_clean":"2024-03-11T21:28:07.790537+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:28:07.790537+0000","last_undegraded":"2024-03-11T21:28:07.790537+0000","last_fullsized":"2024-03-11T21:28:07.790537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.277140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.664468+0000","last_change":"2024-03-11T21:27:32.754174+0000","last_active":"2024-03-11T21:28:07.664468+0000","last_peered":"2024-03-11T21:28:07.664468+0000","last_clean":"2024-03-11T21:28:07.664468+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:28:07.664468+0000","last_undegraded":"2024-03-11T21:28:07.664468+0000","last_fullsized":"2024-03-11T21:28:07.664468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.753964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:45.941808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788956+0000","last_change":"2024-03-11T21:26:39.219283+0000","last_active":"2024-03-11T21:28:07.788956+0000","last_peered":"2024-03-11T21:28:07.788956+0000","last_clean":"2024-03-11T21:28:07.788956+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:28:07.788956+0000","last_undegraded":"2024-03-11T21:28:07.788956+0000","last_fullsized":"2024-03-11T21:28:07.788956+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:26:39.218970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663676+0000","last_change":"2024-03-11T21:27:04.011882+0000","last_active":"2024-03-11T21:28:07.663676+0000","last_peered":"2024-03-11T21:28:07.663676+0000","last_clean":"2024-03-11T21:28:07.663676+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:28:07.663676+0000","last_undegraded":"2024-03-11T21:28:07.663676+0000","last_fullsized":"2024-03-11T21:28:07.663676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:27:04.011634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.011634+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.665552+0000","last_change":"2024-03-11T21:27:10.026903+0000","last_active":"2024-03-11T21:28:07.665552+0000","last_peered":"2024-03-11T21:28:07.665552+0000","last_clean":"2024-03-11T21:28:07.665552+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:28:07.665552+0000","last_undegraded":"2024-03-11T21:28:07.665552+0000","last_fullsized":"2024-03-11T21:28:07.665552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.026738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.026738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.995934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.670282+0000","last_change":"2024-03-11T21:26:00.765533+0000","last_active":"2024-03-11T21:28:07.670282+0000","last_peered":"2024-03-11T21:28:07.670282+0000","last_clean":"2024-03-11T21:28:07.670282+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:28:07.670282+0000","last_undegraded":"2024-03-11T21:28:07.670282+0000","last_fullsized":"2024-03-11T21:28:07.670282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:00.765272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:26:00.765272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785493+0000","last_change":"2024-03-11T21:27:31.013122+0000","last_active":"2024-03-11T21:28:07.785493+0000","last_peered":"2024-03-11T21:28:07.785493+0000","last_clean":"2024-03-11T21:28:07.785493+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:28:07.785493+0000","last_undegraded":"2024-03-11T21:28:07.785493+0000","last_fullsized":"2024-03-11T21:28:07.785493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.013013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.013013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.473846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.811019+0000","last_change":"2024-03-11T21:26:20.455065+0000","last_active":"2024-03-11T21:28:07.811019+0000","last_peered":"2024-03-11T21:28:07.811019+0000","last_clean":"2024-03-11T21:28:07.811019+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:28:07.811019+0000","last_undegraded":"2024-03-11T21:28:07.811019+0000","last_fullsized":"2024-03-11T21:28:07.811019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.454747+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.670418+0000","last_change":"2024-03-11T21:26:09.575095+0000","last_active":"2024-03-11T21:28:07.670418+0000","last_peered":"2024-03-11T21:28:07.670418+0000","last_clean":"2024-03-11T21:28:07.670418+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:28:07.670418+0000","last_undegraded":"2024-03-11T21:28:07.670418+0000","last_fullsized":"2024-03-11T21:28:07.670418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:09.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:09.574796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810702+0000","last_change":"2024-03-11T21:27:08.631242+0000","last_active":"2024-03-11T21:28:07.810702+0000","last_peered":"2024-03-11T21:28:07.810702+0000","last_clean":"2024-03-11T21:28:07.810702+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:28:07.810702+0000","last_undegraded":"2024-03-11T21:28:07.810702+0000","last_fullsized":"2024-03-11T21:28:07.810702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.631010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.631010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:38.452241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788653+0000","last_change":"2024-03-11T21:26:41.250247+0000","last_active":"2024-03-11T21:28:07.788653+0000","last_peered":"2024-03-11T21:28:07.788653+0000","last_clean":"2024-03-11T21:28:07.788653+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:28:07.788653+0000","last_undegraded":"2024-03-11T21:28:07.788653+0000","last_fullsized":"2024-03-11T21:28:07.788653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:26:41.250134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.250134+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788927+0000","last_change":"2024-03-11T21:26:38.192925+0000","last_active":"2024-03-11T21:28:07.788927+0000","last_peered":"2024-03-11T21:28:07.788927+0000","last_clean":"2024-03-11T21:28:07.788927+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:28:07.788927+0000","last_undegraded":"2024-03-11T21:28:07.788927+0000","last_fullsized":"2024-03-11T21:28:07.788927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:38.192555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.192555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":129,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807548+0000","last_change":"2024-03-11T21:26:29.967551+0000","last_active":"2024-03-11T21:28:07.807548+0000","last_peered":"2024-03-11T21:28:07.807548+0000","last_clean":"2024-03-11T21:28:07.807548+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:28:07.807548+0000","last_undegraded":"2024-03-11T21:28:07.807548+0000","last_fullsized":"2024-03-11T21:28:07.807548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:26:29.967316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.967316+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.112310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.227522+0000","last_change":"2024-03-11T21:27:08.856139+0000","last_active":"2024-03-11T21:28:08.227522+0000","last_peered":"2024-03-11T21:28:08.227522+0000","last_clean":"2024-03-11T21:28:08.227522+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:28:08.227522+0000","last_undegraded":"2024-03-11T21:28:08.227522+0000","last_fullsized":"2024-03-11T21:28:08.227522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:08.855901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.855901+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.699794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805696+0000","last_change":"2024-03-11T21:27:29.967422+0000","last_active":"2024-03-11T21:28:07.805696+0000","last_peered":"2024-03-11T21:28:07.805696+0000","last_clean":"2024-03-11T21:28:07.805696+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:28:07.805696+0000","last_undegraded":"2024-03-11T21:28:07.805696+0000","last_fullsized":"2024-03-11T21:28:07.805696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.967200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.967200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:53.877336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668218+0000","last_change":"2024-03-11T21:27:28.074983+0000","last_active":"2024-03-11T21:28:07.668218+0000","last_peered":"2024-03-11T21:28:07.668218+0000","last_clean":"2024-03-11T21:28:07.668218+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:28:07.668218+0000","last_undegraded":"2024-03-11T21:28:07.668218+0000","last_fullsized":"2024-03-11T21:28:07.668218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:27:28.074655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:28.074655+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.971833+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.789294+0000","last_change":"2024-03-11T21:26:28.313721+0000","last_active":"2024-03-11T21:28:07.789294+0000","last_peered":"2024-03-11T21:28:07.789294+0000","last_clean":"2024-03-11T21:28:07.789294+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:28:07.789294+0000","last_undegraded":"2024-03-11T21:28:07.789294+0000","last_fullsized":"2024-03-11T21:28:07.789294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:26:28.313489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806264+0000","last_change":"2024-03-11T21:26:20.986742+0000","last_active":"2024-03-11T21:28:07.806264+0000","last_peered":"2024-03-11T21:28:07.806264+0000","last_clean":"2024-03-11T21:28:07.806264+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:28:07.806264+0000","last_undegraded":"2024-03-11T21:28:07.806264+0000","last_fullsized":"2024-03-11T21:28:07.806264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:20.986534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:20.986534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.786796+0000","last_change":"2024-03-11T21:26:48.997049+0000","last_active":"2024-03-11T21:28:07.786796+0000","last_peered":"2024-03-11T21:28:07.786796+0000","last_clean":"2024-03-11T21:28:07.786796+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:28:07.786796+0000","last_undegraded":"2024-03-11T21:28:07.786796+0000","last_fullsized":"2024-03-11T21:28:07.786796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.996626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.996626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.276693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808562+0000","last_change":"2024-03-11T21:26:50.749938+0000","last_active":"2024-03-11T21:28:07.808562+0000","last_peered":"2024-03-11T21:28:07.808562+0000","last_clean":"2024-03-11T21:28:07.808562+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:28:07.808562+0000","last_undegraded":"2024-03-11T21:28:07.808562+0000","last_fullsized":"2024-03-11T21:28:07.808562+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:50.749697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.749697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.985992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805728+0000","last_change":"2024-03-11T21:27:20.660153+0000","last_active":"2024-03-11T21:28:07.805728+0000","last_peered":"2024-03-11T21:28:07.805728+0000","last_clean":"2024-03-11T21:28:07.805728+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:28:07.805728+0000","last_undegraded":"2024-03-11T21:28:07.805728+0000","last_fullsized":"2024-03-11T21:28:07.805728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.660038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.660038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.672421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785258+0000","last_change":"2024-03-11T21:26:29.972923+0000","last_active":"2024-03-11T21:28:07.785258+0000","last_peered":"2024-03-11T21:28:07.785258+0000","last_clean":"2024-03-11T21:28:07.785258+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:28:07.785258+0000","last_undegraded":"2024-03-11T21:28:07.785258+0000","last_fullsized":"2024-03-11T21:28:07.785258+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.972683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809235+0000","last_change":"2024-03-11T21:26:51.718223+0000","last_active":"2024-03-11T21:28:07.809235+0000","last_peered":"2024-03-11T21:28:07.809235+0000","last_clean":"2024-03-11T21:28:07.809235+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:28:07.809235+0000","last_undegraded":"2024-03-11T21:28:07.809235+0000","last_fullsized":"2024-03-11T21:28:07.809235+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.718008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.718008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.184156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":152,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.811111+0000","last_change":"2024-03-11T21:26:52.592522+0000","last_active":"2024-03-11T21:28:07.811111+0000","last_peered":"2024-03-11T21:28:07.811111+0000","last_clean":"2024-03-11T21:28:07.811111+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:28:07.811111+0000","last_undegraded":"2024-03-11T21:28:07.811111+0000","last_fullsized":"2024-03-11T21:28:07.811111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:26:52.592275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.592275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788536+0000","last_change":"2024-03-11T21:27:03.215324+0000","last_active":"2024-03-11T21:28:07.788536+0000","last_peered":"2024-03-11T21:28:07.788536+0000","last_clean":"2024-03-11T21:28:07.788536+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:28:07.788536+0000","last_undegraded":"2024-03-11T21:28:07.788536+0000","last_fullsized":"2024-03-11T21:28:07.788536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.215038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.969404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668803+0000","last_change":"2024-03-11T21:26:14.593548+0000","last_active":"2024-03-11T21:28:07.668803+0000","last_peered":"2024-03-11T21:28:07.668803+0000","last_clean":"2024-03-11T21:28:07.668803+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:28:07.668803+0000","last_undegraded":"2024-03-11T21:28:07.668803+0000","last_fullsized":"2024-03-11T21:28:07.668803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.593259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.593259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669999+0000","last_change":"2024-03-11T21:27:05.957467+0000","last_active":"2024-03-11T21:28:07.669999+0000","last_peered":"2024-03-11T21:28:07.669999+0000","last_clean":"2024-03-11T21:28:07.669999+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:28:07.669999+0000","last_undegraded":"2024-03-11T21:28:07.669999+0000","last_fullsized":"2024-03-11T21:28:07.669999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.957246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.957246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.013602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.664816+0000","last_change":"2024-03-11T21:26:54.979484+0000","last_active":"2024-03-11T21:28:07.664816+0000","last_peered":"2024-03-11T21:28:07.664816+0000","last_clean":"2024-03-11T21:28:07.664816+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:28:07.664816+0000","last_undegraded":"2024-03-11T21:28:07.664816+0000","last_fullsized":"2024-03-11T21:28:07.664816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:26:54.979250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.979250+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.137828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809915+0000","last_change":"2024-03-11T21:27:07.661169+0000","last_active":"2024-03-11T21:28:07.809915+0000","last_peered":"2024-03-11T21:28:07.809915+0000","last_clean":"2024-03-11T21:28:07.809915+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:28:07.809915+0000","last_undegraded":"2024-03-11T21:28:07.809915+0000","last_fullsized":"2024-03-11T21:28:07.809915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.660940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.825652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.789132+0000","last_change":"2024-03-11T21:27:19.230032+0000","last_active":"2024-03-11T21:28:07.789132+0000","last_peered":"2024-03-11T21:28:07.789132+0000","last_clean":"2024-03-11T21:28:07.789132+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:28:07.789132+0000","last_undegraded":"2024-03-11T21:28:07.789132+0000","last_fullsized":"2024-03-11T21:28:07.789132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.229916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.229916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.157420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785905+0000","last_change":"2024-03-11T21:26:54.928990+0000","last_active":"2024-03-11T21:28:07.785905+0000","last_peered":"2024-03-11T21:28:07.785905+0000","last_clean":"2024-03-11T21:28:07.785905+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:28:07.785905+0000","last_undegraded":"2024-03-11T21:28:07.785905+0000","last_fullsized":"2024-03-11T21:28:07.785905+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.928741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809265+0000","last_change":"2024-03-11T21:26:54.618935+0000","last_active":"2024-03-11T21:28:07.809265+0000","last_peered":"2024-03-11T21:28:07.809265+0000","last_clean":"2024-03-11T21:28:07.809265+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:28:07.809265+0000","last_undegraded":"2024-03-11T21:28:07.809265+0000","last_fullsized":"2024-03-11T21:28:07.809265+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:26:54.618687+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.618687+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.713156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.665368+0000","last_change":"2024-03-11T21:26:58.903854+0000","last_active":"2024-03-11T21:28:07.665368+0000","last_peered":"2024-03-11T21:28:07.665368+0000","last_clean":"2024-03-11T21:28:07.665368+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:28:07.665368+0000","last_undegraded":"2024-03-11T21:28:07.665368+0000","last_fullsized":"2024-03-11T21:28:07.665368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.903611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.903611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.500162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":213,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.233156+0000","last_change":"2024-03-11T21:26:56.252517+0000","last_active":"2024-03-11T21:28:08.233156+0000","last_peered":"2024-03-11T21:28:08.233156+0000","last_clean":"2024-03-11T21:28:08.233156+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:28:08.233156+0000","last_undegraded":"2024-03-11T21:28:08.233156+0000","last_fullsized":"2024-03-11T21:28:08.233156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.252414+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.142273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810921+0000","last_change":"2024-03-11T21:26:55.654571+0000","last_active":"2024-03-11T21:28:07.810921+0000","last_peered":"2024-03-11T21:28:07.810921+0000","last_clean":"2024-03-11T21:28:07.810921+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:28:07.810921+0000","last_undegraded":"2024-03-11T21:28:07.810921+0000","last_fullsized":"2024-03-11T21:28:07.810921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.654463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.654463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.184726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810149+0000","last_change":"2024-03-11T21:27:32.551291+0000","last_active":"2024-03-11T21:28:07.810149+0000","last_peered":"2024-03-11T21:28:07.810149+0000","last_clean":"2024-03-11T21:28:07.810149+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:28:07.810149+0000","last_undegraded":"2024-03-11T21:28:07.810149+0000","last_fullsized":"2024-03-11T21:28:07.810149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:27:32.551180+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:26:01.314615+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.551180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:57.389984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810603+0000","last_change":"2024-03-11T21:27:05.686646+0000","last_active":"2024-03-11T21:28:07.810603+0000","last_peered":"2024-03-11T21:28:07.810603+0000","last_clean":"2024-03-11T21:28:07.810603+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:28:07.810603+0000","last_undegraded":"2024-03-11T21:28:07.810603+0000","last_fullsized":"2024-03-11T21:28:07.810603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.686435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:05.686435+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.686435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.382738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.228619+0000","last_change":"2024-03-11T21:27:25.007663+0000","last_active":"2024-03-11T21:28:08.228619+0000","last_peered":"2024-03-11T21:28:08.228619+0000","last_clean":"2024-03-11T21:28:08.228619+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:28:08.228619+0000","last_undegraded":"2024-03-11T21:28:08.228619+0000","last_fullsized":"2024-03-11T21:28:08.228619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:25.007427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.007427+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.826870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806911+0000","last_change":"2024-03-11T21:26:27.752411+0000","last_active":"2024-03-11T21:28:07.806911+0000","last_peered":"2024-03-11T21:28:07.806911+0000","last_clean":"2024-03-11T21:28:07.806911+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:28:07.806911+0000","last_undegraded":"2024-03-11T21:28:07.806911+0000","last_fullsized":"2024-03-11T21:28:07.806911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:27.752198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:27.752198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.184694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806996+0000","last_change":"2024-03-11T21:26:50.713963+0000","last_active":"2024-03-11T21:28:07.806996+0000","last_peered":"2024-03-11T21:28:07.806996+0000","last_clean":"2024-03-11T21:28:07.806996+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:28:07.806996+0000","last_undegraded":"2024-03-11T21:28:07.806996+0000","last_fullsized":"2024-03-11T21:28:07.806996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:50.713726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.713726+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.027950+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807082+0000","last_change":"2024-03-11T21:27:13.605988+0000","last_active":"2024-03-11T21:28:07.807082+0000","last_peered":"2024-03-11T21:28:07.807082+0000","last_clean":"2024-03-11T21:28:07.807082+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:28:07.807082+0000","last_undegraded":"2024-03-11T21:28:07.807082+0000","last_fullsized":"2024-03-11T21:28:07.807082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:13.605762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:13.605762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.100312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808896+0000","last_change":"2024-03-11T21:26:57.676747+0000","last_active":"2024-03-11T21:28:07.808896+0000","last_peered":"2024-03-11T21:28:07.808896+0000","last_clean":"2024-03-11T21:28:07.808896+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:28:07.808896+0000","last_undegraded":"2024-03-11T21:28:07.808896+0000","last_fullsized":"2024-03-11T21:28:07.808896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.676465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.676465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785269+0000","last_change":"2024-03-11T21:26:43.871224+0000","last_active":"2024-03-11T21:28:07.785269+0000","last_peered":"2024-03-11T21:28:07.785269+0000","last_clean":"2024-03-11T21:28:07.785269+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:28:07.785269+0000","last_undegraded":"2024-03-11T21:28:07.785269+0000","last_fullsized":"2024-03-11T21:28:07.785269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:43.871005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:43.871005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785044+0000","last_change":"2024-03-11T21:26:56.922599+0000","last_active":"2024-03-11T21:28:07.785044+0000","last_peered":"2024-03-11T21:28:07.785044+0000","last_clean":"2024-03-11T21:28:07.785044+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:28:07.785044+0000","last_undegraded":"2024-03-11T21:28:07.785044+0000","last_fullsized":"2024-03-11T21:28:07.785044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:26:56.922358+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.922358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.954469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668290+0000","last_change":"2024-03-11T21:27:02.848853+0000","last_active":"2024-03-11T21:28:07.668290+0000","last_peered":"2024-03-11T21:28:07.668290+0000","last_clean":"2024-03-11T21:28:07.668290+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:28:07.668290+0000","last_undegraded":"2024-03-11T21:28:07.668290+0000","last_fullsized":"2024-03-11T21:28:07.668290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.848551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.848551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.623625+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790621+0000","last_change":"2024-03-11T21:27:20.200413+0000","last_active":"2024-03-11T21:28:07.790621+0000","last_peered":"2024-03-11T21:28:07.790621+0000","last_clean":"2024-03-11T21:28:07.790621+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:28:07.790621+0000","last_undegraded":"2024-03-11T21:28:07.790621+0000","last_fullsized":"2024-03-11T21:28:07.790621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.200196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.200196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.303367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.666860+0000","last_change":"2024-03-11T21:26:45.975343+0000","last_active":"2024-03-11T21:28:07.666860+0000","last_peered":"2024-03-11T21:28:07.666860+0000","last_clean":"2024-03-11T21:28:07.666860+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:28:07.666860+0000","last_undegraded":"2024-03-11T21:28:07.666860+0000","last_fullsized":"2024-03-11T21:28:07.666860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.975124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805879+0000","last_change":"2024-03-11T21:27:31.666712+0000","last_active":"2024-03-11T21:28:07.805879+0000","last_peered":"2024-03-11T21:28:07.805879+0000","last_clean":"2024-03-11T21:28:07.805879+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:28:07.805879+0000","last_undegraded":"2024-03-11T21:28:07.805879+0000","last_fullsized":"2024-03-11T21:28:07.805879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:31.666521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.666521+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:39.549841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.789211+0000","last_change":"2024-03-11T21:26:57.223304+0000","last_active":"2024-03-11T21:28:07.789211+0000","last_peered":"2024-03-11T21:28:07.789211+0000","last_clean":"2024-03-11T21:28:07.789211+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:28:07.789211+0000","last_undegraded":"2024-03-11T21:28:07.789211+0000","last_fullsized":"2024-03-11T21:28:07.789211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.223046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.223046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.664493+0000","last_change":"2024-03-11T21:26:49.041325+0000","last_active":"2024-03-11T21:28:07.664493+0000","last_peered":"2024-03-11T21:28:07.664493+0000","last_clean":"2024-03-11T21:28:07.664493+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:28:07.664493+0000","last_undegraded":"2024-03-11T21:28:07.664493+0000","last_fullsized":"2024-03-11T21:28:07.664493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.041020+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.587235+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806988+0000","last_change":"2024-03-11T21:27:19.700859+0000","last_active":"2024-03-11T21:28:07.806988+0000","last_peered":"2024-03-11T21:28:07.806988+0000","last_clean":"2024-03-11T21:28:07.806988+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:28:07.806988+0000","last_undegraded":"2024-03-11T21:28:07.806988+0000","last_fullsized":"2024-03-11T21:28:07.806988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:27:19.700661+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:27:19.700661+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.700661+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:19.769280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.666757+0000","last_change":"2024-03-11T21:27:21.742470+0000","last_active":"2024-03-11T21:28:07.666757+0000","last_peered":"2024-03-11T21:28:07.666757+0000","last_clean":"2024-03-11T21:28:07.666757+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:28:07.666757+0000","last_undegraded":"2024-03-11T21:28:07.666757+0000","last_fullsized":"2024-03-11T21:28:07.666757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.742357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.742357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.709018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809923+0000","last_change":"2024-03-11T21:27:17.623720+0000","last_active":"2024-03-11T21:28:07.809923+0000","last_peered":"2024-03-11T21:28:07.809923+0000","last_clean":"2024-03-11T21:28:07.809923+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:28:07.809923+0000","last_undegraded":"2024-03-11T21:28:07.809923+0000","last_fullsized":"2024-03-11T21:28:07.809923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.623631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.623631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:47.137727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669239+0000","last_change":"2024-03-11T21:26:59.857825+0000","last_active":"2024-03-11T21:28:07.669239+0000","last_peered":"2024-03-11T21:28:07.669239+0000","last_clean":"2024-03-11T21:28:07.669239+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:28:07.669239+0000","last_undegraded":"2024-03-11T21:28:07.669239+0000","last_fullsized":"2024-03-11T21:28:07.669239+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.857611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.857611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.753621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.228644+0000","last_change":"2024-03-11T21:26:45.687511+0000","last_active":"2024-03-11T21:28:08.228644+0000","last_peered":"2024-03-11T21:28:08.228644+0000","last_clean":"2024-03-11T21:28:08.228644+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:28:08.228644+0000","last_undegraded":"2024-03-11T21:28:08.228644+0000","last_fullsized":"2024-03-11T21:28:08.228644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:26:45.687244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.687244+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.188907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668714+0000","last_change":"2024-03-11T21:25:50.631632+0000","last_active":"2024-03-11T21:28:07.668714+0000","last_peered":"2024-03-11T21:28:07.668714+0000","last_clean":"2024-03-11T21:28:07.668714+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:28:07.668714+0000","last_undegraded":"2024-03-11T21:28:07.668714+0000","last_fullsized":"2024-03-11T21:28:07.668714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:25:50.631545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:25:50.631545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.232577+0000","last_change":"2024-03-11T21:27:12.664471+0000","last_active":"2024-03-11T21:28:08.232577+0000","last_peered":"2024-03-11T21:28:08.232577+0000","last_clean":"2024-03-11T21:28:08.232577+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:28:08.232577+0000","last_undegraded":"2024-03-11T21:28:08.232577+0000","last_fullsized":"2024-03-11T21:28:08.232577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.664354+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.849172+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790168+0000","last_change":"2024-03-11T21:26:30.315059+0000","last_active":"2024-03-11T21:28:07.790168+0000","last_peered":"2024-03-11T21:28:07.790168+0000","last_clean":"2024-03-11T21:28:07.790168+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:28:07.790168+0000","last_undegraded":"2024-03-11T21:28:07.790168+0000","last_fullsized":"2024-03-11T21:28:07.790168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:30.314840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:30.314840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807076+0000","last_change":"2024-03-11T21:26:13.487586+0000","last_active":"2024-03-11T21:28:07.807076+0000","last_peered":"2024-03-11T21:28:07.807076+0000","last_clean":"2024-03-11T21:28:07.807076+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:28:07.807076+0000","last_undegraded":"2024-03-11T21:28:07.807076+0000","last_fullsized":"2024-03-11T21:28:07.807076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:13.487359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.487359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:09.184610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663783+0000","last_change":"2024-03-11T21:26:40.103016+0000","last_active":"2024-03-11T21:28:07.663783+0000","last_peered":"2024-03-11T21:28:07.663783+0000","last_clean":"2024-03-11T21:28:07.663783+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:28:07.663783+0000","last_undegraded":"2024-03-11T21:28:07.663783+0000","last_fullsized":"2024-03-11T21:28:07.663783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:26:40.102764+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.102764+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810864+0000","last_change":"2024-03-11T21:26:25.426916+0000","last_active":"2024-03-11T21:28:07.810864+0000","last_peered":"2024-03-11T21:28:07.810864+0000","last_clean":"2024-03-11T21:28:07.810864+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:28:07.810864+0000","last_undegraded":"2024-03-11T21:28:07.810864+0000","last_fullsized":"2024-03-11T21:28:07.810864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:25.426623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:25.426623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807251+0000","last_change":"2024-03-11T21:27:17.710638+0000","last_active":"2024-03-11T21:28:07.807251+0000","last_peered":"2024-03-11T21:28:07.807251+0000","last_clean":"2024-03-11T21:28:07.807251+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:28:07.807251+0000","last_undegraded":"2024-03-11T21:28:07.807251+0000","last_fullsized":"2024-03-11T21:28:07.807251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.710431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.710431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.182795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"84'18","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.667676+0000","last_change":"2024-03-11T21:27:21.121066+0000","last_active":"2024-03-11T21:28:07.667676+0000","last_peered":"2024-03-11T21:28:07.667676+0000","last_clean":"2024-03-11T21:28:07.667676+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:28:07.667676+0000","last_undegraded":"2024-03-11T21:28:07.667676+0000","last_fullsized":"2024-03-11T21:28:07.667676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-11T21:27:21.120787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.120787+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.943211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.667119+0000","last_change":"2024-03-11T21:27:04.805372+0000","last_active":"2024-03-11T21:28:07.667119+0000","last_peered":"2024-03-11T21:28:07.667119+0000","last_clean":"2024-03-11T21:28:07.667119+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:28:07.667119+0000","last_undegraded":"2024-03-11T21:28:07.667119+0000","last_fullsized":"2024-03-11T21:28:07.667119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.805100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.805100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.216891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668641+0000","last_change":"2024-03-11T21:27:12.756161+0000","last_active":"2024-03-11T21:28:07.668641+0000","last_peered":"2024-03-11T21:28:07.668641+0000","last_clean":"2024-03-11T21:28:07.668641+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:28:07.668641+0000","last_undegraded":"2024-03-11T21:28:07.668641+0000","last_fullsized":"2024-03-11T21:28:07.668641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.755925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.755925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.683631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809723+0000","last_change":"2024-03-11T21:27:14.796892+0000","last_active":"2024-03-11T21:28:07.809723+0000","last_peered":"2024-03-11T21:28:07.809723+0000","last_clean":"2024-03-11T21:28:07.809723+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:28:07.809723+0000","last_undegraded":"2024-03-11T21:28:07.809723+0000","last_fullsized":"2024-03-11T21:28:07.809723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:14.796603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.796603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.446263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.791076+0000","last_change":"2024-03-11T21:27:25.263826+0000","last_active":"2024-03-11T21:28:07.791076+0000","last_peered":"2024-03-11T21:28:07.791076+0000","last_clean":"2024-03-11T21:28:07.791076+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:28:07.791076+0000","last_undegraded":"2024-03-11T21:28:07.791076+0000","last_fullsized":"2024-03-11T21:28:07.791076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:25.263676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.263676+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.297940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807672+0000","last_change":"2024-03-11T21:26:52.732913+0000","last_active":"2024-03-11T21:28:07.807672+0000","last_peered":"2024-03-11T21:28:07.807672+0000","last_clean":"2024-03-11T21:28:07.807672+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:28:07.807672+0000","last_undegraded":"2024-03-11T21:28:07.807672+0000","last_fullsized":"2024-03-11T21:28:07.807672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.732676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.732676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:19.391370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":401,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805739+0000","last_change":"2024-03-11T21:26:36.758336+0000","last_active":"2024-03-11T21:28:07.805739+0000","last_peered":"2024-03-11T21:28:07.805739+0000","last_clean":"2024-03-11T21:28:07.805739+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:28:07.805739+0000","last_undegraded":"2024-03-11T21:28:07.805739+0000","last_fullsized":"2024-03-11T21:28:07.805739+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:26:36.758088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:26:36.758088+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.028045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807123+0000","last_change":"2024-03-11T21:26:34.807309+0000","last_active":"2024-03-11T21:28:07.807123+0000","last_peered":"2024-03-11T21:28:07.807123+0000","last_clean":"2024-03-11T21:28:07.807123+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:28:07.807123+0000","last_undegraded":"2024-03-11T21:28:07.807123+0000","last_fullsized":"2024-03-11T21:28:07.807123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:34.807061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:34.807061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.027995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669016+0000","last_change":"2024-03-11T21:26:10.602526+0000","last_active":"2024-03-11T21:28:07.669016+0000","last_peered":"2024-03-11T21:28:07.669016+0000","last_clean":"2024-03-11T21:28:07.669016+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:28:07.669016+0000","last_undegraded":"2024-03-11T21:28:07.669016+0000","last_fullsized":"2024-03-11T21:28:07.669016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:10.602230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:10.602230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088195+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810209+0000","last_change":"2024-03-11T21:26:13.337394+0000","last_active":"2024-03-11T21:28:07.810209+0000","last_peered":"2024-03-11T21:28:07.810209+0000","last_clean":"2024-03-11T21:28:07.810209+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:28:07.810209+0000","last_undegraded":"2024-03-11T21:28:07.810209+0000","last_fullsized":"2024-03-11T21:28:07.810209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:26:13.337090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:13.337090+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805554+0000","last_change":"2024-03-11T21:27:06.618092+0000","last_active":"2024-03-11T21:28:07.805554+0000","last_peered":"2024-03-11T21:28:07.805554+0000","last_clean":"2024-03-11T21:28:07.805554+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:28:07.805554+0000","last_undegraded":"2024-03-11T21:28:07.805554+0000","last_fullsized":"2024-03-11T21:28:07.805554+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.617820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.617820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.715234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809457+0000","last_change":"2024-03-11T21:27:19.924377+0000","last_active":"2024-03-11T21:28:07.809457+0000","last_peered":"2024-03-11T21:28:07.809457+0000","last_clean":"2024-03-11T21:28:07.809457+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:28:07.809457+0000","last_undegraded":"2024-03-11T21:28:07.809457+0000","last_fullsized":"2024-03-11T21:28:07.809457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.924170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.924170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.337440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.667968+0000","last_change":"2024-03-11T21:26:04.760536+0000","last_active":"2024-03-11T21:28:07.667968+0000","last_peered":"2024-03-11T21:28:07.667968+0000","last_clean":"2024-03-11T21:28:07.667968+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:28:07.667968+0000","last_undegraded":"2024-03-11T21:28:07.667968+0000","last_fullsized":"2024-03-11T21:28:07.667968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:04.760262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:04.760262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807372+0000","last_change":"2024-03-11T21:27:07.638125+0000","last_active":"2024-03-11T21:28:07.807372+0000","last_peered":"2024-03-11T21:28:07.807372+0000","last_clean":"2024-03-11T21:28:07.807372+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:28:07.807372+0000","last_undegraded":"2024-03-11T21:28:07.807372+0000","last_fullsized":"2024-03-11T21:28:07.807372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.637915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.637915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.738623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":256,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.233703+0000","last_change":"2024-03-11T21:26:40.725235+0000","last_active":"2024-03-11T21:28:08.233703+0000","last_peered":"2024-03-11T21:28:08.233703+0000","last_clean":"2024-03-11T21:28:08.233703+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:28:08.233703+0000","last_undegraded":"2024-03-11T21:28:08.233703+0000","last_fullsized":"2024-03-11T21:28:08.233703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:26:40.724980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.724980+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:18.150084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785665+0000","last_change":"2024-03-11T21:26:08.930659+0000","last_active":"2024-03-11T21:28:07.785665+0000","last_peered":"2024-03-11T21:28:07.785665+0000","last_clean":"2024-03-11T21:28:07.785665+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:28:07.785665+0000","last_undegraded":"2024-03-11T21:28:07.785665+0000","last_fullsized":"2024-03-11T21:28:07.785665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:08.930415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:08.930415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663590+0000","last_change":"2024-03-11T21:27:23.772704+0000","last_active":"2024-03-11T21:28:07.663590+0000","last_peered":"2024-03-11T21:28:07.663590+0000","last_clean":"2024-03-11T21:28:07.663590+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:28:07.663590+0000","last_undegraded":"2024-03-11T21:28:07.663590+0000","last_fullsized":"2024-03-11T21:28:07.663590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.772573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.772573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.153346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669893+0000","last_change":"2024-03-11T21:27:09.022813+0000","last_active":"2024-03-11T21:28:07.669893+0000","last_peered":"2024-03-11T21:28:07.669893+0000","last_clean":"2024-03-11T21:28:07.669893+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:28:07.669893+0000","last_undegraded":"2024-03-11T21:28:07.669893+0000","last_fullsized":"2024-03-11T21:28:07.669893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:27:09.022585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.022585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.995050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.790702+0000","last_change":"2024-03-11T21:26:24.352686+0000","last_active":"2024-03-11T21:28:07.790702+0000","last_peered":"2024-03-11T21:28:07.790702+0000","last_clean":"2024-03-11T21:28:07.790702+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:28:07.790702+0000","last_undegraded":"2024-03-11T21:28:07.790702+0000","last_fullsized":"2024-03-11T21:28:07.790702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:26:24.352407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:24.352407+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":446,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:16.303158+0000","last_change":"2024-03-11T21:27:06.215128+0000","last_active":"2024-03-11T21:28:16.303158+0000","last_peered":"2024-03-11T21:28:16.303158+0000","last_clean":"2024-03-11T21:28:16.303158+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:28:16.303158+0000","last_undegraded":"2024-03-11T21:28:16.303158+0000","last_fullsized":"2024-03-11T21:28:16.303158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:27:06.214816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.214816+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.841750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663306+0000","last_change":"2024-03-11T21:26:59.951453+0000","last_active":"2024-03-11T21:28:07.663306+0000","last_peered":"2024-03-11T21:28:07.663306+0000","last_clean":"2024-03-11T21:28:07.663306+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:28:07.663306+0000","last_undegraded":"2024-03-11T21:28:07.663306+0000","last_fullsized":"2024-03-11T21:28:07.663306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.951239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.951239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.079618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806426+0000","last_change":"2024-03-11T21:26:55.658685+0000","last_active":"2024-03-11T21:28:07.806426+0000","last_peered":"2024-03-11T21:28:07.806426+0000","last_clean":"2024-03-11T21:28:07.806426+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:28:07.806426+0000","last_undegraded":"2024-03-11T21:28:07.806426+0000","last_fullsized":"2024-03-11T21:28:07.806426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:26:55.658468+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.658468+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.668238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.786663+0000","last_change":"2024-03-11T21:27:12.894010+0000","last_active":"2024-03-11T21:28:07.786663+0000","last_peered":"2024-03-11T21:28:07.786663+0000","last_clean":"2024-03-11T21:28:07.786663+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:28:07.786663+0000","last_undegraded":"2024-03-11T21:28:07.786663+0000","last_fullsized":"2024-03-11T21:28:07.786663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.893784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.231554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810909+0000","last_change":"2024-03-11T21:26:32.529005+0000","last_active":"2024-03-11T21:28:07.810909+0000","last_peered":"2024-03-11T21:28:07.810909+0000","last_clean":"2024-03-11T21:28:07.810909+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:28:07.810909+0000","last_undegraded":"2024-03-11T21:28:07.810909+0000","last_fullsized":"2024-03-11T21:28:07.810909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:26:32.528639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:32.528639+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663609+0000","last_change":"2024-03-11T21:27:19.913202+0000","last_active":"2024-03-11T21:28:07.663609+0000","last_peered":"2024-03-11T21:28:07.663609+0000","last_clean":"2024-03-11T21:28:07.663609+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:28:07.663609+0000","last_undegraded":"2024-03-11T21:28:07.663609+0000","last_fullsized":"2024-03-11T21:28:07.663609+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.913032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.913032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.839406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.232760+0000","last_change":"2024-03-11T21:26:52.285556+0000","last_active":"2024-03-11T21:28:08.232760+0000","last_peered":"2024-03-11T21:28:08.232760+0000","last_clean":"2024-03-11T21:28:08.232760+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:28:08.232760+0000","last_undegraded":"2024-03-11T21:28:08.232760+0000","last_fullsized":"2024-03-11T21:28:08.232760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:52.285152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.285152+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.213264+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.789443+0000","last_change":"2024-03-11T21:26:14.197969+0000","last_active":"2024-03-11T21:28:07.789443+0000","last_peered":"2024-03-11T21:28:07.789443+0000","last_clean":"2024-03-11T21:28:07.789443+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:28:07.789443+0000","last_undegraded":"2024-03-11T21:28:07.789443+0000","last_fullsized":"2024-03-11T21:28:07.789443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:14.197767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:26:14.197767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809628+0000","last_change":"2024-03-11T21:27:03.708920+0000","last_active":"2024-03-11T21:28:07.809628+0000","last_peered":"2024-03-11T21:28:07.809628+0000","last_clean":"2024-03-11T21:28:07.809628+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:28:07.809628+0000","last_undegraded":"2024-03-11T21:28:07.809628+0000","last_fullsized":"2024-03-11T21:28:07.809628+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.708595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.708595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.018137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805836+0000","last_change":"2024-03-11T21:27:25.634056+0000","last_active":"2024-03-11T21:28:07.805836+0000","last_peered":"2024-03-11T21:28:07.805836+0000","last_clean":"2024-03-11T21:28:07.805836+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:28:07.805836+0000","last_undegraded":"2024-03-11T21:28:07.805836+0000","last_fullsized":"2024-03-11T21:28:07.805836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.633909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.849575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.663362+0000","last_change":"2024-03-11T21:26:17.927107+0000","last_active":"2024-03-11T21:28:07.663362+0000","last_peered":"2024-03-11T21:28:07.663362+0000","last_clean":"2024-03-11T21:28:07.663362+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:28:07.663362+0000","last_undegraded":"2024-03-11T21:28:07.663362+0000","last_fullsized":"2024-03-11T21:28:07.663362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:26:17.926863+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:26:17.926863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.088478+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808123+0000","last_change":"2024-03-11T21:27:08.792122+0000","last_active":"2024-03-11T21:28:07.808123+0000","last_peered":"2024-03-11T21:28:07.808123+0000","last_clean":"2024-03-11T21:28:07.808123+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:28:07.808123+0000","last_undegraded":"2024-03-11T21:28:07.808123+0000","last_fullsized":"2024-03-11T21:28:07.808123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.791885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.791885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.094147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.228196+0000","last_change":"2024-03-11T21:27:21.840085+0000","last_active":"2024-03-11T21:28:08.228196+0000","last_peered":"2024-03-11T21:28:08.228196+0000","last_clean":"2024-03-11T21:28:08.228196+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:28:08.228196+0000","last_undegraded":"2024-03-11T21:28:08.228196+0000","last_fullsized":"2024-03-11T21:28:08.228196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:27:21.839970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.839970+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.941185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668137+0000","last_change":"2024-03-11T21:26:41.718271+0000","last_active":"2024-03-11T21:28:07.668137+0000","last_peered":"2024-03-11T21:28:07.668137+0000","last_clean":"2024-03-11T21:28:07.668137+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:28:07.668137+0000","last_undegraded":"2024-03-11T21:28:07.668137+0000","last_fullsized":"2024-03-11T21:28:07.668137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.718071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.718071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.788766+0000","last_change":"2024-03-11T21:27:12.163374+0000","last_active":"2024-03-11T21:28:07.788766+0000","last_peered":"2024-03-11T21:28:07.788766+0000","last_clean":"2024-03-11T21:28:07.788766+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:28:07.788766+0000","last_undegraded":"2024-03-11T21:28:07.788766+0000","last_fullsized":"2024-03-11T21:28:07.788766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:27:12.163097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.163097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.333821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.665401+0000","last_change":"2024-03-11T21:27:27.825989+0000","last_active":"2024-03-11T21:28:07.665401+0000","last_peered":"2024-03-11T21:28:07.665401+0000","last_clean":"2024-03-11T21:28:07.665401+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:28:07.665401+0000","last_undegraded":"2024-03-11T21:28:07.665401+0000","last_fullsized":"2024-03-11T21:28:07.665401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:27.825842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.825842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.417136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":213,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.233117+0000","last_change":"2024-03-11T21:26:49.911665+0000","last_active":"2024-03-11T21:28:08.233117+0000","last_peered":"2024-03-11T21:28:08.233117+0000","last_clean":"2024-03-11T21:28:08.233117+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:28:08.233117+0000","last_undegraded":"2024-03-11T21:28:08.233117+0000","last_fullsized":"2024-03-11T21:28:08.233117+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:26:49.911420+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.911420+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:16.757842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668517+0000","last_change":"2024-03-11T21:26:07.649240+0000","last_active":"2024-03-11T21:28:07.668517+0000","last_peered":"2024-03-11T21:28:07.668517+0000","last_clean":"2024-03-11T21:28:07.668517+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:28:07.668517+0000","last_undegraded":"2024-03-11T21:28:07.668517+0000","last_fullsized":"2024-03-11T21:28:07.668517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:07.649133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:26:07.649133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.088107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.789044+0000","last_change":"2024-03-11T21:26:48.278251+0000","last_active":"2024-03-11T21:28:07.789044+0000","last_peered":"2024-03-11T21:28:07.789044+0000","last_clean":"2024-03-11T21:28:07.789044+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:28:07.789044+0000","last_undegraded":"2024-03-11T21:28:07.789044+0000","last_fullsized":"2024-03-11T21:28:07.789044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:26:48.277915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.277915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669134+0000","last_change":"2024-03-11T21:27:07.817089+0000","last_active":"2024-03-11T21:28:07.669134+0000","last_peered":"2024-03-11T21:28:07.669134+0000","last_clean":"2024-03-11T21:28:07.669134+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:28:07.669134+0000","last_undegraded":"2024-03-11T21:28:07.669134+0000","last_fullsized":"2024-03-11T21:28:07.669134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.816760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.816760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.238678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.669101+0000","last_change":"2024-03-11T21:27:32.095523+0000","last_active":"2024-03-11T21:28:07.669101+0000","last_peered":"2024-03-11T21:28:07.669101+0000","last_clean":"2024-03-11T21:28:07.669101+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:28:07.669101+0000","last_undegraded":"2024-03-11T21:28:07.669101+0000","last_fullsized":"2024-03-11T21:28:07.669101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.095381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.095381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:35.477286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806730+0000","last_change":"2024-03-11T21:26:46.780598+0000","last_active":"2024-03-11T21:28:07.806730+0000","last_peered":"2024-03-11T21:28:07.806730+0000","last_clean":"2024-03-11T21:28:07.806730+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:28:07.806730+0000","last_undegraded":"2024-03-11T21:28:07.806730+0000","last_fullsized":"2024-03-11T21:28:07.806730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.780383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.780383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":183,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.785289+0000","last_change":"2024-03-11T21:27:06.797303+0000","last_active":"2024-03-11T21:28:07.785289+0000","last_peered":"2024-03-11T21:28:07.785289+0000","last_clean":"2024-03-11T21:28:07.785289+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:28:07.785289+0000","last_undegraded":"2024-03-11T21:28:07.785289+0000","last_fullsized":"2024-03-11T21:28:07.785289+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:27:06.797049+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.797049+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.836292+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.805388+0000","last_change":"2024-03-11T21:27:26.598269+0000","last_active":"2024-03-11T21:28:07.805388+0000","last_peered":"2024-03-11T21:28:07.805388+0000","last_clean":"2024-03-11T21:28:07.805388+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:28:07.805388+0000","last_undegraded":"2024-03-11T21:28:07.805388+0000","last_fullsized":"2024-03-11T21:28:07.805388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:26.598056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:26.598056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:47.232807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.667629+0000","last_change":"2024-03-11T21:26:38.936837+0000","last_active":"2024-03-11T21:28:07.667629+0000","last_peered":"2024-03-11T21:28:07.667629+0000","last_clean":"2024-03-11T21:28:07.667629+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:28:07.667629+0000","last_undegraded":"2024-03-11T21:28:07.667629+0000","last_fullsized":"2024-03-11T21:28:07.667629+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.936573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.936573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668012+0000","last_change":"2024-03-11T21:26:59.822204+0000","last_active":"2024-03-11T21:28:07.668012+0000","last_peered":"2024-03-11T21:28:07.668012+0000","last_clean":"2024-03-11T21:28:07.668012+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:28:07.668012+0000","last_undegraded":"2024-03-11T21:28:07.668012+0000","last_fullsized":"2024-03-11T21:28:07.668012+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.821996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.821996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":179,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.807904+0000","last_change":"2024-03-11T21:27:32.698984+0000","last_active":"2024-03-11T21:28:07.807904+0000","last_peered":"2024-03-11T21:28:07.807904+0000","last_clean":"2024-03-11T21:28:07.807904+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:28:07.807904+0000","last_undegraded":"2024-03-11T21:28:07.807904+0000","last_fullsized":"2024-03-11T21:28:07.807904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:27:32.698821+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.698821+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.463525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.810038+0000","last_change":"2024-03-11T21:27:31.506697+0000","last_active":"2024-03-11T21:28:07.810038+0000","last_peered":"2024-03-11T21:28:07.810038+0000","last_clean":"2024-03-11T21:28:07.810038+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:28:07.810038+0000","last_undegraded":"2024-03-11T21:28:07.810038+0000","last_fullsized":"2024-03-11T21:28:07.810038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.506459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:53.521326+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.506459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:56.741802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.667438+0000","last_change":"2024-03-11T21:26:41.906852+0000","last_active":"2024-03-11T21:28:07.667438+0000","last_peered":"2024-03-11T21:28:07.667438+0000","last_clean":"2024-03-11T21:28:07.667438+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:28:07.667438+0000","last_undegraded":"2024-03-11T21:28:07.667438+0000","last_fullsized":"2024-03-11T21:28:07.667438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:41.906678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:41.906678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.670162+0000","last_change":"2024-03-11T21:27:00.897454+0000","last_active":"2024-03-11T21:28:07.670162+0000","last_peered":"2024-03-11T21:28:07.670162+0000","last_clean":"2024-03-11T21:28:07.670162+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:28:07.670162+0000","last_undegraded":"2024-03-11T21:28:07.670162+0000","last_fullsized":"2024-03-11T21:28:07.670162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.897238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.897238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.879305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.808457+0000","last_change":"2024-03-11T21:27:20.912735+0000","last_active":"2024-03-11T21:28:07.808457+0000","last_peered":"2024-03-11T21:28:07.808457+0000","last_clean":"2024-03-11T21:28:07.808457+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:28:07.808457+0000","last_undegraded":"2024-03-11T21:28:07.808457+0000","last_fullsized":"2024-03-11T21:28:07.808457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:20.912377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.912377+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.495526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":135,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.806803+0000","last_change":"2024-03-11T21:27:31.990217+0000","last_active":"2024-03-11T21:28:07.806803+0000","last_peered":"2024-03-11T21:28:07.806803+0000","last_clean":"2024-03-11T21:28:07.806803+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:28:07.806803+0000","last_undegraded":"2024-03-11T21:28:07.806803+0000","last_fullsized":"2024-03-11T21:28:07.806803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.989922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.989922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:51.661495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.668859+0000","last_change":"2024-03-11T21:26:57.835977+0000","last_active":"2024-03-11T21:28:07.668859+0000","last_peered":"2024-03-11T21:28:07.668859+0000","last_clean":"2024-03-11T21:28:07.668859+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:28:07.668859+0000","last_undegraded":"2024-03-11T21:28:07.668859+0000","last_fullsized":"2024-03-11T21:28:07.668859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.835726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.835726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.590001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":160,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:07.809430+0000","last_change":"2024-03-11T21:26:31.873778+0000","last_active":"2024-03-11T21:28:07.809430+0000","last_peered":"2024-03-11T21:28:07.809430+0000","last_clean":"2024-03-11T21:28:07.809430+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:28:07.809430+0000","last_undegraded":"2024-03-11T21:28:07.809430+0000","last_fullsized":"2024-03-11T21:28:07.809430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:26:31.873666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:26:31.873666+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.027803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":204,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-11T21:28:08.227665+0000","last_change":"2024-03-11T21:26:29.009973+0000","last_active":"2024-03-11T21:28:08.227665+0000","last_peered":"2024-03-11T21:28:08.227665+0000","last_clean":"2024-03-11T21:28:08.227665+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:28:08.227665+0000","last_undegraded":"2024-03-11T21:28:08.227665+0000","last_fullsized":"2024-03-11T21:28:08.227665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:26:29.009718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:26:29.009718+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.112470+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841610,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36688,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708464,"statfs":{"total":95995035648,"available":95957467136,"internally_reserved":0,"allocated":1380352,"data_stored":1020440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168659,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716816,"statfs":{"total":95995035648,"available":95966019584,"internally_reserved":0,"allocated":1347584,"data_stored":1018220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":761856,"data_stored":427734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708484,"statfs":{"total":95995035648,"available":95957487616,"internally_reserved":0,"allocated":1359872,"data_stored":1019337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378749,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":428312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":427790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855463,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":790528,"data_stored":429259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495709,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":749568,"data_stored":427772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:28:21.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-11T21:28:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:21.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:21.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:21.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:21.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:21.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:21.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:21.774 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:21.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:21.829 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:21.932 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.127361+0000 osd.5 (osd.5) 123 : cluster 0 2.3 scrub starts 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.130101+0000 osd.5 (osd.5) 124 : cluster 0 2.3 scrub ok 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.177596+0000 osd.3 (osd.3) 131 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.177685+0000 osd.3 (osd.3) 132 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.127361+0000 osd.5 (osd.5) 123 : cluster 0 2.3 scrub starts 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.130101+0000 osd.5 (osd.5) 124 : cluster 0 2.3 scrub ok 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.177596+0000 osd.3 (osd.3) 131 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:21 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.177685+0000 osd.3 (osd.3) 132 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:21.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:22.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:21 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.127361+0000 osd.5 (osd.5) 123 : cluster 0 2.3 scrub starts 2024-03-11T21:28:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:21 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.130101+0000 osd.5 (osd.5) 124 : cluster 0 2.3 scrub ok 2024-03-11T21:28:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:21 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.177596+0000 osd.3 (osd.3) 131 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:21 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.177685+0000 osd.3 (osd.3) 132 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:22.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:22.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:28:22.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:22.194 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:22.205 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:22.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:22.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:22.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:22.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:22.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:22.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:22.730 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:22.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:22.859 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:22.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:22.973 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:22 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.158703+0000 mgr.y (mgr.14550) 200 : cluster 0 pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:22.973 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:22 smithi203 bash[23654]: cluster 2024-03-11T21:28:21.735711+0000 mon.a (mon.0) 909 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:28:22.973 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:22 smithi203 bash[23654]: cluster 2024-03-11T21:28:22.184035+0000 osd.1 (osd.1) 109 : cluster 0 2.4 scrub starts 2024-03-11T21:28:22.973 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:22 smithi203 bash[23654]: cluster 2024-03-11T21:28:22.185714+0000 osd.1 (osd.1) 110 : cluster 0 2.4 scrub ok 2024-03-11T21:28:22.989 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:23.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.158703+0000 mgr.y (mgr.14550) 200 : cluster 0 pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:23.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[29284]: cluster 2024-03-11T21:28:21.735711+0000 mon.a (mon.0) 909 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:28:23.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[29284]: cluster 2024-03-11T21:28:22.184035+0000 osd.1 (osd.1) 109 : cluster 0 2.4 scrub starts 2024-03-11T21:28:23.065 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[29284]: cluster 2024-03-11T21:28:22.185714+0000 osd.1 (osd.1) 110 : cluster 0 2.4 scrub ok 2024-03-11T21:28:23.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.158703+0000 mgr.y (mgr.14550) 200 : cluster 0 pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:23.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[21512]: cluster 2024-03-11T21:28:21.735711+0000 mon.a (mon.0) 909 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:28:23.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[21512]: cluster 2024-03-11T21:28:22.184035+0000 osd.1 (osd.1) 109 : cluster 0 2.4 scrub starts 2024-03-11T21:28:23.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:22 smithi143 bash[21512]: cluster 2024-03-11T21:28:22.185714+0000 osd.1 (osd.1) 110 : cluster 0 2.4 scrub ok 2024-03-11T21:28:23.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:23.121 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:23.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:23.246 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:23.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:28:23.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:23.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.510 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:23.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:23.639 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:23.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:23.770 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:23.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.894 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:23.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:22.985152+0000 osd.6 (osd.6) 73 : cluster 0 2.17 scrub starts 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:22.986787+0000 osd.6 (osd.6) 74 : cluster 0 2.17 scrub ok 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.126638+0000 osd.7 (osd.7) 141 : cluster 0 6.1c scrub starts 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.129593+0000 osd.7 (osd.7) 142 : cluster 0 6.1c scrub ok 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.191123+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:24.009 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:23 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.191236+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:24.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:24.024 INFO:tasks.workunit.client.0.smithi143.stderr:Note: switching to '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'. 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: git switch -c 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr:Or undo this operation with: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: git switch - 2024-03-11T21:28:24.025 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.026 INFO:tasks.workunit.client.0.smithi143.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T21:28:24.026 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-03-11T21:28:24.026 INFO:tasks.workunit.client.0.smithi143.stderr:HEAD is now at 1c349451176 Merge pull request #56077 from idryomov/wip-rbd-mirror-bash 2024-03-11T21:28:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:24.028 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T21:28:24.034 INFO:tasks.workunit.client.0.smithi143.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T21:28:24.036 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T21:28:24.036 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T21:28:24.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:24.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:22.985152+0000 osd.6 (osd.6) 73 : cluster 0 2.17 scrub starts 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:22.986787+0000 osd.6 (osd.6) 74 : cluster 0 2.17 scrub ok 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.126638+0000 osd.7 (osd.7) 141 : cluster 0 6.1c scrub starts 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.129593+0000 osd.7 (osd.7) 142 : cluster 0 6.1c scrub ok 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.191123+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.191236+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:22.985152+0000 osd.6 (osd.6) 73 : cluster 0 2.17 scrub starts 2024-03-11T21:28:24.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:22.986787+0000 osd.6 (osd.6) 74 : cluster 0 2.17 scrub ok 2024-03-11T21:28:24.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.126638+0000 osd.7 (osd.7) 141 : cluster 0 6.1c scrub starts 2024-03-11T21:28:24.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.129593+0000 osd.7 (osd.7) 142 : cluster 0 6.1c scrub ok 2024-03-11T21:28:24.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.191123+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:24.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:23 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.191236+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:24.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:24.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:24.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:24.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:24.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:24.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:24.675 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:24.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:24.802 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:24.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.159410+0000 mgr.y (mgr.14550) 201 : cluster 0 pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.749012+0000 mon.a (mon.0) 910 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.955199+0000 osd.2 (osd.2) 83 : cluster 0 5.b scrub starts 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:23.956698+0000 osd.2 (osd.2) 84 : cluster 0 5.b scrub ok 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.052807+0000 osd.4 (osd.4) 103 : cluster 0 3.9 scrub starts 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.055752+0000 osd.4 (osd.4) 104 : cluster 0 3.9 scrub ok 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.100133+0000 osd.5 (osd.5) 127 : cluster 0 5.1b scrub starts 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.101779+0000 osd.5 (osd.5) 128 : cluster 0 5.1b scrub ok 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.159410+0000 mgr.y (mgr.14550) 201 : cluster 0 pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:28:24.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.749012+0000 mon.a (mon.0) 910 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.955199+0000 osd.2 (osd.2) 83 : cluster 0 5.b scrub starts 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:23.956698+0000 osd.2 (osd.2) 84 : cluster 0 5.b scrub ok 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.052807+0000 osd.4 (osd.4) 103 : cluster 0 3.9 scrub starts 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.055752+0000 osd.4 (osd.4) 104 : cluster 0 3.9 scrub ok 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.100133+0000 osd.5 (osd.5) 127 : cluster 0 5.1b scrub starts 2024-03-11T21:28:24.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:24 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.101779+0000 osd.5 (osd.5) 128 : cluster 0 5.1b scrub ok 2024-03-11T21:28:24.933 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:25.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.159410+0000 mgr.y (mgr.14550) 201 : cluster 0 pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.749012+0000 mon.a (mon.0) 910 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.955199+0000 osd.2 (osd.2) 83 : cluster 0 5.b scrub starts 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:23.956698+0000 osd.2 (osd.2) 84 : cluster 0 5.b scrub ok 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.052807+0000 osd.4 (osd.4) 103 : cluster 0 3.9 scrub starts 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.055752+0000 osd.4 (osd.4) 104 : cluster 0 3.9 scrub ok 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.100133+0000 osd.5 (osd.5) 127 : cluster 0 5.1b scrub starts 2024-03-11T21:28:25.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:24 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.101779+0000 osd.5 (osd.5) 128 : cluster 0 5.1b scrub ok 2024-03-11T21:28:25.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:25.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:25.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:25.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:25.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:25.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.457 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:25.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:25.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.717 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:25.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:25 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.912071+0000 osd.2 (osd.2) 85 : cluster 0 2.10 scrub starts 2024-03-11T21:28:25.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:25 smithi203 bash[23654]: cluster 2024-03-11T21:28:24.913674+0000 osd.2 (osd.2) 86 : cluster 0 2.10 scrub ok 2024-03-11T21:28:25.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:25 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.015931+0000 osd.6 (osd.6) 75 : cluster 0 5.2 scrub starts 2024-03-11T21:28:25.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:25 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.018164+0000 osd.6 (osd.6) 76 : cluster 0 5.2 scrub ok 2024-03-11T21:28:25.836 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:25.837 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:25.847 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:25.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.980 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:26.006 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.912071+0000 osd.2 (osd.2) 85 : cluster 0 2.10 scrub starts 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[29284]: cluster 2024-03-11T21:28:24.913674+0000 osd.2 (osd.2) 86 : cluster 0 2.10 scrub ok 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.015931+0000 osd.6 (osd.6) 75 : cluster 0 5.2 scrub starts 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.018164+0000 osd.6 (osd.6) 76 : cluster 0 5.2 scrub ok 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.912071+0000 osd.2 (osd.2) 85 : cluster 0 2.10 scrub starts 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[21512]: cluster 2024-03-11T21:28:24.913674+0000 osd.2 (osd.2) 86 : cluster 0 2.10 scrub ok 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.015931+0000 osd.6 (osd.6) 75 : cluster 0 5.2 scrub starts 2024-03-11T21:28:26.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:25 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.018164+0000 osd.6 (osd.6) 76 : cluster 0 5.2 scrub ok 2024-03-11T21:28:26.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:26.103 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:26.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:26.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:26.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:26.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:26.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:26.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:26.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.621 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:26.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:26.750 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:26.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.160068+0000 mgr.y (mgr.14550) 202 : cluster 0 pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.489095+0000 osd.0 (osd.0) 109 : cluster 0 2.1f scrub starts 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.491265+0000 osd.0 (osd.0) 110 : cluster 0 2.1f scrub ok 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.750941+0000 osd.2 (osd.2) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:25.751043+0000 osd.2 (osd.2) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.022169+0000 osd.4 (osd.4) 105 : cluster 0 3.1a scrub starts 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.024987+0000 osd.4 (osd.4) 106 : cluster 0 3.1a scrub ok 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.040420+0000 osd.5 (osd.5) 129 : cluster 0 6.19 scrub starts 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.041592+0000 osd.5 (osd.5) 130 : cluster 0 6.19 scrub ok 2024-03-11T21:28:26.865 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:26 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.176034+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-03-11T21:28:26.881 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.160068+0000 mgr.y (mgr.14550) 202 : cluster 0 pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.489095+0000 osd.0 (osd.0) 109 : cluster 0 2.1f scrub starts 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.491265+0000 osd.0 (osd.0) 110 : cluster 0 2.1f scrub ok 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.750941+0000 osd.2 (osd.2) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:25.751043+0000 osd.2 (osd.2) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:26.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.022169+0000 osd.4 (osd.4) 105 : cluster 0 3.1a scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.024987+0000 osd.4 (osd.4) 106 : cluster 0 3.1a scrub ok 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.040420+0000 osd.5 (osd.5) 129 : cluster 0 6.19 scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.041592+0000 osd.5 (osd.5) 130 : cluster 0 6.19 scrub ok 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.176034+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.160068+0000 mgr.y (mgr.14550) 202 : cluster 0 pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.489095+0000 osd.0 (osd.0) 109 : cluster 0 2.1f scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.491265+0000 osd.0 (osd.0) 110 : cluster 0 2.1f scrub ok 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.750941+0000 osd.2 (osd.2) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:25.751043+0000 osd.2 (osd.2) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.022169+0000 osd.4 (osd.4) 105 : cluster 0 3.1a scrub starts 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.024987+0000 osd.4 (osd.4) 106 : cluster 0 3.1a scrub ok 2024-03-11T21:28:26.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.040420+0000 osd.5 (osd.5) 129 : cluster 0 6.19 scrub starts 2024-03-11T21:28:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.041592+0000 osd.5 (osd.5) 130 : cluster 0 6.19 scrub ok 2024-03-11T21:28:26.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:26 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.176034+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-03-11T21:28:26.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.011 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:27.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.142 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:27.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:27.270 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:27.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.402 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:27.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:27.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:27.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.661 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:27.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.790 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:27.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:27 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.180684+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-03-11T21:28:27.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:27 smithi143 bash[29284]: cluster 2024-03-11T21:28:26.759608+0000 mon.a (mon.0) 911 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:28:27.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:27 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.180684+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-03-11T21:28:27.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:27 smithi143 bash[21512]: cluster 2024-03-11T21:28:26.759608+0000 mon.a (mon.0) 911 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:28:27.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:27.921 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:27 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.180684+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-03-11T21:28:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:27 smithi203 bash[23654]: cluster 2024-03-11T21:28:26.759608+0000 mon.a (mon.0) 911 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:28:28.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:28.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:28.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.183 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:28.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:28.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.426 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:28.442 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:28.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:28.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:28.676 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:28.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:28.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:28.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:28.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:28 smithi203 bash[23654]: cluster 2024-03-11T21:28:27.160758+0000 mgr.y (mgr.14550) 203 : cluster 0 pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:28 smithi203 bash[23654]: audit 2024-03-11T21:28:27.923982+0000 mon.c (mon.2) 100 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:28 smithi203 bash[23654]: cluster 2024-03-11T21:28:28.225345+0000 osd.3 (osd.3) 133 : cluster 0 3.7 scrub starts 2024-03-11T21:28:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:28 smithi203 bash[23654]: cluster 2024-03-11T21:28:28.229410+0000 osd.3 (osd.3) 134 : cluster 0 3.7 scrub ok 2024-03-11T21:28:29.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:29.095 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[21512]: cluster 2024-03-11T21:28:27.160758+0000 mgr.y (mgr.14550) 203 : cluster 0 pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[21512]: audit 2024-03-11T21:28:27.923982+0000 mon.c (mon.2) 100 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[21512]: cluster 2024-03-11T21:28:28.225345+0000 osd.3 (osd.3) 133 : cluster 0 3.7 scrub starts 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[21512]: cluster 2024-03-11T21:28:28.229410+0000 osd.3 (osd.3) 134 : cluster 0 3.7 scrub ok 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[29284]: cluster 2024-03-11T21:28:27.160758+0000 mgr.y (mgr.14550) 203 : cluster 0 pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[29284]: audit 2024-03-11T21:28:27.923982+0000 mon.c (mon.2) 100 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[29284]: cluster 2024-03-11T21:28:28.225345+0000 osd.3 (osd.3) 133 : cluster 0 3.7 scrub starts 2024-03-11T21:28:29.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:28 smithi143 bash[29284]: cluster 2024-03-11T21:28:28.229410+0000 osd.3 (osd.3) 134 : cluster 0 3.7 scrub ok 2024-03-11T21:28:29.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:29.225 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:29.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:29.357 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:29.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:29.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:29.541 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:29.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:29.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:29.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:29.746 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:29.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:29.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: audit 2024-03-11T21:28:28.304161+0000 mgr.y (mgr.14550) 204 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:28.495573+0000 osd.0 (osd.0) 111 : cluster 0 3.1 scrub starts 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:28.500153+0000 osd.0 (osd.0) 112 : cluster 0 3.1 scrub ok 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.011333+0000 osd.6 (osd.6) 77 : cluster 0 5.11 scrub starts 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.012643+0000 osd.6 (osd.6) 78 : cluster 0 5.11 scrub ok 2024-03-11T21:28:29.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.134960+0000 osd.4 (osd.4) 107 : cluster 0 3.3 scrub starts 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.137902+0000 osd.4 (osd.4) 108 : cluster 0 3.3 scrub ok 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.238668+0000 osd.3 (osd.3) 135 : cluster 0 6.1f scrub starts 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.240770+0000 osd.3 (osd.3) 136 : cluster 0 6.1f scrub ok 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: audit 2024-03-11T21:28:28.304161+0000 mgr.y (mgr.14550) 204 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:28.495573+0000 osd.0 (osd.0) 111 : cluster 0 3.1 scrub starts 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:28.500153+0000 osd.0 (osd.0) 112 : cluster 0 3.1 scrub ok 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.011333+0000 osd.6 (osd.6) 77 : cluster 0 5.11 scrub starts 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.012643+0000 osd.6 (osd.6) 78 : cluster 0 5.11 scrub ok 2024-03-11T21:28:29.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.134960+0000 osd.4 (osd.4) 107 : cluster 0 3.3 scrub starts 2024-03-11T21:28:29.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.137902+0000 osd.4 (osd.4) 108 : cluster 0 3.3 scrub ok 2024-03-11T21:28:29.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.238668+0000 osd.3 (osd.3) 135 : cluster 0 6.1f scrub starts 2024-03-11T21:28:29.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:29 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.240770+0000 osd.3 (osd.3) 136 : cluster 0 6.1f scrub ok 2024-03-11T21:28:29.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:30.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: audit 2024-03-11T21:28:28.304161+0000 mgr.y (mgr.14550) 204 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:28.495573+0000 osd.0 (osd.0) 111 : cluster 0 3.1 scrub starts 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:28.500153+0000 osd.0 (osd.0) 112 : cluster 0 3.1 scrub ok 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.011333+0000 osd.6 (osd.6) 77 : cluster 0 5.11 scrub starts 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.012643+0000 osd.6 (osd.6) 78 : cluster 0 5.11 scrub ok 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.134960+0000 osd.4 (osd.4) 107 : cluster 0 3.3 scrub starts 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.137902+0000 osd.4 (osd.4) 108 : cluster 0 3.3 scrub ok 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.238668+0000 osd.3 (osd.3) 135 : cluster 0 6.1f scrub starts 2024-03-11T21:28:30.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:29 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.240770+0000 osd.3 (osd.3) 136 : cluster 0 6.1f scrub ok 2024-03-11T21:28:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:30.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.137 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:30.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:30.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:30.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:30.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:30.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:30.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:30.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.161692+0000 mgr.y (mgr.14550) 205 : cluster 0 pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.458282+0000 osd.0 (osd.0) 113 : cluster 0 3.6 scrub starts 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.461060+0000 osd.0 (osd.0) 114 : cluster 0 3.6 scrub ok 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.963261+0000 osd.6 (osd.6) 79 : cluster 0 2.11 scrub starts 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:29.965615+0000 osd.6 (osd.6) 80 : cluster 0 2.11 scrub ok 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:30.130967+0000 osd.4 (osd.4) 109 : cluster 0 3.14 scrub starts 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[21512]: cluster 2024-03-11T21:28:30.134727+0000 osd.4 (osd.4) 110 : cluster 0 3.14 scrub ok 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.161692+0000 mgr.y (mgr.14550) 205 : cluster 0 pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:28:30.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.458282+0000 osd.0 (osd.0) 113 : cluster 0 3.6 scrub starts 2024-03-11T21:28:30.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.461060+0000 osd.0 (osd.0) 114 : cluster 0 3.6 scrub ok 2024-03-11T21:28:30.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.963261+0000 osd.6 (osd.6) 79 : cluster 0 2.11 scrub starts 2024-03-11T21:28:30.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:29.965615+0000 osd.6 (osd.6) 80 : cluster 0 2.11 scrub ok 2024-03-11T21:28:30.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:30.130967+0000 osd.4 (osd.4) 109 : cluster 0 3.14 scrub starts 2024-03-11T21:28:30.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:30 smithi143 bash[29284]: cluster 2024-03-11T21:28:30.134727+0000 osd.4 (osd.4) 110 : cluster 0 3.14 scrub ok 2024-03-11T21:28:30.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:31.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:31.063 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.161692+0000 mgr.y (mgr.14550) 205 : cluster 0 pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:28:31.063 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.458282+0000 osd.0 (osd.0) 113 : cluster 0 3.6 scrub starts 2024-03-11T21:28:31.063 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.461060+0000 osd.0 (osd.0) 114 : cluster 0 3.6 scrub ok 2024-03-11T21:28:31.064 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.963261+0000 osd.6 (osd.6) 79 : cluster 0 2.11 scrub starts 2024-03-11T21:28:31.064 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:29.965615+0000 osd.6 (osd.6) 80 : cluster 0 2.11 scrub ok 2024-03-11T21:28:31.064 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:30.130967+0000 osd.4 (osd.4) 109 : cluster 0 3.14 scrub starts 2024-03-11T21:28:31.064 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:30 smithi203 bash[23654]: cluster 2024-03-11T21:28:30.134727+0000 osd.4 (osd.4) 110 : cluster 0 3.14 scrub ok 2024-03-11T21:28:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:31.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:31.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:31.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:31.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:31.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:31.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:31.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.739 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:31.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:31.869 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:31.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[29284]: cluster 2024-03-11T21:28:30.216819+0000 osd.7 (osd.7) 145 : cluster 0 2.1d scrub starts 2024-03-11T21:28:31.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[29284]: cluster 2024-03-11T21:28:30.219143+0000 osd.7 (osd.7) 146 : cluster 0 2.1d scrub ok 2024-03-11T21:28:31.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[29284]: cluster 2024-03-11T21:28:31.135417+0000 osd.4 (osd.4) 111 : cluster 0 5.1 scrub starts 2024-03-11T21:28:31.873 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[29284]: cluster 2024-03-11T21:28:31.137780+0000 osd.4 (osd.4) 112 : cluster 0 5.1 scrub ok 2024-03-11T21:28:31.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:31.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:31 smithi203 bash[23654]: cluster 2024-03-11T21:28:30.216819+0000 osd.7 (osd.7) 145 : cluster 0 2.1d scrub starts 2024-03-11T21:28:31.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:31 smithi203 bash[23654]: cluster 2024-03-11T21:28:30.219143+0000 osd.7 (osd.7) 146 : cluster 0 2.1d scrub ok 2024-03-11T21:28:31.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:31 smithi203 bash[23654]: cluster 2024-03-11T21:28:31.135417+0000 osd.4 (osd.4) 111 : cluster 0 5.1 scrub starts 2024-03-11T21:28:31.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:31 smithi203 bash[23654]: cluster 2024-03-11T21:28:31.137780+0000 osd.4 (osd.4) 112 : cluster 0 5.1 scrub ok 2024-03-11T21:28:31.999 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:32.034 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[21512]: cluster 2024-03-11T21:28:30.216819+0000 osd.7 (osd.7) 145 : cluster 0 2.1d scrub starts 2024-03-11T21:28:32.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[21512]: cluster 2024-03-11T21:28:30.219143+0000 osd.7 (osd.7) 146 : cluster 0 2.1d scrub ok 2024-03-11T21:28:32.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[21512]: cluster 2024-03-11T21:28:31.135417+0000 osd.4 (osd.4) 111 : cluster 0 5.1 scrub starts 2024-03-11T21:28:32.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:31 smithi143 bash[21512]: cluster 2024-03-11T21:28:31.137780+0000 osd.4 (osd.4) 112 : cluster 0 5.1 scrub ok 2024-03-11T21:28:32.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:32.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:32.260 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:32.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.389 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:32.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:32.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.522 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:32.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:32.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:32.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:32.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:32.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21512]: cluster 2024-03-11T21:28:31.162277+0000 mgr.y (mgr.14550) 206 : cluster 0 pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:32.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21512]: cluster 2024-03-11T21:28:31.448281+0000 osd.0 (osd.0) 115 : cluster 0 3.1b scrub starts 2024-03-11T21:28:32.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21512]: cluster 2024-03-11T21:28:31.452031+0000 osd.0 (osd.0) 116 : cluster 0 3.1b scrub ok 2024-03-11T21:28:32.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.157190+0000 osd.4 (osd.4) 113 : cluster 0 6.17 scrub starts 2024-03-11T21:28:32.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.158812+0000 osd.4 (osd.4) 114 : cluster 0 6.17 scrub ok 2024-03-11T21:28:32.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[29284]: cluster 2024-03-11T21:28:31.162277+0000 mgr.y (mgr.14550) 206 : cluster 0 pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:32.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[29284]: cluster 2024-03-11T21:28:31.448281+0000 osd.0 (osd.0) 115 : cluster 0 3.1b scrub starts 2024-03-11T21:28:32.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[29284]: cluster 2024-03-11T21:28:31.452031+0000 osd.0 (osd.0) 116 : cluster 0 3.1b scrub ok 2024-03-11T21:28:32.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.157190+0000 osd.4 (osd.4) 113 : cluster 0 6.17 scrub starts 2024-03-11T21:28:32.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:32 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.158812+0000 osd.4 (osd.4) 114 : cluster 0 6.17 scrub ok 2024-03-11T21:28:32.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:33.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:32 smithi203 bash[23654]: cluster 2024-03-11T21:28:31.162277+0000 mgr.y (mgr.14550) 206 : cluster 0 pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:32 smithi203 bash[23654]: cluster 2024-03-11T21:28:31.448281+0000 osd.0 (osd.0) 115 : cluster 0 3.1b scrub starts 2024-03-11T21:28:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:32 smithi203 bash[23654]: cluster 2024-03-11T21:28:31.452031+0000 osd.0 (osd.0) 116 : cluster 0 3.1b scrub ok 2024-03-11T21:28:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:32 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.157190+0000 osd.4 (osd.4) 113 : cluster 0 6.17 scrub starts 2024-03-11T21:28:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:32 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.158812+0000 osd.4 (osd.4) 114 : cluster 0 6.17 scrub ok 2024-03-11T21:28:33.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:33.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.164 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:33.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.288 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:33.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.413 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:33.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:33.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:33.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.667 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:33.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.799 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:33.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:33.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.237142+0000 osd.7 (osd.7) 147 : cluster 0 5.10 scrub starts 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.238870+0000 osd.7 (osd.7) 148 : cluster 0 5.10 scrub ok 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.482287+0000 osd.0 (osd.0) 117 : cluster 0 5.5 scrub starts 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: cluster 2024-03-11T21:28:32.484777+0000 osd.0 (osd.0) 118 : cluster 0 5.5 scrub ok 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: audit 2024-03-11T21:28:32.821008+0000 mon.c (mon.2) 101 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: audit 2024-03-11T21:28:33.502244+0000 mon.c (mon.2) 102 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: audit 2024-03-11T21:28:33.503354+0000 mon.c (mon.2) 103 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:33.914 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:33 smithi203 bash[23654]: audit 2024-03-11T21:28:33.511534+0000 mon.a (mon.0) 912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:28:33.930 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:34.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.237142+0000 osd.7 (osd.7) 147 : cluster 0 5.10 scrub starts 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.238870+0000 osd.7 (osd.7) 148 : cluster 0 5.10 scrub ok 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.482287+0000 osd.0 (osd.0) 117 : cluster 0 5.5 scrub starts 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: cluster 2024-03-11T21:28:32.484777+0000 osd.0 (osd.0) 118 : cluster 0 5.5 scrub ok 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: audit 2024-03-11T21:28:32.821008+0000 mon.c (mon.2) 101 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: audit 2024-03-11T21:28:33.502244+0000 mon.c (mon.2) 102 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: audit 2024-03-11T21:28:33.503354+0000 mon.c (mon.2) 103 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:34.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[29284]: audit 2024-03-11T21:28:33.511534+0000 mon.a (mon.0) 912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.237142+0000 osd.7 (osd.7) 147 : cluster 0 5.10 scrub starts 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.238870+0000 osd.7 (osd.7) 148 : cluster 0 5.10 scrub ok 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.482287+0000 osd.0 (osd.0) 117 : cluster 0 5.5 scrub starts 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: cluster 2024-03-11T21:28:32.484777+0000 osd.0 (osd.0) 118 : cluster 0 5.5 scrub ok 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: audit 2024-03-11T21:28:32.821008+0000 mon.c (mon.2) 101 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: audit 2024-03-11T21:28:33.502244+0000 mon.c (mon.2) 102 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: audit 2024-03-11T21:28:33.503354+0000 mon.c (mon.2) 103 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:34.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:33 smithi143 bash[21512]: audit 2024-03-11T21:28:33.511534+0000 mon.a (mon.0) 912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:28:34.057 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:34.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:34.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:34.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:34.373 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T21:28:34.422 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:34.450 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:34.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.581 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:34.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:34.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:34.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[21512]: cluster 2024-03-11T21:28:33.163638+0000 mgr.y (mgr.14550) 207 : cluster 0 pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.107591+0000 osd.4 (osd.4) 115 : cluster 0 5.18 scrub starts 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.109721+0000 osd.4 (osd.4) 116 : cluster 0 5.18 scrub ok 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.292136+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.295185+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:28:34.824 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[29284]: cluster 2024-03-11T21:28:33.163638+0000 mgr.y (mgr.14550) 207 : cluster 0 pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:34.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.107591+0000 osd.4 (osd.4) 115 : cluster 0 5.18 scrub starts 2024-03-11T21:28:34.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.109721+0000 osd.4 (osd.4) 116 : cluster 0 5.18 scrub ok 2024-03-11T21:28:34.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.292136+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:28:34.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:34 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.295185+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:28:34.840 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:34.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:34 smithi203 bash[23654]: cluster 2024-03-11T21:28:33.163638+0000 mgr.y (mgr.14550) 207 : cluster 0 pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:28:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:34 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.107591+0000 osd.4 (osd.4) 115 : cluster 0 5.18 scrub starts 2024-03-11T21:28:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:34 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.109721+0000 osd.4 (osd.4) 116 : cluster 0 5.18 scrub ok 2024-03-11T21:28:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:34 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.292136+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:28:35.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:34 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.295185+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:28:35.040 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T21:28:35.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:35.101 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:35.181 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T21:28:35.184 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T21:28:35.184 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T21:28:35.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:35.231 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:35.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:35.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:35.372 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T21:28:35.412 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-11T21:28:35.412 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T21:28:35.464 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-11T21:28:35.465 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-11T21:28:35.465 DEBUG:teuthology.orchestra.run.smithi143:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-11T21:28:35.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ parallel=1 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' '' = --serial ']' 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ crimson=0 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' '' = --crimson ']' 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ color= 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -t 1 ']' 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-11T21:28:35.523 INFO:tasks.workunit.client.0.smithi143.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-11T21:28:35.524 INFO:tasks.workunit.client.0.smithi143.stderr:+ declare -A pids 2024-03-11T21:28:35.524 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.524 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.525 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_aio 2024-03-11T21:28:35.525 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_aio' 2024-03-11T21:28:35.525 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_aio 2024-03-11T21:28:35.525 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.526 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_aio 2024-03-11T21:28:35.526 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71589 2024-03-11T21:28:35.527 INFO:tasks.workunit.client.0.smithi143.stdout:test api_aio on pid 71589 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_aio on pid 71589' 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71589 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.528 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_aio_pp 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_aio_pp' 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_aio_pp 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-11T21:28:35.529 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_aio: /' 2024-03-11T21:28:35.530 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_aio_pp 2024-03-11T21:28:35.530 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71597 2024-03-11T21:28:35.530 INFO:tasks.workunit.client.0.smithi143.stdout:test api_aio_pp on pid 71597 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_aio_pp on pid 71597' 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71597 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_io 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_io' 2024-03-11T21:28:35.532 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_io 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-11T21:28:35.533 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_io 2024-03-11T21:28:35.534 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71607 2024-03-11T21:28:35.534 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_io on pid 71607' 2024-03-11T21:28:35.534 INFO:tasks.workunit.client.0.smithi143.stdout:test api_io on pid 71607 2024-03-11T21:28:35.535 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71607 2024-03-11T21:28:35.535 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_io_pp 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_io_pp' 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_io_pp 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_io.log 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-11T21:28:35.536 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_io: /' 2024-03-11T21:28:35.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.537 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_io_pp 2024-03-11T21:28:35.537 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71615 2024-03-11T21:28:35.537 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_io_pp on pid 71615' 2024-03-11T21:28:35.537 INFO:tasks.workunit.client.0.smithi143.stdout:test api_io_pp on pid 71615 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71615 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_asio 2024-03-11T21:28:35.538 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_asio' 2024-03-11T21:28:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:35.540 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_asio 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_asio 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71623 2024-03-11T21:28:35.541 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_asio on pid 71623' 2024-03-11T21:28:35.542 INFO:tasks.workunit.client.0.smithi143.stdout:test api_asio on pid 71623 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71623 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_list 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_list' 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.543 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_list 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_asio: /' 2024-03-11T21:28:35.544 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-11T21:28:35.545 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_list 2024-03-11T21:28:35.545 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71631 2024-03-11T21:28:35.545 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_list on pid 71631' 2024-03-11T21:28:35.545 INFO:tasks.workunit.client.0.smithi143.stdout:test api_list on pid 71631 2024-03-11T21:28:35.546 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71631 2024-03-11T21:28:35.546 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.546 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.546 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-11T21:28:35.546 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_lock 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_lock' 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_list.log 2024-03-11T21:28:35.547 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_list: /' 2024-03-11T21:28:35.548 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_lock 2024-03-11T21:28:35.549 INFO:tasks.workunit.client.0.smithi143.stdout:test api_lock on pid 71639 2024-03-11T21:28:35.549 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_lock 2024-03-11T21:28:35.549 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71639 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_lock on pid 71639' 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71639 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.550 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.551 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.551 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_lock_pp 2024-03-11T21:28:35.551 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_lock_pp' 2024-03-11T21:28:35.552 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_lock_pp 2024-03-11T21:28:35.554 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-11T21:28:35.554 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.557 INFO:tasks.workunit.client.0.smithi143.stdout:test api_lock_pp on pid 71647 2024-03-11T21:28:35.557 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_lock_pp 2024-03-11T21:28:35.557 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71647 2024-03-11T21:28:35.558 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_lock_pp on pid 71647' 2024-03-11T21:28:35.558 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71647 2024-03-11T21:28:35.558 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.558 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.558 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-11T21:28:35.559 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.559 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.559 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.559 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.559 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_misc 2024-03-11T21:28:35.560 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_misc' 2024-03-11T21:28:35.560 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-11T21:28:35.563 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-11T21:28:35.571 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_lock: /' 2024-03-11T21:28:35.575 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-11T21:28:35.578 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.579 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-11T21:28:35.579 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_misc 2024-03-11T21:28:35.581 INFO:tasks.workunit.client.0.smithi143.stdout:test api_misc on pid 71702 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_misc 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71702 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_misc on pid 71702' 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71702 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.582 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.587 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_misc_pp 2024-03-11T21:28:35.588 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_misc_pp' 2024-03-11T21:28:35.588 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_misc_pp 2024-03-11T21:28:35.590 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-11T21:28:35.594 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.594 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.594 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.594 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.594 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-11T21:28:35.595 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-11T21:28:35.595 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_misc: /' 2024-03-11T21:28:35.597 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.598 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_misc_pp 2024-03-11T21:28:35.599 INFO:tasks.workunit.client.0.smithi143.stdout:test api_misc_pp on pid 71753 2024-03-11T21:28:35.599 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71753 2024-03-11T21:28:35.599 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_misc_pp on pid 71753' 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71753 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_tier_pp 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-11T21:28:35.600 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_tier_pp' 2024-03-11T21:28:35.601 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.601 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.601 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.601 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.602 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-11T21:28:35.602 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-11T21:28:35.602 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-11T21:28:35.602 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_tier_pp 2024-03-11T21:28:35.602 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.604 INFO:tasks.workunit.client.0.smithi143.stdout:test api_tier_pp on pid 71762 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_tier_pp 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71762 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_tier_pp on pid 71762' 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71762 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-11T21:28:35.605 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_pool 2024-03-11T21:28:35.606 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_pool' 2024-03-11T21:28:35.606 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.606 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.606 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.606 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.607 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.607 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-11T21:28:35.607 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-11T21:28:35.608 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-11T21:28:35.610 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_pool 2024-03-11T21:28:35.611 INFO:tasks.workunit.client.0.smithi143.stdout:test api_pool on pid 71783 2024-03-11T21:28:35.611 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_pool 2024-03-11T21:28:35.611 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71783 2024-03-11T21:28:35.611 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_pool on pid 71783' 2024-03-11T21:28:35.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71783 2024-03-11T21:28:35.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-11T21:28:35.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_pool: /' 2024-03-11T21:28:35.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-11T21:28:35.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-11T21:28:35.619 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_snapshots 2024-03-11T21:28:35.619 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_snapshots' 2024-03-11T21:28:35.620 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_snapshots 2024-03-11T21:28:35.631 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stdout:test api_snapshots on pid 71816 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_snapshots 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71816 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_snapshots on pid 71816' 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71816 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.634 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-11T21:28:35.635 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.635 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.635 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.635 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.635 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-11T21:28:35.636 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_snapshots_pp 2024-03-11T21:28:35.636 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-11T21:28:35.636 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_snapshots_pp' 2024-03-11T21:28:35.636 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_snapshots_pp 2024-03-11T21:28:35.636 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.638 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_snapshots_pp 2024-03-11T21:28:35.639 INFO:tasks.workunit.client.0.smithi143.stdout:test api_snapshots_pp on pid 71824 2024-03-11T21:28:35.639 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71824 2024-03-11T21:28:35.639 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_snapshots_pp on pid 71824' 2024-03-11T21:28:35.639 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71824 2024-03-11T21:28:35.639 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-11T21:28:35.640 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.640 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.640 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_stat 2024-03-11T21:28:35.640 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_stat' 2024-03-11T21:28:35.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.641 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_stat 2024-03-11T21:28:35.641 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.643 INFO:tasks.workunit.client.0.smithi143.stdout:test api_stat on pid 71832 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_stat 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71832 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_stat on pid 71832' 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71832 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.644 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-11T21:28:35.645 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-11T21:28:35.647 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-11T21:28:35.647 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_stat_pp 2024-03-11T21:28:35.647 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_stat_pp' 2024-03-11T21:28:35.647 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-11T21:28:35.649 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_stat_pp 2024-03-11T21:28:35.649 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.651 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_stat_pp 2024-03-11T21:28:35.652 INFO:tasks.workunit.client.0.smithi143.stdout:test api_stat_pp on pid 71850 2024-03-11T21:28:35.652 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71850 2024-03-11T21:28:35.652 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_stat_pp on pid 71850' 2024-03-11T21:28:35.652 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71850 2024-03-11T21:28:35.653 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.653 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.653 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-11T21:28:35.653 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.653 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.654 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.654 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.654 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_watch_notify 2024-03-11T21:28:35.654 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-11T21:28:35.654 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_watch_notify' 2024-03-11T21:28:35.655 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-11T21:28:35.655 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-11T21:28:35.656 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_watch_notify 2024-03-11T21:28:35.659 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-11T21:28:35.661 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.661 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.661 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.661 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.667 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.670 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_stat: /' 2024-03-11T21:28:35.670 INFO:tasks.workunit.client.0.smithi143.stdout:test api_watch_notify on pid 71888 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_watch_notify 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71888 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_watch_notify on pid 71888' 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71888 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-11T21:28:35.671 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-11T21:28:35.672 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_watch_notify_pp 2024-03-11T21:28:35.672 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_watch_notify_pp' 2024-03-11T21:28:35.678 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-11T21:28:35.679 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_watch_notify_pp 2024-03-11T21:28:35.679 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.680 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.680 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.680 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.680 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.682 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_watch_notify_pp 2024-03-11T21:28:35.682 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71924 2024-03-11T21:28:35.682 INFO:tasks.workunit.client.0.smithi143.stdout:test api_watch_notify_pp on pid 71924 2024-03-11T21:28:35.683 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_watch_notify_pp on pid 71924' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71924 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cmd 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cmd' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cmd 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.684 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.685 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-11T21:28:35.686 INFO:tasks.workunit.client.0.smithi143.stdout:test api_cmd on pid 71933 2024-03-11T21:28:35.686 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cmd 2024-03-11T21:28:35.686 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-11T21:28:35.686 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71933 2024-03-11T21:28:35.686 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cmd on pid 71933' 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71933 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-11T21:28:35.687 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.688 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cmd_pp 2024-03-11T21:28:35.688 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cmd_pp' 2024-03-11T21:28:35.688 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.689 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.689 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.689 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-11T21:28:35.689 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cmd: /' 2024-03-11T21:28:35.690 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.691 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-11T21:28:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:35.693 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.695 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cmd_pp 2024-03-11T21:28:35.696 INFO:tasks.workunit.client.0.smithi143.stdout:test api_cmd_pp on pid 71945 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cmd_pp 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71945 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cmd_pp on pid 71945' 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71945 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_service 2024-03-11T21:28:35.697 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_service' 2024-03-11T21:28:35.698 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.698 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-11T21:28:35.698 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_service 2024-03-11T21:28:35.700 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-11T21:28:35.702 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-11T21:28:35.703 INFO:tasks.workunit.client.0.smithi143.stdout:test api_service on pid 71963 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_service 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71963 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_service on pid 71963' 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71963 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.704 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-11T21:28:35.707 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_service_pp 2024-03-11T21:28:35.708 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_service_pp' 2024-03-11T21:28:35.708 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-11T21:28:35.708 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-11T21:28:35.710 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.710 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.710 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.710 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.711 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_service_pp 2024-03-11T21:28:35.711 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.711 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_service: /' 2024-03-11T21:28:35.712 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_service.log 2024-03-11T21:28:35.715 INFO:tasks.workunit.client.0.smithi143.stdout:test api_service_pp on pid 71992 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_service_pp 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71992 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_service_pp on pid 71992' 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=71992 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.716 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_c_write_operations 2024-03-11T21:28:35.717 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-11T21:28:35.717 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_c_write_operations' 2024-03-11T21:28:35.717 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.717 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.717 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.718 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.718 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-11T21:28:35.718 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-11T21:28:35.718 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-11T21:28:35.718 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_c_write_operations 2024-03-11T21:28:35.719 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.719 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-11T21:28:35.721 INFO:tasks.workunit.client.0.smithi143.stdout:test api_c_write_operations on pid 72000 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_c_write_operations 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72000 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_c_write_operations on pid 72000' 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72000 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.722 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.723 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_c_read_operations 2024-03-11T21:28:35.723 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_c_read_operations' 2024-03-11T21:28:35.725 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-11T21:28:35.727 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.727 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.727 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.727 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.728 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-11T21:28:35.731 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_c_read_operations 2024-03-11T21:28:35.735 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.735 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-11T21:28:35.738 INFO:tasks.workunit.client.0.smithi143.stdout:test api_c_read_operations on pid 72033 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_c_read_operations 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72033 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_c_read_operations on pid 72033' 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72033 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.739 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-11T21:28:35.740 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cls_remote_reads 2024-03-11T21:28:35.741 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cls_remote_reads' 2024-03-11T21:28:35.741 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-11T21:28:35.741 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cls_remote_reads 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-11T21:28:35.742 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.744 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-11T21:28:35.744 INFO:tasks.workunit.client.0.smithi143.stdout:test api_cls_remote_reads on pid 72047 2024-03-11T21:28:35.744 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cls_remote_reads 2024-03-11T21:28:35.744 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72047 2024-03-11T21:28:35.745 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cls_remote_reads on pid 72047' 2024-03-11T21:28:35.745 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72047 2024-03-11T21:28:35.745 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.745 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.745 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s list_parallel 2024-03-11T21:28:35.746 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' list_parallel' 2024-03-11T21:28:35.746 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-11T21:28:35.746 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.746 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.747 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.747 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.747 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-11T21:28:35.747 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-11T21:28:35.747 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-11T21:28:35.749 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo list_parallel 2024-03-11T21:28:35.749 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.752 INFO:tasks.workunit.client.0.smithi143.stdout:test list_parallel on pid 72059 2024-03-11T21:28:35.752 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=list_parallel 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72059 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test list_parallel on pid 72059' 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72059 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s open_pools_parallel 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' open_pools_parallel' 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo open_pools_parallel 2024-03-11T21:28:35.753 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.756 INFO:tasks.workunit.client.0.smithi143.stdout:test open_pools_parallel on pid 72065 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=open_pools_parallel 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72065 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test open_pools_parallel on pid 72065' 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72065 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s delete_pools_parallel 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-11T21:28:35.757 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' delete_pools_parallel' 2024-03-11T21:28:35.758 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.759 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.759 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.759 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.759 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.759 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo delete_pools_parallel 2024-03-11T21:28:35.760 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-11T21:28:35.760 INFO:tasks.workunit.client.0.smithi143.stdout:test delete_pools_parallel on pid 72076 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=delete_pools_parallel 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72076 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test delete_pools_parallel on pid 72076' 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72076 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-11T21:28:35.761 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s cls 2024-03-11T21:28:35.762 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' cls' 2024-03-11T21:28:35.762 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-11T21:28:35.762 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.765 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.765 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.765 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.765 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.766 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-11T21:28:35.766 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-11T21:28:35.767 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo cls 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stdout:test cls on pid 72087 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=cls 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72087 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test cls on pid 72087' 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72087 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-11T21:28:35.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-11T21:28:35.772 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ list_parallel: /' 2024-03-11T21:28:35.772 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.772 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.773 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.773 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.773 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-11T21:28:35.775 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-11T21:28:35.775 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-11T21:28:35.775 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s cmd 2024-03-11T21:28:35.775 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-11T21:28:35.775 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' cmd' 2024-03-11T21:28:35.776 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo cmd 2024-03-11T21:28:35.776 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.776 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.777 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.777 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.777 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.778 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_cls 2024-03-11T21:28:35.778 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=cmd 2024-03-11T21:28:35.778 INFO:tasks.workunit.client.0.smithi143.stdout:test cmd on pid 72109 2024-03-11T21:28:35.778 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72109 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test cmd on pid 72109' 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72109 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.779 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_cls.log 2024-03-11T21:28:35.783 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ cls: /' 2024-03-11T21:28:35.788 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.788 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.788 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.788 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.789 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ cmd: /' 2024-03-11T21:28:35.790 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-11T21:28:35.791 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s handler_error 2024-03-11T21:28:35.791 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' handler_error' 2024-03-11T21:28:35.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:35.799 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_cmd 2024-03-11T21:28:35.806 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo handler_error 2024-03-11T21:28:35.809 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.813 INFO:tasks.workunit.client.0.smithi143.stdout:test handler_error on pid 72178 2024-03-11T21:28:35.814 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=handler_error 2024-03-11T21:28:35.814 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72178 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test handler_error on pid 72178' 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72178 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s io 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' io' 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo io 2024-03-11T21:28:35.815 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.816 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=io 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stdout:test io on pid 72197 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72197 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test io on pid 72197' 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72197 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.817 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-11T21:28:35.818 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.818 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.819 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-11T21:28:35.819 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.819 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.819 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_io.log 2024-03-11T21:28:35.820 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.820 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.820 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.820 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.820 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ io: /' 2024-03-11T21:28:35.822 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ handler_error: /' 2024-03-11T21:28:35.822 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_io 2024-03-11T21:28:35.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:35.823 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-11T21:28:35.823 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_handler_error 2024-03-11T21:28:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:35.827 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s ec_io 2024-03-11T21:28:35.827 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' ec_io' 2024-03-11T21:28:35.828 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo ec_io 2024-03-11T21:28:35.829 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stdout:test ec_io on pid 72234 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=ec_io 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72234 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test ec_io on pid 72234' 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72234 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.832 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.847 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s list 2024-03-11T21:28:35.847 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-11T21:28:35.847 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' list' 2024-03-11T21:28:35.848 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.848 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo list 2024-03-11T21:28:35.848 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_ec_io 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.849 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ ec_io: /' 2024-03-11T21:28:35.851 INFO:tasks.workunit.client.0.smithi143.stdout:test list on pid 72282 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=list 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72282 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test list on pid 72282' 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72282 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.852 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.859 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-11T21:28:35.859 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s ec_list 2024-03-11T21:28:35.860 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' ec_list' 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.861 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_list.log 2024-03-11T21:28:35.862 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_list 2024-03-11T21:28:35.862 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ list: /' 2024-03-11T21:28:35.863 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo ec_list 2024-03-11T21:28:35.864 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=ec_list 2024-03-11T21:28:35.865 INFO:tasks.workunit.client.0.smithi143.stdout:test ec_list on pid 72309 2024-03-11T21:28:35.865 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72309 2024-03-11T21:28:35.865 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test ec_list on pid 72309' 2024-03-11T21:28:35.865 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72309 2024-03-11T21:28:35.865 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.866 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.866 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-11T21:28:35.867 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s misc 2024-03-11T21:28:35.867 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' misc' 2024-03-11T21:28:35.868 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.869 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.869 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.869 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.869 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-11T21:28:35.869 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ ec_list: /' 2024-03-11T21:28:35.870 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo misc 2024-03-11T21:28:35.870 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stdout:test misc on pid 72343 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=misc 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72343 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test misc on pid 72343' 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72343 2024-03-11T21:28:35.873 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.874 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.874 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s pool 2024-03-11T21:28:35.874 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' pool' 2024-03-11T21:28:35.874 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo pool 2024-03-11T21:28:35.874 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.877 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_ec_list 2024-03-11T21:28:35.878 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-11T21:28:35.878 INFO:tasks.workunit.client.0.smithi143.stdout:test pool on pid 72349 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=pool 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72349 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test pool on pid 72349' 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72349 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.879 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_misc 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s read_operations 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' read_operations' 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo read_operations 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_misc.log 2024-03-11T21:28:35.880 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-11T21:28:35.881 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.881 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.882 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.882 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.882 INFO:tasks.workunit.client.0.smithi143.stdout:test read_operations on pid 72379 2024-03-11T21:28:35.883 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=read_operations 2024-03-11T21:28:35.883 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72379 2024-03-11T21:28:35.883 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test read_operations on pid 72379' 2024-03-11T21:28:35.883 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72379 2024-03-11T21:28:35.883 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_pool 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_pool.log 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s snapshots 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ pool: /' 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' snapshots' 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ misc: /' 2024-03-11T21:28:35.884 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.885 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo snapshots 2024-03-11T21:28:35.886 INFO:tasks.workunit.client.0.smithi143.stdout:test snapshots on pid 72388 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=snapshots 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72388 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test snapshots on pid 72388' 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72388 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.887 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s watch_notify 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' watch_notify' 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.888 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo watch_notify 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ read_operations: /' 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.889 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_read_operations 2024-03-11T21:28:35.890 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.890 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.890 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.890 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.891 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_snapshots 2024-03-11T21:28:35.891 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=watch_notify 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stdout:test watch_notify on pid 72402 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72402 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test watch_notify on pid 72402' 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72402 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T21:28:35.892 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.893 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-11T21:28:35.893 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.893 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.893 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.894 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.894 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ watch_notify: /' 2024-03-11T21:28:35.895 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-11T21:28:35.896 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_watch_notify 2024-03-11T21:28:35.896 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-11T21:28:35.902 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ snapshots: /' 2024-03-11T21:28:35.906 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s write_operations 2024-03-11T21:28:35.907 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' write_operations' 2024-03-11T21:28:35.908 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-03-11T21:28:35.908 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo write_operations 2024-03-11T21:28:35.913 INFO:tasks.workunit.client.0.smithi143.stdout:test write_operations on pid 72432 2024-03-11T21:28:35.913 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=write_operations 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72432 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test write_operations on pid 72432' 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=72432 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ ret=0 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71762 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71762 2024-03-11T21:28:35.914 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_neorados_write_operations 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-11T21:28:35.916 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ write_operations: /' 2024-03-11T21:28:35.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:35.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:36.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:36.076 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:36.160 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:28:36.160 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:28:36.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:36.200 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:36.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:36.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:36.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:36.462 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:36.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:36.592 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:36.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:36.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.440751+0000 osd.0 (osd.0) 119 : cluster 0 3.17 scrub starts 2024-03-11T21:28:36.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.442892+0000 osd.0 (osd.0) 120 : cluster 0 3.17 scrub ok 2024-03-11T21:28:36.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.869172+0000 osd.6 (osd.6) 81 : cluster 0 2.14 scrub starts 2024-03-11T21:28:36.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.871447+0000 osd.6 (osd.6) 82 : cluster 0 2.14 scrub ok 2024-03-11T21:28:36.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.965490+0000 osd.5 (osd.5) 131 : cluster 0 2.2 scrub starts 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.967657+0000 osd.5 (osd.5) 132 : cluster 0 2.2 scrub ok 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.440751+0000 osd.0 (osd.0) 119 : cluster 0 3.17 scrub starts 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.442892+0000 osd.0 (osd.0) 120 : cluster 0 3.17 scrub ok 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.869172+0000 osd.6 (osd.6) 81 : cluster 0 2.14 scrub starts 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.871447+0000 osd.6 (osd.6) 82 : cluster 0 2.14 scrub ok 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.965490+0000 osd.5 (osd.5) 131 : cluster 0 2.2 scrub starts 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:36 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.967657+0000 osd.5 (osd.5) 132 : cluster 0 2.2 scrub ok 2024-03-11T21:28:36.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.440751+0000 osd.0 (osd.0) 119 : cluster 0 3.17 scrub starts 2024-03-11T21:28:36.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.442892+0000 osd.0 (osd.0) 120 : cluster 0 3.17 scrub ok 2024-03-11T21:28:36.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.869172+0000 osd.6 (osd.6) 81 : cluster 0 2.14 scrub starts 2024-03-11T21:28:36.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.871447+0000 osd.6 (osd.6) 82 : cluster 0 2.14 scrub ok 2024-03-11T21:28:36.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.965490+0000 osd.5 (osd.5) 131 : cluster 0 2.2 scrub starts 2024-03-11T21:28:36.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:36 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.967657+0000 osd.5 (osd.5) 132 : cluster 0 2.2 scrub ok 2024-03-11T21:28:36.723 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:36.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:36.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:36.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:36.981 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:37.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:37.111 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:37.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:37.236 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:37.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:37.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:37.368 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:37.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:37.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:37.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:37.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:37.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:37.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.013 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:38.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.139 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:38.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:38.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:38.398 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:38.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.526 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:38.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:38.639 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:38.656 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:38.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.782 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:38.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:28:38.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:38.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:39.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:39.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:39.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:39.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:39.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:39.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:39.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:39.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:39.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:39.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.802 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:39.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:40.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:40.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:40.184 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:40.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 -- 172.21.15.143:0/1193734427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c107220 msgr2=0x7f998c10edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 --2- 172.21.15.143:0/1193734427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c107220 0x7f998c10edd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f998400b0a0 tx=0x7f998400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 -- 172.21.15.143:0/1193734427 shutdown_connections 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 --2- 172.21.15.143:0/1193734427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c107220 0x7f998c10edd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 --2- 172.21.15.143:0/1193734427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c06a6a0 0x7f998c06aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 --2- 172.21.15.143:0/1193734427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06b050 0x7f998c106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 -- 172.21.15.143:0/1193734427 >> 172.21.15.143:0/1193734427 conn(0x7f998c077de0 msgr2=0x7f998c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 -- 172.21.15.143:0/1193734427 shutdown_connections 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.242+0000 7f999248b700 1 -- 172.21.15.143:0/1193734427 wait complete. 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 librados: starting msgr at 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 librados: starting objecter 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 Processor -- start 2024-03-11T21:28:40.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 -- start start 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 librados: setting wanted keys 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 librados: calling monclient init 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 0x7f998c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 0x7f998c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c107220 0x7f998c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c1116a0 con 0x7f998c06a6a0 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c111520 con 0x7f998c107220 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f999248b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c111820 con 0x7f998c06b050 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 0x7f998c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 0x7f998c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 0x7f998c1012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49318/0 (socket says 172.21.15.143:49318) 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998b7fe700 1 -- 172.21.15.143:0/1230579109 learned_addr learned my addr 172.21.15.143:0/1230579109 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f9990a28700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c107220 0x7f998c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 -- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 msgr2=0x7f998c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 0x7f998c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 -- 172.21.15.143:0/1230579109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c107220 msgr2=0x7f998c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c107220 0x7f998c101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 -- 172.21.15.143:0/1230579109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c1a6130 con 0x7f998c06a6a0 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f998bfff700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 0x7f998c100d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f997400b450 tx=0x7f997400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f99897fa700 10 client.?.objecter ms_handle_connect 0x7f998c06a6a0 2024-03-11T21:28:40.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.246+0000 7f99897fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:40.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:40.315 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:40.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:40.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:40.569 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- 172.21.15.143:0/513697551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0274107e90 msgr2=0x7f0274108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- 172.21.15.143:0/513697551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0274107e90 0x7f0274108270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0264009990 tx=0x7f0264009ca0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- 172.21.15.143:0/513697551 shutdown_connections 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- 172.21.15.143:0/513697551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027406a4b0 0x7f027410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- 172.21.15.143:0/513697551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0274107e90 0x7f0274108270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- 172.21.15.143:0/513697551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0274069b10 0x7f0274069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- 172.21.15.143:0/513697551 >> 172.21.15.143:0/513697551 conn(0x7f0274076d80 msgr2=0x7f0274077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- 172.21.15.143:0/513697551 shutdown_connections 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- 172.21.15.143:0/513697551 wait complete. 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 librados: starting msgr at 2024-03-11T21:28:40.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 librados: starting objecter 2024-03-11T21:28:40.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 Processor -- start 2024-03-11T21:28:40.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- start start 2024-03-11T21:28:40.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 librados: setting wanted keys 2024-03-11T21:28:40.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 librados: calling monclient init 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0274069b10 0x7f027419d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0274107e90 0x7f02741979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274111590 con 0x7f0274107e90 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274111410 con 0x7f0274069b10 2024-03-11T21:28:40.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f027a79a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0274111710 con 0x7f027406a4b0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49326/0 (socket says 172.21.15.143:49326) 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 -- 172.21.15.143:0/3723106609 learned_addr learned my addr 172.21.15.143:0/3723106609 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0274107e90 0x7f02741979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0274069b10 0x7f027419d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0274069b10 msgr2=0x7f027419d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0274069b10 0x7f027419d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0274107e90 msgr2=0x7f02741979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0274107e90 0x7f02741979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274198150 con 0x7f027406a4b0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f026800e9a0 tx=0x7f026800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02717fa700 10 client.?.objecter ms_handle_connect 0x7f027406a4b0 2024-03-11T21:28:40.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.614+0000 7f02717fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800ca10 con 0x7f027406a4b0 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f026800cbb0 con 0x7f027406a4b0 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268018420 con 0x7f027406a4b0 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0274198440 con 0x7f027406a4b0 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 10 client.24929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0274100d40 con 0x7f027406a4b0 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 librados: init done 2024-03-11T21:28:40.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f027404faa0 con 0x7f027406a4b0 2024-03-11T21:28:40.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0268010430 con 0x7f027406a4b0 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 0x7f025c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(138..138 src has 87..138) v4 ==== 8182+0+0 (secure 0 0 0) 0x7f026808e840 con 0x7f027406a4b0 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 osd_map(138..138 src has 87..138) v4 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 3 client.24929.objecter handle_osd_map got epochs [138,138] > 0 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 3 client.24929.objecter handle_osd_map decoding full epoch 138 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 0x7f025c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 20 client.24929.objecter dump_active .. 0 homeless 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v138) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0268093220 con 0x7f027406a4b0 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 10 client.24929.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 1 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0274107520 0x7f027402c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 1 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 0 1 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 4 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 10 client.24929.objecter _send_command 1 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0274062f00 con 0x7f0274107520 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0274107520 0x7f027402c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 0x7f025c071ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0264006010 tx=0x7f026401e040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f025c06f820 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f0274107520 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.618+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0274107520 0x7f027402c6b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.622+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0274062f00 con 0x7f0274107520 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.622+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:28:40.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.622+0000 7f02717fa700 10 client.24929.objecter _finish_command 1 = system:0 2024-03-11T21:28:40.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.622+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 4 2024-03-11T21:28:40.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.622+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 0 1 2024-03-11T21:28:40.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 10 client.24929.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 2 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 2 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741073b0 osd=0 3 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 0 2 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 4 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 10 client.24929.objecter _send_command 2 2024-03-11T21:28:40.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.638+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f027402cbf0 con 0x7f0274107520 2024-03-11T21:28:40.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f027402cbf0 con 0x7f0274107520 2024-03-11T21:28:40.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 10 client.24929.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 4 2024-03-11T21:28:40.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 0 2 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 10 client.24929.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 3 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f02740625e0 0x7f0274100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 3 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 1 3 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 4 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 10 client.24929.objecter _send_command 3 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0274102c50 con 0x7f02740625e0 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f02740625e0 0x7f0274100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02740625e0 2024-03-11T21:28:40.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f02740625e0 0x7f0274100f70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0274102c50 con 0x7f02740625e0 2024-03-11T21:28:40.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:28:40.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 10 client.24929.objecter _finish_command 3 = system:0 2024-03-11T21:28:40.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 4 2024-03-11T21:28:40.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.642+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 1 3 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 10 client.24929.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 4 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 4 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274062470 osd=1 3 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 1 4 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 4 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 10 client.24929.objecter _send_command 4 2024-03-11T21:28:40.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741014b0 con 0x7f02740625e0 2024-03-11T21:28:40.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741014b0 con 0x7f02740625e0 2024-03-11T21:28:40.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 10 client.24929.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 4 2024-03-11T21:28:40.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 1 4 2024-03-11T21:28:40.638 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 10 client.24929.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 5 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f02741093e0 0x7f0274109800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 5 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 2 5 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 4 2024-03-11T21:28:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 10 client.24929.objecter _send_command 5 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741029e0 con 0x7f02741093e0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f02741093e0 0x7f0274109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02741093e0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.662+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f02741093e0 0x7f0274109800 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.666+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741029e0 con 0x7f02741093e0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.666+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.666+0000 7f02717fa700 10 client.24929.objecter _finish_command 5 = system:0 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.666+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 4 2024-03-11T21:28:40.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.666+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 2 5 2024-03-11T21:28:40.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 10 client.24929.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 6 2024-03-11T21:28:40.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 6 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274102870 osd=2 3 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 2 6 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 4 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 10 client.24929.objecter _send_command 6 2024-03-11T21:28:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741029e0 con 0x7f02741093e0 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741029e0 con 0x7f02741093e0 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 10 client.24929.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 4 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 2 6 2024-03-11T21:28:40.658 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 10 client.24929.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 7 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0274108fd0 0x7f02741ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 7 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 3 7 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 4 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 10 client.24929.objecter _send_command 7 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741023f0 con 0x7f0274108fd0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0274108fd0 0x7f02741ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f0274108fd0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.682+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0274108fd0 0x7f02741ad8c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.686+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741023f0 con 0x7f0274108fd0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.686+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.686+0000 7f02717fa700 10 client.24929.objecter _finish_command 7 = system:0 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.686+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 4 2024-03-11T21:28:40.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.686+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 3 7 2024-03-11T21:28:40.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:40.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 10 client.24929.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 8 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 8 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274108e60 osd=3 3 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 3 8 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 4 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 10 client.24929.objecter _send_command 8 2024-03-11T21:28:40.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0274102630 con 0x7f0274108fd0 2024-03-11T21:28:40.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0274102630 con 0x7f0274108fd0 2024-03-11T21:28:40.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f02717fa700 10 client.24929.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 4 2024-03-11T21:28:40.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 3 8 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 10 client.24929.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 9 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f02741aaf00 0x7f02741b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 9 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 4 9 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 4 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 10 client.24929.objecter _send_command 9 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741ab390 con 0x7f02741aaf00 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.702+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f02741aaf00 0x7f02741b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02741aaf00 2024-03-11T21:28:40.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f02741aaf00 0x7f02741b0f90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741ab390 con 0x7f02741aaf00 2024-03-11T21:28:40.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:28:40.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 10 client.24929.objecter _finish_command 9 = system:0 2024-03-11T21:28:40.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 4 2024-03-11T21:28:40.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.706+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 4 9 2024-03-11T21:28:40.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 10 client.24929.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 10 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 10 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f0274102450 osd=4 3 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 4 10 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 4 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 10 client.24929.objecter _send_command 10 2024-03-11T21:28:40.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.722+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741ab3f0 con 0x7f02741aaf00 2024-03-11T21:28:40.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741ab3f0 con 0x7f02741aaf00 2024-03-11T21:28:40.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 10 client.24929.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 4 2024-03-11T21:28:40.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 4 10 2024-03-11T21:28:40.701 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 10 client.24929.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 11 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f02741b19e0 0x7f02741b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 11 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 5 11 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 4 2024-03-11T21:28:40.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 10 client.24929.objecter _send_command 11 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741b6c30 con 0x7f02741b19e0 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f02741b19e0 0x7f02741b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02741b19e0 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.726+0000 7f02737fe700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f02741b19e0 0x7f02741b3dd0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:40.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.730+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741b6c30 con 0x7f02741b19e0 2024-03-11T21:28:40.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.730+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:28:40.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.730+0000 7f02717fa700 10 client.24929.objecter _finish_command 11 = system:0 2024-03-11T21:28:40.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.730+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 4 2024-03-11T21:28:40.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.730+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 5 11 2024-03-11T21:28:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:40.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 10 client.24929.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 12 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 12 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741ab680 osd=5 3 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 5 12 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 4 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 10 client.24929.objecter _send_command 12 2024-03-11T21:28:40.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741016c0 con 0x7f02741b19e0 2024-03-11T21:28:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741016c0 con 0x7f02741b19e0 2024-03-11T21:28:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f02717fa700 10 client.24929.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 4 2024-03-11T21:28:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 5 12 2024-03-11T21:28:40.723 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:28:40.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 10 client.24929.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 13 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f02741aeae0 0x7f02741b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 13 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 6 13 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 4 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 10 client.24929.objecter _send_command 13 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741ba300 con 0x7f02741aeae0 2024-03-11T21:28:40.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.746+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f02741aeae0 0x7f02741b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02741aeae0 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f0278d37700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f02741aeae0 0x7f02741b74a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741ba300 con 0x7f02741aeae0 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 10 client.24929.objecter _finish_command 13 = system:0 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 4 2024-03-11T21:28:40.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.750+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 6 13 2024-03-11T21:28:40.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 10 client.24929.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 14 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 14 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741aa9d0 osd=6 3 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 6 14 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 4 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 10 client.24929.objecter _send_command 14 2024-03-11T21:28:40.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741b79e0 con 0x7f02741aeae0 2024-03-11T21:28:40.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741b79e0 con 0x7f02741aeae0 2024-03-11T21:28:40.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f02717fa700 10 client.24929.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 4 2024-03-11T21:28:40.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 6 14 2024-03-11T21:28:40.743 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 10 client.24929.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 15 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f02741b7ef0 0x7f02741ba390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 15 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 7 15 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 4 2024-03-11T21:28:40.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 10 client.24929.objecter _send_command 15 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.766+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02741bd410 con 0x7f02741b7ef0 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f02741b7ef0 0x7f02741ba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f0273fff700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f02741b7ef0 0x7f02741ba390 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 10 client.24929.objecter ms_handle_connect 0x7f02741b7ef0 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02741bd410 con 0x7f02741b7ef0 2024-03-11T21:28:40.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:28:40.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 10 client.24929.objecter _finish_command 15 = system:0 2024-03-11T21:28:40.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 4 2024-03-11T21:28:40.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.770+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 7 15 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 10 client.24929.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign -1 16 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 20 client.24929.objecter _get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 15 client.24929.objecter _session_command_op_remove -1 16 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 20 client.24929.objecter get_session s=0x7f02741aad90 osd=7 3 2024-03-11T21:28:40.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 15 client.24929.objecter _session_command_op_assign 7 16 2024-03-11T21:28:40.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 4 2024-03-11T21:28:40.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 10 client.24929.objecter _send_command 16 2024-03-11T21:28:40.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.786+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f02741bd470 con 0x7f02741b7ef0 2024-03-11T21:28:40.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f02717fa700 1 -- 172.21.15.143:0/3723106609 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f02741bd470 con 0x7f02741b7ef0 2024-03-11T21:28:40.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f02717fa700 10 client.24929.objecter ms_dispatch 0x7f0274001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:28:40.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f02717fa700 10 client.24929.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:28:40.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f02717fa700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 4 2024-03-11T21:28:40.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f02717fa700 15 client.24929.objecter _session_command_op_remove 7 16 2024-03-11T21:28:40.765 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:28:40.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 librados: watch_flush enter 2024-03-11T21:28:40.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 librados: watch_flush exit 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.0 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0274107520 msgr2=0x7f027402c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0274107520 0x7f027402c6b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741073b0 osd=0 1 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.1 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f02740625e0 msgr2=0x7f0274100f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f02740625e0 0x7f0274100f70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274062470 osd=1 1 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.2 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f02741093e0 msgr2=0x7f0274109800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f02741093e0 0x7f0274109800 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102870 osd=2 1 2024-03-11T21:28:40.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.3 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0274108fd0 msgr2=0x7f02741ad8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0274108fd0 0x7f02741ad8c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274108e60 osd=3 1 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.4 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f02741aaf00 msgr2=0x7f02741b0f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f02741aaf00 0x7f02741b0f90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f0274102450 osd=4 1 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.5 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f02741b19e0 msgr2=0x7f02741b3dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f02741b19e0 0x7f02741b3dd0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741ab680 osd=5 1 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.6 2024-03-11T21:28:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f02741aeae0 msgr2=0x7f02741b74a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f02741aeae0 0x7f02741b74a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aa9d0 osd=6 1 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter close_session for osd.7 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f02741b7ef0 msgr2=0x7f02741ba390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f02741b7ef0 0x7f02741ba390 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter put_session s=0x7f02741aad90 osd=7 1 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 20 client.24929.objecter shutdown clearing up homeless session... 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 10 client.24929.objecter successfully canceled tick 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f0278d37700 1 -- 172.21.15.143:0/3723106609 reap_dead start 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f0278d37700 1 -- 172.21.15.143:0/3723106609 reap_dead start 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f0278d37700 1 -- 172.21.15.143:0/3723106609 reap_dead start 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f0278d37700 1 -- 172.21.15.143:0/3723106609 reap_dead start 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f0278d37700 1 -- 172.21.15.143:0/3723106609 reap_dead start 2024-03-11T21:28:40.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 msgr2=0x7f025c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 0x7f025c071ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0264006010 tx=0x7f026401e040 comp rx=0 tx=0).stop 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 msgr2=0x7f027419dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f026800e9a0 tx=0x7f026800ecb0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 shutdown_connections 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027406a4b0 0x7f027419dd70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 --2- 172.21.15.143:0/3723106609 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f025c06f820 0x7f025c071ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 >> 172.21.15.143:0/3723106609 conn(0x7f0274076d80 msgr2=0x7f0274105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 shutdown_connections 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 -- 172.21.15.143:0/3723106609 wait complete. 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:40.790+0000 7f027a79a700 1 librados: shutdown 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:40.770 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:40.771 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:40.771 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:40.771 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:40.771 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:40.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:40.828 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:40.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:40.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:41.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:41.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:41.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:41.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:41.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:41.348 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:41.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:41.473 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:41.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:41.586 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.920425+0000 osd.2 (osd.2) 89 : cluster 0 5.a scrub starts 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:34.922063+0000 osd.2 (osd.2) 90 : cluster 0 5.a scrub ok 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.099076+0000 osd.4 (osd.4) 117 : cluster 0 6.1e scrub starts 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.101673+0000 osd.4 (osd.4) 118 : cluster 0 6.1e scrub ok 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.164173+0000 mgr.y (mgr.14550) 208 : cluster 0 pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.487595+0000 osd.0 (osd.0) 121 : cluster 0 6.10 scrub starts 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.489801+0000 osd.0 (osd.0) 122 : cluster 0 6.10 scrub ok 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.887249+0000 osd.6 (osd.6) 83 : cluster 0 2.7 scrub starts 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:35.888988+0000 osd.6 (osd.6) 84 : cluster 0 2.7 scrub ok 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: audit 2024-03-11T21:28:35.917661+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: audit 2024-03-11T21:28:35.952656+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.309074+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.310835+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: audit 2024-03-11T21:28:36.538420+0000 mon.a (mon.0) 913 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[21512]: audit 2024-03-11T21:28:36.546617+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974015020 con 0x7f998c06a6a0 2024-03-11T21:28:41.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c1a6420 con 0x7f998c06a6a0 2024-03-11T21:28:41.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f999248b700 10 client.15000.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:41.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c1a6900 con 0x7f998c06a6a0 2024-03-11T21:28:41.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f999248b700 1 librados: init done 2024-03-11T21:28:41.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.614+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9974011cb0 con 0x7f998c06a6a0 2024-03-11T21:28:41.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997401abc0 con 0x7f998c06a6a0 2024-03-11T21:28:41.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c005120 con 0x7f998c06a6a0 2024-03-11T21:28:41.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9974018020 con 0x7f998c06a6a0 2024-03-11T21:28:41.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 0x7f9978071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:41.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(138..138 src has 87..138) v4 ==== 8182+0+0 (secure 0 0 0) 0x7f997408dfc0 con 0x7f998c06a6a0 2024-03-11T21:28:41.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 10 client.15000.objecter ms_dispatch 0x7f998c001550 osd_map(138..138 src has 87..138) v4 2024-03-11T21:28:41.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 3 client.15000.objecter handle_osd_map got epochs [138,138] > 0 2024-03-11T21:28:41.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f99897fa700 3 client.15000.objecter handle_osd_map decoding full epoch 138 2024-03-11T21:28:41.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.618+0000 7f998b7fe700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 0x7f9978071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:41.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.622+0000 7f998b7fe700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 0x7f9978071bb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f998c1086e0 tx=0x7f997c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:41.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.622+0000 7f99897fa700 20 client.15000.objecter dump_active .. 0 homeless 2024-03-11T21:28:41.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.622+0000 7f99897fa700 10 client.15000.objecter ms_handle_connect 0x7f997806f6f0 2024-03-11T21:28:41.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.630+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f997401f020 con 0x7f998c06a6a0 2024-03-11T21:28:41.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:41.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:41.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:41.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.920425+0000 osd.2 (osd.2) 89 : cluster 0 5.a scrub starts 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:34.922063+0000 osd.2 (osd.2) 90 : cluster 0 5.a scrub ok 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.099076+0000 osd.4 (osd.4) 117 : cluster 0 6.1e scrub starts 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.101673+0000 osd.4 (osd.4) 118 : cluster 0 6.1e scrub ok 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.164173+0000 mgr.y (mgr.14550) 208 : cluster 0 pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.487595+0000 osd.0 (osd.0) 121 : cluster 0 6.10 scrub starts 2024-03-11T21:28:41.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.489801+0000 osd.0 (osd.0) 122 : cluster 0 6.10 scrub ok 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.887249+0000 osd.6 (osd.6) 83 : cluster 0 2.7 scrub starts 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:35.888988+0000 osd.6 (osd.6) 84 : cluster 0 2.7 scrub ok 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: audit 2024-03-11T21:28:35.917661+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: audit 2024-03-11T21:28:35.952656+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.309074+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.310835+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: audit 2024-03-11T21:28:36.538420+0000 mon.a (mon.0) 913 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:41 smithi203 bash[23654]: audit 2024-03-11T21:28:36.546617+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:41.874+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7f996c005780 con 0x7f998c06a6a0 2024-03-11T21:28:41.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:41.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.920425+0000 osd.2 (osd.2) 89 : cluster 0 5.a scrub starts 2024-03-11T21:28:41.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:34.922063+0000 osd.2 (osd.2) 90 : cluster 0 5.a scrub ok 2024-03-11T21:28:41.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.099076+0000 osd.4 (osd.4) 117 : cluster 0 6.1e scrub starts 2024-03-11T21:28:41.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.101673+0000 osd.4 (osd.4) 118 : cluster 0 6.1e scrub ok 2024-03-11T21:28:41.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.164173+0000 mgr.y (mgr.14550) 208 : cluster 0 pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.487595+0000 osd.0 (osd.0) 121 : cluster 0 6.10 scrub starts 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.489801+0000 osd.0 (osd.0) 122 : cluster 0 6.10 scrub ok 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.887249+0000 osd.6 (osd.6) 83 : cluster 0 2.7 scrub starts 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:35.888988+0000 osd.6 (osd.6) 84 : cluster 0 2.7 scrub ok 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: audit 2024-03-11T21:28:35.917661+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: audit 2024-03-11T21:28:35.952656+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.309074+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.310835+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: audit 2024-03-11T21:28:36.538420+0000 mon.a (mon.0) 913 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:41 smithi143 bash[29284]: audit 2024-03-11T21:28:36.546617+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:41.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:41.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:42.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:42.118 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:42.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:42.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:42.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.357 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:42.377 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:42.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- 172.21.15.143:0/2805342465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c107220 msgr2=0x7f1f1c10edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2805342465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c107220 0x7f1f1c10edd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400b0a0 tx=0x7f1f1400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- 172.21.15.143:0/2805342465 shutdown_connections 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2805342465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06a6a0 0x7f1f1c06aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2805342465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c107220 0x7f1f1c10edd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2805342465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c06b050 0x7f1f1c106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- 172.21.15.143:0/2805342465 >> 172.21.15.143:0/2805342465 conn(0x7f1f1c077de0 msgr2=0x7f1f1c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- 172.21.15.143:0/2805342465 shutdown_connections 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- 172.21.15.143:0/2805342465 wait complete. 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 librados: starting msgr at 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 librados: starting objecter 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 Processor -- start 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- start start 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 librados: setting wanted keys 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 librados: calling monclient init 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c06a6a0 0x7f1f1c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:42.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 0x7f1f1c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c111670 con 0x7f1f1c06b050 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1114f0 con 0x7f1f1c107220 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f2273b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1117f0 con 0x7f1f1c06a6a0 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39924/0 (socket says 172.21.15.143:39924) 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 -- 172.21.15.143:0/2386525285 learned_addr learned my addr 172.21.15.143:0/2386525285 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:28:42.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1bfff700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c06a6a0 0x7f1f1c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:42.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f20cd8700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 0x7f1f1c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:42.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 -- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c06a6a0 msgr2=0x7f1f1c100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:42.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c06a6a0 0x7f1f1c100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 -- 172.21.15.143:0/2386525285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 msgr2=0x7f1f1c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 0x7f1f1c101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 -- 172.21.15.143:0/2386525285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c1a6120 con 0x7f1f1c06b050 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f20cd8700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 0x7f1f1c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f1b7fe700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d950 tx=0x7f1f1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f197fa700 10 client.?.objecter ms_handle_connect 0x7f1f1c06b050 2024-03-11T21:28:42.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:42.506+0000 7f1f197fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:28:42.499 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:42.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:42.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:42.758 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:42.861 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:42.881 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:42.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.011 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:43.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:43.136 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:43.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:43.263 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:43.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.395 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:43.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:43.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:43.654 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:43.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:43.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:43.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:44.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:44.039 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:44.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:44.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:44.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:44.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:28:44.555 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:44.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:44.686 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:44.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.811 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:44.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:44.944 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:45.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:45.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:45.199 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:45.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:45.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:45.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:45.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:45.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:45.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:45.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:45.729 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:45.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:45.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.991 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:46.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:46.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:46.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:46.254 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:46.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:46.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:46.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:46.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.618+0000 7f9991c8a700 10 client.15000.objecter tick 2024-03-11T21:28:46.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:46.622 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:46.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:46.731 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:46.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10017070 con 0x7f1f1c06b050 2024-03-11T21:28:46.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c1a6410 con 0x7f1f1c06b050 2024-03-11T21:28:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:46.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f2273b700 10 client.15123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:28:46.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c1a68e0 con 0x7f1f1c06b050 2024-03-11T21:28:46.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f2273b700 1 librados: init done 2024-03-11T21:28:46.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.758+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c04faa0 con 0x7f1f1c06b050 2024-03-11T21:28:46.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.774+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1f10009db0 con 0x7f1f1c06b050 2024-03-11T21:28:46.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.774+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10005280 con 0x7f1f1c06b050 2024-03-11T21:28:46.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1f100049c0 con 0x7f1f1c06b050 2024-03-11T21:28:46.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 0x7f1f04071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f1bfff700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 0x7f1f04071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(139..139 src has 87..139) v4 ==== 12596+0+0 (secure 0 0 0) 0x7f1f100590d0 con 0x7f1f1c06b050 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 10 client.15123.objecter ms_dispatch 0x7f1f1c001550 osd_map(139..139 src has 87..139) v4 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 3 client.15123.objecter handle_osd_map got epochs [139,139] > 0 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 3 client.15123.objecter handle_osd_map decoding full epoch 139 2024-03-11T21:28:46.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 20 client.15123.objecter dump_active .. 0 homeless 2024-03-11T21:28:46.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f1bfff700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 0x7f1f04071ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009c60 tx=0x7f1f0c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:28:46.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.778+0000 7f1f197fa700 10 client.15123.objecter ms_handle_connect 0x7f1f0406f820 2024-03-11T21:28:46.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:46.790+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1f1005b730 con 0x7f1f1c06b050 2024-03-11T21:28:46.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:46.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:46.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:36.563818+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:36.566856+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.143:0/2524399149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:36.585338+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.143:0/1026413940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:36.588089+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.143:0/2896411365' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.898409+0000 osd.2 (osd.2) 91 : cluster 0 5.d scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.900556+0000 osd.2 (osd.2) 92 : cluster 0 5.d scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:37.501939+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:37.505714+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:38.302420+0000 osd.3 (osd.3) 137 : cluster 0 6.6 scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:38.304083+0000 osd.3 (osd.3) 138 : cluster 0 6.6 scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:38.878030+0000 osd.2 (osd.2) 93 : cluster 0 2.c scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:38.879679+0000 osd.2 (osd.2) 94 : cluster 0 2.c scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.270091+0000 osd.3 (osd.3) 139 : cluster 0 6.c scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.272584+0000 osd.3 (osd.3) 140 : cluster 0 6.c scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.511319+0000 osd.0 (osd.0) 125 : cluster 0 6.18 scrub starts 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.513454+0000 osd.0 (osd.0) 126 : cluster 0 6.18 scrub ok 2024-03-11T21:28:46.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.300190+0000 osd.3 (osd.3) 141 : cluster 0 3.18 scrub starts 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.301960+0000 osd.3 (osd.3) 142 : cluster 0 3.18 scrub ok 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.491398+0000 osd.0 (osd.0) 127 : cluster 0 6.16 scrub starts 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.492858+0000 osd.0 (osd.0) 128 : cluster 0 6.16 scrub ok 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:40.621804+0000 mon.c (mon.2) 109 : audit 0 from='client.? 172.21.15.143:0/3723106609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.603909+0000 mon.a (mon.0) 915 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:41.619863+0000 mon.a (mon.0) 916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[29284]: audit 2024-03-11T21:28:41.620896+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:36.563818+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:36.566856+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.143:0/2524399149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:36.585338+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.143:0/1026413940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:36.588089+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.143:0/2896411365' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.898409+0000 osd.2 (osd.2) 91 : cluster 0 5.d scrub starts 2024-03-11T21:28:46.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.900556+0000 osd.2 (osd.2) 92 : cluster 0 5.d scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:37.501939+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:37.505714+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:38.302420+0000 osd.3 (osd.3) 137 : cluster 0 6.6 scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:38.304083+0000 osd.3 (osd.3) 138 : cluster 0 6.6 scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:38.878030+0000 osd.2 (osd.2) 93 : cluster 0 2.c scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:38.879679+0000 osd.2 (osd.2) 94 : cluster 0 2.c scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.270091+0000 osd.3 (osd.3) 139 : cluster 0 6.c scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.272584+0000 osd.3 (osd.3) 140 : cluster 0 6.c scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.511319+0000 osd.0 (osd.0) 125 : cluster 0 6.18 scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.513454+0000 osd.0 (osd.0) 126 : cluster 0 6.18 scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.300190+0000 osd.3 (osd.3) 141 : cluster 0 3.18 scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.301960+0000 osd.3 (osd.3) 142 : cluster 0 3.18 scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.491398+0000 osd.0 (osd.0) 127 : cluster 0 6.16 scrub starts 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.492858+0000 osd.0 (osd.0) 128 : cluster 0 6.16 scrub ok 2024-03-11T21:28:46.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:40.621804+0000 mon.c (mon.2) 109 : audit 0 from='client.? 172.21.15.143:0/3723106609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:46.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.603909+0000 mon.a (mon.0) 915 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:28:46.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:41.619863+0000 mon.a (mon.0) 916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:46 smithi143 bash[21512]: audit 2024-03-11T21:28:41.620896+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:46.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:36.563818+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:36.566856+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.143:0/2524399149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:36.585338+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.143:0/1026413940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:36.588089+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.143:0/2896411365' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.898409+0000 osd.2 (osd.2) 91 : cluster 0 5.d scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.900556+0000 osd.2 (osd.2) 92 : cluster 0 5.d scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:37.501939+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:37.505714+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:38.302420+0000 osd.3 (osd.3) 137 : cluster 0 6.6 scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:38.304083+0000 osd.3 (osd.3) 138 : cluster 0 6.6 scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:38.878030+0000 osd.2 (osd.2) 93 : cluster 0 2.c scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:38.879679+0000 osd.2 (osd.2) 94 : cluster 0 2.c scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.270091+0000 osd.3 (osd.3) 139 : cluster 0 6.c scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.272584+0000 osd.3 (osd.3) 140 : cluster 0 6.c scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.511319+0000 osd.0 (osd.0) 125 : cluster 0 6.18 scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.513454+0000 osd.0 (osd.0) 126 : cluster 0 6.18 scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.300190+0000 osd.3 (osd.3) 141 : cluster 0 3.18 scrub starts 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.301960+0000 osd.3 (osd.3) 142 : cluster 0 3.18 scrub ok 2024-03-11T21:28:46.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.491398+0000 osd.0 (osd.0) 127 : cluster 0 6.16 scrub starts 2024-03-11T21:28:46.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.492858+0000 osd.0 (osd.0) 128 : cluster 0 6.16 scrub ok 2024-03-11T21:28:46.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:40.621804+0000 mon.c (mon.2) 109 : audit 0 from='client.? 172.21.15.143:0/3723106609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:46.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.603909+0000 mon.a (mon.0) 915 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:28:46.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:41.619863+0000 mon.a (mon.0) 916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:46.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:46 smithi203 bash[23654]: audit 2024-03-11T21:28:41.620896+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:46.981 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:47.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:47.058+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1f1c1a6cc0 con 0x7f1f1c06b050 2024-03-11T21:28:47.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:47.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:47.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:47.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:47.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:47.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:47.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:47.492 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:47.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:47.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:47.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:47.754 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:47.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:47.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:47.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:48.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:48.133 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:48.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:48.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:48.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:48.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.517 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:48.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:48.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:48.648 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:48.649 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:48.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:48.882 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:48.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:49.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:49.035 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:49.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:49.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:49.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.290 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:49.393 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.419 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:49.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:49.545 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:49.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:49.674 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:49.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:49.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.809+0000 7f99897fa700 1 -- 172.21.15.143:0/1230579109 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v141) v1 ==== 129+0+0 (secure 0 0 0) 0x7f996c005780 con 0x7f998c06a6a0 2024-03-11T21:28:49.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.813+0000 7f999248b700 10 librados: watch_flush enter 2024-03-11T21:28:49.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.813+0000 7f999248b700 10 librados: watch_flush exit 2024-03-11T21:28:49.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.813+0000 7f999248b700 20 client.15000.objecter shutdown clearing up homeless session... 2024-03-11T21:28:49.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.813+0000 7f999248b700 10 client.15000.objecter successfully canceled tick 2024-03-11T21:28:49.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 msgr2=0x7f9978071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:49.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 0x7f9978071bb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f998c1086e0 tx=0x7f997c009260 comp rx=0 tx=0).stop 2024-03-11T21:28:49.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 msgr2=0x7f998c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 0x7f998c100d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f997400b450 tx=0x7f997400b760 comp rx=0 tx=0).stop 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 shutdown_connections 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f997806f6f0 0x7f9978071bb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f998c06a6a0 0x7f998c100d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f998c107220 0x7f998c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:49.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 --2- 172.21.15.143:0/1230579109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f998c06b050 0x7f998c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 >> 172.21.15.143:0/1230579109 conn(0x7f998c077de0 msgr2=0x7f998c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 shutdown_connections 2024-03-11T21:28:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 -- 172.21.15.143:0/1230579109 wait complete. 2024-03-11T21:28:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:49.817+0000 7f999248b700 1 librados: shutdown 2024-03-11T21:28:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pgp_num to 1 2024-03-11T21:28:49.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.892774+0000 osd.6 (osd.6) 85 : cluster 0 3.10 scrub starts 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.896776+0000 osd.6 (osd.6) 86 : cluster 0 3.10 scrub ok 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.905156+0000 osd.5 (osd.5) 133 : cluster 0 2.12 scrub starts 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:36.907717+0000 osd.5 (osd.5) 134 : cluster 0 2.12 scrub ok 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:37.164590+0000 mgr.y (mgr.14550) 209 : cluster 0 pgmap v184: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:37.392779+0000 osd.7 (osd.7) 149 : cluster 0 5.9 scrub starts 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:37.394499+0000 osd.7 (osd.7) 150 : cluster 0 5.9 scrub ok 2024-03-11T21:28:49.836 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.403357+0000 osd.7 (osd.7) 151 : cluster 0 3.13 scrub starts 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.407226+0000 osd.7 (osd.7) 152 : cluster 0 3.13 scrub ok 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.636751+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.143:0/3606906425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.638321+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.143:0/2450754466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.638623+0000 mon.b (mon.1) 36 : audit 1 from='client.? 172.21.15.143:0/1620680136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.644054+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.143:0/1041041831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.651295+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.143:0/204410024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.666300+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.666520+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.666695+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.143:0/1841026234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:41.667197+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.143:0/1792794818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.317529+0000 osd.3 (osd.3) 143 : cluster 0 6.13 scrub starts 2024-03-11T21:28:49.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.318904+0000 osd.3 (osd.3) 144 : cluster 0 6.13 scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:42.952251+0000 mon.c (mon.2) 115 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.001702+0000 osd.2 (osd.2) 95 : cluster 0 6.f scrub starts 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.003990+0000 osd.2 (osd.2) 96 : cluster 0 6.f scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.968078+0000 osd.2 (osd.2) 97 : cluster 0 5.6 scrub starts 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.970136+0000 osd.2 (osd.2) 98 : cluster 0 5.6 scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.316978+0000 osd.3 (osd.3) 145 : cluster 0 6.1b scrub starts 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.318905+0000 osd.3 (osd.3) 146 : cluster 0 6.1b scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.466968+0000 osd.0 (osd.0) 129 : cluster 0 2.13 scrub starts 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.468162+0000 osd.0 (osd.0) 130 : cluster 0 2.13 scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.877721+0000 osd.2 (osd.2) 99 : cluster 0 2.1 scrub starts 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.879370+0000 osd.2 (osd.2) 100 : cluster 0 2.1 scrub ok 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: cluster 2024-03-11T21:28:46.741680+0000 mon.a (mon.0) 918 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:28:49.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:46.760882+0000 mon.a (mon.0) 919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[29284]: audit 2024-03-11T21:28:46.761185+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.892774+0000 osd.6 (osd.6) 85 : cluster 0 3.10 scrub starts 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.896776+0000 osd.6 (osd.6) 86 : cluster 0 3.10 scrub ok 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.905156+0000 osd.5 (osd.5) 133 : cluster 0 2.12 scrub starts 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:36.907717+0000 osd.5 (osd.5) 134 : cluster 0 2.12 scrub ok 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:37.164590+0000 mgr.y (mgr.14550) 209 : cluster 0 pgmap v184: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:37.392779+0000 osd.7 (osd.7) 149 : cluster 0 5.9 scrub starts 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:37.394499+0000 osd.7 (osd.7) 150 : cluster 0 5.9 scrub ok 2024-03-11T21:28:49.839 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.403357+0000 osd.7 (osd.7) 151 : cluster 0 3.13 scrub starts 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.407226+0000 osd.7 (osd.7) 152 : cluster 0 3.13 scrub ok 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.636751+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.143:0/3606906425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.638321+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.143:0/2450754466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.638623+0000 mon.b (mon.1) 36 : audit 1 from='client.? 172.21.15.143:0/1620680136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.644054+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.143:0/1041041831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.651295+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.143:0/204410024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.666300+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.666520+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.666695+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.143:0/1841026234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:41.667197+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.143:0/1792794818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.317529+0000 osd.3 (osd.3) 143 : cluster 0 6.13 scrub starts 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.318904+0000 osd.3 (osd.3) 144 : cluster 0 6.13 scrub ok 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:42.952251+0000 mon.c (mon.2) 115 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.001702+0000 osd.2 (osd.2) 95 : cluster 0 6.f scrub starts 2024-03-11T21:28:49.840 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.003990+0000 osd.2 (osd.2) 96 : cluster 0 6.f scrub ok 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.968078+0000 osd.2 (osd.2) 97 : cluster 0 5.6 scrub starts 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.970136+0000 osd.2 (osd.2) 98 : cluster 0 5.6 scrub ok 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.316978+0000 osd.3 (osd.3) 145 : cluster 0 6.1b scrub starts 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.318905+0000 osd.3 (osd.3) 146 : cluster 0 6.1b scrub ok 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.466968+0000 osd.0 (osd.0) 129 : cluster 0 2.13 scrub starts 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.468162+0000 osd.0 (osd.0) 130 : cluster 0 2.13 scrub ok 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.877721+0000 osd.2 (osd.2) 99 : cluster 0 2.1 scrub starts 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.879370+0000 osd.2 (osd.2) 100 : cluster 0 2.1 scrub ok 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: cluster 2024-03-11T21:28:46.741680+0000 mon.a (mon.0) 918 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:46.760882+0000 mon.a (mon.0) 919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:49 smithi143 bash[21512]: audit 2024-03-11T21:28:46.761185+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:28:49.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.892774+0000 osd.6 (osd.6) 85 : cluster 0 3.10 scrub starts 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.896776+0000 osd.6 (osd.6) 86 : cluster 0 3.10 scrub ok 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.905156+0000 osd.5 (osd.5) 133 : cluster 0 2.12 scrub starts 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:36.907717+0000 osd.5 (osd.5) 134 : cluster 0 2.12 scrub ok 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:37.164590+0000 mgr.y (mgr.14550) 209 : cluster 0 pgmap v184: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:37.392779+0000 osd.7 (osd.7) 149 : cluster 0 5.9 scrub starts 2024-03-11T21:28:49.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:37.394499+0000 osd.7 (osd.7) 150 : cluster 0 5.9 scrub ok 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.403357+0000 osd.7 (osd.7) 151 : cluster 0 3.13 scrub starts 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.407226+0000 osd.7 (osd.7) 152 : cluster 0 3.13 scrub ok 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.636751+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.143:0/3606906425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.638321+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.143:0/2450754466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.638623+0000 mon.b (mon.1) 36 : audit 1 from='client.? 172.21.15.143:0/1620680136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.644054+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.143:0/1041041831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.651295+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.143:0/204410024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.666300+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.666520+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.666695+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.143:0/1841026234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:41.667197+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.143:0/1792794818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.317529+0000 osd.3 (osd.3) 143 : cluster 0 6.13 scrub starts 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.318904+0000 osd.3 (osd.3) 144 : cluster 0 6.13 scrub ok 2024-03-11T21:28:49.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:42.952251+0000 mon.c (mon.2) 115 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.001702+0000 osd.2 (osd.2) 95 : cluster 0 6.f scrub starts 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.003990+0000 osd.2 (osd.2) 96 : cluster 0 6.f scrub ok 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.968078+0000 osd.2 (osd.2) 97 : cluster 0 5.6 scrub starts 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.970136+0000 osd.2 (osd.2) 98 : cluster 0 5.6 scrub ok 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.316978+0000 osd.3 (osd.3) 145 : cluster 0 6.1b scrub starts 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.318905+0000 osd.3 (osd.3) 146 : cluster 0 6.1b scrub ok 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.466968+0000 osd.0 (osd.0) 129 : cluster 0 2.13 scrub starts 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.468162+0000 osd.0 (osd.0) 130 : cluster 0 2.13 scrub ok 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.877721+0000 osd.2 (osd.2) 99 : cluster 0 2.1 scrub starts 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.879370+0000 osd.2 (osd.2) 100 : cluster 0 2.1 scrub ok 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: cluster 2024-03-11T21:28:46.741680+0000 mon.a (mon.0) 918 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:46.760882+0000 mon.a (mon.0) 919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:49.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:49 smithi203 bash[23654]: audit 2024-03-11T21:28:46.761185+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:28:49.950 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-11T21:28:49.950 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] Global test environment set-up. 2024-03-11T21:28:49.950 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-11T21:28:49.951 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] 12 tests from AsioRados (11 ms total) 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] Global test environment tear-down 2024-03-11T21:28:49.952 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (14367 ms total) 2024-03-11T21:28:49.953 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-11T21:28:49.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:50.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:50.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:50.204 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:50.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:50.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:50.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:50.460 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:50.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.584 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:50.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:50.712 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:50.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:50.841 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:50.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:51.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:51.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.149+0000 7f1f197fa700 1 -- 172.21.15.143:0/2386525285 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v142) v1 ==== 101+0+0 (secure 0 0 0) 0x7f1f1005f180 con 0x7f1f1c06b050 2024-03-11T21:28:51.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 10 librados: watch_flush enter 2024-03-11T21:28:51.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 10 librados: watch_flush exit 2024-03-11T21:28:51.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 20 client.15123.objecter shutdown clearing up homeless session... 2024-03-11T21:28:51.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 10 client.15123.objecter successfully canceled tick 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 msgr2=0x7f1f04071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 0x7f1f04071ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009c60 tx=0x7f1f0c009260 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 msgr2=0x7f1f1c1012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d950 tx=0x7f1f1000dc60 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 shutdown_connections 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1f0406f820 0x7f1f04071ce0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c107220 0x7f1f1c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c06a6a0 0x7f1f1c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 --2- 172.21.15.143:0/2386525285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c06b050 0x7f1f1c1012d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:28:51.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.153+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 >> 172.21.15.143:0/2386525285 conn(0x7f1f1c077de0 msgr2=0x7f1f1c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:28:51.132 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.157+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 shutdown_connections 2024-03-11T21:28:51.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.161+0000 7f1f2273b700 1 -- 172.21.15.143:0/2386525285 wait complete. 2024-03-11T21:28:51.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:28:51.161+0000 7f1f2273b700 1 librados: shutdown 2024-03-11T21:28:51.153 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:28:51.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:51.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:38.314035+0000 mgr.y (mgr.14550) 210 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:39.165983+0000 mgr.y (mgr.14550) 211 : cluster 0 pgmap v185: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.309055+0000 osd.1 (osd.1) 115 : cluster 0 2.a scrub starts 2024-03-11T21:28:51.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.310699+0000 osd.1 (osd.1) 116 : cluster 0 2.a scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.396071+0000 osd.7 (osd.7) 153 : cluster 0 5.4 scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.397919+0000 osd.7 (osd.7) 154 : cluster 0 5.4 scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.842277+0000 osd.6 (osd.6) 87 : cluster 0 3.a scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:40.847327+0000 osd.6 (osd.6) 88 : cluster 0 3.a scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.166767+0000 mgr.y (mgr.14550) 212 : cluster 0 pgmap v186: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.438499+0000 osd.7 (osd.7) 155 : cluster 0 1.0 scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.441852+0000 osd.7 (osd.7) 156 : cluster 0 1.0 scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.809688+0000 osd.6 (osd.6) 89 : cluster 0 5.1f scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.812900+0000 osd.6 (osd.6) 90 : cluster 0 5.1f scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.969374+0000 osd.5 (osd.5) 135 : cluster 0 3.1c scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:41.972498+0000 osd.5 (osd.5) 136 : cluster 0 3.1c scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.091948+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.093976+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T21:28:51.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.394413+0000 osd.7 (osd.7) 157 : cluster 0 2.b scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:42.395719+0000 osd.7 (osd.7) 158 : cluster 0 2.b scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.170070+0000 mgr.y (mgr.14550) 213 : cluster 0 pgmap v188: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.360528+0000 osd.7 (osd.7) 159 : cluster 0 3.15 scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:43.364212+0000 osd.7 (osd.7) 160 : cluster 0 3.15 scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.383289+0000 osd.7 (osd.7) 161 : cluster 0 6.3 scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.386493+0000 osd.7 (osd.7) 162 : cluster 0 6.3 scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.889966+0000 osd.6 (osd.6) 91 : cluster 0 2.1a scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:44.892255+0000 osd.6 (osd.6) 92 : cluster 0 2.1a scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.082510+0000 osd.4 (osd.4) 121 : cluster 0 6.1a scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.084981+0000 osd.4 (osd.4) 122 : cluster 0 6.1a scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.171387+0000 mgr.y (mgr.14550) 214 : cluster 0 pgmap v189: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.376453+0000 osd.7 (osd.7) 163 : cluster 0 6.15 scrub starts 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:45.377663+0000 osd.7 (osd.7) 164 : cluster 0 6.15 scrub ok 2024-03-11T21:28:51.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:46.235138+0000 osd.1 (osd.1) 117 : cluster 0 5.19 scrub starts 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:46.237354+0000 osd.1 (osd.1) 118 : cluster 0 5.19 scrub ok 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:46.792842+0000 mon.a (mon.0) 921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:46.792938+0000 mon.a (mon.0) 922 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:46.804641+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:46.816242+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:47.172353+0000 mgr.y (mgr.14550) 215 : cluster 0 pgmap v191: 964 pgs: 304 creating+peering, 132 active+clean, 528 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.170868+0000 osd.1 (osd.1) 119 : cluster 0 6.1 scrub starts 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.172750+0000 osd.1 (osd.1) 120 : cluster 0 6.1 scrub ok 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.303386+0000 osd.3 (osd.3) 147 : cluster 0 2.19 scrub starts 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.305151+0000 osd.3 (osd.3) 148 : cluster 0 2.19 scrub ok 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:48.321842+0000 mgr.y (mgr.14550) 216 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.362848+0000 osd.7 (osd.7) 165 : cluster 0 2.0 scrub starts 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.364800+0000 osd.7 (osd.7) 166 : cluster 0 2.0 scrub ok 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.414844+0000 osd.0 (osd.0) 131 : cluster 0 5.1e scrub starts 2024-03-11T21:28:51.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.415887+0000 osd.0 (osd.0) 132 : cluster 0 5.1e scrub ok 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.817145+0000 osd.5 (osd.5) 137 : cluster 0 3.5 scrub starts 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.820379+0000 osd.5 (osd.5) 138 : cluster 0 3.5 scrub ok 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.841979+0000 osd.2 (osd.2) 101 : cluster 0 6.14 scrub starts 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:48.843700+0000 osd.2 (osd.2) 102 : cluster 0 6.14 scrub ok 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:48.909582+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:49.515940+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:49.516054+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:49.753304+0000 mon.a (mon.0) 923 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: cluster 2024-03-11T21:28:49.756050+0000 mon.a (mon.0) 924 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.785704+0000 mon.a (mon.0) 925 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.787234+0000 mon.a (mon.0) 926 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.787432+0000 mon.a (mon.0) 927 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.787638+0000 mon.a (mon.0) 928 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.788104+0000 mon.a (mon.0) 929 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.788365+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.788686+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.800470+0000 mon.a (mon.0) 932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.800691+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.800903+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.801798+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.801974+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.802268+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.802384+0000 mon.a (mon.0) 938 : audit 0 from='client.? 172.21.15.143:0/3150635234' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.802971+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.804946+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.805486+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.806228+0000 mon.a (mon.0) 942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.218 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[29284]: audit 2024-03-11T21:28:49.814245+0000 mon.a (mon.0) 943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:38.314035+0000 mgr.y (mgr.14550) 210 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:39.165983+0000 mgr.y (mgr.14550) 211 : cluster 0 pgmap v185: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.309055+0000 osd.1 (osd.1) 115 : cluster 0 2.a scrub starts 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.310699+0000 osd.1 (osd.1) 116 : cluster 0 2.a scrub ok 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.396071+0000 osd.7 (osd.7) 153 : cluster 0 5.4 scrub starts 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.397919+0000 osd.7 (osd.7) 154 : cluster 0 5.4 scrub ok 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.842277+0000 osd.6 (osd.6) 87 : cluster 0 3.a scrub starts 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:40.847327+0000 osd.6 (osd.6) 88 : cluster 0 3.a scrub ok 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.166767+0000 mgr.y (mgr.14550) 212 : cluster 0 pgmap v186: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.219 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.438499+0000 osd.7 (osd.7) 155 : cluster 0 1.0 scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.441852+0000 osd.7 (osd.7) 156 : cluster 0 1.0 scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.809688+0000 osd.6 (osd.6) 89 : cluster 0 5.1f scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.812900+0000 osd.6 (osd.6) 90 : cluster 0 5.1f scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.969374+0000 osd.5 (osd.5) 135 : cluster 0 3.1c scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:41.972498+0000 osd.5 (osd.5) 136 : cluster 0 3.1c scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.091948+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.093976+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.394413+0000 osd.7 (osd.7) 157 : cluster 0 2.b scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:42.395719+0000 osd.7 (osd.7) 158 : cluster 0 2.b scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.170070+0000 mgr.y (mgr.14550) 213 : cluster 0 pgmap v188: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.360528+0000 osd.7 (osd.7) 159 : cluster 0 3.15 scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:43.364212+0000 osd.7 (osd.7) 160 : cluster 0 3.15 scrub ok 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.383289+0000 osd.7 (osd.7) 161 : cluster 0 6.3 scrub starts 2024-03-11T21:28:51.220 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.386493+0000 osd.7 (osd.7) 162 : cluster 0 6.3 scrub ok 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.889966+0000 osd.6 (osd.6) 91 : cluster 0 2.1a scrub starts 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:44.892255+0000 osd.6 (osd.6) 92 : cluster 0 2.1a scrub ok 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.082510+0000 osd.4 (osd.4) 121 : cluster 0 6.1a scrub starts 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.084981+0000 osd.4 (osd.4) 122 : cluster 0 6.1a scrub ok 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.171387+0000 mgr.y (mgr.14550) 214 : cluster 0 pgmap v189: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.376453+0000 osd.7 (osd.7) 163 : cluster 0 6.15 scrub starts 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:45.377663+0000 osd.7 (osd.7) 164 : cluster 0 6.15 scrub ok 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:46.235138+0000 osd.1 (osd.1) 117 : cluster 0 5.19 scrub starts 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:46.237354+0000 osd.1 (osd.1) 118 : cluster 0 5.19 scrub ok 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:46.792842+0000 mon.a (mon.0) 921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:46.792938+0000 mon.a (mon.0) 922 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:46.804641+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:46.816242+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:47.172353+0000 mgr.y (mgr.14550) 215 : cluster 0 pgmap v191: 964 pgs: 304 creating+peering, 132 active+clean, 528 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:51.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.170868+0000 osd.1 (osd.1) 119 : cluster 0 6.1 scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.172750+0000 osd.1 (osd.1) 120 : cluster 0 6.1 scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.303386+0000 osd.3 (osd.3) 147 : cluster 0 2.19 scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.305151+0000 osd.3 (osd.3) 148 : cluster 0 2.19 scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:48.321842+0000 mgr.y (mgr.14550) 216 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.362848+0000 osd.7 (osd.7) 165 : cluster 0 2.0 scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.364800+0000 osd.7 (osd.7) 166 : cluster 0 2.0 scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.414844+0000 osd.0 (osd.0) 131 : cluster 0 5.1e scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.415887+0000 osd.0 (osd.0) 132 : cluster 0 5.1e scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.817145+0000 osd.5 (osd.5) 137 : cluster 0 3.5 scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.820379+0000 osd.5 (osd.5) 138 : cluster 0 3.5 scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.841979+0000 osd.2 (osd.2) 101 : cluster 0 6.14 scrub starts 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:48.843700+0000 osd.2 (osd.2) 102 : cluster 0 6.14 scrub ok 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:48.909582+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:49.515940+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:49.516054+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:49.753304+0000 mon.a (mon.0) 923 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: cluster 2024-03-11T21:28:49.756050+0000 mon.a (mon.0) 924 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.785704+0000 mon.a (mon.0) 925 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.787234+0000 mon.a (mon.0) 926 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.787432+0000 mon.a (mon.0) 927 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.787638+0000 mon.a (mon.0) 928 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.788104+0000 mon.a (mon.0) 929 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.788365+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.788686+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.800470+0000 mon.a (mon.0) 932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.800691+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.800903+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.801798+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.801974+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.802268+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.802384+0000 mon.a (mon.0) 938 : audit 0 from='client.? 172.21.15.143:0/3150635234' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.802971+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.804946+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.805486+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.806228+0000 mon.a (mon.0) 942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:51 smithi143 bash[21512]: audit 2024-03-11T21:28:49.814245+0000 mon.a (mon.0) 943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:51.248 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:51.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:51.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:51.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:51.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:38.314035+0000 mgr.y (mgr.14550) 210 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:39.165983+0000 mgr.y (mgr.14550) 211 : cluster 0 pgmap v185: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.309055+0000 osd.1 (osd.1) 115 : cluster 0 2.a scrub starts 2024-03-11T21:28:51.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.310699+0000 osd.1 (osd.1) 116 : cluster 0 2.a scrub ok 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.396071+0000 osd.7 (osd.7) 153 : cluster 0 5.4 scrub starts 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.397919+0000 osd.7 (osd.7) 154 : cluster 0 5.4 scrub ok 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.842277+0000 osd.6 (osd.6) 87 : cluster 0 3.a scrub starts 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:40.847327+0000 osd.6 (osd.6) 88 : cluster 0 3.a scrub ok 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.166767+0000 mgr.y (mgr.14550) 212 : cluster 0 pgmap v186: 196 pgs: 38 creating+peering, 132 active+clean, 26 unknown; 582 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:28:51.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.438499+0000 osd.7 (osd.7) 155 : cluster 0 1.0 scrub starts 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.441852+0000 osd.7 (osd.7) 156 : cluster 0 1.0 scrub ok 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.809688+0000 osd.6 (osd.6) 89 : cluster 0 5.1f scrub starts 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.812900+0000 osd.6 (osd.6) 90 : cluster 0 5.1f scrub ok 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.969374+0000 osd.5 (osd.5) 135 : cluster 0 3.1c scrub starts 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:41.972498+0000 osd.5 (osd.5) 136 : cluster 0 3.1c scrub ok 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.091948+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.093976+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T21:28:51.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.394413+0000 osd.7 (osd.7) 157 : cluster 0 2.b scrub starts 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:42.395719+0000 osd.7 (osd.7) 158 : cluster 0 2.b scrub ok 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.170070+0000 mgr.y (mgr.14550) 213 : cluster 0 pgmap v188: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.360528+0000 osd.7 (osd.7) 159 : cluster 0 3.15 scrub starts 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:43.364212+0000 osd.7 (osd.7) 160 : cluster 0 3.15 scrub ok 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.383289+0000 osd.7 (osd.7) 161 : cluster 0 6.3 scrub starts 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.386493+0000 osd.7 (osd.7) 162 : cluster 0 6.3 scrub ok 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.889966+0000 osd.6 (osd.6) 91 : cluster 0 2.1a scrub starts 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:44.892255+0000 osd.6 (osd.6) 92 : cluster 0 2.1a scrub ok 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.082510+0000 osd.4 (osd.4) 121 : cluster 0 6.1a scrub starts 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.084981+0000 osd.4 (osd.4) 122 : cluster 0 6.1a scrub ok 2024-03-11T21:28:51.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.171387+0000 mgr.y (mgr.14550) 214 : cluster 0 pgmap v189: 580 pgs: 304 creating+peering, 132 active+clean, 144 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.376453+0000 osd.7 (osd.7) 163 : cluster 0 6.15 scrub starts 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:45.377663+0000 osd.7 (osd.7) 164 : cluster 0 6.15 scrub ok 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:46.235138+0000 osd.1 (osd.1) 117 : cluster 0 5.19 scrub starts 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:46.237354+0000 osd.1 (osd.1) 118 : cluster 0 5.19 scrub ok 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:46.792842+0000 mon.a (mon.0) 921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi143-72277-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:46.792938+0000 mon.a (mon.0) 922 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi143-72329-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:46.804641+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:46.816242+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:47.172353+0000 mgr.y (mgr.14550) 215 : cluster 0 pgmap v191: 964 pgs: 304 creating+peering, 132 active+clean, 528 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.170868+0000 osd.1 (osd.1) 119 : cluster 0 6.1 scrub starts 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.172750+0000 osd.1 (osd.1) 120 : cluster 0 6.1 scrub ok 2024-03-11T21:28:51.525 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.303386+0000 osd.3 (osd.3) 147 : cluster 0 2.19 scrub starts 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.305151+0000 osd.3 (osd.3) 148 : cluster 0 2.19 scrub ok 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:48.321842+0000 mgr.y (mgr.14550) 216 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.362848+0000 osd.7 (osd.7) 165 : cluster 0 2.0 scrub starts 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.364800+0000 osd.7 (osd.7) 166 : cluster 0 2.0 scrub ok 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.414844+0000 osd.0 (osd.0) 131 : cluster 0 5.1e scrub starts 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.415887+0000 osd.0 (osd.0) 132 : cluster 0 5.1e scrub ok 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.817145+0000 osd.5 (osd.5) 137 : cluster 0 3.5 scrub starts 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.820379+0000 osd.5 (osd.5) 138 : cluster 0 3.5 scrub ok 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.841979+0000 osd.2 (osd.2) 101 : cluster 0 6.14 scrub starts 2024-03-11T21:28:51.526 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:48.843700+0000 osd.2 (osd.2) 102 : cluster 0 6.14 scrub ok 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:48.909582+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:49.515940+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:49.516054+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:49.753304+0000 mon.a (mon.0) 923 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: cluster 2024-03-11T21:28:49.756050+0000 mon.a (mon.0) 924 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.785704+0000 mon.a (mon.0) 925 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:51.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.787234+0000 mon.a (mon.0) 926 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.787432+0000 mon.a (mon.0) 927 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.787638+0000 mon.a (mon.0) 928 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.788104+0000 mon.a (mon.0) 929 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.788365+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.788686+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.800470+0000 mon.a (mon.0) 932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.800691+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.800903+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.801798+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.801974+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.802268+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.802384+0000 mon.a (mon.0) 938 : audit 0 from='client.? 172.21.15.143:0/3150635234' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:51.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.802971+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:51.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.804946+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:51.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.805486+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:28:51.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.806228+0000 mon.a (mon.0) 942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:51.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:51 smithi203 bash[23654]: audit 2024-03-11T21:28:49.814245+0000 mon.a (mon.0) 943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi143-71594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:51.558 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:51.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.684 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:51.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:51.821 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:51.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-11T21:28:51.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] Global test environment set-up. 2024-03-11T21:28:51.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-11T21:28:51.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-11T21:28:51.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjects (403 ms) 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (52 ms) 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo2,foo3 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo2 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo3 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo4,foo5 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo4 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo5 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo6,foo7 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo7 2024-03-11T21:28:51.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo6 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo4 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo5 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo7 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo6 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo1 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo1 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo2 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo3 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (113 ms) 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 1 0 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 10 0 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 13 0 2024-03-11T21:28:51.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 7 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 14 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 0 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 15 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 11 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 5 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 8 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 6 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 3 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 4 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 12 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 9 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2 0 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (98 ms) 2024-03-11T21:28:51.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: x cursor=MIN 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=1 cursor=12:02547ec2:::1:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=10 cursor=12:52ea6a34:::10:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=13 cursor=12:566253c9:::13:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=7 cursor=12:5c6b0b28:::7:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=14 cursor=12:62a1935d:::14:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=0 cursor=12:6cac518f:::0:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=15 cursor=12:863748b0:::15:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=11 cursor=12:89d3ae78:::11:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=5 cursor=12:b29083e3:::5:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=8 cursor=12:bd63b0f1:::8:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=6 cursor=12:c4fdafeb:::6:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=3 cursor=12:cfc208b3:::3:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=4 cursor=12:d83876eb:::4:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=12 cursor=12:de5d7c5f:::12:head 2024-03-11T21:28:51.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=9 cursor=12:e960b815:::9:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=2 cursor=12:f905c69b:::2:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=1 cursor=12:02547ec2:::1:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=10 cursor=12:52ea6a34:::10:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=13 cursor=12:566253c9:::13:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:566253c9:::13:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=7 cursor=12:5c6b0b28:::7:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-03-11T21:28:51.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-03-11T21:28:51.949 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:51.997 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=14 cursor=12:62a1935d:::14:head 2024-03-11T21:28:51.997 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=0 cursor=12:6cac518f:::0:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=15 cursor=12:863748b0:::15:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:863748b0:::15:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=11 cursor=12:89d3ae78:::11:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=5 cursor=12:b29083e3:::5:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-03-11T21:28:51.998 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=8 cursor=12:bd63b0f1:::8:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=6 cursor=12:c4fdafeb:::6:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=3 cursor=12:cfc208b3:::3:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=4 cursor=12:d83876eb:::4:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=12 cursor=12:de5d7c5f:::12:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-03-11T21:28:51.999 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=9 cursor=12:e960b815:::9:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:e960b815:::9:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=2 cursor=12:f905c69b:::2:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:d83876eb:::4:head expected=12:d83876eb:::4:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=4 expected=4 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:cfc208b3:::3:head expected=12:cfc208b3:::3:head 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-03-11T21:28:52.000 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=3 expected=3 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:f905c69b:::2:head expected=12:f905c69b:::2:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=2 expected=2 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:e960b815:::9:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:e960b815:::9:head expected=12:e960b815:::9:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=9 expected=9 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:de5d7c5f:::12:head expected=12:de5d7c5f:::12:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=12 expected=12 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-03-11T21:28:52.001 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:52ea6a34:::10:head expected=12:52ea6a34:::10:head 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=10 expected=10 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:c4fdafeb:::6:head expected=12:c4fdafeb:::6:head 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=6 expected=6 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:bd63b0f1:::8:head expected=12:bd63b0f1:::8:head 2024-03-11T21:28:52.002 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-03-11T21:28:52.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:52.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:52.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:52.209 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:52.228 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:28:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:52.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: cluster 2024-03-11T21:28:49.179215+0000 mgr.y (mgr.14550) 217 : cluster 0 pgmap v192: 964 pgs: 683 creating+peering, 132 active+clean, 149 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:49.814352+0000 mon.a (mon.0) 944 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:49.814465+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:49.844422+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.143:0/1999332124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: cluster 2024-03-11T21:28:50.359840+0000 osd.0 (osd.0) 133 : cluster 0 5.3 scrub starts 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: cluster 2024-03-11T21:28:50.362368+0000 osd.0 (osd.0) 134 : cluster 0 5.3 scrub ok 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: cluster 2024-03-11T21:28:51.118782+0000 mon.a (mon.0) 946 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:51.119274+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: cluster 2024-03-11T21:28:51.130344+0000 mon.a (mon.0) 947 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:51.131826+0000 mon.a (mon.0) 948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:51.145896+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:51.146259+0000 mon.a (mon.0) 950 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[21512]: audit 2024-03-11T21:28:51.151676+0000 mon.a (mon.0) 951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.347 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: cluster 2024-03-11T21:28:49.179215+0000 mgr.y (mgr.14550) 217 : cluster 0 pgmap v192: 964 pgs: 683 creating+peering, 132 active+clean, 149 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:49.814352+0000 mon.a (mon.0) 944 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:49.814465+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:49.844422+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.143:0/1999332124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: cluster 2024-03-11T21:28:50.359840+0000 osd.0 (osd.0) 133 : cluster 0 5.3 scrub starts 2024-03-11T21:28:52.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: cluster 2024-03-11T21:28:50.362368+0000 osd.0 (osd.0) 134 : cluster 0 5.3 scrub ok 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: cluster 2024-03-11T21:28:51.118782+0000 mon.a (mon.0) 946 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:51.119274+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: cluster 2024-03-11T21:28:51.130344+0000 mon.a (mon.0) 947 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:51.131826+0000 mon.a (mon.0) 948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:51.145896+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:51.146259+0000 mon.a (mon.0) 950 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:52 smithi143 bash[29284]: audit 2024-03-11T21:28:51.151676+0000 mon.a (mon.0) 951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:52.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: cluster 2024-03-11T21:28:49.179215+0000 mgr.y (mgr.14550) 217 : cluster 0 pgmap v192: 964 pgs: 683 creating+peering, 132 active+clean, 149 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:49.814352+0000 mon.a (mon.0) 944 : audit 1 from='client.? 172.21.15.143:0/1230579109' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:49.814465+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi143-71620-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:49.844422+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.143:0/1999332124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: cluster 2024-03-11T21:28:50.359840+0000 osd.0 (osd.0) 133 : cluster 0 5.3 scrub starts 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: cluster 2024-03-11T21:28:50.362368+0000 osd.0 (osd.0) 134 : cluster 0 5.3 scrub ok 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: cluster 2024-03-11T21:28:51.118782+0000 mon.a (mon.0) 946 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:51.119274+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: cluster 2024-03-11T21:28:51.130344+0000 mon.a (mon.0) 947 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:51.131826+0000 mon.a (mon.0) 948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:51.145896+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:51.146259+0000 mon.a (mon.0) 950 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:52 smithi203 bash[23654]: audit 2024-03-11T21:28:51.151676+0000 mon.a (mon.0) 951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:52.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:52.577 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:52.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:52.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:52.822 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:52.845 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:52.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:53.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:53.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:53.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:53.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.353 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:53.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.480 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:53.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:53.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:53.756 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: cluster 2024-03-11T21:28:50.062094+0000 osd.4 (osd.4) 125 : cluster 0 4.1 scrub starts 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: cluster 2024-03-11T21:28:50.063065+0000 osd.4 (osd.4) 126 : cluster 0 4.1 scrub ok 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.153479+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.153570+0000 mon.a (mon.0) 953 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.153633+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.153699+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.154939+0000 mon.a (mon.0) 956 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155038+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155111+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155199+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155282+0000 mon.a (mon.0) 960 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155362+0000 mon.a (mon.0) 961 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155430+0000 mon.a (mon.0) 962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155510+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155592+0000 mon.a (mon.0) 964 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.155711+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.167578+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.143:0/2135306702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.171064+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.143:0/232344682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:51.856080+0000 mon.c (mon.2) 121 : audit 0 from='client.? 172.21.15.143:0/635897100' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:52.137083+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:53.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: cluster 2024-03-11T21:28:52.194315+0000 mon.a (mon.0) 966 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:28:53.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:52.212798+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:53.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:52.213797+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:53.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:53 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251010+0000 mon.a (mon.0) 969 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:53.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.880 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:53.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:54.013 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:54.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:54.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:54.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: cluster 2024-03-11T21:28:50.062094+0000 osd.4 (osd.4) 125 : cluster 0 4.1 scrub starts 2024-03-11T21:28:54.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: cluster 2024-03-11T21:28:50.063065+0000 osd.4 (osd.4) 126 : cluster 0 4.1 scrub ok 2024-03-11T21:28:54.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.153479+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:54.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.153570+0000 mon.a (mon.0) 953 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.153633+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.153699+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.154939+0000 mon.a (mon.0) 956 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155038+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155111+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155199+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155282+0000 mon.a (mon.0) 960 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155362+0000 mon.a (mon.0) 961 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155430+0000 mon.a (mon.0) 962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155510+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155592+0000 mon.a (mon.0) 964 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.155711+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.167578+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.143:0/2135306702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.171064+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.143:0/232344682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:51.856080+0000 mon.c (mon.2) 121 : audit 0 from='client.? 172.21.15.143:0/635897100' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:52.137083+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: cluster 2024-03-11T21:28:52.194315+0000 mon.a (mon.0) 966 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:52.212798+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:52.213797+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251010+0000 mon.a (mon.0) 969 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: cluster 2024-03-11T21:28:50.062094+0000 osd.4 (osd.4) 125 : cluster 0 4.1 scrub starts 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: cluster 2024-03-11T21:28:50.063065+0000 osd.4 (osd.4) 126 : cluster 0 4.1 scrub ok 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.153479+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.143:0/2386525285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.153570+0000 mon.a (mon.0) 953 : audit 1 from='client.? 172.21.15.143:0/1335918999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi143-71853-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.153633+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.143:0/2445418295' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi143-71868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.153699+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi143-71649-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.154939+0000 mon.a (mon.0) 956 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-71773-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155038+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi143-71604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155111+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi143-71612-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155199+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi143-71636-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155282+0000 mon.a (mon.0) 960 : audit 1 from='client.? 172.21.15.143:0/1167506555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi143-71838-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155362+0000 mon.a (mon.0) 961 : audit 1 from='client.? 172.21.15.143:0/2694737065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-72044-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155430+0000 mon.a (mon.0) 962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi143-71641-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155510+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155592+0000 mon.a (mon.0) 964 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.155711+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72050-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.167578+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.143:0/2135306702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.171064+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.143:0/232344682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:51.856080+0000 mon.c (mon.2) 121 : audit 0 from='client.? 172.21.15.143:0/635897100' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:52.137083+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:54.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: cluster 2024-03-11T21:28:52.194315+0000 mon.a (mon.0) 966 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:28:54.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:52.212798+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:52.213797+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:54.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:53 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251010+0000 mon.a (mon.0) 969 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi143-72277-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:54.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:54.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:54.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:54.388 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:54.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=8 list_parallel: process_1_[72118]: starting. 2024-03-11T21:28:54.407 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: creating pool ceph_test_rados_list_parallel.smithi143-72081 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: created object 0... 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: created object 25... 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: created object 49... 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: finishing. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[72118]: shutting down. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: starting. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: listing objects. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: listed object 0... 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: listed object 25... 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: saw 50 objects 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[72119]: shutting down. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: starting. 2024-03-11T21:28:54.408 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: creating pool ceph_test_rados_list_parallel.smithi143-72081 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: created object 0... 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: created object 25... 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: created object 49... 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: finishing. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[74276]: shutting down. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: starting. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: listing objects. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: listed object 0... 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: listed object 25... 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: saw 43 objects 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[74277]: shutting down. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: starting. 2024-03-11T21:28:54.409 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: removed 25 objects... 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: removed half of the objects 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: removed 50 objects... 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: removed 50 objects 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[74278]: shutting down. 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: starting. 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: creating pool ceph_test_rados_list_parallel.smithi143-72081 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: created object 0... 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: created object 25... 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: created object 49... 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: finishing. 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[74456]: shutting down. 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.410 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: starting. 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: listing objects. 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: listed object 0... 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: listed object 25... 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: listed object 50... 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: saw 56 objects 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[74457]: shutting down. 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: starting. 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: added 25 objects... 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: added half of the objects 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: added 50 objects... 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: added 50 objects 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[74458]: shutting down. 2024-03-11T21:28:54.411 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.412 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:54.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:54.523 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:54.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:54.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ********************** handler_error: Running main() from gmock_main.cc 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [----------] Global test environment set-up. 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (18880 ms) 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [----------] 1 test from neocls_handler_error (18880 ms total) 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: 2024-03-11T21:28:54.732 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [----------] Global test environment tear-down 2024-03-11T21:28:54.733 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [==========] 1 test from 1 test suite ran. (18880 ms total) 2024-03-11T21:28:54.733 INFO:tasks.workunit.client.0.smithi143.stdout: handler_error: [ PASSED ] 1 test. 2024-03-11T21:28:54.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (18932 ms) 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (18932 ms total) 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (18932 ms total) 2024-03-11T21:28:54.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-11T21:28:54.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:54.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:55.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:55.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:55.158 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.179 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:55.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:55.308 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:55.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:55.434 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:55.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:55.561 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:55.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:55.685 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:55.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:55.814 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:55.917 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.939 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (9 ms) 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (7 ms) 2024-03-11T21:28:56.016 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (26 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (114 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (32 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2024-03-11T21:28:56.017 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (6 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (19 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (12 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (12 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (277 ms total) 2024-03-11T21:28:56.018 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: 2024-03-11T21:28:56.019 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-11T21:28:56.019 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (20256 ms total) 2024-03-11T21:28:56.019 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-11T21:28:56.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:56.078 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:56.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: cluster 2024-03-11T21:28:51.180315+0000 mgr.y (mgr.14550) 218 : cluster 0 pgmap v195: 1220 pgs: 632 creating+peering, 132 active+clean, 456 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251100+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251176+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251252+0000 mon.a (mon.0) 972 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251329+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.251399+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.278894+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:52.337330+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.138784+0000 mon.c (mon.2) 125 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.687069+0000 mon.a (mon.0) 975 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:28:56.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.710850+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.710951+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.726452+0000 mon.a (mon.0) 978 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.749110+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.750643+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:53.756910+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:54.140336+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:54.232212+0000 mon.a (mon.0) 981 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:54.234141+0000 mon.a (mon.0) 982 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:54.234514+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:56.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:56 smithi203 bash[23654]: audit 2024-03-11T21:28:54.250271+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.205 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout:********* 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: starting. 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: creating pool ceph_test_rados_list_parallel.smithi143-72081 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: created object 0... 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: created object 25... 2024-03-11T21:28:56.220 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: created object 49... 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: finishing. 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[74632]: shutting down. 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: starting. 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: listing objects. 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: listed object 0... 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: listed object 25... 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: listed object 50... 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: listed object 75... 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: saw 98 objects 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[74633]: shutting down. 2024-03-11T21:28:56.221 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: starting. 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: added 25 objects... 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: added half of the objects 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: added 50 objects... 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: added 50 objects 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[74634]: shutting down. 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: starting. 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: removed 25 objects... 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: removed half of the objects 2024-03-11T21:28:56.222 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: removed 50 objects... 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: removed 50 objects 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[74636]: shutting down. 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: starting. 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: added 25 objects... 2024-03-11T21:28:56.223 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: added half of the objects 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: added 50 objects... 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: added 50 objects 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[74635]: shutting down. 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.224 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: starting. 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: creating pool ceph_test_rados_list_parallel.smithi143-72081 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: created object 0... 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: created object 25... 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: created object 49... 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: finishing. 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[74740]: shutting down. 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: starting. 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listing objects. 2024-03-11T21:28:56.225 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 0... 2024-03-11T21:28:56.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:56.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: cluster 2024-03-11T21:28:51.180315+0000 mgr.y (mgr.14550) 218 : cluster 0 pgmap v195: 1220 pgs: 632 creating+peering, 132 active+clean, 456 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251100+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251176+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251252+0000 mon.a (mon.0) 972 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251329+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.414 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.251399+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.278894+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:52.337330+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.138784+0000 mon.c (mon.2) 125 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.687069+0000 mon.a (mon.0) 975 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.710850+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.710951+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.726452+0000 mon.a (mon.0) 978 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:28:56.415 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.749110+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.416 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: cluster 2024-03-11T21:28:51.180315+0000 mgr.y (mgr.14550) 218 : cluster 0 pgmap v195: 1220 pgs: 632 creating+peering, 132 active+clean, 456 unknown; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:28:56.416 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251100+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi143-72329-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:28:56.416 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251176+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi143-71918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.416 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251252+0000 mon.a (mon.0) 972 : audit 1 from='client.? 172.21.15.143:0/4223449851' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.416 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251329+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.143:0/1384281194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi143-71818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.251399+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.278894+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:52.337330+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.138784+0000 mon.c (mon.2) 125 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.687069+0000 mon.a (mon.0) 975 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.710850+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.710951+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71939-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.726452+0000 mon.a (mon.0) 978 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.749110+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.750643+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.417 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:53.756910+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:54.140336+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:54.232212+0000 mon.a (mon.0) 981 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:54.234141+0000 mon.a (mon.0) 982 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:54.234514+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[21512]: audit 2024-03-11T21:28:54.250271+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.750643+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:53.756910+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:54.140336+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:54.232212+0000 mon.a (mon.0) 981 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:54.234141+0000 mon.a (mon.0) 982 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:56.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:54.234514+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:56.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:56 smithi143 bash[29284]: audit 2024-03-11T21:28:54.250271+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:56.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:56.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:56.569 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.592 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:56.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:56.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:56.719 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:56.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.850 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:56.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:56.981 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:57.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:57.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:57.137 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:28:57.137 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:28:57.137 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:28:57.137 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:28:57.137 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:28:57.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.233 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:57.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:57.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:57.503 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.184221+0000 mgr.y (mgr.14550) 219 : cluster 0 pgmap v197: 1140 pgs: 185 creating+peering, 930 active+clean, 25 unknown; 626 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 327 op/s 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.323632+0000 osd.3 (osd.3) 149 : cluster 0 5.0 scrub starts 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.324997+0000 osd.3 (osd.3) 150 : cluster 0 5.0 scrub ok 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.250820+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.682796+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.683138+0000 mgr.y (mgr.14550) 220 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.683693+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.683849+0000 mgr.y (mgr.14550) 221 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.684450+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.684701+0000 mgr.y (mgr.14550) 222 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.685372+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.685526+0000 mgr.y (mgr.14550) 223 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.686104+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.686259+0000 mgr.y (mgr.14550) 224 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.686809+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.686937+0000 mgr.y (mgr.14550) 225 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.687377+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.687512+0000 mgr.y (mgr.14550) 226 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.687890+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.688038+0000 mgr.y (mgr.14550) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.688423+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.554 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.688569+0000 mgr.y (mgr.14550) 228 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.689015+0000 mon.c (mon.2) 138 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.689213+0000 mgr.y (mgr.14550) 229 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.729057+0000 mon.a (mon.0) 985 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:54.817237+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.924387+0000 osd.2 (osd.2) 103 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.926982+0000 osd.2 (osd.2) 104 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:55.143014+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.185077+0000 mgr.y (mgr.14550) 230 : cluster 0 pgmap v200: 940 pgs: 167 creating+peering, 624 active+clean, 149 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 70 KiB/s wr, 439 op/s 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:55.187457+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:55.187970+0000 mon.c (mon.2) 141 : audit 1 from='client.? 172.21.15.143:0/2303779099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.206028+0000 osd.1 (osd.1) 121 : cluster 0 6.4 scrub starts 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.207707+0000 osd.1 (osd.1) 122 : cluster 0 6.4 scrub ok 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.016720+0000 mon.a (mon.0) 986 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:28:57.555 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.037348+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.044942+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.053679+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.054651+0000 mon.a (mon.0) 989 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.085361+0000 mon.c (mon.2) 143 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.088815+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.145460+0000 mon.c (mon.2) 144 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.971588+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.972269+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.973599+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.556 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[21512]: audit 2024-03-11T21:28:56.978875+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.184221+0000 mgr.y (mgr.14550) 219 : cluster 0 pgmap v197: 1140 pgs: 185 creating+peering, 930 active+clean, 25 unknown; 626 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 327 op/s 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.323632+0000 osd.3 (osd.3) 149 : cluster 0 5.0 scrub starts 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.324997+0000 osd.3 (osd.3) 150 : cluster 0 5.0 scrub ok 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.250820+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.682796+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.683138+0000 mgr.y (mgr.14550) 220 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.683693+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.683849+0000 mgr.y (mgr.14550) 221 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.684450+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.684701+0000 mgr.y (mgr.14550) 222 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.685372+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.685526+0000 mgr.y (mgr.14550) 223 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.686104+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.686259+0000 mgr.y (mgr.14550) 224 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.686809+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.686937+0000 mgr.y (mgr.14550) 225 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.687377+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.687512+0000 mgr.y (mgr.14550) 226 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.687890+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.688038+0000 mgr.y (mgr.14550) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.688423+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.688569+0000 mgr.y (mgr.14550) 228 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.558 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.689015+0000 mon.c (mon.2) 138 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.689213+0000 mgr.y (mgr.14550) 229 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.729057+0000 mon.a (mon.0) 985 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:54.817237+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.924387+0000 osd.2 (osd.2) 103 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.926982+0000 osd.2 (osd.2) 104 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:55.143014+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.185077+0000 mgr.y (mgr.14550) 230 : cluster 0 pgmap v200: 940 pgs: 167 creating+peering, 624 active+clean, 149 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 70 KiB/s wr, 439 op/s 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:55.187457+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:55.187970+0000 mon.c (mon.2) 141 : audit 1 from='client.? 172.21.15.143:0/2303779099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.206028+0000 osd.1 (osd.1) 121 : cluster 0 6.4 scrub starts 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.207707+0000 osd.1 (osd.1) 122 : cluster 0 6.4 scrub ok 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.016720+0000 mon.a (mon.0) 986 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.037348+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:57.559 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.044942+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.053679+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.054651+0000 mon.a (mon.0) 989 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.085361+0000 mon.c (mon.2) 143 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.088815+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.145460+0000 mon.c (mon.2) 144 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.971588+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.972269+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.973599+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.560 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:57 smithi143 bash[29284]: audit 2024-03-11T21:28:56.978875+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:57.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:57.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:57.755 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:57.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.184221+0000 mgr.y (mgr.14550) 219 : cluster 0 pgmap v197: 1140 pgs: 185 creating+peering, 930 active+clean, 25 unknown; 626 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 327 op/s 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.323632+0000 osd.3 (osd.3) 149 : cluster 0 5.0 scrub starts 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.324997+0000 osd.3 (osd.3) 150 : cluster 0 5.0 scrub ok 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.250820+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.682796+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.683138+0000 mgr.y (mgr.14550) 220 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.683693+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.683849+0000 mgr.y (mgr.14550) 221 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.684450+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.684701+0000 mgr.y (mgr.14550) 222 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.685372+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.685526+0000 mgr.y (mgr.14550) 223 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.686104+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.686259+0000 mgr.y (mgr.14550) 224 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.686809+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.686937+0000 mgr.y (mgr.14550) 225 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.687377+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.687512+0000 mgr.y (mgr.14550) 226 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:28:57.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.687890+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.688038+0000 mgr.y (mgr.14550) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:28:57.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.688423+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.688569+0000 mgr.y (mgr.14550) 228 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:28:57.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.689015+0000 mon.c (mon.2) 138 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.689213+0000 mgr.y (mgr.14550) 229 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.729057+0000 mon.a (mon.0) 985 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-71868-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:54.817237+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.924387+0000 osd.2 (osd.2) 103 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.926982+0000 osd.2 (osd.2) 104 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:55.143014+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.185077+0000 mgr.y (mgr.14550) 230 : cluster 0 pgmap v200: 940 pgs: 167 creating+peering, 624 active+clean, 149 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 70 KiB/s wr, 439 op/s 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:55.187457+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:55.187970+0000 mon.c (mon.2) 141 : audit 1 from='client.? 172.21.15.143:0/2303779099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.206028+0000 osd.1 (osd.1) 121 : cluster 0 6.4 scrub starts 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.207707+0000 osd.1 (osd.1) 122 : cluster 0 6.4 scrub ok 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.016720+0000 mon.a (mon.0) 986 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.037348+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.044942+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-71929-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.053679+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.054651+0000 mon.a (mon.0) 989 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.085361+0000 mon.c (mon.2) 143 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.088815+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:57.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.145460+0000 mon.c (mon.2) 144 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:57.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.971588+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:57.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.972269+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.973599+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:57.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:57 smithi203 bash[23654]: audit 2024-03-11T21:28:56.978875+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:57.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:57.886 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:57.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:58.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:58.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:58.145 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:58.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:58.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:58.300 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 25... 2024-03-11T21:28:58.300 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 50... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 75... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 100... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: listed object 125... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: saw 150 objects 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[74741]: shutting down. 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: starting. 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: added 25 objects... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: added half of the objects 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: added 50 objects... 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: added 50 objects 2024-03-11T21:28:58.301 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[74742]: shutting down. 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-03-11T21:28:58.302 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******* SUCCESS ********** 2024-03-11T21:28:58.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:58.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:58.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:28:58.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:58.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:58.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:58.672 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:58.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:28:58.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.384902+0000 osd.0 (osd.0) 135 : cluster 0 6.0 scrub starts 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.386059+0000 osd.0 (osd.0) 136 : cluster 0 6.0 scrub ok 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.361204+0000 osd.3 (osd.3) 151 : cluster 0 6.b scrub starts 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.362427+0000 osd.3 (osd.3) 152 : cluster 0 6.b scrub ok 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.430972+0000 osd.0 (osd.0) 137 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.440461+0000 osd.0 (osd.0) 138 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.179826+0000 osd.1 (osd.1) 123 : cluster 0 6.1d scrub starts 2024-03-11T21:28:58.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.181103+0000 osd.1 (osd.1) 124 : cluster 0 6.1d scrub ok 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:56.979517+0000 mon.c (mon.2) 145 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.146559+0000 mon.c (mon.2) 146 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.410115+0000 mon.a (mon.0) 995 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.417362+0000 mon.a (mon.0) 996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.424474+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.424531+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.424570+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.424617+0000 mon.a (mon.0) 1000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.431507+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.431639+0000 mon.c (mon.2) 148 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:58.809 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.882452+0000 mon.c (mon.2) 149 : audit 1 from='client.? 172.21.15.143:0/300389713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.384902+0000 osd.0 (osd.0) 135 : cluster 0 6.0 scrub starts 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.386059+0000 osd.0 (osd.0) 136 : cluster 0 6.0 scrub ok 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.361204+0000 osd.3 (osd.3) 151 : cluster 0 6.b scrub starts 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.362427+0000 osd.3 (osd.3) 152 : cluster 0 6.b scrub ok 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.430972+0000 osd.0 (osd.0) 137 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.440461+0000 osd.0 (osd.0) 138 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.179826+0000 osd.1 (osd.1) 123 : cluster 0 6.1d scrub starts 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.181103+0000 osd.1 (osd.1) 124 : cluster 0 6.1d scrub ok 2024-03-11T21:28:58.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:56.979517+0000 mon.c (mon.2) 145 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.146559+0000 mon.c (mon.2) 146 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.410115+0000 mon.a (mon.0) 995 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.417362+0000 mon.a (mon.0) 996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.424474+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.424531+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.424570+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.424617+0000 mon.a (mon.0) 1000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.431507+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.431639+0000 mon.c (mon.2) 148 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.882452+0000 mon.c (mon.2) 149 : audit 1 from='client.? 172.21.15.143:0/300389713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:57.882942+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:58.148077+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: cluster 2024-03-11T21:28:58.314296+0000 mon.a (mon.0) 1001 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:28:58.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[29284]: audit 2024-03-11T21:28:58.318551+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:58.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:57.882942+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:58.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:58.148077+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:58.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: cluster 2024-03-11T21:28:58.314296+0000 mon.a (mon.0) 1001 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:28:58.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:28:58 smithi143 bash[21512]: audit 2024-03-11T21:28:58.318551+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:58.829 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:58.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:58.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.384902+0000 osd.0 (osd.0) 135 : cluster 0 6.0 scrub starts 2024-03-11T21:28:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.386059+0000 osd.0 (osd.0) 136 : cluster 0 6.0 scrub ok 2024-03-11T21:28:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.361204+0000 osd.3 (osd.3) 151 : cluster 0 6.b scrub starts 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.362427+0000 osd.3 (osd.3) 152 : cluster 0 6.b scrub ok 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.430972+0000 osd.0 (osd.0) 137 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.440461+0000 osd.0 (osd.0) 138 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.179826+0000 osd.1 (osd.1) 123 : cluster 0 6.1d scrub starts 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.181103+0000 osd.1 (osd.1) 124 : cluster 0 6.1d scrub ok 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:56.979517+0000 mon.c (mon.2) 145 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.146559+0000 mon.c (mon.2) 146 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.410115+0000 mon.a (mon.0) 995 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.417362+0000 mon.a (mon.0) 996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.424474+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.424531+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.424570+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-71594-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.424617+0000 mon.a (mon.0) 1000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.431507+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.143:0/3033351644' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.431639+0000 mon.c (mon.2) 148 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.882452+0000 mon.c (mon.2) 149 : audit 1 from='client.? 172.21.15.143:0/300389713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:57.882942+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:58.148077+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: cluster 2024-03-11T21:28:58.314296+0000 mon.a (mon.0) 1001 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:28:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:28:58 smithi203 bash[23654]: audit 2024-03-11T21:28:58.318551+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]: dispatch 2024-03-11T21:28:59.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:59.088 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:59.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:59.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:59.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:28:59.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.344 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:59.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:59.473 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:59.576 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:59.595 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:59.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:59.717 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:59.820 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.844 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:59.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:59.972 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:00.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.094 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:00.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:00.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:00.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:00.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:00.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:00.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:00.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:00.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:00.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:00.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:00.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:01.004 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:01.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:01.133 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:01.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:01.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:01.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 -- 172.21.15.143:0/3643531677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec106dc0 msgr2=0x7f5aec1071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/3643531677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec106dc0 0x7f5aec1071a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098f0 tx=0x7f5adc009c00 comp rx=0 tx=0).stop 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 -- 172.21.15.143:0/3643531677 shutdown_connections 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/3643531677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec106dc0 0x7f5aec1071a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/3643531677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec101700 0x7f5aec10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/3643531677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec100d40 0x7f5aec1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 -- 172.21.15.143:0/3643531677 >> 172.21.15.143:0/3643531677 conn(0x7f5aec076d80 msgr2=0x7f5aec077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:01.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 -- 172.21.15.143:0/3643531677 shutdown_connections 2024-03-11T21:29:01.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.349+0000 7f5af0ba5700 1 -- 172.21.15.143:0/3643531677 wait complete. 2024-03-11T21:29:01.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 librados: starting msgr at 2024-03-11T21:29:01.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 librados: starting objecter 2024-03-11T21:29:01.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 Processor -- start 2024-03-11T21:29:01.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 -- start start 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 librados: setting wanted keys 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 librados: calling monclient init 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec100d40 0x7f5aec1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec101700 0x7f5aec198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 0x7f5aec19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1104e0 con 0x7f5aec100d40 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec110360 con 0x7f5aec101700 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5af0ba5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec110660 con 0x7f5aec106dc0 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 0x7f5aec19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 0x7f5aec19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:41030/0 (socket says 172.21.15.143:41030) 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 learned_addr learned my addr 172.21.15.143:0/989085384 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:01.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec101700 0x7f5aec198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec101700 msgr2=0x7f5aec198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec101700 0x7f5aec198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec100d40 msgr2=0x7f5aec1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec100d40 0x7f5aec1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec19cc30 con 0x7f5aec106dc0 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 0x7f5aec19c530 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007ee0 tx=0x7f5ae400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5adb7fe700 10 client.?.objecter ms_handle_connect 0x7f5aec106dc0 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.353+0000 7f5adb7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400dbd0 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec19cf20 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5af0ba5700 10 client.25289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1111d0 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5af0ba5700 1 librados: init done 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.365+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5aec102400 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.369+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5ae400dd70 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.369+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4026e60 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5ae401a020 con 0x7f5aec106dc0 2024-03-11T21:29:01.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5ad406f820 0x7f5ad4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(149..149 src has 87..149) v4 ==== 21109+0+0 (secure 0 0 0) 0x7f5ae4026430 con 0x7f5aec106dc0 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 osd_map(149..149 src has 87..149) v4 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 3 client.25289.objecter handle_osd_map got epochs [149,149] > 0 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 3 client.25289.objecter handle_osd_map decoding full epoch 149 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5ad406f820 0x7f5ad4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 20 client.25289.objecter dump_active .. 0 homeless 2024-03-11T21:29:01.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v149) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5ae4010070 con 0x7f5aec106dc0 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5ad406f820 0x7f5ad4071ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009680 tx=0x7f5adc009f50 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 1 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5aec0629f0 0x7f5aec062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 1 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 0 1 2024-03-11T21:29:01.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 4 2024-03-11T21:29:01.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.373+0000 7f5af0ba5700 10 client.25289.objecter _send_command 1 2024-03-11T21:29:01.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.377+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec199100 con 0x7f5aec0629f0 2024-03-11T21:29:01.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.381+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5ad406f820 2024-03-11T21:29:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.381+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5aec0629f0 0x7f5aec062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.381+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec0629f0 2024-03-11T21:29:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.381+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5aec0629f0 0x7f5aec062dd0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.385+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec199100 con 0x7f5aec0629f0 2024-03-11T21:29:01.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.385+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:29:01.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.385+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 1 = system:0 2024-03-11T21:29:01.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.385+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 4 2024-03-11T21:29:01.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.385+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 0 1 2024-03-11T21:29:01.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 2 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 2 2024-03-11T21:29:01.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec04faa0 osd=0 3 2024-03-11T21:29:01.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 0 2 2024-03-11T21:29:01.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 4 2024-03-11T21:29:01.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 10 client.25289.objecter _send_command 2 2024-03-11T21:29:01.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.425+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec062470 con 0x7f5aec0629f0 2024-03-11T21:29:01.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.429+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec062470 con 0x7f5aec0629f0 2024-03-11T21:29:01.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.429+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.429+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.429+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 4 2024-03-11T21:29:01.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.429+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 0 2 2024-03-11T21:29:01.406 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 3 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f5aec108790 0x7f5aec1114e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 3 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 1 3 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 4 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 10 client.25289.objecter _send_command 3 2024-03-11T21:29:01.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.433+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec1131c0 con 0x7f5aec108790 2024-03-11T21:29:01.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:29:01.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:29:01.418 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:29:01.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:29:01.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.445+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f5aec108790 0x7f5aec1114e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f5aec108790 0x7f5aec1114e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec108790 2024-03-11T21:29:01.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec1131c0 con 0x7f5aec108790 2024-03-11T21:29:01.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:29:01.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 3 = system:0 2024-03-11T21:29:01.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 4 2024-03-11T21:29:01.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.449+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 1 3 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 4 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 4 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec02d830 osd=1 3 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 1 4 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 4 2024-03-11T21:29:01.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 10 client.25289.objecter _send_command 4 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.477+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec02dbf0 con 0x7f5aec108790 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec02dbf0 con 0x7f5aec108790 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 4 2024-03-11T21:29:01.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 1 4 2024-03-11T21:29:01.461 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:29:01.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 5 2024-03-11T21:29:01.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5aec107e70 0x7f5aec02ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 5 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 2 5 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 4 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 10 client.25289.objecter _send_command 5 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec02d3c0 con 0x7f5aec107e70 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5aec107e70 0x7f5aec02ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5aec107e70 0x7f5aec02ce80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.481+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec107e70 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.485+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec02d3c0 con 0x7f5aec107e70 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.485+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:29:01.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.485+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 5 = system:0 2024-03-11T21:29:01.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.485+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 4 2024-03-11T21:29:01.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.485+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 2 5 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 6 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 6 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec107d00 osd=2 3 2024-03-11T21:29:01.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 2 6 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 4 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 10 client.25289.objecter _send_command 6 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.501+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec02d600 con 0x7f5aec107e70 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec02d600 con 0x7f5aec107e70 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 4 2024-03-11T21:29:01.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 2 6 2024-03-11T21:29:01.480 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 7 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5aec112700 0x7f5aec0729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 7 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 3 7 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 4 2024-03-11T21:29:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 10 client.25289.objecter _send_command 7 2024-03-11T21:29:01.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec072ef0 con 0x7f5aec112700 2024-03-11T21:29:01.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5aec112700 0x7f5aec0729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5aec112700 0x7f5aec0729b0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec112700 2024-03-11T21:29:01.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec072ef0 con 0x7f5aec112700 2024-03-11T21:29:01.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:29:01.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 7 = system:0 2024-03-11T21:29:01.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 4 2024-03-11T21:29:01.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.505+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 3 7 2024-03-11T21:29:01.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:01.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 8 2024-03-11T21:29:01.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 8 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec02cb50 osd=3 3 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 3 8 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 4 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 10 client.25289.objecter _send_command 8 2024-03-11T21:29:01.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec072ef0 con 0x7f5aec112700 2024-03-11T21:29:01.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec072ef0 con 0x7f5aec112700 2024-03-11T21:29:01.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 4 2024-03-11T21:29:01.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 3 8 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 9 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5aec073bd0 0x7f5aec1c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 9 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 4 9 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 4 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 10 client.25289.objecter _send_command 9 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec1c4e60 con 0x7f5aec073bd0 2024-03-11T21:29:01.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.525+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5aec073bd0 0x7f5aec1c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec073bd0 2024-03-11T21:29:01.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5aec073bd0 0x7f5aec1c2000 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec1c4e60 con 0x7f5aec073bd0 2024-03-11T21:29:01.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:29:01.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 9 = system:0 2024-03-11T21:29:01.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 4 2024-03-11T21:29:01.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.529+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 4 9 2024-03-11T21:29:01.522 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:01.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 10 2024-03-11T21:29:01.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 10 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec111c70 osd=4 3 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 4 10 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 4 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 10 client.25289.objecter _send_command 10 2024-03-11T21:29:01.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.545+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec1c2540 con 0x7f5aec073bd0 2024-03-11T21:29:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec1c2540 con 0x7f5aec073bd0 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 4 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 4 10 2024-03-11T21:29:01.525 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 11 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5aec1c2bc0 0x7f5aec1c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 11 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 5 11 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 4 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 10 client.25289.objecter _send_command 11 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec1c7fd0 con 0x7f5aec1c2bc0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5aec1c2bc0 0x7f5aec1c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5ae9d9b700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5aec1c2bc0 0x7f5aec1c4fb0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec1c2bc0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec1c7fd0 con 0x7f5aec1c2bc0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 11 = system:0 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 4 2024-03-11T21:29:01.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.549+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 5 11 2024-03-11T21:29:01.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 12 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 12 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1c2540 osd=5 3 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 5 12 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 4 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 10 client.25289.objecter _send_command 12 2024-03-11T21:29:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec1c6d60 con 0x7f5aec1c2bc0 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec1c6d60 con 0x7f5aec1c2bc0 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 4 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 5 12 2024-03-11T21:29:01.566 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 13 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5aec1c7480 0x7f5aec1c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 13 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 6 13 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 4 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 10 client.25289.objecter _send_command 13 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec1cb3e0 con 0x7f5aec1c7480 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5aec1c7480 0x7f5aec1c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5aead9d700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5aec1c7480 0x7f5aec1c78e0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.589+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec1c7480 2024-03-11T21:29:01.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.593+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec1cb3e0 con 0x7f5aec1c7480 2024-03-11T21:29:01.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.593+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:29:01.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.593+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 13 = system:0 2024-03-11T21:29:01.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.593+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 4 2024-03-11T21:29:01.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.593+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 6 13 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 14 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 14 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1c6d60 osd=6 3 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 6 14 2024-03-11T21:29:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 4 2024-03-11T21:29:01.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 10 client.25289.objecter _send_command 14 2024-03-11T21:29:01.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.609+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec1ca220 con 0x7f5aec1c7480 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec1ca220 con 0x7f5aec1c7480 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 4 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 6 14 2024-03-11T21:29:01.589 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 15 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5aec1ca990 0x7f5aec1cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 15 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 7 15 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 4 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 10 client.25289.objecter _send_command 15 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5aec1ce780 con 0x7f5aec1ca990 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5aec1ca990 0x7f5aec1cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5aea59c700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5aec1ca990 0x7f5aec1cadd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:01.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 10 client.25289.objecter ms_handle_connect 0x7f5aec1ca990 2024-03-11T21:29:01.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5aec1ce780 con 0x7f5aec1ca990 2024-03-11T21:29:01.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:29:01.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 15 = system:0 2024-03-11T21:29:01.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 4 2024-03-11T21:29:01.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.613+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 7 15 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign -1 16 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter _get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_remove -1 16 2024-03-11T21:29:01.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter get_session s=0x7f5aec1ca220 osd=7 3 2024-03-11T21:29:01.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 15 client.25289.objecter _session_command_op_assign 7 16 2024-03-11T21:29:01.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 4 2024-03-11T21:29:01.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter _send_command 16 2024-03-11T21:29:01.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5aec1c5590 con 0x7f5aec1ca990 2024-03-11T21:29:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5adb7fe700 1 -- 172.21.15.143:0/989085384 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5aec1c5590 con 0x7f5aec1ca990 2024-03-11T21:29:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5adb7fe700 10 client.25289.objecter ms_dispatch 0x7f5aec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5adb7fe700 10 client.25289.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5adb7fe700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 4 2024-03-11T21:29:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5adb7fe700 15 client.25289.objecter _session_command_op_remove 7 16 2024-03-11T21:29:01.611 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:29:01.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 librados: watch_flush enter 2024-03-11T21:29:01.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 librados: watch_flush exit 2024-03-11T21:29:01.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.0 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5aec0629f0 msgr2=0x7f5aec062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5aec0629f0 0x7f5aec062dd0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec04faa0 osd=0 1 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.1 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f5aec108790 msgr2=0x7f5aec1114e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f5aec108790 0x7f5aec1114e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02d830 osd=1 1 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.2 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5aec107e70 msgr2=0x7f5aec02ce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5aec107e70 0x7f5aec02ce80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec107d00 osd=2 1 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.3 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5aec112700 msgr2=0x7f5aec0729b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5aec112700 0x7f5aec0729b0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec02cb50 osd=3 1 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.4 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5aec073bd0 msgr2=0x7f5aec1c2000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5aec073bd0 0x7f5aec1c2000 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec111c70 osd=4 1 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.5 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5aec1c2bc0 msgr2=0x7f5aec1c4fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5aec1c2bc0 0x7f5aec1c4fb0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c2540 osd=5 1 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.6 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5aec1c7480 msgr2=0x7f5aec1c78e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5aec1c7480 0x7f5aec1c78e0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1c6d60 osd=6 1 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter close_session for osd.7 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5aec1ca990 msgr2=0x7f5aec1cadd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5aec1ca990 0x7f5aec1cadd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter put_session s=0x7f5aec1ca220 osd=7 1 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 20 client.25289.objecter shutdown clearing up homeless session... 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 10 client.25289.objecter successfully canceled tick 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 reap_dead start 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5ad406f820 msgr2=0x7f5ad4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5ad406f820 0x7f5ad4071ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009680 tx=0x7f5adc009f50 comp rx=0 tx=0).stop 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 reap_dead start 2024-03-11T21:29:01.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 msgr2=0x7f5aec19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 --2- 172.21.15.143:0/989085384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec106dc0 0x7f5aec19c530 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007ee0 tx=0x7f5ae400d3b0 comp rx=0 tx=0).stop 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 reap_dead start 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 reap_dead start 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5aead9d700 1 -- 172.21.15.143:0/989085384 reap_dead start 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 shutdown_connections 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 >> 172.21.15.143:0/989085384 conn(0x7f5aec076d80 msgr2=0x7f5aec0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.633+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 shutdown_connections 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.637+0000 7f5af0ba5700 1 -- 172.21.15.143:0/989085384 wait complete. 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:01.637+0000 7f5af0ba5700 1 librados: shutdown 2024-03-11T21:29:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:01.616 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:01.622 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.186602+0000 mgr.y (mgr.14550) 231 : cluster 0 pgmap v202: 1012 pgs: 167 creating+peering, 624 active+clean, 221 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 57 KiB/s wr, 359 op/s 2024-03-11T21:29:01.622 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[29284]: audit 2024-03-11T21:28:58.319502+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:01.622 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[29284]: audit 2024-03-11T21:28:58.749348+0000 mon.a (mon.0) 1004 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.622 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[29284]: audit 2024-03-11T21:28:58.753460+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[29284]: audit 2024-03-11T21:28:58.753709+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.186602+0000 mgr.y (mgr.14550) 231 : cluster 0 pgmap v202: 1012 pgs: 167 creating+peering, 624 active+clean, 221 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 57 KiB/s wr, 359 op/s 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[21512]: audit 2024-03-11T21:28:58.319502+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[21512]: audit 2024-03-11T21:28:58.749348+0000 mon.a (mon.0) 1004 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[21512]: audit 2024-03-11T21:28:58.753460+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.623 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:01 smithi143 bash[21512]: audit 2024-03-11T21:28:58.753709+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:01.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:01.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:01 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.186602+0000 mgr.y (mgr.14550) 231 : cluster 0 pgmap v202: 1012 pgs: 167 creating+peering, 624 active+clean, 221 unknown; 599 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 57 KiB/s wr, 359 op/s 2024-03-11T21:29:01.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:01 smithi203 bash[23654]: audit 2024-03-11T21:28:58.319502+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:01.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:01 smithi203 bash[23654]: audit 2024-03-11T21:28:58.749348+0000 mon.a (mon.0) 1004 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:01 smithi203 bash[23654]: audit 2024-03-11T21:28:58.753460+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:01.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:01 smithi203 bash[23654]: audit 2024-03-11T21:28:58.753709+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:01.647 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:01.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:01.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:01.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:01.897 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:02.000 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:02.024 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:02.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.153 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:02.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:02.284 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:02.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:02.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:02.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:02.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:02.667 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:02.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:02.790 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:02.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:03.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.043 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:03.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.552048+0000 osd.0 (osd.0) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:03.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.552122+0000 osd.0 (osd.0) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:58.329857+0000 mgr.y (mgr.14550) 232 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:58.755973+0000 mon.a (mon.0) 1007 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:58.756044+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:58.767570+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:58.820596+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:58.949250+0000 osd.2 (osd.2) 105 : cluster 0 5.8 scrub starts 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:58.950287+0000 osd.2 (osd.2) 106 : cluster 0 5.8 scrub ok 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:59.149521+0000 mon.c (mon.2) 152 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.187492+0000 osd.1 (osd.1) 125 : cluster 0 5.12 scrub starts 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.188237+0000 mgr.y (mgr.14550) 233 : cluster 0 pgmap v205: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 111 op/s 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.189242+0000 osd.1 (osd.1) 126 : cluster 0 5.12 scrub ok 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:28:59.243144+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.345197+0000 osd.3 (osd.3) 153 : cluster 0 2.1b scrub starts 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.346495+0000 osd.3 (osd.3) 154 : cluster 0 2.1b scrub ok 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:00.150642+0000 mon.c (mon.2) 154 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.195292+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.197161+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.151790+0000 mon.c (mon.2) 155 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.339031+0000 mon.a (mon.0) 1009 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:29:03.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.349638+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:03.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.552048+0000 osd.0 (osd.0) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.552122+0000 osd.0 (osd.0) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:58.329857+0000 mgr.y (mgr.14550) 232 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:58.755973+0000 mon.a (mon.0) 1007 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:58.756044+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:58.767570+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:58.820596+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:03.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:58.949250+0000 osd.2 (osd.2) 105 : cluster 0 5.8 scrub starts 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:58.950287+0000 osd.2 (osd.2) 106 : cluster 0 5.8 scrub ok 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:59.149521+0000 mon.c (mon.2) 152 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.187492+0000 osd.1 (osd.1) 125 : cluster 0 5.12 scrub starts 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.188237+0000 mgr.y (mgr.14550) 233 : cluster 0 pgmap v205: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 111 op/s 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.189242+0000 osd.1 (osd.1) 126 : cluster 0 5.12 scrub ok 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:28:59.243144+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.345197+0000 osd.3 (osd.3) 153 : cluster 0 2.1b scrub starts 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.346495+0000 osd.3 (osd.3) 154 : cluster 0 2.1b scrub ok 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:00.150642+0000 mon.c (mon.2) 154 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.195292+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.197161+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.151790+0000 mon.c (mon.2) 155 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.169 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.339031+0000 mon.a (mon.0) 1009 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.349638+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.353490+0000 mon.c (mon.2) 156 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.354249+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.354344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.354427+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.354541+0000 mon.a (mon.0) 1014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.362604+0000 mon.a (mon.0) 1015 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.368657+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:01.374726+0000 mon.c (mon.2) 158 : audit 0 from='client.? 172.21.15.143:0/989085384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:02.153471+0000 mon.c (mon.2) 159 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:02.263862+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[29284]: audit 2024-03-11T21:29:02.267091+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.353490+0000 mon.c (mon.2) 156 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:03.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.354249+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.354344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.354427+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.354541+0000 mon.a (mon.0) 1014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.362604+0000 mon.a (mon.0) 1015 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.368657+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:01.374726+0000 mon.c (mon.2) 158 : audit 0 from='client.? 172.21.15.143:0/989085384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:02.153471+0000 mon.c (mon.2) 159 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:02.263862+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:03 smithi143 bash[21512]: audit 2024-03-11T21:29:02.267091+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.173 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:03.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.297 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:03.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:03.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.552048+0000 osd.0 (osd.0) 139 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.552122+0000 osd.0 (osd.0) 140 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:58.329857+0000 mgr.y (mgr.14550) 232 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:58.755973+0000 mon.a (mon.0) 1007 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-71868-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:58.756044+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi143-72277-1"}]': finished 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:58.767570+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:58.820596+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:58.949250+0000 osd.2 (osd.2) 105 : cluster 0 5.8 scrub starts 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:58.950287+0000 osd.2 (osd.2) 106 : cluster 0 5.8 scrub ok 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:59.149521+0000 mon.c (mon.2) 152 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.187492+0000 osd.1 (osd.1) 125 : cluster 0 5.12 scrub starts 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.188237+0000 mgr.y (mgr.14550) 233 : cluster 0 pgmap v205: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 111 op/s 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.189242+0000 osd.1 (osd.1) 126 : cluster 0 5.12 scrub ok 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:28:59.243144+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.345197+0000 osd.3 (osd.3) 153 : cluster 0 2.1b scrub starts 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.346495+0000 osd.3 (osd.3) 154 : cluster 0 2.1b scrub ok 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:00.150642+0000 mon.c (mon.2) 154 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.195292+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.197161+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.151790+0000 mon.c (mon.2) 155 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.339031+0000 mon.a (mon.0) 1009 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.349638+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.353490+0000 mon.c (mon.2) 156 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.354249+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-71620-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:03.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.354344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.143:0/2906136689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.354427+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.354541+0000 mon.a (mon.0) 1014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.362604+0000 mon.a (mon.0) 1015 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.368657+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:01.374726+0000 mon.c (mon.2) 158 : audit 0 from='client.? 172.21.15.143:0/989085384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:02.153471+0000 mon.c (mon.2) 159 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:02.263862+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:03.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:03 smithi203 bash[23654]: audit 2024-03-11T21:29:02.267091+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:03.424 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:03.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:03.557 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:03.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:03.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:03.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:03.818 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:03.921 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:03.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:04.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:04.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:04.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:04.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:04.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:04.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.458 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:04.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:04.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:04.653 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:04.654 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:04.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:04.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:04.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.846 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:04.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:28:58.517272+0000 osd.0 (osd.0) 141 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:29:04.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:28:58.518627+0000 osd.0 (osd.0) 142 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:29:04.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.534553+0000 osd.0 (osd.0) 143 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:29:04.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.537624+0000 osd.0 (osd.0) 144 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.165986+0000 osd.1 (osd.1) 129 : cluster 0 3.4 scrub starts 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.171858+0000 osd.1 (osd.1) 130 : cluster 0 3.4 scrub ok 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.189606+0000 mgr.y (mgr.14550) 234 : cluster 0 pgmap v206: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 25 MiB/s wr, 82 op/s 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.852597+0000 osd.2 (osd.2) 107 : cluster 0 2.e scrub starts 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:01.853850+0000 osd.2 (osd.2) 108 : cluster 0 2.e scrub ok 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:02.197540+0000 osd.1 (osd.1) 131 : cluster 0 3.0 deep-scrub starts 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:02.205084+0000 osd.1 (osd.1) 132 : cluster 0 3.0 deep-scrub ok 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:02.688118+0000 mon.a (mon.0) 1018 : cluster 3 Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: audit 2024-03-11T21:29:02.692920+0000 mon.a (mon.0) 1019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: audit 2024-03-11T21:29:02.704771+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: cluster 2024-03-11T21:29:03.140887+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: audit 2024-03-11T21:29:03.160303+0000 mon.c (mon.2) 161 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[21512]: audit 2024-03-11T21:29:03.596244+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:28:58.517272+0000 osd.0 (osd.0) 141 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:28:58.518627+0000 osd.0 (osd.0) 142 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.534553+0000 osd.0 (osd.0) 143 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.537624+0000 osd.0 (osd.0) 144 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.165986+0000 osd.1 (osd.1) 129 : cluster 0 3.4 scrub starts 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.171858+0000 osd.1 (osd.1) 130 : cluster 0 3.4 scrub ok 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.189606+0000 mgr.y (mgr.14550) 234 : cluster 0 pgmap v206: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 25 MiB/s wr, 82 op/s 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.852597+0000 osd.2 (osd.2) 107 : cluster 0 2.e scrub starts 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:01.853850+0000 osd.2 (osd.2) 108 : cluster 0 2.e scrub ok 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:02.197540+0000 osd.1 (osd.1) 131 : cluster 0 3.0 deep-scrub starts 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:02.205084+0000 osd.1 (osd.1) 132 : cluster 0 3.0 deep-scrub ok 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:02.688118+0000 mon.a (mon.0) 1018 : cluster 3 Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: audit 2024-03-11T21:29:02.692920+0000 mon.a (mon.0) 1019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: audit 2024-03-11T21:29:02.704771+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: cluster 2024-03-11T21:29:03.140887+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:29:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: audit 2024-03-11T21:29:03.160303+0000 mon.c (mon.2) 161 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:04 smithi143 bash[29284]: audit 2024-03-11T21:29:03.596244+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:04.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:04.972 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:05.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.099 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:05.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:29:05.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.240 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:05.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:28:58.517272+0000 osd.0 (osd.0) 141 : cluster 0 6.9 deep-scrub starts 2024-03-11T21:29:05.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:28:58.518627+0000 osd.0 (osd.0) 142 : cluster 0 6.9 deep-scrub ok 2024-03-11T21:29:05.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.534553+0000 osd.0 (osd.0) 143 : cluster 0 3.12 deep-scrub starts 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.537624+0000 osd.0 (osd.0) 144 : cluster 0 3.12 deep-scrub ok 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.165986+0000 osd.1 (osd.1) 129 : cluster 0 3.4 scrub starts 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.171858+0000 osd.1 (osd.1) 130 : cluster 0 3.4 scrub ok 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.189606+0000 mgr.y (mgr.14550) 234 : cluster 0 pgmap v206: 860 pgs: 223 creating+peering, 523 active+clean, 114 unknown; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 25 MiB/s wr, 82 op/s 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.852597+0000 osd.2 (osd.2) 107 : cluster 0 2.e scrub starts 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:01.853850+0000 osd.2 (osd.2) 108 : cluster 0 2.e scrub ok 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:02.197540+0000 osd.1 (osd.1) 131 : cluster 0 3.0 deep-scrub starts 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:02.205084+0000 osd.1 (osd.1) 132 : cluster 0 3.0 deep-scrub ok 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:02.688118+0000 mon.a (mon.0) 1018 : cluster 3 Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: audit 2024-03-11T21:29:02.692920+0000 mon.a (mon.0) 1019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-71604-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: audit 2024-03-11T21:29:02.704771+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.143:0/1005253754' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: cluster 2024-03-11T21:29:03.140887+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: audit 2024-03-11T21:29:03.160303+0000 mon.c (mon.2) 161 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:05.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:04 smithi203 bash[23654]: audit 2024-03-11T21:29:03.596244+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T21:29:05.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:05.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:05.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:05.607 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.627 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:05.730 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.754 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:05.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.882 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:05.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:06.014 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:06.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:06.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:52.068023+0000 osd.4 (osd.4) 127 : cluster 0 6.e scrub starts 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:52.069090+0000 osd.4 (osd.4) 128 : cluster 0 6.e scrub ok 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:52.553247+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:52.553311+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.052398+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:28:53.054162+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:03.192919+0000 mgr.y (mgr.14550) 235 : cluster 0 pgmap v209: 996 pgs: 365 creating+peering, 591 active+clean, 40 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 26 MiB/s wr, 113 op/s 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:04.012413+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:04.012517+0000 mon.a (mon.0) 1023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:04.021098+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:04.161570+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:04.245048+0000 osd.1 (osd.1) 133 : cluster 0 2.d deep-scrub starts 2024-03-11T21:29:06.129 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:04.246467+0000 osd.1 (osd.1) 134 : cluster 0 2.d deep-scrub ok 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:04.368097+0000 osd.3 (osd.3) 155 : cluster 0 3.b scrub starts 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:04.371296+0000 osd.3 (osd.3) 156 : cluster 0 3.b scrub ok 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:04.451646+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:04.879643+0000 mon.a (mon.0) 1025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.162684+0000 mon.c (mon.2) 164 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.314402+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: cluster 2024-03-11T21:29:05.342810+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.359109+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.143:0/3215347298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.363683+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.364329+0000 mon.c (mon.2) 166 : audit 1 from='client.? 172.21.15.143:0/94710998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[29284]: audit 2024-03-11T21:29:05.370977+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:52.068023+0000 osd.4 (osd.4) 127 : cluster 0 6.e scrub starts 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:52.069090+0000 osd.4 (osd.4) 128 : cluster 0 6.e scrub ok 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:52.553247+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:52.553311+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.052398+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:28:53.054162+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:03.192919+0000 mgr.y (mgr.14550) 235 : cluster 0 pgmap v209: 996 pgs: 365 creating+peering, 591 active+clean, 40 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 26 MiB/s wr, 113 op/s 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:04.012413+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:04.012517+0000 mon.a (mon.0) 1023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:04.021098+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:04.161570+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:04.245048+0000 osd.1 (osd.1) 133 : cluster 0 2.d deep-scrub starts 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:04.246467+0000 osd.1 (osd.1) 134 : cluster 0 2.d deep-scrub ok 2024-03-11T21:29:06.131 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:04.368097+0000 osd.3 (osd.3) 155 : cluster 0 3.b scrub starts 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:04.371296+0000 osd.3 (osd.3) 156 : cluster 0 3.b scrub ok 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:04.451646+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:04.879643+0000 mon.a (mon.0) 1025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.162684+0000 mon.c (mon.2) 164 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.314402+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: cluster 2024-03-11T21:29:05.342810+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.359109+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.143:0/3215347298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.363683+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.364329+0000 mon.c (mon.2) 166 : audit 1 from='client.? 172.21.15.143:0/94710998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:05 smithi143 bash[21512]: audit 2024-03-11T21:29:05.370977+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:06.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:52.068023+0000 osd.4 (osd.4) 127 : cluster 0 6.e scrub starts 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:52.069090+0000 osd.4 (osd.4) 128 : cluster 0 6.e scrub ok 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:52.553247+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:52.553311+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.052398+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:28:53.054162+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:03.192919+0000 mgr.y (mgr.14550) 235 : cluster 0 pgmap v209: 996 pgs: 365 creating+peering, 591 active+clean, 40 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 26 MiB/s wr, 113 op/s 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:04.012413+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-71853-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:04.012517+0000 mon.a (mon.0) 1023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-71604-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:04.021098+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:04.161570+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:04.245048+0000 osd.1 (osd.1) 133 : cluster 0 2.d deep-scrub starts 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:04.246467+0000 osd.1 (osd.1) 134 : cluster 0 2.d deep-scrub ok 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:04.368097+0000 osd.3 (osd.3) 155 : cluster 0 3.b scrub starts 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:04.371296+0000 osd.3 (osd.3) 156 : cluster 0 3.b scrub ok 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:04.451646+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:04.879643+0000 mon.a (mon.0) 1025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.162684+0000 mon.c (mon.2) 164 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.314402+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: cluster 2024-03-11T21:29:05.342810+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.359109+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.143:0/3215347298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.363683+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.364329+0000 mon.c (mon.2) 166 : audit 1 from='client.? 172.21.15.143:0/94710998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:05 smithi203 bash[23654]: audit 2024-03-11T21:29:05.370977+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:06.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:06.447 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:06.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:06.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: seed 71620 2024-03-11T21:29:06.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (108 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (27 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-03-11T21:29:06.607 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-11T21:29:06.608 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (9 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (3 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-11T21:29:06.609 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (205 ms total) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (5162 ms) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (65 ms) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (12 ms) 2024-03-11T21:29:06.610 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-11T21:29:06.611 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-03-11T21:29:06.611 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-11T21:29:06.611 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-11T21:29:06.679 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:06.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:06.707 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:06.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:06.838 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:06.849 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-11T21:29:06.849 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5775 ms) 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (26 ms) 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20048 ms) 2024-03-11T21:29:06.850 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: attempt 0 of 20 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5249 ms) 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31098 ms total) 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31098 ms total) 2024-03-11T21:29:06.851 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-11T21:29:06.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:07.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:07.095 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:07.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.222 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:07.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:07.345 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:07.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.472 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:07.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:07.602 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:07.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:07.752 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:07.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.067924+0000 osd.4 (osd.4) 131 : cluster 0 5.e scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.069175+0000 osd.4 (osd.4) 132 : cluster 0 5.e scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.860507+0000 osd.5 (osd.5) 139 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:54.884563+0000 osd.5 (osd.5) 140 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.107164+0000 osd.4 (osd.4) 133 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.108708+0000 osd.4 (osd.4) 134 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.379571+0000 osd.7 (osd.7) 167 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.425471+0000 osd.7 (osd.7) 168 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.868521+0000 osd.5 (osd.5) 141 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:55.869886+0000 osd.5 (osd.5) 142 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.110453+0000 osd.4 (osd.4) 135 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.133987+0000 osd.4 (osd.4) 136 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:07.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.367059+0000 osd.7 (osd.7) 169 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:56.369326+0000 osd.7 (osd.7) 170 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.074453+0000 osd.4 (osd.4) 137 : cluster 0 5.1c scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:57.075779+0000 osd.4 (osd.4) 138 : cluster 0 5.1c scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.323171+0000 osd.7 (osd.7) 171 : cluster 0 3.e deep-scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.327641+0000 osd.7 (osd.7) 172 : cluster 0 3.e deep-scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.696911+0000 osd.6 (osd.6) 95 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:28:59.701513+0000 osd.6 (osd.6) 96 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.659690+0000 osd.6 (osd.6) 97 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:00.663768+0000 osd.6 (osd.6) 98 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:02.324369+0000 osd.7 (osd.7) 173 : cluster 0 2.5 scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:02.325559+0000 osd.7 (osd.7) 174 : cluster 0 2.5 scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:05.193683+0000 mgr.y (mgr.14550) 236 : cluster 0 pgmap v211: 1036 pgs: 333 creating+peering, 591 active+clean, 112 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:05.300759+0000 osd.7 (osd.7) 175 : cluster 0 2.8 scrub starts 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:05.302608+0000 osd.7 (osd.7) 176 : cluster 0 2.8 scrub ok 2024-03-11T21:29:07.871 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.926927+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.067924+0000 osd.4 (osd.4) 131 : cluster 0 5.e scrub starts 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.069175+0000 osd.4 (osd.4) 132 : cluster 0 5.e scrub ok 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.860507+0000 osd.5 (osd.5) 139 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:54.884563+0000 osd.5 (osd.5) 140 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.107164+0000 osd.4 (osd.4) 133 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.108708+0000 osd.4 (osd.4) 134 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:07.872 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.379571+0000 osd.7 (osd.7) 167 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.425471+0000 osd.7 (osd.7) 168 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.868521+0000 osd.5 (osd.5) 141 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:55.869886+0000 osd.5 (osd.5) 142 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.110453+0000 osd.4 (osd.4) 135 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.133987+0000 osd.4 (osd.4) 136 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.367059+0000 osd.7 (osd.7) 169 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:56.369326+0000 osd.7 (osd.7) 170 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.074453+0000 osd.4 (osd.4) 137 : cluster 0 5.1c scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:57.075779+0000 osd.4 (osd.4) 138 : cluster 0 5.1c scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.323171+0000 osd.7 (osd.7) 171 : cluster 0 3.e deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.327641+0000 osd.7 (osd.7) 172 : cluster 0 3.e deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.696911+0000 osd.6 (osd.6) 95 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:28:59.701513+0000 osd.6 (osd.6) 96 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:07.873 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.659690+0000 osd.6 (osd.6) 97 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:00.663768+0000 osd.6 (osd.6) 98 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:02.324369+0000 osd.7 (osd.7) 173 : cluster 0 2.5 scrub starts 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:02.325559+0000 osd.7 (osd.7) 174 : cluster 0 2.5 scrub ok 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:05.193683+0000 mgr.y (mgr.14550) 236 : cluster 0 pgmap v211: 1036 pgs: 333 creating+peering, 591 active+clean, 112 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:05.300759+0000 osd.7 (osd.7) 175 : cluster 0 2.8 scrub starts 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:05.302608+0000 osd.7 (osd.7) 176 : cluster 0 2.8 scrub ok 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.926927+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.935065+0000 mon.a (mon.0) 1031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:07.874 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.936537+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.951965+0000 mon.b (mon.1) 43 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.953243+0000 mon.b (mon.1) 44 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.956483+0000 mon.a (mon.0) 1033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:05.957675+0000 mon.a (mon.0) 1034 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.163784+0000 mon.c (mon.2) 167 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.320742+0000 mon.a (mon.0) 1035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.320815+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.320860+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.320983+0000 mon.a (mon.0) 1038 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.321097+0000 mon.a (mon.0) 1039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: cluster 2024-03-11T21:29:06.326110+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.330800+0000 mon.b (mon.1) 45 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:07.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.343874+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.427296+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.569526+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.576714+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.855677+0000 mon.a (mon.0) 1041 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.857719+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[29284]: audit 2024-03-11T21:29:06.858685+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:07.876 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.935065+0000 mon.a (mon.0) 1031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.936537+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.951965+0000 mon.b (mon.1) 43 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.953243+0000 mon.b (mon.1) 44 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.956483+0000 mon.a (mon.0) 1033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:05.957675+0000 mon.a (mon.0) 1034 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.163784+0000 mon.c (mon.2) 167 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.320742+0000 mon.a (mon.0) 1035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.320815+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.320860+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.320983+0000 mon.a (mon.0) 1038 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.321097+0000 mon.a (mon.0) 1039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: cluster 2024-03-11T21:29:06.326110+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.330800+0000 mon.b (mon.1) 45 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.343874+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.877 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.427296+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:07.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.569526+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:29:07.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.576714+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:07.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.855677+0000 mon.a (mon.0) 1041 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:07.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.857719+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:07.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:07 smithi143 bash[21512]: audit 2024-03-11T21:29:06.858685+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:07.878 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:07.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.002 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.067924+0000 osd.4 (osd.4) 131 : cluster 0 5.e scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.069175+0000 osd.4 (osd.4) 132 : cluster 0 5.e scrub ok 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.860507+0000 osd.5 (osd.5) 139 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:54.884563+0000 osd.5 (osd.5) 140 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.107164+0000 osd.4 (osd.4) 133 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.108708+0000 osd.4 (osd.4) 134 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.379571+0000 osd.7 (osd.7) 167 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.425471+0000 osd.7 (osd.7) 168 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.868521+0000 osd.5 (osd.5) 141 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:55.869886+0000 osd.5 (osd.5) 142 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.110453+0000 osd.4 (osd.4) 135 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:08.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.133987+0000 osd.4 (osd.4) 136 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.367059+0000 osd.7 (osd.7) 169 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:56.369326+0000 osd.7 (osd.7) 170 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.074453+0000 osd.4 (osd.4) 137 : cluster 0 5.1c scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:57.075779+0000 osd.4 (osd.4) 138 : cluster 0 5.1c scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.323171+0000 osd.7 (osd.7) 171 : cluster 0 3.e deep-scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.327641+0000 osd.7 (osd.7) 172 : cluster 0 3.e deep-scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.696911+0000 osd.6 (osd.6) 95 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:28:59.701513+0000 osd.6 (osd.6) 96 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.659690+0000 osd.6 (osd.6) 97 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:00.663768+0000 osd.6 (osd.6) 98 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:02.324369+0000 osd.7 (osd.7) 173 : cluster 0 2.5 scrub starts 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:02.325559+0000 osd.7 (osd.7) 174 : cluster 0 2.5 scrub ok 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:05.193683+0000 mgr.y (mgr.14550) 236 : cluster 0 pgmap v211: 1036 pgs: 333 creating+peering, 591 active+clean, 112 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:29:08.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:05.300759+0000 osd.7 (osd.7) 175 : cluster 0 2.8 scrub starts 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:05.302608+0000 osd.7 (osd.7) 176 : cluster 0 2.8 scrub ok 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.926927+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.935065+0000 mon.a (mon.0) 1031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.936537+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.951965+0000 mon.b (mon.1) 43 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:08.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.953243+0000 mon.b (mon.1) 44 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.956483+0000 mon.a (mon.0) 1033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:05.957675+0000 mon.a (mon.0) 1034 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.163784+0000 mon.c (mon.2) 167 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.320742+0000 mon.a (mon.0) 1035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.320815+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-71594-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.320860+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.320983+0000 mon.a (mon.0) 1038 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi143-72277-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.321097+0000 mon.a (mon.0) 1039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-71612-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: cluster 2024-03-11T21:29:06.326110+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.330800+0000 mon.b (mon.1) 45 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.343874+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.427296+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.569526+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:29:08.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.576714+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.855677+0000 mon.a (mon.0) 1041 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.857719+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:08.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:07 smithi203 bash[23654]: audit 2024-03-11T21:29:06.858685+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:08.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:08.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:08.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:08.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:08.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:08.384 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:08.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.509 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:08.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:08.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:08.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:08.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:08.762 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:08.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:08.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:08.996 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:09.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-11T21:29:09.051 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (812 ms) 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (33 ms) 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (32480 ms) 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (33325 ms total) 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (33325 ms total) 2024-03-11T21:29:09.052 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-11T21:29:09.062 INFO:tasks.workunit.client.0.smithi143.stdout: cls: Running main() from gmock_main.cc 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [----------] Global test environment set-up. 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [ OK ] NeoRadosCls.DNE (22058 ms) 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (11190 ms) 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [----------] 2 tests from NeoRadosCls (33248 ms total) 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [----------] Global test environment tear-down 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [==========] 2 tests from 1 test suite ran. (33248 ms total) 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: cls: [ PASSED ] 2 tests. 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-11T21:29:09.063 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: handle_notify cookie 94257978493856 notify_id 622770257920 notifier_gid 24887 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (21942 ms) 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Trying... 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: handle_notify cookie 94257991223184 notify_id 648540061701 notifier_gid 25238 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Waiting for 3.000000000s 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Timed out. 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Flushing... 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: Flushed... 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (11204 ms) 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (33146 ms total) 2024-03-11T21:29:09.064 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (33147 ms total) 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: Running main() from gmock_main.cc 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [----------] Global test environment set-up. 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (15317 ms) 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (4873 ms) 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (13042 ms) 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [----------] 3 tests from NeoRadosCmd (33232 ms total) 2024-03-11T21:29:09.065 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: 2024-03-11T21:29:09.066 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [----------] Global test environment tear-down 2024-03-11T21:29:09.066 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [==========] 3 tests from 1 test suite ran. (33232 ms total) 2024-03-11T21:29:09.066 INFO:tasks.workunit.client.0.smithi143.stdout: cmd: [ PASSED ] 3 tests. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: starting. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: creating pool ceph_test_rados_open_pools_parallel.smithi143-72073 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: created object 0... 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: created object 25... 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: created object 49... 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: finishing. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[72110]: shutting down. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[72112]: starting. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[72112]: rados_pool_create. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[72112]: rados_ioctx_create. 2024-03-11T21:29:09.070 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[72112]: shutting down. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: starting. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: creating pool ceph_test_rados_open_pools_parallel.smithi143-72073 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: created object 0... 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: created object 25... 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: created object 49... 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: finishing. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[75239]: shutting down. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[75240]: starting. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[75240]: rados_pool_create. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[75240]: rados_ioctx_create. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[75240]: shutting down. 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-03-11T21:29:09.071 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-03-11T21:29:09.072 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-11T21:29:09.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:09.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:09.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:06.858314+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:09.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:06.859836+0000 mon.c (mon.2) 171 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:06.862709+0000 mon.c (mon.2) 172 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:06.873495+0000 mon.c (mon.2) 173 : audit 0 from='client.? 172.21.15.143:0/308768684' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:06.873673+0000 mgr.y (mgr.14550) 237 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:07.164950+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: cluster 2024-03-11T21:29:07.195071+0000 mgr.y (mgr.14550) 238 : cluster 0 pgmap v214: 940 pgs: 252 creating+peering, 480 active+clean, 208 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: cluster 2024-03-11T21:29:07.718067+0000 mon.a (mon.0) 1044 : cluster 3 Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.166529+0000 mon.c (mon.2) 175 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.603881+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: cluster 2024-03-11T21:29:08.606519+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.614458+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.617997+0000 mon.a (mon.0) 1047 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.624635+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.625935+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.631291+0000 mon.c (mon.2) 176 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:09.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:09 smithi203 bash[23654]: audit 2024-03-11T21:29:08.633703+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:06.858314+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:06.859836+0000 mon.c (mon.2) 171 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:06.862709+0000 mon.c (mon.2) 172 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:06.873495+0000 mon.c (mon.2) 173 : audit 0 from='client.? 172.21.15.143:0/308768684' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:06.873673+0000 mgr.y (mgr.14550) 237 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:07.164950+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: cluster 2024-03-11T21:29:07.195071+0000 mgr.y (mgr.14550) 238 : cluster 0 pgmap v214: 940 pgs: 252 creating+peering, 480 active+clean, 208 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: cluster 2024-03-11T21:29:07.718067+0000 mon.a (mon.0) 1044 : cluster 3 Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.166529+0000 mon.c (mon.2) 175 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.603881+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: cluster 2024-03-11T21:29:08.606519+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.614458+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.617997+0000 mon.a (mon.0) 1047 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.624635+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.625935+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.631291+0000 mon.c (mon.2) 176 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:09.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[29284]: audit 2024-03-11T21:29:08.633703+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:06.858314+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:06.859836+0000 mon.c (mon.2) 171 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:06.862709+0000 mon.c (mon.2) 172 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:06.873495+0000 mon.c (mon.2) 173 : audit 0 from='client.? 172.21.15.143:0/308768684' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:06.873673+0000 mgr.y (mgr.14550) 237 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:07.164950+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: cluster 2024-03-11T21:29:07.195071+0000 mgr.y (mgr.14550) 238 : cluster 0 pgmap v214: 940 pgs: 252 creating+peering, 480 active+clean, 208 unknown; 145 MiB data, 727 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: cluster 2024-03-11T21:29:07.718067+0000 mon.a (mon.0) 1044 : cluster 3 Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.166529+0000 mon.c (mon.2) 175 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.603881+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: cluster 2024-03-11T21:29:08.606519+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.614458+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.617997+0000 mon.a (mon.0) 1047 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.624635+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.625935+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.631291+0000 mon.c (mon.2) 176 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:09.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:09 smithi143 bash[21512]: audit 2024-03-11T21:29:08.633703+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:09.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:09.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:09.673 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:09.693 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:09.731 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:09.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:09.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.942 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:10.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.071 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:10.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:10.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:10.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:10.333 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-11T21:29:10.381 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (857 ms) 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (5050 ms) 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (27458 ms) 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:06.858314+0000 mon.b [INF] from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:06.859836+0000 mon.c [INF] from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:06.862709+0000 mon.c [INF] from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:07.718067+0000 mon.a [WRN] Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.603881+0000 mon.b [INF] from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.614458+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.617997+0000 mon.a [INF] from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.624635+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:10.382 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.625935+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.631291+0000 mon.c [INF] from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:08.633703+0000 mon.c [INF] from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.058820+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.059196+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.062486+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.062566+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.062671+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.062758+0000 mon.a [INF] from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.074719+0000 mon.c [INF] from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.074986+0000 mon.c [INF] from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.075321+0000 mon.c [INF] from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.078688+0000 mon.c [INF] from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.079976+0000 mon.c [INF] from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2"}]: dispatch 2024-03-11T21:29:10.383 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.081134+0000 mon.c [INF] from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.091141+0000 mon.c [INF] from='client.? 172.21.15.143:0/2347823758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.547661+0000 mon.a [INF] from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.548349+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.549314+0000 client.admin [INF] onexx 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.549753+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.384 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.553432+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:10.447 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:08.337643+0000 mgr.y (mgr.14550) 239 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:10.447 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.058820+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.447 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.059196+0000 mon.a (mon.0) 1051 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.447 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.062486+0000 mon.a (mon.0) 1052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:10.447 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.062566+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.062671+0000 mon.a (mon.0) 1054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.062758+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.074719+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.074986+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.075321+0000 mon.c (mon.2) 180 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.078688+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.079976+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.081134+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.091141+0000 mon.c (mon.2) 184 : audit 1 from='client.? 172.21.15.143:0/2347823758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.167698+0000 mon.c (mon.2) 185 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: cluster 2024-03-11T21:29:09.545669+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.547661+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.548349+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: cluster 2024-03-11T21:29:09.549314+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-03-11T21:29:10.448 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.549753+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.553432+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.553997+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.554650+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:10 smithi203 bash[23654]: audit 2024-03-11T21:29:09.977755+0000 mon.a (mon.0) 1063 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T21:29:10.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:10.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:08.337643+0000 mgr.y (mgr.14550) 239 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.058820+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.059196+0000 mon.a (mon.0) 1051 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.062486+0000 mon.a (mon.0) 1052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.062566+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.062671+0000 mon.a (mon.0) 1054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.062758+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.074719+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.074986+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.075321+0000 mon.c (mon.2) 180 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.078688+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.079976+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.081134+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:08.337643+0000 mgr.y (mgr.14550) 239 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.058820+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.059196+0000 mon.a (mon.0) 1051 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.062486+0000 mon.a (mon.0) 1052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi143-72277-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.062566+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-71612-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:10.606 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.062671+0000 mon.a (mon.0) 1054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.062758+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.074719+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.074986+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.075321+0000 mon.c (mon.2) 180 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.078688+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.079976+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2"}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.081134+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.091141+0000 mon.c (mon.2) 184 : audit 1 from='client.? 172.21.15.143:0/2347823758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.167698+0000 mon.c (mon.2) 185 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: cluster 2024-03-11T21:29:09.545669+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:29:10.607 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.547661+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.548349+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: cluster 2024-03-11T21:29:09.549314+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.549753+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.553432+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.553997+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.554650+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[29284]: audit 2024-03-11T21:29:09.977755+0000 mon.a (mon.0) 1063 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.091141+0000 mon.c (mon.2) 184 : audit 1 from='client.? 172.21.15.143:0/2347823758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.167698+0000 mon.c (mon.2) 185 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: cluster 2024-03-11T21:29:09.545669+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.547661+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]: dispatch 2024-03-11T21:29:10.608 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.548349+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: cluster 2024-03-11T21:29:09.549314+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.549753+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.553432+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.553997+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.554650+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:10.609 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:10 smithi143 bash[21512]: audit 2024-03-11T21:29:09.977755+0000 mon.a (mon.0) 1063 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T21:29:10.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.706 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:10.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:10.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:10.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:11.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:11.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:11.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.218 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.553997+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": api_stat: Running main() from gmock_main.cc 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] Global test environment set-up. 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.Stat (7 ms) 2024-03-11T21:29:11.259 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6 ms) 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.StatNS (17 ms) 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (2 ms) 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 5 tests from LibRadosStat (36 ms total) 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-11T21:29:11.260 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (5231 ms) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (19 ms) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (250 ms) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (5 ms) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (5505 ms total) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] Global test environment tear-down 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (35564 ms total) 2024-03-11T21:29:11.261 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-11T21:29:11.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.349 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:11.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:11.476 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:11.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:11.601 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:11.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:11.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: seed 71853 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (6 ms) 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (6 ms) 2024-03-11T21:29:11.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (15 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (23 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (52 ms total) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (2560 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (12 ms) 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-11T21:29:11.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (29 ms) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2601 ms total) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (36077 ms total) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (7 ms) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (7 ms) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (10 ms) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (10 ms) 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-11T21:29:11.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (4 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (12 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (5 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (16 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (11 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (11 ms) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (5352 ms total) 2024-03-11T21:29:11.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: 2024-03-11T21:29:11.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-11T21:29:11.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (36192 ms total) 2024-03-11T21:29:11.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-11T21:29:11.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:11.858 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:11.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:11.985 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:12.006 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: cluster 2024-03-11T21:29:09.195741+0000 mgr.y (mgr.14550) 240 : cluster 0 pgmap v217: 636 pgs: 33 creating+peering, 524 active+clean, 79 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s wr, 4 op/s 2024-03-11T21:29:12.006 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: cluster 2024-03-11T21:29:09.485083+0000 osd.3 (osd.3) 157 : cluster 0 3.1e scrub starts 2024-03-11T21:29:12.006 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: cluster 2024-03-11T21:29:09.487583+0000 osd.3 (osd.3) 158 : cluster 0 3.1e scrub ok 2024-03-11T21:29:12.006 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.169561+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.006 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.399397+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.402968+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403059+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403126+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403199+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403276+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403349+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.403434+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.404643+0000 mon.a (mon.0) 1072 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.409144+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.007 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.409339+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: cluster 2024-03-11T21:29:09.195741+0000 mgr.y (mgr.14550) 240 : cluster 0 pgmap v217: 636 pgs: 33 creating+peering, 524 active+clean, 79 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s wr, 4 op/s 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: cluster 2024-03-11T21:29:09.485083+0000 osd.3 (osd.3) 157 : cluster 0 3.1e scrub starts 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: cluster 2024-03-11T21:29:09.487583+0000 osd.3 (osd.3) 158 : cluster 0 3.1e scrub ok 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.169561+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.399397+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.402968+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403059+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403126+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403199+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403276+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403349+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.403434+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:12.008 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.404643+0000 mon.a (mon.0) 1072 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.409144+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.409339+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.409497+0000 mon.c (mon.2) 189 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.411048+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.411171+0000 mon.c (mon.2) 191 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.412387+0000 mon.c (mon.2) 192 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:10.412799+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.170562+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.265817+0000 mon.a (mon.0) 1073 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.272605+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.273055+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.273379+0000 mon.a (mon.0) 1076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.273816+0000 mon.a (mon.0) 1077 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.274126+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.274526+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.275004+0000 mon.a (mon.0) 1080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.275420+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.290709+0000 mon.b (mon.1) 49 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[21512]: audit 2024-03-11T21:29:11.293298+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.409497+0000 mon.c (mon.2) 189 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.411048+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.411171+0000 mon.c (mon.2) 191 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.412387+0000 mon.c (mon.2) 192 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:10.412799+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.170562+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.265817+0000 mon.a (mon.0) 1073 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.272605+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.273055+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.273379+0000 mon.a (mon.0) 1076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.273816+0000 mon.a (mon.0) 1077 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.274126+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.274526+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.275004+0000 mon.a (mon.0) 1080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.275420+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.290709+0000 mon.b (mon.1) 49 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:11 smithi143 bash[29284]: audit 2024-03-11T21:29:11.293298+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: cluster 2024-03-11T21:29:09.195741+0000 mgr.y (mgr.14550) 240 : cluster 0 pgmap v217: 636 pgs: 33 creating+peering, 524 active+clean, 79 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s wr, 4 op/s 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: cluster 2024-03-11T21:29:09.485083+0000 osd.3 (osd.3) 157 : cluster 0 3.1e scrub starts 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: cluster 2024-03-11T21:29:09.487583+0000 osd.3 (osd.3) 158 : cluster 0 3.1e scrub ok 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.169561+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.399397+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.402968+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403059+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.143:0/962391474' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-71868-7"}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403126+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-71641-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403199+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403276+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403349+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.403434+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.404643+0000 mon.a (mon.0) 1072 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.409144+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.143:0/1519391561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.409339+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.409497+0000 mon.c (mon.2) 189 : audit 1 from='client.? 172.21.15.143:0/1168513910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.411048+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.411171+0000 mon.c (mon.2) 191 : audit 1 from='client.? 172.21.15.143:0/2094840568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.412387+0000 mon.c (mon.2) 192 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:10.412799+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.170562+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.265817+0000 mon.a (mon.0) 1073 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.272605+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.273055+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.273379+0000 mon.a (mon.0) 1076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.273816+0000 mon.a (mon.0) 1077 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:12.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.274126+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]: dispatch 2024-03-11T21:29:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.274526+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.275004+0000 mon.a (mon.0) 1080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.275420+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.290709+0000 mon.b (mon.1) 49 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:11 smithi203 bash[23654]: audit 2024-03-11T21:29:11.293298+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:12.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] Global test environment set-up. 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5074 ms) 2024-03-11T21:29:12.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.RegisterLate (37 ms) 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: cluster: 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: id: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: health: HEALTH_WARN 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: nodeep-scrub flag(s) set 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: services: 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: flags nodeep-scrub 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-11T21:29:12.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: data: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: usage: 252 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pgs: 132 active+clean 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: io: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: cluster: 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: id: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: health: HEALTH_WARN 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 26 pool(s) do not have an application enabled 2024-03-11T21:29:12.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: services: 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: data: 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pools: 32 pools, 964 pgs 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: usage: 251 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pgs: 15.456% pgs unknown 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 70.851% pgs not active 2024-03-11T21:29:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 683 creating+peering 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 149 unknown 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 132 active+clean 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: io: 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.StatusFormat (11011 ms) 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.Status (20328 ms) 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] 4 tests from LibRadosService (36450 ms total) 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] Global test environment tear-down 2024-03-11T21:29:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [==========] 4 tests from 1 test suite ran. (36450 ms total) 2024-03-11T21:29:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ PASSED ] 4 tests. 2024-03-11T21:29:12.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:12.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.367 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:12.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:12.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:12.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: starting. 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-72096 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: created object 0... 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: created object 25... 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: created object 49... 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: finishing. 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[72144]: shutting down. 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[72145]: starting. 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[72145]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-72096 2024-03-11T21:29:12.599 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[72145]: shutting down. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: starting. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-72096 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: created object 0... 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: created object 25... 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: created object 49... 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: finishing. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[75236]: shutting down. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: starting. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: listing objects. 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: listed object 0... 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: listed object 25... 2024-03-11T21:29:12.600 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: saw 50 objects 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[75238]: shutting down. 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[75237]: starting. 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[75237]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-72096 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[75237]: shutting down. 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-03-11T21:29:12.601 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-11T21:29:12.618 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:12.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:12.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:12.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.878 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:12.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:13.005 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:13.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.130 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:13.233 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:13.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:13.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:13.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.267874+0000 osd.1 (osd.1) 135 : cluster 0 2.15 scrub starts 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.269216+0000 osd.1 (osd.1) 136 : cluster 0 2.15 scrub ok 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.405507+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.744486+0000 mon.a (mon.0) 1083 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749152+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749332+0000 mon.a (mon.0) 1085 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749429+0000 mon.a (mon.0) 1086 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749531+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749583+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:13.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749666+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.749799+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:11.773769+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:12.171303+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:12.614176+0000 mon.b (mon.1) 50 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.030856+0000 mon.a (mon.0) 1091 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.031503+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.032621+0000 mon.a (mon.0) 1093 : cluster 3 pool 'PoolQuotaPP_smithi143-71604-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.035299+0000 mon.a (mon.0) 1094 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:13.041701+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: audit 2024-03-11T21:29:13.041822+0000 mon.a (mon.0) 1096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:13.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.048604+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.267874+0000 osd.1 (osd.1) 135 : cluster 0 2.15 scrub starts 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.269216+0000 osd.1 (osd.1) 136 : cluster 0 2.15 scrub ok 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.405507+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.744486+0000 mon.a (mon.0) 1083 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749152+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749332+0000 mon.a (mon.0) 1085 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749429+0000 mon.a (mon.0) 1086 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749531+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749583+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749666+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:13.507 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.749799+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:11.773769+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:12.171303+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:12.614176+0000 mon.b (mon.1) 50 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.030856+0000 mon.a (mon.0) 1091 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.031503+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.032621+0000 mon.a (mon.0) 1093 : cluster 3 pool 'PoolQuotaPP_smithi143-71604-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.035299+0000 mon.a (mon.0) 1094 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:13.041701+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: audit 2024-03-11T21:29:13.041822+0000 mon.a (mon.0) 1096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:13.508 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:13 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.048604+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:29:13.520 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:13.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:13.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.267874+0000 osd.1 (osd.1) 135 : cluster 0 2.15 scrub starts 2024-03-11T21:29:13.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.269216+0000 osd.1 (osd.1) 136 : cluster 0 2.15 scrub ok 2024-03-11T21:29:13.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.405507+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.744486+0000 mon.a (mon.0) 1083 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749152+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749332+0000 mon.a (mon.0) 1085 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-71853-7"}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749429+0000 mon.a (mon.0) 1086 : audit 1 from='client.? 172.21.15.143:0/2307213429' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749531+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.143:0/4216810631' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi143-71594-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749583+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-71929-1"}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749666+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-71620-22"}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.749799+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:11.773769+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:12.171303+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:12.614176+0000 mon.b (mon.1) 50 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.030856+0000 mon.a (mon.0) 1091 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:29:13.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.031503+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:13.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.032621+0000 mon.a (mon.0) 1093 : cluster 3 pool 'PoolQuotaPP_smithi143-71604-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T21:29:13.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.035299+0000 mon.a (mon.0) 1094 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:29:13.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:13.041701+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-71641-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:13.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: audit 2024-03-11T21:29:13.041822+0000 mon.a (mon.0) 1096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-71649-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:13.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:13 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.048604+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:29:13.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:13.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:13.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:13.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:13.900 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:14.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:14.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:14.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:14.279 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:14.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:14.424 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:14.532 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.550 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:14.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:14.675 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:14.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:14.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:14.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:15.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.060 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:15.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:15.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:15.320 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:15.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:15.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.570 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:15.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:15.721 INFO:tasks.workunit.client.0.smithi143.stdout: "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-71620-22"}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.554650+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi143-71741-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:09.977755+0000 mon.a [INF] from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:13.054574+0000 mon.c [INF] from='client.? 172.21.15.143:0/1498783647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:13.056678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:13.510783+0000 mon.a [INF] from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:13.940182+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:13.948622+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.369567+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.369642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.369742+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.377623+0000 mon.c [INF] from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.722 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.380290+0000 mon.c [INF] from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.723 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.391375+0000 mon.b [INF] from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.723 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.400933+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.723 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.411914+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:07.332671+0000 osd.7 (osd.7) 177 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:29:15.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:07.334877+0000 osd.7 (osd.7) 178 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:29:15.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:08.363545+0000 osd.7 (osd.7) 179 : cluster 0 3.d scrub starts 2024-03-11T21:29:15.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:08.366428+0000 osd.7 (osd.7) 180 : cluster 0 3.d scrub ok 2024-03-11T21:29:15.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:08.953534+0000 osd.5 (osd.5) 143 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:08.954962+0000 osd.5 (osd.5) 144 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:10.345443+0000 osd.7 (osd.7) 181 : cluster 0 6.5 scrub starts 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:10.347706+0000 osd.7 (osd.7) 182 : cluster 0 6.5 scrub ok 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:10.979107+0000 osd.5 (osd.5) 145 : cluster 0 2.16 scrub starts 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:10.980628+0000 osd.5 (osd.5) 146 : cluster 0 2.16 scrub ok 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.197022+0000 mgr.y (mgr.14550) 241 : cluster 0 pgmap v219: 700 pgs: 33 creating+peering, 492 active+clean, 175 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.345406+0000 osd.7 (osd.7) 183 : cluster 0 6.12 scrub starts 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:11.346830+0000 osd.7 (osd.7) 184 : cluster 0 6.12 scrub ok 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.054574+0000 mon.c (mon.2) 197 : audit 1 from='client.? 172.21.15.143:0/1498783647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.056678+0000 mon.a (mon.0) 1098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.061203+0000 osd.5 (osd.5) 147 : cluster 0 3.16 scrub starts 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.081717+0000 osd.5 (osd.5) 148 : cluster 0 3.16 scrub ok 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.172355+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.201681+0000 mgr.y (mgr.14550) 242 : cluster 0 pgmap v222: 772 pgs: 1 active+clean+snaptrim, 13 creating+activating, 219 creating+peering, 539 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:29:15.805 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.484459+0000 osd.3 (osd.3) 159 : cluster 0 3.8 scrub starts 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:13.487362+0000 osd.3 (osd.3) 160 : cluster 0 3.8 scrub ok 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.510783+0000 mon.a (mon.0) 1099 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.940182+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:07.332671+0000 osd.7 (osd.7) 177 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:07.334877+0000 osd.7 (osd.7) 178 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:08.363545+0000 osd.7 (osd.7) 179 : cluster 0 3.d scrub starts 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:08.366428+0000 osd.7 (osd.7) 180 : cluster 0 3.d scrub ok 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:08.953534+0000 osd.5 (osd.5) 143 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:08.954962+0000 osd.5 (osd.5) 144 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:29:15.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:10.345443+0000 osd.7 (osd.7) 181 : cluster 0 6.5 scrub starts 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:10.347706+0000 osd.7 (osd.7) 182 : cluster 0 6.5 scrub ok 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:10.979107+0000 osd.5 (osd.5) 145 : cluster 0 2.16 scrub starts 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:10.980628+0000 osd.5 (osd.5) 146 : cluster 0 2.16 scrub ok 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.197022+0000 mgr.y (mgr.14550) 241 : cluster 0 pgmap v219: 700 pgs: 33 creating+peering, 492 active+clean, 175 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.345406+0000 osd.7 (osd.7) 183 : cluster 0 6.12 scrub starts 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:11.346830+0000 osd.7 (osd.7) 184 : cluster 0 6.12 scrub ok 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.054574+0000 mon.c (mon.2) 197 : audit 1 from='client.? 172.21.15.143:0/1498783647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.056678+0000 mon.a (mon.0) 1098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.061203+0000 osd.5 (osd.5) 147 : cluster 0 3.16 scrub starts 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.081717+0000 osd.5 (osd.5) 148 : cluster 0 3.16 scrub ok 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.172355+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.201681+0000 mgr.y (mgr.14550) 242 : cluster 0 pgmap v222: 772 pgs: 1 active+clean+snaptrim, 13 creating+activating, 219 creating+peering, 539 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.484459+0000 osd.3 (osd.3) 159 : cluster 0 3.8 scrub starts 2024-03-11T21:29:15.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:13.487362+0000 osd.3 (osd.3) 160 : cluster 0 3.8 scrub ok 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.510783+0000 mon.a (mon.0) 1099 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.940182+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:13.948622+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.066530+0000 osd.5 (osd.5) 149 : cluster 0 5.7 scrub starts 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.068396+0000 osd.5 (osd.5) 150 : cluster 0 5.7 scrub ok 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.173373+0000 mon.c (mon.2) 199 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.808 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.305464+0000 osd.1 (osd.1) 137 : cluster 0 5.c scrub starts 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.307344+0000 osd.1 (osd.1) 138 : cluster 0 5.c scrub ok 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.369567+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.369642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.369742+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.376568+0000 mon.a (mon.0) 1105 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.377623+0000 mon.c (mon.2) 200 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.380290+0000 mon.c (mon.2) 201 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.391375+0000 mon.b (mon.1) 51 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.400933+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.411914+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.418731+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.419617+0000 mon.a (mon.0) 1109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.422440+0000 mon.a (mon.0) 1110 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.433374+0000 mon.b (mon.1) 52 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:15.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[29284]: audit 2024-03-11T21:29:14.440092+0000 mon.a (mon.0) 1111 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:13.948622+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.066530+0000 osd.5 (osd.5) 149 : cluster 0 5.7 scrub starts 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.068396+0000 osd.5 (osd.5) 150 : cluster 0 5.7 scrub ok 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.173373+0000 mon.c (mon.2) 199 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.305464+0000 osd.1 (osd.1) 137 : cluster 0 5.c scrub starts 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.307344+0000 osd.1 (osd.1) 138 : cluster 0 5.c scrub ok 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.369567+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.369642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.369742+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.376568+0000 mon.a (mon.0) 1105 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.377623+0000 mon.c (mon.2) 200 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.380290+0000 mon.c (mon.2) 201 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.391375+0000 mon.b (mon.1) 51 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.400933+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.411914+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.418731+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.419617+0000 mon.a (mon.0) 1109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.422440+0000 mon.a (mon.0) 1110 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.433374+0000 mon.b (mon.1) 52 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:15.811 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:15 smithi143 bash[21512]: audit 2024-03-11T21:29:14.440092+0000 mon.a (mon.0) 1111 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:15.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:15.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:15.974 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:15.975 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:07.332671+0000 osd.7 (osd.7) 177 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:29:15.975 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:07.334877+0000 osd.7 (osd.7) 178 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:29:15.975 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:08.363545+0000 osd.7 (osd.7) 179 : cluster 0 3.d scrub starts 2024-03-11T21:29:15.975 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:08.366428+0000 osd.7 (osd.7) 180 : cluster 0 3.d scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:08.953534+0000 osd.5 (osd.5) 143 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:08.954962+0000 osd.5 (osd.5) 144 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:10.345443+0000 osd.7 (osd.7) 181 : cluster 0 6.5 scrub starts 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:10.347706+0000 osd.7 (osd.7) 182 : cluster 0 6.5 scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:10.979107+0000 osd.5 (osd.5) 145 : cluster 0 2.16 scrub starts 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:10.980628+0000 osd.5 (osd.5) 146 : cluster 0 2.16 scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.197022+0000 mgr.y (mgr.14550) 241 : cluster 0 pgmap v219: 700 pgs: 33 creating+peering, 492 active+clean, 175 unknown; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.345406+0000 osd.7 (osd.7) 183 : cluster 0 6.12 scrub starts 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:11.346830+0000 osd.7 (osd.7) 184 : cluster 0 6.12 scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.054574+0000 mon.c (mon.2) 197 : audit 1 from='client.? 172.21.15.143:0/1498783647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.056678+0000 mon.a (mon.0) 1098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.061203+0000 osd.5 (osd.5) 147 : cluster 0 3.16 scrub starts 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.081717+0000 osd.5 (osd.5) 148 : cluster 0 3.16 scrub ok 2024-03-11T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.172355+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.201681+0000 mgr.y (mgr.14550) 242 : cluster 0 pgmap v222: 772 pgs: 1 active+clean+snaptrim, 13 creating+activating, 219 creating+peering, 539 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-03-11T21:29:15.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.484459+0000 osd.3 (osd.3) 159 : cluster 0 3.8 scrub starts 2024-03-11T21:29:15.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:13.487362+0000 osd.3 (osd.3) 160 : cluster 0 3.8 scrub ok 2024-03-11T21:29:15.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.510783+0000 mon.a (mon.0) 1099 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T21:29:15.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.940182+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:15.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:13.948622+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.066530+0000 osd.5 (osd.5) 149 : cluster 0 5.7 scrub starts 2024-03-11T21:29:15.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.068396+0000 osd.5 (osd.5) 150 : cluster 0 5.7 scrub ok 2024-03-11T21:29:15.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.173373+0000 mon.c (mon.2) 199 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:15.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.305464+0000 osd.1 (osd.1) 137 : cluster 0 5.c scrub starts 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.307344+0000 osd.1 (osd.1) 138 : cluster 0 5.c scrub ok 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.369567+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.369642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi143-71818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.369742+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.376568+0000 mon.a (mon.0) 1105 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.377623+0000 mon.c (mon.2) 200 : audit 1 from='client.? 172.21.15.143:0/766040946' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.380290+0000 mon.c (mon.2) 201 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.391375+0000 mon.b (mon.1) 51 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.400933+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.411914+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.418731+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.419617+0000 mon.a (mon.0) 1109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.422440+0000 mon.a (mon.0) 1110 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:15.979 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.433374+0000 mon.b (mon.1) 52 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:15.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:15 smithi203 bash[23654]: audit 2024-03-11T21:29:14.440092+0000 mon.a (mon.0) 1111 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:16.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:16.112 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:16.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:16.241 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:16.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:16.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:16.472 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.504 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:16.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.647 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:16.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:16.775 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.508196+0000 osd.3 (osd.3) 161 : cluster 0 5.14 scrub starts 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:14.510127+0000 osd.3 (osd.3) 162 : cluster 0 5.14 scrub ok 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:15.116661+0000 osd.5 (osd.5) 151 : cluster 0 3.1d scrub starts 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:15.121057+0000 osd.5 (osd.5) 152 : cluster 0 3.1d scrub ok 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.174497+0000 mon.c (mon.2) 202 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:15.512221+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.750207+0000 mon.a (mon.0) 1112 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.753979+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.754044+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:16.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.754118+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.754198+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.754286+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.763288+0000 mon.b (mon.1) 53 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.764607+0000 mon.b (mon.1) 54 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.765583+0000 mon.b (mon.1) 55 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.766328+0000 mon.b (mon.1) 56 : audit 1 from='client.? 172.21.15.143:0/813925170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.769234+0000 mon.b (mon.1) 57 : audit 1 from='client.? 172.21.15.143:0/1419840769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:15.931637+0000 mon.c (mon.2) 203 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.175584+0000 mon.c (mon.2) 204 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: cluster 2024-03-11T21:29:16.261014+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.270270+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.285854+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:16.799 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.288382+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.418731+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.419617+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.422440+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.433374+0000 mon.b [INF] from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:14.440092+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.512221+0000 client.admin [INF] threexx 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.750207+0000 mon.a [INF] from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.753979+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.754044+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.754118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.754198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:16.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.754286+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:16.821 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.763288+0000 mon.b [INF] from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:16.821 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.764607+0000 mon.b [INF] from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:16.821 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.765583+0000 mon.b [INF] from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:16.821 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.766328+0000 mon.b [INF] from='client.? 172.21.15.143:0/813925170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:16.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.508196+0000 osd.3 (osd.3) 161 : cluster 0 5.14 scrub starts 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:14.510127+0000 osd.3 (osd.3) 162 : cluster 0 5.14 scrub ok 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:15.116661+0000 osd.5 (osd.5) 151 : cluster 0 3.1d scrub starts 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:15.121057+0000 osd.5 (osd.5) 152 : cluster 0 3.1d scrub ok 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.174497+0000 mon.c (mon.2) 202 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:15.512221+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.750207+0000 mon.a (mon.0) 1112 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.753979+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.754044+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.754118+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:17.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.754198+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.754286+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.763288+0000 mon.b (mon.1) 53 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.764607+0000 mon.b (mon.1) 54 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.765583+0000 mon.b (mon.1) 55 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.766328+0000 mon.b (mon.1) 56 : audit 1 from='client.? 172.21.15.143:0/813925170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.769234+0000 mon.b (mon.1) 57 : audit 1 from='client.? 172.21.15.143:0/1419840769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:15.931637+0000 mon.c (mon.2) 203 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.175584+0000 mon.c (mon.2) 204 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: cluster 2024-03-11T21:29:16.261014+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.270270+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.285854+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.288382+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:17.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.294200+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:16 smithi203 bash[23654]: audit 2024-03-11T21:29:16.295546+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.508196+0000 osd.3 (osd.3) 161 : cluster 0 5.14 scrub starts 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:14.510127+0000 osd.3 (osd.3) 162 : cluster 0 5.14 scrub ok 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:15.116661+0000 osd.5 (osd.5) 151 : cluster 0 3.1d scrub starts 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:15.121057+0000 osd.5 (osd.5) 152 : cluster 0 3.1d scrub ok 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.174497+0000 mon.c (mon.2) 202 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:15.512221+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.750207+0000 mon.a (mon.0) 1112 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.753979+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi143-71741-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:29:17.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.754044+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.754118+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.754198+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.754286+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.763288+0000 mon.b (mon.1) 53 : audit 1 from='client.? 172.21.15.143:0/1682110848' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.764607+0000 mon.b (mon.1) 54 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.765583+0000 mon.b (mon.1) 55 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.766328+0000 mon.b (mon.1) 56 : audit 1 from='client.? 172.21.15.143:0/813925170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.053 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.294200+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[21512]: audit 2024-03-11T21:29:16.295546+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.769234+0000 mon.b (mon.1) 57 : audit 1 from='client.? 172.21.15.143:0/1419840769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:15.931637+0000 mon.c (mon.2) 203 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.175584+0000 mon.c (mon.2) 204 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: cluster 2024-03-11T21:29:16.261014+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.270270+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.285854+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.288382+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:17.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.294200+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.055 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:16 smithi143 bash[29284]: audit 2024-03-11T21:29:16.295546+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:17.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:17.188 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:17.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:17.320 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.769234+0000 mon.b [INF] from='client.? pool: Running main() from gmock_main.cc 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [----------] Global test environment set-up. 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolList (17768 ms) 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4182 ms) 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (6139 ms) 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5058 ms) 2024-03-11T21:29:17.344 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (5303 ms) 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2980 ms) 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [----------] 6 tests from NeoRadosPools (41430 ms total) 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [----------] Global test environment tear-down 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [==========] 6 tests from 1 test suite ran. (41430 ms total) 2024-03-11T21:29:17.345 INFO:tasks.workunit.client.0.smithi143.stdout: pool: [ PASSED ] 6 tests. 2024-03-11T21:29:17.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:17.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:17.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:17.603 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:17.706 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:17.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: cluster 2024-03-11T21:29:15.202509+0000 mgr.y (mgr.14550) 243 : cluster 0 pgmap v224: 740 pgs: 96 unknown, 1 active+clean+snaptrim, 5 creating+activating, 150 creating+peering, 488 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.304222+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.818503+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.818616+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.818695+0000 mon.a (mon.0) 1127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.818772+0000 mon.a (mon.0) 1128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.818839+0000 mon.a (mon.0) 1129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.823934+0000 mon.a (mon.0) 1130 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.849667+0000 mon.b (mon.1) 58 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:17.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.856937+0000 mon.b (mon.1) 59 : audit 1 from='client.? 172.21.15.143:0/2616861557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:17.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:16.859893+0000 mon.c (mon.2) 205 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:17.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: audit 2024-03-11T21:29:17.177698+0000 mon.c (mon.2) 206 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:17.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[29284]: cluster 2024-03-11T21:29:17.359627+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:29:17.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:17.932 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:18.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:18.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:18.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: cluster 2024-03-11T21:29:15.202509+0000 mgr.y (mgr.14550) 243 : cluster 0 pgmap v224: 740 pgs: 96 unknown, 1 active+clean+snaptrim, 5 creating+activating, 150 creating+peering, 488 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.304222+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.818503+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.818616+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.818695+0000 mon.a (mon.0) 1127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.818772+0000 mon.a (mon.0) 1128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.818839+0000 mon.a (mon.0) 1129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.823934+0000 mon.a (mon.0) 1130 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.849667+0000 mon.b (mon.1) 58 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.856937+0000 mon.b (mon.1) 59 : audit 1 from='client.? 172.21.15.143:0/2616861557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:16.859893+0000 mon.c (mon.2) 205 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: audit 2024-03-11T21:29:17.177698+0000 mon.c (mon.2) 206 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:18.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:17 smithi203 bash[23654]: cluster 2024-03-11T21:29:17.359627+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:29:18.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: cluster 2024-03-11T21:29:15.202509+0000 mgr.y (mgr.14550) 243 : cluster 0 pgmap v224: 740 pgs: 96 unknown, 1 active+clean+snaptrim, 5 creating+activating, 150 creating+peering, 488 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.304222+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.818503+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.818616+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.818695+0000 mon.a (mon.0) 1127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.818772+0000 mon.a (mon.0) 1128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.818839+0000 mon.a (mon.0) 1129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.823934+0000 mon.a (mon.0) 1130 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.849667+0000 mon.b (mon.1) 58 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.856937+0000 mon.b (mon.1) 59 : audit 1 from='client.? 172.21.15.143:0/2616861557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:16.859893+0000 mon.c (mon.2) 205 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: audit 2024-03-11T21:29:17.177698+0000 mon.c (mon.2) 206 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:18.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:17 smithi143 bash[21512]: cluster 2024-03-11T21:29:17.359627+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:29:18.188 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:18.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:18.314 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:18.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.440 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:18.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:18.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:18.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:18.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:18.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:18.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:19.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:19.067 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:19.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:19.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: cluster 2024-03-11T21:29:16.232132+0000 osd.1 (osd.1) 139 : cluster 0 2.6 scrub starts 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: cluster 2024-03-11T21:29:16.233733+0000 osd.1 (osd.1) 140 : cluster 0 2.6 scrub ok 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: cluster 2024-03-11T21:29:17.826178+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:17.890270+0000 mon.a (mon.0) 1132 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:17.910331+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: cluster 2024-03-11T21:29:17.920946+0000 mon.a (mon.0) 1134 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:17.932934+0000 mon.c (mon.2) 207 : audit 1 from='client.? 172.21.15.143:0/2823279075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:18.179608+0000 mon.c (mon.2) 208 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:18.464123+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:19.349 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:18.475836+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:19.350 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:18.484344+0000 mon.a (mon.0) 1137 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:19.350 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:19 smithi203 bash[23654]: audit 2024-03-11T21:29:18.484680+0000 mon.c (mon.2) 209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:19.363 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:19.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: cluster 2024-03-11T21:29:16.232132+0000 osd.1 (osd.1) 139 : cluster 0 2.6 scrub starts 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: cluster 2024-03-11T21:29:16.233733+0000 osd.1 (osd.1) 140 : cluster 0 2.6 scrub ok 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: cluster 2024-03-11T21:29:17.826178+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:17.890270+0000 mon.a (mon.0) 1132 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:17.910331+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: cluster 2024-03-11T21:29:17.920946+0000 mon.a (mon.0) 1134 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:17.932934+0000 mon.c (mon.2) 207 : audit 1 from='client.? 172.21.15.143:0/2823279075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:18.179608+0000 mon.c (mon.2) 208 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:18.464123+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:18.475836+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:18.484344+0000 mon.a (mon.0) 1137 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[21512]: audit 2024-03-11T21:29:18.484680+0000 mon.c (mon.2) 209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: cluster 2024-03-11T21:29:16.232132+0000 osd.1 (osd.1) 139 : cluster 0 2.6 scrub starts 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: cluster 2024-03-11T21:29:16.233733+0000 osd.1 (osd.1) 140 : cluster 0 2.6 scrub ok 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: cluster 2024-03-11T21:29:17.826178+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:17.890270+0000 mon.a (mon.0) 1132 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:17.910331+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: cluster 2024-03-11T21:29:17.920946+0000 mon.a (mon.0) 1134 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:17.932934+0000 mon.c (mon.2) 207 : audit 1 from='client.? 172.21.15.143:0/2823279075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:18.179608+0000 mon.c (mon.2) 208 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:18.464123+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:18.475836+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:18.484344+0000 mon.a (mon.0) 1137 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:19.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:19 smithi143 bash[29284]: audit 2024-03-11T21:29:18.484680+0000 mon.c (mon.2) 209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:19.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:19.491 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:19.594 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:19.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:19.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:20.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.084 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:20.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:20.228 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:20.230 INFO:tasks.workunit.client.0.smithi143.stdout: list: Running main() from gmock_main.cc 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [----------] Global test environment set-up. 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [----------] 3 tests from NeoradosList 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ OK ] NeoradosList.ListObjects (22866 ms) 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ OK ] NeoradosList.ListObjectsNS (9847 ms) 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ OK ] NeoradosList.ListObjectsMany (11637 ms) 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [----------] 3 tests from NeoradosList (44350 ms total) 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [----------] Global test environment tear-down 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [==========] 3 tests from 1 test suite ran. (44350 ms total) 2024-03-11T21:29:20.231 INFO:tasks.workunit.client.0.smithi143.stdout: list: [ PASSED ] 3 tests. 2024-03-11T21:29:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:16.090094+0000 osd.5 (osd.5) 153 : cluster 0 5.f scrub starts 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:16.091343+0000 osd.5 (osd.5) 154 : cluster 0 5.f scrub ok 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:17.203393+0000 mgr.y (mgr.14550) 244 : cluster 0 pgmap v227: 712 pgs: 164 unknown, 5 creating+activating, 86 creating+peering, 457 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:18.037535+0000 osd.5 (osd.5) 155 : cluster 0 6.d scrub starts 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:18.039054+0000 osd.5 (osd.5) 156 : cluster 0 6.d scrub ok 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:18.344844+0000 mgr.y (mgr.14550) 245 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.047008+0000 mon.a (mon.0) 1138 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.053996+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T21:29:20.236 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.071423+0000 mon.a (mon.0) 1140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.071561+0000 mon.a (mon.0) 1141 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: cluster 2024-03-11T21:29:19.086664+0000 mon.a (mon.0) 1142 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.088731+0000 mon.a (mon.0) 1143 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.135767+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.135918+0000 mon.c (mon.2) 210 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.136234+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.142422+0000 mon.a (mon.0) 1145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.142758+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.143026+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[21512]: audit 2024-03-11T21:29:19.186623+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:16.090094+0000 osd.5 (osd.5) 153 : cluster 0 5.f scrub starts 2024-03-11T21:29:20.237 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:16.091343+0000 osd.5 (osd.5) 154 : cluster 0 5.f scrub ok 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:17.203393+0000 mgr.y (mgr.14550) 244 : cluster 0 pgmap v227: 712 pgs: 164 unknown, 5 creating+activating, 86 creating+peering, 457 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:18.037535+0000 osd.5 (osd.5) 155 : cluster 0 6.d scrub starts 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:18.039054+0000 osd.5 (osd.5) 156 : cluster 0 6.d scrub ok 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:18.344844+0000 mgr.y (mgr.14550) 245 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.047008+0000 mon.a (mon.0) 1138 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.053996+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.071423+0000 mon.a (mon.0) 1140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.071561+0000 mon.a (mon.0) 1141 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: cluster 2024-03-11T21:29:19.086664+0000 mon.a (mon.0) 1142 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.088731+0000 mon.a (mon.0) 1143 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.135767+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.135918+0000 mon.c (mon.2) 210 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.136234+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.239 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.142422+0000 mon.a (mon.0) 1145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.239 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.142758+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.239 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.143026+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.239 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:20 smithi143 bash[29284]: audit 2024-03-11T21:29:19.186623+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:20.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:20.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:20.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:16.090094+0000 osd.5 (osd.5) 153 : cluster 0 5.f scrub starts 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:16.091343+0000 osd.5 (osd.5) 154 : cluster 0 5.f scrub ok 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:17.203393+0000 mgr.y (mgr.14550) 244 : cluster 0 pgmap v227: 712 pgs: 164 unknown, 5 creating+activating, 86 creating+peering, 457 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:18.037535+0000 osd.5 (osd.5) 155 : cluster 0 6.d scrub starts 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:18.039054+0000 osd.5 (osd.5) 156 : cluster 0 6.d scrub ok 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:18.344844+0000 mgr.y (mgr.14550) 245 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.047008+0000 mon.a (mon.0) 1138 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.053996+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.143:0/4264182251' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.071423+0000 mon.a (mon.0) 1140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.071561+0000 mon.a (mon.0) 1141 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi143-72329-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: cluster 2024-03-11T21:29:19.086664+0000 mon.a (mon.0) 1142 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.088731+0000 mon.a (mon.0) 1143 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.135767+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:29:20.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.135918+0000 mon.c (mon.2) 210 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.136234+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.142422+0000 mon.a (mon.0) 1145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.142758+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.143026+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:20 smithi203 bash[23654]: audit 2024-03-11T21:29:19.186623+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:20.527 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:20.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:20.652 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:20.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:20.775 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2874006262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880aab70 msgr2=0x7fbe880aaf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2874006262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880aab70 0x7fbe880aaf50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009de0 tx=0x7fbe8000b540 comp rx=0 tx=0).stop 2024-03-11T21:29:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2874006262 shutdown_connections 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2874006262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880aab70 0x7fbe880aaf50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2874006262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a5790 0x7fbe8800bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2874006262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe880a4dd0 0x7fbe880a5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2874006262 >> 172.21.15.143:0/2874006262 conn(0x7fbe8801a690 msgr2=0x7fbe8801aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2874006262 shutdown_connections 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2874006262 wait complete. 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 librados: starting msgr at 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 librados: starting objecter 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 Processor -- start 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- start start 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 librados: setting wanted keys 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 librados: calling monclient init 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880a5790 0x7fbe8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe880aab70 0x7fbe88005530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800ef30 con 0x7fbe880a5790 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800edb0 con 0x7fbe880a4dd0 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800f0b0 con 0x7fbe880aab70 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.143:44944/0 (socket says 172.21.15.143:44944) 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 -- 172.21.15.143:0/2791333235 learned_addr learned my addr 172.21.15.143:0/2791333235 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 -- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe880aab70 msgr2=0x7fbe88005530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe880aab70 0x7fbe88005530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 -- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880a5790 msgr2=0x7fbe8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880a5790 0x7fbe8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 -- 172.21.15.143:0/2791333235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe88005cf0 con 0x7fbe880a4dd0 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8e59c700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbe800038b0 tx=0x7fbe80003990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe7f7fe700 10 client.?.objecter ms_handle_connect 0x7fbe880a4dd0 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe7f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80027070 con 0x7fbe880a4dd0 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe88005f80 con 0x7fbe880a4dd0 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 10 client.28081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe881586f0 con 0x7fbe880a4dd0 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.812+0000 7fbe8f59e700 1 librados: init done 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.816+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fbe80022470 con 0x7fbe880a4dd0 2024-03-11T21:29:20.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.816+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8000f6c0 con 0x7fbe880a4dd0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70005120 con 0x7fbe880a4dd0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbe80045870 con 0x7fbe880a4dd0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 0x7fbe78071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe8dd9b700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 0x7fbe78071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(164..164 src has 87..164) v4 ==== 16859+0+0 (secure 0 0 0) 0x7fbe80091f50 con 0x7fbe880a4dd0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 10 client.28081.objecter ms_dispatch 0x7fbe90001550 osd_map(164..164 src has 87..164) v4 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 3 client.28081.objecter handle_osd_map got epochs [164,164] > 0 2024-03-11T21:29:20.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 3 client.28081.objecter handle_osd_map decoding full epoch 164 2024-03-11T21:29:20.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.820+0000 7fbe7f7fe700 20 client.28081.objecter dump_active .. 0 homeless 2024-03-11T21:29:20.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.824+0000 7fbe8dd9b700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 0x7fbe78071bb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006cf0 tx=0x7fbe84006d20 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:20.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.824+0000 7fbe7f7fe700 10 client.28081.objecter ms_handle_connect 0x7fbe7806f6f0 2024-03-11T21:29:20.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:20.828+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fbe80013070 con 0x7fbe880a4dd0 2024-03-11T21:29:20.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:21.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:21.049 INFO:tasks.workunit.client.0.smithi143.stdout:172.21.15.143:0/1419840769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:21.049 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:15.931637+0000 mon.c [INF] from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71929-6", "pg_num": 4}]: dispatch 2024-03-11T21:29:21.049 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:16.270270+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi143-72329-1"}]: dispatch 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:16.285854+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi143-71918-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:16.288382+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-4", "overlaypool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:16.294200+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi143-71594-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-03-11T21:29:16.295546+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (11977 ms) 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (45342 ms total) 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (45342 ms total) 2024-03-11T21:29:21.050 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-11T21:29:21.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:21.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.240+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fbe700056e0 con 0x7fbe880a4dd0 2024-03-11T21:29:21.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.248+0000 7fbe7f7fe700 1 -- 172.21.15.143:0/2791333235 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v164) v1 ==== 81+0+101938 (secure 0 0 0) 0x7fbe8005d120 con 0x7fbe880a4dd0 2024-03-11T21:29:21.226 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:29:21.237 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:29:21.237 INFO:teuthology.orchestra.run.smithi143.stdout: "epoch": 164, 2024-03-11T21:29:21.237 INFO:teuthology.orchestra.run.smithi143.stdout: "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:29:21.237 INFO:teuthology.orchestra.run.smithi143.stdout: "created": "2024-03-11T21:12:47.384719+0000", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "modified": "2024-03-11T21:29:20.127552+0000", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "last_up_change": "2024-03-11T21:22:21.063566+0000", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "last_in_change": "2024-03-11T21:21:55.456354+0000", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_num": 5799936, 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_set": [ 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "pglog_hardlimit", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "purged_snapdirs", 2024-03-11T21:29:21.238 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_deletes", 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "sortbitwise" 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_version": 28, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_snapshot": "", 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_max": 102, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "max_osd": 8, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "min_compat_client": "luminous", 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "require_osd_release": "squid", 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "allow_crimson": false, 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: "pools": [ 2024-03-11T21:29:21.239 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 1, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".mgr", 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:18:08.036690+0000", 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.240 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 1, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 1, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 1, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 1, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 1, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.241 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "141", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.243 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.244 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_max": 32, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_min": 1 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr": {} 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 7.8899998664855957, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 7.8899998664855957, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 3, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 3, 2024-03-11T21:29:21.245 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 2, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".rgw.root", 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:01.432928+0000", 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.246 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.247 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "78", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.248 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.249 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.251 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 3, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:03.441476+0000", 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.252 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.253 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "80", 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.254 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.255 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.256 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.257 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 4, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "datapool", 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:04.441432+0000", 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:29:21.258 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.259 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 3, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 3, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 3, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 3, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 3, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.260 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "87", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 2, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 87, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.261 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.262 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.263 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd": {} 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2.6500000953674316, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2.6500000953674316, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.264 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 5, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:05.388543+0000", 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.265 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.266 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "82", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.267 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.268 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.269 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.25, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.25, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.25, 2024-03-11T21:29:21.270 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.25, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 6, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:07.642696+0000", 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.271 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.272 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "84", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.273 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.274 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.275 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_bias": 4 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.276 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 9, 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-11T21:29:21.277 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:36.978109+0000", 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.280 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.281 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.282 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 12, 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosList_smithi143-71636-1", 2024-03-11T21:29:21.283 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:38.230956+0000", 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.284 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.285 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.286 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.287 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.288 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 13, 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosMiscPP_smithi143-71756-1", 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:38.643455+0000", 2024-03-11T21:29:21.289 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.290 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.291 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.292 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.294 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.295 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 18, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosTierPP_smithi143-71773-1", 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:40.754537+0000", 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.296 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.297 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.298 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.299 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.300 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.301 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 39, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "SnapListsmithi143-72399-1", 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:49.753089+0000", 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.302 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.303 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "153", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.304 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.305 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.306 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.307 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2, 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2, 2024-03-11T21:29:21.308 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2, 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2, 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.309 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 52, 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "PoolQuotaPP_smithi143-71604-2", 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:56.015048+0000", 2024-03-11T21:29:21.310 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.311 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.312 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.313 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.314 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "157", 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.315 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.316 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.317 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.318 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 57, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-4", 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:58.311890+0000", 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "161", 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "161", 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "161", 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "161", 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 160, 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [ 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: 76 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": 76, 2024-03-11T21:29:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": 76, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 69, 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosMisc_smithi143-71741-1", 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:03.594147+0000", 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "160", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "app1": { 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "key2": "value2" 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "app2": {}, 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2, 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2, 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2, 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2, 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 76, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-4-cache", 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:07.281070+0000", 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 9, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,incomplete_clones", 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "161", 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "161", 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "161", 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 160, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": 57, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "writeback", 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 89, 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosSnapshotsSelfManaged_smithi143-71818-6", 2024-03-11T21:29:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:13.028596+0000", 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 3, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 164, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 90, 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:13.507862+0000", 2024-03-11T21:29:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "164", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "164", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "164", 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 163, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [ 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: 97 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": 97, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": 97, 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 96, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71794-5", 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:16.260147+0000", 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 97, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71929-6", 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:16.802774+0000", 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 4, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 4, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 4, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 4, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 4, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "164", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "164", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "164", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 163, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": 90, 2024-03-11T21:29:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.75, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 98, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosWatchNotifyEC_smithi143-71918-12", 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:17.359374+0000", 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 3, 2024-03-11T21:29:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 1, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 8, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 8, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 8, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 8, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 8, 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "162", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12", 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 8192, 2024-03-11T21:29:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {} 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 99, 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7", 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:17.886593+0000", 2024-03-11T21:29:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 100, 2024-03-11T21:29:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "SparseReadsmithi143-72205-4", 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:18.459721+0000", 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "163", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 101, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "LongNamespacesmithi143-72351-5", 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:19.010861+0000", 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "163", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 102, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "Readsmithi143-72391-5", 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:19.689743+0000", 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 32769, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,creating", 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:29:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:29:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "164", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:29:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:29:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:29:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:29:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "osds": [ 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:29:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d", 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 8, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6802", 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6803", 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6804", 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6805", 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6808", 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6809", 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6806", 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6807", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6803/2903432023", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6805/2903432023", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6809/2903432023", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6807/2903432023", 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02", 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 13, 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6810", 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6811", 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6812", 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6813", 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6816", 2024-03-11T21:29:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6817", 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6814", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6815", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 397942214 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6811/397942214", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6813/397942214", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6817/397942214", 2024-03-11T21:29:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6815/397942214", 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d", 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.420 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 19, 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6818", 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6819", 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.421 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6820", 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6821", 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.422 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6824", 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.423 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6825", 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6822", 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.424 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6823", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6819/2592131625", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6821/2592131625", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6825/2592131625", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6823/2592131625", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.425 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc", 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 27, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.426 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6826", 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6827", 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.427 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6828", 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6829", 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.428 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6832", 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6833", 2024-03-11T21:29:21.429 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6830", 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.430 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6831", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6827/2146571980", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6829/2146571980", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6833/2146571980", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6831/2146571980", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.431 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "d7b9e088-d377-4335-8863-11d581828f7e", 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 35, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.432 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6800", 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6801", 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.434 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6802", 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6803", 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.435 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6806", 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6807", 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.436 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6804", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6805", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6801/2090835691", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6803/2090835691", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6807/2090835691", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6805/2090835691", 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.437 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01", 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.438 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 46, 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.439 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6808", 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6809", 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.440 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6810", 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6811", 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6814", 2024-03-11T21:29:21.441 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6815", 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6812", 2024-03-11T21:29:21.442 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6813", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6809/2432038345", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6811/2432038345", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2432038345", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6813/2432038345", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.443 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241", 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.444 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 56, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.445 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6816", 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.446 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6817", 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6818", 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.447 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6819", 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.448 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6822", 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6823", 2024-03-11T21:29:21.449 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6820", 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.450 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6821", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6817/513748834", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6819/513748834", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6823/513748834", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6821/513748834", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "88b53171-69d2-4922-a787-e63d44d01c60", 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 66, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 163, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6824", 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6825", 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.453 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6826", 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6827", 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:29:21.454 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6830", 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6831", 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.455 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6828", 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6829", 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:29:21.456 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6825/1251497952", 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6827/1251497952", 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6831/1251497952", 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6829/1251497952", 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_xinfo": [ 2024-03-11T21:29:21.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:57.552126+0000", 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.458 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:06.790655+0000", 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.459 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:25.751047+0000", 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.460 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:21.177688+0000", 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.461 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:49.516058+0000", 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:23.191241+0000", 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.463 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:52.553316+0000", 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:18.149979+0000", 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.464 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap": [], 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_items": [ 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "2.18", 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.465 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_primaries": [], 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_temp": [], 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_temp": [], 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "blocklist": { 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3638739149": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4274199357": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3217951344": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3912416566": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/906834629": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2791822486": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:29:21.466 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2190104670": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4171894415": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2845155916": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2913593434": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/143435387": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2930358289": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3638739149": "2024-03-12T21:23:57.022184+0000" 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "range_blocklist": {}, 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profiles": { 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "default": { 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.467 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van" 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "testprofile-LibRadosIoEC_smithi143-71612-16": { 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-device-class": "", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-num-failure-domains": "0", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-osds-per-failure-domain": "0", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-root": "default", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.468 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "w": "8" 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "testprofile-LibRadosLockECPP_smithi143-71649-10": { 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-device-class": "", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-num-failure-domains": "0", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-osds-per-failure-domain": "0", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-root": "default", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van", 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "w": "8" 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.469 INFO:teuthology.orchestra.run.smithi143.stdout: "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12": { 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-device-class": "", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-num-failure-domains": "0", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-osds-per-failure-domain": "0", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-root": "default", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van", 2024-03-11T21:29:21.470 INFO:teuthology.orchestra.run.smithi143.stdout: "w": "8" 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "testprofile-ListObjectsNSsmithi143-72329-2": { 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-device-class": "", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-num-failure-domains": "0", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-osds-per-failure-domain": "0", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-root": "default", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van", 2024-03-11T21:29:21.471 INFO:teuthology.orchestra.run.smithi143.stdout: "w": "8" 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "testprofile-ReadOpsmithi143-72277-2": { 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-device-class": "", 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-num-failure-domains": "0", 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-osds-per-failure-domain": "0", 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-root": "default", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van", 2024-03-11T21:29:21.473 INFO:teuthology.orchestra.run.smithi143.stdout: "w": "8" 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps_queue": [ 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 28, 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:29:21.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 5, 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 35, 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 1, 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:29:21.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 89, 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: "new_removed_snaps": [ 2024-03-11T21:29:21.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 89, 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "new_purged_snaps": [], 2024-03-11T21:29:21.477 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_node_flags": {}, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "device_class_flags": {}, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode": { 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_enabled": false, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_bucket_count": 0, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_bucket": 0 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:29:21.478 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 10 librados: watch_flush enter 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 10 librados: watch_flush exit 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 20 client.28081.objecter shutdown clearing up homeless session... 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 10 client.28081.objecter successfully canceled tick 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 msgr2=0x7fbe78071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 0x7fbe78071bb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006cf0 tx=0x7fbe84006d20 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 msgr2=0x7fbe8800b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.260+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbe800038b0 tx=0x7fbe80003990 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 shutdown_connections 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f6f0 0x7fbe78071bb0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe880aab70 0x7fbe88005530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe880a4dd0 0x7fbe8800b480 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 --2- 172.21.15.143:0/2791333235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe880a5790 0x7fbe8800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.264+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 >> 172.21.15.143:0/2791333235 conn(0x7fbe8801a690 msgr2=0x7fbe880a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:21.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.268+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 shutdown_connections 2024-03-11T21:29:21.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.268+0000 7fbe8f59e700 1 -- 172.21.15.143:0/2791333235 wait complete. 2024-03-11T21:29:21.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.268+0000 7fbe8f59e700 1 librados: shutdown 2024-03-11T21:29:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.404+0000 7fa9dcf99700 1 -- 172.21.15.143:0/1751254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 msgr2=0x7fa9d806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.404+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/1751254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806ae90 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099c0 tx=0x7fa9cc009cd0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- 172.21.15.143:0/1751254630 shutdown_connections 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/1751254630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d8104a10 0x7fa9d8104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/1751254630 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8106e90 0x7fa9d806a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/1751254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806ae90 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- 172.21.15.143:0/1751254630 >> 172.21.15.143:0/1751254630 conn(0x7fa9d8077de0 msgr2=0x7fa9d80781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- 172.21.15.143:0/1751254630 shutdown_connections 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- 172.21.15.143:0/1751254630 wait complete. 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 librados: starting msgr at 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 librados: starting objecter 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 Processor -- start 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- start start 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 librados: setting wanted keys 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 librados: calling monclient init 2024-03-11T21:29:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d8106e90 0x7fa9d81b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8117df0 con 0x7fa9d8106e90 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8117c70 con 0x7fa9d806aa10 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9dcf99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8117f70 con 0x7fa9d8104a10 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33628/0 (socket says 172.21.15.143:33628) 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d5d9b700 1 -- 172.21.15.143:0/3463061929 learned_addr learned my addr 172.21.15.143:0/3463061929 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 -- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 msgr2=0x7fa9d806c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 -- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d8106e90 msgr2=0x7fa9d81b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d8106e90 0x7fa9d81b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d659c700 1 -- 172.21.15.143:0/3463061929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d81b2d50 con 0x7fa9d806aa10 2024-03-11T21:29:21.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.408+0000 7fa9d5d9b700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.412+0000 7fa9d659c700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806c260 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000bd70 tx=0x7fa9d000bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.412+0000 7fa9c77fe700 10 client.?.objecter ms_handle_connect 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.412+0000 7fa9c77fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0019070 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d81b3040 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9dcf99700 10 client.28489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d81b3550 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9dcf99700 1 librados: init done 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa9d000cc30 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00125a0 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.416+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa9d001c070 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 0x7fa9c0071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b8005150 con 0x7fa9d806aa10 2024-03-11T21:29:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(164..164 src has 87..164) v4 ==== 16859+0+0 (secure 0 0 0) 0x7fa9d008fdb0 con 0x7fa9d806aa10 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 10 client.28489.objecter ms_dispatch 0x7fa9d8001550 osd_map(164..164 src has 87..164) v4 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 3 client.28489.objecter handle_osd_map got epochs [164,164] > 0 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 3 client.28489.objecter handle_osd_map decoding full epoch 164 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9c77fe700 20 client.28489.objecter dump_active .. 0 homeless 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.420+0000 7fa9d5d9b700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 0x7fa9c0071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.424+0000 7fa9d5d9b700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 0x7fa9c0071cc0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa9d806d850 tx=0x7fa9cc005fb0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.424+0000 7fa9c77fe700 10 client.28489.objecter ms_handle_connect 0x7fa9c006f800 2024-03-11T21:29:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.440+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa9d005aea0 con 0x7fa9d806aa10 2024-03-11T21:29:21.600 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:21.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:21.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:21.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:21.876+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa9b8005440 con 0x7fa9d806aa10 2024-03-11T21:29:21.915 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:29:21.916 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:29:21.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:21.974 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:22.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:22.101 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:19.204248+0000 mgr.y (mgr.14550) 246 : cluster 0 pgmap v230: 808 pgs: 64 unknown, 89 creating+peering, 655 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 13 KiB/s wr, 23 op/s 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:19.354894+0000 osd.7 (osd.7) 185 : cluster 0 3.11 scrub starts 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:19.356697+0000 osd.7 (osd.7) 186 : cluster 0 3.11 scrub ok 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:20.129901+0000 mon.a (mon.0) 1148 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:20.129930+0000 mon.a (mon.0) 1149 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.140815+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.140945+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]': finished 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.141025+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:22.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.141084+0000 mon.a (mon.0) 1153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.141149+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.237251+0000 mon.c (mon.2) 213 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.238124+0000 mon.c (mon.2) 214 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.238596+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.252689+0000 mon.c (mon.2) 216 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.490518+0000 mon.b (mon.1) 60 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: cluster 2024-03-11T21:29:20.804762+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.804890+0000 mon.b (mon.1) 61 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.805101+0000 mon.b (mon.1) 62 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.813859+0000 mon.a (mon.0) 1156 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.817788+0000 mon.a (mon.0) 1157 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.826087+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.852121+0000 mon.a (mon.0) 1159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.852313+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:20.852471+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:21.241412+0000 mon.b (mon.1) 63 : audit 0 from='client.? 172.21.15.143:0/2791333235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21512]: audit 2024-03-11T21:29:21.246454+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.190 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:19.204248+0000 mgr.y (mgr.14550) 246 : cluster 0 pgmap v230: 808 pgs: 64 unknown, 89 creating+peering, 655 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 13 KiB/s wr, 23 op/s 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:19.354894+0000 osd.7 (osd.7) 185 : cluster 0 3.11 scrub starts 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:19.356697+0000 osd.7 (osd.7) 186 : cluster 0 3.11 scrub ok 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:20.129901+0000 mon.a (mon.0) 1148 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:20.129930+0000 mon.a (mon.0) 1149 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.140815+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.140945+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]': finished 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.141025+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.141084+0000 mon.a (mon.0) 1153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.141149+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:22.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.237251+0000 mon.c (mon.2) 213 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.238124+0000 mon.c (mon.2) 214 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.238596+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.252689+0000 mon.c (mon.2) 216 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.490518+0000 mon.b (mon.1) 60 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: cluster 2024-03-11T21:29:20.804762+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.804890+0000 mon.b (mon.1) 61 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.805101+0000 mon.b (mon.1) 62 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.813859+0000 mon.a (mon.0) 1156 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.817788+0000 mon.a (mon.0) 1157 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.826087+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.852121+0000 mon.a (mon.0) 1159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.852313+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:20.852471+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:21.241412+0000 mon.b (mon.1) 63 : audit 0 from='client.? 172.21.15.143:0/2791333235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:29:22.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:22 smithi143 bash[29284]: audit 2024-03-11T21:29:21.246454+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: Running main() from gmock_main.cc 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] Global test environment set-up. 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-11T21:29:22.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (269 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockShared (61 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1005 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1005 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (9 ms) 2024-03-11T21:29:22.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLock (2362 ms total) 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1447 ms) 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (11 ms) 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1007 ms) 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-11T21:29:22.195 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1008 ms) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (9 ms) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (11 ms) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (7 ms) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (5 ms) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3505 ms total) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] Global test environment tear-down 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (46612 ms total) 2024-03-11T21:29:22.196 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-11T21:29:22.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.270 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:19.204248+0000 mgr.y (mgr.14550) 246 : cluster 0 pgmap v230: 808 pgs: 64 unknown, 89 creating+peering, 655 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 13 KiB/s wr, 23 op/s 2024-03-11T21:29:22.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:19.354894+0000 osd.7 (osd.7) 185 : cluster 0 3.11 scrub starts 2024-03-11T21:29:22.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:19.356697+0000 osd.7 (osd.7) 186 : cluster 0 3.11 scrub ok 2024-03-11T21:29:22.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:20.129901+0000 mon.a (mon.0) 1148 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:20.129930+0000 mon.a (mon.0) 1149 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.140815+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi143-71838-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.140945+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-4-cache", "mode": "writeback"}]': finished 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.141025+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.141084+0000 mon.a (mon.0) 1153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.141149+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "overlaypool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:22.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.237251+0000 mon.c (mon.2) 213 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.238124+0000 mon.c (mon.2) 214 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.238596+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.143:0/2890051951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.252689+0000 mon.c (mon.2) 216 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.490518+0000 mon.b (mon.1) 60 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: cluster 2024-03-11T21:29:20.804762+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.804890+0000 mon.b (mon.1) 61 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.805101+0000 mon.b (mon.1) 62 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.813859+0000 mon.a (mon.0) 1156 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.817788+0000 mon.a (mon.0) 1157 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.826087+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:22.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.852121+0000 mon.a (mon.0) 1159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]: dispatch 2024-03-11T21:29:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.852313+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:20.852471+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:21.241412+0000 mon.b (mon.1) 63 : audit 0 from='client.? 172.21.15.143:0/2791333235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:29:22.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:22 smithi203 bash[23654]: audit 2024-03-11T21:29:21.246454+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:22.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:22.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:22.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:22.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:22.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:22.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:22.859 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:22.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:22.989 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:23.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.115 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:23.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:23.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:23.338 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: Running main() from gmock_main.cc 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [----------] Global test environment set-up. 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (22873 ms) 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (24589 ms) 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (47462 ms total) 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [----------] Global test environment tear-down 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (47462 ms total) 2024-03-11T21:29:23.339 INFO:tasks.workunit.client.0.smithi143.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-11T21:29:23.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.344+0000 7fa9c77fe700 1 -- 172.21.15.143:0/3463061929 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v166) v1 ==== 87+0+0 (secure 0 0 0) 0x7fa9d000cde0 con 0x7fa9d806aa10 2024-03-11T21:29:23.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.348+0000 7fa9dcf99700 10 librados: watch_flush enter 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.352+0000 7fa9dcf99700 10 librados: watch_flush exit 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.352+0000 7fa9dcf99700 20 client.28489.objecter shutdown clearing up homeless session... 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.352+0000 7fa9dcf99700 10 client.28489.objecter successfully canceled tick 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.352+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 msgr2=0x7fa9c0071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.352+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 0x7fa9c0071cc0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa9d806d850 tx=0x7fa9cc005fb0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 msgr2=0x7fa9d806c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806c260 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000bd70 tx=0x7fa9d000bda0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 shutdown_connections 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9c006f800 0x7fa9c0071cc0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d8106e90 0x7fa9d81b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d806aa10 0x7fa9d806c260 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 --2- 172.21.15.143:0/3463061929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d8104a10 0x7fa9d806c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.356+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 >> 172.21.15.143:0/3463061929 conn(0x7fa9d8077de0 msgr2=0x7fa9d80fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.364+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 shutdown_connections 2024-03-11T21:29:23.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.364+0000 7fa9dcf99700 1 -- 172.21.15.143:0/3463061929 wait complete. 2024-03-11T21:29:23.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: cluster 2024-03-11T21:29:20.444907+0000 osd.1 (osd.1) 141 : cluster 0 3.19 scrub starts 2024-03-11T21:29:23.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: cluster 2024-03-11T21:29:20.448103+0000 osd.1 (osd.1) 142 : cluster 0 3.19 scrub ok 2024-03-11T21:29:23.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: cluster 2024-03-11T21:29:21.204992+0000 mgr.y (mgr.14550) 247 : cluster 0 pgmap v232: 656 pgs: 96 unknown, 76 creating+peering, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:29:23.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:21.877876+0000 mon.b (mon.1) 64 : audit 1 from='client.? 172.21.15.143:0/3463061929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.122257+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126599+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126683+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126743+0000 mon.a (mon.0) 1165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126796+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126847+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126900+0000 mon.a (mon.0) 1168 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.126951+0000 mon.a (mon.0) 1169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.149174+0000 mon.b (mon.1) 65 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.157958+0000 mon.b (mon.1) 66 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.158662+0000 mon.b (mon.1) 67 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.163425+0000 mon.b (mon.1) 68 : audit 1 from='client.? 172.21.15.143:0/1307145598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.166927+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.172898+0000 mon.b (mon.1) 69 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.173309+0000 mon.b (mon.1) 70 : audit 1 from='client.? 172.21.15.143:0/3628656352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.260205+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: cluster 2024-03-11T21:29:22.740516+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: cluster 2024-03-11T21:29:22.755483+0000 mon.a (mon.0) 1171 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.778497+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.786135+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.787211+0000 mon.a (mon.0) 1174 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.787622+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.358 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:23 smithi203 bash[23654]: audit 2024-03-11T21:29:22.788110+0000 mon.a (mon.0) 1176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:23.364+0000 7fa9dcf99700 1 librados: shutdown 2024-03-11T21:29:23.365 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: seed 71649 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (317 ms) 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (6 ms) 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1012 ms) 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-11T21:29:23.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1008 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2361 ms total) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1455 ms) 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-11T21:29:23.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (1377 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1025 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1045 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (28 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (7 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (7 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (4947 ms total) 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: 2024-03-11T21:29:23.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-11T21:29:23.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (47789 ms total) 2024-03-11T21:29:23.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-11T21:29:23.379 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: Running main() from gmock_main.cc 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] Global test environment set-up. 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (288 ms) 2024-03-11T21:29:23.392 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (53 ms) 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (18 ms) 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-11T21:29:23.393 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (9 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (11 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (13 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (8 ms) 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-11T21:29:23.394 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RmXattr (18 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 14 tests from LibRadosIo (439 ms total) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (6855 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (13 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-11T21:29:23.395 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (6 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (15 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (14 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (18 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (8 ms) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 10 tests from LibRadosIoEC (6952 ms total) 2024-03-11T21:29:23.396 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: 2024-03-11T21:29:23.397 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] Global test environment tear-down 2024-03-11T21:29:23.397 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [==========] 24 tests from 2 test suites ran. (47835 ms total) 2024-03-11T21:29:23.397 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ PASSED ] 24 tests. 2024-03-11T21:29:23.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: cluster 2024-03-11T21:29:20.444907+0000 osd.1 (osd.1) 141 : cluster 0 3.19 scrub starts 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: cluster 2024-03-11T21:29:20.448103+0000 osd.1 (osd.1) 142 : cluster 0 3.19 scrub ok 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: cluster 2024-03-11T21:29:21.204992+0000 mgr.y (mgr.14550) 247 : cluster 0 pgmap v232: 656 pgs: 96 unknown, 76 creating+peering, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:21.877876+0000 mon.b (mon.1) 64 : audit 1 from='client.? 172.21.15.143:0/3463061929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.122257+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126599+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126683+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]': finished 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126743+0000 mon.a (mon.0) 1165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:23.502 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126796+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126847+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]': finished 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126900+0000 mon.a (mon.0) 1168 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.126951+0000 mon.a (mon.0) 1169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.149174+0000 mon.b (mon.1) 65 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.157958+0000 mon.b (mon.1) 66 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.158662+0000 mon.b (mon.1) 67 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.163425+0000 mon.b (mon.1) 68 : audit 1 from='client.? 172.21.15.143:0/1307145598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.166927+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.172898+0000 mon.b (mon.1) 69 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.173309+0000 mon.b (mon.1) 70 : audit 1 from='client.? 172.21.15.143:0/3628656352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.260205+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: cluster 2024-03-11T21:29:22.740516+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: cluster 2024-03-11T21:29:22.755483+0000 mon.a (mon.0) 1171 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:23.503 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.778497+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.786135+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.787211+0000 mon.a (mon.0) 1174 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.787622+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[21512]: audit 2024-03-11T21:29:22.788110+0000 mon.a (mon.0) 1176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: cluster 2024-03-11T21:29:20.444907+0000 osd.1 (osd.1) 141 : cluster 0 3.19 scrub starts 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: cluster 2024-03-11T21:29:20.448103+0000 osd.1 (osd.1) 142 : cluster 0 3.19 scrub ok 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: cluster 2024-03-11T21:29:21.204992+0000 mgr.y (mgr.14550) 247 : cluster 0 pgmap v232: 656 pgs: 96 unknown, 76 creating+peering, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:21.877876+0000 mon.b (mon.1) 64 : audit 1 from='client.? 172.21.15.143:0/3463061929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.122257+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126599+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi143-72329-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:23.504 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126683+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71929-6", "mode": "writeback"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126743+0000 mon.a (mon.0) 1165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi143-71641-10"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126796+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126847+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-4"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126900+0000 mon.a (mon.0) 1168 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.126951+0000 mon.a (mon.0) 1169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.149174+0000 mon.b (mon.1) 65 : audit 1 from='client.? 172.21.15.143:0/3291721753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.157958+0000 mon.b (mon.1) 66 : audit 1 from='client.? 172.21.15.143:0/232821597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.158662+0000 mon.b (mon.1) 67 : audit 1 from='client.? 172.21.15.143:0/1903185492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.163425+0000 mon.b (mon.1) 68 : audit 1 from='client.? 172.21.15.143:0/1307145598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.166927+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.143:0/756023775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.172898+0000 mon.b (mon.1) 69 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.173309+0000 mon.b (mon.1) 70 : audit 1 from='client.? 172.21.15.143:0/3628656352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.260205+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:23.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: cluster 2024-03-11T21:29:22.740516+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: cluster 2024-03-11T21:29:22.755483+0000 mon.a (mon.0) 1171 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.778497+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]: dispatch 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.786135+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]: dispatch 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.787211+0000 mon.a (mon.0) 1174 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]: dispatch 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.787622+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]: dispatch 2024-03-11T21:29:23.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:23 smithi143 bash[29284]: audit 2024-03-11T21:29:22.788110+0000 mon.a (mon.0) 1176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:23.566 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:23.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:23.702 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:23.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:23.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:23.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.954 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:24.036 INFO:tasks.thrashosds.thrasher:j is {'epoch': 164, 'fsid': 'eebe2e36-dfeb-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:12:47.384719+0000', 'modified': '2024-03-11T21:29:20.127552+0000', 'last_up_change': '2024-03-11T21:22:21.063566+0000', 'last_in_change': '2024-03-11T21:21:55.456354+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 102, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:18:08.036690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:23:01.432928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:23:03.441476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:23:04.441432+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:23:05.388543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:23:07.642696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-11T21:28:36.978109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'LibRadosList_smithi143-71636-1', 'create_time': '2024-03-11T21:28:38.230956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'LibRadosMiscPP_smithi143-71756-1', 'create_time': '2024-03-11T21:28:38.643455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'LibRadosTierPP_smithi143-71773-1', 'create_time': '2024-03-11T21:28:40.754537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 39, 'pool_name': 'SnapListsmithi143-72399-1', 'create_time': '2024-03-11T21:28:49.753089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 52, 'pool_name': 'PoolQuotaPP_smithi143-71604-2', 'create_time': '2024-03-11T21:28:56.015048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 57, 'pool_name': 'test-rados-api-smithi143-71773-4', 'create_time': '2024-03-11T21:28:58.311890+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '161', 'last_force_op_resend_prenautilus': '161', 'last_force_op_resend_preluminous': '161', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 160, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [76], 'tier_of': -1, 'read_tier': 76, 'write_tier': 76, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 69, 'pool_name': 'LibRadosMisc_smithi143-71741-1', 'create_time': '2024-03-11T21:29:03.594147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'app1': {'key2': 'value2'}, 'app2': {}, 'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 76, 'pool_name': 'test-rados-api-smithi143-71773-4-cache', 'create_time': '2024-03-11T21:29:07.281070+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '161', 'last_force_op_resend_prenautilus': '161', 'last_force_op_resend_preluminous': '161', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 160, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 57, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 89, 'pool_name': 'LibRadosSnapshotsSelfManaged_smithi143-71818-6', 'create_time': '2024-03-11T21:29:13.028596+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 90, 'pool_name': 'LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4', 'create_time': '2024-03-11T21:29:13.507862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '164', 'last_force_op_resend_prenautilus': '164', 'last_force_op_resend_preluminous': '164', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [97], 'tier_of': -1, 'read_tier': 97, 'write_tier': 97, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 96, 'pool_name': 'test-rados-api-smithi143-71794-5', 'create_time': '2024-03-11T21:29:16.260147+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 97, 'pool_name': 'test-rados-api-smithi143-71929-6', 'create_time': '2024-03-11T21:29:16.802774+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '164', 'last_force_op_resend_prenautilus': '164', 'last_force_op_resend_preluminous': '164', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 90, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 98, 'pool_name': 'LibRadosWatchNotifyEC_smithi143-71918-12', 'create_time': '2024-03-11T21:29:17.359374+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosWatchNotifyEC_smithi143-71918-12', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 99, 'pool_name': 'LibRadosSnapshotsSelfManagedPP_smithi143-71838-7', 'create_time': '2024-03-11T21:29:17.886593+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 100, 'pool_name': 'SparseReadsmithi143-72205-4', 'create_time': '2024-03-11T21:29:18.459721+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 101, 'pool_name': 'LongNamespacesmithi143-72351-5', 'create_time': '2024-03-11T21:29:19.010861+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 102, 'pool_name': 'Readsmithi143-72391-5', 'create_time': '2024-03-11T21:29:19.689743+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6802', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6803', 'nonce': 2903432023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6804', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6805', 'nonce': 2903432023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6808', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6809', 'nonce': 2903432023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6806', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6807', 'nonce': 2903432023}]}, 'public_addr': '172.21.15.143:6803/2903432023', 'cluster_addr': '172.21.15.143:6805/2903432023', 'heartbeat_back_addr': '172.21.15.143:6809/2903432023', 'heartbeat_front_addr': '172.21.15.143:6807/2903432023', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2948dc44-68d3-4b3d-92bf-2b0c46374f02', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6810', 'nonce': 397942214}, {'type': 'v1', 'addr': '172.21.15.143:6811', 'nonce': 397942214}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6812', 'nonce': 397942214}, {'type': 'v1', 'addr': '172.21.15.143:6813', 'nonce': 397942214}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6816', 'nonce': 397942214}, {'type': 'v1', 'addr': '172.21.15.143:6817', 'nonce': 397942214}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6814', 'nonce': 397942214}, {'type': 'v1', 'addr': '172.21.15.143:6815', 'nonce': 397942214}]}, 'public_addr': '172.21.15.143:6811/397942214', 'cluster_addr': '172.21.15.143:6813/397942214', 'heartbeat_back_addr': '172.21.15.143:6817/397942214', 'heartbeat_front_addr': '172.21.15.143:6815/397942214', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7217589a-5835-407b-91a7-f5c055a48f3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6818', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6819', 'nonce': 2592131625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6820', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6821', 'nonce': 2592131625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6824', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6825', 'nonce': 2592131625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6822', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6823', 'nonce': 2592131625}]}, 'public_addr': '172.21.15.143:6819/2592131625', 'cluster_addr': '172.21.15.143:6821/2592131625', 'heartbeat_back_addr': '172.21.15.143:6825/2592131625', 'heartbeat_front_addr': '172.21.15.143:6823/2592131625', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a3e7177f-a5a0-44ad-a65d-9aeac3a680cc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6826', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6827', 'nonce': 2146571980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6828', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6829', 'nonce': 2146571980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2146571980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6830', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6831', 'nonce': 2146571980}]}, 'public_addr': '172.21.15.143:6827/2146571980', 'cluster_addr': '172.21.15.143:6829/2146571980', 'heartbeat_back_addr': '172.21.15.143:6833/2146571980', 'heartbeat_front_addr': '172.21.15.143:6831/2146571980', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd7b9e088-d377-4335-8863-11d581828f7e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6800', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 2090835691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6802', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 2090835691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6806', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 2090835691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6804', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 2090835691}]}, 'public_addr': '172.21.15.203:6801/2090835691', 'cluster_addr': '172.21.15.203:6803/2090835691', 'heartbeat_back_addr': '172.21.15.203:6807/2090835691', 'heartbeat_front_addr': '172.21.15.203:6805/2090835691', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b3b2a75c-68a5-4e00-8068-622e1ebbaa01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6808', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 2432038345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6810', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 2432038345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6814', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2432038345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6812', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2432038345}]}, 'public_addr': '172.21.15.203:6809/2432038345', 'cluster_addr': '172.21.15.203:6811/2432038345', 'heartbeat_back_addr': '172.21.15.203:6815/2432038345', 'heartbeat_front_addr': '172.21.15.203:6813/2432038345', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ceffb234-db0c-4ffa-a142-9b201df9b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6816', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6817', 'nonce': 513748834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6818', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6819', 'nonce': 513748834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6822', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6823', 'nonce': 513748834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6820', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6821', 'nonce': 513748834}]}, 'public_addr': '172.21.15.203:6817/513748834', 'cluster_addr': '172.21.15.203:6819/513748834', 'heartbeat_back_addr': '172.21.15.203:6823/513748834', 'heartbeat_front_addr': '172.21.15.203:6821/513748834', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '88b53171-69d2-4922-a787-e63d44d01c60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6824', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6825', 'nonce': 1251497952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6826', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6827', 'nonce': 1251497952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6830', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6831', 'nonce': 1251497952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6828', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6829', 'nonce': 1251497952}]}, 'public_addr': '172.21.15.203:6825/1251497952', 'cluster_addr': '172.21.15.203:6827/1251497952', 'heartbeat_back_addr': '172.21.15.203:6831/1251497952', 'heartbeat_front_addr': '172.21.15.203:6829/1251497952', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:57.552126+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:06.790655+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:25.751047+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:21.177688+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:49.516058+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:23.191241+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:52.553316+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:18.149979+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.143:6801/3638739149': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4274199357': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/3217951344': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/3912416566': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/906834629': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2791822486': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2190104670': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4171894415': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2845155916': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2913593434': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/143435387': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2930358289': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6800/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3638739149': '2024-03-12T21:23:57.022184+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosIoEC_smithi143-71612-16': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-LibRadosLockECPP_smithi143-71649-10': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-LibRadosWatchNotifyEC_smithi143-71918-12': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-ListObjectsNSsmithi143-72329-2': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-ReadOpsmithi143-72277-2': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 28, 'snaps': [{'begin': 5, 'length': 2}]}, {'pool': 35, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 89, 'snaps': [{'begin': 3, 'length': 1}]}], 'new_removed_snaps': [{'pool': 89, 'snaps': [{'begin': 3, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:29:24.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:29:24.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:29:24.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:24.317 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:24.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:24.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:24.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- 172.21.15.143:0/561399310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54108de0 msgr2=0x7f9c54109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/561399310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54108de0 0x7f9c54109260 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9c440098f0 tx=0x7f9c44009c00 comp rx=0 tx=0).stop 2024-03-11T21:29:24.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- 172.21.15.143:0/561399310 shutdown_connections 2024-03-11T21:29:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/561399310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54108de0 0x7f9c54109260 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/561399310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c54110580 0x7f9c5410e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/561399310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5410e800 0x7f9c5410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- 172.21.15.143:0/561399310 >> 172.21.15.143:0/561399310 conn(0x7f9c5406cf60 msgr2=0x7f9c5406d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- 172.21.15.143:0/561399310 shutdown_connections 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- 172.21.15.143:0/561399310 wait complete. 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 librados: starting msgr at 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 librados: starting objecter 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 Processor -- start 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- start start 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 librados: setting wanted keys 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 librados: calling monclient init 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54110580 0x7f9c541acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541a6800 0x7f9c541a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541205f0 con 0x7f9c54110580 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c54120470 con 0x7f9c5410e800 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.544+0000 7f9c5a25f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c54120770 con 0x7f9c541a6800 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.143:42694/0 (socket says 172.21.15.143:42694) 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 -- 172.21.15.143:0/4249843099 learned_addr learned my addr 172.21.15.143:0/4249843099 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541a6800 msgr2=0x7f9c541a6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541a6800 0x7f9c541a6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54110580 msgr2=0x7f9c541acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c54110580 0x7f9c541acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.548+0000 7f9c537fe700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c541a73c0 con 0x7f9c5410e800 2024-03-11T21:29:24.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800b530 tx=0x7f9c4800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c50ff9700 10 client.?.objecter ms_handle_connect 0x7f9c5410e800 2024-03-11T21:29:24.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c50ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48015020 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9c48011cb0 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.552+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48010d20 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c54110da0 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c5a25f700 10 client.30664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c54111290 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c5a25f700 1 librados: init done 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9c5406e500 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.556+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9c48018020 con 0x7f9c5410e800 2024-03-11T21:29:24.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 0x7f9c3c071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(166..166 src has 87..166) v4 ==== 15379+0+0 (secure 0 0 0) 0x7f9c480908e0 con 0x7f9c5410e800 2024-03-11T21:29:24.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 osd_map(166..166 src has 87..166) v4 2024-03-11T21:29:24.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 3 client.30664.objecter handle_osd_map got epochs [166,166] > 0 2024-03-11T21:29:24.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 3 client.30664.objecter handle_osd_map decoding full epoch 166 2024-03-11T21:29:24.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.560+0000 7f9c50ff9700 20 client.30664.objecter dump_active .. 0 homeless 2024-03-11T21:29:24.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.564+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 0x7f9c3c071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.568+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 0x7f9c3c071db0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9c5410e7b0 tx=0x7f9c4c00b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.568+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c3c06f8f0 2024-03-11T21:29:24.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:24.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v166) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9c4801f020 con 0x7f9c5410e800 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 1 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c54062f00 0x7f9c54114740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 1 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 0 1 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 4 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 1 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c541a7d30 con 0x7f9c54062f00 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.572+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c54062f00 0x7f9c54114740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.553 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c54062f00 0x7f9c54114740 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c54062f00 2024-03-11T21:29:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c541a7d30 con 0x7f9c54062f00 2024-03-11T21:29:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:29:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 1 = system:0 2024-03-11T21:29:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 4 2024-03-11T21:29:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.576+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 0 1 2024-03-11T21:29:24.589 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:24.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 2 2024-03-11T21:29:24.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 2 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c5404faa0 osd=0 3 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 0 2 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 4 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 2 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c54062470 con 0x7f9c54062f00 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c54062470 con 0x7f9c54062f00 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 4 2024-03-11T21:29:24.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.668+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 0 2 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 3 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c54115e00 0x7f9c54118230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 3 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 1 3 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 4 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 3 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c54112400 con 0x7f9c54115e00 2024-03-11T21:29:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c54115e00 0x7f9c54118230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c54115e00 0x7f9c54118230 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c54115e00 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c54112400 con 0x7f9c54115e00 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 3 = system:0 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 4 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.672+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 1 3 2024-03-11T21:29:24.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.272102+0000 mon.c (mon.2) 220 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: cluster 2024-03-11T21:29:23.328606+0000 mon.a (mon.0) 1177 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.330395+0000 mon.a (mon.0) 1178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.342963+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.343056+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.343128+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.343198+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.343274+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.343353+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: cluster 2024-03-11T21:29:23.357415+0000 mon.a (mon.0) 1185 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:23.376956+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:24.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[21512]: audit 2024-03-11T21:29:24.273345+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.272102+0000 mon.c (mon.2) 220 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: cluster 2024-03-11T21:29:23.328606+0000 mon.a (mon.0) 1177 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.330395+0000 mon.a (mon.0) 1178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.342963+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.343056+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.343128+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.343198+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.343274+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.343353+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: cluster 2024-03-11T21:29:23.357415+0000 mon.a (mon.0) 1185 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:23.376956+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:24.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:24 smithi143 bash[29284]: audit 2024-03-11T21:29:24.273345+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 4 2024-03-11T21:29:24.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 4 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c54111d80 osd=1 3 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 1 4 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 4 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 4 2024-03-11T21:29:24.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.708+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c54111800 con 0x7f9c54115e00 2024-03-11T21:29:24.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:24.696 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c54111800 con 0x7f9c54115e00 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 4 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 1 4 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 5 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c54115960 0x7f9c541198f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 5 2024-03-11T21:29:24.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 2 5 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 4 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 5 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c54119e30 con 0x7f9c54115960 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.716+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c54115960 0x7f9c541198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c54115960 0x7f9c541198f0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c54115960 2024-03-11T21:29:24.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c54119e30 con 0x7f9c54115960 2024-03-11T21:29:24.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:29:24.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 5 = system:0 2024-03-11T21:29:24.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 4 2024-03-11T21:29:24.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.720+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 2 5 2024-03-11T21:29:24.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.272102+0000 mon.c (mon.2) 220 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: cluster 2024-03-11T21:29:23.328606+0000 mon.a (mon.0) 1177 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:24.708 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.330395+0000 mon.a (mon.0) 1178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.342963+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.343056+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi143-71649-10"}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.343128+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi143-72277-2"}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.343198+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-4", "tierpool": "test-rados-api-smithi143-71773-4-cache"}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.343274+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-71612-16"}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.343353+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: cluster 2024-03-11T21:29:23.357415+0000 mon.a (mon.0) 1185 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:23.376956+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:24.709 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:24 smithi203 bash[23654]: audit 2024-03-11T21:29:24.273345+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:24.722 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 6 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 6 2024-03-11T21:29:24.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c541125b0 osd=2 3 2024-03-11T21:29:24.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 2 6 2024-03-11T21:29:24.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 4 2024-03-11T21:29:24.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 6 2024-03-11T21:29:24.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c5411a230 con 0x7f9c54115960 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c5411a230 con 0x7f9c54115960 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 4 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.768+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 2 6 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 7 2024-03-11T21:29:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c541c6b80 0x7f9c541c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 7 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c541c6b80 0x7f9c541c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 3 7 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 4 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 7 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c541cbfc0 con 0x7f9c541c6b80 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c541c6b80 0x7f9c541c8f70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c541c6b80 2024-03-11T21:29:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c541cbfc0 con 0x7f9c541c6b80 2024-03-11T21:29:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:29:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 7 = system:0 2024-03-11T21:29:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 4 2024-03-11T21:29:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.772+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 3 7 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 8 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 8 2024-03-11T21:29:24.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c541c6500 osd=3 3 2024-03-11T21:29:24.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 3 8 2024-03-11T21:29:24.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 4 2024-03-11T21:29:24.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 8 2024-03-11T21:29:24.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.800+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c54114f90 con 0x7f9c541c6b80 2024-03-11T21:29:24.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.812+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c54114f90 con 0x7f9c541c6b80 2024-03-11T21:29:24.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.812+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.812+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.812+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 4 2024-03-11T21:29:24.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.812+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 3 8 2024-03-11T21:29:24.792 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 10 client.30664.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign -1 9 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c2c002ff0 0x7f9c2c0054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_remove -1 9 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 20 client.30664.objecter get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign 4 9 2024-03-11T21:29:24.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 4 2024-03-11T21:29:24.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 10 client.30664.objecter _send_command 9 2024-03-11T21:29:24.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c2c008530 con 0x7f9c2c002ff0 2024-03-11T21:29:24.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.816+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c2c002ff0 0x7f9c2c0054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.824+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c2c002ff0 0x7f9c2c0054b0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.824+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c2c002ff0 2024-03-11T21:29:24.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.832+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c2c008530 con 0x7f9c2c002ff0 2024-03-11T21:29:24.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.832+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:29:24.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.832+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 9 = system:0 2024-03-11T21:29:24.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.832+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 4 2024-03-11T21:29:24.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.832+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 4 9 2024-03-11T21:29:24.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:24.839 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:29:24.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 10 client.30664.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign -1 10 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_remove -1 10 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 20 client.30664.objecter get_session s=0x7f9c2c0028e0 osd=4 3 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign 4 10 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 4 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 10 client.30664.objecter _send_command 10 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.852+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c2c001700 con 0x7f9c2c002ff0 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c2c001700 con 0x7f9c2c002ff0 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 4 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 4 10 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 10 client.30664.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign -1 11 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c2c0061b0 0x7f9c2c006660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_remove -1 11 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 20 client.30664.objecter get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign 5 11 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 4 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 10 client.30664.objecter _send_command 11 2024-03-11T21:29:24.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.856+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c2c00b8f0 con 0x7f9c2c0061b0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.860+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c2c0061b0 0x7f9c2c006660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.860+0000 7f9c52ffd700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c2c0061b0 0x7f9c2c006660 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.860+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c2c0061b0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.864+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c2c00b8f0 con 0x7f9c2c0061b0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.864+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.864+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 11 = system:0 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.864+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 4 2024-03-11T21:29:24.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.864+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 5 11 2024-03-11T21:29:24.865 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 10 client.30664.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign -1 12 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 20 client.30664.objecter _get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_remove -1 12 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 20 client.30664.objecter get_session s=0x7f9c2c005b30 osd=5 3 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 15 client.30664.objecter _session_command_op_assign 5 12 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 4 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 10 client.30664.objecter _send_command 12 2024-03-11T21:29:24.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.900+0000 7f9c5a25f700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c2c001390 con 0x7f9c2c0061b0 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.908+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c2c001390 con 0x7f9c2c0061b0 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.908+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.908+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.908+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 4 2024-03-11T21:29:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.908+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 5 12 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 13 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c2c006d40 0x7f9c2c0071f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 13 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 6 13 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 4 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 13 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c2c00b6a0 con 0x7f9c2c006d40 2024-03-11T21:29:24.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.912+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c2c006d40 0x7f9c2c0071f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.916+0000 7f9c53fff700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c2c006d40 0x7f9c2c0071f0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.916+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c2c006d40 2024-03-11T21:29:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.924+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c4401d040 con 0x7f9c2c006d40 2024-03-11T21:29:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.924+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:29:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.924+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 13 = system:0 2024-03-11T21:29:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.924+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 4 2024-03-11T21:29:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.924+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 6 13 2024-03-11T21:29:24.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 14 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 14 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c2c00b920 osd=6 3 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 6 14 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 4 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 14 2024-03-11T21:29:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.960+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c2c0077c0 con 0x7f9c2c006d40 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c4401d020 con 0x7f9c2c006d40 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 4 2024-03-11T21:29:24.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 6 14 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 15 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c2c00f2e0 0x7f9c2c0116d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 15 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 7 15 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 4 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.964+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 15 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c2c0082f0 con 0x7f9c2c00f2e0 2024-03-11T21:29:24.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c2c00f2e0 0x7f9c2c0116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:24.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c537fe700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c2c00f2e0 0x7f9c2c0116d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:24.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 10 client.30664.objecter ms_handle_connect 0x7f9c2c00f2e0 2024-03-11T21:29:24.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c2c0082f0 con 0x7f9c2c00f2e0 2024-03-11T21:29:24.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:29:24.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 15 = system:0 2024-03-11T21:29:24.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 4 2024-03-11T21:29:24.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:24.968+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 7 15 2024-03-11T21:29:24.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 10 client.30664.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign -1 16 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 20 client.30664.objecter _get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_remove -1 16 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 20 client.30664.objecter get_session s=0x7f9c2c007a50 osd=7 3 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 15 client.30664.objecter _session_command_op_assign 7 16 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 4 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 10 client.30664.objecter _send_command 16 2024-03-11T21:29:24.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.004+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9c2c011c10 con 0x7f9c2c00f2e0 2024-03-11T21:29:24.989 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c50ff9700 1 -- 172.21.15.143:0/4249843099 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9c2c011c10 con 0x7f9c2c00f2e0 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c50ff9700 10 client.30664.objecter ms_dispatch 0x7f9c54001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c50ff9700 10 client.30664.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c50ff9700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 4 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c50ff9700 15 client.30664.objecter _session_command_op_remove 7 16 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 librados: watch_flush enter 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 librados: watch_flush exit 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.0 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c54062f00 msgr2=0x7f9c54114740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9c54062f00 0x7f9c54114740 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c5404faa0 osd=0 1 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.1 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c54115e00 msgr2=0x7f9c54118230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f9c54115e00 0x7f9c54118230 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c54111d80 osd=1 1 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.2 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c54115960 msgr2=0x7f9c541198f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9c54115960 0x7f9c541198f0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c53fff700 1 -- 172.21.15.143:0/4249843099 reap_dead start 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541125b0 osd=2 1 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.3 2024-03-11T21:29:24.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c541c6b80 msgr2=0x7f9c541c8f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9c541c6b80 0x7f9c541c8f70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c541c6500 osd=3 1 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.4 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c2c002ff0 msgr2=0x7f9c2c0054b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9c2c002ff0 0x7f9c2c0054b0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c0028e0 osd=4 1 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.5 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c2c0061b0 msgr2=0x7f9c2c006660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9c2c0061b0 0x7f9c2c006660 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c005b30 osd=5 1 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.6 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c2c006d40 msgr2=0x7f9c2c0071f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9c2c006d40 0x7f9c2c0071f0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c00b920 osd=6 1 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter close_session for osd.7 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c2c00f2e0 msgr2=0x7f9c2c0116d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9c2c00f2e0 0x7f9c2c0116d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c53fff700 1 -- 172.21.15.143:0/4249843099 reap_dead start 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter put_session s=0x7f9c2c007a50 osd=7 1 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 20 client.30664.objecter shutdown clearing up homeless session... 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 10 client.30664.objecter successfully canceled tick 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 msgr2=0x7f9c3c071db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.008+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 0x7f9c3c071db0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9c5410e7b0 tx=0x7f9c4c00b040 comp rx=0 tx=0).stop 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 msgr2=0x7f9c541ac680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:24.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800b530 tx=0x7f9c4800b8f0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 shutdown_connections 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c5410e800 0x7f9c541ac680 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 --2- 172.21.15.143:0/4249843099 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c3c06f8f0 0x7f9c3c071db0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 >> 172.21.15.143:0/4249843099 conn(0x7f9c5406cf60 msgr2=0x7f9c5410bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 shutdown_connections 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 -- 172.21.15.143:0/4249843099 wait complete. 2024-03-11T21:29:24.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:25.012+0000 7f9c3a7fc700 1 librados: shutdown 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:25.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:25.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:25.333 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (10535 ms) 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-11T21:29:25.425 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (23 ms) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (10558 ms total) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (95 ms) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (6377 ms) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (16 ms) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-03-11T21:29:25.426 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721927856 notify_id 708669603843 notifier_gid 25400 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (8 ms) 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721927856 notify_id 708669603844 notifier_gid 25400 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (14 ms) 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721927808 notify_id 708669603842 notifier_gid 25400 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (7 ms) 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721912352 notify_id 708669603842 notifier_gid 25400 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (46 ms) 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721912352 notify_id 708669603845 notifier_gid 25400 2024-03-11T21:29:25.427 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (11 ms) 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721912352 notify_id 708669603846 notifier_gid 25400 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (25 ms) 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: trying... 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721912352 notify_id 708669603847 notifier_gid 25400 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: timed out 2024-03-11T21:29:25.428 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-03-11T21:29:25.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:25.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:25.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:25.764 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:25.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:25.904 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:22.003271+0000 osd.5 (osd.5) 157 : cluster 0 5.15 scrub starts 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:22.004967+0000 osd.5 (osd.5) 158 : cluster 0 5.15 scrub ok 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:23.207916+0000 mgr.y (mgr.14550) 248 : cluster 0 pgmap v234: 592 pgs: 72 creating+peering, 520 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:24.257195+0000 osd.3 (osd.3) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:24.257271+0000 osd.3 (osd.3) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:24.558972+0000 mon.b (mon.1) 71 : audit 0 from='client.? 172.21.15.143:0/4249843099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:24.582398+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:24.582494+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:24.590257+0000 mon.b (mon.1) 72 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:24.596166+0000 mon.b (mon.1) 73 : audit 1 from='client.? 172.21.15.143:0/3030443100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.113965+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.138480+0000 mon.c (mon.2) 223 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:25.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.151491+0000 mon.c (mon.2) 224 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.164382+0000 mon.c (mon.2) 225 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.176834+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.181540+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.184769+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.188846+0000 mon.c (mon.2) 229 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:22.003271+0000 osd.5 (osd.5) 157 : cluster 0 5.15 scrub starts 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:22.004967+0000 osd.5 (osd.5) 158 : cluster 0 5.15 scrub ok 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:23.207916+0000 mgr.y (mgr.14550) 248 : cluster 0 pgmap v234: 592 pgs: 72 creating+peering, 520 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:24.257195+0000 osd.3 (osd.3) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:24.257271+0000 osd.3 (osd.3) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:25.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:24.558972+0000 mon.b (mon.1) 71 : audit 0 from='client.? 172.21.15.143:0/4249843099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:24.582398+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:24.582494+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:24.590257+0000 mon.b (mon.1) 72 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:24.596166+0000 mon.b (mon.1) 73 : audit 1 from='client.? 172.21.15.143:0/3030443100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.113965+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.138480+0000 mon.c (mon.2) 223 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.151491+0000 mon.c (mon.2) 224 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.164382+0000 mon.c (mon.2) 225 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.176834+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.181540+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.184769+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.188846+0000 mon.c (mon.2) 229 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.191386+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:25.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.195093+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.255921+0000 mon.a (mon.0) 1189 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.267884+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.274745+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[29284]: audit 2024-03-11T21:29:25.277238+0000 mon.c (mon.2) 232 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.191386+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.195093+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.255921+0000 mon.a (mon.0) 1189 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.267884+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.274745+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:25.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:25 smithi143 bash[21512]: audit 2024-03-11T21:29:25.277238+0000 mon.c (mon.2) 232 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:25.954 INFO:tasks.workunit.client.0.smithi143.stdout: a api_watch_notify: Running main() from gmock_main.cc 2024-03-11T21:29:25.954 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-11T21:29:25.954 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-11T21:29:25.954 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-11T21:29:25.954 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify_test_cb 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (2630 ms) 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94059017255248 err -107 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (5 ms) 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94059017266864 err -107 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (5 ms) 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257921 cookie 94059017284672 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6 ms) 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-11T21:29:25.955 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257921 cookie 94059017295952 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257922 cookie 94059017288000 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (4 ms) 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257920 cookie 94059017288000 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257920 cookie 94059017289056 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257921 cookie 94059017288928 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 622770257922 cookie 94059017288928 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3007 ms) 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-11T21:29:25.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94059017290256 err -107 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 24730 notify_id 644245094404 cookie 94059017290256 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5011 ms) 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94059017288928 err -107 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1011 ms) 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11692 ms total) 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify_test_cb 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1881 ms) 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1881 ms total) 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: 2024-03-11T21:29:25.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-11T21:29:25.958 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (50233 ms total) 2024-03-11T21:29:25.958 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-11T21:29:26.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:26.066 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:26.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:22.003271+0000 osd.5 (osd.5) 157 : cluster 0 5.15 scrub starts 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:22.004967+0000 osd.5 (osd.5) 158 : cluster 0 5.15 scrub ok 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:23.207916+0000 mgr.y (mgr.14550) 248 : cluster 0 pgmap v234: 592 pgs: 72 creating+peering, 520 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:24.257195+0000 osd.3 (osd.3) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:24.257271+0000 osd.3 (osd.3) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:24.558972+0000 mon.b (mon.1) 71 : audit 0 from='client.? 172.21.15.143:0/4249843099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:24.582398+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:24.582494+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi143-71594-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:24.590257+0000 mon.b (mon.1) 72 : audit 1 from='client.? 172.21.15.143:0/888575219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:24.596166+0000 mon.b (mon.1) 73 : audit 1 from='client.? 172.21.15.143:0/3030443100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.113965+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.138480+0000 mon.c (mon.2) 223 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:26.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.151491+0000 mon.c (mon.2) 224 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.164382+0000 mon.c (mon.2) 225 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.176834+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.181540+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.184769+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.188846+0000 mon.c (mon.2) 229 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.191386+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.195093+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.255921+0000 mon.a (mon.0) 1189 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.267884+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.274745+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:26.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:25 smithi203 bash[23654]: audit 2024-03-11T21:29:25.277238+0000 mon.c (mon.2) 232 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:26.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:26.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:26.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:26.744 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:26.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:26.906 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:29:26.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:27.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:27.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:27.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.124033+0000 mgr.y (mgr.14550) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:27.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.140159+0000 mgr.y (mgr.14550) 250 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.152029+0000 mgr.y (mgr.14550) 251 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.168029+0000 mgr.y (mgr.14550) 252 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.178730+0000 mgr.y (mgr.14550) 253 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.184013+0000 mgr.y (mgr.14550) 254 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.188028+0000 mgr.y (mgr.14550) 255 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.190828+0000 mgr.y (mgr.14550) 256 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:27.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.193830+0000 mgr.y (mgr.14550) 257 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:27.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:27.195 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:27.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:27.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.124033+0000 mgr.y (mgr.14550) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:27.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.140159+0000 mgr.y (mgr.14550) 250 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:27.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.152029+0000 mgr.y (mgr.14550) 251 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.168029+0000 mgr.y (mgr.14550) 252 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.178730+0000 mgr.y (mgr.14550) 253 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.184013+0000 mgr.y (mgr.14550) 254 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.188028+0000 mgr.y (mgr.14550) 255 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.190828+0000 mgr.y (mgr.14550) 256 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.193830+0000 mgr.y (mgr.14550) 257 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.196242+0000 mgr.y (mgr.14550) 258 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.208992+0000 mgr.y (mgr.14550) 259 : cluster 0 pgmap v237: 688 pgs: 128 unknown, 72 creating+peering, 488 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 3.2 KiB/s wr, 5 op/s 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.701369+0000 osd.0 (osd.0) 145 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.783414+0000 osd.0 (osd.0) 146 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.916192+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.916281+0000 mon.a (mon.0) 1193 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:27.308 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.922540+0000 mon.a (mon.0) 1194 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:29:27.309 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.929792+0000 mon.a (mon.0) 1195 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:27.309 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.954692+0000 mon.b (mon.1) 74 : audit 1 from='client.? 172.21.15.143:0/1328570527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.309 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:25.959453+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.309 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:27 smithi203 bash[23654]: audit 2024-03-11T21:29:26.284289+0000 mon.c (mon.2) 233 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:27.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:27.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.196242+0000 mgr.y (mgr.14550) 258 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.208992+0000 mgr.y (mgr.14550) 259 : cluster 0 pgmap v237: 688 pgs: 128 unknown, 72 creating+peering, 488 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 3.2 KiB/s wr, 5 op/s 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.701369+0000 osd.0 (osd.0) 145 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.783414+0000 osd.0 (osd.0) 146 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.916192+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.916281+0000 mon.a (mon.0) 1193 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.922540+0000 mon.a (mon.0) 1194 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.929792+0000 mon.a (mon.0) 1195 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.954692+0000 mon.b (mon.1) 74 : audit 1 from='client.? 172.21.15.143:0/1328570527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:25.959453+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[21512]: audit 2024-03-11T21:29:26.284289+0000 mon.c (mon.2) 233 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.124033+0000 mgr.y (mgr.14550) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.140159+0000 mgr.y (mgr.14550) 250 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.152029+0000 mgr.y (mgr.14550) 251 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.168029+0000 mgr.y (mgr.14550) 252 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.178730+0000 mgr.y (mgr.14550) 253 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.184013+0000 mgr.y (mgr.14550) 254 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.188028+0000 mgr.y (mgr.14550) 255 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-03-11T21:29:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.190828+0000 mgr.y (mgr.14550) 256 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.193830+0000 mgr.y (mgr.14550) 257 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.196242+0000 mgr.y (mgr.14550) 258 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.208992+0000 mgr.y (mgr.14550) 259 : cluster 0 pgmap v237: 688 pgs: 128 unknown, 72 creating+peering, 488 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 3.2 KiB/s wr, 5 op/s 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.701369+0000 osd.0 (osd.0) 145 : cluster 0 13.3 deep-scrub starts 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.783414+0000 osd.0 (osd.0) 146 : cluster 0 13.3 deep-scrub ok 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.916192+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi143-71918-12"}]': finished 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.916281+0000 mon.a (mon.0) 1193 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.922540+0000 mon.a (mon.0) 1194 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.929792+0000 mon.a (mon.0) 1195 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.954692+0000 mon.b (mon.1) 74 : audit 1 from='client.? 172.21.15.143:0/1328570527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:25.959453+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:27.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:27 smithi143 bash[29284]: audit 2024-03-11T21:29:26.284289+0000 mon.c (mon.2) 233 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:27.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:27.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:27.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:28.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.105 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:28.205 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:29:28.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:28.237 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:28.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:28.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.328754+0000 osd.7 (osd.7) 187 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:28.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.378515+0000 osd.7 (osd.7) 188 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.557848+0000 osd.6 (osd.6) 99 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.566046+0000 osd.6 (osd.6) 100 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.709293+0000 osd.2 (osd.2) 109 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:25.718760+0000 osd.2 (osd.2) 110 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.044864+0000 osd.5 (osd.5) 159 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.046635+0000 osd.5 (osd.5) 160 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.050169+0000 osd.4 (osd.4) 139 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.053577+0000 osd.4 (osd.4) 140 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.371094+0000 osd.7 (osd.7) 189 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.374610+0000 osd.7 (osd.7) 190 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.517316+0000 osd.6 (osd.6) 101 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:26.520600+0000 osd.6 (osd.6) 102 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.034266+0000 osd.4 (osd.4) 141 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:28.370 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.036874+0000 osd.5 (osd.5) 161 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.039500+0000 osd.5 (osd.5) 162 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.082785+0000 osd.4 (osd.4) 142 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:27.157747+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:27.157827+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.229743+0000 mgr.y (mgr.14550) 260 : cluster 0 pgmap v240: 616 pgs: 192 unknown, 424 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:27.291525+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: cluster 2024-03-11T21:29:27.818773+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:27.824728+0000 mon.a (mon.0) 1200 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:27.834786+0000 mon.a (mon.0) 1201 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:28.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[29284]: audit 2024-03-11T21:29:28.316848+0000 mon.c (mon.2) 235 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.328754+0000 osd.7 (osd.7) 187 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.378515+0000 osd.7 (osd.7) 188 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.557848+0000 osd.6 (osd.6) 99 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.566046+0000 osd.6 (osd.6) 100 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.709293+0000 osd.2 (osd.2) 109 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:25.718760+0000 osd.2 (osd.2) 110 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.044864+0000 osd.5 (osd.5) 159 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.046635+0000 osd.5 (osd.5) 160 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.050169+0000 osd.4 (osd.4) 139 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.053577+0000 osd.4 (osd.4) 140 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:28.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.371094+0000 osd.7 (osd.7) 189 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.374610+0000 osd.7 (osd.7) 190 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.517316+0000 osd.6 (osd.6) 101 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:26.520600+0000 osd.6 (osd.6) 102 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.034266+0000 osd.4 (osd.4) 141 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.036874+0000 osd.5 (osd.5) 161 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.039500+0000 osd.5 (osd.5) 162 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.082785+0000 osd.4 (osd.4) 142 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:27.157747+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:27.157827+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.229743+0000 mgr.y (mgr.14550) 260 : cluster 0 pgmap v240: 616 pgs: 192 unknown, 424 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:27.291525+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: cluster 2024-03-11T21:29:27.818773+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:27.824728+0000 mon.a (mon.0) 1200 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:27.834786+0000 mon.a (mon.0) 1201 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:28.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:28 smithi143 bash[21512]: audit 2024-03-11T21:29:28.316848+0000 mon.c (mon.2) 235 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.328754+0000 osd.7 (osd.7) 187 : cluster 0 13.7 deep-scrub starts 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.378515+0000 osd.7 (osd.7) 188 : cluster 0 13.7 deep-scrub ok 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.557848+0000 osd.6 (osd.6) 99 : cluster 0 13.1 deep-scrub starts 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.566046+0000 osd.6 (osd.6) 100 : cluster 0 13.1 deep-scrub ok 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.709293+0000 osd.2 (osd.2) 109 : cluster 0 13.8 deep-scrub starts 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:25.718760+0000 osd.2 (osd.2) 110 : cluster 0 13.8 deep-scrub ok 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.044864+0000 osd.5 (osd.5) 159 : cluster 0 13.9 deep-scrub starts 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.046635+0000 osd.5 (osd.5) 160 : cluster 0 13.9 deep-scrub ok 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.050169+0000 osd.4 (osd.4) 139 : cluster 0 13.5 deep-scrub starts 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.053577+0000 osd.4 (osd.4) 140 : cluster 0 13.5 deep-scrub ok 2024-03-11T21:29:28.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.371094+0000 osd.7 (osd.7) 189 : cluster 0 13.0 deep-scrub starts 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.374610+0000 osd.7 (osd.7) 190 : cluster 0 13.0 deep-scrub ok 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.517316+0000 osd.6 (osd.6) 101 : cluster 0 13.4 deep-scrub starts 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:26.520600+0000 osd.6 (osd.6) 102 : cluster 0 13.4 deep-scrub ok 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.034266+0000 osd.4 (osd.4) 141 : cluster 0 13.6 deep-scrub starts 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.036874+0000 osd.5 (osd.5) 161 : cluster 0 13.2 deep-scrub starts 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.039500+0000 osd.5 (osd.5) 162 : cluster 0 13.2 deep-scrub ok 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.082785+0000 osd.4 (osd.4) 142 : cluster 0 13.6 deep-scrub ok 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:27.157747+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:27.157827+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.229743+0000 mgr.y (mgr.14550) 260 : cluster 0 pgmap v240: 616 pgs: 192 unknown, 424 active+clean; 145 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:27.291525+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: cluster 2024-03-11T21:29:27.818773+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:29:28.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:27.824728+0000 mon.a (mon.0) 1200 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]: dispatch 2024-03-11T21:29:28.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:27.834786+0000 mon.a (mon.0) 1201 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:28.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:28 smithi203 bash[23654]: audit 2024-03-11T21:29:28.316848+0000 mon.c (mon.2) 235 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:28.541 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:28.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.785 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:28.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:29.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:29.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:29.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:29.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:29.293 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:29.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:29.451 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:29.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:29.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:29.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:29.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:29.736 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:29.737 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:29.739 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:29.839 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:29.913 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:30.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:30.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:30.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:30.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: audit 2024-03-11T21:29:28.353522+0000 mgr.y (mgr.14550) 261 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: cluster 2024-03-11T21:29:28.386155+0000 mon.a (mon.0) 1202 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: audit 2024-03-11T21:29:28.404459+0000 mon.a (mon.0) 1203 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: audit 2024-03-11T21:29:28.404625+0000 mon.a (mon.0) 1204 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: cluster 2024-03-11T21:29:28.409542+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: audit 2024-03-11T21:29:28.437942+0000 mon.b (mon.1) 75 : audit 1 from='client.? 172.21.15.143:0/560179910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[21512]: audit 2024-03-11T21:29:29.053166+0000 mon.a (mon.0) 1206 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: audit 2024-03-11T21:29:28.353522+0000 mgr.y (mgr.14550) 261 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: cluster 2024-03-11T21:29:28.386155+0000 mon.a (mon.0) 1202 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: audit 2024-03-11T21:29:28.404459+0000 mon.a (mon.0) 1203 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: audit 2024-03-11T21:29:28.404625+0000 mon.a (mon.0) 1204 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: cluster 2024-03-11T21:29:28.409542+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: audit 2024-03-11T21:29:28.437942+0000 mon.b (mon.1) 75 : audit 1 from='client.? 172.21.15.143:0/560179910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:30 smithi143 bash[29284]: audit 2024-03-11T21:29:29.053166+0000 mon.a (mon.0) 1206 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:30.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: audit 2024-03-11T21:29:28.353522+0000 mgr.y (mgr.14550) 261 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: cluster 2024-03-11T21:29:28.386155+0000 mon.a (mon.0) 1202 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: audit 2024-03-11T21:29:28.404459+0000 mon.a (mon.0) 1203 : audit 1 from='client.? 172.21.15.143:0/1675038819' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi143-72329-2"}]': finished 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: audit 2024-03-11T21:29:28.404625+0000 mon.a (mon.0) 1204 : audit 1 from='client.? 172.21.15.143:0/1918878702' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: cluster 2024-03-11T21:29:28.409542+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: audit 2024-03-11T21:29:28.437942+0000 mon.b (mon.1) 75 : audit 1 from='client.? 172.21.15.143:0/560179910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:30 smithi203 bash[23654]: audit 2024-03-11T21:29:29.053166+0000 mon.a (mon.0) 1206 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:30.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:30.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:30.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:30.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:30.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:31.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:31.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:31.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:31.172 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:31.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:31.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:31.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:31.523 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:31.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[21512]: audit 2024-03-11T21:29:29.074790+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[21512]: cluster 2024-03-11T21:29:29.230442+0000 mgr.y (mgr.14550) 262 : cluster 0 pgmap v243: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 1 op/s 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[21512]: audit 2024-03-11T21:29:29.320129+0000 mon.c (mon.2) 236 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[21512]: cluster 2024-03-11T21:29:29.660977+0000 mon.a (mon.0) 1208 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[21512]: audit 2024-03-11T21:29:30.256725+0000 mon.a (mon.0) 1209 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[29284]: audit 2024-03-11T21:29:29.074790+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[29284]: cluster 2024-03-11T21:29:29.230442+0000 mgr.y (mgr.14550) 262 : cluster 0 pgmap v243: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 1 op/s 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[29284]: audit 2024-03-11T21:29:29.320129+0000 mon.c (mon.2) 236 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:31.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[29284]: cluster 2024-03-11T21:29:29.660977+0000 mon.a (mon.0) 1208 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:29:31.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:31 smithi143 bash[29284]: audit 2024-03-11T21:29:30.256725+0000 mon.a (mon.0) 1209 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:31.714 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:31.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:31 smithi203 bash[23654]: audit 2024-03-11T21:29:29.074790+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi143-71594-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:31.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:31 smithi203 bash[23654]: cluster 2024-03-11T21:29:29.230442+0000 mgr.y (mgr.14550) 262 : cluster 0 pgmap v243: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 1 op/s 2024-03-11T21:29:31.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:31 smithi203 bash[23654]: audit 2024-03-11T21:29:29.320129+0000 mon.c (mon.2) 236 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:31.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:31 smithi203 bash[23654]: cluster 2024-03-11T21:29:29.660977+0000 mon.a (mon.0) 1208 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:29:31.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:31 smithi203 bash[23654]: audit 2024-03-11T21:29:30.256725+0000 mon.a (mon.0) 1209 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:31.775 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:31.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:31.899 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:32.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:32.034 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:32.078 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:32.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:32.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:32.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:32.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:32.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:32.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:32.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:32.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:30.280322+0000 mon.c (mon.2) 237 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:30.321960+0000 mon.c (mon.2) 238 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:31.324170+0000 mon.c (mon.2) 239 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: cluster 2024-03-11T21:29:31.361219+0000 mon.a (mon.0) 1210 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:31.392705+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: cluster 2024-03-11T21:29:31.397086+0000 mon.a (mon.0) 1212 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:31.428344+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:31.514620+0000 mon.b (mon.1) 76 : audit 1 from='client.? 172.21.15.143:0/1840095906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[29284]: audit 2024-03-11T21:29:32.325298+0000 mon.c (mon.2) 240 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:30.280322+0000 mon.c (mon.2) 237 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:30.321960+0000 mon.c (mon.2) 238 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:31.324170+0000 mon.c (mon.2) 239 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: cluster 2024-03-11T21:29:31.361219+0000 mon.a (mon.0) 1210 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:31.392705+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: cluster 2024-03-11T21:29:31.397086+0000 mon.a (mon.0) 1212 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:31.428344+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:31.514620+0000 mon.b (mon.1) 76 : audit 1 from='client.? 172.21.15.143:0/1840095906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:32 smithi143 bash[21512]: audit 2024-03-11T21:29:32.325298+0000 mon.c (mon.2) 240 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:30.280322+0000 mon.c (mon.2) 237 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:30.321960+0000 mon.c (mon.2) 238 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:31.324170+0000 mon.c (mon.2) 239 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: cluster 2024-03-11T21:29:31.361219+0000 mon.a (mon.0) 1210 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:31.392705+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi143-72329-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: cluster 2024-03-11T21:29:31.397086+0000 mon.a (mon.0) 1212 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:29:32.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:31.428344+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:31.514620+0000 mon.b (mon.1) 76 : audit 1 from='client.? 172.21.15.143:0/1840095906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:32.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:32 smithi203 bash[23654]: audit 2024-03-11T21:29:32.325298+0000 mon.c (mon.2) 240 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:32.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:32.797 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:32.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.157 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:33.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:33.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:33.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[21512]: cluster 2024-03-11T21:29:31.231179+0000 mgr.y (mgr.14550) 263 : cluster 0 pgmap v245: 552 pgs: 1 active+clean+snaptrim, 160 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-11T21:29:33.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[21512]: audit 2024-03-11T21:29:32.553586+0000 mon.a (mon.0) 1214 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:33.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[21512]: audit 2024-03-11T21:29:32.571778+0000 mon.a (mon.0) 1215 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[29284]: cluster 2024-03-11T21:29:31.231179+0000 mgr.y (mgr.14550) 263 : cluster 0 pgmap v245: 552 pgs: 1 active+clean+snaptrim, 160 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-11T21:29:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[29284]: audit 2024-03-11T21:29:32.553586+0000 mon.a (mon.0) 1214 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:33.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:33 smithi143 bash[29284]: audit 2024-03-11T21:29:32.571778+0000 mon.a (mon.0) 1215 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:33 smithi203 bash[23654]: cluster 2024-03-11T21:29:31.231179+0000 mgr.y (mgr.14550) 263 : cluster 0 pgmap v245: 552 pgs: 1 active+clean+snaptrim, 160 unknown, 391 active+clean; 217 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 2 op/s 2024-03-11T21:29:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:33 smithi203 bash[23654]: audit 2024-03-11T21:29:32.553586+0000 mon.a (mon.0) 1214 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:33 smithi203 bash[23654]: audit 2024-03-11T21:29:32.571778+0000 mon.a (mon.0) 1215 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:34.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:34.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:34.316 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:34.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.446 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:34.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:34.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:34.799 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:34.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.930 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:35.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:35.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:35.173 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:35.224 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:35.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:35.362 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:35.380 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[21512]: audit 2024-03-11T21:29:32.572269+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:35.380 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[21512]: audit 2024-03-11T21:29:33.331740+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:35.380 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[21512]: audit 2024-03-11T21:29:33.572613+0000 mon.c (mon.2) 242 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:35.380 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[21512]: cluster 2024-03-11T21:29:33.682833+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:29:35.475 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:35.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:35.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:35.645 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[29284]: audit 2024-03-11T21:29:32.572269+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:35.645 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[29284]: audit 2024-03-11T21:29:33.331740+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:35.645 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[29284]: audit 2024-03-11T21:29:33.572613+0000 mon.c (mon.2) 242 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:35.645 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:35 smithi143 bash[29284]: cluster 2024-03-11T21:29:33.682833+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:29:35.669 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:35.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:35 smithi203 bash[23654]: audit 2024-03-11T21:29:32.572269+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:35.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:35 smithi203 bash[23654]: audit 2024-03-11T21:29:33.331740+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:35.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:35 smithi203 bash[23654]: audit 2024-03-11T21:29:33.572613+0000 mon.c (mon.2) 242 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:35.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:35 smithi203 bash[23654]: cluster 2024-03-11T21:29:33.682833+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:29:35.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:35.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:36.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:36.080 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:36.183 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:36.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:36.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:36.391 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:36.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:36.619 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:36.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:36.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:36.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:36.982 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:37.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:31.801788+0000 osd.2 (osd.2) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:31.801877+0000 osd.2 (osd.2) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:32.654048+0000 osd.5 (osd.5) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:32.654107+0000 osd.5 (osd.5) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:33.233545+0000 mgr.y (mgr.14550) 264 : cluster 0 pgmap v248: 584 pgs: 64 creating+peering, 24 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:33.542976+0000 osd.1 (osd.1) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:33.543071+0000 osd.1 (osd.1) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:33.741905+0000 mon.a (mon.0) 1218 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:33.742002+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:33.754502+0000 mon.b (mon.1) 77 : audit 1 from='client.? 172.21.15.143:0/1963926694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:34.353717+0000 mon.c (mon.2) 243 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:34.620126+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.215601+0000 mon.c (mon.2) 245 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.222773+0000 mon.c (mon.2) 246 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:35.234460+0000 mgr.y (mgr.14550) 265 : cluster 0 pgmap v250: 624 pgs: 64 creating+peering, 64 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:35.351215+0000 mon.a (mon.0) 1220 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.355642+0000 mon.c (mon.2) 247 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:35.371070+0000 mon.a (mon.0) 1221 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.395241+0000 mon.a (mon.0) 1222 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.419297+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.466897+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.500795+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.510190+0000 mon.a (mon.0) 1225 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.514725+0000 mon.a (mon.0) 1226 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.516351+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.517513+0000 mon.c (mon.2) 249 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:35.520588+0000 mon.a (mon.0) 1228 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.361427+0000 mon.c (mon.2) 250 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.370470+0000 mon.a (mon.0) 1229 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.370562+0000 mon.a (mon.0) 1230 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.370642+0000 mon.a (mon.0) 1231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]': finished 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.370707+0000 mon.a (mon.0) 1232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.230 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.370779+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: cluster 2024-03-11T21:29:36.378754+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[29284]: audit 2024-03-11T21:29:36.397377+0000 mon.a (mon.0) 1235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:31.801788+0000 osd.2 (osd.2) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:31.801877+0000 osd.2 (osd.2) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:32.654048+0000 osd.5 (osd.5) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:32.654107+0000 osd.5 (osd.5) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:33.233545+0000 mgr.y (mgr.14550) 264 : cluster 0 pgmap v248: 584 pgs: 64 creating+peering, 24 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-03-11T21:29:37.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:33.542976+0000 osd.1 (osd.1) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:33.543071+0000 osd.1 (osd.1) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:33.741905+0000 mon.a (mon.0) 1218 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:33.742002+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:33.754502+0000 mon.b (mon.1) 77 : audit 1 from='client.? 172.21.15.143:0/1963926694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:34.353717+0000 mon.c (mon.2) 243 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:34.620126+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.215601+0000 mon.c (mon.2) 245 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.222773+0000 mon.c (mon.2) 246 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:37.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:35.234460+0000 mgr.y (mgr.14550) 265 : cluster 0 pgmap v250: 624 pgs: 64 creating+peering, 64 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:35.351215+0000 mon.a (mon.0) 1220 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.355642+0000 mon.c (mon.2) 247 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:35.371070+0000 mon.a (mon.0) 1221 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.395241+0000 mon.a (mon.0) 1222 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.419297+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.466897+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.500795+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.510190+0000 mon.a (mon.0) 1225 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.514725+0000 mon.a (mon.0) 1226 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.516351+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.517513+0000 mon.c (mon.2) 249 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:35.520588+0000 mon.a (mon.0) 1228 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.361427+0000 mon.c (mon.2) 250 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.370470+0000 mon.a (mon.0) 1229 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.370562+0000 mon.a (mon.0) 1230 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.370642+0000 mon.a (mon.0) 1231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]': finished 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.370707+0000 mon.a (mon.0) 1232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.370779+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: cluster 2024-03-11T21:29:36.378754+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:29:37.234 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:37 smithi143 bash[21512]: audit 2024-03-11T21:29:36.397377+0000 mon.a (mon.0) 1235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:37.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.329 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:37.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:31.801788+0000 osd.2 (osd.2) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:31.801877+0000 osd.2 (osd.2) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:32.654048+0000 osd.5 (osd.5) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:32.654107+0000 osd.5 (osd.5) 164 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:33.233545+0000 mgr.y (mgr.14550) 264 : cluster 0 pgmap v248: 584 pgs: 64 creating+peering, 24 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:33.542976+0000 osd.1 (osd.1) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:33.543071+0000 osd.1 (osd.1) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:33.741905+0000 mon.a (mon.0) 1218 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi143-72329-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:33.742002+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi143-71604-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:33.754502+0000 mon.b (mon.1) 77 : audit 1 from='client.? 172.21.15.143:0/1963926694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:34.353717+0000 mon.c (mon.2) 243 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.449 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:34.620126+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.215601+0000 mon.c (mon.2) 245 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.222773+0000 mon.c (mon.2) 246 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:35.234460+0000 mgr.y (mgr.14550) 265 : cluster 0 pgmap v250: 624 pgs: 64 creating+peering, 64 unknown, 496 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:35.351215+0000 mon.a (mon.0) 1220 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.355642+0000 mon.c (mon.2) 247 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:35.371070+0000 mon.a (mon.0) 1221 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.395241+0000 mon.a (mon.0) 1222 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.419297+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.466897+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.500795+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]: dispatch 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.510190+0000 mon.a (mon.0) 1225 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.514725+0000 mon.a (mon.0) 1226 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:37.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.516351+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.517513+0000 mon.c (mon.2) 249 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:35.520588+0000 mon.a (mon.0) 1228 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.361427+0000 mon.c (mon.2) 250 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.370470+0000 mon.a (mon.0) 1229 : audit 1 from='client.? 172.21.15.143:0/484288227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi143-71594-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.370562+0000 mon.a (mon.0) 1230 : audit 1 from='client.? 172.21.15.143:0/1545131348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-72017-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.370642+0000 mon.a (mon.0) 1231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4"}]': finished 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.370707+0000 mon.a (mon.0) 1232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.370779+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: cluster 2024-03-11T21:29:36.378754+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:29:37.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:37 smithi203 bash[23654]: audit 2024-03-11T21:29:36.397377+0000 mon.a (mon.0) 1235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:37.461 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:37.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.736 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:37.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:37.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:38.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:38.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:38.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:38.368 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: audit 2024-03-11T21:29:36.394846+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: audit 2024-03-11T21:29:36.613677+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: cluster 2024-03-11T21:29:37.097649+0000 mon.a (mon.0) 1236 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: audit 2024-03-11T21:29:37.139372+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: audit 2024-03-11T21:29:37.362736+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:38.392 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[21512]: cluster 2024-03-11T21:29:37.714638+0000 mon.a (mon.0) 1238 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: audit 2024-03-11T21:29:36.394846+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: audit 2024-03-11T21:29:36.613677+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: cluster 2024-03-11T21:29:37.097649+0000 mon.a (mon.0) 1236 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: audit 2024-03-11T21:29:37.139372+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: audit 2024-03-11T21:29:37.362736+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:38.393 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:38 smithi143 bash[29284]: cluster 2024-03-11T21:29:37.714638+0000 mon.a (mon.0) 1238 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:38.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:38.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: audit 2024-03-11T21:29:36.394846+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]: dispatch 2024-03-11T21:29:38.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: audit 2024-03-11T21:29:36.613677+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: cluster 2024-03-11T21:29:37.097649+0000 mon.a (mon.0) 1236 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:38.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: audit 2024-03-11T21:29:37.139372+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:38.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: audit 2024-03-11T21:29:37.362736+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:38.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[23654]: cluster 2024-03-11T21:29:37.714638+0000 mon.a (mon.0) 1238 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:38.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:38.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:38.995 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:39.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:39.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:39.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:39.315 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:39.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:39.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:39.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:39.568 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: cluster 2024-03-11T21:29:37.235027+0000 mgr.y (mgr.14550) 266 : cluster 0 pgmap v253: 624 pgs: 18 creating+peering, 182 unknown, 424 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:29:39.568 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:37.725991+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:39.568 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:37.726092+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:39.568 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:37.745085+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.568 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:37.749378+0000 mon.c (mon.2) 255 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: cluster 2024-03-11T21:29:38.336202+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:38.339354+0000 mon.a (mon.0) 1242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:38.358284+0000 mon.a (mon.0) 1243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:38.370704+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:38.948367+0000 mon.a (mon.0) 1244 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: audit 2024-03-11T21:29:38.948477+0000 mon.a (mon.0) 1245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[29284]: cluster 2024-03-11T21:29:38.956052+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:29:39.569 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: cluster 2024-03-11T21:29:37.235027+0000 mgr.y (mgr.14550) 266 : cluster 0 pgmap v253: 624 pgs: 18 creating+peering, 182 unknown, 424 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:37.725991+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:37.726092+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:37.745085+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:37.749378+0000 mon.c (mon.2) 255 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: cluster 2024-03-11T21:29:38.336202+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:38.339354+0000 mon.a (mon.0) 1242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:38.358284+0000 mon.a (mon.0) 1243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:38.370704+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:38.948367+0000 mon.a (mon.0) 1244 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: audit 2024-03-11T21:29:38.948477+0000 mon.a (mon.0) 1245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:39.570 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:39 smithi143 bash[21512]: cluster 2024-03-11T21:29:38.956052+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:29:39.606 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:39.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:39.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: cluster 2024-03-11T21:29:37.235027+0000 mgr.y (mgr.14550) 266 : cluster 0 pgmap v253: 624 pgs: 18 creating+peering, 182 unknown, 424 active+clean; 217 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:29:39.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:37.725991+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi143-71929-4", "tierpool": "test-rados-api-smithi143-71929-6"}]': finished 2024-03-11T21:29:39.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:37.726092+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:37.745085+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.143:0/622293172' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:37.749378+0000 mon.c (mon.2) 255 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: cluster 2024-03-11T21:29:38.336202+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:38.339354+0000 mon.a (mon.0) 1242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:38.358284+0000 mon.a (mon.0) 1243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:38.370704+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:39.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:38.948367+0000 mon.a (mon.0) 1244 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi143-71929-6", "pool2": "test-rados-api-smithi143-71929-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:29:39.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: audit 2024-03-11T21:29:38.948477+0000 mon.a (mon.0) 1245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:39.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:39 smithi203 bash[23654]: cluster 2024-03-11T21:29:38.956052+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:29:39.732 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:39.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:39.859 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:39.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:40.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:40.124 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:40.207 INFO:tasks.workunit.client.0.smithi143.stdout:pi_watch_notify_pp: flushed 2024-03-11T21:29:40.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3013 ms) 2024-03-11T21:29:40.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-11T21:29:40.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: trying... 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721970320 notify_id 717259538432 notifier_gid 25400 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: timed out 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushed 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3019 ms) 2024-03-11T21:29:40.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: List watches 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721970320 notify_id 730144440323 notifier_gid 25400 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 done 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: watch_check 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: unwatch2 2024-03-11T21:29:40.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-03-11T21:29:40.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: done 2024-03-11T21:29:40.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3129 ms) 2024-03-11T21:29:40.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-11T21:29:40.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: List watches 2024-03-11T21:29:40.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94860721970320 notify_id 743029342212 notifier_gid 25400 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 done 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: watch_check 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: unwatch2 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: done 2024-03-11T21:29:40.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3017 ms) 2024-03-11T21:29:40.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18782 ms total) 2024-03-11T21:29:40.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: 2024-03-11T21:29:40.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-11T21:29:40.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (64500 ms total) 2024-03-11T21:29:40.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-11T21:29:40.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:40.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:40.430 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:40.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:40.553 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:40.656 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:40.694 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:40.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:40.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:38.361003+0000 mgr.y (mgr.14550) 267 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:40.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:38.969549+0000 mon.b (mon.1) 78 : audit 1 from='client.? 172.21.15.143:0/267256569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:39.119852+0000 mon.c (mon.2) 257 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: cluster 2024-03-11T21:29:39.235853+0000 mgr.y (mgr.14550) 268 : cluster 0 pgmap v256: 684 pgs: 42 creating+peering, 86 unknown, 556 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:39.375391+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:39.566678+0000 mon.a (mon.0) 1247 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:39.578438+0000 mon.a (mon.0) 1248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.192165+0000 mon.a (mon.0) 1249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.192255+0000 mon.a (mon.0) 1250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: cluster 2024-03-11T21:29:40.202994+0000 mon.a (mon.0) 1251 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.208727+0000 mon.b (mon.1) 79 : audit 1 from='client.? 172.21.15.143:0/2761071479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.219642+0000 mon.a (mon.0) 1252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.222311+0000 mon.c (mon.2) 259 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.225005+0000 mon.a (mon.0) 1253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.807 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.244797+0000 mon.a (mon.0) 1254 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.248086+0000 mon.a (mon.0) 1255 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.248756+0000 mon.a (mon.0) 1256 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:40.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:40 smithi203 bash[23654]: audit 2024-03-11T21:29:40.404156+0000 mon.c (mon.2) 260 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:40.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:40.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:38.361003+0000 mgr.y (mgr.14550) 267 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:38.969549+0000 mon.b (mon.1) 78 : audit 1 from='client.? 172.21.15.143:0/267256569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:39.119852+0000 mon.c (mon.2) 257 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: cluster 2024-03-11T21:29:39.235853+0000 mgr.y (mgr.14550) 268 : cluster 0 pgmap v256: 684 pgs: 42 creating+peering, 86 unknown, 556 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:39.375391+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:39.566678+0000 mon.a (mon.0) 1247 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:39.578438+0000 mon.a (mon.0) 1248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.192165+0000 mon.a (mon.0) 1249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.192255+0000 mon.a (mon.0) 1250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: cluster 2024-03-11T21:29:40.202994+0000 mon.a (mon.0) 1251 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.208727+0000 mon.b (mon.1) 79 : audit 1 from='client.? 172.21.15.143:0/2761071479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.219642+0000 mon.a (mon.0) 1252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.222311+0000 mon.c (mon.2) 259 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.225005+0000 mon.a (mon.0) 1253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.244797+0000 mon.a (mon.0) 1254 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.248086+0000 mon.a (mon.0) 1255 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.248756+0000 mon.a (mon.0) 1256 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[21512]: audit 2024-03-11T21:29:40.404156+0000 mon.c (mon.2) 260 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:38.361003+0000 mgr.y (mgr.14550) 267 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:38.969549+0000 mon.b (mon.1) 78 : audit 1 from='client.? 172.21.15.143:0/267256569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:39.119852+0000 mon.c (mon.2) 257 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: cluster 2024-03-11T21:29:39.235853+0000 mgr.y (mgr.14550) 268 : cluster 0 pgmap v256: 684 pgs: 42 creating+peering, 86 unknown, 556 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:39.375391+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:39.566678+0000 mon.a (mon.0) 1247 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:39.578438+0000 mon.a (mon.0) 1248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.192165+0000 mon.a (mon.0) 1249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi143-71604-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.192255+0000 mon.a (mon.0) 1250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: cluster 2024-03-11T21:29:40.202994+0000 mon.a (mon.0) 1251 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.208727+0000 mon.b (mon.1) 79 : audit 1 from='client.? 172.21.15.143:0/2761071479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.219642+0000 mon.a (mon.0) 1252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.222311+0000 mon.c (mon.2) 259 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.225005+0000 mon.a (mon.0) 1253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.244797+0000 mon.a (mon.0) 1254 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.248086+0000 mon.a (mon.0) 1255 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.248756+0000 mon.a (mon.0) 1256 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:40.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:40 smithi143 bash[29284]: audit 2024-03-11T21:29:40.404156+0000 mon.c (mon.2) 260 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:40.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:41.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:41.063 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:41.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.402 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:41.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:41.578 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:41.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.768 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:41.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:41.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.153 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:42.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.277 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:42.403 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:42.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:42.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:42.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.665 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:42.789 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:42.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:42.919 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:43.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:43.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:43.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:43.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:43.299 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:43.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:43.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:43.554 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:43.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.680 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:43.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.196845+0000 mon.a (mon.0) 1257 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.196931+0000 mon.a (mon.0) 1258 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.196993+0000 mon.a (mon.0) 1259 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: cluster 2024-03-11T21:29:41.202074+0000 mon.a (mon.0) 1260 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.205634+0000 mon.a (mon.0) 1261 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.215710+0000 mon.a (mon.0) 1262 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:43.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:43 smithi203 bash[23654]: audit 2024-03-11T21:29:41.407236+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:43.815 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:43.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.196845+0000 mon.a (mon.0) 1257 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.196931+0000 mon.a (mon.0) 1258 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.196993+0000 mon.a (mon.0) 1259 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: cluster 2024-03-11T21:29:41.202074+0000 mon.a (mon.0) 1260 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.205634+0000 mon.a (mon.0) 1261 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.215710+0000 mon.a (mon.0) 1262 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:43.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[29284]: audit 2024-03-11T21:29:41.407236+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.196845+0000 mon.a (mon.0) 1257 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi143-71594-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.196931+0000 mon.a (mon.0) 1258 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-7"}]': finished 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.196993+0000 mon.a (mon.0) 1259 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: cluster 2024-03-11T21:29:41.202074+0000 mon.a (mon.0) 1260 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.205634+0000 mon.a (mon.0) 1261 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.215710+0000 mon.a (mon.0) 1262 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:43.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:43 smithi143 bash[21512]: audit 2024-03-11T21:29:41.407236+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:43.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:44.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.071 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:44.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.198 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:44.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:44.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:44.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:44.458 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:44.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:44.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:44.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:44.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.839 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:44.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.964 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:45.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: Running main() from gmock_main.cc 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [----------] Global test environment set-up. 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-11T21:29:45.079 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (40917 ms) 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (11592 ms) 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (16668 ms) 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [----------] 3 tests from NeoradosECList (69177 ms total) 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [----------] Global test environment tear-down 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (69177 ms total) 2024-03-11T21:29:45.080 INFO:tasks.workunit.client.0.smithi143.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-11T21:29:45.086 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: Running main() from gmock_main.cc 2024-03-11T21:29:45.086 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-11T21:29:45.086 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [----------] Global test environment set-up. 2024-03-11T21:29:45.086 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (17671 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (8972 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (5908 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (12195 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (5125 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4304 ms) 2024-03-11T21:29:45.087 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (14869 ms) 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (69044 ms total) 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [----------] Global test environment tear-down 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (69044 ms total) 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-11T21:29:45.088 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (18963 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (14348 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (5303 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (6416 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (5150 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (5458 ms) 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-11T21:29:45.089 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (13691 ms) 2024-03-11T21:29:45.090 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (69329 ms total) 2024-03-11T21:29:45.090 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: 2024-03-11T21:29:45.090 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-11T21:29:45.090 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (69330 ms total) 2024-03-11T21:29:45.090 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-11T21:29:45.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:45.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:45.213 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/1626949383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d81060a0 msgr2=0x7fe3d810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/1626949383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d81060a0 0x7fe3d810dc70 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b0d0 tx=0x7fe3d000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/1626949383 shutdown_connections 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/1626949383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d8100fc0 0x7fe3d81013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/1626949383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d81060a0 0x7fe3d810dc70 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/1626949383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8101970 0x7fe3d81059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/1626949383 >> 172.21.15.143:0/1626949383 conn(0x7fe3d8077de0 msgr2=0x7fe3d80781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/1626949383 shutdown_connections 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/1626949383 wait complete. 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 librados: starting msgr at 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 librados: starting objecter 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 Processor -- start 2024-03-11T21:29:45.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- start start 2024-03-11T21:29:45.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 librados: setting wanted keys 2024-03-11T21:29:45.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 librados: calling monclient init 2024-03-11T21:29:45.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8100fc0 0x7fe3d819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 0x7fe3d819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 0x7fe3d81969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8110590 con 0x7fe3d81060a0 2024-03-11T21:29:45.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8110410 con 0x7fe3d8100fc0 2024-03-11T21:29:45.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3dcdf9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8110710 con 0x7fe3d8101970 2024-03-11T21:29:45.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d6d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 0x7fe3d81969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 0x7fe3d819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d6d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 0x7fe3d81969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39136/0 (socket says 172.21.15.143:39136) 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d6d9d700 1 -- 172.21.15.143:0/3137414714 learned_addr learned my addr 172.21.15.143:0/3137414714 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d659c700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8100fc0 0x7fe3d819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 -- 172.21.15.143:0/3137414714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8100fc0 msgr2=0x7fe3d819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8100fc0 0x7fe3d819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 -- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 msgr2=0x7fe3d81969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 0x7fe3d81969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.259+0000 7fe3d5d9b700 1 -- 172.21.15.143:0/3137414714 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8197100 con 0x7fe3d8101970 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3d5d9b700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 0x7fe3d819cd20 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00d720 tx=0x7fe3cc00da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 10 client.?.objecter ms_handle_connect 0x7fe3d8101970 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:45.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc016070 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d81973f0 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe3cc005020 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc00f990 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3dcdf9700 10 client.40742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d806a3a0 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3dcdf9700 1 librados: init done 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d804faa0 con 0x7fe3d8101970 2024-03-11T21:29:45.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe3cc00fb30 con 0x7fe3d8101970 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 0x7fe3c0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(184..184 src has 87..184) v4 ==== 13784+0+0 (secure 0 0 0) 0x7fe3cc08f580 con 0x7fe3d8101970 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 10 client.40742.objecter ms_dispatch 0x7fe3d8001550 osd_map(184..184 src has 87..184) v4 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 3 client.40742.objecter handle_osd_map got epochs [184,184] > 0 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 3 client.40742.objecter handle_osd_map decoding full epoch 184 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3d659c700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 0x7fe3c0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 20 client.40742.objecter dump_active .. 0 homeless 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3c77fe700 10 client.40742.objecter ms_handle_connect 0x7fe3c006f820 2024-03-11T21:29:45.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.263+0000 7fe3d659c700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 0x7fe3c0071ce0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800e630 tx=0x7fe3c8018040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:45.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.267+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe3cc00fe20 con 0x7fe3d8101970 2024-03-11T21:29:45.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.325 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: cluster 2024-03-11T21:29:41.237644+0000 mgr.y (mgr.14550) 269 : cluster 0 pgmap v259: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:42.408259+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: cluster 2024-03-11T21:29:42.429038+0000 mon.a (mon.0) 1263 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:42.431596+0000 mon.a (mon.0) 1264 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:42.880174+0000 mon.b (mon.1) 80 : audit 1 from='client.? 172.21.15.143:0/1315336659' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:42.882528+0000 mon.c (mon.2) 263 : audit 1 from='client.? 172.21.15.143:0/498644794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:43.409133+0000 mon.c (mon.2) 264 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: cluster 2024-03-11T21:29:43.742422+0000 mon.a (mon.0) 1265 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:43.748085+0000 mon.a (mon.0) 1266 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:43.748939+0000 mon.a (mon.0) 1267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:43.749592+0000 mon.a (mon.0) 1268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:43.762631+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: audit 2024-03-11T21:29:44.410098+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[29284]: cluster 2024-03-11T21:29:44.638612+0000 mon.a (mon.0) 1270 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: cluster 2024-03-11T21:29:41.237644+0000 mgr.y (mgr.14550) 269 : cluster 0 pgmap v259: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:42.408259+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: cluster 2024-03-11T21:29:42.429038+0000 mon.a (mon.0) 1263 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:42.431596+0000 mon.a (mon.0) 1264 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:42.880174+0000 mon.b (mon.1) 80 : audit 1 from='client.? 172.21.15.143:0/1315336659' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:42.882528+0000 mon.c (mon.2) 263 : audit 1 from='client.? 172.21.15.143:0/498644794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:43.409133+0000 mon.c (mon.2) 264 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: cluster 2024-03-11T21:29:43.742422+0000 mon.a (mon.0) 1265 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:43.748085+0000 mon.a (mon.0) 1266 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:43.748939+0000 mon.a (mon.0) 1267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:43.749592+0000 mon.a (mon.0) 1268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:43.762631+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:45.327 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: audit 2024-03-11T21:29:44.410098+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.328 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:45 smithi143 bash[21512]: cluster 2024-03-11T21:29:44.638612+0000 mon.a (mon.0) 1270 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:29:45.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:45.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:45.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.475+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3d80ff240 con 0x7fe3c006f820 2024-03-11T21:29:45.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:45.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.539+0000 7fe3c77fe700 1 -- 172.21.15.143:0/3137414714 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1463066 (secure 0 0 0) 0x7fe3d80ff240 con 0x7fe3c006f820 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: cluster 2024-03-11T21:29:41.237644+0000 mgr.y (mgr.14550) 269 : cluster 0 pgmap v259: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:42.408259+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: cluster 2024-03-11T21:29:42.429038+0000 mon.a (mon.0) 1263 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:42.431596+0000 mon.a (mon.0) 1264 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:42.880174+0000 mon.b (mon.1) 80 : audit 1 from='client.? 172.21.15.143:0/1315336659' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:42.882528+0000 mon.c (mon.2) 263 : audit 1 from='client.? 172.21.15.143:0/498644794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:43.409133+0000 mon.c (mon.2) 264 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: cluster 2024-03-11T21:29:43.742422+0000 mon.a (mon.0) 1265 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:43.748085+0000 mon.a (mon.0) 1266 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:43.748939+0000 mon.a (mon.0) 1267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:43.749592+0000 mon.a (mon.0) 1268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:43.762631+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi143-71818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: audit 2024-03-11T21:29:44.410098+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:45 smithi203 bash[23654]: cluster 2024-03-11T21:29:44.638612+0000 mon.a (mon.0) 1270 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:29:45.534 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 10 librados: watch_flush enter 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 10 librados: watch_flush exit 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 20 client.40742.objecter shutdown clearing up homeless session... 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 10 client.40742.objecter successfully canceled tick 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 msgr2=0x7fe3c0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 0x7fe3c0071ce0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800e630 tx=0x7fe3c8018040 comp rx=0 tx=0).stop 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 msgr2=0x7fe3d819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 0x7fe3d819cd20 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00d720 tx=0x7fe3cc00da30 comp rx=0 tx=0).stop 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 shutdown_connections 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe3c006f820 0x7fe3c0071ce0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3d81060a0 0x7fe3d81969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3d8100fc0 0x7fe3d819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 --2- 172.21.15.143:0/3137414714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d8101970 0x7fe3d819cd20 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.551+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 >> 172.21.15.143:0/3137414714 conn(0x7fe3d8077de0 msgr2=0x7fe3d80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.555+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 shutdown_connections 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.555+0000 7fe3dcdf9700 1 -- 172.21.15.143:0/3137414714 wait complete. 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.555+0000 7fe3dcdf9700 1 librados: shutdown 2024-03-11T21:29:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:29:45.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.602 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:45.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.730 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3252925049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 msgr2=0x7fc5801022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- 172.21.15.143:0/3252925049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5801022f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fc574009990 tx=0x7fc574009ca0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3252925049 shutdown_connections 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- 172.21.15.143:0/3252925049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580102830 0x7fc58010ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- 172.21.15.143:0/3252925049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580107ed0 0x7fc5801082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- 172.21.15.143:0/3252925049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5801022f0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3252925049 >> 172.21.15.143:0/3252925049 conn(0x7fc580078f90 msgr2=0x7fc5800793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:45.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3252925049 shutdown_connections 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3252925049 wait complete. 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 librados: starting msgr at 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 librados: starting objecter 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 Processor -- start 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- start start 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 librados: setting wanted keys 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 librados: calling monclient init 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5800740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 0x7fc580074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580111650 con 0x7fc580101e90 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801114d0 con 0x7fc580102830 2024-03-11T21:29:45.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801117d0 con 0x7fc580107ed0 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5800740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37962/0 (socket says 172.21.15.143:37962) 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 -- 172.21.15.143:0/3441207891 learned_addr learned my addr 172.21.15.143:0/3441207891 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57dd9b700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 0x7fc580074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 -- 172.21.15.143:0/3441207891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 msgr2=0x7fc580074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 0x7fc580074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 -- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 msgr2=0x7fc5800740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5800740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 -- 172.21.15.143:0/3441207891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5801a61d0 con 0x7fc580107ed0 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57e59c700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57dd9b700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 0x7fc580074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:29:45.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc57ed9d700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fc57000e970 tx=0x7fc57000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc56f7fe700 10 client.?.objecter ms_handle_connect 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc56f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570004a30 con 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc5700054a0 con 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000cbb0 con 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5801a64c0 con 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 10 client.40748.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5801a69b0 con 0x7fc580107ed0 2024-03-11T21:29:45.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 librados: init done 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.799+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58004faa0 con 0x7fc580107ed0 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc570005000 con 0x7fc580107ed0 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 0x7fc568071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(185..185 src has 87..185) v4 ==== 14183+0+0 (secure 0 0 0) 0x7fc570090100 con 0x7fc580107ed0 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 10 client.40748.objecter ms_dispatch 0x7fc580001550 osd_map(185..185 src has 87..185) v4 2024-03-11T21:29:45.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 3 client.40748.objecter handle_osd_map got epochs [185,185] > 0 2024-03-11T21:29:45.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 3 client.40748.objecter handle_osd_map decoding full epoch 185 2024-03-11T21:29:45.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc57e59c700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 0x7fc568071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:45.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 20 client.40748.objecter dump_active .. 0 homeless 2024-03-11T21:29:45.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc56f7fe700 10 client.40748.objecter ms_handle_connect 0x7fc56806f820 2024-03-11T21:29:45.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.803+0000 7fc57e59c700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 0x7fc568071ce0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fc57800a7b0 tx=0x7fc57800d040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:45.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:45.807+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc57005bb80 con 0x7fc580107ed0 2024-03-11T21:29:45.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.849 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:45.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.975 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:46.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.023+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc5801a6c70 con 0x7fc580107ed0 2024-03-11T21:29:46.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:43.240555+0000 mgr.y (mgr.14550) 270 : cluster 0 pgmap v261: 516 pgs: 45 creating+peering, 19 unknown, 452 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:45.046064+0000 osd.7 (osd.7) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:45.046174+0000 osd.7 (osd.7) 192 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.099711+0000 mon.a (mon.0) 1271 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.099782+0000 mon.a (mon.0) 1272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.099853+0000 mon.a (mon.0) 1273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:45.104907+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.128531+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.143:0/2472074103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.128843+0000 mon.c (mon.2) 267 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:45.241650+0000 mgr.y (mgr.14550) 271 : cluster 0 pgmap v264: 588 pgs: 45 creating+peering, 187 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.411193+0000 mon.c (mon.2) 268 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.478800+0000 mgr.y (mgr.14550) 272 : audit 0 from='client.40742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: cluster 2024-03-11T21:29:45.578218+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.580416+0000 mon.a (mon.0) 1276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:45.595079+0000 mon.a (mon.0) 1277 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:46.028138+0000 mon.c (mon.2) 269 : audit 1 from='client.? 172.21.15.143:0/3441207891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[21512]: audit 2024-03-11T21:29:46.028422+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:43.240555+0000 mgr.y (mgr.14550) 270 : cluster 0 pgmap v261: 516 pgs: 45 creating+peering, 19 unknown, 452 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:45.046064+0000 osd.7 (osd.7) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:45.046174+0000 osd.7 (osd.7) 192 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:46.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.099711+0000 mon.a (mon.0) 1271 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.099782+0000 mon.a (mon.0) 1272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.099853+0000 mon.a (mon.0) 1273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:45.104907+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.128531+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.143:0/2472074103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.128843+0000 mon.c (mon.2) 267 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:45.241650+0000 mgr.y (mgr.14550) 271 : cluster 0 pgmap v264: 588 pgs: 45 creating+peering, 187 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.411193+0000 mon.c (mon.2) 268 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.478800+0000 mgr.y (mgr.14550) 272 : audit 0 from='client.40742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: cluster 2024-03-11T21:29:45.578218+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.580416+0000 mon.a (mon.0) 1276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:45.595079+0000 mon.a (mon.0) 1277 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:46.028138+0000 mon.c (mon.2) 269 : audit 1 from='client.? 172.21.15.143:0/3441207891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.095 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:46 smithi143 bash[29284]: audit 2024-03-11T21:29:46.028422+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.105 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:46.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:46.234 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:29:46.234 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:29:46.234 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:29:46.234 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:29:46.236 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:46.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:46.350 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:43.240555+0000 mgr.y (mgr.14550) 270 : cluster 0 pgmap v261: 516 pgs: 45 creating+peering, 19 unknown, 452 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:29:46.350 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:45.046064+0000 osd.7 (osd.7) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:29:46.350 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:45.046174+0000 osd.7 (osd.7) 192 : cluster 0 purged_snaps scrub ok 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.099711+0000 mon.a (mon.0) 1271 : audit 1 from='client.? 172.21.15.143:0/2361743075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi143-72329-3"}]': finished 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.099782+0000 mon.a (mon.0) 1272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi143-71604-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.099853+0000 mon.a (mon.0) 1273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:45.104907+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.128531+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.143:0/2472074103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.128843+0000 mon.c (mon.2) 267 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:45.241650+0000 mgr.y (mgr.14550) 271 : cluster 0 pgmap v264: 588 pgs: 45 creating+peering, 187 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.411193+0000 mon.c (mon.2) 268 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.478800+0000 mgr.y (mgr.14550) 272 : audit 0 from='client.40742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: cluster 2024-03-11T21:29:45.578218+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.580416+0000 mon.a (mon.0) 1276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:45.595079+0000 mon.a (mon.0) 1277 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:46.028138+0000 mon.c (mon.2) 269 : audit 1 from='client.? 172.21.15.143:0/3441207891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:46 smithi203 bash[23654]: audit 2024-03-11T21:29:46.028422+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:46.365 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:46.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:46.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:46.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.575+0000 7fc56f7fe700 1 -- 172.21.15.143:0/3441207891 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v186) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc57005f5d0 con 0x7fc580107ed0 2024-03-11T21:29:46.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 10 librados: watch_flush enter 2024-03-11T21:29:46.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 10 librados: watch_flush exit 2024-03-11T21:29:46.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 20 client.40748.objecter shutdown clearing up homeless session... 2024-03-11T21:29:46.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 10 client.40748.objecter successfully canceled tick 2024-03-11T21:29:46.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 msgr2=0x7fc568071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:46.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 0x7fc568071ce0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fc57800a7b0 tx=0x7fc57800d040 comp rx=0 tx=0).stop 2024-03-11T21:29:46.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 msgr2=0x7fc580074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:46.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.583+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fc57000e970 tx=0x7fc57000ec80 comp rx=0 tx=0).stop 2024-03-11T21:29:46.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 shutdown_connections 2024-03-11T21:29:46.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc56806f820 0x7fc568071ce0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc580101e90 0x7fc5800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc580107ed0 0x7fc580074b60 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 --2- 172.21.15.143:0/3441207891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc580102830 0x7fc580074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.599+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 >> 172.21.15.143:0/3441207891 conn(0x7fc580078f90 msgr2=0x7fc580105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:46.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.603+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 shutdown_connections 2024-03-11T21:29:46.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.603+0000 7fc584dba700 1 -- 172.21.15.143:0/3441207891 wait complete. 2024-03-11T21:29:46.578 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-11T21:29:46.578 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] Global test environment set-up. 2024-03-11T21:29:46.578 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-11T21:29:46.578 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:28:36.554+0000 7fdf48260880 -1 failed for service _ceph-mon._tcp 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:28:36.554+0000 7fdf48260880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (903 ms) 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2010 ms) 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2913 ms total) 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:29:46.579 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 0x56211bafa2a0 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 0x56211be38a50 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 2 aios 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: waiting 0x56211bafa2a0 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: waiting 0x56211be38a50 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: done. 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (22814 ms) 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (22814 ms total) 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-11T21:29:46.580 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.Exec (1199 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (18 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.Applications (9267 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (29270 ms) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 8 tests from LibRadosMisc (39760 ms total) 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-03-11T21:29:46.581 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] Global test environment tear-down 2024-03-11T21:29:46.582 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (70934 ms total) 2024-03-11T21:29:46.582 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-11T21:29:46.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:46.607+0000 7fc584dba700 1 librados: shutdown 2024-03-11T21:29:46.586 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:29:46.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:46.634 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:46.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:46.757 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:46.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:46.886 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:46.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- 172.21.15.143:0/119674429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c01071c0 msgr2=0x7f20c010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- 172.21.15.143:0/119674429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c01071c0 0x7f20c010ed40 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f20b800b0d0 tx=0x7f20b800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- 172.21.15.143:0/119674429 shutdown_connections 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- 172.21.15.143:0/119674429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c006a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- 172.21.15.143:0/119674429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c01071c0 0x7f20c010ed40 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- 172.21.15.143:0/119674429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c006a720 0x7f20c0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- 172.21.15.143:0/119674429 >> 172.21.15.143:0/119674429 conn(0x7f20c0076d80 msgr2=0x7f20c0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- 172.21.15.143:0/119674429 shutdown_connections 2024-03-11T21:29:46.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- 172.21.15.143:0/119674429 wait complete. 2024-03-11T21:29:46.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 librados: starting msgr at 2024-03-11T21:29:46.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 librados: starting objecter 2024-03-11T21:29:46.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 Processor -- start 2024-03-11T21:29:46.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- start start 2024-03-11T21:29:46.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 librados: setting wanted keys 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 librados: calling monclient init 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c006a720 0x7f20c019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c01071c0 0x7f20c019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c01116d0 con 0x7f20c0069d70 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c0111550 con 0x7f20c01071c0 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c8222700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c0111850 con 0x7f20c006a720 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c5fbe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c5fbe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39252/0 (socket says 172.21.15.143:39252) 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.015+0000 7f20c5fbe700 1 -- 172.21.15.143:0/3613364002 learned_addr learned my addr 172.21.15.143:0/3613364002 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c006a720 0x7f20c019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c01071c0 0x7f20c019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c006a720 msgr2=0x7f20c019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:46.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20c006a720 0x7f20c019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c01071c0 msgr2=0x7f20c019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c01071c0 0x7f20c019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c019ee20 con 0x7f20c0069d70 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00e9a0 tx=0x7f20bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20b6ffd700 10 client.?.objecter ms_handle_connect 0x7f20c0069d70 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.019+0000 7f20b6ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:29:47.017 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:47.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:47.148 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:47.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:47.278 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:47.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:47.407 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:47.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.487+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00cbd0 con 0x7f20c0069d70 2024-03-11T21:29:47.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.487+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c019f110 con 0x7f20c0069d70 2024-03-11T21:29:47.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.487+0000 7f20c8222700 10 client.29589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:29:47.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.487+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c01a6990 con 0x7f20c0069d70 2024-03-11T21:29:47.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.487+0000 7f20c8222700 1 librados: init done 2024-03-11T21:29:47.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.491+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f20c004faa0 con 0x7f20c0069d70 2024-03-11T21:29:47.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.495+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f20bc00cd70 con 0x7f20c0069d70 2024-03-11T21:29:47.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.495+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc026060 con 0x7f20c0069d70 2024-03-11T21:29:47.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f20bc010430 con 0x7f20c0069d70 2024-03-11T21:29:47.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 0x7f20ac071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(186..186 src has 87..186) v4 ==== 12348+0+0 (secure 0 0 0) 0x7f20bc0903c0 con 0x7f20c0069d70 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 osd_map(186..186 src has 87..186) v4 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 3 client.29589.objecter handle_osd_map got epochs [186,186] > 0 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 3 client.29589.objecter handle_osd_map decoding full epoch 186 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 0x7f20ac071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 20 client.29589.objecter dump_active .. 0 homeless 2024-03-11T21:29:47.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v186) v1 ==== 54+0+16 (secure 0 0 0) 0x7f20bc090810 con 0x7f20c0069d70 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 10 client.29589.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 1 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f20c00629f0 0x7f20c0062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 1 2024-03-11T21:29:47.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 0 1 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 4 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 10 client.29589.objecter _send_command 1 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01a9e40 con 0x7f20c00629f0 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f20c00629f0 0x7f20c0062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20ac06f820 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.499+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 0x7f20ac071ce0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f20c0108510 tx=0x7f20b0009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c00629f0 2024-03-11T21:29:47.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f20c00629f0 0x7f20c0062dd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01a9e40 con 0x7f20c00629f0 2024-03-11T21:29:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:29:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 1 = system:0 2024-03-11T21:29:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 4 2024-03-11T21:29:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.503+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 0 1 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 10 client.29589.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 2 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 2 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c019b170 osd=0 3 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 0 2 2024-03-11T21:29:47.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 4 2024-03-11T21:29:47.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 10 client.29589.objecter _send_command 2 2024-03-11T21:29:47.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.523+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c0062470 con 0x7f20c00629f0 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c0062470 con 0x7f20c00629f0 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 4 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 0 2 2024-03-11T21:29:47.503 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 10 client.29589.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 3 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f20c002c6b0 0x7f20c002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 3 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 1 3 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 4 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 10 client.29589.objecter _send_command 3 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c00ffa00 con 0x7f20c002c6b0 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.527+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f20c002c6b0 0x7f20c002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c002c6b0 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f20c002c6b0 0x7f20c002ca90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c00ffa00 con 0x7f20c002c6b0 2024-03-11T21:29:47.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:29:47.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 3 = system:0 2024-03-11T21:29:47.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 4 2024-03-11T21:29:47.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.531+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 1 3 2024-03-11T21:29:47.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 10 client.29589.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 4 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 4 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c00ff380 osd=1 3 2024-03-11T21:29:47.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 1 4 2024-03-11T21:29:47.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 4 2024-03-11T21:29:47.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 10 client.29589.objecter _send_command 4 2024-03-11T21:29:47.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.555+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c00fee00 con 0x7f20c002c6b0 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c00fee00 con 0x7f20c002c6b0 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 4 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 1 4 2024-03-11T21:29:47.532 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:29:47.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:47.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 10 client.29589.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 5 2024-03-11T21:29:47.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f20c01ac000 0x7f20c01ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 5 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 2 5 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 4 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 10 client.29589.objecter _send_command 5 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c002dcb0 con 0x7f20c01ac000 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f20c01ac000 0x7f20c01ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01ac000 2024-03-11T21:29:47.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f20c01ac000 0x7f20c01ae3f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:47.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c002dcb0 con 0x7f20c01ac000 2024-03-11T21:29:47.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:29:47.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 5 = system:0 2024-03-11T21:29:47.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 4 2024-03-11T21:29:47.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.559+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 2 5 2024-03-11T21:29:47.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 10 client.29589.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 6 2024-03-11T21:29:47.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 6 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c00ffbb0 osd=2 3 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 2 6 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 4 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 10 client.29589.objecter _send_command 6 2024-03-11T21:29:47.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c002d220 con 0x7f20c01ac000 2024-03-11T21:29:47.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c002d220 con 0x7f20c01ac000 2024-03-11T21:29:47.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 4 2024-03-11T21:29:47.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 2 6 2024-03-11T21:29:47.575 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 10 client.29589.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 7 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f20c01afab0 0x7f20c01b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 7 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 3 7 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 4 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 10 client.29589.objecter _send_command 7 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01a9c70 con 0x7f20c01afab0 2024-03-11T21:29:47.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.595+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f20c01afab0 0x7f20c01b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01afab0 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f20c01afab0 0x7f20c01b1ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01a9c70 con 0x7f20c01afab0 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 7 = system:0 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 4 2024-03-11T21:29:47.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.599+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 3 7 2024-03-11T21:29:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 10 client.29589.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 8 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 8 2024-03-11T21:29:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a9490 osd=3 3 2024-03-11T21:29:47.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 3 8 2024-03-11T21:29:47.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 4 2024-03-11T21:29:47.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 10 client.29589.objecter _send_command 8 2024-03-11T21:29:47.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c01a8f10 con 0x7f20c01afab0 2024-03-11T21:29:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c01a8f10 con 0x7f20c01afab0 2024-03-11T21:29:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 4 2024-03-11T21:29:47.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 3 8 2024-03-11T21:29:47.593 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:29:47.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 10 client.29589.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 9 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f20c01af610 0x7f20c01b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 9 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 4 9 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 4 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 10 client.29589.objecter _send_command 9 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.615+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01a8e30 con 0x7f20c01af610 2024-03-11T21:29:47.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f20c01af610 0x7f20c01b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01af610 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f20c01af610 0x7f20c01b3560 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01a8e30 con 0x7f20c01af610 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 9 = system:0 2024-03-11T21:29:47.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 4 2024-03-11T21:29:47.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.619+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 4 9 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 10 client.29589.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 10 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 10 2024-03-11T21:29:47.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a8cc0 osd=4 3 2024-03-11T21:29:47.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 4 10 2024-03-11T21:29:47.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 4 2024-03-11T21:29:47.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 10 client.29589.objecter _send_command 10 2024-03-11T21:29:47.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.635+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c01a8e30 con 0x7f20c01af610 2024-03-11T21:29:47.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.639+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c01a8e30 con 0x7f20c01af610 2024-03-11T21:29:47.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.639+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.639+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.639+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 4 2024-03-11T21:29:47.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.639+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 4 10 2024-03-11T21:29:47.617 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 10 client.29589.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 11 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f20c01b4000 0x7f20c01b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 11 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 5 11 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 4 2024-03-11T21:29:47.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 10 client.29589.objecter _send_command 11 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01b4b00 con 0x7f20c01b4000 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f20c01b4000 0x7f20c01b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20c57bd700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f20c01b4000 0x7f20c01b4440 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01b4000 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01b4b00 con 0x7f20c01b4000 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 11 = system:0 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 4 2024-03-11T21:29:47.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.643+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 5 11 2024-03-11T21:29:47.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.675+0000 7f20c8222700 10 client.29589.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 12 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 12 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01a8e30 osd=5 3 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 5 12 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 4 2024-03-11T21:29:47.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 10 client.29589.objecter _send_command 12 2024-03-11T21:29:47.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.679+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c01b8db0 con 0x7f20c01b4000 2024-03-11T21:29:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c01b8db0 con 0x7f20c01b4000 2024-03-11T21:29:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 4 2024-03-11T21:29:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 5 12 2024-03-11T21:29:47.657 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:29:47.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 10 client.29589.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 13 2024-03-11T21:29:47.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f20c01b9430 0x7f20c01b9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 13 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 6 13 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 4 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 10 client.29589.objecter _send_command 13 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01be890 con 0x7f20c01b9430 2024-03-11T21:29:47.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f20c01b9430 0x7f20c01b9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.659 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:47.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01b9430 2024-03-11T21:29:47.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20c67bf700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f20c01b9430 0x7f20c01b9850 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:47.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01be890 con 0x7f20c01b9430 2024-03-11T21:29:47.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:29:47.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 13 = system:0 2024-03-11T21:29:47.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 4 2024-03-11T21:29:47.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.683+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 6 13 2024-03-11T21:29:47.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 10 client.29589.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 14 2024-03-11T21:29:47.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 14 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01b8db0 osd=6 3 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 6 14 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 4 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 10 client.29589.objecter _send_command 14 2024-03-11T21:29:47.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.703+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c01b9de0 con 0x7f20c01b9430 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c01b9de0 con 0x7f20c01b9430 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 4 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 6 14 2024-03-11T21:29:47.681 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 10 client.29589.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 15 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f20c01bb1a0 0x7f20c01ba500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 15 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 7 15 2024-03-11T21:29:47.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 4 2024-03-11T21:29:47.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 10 client.29589.objecter _send_command 15 2024-03-11T21:29:47.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20c01baba0 con 0x7f20c01bb1a0 2024-03-11T21:29:47.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f20c01bb1a0 0x7f20c01ba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:29:47.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20b6ffd700 10 client.29589.objecter ms_handle_connect 0x7f20c01bb1a0 2024-03-11T21:29:47.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.707+0000 7f20c5fbe700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f20c01bb1a0 0x7f20c01ba500 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:29:47.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.711+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20c01baba0 con 0x7f20c01bb1a0 2024-03-11T21:29:47.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.711+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:29:47.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.711+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 15 = system:0 2024-03-11T21:29:47.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.711+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 4 2024-03-11T21:29:47.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.711+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 7 15 2024-03-11T21:29:47.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 10 client.29589.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:29:47.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign -1 16 2024-03-11T21:29:47.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 20 client.29589.objecter _get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 15 client.29589.objecter _session_command_op_remove -1 16 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 20 client.29589.objecter get_session s=0x7f20c01b9de0 osd=7 3 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 15 client.29589.objecter _session_command_op_assign 7 16 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 4 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 10 client.29589.objecter _send_command 16 2024-03-11T21:29:47.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.727+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f20c01badb0 con 0x7f20c01bb1a0 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20b6ffd700 1 -- 172.21.15.143:0/3613364002 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f20c01badb0 con 0x7f20c01bb1a0 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20b6ffd700 10 client.29589.objecter ms_dispatch 0x7f20c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20b6ffd700 10 client.29589.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20b6ffd700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 4 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20b6ffd700 15 client.29589.objecter _session_command_op_remove 7 16 2024-03-11T21:29:47.705 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:29:47.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 librados: watch_flush enter 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 librados: watch_flush exit 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.0 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f20c00629f0 msgr2=0x7f20c0062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f20c00629f0 0x7f20c0062dd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c019b170 osd=0 1 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.1 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f20c002c6b0 msgr2=0x7f20c002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f20c002c6b0 0x7f20c002ca90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ff380 osd=1 1 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.2 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f20c01ac000 msgr2=0x7f20c01ae3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f20c01ac000 0x7f20c01ae3f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c00ffbb0 osd=2 1 2024-03-11T21:29:47.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.3 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f20c01afab0 msgr2=0x7f20c01b1ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f20c01afab0 0x7f20c01b1ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a9490 osd=3 1 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.4 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f20c01af610 msgr2=0x7f20c01b3560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f20c01af610 0x7f20c01b3560 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8cc0 osd=4 1 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.5 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f20c01b4000 msgr2=0x7f20c01b4440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f20c01b4000 0x7f20c01b4440 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c67bf700 1 -- 172.21.15.143:0/3613364002 reap_dead start 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01a8e30 osd=5 1 2024-03-11T21:29:47.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.6 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c67bf700 1 -- 172.21.15.143:0/3613364002 reap_dead start 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f20c01b9430 msgr2=0x7f20c01b9850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f20c01b9430 0x7f20c01b9850 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b8db0 osd=6 1 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter close_session for osd.7 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f20c01bb1a0 msgr2=0x7f20c01ba500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f20c01bb1a0 0x7f20c01ba500 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter put_session s=0x7f20c01b9de0 osd=7 1 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 20 client.29589.objecter shutdown clearing up homeless session... 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 10 client.29589.objecter successfully canceled tick 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c67bf700 1 -- 172.21.15.143:0/3613364002 reap_dead start 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c67bf700 1 -- 172.21.15.143:0/3613364002 reap_dead start 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 msgr2=0x7f20ac071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 0x7f20ac071ce0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f20c0108510 tx=0x7f20b0009190 comp rx=0 tx=0).stop 2024-03-11T21:29:47.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 msgr2=0x7f20c0199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00e9a0 tx=0x7f20bc00ecb0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 shutdown_connections 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20c0069d70 0x7f20c0199e10 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 --2- 172.21.15.143:0/3613364002 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f20ac06f820 0x7f20ac071ce0 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 >> 172.21.15.143:0/3613364002 conn(0x7f20c0076d80 msgr2=0x7f20c010d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 shutdown_connections 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 -- 172.21.15.143:0/3613364002 wait complete. 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:29:47.731+0000 7f20c8222700 1 librados: shutdown 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:47.710 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:47.711 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:47.711 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:47.711 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: audit 2024-03-11T21:29:46.412373+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: cluster 2024-03-11T21:29:46.571446+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: audit 2024-03-11T21:29:46.575383+0000 mon.a (mon.0) 1280 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: audit 2024-03-11T21:29:46.575603+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: audit 2024-03-11T21:29:46.575728+0000 mon.a (mon.0) 1282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: cluster 2024-03-11T21:29:46.583280+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:29:47.717 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[29284]: audit 2024-03-11T21:29:46.698476+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: audit 2024-03-11T21:29:46.412373+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: cluster 2024-03-11T21:29:46.571446+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: audit 2024-03-11T21:29:46.575383+0000 mon.a (mon.0) 1280 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: audit 2024-03-11T21:29:46.575603+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: audit 2024-03-11T21:29:46.575728+0000 mon.a (mon.0) 1282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: cluster 2024-03-11T21:29:46.583280+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:29:47.718 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:47 smithi143 bash[21512]: audit 2024-03-11T21:29:46.698476+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:47.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:47.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: audit 2024-03-11T21:29:46.412373+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:47.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: cluster 2024-03-11T21:29:46.571446+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:47.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: audit 2024-03-11T21:29:46.575383+0000 mon.a (mon.0) 1280 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi143-71594-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: audit 2024-03-11T21:29:46.575603+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:47.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: audit 2024-03-11T21:29:46.575728+0000 mon.a (mon.0) 1282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:47.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: cluster 2024-03-11T21:29:46.583280+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:29:47.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:47 smithi203 bash[23654]: audit 2024-03-11T21:29:46.698476+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:47.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:47.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:47.914 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:48.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:48.039 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:48.142 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:48.165 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:48.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:48.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:48.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: cluster 2024-03-11T21:29:47.242694+0000 mgr.y (mgr.14550) 273 : cluster 0 pgmap v267: 460 pgs: 25 creating+peering, 111 unknown, 324 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:47.413810+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:47.502306+0000 mon.a (mon.0) 1284 : audit 0 from='client.? 172.21.15.143:0/3613364002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:47.938561+0000 mon.a (mon.0) 1285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: cluster 2024-03-11T21:29:47.949010+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:47.963291+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.143:0/1982255960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:47.965856+0000 mon.a (mon.0) 1287 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[29284]: audit 2024-03-11T21:29:48.415451+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: cluster 2024-03-11T21:29:47.242694+0000 mgr.y (mgr.14550) 273 : cluster 0 pgmap v267: 460 pgs: 25 creating+peering, 111 unknown, 324 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:47.413810+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:47.502306+0000 mon.a (mon.0) 1284 : audit 0 from='client.? 172.21.15.143:0/3613364002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:48.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:47.938561+0000 mon.a (mon.0) 1285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:48.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: cluster 2024-03-11T21:29:47.949010+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:29:48.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:47.963291+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.143:0/1982255960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:47.965856+0000 mon.a (mon.0) 1287 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:48 smithi143 bash[21512]: audit 2024-03-11T21:29:48.415451+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:48.666 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:48.666 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: cluster 2024-03-11T21:29:47.242694+0000 mgr.y (mgr.14550) 273 : cluster 0 pgmap v267: 460 pgs: 25 creating+peering, 111 unknown, 324 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:48.666 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:47.413810+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:47.502306+0000 mon.a (mon.0) 1284 : audit 0 from='client.? 172.21.15.143:0/3613364002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:47.938561+0000 mon.a (mon.0) 1285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: cluster 2024-03-11T21:29:47.949010+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:47.963291+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.143:0/1982255960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:47.965856+0000 mon.a (mon.0) 1287 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:48.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:48 smithi203 bash[23654]: audit 2024-03-11T21:29:48.415451+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:48.680 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:48.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:48.814 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:48.916 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:48.963 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:49.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:49.086 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:49.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:49.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:49.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:49.472 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:49.576 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:49.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:49.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:49.729 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:49.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:49.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.965 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-03-11T21:29:45.240788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":228148602,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":2360,"num_read_kb":50350,"num_write":1798,"num_write_kb":230289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1720,"ondisk_log_size":1720,"up":1203,"acting":1203,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1661,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1054668,"kb_used_data":685876,"kb_used_omap":12,"kb_used_meta":368691,"kb_avail":748906548,"statfs":{"total":767960285184,"available":766880305152,"internally_reserved":0,"allocated":702337024,"data_stored":694143842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":377540146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1047539,"num_objects":-4,"num_object_clones":-2,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4,"num_write_kb":1019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031382"},"pg_stats":[{"pgid":"142.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.18","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.445044+0000","last_change":"2024-03-11T21:29:42.445044+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.445044+0000","last_undegraded":"2024-03-11T21:29:42.445044+0000","last_fullsized":"2024-03-11T21:29:42.445044+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"142.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.16","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.446410+0000","last_change":"2024-03-11T21:29:42.446410+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.446410+0000","last_undegraded":"2024-03-11T21:29:42.446410+0000","last_fullsized":"2024-03-11T21:29:42.446410+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"135.19","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432634+0000","last_change":"2024-03-11T21:29:42.432634+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432634+0000","last_undegraded":"2024-03-11T21:29:42.432634+0000","last_fullsized":"2024-03-11T21:29:42.432634+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.15","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.437099+0000","last_change":"2024-03-11T21:29:42.437099+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.437099+0000","last_undegraded":"2024-03-11T21:29:42.437099+0000","last_fullsized":"2024-03-11T21:29:42.437099+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"135.1a","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.435627+0000","last_change":"2024-03-11T21:29:42.435627+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.435627+0000","last_undegraded":"2024-03-11T21:29:42.435627+0000","last_fullsized":"2024-03-11T21:29:42.435627+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"142.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.14","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.433523+0000","last_change":"2024-03-11T21:29:42.433523+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.433523+0000","last_undegraded":"2024-03-11T21:29:42.433523+0000","last_fullsized":"2024-03-11T21:29:42.433523+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"135.1b","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.439574+0000","last_change":"2024-03-11T21:29:42.439574+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.439574+0000","last_undegraded":"2024-03-11T21:29:42.439574+0000","last_fullsized":"2024-03-11T21:29:42.439574+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.13","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.445388+0000","last_change":"2024-03-11T21:29:42.445388+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.445388+0000","last_undegraded":"2024-03-11T21:29:42.445388+0000","last_fullsized":"2024-03-11T21:29:42.445388+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"135.1c","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.445616+0000","last_change":"2024-03-11T21:29:42.445616+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.445616+0000","last_undegraded":"2024-03-11T21:29:42.445616+0000","last_fullsized":"2024-03-11T21:29:42.445616+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"142.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.1d","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438098+0000","last_change":"2024-03-11T21:29:42.438098+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438098+0000","last_undegraded":"2024-03-11T21:29:42.438098+0000","last_fullsized":"2024-03-11T21:29:42.438098+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.1e","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438840+0000","last_change":"2024-03-11T21:29:42.438840+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438840+0000","last_undegraded":"2024-03-11T21:29:42.438840+0000","last_fullsized":"2024-03-11T21:29:42.438840+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.10","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.449416+0000","last_change":"2024-03-11T21:29:42.449416+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.449416+0000","last_undegraded":"2024-03-11T21:29:42.449416+0000","last_fullsized":"2024-03-11T21:29:42.449416+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"135.1f","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438200+0000","last_change":"2024-03-11T21:29:42.438200+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438200+0000","last_undegraded":"2024-03-11T21:29:42.438200+0000","last_fullsized":"2024-03-11T21:29:42.438200+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.10","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.435934+0000","last_change":"2024-03-11T21:29:42.435934+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.435934+0000","last_undegraded":"2024-03-11T21:29:42.435934+0000","last_fullsized":"2024-03-11T21:29:42.435934+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"142.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.11","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432025+0000","last_change":"2024-03-11T21:29:42.432025+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432025+0000","last_undegraded":"2024-03-11T21:29:42.432025+0000","last_fullsized":"2024-03-11T21:29:42.432025+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1d","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432856+0000","last_change":"2024-03-11T21:29:42.432856+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432856+0000","last_undegraded":"2024-03-11T21:29:42.432856+0000","last_fullsized":"2024-03-11T21:29:42.432856+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"135.12","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.435833+0000","last_change":"2024-03-11T21:29:42.435833+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.435833+0000","last_undegraded":"2024-03-11T21:29:42.435833+0000","last_fullsized":"2024-03-11T21:29:42.435833+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1b","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.440014+0000","last_change":"2024-03-11T21:29:42.440014+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.440014+0000","last_undegraded":"2024-03-11T21:29:42.440014+0000","last_fullsized":"2024-03-11T21:29:42.440014+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"135.14","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432176+0000","last_change":"2024-03-11T21:29:42.432176+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432176+0000","last_undegraded":"2024-03-11T21:29:42.432176+0000","last_fullsized":"2024-03-11T21:29:42.432176+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1a","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.449876+0000","last_change":"2024-03-11T21:29:42.449876+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.449876+0000","last_undegraded":"2024-03-11T21:29:42.449876+0000","last_fullsized":"2024-03-11T21:29:42.449876+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"135.15","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.554058+0000","last_change":"2024-03-11T21:29:42.554058+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.554058+0000","last_undegraded":"2024-03-11T21:29:42.554058+0000","last_fullsized":"2024-03-11T21:29:42.554058+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"142.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.19","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.433522+0000","last_change":"2024-03-11T21:29:42.433522+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.433522+0000","last_undegraded":"2024-03-11T21:29:42.433522+0000","last_fullsized":"2024-03-11T21:29:42.433522+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"135.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.17","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432364+0000","last_change":"2024-03-11T21:29:42.432364+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432364+0000","last_undegraded":"2024-03-11T21:29:42.432364+0000","last_fullsized":"2024-03-11T21:29:42.432364+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.7","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438592+0000","last_change":"2024-03-11T21:29:42.438592+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438592+0000","last_undegraded":"2024-03-11T21:29:42.438592+0000","last_fullsized":"2024-03-11T21:29:42.438592+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"135.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.9","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.449496+0000","last_change":"2024-03-11T21:29:42.449496+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.449496+0000","last_undegraded":"2024-03-11T21:29:42.449496+0000","last_fullsized":"2024-03-11T21:29:42.449496+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"142.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.5","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438507+0000","last_change":"2024-03-11T21:29:42.438507+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438507+0000","last_undegraded":"2024-03-11T21:29:42.438507+0000","last_fullsized":"2024-03-11T21:29:42.438507+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"135.a","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432611+0000","last_change":"2024-03-11T21:29:42.432611+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432611+0000","last_undegraded":"2024-03-11T21:29:42.432611+0000","last_fullsized":"2024-03-11T21:29:42.432611+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.4","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.436377+0000","last_change":"2024-03-11T21:29:42.436377+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.436377+0000","last_undegraded":"2024-03-11T21:29:42.436377+0000","last_fullsized":"2024-03-11T21:29:42.436377+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"135.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.3","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432184+0000","last_change":"2024-03-11T21:29:42.432184+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432184+0000","last_undegraded":"2024-03-11T21:29:42.432184+0000","last_fullsized":"2024-03-11T21:29:42.432184+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"135.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.2","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.433168+0000","last_change":"2024-03-11T21:29:42.433168+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.433168+0000","last_undegraded":"2024-03-11T21:29:42.433168+0000","last_fullsized":"2024-03-11T21:29:42.433168+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"135.d","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432407+0000","last_change":"2024-03-11T21:29:42.432407+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432407+0000","last_undegraded":"2024-03-11T21:29:42.432407+0000","last_fullsized":"2024-03-11T21:29:42.432407+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.e","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.450825+0000","last_change":"2024-03-11T21:29:42.450825+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.450825+0000","last_undegraded":"2024-03-11T21:29:42.450825+0000","last_fullsized":"2024-03-11T21:29:42.450825+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"142.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"137.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.0","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.435811+0000","last_change":"2024-03-11T21:29:42.435811+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.435811+0000","last_undegraded":"2024-03-11T21:29:42.435811+0000","last_fullsized":"2024-03-11T21:29:42.435811+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"135.f","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.437316+0000","last_change":"2024-03-11T21:29:42.437316+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.437316+0000","last_undegraded":"2024-03-11T21:29:42.437316+0000","last_fullsized":"2024-03-11T21:29:42.437316+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"142.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.f","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.446530+0000","last_change":"2024-03-11T21:29:42.446530+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.446530+0000","last_undegraded":"2024-03-11T21:29:42.446530+0000","last_fullsized":"2024-03-11T21:29:42.446530+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"135.0","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.435422+0000","last_change":"2024-03-11T21:29:42.435422+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.435422+0000","last_undegraded":"2024-03-11T21:29:42.435422+0000","last_fullsized":"2024-03-11T21:29:42.435422+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"142.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.d","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.446450+0000","last_change":"2024-03-11T21:29:42.446450+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.446450+0000","last_undegraded":"2024-03-11T21:29:42.446450+0000","last_fullsized":"2024-03-11T21:29:42.446450+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"135.2","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.432039+0000","last_change":"2024-03-11T21:29:42.432039+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.432039+0000","last_undegraded":"2024-03-11T21:29:42.432039+0000","last_fullsized":"2024-03-11T21:29:42.432039+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"142.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.c","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.436783+0000","last_change":"2024-03-11T21:29:42.436783+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.436783+0000","last_undegraded":"2024-03-11T21:29:42.436783+0000","last_fullsized":"2024-03-11T21:29:42.436783+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"135.3","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.444881+0000","last_change":"2024-03-11T21:29:42.444881+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.444881+0000","last_undegraded":"2024-03-11T21:29:42.444881+0000","last_fullsized":"2024-03-11T21:29:42.444881+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"142.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.b","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.436625+0000","last_change":"2024-03-11T21:29:42.436625+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.436625+0000","last_undegraded":"2024-03-11T21:29:42.436625+0000","last_fullsized":"2024-03-11T21:29:42.436625+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"135.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"142.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.a","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.446745+0000","last_change":"2024-03-11T21:29:42.446745+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.446745+0000","last_undegraded":"2024-03-11T21:29:42.446745+0000","last_fullsized":"2024-03-11T21:29:42.446745+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"135.5","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.552693+0000","last_change":"2024-03-11T21:29:42.552693+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.552693+0000","last_undegraded":"2024-03-11T21:29:42.552693+0000","last_fullsized":"2024-03-11T21:29:42.552693+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"142.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.6","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.438447+0000","last_change":"2024-03-11T21:29:42.438447+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.438447+0000","last_undegraded":"2024-03-11T21:29:42.438447+0000","last_fullsized":"2024-03-11T21:29:42.438447+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"142.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"141.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"140.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:45.088676+0000","last_change":"2024-03-11T21:29:45.088676+0000","last_active":"2024-03-11T21:29:45.088676+0000","last_peered":"2024-03-11T21:29:45.088676+0000","last_clean":"2024-03-11T21:29:45.088676+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:45.088676+0000","last_undegraded":"2024-03-11T21:29:45.088676+0000","last_fullsized":"2024-03-11T21:29:45.088676+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:45.088676+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.088676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"139.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:43.743422+0000","last_change":"2024-03-11T21:29:43.743422+0000","last_active":"2024-03-11T21:29:43.743422+0000","last_peered":"2024-03-11T21:29:43.743422+0000","last_clean":"2024-03-11T21:29:43.743422+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:43.743422+0000","last_undegraded":"2024-03-11T21:29:43.743422+0000","last_fullsized":"2024-03-11T21:29:43.743422+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:43.743422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.743422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"136.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:29:42.426965+0000","last_change":"2024-03-11T21:29:42.426965+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.426965+0000","last_undegraded":"2024-03-11T21:29:42.426965+0000","last_fullsized":"2024-03-11T21:29:42.426965+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"135.7","version":"0'0","reported_seq":1,"reported_epoch":182,"state":"creating+peering","last_fresh":"2024-03-11T21:29:42.436171+0000","last_change":"2024-03-11T21:29:42.436171+0000","last_active":"2024-03-11T21:29:42.426965+0000","last_peered":"2024-03-11T21:29:42.426965+0000","last_clean":"2024-03-11T21:29:42.426965+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:29:42.436171+0000","last_undegraded":"2024-03-11T21:29:42.436171+0000","last_fullsized":"2024-03-11T21:29:42.436171+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.426965+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.426965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.9","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.460790+0000","last_change":"2024-03-11T21:29:32.625214+0000","last_active":"2024-03-11T21:29:42.460790+0000","last_peered":"2024-03-11T21:29:42.460790+0000","last_clean":"2024-03-11T21:29:42.460790+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:29:42.460790+0000","last_undegraded":"2024-03-11T21:29:42.460790+0000","last_fullsized":"2024-03-11T21:29:42.460790+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.417288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"99.1c","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.467060+0000","last_change":"2024-03-11T21:29:41.251022+0000","last_active":"2024-03-11T21:29:42.467060+0000","last_peered":"2024-03-11T21:29:42.467060+0000","last_clean":"2024-03-11T21:29:42.467060+0000","last_became_active":"2024-03-11T21:29:20.265928+0000","last_became_peered":"2024-03-11T21:29:20.265928+0000","last_unstale":"2024-03-11T21:29:42.467060+0000","last_undegraded":"2024-03-11T21:29:42.467060+0000","last_fullsized":"2024-03-11T21:29:42.467060+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:37.879815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086722310000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.8","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443848+0000","last_change":"2024-03-11T21:29:32.605761+0000","last_active":"2024-03-11T21:29:42.443848+0000","last_peered":"2024-03-11T21:29:42.443848+0000","last_clean":"2024-03-11T21:29:42.443848+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:29:42.443848+0000","last_undegraded":"2024-03-11T21:29:42.443848+0000","last_fullsized":"2024-03-11T21:29:42.443848+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:51.934769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"99.1d","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684446+0000","last_change":"2024-03-11T21:29:41.298630+0000","last_active":"2024-03-11T21:29:42.684446+0000","last_peered":"2024-03-11T21:29:42.684446+0000","last_clean":"2024-03-11T21:29:42.684446+0000","last_became_active":"2024-03-11T21:29:20.398218+0000","last_became_peered":"2024-03-11T21:29:20.398218+0000","last_unstale":"2024-03-11T21:29:42.684446+0000","last_undegraded":"2024-03-11T21:29:42.684446+0000","last_fullsized":"2024-03-11T21:29:42.684446+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:25.973406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025917862,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.b","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461464+0000","last_change":"2024-03-11T21:29:32.639335+0000","last_active":"2024-03-11T21:29:42.461464+0000","last_peered":"2024-03-11T21:29:42.461464+0000","last_clean":"2024-03-11T21:29:42.461464+0000","last_became_active":"2024-03-11T21:29:32.638921+0000","last_became_peered":"2024-03-11T21:29:32.638921+0000","last_unstale":"2024-03-11T21:29:42.461464+0000","last_undegraded":"2024-03-11T21:29:42.461464+0000","last_fullsized":"2024-03-11T21:29:42.461464+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.952749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.1e","version":"169'6","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684624+0000","last_change":"2024-03-11T21:29:41.303681+0000","last_active":"2024-03-11T21:29:42.684624+0000","last_peered":"2024-03-11T21:29:42.684624+0000","last_clean":"2024-03-11T21:29:42.684624+0000","last_became_active":"2024-03-11T21:29:19.213433+0000","last_became_peered":"2024-03-11T21:29:19.213433+0000","last_unstale":"2024-03-11T21:29:42.684624+0000","last_undegraded":"2024-03-11T21:29:42.684624+0000","last_fullsized":"2024-03-11T21:29:42.684624+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:39.371243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014118639000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.a","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442659+0000","last_change":"2024-03-11T21:29:32.654519+0000","last_active":"2024-03-11T21:29:42.442659+0000","last_peered":"2024-03-11T21:29:42.442659+0000","last_clean":"2024-03-11T21:29:42.442659+0000","last_became_active":"2024-03-11T21:29:32.654069+0000","last_became_peered":"2024-03-11T21:29:32.654069+0000","last_unstale":"2024-03-11T21:29:42.442659+0000","last_undegraded":"2024-03-11T21:29:42.442659+0000","last_fullsized":"2024-03-11T21:29:42.442659+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:56.354110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"99.1f","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560263+0000","last_change":"2024-03-11T21:29:41.317161+0000","last_active":"2024-03-11T21:29:42.560263+0000","last_peered":"2024-03-11T21:29:42.560263+0000","last_clean":"2024-03-11T21:29:42.560263+0000","last_became_active":"2024-03-11T21:29:20.358271+0000","last_became_peered":"2024-03-11T21:29:20.358271+0000","last_unstale":"2024-03-11T21:29:42.560263+0000","last_undegraded":"2024-03-11T21:29:42.560263+0000","last_fullsized":"2024-03-11T21:29:42.560263+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.085261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034336242000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.d","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.677900+0000","last_change":"2024-03-11T21:29:32.624676+0000","last_active":"2024-03-11T21:29:42.677900+0000","last_peered":"2024-03-11T21:29:42.677900+0000","last_clean":"2024-03-11T21:29:42.677900+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:29:42.677900+0000","last_undegraded":"2024-03-11T21:29:42.677900+0000","last_fullsized":"2024-03-11T21:29:42.677900+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:34.534481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"99.18","version":"180'20","reported_seq":111,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:43.053604+0000","last_change":"2024-03-11T21:29:43.053953+0000","last_active":"2024-03-11T21:29:43.053604+0000","last_peered":"2024-03-11T21:29:43.053604+0000","last_clean":"2024-03-11T21:29:43.053604+0000","last_became_active":"2024-03-11T21:29:20.238204+0000","last_became_peered":"2024-03-11T21:29:20.238204+0000","last_unstale":"2024-03-11T21:29:43.053604+0000","last_undegraded":"2024-03-11T21:29:43.053604+0000","last_fullsized":"2024-03-11T21:29:43.053604+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:34.185467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50416650399999996,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.c","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.451975+0000","last_change":"2024-03-11T21:29:32.636498+0000","last_active":"2024-03-11T21:29:42.451975+0000","last_peered":"2024-03-11T21:29:42.451975+0000","last_clean":"2024-03-11T21:29:42.451975+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:29:42.451975+0000","last_undegraded":"2024-03-11T21:29:42.451975+0000","last_fullsized":"2024-03-11T21:29:42.451975+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:36.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"99.19","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.435344+0000","last_change":"2024-03-11T21:29:41.219628+0000","last_active":"2024-03-11T21:29:42.435344+0000","last_peered":"2024-03-11T21:29:42.435344+0000","last_clean":"2024-03-11T21:29:42.435344+0000","last_became_active":"2024-03-11T21:29:20.230756+0000","last_became_peered":"2024-03-11T21:29:20.230756+0000","last_unstale":"2024-03-11T21:29:42.435344+0000","last_undegraded":"2024-03-11T21:29:42.435344+0000","last_fullsized":"2024-03-11T21:29:42.435344+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:22.072413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047504940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.f","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.440142+0000","last_change":"2024-03-11T21:29:32.602579+0000","last_active":"2024-03-11T21:29:42.440142+0000","last_peered":"2024-03-11T21:29:42.440142+0000","last_clean":"2024-03-11T21:29:42.440142+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:29:42.440142+0000","last_undegraded":"2024-03-11T21:29:42.440142+0000","last_fullsized":"2024-03-11T21:29:42.440142+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.343162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"99.1a","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466317+0000","last_change":"2024-03-11T21:29:41.226225+0000","last_active":"2024-03-11T21:29:42.466317+0000","last_peered":"2024-03-11T21:29:42.466317+0000","last_clean":"2024-03-11T21:29:42.466317+0000","last_became_active":"2024-03-11T21:29:20.202298+0000","last_became_peered":"2024-03-11T21:29:20.202298+0000","last_unstale":"2024-03-11T21:29:42.466317+0000","last_undegraded":"2024-03-11T21:29:42.466317+0000","last_fullsized":"2024-03-11T21:29:42.466317+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096643110000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.e","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443159+0000","last_change":"2024-03-11T21:29:32.606674+0000","last_active":"2024-03-11T21:29:42.443159+0000","last_peered":"2024-03-11T21:29:42.443159+0000","last_clean":"2024-03-11T21:29:42.443159+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:29:42.443159+0000","last_undegraded":"2024-03-11T21:29:42.443159+0000","last_fullsized":"2024-03-11T21:29:42.443159+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:55.055012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"99.1b","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552175+0000","last_change":"2024-03-11T21:29:41.306721+0000","last_active":"2024-03-11T21:29:42.552175+0000","last_peered":"2024-03-11T21:29:42.552175+0000","last_clean":"2024-03-11T21:29:42.552175+0000","last_became_active":"2024-03-11T21:29:20.236012+0000","last_became_peered":"2024-03-11T21:29:20.236012+0000","last_unstale":"2024-03-11T21:29:42.552175+0000","last_undegraded":"2024-03-11T21:29:42.552175+0000","last_fullsized":"2024-03-11T21:29:42.552175+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:42.481757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.075582957000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672243+0000","last_change":"2024-03-11T21:29:32.626486+0000","last_active":"2024-03-11T21:29:42.672243+0000","last_peered":"2024-03-11T21:29:42.672243+0000","last_clean":"2024-03-11T21:29:42.672243+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:29:42.672243+0000","last_undegraded":"2024-03-11T21:29:42.672243+0000","last_fullsized":"2024-03-11T21:29:42.672243+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.678606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.14","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447066+0000","last_change":"2024-03-11T21:29:41.246933+0000","last_active":"2024-03-11T21:29:42.447066+0000","last_peered":"2024-03-11T21:29:42.447066+0000","last_clean":"2024-03-11T21:29:42.447066+0000","last_became_active":"2024-03-11T21:29:20.258757+0000","last_became_peered":"2024-03-11T21:29:20.258757+0000","last_unstale":"2024-03-11T21:29:42.447066+0000","last_undegraded":"2024-03-11T21:29:42.447066+0000","last_fullsized":"2024-03-11T21:29:42.447066+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.846905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059684380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.0","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672062+0000","last_change":"2024-03-11T21:29:32.660797+0000","last_active":"2024-03-11T21:29:42.672062+0000","last_peered":"2024-03-11T21:29:42.672062+0000","last_clean":"2024-03-11T21:29:42.672062+0000","last_became_active":"2024-03-11T21:29:32.660539+0000","last_became_peered":"2024-03-11T21:29:32.660539+0000","last_unstale":"2024-03-11T21:29:42.672062+0000","last_undegraded":"2024-03-11T21:29:42.672062+0000","last_fullsized":"2024-03-11T21:29:42.672062+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.341628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.15","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452021+0000","last_change":"2024-03-11T21:29:41.248172+0000","last_active":"2024-03-11T21:29:42.452021+0000","last_peered":"2024-03-11T21:29:42.452021+0000","last_clean":"2024-03-11T21:29:42.452021+0000","last_became_active":"2024-03-11T21:29:20.214600+0000","last_became_peered":"2024-03-11T21:29:20.214600+0000","last_unstale":"2024-03-11T21:29:42.452021+0000","last_undegraded":"2024-03-11T21:29:42.452021+0000","last_fullsized":"2024-03-11T21:29:42.452021+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:19.462626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059012200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.3","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461576+0000","last_change":"2024-03-11T21:29:32.649490+0000","last_active":"2024-03-11T21:29:42.461576+0000","last_peered":"2024-03-11T21:29:42.461576+0000","last_clean":"2024-03-11T21:29:42.461576+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:29:42.461576+0000","last_undegraded":"2024-03-11T21:29:42.461576+0000","last_fullsized":"2024-03-11T21:29:42.461576+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:58.708586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.16","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448130+0000","last_change":"2024-03-11T21:29:41.245503+0000","last_active":"2024-03-11T21:29:42.448130+0000","last_peered":"2024-03-11T21:29:42.448130+0000","last_clean":"2024-03-11T21:29:42.448130+0000","last_became_active":"2024-03-11T21:29:20.398752+0000","last_became_peered":"2024-03-11T21:29:20.398752+0000","last_unstale":"2024-03-11T21:29:42.448130+0000","last_undegraded":"2024-03-11T21:29:42.448130+0000","last_fullsized":"2024-03-11T21:29:42.448130+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:40.570382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047298460000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.2","version":"177'1","reported_seq":34,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442971+0000","last_change":"2024-03-11T21:29:32.631702+0000","last_active":"2024-03-11T21:29:42.442971+0000","last_peered":"2024-03-11T21:29:42.442971+0000","last_clean":"2024-03-11T21:29:42.442971+0000","last_became_active":"2024-03-11T21:29:32.631274+0000","last_became_peered":"2024-03-11T21:29:32.631274+0000","last_unstale":"2024-03-11T21:29:42.442971+0000","last_undegraded":"2024-03-11T21:29:42.442971+0000","last_fullsized":"2024-03-11T21:29:42.442971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:58.202033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"99.17","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.440306+0000","last_change":"2024-03-11T21:29:41.240707+0000","last_active":"2024-03-11T21:29:42.440306+0000","last_peered":"2024-03-11T21:29:42.440306+0000","last_clean":"2024-03-11T21:29:42.440306+0000","last_became_active":"2024-03-11T21:29:20.391082+0000","last_became_peered":"2024-03-11T21:29:20.391082+0000","last_unstale":"2024-03-11T21:29:42.440306+0000","last_undegraded":"2024-03-11T21:29:42.440306+0000","last_fullsized":"2024-03-11T21:29:42.440306+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:25.655153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082720910000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.5","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.678450+0000","last_change":"2024-03-11T21:29:32.610516+0000","last_active":"2024-03-11T21:29:42.678450+0000","last_peered":"2024-03-11T21:29:42.678450+0000","last_clean":"2024-03-11T21:29:42.678450+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:29:42.678450+0000","last_undegraded":"2024-03-11T21:29:42.678450+0000","last_fullsized":"2024-03-11T21:29:42.678450+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:38.561152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"99.10","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.434718+0000","last_change":"2024-03-11T21:29:41.216071+0000","last_active":"2024-03-11T21:29:42.434718+0000","last_peered":"2024-03-11T21:29:42.434718+0000","last_clean":"2024-03-11T21:29:42.434718+0000","last_became_active":"2024-03-11T21:29:20.206566+0000","last_became_peered":"2024-03-11T21:29:20.206566+0000","last_unstale":"2024-03-11T21:29:42.434718+0000","last_undegraded":"2024-03-11T21:29:42.434718+0000","last_fullsized":"2024-03-11T21:29:42.434718+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.884100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061193189999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.4","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.687144+0000","last_change":"2024-03-11T21:29:32.662811+0000","last_active":"2024-03-11T21:29:42.687144+0000","last_peered":"2024-03-11T21:29:42.687144+0000","last_clean":"2024-03-11T21:29:42.687144+0000","last_became_active":"2024-03-11T21:29:32.662525+0000","last_became_peered":"2024-03-11T21:29:32.662525+0000","last_unstale":"2024-03-11T21:29:42.687144+0000","last_undegraded":"2024-03-11T21:29:42.687144+0000","last_fullsized":"2024-03-11T21:29:42.687144+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:52.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.11","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458349+0000","last_change":"2024-03-11T21:29:41.230587+0000","last_active":"2024-03-11T21:29:42.458349+0000","last_peered":"2024-03-11T21:29:42.458349+0000","last_clean":"2024-03-11T21:29:42.458349+0000","last_became_active":"2024-03-11T21:29:20.348388+0000","last_became_peered":"2024-03-11T21:29:20.348388+0000","last_unstale":"2024-03-11T21:29:42.458349+0000","last_undegraded":"2024-03-11T21:29:42.458349+0000","last_fullsized":"2024-03-11T21:29:42.458349+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.998791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023633109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.7","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.772396+0000","last_change":"2024-03-11T21:29:32.640524+0000","last_active":"2024-03-11T21:29:42.772396+0000","last_peered":"2024-03-11T21:29:42.772396+0000","last_clean":"2024-03-11T21:29:42.772396+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:29:42.772396+0000","last_undegraded":"2024-03-11T21:29:42.772396+0000","last_fullsized":"2024-03-11T21:29:42.772396+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:36.631818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"99.12","version":"182'1","reported_seq":91,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.440905+0000","last_change":"2024-03-11T21:29:41.215523+0000","last_active":"2024-03-11T21:29:42.440905+0000","last_peered":"2024-03-11T21:29:42.440905+0000","last_clean":"2024-03-11T21:29:42.440905+0000","last_became_active":"2024-03-11T21:29:20.214494+0000","last_became_peered":"2024-03-11T21:29:20.214494+0000","last_unstale":"2024-03-11T21:29:42.440905+0000","last_undegraded":"2024-03-11T21:29:42.440905+0000","last_fullsized":"2024-03-11T21:29:42.440905+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.216487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051836360000000001,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.6","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.762160+0000","last_change":"2024-03-11T21:29:32.660722+0000","last_active":"2024-03-11T21:29:42.762160+0000","last_peered":"2024-03-11T21:29:42.762160+0000","last_clean":"2024-03-11T21:29:42.762160+0000","last_became_active":"2024-03-11T21:29:32.660445+0000","last_became_peered":"2024-03-11T21:29:32.660445+0000","last_unstale":"2024-03-11T21:29:42.762160+0000","last_undegraded":"2024-03-11T21:29:42.762160+0000","last_fullsized":"2024-03-11T21:29:42.762160+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:42.241705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"99.13","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.670737+0000","last_change":"2024-03-11T21:29:41.286295+0000","last_active":"2024-03-11T21:29:42.670737+0000","last_peered":"2024-03-11T21:29:42.670737+0000","last_clean":"2024-03-11T21:29:42.670737+0000","last_became_active":"2024-03-11T21:29:20.391583+0000","last_became_peered":"2024-03-11T21:29:20.391583+0000","last_unstale":"2024-03-11T21:29:42.670737+0000","last_undegraded":"2024-03-11T21:29:42.670737+0000","last_fullsized":"2024-03-11T21:29:42.670737+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:29.435835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024887657000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.19","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.667786+0000","last_change":"2024-03-11T21:29:32.629369+0000","last_active":"2024-03-11T21:29:42.667786+0000","last_peered":"2024-03-11T21:29:42.667786+0000","last_clean":"2024-03-11T21:29:42.667786+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:29:42.667786+0000","last_undegraded":"2024-03-11T21:29:42.667786+0000","last_fullsized":"2024-03-11T21:29:42.667786+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:52.619580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.c","version":"0'0","reported_seq":92,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.772811+0000","last_change":"2024-03-11T21:29:41.243529+0000","last_active":"2024-03-11T21:29:42.772811+0000","last_peered":"2024-03-11T21:29:42.772811+0000","last_clean":"2024-03-11T21:29:42.772811+0000","last_became_active":"2024-03-11T21:29:19.149814+0000","last_became_peered":"2024-03-11T21:29:19.149814+0000","last_unstale":"2024-03-11T21:29:42.772811+0000","last_undegraded":"2024-03-11T21:29:42.772811+0000","last_fullsized":"2024-03-11T21:29:42.772811+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.372972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028523749000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.18","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.671782+0000","last_change":"2024-03-11T21:29:32.643824+0000","last_active":"2024-03-11T21:29:42.671782+0000","last_peered":"2024-03-11T21:29:42.671782+0000","last_clean":"2024-03-11T21:29:42.671782+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:29:42.671782+0000","last_undegraded":"2024-03-11T21:29:42.671782+0000","last_fullsized":"2024-03-11T21:29:42.671782+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:47.966430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.d","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.555792+0000","last_change":"2024-03-11T21:29:41.307199+0000","last_active":"2024-03-11T21:29:42.555792+0000","last_peered":"2024-03-11T21:29:42.555792+0000","last_clean":"2024-03-11T21:29:42.555792+0000","last_became_active":"2024-03-11T21:29:20.415437+0000","last_became_peered":"2024-03-11T21:29:20.415437+0000","last_unstale":"2024-03-11T21:29:42.555792+0000","last_undegraded":"2024-03-11T21:29:42.555792+0000","last_fullsized":"2024-03-11T21:29:42.555792+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:46.749967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.040013102000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1b","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.551971+0000","last_change":"2024-03-11T21:29:32.670508+0000","last_active":"2024-03-11T21:29:42.551971+0000","last_peered":"2024-03-11T21:29:42.551971+0000","last_clean":"2024-03-11T21:29:42.551971+0000","last_became_active":"2024-03-11T21:29:32.670334+0000","last_became_peered":"2024-03-11T21:29:32.670334+0000","last_unstale":"2024-03-11T21:29:42.551971+0000","last_undegraded":"2024-03-11T21:29:42.551971+0000","last_fullsized":"2024-03-11T21:29:42.551971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.836820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"99.e","version":"0'0","reported_seq":92,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.762801+0000","last_change":"2024-03-11T21:29:41.226845+0000","last_active":"2024-03-11T21:29:42.762801+0000","last_peered":"2024-03-11T21:29:42.762801+0000","last_clean":"2024-03-11T21:29:42.762801+0000","last_became_active":"2024-03-11T21:29:19.148818+0000","last_became_peered":"2024-03-11T21:29:19.148818+0000","last_unstale":"2024-03-11T21:29:42.762801+0000","last_undegraded":"2024-03-11T21:29:42.762801+0000","last_fullsized":"2024-03-11T21:29:42.762801+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:42.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024285550000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1a","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.667388+0000","last_change":"2024-03-11T21:29:32.654543+0000","last_active":"2024-03-11T21:29:42.667388+0000","last_peered":"2024-03-11T21:29:42.667388+0000","last_clean":"2024-03-11T21:29:42.667388+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:29:42.667388+0000","last_undegraded":"2024-03-11T21:29:42.667388+0000","last_fullsized":"2024-03-11T21:29:42.667388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:55.209904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"99.f","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560183+0000","last_change":"2024-03-11T21:29:41.316265+0000","last_active":"2024-03-11T21:29:42.560183+0000","last_peered":"2024-03-11T21:29:42.560183+0000","last_clean":"2024-03-11T21:29:42.560183+0000","last_became_active":"2024-03-11T21:29:20.276177+0000","last_became_peered":"2024-03-11T21:29:20.276177+0000","last_unstale":"2024-03-11T21:29:42.560183+0000","last_undegraded":"2024-03-11T21:29:42.560183+0000","last_fullsized":"2024-03-11T21:29:42.560183+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.081759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034084204999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1d","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454504+0000","last_change":"2024-03-11T21:29:32.661244+0000","last_active":"2024-03-11T21:29:42.454504+0000","last_peered":"2024-03-11T21:29:42.454504+0000","last_clean":"2024-03-11T21:29:42.454504+0000","last_became_active":"2024-03-11T21:29:32.660956+0000","last_became_peered":"2024-03-11T21:29:32.660956+0000","last_unstale":"2024-03-11T21:29:42.454504+0000","last_undegraded":"2024-03-11T21:29:42.454504+0000","last_fullsized":"2024-03-11T21:29:42.454504+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:48.758859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"99.8","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.660916+0000","last_change":"2024-03-11T21:29:41.281308+0000","last_active":"2024-03-11T21:29:42.660916+0000","last_peered":"2024-03-11T21:29:42.660916+0000","last_clean":"2024-03-11T21:29:42.660916+0000","last_became_active":"2024-03-11T21:29:20.413543+0000","last_became_peered":"2024-03-11T21:29:20.413543+0000","last_unstale":"2024-03-11T21:29:42.660916+0000","last_undegraded":"2024-03-11T21:29:42.660916+0000","last_fullsized":"2024-03-11T21:29:42.660916+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.375929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027137205000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1c","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466841+0000","last_change":"2024-03-11T21:29:32.654668+0000","last_active":"2024-03-11T21:29:42.466841+0000","last_peered":"2024-03-11T21:29:42.466841+0000","last_clean":"2024-03-11T21:29:42.466841+0000","last_became_active":"2024-03-11T21:29:32.654093+0000","last_became_peered":"2024-03-11T21:29:32.654093+0000","last_unstale":"2024-03-11T21:29:42.466841+0000","last_undegraded":"2024-03-11T21:29:42.466841+0000","last_fullsized":"2024-03-11T21:29:42.466841+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.162854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.9","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448088+0000","last_change":"2024-03-11T21:29:41.234220+0000","last_active":"2024-03-11T21:29:42.448088+0000","last_peered":"2024-03-11T21:29:42.448088+0000","last_clean":"2024-03-11T21:29:42.448088+0000","last_became_active":"2024-03-11T21:29:20.399380+0000","last_became_peered":"2024-03-11T21:29:20.399380+0000","last_unstale":"2024-03-11T21:29:42.448088+0000","last_undegraded":"2024-03-11T21:29:42.448088+0000","last_fullsized":"2024-03-11T21:29:42.448088+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:31.966903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027516760000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1f","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559962+0000","last_change":"2024-03-11T21:29:32.668228+0000","last_active":"2024-03-11T21:29:42.559962+0000","last_peered":"2024-03-11T21:29:42.559962+0000","last_clean":"2024-03-11T21:29:42.559962+0000","last_became_active":"2024-03-11T21:29:32.666162+0000","last_became_peered":"2024-03-11T21:29:32.666162+0000","last_unstale":"2024-03-11T21:29:42.559962+0000","last_undegraded":"2024-03-11T21:29:42.559962+0000","last_fullsized":"2024-03-11T21:29:42.559962+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:50.705068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"99.a","version":"173'9","reported_seq":104,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441003+0000","last_change":"2024-03-11T21:29:41.232561+0000","last_active":"2024-03-11T21:29:42.441003+0000","last_peered":"2024-03-11T21:29:42.441003+0000","last_clean":"2024-03-11T21:29:42.441003+0000","last_became_active":"2024-03-11T21:29:20.281226+0000","last_became_peered":"2024-03-11T21:29:20.281226+0000","last_unstale":"2024-03-11T21:29:42.441003+0000","last_undegraded":"2024-03-11T21:29:42.441003+0000","last_fullsized":"2024-03-11T21:29:42.441003+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:47.124329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016718980000000001,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.1e","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.470953+0000","last_change":"2024-03-11T21:29:32.618543+0000","last_active":"2024-03-11T21:29:42.470953+0000","last_peered":"2024-03-11T21:29:42.470953+0000","last_clean":"2024-03-11T21:29:42.470953+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:29:42.470953+0000","last_undegraded":"2024-03-11T21:29:42.470953+0000","last_fullsized":"2024-03-11T21:29:42.470953+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.084214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.b","version":"0'0","reported_seq":92,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458909+0000","last_change":"2024-03-11T21:29:41.262708+0000","last_active":"2024-03-11T21:29:42.458909+0000","last_peered":"2024-03-11T21:29:42.458909+0000","last_clean":"2024-03-11T21:29:42.458909+0000","last_became_active":"2024-03-11T21:29:19.144185+0000","last_became_peered":"2024-03-11T21:29:19.144185+0000","last_unstale":"2024-03-11T21:29:42.458909+0000","last_undegraded":"2024-03-11T21:29:42.458909+0000","last_fullsized":"2024-03-11T21:29:42.458909+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.965065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011537571,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.11","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.469067+0000","last_change":"2024-03-11T21:29:32.654693+0000","last_active":"2024-03-11T21:29:42.469067+0000","last_peered":"2024-03-11T21:29:42.469067+0000","last_clean":"2024-03-11T21:29:42.469067+0000","last_became_active":"2024-03-11T21:29:32.654370+0000","last_became_peered":"2024-03-11T21:29:32.654370+0000","last_unstale":"2024-03-11T21:29:42.469067+0000","last_undegraded":"2024-03-11T21:29:42.469067+0000","last_fullsized":"2024-03-11T21:29:42.469067+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:46.988272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.4","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.675481+0000","last_change":"2024-03-11T21:29:41.280569+0000","last_active":"2024-03-11T21:29:42.675481+0000","last_peered":"2024-03-11T21:29:42.675481+0000","last_clean":"2024-03-11T21:29:42.675481+0000","last_became_active":"2024-03-11T21:29:20.403523+0000","last_became_peered":"2024-03-11T21:29:20.403523+0000","last_unstale":"2024-03-11T21:29:42.675481+0000","last_undegraded":"2024-03-11T21:29:42.675481+0000","last_fullsized":"2024-03-11T21:29:42.675481+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:37.744904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023318220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.10","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443730+0000","last_change":"2024-03-11T21:29:32.623652+0000","last_active":"2024-03-11T21:29:42.443730+0000","last_peered":"2024-03-11T21:29:42.443730+0000","last_clean":"2024-03-11T21:29:42.443730+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:29:42.443730+0000","last_undegraded":"2024-03-11T21:29:42.443730+0000","last_fullsized":"2024-03-11T21:29:42.443730+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:00.087381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"99.5","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466197+0000","last_change":"2024-03-11T21:29:41.255969+0000","last_active":"2024-03-11T21:29:42.466197+0000","last_peered":"2024-03-11T21:29:42.466197+0000","last_clean":"2024-03-11T21:29:42.466197+0000","last_became_active":"2024-03-11T21:29:20.348172+0000","last_became_peered":"2024-03-11T21:29:20.348172+0000","last_unstale":"2024-03-11T21:29:42.466197+0000","last_undegraded":"2024-03-11T21:29:42.466197+0000","last_fullsized":"2024-03-11T21:29:42.466197+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:38.521684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010003323,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.13","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449645+0000","last_change":"2024-03-11T21:29:32.636980+0000","last_active":"2024-03-11T21:29:42.449645+0000","last_peered":"2024-03-11T21:29:42.449645+0000","last_clean":"2024-03-11T21:29:42.449645+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:29:42.449645+0000","last_undegraded":"2024-03-11T21:29:42.449645+0000","last_fullsized":"2024-03-11T21:29:42.449645+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:58.456026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"99.6","version":"0'0","reported_seq":92,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683263+0000","last_change":"2024-03-11T21:29:41.311810+0000","last_active":"2024-03-11T21:29:42.683263+0000","last_peered":"2024-03-11T21:29:42.683263+0000","last_clean":"2024-03-11T21:29:42.683263+0000","last_became_active":"2024-03-11T21:29:19.178530+0000","last_became_peered":"2024-03-11T21:29:19.178530+0000","last_unstale":"2024-03-11T21:29:42.683263+0000","last_undegraded":"2024-03-11T21:29:42.683263+0000","last_fullsized":"2024-03-11T21:29:42.683263+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.708700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022083001000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.12","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466241+0000","last_change":"2024-03-11T21:29:32.636504+0000","last_active":"2024-03-11T21:29:42.466241+0000","last_peered":"2024-03-11T21:29:42.466241+0000","last_clean":"2024-03-11T21:29:42.466241+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:29:42.466241+0000","last_undegraded":"2024-03-11T21:29:42.466241+0000","last_fullsized":"2024-03-11T21:29:42.466241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:48.443525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.7","version":"0'0","reported_seq":92,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.686799+0000","last_change":"2024-03-11T21:29:41.303468+0000","last_active":"2024-03-11T21:29:42.686799+0000","last_peered":"2024-03-11T21:29:42.686799+0000","last_clean":"2024-03-11T21:29:42.686799+0000","last_became_active":"2024-03-11T21:29:19.189952+0000","last_became_peered":"2024-03-11T21:29:19.189952+0000","last_unstale":"2024-03-11T21:29:42.686799+0000","last_undegraded":"2024-03-11T21:29:42.686799+0000","last_fullsized":"2024-03-11T21:29:42.686799+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:19.134547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017682297,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.15","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.678056+0000","last_change":"2024-03-11T21:29:32.610303+0000","last_active":"2024-03-11T21:29:42.678056+0000","last_peered":"2024-03-11T21:29:42.678056+0000","last_clean":"2024-03-11T21:29:42.678056+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:29:42.678056+0000","last_undegraded":"2024-03-11T21:29:42.678056+0000","last_fullsized":"2024-03-11T21:29:42.678056+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:51.751347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"99.0","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.438192+0000","last_change":"2024-03-11T21:29:41.230882+0000","last_active":"2024-03-11T21:29:42.438192+0000","last_peered":"2024-03-11T21:29:42.438192+0000","last_clean":"2024-03-11T21:29:42.438192+0000","last_became_active":"2024-03-11T21:29:20.259101+0000","last_became_peered":"2024-03-11T21:29:20.259101+0000","last_unstale":"2024-03-11T21:29:42.438192+0000","last_undegraded":"2024-03-11T21:29:42.438192+0000","last_fullsized":"2024-03-11T21:29:42.438192+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:47.826030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012319449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.14","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449507+0000","last_change":"2024-03-11T21:29:32.639319+0000","last_active":"2024-03-11T21:29:42.449507+0000","last_peered":"2024-03-11T21:29:42.449507+0000","last_clean":"2024-03-11T21:29:42.449507+0000","last_became_active":"2024-03-11T21:29:32.638805+0000","last_became_peered":"2024-03-11T21:29:32.638805+0000","last_unstale":"2024-03-11T21:29:42.449507+0000","last_undegraded":"2024-03-11T21:29:42.449507+0000","last_fullsized":"2024-03-11T21:29:42.449507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:57.661337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"99.1","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552733+0000","last_change":"2024-03-11T21:29:41.315830+0000","last_active":"2024-03-11T21:29:42.552733+0000","last_peered":"2024-03-11T21:29:42.552733+0000","last_clean":"2024-03-11T21:29:42.552733+0000","last_became_active":"2024-03-11T21:29:20.400829+0000","last_became_peered":"2024-03-11T21:29:20.400829+0000","last_unstale":"2024-03-11T21:29:42.552733+0000","last_undegraded":"2024-03-11T21:29:42.552733+0000","last_fullsized":"2024-03-11T21:29:42.552733+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:34.949017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048472940999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.17","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.679696+0000","last_change":"2024-03-11T21:29:32.648189+0000","last_active":"2024-03-11T21:29:42.679696+0000","last_peered":"2024-03-11T21:29:42.679696+0000","last_clean":"2024-03-11T21:29:42.679696+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:29:42.679696+0000","last_undegraded":"2024-03-11T21:29:42.679696+0000","last_fullsized":"2024-03-11T21:29:42.679696+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:47.458714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"99.2","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443388+0000","last_change":"2024-03-11T21:29:41.224526+0000","last_active":"2024-03-11T21:29:42.443388+0000","last_peered":"2024-03-11T21:29:42.443388+0000","last_clean":"2024-03-11T21:29:42.443388+0000","last_became_active":"2024-03-11T21:29:20.427234+0000","last_became_peered":"2024-03-11T21:29:20.427234+0000","last_unstale":"2024-03-11T21:29:42.443388+0000","last_undegraded":"2024-03-11T21:29:42.443388+0000","last_fullsized":"2024-03-11T21:29:42.443388+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:39.866542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049170400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"118.16","version":"0'0","reported_seq":32,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.471056+0000","last_change":"2024-03-11T21:29:32.649595+0000","last_active":"2024-03-11T21:29:42.471056+0000","last_peered":"2024-03-11T21:29:42.471056+0000","last_clean":"2024-03-11T21:29:42.471056+0000","last_became_active":"2024-03-11T21:29:32.649241+0000","last_became_peered":"2024-03-11T21:29:32.649241+0000","last_unstale":"2024-03-11T21:29:42.471056+0000","last_undegraded":"2024-03-11T21:29:42.471056+0000","last_fullsized":"2024-03-11T21:29:42.471056+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:49.709466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"99.3","version":"0'0","reported_seq":90,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.463912+0000","last_change":"2024-03-11T21:29:41.236231+0000","last_active":"2024-03-11T21:29:42.463912+0000","last_peered":"2024-03-11T21:29:42.463912+0000","last_clean":"2024-03-11T21:29:42.463912+0000","last_became_active":"2024-03-11T21:29:20.267293+0000","last_became_peered":"2024-03-11T21:29:20.267293+0000","last_unstale":"2024-03-11T21:29:42.463912+0000","last_undegraded":"2024-03-11T21:29:42.463912+0000","last_fullsized":"2024-03-11T21:29:42.463912+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:17.887691+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.887691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:28.955585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"69.1a","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441381+0000","last_change":"2024-03-11T21:29:06.521753+0000","last_active":"2024-03-11T21:29:42.441381+0000","last_peered":"2024-03-11T21:29:42.441381+0000","last_clean":"2024-03-11T21:29:42.441381+0000","last_became_active":"2024-03-11T21:29:06.500113+0000","last_became_peered":"2024-03-11T21:29:06.500113+0000","last_unstale":"2024-03-11T21:29:42.441381+0000","last_undegraded":"2024-03-11T21:29:42.441381+0000","last_fullsized":"2024-03-11T21:29:42.441381+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.982933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"69.1b","version":"153'1","reported_seq":77,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552249+0000","last_change":"2024-03-11T21:29:05.399839+0000","last_active":"2024-03-11T21:29:42.552249+0000","last_peered":"2024-03-11T21:29:42.552249+0000","last_clean":"2024-03-11T21:29:42.552249+0000","last_became_active":"2024-03-11T21:29:05.399541+0000","last_became_peered":"2024-03-11T21:29:05.399541+0000","last_unstale":"2024-03-11T21:29:42.552249+0000","last_undegraded":"2024-03-11T21:29:42.552249+0000","last_fullsized":"2024-03-11T21:29:42.552249+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:28.410705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.18","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661509+0000","last_change":"2024-03-11T21:29:06.577883+0000","last_active":"2024-03-11T21:29:42.661509+0000","last_peered":"2024-03-11T21:29:42.661509+0000","last_clean":"2024-03-11T21:29:42.661509+0000","last_became_active":"2024-03-11T21:29:06.577604+0000","last_became_peered":"2024-03-11T21:29:06.577604+0000","last_unstale":"2024-03-11T21:29:42.661509+0000","last_undegraded":"2024-03-11T21:29:42.661509+0000","last_fullsized":"2024-03-11T21:29:42.661509+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.873608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"69.19","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447739+0000","last_change":"2024-03-11T21:29:06.540387+0000","last_active":"2024-03-11T21:29:42.447739+0000","last_peered":"2024-03-11T21:29:42.447739+0000","last_clean":"2024-03-11T21:29:42.447739+0000","last_became_active":"2024-03-11T21:29:06.538699+0000","last_became_peered":"2024-03-11T21:29:06.538699+0000","last_unstale":"2024-03-11T21:29:42.447739+0000","last_undegraded":"2024-03-11T21:29:42.447739+0000","last_fullsized":"2024-03-11T21:29:42.447739+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:25.638116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"69.1e","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559687+0000","last_change":"2024-03-11T21:29:05.423381+0000","last_active":"2024-03-11T21:29:42.559687+0000","last_peered":"2024-03-11T21:29:42.559687+0000","last_clean":"2024-03-11T21:29:42.559687+0000","last_became_active":"2024-03-11T21:29:05.422986+0000","last_became_peered":"2024-03-11T21:29:05.422986+0000","last_unstale":"2024-03-11T21:29:42.559687+0000","last_undegraded":"2024-03-11T21:29:42.559687+0000","last_fullsized":"2024-03-11T21:29:42.559687+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.589371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.1f","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.473005+0000","last_change":"2024-03-11T21:29:06.490576+0000","last_active":"2024-03-11T21:29:42.473005+0000","last_peered":"2024-03-11T21:29:42.473005+0000","last_clean":"2024-03-11T21:29:42.473005+0000","last_became_active":"2024-03-11T21:29:06.490318+0000","last_became_peered":"2024-03-11T21:29:06.490318+0000","last_unstale":"2024-03-11T21:29:42.473005+0000","last_undegraded":"2024-03-11T21:29:42.473005+0000","last_fullsized":"2024-03-11T21:29:42.473005+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.988596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"69.1c","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454970+0000","last_change":"2024-03-11T21:29:06.497008+0000","last_active":"2024-03-11T21:29:42.454970+0000","last_peered":"2024-03-11T21:29:42.454970+0000","last_clean":"2024-03-11T21:29:42.454970+0000","last_became_active":"2024-03-11T21:29:06.496524+0000","last_became_peered":"2024-03-11T21:29:06.496524+0000","last_unstale":"2024-03-11T21:29:42.454970+0000","last_undegraded":"2024-03-11T21:29:42.454970+0000","last_fullsized":"2024-03-11T21:29:42.454970+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.235650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"69.1d","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444616+0000","last_change":"2024-03-11T21:29:06.511620+0000","last_active":"2024-03-11T21:29:42.444616+0000","last_peered":"2024-03-11T21:29:42.444616+0000","last_clean":"2024-03-11T21:29:42.444616+0000","last_became_active":"2024-03-11T21:29:06.496319+0000","last_became_peered":"2024-03-11T21:29:06.496319+0000","last_unstale":"2024-03-11T21:29:42.444616+0000","last_undegraded":"2024-03-11T21:29:42.444616+0000","last_fullsized":"2024-03-11T21:29:42.444616+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:10.761689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"69.12","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466345+0000","last_change":"2024-03-11T21:29:06.504935+0000","last_active":"2024-03-11T21:29:42.466345+0000","last_peered":"2024-03-11T21:29:42.466345+0000","last_clean":"2024-03-11T21:29:42.466345+0000","last_became_active":"2024-03-11T21:29:06.504288+0000","last_became_peered":"2024-03-11T21:29:06.504288+0000","last_unstale":"2024-03-11T21:29:42.466345+0000","last_undegraded":"2024-03-11T21:29:42.466345+0000","last_fullsized":"2024-03-11T21:29:42.466345+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.069822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"69.13","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444341+0000","last_change":"2024-03-11T21:29:06.527434+0000","last_active":"2024-03-11T21:29:42.444341+0000","last_peered":"2024-03-11T21:29:42.444341+0000","last_clean":"2024-03-11T21:29:42.444341+0000","last_became_active":"2024-03-11T21:29:06.526743+0000","last_became_peered":"2024-03-11T21:29:06.526743+0000","last_unstale":"2024-03-11T21:29:42.444341+0000","last_undegraded":"2024-03-11T21:29:42.444341+0000","last_fullsized":"2024-03-11T21:29:42.444341+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:22.324927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"69.10","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672952+0000","last_change":"2024-03-11T21:29:06.494392+0000","last_active":"2024-03-11T21:29:42.672952+0000","last_peered":"2024-03-11T21:29:42.672952+0000","last_clean":"2024-03-11T21:29:42.672952+0000","last_became_active":"2024-03-11T21:29:06.493130+0000","last_became_peered":"2024-03-11T21:29:06.493130+0000","last_unstale":"2024-03-11T21:29:42.672952+0000","last_undegraded":"2024-03-11T21:29:42.672952+0000","last_fullsized":"2024-03-11T21:29:42.672952+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:27.354837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"69.11","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.436626+0000","last_change":"2024-03-11T21:29:06.504394+0000","last_active":"2024-03-11T21:29:42.436626+0000","last_peered":"2024-03-11T21:29:42.436626+0000","last_clean":"2024-03-11T21:29:42.436626+0000","last_became_active":"2024-03-11T21:29:06.503851+0000","last_became_peered":"2024-03-11T21:29:06.503851+0000","last_unstale":"2024-03-11T21:29:42.436626+0000","last_undegraded":"2024-03-11T21:29:42.436626+0000","last_fullsized":"2024-03-11T21:29:42.436626+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.254911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"69.16","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442333+0000","last_change":"2024-03-11T21:29:06.505606+0000","last_active":"2024-03-11T21:29:42.442333+0000","last_peered":"2024-03-11T21:29:42.442333+0000","last_clean":"2024-03-11T21:29:42.442333+0000","last_became_active":"2024-03-11T21:29:06.505293+0000","last_became_peered":"2024-03-11T21:29:06.505293+0000","last_unstale":"2024-03-11T21:29:42.442333+0000","last_undegraded":"2024-03-11T21:29:42.442333+0000","last_fullsized":"2024-03-11T21:29:42.442333+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.562288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"69.17","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.680982+0000","last_change":"2024-03-11T21:29:06.535303+0000","last_active":"2024-03-11T21:29:42.680982+0000","last_peered":"2024-03-11T21:29:42.680982+0000","last_clean":"2024-03-11T21:29:42.680982+0000","last_became_active":"2024-03-11T21:29:06.534788+0000","last_became_peered":"2024-03-11T21:29:06.534788+0000","last_unstale":"2024-03-11T21:29:42.680982+0000","last_undegraded":"2024-03-11T21:29:42.680982+0000","last_fullsized":"2024-03-11T21:29:42.680982+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.708777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"69.14","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.468752+0000","last_change":"2024-03-11T21:29:06.500595+0000","last_active":"2024-03-11T21:29:42.468752+0000","last_peered":"2024-03-11T21:29:42.468752+0000","last_clean":"2024-03-11T21:29:42.468752+0000","last_became_active":"2024-03-11T21:29:06.500303+0000","last_became_peered":"2024-03-11T21:29:06.500303+0000","last_unstale":"2024-03-11T21:29:42.468752+0000","last_undegraded":"2024-03-11T21:29:42.468752+0000","last_fullsized":"2024-03-11T21:29:42.468752+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:25.360775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"69.15","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.556228+0000","last_change":"2024-03-11T21:29:05.405336+0000","last_active":"2024-03-11T21:29:42.556228+0000","last_peered":"2024-03-11T21:29:42.556228+0000","last_clean":"2024-03-11T21:29:42.556228+0000","last_became_active":"2024-03-11T21:29:05.405043+0000","last_became_peered":"2024-03-11T21:29:05.405043+0000","last_unstale":"2024-03-11T21:29:42.556228+0000","last_undegraded":"2024-03-11T21:29:42.556228+0000","last_fullsized":"2024-03-11T21:29:42.556228+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:26.400862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.a","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443291+0000","last_change":"2024-03-11T21:29:06.554983+0000","last_active":"2024-03-11T21:29:42.443291+0000","last_peered":"2024-03-11T21:29:42.443291+0000","last_clean":"2024-03-11T21:29:42.443291+0000","last_became_active":"2024-03-11T21:29:06.554678+0000","last_became_peered":"2024-03-11T21:29:06.554678+0000","last_unstale":"2024-03-11T21:29:42.443291+0000","last_undegraded":"2024-03-11T21:29:42.443291+0000","last_fullsized":"2024-03-11T21:29:42.443291+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.388574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"69.b","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552315+0000","last_change":"2024-03-11T21:29:05.400761+0000","last_active":"2024-03-11T21:29:42.552315+0000","last_peered":"2024-03-11T21:29:42.552315+0000","last_clean":"2024-03-11T21:29:42.552315+0000","last_became_active":"2024-03-11T21:29:05.400396+0000","last_became_peered":"2024-03-11T21:29:05.400396+0000","last_unstale":"2024-03-11T21:29:42.552315+0000","last_undegraded":"2024-03-11T21:29:42.552315+0000","last_fullsized":"2024-03-11T21:29:42.552315+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:29.072832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.8","version":"153'3","reported_seq":78,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.438434+0000","last_change":"2024-03-11T21:29:06.485952+0000","last_active":"2024-03-11T21:29:42.438434+0000","last_peered":"2024-03-11T21:29:42.438434+0000","last_clean":"2024-03-11T21:29:42.438434+0000","last_became_active":"2024-03-11T21:29:06.485661+0000","last_became_peered":"2024-03-11T21:29:06.485661+0000","last_unstale":"2024-03-11T21:29:42.438434+0000","last_undegraded":"2024-03-11T21:29:42.438434+0000","last_fullsized":"2024-03-11T21:29:42.438434+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:14.277067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"69.9","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.435202+0000","last_change":"2024-03-11T21:29:06.504135+0000","last_active":"2024-03-11T21:29:42.435202+0000","last_peered":"2024-03-11T21:29:42.435202+0000","last_clean":"2024-03-11T21:29:42.435202+0000","last_became_active":"2024-03-11T21:29:06.494948+0000","last_became_peered":"2024-03-11T21:29:06.494948+0000","last_unstale":"2024-03-11T21:29:42.435202+0000","last_undegraded":"2024-03-11T21:29:42.435202+0000","last_fullsized":"2024-03-11T21:29:42.435202+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:28.004172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"69.e","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.558779+0000","last_change":"2024-03-11T21:29:05.403051+0000","last_active":"2024-03-11T21:29:42.558779+0000","last_peered":"2024-03-11T21:29:42.558779+0000","last_clean":"2024-03-11T21:29:42.558779+0000","last_became_active":"2024-03-11T21:29:05.402791+0000","last_became_peered":"2024-03-11T21:29:05.402791+0000","last_unstale":"2024-03-11T21:29:42.558779+0000","last_undegraded":"2024-03-11T21:29:42.558779+0000","last_fullsized":"2024-03-11T21:29:42.558779+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:28.709291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.f","version":"153'1","reported_seq":76,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688365+0000","last_change":"2024-03-11T21:29:06.500854+0000","last_active":"2024-03-11T21:29:42.688365+0000","last_peered":"2024-03-11T21:29:42.688365+0000","last_clean":"2024-03-11T21:29:42.688365+0000","last_became_active":"2024-03-11T21:29:06.500409+0000","last_became_peered":"2024-03-11T21:29:06.500409+0000","last_unstale":"2024-03-11T21:29:42.688365+0000","last_undegraded":"2024-03-11T21:29:42.688365+0000","last_fullsized":"2024-03-11T21:29:42.688365+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:23.693208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"69.c","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.677968+0000","last_change":"2024-03-11T21:29:06.543045+0000","last_active":"2024-03-11T21:29:42.677968+0000","last_peered":"2024-03-11T21:29:42.677968+0000","last_clean":"2024-03-11T21:29:42.677968+0000","last_became_active":"2024-03-11T21:29:06.540575+0000","last_became_peered":"2024-03-11T21:29:06.540575+0000","last_unstale":"2024-03-11T21:29:42.677968+0000","last_undegraded":"2024-03-11T21:29:42.677968+0000","last_fullsized":"2024-03-11T21:29:42.677968+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.353080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"69.d","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.681021+0000","last_change":"2024-03-11T21:29:06.519157+0000","last_active":"2024-03-11T21:29:42.681021+0000","last_peered":"2024-03-11T21:29:42.681021+0000","last_clean":"2024-03-11T21:29:42.681021+0000","last_became_active":"2024-03-11T21:29:06.518861+0000","last_became_peered":"2024-03-11T21:29:06.518861+0000","last_unstale":"2024-03-11T21:29:42.681021+0000","last_undegraded":"2024-03-11T21:29:42.681021+0000","last_fullsized":"2024-03-11T21:29:42.681021+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:12.693481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"69.2","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552062+0000","last_change":"2024-03-11T21:29:05.373810+0000","last_active":"2024-03-11T21:29:42.552062+0000","last_peered":"2024-03-11T21:29:42.552062+0000","last_clean":"2024-03-11T21:29:42.552062+0000","last_became_active":"2024-03-11T21:29:05.373532+0000","last_became_peered":"2024-03-11T21:29:05.373532+0000","last_unstale":"2024-03-11T21:29:42.552062+0000","last_undegraded":"2024-03-11T21:29:42.552062+0000","last_fullsized":"2024-03-11T21:29:42.552062+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.007318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.3","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.670978+0000","last_change":"2024-03-11T21:29:06.513583+0000","last_active":"2024-03-11T21:29:42.670978+0000","last_peered":"2024-03-11T21:29:42.670978+0000","last_clean":"2024-03-11T21:29:42.670978+0000","last_became_active":"2024-03-11T21:29:06.513172+0000","last_became_peered":"2024-03-11T21:29:06.513172+0000","last_unstale":"2024-03-11T21:29:42.670978+0000","last_undegraded":"2024-03-11T21:29:42.670978+0000","last_fullsized":"2024-03-11T21:29:42.670978+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:29.972425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"69.0","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448741+0000","last_change":"2024-03-11T21:29:06.453788+0000","last_active":"2024-03-11T21:29:42.448741+0000","last_peered":"2024-03-11T21:29:42.448741+0000","last_clean":"2024-03-11T21:29:42.448741+0000","last_became_active":"2024-03-11T21:29:06.448023+0000","last_became_peered":"2024-03-11T21:29:06.448023+0000","last_unstale":"2024-03-11T21:29:42.448741+0000","last_undegraded":"2024-03-11T21:29:42.448741+0000","last_fullsized":"2024-03-11T21:29:42.448741+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"69.1","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.669625+0000","last_change":"2024-03-11T21:29:06.494242+0000","last_active":"2024-03-11T21:29:42.669625+0000","last_peered":"2024-03-11T21:29:42.669625+0000","last_clean":"2024-03-11T21:29:42.669625+0000","last_became_active":"2024-03-11T21:29:06.490278+0000","last_became_peered":"2024-03-11T21:29:06.490278+0000","last_unstale":"2024-03-11T21:29:42.669625+0000","last_undegraded":"2024-03-11T21:29:42.669625+0000","last_fullsized":"2024-03-11T21:29:42.669625+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.585592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"69.6","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.678056+0000","last_change":"2024-03-11T21:29:06.541612+0000","last_active":"2024-03-11T21:29:42.678056+0000","last_peered":"2024-03-11T21:29:42.678056+0000","last_clean":"2024-03-11T21:29:42.678056+0000","last_became_active":"2024-03-11T21:29:06.540599+0000","last_became_peered":"2024-03-11T21:29:06.540599+0000","last_unstale":"2024-03-11T21:29:42.678056+0000","last_undegraded":"2024-03-11T21:29:42.678056+0000","last_fullsized":"2024-03-11T21:29:42.678056+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:26.659807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"69.7","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560305+0000","last_change":"2024-03-11T21:29:05.417927+0000","last_active":"2024-03-11T21:29:42.560305+0000","last_peered":"2024-03-11T21:29:42.560305+0000","last_clean":"2024-03-11T21:29:42.560305+0000","last_became_active":"2024-03-11T21:29:05.417650+0000","last_became_peered":"2024-03-11T21:29:05.417650+0000","last_unstale":"2024-03-11T21:29:42.560305+0000","last_undegraded":"2024-03-11T21:29:42.560305+0000","last_fullsized":"2024-03-11T21:29:42.560305+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:17.864044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.4","version":"0'0","reported_seq":75,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554209+0000","last_change":"2024-03-11T21:29:05.395573+0000","last_active":"2024-03-11T21:29:42.554209+0000","last_peered":"2024-03-11T21:29:42.554209+0000","last_clean":"2024-03-11T21:29:42.554209+0000","last_became_active":"2024-03-11T21:29:05.395264+0000","last_became_peered":"2024-03-11T21:29:05.395264+0000","last_unstale":"2024-03-11T21:29:42.554209+0000","last_undegraded":"2024-03-11T21:29:42.554209+0000","last_fullsized":"2024-03-11T21:29:42.554209+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:12.312072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"69.5","version":"0'0","reported_seq":74,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452228+0000","last_change":"2024-03-11T21:29:06.568626+0000","last_active":"2024-03-11T21:29:42.452228+0000","last_peered":"2024-03-11T21:29:42.452228+0000","last_clean":"2024-03-11T21:29:42.452228+0000","last_became_active":"2024-03-11T21:29:06.565601+0000","last_became_peered":"2024-03-11T21:29:06.565601+0000","last_unstale":"2024-03-11T21:29:42.452228+0000","last_undegraded":"2024-03-11T21:29:42.452228+0000","last_fullsized":"2024-03-11T21:29:42.452228+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.004563+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.004563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:12.039776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449383+0000","last_change":"2024-03-11T21:28:57.551635+0000","last_active":"2024-03-11T21:29:42.449383+0000","last_peered":"2024-03-11T21:29:42.449383+0000","last_clean":"2024-03-11T21:29:42.449383+0000","last_became_active":"2024-03-11T21:28:57.551331+0000","last_became_peered":"2024-03-11T21:28:57.551331+0000","last_unstale":"2024-03-11T21:29:42.449383+0000","last_undegraded":"2024-03-11T21:29:42.449383+0000","last_fullsized":"2024-03-11T21:29:42.449383+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:17.992645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447218+0000","last_change":"2024-03-11T21:28:54.794911+0000","last_active":"2024-03-11T21:29:42.447218+0000","last_peered":"2024-03-11T21:29:42.447218+0000","last_clean":"2024-03-11T21:29:42.447218+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:29:42.447218+0000","last_undegraded":"2024-03-11T21:29:42.447218+0000","last_fullsized":"2024-03-11T21:29:42.447218+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:12.003174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441036+0000","last_change":"2024-03-11T21:28:54.808154+0000","last_active":"2024-03-11T21:29:42.441036+0000","last_peered":"2024-03-11T21:29:42.441036+0000","last_clean":"2024-03-11T21:29:42.441036+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:29:42.441036+0000","last_undegraded":"2024-03-11T21:29:42.441036+0000","last_fullsized":"2024-03-11T21:29:42.441036+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.093974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458590+0000","last_change":"2024-03-11T21:28:57.522457+0000","last_active":"2024-03-11T21:29:42.458590+0000","last_peered":"2024-03-11T21:29:42.458590+0000","last_clean":"2024-03-11T21:29:42.458590+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:29:42.458590+0000","last_undegraded":"2024-03-11T21:29:42.458590+0000","last_fullsized":"2024-03-11T21:29:42.458590+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:17.525111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554791+0000","last_change":"2024-03-11T21:28:53.800290+0000","last_active":"2024-03-11T21:29:42.554791+0000","last_peered":"2024-03-11T21:29:42.554791+0000","last_clean":"2024-03-11T21:29:42.554791+0000","last_became_active":"2024-03-11T21:28:53.798891+0000","last_became_peered":"2024-03-11T21:28:53.798891+0000","last_unstale":"2024-03-11T21:29:42.554791+0000","last_undegraded":"2024-03-11T21:29:42.554791+0000","last_fullsized":"2024-03-11T21:29:42.554791+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.117601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.455384+0000","last_change":"2024-03-11T21:28:54.833928+0000","last_active":"2024-03-11T21:29:42.455384+0000","last_peered":"2024-03-11T21:29:42.455384+0000","last_clean":"2024-03-11T21:29:42.455384+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:29:42.455384+0000","last_undegraded":"2024-03-11T21:29:42.455384+0000","last_fullsized":"2024-03-11T21:29:42.455384+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.939874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448324+0000","last_change":"2024-03-11T21:28:54.813546+0000","last_active":"2024-03-11T21:29:42.448324+0000","last_peered":"2024-03-11T21:29:42.448324+0000","last_clean":"2024-03-11T21:29:42.448324+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:29:42.448324+0000","last_undegraded":"2024-03-11T21:29:42.448324+0000","last_fullsized":"2024-03-11T21:29:42.448324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.621905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.680396+0000","last_change":"2024-03-11T21:28:54.853417+0000","last_active":"2024-03-11T21:29:42.680396+0000","last_peered":"2024-03-11T21:29:42.680396+0000","last_clean":"2024-03-11T21:29:42.680396+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:29:42.680396+0000","last_undegraded":"2024-03-11T21:29:42.680396+0000","last_fullsized":"2024-03-11T21:29:42.680396+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.412134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.435929+0000","last_change":"2024-03-11T21:28:53.811402+0000","last_active":"2024-03-11T21:29:42.435929+0000","last_peered":"2024-03-11T21:29:42.435929+0000","last_clean":"2024-03-11T21:29:42.435929+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:29:42.435929+0000","last_undegraded":"2024-03-11T21:29:42.435929+0000","last_fullsized":"2024-03-11T21:29:42.435929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.199839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453960+0000","last_change":"2024-03-11T21:28:57.526000+0000","last_active":"2024-03-11T21:29:42.453960+0000","last_peered":"2024-03-11T21:29:42.453960+0000","last_clean":"2024-03-11T21:29:42.453960+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:29:42.453960+0000","last_undegraded":"2024-03-11T21:29:42.453960+0000","last_fullsized":"2024-03-11T21:29:42.453960+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.670734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.668090+0000","last_change":"2024-03-11T21:28:54.872929+0000","last_active":"2024-03-11T21:29:42.668090+0000","last_peered":"2024-03-11T21:29:42.668090+0000","last_clean":"2024-03-11T21:29:42.668090+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:29:42.668090+0000","last_undegraded":"2024-03-11T21:29:42.668090+0000","last_fullsized":"2024-03-11T21:29:42.668090+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.073668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.463679+0000","last_change":"2024-03-11T21:28:53.856740+0000","last_active":"2024-03-11T21:29:42.463679+0000","last_peered":"2024-03-11T21:29:42.463679+0000","last_clean":"2024-03-11T21:29:42.463679+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:29:42.463679+0000","last_undegraded":"2024-03-11T21:29:42.463679+0000","last_fullsized":"2024-03-11T21:29:42.463679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.222353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.686772+0000","last_change":"2024-03-11T21:28:54.867287+0000","last_active":"2024-03-11T21:29:42.686772+0000","last_peered":"2024-03-11T21:29:42.686772+0000","last_clean":"2024-03-11T21:29:42.686772+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:29:42.686772+0000","last_undegraded":"2024-03-11T21:29:42.686772+0000","last_fullsized":"2024-03-11T21:29:42.686772+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.686167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453421+0000","last_change":"2024-03-11T21:28:53.826473+0000","last_active":"2024-03-11T21:29:42.453421+0000","last_peered":"2024-03-11T21:29:42.453421+0000","last_clean":"2024-03-11T21:29:42.453421+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:29:42.453421+0000","last_undegraded":"2024-03-11T21:29:42.453421+0000","last_fullsized":"2024-03-11T21:29:42.453421+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.963254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.678533+0000","last_change":"2024-03-11T21:28:54.862519+0000","last_active":"2024-03-11T21:29:42.678533+0000","last_peered":"2024-03-11T21:29:42.678533+0000","last_clean":"2024-03-11T21:29:42.678533+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:29:42.678533+0000","last_undegraded":"2024-03-11T21:29:42.678533+0000","last_fullsized":"2024-03-11T21:29:42.678533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.403977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.776732+0000","last_change":"2024-03-11T21:28:57.548976+0000","last_active":"2024-03-11T21:29:42.776732+0000","last_peered":"2024-03-11T21:29:42.776732+0000","last_clean":"2024-03-11T21:29:42.776732+0000","last_became_active":"2024-03-11T21:28:57.548665+0000","last_became_peered":"2024-03-11T21:28:57.548665+0000","last_unstale":"2024-03-11T21:29:42.776732+0000","last_undegraded":"2024-03-11T21:29:42.776732+0000","last_fullsized":"2024-03-11T21:29:42.776732+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.057221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.438652+0000","last_change":"2024-03-11T21:28:54.833916+0000","last_active":"2024-03-11T21:29:42.438652+0000","last_peered":"2024-03-11T21:29:42.438652+0000","last_clean":"2024-03-11T21:29:42.438652+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:29:42.438652+0000","last_undegraded":"2024-03-11T21:29:42.438652+0000","last_fullsized":"2024-03-11T21:29:42.438652+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.324797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.467184+0000","last_change":"2024-03-11T21:28:53.852957+0000","last_active":"2024-03-11T21:29:42.467184+0000","last_peered":"2024-03-11T21:29:42.467184+0000","last_clean":"2024-03-11T21:29:42.467184+0000","last_became_active":"2024-03-11T21:28:53.852502+0000","last_became_peered":"2024-03-11T21:28:53.852502+0000","last_unstale":"2024-03-11T21:29:42.467184+0000","last_undegraded":"2024-03-11T21:29:42.467184+0000","last_fullsized":"2024-03-11T21:29:42.467184+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.430274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.767011+0000","last_change":"2024-03-11T21:28:57.495142+0000","last_active":"2024-03-11T21:29:42.767011+0000","last_peered":"2024-03-11T21:29:42.767011+0000","last_clean":"2024-03-11T21:29:42.767011+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:29:42.767011+0000","last_undegraded":"2024-03-11T21:29:42.767011+0000","last_fullsized":"2024-03-11T21:29:42.767011+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:14.516938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.457803+0000","last_change":"2024-03-11T21:28:53.856809+0000","last_active":"2024-03-11T21:29:42.457803+0000","last_peered":"2024-03-11T21:29:42.457803+0000","last_clean":"2024-03-11T21:29:42.457803+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:29:42.457803+0000","last_undegraded":"2024-03-11T21:29:42.457803+0000","last_fullsized":"2024-03-11T21:29:42.457803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.497975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554516+0000","last_change":"2024-03-11T21:28:53.801492+0000","last_active":"2024-03-11T21:29:42.554516+0000","last_peered":"2024-03-11T21:29:42.554516+0000","last_clean":"2024-03-11T21:29:42.554516+0000","last_became_active":"2024-03-11T21:28:53.801323+0000","last_became_peered":"2024-03-11T21:28:53.801323+0000","last_unstale":"2024-03-11T21:29:42.554516+0000","last_undegraded":"2024-03-11T21:29:42.554516+0000","last_fullsized":"2024-03-11T21:29:42.554516+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.055484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.771139+0000","last_change":"2024-03-11T21:28:57.569751+0000","last_active":"2024-03-11T21:29:42.771139+0000","last_peered":"2024-03-11T21:29:42.771139+0000","last_clean":"2024-03-11T21:29:42.771139+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:29:42.771139+0000","last_undegraded":"2024-03-11T21:29:42.771139+0000","last_fullsized":"2024-03-11T21:29:42.771139+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.392509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.766544+0000","last_change":"2024-03-11T21:28:57.547048+0000","last_active":"2024-03-11T21:29:42.766544+0000","last_peered":"2024-03-11T21:29:42.766544+0000","last_clean":"2024-03-11T21:29:42.766544+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:29:42.766544+0000","last_undegraded":"2024-03-11T21:29:42.766544+0000","last_fullsized":"2024-03-11T21:29:42.766544+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.452079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688566+0000","last_change":"2024-03-11T21:28:54.850648+0000","last_active":"2024-03-11T21:29:42.688566+0000","last_peered":"2024-03-11T21:29:42.688566+0000","last_clean":"2024-03-11T21:29:42.688566+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:29:42.688566+0000","last_undegraded":"2024-03-11T21:29:42.688566+0000","last_fullsized":"2024-03-11T21:29:42.688566+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.726464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444259+0000","last_change":"2024-03-11T21:28:54.822826+0000","last_active":"2024-03-11T21:29:42.444259+0000","last_peered":"2024-03-11T21:29:42.444259+0000","last_clean":"2024-03-11T21:29:42.444259+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:29:42.444259+0000","last_undegraded":"2024-03-11T21:29:42.444259+0000","last_fullsized":"2024-03-11T21:29:42.444259+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.895837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.467284+0000","last_change":"2024-03-11T21:28:53.852772+0000","last_active":"2024-03-11T21:29:42.467284+0000","last_peered":"2024-03-11T21:29:42.467284+0000","last_clean":"2024-03-11T21:29:42.467284+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:29:42.467284+0000","last_undegraded":"2024-03-11T21:29:42.467284+0000","last_fullsized":"2024-03-11T21:29:42.467284+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.600820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":85,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.767251+0000","last_change":"2024-03-11T21:28:57.523727+0000","last_active":"2024-03-11T21:29:42.767251+0000","last_peered":"2024-03-11T21:29:42.767251+0000","last_clean":"2024-03-11T21:29:42.767251+0000","last_became_active":"2024-03-11T21:28:57.523449+0000","last_became_peered":"2024-03-11T21:28:57.523449+0000","last_unstale":"2024-03-11T21:29:42.767251+0000","last_undegraded":"2024-03-11T21:29:42.767251+0000","last_fullsized":"2024-03-11T21:29:42.767251+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453441+0000","last_change":"2024-03-11T21:28:53.827705+0000","last_active":"2024-03-11T21:29:42.453441+0000","last_peered":"2024-03-11T21:29:42.453441+0000","last_clean":"2024-03-11T21:29:42.453441+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:29:42.453441+0000","last_undegraded":"2024-03-11T21:29:42.453441+0000","last_fullsized":"2024-03-11T21:29:42.453441+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:10.875350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.687449+0000","last_change":"2024-03-11T21:28:54.861935+0000","last_active":"2024-03-11T21:29:42.687449+0000","last_peered":"2024-03-11T21:29:42.687449+0000","last_clean":"2024-03-11T21:29:42.687449+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:29:42.687449+0000","last_undegraded":"2024-03-11T21:29:42.687449+0000","last_fullsized":"2024-03-11T21:29:42.687449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.800445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461727+0000","last_change":"2024-03-11T21:28:53.852479+0000","last_active":"2024-03-11T21:29:42.461727+0000","last_peered":"2024-03-11T21:29:42.461727+0000","last_clean":"2024-03-11T21:29:42.461727+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:29:42.461727+0000","last_undegraded":"2024-03-11T21:29:42.461727+0000","last_fullsized":"2024-03-11T21:29:42.461727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.423252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":87,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.678812+0000","last_change":"2024-03-11T21:28:54.869980+0000","last_active":"2024-03-11T21:29:42.678812+0000","last_peered":"2024-03-11T21:29:42.678812+0000","last_clean":"2024-03-11T21:29:42.678812+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:29:42.678812+0000","last_undegraded":"2024-03-11T21:29:42.678812+0000","last_fullsized":"2024-03-11T21:29:42.678812+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.469740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":88,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560648+0000","last_change":"2024-03-11T21:28:53.842332+0000","last_active":"2024-03-11T21:29:42.560648+0000","last_peered":"2024-03-11T21:29:42.560648+0000","last_clean":"2024-03-11T21:29:42.560648+0000","last_became_active":"2024-03-11T21:28:53.842023+0000","last_became_peered":"2024-03-11T21:28:53.842023+0000","last_unstale":"2024-03-11T21:29:42.560648+0000","last_undegraded":"2024-03-11T21:29:42.560648+0000","last_fullsized":"2024-03-11T21:29:42.560648+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.932379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"167'4","reported_seq":115,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.451340+0000","last_change":"2024-03-11T21:29:25.566191+0000","last_active":"2024-03-11T21:29:42.451340+0000","last_peered":"2024-03-11T21:29:42.451340+0000","last_clean":"2024-03-11T21:29:42.451340+0000","last_became_active":"2024-03-11T21:28:51.397445+0000","last_became_peered":"2024-03-11T21:28:51.397445+0000","last_unstale":"2024-03-11T21:29:42.451340+0000","last_undegraded":"2024-03-11T21:29:42.451340+0000","last_fullsized":"2024-03-11T21:29:42.451340+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'4","last_scrub_stamp":"2024-03-11T21:29:25.566057+0000","last_deep_scrub":"167'4","last_deep_scrub_stamp":"2024-03-11T21:29:25.566057+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.566057+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:46.338532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.434899+0000","last_change":"2024-03-11T21:28:51.351930+0000","last_active":"2024-03-11T21:29:42.434899+0000","last_peered":"2024-03-11T21:29:42.434899+0000","last_clean":"2024-03-11T21:29:42.434899+0000","last_became_active":"2024-03-11T21:28:51.348023+0000","last_became_peered":"2024-03-11T21:28:51.348023+0000","last_unstale":"2024-03-11T21:29:42.434899+0000","last_undegraded":"2024-03-11T21:29:42.434899+0000","last_fullsized":"2024-03-11T21:29:42.434899+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:52.053436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683151+0000","last_change":"2024-03-11T21:28:51.388861+0000","last_active":"2024-03-11T21:29:42.683151+0000","last_peered":"2024-03-11T21:29:42.683151+0000","last_clean":"2024-03-11T21:29:42.683151+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:29:42.683151+0000","last_undegraded":"2024-03-11T21:29:42.683151+0000","last_fullsized":"2024-03-11T21:29:42.683151+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.084176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":330,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.469361+0000","last_change":"2024-03-11T21:28:37.394649+0000","last_active":"2024-03-11T21:29:42.469361+0000","last_peered":"2024-03-11T21:29:42.469361+0000","last_clean":"2024-03-11T21:29:42.469361+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:29:42.469361+0000","last_undegraded":"2024-03-11T21:29:42.469361+0000","last_fullsized":"2024-03-11T21:29:42.469361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:37.394505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.394505+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.899182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.472727+0000","last_change":"2024-03-11T21:28:26.181008+0000","last_active":"2024-03-11T21:29:42.472727+0000","last_peered":"2024-03-11T21:29:42.472727+0000","last_clean":"2024-03-11T21:29:42.472727+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:29:42.472727+0000","last_undegraded":"2024-03-11T21:29:42.472727+0000","last_fullsized":"2024-03-11T21:29:42.472727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:28:26.180694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.180694+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.711329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.679099+0000","last_change":"2024-03-11T21:29:01.853935+0000","last_active":"2024-03-11T21:29:42.679099+0000","last_peered":"2024-03-11T21:29:42.679099+0000","last_clean":"2024-03-11T21:29:42.679099+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:29:42.679099+0000","last_undegraded":"2024-03-11T21:29:42.679099+0000","last_fullsized":"2024-03-11T21:29:42.679099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:01.853854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:01.853854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.945783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442691+0000","last_change":"2024-03-11T21:27:31.990217+0000","last_active":"2024-03-11T21:29:42.442691+0000","last_peered":"2024-03-11T21:29:42.442691+0000","last_clean":"2024-03-11T21:29:42.442691+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:29:42.442691+0000","last_undegraded":"2024-03-11T21:29:42.442691+0000","last_fullsized":"2024-03-11T21:29:42.442691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.989922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.989922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":111,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442060+0000","last_change":"2024-03-11T21:29:27.039773+0000","last_active":"2024-03-11T21:29:42.442060+0000","last_peered":"2024-03-11T21:29:42.442060+0000","last_clean":"2024-03-11T21:29:42.442060+0000","last_became_active":"2024-03-11T21:28:51.408595+0000","last_became_peered":"2024-03-11T21:28:51.408595+0000","last_unstale":"2024-03-11T21:29:42.442060+0000","last_undegraded":"2024-03-11T21:29:42.442060+0000","last_fullsized":"2024-03-11T21:29:42.442060+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:27.039510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:27.039510+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.039510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:49.503163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683894+0000","last_change":"2024-03-11T21:28:51.326889+0000","last_active":"2024-03-11T21:29:42.683894+0000","last_peered":"2024-03-11T21:29:42.683894+0000","last_clean":"2024-03-11T21:29:42.683894+0000","last_became_active":"2024-03-11T21:28:51.326593+0000","last_became_peered":"2024-03-11T21:28:51.326593+0000","last_unstale":"2024-03-11T21:29:42.683894+0000","last_undegraded":"2024-03-11T21:29:42.683894+0000","last_fullsized":"2024-03-11T21:29:42.683894+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.529836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448664+0000","last_change":"2024-03-11T21:28:51.349975+0000","last_active":"2024-03-11T21:29:42.448664+0000","last_peered":"2024-03-11T21:29:42.448664+0000","last_clean":"2024-03-11T21:29:42.448664+0000","last_became_active":"2024-03-11T21:28:51.346539+0000","last_became_peered":"2024-03-11T21:28:51.346539+0000","last_unstale":"2024-03-11T21:29:42.448664+0000","last_undegraded":"2024-03-11T21:29:42.448664+0000","last_fullsized":"2024-03-11T21:29:42.448664+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.114834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452482+0000","last_change":"2024-03-11T21:27:20.912735+0000","last_active":"2024-03-11T21:29:42.452482+0000","last_peered":"2024-03-11T21:29:42.452482+0000","last_clean":"2024-03-11T21:29:42.452482+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:29:42.452482+0000","last_undegraded":"2024-03-11T21:29:42.452482+0000","last_fullsized":"2024-03-11T21:29:42.452482+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:20.912377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.912377+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559037+0000","last_change":"2024-03-11T21:29:04.246592+0000","last_active":"2024-03-11T21:29:42.559037+0000","last_peered":"2024-03-11T21:29:42.559037+0000","last_clean":"2024-03-11T21:29:42.559037+0000","last_became_active":"2024-03-11T21:23:02.925089+0000","last_became_peered":"2024-03-11T21:23:02.925089+0000","last_unstale":"2024-03-11T21:29:42.559037+0000","last_undegraded":"2024-03-11T21:29:42.559037+0000","last_fullsized":"2024-03-11T21:29:42.559037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.246471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:19.156597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661822+0000","last_change":"2024-03-11T21:28:34.922184+0000","last_active":"2024-03-11T21:29:42.661822+0000","last_peered":"2024-03-11T21:29:42.661822+0000","last_clean":"2024-03-11T21:29:42.661822+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:29:42.661822+0000","last_undegraded":"2024-03-11T21:29:42.661822+0000","last_fullsized":"2024-03-11T21:29:42.661822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.922068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.922068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.766157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":244,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.682426+0000","last_change":"2024-03-11T21:28:58.518723+0000","last_active":"2024-03-11T21:29:42.682426+0000","last_peered":"2024-03-11T21:29:42.682426+0000","last_clean":"2024-03-11T21:29:42.682426+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:29:42.682426+0000","last_undegraded":"2024-03-11T21:29:42.682426+0000","last_fullsized":"2024-03-11T21:29:42.682426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:28:58.518630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:23.237603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"167'1004","reported_seq":2118,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684862+0000","last_change":"2024-03-11T21:29:25.783776+0000","last_active":"2024-03-11T21:29:42.684862+0000","last_peered":"2024-03-11T21:29:42.684862+0000","last_clean":"2024-03-11T21:29:42.684862+0000","last_became_active":"2024-03-11T21:28:51.293289+0000","last_became_peered":"2024-03-11T21:28:51.293289+0000","last_unstale":"2024-03-11T21:29:42.684862+0000","last_undegraded":"2024-03-11T21:29:42.684862+0000","last_fullsized":"2024-03-11T21:29:42.684862+0000","mapping_epoch":139,"log_start":"142'200","ondisk_log_start":"142'200","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'1004","last_scrub_stamp":"2024-03-11T21:29:25.783422+0000","last_deep_scrub":"167'1004","last_deep_scrub_stamp":"2024-03-11T21:29:25.783422+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.783422+0000","objects_scrubbed":2,"log_size":804,"log_dups_size":200,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.327590+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1001,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452694+0000","last_change":"2024-03-11T21:28:51.402701+0000","last_active":"2024-03-11T21:29:42.452694+0000","last_peered":"2024-03-11T21:29:42.452694+0000","last_clean":"2024-03-11T21:29:42.452694+0000","last_became_active":"2024-03-11T21:28:51.402402+0000","last_became_peered":"2024-03-11T21:28:51.402402+0000","last_unstale":"2024-03-11T21:29:42.452694+0000","last_undegraded":"2024-03-11T21:29:42.452694+0000","last_fullsized":"2024-03-11T21:29:42.452694+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.877659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452673+0000","last_change":"2024-03-11T21:28:51.382103+0000","last_active":"2024-03-11T21:29:42.452673+0000","last_peered":"2024-03-11T21:29:42.452673+0000","last_clean":"2024-03-11T21:29:42.452673+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:29:42.452673+0000","last_undegraded":"2024-03-11T21:29:42.452673+0000","last_fullsized":"2024-03-11T21:29:42.452673+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.437125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":287,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462520+0000","last_change":"2024-03-11T21:29:08.366566+0000","last_active":"2024-03-11T21:29:42.462520+0000","last_peered":"2024-03-11T21:29:42.462520+0000","last_clean":"2024-03-11T21:29:42.462520+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:29:42.462520+0000","last_undegraded":"2024-03-11T21:29:42.462520+0000","last_fullsized":"2024-03-11T21:29:42.462520+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:29:08.366434+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:29:08.366434+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.316999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.676241+0000","last_change":"2024-03-11T21:28:38.879798+0000","last_active":"2024-03-11T21:29:42.676241+0000","last_peered":"2024-03-11T21:29:42.676241+0000","last_clean":"2024-03-11T21:29:42.676241+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:29:42.676241+0000","last_undegraded":"2024-03-11T21:29:42.676241+0000","last_fullsized":"2024-03-11T21:29:42.676241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:38.879683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:38.879683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.646344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":246,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672415+0000","last_change":"2024-03-11T21:28:23.956834+0000","last_active":"2024-03-11T21:29:42.672415+0000","last_peered":"2024-03-11T21:29:42.672415+0000","last_clean":"2024-03-11T21:29:42.672415+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:29:42.672415+0000","last_undegraded":"2024-03-11T21:29:42.672415+0000","last_fullsized":"2024-03-11T21:29:42.672415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.956703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.956703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452464+0000","last_change":"2024-03-11T21:27:26.598269+0000","last_active":"2024-03-11T21:29:42.452464+0000","last_peered":"2024-03-11T21:29:42.452464+0000","last_clean":"2024-03-11T21:29:42.452464+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:29:42.452464+0000","last_undegraded":"2024-03-11T21:29:42.452464+0000","last_fullsized":"2024-03-11T21:29:42.452464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:26.598056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:26.598056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.254927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.4","version":"148'2","reported_seq":116,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.450064+0000","last_change":"2024-03-11T21:29:26.520882+0000","last_active":"2024-03-11T21:29:42.450064+0000","last_peered":"2024-03-11T21:29:42.450064+0000","last_clean":"2024-03-11T21:29:42.450064+0000","last_became_active":"2024-03-11T21:28:51.442242+0000","last_became_peered":"2024-03-11T21:28:51.442242+0000","last_unstale":"2024-03-11T21:29:42.450064+0000","last_undegraded":"2024-03-11T21:29:42.450064+0000","last_fullsized":"2024-03-11T21:29:42.450064+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'2","last_scrub_stamp":"2024-03-11T21:29:26.520611+0000","last_deep_scrub":"148'2","last_deep_scrub_stamp":"2024-03-11T21:29:26.520611+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.520611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:40.113539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454914+0000","last_change":"2024-03-11T21:28:51.443779+0000","last_active":"2024-03-11T21:29:42.454914+0000","last_peered":"2024-03-11T21:29:42.454914+0000","last_clean":"2024-03-11T21:29:42.454914+0000","last_became_active":"2024-03-11T21:28:51.443285+0000","last_became_peered":"2024-03-11T21:28:51.443285+0000","last_unstale":"2024-03-11T21:29:42.454914+0000","last_undegraded":"2024-03-11T21:29:42.454914+0000","last_fullsized":"2024-03-11T21:29:42.454914+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.255430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452735+0000","last_change":"2024-03-11T21:28:51.416026+0000","last_active":"2024-03-11T21:29:42.452735+0000","last_peered":"2024-03-11T21:29:42.452735+0000","last_clean":"2024-03-11T21:29:42.452735+0000","last_became_active":"2024-03-11T21:28:51.393061+0000","last_became_peered":"2024-03-11T21:28:51.393061+0000","last_unstale":"2024-03-11T21:29:42.452735+0000","last_undegraded":"2024-03-11T21:29:42.452735+0000","last_fullsized":"2024-03-11T21:29:42.452735+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.112981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":291,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443733+0000","last_change":"2024-03-11T21:28:40.847588+0000","last_active":"2024-03-11T21:29:42.443733+0000","last_peered":"2024-03-11T21:29:42.443733+0000","last_clean":"2024-03-11T21:29:42.443733+0000","last_became_active":"2024-03-11T21:23:04.941845+0000","last_became_peered":"2024-03-11T21:23:04.941845+0000","last_unstale":"2024-03-11T21:29:42.443733+0000","last_undegraded":"2024-03-11T21:29:42.443733+0000","last_fullsized":"2024-03-11T21:29:42.443733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:28:40.847340+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:28:40.847340+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.221629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462470+0000","last_change":"2024-03-11T21:28:42.395827+0000","last_active":"2024-03-11T21:29:42.462470+0000","last_peered":"2024-03-11T21:29:42.462470+0000","last_clean":"2024-03-11T21:29:42.462470+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:29:42.462470+0000","last_undegraded":"2024-03-11T21:29:42.462470+0000","last_fullsized":"2024-03-11T21:29:42.462470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:42.395724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:42.395724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.334377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554616+0000","last_change":"2024-03-11T21:29:14.307544+0000","last_active":"2024-03-11T21:29:42.554616+0000","last_peered":"2024-03-11T21:29:42.554616+0000","last_clean":"2024-03-11T21:29:42.554616+0000","last_became_active":"2024-03-11T21:23:07.100323+0000","last_became_peered":"2024-03-11T21:23:07.100323+0000","last_unstale":"2024-03-11T21:29:42.554616+0000","last_undegraded":"2024-03-11T21:29:42.554616+0000","last_fullsized":"2024-03-11T21:29:42.554616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:14.307354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.307354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:31.976690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":242,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.682023+0000","last_change":"2024-03-11T21:28:43.004205+0000","last_active":"2024-03-11T21:29:42.682023+0000","last_peered":"2024-03-11T21:29:42.682023+0000","last_clean":"2024-03-11T21:29:42.682023+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:29:42.682023+0000","last_undegraded":"2024-03-11T21:29:42.682023+0000","last_fullsized":"2024-03-11T21:29:42.682023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:43.004003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:43.004003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.195219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":111,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452528+0000","last_change":"2024-03-11T21:29:26.053722+0000","last_active":"2024-03-11T21:29:42.452528+0000","last_peered":"2024-03-11T21:29:42.452528+0000","last_clean":"2024-03-11T21:29:42.452528+0000","last_became_active":"2024-03-11T21:28:51.323433+0000","last_became_peered":"2024-03-11T21:28:51.323433+0000","last_unstale":"2024-03-11T21:29:42.452528+0000","last_undegraded":"2024-03-11T21:29:42.452528+0000","last_fullsized":"2024-03-11T21:29:42.452528+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:26.053584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:26.053584+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.053584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:26.215652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.450186+0000","last_change":"2024-03-11T21:28:51.486608+0000","last_active":"2024-03-11T21:29:42.450186+0000","last_peered":"2024-03-11T21:29:42.450186+0000","last_clean":"2024-03-11T21:29:42.450186+0000","last_became_active":"2024-03-11T21:28:51.485070+0000","last_became_peered":"2024-03-11T21:28:51.485070+0000","last_unstale":"2024-03-11T21:29:42.450186+0000","last_undegraded":"2024-03-11T21:29:42.450186+0000","last_fullsized":"2024-03-11T21:29:42.450186+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.053946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"141'7","reported_seq":107,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.456534+0000","last_change":"2024-03-11T21:28:49.936800+0000","last_active":"2024-03-11T21:29:42.456534+0000","last_peered":"2024-03-11T21:29:42.456534+0000","last_clean":"2024-03-11T21:29:42.456534+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:29:42.456534+0000","last_undegraded":"2024-03-11T21:29:42.456534+0000","last_fullsized":"2024-03-11T21:29:42.456534+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.480728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458941+0000","last_change":"2024-03-11T21:29:04.371556+0000","last_active":"2024-03-11T21:29:42.458941+0000","last_peered":"2024-03-11T21:29:42.458941+0000","last_clean":"2024-03-11T21:29:42.458941+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:29:42.458941+0000","last_undegraded":"2024-03-11T21:29:42.458941+0000","last_fullsized":"2024-03-11T21:29:42.458941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:29:04.371307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:04.371307+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.309222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552503+0000","last_change":"2024-03-11T21:28:40.310802+0000","last_active":"2024-03-11T21:29:42.552503+0000","last_peered":"2024-03-11T21:29:42.552503+0000","last_clean":"2024-03-11T21:29:42.552503+0000","last_became_active":"2024-03-11T21:23:02.923595+0000","last_became_peered":"2024-03-11T21:23:02.923595+0000","last_unstale":"2024-03-11T21:29:42.552503+0000","last_undegraded":"2024-03-11T21:29:42.552503+0000","last_fullsized":"2024-03-11T21:29:42.552503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:40.310703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:28:40.310703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.487501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":337,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684163+0000","last_change":"2024-03-11T21:28:36.900652+0000","last_active":"2024-03-11T21:29:42.684163+0000","last_peered":"2024-03-11T21:29:42.684163+0000","last_clean":"2024-03-11T21:29:42.684163+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:29:42.684163+0000","last_undegraded":"2024-03-11T21:29:42.684163+0000","last_fullsized":"2024-03-11T21:29:42.684163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:28:36.900561+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.900561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447185+0000","last_change":"2024-03-11T21:28:52.069197+0000","last_active":"2024-03-11T21:29:42.447185+0000","last_peered":"2024-03-11T21:29:42.447185+0000","last_clean":"2024-03-11T21:29:42.447185+0000","last_became_active":"2024-03-11T21:23:09.150916+0000","last_became_peered":"2024-03-11T21:23:09.150916+0000","last_unstale":"2024-03-11T21:29:42.447185+0000","last_undegraded":"2024-03-11T21:29:42.447185+0000","last_fullsized":"2024-03-11T21:29:42.447185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:52.069094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:52.069094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.728581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"168'2","reported_seq":116,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443733+0000","last_change":"2024-03-11T21:29:27.082901+0000","last_active":"2024-03-11T21:29:42.443733+0000","last_peered":"2024-03-11T21:29:42.443733+0000","last_clean":"2024-03-11T21:29:42.443733+0000","last_became_active":"2024-03-11T21:28:51.408612+0000","last_became_peered":"2024-03-11T21:28:51.408612+0000","last_unstale":"2024-03-11T21:29:42.443733+0000","last_undegraded":"2024-03-11T21:29:42.443733+0000","last_fullsized":"2024-03-11T21:29:42.443733+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'2","last_scrub_stamp":"2024-03-11T21:29:27.082793+0000","last_deep_scrub":"168'2","last_deep_scrub_stamp":"2024-03-11T21:29:27.082793+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.082793+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:53.122357+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560534+0000","last_change":"2024-03-11T21:28:51.436656+0000","last_active":"2024-03-11T21:29:42.560534+0000","last_peered":"2024-03-11T21:29:42.560534+0000","last_clean":"2024-03-11T21:29:42.560534+0000","last_became_active":"2024-03-11T21:28:51.432471+0000","last_became_peered":"2024-03-11T21:28:51.432471+0000","last_unstale":"2024-03-11T21:29:42.560534+0000","last_undegraded":"2024-03-11T21:29:42.560534+0000","last_fullsized":"2024-03-11T21:29:42.560534+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.779121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.771770+0000","last_change":"2024-03-11T21:28:49.958316+0000","last_active":"2024-03-11T21:29:42.771770+0000","last_peered":"2024-03-11T21:29:42.771770+0000","last_clean":"2024-03-11T21:29:42.771770+0000","last_became_active":"2024-03-11T21:28:49.955426+0000","last_became_peered":"2024-03-11T21:28:49.955426+0000","last_unstale":"2024-03-11T21:29:42.771770+0000","last_undegraded":"2024-03-11T21:29:42.771770+0000","last_fullsized":"2024-03-11T21:29:42.771770+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.939807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449768+0000","last_change":"2024-03-11T21:29:13.487479+0000","last_active":"2024-03-11T21:29:42.449768+0000","last_peered":"2024-03-11T21:29:42.449768+0000","last_clean":"2024-03-11T21:29:42.449768+0000","last_became_active":"2024-03-11T21:23:04.949622+0000","last_became_peered":"2024-03-11T21:23:04.949622+0000","last_unstale":"2024-03-11T21:29:42.449768+0000","last_undegraded":"2024-03-11T21:29:42.449768+0000","last_fullsized":"2024-03-11T21:29:42.449768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:13.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.487366+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:22.223569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552718+0000","last_change":"2024-03-11T21:29:00.197303+0000","last_active":"2024-03-11T21:29:42.552718+0000","last_peered":"2024-03-11T21:29:42.552718+0000","last_clean":"2024-03-11T21:29:42.552718+0000","last_became_active":"2024-03-11T21:23:02.923296+0000","last_became_peered":"2024-03-11T21:23:02.923296+0000","last_unstale":"2024-03-11T21:29:42.552718+0000","last_undegraded":"2024-03-11T21:29:42.552718+0000","last_fullsized":"2024-03-11T21:29:42.552718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:00.197168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:00.197168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.349626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":337,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443803+0000","last_change":"2024-03-11T21:28:54.069307+0000","last_active":"2024-03-11T21:29:42.443803+0000","last_peered":"2024-03-11T21:29:42.443803+0000","last_clean":"2024-03-11T21:29:42.443803+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:29:42.443803+0000","last_undegraded":"2024-03-11T21:29:42.443803+0000","last_fullsized":"2024-03-11T21:29:42.443803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:54.069179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.069179+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.164085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444970+0000","last_change":"2024-03-11T21:29:18.039166+0000","last_active":"2024-03-11T21:29:42.444970+0000","last_peered":"2024-03-11T21:29:42.444970+0000","last_clean":"2024-03-11T21:29:42.444970+0000","last_became_active":"2024-03-11T21:23:09.134706+0000","last_became_peered":"2024-03-11T21:23:09.134706+0000","last_unstale":"2024-03-11T21:29:42.444970+0000","last_undegraded":"2024-03-11T21:29:42.444970+0000","last_fullsized":"2024-03-11T21:29:42.444970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:18.039059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:29:18.039059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:19.369629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"167'4","reported_seq":120,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.472347+0000","last_change":"2024-03-11T21:29:25.378639+0000","last_active":"2024-03-11T21:29:42.472347+0000","last_peered":"2024-03-11T21:29:42.472347+0000","last_clean":"2024-03-11T21:29:42.472347+0000","last_became_active":"2024-03-11T21:28:51.394014+0000","last_became_peered":"2024-03-11T21:28:51.394014+0000","last_unstale":"2024-03-11T21:29:42.472347+0000","last_undegraded":"2024-03-11T21:29:42.472347+0000","last_fullsized":"2024-03-11T21:29:42.472347+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'4","last_scrub_stamp":"2024-03-11T21:29:25.378523+0000","last_deep_scrub":"167'4","last_deep_scrub_stamp":"2024-03-11T21:29:25.378523+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.378523+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:53.435907+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166860,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":24577,"num_write":4,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"142'9","reported_seq":102,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.558924+0000","last_change":"2024-03-11T21:28:51.430084+0000","last_active":"2024-03-11T21:29:42.558924+0000","last_peered":"2024-03-11T21:29:42.558924+0000","last_clean":"2024-03-11T21:29:42.558924+0000","last_became_active":"2024-03-11T21:28:51.428846+0000","last_became_peered":"2024-03-11T21:28:51.428846+0000","last_unstale":"2024-03-11T21:29:42.558924+0000","last_undegraded":"2024-03-11T21:29:42.558924+0000","last_fullsized":"2024-03-11T21:29:42.558924+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.476909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462052+0000","last_change":"2024-03-11T21:28:51.495976+0000","last_active":"2024-03-11T21:29:42.462052+0000","last_peered":"2024-03-11T21:29:42.462052+0000","last_clean":"2024-03-11T21:29:42.462052+0000","last_became_active":"2024-03-11T21:28:51.495249+0000","last_became_peered":"2024-03-11T21:28:51.495249+0000","last_unstale":"2024-03-11T21:29:42.462052+0000","last_undegraded":"2024-03-11T21:29:42.462052+0000","last_fullsized":"2024-03-11T21:29:42.462052+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.447756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":272,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.436583+0000","last_change":"2024-03-11T21:28:24.055897+0000","last_active":"2024-03-11T21:29:42.436583+0000","last_peered":"2024-03-11T21:29:42.436583+0000","last_clean":"2024-03-11T21:29:42.436583+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:29:42.436583+0000","last_undegraded":"2024-03-11T21:29:42.436583+0000","last_fullsized":"2024-03-11T21:29:42.436583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:28:24.055759+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.055759+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.046651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452730+0000","last_change":"2024-03-11T21:29:05.302929+0000","last_active":"2024-03-11T21:29:42.452730+0000","last_peered":"2024-03-11T21:29:42.452730+0000","last_clean":"2024-03-11T21:29:42.452730+0000","last_became_active":"2024-03-11T21:23:02.930090+0000","last_became_peered":"2024-03-11T21:23:02.930090+0000","last_unstale":"2024-03-11T21:29:42.452730+0000","last_undegraded":"2024-03-11T21:29:42.452730+0000","last_fullsized":"2024-03-11T21:29:42.452730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:05.302619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:29:05.302619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.156964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453608+0000","last_change":"2024-03-11T21:29:16.091449+0000","last_active":"2024-03-11T21:29:42.453608+0000","last_peered":"2024-03-11T21:29:42.453608+0000","last_clean":"2024-03-11T21:29:42.453608+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:29:42.453608+0000","last_undegraded":"2024-03-11T21:29:42.453608+0000","last_fullsized":"2024-03-11T21:29:42.453608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:16.091348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.091348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.775118+0000","last_change":"2024-03-11T21:28:39.272868+0000","last_active":"2024-03-11T21:29:42.775118+0000","last_peered":"2024-03-11T21:29:42.775118+0000","last_clean":"2024-03-11T21:29:42.775118+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:29:42.775118+0000","last_undegraded":"2024-03-11T21:29:42.775118+0000","last_fullsized":"2024-03-11T21:29:42.775118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:39.272593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:39.272593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.200003+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"167'13","reported_seq":126,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661990+0000","last_change":"2024-03-11T21:29:25.718951+0000","last_active":"2024-03-11T21:29:42.661990+0000","last_peered":"2024-03-11T21:29:42.661990+0000","last_clean":"2024-03-11T21:29:42.661990+0000","last_became_active":"2024-03-11T21:28:49.893206+0000","last_became_peered":"2024-03-11T21:28:49.893206+0000","last_unstale":"2024-03-11T21:29:42.661990+0000","last_undegraded":"2024-03-11T21:29:42.661990+0000","last_fullsized":"2024-03-11T21:29:42.661990+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-11T21:29:25.718768+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-11T21:29:25.718768+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.718768+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:53.049819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461076+0000","last_change":"2024-03-11T21:28:49.938085+0000","last_active":"2024-03-11T21:29:42.461076+0000","last_peered":"2024-03-11T21:29:42.461076+0000","last_clean":"2024-03-11T21:29:42.461076+0000","last_became_active":"2024-03-11T21:28:49.937485+0000","last_became_peered":"2024-03-11T21:28:49.937485+0000","last_unstale":"2024-03-11T21:29:42.461076+0000","last_undegraded":"2024-03-11T21:29:42.461076+0000","last_fullsized":"2024-03-11T21:29:42.461076+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.048516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.775701+0000","last_change":"2024-03-11T21:28:49.941607+0000","last_active":"2024-03-11T21:29:42.775701+0000","last_peered":"2024-03-11T21:29:42.775701+0000","last_clean":"2024-03-11T21:29:42.775701+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:29:42.775701+0000","last_undegraded":"2024-03-11T21:29:42.775701+0000","last_fullsized":"2024-03-11T21:29:42.775701+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.768697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"84'11","reported_seq":338,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.450064+0000","last_change":"2024-03-11T21:28:53.325157+0000","last_active":"2024-03-11T21:29:42.450064+0000","last_peered":"2024-03-11T21:29:42.450064+0000","last_clean":"2024-03-11T21:29:42.450064+0000","last_became_active":"2024-03-11T21:23:07.077500+0000","last_became_peered":"2024-03-11T21:23:07.077500+0000","last_unstale":"2024-03-11T21:29:42.450064+0000","last_undegraded":"2024-03-11T21:29:42.450064+0000","last_fullsized":"2024-03-11T21:29:42.450064+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:53.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:53.325007+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.067307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.436716+0000","last_change":"2024-03-11T21:28:50.063147+0000","last_active":"2024-03-11T21:29:42.436716+0000","last_peered":"2024-03-11T21:29:42.436716+0000","last_clean":"2024-03-11T21:29:42.436716+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:29:42.436716+0000","last_undegraded":"2024-03-11T21:29:42.436716+0000","last_fullsized":"2024-03-11T21:29:42.436716+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:50.063069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:28:50.063069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.094104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684105+0000","last_change":"2024-03-11T21:28:29.461226+0000","last_active":"2024-03-11T21:29:42.684105+0000","last_peered":"2024-03-11T21:29:42.684105+0000","last_clean":"2024-03-11T21:29:42.684105+0000","last_became_active":"2024-03-11T21:23:04.944212+0000","last_became_peered":"2024-03-11T21:23:04.944212+0000","last_unstale":"2024-03-11T21:29:42.684105+0000","last_undegraded":"2024-03-11T21:29:42.684105+0000","last_fullsized":"2024-03-11T21:29:42.684105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:28:29.461067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.461067+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.063348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454717+0000","last_change":"2024-03-11T21:28:35.889150+0000","last_active":"2024-03-11T21:29:42.454717+0000","last_peered":"2024-03-11T21:29:42.454717+0000","last_clean":"2024-03-11T21:29:42.454717+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:29:42.454717+0000","last_undegraded":"2024-03-11T21:29:42.454717+0000","last_fullsized":"2024-03-11T21:29:42.454717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:28:35.888993+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.888993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.802448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":246,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462230+0000","last_change":"2024-03-11T21:28:44.386906+0000","last_active":"2024-03-11T21:29:42.462230+0000","last_peered":"2024-03-11T21:29:42.462230+0000","last_clean":"2024-03-11T21:29:42.462230+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:29:42.462230+0000","last_undegraded":"2024-03-11T21:29:42.462230+0000","last_fullsized":"2024-03-11T21:29:42.462230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:28:44.386512+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:28:44.386512+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.985830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":111,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.440935+0000","last_change":"2024-03-11T21:29:26.046726+0000","last_active":"2024-03-11T21:29:42.440935+0000","last_peered":"2024-03-11T21:29:42.440935+0000","last_clean":"2024-03-11T21:29:42.440935+0000","last_became_active":"2024-03-11T21:28:51.352004+0000","last_became_peered":"2024-03-11T21:28:51.352004+0000","last_unstale":"2024-03-11T21:29:42.440935+0000","last_undegraded":"2024-03-11T21:29:42.440935+0000","last_fullsized":"2024-03-11T21:29:42.440935+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:26.046639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:26.046639+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.046639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:46.675133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"142'2","reported_seq":97,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449964+0000","last_change":"2024-03-11T21:28:49.911630+0000","last_active":"2024-03-11T21:29:42.449964+0000","last_peered":"2024-03-11T21:29:42.449964+0000","last_clean":"2024-03-11T21:29:42.449964+0000","last_became_active":"2024-03-11T21:28:49.910995+0000","last_became_peered":"2024-03-11T21:28:49.910995+0000","last_unstale":"2024-03-11T21:29:42.449964+0000","last_undegraded":"2024-03-11T21:29:42.449964+0000","last_fullsized":"2024-03-11T21:29:42.449964+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.683987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462095+0000","last_change":"2024-03-11T21:28:51.473342+0000","last_active":"2024-03-11T21:29:42.462095+0000","last_peered":"2024-03-11T21:29:42.462095+0000","last_clean":"2024-03-11T21:29:42.462095+0000","last_became_active":"2024-03-11T21:28:51.473078+0000","last_became_peered":"2024-03-11T21:28:51.473078+0000","last_unstale":"2024-03-11T21:29:42.462095+0000","last_undegraded":"2024-03-11T21:29:42.462095+0000","last_fullsized":"2024-03-11T21:29:42.462095+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.024046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.436529+0000","last_change":"2024-03-11T21:28:31.138011+0000","last_active":"2024-03-11T21:29:42.436529+0000","last_peered":"2024-03-11T21:29:42.436529+0000","last_clean":"2024-03-11T21:29:42.436529+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:29:42.436529+0000","last_undegraded":"2024-03-11T21:29:42.436529+0000","last_fullsized":"2024-03-11T21:29:42.436529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:31.137790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:31.137790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.376397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":632,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449907+0000","last_change":"2024-03-11T21:27:06.215128+0000","last_active":"2024-03-11T21:29:42.449907+0000","last_peered":"2024-03-11T21:29:42.449907+0000","last_clean":"2024-03-11T21:29:42.449907+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:29:42.449907+0000","last_undegraded":"2024-03-11T21:29:42.449907+0000","last_fullsized":"2024-03-11T21:29:42.449907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:27:06.214816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.214816+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.776990+0000","last_change":"2024-03-11T21:28:28.229695+0000","last_active":"2024-03-11T21:29:42.776990+0000","last_peered":"2024-03-11T21:29:42.776990+0000","last_clean":"2024-03-11T21:29:42.776990+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:29:42.776990+0000","last_undegraded":"2024-03-11T21:29:42.776990+0000","last_fullsized":"2024-03-11T21:29:42.776990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:28:28.229421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.229421+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.083937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559029+0000","last_change":"2024-03-11T21:29:16.233835+0000","last_active":"2024-03-11T21:29:42.559029+0000","last_peered":"2024-03-11T21:29:42.559029+0000","last_clean":"2024-03-11T21:29:42.559029+0000","last_became_active":"2024-03-11T21:23:02.929385+0000","last_became_peered":"2024-03-11T21:23:02.929385+0000","last_unstale":"2024-03-11T21:29:42.559029+0000","last_undegraded":"2024-03-11T21:29:42.559029+0000","last_fullsized":"2024-03-11T21:29:42.559029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:29:16.233739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.233739+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.242229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":244,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444154+0000","last_change":"2024-03-11T21:28:53.054433+0000","last_active":"2024-03-11T21:29:42.444154+0000","last_peered":"2024-03-11T21:29:42.444154+0000","last_clean":"2024-03-11T21:29:42.444154+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:29:42.444154+0000","last_undegraded":"2024-03-11T21:29:42.444154+0000","last_fullsized":"2024-03-11T21:29:42.444154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:53.054172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:53.054172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.130559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"142'1","reported_seq":96,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661700+0000","last_change":"2024-03-11T21:28:49.940466+0000","last_active":"2024-03-11T21:29:42.661700+0000","last_peered":"2024-03-11T21:29:42.661700+0000","last_clean":"2024-03-11T21:29:42.661700+0000","last_became_active":"2024-03-11T21:28:49.937281+0000","last_became_peered":"2024-03-11T21:28:49.937281+0000","last_unstale":"2024-03-11T21:29:42.661700+0000","last_undegraded":"2024-03-11T21:29:42.661700+0000","last_fullsized":"2024-03-11T21:29:42.661700+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.028987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"142'2","reported_seq":97,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.762523+0000","last_change":"2024-03-11T21:28:49.917499+0000","last_active":"2024-03-11T21:29:42.762523+0000","last_peered":"2024-03-11T21:29:42.762523+0000","last_clean":"2024-03-11T21:29:42.762523+0000","last_became_active":"2024-03-11T21:28:49.917221+0000","last_became_peered":"2024-03-11T21:28:49.917221+0000","last_unstale":"2024-03-11T21:29:42.762523+0000","last_undegraded":"2024-03-11T21:29:42.762523+0000","last_fullsized":"2024-03-11T21:29:42.762523+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.699623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.471918+0000","last_change":"2024-03-11T21:28:51.476553+0000","last_active":"2024-03-11T21:29:42.471918+0000","last_peered":"2024-03-11T21:29:42.471918+0000","last_clean":"2024-03-11T21:29:42.471918+0000","last_became_active":"2024-03-11T21:28:51.475222+0000","last_became_peered":"2024-03-11T21:28:51.475222+0000","last_unstale":"2024-03-11T21:29:42.471918+0000","last_undegraded":"2024-03-11T21:29:42.471918+0000","last_fullsized":"2024-03-11T21:29:42.471918+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.462218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443637+0000","last_change":"2024-03-11T21:28:25.018416+0000","last_active":"2024-03-11T21:29:42.443637+0000","last_peered":"2024-03-11T21:29:42.443637+0000","last_clean":"2024-03-11T21:29:42.443637+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:29:42.443637+0000","last_undegraded":"2024-03-11T21:29:42.443637+0000","last_fullsized":"2024-03-11T21:29:42.443637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.018198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.018198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.575381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":382,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554414+0000","last_change":"2024-03-11T21:29:01.172157+0000","last_active":"2024-03-11T21:29:42.554414+0000","last_peered":"2024-03-11T21:29:42.554414+0000","last_clean":"2024-03-11T21:29:42.554414+0000","last_became_active":"2024-03-11T21:23:05.023439+0000","last_became_peered":"2024-03-11T21:23:05.023439+0000","last_unstale":"2024-03-11T21:29:42.554414+0000","last_undegraded":"2024-03-11T21:29:42.554414+0000","last_fullsized":"2024-03-11T21:29:42.554414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T21:29:01.171872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:01.171872+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.084949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462415+0000","last_change":"2024-03-11T21:29:02.325660+0000","last_active":"2024-03-11T21:29:42.462415+0000","last_peered":"2024-03-11T21:29:42.462415+0000","last_clean":"2024-03-11T21:29:42.462415+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:29:42.462415+0000","last_undegraded":"2024-03-11T21:29:42.462415+0000","last_fullsized":"2024-03-11T21:29:42.462415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:02.325563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:02.325563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.881802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.553802+0000","last_change":"2024-03-11T21:28:48.172948+0000","last_active":"2024-03-11T21:29:42.553802+0000","last_peered":"2024-03-11T21:29:42.553802+0000","last_clean":"2024-03-11T21:29:42.553802+0000","last_became_active":"2024-03-11T21:23:09.127231+0000","last_became_peered":"2024-03-11T21:23:09.127231+0000","last_unstale":"2024-03-11T21:29:42.553802+0000","last_undegraded":"2024-03-11T21:29:42.553802+0000","last_fullsized":"2024-03-11T21:29:42.553802+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.172758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.172758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.476811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684489+0000","last_change":"2024-03-11T21:28:51.345067+0000","last_active":"2024-03-11T21:29:42.684489+0000","last_peered":"2024-03-11T21:29:42.684489+0000","last_clean":"2024-03-11T21:29:42.684489+0000","last_became_active":"2024-03-11T21:28:51.343036+0000","last_became_peered":"2024-03-11T21:28:51.343036+0000","last_unstale":"2024-03-11T21:29:42.684489+0000","last_undegraded":"2024-03-11T21:29:42.684489+0000","last_fullsized":"2024-03-11T21:29:42.684489+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.489438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683658+0000","last_change":"2024-03-11T21:28:51.439914+0000","last_active":"2024-03-11T21:29:42.683658+0000","last_peered":"2024-03-11T21:29:42.683658+0000","last_clean":"2024-03-11T21:29:42.683658+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:29:42.683658+0000","last_undegraded":"2024-03-11T21:29:42.683658+0000","last_fullsized":"2024-03-11T21:29:42.683658+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.707137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453544+0000","last_change":"2024-03-11T21:29:14.068625+0000","last_active":"2024-03-11T21:29:42.453544+0000","last_peered":"2024-03-11T21:29:42.453544+0000","last_clean":"2024-03-11T21:29:42.453544+0000","last_became_active":"2024-03-11T21:23:07.087430+0000","last_became_peered":"2024-03-11T21:23:07.087430+0000","last_unstale":"2024-03-11T21:29:42.453544+0000","last_undegraded":"2024-03-11T21:29:42.453544+0000","last_fullsized":"2024-03-11T21:29:42.453544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:14.068405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.068405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.432113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452516+0000","last_change":"2024-03-11T21:28:48.365055+0000","last_active":"2024-03-11T21:29:42.452516+0000","last_peered":"2024-03-11T21:29:42.452516+0000","last_clean":"2024-03-11T21:29:42.452516+0000","last_became_active":"2024-03-11T21:23:02.933265+0000","last_became_peered":"2024-03-11T21:23:02.933265+0000","last_unstale":"2024-03-11T21:29:42.452516+0000","last_undegraded":"2024-03-11T21:29:42.452516+0000","last_fullsized":"2024-03-11T21:29:42.452516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.364816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.364816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.344749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688301+0000","last_change":"2024-03-11T21:28:28.500423+0000","last_active":"2024-03-11T21:29:42.688301+0000","last_peered":"2024-03-11T21:29:42.688301+0000","last_clean":"2024-03-11T21:29:42.688301+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:29:42.688301+0000","last_undegraded":"2024-03-11T21:29:42.688301+0000","last_fullsized":"2024-03-11T21:29:42.688301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:28:28.500168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.500168+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554707+0000","last_change":"2024-03-11T21:28:55.207904+0000","last_active":"2024-03-11T21:29:42.554707+0000","last_peered":"2024-03-11T21:29:42.554707+0000","last_clean":"2024-03-11T21:29:42.554707+0000","last_became_active":"2024-03-11T21:23:09.141604+0000","last_became_peered":"2024-03-11T21:23:09.141604+0000","last_unstale":"2024-03-11T21:29:42.554707+0000","last_undegraded":"2024-03-11T21:29:42.554707+0000","last_fullsized":"2024-03-11T21:29:42.554707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:55.207711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:55.207711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.197071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"144'4","reported_seq":105,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454438+0000","last_change":"2024-03-11T21:28:51.386273+0000","last_active":"2024-03-11T21:29:42.454438+0000","last_peered":"2024-03-11T21:29:42.454438+0000","last_clean":"2024-03-11T21:29:42.454438+0000","last_became_active":"2024-03-11T21:28:51.383996+0000","last_became_peered":"2024-03-11T21:28:51.383996+0000","last_unstale":"2024-03-11T21:29:42.454438+0000","last_undegraded":"2024-03-11T21:29:42.454438+0000","last_fullsized":"2024-03-11T21:29:42.454438+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.160852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165836,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.463053+0000","last_change":"2024-03-11T21:28:51.478245+0000","last_active":"2024-03-11T21:29:42.463053+0000","last_peered":"2024-03-11T21:29:42.463053+0000","last_clean":"2024-03-11T21:29:42.463053+0000","last_became_active":"2024-03-11T21:28:51.476878+0000","last_became_peered":"2024-03-11T21:28:51.476878+0000","last_unstale":"2024-03-11T21:29:42.463053+0000","last_undegraded":"2024-03-11T21:29:42.463053+0000","last_fullsized":"2024-03-11T21:29:42.463053+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.272480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449483+0000","last_change":"2024-03-11T21:28:49.941751+0000","last_active":"2024-03-11T21:29:42.449483+0000","last_peered":"2024-03-11T21:29:42.449483+0000","last_clean":"2024-03-11T21:29:42.449483+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:29:42.449483+0000","last_undegraded":"2024-03-11T21:29:42.449483+0000","last_fullsized":"2024-03-11T21:29:42.449483+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":240,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.469125+0000","last_change":"2024-03-11T21:28:40.398142+0000","last_active":"2024-03-11T21:29:42.469125+0000","last_peered":"2024-03-11T21:29:42.469125+0000","last_clean":"2024-03-11T21:29:42.469125+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:29:42.469125+0000","last_undegraded":"2024-03-11T21:29:42.469125+0000","last_fullsized":"2024-03-11T21:29:42.469125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:40.397932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:40.397932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.716423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":511,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452628+0000","last_change":"2024-03-11T21:28:41.442100+0000","last_active":"2024-03-11T21:29:42.452628+0000","last_peered":"2024-03-11T21:29:42.452628+0000","last_clean":"2024-03-11T21:29:42.452628+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:29:42.452628+0000","last_undegraded":"2024-03-11T21:29:42.452628+0000","last_fullsized":"2024-03-11T21:29:42.452628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:28:41.441866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.441866+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.005513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452341+0000","last_change":"2024-03-11T21:28:21.130354+0000","last_active":"2024-03-11T21:29:42.452341+0000","last_peered":"2024-03-11T21:29:42.452341+0000","last_clean":"2024-03-11T21:29:42.452341+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:29:42.452341+0000","last_undegraded":"2024-03-11T21:29:42.452341+0000","last_fullsized":"2024-03-11T21:29:42.452341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:21.130135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:28:21.130135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.772641+0000","last_change":"2024-03-11T21:27:25.263826+0000","last_active":"2024-03-11T21:29:42.772641+0000","last_peered":"2024-03-11T21:29:42.772641+0000","last_clean":"2024-03-11T21:29:42.772641+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:29:42.772641+0000","last_undegraded":"2024-03-11T21:29:42.772641+0000","last_fullsized":"2024-03-11T21:29:42.772641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:27:25.263676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.263676+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453715+0000","last_change":"2024-03-11T21:27:14.796892+0000","last_active":"2024-03-11T21:29:42.453715+0000","last_peered":"2024-03-11T21:29:42.453715+0000","last_clean":"2024-03-11T21:29:42.453715+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:29:42.453715+0000","last_undegraded":"2024-03-11T21:29:42.453715+0000","last_fullsized":"2024-03-11T21:29:42.453715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:14.796603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.796603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554310+0000","last_change":"2024-03-11T21:28:51.297745+0000","last_active":"2024-03-11T21:29:42.554310+0000","last_peered":"2024-03-11T21:29:42.554310+0000","last_clean":"2024-03-11T21:29:42.554310+0000","last_became_active":"2024-03-11T21:28:51.292188+0000","last_became_peered":"2024-03-11T21:28:51.292188+0000","last_unstale":"2024-03-11T21:29:42.554310+0000","last_undegraded":"2024-03-11T21:29:42.554310+0000","last_fullsized":"2024-03-11T21:29:42.554310+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.597627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454605+0000","last_change":"2024-03-11T21:28:51.389227+0000","last_active":"2024-03-11T21:29:42.454605+0000","last_peered":"2024-03-11T21:29:42.454605+0000","last_clean":"2024-03-11T21:29:42.454605+0000","last_became_active":"2024-03-11T21:28:51.384917+0000","last_became_peered":"2024-03-11T21:28:51.384917+0000","last_unstale":"2024-03-11T21:29:42.454605+0000","last_undegraded":"2024-03-11T21:29:42.454605+0000","last_fullsized":"2024-03-11T21:29:42.454605+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.698923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441138+0000","last_change":"2024-03-11T21:28:51.377051+0000","last_active":"2024-03-11T21:29:42.441138+0000","last_peered":"2024-03-11T21:29:42.441138+0000","last_clean":"2024-03-11T21:29:42.441138+0000","last_became_active":"2024-03-11T21:28:51.374924+0000","last_became_peered":"2024-03-11T21:28:51.374924+0000","last_unstale":"2024-03-11T21:29:42.441138+0000","last_undegraded":"2024-03-11T21:29:42.441138+0000","last_fullsized":"2024-03-11T21:29:42.441138+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.151792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.679402+0000","last_change":"2024-03-11T21:28:43.970353+0000","last_active":"2024-03-11T21:29:42.679402+0000","last_peered":"2024-03-11T21:29:42.679402+0000","last_clean":"2024-03-11T21:29:42.679402+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:29:42.679402+0000","last_undegraded":"2024-03-11T21:29:42.679402+0000","last_fullsized":"2024-03-11T21:29:42.679402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:43.970146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:43.970146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.003147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461202+0000","last_change":"2024-03-11T21:28:45.879467+0000","last_active":"2024-03-11T21:29:42.461202+0000","last_peered":"2024-03-11T21:29:42.461202+0000","last_clean":"2024-03-11T21:29:42.461202+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:29:42.461202+0000","last_undegraded":"2024-03-11T21:29:42.461202+0000","last_fullsized":"2024-03-11T21:29:42.461202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:45.879374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:45.879374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.534702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"149'26","reported_seq":292,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453774+0000","last_change":"2024-03-11T21:29:02.205175+0000","last_active":"2024-03-11T21:29:42.453774+0000","last_peered":"2024-03-11T21:29:42.453774+0000","last_clean":"2024-03-11T21:29:42.453774+0000","last_became_active":"2024-03-11T21:23:05.026287+0000","last_became_peered":"2024-03-11T21:23:05.026287+0000","last_unstale":"2024-03-11T21:29:42.453774+0000","last_undegraded":"2024-03-11T21:29:42.453774+0000","last_fullsized":"2024-03-11T21:29:42.453774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:29:02.205088+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:21.844962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":244,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462309+0000","last_change":"2024-03-11T21:29:10.347962+0000","last_active":"2024-03-11T21:29:42.462309+0000","last_peered":"2024-03-11T21:29:42.462309+0000","last_clean":"2024-03-11T21:29:42.462309+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:29:42.462309+0000","last_undegraded":"2024-03-11T21:29:42.462309+0000","last_fullsized":"2024-03-11T21:29:42.462309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:10.347717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:29:10.347717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:31.073525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.472559+0000","last_change":"2024-03-11T21:28:51.424113+0000","last_active":"2024-03-11T21:29:42.472559+0000","last_peered":"2024-03-11T21:29:42.472559+0000","last_clean":"2024-03-11T21:29:42.472559+0000","last_became_active":"2024-03-11T21:28:51.417879+0000","last_became_peered":"2024-03-11T21:28:51.417879+0000","last_unstale":"2024-03-11T21:29:42.472559+0000","last_undegraded":"2024-03-11T21:29:42.472559+0000","last_fullsized":"2024-03-11T21:29:42.472559+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.586125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688746+0000","last_change":"2024-03-11T21:28:51.394594+0000","last_active":"2024-03-11T21:29:42.688746+0000","last_peered":"2024-03-11T21:29:42.688746+0000","last_clean":"2024-03-11T21:29:42.688746+0000","last_became_active":"2024-03-11T21:28:51.394292+0000","last_became_peered":"2024-03-11T21:28:51.394292+0000","last_unstale":"2024-03-11T21:29:42.688746+0000","last_undegraded":"2024-03-11T21:29:42.688746+0000","last_fullsized":"2024-03-11T21:29:42.688746+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.337301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672827+0000","last_change":"2024-03-11T21:28:51.400250+0000","last_active":"2024-03-11T21:29:42.672827+0000","last_peered":"2024-03-11T21:29:42.672827+0000","last_clean":"2024-03-11T21:29:42.672827+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:29:42.672827+0000","last_undegraded":"2024-03-11T21:29:42.672827+0000","last_fullsized":"2024-03-11T21:29:42.672827+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.606108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":240,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.682880+0000","last_change":"2024-03-11T21:28:32.485022+0000","last_active":"2024-03-11T21:29:42.682880+0000","last_peered":"2024-03-11T21:29:42.682880+0000","last_clean":"2024-03-11T21:29:42.682880+0000","last_became_active":"2024-03-11T21:23:07.086583+0000","last_became_peered":"2024-03-11T21:23:07.086583+0000","last_unstale":"2024-03-11T21:29:42.682880+0000","last_undegraded":"2024-03-11T21:29:42.682880+0000","last_fullsized":"2024-03-11T21:29:42.682880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.484788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.484788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.153491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":296,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453122+0000","last_change":"2024-03-11T21:28:29.138041+0000","last_active":"2024-03-11T21:29:42.453122+0000","last_peered":"2024-03-11T21:29:42.453122+0000","last_clean":"2024-03-11T21:29:42.453122+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:29:42.453122+0000","last_undegraded":"2024-03-11T21:29:42.453122+0000","last_fullsized":"2024-03-11T21:29:42.453122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:28:29.137906+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.137906+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.564245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441619+0000","last_change":"2024-03-11T21:28:34.967885+0000","last_active":"2024-03-11T21:29:42.441619+0000","last_peered":"2024-03-11T21:29:42.441619+0000","last_clean":"2024-03-11T21:29:42.441619+0000","last_became_active":"2024-03-11T21:23:02.935380+0000","last_became_peered":"2024-03-11T21:23:02.935380+0000","last_unstale":"2024-03-11T21:29:42.441619+0000","last_undegraded":"2024-03-11T21:29:42.441619+0000","last_fullsized":"2024-03-11T21:29:42.441619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.967689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.967689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.770778+0000","last_change":"2024-03-11T21:28:38.304169+0000","last_active":"2024-03-11T21:29:42.770778+0000","last_peered":"2024-03-11T21:29:42.770778+0000","last_clean":"2024-03-11T21:29:42.770778+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:29:42.770778+0000","last_undegraded":"2024-03-11T21:29:42.770778+0000","last_fullsized":"2024-03-11T21:29:42.770778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:38.304088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:38.304088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"144'2","reported_seq":97,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452759+0000","last_change":"2024-03-11T21:28:51.384723+0000","last_active":"2024-03-11T21:29:42.452759+0000","last_peered":"2024-03-11T21:29:42.452759+0000","last_clean":"2024-03-11T21:29:42.452759+0000","last_became_active":"2024-03-11T21:28:51.384441+0000","last_became_peered":"2024-03-11T21:28:51.384441+0000","last_unstale":"2024-03-11T21:29:42.452759+0000","last_undegraded":"2024-03-11T21:29:42.452759+0000","last_fullsized":"2024-03-11T21:29:42.452759+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.107368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"144'1","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452511+0000","last_change":"2024-03-11T21:28:51.421124+0000","last_active":"2024-03-11T21:29:42.452511+0000","last_peered":"2024-03-11T21:29:42.452511+0000","last_clean":"2024-03-11T21:29:42.452511+0000","last_became_active":"2024-03-11T21:28:51.420792+0000","last_became_peered":"2024-03-11T21:28:51.420792+0000","last_unstale":"2024-03-11T21:29:42.452511+0000","last_undegraded":"2024-03-11T21:29:42.452511+0000","last_fullsized":"2024-03-11T21:29:42.452511+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.944361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"142'10","reported_seq":105,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661644+0000","last_change":"2024-03-11T21:28:49.940619+0000","last_active":"2024-03-11T21:29:42.661644+0000","last_peered":"2024-03-11T21:29:42.661644+0000","last_clean":"2024-03-11T21:29:42.661644+0000","last_became_active":"2024-03-11T21:28:49.937682+0000","last_became_peered":"2024-03-11T21:28:49.937682+0000","last_unstale":"2024-03-11T21:29:42.661644+0000","last_undegraded":"2024-03-11T21:29:42.661644+0000","last_fullsized":"2024-03-11T21:29:42.661644+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.331922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452754+0000","last_change":"2024-03-11T21:28:51.328541+0000","last_active":"2024-03-11T21:29:42.452754+0000","last_peered":"2024-03-11T21:29:42.452754+0000","last_clean":"2024-03-11T21:29:42.452754+0000","last_became_active":"2024-03-11T21:28:51.325987+0000","last_became_peered":"2024-03-11T21:28:51.325987+0000","last_unstale":"2024-03-11T21:29:42.452754+0000","last_undegraded":"2024-03-11T21:29:42.452754+0000","last_fullsized":"2024-03-11T21:29:42.452754+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.337576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":343,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683288+0000","last_change":"2024-03-11T21:28:50.362613+0000","last_active":"2024-03-11T21:29:42.683288+0000","last_peered":"2024-03-11T21:29:42.683288+0000","last_clean":"2024-03-11T21:29:42.683288+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:29:42.683288+0000","last_undegraded":"2024-03-11T21:29:42.683288+0000","last_fullsized":"2024-03-11T21:29:42.683288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:28:50.362381+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:28:50.362381+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.885376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552731+0000","last_change":"2024-03-11T21:28:34.295455+0000","last_active":"2024-03-11T21:29:42.552731+0000","last_peered":"2024-03-11T21:29:42.552731+0000","last_clean":"2024-03-11T21:29:42.552731+0000","last_became_active":"2024-03-11T21:23:06.071369+0000","last_became_peered":"2024-03-11T21:23:06.071369+0000","last_unstale":"2024-03-11T21:29:42.552731+0000","last_undegraded":"2024-03-11T21:29:42.552731+0000","last_fullsized":"2024-03-11T21:29:42.552731+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:28:34.295197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.295197+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.009264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":354,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.445077+0000","last_change":"2024-03-11T21:28:48.820513+0000","last_active":"2024-03-11T21:29:42.445077+0000","last_peered":"2024-03-11T21:29:42.445077+0000","last_clean":"2024-03-11T21:29:42.445077+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:29:42.445077+0000","last_undegraded":"2024-03-11T21:29:42.445077+0000","last_fullsized":"2024-03-11T21:29:42.445077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:28:48.820383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.820383+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:10.111797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554515+0000","last_change":"2024-03-11T21:28:22.185924+0000","last_active":"2024-03-11T21:29:42.554515+0000","last_peered":"2024-03-11T21:29:42.554515+0000","last_clean":"2024-03-11T21:29:42.554515+0000","last_became_active":"2024-03-11T21:23:02.932104+0000","last_became_peered":"2024-03-11T21:23:02.932104+0000","last_unstale":"2024-03-11T21:29:42.554515+0000","last_undegraded":"2024-03-11T21:29:42.554515+0000","last_fullsized":"2024-03-11T21:29:42.554515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:22.185727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:22.185727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":244,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.673042+0000","last_change":"2024-03-11T21:28:53.386165+0000","last_active":"2024-03-11T21:29:42.673042+0000","last_peered":"2024-03-11T21:29:42.673042+0000","last_clean":"2024-03-11T21:29:42.673042+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:29:42.673042+0000","last_undegraded":"2024-03-11T21:29:42.673042+0000","last_fullsized":"2024-03-11T21:29:42.673042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:53.386063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:28:53.386063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.246055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"168'6","reported_seq":117,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452256+0000","last_change":"2024-03-11T21:29:26.374743+0000","last_active":"2024-03-11T21:29:42.452256+0000","last_peered":"2024-03-11T21:29:42.452256+0000","last_clean":"2024-03-11T21:29:42.452256+0000","last_became_active":"2024-03-11T21:28:51.471832+0000","last_became_peered":"2024-03-11T21:28:51.471832+0000","last_unstale":"2024-03-11T21:29:42.452256+0000","last_undegraded":"2024-03-11T21:29:42.452256+0000","last_fullsized":"2024-03-11T21:29:42.452256+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'6","last_scrub_stamp":"2024-03-11T21:29:26.374621+0000","last_deep_scrub":"168'6","last_deep_scrub_stamp":"2024-03-11T21:29:26.374621+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.374621+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.990530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447841+0000","last_change":"2024-03-11T21:28:51.365938+0000","last_active":"2024-03-11T21:29:42.447841+0000","last_peered":"2024-03-11T21:29:42.447841+0000","last_clean":"2024-03-11T21:29:42.447841+0000","last_became_active":"2024-03-11T21:28:51.365615+0000","last_became_peered":"2024-03-11T21:28:51.365615+0000","last_unstale":"2024-03-11T21:29:42.447841+0000","last_undegraded":"2024-03-11T21:29:42.447841+0000","last_fullsized":"2024-03-11T21:29:42.447841+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.234686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554413+0000","last_change":"2024-03-11T21:28:51.337186+0000","last_active":"2024-03-11T21:29:42.554413+0000","last_peered":"2024-03-11T21:29:42.554413+0000","last_clean":"2024-03-11T21:29:42.554413+0000","last_became_active":"2024-03-11T21:28:51.313606+0000","last_became_peered":"2024-03-11T21:28:51.313606+0000","last_unstale":"2024-03-11T21:29:42.554413+0000","last_undegraded":"2024-03-11T21:29:42.554413+0000","last_fullsized":"2024-03-11T21:29:42.554413+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.448195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.661261+0000","last_change":"2024-03-11T21:28:58.950391+0000","last_active":"2024-03-11T21:29:42.661261+0000","last_peered":"2024-03-11T21:29:42.661261+0000","last_clean":"2024-03-11T21:29:42.661261+0000","last_became_active":"2024-03-11T21:23:07.064118+0000","last_became_peered":"2024-03-11T21:23:07.064118+0000","last_unstale":"2024-03-11T21:29:42.661261+0000","last_undegraded":"2024-03-11T21:29:42.661261+0000","last_fullsized":"2024-03-11T21:29:42.661261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:58.950292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:28:58.950292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.623112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":275,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.472281+0000","last_change":"2024-03-11T21:28:59.327951+0000","last_active":"2024-03-11T21:29:42.472281+0000","last_peered":"2024-03-11T21:29:42.472281+0000","last_clean":"2024-03-11T21:29:42.472281+0000","last_became_active":"2024-03-11T21:23:04.941932+0000","last_became_peered":"2024-03-11T21:23:04.941932+0000","last_unstale":"2024-03-11T21:29:42.472281+0000","last_undegraded":"2024-03-11T21:29:42.472281+0000","last_fullsized":"2024-03-11T21:29:42.472281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.327656+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.449333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":273,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.440796+0000","last_change":"2024-03-11T21:28:42.094124+0000","last_active":"2024-03-11T21:29:42.440796+0000","last_peered":"2024-03-11T21:29:42.440796+0000","last_clean":"2024-03-11T21:29:42.440796+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:29:42.440796+0000","last_undegraded":"2024-03-11T21:29:42.440796+0000","last_fullsized":"2024-03-11T21:29:42.440796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:28:42.093981+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:28:42.093981+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.732173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.761363+0000","last_change":"2024-03-11T21:28:54.362549+0000","last_active":"2024-03-11T21:29:42.761363+0000","last_peered":"2024-03-11T21:29:42.761363+0000","last_clean":"2024-03-11T21:29:42.761363+0000","last_became_active":"2024-03-11T21:23:09.157392+0000","last_became_peered":"2024-03-11T21:23:09.157392+0000","last_unstale":"2024-03-11T21:29:42.761363+0000","last_undegraded":"2024-03-11T21:29:42.761363+0000","last_fullsized":"2024-03-11T21:29:42.761363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:54.362432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.362432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:16.993999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442137+0000","last_change":"2024-03-11T21:28:51.406495+0000","last_active":"2024-03-11T21:29:42.442137+0000","last_peered":"2024-03-11T21:29:42.442137+0000","last_clean":"2024-03-11T21:29:42.442137+0000","last_became_active":"2024-03-11T21:28:51.406203+0000","last_became_peered":"2024-03-11T21:28:51.406203+0000","last_unstale":"2024-03-11T21:29:42.442137+0000","last_undegraded":"2024-03-11T21:29:42.442137+0000","last_fullsized":"2024-03-11T21:29:42.442137+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.934231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.761975+0000","last_change":"2024-03-11T21:28:49.948690+0000","last_active":"2024-03-11T21:29:42.761975+0000","last_peered":"2024-03-11T21:29:42.761975+0000","last_clean":"2024-03-11T21:29:42.761975+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:29:42.761975+0000","last_undegraded":"2024-03-11T21:29:42.761975+0000","last_fullsized":"2024-03-11T21:29:42.761975+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.512332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.469189+0000","last_change":"2024-03-11T21:29:19.356816+0000","last_active":"2024-03-11T21:29:42.469189+0000","last_peered":"2024-03-11T21:29:42.469189+0000","last_clean":"2024-03-11T21:29:42.469189+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:29:42.469189+0000","last_undegraded":"2024-03-11T21:29:42.469189+0000","last_fullsized":"2024-03-11T21:29:42.469189+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:29:19.356702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.356702+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.354746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.660998+0000","last_change":"2024-03-11T21:28:24.913783+0000","last_active":"2024-03-11T21:29:42.660998+0000","last_peered":"2024-03-11T21:29:42.660998+0000","last_clean":"2024-03-11T21:29:42.660998+0000","last_became_active":"2024-03-11T21:23:02.930433+0000","last_became_peered":"2024-03-11T21:23:02.930433+0000","last_unstale":"2024-03-11T21:29:42.660998+0000","last_undegraded":"2024-03-11T21:29:42.660998+0000","last_fullsized":"2024-03-11T21:29:42.660998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:24.913679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.913679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.777816+0000","last_change":"2024-03-11T21:27:20.200413+0000","last_active":"2024-03-11T21:29:42.777816+0000","last_peered":"2024-03-11T21:29:42.777816+0000","last_clean":"2024-03-11T21:29:42.777816+0000","last_became_active":"2024-03-11T21:23:07.088296+0000","last_became_peered":"2024-03-11T21:23:07.088296+0000","last_unstale":"2024-03-11T21:29:42.777816+0000","last_undegraded":"2024-03-11T21:29:42.777816+0000","last_fullsized":"2024-03-11T21:29:42.777816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.200196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.200196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":242,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.676680+0000","last_change":"2024-03-11T21:28:48.843905+0000","last_active":"2024-03-11T21:29:42.676680+0000","last_peered":"2024-03-11T21:29:42.676680+0000","last_clean":"2024-03-11T21:29:42.676680+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:29:42.676680+0000","last_undegraded":"2024-03-11T21:29:42.676680+0000","last_fullsized":"2024-03-11T21:29:42.676680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.843713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.843713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.937146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.451759+0000","last_change":"2024-03-11T21:28:51.414138+0000","last_active":"2024-03-11T21:29:42.451759+0000","last_peered":"2024-03-11T21:29:42.451759+0000","last_clean":"2024-03-11T21:29:42.451759+0000","last_became_active":"2024-03-11T21:28:51.413871+0000","last_became_peered":"2024-03-11T21:28:51.413871+0000","last_unstale":"2024-03-11T21:29:42.451759+0000","last_undegraded":"2024-03-11T21:29:42.451759+0000","last_fullsized":"2024-03-11T21:29:42.451759+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:05.249719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1f","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454559+0000","last_change":"2024-03-11T21:28:51.386509+0000","last_active":"2024-03-11T21:29:42.454559+0000","last_peered":"2024-03-11T21:29:42.454559+0000","last_clean":"2024-03-11T21:29:42.454559+0000","last_became_active":"2024-03-11T21:28:51.384402+0000","last_became_peered":"2024-03-11T21:28:51.384402+0000","last_unstale":"2024-03-11T21:29:42.454559+0000","last_undegraded":"2024-03-11T21:29:42.454559+0000","last_fullsized":"2024-03-11T21:29:42.454559+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.257683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.772844+0000","last_change":"2024-03-11T21:28:49.958604+0000","last_active":"2024-03-11T21:29:42.772844+0000","last_peered":"2024-03-11T21:29:42.772844+0000","last_clean":"2024-03-11T21:29:42.772844+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:29:42.772844+0000","last_undegraded":"2024-03-11T21:29:42.772844+0000","last_fullsized":"2024-03-11T21:29:42.772844+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.169959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.438798+0000","last_change":"2024-03-11T21:28:36.897072+0000","last_active":"2024-03-11T21:29:42.438798+0000","last_peered":"2024-03-11T21:29:42.438798+0000","last_clean":"2024-03-11T21:29:42.438798+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:29:42.438798+0000","last_undegraded":"2024-03-11T21:29:42.438798+0000","last_fullsized":"2024-03-11T21:29:42.438798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:28:36.896795+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.896795+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.450968+0000","last_change":"2024-03-11T21:28:29.965841+0000","last_active":"2024-03-11T21:29:42.450968+0000","last_peered":"2024-03-11T21:29:42.450968+0000","last_clean":"2024-03-11T21:29:42.450968+0000","last_became_active":"2024-03-11T21:23:02.925315+0000","last_became_peered":"2024-03-11T21:23:02.925315+0000","last_unstale":"2024-03-11T21:29:42.450968+0000","last_undegraded":"2024-03-11T21:29:42.450968+0000","last_fullsized":"2024-03-11T21:29:42.450968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.965626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.965626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.833408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448952+0000","last_change":"2024-03-11T21:29:08.955092+0000","last_active":"2024-03-11T21:29:42.448952+0000","last_peered":"2024-03-11T21:29:42.448952+0000","last_clean":"2024-03-11T21:29:42.448952+0000","last_became_active":"2024-03-11T21:23:07.084155+0000","last_became_peered":"2024-03-11T21:23:07.084155+0000","last_unstale":"2024-03-11T21:29:42.448952+0000","last_undegraded":"2024-03-11T21:29:42.448952+0000","last_fullsized":"2024-03-11T21:29:42.448952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:29:08.954967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.388120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462204+0000","last_change":"2024-03-11T21:28:45.377770+0000","last_active":"2024-03-11T21:29:42.462204+0000","last_peered":"2024-03-11T21:29:42.462204+0000","last_clean":"2024-03-11T21:29:42.462204+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:29:42.462204+0000","last_undegraded":"2024-03-11T21:29:42.462204+0000","last_fullsized":"2024-03-11T21:29:42.462204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:45.377668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:45.377668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.941167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1e","version":"144'3","reported_seq":100,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.442125+0000","last_change":"2024-03-11T21:28:51.343644+0000","last_active":"2024-03-11T21:29:42.442125+0000","last_peered":"2024-03-11T21:29:42.442125+0000","last_clean":"2024-03-11T21:29:42.442125+0000","last_became_active":"2024-03-11T21:28:51.342287+0000","last_became_peered":"2024-03-11T21:28:51.342287+0000","last_unstale":"2024-03-11T21:29:42.442125+0000","last_undegraded":"2024-03-11T21:29:42.442125+0000","last_fullsized":"2024-03-11T21:29:42.442125+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:52.883916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":27325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.687697+0000","last_change":"2024-03-11T21:28:51.307699+0000","last_active":"2024-03-11T21:29:42.687697+0000","last_peered":"2024-03-11T21:29:42.687697+0000","last_clean":"2024-03-11T21:29:42.687697+0000","last_became_active":"2024-03-11T21:28:51.305904+0000","last_became_peered":"2024-03-11T21:28:51.305904+0000","last_unstale":"2024-03-11T21:29:42.687697+0000","last_undegraded":"2024-03-11T21:29:42.687697+0000","last_fullsized":"2024-03-11T21:29:42.687697+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.563052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.457182+0000","last_change":"2024-03-11T21:28:49.938600+0000","last_active":"2024-03-11T21:29:42.457182+0000","last_peered":"2024-03-11T21:29:42.457182+0000","last_clean":"2024-03-11T21:29:42.457182+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:29:42.457182+0000","last_undegraded":"2024-03-11T21:29:42.457182+0000","last_fullsized":"2024-03-11T21:29:42.457182+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.604956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462610+0000","last_change":"2024-03-11T21:28:39.407473+0000","last_active":"2024-03-11T21:29:42.462610+0000","last_peered":"2024-03-11T21:29:42.462610+0000","last_clean":"2024-03-11T21:29:42.462610+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:29:42.462610+0000","last_undegraded":"2024-03-11T21:29:42.462610+0000","last_fullsized":"2024-03-11T21:29:42.462610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:39.407239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:39.407239+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.358532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441469+0000","last_change":"2024-03-11T21:28:36.908078+0000","last_active":"2024-03-11T21:29:42.441469+0000","last_peered":"2024-03-11T21:29:42.441469+0000","last_clean":"2024-03-11T21:29:42.441469+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:29:42.441469+0000","last_undegraded":"2024-03-11T21:29:42.441469+0000","last_fullsized":"2024-03-11T21:29:42.441469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:36.907739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.907739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.196001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"84'11","reported_seq":337,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448804+0000","last_change":"2024-03-11T21:29:22.005088+0000","last_active":"2024-03-11T21:29:42.448804+0000","last_peered":"2024-03-11T21:29:42.448804+0000","last_clean":"2024-03-11T21:29:42.448804+0000","last_became_active":"2024-03-11T21:23:07.084601+0000","last_became_peered":"2024-03-11T21:23:07.084601+0000","last_unstale":"2024-03-11T21:29:42.448804+0000","last_undegraded":"2024-03-11T21:29:42.448804+0000","last_fullsized":"2024-03-11T21:29:42.448804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:29:22.004972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:22.004972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:50.645199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684401+0000","last_change":"2024-03-11T21:28:40.492998+0000","last_active":"2024-03-11T21:29:42.684401+0000","last_peered":"2024-03-11T21:29:42.684401+0000","last_clean":"2024-03-11T21:29:42.684401+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:29:42.684401+0000","last_undegraded":"2024-03-11T21:29:42.684401+0000","last_fullsized":"2024-03-11T21:29:42.684401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:40.492866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:05.686435+0000","last_clean_scrub_stamp":"2024-03-11T21:28:40.492866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.425247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"142'1","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.463210+0000","last_change":"2024-03-11T21:28:51.494676+0000","last_active":"2024-03-11T21:29:42.463210+0000","last_peered":"2024-03-11T21:29:42.463210+0000","last_clean":"2024-03-11T21:29:42.463210+0000","last_became_active":"2024-03-11T21:28:51.494360+0000","last_became_peered":"2024-03-11T21:28:51.494360+0000","last_unstale":"2024-03-11T21:29:42.463210+0000","last_undegraded":"2024-03-11T21:29:42.463210+0000","last_fullsized":"2024-03-11T21:29:42.463210+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.769204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1d","version":"142'7","reported_seq":100,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.683664+0000","last_change":"2024-03-11T21:28:51.418349+0000","last_active":"2024-03-11T21:29:42.683664+0000","last_peered":"2024-03-11T21:29:42.683664+0000","last_clean":"2024-03-11T21:29:42.683664+0000","last_became_active":"2024-03-11T21:28:51.389315+0000","last_became_peered":"2024-03-11T21:28:51.389315+0000","last_unstale":"2024-03-11T21:29:42.683664+0000","last_undegraded":"2024-03-11T21:29:42.683664+0000","last_fullsized":"2024-03-11T21:29:42.683664+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.912337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.439081+0000","last_change":"2024-03-11T21:28:51.358883+0000","last_active":"2024-03-11T21:29:42.439081+0000","last_peered":"2024-03-11T21:29:42.439081+0000","last_clean":"2024-03-11T21:29:42.439081+0000","last_became_active":"2024-03-11T21:28:51.357900+0000","last_became_peered":"2024-03-11T21:28:51.357900+0000","last_unstale":"2024-03-11T21:29:42.439081+0000","last_undegraded":"2024-03-11T21:29:42.439081+0000","last_fullsized":"2024-03-11T21:29:42.439081+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.015024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.671592+0000","last_change":"2024-03-11T21:29:00.537875+0000","last_active":"2024-03-11T21:29:42.671592+0000","last_peered":"2024-03-11T21:29:42.671592+0000","last_clean":"2024-03-11T21:29:42.671592+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:29:42.671592+0000","last_undegraded":"2024-03-11T21:29:42.671592+0000","last_fullsized":"2024-03-11T21:29:42.671592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:29:00.537639+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.827411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.682455+0000","last_change":"2024-03-11T21:28:44.468282+0000","last_active":"2024-03-11T21:29:42.682455+0000","last_peered":"2024-03-11T21:29:42.682455+0000","last_clean":"2024-03-11T21:29:42.682455+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:29:42.682455+0000","last_undegraded":"2024-03-11T21:29:42.682455+0000","last_fullsized":"2024-03-11T21:29:42.682455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:44.468166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:28:44.468166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.911914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":338,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.775105+0000","last_change":"2024-03-11T21:29:14.510232+0000","last_active":"2024-03-11T21:29:42.775105+0000","last_peered":"2024-03-11T21:29:42.775105+0000","last_clean":"2024-03-11T21:29:42.775105+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:29:42.775105+0000","last_undegraded":"2024-03-11T21:29:42.775105+0000","last_fullsized":"2024-03-11T21:29:42.775105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:29:14.510132+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.510132+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:27.070766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441487+0000","last_change":"2024-03-11T21:28:32.158975+0000","last_active":"2024-03-11T21:29:42.441487+0000","last_peered":"2024-03-11T21:29:42.441487+0000","last_clean":"2024-03-11T21:29:42.441487+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:29:42.441487+0000","last_undegraded":"2024-03-11T21:29:42.441487+0000","last_fullsized":"2024-03-11T21:29:42.441487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.158822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.158822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.046668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"144'5","reported_seq":103,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448348+0000","last_change":"2024-03-11T21:28:51.441861+0000","last_active":"2024-03-11T21:29:42.448348+0000","last_peered":"2024-03-11T21:29:42.448348+0000","last_clean":"2024-03-11T21:29:42.448348+0000","last_became_active":"2024-03-11T21:28:51.441360+0000","last_became_peered":"2024-03-11T21:28:51.441360+0000","last_unstale":"2024-03-11T21:29:42.448348+0000","last_undegraded":"2024-03-11T21:29:42.448348+0000","last_fullsized":"2024-03-11T21:29:42.448348+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.089552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1a","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443715+0000","last_change":"2024-03-11T21:28:51.370386+0000","last_active":"2024-03-11T21:29:42.443715+0000","last_peered":"2024-03-11T21:29:42.443715+0000","last_clean":"2024-03-11T21:29:42.443715+0000","last_became_active":"2024-03-11T21:28:51.365521+0000","last_became_peered":"2024-03-11T21:28:51.365521+0000","last_unstale":"2024-03-11T21:29:42.443715+0000","last_undegraded":"2024-03-11T21:29:42.443715+0000","last_fullsized":"2024-03-11T21:29:42.443715+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.366074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.560536+0000","last_change":"2024-03-11T21:28:51.408986+0000","last_active":"2024-03-11T21:29:42.560536+0000","last_peered":"2024-03-11T21:29:42.560536+0000","last_clean":"2024-03-11T21:29:42.560536+0000","last_became_active":"2024-03-11T21:28:51.408337+0000","last_became_peered":"2024-03-11T21:28:51.408337+0000","last_unstale":"2024-03-11T21:29:42.560536+0000","last_undegraded":"2024-03-11T21:29:42.560536+0000","last_fullsized":"2024-03-11T21:29:42.560536+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.698511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462761+0000","last_change":"2024-03-11T21:28:43.364473+0000","last_active":"2024-03-11T21:29:42.462761+0000","last_peered":"2024-03-11T21:29:42.462761+0000","last_clean":"2024-03-11T21:29:42.462761+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:29:42.462761+0000","last_undegraded":"2024-03-11T21:29:42.462761+0000","last_fullsized":"2024-03-11T21:29:42.462761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:28:43.364223+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:28:43.364223+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.597237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.450444+0000","last_change":"2024-03-11T21:28:34.871679+0000","last_active":"2024-03-11T21:29:42.450444+0000","last_peered":"2024-03-11T21:29:42.450444+0000","last_clean":"2024-03-11T21:29:42.450444+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:29:42.450444+0000","last_undegraded":"2024-03-11T21:29:42.450444+0000","last_fullsized":"2024-03-11T21:29:42.450444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.871457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.871457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.425614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.761061+0000","last_change":"2024-03-11T21:27:19.230032+0000","last_active":"2024-03-11T21:29:42.761061+0000","last_peered":"2024-03-11T21:29:42.761061+0000","last_clean":"2024-03-11T21:29:42.761061+0000","last_became_active":"2024-03-11T21:23:07.072078+0000","last_became_peered":"2024-03-11T21:23:07.072078+0000","last_unstale":"2024-03-11T21:29:42.761061+0000","last_undegraded":"2024-03-11T21:29:42.761061+0000","last_fullsized":"2024-03-11T21:29:42.761061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:19.229916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.229916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.672850+0000","last_change":"2024-03-11T21:28:35.490060+0000","last_active":"2024-03-11T21:29:42.672850+0000","last_peered":"2024-03-11T21:29:42.672850+0000","last_clean":"2024-03-11T21:29:42.672850+0000","last_became_active":"2024-03-11T21:23:09.159264+0000","last_became_peered":"2024-03-11T21:23:09.159264+0000","last_unstale":"2024-03-11T21:29:42.672850+0000","last_undegraded":"2024-03-11T21:29:42.672850+0000","last_fullsized":"2024-03-11T21:29:42.672850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.489814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.489814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.673585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458396+0000","last_change":"2024-03-11T21:28:51.343855+0000","last_active":"2024-03-11T21:29:42.458396+0000","last_peered":"2024-03-11T21:29:42.458396+0000","last_clean":"2024-03-11T21:29:42.458396+0000","last_became_active":"2024-03-11T21:28:51.340081+0000","last_became_peered":"2024-03-11T21:28:51.340081+0000","last_unstale":"2024-03-11T21:29:42.458396+0000","last_undegraded":"2024-03-11T21:29:42.458396+0000","last_fullsized":"2024-03-11T21:29:42.458396+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.607166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1b","version":"142'10","reported_seq":103,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452612+0000","last_change":"2024-03-11T21:28:51.396347+0000","last_active":"2024-03-11T21:29:42.452612+0000","last_peered":"2024-03-11T21:29:42.452612+0000","last_clean":"2024-03-11T21:29:42.452612+0000","last_became_active":"2024-03-11T21:28:51.396052+0000","last_became_peered":"2024-03-11T21:28:51.396052+0000","last_unstale":"2024-03-11T21:29:42.452612+0000","last_undegraded":"2024-03-11T21:29:42.452612+0000","last_fullsized":"2024-03-11T21:29:42.452612+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.332183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684013+0000","last_change":"2024-03-11T21:28:51.397033+0000","last_active":"2024-03-11T21:29:42.684013+0000","last_peered":"2024-03-11T21:29:42.684013+0000","last_clean":"2024-03-11T21:29:42.684013+0000","last_became_active":"2024-03-11T21:28:51.388727+0000","last_became_peered":"2024-03-11T21:28:51.388727+0000","last_unstale":"2024-03-11T21:29:42.684013+0000","last_undegraded":"2024-03-11T21:29:42.684013+0000","last_fullsized":"2024-03-11T21:29:42.684013+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.023029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":266,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452347+0000","last_change":"2024-03-11T21:28:30.134877+0000","last_active":"2024-03-11T21:29:42.452347+0000","last_peered":"2024-03-11T21:29:42.452347+0000","last_clean":"2024-03-11T21:29:42.452347+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:29:42.452347+0000","last_undegraded":"2024-03-11T21:29:42.452347+0000","last_fullsized":"2024-03-11T21:29:42.452347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:28:30.134732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.134732+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.046687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.558802+0000","last_change":"2024-03-11T21:29:11.269321+0000","last_active":"2024-03-11T21:29:42.558802+0000","last_peered":"2024-03-11T21:29:42.558802+0000","last_clean":"2024-03-11T21:29:42.558802+0000","last_became_active":"2024-03-11T21:23:02.933501+0000","last_became_peered":"2024-03-11T21:23:02.933501+0000","last_unstale":"2024-03-11T21:29:42.558802+0000","last_undegraded":"2024-03-11T21:29:42.558802+0000","last_fullsized":"2024-03-11T21:29:42.558802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.269220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.269220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:39.924405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554636+0000","last_change":"2024-03-11T21:28:59.189396+0000","last_active":"2024-03-11T21:29:42.554636+0000","last_peered":"2024-03-11T21:29:42.554636+0000","last_clean":"2024-03-11T21:29:42.554636+0000","last_became_active":"2024-03-11T21:23:07.082887+0000","last_became_peered":"2024-03-11T21:23:07.082887+0000","last_unstale":"2024-03-11T21:29:42.554636+0000","last_undegraded":"2024-03-11T21:29:42.554636+0000","last_fullsized":"2024-03-11T21:29:42.554636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:59.189248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.189248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:19.554879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.456749+0000","last_change":"2024-03-11T21:27:03.215324+0000","last_active":"2024-03-11T21:29:42.456749+0000","last_peered":"2024-03-11T21:29:42.456749+0000","last_clean":"2024-03-11T21:29:42.456749+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:29:42.456749+0000","last_undegraded":"2024-03-11T21:29:42.456749+0000","last_fullsized":"2024-03-11T21:29:42.456749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.215038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"144'3","reported_seq":97,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554579+0000","last_change":"2024-03-11T21:28:51.246583+0000","last_active":"2024-03-11T21:29:42.554579+0000","last_peered":"2024-03-11T21:29:42.554579+0000","last_clean":"2024-03-11T21:29:42.554579+0000","last_became_active":"2024-03-11T21:28:51.246300+0000","last_became_peered":"2024-03-11T21:28:51.246300+0000","last_unstale":"2024-03-11T21:29:42.554579+0000","last_undegraded":"2024-03-11T21:29:42.554579+0000","last_fullsized":"2024-03-11T21:29:42.554579+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.850886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2807,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.18","version":"142'2","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453676+0000","last_change":"2024-03-11T21:28:51.312646+0000","last_active":"2024-03-11T21:29:42.453676+0000","last_peered":"2024-03-11T21:29:42.453676+0000","last_clean":"2024-03-11T21:29:42.453676+0000","last_became_active":"2024-03-11T21:28:51.309027+0000","last_became_peered":"2024-03-11T21:28:51.309027+0000","last_unstale":"2024-03-11T21:29:42.453676+0000","last_undegraded":"2024-03-11T21:29:42.453676+0000","last_fullsized":"2024-03-11T21:29:42.453676+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.133262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.681436+0000","last_change":"2024-03-11T21:28:51.389398+0000","last_active":"2024-03-11T21:29:42.681436+0000","last_peered":"2024-03-11T21:29:42.681436+0000","last_clean":"2024-03-11T21:29:42.681436+0000","last_became_active":"2024-03-11T21:28:51.388792+0000","last_became_peered":"2024-03-11T21:28:51.388792+0000","last_unstale":"2024-03-11T21:29:42.681436+0000","last_undegraded":"2024-03-11T21:29:42.681436+0000","last_fullsized":"2024-03-11T21:29:42.681436+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:07.091291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.690208+0000","last_change":"2024-03-11T21:28:34.442997+0000","last_active":"2024-03-11T21:29:42.690208+0000","last_peered":"2024-03-11T21:29:42.690208+0000","last_clean":"2024-03-11T21:29:42.690208+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:29:42.690208+0000","last_undegraded":"2024-03-11T21:29:42.690208+0000","last_fullsized":"2024-03-11T21:29:42.690208+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:28:34.442896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.442896+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449371+0000","last_change":"2024-03-11T21:29:10.980732+0000","last_active":"2024-03-11T21:29:42.449371+0000","last_peered":"2024-03-11T21:29:42.449371+0000","last_clean":"2024-03-11T21:29:42.449371+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:29:42.449371+0000","last_undegraded":"2024-03-11T21:29:42.449371+0000","last_fullsized":"2024-03-11T21:29:42.449371+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:10.980632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:10.980632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:28.323853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":246,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448393+0000","last_change":"2024-03-11T21:28:29.012798+0000","last_active":"2024-03-11T21:29:42.448393+0000","last_peered":"2024-03-11T21:29:42.448393+0000","last_clean":"2024-03-11T21:29:42.448393+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:29:42.448393+0000","last_undegraded":"2024-03-11T21:29:42.448393+0000","last_fullsized":"2024-03-11T21:29:42.448393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.012647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.012647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.899127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466628+0000","last_change":"2024-03-11T21:29:11.346981+0000","last_active":"2024-03-11T21:29:42.466628+0000","last_peered":"2024-03-11T21:29:42.466628+0000","last_clean":"2024-03-11T21:29:42.466628+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:29:42.466628+0000","last_undegraded":"2024-03-11T21:29:42.466628+0000","last_fullsized":"2024-03-11T21:29:42.466628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.346840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.346840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:40.102939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.457291+0000","last_change":"2024-03-11T21:28:49.936115+0000","last_active":"2024-03-11T21:29:42.457291+0000","last_peered":"2024-03-11T21:29:42.457291+0000","last_clean":"2024-03-11T21:29:42.457291+0000","last_became_active":"2024-03-11T21:28:49.935829+0000","last_became_peered":"2024-03-11T21:28:49.935829+0000","last_unstale":"2024-03-11T21:29:42.457291+0000","last_undegraded":"2024-03-11T21:29:42.457291+0000","last_fullsized":"2024-03-11T21:29:42.457291+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.741426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.673325+0000","last_change":"2024-03-11T21:28:51.400497+0000","last_active":"2024-03-11T21:29:42.673325+0000","last_peered":"2024-03-11T21:29:42.673325+0000","last_clean":"2024-03-11T21:29:42.673325+0000","last_became_active":"2024-03-11T21:28:51.395884+0000","last_became_peered":"2024-03-11T21:28:51.395884+0000","last_unstale":"2024-03-11T21:29:42.673325+0000","last_undegraded":"2024-03-11T21:29:42.673325+0000","last_fullsized":"2024-03-11T21:29:42.673325+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688194+0000","last_change":"2024-03-11T21:28:51.270988+0000","last_active":"2024-03-11T21:29:42.688194+0000","last_peered":"2024-03-11T21:29:42.688194+0000","last_clean":"2024-03-11T21:29:42.688194+0000","last_became_active":"2024-03-11T21:28:51.270175+0000","last_became_peered":"2024-03-11T21:28:51.270175+0000","last_unstale":"2024-03-11T21:29:42.688194+0000","last_undegraded":"2024-03-11T21:29:42.688194+0000","last_fullsized":"2024-03-11T21:29:42.688194+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.993043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448853+0000","last_change":"2024-03-11T21:29:13.082319+0000","last_active":"2024-03-11T21:29:42.448853+0000","last_peered":"2024-03-11T21:29:42.448853+0000","last_clean":"2024-03-11T21:29:42.448853+0000","last_became_active":"2024-03-11T21:23:04.934431+0000","last_became_peered":"2024-03-11T21:23:04.934431+0000","last_unstale":"2024-03-11T21:29:42.448853+0000","last_undegraded":"2024-03-11T21:29:42.448853+0000","last_fullsized":"2024-03-11T21:29:42.448853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:29:13.081725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.081725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.481460+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454673+0000","last_change":"2024-03-11T21:28:22.986925+0000","last_active":"2024-03-11T21:29:42.454673+0000","last_peered":"2024-03-11T21:29:42.454673+0000","last_clean":"2024-03-11T21:29:42.454673+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:29:42.454673+0000","last_undegraded":"2024-03-11T21:29:42.454673+0000","last_fullsized":"2024-03-11T21:29:42.454673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:22.986791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:28:22.986791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.700081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452995+0000","last_change":"2024-03-11T21:28:32.238983+0000","last_active":"2024-03-11T21:29:42.452995+0000","last_peered":"2024-03-11T21:29:42.452995+0000","last_clean":"2024-03-11T21:29:42.452995+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:29:42.452995+0000","last_undegraded":"2024-03-11T21:29:42.452995+0000","last_fullsized":"2024-03-11T21:29:42.452995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.238875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.238875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.404261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.761709+0000","last_change":"2024-03-11T21:28:42.318986+0000","last_active":"2024-03-11T21:29:42.761709+0000","last_peered":"2024-03-11T21:29:42.761709+0000","last_clean":"2024-03-11T21:29:42.761709+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:29:42.761709+0000","last_undegraded":"2024-03-11T21:29:42.761709+0000","last_fullsized":"2024-03-11T21:29:42.761709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:42.318908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:28:42.318908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.936241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452895+0000","last_change":"2024-03-11T21:28:51.402121+0000","last_active":"2024-03-11T21:29:42.452895+0000","last_peered":"2024-03-11T21:29:42.452895+0000","last_clean":"2024-03-11T21:29:42.452895+0000","last_became_active":"2024-03-11T21:28:51.400779+0000","last_became_peered":"2024-03-11T21:28:51.400779+0000","last_unstale":"2024-03-11T21:29:42.452895+0000","last_undegraded":"2024-03-11T21:29:42.452895+0000","last_fullsized":"2024-03-11T21:29:42.452895+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.543655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.448508+0000","last_change":"2024-03-11T21:28:51.378811+0000","last_active":"2024-03-11T21:29:42.448508+0000","last_peered":"2024-03-11T21:29:42.448508+0000","last_clean":"2024-03-11T21:29:42.448508+0000","last_became_active":"2024-03-11T21:28:51.377270+0000","last_became_peered":"2024-03-11T21:28:51.377270+0000","last_unstale":"2024-03-11T21:29:42.448508+0000","last_undegraded":"2024-03-11T21:29:42.448508+0000","last_fullsized":"2024-03-11T21:29:42.448508+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.375381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462335+0000","last_change":"2024-03-11T21:28:51.424576+0000","last_active":"2024-03-11T21:29:42.462335+0000","last_peered":"2024-03-11T21:29:42.462335+0000","last_clean":"2024-03-11T21:29:42.462335+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:29:42.462335+0000","last_undegraded":"2024-03-11T21:29:42.462335+0000","last_fullsized":"2024-03-11T21:29:42.462335+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.341722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552443+0000","last_change":"2024-03-11T21:29:20.448262+0000","last_active":"2024-03-11T21:29:42.552443+0000","last_peered":"2024-03-11T21:29:42.552443+0000","last_clean":"2024-03-11T21:29:42.552443+0000","last_became_active":"2024-03-11T21:23:04.950793+0000","last_became_peered":"2024-03-11T21:23:04.950793+0000","last_unstale":"2024-03-11T21:29:42.552443+0000","last_undegraded":"2024-03-11T21:29:42.552443+0000","last_fullsized":"2024-03-11T21:29:42.552443+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:20.448110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.448110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:48.364973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444131+0000","last_change":"2024-03-11T21:27:29.967422+0000","last_active":"2024-03-11T21:29:42.444131+0000","last_peered":"2024-03-11T21:29:42.444131+0000","last_clean":"2024-03-11T21:29:42.444131+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:29:42.444131+0000","last_undegraded":"2024-03-11T21:29:42.444131+0000","last_fullsized":"2024-03-11T21:29:42.444131+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.967200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.967200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":340,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454773+0000","last_change":"2024-03-11T21:28:41.813260+0000","last_active":"2024-03-11T21:29:42.454773+0000","last_peered":"2024-03-11T21:29:42.454773+0000","last_clean":"2024-03-11T21:29:42.454773+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:29:42.454773+0000","last_undegraded":"2024-03-11T21:29:42.454773+0000","last_fullsized":"2024-03-11T21:29:42.454773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:41.812918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.812918+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.681111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":237,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.469229+0000","last_change":"2024-03-11T21:28:23.129847+0000","last_active":"2024-03-11T21:29:42.469229+0000","last_peered":"2024-03-11T21:29:42.469229+0000","last_clean":"2024-03-11T21:29:42.469229+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:29:42.469229+0000","last_undegraded":"2024-03-11T21:29:42.469229+0000","last_fullsized":"2024-03-11T21:29:42.469229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:28:23.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.129604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.254992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452574+0000","last_change":"2024-03-11T21:28:51.341703+0000","last_active":"2024-03-11T21:29:42.452574+0000","last_peered":"2024-03-11T21:29:42.452574+0000","last_clean":"2024-03-11T21:29:42.452574+0000","last_became_active":"2024-03-11T21:28:51.340317+0000","last_became_peered":"2024-03-11T21:28:51.340317+0000","last_unstale":"2024-03-11T21:29:42.452574+0000","last_undegraded":"2024-03-11T21:29:42.452574+0000","last_fullsized":"2024-03-11T21:29:42.452574+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.936235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.17","version":"142'5","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454828+0000","last_change":"2024-03-11T21:28:51.400711+0000","last_active":"2024-03-11T21:29:42.454828+0000","last_peered":"2024-03-11T21:29:42.454828+0000","last_clean":"2024-03-11T21:29:42.454828+0000","last_became_active":"2024-03-11T21:28:51.400553+0000","last_became_peered":"2024-03-11T21:28:51.400553+0000","last_unstale":"2024-03-11T21:29:42.454828+0000","last_undegraded":"2024-03-11T21:29:42.454828+0000","last_fullsized":"2024-03-11T21:29:42.454828+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.515489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441252+0000","last_change":"2024-03-11T21:28:51.338396+0000","last_active":"2024-03-11T21:29:42.441252+0000","last_peered":"2024-03-11T21:29:42.441252+0000","last_clean":"2024-03-11T21:29:42.441252+0000","last_became_active":"2024-03-11T21:28:51.337261+0000","last_became_peered":"2024-03-11T21:28:51.337261+0000","last_unstale":"2024-03-11T21:29:42.441252+0000","last_undegraded":"2024-03-11T21:29:42.441252+0000","last_fullsized":"2024-03-11T21:29:42.441252+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.850018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.449599+0000","last_change":"2024-03-11T21:28:40.302063+0000","last_active":"2024-03-11T21:29:42.449599+0000","last_peered":"2024-03-11T21:29:42.449599+0000","last_clean":"2024-03-11T21:29:42.449599+0000","last_became_active":"2024-03-11T21:23:04.950912+0000","last_became_peered":"2024-03-11T21:23:04.950912+0000","last_unstale":"2024-03-11T21:29:42.449599+0000","last_undegraded":"2024-03-11T21:29:42.449599+0000","last_fullsized":"2024-03-11T21:29:42.449599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:28:40.301964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:40.301964+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.276356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.456838+0000","last_change":"2024-03-11T21:28:48.305255+0000","last_active":"2024-03-11T21:29:42.456838+0000","last_peered":"2024-03-11T21:29:42.456838+0000","last_clean":"2024-03-11T21:29:42.456838+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:29:42.456838+0000","last_undegraded":"2024-03-11T21:29:42.456838+0000","last_fullsized":"2024-03-11T21:29:42.456838+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:28:48.305156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.305156+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.786717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.684318+0000","last_change":"2024-03-11T21:28:48.416010+0000","last_active":"2024-03-11T21:29:42.684318+0000","last_peered":"2024-03-11T21:29:42.684318+0000","last_clean":"2024-03-11T21:29:42.684318+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:29:42.684318+0000","last_undegraded":"2024-03-11T21:29:42.684318+0000","last_fullsized":"2024-03-11T21:29:42.684318+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.415892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.415892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:12.444012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559268+0000","last_change":"2024-03-11T21:28:56.181217+0000","last_active":"2024-03-11T21:29:42.559268+0000","last_peered":"2024-03-11T21:29:42.559268+0000","last_clean":"2024-03-11T21:29:42.559268+0000","last_became_active":"2024-03-11T21:23:09.152647+0000","last_became_peered":"2024-03-11T21:23:09.152647+0000","last_unstale":"2024-03-11T21:29:42.559268+0000","last_undegraded":"2024-03-11T21:29:42.559268+0000","last_fullsized":"2024-03-11T21:29:42.559268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:56.181111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:56.181111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.639235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.767028+0000","last_change":"2024-03-11T21:28:49.946437+0000","last_active":"2024-03-11T21:29:42.767028+0000","last_peered":"2024-03-11T21:29:42.767028+0000","last_clean":"2024-03-11T21:29:42.767028+0000","last_became_active":"2024-03-11T21:28:49.945367+0000","last_became_peered":"2024-03-11T21:28:49.945367+0000","last_unstale":"2024-03-11T21:29:42.767028+0000","last_undegraded":"2024-03-11T21:29:42.767028+0000","last_fullsized":"2024-03-11T21:29:42.767028+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.266111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"142'2","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452185+0000","last_change":"2024-03-11T21:28:51.412031+0000","last_active":"2024-03-11T21:29:42.452185+0000","last_peered":"2024-03-11T21:29:42.452185+0000","last_clean":"2024-03-11T21:29:42.452185+0000","last_became_active":"2024-03-11T21:28:51.411309+0000","last_became_peered":"2024-03-11T21:28:51.411309+0000","last_unstale":"2024-03-11T21:29:42.452185+0000","last_undegraded":"2024-03-11T21:29:42.452185+0000","last_fullsized":"2024-03-11T21:29:42.452185+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.116001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.468969+0000","last_change":"2024-03-11T21:28:51.409644+0000","last_active":"2024-03-11T21:29:42.468969+0000","last_peered":"2024-03-11T21:29:42.468969+0000","last_clean":"2024-03-11T21:29:42.468969+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:29:42.468969+0000","last_undegraded":"2024-03-11T21:29:42.468969+0000","last_fullsized":"2024-03-11T21:29:42.468969+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:58.016220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.682688+0000","last_change":"2024-03-11T21:28:31.452297+0000","last_active":"2024-03-11T21:29:42.682688+0000","last_peered":"2024-03-11T21:29:42.682688+0000","last_clean":"2024-03-11T21:29:42.682688+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:29:42.682688+0000","last_undegraded":"2024-03-11T21:29:42.682688+0000","last_fullsized":"2024-03-11T21:29:42.682688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:28:31.452044+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:28:31.452044+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443544+0000","last_change":"2024-03-11T21:28:44.892608+0000","last_active":"2024-03-11T21:29:42.443544+0000","last_peered":"2024-03-11T21:29:42.443544+0000","last_clean":"2024-03-11T21:29:42.443544+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:29:42.443544+0000","last_undegraded":"2024-03-11T21:29:42.443544+0000","last_fullsized":"2024-03-11T21:29:42.443544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:44.892265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:28:44.892265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.492754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.559107+0000","last_change":"2024-03-11T21:28:36.310920+0000","last_active":"2024-03-11T21:29:42.559107+0000","last_peered":"2024-03-11T21:29:42.559107+0000","last_clean":"2024-03-11T21:29:42.559107+0000","last_became_active":"2024-03-11T21:23:07.098776+0000","last_became_peered":"2024-03-11T21:23:07.098776+0000","last_unstale":"2024-03-11T21:29:42.559107+0000","last_undegraded":"2024-03-11T21:29:42.559107+0000","last_fullsized":"2024-03-11T21:29:42.559107+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:36.310840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.310840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.124080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":244,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.447349+0000","last_change":"2024-03-11T21:28:35.101814+0000","last_active":"2024-03-11T21:29:42.447349+0000","last_peered":"2024-03-11T21:29:42.447349+0000","last_clean":"2024-03-11T21:29:42.447349+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:29:42.447349+0000","last_undegraded":"2024-03-11T21:29:42.447349+0000","last_fullsized":"2024-03-11T21:29:42.447349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.101679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.101679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.552026+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"167'1","reported_seq":94,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453153+0000","last_change":"2024-03-11T21:28:51.297639+0000","last_active":"2024-03-11T21:29:42.453153+0000","last_peered":"2024-03-11T21:29:42.453153+0000","last_clean":"2024-03-11T21:29:42.453153+0000","last_became_active":"2024-03-11T21:28:51.295977+0000","last_became_peered":"2024-03-11T21:28:51.295977+0000","last_unstale":"2024-03-11T21:29:42.453153+0000","last_undegraded":"2024-03-11T21:29:42.453153+0000","last_fullsized":"2024-03-11T21:29:42.453153+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.481734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.454821+0000","last_change":"2024-03-11T21:28:51.444581+0000","last_active":"2024-03-11T21:29:42.454821+0000","last_peered":"2024-03-11T21:29:42.454821+0000","last_clean":"2024-03-11T21:29:42.454821+0000","last_became_active":"2024-03-11T21:28:51.444296+0000","last_became_peered":"2024-03-11T21:28:51.444296+0000","last_unstale":"2024-03-11T21:29:42.454821+0000","last_undegraded":"2024-03-11T21:29:42.454821+0000","last_fullsized":"2024-03-11T21:29:42.454821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:52.749849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.434860+0000","last_change":"2024-03-11T21:28:51.388343+0000","last_active":"2024-03-11T21:29:42.434860+0000","last_peered":"2024-03-11T21:29:42.434860+0000","last_clean":"2024-03-11T21:29:42.434860+0000","last_became_active":"2024-03-11T21:28:51.388058+0000","last_became_peered":"2024-03-11T21:28:51.388058+0000","last_unstale":"2024-03-11T21:29:42.434860+0000","last_undegraded":"2024-03-11T21:29:42.434860+0000","last_fullsized":"2024-03-11T21:29:42.434860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.844341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444416+0000","last_change":"2024-03-11T21:28:26.025111+0000","last_active":"2024-03-11T21:29:42.444416+0000","last_peered":"2024-03-11T21:29:42.444416+0000","last_clean":"2024-03-11T21:29:42.444416+0000","last_became_active":"2024-03-11T21:23:05.020296+0000","last_became_peered":"2024-03-11T21:23:05.020296+0000","last_unstale":"2024-03-11T21:29:42.444416+0000","last_undegraded":"2024-03-11T21:29:42.444416+0000","last_fullsized":"2024-03-11T21:29:42.444416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:28:26.024992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.024992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.046629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458856+0000","last_change":"2024-03-11T21:28:59.346617+0000","last_active":"2024-03-11T21:29:42.458856+0000","last_peered":"2024-03-11T21:29:42.458856+0000","last_clean":"2024-03-11T21:29:42.458856+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:29:42.458856+0000","last_undegraded":"2024-03-11T21:29:42.458856+0000","last_fullsized":"2024-03-11T21:29:42.458856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:59.346500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.346500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.076449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452456+0000","last_change":"2024-03-11T21:28:57.075890+0000","last_active":"2024-03-11T21:29:42.452456+0000","last_peered":"2024-03-11T21:29:42.452456+0000","last_clean":"2024-03-11T21:29:42.452456+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:29:42.452456+0000","last_undegraded":"2024-03-11T21:29:42.452456+0000","last_fullsized":"2024-03-11T21:29:42.452456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:57.075783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:28:57.075783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:04.498717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":236,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.777255+0000","last_change":"2024-03-11T21:28:29.240899+0000","last_active":"2024-03-11T21:29:42.777255+0000","last_peered":"2024-03-11T21:29:42.777255+0000","last_clean":"2024-03-11T21:29:42.777255+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:29:42.777255+0000","last_undegraded":"2024-03-11T21:29:42.777255+0000","last_fullsized":"2024-03-11T21:29:42.777255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.240776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.240776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.300625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.688331+0000","last_change":"2024-03-11T21:28:51.429299+0000","last_active":"2024-03-11T21:29:42.688331+0000","last_peered":"2024-03-11T21:29:42.688331+0000","last_clean":"2024-03-11T21:29:42.688331+0000","last_became_active":"2024-03-11T21:28:51.428710+0000","last_became_peered":"2024-03-11T21:28:51.428710+0000","last_unstale":"2024-03-11T21:29:42.688331+0000","last_undegraded":"2024-03-11T21:29:42.688331+0000","last_fullsized":"2024-03-11T21:29:42.688331+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.599859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":95,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.772966+0000","last_change":"2024-03-11T21:28:49.958023+0000","last_active":"2024-03-11T21:29:42.772966+0000","last_peered":"2024-03-11T21:29:42.772966+0000","last_clean":"2024-03-11T21:29:42.772966+0000","last_became_active":"2024-03-11T21:28:49.955021+0000","last_became_peered":"2024-03-11T21:28:49.955021+0000","last_unstale":"2024-03-11T21:29:42.772966+0000","last_undegraded":"2024-03-11T21:29:42.772966+0000","last_fullsized":"2024-03-11T21:29:42.772966+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.279432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.472568+0000","last_change":"2024-03-11T21:28:51.472372+0000","last_active":"2024-03-11T21:29:42.472568+0000","last_peered":"2024-03-11T21:29:42.472568+0000","last_clean":"2024-03-11T21:29:42.472568+0000","last_became_active":"2024-03-11T21:28:51.471787+0000","last_became_peered":"2024-03-11T21:28:51.471787+0000","last_unstale":"2024-03-11T21:29:42.472568+0000","last_undegraded":"2024-03-11T21:29:42.472568+0000","last_fullsized":"2024-03-11T21:29:42.472568+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.503703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":274,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.445254+0000","last_change":"2024-03-11T21:29:15.121395+0000","last_active":"2024-03-11T21:29:42.445254+0000","last_peered":"2024-03-11T21:29:42.445254+0000","last_clean":"2024-03-11T21:29:42.445254+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:29:42.445254+0000","last_undegraded":"2024-03-11T21:29:42.445254+0000","last_fullsized":"2024-03-11T21:29:42.445254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:29:15.121067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:15.121067+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:34.513372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.468980+0000","last_change":"2024-03-11T21:29:07.335102+0000","last_active":"2024-03-11T21:29:42.468980+0000","last_peered":"2024-03-11T21:29:42.468980+0000","last_clean":"2024-03-11T21:29:42.468980+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:29:42.468980+0000","last_undegraded":"2024-03-11T21:29:42.468980+0000","last_fullsized":"2024-03-11T21:29:42.468980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:07.334888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:07.334888+0000","last_clean_scrub_stamp":"2024-03-11T21:29:07.334888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:08.312753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452230+0000","last_change":"2024-03-11T21:28:24.101901+0000","last_active":"2024-03-11T21:29:42.452230+0000","last_peered":"2024-03-11T21:29:42.452230+0000","last_clean":"2024-03-11T21:29:42.452230+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:29:42.452230+0000","last_undegraded":"2024-03-11T21:29:42.452230+0000","last_fullsized":"2024-03-11T21:29:42.452230+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:24.101784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.101784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.673293+0000","last_change":"2024-03-11T21:28:39.513784+0000","last_active":"2024-03-11T21:29:42.673293+0000","last_peered":"2024-03-11T21:29:42.673293+0000","last_clean":"2024-03-11T21:29:42.673293+0000","last_became_active":"2024-03-11T21:23:09.156753+0000","last_became_peered":"2024-03-11T21:23:09.156753+0000","last_unstale":"2024-03-11T21:29:42.673293+0000","last_undegraded":"2024-03-11T21:29:42.673293+0000","last_fullsized":"2024-03-11T21:29:42.673293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:39.513463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:39.513463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.893190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"167'2","reported_seq":98,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462783+0000","last_change":"2024-03-11T21:28:51.379640+0000","last_active":"2024-03-11T21:29:42.462783+0000","last_peered":"2024-03-11T21:29:42.462783+0000","last_clean":"2024-03-11T21:29:42.462783+0000","last_became_active":"2024-03-11T21:28:51.379326+0000","last_became_peered":"2024-03-11T21:28:51.379326+0000","last_unstale":"2024-03-11T21:29:42.462783+0000","last_undegraded":"2024-03-11T21:29:42.462783+0000","last_fullsized":"2024-03-11T21:29:42.462783+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.910274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165835,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.13","version":"142'9","reported_seq":102,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552424+0000","last_change":"2024-03-11T21:28:51.474798+0000","last_active":"2024-03-11T21:29:42.552424+0000","last_peered":"2024-03-11T21:29:42.552424+0000","last_clean":"2024-03-11T21:29:42.552424+0000","last_became_active":"2024-03-11T21:28:51.443928+0000","last_became_peered":"2024-03-11T21:28:51.443928+0000","last_unstale":"2024-03-11T21:29:42.552424+0000","last_undegraded":"2024-03-11T21:29:42.552424+0000","last_fullsized":"2024-03-11T21:29:42.552424+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:06.169475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.443618+0000","last_change":"2024-03-11T21:28:51.308794+0000","last_active":"2024-03-11T21:29:42.443618+0000","last_peered":"2024-03-11T21:29:42.443618+0000","last_clean":"2024-03-11T21:29:42.443618+0000","last_became_active":"2024-03-11T21:28:51.308503+0000","last_became_peered":"2024-03-11T21:28:51.308503+0000","last_unstale":"2024-03-11T21:29:42.443618+0000","last_undegraded":"2024-03-11T21:29:42.443618+0000","last_fullsized":"2024-03-11T21:29:42.443618+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.274795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.452656+0000","last_change":"2024-03-11T21:28:41.972620+0000","last_active":"2024-03-11T21:29:42.452656+0000","last_peered":"2024-03-11T21:29:42.452656+0000","last_clean":"2024-03-11T21:29:42.452656+0000","last_became_active":"2024-03-11T21:23:04.933432+0000","last_became_peered":"2024-03-11T21:23:04.933432+0000","last_unstale":"2024-03-11T21:29:42.452656+0000","last_undegraded":"2024-03-11T21:29:42.452656+0000","last_fullsized":"2024-03-11T21:29:42.452656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:28:41.972502+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.972502+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.170901+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.462631+0000","last_change":"2024-03-11T21:28:30.219450+0000","last_active":"2024-03-11T21:29:42.462631+0000","last_peered":"2024-03-11T21:29:42.462631+0000","last_clean":"2024-03-11T21:29:42.462631+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:29:42.462631+0000","last_undegraded":"2024-03-11T21:29:42.462631+0000","last_fullsized":"2024-03-11T21:29:42.462631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.219153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.219153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.254956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.466513+0000","last_change":"2024-03-11T21:27:18.708546+0000","last_active":"2024-03-11T21:29:42.466513+0000","last_peered":"2024-03-11T21:29:42.466513+0000","last_clean":"2024-03-11T21:29:42.466513+0000","last_became_active":"2024-03-11T21:23:07.091106+0000","last_became_peered":"2024-03-11T21:23:07.091106+0000","last_unstale":"2024-03-11T21:29:42.466513+0000","last_undegraded":"2024-03-11T21:29:42.466513+0000","last_fullsized":"2024-03-11T21:29:42.466513+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.708450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.708450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.254892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441394+0000","last_change":"2024-03-11T21:28:26.041698+0000","last_active":"2024-03-11T21:29:42.441394+0000","last_peered":"2024-03-11T21:29:42.441394+0000","last_clean":"2024-03-11T21:29:42.441394+0000","last_became_active":"2024-03-11T21:23:09.160077+0000","last_became_peered":"2024-03-11T21:23:09.160077+0000","last_unstale":"2024-03-11T21:29:42.441394+0000","last_undegraded":"2024-03-11T21:29:42.441394+0000","last_fullsized":"2024-03-11T21:29:42.441394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.041597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.041597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.040590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.668498+0000","last_change":"2024-03-11T21:28:51.314095+0000","last_active":"2024-03-11T21:29:42.668498+0000","last_peered":"2024-03-11T21:29:42.668498+0000","last_clean":"2024-03-11T21:29:42.668498+0000","last_became_active":"2024-03-11T21:28:51.310942+0000","last_became_peered":"2024-03-11T21:28:51.310942+0000","last_unstale":"2024-03-11T21:29:42.668498+0000","last_undegraded":"2024-03-11T21:29:42.668498+0000","last_fullsized":"2024-03-11T21:29:42.668498+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:52.376062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.685862+0000","last_change":"2024-03-11T21:28:51.452334+0000","last_active":"2024-03-11T21:29:42.685862+0000","last_peered":"2024-03-11T21:29:42.685862+0000","last_clean":"2024-03-11T21:29:42.685862+0000","last_became_active":"2024-03-11T21:28:51.452176+0000","last_became_peered":"2024-03-11T21:28:51.452176+0000","last_unstale":"2024-03-11T21:29:42.685862+0000","last_undegraded":"2024-03-11T21:29:42.685862+0000","last_fullsized":"2024-03-11T21:29:42.685862+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.434727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.461994+0000","last_change":"2024-03-11T21:28:51.473064+0000","last_active":"2024-03-11T21:29:42.461994+0000","last_peered":"2024-03-11T21:29:42.461994+0000","last_clean":"2024-03-11T21:29:42.461994+0000","last_became_active":"2024-03-11T21:28:51.472075+0000","last_became_peered":"2024-03-11T21:28:51.472075+0000","last_unstale":"2024-03-11T21:29:42.461994+0000","last_undegraded":"2024-03-11T21:29:42.461994+0000","last_fullsized":"2024-03-11T21:29:42.461994+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.643572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"121'16","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.690654+0000","last_change":"2024-03-11T21:28:37.505958+0000","last_active":"2024-03-11T21:29:42.690654+0000","last_peered":"2024-03-11T21:29:42.690654+0000","last_clean":"2024-03-11T21:29:42.690654+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:29:42.690654+0000","last_undegraded":"2024-03-11T21:29:42.690654+0000","last_fullsized":"2024-03-11T21:29:42.690654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:28:37.505724+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.505724+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.954730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.767628+0000","last_change":"2024-03-11T21:27:32.238387+0000","last_active":"2024-03-11T21:29:42.767628+0000","last_peered":"2024-03-11T21:29:42.767628+0000","last_clean":"2024-03-11T21:29:42.767628+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:29:42.767628+0000","last_undegraded":"2024-03-11T21:29:42.767628+0000","last_fullsized":"2024-03-11T21:29:42.767628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.238293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.238293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.815701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552626+0000","last_change":"2024-03-11T21:28:46.237591+0000","last_active":"2024-03-11T21:29:42.552626+0000","last_peered":"2024-03-11T21:29:42.552626+0000","last_clean":"2024-03-11T21:29:42.552626+0000","last_became_active":"2024-03-11T21:23:07.081069+0000","last_became_peered":"2024-03-11T21:23:07.081069+0000","last_unstale":"2024-03-11T21:29:42.552626+0000","last_undegraded":"2024-03-11T21:29:42.552626+0000","last_fullsized":"2024-03-11T21:29:42.552626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:46.237367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:46.237367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:11.804681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.444044+0000","last_change":"2024-03-11T21:28:45.085235+0000","last_active":"2024-03-11T21:29:42.444044+0000","last_peered":"2024-03-11T21:29:42.444044+0000","last_clean":"2024-03-11T21:29:42.444044+0000","last_became_active":"2024-03-11T21:23:09.150499+0000","last_became_peered":"2024-03-11T21:23:09.150499+0000","last_unstale":"2024-03-11T21:29:42.444044+0000","last_undegraded":"2024-03-11T21:29:42.444044+0000","last_fullsized":"2024-03-11T21:29:42.444044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:45.084991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:45.084991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:03.677502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.552539+0000","last_change":"2024-03-11T21:28:51.433225+0000","last_active":"2024-03-11T21:29:42.552539+0000","last_peered":"2024-03-11T21:29:42.552539+0000","last_clean":"2024-03-11T21:29:42.552539+0000","last_became_active":"2024-03-11T21:28:51.409055+0000","last_became_peered":"2024-03-11T21:28:51.409055+0000","last_unstale":"2024-03-11T21:29:42.552539+0000","last_undegraded":"2024-03-11T21:29:42.552539+0000","last_fullsized":"2024-03-11T21:29:42.552539+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:02.287962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"142'7","reported_seq":100,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.441052+0000","last_change":"2024-03-11T21:28:51.367540+0000","last_active":"2024-03-11T21:29:42.441052+0000","last_peered":"2024-03-11T21:29:42.441052+0000","last_clean":"2024-03-11T21:29:42.441052+0000","last_became_active":"2024-03-11T21:28:51.366739+0000","last_became_peered":"2024-03-11T21:28:51.366739+0000","last_unstale":"2024-03-11T21:29:42.441052+0000","last_undegraded":"2024-03-11T21:29:42.441052+0000","last_fullsized":"2024-03-11T21:29:42.441052+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:09.947478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":93,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.554316+0000","last_change":"2024-03-11T21:28:51.337401+0000","last_active":"2024-03-11T21:29:42.554316+0000","last_peered":"2024-03-11T21:29:42.554316+0000","last_clean":"2024-03-11T21:29:42.554316+0000","last_became_active":"2024-03-11T21:28:51.313778+0000","last_became_peered":"2024-03-11T21:28:51.313778+0000","last_unstale":"2024-03-11T21:29:42.554316+0000","last_undegraded":"2024-03-11T21:29:42.554316+0000","last_fullsized":"2024-03-11T21:29:42.554316+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.771097+0000","last_change":"2024-03-11T21:29:09.487728+0000","last_active":"2024-03-11T21:29:42.771097+0000","last_peered":"2024-03-11T21:29:42.771097+0000","last_clean":"2024-03-11T21:29:42.771097+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:29:42.771097+0000","last_undegraded":"2024-03-11T21:29:42.771097+0000","last_fullsized":"2024-03-11T21:29:42.771097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:29:09.487592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:09.487592+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:20.434793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.690712+0000","last_change":"2024-03-11T21:28:25.491444+0000","last_active":"2024-03-11T21:29:42.690712+0000","last_peered":"2024-03-11T21:29:42.690712+0000","last_clean":"2024-03-11T21:29:42.690712+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:29:42.690712+0000","last_undegraded":"2024-03-11T21:29:42.690712+0000","last_fullsized":"2024-03-11T21:29:42.690712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:28:25.491271+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.491271+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.434962+0000","last_change":"2024-03-11T21:28:34.109955+0000","last_active":"2024-03-11T21:29:42.434962+0000","last_peered":"2024-03-11T21:29:42.434962+0000","last_clean":"2024-03-11T21:29:42.434962+0000","last_became_active":"2024-03-11T21:23:07.093871+0000","last_became_peered":"2024-03-11T21:23:07.093871+0000","last_unstale":"2024-03-11T21:29:42.434962+0000","last_undegraded":"2024-03-11T21:29:42.434962+0000","last_fullsized":"2024-03-11T21:29:42.434962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.109733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.109733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.524106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":235,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.458762+0000","last_change":"2024-03-11T21:28:44.319129+0000","last_active":"2024-03-11T21:29:42.458762+0000","last_peered":"2024-03-11T21:29:42.458762+0000","last_clean":"2024-03-11T21:29:42.458762+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:29:42.458762+0000","last_undegraded":"2024-03-11T21:29:42.458762+0000","last_fullsized":"2024-03-11T21:29:42.458762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:44.318914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:44.318914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:00.636095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"144'7","reported_seq":108,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-11T21:29:42.453033+0000","last_change":"2024-03-11T21:28:51.248692+0000","last_active":"2024-03-11T21:29:42.453033+0000","last_peered":"2024-03-11T21:29:42.453033+0000","last_clean":"2024-03-11T21:29:42.453033+0000","last_became_active":"2024-03-11T21:28:51.248169+0000","last_became_peered":"2024-03-11T21:28:51.248169+0000","last_unstale":"2024-03-11T21:29:42.453033+0000","last_undegraded":"2024-03-11T21:29:42.453033+0000","last_fullsized":"2024-03-11T21:29:42.453033+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:28:41.604986+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.814494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165850,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":142,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":141,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":140,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":139,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":138,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":137,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":136,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":60,"acting":60,"num_store_stats":5},{"poolid":135,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":75,"acting":75,"num_store_stats":5},{"poolid":99,"num_pg":32,"stat_sum":{"num_bytes":1050240,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":23,"num_write_kb":1045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3244032,"data_stored":3150720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":96,"acting":96,"num_store_stats":8},{"poolid":69,"num_pg":32,"stat_sum":{"num_bytes":651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679809024,"data_stored":679497786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":462,"ondisk_log_size":462,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841627,"num_pgs":191,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129988,"kb_used_data":76604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93615164,"statfs":{"total":95995035648,"available":95861927936,"internally_reserved":0,"allocated":78442496,"data_stored":77425263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168676,"num_pgs":159,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132376,"kb_used_data":101200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93612776,"statfs":{"total":95995035648,"available":95859482624,"internally_reserved":0,"allocated":103628800,"data_stored":102590371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":235,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82672,"kb_used_data":51488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93662480,"statfs":{"total":95995035648,"available":95910379520,"internally_reserved":0,"allocated":52723712,"data_stored":51685804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":217,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109572,"kb_used_data":52028,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93635580,"statfs":{"total":95995035648,"available":95882833920,"internally_reserved":0,"allocated":53276672,"data_stored":52276259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378766,"num_pgs":197,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203196,"kb_used_data":149808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93541956,"statfs":{"total":95995035648,"available":95786962944,"internally_reserved":0,"allocated":153403392,"data_stored":152348336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":232,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125240,"kb_used_data":76012,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93619912,"statfs":{"total":95995035648,"available":95866789888,"internally_reserved":0,"allocated":77836288,"data_stored":76850489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855480,"num_pgs":198,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159248,"kb_used_data":101700,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93585904,"statfs":{"total":95995035648,"available":95831965696,"internally_reserved":0,"allocated":104140800,"data_stored":103068747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495726,"num_pgs":232,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112376,"kb_used_data":77036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93632776,"statfs":{"total":95995035648,"available":95879962624,"internally_reserved":0,"allocated":78884864,"data_stored":77898573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50352128,"data_stored":50334527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50334508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151076864,"data_stored":150997874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75513856,"data_stored":75500292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":100708352,"data_stored":100667237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75542528,"data_stored":75500346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100664399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":75530240,"data_stored":75498603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":69,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1077248,"data_stored":1049856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":1050240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1052672,"data_stored":1048960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":99,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":135,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":135,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":135,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":135,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":135,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":136,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":136,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":136,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":136,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":136,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:29:49.989 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-03-11T21:29:49.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: cluster 2024-03-11T21:29:48.492403+0000 mon.a (mon.0) 1288 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.946291+0000 mon.a (mon.0) 1289 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.946550+0000 mon.a (mon.0) 1290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: cluster 2024-03-11T21:29:48.962048+0000 mon.a (mon.0) 1291 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.962159+0000 mon.c (mon.2) 275 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.979426+0000 mon.c (mon.2) 276 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.984239+0000 mon.a (mon.0) 1292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:48.984921+0000 mon.a (mon.0) 1293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:49.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:49 smithi203 bash[23654]: audit 2024-03-11T21:29:49.416951+0000 mon.c (mon.2) 277 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.992 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:49.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:50.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:29:50.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: cluster 2024-03-11T21:29:48.492403+0000 mon.a (mon.0) 1288 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:50.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.946291+0000 mon.a (mon.0) 1289 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.946550+0000 mon.a (mon.0) 1290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: cluster 2024-03-11T21:29:48.962048+0000 mon.a (mon.0) 1291 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.962159+0000 mon.c (mon.2) 275 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.979426+0000 mon.c (mon.2) 276 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.984239+0000 mon.a (mon.0) 1292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:48.984921+0000 mon.a (mon.0) 1293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:50.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[29284]: audit 2024-03-11T21:29:49.416951+0000 mon.c (mon.2) 277 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: cluster 2024-03-11T21:29:48.492403+0000 mon.a (mon.0) 1288 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.946291+0000 mon.a (mon.0) 1289 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.946550+0000 mon.a (mon.0) 1290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi143-71604-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: cluster 2024-03-11T21:29:48.962048+0000 mon.a (mon.0) 1291 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.962159+0000 mon.c (mon.2) 275 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.979426+0000 mon.c (mon.2) 276 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.984239+0000 mon.a (mon.0) 1292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:50.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:48.984921+0000 mon.a (mon.0) 1293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:50.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:49 smithi143 bash[21512]: audit 2024-03-11T21:29:49.416951+0000 mon.c (mon.2) 277 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:50.126 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:50.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:50.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:50.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:50.516 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:50.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.642 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:50.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.770 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:50.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:50.893 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:50.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:51.019 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:51.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.143 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:51.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:51.266 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:51.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:51.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:51.526 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:51.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:51.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:51.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:51.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:48.371064+0000 mgr.y (mgr.14550) 274 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: cluster 2024-03-11T21:29:49.243341+0000 mgr.y (mgr.14550) 275 : cluster 0 pgmap v270: 460 pgs: 64 creating+peering, 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.350216+0000 mon.a (mon.0) 1294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.350436+0000 mon.a (mon.0) 1295 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.355745+0000 mon.c (mon.2) 278 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.367015+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.143:0/3975499080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: cluster 2024-03-11T21:29:50.367425+0000 mon.a (mon.0) 1296 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.371078+0000 mon.a (mon.0) 1297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.383114+0000 mon.a (mon.0) 1298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.418279+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: cluster 2024-03-11T21:29:50.806757+0000 mon.a (mon.0) 1299 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.808653+0000 mon.a (mon.0) 1300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]': finished 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.808726+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: cluster 2024-03-11T21:29:50.816177+0000 mon.a (mon.0) 1302 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:29:51.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.818276+0000 mon.c (mon.2) 281 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:51.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.834393+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:51.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.836292+0000 mon.c (mon.2) 282 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:51.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:51 smithi203 bash[23654]: audit 2024-03-11T21:29:50.844840+0000 mon.a (mon.0) 1304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:51.785 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:51.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.910 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:48.371064+0000 mgr.y (mgr.14550) 274 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: cluster 2024-03-11T21:29:49.243341+0000 mgr.y (mgr.14550) 275 : cluster 0 pgmap v270: 460 pgs: 64 creating+peering, 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.350216+0000 mon.a (mon.0) 1294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.350436+0000 mon.a (mon.0) 1295 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.355745+0000 mon.c (mon.2) 278 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.367015+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.143:0/3975499080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: cluster 2024-03-11T21:29:50.367425+0000 mon.a (mon.0) 1296 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.371078+0000 mon.a (mon.0) 1297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.383114+0000 mon.a (mon.0) 1298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.418279+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: cluster 2024-03-11T21:29:50.806757+0000 mon.a (mon.0) 1299 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.808653+0000 mon.a (mon.0) 1300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]': finished 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.808726+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: cluster 2024-03-11T21:29:50.816177+0000 mon.a (mon.0) 1302 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.818276+0000 mon.c (mon.2) 281 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.834393+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:48.371064+0000 mgr.y (mgr.14550) 274 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: cluster 2024-03-11T21:29:49.243341+0000 mgr.y (mgr.14550) 275 : cluster 0 pgmap v270: 460 pgs: 64 creating+peering, 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.350216+0000 mon.a (mon.0) 1294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.350436+0000 mon.a (mon.0) 1295 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71794-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.355745+0000 mon.c (mon.2) 278 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.367015+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.143:0/3975499080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: cluster 2024-03-11T21:29:50.367425+0000 mon.a (mon.0) 1296 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.371078+0000 mon.a (mon.0) 1297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.383114+0000 mon.a (mon.0) 1298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.418279+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: cluster 2024-03-11T21:29:50.806757+0000 mon.a (mon.0) 1299 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.808653+0000 mon.a (mon.0) 1300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-9", "mode": "writeback"}]': finished 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.808726+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi143-71594-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: cluster 2024-03-11T21:29:50.816177+0000 mon.a (mon.0) 1302 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:29:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.818276+0000 mon.c (mon.2) 281 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:51.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.834393+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:29:51.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.836292+0000 mon.c (mon.2) 282 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:51.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[21512]: audit 2024-03-11T21:29:50.844840+0000 mon.a (mon.0) 1304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:51.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.836292+0000 mon.c (mon.2) 282 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:51.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:51 smithi143 bash[29284]: audit 2024-03-11T21:29:50.844840+0000 mon.a (mon.0) 1304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T21:29:52.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:52.039 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:52.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:29:52.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:29:52.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:52.169 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:52.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:52.299 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:52.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:52.408 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:29:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:52.419 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:52.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:52.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:52.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:52.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:50.917482+0000 mon.c (mon.2) 283 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: cluster 2024-03-11T21:29:51.244338+0000 mgr.y (mgr.14550) 276 : cluster 0 pgmap v273: 524 pgs: 160 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:29:52.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:52.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:52.802 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:50.917482+0000 mon.c (mon.2) 283 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.802 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: cluster 2024-03-11T21:29:51.244338+0000 mgr.y (mgr.14550) 276 : cluster 0 pgmap v273: 524 pgs: 160 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:29:52.802 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.284487+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.802 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.419420+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.812190+0000 mon.a (mon.0) 1306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]': finished 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.812263+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: cluster 2024-03-11T21:29:51.816638+0000 mon.a (mon.0) 1308 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.818493+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.818848+0000 mon.c (mon.2) 286 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.825797+0000 mon.a (mon.0) 1309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.826060+0000 mon.a (mon.0) 1310 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:51.826526+0000 mon.a (mon.0) 1311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:52 smithi203 bash[23654]: audit 2024-03-11T21:29:52.420531+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:52.826 INFO:tasks.workunit.client.0.smithi143.stdout: misc: Running main() from gmock_main.cc 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [----------] Global test environment set-up. 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.Version (15270 ms) 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (15157 ms) 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.LongName (6706 ms) 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3803 ms) 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (5325 ms) 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-11T21:29:52.827 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3763 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.Exec (5453 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (5004 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4811 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.BigObject (5385 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2377 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3865 ms) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [----------] 12 tests from NeoRadosMisc (76920 ms total) 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [----------] Global test environment tear-down 2024-03-11T21:29:52.828 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [==========] 12 tests from 1 test suite ran. (76920 ms total) 2024-03-11T21:29:52.829 INFO:tasks.workunit.client.0.smithi143.stdout: misc: [ PASSED ] 12 tests. 2024-03-11T21:29:52.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.284487+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.419420+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.812190+0000 mon.a (mon.0) 1306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]': finished 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.812263+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: cluster 2024-03-11T21:29:51.816638+0000 mon.a (mon.0) 1308 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.818493+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.818848+0000 mon.c (mon.2) 286 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.825797+0000 mon.a (mon.0) 1309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.826060+0000 mon.a (mon.0) 1310 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:51.826526+0000 mon.a (mon.0) 1311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[21512]: audit 2024-03-11T21:29:52.420531+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:50.917482+0000 mon.c (mon.2) 283 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: cluster 2024-03-11T21:29:51.244338+0000 mgr.y (mgr.14550) 276 : cluster 0 pgmap v273: 524 pgs: 160 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.284487+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.419420+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.812190+0000 mon.a (mon.0) 1306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache", "force_nonempty":""}]': finished 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.812263+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: cluster 2024-03-11T21:29:51.816638+0000 mon.a (mon.0) 1308 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.818493+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.818848+0000 mon.c (mon.2) 286 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.825797+0000 mon.a (mon.0) 1309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]: dispatch 2024-03-11T21:29:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.826060+0000 mon.a (mon.0) 1310 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:51.826526+0000 mon.a (mon.0) 1311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:52.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:52 smithi143 bash[29284]: audit 2024-03-11T21:29:52.420531+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:53.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:53.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:53.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:53.199 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:53.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:53.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:53.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:53.461 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:53.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.587 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:53.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.714 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:53.817 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: cluster 2024-03-11T21:29:52.813559+0000 mon.a (mon.0) 1312 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:53.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: audit 2024-03-11T21:29:52.816152+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:53.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: audit 2024-03-11T21:29:52.816419+0000 mon.a (mon.0) 1314 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:53.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: audit 2024-03-11T21:29:52.816629+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:53.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: cluster 2024-03-11T21:29:52.825530+0000 mon.a (mon.0) 1316 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:29:53.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: audit 2024-03-11T21:29:52.853277+0000 mon.c (mon.2) 288 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:53.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[21512]: audit 2024-03-11T21:29:52.854271+0000 mon.a (mon.0) 1317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: cluster 2024-03-11T21:29:52.813559+0000 mon.a (mon.0) 1312 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: audit 2024-03-11T21:29:52.816152+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: audit 2024-03-11T21:29:52.816419+0000 mon.a (mon.0) 1314 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: audit 2024-03-11T21:29:52.816629+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: cluster 2024-03-11T21:29:52.825530+0000 mon.a (mon.0) 1316 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: audit 2024-03-11T21:29:52.853277+0000 mon.c (mon.2) 288 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:53.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:53 smithi143 bash[29284]: audit 2024-03-11T21:29:52.854271+0000 mon.a (mon.0) 1317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:53.868 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:53.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:54.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:54.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: cluster 2024-03-11T21:29:52.813559+0000 mon.a (mon.0) 1312 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:29:54.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: audit 2024-03-11T21:29:52.816152+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-9"}]': finished 2024-03-11T21:29:54.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: audit 2024-03-11T21:29:52.816419+0000 mon.a (mon.0) 1314 : audit 1 from='client.? 172.21.15.143:0/4000691228' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:54.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: audit 2024-03-11T21:29:52.816629+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71794-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:54.003 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: cluster 2024-03-11T21:29:52.825530+0000 mon.a (mon.0) 1316 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:29:54.003 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: audit 2024-03-11T21:29:52.853277+0000 mon.c (mon.2) 288 : audit 1 from='client.? 172.21.15.143:0/2766804453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:54.003 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:53 smithi203 bash[23654]: audit 2024-03-11T21:29:52.854271+0000 mon.a (mon.0) 1317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]: dispatch 2024-03-11T21:29:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:54.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.126 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:54.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:54.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:54.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.378 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:54.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:54.508 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:54.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:54.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.763 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:54.826 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:54.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.884 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:54.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:54.991 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:29:54.991 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:29:55.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:55.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:55.141 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:55.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:55.370 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:55.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: cluster 2024-03-11T21:29:53.245884+0000 mgr.y (mgr.14550) 277 : cluster 0 pgmap v276: 492 pgs: 1 active+clean+snaptrim, 19 unknown, 472 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: audit 2024-03-11T21:29:53.421651+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: cluster 2024-03-11T21:29:53.816051+0000 mon.a (mon.0) 1318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: audit 2024-03-11T21:29:53.824422+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]': finished 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: cluster 2024-03-11T21:29:53.828104+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:29:55.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: audit 2024-03-11T21:29:53.831546+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.143:0/584453684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: audit 2024-03-11T21:29:54.267196+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[21512]: audit 2024-03-11T21:29:54.423042+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: cluster 2024-03-11T21:29:53.245884+0000 mgr.y (mgr.14550) 277 : cluster 0 pgmap v276: 492 pgs: 1 active+clean+snaptrim, 19 unknown, 472 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: audit 2024-03-11T21:29:53.421651+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: cluster 2024-03-11T21:29:53.816051+0000 mon.a (mon.0) 1318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: audit 2024-03-11T21:29:53.824422+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]': finished 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: cluster 2024-03-11T21:29:53.828104+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: audit 2024-03-11T21:29:53.831546+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.143:0/584453684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: audit 2024-03-11T21:29:54.267196+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:55 smithi143 bash[29284]: audit 2024-03-11T21:29:54.423042+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:55.516 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: cluster 2024-03-11T21:29:53.245884+0000 mgr.y (mgr.14550) 277 : cluster 0 pgmap v276: 492 pgs: 1 active+clean+snaptrim, 19 unknown, 472 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: audit 2024-03-11T21:29:53.421651+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: cluster 2024-03-11T21:29:53.816051+0000 mon.a (mon.0) 1318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: audit 2024-03-11T21:29:53.824422+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71794-10", "tierpool":"test-rados-api-smithi143-71794-10-cache"}]': finished 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: cluster 2024-03-11T21:29:53.828104+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:29:55.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: audit 2024-03-11T21:29:53.831546+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.143:0/584453684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: audit 2024-03-11T21:29:54.267196+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:55.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:55 smithi203 bash[23654]: audit 2024-03-11T21:29:54.423042+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:55.619 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:55.645 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:55.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:55.774 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:55.812 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: Running main() from gmock_main.cc 2024-03-11T21:29:55.812 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-11T21:29:55.812 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] Global test environment set-up. 2024-03-11T21:29:55.812 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolList (20383 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7133 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5005 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (6206 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (7830 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (9304 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (9714 ms) 2024-03-11T21:29:55.813 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6722 ms) 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7881 ms) 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] 9 tests from LibRadosPools (80178 ms total) 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] Global test environment tear-down 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (80178 ms total) 2024-03-11T21:29:55.814 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-11T21:29:55.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:55.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:56.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.022 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:56.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:56.147 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:56.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:56.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:56.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:56.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:56.536 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:56.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:56.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:56.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:56.787 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:56.850 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:56.852 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:56.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.211375+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.213507+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: cluster 2024-03-11T21:29:55.219339+0000 mon.a (mon.0) 1323 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.228450+0000 mon.a (mon.0) 1324 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.239276+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.240087+0000 mon.c (mon.2) 294 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.240935+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.241320+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.423925+0000 mon.c (mon.2) 295 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.812906+0000 mon.a (mon.0) 1327 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.813123+0000 mon.a (mon.0) 1328 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.813351+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: cluster 2024-03-11T21:29:55.824414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:29:56.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.832368+0000 mon.a (mon.0) 1331 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.835290+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2"}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.860854+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[29284]: audit 2024-03-11T21:29:55.861522+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.211375+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.213507+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: cluster 2024-03-11T21:29:55.219339+0000 mon.a (mon.0) 1323 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.228450+0000 mon.a (mon.0) 1324 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.239276+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.240087+0000 mon.c (mon.2) 294 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.240935+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.241320+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.423925+0000 mon.c (mon.2) 295 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.812906+0000 mon.a (mon.0) 1327 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.813123+0000 mon.a (mon.0) 1328 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.813351+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: cluster 2024-03-11T21:29:55.824414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.832368+0000 mon.a (mon.0) 1331 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.835290+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2"}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.860854+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:56 smithi143 bash[21512]: audit 2024-03-11T21:29:55.861522+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:56.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:57.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:57.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.211375+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:29:57.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.213507+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi143-71594-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: cluster 2024-03-11T21:29:55.219339+0000 mon.a (mon.0) 1323 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.228450+0000 mon.a (mon.0) 1324 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.239276+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.240087+0000 mon.c (mon.2) 294 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.240935+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.241320+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.423925+0000 mon.c (mon.2) 295 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.812906+0000 mon.a (mon.0) 1327 : audit 1 from='client.? 172.21.15.143:0/657721367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi143-71604-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.813123+0000 mon.a (mon.0) 1328 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.813351+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: cluster 2024-03-11T21:29:55.824414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:29:57.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.832368+0000 mon.a (mon.0) 1331 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:57.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.835290+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2"}]: dispatch 2024-03-11T21:29:57.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.860854+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:56 smithi203 bash[23654]: audit 2024-03-11T21:29:55.861522+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:57.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:57.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:57.302 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:57.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:57.431 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:57.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:57.553 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:57.655 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:57.681 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:57.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:57.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:57.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: cluster 2024-03-11T21:29:55.246582+0000 mgr.y (mgr.14550) 278 : cluster 0 pgmap v279: 524 pgs: 1 active+clean+snaptrim, 160 unknown, 363 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.424826+0000 mon.c (mon.2) 298 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.815429+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.815508+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: cluster 2024-03-11T21:29:56.821678+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.831039+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.832862+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.833390+0000 mon.c (mon.2) 300 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.834088+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.834291+0000 mon.a (mon.0) 1338 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:56.905877+0000 mon.c (mon.2) 301 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:57.426175+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[21512]: audit 2024-03-11T21:29:57.727025+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: cluster 2024-03-11T21:29:55.246582+0000 mgr.y (mgr.14550) 278 : cluster 0 pgmap v279: 524 pgs: 1 active+clean+snaptrim, 160 unknown, 363 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.424826+0000 mon.c (mon.2) 298 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.815429+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.815508+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: cluster 2024-03-11T21:29:56.821678+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.831039+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.832862+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.833390+0000 mon.c (mon.2) 300 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.834088+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.834291+0000 mon.a (mon.0) 1338 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:56.905877+0000 mon.c (mon.2) 301 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:57.426175+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:57 smithi143 bash[29284]: audit 2024-03-11T21:29:57.727025+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:57.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:58.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: cluster 2024-03-11T21:29:55.246582+0000 mgr.y (mgr.14550) 278 : cluster 0 pgmap v279: 524 pgs: 1 active+clean+snaptrim, 160 unknown, 363 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:29:58.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.424826+0000 mon.c (mon.2) 298 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.815429+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.815508+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi143-71756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: cluster 2024-03-11T21:29:56.821678+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.831039+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.832862+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.833390+0000 mon.c (mon.2) 300 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.834088+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.834291+0000 mon.a (mon.0) 1338 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:56.905877+0000 mon.c (mon.2) 301 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:57.426175+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:57 smithi203 bash[23654]: audit 2024-03-11T21:29:57.727025+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:29:58.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:58.080 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:58.183 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:58.205 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:58.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:58.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:58.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:58.465 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:58.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:58.595 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:58.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:58.724 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:58.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:29:58.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:58.871 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:58.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:58.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:59.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: cluster 2024-03-11T21:29:57.247027+0000 mgr.y (mgr.14550) 279 : cluster 0 pgmap v282: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.829273+0000 mon.a (mon.0) 1340 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.829413+0000 mon.a (mon.0) 1341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.829615+0000 mon.a (mon.0) 1342 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.829730+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: cluster 2024-03-11T21:29:57.835795+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.840571+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.843587+0000 mon.c (mon.2) 304 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.844304+0000 mon.c (mon.2) 305 : audit 1 from='client.? 172.21.15.143:0/4166909090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.863375+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.864002+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.864389+0000 mon.a (mon.0) 1347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.916764+0000 mon.b (mon.1) 81 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.918574+0000 mon.b (mon.1) 82 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.919368+0000 mon.b (mon.1) 83 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.920032+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.921699+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:57.922425+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:29:58 smithi203 bash[23654]: audit 2024-03-11T21:29:58.427677+0000 mon.c (mon.2) 306 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:59.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:59.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: cluster 2024-03-11T21:29:57.247027+0000 mgr.y (mgr.14550) 279 : cluster 0 pgmap v282: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.829273+0000 mon.a (mon.0) 1340 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.829413+0000 mon.a (mon.0) 1341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.829615+0000 mon.a (mon.0) 1342 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.829730+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: cluster 2024-03-11T21:29:57.835795+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.840571+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.843587+0000 mon.c (mon.2) 304 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.844304+0000 mon.c (mon.2) 305 : audit 1 from='client.? 172.21.15.143:0/4166909090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.863375+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.864002+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.864389+0000 mon.a (mon.0) 1347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.916764+0000 mon.b (mon.1) 81 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.918574+0000 mon.b (mon.1) 82 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: cluster 2024-03-11T21:29:57.247027+0000 mgr.y (mgr.14550) 279 : cluster 0 pgmap v282: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.829273+0000 mon.a (mon.0) 1340 : audit 1 from='client.? 172.21.15.143:0/202579008' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi143-71818-10"}]': finished 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.829413+0000 mon.a (mon.0) 1341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.829615+0000 mon.a (mon.0) 1342 : audit 1 from='client.? 172.21.15.143:0/2461441470' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi143-71594-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.829730+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: cluster 2024-03-11T21:29:57.835795+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.840571+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.843587+0000 mon.c (mon.2) 304 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.844304+0000 mon.c (mon.2) 305 : audit 1 from='client.? 172.21.15.143:0/4166909090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.863375+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.864002+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.864389+0000 mon.a (mon.0) 1347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.916764+0000 mon.b (mon.1) 81 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.918574+0000 mon.b (mon.1) 82 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.919368+0000 mon.b (mon.1) 83 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.920032+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.921699+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:57.922425+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[21512]: audit 2024-03-11T21:29:58.427677+0000 mon.c (mon.2) 306 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:59.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.919368+0000 mon.b (mon.1) 83 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.920032+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.921699+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:29:59.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:57.922425+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:29:59.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:29:58 smithi143 bash[29284]: audit 2024-03-11T21:29:58.427677+0000 mon.c (mon.2) 306 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:29:59.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:59.261 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:59.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:59.488 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:59.511 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:59.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:59.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:59.640 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:59.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:59.770 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:59.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:59.894 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:59.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:00.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:00.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:00.150 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:00.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:00.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:00.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:00.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.380294+0000 mgr.y (mgr.14550) 280 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.833022+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.833318+0000 mon.a (mon.0) 1352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.833587+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.833844+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.841428+0000 mon.b (mon.1) 84 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: cluster 2024-03-11T21:29:58.843906+0000 mon.a (mon.0) 1355 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:30:00.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.848308+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.858648+0000 mon.a (mon.0) 1356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.859474+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: cluster 2024-03-11T21:29:58.895188+0000 osd.0 (osd.0) 147 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: cluster 2024-03-11T21:29:58.895280+0000 osd.0 (osd.0) 148 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.896815+0000 mon.c (mon.2) 308 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:58.897572+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:59.429073+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: cluster 2024-03-11T21:29:59.834548+0000 mon.a (mon.0) 1359 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[21512]: audit 2024-03-11T21:29:59.836958+0000 mon.a (mon.0) 1360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]': finished 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.380294+0000 mgr.y (mgr.14550) 280 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.833022+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.833318+0000 mon.a (mon.0) 1352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:30:00.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.833587+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.833844+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.841428+0000 mon.b (mon.1) 84 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: cluster 2024-03-11T21:29:58.843906+0000 mon.a (mon.0) 1355 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.848308+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.858648+0000 mon.a (mon.0) 1356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.859474+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: cluster 2024-03-11T21:29:58.895188+0000 osd.0 (osd.0) 147 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: cluster 2024-03-11T21:29:58.895280+0000 osd.0 (osd.0) 148 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.896815+0000 mon.c (mon.2) 308 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:58.897572+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:59.429073+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: cluster 2024-03-11T21:29:59.834548+0000 mon.a (mon.0) 1359 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:00.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:00 smithi143 bash[29284]: audit 2024-03-11T21:29:59.836958+0000 mon.a (mon.0) 1360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]': finished 2024-03-11T21:30:00.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:00.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:00.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.380294+0000 mgr.y (mgr.14550) 280 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.833022+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.833318+0000 mon.a (mon.0) 1352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.833587+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi143-71604-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.833844+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.841428+0000 mon.b (mon.1) 84 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: cluster 2024-03-11T21:29:58.843906+0000 mon.a (mon.0) 1355 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.848308+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.858648+0000 mon.a (mon.0) 1356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.859474+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: cluster 2024-03-11T21:29:58.895188+0000 osd.0 (osd.0) 147 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: cluster 2024-03-11T21:29:58.895280+0000 osd.0 (osd.0) 148 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.896815+0000 mon.c (mon.2) 308 : audit 1 from='client.? 172.21.15.143:0/4179355585' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:58.897572+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:30:00.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:59.429073+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:00.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: cluster 2024-03-11T21:29:59.834548+0000 mon.a (mon.0) 1359 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:00.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:00 smithi203 bash[23654]: audit 2024-03-11T21:29:59.836958+0000 mon.a (mon.0) 1360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-11", "mode": "writeback"}]': finished 2024-03-11T21:30:00.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:00.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:00.788 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:00.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:00.913 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:01.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:01.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:01.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:01.177 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: seed 71756 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-03-11T21:30:01.190 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (328 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (28 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (11 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (15 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (8 ms) 2024-03-11T21:30:01.191 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (16 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (7 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2272 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (418 ms) 2024-03-11T21:30:01.192 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: done waiting, doing copies 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: done waiting 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60902 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (8 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4644 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-11T21:30:01.193 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-11T21:30:01.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (1 ms) 2024-03-11T21:30:01.260 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:01.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:01.304 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:01.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:01.431 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:01.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:29:59.247893+0000 mgr.y (mgr.14550) 281 : cluster 0 pgmap v285: 420 pgs: 64 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:29:59.837166+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.302675+0000 mon.a (mon.0) 1362 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.303182+0000 mon.a (mon.0) 1363 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.303864+0000 mon.a (mon.0) 1364 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:00.304127+0000 mon.c (mon.2) 310 : audit 1 from='client.? 172.21.15.143:0/2739817085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304268+0000 mon.a (mon.0) 1366 : cluster 3 pool 'test-rados-api-smithi143-71773-11' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304290+0000 mon.a (mon.0) 1367 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304305+0000 mon.a (mon.0) 1368 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304324+0000 mon.a (mon.0) 1369 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304337+0000 mon.a (mon.0) 1370 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304354+0000 mon.a (mon.0) 1371 : cluster 3 application not enabled on pool 'IsSafe_smithi143-71594-13' 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304366+0000 mon.a (mon.0) 1372 : cluster 3 application not enabled on pool 'GetXattrssmithi143-72391-13' 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:00.304390+0000 mon.a (mon.0) 1373 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:00.312048+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:00.430060+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:01.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.192227+0000 mon.a (mon.0) 1375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:29:59.247893+0000 mgr.y (mgr.14550) 281 : cluster 0 pgmap v285: 420 pgs: 64 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:29:59.837166+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.302675+0000 mon.a (mon.0) 1362 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.303182+0000 mon.a (mon.0) 1363 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.303864+0000 mon.a (mon.0) 1364 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:00.304127+0000 mon.c (mon.2) 310 : audit 1 from='client.? 172.21.15.143:0/2739817085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304268+0000 mon.a (mon.0) 1366 : cluster 3 pool 'test-rados-api-smithi143-71773-11' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304290+0000 mon.a (mon.0) 1367 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304305+0000 mon.a (mon.0) 1368 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304324+0000 mon.a (mon.0) 1369 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304337+0000 mon.a (mon.0) 1370 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304354+0000 mon.a (mon.0) 1371 : cluster 3 application not enabled on pool 'IsSafe_smithi143-71594-13' 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304366+0000 mon.a (mon.0) 1372 : cluster 3 application not enabled on pool 'GetXattrssmithi143-72391-13' 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:00.304390+0000 mon.a (mon.0) 1373 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:00.312048+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:00.430060+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.192227+0000 mon.a (mon.0) 1375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.192550+0000 mon.a (mon.0) 1376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:01.207829+0000 mon.a (mon.0) 1377 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.218257+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.143:0/3361103118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.228503+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.270600+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.276413+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.277270+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.296139+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[21512]: audit 2024-03-11T21:30:01.296716+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.192550+0000 mon.a (mon.0) 1376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:01.207829+0000 mon.a (mon.0) 1377 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.218257+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.143:0/3361103118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.228503+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.270600+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.276413+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.277270+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:01.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.296139+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:01 smithi143 bash[29284]: audit 2024-03-11T21:30:01.296716+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:01.691 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:29:59.247893+0000 mgr.y (mgr.14550) 281 : cluster 0 pgmap v285: 420 pgs: 64 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:29:59.837166+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi143-71756-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.302675+0000 mon.a (mon.0) 1362 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.303182+0000 mon.a (mon.0) 1363 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.303864+0000 mon.a (mon.0) 1364 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:00.304127+0000 mon.c (mon.2) 310 : audit 1 from='client.? 172.21.15.143:0/2739817085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304268+0000 mon.a (mon.0) 1366 : cluster 3 pool 'test-rados-api-smithi143-71773-11' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304290+0000 mon.a (mon.0) 1367 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304305+0000 mon.a (mon.0) 1368 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304324+0000 mon.a (mon.0) 1369 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304337+0000 mon.a (mon.0) 1370 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304354+0000 mon.a (mon.0) 1371 : cluster 3 application not enabled on pool 'IsSafe_smithi143-71594-13' 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304366+0000 mon.a (mon.0) 1372 : cluster 3 application not enabled on pool 'GetXattrssmithi143-72391-13' 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:00.304390+0000 mon.a (mon.0) 1373 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:00.312048+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:00.430060+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.192227+0000 mon.a (mon.0) 1375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi143-71818-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.192550+0000 mon.a (mon.0) 1376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi143-71594-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:01.207829+0000 mon.a (mon.0) 1377 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.218257+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.143:0/3361103118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.228503+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:01.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.270600+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.276413+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:01.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.277270+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:01.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.296139+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:01 smithi203 bash[23654]: audit 2024-03-11T21:30:01.296716+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:01.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:01.821 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:01.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:01.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:02.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:02.076 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:02.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP read_operations: Running main() from gmock_main.cc 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [----------] Global test environment set-up. 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (22394 ms) 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-11T21:30:02.194 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (7019 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3732 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (7776 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (7762 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3831 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (6984 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3546 ms) 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-11T21:30:02.195 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3907 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (6101 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3848 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3002 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3024 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3350 ms) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (86278 ms total) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [----------] Global test environment tear-down 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (86278 ms total) 2024-03-11T21:30:02.196 INFO:tasks.workunit.client.0.smithi143.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-11T21:30:02.216 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:02.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:02.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:02.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.606 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:02.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.730 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:02.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:02.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:02.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: cluster 2024-03-11T21:30:01.248626+0000 mgr.y (mgr.14550) 282 : cluster 0 pgmap v288: 460 pgs: 136 unknown, 324 active+clean; 586 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:01.431208+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.200607+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.200732+0000 mon.a (mon.0) 1384 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.200782+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: cluster 2024-03-11T21:30:02.206881+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.208757+0000 mon.a (mon.0) 1387 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.209779+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:02 smithi203 bash[23654]: audit 2024-03-11T21:30:02.213222+0000 mon.a (mon.0) 1388 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: cluster 2024-03-11T21:30:01.248626+0000 mgr.y (mgr.14550) 282 : cluster 0 pgmap v288: 460 pgs: 136 unknown, 324 active+clean; 586 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:01.431208+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.200607+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.200732+0000 mon.a (mon.0) 1384 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.200782+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: cluster 2024-03-11T21:30:02.206881+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:30:02.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.208757+0000 mon.a (mon.0) 1387 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.209779+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[29284]: audit 2024-03-11T21:30:02.213222+0000 mon.a (mon.0) 1388 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: cluster 2024-03-11T21:30:01.248626+0000 mgr.y (mgr.14550) 282 : cluster 0 pgmap v288: 460 pgs: 136 unknown, 324 active+clean; 586 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:01.431208+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.200607+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi143-71604-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.200732+0000 mon.a (mon.0) 1384 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.200782+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: cluster 2024-03-11T21:30:02.206881+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:30:02.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.208757+0000 mon.a (mon.0) 1387 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:02.982 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.209779+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.982 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:02 smithi143 bash[21512]: audit 2024-03-11T21:30:02.213222+0000 mon.a (mon.0) 1388 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]: dispatch 2024-03-11T21:30:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:03.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:03.118 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:03.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:03.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:03.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:03.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:03.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:03.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:03.639 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:03.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:03.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:03.898 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:04.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:04.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:04.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:04.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:04.293 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:04.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:04.418 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:04.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: audit 2024-03-11T21:30:02.432537+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: cluster 2024-03-11T21:30:03.203295+0000 mon.a (mon.0) 1389 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: audit 2024-03-11T21:30:03.205886+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: audit 2024-03-11T21:30:03.433202+0000 mon.c (mon.2) 317 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: cluster 2024-03-11T21:30:03.659159+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: audit 2024-03-11T21:30:03.661488+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.143:0/452795439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:04 smithi203 bash[23654]: audit 2024-03-11T21:30:03.662846+0000 mon.a (mon.0) 1392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.558 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:04.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: audit 2024-03-11T21:30:02.432537+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: cluster 2024-03-11T21:30:03.203295+0000 mon.a (mon.0) 1389 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: audit 2024-03-11T21:30:03.205886+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: audit 2024-03-11T21:30:03.433202+0000 mon.c (mon.2) 317 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: cluster 2024-03-11T21:30:03.659159+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: audit 2024-03-11T21:30:03.661488+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.143:0/452795439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[29284]: audit 2024-03-11T21:30:03.662846+0000 mon.a (mon.0) 1392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: audit 2024-03-11T21:30:02.432537+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: cluster 2024-03-11T21:30:03.203295+0000 mon.a (mon.0) 1389 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: audit 2024-03-11T21:30:03.205886+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-11"}]': finished 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: audit 2024-03-11T21:30:03.433202+0000 mon.c (mon.2) 317 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: cluster 2024-03-11T21:30:03.659159+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: audit 2024-03-11T21:30:03.661488+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.143:0/452795439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:04 smithi143 bash[21512]: audit 2024-03-11T21:30:03.662846+0000 mon.a (mon.0) 1392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:04.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:04.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:04.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:04.920 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:04.943 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:05.046 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.075 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:05.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:05.205 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:05.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:05.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:05.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:05.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:05.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:05.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:05.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.738 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:05.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: cluster 2024-03-11T21:30:03.249709+0000 mgr.y (mgr.14550) 283 : cluster 0 pgmap v291: 364 pgs: 5 active+clean+snaptrim, 32 unknown, 327 active+clean; 594 KiB data, 513 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:05.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:04.434215+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:04.556102+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:04.556190+0000 mon.a (mon.0) 1394 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: cluster 2024-03-11T21:30:04.561390+0000 mon.a (mon.0) 1395 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:04.574424+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.143:0/3164395311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:05.406908+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:05 smithi203 bash[23654]: audit 2024-03-11T21:30:05.435739+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: cluster 2024-03-11T21:30:03.249709+0000 mgr.y (mgr.14550) 283 : cluster 0 pgmap v291: 364 pgs: 5 active+clean+snaptrim, 32 unknown, 327 active+clean; 594 KiB data, 513 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:04.434215+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:04.556102+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:04.556190+0000 mon.a (mon.0) 1394 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: cluster 2024-03-11T21:30:04.561390+0000 mon.a (mon.0) 1395 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:30:05.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:04.574424+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.143:0/3164395311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:05.406908+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[21512]: audit 2024-03-11T21:30:05.435739+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: cluster 2024-03-11T21:30:03.249709+0000 mgr.y (mgr.14550) 283 : cluster 0 pgmap v291: 364 pgs: 5 active+clean+snaptrim, 32 unknown, 327 active+clean; 594 KiB data, 513 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:04.434215+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:04.556102+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:04.556190+0000 mon.a (mon.0) 1394 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi143-71594-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: cluster 2024-03-11T21:30:04.561390+0000 mon.a (mon.0) 1395 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:04.574424+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.143:0/3164395311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:05.406908+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:05.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:05 smithi143 bash[29284]: audit 2024-03-11T21:30:05.435739+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:05.868 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:05.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:05.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:06.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:06.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:06.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:06.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:06.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:06.517 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:06.594 INFO:tasks.workunit.client.0.smithi143.stdout: io: Running main() from gmock_main.cc 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [----------] Global test environment set-up. 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.Limits (22014 ms) 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (8454 ms) 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.ReadOp (10517 ms) 2024-03-11T21:30:06.595 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.SparseRead (5281 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.RoundTrip (6289 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (7990 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (4816 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (5363 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3784 ms) 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-11T21:30:06.596 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.Trunc (3455 ms) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.Remove (3000 ms) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3480 ms) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.RmXattr (3351 ms) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2951 ms) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [----------] 14 tests from NeoRadosIo (90746 ms total) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [----------] Global test environment tear-down 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [==========] 14 tests from 1 test suite ran. (90748 ms total) 2024-03-11T21:30:06.597 INFO:tasks.workunit.client.0.smithi143.stdout: io: [ PASSED ] 14 tests. 2024-03-11T21:30:06.620 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:06.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: cluster 2024-03-11T21:30:05.250712+0000 mgr.y (mgr.14550) 284 : cluster 0 pgmap v293: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: cluster 2024-03-11T21:30:05.556144+0000 mon.a (mon.0) 1397 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:05.569523+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: cluster 2024-03-11T21:30:05.578780+0000 mon.a (mon.0) 1399 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:05.587391+0000 mon.c (mon.2) 322 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:05.593172+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:05.594715+0000 mon.a (mon.0) 1401 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:05.864615+0000 mon.c (mon.2) 323 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:06.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:06 smithi203 bash[23654]: audit 2024-03-11T21:30:06.437188+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:06.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: cluster 2024-03-11T21:30:05.250712+0000 mgr.y (mgr.14550) 284 : cluster 0 pgmap v293: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: cluster 2024-03-11T21:30:05.556144+0000 mon.a (mon.0) 1397 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:05.569523+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: cluster 2024-03-11T21:30:05.578780+0000 mon.a (mon.0) 1399 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:05.587391+0000 mon.c (mon.2) 322 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:05.593172+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:05.594715+0000 mon.a (mon.0) 1401 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:05.864615+0000 mon.c (mon.2) 323 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:06.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[29284]: audit 2024-03-11T21:30:06.437188+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: cluster 2024-03-11T21:30:05.250712+0000 mgr.y (mgr.14550) 284 : cluster 0 pgmap v293: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: cluster 2024-03-11T21:30:05.556144+0000 mon.a (mon.0) 1397 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:05.569523+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi143-71604-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: cluster 2024-03-11T21:30:05.578780+0000 mon.a (mon.0) 1399 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:05.587391+0000 mon.c (mon.2) 322 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:05.593172+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:05.594715+0000 mon.a (mon.0) 1401 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:05.864615+0000 mon.c (mon.2) 323 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:06.788 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:06 smithi143 bash[21512]: audit 2024-03-11T21:30:06.437188+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:06.888 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:06.910 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:07.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:07.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:07.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:07.169 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:07.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.298 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:07.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:07.429 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:07.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:07.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:07.701 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:07.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:07.818 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: audit 2024-03-11T21:30:06.575293+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: audit 2024-03-11T21:30:06.575519+0000 mon.a (mon.0) 1403 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: cluster 2024-03-11T21:30:06.583422+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: audit 2024-03-11T21:30:06.590132+0000 mon.a (mon.0) 1405 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: audit 2024-03-11T21:30:06.596178+0000 mon.a (mon.0) 1406 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[29284]: audit 2024-03-11T21:30:07.438595+0000 mon.c (mon.2) 325 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: audit 2024-03-11T21:30:06.575293+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: audit 2024-03-11T21:30:06.575519+0000 mon.a (mon.0) 1403 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: cluster 2024-03-11T21:30:06.583422+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: audit 2024-03-11T21:30:06.590132+0000 mon.a (mon.0) 1405 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: audit 2024-03-11T21:30:06.596178+0000 mon.a (mon.0) 1406 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:07 smithi143 bash[21512]: audit 2024-03-11T21:30:07.438595+0000 mon.c (mon.2) 325 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:07.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:07.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: audit 2024-03-11T21:30:06.575293+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: audit 2024-03-11T21:30:06.575519+0000 mon.a (mon.0) 1403 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:07.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: cluster 2024-03-11T21:30:06.583422+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:30:07.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: audit 2024-03-11T21:30:06.590132+0000 mon.a (mon.0) 1405 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: audit 2024-03-11T21:30:06.596178+0000 mon.a (mon.0) 1406 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:07.948 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:07 smithi203 bash[23654]: audit 2024-03-11T21:30:07.438595+0000 mon.c (mon.2) 325 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:07.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:08.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:08.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:08.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:08.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:08.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:08.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:08.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:08.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:08.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:08.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:08.632 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:08.698 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:08.699 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:08.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:08.750 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: cluster 2024-03-11T21:30:07.251649+0000 mgr.y (mgr.14550) 285 : cluster 0 pgmap v296: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:08.750 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: audit 2024-03-11T21:30:07.589212+0000 mon.a (mon.0) 1407 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.750 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: audit 2024-03-11T21:30:07.589480+0000 mon.a (mon.0) 1408 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.750 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: cluster 2024-03-11T21:30:07.598930+0000 mon.a (mon.0) 1409 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:30:08.750 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: audit 2024-03-11T21:30:07.601927+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.143:0/3419979794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: audit 2024-03-11T21:30:07.603382+0000 mon.a (mon.0) 1410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[29284]: audit 2024-03-11T21:30:08.440235+0000 mon.c (mon.2) 327 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: cluster 2024-03-11T21:30:07.251649+0000 mgr.y (mgr.14550) 285 : cluster 0 pgmap v296: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: audit 2024-03-11T21:30:07.589212+0000 mon.a (mon.0) 1407 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: audit 2024-03-11T21:30:07.589480+0000 mon.a (mon.0) 1408 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: cluster 2024-03-11T21:30:07.598930+0000 mon.a (mon.0) 1409 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: audit 2024-03-11T21:30:07.601927+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.143:0/3419979794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: audit 2024-03-11T21:30:07.603382+0000 mon.a (mon.0) 1410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:08 smithi143 bash[21512]: audit 2024-03-11T21:30:08.440235+0000 mon.c (mon.2) 327 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: cluster 2024-03-11T21:30:07.251649+0000 mgr.y (mgr.14550) 285 : cluster 0 pgmap v296: 404 pgs: 2 active+clean+snaptrim, 104 unknown, 298 active+clean; 594 KiB data, 449 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: audit 2024-03-11T21:30:07.589212+0000 mon.a (mon.0) 1407 : audit 1 from='client.? 172.21.15.143:0/1518096148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi143-71594-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: audit 2024-03-11T21:30:07.589480+0000 mon.a (mon.0) 1408 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: cluster 2024-03-11T21:30:07.598930+0000 mon.a (mon.0) 1409 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: audit 2024-03-11T21:30:07.601927+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.143:0/3419979794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: audit 2024-03-11T21:30:07.603382+0000 mon.a (mon.0) 1410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:08.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:08 smithi203 bash[23654]: audit 2024-03-11T21:30:08.440235+0000 mon.c (mon.2) 327 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:08.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:08.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:08.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:09.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:09.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:09.260 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:09.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:09.393 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:09.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.527 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:09.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.649 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[29284]: audit 2024-03-11T21:30:08.388085+0000 mgr.y (mgr.14550) 286 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[29284]: audit 2024-03-11T21:30:08.599138+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[29284]: cluster 2024-03-11T21:30:08.609073+0000 mon.a (mon.0) 1412 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[29284]: audit 2024-03-11T21:30:09.442014+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[21512]: audit 2024-03-11T21:30:08.388085+0000 mgr.y (mgr.14550) 286 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[21512]: audit 2024-03-11T21:30:08.599138+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[21512]: cluster 2024-03-11T21:30:08.609073+0000 mon.a (mon.0) 1412 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:30:09.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:09 smithi143 bash[21512]: audit 2024-03-11T21:30:09.442014+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:09.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.775 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:09.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 -- 172.21.15.143:0/762493285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 msgr2=0x7fbbe010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:09.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/762493285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe010db70 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b0a0 tx=0x7fbbd800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 -- 172.21.15.143:0/762493285 shutdown_connections 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/762493285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0106dc0 0x7fbbe01071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/762493285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe010db70 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/762493285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0100d40 0x7fbbe01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 -- 172.21.15.143:0/762493285 >> 172.21.15.143:0/762493285 conn(0x7fbbe0076d80 msgr2=0x7fbbe0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 -- 172.21.15.143:0/762493285 shutdown_connections 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 -- 172.21.15.143:0/762493285 wait complete. 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 librados: starting msgr at 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 librados: starting objecter 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.886+0000 7fbbe5c80700 1 Processor -- start 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- start start 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 librados: setting wanted keys 2024-03-11T21:30:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 librados: calling monclient init 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0100d40 0x7fbbe019b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0106dc0 0x7fbbe0195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f460 con 0x7fbbe0106dc0 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f2e0 con 0x7fbbe0100d40 2024-03-11T21:30:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f5e0 con 0x7fbbe0101700 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe019bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:59098/0 (socket says 172.21.15.143:59098) 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 -- 172.21.15.143:0/3662532915 learned_addr learned my addr 172.21.15.143:0/3662532915 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0106dc0 0x7fbbe0195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0100d40 0x7fbbe019b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0100d40 msgr2=0x7fbbe019b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0100d40 0x7fbbe019b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0106dc0 msgr2=0x7fbbe0195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0106dc0 0x7fbbe0195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe0196000 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe019bb40 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400d980 tx=0x7fbbd400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 10 client.?.objecter ms_handle_connect 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4017070 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe01962f0 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fbbd4005020 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4009ad0 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 10 client.40895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe01a2580 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 librados: init done 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbbe0102400 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbbd4009c70 con 0x7fbbe0101700 2024-03-11T21:30:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.890+0000 7fbbdcff9700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbbc806f820 0x7fbbc8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(208..208 src has 87..208) v4 ==== 11329+0+0 (secure 0 0 0) 0x7fbbd408f010 con 0x7fbbe0101700 2024-03-11T21:30:09.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 osd_map(208..208 src has 87..208) v4 2024-03-11T21:30:09.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 3 client.40895.objecter handle_osd_map got epochs [208,208] > 0 2024-03-11T21:30:09.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 3 client.40895.objecter handle_osd_map decoding full epoch 208 2024-03-11T21:30:09.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbbc806f820 0x7fbbc8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 20 client.40895.objecter dump_active .. 0 homeless 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v208) v1 ==== 54+0+16 (secure 0 0 0) 0x7fbbd4020020 con 0x7fbbe0101700 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbc806f820 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbbc806f820 0x7fbbc8071ce0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000be00 tx=0x7fbbd000d040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 1 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fbbe0062c40 0x7fbbe0107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 1 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 0 1 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 4 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 10 client.40895.objecter _send_command 1 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe004faa0 con 0x7fbbe0062c40 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fbbe0062c40 0x7fbbe0107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe0062c40 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fbbe0062c40 0x7fbbe0107ab0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe004faa0 con 0x7fbbe0062c40 2024-03-11T21:30:09.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:30:09.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 1 = system:0 2024-03-11T21:30:09.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 4 2024-03-11T21:30:09.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.894+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 0 1 2024-03-11T21:30:09.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 2 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 2 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe0196660 osd=0 3 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 0 2 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 4 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 10 client.40895.objecter _send_command 2 2024-03-11T21:30:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe0196c00 con 0x7fbbe0062c40 2024-03-11T21:30:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe0196c00 con 0x7fbbe0062c40 2024-03-11T21:30:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:09.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:09.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 4 2024-03-11T21:30:09.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 0 2 2024-03-11T21:30:09.892 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.914+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7fbbe002ce80 0x7fbbe002d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 1 3 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 4 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 10 client.40895.objecter _send_command 3 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe0109170 con 0x7fbbe002ce80 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7fbbe002ce80 0x7fbbe002d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe002ce80 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7fbbe002ce80 0x7fbbe002d260 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe0109170 con 0x7fbbe002ce80 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 3 = system:0 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 4 2024-03-11T21:30:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.918+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 1 3 2024-03-11T21:30:09.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:09.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 4 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 4 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe002dd40 osd=1 3 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 1 4 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 4 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 10 client.40895.objecter _send_command 4 2024-03-11T21:30:09.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.938+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe002c900 con 0x7fbbe002ce80 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe002c900 con 0x7fbbe002ce80 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 4 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 1 4 2024-03-11T21:30:09.917 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 5 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fbbe0108d20 0x7fbbe01a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 5 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 2 5 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 4 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 10 client.40895.objecter _send_command 5 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01ac990 con 0x7fbbe0108d20 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fbbe0108d20 0x7fbbe01a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe0108d20 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fbbe0108d20 0x7fbbe01a9900 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01ac990 con 0x7fbbe0108d20 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 5 = system:0 2024-03-11T21:30:09.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 4 2024-03-11T21:30:09.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.942+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 2 5 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 6 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 6 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe002cd00 osd=2 3 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 2 6 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 4 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 10 client.40895.objecter _send_command 6 2024-03-11T21:30:09.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.958+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe0108300 con 0x7fbbe0108d20 2024-03-11T21:30:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe0108300 con 0x7fbbe0108d20 2024-03-11T21:30:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 4 2024-03-11T21:30:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 2 6 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 7 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fbbe01ab0b0 0x7fbbe01ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 7 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 3 7 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 4 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 10 client.40895.objecter _send_command 7 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01aba70 con 0x7fbbe01ab0b0 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fbbe01ab0b0 0x7fbbe01ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fbbe01ab0b0 0x7fbbe01ab530 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe01ab0b0 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01aba70 con 0x7fbbe01ab0b0 2024-03-11T21:30:09.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:30:09.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 7 = system:0 2024-03-11T21:30:09.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 4 2024-03-11T21:30:09.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.970+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 3 7 2024-03-11T21:30:09.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:09.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 8 2024-03-11T21:30:09.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 8 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe0108590 osd=3 3 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 3 8 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 4 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 10 client.40895.objecter _send_command 8 2024-03-11T21:30:09.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe01aba70 con 0x7fbbe01ab0b0 2024-03-11T21:30:09.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe01aba70 con 0x7fbbe01ab0b0 2024-03-11T21:30:09.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:09.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:09.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 4 2024-03-11T21:30:09.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 3 8 2024-03-11T21:30:09.971 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:30:09.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 9 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fbbe01a6d80 0x7fbbe01b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 9 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 4 9 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 4 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 10 client.40895.objecter _send_command 9 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01b3bc0 con 0x7fbbe01a6d80 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fbbe01a6d80 0x7fbbe01b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fbbe01a6d80 0x7fbbe01b0bf0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:09.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.994+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe01a6d80 2024-03-11T21:30:09.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.998+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01b3bc0 con 0x7fbbe01a6d80 2024-03-11T21:30:09.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.998+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:30:09.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.998+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 9 = system:0 2024-03-11T21:30:09.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.998+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 4 2024-03-11T21:30:09.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:09.998+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 4 9 2024-03-11T21:30:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 10 2024-03-11T21:30:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 10 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01a6c10 osd=4 3 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 4 10 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 4 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 10 client.40895.objecter _send_command 10 2024-03-11T21:30:09.992 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.014+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe01a6a20 con 0x7fbbe01a6d80 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe01a6a20 con 0x7fbbe01a6d80 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 4 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 4 10 2024-03-11T21:30:09.997 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 11 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fbbe01b2ec0 0x7fbbe01b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 11 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 5 11 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 4 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 10 client.40895.objecter _send_command 11 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01b6f90 con 0x7fbbe01b2ec0 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fbbe01b2ec0 0x7fbbe01b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe01b2ec0 2024-03-11T21:30:09.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.022+0000 7fbbdeffd700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fbbe01b2ec0 0x7fbbe01b3320 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:10.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.026+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01b6f90 con 0x7fbbe01b2ec0 2024-03-11T21:30:10.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.026+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:30:10.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.026+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 11 = system:0 2024-03-11T21:30:10.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.026+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 4 2024-03-11T21:30:10.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.026+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 5 11 2024-03-11T21:30:10.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:09 smithi203 bash[23654]: audit 2024-03-11T21:30:08.388085+0000 mgr.y (mgr.14550) 286 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:10.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:09 smithi203 bash[23654]: audit 2024-03-11T21:30:08.599138+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi143-71604-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:10.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:09 smithi203 bash[23654]: cluster 2024-03-11T21:30:08.609073+0000 mon.a (mon.0) 1412 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:30:10.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:09 smithi203 bash[23654]: audit 2024-03-11T21:30:09.442014+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:10.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:10.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 12 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 12 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01a6a20 osd=5 3 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 5 12 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 4 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 10 client.40895.objecter _send_command 12 2024-03-11T21:30:10.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.042+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe01b1180 con 0x7fbbe01b2ec0 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe01b1180 con 0x7fbbe01b2ec0 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 4 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 5 12 2024-03-11T21:30:10.022 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 13 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fbbe01b1690 0x7fbbe01b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 13 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 6 13 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 4 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 10 client.40895.objecter _send_command 13 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01b2030 con 0x7fbbe01b1690 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fbbe01b1690 0x7fbbe01b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe01b1690 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.046+0000 7fbbdffff700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fbbe01b1690 0x7fbbe01b1af0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.050+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01b2030 con 0x7fbbe01b1690 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.050+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.050+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 13 = system:0 2024-03-11T21:30:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.050+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 4 2024-03-11T21:30:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.050+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 6 13 2024-03-11T21:30:10.033 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:10.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 14 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 14 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01b24e0 osd=6 3 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 6 14 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 4 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 10 client.40895.objecter _send_command 14 2024-03-11T21:30:10.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe01b2030 con 0x7fbbe01b1690 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe01b2030 con 0x7fbbe01b1690 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 4 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 6 14 2024-03-11T21:30:10.043 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 15 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fbbe01bc740 0x7fbbe01beb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 15 2024-03-11T21:30:10.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 7 15 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 4 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 10 client.40895.objecter _send_command 15 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe01c1990 con 0x7fbbe01bc740 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fbbe01bc740 0x7fbbe01beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdcff9700 10 client.40895.objecter ms_handle_connect 0x7fbbe01bc740 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.066+0000 7fbbdf7fe700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fbbe01bc740 0x7fbbe01beb30 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.070+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe01c1990 con 0x7fbbe01bc740 2024-03-11T21:30:10.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.070+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:30:10.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.070+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 15 = system:0 2024-03-11T21:30:10.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.070+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 4 2024-03-11T21:30:10.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.070+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 7 15 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 10 client.40895.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign -1 16 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 20 client.40895.objecter _get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_remove -1 16 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 20 client.40895.objecter get_session s=0x7fbbe01b2030 osd=7 3 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 15 client.40895.objecter _session_command_op_assign 7 16 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 4 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 10 client.40895.objecter _send_command 16 2024-03-11T21:30:10.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.086+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbbe01bf070 con 0x7fbbe01bc740 2024-03-11T21:30:10.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdcff9700 1 -- 172.21.15.143:0/3662532915 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fbbe01bf070 con 0x7fbbe01bc740 2024-03-11T21:30:10.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdcff9700 10 client.40895.objecter ms_dispatch 0x7fbbe0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdcff9700 10 client.40895.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdcff9700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 4 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdcff9700 15 client.40895.objecter _session_command_op_remove 7 16 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 librados: watch_flush enter 2024-03-11T21:30:10.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 librados: watch_flush exit 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.0 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fbbe0062c40 msgr2=0x7fbbe0107ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fbbe0062c40 0x7fbbe0107ab0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0196660 osd=0 1 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.1 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7fbbe002ce80 msgr2=0x7fbbe002d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7fbbe002ce80 0x7fbbe002d260 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002dd40 osd=1 1 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.2 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fbbe0108d20 msgr2=0x7fbbe01a9900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fbbe0108d20 0x7fbbe01a9900 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe002cd00 osd=2 1 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.3 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fbbe01ab0b0 msgr2=0x7fbbe01ab530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fbbe01ab0b0 0x7fbbe01ab530 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe0108590 osd=3 1 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.4 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdffff700 1 -- 172.21.15.143:0/3662532915 reap_dead start 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fbbe01a6d80 msgr2=0x7fbbe01b0bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fbbe01a6d80 0x7fbbe01b0bf0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6c10 osd=4 1 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdffff700 1 -- 172.21.15.143:0/3662532915 reap_dead start 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.5 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fbbe01b2ec0 msgr2=0x7fbbe01b3320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fbbe01b2ec0 0x7fbbe01b3320 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01a6a20 osd=5 1 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.6 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fbbe01b1690 msgr2=0x7fbbe01b1af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fbbe01b1690 0x7fbbe01b1af0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b24e0 osd=6 1 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter close_session for osd.7 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fbbe01bc740 msgr2=0x7fbbe01beb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fbbe01bc740 0x7fbbe01beb30 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:10.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter put_session s=0x7fbbe01b2030 osd=7 1 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 20 client.40895.objecter shutdown clearing up homeless session... 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 10 client.40895.objecter successfully canceled tick 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbbc806f820 msgr2=0x7fbbc8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbbc806f820 0x7fbbc8071ce0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000be00 tx=0x7fbbd000d040 comp rx=0 tx=0).stop 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 msgr2=0x7fbbe019bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 --2- 172.21.15.143:0/3662532915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbe0101700 0x7fbbe019bb40 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400d980 tx=0x7fbbd400dc90 comp rx=0 tx=0).stop 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbdffff700 1 -- 172.21.15.143:0/3662532915 reap_dead start 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 shutdown_connections 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 >> 172.21.15.143:0/3662532915 conn(0x7fbbe0076d80 msgr2=0x7fbbe00ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 shutdown_connections 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 -- 172.21.15.143:0/3662532915 wait complete. 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:10.090+0000 7fbbe5c80700 1 librados: shutdown 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:10.069 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:10.070 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:10.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:10.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:10.385 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:10.409 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:10.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:10.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:10.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:10.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: cluster 2024-03-11T21:30:09.252639+0000 mgr.y (mgr.14550) 287 : cluster 0 pgmap v299: 372 pgs: 32 creating+peering, 340 active+clean; 596 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 56 KiB/s wr, 52 op/s 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: cluster 2024-03-11T21:30:09.613295+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: audit 2024-03-11T21:30:09.618999+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: audit 2024-03-11T21:30:09.620521+0000 mon.a (mon.0) 1415 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: audit 2024-03-11T21:30:09.895625+0000 mon.c (mon.2) 329 : audit 0 from='client.? 172.21.15.143:0/3662532915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: cluster 2024-03-11T21:30:10.045969+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: cluster 2024-03-11T21:30:10.046830+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:10.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:10 smithi203 bash[23654]: audit 2024-03-11T21:30:10.442885+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:10.793 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:10.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:10.918 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:10.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: cluster 2024-03-11T21:30:09.252639+0000 mgr.y (mgr.14550) 287 : cluster 0 pgmap v299: 372 pgs: 32 creating+peering, 340 active+clean; 596 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 56 KiB/s wr, 52 op/s 2024-03-11T21:30:10.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: cluster 2024-03-11T21:30:09.613295+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:30:10.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: audit 2024-03-11T21:30:09.618999+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:10.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: audit 2024-03-11T21:30:09.620521+0000 mon.a (mon.0) 1415 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: audit 2024-03-11T21:30:09.895625+0000 mon.c (mon.2) 329 : audit 0 from='client.? 172.21.15.143:0/3662532915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: cluster 2024-03-11T21:30:10.045969+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: cluster 2024-03-11T21:30:10.046830+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[21512]: audit 2024-03-11T21:30:10.442885+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: cluster 2024-03-11T21:30:09.252639+0000 mgr.y (mgr.14550) 287 : cluster 0 pgmap v299: 372 pgs: 32 creating+peering, 340 active+clean; 596 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 56 KiB/s wr, 52 op/s 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: cluster 2024-03-11T21:30:09.613295+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: audit 2024-03-11T21:30:09.618999+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: audit 2024-03-11T21:30:09.620521+0000 mon.a (mon.0) 1415 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: audit 2024-03-11T21:30:09.895625+0000 mon.c (mon.2) 329 : audit 0 from='client.? 172.21.15.143:0/3662532915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:10.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: cluster 2024-03-11T21:30:10.045969+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:10.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: cluster 2024-03-11T21:30:10.046830+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:10.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:10 smithi143 bash[29284]: audit 2024-03-11T21:30:10.442885+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:11.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:11.047 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:11.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:11.184 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:11.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.314 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:11.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:11.445 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:11.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:11.676 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:11.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:11.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:11.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: cluster 2024-03-11T21:30:09.780751+0000 osd.4 (osd.4) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: cluster 2024-03-11T21:30:09.782523+0000 osd.4 (osd.4) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.612291+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.612534+0000 mon.a (mon.0) 1417 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: cluster 2024-03-11T21:30:10.623659+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.625535+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.143:0/3901840629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:11.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.625587+0000 mon.a (mon.0) 1419 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.625770+0000 mon.c (mon.2) 332 : audit 1 from='client.? 172.21.15.143:0/3248257188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.627884+0000 mon.b (mon.1) 85 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.638464+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.638787+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.640186+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.900077+0000 mon.c (mon.2) 333 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:10.900532+0000 mon.a (mon.0) 1423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: cluster 2024-03-11T21:30:11.193871+0000 mon.a (mon.0) 1424 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.201254+0000 mon.a (mon.0) 1425 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.201442+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.201704+0000 mon.a (mon.0) 1427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.201942+0000 mon.a (mon.0) 1428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.202154+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:11.963 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.205448+0000 mon.b (mon.1) 86 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:11.964 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: cluster 2024-03-11T21:30:11.211624+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:30:11.964 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.217381+0000 mon.a (mon.0) 1431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:11.964 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.217689+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:11.964 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:11 smithi203 bash[23654]: audit 2024-03-11T21:30:11.220853+0000 mon.a (mon.0) 1432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:12.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:12.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: cluster 2024-03-11T21:30:09.780751+0000 osd.4 (osd.4) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: cluster 2024-03-11T21:30:09.782523+0000 osd.4 (osd.4) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.612291+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.612534+0000 mon.a (mon.0) 1417 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: cluster 2024-03-11T21:30:10.623659+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.625535+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.143:0/3901840629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.625587+0000 mon.a (mon.0) 1419 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.625770+0000 mon.c (mon.2) 332 : audit 1 from='client.? 172.21.15.143:0/3248257188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.627884+0000 mon.b (mon.1) 85 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.638464+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.638787+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.640186+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.900077+0000 mon.c (mon.2) 333 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:10.900532+0000 mon.a (mon.0) 1423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: cluster 2024-03-11T21:30:11.193871+0000 mon.a (mon.0) 1424 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: cluster 2024-03-11T21:30:09.780751+0000 osd.4 (osd.4) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: cluster 2024-03-11T21:30:09.782523+0000 osd.4 (osd.4) 144 : cluster 0 purged_snaps scrub ok 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.612291+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.612534+0000 mon.a (mon.0) 1417 : audit 1 from='client.? 172.21.15.143:0/2190719162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi143-71594-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: cluster 2024-03-11T21:30:10.623659+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.625535+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.143:0/3901840629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.625587+0000 mon.a (mon.0) 1419 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]: dispatch 2024-03-11T21:30:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.625770+0000 mon.c (mon.2) 332 : audit 1 from='client.? 172.21.15.143:0/3248257188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.627884+0000 mon.b (mon.1) 85 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.638464+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.638787+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.640186+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.900077+0000 mon.c (mon.2) 333 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:10.900532+0000 mon.a (mon.0) 1423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: cluster 2024-03-11T21:30:11.193871+0000 mon.a (mon.0) 1424 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.201254+0000 mon.a (mon.0) 1425 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.201442+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.201704+0000 mon.a (mon.0) 1427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.201942+0000 mon.a (mon.0) 1428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.202154+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.205448+0000 mon.b (mon.1) 86 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: cluster 2024-03-11T21:30:11.211624+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.217381+0000 mon.a (mon.0) 1431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.217689+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[29284]: audit 2024-03-11T21:30:11.220853+0000 mon.a (mon.0) 1432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.201254+0000 mon.a (mon.0) 1425 : audit 1 from='client.? 172.21.15.143:0/1374067002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71756-24"}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.201442+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71838-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.201704+0000 mon.a (mon.0) 1427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi143-71604-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.201942+0000 mon.a (mon.0) 1428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:12.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.202154+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.205448+0000 mon.b (mon.1) 86 : audit 1 from='client.? 172.21.15.143:0/3347152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: cluster 2024-03-11T21:30:11.211624+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:30:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.217381+0000 mon.a (mon.0) 1431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]: dispatch 2024-03-11T21:30:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.217689+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:11 smithi143 bash[21512]: audit 2024-03-11T21:30:11.220853+0000 mon.a (mon.0) 1432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:12.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:12.209 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (7678 ms) 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3071 ms) 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1885 ms) 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (3209 ms) 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (15843 ms total) 2024-03-11T21:30:12.210 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (7889 ms) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6319 ms) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (10464 ms) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (24672 ms total) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2862 ms) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2418 ms) 2024-03-11T21:30:12.211 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2462 ms) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2389 ms) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (10131 ms total) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4441 ms) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4116 ms) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8557 ms total) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (96549 ms total) 2024-03-11T21:30:12.212 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-11T21:30:12.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:12.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:12.365 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:12.435 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:12.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:12.490 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:12.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:12.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:12.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:12.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:12.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:30:12.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:12.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:12.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:12.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: cluster 2024-03-11T21:30:11.253679+0000 mgr.y (mgr.14550) 288 : cluster 0 pgmap v303: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 46 op/s 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:11.443745+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.207167+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.207460+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.218763+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: cluster 2024-03-11T21:30:12.226436+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.227797+0000 mon.b (mon.1) 87 : audit 1 from='client.? 172.21.15.143:0/1725112207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.230623+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.143:0/3281351558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.236205+0000 mon.a (mon.0) 1436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.236811+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.246270+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:12 smithi203 bash[23654]: audit 2024-03-11T21:30:12.444848+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.045 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: cluster 2024-03-11T21:30:11.253679+0000 mgr.y (mgr.14550) 288 : cluster 0 pgmap v303: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 46 op/s 2024-03-11T21:30:13.046 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:11.443745+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.046 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.207167+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:13.046 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.207460+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:13.046 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.218763+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.046 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: cluster 2024-03-11T21:30:12.226436+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.227797+0000 mon.b (mon.1) 87 : audit 1 from='client.? 172.21.15.143:0/1725112207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.230623+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.143:0/3281351558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.236205+0000 mon.a (mon.0) 1436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.236811+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.246270+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[29284]: audit 2024-03-11T21:30:12.444848+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: cluster 2024-03-11T21:30:11.253679+0000 mgr.y (mgr.14550) 288 : cluster 0 pgmap v303: 356 pgs: 64 unknown, 292 active+clean; 586 KiB data, 461 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 46 op/s 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:11.443745+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.207167+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi143-71818-15"}]': finished 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.207460+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.218763+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: cluster 2024-03-11T21:30:12.226436+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.227797+0000 mon.b (mon.1) 87 : audit 1 from='client.? 172.21.15.143:0/1725112207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.230623+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.143:0/3281351558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.236205+0000 mon.a (mon.0) 1436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.236811+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.246270+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:12 smithi143 bash[21512]: audit 2024-03-11T21:30:12.444848+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:13.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- 172.21.15.143:0/3465676174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854806a6a0 msgr2=0x7f854806aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:13.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 --2- 172.21.15.143:0/3465676174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854806a6a0 0x7f854806aa80 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f8530009990 tx=0x7f8530009ca0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- 172.21.15.143:0/3465676174 shutdown_connections 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 --2- 172.21.15.143:0/3465676174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8548107220 0x7f854810edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 --2- 172.21.15.143:0/3465676174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854806a6a0 0x7f854806aa80 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 --2- 172.21.15.143:0/3465676174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806b050 0x7f8548106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- 172.21.15.143:0/3465676174 >> 172.21.15.143:0/3465676174 conn(0x7f8548077de0 msgr2=0x7f85480781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- 172.21.15.143:0/3465676174 shutdown_connections 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- 172.21.15.143:0/3465676174 wait complete. 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 librados: starting msgr at 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 librados: starting objecter 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 Processor -- start 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 -- start start 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 librados: setting wanted keys 2024-03-11T21:30:13.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.134+0000 7f854ce1e700 1 librados: calling monclient init 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806a6a0 0x7f854819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 0x7f854819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8548111650 con 0x7f854806b050 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85481114d0 con 0x7f854806a6a0 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85481117d0 con 0x7f8548107220 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:59178/0 (socket says 172.21.15.143:59178) 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 -- 172.21.15.143:0/1695830874 learned_addr learned my addr 172.21.15.143:0/1695830874 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8545d9b700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 0x7f854819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854659c700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806a6a0 0x7f854819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 -- 172.21.15.143:0/1695830874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806a6a0 msgr2=0x7f854819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:13.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806a6a0 0x7f854819d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 -- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 msgr2=0x7f854819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:13.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 0x7f854819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:13.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 -- 172.21.15.143:0/1695830874 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8548198180 con 0x7f8548107220 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8545d9b700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 0x7f854819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f8546d9d700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f854000ef30 tx=0x7f854000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 10 client.?.objecter ms_handle_connect 0x7f8548107220 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400092e0 con 0x7f8548107220 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8540004bb0 con 0x7f8548107220 2024-03-11T21:30:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8548198410 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 10 client.40928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85481a4730 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 librados: init done 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540004020 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528005120 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f854001b030 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 0x7f8534071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(211..211 src has 87..211) v4 ==== 10685+0+0 (secure 0 0 0) 0x7f854008e950 con 0x7f8548107220 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854659c700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 0x7f8534071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 10 client.40928.objecter ms_dispatch 0x7f8548001550 osd_map(211..211 src has 87..211) v4 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 3 client.40928.objecter handle_osd_map got epochs [211,211] > 0 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 3 client.40928.objecter handle_osd_map decoding full epoch 211 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 20 client.40928.objecter dump_active .. 0 homeless 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f854659c700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 0x7f8534071ce0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f853000b2e0 tx=0x7f853001a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:13.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.138+0000 7f853f7fe700 10 client.40928.objecter ms_handle_connect 0x7f853406f820 2024-03-11T21:30:13.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.146+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8540090e20 con 0x7f8548107220 2024-03-11T21:30:13.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:13.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:13.257 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:13.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.382 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:13.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:13.414+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f85280056e0 con 0x7f8548107220 2024-03-11T21:30:13.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:13.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:13.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.758 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:13.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:13.880 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:13.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: cluster 2024-03-11T21:30:13.207785+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.209570+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]': finished 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.209638+0000 mon.a (mon.0) 1441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.209712+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: cluster 2024-03-11T21:30:13.220497+0000 mon.a (mon.0) 1443 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.221503+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.143:0/1063109346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.237348+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.417598+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.143:0/1695830874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.417906+0000 mon.a (mon.0) 1445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[29284]: audit 2024-03-11T21:30:13.446034+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:13.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: cluster 2024-03-11T21:30:13.207785+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.209570+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]': finished 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.209638+0000 mon.a (mon.0) 1441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.209712+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: cluster 2024-03-11T21:30:13.220497+0000 mon.a (mon.0) 1443 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.221503+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.143:0/1063109346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.237348+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.417598+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.143:0/1695830874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.417906+0000 mon.a (mon.0) 1445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:13.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:13 smithi143 bash[21512]: audit 2024-03-11T21:30:13.446034+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:14.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: cluster 2024-03-11T21:30:13.207785+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.209570+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-13", "mode": "writeback"}]': finished 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.209638+0000 mon.a (mon.0) 1441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi143-71756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.209712+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi143-71594-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: cluster 2024-03-11T21:30:13.220497+0000 mon.a (mon.0) 1443 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.221503+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.143:0/1063109346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.237348+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:14.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.417598+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.143:0/1695830874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.417906+0000 mon.a (mon.0) 1445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:13 smithi203 bash[23654]: audit 2024-03-11T21:30:13.446034+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:14.138 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:14.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.222+0000 7f853f7fe700 1 -- 172.21.15.143:0/1695830874 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v213) v1 ==== 91+0+0 (secure 0 0 0) 0x7f8540022020 con 0x7f8548107220 2024-03-11T21:30:14.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 10 librados: watch_flush enter 2024-03-11T21:30:14.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 10 librados: watch_flush exit 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 20 client.40928.objecter shutdown clearing up homeless session... 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 10 client.40928.objecter successfully canceled tick 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 msgr2=0x7f8534071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 0x7f8534071ce0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f853000b2e0 tx=0x7f853001a040 comp rx=0 tx=0).stop 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 msgr2=0x7f8548197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:14.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.226+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f854000ef30 tx=0x7f854000c3a0 comp rx=0 tx=0).stop 2024-03-11T21:30:14.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 shutdown_connections 2024-03-11T21:30:14.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f853406f820 0x7f8534071ce0 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:14.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8548107220 0x7f8548197a20 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:14.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f854806a6a0 0x7f854819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:14.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 --2- 172.21.15.143:0/1695830874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854806b050 0x7f854819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:14.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.234+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 >> 172.21.15.143:0/1695830874 conn(0x7f8548077de0 msgr2=0x7f85480feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:14.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.242+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 shutdown_connections 2024-03-11T21:30:14.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.242+0000 7f854ce1e700 1 -- 172.21.15.143:0/1695830874 wait complete. 2024-03-11T21:30:14.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:14.250+0000 7f854ce1e700 1 librados: shutdown 2024-03-11T21:30:14.231 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:30:14.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:14.287 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:14.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:14.410 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:14.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:14.541 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:14.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:14.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:14.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:14.802 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:14.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:14.933 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:15.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.060 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:15.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: cluster 2024-03-11T21:30:13.255855+0000 mgr.y (mgr.14550) 289 : cluster 0 pgmap v306: 420 pgs: 58 creating+peering, 42 unknown, 320 active+clean; 589 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 60 KiB/s wr, 68 op/s 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.210535+0000 mon.a (mon.0) 1446 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: audit 2024-03-11T21:30:14.218395+0000 mon.a (mon.0) 1447 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: audit 2024-03-11T21:30:14.218521+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.225884+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:30:15.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:15 smithi203 bash[23654]: audit 2024-03-11T21:30:14.319023+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:15.187 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:15.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:15.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:15.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: cluster 2024-03-11T21:30:13.255855+0000 mgr.y (mgr.14550) 289 : cluster 0 pgmap v306: 420 pgs: 58 creating+peering, 42 unknown, 320 active+clean; 589 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 60 KiB/s wr, 68 op/s 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.210535+0000 mon.a (mon.0) 1446 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: audit 2024-03-11T21:30:14.218395+0000 mon.a (mon.0) 1447 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: audit 2024-03-11T21:30:14.218521+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.225884+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[21512]: audit 2024-03-11T21:30:14.319023+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: cluster 2024-03-11T21:30:13.255855+0000 mgr.y (mgr.14550) 289 : cluster 0 pgmap v306: 420 pgs: 58 creating+peering, 42 unknown, 320 active+clean; 589 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 60 KiB/s wr, 68 op/s 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.210535+0000 mon.a (mon.0) 1446 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:15.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: audit 2024-03-11T21:30:14.218395+0000 mon.a (mon.0) 1447 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi143-71604-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:15.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: audit 2024-03-11T21:30:14.218521+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:15.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.225884+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:30:15.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:15 smithi143 bash[29284]: audit 2024-03-11T21:30:14.319023+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:15.430 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:15.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.557 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:15.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:15.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:15.809 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:15.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.936 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:16.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:16.064 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:16.168 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:16.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.402795+0000 osd.6 (osd.6) 105 : cluster 0 2.1a scrub starts 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.403844+0000 osd.6 (osd.6) 106 : cluster 0 2.1a scrub ok 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.420446+0000 mon.c (mon.2) 343 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.420689+0000 mgr.y (mgr.14550) 290 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.421195+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.421329+0000 mgr.y (mgr.14550) 291 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.421741+0000 mon.c (mon.2) 345 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.421873+0000 mgr.y (mgr.14550) 292 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.422242+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.422361+0000 mgr.y (mgr.14550) 293 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.422726+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.422846+0000 mgr.y (mgr.14550) 294 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.423210+0000 mon.c (mon.2) 348 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.423326+0000 mgr.y (mgr.14550) 295 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.423688+0000 mon.c (mon.2) 349 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.423804+0000 mgr.y (mgr.14550) 296 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.424205+0000 mon.c (mon.2) 350 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.424321+0000 mgr.y (mgr.14550) 297 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.424673+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.424785+0000 mgr.y (mgr.14550) 298 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.425234+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.425367+0000 mgr.y (mgr.14550) 299 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:14.446793+0000 mon.c (mon.2) 353 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.230 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.512903+0000 osd.7 (osd.7) 193 : cluster 0 173.4 scrub starts 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.517157+0000 osd.7 (osd.7) 194 : cluster 0 173.4 scrub ok 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.689729+0000 osd.2 (osd.2) 113 : cluster 0 173.8 scrub starts 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.692671+0000 osd.2 (osd.2) 114 : cluster 0 173.8 scrub ok 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.941400+0000 osd.3 (osd.3) 165 : cluster 0 173.6 scrub starts 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:14.948449+0000 osd.3 (osd.3) 166 : cluster 0 173.6 scrub ok 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.007896+0000 osd.0 (osd.0) 149 : cluster 0 173.9 scrub starts 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.011459+0000 osd.0 (osd.0) 150 : cluster 0 173.9 scrub ok 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.060639+0000 osd.4 (osd.4) 145 : cluster 0 173.1 scrub starts 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.064282+0000 osd.4 (osd.4) 146 : cluster 0 173.1 scrub ok 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.180853+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.182933+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.183400+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.184285+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.184873+0000 mon.a (mon.0) 1452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:16.314 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:16.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:30:16.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.222411+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.231538+0000 mon.c (mon.2) 356 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.256616+0000 mgr.y (mgr.14550) 300 : cluster 0 pgmap v309: 356 pgs: 64 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 75 KiB/s wr, 78 op/s 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.260322+0000 mon.a (mon.0) 1454 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.261819+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.143:0/1900498068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.262263+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.266867+0000 mon.a (mon.0) 1456 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.268690+0000 mon.a (mon.0) 1457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: audit 2024-03-11T21:30:15.447537+0000 mon.c (mon.2) 358 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.720310+0000 osd.2 (osd.2) 115 : cluster 0 6.14 scrub starts 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.721927+0000 osd.2 (osd.2) 116 : cluster 0 6.14 scrub ok 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.402795+0000 osd.6 (osd.6) 105 : cluster 0 2.1a scrub starts 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.403844+0000 osd.6 (osd.6) 106 : cluster 0 2.1a scrub ok 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.420446+0000 mon.c (mon.2) 343 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.420689+0000 mgr.y (mgr.14550) 290 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.421195+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.421329+0000 mgr.y (mgr.14550) 291 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.421741+0000 mon.c (mon.2) 345 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.421873+0000 mgr.y (mgr.14550) 292 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.422242+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.422361+0000 mgr.y (mgr.14550) 293 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.422726+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.422846+0000 mgr.y (mgr.14550) 294 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.423210+0000 mon.c (mon.2) 348 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.423326+0000 mgr.y (mgr.14550) 295 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.423688+0000 mon.c (mon.2) 349 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.423804+0000 mgr.y (mgr.14550) 296 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.424205+0000 mon.c (mon.2) 350 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.424321+0000 mgr.y (mgr.14550) 297 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.424673+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.424785+0000 mgr.y (mgr.14550) 298 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.425234+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.425367+0000 mgr.y (mgr.14550) 299 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:14.446793+0000 mon.c (mon.2) 353 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.512903+0000 osd.7 (osd.7) 193 : cluster 0 173.4 scrub starts 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.517157+0000 osd.7 (osd.7) 194 : cluster 0 173.4 scrub ok 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.689729+0000 osd.2 (osd.2) 113 : cluster 0 173.8 scrub starts 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.692671+0000 osd.2 (osd.2) 114 : cluster 0 173.8 scrub ok 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.941400+0000 osd.3 (osd.3) 165 : cluster 0 173.6 scrub starts 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:14.948449+0000 osd.3 (osd.3) 166 : cluster 0 173.6 scrub ok 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.007896+0000 osd.0 (osd.0) 149 : cluster 0 173.9 scrub starts 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.011459+0000 osd.0 (osd.0) 150 : cluster 0 173.9 scrub ok 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.060639+0000 osd.4 (osd.4) 145 : cluster 0 173.1 scrub starts 2024-03-11T21:30:16.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.064282+0000 osd.4 (osd.4) 146 : cluster 0 173.1 scrub ok 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.180853+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.182933+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.183400+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.184285+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.184873+0000 mon.a (mon.0) 1452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.222411+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.231538+0000 mon.c (mon.2) 356 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.256616+0000 mgr.y (mgr.14550) 300 : cluster 0 pgmap v309: 356 pgs: 64 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 75 KiB/s wr, 78 op/s 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.260322+0000 mon.a (mon.0) 1454 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.261819+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.143:0/1900498068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.262263+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.266867+0000 mon.a (mon.0) 1456 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.268690+0000 mon.a (mon.0) 1457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: audit 2024-03-11T21:30:15.447537+0000 mon.c (mon.2) 358 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.720310+0000 osd.2 (osd.2) 115 : cluster 0 6.14 scrub starts 2024-03-11T21:30:16.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:16 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.721927+0000 osd.2 (osd.2) 116 : cluster 0 6.14 scrub ok 2024-03-11T21:30:16.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:16.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.402795+0000 osd.6 (osd.6) 105 : cluster 0 2.1a scrub starts 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.403844+0000 osd.6 (osd.6) 106 : cluster 0 2.1a scrub ok 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.420446+0000 mon.c (mon.2) 343 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.420689+0000 mgr.y (mgr.14550) 290 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.421195+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.421329+0000 mgr.y (mgr.14550) 291 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.421741+0000 mon.c (mon.2) 345 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.421873+0000 mgr.y (mgr.14550) 292 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.422242+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.422361+0000 mgr.y (mgr.14550) 293 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.422726+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.422846+0000 mgr.y (mgr.14550) 294 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.423210+0000 mon.c (mon.2) 348 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.423326+0000 mgr.y (mgr.14550) 295 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.423688+0000 mon.c (mon.2) 349 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.423804+0000 mgr.y (mgr.14550) 296 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.424205+0000 mon.c (mon.2) 350 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.424321+0000 mgr.y (mgr.14550) 297 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.424673+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.424785+0000 mgr.y (mgr.14550) 298 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.425234+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.425367+0000 mgr.y (mgr.14550) 299 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:14.446793+0000 mon.c (mon.2) 353 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.512903+0000 osd.7 (osd.7) 193 : cluster 0 173.4 scrub starts 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.517157+0000 osd.7 (osd.7) 194 : cluster 0 173.4 scrub ok 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.689729+0000 osd.2 (osd.2) 113 : cluster 0 173.8 scrub starts 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.692671+0000 osd.2 (osd.2) 114 : cluster 0 173.8 scrub ok 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.941400+0000 osd.3 (osd.3) 165 : cluster 0 173.6 scrub starts 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:14.948449+0000 osd.3 (osd.3) 166 : cluster 0 173.6 scrub ok 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.007896+0000 osd.0 (osd.0) 149 : cluster 0 173.9 scrub starts 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.011459+0000 osd.0 (osd.0) 150 : cluster 0 173.9 scrub ok 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.060639+0000 osd.4 (osd.4) 145 : cluster 0 173.1 scrub starts 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.064282+0000 osd.4 (osd.4) 146 : cluster 0 173.1 scrub ok 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.180853+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.182933+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.183400+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.184285+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.184873+0000 mon.a (mon.0) 1452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.222411+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.231538+0000 mon.c (mon.2) 356 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.256616+0000 mgr.y (mgr.14550) 300 : cluster 0 pgmap v309: 356 pgs: 64 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 75 KiB/s wr, 78 op/s 2024-03-11T21:30:16.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.260322+0000 mon.a (mon.0) 1454 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.261819+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.143:0/1900498068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.262263+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.266867+0000 mon.a (mon.0) 1456 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.268690+0000 mon.a (mon.0) 1457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: audit 2024-03-11T21:30:15.447537+0000 mon.c (mon.2) 358 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.720310+0000 osd.2 (osd.2) 115 : cluster 0 6.14 scrub starts 2024-03-11T21:30:16.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:16 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.721927+0000 osd.2 (osd.2) 116 : cluster 0 6.14 scrub ok 2024-03-11T21:30:16.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:16.563 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:16.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:16.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- 172.21.15.143:0/1769133299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102a00 msgr2=0x7f5084106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- 172.21.15.143:0/1769133299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102a00 0x7f5084106a50 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f50740099c0 tx=0x7f5074009cd0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- 172.21.15.143:0/1769133299 shutdown_connections 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- 172.21.15.143:0/1769133299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084102050 0x7f5084102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- 172.21.15.143:0/1769133299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084107130 0x7f508410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- 172.21.15.143:0/1769133299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102a00 0x7f5084106a50 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- 172.21.15.143:0/1769133299 >> 172.21.15.143:0/1769133299 conn(0x7f50840791b0 msgr2=0x7f50840795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:16.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- 172.21.15.143:0/1769133299 shutdown_connections 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- 172.21.15.143:0/1769133299 wait complete. 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 librados: starting msgr at 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 librados: starting objecter 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 Processor -- start 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- start start 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 librados: setting wanted keys 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 librados: calling monclient init 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084102a00 0x7f508419dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50841115f0 con 0x7f5084107130 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5084111470 con 0x7f5084102a00 2024-03-11T21:30:16.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508bec3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5084111770 con 0x7f5084102050 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f5089c5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f5089c5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:59286/0 (socket says 172.21.15.143:59286) 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f5089c5f700 1 -- 172.21.15.143:0/2330446057 learned_addr learned my addr 172.21.15.143:0/2330446057 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41616/0 (socket says 172.21.15.143:41616) 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508945e700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084102a00 0x7f508419dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 -- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 msgr2=0x7f508419d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 -- 172.21.15.143:0/2330446057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084102a00 msgr2=0x7f508419dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084102a00 0x7f508419dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:16.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 -- 172.21.15.143:0/2330446057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084198150 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f5089c5f700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f508a460700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f507c007f10 tx=0x7f507c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f507affd700 10 client.?.objecter ms_handle_connect 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.790+0000 7f507affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c004db0 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5084198440 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f507c009e20 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f508bec3700 10 client.29787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c015400 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50841a4700 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f508bec3700 1 librados: init done 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508404faa0 con 0x7f5084107130 2024-03-11T21:30:16.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f507c00db20 con 0x7f5084107130 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 0x7f5070071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(215..215 src has 87..215) v4 ==== 11177+0+0 (secure 0 0 0) 0x7f507c08f1a0 con 0x7f5084107130 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 10 client.29787.objecter ms_dispatch 0x7f5084001550 osd_map(215..215 src has 87..215) v4 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 3 client.29787.objecter handle_osd_map got epochs [215,215] > 0 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 3 client.29787.objecter handle_osd_map decoding full epoch 215 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f5089c5f700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 0x7f5070071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 20 client.29787.objecter dump_active .. 0 homeless 2024-03-11T21:30:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f5089c5f700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 0x7f5070071ce0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f5084198b30 tx=0x7f5080009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.794+0000 7f507affd700 10 client.29787.objecter ms_handle_connect 0x7f507006f820 2024-03-11T21:30:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:16.798+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f507c010070 con 0x7f5084107130 2024-03-11T21:30:16.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:16.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:16.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:16.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:16.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.014+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f50841a49f0 con 0x7f5084107130 2024-03-11T21:30:17.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:17.080 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:17.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:17.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.226+0000 7f507affd700 1 -- 172.21.15.143:0/2330446057 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v216) v1 ==== 113+0+0 (secure 0 0 0) 0x7f507c05b860 con 0x7f5084107130 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 10 librados: watch_flush enter 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 10 librados: watch_flush exit 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 20 client.29787.objecter shutdown clearing up homeless session... 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 10 client.29787.objecter successfully canceled tick 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 msgr2=0x7f5070071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 0x7f5070071ce0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f5084198b30 tx=0x7f5080009260 comp rx=0 tx=0).stop 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 msgr2=0x7f50841979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.234+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f507c007f10 tx=0x7f507c00d3b0 comp rx=0 tx=0).stop 2024-03-11T21:30:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 shutdown_connections 2024-03-11T21:30:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f507006f820 0x7f5070071ce0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5084107130 0x7f50841979f0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5084102050 0x7f508419d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 --2- 172.21.15.143:0/2330446057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5084102a00 0x7f508419dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.238+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 >> 172.21.15.143:0/2330446057 conn(0x7f50840791b0 msgr2=0x7f50840ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.246+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 shutdown_connections 2024-03-11T21:30:17.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.246+0000 7f508bec3700 1 -- 172.21.15.143:0/2330446057 wait complete. 2024-03-11T21:30:17.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:17.258+0000 7f508bec3700 1 librados: shutdown 2024-03-11T21:30:17.233 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:17.237 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:30:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.533638+0000 osd.7 (osd.7) 195 : cluster 0 173.2 scrub starts 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:15.535813+0000 osd.7 (osd.7) 196 : cluster 0 173.2 scrub ok 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.052816+0000 osd.4 (osd.4) 147 : cluster 0 173.0 scrub starts 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.058287+0000 osd.4 (osd.4) 148 : cluster 0 173.0 scrub ok 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.223669+0000 mon.a (mon.0) 1458 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:16.225051+0000 mon.a (mon.0) 1459 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:16.225151+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.270 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:16.448452+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:17.271 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.670362+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:30:17.271 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:16.671413+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.143:0/608065025' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.271 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:16.680412+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.271 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[21512]: audit 2024-03-11T21:30:17.016499+0000 mon.a (mon.0) 1463 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.533638+0000 osd.7 (osd.7) 195 : cluster 0 173.2 scrub starts 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:15.535813+0000 osd.7 (osd.7) 196 : cluster 0 173.2 scrub ok 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.052816+0000 osd.4 (osd.4) 147 : cluster 0 173.0 scrub starts 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.058287+0000 osd.4 (osd.4) 148 : cluster 0 173.0 scrub ok 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.223669+0000 mon.a (mon.0) 1458 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:16.225051+0000 mon.a (mon.0) 1459 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:16.225151+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:16.448452+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.670362+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:30:17.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:16.671413+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.143:0/608065025' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:16.680412+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:17 smithi143 bash[29284]: audit 2024-03-11T21:30:17.016499+0000 mon.a (mon.0) 1463 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:30:17.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:17.358 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:17.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:17.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:17.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.533638+0000 osd.7 (osd.7) 195 : cluster 0 173.2 scrub starts 2024-03-11T21:30:17.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:15.535813+0000 osd.7 (osd.7) 196 : cluster 0 173.2 scrub ok 2024-03-11T21:30:17.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.052816+0000 osd.4 (osd.4) 147 : cluster 0 173.0 scrub starts 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.058287+0000 osd.4 (osd.4) 148 : cluster 0 173.0 scrub ok 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.223669+0000 mon.a (mon.0) 1458 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:16.225051+0000 mon.a (mon.0) 1459 : audit 1 from='client.? 172.21.15.143:0/1931823432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi143-71594-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:16.225151+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi143-71756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:16.448452+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.670362+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:16.671413+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.143:0/608065025' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:16.680412+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:17.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:17 smithi203 bash[23654]: audit 2024-03-11T21:30:17.016499+0000 mon.a (mon.0) 1463 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:30:17.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:17.617 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:17.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:17.744 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:17.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:17.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:17.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:18.001 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:18.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:18.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:18.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:18.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:18.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:18.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.527682+0000 osd.7 (osd.7) 197 : cluster 0 173.7 scrub starts 2024-03-11T21:30:18.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.532652+0000 osd.7 (osd.7) 198 : cluster 0 173.7 scrub ok 2024-03-11T21:30:18.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: audit 2024-03-11T21:30:17.229528+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: audit 2024-03-11T21:30:17.229748+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: audit 2024-03-11T21:30:17.229896+0000 mon.a (mon.0) 1466 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.235101+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[21512]: audit 2024-03-11T21:30:17.449524+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.527682+0000 osd.7 (osd.7) 197 : cluster 0 173.7 scrub starts 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.532652+0000 osd.7 (osd.7) 198 : cluster 0 173.7 scrub ok 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: audit 2024-03-11T21:30:17.229528+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: audit 2024-03-11T21:30:17.229748+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: audit 2024-03-11T21:30:17.229896+0000 mon.a (mon.0) 1466 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.235101+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:30:18.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:18 smithi143 bash[29284]: audit 2024-03-11T21:30:17.449524+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:18.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:18.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.527682+0000 osd.7 (osd.7) 197 : cluster 0 173.7 scrub starts 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.532652+0000 osd.7 (osd.7) 198 : cluster 0 173.7 scrub ok 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: audit 2024-03-11T21:30:17.229528+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi143-71838-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: audit 2024-03-11T21:30:17.229748+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: audit 2024-03-11T21:30:17.229896+0000 mon.a (mon.0) 1466 : audit 1 from='client.? 172.21.15.143:0/2330446057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.235101+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:30:18.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:18 smithi203 bash[23654]: audit 2024-03-11T21:30:17.449524+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:18.523 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:18.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:30:18.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:30:18.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:18.653 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:18.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:18.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:18.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:19.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:19.033 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:19.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:19.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:19.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.987032+0000 osd.0 (osd.0) 151 : cluster 0 2.1f scrub starts 2024-03-11T21:30:19.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:16.988894+0000 osd.0 (osd.0) 152 : cluster 0 2.1f scrub ok 2024-03-11T21:30:19.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.070455+0000 osd.5 (osd.5) 165 : cluster 0 6.7 scrub starts 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.072752+0000 osd.5 (osd.5) 166 : cluster 0 6.7 scrub ok 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.080666+0000 osd.4 (osd.4) 149 : cluster 0 173.3 scrub starts 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.089060+0000 osd.4 (osd.4) 150 : cluster 0 173.3 scrub ok 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.263447+0000 mgr.y (mgr.14550) 301 : cluster 0 pgmap v312: 332 pgs: 40 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.559218+0000 osd.7 (osd.7) 199 : cluster 0 173.5 scrub starts 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.561175+0000 osd.7 (osd.7) 200 : cluster 0 173.5 scrub ok 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.904854+0000 osd.3 (osd.3) 167 : cluster 0 4.0 scrub starts 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:17.907156+0000 osd.3 (osd.3) 168 : cluster 0 4.0 scrub ok 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.025447+0000 osd.5 (osd.5) 167 : cluster 0 3.c scrub starts 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.029367+0000 osd.5 (osd.5) 168 : cluster 0 3.c scrub ok 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.243119+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.248050+0000 mon.b (mon.1) 88 : audit 1 from='client.? 172.21.15.143:0/4045493541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.249444+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.143:0/2925604180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.253011+0000 mon.a (mon.0) 1469 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.253357+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.395921+0000 mgr.y (mgr.14550) 302 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: audit 2024-03-11T21:30:18.450593+0000 mon.c (mon.2) 363 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.737244+0000 osd.2 (osd.2) 117 : cluster 0 2.10 scrub starts 2024-03-11T21:30:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:19 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.738295+0000 osd.2 (osd.2) 118 : cluster 0 2.10 scrub ok 2024-03-11T21:30:19.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.987032+0000 osd.0 (osd.0) 151 : cluster 0 2.1f scrub starts 2024-03-11T21:30:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:16.988894+0000 osd.0 (osd.0) 152 : cluster 0 2.1f scrub ok 2024-03-11T21:30:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.070455+0000 osd.5 (osd.5) 165 : cluster 0 6.7 scrub starts 2024-03-11T21:30:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.072752+0000 osd.5 (osd.5) 166 : cluster 0 6.7 scrub ok 2024-03-11T21:30:19.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.080666+0000 osd.4 (osd.4) 149 : cluster 0 173.3 scrub starts 2024-03-11T21:30:19.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.089060+0000 osd.4 (osd.4) 150 : cluster 0 173.3 scrub ok 2024-03-11T21:30:19.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.263447+0000 mgr.y (mgr.14550) 301 : cluster 0 pgmap v312: 332 pgs: 40 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-11T21:30:19.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.559218+0000 osd.7 (osd.7) 199 : cluster 0 173.5 scrub starts 2024-03-11T21:30:19.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.561175+0000 osd.7 (osd.7) 200 : cluster 0 173.5 scrub ok 2024-03-11T21:30:19.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.904854+0000 osd.3 (osd.3) 167 : cluster 0 4.0 scrub starts 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:17.907156+0000 osd.3 (osd.3) 168 : cluster 0 4.0 scrub ok 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.025447+0000 osd.5 (osd.5) 167 : cluster 0 3.c scrub starts 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.029367+0000 osd.5 (osd.5) 168 : cluster 0 3.c scrub ok 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.243119+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.248050+0000 mon.b (mon.1) 88 : audit 1 from='client.? 172.21.15.143:0/4045493541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.249444+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.143:0/2925604180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.253011+0000 mon.a (mon.0) 1469 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.253357+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.395921+0000 mgr.y (mgr.14550) 302 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: audit 2024-03-11T21:30:18.450593+0000 mon.c (mon.2) 363 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.737244+0000 osd.2 (osd.2) 117 : cluster 0 2.10 scrub starts 2024-03-11T21:30:19.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.738295+0000 osd.2 (osd.2) 118 : cluster 0 2.10 scrub ok 2024-03-11T21:30:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.987032+0000 osd.0 (osd.0) 151 : cluster 0 2.1f scrub starts 2024-03-11T21:30:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:16.988894+0000 osd.0 (osd.0) 152 : cluster 0 2.1f scrub ok 2024-03-11T21:30:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.070455+0000 osd.5 (osd.5) 165 : cluster 0 6.7 scrub starts 2024-03-11T21:30:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.072752+0000 osd.5 (osd.5) 166 : cluster 0 6.7 scrub ok 2024-03-11T21:30:19.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.080666+0000 osd.4 (osd.4) 149 : cluster 0 173.3 scrub starts 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.089060+0000 osd.4 (osd.4) 150 : cluster 0 173.3 scrub ok 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.263447+0000 mgr.y (mgr.14550) 301 : cluster 0 pgmap v312: 332 pgs: 40 unknown, 292 active+clean; 590 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.559218+0000 osd.7 (osd.7) 199 : cluster 0 173.5 scrub starts 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.561175+0000 osd.7 (osd.7) 200 : cluster 0 173.5 scrub ok 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.904854+0000 osd.3 (osd.3) 167 : cluster 0 4.0 scrub starts 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:17.907156+0000 osd.3 (osd.3) 168 : cluster 0 4.0 scrub ok 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.025447+0000 osd.5 (osd.5) 167 : cluster 0 3.c scrub starts 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.029367+0000 osd.5 (osd.5) 168 : cluster 0 3.c scrub ok 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.243119+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.248050+0000 mon.b (mon.1) 88 : audit 1 from='client.? 172.21.15.143:0/4045493541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.249444+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.143:0/2925604180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.253011+0000 mon.a (mon.0) 1469 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.253357+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:19.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.395921+0000 mgr.y (mgr.14550) 302 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:19.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: audit 2024-03-11T21:30:18.450593+0000 mon.c (mon.2) 363 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:19.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.737244+0000 osd.2 (osd.2) 117 : cluster 0 2.10 scrub starts 2024-03-11T21:30:19.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:19 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.738295+0000 osd.2 (osd.2) 118 : cluster 0 2.10 scrub ok 2024-03-11T21:30:19.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:19.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:19.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.727 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:19.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:19.858 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:19.961 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:19.990 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:20.032 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:20.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:20.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:20.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.715404+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.716490+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.880077+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:18.881357+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: audit 2024-03-11T21:30:19.242608+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: audit 2024-03-11T21:30:19.242838+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.253987+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.264148+0000 mgr.y (mgr.14550) 303 : cluster 0 pgmap v315: 396 pgs: 13 creating+peering, 91 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: audit 2024-03-11T21:30:19.269781+0000 mon.a (mon.0) 1474 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: audit 2024-03-11T21:30:19.452170+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.561389+0000 osd.7 (osd.7) 201 : cluster 0 5.1a scrub starts 2024-03-11T21:30:20.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.563574+0000 osd.7 (osd.7) 202 : cluster 0 5.1a scrub ok 2024-03-11T21:30:20.281 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:20.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:20.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.715404+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:30:20.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.716490+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.880077+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:18.881357+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: audit 2024-03-11T21:30:19.242608+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: audit 2024-03-11T21:30:19.242838+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.253987+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.264148+0000 mgr.y (mgr.14550) 303 : cluster 0 pgmap v315: 396 pgs: 13 creating+peering, 91 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: audit 2024-03-11T21:30:19.269781+0000 mon.a (mon.0) 1474 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: audit 2024-03-11T21:30:19.452170+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.561389+0000 osd.7 (osd.7) 201 : cluster 0 5.1a scrub starts 2024-03-11T21:30:20.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:20 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.563574+0000 osd.7 (osd.7) 202 : cluster 0 5.1a scrub ok 2024-03-11T21:30:20.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:20.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.715404+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:30:20.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.716490+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:30:20.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.880077+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-03-11T21:30:20.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:18.881357+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-03-11T21:30:20.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: audit 2024-03-11T21:30:19.242608+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71594-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: audit 2024-03-11T21:30:19.242838+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi143-71756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.253987+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.264148+0000 mgr.y (mgr.14550) 303 : cluster 0 pgmap v315: 396 pgs: 13 creating+peering, 91 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: audit 2024-03-11T21:30:19.269781+0000 mon.a (mon.0) 1474 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: audit 2024-03-11T21:30:19.452170+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.561389+0000 osd.7 (osd.7) 201 : cluster 0 5.1a scrub starts 2024-03-11T21:30:20.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:20 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.563574+0000 osd.7 (osd.7) 202 : cluster 0 5.1a scrub ok 2024-03-11T21:30:20.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:20.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:20.669 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:20.772 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:20.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:20.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:21.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:21.060 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:21.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:21.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:21.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:21.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:21.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:21.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.607 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:21.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:21.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.780559+0000 osd.2 (osd.2) 119 : cluster 0 5.b scrub starts 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:19.782224+0000 osd.2 (osd.2) 120 : cluster 0 5.b scrub ok 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.246039+0000 mon.a (mon.0) 1475 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:20.253949+0000 mon.a (mon.0) 1476 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.328669+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.453612+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:20.737174+0000 osd.2 (osd.2) 121 : cluster 0 5.d scrub starts 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:20.740175+0000 osd.2 (osd.2) 122 : cluster 0 5.d scrub ok 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.760285+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.761433+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.761727+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.762435+0000 mon.c (mon.2) 368 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.762755+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.890568+0000 mon.a (mon.0) 1480 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:20.893544+0000 mon.c (mon.2) 369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:21.207758+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:21.212974+0000 mon.c (mon.2) 370 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.780559+0000 osd.2 (osd.2) 119 : cluster 0 5.b scrub starts 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:19.782224+0000 osd.2 (osd.2) 120 : cluster 0 5.b scrub ok 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.246039+0000 mon.a (mon.0) 1475 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:20.253949+0000 mon.a (mon.0) 1476 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:30:21.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.328669+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.453612+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:20.737174+0000 osd.2 (osd.2) 121 : cluster 0 5.d scrub starts 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:20.740175+0000 osd.2 (osd.2) 122 : cluster 0 5.d scrub ok 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.760285+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.761433+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.761727+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.762435+0000 mon.c (mon.2) 368 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.762755+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.890568+0000 mon.a (mon.0) 1480 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:20.893544+0000 mon.c (mon.2) 369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:21.207758+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:21.212974+0000 mon.c (mon.2) 370 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.228717+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:21.229532+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:21.236971+0000 mon.c (mon.2) 371 : audit 1 from='client.? 172.21.15.143:0/2451351906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.854 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[29284]: audit 2024-03-11T21:30:21.237555+0000 mon.a (mon.0) 1484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.228717+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:30:21.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:21.229532+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:21.236971+0000 mon.c (mon.2) 371 : audit 1 from='client.? 172.21.15.143:0/2451351906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:21 smithi143 bash[21512]: audit 2024-03-11T21:30:21.237555+0000 mon.a (mon.0) 1484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:21.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.780559+0000 osd.2 (osd.2) 119 : cluster 0 5.b scrub starts 2024-03-11T21:30:21.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:19.782224+0000 osd.2 (osd.2) 120 : cluster 0 5.b scrub ok 2024-03-11T21:30:21.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.246039+0000 mon.a (mon.0) 1475 : audit 1 from='client.? 172.21.15.143:0/2177063064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi143-71604-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:21.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:20.253949+0000 mon.a (mon.0) 1476 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:30:21.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.328669+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.453612+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:20.737174+0000 osd.2 (osd.2) 121 : cluster 0 5.d scrub starts 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:20.740175+0000 osd.2 (osd.2) 122 : cluster 0 5.d scrub ok 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.760285+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.761433+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.761727+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.762435+0000 mon.c (mon.2) 368 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.762755+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.890568+0000 mon.a (mon.0) 1480 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:20.893544+0000 mon.c (mon.2) 369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:21.207758+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:21.212974+0000 mon.c (mon.2) 370 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.228717+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:30:21.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:21.229532+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:21.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:21.236971+0000 mon.c (mon.2) 371 : audit 1 from='client.? 172.21.15.143:0/2451351906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:21 smithi203 bash[23654]: audit 2024-03-11T21:30:21.237555+0000 mon.a (mon.0) 1484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:21.989 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:22.068 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:22.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:22.102 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:22.115 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:22.218 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.246 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:22.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:22.378 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:22.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:22.508 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:22.612 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:22.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.766 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:22.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.264642+0000 mgr.y (mgr.14550) 304 : cluster 0 pgmap v318: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: audit 2024-03-11T21:30:21.455070+0000 mon.c (mon.2) 372 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.702884+0000 osd.2 (osd.2) 123 : cluster 0 6.f scrub starts 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.703882+0000 osd.2 (osd.2) 124 : cluster 0 6.f scrub ok 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.707472+0000 osd.1 (osd.1) 147 : cluster 0 9.14 scrub starts 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:21.708536+0000 osd.1 (osd.1) 148 : cluster 0 9.14 scrub ok 2024-03-11T21:30:22.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: audit 2024-03-11T21:30:22.218222+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: audit 2024-03-11T21:30:22.456683+0000 mon.c (mon.2) 373 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: cluster 2024-03-11T21:30:22.657080+0000 mon.a (mon.0) 1486 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[29284]: audit 2024-03-11T21:30:22.661474+0000 mon.a (mon.0) 1487 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.264642+0000 mgr.y (mgr.14550) 304 : cluster 0 pgmap v318: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: audit 2024-03-11T21:30:21.455070+0000 mon.c (mon.2) 372 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.702884+0000 osd.2 (osd.2) 123 : cluster 0 6.f scrub starts 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.703882+0000 osd.2 (osd.2) 124 : cluster 0 6.f scrub ok 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.707472+0000 osd.1 (osd.1) 147 : cluster 0 9.14 scrub starts 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:21.708536+0000 osd.1 (osd.1) 148 : cluster 0 9.14 scrub ok 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: audit 2024-03-11T21:30:22.218222+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: audit 2024-03-11T21:30:22.456683+0000 mon.c (mon.2) 373 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:22.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: cluster 2024-03-11T21:30:22.657080+0000 mon.a (mon.0) 1486 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:30:22.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:22 smithi143 bash[21512]: audit 2024-03-11T21:30:22.661474+0000 mon.a (mon.0) 1487 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:22.891 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:22.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.264642+0000 mgr.y (mgr.14550) 304 : cluster 0 pgmap v318: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 74 op/s 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: audit 2024-03-11T21:30:21.455070+0000 mon.c (mon.2) 372 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.702884+0000 osd.2 (osd.2) 123 : cluster 0 6.f scrub starts 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.703882+0000 osd.2 (osd.2) 124 : cluster 0 6.f scrub ok 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.707472+0000 osd.1 (osd.1) 147 : cluster 0 9.14 scrub starts 2024-03-11T21:30:23.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:21.708536+0000 osd.1 (osd.1) 148 : cluster 0 9.14 scrub ok 2024-03-11T21:30:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: audit 2024-03-11T21:30:22.218222+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi143-71594-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: audit 2024-03-11T21:30:22.456683+0000 mon.c (mon.2) 373 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: cluster 2024-03-11T21:30:22.657080+0000 mon.a (mon.0) 1486 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:30:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:22 smithi203 bash[23654]: audit 2024-03-11T21:30:22.661474+0000 mon.a (mon.0) 1487 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:23.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:23.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:23.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:23.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:23.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:23.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:23.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:23.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:23.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:23.783 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:23.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.911 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: cluster 2024-03-11T21:30:22.360765+0000 osd.6 (osd.6) 107 : cluster 0 2.14 scrub starts 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: cluster 2024-03-11T21:30:22.363078+0000 osd.6 (osd.6) 108 : cluster 0 2.14 scrub ok 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: cluster 2024-03-11T21:30:22.715834+0000 osd.2 (osd.2) 125 : cluster 0 5.6 scrub starts 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: cluster 2024-03-11T21:30:22.717519+0000 osd.2 (osd.2) 126 : cluster 0 5.6 scrub ok 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: audit 2024-03-11T21:30:23.222253+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:23.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: audit 2024-03-11T21:30:23.222543+0000 mon.a (mon.0) 1489 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.231784+0000 mon.a (mon.0) 1490 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: cluster 2024-03-11T21:30:22.360765+0000 osd.6 (osd.6) 107 : cluster 0 2.14 scrub starts 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: cluster 2024-03-11T21:30:22.363078+0000 osd.6 (osd.6) 108 : cluster 0 2.14 scrub ok 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: cluster 2024-03-11T21:30:22.715834+0000 osd.2 (osd.2) 125 : cluster 0 5.6 scrub starts 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: cluster 2024-03-11T21:30:22.717519+0000 osd.2 (osd.2) 126 : cluster 0 5.6 scrub ok 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: audit 2024-03-11T21:30:23.222253+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: audit 2024-03-11T21:30:23.222543+0000 mon.a (mon.0) 1489 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:23.913 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:23 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.231784+0000 mon.a (mon.0) 1490 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:30:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:24.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: cluster 2024-03-11T21:30:22.360765+0000 osd.6 (osd.6) 107 : cluster 0 2.14 scrub starts 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: cluster 2024-03-11T21:30:22.363078+0000 osd.6 (osd.6) 108 : cluster 0 2.14 scrub ok 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: cluster 2024-03-11T21:30:22.715834+0000 osd.2 (osd.2) 125 : cluster 0 5.6 scrub starts 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: cluster 2024-03-11T21:30:22.717519+0000 osd.2 (osd.2) 126 : cluster 0 5.6 scrub ok 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: audit 2024-03-11T21:30:23.222253+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi143-71756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: audit 2024-03-11T21:30:23.222543+0000 mon.a (mon.0) 1489 : audit 1 from='client.? 172.21.15.143:0/1438445183' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi143-71604-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:24.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:23 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.231784+0000 mon.a (mon.0) 1490 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:30:24.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:24.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:24.172 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:24.226 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:24.227 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:24.228 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-11T21:30:24.229 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-11T21:30:24.229 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-11T21:30:24.229 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-11T21:30:24.229 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: seed 71838 2024-03-11T21:30:24.229 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (3820 ms) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (5346 ms) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3071 ms) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1887 ms) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (8039 ms) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (22163 ms total) 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-11T21:30:24.230 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (7069 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5293 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8642 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (6708 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2942 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi143-71838-7 2024-03-11T21:30:24.231 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17945 ms) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (48599 ms total) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4670 ms) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4679 ms total) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2987 ms) 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-11T21:30:24.232 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2402 ms) 2024-03-11T21:30:24.233 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-11T21:30:24.233 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1597 ms) 2024-03-11T21:30:24.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.298 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:24.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:24.429 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:24.532 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:24.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:24.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.690 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:24.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:24.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.267938+0000 mgr.y (mgr.14550) 305 : cluster 0 pgmap v321: 340 pgs: 29 creating+peering, 11 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.393691+0000 osd.6 (osd.6) 109 : cluster 0 3.10 scrub starts 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.397302+0000 osd.6 (osd.6) 110 : cluster 0 3.10 scrub ok 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: audit 2024-03-11T21:30:23.458253+0000 mon.c (mon.2) 374 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.713438+0000 osd.2 (osd.2) 127 : cluster 0 39.1f scrub starts 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:23.715433+0000 osd.2 (osd.2) 128 : cluster 0 39.1f scrub ok 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.235864+0000 mon.a (mon.0) 1491 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: audit 2024-03-11T21:30:24.260261+0000 mon.c (mon.2) 375 : audit 1 from='client.? 172.21.15.143:0/3332855013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: audit 2024-03-11T21:30:24.261289+0000 mon.a (mon.0) 1492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[21512]: audit 2024-03-11T21:30:24.459770+0000 mon.c (mon.2) 376 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.802 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.267938+0000 mgr.y (mgr.14550) 305 : cluster 0 pgmap v321: 340 pgs: 29 creating+peering, 11 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.393691+0000 osd.6 (osd.6) 109 : cluster 0 3.10 scrub starts 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.397302+0000 osd.6 (osd.6) 110 : cluster 0 3.10 scrub ok 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: audit 2024-03-11T21:30:23.458253+0000 mon.c (mon.2) 374 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.713438+0000 osd.2 (osd.2) 127 : cluster 0 39.1f scrub starts 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:23.715433+0000 osd.2 (osd.2) 128 : cluster 0 39.1f scrub ok 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.235864+0000 mon.a (mon.0) 1491 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: audit 2024-03-11T21:30:24.260261+0000 mon.c (mon.2) 375 : audit 1 from='client.? 172.21.15.143:0/3332855013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: audit 2024-03-11T21:30:24.261289+0000 mon.a (mon.0) 1492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:24 smithi143 bash[29284]: audit 2024-03-11T21:30:24.459770+0000 mon.c (mon.2) 376 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:24.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.267938+0000 mgr.y (mgr.14550) 305 : cluster 0 pgmap v321: 340 pgs: 29 creating+peering, 11 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.393691+0000 osd.6 (osd.6) 109 : cluster 0 3.10 scrub starts 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.397302+0000 osd.6 (osd.6) 110 : cluster 0 3.10 scrub ok 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: audit 2024-03-11T21:30:23.458253+0000 mon.c (mon.2) 374 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.713438+0000 osd.2 (osd.2) 127 : cluster 0 39.1f scrub starts 2024-03-11T21:30:24.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:23.715433+0000 osd.2 (osd.2) 128 : cluster 0 39.1f scrub ok 2024-03-11T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.235864+0000 mon.a (mon.0) 1491 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: audit 2024-03-11T21:30:24.260261+0000 mon.c (mon.2) 375 : audit 1 from='client.? 172.21.15.143:0/3332855013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: audit 2024-03-11T21:30:24.261289+0000 mon.a (mon.0) 1492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:24 smithi203 bash[23654]: audit 2024-03-11T21:30:24.459770+0000 mon.c (mon.2) 376 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:24.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:25.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:25.074 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:25.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:25.205 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:25.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:25.332 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:25.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.461 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:25.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:25.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:25.828 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:25.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.431950+0000 osd.6 (osd.6) 111 : cluster 0 12.16 scrub starts 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.434656+0000 osd.6 (osd.6) 112 : cluster 0 12.16 scrub ok 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.466068+0000 osd.7 (osd.7) 203 : cluster 0 6.8 scrub starts 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.467775+0000 osd.7 (osd.7) 204 : cluster 0 6.8 scrub ok 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.694829+0000 osd.1 (osd.1) 149 : cluster 0 9.1f scrub starts 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.696891+0000 osd.1 (osd.1) 150 : cluster 0 9.1f scrub ok 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.757361+0000 osd.2 (osd.2) 129 : cluster 0 39.6 scrub starts 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.759281+0000 osd.2 (osd.2) 130 : cluster 0 39.6 scrub ok 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.995039+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:24.997666+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: audit 2024-03-11T21:30:25.235835+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:25.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.247311+0000 mon.a (mon.0) 1494 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:30:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: audit 2024-03-11T21:30:25.251775+0000 mon.c (mon.2) 377 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: audit 2024-03-11T21:30:25.254955+0000 mon.a (mon.0) 1495 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: audit 2024-03-11T21:30:25.255900+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:25 smithi203 bash[23654]: audit 2024-03-11T21:30:25.461256+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:25.983 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.431950+0000 osd.6 (osd.6) 111 : cluster 0 12.16 scrub starts 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.434656+0000 osd.6 (osd.6) 112 : cluster 0 12.16 scrub ok 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.466068+0000 osd.7 (osd.7) 203 : cluster 0 6.8 scrub starts 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.467775+0000 osd.7 (osd.7) 204 : cluster 0 6.8 scrub ok 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.694829+0000 osd.1 (osd.1) 149 : cluster 0 9.1f scrub starts 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.696891+0000 osd.1 (osd.1) 150 : cluster 0 9.1f scrub ok 2024-03-11T21:30:25.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.757361+0000 osd.2 (osd.2) 129 : cluster 0 39.6 scrub starts 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.759281+0000 osd.2 (osd.2) 130 : cluster 0 39.6 scrub ok 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.995039+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:24.997666+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: audit 2024-03-11T21:30:25.235835+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.247311+0000 mon.a (mon.0) 1494 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: audit 2024-03-11T21:30:25.251775+0000 mon.c (mon.2) 377 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: audit 2024-03-11T21:30:25.254955+0000 mon.a (mon.0) 1495 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: audit 2024-03-11T21:30:25.255900+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[21512]: audit 2024-03-11T21:30:25.461256+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.431950+0000 osd.6 (osd.6) 111 : cluster 0 12.16 scrub starts 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.434656+0000 osd.6 (osd.6) 112 : cluster 0 12.16 scrub ok 2024-03-11T21:30:25.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.466068+0000 osd.7 (osd.7) 203 : cluster 0 6.8 scrub starts 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.467775+0000 osd.7 (osd.7) 204 : cluster 0 6.8 scrub ok 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.694829+0000 osd.1 (osd.1) 149 : cluster 0 9.1f scrub starts 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.696891+0000 osd.1 (osd.1) 150 : cluster 0 9.1f scrub ok 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.757361+0000 osd.2 (osd.2) 129 : cluster 0 39.6 scrub starts 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.759281+0000 osd.2 (osd.2) 130 : cluster 0 39.6 scrub ok 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.995039+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:24.997666+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: audit 2024-03-11T21:30:25.235835+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi143-71594-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.247311+0000 mon.a (mon.0) 1494 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: audit 2024-03-11T21:30:25.251775+0000 mon.c (mon.2) 377 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: audit 2024-03-11T21:30:25.254955+0000 mon.a (mon.0) 1495 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: audit 2024-03-11T21:30:25.255900+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:25.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:25 smithi143 bash[29284]: audit 2024-03-11T21:30:25.461256+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:26.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.113 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:26.217 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:26.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:26.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:26.488 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:26.511 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:26.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:26.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:26.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.269164+0000 mgr.y (mgr.14550) 306 : cluster 0 pgmap v324: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.448063+0000 osd.6 (osd.6) 113 : cluster 0 5.2 scrub starts 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.450291+0000 osd.6 (osd.6) 114 : cluster 0 5.2 scrub ok 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.477591+0000 osd.7 (osd.7) 205 : cluster 0 2.1d scrub starts 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:25.479168+0000 osd.7 (osd.7) 206 : cluster 0 2.1d scrub ok 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: audit 2024-03-11T21:30:26.210309+0000 mon.a (mon.0) 1497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: audit 2024-03-11T21:30:26.210592+0000 mon.a (mon.0) 1498 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: audit 2024-03-11T21:30:26.216702+0000 mon.c (mon.2) 379 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: cluster 2024-03-11T21:30:26.220914+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: audit 2024-03-11T21:30:26.245451+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:26.797 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[21512]: audit 2024-03-11T21:30:26.462716+0000 mon.c (mon.2) 380 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.269164+0000 mgr.y (mgr.14550) 306 : cluster 0 pgmap v324: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.448063+0000 osd.6 (osd.6) 113 : cluster 0 5.2 scrub starts 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.450291+0000 osd.6 (osd.6) 114 : cluster 0 5.2 scrub ok 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.477591+0000 osd.7 (osd.7) 205 : cluster 0 2.1d scrub starts 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:25.479168+0000 osd.7 (osd.7) 206 : cluster 0 2.1d scrub ok 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: audit 2024-03-11T21:30:26.210309+0000 mon.a (mon.0) 1497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: audit 2024-03-11T21:30:26.210592+0000 mon.a (mon.0) 1498 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: audit 2024-03-11T21:30:26.216702+0000 mon.c (mon.2) 379 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: cluster 2024-03-11T21:30:26.220914+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: audit 2024-03-11T21:30:26.245451+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:26.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:26 smithi143 bash[29284]: audit 2024-03-11T21:30:26.462716+0000 mon.c (mon.2) 380 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:26.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:26.900 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:27.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.269164+0000 mgr.y (mgr.14550) 306 : cluster 0 pgmap v324: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T21:30:27.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.448063+0000 osd.6 (osd.6) 113 : cluster 0 5.2 scrub starts 2024-03-11T21:30:27.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.450291+0000 osd.6 (osd.6) 114 : cluster 0 5.2 scrub ok 2024-03-11T21:30:27.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.477591+0000 osd.7 (osd.7) 205 : cluster 0 2.1d scrub starts 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:25.479168+0000 osd.7 (osd.7) 206 : cluster 0 2.1d scrub ok 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: audit 2024-03-11T21:30:26.210309+0000 mon.a (mon.0) 1497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: audit 2024-03-11T21:30:26.210592+0000 mon.a (mon.0) 1498 : audit 1 from='client.? 172.21.15.143:0/1615863527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi143-71604-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: audit 2024-03-11T21:30:26.216702+0000 mon.c (mon.2) 379 : audit 1 from='client.? 172.21.15.143:0/3284700860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: cluster 2024-03-11T21:30:26.220914+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: audit 2024-03-11T21:30:26.245451+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]: dispatch 2024-03-11T21:30:27.015 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:26 smithi203 bash[23654]: audit 2024-03-11T21:30:26.462716+0000 mon.c (mon.2) 380 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:27.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:27.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:27.205 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pP (68689 ms total) 2024-03-11T21:30:27.205 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.205 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (298 ms) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (299 ms total) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (54 ms) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (14 ms) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (68 ms total) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (61 ms) 2024-03-11T21:30:27.206 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (10 ms) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (71 ms total) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (82 ms) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (85 ms total) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1028 ms) 2024-03-11T21:30:27.207 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1028 ms total) 2024-03-11T21:30:27.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-03-11T21:30:27.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-11T21:30:27.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (111552 ms total) 2024-03-11T21:30:27.208 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-11T21:30:27.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:27.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:27.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:27.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:27.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:27.671 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:27.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.800 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:27.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:27.931 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:28.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:28.059 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:28.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:28.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:28.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.026096+0000 osd.0 (osd.0) 153 : cluster 0 3.17 scrub starts 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.029862+0000 osd.0 (osd.0) 154 : cluster 0 3.17 scrub ok 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.213512+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.217707+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.222335+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.226983+0000 mon.a (mon.0) 1503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.233863+0000 mon.c (mon.2) 382 : audit 1 from='client.? 172.21.15.143:0/2930149780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.234438+0000 mon.a (mon.0) 1504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.269767+0000 mgr.y (mgr.14550) 307 : cluster 0 pgmap v327: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: audit 2024-03-11T21:30:27.464173+0000 mon.c (mon.2) 383 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.497404+0000 osd.7 (osd.7) 207 : cluster 0 6.1c scrub starts 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.499418+0000 osd.7 (osd.7) 208 : cluster 0 6.1c scrub ok 2024-03-11T21:30:28.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.781721+0000 osd.3 (osd.3) 171 : cluster 0 6.11 scrub starts 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.783927+0000 osd.3 (osd.3) 172 : cluster 0 6.11 scrub ok 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.026096+0000 osd.0 (osd.0) 153 : cluster 0 3.17 scrub starts 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.029862+0000 osd.0 (osd.0) 154 : cluster 0 3.17 scrub ok 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.213512+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.217707+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.222335+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.226983+0000 mon.a (mon.0) 1503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.233863+0000 mon.c (mon.2) 382 : audit 1 from='client.? 172.21.15.143:0/2930149780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.234438+0000 mon.a (mon.0) 1504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.269767+0000 mgr.y (mgr.14550) 307 : cluster 0 pgmap v327: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: audit 2024-03-11T21:30:27.464173+0000 mon.c (mon.2) 383 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.497404+0000 osd.7 (osd.7) 207 : cluster 0 6.1c scrub starts 2024-03-11T21:30:28.305 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.499418+0000 osd.7 (osd.7) 208 : cluster 0 6.1c scrub ok 2024-03-11T21:30:28.305 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.781721+0000 osd.3 (osd.3) 171 : cluster 0 6.11 scrub starts 2024-03-11T21:30:28.305 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:28 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.783927+0000 osd.3 (osd.3) 172 : cluster 0 6.11 scrub ok 2024-03-11T21:30:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:28.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:28.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:28.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.026096+0000 osd.0 (osd.0) 153 : cluster 0 3.17 scrub starts 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.029862+0000 osd.0 (osd.0) 154 : cluster 0 3.17 scrub ok 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.213512+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi143-71756-36"}]': finished 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.217707+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.222335+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.226983+0000 mon.a (mon.0) 1503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.233863+0000 mon.c (mon.2) 382 : audit 1 from='client.? 172.21.15.143:0/2930149780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.234438+0000 mon.a (mon.0) 1504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.269767+0000 mgr.y (mgr.14550) 307 : cluster 0 pgmap v327: 324 pgs: 32 unknown, 292 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:28.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: audit 2024-03-11T21:30:27.464173+0000 mon.c (mon.2) 383 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.497404+0000 osd.7 (osd.7) 207 : cluster 0 6.1c scrub starts 2024-03-11T21:30:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.499418+0000 osd.7 (osd.7) 208 : cluster 0 6.1c scrub ok 2024-03-11T21:30:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.781721+0000 osd.3 (osd.3) 171 : cluster 0 6.11 scrub starts 2024-03-11T21:30:28.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:28 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.783927+0000 osd.3 (osd.3) 172 : cluster 0 6.11 scrub ok 2024-03-11T21:30:28.450 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:28.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:28.579 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:28.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:28.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:28.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:28.838 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:28.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:28.970 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:29.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:29.101 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:29.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:29.248 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:29.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:29.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:29.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.486477+0000 osd.6 (osd.6) 115 : cluster 0 2.17 scrub starts 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:27.487772+0000 osd.6 (osd.6) 116 : cluster 0 2.17 scrub ok 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.022537+0000 osd.5 (osd.5) 169 : cluster 0 6.a scrub starts 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.024060+0000 osd.5 (osd.5) 170 : cluster 0 6.a scrub ok 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.074714+0000 osd.0 (osd.0) 155 : cluster 0 3.1b scrub starts 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.077396+0000 osd.0 (osd.0) 156 : cluster 0 3.1b scrub ok 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.216034+0000 mon.a (mon.0) 1505 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.217817+0000 mon.a (mon.0) 1506 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.218085+0000 mon.a (mon.0) 1507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.487 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.233381+0000 mon.c (mon.2) 384 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.236064+0000 mon.a (mon.0) 1508 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.242606+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.143:0/1463531488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.242686+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.265117+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.403836+0000 mgr.y (mgr.14550) 308 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:28.465783+0000 mon.c (mon.2) 386 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.521073+0000 osd.6 (osd.6) 117 : cluster 0 12.1f scrub starts 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.524581+0000 osd.6 (osd.6) 118 : cluster 0 12.1f scrub ok 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.808057+0000 osd.3 (osd.3) 173 : cluster 0 5.13 scrub starts 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:28.810549+0000 osd.3 (osd.3) 174 : cluster 0 5.13 scrub ok 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:29.221822+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: audit 2024-03-11T21:30:29.222079+0000 mon.a (mon.0) 1512 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.488 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.230997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:30:29.489 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.486477+0000 osd.6 (osd.6) 115 : cluster 0 2.17 scrub starts 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:27.487772+0000 osd.6 (osd.6) 116 : cluster 0 2.17 scrub ok 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.022537+0000 osd.5 (osd.5) 169 : cluster 0 6.a scrub starts 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.024060+0000 osd.5 (osd.5) 170 : cluster 0 6.a scrub ok 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.074714+0000 osd.0 (osd.0) 155 : cluster 0 3.1b scrub starts 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.077396+0000 osd.0 (osd.0) 156 : cluster 0 3.1b scrub ok 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.216034+0000 mon.a (mon.0) 1505 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.217817+0000 mon.a (mon.0) 1506 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.218085+0000 mon.a (mon.0) 1507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.233381+0000 mon.c (mon.2) 384 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.236064+0000 mon.a (mon.0) 1508 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.242606+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.143:0/1463531488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.242686+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.265117+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.490 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.403836+0000 mgr.y (mgr.14550) 308 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:28.465783+0000 mon.c (mon.2) 386 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.521073+0000 osd.6 (osd.6) 117 : cluster 0 12.1f scrub starts 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.524581+0000 osd.6 (osd.6) 118 : cluster 0 12.1f scrub ok 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.808057+0000 osd.3 (osd.3) 173 : cluster 0 5.13 scrub starts 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:28.810549+0000 osd.3 (osd.3) 174 : cluster 0 5.13 scrub ok 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:29.221822+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: audit 2024-03-11T21:30:29.222079+0000 mon.a (mon.0) 1512 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.491 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:29 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.230997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:30:29.498 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:29.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.486477+0000 osd.6 (osd.6) 115 : cluster 0 2.17 scrub starts 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:27.487772+0000 osd.6 (osd.6) 116 : cluster 0 2.17 scrub ok 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.022537+0000 osd.5 (osd.5) 169 : cluster 0 6.a scrub starts 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.024060+0000 osd.5 (osd.5) 170 : cluster 0 6.a scrub ok 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.074714+0000 osd.0 (osd.0) 155 : cluster 0 3.1b scrub starts 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.077396+0000 osd.0 (osd.0) 156 : cluster 0 3.1b scrub ok 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.216034+0000 mon.a (mon.0) 1505 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.217817+0000 mon.a (mon.0) 1506 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.218085+0000 mon.a (mon.0) 1507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi143-71594-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.233381+0000 mon.c (mon.2) 384 : audit 1 from='client.? 172.21.15.143:0/334886417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.236064+0000 mon.a (mon.0) 1508 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.242606+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.143:0/1463531488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.242686+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]: dispatch 2024-03-11T21:30:29.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.265117+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.403836+0000 mgr.y (mgr.14550) 308 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:28.465783+0000 mon.c (mon.2) 386 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.521073+0000 osd.6 (osd.6) 117 : cluster 0 12.1f scrub starts 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.524581+0000 osd.6 (osd.6) 118 : cluster 0 12.1f scrub ok 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.808057+0000 osd.3 (osd.3) 173 : cluster 0 5.13 scrub starts 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:28.810549+0000 osd.3 (osd.3) 174 : cluster 0 5.13 scrub ok 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:29.221822+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi143-71838-16"}]': finished 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: audit 2024-03-11T21:30:29.222079+0000 mon.a (mon.0) 1512 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi143-71604-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:29.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:29 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.230997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:30:29.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:29.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:29.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:29.758 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:29.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:29.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:29.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:30.016 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:30.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:30.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:30.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.014428+0000 osd.5 (osd.5) 171 : cluster 0 2.18 scrub starts 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.016588+0000 osd.5 (osd.5) 172 : cluster 0 2.18 scrub ok 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.031620+0000 osd.4 (osd.4) 153 : cluster 0 3.9 scrub starts 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.035460+0000 osd.4 (osd.4) 154 : cluster 0 3.9 scrub ok 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.272662+0000 mgr.y (mgr.14550) 309 : cluster 0 pgmap v330: 324 pgs: 6 creating+peering, 26 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.294957+0000 mon.c (mon.2) 387 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.467280+0000 mon.c (mon.2) 388 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:30.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.683924+0000 osd.2 (osd.2) 131 : cluster 0 12.9 scrub starts 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.685896+0000 osd.2 (osd.2) 132 : cluster 0 12.9 scrub ok 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.736593+0000 mon.a (mon.0) 1514 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.737747+0000 mon.c (mon.2) 389 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.738100+0000 mon.a (mon.0) 1515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.738727+0000 mon.c (mon.2) 390 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[21512]: audit 2024-03-11T21:30:29.739171+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.014428+0000 osd.5 (osd.5) 171 : cluster 0 2.18 scrub starts 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.016588+0000 osd.5 (osd.5) 172 : cluster 0 2.18 scrub ok 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.031620+0000 osd.4 (osd.4) 153 : cluster 0 3.9 scrub starts 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.035460+0000 osd.4 (osd.4) 154 : cluster 0 3.9 scrub ok 2024-03-11T21:30:30.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.272662+0000 mgr.y (mgr.14550) 309 : cluster 0 pgmap v330: 324 pgs: 6 creating+peering, 26 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.294957+0000 mon.c (mon.2) 387 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.467280+0000 mon.c (mon.2) 388 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.683924+0000 osd.2 (osd.2) 131 : cluster 0 12.9 scrub starts 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.685896+0000 osd.2 (osd.2) 132 : cluster 0 12.9 scrub ok 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.736593+0000 mon.a (mon.0) 1514 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.737747+0000 mon.c (mon.2) 389 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.738100+0000 mon.a (mon.0) 1515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.738727+0000 mon.c (mon.2) 390 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:30 smithi143 bash[29284]: audit 2024-03-11T21:30:29.739171+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.288 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:30.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:30.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.014428+0000 osd.5 (osd.5) 171 : cluster 0 2.18 scrub starts 2024-03-11T21:30:30.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.016588+0000 osd.5 (osd.5) 172 : cluster 0 2.18 scrub ok 2024-03-11T21:30:30.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.031620+0000 osd.4 (osd.4) 153 : cluster 0 3.9 scrub starts 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.035460+0000 osd.4 (osd.4) 154 : cluster 0 3.9 scrub ok 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.272662+0000 mgr.y (mgr.14550) 309 : cluster 0 pgmap v330: 324 pgs: 6 creating+peering, 26 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.294957+0000 mon.c (mon.2) 387 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.467280+0000 mon.c (mon.2) 388 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.683924+0000 osd.2 (osd.2) 131 : cluster 0 12.9 scrub starts 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.685896+0000 osd.2 (osd.2) 132 : cluster 0 12.9 scrub ok 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.736593+0000 mon.a (mon.0) 1514 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.737747+0000 mon.c (mon.2) 389 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.738100+0000 mon.a (mon.0) 1515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.738727+0000 mon.c (mon.2) 390 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:30 smithi203 bash[23654]: audit 2024-03-11T21:30:29.739171+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:30.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:30.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:30.680 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:30.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:30.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:30.940 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:31.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.070 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.474 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24836.0:2 25.1 25:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-11T21:30:31.485 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 104.70263943800001, 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6763401199999999, 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24836", 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/184328842", 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.486 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807044+0000", 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.843e-06 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807048+0000", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7269999999999998e-06 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807058+0000", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7999999999999993e-06 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807067+0000", 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9590000000000001e-06 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:49.771853+0000", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9647853340000001 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:49.771874+0000", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1265999999999999e-05 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.358550+0000", 2024-03-11T21:30:31.489 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5866757090000001 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.358659+0000", 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000109337 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.371175+0000", 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012516042999999999 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:30:31.490 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.378988+0000", 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0078134309999999992 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.423413+0000", 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.044424287 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483276+0000", 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.059863422999999999 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.491 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483335+0000", 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8535000000000001e-05 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483382+0000", 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7426000000000003e-05 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24730.0:2 36.1c 36:39ce3343:test-rados-api-smithi143-71918-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-11T21:30:31.492 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 99.186323427000005, 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.577233369, 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24730", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1999332124", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323359+0000", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5039999999999999e-06 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323366+0000", 2024-03-11T21:30:31.494 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2489999999999998e-06 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.325811+0000", 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024457099999999998 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.325825+0000", 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3283e-05 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.495 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.326088+0000", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026380000000000002 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.326098+0000", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1339999999999993e-06 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885403+0000", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5593055150000001 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885537+0000", 2024-03-11T21:30:31.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000134141 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885675+0000", 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000137236 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.891300+0000", 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056256520000000001 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.891791+0000", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00049041999999999998 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900495+0000", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0087042160000000007 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900549+0000", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4632000000000001e-05 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900591+0000", 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1876999999999999e-05 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.24743.0:2070 13.1c e144/139 13:3b97e158:test-rados-api-smithi143-71756-17::big2:head v 144'2, mlcod=0'0)", 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:54.445370+0000", 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 97.064311296, 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.091493930000000001, 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.499 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.445370+0000", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.445370+0000", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.445371+0000", 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.274e-06 2024-03-11T21:30:31.500 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.484105+0000", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.038733953000000002 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.484126+0000", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0537e-05 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.484137+0000", 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0791999999999999e-05 2024-03-11T21:30:31.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.484204+0000", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7628999999999999e-05 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.484223+0000", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.893e-05 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.536812+0000", 2024-03-11T21:30:31.502 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.052589097000000001 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.536864+0000", 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.1718000000000002e-05 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.25139.0:2 51.15 51:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:30:31.503 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 95.455374258000006, 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8283110329999999, 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.25139", 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2889742841", 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.504 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054308+0000", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.850000000000001e-07 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054313+0000", 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2109999999999997e-06 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054328+0000", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4766999999999999e-05 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054337+0000", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.02e-06 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.088502+0000", 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.034165244999999997 2024-03-11T21:30:31.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.088513+0000", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0225e-05 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.869905+0000", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7813922409999998 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870009+0000", 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000104211 2024-03-11T21:30:31.507 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870101+0000", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2275000000000007e-05 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870700+0000", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00059873000000000001 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.871403+0000", 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00070267699999999997 2024-03-11T21:30:31.508 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882550+0000", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011147527000000001 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882585+0000", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.5185000000000003e-05 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882618+0000", 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2734000000000002e-05 2024-03-11T21:30:31.509 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23010.0:2 121.15 121:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 57.770213914000003, 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8154935549999998, 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23010", 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2727152565", 2024-03-11T21:30:31.510 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739470+0000", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0810000000000001e-06 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739478+0000", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1980000000000001e-06 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739495+0000", 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7447000000000001e-05 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.512 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739508+0000", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.271e-05 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739585+0000", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7406999999999999e-05 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739596+0000", 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0583e-05 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.513 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.529742+0000", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7901457499999998 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.529867+0000", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000125258 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.530003+0000", 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013603000000000001 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.514 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.551591+0000", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.021587688000000001 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.552517+0000", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00092659700000000003 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554865+0000", 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002348232 2024-03-11T21:30:31.515 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554920+0000", 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4293999999999999e-05 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554961+0000", 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1279999999999998e-05 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.516 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:1 123.6s0 123:653d3f2f:::000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 56.011899702000001, 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0403799309999999, 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497784+0000", 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.88e-06 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.518 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497789+0000", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6980000000000003e-06 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497818+0000", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8280999999999999e-05 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497829+0000", 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1199999999999999e-05 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497903+0000", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4468999999999995e-05 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497908+0000", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4460000000000003e-06 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493194+0000", 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.99528657200000004 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.520 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493271+0000", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7151999999999996e-05 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493536+0000", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026473799999999999 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.499652+0000", 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0061159090000000001 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.538093+0000", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.038441365999999998 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.538162+0000", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8219999999999994e-05 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.e epoch 178/176 rep_tid 597 entries 178'7 (0'0) error 126:748fcfa5:::test_252:head by client.23697.0:259 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 177'5)", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:30:31.522 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 52.904586025999997, 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15239376199999999, 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.523 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605096+0000", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.13e-06 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605108+0000", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1088e-05 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605129+0000", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1834000000000001e-05 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605144+0000", 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5088e-05 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605297+0000", 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000152299 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757489+0000", 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15219232299999999 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.525 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:268 123.6s0 123:680f0a60:::267:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 52.904149940000003, 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.153234183, 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 268 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.526 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605533+0000", 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5710000000000001e-06 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605540+0000", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1250000000000004e-06 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605554+0000", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.379e-05 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605563+0000", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.8170000000000005e-06 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606212+0000", 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00064885599999999996 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606345+0000", 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013343599999999999 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606687+0000", 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000342381 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758441+0000", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15175385299999999 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758507+0000", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6124000000000006e-05 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758766+0000", 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025823000000000003 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.530 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 178/175 ECSubWriteReply(tid=407, last_complete=178'30, committed=1, applied=1))", 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.609188+0000", 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 52.900493429999997, 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.148502575, 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.609188+0000", 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.609188+0000", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.609189+0000", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2920000000000001e-06 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.609193+0000", 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1799999999999998e-06 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.532 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757599+0000", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.14840568800000001 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757611+0000", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2089000000000001e-05 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757668+0000", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7287000000000002e-05 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.533 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757691+0000", 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2039e-05 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 178/175 ECSubWriteReply(tid=407, last_complete=178'30, committed=1, applied=1))", 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.610229+0000", 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 52.899451904999999, 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.14750761100000001, 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.534 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.610229+0000", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.610229+0000", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.610230+0000", 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.001e-06 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.610236+0000", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7209999999999999e-06 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757664+0000", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.14742764899999999 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757673+0000", 2024-03-11T21:30:31.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.0620000000000007e-06 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757721+0000", 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7676999999999998e-05 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757737+0000", 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6501e-05 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(121.0 epoch 178/174 rep_tid 598 entries 178'13 (0'0) error 121:0326b288:::test_248:head by client.23010.0:255 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 178'11)", 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.613079+0000", 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 52.896602758999997, 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.14575892500000001, 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.538 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.613079+0000", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.613079+0000", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.613080+0000", 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.13e-06 2024-03-11T21:30:31.539 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.613093+0000", 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3508e-05 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757564+0000", 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.14447075600000001 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757590+0000", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5783e-05 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757733+0000", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000143468 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758838+0000", 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00110428 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.541 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:413 123.6s0 123:7760f31e:::412:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.966974870000001, 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.183137513, 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:30:31.542 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 413 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542708+0000", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7570000000000001e-06 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542716+0000", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9230000000000002e-06 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542732+0000", 2024-03-11T21:30:31.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5624000000000002e-05 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542740+0000", 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8699999999999992e-06 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542795+0000", 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4852000000000003e-05 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.545 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542888+0000", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.3542000000000007e-05 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543159+0000", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027125599999999999 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725710+0000", 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18255065400000001 2024-03-11T21:30:31.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725798+0000", 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.7621999999999996e-05 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725844+0000", 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6412999999999998e-05 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.14 epoch 179/176 rep_tid 673 entries 179'13 (0'0) error 126:2884fdb0:::test_383:head by client.23697.0:390 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 179'11)", 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.966710220000003, 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.182524724, 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.548 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542972+0000", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0890000000000001e-06 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542984+0000", 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2224e-05 2024-03-11T21:30:31.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543008+0000", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3124000000000001e-05 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543022+0000", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4521e-05 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543225+0000", 2024-03-11T21:30:31.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020285100000000001 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725496+0000", 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18227091500000001 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 179/175 ECSubWriteReply(tid=485, last_complete=179'50, committed=1, applied=1))", 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:30:31.551 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.965909975999999, 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18223889700000001, 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.552 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543772+0000", 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6300000000000005e-07 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543776+0000", 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0799999999999999e-06 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725718+0000", 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18194215899999999 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725726+0000", 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5889999999999997e-06 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725756+0000", 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0587999999999999e-05 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.726010+0000", 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000253818 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 179/175 ECSubWriteReply(tid=485, last_complete=179'50, committed=1, applied=1))", 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.964776573000002, 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18037215600000001, 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.555 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544906+0000", 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7699999999999992e-07 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544910+0000", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3409999999999996e-06 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725101+0000", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18019127800000001 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725118+0000", 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6705e-05 2024-03-11T21:30:31.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725263+0000", 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014521099999999999 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725277+0000", 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3644e-05 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.558 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23010.0:428 121.14 121:2a92f4ca:::test_421:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.494223009000002, 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.294164496, 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23010", 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2727152565", 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 428 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.559 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.560 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015459+0000", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0759999999999999e-06 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015465+0000", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.054e-06 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015477+0000", 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1257000000000001e-05 2024-03-11T21:30:31.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015485+0000", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.79e-06 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015740+0000", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025537800000000001 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015907+0000", 2024-03-11T21:30:31.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016721800000000001 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.309623+0000", 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.29371572299999998 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing_reply(121.14 epoch 179/174 rep_tid 537 lcod 179'15)", 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.492498015000002, 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19906863499999999, 2024-03-11T21:30:31.563 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.564 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017184+0000", 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3300000000000002e-07 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017187+0000", 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.27e-06 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017198+0000", 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1039999999999999e-05 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017206+0000", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2849999999999997e-06 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017273+0000", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.745e-05 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216252+0000", 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19897895700000001 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.566 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWrite(123.4s2 179/175 ECSubWrite(tid=447, reqid=client.22002.0:483, at_version=0'0, trim_to=0'0, roll_forward_to=179'52))", 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.489555301000003, 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19683077800000001, 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020127+0000", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.387e-06 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020144+0000", 2024-03-11T21:30:31.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6331999999999999e-05 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216496+0000", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.196352472 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216507+0000", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0541e-05 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216557+0000", 2024-03-11T21:30:31.569 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.0590999999999997e-05 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216567+0000", 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.448e-06 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216928+0000", 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00036090599999999998 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.570 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216957+0000", 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9101000000000001e-05 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.1f epoch 179/176 rep_tid 708 entries 179'12 (0'0) error 126:fdf52ff4:::test_450:head by client.23697.0:457 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 179'10)", 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 51.486584942999997, 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19388946500000001, 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.571 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023098+0000", 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8250000000000001e-06 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023115+0000", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6483000000000001e-05 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216307+0000", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19319249399999999 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216335+0000", 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7846000000000001e-05 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216434+0000", 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9415999999999998e-05 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216986+0000", 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00055140099999999997 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23010.0:592 121.15 121:aeef9581:::test_585:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e180)", 2024-03-11T21:30:31.574 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:41.210870+0000", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 50.298811360999998, 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.090684083999999998, 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23010", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2727152565", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 592 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210870+0000", 2024-03-11T21:30:31.575 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210870+0000", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210872+0000", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.66e-06 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210881+0000", 2024-03-11T21:30:31.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9619999999999999e-06 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210899+0000", 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.806e-05 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210910+0000", 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1134e-05 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:30:31.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.210973+0000", 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3491000000000006e-05 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.211114+0000", 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000140412 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.301554+0000", 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.090440364999999995 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.578 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.579 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:30:31.579 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:30:31.579 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:30:31.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:31.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:31.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.708 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.720111+0000 osd.1 (osd.1) 151 : cluster 0 12.6 scrub starts 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.722879+0000 osd.1 (osd.1) 152 : cluster 0 12.6 scrub ok 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: cluster 2024-03-11T21:30:29.995801+0000 osd.0 (osd.0) 157 : cluster 0 3.1 scrub starts 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.000031+0000 osd.0 (osd.0) 158 : cluster 0 3.1 scrub ok 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:30.269817+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.283539+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:30.289766+0000 mon.a (mon.0) 1519 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:30.298723+0000 mon.c (mon.2) 391 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:30.300156+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:30.468405+0000 mon.c (mon.2) 392 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:31.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[21512]: audit 2024-03-11T21:30:31.212957+0000 mon.a (mon.0) 1521 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.720111+0000 osd.1 (osd.1) 151 : cluster 0 12.6 scrub starts 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.722879+0000 osd.1 (osd.1) 152 : cluster 0 12.6 scrub ok 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: cluster 2024-03-11T21:30:29.995801+0000 osd.0 (osd.0) 157 : cluster 0 3.1 scrub starts 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.000031+0000 osd.0 (osd.0) 158 : cluster 0 3.1 scrub ok 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:30.269817+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.283539+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:30.289766+0000 mon.a (mon.0) 1519 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:30.298723+0000 mon.c (mon.2) 391 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:30.300156+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:30.468405+0000 mon.c (mon.2) 392 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:31.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:31 smithi143 bash[29284]: audit 2024-03-11T21:30:31.212957+0000 mon.a (mon.0) 1521 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:31.719 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.843 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:31.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.972 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:32.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.720111+0000 osd.1 (osd.1) 151 : cluster 0 12.6 scrub starts 2024-03-11T21:30:32.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.722879+0000 osd.1 (osd.1) 152 : cluster 0 12.6 scrub ok 2024-03-11T21:30:32.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: cluster 2024-03-11T21:30:29.995801+0000 osd.0 (osd.0) 157 : cluster 0 3.1 scrub starts 2024-03-11T21:30:32.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.000031+0000 osd.0 (osd.0) 158 : cluster 0 3.1 scrub ok 2024-03-11T21:30:32.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:30.269817+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.283539+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:30.289766+0000 mon.a (mon.0) 1519 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:30.298723+0000 mon.c (mon.2) 391 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:30.300156+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:30.468405+0000 mon.c (mon.2) 392 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:31 smithi203 bash[23654]: audit 2024-03-11T21:30:31.212957+0000 mon.a (mon.0) 1521 : audit 1 from='client.? 172.21.15.143:0/3132407875' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi143-71594-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:32.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.104 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:32.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:32.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:32.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:32.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:32.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.517 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:32.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:32.646 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:32.749 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:32.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.889429+0000 osd.3 (osd.3) 175 : cluster 0 2.1e scrub starts 2024-03-11T21:30:32.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.891756+0000 osd.3 (osd.3) 176 : cluster 0 2.1e scrub ok 2024-03-11T21:30:32.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.990709+0000 osd.0 (osd.0) 159 : cluster 0 12.19 scrub starts 2024-03-11T21:30:32.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:30.992664+0000 osd.0 (osd.0) 160 : cluster 0 12.19 scrub ok 2024-03-11T21:30:32.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.273565+0000 mgr.y (mgr.14550) 310 : cluster 0 pgmap v333: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: audit 2024-03-11T21:30:31.469849+0000 mon.c (mon.2) 393 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.530603+0000 osd.6 (osd.6) 119 : cluster 0 2.7 scrub starts 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.533795+0000 osd.6 (osd.6) 120 : cluster 0 2.7 scrub ok 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.665945+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: audit 2024-03-11T21:30:31.675823+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: audit 2024-03-11T21:30:32.216733+0000 mon.a (mon.0) 1524 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: audit 2024-03-11T21:30:32.217089+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[21512]: cluster 2024-03-11T21:30:32.226412+0000 mon.a (mon.0) 1526 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.889429+0000 osd.3 (osd.3) 175 : cluster 0 2.1e scrub starts 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.891756+0000 osd.3 (osd.3) 176 : cluster 0 2.1e scrub ok 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.990709+0000 osd.0 (osd.0) 159 : cluster 0 12.19 scrub starts 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:30.992664+0000 osd.0 (osd.0) 160 : cluster 0 12.19 scrub ok 2024-03-11T21:30:32.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.273565+0000 mgr.y (mgr.14550) 310 : cluster 0 pgmap v333: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: audit 2024-03-11T21:30:31.469849+0000 mon.c (mon.2) 393 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.530603+0000 osd.6 (osd.6) 119 : cluster 0 2.7 scrub starts 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.533795+0000 osd.6 (osd.6) 120 : cluster 0 2.7 scrub ok 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.665945+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: audit 2024-03-11T21:30:31.675823+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: audit 2024-03-11T21:30:32.216733+0000 mon.a (mon.0) 1524 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: audit 2024-03-11T21:30:32.217089+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:32.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:32 smithi143 bash[29284]: cluster 2024-03-11T21:30:32.226412+0000 mon.a (mon.0) 1526 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:30:32.771 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:32.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:32.899 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:33.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:33.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.889429+0000 osd.3 (osd.3) 175 : cluster 0 2.1e scrub starts 2024-03-11T21:30:33.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.891756+0000 osd.3 (osd.3) 176 : cluster 0 2.1e scrub ok 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.990709+0000 osd.0 (osd.0) 159 : cluster 0 12.19 scrub starts 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:30.992664+0000 osd.0 (osd.0) 160 : cluster 0 12.19 scrub ok 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.273565+0000 mgr.y (mgr.14550) 310 : cluster 0 pgmap v333: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: audit 2024-03-11T21:30:31.469849+0000 mon.c (mon.2) 393 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.530603+0000 osd.6 (osd.6) 119 : cluster 0 2.7 scrub starts 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.533795+0000 osd.6 (osd.6) 120 : cluster 0 2.7 scrub ok 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.665945+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: audit 2024-03-11T21:30:31.675823+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: audit 2024-03-11T21:30:32.216733+0000 mon.a (mon.0) 1524 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: audit 2024-03-11T21:30:32.217089+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.143:0/2674272180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi143-71604-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:33.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:32 smithi203 bash[23654]: cluster 2024-03-11T21:30:32.226412+0000 mon.a (mon.0) 1526 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:30:33.026 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:33.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:33.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:33.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:33.279 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:33.383 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:33.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:33.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:33.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:33.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 -- 172.21.15.143:0/3438948984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928107f30 msgr2=0x7f7928108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 --2- 172.21.15.143:0/3438948984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928107f30 0x7f7928108310 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f7920009990 tx=0x7f7920009ca0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 -- 172.21.15.143:0/3438948984 shutdown_connections 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 --2- 172.21.15.143:0/3438948984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7928102880 0x7f792810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 --2- 172.21.15.143:0/3438948984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928107f30 0x7f7928108310 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 --2- 172.21.15.143:0/3438948984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f7928102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 -- 172.21.15.143:0/3438948984 >> 172.21.15.143:0/3438948984 conn(0x7f7928078eb0 msgr2=0x7f79280792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 -- 172.21.15.143:0/3438948984 shutdown_connections 2024-03-11T21:30:33.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.642+0000 7f792f151700 1 -- 172.21.15.143:0/3438948984 wait complete. 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 librados: starting msgr at 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 librados: starting objecter 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 Processor -- start 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 -- start start 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 librados: setting wanted keys 2024-03-11T21:30:33.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 librados: calling monclient init 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f79280740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928102880 0x7f7928074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7928107f30 0x7f7928074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79281116a0 con 0x7f7928101ec0 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7928111520 con 0x7f7928107f30 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792f151700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7928111820 con 0x7f7928102880 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f79280740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f79280740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52120/0 (socket says 172.21.15.143:52120) 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 -- 172.21.15.143:0/3903967696 learned_addr learned my addr 172.21.15.143:0/3903967696 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928102880 0x7f7928074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7928107f30 0x7f7928074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928102880 msgr2=0x7f7928074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7928102880 0x7f7928074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7928107f30 msgr2=0x7f7928074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7928107f30 0x7f7928074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79281a6170 con 0x7f7928101ec0 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f79280740e0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f7925ffb700 10 client.?.objecter ms_handle_connect 0x7f7928101ec0 2024-03-11T21:30:33.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.646+0000 7f7925ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:33.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:33.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:33.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920027070 con 0x7f7928101ec0 2024-03-11T21:30:33.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79281a6400 con 0x7f7928101ec0 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f792f151700 10 client.29856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79281a68b0 con 0x7f7928101ec0 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f792f151700 1 librados: init done 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7920003fc0 con 0x7f7928101ec0 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920022980 con 0x7f7928101ec0 2024-03-11T21:30:33.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7908005100 con 0x7f7928101ec0 2024-03-11T21:30:33.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f792002a070 con 0x7f7928101ec0 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f791406f820 0x7f7914071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(232..232 src has 87..232) v4 ==== 10899+0+0 (secure 0 0 0) 0x7f792008ffc0 con 0x7f7928101ec0 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 osd_map(232..232 src has 87..232) v4 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 3 client.29856.objecter handle_osd_map got epochs [232,232] > 0 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 3 client.29856.objecter handle_osd_map decoding full epoch 232 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f791406f820 0x7f7914071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 20 client.29856.objecter dump_active .. 0 homeless 2024-03-11T21:30:33.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.682+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v232) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7920013070 con 0x7f7928101ec0 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 10 client.29856.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 1 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7908004730 0x7f7908004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 1 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 0 1 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 4 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 10 client.29856.objecter _send_command 1 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7908008b70 con 0x7f7908004730 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f791406f820 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f791406f820 0x7f7914071ce0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f7910005950 tx=0x7f79100058e0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7908004730 0x7f7908004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7908004730 0x7f7908004b70 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f7908004730 2024-03-11T21:30:33.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7908008b70 con 0x7f7908004730 2024-03-11T21:30:33.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:30:33.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 10 client.29856.objecter _finish_command 1 = system:0 2024-03-11T21:30:33.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 4 2024-03-11T21:30:33.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.686+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 0 1 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 10 client.29856.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 2 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 2 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080053e0 osd=0 3 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 0 2 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 4 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 10 client.29856.objecter _send_command 2 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7908008510 con 0x7f7908004730 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7908008510 con 0x7f7908004730 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f7925ffb700 10 client.29856.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 4 2024-03-11T21:30:33.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 0 2 2024-03-11T21:30:33.680 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 10 client.29856.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f7908002ca0 0x7f7908003130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 1 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 4 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 10 client.29856.objecter _send_command 3 2024-03-11T21:30:33.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79080038d0 con 0x7f7908002ca0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.702+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f7908002ca0 0x7f7908003130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f7908002ca0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f7908002ca0 0x7f7908003130 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.1 v2:172.21.15.143:6810/397942214 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f79080038d0 con 0x7f7908002ca0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 10 client.29856.objecter _finish_command 3 = system:0 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 4 2024-03-11T21:30:33.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.706+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 1 3 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 10 client.29856.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 4 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 4 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080089c0 osd=1 3 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 1 4 2024-03-11T21:30:33.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 4 2024-03-11T21:30:33.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 10 client.29856.objecter _send_command 4 2024-03-11T21:30:33.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f79080042e0 con 0x7f7908002ca0 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.1 v2:172.21.15.143:6810/397942214 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f79080042e0 con 0x7f7908002ca0 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 10 client.29856.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 4 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 1 4 2024-03-11T21:30:33.699 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:30:33.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 10 client.29856.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 5 2024-03-11T21:30:33.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f790800b240 0x7f790800d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 5 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 2 5 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 4 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 10 client.29856.objecter _send_command 5 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7908010490 con 0x7f790800b240 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f790800b240 0x7f790800d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f790800b240 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.722+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f790800b240 0x7f790800d630 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.726+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7908010490 con 0x7f790800b240 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.726+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:30:33.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.726+0000 7f7925ffb700 10 client.29856.objecter _finish_command 5 = system:0 2024-03-11T21:30:33.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.726+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 4 2024-03-11T21:30:33.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.726+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 2 5 2024-03-11T21:30:33.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 10 client.29856.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 6 2024-03-11T21:30:33.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 6 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f7908004570 osd=2 3 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 2 6 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 4 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 10 client.29856.objecter _send_command 6 2024-03-11T21:30:33.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.742+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7908003e50 con 0x7f790800b240 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7908003e50 con 0x7f790800b240 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 10 client.29856.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 4 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 2 6 2024-03-11T21:30:33.720 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:30:33.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 10 client.29856.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 7 2024-03-11T21:30:33.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f790800ecf0 0x7f790800f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 7 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 3 7 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 4 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 10 client.29856.objecter _send_command 7 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f790800f680 con 0x7f790800ecf0 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f790800ecf0 0x7f790800f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f790800ecf0 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f790800ecf0 0x7f790800f140 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f790800f680 con 0x7f790800ecf0 2024-03-11T21:30:33.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:30:33.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 10 client.29856.objecter _finish_command 7 = system:0 2024-03-11T21:30:33.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 4 2024-03-11T21:30:33.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.746+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 3 7 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 10 client.29856.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 8 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 8 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080040e0 osd=3 3 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 3 8 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 4 2024-03-11T21:30:33.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 10 client.29856.objecter _send_command 8 2024-03-11T21:30:33.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7908010090 con 0x7f790800ecf0 2024-03-11T21:30:33.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7908010090 con 0x7f790800ecf0 2024-03-11T21:30:33.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f7925ffb700 10 client.29856.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 4 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 3 8 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 10 client.29856.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 9 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7908012680 0x7f7908014a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 9 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 4 9 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 4 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 10 client.29856.objecter _send_command 9 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.762+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7908017920 con 0x7f7908012680 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7908012680 0x7f7908014a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f7908012680 2024-03-11T21:30:33.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7908012680 0x7f7908014a70 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7908017920 con 0x7f7908012680 2024-03-11T21:30:33.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:30:33.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 10 client.29856.objecter _finish_command 9 = system:0 2024-03-11T21:30:33.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 4 2024-03-11T21:30:33.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.766+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 4 9 2024-03-11T21:30:33.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:33.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 10 client.29856.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 10 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 10 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080102f0 osd=4 3 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 4 10 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 4 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 10 client.29856.objecter _send_command 10 2024-03-11T21:30:33.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.782+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7908014fb0 con 0x7f7908012680 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.786+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7908014fb0 con 0x7f7908012680 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.786+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.786+0000 7f7925ffb700 10 client.29856.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.786+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 4 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.786+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 4 10 2024-03-11T21:30:33.764 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:30:33.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 10 client.29856.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 11 2024-03-11T21:30:33.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f79080156d0 0x7f7908015b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 11 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 5 11 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 4 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 10 client.29856.objecter _send_command 11 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f790801ad40 con 0x7f79080156d0 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f79080156d0 0x7f7908015b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f79080156d0 2024-03-11T21:30:33.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7927fff700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f79080156d0 0x7f7908015b10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f790801ad40 con 0x7f79080156d0 2024-03-11T21:30:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:30:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 10 client.29856.objecter _finish_command 11 = system:0 2024-03-11T21:30:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 4 2024-03-11T21:30:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.790+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 5 11 2024-03-11T21:30:33.771 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 10 client.29856.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 12 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 12 2024-03-11T21:30:33.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f7908014fb0 osd=5 3 2024-03-11T21:30:33.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 5 12 2024-03-11T21:30:33.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 4 2024-03-11T21:30:33.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 10 client.29856.objecter _send_command 12 2024-03-11T21:30:33.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.810+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f790801ad40 con 0x7f79080156d0 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f790801ad40 con 0x7f79080156d0 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 10 client.29856.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 4 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 5 12 2024-03-11T21:30:33.789 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:30:33.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 10 client.29856.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 13 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f790801a130 0x7f790801a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 13 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 6 13 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 4 2024-03-11T21:30:33.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 10 client.29856.objecter _send_command 13 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f790801e0b0 con 0x7f790801a130 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f790801a130 0x7f790801a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f790801a130 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.814+0000 7f792d6ee700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f790801a130 0x7f790801a590 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.818+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f790801e0b0 con 0x7f790801a130 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.818+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:30:33.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.818+0000 7f7925ffb700 10 client.29856.objecter _finish_command 13 = system:0 2024-03-11T21:30:33.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.818+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 4 2024-03-11T21:30:33.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.818+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 6 13 2024-03-11T21:30:33.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 10 client.29856.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 14 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 14 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f790801ad40 osd=6 3 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 6 14 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 4 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 10 client.29856.objecter _send_command 14 2024-03-11T21:30:33.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.834+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f79080160a0 con 0x7f790801a130 2024-03-11T21:30:33.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f79080160a0 con 0x7f790801a130 2024-03-11T21:30:33.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 10 client.29856.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 4 2024-03-11T21:30:33.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 6 14 2024-03-11T21:30:33.814 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 10 client.29856.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 15 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f79080167c0 0x7f7908016c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 15 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 7 15 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 4 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 10 client.29856.objecter _send_command 15 2024-03-11T21:30:33.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7908017340 con 0x7f79080167c0 2024-03-11T21:30:33.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f79080167c0 0x7f7908016c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:33.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f7925ffb700 10 client.29856.objecter ms_handle_connect 0x7f79080167c0 2024-03-11T21:30:33.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.838+0000 7f792ceed700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f79080167c0 0x7f7908016c20 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:33.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.842+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7908017340 con 0x7f79080167c0 2024-03-11T21:30:33.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.842+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:30:33.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.842+0000 7f7925ffb700 10 client.29856.objecter _finish_command 15 = system:0 2024-03-11T21:30:33.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.842+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 4 2024-03-11T21:30:33.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.842+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 7 15 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 10 client.29856.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign -1 16 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 20 client.29856.objecter _get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 15 client.29856.objecter _session_command_op_remove -1 16 2024-03-11T21:30:33.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 20 client.29856.objecter get_session s=0x7f79080160a0 osd=7 3 2024-03-11T21:30:33.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 15 client.29856.objecter _session_command_op_assign 7 16 2024-03-11T21:30:33.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 4 2024-03-11T21:30:33.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 10 client.29856.objecter _send_command 16 2024-03-11T21:30:33.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.858+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7908017300 con 0x7f79080167c0 2024-03-11T21:30:33.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f7925ffb700 1 -- 172.21.15.143:0/3903967696 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7908017300 con 0x7f79080167c0 2024-03-11T21:30:33.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f7925ffb700 10 client.29856.objecter ms_dispatch 0x7f7928001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:30:33.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f7925ffb700 10 client.29856.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:30:33.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f7925ffb700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 4 2024-03-11T21:30:33.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f7925ffb700 15 client.29856.objecter _session_command_op_remove 7 16 2024-03-11T21:30:33.838 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:30:33.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 librados: watch_flush enter 2024-03-11T21:30:33.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 librados: watch_flush exit 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.0 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7908004730 msgr2=0x7f7908004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7908004730 0x7f7908004b70 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080053e0 osd=0 1 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.1 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f7908002ca0 msgr2=0x7f7908003130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6810/397942214,v1:172.21.15.143:6811/397942214] conn(0x7f7908002ca0 0x7f7908003130 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080089c0 osd=1 1 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.2 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f790800b240 msgr2=0x7f790800d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f790800b240 0x7f790800d630 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908004570 osd=2 1 2024-03-11T21:30:33.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.3 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f790800ecf0 msgr2=0x7f790800f140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f790800ecf0 0x7f790800f140 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080040e0 osd=3 1 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.4 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7908012680 msgr2=0x7f7908014a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7908012680 0x7f7908014a70 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080102f0 osd=4 1 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.5 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792d6ee700 1 -- 172.21.15.143:0/3903967696 reap_dead start 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f79080156d0 msgr2=0x7f7908015b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f79080156d0 0x7f7908015b10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792d6ee700 1 -- 172.21.15.143:0/3903967696 reap_dead start 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792d6ee700 1 -- 172.21.15.143:0/3903967696 reap_dead start 2024-03-11T21:30:33.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f7908014fb0 osd=5 1 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.6 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f790801a130 msgr2=0x7f790801a590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f790801a130 0x7f790801a590 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f790801ad40 osd=6 1 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter close_session for osd.7 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f79080167c0 msgr2=0x7f7908016c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f79080167c0 0x7f7908016c20 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter put_session s=0x7f79080160a0 osd=7 1 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 20 client.29856.objecter shutdown clearing up homeless session... 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 10 client.29856.objecter successfully canceled tick 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f791406f820 msgr2=0x7f7914071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f791406f820 0x7f7914071ce0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f7910005950 tx=0x7f79100058e0 comp rx=0 tx=0).stop 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 msgr2=0x7f79280740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:33.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 --2- 172.21.15.143:0/3903967696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7928101ec0 0x7f79280740e0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920004580 comp rx=0 tx=0).stop 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792d6ee700 1 -- 172.21.15.143:0/3903967696 reap_dead start 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 shutdown_connections 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 >> 172.21.15.143:0/3903967696 conn(0x7f7928078eb0 msgr2=0x7f7928108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 shutdown_connections 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 -- 172.21.15.143:0/3903967696 wait complete. 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:33.862+0000 7f792f151700 1 librados: shutdown 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:33.843 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:33.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:33.902 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:34.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:34.016 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.974601+0000 osd.4 (osd.4) 155 : cluster 0 6.17 scrub starts 2024-03-11T21:30:34.016 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:31.977037+0000 osd.4 (osd.4) 156 : cluster 0 6.17 scrub ok 2024-03-11T21:30:34.016 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:32.446346+0000 osd.7 (osd.7) 209 : cluster 0 9.15 scrub starts 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:32.448209+0000 osd.7 (osd.7) 210 : cluster 0 9.15 scrub ok 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: audit 2024-03-11T21:30:32.470796+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.022441+0000 osd.5 (osd.5) 173 : cluster 0 5.1b scrub starts 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.024677+0000 osd.5 (osd.5) 174 : cluster 0 5.1b scrub ok 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.223438+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: audit 2024-03-11T21:30:33.240180+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.143:0/320300398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: audit 2024-03-11T21:30:33.241318+0000 mon.a (mon.0) 1528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: audit 2024-03-11T21:30:33.471765+0000 mon.c (mon.2) 396 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[29284]: audit 2024-03-11T21:30:33.686301+0000 mon.a (mon.0) 1529 : audit 0 from='client.? 172.21.15.143:0/3903967696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:34.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.974601+0000 osd.4 (osd.4) 155 : cluster 0 6.17 scrub starts 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:31.977037+0000 osd.4 (osd.4) 156 : cluster 0 6.17 scrub ok 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:32.446346+0000 osd.7 (osd.7) 209 : cluster 0 9.15 scrub starts 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:32.448209+0000 osd.7 (osd.7) 210 : cluster 0 9.15 scrub ok 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: audit 2024-03-11T21:30:32.470796+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.022441+0000 osd.5 (osd.5) 173 : cluster 0 5.1b scrub starts 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.024677+0000 osd.5 (osd.5) 174 : cluster 0 5.1b scrub ok 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.223438+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: audit 2024-03-11T21:30:33.240180+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.143:0/320300398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: audit 2024-03-11T21:30:33.241318+0000 mon.a (mon.0) 1528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: audit 2024-03-11T21:30:33.471765+0000 mon.c (mon.2) 396 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.018 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:33 smithi143 bash[21512]: audit 2024-03-11T21:30:33.686301+0000 mon.a (mon.0) 1529 : audit 0 from='client.? 172.21.15.143:0/3903967696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.974601+0000 osd.4 (osd.4) 155 : cluster 0 6.17 scrub starts 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:31.977037+0000 osd.4 (osd.4) 156 : cluster 0 6.17 scrub ok 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:32.446346+0000 osd.7 (osd.7) 209 : cluster 0 9.15 scrub starts 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:32.448209+0000 osd.7 (osd.7) 210 : cluster 0 9.15 scrub ok 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: audit 2024-03-11T21:30:32.470796+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.022441+0000 osd.5 (osd.5) 173 : cluster 0 5.1b scrub starts 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.024677+0000 osd.5 (osd.5) 174 : cluster 0 5.1b scrub ok 2024-03-11T21:30:34.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.223438+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:30:34.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: audit 2024-03-11T21:30:33.240180+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.143:0/320300398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: audit 2024-03-11T21:30:33.241318+0000 mon.a (mon.0) 1528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: audit 2024-03-11T21:30:33.471765+0000 mon.c (mon.2) 396 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:33 smithi203 bash[23654]: audit 2024-03-11T21:30:33.686301+0000 mon.a (mon.0) 1529 : audit 0 from='client.? 172.21.15.143:0/3903967696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:34.031 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:34.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:34.162 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:34.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:34.288 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:34.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:34.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:34.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:34.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:34.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:34.791 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:32.859072+0000 osd.3 (osd.3) 177 : cluster 0 3.2 scrub starts 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:32.861906+0000 osd.3 (osd.3) 178 : cluster 0 3.2 scrub ok 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.274058+0000 mgr.y (mgr.14550) 311 : cluster 0 pgmap v336: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.823967+0000 osd.3 (osd.3) 179 : cluster 0 6.6 scrub starts 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:33.825471+0000 osd.3 (osd.3) 180 : cluster 0 6.6 scrub ok 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.219332+0000 mon.a (mon.0) 1530 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: audit 2024-03-11T21:30:34.227516+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.231329+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: audit 2024-03-11T21:30:34.239486+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.143:0/2723568854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: audit 2024-03-11T21:30:34.240306+0000 mon.a (mon.0) 1533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:34.792 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:34 smithi203 bash[23654]: audit 2024-03-11T21:30:34.472875+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:34.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:34.910 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:34.939 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:35.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:35.069 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:35.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:35.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:32.859072+0000 osd.3 (osd.3) 177 : cluster 0 3.2 scrub starts 2024-03-11T21:30:35.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:32.861906+0000 osd.3 (osd.3) 178 : cluster 0 3.2 scrub ok 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.274058+0000 mgr.y (mgr.14550) 311 : cluster 0 pgmap v336: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.823967+0000 osd.3 (osd.3) 179 : cluster 0 6.6 scrub starts 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:33.825471+0000 osd.3 (osd.3) 180 : cluster 0 6.6 scrub ok 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.219332+0000 mon.a (mon.0) 1530 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: audit 2024-03-11T21:30:34.227516+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.231329+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: audit 2024-03-11T21:30:34.239486+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.143:0/2723568854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: audit 2024-03-11T21:30:34.240306+0000 mon.a (mon.0) 1533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[21512]: audit 2024-03-11T21:30:34.472875+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:35.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:32.859072+0000 osd.3 (osd.3) 177 : cluster 0 3.2 scrub starts 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:32.861906+0000 osd.3 (osd.3) 178 : cluster 0 3.2 scrub ok 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.274058+0000 mgr.y (mgr.14550) 311 : cluster 0 pgmap v336: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.823967+0000 osd.3 (osd.3) 179 : cluster 0 6.6 scrub starts 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:33.825471+0000 osd.3 (osd.3) 180 : cluster 0 6.6 scrub ok 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.219332+0000 mon.a (mon.0) 1530 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: audit 2024-03-11T21:30:34.227516+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi143-71594-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.231329+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: audit 2024-03-11T21:30:34.239486+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.143:0/2723568854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: audit 2024-03-11T21:30:34.240306+0000 mon.a (mon.0) 1533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:35.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:34 smithi143 bash[29284]: audit 2024-03-11T21:30:34.472875+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:35.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:35.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:35.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:35.455 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:35.520 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:30:35.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:35.583 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:35.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:35.709 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:35.811 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:35.820 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.149005+0000 osd.2 (osd.2) 133 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:35.820 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.150757+0000 osd.2 (osd.2) 134 : cluster 0 purged_snaps scrub found 86 strays 2024-03-11T21:30:35.820 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.672988+0000 osd.2 (osd.2) 135 : cluster 0 5.8 scrub starts 2024-03-11T21:30:35.820 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: cluster 2024-03-11T21:30:34.675259+0000 osd.2 (osd.2) 136 : cluster 0 5.8 scrub ok 2024-03-11T21:30:35.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.149005+0000 osd.2 (osd.2) 133 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:35.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.150757+0000 osd.2 (osd.2) 134 : cluster 0 purged_snaps scrub found 86 strays 2024-03-11T21:30:35.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.672988+0000 osd.2 (osd.2) 135 : cluster 0 5.8 scrub starts 2024-03-11T21:30:35.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: cluster 2024-03-11T21:30:34.675259+0000 osd.2 (osd.2) 136 : cluster 0 5.8 scrub ok 2024-03-11T21:30:35.833 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:35.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.149005+0000 osd.2 (osd.2) 133 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.150757+0000 osd.2 (osd.2) 134 : cluster 0 purged_snaps scrub found 86 strays 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.672988+0000 osd.2 (osd.2) 135 : cluster 0 5.8 scrub starts 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: cluster 2024-03-11T21:30:34.675259+0000 osd.2 (osd.2) 136 : cluster 0 5.8 scrub ok 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: audit 2024-03-11T21:30:35.230905+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:35.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:35 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.240270+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:30:35.963 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:36.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:36.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: audit 2024-03-11T21:30:35.230905+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:36.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.240270+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:30:36.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: audit 2024-03-11T21:30:35.230905+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi143-71604-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:36.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:35 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.240270+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:30:36.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:36.235 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:36.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.362 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:36.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:36.492 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:36.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:36.615 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:36.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:36.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:36.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:36.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.061787+0000 osd.5 (osd.5) 175 : cluster 0 2.3 scrub starts 2024-03-11T21:30:36.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.064125+0000 osd.5 (osd.5) 176 : cluster 0 2.3 scrub ok 2024-03-11T21:30:36.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.274719+0000 mgr.y (mgr.14550) 312 : cluster 0 pgmap v339: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:35.474314+0000 mon.c (mon.2) 399 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:35.715705+0000 mon.c (mon.2) 400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.815724+0000 osd.3 (osd.3) 181 : cluster 0 12.8 scrub starts 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.816638+0000 osd.3 (osd.3) 182 : cluster 0 12.8 scrub ok 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.847663+0000 osd.1 (osd.1) 153 : cluster 0 4.2 scrub starts 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.849625+0000 osd.1 (osd.1) 154 : cluster 0 4.2 scrub ok 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:35.926762+0000 mon.c (mon.2) 401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.243504+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.251457+0000 mon.c (mon.2) 402 : audit 1 from='client.? 172.21.15.143:0/1194024154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.252014+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.406036+0000 mon.c (mon.2) 403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:30:36.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.407864+0000 mon.c (mon.2) 404 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.475343+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[21512]: audit 2024-03-11T21:30:36.732908+0000 mon.a (mon.0) 1538 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.061787+0000 osd.5 (osd.5) 175 : cluster 0 2.3 scrub starts 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.064125+0000 osd.5 (osd.5) 176 : cluster 0 2.3 scrub ok 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.274719+0000 mgr.y (mgr.14550) 312 : cluster 0 pgmap v339: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:35.474314+0000 mon.c (mon.2) 399 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:35.715705+0000 mon.c (mon.2) 400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.815724+0000 osd.3 (osd.3) 181 : cluster 0 12.8 scrub starts 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.816638+0000 osd.3 (osd.3) 182 : cluster 0 12.8 scrub ok 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.847663+0000 osd.1 (osd.1) 153 : cluster 0 4.2 scrub starts 2024-03-11T21:30:36.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.849625+0000 osd.1 (osd.1) 154 : cluster 0 4.2 scrub ok 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:35.926762+0000 mon.c (mon.2) 401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.243504+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.251457+0000 mon.c (mon.2) 402 : audit 1 from='client.? 172.21.15.143:0/1194024154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.252014+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.406036+0000 mon.c (mon.2) 403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.407864+0000 mon.c (mon.2) 404 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.475343+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:36 smithi143 bash[29284]: audit 2024-03-11T21:30:36.732908+0000 mon.a (mon.0) 1538 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:36.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:37.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.061787+0000 osd.5 (osd.5) 175 : cluster 0 2.3 scrub starts 2024-03-11T21:30:37.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.064125+0000 osd.5 (osd.5) 176 : cluster 0 2.3 scrub ok 2024-03-11T21:30:37.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.274719+0000 mgr.y (mgr.14550) 312 : cluster 0 pgmap v339: 332 pgs: 8 creating+peering, 32 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:35.474314+0000 mon.c (mon.2) 399 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:35.715705+0000 mon.c (mon.2) 400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.815724+0000 osd.3 (osd.3) 181 : cluster 0 12.8 scrub starts 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.816638+0000 osd.3 (osd.3) 182 : cluster 0 12.8 scrub ok 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.847663+0000 osd.1 (osd.1) 153 : cluster 0 4.2 scrub starts 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.849625+0000 osd.1 (osd.1) 154 : cluster 0 4.2 scrub ok 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:35.926762+0000 mon.c (mon.2) 401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.243504+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.251457+0000 mon.c (mon.2) 402 : audit 1 from='client.? 172.21.15.143:0/1194024154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.252014+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.406036+0000 mon.c (mon.2) 403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.407864+0000 mon.c (mon.2) 404 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.475343+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:37.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:36 smithi203 bash[23654]: audit 2024-03-11T21:30:36.732908+0000 mon.a (mon.0) 1538 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:37.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.136 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:37.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:37.274 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:37.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:37.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:37.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:37.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:37.661 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:37.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:37.793 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:37.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:37.907 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.922739+0000 osd.0 (osd.0) 161 : cluster 0 3.1f scrub starts 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:35.925532+0000 osd.0 (osd.0) 162 : cluster 0 3.1f scrub ok 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.661417+0000 osd.6 (osd.6) 121 : cluster 0 5.11 scrub starts 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.663910+0000 osd.6 (osd.6) 122 : cluster 0 5.11 scrub ok 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.775886+0000 osd.3 (osd.3) 183 : cluster 0 6.1f scrub starts 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.777103+0000 osd.3 (osd.3) 184 : cluster 0 6.1f scrub ok 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.868845+0000 osd.1 (osd.1) 155 : cluster 0 12.18 scrub starts 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.871157+0000 osd.1 (osd.1) 156 : cluster 0 12.18 scrub ok 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.006960+0000 osd.5 (osd.5) 177 : cluster 0 6.19 scrub starts 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.009596+0000 osd.5 (osd.5) 178 : cluster 0 6.19 scrub ok 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: audit 2024-03-11T21:30:37.239512+0000 mon.a (mon.0) 1539 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.245973+0000 mon.a (mon.0) 1540 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:30:37.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: audit 2024-03-11T21:30:37.252768+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[21512]: audit 2024-03-11T21:30:37.476948+0000 mon.c (mon.2) 406 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.922739+0000 osd.0 (osd.0) 161 : cluster 0 3.1f scrub starts 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:35.925532+0000 osd.0 (osd.0) 162 : cluster 0 3.1f scrub ok 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.661417+0000 osd.6 (osd.6) 121 : cluster 0 5.11 scrub starts 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.663910+0000 osd.6 (osd.6) 122 : cluster 0 5.11 scrub ok 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.775886+0000 osd.3 (osd.3) 183 : cluster 0 6.1f scrub starts 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.777103+0000 osd.3 (osd.3) 184 : cluster 0 6.1f scrub ok 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.868845+0000 osd.1 (osd.1) 155 : cluster 0 12.18 scrub starts 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.871157+0000 osd.1 (osd.1) 156 : cluster 0 12.18 scrub ok 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.006960+0000 osd.5 (osd.5) 177 : cluster 0 6.19 scrub starts 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.009596+0000 osd.5 (osd.5) 178 : cluster 0 6.19 scrub ok 2024-03-11T21:30:37.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: audit 2024-03-11T21:30:37.239512+0000 mon.a (mon.0) 1539 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.245973+0000 mon.a (mon.0) 1540 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:30:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: audit 2024-03-11T21:30:37.252768+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:37 smithi143 bash[29284]: audit 2024-03-11T21:30:37.476948+0000 mon.c (mon.2) 406 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:37.922 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.922739+0000 osd.0 (osd.0) 161 : cluster 0 3.1f scrub starts 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:35.925532+0000 osd.0 (osd.0) 162 : cluster 0 3.1f scrub ok 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.661417+0000 osd.6 (osd.6) 121 : cluster 0 5.11 scrub starts 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.663910+0000 osd.6 (osd.6) 122 : cluster 0 5.11 scrub ok 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.775886+0000 osd.3 (osd.3) 183 : cluster 0 6.1f scrub starts 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.777103+0000 osd.3 (osd.3) 184 : cluster 0 6.1f scrub ok 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.868845+0000 osd.1 (osd.1) 155 : cluster 0 12.18 scrub starts 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.871157+0000 osd.1 (osd.1) 156 : cluster 0 12.18 scrub ok 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.006960+0000 osd.5 (osd.5) 177 : cluster 0 6.19 scrub starts 2024-03-11T21:30:38.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.009596+0000 osd.5 (osd.5) 178 : cluster 0 6.19 scrub ok 2024-03-11T21:30:38.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: audit 2024-03-11T21:30:37.239512+0000 mon.a (mon.0) 1539 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi143-71594-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:38.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.245973+0000 mon.a (mon.0) 1540 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:30:38.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: audit 2024-03-11T21:30:37.252768+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:38.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:37 smithi203 bash[23654]: audit 2024-03-11T21:30:37.476948+0000 mon.c (mon.2) 406 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:38.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:38.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:38.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:38.177 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: Running main() from gmock_main.cc 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] Global test environment set-up. 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.TooBig (16707 ms) 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-11T21:30:38.236 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (10442 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (6375 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (5294 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (6424 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (6375 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4329 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (6849 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4517 ms) 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-11T21:30:38.237 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (6094 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3877 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2981 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3030 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3350 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.Flush (3375 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3023 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2607 ms) 2024-03-11T21:30:38.238 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3025 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2998 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3015 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2984 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2986 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3013 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2995 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3008 ms) 2024-03-11T21:30:38.239 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-11T21:30:38.240 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3018 ms) 2024-03-11T21:30:38.240 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 26 tests from LibRadosAio (122691 ms total) 2024-03-11T21:30:38.240 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: 2024-03-11T21:30:38.240 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-11T21:30:38.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:38.305 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:38.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:38.429 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:38.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.561 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:38.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:38.674 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:38.689 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:38.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.940939+0000 osd.0 (osd.0) 163 : cluster 0 9.1e scrub starts 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:36.943159+0000 osd.0 (osd.0) 164 : cluster 0 9.1e scrub ok 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.275297+0000 mgr.y (mgr.14550) 313 : cluster 0 pgmap v342: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.345211+0000 osd.7 (osd.7) 211 : cluster 0 5.4 scrub starts 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.346204+0000 osd.7 (osd.7) 212 : cluster 0 5.4 scrub ok 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.989478+0000 osd.5 (osd.5) 179 : cluster 0 2.2 scrub starts 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.991571+0000 osd.5 (osd.5) 180 : cluster 0 2.2 scrub ok 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: audit 2024-03-11T21:30:38.247776+0000 mon.a (mon.0) 1542 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: cluster 2024-03-11T21:30:38.254403+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: audit 2024-03-11T21:30:38.299804+0000 mon.a (mon.0) 1544 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: audit 2024-03-11T21:30:38.300818+0000 mon.a (mon.0) 1545 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: audit 2024-03-11T21:30:38.301400+0000 mon.a (mon.0) 1546 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:38.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:38 smithi203 bash[23654]: audit 2024-03-11T21:30:38.478158+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:38.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:38.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- 172.21.15.143:0/2950692809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 msgr2=0x7f752c1022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- 172.21.15.143:0/2950692809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c1022f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f751c009990 tx=0x7f751c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- 172.21.15.143:0/2950692809 shutdown_connections 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- 172.21.15.143:0/2950692809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c102830 0x7f752c10ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- 172.21.15.143:0/2950692809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c107ea0 0x7f752c108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- 172.21.15.143:0/2950692809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c1022f0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- 172.21.15.143:0/2950692809 >> 172.21.15.143:0/2950692809 conn(0x7f752c078eb0 msgr2=0x7f752c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- 172.21.15.143:0/2950692809 shutdown_connections 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- 172.21.15.143:0/2950692809 wait complete. 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 librados: starting msgr at 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 librados: starting objecter 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 Processor -- start 2024-03-11T21:30:38.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- start start 2024-03-11T21:30:38.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 librados: setting wanted keys 2024-03-11T21:30:38.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 librados: calling monclient init 2024-03-11T21:30:38.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c102830 0x7f752c19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 0x7f752c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c111620 con 0x7f752c101e90 2024-03-11T21:30:38.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c1114a0 con 0x7f752c102830 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7533aca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752c1117a0 con 0x7f752c107ea0 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7532067700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 0x7f752c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52168/0 (socket says 172.21.15.143:52168) 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 -- 172.21.15.143:0/3379093067 learned_addr learned my addr 172.21.15.143:0/3379093067 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531065700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c102830 0x7f752c19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 -- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 msgr2=0x7f752c197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 0x7f752c197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 -- 172.21.15.143:0/3379093067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c102830 msgr2=0x7f752c19dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c102830 0x7f752c19dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 -- 172.21.15.143:0/3379093067 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c1980d0 con 0x7f752c101e90 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7532067700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 0x7f752c197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7531866700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f752800c9b0 tx=0x7f752800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:38.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7522ffd700 10 client.?.objecter ms_handle_connect 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.929+0000 7f7522ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528007b00 con 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c1983c0 con 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7528007ca0 con 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528018660 con 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7533aca700 10 client.29877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c1a4680 con 0x7f752c101e90 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7533aca700 1 librados: init done 2024-03-11T21:30:38.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c04faa0 con 0x7f752c101e90 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f752800f450 con 0x7f752c101e90 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 0x7f7518071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(237..237 src has 87..237) v4 ==== 10942+0+0 (secure 0 0 0) 0x7f752808ef40 con 0x7f752c101e90 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 10 client.29877.objecter ms_dispatch 0x7f752c001550 osd_map(237..237 src has 87..237) v4 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 3 client.29877.objecter handle_osd_map got epochs [237,237] > 0 2024-03-11T21:30:38.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 3 client.29877.objecter handle_osd_map decoding full epoch 237 2024-03-11T21:30:38.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7531065700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 0x7f7518071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 20 client.29877.objecter dump_active .. 0 homeless 2024-03-11T21:30:38.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7531065700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 0x7f7518071ce0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f752c1079e0 tx=0x7f751c01a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:38.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.933+0000 7f7522ffd700 10 client.29877.objecter ms_handle_connect 0x7f751806f820 2024-03-11T21:30:38.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.941+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7528014030 con 0x7f752c101e90 2024-03-11T21:30:38.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.951 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:38.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- 172.21.15.143:0/451083978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca006a6a0 msgr2=0x7f5ca006aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/451083978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca006a6a0 0x7f5ca006aa80 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009990 tx=0x7f5c90009ca0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- 172.21.15.143:0/451083978 shutdown_connections 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/451083978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca006a6a0 0x7f5ca006aa80 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/451083978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca0107220 0x7f5ca010edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/451083978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 0x7f5ca0106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- 172.21.15.143:0/451083978 >> 172.21.15.143:0/451083978 conn(0x7f5ca0077de0 msgr2=0x7f5ca00781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- 172.21.15.143:0/451083978 shutdown_connections 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- 172.21.15.143:0/451083978 wait complete. 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 librados: starting msgr at 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 librados: starting objecter 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 Processor -- start 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- start start 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 librados: setting wanted keys 2024-03-11T21:30:38.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 librados: calling monclient init 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 0x7f5ca01012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 0x7f5ca0101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0111630 con 0x7f5ca0107220 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01114b0 con 0x7f5ca006b050 2024-03-11T21:30:38.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca87bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01117b0 con 0x7f5ca006a6a0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47012/0 (socket says 172.21.15.143:47012) 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 -- 172.21.15.143:0/476846038 learned_addr learned my addr 172.21.15.143:0/476846038 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6d59700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 0x7f5ca0101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca5d57700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 0x7f5ca01012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 -- 172.21.15.143:0/476846038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 msgr2=0x7f5ca01012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 0x7f5ca01012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 -- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 msgr2=0x7f5ca0101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 0x7f5ca0101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 -- 172.21.15.143:0/476846038 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca01a6140 con 0x7f5ca006a6a0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6d59700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 0x7f5ca0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5ca6558700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f5c900057f0 tx=0x7f5c90005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:38.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5c977fe700 10 client.?.objecter ms_handle_connect 0x7f5ca006a6a0 2024-03-11T21:30:38.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5c977fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.977+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90026070 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca01a63d0 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca87bc700 10 client.41084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca01a6900 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca87bc700 1 librados: init done 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5c90021410 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90021e60 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5c90029070 con 0x7f5ca006a6a0 2024-03-11T21:30:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca01035b0 con 0x7f5ca006a6a0 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 0x7f5c8c071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(237..237 src has 87..237) v4 ==== 10942+0+0 (secure 0 0 0) 0x7f5c9008fa40 con 0x7f5ca006a6a0 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 10 client.41084.objecter ms_dispatch 0x7f5ca0001550 osd_map(237..237 src has 87..237) v4 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 3 client.41084.objecter handle_osd_map got epochs [237,237] > 0 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 3 client.41084.objecter handle_osd_map decoding full epoch 237 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca5d57700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 0x7f5c8c071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 20 client.41084.objecter dump_active .. 0 homeless 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5c977fe700 10 client.41084.objecter ms_handle_connect 0x7f5c8c06f6f0 2024-03-11T21:30:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.981+0000 7f5ca5d57700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 0x7f5c8c071bb0 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f5ca01086c0 tx=0x7f5c9c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:30:38.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:38.985+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5c90013070 con 0x7f5ca006a6a0 2024-03-11T21:30:39.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.080 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:39.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.165+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f752c1a4910 con 0x7f752c101e90 2024-03-11T21:30:39.183 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.184 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.205+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f5ca004faa0 con 0x7f5ca006a6a0 2024-03-11T21:30:39.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.940939+0000 osd.0 (osd.0) 163 : cluster 0 9.1e scrub starts 2024-03-11T21:30:39.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:36.943159+0000 osd.0 (osd.0) 164 : cluster 0 9.1e scrub ok 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.275297+0000 mgr.y (mgr.14550) 313 : cluster 0 pgmap v342: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.345211+0000 osd.7 (osd.7) 211 : cluster 0 5.4 scrub starts 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.346204+0000 osd.7 (osd.7) 212 : cluster 0 5.4 scrub ok 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.989478+0000 osd.5 (osd.5) 179 : cluster 0 2.2 scrub starts 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.991571+0000 osd.5 (osd.5) 180 : cluster 0 2.2 scrub ok 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: audit 2024-03-11T21:30:38.247776+0000 mon.a (mon.0) 1542 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: cluster 2024-03-11T21:30:38.254403+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: audit 2024-03-11T21:30:38.299804+0000 mon.a (mon.0) 1544 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: audit 2024-03-11T21:30:38.300818+0000 mon.a (mon.0) 1545 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: audit 2024-03-11T21:30:38.301400+0000 mon.a (mon.0) 1546 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:39.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[29284]: audit 2024-03-11T21:30:38.478158+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.940939+0000 osd.0 (osd.0) 163 : cluster 0 9.1e scrub starts 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:36.943159+0000 osd.0 (osd.0) 164 : cluster 0 9.1e scrub ok 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.275297+0000 mgr.y (mgr.14550) 313 : cluster 0 pgmap v342: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.345211+0000 osd.7 (osd.7) 211 : cluster 0 5.4 scrub starts 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.346204+0000 osd.7 (osd.7) 212 : cluster 0 5.4 scrub ok 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.989478+0000 osd.5 (osd.5) 179 : cluster 0 2.2 scrub starts 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.991571+0000 osd.5 (osd.5) 180 : cluster 0 2.2 scrub ok 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: audit 2024-03-11T21:30:38.247776+0000 mon.a (mon.0) 1542 : audit 1 from='client.? 172.21.15.143:0/1861040220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi143-71604-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: cluster 2024-03-11T21:30:38.254403+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: audit 2024-03-11T21:30:38.299804+0000 mon.a (mon.0) 1544 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: audit 2024-03-11T21:30:38.300818+0000 mon.a (mon.0) 1545 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: audit 2024-03-11T21:30:38.301400+0000 mon.a (mon.0) 1546 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:39.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:38 smithi143 bash[21512]: audit 2024-03-11T21:30:38.478158+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:39.203 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.245+0000 7f7522ffd700 1 -- 172.21.15.143:0/3379093067 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v238) v1 ==== 101+0+0 (secure 0 0 0) 0x7f752805b670 con 0x7f752c101e90 2024-03-11T21:30:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 10 librados: watch_flush enter 2024-03-11T21:30:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 10 librados: watch_flush exit 2024-03-11T21:30:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 20 client.29877.objecter shutdown clearing up homeless session... 2024-03-11T21:30:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 10 client.29877.objecter successfully canceled tick 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 msgr2=0x7f7518071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 0x7f7518071ce0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f752c1079e0 tx=0x7f751c01a040 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 msgr2=0x7f752c19d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f752800c9b0 tx=0x7f752800cd70 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 shutdown_connections 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f751806f820 0x7f7518071ce0 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f752c101e90 0x7f752c19d7b0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f752c107ea0 0x7f752c197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 --2- 172.21.15.143:0/3379093067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752c102830 0x7f752c19dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 >> 172.21.15.143:0/3379093067 conn(0x7f752c078eb0 msgr2=0x7f752c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 shutdown_connections 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 -- 172.21.15.143:0/3379093067 wait complete. 2024-03-11T21:30:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f7533aca700 1 librados: shutdown 2024-03-11T21:30:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:30:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.249+0000 7f5c977fe700 1 -- 172.21.15.143:0/476846038 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v238) v1 ==== 113+0+0 (secure 0 0 0) 0x7f5c9005fc40 con 0x7f5ca006a6a0 2024-03-11T21:30:39.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.253+0000 7f5ca87bc700 10 librados: watch_flush enter 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 10 librados: watch_flush exit 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 20 client.41084.objecter shutdown clearing up homeless session... 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 10 client.41084.objecter successfully canceled tick 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 msgr2=0x7f5c8c071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 0x7f5c8c071bb0 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f5ca01086c0 tx=0x7f5c9c009260 comp rx=0 tx=0).stop 2024-03-11T21:30:39.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 msgr2=0x7f5ca0100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:30:39.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f5c900057f0 tx=0x7f5c90005e80 comp rx=0 tx=0).stop 2024-03-11T21:30:39.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 shutdown_connections 2024-03-11T21:30:39.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5c8c06f6f0 0x7f5c8c071bb0 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ca0107220 0x7f5ca0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ca006a6a0 0x7f5ca0100d90 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 --2- 172.21.15.143:0/476846038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ca006b050 0x7f5ca01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 >> 172.21.15.143:0/476846038 conn(0x7f5ca0077de0 msgr2=0x7f5ca00feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 shutdown_connections 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 -- 172.21.15.143:0/476846038 wait complete. 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:30:39.261+0000 7f5ca87bc700 1 librados: shutdown 2024-03-11T21:30:39.240 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:30:39.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:39.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:39.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.460 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:39.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:30:39.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:39.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:39.720 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.960232+0000 osd.0 (osd.0) 165 : cluster 0 12.1d scrub starts 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:37.962375+0000 osd.0 (osd.0) 166 : cluster 0 12.1d scrub ok 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:38.388124+0000 osd.7 (osd.7) 213 : cluster 0 5.10 scrub starts 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:38.390398+0000 osd.7 (osd.7) 214 : cluster 0 5.10 scrub ok 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:38.411882+0000 mgr.y (mgr.14550) 314 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:38.860908+0000 osd.4 (osd.4) 157 : cluster 0 3.14 scrub starts 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:38.862993+0000 osd.4 (osd.4) 158 : cluster 0 3.14 scrub ok 2024-03-11T21:30:39.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.005511+0000 osd.5 (osd.5) 181 : cluster 0 9.12 scrub starts 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.006800+0000 osd.5 (osd.5) 182 : cluster 0 9.12 scrub ok 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.168628+0000 mon.a (mon.0) 1547 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.212138+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.143:0/476846038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.212409+0000 mon.a (mon.0) 1548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.248543+0000 mon.a (mon.0) 1549 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.250098+0000 mon.a (mon.0) 1550 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.250179+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.250220+0000 mon.a (mon.0) 1552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.257215+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:30:39.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.259345+0000 mon.a (mon.0) 1554 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[21512]: audit 2024-03-11T21:30:39.479237+0000 mon.c (mon.2) 409 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:39.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:39.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.960232+0000 osd.0 (osd.0) 165 : cluster 0 12.1d scrub starts 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:37.962375+0000 osd.0 (osd.0) 166 : cluster 0 12.1d scrub ok 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:38.388124+0000 osd.7 (osd.7) 213 : cluster 0 5.10 scrub starts 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:38.390398+0000 osd.7 (osd.7) 214 : cluster 0 5.10 scrub ok 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:38.411882+0000 mgr.y (mgr.14550) 314 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:38.860908+0000 osd.4 (osd.4) 157 : cluster 0 3.14 scrub starts 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:38.862993+0000 osd.4 (osd.4) 158 : cluster 0 3.14 scrub ok 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.005511+0000 osd.5 (osd.5) 181 : cluster 0 9.12 scrub starts 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.006800+0000 osd.5 (osd.5) 182 : cluster 0 9.12 scrub ok 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.168628+0000 mon.a (mon.0) 1547 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.212138+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.143:0/476846038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.212409+0000 mon.a (mon.0) 1548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.248543+0000 mon.a (mon.0) 1549 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.250098+0000 mon.a (mon.0) 1550 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.250179+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.250220+0000 mon.a (mon.0) 1552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.257215+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.259345+0000 mon.a (mon.0) 1554 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:39 smithi143 bash[29284]: audit 2024-03-11T21:30:39.479237+0000 mon.c (mon.2) 409 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:39.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.960232+0000 osd.0 (osd.0) 165 : cluster 0 12.1d scrub starts 2024-03-11T21:30:39.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:37.962375+0000 osd.0 (osd.0) 166 : cluster 0 12.1d scrub ok 2024-03-11T21:30:39.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:38.388124+0000 osd.7 (osd.7) 213 : cluster 0 5.10 scrub starts 2024-03-11T21:30:39.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:38.390398+0000 osd.7 (osd.7) 214 : cluster 0 5.10 scrub ok 2024-03-11T21:30:39.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:38.411882+0000 mgr.y (mgr.14550) 314 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:38.860908+0000 osd.4 (osd.4) 157 : cluster 0 3.14 scrub starts 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:38.862993+0000 osd.4 (osd.4) 158 : cluster 0 3.14 scrub ok 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.005511+0000 osd.5 (osd.5) 181 : cluster 0 9.12 scrub starts 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.006800+0000 osd.5 (osd.5) 182 : cluster 0 9.12 scrub ok 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.168628+0000 mon.a (mon.0) 1547 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.212138+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.143:0/476846038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.212409+0000 mon.a (mon.0) 1548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.248543+0000 mon.a (mon.0) 1549 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.250098+0000 mon.a (mon.0) 1550 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi143-71594-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.250179+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.143:0/3379093067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.250220+0000 mon.a (mon.0) 1552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.257215+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.259345+0000 mon.a (mon.0) 1554 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:39.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:39 smithi203 bash[23654]: audit 2024-03-11T21:30:39.479237+0000 mon.c (mon.2) 409 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:39.982 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:40.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.113 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:40.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:40.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:40.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:40.380 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:40.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:40.506 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:40.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:40.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.751 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.275865+0000 mgr.y (mgr.14550) 315 : cluster 0 pgmap v345: 300 pgs: 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.751468+0000 osd.2 (osd.2) 137 : cluster 0 5.a scrub starts 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.753891+0000 osd.2 (osd.2) 138 : cluster 0 5.a scrub ok 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.250272+0000 mon.a (mon.0) 1555 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: audit 2024-03-11T21:30:40.480640+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: audit 2024-03-11T21:30:40.481903+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.708795+0000 mon.a (mon.0) 1556 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: audit 2024-03-11T21:30:40.709809+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: audit 2024-03-11T21:30:40.711612+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.143:0/2373953823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[29284]: audit 2024-03-11T21:30:40.712331+0000 mon.a (mon.0) 1558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.275865+0000 mgr.y (mgr.14550) 315 : cluster 0 pgmap v345: 300 pgs: 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.751468+0000 osd.2 (osd.2) 137 : cluster 0 5.a scrub starts 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.753891+0000 osd.2 (osd.2) 138 : cluster 0 5.a scrub ok 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.250272+0000 mon.a (mon.0) 1555 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: audit 2024-03-11T21:30:40.480640+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: audit 2024-03-11T21:30:40.481903+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.708795+0000 mon.a (mon.0) 1556 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:30:40.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: audit 2024-03-11T21:30:40.709809+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: audit 2024-03-11T21:30:40.711612+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.143:0/2373953823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:40.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:40 smithi143 bash[21512]: audit 2024-03-11T21:30:40.712331+0000 mon.a (mon.0) 1558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:40.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:40.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:40.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:40.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.275865+0000 mgr.y (mgr.14550) 315 : cluster 0 pgmap v345: 300 pgs: 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.751468+0000 osd.2 (osd.2) 137 : cluster 0 5.a scrub starts 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.753891+0000 osd.2 (osd.2) 138 : cluster 0 5.a scrub ok 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.250272+0000 mon.a (mon.0) 1555 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: audit 2024-03-11T21:30:40.480640+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: audit 2024-03-11T21:30:40.481903+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.708795+0000 mon.a (mon.0) 1556 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: audit 2024-03-11T21:30:40.709809+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:30:40.998 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: audit 2024-03-11T21:30:40.711612+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.143:0/2373953823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:40.999 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:40 smithi203 bash[23654]: audit 2024-03-11T21:30:40.712331+0000 mon.a (mon.0) 1558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:41.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:41.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:41.144 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:41.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:41.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:41.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:41.393 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:41.469 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:41.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:41.524 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:41.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:41.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:41.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.785 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:41.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.923259+0000 osd.0 (osd.0) 167 : cluster 0 12.3 scrub starts 2024-03-11T21:30:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:39.934416+0000 osd.0 (osd.0) 168 : cluster 0 12.3 scrub ok 2024-03-11T21:30:41.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.749328+0000 osd.3 (osd.3) 185 : cluster 0 9.1b scrub starts 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.750731+0000 osd.3 (osd.3) 186 : cluster 0 9.1b scrub ok 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.894670+0000 osd.1 (osd.1) 157 : cluster 0 39.1c scrub starts 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.897134+0000 osd.1 (osd.1) 158 : cluster 0 39.1c scrub ok 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.908090+0000 osd.4 (osd.4) 159 : cluster 0 6.1e scrub starts 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:40.909351+0000 osd.4 (osd.4) 160 : cluster 0 6.1e scrub ok 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.218117+0000 mon.a (mon.0) 1559 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.218426+0000 mon.a (mon.0) 1560 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.218582+0000 mon.a (mon.0) 1561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.223591+0000 mon.a (mon.0) 1562 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.226358+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.227982+0000 mon.c (mon.2) 414 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.229345+0000 mon.c (mon.2) 415 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.233522+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.276902+0000 mon.c (mon.2) 416 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[21512]: audit 2024-03-11T21:30:41.278263+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.923259+0000 osd.0 (osd.0) 167 : cluster 0 12.3 scrub starts 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:39.934416+0000 osd.0 (osd.0) 168 : cluster 0 12.3 scrub ok 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.749328+0000 osd.3 (osd.3) 185 : cluster 0 9.1b scrub starts 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.750731+0000 osd.3 (osd.3) 186 : cluster 0 9.1b scrub ok 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.894670+0000 osd.1 (osd.1) 157 : cluster 0 39.1c scrub starts 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.897134+0000 osd.1 (osd.1) 158 : cluster 0 39.1c scrub ok 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.908090+0000 osd.4 (osd.4) 159 : cluster 0 6.1e scrub starts 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:40.909351+0000 osd.4 (osd.4) 160 : cluster 0 6.1e scrub ok 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.218117+0000 mon.a (mon.0) 1559 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.218426+0000 mon.a (mon.0) 1560 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:30:41.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.218582+0000 mon.a (mon.0) 1561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.223591+0000 mon.a (mon.0) 1562 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.226358+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.227982+0000 mon.c (mon.2) 414 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.229345+0000 mon.c (mon.2) 415 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.233522+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.276902+0000 mon.c (mon.2) 416 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:41.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:41 smithi143 bash[29284]: audit 2024-03-11T21:30:41.278263+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:42.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.923259+0000 osd.0 (osd.0) 167 : cluster 0 12.3 scrub starts 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:39.934416+0000 osd.0 (osd.0) 168 : cluster 0 12.3 scrub ok 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.749328+0000 osd.3 (osd.3) 185 : cluster 0 9.1b scrub starts 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.750731+0000 osd.3 (osd.3) 186 : cluster 0 9.1b scrub ok 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.894670+0000 osd.1 (osd.1) 157 : cluster 0 39.1c scrub starts 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.897134+0000 osd.1 (osd.1) 158 : cluster 0 39.1c scrub ok 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.908090+0000 osd.4 (osd.4) 159 : cluster 0 6.1e scrub starts 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:40.909351+0000 osd.4 (osd.4) 160 : cluster 0 6.1e scrub ok 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.218117+0000 mon.a (mon.0) 1559 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi143-71594-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.218426+0000 mon.a (mon.0) 1560 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.218582+0000 mon.a (mon.0) 1561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi143-71604-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.223591+0000 mon.a (mon.0) 1562 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:30:42.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.226358+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:42.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.227982+0000 mon.c (mon.2) 414 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:42.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.229345+0000 mon.c (mon.2) 415 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:42.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.233522+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:30:42.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.276902+0000 mon.c (mon.2) 416 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:42.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:41 smithi203 bash[23654]: audit 2024-03-11T21:30:41.278263+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:30:42.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.049 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:42.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:42.178 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:42.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.304 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:42.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:42.438 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.439 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:42.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:42.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:42.569 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:42.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:42.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:42.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.276518+0000 mgr.y (mgr.14550) 316 : cluster 0 pgmap v348: 340 pgs: 40 unknown, 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.878583+0000 osd.0 (osd.0) 169 : cluster 0 12.1c scrub starts 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.880849+0000 osd.0 (osd.0) 170 : cluster 0 12.1c scrub ok 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.895797+0000 osd.1 (osd.1) 159 : cluster 0 12.13 scrub starts 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:41.899049+0000 osd.1 (osd.1) 160 : cluster 0 12.13 scrub ok 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: audit 2024-03-11T21:30:42.220190+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:30:42.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: audit 2024-03-11T21:30:42.220299+0000 mon.a (mon.0) 1566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:30:42.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.224455+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:30:42.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:42 smithi203 bash[23654]: audit 2024-03-11T21:30:42.227974+0000 mon.c (mon.2) 417 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:42.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:42.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:42.991 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.276518+0000 mgr.y (mgr.14550) 316 : cluster 0 pgmap v348: 340 pgs: 40 unknown, 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.878583+0000 osd.0 (osd.0) 169 : cluster 0 12.1c scrub starts 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.880849+0000 osd.0 (osd.0) 170 : cluster 0 12.1c scrub ok 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.895797+0000 osd.1 (osd.1) 159 : cluster 0 12.13 scrub starts 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:41.899049+0000 osd.1 (osd.1) 160 : cluster 0 12.13 scrub ok 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: audit 2024-03-11T21:30:42.220190+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: audit 2024-03-11T21:30:42.220299+0000 mon.a (mon.0) 1566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.224455+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:30:43.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[21512]: audit 2024-03-11T21:30:42.227974+0000 mon.c (mon.2) 417 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.276518+0000 mgr.y (mgr.14550) 316 : cluster 0 pgmap v348: 340 pgs: 40 unknown, 300 active+clean; 616 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.878583+0000 osd.0 (osd.0) 169 : cluster 0 12.1c scrub starts 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.880849+0000 osd.0 (osd.0) 170 : cluster 0 12.1c scrub ok 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.895797+0000 osd.1 (osd.1) 159 : cluster 0 12.13 scrub starts 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:41.899049+0000 osd.1 (osd.1) 160 : cluster 0 12.13 scrub ok 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: audit 2024-03-11T21:30:42.220190+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: audit 2024-03-11T21:30:42.220299+0000 mon.a (mon.0) 1566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.224455+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:30:43.004 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:42 smithi143 bash[29284]: audit 2024-03-11T21:30:42.227974+0000 mon.c (mon.2) 417 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:30:43.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:43.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:43.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:43.233 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:43.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:43.362 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:43.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:43.493 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:43.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:43.622 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:43.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:43.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:43.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:43.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:43.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:43.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.545087+0000 osd.6 (osd.6) 123 : cluster 0 12.5 deep-scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.546459+0000 osd.6 (osd.6) 124 : cluster 0 12.5 deep-scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.644579+0000 osd.2 (osd.2) 139 : cluster 0 12.a scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.647183+0000 osd.2 (osd.2) 140 : cluster 0 12.a scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.816469+0000 osd.3 (osd.3) 187 : cluster 0 9.2 scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.818567+0000 osd.3 (osd.3) 188 : cluster 0 9.2 scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.845028+0000 osd.0 (osd.0) 171 : cluster 0 9.5 scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.847464+0000 osd.0 (osd.0) 172 : cluster 0 9.5 scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.863944+0000 osd.1 (osd.1) 161 : cluster 0 12.7 deep-scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.866560+0000 osd.1 (osd.1) 162 : cluster 0 12.7 deep-scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.916193+0000 osd.4 (osd.4) 161 : cluster 0 12.2 scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.918553+0000 osd.4 (osd.4) 162 : cluster 0 12.2 scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.935945+0000 osd.5 (osd.5) 183 : cluster 0 3.1c scrub starts 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:42.940243+0000 osd.5 (osd.5) 184 : cluster 0 3.1c scrub ok 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.226258+0000 mon.a (mon.0) 1568 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:30:43.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.232719+0000 osd.1 (osd.1) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:43.997 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.233724+0000 osd.1 (osd.1) 164 : cluster 0 purged_snaps scrub found 60 strays 2024-03-11T21:30:43.997 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: audit 2024-03-11T21:30:43.235515+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:43.997 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:43 smithi203 bash[23654]: audit 2024-03-11T21:30:43.236957+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:44.010 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:44.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:44.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.545087+0000 osd.6 (osd.6) 123 : cluster 0 12.5 deep-scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.546459+0000 osd.6 (osd.6) 124 : cluster 0 12.5 deep-scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.644579+0000 osd.2 (osd.2) 139 : cluster 0 12.a scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.647183+0000 osd.2 (osd.2) 140 : cluster 0 12.a scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.816469+0000 osd.3 (osd.3) 187 : cluster 0 9.2 scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.818567+0000 osd.3 (osd.3) 188 : cluster 0 9.2 scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.845028+0000 osd.0 (osd.0) 171 : cluster 0 9.5 scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.847464+0000 osd.0 (osd.0) 172 : cluster 0 9.5 scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.863944+0000 osd.1 (osd.1) 161 : cluster 0 12.7 deep-scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.866560+0000 osd.1 (osd.1) 162 : cluster 0 12.7 deep-scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.916193+0000 osd.4 (osd.4) 161 : cluster 0 12.2 scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.918553+0000 osd.4 (osd.4) 162 : cluster 0 12.2 scrub ok 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.935945+0000 osd.5 (osd.5) 183 : cluster 0 3.1c scrub starts 2024-03-11T21:30:44.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:42.940243+0000 osd.5 (osd.5) 184 : cluster 0 3.1c scrub ok 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.226258+0000 mon.a (mon.0) 1568 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.232719+0000 osd.1 (osd.1) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.233724+0000 osd.1 (osd.1) 164 : cluster 0 purged_snaps scrub found 60 strays 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: audit 2024-03-11T21:30:43.235515+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[29284]: audit 2024-03-11T21:30:43.236957+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.545087+0000 osd.6 (osd.6) 123 : cluster 0 12.5 deep-scrub starts 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.546459+0000 osd.6 (osd.6) 124 : cluster 0 12.5 deep-scrub ok 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.644579+0000 osd.2 (osd.2) 139 : cluster 0 12.a scrub starts 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.647183+0000 osd.2 (osd.2) 140 : cluster 0 12.a scrub ok 2024-03-11T21:30:44.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.816469+0000 osd.3 (osd.3) 187 : cluster 0 9.2 scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.818567+0000 osd.3 (osd.3) 188 : cluster 0 9.2 scrub ok 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.845028+0000 osd.0 (osd.0) 171 : cluster 0 9.5 scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.847464+0000 osd.0 (osd.0) 172 : cluster 0 9.5 scrub ok 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.863944+0000 osd.1 (osd.1) 161 : cluster 0 12.7 deep-scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.866560+0000 osd.1 (osd.1) 162 : cluster 0 12.7 deep-scrub ok 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.916193+0000 osd.4 (osd.4) 161 : cluster 0 12.2 scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.918553+0000 osd.4 (osd.4) 162 : cluster 0 12.2 scrub ok 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.935945+0000 osd.5 (osd.5) 183 : cluster 0 3.1c scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:42.940243+0000 osd.5 (osd.5) 184 : cluster 0 3.1c scrub ok 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.226258+0000 mon.a (mon.0) 1568 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.232719+0000 osd.1 (osd.1) 163 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.233724+0000 osd.1 (osd.1) 164 : cluster 0 purged_snaps scrub found 60 strays 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: audit 2024-03-11T21:30:43.235515+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:44.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:43 smithi143 bash[21512]: audit 2024-03-11T21:30:43.236957+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:44.140 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:44.243 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:44.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:44.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:44.402 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:44.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:44.533 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:44.626 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:44.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:44.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:44.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.277301+0000 mgr.y (mgr.14550) 317 : cluster 0 pgmap v351: 332 pgs: 18 peering, 32 unknown, 282 active+clean; 624 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.808391+0000 osd.0 (osd.0) 173 : cluster 0 12.1b deep-scrub starts 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: cluster 2024-03-11T21:30:43.812573+0000 osd.0 (osd.0) 174 : cluster 0 12.1b deep-scrub ok 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: cluster 2024-03-11T21:30:44.222653+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: audit 2024-03-11T21:30:44.231517+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: audit 2024-03-11T21:30:44.231608+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: cluster 2024-03-11T21:30:44.236078+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:30:44.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[21512]: audit 2024-03-11T21:30:44.238614+0000 mon.a (mon.0) 1575 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.277301+0000 mgr.y (mgr.14550) 317 : cluster 0 pgmap v351: 332 pgs: 18 peering, 32 unknown, 282 active+clean; 624 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.808391+0000 osd.0 (osd.0) 173 : cluster 0 12.1b deep-scrub starts 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: cluster 2024-03-11T21:30:43.812573+0000 osd.0 (osd.0) 174 : cluster 0 12.1b deep-scrub ok 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: cluster 2024-03-11T21:30:44.222653+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: audit 2024-03-11T21:30:44.231517+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: audit 2024-03-11T21:30:44.231608+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: cluster 2024-03-11T21:30:44.236078+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:30:44.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:44 smithi143 bash[29284]: audit 2024-03-11T21:30:44.238614+0000 mon.a (mon.0) 1575 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:44.914 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.277301+0000 mgr.y (mgr.14550) 317 : cluster 0 pgmap v351: 332 pgs: 18 peering, 32 unknown, 282 active+clean; 624 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-11T21:30:45.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.808391+0000 osd.0 (osd.0) 173 : cluster 0 12.1b deep-scrub starts 2024-03-11T21:30:45.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: cluster 2024-03-11T21:30:43.812573+0000 osd.0 (osd.0) 174 : cluster 0 12.1b deep-scrub ok 2024-03-11T21:30:45.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: cluster 2024-03-11T21:30:44.222653+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: audit 2024-03-11T21:30:44.231517+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: audit 2024-03-11T21:30:44.231608+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.143:0/2471290473' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi143-71604-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: cluster 2024-03-11T21:30:44.236078+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:30:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:44 smithi203 bash[23654]: audit 2024-03-11T21:30:44.238614+0000 mon.a (mon.0) 1575 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]: dispatch 2024-03-11T21:30:45.045 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:45.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:45.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.302 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:45.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.427 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:45.542 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:45.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:45.658 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.679 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:45.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: cluster 2024-03-11T21:30:44.288667+0000 osd.7 (osd.7) 215 : cluster 0 3.f scrub starts 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: cluster 2024-03-11T21:30:44.296424+0000 osd.7 (osd.7) 216 : cluster 0 3.f scrub ok 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.234074+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.238277+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.246880+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.247413+0000 mon.a (mon.0) 1578 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:45.919 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.281828+0000 mon.c (mon.2) 419 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.282601+0000 mon.a (mon.0) 1579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.284657+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.285737+0000 mon.a (mon.0) 1580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.290645+0000 mon.c (mon.2) 421 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[21512]: audit 2024-03-11T21:30:45.719209+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: cluster 2024-03-11T21:30:44.288667+0000 osd.7 (osd.7) 215 : cluster 0 3.f scrub starts 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: cluster 2024-03-11T21:30:44.296424+0000 osd.7 (osd.7) 216 : cluster 0 3.f scrub ok 2024-03-11T21:30:45.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.234074+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.238277+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.246880+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.247413+0000 mon.a (mon.0) 1578 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.281828+0000 mon.c (mon.2) 419 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.282601+0000 mon.a (mon.0) 1579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.284657+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.285737+0000 mon.a (mon.0) 1580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.290645+0000 mon.c (mon.2) 421 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:45.921 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:45 smithi143 bash[29284]: audit 2024-03-11T21:30:45.719209+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:45.932 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:46.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: cluster 2024-03-11T21:30:44.288667+0000 osd.7 (osd.7) 215 : cluster 0 3.f scrub starts 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: cluster 2024-03-11T21:30:44.296424+0000 osd.7 (osd.7) 216 : cluster 0 3.f scrub ok 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.234074+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.143:0/967434721' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi143-71594-27"}]': finished 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.238277+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.246880+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.247413+0000 mon.a (mon.0) 1578 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.281828+0000 mon.c (mon.2) 419 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.282601+0000 mon.a (mon.0) 1579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.284657+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.285737+0000 mon.a (mon.0) 1580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.290645+0000 mon.c (mon.2) 421 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:45 smithi203 bash[23654]: audit 2024-03-11T21:30:45.719209+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:46.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:46.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:46.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.193 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:46.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:46.316 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:46.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:46.448 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:46.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:46.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:46.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:46.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.277973+0000 mgr.y (mgr.14550) 318 : cluster 0 pgmap v354: 292 pgs: 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.610429+0000 osd.2 (osd.2) 141 : cluster 0 12.1a deep-scrub starts 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.612495+0000 osd.2 (osd.2) 142 : cluster 0 12.1a deep-scrub ok 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.808386+0000 osd.1 (osd.1) 165 : cluster 0 12.e scrub starts 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.810561+0000 osd.1 (osd.1) 166 : cluster 0 12.e scrub ok 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.830867+0000 osd.0 (osd.0) 175 : cluster 0 5.5 scrub starts 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:45.832414+0000 osd.0 (osd.0) 176 : cluster 0 5.5 scrub ok 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.217417+0000 mon.a (mon.0) 1582 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.245848+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.246051+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.255792+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:46.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.256040+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:30:46.821 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.262290+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.821 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.262587+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:46.821 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.263276+0000 mon.a (mon.0) 1587 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:46.821 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:46 smithi203 bash[23654]: audit 2024-03-11T21:30:46.263929+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:46.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:46.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:46.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:47.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.277973+0000 mgr.y (mgr.14550) 318 : cluster 0 pgmap v354: 292 pgs: 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.610429+0000 osd.2 (osd.2) 141 : cluster 0 12.1a deep-scrub starts 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.612495+0000 osd.2 (osd.2) 142 : cluster 0 12.1a deep-scrub ok 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.808386+0000 osd.1 (osd.1) 165 : cluster 0 12.e scrub starts 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.810561+0000 osd.1 (osd.1) 166 : cluster 0 12.e scrub ok 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.830867+0000 osd.0 (osd.0) 175 : cluster 0 5.5 scrub starts 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:45.832414+0000 osd.0 (osd.0) 176 : cluster 0 5.5 scrub ok 2024-03-11T21:30:47.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.217417+0000 mon.a (mon.0) 1582 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.245848+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.246051+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.255792+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.256040+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.262290+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.262587+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.263276+0000 mon.a (mon.0) 1587 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[29284]: audit 2024-03-11T21:30:46.263929+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.277973+0000 mgr.y (mgr.14550) 318 : cluster 0 pgmap v354: 292 pgs: 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:47.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.610429+0000 osd.2 (osd.2) 141 : cluster 0 12.1a deep-scrub starts 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.612495+0000 osd.2 (osd.2) 142 : cluster 0 12.1a deep-scrub ok 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.808386+0000 osd.1 (osd.1) 165 : cluster 0 12.e scrub starts 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.810561+0000 osd.1 (osd.1) 166 : cluster 0 12.e scrub ok 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.830867+0000 osd.0 (osd.0) 175 : cluster 0 5.5 scrub starts 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:45.832414+0000 osd.0 (osd.0) 176 : cluster 0 5.5 scrub ok 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.217417+0000 mon.a (mon.0) 1582 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.245848+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.246051+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi143-71594-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.255792+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.143:0/2249686298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.256040+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.262290+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.262587+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]: dispatch 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.263276+0000 mon.a (mon.0) 1587 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:47.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:46 smithi143 bash[21512]: audit 2024-03-11T21:30:46.263929+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:47.095 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:47.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:47.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWritp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1970 ms) 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8956 ms total) 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4016 ms) 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4441 ms) 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-11T21:30:47.264 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3529 ms) 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11986 ms total) 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (131577 ms total) 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-11T21:30:47.265 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-11T21:30:47.331 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: In function 'virtual void PG::on_active_advmap(const OSDMapRef&)' thread 7f56f787f700 time 2024-03-11T21:30:47.286612+0000 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: 1901: FAILED ceph_assert(!bad || !cct->_conf->osd_debug_verify_cached_snaps) 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x562e059abc97] 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.341 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: clone() 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug 2024-03-11T21:30:47.281+0000 7f56f787f700 -1 osd.1 pg_epoch: 246 pg[118.1b( v 244'13 (0'0,244'13] local-lis/les=172/174 n=3 ec=172/172 lis/c=172/172 les/c/f=174/174/0 sis=172) [1,5,2] r=0 lpr=172 crt=244'13 lcod 242'11 mlcod 242'11 active+clean+snaptrim trimq=[4~3](4)] on_active_advmap removed_snaps already contains [4~1] 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: *** Caught signal (Aborted) ** 2024-03-11T21:30:47.342 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: in thread 7f56f787f700 thread_name:tp_osd_tp 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug 2024-03-11T21:30:47.305+0000 7f56f787f700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: In function 'virtual void PG::on_active_advmap(const OSDMapRef&)' thread 7f56f787f700 time 2024-03-11T21:30:47.286612+0000 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: 1901: FAILED ceph_assert(!bad || !cct->_conf->osd_debug_verify_cached_snaps) 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x562e059abc97] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.343 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: clone() 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f571770bd20] 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: gsignal() 2024-03-11T21:30:47.344 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: abort() 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x562e059abcf1] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 17: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.345 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 18: clone() 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug 2024-03-11T21:30:47.341+0000 7f56f787f700 -1 *** Caught signal (Aborted) ** 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: in thread 7f56f787f700 thread_name:tp_osd_tp 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f571770bd20] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: gsignal() 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: abort() 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x562e059abcf1] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.346 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 17: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 18: clone() 2024-03-11T21:30:47.347 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T21:30:47.356 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:47.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:47.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:47.520 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:47.520 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -2979> 2024-03-11T21:30:47.281+0000 7f56f787f700 -1 osd.1 pg_epoch: 246 pg[118.1b( v 244'13 (0'0,244'13] local-lis/les=172/174 n=3 ec=172/172 lis/c=172/172 les/c/f=174/174/0 sis=172) [1,5,2] r=0 lpr=172 crt=244'13 lcod 242'11 mlcod 242'11 active+clean+snaptrim trimq=[4~3](4)] on_active_advmap removed_snaps already contains [4~1] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -1345> 2024-03-11T21:30:47.305+0000 7f56f787f700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: In function 'virtual void PG::on_active_advmap(const OSDMapRef&)' thread 7f56f787f700 time 2024-03-11T21:30:47.286612+0000 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: 1901: FAILED ceph_assert(!bad || !cct->_conf->osd_debug_verify_cached_snaps) 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x562e059abc97] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.532 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: clone() 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -203> 2024-03-11T21:30:47.341+0000 7f56f787f700 -1 *** Caught signal (Aborted) ** 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: in thread 7f56f787f700 thread_name:tp_osd_tp 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f571770bd20] 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: gsignal() 2024-03-11T21:30:47.533 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: abort() 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x562e059abcf1] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.534 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 17: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 18: clone() 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -9999> 2024-03-11T21:30:47.281+0000 7f56f787f700 -1 osd.1 pg_epoch: 246 pg[118.1b( v 244'13 (0'0,244'13] local-lis/les=172/174 n=3 ec=172/172 lis/c=172/172 les/c/f=174/174/0 sis=172) [1,5,2] r=0 lpr=172 crt=244'13 lcod 242'11 mlcod 242'11 active+clean+snaptrim trimq=[4~3](4)] on_active_advmap removed_snaps already contains [4~1] 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -9998> 2024-03-11T21:30:47.305+0000 7f56f787f700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: In function 'virtual void PG::on_active_advmap(const OSDMapRef&)' thread 7f56f787f700 time 2024-03-11T21:30:47.286612+0000 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el8/BUILD/ceph-19.0.0-1979-g2b0101eb/src/osd/PG.cc: 1901: FAILED ceph_assert(!bad || !cct->_conf->osd_debug_verify_cached_snaps) 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x562e059abc97] 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.535 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.536 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: clone() 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: debug -9997> 2024-03-11T21:30:47.341+0000 7f56f787f700 -1 *** Caught signal (Aborted) ** 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: in thread 7f56f787f700 thread_name:tp_osd_tp 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f571770bd20] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 2: gsignal() 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 3: abort() 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x562e059abcf1] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 5: /usr/bin/ceph-osd(+0x672e5d) [0x562e059abe5d] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 6: (PG::on_active_advmap(std::shared_ptr const&)+0xd33) [0x562e05ba71a3] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 7: (PeeringState::Active::react(PeeringState::AdvMap const&)+0x183) [0x562e05e53183] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 8: (boost::statechart::simple_state::react_impl(boost::statechart::event_base const&, void const*)+0xb5) [0x562e05e9f3c5] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 9: (PeeringState::advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0x276) [0x562e05e37046] 2024-03-11T21:30:47.537 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 10: (PG::handle_advance_map(std::shared_ptr, std::shared_ptr, std::vector >&, int, std::vector >&, int, PeeringCtx&)+0xfa) [0x562e05b9ad2a] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 11: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PeeringCtx&)+0x371) [0x562e05ad6871] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x299) [0x562e05ad8e29] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 13: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x52) [0x562e05e0a492] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x113d) [0x562e05acaafd] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562e061dbe5b] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e061def24] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 17: /lib64/libpthread.so.0(+0x81ca) [0x7f57177011ca] 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: 18: clone() 2024-03-11T21:30:47.538 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[37804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T21:30:47.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:47.587 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:47.609 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:47.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:47.735 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:47.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:47.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.324155+0000 osd.7 (osd.7) 217 : cluster 0 3.13 scrub starts 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.325862+0000 osd.7 (osd.7) 218 : cluster 0 3.13 scrub ok 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.641846+0000 osd.6 (osd.6) 125 : cluster 0 12.d scrub starts 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.644286+0000 osd.6 (osd.6) 126 : cluster 0 12.d scrub ok 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.911241+0000 osd.5 (osd.5) 185 : cluster 0 2.12 scrub starts 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:46.912724+0000 osd.5 (osd.5) 186 : cluster 0 2.12 scrub ok 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: audit 2024-03-11T21:30:47.251691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: audit 2024-03-11T21:30:47.251911+0000 mon.a (mon.0) 1590 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.742078+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.748591+0000 mon.a (mon.0) 1592 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.748688+0000 mon.a (mon.0) 1593 : cluster 1 osd.1 failed (root=default,host=smithi143) (connection refused reported by osd.0) 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.749066+0000 mon.a (mon.0) 1594 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.851 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.749559+0000 mon.a (mon.0) 1595 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.749833+0000 mon.a (mon.0) 1596 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.750327+0000 mon.a (mon.0) 1597 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.750551+0000 mon.a (mon.0) 1598 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.751391+0000 mon.a (mon.0) 1599 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.753917+0000 mon.a (mon.0) 1600 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.754230+0000 mon.a (mon.0) 1601 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.754503+0000 mon.a (mon.0) 1602 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.754786+0000 mon.a (mon.0) 1603 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.755078+0000 mon.a (mon.0) 1604 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.755368+0000 mon.a (mon.0) 1605 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.755637+0000 mon.a (mon.0) 1606 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:47 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.755917+0000 mon.a (mon.0) 1607 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.865 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:47.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.324155+0000 osd.7 (osd.7) 217 : cluster 0 3.13 scrub starts 2024-03-11T21:30:47.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.325862+0000 osd.7 (osd.7) 218 : cluster 0 3.13 scrub ok 2024-03-11T21:30:47.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.641846+0000 osd.6 (osd.6) 125 : cluster 0 12.d scrub starts 2024-03-11T21:30:47.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.644286+0000 osd.6 (osd.6) 126 : cluster 0 12.d scrub ok 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.911241+0000 osd.5 (osd.5) 185 : cluster 0 2.12 scrub starts 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:46.912724+0000 osd.5 (osd.5) 186 : cluster 0 2.12 scrub ok 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: audit 2024-03-11T21:30:47.251691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: audit 2024-03-11T21:30:47.251911+0000 mon.a (mon.0) 1590 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.742078+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.748591+0000 mon.a (mon.0) 1592 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.748688+0000 mon.a (mon.0) 1593 : cluster 1 osd.1 failed (root=default,host=smithi143) (connection refused reported by osd.0) 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.749066+0000 mon.a (mon.0) 1594 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.749559+0000 mon.a (mon.0) 1595 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.749833+0000 mon.a (mon.0) 1596 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.750327+0000 mon.a (mon.0) 1597 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.750551+0000 mon.a (mon.0) 1598 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.898 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.751391+0000 mon.a (mon.0) 1599 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.753917+0000 mon.a (mon.0) 1600 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.754230+0000 mon.a (mon.0) 1601 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.754503+0000 mon.a (mon.0) 1602 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.754786+0000 mon.a (mon.0) 1603 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.755078+0000 mon.a (mon.0) 1604 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.755368+0000 mon.a (mon.0) 1605 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.755637+0000 mon.a (mon.0) 1606 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.755917+0000 mon.a (mon.0) 1607 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.324155+0000 osd.7 (osd.7) 217 : cluster 0 3.13 scrub starts 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.325862+0000 osd.7 (osd.7) 218 : cluster 0 3.13 scrub ok 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.641846+0000 osd.6 (osd.6) 125 : cluster 0 12.d scrub starts 2024-03-11T21:30:47.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.644286+0000 osd.6 (osd.6) 126 : cluster 0 12.d scrub ok 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.911241+0000 osd.5 (osd.5) 185 : cluster 0 2.12 scrub starts 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:46.912724+0000 osd.5 (osd.5) 186 : cluster 0 2.12 scrub ok 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: audit 2024-03-11T21:30:47.251691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi143-71838-21"}]': finished 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: audit 2024-03-11T21:30:47.251911+0000 mon.a (mon.0) 1590 : audit 1 from='client.? 172.21.15.143:0/3937727249' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi143-71604-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.742078+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.748591+0000 mon.a (mon.0) 1592 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.748688+0000 mon.a (mon.0) 1593 : cluster 1 osd.1 failed (root=default,host=smithi143) (connection refused reported by osd.0) 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.749066+0000 mon.a (mon.0) 1594 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.749559+0000 mon.a (mon.0) 1595 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.749833+0000 mon.a (mon.0) 1596 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.750327+0000 mon.a (mon.0) 1597 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.750551+0000 mon.a (mon.0) 1598 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.751391+0000 mon.a (mon.0) 1599 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.753917+0000 mon.a (mon.0) 1600 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.754230+0000 mon.a (mon.0) 1601 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.754503+0000 mon.a (mon.0) 1602 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.754786+0000 mon.a (mon.0) 1603 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.755078+0000 mon.a (mon.0) 1604 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.755368+0000 mon.a (mon.0) 1605 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.755637+0000 mon.a (mon.0) 1606 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:47.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:47 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.755917+0000 mon.a (mon.0) 1607 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:47.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:47.997 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:48.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:48.126 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:48.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.263 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:48.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.390 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:48.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:48.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:48.647 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:48.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:48.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:48.776 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:48.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:48.891 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.278899+0000 mgr.y (mgr.14550) 319 : cluster 0 pgmap v357: 324 pgs: 32 unknown, 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.777540+0000 osd.3 (osd.3) 189 : cluster 0 3.7 scrub starts 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.779293+0000 osd.3 (osd.3) 190 : cluster 0 3.7 scrub ok 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.895307+0000 osd.4 (osd.4) 163 : cluster 0 12.0 scrub starts 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:47.897064+0000 osd.4 (osd.4) 164 : cluster 0 12.0 scrub ok 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.064862+0000 mon.a (mon.0) 1608 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.065342+0000 mon.a (mon.0) 1609 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.065581+0000 mon.a (mon.0) 1610 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.065725+0000 mon.a (mon.0) 1611 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.066016+0000 mon.a (mon.0) 1612 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.066233+0000 mon.a (mon.0) 1613 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.066513+0000 mon.a (mon.0) 1614 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.066727+0000 mon.a (mon.0) 1615 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:48.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.066935+0000 mon.a (mon.0) 1616 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.067154+0000 mon.a (mon.0) 1617 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.067431+0000 mon.a (mon.0) 1618 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.067704+0000 mon.a (mon.0) 1619 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.068013+0000 mon.a (mon.0) 1620 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.068293+0000 mon.a (mon.0) 1621 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.068567+0000 mon.a (mon.0) 1622 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.254328+0000 mon.a (mon.0) 1623 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: audit 2024-03-11T21:30:48.264418+0000 mon.a (mon.0) 1624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.272297+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.278899+0000 mgr.y (mgr.14550) 319 : cluster 0 pgmap v357: 324 pgs: 32 unknown, 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.777540+0000 osd.3 (osd.3) 189 : cluster 0 3.7 scrub starts 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.779293+0000 osd.3 (osd.3) 190 : cluster 0 3.7 scrub ok 2024-03-11T21:30:48.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.895307+0000 osd.4 (osd.4) 163 : cluster 0 12.0 scrub starts 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:47.897064+0000 osd.4 (osd.4) 164 : cluster 0 12.0 scrub ok 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.064862+0000 mon.a (mon.0) 1608 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.065342+0000 mon.a (mon.0) 1609 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.065581+0000 mon.a (mon.0) 1610 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.065725+0000 mon.a (mon.0) 1611 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.066016+0000 mon.a (mon.0) 1612 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.066233+0000 mon.a (mon.0) 1613 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.066513+0000 mon.a (mon.0) 1614 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.066727+0000 mon.a (mon.0) 1615 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.066935+0000 mon.a (mon.0) 1616 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.067154+0000 mon.a (mon.0) 1617 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.067431+0000 mon.a (mon.0) 1618 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.067704+0000 mon.a (mon.0) 1619 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:48.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.068013+0000 mon.a (mon.0) 1620 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:48.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.068293+0000 mon.a (mon.0) 1621 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:48.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.068567+0000 mon.a (mon.0) 1622 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:48.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.254328+0000 mon.a (mon.0) 1623 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:30:48.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: audit 2024-03-11T21:30:48.264418+0000 mon.a (mon.0) 1624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:48.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:48 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.272297+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-03-11T21:30:48.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:49.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.278899+0000 mgr.y (mgr.14550) 319 : cluster 0 pgmap v357: 324 pgs: 32 unknown, 18 peering, 274 active+clean; 592 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 95 B/s, 1 objects/s recovering 2024-03-11T21:30:49.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.777540+0000 osd.3 (osd.3) 189 : cluster 0 3.7 scrub starts 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.779293+0000 osd.3 (osd.3) 190 : cluster 0 3.7 scrub ok 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.895307+0000 osd.4 (osd.4) 163 : cluster 0 12.0 scrub starts 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:47.897064+0000 osd.4 (osd.4) 164 : cluster 0 12.0 scrub ok 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.064862+0000 mon.a (mon.0) 1608 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.065342+0000 mon.a (mon.0) 1609 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.065581+0000 mon.a (mon.0) 1610 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.065725+0000 mon.a (mon.0) 1611 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.066016+0000 mon.a (mon.0) 1612 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.066233+0000 mon.a (mon.0) 1613 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.066513+0000 mon.a (mon.0) 1614 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.066727+0000 mon.a (mon.0) 1615 : cluster 0 osd.1 reported immediately failed by osd.2 2024-03-11T21:30:49.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.066935+0000 mon.a (mon.0) 1616 : cluster 0 osd.1 reported immediately failed by osd.3 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.067154+0000 mon.a (mon.0) 1617 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.067431+0000 mon.a (mon.0) 1618 : cluster 0 osd.1 reported immediately failed by osd.0 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.067704+0000 mon.a (mon.0) 1619 : cluster 0 osd.1 reported immediately failed by osd.5 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.068013+0000 mon.a (mon.0) 1620 : cluster 0 osd.1 reported immediately failed by osd.6 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.068293+0000 mon.a (mon.0) 1621 : cluster 0 osd.1 reported immediately failed by osd.7 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.068567+0000 mon.a (mon.0) 1622 : cluster 0 osd.1 reported immediately failed by osd.4 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.254328+0000 mon.a (mon.0) 1623 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: audit 2024-03-11T21:30:48.264418+0000 mon.a (mon.0) 1624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi143-71594-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:49.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:48 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.272297+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-03-11T21:30:49.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:49.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.166 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:49.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:49.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:49.394 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:49.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:49.544 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:49.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:49.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:49.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:49.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.238845+0000 osd.7 (osd.7) 219 : cluster 0 1.0 scrub starts 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.243358+0000 osd.7 (osd.7) 220 : cluster 0 1.0 scrub ok 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: audit 2024-03-11T21:30:48.419723+0000 mgr.y (mgr.14550) 320 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.623366+0000 osd.6 (osd.6) 127 : cluster 0 12.1 scrub starts 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.625562+0000 osd.6 (osd.6) 128 : cluster 0 12.1 scrub ok 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.776065+0000 osd.0 (osd.0) 177 : cluster 0 9.b scrub starts 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.777767+0000 osd.0 (osd.0) 178 : cluster 0 9.b scrub ok 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.789211+0000 osd.3 (osd.3) 191 : cluster 0 9.1a scrub starts 2024-03-11T21:30:49.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.791026+0000 osd.3 (osd.3) 192 : cluster 0 9.1a scrub ok 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.888305+0000 osd.4 (osd.4) 165 : cluster 0 3.3 scrub starts 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:48.891641+0000 osd.4 (osd.4) 166 : cluster 0 3.3 scrub ok 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.272369+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: audit 2024-03-11T21:30:49.278285+0000 mon.a (mon.0) 1627 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: audit 2024-03-11T21:30:49.482638+0000 mon.c (mon.2) 424 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:49 smithi203 bash[23654]: audit 2024-03-11T21:30:49.482988+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.238845+0000 osd.7 (osd.7) 219 : cluster 0 1.0 scrub starts 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.243358+0000 osd.7 (osd.7) 220 : cluster 0 1.0 scrub ok 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: audit 2024-03-11T21:30:48.419723+0000 mgr.y (mgr.14550) 320 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.623366+0000 osd.6 (osd.6) 127 : cluster 0 12.1 scrub starts 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.625562+0000 osd.6 (osd.6) 128 : cluster 0 12.1 scrub ok 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.776065+0000 osd.0 (osd.0) 177 : cluster 0 9.b scrub starts 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.777767+0000 osd.0 (osd.0) 178 : cluster 0 9.b scrub ok 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.789211+0000 osd.3 (osd.3) 191 : cluster 0 9.1a scrub starts 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.791026+0000 osd.3 (osd.3) 192 : cluster 0 9.1a scrub ok 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.888305+0000 osd.4 (osd.4) 165 : cluster 0 3.3 scrub starts 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:48.891641+0000 osd.4 (osd.4) 166 : cluster 0 3.3 scrub ok 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.272369+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: audit 2024-03-11T21:30:49.278285+0000 mon.a (mon.0) 1627 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: audit 2024-03-11T21:30:49.482638+0000 mon.c (mon.2) 424 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[29284]: audit 2024-03-11T21:30:49.482988+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.238845+0000 osd.7 (osd.7) 219 : cluster 0 1.0 scrub starts 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.243358+0000 osd.7 (osd.7) 220 : cluster 0 1.0 scrub ok 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: audit 2024-03-11T21:30:48.419723+0000 mgr.y (mgr.14550) 320 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.623366+0000 osd.6 (osd.6) 127 : cluster 0 12.1 scrub starts 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.625562+0000 osd.6 (osd.6) 128 : cluster 0 12.1 scrub ok 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.776065+0000 osd.0 (osd.0) 177 : cluster 0 9.b scrub starts 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.777767+0000 osd.0 (osd.0) 178 : cluster 0 9.b scrub ok 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.789211+0000 osd.3 (osd.3) 191 : cluster 0 9.1a scrub starts 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.791026+0000 osd.3 (osd.3) 192 : cluster 0 9.1a scrub ok 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.888305+0000 osd.4 (osd.4) 165 : cluster 0 3.3 scrub starts 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:48.891641+0000 osd.4 (osd.4) 166 : cluster 0 3.3 scrub ok 2024-03-11T21:30:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.272369+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-03-11T21:30:49.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: audit 2024-03-11T21:30:49.278285+0000 mon.a (mon.0) 1627 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:49.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: audit 2024-03-11T21:30:49.482638+0000 mon.c (mon.2) 424 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:49 smithi143 bash[21512]: audit 2024-03-11T21:30:49.482988+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:50.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:50.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:50.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:50.275 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:50.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:50.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:50.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:50.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:50.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:50.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:50.923 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.279757+0000 mgr.y (mgr.14550) 321 : cluster 0 pgmap v360: 332 pgs: 14 peering, 226 active+clean, 3 creating+peering, 37 unknown, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.578315+0000 osd.6 (osd.6) 129 : cluster 0 12.11 scrub starts 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.579984+0000 osd.6 (osd.6) 130 : cluster 0 12.11 scrub ok 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.768280+0000 osd.0 (osd.0) 179 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:49.769825+0000 osd.0 (osd.0) 180 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.267087+0000 mon.a (mon.0) 1629 : cluster 3 Health check update: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.277562+0000 mon.a (mon.0) 1630 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.277725+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.285783+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.285849+0000 mon.c (mon.2) 425 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.287209+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:50.924 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.279757+0000 mgr.y (mgr.14550) 321 : cluster 0 pgmap v360: 332 pgs: 14 peering, 226 active+clean, 3 creating+peering, 37 unknown, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.578315+0000 osd.6 (osd.6) 129 : cluster 0 12.11 scrub starts 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.579984+0000 osd.6 (osd.6) 130 : cluster 0 12.11 scrub ok 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.768280+0000 osd.0 (osd.0) 179 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:49.769825+0000 osd.0 (osd.0) 180 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.267087+0000 mon.a (mon.0) 1629 : cluster 3 Health check update: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.277562+0000 mon.a (mon.0) 1630 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.277725+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.285783+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.285849+0000 mon.c (mon.2) 425 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:50.925 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.287209+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:50.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.279757+0000 mgr.y (mgr.14550) 321 : cluster 0 pgmap v360: 332 pgs: 14 peering, 226 active+clean, 3 creating+peering, 37 unknown, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.578315+0000 osd.6 (osd.6) 129 : cluster 0 12.11 scrub starts 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.579984+0000 osd.6 (osd.6) 130 : cluster 0 12.11 scrub ok 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.768280+0000 osd.0 (osd.0) 179 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:49.769825+0000 osd.0 (osd.0) 180 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.267087+0000 mon.a (mon.0) 1629 : cluster 3 Health check update: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:51.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.277562+0000 mon.a (mon.0) 1630 : audit 1 from='client.? 172.21.15.143:0/4122401185' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi143-71604-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.277725+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.285783+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.285849+0000 mon.c (mon.2) 425 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.287209+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]: dispatch 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.292126+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: audit 2024-03-11T21:30:50.299494+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.696358+0000 osd.2 (osd.2) 143 : cluster 0 2.c deep-scrub starts 2024-03-11T21:30:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:50 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.697851+0000 osd.2 (osd.2) 144 : cluster 0 2.c deep-scrub ok 2024-03-11T21:30:51.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:51.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:51.134 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.292126+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.134 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: audit 2024-03-11T21:30:50.299494+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.134 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.696358+0000 osd.2 (osd.2) 143 : cluster 0 2.c deep-scrub starts 2024-03-11T21:30:51.134 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.697851+0000 osd.2 (osd.2) 144 : cluster 0 2.c deep-scrub ok 2024-03-11T21:30:51.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.292126+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: audit 2024-03-11T21:30:50.299494+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:51.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.696358+0000 osd.2 (osd.2) 143 : cluster 0 2.c deep-scrub starts 2024-03-11T21:30:51.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:50 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.697851+0000 osd.2 (osd.2) 144 : cluster 0 2.c deep-scrub ok 2024-03-11T21:30:51.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:51.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:51.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.312 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:51.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:51.434 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:51.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:51.561 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:51.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:51.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:51.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:51.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:51.952 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:52.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:52.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:52.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:30:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:52.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.880933+0000 osd.4 (osd.4) 167 : cluster 0 12.12 scrub starts 2024-03-11T21:30:52.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.883834+0000 osd.4 (osd.4) 168 : cluster 0 12.12 scrub ok 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:50.954622+0000 mon.a (mon.0) 1635 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:50.956154+0000 mon.c (mon.2) 427 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.965414+0000 osd.5 (osd.5) 187 : cluster 0 39.1a scrub starts 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:50.966576+0000 osd.5 (osd.5) 188 : cluster 0 39.1a scrub ok 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.219870+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:51.221583+0000 mon.a (mon.0) 1637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:51.221648+0000 mon.a (mon.0) 1638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.225253+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:51.226829+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:51.237872+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.769567+0000 osd.0 (osd.0) 181 : cluster 0 39.12 scrub starts 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.770701+0000 osd.0 (osd.0) 182 : cluster 0 39.12 scrub ok 2024-03-11T21:30:52.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.880933+0000 osd.4 (osd.4) 167 : cluster 0 12.12 scrub starts 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.883834+0000 osd.4 (osd.4) 168 : cluster 0 12.12 scrub ok 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: audit 2024-03-11T21:30:50.954622+0000 mon.a (mon.0) 1635 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: audit 2024-03-11T21:30:50.956154+0000 mon.c (mon.2) 427 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.965414+0000 osd.5 (osd.5) 187 : cluster 0 39.1a scrub starts 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:51 smithi143 bash[21512]: cluster 2024-03-11T21:30:50.966576+0000 osd.5 (osd.5) 188 : cluster 0 39.1a scrub ok 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.219870+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:51.221583+0000 mon.a (mon.0) 1637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:51.221648+0000 mon.a (mon.0) 1638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.225253+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:51.226829+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:51.237872+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.769567+0000 osd.0 (osd.0) 181 : cluster 0 39.12 scrub starts 2024-03-11T21:30:52.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.770701+0000 osd.0 (osd.0) 182 : cluster 0 39.12 scrub ok 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.880933+0000 osd.4 (osd.4) 167 : cluster 0 12.12 scrub starts 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.883834+0000 osd.4 (osd.4) 168 : cluster 0 12.12 scrub ok 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:50.954622+0000 mon.a (mon.0) 1635 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:50.956154+0000 mon.c (mon.2) 427 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.965414+0000 osd.5 (osd.5) 187 : cluster 0 39.1a scrub starts 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:50.966576+0000 osd.5 (osd.5) 188 : cluster 0 39.1a scrub ok 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.219870+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:51.221583+0000 mon.a (mon.0) 1637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-13"}]': finished 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:51.221648+0000 mon.a (mon.0) 1638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:52.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.225253+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-03-11T21:30:52.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:51.226829+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.143:0/3605250412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: audit 2024-03-11T21:30:51.237872+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]: dispatch 2024-03-11T21:30:52.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.769567+0000 osd.0 (osd.0) 181 : cluster 0 39.12 scrub starts 2024-03-11T21:30:52.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:51 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.770701+0000 osd.0 (osd.0) 182 : cluster 0 39.12 scrub ok 2024-03-11T21:30:52.228 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:52.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:52.361 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:52.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:52.484 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:52.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:52.613 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:52.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:52.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:52.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:52.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:30:52.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:53.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:53.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:53.132 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:53.185 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:30:52 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1.service: Main process exited, code=exited, status=134/n/a 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.279616+0000 osd.7 (osd.7) 221 : cluster 0 2.b scrub starts 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.280820+0000 mgr.y (mgr.14550) 322 : cluster 0 pgmap v363: 292 pgs: 14 peering, 226 active+clean, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.296237+0000 osd.7 (osd.7) 222 : cluster 0 2.b scrub ok 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.552385+0000 osd.6 (osd.6) 131 : cluster 0 9.18 scrub starts 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.554529+0000 osd.6 (osd.6) 132 : cluster 0 9.18 scrub ok 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.970026+0000 osd.5 (osd.5) 189 : cluster 0 9.7 scrub starts 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:51.971614+0000 osd.5 (osd.5) 190 : cluster 0 9.7 scrub ok 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.234557+0000 mon.a (mon.0) 1641 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.235832+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.241450+0000 mon.a (mon.0) 1643 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.244059+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.143:0/3345876625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.247739+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.283281+0000 mon.c (mon.2) 430 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.471236+0000 osd.3 (osd.3) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.471451+0000 osd.3 (osd.3) 194 : cluster 0 purged_snaps scrub found 16 strays 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.686290+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.686998+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.687359+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.688048+0000 mon.c (mon.2) 432 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: audit 2024-03-11T21:30:52.688436+0000 mon.a (mon.0) 1647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.792161+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub starts 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.794199+0000 osd.3 (osd.3) 196 : cluster 0 6.13 scrub ok 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.809845+0000 osd.0 (osd.0) 183 : cluster 0 3.12 scrub starts 2024-03-11T21:30:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.812926+0000 osd.0 (osd.0) 184 : cluster 0 3.12 scrub ok 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.279616+0000 osd.7 (osd.7) 221 : cluster 0 2.b scrub starts 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.280820+0000 mgr.y (mgr.14550) 322 : cluster 0 pgmap v363: 292 pgs: 14 peering, 226 active+clean, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.296237+0000 osd.7 (osd.7) 222 : cluster 0 2.b scrub ok 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.552385+0000 osd.6 (osd.6) 131 : cluster 0 9.18 scrub starts 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.554529+0000 osd.6 (osd.6) 132 : cluster 0 9.18 scrub ok 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.970026+0000 osd.5 (osd.5) 189 : cluster 0 9.7 scrub starts 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:51.971614+0000 osd.5 (osd.5) 190 : cluster 0 9.7 scrub ok 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.234557+0000 mon.a (mon.0) 1641 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.235832+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.241450+0000 mon.a (mon.0) 1643 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-03-11T21:30:53.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.244059+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.143:0/3345876625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.247739+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.283281+0000 mon.c (mon.2) 430 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.471236+0000 osd.3 (osd.3) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.471451+0000 osd.3 (osd.3) 194 : cluster 0 purged_snaps scrub found 16 strays 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.686290+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.686998+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.687359+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.688048+0000 mon.c (mon.2) 432 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: audit 2024-03-11T21:30:52.688436+0000 mon.a (mon.0) 1647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.792161+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub starts 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.794199+0000 osd.3 (osd.3) 196 : cluster 0 6.13 scrub ok 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.809845+0000 osd.0 (osd.0) 183 : cluster 0 3.12 scrub starts 2024-03-11T21:30:53.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:52 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.812926+0000 osd.0 (osd.0) 184 : cluster 0 3.12 scrub ok 2024-03-11T21:30:53.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.279616+0000 osd.7 (osd.7) 221 : cluster 0 2.b scrub starts 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.280820+0000 mgr.y (mgr.14550) 322 : cluster 0 pgmap v363: 292 pgs: 14 peering, 226 active+clean, 14 active+clean+snaptrim_wait, 27 stale+active+clean, 7 active+clean+snaptrim, 4 stale+peering; 589 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.296237+0000 osd.7 (osd.7) 222 : cluster 0 2.b scrub ok 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.552385+0000 osd.6 (osd.6) 131 : cluster 0 9.18 scrub starts 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.554529+0000 osd.6 (osd.6) 132 : cluster 0 9.18 scrub ok 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.970026+0000 osd.5 (osd.5) 189 : cluster 0 9.7 scrub starts 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:51.971614+0000 osd.5 (osd.5) 190 : cluster 0 9.7 scrub ok 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.234557+0000 mon.a (mon.0) 1641 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.235832+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi143-71594-28"}]': finished 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.241450+0000 mon.a (mon.0) 1643 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-03-11T21:30:53.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.244059+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.143:0/3345876625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.247739+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.283281+0000 mon.c (mon.2) 430 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.471236+0000 osd.3 (osd.3) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.471451+0000 osd.3 (osd.3) 194 : cluster 0 purged_snaps scrub found 16 strays 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.686290+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.686998+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.687359+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.688048+0000 mon.c (mon.2) 432 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: audit 2024-03-11T21:30:52.688436+0000 mon.a (mon.0) 1647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.792161+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub starts 2024-03-11T21:30:53.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.794199+0000 osd.3 (osd.3) 196 : cluster 0 6.13 scrub ok 2024-03-11T21:30:53.254 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.809845+0000 osd.0 (osd.0) 183 : cluster 0 3.12 scrub starts 2024-03-11T21:30:53.254 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:52 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.812926+0000 osd.0 (osd.0) 184 : cluster 0 3.12 scrub ok 2024-03-11T21:30:53.270 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:53.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:53.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:53.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:53.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:53.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.794 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:53.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.924 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:54.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:54.054 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:54.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:54.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:54.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:54.318 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:54.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:54.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:54.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:54.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (18158 ms) 2024-03-11T21:30:54.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (28490 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (14186 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4815 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (5364 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (4235 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3018 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2992 ms) 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-11T21:30:54.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3477 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3353 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2938 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3014 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2616 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3031 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2990 ms) 2024-03-11T21:30:54.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2984 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3012 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2981 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3070 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2927 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3037 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2996 ms) 2024-03-11T21:30:54.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2971 ms) 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3020 ms) 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3033 ms) 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2943 ms) 2024-03-11T21:30:54.687 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-11T21:30:54.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:54.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:54.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.895865+0000 osd.4 (osd.4) 169 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:30:54.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:52.898346+0000 osd.4 (osd.4) 170 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:30:54.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.238539+0000 mon.a (mon.0) 1648 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:54.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.238642+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:54.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:53.242473+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.246437+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.247217+0000 mon.a (mon.0) 1651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.251703+0000 mon.c (mon.2) 434 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: audit 2024-03-11T21:30:53.252404+0000 mon.a (mon.0) 1652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:53.281827+0000 mgr.y (mgr.14550) 323 : cluster 0 pgmap v366: 324 pgs: 28 active+undersized+degraded, 56 active+undersized, 27 creating+peering, 37 unknown, 11 peering, 161 active+clean, 3 active+clean+snaptrim, 1 stale+peering; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 162/1281 objects degraded (12.646%) 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:53.860191+0000 osd.0 (osd.0) 185 : cluster 0 9.9 scrub starts 2024-03-11T21:30:54.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:54 smithi203 bash[23654]: cluster 2024-03-11T21:30:53.861696+0000 osd.0 (osd.0) 186 : cluster 0 9.9 scrub ok 2024-03-11T21:30:54.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:54.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:54.956 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:55.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:55.086 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.895865+0000 osd.4 (osd.4) 169 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:52.898346+0000 osd.4 (osd.4) 170 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.238539+0000 mon.a (mon.0) 1648 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.238642+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:53.242473+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.246437+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.247217+0000 mon.a (mon.0) 1651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.251703+0000 mon.c (mon.2) 434 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: audit 2024-03-11T21:30:53.252404+0000 mon.a (mon.0) 1652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:53.281827+0000 mgr.y (mgr.14550) 323 : cluster 0 pgmap v366: 324 pgs: 28 active+undersized+degraded, 56 active+undersized, 27 creating+peering, 37 unknown, 11 peering, 161 active+clean, 3 active+clean+snaptrim, 1 stale+peering; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 162/1281 objects degraded (12.646%) 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:53.860191+0000 osd.0 (osd.0) 185 : cluster 0 9.9 scrub starts 2024-03-11T21:30:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[21512]: cluster 2024-03-11T21:30:53.861696+0000 osd.0 (osd.0) 186 : cluster 0 9.9 scrub ok 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.895865+0000 osd.4 (osd.4) 169 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:52.898346+0000 osd.4 (osd.4) 170 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.238539+0000 mon.a (mon.0) 1648 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi143-71604-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.238642+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi143-71594-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:53.242473+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.246437+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.247217+0000 mon.a (mon.0) 1651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.251703+0000 mon.c (mon.2) 434 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: audit 2024-03-11T21:30:53.252404+0000 mon.a (mon.0) 1652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:53.281827+0000 mgr.y (mgr.14550) 323 : cluster 0 pgmap v366: 324 pgs: 28 active+undersized+degraded, 56 active+undersized, 27 creating+peering, 37 unknown, 11 peering, 161 active+clean, 3 active+clean+snaptrim, 1 stale+peering; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 162/1281 objects degraded (12.646%) 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:53.860191+0000 osd.0 (osd.0) 185 : cluster 0 9.9 scrub starts 2024-03-11T21:30:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:54 smithi143 bash[29284]: cluster 2024-03-11T21:30:53.861696+0000 osd.0 (osd.0) 186 : cluster 0 9.9 scrub ok 2024-03-11T21:30:55.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:55.213 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:55.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:55.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:55.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:55.480 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:55.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:55.613 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:55.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:55.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:55.847 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:55.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.241172+0000 mon.a (mon.0) 1653 : cluster 3 Health check failed: Degraded data redundancy: 162/1281 objects degraded (12.646%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:54.242727+0000 mon.a (mon.0) 1654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.332154+0000 osd.7 (osd.7) 223 : cluster 0 9.13 scrub starts 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.336302+0000 osd.7 (osd.7) 224 : cluster 0 9.13 scrub ok 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.706960+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.939296+0000 osd.5 (osd.5) 191 : cluster 0 3.5 scrub starts 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.944293+0000 osd.5 (osd.5) 192 : cluster 0 3.5 scrub ok 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.962373+0000 osd.4 (osd.4) 171 : cluster 0 39.10 scrub starts 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:54.963565+0000 osd.4 (osd.4) 172 : cluster 0 39.10 scrub ok 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:55.245195+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.252999+0000 mon.a (mon.0) 1657 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:55.255103+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.143:0/4276698049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:55.255574+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:55.288336+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:55.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[21512]: audit 2024-03-11T21:30:55.288692+0000 mon.a (mon.0) 1659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.241172+0000 mon.a (mon.0) 1653 : cluster 3 Health check failed: Degraded data redundancy: 162/1281 objects degraded (12.646%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:54.242727+0000 mon.a (mon.0) 1654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.332154+0000 osd.7 (osd.7) 223 : cluster 0 9.13 scrub starts 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.336302+0000 osd.7 (osd.7) 224 : cluster 0 9.13 scrub ok 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.706960+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.939296+0000 osd.5 (osd.5) 191 : cluster 0 3.5 scrub starts 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.944293+0000 osd.5 (osd.5) 192 : cluster 0 3.5 scrub ok 2024-03-11T21:30:55.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.962373+0000 osd.4 (osd.4) 171 : cluster 0 39.10 scrub starts 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:54.963565+0000 osd.4 (osd.4) 172 : cluster 0 39.10 scrub ok 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:55.245195+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.252999+0000 mon.a (mon.0) 1657 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:55.255103+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.143:0/4276698049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:55.255574+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:55.288336+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:55.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:55 smithi143 bash[29284]: audit 2024-03-11T21:30:55.288692+0000 mon.a (mon.0) 1659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:55.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:55.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:56.007 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:56.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.241172+0000 mon.a (mon.0) 1653 : cluster 3 Health check failed: Degraded data redundancy: 162/1281 objects degraded (12.646%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T21:30:56.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:54.242727+0000 mon.a (mon.0) 1654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.332154+0000 osd.7 (osd.7) 223 : cluster 0 9.13 scrub starts 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.336302+0000 osd.7 (osd.7) 224 : cluster 0 9.13 scrub ok 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.706960+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.939296+0000 osd.5 (osd.5) 191 : cluster 0 3.5 scrub starts 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.944293+0000 osd.5 (osd.5) 192 : cluster 0 3.5 scrub ok 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.962373+0000 osd.4 (osd.4) 171 : cluster 0 39.10 scrub starts 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:54.963565+0000 osd.4 (osd.4) 172 : cluster 0 39.10 scrub ok 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:55.245195+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi143-71594-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.252999+0000 mon.a (mon.0) 1657 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:55.255103+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.143:0/4276698049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:55.255574+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:55.288336+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:55 smithi203 bash[23654]: audit 2024-03-11T21:30:55.288692+0000 mon.a (mon.0) 1659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:30:56.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:56.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:56.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.280 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:56.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:56.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:56.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:56.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:56.663 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:56.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.795 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:56.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.282509+0000 mgr.y (mgr.14550) 324 : cluster 0 pgmap v369: 332 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 67 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.791307+0000 osd.3 (osd.3) 197 : cluster 0 39.2 scrub starts 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.793786+0000 osd.3 (osd.3) 198 : cluster 0 39.2 scrub ok 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.221176+0000 mon.a (mon.0) 1660 : cluster 3 Health check update: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.245326+0000 mon.a (mon.0) 1661 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: audit 2024-03-11T21:30:56.256389+0000 mon.a (mon.0) 1662 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: audit 2024-03-11T21:30:56.256588+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.266745+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: audit 2024-03-11T21:30:56.267453+0000 mon.c (mon.2) 437 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:56.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[29284]: audit 2024-03-11T21:30:56.268175+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.282509+0000 mgr.y (mgr.14550) 324 : cluster 0 pgmap v369: 332 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 67 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.791307+0000 osd.3 (osd.3) 197 : cluster 0 39.2 scrub starts 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.793786+0000 osd.3 (osd.3) 198 : cluster 0 39.2 scrub ok 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.221176+0000 mon.a (mon.0) 1660 : cluster 3 Health check update: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.245326+0000 mon.a (mon.0) 1661 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: audit 2024-03-11T21:30:56.256389+0000 mon.a (mon.0) 1662 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: audit 2024-03-11T21:30:56.256588+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.266745+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: audit 2024-03-11T21:30:56.267453+0000 mon.c (mon.2) 437 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:56.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:56 smithi143 bash[21512]: audit 2024-03-11T21:30:56.268175+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:56.923 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:57.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.282509+0000 mgr.y (mgr.14550) 324 : cluster 0 pgmap v369: 332 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 67 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:57.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.791307+0000 osd.3 (osd.3) 197 : cluster 0 39.2 scrub starts 2024-03-11T21:30:57.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.793786+0000 osd.3 (osd.3) 198 : cluster 0 39.2 scrub ok 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.221176+0000 mon.a (mon.0) 1660 : cluster 3 Health check update: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.245326+0000 mon.a (mon.0) 1661 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: audit 2024-03-11T21:30:56.256389+0000 mon.a (mon.0) 1662 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi143-71604-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: audit 2024-03-11T21:30:56.256588+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.266745+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: audit 2024-03-11T21:30:56.267453+0000 mon.c (mon.2) 437 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:57.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:56 smithi203 bash[23654]: audit 2024-03-11T21:30:56.268175+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:30:57.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:57.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:57.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:57.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:57.312 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:57.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:57.440 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:57.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:57.565 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:57.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:57.695 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:57.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:57.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:57.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.820196+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub starts 2024-03-11T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: cluster 2024-03-11T21:30:55.823160+0000 osd.2 (osd.2) 146 : cluster 0 2.e scrub ok 2024-03-11T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.546155+0000 osd.6 (osd.6) 133 : cluster 0 5.1f scrub starts 2024-03-11T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.549035+0000 osd.6 (osd.6) 134 : cluster 0 5.1f scrub ok 2024-03-11T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: audit 2024-03-11T21:30:57.259174+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: cluster 2024-03-11T21:30:57.263146+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: audit 2024-03-11T21:30:57.266555+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: audit 2024-03-11T21:30:57.268033+0000 mon.a (mon.0) 1668 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: audit 2024-03-11T21:30:57.268275+0000 mon.c (mon.2) 439 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[21512]: audit 2024-03-11T21:30:57.270437+0000 mon.a (mon.0) 1669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.820196+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub starts 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: cluster 2024-03-11T21:30:55.823160+0000 osd.2 (osd.2) 146 : cluster 0 2.e scrub ok 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.546155+0000 osd.6 (osd.6) 133 : cluster 0 5.1f scrub starts 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.549035+0000 osd.6 (osd.6) 134 : cluster 0 5.1f scrub ok 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: audit 2024-03-11T21:30:57.259174+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: cluster 2024-03-11T21:30:57.263146+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:30:57.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: audit 2024-03-11T21:30:57.266555+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: audit 2024-03-11T21:30:57.268033+0000 mon.a (mon.0) 1668 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: audit 2024-03-11T21:30:57.268275+0000 mon.c (mon.2) 439 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:57.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:57 smithi143 bash[29284]: audit 2024-03-11T21:30:57.270437+0000 mon.a (mon.0) 1669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:57.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:58.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.820196+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub starts 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: cluster 2024-03-11T21:30:55.823160+0000 osd.2 (osd.2) 146 : cluster 0 2.e scrub ok 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.546155+0000 osd.6 (osd.6) 133 : cluster 0 5.1f scrub starts 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.549035+0000 osd.6 (osd.6) 134 : cluster 0 5.1f scrub ok 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: audit 2024-03-11T21:30:57.259174+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: cluster 2024-03-11T21:30:57.263146+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: audit 2024-03-11T21:30:57.266555+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: audit 2024-03-11T21:30:57.268033+0000 mon.a (mon.0) 1668 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]: dispatch 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: audit 2024-03-11T21:30:57.268275+0000 mon.c (mon.2) 439 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:57 smithi203 bash[23654]: audit 2024-03-11T21:30:57.270437+0000 mon.a (mon.0) 1669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:58.085 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:58.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:58.216 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:58.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:58.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:58.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:58.474 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:58.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:30:58.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:58.605 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:58.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.972472+0000 osd.5 (osd.5) 193 : cluster 0 39.0 scrub starts 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:56.974785+0000 osd.5 (osd.5) 194 : cluster 0 39.0 scrub ok 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:57.283156+0000 mgr.y (mgr.14550) 325 : cluster 0 pgmap v372: 292 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 27 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:57.520483+0000 osd.6 (osd.6) 135 : cluster 0 39.8 scrub starts 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:57.522681+0000 osd.6 (osd.6) 136 : cluster 0 39.8 scrub ok 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.259325+0000 mon.a (mon.0) 1670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.261251+0000 mon.a (mon.0) 1671 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:58.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.269699+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]': finished 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.269900+0000 mon.a (mon.0) 1673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.277322+0000 mon.a (mon.0) 1674 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.293743+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.294943+0000 mon.c (mon.2) 441 : audit 1 from='client.? 172.21.15.143:0/41447402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.294994+0000 mon.a (mon.0) 1675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[21512]: audit 2024-03-11T21:30:58.295856+0000 mon.a (mon.0) 1676 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.972472+0000 osd.5 (osd.5) 193 : cluster 0 39.0 scrub starts 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:56.974785+0000 osd.5 (osd.5) 194 : cluster 0 39.0 scrub ok 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:57.283156+0000 mgr.y (mgr.14550) 325 : cluster 0 pgmap v372: 292 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 27 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:57.520483+0000 osd.6 (osd.6) 135 : cluster 0 39.8 scrub starts 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:57.522681+0000 osd.6 (osd.6) 136 : cluster 0 39.8 scrub ok 2024-03-11T21:30:58.720 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.259325+0000 mon.a (mon.0) 1670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.261251+0000 mon.a (mon.0) 1671 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.269699+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]': finished 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.269900+0000 mon.a (mon.0) 1673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.277322+0000 mon.a (mon.0) 1674 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.293743+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.294943+0000 mon.c (mon.2) 441 : audit 1 from='client.? 172.21.15.143:0/41447402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.294994+0000 mon.a (mon.0) 1675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:58.721 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:58 smithi143 bash[29284]: audit 2024-03-11T21:30:58.295856+0000 mon.a (mon.0) 1676 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:58.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:58.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.862 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:58.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:59.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.972472+0000 osd.5 (osd.5) 193 : cluster 0 39.0 scrub starts 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:56.974785+0000 osd.5 (osd.5) 194 : cluster 0 39.0 scrub ok 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:57.283156+0000 mgr.y (mgr.14550) 325 : cluster 0 pgmap v372: 292 pgs: 33 active+undersized+degraded, 63 active+undersized, 5 creating+peering, 27 unknown, 161 active+clean, 3 active+clean+snaptrim; 587 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 180/1281 objects degraded (14.052%) 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:57.520483+0000 osd.6 (osd.6) 135 : cluster 0 39.8 scrub starts 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:57.522681+0000 osd.6 (osd.6) 136 : cluster 0 39.8 scrub ok 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.259325+0000 mon.a (mon.0) 1670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.261251+0000 mon.a (mon.0) 1671 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.269699+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-15", "mode": "writeback"}]': finished 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.269900+0000 mon.a (mon.0) 1673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.277322+0000 mon.a (mon.0) 1674 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.293743+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.143:0/3084097275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.294943+0000 mon.c (mon.2) 441 : audit 1 from='client.? 172.21.15.143:0/41447402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.294994+0000 mon.a (mon.0) 1675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]: dispatch 2024-03-11T21:30:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:58 smithi203 bash[23654]: audit 2024-03-11T21:30:58.295856+0000 mon.a (mon.0) 1676 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:30:59.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:59.123 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:59.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:59.271 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:59.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:59.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.529 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:59.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:59.661 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:59.764 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:59.788 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:59.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:59.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:58.428656+0000 mgr.y (mgr.14550) 326 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.475910+0000 osd.6 (osd.6) 137 : cluster 0 39.1d scrub starts 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.478175+0000 osd.6 (osd.6) 138 : cluster 0 39.1d scrub ok 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.881080+0000 osd.0 (osd.0) 187 : cluster 0 2.13 scrub starts 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.882815+0000 osd.0 (osd.0) 188 : cluster 0 2.13 scrub ok 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.910253+0000 osd.4 (osd.4) 173 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:30:59.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:58.912120+0000 osd.4 (osd.4) 174 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.273320+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.273446+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.278324+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.318669+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.319151+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.320252+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.320501+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.321006+0000 mon.c (mon.2) 444 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:59.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.321421+0000 mon.a (mon.0) 1682 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.357235+0000 mon.c (mon.2) 445 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[21512]: audit 2024-03-11T21:30:59.357628+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:58.428656+0000 mgr.y (mgr.14550) 326 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.475910+0000 osd.6 (osd.6) 137 : cluster 0 39.1d scrub starts 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.478175+0000 osd.6 (osd.6) 138 : cluster 0 39.1d scrub ok 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.881080+0000 osd.0 (osd.0) 187 : cluster 0 2.13 scrub starts 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.882815+0000 osd.0 (osd.0) 188 : cluster 0 2.13 scrub ok 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.910253+0000 osd.4 (osd.4) 173 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:30:59.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:58.912120+0000 osd.4 (osd.4) 174 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.273320+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.273446+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.278324+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.318669+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.319151+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.320252+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.320501+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.321006+0000 mon.c (mon.2) 444 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.321421+0000 mon.a (mon.0) 1682 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.357235+0000 mon.c (mon.2) 445 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:30:59.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:30:59 smithi143 bash[29284]: audit 2024-03-11T21:30:59.357628+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:00.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:58.428656+0000 mgr.y (mgr.14550) 326 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:00.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.475910+0000 osd.6 (osd.6) 137 : cluster 0 39.1d scrub starts 2024-03-11T21:31:00.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.478175+0000 osd.6 (osd.6) 138 : cluster 0 39.1d scrub ok 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.881080+0000 osd.0 (osd.0) 187 : cluster 0 2.13 scrub starts 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.882815+0000 osd.0 (osd.0) 188 : cluster 0 2.13 scrub ok 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.910253+0000 osd.4 (osd.4) 173 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:58.912120+0000 osd.4 (osd.4) 174 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.273320+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi143-71594-29"}]': finished 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.273446+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi143-71604-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.278324+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.318669+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.319151+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.320252+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.320501+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.321006+0000 mon.c (mon.2) 444 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.321421+0000 mon.a (mon.0) 1682 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.357235+0000 mon.c (mon.2) 445 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:30:59 smithi203 bash[23654]: audit 2024-03-11T21:30:59.357628+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:00.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:00.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:00.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:00.176 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:00.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.300 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.427 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:00.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:00.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.692 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:00.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:00.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:00.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.284045+0000 mgr.y (mgr.14550) 327 : cluster 0 pgmap v375: 324 pgs: 33 active+undersized+degraded, 74 active+undersized, 4 creating+peering, 28 unknown, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:00.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.342545+0000 osd.7 (osd.7) 225 : cluster 0 9.11 scrub starts 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.344472+0000 osd.7 (osd.7) 226 : cluster 0 9.11 scrub ok 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.447078+0000 osd.6 (osd.6) 139 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.448400+0000 osd.6 (osd.6) 140 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.895513+0000 osd.0 (osd.0) 189 : cluster 0 39.4 scrub starts 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.897028+0000 osd.0 (osd.0) 190 : cluster 0 39.4 scrub ok 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.273294+0000 mon.a (mon.0) 1684 : cluster 3 Health check update: Degraded data redundancy: 182/1257 objects degraded (14.479%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.280172+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.280294+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.284167+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.285563+0000 mon.c (mon.2) 446 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.286081+0000 mon.a (mon.0) 1688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:00.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.288438+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:00.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:00 smithi203 bash[23654]: audit 2024-03-11T21:31:00.288813+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:00.946 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:00.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.284045+0000 mgr.y (mgr.14550) 327 : cluster 0 pgmap v375: 324 pgs: 33 active+undersized+degraded, 74 active+undersized, 4 creating+peering, 28 unknown, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:00.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.342545+0000 osd.7 (osd.7) 225 : cluster 0 9.11 scrub starts 2024-03-11T21:31:00.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.344472+0000 osd.7 (osd.7) 226 : cluster 0 9.11 scrub ok 2024-03-11T21:31:00.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.447078+0000 osd.6 (osd.6) 139 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.448400+0000 osd.6 (osd.6) 140 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.895513+0000 osd.0 (osd.0) 189 : cluster 0 39.4 scrub starts 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.897028+0000 osd.0 (osd.0) 190 : cluster 0 39.4 scrub ok 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.273294+0000 mon.a (mon.0) 1684 : cluster 3 Health check update: Degraded data redundancy: 182/1257 objects degraded (14.479%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.280172+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.280294+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.284167+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:31:00.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.285563+0000 mon.c (mon.2) 446 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:01.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:01.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:01.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.286081+0000 mon.a (mon.0) 1688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:01.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.288438+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:01.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[21512]: audit 2024-03-11T21:31:00.288813+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.284045+0000 mgr.y (mgr.14550) 327 : cluster 0 pgmap v375: 324 pgs: 33 active+undersized+degraded, 74 active+undersized, 4 creating+peering, 28 unknown, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.342545+0000 osd.7 (osd.7) 225 : cluster 0 9.11 scrub starts 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.344472+0000 osd.7 (osd.7) 226 : cluster 0 9.11 scrub ok 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.447078+0000 osd.6 (osd.6) 139 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.448400+0000 osd.6 (osd.6) 140 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.895513+0000 osd.0 (osd.0) 189 : cluster 0 39.4 scrub starts 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.897028+0000 osd.0 (osd.0) 190 : cluster 0 39.4 scrub ok 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.273294+0000 mon.a (mon.0) 1684 : cluster 3 Health check update: Degraded data redundancy: 182/1257 objects degraded (14.479%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: audit 2024-03-11T21:31:00.280172+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi143-71594-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: audit 2024-03-11T21:31:00.280294+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:01.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.284167+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:31:01.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:00 smithi143 bash[29284]: audit 2024-03-11T21:31:00.285563+0000 mon.c (mon.2) 446 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:01.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:00.286081+0000 mon.a (mon.0) 1688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:01.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:00.288438+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:01.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:00.288813+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]: dispatch 2024-03-11T21:31:01.212 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:01.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:01.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:01.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:01.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:01.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/3075458260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 msgr2=0x7f7ec8110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:01.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- 172.21.15.143:0/3075458260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec8110f40 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000b0d0 tx=0x7f7ec000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/3075458260 shutdown_connections 2024-03-11T21:31:01.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- 172.21.15.143:0/3075458260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec8110f40 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- 172.21.15.143:0/3075458260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8104290 0x7f7ec8104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- 172.21.15.143:0/3075458260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ec8104c40 0x7f7ec8108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/3075458260 >> 172.21.15.143:0/3075458260 conn(0x7f7ec80ffb40 msgr2=0x7f7ec8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/3075458260 shutdown_connections 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/3075458260 wait complete. 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 librados: starting msgr at 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 librados: starting objecter 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 Processor -- start 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- start start 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 librados: setting wanted keys 2024-03-11T21:31:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 librados: calling monclient init 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ec8104290 0x7f7ec8199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8104c40 0x7f7ec819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8113870 con 0x7f7ec8104290 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81136f0 con 0x7f7ec8104c40 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81139f0 con 0x7f7ec8109370 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:32778/0 (socket says 172.21.15.143:32778) 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecdce0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ec8104290 0x7f7ec8199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 -- 172.21.15.143:0/2481286402 learned_addr learned my addr 172.21.15.143:0/2481286402 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 -- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8104c40 msgr2=0x7f7ec819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:01.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8104c40 0x7f7ec819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 -- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ec8104290 msgr2=0x7f7ec8199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ec8104290 0x7f7ec8199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec819ed90 con 0x7f7ec8109370 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ec8109370 0x7f7ec819e650 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0007a20 tx=0x7f7ec0007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ebeffd700 10 client.?.objecter ms_handle_connect 0x7f7ec8109370 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ebeffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec001f070 con 0x7f7ec8109370 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7ec0003c00 con 0x7f7ec8109370 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec001b6a0 con 0x7f7ec8109370 2024-03-11T21:31:01.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec819f020 con 0x7f7ec8109370 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 10 client.41153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81a6900 con 0x7f7ec8109370 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.581+0000 7f7ecff44700 1 librados: init done 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7ec804faa0 con 0x7f7ec8109370 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7ec0022070 con 0x7f7ec8109370 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7eb406f820 0x7f7eb4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(260..260 src has 87..260) v4 ==== 10514+0+0 (secure 0 0 0) 0x7f7ec008fbd0 con 0x7f7ec8109370 2024-03-11T21:31:01.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 osd_map(260..260 src has 87..260) v4 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 3 client.41153.objecter handle_osd_map got epochs [260,260] > 0 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 3 client.41153.objecter handle_osd_map decoding full epoch 260 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7eb406f820 0x7f7eb4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 20 client.41153.objecter dump_active .. 0 homeless 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v260) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7ec0095220 con 0x7f7ec8109370 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 10 client.41153.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 1 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7eb406f820 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7eb406f820 0x7f7eb4071ce0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4004630 tx=0x7f7ec400b3c0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7ec80629f0 0x7f7ec8062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 1 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 0 1 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 4 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 10 client.41153.objecter _send_command 1 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81a9c60 con 0x7f7ec80629f0 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7ec80629f0 0x7f7ec8062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec80629f0 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7ec80629f0 0x7f7ec8062dd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81a9c60 con 0x7f7ec80629f0 2024-03-11T21:31:01.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:31:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 1 = system:0 2024-03-11T21:31:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 4 2024-03-11T21:31:01.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.585+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 0 1 2024-03-11T21:31:01.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 2 2024-03-11T21:31:01.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 2 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec819b0e0 osd=0 3 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 0 2 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 4 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _send_command 2 2024-03-11T21:31:01.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec8062470 con 0x7f7ec80629f0 2024-03-11T21:31:01.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:01.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec8062470 con 0x7f7ec80629f0 2024-03-11T21:31:01.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec819b0e0 osd=0 4 2024-03-11T21:31:01.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 0 2 2024-03-11T21:31:01.583 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 3 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter _get_session osd=-1 returning homeless 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=261}) v3 -- 0x7f7ec810b6c0 con 0x7f7ec8109370 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ec810ba60 con 0x7f7ec8109370 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_get_version_reply(handle=1 version=260) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec0029030 con 0x7f7ec8109370 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebffff700 10 client.41153.objecter _check_command_map_dne tid 3 current 260 map_dne_bound 260 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebffff700 10 client.41153.objecter _finish_command 3 = osd:6 osd down 2024-03-11T21:31:01.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebffff700 15 client.41153.objecter _session_command_op_remove -1 3 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 4 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7ec802cbc0 0x7f7ec802cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 4 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 2 4 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 4 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 10 client.41153.objecter _send_command 4 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec802d520 con 0x7f7ec802cbc0 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7ec802cbc0 0x7f7ec802cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec802cbc0 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7ec802cbc0 0x7f7ec802cfe0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec802d520 con 0x7f7ec802cbc0 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 4: 0 ) v1 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 4 = system:0 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 4 2024-03-11T21:31:01.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.605+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 2 4 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 10 client.41153.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 5 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 5 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec810bef0 osd=2 3 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 2 5 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 4 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 10 client.41153.objecter _send_command 5 2024-03-11T21:31:01.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec802d980 con 0x7f7ec802cbc0 2024-03-11T21:31:01.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec802d980 con 0x7f7ec802cbc0 2024-03-11T21:31:01.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec810bef0 osd=2 4 2024-03-11T21:31:01.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 2 5 2024-03-11T21:31:01.603 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:31:01.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 10 client.41153.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 6 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7ec81ac4a0 0x7f7ec81ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 6 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 3 6 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 4 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 10 client.41153.objecter _send_command 6 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81b16f0 con 0x7f7ec81ac4a0 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7ec81ac4a0 0x7f7ec81ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec81ac4a0 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.625+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7ec81ac4a0 0x7f7ec81ae890 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.629+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81b16f0 con 0x7f7ec81ac4a0 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.629+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 6: 0 ) v1 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.629+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 6 = system:0 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.629+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 4 2024-03-11T21:31:01.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.629+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 3 6 2024-03-11T21:31:01.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 10 client.41153.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 7 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 7 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81abe20 osd=3 3 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 3 7 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 4 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 10 client.41153.objecter _send_command 7 2024-03-11T21:31:01.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec810b1a0 con 0x7f7ec81ac4a0 2024-03-11T21:31:01.623 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec810b1a0 con 0x7f7ec81ac4a0 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81abe20 osd=3 4 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 3 7 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 10 client.41153.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 8 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7ec81aff50 0x7f7ec81b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 8 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 4 8 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 4 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 10 client.41153.objecter _send_command 8 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.645+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81a9a90 con 0x7f7ec81aff50 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7ec81aff50 0x7f7ec81b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec81aff50 2024-03-11T21:31:01.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7ec81aff50 0x7f7ec81b0370 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81a9a90 con 0x7f7ec81aff50 2024-03-11T21:31:01.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 8: 0 ) v1 2024-03-11T21:31:01.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 8 = system:0 2024-03-11T21:31:01.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 4 2024-03-11T21:31:01.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.649+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 4 8 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 10 client.41153.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 9 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 9 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81a92b0 osd=4 3 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 4 9 2024-03-11T21:31:01.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 4 2024-03-11T21:31:01.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 10 client.41153.objecter _send_command 9 2024-03-11T21:31:01.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.665+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec81a8d30 con 0x7f7ec81aff50 2024-03-11T21:31:01.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec81a8d30 con 0x7f7ec81aff50 2024-03-11T21:31:01.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81a92b0 osd=4 4 2024-03-11T21:31:01.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 4 9 2024-03-11T21:31:01.648 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 10 client.41153.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 10 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7ec81b1080 0x7f7ec81b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 10 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 5 10 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 4 2024-03-11T21:31:01.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 10 client.41153.objecter _send_command 10 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.669+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81b8720 con 0x7f7ec81b1080 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7ec81b1080 0x7f7ec81b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec81b1080 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ecdce0700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7ec81b1080 0x7f7ec81b58c0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81b8720 con 0x7f7ec81b1080 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 10: 0 ) v1 2024-03-11T21:31:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 10 = system:0 2024-03-11T21:31:01.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 4 2024-03-11T21:31:01.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.673+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 5 10 2024-03-11T21:31:01.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 10 client.41153.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 11 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 11 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81a9a90 osd=5 3 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 5 11 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 4 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 10 client.41153.objecter _send_command 11 2024-03-11T21:31:01.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.693+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec81b0b00 con 0x7f7ec81b1080 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec81b0b00 con 0x7f7ec81b1080 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81a9a90 osd=5 4 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 5 11 2024-03-11T21:31:01.673 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 10 client.41153.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 12 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7ec81b0c70 0x7f7ec81b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 12 2024-03-11T21:31:01.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 6 12 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 4 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 10 client.41153.objecter _send_command 12 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81afaf0 con 0x7f7ec81b0c70 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7ec81b0c70 0x7f7ec81b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec81b0c70 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ecd4df700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7ec81b0c70 0x7f7ec81b8f90 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81afaf0 con 0x7f7ec81b0c70 2024-03-11T21:31:01.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 12: 0 ) v1 2024-03-11T21:31:01.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 12 = system:0 2024-03-11T21:31:01.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 4 2024-03-11T21:31:01.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.697+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 6 12 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 10 client.41153.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 13 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 13 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81b0b00 osd=6 3 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 6 13 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 4 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 10 client.41153.objecter _send_command 13 2024-03-11T21:31:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec81afd30 con 0x7f7ec81b0c70 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec81afd30 con 0x7f7ec81b0c70 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81b0b00 osd=6 4 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 6 13 2024-03-11T21:31:01.695 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:31:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 10 client.41153.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 14 2024-03-11T21:31:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7ec81b99e0 0x7f7ec81bbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 14 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 7 14 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 4 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 10 client.41153.objecter _send_command 14 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec81bed50 con 0x7f7ec81b99e0 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.717+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7ec81b99e0 0x7f7ec81bbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 10 client.41153.objecter ms_handle_connect 0x7f7ec81b99e0 2024-03-11T21:31:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ece4e1700 1 --2- 172.21.15.143:0/2481286402 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7ec81b99e0 0x7f7ec81bbdd0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ec81bed50 con 0x7f7ec81b99e0 2024-03-11T21:31:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 14: 0 ) v1 2024-03-11T21:31:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 14 = system:0 2024-03-11T21:31:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 4 2024-03-11T21:31:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.721+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 7 14 2024-03-11T21:31:01.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 10 client.41153.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign -1 15 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 20 client.41153.objecter _get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_remove -1 15 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 20 client.41153.objecter get_session s=0x7f7ec81afc00 osd=7 3 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 15 client.41153.objecter _session_command_op_assign 7 15 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 4 2024-03-11T21:31:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 10 client.41153.objecter _send_command 15 2024-03-11T21:31:01.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.737+0000 7f7ecff44700 1 -- 172.21.15.143:0/2481286402 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7ec81bdae0 con 0x7f7ec81b99e0 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.741+0000 7f7ebeffd700 1 -- 172.21.15.143:0/2481286402 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ec0074e50 con 0x7f7ec81b99e0 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.741+0000 7f7ebeffd700 10 client.41153.objecter ms_dispatch 0x7f7ec8001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.741+0000 7f7ebeffd700 10 client.41153.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.741+0000 7f7ebeffd700 20 client.41153.objecter put_session s=0x7f7ec81afc00 osd=7 4 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:01.741+0000 7f7ebeffd700 15 client.41153.objecter _session_command_op_remove 7 15 2024-03-11T21:31:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:01.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:01.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:01.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:01.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:01.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.978 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.839675+0000 osd.2 (osd.2) 147 : cluster 0 2.1 scrub starts 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:30:59.840713+0000 osd.2 (osd.2) 148 : cluster 0 2.1 scrub ok 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.308068+0000 osd.5 (osd.5) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.308521+0000 osd.5 (osd.5) 196 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:00.921608+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.944066+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.964908+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.225768+0000 mon.a (mon.0) 1690 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:01.228655+0000 mon.a (mon.0) 1691 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:01.588922+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.143:0/2481286402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.686049+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:31:01.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[29284]: audit 2024-03-11T21:31:01.689265+0000 mon.a (mon.0) 1693 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.839675+0000 osd.2 (osd.2) 147 : cluster 0 2.1 scrub starts 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:30:59.840713+0000 osd.2 (osd.2) 148 : cluster 0 2.1 scrub ok 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.308068+0000 osd.5 (osd.5) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.308521+0000 osd.5 (osd.5) 196 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: audit 2024-03-11T21:31:00.921608+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.944066+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.964908+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.225768+0000 mon.a (mon.0) 1690 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: audit 2024-03-11T21:31:01.228655+0000 mon.a (mon.0) 1691 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: audit 2024-03-11T21:31:01.588922+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.143:0/2481286402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.686049+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:31:01.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:01 smithi143 bash[21512]: audit 2024-03-11T21:31:01.689265+0000 mon.a (mon.0) 1693 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.839675+0000 osd.2 (osd.2) 147 : cluster 0 2.1 scrub starts 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:30:59.840713+0000 osd.2 (osd.2) 148 : cluster 0 2.1 scrub ok 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.308068+0000 osd.5 (osd.5) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.308521+0000 osd.5 (osd.5) 196 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: audit 2024-03-11T21:31:00.921608+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.944066+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.964908+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.225768+0000 mon.a (mon.0) 1690 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: audit 2024-03-11T21:31:01.228655+0000 mon.a (mon.0) 1691 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-15"}]': finished 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: audit 2024-03-11T21:31:01.588922+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.143:0/2481286402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.686049+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:31:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:01 smithi203 bash[23654]: audit 2024-03-11T21:31:01.689265+0000 mon.a (mon.0) 1693 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:02.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.109 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:02.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:02.258 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:02.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:02.388 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:02.435 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:02.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:02.516 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:02.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.646 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:02.749 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:02.759 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.851825+0000 osd.2 (osd.2) 149 : cluster 0 39.16 scrub starts 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:00.855892+0000 osd.2 (osd.2) 150 : cluster 0 39.16 scrub ok 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.284721+0000 mgr.y (mgr.14550) 328 : cluster 0 pgmap v378: 324 pgs: 32 unknown, 33 active+undersized+degraded, 74 active+undersized, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.655489+0000 osd.3 (osd.3) 199 : cluster 0 39.11 scrub starts 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.656418+0000 osd.3 (osd.3) 200 : cluster 0 39.11 scrub ok 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.832483+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.832677+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub found 15 strays 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.847875+0000 osd.0 (osd.0) 193 : cluster 0 12.b deep-scrub starts 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.849444+0000 osd.0 (osd.0) 194 : cluster 0 12.b deep-scrub ok 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.878824+0000 osd.7 (osd.7) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:01.879233+0000 osd.7 (osd.7) 228 : cluster 0 purged_snaps scrub found 12 strays 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: audit 2024-03-11T21:31:01.922401+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: audit 2024-03-11T21:31:02.238365+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: audit 2024-03-11T21:31:02.238606+0000 mon.a (mon.0) 1695 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:02.760 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.245416+0000 mon.a (mon.0) 1696 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.851825+0000 osd.2 (osd.2) 149 : cluster 0 39.16 scrub starts 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:00.855892+0000 osd.2 (osd.2) 150 : cluster 0 39.16 scrub ok 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.284721+0000 mgr.y (mgr.14550) 328 : cluster 0 pgmap v378: 324 pgs: 32 unknown, 33 active+undersized+degraded, 74 active+undersized, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.655489+0000 osd.3 (osd.3) 199 : cluster 0 39.11 scrub starts 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.656418+0000 osd.3 (osd.3) 200 : cluster 0 39.11 scrub ok 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.832483+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.832677+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub found 15 strays 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.847875+0000 osd.0 (osd.0) 193 : cluster 0 12.b deep-scrub starts 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.849444+0000 osd.0 (osd.0) 194 : cluster 0 12.b deep-scrub ok 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.878824+0000 osd.7 (osd.7) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:01.879233+0000 osd.7 (osd.7) 228 : cluster 0 purged_snaps scrub found 12 strays 2024-03-11T21:31:02.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: audit 2024-03-11T21:31:01.922401+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:02.762 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: audit 2024-03-11T21:31:02.238365+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:02.762 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: audit 2024-03-11T21:31:02.238606+0000 mon.a (mon.0) 1695 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:02.762 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:02 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.245416+0000 mon.a (mon.0) 1696 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:31:02.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:02.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.901 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.851825+0000 osd.2 (osd.2) 149 : cluster 0 39.16 scrub starts 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:00.855892+0000 osd.2 (osd.2) 150 : cluster 0 39.16 scrub ok 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.284721+0000 mgr.y (mgr.14550) 328 : cluster 0 pgmap v378: 324 pgs: 32 unknown, 33 active+undersized+degraded, 74 active+undersized, 183 active+clean, 2 active+clean+snaptrim; 586 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 182/1257 objects degraded (14.479%) 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.655489+0000 osd.3 (osd.3) 199 : cluster 0 39.11 scrub starts 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.656418+0000 osd.3 (osd.3) 200 : cluster 0 39.11 scrub ok 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.832483+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.832677+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub found 15 strays 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.847875+0000 osd.0 (osd.0) 193 : cluster 0 12.b deep-scrub starts 2024-03-11T21:31:02.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.849444+0000 osd.0 (osd.0) 194 : cluster 0 12.b deep-scrub ok 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.878824+0000 osd.7 (osd.7) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:01.879233+0000 osd.7 (osd.7) 228 : cluster 0 purged_snaps scrub found 12 strays 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: audit 2024-03-11T21:31:01.922401+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: audit 2024-03-11T21:31:02.238365+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi143-71594-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: audit 2024-03-11T21:31:02.238606+0000 mon.a (mon.0) 1695 : audit 1 from='client.? 172.21.15.143:0/2636554593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi143-71604-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:02.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:02 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.245416+0000 mon.a (mon.0) 1696 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:31:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:03.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:03.029 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:03.133 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:03.153 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/4073492257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc102880 msgr2=0x7f4dcc10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/4073492257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc102880 0x7f4dcc10ec70 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400b0a0 tx=0x7f4dc400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/4073492257 shutdown_connections 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/4073492257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc102880 0x7f4dcc10ec70 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/4073492257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc107f30 0x7f4dcc108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/4073492257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc101ec0 0x7f4dcc102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/4073492257 >> 172.21.15.143:0/4073492257 conn(0x7f4dcc078eb0 msgr2=0x7f4dcc0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/4073492257 shutdown_connections 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/4073492257 wait complete. 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 librados: starting msgr at 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 librados: starting objecter 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.233+0000 7f4dd0b7d700 1 Processor -- start 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 -- start start 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 librados: setting wanted keys 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 librados: calling monclient init 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 0x7f4dcc19d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc102880 0x7f4dcc19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 0x7f4dcc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc1116a0 con 0x7f4dcc101ec0 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc111520 con 0x7f4dcc102880 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dd0b7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc111820 con 0x7f4dcc107f30 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dca59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 0x7f4dcc19d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dca59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 0x7f4dcc19d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41090/0 (socket says 172.21.15.143:41090) 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 0x7f4dcc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dca59c700 1 -- 172.21.15.143:0/280315393 learned_addr learned my addr 172.21.15.143:0/280315393 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dc9d9b700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc102880 0x7f4dcc19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 -- 172.21.15.143:0/280315393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc102880 msgr2=0x7f4dcc19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc102880 0x7f4dcc19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 -- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 msgr2=0x7f4dcc19d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 0x7f4dcc19d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 -- 172.21.15.143:0/280315393 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc198170 con 0x7f4dcc107f30 2024-03-11T21:31:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dcad9d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 0x7f4dcc197a10 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400bfd0 tx=0x7f4dc4007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dbb7fe700 10 client.?.objecter ms_handle_connect 0x7f4dcc107f30 2024-03-11T21:31:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.237+0000 7f4dbb7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4021020 con 0x7f4dcc107f30 2024-03-11T21:31:03.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4dc4004c50 con 0x7f4dcc107f30 2024-03-11T21:31:03.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4005050 con 0x7f4dcc107f30 2024-03-11T21:31:03.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dcc198400 con 0x7f4dcc107f30 2024-03-11T21:31:03.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dd0b7d700 10 client.41159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:03.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc1a4720 con 0x7f4dcc107f30 2024-03-11T21:31:03.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.241+0000 7f4dd0b7d700 1 librados: init done 2024-03-11T21:31:03.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.245+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc103550 con 0x7f4dcc107f30 2024-03-11T21:31:03.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4dc4024020 con 0x7f4dcc107f30 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 0x7f4db4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(262..262 src has 87..262) v4 ==== 10534+0+0 (secure 0 0 0) 0x7f4dc400f020 con 0x7f4dcc107f30 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 10 client.41159.objecter ms_dispatch 0x7f4dcc001550 osd_map(262..262 src has 87..262) v4 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 3 client.41159.objecter handle_osd_map got epochs [262,262] > 0 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 3 client.41159.objecter handle_osd_map decoding full epoch 262 2024-03-11T21:31:03.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 20 client.41159.objecter dump_active .. 0 homeless 2024-03-11T21:31:03.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dca59c700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 0x7f4db4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:03.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dbb7fe700 10 client.41159.objecter ms_handle_connect 0x7f4db406f820 2024-03-11T21:31:03.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.249+0000 7f4dca59c700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 0x7f4db4071ce0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc198b50 tx=0x7f4dbc009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:03.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.253+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4dc4011e50 con 0x7f4dcc107f30 2024-03-11T21:31:03.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:03.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:03.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:03.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:03.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:03.489+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f4dcc198f40 con 0x7f4dcc107f30 2024-03-11T21:31:03.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:03.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:03.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:03.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:03.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:03.775 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.845037+0000 osd.4 (osd.4) 175 : cluster 0 5.1 scrub starts 2024-03-11T21:31:03.775 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.847430+0000 osd.0 (osd.0) 195 : cluster 0 39.14 scrub starts 2024-03-11T21:31:03.775 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.847682+0000 osd.4 (osd.4) 176 : cluster 0 5.1 scrub ok 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.848974+0000 osd.0 (osd.0) 196 : cluster 0 39.14 scrub ok 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: audit 2024-03-11T21:31:02.923619+0000 mon.c (mon.2) 451 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.924485+0000 osd.5 (osd.5) 199 : cluster 0 118.e scrub starts 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.926766+0000 osd.5 (osd.5) 200 : cluster 0 118.e scrub ok 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: cluster 2024-03-11T21:31:03.244920+0000 mon.a (mon.0) 1697 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: audit 2024-03-11T21:31:03.250136+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: audit 2024-03-11T21:31:03.256570+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: audit 2024-03-11T21:31:03.493367+0000 mon.c (mon.2) 453 : audit 1 from='client.? 172.21.15.143:0/280315393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[21512]: audit 2024-03-11T21:31:03.709925+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.845037+0000 osd.4 (osd.4) 175 : cluster 0 5.1 scrub starts 2024-03-11T21:31:03.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.847430+0000 osd.0 (osd.0) 195 : cluster 0 39.14 scrub starts 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.847682+0000 osd.4 (osd.4) 176 : cluster 0 5.1 scrub ok 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.848974+0000 osd.0 (osd.0) 196 : cluster 0 39.14 scrub ok 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: audit 2024-03-11T21:31:02.923619+0000 mon.c (mon.2) 451 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.924485+0000 osd.5 (osd.5) 199 : cluster 0 118.e scrub starts 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.926766+0000 osd.5 (osd.5) 200 : cluster 0 118.e scrub ok 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: cluster 2024-03-11T21:31:03.244920+0000 mon.a (mon.0) 1697 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: audit 2024-03-11T21:31:03.250136+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: audit 2024-03-11T21:31:03.256570+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: audit 2024-03-11T21:31:03.493367+0000 mon.c (mon.2) 453 : audit 1 from='client.? 172.21.15.143:0/280315393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:03 smithi143 bash[29284]: audit 2024-03-11T21:31:03.709925+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.788 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:03.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:03.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.845037+0000 osd.4 (osd.4) 175 : cluster 0 5.1 scrub starts 2024-03-11T21:31:03.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.847430+0000 osd.0 (osd.0) 195 : cluster 0 39.14 scrub starts 2024-03-11T21:31:03.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.847682+0000 osd.4 (osd.4) 176 : cluster 0 5.1 scrub ok 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.848974+0000 osd.0 (osd.0) 196 : cluster 0 39.14 scrub ok 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: audit 2024-03-11T21:31:02.923619+0000 mon.c (mon.2) 451 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.924485+0000 osd.5 (osd.5) 199 : cluster 0 118.e scrub starts 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.926766+0000 osd.5 (osd.5) 200 : cluster 0 118.e scrub ok 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: cluster 2024-03-11T21:31:03.244920+0000 mon.a (mon.0) 1697 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: audit 2024-03-11T21:31:03.250136+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: audit 2024-03-11T21:31:03.256570+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:03.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: audit 2024-03-11T21:31:03.493367+0000 mon.c (mon.2) 453 : audit 1 from='client.? 172.21.15.143:0/280315393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.905 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:03 smithi203 bash[23654]: audit 2024-03-11T21:31:03.709925+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:31:03.917 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:04.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.049 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:04.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.179 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:04.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.245+0000 7f4dbb7fe700 1 -- 172.21.15.143:0/280315393 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v263) v1 ==== 113+0+0 (secure 0 0 0) 0x7f4dc405c8d0 con 0x7f4dcc107f30 2024-03-11T21:31:04.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 10 librados: watch_flush enter 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 10 librados: watch_flush exit 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 20 client.41159.objecter shutdown clearing up homeless session... 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 10 client.41159.objecter successfully canceled tick 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 msgr2=0x7f4db4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 0x7f4db4071ce0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc198b50 tx=0x7f4dbc009290 comp rx=0 tx=0).stop 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 msgr2=0x7f4dcc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 0x7f4dcc197a10 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400bfd0 tx=0x7f4dc4007a20 comp rx=0 tx=0).stop 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 shutdown_connections 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4db406f820 0x7f4db4071ce0 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dcc101ec0 0x7f4dcc19d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dcc107f30 0x7f4dcc197a10 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 --2- 172.21.15.143:0/280315393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dcc102880 0x7f4dcc19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:04.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 >> 172.21.15.143:0/280315393 conn(0x7f4dcc078eb0 msgr2=0x7f4dcc105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:04.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 shutdown_connections 2024-03-11T21:31:04.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 -- 172.21.15.143:0/280315393 wait complete. 2024-03-11T21:31:04.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:04.249+0000 7f4dd0b7d700 1 librados: shutdown 2024-03-11T21:31:04.229 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:31:04.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:04.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:04.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:04.441 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:04.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:04.573 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:04.676 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:04.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:04.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.811755+0000 osd.2 (osd.2) 151 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: cluster 2024-03-11T21:31:02.819196+0000 osd.2 (osd.2) 152 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: cluster 2024-03-11T21:31:03.285355+0000 mgr.y (mgr.14550) 329 : cluster 0 pgmap v381: 300 pgs: 1 active+clean+snaptrim, 7 creating+peering, 33 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:03.925406+0000 mon.c (mon.2) 454 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: cluster 2024-03-11T21:31:04.242382+0000 mon.a (mon.0) 1700 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.245611+0000 mon.a (mon.0) 1701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.245699+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: cluster 2024-03-11T21:31:04.250463+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:31:04.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.252525+0000 mon.c (mon.2) 455 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.252725+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.259606+0000 mon.a (mon.0) 1704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.259899+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.296956+0000 mon.c (mon.2) 457 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[21512]: audit 2024-03-11T21:31:04.297345+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.811755+0000 osd.2 (osd.2) 151 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: cluster 2024-03-11T21:31:02.819196+0000 osd.2 (osd.2) 152 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: cluster 2024-03-11T21:31:03.285355+0000 mgr.y (mgr.14550) 329 : cluster 0 pgmap v381: 300 pgs: 1 active+clean+snaptrim, 7 creating+peering, 33 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:04.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:03.925406+0000 mon.c (mon.2) 454 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: cluster 2024-03-11T21:31:04.242382+0000 mon.a (mon.0) 1700 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.245611+0000 mon.a (mon.0) 1701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.245699+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: cluster 2024-03-11T21:31:04.250463+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.252525+0000 mon.c (mon.2) 455 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.252725+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.259606+0000 mon.a (mon.0) 1704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:04.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.259899+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:04.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.296956+0000 mon.c (mon.2) 457 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:04.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:04 smithi143 bash[29284]: audit 2024-03-11T21:31:04.297345+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:04.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.811755+0000 osd.2 (osd.2) 151 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: cluster 2024-03-11T21:31:02.819196+0000 osd.2 (osd.2) 152 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: cluster 2024-03-11T21:31:03.285355+0000 mgr.y (mgr.14550) 329 : cluster 0 pgmap v381: 300 pgs: 1 active+clean+snaptrim, 7 creating+peering, 33 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:03.925406+0000 mon.c (mon.2) 454 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: cluster 2024-03-11T21:31:04.242382+0000 mon.a (mon.0) 1700 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.245611+0000 mon.a (mon.0) 1701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.245699+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: cluster 2024-03-11T21:31:04.250463+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.252525+0000 mon.c (mon.2) 455 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.252725+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.259606+0000 mon.a (mon.0) 1704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.259899+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.296956+0000 mon.c (mon.2) 457 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:04 smithi203 bash[23654]: audit 2024-03-11T21:31:04.297345+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:05.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:05.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:31:05.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:31:05.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:05.228 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:05.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:05.397 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:05 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1.service: Failed with result 'exit-code'. 2024-03-11T21:31:05.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:05.483 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:05.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- 172.21.15.143:0/2927570359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde64106ef0 msgr2=0x7fde641072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:05.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- 172.21.15.143:0/2927570359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde64106ef0 0x7fde641072d0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fde600099c0 tx=0x7fde60009cd0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- 172.21.15.143:0/2927570359 shutdown_connections 2024-03-11T21:31:05.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- 172.21.15.143:0/2927570359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde64106ef0 0x7fde641072d0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- 172.21.15.143:0/2927570359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde641019c0 0x7fde6410db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- 172.21.15.143:0/2927570359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64101000 0x7fde64101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- 172.21.15.143:0/2927570359 >> 172.21.15.143:0/2927570359 conn(0x7fde64077a40 msgr2=0x7fde64077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:05.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- 172.21.15.143:0/2927570359 shutdown_connections 2024-03-11T21:31:05.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- 172.21.15.143:0/2927570359 wait complete. 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 librados: starting msgr at 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 librados: starting objecter 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 Processor -- start 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 -- start start 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 librados: setting wanted keys 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 librados: calling monclient init 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde64101000 0x7fde64102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.584+0000 7fde6bfaf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde641019c0 0x7fde64102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde641105d0 con 0x7fde641019c0 2024-03-11T21:31:05.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde64110450 con 0x7fde64101000 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde64110750 con 0x7fde64106ef0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58032/0 (socket says 172.21.15.143:58032) 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 -- 172.21.15.143:0/678565147 learned_addr learned my addr 172.21.15.143:0/678565147 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6954a700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde641019c0 0x7fde64102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde69d4b700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde64101000 0x7fde64102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 -- 172.21.15.143:0/678565147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde64101000 msgr2=0x7fde64102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde64101000 0x7fde64102a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 -- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde641019c0 msgr2=0x7fde64102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde641019c0 0x7fde64102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 -- 172.21.15.143:0/678565147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde641a6130 con 0x7fde64106ef0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6a54c700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fde5c007ee0 tx=0x7fde5c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 10 client.?.objecter ms_handle_connect 0x7fde64106ef0 2024-03-11T21:31:05.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00dbd0 con 0x7fde64106ef0 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fde5c00dd70 con 0x7fde64106ef0 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 -- 172.21.15.143:0/678565147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde641a6420 con 0x7fde64106ef0 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c005840 con 0x7fde64106ef0 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 10 client.41171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 -- 172.21.15.143:0/678565147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde641a6910 con 0x7fde64106ef0 2024-03-11T21:31:05.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde6bfaf700 1 librados: init done 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c005120 con 0x7fde64106ef0 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fde5c01a020 con 0x7fde64106ef0 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 0x7fde50071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(264..264 src has 87..264) v4 ==== 10203+0+0 (secure 0 0 0) 0x7fde5c08e480 con 0x7fde64106ef0 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde69d4b700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 0x7fde50071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 10 client.41171.objecter ms_dispatch 0x7fde64001550 osd_map(264..264 src has 87..264) v4 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 3 client.41171.objecter handle_osd_map got epochs [264,264] > 0 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 3 client.41171.objecter handle_osd_map decoding full epoch 264 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 20 client.41171.objecter dump_active .. 0 homeless 2024-03-11T21:31:05.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde5affd700 10 client.41171.objecter ms_handle_connect 0x7fde5006f820 2024-03-11T21:31:05.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.588+0000 7fde69d4b700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 0x7fde50071ce0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7fde600059d0 tx=0x7fde60005940 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.596+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fde5c010070 con 0x7fde64106ef0 2024-03-11T21:31:05.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:05.610 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:05.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:05.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:05.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:05.824+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fde4c005410 con 0x7fde64106ef0 2024-03-11T21:31:05.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:05.867 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:05.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: cluster 2024-03-11T21:31:04.353567+0000 osd.7 (osd.7) 229 : cluster 0 6.3 scrub starts 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: cluster 2024-03-11T21:31:04.354962+0000 osd.7 (osd.7) 230 : cluster 0 6.3 scrub ok 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:04.927222+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.248848+0000 mon.a (mon.0) 1707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.249095+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.249357+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.253745+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.253866+0000 mon.c (mon.2) 460 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.256692+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.259127+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[29284]: audit 2024-03-11T21:31:05.259729+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: cluster 2024-03-11T21:31:04.353567+0000 osd.7 (osd.7) 229 : cluster 0 6.3 scrub starts 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: cluster 2024-03-11T21:31:04.354962+0000 osd.7 (osd.7) 230 : cluster 0 6.3 scrub ok 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:04.927222+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.248848+0000 mon.a (mon.0) 1707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.249095+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.249357+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.253745+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.253866+0000 mon.c (mon.2) 460 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.256692+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.259127+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:05.980 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:05 smithi143 bash[21512]: audit 2024-03-11T21:31:05.259729+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:05.989 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:06.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: cluster 2024-03-11T21:31:04.353567+0000 osd.7 (osd.7) 229 : cluster 0 6.3 scrub starts 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: cluster 2024-03-11T21:31:04.354962+0000 osd.7 (osd.7) 230 : cluster 0 6.3 scrub ok 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:04.927222+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.248848+0000 mon.a (mon.0) 1707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.249095+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi143-71604-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.249357+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.253745+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.143:0/2586124918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.253866+0000 mon.c (mon.2) 460 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.256692+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.259127+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]: dispatch 2024-03-11T21:31:06.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:05 smithi203 bash[23654]: audit 2024-03-11T21:31:05.259729+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:06.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:06.117 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:06.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:31:06.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.244 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:06.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.288+0000 7fde5affd700 1 -- 172.21.15.143:0/678565147 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v265) v1 ==== 87+0+0 (secure 0 0 0) 0x7fde5c05ae90 con 0x7fde64106ef0 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 10 librados: watch_flush enter 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 10 librados: watch_flush exit 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 20 client.41171.objecter shutdown clearing up homeless session... 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 10 client.41171.objecter successfully canceled tick 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 msgr2=0x7fde50071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 0x7fde50071ce0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7fde600059d0 tx=0x7fde60005940 comp rx=0 tx=0).stop 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 msgr2=0x7fde641034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:06.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.292+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fde5c007ee0 tx=0x7fde5c00d3b0 comp rx=0 tx=0).stop 2024-03-11T21:31:06.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 shutdown_connections 2024-03-11T21:31:06.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fde5006f820 0x7fde50071ce0 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:06.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde64106ef0 0x7fde641034b0 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde64101000 0x7fde64102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 --2- 172.21.15.143:0/678565147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde641019c0 0x7fde64102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 >> 172.21.15.143:0/678565147 conn(0x7fde64077a40 msgr2=0x7fde640fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 shutdown_connections 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 -- 172.21.15.143:0/678565147 wait complete. 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:06.296+0000 7fde58ff9700 1 librados: shutdown 2024-03-11T21:31:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:31:06.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:06.374 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:06.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:06.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:06.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.764 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:06.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:06.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:06.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.286014+0000 mgr.y (mgr.14550) 330 : cluster 0 pgmap v384: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.360496+0000 osd.7 (osd.7) 231 : cluster 0 5.9 scrub starts 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.363044+0000 osd.7 (osd.7) 232 : cluster 0 5.9 scrub ok 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.575298+0000 osd.6 (osd.6) 141 : cluster 0 39.19 scrub starts 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.576554+0000 osd.6 (osd.6) 142 : cluster 0 39.19 scrub ok 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762521+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{ 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762567+0000 mon.c (mon.2) 462 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762601+0000 mon.c (mon.2) 463 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762639+0000 mon.c (mon.2) 464 : audit 1 "var": "eio", 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762674+0000 mon.c (mon.2) 465 : audit 1 "val": "true" 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.762708+0000 mon.c (mon.2) 466 : audit 1 }]: dispatch 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763432+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763502+0000 mon.a (mon.0) 1714 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763548+0000 mon.a (mon.0) 1715 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763593+0000 mon.a (mon.0) 1716 : audit 1 "var": "eio", 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763633+0000 mon.a (mon.0) 1717 : audit 1 "val": "true" 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.763736+0000 mon.a (mon.0) 1718 : audit 1 }]: dispatch 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.764654+0000 osd.2 (osd.2) 153 : cluster 0 118.d scrub starts 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.766451+0000 osd.2 (osd.2) 154 : cluster 0 118.d scrub ok 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.817377+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.819552+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.829260+0000 mon.c (mon.2) 467 : audit 1 from='client.? 172.21.15.143:0/678565147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.829627+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.885598+0000 osd.5 (osd.5) 201 : cluster 0 118.10 scrub starts 2024-03-11T21:31:06.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:05.888494+0000 osd.5 (osd.5) 202 : cluster 0 118.10 scrub ok 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.286014+0000 mgr.y (mgr.14550) 330 : cluster 0 pgmap v384: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.360496+0000 osd.7 (osd.7) 231 : cluster 0 5.9 scrub starts 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.363044+0000 osd.7 (osd.7) 232 : cluster 0 5.9 scrub ok 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.575298+0000 osd.6 (osd.6) 141 : cluster 0 39.19 scrub starts 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.576554+0000 osd.6 (osd.6) 142 : cluster 0 39.19 scrub ok 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762521+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{ 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762567+0000 mon.c (mon.2) 462 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762601+0000 mon.c (mon.2) 463 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762639+0000 mon.c (mon.2) 464 : audit 1 "var": "eio", 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762674+0000 mon.c (mon.2) 465 : audit 1 "val": "true" 2024-03-11T21:31:06.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.762708+0000 mon.c (mon.2) 466 : audit 1 }]: dispatch 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763432+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763502+0000 mon.a (mon.0) 1714 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763548+0000 mon.a (mon.0) 1715 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763593+0000 mon.a (mon.0) 1716 : audit 1 "var": "eio", 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763633+0000 mon.a (mon.0) 1717 : audit 1 "val": "true" 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.763736+0000 mon.a (mon.0) 1718 : audit 1 }]: dispatch 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.764654+0000 osd.2 (osd.2) 153 : cluster 0 118.d scrub starts 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.766451+0000 osd.2 (osd.2) 154 : cluster 0 118.d scrub ok 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.817377+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.819552+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.829260+0000 mon.c (mon.2) 467 : audit 1 from='client.? 172.21.15.143:0/678565147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.829627+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.885598+0000 osd.5 (osd.5) 201 : cluster 0 118.10 scrub starts 2024-03-11T21:31:06.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:05.888494+0000 osd.5 (osd.5) 202 : cluster 0 118.10 scrub ok 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.928637+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.975601+0000 mon.a (mon.0) 1720 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:05.979634+0000 mon.c (mon.2) 469 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:06.227854+0000 mon.a (mon.0) 1721 : cluster 3 Health check update: Degraded data redundancy: 182/1239 objects degraded (14.689%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:06.250172+0000 mon.a (mon.0) 1722 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261064+0000 mon.a (mon.0) 1723 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261187+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261302+0000 mon.a (mon.0) 1725 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261328+0000 mon.a (mon.0) 1726 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261344+0000 mon.a (mon.0) 1727 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261361+0000 mon.a (mon.0) 1728 : audit 1 "var": "eio", 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261373+0000 mon.a (mon.0) 1729 : audit 1 "val": "true" 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261405+0000 mon.a (mon.0) 1730 : audit 1 }]': finished 2024-03-11T21:31:06.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.261514+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: cluster 2024-03-11T21:31:06.283605+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.293404+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.293831+0000 mon.a (mon.0) 1733 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.319996+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.320508+0000 mon.a (mon.0) 1735 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[21512]: audit 2024-03-11T21:31:06.320956+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.928637+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.975601+0000 mon.a (mon.0) 1720 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:05.979634+0000 mon.c (mon.2) 469 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:06.227854+0000 mon.a (mon.0) 1721 : cluster 3 Health check update: Degraded data redundancy: 182/1239 objects degraded (14.689%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:06.250172+0000 mon.a (mon.0) 1722 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:06.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261064+0000 mon.a (mon.0) 1723 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261187+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261302+0000 mon.a (mon.0) 1725 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261328+0000 mon.a (mon.0) 1726 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261344+0000 mon.a (mon.0) 1727 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261361+0000 mon.a (mon.0) 1728 : audit 1 "var": "eio", 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261373+0000 mon.a (mon.0) 1729 : audit 1 "val": "true" 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261405+0000 mon.a (mon.0) 1730 : audit 1 }]': finished 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.261514+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: cluster 2024-03-11T21:31:06.283605+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.293404+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.293831+0000 mon.a (mon.0) 1733 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.319996+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.320508+0000 mon.a (mon.0) 1735 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:06 smithi143 bash[29284]: audit 2024-03-11T21:31:06.320956+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:06.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.286014+0000 mgr.y (mgr.14550) 330 : cluster 0 pgmap v384: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.360496+0000 osd.7 (osd.7) 231 : cluster 0 5.9 scrub starts 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.363044+0000 osd.7 (osd.7) 232 : cluster 0 5.9 scrub ok 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.575298+0000 osd.6 (osd.6) 141 : cluster 0 39.19 scrub starts 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.576554+0000 osd.6 (osd.6) 142 : cluster 0 39.19 scrub ok 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762521+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.143:0/2113807600' entity='client.admin' cmd=[{ 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762567+0000 mon.c (mon.2) 462 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:07.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762601+0000 mon.c (mon.2) 463 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762639+0000 mon.c (mon.2) 464 : audit 1 "var": "eio", 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762674+0000 mon.c (mon.2) 465 : audit 1 "val": "true" 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.762708+0000 mon.c (mon.2) 466 : audit 1 }]: dispatch 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763432+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763502+0000 mon.a (mon.0) 1714 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763548+0000 mon.a (mon.0) 1715 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763593+0000 mon.a (mon.0) 1716 : audit 1 "var": "eio", 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763633+0000 mon.a (mon.0) 1717 : audit 1 "val": "true" 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.763736+0000 mon.a (mon.0) 1718 : audit 1 }]: dispatch 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.764654+0000 osd.2 (osd.2) 153 : cluster 0 118.d scrub starts 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.766451+0000 osd.2 (osd.2) 154 : cluster 0 118.d scrub ok 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.817377+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.819552+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.829260+0000 mon.c (mon.2) 467 : audit 1 from='client.? 172.21.15.143:0/678565147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:07.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.829627+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.885598+0000 osd.5 (osd.5) 201 : cluster 0 118.10 scrub starts 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:05.888494+0000 osd.5 (osd.5) 202 : cluster 0 118.10 scrub ok 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.928637+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.975601+0000 mon.a (mon.0) 1720 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:05.979634+0000 mon.c (mon.2) 469 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:06.227854+0000 mon.a (mon.0) 1721 : cluster 3 Health check update: Degraded data redundancy: 182/1239 objects degraded (14.689%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:06.250172+0000 mon.a (mon.0) 1722 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261064+0000 mon.a (mon.0) 1723 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi143-71594-30"}]': finished 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261187+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:07.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261302+0000 mon.a (mon.0) 1725 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261328+0000 mon.a (mon.0) 1726 : audit 1 "prefix": "osd pool set", 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261344+0000 mon.a (mon.0) 1727 : audit 1 "pool": "PoolEIOFlag_smithi143-71604-33", 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261361+0000 mon.a (mon.0) 1728 : audit 1 "var": "eio", 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261373+0000 mon.a (mon.0) 1729 : audit 1 "val": "true" 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261405+0000 mon.a (mon.0) 1730 : audit 1 }]': finished 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.261514+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: cluster 2024-03-11T21:31:06.283605+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.293404+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.293831+0000 mon.a (mon.0) 1733 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]: dispatch 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.319996+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.320508+0000 mon.a (mon.0) 1735 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:06 smithi203 bash[23654]: audit 2024-03-11T21:31:06.320956+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:07.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:07.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:07.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:07.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:31:07.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:07.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:07.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:07.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:07.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:07.624 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:07.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:07.678 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:07.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:07.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:07.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:07.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:06.929876+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:07.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: cluster 2024-03-11T21:31:07.261359+0000 mon.a (mon.0) 1737 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:07.263043+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]': finished 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:07.263136+0000 mon.a (mon.0) 1739 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: cluster 2024-03-11T21:31:07.266970+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:07.268242+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:07.413327+0000 mon.c (mon.2) 472 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.919 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:07 smithi203 bash[23654]: audit 2024-03-11T21:31:07.731867+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:06.929876+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: cluster 2024-03-11T21:31:07.261359+0000 mon.a (mon.0) 1737 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:07.263043+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]': finished 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:07.263136+0000 mon.a (mon.0) 1739 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: cluster 2024-03-11T21:31:07.266970+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:07.268242+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:07.413327+0000 mon.c (mon.2) 472 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[21512]: audit 2024-03-11T21:31:07.731867+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:06.929876+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: cluster 2024-03-11T21:31:07.261359+0000 mon.a (mon.0) 1737 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:07.263043+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-17", "mode": "writeback"}]': finished 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:07.263136+0000 mon.a (mon.0) 1739 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi143-71594-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: cluster 2024-03-11T21:31:07.266970+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:07.268242+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:07.413327+0000 mon.c (mon.2) 472 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:07 smithi143 bash[29284]: audit 2024-03-11T21:31:07.731867+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:07.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:08.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:08.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:08.197 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:08.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.324 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:08.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:08.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:08.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:08.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:08.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:08.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:08.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:08.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: cluster 2024-03-11T21:31:07.286612+0000 mgr.y (mgr.14550) 331 : cluster 0 pgmap v387: 292 pgs: 1 active+clean+snaptrim, 6 creating+peering, 26 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:07.931709+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:08.272319+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:08.282333+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: cluster 2024-03-11T21:31:08.723461+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:08.724712+0000 mon.a (mon.0) 1745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:08.726738+0000 mon.c (mon.2) 475 : audit 1 from='client.? 172.21.15.143:0/3740019788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[21512]: audit 2024-03-11T21:31:08.727911+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: cluster 2024-03-11T21:31:07.286612+0000 mgr.y (mgr.14550) 331 : cluster 0 pgmap v387: 292 pgs: 1 active+clean+snaptrim, 6 creating+peering, 26 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:08.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:07.931709+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:08.272319+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:08.282333+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: cluster 2024-03-11T21:31:08.723461+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:08.724712+0000 mon.a (mon.0) 1745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:08.726738+0000 mon.c (mon.2) 475 : audit 1 from='client.? 172.21.15.143:0/3740019788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:08.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:08 smithi143 bash[29284]: audit 2024-03-11T21:31:08.727911+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:08.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:09.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: cluster 2024-03-11T21:31:07.286612+0000 mgr.y (mgr.14550) 331 : cluster 0 pgmap v387: 292 pgs: 1 active+clean+snaptrim, 6 creating+peering, 26 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s; 182/1239 objects degraded (14.689%) 2024-03-11T21:31:09.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:07.931709+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:09.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:08.272319+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:09.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:08.282333+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:09.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: cluster 2024-03-11T21:31:08.723461+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:31:09.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:08.724712+0000 mon.a (mon.0) 1745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]: dispatch 2024-03-11T21:31:09.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:08.726738+0000 mon.c (mon.2) 475 : audit 1 from='client.? 172.21.15.143:0/3740019788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:09.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:08 smithi203 bash[23654]: audit 2024-03-11T21:31:08.727911+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:09.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:09.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:09.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:09.209 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:09.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:31:09.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:09.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:09.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.469 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:09.595 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:09.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.727 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:09.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.858 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:09.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: audit 2024-03-11T21:31:08.440142+0000 mgr.y (mgr.14550) 332 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:09.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: audit 2024-03-11T21:31:08.933349+0000 mon.c (mon.2) 476 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: cluster 2024-03-11T21:31:09.273044+0000 mon.a (mon.0) 1747 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: audit 2024-03-11T21:31:09.274821+0000 mon.a (mon.0) 1748 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: audit 2024-03-11T21:31:09.274930+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: audit 2024-03-11T21:31:09.275024+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[29284]: cluster 2024-03-11T21:31:09.279421+0000 mon.a (mon.0) 1751 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: audit 2024-03-11T21:31:08.440142+0000 mgr.y (mgr.14550) 332 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: audit 2024-03-11T21:31:08.933349+0000 mon.c (mon.2) 476 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: cluster 2024-03-11T21:31:09.273044+0000 mon.a (mon.0) 1747 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: audit 2024-03-11T21:31:09.274821+0000 mon.a (mon.0) 1748 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:09.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: audit 2024-03-11T21:31:09.274930+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: audit 2024-03-11T21:31:09.275024+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:09 smithi143 bash[21512]: cluster 2024-03-11T21:31:09.279421+0000 mon.a (mon.0) 1751 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:31:09.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:09.986 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:10.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: audit 2024-03-11T21:31:08.440142+0000 mgr.y (mgr.14550) 332 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: audit 2024-03-11T21:31:08.933349+0000 mon.c (mon.2) 476 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: cluster 2024-03-11T21:31:09.273044+0000 mon.a (mon.0) 1747 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: audit 2024-03-11T21:31:09.274821+0000 mon.a (mon.0) 1748 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi143-71594-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: audit 2024-03-11T21:31:09.274930+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-17"}]': finished 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: audit 2024-03-11T21:31:09.275024+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi143-71604-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:10.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:09 smithi203 bash[23654]: cluster 2024-03-11T21:31:09.279421+0000 mon.a (mon.0) 1751 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:31:10.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.113 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:10.238 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:10.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.703 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:10.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:10.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:10.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:10.866 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:10.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:10.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:11.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:11.123 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:11.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:11.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:11.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:11.370 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:11 smithi203 bash[23654]: cluster 2024-03-11T21:31:09.287489+0000 mgr.y (mgr.14550) 333 : cluster 0 pgmap v390: 332 pgs: 1 active+clean+snaptrim, 40 unknown, 33 active+undersized+degraded, 78 active+undersized, 180 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:11.370 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:11 smithi203 bash[23654]: audit 2024-03-11T21:31:09.935143+0000 mon.c (mon.2) 477 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:11.370 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:11 smithi203 bash[23654]: cluster 2024-03-11T21:31:10.274890+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:11.370 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:11 smithi203 bash[23654]: cluster 2024-03-11T21:31:10.289686+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:31:11.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[29284]: cluster 2024-03-11T21:31:09.287489+0000 mgr.y (mgr.14550) 333 : cluster 0 pgmap v390: 332 pgs: 1 active+clean+snaptrim, 40 unknown, 33 active+undersized+degraded, 78 active+undersized, 180 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[29284]: audit 2024-03-11T21:31:09.935143+0000 mon.c (mon.2) 477 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[29284]: cluster 2024-03-11T21:31:10.274890+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[29284]: cluster 2024-03-11T21:31:10.289686+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:31:11.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[21512]: cluster 2024-03-11T21:31:09.287489+0000 mgr.y (mgr.14550) 333 : cluster 0 pgmap v390: 332 pgs: 1 active+clean+snaptrim, 40 unknown, 33 active+undersized+degraded, 78 active+undersized, 180 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:11.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[21512]: audit 2024-03-11T21:31:09.935143+0000 mon.c (mon.2) 477 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:11.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[21512]: cluster 2024-03-11T21:31:10.274890+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:11.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:11 smithi143 bash[21512]: cluster 2024-03-11T21:31:10.289686+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:31:11.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:11.514 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:11.617 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:11.645 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:11.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:11.776 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:11.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:11.907 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:12.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:12.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:12.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:12.168 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:12.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:12.279 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:12.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:10.936765+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: cluster 2024-03-11T21:31:11.229271+0000 mon.a (mon.0) 1754 : cluster 3 Health check update: Degraded data redundancy: 183/1230 objects degraded (14.878%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: cluster 2024-03-11T21:31:11.246366+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:11.255234+0000 mon.a (mon.0) 1756 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:11.261702+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:11.273163+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:11.280736+0000 mon.a (mon.0) 1758 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: cluster 2024-03-11T21:31:11.288170+0000 mgr.y (mgr.14550) 334 : cluster 0 pgmap v393: 324 pgs: 1 active+clean+snaptrim, 64 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[29284]: audit 2024-03-11T21:31:11.938241+0000 mon.c (mon.2) 480 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:10.936765+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: cluster 2024-03-11T21:31:11.229271+0000 mon.a (mon.0) 1754 : cluster 3 Health check update: Degraded data redundancy: 183/1230 objects degraded (14.878%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: cluster 2024-03-11T21:31:11.246366+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:11.255234+0000 mon.a (mon.0) 1756 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:12.280 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:11.261702+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:11.273163+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:11.280736+0000 mon.a (mon.0) 1758 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: cluster 2024-03-11T21:31:11.288170+0000 mgr.y (mgr.14550) 334 : cluster 0 pgmap v393: 324 pgs: 1 active+clean+snaptrim, 64 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:12.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:12 smithi143 bash[21512]: audit 2024-03-11T21:31:11.938241+0000 mon.c (mon.2) 480 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:12.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:10.936765+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: cluster 2024-03-11T21:31:11.229271+0000 mon.a (mon.0) 1754 : cluster 3 Health check update: Degraded data redundancy: 183/1230 objects degraded (14.878%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: cluster 2024-03-11T21:31:11.246366+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:11.255234+0000 mon.a (mon.0) 1756 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:11.261702+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:11.273163+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:11.280736+0000 mon.a (mon.0) 1758 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: cluster 2024-03-11T21:31:11.288170+0000 mgr.y (mgr.14550) 334 : cluster 0 pgmap v393: 324 pgs: 1 active+clean+snaptrim, 64 unknown, 33 active+undersized+degraded, 63 active+undersized, 163 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 183/1230 objects degraded (14.878%) 2024-03-11T21:31:12.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:12 smithi203 bash[23654]: audit 2024-03-11T21:31:11.938241+0000 mon.c (mon.2) 480 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:12.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:12.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:12.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:12.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:12.676 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:12.702 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:31:12.702 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:31:12.702 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:31:12.702 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:31:12.702 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:31:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:12.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:12.800 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:12.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:12.929 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:13.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:13.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:13.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:13.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:13.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.237873+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.238096+0000 mon.a (mon.0) 1760 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.238228+0000 mon.a (mon.0) 1761 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: cluster 2024-03-11T21:31:12.246416+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.248534+0000 mon.a (mon.0) 1763 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.252464+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.303 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.253301+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.304 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:13 smithi203 bash[23654]: audit 2024-03-11T21:31:12.940069+0000 mon.c (mon.2) 482 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:13.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:13.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:13.446 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.237873+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.238096+0000 mon.a (mon.0) 1760 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.238228+0000 mon.a (mon.0) 1761 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: cluster 2024-03-11T21:31:12.246416+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.248534+0000 mon.a (mon.0) 1763 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.252464+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.253301+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.454 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[21512]: audit 2024-03-11T21:31:12.940069+0000 mon.c (mon.2) 482 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.237873+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.238096+0000 mon.a (mon.0) 1760 : audit 1 from='client.? 172.21.15.143:0/3437465916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi143-71604-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.238228+0000 mon.a (mon.0) 1761 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: cluster 2024-03-11T21:31:12.246416+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.248534+0000 mon.a (mon.0) 1763 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]: dispatch 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.252464+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.253301+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:13.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:13 smithi143 bash[29284]: audit 2024-03-11T21:31:12.940069+0000 mon.c (mon.2) 482 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:13.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:13.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:13.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:13.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:13.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:13.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:14.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:14.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:14.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.222 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:14.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:14.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.247105+0000 mon.a (mon.0) 1765 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.247258+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: cluster 2024-03-11T21:31:13.253773+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.254636+0000 mon.c (mon.2) 483 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.255718+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.281936+0000 mon.a (mon.0) 1769 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.283366+0000 mon.a (mon.0) 1770 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: cluster 2024-03-11T21:31:13.288865+0000 mgr.y (mgr.14550) 335 : cluster 0 pgmap v396: 292 pgs: 1 active+clean+snaptrim, 3 unknown, 33 active+undersized+degraded, 78 active+undersized, 177 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.720673+0000 mon.a (mon.0) 1771 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:14.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:14 smithi203 bash[23654]: audit 2024-03-11T21:31:13.942016+0000 mon.c (mon.2) 484 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:14.352 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:14.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.247105+0000 mon.a (mon.0) 1765 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:14.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.247258+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:14.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: cluster 2024-03-11T21:31:13.253773+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:31:14.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.254636+0000 mon.c (mon.2) 483 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.255718+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.281936+0000 mon.a (mon.0) 1769 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.283366+0000 mon.a (mon.0) 1770 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: cluster 2024-03-11T21:31:13.288865+0000 mgr.y (mgr.14550) 335 : cluster 0 pgmap v396: 292 pgs: 1 active+clean+snaptrim, 3 unknown, 33 active+undersized+degraded, 78 active+undersized, 177 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.720673+0000 mon.a (mon.0) 1771 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[29284]: audit 2024-03-11T21:31:13.942016+0000 mon.c (mon.2) 484 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.247105+0000 mon.a (mon.0) 1765 : audit 1 from='client.? 172.21.15.143:0/2899448680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi143-71594-31"}]': finished 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.247258+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: cluster 2024-03-11T21:31:13.253773+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.254636+0000 mon.c (mon.2) 483 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.468 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.255718+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:14.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.281936+0000 mon.a (mon.0) 1769 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.283366+0000 mon.a (mon.0) 1770 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:14.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: cluster 2024-03-11T21:31:13.288865+0000 mgr.y (mgr.14550) 335 : cluster 0 pgmap v396: 292 pgs: 1 active+clean+snaptrim, 3 unknown, 33 active+undersized+degraded, 78 active+undersized, 177 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:14.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.720673+0000 mon.a (mon.0) 1771 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:14.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:14 smithi143 bash[21512]: audit 2024-03-11T21:31:13.942016+0000 mon.c (mon.2) 484 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:14.480 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:14.583 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:14.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:14.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:14.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:14.872 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:14.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:15.002 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:15.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:15.134 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:15.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:15.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:15.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:31:15.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:15.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:15.427 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:15 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1.service: Scheduled restart job, restart counter is at 1. 2024-03-11T21:31:15.427 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:15 smithi143 systemd[1]: Stopped Ceph osd.1 for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:31:15.427 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:15 smithi143 systemd[1]: Started Ceph osd.1 for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:31:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:15.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:15.549 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:15.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:15.676 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:15.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:15.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:15.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:15.927 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:16.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.256162+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:16.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.256288+0000 mon.a (mon.0) 1773 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:16.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.261660+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: cluster 2024-03-11T21:31:14.262168+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:31:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.263468+0000 mon.a (mon.0) 1775 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.263939+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.265313+0000 mon.c (mon.2) 486 : audit 1 from='client.? 172.21.15.143:0/4149958807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.265756+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:15 smithi203 bash[23654]: audit 2024-03-11T21:31:14.944583+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:16.057 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:16.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:16.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.256162+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:16.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.256288+0000 mon.a (mon.0) 1773 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:16.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.261660+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: cluster 2024-03-11T21:31:14.262168+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.263468+0000 mon.a (mon.0) 1775 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.263939+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.265313+0000 mon.c (mon.2) 486 : audit 1 from='client.? 172.21.15.143:0/4149958807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.265756+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[29284]: audit 2024-03-11T21:31:14.944583+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.256162+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.256288+0000 mon.a (mon.0) 1773 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi143-71594-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.261660+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: cluster 2024-03-11T21:31:14.262168+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:31:16.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.263468+0000 mon.a (mon.0) 1775 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:16.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.263939+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]: dispatch 2024-03-11T21:31:16.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.265313+0000 mon.c (mon.2) 486 : audit 1 from='client.? 172.21.15.143:0/4149958807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.265756+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:15 smithi143 bash[21512]: audit 2024-03-11T21:31:14.944583+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.188 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:16.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:16.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:16.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:16.412 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:16.436 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:16.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:16.565 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:16.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:16.692 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:16.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:16.820 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:16.866 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:16.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:15.256926+0000 mon.a (mon.0) 1778 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:16.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:15.259931+0000 mon.a (mon.0) 1779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]': finished 2024-03-11T21:31:16.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:15.260063+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:16.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:15.289513+0000 mgr.y (mgr.14550) 336 : cluster 0 pgmap v399: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:15.704962+0000 mon.a (mon.0) 1781 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:15.756451+0000 mon.c (mon.2) 488 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:15.946133+0000 mon.c (mon.2) 489 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.167497+0000 mon.a (mon.0) 1782 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:16.234654+0000 mon.a (mon.0) 1783 : cluster 3 Health check update: Degraded data redundancy: 183/1212 objects degraded (15.099%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:16.234721+0000 mon.a (mon.0) 1784 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.271554+0000 mon.a (mon.0) 1785 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.271702+0000 mon.a (mon.0) 1786 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: cluster 2024-03-11T21:31:16.276081+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:31:16.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.276755+0000 mon.c (mon.2) 490 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.279006+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.279400+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.143:0/1984573492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[29284]: audit 2024-03-11T21:31:16.279879+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:15.256926+0000 mon.a (mon.0) 1778 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:15.259931+0000 mon.a (mon.0) 1779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]': finished 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:15.260063+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:15.289513+0000 mgr.y (mgr.14550) 336 : cluster 0 pgmap v399: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:15.704962+0000 mon.a (mon.0) 1781 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:31:16.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:15.756451+0000 mon.c (mon.2) 488 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:15.946133+0000 mon.c (mon.2) 489 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.167497+0000 mon.a (mon.0) 1782 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:16.234654+0000 mon.a (mon.0) 1783 : cluster 3 Health check update: Degraded data redundancy: 183/1212 objects degraded (15.099%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:16.234721+0000 mon.a (mon.0) 1784 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.271554+0000 mon.a (mon.0) 1785 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.271702+0000 mon.a (mon.0) 1786 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: cluster 2024-03-11T21:31:16.276081+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.276755+0000 mon.c (mon.2) 490 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.279006+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.279400+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.143:0/1984573492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:16 smithi143 bash[21512]: audit 2024-03-11T21:31:16.279879+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:16.949 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:15.256926+0000 mon.a (mon.0) 1778 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:15.259931+0000 mon.a (mon.0) 1779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-19", "mode": "writeback"}]': finished 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:15.260063+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:15.289513+0000 mgr.y (mgr.14550) 336 : cluster 0 pgmap v399: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:15.704962+0000 mon.a (mon.0) 1781 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:15.756451+0000 mon.c (mon.2) 488 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:15.946133+0000 mon.c (mon.2) 489 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.167497+0000 mon.a (mon.0) 1782 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:16.234654+0000 mon.a (mon.0) 1783 : cluster 3 Health check update: Degraded data redundancy: 183/1212 objects degraded (15.099%), 33 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:16.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:16.234721+0000 mon.a (mon.0) 1784 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.271554+0000 mon.a (mon.0) 1785 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi143-71594-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.271702+0000 mon.a (mon.0) 1786 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: cluster 2024-03-11T21:31:16.276081+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.276755+0000 mon.c (mon.2) 490 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.279006+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]: dispatch 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.279400+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.143:0/1984573492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.952 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:16 smithi203 bash[23654]: audit 2024-03-11T21:31:16.279879+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:17.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:17.081 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:17.183 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:17.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:17.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:17.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:17.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:17.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:17.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:17.586 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:17.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:17.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:17.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:17.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:17.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[21512]: audit 2024-03-11T21:31:16.947368+0000 mon.c (mon.2) 492 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:17.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[21512]: cluster 2024-03-11T21:31:17.272130+0000 mon.a (mon.0) 1790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[21512]: audit 2024-03-11T21:31:17.274165+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[21512]: audit 2024-03-11T21:31:17.274245+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[21512]: cluster 2024-03-11T21:31:17.278871+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[29284]: audit 2024-03-11T21:31:16.947368+0000 mon.c (mon.2) 492 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[29284]: cluster 2024-03-11T21:31:17.272130+0000 mon.a (mon.0) 1790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[29284]: audit 2024-03-11T21:31:17.274165+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[29284]: audit 2024-03-11T21:31:17.274245+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:17.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:17 smithi143 bash[29284]: cluster 2024-03-11T21:31:17.278871+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:31:17.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:17.971 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:18.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:17 smithi203 bash[23654]: audit 2024-03-11T21:31:16.947368+0000 mon.c (mon.2) 492 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:18.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:17 smithi203 bash[23654]: cluster 2024-03-11T21:31:17.272130+0000 mon.a (mon.0) 1790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:18.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:17 smithi203 bash[23654]: audit 2024-03-11T21:31:17.274165+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-19"}]': finished 2024-03-11T21:31:18.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:17 smithi203 bash[23654]: audit 2024-03-11T21:31:17.274245+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:18.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:17 smithi203 bash[23654]: cluster 2024-03-11T21:31:17.278871+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:31:18.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.099 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:18.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:18.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:18.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:18.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:18.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:18.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:18.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:18.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:18.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:18.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[21512]: cluster 2024-03-11T21:31:17.290341+0000 mgr.y (mgr.14550) 337 : cluster 0 pgmap v402: 364 pgs: 72 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:18.850 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[21512]: audit 2024-03-11T21:31:17.948591+0000 mon.c (mon.2) 493 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[21512]: cluster 2024-03-11T21:31:18.283584+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[21512]: audit 2024-03-11T21:31:18.288254+0000 mon.a (mon.0) 1795 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[21512]: audit 2024-03-11T21:31:18.297307+0000 mon.a (mon.0) 1796 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[29284]: cluster 2024-03-11T21:31:17.290341+0000 mgr.y (mgr.14550) 337 : cluster 0 pgmap v402: 364 pgs: 72 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[29284]: audit 2024-03-11T21:31:17.948591+0000 mon.c (mon.2) 493 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[29284]: cluster 2024-03-11T21:31:18.283584+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[29284]: audit 2024-03-11T21:31:18.288254+0000 mon.a (mon.0) 1795 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:18.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:18 smithi143 bash[29284]: audit 2024-03-11T21:31:18.297307+0000 mon.a (mon.0) 1796 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:18.864 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:18.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:19.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[23654]: cluster 2024-03-11T21:31:17.290341+0000 mgr.y (mgr.14550) 337 : cluster 0 pgmap v402: 364 pgs: 72 unknown, 1 active+clean+snaptrim, 33 active+undersized+degraded, 79 active+undersized, 179 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 183/1212 objects degraded (15.099%) 2024-03-11T21:31:19.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[23654]: audit 2024-03-11T21:31:17.948591+0000 mon.c (mon.2) 493 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:19.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[23654]: cluster 2024-03-11T21:31:18.283584+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-03-11T21:31:19.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[23654]: audit 2024-03-11T21:31:18.288254+0000 mon.a (mon.0) 1795 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:19.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:18 smithi203 bash[23654]: audit 2024-03-11T21:31:18.297307+0000 mon.a (mon.0) 1796 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:19.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:19.265 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:19.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:19.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:19.524 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:19.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.655 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:19.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:19.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:18.450278+0000 mgr.y (mgr.14550) 338 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:18.950277+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:19.282048+0000 mon.a (mon.0) 1797 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:19.282232+0000 mon.a (mon.0) 1798 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: cluster 2024-03-11T21:31:19.293187+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:19.297274+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:19.297367+0000 mon.a (mon.0) 1800 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:19 smithi203 bash[23654]: audit 2024-03-11T21:31:19.307330+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:19.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:19.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:18.450278+0000 mgr.y (mgr.14550) 338 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:19.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:18.950277+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:19.282048+0000 mon.a (mon.0) 1797 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:19.282232+0000 mon.a (mon.0) 1798 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: cluster 2024-03-11T21:31:19.293187+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:19.297274+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:19.297367+0000 mon.a (mon.0) 1800 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[21512]: audit 2024-03-11T21:31:19.307330+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:18.450278+0000 mgr.y (mgr.14550) 338 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:18.950277+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:19.282048+0000 mon.a (mon.0) 1797 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:19.282232+0000 mon.a (mon.0) 1798 : audit 1 from='client.? 172.21.15.143:0/608680973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi143-71604-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: cluster 2024-03-11T21:31:19.293187+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-03-11T21:31:19.899 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:19.297274+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:19.297367+0000 mon.a (mon.0) 1800 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]: dispatch 2024-03-11T21:31:19.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:19 smithi143 bash[29284]: audit 2024-03-11T21:31:19.307330+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:19.914 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:20.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.045 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:20.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:20.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:20.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:20.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:20.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:20.436 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:20.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:20.565 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:20.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:20.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:20.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:20.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: cluster 2024-03-11T21:31:19.292173+0000 mgr.y (mgr.14550) 339 : cluster 0 pgmap v405: 388 pgs: 1 active+clean+snaptrim, 3 creating+peering, 61 unknown, 34 active+undersized+degraded, 87 active+undersized, 202 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:20.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:19.951617+0000 mon.c (mon.2) 496 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.291521+0000 mon.a (mon.0) 1802 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.291606+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: cluster 2024-03-11T21:31:20.295805+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.339145+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.339618+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.340117+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.340355+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.340801+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.341031+0000 mon.a (mon.0) 1807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.408822+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[29284]: audit 2024-03-11T21:31:20.409487+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: cluster 2024-03-11T21:31:19.292173+0000 mgr.y (mgr.14550) 339 : cluster 0 pgmap v405: 388 pgs: 1 active+clean+snaptrim, 3 creating+peering, 61 unknown, 34 active+undersized+degraded, 87 active+undersized, 202 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:19.951617+0000 mon.c (mon.2) 496 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.291521+0000 mon.a (mon.0) 1802 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.291606+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: cluster 2024-03-11T21:31:20.295805+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.339145+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.339618+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.340117+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.340355+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:20.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.340801+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.341031+0000 mon.a (mon.0) 1807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.408822+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:20.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:20 smithi143 bash[21512]: audit 2024-03-11T21:31:20.409487+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:20.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:21.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: cluster 2024-03-11T21:31:19.292173+0000 mgr.y (mgr.14550) 339 : cluster 0 pgmap v405: 388 pgs: 1 active+clean+snaptrim, 3 creating+peering, 61 unknown, 34 active+undersized+degraded, 87 active+undersized, 202 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:19.951617+0000 mon.c (mon.2) 496 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.291521+0000 mon.a (mon.0) 1802 : audit 1 from='client.? 172.21.15.143:0/2134772814' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi143-71594-32"}]': finished 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.291606+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: cluster 2024-03-11T21:31:20.295805+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.339145+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.339618+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.340117+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.340355+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.340801+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.341031+0000 mon.a (mon.0) 1807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:21.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.408822+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:21.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:20 smithi203 bash[23654]: audit 2024-03-11T21:31:20.409487+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:21.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.110 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:21.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.245 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:21.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:21.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:21.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:21.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:21.704 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:21.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:21.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:21.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:21.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:22.010 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:20.952878+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.000694+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.004710+0000 mon.c (mon.2) 502 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: cluster 2024-03-11T21:31:21.237141+0000 mon.a (mon.0) 1810 : cluster 3 Health check update: Degraded data redundancy: 185/1206 objects degraded (15.340%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: cluster 2024-03-11T21:31:21.237171+0000 mon.a (mon.0) 1811 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.241171+0000 mon.a (mon.0) 1812 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:22.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.241248+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:22.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: cluster 2024-03-11T21:31:21.246221+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-03-11T21:31:22.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.251585+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.252517+0000 mon.c (mon.2) 504 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.260583+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:21 smithi203 bash[23654]: audit 2024-03-11T21:31:21.260774+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T21:31:22.080 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[149147]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:20.952878+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.000694+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.004710+0000 mon.c (mon.2) 502 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: cluster 2024-03-11T21:31:21.237141+0000 mon.a (mon.0) 1810 : cluster 3 Health check update: Degraded data redundancy: 185/1206 objects degraded (15.340%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: cluster 2024-03-11T21:31:21.237171+0000 mon.a (mon.0) 1811 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.241171+0000 mon.a (mon.0) 1812 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.241248+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: cluster 2024-03-11T21:31:21.246221+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.251585+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.252517+0000 mon.c (mon.2) 504 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.260583+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[29284]: audit 2024-03-11T21:31:21.260774+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:20.952878+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.000694+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.004710+0000 mon.c (mon.2) 502 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: cluster 2024-03-11T21:31:21.237141+0000 mon.a (mon.0) 1810 : cluster 3 Health check update: Degraded data redundancy: 185/1206 objects degraded (15.340%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: cluster 2024-03-11T21:31:21.237171+0000 mon.a (mon.0) 1811 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.241171+0000 mon.a (mon.0) 1812 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi143-71594-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.241248+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: cluster 2024-03-11T21:31:21.246221+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-03-11T21:31:22.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.251585+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.252517+0000 mon.c (mon.2) 504 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.260583+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:22.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:21 smithi143 bash[21512]: audit 2024-03-11T21:31:21.260774+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:22.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:22.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:22.284 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:22.377 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:22.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:22.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:22.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:22.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:22.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.671 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:22.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:22.795 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:22.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 -- 172.21.15.143:0/3672015816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50102850 msgr2=0x7f8d5010ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 --2- 172.21.15.143:0/3672015816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50102850 0x7f8d5010ecc0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f8d48009f90 tx=0x7f8d4800b330 comp rx=0 tx=0).stop 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 -- 172.21.15.143:0/3672015816 shutdown_connections 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 --2- 172.21.15.143:0/3672015816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50102850 0x7f8d5010ecc0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 --2- 172.21.15.143:0/3672015816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50107ea0 0x7f8d50108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 --2- 172.21.15.143:0/3672015816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50101e90 0x7f8d50102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 -- 172.21.15.143:0/3672015816 >> 172.21.15.143:0/3672015816 conn(0x7f8d50078f40 msgr2=0x7f8d50079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 -- 172.21.15.143:0/3672015816 shutdown_connections 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.036+0000 7f8d57196700 1 -- 172.21.15.143:0/3672015816 wait complete. 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 librados: starting msgr at 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 librados: starting objecter 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 Processor -- start 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- start start 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 librados: setting wanted keys 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 librados: calling monclient init 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50102850 0x7f8d50074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50107ea0 0x7f8d50074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50111620 con 0x7f8d50107ea0 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d501114a0 con 0x7f8d50102850 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d501117a0 con 0x7f8d50101e90 2024-03-11T21:31:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:23.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48794/0 (socket says 172.21.15.143:48794) 2024-03-11T21:31:23.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d55733700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50107ea0 0x7f8d50074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:23.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 -- 172.21.15.143:0/3049121882 learned_addr learned my addr 172.21.15.143:0/3049121882 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:23.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4ffff700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50102850 0x7f8d50074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:23.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 -- 172.21.15.143:0/3049121882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50102850 msgr2=0x7f8d50074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:23.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50102850 0x7f8d50074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 -- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50107ea0 msgr2=0x7f8d50074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50107ea0 0x7f8d50074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 -- 172.21.15.143:0/3049121882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d501a5f60 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d54f32700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d950 tx=0x7f8d4400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 10 client.?.objecter ms_handle_connect 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44016070 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8d44009dc0 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400f3c0 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d501a6250 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 10 client.41231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d501a6740 con 0x7f8d50101e90 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 librados: init done 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5004faa0 con 0x7f8d50101e90 2024-03-11T21:31:23.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8d44019070 con 0x7f8d50101e90 2024-03-11T21:31:23.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.040+0000 7f8d4dffb700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 0x7f8d40071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:23.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(281..281 src has 87..281) v4 ==== 10138+0+0 (secure 0 0 0) 0x7f8d4408e740 con 0x7f8d50101e90 2024-03-11T21:31:23.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 10 client.41231.objecter ms_dispatch 0x7f8d50001550 osd_map(281..281 src has 87..281) v4 2024-03-11T21:31:23.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 3 client.41231.objecter handle_osd_map got epochs [281,281] > 0 2024-03-11T21:31:23.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 3 client.41231.objecter handle_osd_map decoding full epoch 281 2024-03-11T21:31:23.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4ffff700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 0x7f8d40071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:23.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 20 client.41231.objecter dump_active .. 0 homeless 2024-03-11T21:31:23.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4ffff700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 0x7f8d40071ce0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f8d50079620 tx=0x7f8d3c009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:23.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.044+0000 7f8d4dffb700 10 client.41231.objecter ms_handle_connect 0x7f8d4006f820 2024-03-11T21:31:23.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:23.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.048+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8d4405b2b0 con 0x7f8d50101e90 2024-03-11T21:31:23.050 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:23.153 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:23.173 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:23.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:23.284+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f8d501a69a0 con 0x7f8d50101e90 2024-03-11T21:31:23.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:23.298 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:23.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: cluster 2024-03-11T21:31:21.292986+0000 mgr.y (mgr.14550) 340 : cluster 0 pgmap v408: 324 pgs: 1 active+clean+snaptrim, 32 unknown, 34 active+undersized+degraded, 75 active+undersized, 182 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:23.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: audit 2024-03-11T21:31:21.954685+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:23.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: audit 2024-03-11T21:31:22.256187+0000 mon.a (mon.0) 1817 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:23.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: audit 2024-03-11T21:31:22.267725+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: cluster 2024-03-11T21:31:22.725846+0000 mon.a (mon.0) 1818 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-03-11T21:31:23.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:23 smithi203 bash[23654]: audit 2024-03-11T21:31:22.726812+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.425 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:23.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:23.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:23.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: cluster 2024-03-11T21:31:21.292986+0000 mgr.y (mgr.14550) 340 : cluster 0 pgmap v408: 324 pgs: 1 active+clean+snaptrim, 32 unknown, 34 active+undersized+degraded, 75 active+undersized, 182 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: audit 2024-03-11T21:31:21.954685+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: audit 2024-03-11T21:31:22.256187+0000 mon.a (mon.0) 1817 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: audit 2024-03-11T21:31:22.267725+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: cluster 2024-03-11T21:31:22.725846+0000 mon.a (mon.0) 1818 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-03-11T21:31:23.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[29284]: audit 2024-03-11T21:31:22.726812+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: cluster 2024-03-11T21:31:21.292986+0000 mgr.y (mgr.14550) 340 : cluster 0 pgmap v408: 324 pgs: 1 active+clean+snaptrim, 32 unknown, 34 active+undersized+degraded, 75 active+undersized, 182 active+clean; 586 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 185/1206 objects degraded (15.340%) 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: audit 2024-03-11T21:31:21.954685+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: audit 2024-03-11T21:31:22.256187+0000 mon.a (mon.0) 1817 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: audit 2024-03-11T21:31:22.267725+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: cluster 2024-03-11T21:31:22.725846+0000 mon.a (mon.0) 1818 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-03-11T21:31:23.764 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:23 smithi143 bash[21512]: audit 2024-03-11T21:31:22.726812+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]: dispatch 2024-03-11T21:31:23.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:23.819 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:23.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.948 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:24.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:24.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:24.193 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:22.955934+0000 mon.c (mon.2) 507 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: cluster 2024-03-11T21:31:23.268876+0000 mon.a (mon.0) 1820 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.270551+0000 mon.a (mon.0) 1821 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.270674+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]': finished 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: cluster 2024-03-11T21:31:23.274357+0000 mon.a (mon.0) 1823 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.275583+0000 mon.c (mon.2) 508 : audit 1 from='client.? 172.21.15.143:0/3379144069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.291882+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.143:0/3049121882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: cluster 2024-03-11T21:31:23.293652+0000 mgr.y (mgr.14550) 341 : cluster 0 pgmap v411: 332 pgs: 1 active+clean+snaptrim, 45 unknown, 34 active+undersized+degraded, 69 active+undersized, 183 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.296582+0000 mon.a (mon.0) 1824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.194 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.296862+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.195 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.388075+0000 mon.c (mon.2) 510 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.195 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.388453+0000 mon.a (mon.0) 1826 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.195 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:24 smithi203 bash[23654]: audit 2024-03-11T21:31:23.957050+0000 mon.c (mon.2) 511 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.209 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:24.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.276+0000 7f8d4dffb700 1 -- 172.21.15.143:0/3049121882 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v283) v1 ==== 113+0+0 (secure 0 0 0) 0x7f8d4405ed00 con 0x7f8d50101e90 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 10 librados: watch_flush enter 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 10 librados: watch_flush exit 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 20 client.41231.objecter shutdown clearing up homeless session... 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 10 client.41231.objecter successfully canceled tick 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 msgr2=0x7f8d40071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 0x7f8d40071ce0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f8d50079620 tx=0x7f8d3c009310 comp rx=0 tx=0).stop 2024-03-11T21:31:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 msgr2=0x7f8d500740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:24.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d950 tx=0x7f8d4400dc60 comp rx=0 tx=0).stop 2024-03-11T21:31:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 shutdown_connections 2024-03-11T21:31:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d4006f820 0x7f8d40071ce0 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d50107ea0 0x7f8d50074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d50101e90 0x7f8d500740e0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 --2- 172.21.15.143:0/3049121882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d50102850 0x7f8d50074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 >> 172.21.15.143:0/3049121882 conn(0x7f8d50078f40 msgr2=0x7f8d50105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 shutdown_connections 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 -- 172.21.15.143:0/3049121882 wait complete. 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:24.288+0000 7f8d57196700 1 librados: shutdown 2024-03-11T21:31:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:31:24.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.331 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:24.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:24.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:24.593 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:24.601 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:22.955934+0000 mon.c (mon.2) 507 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: cluster 2024-03-11T21:31:23.268876+0000 mon.a (mon.0) 1820 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.270551+0000 mon.a (mon.0) 1821 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.270674+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]': finished 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: cluster 2024-03-11T21:31:23.274357+0000 mon.a (mon.0) 1823 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.275583+0000 mon.c (mon.2) 508 : audit 1 from='client.? 172.21.15.143:0/3379144069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.291882+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.143:0/3049121882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: cluster 2024-03-11T21:31:23.293652+0000 mgr.y (mgr.14550) 341 : cluster 0 pgmap v411: 332 pgs: 1 active+clean+snaptrim, 45 unknown, 34 active+undersized+degraded, 69 active+undersized, 183 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.296582+0000 mon.a (mon.0) 1824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.296862+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.388075+0000 mon.c (mon.2) 510 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.388453+0000 mon.a (mon.0) 1826 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[21512]: audit 2024-03-11T21:31:23.957050+0000 mon.c (mon.2) 511 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:22.955934+0000 mon.c (mon.2) 507 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: cluster 2024-03-11T21:31:23.268876+0000 mon.a (mon.0) 1820 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.270551+0000 mon.a (mon.0) 1821 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi143-71594-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.270674+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-21", "mode": "writeback"}]': finished 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: cluster 2024-03-11T21:31:23.274357+0000 mon.a (mon.0) 1823 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.275583+0000 mon.c (mon.2) 508 : audit 1 from='client.? 172.21.15.143:0/3379144069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.291882+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.143:0/3049121882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: cluster 2024-03-11T21:31:23.293652+0000 mgr.y (mgr.14550) 341 : cluster 0 pgmap v411: 332 pgs: 1 active+clean+snaptrim, 45 unknown, 34 active+undersized+degraded, 69 active+undersized, 183 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.296582+0000 mon.a (mon.0) 1824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.296862+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.388075+0000 mon.c (mon.2) 510 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.388453+0000 mon.a (mon.0) 1826 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:24.603 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:24 smithi143 bash[29284]: audit 2024-03-11T21:31:23.957050+0000 mon.c (mon.2) 511 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:24.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:24.721 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:24.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.849 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:24.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:24.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:25.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:25.109 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:25.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:25.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:25.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.277710+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.277780+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.277846+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: cluster 2024-03-11T21:31:24.281056+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.282111+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.283636+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[29284]: audit 2024-03-11T21:31:24.958833+0000 mon.c (mon.2) 513 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:25.353 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.277710+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.277780+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.277846+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: cluster 2024-03-11T21:31:24.281056+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.282111+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.283636+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:25 smithi143 bash[21512]: audit 2024-03-11T21:31:24.958833+0000 mon.c (mon.2) 513 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:25.365 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:25.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:25.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:25.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.277710+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.277780+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.277846+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: cluster 2024-03-11T21:31:24.281056+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.282111+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.283636+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]: dispatch 2024-03-11T21:31:25.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:25 smithi203 bash[23654]: audit 2024-03-11T21:31:24.958833+0000 mon.c (mon.2) 513 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:25.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:25.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:25.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:25.751 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:25.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:25.880 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:25.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:26.013 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:26.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.148 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:26.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:26.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:26.296 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:26.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:26.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:26.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:26.548 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:26.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:26.675 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:26.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:26.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:26.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:26.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: cluster 2024-03-11T21:31:25.279474+0000 mon.a (mon.0) 1832 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.282064+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: cluster 2024-03-11T21:31:25.294797+0000 mon.a (mon.0) 1834 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: cluster 2024-03-11T21:31:25.295788+0000 mgr.y (mgr.14550) 342 : cluster 0 pgmap v414: 356 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 69 active+undersized, 188 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.296235+0000 mon.c (mon.2) 514 : audit 1 from='client.? 172.21.15.143:0/2558581496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.298164+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.299166+0000 mon.a (mon.0) 1835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.301934+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:25.960286+0000 mon.c (mon.2) 516 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: cluster 2024-03-11T21:31:26.238865+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: Degraded data redundancy: 185/1188 objects degraded (15.572%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: cluster 2024-03-11T21:31:26.238919+0000 mon.a (mon.0) 1838 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:27.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:26 smithi203 bash[23654]: audit 2024-03-11T21:31:26.291818+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:27.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: cluster 2024-03-11T21:31:25.279474+0000 mon.a (mon.0) 1832 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.282064+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: cluster 2024-03-11T21:31:25.294797+0000 mon.a (mon.0) 1834 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: cluster 2024-03-11T21:31:25.295788+0000 mgr.y (mgr.14550) 342 : cluster 0 pgmap v414: 356 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 69 active+undersized, 188 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.296235+0000 mon.c (mon.2) 514 : audit 1 from='client.? 172.21.15.143:0/2558581496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.298164+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.299166+0000 mon.a (mon.0) 1835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.301934+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:25.960286+0000 mon.c (mon.2) 516 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: cluster 2024-03-11T21:31:26.238865+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: Degraded data redundancy: 185/1188 objects degraded (15.572%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: cluster 2024-03-11T21:31:26.238919+0000 mon.a (mon.0) 1838 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[21512]: audit 2024-03-11T21:31:26.291818+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: cluster 2024-03-11T21:31:25.279474+0000 mon.a (mon.0) 1832 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.282064+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-21"}]': finished 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: cluster 2024-03-11T21:31:25.294797+0000 mon.a (mon.0) 1834 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: cluster 2024-03-11T21:31:25.295788+0000 mgr.y (mgr.14550) 342 : cluster 0 pgmap v414: 356 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 69 active+undersized, 188 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.296235+0000 mon.c (mon.2) 514 : audit 1 from='client.? 172.21.15.143:0/2558581496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.298164+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.299166+0000 mon.a (mon.0) 1835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.301934+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:25.960286+0000 mon.c (mon.2) 516 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:27.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: cluster 2024-03-11T21:31:26.238865+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: Degraded data redundancy: 185/1188 objects degraded (15.572%), 34 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:27.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: cluster 2024-03-11T21:31:26.238919+0000 mon.a (mon.0) 1838 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:27.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:26 smithi143 bash[29284]: audit 2024-03-11T21:31:26.291818+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:27.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:27.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:27.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:27.325 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:27.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:27.456 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:27.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:27.586 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:27.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:27.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:26.292090+0000 mon.a (mon.0) 1840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:26.300357+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: cluster 2024-03-11T21:31:26.300359+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:26.314795+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:26.961573+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.297506+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: cluster 2024-03-11T21:31:27.307816+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-03-11T21:31:27.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.312039+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.313575+0000 mon.c (mon.2) 520 : audit 1 from='client.? 172.21.15.143:0/1782337969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.328273+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.330368+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.343499+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.344030+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.345267+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.345911+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.346534+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[21512]: audit 2024-03-11T21:31:27.346855+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:26.292090+0000 mon.a (mon.0) 1840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:26.300357+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: cluster 2024-03-11T21:31:26.300359+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:26.314795+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:26.961573+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.297506+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: cluster 2024-03-11T21:31:27.307816+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.312039+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.313575+0000 mon.c (mon.2) 520 : audit 1 from='client.? 172.21.15.143:0/1782337969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.328273+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.330368+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.343499+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.344030+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.345267+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.345911+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:27.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.346534+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:27.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:27 smithi143 bash[29284]: audit 2024-03-11T21:31:27.346855+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:27.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:27.974 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:26.292090+0000 mon.a (mon.0) 1840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:26.300357+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.143:0/1445291562' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: cluster 2024-03-11T21:31:26.300359+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-03-11T21:31:28.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:26.314795+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:26.961573+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.297506+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi143-71594-33"}]': finished 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: cluster 2024-03-11T21:31:27.307816+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.312039+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.313575+0000 mon.c (mon.2) 520 : audit 1 from='client.? 172.21.15.143:0/1782337969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.328273+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.330368+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.343499+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.344030+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.345267+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.345911+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.346534+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:28.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:27 smithi203 bash[23654]: audit 2024-03-11T21:31:27.346855+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:28.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:28.105 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:28.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:28.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:28.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:28.358 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:28.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:28.484 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:28.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:28.613 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:28.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:28.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:28.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:28.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:28.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: cluster 2024-03-11T21:31:27.297142+0000 mgr.y (mgr.14550) 343 : cluster 0 pgmap v416: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 62 active+undersized, 163 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:27.963309+0000 mon.c (mon.2) 524 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:28.301070+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:28.301324+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:28.301468+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:28.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: cluster 2024-03-11T21:31:28.308942+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:28.313035+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[21512]: audit 2024-03-11T21:31:28.323189+0000 mon.a (mon.0) 1854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: cluster 2024-03-11T21:31:27.297142+0000 mgr.y (mgr.14550) 343 : cluster 0 pgmap v416: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 62 active+undersized, 163 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:27.963309+0000 mon.c (mon.2) 524 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:28.301070+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:28.301324+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:28.301468+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: cluster 2024-03-11T21:31:28.308942+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:28.313035+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:28 smithi143 bash[29284]: audit 2024-03-11T21:31:28.323189+0000 mon.a (mon.0) 1854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:31:28.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: cluster 2024-03-11T21:31:27.297142+0000 mgr.y (mgr.14550) 343 : cluster 0 pgmap v416: 324 pgs: 1 active+clean+snaptrim, 6 creating+peering, 58 unknown, 34 active+undersized+degraded, 62 active+undersized, 163 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 185/1188 objects degraded (15.572%) 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:27.963309+0000 mon.c (mon.2) 524 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:28.301070+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:28.301324+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:28.301468+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi143-71594-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:28.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: cluster 2024-03-11T21:31:28.308942+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-03-11T21:31:28.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:28.313035+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:28.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:28 smithi203 bash[23654]: audit 2024-03-11T21:31:28.323189+0000 mon.a (mon.0) 1854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:29.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 -- 172.21.15.143:0/121265158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928106e00 msgr2=0x7f09281071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 --2- 172.21.15.143:0/121265158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928106e00 0x7f09281071e0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f09180099c0 tx=0x7f0918009cd0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 -- 172.21.15.143:0/121265158 shutdown_connections 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 --2- 172.21.15.143:0/121265158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928106e00 0x7f09281071e0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 --2- 172.21.15.143:0/121265158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0928100cf0 0x7f092810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 --2- 172.21.15.143:0/121265158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0928100330 0x7f09281007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 -- 172.21.15.143:0/121265158 >> 172.21.15.143:0/121265158 conn(0x7f09280772e0 msgr2=0x7f09280fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:29.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 -- 172.21.15.143:0/121265158 shutdown_connections 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.096+0000 7f092d91f700 1 -- 172.21.15.143:0/121265158 wait complete. 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 librados: starting msgr at 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 librados: starting objecter 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 Processor -- start 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- start start 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 librados: setting wanted keys 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 librados: calling monclient init 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 0x7f092819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0928100cf0 0x7f092819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0928106e00 0x7f0928197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09281116d0 con 0x7f0928100330 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0928111550 con 0x7f0928100cf0 2024-03-11T21:31:29.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0928111850 con 0x7f0928106e00 2024-03-11T21:31:29.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f0926ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 0x7f092819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f0926ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 0x7f092819d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33644/0 (socket says 172.21.15.143:33644) 2024-03-11T21:31:29.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f0926ffd700 1 -- 172.21.15.143:0/2680037437 learned_addr learned my addr 172.21.15.143:0/2680037437 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:29.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0928106e00 0x7f0928197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0928100cf0 0x7f092819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 -- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0928100cf0 msgr2=0x7f092819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0928100cf0 0x7f092819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 -- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 msgr2=0x7f092819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 0x7f092819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928198200 con 0x7f0928106e00 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0928100330 0x7f092819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0928106e00 0x7f0928197aa0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f0920007f10 tx=0x7f092000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 10 client.?.objecter ms_handle_connect 0x7f0928106e00 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000ddd0 con 0x7f0928106e00 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0920009e20 con 0x7f0928106e00 2024-03-11T21:31:29.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09281984f0 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920015400 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 10 client.41258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09281a47b0 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 librados: init done 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f092804faa0 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.100+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0920004ad0 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f091006f820 0x7f0910071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(287..287 src has 87..287) v4 ==== 11174+0+0 (secure 0 0 0) 0x7f092008f600 con 0x7f0928106e00 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 osd_map(287..287 src has 87..287) v4 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 3 client.41258.objecter handle_osd_map got epochs [287,287] > 0 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 3 client.41258.objecter handle_osd_map decoding full epoch 287 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f091006f820 0x7f0910071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 20 client.41258.objecter dump_active .. 0 homeless 2024-03-11T21:31:29.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v287) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0920010070 con 0x7f0928106e00 2024-03-11T21:31:29.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 10 client.41258.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 1 2024-03-11T21:31:29.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f091006f820 0x7f0910071ce0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f09180037a0 tx=0x7f0918019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f091006f820 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f09280629f0 0x7f0928062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 1 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 0 1 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 4 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 10 client.41258.objecter _send_command 1 2024-03-11T21:31:29.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09281a7c30 con 0x7f09280629f0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f09280629f0 0x7f0928062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f09280629f0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f09280629f0 0x7f0928062dd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09281a7c30 con 0x7f09280629f0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 10 client.41258.objecter _finish_command 1 = system:0 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 4 2024-03-11T21:31:29.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.104+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 0 1 2024-03-11T21:31:29.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 2 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 2 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928198fd0 osd=0 3 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 0 2 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 4 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _send_command 2 2024-03-11T21:31:29.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0928062470 con 0x7f09280629f0 2024-03-11T21:31:29.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0928062470 con 0x7f09280629f0 2024-03-11T21:31:29.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 10 client.41258.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928198fd0 osd=0 4 2024-03-11T21:31:29.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 0 2 2024-03-11T21:31:29.103 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:31:29.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 3 2024-03-11T21:31:29.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter _get_session osd=-1 returning homeless 2024-03-11T21:31:29.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:29.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=288}) v3 -- 0x7f09281093b0 con 0x7f0928106e00 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09281096f0 con 0x7f0928106e00 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_get_version_reply(handle=1 version=287) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0920023070 con 0x7f0928106e00 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f09257fa700 10 client.41258.objecter _check_command_map_dne tid 3 current 287 map_dne_bound 287 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f09257fa700 10 client.41258.objecter _finish_command 3 = osd:6 osd down 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f09257fa700 15 client.41258.objecter _session_command_op_remove -1 3 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 4 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f092802cbc0 0x7f092802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 4 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 2 4 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 4 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 10 client.41258.objecter _send_command 4 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f092802d4e0 con 0x7f092802cbc0 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.124+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f092802cbc0 0x7f092802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f092802cbc0 0x7f092802cfa0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f092802cbc0 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f092802d4e0 con 0x7f092802cbc0 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 4: 0 ) v1 2024-03-11T21:31:29.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 10 client.41258.objecter _finish_command 4 = system:0 2024-03-11T21:31:29.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 4 2024-03-11T21:31:29.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.128+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 2 4 2024-03-11T21:31:29.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 10 client.41258.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 5 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 5 2024-03-11T21:31:29.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928109bb0 osd=2 3 2024-03-11T21:31:29.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 2 5 2024-03-11T21:31:29.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 4 2024-03-11T21:31:29.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 10 client.41258.objecter _send_command 5 2024-03-11T21:31:29.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f092802d940 con 0x7f092802cbc0 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f092802d940 con 0x7f092802cbc0 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 10 client.41258.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928109bb0 osd=2 4 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 2 5 2024-03-11T21:31:29.127 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 10 client.41258.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 6 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0928101ac0 0x7f09281abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 6 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 3 6 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 4 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 10 client.41258.objecter _send_command 6 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09281ac340 con 0x7f0928101ac0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0928101ac0 0x7f09281abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0928101ac0 0x7f09281abe00 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.148+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f0928101ac0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.152+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09281ac340 con 0x7f0928101ac0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.152+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 6: 0 ) v1 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.152+0000 7f092c91d700 10 client.41258.objecter _finish_command 6 = system:0 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.152+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 4 2024-03-11T21:31:29.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.152+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 3 6 2024-03-11T21:31:29.133 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:29.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 10 client.41258.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 7 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 7 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928101440 osd=3 3 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 3 7 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 4 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 10 client.41258.objecter _send_command 7 2024-03-11T21:31:29.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09281ac520 con 0x7f0928101ac0 2024-03-11T21:31:29.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f09281ac520 con 0x7f0928101ac0 2024-03-11T21:31:29.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092c91d700 10 client.41258.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928101440 osd=3 4 2024-03-11T21:31:29.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 3 7 2024-03-11T21:31:29.148 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:31:29.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 10 client.41258.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 8 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f09281a7280 0x7f09281a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 8 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 4 8 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 4 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 10 client.41258.objecter _send_command 8 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09281091f0 con 0x7f09281a7280 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.168+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f09281a7280 0x7f09281a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f09281a7280 2024-03-11T21:31:29.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f09281a7280 0x7f09281a76e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09281096f0 con 0x7f09281a7280 2024-03-11T21:31:29.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 8: 0 ) v1 2024-03-11T21:31:29.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 10 client.41258.objecter _finish_command 8 = system:0 2024-03-11T21:31:29.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 4 2024-03-11T21:31:29.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.172+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 4 8 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 10 client.41258.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 9 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 9 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928109080 osd=4 3 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 4 9 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 4 2024-03-11T21:31:29.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 10 client.41258.objecter _send_command 9 2024-03-11T21:31:29.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.188+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09281a6d00 con 0x7f09281a7280 2024-03-11T21:31:29.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f09281a6d00 con 0x7f09281a7280 2024-03-11T21:31:29.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 10 client.41258.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928109080 osd=4 4 2024-03-11T21:31:29.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 4 9 2024-03-11T21:31:29.170 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:31:29.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 10 client.41258.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 10 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f09281a6e70 0x7f09281b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 10 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 5 10 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 4 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 10 client.41258.objecter _send_command 10 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0928102d80 con 0x7f09281a6e70 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f09281a6e70 0x7f09281b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f09281a6e70 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f0926ffd700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f09281a6e70 0x7f09281b5cc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0928102d80 con 0x7f09281a6e70 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 10: 0 ) v1 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 10 client.41258.objecter _finish_command 10 = system:0 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 4 2024-03-11T21:31:29.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.192+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 5 10 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 10 client.41258.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 11 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 11 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f09281a6d00 osd=5 3 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 5 11 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 4 2024-03-11T21:31:29.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 10 client.41258.objecter _send_command 11 2024-03-11T21:31:29.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.212+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0928102d80 con 0x7f09281a6e70 2024-03-11T21:31:29.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0928102d80 con 0x7f09281a6e70 2024-03-11T21:31:29.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 10 client.41258.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f09281a6d00 osd=5 4 2024-03-11T21:31:29.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 5 11 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 10 client.41258.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 12 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f09281b7ee0 0x7f09281b8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 12 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 6 12 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 4 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 10 client.41258.objecter _send_command 12 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09281bbd60 con 0x7f09281b7ee0 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f09281b7ee0 0x7f09281b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f09267fc700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f09281b7ee0 0x7f09281b8300 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.216+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f09281b7ee0 2024-03-11T21:31:29.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.220+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09281bbd60 con 0x7f09281b7ee0 2024-03-11T21:31:29.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.220+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 12: 0 ) v1 2024-03-11T21:31:29.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.220+0000 7f092c91d700 10 client.41258.objecter _finish_command 12 = system:0 2024-03-11T21:31:29.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.220+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 4 2024-03-11T21:31:29.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.220+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 6 12 2024-03-11T21:31:29.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 10 client.41258.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 13 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 13 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f0928102d80 osd=6 3 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 6 13 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 4 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 10 client.41258.objecter _send_command 13 2024-03-11T21:31:29.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.240+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09281b6200 con 0x7f09281b7ee0 2024-03-11T21:31:29.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f09281b6200 con 0x7f09281b7ee0 2024-03-11T21:31:29.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 10 client.41258.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f0928102d80 osd=6 4 2024-03-11T21:31:29.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 6 13 2024-03-11T21:31:29.222 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:31:29.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 10 client.41258.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 14 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f09281b6920 0x7f09281b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 14 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 7 14 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 4 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 10 client.41258.objecter _send_command 14 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09281b7400 con 0x7f09281b6920 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f09281b6920 0x7f09281b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f092c91d700 10 client.41258.objecter ms_handle_connect 0x7f09281b6920 2024-03-11T21:31:29.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.244+0000 7f09277fe700 1 --2- 172.21.15.143:0/2680037437 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f09281b6920 0x7f09281b6d60 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:29.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.248+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09281b7400 con 0x7f09281b6920 2024-03-11T21:31:29.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.248+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 14: 0 ) v1 2024-03-11T21:31:29.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.248+0000 7f092c91d700 10 client.41258.objecter _finish_command 14 = system:0 2024-03-11T21:31:29.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.248+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 4 2024-03-11T21:31:29.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.248+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 7 14 2024-03-11T21:31:29.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 10 client.41258.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign -1 15 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 20 client.41258.objecter _get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 15 client.41258.objecter _session_command_op_remove -1 15 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 20 client.41258.objecter get_session s=0x7f09281b6200 osd=7 3 2024-03-11T21:31:29.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 15 client.41258.objecter _session_command_op_assign 7 15 2024-03-11T21:31:29.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 4 2024-03-11T21:31:29.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 10 client.41258.objecter _send_command 15 2024-03-11T21:31:29.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.264+0000 7f092d91f700 1 -- 172.21.15.143:0/2680037437 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09281b7400 con 0x7f09281b6920 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.268+0000 7f092c91d700 1 -- 172.21.15.143:0/2680037437 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f09281b7400 con 0x7f09281b6920 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.268+0000 7f092c91d700 10 client.41258.objecter ms_dispatch 0x7f0928001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.268+0000 7f092c91d700 10 client.41258.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.268+0000 7f092c91d700 20 client.41258.objecter put_session s=0x7f09281b6200 osd=7 4 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:29.268+0000 7f092c91d700 15 client.41258.objecter _session_command_op_remove 7 15 2024-03-11T21:31:29.248 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:29.249 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:29.262 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:29.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:29.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:29.488 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:29.509 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:29.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:29.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:29.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:29.768 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:29.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:29.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:29.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:28.460062+0000 mgr.y (mgr.14550) 344 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:29.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:28.964793+0000 mon.c (mon.2) 526 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:29.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:29.107121+0000 mon.c (mon.2) 527 : audit 0 from='client.? 172.21.15.143:0/2680037437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: cluster 2024-03-11T21:31:29.743625+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:29.747773+0000 mon.c (mon.2) 528 : audit 1 from='client.? 172.21.15.143:0/364027105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:29.748331+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:29.756318+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[29284]: audit 2024-03-11T21:31:29.756757+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:28.460062+0000 mgr.y (mgr.14550) 344 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:28.964793+0000 mon.c (mon.2) 526 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:29.107121+0000 mon.c (mon.2) 527 : audit 0 from='client.? 172.21.15.143:0/2680037437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: cluster 2024-03-11T21:31:29.743625+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-03-11T21:31:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:29.747773+0000 mon.c (mon.2) 528 : audit 1 from='client.? 172.21.15.143:0/364027105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:29.748331+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:29.756318+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:29.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:29 smithi143 bash[21512]: audit 2024-03-11T21:31:29.756757+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:30.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:28.460062+0000 mgr.y (mgr.14550) 344 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:28.964793+0000 mon.c (mon.2) 526 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:29.107121+0000 mon.c (mon.2) 527 : audit 0 from='client.? 172.21.15.143:0/2680037437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: cluster 2024-03-11T21:31:29.743625+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:29.747773+0000 mon.c (mon.2) 528 : audit 1 from='client.? 172.21.15.143:0/364027105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:29.748331+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:29.756318+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:29 smithi203 bash[23654]: audit 2024-03-11T21:31:29.756757+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:30.030 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:30.133 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.161 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:30.264 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:30.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:30.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.413 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:30.516 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:30.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.665 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:30.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:30.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:30.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: cluster 2024-03-11T21:31:29.298052+0000 mgr.y (mgr.14550) 345 : cluster 0 pgmap v419: 388 pgs: 1 active+clean+snaptrim, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 765 B/s wr, 2 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:30.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:29.965935+0000 mon.c (mon.2) 530 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:30.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:30.319068+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:30.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:30.319199+0000 mon.a (mon.0) 1859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:30.319282+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: cluster 2024-03-11T21:31:30.324490+0000 mon.a (mon.0) 1861 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:30.327055+0000 mon.c (mon.2) 531 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[29284]: audit 2024-03-11T21:31:30.330281+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: cluster 2024-03-11T21:31:29.298052+0000 mgr.y (mgr.14550) 345 : cluster 0 pgmap v419: 388 pgs: 1 active+clean+snaptrim, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 765 B/s wr, 2 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:29.965935+0000 mon.c (mon.2) 530 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:30.319068+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:30.319199+0000 mon.a (mon.0) 1859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:30.319282+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: cluster 2024-03-11T21:31:30.324490+0000 mon.a (mon.0) 1861 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:30.327055+0000 mon.c (mon.2) 531 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:30.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[21512]: audit 2024-03-11T21:31:30.330281+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:30.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:31.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:31.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:31.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: cluster 2024-03-11T21:31:29.298052+0000 mgr.y (mgr.14550) 345 : cluster 0 pgmap v419: 388 pgs: 1 active+clean+snaptrim, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 765 B/s wr, 2 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:29.965935+0000 mon.c (mon.2) 530 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:30.319068+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi143-71594-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:30.319199+0000 mon.a (mon.0) 1859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:30.319282+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: cluster 2024-03-11T21:31:30.324490+0000 mon.a (mon.0) 1861 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:30.327055+0000 mon.c (mon.2) 531 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:31.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:30 smithi203 bash[23654]: audit 2024-03-11T21:31:30.330281+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:31.185 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:30 smithi143 bash[150181]: debug 2024-03-11T21:31:30.912+0000 7f3a4f5cb700 1 -- 172.21.15.143:0/3256203056 <== mon.0 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556848441680 con 0x55684847c800 2024-03-11T21:31:31.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:31.295 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:31.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:31.423 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:31.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:31.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- 172.21.15.143:0/1377250632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0100d40 msgr2=0x7f65f01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- 172.21.15.143:0/1377250632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0100d40 0x7f65f01011c0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f65e800b0a0 tx=0x7f65e800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- 172.21.15.143:0/1377250632 shutdown_connections 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- 172.21.15.143:0/1377250632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0101700 0x7f65f010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- 172.21.15.143:0/1377250632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0106dc0 0x7f65f01071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- 172.21.15.143:0/1377250632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0100d40 0x7f65f01011c0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- 172.21.15.143:0/1377250632 >> 172.21.15.143:0/1377250632 conn(0x7f65f0076d80 msgr2=0x7f65f0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- 172.21.15.143:0/1377250632 shutdown_connections 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- 172.21.15.143:0/1377250632 wait complete. 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 librados: starting msgr at 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 librados: starting objecter 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 Processor -- start 2024-03-11T21:31:31.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- start start 2024-03-11T21:31:31.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 librados: setting wanted keys 2024-03-11T21:31:31.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 librados: calling monclient init 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 0x7f65f019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0101700 0x7f65f019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0110520 con 0x7f65f0106dc0 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01103a0 con 0x7f65f0101700 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.620+0000 7f65f78b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01106a0 con 0x7f65f0100d40 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5e52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:31.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 0x7f65f019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5e52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33694/0 (socket says 172.21.15.143:33694) 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5e52700 1 -- 172.21.15.143:0/3774517188 learned_addr learned my addr 172.21.15.143:0/3774517188 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f4e50700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0101700 0x7f65f019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 -- 172.21.15.143:0/3774517188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0101700 msgr2=0x7f65f019dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0101700 0x7f65f019dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 -- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 msgr2=0x7f65f01979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 -- 172.21.15.143:0/3774517188 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0198100 con 0x7f65f0100d40 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5e52700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f5651700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 0x7f65f019d7e0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00d750 tx=0x7f65ec00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 10 client.?.objecter ms_handle_connect 0x7f65f0100d40 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec016070 con 0x7f65f0100d40 2024-03-11T21:31:31.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f65ec004830 con 0x7f65f0100d40 2024-03-11T21:31:31.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f01983f0 con 0x7f65f0100d40 2024-03-11T21:31:31.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec005510 con 0x7f65f0100d40 2024-03-11T21:31:31.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f78b5700 10 client.41273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:31.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f01a46b0 con 0x7f65f0100d40 2024-03-11T21:31:31.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f78b5700 1 librados: init done 2024-03-11T21:31:31.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f004faa0 con 0x7f65f0100d40 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f65ec004220 con 0x7f65f0100d40 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 0x7f65dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(290..290 src has 87..290) v4 ==== 12446+0+0 (secure 0 0 0) 0x7f65ec08f1f0 con 0x7f65f0100d40 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 10 client.41273.objecter ms_dispatch 0x7f65f0001550 osd_map(290..290 src has 87..290) v4 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 3 client.41273.objecter handle_osd_map got epochs [290,290] > 0 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 3 client.41273.objecter handle_osd_map decoding full epoch 290 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f4e50700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 0x7f65dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 20 client.41273.objecter dump_active .. 0 homeless 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65e27fc700 10 client.41273.objecter ms_handle_connect 0x7f65dc06f820 2024-03-11T21:31:31.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.624+0000 7f65f4e50700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 0x7f65dc071ce0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f65f0077530 tx=0x7f65e80060a0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:31.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.628+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f65ec01f020 con 0x7f65f0100d40 2024-03-11T21:31:31.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:31.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:31.685 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[150181]: debug 2024-03-11T21:31:31.464+0000 7f3a587637c0 -1 Falling back to public interface 2024-03-11T21:31:31.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:31.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:31.856+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f65f0198ed0 con 0x7f65f0100d40 2024-03-11T21:31:31.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:31.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:32.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: cluster 2024-03-11T21:31:29.816221+0000 osd.4 (osd.4) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: cluster 2024-03-11T21:31:29.816666+0000 osd.4 (osd.4) 180 : cluster 0 purged_snaps scrub found 17 strays 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:30.967508+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.240713+0000 mon.a (mon.0) 1863 : cluster 3 Health check update: Degraded data redundancy: 187/1185 objects degraded (15.781%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.240779+0000 mon.a (mon.0) 1864 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:31.245378+0000 mon.a (mon.0) 1865 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:32.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.251139+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:31.259443+0000 mon.c (mon.2) 533 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:31.266606+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.143:0/4006562154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:31.267735+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[21512]: audit 2024-03-11T21:31:31.268103+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: cluster 2024-03-11T21:31:29.816221+0000 osd.4 (osd.4) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: cluster 2024-03-11T21:31:29.816666+0000 osd.4 (osd.4) 180 : cluster 0 purged_snaps scrub found 17 strays 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:30.967508+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:32.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.240713+0000 mon.a (mon.0) 1863 : cluster 3 Health check update: Degraded data redundancy: 187/1185 objects degraded (15.781%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.240779+0000 mon.a (mon.0) 1864 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:31.245378+0000 mon.a (mon.0) 1865 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.251139+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:31.259443+0000 mon.c (mon.2) 533 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:31.266606+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.143:0/4006562154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:31.267735+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.050 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:31 smithi143 bash[29284]: audit 2024-03-11T21:31:31.268103+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:32.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: cluster 2024-03-11T21:31:29.816221+0000 osd.4 (osd.4) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: cluster 2024-03-11T21:31:29.816666+0000 osd.4 (osd.4) 180 : cluster 0 purged_snaps scrub found 17 strays 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:30.967508+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.240713+0000 mon.a (mon.0) 1863 : cluster 3 Health check update: Degraded data redundancy: 187/1185 objects degraded (15.781%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.240779+0000 mon.a (mon.0) 1864 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:32.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:31.245378+0000 mon.a (mon.0) 1865 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:32.177 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.251139+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-03-11T21:31:32.177 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:31.259443+0000 mon.c (mon.2) 533 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.177 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:31.266606+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.143:0/4006562154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.177 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:31.267735+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]: dispatch 2024-03-11T21:31:32.177 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:31 smithi203 bash[23654]: audit 2024-03-11T21:31:31.268103+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:32.193 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:32.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.252+0000 7f65e27fc700 1 -- 172.21.15.143:0/3774517188 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v291) v1 ==== 97+0+0 (secure 0 0 0) 0x7f65ec05b3c0 con 0x7f65f0100d40 2024-03-11T21:31:32.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.260+0000 7f65f78b5700 10 librados: watch_flush enter 2024-03-11T21:31:32.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 10 librados: watch_flush exit 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 20 client.41273.objecter shutdown clearing up homeless session... 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 10 client.41273.objecter successfully canceled tick 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 msgr2=0x7f65dc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 0x7f65dc071ce0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f65f0077530 tx=0x7f65e80060a0 comp rx=0 tx=0).stop 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 msgr2=0x7f65f019d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:32.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.268+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 0x7f65f019d7e0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00d750 tx=0x7f65ec00da60 comp rx=0 tx=0).stop 2024-03-11T21:31:32.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 shutdown_connections 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f65dc06f820 0x7f65dc071ce0 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65f0106dc0 0x7f65f01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65f0100d40 0x7f65f019d7e0 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 --2- 172.21.15.143:0/3774517188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65f0101700 0x7f65f019dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 >> 172.21.15.143:0/3774517188 conn(0x7f65f0076d80 msgr2=0x7f65f0069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 shutdown_connections 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 -- 172.21.15.143:0/3774517188 wait complete. 2024-03-11T21:31:32.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:32.272+0000 7f65f78b5700 1 librados: shutdown 2024-03-11T21:31:32.254 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:31:32.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:32.304 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:32.315 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:32.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:32.445 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:32.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.577 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:32.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:32.707 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:32.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:31:32.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:31:32.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:32.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:33.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.247174+0000 osd.6 (osd.6) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.247519+0000 osd.6 (osd.6) 144 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:31.298788+0000 mgr.y (mgr.14550) 346 : cluster 0 pgmap v423: 460 pgs: 1 active+clean+snaptrim, 72 unknown, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 768 B/s wr, 3 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:31.861325+0000 mon.c (mon.2) 535 : audit 1 from='client.? 172.21.15.143:0/3774517188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:31.861609+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:31.968572+0000 mon.c (mon.2) 536 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:32.246038+0000 mon.a (mon.0) 1870 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:32.246104+0000 mon.a (mon.0) 1871 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:32.248434+0000 mon.a (mon.0) 1872 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]': finished 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:32.248662+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:32.248824+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: cluster 2024-03-11T21:31:32.252853+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-03-11T21:31:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:32.275883+0000 mon.c (mon.2) 537 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[29284]: audit 2024-03-11T21:31:32.278126+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.247174+0000 osd.6 (osd.6) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.247519+0000 osd.6 (osd.6) 144 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:31.298788+0000 mgr.y (mgr.14550) 346 : cluster 0 pgmap v423: 460 pgs: 1 active+clean+snaptrim, 72 unknown, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 768 B/s wr, 3 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:31.861325+0000 mon.c (mon.2) 535 : audit 1 from='client.? 172.21.15.143:0/3774517188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:31.861609+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:31.968572+0000 mon.c (mon.2) 536 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:32.246038+0000 mon.a (mon.0) 1870 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:33.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:32.246104+0000 mon.a (mon.0) 1871 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:32.248434+0000 mon.a (mon.0) 1872 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]': finished 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:32.248662+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:32.248824+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: cluster 2024-03-11T21:31:32.252853+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:32.275883+0000 mon.c (mon.2) 537 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:32 smithi143 bash[21512]: audit 2024-03-11T21:31:32.278126+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:33.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:33.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.247174+0000 osd.6 (osd.6) 143 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.247519+0000 osd.6 (osd.6) 144 : cluster 0 purged_snaps scrub found 10 strays 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:31.298788+0000 mgr.y (mgr.14550) 346 : cluster 0 pgmap v423: 460 pgs: 1 active+clean+snaptrim, 72 unknown, 52 creating+peering, 35 active+undersized+degraded, 89 active+undersized, 211 active+clean; 586 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 768 B/s wr, 3 op/s; 187/1185 objects degraded (15.781%) 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:31.861325+0000 mon.c (mon.2) 535 : audit 1 from='client.? 172.21.15.143:0/3774517188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:31.861609+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:31.968572+0000 mon.c (mon.2) 536 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:32.246038+0000 mon.a (mon.0) 1870 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:33.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:32.246104+0000 mon.a (mon.0) 1871 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:32.248434+0000 mon.a (mon.0) 1872 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-23", "mode": "writeback"}]': finished 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:32.248662+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi143-71604-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:32.248824+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: cluster 2024-03-11T21:31:32.252853+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:32.275883+0000 mon.c (mon.2) 537 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:32 smithi203 bash[23654]: audit 2024-03-11T21:31:32.278126+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:33.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:33.483 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:33.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:33.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:33.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:33.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:33.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:33.870 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:32.970215+0000 mon.c (mon.2) 538 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:33.273617+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: cluster 2024-03-11T21:31:33.278103+0000 mon.a (mon.0) 1878 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-03-11T21:31:33.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:33.278844+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:33.279288+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:33.374201+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[29284]: audit 2024-03-11T21:31:33.374602+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:32.970215+0000 mon.c (mon.2) 538 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:33.273617+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: cluster 2024-03-11T21:31:33.278103+0000 mon.a (mon.0) 1878 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:33.278844+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:33.279288+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:33.374201+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:33.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:33 smithi143 bash[21512]: audit 2024-03-11T21:31:33.374602+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:33.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:32.970215+0000 mon.c (mon.2) 538 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:33.273617+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: cluster 2024-03-11T21:31:33.278103+0000 mon.a (mon.0) 1878 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:33.278844+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.143:0/2914019639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:33.279288+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]: dispatch 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:33.374201+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:33.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:33 smithi203 bash[23654]: audit 2024-03-11T21:31:33.374602+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:34.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:34.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:34.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:34.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:34.262 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:34.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.394 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:34.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:34.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:34.759 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:31:34.759 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:31:34.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:34.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:31:34.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:34.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:35.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:35.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: cluster 2024-03-11T21:31:33.299534+0000 mgr.y (mgr.14550) 347 : cluster 0 pgmap v426: 420 pgs: 5 unknown, 35 active+undersized+degraded, 115 active+undersized, 265 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:35.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:33.971684+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.282724+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.282810+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: cluster 2024-03-11T21:31:34.286891+0000 mon.a (mon.0) 1883 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.289735+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.290273+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.312466+0000 mon.a (mon.0) 1885 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.313442+0000 mon.a (mon.0) 1886 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:34 smithi203 bash[23654]: audit 2024-03-11T21:31:34.313832+0000 mon.a (mon.0) 1887 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:35.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:35.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.155 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[150181]: debug 2024-03-11T21:31:35.040+0000 7f3a587637c0 -1 osd.1 246 log_to_monitors true 2024-03-11T21:31:35.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: cluster 2024-03-11T21:31:33.299534+0000 mgr.y (mgr.14550) 347 : cluster 0 pgmap v426: 420 pgs: 5 unknown, 35 active+undersized+degraded, 115 active+undersized, 265 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:35.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:33.971684+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.282724+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.282810+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: cluster 2024-03-11T21:31:34.286891+0000 mon.a (mon.0) 1883 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.289735+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.290273+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.312466+0000 mon.a (mon.0) 1885 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.313442+0000 mon.a (mon.0) 1886 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[29284]: audit 2024-03-11T21:31:34.313832+0000 mon.a (mon.0) 1887 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: cluster 2024-03-11T21:31:33.299534+0000 mgr.y (mgr.14550) 347 : cluster 0 pgmap v426: 420 pgs: 5 unknown, 35 active+undersized+degraded, 115 active+undersized, 265 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:33.971684+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:35.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.282724+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi143-71594-34"}]': finished 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.282810+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: cluster 2024-03-11T21:31:34.286891+0000 mon.a (mon.0) 1883 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.289735+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.290273+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]: dispatch 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.312466+0000 mon.a (mon.0) 1885 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.313442+0000 mon.a (mon.0) 1886 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:35.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:34 smithi143 bash[21512]: audit 2024-03-11T21:31:34.313832+0000 mon.a (mon.0) 1887 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:35.169 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:35.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.304 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:35.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.431 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:35.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.557 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:35.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.682 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:35.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.809 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:35.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:31:35.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:36.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:36.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:36.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:31:36.122 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[150181]: debug 2024-03-11T21:31:35.884+0000 7f3a45bb3700 -1 osd.1 294 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:31:36.122 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:34.973053+0000 mon.c (mon.2) 543 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.050269+0000 mon.c (mon.2) 544 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.050552+0000 mon.a (mon.0) 1888 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: cluster 2024-03-11T21:31:35.283891+0000 mon.a (mon.0) 1889 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.285894+0000 mon.a (mon.0) 1890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.285955+0000 mon.a (mon.0) 1891 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.286000+0000 mon.a (mon.0) 1892 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: cluster 2024-03-11T21:31:35.288604+0000 mon.a (mon.0) 1893 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.289259+0000 mon.a (mon.0) 1894 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.297203+0000 mon.c (mon.2) 545 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.297824+0000 mon.a (mon.0) 1895 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[21512]: audit 2024-03-11T21:31:35.789500+0000 mon.a (mon.0) 1896 : audit 1 from='osd.1 ' entity='osd.1' 2024-03-11T21:31:36.123 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:34.973053+0000 mon.c (mon.2) 543 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.050269+0000 mon.c (mon.2) 544 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.050552+0000 mon.a (mon.0) 1888 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: cluster 2024-03-11T21:31:35.283891+0000 mon.a (mon.0) 1889 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.285894+0000 mon.a (mon.0) 1890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.285955+0000 mon.a (mon.0) 1891 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.286000+0000 mon.a (mon.0) 1892 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: cluster 2024-03-11T21:31:35.288604+0000 mon.a (mon.0) 1893 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.289259+0000 mon.a (mon.0) 1894 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.297203+0000 mon.c (mon.2) 545 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.297824+0000 mon.a (mon.0) 1895 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:35 smithi143 bash[29284]: audit 2024-03-11T21:31:35.789500+0000 mon.a (mon.0) 1896 : audit 1 from='osd.1 ' entity='osd.1' 2024-03-11T21:31:36.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.185 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:34.973053+0000 mon.c (mon.2) 543 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.050269+0000 mon.c (mon.2) 544 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.050552+0000 mon.a (mon.0) 1888 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: cluster 2024-03-11T21:31:35.283891+0000 mon.a (mon.0) 1889 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.285894+0000 mon.a (mon.0) 1890 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-23"}]': finished 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.285955+0000 mon.a (mon.0) 1891 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi143-71594-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.286000+0000 mon.a (mon.0) 1892 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: cluster 2024-03-11T21:31:35.288604+0000 mon.a (mon.0) 1893 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.289259+0000 mon.a (mon.0) 1894 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:36.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.297203+0000 mon.c (mon.2) 545 : audit 1 from='osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.297824+0000 mon.a (mon.0) 1895 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-11T21:31:36.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:35 smithi203 bash[23654]: audit 2024-03-11T21:31:35.789500+0000 mon.a (mon.0) 1896 : audit 1 from='osd.1 ' entity='osd.1' 2024-03-11T21:31:36.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.337 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:36.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:36.456 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:36.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:36.585 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:36.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.713 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:36.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:36.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:37.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:37.098 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:37.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.209 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: cluster 2024-03-11T21:31:35.300404+0000 mgr.y (mgr.14550) 348 : cluster 0 pgmap v429: 356 pgs: 36 active+undersized+degraded, 96 active+undersized, 224 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: audit 2024-03-11T21:31:35.974133+0000 mon.c (mon.2) 546 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: audit 2024-03-11T21:31:36.025015+0000 mon.c (mon.2) 547 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: cluster 2024-03-11T21:31:36.243267+0000 mon.a (mon.0) 1897 : cluster 3 Health check update: Degraded data redundancy: 189/1203 objects degraded (15.711%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: cluster 2024-03-11T21:31:36.321283+0000 mon.a (mon.0) 1898 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: audit 2024-03-11T21:31:36.336092+0000 mon.c (mon.2) 548 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: cluster 2024-03-11T21:31:36.766652+0000 mon.a (mon.0) 1899 : cluster 1 osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] boot 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[21512]: cluster 2024-03-11T21:31:36.766698+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: cluster 2024-03-11T21:31:35.300404+0000 mgr.y (mgr.14550) 348 : cluster 0 pgmap v429: 356 pgs: 36 active+undersized+degraded, 96 active+undersized, 224 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: audit 2024-03-11T21:31:35.974133+0000 mon.c (mon.2) 546 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: audit 2024-03-11T21:31:36.025015+0000 mon.c (mon.2) 547 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: cluster 2024-03-11T21:31:36.243267+0000 mon.a (mon.0) 1897 : cluster 3 Health check update: Degraded data redundancy: 189/1203 objects degraded (15.711%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:37.211 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: cluster 2024-03-11T21:31:36.321283+0000 mon.a (mon.0) 1898 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:31:37.211 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: audit 2024-03-11T21:31:36.336092+0000 mon.c (mon.2) 548 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:31:37.211 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: cluster 2024-03-11T21:31:36.766652+0000 mon.a (mon.0) 1899 : cluster 1 osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] boot 2024-03-11T21:31:37.211 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:37 smithi143 bash[29284]: cluster 2024-03-11T21:31:36.766698+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:31:37.221 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:37.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:37.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:37.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.482 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: cluster 2024-03-11T21:31:35.300404+0000 mgr.y (mgr.14550) 348 : cluster 0 pgmap v429: 356 pgs: 36 active+undersized+degraded, 96 active+undersized, 224 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: audit 2024-03-11T21:31:35.974133+0000 mon.c (mon.2) 546 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: audit 2024-03-11T21:31:36.025015+0000 mon.c (mon.2) 547 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: cluster 2024-03-11T21:31:36.243267+0000 mon.a (mon.0) 1897 : cluster 3 Health check update: Degraded data redundancy: 189/1203 objects degraded (15.711%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: cluster 2024-03-11T21:31:36.321283+0000 mon.a (mon.0) 1898 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: audit 2024-03-11T21:31:36.336092+0000 mon.c (mon.2) 548 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: cluster 2024-03-11T21:31:36.766652+0000 mon.a (mon.0) 1899 : cluster 1 osd.1 [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] boot 2024-03-11T21:31:37.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:37 smithi203 bash[23654]: cluster 2024-03-11T21:31:36.766698+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:31:37.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.606 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:37.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:37.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:37.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:37.866 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:37.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:38.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:38.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:38.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:36.792494+0000 mon.c (mon.2) 549 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:36.975006+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: cluster 2024-03-11T21:31:37.300999+0000 mgr.y (mgr.14550) 349 : cluster 0 pgmap v431: 292 pgs: 35 active+undersized+degraded, 73 active+undersized, 184 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 254 B/s wr, 0 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.329459+0000 mon.a (mon.0) 1901 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: cluster 2024-03-11T21:31:37.334508+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.342552+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.344941+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.476057+0000 mon.c (mon.2) 552 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.476865+0000 mon.c (mon.2) 553 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.825930+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:38.237 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[21512]: audit 2024-03-11T21:31:37.976473+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:36.792494+0000 mon.c (mon.2) 549 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:36.975006+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: cluster 2024-03-11T21:31:37.300999+0000 mgr.y (mgr.14550) 349 : cluster 0 pgmap v431: 292 pgs: 35 active+undersized+degraded, 73 active+undersized, 184 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 254 B/s wr, 0 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.329459+0000 mon.a (mon.0) 1901 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: cluster 2024-03-11T21:31:37.334508+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.342552+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.344941+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.476057+0000 mon.c (mon.2) 552 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.476865+0000 mon.c (mon.2) 553 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.825930+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:38.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:38 smithi143 bash[29284]: audit 2024-03-11T21:31:37.976473+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.252 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:38.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:36.792494+0000 mon.c (mon.2) 549 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:31:38.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:36.975006+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: cluster 2024-03-11T21:31:37.300999+0000 mgr.y (mgr.14550) 349 : cluster 0 pgmap v431: 292 pgs: 35 active+undersized+degraded, 73 active+undersized, 184 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 254 B/s wr, 0 op/s; 189/1203 objects degraded (15.711%) 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.329459+0000 mon.a (mon.0) 1901 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi143-71594-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: cluster 2024-03-11T21:31:37.334508+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.342552+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.344941+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.476057+0000 mon.c (mon.2) 552 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.476865+0000 mon.c (mon.2) 553 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.825930+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:38.373 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[23654]: audit 2024-03-11T21:31:37.976473+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:38.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:38.415 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:38.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.509 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:38.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:38.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:38.636 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:38.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.765 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:38.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:38.898 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:39.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.028 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:39.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.157 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:39.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:39.289 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:39.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:39.402 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: audit 2024-03-11T21:31:38.334568+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:39.402 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: cluster 2024-03-11T21:31:38.347448+0000 mon.a (mon.0) 1906 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:31:39.402 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: audit 2024-03-11T21:31:38.349415+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.143:0/990177932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.402 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: audit 2024-03-11T21:31:38.353555+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.402 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: audit 2024-03-11T21:31:38.469295+0000 mgr.y (mgr.14550) 350 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[29284]: audit 2024-03-11T21:31:38.977903+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: audit 2024-03-11T21:31:38.334568+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: cluster 2024-03-11T21:31:38.347448+0000 mon.a (mon.0) 1906 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: audit 2024-03-11T21:31:38.349415+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.143:0/990177932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: audit 2024-03-11T21:31:38.353555+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: audit 2024-03-11T21:31:38.469295+0000 mgr.y (mgr.14550) 350 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:39.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:39 smithi143 bash[21512]: audit 2024-03-11T21:31:38.977903+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:39.415 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:39.427 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:39.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.545 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:39.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: audit 2024-03-11T21:31:38.334568+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: cluster 2024-03-11T21:31:38.347448+0000 mon.a (mon.0) 1906 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: audit 2024-03-11T21:31:38.349415+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.143:0/990177932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: audit 2024-03-11T21:31:38.353555+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: audit 2024-03-11T21:31:38.469295+0000 mgr.y (mgr.14550) 350 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:39.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:39 smithi203 bash[23654]: audit 2024-03-11T21:31:38.977903+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:39.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:39.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:39.805 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:39.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:39.937 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:40.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:40.065 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:40.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:40.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:40.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:40.323 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:40.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:40.445 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:40.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: cluster 2024-03-11T21:31:39.302379+0000 mgr.y (mgr.14550) 351 : cluster 0 pgmap v434: 332 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 31 creating+peering, 34 unknown, 11 active+undersized+degraded, 23 active+undersized, 228 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 40/1128 objects degraded (3.546%); 73 B/s, 3 objects/s recovering 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: cluster 2024-03-11T21:31:39.334454+0000 mon.a (mon.0) 1908 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: audit 2024-03-11T21:31:39.345035+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: cluster 2024-03-11T21:31:39.356063+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: audit 2024-03-11T21:31:39.360786+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: audit 2024-03-11T21:31:39.405172+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: audit 2024-03-11T21:31:39.405895+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:40 smithi203 bash[23654]: audit 2024-03-11T21:31:39.979024+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:40.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:40.581 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:40.672 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:31:40.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:40.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: cluster 2024-03-11T21:31:39.302379+0000 mgr.y (mgr.14550) 351 : cluster 0 pgmap v434: 332 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 31 creating+peering, 34 unknown, 11 active+undersized+degraded, 23 active+undersized, 228 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 40/1128 objects degraded (3.546%); 73 B/s, 3 objects/s recovering 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: cluster 2024-03-11T21:31:39.334454+0000 mon.a (mon.0) 1908 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: audit 2024-03-11T21:31:39.345035+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: cluster 2024-03-11T21:31:39.356063+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: audit 2024-03-11T21:31:39.360786+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: audit 2024-03-11T21:31:39.405172+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: audit 2024-03-11T21:31:39.405895+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[21512]: audit 2024-03-11T21:31:39.979024+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: cluster 2024-03-11T21:31:39.302379+0000 mgr.y (mgr.14550) 351 : cluster 0 pgmap v434: 332 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 31 creating+peering, 34 unknown, 11 active+undersized+degraded, 23 active+undersized, 228 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 40/1128 objects degraded (3.546%); 73 B/s, 3 objects/s recovering 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: cluster 2024-03-11T21:31:39.334454+0000 mon.a (mon.0) 1908 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: audit 2024-03-11T21:31:39.345035+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi143-71604-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: cluster 2024-03-11T21:31:39.356063+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:31:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: audit 2024-03-11T21:31:39.360786+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: audit 2024-03-11T21:31:39.405172+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: audit 2024-03-11T21:31:39.405895+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:40 smithi143 bash[29284]: audit 2024-03-11T21:31:39.979024+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:40.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:40.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:40.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:40.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:40.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:41.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:41.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:41.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:41.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:41.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.347124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.347193+0000 mon.a (mon.0) 1914 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:41.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: cluster 2024-03-11T21:31:40.350986+0000 mon.a (mon.0) 1915 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:31:41.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:41.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:41.499 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:41.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.352066+0000 mon.a (mon.0) 1916 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.356674+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.380657+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:40.980282+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: cluster 2024-03-11T21:31:41.244488+0000 mon.a (mon.0) 1918 : cluster 3 Health check update: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.248438+0000 mon.a (mon.0) 1919 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.248533+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.255672+0000 mon.c (mon.2) 561 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: cluster 2024-03-11T21:31:41.258682+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:31:41.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.261659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.262125+0000 mon.a (mon.0) 1923 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.289103+0000 mon.a (mon.0) 1924 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.290612+0000 mon.a (mon.0) 1925 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.290985+0000 mon.a (mon.0) 1926 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.303251+0000 mon.c (mon.2) 562 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[29284]: audit 2024-03-11T21:31:41.304372+0000 mon.a (mon.0) 1927 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.347124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.347193+0000 mon.a (mon.0) 1914 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:41.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: cluster 2024-03-11T21:31:40.350986+0000 mon.a (mon.0) 1915 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.352066+0000 mon.a (mon.0) 1916 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.356674+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.380657+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:40.980282+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: cluster 2024-03-11T21:31:41.244488+0000 mon.a (mon.0) 1918 : cluster 3 Health check update: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.248438+0000 mon.a (mon.0) 1919 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.248533+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.255672+0000 mon.c (mon.2) 561 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: cluster 2024-03-11T21:31:41.258682+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.261659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.262125+0000 mon.a (mon.0) 1923 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.289103+0000 mon.a (mon.0) 1924 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.290612+0000 mon.a (mon.0) 1925 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.290985+0000 mon.a (mon.0) 1926 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:41.617 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.303251+0000 mon.c (mon.2) 562 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.617 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:41 smithi143 bash[21512]: audit 2024-03-11T21:31:41.304372+0000 mon.a (mon.0) 1927 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.628 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:41.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:41.743 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.347124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.347193+0000 mon.a (mon.0) 1914 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: cluster 2024-03-11T21:31:40.350986+0000 mon.a (mon.0) 1915 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.352066+0000 mon.a (mon.0) 1916 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]: dispatch 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.356674+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.380657+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:40.980282+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: cluster 2024-03-11T21:31:41.244488+0000 mon.a (mon.0) 1918 : cluster 3 Health check update: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded (PG_DEGRADED) 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.248438+0000 mon.a (mon.0) 1919 : audit 1 from='client.? 172.21.15.143:0/950557910' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi143-71594-35"}]': finished 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.248533+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.255672+0000 mon.c (mon.2) 561 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: cluster 2024-03-11T21:31:41.258682+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.261659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.262125+0000 mon.a (mon.0) 1923 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.289103+0000 mon.a (mon.0) 1924 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.290612+0000 mon.a (mon.0) 1925 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.290985+0000 mon.a (mon.0) 1926 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.303251+0000 mon.c (mon.2) 562 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:41 smithi203 bash[23654]: audit 2024-03-11T21:31:41.304372+0000 mon.a (mon.0) 1927 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T21:31:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:41.861 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:41.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:41.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:42.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:42.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:42.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:42.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:42.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.290 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:42.394 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: cluster 2024-03-11T21:31:41.302904+0000 mgr.y (mgr.14550) 352 : cluster 0 pgmap v438: 324 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 32 unknown, 22 creating+peering, 265 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 203 B/s, 4 objects/s recovering 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:41.982098+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: cluster 2024-03-11T21:31:42.248485+0000 mon.a (mon.0) 1928 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded) 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: cluster 2024-03-11T21:31:42.249454+0000 mon.a (mon.0) 1929 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.256256+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]': finished 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.256366+0000 mon.a (mon.0) 1931 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.256449+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:42.402 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.256503+0000 mon.a (mon.0) 1933 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T21:31:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: cluster 2024-03-11T21:31:41.302904+0000 mgr.y (mgr.14550) 352 : cluster 0 pgmap v438: 324 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 32 unknown, 22 creating+peering, 265 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 203 B/s, 4 objects/s recovering 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:41.982098+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: cluster 2024-03-11T21:31:42.248485+0000 mon.a (mon.0) 1928 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded) 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: cluster 2024-03-11T21:31:42.249454+0000 mon.a (mon.0) 1929 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.256256+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]': finished 2024-03-11T21:31:42.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.256366+0000 mon.a (mon.0) 1931 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.256449+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.256503+0000 mon.a (mon.0) 1933 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: cluster 2024-03-11T21:31:42.259591+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.260352+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.323377+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:42 smithi203 bash[23654]: audit 2024-03-11T21:31:42.324317+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: cluster 2024-03-11T21:31:42.259591+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.260352+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.323377+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[21512]: audit 2024-03-11T21:31:42.324317+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: cluster 2024-03-11T21:31:41.302904+0000 mgr.y (mgr.14550) 352 : cluster 0 pgmap v438: 324 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 32 unknown, 22 creating+peering, 265 active+clean; 586 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 203 B/s, 4 objects/s recovering 2024-03-11T21:31:42.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:41.982098+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: cluster 2024-03-11T21:31:42.248485+0000 mon.a (mon.0) 1928 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1128 objects degraded (3.546%), 11 pgs degraded) 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: cluster 2024-03-11T21:31:42.249454+0000 mon.a (mon.0) 1929 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.256256+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-25", "mode": "writeback"}]': finished 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.256366+0000 mon.a (mon.0) 1931 : audit 1 from='client.? 172.21.15.143:0/744436211' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi143-71604-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.256449+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi143-71594-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.256503+0000 mon.a (mon.0) 1933 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: cluster 2024-03-11T21:31:42.259591+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.260352+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.323377+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:42 smithi143 bash[29284]: audit 2024-03-11T21:31:42.324317+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:42.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:42.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:42.685 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:42.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:42.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:42.946 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:43.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:43.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:43.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:43.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:43.306 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:43.330 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:43.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:43.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:43.578 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:43.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:43.706 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:43.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:43.831 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:43.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:43 smithi143 bash[21512]: audit 2024-03-11T21:31:42.983916+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:43.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:43 smithi143 bash[21512]: audit 2024-03-11T21:31:43.260144+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:43.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:43 smithi143 bash[29284]: audit 2024-03-11T21:31:42.983916+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:43.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:43 smithi143 bash[29284]: audit 2024-03-11T21:31:43.260144+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:43.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:43.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:43.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:43 smithi203 bash[23654]: audit 2024-03-11T21:31:42.983916+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:43.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:43 smithi203 bash[23654]: audit 2024-03-11T21:31:43.260144+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:44.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:44.096 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:44.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.224 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:44.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:44.354 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:44.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.484 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:44.527 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:31:44.527 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:31:44.587 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:44.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:44.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.271088+0000 mon.c (mon.2) 566 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:44.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: cluster 2024-03-11T21:31:43.304133+0000 mgr.y (mgr.14550) 353 : cluster 0 pgmap v441: 292 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 287 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 517 B/s wr, 2 op/s; 129 B/s, 1 objects/s recovering 2024-03-11T21:31:44.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: cluster 2024-03-11T21:31:43.728626+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:31:44.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.729905+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.799630+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.799963+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.800679+0000 mon.c (mon.2) 568 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.800955+0000 mon.a (mon.0) 1941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.801456+0000 mon.c (mon.2) 569 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.801740+0000 mon.a (mon.0) 1942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:43.985458+0000 mon.c (mon.2) 570 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: cluster 2024-03-11T21:31:44.260857+0000 mon.a (mon.0) 1943 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:44.729 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:44.268087+0000 mon.a (mon.0) 1944 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:44.268198+0000 mon.a (mon.0) 1945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:44.268321+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: cluster 2024-03-11T21:31:44.273306+0000 mon.a (mon.0) 1947 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:44.274175+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[21512]: audit 2024-03-11T21:31:44.277168+0000 mon.a (mon.0) 1948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.730 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.271088+0000 mon.c (mon.2) 566 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: cluster 2024-03-11T21:31:43.304133+0000 mgr.y (mgr.14550) 353 : cluster 0 pgmap v441: 292 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 287 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 517 B/s wr, 2 op/s; 129 B/s, 1 objects/s recovering 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: cluster 2024-03-11T21:31:43.728626+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.729905+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.799630+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.799963+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.800679+0000 mon.c (mon.2) 568 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.800955+0000 mon.a (mon.0) 1941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.801456+0000 mon.c (mon.2) 569 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:44.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.801740+0000 mon.a (mon.0) 1942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:43.985458+0000 mon.c (mon.2) 570 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: cluster 2024-03-11T21:31:44.260857+0000 mon.a (mon.0) 1943 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:44.268087+0000 mon.a (mon.0) 1944 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:44.268198+0000 mon.a (mon.0) 1945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:44.268321+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: cluster 2024-03-11T21:31:44.273306+0000 mon.a (mon.0) 1947 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:44.274175+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:44 smithi143 bash[29284]: audit 2024-03-11T21:31:44.277168+0000 mon.a (mon.0) 1948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:44.740 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:44.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.870 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:44.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.995 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.271088+0000 mon.c (mon.2) 566 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: cluster 2024-03-11T21:31:43.304133+0000 mgr.y (mgr.14550) 353 : cluster 0 pgmap v441: 292 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 287 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 517 B/s wr, 2 op/s; 129 B/s, 1 objects/s recovering 2024-03-11T21:31:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: cluster 2024-03-11T21:31:43.728626+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:31:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.729905+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.799630+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.799963+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.800679+0000 mon.c (mon.2) 568 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.800955+0000 mon.a (mon.0) 1941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.801456+0000 mon.c (mon.2) 569 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.801740+0000 mon.a (mon.0) 1942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:43.985458+0000 mon.c (mon.2) 570 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: cluster 2024-03-11T21:31:44.260857+0000 mon.a (mon.0) 1943 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:44.268087+0000 mon.a (mon.0) 1944 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi143-71594-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:44.268198+0000 mon.a (mon.0) 1945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-25"}]': finished 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:44.268321+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:45.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: cluster 2024-03-11T21:31:44.273306+0000 mon.a (mon.0) 1947 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:31:45.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:44.274175+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:44 smithi203 bash[23654]: audit 2024-03-11T21:31:44.277168+0000 mon.a (mon.0) 1948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:45.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.125 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:45.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:31:45.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:45.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.392 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:45.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:45.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:45.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.646 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:45.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:45.777 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:45.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:45.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:46.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:46.032 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:46.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:46.144 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[29284]: audit 2024-03-11T21:31:44.987140+0000 mon.c (mon.2) 572 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[29284]: audit 2024-03-11T21:31:45.305717+0000 mon.c (mon.2) 573 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[29284]: cluster 2024-03-11T21:31:45.718565+0000 mon.a (mon.0) 1949 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[29284]: audit 2024-03-11T21:31:45.720718+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[21512]: audit 2024-03-11T21:31:44.987140+0000 mon.c (mon.2) 572 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[21512]: audit 2024-03-11T21:31:45.305717+0000 mon.c (mon.2) 573 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[21512]: cluster 2024-03-11T21:31:45.718565+0000 mon.a (mon.0) 1949 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:31:46.145 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:46 smithi143 bash[21512]: audit 2024-03-11T21:31:45.720718+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.157 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:46.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:46.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:46 smithi203 bash[23654]: audit 2024-03-11T21:31:44.987140+0000 mon.c (mon.2) 572 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:46.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:46 smithi203 bash[23654]: audit 2024-03-11T21:31:45.305717+0000 mon.c (mon.2) 573 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:46 smithi203 bash[23654]: cluster 2024-03-11T21:31:45.718565+0000 mon.a (mon.0) 1949 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:31:46.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:46 smithi203 bash[23654]: audit 2024-03-11T21:31:45.720718+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T21:31:46.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:46.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:46.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:46.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:46.553 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:46.656 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:46.684 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:46.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:46.814 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:46.917 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:46.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:47.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:47.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:47.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:47.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: cluster 2024-03-11T21:31:45.305213+0000 mgr.y (mgr.14550) 354 : cluster 0 pgmap v444: 267 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:47.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:45.988291+0000 mon.c (mon.2) 574 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: cluster 2024-03-11T21:31:46.247814+0000 mon.a (mon.0) 1951 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:47.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.280949+0000 mon.a (mon.0) 1952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.281262+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: cluster 2024-03-11T21:31:46.289847+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.296017+0000 mon.c (mon.2) 575 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.302414+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.303320+0000 mon.a (mon.0) 1956 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[21512]: audit 2024-03-11T21:31:46.989844+0000 mon.c (mon.2) 576 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: cluster 2024-03-11T21:31:45.305213+0000 mgr.y (mgr.14550) 354 : cluster 0 pgmap v444: 267 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:45.988291+0000 mon.c (mon.2) 574 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: cluster 2024-03-11T21:31:46.247814+0000 mon.a (mon.0) 1951 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:47.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.280949+0000 mon.a (mon.0) 1952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.281262+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: cluster 2024-03-11T21:31:46.289847+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.296017+0000 mon.c (mon.2) 575 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.302414+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.303320+0000 mon.a (mon.0) 1956 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:47 smithi143 bash[29284]: audit 2024-03-11T21:31:46.989844+0000 mon.c (mon.2) 576 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.204 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:47.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: cluster 2024-03-11T21:31:45.305213+0000 mgr.y (mgr.14550) 354 : cluster 0 pgmap v444: 267 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:45.988291+0000 mon.c (mon.2) 574 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: cluster 2024-03-11T21:31:46.247814+0000 mon.a (mon.0) 1951 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.280949+0000 mon.a (mon.0) 1952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.281262+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: cluster 2024-03-11T21:31:46.289847+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.296017+0000 mon.c (mon.2) 575 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.317 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.302414+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:47.318 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.303320+0000 mon.a (mon.0) 1956 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:47.318 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:47 smithi203 bash[23654]: audit 2024-03-11T21:31:46.989844+0000 mon.c (mon.2) 576 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:47.333 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:47.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:47.458 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:47.561 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:47.692 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:47.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:47.849 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:47.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:48.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:48.106 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:48.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:48.238 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:48.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.352 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: audit 2024-03-11T21:31:47.284367+0000 mon.a (mon.0) 1957 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:48.352 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: audit 2024-03-11T21:31:47.284535+0000 mon.a (mon.0) 1958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:48.352 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: cluster 2024-03-11T21:31:47.293670+0000 mon.a (mon.0) 1959 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:31:48.352 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: audit 2024-03-11T21:31:47.299537+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:48.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: cluster 2024-03-11T21:31:47.309223+0000 mgr.y (mgr.14550) 355 : cluster 0 pgmap v447: 299 pgs: 40 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:48.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[23654]: audit 2024-03-11T21:31:47.991124+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:48.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:48.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.496 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:48.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:48.610 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:48.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:48.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: audit 2024-03-11T21:31:47.284367+0000 mon.a (mon.0) 1957 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: audit 2024-03-11T21:31:47.284535+0000 mon.a (mon.0) 1958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: cluster 2024-03-11T21:31:47.293670+0000 mon.a (mon.0) 1959 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: audit 2024-03-11T21:31:47.299537+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: cluster 2024-03-11T21:31:47.309223+0000 mgr.y (mgr.14550) 355 : cluster 0 pgmap v447: 299 pgs: 40 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[21512]: audit 2024-03-11T21:31:47.991124+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: audit 2024-03-11T21:31:47.284367+0000 mon.a (mon.0) 1957 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: audit 2024-03-11T21:31:47.284535+0000 mon.a (mon.0) 1958 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: cluster 2024-03-11T21:31:47.293670+0000 mon.a (mon.0) 1959 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: audit 2024-03-11T21:31:47.299537+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]: dispatch 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: cluster 2024-03-11T21:31:47.309223+0000 mgr.y (mgr.14550) 355 : cluster 0 pgmap v447: 299 pgs: 40 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 254 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:48.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:48 smithi143 bash[29284]: audit 2024-03-11T21:31:47.991124+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:48.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:48.754 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:48.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:48.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:49.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:49.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:49.146 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:49.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.281 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:49.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.411 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:49.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:49.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.286861+0000 mon.a (mon.0) 1961 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: cluster 2024-03-11T21:31:48.293838+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.296493+0000 mon.c (mon.2) 578 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.304374+0000 mon.a (mon.0) 1963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.319894+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.322379+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.322890+0000 mon.a (mon.0) 1966 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.349264+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.349693+0000 mon.a (mon.0) 1967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.480385+0000 mgr.y (mgr.14550) 356 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:49 smithi203 bash[23654]: audit 2024-03-11T21:31:48.992673+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.286861+0000 mon.a (mon.0) 1961 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: cluster 2024-03-11T21:31:48.293838+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.296493+0000 mon.c (mon.2) 578 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.304374+0000 mon.a (mon.0) 1963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.319894+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.322379+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.322890+0000 mon.a (mon.0) 1966 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:49.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.349264+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.349693+0000 mon.a (mon.0) 1967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.480385+0000 mgr.y (mgr.14550) 356 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[21512]: audit 2024-03-11T21:31:48.992673+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.286861+0000 mon.a (mon.0) 1961 : audit 1 from='client.? 172.21.15.143:0/1000515913' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi143-71594-36"}]': finished 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: cluster 2024-03-11T21:31:48.293838+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.296493+0000 mon.c (mon.2) 578 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.304374+0000 mon.a (mon.0) 1963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.319894+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.322379+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.322890+0000 mon.a (mon.0) 1966 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.349264+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.349693+0000 mon.a (mon.0) 1967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:31:49.523 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.480385+0000 mgr.y (mgr.14550) 356 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:49.523 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:49 smithi143 bash[29284]: audit 2024-03-11T21:31:48.992673+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:49.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:49.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/4192396551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea010e8f0 msgr2=0x7f5ea0110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/4192396551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea010e8f0 0x7f5ea0110ce0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009f90 tx=0x7f5e9800b330 comp rx=0 tx=0).stop 2024-03-11T21:31:49.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/4192396551 shutdown_connections 2024-03-11T21:31:49.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/4192396551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea010e8f0 0x7f5ea0110ce0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/4192396551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea0101850 0x7f5ea0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/4192396551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/4192396551 >> 172.21.15.143:0/4192396551 conn(0x7f5ea0076d80 msgr2=0x7f5ea0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:49.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/4192396551 shutdown_connections 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/4192396551 wait complete. 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 librados: starting msgr at 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 librados: starting objecter 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 Processor -- start 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- start start 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 librados: setting wanted keys 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 librados: calling monclient init 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 0x7f5ea0199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea010e8f0 0x7f5ea019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea0113850 con 0x7f5ea0101850 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01136d0 con 0x7f5ea010e8f0 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01139d0 con 0x7f5ea0102170 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 0x7f5ea0199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea019a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:39672/0 (socket says 172.21.15.143:39672) 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9e59c700 1 -- 172.21.15.143:0/3431713019 learned_addr learned my addr 172.21.15.143:0/3431713019 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea010e8f0 0x7f5ea019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 msgr2=0x7f5ea019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea010e8f0 msgr2=0x7f5ea019e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea010e8f0 0x7f5ea019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea019ee30 con 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ea0102170 0x7f5ea019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 0x7f5ea0199e20 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b450 tx=0x7f5e9400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e87fff700 10 client.?.objecter ms_handle_connect 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e87fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94015020 con 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5e94011cb0 con 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea019f120 con 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94010c40 con 0x7f5ea0101850 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 10 client.30117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:49.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea01a69a0 con 0x7f5ea0101850 2024-03-11T21:31:49.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 librados: init done 2024-03-11T21:31:49.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.635+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5ea004faa0 con 0x7f5ea0101850 2024-03-11T21:31:49.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5e94018020 con 0x7f5ea0101850 2024-03-11T21:31:49.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 0x7f5e88071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(308..308 src has 87..308) v4 ==== 10185+0+0 (secure 0 0 0) 0x7f5e9408e3a0 con 0x7f5ea0101850 2024-03-11T21:31:49.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 0x7f5e88071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 osd_map(308..308 src has 87..308) v4 2024-03-11T21:31:49.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 3 client.30117.objecter handle_osd_map got epochs [308,308] > 0 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 3 client.30117.objecter handle_osd_map decoding full epoch 308 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 20 client.30117.objecter dump_active .. 0 homeless 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v308) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5e9401f020 con 0x7f5ea0101850 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 0x7f5e88071ce0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009750 tx=0x7f5e90009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5e8806f820 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 1 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5ea00629f0 0x7f5ea0062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 1 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 0 1 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 4 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 10 client.30117.objecter _send_command 1 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01a9d60 con 0x7f5ea00629f0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5ea00629f0 0x7f5ea0062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5ea00629f0 0x7f5ea0062dd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea00629f0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01a9d60 con 0x7f5ea00629f0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 10 client.30117.objecter _finish_command 1 = system:0 2024-03-11T21:31:49.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 4 2024-03-11T21:31:49.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.639+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 0 1 2024-03-11T21:31:49.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 2 2024-03-11T21:31:49.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 2 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea019b180 osd=0 3 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 0 2 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 4 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 10 client.30117.objecter _send_command 2 2024-03-11T21:31:49.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea0062470 con 0x7f5ea00629f0 2024-03-11T21:31:49.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea0062470 con 0x7f5ea00629f0 2024-03-11T21:31:49.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 10 client.30117.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 4 2024-03-11T21:31:49.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 0 2 2024-03-11T21:31:49.635 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:31:49.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f5ea002c6b0 0x7f5ea002cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 1 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 4 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 10 client.30117.objecter _send_command 3 2024-03-11T21:31:49.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01091c0 con 0x7f5ea002c6b0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f5ea002c6b0 0x7f5ea002cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea002c6b0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f5ea002c6b0 0x7f5ea002cab0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01091c0 con 0x7f5ea002c6b0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 10 client.30117.objecter _finish_command 3 = system:0 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 4 2024-03-11T21:31:49.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.659+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 1 3 2024-03-11T21:31:49.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 4 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 4 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea0108b40 osd=1 3 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 1 4 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 4 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 10 client.30117.objecter _send_command 4 2024-03-11T21:31:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01085c0 con 0x7f5ea002c6b0 2024-03-11T21:31:49.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01085c0 con 0x7f5ea002c6b0 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.679+0000 7f5e87fff700 10 client.30117.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 4 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 1 4 2024-03-11T21:31:49.657 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 5 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5ea01abf20 0x7f5ea01ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 5 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 2 5 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 4 2024-03-11T21:31:49.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 10 client.30117.objecter _send_command 5 2024-03-11T21:31:49.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01b1170 con 0x7f5ea01abf20 2024-03-11T21:31:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5ea01abf20 0x7f5ea01ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01abf20 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5ea01abf20 0x7f5ea01ae310 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01b1170 con 0x7f5ea01abf20 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:31:49.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 10 client.30117.objecter _finish_command 5 = system:0 2024-03-11T21:31:49.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 4 2024-03-11T21:31:49.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.683+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 2 5 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 6 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 6 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea0109370 osd=2 3 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 2 6 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 4 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 10 client.30117.objecter _send_command 6 2024-03-11T21:31:49.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.699+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea002d240 con 0x7f5ea01abf20 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea002d240 con 0x7f5ea01abf20 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 10 client.30117.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 4 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 2 6 2024-03-11T21:31:49.678 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 7 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5ea01a98c0 0x7f5ea01af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 7 2024-03-11T21:31:49.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 3 7 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 4 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 10 client.30117.objecter _send_command 7 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01b4b20 con 0x7f5ea01a98c0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5ea01a98c0 0x7f5ea01af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01a98c0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5ea01a98c0 0x7f5ea01af9d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01b4b20 con 0x7f5ea01a98c0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 10 client.30117.objecter _finish_command 7 = system:0 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 4 2024-03-11T21:31:49.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.703+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 3 7 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 8 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 8 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea002d640 osd=3 3 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 3 8 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 4 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 10 client.30117.objecter _send_command 8 2024-03-11T21:31:49.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01a8e30 con 0x7f5ea01a98c0 2024-03-11T21:31:49.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01a8e30 con 0x7f5ea01a98c0 2024-03-11T21:31:49.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 10 client.30117.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 4 2024-03-11T21:31:49.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 3 8 2024-03-11T21:31:49.699 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:31:49.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 9 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5ea01a8be0 0x7f5ea01b3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 9 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 4 9 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 4 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 10 client.30117.objecter _send_command 9 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01af640 con 0x7f5ea01a8be0 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5ea01a8be0 0x7f5ea01b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5ea01a8be0 0x7f5ea01b3330 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.723+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01a8be0 2024-03-11T21:31:49.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.727+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01af640 con 0x7f5ea01a8be0 2024-03-11T21:31:49.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.727+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:31:49.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.727+0000 7f5e87fff700 10 client.30117.objecter _finish_command 9 = system:0 2024-03-11T21:31:49.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.727+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 4 2024-03-11T21:31:49.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.727+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 4 9 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 10 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 10 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b4b80 osd=4 3 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 4 10 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 4 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 10 client.30117.objecter _send_command 10 2024-03-11T21:31:49.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.743+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01af640 con 0x7f5ea01a8be0 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01af640 con 0x7f5ea01a8be0 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e87fff700 10 client.30117.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 4 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 4 10 2024-03-11T21:31:49.723 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 11 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5ea01b3d80 0x7f5ea01b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 11 2024-03-11T21:31:49.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 5 11 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 4 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 10 client.30117.objecter _send_command 11 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01b47b0 con 0x7f5ea01b3d80 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.747+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5ea01b3d80 0x7f5ea01b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01b3d80 2024-03-11T21:31:49.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e9e59c700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5ea01b3d80 0x7f5ea01b41e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01b47b0 con 0x7f5ea01b3d80 2024-03-11T21:31:49.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:31:49.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 10 client.30117.objecter _finish_command 11 = system:0 2024-03-11T21:31:49.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 4 2024-03-11T21:31:49.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.751+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 5 11 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 12 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 12 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01af850 osd=5 3 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 5 12 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 4 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 10 client.30117.objecter _send_command 12 2024-03-11T21:31:49.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.771+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01b47b0 con 0x7f5ea01b3d80 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01b47b0 con 0x7f5ea01b3d80 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 10 client.30117.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 4 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 5 12 2024-03-11T21:31:49.749 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 13 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5ea01b9260 0x7f5ea01b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 13 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 6 13 2024-03-11T21:31:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 4 2024-03-11T21:31:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 10 client.30117.objecter _send_command 13 2024-03-11T21:31:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01be850 con 0x7f5ea01b9260 2024-03-11T21:31:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5ea01b9260 0x7f5ea01b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01b9260 2024-03-11T21:31:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.775+0000 7f5e9f59e700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5ea01b9260 0x7f5ea01b96a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.779+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01be850 con 0x7f5ea01b9260 2024-03-11T21:31:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.779+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:31:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.779+0000 7f5e87fff700 10 client.30117.objecter _finish_command 13 = system:0 2024-03-11T21:31:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.779+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 4 2024-03-11T21:31:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.779+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 6 13 2024-03-11T21:31:49.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:49.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 14 2024-03-11T21:31:49.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 14 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b47b0 osd=6 3 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 6 14 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 4 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 10 client.30117.objecter _send_command 14 2024-03-11T21:31:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.799+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01b9cf0 con 0x7f5ea01b9260 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01b9cf0 con 0x7f5ea01b9260 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 10 client.30117.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 4 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 6 14 2024-03-11T21:31:49.777 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 15 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5ea01ba370 0x7f5ea01ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 15 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 7 15 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 4 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 10 client.30117.objecter _send_command 15 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea01bae90 con 0x7f5ea01ba370 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5ea01ba370 0x7f5ea01ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e9ed9d700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5ea01ba370 0x7f5ea01ba7b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.803+0000 7f5e87fff700 10 client.30117.objecter ms_handle_connect 0x7f5ea01ba370 2024-03-11T21:31:49.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.807+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea01bae90 con 0x7f5ea01ba370 2024-03-11T21:31:49.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.807+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:31:49.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.807+0000 7f5e87fff700 10 client.30117.objecter _finish_command 15 = system:0 2024-03-11T21:31:49.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.807+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 4 2024-03-11T21:31:49.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.807+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 7 15 2024-03-11T21:31:49.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:49.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 10 client.30117.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign -1 16 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 20 client.30117.objecter _get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_remove -1 16 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 20 client.30117.objecter get_session s=0x7f5ea01b9cf0 osd=7 3 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 15 client.30117.objecter _session_command_op_assign 7 16 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 4 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 10 client.30117.objecter _send_command 16 2024-03-11T21:31:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.823+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5ea01bb070 con 0x7f5ea01ba370 2024-03-11T21:31:49.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e87fff700 1 -- 172.21.15.143:0/3431713019 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5ea01bb070 con 0x7f5ea01ba370 2024-03-11T21:31:49.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e87fff700 10 client.30117.objecter ms_dispatch 0x7f5ea0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:31:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e87fff700 10 client.30117.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:31:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e87fff700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 4 2024-03-11T21:31:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e87fff700 15 client.30117.objecter _session_command_op_remove 7 16 2024-03-11T21:31:49.803 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:31:49.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 librados: watch_flush enter 2024-03-11T21:31:49.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 librados: watch_flush exit 2024-03-11T21:31:49.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.0 2024-03-11T21:31:49.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5ea00629f0 msgr2=0x7f5ea0062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f5ea00629f0 0x7f5ea0062dd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea019b180 osd=0 1 2024-03-11T21:31:49.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.1 2024-03-11T21:31:49.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f5ea002c6b0 msgr2=0x7f5ea002cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f5ea002c6b0 0x7f5ea002cab0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0108b40 osd=1 1 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.2 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5ea01abf20 msgr2=0x7f5ea01ae310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f5ea01abf20 0x7f5ea01ae310 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea0109370 osd=2 1 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.3 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5ea01a98c0 msgr2=0x7f5ea01af9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f5ea01a98c0 0x7f5ea01af9d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea002d640 osd=3 1 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.4 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5ea01a8be0 msgr2=0x7f5ea01b3330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f5ea01a8be0 0x7f5ea01b3330 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b4b80 osd=4 1 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.5 2024-03-11T21:31:49.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5ea01b3d80 msgr2=0x7f5ea01b41e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f5ea01b3d80 0x7f5ea01b41e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01af850 osd=5 1 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.6 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5ea01b9260 msgr2=0x7f5ea01b96a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f5ea01b9260 0x7f5ea01b96a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b47b0 osd=6 1 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter close_session for osd.7 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5ea01ba370 msgr2=0x7f5ea01ba7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e9f59e700 1 -- 172.21.15.143:0/3431713019 reap_dead start 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5ea01ba370 0x7f5ea01ba7b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e9f59e700 1 -- 172.21.15.143:0/3431713019 reap_dead start 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter put_session s=0x7f5ea01b9cf0 osd=7 1 2024-03-11T21:31:49.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 20 client.30117.objecter shutdown clearing up homeless session... 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 10 client.30117.objecter successfully canceled tick 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e9f59e700 1 -- 172.21.15.143:0/3431713019 reap_dead start 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5e9f59e700 1 -- 172.21.15.143:0/3431713019 reap_dead start 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 msgr2=0x7f5e88071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 0x7f5e88071ce0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009750 tx=0x7f5e90009310 comp rx=0 tx=0).stop 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 msgr2=0x7f5ea0199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 0x7f5ea0199e20 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b450 tx=0x7f5e9400b760 comp rx=0 tx=0).stop 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 shutdown_connections 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ea0101850 0x7f5ea0199e20 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 --2- 172.21.15.143:0/3431713019 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e8806f820 0x7f5e88071ce0 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 >> 172.21.15.143:0/3431713019 conn(0x7f5ea0076d80 msgr2=0x7f5ea010fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 shutdown_connections 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 -- 172.21.15.143:0/3431713019 wait complete. 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.827+0000 7f5ea50d0700 1 librados: shutdown 2024-03-11T21:31:49.809 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:49.810 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:49.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/2274242187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08102170 msgr2=0x7f1e0810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/2274242187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08102170 0x7f1e0810e3b0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f1df40099c0 tx=0x7f1df4009cd0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/2274242187 shutdown_connections 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/2274242187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e08101850 0x7f1e08101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/2274242187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e0810e8f0 0x7f1e08110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/2274242187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08102170 0x7f1e0810e3b0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/2274242187 >> 172.21.15.143:0/2274242187 conn(0x7f1e08076d80 msgr2=0x7f1e08077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/2274242187 shutdown_connections 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/2274242187 wait complete. 2024-03-11T21:31:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 librados: starting msgr at 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 librados: starting objecter 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 Processor -- start 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- start start 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 librados: setting wanted keys 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 librados: calling monclient init 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 0x7f1e08199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e08102170 0x7f1e0819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08113840 con 0x7f1e0810e8f0 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081136c0 con 0x7f1e08102170 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081139c0 con 0x7f1e08101850 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0d060700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0d060700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46796/0 (socket says 172.21.15.143:46796) 2024-03-11T21:31:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 0x7f1e08199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0d060700 1 -- 172.21.15.143:0/409860231 learned_addr learned my addr 172.21.15.143:0/409860231 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e07fff700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e08102170 0x7f1e0819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 -- 172.21.15.143:0/409860231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e08102170 msgr2=0x7f1e0819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e08102170 0x7f1e0819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 -- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 msgr2=0x7f1e0819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 -- 172.21.15.143:0/409860231 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0819ee60 con 0x7f1e08101850 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0d060700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0c85f700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 0x7f1e08199e50 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00e9f0 tx=0x7f1dfc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e05ffb700 10 client.?.objecter ms_handle_connect 0x7f1e08101850 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e05ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc00c9c0 con 0x7f1e08101850 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1dfc010430 con 0x7f1e08101850 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc018680 con 0x7f1e08101850 2024-03-11T21:31:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0819f150 con 0x7f1e08101850 2024-03-11T21:31:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 10 client.41324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e081a69d0 con 0x7f1e08101850 2024-03-11T21:31:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 librados: init done 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.935+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0804faa0 con 0x7f1e08101850 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1dfc0105e0 con 0x7f1e08101850 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 0x7f1df8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(308..308 src has 87..308) v4 ==== 10185+0+0 (secure 0 0 0) 0x7f1dfc08f910 con 0x7f1e08101850 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 10 client.41324.objecter ms_dispatch 0x7f1e08001550 osd_map(308..308 src has 87..308) v4 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 3 client.41324.objecter handle_osd_map got epochs [308,308] > 0 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 3 client.41324.objecter handle_osd_map decoding full epoch 308 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e07fff700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 0x7f1df8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 20 client.41324.objecter dump_active .. 0 homeless 2024-03-11T21:31:49.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e05ffb700 10 client.41324.objecter ms_handle_connect 0x7f1df806f820 2024-03-11T21:31:49.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.939+0000 7f1e07fff700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 0x7f1df8071ce0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f1df400b580 tx=0x7f1df4019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:49.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:49.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:49.943+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1dfc026070 con 0x7f1e08101850 2024-03-11T21:31:50.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:31:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/531670725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 msgr2=0x7f202406a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- 172.21.15.143:0/531670725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202406a150 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f20200099c0 tx=0x7f2020009cd0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/531670725 shutdown_connections 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- 172.21.15.143:0/531670725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20241071c0 0x7f202410ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- 172.21.15.143:0/531670725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202406a150 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- 172.21.15.143:0/531670725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f202406a720 0x7f2024106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/531670725 >> 172.21.15.143:0/531670725 conn(0x7f2024076d80 msgr2=0x7f2024077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/531670725 shutdown_connections 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/531670725 wait complete. 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 librados: starting msgr at 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 librados: starting objecter 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 Processor -- start 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- start start 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 librados: setting wanted keys 2024-03-11T21:31:50.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 librados: calling monclient init 2024-03-11T21:31:50.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:50.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:50.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241071c0 0x7f2024197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:50.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024111630 con 0x7f202406a720 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20241114b0 con 0x7f20241071c0 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20241117b0 con 0x7f2024069d70 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f2029b0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f2029b0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46808/0 (socket says 172.21.15.143:46808) 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f2029b0d700 1 -- 172.21.15.143:0/2803841966 learned_addr learned my addr 172.21.15.143:0/2803841966 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202ab0f700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241071c0 0x7f2024197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 -- 172.21.15.143:0/2803841966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241071c0 msgr2=0x7f2024197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241071c0 0x7f2024197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 -- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 msgr2=0x7f202419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 -- 172.21.15.143:0/2803841966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024198260 con 0x7f2024069d70 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f2029b0d700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:31:50.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202a30e700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202419d940 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f20200099c0 tx=0x7f202000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f201b7fe700 10 client.?.objecter ms_handle_connect 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f201b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020009e60 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f202000fdf0 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20241984f0 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 10 client.41330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020003a60 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20241a4810 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.083+0000 7f202c572700 1 librados: init done 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024103570 con 0x7f2024069d70 2024-03-11T21:31:50.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f202000f970 con 0x7f2024069d70 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 0x7f2010071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(308..308 src has 87..308) v4 ==== 10185+0+0 (secure 0 0 0) 0x7f202008fec0 con 0x7f2024069d70 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 10 client.41330.objecter ms_dispatch 0x7f2024001550 osd_map(308..308 src has 87..308) v4 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 3 client.41330.objecter handle_osd_map got epochs [308,308] > 0 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 3 client.41330.objecter handle_osd_map decoding full epoch 308 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f2029b0d700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 0x7f2010071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:31:50.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 20 client.41330.objecter dump_active .. 0 homeless 2024-03-11T21:31:50.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f201b7fe700 10 client.41330.objecter ms_handle_connect 0x7f201006f6f0 2024-03-11T21:31:50.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.087+0000 7f2029b0d700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 0x7f2010071bb0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f2024108630 tx=0x7f2014009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:31:50.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.091+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2020013070 con 0x7f2024069d70 2024-03-11T21:31:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.167+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f1e081a6c60 con 0x7f1e08101850 2024-03-11T21:31:50.153 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.171 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:50.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.311+0000 7f1e05ffb700 1 -- 172.21.15.143:0/409860231 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v309) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1dfc05c3b0 con 0x7f1e08101850 2024-03-11T21:31:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.315+0000 7f1e0eac3700 10 librados: watch_flush enter 2024-03-11T21:31:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.315+0000 7f1e0eac3700 10 librados: watch_flush exit 2024-03-11T21:31:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.315+0000 7f1e0eac3700 20 client.41324.objecter shutdown clearing up homeless session... 2024-03-11T21:31:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.315+0000 7f1e0eac3700 10 client.41324.objecter successfully canceled tick 2024-03-11T21:31:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 msgr2=0x7f1df8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 0x7f1df8071ce0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f1df400b580 tx=0x7f1df4019040 comp rx=0 tx=0).stop 2024-03-11T21:31:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 msgr2=0x7f1e08199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 0x7f1e08199e50 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00e9f0 tx=0x7f1dfc00ed00 comp rx=0 tx=0).stop 2024-03-11T21:31:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 shutdown_connections 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1df806f820 0x7f1df8071ce0 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e0810e8f0 0x7f1e0819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e08101850 0x7f1e08199e50 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 --2- 172.21.15.143:0/409860231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e08102170 0x7f1e0819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 >> 172.21.15.143:0/409860231 conn(0x7f1e08076d80 msgr2=0x7f1e0810fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 shutdown_connections 2024-03-11T21:31:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f1e0eac3700 1 -- 172.21.15.143:0/409860231 wait complete. 2024-03-11T21:31:50.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.323+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f202404faa0 con 0x7f2024069d70 2024-03-11T21:31:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:50.327+0000 7f1e0eac3700 1 librados: shutdown 2024-03-11T21:31:50.301 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:31:50.306 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:50.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:50.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:50.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.565 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:50.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:50.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:50.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:50.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: cluster 2024-03-11T21:31:48.829547+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:50.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: cluster 2024-03-11T21:31:48.829890+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:50.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.301214+0000 mon.a (mon.0) 1968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.301303+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.301375+0000 mon.a (mon.0) 1970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: cluster 2024-03-11T21:31:49.306204+0000 mon.a (mon.0) 1971 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.308313+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.308581+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: cluster 2024-03-11T21:31:49.309960+0000 mgr.y (mgr.14550) 357 : cluster 0 pgmap v450: 290 pgs: 1 active+clean+snaptrim_wait, 3 unknown, 4 active+clean+snaptrim, 282 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.315187+0000 mon.c (mon.2) 582 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.315484+0000 mon.c (mon.2) 583 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.316089+0000 mon.a (mon.0) 1973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.317094+0000 mon.a (mon.0) 1974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.317297+0000 mon.a (mon.0) 1975 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:50.808 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.640691+0000 mon.a (mon.0) 1976 : audit 0 from='client.? 172.21.15.143:0/3431713019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: cluster 2024-03-11T21:31:48.829547+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: cluster 2024-03-11T21:31:48.829890+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.301214+0000 mon.a (mon.0) 1968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.301303+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.301375+0000 mon.a (mon.0) 1970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: cluster 2024-03-11T21:31:49.306204+0000 mon.a (mon.0) 1971 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.308313+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.308581+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: cluster 2024-03-11T21:31:49.309960+0000 mgr.y (mgr.14550) 357 : cluster 0 pgmap v450: 290 pgs: 1 active+clean+snaptrim_wait, 3 unknown, 4 active+clean+snaptrim, 282 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.315187+0000 mon.c (mon.2) 582 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.315484+0000 mon.c (mon.2) 583 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.316089+0000 mon.a (mon.0) 1973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:50.809 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.317094+0000 mon.a (mon.0) 1974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.317297+0000 mon.a (mon.0) 1975 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.640691+0000 mon.a (mon.0) 1976 : audit 0 from='client.? 172.21.15.143:0/3431713019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:49.994213+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:50.169984+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.143:0/409860231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:50.170360+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:50.810 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[21512]: audit 2024-03-11T21:31:50.304460+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:50.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:49.994213+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:50.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:50.169984+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.143:0/409860231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:50.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:50.170360+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:50.811 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:50 smithi143 bash[29284]: audit 2024-03-11T21:31:50.304460+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:50.819 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:50.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:50.944 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: cluster 2024-03-11T21:31:48.829547+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: cluster 2024-03-11T21:31:48.829890+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.301214+0000 mon.a (mon.0) 1968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.301303+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.301375+0000 mon.a (mon.0) 1970 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: cluster 2024-03-11T21:31:49.306204+0000 mon.a (mon.0) 1971 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.308313+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.308581+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.143:0/3426972044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: cluster 2024-03-11T21:31:49.309960+0000 mgr.y (mgr.14550) 357 : cluster 0 pgmap v450: 290 pgs: 1 active+clean+snaptrim_wait, 3 unknown, 4 active+clean+snaptrim, 282 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.315187+0000 mon.c (mon.2) 582 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.315484+0000 mon.c (mon.2) 583 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.316089+0000 mon.a (mon.0) 1973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.317094+0000 mon.a (mon.0) 1974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.317297+0000 mon.a (mon.0) 1975 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T21:31:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.640691+0000 mon.a (mon.0) 1976 : audit 0 from='client.? 172.21.15.143:0/3431713019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:49.994213+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:50.169984+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.143:0/409860231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:50.170360+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:31:51.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:50 smithi203 bash[23654]: audit 2024-03-11T21:31:50.304460+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-46"}]': finished 2024-03-11T21:31:51.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:51.069 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:51.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:51.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:51.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.259+0000 7f201b7fe700 1 -- 172.21.15.143:0/2803841966 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v310) v1 ==== 91+0+0 (secure 0 0 0) 0x7f202005c960 con 0x7f2024069d70 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 10 librados: watch_flush enter 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 10 librados: watch_flush exit 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 20 client.41330.objecter shutdown clearing up homeless session... 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 10 client.41330.objecter successfully canceled tick 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 msgr2=0x7f2010071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 0x7f2010071bb0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f2024108630 tx=0x7f2014009260 comp rx=0 tx=0).stop 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 msgr2=0x7f202419d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202419d940 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f20200099c0 tx=0x7f202000f690 comp rx=0 tx=0).stop 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 shutdown_connections 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f201006f6f0 0x7f2010071bb0 secure :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f2024108630 tx=0x7f2014009260 comp rx=0 tx=0).stop 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241071c0 0x7f2024197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2024069d70 0x7f202419d940 secure :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f20200099c0 tx=0x7f202000f690 comp rx=0 tx=0).stop 2024-03-11T21:31:51.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 --2- 172.21.15.143:0/2803841966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202406a720 0x7f202419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:31:51.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.263+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 >> 172.21.15.143:0/2803841966 conn(0x7f2024076d80 msgr2=0x7f20240febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:31:51.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.275+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 shutdown_connections 2024-03-11T21:31:51.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.275+0000 7f202c572700 1 -- 172.21.15.143:0/2803841966 wait complete. 2024-03-11T21:31:51.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:31:51.275+0000 7f202c572700 1 librados: shutdown 2024-03-11T21:31:51.252 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:31:51.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:51.324 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:51.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:51.452 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:51.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:51.580 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:51.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:51.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:51.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:51.841 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:51.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:51.974 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.304642+0000 mon.a (mon.0) 1979 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:52.051 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.304784+0000 mon.a (mon.0) 1980 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.304917+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.315866+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.332219+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.143:0/2803841966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.344536+0000 mon.c (mon.2) 588 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: cluster 2024-03-11T21:31:50.734099+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.736617+0000 mon.a (mon.0) 1983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.742050+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.749057+0000 mon.a (mon.0) 1985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.749899+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.750235+0000 mon.a (mon.0) 1986 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.750898+0000 mon.c (mon.2) 590 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.751223+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:50.995217+0000 mon.c (mon.2) 591 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:52.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.045497+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.047570+0000 mon.c (mon.2) 592 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.251330+0000 mon.a (mon.0) 1989 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.251407+0000 mon.a (mon.0) 1990 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.254875+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.255067+0000 mon.a (mon.0) 1992 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]': finished 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.255234+0000 mon.a (mon.0) 1993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.255355+0000 mon.a (mon.0) 1994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.266607+0000 mon.a (mon.0) 1995 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.269781+0000 mon.c (mon.2) 593 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[29284]: audit 2024-03-11T21:31:51.274234+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.304642+0000 mon.a (mon.0) 1979 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.304784+0000 mon.a (mon.0) 1980 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.304917+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.315866+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.332219+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.143:0/2803841966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.344536+0000 mon.c (mon.2) 588 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: cluster 2024-03-11T21:31:50.734099+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.736617+0000 mon.a (mon.0) 1983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.742050+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.749057+0000 mon.a (mon.0) 1985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.749899+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.750235+0000 mon.a (mon.0) 1986 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.750898+0000 mon.c (mon.2) 590 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.751223+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:50.995217+0000 mon.c (mon.2) 591 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.045497+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.047570+0000 mon.c (mon.2) 592 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.251330+0000 mon.a (mon.0) 1989 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.251407+0000 mon.a (mon.0) 1990 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.254875+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.255067+0000 mon.a (mon.0) 1992 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]': finished 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.255234+0000 mon.a (mon.0) 1993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.255355+0000 mon.a (mon.0) 1994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.266607+0000 mon.a (mon.0) 1995 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.269781+0000 mon.c (mon.2) 593 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:51 smithi143 bash[21512]: audit 2024-03-11T21:31:51.274234+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:52.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:31:52.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.304642+0000 mon.a (mon.0) 1979 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.304784+0000 mon.a (mon.0) 1980 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.304917+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.315866+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.332219+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.143:0/2803841966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.344536+0000 mon.c (mon.2) 588 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: cluster 2024-03-11T21:31:50.734099+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:31:52.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.736617+0000 mon.a (mon.0) 1983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.742050+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.749057+0000 mon.a (mon.0) 1985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.749899+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.750235+0000 mon.a (mon.0) 1986 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.750898+0000 mon.c (mon.2) 590 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.751223+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:50.995217+0000 mon.c (mon.2) 591 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.045497+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.047570+0000 mon.c (mon.2) 592 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.251330+0000 mon.a (mon.0) 1989 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.251407+0000 mon.a (mon.0) 1990 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.254875+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi143-71594-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.255067+0000 mon.a (mon.0) 1992 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-27", "mode": "writeback"}]': finished 2024-03-11T21:31:52.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.255234+0000 mon.a (mon.0) 1993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:52.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.255355+0000 mon.a (mon.0) 1994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi143-71604-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:52.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.266607+0000 mon.a (mon.0) 1995 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:31:52.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.269781+0000 mon.c (mon.2) 593 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:51 smithi203 bash[23654]: audit 2024-03-11T21:31:51.274234+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:52.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:52.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:52.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:52.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.423 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:31:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:52.436 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:52.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:52.566 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:52.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:52.697 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:52.801 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:52.825 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:52.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.310501+0000 mgr.y (mgr.14550) 358 : cluster 0 pgmap v453: 298 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 285 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.385269+0000 osd.7 (osd.7) 233 : cluster 0 118.1e scrub starts 2024-03-11T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.388395+0000 osd.7 (osd.7) 234 : cluster 0 118.1e scrub ok 2024-03-11T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.832064+0000 osd.5 (osd.5) 203 : cluster 0 9.6 scrub starts 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:51.834241+0000 osd.5 (osd.5) 204 : cluster 0 9.6 scrub ok 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: audit 2024-03-11T21:31:51.996481+0000 mon.c (mon.2) 594 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:52.727394+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[29284]: cluster 2024-03-11T21:31:52.727729+0000 mon.a (mon.0) 1998 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.310501+0000 mgr.y (mgr.14550) 358 : cluster 0 pgmap v453: 298 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 285 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.385269+0000 osd.7 (osd.7) 233 : cluster 0 118.1e scrub starts 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.388395+0000 osd.7 (osd.7) 234 : cluster 0 118.1e scrub ok 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.832064+0000 osd.5 (osd.5) 203 : cluster 0 9.6 scrub starts 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:51.834241+0000 osd.5 (osd.5) 204 : cluster 0 9.6 scrub ok 2024-03-11T21:31:52.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: audit 2024-03-11T21:31:51.996481+0000 mon.c (mon.2) 594 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:52.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:52.727394+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:31:52.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:52 smithi143 bash[21512]: cluster 2024-03-11T21:31:52.727729+0000 mon.a (mon.0) 1998 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:52.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.310501+0000 mgr.y (mgr.14550) 358 : cluster 0 pgmap v453: 298 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 285 active+clean; 586 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.385269+0000 osd.7 (osd.7) 233 : cluster 0 118.1e scrub starts 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.388395+0000 osd.7 (osd.7) 234 : cluster 0 118.1e scrub ok 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.832064+0000 osd.5 (osd.5) 203 : cluster 0 9.6 scrub starts 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:51.834241+0000 osd.5 (osd.5) 204 : cluster 0 9.6 scrub ok 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: audit 2024-03-11T21:31:51.996481+0000 mon.c (mon.2) 594 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:52.727394+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:31:53.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:52 smithi203 bash[23654]: cluster 2024-03-11T21:31:52.727729+0000 mon.a (mon.0) 1998 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:53.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:53.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:53.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:31:53.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:53.207 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:53.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.330 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:31:53.330 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:31:53.331 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:31:53.331 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:31:53.331 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:31:53.332 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:53.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:53.456 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:53.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:53.584 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:53.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:53.713 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: cluster 2024-03-11T21:31:52.558256+0000 osd.6 (osd.6) 145 : cluster 0 12.15 scrub starts 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: cluster 2024-03-11T21:31:52.560755+0000 osd.6 (osd.6) 146 : cluster 0 12.15 scrub ok 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:52.744594+0000 mon.c (mon.2) 595 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:52.744990+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:52.997833+0000 mon.c (mon.2) 596 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.262606+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.262737+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.267415+0000 mon.a (mon.0) 2002 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:31:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.269528+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.270027+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.272972+0000 mon.a (mon.0) 2004 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.311990+0000 mon.c (mon.2) 598 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:53 smithi203 bash[23654]: audit 2024-03-11T21:31:53.314955+0000 mon.a (mon.0) 2005 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.817 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:53.829 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: cluster 2024-03-11T21:31:52.558256+0000 osd.6 (osd.6) 145 : cluster 0 12.15 scrub starts 2024-03-11T21:31:53.829 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: cluster 2024-03-11T21:31:52.560755+0000 osd.6 (osd.6) 146 : cluster 0 12.15 scrub ok 2024-03-11T21:31:53.829 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:52.744594+0000 mon.c (mon.2) 595 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.829 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:52.744990+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:52.997833+0000 mon.c (mon.2) 596 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.262606+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.262737+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.267415+0000 mon.a (mon.0) 2002 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.269528+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.270027+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.272972+0000 mon.a (mon.0) 2004 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.311990+0000 mon.c (mon.2) 598 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[21512]: audit 2024-03-11T21:31:53.314955+0000 mon.a (mon.0) 2005 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: cluster 2024-03-11T21:31:52.558256+0000 osd.6 (osd.6) 145 : cluster 0 12.15 scrub starts 2024-03-11T21:31:53.830 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: cluster 2024-03-11T21:31:52.560755+0000 osd.6 (osd.6) 146 : cluster 0 12.15 scrub ok 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:52.744594+0000 mon.c (mon.2) 595 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:52.744990+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:52.997833+0000 mon.c (mon.2) 596 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.262606+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi143-71604-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.262737+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.267415+0000 mon.a (mon.0) 2002 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.269528+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.270027+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.272972+0000 mon.a (mon.0) 2004 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.311990+0000 mon.c (mon.2) 598 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:53 smithi143 bash[29284]: audit 2024-03-11T21:31:53.314955+0000 mon.a (mon.0) 2005 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T21:31:53.844 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:53.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.973 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:54.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:54.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:31:54.105 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:54.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:54.231 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:54.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:54.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:54.484 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:54.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:54.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:31:54.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.744 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:54.847 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:54.869 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:54.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.093617+0000 osd.1 (osd.1) 1 : cluster 0 12.f scrub starts 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.097171+0000 osd.1 (osd.1) 2 : cluster 0 12.f scrub ok 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.311371+0000 mgr.y (mgr.14550) 359 : cluster 0 pgmap v456: 297 pgs: 1 peering, 8 unknown, 9 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 275 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.403186+0000 osd.7 (osd.7) 235 : cluster 0 9.e scrub starts 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.404318+0000 osd.7 (osd.7) 236 : cluster 0 9.e scrub ok 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.543786+0000 osd.6 (osd.6) 147 : cluster 0 12.4 scrub starts 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.545520+0000 osd.6 (osd.6) 148 : cluster 0 12.4 scrub ok 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: audit 2024-03-11T21:31:53.998777+0000 mon.c (mon.2) 599 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.263993+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: audit 2024-03-11T21:31:54.271913+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:54.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: audit 2024-03-11T21:31:54.272152+0000 mon.a (mon.0) 2008 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:54.988 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: audit 2024-03-11T21:31:54.272306+0000 mon.a (mon.0) 2009 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T21:31:54.988 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.279686+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:31:54.988 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:54 smithi203 bash[23654]: audit 2024-03-11T21:31:54.283620+0000 mon.a (mon.0) 2011 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:55.002 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:55.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.093617+0000 osd.1 (osd.1) 1 : cluster 0 12.f scrub starts 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.097171+0000 osd.1 (osd.1) 2 : cluster 0 12.f scrub ok 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.311371+0000 mgr.y (mgr.14550) 359 : cluster 0 pgmap v456: 297 pgs: 1 peering, 8 unknown, 9 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 275 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.403186+0000 osd.7 (osd.7) 235 : cluster 0 9.e scrub starts 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.404318+0000 osd.7 (osd.7) 236 : cluster 0 9.e scrub ok 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.543786+0000 osd.6 (osd.6) 147 : cluster 0 12.4 scrub starts 2024-03-11T21:31:55.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.545520+0000 osd.6 (osd.6) 148 : cluster 0 12.4 scrub ok 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: audit 2024-03-11T21:31:53.998777+0000 mon.c (mon.2) 599 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.263993+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: audit 2024-03-11T21:31:54.271913+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: audit 2024-03-11T21:31:54.272152+0000 mon.a (mon.0) 2008 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: audit 2024-03-11T21:31:54.272306+0000 mon.a (mon.0) 2009 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.279686+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[21512]: audit 2024-03-11T21:31:54.283620+0000 mon.a (mon.0) 2011 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:55.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.093617+0000 osd.1 (osd.1) 1 : cluster 0 12.f scrub starts 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.097171+0000 osd.1 (osd.1) 2 : cluster 0 12.f scrub ok 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.311371+0000 mgr.y (mgr.14550) 359 : cluster 0 pgmap v456: 297 pgs: 1 peering, 8 unknown, 9 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 275 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.403186+0000 osd.7 (osd.7) 235 : cluster 0 9.e scrub starts 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.404318+0000 osd.7 (osd.7) 236 : cluster 0 9.e scrub ok 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.543786+0000 osd.6 (osd.6) 147 : cluster 0 12.4 scrub starts 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.545520+0000 osd.6 (osd.6) 148 : cluster 0 12.4 scrub ok 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: audit 2024-03-11T21:31:53.998777+0000 mon.c (mon.2) 599 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.263993+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: audit 2024-03-11T21:31:54.271913+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-27"}]': finished 2024-03-11T21:31:55.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: audit 2024-03-11T21:31:54.272152+0000 mon.a (mon.0) 2008 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:55.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: audit 2024-03-11T21:31:54.272306+0000 mon.a (mon.0) 2009 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T21:31:55.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.279686+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:31:55.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:54 smithi143 bash[29284]: audit 2024-03-11T21:31:54.283620+0000 mon.a (mon.0) 2011 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]: dispatch 2024-03-11T21:31:55.130 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:55.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:55.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:55.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:55.397 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:55.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:55.531 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:55.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:55.663 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:55.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.901082+0000 osd.2 (osd.2) 157 : cluster 0 118.5 scrub starts 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:53.903200+0000 osd.2 (osd.2) 158 : cluster 0 118.5 scrub ok 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.448727+0000 osd.7 (osd.7) 237 : cluster 0 9.d scrub starts 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.451217+0000 osd.7 (osd.7) 238 : cluster 0 9.d scrub ok 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.546470+0000 osd.3 (osd.3) 201 : cluster 0 3.18 scrub starts 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.548269+0000 osd.3 (osd.3) 202 : cluster 0 3.18 scrub ok 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.588968+0000 osd.6 (osd.6) 149 : cluster 0 3.a scrub starts 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:54.594106+0000 osd.6 (osd.6) 150 : cluster 0 3.a scrub ok 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: audit 2024-03-11T21:31:55.000110+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: audit 2024-03-11T21:31:55.275472+0000 mon.a (mon.0) 2012 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.282684+0000 mon.a (mon.0) 2013 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:31:55.776 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[21512]: audit 2024-03-11T21:31:55.292721+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.901082+0000 osd.2 (osd.2) 157 : cluster 0 118.5 scrub starts 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:53.903200+0000 osd.2 (osd.2) 158 : cluster 0 118.5 scrub ok 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.448727+0000 osd.7 (osd.7) 237 : cluster 0 9.d scrub starts 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.451217+0000 osd.7 (osd.7) 238 : cluster 0 9.d scrub ok 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.546470+0000 osd.3 (osd.3) 201 : cluster 0 3.18 scrub starts 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.548269+0000 osd.3 (osd.3) 202 : cluster 0 3.18 scrub ok 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.588968+0000 osd.6 (osd.6) 149 : cluster 0 3.a scrub starts 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:54.594106+0000 osd.6 (osd.6) 150 : cluster 0 3.a scrub ok 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: audit 2024-03-11T21:31:55.000110+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: audit 2024-03-11T21:31:55.275472+0000 mon.a (mon.0) 2012 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:55.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.282684+0000 mon.a (mon.0) 2013 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:31:55.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:55 smithi143 bash[29284]: audit 2024-03-11T21:31:55.292721+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:55.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:55.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.901082+0000 osd.2 (osd.2) 157 : cluster 0 118.5 scrub starts 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:53.903200+0000 osd.2 (osd.2) 158 : cluster 0 118.5 scrub ok 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.448727+0000 osd.7 (osd.7) 237 : cluster 0 9.d scrub starts 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.451217+0000 osd.7 (osd.7) 238 : cluster 0 9.d scrub ok 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.546470+0000 osd.3 (osd.3) 201 : cluster 0 3.18 scrub starts 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.548269+0000 osd.3 (osd.3) 202 : cluster 0 3.18 scrub ok 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.588968+0000 osd.6 (osd.6) 149 : cluster 0 3.a scrub starts 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:54.594106+0000 osd.6 (osd.6) 150 : cluster 0 3.a scrub ok 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: audit 2024-03-11T21:31:55.000110+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: audit 2024-03-11T21:31:55.275472+0000 mon.a (mon.0) 2012 : audit 1 from='client.? 172.21.15.143:0/3824385048' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi143-71594-37"}]': finished 2024-03-11T21:31:55.901 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.282684+0000 mon.a (mon.0) 2013 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:31:55.902 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:55 smithi203 bash[23654]: audit 2024-03-11T21:31:55.292721+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:55.917 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:56.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:56.047 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:56.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.177 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:56.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:56.406 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:56.431 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:56.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:56.562 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:56.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:56.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:56.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.092489+0000 osd.1 (osd.1) 3 : cluster 0 9.0 scrub starts 2024-03-11T21:31:56.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.095098+0000 osd.1 (osd.1) 4 : cluster 0 9.0 scrub ok 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.312632+0000 mgr.y (mgr.14550) 360 : cluster 0 pgmap v459: 257 pgs: 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 249 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.315024+0000 mon.b (mon.1) 89 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.759583+0000 mon.a (mon.0) 2014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.760281+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.765030+0000 mon.b (mon.1) 90 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.766618+0000 mon.b (mon.1) 91 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.768680+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:55.770432+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.941997+0000 osd.5 (osd.5) 205 : cluster 0 9.f scrub starts 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:55.943477+0000 osd.5 (osd.5) 206 : cluster 0 9.f scrub ok 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.001385+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.259330+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.259529+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:56.809 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: cluster 2024-03-11T21:31:56.267383+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.268621+0000 mon.b (mon.1) 92 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.271616+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.273002+0000 mon.c (mon.2) 604 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.290490+0000 mon.a (mon.0) 2021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.291930+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:56.810 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:56 smithi203 bash[23654]: audit 2024-03-11T21:31:56.293341+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:56.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:56.918 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:56.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:56.951 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:57.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:57.081 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:57.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:57.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.092489+0000 osd.1 (osd.1) 3 : cluster 0 9.0 scrub starts 2024-03-11T21:31:57.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.095098+0000 osd.1 (osd.1) 4 : cluster 0 9.0 scrub ok 2024-03-11T21:31:57.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.312632+0000 mgr.y (mgr.14550) 360 : cluster 0 pgmap v459: 257 pgs: 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 249 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:31:57.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.315024+0000 mon.b (mon.1) 89 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.759583+0000 mon.a (mon.0) 2014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.760281+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.765030+0000 mon.b (mon.1) 90 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.766618+0000 mon.b (mon.1) 91 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.768680+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:55.770432+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.941997+0000 osd.5 (osd.5) 205 : cluster 0 9.f scrub starts 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:55.943477+0000 osd.5 (osd.5) 206 : cluster 0 9.f scrub ok 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.001385+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.259330+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.259529+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:57.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: cluster 2024-03-11T21:31:56.267383+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.268621+0000 mon.b (mon.1) 92 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.092489+0000 osd.1 (osd.1) 3 : cluster 0 9.0 scrub starts 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.095098+0000 osd.1 (osd.1) 4 : cluster 0 9.0 scrub ok 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.312632+0000 mgr.y (mgr.14550) 360 : cluster 0 pgmap v459: 257 pgs: 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 249 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.315024+0000 mon.b (mon.1) 89 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.759583+0000 mon.a (mon.0) 2014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.760281+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.765030+0000 mon.b (mon.1) 90 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.766618+0000 mon.b (mon.1) 91 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.768680+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:55.770432+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.941997+0000 osd.5 (osd.5) 205 : cluster 0 9.f scrub starts 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:55.943477+0000 osd.5 (osd.5) 206 : cluster 0 9.f scrub ok 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.001385+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.259330+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.259529+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi143-71594-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: cluster 2024-03-11T21:31:56.267383+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:31:57.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.268621+0000 mon.b (mon.1) 92 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.271616+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.273002+0000 mon.c (mon.2) 604 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:57.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.290490+0000 mon.a (mon.0) 2021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.291930+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:57.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[21512]: audit 2024-03-11T21:31:56.293341+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.271616+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.143:0/1928853474' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.273002+0000 mon.c (mon.2) 604 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:57.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.290490+0000 mon.a (mon.0) 2021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]: dispatch 2024-03-11T21:31:57.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.291930+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:57.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:56 smithi143 bash[29284]: audit 2024-03-11T21:31:56.293341+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:31:57.222 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:57.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:57.345 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:57.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.481 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:57.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.613 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:57.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:57.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: cluster 2024-03-11T21:31:56.388737+0000 osd.7 (osd.7) 239 : cluster 0 9.3 scrub starts 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: cluster 2024-03-11T21:31:56.390960+0000 osd.7 (osd.7) 240 : cluster 0 9.3 scrub ok 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.003048+0000 mon.c (mon.2) 605 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.261750+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.261840+0000 mon.a (mon.0) 2025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.311220+0000 mon.c (mon.2) 606 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.313811+0000 mon.c (mon.2) 607 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.710467+0000 mon.a (mon.0) 2026 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.712644+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.713993+0000 mon.a (mon.0) 2028 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.715076+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.715667+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.716346+0000 mon.c (mon.2) 609 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[29284]: audit 2024-03-11T21:31:57.716743+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: cluster 2024-03-11T21:31:56.388737+0000 osd.7 (osd.7) 239 : cluster 0 9.3 scrub starts 2024-03-11T21:31:57.857 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: cluster 2024-03-11T21:31:56.390960+0000 osd.7 (osd.7) 240 : cluster 0 9.3 scrub ok 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.003048+0000 mon.c (mon.2) 605 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.261750+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.261840+0000 mon.a (mon.0) 2025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.311220+0000 mon.c (mon.2) 606 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.313811+0000 mon.c (mon.2) 607 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.710467+0000 mon.a (mon.0) 2026 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.712644+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.713993+0000 mon.a (mon.0) 2028 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.715076+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.715667+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.716346+0000 mon.c (mon.2) 609 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:57 smithi143 bash[21512]: audit 2024-03-11T21:31:57.716743+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.871 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:57.936 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:57.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:57.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: cluster 2024-03-11T21:31:56.388737+0000 osd.7 (osd.7) 239 : cluster 0 9.3 scrub starts 2024-03-11T21:31:57.985 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: cluster 2024-03-11T21:31:56.390960+0000 osd.7 (osd.7) 240 : cluster 0 9.3 scrub ok 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.003048+0000 mon.c (mon.2) 605 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.261750+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi143-71604-47"}]': finished 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.261840+0000 mon.a (mon.0) 2025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.311220+0000 mon.c (mon.2) 606 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.313811+0000 mon.c (mon.2) 607 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.710467+0000 mon.a (mon.0) 2026 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.712644+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.713993+0000 mon.a (mon.0) 2028 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.715076+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.715667+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:57.986 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.716346+0000 mon.c (mon.2) 609 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:57.987 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:57 smithi203 bash[23654]: audit 2024-03-11T21:31:57.716743+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:31:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:58.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:58.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:58.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:58.397 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:58.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:58.527 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:58.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:58.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:31:58.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:58.786 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:58.853 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:31:58.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.313371+0000 mgr.y (mgr.14550) 361 : cluster 0 pgmap v462: 288 pgs: 32 unknown, 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 248 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.631408+0000 osd.3 (osd.3) 203 : cluster 0 9.19 scrub starts 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.633425+0000 osd.3 (osd.3) 204 : cluster 0 9.19 scrub ok 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.947340+0000 osd.5 (osd.5) 207 : cluster 0 9.a scrub starts 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:57.949760+0000 osd.5 (osd.5) 208 : cluster 0 9.a scrub ok 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.004042+0000 mon.c (mon.2) 610 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.033653+0000 osd.1 (osd.1) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.034216+0000 osd.1 (osd.1) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.273268+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:31:58.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.273521+0000 mon.a (mon.0) 2032 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.273715+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.283263+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.284109+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: audit 2024-03-11T21:31:58.286614+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.670468+0000 osd.4 (osd.4) 181 : cluster 0 5.18 scrub starts 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.313371+0000 mgr.y (mgr.14550) 361 : cluster 0 pgmap v462: 288 pgs: 32 unknown, 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 248 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.631408+0000 osd.3 (osd.3) 203 : cluster 0 9.19 scrub starts 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.633425+0000 osd.3 (osd.3) 204 : cluster 0 9.19 scrub ok 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.947340+0000 osd.5 (osd.5) 207 : cluster 0 9.a scrub starts 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:57.949760+0000 osd.5 (osd.5) 208 : cluster 0 9.a scrub ok 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.004042+0000 mon.c (mon.2) 610 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:58.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.033653+0000 osd.1 (osd.1) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.034216+0000 osd.1 (osd.1) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.273268+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.273521+0000 mon.a (mon.0) 2032 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.273715+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.283263+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.284109+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: audit 2024-03-11T21:31:58.286614+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:58.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:58 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.670468+0000 osd.4 (osd.4) 181 : cluster 0 5.18 scrub starts 2024-03-11T21:31:58.917 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:58.957 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:59.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.313371+0000 mgr.y (mgr.14550) 361 : cluster 0 pgmap v462: 288 pgs: 32 unknown, 1 peering, 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 248 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.631408+0000 osd.3 (osd.3) 203 : cluster 0 9.19 scrub starts 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.633425+0000 osd.3 (osd.3) 204 : cluster 0 9.19 scrub ok 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.947340+0000 osd.5 (osd.5) 207 : cluster 0 9.a scrub starts 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:57.949760+0000 osd.5 (osd.5) 208 : cluster 0 9.a scrub ok 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.004042+0000 mon.c (mon.2) 610 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.033653+0000 osd.1 (osd.1) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.034216+0000 osd.1 (osd.1) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.273268+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi143-71594-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.273521+0000 mon.a (mon.0) 2032 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.273715+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi143-71604-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.283263+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.284109+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:31:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: audit 2024-03-11T21:31:58.286614+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:31:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:58 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.670468+0000 osd.4 (osd.4) 181 : cluster 0 5.18 scrub starts 2024-03-11T21:31:59.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:59.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.174 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:59.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:59.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:59.429 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:59.532 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:59.557 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:59.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:59.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.816 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:59.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.090520+0000 osd.1 (osd.1) 7 : cluster 0 9.4 scrub starts 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.092731+0000 osd.1 (osd.1) 8 : cluster 0 9.4 scrub ok 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: audit 2024-03-11T21:31:58.492002+0000 mgr.y (mgr.14550) 362 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.623811+0000 osd.6 (osd.6) 151 : cluster 0 12.14 scrub starts 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.626000+0000 osd.6 (osd.6) 152 : cluster 0 12.14 scrub ok 2024-03-11T21:32:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.672786+0000 osd.4 (osd.4) 182 : cluster 0 5.18 scrub ok 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: audit 2024-03-11T21:31:59.004956+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.043312+0000 osd.1 (osd.1) 9 : cluster 0 12.17 scrub starts 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.045210+0000 osd.1 (osd.1) 10 : cluster 0 12.17 scrub ok 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.742791+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: audit 2024-03-11T21:31:59.746699+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:31:59 smithi203 bash[23654]: audit 2024-03-11T21:31:59.748370+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.090520+0000 osd.1 (osd.1) 7 : cluster 0 9.4 scrub starts 2024-03-11T21:32:00.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.092731+0000 osd.1 (osd.1) 8 : cluster 0 9.4 scrub ok 2024-03-11T21:32:00.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: audit 2024-03-11T21:31:58.492002+0000 mgr.y (mgr.14550) 362 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:00.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.623811+0000 osd.6 (osd.6) 151 : cluster 0 12.14 scrub starts 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.626000+0000 osd.6 (osd.6) 152 : cluster 0 12.14 scrub ok 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.672786+0000 osd.4 (osd.4) 182 : cluster 0 5.18 scrub ok 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: audit 2024-03-11T21:31:59.004956+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.043312+0000 osd.1 (osd.1) 9 : cluster 0 12.17 scrub starts 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.045210+0000 osd.1 (osd.1) 10 : cluster 0 12.17 scrub ok 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.742791+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: audit 2024-03-11T21:31:59.746699+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[21512]: audit 2024-03-11T21:31:59.748370+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.090520+0000 osd.1 (osd.1) 7 : cluster 0 9.4 scrub starts 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.092731+0000 osd.1 (osd.1) 8 : cluster 0 9.4 scrub ok 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: audit 2024-03-11T21:31:58.492002+0000 mgr.y (mgr.14550) 362 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:00.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.623811+0000 osd.6 (osd.6) 151 : cluster 0 12.14 scrub starts 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.626000+0000 osd.6 (osd.6) 152 : cluster 0 12.14 scrub ok 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.672786+0000 osd.4 (osd.4) 182 : cluster 0 5.18 scrub ok 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: audit 2024-03-11T21:31:59.004956+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.043312+0000 osd.1 (osd.1) 9 : cluster 0 12.17 scrub starts 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.045210+0000 osd.1 (osd.1) 10 : cluster 0 12.17 scrub ok 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.742791+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: audit 2024-03-11T21:31:59.746699+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:31:59 smithi143 bash[29284]: audit 2024-03-11T21:31:59.748370+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:00.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:00.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:00.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:00.209 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:00.312 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:00.332 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:00.435 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:00.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:00.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:00.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:00.723 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:00.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:00.849 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:00.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:00.974 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.975026+0000 osd.0 (osd.0) 197 : cluster 0 3.6 scrub starts 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:58.977536+0000 osd.0 (osd.0) 198 : cluster 0 3.6 scrub ok 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.314314+0000 mgr.y (mgr.14550) 363 : cluster 0 pgmap v465: 296 pgs: 2 peering, 12 unknown, 282 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.449427+0000 osd.7 (osd.7) 241 : cluster 0 9.17 scrub starts 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.451342+0000 osd.7 (osd.7) 242 : cluster 0 9.17 scrub ok 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.587586+0000 osd.6 (osd.6) 153 : cluster 0 2.11 scrub starts 2024-03-11T21:32:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.593020+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub ok 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.701561+0000 osd.4 (osd.4) 183 : cluster 0 12.10 scrub starts 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.709778+0000 osd.4 (osd.4) 184 : cluster 0 12.10 scrub ok 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.992613+0000 osd.5 (osd.5) 209 : cluster 0 3.16 scrub starts 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:31:59.996233+0000 osd.5 (osd.5) 210 : cluster 0 3.16 scrub ok 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.006003+0000 mon.c (mon.2) 614 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:32:00.276910+0000 mon.a (mon.0) 2038 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.285743+0000 mon.a (mon.0) 2039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.285877+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.290481+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: cluster 2024-03-11T21:32:00.290875+0000 mon.a (mon.0) 2041 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:32:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.292783+0000 mon.b (mon.1) 93 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.294155+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:00 smithi203 bash[23654]: audit 2024-03-11T21:32:00.300638+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:01.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.975026+0000 osd.0 (osd.0) 197 : cluster 0 3.6 scrub starts 2024-03-11T21:32:01.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:58.977536+0000 osd.0 (osd.0) 198 : cluster 0 3.6 scrub ok 2024-03-11T21:32:01.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.314314+0000 mgr.y (mgr.14550) 363 : cluster 0 pgmap v465: 296 pgs: 2 peering, 12 unknown, 282 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T21:32:01.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.449427+0000 osd.7 (osd.7) 241 : cluster 0 9.17 scrub starts 2024-03-11T21:32:01.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.451342+0000 osd.7 (osd.7) 242 : cluster 0 9.17 scrub ok 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.587586+0000 osd.6 (osd.6) 153 : cluster 0 2.11 scrub starts 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.593020+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub ok 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.701561+0000 osd.4 (osd.4) 183 : cluster 0 12.10 scrub starts 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.709778+0000 osd.4 (osd.4) 184 : cluster 0 12.10 scrub ok 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.992613+0000 osd.5 (osd.5) 209 : cluster 0 3.16 scrub starts 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:31:59.996233+0000 osd.5 (osd.5) 210 : cluster 0 3.16 scrub ok 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.006003+0000 mon.c (mon.2) 614 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:32:00.276910+0000 mon.a (mon.0) 2038 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.285743+0000 mon.a (mon.0) 2039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.285877+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.290481+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: cluster 2024-03-11T21:32:00.290875+0000 mon.a (mon.0) 2041 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.292783+0000 mon.b (mon.1) 93 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.294155+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[21512]: audit 2024-03-11T21:32:00.300638+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.975026+0000 osd.0 (osd.0) 197 : cluster 0 3.6 scrub starts 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:58.977536+0000 osd.0 (osd.0) 198 : cluster 0 3.6 scrub ok 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.314314+0000 mgr.y (mgr.14550) 363 : cluster 0 pgmap v465: 296 pgs: 2 peering, 12 unknown, 282 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.449427+0000 osd.7 (osd.7) 241 : cluster 0 9.17 scrub starts 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.451342+0000 osd.7 (osd.7) 242 : cluster 0 9.17 scrub ok 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.587586+0000 osd.6 (osd.6) 153 : cluster 0 2.11 scrub starts 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.593020+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub ok 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.701561+0000 osd.4 (osd.4) 183 : cluster 0 12.10 scrub starts 2024-03-11T21:32:01.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.709778+0000 osd.4 (osd.4) 184 : cluster 0 12.10 scrub ok 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.992613+0000 osd.5 (osd.5) 209 : cluster 0 3.16 scrub starts 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:31:59.996233+0000 osd.5 (osd.5) 210 : cluster 0 3.16 scrub ok 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.006003+0000 mon.c (mon.2) 614 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:32:00.276910+0000 mon.a (mon.0) 2038 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.285743+0000 mon.a (mon.0) 2039 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi143-71604-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.285877+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.290481+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: cluster 2024-03-11T21:32:00.290875+0000 mon.a (mon.0) 2041 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.292783+0000 mon.b (mon.1) 93 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.294155+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:01.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:00 smithi143 bash[29284]: audit 2024-03-11T21:32:00.300638+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.104 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:01.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:01.234 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:01.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:01.357 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:01.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:01.488 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:01.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:01.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:01.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:01.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:01.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:01.879 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:01.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:01.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.006910+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:01.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.014926+0000 osd.1 (osd.1) 11 : cluster 0 5.1d scrub starts 2024-03-11T21:32:01.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.016234+0000 osd.1 (osd.1) 12 : cluster 0 5.1d scrub ok 2024-03-11T21:32:01.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.262509+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:01.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.262696+0000 mon.a (mon.0) 2045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.265428+0000 mon.b (mon.1) 94 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.273672+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.276692+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.280010+0000 mon.a (mon.0) 2047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.280926+0000 mon.a (mon.0) 2048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:01.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.315653+0000 mon.c (mon.2) 618 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:01.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:01 smithi203 bash[23654]: audit 2024-03-11T21:32:01.317517+0000 mon.a (mon.0) 2049 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:02.008 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:02.098 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.006910+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:02.098 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.014926+0000 osd.1 (osd.1) 11 : cluster 0 5.1d scrub starts 2024-03-11T21:32:02.098 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.016234+0000 osd.1 (osd.1) 12 : cluster 0 5.1d scrub ok 2024-03-11T21:32:02.098 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.262509+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:02.098 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.262696+0000 mon.a (mon.0) 2045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.265428+0000 mon.b (mon.1) 94 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.273672+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.276692+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.280010+0000 mon.a (mon.0) 2047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.280926+0000 mon.a (mon.0) 2048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.315653+0000 mon.c (mon.2) 618 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[21512]: audit 2024-03-11T21:32:01.317517+0000 mon.a (mon.0) 2049 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.006910+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.014926+0000 osd.1 (osd.1) 11 : cluster 0 5.1d scrub starts 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.016234+0000 osd.1 (osd.1) 12 : cluster 0 5.1d scrub ok 2024-03-11T21:32:02.099 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.262509+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.262696+0000 mon.a (mon.0) 2045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.265428+0000 mon.b (mon.1) 94 : audit 1 from='client.? 172.21.15.143:0/3624704405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.273672+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.276692+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.280010+0000 mon.a (mon.0) 2047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]: dispatch 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.280926+0000 mon.a (mon.0) 2048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]: dispatch 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.315653+0000 mon.c (mon.2) 618 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:02.100 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:01 smithi143 bash[29284]: audit 2024-03-11T21:32:01.317517+0000 mon.a (mon.0) 2049 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:02.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:02.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:02.272 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24679.0:2 22.0 22:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-11T21:32:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:46.802676+0000", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 195.556183086, 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6394070149999997, 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24679", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1821030059", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802676+0000", 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802676+0000", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802678+0000", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9269999999999999e-06 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802686+0000", 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7929999999999998e-06 2024-03-11T21:32:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802698+0000", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2041999999999999e-05 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.802712+0000", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3648e-05 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.836587+0000", 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033874865999999997 2024-03-11T21:32:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:46.836599+0000", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2463000000000001e-05 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.321125+0000", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4845259669999997 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.322846+0000", 2024-03-11T21:32:02.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001720795 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.442083+0000", 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.119237514 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(osd.7.0:196 13.7 e144/139 -15:e20ebceb:::temp_13.7_0_24289_1:head v 0'0, mlcod=0'0)", 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.171457+0000", 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 188.18740271600001, 2024-03-11T21:32:02.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.032743415999999997, 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.171457+0000", 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.171457+0000", 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.171459+0000", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.79e-06 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.176822+0000", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053634970000000001 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.176883+0000", 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0423000000000001e-05 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.176897+0000", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4041e-05 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.176968+0000", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1675999999999996e-05 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.176990+0000", 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1610000000000001e-05 2024-03-11T21:32:02.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.204150+0000", 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027159814000000001 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.204200+0000", 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0565000000000001e-05 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.347 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(osd.7.0:197 13.7 e144/139 -15:e20ebceb:::temp_13.7_0_24289_1:head v 0'0, mlcod=0'0)", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.194560+0000", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 188.16429919300001, 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.031507979999999998, 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.194560+0000", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.194560+0000", 2024-03-11T21:32:02.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.194562+0000", 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.826e-06 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.199273+0000", 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047104169999999997 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.199302+0000", 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9549000000000001e-05 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.199314+0000", 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1963e-05 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.199395+0000", 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1227e-05 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.199419+0000", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3850999999999999e-05 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.226004+0000", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026584412000000002 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.226068+0000", 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4734999999999996e-05 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.351 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(osd.7.0:198 13.7 e144/139 -15:e20ebceb:::temp_13.7_0_24289_1:head v 0'0, mlcod=0'0)", 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.212935+0000", 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 188.14592499400001, 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027996508, 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.212935+0000", 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.212935+0000", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.212936+0000", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.029e-06 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.216357+0000", 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034218880000000001 2024-03-11T21:32:02.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.216388+0000", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0864000000000001e-05 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.216401+0000", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2514000000000001e-05 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.216477+0000", 2024-03-11T21:32:02.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6224999999999996e-05 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.216499+0000", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.215e-05 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.240883+0000", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.024383950000000001 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.240931+0000", 2024-03-11T21:32:02.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7888e-05 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24743.0:2074 13.1e 13:7a94f9ba:test-rados-api-smithi143-71756-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.559891+0000", 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 187.79896840000001, 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.099049886000000004, 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24743", 2024-03-11T21:32:02.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/4179355585", 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2074 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.559891+0000", 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.559891+0000", 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.559892+0000", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9600000000000008e-07 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584241+0000", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.024348776999999999 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584259+0000", 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7842e-05 2024-03-11T21:32:02.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584267+0000", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9489999999999996e-06 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584330+0000", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3675000000000003e-05 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584460+0000", 2024-03-11T21:32:02.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012997900000000001 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584588+0000", 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012725799999999999 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650185+0000", 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.065597484999999997 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650659+0000", 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00047378399999999999 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.658857+0000", 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0081981599999999995 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.658906+0000", 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9041e-05 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.658941+0000", 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4940000000000001e-05 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.25169.0:32 21.c e144/140 21:37619e84:::7.obj3:head v 144'12, mlcod=144'10)", 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.606375+0000", 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 187.75248458199999, 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043745891000000002, 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606375+0000", 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606375+0000", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606376+0000", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0720000000000001e-06 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606387+0000", 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0808000000000001e-05 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606423+0000", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6155999999999997e-05 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606434+0000", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1442e-05 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606501+0000", 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6429999999999999e-05 2024-03-11T21:32:02.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.606517+0000", 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6271000000000001e-05 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650064+0000", 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043546995999999998 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650121+0000", 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6715999999999997e-05 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25400.0:68 97.3 97:c092418b:test-rados-api-smithi143-71929-18::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:28.469018+0000", 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 153.88984145200001, 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.102954561, 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.367 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25400", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/622293172", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 68 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469018+0000", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469018+0000", 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469020+0000", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0420000000000001e-06 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469026+0000", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1190000000000002e-06 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469038+0000", 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2012e-05 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469047+0000", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.337e-06 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469575+0000", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00052796600000000005 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.469846+0000", 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027138599999999998 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.477151+0000", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073049209999999998 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.477250+0000", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9600999999999997e-05 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.477413+0000", 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000162091 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.481173+0000", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037607790000000001 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.530348+0000", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.049175117999999997 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.571882+0000", 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.04153316 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.571924+0000", 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2234999999999998e-05 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.571973+0000", 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8794000000000001e-05 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:3 118.2 118:453056ba:test-rados-api-smithi143-71773-8::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:02.373 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:36.463115+0000", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 145.895744106, 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14016732700000001, 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463115+0000", 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463115+0000", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463117+0000", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.663e-06 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463124+0000", 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5490000000000003e-06 2024-03-11T21:32:02.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463134+0000", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0297e-05 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463144+0000", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0558e-05 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463200+0000", 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5024e-05 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463371+0000", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000171504 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.463568+0000", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019651299999999999 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.464491+0000", 2024-03-11T21:32:02.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00092338999999999995 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.464947+0000", 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00045640699999999998 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.603177+0000", 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13822996300000001 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.603236+0000", 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8853e-05 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.603283+0000", 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6606000000000001e-05 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.22002.0:212 123.2s0 123:451e2d00:::211:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:37.729057+0000", 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 144.629802748, 2024-03-11T21:32:02.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.049962103000000001, 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.22002", 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 212 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729057+0000", 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729057+0000", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729058+0000", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.393e-06 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729064+0000", 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7169999999999996e-06 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729076+0000", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1989e-05 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729086+0000", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9170000000000001e-06 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729142+0000", 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5858e-05 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729264+0000", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000122168 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.729617+0000", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000353471 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.730404+0000", 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00078717999999999995 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.778945+0000", 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.048540888999999997 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.779019+0000", 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3521000000000003e-05 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.384 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.38716.0:2 130.0 130:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:37.773226+0000", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 144.585633938, 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.53211611, 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.38716", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1159253884", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773226+0000", 2024-03-11T21:32:02.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773226+0000", 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773227+0000", 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5990000000000001e-06 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773234+0000", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.562e-06 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773245+0000", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0961999999999999e-05 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773255+0000", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0767e-05 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773303+0000", 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7039999999999997e-05 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.773308+0000", 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.857e-06 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.291898+0000", 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5185892430000001 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.292037+0000", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013941800000000001 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.292189+0000", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015168000000000001 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.292578+0000", 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000389071 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.293133+0000", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055503799999999999 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.305283+0000", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012150242 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.305317+0000", 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3974999999999999e-05 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.305342+0000", 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4655999999999999e-05 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.22002.0:238 123.5s0 123:a96ce6a2:::237:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:38.077540+0000", 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 144.281319669, 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.33662774899999998, 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.391 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.22002", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 238 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077540+0000", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077540+0000", 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077541+0000", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9400000000000005e-07 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077544+0000", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5700000000000001e-06 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077553+0000", 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0259999999999996e-06 2024-03-11T21:32:02.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077561+0000", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3560000000000004e-06 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077607+0000", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6026000000000003e-05 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077680+0000", 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3808999999999998e-05 2024-03-11T21:32:02.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.077882+0000", 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020127799999999999 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.078149+0000", 2024-03-11T21:32:02.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000267359 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.414120+0000", 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.335970781 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.414168+0000", 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7849999999999998e-05 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.396 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:300 126.1a 126:5b56e481:::test_293:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e178)", 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:38.958792+0000", 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 143.400067428, 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046531761999999997, 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 300 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.958792+0000", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.958792+0000", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.958794+0000", 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.979e-06 2024-03-11T21:32:02.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.958804+0000", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6330000000000008e-06 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.958838+0000", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4221000000000003e-05 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.960085+0000", 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012466440000000001 2024-03-11T21:32:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.961239+0000", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011545469999999999 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.961528+0000", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000289356 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.005324+0000", 2024-03-11T21:32:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043795382000000001 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40157.0:2 136.3 136:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:45.180130+0000", 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 137.178729543, 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.039659011000000001, 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40157", 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/498644794", 2024-03-11T21:32:02.401 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180130+0000", 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180130+0000", 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180132+0000", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9209999999999999e-06 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180139+0000", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4730000000000003e-06 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180152+0000", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.294e-05 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.180165+0000", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2764e-05 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.197661+0000", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017496397 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.197673+0000", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1479e-05 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.204811+0000", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071378789999999998 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.205390+0000", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00057906699999999995 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.205569+0000", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017941699999999999 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.207255+0000", 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016852410000000001 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.218348+0000", 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011093418000000001 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.219707+0000", 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001359388 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.219746+0000", 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8467000000000003e-05 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.219789+0000", 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.316e-05 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40952.0:1 185.6s0 185:7ba9627c:test-rados-api-smithi143-71838-17::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:32:02.407 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:17.270458+0000", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 105.088401906, 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0234499109999999, 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40952", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/334886417", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270458+0000", 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270458+0000", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270460+0000", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.886e-06 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270466+0000", 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.674e-06 2024-03-11T21:32:02.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270479+0000", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3268000000000001e-05 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270491+0000", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1946e-05 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270584+0000", 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3028000000000004e-05 2024-03-11T21:32:02.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:17.270589+0000", 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8040000000000001e-06 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.290921+0000", 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0203313730000001 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.291072+0000", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015167699999999999 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.291548+0000", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00047616899999999997 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.293622+0000", 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020733040000000002 2024-03-11T21:32:02.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.293672+0000", 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9812999999999999e-05 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:18.293908+0000", 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023596899999999999 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.413 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40988.0:2 189.6 189:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:22.260607+0000", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 100.098252465, 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027457591, 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40988", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2451351906", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.414 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260607+0000", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260607+0000", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260610+0000", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9440000000000001e-06 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.415 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260623+0000", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3202e-05 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260658+0000", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4912e-05 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.260684+0000", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6353e-05 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.275769+0000", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015084238999999999 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.276012+0000", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024373400000000001 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.276317+0000", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00030457699999999999 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.278721+0000", 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024038470000000002 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.282095+0000", 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0033739070000000002 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.287861+0000", 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.005766055 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.287963+0000", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102093 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:22.288065+0000", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101728 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.29808.0:2 190.13 190:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:23.239620+0000", 2024-03-11T21:32:02.419 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 99.119239868999998, 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.036691080000000001, 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.29808", 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1438445183", 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.239620+0000", 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.420 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.239620+0000", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.239622+0000", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3360000000000002e-06 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.239630+0000", 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0809999999999994e-06 2024-03-11T21:32:02.421 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.240152+0000", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00052196999999999996 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.240172+0000", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0018e-05 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.240268+0000", 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6378000000000005e-05 2024-03-11T21:32:02.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.240288+0000", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9660000000000002e-05 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.252479+0000", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01219075 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.252618+0000", 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000138942 2024-03-11T21:32:02.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.252797+0000", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017966 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.253609+0000", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00081143100000000004 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.253790+0000", 2024-03-11T21:32:02.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000181127 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.276211+0000", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022421197 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.276268+0000", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7021000000000002e-05 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.276311+0000", 2024-03-11T21:32:02.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509000000000002e-05 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40952.0:13 185.5s0 185:ad670e5a:test-rados-api-smithi143-71838-20::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:24.252276+0000", 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 98.106583047000001, 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.023776664999999999, 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.426 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40952", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/334886417", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 13 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252276+0000", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252276+0000", 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.427 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252278+0000", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.545e-06 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252285+0000", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0419999999999997e-06 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.253550+0000", 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001264989 2024-03-11T21:32:02.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.253562+0000", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2407e-05 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.261110+0000", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075473650000000003 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.261260+0000", 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015061400000000001 2024-03-11T21:32:02.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.261745+0000", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00048430400000000002 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.263578+0000", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018328559999999999 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.275971+0000", 2024-03-11T21:32:02.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012393322999999999 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.276053+0000", 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2219999999999995e-05 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:1501 118.2 118:46c6f3d6:test-rados-api-smithi143-71773-18::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:04.249147+0000", 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 58.109713018999997, 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044859611000000001, 2024-03-11T21:32:02.431 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1501 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249147+0000", 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249147+0000", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249149+0000", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8890000000000002e-06 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249165+0000", 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.607e-05 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249194+0000", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8799e-05 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249219+0000", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4491999999999999e-05 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249472+0000", 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025318200000000002 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.249499+0000", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7086000000000001e-05 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.258070+0000", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0085706010000000006 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.258426+0000", 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035632099999999999 2024-03-11T21:32:02.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 7", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.258864+0000", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00043758600000000001 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.280303+0000", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021439512000000001 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.293512+0000", 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013208892999999999 2024-03-11T21:32:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.293711+0000", 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000198576 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:04.294006+0000", 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029560399999999998 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:1777 226.16 226:69867c8b:test-rados-api-smithi143-71773-22::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e282)", 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:23.302397+0000", 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 39.056462123999999, 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033784692999999998, 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1777 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302397+0000", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302397+0000", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302401+0000", 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9060000000000004e-06 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302417+0000", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5696999999999999e-05 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302450+0000", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3475000000000001e-05 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302476+0000", 2024-03-11T21:32:02.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5839000000000001e-05 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302667+0000", 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000190411 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.302683+0000", 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6308999999999999e-05 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.325808+0000", 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.023125340000000001 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.326380+0000", 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00057119799999999997 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.334921+0000", 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0085417949999999996 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.335096+0000", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017416799999999999 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.335720+0000", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00062406400000000002 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.335855+0000", 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013504599999999999 2024-03-11T21:32:02.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.336182+0000", 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032744499999999999 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.30111.0:1 242.6s0 242:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:51.280502+0000", 2024-03-11T21:32:02.444 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 11.078357252, 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.041362302, 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.30111", 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3824385048", 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280502+0000", 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280502+0000", 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280504+0000", 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8649999999999999e-06 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:02.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280521+0000", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6798000000000001e-05 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280531+0000", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.046e-05 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.280541+0000", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9329999999999996e-06 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.282090+0000", 2024-03-11T21:32:02.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015483420000000001 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:51.282093+0000", 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0240000000000002e-06 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.319522+0000", 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.037429409 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:02.448 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.319681+0000", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015903 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.320110+0000", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042913199999999998 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.320577+0000", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00046631500000000002 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:02.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.321814+0000", 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00123767 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:52.321865+0000", 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0324000000000001e-05 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:02.450 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:32:02.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:02.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:02.475 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:02.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:02.603 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:02.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.734 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:02.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.860 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:02.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:02.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.012641+0000 osd.0 (osd.0) 199 : cluster 0 6.18 scrub starts 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.014164+0000 osd.0 (osd.0) 200 : cluster 0 6.18 scrub ok 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.315019+0000 mgr.y (mgr.14550) 364 : cluster 0 pgmap v468: 295 pgs: 1 peering, 1 creating+peering, 7 unknown, 286 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.852273+0000 osd.2 (osd.2) 159 : cluster 0 118.17 scrub starts 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:01.854459+0000 osd.2 (osd.2) 160 : cluster 0 118.17 scrub ok 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.008142+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.263233+0000 mon.a (mon.0) 2050 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.265113+0000 mon.a (mon.0) 2051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.265196+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]': finished 2024-03-11T21:32:02.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.265276+0000 mon.a (mon.0) 2053 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.270240+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.272418+0000 mon.c (mon.2) 620 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.273690+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.301974+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.302441+0000 mon.a (mon.0) 2056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.305084+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.305492+0000 mon.a (mon.0) 2057 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.306181+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[29284]: audit 2024-03-11T21:32:02.306633+0000 mon.a (mon.0) 2058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.965 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.012641+0000 osd.0 (osd.0) 199 : cluster 0 6.18 scrub starts 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.014164+0000 osd.0 (osd.0) 200 : cluster 0 6.18 scrub ok 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.315019+0000 mgr.y (mgr.14550) 364 : cluster 0 pgmap v468: 295 pgs: 1 peering, 1 creating+peering, 7 unknown, 286 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.852273+0000 osd.2 (osd.2) 159 : cluster 0 118.17 scrub starts 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:01.854459+0000 osd.2 (osd.2) 160 : cluster 0 118.17 scrub ok 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.008142+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.263233+0000 mon.a (mon.0) 2050 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.265113+0000 mon.a (mon.0) 2051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.265196+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]': finished 2024-03-11T21:32:02.966 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.265276+0000 mon.a (mon.0) 2053 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.270240+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.272418+0000 mon.c (mon.2) 620 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.273690+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.301974+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.302441+0000 mon.a (mon.0) 2056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.305084+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.305492+0000 mon.a (mon.0) 2057 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.306181+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:02 smithi143 bash[21512]: audit 2024-03-11T21:32:02.306633+0000 mon.a (mon.0) 2058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.012641+0000 osd.0 (osd.0) 199 : cluster 0 6.18 scrub starts 2024-03-11T21:32:02.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.014164+0000 osd.0 (osd.0) 200 : cluster 0 6.18 scrub ok 2024-03-11T21:32:02.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.315019+0000 mgr.y (mgr.14550) 364 : cluster 0 pgmap v468: 295 pgs: 1 peering, 1 creating+peering, 7 unknown, 286 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:02.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.852273+0000 osd.2 (osd.2) 159 : cluster 0 118.17 scrub starts 2024-03-11T21:32:02.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:01.854459+0000 osd.2 (osd.2) 160 : cluster 0 118.17 scrub ok 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.008142+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.263233+0000 mon.a (mon.0) 2050 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.265113+0000 mon.a (mon.0) 2051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi143-71594-38"}]': finished 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.265196+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-29", "mode": "writeback"}]': finished 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.265276+0000 mon.a (mon.0) 2053 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.270240+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.272418+0000 mon.c (mon.2) 620 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.273690+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.301974+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.302441+0000 mon.a (mon.0) 2056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.305084+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.305492+0000 mon.a (mon.0) 2057 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.306181+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.978 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:02 smithi203 bash[23654]: audit 2024-03-11T21:32:02.306633+0000 mon.a (mon.0) 2058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:03.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.120 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:32:03.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:03.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:03.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:03.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:03.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:03.762 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:03.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.050819+0000 osd.1 (osd.1) 13 : cluster 0 5.19 scrub starts 2024-03-11T21:32:03.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.052853+0000 osd.1 (osd.1) 14 : cluster 0 5.19 scrub ok 2024-03-11T21:32:03.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.600173+0000 osd.3 (osd.3) 205 : cluster 0 39.17 scrub starts 2024-03-11T21:32:03.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.603941+0000 osd.3 (osd.3) 206 : cluster 0 39.17 scrub ok 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.893461+0000 osd.2 (osd.2) 161 : cluster 0 118.9 scrub starts 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:02.895299+0000 osd.2 (osd.2) 162 : cluster 0 118.9 scrub ok 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.010146+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.269705+0000 mon.a (mon.0) 2059 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.269808+0000 mon.a (mon.0) 2060 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: cluster 2024-03-11T21:32:03.274525+0000 mon.a (mon.0) 2061 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.283627+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.284033+0000 mon.c (mon.2) 626 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.284194+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[21512]: audit 2024-03-11T21:32:03.284850+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.050819+0000 osd.1 (osd.1) 13 : cluster 0 5.19 scrub starts 2024-03-11T21:32:03.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.052853+0000 osd.1 (osd.1) 14 : cluster 0 5.19 scrub ok 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.600173+0000 osd.3 (osd.3) 205 : cluster 0 39.17 scrub starts 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.603941+0000 osd.3 (osd.3) 206 : cluster 0 39.17 scrub ok 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.893461+0000 osd.2 (osd.2) 161 : cluster 0 118.9 scrub starts 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:02.895299+0000 osd.2 (osd.2) 162 : cluster 0 118.9 scrub ok 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.010146+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.269705+0000 mon.a (mon.0) 2059 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.269808+0000 mon.a (mon.0) 2060 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: cluster 2024-03-11T21:32:03.274525+0000 mon.a (mon.0) 2061 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.283627+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.284033+0000 mon.c (mon.2) 626 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.284194+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:03 smithi143 bash[29284]: audit 2024-03-11T21:32:03.284850+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:03.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:04.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.050819+0000 osd.1 (osd.1) 13 : cluster 0 5.19 scrub starts 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.052853+0000 osd.1 (osd.1) 14 : cluster 0 5.19 scrub ok 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.600173+0000 osd.3 (osd.3) 205 : cluster 0 39.17 scrub starts 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.603941+0000 osd.3 (osd.3) 206 : cluster 0 39.17 scrub ok 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.893461+0000 osd.2 (osd.2) 161 : cluster 0 118.9 scrub starts 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:02.895299+0000 osd.2 (osd.2) 162 : cluster 0 118.9 scrub ok 2024-03-11T21:32:04.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.010146+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.269705+0000 mon.a (mon.0) 2059 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.269808+0000 mon.a (mon.0) 2060 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi143-71594-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: cluster 2024-03-11T21:32:03.274525+0000 mon.a (mon.0) 2061 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.283627+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.143:0/3080125293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.284033+0000 mon.c (mon.2) 626 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:04.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.284194+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]: dispatch 2024-03-11T21:32:04.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:03 smithi203 bash[23654]: audit 2024-03-11T21:32:03.284850+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:04.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:04.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.156 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:04.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.289 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:04.393 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:04.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:04.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:04.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:04.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:04.678 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:04.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:04.809 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:04.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: cluster 2024-03-11T21:32:03.315886+0000 mgr.y (mgr.14550) 365 : cluster 0 pgmap v471: 287 pgs: 3 peering, 284 active+clean; 8.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:32:04.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.012295+0000 mon.c (mon.2) 627 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:04.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.279739+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:04.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.343748+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:04.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.746483+0000 mon.a (mon.0) 2065 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:32:04.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.747653+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:04.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.748579+0000 mon.c (mon.2) 629 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:04.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.748945+0000 mon.a (mon.0) 2067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:04.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.749581+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:04.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:04 smithi203 bash[23654]: audit 2024-03-11T21:32:04.749945+0000 mon.a (mon.0) 2068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:04.937 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:05.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:05.052 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: cluster 2024-03-11T21:32:03.315886+0000 mgr.y (mgr.14550) 365 : cluster 0 pgmap v471: 287 pgs: 3 peering, 284 active+clean; 8.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.012295+0000 mon.c (mon.2) 627 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.279739+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.343748+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.746483+0000 mon.a (mon.0) 2065 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.747653+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.748579+0000 mon.c (mon.2) 629 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.748945+0000 mon.a (mon.0) 2067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.749581+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[29284]: audit 2024-03-11T21:32:04.749945+0000 mon.a (mon.0) 2068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: cluster 2024-03-11T21:32:03.315886+0000 mgr.y (mgr.14550) 365 : cluster 0 pgmap v471: 287 pgs: 3 peering, 284 active+clean; 8.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T21:32:05.053 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.012295+0000 mon.c (mon.2) 627 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.279739+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi143-71604-48"}]': finished 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.343748+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.746483+0000 mon.a (mon.0) 2065 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.747653+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.748579+0000 mon.c (mon.2) 629 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.748945+0000 mon.a (mon.0) 2067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.749581+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:05.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:04 smithi143 bash[21512]: audit 2024-03-11T21:32:04.749945+0000 mon.a (mon.0) 2068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:05.069 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:05.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:05.201 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:05.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:05.329 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:05.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:05.455 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:05.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:05.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:05.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:05.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:05.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.028690+0000 osd.3 (osd.3) 207 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:05.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.029071+0000 osd.3 (osd.3) 208 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:05.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.057480+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:32:05.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.059131+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.561507+0000 osd.7 (osd.7) 243 : cluster 0 39.9 scrub starts 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.563492+0000 osd.7 (osd.7) 244 : cluster 0 39.9 scrub ok 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.845037+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:04.847082+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.013969+0000 mon.c (mon.2) 631 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.284532+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.284811+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.290187+0000 mon.c (mon.2) 632 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.291654+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.292862+0000 mon.a (mon.0) 2072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.317059+0000 mon.c (mon.2) 633 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.318409+0000 mon.a (mon.0) 2073 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.434742+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:05 smithi203 bash[23654]: audit 2024-03-11T21:32:05.435156+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.847 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.028690+0000 osd.3 (osd.3) 207 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.029071+0000 osd.3 (osd.3) 208 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.057480+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.059131+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.561507+0000 osd.7 (osd.7) 243 : cluster 0 39.9 scrub starts 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.563492+0000 osd.7 (osd.7) 244 : cluster 0 39.9 scrub ok 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.845037+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:04.847082+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.013969+0000 mon.c (mon.2) 631 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.284532+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.284811+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:05.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.290187+0000 mon.c (mon.2) 632 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.291654+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.292862+0000 mon.a (mon.0) 2072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.317059+0000 mon.c (mon.2) 633 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.318409+0000 mon.a (mon.0) 2073 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.434742+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[29284]: audit 2024-03-11T21:32:05.435156+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.028690+0000 osd.3 (osd.3) 207 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.029071+0000 osd.3 (osd.3) 208 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.057480+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.059131+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.561507+0000 osd.7 (osd.7) 243 : cluster 0 39.9 scrub starts 2024-03-11T21:32:05.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.563492+0000 osd.7 (osd.7) 244 : cluster 0 39.9 scrub ok 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.845037+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:04.847082+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.013969+0000 mon.c (mon.2) 631 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.284532+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi143-71594-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.284811+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi143-71604-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.290187+0000 mon.c (mon.2) 632 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.291654+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.292862+0000 mon.a (mon.0) 2072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.317059+0000 mon.c (mon.2) 633 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.318409+0000 mon.a (mon.0) 2073 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.434742+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:05 smithi143 bash[21512]: audit 2024-03-11T21:32:05.435156+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:05.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.975 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:06.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.106 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:06.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:06.237 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:06.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:06.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:06.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:06.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:06.621 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:06.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:06.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:06.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.883 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:06.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:07.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.071980+0000 osd.1 (osd.1) 17 : cluster 0 3.4 scrub starts 2024-03-11T21:32:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.077392+0000 osd.1 (osd.1) 18 : cluster 0 3.4 scrub ok 2024-03-11T21:32:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.316533+0000 mgr.y (mgr.14550) 366 : cluster 0 pgmap v474: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 1.2 KiB/s wr, 20 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.610293+0000 osd.6 (osd.6) 155 : cluster 0 12.c scrub starts 2024-03-11T21:32:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.612180+0000 osd.6 (osd.6) 156 : cluster 0 12.c scrub ok 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.836261+0000 osd.5 (osd.5) 211 : cluster 0 6.d scrub starts 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.838027+0000 osd.5 (osd.5) 212 : cluster 0 6.d scrub ok 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.873607+0000 osd.2 (osd.2) 165 : cluster 0 6.f scrub starts 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:05.875982+0000 osd.2 (osd.2) 166 : cluster 0 6.f scrub ok 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.015677+0000 mon.c (mon.2) 635 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.067070+0000 mon.a (mon.0) 2075 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.071328+0000 mon.c (mon.2) 636 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.261830+0000 mon.a (mon.0) 2076 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.289146+0000 mon.a (mon.0) 2077 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.289304+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.304772+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.758873+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:32:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:06 smithi203 bash[23654]: audit 2024-03-11T21:32:06.760707+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.071980+0000 osd.1 (osd.1) 17 : cluster 0 3.4 scrub starts 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.077392+0000 osd.1 (osd.1) 18 : cluster 0 3.4 scrub ok 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.316533+0000 mgr.y (mgr.14550) 366 : cluster 0 pgmap v474: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 1.2 KiB/s wr, 20 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.610293+0000 osd.6 (osd.6) 155 : cluster 0 12.c scrub starts 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.612180+0000 osd.6 (osd.6) 156 : cluster 0 12.c scrub ok 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.836261+0000 osd.5 (osd.5) 211 : cluster 0 6.d scrub starts 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.838027+0000 osd.5 (osd.5) 212 : cluster 0 6.d scrub ok 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.873607+0000 osd.2 (osd.2) 165 : cluster 0 6.f scrub starts 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:05.875982+0000 osd.2 (osd.2) 166 : cluster 0 6.f scrub ok 2024-03-11T21:32:07.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: audit 2024-03-11T21:32:06.015677+0000 mon.c (mon.2) 635 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: audit 2024-03-11T21:32:06.067070+0000 mon.a (mon.0) 2075 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: audit 2024-03-11T21:32:06.071328+0000 mon.c (mon.2) 636 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.261830+0000 mon.a (mon.0) 2076 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: audit 2024-03-11T21:32:06.289146+0000 mon.a (mon.0) 2077 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[21512]: audit 2024-03-11T21:32:06.289304+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:06.304772+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.758873+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:06.760707+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.110 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.071980+0000 osd.1 (osd.1) 17 : cluster 0 3.4 scrub starts 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.077392+0000 osd.1 (osd.1) 18 : cluster 0 3.4 scrub ok 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.316533+0000 mgr.y (mgr.14550) 366 : cluster 0 pgmap v474: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 1.2 KiB/s wr, 20 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.610293+0000 osd.6 (osd.6) 155 : cluster 0 12.c scrub starts 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.612180+0000 osd.6 (osd.6) 156 : cluster 0 12.c scrub ok 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.836261+0000 osd.5 (osd.5) 211 : cluster 0 6.d scrub starts 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.838027+0000 osd.5 (osd.5) 212 : cluster 0 6.d scrub ok 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.873607+0000 osd.2 (osd.2) 165 : cluster 0 6.f scrub starts 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:05.875982+0000 osd.2 (osd.2) 166 : cluster 0 6.f scrub ok 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: audit 2024-03-11T21:32:06.015677+0000 mon.c (mon.2) 635 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: audit 2024-03-11T21:32:06.067070+0000 mon.a (mon.0) 2075 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: audit 2024-03-11T21:32:06.071328+0000 mon.c (mon.2) 636 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.261830+0000 mon.a (mon.0) 2076 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: audit 2024-03-11T21:32:06.289146+0000 mon.a (mon.0) 2077 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:06 smithi143 bash[29284]: audit 2024-03-11T21:32:06.289304+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:07.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:06.304772+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.758873+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:32:07.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:06.760707+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]: dispatch 2024-03-11T21:32:07.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.139 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:07.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:07.274 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:07.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:07.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:07.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:07.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:07.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.667 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:07.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:07.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.312872+0000 osd.7 (osd.7) 245 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.313107+0000 osd.7 (osd.7) 246 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.798597+0000 osd.5 (osd.5) 213 : cluster 0 5.16 scrub starts 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.800425+0000 osd.5 (osd.5) 214 : cluster 0 5.16 scrub ok 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.994558+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.996509+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: audit 2024-03-11T21:32:07.017198+0000 mon.c (mon.2) 638 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.290051+0000 mon.a (mon.0) 2081 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: audit 2024-03-11T21:32:07.292557+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: audit 2024-03-11T21:32:07.292785+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.300118+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: audit 2024-03-11T21:32:07.303275+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: audit 2024-03-11T21:32:07.304101+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.369308+0000 osd.5 (osd.5) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.785 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:07 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.369545+0000 osd.5 (osd.5) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:07.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:07.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.924 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:07.930 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:07.958 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.312872+0000 osd.7 (osd.7) 245 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.959 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.313107+0000 osd.7 (osd.7) 246 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:07.959 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.798597+0000 osd.5 (osd.5) 213 : cluster 0 5.16 scrub starts 2024-03-11T21:32:07.959 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.800425+0000 osd.5 (osd.5) 214 : cluster 0 5.16 scrub ok 2024-03-11T21:32:07.959 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.994558+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:32:07.959 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.996509+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:32:07.960 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:07.017198+0000 mon.c (mon.2) 638 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.960 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.290051+0000 mon.a (mon.0) 2081 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:07.960 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:07.292557+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:07.960 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:07.292785+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:07.960 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.300118+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:32:07.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:07.303275+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: audit 2024-03-11T21:32:07.304101+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.369308+0000 osd.5 (osd.5) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.369545+0000 osd.5 (osd.5) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:07.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.312872+0000 osd.7 (osd.7) 245 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.313107+0000 osd.7 (osd.7) 246 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:07.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.798597+0000 osd.5 (osd.5) 213 : cluster 0 5.16 scrub starts 2024-03-11T21:32:07.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.800425+0000 osd.5 (osd.5) 214 : cluster 0 5.16 scrub ok 2024-03-11T21:32:07.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.994558+0000 osd.1 (osd.1) 19 : cluster 0 2.d scrub starts 2024-03-11T21:32:07.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.996509+0000 osd.1 (osd.1) 20 : cluster 0 2.d scrub ok 2024-03-11T21:32:07.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:07.017198+0000 mon.c (mon.2) 638 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:07.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.290051+0000 mon.a (mon.0) 2081 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:07.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:07.292557+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi143-71604-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:07.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:07.292785+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-29"}]': finished 2024-03-11T21:32:07.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.300118+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:32:07.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:07.303275+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: audit 2024-03-11T21:32:07.304101+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:07.964 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.369308+0000 osd.5 (osd.5) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:07.965 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:07 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.369545+0000 osd.5 (osd.5) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:08.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.062 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:08.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:08.193 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:08.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:08.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:08.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:08.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:08.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.722 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:08.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:08.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:08.855 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/1361969794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984101700 msgr2=0x7f898410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- 172.21.15.143:0/1361969794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984101700 0x7f898410db70 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f897c00b0a0 tx=0x7f897c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/1361969794 shutdown_connections 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- 172.21.15.143:0/1361969794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984101700 0x7f898410db70 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- 172.21.15.143:0/1361969794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f89841071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- 172.21.15.143:0/1361969794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984100d40 0x7f89841011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/1361969794 >> 172.21.15.143:0/1361969794 conn(0x7f8984076d80 msgr2=0x7f8984077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/1361969794 shutdown_connections 2024-03-11T21:32:08.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/1361969794 wait complete. 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 librados: starting msgr at 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 librados: starting objecter 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 Processor -- start 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- start start 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 librados: setting wanted keys 2024-03-11T21:32:08.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 librados: calling monclient init 2024-03-11T21:32:08.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 0x7f8984108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f8984108690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8984110570 con 0x7f8984100d40 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89841103f0 con 0x7f8984106dc0 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89841106f0 con 0x7f8984101700 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56060/0 (socket says 172.21.15.143:56060) 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 -- 172.21.15.143:0/3181800047 learned_addr learned my addr 172.21.15.143:0/3181800047 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f897b7fe700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 0x7f8984108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898895c700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f8984108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:08.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 -- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 msgr2=0x7f8984108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 0x7f8984108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 -- 172.21.15.143:0/3181800047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 msgr2=0x7f8984108690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f8984108690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 -- 172.21.15.143:0/3181800047 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89841a6100 con 0x7f8984100d40 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f897b7fe700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 0x7f8984108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898895c700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f8984108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8983fff700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f897400d950 tx=0x7f897400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8981ffb700 10 client.?.objecter ms_handle_connect 0x7f8984100d40 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8981ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974016070 con 0x7f8984100d40 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89841a63f0 con 0x7f8984100d40 2024-03-11T21:32:08.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8974009dc0 con 0x7f8984100d40 2024-03-11T21:32:08.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897400f3c0 con 0x7f8984100d40 2024-03-11T21:32:08.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 10 client.30162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:08.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89841a6980 con 0x7f8984100d40 2024-03-11T21:32:08.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.963+0000 7f898a3bf700 1 librados: init done 2024-03-11T21:32:08.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898404faa0 con 0x7f8984100d40 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8974019070 con 0x7f8984100d40 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 0x7f8970071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(327..327 src has 87..327) v4 ==== 10357+0+0 (secure 0 0 0) 0x7f897408e940 con 0x7f8984100d40 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 10 client.30162.objecter ms_dispatch 0x7f8984001550 osd_map(327..327 src has 87..327) v4 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 3 client.30162.objecter handle_osd_map got epochs [327,327] > 0 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 3 client.30162.objecter handle_osd_map decoding full epoch 327 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f897b7fe700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 0x7f8970071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 20 client.30162.objecter dump_active .. 0 homeless 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f8981ffb700 10 client.30162.objecter ms_handle_connect 0x7f897006f820 2024-03-11T21:32:08.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.967+0000 7f897b7fe700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 0x7f8970071ce0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f896c009680 tx=0x7f896c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:08.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:08.971+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f897405b2b0 con 0x7f8984100d40 2024-03-11T21:32:08.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.978 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:09.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.987796+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:06.989714+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.317374+0000 mgr.y (mgr.14550) 367 : cluster 0 pgmap v477: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.551175+0000 osd.6 (osd.6) 157 : cluster 0 118.1d scrub starts 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.554677+0000 osd.6 (osd.6) 158 : cluster 0 118.1d scrub ok 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.646309+0000 osd.3 (osd.3) 209 : cluster 0 39.d scrub starts 2024-03-11T21:32:09.067 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.648787+0000 osd.3 (osd.3) 210 : cluster 0 39.d scrub ok 2024-03-11T21:32:09.068 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.731265+0000 osd.4 (osd.4) 185 : cluster 0 6.1a scrub starts 2024-03-11T21:32:09.068 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.733546+0000 osd.4 (osd.4) 186 : cluster 0 6.1a scrub ok 2024-03-11T21:32:09.068 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.801609+0000 osd.2 (osd.2) 167 : cluster 0 5.d scrub starts 2024-03-11T21:32:09.068 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.803823+0000 osd.2 (osd.2) 168 : cluster 0 5.d scrub ok 2024-03-11T21:32:09.068 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.812015+0000 osd.5 (osd.5) 217 : cluster 0 5.7 scrub starts 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.813600+0000 osd.5 (osd.5) 218 : cluster 0 5.7 scrub ok 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.974412+0000 osd.1 (osd.1) 21 : cluster 0 6.1 scrub starts 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:07.975728+0000 osd.1 (osd.1) 22 : cluster 0 6.1 scrub ok 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: audit 2024-03-11T21:32:08.018949+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: audit 2024-03-11T21:32:08.300667+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: audit 2024-03-11T21:32:08.306573+0000 mon.c (mon.2) 641 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: cluster 2024-03-11T21:32:08.308666+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:32:09.069 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:08 smithi203 bash[23654]: audit 2024-03-11T21:32:08.313115+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:09.104 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:09.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.203+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f89841a6be0 con 0x7f8984100d40 2024-03-11T21:32:09.208 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:09.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:09.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.303+0000 7f8981ffb700 1 -- 172.21.15.143:0/3181800047 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v328) v1 ==== 113+0+0 (secure 0 0 0) 0x7f897405ed00 con 0x7f8984100d40 2024-03-11T21:32:09.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.307+0000 7f898a3bf700 10 librados: watch_flush enter 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 10 librados: watch_flush exit 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 20 client.30162.objecter shutdown clearing up homeless session... 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 10 client.30162.objecter successfully canceled tick 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 msgr2=0x7f8970071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 0x7f8970071ce0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f896c009680 tx=0x7f896c009260 comp rx=0 tx=0).stop 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 msgr2=0x7f8984107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.311+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f897400d950 tx=0x7f897400dc60 comp rx=0 tx=0).stop 2024-03-11T21:32:09.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 shutdown_connections 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f897006f820 0x7f8970071ce0 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8984100d40 0x7f8984107c10 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8984106dc0 0x7f8984108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 --2- 172.21.15.143:0/3181800047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8984101700 0x7f8984108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 >> 172.21.15.143:0/3181800047 conn(0x7f8984076d80 msgr2=0x7f89840fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 shutdown_connections 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 -- 172.21.15.143:0/3181800047 wait complete. 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.331+0000 7f898a3bf700 1 librados: shutdown 2024-03-11T21:32:09.305 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:32:09.320 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.987796+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:32:09.320 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:06.989714+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.317374+0000 mgr.y (mgr.14550) 367 : cluster 0 pgmap v477: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.551175+0000 osd.6 (osd.6) 157 : cluster 0 118.1d scrub starts 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.554677+0000 osd.6 (osd.6) 158 : cluster 0 118.1d scrub ok 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.646309+0000 osd.3 (osd.3) 209 : cluster 0 39.d scrub starts 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.648787+0000 osd.3 (osd.3) 210 : cluster 0 39.d scrub ok 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.731265+0000 osd.4 (osd.4) 185 : cluster 0 6.1a scrub starts 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.733546+0000 osd.4 (osd.4) 186 : cluster 0 6.1a scrub ok 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.801609+0000 osd.2 (osd.2) 167 : cluster 0 5.d scrub starts 2024-03-11T21:32:09.321 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.803823+0000 osd.2 (osd.2) 168 : cluster 0 5.d scrub ok 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.812015+0000 osd.5 (osd.5) 217 : cluster 0 5.7 scrub starts 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.813600+0000 osd.5 (osd.5) 218 : cluster 0 5.7 scrub ok 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.974412+0000 osd.1 (osd.1) 21 : cluster 0 6.1 scrub starts 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:07.975728+0000 osd.1 (osd.1) 22 : cluster 0 6.1 scrub ok 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:08.018949+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:08.300667+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:08.306573+0000 mon.c (mon.2) 641 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:08.308666+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:32:09.322 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:08.313115+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.987796+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:06.989714+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.317374+0000 mgr.y (mgr.14550) 367 : cluster 0 pgmap v477: 294 pgs: 8 unknown, 1 peering, 285 active+clean; 8.6 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.551175+0000 osd.6 (osd.6) 157 : cluster 0 118.1d scrub starts 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.554677+0000 osd.6 (osd.6) 158 : cluster 0 118.1d scrub ok 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.646309+0000 osd.3 (osd.3) 209 : cluster 0 39.d scrub starts 2024-03-11T21:32:09.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.648787+0000 osd.3 (osd.3) 210 : cluster 0 39.d scrub ok 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.731265+0000 osd.4 (osd.4) 185 : cluster 0 6.1a scrub starts 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.733546+0000 osd.4 (osd.4) 186 : cluster 0 6.1a scrub ok 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.801609+0000 osd.2 (osd.2) 167 : cluster 0 5.d scrub starts 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.803823+0000 osd.2 (osd.2) 168 : cluster 0 5.d scrub ok 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.812015+0000 osd.5 (osd.5) 217 : cluster 0 5.7 scrub starts 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.813600+0000 osd.5 (osd.5) 218 : cluster 0 5.7 scrub ok 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.974412+0000 osd.1 (osd.1) 21 : cluster 0 6.1 scrub starts 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:07.975728+0000 osd.1 (osd.1) 22 : cluster 0 6.1 scrub ok 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:08.018949+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:08.300667+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:08.306573+0000 mon.c (mon.2) 641 : audit 1 from='client.? 172.21.15.143:0/2500018052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:08.308666+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:32:09.325 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:08.313115+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]: dispatch 2024-03-11T21:32:09.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:09.357 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:09.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:09.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:09.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:09.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:09.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:09.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/1667682778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 msgr2=0x7fe9a4110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- 172.21.15.143:0/1667682778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 0x7fe9a4110f30 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009fc0 tx=0x7fe99c00b330 comp rx=0 tx=0).stop 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/1667682778 shutdown_connections 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- 172.21.15.143:0/1667682778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104280 0x7fe9a4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- 172.21.15.143:0/1667682778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 0x7fe9a4110f30 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- 172.21.15.143:0/1667682778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104c30 0x7fe9a4108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/1667682778 >> 172.21.15.143:0/1667682778 conn(0x7fe9a40ffb10 msgr2=0x7fe9a4101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/1667682778 shutdown_connections 2024-03-11T21:32:09.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/1667682778 wait complete. 2024-03-11T21:32:09.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 librados: starting msgr at 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 librados: starting objecter 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 Processor -- start 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- start start 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 librados: setting wanted keys 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 librados: calling monclient init 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104280 0x7fe9a4199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 0x7fe9a419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a41138b0 con 0x7fe9a4104c30 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4113730 con 0x7fe9a4104280 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4113a30 con 0x7fe9a4109360 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56094/0 (socket says 172.21.15.143:56094) 2024-03-11T21:32:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 -- 172.21.15.143:0/3174280466 learned_addr learned my addr 172.21.15.143:0/3174280466 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a9267700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 0x7fe9a419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104280 0x7fe9a4199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 msgr2=0x7fe9a419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9a4109360 0x7fe9a419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104280 msgr2=0x7fe9a4199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104280 0x7fe9a4199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a419ee00 con 0x7fe9a4104c30 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a4104280 0x7fe9a4199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe99800e970 tx=0x7fe99800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a1ffb700 10 client.?.objecter ms_handle_connect 0x7fe9a4104c30 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a1ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe998004a30 con 0x7fe9a4104c30 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a419f0f0 con 0x7fe9a4104c30 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe99800cc60 con 0x7fe9a4104c30 2024-03-11T21:32:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe998005490 con 0x7fe9a4104c30 2024-03-11T21:32:09.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 10 client.30174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:09.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a41a6970 con 0x7fe9a4104c30 2024-03-11T21:32:09.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.803+0000 7fe9aacca700 1 librados: init done 2024-03-11T21:32:09.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe9a404faa0 con 0x7fe9a4104c30 2024-03-11T21:32:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe99800c7e0 con 0x7fe9a4104c30 2024-03-11T21:32:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 0x7fe994071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(328..328 src has 87..328) v4 ==== 10211+0+0 (secure 0 0 0) 0x7fe99808f250 con 0x7fe9a4104c30 2024-03-11T21:32:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 osd_map(328..328 src has 87..328) v4 2024-03-11T21:32:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 0x7fe994071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 3 client.30174.objecter handle_osd_map got epochs [328,328] > 0 2024-03-11T21:32:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 3 client.30174.objecter handle_osd_map decoding full epoch 328 2024-03-11T21:32:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 20 client.30174.objecter dump_active .. 0 homeless 2024-03-11T21:32:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v328) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe99808f6a0 con 0x7fe9a4104c30 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe99406f820 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 0x7fe994071ce0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fe990009750 tx=0x7fe990009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9aacca700 10 client.30174.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.807+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 1 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe9a40629f0 0x7fe9a4062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 1 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 0 1 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 4 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 10 client.30174.objecter _send_command 1 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41a9d90 con 0x7fe9a40629f0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe9a40629f0 0x7fe9a4062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a40629f0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe9a40629f0 0x7fe9a4062dd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41a9d90 con 0x7fe9a40629f0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 1 = system:0 2024-03-11T21:32:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 4 2024-03-11T21:32:09.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.811+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 0 1 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 10 client.30174.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 2 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 2 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a419b150 osd=0 3 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 0 2 2024-03-11T21:32:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 4 2024-03-11T21:32:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 10 client.30174.objecter _send_command 2 2024-03-11T21:32:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.827+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a4062470 con 0x7fe9a40629f0 2024-03-11T21:32:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a4062470 con 0x7fe9a40629f0 2024-03-11T21:32:09.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 4 2024-03-11T21:32:09.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 0 2 2024-03-11T21:32:09.805 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 10 client.30174.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe9a402c6b0 0x7fe9a402ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 1 3 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 4 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 10 client.30174.objecter _send_command 3 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a410bc80 con 0x7fe9a402c6b0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe9a402c6b0 0x7fe9a402ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a402c6b0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe9a402c6b0 0x7fe9a402ca90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a410bc80 con 0x7fe9a402c6b0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 3 = system:0 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 4 2024-03-11T21:32:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.831+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 1 3 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 10 client.30174.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 4 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 4 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a410b600 osd=1 3 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 1 4 2024-03-11T21:32:09.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 4 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 10 client.30174.objecter _send_command 4 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.847+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a410b080 con 0x7fe9a402c6b0 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a410b080 con 0x7fe9a402c6b0 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 4 2024-03-11T21:32:09.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 1 4 2024-03-11T21:32:09.826 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 10 client.30174.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 5 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe9a41abf50 0x7fe9a41ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 5 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 2 5 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 4 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 10 client.30174.objecter _send_command 5 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41b11a0 con 0x7fe9a41abf50 2024-03-11T21:32:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe9a41abf50 0x7fe9a41ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe9a41abf50 0x7fe9a41ae340 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41abf50 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41b11a0 con 0x7fe9a41abf50 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 5 = system:0 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 4 2024-03-11T21:32:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.851+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 2 5 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 10 client.30174.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 6 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 6 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a410b480 osd=2 3 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 2 6 2024-03-11T21:32:09.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 4 2024-03-11T21:32:09.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 10 client.30174.objecter _send_command 6 2024-03-11T21:32:09.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.867+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a402d220 con 0x7fe9a41abf50 2024-03-11T21:32:09.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a402d220 con 0x7fe9a41abf50 2024-03-11T21:32:09.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 4 2024-03-11T21:32:09.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 2 6 2024-03-11T21:32:09.846 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 10 client.30174.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 7 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe9a41afa00 0x7fe9a41afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 7 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 3 7 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 4 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 10 client.30174.objecter _send_command 7 2024-03-11T21:32:09.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41a9ab0 con 0x7fe9a41afa00 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe9a41afa00 0x7fe9a41afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41afa00 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe9a41afa00 0x7fe9a41afe20 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41a9ab0 con 0x7fe9a41afa00 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 7 = system:0 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 4 2024-03-11T21:32:09.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.871+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 3 7 2024-03-11T21:32:09.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:09.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:08.500620+0000 mgr.y (mgr.14550) 368 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:09.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: cluster 2024-03-11T21:32:08.742271+0000 osd.4 (osd.4) 187 : cluster 0 6.e scrub starts 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: cluster 2024-03-11T21:32:08.744410+0000 osd.4 (osd.4) 188 : cluster 0 6.e scrub ok 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.019830+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.207010+0000 mon.a (mon.0) 2089 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.305428+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.305573+0000 mon.a (mon.0) 2091 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.310179+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.313517+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.316578+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.316839+0000 mon.c (mon.2) 644 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.317529+0000 mon.a (mon.0) 2094 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.318777+0000 mon.c (mon.2) 645 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.325560+0000 mon.a (mon.0) 2095 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.340292+0000 mon.c (mon.2) 646 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.340682+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.341423+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.341760+0000 mon.a (mon.0) 2097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.342246+0000 mon.c (mon.2) 648 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:09 smithi203 bash[23654]: audit 2024-03-11T21:32:09.342567+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 10 client.30174.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 8 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 8 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a93e0 osd=3 3 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 3 8 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 4 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 10 client.30174.objecter _send_command 8 2024-03-11T21:32:09.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a41a8e60 con 0x7fe9a41afa00 2024-03-11T21:32:09.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a41a8e60 con 0x7fe9a41afa00 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 4 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 3 8 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:32:09.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 10 client.30174.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 9 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe9a41b3360 0x7fe9a41b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 9 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 4 9 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 4 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 10 client.30174.objecter _send_command 9 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41a8d80 con 0x7fe9a41b3360 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe9a41b3360 0x7fe9a41b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41b3360 2024-03-11T21:32:09.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.891+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe9a41b3360 0x7fe9a41b5750 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.895+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41a8d80 con 0x7fe9a41b3360 2024-03-11T21:32:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.895+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:32:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.895+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 9 = system:0 2024-03-11T21:32:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.895+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 4 2024-03-11T21:32:09.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.895+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 4 9 2024-03-11T21:32:09.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 10 client.30174.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 10 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 10 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a8c10 osd=4 3 2024-03-11T21:32:09.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 4 10 2024-03-11T21:32:09.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 4 2024-03-11T21:32:09.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 10 client.30174.objecter _send_command 10 2024-03-11T21:32:09.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.911+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a41a8d80 con 0x7fe9a41b3360 2024-03-11T21:32:09.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a41a8d80 con 0x7fe9a41b3360 2024-03-11T21:32:09.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 4 2024-03-11T21:32:09.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 4 10 2024-03-11T21:32:09.889 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:32:09.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 10 client.30174.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 11 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe9a41b61a0 0x7fe9a41b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 11 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 5 11 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 4 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 10 client.30174.objecter _send_command 11 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41bb790 con 0x7fe9a41b61a0 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe9a41b61a0 0x7fe9a41b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a3fff700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe9a41b61a0 0x7fe9a41b6600 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41b61a0 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41bb790 con 0x7fe9a41b61a0 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 11 = system:0 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 4 2024-03-11T21:32:09.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.915+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 5 11 2024-03-11T21:32:09.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 10 client.30174.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 12 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 12 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41a8d80 osd=5 3 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 5 12 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 4 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 10 client.30174.objecter _send_command 12 2024-03-11T21:32:09.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.931+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a41b6c50 con 0x7fe9a41b61a0 2024-03-11T21:32:09.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a41b6c50 con 0x7fe9a41b61a0 2024-03-11T21:32:09.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 4 2024-03-11T21:32:09.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 5 12 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 10 client.30174.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 13 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe9a41b7370 0x7fe9a41b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 13 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 6 13 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 4 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 10 client.30174.objecter _send_command 13 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41b7e20 con 0x7fe9a41b7370 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe9a41b7370 0x7fe9a41b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41b7370 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.935+0000 7fe9a9267700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe9a41b7370 0x7fe9a41b7790 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.939+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41b7e20 con 0x7fe9a41b7370 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.939+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:32:09.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.939+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 13 = system:0 2024-03-11T21:32:09.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.939+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 4 2024-03-11T21:32:09.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.939+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 6 13 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 10 client.30174.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 14 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 14 2024-03-11T21:32:09.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41b6c50 osd=6 3 2024-03-11T21:32:09.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 6 14 2024-03-11T21:32:09.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 4 2024-03-11T21:32:09.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 10 client.30174.objecter _send_command 14 2024-03-11T21:32:09.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a41b7e20 con 0x7fe9a41b7370 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a41b7e20 con 0x7fe9a41b7370 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 4 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.955+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 6 14 2024-03-11T21:32:09.932 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 10 client.30174.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 15 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe9a41b7f90 0x7fe9a41beb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 15 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 7 15 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 4 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 10 client.30174.objecter _send_command 15 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9a41c1b60 con 0x7fe9a41b7f90 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe9a41b7f90 0x7fe9a41beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 10 client.30174.objecter ms_handle_connect 0x7fe9a41b7f90 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a8a66700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe9a41b7f90 0x7fe9a41beb40 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9a41c1b60 con 0x7fe9a41b7f90 2024-03-11T21:32:09.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:32:09.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 15 = system:0 2024-03-11T21:32:09.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 4 2024-03-11T21:32:09.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.959+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 7 15 2024-03-11T21:32:09.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:08.500620+0000 mgr.y (mgr.14550) 368 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:09.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:08.742271+0000 osd.4 (osd.4) 187 : cluster 0 6.e scrub starts 2024-03-11T21:32:09.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:08.744410+0000 osd.4 (osd.4) 188 : cluster 0 6.e scrub ok 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.019830+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.207010+0000 mon.a (mon.0) 2089 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.305428+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.305573+0000 mon.a (mon.0) 2091 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.310179+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.313517+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.316578+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.316839+0000 mon.c (mon.2) 644 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.317529+0000 mon.a (mon.0) 2094 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.318777+0000 mon.c (mon.2) 645 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.325560+0000 mon.a (mon.0) 2095 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.340292+0000 mon.c (mon.2) 646 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.340682+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.341423+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.341760+0000 mon.a (mon.0) 2097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.342246+0000 mon.c (mon.2) 648 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[29284]: audit 2024-03-11T21:32:09.342567+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:08.500620+0000 mgr.y (mgr.14550) 368 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:08.742271+0000 osd.4 (osd.4) 187 : cluster 0 6.e scrub starts 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:08.744410+0000 osd.4 (osd.4) 188 : cluster 0 6.e scrub ok 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.019830+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.207010+0000 mon.a (mon.0) 2089 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.305428+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi143-71594-39"}]': finished 2024-03-11T21:32:09.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.305573+0000 mon.a (mon.0) 2091 : audit 1 from='client.? 172.21.15.143:0/3181800047' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.310179+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.313517+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.316578+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.316839+0000 mon.c (mon.2) 644 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.317529+0000 mon.a (mon.0) 2094 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.318777+0000 mon.c (mon.2) 645 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.325560+0000 mon.a (mon.0) 2095 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.340292+0000 mon.c (mon.2) 646 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.340682+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.341423+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.341760+0000 mon.a (mon.0) 2097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.342246+0000 mon.c (mon.2) 648 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:09 smithi143 bash[21512]: audit 2024-03-11T21:32:09.342567+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:09.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 10 client.30174.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:09.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign -1 16 2024-03-11T21:32:09.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 20 client.30174.objecter _get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_remove -1 16 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 20 client.30174.objecter get_session s=0x7fe9a41b7e20 osd=7 3 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 15 client.30174.objecter _session_command_op_assign 7 16 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 4 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 10 client.30174.objecter _send_command 16 2024-03-11T21:32:09.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.975+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe9a41c08f0 con 0x7fe9a41b7f90 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a1ffb700 1 -- 172.21.15.143:0/3174280466 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe9a41c08f0 con 0x7fe9a41b7f90 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a1ffb700 10 client.30174.objecter ms_dispatch 0x7fe9a4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a1ffb700 10 client.30174.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a1ffb700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 4 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a1ffb700 15 client.30174.objecter _session_command_op_remove 7 16 2024-03-11T21:32:09.954 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:32:09.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 librados: watch_flush enter 2024-03-11T21:32:09.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 librados: watch_flush exit 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.0 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe9a40629f0 msgr2=0x7fe9a4062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe9a40629f0 0x7fe9a4062dd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a419b150 osd=0 1 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.1 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe9a402c6b0 msgr2=0x7fe9a402ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe9a402c6b0 0x7fe9a402ca90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b600 osd=1 1 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.2 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe9a41abf50 msgr2=0x7fe9a41ae340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe9a41abf50 0x7fe9a41ae340 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a410b480 osd=2 1 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.3 2024-03-11T21:32:09.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe9a41afa00 msgr2=0x7fe9a41afe20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe9a41afa00 0x7fe9a41afe20 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a93e0 osd=3 1 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.4 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe9a41b3360 msgr2=0x7fe9a41b5750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe9a41b3360 0x7fe9a41b5750 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8c10 osd=4 1 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.5 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe9a41b61a0 msgr2=0x7fe9a41b6600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe9a41b61a0 0x7fe9a41b6600 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a9267700 1 -- 172.21.15.143:0/3174280466 reap_dead start 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41a8d80 osd=5 1 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.6 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe9a41b7370 msgr2=0x7fe9a41b7790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe9a41b7370 0x7fe9a41b7790 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a9267700 1 -- 172.21.15.143:0/3174280466 reap_dead start 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b6c50 osd=6 1 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter close_session for osd.7 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe9a41b7f90 msgr2=0x7fe9a41beb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe9a41b7f90 0x7fe9a41beb40 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter put_session s=0x7fe9a41b7e20 osd=7 1 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 20 client.30174.objecter shutdown clearing up homeless session... 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a9267700 1 -- 172.21.15.143:0/3174280466 reap_dead start 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 10 client.30174.objecter successfully canceled tick 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9a9267700 1 -- 172.21.15.143:0/3174280466 reap_dead start 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 msgr2=0x7fe994071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 0x7fe994071ce0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fe990009750 tx=0x7fe990009310 comp rx=0 tx=0).stop 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 msgr2=0x7fe9a419a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe99800e970 tx=0x7fe99800ec80 comp rx=0 tx=0).stop 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 shutdown_connections 2024-03-11T21:32:09.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9a4104c30 0x7fe9a419a330 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 --2- 172.21.15.143:0/3174280466 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe99406f820 0x7fe994071ce0 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 >> 172.21.15.143:0/3174280466 conn(0x7fe9a40ffb10 msgr2=0x7fe9a41001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 shutdown_connections 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 -- 172.21.15.143:0/3174280466 wait complete. 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:09.979+0000 7fe9aacca700 1 librados: shutdown 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:09.959 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:09.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:09.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:10.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:10.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:10.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:10.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:10.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:10.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:10.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:10.517 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:10.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:10.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:10.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:10.780 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:10.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:10.911 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:11.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:11.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.318300+0000 mgr.y (mgr.14550) 369 : cluster 0 pgmap v480: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:11.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.569948+0000 osd.6 (osd.6) 159 : cluster 0 2.14 scrub starts 2024-03-11T21:32:11.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.571251+0000 osd.6 (osd.6) 160 : cluster 0 2.14 scrub ok 2024-03-11T21:32:11.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.755562+0000 osd.5 (osd.5) 219 : cluster 0 3.1d scrub starts 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.759610+0000 osd.5 (osd.5) 220 : cluster 0 3.1d scrub ok 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.767448+0000 osd.4 (osd.4) 189 : cluster 0 5.1c scrub starts 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:09.768604+0000 osd.4 (osd.4) 190 : cluster 0 5.1c scrub ok 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:09.808943+0000 mon.a (mon.0) 2099 : audit 0 from='client.? 172.21.15.143:0/3174280466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.020728+0000 mon.c (mon.2) 649 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.308660+0000 mon.a (mon.0) 2100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.308903+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.309073+0000 mon.a (mon.0) 2102 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.309224+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.317655+0000 mon.a (mon.0) 2104 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.320466+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.322804+0000 mon.c (mon.2) 651 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.334550+0000 mon.a (mon.0) 2105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.336848+0000 mon.a (mon.0) 2106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.338413+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:10 smithi203 bash[23654]: audit 2024-03-11T21:32:10.341048+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.038 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.318300+0000 mgr.y (mgr.14550) 369 : cluster 0 pgmap v480: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.569948+0000 osd.6 (osd.6) 159 : cluster 0 2.14 scrub starts 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.571251+0000 osd.6 (osd.6) 160 : cluster 0 2.14 scrub ok 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.755562+0000 osd.5 (osd.5) 219 : cluster 0 3.1d scrub starts 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.759610+0000 osd.5 (osd.5) 220 : cluster 0 3.1d scrub ok 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.767448+0000 osd.4 (osd.4) 189 : cluster 0 5.1c scrub starts 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:09.768604+0000 osd.4 (osd.4) 190 : cluster 0 5.1c scrub ok 2024-03-11T21:32:11.106 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:09.808943+0000 mon.a (mon.0) 2099 : audit 0 from='client.? 172.21.15.143:0/3174280466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.020728+0000 mon.c (mon.2) 649 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.308660+0000 mon.a (mon.0) 2100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.308903+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.309073+0000 mon.a (mon.0) 2102 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.309224+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.317655+0000 mon.a (mon.0) 2104 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.320466+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.322804+0000 mon.c (mon.2) 651 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.334550+0000 mon.a (mon.0) 2105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.336848+0000 mon.a (mon.0) 2106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.338413+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.107 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[29284]: audit 2024-03-11T21:32:10.341048+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.318300+0000 mgr.y (mgr.14550) 369 : cluster 0 pgmap v480: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.569948+0000 osd.6 (osd.6) 159 : cluster 0 2.14 scrub starts 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.571251+0000 osd.6 (osd.6) 160 : cluster 0 2.14 scrub ok 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.755562+0000 osd.5 (osd.5) 219 : cluster 0 3.1d scrub starts 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.759610+0000 osd.5 (osd.5) 220 : cluster 0 3.1d scrub ok 2024-03-11T21:32:11.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.767448+0000 osd.4 (osd.4) 189 : cluster 0 5.1c scrub starts 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:09.768604+0000 osd.4 (osd.4) 190 : cluster 0 5.1c scrub ok 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:09.808943+0000 mon.a (mon.0) 2099 : audit 0 from='client.? 172.21.15.143:0/3174280466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.020728+0000 mon.c (mon.2) 649 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.308660+0000 mon.a (mon.0) 2100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.308903+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.309073+0000 mon.a (mon.0) 2102 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.309224+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi143-71594-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.317655+0000 mon.a (mon.0) 2104 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.320466+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.322804+0000 mon.c (mon.2) 651 : audit 1 from='client.? 172.21.15.143:0/2763754164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.334550+0000 mon.a (mon.0) 2105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:11.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.336848+0000 mon.a (mon.0) 2106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]: dispatch 2024-03-11T21:32:11.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.338413+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:10 smithi143 bash[21512]: audit 2024-03-11T21:32:10.341048+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:11.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:11.168 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:11.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:11.281 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:11.281 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:11.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:11.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:32:11.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:11.418 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:11.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:11.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:11.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.671 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:11.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- 172.21.15.143:0/3637472739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe8102a70 msgr2=0x7f8fe8106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:11.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- 172.21.15.143:0/3637472739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe8102a70 0x7f8fe8106a60 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f8fd80099c0 tx=0x7f8fd8009cd0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- 172.21.15.143:0/3637472739 shutdown_connections 2024-03-11T21:32:11.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- 172.21.15.143:0/3637472739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe8107140 0x7f8fe810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- 172.21.15.143:0/3637472739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe81020c0 0x7f8fe81024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- 172.21.15.143:0/3637472739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe8102a70 0x7f8fe8106a60 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- 172.21.15.143:0/3637472739 >> 172.21.15.143:0/3637472739 conn(0x7f8fe8078f00 msgr2=0x7f8fe8079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:11.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- 172.21.15.143:0/3637472739 shutdown_connections 2024-03-11T21:32:11.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- 172.21.15.143:0/3637472739 wait complete. 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 librados: starting msgr at 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 librados: starting objecter 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 Processor -- start 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- start start 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 librados: setting wanted keys 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 librados: calling monclient init 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe8102a70 0x7f8fe819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:11.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe8107140 0x7f8fe8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8111690 con 0x7f8fe81020c0 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8111510 con 0x7f8fe8107140 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.743+0000 7f8fee35b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8111810 con 0x7f8fe8102a70 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56120/0 (socket says 172.21.15.143:56120) 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 -- 172.21.15.143:0/1419334317 learned_addr learned my addr 172.21.15.143:0/1419334317 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fec8f8700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe8107140 0x7f8fe8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 -- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe8102a70 msgr2=0x7f8fe819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe8102a70 0x7f8fe819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 -- 172.21.15.143:0/1419334317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe8107140 msgr2=0x7f8fe8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe8107140 0x7f8fe8197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 -- 172.21.15.143:0/1419334317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe81981e0 con 0x7f8fe81020c0 2024-03-11T21:32:11.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe7fff700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c9b0 tx=0x7f8fdc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 10 client.?.objecter ms_handle_connect 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc007b00 con 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe81984d0 con 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8fdc007ca0 con 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc018660 con 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fee35b700 10 client.30186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe81a4790 con 0x7f8fe81020c0 2024-03-11T21:32:11.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fee35b700 1 librados: init done 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.747+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8fdc00f450 con 0x7f8fe81020c0 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 0x7f8fd0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(330..330 src has 87..330) v4 ==== 10230+0+0 (secure 0 0 0) 0x7f8fdc08ece0 con 0x7f8fe81020c0 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 10 client.30186.objecter ms_dispatch 0x7f8fe8001550 osd_map(330..330 src has 87..330) v4 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 3 client.30186.objecter handle_osd_map got epochs [330,330] > 0 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 3 client.30186.objecter handle_osd_map decoding full epoch 330 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe77fe700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 0x7f8fd0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:11.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 20 client.30186.objecter dump_active .. 0 homeless 2024-03-11T21:32:11.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe77fe700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 0x7f8fd0071ce0 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7f8fd80098e0 tx=0x7f8fd8000bc0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:11.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fe57fa700 10 client.30186.objecter ms_handle_connect 0x7f8fd006f820 2024-03-11T21:32:11.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.751+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe8103540 con 0x7f8fe81020c0 2024-03-11T21:32:11.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:11.763+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8fdc014030 con 0x7f8fe81020c0 2024-03-11T21:32:11.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.541578+0000 osd.7 (osd.7) 247 : cluster 0 2.5 scrub starts 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.543722+0000 osd.7 (osd.7) 248 : cluster 0 2.5 scrub ok 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.545432+0000 osd.6 (osd.6) 161 : cluster 0 2.1a scrub starts 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.547670+0000 osd.6 (osd.6) 162 : cluster 0 2.1a scrub ok 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.684808+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-03-11T21:32:11.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:10.686757+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.022111+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.263587+0000 mon.a (mon.0) 2108 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.268162+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.268313+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.279210+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.315886+0000 mon.c (mon.2) 655 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.739236+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.740344+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.740719+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:11.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.541578+0000 osd.7 (osd.7) 247 : cluster 0 2.5 scrub starts 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.543722+0000 osd.7 (osd.7) 248 : cluster 0 2.5 scrub ok 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.545432+0000 osd.6 (osd.6) 161 : cluster 0 2.1a scrub starts 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.547670+0000 osd.6 (osd.6) 162 : cluster 0 2.1a scrub ok 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.684808+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:10.686757+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.022111+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.263587+0000 mon.a (mon.0) 2108 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:11.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.268162+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.268313+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.279210+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.315886+0000 mon.c (mon.2) 655 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.739236+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.740344+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.740719+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.745905+0000 mon.c (mon.2) 656 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.746849+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.756545+0000 mon.c (mon.2) 657 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: audit 2024-03-11T21:32:11.764410+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.767207+0000 osd.2 (osd.2) 169 : cluster 0 5.6 scrub starts 2024-03-11T21:32:11.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.769131+0000 osd.2 (osd.2) 170 : cluster 0 5.6 scrub ok 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.541578+0000 osd.7 (osd.7) 247 : cluster 0 2.5 scrub starts 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.543722+0000 osd.7 (osd.7) 248 : cluster 0 2.5 scrub ok 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.545432+0000 osd.6 (osd.6) 161 : cluster 0 2.1a scrub starts 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.547670+0000 osd.6 (osd.6) 162 : cluster 0 2.1a scrub ok 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.684808+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:10.686757+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.022111+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.263587+0000 mon.a (mon.0) 2108 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.268162+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi143-71604-49"}]': finished 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.268313+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.279210+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.315886+0000 mon.c (mon.2) 655 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.739236+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.740344+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.740719+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.745905+0000 mon.c (mon.2) 656 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.746849+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.756545+0000 mon.c (mon.2) 657 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: audit 2024-03-11T21:32:11.764410+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.767207+0000 osd.2 (osd.2) 169 : cluster 0 5.6 scrub starts 2024-03-11T21:32:11.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:11 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.769131+0000 osd.2 (osd.2) 170 : cluster 0 5.6 scrub ok 2024-03-11T21:32:11.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:11.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.003+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8fe8198fb0 con 0x7f8fe81020c0 2024-03-11T21:32:12.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:12.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:12.071 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.745905+0000 mon.c (mon.2) 656 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.071 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.746849+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.071 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.756545+0000 mon.c (mon.2) 657 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:12.071 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: audit 2024-03-11T21:32:11.764410+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:12.071 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.767207+0000 osd.2 (osd.2) 169 : cluster 0 5.6 scrub starts 2024-03-11T21:32:12.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:11 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.769131+0000 osd.2 (osd.2) 170 : cluster 0 5.6 scrub ok 2024-03-11T21:32:12.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:32:12.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:12.179 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:12.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:12.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.271+0000 7f8fe57fa700 1 -- 172.21.15.143:0/1419334317 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v331) v1 ==== 101+0+0 (secure 0 0 0) 0x7f8fdc05b6d0 con 0x7f8fe81020c0 2024-03-11T21:32:12.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 10 librados: watch_flush enter 2024-03-11T21:32:12.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 10 librados: watch_flush exit 2024-03-11T21:32:12.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 20 client.30186.objecter shutdown clearing up homeless session... 2024-03-11T21:32:12.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 10 client.30186.objecter successfully canceled tick 2024-03-11T21:32:12.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 msgr2=0x7f8fd0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:12.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 0x7f8fd0071ce0 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7f8fd80098e0 tx=0x7f8fd8000bc0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 msgr2=0x7f8fe819d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:12.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c9b0 tx=0x7f8fdc00ccc0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 shutdown_connections 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd006f820 0x7f8fd0071ce0 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe81020c0 0x7f8fe819d8c0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe8107140 0x7f8fe8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 --2- 172.21.15.143:0/1419334317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe8102a70 0x7f8fe819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:12.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.275+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 >> 172.21.15.143:0/1419334317 conn(0x7f8fe8078f00 msgr2=0x7f8fe80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:12.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.283+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 shutdown_connections 2024-03-11T21:32:12.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.283+0000 7f8fee35b700 1 -- 172.21.15.143:0/1419334317 wait complete. 2024-03-11T21:32:12.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:12.283+0000 7f8fee35b700 1 librados: shutdown 2024-03-11T21:32:12.262 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:32:12.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:12.302 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:12.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:12.427 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:12.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:12.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:12.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:12.686 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:12.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:12.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.318820+0000 mgr.y (mgr.14550) 370 : cluster 0 pgmap v483: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:12.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.662181+0000 osd.3 (osd.3) 213 : cluster 0 9.c scrub starts 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.663525+0000 osd.3 (osd.3) 214 : cluster 0 9.c scrub ok 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.927196+0000 osd.1 (osd.1) 23 : cluster 0 2.15 scrub starts 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:11.929618+0000 osd.1 (osd.1) 24 : cluster 0 2.15 scrub ok 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.007443+0000 mon.a (mon.0) 2116 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.023700+0000 mon.c (mon.2) 658 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.269034+0000 mon.a (mon.0) 2117 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.271428+0000 mon.a (mon.0) 2118 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.271680+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.271816+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.272006+0000 mon.a (mon.0) 2121 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.280174+0000 mon.a (mon.0) 2122 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:32:12.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.280976+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.282125+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.289693+0000 mon.c (mon.2) 660 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[21512]: audit 2024-03-11T21:32:12.290328+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.318820+0000 mgr.y (mgr.14550) 370 : cluster 0 pgmap v483: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.662181+0000 osd.3 (osd.3) 213 : cluster 0 9.c scrub starts 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.663525+0000 osd.3 (osd.3) 214 : cluster 0 9.c scrub ok 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.927196+0000 osd.1 (osd.1) 23 : cluster 0 2.15 scrub starts 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:11.929618+0000 osd.1 (osd.1) 24 : cluster 0 2.15 scrub ok 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.007443+0000 mon.a (mon.0) 2116 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.023700+0000 mon.c (mon.2) 658 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.269034+0000 mon.a (mon.0) 2117 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:12.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.271428+0000 mon.a (mon.0) 2118 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.271680+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.271816+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.272006+0000 mon.a (mon.0) 2121 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.280174+0000 mon.a (mon.0) 2122 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.280976+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.282125+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.289693+0000 mon.c (mon.2) 660 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:12.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:12 smithi143 bash[29284]: audit 2024-03-11T21:32:12.290328+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:12.817 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:12.921 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.318820+0000 mgr.y (mgr.14550) 370 : cluster 0 pgmap v483: 285 pgs: 1 clean+premerge+peered, 32 unknown, 252 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.662181+0000 osd.3 (osd.3) 213 : cluster 0 9.c scrub starts 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.663525+0000 osd.3 (osd.3) 214 : cluster 0 9.c scrub ok 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.927196+0000 osd.1 (osd.1) 23 : cluster 0 2.15 scrub starts 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:11.929618+0000 osd.1 (osd.1) 24 : cluster 0 2.15 scrub ok 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.007443+0000 mon.a (mon.0) 2116 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.023700+0000 mon.c (mon.2) 658 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.269034+0000 mon.a (mon.0) 2117 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.271428+0000 mon.a (mon.0) 2118 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi143-71594-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.271680+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.271816+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi143-71604-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.272006+0000 mon.a (mon.0) 2121 : audit 1 from='client.? 172.21.15.143:0/1419334317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.280174+0000 mon.a (mon.0) 2122 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.280976+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.282125+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.289693+0000 mon.c (mon.2) 660 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:13.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:12 smithi203 bash[23654]: audit 2024-03-11T21:32:12.290328+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]: dispatch 2024-03-11T21:32:13.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:13.081 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:13.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:13.211 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:13.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:13.336 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:13.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:13.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:13.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:13.587 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:13.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:13.718 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:13.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:13.848 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:13.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.548317+0000 osd.6 (osd.6) 163 : cluster 0 5.11 scrub starts 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.550615+0000 osd.6 (osd.6) 164 : cluster 0 5.11 scrub ok 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.798709+0000 osd.2 (osd.2) 171 : cluster 0 5.8 scrub starts 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.801110+0000 osd.2 (osd.2) 172 : cluster 0 5.8 scrub ok 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.025118+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:13.273102+0000 mon.a (mon.0) 2125 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.275887+0000 mon.a (mon.0) 2126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]': finished 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.320153+0000 mon.c (mon.2) 662 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.422653+0000 mon.c (mon.2) 663 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: cluster 2024-03-11T21:32:13.739890+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.742697+0000 mon.a (mon.0) 2128 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:13.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[29284]: audit 2024-03-11T21:32:13.742977+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.548317+0000 osd.6 (osd.6) 163 : cluster 0 5.11 scrub starts 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.550615+0000 osd.6 (osd.6) 164 : cluster 0 5.11 scrub ok 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.798709+0000 osd.2 (osd.2) 171 : cluster 0 5.8 scrub starts 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.801110+0000 osd.2 (osd.2) 172 : cluster 0 5.8 scrub ok 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.025118+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:13.273102+0000 mon.a (mon.0) 2125 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.275887+0000 mon.a (mon.0) 2126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]': finished 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.320153+0000 mon.c (mon.2) 662 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.422653+0000 mon.c (mon.2) 663 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: cluster 2024-03-11T21:32:13.739890+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.742697+0000 mon.a (mon.0) 2128 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:13.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:13 smithi143 bash[21512]: audit 2024-03-11T21:32:13.742977+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:13.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.548317+0000 osd.6 (osd.6) 163 : cluster 0 5.11 scrub starts 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.550615+0000 osd.6 (osd.6) 164 : cluster 0 5.11 scrub ok 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.798709+0000 osd.2 (osd.2) 171 : cluster 0 5.8 scrub starts 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.801110+0000 osd.2 (osd.2) 172 : cluster 0 5.8 scrub ok 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.025118+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:13.273102+0000 mon.a (mon.0) 2125 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.275887+0000 mon.a (mon.0) 2126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-31", "mode": "writeback"}]': finished 2024-03-11T21:32:14.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.320153+0000 mon.c (mon.2) 662 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:14.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.422653+0000 mon.c (mon.2) 663 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:14.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: cluster 2024-03-11T21:32:13.739890+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:32:14.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.742697+0000 mon.a (mon.0) 2128 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T21:32:14.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:13 smithi203 bash[23654]: audit 2024-03-11T21:32:13.742977+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:14.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:14.110 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:14.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.242 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:32:14.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:14.502 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:14.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:14.633 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:14.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:14.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:14.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.997159+0000 osd.0 (osd.0) 203 : cluster 0 9.1c scrub starts 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:12.998638+0000 osd.0 (osd.0) 204 : cluster 0 9.1c scrub ok 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:13.319518+0000 mgr.y (mgr.14550) 371 : cluster 0 pgmap v486: 292 pgs: 1 peering, 6 creating+peering, 7 unknown, 278 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:13.458523+0000 osd.7 (osd.7) 249 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:13.461141+0000 osd.7 (osd.7) 250 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:32:15.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.026361+0000 mon.c (mon.2) 664 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.286746+0000 mon.a (mon.0) 2130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.287062+0000 mon.a (mon.0) 2131 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.287275+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.293981+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.298260+0000 mon.c (mon.2) 665 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.301166+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.305187+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:14 smithi203 bash[23654]: audit 2024-03-11T21:32:14.306203+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.019 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:15.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.150 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.997159+0000 osd.0 (osd.0) 203 : cluster 0 9.1c scrub starts 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:12.998638+0000 osd.0 (osd.0) 204 : cluster 0 9.1c scrub ok 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:13.319518+0000 mgr.y (mgr.14550) 371 : cluster 0 pgmap v486: 292 pgs: 1 peering, 6 creating+peering, 7 unknown, 278 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:13.458523+0000 osd.7 (osd.7) 249 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:13.461141+0000 osd.7 (osd.7) 250 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.026361+0000 mon.c (mon.2) 664 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.286746+0000 mon.a (mon.0) 2130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.287062+0000 mon.a (mon.0) 2131 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.287275+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:15.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.293981+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.298260+0000 mon.c (mon.2) 665 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.301166+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.305187+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[21512]: audit 2024-03-11T21:32:14.306203+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.997159+0000 osd.0 (osd.0) 203 : cluster 0 9.1c scrub starts 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:12.998638+0000 osd.0 (osd.0) 204 : cluster 0 9.1c scrub ok 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:13.319518+0000 mgr.y (mgr.14550) 371 : cluster 0 pgmap v486: 292 pgs: 1 peering, 6 creating+peering, 7 unknown, 278 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:13.458523+0000 osd.7 (osd.7) 249 : cluster 0 2.1c deep-scrub starts 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:13.461141+0000 osd.7 (osd.7) 250 : cluster 0 2.1c deep-scrub ok 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.026361+0000 mon.c (mon.2) 664 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.286746+0000 mon.a (mon.0) 2130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi143-71604-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.287062+0000 mon.a (mon.0) 2131 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.287275+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.293981+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.298260+0000 mon.c (mon.2) 665 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.301166+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]: dispatch 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.305187+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:14 smithi143 bash[29284]: audit 2024-03-11T21:32:14.306203+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:15.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:15.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:32:15.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:15.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:15.419 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:15.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:15.544 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:15.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:15.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:15.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:15.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:15.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:15.935 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:16.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.691757+0000 osd.4 (osd.4) 191 : cluster 0 39.b scrub starts 2024-03-11T21:32:16.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.692066+0000 osd.3 (osd.3) 215 : cluster 0 9.1 deep-scrub starts 2024-03-11T21:32:16.050 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.693581+0000 osd.4 (osd.4) 192 : cluster 0 39.b scrub ok 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.697104+0000 osd.3 (osd.3) 216 : cluster 0 9.1 deep-scrub ok 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.837046+0000 osd.5 (osd.5) 221 : cluster 0 118.13 scrub starts 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.839181+0000 osd.5 (osd.5) 222 : cluster 0 118.13 scrub ok 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: audit 2024-03-11T21:32:15.027808+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.287257+0000 mon.a (mon.0) 2136 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: audit 2024-03-11T21:32:15.289460+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: audit 2024-03-11T21:32:15.289695+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.299686+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: audit 2024-03-11T21:32:15.305453+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: audit 2024-03-11T21:32:15.305962+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.650103+0000 osd.4 (osd.4) 193 : cluster 0 9.16 scrub starts 2024-03-11T21:32:16.051 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:15 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.652307+0000 osd.4 (osd.4) 194 : cluster 0 9.16 scrub ok 2024-03-11T21:32:16.066 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:16.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.181 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.691757+0000 osd.4 (osd.4) 191 : cluster 0 39.b scrub starts 2024-03-11T21:32:16.181 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.692066+0000 osd.3 (osd.3) 215 : cluster 0 9.1 deep-scrub starts 2024-03-11T21:32:16.181 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.693581+0000 osd.4 (osd.4) 192 : cluster 0 39.b scrub ok 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.697104+0000 osd.3 (osd.3) 216 : cluster 0 9.1 deep-scrub ok 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.837046+0000 osd.5 (osd.5) 221 : cluster 0 118.13 scrub starts 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.839181+0000 osd.5 (osd.5) 222 : cluster 0 118.13 scrub ok 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: audit 2024-03-11T21:32:15.027808+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.287257+0000 mon.a (mon.0) 2136 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: audit 2024-03-11T21:32:15.289460+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: audit 2024-03-11T21:32:15.289695+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.299686+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: audit 2024-03-11T21:32:15.305453+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: audit 2024-03-11T21:32:15.305962+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.650103+0000 osd.4 (osd.4) 193 : cluster 0 9.16 scrub starts 2024-03-11T21:32:16.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.652307+0000 osd.4 (osd.4) 194 : cluster 0 9.16 scrub ok 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.691757+0000 osd.4 (osd.4) 191 : cluster 0 39.b scrub starts 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.692066+0000 osd.3 (osd.3) 215 : cluster 0 9.1 deep-scrub starts 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.693581+0000 osd.4 (osd.4) 192 : cluster 0 39.b scrub ok 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.697104+0000 osd.3 (osd.3) 216 : cluster 0 9.1 deep-scrub ok 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.837046+0000 osd.5 (osd.5) 221 : cluster 0 118.13 scrub starts 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.839181+0000 osd.5 (osd.5) 222 : cluster 0 118.13 scrub ok 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: audit 2024-03-11T21:32:15.027808+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:16.183 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.287257+0000 mon.a (mon.0) 2136 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: audit 2024-03-11T21:32:15.289460+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-31"}]': finished 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: audit 2024-03-11T21:32:15.289695+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.299686+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: audit 2024-03-11T21:32:15.305453+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.143:0/611202346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: audit 2024-03-11T21:32:15.305962+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]: dispatch 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.650103+0000 osd.4 (osd.4) 193 : cluster 0 9.16 scrub starts 2024-03-11T21:32:16.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:15 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.652307+0000 osd.4 (osd.4) 194 : cluster 0 9.16 scrub ok 2024-03-11T21:32:16.198 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:16.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.327 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:16.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:16.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:16.591 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:16.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:16.769 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:16.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:16.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:16.857 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:16.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.973706+0000 osd.0 (osd.0) 205 : cluster 0 39.f scrub starts 2024-03-11T21:32:16.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:14.975961+0000 osd.0 (osd.0) 206 : cluster 0 39.f scrub ok 2024-03-11T21:32:16.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.320294+0000 mgr.y (mgr.14550) 372 : cluster 0 pgmap v489: 292 pgs: 1 peering, 8 unknown, 283 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:32:16.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.564007+0000 osd.6 (osd.6) 165 : cluster 0 3.10 scrub starts 2024-03-11T21:32:16.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.567433+0000 osd.6 (osd.6) 166 : cluster 0 3.10 scrub ok 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.720714+0000 osd.3 (osd.3) 217 : cluster 0 39.e scrub starts 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.722438+0000 osd.3 (osd.3) 218 : cluster 0 39.e scrub ok 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.790184+0000 osd.2 (osd.2) 173 : cluster 0 39.1f scrub starts 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.793897+0000 osd.2 (osd.2) 174 : cluster 0 39.1f scrub ok 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.852808+0000 osd.5 (osd.5) 223 : cluster 0 5.15 scrub starts 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.854910+0000 osd.5 (osd.5) 224 : cluster 0 5.15 scrub ok 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.029214+0000 mon.c (mon.2) 669 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.297335+0000 mon.a (mon.0) 2141 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.301387+0000 mon.a (mon.0) 2142 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.307147+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.747267+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.748849+0000 mon.a (mon.0) 2144 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.750363+0000 mon.a (mon.0) 2145 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:16.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:16 smithi203 bash[23654]: audit 2024-03-11T21:32:16.750790+0000 mon.a (mon.0) 2146 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:16.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:16.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:17.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.973706+0000 osd.0 (osd.0) 205 : cluster 0 39.f scrub starts 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:14.975961+0000 osd.0 (osd.0) 206 : cluster 0 39.f scrub ok 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.320294+0000 mgr.y (mgr.14550) 372 : cluster 0 pgmap v489: 292 pgs: 1 peering, 8 unknown, 283 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.564007+0000 osd.6 (osd.6) 165 : cluster 0 3.10 scrub starts 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.567433+0000 osd.6 (osd.6) 166 : cluster 0 3.10 scrub ok 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.720714+0000 osd.3 (osd.3) 217 : cluster 0 39.e scrub starts 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.722438+0000 osd.3 (osd.3) 218 : cluster 0 39.e scrub ok 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.790184+0000 osd.2 (osd.2) 173 : cluster 0 39.1f scrub starts 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.793897+0000 osd.2 (osd.2) 174 : cluster 0 39.1f scrub ok 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.852808+0000 osd.5 (osd.5) 223 : cluster 0 5.15 scrub starts 2024-03-11T21:32:17.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.854910+0000 osd.5 (osd.5) 224 : cluster 0 5.15 scrub ok 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.029214+0000 mon.c (mon.2) 669 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.297335+0000 mon.a (mon.0) 2141 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.301387+0000 mon.a (mon.0) 2142 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.307147+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.747267+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.748849+0000 mon.a (mon.0) 2144 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.750363+0000 mon.a (mon.0) 2145 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[21512]: audit 2024-03-11T21:32:16.750790+0000 mon.a (mon.0) 2146 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:17.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.973706+0000 osd.0 (osd.0) 205 : cluster 0 39.f scrub starts 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:14.975961+0000 osd.0 (osd.0) 206 : cluster 0 39.f scrub ok 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.320294+0000 mgr.y (mgr.14550) 372 : cluster 0 pgmap v489: 292 pgs: 1 peering, 8 unknown, 283 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.564007+0000 osd.6 (osd.6) 165 : cluster 0 3.10 scrub starts 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.567433+0000 osd.6 (osd.6) 166 : cluster 0 3.10 scrub ok 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.720714+0000 osd.3 (osd.3) 217 : cluster 0 39.e scrub starts 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.722438+0000 osd.3 (osd.3) 218 : cluster 0 39.e scrub ok 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.790184+0000 osd.2 (osd.2) 173 : cluster 0 39.1f scrub starts 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.793897+0000 osd.2 (osd.2) 174 : cluster 0 39.1f scrub ok 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.852808+0000 osd.5 (osd.5) 223 : cluster 0 5.15 scrub starts 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.854910+0000 osd.5 (osd.5) 224 : cluster 0 5.15 scrub ok 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.029214+0000 mon.c (mon.2) 669 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.297335+0000 mon.a (mon.0) 2141 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi143-71594-40"}]': finished 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.301387+0000 mon.a (mon.0) 2142 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:32:17.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.307147+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.747267+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.748849+0000 mon.a (mon.0) 2144 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:17.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.750363+0000 mon.a (mon.0) 2145 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:17.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:16 smithi143 bash[29284]: audit 2024-03-11T21:32:16.750790+0000 mon.a (mon.0) 2146 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:17.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:17.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:17.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.357 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:17.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:17.489 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:17.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:17.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:17.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:17.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.945895+0000 osd.0 (osd.0) 207 : cluster 0 5.3 scrub starts 2024-03-11T21:32:17.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:15.949096+0000 osd.0 (osd.0) 208 : cluster 0 5.3 scrub ok 2024-03-11T21:32:17.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.759712+0000 osd.3 (osd.3) 219 : cluster 0 9.8 scrub starts 2024-03-11T21:32:17.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.760910+0000 osd.3 (osd.3) 220 : cluster 0 9.8 scrub ok 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.780998+0000 osd.2 (osd.2) 175 : cluster 0 5.b scrub starts 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.782396+0000 osd.2 (osd.2) 176 : cluster 0 5.b scrub ok 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.906599+0000 osd.0 (osd.0) 209 : cluster 0 6.0 scrub starts 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:16.907557+0000 osd.0 (osd.0) 210 : cluster 0 6.0 scrub ok 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.030689+0000 mon.c (mon.2) 671 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.300919+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.301051+0000 mon.a (mon.0) 2148 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: cluster 2024-03-11T21:32:17.305591+0000 mon.a (mon.0) 2149 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.306310+0000 mon.a (mon.0) 2150 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.312716+0000 mon.c (mon.2) 672 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.313464+0000 mon.a (mon.0) 2151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.315569+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.316389+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:17.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.321497+0000 mon.c (mon.2) 674 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:17.864 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:17 smithi203 bash[23654]: audit 2024-03-11T21:32:17.323334+0000 mon.a (mon.0) 2153 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:17.878 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:17.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.010 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:18.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.945895+0000 osd.0 (osd.0) 207 : cluster 0 5.3 scrub starts 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:15.949096+0000 osd.0 (osd.0) 208 : cluster 0 5.3 scrub ok 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.759712+0000 osd.3 (osd.3) 219 : cluster 0 9.8 scrub starts 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.760910+0000 osd.3 (osd.3) 220 : cluster 0 9.8 scrub ok 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.780998+0000 osd.2 (osd.2) 175 : cluster 0 5.b scrub starts 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.782396+0000 osd.2 (osd.2) 176 : cluster 0 5.b scrub ok 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.906599+0000 osd.0 (osd.0) 209 : cluster 0 6.0 scrub starts 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:16.907557+0000 osd.0 (osd.0) 210 : cluster 0 6.0 scrub ok 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.030689+0000 mon.c (mon.2) 671 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.300919+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.301051+0000 mon.a (mon.0) 2148 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: cluster 2024-03-11T21:32:17.305591+0000 mon.a (mon.0) 2149 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:32:18.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.306310+0000 mon.a (mon.0) 2150 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.312716+0000 mon.c (mon.2) 672 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.313464+0000 mon.a (mon.0) 2151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.315569+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.316389+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.321497+0000 mon.c (mon.2) 674 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[29284]: audit 2024-03-11T21:32:17.323334+0000 mon.a (mon.0) 2153 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.945895+0000 osd.0 (osd.0) 207 : cluster 0 5.3 scrub starts 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:15.949096+0000 osd.0 (osd.0) 208 : cluster 0 5.3 scrub ok 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.759712+0000 osd.3 (osd.3) 219 : cluster 0 9.8 scrub starts 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.760910+0000 osd.3 (osd.3) 220 : cluster 0 9.8 scrub ok 2024-03-11T21:32:18.126 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.780998+0000 osd.2 (osd.2) 175 : cluster 0 5.b scrub starts 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.782396+0000 osd.2 (osd.2) 176 : cluster 0 5.b scrub ok 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.906599+0000 osd.0 (osd.0) 209 : cluster 0 6.0 scrub starts 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:16.907557+0000 osd.0 (osd.0) 210 : cluster 0 6.0 scrub ok 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.030689+0000 mon.c (mon.2) 671 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.300919+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.301051+0000 mon.a (mon.0) 2148 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi143-71594-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: cluster 2024-03-11T21:32:17.305591+0000 mon.a (mon.0) 2149 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.306310+0000 mon.a (mon.0) 2150 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.312716+0000 mon.c (mon.2) 672 : audit 1 from='client.? 172.21.15.143:0/1388469412' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.313464+0000 mon.a (mon.0) 2151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.315569+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.316389+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.321497+0000 mon.c (mon.2) 674 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:18.127 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:17 smithi143 bash[21512]: audit 2024-03-11T21:32:17.323334+0000 mon.a (mon.0) 2153 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T21:32:18.137 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:18.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:18.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:18.391 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:18.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:18.513 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:18.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:18.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:18.641 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:18.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:18.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:18.905 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:19.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:19.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:19.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:19.160 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:19.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:19.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:19.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:19.409 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:19.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: cluster 2024-03-11T21:32:17.320946+0000 mgr.y (mgr.14550) 373 : cluster 0 pgmap v492: 283 pgs: 32 unknown, 1 clean+premerge+peered, 1 peering, 249 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: cluster 2024-03-11T21:32:17.941976+0000 osd.1 (osd.1) 25 : cluster 0 6.4 scrub starts 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: cluster 2024-03-11T21:32:17.944132+0000 osd.1 (osd.1) 26 : cluster 0 6.4 scrub ok 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.032472+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.313074+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.313328+0000 mon.a (mon.0) 2155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.313553+0000 mon.a (mon.0) 2156 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.331481+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: cluster 2024-03-11T21:32:18.771792+0000 mon.a (mon.0) 2157 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:32:19.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:19 smithi203 bash[23654]: audit 2024-03-11T21:32:18.773211+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:19.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:19.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: cluster 2024-03-11T21:32:17.320946+0000 mgr.y (mgr.14550) 373 : cluster 0 pgmap v492: 283 pgs: 32 unknown, 1 clean+premerge+peered, 1 peering, 249 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: cluster 2024-03-11T21:32:17.941976+0000 osd.1 (osd.1) 25 : cluster 0 6.4 scrub starts 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: cluster 2024-03-11T21:32:17.944132+0000 osd.1 (osd.1) 26 : cluster 0 6.4 scrub ok 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.032472+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.313074+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.313328+0000 mon.a (mon.0) 2155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.313553+0000 mon.a (mon.0) 2156 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.331481+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: cluster 2024-03-11T21:32:18.771792+0000 mon.a (mon.0) 2157 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[29284]: audit 2024-03-11T21:32:18.773211+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: cluster 2024-03-11T21:32:17.320946+0000 mgr.y (mgr.14550) 373 : cluster 0 pgmap v492: 283 pgs: 32 unknown, 1 clean+premerge+peered, 1 peering, 249 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: cluster 2024-03-11T21:32:17.941976+0000 osd.1 (osd.1) 25 : cluster 0 6.4 scrub starts 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: cluster 2024-03-11T21:32:17.944132+0000 osd.1 (osd.1) 26 : cluster 0 6.4 scrub ok 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.032472+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.313074+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi143-71604-50"}]': finished 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.313328+0000 mon.a (mon.0) 2155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.313553+0000 mon.a (mon.0) 2156 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T21:32:19.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.331481+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: cluster 2024-03-11T21:32:18.771792+0000 mon.a (mon.0) 2157 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:32:19.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:19 smithi143 bash[21512]: audit 2024-03-11T21:32:18.773211+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:19.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:19.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:19.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:19.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:20.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:20.053 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:20.066 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:20.066 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:20.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:20.182 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:20.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:18.511795+0000 mgr.y (mgr.14550) 374 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:18.864826+0000 osd.0 (osd.0) 211 : cluster 0 9.1d scrub starts 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:18.866694+0000 osd.0 (osd.0) 212 : cluster 0 9.1d scrub ok 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:18.955880+0000 osd.1 (osd.1) 27 : cluster 0 2.9 scrub starts 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:18.957063+0000 osd.1 (osd.1) 28 : cluster 0 2.9 scrub ok 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.034257+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.300 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.315344+0000 mon.a (mon.0) 2159 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.315444+0000 mon.a (mon.0) 2160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.319026+0000 mon.a (mon.0) 2161 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.321727+0000 mgr.y (mgr.14550) 375 : cluster 0 pgmap v495: 323 pgs: 6 creating+activating, 22 creating+peering, 43 unknown, 1 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.324515+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.325267+0000 mon.c (mon.2) 679 : audit 1 from='client.? 172.21.15.143:0/2224974498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.325494+0000 mon.a (mon.0) 2162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:19.327877+0000 mon.a (mon.0) 2163 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.433352+0000 osd.7 (osd.7) 251 : cluster 0 39.5 deep-scrub starts 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.435384+0000 osd.7 (osd.7) 252 : cluster 0 39.5 deep-scrub ok 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[29284]: audit 2024-03-11T21:32:20.036047+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.301 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:18.511795+0000 mgr.y (mgr.14550) 374 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:18.864826+0000 osd.0 (osd.0) 211 : cluster 0 9.1d scrub starts 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:18.866694+0000 osd.0 (osd.0) 212 : cluster 0 9.1d scrub ok 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:18.955880+0000 osd.1 (osd.1) 27 : cluster 0 2.9 scrub starts 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:18.957063+0000 osd.1 (osd.1) 28 : cluster 0 2.9 scrub ok 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.034257+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.315344+0000 mon.a (mon.0) 2159 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.315444+0000 mon.a (mon.0) 2160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.319026+0000 mon.a (mon.0) 2161 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.321727+0000 mgr.y (mgr.14550) 375 : cluster 0 pgmap v495: 323 pgs: 6 creating+activating, 22 creating+peering, 43 unknown, 1 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.324515+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.302 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.325267+0000 mon.c (mon.2) 679 : audit 1 from='client.? 172.21.15.143:0/2224974498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.325494+0000 mon.a (mon.0) 2162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:19.327877+0000 mon.a (mon.0) 2163 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.433352+0000 osd.7 (osd.7) 251 : cluster 0 39.5 deep-scrub starts 2024-03-11T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.435384+0000 osd.7 (osd.7) 252 : cluster 0 39.5 deep-scrub ok 2024-03-11T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:20 smithi143 bash[21512]: audit 2024-03-11T21:32:20.036047+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.318 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:20.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:20.448 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:20.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:18.511795+0000 mgr.y (mgr.14550) 374 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:20.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:18.864826+0000 osd.0 (osd.0) 211 : cluster 0 9.1d scrub starts 2024-03-11T21:32:20.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:18.866694+0000 osd.0 (osd.0) 212 : cluster 0 9.1d scrub ok 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:18.955880+0000 osd.1 (osd.1) 27 : cluster 0 2.9 scrub starts 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:18.957063+0000 osd.1 (osd.1) 28 : cluster 0 2.9 scrub ok 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.034257+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.315344+0000 mon.a (mon.0) 2159 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi143-71594-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.315444+0000 mon.a (mon.0) 2160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.319026+0000 mon.a (mon.0) 2161 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.321727+0000 mgr.y (mgr.14550) 375 : cluster 0 pgmap v495: 323 pgs: 6 creating+activating, 22 creating+peering, 43 unknown, 1 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.324515+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.325267+0000 mon.c (mon.2) 679 : audit 1 from='client.? 172.21.15.143:0/2224974498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.325494+0000 mon.a (mon.0) 2162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:19.327877+0000 mon.a (mon.0) 2163 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:20.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.433352+0000 osd.7 (osd.7) 251 : cluster 0 39.5 deep-scrub starts 2024-03-11T21:32:20.472 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.435384+0000 osd.7 (osd.7) 252 : cluster 0 39.5 deep-scrub ok 2024-03-11T21:32:20.472 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:20 smithi203 bash[23654]: audit 2024-03-11T21:32:20.036047+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:20.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:20.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:20.670 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:20.670 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:20.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:20.710 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:20.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:20.841 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:20.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:20.969 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:21.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:21.101 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:21.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:21.230 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:21.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:21.359 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:21.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.960179+0000 osd.5 (osd.5) 225 : cluster 0 5.f scrub starts 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:19.961468+0000 osd.5 (osd.5) 226 : cluster 0 5.f scrub ok 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:20.315426+0000 mon.a (mon.0) 2164 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:20.325646+0000 mon.a (mon.0) 2165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:20.325977+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:20.338620+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:20.341728+0000 mon.a (mon.0) 2167 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:20.352121+0000 mon.a (mon.0) 2168 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:20.979236+0000 osd.1 (osd.1) 29 : cluster 0 3.0 scrub starts 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: cluster 2024-03-11T21:32:20.982411+0000 osd.1 (osd.1) 30 : cluster 0 3.0 scrub ok 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:21.037844+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:21.103670+0000 mon.a (mon.0) 2169 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[29284]: audit 2024-03-11T21:32:21.107101+0000 mon.c (mon.2) 683 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:21.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.960179+0000 osd.5 (osd.5) 225 : cluster 0 5.f scrub starts 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:19.961468+0000 osd.5 (osd.5) 226 : cluster 0 5.f scrub ok 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:20.315426+0000 mon.a (mon.0) 2164 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:20.325646+0000 mon.a (mon.0) 2165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:20.325977+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:20.338620+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:20.341728+0000 mon.a (mon.0) 2167 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:20.352121+0000 mon.a (mon.0) 2168 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:20.979236+0000 osd.1 (osd.1) 29 : cluster 0 3.0 scrub starts 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: cluster 2024-03-11T21:32:20.982411+0000 osd.1 (osd.1) 30 : cluster 0 3.0 scrub ok 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:21.037844+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:21.103670+0000 mon.a (mon.0) 2169 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:21.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:21 smithi143 bash[21512]: audit 2024-03-11T21:32:21.107101+0000 mon.c (mon.2) 683 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:21.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:32:21.463 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:21.490 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:21.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.960179+0000 osd.5 (osd.5) 225 : cluster 0 5.f scrub starts 2024-03-11T21:32:21.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:19.961468+0000 osd.5 (osd.5) 226 : cluster 0 5.f scrub ok 2024-03-11T21:32:21.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:20.315426+0000 mon.a (mon.0) 2164 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:21.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:20.325646+0000 mon.a (mon.0) 2165 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:20.325977+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi143-71604-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:20.338620+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:20.341728+0000 mon.a (mon.0) 2167 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:20.352121+0000 mon.a (mon.0) 2168 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]: dispatch 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:20.979236+0000 osd.1 (osd.1) 29 : cluster 0 3.0 scrub starts 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: cluster 2024-03-11T21:32:20.982411+0000 osd.1 (osd.1) 30 : cluster 0 3.0 scrub ok 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:21.037844+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:21.103670+0000 mon.a (mon.0) 2169 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:21.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:21 smithi203 bash[23654]: audit 2024-03-11T21:32:21.107101+0000 mon.c (mon.2) 683 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:21.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:21.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:21.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:21.871 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:21.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:22.002 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:22.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:22.132 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:22.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:22.266 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:22.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:22.381 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: cluster 2024-03-11T21:32:21.268043+0000 mon.a (mon.0) 2170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.269766+0000 mon.a (mon.0) 2171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]': finished 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: cluster 2024-03-11T21:32:21.276513+0000 mon.a (mon.0) 2172 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.280474+0000 mon.a (mon.0) 2173 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.318385+0000 mon.a (mon.0) 2174 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.319848+0000 mon.a (mon.0) 2175 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.320333+0000 mon.a (mon.0) 2176 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: cluster 2024-03-11T21:32:21.322231+0000 mgr.y (mgr.14550) 376 : cluster 0 pgmap v498: 282 pgs: 1 peering, 6 creating+activating, 22 creating+peering, 253 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.322693+0000 mon.c (mon.2) 684 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.324028+0000 mon.a (mon.0) 2177 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.595416+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:21.595939+0000 mon.a (mon.0) 2178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:22 smithi203 bash[23654]: audit 2024-03-11T21:32:22.039743+0000 mon.c (mon.2) 686 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:22.397 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:22.430 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: cluster 2024-03-11T21:32:21.268043+0000 mon.a (mon.0) 2170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.269766+0000 mon.a (mon.0) 2171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]': finished 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: cluster 2024-03-11T21:32:21.276513+0000 mon.a (mon.0) 2172 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.280474+0000 mon.a (mon.0) 2173 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.318385+0000 mon.a (mon.0) 2174 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.319848+0000 mon.a (mon.0) 2175 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.320333+0000 mon.a (mon.0) 2176 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: cluster 2024-03-11T21:32:21.322231+0000 mgr.y (mgr.14550) 376 : cluster 0 pgmap v498: 282 pgs: 1 peering, 6 creating+activating, 22 creating+peering, 253 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.322693+0000 mon.c (mon.2) 684 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.324028+0000 mon.a (mon.0) 2177 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.595416+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:21.595939+0000 mon.a (mon.0) 2178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[21512]: audit 2024-03-11T21:32:22.039743+0000 mon.c (mon.2) 686 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: cluster 2024-03-11T21:32:21.268043+0000 mon.a (mon.0) 2170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.269766+0000 mon.a (mon.0) 2171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-33", "mode": "writeback"}]': finished 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: cluster 2024-03-11T21:32:21.276513+0000 mon.a (mon.0) 2172 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.280474+0000 mon.a (mon.0) 2173 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.318385+0000 mon.a (mon.0) 2174 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.319848+0000 mon.a (mon.0) 2175 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.320333+0000 mon.a (mon.0) 2176 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: cluster 2024-03-11T21:32:21.322231+0000 mgr.y (mgr.14550) 376 : cluster 0 pgmap v498: 282 pgs: 1 peering, 6 creating+activating, 22 creating+peering, 253 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.322693+0000 mon.c (mon.2) 684 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.324028+0000 mon.a (mon.0) 2177 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:32:22.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.595416+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:21.595939+0000 mon.a (mon.0) 2178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:22.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:22 smithi143 bash[29284]: audit 2024-03-11T21:32:22.039743+0000 mon.c (mon.2) 686 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:22.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:22.527 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:22.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/1730026682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0010e810 msgr2=0x7f7e00110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- 172.21.15.143:0/1730026682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0010e810 0x7f7e00110c00 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f7df800b0a0 tx=0x7f7df800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/1730026682 shutdown_connections 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- 172.21.15.143:0/1730026682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e00074010 0x7f7e000743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- 172.21.15.143:0/1730026682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0010e810 0x7f7e00110c00 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- 172.21.15.143:0/1730026682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074930 0x7f7e0010e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/1730026682 >> 172.21.15.143:0/1730026682 conn(0x7f7e00078f00 msgr2=0x7f7e00079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/1730026682 shutdown_connections 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/1730026682 wait complete. 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 librados: starting msgr at 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 librados: starting objecter 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 Processor -- start 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- start start 2024-03-11T21:32:22.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 librados: setting wanted keys 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 librados: calling monclient init 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 0x7f7e00199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001137e0 con 0x7f7e0010e810 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00113660 con 0x7f7e00074010 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00113960 con 0x7f7e00074930 2024-03-11T21:32:22.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e04e33700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58488/0 (socket says 172.21.15.143:58488) 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e04e33700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:39920/0 (socket says 172.21.15.143:39920) 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 -- 172.21.15.143:0/363734415 learned_addr learned my addr 172.21.15.143:0/363734415 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05634700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 0x7f7e00199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 -- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 msgr2=0x7f7e0019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 -- 172.21.15.143:0/363734415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 msgr2=0x7f7e00199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 0x7f7e00199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 -- 172.21.15.143:0/363734415 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0019ed90 con 0x7f7e0010e810 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05634700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 0x7f7e00199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e04e33700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e05e35700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f7df8007a20 tx=0x7f7df8007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:22.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7df67fc700 10 client.?.objecter ms_handle_connect 0x7f7e0010e810 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7df67fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8021020 con 0x7f7e0010e810 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0019f020 con 0x7f7e0010e810 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 10 client.30210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e001a6900 con 0x7f7e0010e810 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 librados: init done 2024-03-11T21:32:22.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7df8004c50 con 0x7f7e0010e810 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8005050 con 0x7f7e0010e810 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.594+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de4005120 con 0x7f7e0010e810 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7df8024020 con 0x7f7e0010e810 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 0x7f7dec071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(341..341 src has 87..341) v4 ==== 10215+0+0 (secure 0 0 0) 0x7f7df800f020 con 0x7f7e0010e810 2024-03-11T21:32:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 10 client.30210.objecter ms_dispatch 0x7f7e00001550 osd_map(341..341 src has 87..341) v4 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 3 client.30210.objecter handle_osd_map got epochs [341,341] > 0 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 3 client.30210.objecter handle_osd_map decoding full epoch 341 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7e05634700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 0x7f7dec071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 20 client.30210.objecter dump_active .. 0 homeless 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7df67fc700 10 client.30210.objecter ms_handle_connect 0x7f7dec06f820 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.598+0000 7f7e05634700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 0x7f7dec071ce0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009be0 tx=0x7f7df0009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:22.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.602+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7df802b070 con 0x7f7e0010e810 2024-03-11T21:32:22.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:22.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.777 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:22.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:22.834+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f7de40056e0 con 0x7f7e0010e810 2024-03-11T21:32:22.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:23.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:23.024 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:23.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:23.157 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:23.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.278+0000 7f7df67fc700 1 -- 172.21.15.143:0/363734415 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v342) v1 ==== 113+0+0 (secure 0 0 0) 0x7f7df805c2a0 con 0x7f7e0010e810 2024-03-11T21:32:23.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.282+0000 7f7e07898700 10 librados: watch_flush enter 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 10 librados: watch_flush exit 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 20 client.30210.objecter shutdown clearing up homeless session... 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 10 client.30210.objecter successfully canceled tick 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 msgr2=0x7f7dec071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 0x7f7dec071ce0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009be0 tx=0x7f7df0009260 comp rx=0 tx=0).stop 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 msgr2=0x7f7e0019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f7df8007a20 tx=0x7f7df8007a50 comp rx=0 tx=0).stop 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 shutdown_connections 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7dec06f820 0x7f7dec071ce0 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0010e810 0x7f7e0019e650 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00074010 0x7f7e00199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 --2- 172.21.15.143:0/363734415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e00074930 0x7f7e0019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 >> 172.21.15.143:0/363734415 conn(0x7f7e00078f00 msgr2=0x7f7e00106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 shutdown_connections 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 -- 172.21.15.143:0/363734415 wait complete. 2024-03-11T21:32:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:23.286+0000 7f7e07898700 1 librados: shutdown 2024-03-11T21:32:23.266 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:32:23.288 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:23.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:23.402 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.033887+0000 osd.5 (osd.5) 227 : cluster 0 118.8 scrub starts 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.035730+0000 osd.5 (osd.5) 228 : cluster 0 118.8 scrub ok 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.272611+0000 mon.a (mon.0) 2179 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.272782+0000 mon.a (mon.0) 2180 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.272979+0000 mon.a (mon.0) 2181 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.273145+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.278749+0000 mon.a (mon.0) 2183 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.279923+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.283128+0000 mon.a (mon.0) 2184 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.284409+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.284939+0000 mon.a (mon.0) 2186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.369783+0000 osd.7 (osd.7) 253 : cluster 0 3.15 scrub starts 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.373442+0000 osd.7 (osd.7) 254 : cluster 0 3.15 scrub ok 2024-03-11T21:32:23.403 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.806156+0000 osd.3 (osd.3) 221 : cluster 0 6.c scrub starts 2024-03-11T21:32:23.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.807719+0000 osd.3 (osd.3) 222 : cluster 0 6.c scrub ok 2024-03-11T21:32:23.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:22.837866+0000 mon.a (mon.0) 2187 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:32:23.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.986334+0000 osd.1 (osd.1) 31 : cluster 0 39.c scrub starts 2024-03-11T21:32:23.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: cluster 2024-03-11T21:32:22.987707+0000 osd.1 (osd.1) 32 : cluster 0 39.c scrub ok 2024-03-11T21:32:23.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:23 smithi203 bash[23654]: audit 2024-03-11T21:32:23.041230+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:23.414 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:23.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:23.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:23.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:23.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.033887+0000 osd.5 (osd.5) 227 : cluster 0 118.8 scrub starts 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.035730+0000 osd.5 (osd.5) 228 : cluster 0 118.8 scrub ok 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.272611+0000 mon.a (mon.0) 2179 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.272782+0000 mon.a (mon.0) 2180 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.272979+0000 mon.a (mon.0) 2181 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.273145+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.278749+0000 mon.a (mon.0) 2183 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.279923+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.283128+0000 mon.a (mon.0) 2184 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:23.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.284409+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.284939+0000 mon.a (mon.0) 2186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.369783+0000 osd.7 (osd.7) 253 : cluster 0 3.15 scrub starts 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.373442+0000 osd.7 (osd.7) 254 : cluster 0 3.15 scrub ok 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.806156+0000 osd.3 (osd.3) 221 : cluster 0 6.c scrub starts 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.807719+0000 osd.3 (osd.3) 222 : cluster 0 6.c scrub ok 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:22.837866+0000 mon.a (mon.0) 2187 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.986334+0000 osd.1 (osd.1) 31 : cluster 0 39.c scrub starts 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: cluster 2024-03-11T21:32:22.987707+0000 osd.1 (osd.1) 32 : cluster 0 39.c scrub ok 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[21512]: audit 2024-03-11T21:32:23.041230+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.033887+0000 osd.5 (osd.5) 227 : cluster 0 118.8 scrub starts 2024-03-11T21:32:23.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.035730+0000 osd.5 (osd.5) 228 : cluster 0 118.8 scrub ok 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.272611+0000 mon.a (mon.0) 2179 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.272782+0000 mon.a (mon.0) 2180 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.272979+0000 mon.a (mon.0) 2181 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.273145+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.278749+0000 mon.a (mon.0) 2183 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.279923+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.283128+0000 mon.a (mon.0) 2184 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]: dispatch 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.284409+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.284939+0000 mon.a (mon.0) 2186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]: dispatch 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.369783+0000 osd.7 (osd.7) 253 : cluster 0 3.15 scrub starts 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.373442+0000 osd.7 (osd.7) 254 : cluster 0 3.15 scrub ok 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.806156+0000 osd.3 (osd.3) 221 : cluster 0 6.c scrub starts 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.807719+0000 osd.3 (osd.3) 222 : cluster 0 6.c scrub ok 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:22.837866+0000 mon.a (mon.0) 2187 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:32:23.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.986334+0000 osd.1 (osd.1) 31 : cluster 0 39.c scrub starts 2024-03-11T21:32:23.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: cluster 2024-03-11T21:32:22.987707+0000 osd.1 (osd.1) 32 : cluster 0 39.c scrub ok 2024-03-11T21:32:23.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:23 smithi143 bash[29284]: audit 2024-03-11T21:32:23.041230+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:23.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:23.802 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:23.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:23.930 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:24.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:24.060 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:24.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:24.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:24.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:24.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:24.440 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:24.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.012566+0000 osd.0 (osd.0) 213 : cluster 0 5.1e scrub starts 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.013979+0000 osd.0 (osd.0) 214 : cluster 0 5.1e scrub ok 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.039469+0000 osd.5 (osd.5) 229 : cluster 0 3.c scrub starts 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.043645+0000 osd.5 (osd.5) 230 : cluster 0 3.c scrub ok 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.274237+0000 mon.a (mon.0) 2188 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.283192+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.283269+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.283360+0000 mon.a (mon.0) 2191 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.323044+0000 mgr.y (mgr.14550) 377 : cluster 0 pgmap v501: 282 pgs: 3 peering, 6 creating+activating, 273 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.711698+0000 mon.a (mon.0) 2192 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.730530+0000 osd.2 (osd.2) 177 : cluster 0 12.9 scrub starts 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: cluster 2024-03-11T21:32:23.731765+0000 osd.2 (osd.2) 178 : cluster 0 12.9 scrub ok 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.734661+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.734983+0000 mon.a (mon.0) 2193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.735756+0000 mon.c (mon.2) 690 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.736129+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.736608+0000 mon.c (mon.2) 691 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:23.736880+0000 mon.a (mon.0) 2195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:24 smithi203 bash[23654]: audit 2024-03-11T21:32:24.042587+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:24.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:24.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:24.697 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:24.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.012566+0000 osd.0 (osd.0) 213 : cluster 0 5.1e scrub starts 2024-03-11T21:32:24.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.013979+0000 osd.0 (osd.0) 214 : cluster 0 5.1e scrub ok 2024-03-11T21:32:24.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.039469+0000 osd.5 (osd.5) 229 : cluster 0 3.c scrub starts 2024-03-11T21:32:24.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.043645+0000 osd.5 (osd.5) 230 : cluster 0 3.c scrub ok 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.274237+0000 mon.a (mon.0) 2188 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.283192+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.283269+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.283360+0000 mon.a (mon.0) 2191 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.323044+0000 mgr.y (mgr.14550) 377 : cluster 0 pgmap v501: 282 pgs: 3 peering, 6 creating+activating, 273 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.711698+0000 mon.a (mon.0) 2192 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.730530+0000 osd.2 (osd.2) 177 : cluster 0 12.9 scrub starts 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: cluster 2024-03-11T21:32:23.731765+0000 osd.2 (osd.2) 178 : cluster 0 12.9 scrub ok 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.734661+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.734983+0000 mon.a (mon.0) 2193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.735756+0000 mon.c (mon.2) 690 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.736129+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.736608+0000 mon.c (mon.2) 691 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:23.736880+0000 mon.a (mon.0) 2195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[21512]: audit 2024-03-11T21:32:24.042587+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.012566+0000 osd.0 (osd.0) 213 : cluster 0 5.1e scrub starts 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.013979+0000 osd.0 (osd.0) 214 : cluster 0 5.1e scrub ok 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.039469+0000 osd.5 (osd.5) 229 : cluster 0 3.c scrub starts 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.043645+0000 osd.5 (osd.5) 230 : cluster 0 3.c scrub ok 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.274237+0000 mon.a (mon.0) 2188 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.283192+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.143:0/370200603' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi143-71594-41"}]': finished 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.283269+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-33"}]': finished 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.283360+0000 mon.a (mon.0) 2191 : audit 1 from='client.? 172.21.15.143:0/363734415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.323044+0000 mgr.y (mgr.14550) 377 : cluster 0 pgmap v501: 282 pgs: 3 peering, 6 creating+activating, 273 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:24.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.711698+0000 mon.a (mon.0) 2192 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.730530+0000 osd.2 (osd.2) 177 : cluster 0 12.9 scrub starts 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: cluster 2024-03-11T21:32:23.731765+0000 osd.2 (osd.2) 178 : cluster 0 12.9 scrub ok 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.734661+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.734983+0000 mon.a (mon.0) 2193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.735756+0000 mon.c (mon.2) 690 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.736129+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.736608+0000 mon.c (mon.2) 691 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:23.736880+0000 mon.a (mon.0) 2195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:24.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:24 smithi143 bash[29284]: audit 2024-03-11T21:32:24.042587+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:24.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:24.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:24.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:24.953 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:25.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:25.083 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:25.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:25.214 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:25.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:25.346 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:25.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.090179+0000 osd.5 (osd.5) 231 : cluster 0 6.a scrub starts 2024-03-11T21:32:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.092556+0000 osd.5 (osd.5) 232 : cluster 0 6.a scrub ok 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: audit 2024-03-11T21:32:24.296412+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: audit 2024-03-11T21:32:24.296643+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.304436+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: audit 2024-03-11T21:32:24.310748+0000 mon.c (mon.2) 693 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.663631+0000 osd.0 (osd.0) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.663981+0000 osd.0 (osd.0) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: audit 2024-03-11T21:32:24.755496+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.829525+0000 osd.3 (osd.3) 223 : cluster 0 2.1b scrub starts 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: cluster 2024-03-11T21:32:24.831521+0000 osd.3 (osd.3) 224 : cluster 0 2.1b scrub ok 2024-03-11T21:32:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:25 smithi203 bash[23654]: audit 2024-03-11T21:32:25.044071+0000 mon.c (mon.2) 694 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:25.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:25.577 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.090179+0000 osd.5 (osd.5) 231 : cluster 0 6.a scrub starts 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.092556+0000 osd.5 (osd.5) 232 : cluster 0 6.a scrub ok 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: audit 2024-03-11T21:32:24.296412+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: audit 2024-03-11T21:32:24.296643+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.304436+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: audit 2024-03-11T21:32:24.310748+0000 mon.c (mon.2) 693 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.663631+0000 osd.0 (osd.0) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.663981+0000 osd.0 (osd.0) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: audit 2024-03-11T21:32:24.755496+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.829525+0000 osd.3 (osd.3) 223 : cluster 0 2.1b scrub starts 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: cluster 2024-03-11T21:32:24.831521+0000 osd.3 (osd.3) 224 : cluster 0 2.1b scrub ok 2024-03-11T21:32:25.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[21512]: audit 2024-03-11T21:32:25.044071+0000 mon.c (mon.2) 694 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.090179+0000 osd.5 (osd.5) 231 : cluster 0 6.a scrub starts 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.092556+0000 osd.5 (osd.5) 232 : cluster 0 6.a scrub ok 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: audit 2024-03-11T21:32:24.296412+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi143-71604-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: audit 2024-03-11T21:32:24.296643+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi143-71594-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.304436+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: audit 2024-03-11T21:32:24.310748+0000 mon.c (mon.2) 693 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.663631+0000 osd.0 (osd.0) 215 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.663981+0000 osd.0 (osd.0) 216 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: audit 2024-03-11T21:32:24.755496+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.829525+0000 osd.3 (osd.3) 223 : cluster 0 2.1b scrub starts 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: cluster 2024-03-11T21:32:24.831521+0000 osd.3 (osd.3) 224 : cluster 0 2.1b scrub ok 2024-03-11T21:32:25.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:25 smithi143 bash[29284]: audit 2024-03-11T21:32:25.044071+0000 mon.c (mon.2) 694 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:25.601 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:25.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.728 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:25.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:25.855 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:25.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:32:25.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.985 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:26.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:26.113 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:26.199 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:26.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:26.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:26.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:26.371 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:26.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:26.484 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.046292+0000 osd.0 (osd.0) 217 : cluster 0 6.9 scrub starts 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.048692+0000 osd.0 (osd.0) 218 : cluster 0 6.9 scrub ok 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.324268+0000 mgr.y (mgr.14550) 378 : cluster 0 pgmap v504: 289 pgs: 40 unknown, 2 peering, 247 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: audit 2024-03-11T21:32:25.325005+0000 mon.c (mon.2) 695 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: audit 2024-03-11T21:32:25.341913+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.432749+0000 osd.7 (osd.7) 255 : cluster 0 39.13 scrub starts 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.434772+0000 osd.7 (osd.7) 256 : cluster 0 39.13 scrub ok 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.653590+0000 osd.4 (osd.4) 195 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.655673+0000 osd.4 (osd.4) 196 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.704195+0000 osd.6 (osd.6) 167 : cluster 0 12.5 scrub starts 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.705931+0000 osd.6 (osd.6) 168 : cluster 0 12.5 scrub ok 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:25.785773+0000 mon.a (mon.0) 2200 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:32:26.485 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: audit 2024-03-11T21:32:25.789043+0000 mon.a (mon.0) 2201 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.486 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: audit 2024-03-11T21:32:25.789642+0000 mon.a (mon.0) 2202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.486 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: audit 2024-03-11T21:32:26.045418+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:26.486 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:26 smithi203 bash[23654]: cluster 2024-03-11T21:32:26.269207+0000 mon.a (mon.0) 2203 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:26.501 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.046292+0000 osd.0 (osd.0) 217 : cluster 0 6.9 scrub starts 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.048692+0000 osd.0 (osd.0) 218 : cluster 0 6.9 scrub ok 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.324268+0000 mgr.y (mgr.14550) 378 : cluster 0 pgmap v504: 289 pgs: 40 unknown, 2 peering, 247 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: audit 2024-03-11T21:32:25.325005+0000 mon.c (mon.2) 695 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: audit 2024-03-11T21:32:25.341913+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.432749+0000 osd.7 (osd.7) 255 : cluster 0 39.13 scrub starts 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.434772+0000 osd.7 (osd.7) 256 : cluster 0 39.13 scrub ok 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.653590+0000 osd.4 (osd.4) 195 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:32:26.564 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.655673+0000 osd.4 (osd.4) 196 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.704195+0000 osd.6 (osd.6) 167 : cluster 0 12.5 scrub starts 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.705931+0000 osd.6 (osd.6) 168 : cluster 0 12.5 scrub ok 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:25.785773+0000 mon.a (mon.0) 2200 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: audit 2024-03-11T21:32:25.789043+0000 mon.a (mon.0) 2201 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: audit 2024-03-11T21:32:25.789642+0000 mon.a (mon.0) 2202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: audit 2024-03-11T21:32:26.045418+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[21512]: cluster 2024-03-11T21:32:26.269207+0000 mon.a (mon.0) 2203 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.046292+0000 osd.0 (osd.0) 217 : cluster 0 6.9 scrub starts 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.048692+0000 osd.0 (osd.0) 218 : cluster 0 6.9 scrub ok 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.324268+0000 mgr.y (mgr.14550) 378 : cluster 0 pgmap v504: 289 pgs: 40 unknown, 2 peering, 247 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: audit 2024-03-11T21:32:25.325005+0000 mon.c (mon.2) 695 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: audit 2024-03-11T21:32:25.341913+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.432749+0000 osd.7 (osd.7) 255 : cluster 0 39.13 scrub starts 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.434772+0000 osd.7 (osd.7) 256 : cluster 0 39.13 scrub ok 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.653590+0000 osd.4 (osd.4) 195 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.655673+0000 osd.4 (osd.4) 196 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.704195+0000 osd.6 (osd.6) 167 : cluster 0 12.5 scrub starts 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.705931+0000 osd.6 (osd.6) 168 : cluster 0 12.5 scrub ok 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:25.785773+0000 mon.a (mon.0) 2200 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: audit 2024-03-11T21:32:25.789043+0000 mon.a (mon.0) 2201 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: audit 2024-03-11T21:32:25.789642+0000 mon.a (mon.0) 2202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: audit 2024-03-11T21:32:26.045418+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:26.566 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:26 smithi143 bash[29284]: cluster 2024-03-11T21:32:26.269207+0000 mon.a (mon.0) 2203 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:26.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:26.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:26.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:26.762 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:26.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:26.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:26.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.020 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:27.124 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:27.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:27.285 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:27.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.413 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: cluster 2024-03-11T21:32:26.157361+0000 osd.5 (osd.5) 233 : cluster 0 6.7 scrub starts 2024-03-11T21:32:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: cluster 2024-03-11T21:32:26.159462+0000 osd.5 (osd.5) 234 : cluster 0 6.7 scrub ok 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.327184+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.327405+0000 mon.a (mon.0) 2205 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.327670+0000 mon.a (mon.0) 2206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: cluster 2024-03-11T21:32:26.343416+0000 mon.a (mon.0) 2207 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.347070+0000 mon.c (mon.2) 698 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.353307+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:26.366868+0000 mon.a (mon.0) 2209 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[21512]: audit 2024-03-11T21:32:27.047138+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: cluster 2024-03-11T21:32:26.157361+0000 osd.5 (osd.5) 233 : cluster 0 6.7 scrub starts 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: cluster 2024-03-11T21:32:26.159462+0000 osd.5 (osd.5) 234 : cluster 0 6.7 scrub ok 2024-03-11T21:32:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.327184+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.327405+0000 mon.a (mon.0) 2205 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.327670+0000 mon.a (mon.0) 2206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: cluster 2024-03-11T21:32:26.343416+0000 mon.a (mon.0) 2207 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.347070+0000 mon.c (mon.2) 698 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.353307+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:26.366868+0000 mon.a (mon.0) 2209 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:27 smithi143 bash[29284]: audit 2024-03-11T21:32:27.047138+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:27.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:27.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: cluster 2024-03-11T21:32:26.157361+0000 osd.5 (osd.5) 233 : cluster 0 6.7 scrub starts 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: cluster 2024-03-11T21:32:26.159462+0000 osd.5 (osd.5) 234 : cluster 0 6.7 scrub ok 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.327184+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi143-71594-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.327405+0000 mon.a (mon.0) 2205 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.327670+0000 mon.a (mon.0) 2206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: cluster 2024-03-11T21:32:26.343416+0000 mon.a (mon.0) 2207 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.347070+0000 mon.c (mon.2) 698 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.353307+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:26.366868+0000 mon.a (mon.0) 2209 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:27 smithi203 bash[23654]: audit 2024-03-11T21:32:27.047138+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:27.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:27.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:27.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:27.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:27.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:27.929 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:28.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.060 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:28.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 -- 172.21.15.143:0/3022652643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c102850 msgr2=0x7f656c10ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 --2- 172.21.15.143:0/3022652643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c102850 0x7f656c10ecc0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f6564009fc0 tx=0x7f656400b330 comp rx=0 tx=0).stop 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 -- 172.21.15.143:0/3022652643 shutdown_connections 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 --2- 172.21.15.143:0/3022652643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c102850 0x7f656c10ecc0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 --2- 172.21.15.143:0/3022652643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 0x7f656c108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 --2- 172.21.15.143:0/3022652643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c101e90 0x7f656c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 -- 172.21.15.143:0/3022652643 >> 172.21.15.143:0/3022652643 conn(0x7f656c078f40 msgr2=0x7f656c079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 -- 172.21.15.143:0/3022652643 shutdown_connections 2024-03-11T21:32:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.278+0000 7f6573c27700 1 -- 172.21.15.143:0/3022652643 wait complete. 2024-03-11T21:32:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 librados: starting msgr at 2024-03-11T21:32:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 librados: starting objecter 2024-03-11T21:32:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 Processor -- start 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- start start 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 librados: setting wanted keys 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 librados: calling monclient init 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c102850 0x7f656c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 0x7f656c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:60874/0 (socket says 172.21.15.143:60874) 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65721c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 0x7f656c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c111620 con 0x7f656c101e90 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1114a0 con 0x7f656c102850 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1117a0 con 0x7f656c107ea0 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 -- 172.21.15.143:0/1877963555 learned_addr learned my addr 172.21.15.143:0/1877963555 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65721c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 0x7f656c074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37484/0 (socket says 172.21.15.143:37484) 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c102850 0x7f656c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 msgr2=0x7f656c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f656c107ea0 0x7f656c074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c102850 msgr2=0x7f656c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c102850 0x7f656c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c1a5f10 con 0x7f656c101e90 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f656c102850 0x7f656c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f656800ec10 tx=0x7f656800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 10 client.?.objecter ms_handle_connect 0x7f656c101e90 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800cd40 con 0x7f656c101e90 2024-03-11T21:32:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f65680044a0 con 0x7f656c101e90 2024-03-11T21:32:28.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c1a6200 con 0x7f656c101e90 2024-03-11T21:32:28.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568010430 con 0x7f656c101e90 2024-03-11T21:32:28.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 10 client.30222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:28.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c1a6740 con 0x7f656c101e90 2024-03-11T21:32:28.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 librados: init done 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f656c04faa0 con 0x7f656c101e90 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.282+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f6568004000 con 0x7f656c101e90 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f655806f820 0x7f6558071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(346..346 src has 87..346) v4 ==== 10727+0+0 (secure 0 0 0) 0x7f656808f610 con 0x7f656c101e90 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f655806f820 0x7f6558071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 osd_map(346..346 src has 87..346) v4 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 3 client.30222.objecter handle_osd_map got epochs [346,346] > 0 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 3 client.30222.objecter handle_osd_map decoding full epoch 346 2024-03-11T21:32:28.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 20 client.30222.objecter dump_active .. 0 homeless 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v346) v1 ==== 54+0+16 (secure 0 0 0) 0x7f656808fa60 con 0x7f656c101e90 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f655806f820 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f655806f820 0x7f6558071ce0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f655c005fd0 tx=0x7f655c005e30 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 10 client.30222.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 1 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f656c0629f0 0x7f656c062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 1 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 0 1 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 4 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 10 client.30222.objecter _send_command 1 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1a9b90 con 0x7f656c0629f0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f656c0629f0 0x7f656c062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c0629f0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f656c0629f0 0x7f656c062dd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1a9b90 con 0x7f656c0629f0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 10 client.30222.objecter _finish_command 1 = system:0 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 4 2024-03-11T21:32:28.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.286+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 0 1 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 10 client.30222.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 2 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 2 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c075610 osd=0 3 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 0 2 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 4 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 10 client.30222.objecter _send_command 2 2024-03-11T21:32:28.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c062470 con 0x7f656c0629f0 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c062470 con 0x7f656c0629f0 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 10 client.30222.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 4 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 0 2 2024-03-11T21:32:28.283 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 10 client.30222.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 3 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f656c1098e0 0x7f656c02c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 3 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 1 3 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 4 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 10 client.30222.objecter _send_command 3 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1ad4f0 con 0x7f656c1098e0 2024-03-11T21:32:28.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f656c1098e0 0x7f656c02c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f656c1098e0 0x7f656c02c6b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1098e0 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1ad4f0 con 0x7f656c1098e0 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 10 client.30222.objecter _finish_command 3 = system:0 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 4 2024-03-11T21:32:28.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.306+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 1 3 2024-03-11T21:32:28.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:28.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 10 client.30222.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 4 2024-03-11T21:32:28.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 4 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c062870 osd=1 3 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 1 4 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 4 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 10 client.30222.objecter _send_command 4 2024-03-11T21:32:28.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c108e50 con 0x7f656c1098e0 2024-03-11T21:32:28.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c108e50 con 0x7f656c1098e0 2024-03-11T21:32:28.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6562ffd700 10 client.30222.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 4 2024-03-11T21:32:28.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 1 4 2024-03-11T21:32:28.304 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:32:28.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 -- 172.21.15.143:0/3874310429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806a440 msgr2=0x7fc80806a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/3874310429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806a440 0x7fc80806a820 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009990 tx=0x7fc7f8009ca0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 -- 172.21.15.143:0/3874310429 shutdown_connections 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/3874310429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80810e870 0x7fc808110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/3874310429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806a440 0x7fc80806a820 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/3874310429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80806adf0 0x7fc80810e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 -- 172.21.15.143:0/3874310429 >> 172.21.15.143:0/3874310429 conn(0x7fc808077de0 msgr2=0x7fc8080781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 -- 172.21.15.143:0/3874310429 shutdown_connections 2024-03-11T21:32:28.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 -- 172.21.15.143:0/3874310429 wait complete. 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 10 client.30222.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 5 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f656c1abd50 0x7f656c1ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 5 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 2 5 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 4 2024-03-11T21:32:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 10 client.30222.objecter _send_command 5 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1a9860 con 0x7f656c1abd50 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f656c1abd50 0x7f656c1ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 librados: starting msgr at 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.326+0000 7fc80d3ca700 1 librados: starting objecter 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f656c1abd50 0x7f656c1ac170 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1abd50 2024-03-11T21:32:28.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 Processor -- start 2024-03-11T21:32:28.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- start start 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 librados: setting wanted keys 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 librados: calling monclient init 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 0x7fc80819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80810e870 0x7fc80819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808113800 con 0x7fc80806a440 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808113680 con 0x7fc80810e870 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc808113980 con 0x7fc80806adf0 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc806ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 0x7fc80819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc806ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:60884/0 (socket says 172.21.15.143:60884) 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc806ffd700 1 -- 172.21.15.143:0/9485946 learned_addr learned my addr 172.21.15.143:0/9485946 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8077fe700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80810e870 0x7fc80819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 -- 172.21.15.143:0/9485946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80810e870 msgr2=0x7fc80819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80810e870 0x7fc80819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 -- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 msgr2=0x7fc808199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 -- 172.21.15.143:0/9485946 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80819edc0 con 0x7fc80806adf0 2024-03-11T21:32:28.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc806ffd700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:32:28.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1a9860 con 0x7f656c1abd50 2024-03-11T21:32:28.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:32:28.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 10 client.30222.objecter _finish_command 5 = system:0 2024-03-11T21:32:28.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 4 2024-03-11T21:32:28.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 2 5 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc8067fc700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 0x7fc80819a2f0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000d950 tx=0x7fc7f000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 10 client.?.objecter ms_handle_connect 0x7fc80806adf0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0017070 con 0x7fc80806adf0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc7f0009db0 con 0x7fc80806adf0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0010cd0 con 0x7fc80806adf0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80819f0b0 con 0x7fc80806adf0 2024-03-11T21:32:28.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 10 client.41429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808102ed0 con 0x7fc80806adf0 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 librados: init done 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80804faa0 con 0x7fc80806adf0 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc7f001a070 con 0x7fc80806adf0 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 0x7fc7f4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(346..346 src has 87..346) v4 ==== 10727+0+0 (secure 0 0 0) 0x7fc7f008e8e0 con 0x7fc80806adf0 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 10 client.41429.objecter ms_dispatch 0x7fc808001550 osd_map(346..346 src has 87..346) v4 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 3 client.41429.objecter handle_osd_map got epochs [346,346] > 0 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.330+0000 7fc7fffff700 3 client.41429.objecter handle_osd_map decoding full epoch 346 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.334+0000 7fc7fffff700 20 client.41429.objecter dump_active .. 0 homeless 2024-03-11T21:32:28.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.338+0000 7fc806ffd700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 0x7fc7f4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.338+0000 7fc7fffff700 10 client.41429.objecter ms_handle_connect 0x7fc7f406f820 2024-03-11T21:32:28.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.338+0000 7fc806ffd700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 0x7fc7f4071ce0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b2e0 tx=0x7fc7f8022040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.338+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc7f005b100 con 0x7fc80806adf0 2024-03-11T21:32:28.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:28.332 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: cluster 2024-03-11T21:32:27.325286+0000 mgr.y (mgr.14550) 379 : cluster 0 pgmap v506: 289 pgs: 2 creating+peering, 38 unknown, 1 peering, 248 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:28.332 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:27.336125+0000 mon.a (mon.0) 2210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:28.332 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:27.336210+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: cluster 2024-03-11T21:32:27.343290+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:27.343885+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:27.352006+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:27.353234+0000 mon.a (mon.0) 2214 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:28.049096+0000 mon.c (mon.2) 701 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:28.333 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[29284]: audit 2024-03-11T21:32:28.287442+0000 mon.a (mon.0) 2215 : audit 0 from='client.? 172.21.15.143:0/1877963555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 10 client.30222.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 6 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 6 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1a91e0 osd=2 3 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 2 6 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 4 2024-03-11T21:32:28.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 10 client.30222.objecter _send_command 6 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c1a8c60 con 0x7f656c1abd50 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c1a8c60 con 0x7f656c1abd50 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 10 client.30222.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 4 2024-03-11T21:32:28.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 2 6 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 10 client.30222.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 7 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f656c1af6b0 0x7f656c1b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 7 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 3 7 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 4 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 10 client.30222.objecter _send_command 7 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1b4900 con 0x7f656c1af6b0 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f656c1af6b0 0x7f656c1b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1af6b0 2024-03-11T21:32:28.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.366+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f656c1af6b0 0x7f656c1b1aa0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.370+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1b4900 con 0x7f656c1af6b0 2024-03-11T21:32:28.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.370+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:32:28.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.370+0000 7f6562ffd700 10 client.30222.objecter _finish_command 7 = system:0 2024-03-11T21:32:28.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.370+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 4 2024-03-11T21:32:28.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.370+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 3 7 2024-03-11T21:32:28.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 10 client.30222.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 8 2024-03-11T21:32:28.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 8 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1a8eb0 osd=3 3 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 3 8 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 4 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 10 client.30222.objecter _send_command 8 2024-03-11T21:32:28.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c1ac9c0 con 0x7f656c1af6b0 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c1ac9c0 con 0x7f656c1af6b0 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 10 client.30222.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 4 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 3 8 2024-03-11T21:32:28.374 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 10 client.30222.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 9 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f656c02d8d0 0x7f656c1b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 9 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 4 9 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 4 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 10 client.30222.objecter _send_command 9 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1aca00 con 0x7f656c02d8d0 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f656c02d8d0 0x7f656c1b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f656c02d8d0 0x7f656c1b5170 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c02d8d0 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1aca00 con 0x7f656c02d8d0 2024-03-11T21:32:28.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.398+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:32:28.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.402+0000 7f6562ffd700 10 client.30222.objecter _finish_command 9 = system:0 2024-03-11T21:32:28.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.402+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 4 2024-03-11T21:32:28.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.402+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 4 9 2024-03-11T21:32:28.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 10 client.30222.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 10 2024-03-11T21:32:28.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 10 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1ac710 osd=4 3 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 4 10 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 4 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 10 client.30222.objecter _send_command 10 2024-03-11T21:32:28.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c02ce40 con 0x7f656c02d8d0 2024-03-11T21:32:28.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c02ce40 con 0x7f656c02d8d0 2024-03-11T21:32:28.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6562ffd700 10 client.30222.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 4 2024-03-11T21:32:28.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 4 10 2024-03-11T21:32:28.396 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:32:28.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 10 client.30222.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 11 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f656c1b2cc0 0x7f656c1b6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 11 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 5 11 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 4 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 10 client.30222.objecter _send_command 11 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c02cd60 con 0x7f656c1b2cc0 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.418+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f656c1b2cc0 0x7f656c1b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1b2cc0 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f65711c2700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f656c1b2cc0 0x7f656c1b6830 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c02cd60 con 0x7f656c1b2cc0 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 10 client.30222.objecter _finish_command 11 = system:0 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 4 2024-03-11T21:32:28.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.422+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 5 11 2024-03-11T21:32:28.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:28.429 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 10 client.30222.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 12 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 12 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c02cbf0 osd=5 3 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 5 12 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 4 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 10 client.30222.objecter _send_command 12 2024-03-11T21:32:28.430 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c02cd60 con 0x7f656c1b2cc0 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c02cd60 con 0x7f656c1b2cc0 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6562ffd700 10 client.30222.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 4 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 5 12 2024-03-11T21:32:28.431 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 10 client.30222.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 13 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f656c1b7280 0x7f656c1b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 13 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 6 13 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 4 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 10 client.30222.objecter _send_command 13 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1b7d00 con 0x7f656c1b7280 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.454+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f656c1b7280 0x7f656c1b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1b7280 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f65721c4700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f656c1b7280 0x7f656c1b76e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1b7d00 con 0x7f656c1b7280 2024-03-11T21:32:28.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:32:28.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 10 client.30222.objecter _finish_command 13 = system:0 2024-03-11T21:32:28.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 4 2024-03-11T21:32:28.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.458+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 6 13 2024-03-11T21:32:28.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: cluster 2024-03-11T21:32:27.325286+0000 mgr.y (mgr.14550) 379 : cluster 0 pgmap v506: 289 pgs: 2 creating+peering, 38 unknown, 1 peering, 248 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:27.336125+0000 mon.a (mon.0) 2210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:27.336210+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: cluster 2024-03-11T21:32:27.343290+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:27.343885+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:27.352006+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:27.353234+0000 mon.a (mon.0) 2214 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:28.049096+0000 mon.c (mon.2) 701 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:28.442 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[23654]: audit 2024-03-11T21:32:28.287442+0000 mon.a (mon.0) 2215 : audit 0 from='client.? 172.21.15.143:0/1877963555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:28.446 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 10 client.30222.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 14 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 14 2024-03-11T21:32:28.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c02cd60 osd=6 3 2024-03-11T21:32:28.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 6 14 2024-03-11T21:32:28.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 4 2024-03-11T21:32:28.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 10 client.30222.objecter _send_command 14 2024-03-11T21:32:28.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c1bc080 con 0x7f656c1b7280 2024-03-11T21:32:28.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c1bc080 con 0x7f656c1b7280 2024-03-11T21:32:28.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6562ffd700 10 client.30222.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 4 2024-03-11T21:32:28.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 6 14 2024-03-11T21:32:28.456 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 10 client.30222.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 15 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f656c1bc700 0x7f656c1beaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 15 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 7 15 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 4 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 10 client.30222.objecter _send_command 15 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1c1a10 con 0x7f656c1bc700 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.478+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f656c1bc700 0x7f656c1beaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 10 client.30222.objecter ms_handle_connect 0x7f656c1bc700 2024-03-11T21:32:28.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f65719c3700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f656c1bc700 0x7f656c1beaf0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:28.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f656c1c1a10 con 0x7f656c1bc700 2024-03-11T21:32:28.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:32:28.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 10 client.30222.objecter _finish_command 15 = system:0 2024-03-11T21:32:28.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 4 2024-03-11T21:32:28.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.482+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 7 15 2024-03-11T21:32:28.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 10 client.30222.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:32:28.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign -1 16 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 20 client.30222.objecter _get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 15 client.30222.objecter _session_command_op_remove -1 16 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 20 client.30222.objecter get_session s=0x7f656c1bc080 osd=7 3 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 15 client.30222.objecter _session_command_op_assign 7 16 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 4 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 10 client.30222.objecter _send_command 16 2024-03-11T21:32:28.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.498+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f656c1c1a10 con 0x7f656c1bc700 2024-03-11T21:32:28.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6562ffd700 1 -- 172.21.15.143:0/1877963555 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c1c1a10 con 0x7f656c1bc700 2024-03-11T21:32:28.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6562ffd700 10 client.30222.objecter ms_dispatch 0x7f656c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:28.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6562ffd700 10 client.30222.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:28.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6562ffd700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 4 2024-03-11T21:32:28.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6562ffd700 15 client.30222.objecter _session_command_op_remove 7 16 2024-03-11T21:32:28.477 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:32:28.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 librados: watch_flush enter 2024-03-11T21:32:28.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 librados: watch_flush exit 2024-03-11T21:32:28.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.0 2024-03-11T21:32:28.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f656c0629f0 msgr2=0x7f656c062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f656c0629f0 0x7f656c062dd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c075610 osd=0 1 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.1 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f656c1098e0 msgr2=0x7f656c02c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f656c1098e0 0x7f656c02c6b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c062870 osd=1 1 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.2 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f656c1abd50 msgr2=0x7f656c1ac170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f656c1abd50 0x7f656c1ac170 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a91e0 osd=2 1 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.3 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f656c1af6b0 msgr2=0x7f656c1b1aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f656c1af6b0 0x7f656c1b1aa0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1a8eb0 osd=3 1 2024-03-11T21:32:28.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.4 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f656c02d8d0 msgr2=0x7f656c1b5170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f656c02d8d0 0x7f656c1b5170 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1ac710 osd=4 1 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.5 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f656c1b2cc0 msgr2=0x7f656c1b6830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f656c1b2cc0 0x7f656c1b6830 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cbf0 osd=5 1 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.6 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f656c1b7280 msgr2=0x7f656c1b76e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f656c1b7280 0x7f656c1b76e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c02cd60 osd=6 1 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter close_session for osd.7 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f656c1bc700 msgr2=0x7f656c1beaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f656c1bc700 0x7f656c1beaf0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter put_session s=0x7f656c1bc080 osd=7 1 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 20 client.30222.objecter shutdown clearing up homeless session... 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 10 client.30222.objecter successfully canceled tick 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f65721c4700 1 -- 172.21.15.143:0/1877963555 reap_dead start 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f65721c4700 1 -- 172.21.15.143:0/1877963555 reap_dead start 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f65721c4700 1 -- 172.21.15.143:0/1877963555 reap_dead start 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f65721c4700 1 -- 172.21.15.143:0/1877963555 reap_dead start 2024-03-11T21:32:28.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f655806f820 msgr2=0x7f6558071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f655806f820 0x7f6558071ce0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f655c005fd0 tx=0x7f655c005e30 comp rx=0 tx=0).stop 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f65721c4700 1 -- 172.21.15.143:0/1877963555 reap_dead start 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 msgr2=0x7f656c0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f656800ec10 tx=0x7f656800ef20 comp rx=0 tx=0).stop 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 shutdown_connections 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 --2- 172.21.15.143:0/1877963555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f656c101e90 0x7f656c0740e0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 >> 172.21.15.143:0/1877963555 conn(0x7f656c078f40 msgr2=0x7f656c105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 shutdown_connections 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 -- 172.21.15.143:0/1877963555 wait complete. 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.502+0000 7f6573c27700 1 librados: shutdown 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:28.482 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:28.483 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:28.483 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:28.483 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:28.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:28.577 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:28.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:28.602+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fc8081031f0 con 0x7fc80806adf0 2024-03-11T21:32:28.578 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:28.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: cluster 2024-03-11T21:32:27.325286+0000 mgr.y (mgr.14550) 379 : cluster 0 pgmap v506: 289 pgs: 2 creating+peering, 38 unknown, 1 peering, 248 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:27.336125+0000 mon.a (mon.0) 2210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:27.336210+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: cluster 2024-03-11T21:32:27.343290+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:27.343885+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]: dispatch 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:27.352006+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:27.353234+0000 mon.a (mon.0) 2214 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:28.049096+0000 mon.c (mon.2) 701 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:28.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:28 smithi143 bash[21512]: audit 2024-03-11T21:32:28.287442+0000 mon.a (mon.0) 2215 : audit 0 from='client.? 172.21.15.143:0/1877963555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:28.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:28.812 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:28.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:29.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:29.098 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:29.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:29.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:29.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:29.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.354+0000 7fc7fffff700 1 -- 172.21.15.143:0/9485946 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v348) v1 ==== 87+0+0 (secure 0 0 0) 0x7fc7f005eb50 con 0x7fc80806adf0 2024-03-11T21:32:29.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 10 librados: watch_flush enter 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 10 librados: watch_flush exit 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 20 client.41429.objecter shutdown clearing up homeless session... 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 10 client.41429.objecter successfully canceled tick 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 msgr2=0x7fc7f4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 0x7fc7f4071ce0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b2e0 tx=0x7fc7f8022040 comp rx=0 tx=0).stop 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 msgr2=0x7fc80819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:29.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 0x7fc80819a2f0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000d950 tx=0x7fc7f000dc60 comp rx=0 tx=0).stop 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 shutdown_connections 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc7f406f820 0x7fc7f4071ce0 secure :-1 s=CLOSED pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b2e0 tx=0x7fc7f8022040 comp rx=0 tx=0).stop 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc80806a440 0x7fc808199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc80810e870 0x7fc80819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 --2- 172.21.15.143:0/9485946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc80806adf0 0x7fc80819a2f0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.358+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 >> 172.21.15.143:0/9485946 conn(0x7fc808077de0 msgr2=0x7fc8080ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.362+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 shutdown_connections 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.362+0000 7fc80d3ca700 1 -- 172.21.15.143:0/9485946 wait complete. 2024-03-11T21:32:29.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:29.362+0000 7fc80d3ca700 1 librados: shutdown 2024-03-11T21:32:29.342 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:32:29.363 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:29.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:27.921078+0000 osd.3 (osd.3) 225 : cluster 0 39.18 scrub starts 2024-03-11T21:32:29.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:27.923312+0000 osd.3 (osd.3) 226 : cluster 0 39.18 scrub ok 2024-03-11T21:32:29.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.341995+0000 mon.a (mon.0) 2216 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:29.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.342072+0000 mon.a (mon.0) 2217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:29.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.346057+0000 mon.a (mon.0) 2218 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.347891+0000 mon.c (mon.2) 702 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.348242+0000 mon.c (mon.2) 703 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.348956+0000 mon.a (mon.0) 2219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.349243+0000 mon.a (mon.0) 2220 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.376252+0000 mon.c (mon.2) 704 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.376743+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.377311+0000 mon.c (mon.2) 705 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.377689+0000 mon.a (mon.0) 2222 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.378169+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.378543+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.523040+0000 mgr.y (mgr.14550) 380 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:27.921078+0000 osd.3 (osd.3) 225 : cluster 0 39.18 scrub starts 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:27.923312+0000 osd.3 (osd.3) 226 : cluster 0 39.18 scrub ok 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.341995+0000 mon.a (mon.0) 2216 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.342072+0000 mon.a (mon.0) 2217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.346057+0000 mon.a (mon.0) 2218 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.347891+0000 mon.c (mon.2) 702 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.348242+0000 mon.c (mon.2) 703 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.348956+0000 mon.a (mon.0) 2219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.349243+0000 mon.a (mon.0) 2220 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.376252+0000 mon.c (mon.2) 704 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.376743+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.377311+0000 mon.c (mon.2) 705 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.377689+0000 mon.a (mon.0) 2222 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.378169+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.378543+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.523040+0000 mgr.y (mgr.14550) 380 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.606392+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.143:0/9485946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:28.606780+0000 mon.a (mon.0) 2224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.776048+0000 osd.6 (osd.6) 169 : cluster 0 2.17 scrub starts 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.777618+0000 osd.6 (osd.6) 170 : cluster 0 2.17 scrub ok 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.779851+0000 osd.2 (osd.2) 179 : cluster 0 5.a scrub starts 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.975219+0000 osd.1 (osd.1) 33 : cluster 0 39.7 scrub starts 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.978716+0000 osd.1 (osd.1) 34 : cluster 0 39.7 scrub ok 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:29.050801+0000 mon.c (mon.2) 708 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:29.327204+0000 mon.c (mon.2) 709 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:29.329720+0000 mon.a (mon.0) 2225 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.342638+0000 mon.a (mon.0) 2226 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:29.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.342727+0000 mon.a (mon.0) 2227 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[21512]: audit 2024-03-11T21:32:29.345158+0000 mon.a (mon.0) 2228 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]': finished 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.606392+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.143:0/9485946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:28.606780+0000 mon.a (mon.0) 2224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.776048+0000 osd.6 (osd.6) 169 : cluster 0 2.17 scrub starts 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.777618+0000 osd.6 (osd.6) 170 : cluster 0 2.17 scrub ok 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.779851+0000 osd.2 (osd.2) 179 : cluster 0 5.a scrub starts 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.975219+0000 osd.1 (osd.1) 33 : cluster 0 39.7 scrub starts 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.978716+0000 osd.1 (osd.1) 34 : cluster 0 39.7 scrub ok 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:29.050801+0000 mon.c (mon.2) 708 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:29.327204+0000 mon.c (mon.2) 709 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.440 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:29.329720+0000 mon.a (mon.0) 2225 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.342638+0000 mon.a (mon.0) 2226 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.342727+0000 mon.a (mon.0) 2227 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:29.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:29 smithi143 bash[29284]: audit 2024-03-11T21:32:29.345158+0000 mon.a (mon.0) 2228 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]': finished 2024-03-11T21:32:29.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:29.490 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:29.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:29.603 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:27.921078+0000 osd.3 (osd.3) 225 : cluster 0 39.18 scrub starts 2024-03-11T21:32:29.603 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:27.923312+0000 osd.3 (osd.3) 226 : cluster 0 39.18 scrub ok 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.341995+0000 mon.a (mon.0) 2216 : audit 1 from='client.? 172.21.15.143:0/2469857422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi143-71604-52"}]': finished 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.342072+0000 mon.a (mon.0) 2217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.346057+0000 mon.a (mon.0) 2218 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.347891+0000 mon.c (mon.2) 702 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.348242+0000 mon.c (mon.2) 703 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.348956+0000 mon.a (mon.0) 2219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.349243+0000 mon.a (mon.0) 2220 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.376252+0000 mon.c (mon.2) 704 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.376743+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.377311+0000 mon.c (mon.2) 705 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.377689+0000 mon.a (mon.0) 2222 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:29.604 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.378169+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.378543+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.523040+0000 mgr.y (mgr.14550) 380 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.606392+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.143:0/9485946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:28.606780+0000 mon.a (mon.0) 2224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.776048+0000 osd.6 (osd.6) 169 : cluster 0 2.17 scrub starts 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.777618+0000 osd.6 (osd.6) 170 : cluster 0 2.17 scrub ok 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.779851+0000 osd.2 (osd.2) 179 : cluster 0 5.a scrub starts 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.975219+0000 osd.1 (osd.1) 33 : cluster 0 39.7 scrub starts 2024-03-11T21:32:29.605 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.978716+0000 osd.1 (osd.1) 34 : cluster 0 39.7 scrub ok 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:29.050801+0000 mon.c (mon.2) 708 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:29.327204+0000 mon.c (mon.2) 709 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:29.329720+0000 mon.a (mon.0) 2225 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.342638+0000 mon.a (mon.0) 2226 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.342727+0000 mon.a (mon.0) 2227 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:29.606 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:29 smithi203 bash[23654]: audit 2024-03-11T21:32:29.345158+0000 mon.a (mon.0) 2228 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-35", "mode": "writeback"}]': finished 2024-03-11T21:32:29.615 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:29.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.720 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24881.0:2 39.0 39.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 219.89488831700001, 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.450910398, 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24881", 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2563940939", 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.721 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845574+0000", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0280000000000001e-06 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845587+0000", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2883e-05 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845617+0000", 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9745000000000001e-05 2024-03-11T21:32:29.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845640+0000", 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3057e-05 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.296469+0000", 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4508295920000001 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.296481+0000", 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2092999999999999e-05 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24704.0:2 8.d 8:b26ffda6:test-rados-api-smithi143-71620-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 219.88149807600001, 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079922434000000001, 2024-03-11T21:32:29.725 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24704", 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2524399149", 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:32:29.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.727 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858964+0000", 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7319999999999998e-06 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858979+0000", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.483e-05 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.859005+0000", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5877e-05 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.859027+0000", 2024-03-11T21:32:29.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1950000000000002e-05 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.881850+0000", 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022822952000000001 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.881861+0000", 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1335000000000001e-05 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927087+0000", 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.045226508999999998 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927217+0000", 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000129985 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927327+0000", 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109405 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927718+0000", 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000391419 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927945+0000", 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022644399999999999 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938814+0000", 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010869838 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938852+0000", 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7960000000000002e-05 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938884+0000", 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1198000000000002e-05 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.735 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24610.0:2 18.b 18:d5d10dbd:test-rados-api-smithi143-71773-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 218.49026939199999, 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17578144600000001, 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24610", 2024-03-11T21:32:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2450754466", 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250192+0000", 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9530000000000002e-06 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250208+0000", 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6571000000000001e-05 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.252972+0000", 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027639729999999999 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.252988+0000", 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5387e-05 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.301037+0000", 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.048049341000000002 2024-03-11T21:32:29.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.301047+0000", 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6290000000000005e-06 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.391847+0000", 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.090799885999999996 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.392130+0000", 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028327700000000002 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.425971+0000", 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033841428999999999 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.741 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24706.0:1 44.6s0 44:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 217.42320489100001, 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1487131640000001, 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24706", 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1682110848", 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317256+0000", 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7039999999999999e-06 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317263+0000", 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1570000000000003e-06 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317277+0000", 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4049000000000001e-05 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317293+0000", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5897e-05 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317535+0000", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000241609 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317539+0000", 2024-03-11T21:32:29.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2599999999999999e-06 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.459236+0000", 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1416968569999995 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.459367+0000", 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130815 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.462593+0000", 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003226591 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.463579+0000", 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00098520500000000007 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.465848+0000", 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022699159999999999 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.465968+0000", 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119104 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25037.0:3 41.14 41:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:32:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 215.904845972, 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.639917509, 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25037", 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2135306702", 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835615+0000", 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5820000000000001e-06 2024-03-11T21:32:29.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835621+0000", 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1959999999999996e-06 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835638+0000", 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7345000000000001e-05 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835650+0000", 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1102e-05 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835724+0000", 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4890999999999998e-05 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835734+0000", 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9119999999999993e-06 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.475447+0000", 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6397126469999996 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.475531+0000", 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3833999999999997e-05 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24743.0:2061 13.b 13:d04d64e0:test-rados-api-smithi143-71756-16::big.copy:head [copy-from ver 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 215.75560614299999, 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.099396153000000001, 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24743", 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/4179355585", 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2061 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.754 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984855+0000", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0320000000000002e-06 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984867+0000", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1814000000000001e-05 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984897+0000", 2024-03-11T21:32:29.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9697e-05 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984911+0000", 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4501999999999999e-05 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984977+0000", 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6087999999999995e-05 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.985060+0000", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2614000000000006e-05 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.058952+0000", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073892422999999999 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.061358+0000", 2024-03-11T21:32:29.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024060510000000002 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084061+0000", 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022702994000000001 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084189+0000", 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012778 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084222+0000", 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3139000000000002e-05 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084249+0000", 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7019e-05 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25046.0:4 42.1f 42:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 210.87810256700001, 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6107376059999998, 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25046", 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/232344682", 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4 2024-03-11T21:32:29.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862359+0000", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3020000000000002e-06 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862365+0000", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4350000000000002e-06 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.865086+0000", 2024-03-11T21:32:29.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002720311 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.865103+0000", 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6869000000000001e-05 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.874460+0000", 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0093571869999999994 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.874473+0000", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3203999999999999e-05 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.472983+0000", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.598509559 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.473094+0000", 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111739 2024-03-11T21:32:29.765 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:11 99.a 99:5634a9cf:test-rados-api-smithi143-71838-9::foo:head [write 0~128] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 181.28101819400001, 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069531287999999997, 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:32:29.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 11 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459443+0000", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1289999999999999e-06 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459453+0000", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9820000000000003e-06 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459471+0000", 2024-03-11T21:32:29.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7869000000000001e-05 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459483+0000", 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2266000000000001e-05 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.460708+0000", 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012245190000000001 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.461176+0000", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000467863 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.461353+0000", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017721999999999999 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.475346+0000", 2024-03-11T21:32:29.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013993017999999999 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.485203+0000", 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098573140000000007 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528895+0000", 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043691299000000003 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528936+0000", 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1437000000000003e-05 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528972+0000", 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6371999999999997e-05 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.772 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:16 99.a 99:5634a9cf:test-rados-api-smithi143-71838-9::foo:head [write 128~128] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 180.643314613, 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.081470465000000006, 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 16 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097148+0000", 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9730000000000002e-06 2024-03-11T21:32:29.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097161+0000", 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3337e-05 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097189+0000", 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8345999999999999e-05 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.098332+0000", 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011424009999999999 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.099046+0000", 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000714626 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.099059+0000", 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3112000000000001e-05 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148091+0000", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.049031673999999997 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148268+0000", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017690800000000001 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148619+0000", 2024-03-11T21:32:29.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035063 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.153402+0000", 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047830279999999999 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.156195+0000", 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027934739999999998 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178525+0000", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022330236999999999 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178573+0000", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7401000000000002e-05 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178615+0000", 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2317999999999997e-05 2024-03-11T21:32:29.779 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34496.0:2 122.0 122:0384ab8e:::oid:head [write 0~0,setxattr key1 (5) in=4b,omap-clear] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 177.12387941, 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8526834349999999, 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34496", 2024-03-11T21:32:29.780 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1432393101", 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616582+0000", 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8080000000000001e-06 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616590+0000", 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4689999999999997e-06 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616604+0000", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4267000000000001e-05 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616616+0000", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2012e-05 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616820+0000", 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020306999999999999 2024-03-11T21:32:29.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616839+0000", 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9046e-05 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463381+0000", 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.846542758 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463537+0000", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015549499999999999 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463700+0000", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016336899999999999 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.464262+0000", 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00056225600000000002 2024-03-11T21:32:29.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.468013+0000", 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037509230000000002 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469133+0000", 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001119854 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469209+0000", 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5422999999999999e-05 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469263+0000", 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4685000000000002e-05 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.787 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23730.0:2 127.6 127:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.30982449000001, 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15423601100000001, 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23730", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/484288227", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430637+0000", 2024-03-11T21:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.843e-06 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430645+0000", 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6219999999999998e-06 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.431014+0000", 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00036898900000000001 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.431746+0000", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00073208599999999996 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.450995+0000", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019249097999999999 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.451006+0000", 2024-03-11T21:32:29.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1021e-05 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565207+0000", 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11420063699999999 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565343+0000", 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000135456 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565497+0000", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000154384 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565997+0000", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049974800000000005 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.566648+0000", 2024-03-11T21:32:29.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00065155900000000001 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584801+0000", 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018152435000000001 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584843+0000", 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2166000000000003e-05 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584871+0000", 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7966999999999998e-05 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:259 126.e 126:748fcfa5:::test_252:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e178)", 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.141087478, 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15545046300000001, 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.795 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 259 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599373+0000", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.497e-06 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599382+0000", 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5019999999999992e-06 2024-03-11T21:32:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599397+0000", 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5291e-05 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599404+0000", 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9829999999999999e-06 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599454+0000", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0043999999999997e-05 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599552+0000", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7511999999999997e-05 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.754822+0000", 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15527063399999999 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:390 126.14 126:2884fdb0:::test_383:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 170.202366063, 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.184484805, 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:32:29.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 390 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538094+0000", 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0049999999999999e-06 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538101+0000", 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5400000000000001e-06 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538115+0000", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3743e-05 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538122+0000", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7219999999999999e-06 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538186+0000", 2024-03-11T21:32:29.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3812999999999994e-05 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538309+0000", 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012344299999999999 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.722578+0000", 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18426853900000001 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.804 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:457 126.1f 126:fdf52ff4:::test_450:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 169.72108219899999, 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.22404676100000001, 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:32:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 457 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019378+0000", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0500000000000003e-07 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019381+0000", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2119999999999999e-06 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019388+0000", 2024-03-11T21:32:29.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019393+0000", 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.135e-06 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019441+0000", 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8386000000000001e-05 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019513+0000", 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1167000000000005e-05 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.243424+0000", 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.22391118400000001 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.809 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:53 99.12 99:4cecbdbb:test-rados-api-smithi143-71838-11::foo:head [stat,delete] snapc a=[a,9] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.61459531099999, 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088322633999999997, 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 53 2024-03-11T21:32:29.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125867+0000", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7369999999999998e-06 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125879+0000", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2309e-05 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125908+0000", 2024-03-11T21:32:29.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9062999999999999e-05 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125939+0000", 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1127000000000002e-05 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.126101+0000", 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000161924 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.126122+0000", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0843e-05 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181570+0000", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.055448099000000001 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181708+0000", 2024-03-11T21:32:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013806499999999999 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181990+0000", 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028172199999999998 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.184022+0000", 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002032491 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.196112+0000", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012089177 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214070+0000", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017958096 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214132+0000", 2024-03-11T21:32:29.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2129000000000001e-05 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214187+0000", 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4852000000000003e-05 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:109 118.10 118:0e6d2b0c:test-rados-api-smithi143-71773-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-11T21:32:29.817 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 163.15592179699999, 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.103370712, 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584540+0000", 2024-03-11T21:32:29.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4229999999999999e-06 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584554+0000", 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4113e-05 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584582+0000", 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7679e-05 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584607+0000", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5684e-05 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584990+0000", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000382471 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.585007+0000", 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7200000000000001e-05 2024-03-11T21:32:29.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.647849+0000", 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.062841852000000004 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.648005+0000", 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015635700000000001 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.648161+0000", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015576500000000001 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.676335+0000", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.028173650000000001 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.680739+0000", 2024-03-11T21:32:29.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004404771 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687805+0000", 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070651589999999997 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687863+0000", 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8025000000000001e-05 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687908+0000", 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5562999999999997e-05 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:773 118.13 118:ca30d8bc:test-rados-api-smithi143-71773-14::foo0:head [writefull 0~11] snapc 6=[6,5,4,3] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 139.10942201899999, 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100009098, 2024-03-11T21:32:29.825 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 773 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631041+0000", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6899999999999998e-06 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631060+0000", 2024-03-11T21:32:29.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8953e-05 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631093+0000", 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2598999999999997e-05 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631120+0000", 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7223e-05 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631322+0000", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000202394 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631330+0000", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2700000000000004e-06 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.652699+0000", 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021368809999999999 2024-03-11T21:32:29.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.652966+0000", 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000266419 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.653366+0000", 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00040020600000000001 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.656073+0000", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002707344 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.724089+0000", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.068015961999999999 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.730923+0000", 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0068343570000000001 2024-03-11T21:32:29.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.730997+0000", 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3328e-05 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.731046+0000", 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9543000000000003e-05 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40991.0:1 191.4s0 191:3d1c1d8f:test-rados-api-smithi143-71756-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 126.48913772100001, 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.011248849, 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40991", 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3284700860", 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.833 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251324+0000", 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2060000000000001e-06 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251335+0000", 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1275e-05 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251355+0000", 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0013999999999998e-05 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251371+0000", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5851000000000001e-05 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251472+0000", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101586 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251477+0000", 2024-03-11T21:32:29.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8509999999999998e-06 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247183+0000", 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99570569600000003 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247325+0000", 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000141528 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247806+0000", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00048166799999999998 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252955+0000", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051488669999999997 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.262484+0000", 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0095285509999999997 2024-03-11T21:32:29.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.262570+0000", 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6755999999999994e-05 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41141.0:1 215.6s0 215:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:32:29.839 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.492432378000004, 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.060282137, 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41141", 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2586124918", 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248030+0000", 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9699999999999999e-06 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248046+0000", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6478999999999999e-05 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248091+0000", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4422000000000003e-05 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248118+0000", 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7543e-05 2024-03-11T21:32:29.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.264982+0000", 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016863974 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.264990+0000", 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.306e-06 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.299012+0000", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0340223479999999 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.299309+0000", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000297416 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.300363+0000", 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001053153 2024-03-11T21:32:29.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.306158+0000", 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057951310000000002 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.308200+0000", 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020427129999999998 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.308309+0000", 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108682 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.29997.0:1 223.6s0 223:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 73.457613792999993, 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0641188610000001, 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:29.846 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.29997", 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2134772814", 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282847+0000", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9659999999999999e-06 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282854+0000", 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.99e-06 2024-03-11T21:32:29.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282867+0000", 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2959000000000001e-05 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282965+0000", 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7283999999999999e-05 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.283039+0000", 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4071000000000003e-05 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.283043+0000", 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4510000000000002e-06 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.340931+0000", 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0578876260000001 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.341105+0000", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017386499999999999 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.342266+0000", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011612949999999999 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.344603+0000", 2024-03-11T21:32:29.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023371720000000002 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.346859+0000", 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022555769999999999 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.346964+0000", 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105605 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.852 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.853 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:32:29.853 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:32:29.853 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:32:29.875 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:29.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:30.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:30.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:30.139 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:30.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:30.270 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:30.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAe 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6985 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6999 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7049 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7000 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6962 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7049 ms) 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-11T21:32:30.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7005 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6982 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6971 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7035 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6989 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6985 ms) 2024-03-11T21:32:30.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7041 ms) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6994 ms) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7406 ms) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6653 ms) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112105 ms total) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] Global test environment tear-down 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (234796 ms total) 2024-03-11T21:32:30.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-11T21:32:30.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:30.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:30.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:30.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:30.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:32:30.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:30.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:30.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:30.802 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:30.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:30.929 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:31.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.781320+0000 osd.2 (osd.2) 180 : cluster 0 5.a scrub ok 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.902000+0000 osd.3 (osd.3) 227 : cluster 0 3.b scrub starts 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:28.904814+0000 osd.3 (osd.3) 228 : cluster 0 3.b scrub ok 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.157643+0000 osd.5 (osd.5) 235 : cluster 0 2.18 scrub starts 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.159702+0000 osd.5 (osd.5) 236 : cluster 0 2.18 scrub ok 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.326529+0000 mgr.y (mgr.14550) 381 : cluster 0 pgmap v509: 280 pgs: 1 clean+premerge+peered, 2 creating+peering, 277 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.345375+0000 mon.a (mon.0) 2229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.345602+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.345847+0000 mon.a (mon.0) 2231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.346044+0000 mon.a (mon.0) 2232 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.358137+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.358573+0000 mon.c (mon.2) 711 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: cluster 2024-03-11T21:32:29.362178+0000 mon.a (mon.0) 2233 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.364636+0000 mon.a (mon.0) 2234 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:29.365274+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:30.052225+0000 mon.c (mon.2) 712 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[29284]: audit 2024-03-11T21:32:30.348036+0000 mon.a (mon.0) 2236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.781320+0000 osd.2 (osd.2) 180 : cluster 0 5.a scrub ok 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.902000+0000 osd.3 (osd.3) 227 : cluster 0 3.b scrub starts 2024-03-11T21:32:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:28.904814+0000 osd.3 (osd.3) 228 : cluster 0 3.b scrub ok 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.157643+0000 osd.5 (osd.5) 235 : cluster 0 2.18 scrub starts 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.159702+0000 osd.5 (osd.5) 236 : cluster 0 2.18 scrub ok 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.326529+0000 mgr.y (mgr.14550) 381 : cluster 0 pgmap v509: 280 pgs: 1 clean+premerge+peered, 2 creating+peering, 277 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.345375+0000 mon.a (mon.0) 2229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.345602+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.345847+0000 mon.a (mon.0) 2231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.346044+0000 mon.a (mon.0) 2232 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.358137+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.358573+0000 mon.c (mon.2) 711 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: cluster 2024-03-11T21:32:29.362178+0000 mon.a (mon.0) 2233 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.364636+0000 mon.a (mon.0) 2234 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:29.365274+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:30.052225+0000 mon.c (mon.2) 712 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:31.045 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:30 smithi143 bash[21512]: audit 2024-03-11T21:32:30.348036+0000 mon.a (mon.0) 2236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.054 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:31.158 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:31.185 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.781320+0000 osd.2 (osd.2) 180 : cluster 0 5.a scrub ok 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.902000+0000 osd.3 (osd.3) 227 : cluster 0 3.b scrub starts 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:28.904814+0000 osd.3 (osd.3) 228 : cluster 0 3.b scrub ok 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.157643+0000 osd.5 (osd.5) 235 : cluster 0 2.18 scrub starts 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.159702+0000 osd.5 (osd.5) 236 : cluster 0 2.18 scrub ok 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.326529+0000 mgr.y (mgr.14550) 381 : cluster 0 pgmap v509: 280 pgs: 1 clean+premerge+peered, 2 creating+peering, 277 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:31.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.345375+0000 mon.a (mon.0) 2229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.345602+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.345847+0000 mon.a (mon.0) 2231 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.346044+0000 mon.a (mon.0) 2232 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.358137+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.143:0/1535095685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.358573+0000 mon.c (mon.2) 711 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: cluster 2024-03-11T21:32:29.362178+0000 mon.a (mon.0) 2233 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.364636+0000 mon.a (mon.0) 2234 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]: dispatch 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:29.365274+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:30.052225+0000 mon.c (mon.2) 712 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:31.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:30 smithi203 bash[23654]: audit 2024-03-11T21:32:30.348036+0000 mon.a (mon.0) 2236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi143-71594-42"}]': finished 2024-03-11T21:32:31.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:31.311 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:31.415 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:31.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:31.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:31.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:31.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:31.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:31.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:32:31.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.962 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: cluster 2024-03-11T21:32:30.810288+0000 mon.a (mon.0) 2237 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:32:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: cluster 2024-03-11T21:32:31.043962+0000 osd.4 (osd.4) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: cluster 2024-03-11T21:32:31.044394+0000 osd.4 (osd.4) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:32.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.053576+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.274292+0000 mon.a (mon.0) 2238 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: cluster 2024-03-11T21:32:31.279189+0000 mon.a (mon.0) 2239 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.327779+0000 mon.c (mon.2) 714 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.330969+0000 mon.a (mon.0) 2240 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.381129+0000 mon.c (mon.2) 715 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:31 smithi203 bash[23654]: audit 2024-03-11T21:32:31.381486+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: cluster 2024-03-11T21:32:30.810288+0000 mon.a (mon.0) 2237 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: cluster 2024-03-11T21:32:31.043962+0000 osd.4 (osd.4) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: cluster 2024-03-11T21:32:31.044394+0000 osd.4 (osd.4) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.053576+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.274292+0000 mon.a (mon.0) 2238 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: cluster 2024-03-11T21:32:31.279189+0000 mon.a (mon.0) 2239 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.327779+0000 mon.c (mon.2) 714 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.330969+0000 mon.a (mon.0) 2240 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.381129+0000 mon.c (mon.2) 715 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[21512]: audit 2024-03-11T21:32:31.381486+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: cluster 2024-03-11T21:32:30.810288+0000 mon.a (mon.0) 2237 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:32:32.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: cluster 2024-03-11T21:32:31.043962+0000 osd.4 (osd.4) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: cluster 2024-03-11T21:32:31.044394+0000 osd.4 (osd.4) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.053576+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.274292+0000 mon.a (mon.0) 2238 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi143-71604-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: cluster 2024-03-11T21:32:31.279189+0000 mon.a (mon.0) 2239 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.327779+0000 mon.c (mon.2) 714 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.330969+0000 mon.a (mon.0) 2240 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.381129+0000 mon.c (mon.2) 715 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:31 smithi143 bash[29284]: audit 2024-03-11T21:32:31.381486+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:32.085 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:32.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.214 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:32.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.342 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:32.409 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:32.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:32.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:32:32.569 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:32.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:32.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.749 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:32.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:32.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: cluster 2024-03-11T21:32:31.327244+0000 mgr.y (mgr.14550) 382 : cluster 0 pgmap v513: 287 pgs: 8 unknown, 1 clean+premerge+peered, 278 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: audit 2024-03-11T21:32:32.054550+0000 mon.c (mon.2) 716 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: audit 2024-03-11T21:32:32.276797+0000 mon.a (mon.0) 2242 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: audit 2024-03-11T21:32:32.276891+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: cluster 2024-03-11T21:32:32.280429+0000 mon.a (mon.0) 2244 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: cluster 2024-03-11T21:32:32.286278+0000 mon.a (mon.0) 2245 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: audit 2024-03-11T21:32:32.313459+0000 mon.c (mon.2) 717 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[21512]: audit 2024-03-11T21:32:32.313889+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: cluster 2024-03-11T21:32:31.327244+0000 mgr.y (mgr.14550) 382 : cluster 0 pgmap v513: 287 pgs: 8 unknown, 1 clean+premerge+peered, 278 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: audit 2024-03-11T21:32:32.054550+0000 mon.c (mon.2) 716 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:32.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: audit 2024-03-11T21:32:32.276797+0000 mon.a (mon.0) 2242 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T21:32:32.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: audit 2024-03-11T21:32:32.276891+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:32.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: cluster 2024-03-11T21:32:32.280429+0000 mon.a (mon.0) 2244 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:32:32.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: cluster 2024-03-11T21:32:32.286278+0000 mon.a (mon.0) 2245 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:32.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: audit 2024-03-11T21:32:32.313459+0000 mon.c (mon.2) 717 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:32.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:32 smithi143 bash[29284]: audit 2024-03-11T21:32:32.313889+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:33.005 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:33.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:33.120 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: cluster 2024-03-11T21:32:31.327244+0000 mgr.y (mgr.14550) 382 : cluster 0 pgmap v513: 287 pgs: 8 unknown, 1 clean+premerge+peered, 278 active+clean; 4.6 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:32:33.120 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: audit 2024-03-11T21:32:32.054550+0000 mon.c (mon.2) 716 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:33.120 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: audit 2024-03-11T21:32:32.276797+0000 mon.a (mon.0) 2242 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T21:32:33.121 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: audit 2024-03-11T21:32:32.276891+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:33.121 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: cluster 2024-03-11T21:32:32.280429+0000 mon.a (mon.0) 2244 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:32:33.121 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: cluster 2024-03-11T21:32:32.286278+0000 mon.a (mon.0) 2245 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:33.121 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: audit 2024-03-11T21:32:32.313459+0000 mon.c (mon.2) 717 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:33.121 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:32 smithi203 bash[23654]: audit 2024-03-11T21:32:32.313889+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:33.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:33.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:33.273 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:33.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:33.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:33.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:33.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:33.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:33.665 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:33.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:33.793 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:33.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:33.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:33.963 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[21512]: audit 2024-03-11T21:32:33.055780+0000 mon.c (mon.2) 718 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:33.964 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[21512]: audit 2024-03-11T21:32:33.280098+0000 mon.a (mon.0) 2247 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:33.964 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[21512]: cluster 2024-03-11T21:32:33.287649+0000 mon.a (mon.0) 2248 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:32:33.964 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[21512]: audit 2024-03-11T21:32:33.295707+0000 mon.c (mon.2) 719 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:33.964 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[21512]: audit 2024-03-11T21:32:33.297544+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:34.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:34.054 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:34.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:34.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:34.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:33 smithi203 bash[23654]: audit 2024-03-11T21:32:33.055780+0000 mon.c (mon.2) 718 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:34.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:33 smithi203 bash[23654]: audit 2024-03-11T21:32:33.280098+0000 mon.a (mon.0) 2247 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:34.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:33 smithi203 bash[23654]: cluster 2024-03-11T21:32:33.287649+0000 mon.a (mon.0) 2248 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:32:34.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:33 smithi203 bash[23654]: audit 2024-03-11T21:32:33.295707+0000 mon.c (mon.2) 719 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:34.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:33 smithi203 bash[23654]: audit 2024-03-11T21:32:33.297544+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:34.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:34.318 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:34.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:34.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[29284]: audit 2024-03-11T21:32:33.055780+0000 mon.c (mon.2) 718 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:34.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[29284]: audit 2024-03-11T21:32:33.280098+0000 mon.a (mon.0) 2247 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:34.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[29284]: cluster 2024-03-11T21:32:33.287649+0000 mon.a (mon.0) 2248 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:32:34.434 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[29284]: audit 2024-03-11T21:32:33.295707+0000 mon.c (mon.2) 719 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:34.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:33 smithi143 bash[29284]: audit 2024-03-11T21:32:33.297544+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:34.449 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:34.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:34.579 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:34.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:34.709 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:34.812 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:34.838 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:34.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:34.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: cluster 2024-03-11T21:32:33.328314+0000 mgr.y (mgr.14550) 383 : cluster 0 pgmap v516: 279 pgs: 2 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-11T21:32:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.057166+0000 mon.c (mon.2) 720 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.290657+0000 mon.a (mon.0) 2250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.295015+0000 mon.c (mon.2) 721 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: cluster 2024-03-11T21:32:34.298832+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:32:35.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.300672+0000 mon.a (mon.0) 2252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.362012+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:34 smithi203 bash[23654]: audit 2024-03-11T21:32:34.362648+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:35.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: cluster 2024-03-11T21:32:33.328314+0000 mgr.y (mgr.14550) 383 : cluster 0 pgmap v516: 279 pgs: 2 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-11T21:32:35.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.057166+0000 mon.c (mon.2) 720 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.290657+0000 mon.a (mon.0) 2250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.295015+0000 mon.c (mon.2) 721 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: cluster 2024-03-11T21:32:34.298832+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.300672+0000 mon.a (mon.0) 2252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.362012+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[21512]: audit 2024-03-11T21:32:34.362648+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: cluster 2024-03-11T21:32:33.328314+0000 mgr.y (mgr.14550) 383 : cluster 0 pgmap v516: 279 pgs: 2 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.057166+0000 mon.c (mon.2) 720 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.290657+0000 mon.a (mon.0) 2250 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.295015+0000 mon.c (mon.2) 721 : audit 1 from='client.? 172.21.15.143:0/126623859' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: cluster 2024-03-11T21:32:34.298832+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:32:35.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.300672+0000 mon.a (mon.0) 2252 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]: dispatch 2024-03-11T21:32:35.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.362012+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:34 smithi143 bash[29284]: audit 2024-03-11T21:32:34.362648+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:35.092 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:35.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/626092967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 msgr2=0x7fe19010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- 172.21.15.143:0/626092967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19010dc70 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fe188009fc0 tx=0x7fe18800b330 comp rx=0 tx=0).stop 2024-03-11T21:32:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/626092967 shutdown_connections 2024-03-11T21:32:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- 172.21.15.143:0/626092967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19010dc70 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- 172.21.15.143:0/626092967 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 0x7fe1901013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- 172.21.15.143:0/626092967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe1901059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/626092967 >> 172.21.15.143:0/626092967 conn(0x7fe190077de0 msgr2=0x7fe1900781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/626092967 shutdown_connections 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/626092967 wait complete. 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 librados: starting msgr at 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 librados: starting objecter 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 Processor -- start 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- start start 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 librados: setting wanted keys 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 librados: calling monclient init 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 0x7fe190199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe19019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:35.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1901105e0 con 0x7fe1901060a0 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe190110460 con 0x7fe190100fc0 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe190110760 con 0x7fe190101970 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33398/0 (socket says 172.21.15.143:33398) 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 -- 172.21.15.143:0/496519948 learned_addr learned my addr 172.21.15.143:0/496519948 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe19650f700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 0x7fe190199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 -- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 msgr2=0x7fe19019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:35.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe195d0e700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe19019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe19019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 -- 172.21.15.143:0/496519948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 msgr2=0x7fe190199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 0x7fe190199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 -- 172.21.15.143:0/496519948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19019ee70 con 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe195d0e700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe19019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe196d10700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fe18800cc30 tx=0x7fe18800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe1877fe700 10 client.?.objecter ms_handle_connect 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe1877fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188021070 con 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19019f100 con 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe188007c00 con 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 10 client.30240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880052a0 con 0x7fe1901060a0 2024-03-11T21:32:35.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1901a69e0 con 0x7fe1901060a0 2024-03-11T21:32:35.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.118+0000 7fe198773700 1 librados: init done 2024-03-11T21:32:35.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19004faa0 con 0x7fe1901060a0 2024-03-11T21:32:35.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe188024070 con 0x7fe1901060a0 2024-03-11T21:32:35.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 0x7fe17c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:35.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe19650f700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 0x7fe17c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(353..353 src has 87..353) v4 ==== 9996+0+0 (secure 0 0 0) 0x7fe18808fdb0 con 0x7fe1901060a0 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 10 client.30240.objecter ms_dispatch 0x7fe190001550 osd_map(353..353 src has 87..353) v4 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 3 client.30240.objecter handle_osd_map got epochs [353,353] > 0 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 3 client.30240.objecter handle_osd_map decoding full epoch 353 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 20 client.30240.objecter dump_active .. 0 homeless 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe19650f700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 0x7fe17c071ce0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7fe18c005950 tx=0x7fe18c0058e0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:35.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.122+0000 7fe1877fe700 10 client.30240.objecter ms_handle_connect 0x7fe17c06f820 2024-03-11T21:32:35.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.126+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe188094070 con 0x7fe1901060a0 2024-03-11T21:32:35.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:35.223 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:35.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:35.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:35.362+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fe19002d830 con 0x7fe1901060a0 2024-03-11T21:32:35.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:35.384 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:35.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:35.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:35.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:35.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:35.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:35.739 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:35.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:35.869 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.058738+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.295507+0000 mon.a (mon.0) 2254 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.295665+0000 mon.a (mon.0) 2255 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: cluster 2024-03-11T21:32:35.301903+0000 mon.a (mon.0) 2256 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.307384+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.310424+0000 mon.a (mon.0) 2257 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.329313+0000 mon.c (mon.2) 725 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.329684+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.330895+0000 mon.a (mon.0) 2258 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.331104+0000 mon.a (mon.0) 2259 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.331608+0000 mon.c (mon.2) 727 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.331861+0000 mon.a (mon.0) 2260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.332357+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.332629+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[29284]: audit 2024-03-11T21:32:35.367785+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.058738+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.295507+0000 mon.a (mon.0) 2254 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.295665+0000 mon.a (mon.0) 2255 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: cluster 2024-03-11T21:32:35.301903+0000 mon.a (mon.0) 2256 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.307384+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.310424+0000 mon.a (mon.0) 2257 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.329313+0000 mon.c (mon.2) 725 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.329684+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.330895+0000 mon.a (mon.0) 2258 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.331104+0000 mon.a (mon.0) 2259 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.331608+0000 mon.c (mon.2) 727 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.331861+0000 mon.a (mon.0) 2260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.332357+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.332629+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:35 smithi143 bash[21512]: audit 2024-03-11T21:32:35.367785+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:32:35.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:35.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.058738+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:35.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.295507+0000 mon.a (mon.0) 2254 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi143-71604-53"}]': finished 2024-03-11T21:32:35.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.295665+0000 mon.a (mon.0) 2255 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: cluster 2024-03-11T21:32:35.301903+0000 mon.a (mon.0) 2256 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.307384+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.310424+0000 mon.a (mon.0) 2257 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.329313+0000 mon.c (mon.2) 725 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.329684+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.330895+0000 mon.a (mon.0) 2258 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.331104+0000 mon.a (mon.0) 2259 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.331608+0000 mon.c (mon.2) 727 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.331861+0000 mon.a (mon.0) 2260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.332357+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.332629+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:35 smithi203 bash[23654]: audit 2024-03-11T21:32:35.367785+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:32:35.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:36.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.130 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:36.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.262 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:36.276 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.302+0000 7fe1877fe700 1 -- 172.21.15.143:0/496519948 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v355) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe18805c830 con 0x7fe1901060a0 2024-03-11T21:32:36.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.302+0000 7fe198773700 10 librados: watch_flush enter 2024-03-11T21:32:36.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 10 librados: watch_flush exit 2024-03-11T21:32:36.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 20 client.30240.objecter shutdown clearing up homeless session... 2024-03-11T21:32:36.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 10 client.30240.objecter successfully canceled tick 2024-03-11T21:32:36.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 msgr2=0x7fe17c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:36.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 0x7fe17c071ce0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7fe18c005950 tx=0x7fe18c0058e0 comp rx=0 tx=0).stop 2024-03-11T21:32:36.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 msgr2=0x7fe19019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:36.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fe18800cc30 tx=0x7fe18800cd10 comp rx=0 tx=0).stop 2024-03-11T21:32:36.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 shutdown_connections 2024-03-11T21:32:36.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe17c06f820 0x7fe17c071ce0 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:36.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1901060a0 0x7fe19019e730 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:36.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe190100fc0 0x7fe190199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:36.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 --2- 172.21.15.143:0/496519948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe190101970 0x7fe19019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:36.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.306+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 >> 172.21.15.143:0/496519948 conn(0x7fe190077de0 msgr2=0x7fe19010c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:36.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.310+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 shutdown_connections 2024-03-11T21:32:36.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.310+0000 7fe198773700 1 -- 172.21.15.143:0/496519948 wait complete. 2024-03-11T21:32:36.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:36.310+0000 7fe198773700 1 librados: shutdown 2024-03-11T21:32:36.292 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:32:36.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.388 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:36.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.518 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:36.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:36.642 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:36.643 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:36.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:36.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.772 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:36.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.887 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: cluster 2024-03-11T21:32:35.328944+0000 mgr.y (mgr.14550) 384 : cluster 0 pgmap v519: 278 pgs: 1 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.060021+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.143581+0000 mon.a (mon.0) 2263 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.147370+0000 mon.c (mon.2) 730 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: cluster 2024-03-11T21:32:36.296670+0000 mon.a (mon.0) 2264 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.303617+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.303731+0000 mon.a (mon.0) 2266 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.303820+0000 mon.a (mon.0) 2267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.303920+0000 mon.a (mon.0) 2268 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: cluster 2024-03-11T21:32:36.308272+0000 mon.a (mon.0) 2269 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.313300+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:36 smithi203 bash[23654]: audit 2024-03-11T21:32:36.315255+0000 mon.a (mon.0) 2270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: cluster 2024-03-11T21:32:35.328944+0000 mgr.y (mgr.14550) 384 : cluster 0 pgmap v519: 278 pgs: 1 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.060021+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.143581+0000 mon.a (mon.0) 2263 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.147370+0000 mon.c (mon.2) 730 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: cluster 2024-03-11T21:32:36.296670+0000 mon.a (mon.0) 2264 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.303617+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.303731+0000 mon.a (mon.0) 2266 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.303820+0000 mon.a (mon.0) 2267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.303920+0000 mon.a (mon.0) 2268 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: cluster 2024-03-11T21:32:36.308272+0000 mon.a (mon.0) 2269 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.313300+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[21512]: audit 2024-03-11T21:32:36.315255+0000 mon.a (mon.0) 2270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.890 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: cluster 2024-03-11T21:32:35.328944+0000 mgr.y (mgr.14550) 384 : cluster 0 pgmap v519: 278 pgs: 1 peering, 277 active+clean; 4.6 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.060021+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.143581+0000 mon.a (mon.0) 2263 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.147370+0000 mon.c (mon.2) 730 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: cluster 2024-03-11T21:32:36.296670+0000 mon.a (mon.0) 2264 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.303617+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-35"}]': finished 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.303731+0000 mon.a (mon.0) 2266 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.303820+0000 mon.a (mon.0) 2267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi143-71604-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.303920+0000 mon.a (mon.0) 2268 : audit 1 from='client.? 172.21.15.143:0/496519948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: cluster 2024-03-11T21:32:36.308272+0000 mon.a (mon.0) 2269 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.313300+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.891 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:36 smithi143 bash[29284]: audit 2024-03-11T21:32:36.315255+0000 mon.a (mon.0) 2270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:36.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:37.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:37.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.166 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:37.297 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:37.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:37.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:37.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:37.782 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:37.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:38.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:38.053 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:38.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:38.181 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:38.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:38.316 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:38.389 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:32:38.389 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:32:38.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:38.444 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:38 smithi143 bash[29284]: audit 2024-03-11T21:32:37.061533+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:38.444 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:38 smithi143 bash[29284]: cluster 2024-03-11T21:32:37.768023+0000 mon.a (mon.0) 2271 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:32:38.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:38 smithi143 bash[21512]: audit 2024-03-11T21:32:37.061533+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:38.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:38 smithi143 bash[21512]: cluster 2024-03-11T21:32:37.768023+0000 mon.a (mon.0) 2271 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:32:38.457 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:38.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:38 smithi203 bash[23654]: audit 2024-03-11T21:32:37.061533+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:38.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:38 smithi203 bash[23654]: cluster 2024-03-11T21:32:37.768023+0000 mon.a (mon.0) 2271 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:32:38.560 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:38.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:38.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:38.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:38.730 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:32:38.730 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:32:38.730 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:32:38.730 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:32:38.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:38.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:38.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:38.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:38.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:39.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:39.090 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:39.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: cluster 2024-03-11T21:32:37.329626+0000 mgr.y (mgr.14550) 385 : cluster 0 pgmap v522: 246 pgs: 1 peering, 245 active+clean; 4.6 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:37.884102+0000 mon.c (mon.2) 733 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.062750+0000 mon.c (mon.2) 734 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.310322+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: cluster 2024-03-11T21:32:38.318399+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:32:39.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.325197+0000 mon.c (mon.2) 735 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.337513+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.532956+0000 mgr.y (mgr.14550) 386 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.558171+0000 mon.c (mon.2) 736 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.559786+0000 mon.c (mon.2) 737 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:38.573132+0000 mon.a (mon.0) 2275 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:39.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:39 smithi203 bash[23654]: audit 2024-03-11T21:32:39.063962+0000 mon.c (mon.2) 738 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:39.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:39.338 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: cluster 2024-03-11T21:32:37.329626+0000 mgr.y (mgr.14550) 385 : cluster 0 pgmap v522: 246 pgs: 1 peering, 245 active+clean; 4.6 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:37.884102+0000 mon.c (mon.2) 733 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.062750+0000 mon.c (mon.2) 734 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.310322+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: cluster 2024-03-11T21:32:38.318399+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.325197+0000 mon.c (mon.2) 735 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.339 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.337513+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.340 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.532956+0000 mgr.y (mgr.14550) 386 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:39.340 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.558171+0000 mon.c (mon.2) 736 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:39.340 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.559786+0000 mon.c (mon.2) 737 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:39.340 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:38.573132+0000 mon.a (mon.0) 2275 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:39.340 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[29284]: audit 2024-03-11T21:32:39.063962+0000 mon.c (mon.2) 738 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: cluster 2024-03-11T21:32:37.329626+0000 mgr.y (mgr.14550) 385 : cluster 0 pgmap v522: 246 pgs: 1 peering, 245 active+clean; 4.6 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:37.884102+0000 mon.c (mon.2) 733 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.062750+0000 mon.c (mon.2) 734 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.310322+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi143-71604-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: cluster 2024-03-11T21:32:38.318399+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:32:39.341 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.325197+0000 mon.c (mon.2) 735 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.337513+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.532956+0000 mgr.y (mgr.14550) 386 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.558171+0000 mon.c (mon.2) 736 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.559786+0000 mon.c (mon.2) 737 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:38.573132+0000 mon.a (mon.0) 2275 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:32:39.342 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:39 smithi143 bash[21512]: audit 2024-03-11T21:32:39.063962+0000 mon.c (mon.2) 738 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:39.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:39.453 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:39.475 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:39.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:32:39.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:39.604 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:39.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.729 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:39.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:39.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:39.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:39.991 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:40.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:40.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:40.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.257 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:40.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:40.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:40.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:40.518 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:39.314333+0000 mon.a (mon.0) 2276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: cluster 2024-03-11T21:32:39.318171+0000 mon.a (mon.0) 2277 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:39.321076+0000 mon.c (mon.2) 739 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:39.321614+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: cluster 2024-03-11T21:32:39.330210+0000 mgr.y (mgr.14550) 387 : cluster 0 pgmap v525: 285 pgs: 7 creating+peering, 33 unknown, 1 clean+premerge+peered, 244 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:39.330686+0000 mon.c (mon.2) 740 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:39.332765+0000 mon.a (mon.0) 2279 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:40 smithi203 bash[23654]: audit 2024-03-11T21:32:40.065291+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:40.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:39.314333+0000 mon.a (mon.0) 2276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: cluster 2024-03-11T21:32:39.318171+0000 mon.a (mon.0) 2277 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:39.321076+0000 mon.c (mon.2) 739 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:39.321614+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: cluster 2024-03-11T21:32:39.330210+0000 mgr.y (mgr.14550) 387 : cluster 0 pgmap v525: 285 pgs: 7 creating+peering, 33 unknown, 1 clean+premerge+peered, 244 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:32:40.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:39.330686+0000 mon.c (mon.2) 740 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:39.332765+0000 mon.a (mon.0) 2279 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[29284]: audit 2024-03-11T21:32:40.065291+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:39.314333+0000 mon.a (mon.0) 2276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: cluster 2024-03-11T21:32:39.318171+0000 mon.a (mon.0) 2277 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:39.321076+0000 mon.c (mon.2) 739 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:39.321614+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:40.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: cluster 2024-03-11T21:32:39.330210+0000 mgr.y (mgr.14550) 387 : cluster 0 pgmap v525: 285 pgs: 7 creating+peering, 33 unknown, 1 clean+premerge+peered, 244 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:32:40.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:39.330686+0000 mon.c (mon.2) 740 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:39.332765+0000 mon.a (mon.0) 2279 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T21:32:40.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:40 smithi143 bash[21512]: audit 2024-03-11T21:32:40.065291+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:40.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:40.781 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:40.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.913 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:41.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:41.045 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:41.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:41.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.302 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:41.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: cluster 2024-03-11T21:32:40.315675+0000 mon.a (mon.0) 2280 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.317133+0000 mon.a (mon.0) 2281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.317499+0000 mon.a (mon.0) 2282 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: cluster 2024-03-11T21:32:40.327509+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.328055+0000 mon.c (mon.2) 742 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.415 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.333378+0000 mon.c (mon.2) 743 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.337035+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:40.350349+0000 mon.a (mon.0) 2285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.066901+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.279906+0000 mon.a (mon.0) 2286 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.280177+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.289865+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.290289+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: cluster 2024-03-11T21:32:41.290390+0000 mon.a (mon.0) 2288 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.292557+0000 mon.a (mon.0) 2289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.416 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:41 smithi203 bash[23654]: audit 2024-03-11T21:32:41.293464+0000 mon.a (mon.0) 2290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.428 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:41.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:41.558 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:41.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: cluster 2024-03-11T21:32:40.315675+0000 mon.a (mon.0) 2280 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.317133+0000 mon.a (mon.0) 2281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.317499+0000 mon.a (mon.0) 2282 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: cluster 2024-03-11T21:32:40.327509+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.328055+0000 mon.c (mon.2) 742 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.333378+0000 mon.c (mon.2) 743 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.337035+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:40.350349+0000 mon.a (mon.0) 2285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.066901+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.279906+0000 mon.a (mon.0) 2286 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.280177+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.289865+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.290289+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: cluster 2024-03-11T21:32:41.290390+0000 mon.a (mon.0) 2288 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.292557+0000 mon.a (mon.0) 2289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[29284]: audit 2024-03-11T21:32:41.293464+0000 mon.a (mon.0) 2290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.673 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: cluster 2024-03-11T21:32:40.315675+0000 mon.a (mon.0) 2280 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.317133+0000 mon.a (mon.0) 2281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.317499+0000 mon.a (mon.0) 2282 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: cluster 2024-03-11T21:32:40.327509+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.328055+0000 mon.c (mon.2) 742 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.333378+0000 mon.c (mon.2) 743 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.337035+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:40.350349+0000 mon.a (mon.0) 2285 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.066901+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.279906+0000 mon.a (mon.0) 2286 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.280177+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.289865+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.290289+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.143:0/3006824523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: cluster 2024-03-11T21:32:41.290390+0000 mon.a (mon.0) 2288 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:32:41.675 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.292557+0000 mon.a (mon.0) 2289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]: dispatch 2024-03-11T21:32:41.675 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:41 smithi143 bash[21512]: audit 2024-03-11T21:32:41.293464+0000 mon.a (mon.0) 2290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]: dispatch 2024-03-11T21:32:41.685 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:41.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:41.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:41.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:42.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:42.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:42.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:42.205 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:42.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:42.333 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:42.334 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:42.334 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: cluster 2024-03-11T21:32:41.330818+0000 mgr.y (mgr.14550) 388 : cluster 0 pgmap v528: 277 pgs: 5 creating+peering, 24 unknown, 1 clean+premerge+peered, 247 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.068732+0000 mon.c (mon.2) 747 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: cluster 2024-03-11T21:32:42.281284+0000 mon.a (mon.0) 2291 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.289827+0000 mon.a (mon.0) 2292 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]': finished 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.289907+0000 mon.a (mon.0) 2293 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: cluster 2024-03-11T21:32:42.295268+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.322565+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.335 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.322997+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.323643+0000 mon.c (mon.2) 749 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.324021+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.324623+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[21512]: audit 2024-03-11T21:32:42.324944+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: cluster 2024-03-11T21:32:41.330818+0000 mgr.y (mgr.14550) 388 : cluster 0 pgmap v528: 277 pgs: 5 creating+peering, 24 unknown, 1 clean+premerge+peered, 247 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.068732+0000 mon.c (mon.2) 747 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: cluster 2024-03-11T21:32:42.281284+0000 mon.a (mon.0) 2291 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.289827+0000 mon.a (mon.0) 2292 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]': finished 2024-03-11T21:32:42.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.289907+0000 mon.a (mon.0) 2293 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: cluster 2024-03-11T21:32:42.295268+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.322565+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.322997+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.323643+0000 mon.c (mon.2) 749 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.324021+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.324623+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.337 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:42 smithi143 bash[29284]: audit 2024-03-11T21:32:42.324944+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:42.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:42.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:42.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:42.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 -- 172.21.15.143:0/2403383227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 msgr2=0x7faa601082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 --2- 172.21.15.143:0/2403383227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa601082c0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009990 tx=0x7faa5c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 -- 172.21.15.143:0/2403383227 shutdown_connections 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 --2- 172.21.15.143:0/2403383227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 0x7faa6010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 --2- 172.21.15.143:0/2403383227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa601082c0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 --2- 172.21.15.143:0/2403383227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa601022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 -- 172.21.15.143:0/2403383227 >> 172.21.15.143:0/2403383227 conn(0x7faa60078f90 msgr2=0x7faa600793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.686+0000 7faa678c3700 1 -- 172.21.15.143:0/2403383227 shutdown_connections 2024-03-11T21:32:42.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- 172.21.15.143:0/2403383227 wait complete. 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 librados: starting msgr at 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 librados: starting objecter 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 Processor -- start 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- start start 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 librados: setting wanted keys 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 librados: calling monclient init 2024-03-11T21:32:42.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa600740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 0x7faa60074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa60111630 con 0x7faa60101e60 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa601114b0 con 0x7faa60102820 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa601117b0 con 0x7faa60107ee0 2024-03-11T21:32:42.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa65e60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa600740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa65e60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33438/0 (socket says 172.21.15.143:33438) 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa65e60700 1 -- 172.21.15.143:0/132804148 learned_addr learned my addr 172.21.15.143:0/132804148 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa64e5e700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 0x7faa60074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 -- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 msgr2=0x7faa60074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 -- 172.21.15.143:0/132804148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 msgr2=0x7faa60074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 0x7faa60074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 -- 172.21.15.143:0/132804148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa601aa5a0 con 0x7faa60101e60 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa65e60700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa6565f700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa600740e0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0059d0 tx=0x7faa5c00f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa567fc700 10 client.?.objecter ms_handle_connect 0x7faa60101e60 2024-03-11T21:32:42.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa567fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c027070 con 0x7faa60101e60 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7faa5c022470 con 0x7faa60101e60 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa601aa7d0 con 0x7faa60101e60 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c017650 con 0x7faa60101e60 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 10 client.30258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa601aacd0 con 0x7faa60101e60 2024-03-11T21:32:42.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 librados: init done 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.690+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa60103520 con 0x7faa60101e60 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7faa5c02a070 con 0x7faa60101e60 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 0x7faa4c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(361..361 src has 87..361) v4 ==== 9800+0+0 (secure 0 0 0) 0x7faa5c08fc20 con 0x7faa60101e60 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa64e5e700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 0x7faa4c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 10 client.30258.objecter ms_dispatch 0x7faa60001550 osd_map(361..361 src has 87..361) v4 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 3 client.30258.objecter handle_osd_map got epochs [361,361] > 0 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 3 client.30258.objecter handle_osd_map decoding full epoch 361 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 20 client.30258.objecter dump_active .. 0 homeless 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa64e5e700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 0x7faa4c071ce0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7faa600758a0 tx=0x7faa50009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.694+0000 7faa567fc700 10 client.30258.objecter ms_handle_connect 0x7faa4c06f820 2024-03-11T21:32:42.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.698+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7faa5c013070 con 0x7faa60101e60 2024-03-11T21:32:42.695 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:42.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 -- 172.21.15.143:0/1274792842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec101700 msgr2=0x7f3dec10db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/1274792842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec101700 0x7f3dec10db70 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009f90 tx=0x7f3de400b330 comp rx=0 tx=0).stop 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 -- 172.21.15.143:0/1274792842 shutdown_connections 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/1274792842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec101700 0x7f3dec10db70 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/1274792842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3dec106dc0 0x7f3dec1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/1274792842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 -- 172.21.15.143:0/1274792842 >> 172.21.15.143:0/1274792842 conn(0x7f3dec076d80 msgr2=0x7f3dec077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 -- 172.21.15.143:0/1274792842 shutdown_connections 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 -- 172.21.15.143:0/1274792842 wait complete. 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 librados: starting msgr at 2024-03-11T21:32:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.790+0000 7f3df0fe4700 1 librados: starting objecter 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 Processor -- start 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- start start 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 librados: setting wanted keys 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 librados: calling monclient init 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3dec101700 0x7f3dec198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec106dc0 0x7f3dec1a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec110570 con 0x7f3dec100d40 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec1103f0 con 0x7f3dec101700 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec1106f0 con 0x7f3dec106dc0 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33464/0 (socket says 172.21.15.143:33464) 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 -- 172.21.15.143:0/810622580 learned_addr learned my addr 172.21.15.143:0/810622580 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec106dc0 0x7f3dec1a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3dec101700 0x7f3dec198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec106dc0 msgr2=0x7f3dec1a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dec106dc0 0x7f3dec1a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3dec101700 msgr2=0x7f3dec198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3dec101700 0x7f3dec198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec1967e0 con 0x7f3dec100d40 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00d980 tx=0x7f3ddc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 10 client.?.objecter ms_handle_connect 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc017070 con 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3ddc005020 con 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc00fa30 con 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dec196ad0 con 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 10 client.30264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dec196fc0 con 0x7f3dec100d40 2024-03-11T21:32:42.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 librados: init done 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: cluster 2024-03-11T21:32:41.330818+0000 mgr.y (mgr.14550) 388 : cluster 0 pgmap v528: 277 pgs: 5 creating+peering, 24 unknown, 1 clean+premerge+peered, 247 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.068732+0000 mon.c (mon.2) 747 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: cluster 2024-03-11T21:32:42.281284+0000 mon.a (mon.0) 2291 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.289827+0000 mon.a (mon.0) 2292 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-37", "mode": "writeback"}]': finished 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.289907+0000 mon.a (mon.0) 2293 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi143-71604-54"}]': finished 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: cluster 2024-03-11T21:32:42.295268+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.322565+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.322997+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.323643+0000 mon.c (mon.2) 749 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.324021+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.324623+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:42 smithi203 bash[23654]: audit 2024-03-11T21:32:42.324944+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3dec102400 con 0x7f3dec100d40 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.794+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3ddc004a30 con 0x7f3dec100d40 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 0x7f3dd4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(361..361 src has 87..361) v4 ==== 9800+0+0 (secure 0 0 0) 0x7f3ddc08eec0 con 0x7f3dec100d40 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 osd_map(361..361 src has 87..361) v4 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 3 client.30264.objecter handle_osd_map got epochs [361,361] > 0 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 3 client.30264.objecter handle_osd_map decoding full epoch 361 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 0x7f3dd4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 20 client.30264.objecter dump_active .. 0 homeless 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v361) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3ddc020020 con 0x7f3dec100d40 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 1 2024-03-11T21:32:42.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3dec062c40 0x7f3dec02c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 1 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 0 1 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 4 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 10 client.30264.objecter _send_command 1 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec109100 con 0x7f3dec062c40 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dd406f820 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 0x7f3dd4071ce0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009750 tx=0x7f3de0009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3dec062c40 0x7f3dec02c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec062c40 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3dec062c40 0x7f3dec02c6b0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec109100 con 0x7f3dec062c40 2024-03-11T21:32:42.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:32:42.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 1 = system:0 2024-03-11T21:32:42.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 4 2024-03-11T21:32:42.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.798+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 0 1 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 2 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 2 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec04faa0 osd=0 3 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 0 2 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 4 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 10 client.30264.objecter _send_command 2 2024-03-11T21:32:42.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.814+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec06a0b0 con 0x7f3dec062c40 2024-03-11T21:32:42.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec06a0b0 con 0x7f3dec062c40 2024-03-11T21:32:42.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 4 2024-03-11T21:32:42.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 0 2 2024-03-11T21:32:42.794 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3dec062830 0x7f3dec107960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 1 3 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 4 2024-03-11T21:32:42.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 10 client.30264.objecter _send_command 3 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec197f60 con 0x7f3dec062830 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3dec062830 0x7f3dec107960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec062830 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3dec062830 0x7f3dec107960 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec197f60 con 0x7f3dec062830 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 3 = system:0 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 4 2024-03-11T21:32:42.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.818+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 1 3 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 4 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 4 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec0626c0 osd=1 3 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 1 4 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 4 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 10 client.30264.objecter _send_command 4 2024-03-11T21:32:42.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec02cbf0 con 0x7f3dec062830 2024-03-11T21:32:42.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec02cbf0 con 0x7f3dec062830 2024-03-11T21:32:42.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 4 2024-03-11T21:32:42.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 1 4 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 5 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3dec197640 0x7f3dec1ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 5 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 2 5 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 4 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 10 client.30264.objecter _send_command 5 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec02dae0 con 0x7f3dec197640 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3dec197640 0x7f3dec1ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec197640 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.838+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3dec197640 0x7f3dec1ae3f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.842+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec02dae0 con 0x7f3dec197640 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.842+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.842+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 5 = system:0 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.842+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 4 2024-03-11T21:32:42.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.842+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 2 5 2024-03-11T21:32:42.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 6 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.834 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 6 2024-03-11T21:32:42.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1974d0 osd=2 3 2024-03-11T21:32:42.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 2 6 2024-03-11T21:32:42.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 4 2024-03-11T21:32:42.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 10 client.30264.objecter _send_command 6 2024-03-11T21:32:42.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.858+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec02d050 con 0x7f3dec197640 2024-03-11T21:32:42.836 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec02d050 con 0x7f3dec197640 2024-03-11T21:32:42.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 4 2024-03-11T21:32:42.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 2 6 2024-03-11T21:32:42.837 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 7 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3dec108b80 0x7f3dec1b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 7 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 3 7 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 4 2024-03-11T21:32:42.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 10 client.30264.objecter _send_command 7 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec1b4a30 con 0x7f3dec108b80 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3dec108b80 0x7f3dec1b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3dec108b80 0x7f3dec1b1ac0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec108b80 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec1b4a30 con 0x7f3dec108b80 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 7 = system:0 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 4 2024-03-11T21:32:42.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.862+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 3 7 2024-03-11T21:32:42.841 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 8 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 8 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec02d450 osd=3 3 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 3 8 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 4 2024-03-11T21:32:42.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 10 client.30264.objecter _send_command 8 2024-03-11T21:32:42.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec108150 con 0x7f3dec108b80 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec108150 con 0x7f3dec108b80 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 4 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 3 8 2024-03-11T21:32:42.859 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 9 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3dec1af610 0x7f3dec1b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 9 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 4 9 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 4 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 10 client.30264.objecter _send_command 9 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec1b8420 con 0x7f3dec1af610 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.882+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3dec1af610 0x7f3dec1b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3dec1af610 0x7f3dec1b2050 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec1af610 2024-03-11T21:32:42.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec1b8420 con 0x7f3dec1af610 2024-03-11T21:32:42.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:32:42.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 9 = system:0 2024-03-11T21:32:42.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 4 2024-03-11T21:32:42.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.886+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 4 9 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 10 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 10 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec107f00 osd=4 3 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 4 10 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 4 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 10 client.30264.objecter _send_command 10 2024-03-11T21:32:42.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.902+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec1aeb80 con 0x7f3dec1af610 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.906+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec1aeb80 con 0x7f3dec1af610 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.906+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.906+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 4 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 4 10 2024-03-11T21:32:42.885 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 11 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3dec1b4500 0x7f3dec1b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 11 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 5 11 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 4 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 10 client.30264.objecter _send_command 11 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec1aeaa0 con 0x7f3dec1b4500 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3dec1b4500 0x7f3dec1b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec1b4500 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3de9d9b700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3dec1b4500 0x7f3dec1b8c90 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec1aeaa0 con 0x7f3dec1b4500 2024-03-11T21:32:42.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:32:42.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 11 = system:0 2024-03-11T21:32:42.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 4 2024-03-11T21:32:42.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.910+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 5 11 2024-03-11T21:32:42.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 12 2024-03-11T21:32:42.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 12 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1ae930 osd=5 3 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 5 12 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 4 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 10 client.30264.objecter _send_command 12 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec1aeaa0 con 0x7f3dec1b4500 2024-03-11T21:32:42.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.930+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7faa60075610 con 0x7faa60101e60 2024-03-11T21:32:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec1aeaa0 con 0x7f3dec1b4500 2024-03-11T21:32:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 4 2024-03-11T21:32:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 5 12 2024-03-11T21:32:42.909 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:32:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 13 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3dec1b96e0 0x7f3dec1bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 13 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 6 13 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 4 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 10 client.30264.objecter _send_command 13 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec1beb50 con 0x7f3dec1b96e0 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3dec1b96e0 0x7f3dec1bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec1b96e0 2024-03-11T21:32:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3dead9d700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3dec1b96e0 0x7f3dec1bbb50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec1beb50 con 0x7f3dec1b96e0 2024-03-11T21:32:42.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:32:42.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 13 = system:0 2024-03-11T21:32:42.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 4 2024-03-11T21:32:42.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.934+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 6 13 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 14 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 14 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1aeaa0 osd=6 3 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 6 14 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 4 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.950+0000 7f3df0fe4700 10 client.30264.objecter _send_command 14 2024-03-11T21:32:42.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.954+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec1bd920 con 0x7f3dec1b96e0 2024-03-11T21:32:42.932 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:32:42.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec1bd920 con 0x7f3dec1b96e0 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 4 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 6 14 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 15 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3dec1bdfa0 0x7f3dec1be400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 15 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 7 15 2024-03-11T21:32:42.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 4 2024-03-11T21:32:42.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 10 client.30264.objecter _send_command 15 2024-03-11T21:32:42.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dec1c1ec0 con 0x7f3dec1bdfa0 2024-03-11T21:32:42.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3dec1bdfa0 0x7f3dec1be400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:42.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3dea59c700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3dec1bdfa0 0x7f3dec1be400 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:42.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 10 client.30264.objecter ms_handle_connect 0x7f3dec1bdfa0 2024-03-11T21:32:42.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3dec1c1ec0 con 0x7f3dec1bdfa0 2024-03-11T21:32:42.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:32:42.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 15 = system:0 2024-03-11T21:32:42.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 4 2024-03-11T21:32:42.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.958+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 7 15 2024-03-11T21:32:42.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:42.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 10 client.30264.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:32:42.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign -1 16 2024-03-11T21:32:42.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 20 client.30264.objecter _get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_remove -1 16 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 20 client.30264.objecter get_session s=0x7f3dec1bd920 osd=7 3 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 15 client.30264.objecter _session_command_op_assign 7 16 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 4 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 10 client.30264.objecter _send_command 16 2024-03-11T21:32:42.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.974+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3dec1bc150 con 0x7f3dec1bdfa0 2024-03-11T21:32:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3ddb7fe700 1 -- 172.21.15.143:0/810622580 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3dec1bc150 con 0x7f3dec1bdfa0 2024-03-11T21:32:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3ddb7fe700 10 client.30264.objecter ms_dispatch 0x7f3dec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:32:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3ddb7fe700 10 client.30264.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:32:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3ddb7fe700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 4 2024-03-11T21:32:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3ddb7fe700 15 client.30264.objecter _session_command_op_remove 7 16 2024-03-11T21:32:42.956 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 10 librados: watch_flush enter 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 10 librados: watch_flush exit 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.0 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3dec062c40 msgr2=0x7f3dec02c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3dec062c40 0x7f3dec02c6b0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec04faa0 osd=0 1 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.1 2024-03-11T21:32:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3dec062830 msgr2=0x7f3dec107960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3dec062830 0x7f3dec107960 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec0626c0 osd=1 1 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.2 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3dec197640 msgr2=0x7f3dec1ae3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.978+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3dec197640 0x7f3dec1ae3f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1974d0 osd=2 1 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.3 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3dec108b80 msgr2=0x7f3dec1b1ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3dec108b80 0x7f3dec1b1ac0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec02d450 osd=3 1 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.4 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3dec1af610 msgr2=0x7f3dec1b2050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3dec1af610 0x7f3dec1b2050 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec107f00 osd=4 1 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.5 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3dec1b4500 msgr2=0x7f3dec1b8c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3dec1b4500 0x7f3dec1b8c90 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1ae930 osd=5 1 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.6 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3dec1b96e0 msgr2=0x7f3dec1bbb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3dec1b96e0 0x7f3dec1bbb50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1aeaa0 osd=6 1 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter close_session for osd.7 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3dec1bdfa0 msgr2=0x7f3dec1be400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3dec1bdfa0 0x7f3dec1be400 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter put_session s=0x7f3dec1bd920 osd=7 1 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 20 client.30264.objecter shutdown clearing up homeless session... 2024-03-11T21:32:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 10 client.30264.objecter successfully canceled tick 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3dead9d700 1 -- 172.21.15.143:0/810622580 reap_dead start 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3dead9d700 1 -- 172.21.15.143:0/810622580 reap_dead start 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3dead9d700 1 -- 172.21.15.143:0/810622580 reap_dead start 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3dead9d700 1 -- 172.21.15.143:0/810622580 reap_dead start 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3dead9d700 1 -- 172.21.15.143:0/810622580 reap_dead start 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 msgr2=0x7f3dd4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 0x7f3dd4071ce0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009750 tx=0x7f3de0009310 comp rx=0 tx=0).stop 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 msgr2=0x7f3dec1989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00d980 tx=0x7f3ddc00dc90 comp rx=0 tx=0).stop 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 shutdown_connections 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dec100d40 0x7f3dec1989f0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 --2- 172.21.15.143:0/810622580 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3dd406f820 0x7f3dd4071ce0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 >> 172.21.15.143:0/810622580 conn(0x7f3dec076d80 msgr2=0x7f3dec0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 shutdown_connections 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 -- 172.21.15.143:0/810622580 wait complete. 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:42.982+0000 7f3df0fe4700 1 librados: shutdown 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:42.972 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:43.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.095 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.137 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:43.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.290+0000 7faa567fc700 1 -- 172.21.15.143:0/132804148 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v362) v1 ==== 97+0+0 (secure 0 0 0) 0x7faa5c05c900 con 0x7faa60101e60 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 10 librados: watch_flush enter 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 10 librados: watch_flush exit 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 20 client.30258.objecter shutdown clearing up homeless session... 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 10 client.30258.objecter successfully canceled tick 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 msgr2=0x7faa4c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 0x7faa4c071ce0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7faa600758a0 tx=0x7faa50009260 comp rx=0 tx=0).stop 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 msgr2=0x7faa600740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:43.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa600740e0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0059d0 tx=0x7faa5c00f460 comp rx=0 tx=0).stop 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 shutdown_connections 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faa4c06f820 0x7faa4c071ce0 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa60101e60 0x7faa600740e0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa60107ee0 0x7faa60074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 --2- 172.21.15.143:0/132804148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60102820 0x7faa60074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:43.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 >> 172.21.15.143:0/132804148 conn(0x7faa60078f90 msgr2=0x7faa60108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:43.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 shutdown_connections 2024-03-11T21:32:43.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 -- 172.21.15.143:0/132804148 wait complete. 2024-03-11T21:32:43.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:43.294+0000 7faa678c3700 1 librados: shutdown 2024-03-11T21:32:43.273 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:32:43.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:43.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:43.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:42.365343+0000 mon.c (mon.2) 751 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:42.365812+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:42.799018+0000 mon.a (mon.0) 2299 : audit 0 from='client.? 172.21.15.143:0/810622580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:42.932732+0000 mon.a (mon.0) 2300 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.070356+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: cluster 2024-03-11T21:32:43.290301+0000 mon.a (mon.0) 2301 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.292499+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.292653+0000 mon.a (mon.0) 2303 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.292847+0000 mon.a (mon.0) 2304 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: cluster 2024-03-11T21:32:43.297715+0000 mon.a (mon.0) 2305 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.298811+0000 mon.c (mon.2) 753 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.299137+0000 mon.c (mon.2) 754 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.299549+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.299798+0000 mon.a (mon.0) 2307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.331832+0000 mon.c (mon.2) 755 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[29284]: audit 2024-03-11T21:32:43.333482+0000 mon.a (mon.0) 2308 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:42.365343+0000 mon.c (mon.2) 751 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:42.365812+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:42.799018+0000 mon.a (mon.0) 2299 : audit 0 from='client.? 172.21.15.143:0/810622580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:42.932732+0000 mon.a (mon.0) 2300 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.070356+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: cluster 2024-03-11T21:32:43.290301+0000 mon.a (mon.0) 2301 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.292499+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.292653+0000 mon.a (mon.0) 2303 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:43.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.292847+0000 mon.a (mon.0) 2304 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: cluster 2024-03-11T21:32:43.297715+0000 mon.a (mon.0) 2305 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.298811+0000 mon.c (mon.2) 753 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.299137+0000 mon.c (mon.2) 754 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.299549+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.299798+0000 mon.a (mon.0) 2307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.331832+0000 mon.c (mon.2) 755 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.438 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:43 smithi143 bash[21512]: audit 2024-03-11T21:32:43.333482+0000 mon.a (mon.0) 2308 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.479 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:43.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:43.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:43.740 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:43.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:42.365343+0000 mon.c (mon.2) 751 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:42.365812+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:42.799018+0000 mon.a (mon.0) 2299 : audit 0 from='client.? 172.21.15.143:0/810622580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:42.932732+0000 mon.a (mon.0) 2300 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.070356+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: cluster 2024-03-11T21:32:43.290301+0000 mon.a (mon.0) 2301 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.292499+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi143-71604-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.292653+0000 mon.a (mon.0) 2303 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.292847+0000 mon.a (mon.0) 2304 : audit 1 from='client.? 172.21.15.143:0/132804148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: cluster 2024-03-11T21:32:43.297715+0000 mon.a (mon.0) 2305 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.298811+0000 mon.c (mon.2) 753 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.299137+0000 mon.c (mon.2) 754 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.299549+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]: dispatch 2024-03-11T21:32:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.299798+0000 mon.a (mon.0) 2307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.331832+0000 mon.c (mon.2) 755 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:43 smithi203 bash[23654]: audit 2024-03-11T21:32:43.333482+0000 mon.a (mon.0) 2308 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T21:32:43.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:43.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:43.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:44.008 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:44.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:44.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:44.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.286 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:44.341 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:44.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:44.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:44.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.540 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:44.643 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:44.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:44.786 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[21512]: cluster 2024-03-11T21:32:43.331443+0000 mgr.y (mgr.14550) 389 : cluster 0 pgmap v531: 276 pgs: 1 peering, 5 creating+peering, 270 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:44.786 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[21512]: audit 2024-03-11T21:32:44.071320+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:44.786 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[21512]: cluster 2024-03-11T21:32:44.293265+0000 mon.a (mon.0) 2309 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:44.787 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[21512]: audit 2024-03-11T21:32:44.295135+0000 mon.a (mon.0) 2310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:44.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[29284]: cluster 2024-03-11T21:32:43.331443+0000 mgr.y (mgr.14550) 389 : cluster 0 pgmap v531: 276 pgs: 1 peering, 5 creating+peering, 270 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:44.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[29284]: audit 2024-03-11T21:32:44.071320+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:44.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[29284]: cluster 2024-03-11T21:32:44.293265+0000 mon.a (mon.0) 2309 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:44.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:44 smithi143 bash[29284]: audit 2024-03-11T21:32:44.295135+0000 mon.a (mon.0) 2310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:44.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:44.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:44.927 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:44 smithi203 bash[23654]: cluster 2024-03-11T21:32:43.331443+0000 mgr.y (mgr.14550) 389 : cluster 0 pgmap v531: 276 pgs: 1 peering, 5 creating+peering, 270 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:44 smithi203 bash[23654]: audit 2024-03-11T21:32:44.071320+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:44 smithi203 bash[23654]: cluster 2024-03-11T21:32:44.293265+0000 mon.a (mon.0) 2309 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:45.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:44 smithi203 bash[23654]: audit 2024-03-11T21:32:44.295135+0000 mon.a (mon.0) 2310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-37"}]': finished 2024-03-11T21:32:45.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:45.058 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:45.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:45.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:45.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:45.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:45.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:45.452 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:45.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:45.576 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:45.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:45.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:45.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:45.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:45.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:45.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[21512]: audit 2024-03-11T21:32:44.295216+0000 mon.a (mon.0) 2311 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T21:32:45.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[21512]: cluster 2024-03-11T21:32:44.745307+0000 mon.a (mon.0) 2312 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:32:45.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[21512]: audit 2024-03-11T21:32:45.072625+0000 mon.c (mon.2) 757 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:45.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[21512]: audit 2024-03-11T21:32:45.298484+0000 mon.a (mon.0) 2313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:45.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[21512]: cluster 2024-03-11T21:32:45.306763+0000 mon.a (mon.0) 2314 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:32:45.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[29284]: audit 2024-03-11T21:32:44.295216+0000 mon.a (mon.0) 2311 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T21:32:45.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[29284]: cluster 2024-03-11T21:32:44.745307+0000 mon.a (mon.0) 2312 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:32:45.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[29284]: audit 2024-03-11T21:32:45.072625+0000 mon.c (mon.2) 757 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:45.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[29284]: audit 2024-03-11T21:32:45.298484+0000 mon.a (mon.0) 2313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:45.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:45 smithi143 bash[29284]: cluster 2024-03-11T21:32:45.306763+0000 mon.a (mon.0) 2314 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:32:45.956 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:45 smithi203 bash[23654]: audit 2024-03-11T21:32:44.295216+0000 mon.a (mon.0) 2311 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T21:32:46.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:45 smithi203 bash[23654]: cluster 2024-03-11T21:32:44.745307+0000 mon.a (mon.0) 2312 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:32:46.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:45 smithi203 bash[23654]: audit 2024-03-11T21:32:45.072625+0000 mon.c (mon.2) 757 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:46.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:45 smithi203 bash[23654]: audit 2024-03-11T21:32:45.298484+0000 mon.a (mon.0) 2313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi143-71604-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:46.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:45 smithi203 bash[23654]: cluster 2024-03-11T21:32:45.306763+0000 mon.a (mon.0) 2314 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:32:46.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:46.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:46.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:46.318 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:46.336 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:46.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:46.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:46.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.593 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:46.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:46.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:46.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:46.856 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:46.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: cluster 2024-03-11T21:32:45.332099+0000 mgr.y (mgr.14550) 390 : cluster 0 pgmap v534: 252 pgs: 8 unknown, 1 peering, 243 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: audit 2024-03-11T21:32:46.074185+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: cluster 2024-03-11T21:32:46.279248+0000 mon.a (mon.0) 2315 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: cluster 2024-03-11T21:32:46.307684+0000 mon.a (mon.0) 2316 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: audit 2024-03-11T21:32:46.308828+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:46 smithi203 bash[23654]: audit 2024-03-11T21:32:46.315051+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.089 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.117 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:47.231 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: cluster 2024-03-11T21:32:45.332099+0000 mgr.y (mgr.14550) 390 : cluster 0 pgmap v534: 252 pgs: 8 unknown, 1 peering, 243 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:46.074185+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: cluster 2024-03-11T21:32:46.279248+0000 mon.a (mon.0) 2315 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: cluster 2024-03-11T21:32:46.307684+0000 mon.a (mon.0) 2316 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:46.308828+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:46.315051+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: cluster 2024-03-11T21:32:45.332099+0000 mgr.y (mgr.14550) 390 : cluster 0 pgmap v534: 252 pgs: 8 unknown, 1 peering, 243 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:46.074185+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: cluster 2024-03-11T21:32:46.279248+0000 mon.a (mon.0) 2315 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: cluster 2024-03-11T21:32:46.307684+0000 mon.a (mon.0) 2316 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:32:47.232 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:46.308828+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.233 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:46.315051+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:47.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:47.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:47.371 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:47.379 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:47.379 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:47.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:47.494 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:47.597 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:32:47.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:47.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:32:47.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:47.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.075901+0000 mon.c (mon.2) 760 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.306532+0000 mon.a (mon.0) 2318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: cluster 2024-03-11T21:32:47.311891+0000 mon.a (mon.0) 2319 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.319207+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.320213+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.322615+0000 mon.c (mon.2) 762 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.323449+0000 mon.a (mon.0) 2321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.333716+0000 mon.c (mon.2) 763 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:47.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[21512]: audit 2024-03-11T21:32:47.336727+0000 mon.a (mon.0) 2322 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.075901+0000 mon.c (mon.2) 760 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.306532+0000 mon.a (mon.0) 2318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: cluster 2024-03-11T21:32:47.311891+0000 mon.a (mon.0) 2319 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.319207+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.320213+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.322615+0000 mon.c (mon.2) 762 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.323449+0000 mon.a (mon.0) 2321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.333716+0000 mon.c (mon.2) 763 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:47.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:47 smithi143 bash[29284]: audit 2024-03-11T21:32:47.336727+0000 mon.a (mon.0) 2322 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:47.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:47.995 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:48.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.075901+0000 mon.c (mon.2) 760 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:48.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.306532+0000 mon.a (mon.0) 2318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:48.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: cluster 2024-03-11T21:32:47.311891+0000 mon.a (mon.0) 2319 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:32:48.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.319207+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.320213+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.322615+0000 mon.c (mon.2) 762 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.323449+0000 mon.a (mon.0) 2321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.333716+0000 mon.c (mon.2) 763 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:48.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:47 smithi203 bash[23654]: audit 2024-03-11T21:32:47.336727+0000 mon.a (mon.0) 2322 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T21:32:48.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.123 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:48.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:32:48.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.253 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:48.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:48.378 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:48.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.508 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:48.612 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:48.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:48.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.770 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: cluster 2024-03-11T21:32:47.332708+0000 mgr.y (mgr.14550) 391 : cluster 0 pgmap v537: 275 pgs: 1 clean+premerge+peered, 32 unknown, 1 peering, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.077259+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.314030+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.314177+0000 mon.a (mon.0) 2324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.314286+0000 mon.a (mon.0) 2325 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: cluster 2024-03-11T21:32:48.319185+0000 mon.a (mon.0) 2326 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.319888+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.320319+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.320593+0000 mon.a (mon.0) 2327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:48 smithi203 bash[23654]: audit 2024-03-11T21:32:48.321557+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:48.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:48.903 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:48.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: cluster 2024-03-11T21:32:47.332708+0000 mgr.y (mgr.14550) 391 : cluster 0 pgmap v537: 275 pgs: 1 clean+premerge+peered, 32 unknown, 1 peering, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.077259+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.314030+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.314177+0000 mon.a (mon.0) 2324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.314286+0000 mon.a (mon.0) 2325 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: cluster 2024-03-11T21:32:48.319185+0000 mon.a (mon.0) 2326 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.319888+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.915 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.320319+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.320593+0000 mon.a (mon.0) 2327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[21512]: audit 2024-03-11T21:32:48.321557+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: cluster 2024-03-11T21:32:47.332708+0000 mgr.y (mgr.14550) 391 : cluster 0 pgmap v537: 275 pgs: 1 clean+premerge+peered, 32 unknown, 1 peering, 241 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.077259+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.314030+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.314177+0000 mon.a (mon.0) 2324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:48.916 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.314286+0000 mon.a (mon.0) 2325 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T21:32:48.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: cluster 2024-03-11T21:32:48.319185+0000 mon.a (mon.0) 2326 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:32:48.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.319888+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.143:0/2783444678' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.320319+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:48.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.320593+0000 mon.a (mon.0) 2327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]: dispatch 2024-03-11T21:32:48.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:48 smithi143 bash[29284]: audit 2024-03-11T21:32:48.321557+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:49.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:49.028 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:49.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/3321371663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 msgr2=0x7fb27410ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:49.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- 172.21.15.143:0/3321371663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb27410ecb0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009f90 tx=0x7fb26c00b330 comp rx=0 tx=0).stop 2024-03-11T21:32:49.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/3321371663 shutdown_connections 2024-03-11T21:32:49.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- 172.21.15.143:0/3321371663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb27410ecb0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- 172.21.15.143:0/3321371663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb274107e90 0x7fb274108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- 172.21.15.143:0/3321371663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb27406a4e0 0x7fb27406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/3321371663 >> 172.21.15.143:0/3321371663 conn(0x7fb274077bd0 msgr2=0x7fb274077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:49.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/3321371663 shutdown_connections 2024-03-11T21:32:49.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/3321371663 wait complete. 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 librados: starting msgr at 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 librados: starting objecter 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 Processor -- start 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- start start 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 librados: setting wanted keys 2024-03-11T21:32:49.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 librados: calling monclient init 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb27406a4e0 0x7fb274101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb2741015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274111610 con 0x7fb274107e90 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274111490 con 0x7fb27406a4e0 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb274111790 con 0x7fb27406aea0 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2789a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb2741015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2789a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb2741015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43070/0 (socket says 172.21.15.143:43070) 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2789a3700 1 -- 172.21.15.143:0/2708687838 learned_addr learned my addr 172.21.15.143:0/2708687838 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40170/0 (socket says 172.21.15.143:40170) 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2791a4700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb27406a4e0 0x7fb274101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 -- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 msgr2=0x7fb2741015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:49.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb2741015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 -- 172.21.15.143:0/2708687838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb27406a4e0 msgr2=0x7fb274101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb27406a4e0 0x7fb274101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 -- 172.21.15.143:0/2708687838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2741a60c0 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb2799a5700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009790 tx=0x7fb26c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb26a7fc700 10 client.?.objecter ms_handle_connect 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb26a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c021070 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2741a6350 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb26c01d450 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0052a0 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 10 client.30270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2741a6800 con 0x7fb274107e90 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.118+0000 7fb27b408700 1 librados: init done 2024-03-11T21:32:49.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb274103520 con 0x7fb274107e90 2024-03-11T21:32:49.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb26c024070 con 0x7fb274107e90 2024-03-11T21:32:49.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 0x7fb260071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(367..367 src has 87..367) v4 ==== 9903+0+0 (secure 0 0 0) 0x7fb26c00f020 con 0x7fb274107e90 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 10 client.30270.objecter ms_dispatch 0x7fb274001550 osd_map(367..367 src has 87..367) v4 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 3 client.30270.objecter handle_osd_map got epochs [367,367] > 0 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 3 client.30270.objecter handle_osd_map decoding full epoch 367 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb2791a4700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 0x7fb260071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 20 client.30270.objecter dump_active .. 0 homeless 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb2791a4700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 0x7fb260071ce0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fb270005fd0 tx=0x7fb270005d90 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:32:49.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.122+0000 7fb26a7fc700 10 client.30270.objecter ms_handle_connect 0x7fb26006f820 2024-03-11T21:32:49.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.126+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb26c02b070 con 0x7fb274107e90 2024-03-11T21:32:49.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.158 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:49.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:49.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.350+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb274109300 con 0x7fb26006f820 2024-03-11T21:32:49.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.370+0000 7fb26a7fc700 1 -- 172.21.15.143:0/2708687838 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+709479 (secure 0 0 0) 0x7fb274109300 con 0x7fb26006f820 2024-03-11T21:32:49.357 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:32:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 10 librados: watch_flush enter 2024-03-11T21:32:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 10 librados: watch_flush exit 2024-03-11T21:32:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 20 client.30270.objecter shutdown clearing up homeless session... 2024-03-11T21:32:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 10 client.30270.objecter successfully canceled tick 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 msgr2=0x7fb260071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 0x7fb260071ce0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fb270005fd0 tx=0x7fb270005d90 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 msgr2=0x7fb274101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009790 tx=0x7fb26c00ccb0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 shutdown_connections 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb26006f820 0x7fb260071ce0 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274107e90 0x7fb274101b10 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb27406a4e0 0x7fb274101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 --2- 172.21.15.143:0/2708687838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb27406aea0 0x7fb2741015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 >> 172.21.15.143:0/2708687838 conn(0x7fb274077bd0 msgr2=0x7fb274108bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 shutdown_connections 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 -- 172.21.15.143:0/2708687838 wait complete. 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:32:49.378+0000 7fb27b408700 1 librados: shutdown 2024-03-11T21:32:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:32:49.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:49.418 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:49.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:49.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:49.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:49.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:48.544269+0000 mgr.y (mgr.14550) 392 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:49.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.078187+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:49.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.316293+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.316455+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: cluster 2024-03-11T21:32:49.320241+0000 mon.a (mon.0) 2331 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.321999+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.322856+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.342456+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.342777+0000 mon.a (mon.0) 2333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.343862+0000 mon.c (mon.2) 770 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.344192+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.344668+0000 mon.c (mon.2) 771 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:49 smithi203 bash[23654]: audit 2024-03-11T21:32:49.344959+0000 mon.a (mon.0) 2335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:48.544269+0000 mgr.y (mgr.14550) 392 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.078187+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.316293+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.316455+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:49.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: cluster 2024-03-11T21:32:49.320241+0000 mon.a (mon.0) 2331 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.321999+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.322856+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.342456+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.342777+0000 mon.a (mon.0) 2333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.343862+0000 mon.c (mon.2) 770 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.344192+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.344668+0000 mon.c (mon.2) 771 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[29284]: audit 2024-03-11T21:32:49.344959+0000 mon.a (mon.0) 2335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:48.544269+0000 mgr.y (mgr.14550) 392 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.078187+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.316293+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi143-71604-55"}]': finished 2024-03-11T21:32:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.316455+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: cluster 2024-03-11T21:32:49.320241+0000 mon.a (mon.0) 2331 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.321999+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.322856+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.342456+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.342777+0000 mon.a (mon.0) 2333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.343862+0000 mon.c (mon.2) 770 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.344192+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.344668+0000 mon.c (mon.2) 771 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:49 smithi143 bash[21512]: audit 2024-03-11T21:32:49.344959+0000 mon.a (mon.0) 2335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:49.937 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:50.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:50.064 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:50.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:50.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:50.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:50.329 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:50.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:50.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:50.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:50.591 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:50.637 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-03-11T21:32:49.332986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793833,"num_objects":337,"num_object_clones":2,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1618,"num_read_kb":1369,"num_write":1297,"num_write_kb":11273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1772680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2519,"ondisk_log_size":2519,"up":813,"acting":813,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":817,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":585884,"kb_used_data":61928,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749375332,"statfs":{"total":767960285184,"available":767360339968,"internally_reserved":0,"allocated":63414272,"data_stored":44047703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-128,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-7,"num_write_kb":-4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001310"},"pg_stats":[{"pgid":"262.1f","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340444+0000","last_change":"2024-03-11T21:32:47.331859+0000","last_active":"2024-03-11T21:32:47.340444+0000","last_peered":"2024-03-11T21:32:47.340444+0000","last_clean":"2024-03-11T21:32:47.340444+0000","last_became_active":"2024-03-11T21:32:47.331528+0000","last_became_peered":"2024-03-11T21:32:47.331528+0000","last_unstale":"2024-03-11T21:32:47.340444+0000","last_undegraded":"2024-03-11T21:32:47.340444+0000","last_fullsized":"2024-03-11T21:32:47.340444+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:08.680619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.1e","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335940+0000","last_change":"2024-03-11T21:32:47.335554+0000","last_active":"2024-03-11T21:32:47.335940+0000","last_peered":"2024-03-11T21:32:47.335940+0000","last_clean":"2024-03-11T21:32:47.335940+0000","last_became_active":"2024-03-11T21:32:47.335263+0000","last_became_peered":"2024-03-11T21:32:47.335263+0000","last_unstale":"2024-03-11T21:32:47.335940+0000","last_undegraded":"2024-03-11T21:32:47.335940+0000","last_fullsized":"2024-03-11T21:32:47.335940+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:10.126154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"262.1d","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340440+0000","last_change":"2024-03-11T21:32:47.326160+0000","last_active":"2024-03-11T21:32:47.340440+0000","last_peered":"2024-03-11T21:32:47.340440+0000","last_clean":"2024-03-11T21:32:47.340440+0000","last_became_active":"2024-03-11T21:32:47.325859+0000","last_became_peered":"2024-03-11T21:32:47.325859+0000","last_unstale":"2024-03-11T21:32:47.340440+0000","last_undegraded":"2024-03-11T21:32:47.340440+0000","last_fullsized":"2024-03-11T21:32:47.340440+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:09.850467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.1c","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.339428+0000","last_change":"2024-03-11T21:32:47.337965+0000","last_active":"2024-03-11T21:32:47.339428+0000","last_peered":"2024-03-11T21:32:47.339428+0000","last_clean":"2024-03-11T21:32:47.339428+0000","last_became_active":"2024-03-11T21:32:47.337710+0000","last_became_peered":"2024-03-11T21:32:47.337710+0000","last_unstale":"2024-03-11T21:32:47.339428+0000","last_undegraded":"2024-03-11T21:32:47.339428+0000","last_fullsized":"2024-03-11T21:32:47.339428+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:14.296558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"262.1b","version":"0'0","reported_seq":13,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325314+0000","last_change":"2024-03-11T21:32:47.342877+0000","last_active":"2024-03-11T21:32:48.325314+0000","last_peered":"2024-03-11T21:32:48.325314+0000","last_clean":"2024-03-11T21:32:48.325314+0000","last_became_active":"2024-03-11T21:32:47.342526+0000","last_became_peered":"2024-03-11T21:32:47.342526+0000","last_unstale":"2024-03-11T21:32:48.325314+0000","last_undegraded":"2024-03-11T21:32:48.325314+0000","last_fullsized":"2024-03-11T21:32:48.325314+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:46.932124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"262.1a","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340147+0000","last_change":"2024-03-11T21:32:47.325086+0000","last_active":"2024-03-11T21:32:47.340147+0000","last_peered":"2024-03-11T21:32:47.340147+0000","last_clean":"2024-03-11T21:32:47.340147+0000","last_became_active":"2024-03-11T21:32:47.324171+0000","last_became_peered":"2024-03-11T21:32:47.324171+0000","last_unstale":"2024-03-11T21:32:47.340147+0000","last_undegraded":"2024-03-11T21:32:47.340147+0000","last_fullsized":"2024-03-11T21:32:47.340147+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:46.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:32:46.298571+0000","last_change":"2024-03-11T21:32:46.298571+0000","last_active":"2024-03-11T21:32:46.298571+0000","last_peered":"2024-03-11T21:32:46.298571+0000","last_clean":"2024-03-11T21:32:46.298571+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:32:46.298571+0000","last_undegraded":"2024-03-11T21:32:46.298571+0000","last_fullsized":"2024-03-11T21:32:46.298571+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.18","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.346550+0000","last_change":"2024-03-11T21:32:47.346168+0000","last_active":"2024-03-11T21:32:47.346550+0000","last_peered":"2024-03-11T21:32:47.346550+0000","last_clean":"2024-03-11T21:32:47.346550+0000","last_became_active":"2024-03-11T21:32:47.345731+0000","last_became_peered":"2024-03-11T21:32:47.345731+0000","last_unstale":"2024-03-11T21:32:47.346550+0000","last_undegraded":"2024-03-11T21:32:47.346550+0000","last_fullsized":"2024-03-11T21:32:47.346550+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:50.402997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"262.17","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.344308+0000","last_change":"2024-03-11T21:32:47.343821+0000","last_active":"2024-03-11T21:32:47.344308+0000","last_peered":"2024-03-11T21:32:47.344308+0000","last_clean":"2024-03-11T21:32:47.344308+0000","last_became_active":"2024-03-11T21:32:47.340165+0000","last_became_peered":"2024-03-11T21:32:47.340165+0000","last_unstale":"2024-03-11T21:32:47.344308+0000","last_undegraded":"2024-03-11T21:32:47.344308+0000","last_fullsized":"2024-03-11T21:32:47.344308+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:51.236544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"262.16","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337071+0000","last_change":"2024-03-11T21:32:47.336633+0000","last_active":"2024-03-11T21:32:47.337071+0000","last_peered":"2024-03-11T21:32:47.337071+0000","last_clean":"2024-03-11T21:32:47.337071+0000","last_became_active":"2024-03-11T21:32:47.336143+0000","last_became_peered":"2024-03-11T21:32:47.336143+0000","last_unstale":"2024-03-11T21:32:47.337071+0000","last_undegraded":"2024-03-11T21:32:47.337071+0000","last_fullsized":"2024-03-11T21:32:47.337071+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:14.561281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"262.15","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.339385+0000","last_change":"2024-03-11T21:32:47.335760+0000","last_active":"2024-03-11T21:32:47.339385+0000","last_peered":"2024-03-11T21:32:47.339385+0000","last_clean":"2024-03-11T21:32:47.339385+0000","last_became_active":"2024-03-11T21:32:47.335369+0000","last_became_peered":"2024-03-11T21:32:47.335369+0000","last_unstale":"2024-03-11T21:32:47.339385+0000","last_undegraded":"2024-03-11T21:32:47.339385+0000","last_fullsized":"2024-03-11T21:32:47.339385+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:54.670176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"262.14","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340581+0000","last_change":"2024-03-11T21:32:47.339870+0000","last_active":"2024-03-11T21:32:47.340581+0000","last_peered":"2024-03-11T21:32:47.340581+0000","last_clean":"2024-03-11T21:32:47.340581+0000","last_became_active":"2024-03-11T21:32:47.339577+0000","last_became_peered":"2024-03-11T21:32:47.339577+0000","last_unstale":"2024-03-11T21:32:47.340581+0000","last_undegraded":"2024-03-11T21:32:47.340581+0000","last_fullsized":"2024-03-11T21:32:47.340581+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:57.796327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"262.13","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327411+0000","last_change":"2024-03-11T21:32:47.322943+0000","last_active":"2024-03-11T21:32:47.327411+0000","last_peered":"2024-03-11T21:32:47.327411+0000","last_clean":"2024-03-11T21:32:47.327411+0000","last_became_active":"2024-03-11T21:32:47.322469+0000","last_became_peered":"2024-03-11T21:32:47.322469+0000","last_unstale":"2024-03-11T21:32:47.327411+0000","last_undegraded":"2024-03-11T21:32:47.327411+0000","last_fullsized":"2024-03-11T21:32:47.327411+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:08.505390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.12","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.342387+0000","last_change":"2024-03-11T21:32:47.342025+0000","last_active":"2024-03-11T21:32:47.342387+0000","last_peered":"2024-03-11T21:32:47.342387+0000","last_clean":"2024-03-11T21:32:47.342387+0000","last_became_active":"2024-03-11T21:32:47.341686+0000","last_became_peered":"2024-03-11T21:32:47.341686+0000","last_unstale":"2024-03-11T21:32:47.342387+0000","last_undegraded":"2024-03-11T21:32:47.342387+0000","last_fullsized":"2024-03-11T21:32:47.342387+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:06.240636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"262.11","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.341622+0000","last_change":"2024-03-11T21:32:47.331169+0000","last_active":"2024-03-11T21:32:47.341622+0000","last_peered":"2024-03-11T21:32:47.341622+0000","last_clean":"2024-03-11T21:32:47.341622+0000","last_became_active":"2024-03-11T21:32:47.330798+0000","last_became_peered":"2024-03-11T21:32:47.330798+0000","last_unstale":"2024-03-11T21:32:47.341622+0000","last_undegraded":"2024-03-11T21:32:47.341622+0000","last_fullsized":"2024-03-11T21:32:47.341622+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:11.774449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.10","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.341678+0000","last_change":"2024-03-11T21:32:47.340548+0000","last_active":"2024-03-11T21:32:47.341678+0000","last_peered":"2024-03-11T21:32:47.341678+0000","last_clean":"2024-03-11T21:32:47.341678+0000","last_became_active":"2024-03-11T21:32:47.340043+0000","last_became_peered":"2024-03-11T21:32:47.340043+0000","last_unstale":"2024-03-11T21:32:47.341678+0000","last_undegraded":"2024-03-11T21:32:47.341678+0000","last_fullsized":"2024-03-11T21:32:47.341678+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:15.425381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"262.f","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.347382+0000","last_change":"2024-03-11T21:32:47.346982+0000","last_active":"2024-03-11T21:32:47.347382+0000","last_peered":"2024-03-11T21:32:47.347382+0000","last_clean":"2024-03-11T21:32:47.347382+0000","last_became_active":"2024-03-11T21:32:47.346543+0000","last_became_peered":"2024-03-11T21:32:47.346543+0000","last_unstale":"2024-03-11T21:32:47.347382+0000","last_undegraded":"2024-03-11T21:32:47.347382+0000","last_fullsized":"2024-03-11T21:32:47.347382+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:08.931300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"262.e","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340673+0000","last_change":"2024-03-11T21:32:47.327635+0000","last_active":"2024-03-11T21:32:47.340673+0000","last_peered":"2024-03-11T21:32:47.340673+0000","last_clean":"2024-03-11T21:32:47.340673+0000","last_became_active":"2024-03-11T21:32:47.326791+0000","last_became_peered":"2024-03-11T21:32:47.326791+0000","last_unstale":"2024-03-11T21:32:47.340673+0000","last_undegraded":"2024-03-11T21:32:47.340673+0000","last_fullsized":"2024-03-11T21:32:47.340673+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:55.429281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.d","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340512+0000","last_change":"2024-03-11T21:32:47.328612+0000","last_active":"2024-03-11T21:32:47.340512+0000","last_peered":"2024-03-11T21:32:47.340512+0000","last_clean":"2024-03-11T21:32:47.340512+0000","last_became_active":"2024-03-11T21:32:47.323939+0000","last_became_peered":"2024-03-11T21:32:47.323939+0000","last_unstale":"2024-03-11T21:32:47.340512+0000","last_undegraded":"2024-03-11T21:32:47.340512+0000","last_fullsized":"2024-03-11T21:32:47.340512+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:10.539068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:32:46.298571+0000","last_change":"2024-03-11T21:32:46.298571+0000","last_active":"2024-03-11T21:32:46.298571+0000","last_peered":"2024-03-11T21:32:46.298571+0000","last_clean":"2024-03-11T21:32:46.298571+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:32:46.298571+0000","last_undegraded":"2024-03-11T21:32:46.298571+0000","last_fullsized":"2024-03-11T21:32:46.298571+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:32:46.298571+0000","last_change":"2024-03-11T21:32:46.298571+0000","last_active":"2024-03-11T21:32:46.298571+0000","last_peered":"2024-03-11T21:32:46.298571+0000","last_clean":"2024-03-11T21:32:46.298571+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:32:46.298571+0000","last_undegraded":"2024-03-11T21:32:46.298571+0000","last_fullsized":"2024-03-11T21:32:46.298571+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:32:46.298571+0000","last_change":"2024-03-11T21:32:46.298571+0000","last_active":"2024-03-11T21:32:46.298571+0000","last_peered":"2024-03-11T21:32:46.298571+0000","last_clean":"2024-03-11T21:32:46.298571+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:32:46.298571+0000","last_undegraded":"2024-03-11T21:32:46.298571+0000","last_fullsized":"2024-03-11T21:32:46.298571+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.9","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337115+0000","last_change":"2024-03-11T21:32:47.330562+0000","last_active":"2024-03-11T21:32:47.337115+0000","last_peered":"2024-03-11T21:32:47.337115+0000","last_clean":"2024-03-11T21:32:47.337115+0000","last_became_active":"2024-03-11T21:32:47.326734+0000","last_became_peered":"2024-03-11T21:32:47.326734+0000","last_unstale":"2024-03-11T21:32:47.337115+0000","last_undegraded":"2024-03-11T21:32:47.337115+0000","last_fullsized":"2024-03-11T21:32:47.337115+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:11.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"262.8","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.341964+0000","last_change":"2024-03-11T21:32:47.341428+0000","last_active":"2024-03-11T21:32:47.341964+0000","last_peered":"2024-03-11T21:32:47.341964+0000","last_clean":"2024-03-11T21:32:47.341964+0000","last_became_active":"2024-03-11T21:32:47.341069+0000","last_became_peered":"2024-03-11T21:32:47.341069+0000","last_unstale":"2024-03-11T21:32:47.341964+0000","last_undegraded":"2024-03-11T21:32:47.341964+0000","last_fullsized":"2024-03-11T21:32:47.341964+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.682168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"262.2","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340174+0000","last_change":"2024-03-11T21:32:47.327171+0000","last_active":"2024-03-11T21:32:47.340174+0000","last_peered":"2024-03-11T21:32:47.340174+0000","last_clean":"2024-03-11T21:32:47.340174+0000","last_became_active":"2024-03-11T21:32:47.324578+0000","last_became_peered":"2024-03-11T21:32:47.324578+0000","last_unstale":"2024-03-11T21:32:47.340174+0000","last_undegraded":"2024-03-11T21:32:47.340174+0000","last_fullsized":"2024-03-11T21:32:47.340174+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:10.124931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"262.3","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340698+0000","last_change":"2024-03-11T21:32:47.340118+0000","last_active":"2024-03-11T21:32:47.340698+0000","last_peered":"2024-03-11T21:32:47.340698+0000","last_clean":"2024-03-11T21:32:47.340698+0000","last_became_active":"2024-03-11T21:32:47.339712+0000","last_became_peered":"2024-03-11T21:32:47.339712+0000","last_unstale":"2024-03-11T21:32:47.340698+0000","last_undegraded":"2024-03-11T21:32:47.340698+0000","last_fullsized":"2024-03-11T21:32:47.340698+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:01.000694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"262.0","version":"0'0","reported_seq":13,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324729+0000","last_change":"2024-03-11T21:32:47.333825+0000","last_active":"2024-03-11T21:32:48.324729+0000","last_peered":"2024-03-11T21:32:48.324729+0000","last_clean":"2024-03-11T21:32:48.324729+0000","last_became_active":"2024-03-11T21:32:47.333279+0000","last_became_peered":"2024-03-11T21:32:47.333279+0000","last_unstale":"2024-03-11T21:32:48.324729+0000","last_undegraded":"2024-03-11T21:32:48.324729+0000","last_fullsized":"2024-03-11T21:32:48.324729+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.867228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"262.1","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337087+0000","last_change":"2024-03-11T21:32:47.327057+0000","last_active":"2024-03-11T21:32:47.337087+0000","last_peered":"2024-03-11T21:32:47.337087+0000","last_clean":"2024-03-11T21:32:47.337087+0000","last_became_active":"2024-03-11T21:32:47.326302+0000","last_became_peered":"2024-03-11T21:32:47.326302+0000","last_unstale":"2024-03-11T21:32:47.337087+0000","last_undegraded":"2024-03-11T21:32:47.337087+0000","last_fullsized":"2024-03-11T21:32:47.337087+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:06.294079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"262.6","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335886+0000","last_change":"2024-03-11T21:32:47.333694+0000","last_active":"2024-03-11T21:32:47.335886+0000","last_peered":"2024-03-11T21:32:47.335886+0000","last_clean":"2024-03-11T21:32:47.335886+0000","last_became_active":"2024-03-11T21:32:47.332850+0000","last_became_peered":"2024-03-11T21:32:47.332850+0000","last_unstale":"2024-03-11T21:32:47.335886+0000","last_undegraded":"2024-03-11T21:32:47.335886+0000","last_fullsized":"2024-03-11T21:32:47.335886+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:12.840954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"262.7","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334247+0000","last_change":"2024-03-11T21:32:47.331361+0000","last_active":"2024-03-11T21:32:47.334247+0000","last_peered":"2024-03-11T21:32:47.334247+0000","last_clean":"2024-03-11T21:32:47.334247+0000","last_became_active":"2024-03-11T21:32:47.330929+0000","last_became_peered":"2024-03-11T21:32:47.330929+0000","last_unstale":"2024-03-11T21:32:47.334247+0000","last_undegraded":"2024-03-11T21:32:47.334247+0000","last_fullsized":"2024-03-11T21:32:47.334247+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:01.911833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"262.4","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.344011+0000","last_change":"2024-03-11T21:32:47.343378+0000","last_active":"2024-03-11T21:32:47.344011+0000","last_peered":"2024-03-11T21:32:47.344011+0000","last_clean":"2024-03-11T21:32:47.344011+0000","last_became_active":"2024-03-11T21:32:47.340270+0000","last_became_peered":"2024-03-11T21:32:47.340270+0000","last_unstale":"2024-03-11T21:32:47.344011+0000","last_undegraded":"2024-03-11T21:32:47.344011+0000","last_fullsized":"2024-03-11T21:32:47.344011+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:15.371210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"262.5","version":"0'0","reported_seq":10,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340106+0000","last_change":"2024-03-11T21:32:47.337335+0000","last_active":"2024-03-11T21:32:47.340106+0000","last_peered":"2024-03-11T21:32:47.340106+0000","last_clean":"2024-03-11T21:32:47.340106+0000","last_became_active":"2024-03-11T21:32:47.337091+0000","last_became_peered":"2024-03-11T21:32:47.337091+0000","last_unstale":"2024-03-11T21:32:47.340106+0000","last_undegraded":"2024-03-11T21:32:47.340106+0000","last_fullsized":"2024-03-11T21:32:47.340106+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:46.298571+0000","last_clean_scrub_stamp":"2024-03-11T21:32:46.298571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:02.274307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.9","version":"245'17","reported_seq":565,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.309809+0000","last_change":"2024-03-11T21:32:34.298539+0000","last_active":"2024-03-11T21:32:47.309809+0000","last_peered":"2024-03-11T21:32:47.309809+0000","last_clean":"2024-03-11T21:32:47.309809+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:32:47.309809+0000","last_undegraded":"2024-03-11T21:32:47.309809+0000","last_fullsized":"2024-03-11T21:32:47.309809+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-03-11T21:32:02.895303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.895303+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:10.325048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001660573,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"332'18","reported_seq":557,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.317798+0000","last_change":"2024-03-11T21:32:34.299549+0000","last_active":"2024-03-11T21:32:47.317798+0000","last_peered":"2024-03-11T21:32:47.317798+0000","last_clean":"2024-03-11T21:32:47.317798+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:32:47.317798+0000","last_undegraded":"2024-03-11T21:32:47.317798+0000","last_fullsized":"2024-03-11T21:32:47.317798+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'18","last_scrub_stamp":"2024-03-11T21:32:22.035738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.035738+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:24.772394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076326579999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"296'34","reported_seq":570,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340494+0000","last_change":"2024-03-11T21:32:34.335857+0000","last_active":"2024-03-11T21:32:47.340494+0000","last_peered":"2024-03-11T21:32:47.340494+0000","last_clean":"2024-03-11T21:32:47.340494+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:32:47.340494+0000","last_undegraded":"2024-03-11T21:32:47.340494+0000","last_fullsized":"2024-03-11T21:32:47.340494+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024159609999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"310'52","reported_seq":594,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310658+0000","last_change":"2024-03-11T21:32:34.308763+0000","last_active":"2024-03-11T21:32:47.310658+0000","last_peered":"2024-03-11T21:32:47.310658+0000","last_clean":"2024-03-11T21:32:47.310658+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:32:47.310658+0000","last_undegraded":"2024-03-11T21:32:47.310658+0000","last_fullsized":"2024-03-11T21:32:47.310658+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.672997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043833079999999998,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"245'51","reported_seq":591,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327871+0000","last_change":"2024-03-11T21:32:34.306547+0000","last_active":"2024-03-11T21:32:47.327871+0000","last_peered":"2024-03-11T21:32:47.327871+0000","last_clean":"2024-03-11T21:32:47.327871+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:32:47.327871+0000","last_undegraded":"2024-03-11T21:32:47.327871+0000","last_fullsized":"2024-03-11T21:32:47.327871+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:31:05.766455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.766455+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022023799999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":561,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.315347+0000","last_change":"2024-03-11T21:32:34.323770+0000","last_active":"2024-03-11T21:32:47.315347+0000","last_peered":"2024-03-11T21:32:47.315347+0000","last_clean":"2024-03-11T21:32:47.315347+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:32:47.315347+0000","last_undegraded":"2024-03-11T21:32:47.315347+0000","last_fullsized":"2024-03-11T21:32:47.315347+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.672976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036499739999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"353'183","reported_seq":687,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.315750+0000","last_change":"2024-03-11T21:32:36.339723+0000","last_active":"2024-03-11T21:32:47.315750+0000","last_peered":"2024-03-11T21:32:47.315750+0000","last_clean":"2024-03-11T21:32:47.315750+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:32:47.315750+0000","last_undegraded":"2024-03-11T21:32:47.315750+0000","last_fullsized":"2024-03-11T21:32:47.315750+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029727594,"stat_sum":{"num_bytes":117,"num_objects":13,"num_object_clones":1,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":55,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":630,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.325174+0000","last_change":"2024-03-11T21:32:34.312125+0000","last_active":"2024-03-11T21:32:47.325174+0000","last_peered":"2024-03-11T21:32:47.325174+0000","last_clean":"2024-03-11T21:32:47.325174+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:32:47.325174+0000","last_undegraded":"2024-03-11T21:32:47.325174+0000","last_fullsized":"2024-03-11T21:32:47.325174+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'102","last_scrub_stamp":"2024-03-11T21:31:02.926776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.926776+0000","objects_scrubbed":6,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.208104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013258202,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"247'17","reported_seq":552,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324550+0000","last_change":"2024-03-11T21:32:34.353151+0000","last_active":"2024-03-11T21:32:48.324550+0000","last_peered":"2024-03-11T21:32:48.324550+0000","last_clean":"2024-03-11T21:32:48.324550+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:32:48.324550+0000","last_undegraded":"2024-03-11T21:32:48.324550+0000","last_fullsized":"2024-03-11T21:32:48.324550+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077514410000000004,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":591,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324332+0000","last_change":"2024-03-11T21:32:34.352438+0000","last_active":"2024-03-11T21:32:48.324332+0000","last_peered":"2024-03-11T21:32:48.324332+0000","last_clean":"2024-03-11T21:32:48.324332+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:32:48.324332+0000","last_undegraded":"2024-03-11T21:32:48.324332+0000","last_fullsized":"2024-03-11T21:32:48.324332+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076282429999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"247'119","reported_seq":630,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.333704+0000","last_change":"2024-03-11T21:32:34.336215+0000","last_active":"2024-03-11T21:32:47.333704+0000","last_peered":"2024-03-11T21:32:47.333704+0000","last_clean":"2024-03-11T21:32:47.333704+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:32:47.333704+0000","last_undegraded":"2024-03-11T21:32:47.333704+0000","last_fullsized":"2024-03-11T21:32:47.333704+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026596910000000001,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"306'21","reported_seq":569,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311261+0000","last_change":"2024-03-11T21:32:34.309416+0000","last_active":"2024-03-11T21:32:47.311261+0000","last_peered":"2024-03-11T21:32:47.311261+0000","last_clean":"2024-03-11T21:32:47.311261+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:32:47.311261+0000","last_undegraded":"2024-03-11T21:32:47.311261+0000","last_fullsized":"2024-03-11T21:32:47.311261+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041327550000000001,"stat_sum":{"num_bytes":26,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"0'0","reported_seq":539,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327907+0000","last_change":"2024-03-11T21:32:34.307693+0000","last_active":"2024-03-11T21:32:47.327907+0000","last_peered":"2024-03-11T21:32:47.327907+0000","last_clean":"2024-03-11T21:32:47.327907+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:32:47.327907+0000","last_undegraded":"2024-03-11T21:32:47.327907+0000","last_fullsized":"2024-03-11T21:32:47.327907+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.903210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.903210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.418420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002624487,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":570,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325653+0000","last_change":"2024-03-11T21:32:34.354250+0000","last_active":"2024-03-11T21:32:48.325653+0000","last_peered":"2024-03-11T21:32:48.325653+0000","last_clean":"2024-03-11T21:32:48.325653+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:32:48.325653+0000","last_undegraded":"2024-03-11T21:32:48.325653+0000","last_fullsized":"2024-03-11T21:32:48.325653+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080892789999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"311'39","reported_seq":563,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334774+0000","last_change":"2024-03-11T21:32:34.320170+0000","last_active":"2024-03-11T21:32:47.334774+0000","last_peered":"2024-03-11T21:32:47.334774+0000","last_clean":"2024-03-11T21:32:47.334774+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:32:47.334774+0000","last_undegraded":"2024-03-11T21:32:47.334774+0000","last_fullsized":"2024-03-11T21:32:47.334774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001055535,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":607,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334688+0000","last_change":"2024-03-11T21:32:34.321070+0000","last_active":"2024-03-11T21:32:47.334688+0000","last_peered":"2024-03-11T21:32:47.334688+0000","last_clean":"2024-03-11T21:32:47.334688+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:32:47.334688+0000","last_undegraded":"2024-03-11T21:32:47.334688+0000","last_fullsized":"2024-03-11T21:32:47.334688+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789856,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":550,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324496+0000","last_change":"2024-03-11T21:32:34.352506+0000","last_active":"2024-03-11T21:32:48.324496+0000","last_peered":"2024-03-11T21:32:48.324496+0000","last_clean":"2024-03-11T21:32:48.324496+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:32:48.324496+0000","last_undegraded":"2024-03-11T21:32:48.324496+0000","last_fullsized":"2024-03-11T21:32:48.324496+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078384049999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"292'107","reported_seq":621,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324246+0000","last_change":"2024-03-11T21:32:34.350993+0000","last_active":"2024-03-11T21:32:48.324246+0000","last_peered":"2024-03-11T21:32:48.324246+0000","last_clean":"2024-03-11T21:32:48.324246+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:32:48.324246+0000","last_undegraded":"2024-03-11T21:32:48.324246+0000","last_fullsized":"2024-03-11T21:32:48.324246+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067123959999999998,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":429,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.312187+0000","last_change":"2024-03-11T21:32:34.301841+0000","last_active":"2024-03-11T21:32:45.312187+0000","last_peered":"2024-03-11T21:32:45.312187+0000","last_clean":"2024-03-11T21:32:45.312187+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:32:45.312187+0000","last_undegraded":"2024-03-11T21:32:45.312187+0000","last_fullsized":"2024-03-11T21:32:45.312187+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005553105,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":586,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325157+0000","last_change":"2024-03-11T21:32:34.351742+0000","last_active":"2024-03-11T21:32:48.325157+0000","last_peered":"2024-03-11T21:32:48.325157+0000","last_clean":"2024-03-11T21:32:48.325157+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:32:48.325157+0000","last_undegraded":"2024-03-11T21:32:48.325157+0000","last_fullsized":"2024-03-11T21:32:48.325157+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071306190000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":623,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337742+0000","last_change":"2024-03-11T21:32:34.326566+0000","last_active":"2024-03-11T21:32:47.337742+0000","last_peered":"2024-03-11T21:32:47.337742+0000","last_clean":"2024-03-11T21:32:47.337742+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:32:47.337742+0000","last_undegraded":"2024-03-11T21:32:47.337742+0000","last_fullsized":"2024-03-11T21:32:47.337742+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:32:07.554683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.554683+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.283700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096827370000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"311'49","reported_seq":595,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335256+0000","last_change":"2024-03-11T21:32:34.335870+0000","last_active":"2024-03-11T21:32:47.335256+0000","last_peered":"2024-03-11T21:32:47.335256+0000","last_clean":"2024-03-11T21:32:47.335256+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:32:47.335256+0000","last_undegraded":"2024-03-11T21:32:47.335256+0000","last_fullsized":"2024-03-11T21:32:47.335256+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023272950000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"296'34","reported_seq":441,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315964+0000","last_change":"2024-03-11T21:32:34.311124+0000","last_active":"2024-03-11T21:32:45.315964+0000","last_peered":"2024-03-11T21:32:45.315964+0000","last_clean":"2024-03-11T21:32:45.315964+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:32:45.315964+0000","last_undegraded":"2024-03-11T21:32:45.315964+0000","last_fullsized":"2024-03-11T21:32:45.315964+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011782635,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"361'209","reported_seq":701,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340296+0000","last_change":"2024-03-11T21:32:34.338193+0000","last_active":"2024-03-11T21:32:47.340296+0000","last_peered":"2024-03-11T21:32:47.340296+0000","last_clean":"2024-03-11T21:32:47.340296+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:32:47.340296+0000","last_undegraded":"2024-03-11T21:32:47.340296+0000","last_fullsized":"2024-03-11T21:32:47.340296+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.426186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829950000000001,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"296'34","reported_seq":561,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329847+0000","last_change":"2024-03-11T21:32:34.335400+0000","last_active":"2024-03-11T21:32:47.329847+0000","last_peered":"2024-03-11T21:32:47.329847+0000","last_clean":"2024-03-11T21:32:47.329847+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:32:47.329847+0000","last_undegraded":"2024-03-11T21:32:47.329847+0000","last_fullsized":"2024-03-11T21:32:47.329847+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024394040000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"324'14","reported_seq":604,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.317622+0000","last_change":"2024-03-11T21:32:34.298557+0000","last_active":"2024-03-11T21:32:47.317622+0000","last_peered":"2024-03-11T21:32:47.317622+0000","last_clean":"2024-03-11T21:32:47.317622+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:32:47.317622+0000","last_undegraded":"2024-03-11T21:32:47.317622+0000","last_fullsized":"2024-03-11T21:32:47.317622+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'1","last_scrub_stamp":"2024-03-11T21:31:05.888507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.888507+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.208133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0069698629999999998,"stat_sum":{"num_bytes":4194312,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"247'85","reported_seq":608,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.320171+0000","last_change":"2024-03-11T21:32:34.312524+0000","last_active":"2024-03-11T21:32:47.320171+0000","last_peered":"2024-03-11T21:32:47.320171+0000","last_clean":"2024-03-11T21:32:47.320171+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:32:47.320171+0000","last_undegraded":"2024-03-11T21:32:47.320171+0000","last_fullsized":"2024-03-11T21:32:47.320171+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'85","last_scrub_stamp":"2024-03-11T21:32:14.839185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.839185+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.235118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018288136999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"298'64","reported_seq":593,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.332014+0000","last_change":"2024-03-11T21:32:34.332276+0000","last_active":"2024-03-11T21:32:47.332014+0000","last_peered":"2024-03-11T21:32:47.332014+0000","last_clean":"2024-03-11T21:32:47.332014+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:32:47.332014+0000","last_undegraded":"2024-03-11T21:32:47.332014+0000","last_fullsized":"2024-03-11T21:32:47.332014+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016228219999999999,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":605,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327702+0000","last_change":"2024-03-11T21:32:34.307434+0000","last_active":"2024-03-11T21:32:47.327702+0000","last_peered":"2024-03-11T21:32:47.327702+0000","last_clean":"2024-03-11T21:32:47.327702+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:32:47.327702+0000","last_undegraded":"2024-03-11T21:32:47.327702+0000","last_fullsized":"2024-03-11T21:32:47.327702+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.819204+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029485240000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":587,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.315054+0000","last_change":"2024-03-11T21:32:34.315488+0000","last_active":"2024-03-11T21:32:47.315054+0000","last_peered":"2024-03-11T21:32:47.315054+0000","last_clean":"2024-03-11T21:32:47.315054+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:32:47.315054+0000","last_undegraded":"2024-03-11T21:32:47.315054+0000","last_fullsized":"2024-03-11T21:32:47.315054+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001496713,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"245'51","reported_seq":589,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311714+0000","last_change":"2024-03-11T21:32:34.308970+0000","last_active":"2024-03-11T21:32:47.311714+0000","last_peered":"2024-03-11T21:32:47.311714+0000","last_clean":"2024-03-11T21:32:47.311714+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:32:47.311714+0000","last_undegraded":"2024-03-11T21:32:47.311714+0000","last_fullsized":"2024-03-11T21:32:47.311714+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:32:01.854464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.854464+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.344378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001987904,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"282'2","reported_seq":544,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340534+0000","last_change":"2024-03-11T21:32:34.338218+0000","last_active":"2024-03-11T21:32:47.340534+0000","last_peered":"2024-03-11T21:32:47.340534+0000","last_clean":"2024-03-11T21:32:47.340534+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:32:47.340534+0000","last_undegraded":"2024-03-11T21:32:47.340534+0000","last_fullsized":"2024-03-11T21:32:47.340534+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011248230000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.323891+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:32:47.323891+0000","last_peered":"2024-03-11T21:32:47.323891+0000","last_clean":"2024-03-11T21:32:47.323891+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:32:47.323891+0000","last_undegraded":"2024-03-11T21:32:47.323891+0000","last_fullsized":"2024-03-11T21:32:47.323891+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.119355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.330835+0000","last_change":"2024-03-11T21:31:05.576659+0000","last_active":"2024-03-11T21:32:47.330835+0000","last_peered":"2024-03-11T21:32:47.330835+0000","last_clean":"2024-03-11T21:32:47.330835+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:32:47.330835+0000","last_undegraded":"2024-03-11T21:32:47.330835+0000","last_fullsized":"2024-03-11T21:32:47.330835+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.576558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.576558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":463,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329518+0000","last_change":"2024-03-11T21:30:50.966683+0000","last_active":"2024-03-11T21:32:47.329518+0000","last_peered":"2024-03-11T21:32:47.329518+0000","last_clean":"2024-03-11T21:32:47.329518+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:32:47.329518+0000","last_undegraded":"2024-03-11T21:32:47.329518+0000","last_fullsized":"2024-03-11T21:32:47.329518+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:50.966582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:50.966582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":453,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327814+0000","last_change":"2024-03-11T21:28:57.522457+0000","last_active":"2024-03-11T21:32:47.327814+0000","last_peered":"2024-03-11T21:32:47.327814+0000","last_clean":"2024-03-11T21:32:47.327814+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:32:47.327814+0000","last_undegraded":"2024-03-11T21:32:47.327814+0000","last_fullsized":"2024-03-11T21:32:47.327814+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":358,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.314871+0000","last_change":"2024-03-11T21:31:37.402407+0000","last_active":"2024-03-11T21:32:45.314871+0000","last_peered":"2024-03-11T21:32:45.314871+0000","last_clean":"2024-03-11T21:32:45.314871+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:32:45.314871+0000","last_undegraded":"2024-03-11T21:32:45.314871+0000","last_fullsized":"2024-03-11T21:32:45.314871+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.897147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.897147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338357+0000","last_change":"2024-03-11T21:30:58.478329+0000","last_active":"2024-03-11T21:32:47.338357+0000","last_peered":"2024-03-11T21:32:47.338357+0000","last_clean":"2024-03-11T21:32:47.338357+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:32:47.338357+0000","last_undegraded":"2024-03-11T21:32:47.338357+0000","last_fullsized":"2024-03-11T21:32:47.338357+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.478183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.478183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.339068+0000","last_change":"2024-03-11T21:30:59.448512+0000","last_active":"2024-03-11T21:32:47.339068+0000","last_peered":"2024-03-11T21:32:47.339068+0000","last_clean":"2024-03-11T21:32:47.339068+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:32:47.339068+0000","last_undegraded":"2024-03-11T21:32:47.339068+0000","last_fullsized":"2024-03-11T21:32:47.339068+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.448405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":472,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312208+0000","last_change":"2024-03-11T21:32:15.794130+0000","last_active":"2024-03-11T21:32:47.312208+0000","last_peered":"2024-03-11T21:32:47.312208+0000","last_clean":"2024-03-11T21:32:47.312208+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:32:47.312208+0000","last_undegraded":"2024-03-11T21:32:47.312208+0000","last_fullsized":"2024-03-11T21:32:47.312208+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.793910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.793910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.911787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":465,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312046+0000","last_change":"2024-03-11T21:30:54.963682+0000","last_active":"2024-03-11T21:32:47.312046+0000","last_peered":"2024-03-11T21:32:47.312046+0000","last_clean":"2024-03-11T21:32:47.312046+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:32:47.312046+0000","last_undegraded":"2024-03-11T21:32:47.312046+0000","last_fullsized":"2024-03-11T21:32:47.312046+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.963569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.963569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":461,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326113+0000","last_change":"2024-03-11T21:31:01.656496+0000","last_active":"2024-03-11T21:32:47.326113+0000","last_peered":"2024-03-11T21:32:47.326113+0000","last_clean":"2024-03-11T21:32:47.326113+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:32:47.326113+0000","last_undegraded":"2024-03-11T21:32:47.326113+0000","last_fullsized":"2024-03-11T21:32:47.326113+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.656422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.656422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325357+0000","last_change":"2024-03-11T21:30:51.770965+0000","last_active":"2024-03-11T21:32:48.325357+0000","last_peered":"2024-03-11T21:32:48.325357+0000","last_clean":"2024-03-11T21:32:48.325357+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:32:48.325357+0000","last_undegraded":"2024-03-11T21:32:48.325357+0000","last_fullsized":"2024-03-11T21:32:48.325357+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.770705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.770705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":463,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335716+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:32:47.335716+0000","last_peered":"2024-03-11T21:32:47.335716+0000","last_clean":"2024-03-11T21:32:47.335716+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:32:47.335716+0000","last_undegraded":"2024-03-11T21:32:47.335716+0000","last_fullsized":"2024-03-11T21:32:47.335716+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.448006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325526+0000","last_change":"2024-03-11T21:31:02.849078+0000","last_active":"2024-03-11T21:32:48.325526+0000","last_peered":"2024-03-11T21:32:48.325526+0000","last_clean":"2024-03-11T21:32:48.325526+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:32:48.325526+0000","last_undegraded":"2024-03-11T21:32:48.325526+0000","last_fullsized":"2024-03-11T21:32:48.325526+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.848978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.848978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":457,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.320194+0000","last_change":"2024-03-11T21:28:53.826473+0000","last_active":"2024-03-11T21:32:47.320194+0000","last_peered":"2024-03-11T21:32:47.320194+0000","last_clean":"2024-03-11T21:32:47.320194+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:32:47.320194+0000","last_undegraded":"2024-03-11T21:32:47.320194+0000","last_fullsized":"2024-03-11T21:32:47.320194+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.672931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311278+0000","last_change":"2024-03-11T21:31:00.856017+0000","last_active":"2024-03-11T21:32:47.311278+0000","last_peered":"2024-03-11T21:32:47.311278+0000","last_clean":"2024-03-11T21:32:47.311278+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:32:47.311278+0000","last_undegraded":"2024-03-11T21:32:47.311278+0000","last_fullsized":"2024-03-11T21:32:47.311278+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.855899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.855899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122919+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":461,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335539+0000","last_change":"2024-03-11T21:32:02.604095+0000","last_active":"2024-03-11T21:32:47.335539+0000","last_peered":"2024-03-11T21:32:47.335539+0000","last_clean":"2024-03-11T21:32:47.335539+0000","last_became_active":"2024-03-11T21:28:57.548665+0000","last_became_peered":"2024-03-11T21:28:57.548665+0000","last_unstale":"2024-03-11T21:32:47.335539+0000","last_undegraded":"2024-03-11T21:32:47.335539+0000","last_fullsized":"2024-03-11T21:32:47.335539+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.603946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.603946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.775791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326741+0000","last_change":"2024-03-11T21:30:57.522881+0000","last_active":"2024-03-11T21:32:47.326741+0000","last_peered":"2024-03-11T21:32:47.326741+0000","last_clean":"2024-03-11T21:32:47.326741+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:32:47.326741+0000","last_undegraded":"2024-03-11T21:32:47.326741+0000","last_fullsized":"2024-03-11T21:32:47.326741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:57.522689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.522689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752490+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":463,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329545+0000","last_change":"2024-03-11T21:32:04.563626+0000","last_active":"2024-03-11T21:32:47.329545+0000","last_peered":"2024-03-11T21:32:47.329545+0000","last_clean":"2024-03-11T21:32:47.329545+0000","last_became_active":"2024-03-11T21:28:53.852502+0000","last_became_peered":"2024-03-11T21:28:53.852502+0000","last_unstale":"2024-03-11T21:32:47.329545+0000","last_undegraded":"2024-03-11T21:32:47.329545+0000","last_fullsized":"2024-03-11T21:32:47.329545+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:34.350718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":453,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326777+0000","last_change":"2024-03-11T21:28:57.495142+0000","last_active":"2024-03-11T21:32:47.326777+0000","last_peered":"2024-03-11T21:32:47.326777+0000","last_clean":"2024-03-11T21:32:47.326777+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:32:47.326777+0000","last_undegraded":"2024-03-11T21:32:47.326777+0000","last_fullsized":"2024-03-11T21:32:47.326777+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":465,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.313650+0000","last_change":"2024-03-11T21:32:14.693680+0000","last_active":"2024-03-11T21:32:47.313650+0000","last_peered":"2024-03-11T21:32:47.313650+0000","last_clean":"2024-03-11T21:32:47.313650+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:32:47.313650+0000","last_undegraded":"2024-03-11T21:32:47.313650+0000","last_fullsized":"2024-03-11T21:32:47.313650+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.693586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.693586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.646372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":334,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315252+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:32:45.315252+0000","last_peered":"2024-03-11T21:32:45.315252+0000","last_clean":"2024-03-11T21:32:45.315252+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:32:45.315252+0000","last_undegraded":"2024-03-11T21:32:45.315252+0000","last_fullsized":"2024-03-11T21:32:45.315252+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.546676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":461,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334318+0000","last_change":"2024-03-11T21:32:07.649024+0000","last_active":"2024-03-11T21:32:47.334318+0000","last_peered":"2024-03-11T21:32:47.334318+0000","last_clean":"2024-03-11T21:32:47.334318+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:32:47.334318+0000","last_undegraded":"2024-03-11T21:32:47.334318+0000","last_fullsized":"2024-03-11T21:32:47.334318+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.648800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.648800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.940650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":461,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334831+0000","last_change":"2024-03-11T21:32:15.722549+0000","last_active":"2024-03-11T21:32:47.334831+0000","last_peered":"2024-03-11T21:32:47.334831+0000","last_clean":"2024-03-11T21:32:47.334831+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:32:47.334831+0000","last_undegraded":"2024-03-11T21:32:47.334831+0000","last_fullsized":"2024-03-11T21:32:47.334831+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.722443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.722443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.727090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325986+0000","last_change":"2024-03-11T21:32:14.976068+0000","last_active":"2024-03-11T21:32:48.325986+0000","last_peered":"2024-03-11T21:32:48.325986+0000","last_clean":"2024-03-11T21:32:48.325986+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:32:48.325986+0000","last_undegraded":"2024-03-11T21:32:48.325986+0000","last_fullsized":"2024-03-11T21:32:48.325986+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.975966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.975966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:40.129592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":463,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.318408+0000","last_change":"2024-03-11T21:30:56.975013+0000","last_active":"2024-03-11T21:32:47.318408+0000","last_peered":"2024-03-11T21:32:47.318408+0000","last_clean":"2024-03-11T21:32:47.318408+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:32:47.318408+0000","last_undegraded":"2024-03-11T21:32:47.318408+0000","last_fullsized":"2024-03-11T21:32:47.318408+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:56.974794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.974794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":455,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329662+0000","last_change":"2024-03-11T21:28:53.852772+0000","last_active":"2024-03-11T21:32:47.329662+0000","last_peered":"2024-03-11T21:32:47.329662+0000","last_clean":"2024-03-11T21:32:47.329662+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:32:47.329662+0000","last_undegraded":"2024-03-11T21:32:47.329662+0000","last_fullsized":"2024-03-11T21:32:47.329662+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.420917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":461,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326890+0000","last_change":"2024-03-11T21:30:55.794089+0000","last_active":"2024-03-11T21:32:47.326890+0000","last_peered":"2024-03-11T21:32:47.326890+0000","last_clean":"2024-03-11T21:32:47.326890+0000","last_became_active":"2024-03-11T21:28:57.523449+0000","last_became_peered":"2024-03-11T21:28:57.523449+0000","last_unstale":"2024-03-11T21:32:47.326890+0000","last_undegraded":"2024-03-11T21:32:47.326890+0000","last_fullsized":"2024-03-11T21:32:47.326890+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.793796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.793796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":457,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.313404+0000","last_change":"2024-03-11T21:28:53.827705+0000","last_active":"2024-03-11T21:32:47.313404+0000","last_peered":"2024-03-11T21:32:47.313404+0000","last_clean":"2024-03-11T21:32:47.313404+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:32:47.313404+0000","last_undegraded":"2024-03-11T21:32:47.313404+0000","last_fullsized":"2024-03-11T21:32:47.313404+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.672906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325684+0000","last_change":"2024-03-11T21:30:59.897121+0000","last_active":"2024-03-11T21:32:48.325684+0000","last_peered":"2024-03-11T21:32:48.325684+0000","last_clean":"2024-03-11T21:32:48.325684+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:32:48.325684+0000","last_undegraded":"2024-03-11T21:32:48.325684+0000","last_fullsized":"2024-03-11T21:32:48.325684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.897032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.897032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":463,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338364+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:32:47.338364+0000","last_peered":"2024-03-11T21:32:47.338364+0000","last_clean":"2024-03-11T21:32:47.338364+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:32:47.338364+0000","last_undegraded":"2024-03-11T21:32:47.338364+0000","last_fullsized":"2024-03-11T21:32:47.338364+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:45.699273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":464,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311394+0000","last_change":"2024-03-11T21:30:24.759417+0000","last_active":"2024-03-11T21:32:47.311394+0000","last_peered":"2024-03-11T21:32:47.311394+0000","last_clean":"2024-03-11T21:32:47.311394+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:32:47.311394+0000","last_undegraded":"2024-03-11T21:32:47.311394+0000","last_fullsized":"2024-03-11T21:32:47.311394+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.759285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.759285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":334,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.319018+0000","last_change":"2024-03-11T21:32:28.978945+0000","last_active":"2024-03-11T21:32:45.319018+0000","last_peered":"2024-03-11T21:32:45.319018+0000","last_clean":"2024-03-11T21:32:45.319018+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:32:45.319018+0000","last_undegraded":"2024-03-11T21:32:45.319018+0000","last_fullsized":"2024-03-11T21:32:45.319018+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.470081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"259'7","reported_seq":524,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311793+0000","last_change":"2024-03-11T21:32:43.306060+0000","last_active":"2024-03-11T21:32:47.311793+0000","last_peered":"2024-03-11T21:32:47.311793+0000","last_clean":"2024-03-11T21:32:47.311793+0000","last_became_active":"2024-03-11T21:32:43.305002+0000","last_became_peered":"2024-03-11T21:32:43.305002+0000","last_unstale":"2024-03-11T21:32:47.311793+0000","last_undegraded":"2024-03-11T21:32:47.311793+0000","last_fullsized":"2024-03-11T21:32:47.311793+0000","mapping_epoch":139,"log_start":"259'7","ondisk_log_start":"259'7","created":139,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:47.897070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.897070+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":7,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":472,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326174+0000","last_change":"2024-03-11T21:30:42.847714+0000","last_active":"2024-03-11T21:32:48.326174+0000","last_peered":"2024-03-11T21:32:48.326174+0000","last_clean":"2024-03-11T21:32:48.326174+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:32:48.326174+0000","last_undegraded":"2024-03-11T21:32:48.326174+0000","last_fullsized":"2024-03-11T21:32:48.326174+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.847476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.847476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.986334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":742,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.330463+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:32:47.330463+0000","last_peered":"2024-03-11T21:32:47.330463+0000","last_clean":"2024-03-11T21:32:47.330463+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:32:47.330463+0000","last_undegraded":"2024-03-11T21:32:47.330463+0000","last_fullsized":"2024-03-11T21:32:47.330463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.345971+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:32:47.345971+0000","last_peered":"2024-03-11T21:32:47.345971+0000","last_clean":"2024-03-11T21:32:47.345971+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:32:47.345971+0000","last_undegraded":"2024-03-11T21:32:47.345971+0000","last_fullsized":"2024-03-11T21:32:47.345971+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422122+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.314065+0000","last_change":"2024-03-11T21:30:55.823529+0000","last_active":"2024-03-11T21:32:47.314065+0000","last_peered":"2024-03-11T21:32:47.314065+0000","last_clean":"2024-03-11T21:32:47.314065+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:32:47.314065+0000","last_undegraded":"2024-03-11T21:32:47.314065+0000","last_fullsized":"2024-03-11T21:32:47.314065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.823180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.823180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":620,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329870+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:32:47.329870+0000","last_peered":"2024-03-11T21:32:47.329870+0000","last_clean":"2024-03-11T21:32:47.329870+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:32:47.329870+0000","last_undegraded":"2024-03-11T21:32:47.329870+0000","last_fullsized":"2024-03-11T21:32:47.329870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.496671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"259'11","reported_seq":525,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326258+0000","last_change":"2024-03-11T21:32:32.305524+0000","last_active":"2024-03-11T21:32:48.326258+0000","last_peered":"2024-03-11T21:32:48.326258+0000","last_clean":"2024-03-11T21:32:48.326258+0000","last_became_active":"2024-03-11T21:32:32.305127+0000","last_became_peered":"2024-03-11T21:32:32.305127+0000","last_unstale":"2024-03-11T21:32:48.326258+0000","last_undegraded":"2024-03-11T21:32:48.326258+0000","last_fullsized":"2024-03-11T21:32:48.326258+0000","mapping_epoch":139,"log_start":"259'11","ondisk_log_start":"259'11","created":139,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:39.934432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:39.934432+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":478,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326224+0000","last_change":"2024-03-11T21:31:51.834491+0000","last_active":"2024-03-11T21:32:47.326224+0000","last_peered":"2024-03-11T21:32:47.326224+0000","last_clean":"2024-03-11T21:32:47.326224+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:32:47.326224+0000","last_undegraded":"2024-03-11T21:32:47.326224+0000","last_fullsized":"2024-03-11T21:32:47.326224+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.834251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.834251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.072172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.323679+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:32:47.323679+0000","last_peered":"2024-03-11T21:32:47.323679+0000","last_clean":"2024-03-11T21:32:47.323679+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:32:47.323679+0000","last_undegraded":"2024-03-11T21:32:47.323679+0000","last_fullsized":"2024-03-11T21:32:47.323679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.006264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":516,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315731+0000","last_change":"2024-03-11T21:32:06.996760+0000","last_active":"2024-03-11T21:32:45.315731+0000","last_peered":"2024-03-11T21:32:45.315731+0000","last_clean":"2024-03-11T21:32:45.315731+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:32:45.315731+0000","last_undegraded":"2024-03-11T21:32:45.315731+0000","last_fullsized":"2024-03-11T21:32:45.315731+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.996518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.996518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.387438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311677+0000","last_change":"2024-03-11T21:32:28.781453+0000","last_active":"2024-03-11T21:32:47.311677+0000","last_peered":"2024-03-11T21:32:47.311677+0000","last_clean":"2024-03-11T21:32:47.311677+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:32:47.311677+0000","last_undegraded":"2024-03-11T21:32:47.311677+0000","last_fullsized":"2024-03-11T21:32:47.311677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.324762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":623,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324867+0000","last_change":"2024-03-11T21:32:25.048822+0000","last_active":"2024-03-11T21:32:48.324867+0000","last_peered":"2024-03-11T21:32:48.324867+0000","last_clean":"2024-03-11T21:32:48.324867+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:32:48.324867+0000","last_undegraded":"2024-03-11T21:32:48.324867+0000","last_fullsized":"2024-03-11T21:32:48.324867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.048696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.048696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:46.961096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":516,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.313881+0000","last_change":"2024-03-11T21:32:35.401156+0000","last_active":"2024-03-11T21:32:47.313881+0000","last_peered":"2024-03-11T21:32:47.313881+0000","last_clean":"2024-03-11T21:32:47.313881+0000","last_became_active":"2024-03-11T21:32:35.400680+0000","last_became_peered":"2024-03-11T21:32:35.400680+0000","last_unstale":"2024-03-11T21:32:47.313881+0000","last_undegraded":"2024-03-11T21:32:47.313881+0000","last_fullsized":"2024-03-11T21:32:47.313881+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.918563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.918563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":469,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324543+0000","last_change":"2024-03-11T21:30:51.971724+0000","last_active":"2024-03-11T21:32:47.324543+0000","last_peered":"2024-03-11T21:32:47.324543+0000","last_clean":"2024-03-11T21:32:47.324543+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:32:47.324543+0000","last_undegraded":"2024-03-11T21:32:47.324543+0000","last_fullsized":"2024-03-11T21:32:47.324543+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.971619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.971619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338828+0000","last_change":"2024-03-11T21:29:08.366566+0000","last_active":"2024-03-11T21:32:47.338828+0000","last_peered":"2024-03-11T21:32:47.338828+0000","last_clean":"2024-03-11T21:32:47.338828+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:32:47.338828+0000","last_undegraded":"2024-03-11T21:32:47.338828+0000","last_fullsized":"2024-03-11T21:32:47.338828+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:29:08.366434+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:29:08.366434+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.420813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311667+0000","last_change":"2024-03-11T21:30:50.697950+0000","last_active":"2024-03-11T21:32:47.311667+0000","last_peered":"2024-03-11T21:32:47.311667+0000","last_clean":"2024-03-11T21:32:47.311667+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:32:47.311667+0000","last_undegraded":"2024-03-11T21:32:47.311667+0000","last_fullsized":"2024-03-11T21:32:47.311667+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:30:50.697855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310983+0000","last_change":"2024-03-11T21:32:16.782474+0000","last_active":"2024-03-11T21:32:47.310983+0000","last_peered":"2024-03-11T21:32:47.310983+0000","last_clean":"2024-03-11T21:32:47.310983+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:32:47.310983+0000","last_undegraded":"2024-03-11T21:32:47.310983+0000","last_fullsized":"2024-03-11T21:32:47.310983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.782400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.782400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.897094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":611,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324793+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:32:47.324793+0000","last_peered":"2024-03-11T21:32:47.324793+0000","last_clean":"2024-03-11T21:32:47.324793+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:32:47.324793+0000","last_undegraded":"2024-03-11T21:32:47.324793+0000","last_fullsized":"2024-03-11T21:32:47.324793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":533,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337858+0000","last_change":"2024-03-11T21:32:25.706118+0000","last_active":"2024-03-11T21:32:47.337858+0000","last_peered":"2024-03-11T21:32:47.337858+0000","last_clean":"2024-03-11T21:32:47.337858+0000","last_became_active":"2024-03-11T21:32:25.349196+0000","last_became_peered":"2024-03-11T21:32:25.349196+0000","last_unstale":"2024-03-11T21:32:47.337858+0000","last_undegraded":"2024-03-11T21:32:47.337858+0000","last_fullsized":"2024-03-11T21:32:47.337858+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.705939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:42.546466+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.705939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:53.802750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":335,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.308298+0000","last_change":"2024-03-11T21:31:55.095325+0000","last_active":"2024-03-11T21:32:45.308298+0000","last_peered":"2024-03-11T21:32:45.308298+0000","last_clean":"2024-03-11T21:32:45.308298+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:32:45.308298+0000","last_undegraded":"2024-03-11T21:32:45.308298+0000","last_fullsized":"2024-03-11T21:32:45.308298+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.095111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.095111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.214739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327979+0000","last_change":"2024-03-11T21:31:54.594364+0000","last_active":"2024-03-11T21:32:47.327979+0000","last_peered":"2024-03-11T21:32:47.327979+0000","last_clean":"2024-03-11T21:32:47.327979+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:32:47.327979+0000","last_undegraded":"2024-03-11T21:32:47.327979+0000","last_fullsized":"2024-03-11T21:32:47.327979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:31:54.594116+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.594116+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:55.246404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":631,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334764+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:32:47.334764+0000","last_peered":"2024-03-11T21:32:47.334764+0000","last_clean":"2024-03-11T21:32:47.334764+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:32:47.334764+0000","last_undegraded":"2024-03-11T21:32:47.334764+0000","last_fullsized":"2024-03-11T21:32:47.334764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422289+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":514,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315129+0000","last_change":"2024-03-11T21:31:37.399161+0000","last_active":"2024-03-11T21:32:45.315129+0000","last_peered":"2024-03-11T21:32:45.315129+0000","last_clean":"2024-03-11T21:32:45.315129+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:32:45.315129+0000","last_undegraded":"2024-03-11T21:32:45.315129+0000","last_fullsized":"2024-03-11T21:32:45.315129+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:14.307354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.307354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":628,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312618+0000","last_change":"2024-03-11T21:32:05.876383+0000","last_active":"2024-03-11T21:32:47.312618+0000","last_peered":"2024-03-11T21:32:47.312618+0000","last_clean":"2024-03-11T21:32:47.312618+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:32:47.312618+0000","last_undegraded":"2024-03-11T21:32:47.312618+0000","last_fullsized":"2024-03-11T21:32:47.312618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.876018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.876018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.769278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"142'2","reported_seq":524,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.333573+0000","last_change":"2024-03-11T21:32:29.376547+0000","last_active":"2024-03-11T21:32:47.333573+0000","last_peered":"2024-03-11T21:32:47.333573+0000","last_clean":"2024-03-11T21:32:47.333573+0000","last_became_active":"2024-03-11T21:32:29.375562+0000","last_became_peered":"2024-03-11T21:32:29.375562+0000","last_unstale":"2024-03-11T21:32:47.333573+0000","last_undegraded":"2024-03-11T21:32:47.333573+0000","last_fullsized":"2024-03-11T21:32:47.333573+0000","mapping_epoch":295,"log_start":"142'2","ondisk_log_start":"142'2","created":139,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.545525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.545525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":2,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.059390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":484,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326379+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:32:47.326379+0000","last_peered":"2024-03-11T21:32:47.326379+0000","last_clean":"2024-03-11T21:32:47.326379+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:32:47.326379+0000","last_undegraded":"2024-03-11T21:32:47.326379+0000","last_fullsized":"2024-03-11T21:32:47.326379+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.202895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328325+0000","last_change":"2024-03-11T21:32:28.904947+0000","last_active":"2024-03-11T21:32:47.328325+0000","last_peered":"2024-03-11T21:32:47.328325+0000","last_clean":"2024-03-11T21:32:47.328325+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:32:47.328325+0000","last_undegraded":"2024-03-11T21:32:47.328325+0000","last_fullsized":"2024-03-11T21:32:47.328325+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:32:28.904821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.904821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.529402+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":482,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.310772+0000","last_change":"2024-03-11T21:32:04.059244+0000","last_active":"2024-03-11T21:32:45.310772+0000","last_peered":"2024-03-11T21:32:45.310772+0000","last_clean":"2024-03-11T21:32:45.310772+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:32:45.310772+0000","last_undegraded":"2024-03-11T21:32:45.310772+0000","last_fullsized":"2024-03-11T21:32:45.310772+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.059137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:17.202132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":760,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328251+0000","last_change":"2024-03-11T21:32:07.803946+0000","last_active":"2024-03-11T21:32:47.328251+0000","last_peered":"2024-03-11T21:32:47.328251+0000","last_clean":"2024-03-11T21:32:47.328251+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:32:47.328251+0000","last_undegraded":"2024-03-11T21:32:47.328251+0000","last_fullsized":"2024-03-11T21:32:47.328251+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:32:07.803828+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.803828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.048925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":630,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324167+0000","last_change":"2024-03-11T21:32:08.744653+0000","last_active":"2024-03-11T21:32:47.324167+0000","last_peered":"2024-03-11T21:32:47.324167+0000","last_clean":"2024-03-11T21:32:47.324167+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:32:47.324167+0000","last_undegraded":"2024-03-11T21:32:47.324167+0000","last_fullsized":"2024-03-11T21:32:47.324167+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.744422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.744422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.054417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"259'7","reported_seq":417,"reported_epoch":364,"state":"clean+premerge+peered","last_fresh":"2024-03-11T21:32:45.336815+0000","last_change":"2024-03-11T21:32:45.336815+0000","last_active":"2024-03-11T21:32:44.301958+0000","last_peered":"2024-03-11T21:32:45.336815+0000","last_clean":"2024-03-11T21:32:45.336815+0000","last_became_active":"2024-03-11T21:32:17.323541+0000","last_became_peered":"2024-03-11T21:32:45.336303+0000","last_unstale":"2024-03-11T21:32:45.336815+0000","last_undegraded":"2024-03-11T21:32:45.336815+0000","last_fullsized":"2024-03-11T21:32:45.336815+0000","mapping_epoch":295,"log_start":"259'7","ondisk_log_start":"259'7","created":139,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.866571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:42.866571+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.866571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":7,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":479,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328340+0000","last_change":"2024-03-11T21:31:37.390219+0000","last_active":"2024-03-11T21:32:47.328340+0000","last_peered":"2024-03-11T21:32:47.328340+0000","last_clean":"2024-03-11T21:32:47.328340+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:32:47.328340+0000","last_undegraded":"2024-03-11T21:32:47.328340+0000","last_fullsized":"2024-03-11T21:32:47.328340+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.818577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.818577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324603+0000","last_change":"2024-03-11T21:31:37.390418+0000","last_active":"2024-03-11T21:32:47.324603+0000","last_peered":"2024-03-11T21:32:47.324603+0000","last_clean":"2024-03-11T21:32:47.324603+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:32:47.324603+0000","last_undegraded":"2024-03-11T21:32:47.324603+0000","last_fullsized":"2024-03-11T21:32:47.324603+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:13.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.487366+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":510,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.311256+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:32:45.311256+0000","last_peered":"2024-03-11T21:32:45.311256+0000","last_clean":"2024-03-11T21:32:45.311256+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:32:45.311256+0000","last_undegraded":"2024-03-11T21:32:45.311256+0000","last_fullsized":"2024-03-11T21:32:45.311256+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:48.923389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":744,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.323499+0000","last_change":"2024-03-11T21:28:54.069307+0000","last_active":"2024-03-11T21:32:47.323499+0000","last_peered":"2024-03-11T21:32:47.323499+0000","last_clean":"2024-03-11T21:32:47.323499+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:32:47.323499+0000","last_undegraded":"2024-03-11T21:32:47.323499+0000","last_fullsized":"2024-03-11T21:32:47.323499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:54.069179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.069179+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.672955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":629,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324532+0000","last_change":"2024-03-11T21:32:05.838146+0000","last_active":"2024-03-11T21:32:47.324532+0000","last_peered":"2024-03-11T21:32:47.324532+0000","last_clean":"2024-03-11T21:32:47.324532+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:32:47.324532+0000","last_undegraded":"2024-03-11T21:32:47.324532+0000","last_fullsized":"2024-03-11T21:32:47.324532+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.838032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.838032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.307236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"259'11","reported_seq":397,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.316811+0000","last_change":"2024-03-11T21:32:21.312533+0000","last_active":"2024-03-11T21:32:45.316811+0000","last_peered":"2024-03-11T21:32:45.316811+0000","last_clean":"2024-03-11T21:32:45.316811+0000","last_became_active":"2024-03-11T21:32:21.312018+0000","last_became_peered":"2024-03-11T21:32:21.312018+0000","last_unstale":"2024-03-11T21:32:45.316811+0000","last_undegraded":"2024-03-11T21:32:45.316811+0000","last_fullsized":"2024-03-11T21:32:45.316811+0000","mapping_epoch":295,"log_start":"259'11","ondisk_log_start":"259'11","created":139,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'9","last_scrub_stamp":"2024-03-11T21:30:29.722893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:29.722893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":477,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334047+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:32:47.334047+0000","last_peered":"2024-03-11T21:32:47.334047+0000","last_clean":"2024-03-11T21:32:47.334047+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:32:47.334047+0000","last_undegraded":"2024-03-11T21:32:47.334047+0000","last_fullsized":"2024-03-11T21:32:47.334047+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.054249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":649,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311128+0000","last_change":"2024-03-11T21:30:29.035776+0000","last_active":"2024-03-11T21:32:47.311128+0000","last_peered":"2024-03-11T21:32:47.311128+0000","last_clean":"2024-03-11T21:32:47.311128+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:32:47.311128+0000","last_undegraded":"2024-03-11T21:32:47.311128+0000","last_fullsized":"2024-03-11T21:32:47.311128+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:30:29.035470+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:30:29.035470+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329313+0000","last_change":"2024-03-11T21:31:37.402980+0000","last_active":"2024-03-11T21:32:47.329313+0000","last_peered":"2024-03-11T21:32:47.329313+0000","last_clean":"2024-03-11T21:32:47.329313+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:32:47.329313+0000","last_undegraded":"2024-03-11T21:32:47.329313+0000","last_fullsized":"2024-03-11T21:32:47.329313+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:05.302619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:29:05.302619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":624,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326078+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:32:47.326078+0000","last_peered":"2024-03-11T21:32:47.326078+0000","last_clean":"2024-03-11T21:32:47.326078+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:32:47.326078+0000","last_undegraded":"2024-03-11T21:32:47.326078+0000","last_fullsized":"2024-03-11T21:32:47.326078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:49.570886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335669+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:32:47.335669+0000","last_peered":"2024-03-11T21:32:47.335669+0000","last_clean":"2024-03-11T21:32:47.335669+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:32:47.335669+0000","last_undegraded":"2024-03-11T21:32:47.335669+0000","last_fullsized":"2024-03-11T21:32:47.335669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:34.315898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":526,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310090+0000","last_change":"2024-03-11T21:32:23.731916+0000","last_active":"2024-03-11T21:32:47.310090+0000","last_peered":"2024-03-11T21:32:47.310090+0000","last_clean":"2024-03-11T21:32:47.310090+0000","last_became_active":"2024-03-11T21:32:09.326360+0000","last_became_peered":"2024-03-11T21:32:09.326360+0000","last_unstale":"2024-03-11T21:32:47.310090+0000","last_undegraded":"2024-03-11T21:32:47.310090+0000","last_fullsized":"2024-03-11T21:32:47.310090+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.731769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.731769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.684775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335798+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:32:47.335798+0000","last_peered":"2024-03-11T21:32:47.335798+0000","last_clean":"2024-03-11T21:32:47.335798+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:32:47.335798+0000","last_undegraded":"2024-03-11T21:32:47.335798+0000","last_fullsized":"2024-03-11T21:32:47.335798+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:38.667213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":753,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324910+0000","last_change":"2024-03-11T21:31:37.390682+0000","last_active":"2024-03-11T21:32:47.324910+0000","last_peered":"2024-03-11T21:32:47.324910+0000","last_clean":"2024-03-11T21:32:47.324910+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:32:47.324910+0000","last_undegraded":"2024-03-11T21:32:47.324910+0000","last_fullsized":"2024-03-11T21:32:47.324910+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:53.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:53.325007+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":634,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311235+0000","last_change":"2024-03-11T21:30:52.898564+0000","last_active":"2024-03-11T21:32:47.311235+0000","last_peered":"2024-03-11T21:32:47.311235+0000","last_clean":"2024-03-11T21:32:47.311235+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:32:47.311235+0000","last_undegraded":"2024-03-11T21:32:47.311235+0000","last_fullsized":"2024-03-11T21:32:47.311235+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.898357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":658,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326067+0000","last_change":"2024-03-11T21:31:58.977677+0000","last_active":"2024-03-11T21:32:48.326067+0000","last_peered":"2024-03-11T21:32:48.326067+0000","last_clean":"2024-03-11T21:32:48.326067+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:32:48.326067+0000","last_undegraded":"2024-03-11T21:32:48.326067+0000","last_fullsized":"2024-03-11T21:32:48.326067+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:31:58.977541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.977541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.700906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.344085+0000","last_change":"2024-03-11T21:30:31.534053+0000","last_active":"2024-03-11T21:32:47.344085+0000","last_peered":"2024-03-11T21:32:47.344085+0000","last_clean":"2024-03-11T21:32:47.344085+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:32:47.344085+0000","last_undegraded":"2024-03-11T21:32:47.344085+0000","last_fullsized":"2024-03-11T21:32:47.344085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:30:31.533807+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:30:31.533807+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":621,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334373+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:32:47.334373+0000","last_peered":"2024-03-11T21:32:47.334373+0000","last_clean":"2024-03-11T21:32:47.334373+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:32:47.334373+0000","last_undegraded":"2024-03-11T21:32:47.334373+0000","last_fullsized":"2024-03-11T21:32:47.334373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"142'2","reported_seq":519,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324792+0000","last_change":"2024-03-11T21:32:13.306779+0000","last_active":"2024-03-11T21:32:47.324792+0000","last_peered":"2024-03-11T21:32:47.324792+0000","last_clean":"2024-03-11T21:32:47.324792+0000","last_became_active":"2024-03-11T21:32:13.305739+0000","last_became_peered":"2024-03-11T21:32:13.305739+0000","last_unstale":"2024-03-11T21:32:47.324792+0000","last_undegraded":"2024-03-11T21:32:47.324792+0000","last_fullsized":"2024-03-11T21:32:47.324792+0000","mapping_epoch":139,"log_start":"142'2","ondisk_log_start":"142'2","created":139,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'2","last_scrub_stamp":"2024-03-11T21:30:35.816643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.816643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":477,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338516+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:32:47.338516+0000","last_peered":"2024-03-11T21:32:47.338516+0000","last_clean":"2024-03-11T21:32:47.338516+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:32:47.338516+0000","last_undegraded":"2024-03-11T21:32:47.338516+0000","last_fullsized":"2024-03-11T21:32:47.338516+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.151214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":625,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311022+0000","last_change":"2024-03-11T21:31:02.847904+0000","last_active":"2024-03-11T21:32:47.311022+0000","last_peered":"2024-03-11T21:32:47.311022+0000","last_clean":"2024-03-11T21:32:47.311022+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:32:47.311022+0000","last_undegraded":"2024-03-11T21:32:47.311022+0000","last_fullsized":"2024-03-11T21:32:47.311022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.847692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.847692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1192,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324606+0000","last_change":"2024-03-11T21:30:17.907380+0000","last_active":"2024-03-11T21:32:47.324606+0000","last_peered":"2024-03-11T21:32:47.324606+0000","last_clean":"2024-03-11T21:32:47.324606+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:32:47.324606+0000","last_undegraded":"2024-03-11T21:32:47.324606+0000","last_fullsized":"2024-03-11T21:32:47.324606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:30:17.907168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.907168+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":564,"num_read_kb":559,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335790+0000","last_change":"2024-03-11T21:30:47.779423+0000","last_active":"2024-03-11T21:32:47.335790+0000","last_peered":"2024-03-11T21:32:47.335790+0000","last_clean":"2024-03-11T21:32:47.335790+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:32:47.335790+0000","last_undegraded":"2024-03-11T21:32:47.335790+0000","last_fullsized":"2024-03-11T21:32:47.335790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:30:47.779298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.779298+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":527,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.313995+0000","last_change":"2024-03-11T21:31:37.408692+0000","last_active":"2024-03-11T21:32:45.313995+0000","last_peered":"2024-03-11T21:32:45.313995+0000","last_clean":"2024-03-11T21:32:45.313995+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:32:45.313995+0000","last_undegraded":"2024-03-11T21:32:45.313995+0000","last_fullsized":"2024-03-11T21:32:45.313995+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:29:16.233739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.233739+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":621,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312163+0000","last_change":"2024-03-11T21:30:58.912377+0000","last_active":"2024-03-11T21:32:47.312163+0000","last_peered":"2024-03-11T21:32:47.312163+0000","last_clean":"2024-03-11T21:32:47.312163+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:32:47.312163+0000","last_undegraded":"2024-03-11T21:32:47.312163+0000","last_fullsized":"2024-03-11T21:32:47.312163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.912133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"259'16","reported_seq":319,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325340+0000","last_change":"2024-03-11T21:32:01.314007+0000","last_active":"2024-03-11T21:32:48.325340+0000","last_peered":"2024-03-11T21:32:48.325340+0000","last_clean":"2024-03-11T21:32:48.325340+0000","last_became_active":"2024-03-11T21:32:01.309399+0000","last_became_peered":"2024-03-11T21:32:01.309399+0000","last_unstale":"2024-03-11T21:32:48.325340+0000","last_undegraded":"2024-03-11T21:32:48.325340+0000","last_fullsized":"2024-03-11T21:32:48.325340+0000","mapping_epoch":241,"log_start":"259'16","ondisk_log_start":"259'16","created":139,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'2","last_scrub_stamp":"2024-03-11T21:31:01.849450+0000","last_deep_scrub":"142'2","last_deep_scrub_stamp":"2024-03-11T21:31:01.849450+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.849450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":477,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.340773+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:32:47.340773+0000","last_peered":"2024-03-11T21:32:47.340773+0000","last_clean":"2024-03-11T21:32:47.340773+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:32:47.340773+0000","last_undegraded":"2024-03-11T21:32:47.340773+0000","last_fullsized":"2024-03-11T21:32:47.340773+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:55.539607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":616,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327844+0000","last_change":"2024-03-11T21:30:25.450518+0000","last_active":"2024-03-11T21:32:47.327844+0000","last_peered":"2024-03-11T21:32:47.327844+0000","last_clean":"2024-03-11T21:32:47.327844+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:32:47.327844+0000","last_undegraded":"2024-03-11T21:32:47.327844+0000","last_fullsized":"2024-03-11T21:32:47.327844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.450301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.450301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"297'32","reported_seq":660,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.316117+0000","last_change":"2024-03-11T21:32:05.077652+0000","last_active":"2024-03-11T21:32:45.316117+0000","last_peered":"2024-03-11T21:32:45.316117+0000","last_clean":"2024-03-11T21:32:45.316117+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:32:45.316117+0000","last_undegraded":"2024-03-11T21:32:45.316117+0000","last_fullsized":"2024-03-11T21:32:45.316117+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'32","last_scrub_stamp":"2024-03-11T21:32:05.077406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.077406+0000","objects_scrubbed":10,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.334401+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":631,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338719+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:32:47.338719+0000","last_peered":"2024-03-11T21:32:47.338719+0000","last_clean":"2024-03-11T21:32:47.338719+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:32:47.338719+0000","last_undegraded":"2024-03-11T21:32:47.338719+0000","last_fullsized":"2024-03-11T21:32:47.338719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.436858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":474,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.310784+0000","last_change":"2024-03-11T21:32:07.975839+0000","last_active":"2024-03-11T21:32:45.310784+0000","last_peered":"2024-03-11T21:32:45.310784+0000","last_clean":"2024-03-11T21:32:45.310784+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:32:45.310784+0000","last_undegraded":"2024-03-11T21:32:45.310784+0000","last_fullsized":"2024-03-11T21:32:45.310784+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.975732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.975732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:13.100368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":214,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315498+0000","last_change":"2024-03-11T21:31:49.459162+0000","last_active":"2024-03-11T21:32:45.315498+0000","last_peered":"2024-03-11T21:32:45.315498+0000","last_clean":"2024-03-11T21:32:45.315498+0000","last_became_active":"2024-03-11T21:31:49.458836+0000","last_became_peered":"2024-03-11T21:31:49.458836+0000","last_unstale":"2024-03-11T21:32:45.315498+0000","last_undegraded":"2024-03-11T21:32:45.315498+0000","last_fullsized":"2024-03-11T21:32:45.315498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:45.810580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:45.810580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":472,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326105+0000","last_change":"2024-03-11T21:30:48.777877+0000","last_active":"2024-03-11T21:32:48.326105+0000","last_peered":"2024-03-11T21:32:48.326105+0000","last_clean":"2024-03-11T21:32:48.326105+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:32:48.326105+0000","last_undegraded":"2024-03-11T21:32:48.326105+0000","last_fullsized":"2024-03-11T21:32:48.326105+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.777772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.777772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":633,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.325963+0000","last_change":"2024-03-11T21:32:07.813703+0000","last_active":"2024-03-11T21:32:47.325963+0000","last_peered":"2024-03-11T21:32:47.325963+0000","last_clean":"2024-03-11T21:32:47.325963+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:32:47.325963+0000","last_undegraded":"2024-03-11T21:32:47.325963+0000","last_fullsized":"2024-03-11T21:32:47.325963+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.813604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.813604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.697988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329078+0000","last_change":"2024-03-11T21:31:37.403229+0000","last_active":"2024-03-11T21:32:47.329078+0000","last_peered":"2024-03-11T21:32:47.329078+0000","last_clean":"2024-03-11T21:32:47.329078+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:32:47.329078+0000","last_undegraded":"2024-03-11T21:32:47.329078+0000","last_fullsized":"2024-03-11T21:32:47.329078+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.364816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.364816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.420745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":644,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324746+0000","last_change":"2024-03-11T21:30:30.000308+0000","last_active":"2024-03-11T21:32:48.324746+0000","last_peered":"2024-03-11T21:32:48.324746+0000","last_clean":"2024-03-11T21:32:48.324746+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:32:48.324746+0000","last_undegraded":"2024-03-11T21:32:48.324746+0000","last_fullsized":"2024-03-11T21:32:48.324746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:30:30.000054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.000054+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.985812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":484,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315563+0000","last_change":"2024-03-11T21:32:17.944438+0000","last_active":"2024-03-11T21:32:45.315563+0000","last_peered":"2024-03-11T21:32:45.315563+0000","last_clean":"2024-03-11T21:32:45.315563+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:32:45.315563+0000","last_undegraded":"2024-03-11T21:32:45.315563+0000","last_fullsized":"2024-03-11T21:32:45.315563+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.944141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.944141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.417735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"259'9","reported_seq":331,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.337114+0000","last_change":"2024-03-11T21:31:53.275348+0000","last_active":"2024-03-11T21:32:47.337114+0000","last_peered":"2024-03-11T21:32:47.337114+0000","last_clean":"2024-03-11T21:32:47.337114+0000","last_became_active":"2024-03-11T21:31:53.275084+0000","last_became_peered":"2024-03-11T21:31:53.275084+0000","last_unstale":"2024-03-11T21:32:47.337114+0000","last_undegraded":"2024-03-11T21:32:47.337114+0000","last_fullsized":"2024-03-11T21:32:47.337114+0000","mapping_epoch":295,"log_start":"259'9","ondisk_log_start":"259'9","created":139,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:46.644291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.644291+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324027+0000","last_change":"2024-03-11T21:32:16.760994+0000","last_active":"2024-03-11T21:32:47.324027+0000","last_peered":"2024-03-11T21:32:47.324027+0000","last_clean":"2024-03-11T21:32:47.324027+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:32:47.324027+0000","last_undegraded":"2024-03-11T21:32:47.324027+0000","last_fullsized":"2024-03-11T21:32:47.324027+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.760914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.760914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.177261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":615,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.336512+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:32:47.336512+0000","last_peered":"2024-03-11T21:32:47.336512+0000","last_clean":"2024-03-11T21:32:47.336512+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:32:47.336512+0000","last_undegraded":"2024-03-11T21:32:47.336512+0000","last_fullsized":"2024-03-11T21:32:47.336512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":887,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329210+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:32:47.329210+0000","last_peered":"2024-03-11T21:32:47.329210+0000","last_clean":"2024-03-11T21:32:47.329210+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:32:47.329210+0000","last_undegraded":"2024-03-11T21:32:47.329210+0000","last_fullsized":"2024-03-11T21:32:47.329210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.320760+0000","last_change":"2024-03-11T21:30:35.064347+0000","last_active":"2024-03-11T21:32:47.320760+0000","last_peered":"2024-03-11T21:32:47.320760+0000","last_clean":"2024-03-11T21:32:47.320760+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:32:47.320760+0000","last_undegraded":"2024-03-11T21:32:47.320760+0000","last_fullsized":"2024-03-11T21:32:47.320760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:35.064136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.064136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.305941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":635,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328089+0000","last_change":"2024-03-11T21:30:32.862005+0000","last_active":"2024-03-11T21:32:47.328089+0000","last_peered":"2024-03-11T21:32:47.328089+0000","last_clean":"2024-03-11T21:32:47.328089+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:32:47.328089+0000","last_undegraded":"2024-03-11T21:32:47.328089+0000","last_fullsized":"2024-03-11T21:32:47.328089+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:32.861910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.861910+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":620,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328912+0000","last_change":"2024-03-11T21:32:26.159690+0000","last_active":"2024-03-11T21:32:47.328912+0000","last_peered":"2024-03-11T21:32:47.328912+0000","last_clean":"2024-03-11T21:32:47.328912+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:32:47.328912+0000","last_undegraded":"2024-03-11T21:32:47.328912+0000","last_fullsized":"2024-03-11T21:32:47.328912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:26.159472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:26.159472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.482009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":478,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329613+0000","last_change":"2024-03-11T21:31:57.949982+0000","last_active":"2024-03-11T21:32:47.329613+0000","last_peered":"2024-03-11T21:32:47.329613+0000","last_clean":"2024-03-11T21:32:47.329613+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:32:47.329613+0000","last_undegraded":"2024-03-11T21:32:47.329613+0000","last_fullsized":"2024-03-11T21:32:47.329613+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.949771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.949771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:00.387949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327385+0000","last_change":"2024-03-11T21:32:11.769286+0000","last_active":"2024-03-11T21:32:47.327385+0000","last_peered":"2024-03-11T21:32:47.327385+0000","last_clean":"2024-03-11T21:32:47.327385+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:32:47.327385+0000","last_undegraded":"2024-03-11T21:32:47.327385+0000","last_fullsized":"2024-03-11T21:32:47.327385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.769136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.769136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.591393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310132+0000","last_change":"2024-03-11T21:30:59.840809+0000","last_active":"2024-03-11T21:32:47.310132+0000","last_peered":"2024-03-11T21:32:47.310132+0000","last_clean":"2024-03-11T21:32:47.310132+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:32:47.310132+0000","last_undegraded":"2024-03-11T21:32:47.310132+0000","last_fullsized":"2024-03-11T21:32:47.310132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.840718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.840718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"149'26","reported_seq":548,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.308397+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:32:45.308397+0000","last_peered":"2024-03-11T21:32:45.308397+0000","last_clean":"2024-03-11T21:32:45.308397+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:32:45.308397+0000","last_undegraded":"2024-03-11T21:32:45.308397+0000","last_fullsized":"2024-03-11T21:32:45.308397+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.837252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":611,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338700+0000","last_change":"2024-03-11T21:29:10.347962+0000","last_active":"2024-03-11T21:32:47.338700+0000","last_peered":"2024-03-11T21:32:47.338700+0000","last_clean":"2024-03-11T21:32:47.338700+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:32:47.338700+0000","last_undegraded":"2024-03-11T21:32:47.338700+0000","last_fullsized":"2024-03-11T21:32:47.338700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:10.347717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:29:10.347717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":320,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.333068+0000","last_change":"2024-03-11T21:32:05.612359+0000","last_active":"2024-03-11T21:32:47.333068+0000","last_peered":"2024-03-11T21:32:47.333068+0000","last_clean":"2024-03-11T21:32:47.333068+0000","last_became_active":"2024-03-11T21:31:57.299477+0000","last_became_peered":"2024-03-11T21:31:57.299477+0000","last_unstale":"2024-03-11T21:32:47.333068+0000","last_undegraded":"2024-03-11T21:32:47.333068+0000","last_fullsized":"2024-03-11T21:32:47.333068+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.612189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.612189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.877255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":472,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324674+0000","last_change":"2024-03-11T21:30:53.861799+0000","last_active":"2024-03-11T21:32:48.324674+0000","last_peered":"2024-03-11T21:32:48.324674+0000","last_clean":"2024-03-11T21:32:48.324674+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:32:48.324674+0000","last_undegraded":"2024-03-11T21:32:48.324674+0000","last_fullsized":"2024-03-11T21:32:48.324674+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:53.861701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:53.861701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":628,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326191+0000","last_change":"2024-03-11T21:31:37.429746+0000","last_active":"2024-03-11T21:32:48.326191+0000","last_peered":"2024-03-11T21:32:48.326191+0000","last_clean":"2024-03-11T21:32:48.326191+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:32:48.326191+0000","last_undegraded":"2024-03-11T21:32:48.326191+0000","last_fullsized":"2024-03-11T21:32:48.326191+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:45.832418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:45.832418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.986446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":673,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312962+0000","last_change":"2024-03-11T21:30:48.891794+0000","last_active":"2024-03-11T21:32:47.312962+0000","last_peered":"2024-03-11T21:32:47.312962+0000","last_clean":"2024-03-11T21:32:47.312962+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:32:47.312962+0000","last_undegraded":"2024-03-11T21:32:47.312962+0000","last_fullsized":"2024-03-11T21:32:47.312962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:30:48.891648+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.891648+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":633,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.330044+0000","last_change":"2024-03-11T21:31:37.388733+0000","last_active":"2024-03-11T21:32:47.330044+0000","last_peered":"2024-03-11T21:32:47.330044+0000","last_clean":"2024-03-11T21:32:47.330044+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:32:47.330044+0000","last_undegraded":"2024-03-11T21:32:47.330044+0000","last_fullsized":"2024-03-11T21:32:47.330044+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.991582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.991582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.305677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335089+0000","last_change":"2024-03-11T21:30:33.825558+0000","last_active":"2024-03-11T21:32:47.335089+0000","last_peered":"2024-03-11T21:32:47.335089+0000","last_clean":"2024-03-11T21:32:47.335089+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:32:47.335089+0000","last_undegraded":"2024-03-11T21:32:47.335089+0000","last_fullsized":"2024-03-11T21:32:47.335089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:33.825475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:33.825475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"259'14","reported_seq":532,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311585+0000","last_change":"2024-03-11T21:32:05.389256+0000","last_active":"2024-03-11T21:32:47.311585+0000","last_peered":"2024-03-11T21:32:47.311585+0000","last_clean":"2024-03-11T21:32:47.311585+0000","last_became_active":"2024-03-11T21:32:05.388874+0000","last_became_peered":"2024-03-11T21:32:05.388874+0000","last_unstale":"2024-03-11T21:32:47.311585+0000","last_undegraded":"2024-03-11T21:32:47.311585+0000","last_fullsized":"2024-03-11T21:32:47.311585+0000","mapping_epoch":139,"log_start":"259'14","ondisk_log_start":"259'14","created":139,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'10","last_scrub_stamp":"2024-03-11T21:30:42.647195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.647195+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":22,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":478,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324642+0000","last_change":"2024-03-11T21:31:55.943647+0000","last_active":"2024-03-11T21:32:47.324642+0000","last_peered":"2024-03-11T21:32:47.324642+0000","last_clean":"2024-03-11T21:32:47.324642+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:32:47.324642+0000","last_undegraded":"2024-03-11T21:32:47.324642+0000","last_fullsized":"2024-03-11T21:32:47.324642+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.943487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.943487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.360467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":759,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326069+0000","last_change":"2024-03-11T21:32:15.949343+0000","last_active":"2024-03-11T21:32:48.326069+0000","last_peered":"2024-03-11T21:32:48.326069+0000","last_clean":"2024-03-11T21:32:48.326069+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:32:48.326069+0000","last_undegraded":"2024-03-11T21:32:48.326069+0000","last_fullsized":"2024-03-11T21:32:48.326069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:32:15.949107+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.949107+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.139215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":511,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.310380+0000","last_change":"2024-03-11T21:31:37.399181+0000","last_active":"2024-03-11T21:32:45.310380+0000","last_peered":"2024-03-11T21:32:45.310380+0000","last_clean":"2024-03-11T21:32:45.310380+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:32:45.310380+0000","last_undegraded":"2024-03-11T21:32:45.310380+0000","last_fullsized":"2024-03-11T21:32:45.310380+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:30:35.849630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.849630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":767,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324636+0000","last_change":"2024-03-11T21:30:54.944552+0000","last_active":"2024-03-11T21:32:47.324636+0000","last_peered":"2024-03-11T21:32:47.324636+0000","last_clean":"2024-03-11T21:32:47.324636+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:32:47.324636+0000","last_undegraded":"2024-03-11T21:32:47.324636+0000","last_fullsized":"2024-03-11T21:32:47.324636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:30:54.944305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.944305+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":482,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315370+0000","last_change":"2024-03-11T21:31:37.399025+0000","last_active":"2024-03-11T21:32:45.315370+0000","last_peered":"2024-03-11T21:32:45.315370+0000","last_clean":"2024-03-11T21:32:45.315370+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:32:45.315370+0000","last_undegraded":"2024-03-11T21:32:45.315370+0000","last_fullsized":"2024-03-11T21:32:45.315370+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.716495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.716495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":623,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324791+0000","last_change":"2024-03-11T21:32:16.907647+0000","last_active":"2024-03-11T21:32:48.324791+0000","last_peered":"2024-03-11T21:32:48.324791+0000","last_clean":"2024-03-11T21:32:48.324791+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:32:48.324791+0000","last_undegraded":"2024-03-11T21:32:48.324791+0000","last_fullsized":"2024-03-11T21:32:48.324791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.907562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.907562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:37.130510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"259'9","reported_seq":523,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.331116+0000","last_change":"2024-03-11T21:32:39.363387+0000","last_active":"2024-03-11T21:32:47.331116+0000","last_peered":"2024-03-11T21:32:47.331116+0000","last_clean":"2024-03-11T21:32:47.331116+0000","last_became_active":"2024-03-11T21:32:39.359239+0000","last_became_peered":"2024-03-11T21:32:39.359239+0000","last_unstale":"2024-03-11T21:32:47.331116+0000","last_undegraded":"2024-03-11T21:32:47.331116+0000","last_fullsized":"2024-03-11T21:32:47.331116+0000","mapping_epoch":139,"log_start":"259'9","ondisk_log_start":"259'9","created":139,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:48.625569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.625569+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":335,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315473+0000","last_change":"2024-03-11T21:31:58.092962+0000","last_active":"2024-03-11T21:32:45.315473+0000","last_peered":"2024-03-11T21:32:45.315473+0000","last_clean":"2024-03-11T21:32:45.315473+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:32:45.315473+0000","last_undegraded":"2024-03-11T21:32:45.315473+0000","last_fullsized":"2024-03-11T21:32:45.315473+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.092744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.092744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.013717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310546+0000","last_change":"2024-03-11T21:32:12.801345+0000","last_active":"2024-03-11T21:32:47.310546+0000","last_peered":"2024-03-11T21:32:47.310546+0000","last_clean":"2024-03-11T21:32:47.310546+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:32:47.310546+0000","last_undegraded":"2024-03-11T21:32:47.310546+0000","last_fullsized":"2024-03-11T21:32:47.310546+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.497490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":652,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.341002+0000","last_change":"2024-03-11T21:31:37.405413+0000","last_active":"2024-03-11T21:32:47.341002+0000","last_peered":"2024-03-11T21:32:47.341002+0000","last_clean":"2024-03-11T21:32:47.341002+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:32:47.341002+0000","last_undegraded":"2024-03-11T21:32:47.341002+0000","last_fullsized":"2024-03-11T21:32:47.341002+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.327656+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":651,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.316477+0000","last_change":"2024-03-11T21:31:05.819709+0000","last_active":"2024-03-11T21:32:47.316477+0000","last_peered":"2024-03-11T21:32:47.316477+0000","last_clean":"2024-03-11T21:32:47.316477+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:32:47.316477+0000","last_undegraded":"2024-03-11T21:32:47.316477+0000","last_fullsized":"2024-03-11T21:32:47.316477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:31:05.819557+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.819557+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":621,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328609+0000","last_change":"2024-03-11T21:31:37.406030+0000","last_active":"2024-03-11T21:32:47.328609+0000","last_peered":"2024-03-11T21:32:47.328609+0000","last_clean":"2024-03-11T21:32:47.328609+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:32:47.328609+0000","last_undegraded":"2024-03-11T21:32:47.328609+0000","last_fullsized":"2024-03-11T21:32:47.328609+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:54.362432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.362432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328833+0000","last_change":"2024-03-11T21:30:40.750808+0000","last_active":"2024-03-11T21:32:47.328833+0000","last_peered":"2024-03-11T21:32:47.328833+0000","last_clean":"2024-03-11T21:32:47.328833+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:32:47.328833+0000","last_undegraded":"2024-03-11T21:32:47.328833+0000","last_fullsized":"2024-03-11T21:32:47.328833+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.750735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.750735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":638,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.330211+0000","last_change":"2024-03-11T21:29:19.356816+0000","last_active":"2024-03-11T21:32:47.330211+0000","last_peered":"2024-03-11T21:32:47.330211+0000","last_clean":"2024-03-11T21:32:47.330211+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:32:47.330211+0000","last_undegraded":"2024-03-11T21:32:47.330211+0000","last_fullsized":"2024-03-11T21:32:47.330211+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:29:19.356702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.356702+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.420980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.310416+0000","last_change":"2024-03-11T21:31:37.405137+0000","last_active":"2024-03-11T21:32:47.310416+0000","last_peered":"2024-03-11T21:32:47.310416+0000","last_clean":"2024-03-11T21:32:47.310416+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:32:47.310416+0000","last_undegraded":"2024-03-11T21:32:47.310416+0000","last_fullsized":"2024-03-11T21:32:47.310416+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.738299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.738299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.122778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":625,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.336143+0000","last_change":"2024-03-11T21:31:37.406910+0000","last_active":"2024-03-11T21:32:47.336143+0000","last_peered":"2024-03-11T21:32:47.336143+0000","last_clean":"2024-03-11T21:32:47.336143+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:32:47.336143+0000","last_undegraded":"2024-03-11T21:32:47.336143+0000","last_fullsized":"2024-03-11T21:32:47.336143+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.881361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.881361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":628,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.312206+0000","last_change":"2024-03-11T21:32:04.847327+0000","last_active":"2024-03-11T21:32:47.312206+0000","last_peered":"2024-03-11T21:32:47.312206+0000","last_clean":"2024-03-11T21:32:47.312206+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:32:47.312206+0000","last_undegraded":"2024-03-11T21:32:47.312206+0000","last_fullsized":"2024-03-11T21:32:47.312206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.847094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.847094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.764379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328435+0000","last_change":"2024-03-11T21:30:48.791220+0000","last_active":"2024-03-11T21:32:47.328435+0000","last_peered":"2024-03-11T21:32:47.328435+0000","last_clean":"2024-03-11T21:32:47.328435+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:32:47.328435+0000","last_undegraded":"2024-03-11T21:32:47.328435+0000","last_fullsized":"2024-03-11T21:32:47.328435+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.791036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326326+0000","last_change":"2024-03-11T21:32:15.567675+0000","last_active":"2024-03-11T21:32:47.326326+0000","last_peered":"2024-03-11T21:32:47.326326+0000","last_clean":"2024-03-11T21:32:47.326326+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:32:47.326326+0000","last_undegraded":"2024-03-11T21:32:47.326326+0000","last_fullsized":"2024-03-11T21:32:47.326326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:32:15.567443+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.567443+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.706289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.331473+0000","last_change":"2024-03-11T21:31:59.593413+0000","last_active":"2024-03-11T21:32:47.331473+0000","last_peered":"2024-03-11T21:32:47.331473+0000","last_clean":"2024-03-11T21:32:47.331473+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:32:47.331473+0000","last_undegraded":"2024-03-11T21:32:47.331473+0000","last_fullsized":"2024-03-11T21:32:47.331473+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.593037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.593037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.709416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":633,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326758+0000","last_change":"2024-03-11T21:32:06.800643+0000","last_active":"2024-03-11T21:32:47.326758+0000","last_peered":"2024-03-11T21:32:47.326758+0000","last_clean":"2024-03-11T21:32:47.326758+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:32:47.326758+0000","last_undegraded":"2024-03-11T21:32:47.326758+0000","last_fullsized":"2024-03-11T21:32:47.326758+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.800436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.800436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.924396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":610,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338615+0000","last_change":"2024-03-11T21:28:45.377770+0000","last_active":"2024-03-11T21:32:47.338615+0000","last_peered":"2024-03-11T21:32:47.338615+0000","last_clean":"2024-03-11T21:32:47.338615+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:32:47.338615+0000","last_undegraded":"2024-03-11T21:32:47.338615+0000","last_fullsized":"2024-03-11T21:32:47.338615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:45.377668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:45.377668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.420680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326854+0000","last_change":"2024-03-11T21:31:57.633633+0000","last_active":"2024-03-11T21:32:47.326854+0000","last_peered":"2024-03-11T21:32:47.326854+0000","last_clean":"2024-03-11T21:32:47.326854+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:32:47.326854+0000","last_undegraded":"2024-03-11T21:32:47.326854+0000","last_fullsized":"2024-03-11T21:32:47.326854+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.633438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.633438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.657983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334947+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:32:47.334947+0000","last_peered":"2024-03-11T21:32:47.334947+0000","last_clean":"2024-03-11T21:32:47.334947+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:32:47.334947+0000","last_undegraded":"2024-03-11T21:32:47.334947+0000","last_fullsized":"2024-03-11T21:32:47.334947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329966+0000","last_change":"2024-03-11T21:30:46.912846+0000","last_active":"2024-03-11T21:32:47.329966+0000","last_peered":"2024-03-11T21:32:47.329966+0000","last_clean":"2024-03-11T21:32:47.329966+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:32:47.329966+0000","last_undegraded":"2024-03-11T21:32:47.329966+0000","last_fullsized":"2024-03-11T21:32:47.329966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:46.912728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.912728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.305769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":761,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324977+0000","last_change":"2024-03-11T21:32:15.855148+0000","last_active":"2024-03-11T21:32:47.324977+0000","last_peered":"2024-03-11T21:32:47.324977+0000","last_clean":"2024-03-11T21:32:47.324977+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:32:47.324977+0000","last_undegraded":"2024-03-11T21:32:47.324977+0000","last_fullsized":"2024-03-11T21:32:47.324977+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:32:15.854918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.854918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.754470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":622,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326545+0000","last_change":"2024-03-11T21:30:49.769958+0000","last_active":"2024-03-11T21:32:48.326545+0000","last_peered":"2024-03-11T21:32:48.326545+0000","last_clean":"2024-03-11T21:32:48.326545+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:32:48.326545+0000","last_undegraded":"2024-03-11T21:32:48.326545+0000","last_fullsized":"2024-03-11T21:32:48.326545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:30:49.769830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326623+0000","last_change":"2024-03-11T21:30:51.554762+0000","last_active":"2024-03-11T21:32:47.326623+0000","last_peered":"2024-03-11T21:32:47.326623+0000","last_clean":"2024-03-11T21:32:47.326623+0000","last_became_active":"2024-03-11T21:28:51.357900+0000","last_became_peered":"2024-03-11T21:28:51.357900+0000","last_unstale":"2024-03-11T21:32:47.326623+0000","last_undegraded":"2024-03-11T21:32:47.326623+0000","last_fullsized":"2024-03-11T21:32:47.326623+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.554539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.554539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":651,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325492+0000","last_change":"2024-03-11T21:30:52.813167+0000","last_active":"2024-03-11T21:32:48.325492+0000","last_peered":"2024-03-11T21:32:48.325492+0000","last_clean":"2024-03-11T21:32:48.325492+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:32:48.325492+0000","last_undegraded":"2024-03-11T21:32:48.325492+0000","last_fullsized":"2024-03-11T21:32:48.325492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:30:52.812936+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.812936+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":635,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325803+0000","last_change":"2024-03-11T21:30:58.882981+0000","last_active":"2024-03-11T21:32:48.325803+0000","last_peered":"2024-03-11T21:32:48.325803+0000","last_clean":"2024-03-11T21:32:48.325803+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:32:48.325803+0000","last_undegraded":"2024-03-11T21:32:48.325803+0000","last_fullsized":"2024-03-11T21:32:48.325803+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.882820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.882820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.987358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":744,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335639+0000","last_change":"2024-03-11T21:29:14.510232+0000","last_active":"2024-03-11T21:32:47.335639+0000","last_peered":"2024-03-11T21:32:47.335639+0000","last_clean":"2024-03-11T21:32:47.335639+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:32:47.335639+0000","last_undegraded":"2024-03-11T21:32:47.335639+0000","last_fullsized":"2024-03-11T21:32:47.335639+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:29:14.510132+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.510132+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":620,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.317722+0000","last_change":"2024-03-11T21:30:31.977271+0000","last_active":"2024-03-11T21:32:47.317722+0000","last_peered":"2024-03-11T21:32:47.317722+0000","last_clean":"2024-03-11T21:32:47.317722+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:32:47.317722+0000","last_undegraded":"2024-03-11T21:32:47.317722+0000","last_fullsized":"2024-03-11T21:32:47.317722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:31.977047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:31.977047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673167+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":331,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.319187+0000","last_change":"2024-03-11T21:31:37.401438+0000","last_active":"2024-03-11T21:32:45.319187+0000","last_peered":"2024-03-11T21:32:45.319187+0000","last_clean":"2024-03-11T21:32:45.319187+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:32:45.319187+0000","last_undegraded":"2024-03-11T21:32:45.319187+0000","last_fullsized":"2024-03-11T21:32:45.319187+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.696902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.696902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.339008+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:32:47.339008+0000","last_peered":"2024-03-11T21:32:47.339008+0000","last_clean":"2024-03-11T21:32:47.339008+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:32:47.339008+0000","last_undegraded":"2024-03-11T21:32:47.339008+0000","last_fullsized":"2024-03-11T21:32:47.339008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.634197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":640,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.333940+0000","last_change":"2024-03-11T21:32:09.571388+0000","last_active":"2024-03-11T21:32:47.333940+0000","last_peered":"2024-03-11T21:32:47.333940+0000","last_clean":"2024-03-11T21:32:47.333940+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:32:47.333940+0000","last_undegraded":"2024-03-11T21:32:47.333940+0000","last_fullsized":"2024-03-11T21:32:47.333940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.571257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.571257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.587765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":625,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328507+0000","last_change":"2024-03-11T21:31:37.402794+0000","last_active":"2024-03-11T21:32:47.328507+0000","last_peered":"2024-03-11T21:32:47.328507+0000","last_clean":"2024-03-11T21:32:47.328507+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:32:47.328507+0000","last_undegraded":"2024-03-11T21:32:47.328507+0000","last_fullsized":"2024-03-11T21:32:47.328507+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:28.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:28.810562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":632,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324707+0000","last_change":"2024-03-11T21:32:06.989987+0000","last_active":"2024-03-11T21:32:48.324707+0000","last_peered":"2024-03-11T21:32:48.324707+0000","last_clean":"2024-03-11T21:32:48.324707+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:32:48.324707+0000","last_undegraded":"2024-03-11T21:32:48.324707+0000","last_fullsized":"2024-03-11T21:32:48.324707+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.989724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.989724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.368810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":473,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326159+0000","last_change":"2024-03-11T21:30:36.943381+0000","last_active":"2024-03-11T21:32:48.326159+0000","last_peered":"2024-03-11T21:32:48.326159+0000","last_clean":"2024-03-11T21:32:48.326159+0000","last_became_active":"2024-03-11T21:28:51.388727+0000","last_became_peered":"2024-03-11T21:28:51.388727+0000","last_unstale":"2024-03-11T21:32:48.326159+0000","last_undegraded":"2024-03-11T21:32:48.326159+0000","last_fullsized":"2024-03-11T21:32:48.326159+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.943172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.986224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.316275+0000","last_change":"2024-03-11T21:30:38.863143+0000","last_active":"2024-03-11T21:32:47.316275+0000","last_peered":"2024-03-11T21:32:47.316275+0000","last_clean":"2024-03-11T21:32:47.316275+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:32:47.316275+0000","last_undegraded":"2024-03-11T21:32:47.316275+0000","last_fullsized":"2024-03-11T21:32:47.316275+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:38.862999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.862999+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":526,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.316494+0000","last_change":"2024-03-11T21:32:11.929832+0000","last_active":"2024-03-11T21:32:45.316494+0000","last_peered":"2024-03-11T21:32:45.316494+0000","last_clean":"2024-03-11T21:32:45.316494+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:32:45.316494+0000","last_undegraded":"2024-03-11T21:32:45.316494+0000","last_fullsized":"2024-03-11T21:32:45.316494+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.929628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.929628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.408544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":486,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.310250+0000","last_change":"2024-03-11T21:31:37.400845+0000","last_active":"2024-03-11T21:32:45.310250+0000","last_peered":"2024-03-11T21:32:45.310250+0000","last_clean":"2024-03-11T21:32:45.310250+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:32:45.310250+0000","last_undegraded":"2024-03-11T21:32:45.310250+0000","last_fullsized":"2024-03-11T21:32:45.310250+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:59.189248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.189248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326489+0000","last_change":"2024-03-11T21:30:27.784291+0000","last_active":"2024-03-11T21:32:47.326489+0000","last_peered":"2024-03-11T21:32:47.326489+0000","last_clean":"2024-03-11T21:32:47.326489+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:32:47.326489+0000","last_undegraded":"2024-03-11T21:32:47.326489+0000","last_fullsized":"2024-03-11T21:32:47.326489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:27.783939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.783939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":472,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325774+0000","last_change":"2024-03-11T21:32:18.866826+0000","last_active":"2024-03-11T21:32:48.325774+0000","last_peered":"2024-03-11T21:32:48.325774+0000","last_clean":"2024-03-11T21:32:48.325774+0000","last_became_active":"2024-03-11T21:28:51.388792+0000","last_became_peered":"2024-03-11T21:28:51.388792+0000","last_unstale":"2024-03-11T21:32:48.325774+0000","last_undegraded":"2024-03-11T21:32:48.325774+0000","last_fullsized":"2024-03-11T21:32:48.325774+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.866701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.866701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:37.142390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":640,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326301+0000","last_change":"2024-03-11T21:30:27.030104+0000","last_active":"2024-03-11T21:32:48.326301+0000","last_peered":"2024-03-11T21:32:48.326301+0000","last_clean":"2024-03-11T21:32:48.326301+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:32:48.326301+0000","last_undegraded":"2024-03-11T21:32:48.326301+0000","last_fullsized":"2024-03-11T21:32:48.326301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:30:27.029873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.029873+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.986017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329313+0000","last_change":"2024-03-11T21:31:00.965260+0000","last_active":"2024-03-11T21:32:47.329313+0000","last_peered":"2024-03-11T21:32:47.329313+0000","last_clean":"2024-03-11T21:32:47.329313+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:32:47.329313+0000","last_undegraded":"2024-03-11T21:32:47.329313+0000","last_fullsized":"2024-03-11T21:32:47.329313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.964917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.964917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306738+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.331377+0000","last_change":"2024-03-11T21:32:12.550833+0000","last_active":"2024-03-11T21:32:47.331377+0000","last_peered":"2024-03-11T21:32:47.331377+0000","last_clean":"2024-03-11T21:32:47.331377+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:32:47.331377+0000","last_undegraded":"2024-03-11T21:32:47.331377+0000","last_fullsized":"2024-03-11T21:32:47.331377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.550625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.550625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.339924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":611,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.332982+0000","last_change":"2024-03-11T21:29:11.346981+0000","last_active":"2024-03-11T21:32:47.332982+0000","last_peered":"2024-03-11T21:32:47.332982+0000","last_clean":"2024-03-11T21:32:47.332982+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:32:47.332982+0000","last_undegraded":"2024-03-11T21:32:47.332982+0000","last_fullsized":"2024-03-11T21:32:47.332982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.346840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.346840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":481,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325883+0000","last_change":"2024-03-11T21:32:12.998779+0000","last_active":"2024-03-11T21:32:48.325883+0000","last_peered":"2024-03-11T21:32:48.325883+0000","last_clean":"2024-03-11T21:32:48.325883+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:32:48.325883+0000","last_undegraded":"2024-03-11T21:32:48.325883+0000","last_fullsized":"2024-03-11T21:32:48.325883+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.998644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.998644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:14.669863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326387+0000","last_change":"2024-03-11T21:31:59.996470+0000","last_active":"2024-03-11T21:32:47.326387+0000","last_peered":"2024-03-11T21:32:47.326387+0000","last_clean":"2024-03-11T21:32:47.326387+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:32:47.326387+0000","last_undegraded":"2024-03-11T21:32:47.326387+0000","last_fullsized":"2024-03-11T21:32:47.326387+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:59.996246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.996246+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.079844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":640,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.343393+0000","last_change":"2024-03-11T21:32:28.777744+0000","last_active":"2024-03-11T21:32:47.343393+0000","last_peered":"2024-03-11T21:32:47.343393+0000","last_clean":"2024-03-11T21:32:47.343393+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:32:47.343393+0000","last_undegraded":"2024-03-11T21:32:47.343393+0000","last_fullsized":"2024-03-11T21:32:47.343393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.777629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.777629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.837185+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":615,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329479+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:32:47.329479+0000","last_peered":"2024-03-11T21:32:47.329479+0000","last_clean":"2024-03-11T21:32:47.329479+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:32:47.329479+0000","last_undegraded":"2024-03-11T21:32:47.329479+0000","last_fullsized":"2024-03-11T21:32:47.329479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328720+0000","last_change":"2024-03-11T21:30:52.794455+0000","last_active":"2024-03-11T21:32:47.328720+0000","last_peered":"2024-03-11T21:32:47.328720+0000","last_clean":"2024-03-11T21:32:47.328720+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:32:47.328720+0000","last_undegraded":"2024-03-11T21:32:47.328720+0000","last_fullsized":"2024-03-11T21:32:47.328720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.794209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.794209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":468,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334582+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:32:47.334582+0000","last_peered":"2024-03-11T21:32:47.334582+0000","last_clean":"2024-03-11T21:32:47.334582+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:32:47.334582+0000","last_undegraded":"2024-03-11T21:32:47.334582+0000","last_fullsized":"2024-03-11T21:32:47.334582+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422058+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":554,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.311064+0000","last_change":"2024-03-11T21:31:37.386896+0000","last_active":"2024-03-11T21:32:45.311064+0000","last_peered":"2024-03-11T21:32:45.311064+0000","last_clean":"2024-03-11T21:32:45.311064+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:32:45.311064+0000","last_undegraded":"2024-03-11T21:32:45.311064+0000","last_fullsized":"2024-03-11T21:32:45.311064+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:20.448110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.448110+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":638,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.318216+0000","last_change":"2024-03-11T21:32:29.159917+0000","last_active":"2024-03-11T21:32:47.318216+0000","last_peered":"2024-03-11T21:32:47.318216+0000","last_clean":"2024-03-11T21:32:47.318216+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:32:47.318216+0000","last_undegraded":"2024-03-11T21:32:47.318216+0000","last_fullsized":"2024-03-11T21:32:47.318216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:29.159716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:29.159716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.410503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":754,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.343465+0000","last_change":"2024-03-11T21:30:56.549284+0000","last_active":"2024-03-11T21:32:47.343465+0000","last_peered":"2024-03-11T21:32:47.343465+0000","last_clean":"2024-03-11T21:32:47.343465+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:32:47.343465+0000","last_undegraded":"2024-03-11T21:32:47.343465+0000","last_fullsized":"2024-03-11T21:32:47.343465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:56.549048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.549048+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.752448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.336669+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:32:47.336669+0000","last_peered":"2024-03-11T21:32:47.336669+0000","last_clean":"2024-03-11T21:32:47.336669+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:32:47.336669+0000","last_undegraded":"2024-03-11T21:32:47.336669+0000","last_fullsized":"2024-03-11T21:32:47.336669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":478,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329709+0000","last_change":"2024-03-11T21:31:37.387585+0000","last_active":"2024-03-11T21:32:47.329709+0000","last_peered":"2024-03-11T21:32:47.329709+0000","last_clean":"2024-03-11T21:32:47.329709+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:32:47.329709+0000","last_undegraded":"2024-03-11T21:32:47.329709+0000","last_fullsized":"2024-03-11T21:32:47.329709+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:39.006804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:39.006804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.305857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324405+0000","last_change":"2024-03-11T21:31:54.548395+0000","last_active":"2024-03-11T21:32:47.324405+0000","last_peered":"2024-03-11T21:32:47.324405+0000","last_clean":"2024-03-11T21:32:47.324405+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:32:47.324405+0000","last_undegraded":"2024-03-11T21:32:47.324405+0000","last_fullsized":"2024-03-11T21:32:47.324405+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:54.548275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.548275+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:22.947459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":625,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.326502+0000","last_change":"2024-03-11T21:28:48.305255+0000","last_active":"2024-03-11T21:32:47.326502+0000","last_peered":"2024-03-11T21:32:47.326502+0000","last_clean":"2024-03-11T21:32:47.326502+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:32:47.326502+0000","last_undegraded":"2024-03-11T21:32:47.326502+0000","last_fullsized":"2024-03-11T21:32:47.326502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:28:48.305156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.305156+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":627,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326474+0000","last_change":"2024-03-11T21:32:23.014073+0000","last_active":"2024-03-11T21:32:48.326474+0000","last_peered":"2024-03-11T21:32:48.326474+0000","last_clean":"2024-03-11T21:32:48.326474+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:32:48.326474+0000","last_undegraded":"2024-03-11T21:32:48.326474+0000","last_fullsized":"2024-03-11T21:32:48.326474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.013984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.013984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.212286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":477,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.316184+0000","last_change":"2024-03-11T21:31:37.396975+0000","last_active":"2024-03-11T21:32:45.316184+0000","last_peered":"2024-03-11T21:32:45.316184+0000","last_clean":"2024-03-11T21:32:45.316184+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:32:45.316184+0000","last_undegraded":"2024-03-11T21:32:45.316184+0000","last_fullsized":"2024-03-11T21:32:45.316184+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:56.181111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:56.181111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":468,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.329959+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:32:47.329959+0000","last_peered":"2024-03-11T21:32:47.329959+0000","last_clean":"2024-03-11T21:32:47.329959+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:32:47.329959+0000","last_undegraded":"2024-03-11T21:32:47.329959+0000","last_fullsized":"2024-03-11T21:32:47.329959+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.422403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":634,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.325886+0000","last_change":"2024-03-11T21:30:28.077530+0000","last_active":"2024-03-11T21:32:48.325886+0000","last_peered":"2024-03-11T21:32:48.325886+0000","last_clean":"2024-03-11T21:32:48.325886+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:32:48.325886+0000","last_undegraded":"2024-03-11T21:32:48.325886+0000","last_fullsized":"2024-03-11T21:32:48.325886+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:30:28.077400+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:30:28.077400+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.985913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":639,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.327522+0000","last_change":"2024-03-11T21:32:10.547894+0000","last_active":"2024-03-11T21:32:47.327522+0000","last_peered":"2024-03-11T21:32:47.327522+0000","last_clean":"2024-03-11T21:32:47.327522+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:32:47.327522+0000","last_undegraded":"2024-03-11T21:32:47.327522+0000","last_fullsized":"2024-03-11T21:32:47.327522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.547682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.547682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.675268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":471,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.319542+0000","last_change":"2024-03-11T21:32:01.016344+0000","last_active":"2024-03-11T21:32:45.319542+0000","last_peered":"2024-03-11T21:32:45.319542+0000","last_clean":"2024-03-11T21:32:45.319542+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:32:45.319542+0000","last_undegraded":"2024-03-11T21:32:45.319542+0000","last_fullsized":"2024-03-11T21:32:45.319542+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.016240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.016240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:01.498002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":621,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324499+0000","last_change":"2024-03-11T21:30:40.909457+0000","last_active":"2024-03-11T21:32:47.324499+0000","last_peered":"2024-03-11T21:32:47.324499+0000","last_clean":"2024-03-11T21:32:47.324499+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:32:47.324499+0000","last_undegraded":"2024-03-11T21:32:47.324499+0000","last_fullsized":"2024-03-11T21:32:47.324499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.909356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.909356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311691+0000","last_change":"2024-03-11T21:32:25.655891+0000","last_active":"2024-03-11T21:32:47.311691+0000","last_peered":"2024-03-11T21:32:47.311691+0000","last_clean":"2024-03-11T21:32:47.311691+0000","last_became_active":"2024-03-11T21:28:51.388058+0000","last_became_peered":"2024-03-11T21:28:51.388058+0000","last_unstale":"2024-03-11T21:32:47.311691+0000","last_undegraded":"2024-03-11T21:32:47.311691+0000","last_fullsized":"2024-03-11T21:32:47.311691+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.655683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:53.974261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.313711+0000","last_change":"2024-03-11T21:31:37.378913+0000","last_active":"2024-03-11T21:32:47.313711+0000","last_peered":"2024-03-11T21:32:47.313711+0000","last_clean":"2024-03-11T21:32:47.313711+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:32:47.313711+0000","last_undegraded":"2024-03-11T21:32:47.313711+0000","last_fullsized":"2024-03-11T21:32:47.313711+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:30:24.997673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.997673+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:50.673084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":631,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328221+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:32:47.328221+0000","last_peered":"2024-03-11T21:32:47.328221+0000","last_clean":"2024-03-11T21:32:47.328221+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:32:47.328221+0000","last_undegraded":"2024-03-11T21:32:47.328221+0000","last_fullsized":"2024-03-11T21:32:47.328221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:50.107199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":624,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.316655+0000","last_change":"2024-03-11T21:32:09.768728+0000","last_active":"2024-03-11T21:32:47.316655+0000","last_peered":"2024-03-11T21:32:47.316655+0000","last_clean":"2024-03-11T21:32:47.316655+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:32:47.316655+0000","last_undegraded":"2024-03-11T21:32:47.316655+0000","last_fullsized":"2024-03-11T21:32:47.316655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.768608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.768608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.029403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":612,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335907+0000","last_change":"2024-03-11T21:30:36.777188+0000","last_active":"2024-03-11T21:32:47.335907+0000","last_peered":"2024-03-11T21:32:47.335907+0000","last_clean":"2024-03-11T21:32:47.335907+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:32:47.335907+0000","last_undegraded":"2024-03-11T21:32:47.335907+0000","last_fullsized":"2024-03-11T21:32:47.335907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.777107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.777107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":477,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.345832+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:32:47.345832+0000","last_peered":"2024-03-11T21:32:47.345832+0000","last_clean":"2024-03-11T21:32:47.345832+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:32:47.345832+0000","last_undegraded":"2024-03-11T21:32:47.345832+0000","last_fullsized":"2024-03-11T21:32:47.345832+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:00.106275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":650,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324812+0000","last_change":"2024-03-11T21:32:09.759891+0000","last_active":"2024-03-11T21:32:47.324812+0000","last_peered":"2024-03-11T21:32:47.324812+0000","last_clean":"2024-03-11T21:32:47.324812+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:32:47.324812+0000","last_undegraded":"2024-03-11T21:32:47.324812+0000","last_fullsized":"2024-03-11T21:32:47.324812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:32:09.759634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.759634+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.521604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":630,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335982+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:32:47.335982+0000","last_peered":"2024-03-11T21:32:47.335982+0000","last_clean":"2024-03-11T21:32:47.335982+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:32:47.335982+0000","last_undegraded":"2024-03-11T21:32:47.335982+0000","last_fullsized":"2024-03-11T21:32:47.335982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.467463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":615,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.320605+0000","last_change":"2024-03-11T21:30:33.024945+0000","last_active":"2024-03-11T21:32:47.320605+0000","last_peered":"2024-03-11T21:32:47.320605+0000","last_clean":"2024-03-11T21:32:47.320605+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:32:47.320605+0000","last_undegraded":"2024-03-11T21:32:47.320605+0000","last_fullsized":"2024-03-11T21:32:47.320605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:33.024687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:33.024687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":631,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.324832+0000","last_change":"2024-03-11T21:32:01.014273+0000","last_active":"2024-03-11T21:32:48.324832+0000","last_peered":"2024-03-11T21:32:48.324832+0000","last_clean":"2024-03-11T21:32:48.324832+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:32:48.324832+0000","last_undegraded":"2024-03-11T21:32:48.324832+0000","last_fullsized":"2024-03-11T21:32:48.324832+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.014168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.014168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:13.372767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":479,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.322679+0000","last_change":"2024-03-11T21:32:15.652648+0000","last_active":"2024-03-11T21:32:47.322679+0000","last_peered":"2024-03-11T21:32:47.322679+0000","last_clean":"2024-03-11T21:32:47.322679+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:32:47.322679+0000","last_undegraded":"2024-03-11T21:32:47.322679+0000","last_fullsized":"2024-03-11T21:32:47.322679+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.652323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.652323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:22.960005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":668,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.324004+0000","last_change":"2024-03-11T21:31:37.388352+0000","last_active":"2024-03-11T21:32:47.324004+0000","last_peered":"2024-03-11T21:32:47.324004+0000","last_clean":"2024-03-11T21:32:47.324004+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:32:47.324004+0000","last_undegraded":"2024-03-11T21:32:47.324004+0000","last_fullsized":"2024-03-11T21:32:47.324004+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:30:42.940256+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.940256+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":630,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338918+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:32:47.338918+0000","last_peered":"2024-03-11T21:32:47.338918+0000","last_clean":"2024-03-11T21:32:47.338918+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:32:47.338918+0000","last_undegraded":"2024-03-11T21:32:47.338918+0000","last_fullsized":"2024-03-11T21:32:47.338918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":623,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.332680+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:32:47.332680+0000","last_peered":"2024-03-11T21:32:47.332680+0000","last_clean":"2024-03-11T21:32:47.332680+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:32:47.332680+0000","last_undegraded":"2024-03-11T21:32:47.332680+0000","last_fullsized":"2024-03-11T21:32:47.332680+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":620,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.321313+0000","last_change":"2024-03-11T21:31:37.387295+0000","last_active":"2024-03-11T21:32:47.321313+0000","last_peered":"2024-03-11T21:32:47.321313+0000","last_clean":"2024-03-11T21:32:47.321313+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:32:47.321313+0000","last_undegraded":"2024-03-11T21:32:47.321313+0000","last_fullsized":"2024-03-11T21:32:47.321313+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.009609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.009609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.306390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":477,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.338498+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:32:47.338498+0000","last_peered":"2024-03-11T21:32:47.338498+0000","last_clean":"2024-03-11T21:32:47.338498+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:32:47.338498+0000","last_undegraded":"2024-03-11T21:32:47.338498+0000","last_fullsized":"2024-03-11T21:32:47.338498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:51.421819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"121'16","reported_seq":655,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326385+0000","last_change":"2024-03-11T21:30:35.925775+0000","last_active":"2024-03-11T21:32:48.326385+0000","last_peered":"2024-03-11T21:32:48.326385+0000","last_clean":"2024-03-11T21:32:48.326385+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:32:48.326385+0000","last_undegraded":"2024-03-11T21:32:48.326385+0000","last_fullsized":"2024-03-11T21:32:48.326385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:30:35.925542+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.925542+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.986562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":624,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.334969+0000","last_change":"2024-03-11T21:30:30.892128+0000","last_active":"2024-03-11T21:32:47.334969+0000","last_peered":"2024-03-11T21:32:47.334969+0000","last_clean":"2024-03-11T21:32:47.334969+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:32:47.334969+0000","last_undegraded":"2024-03-11T21:32:47.334969+0000","last_fullsized":"2024-03-11T21:32:47.334969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:30.891767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.891767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":475,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.311418+0000","last_change":"2024-03-11T21:32:02.053080+0000","last_active":"2024-03-11T21:32:45.311418+0000","last_peered":"2024-03-11T21:32:45.311418+0000","last_clean":"2024-03-11T21:32:45.311418+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:32:45.311418+0000","last_undegraded":"2024-03-11T21:32:45.311418+0000","last_fullsized":"2024-03-11T21:32:45.311418+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.052865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.052865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.499264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":629,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311884+0000","last_change":"2024-03-11T21:32:07.733765+0000","last_active":"2024-03-11T21:32:47.311884+0000","last_peered":"2024-03-11T21:32:47.311884+0000","last_clean":"2024-03-11T21:32:47.311884+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:32:47.311884+0000","last_undegraded":"2024-03-11T21:32:47.311884+0000","last_fullsized":"2024-03-11T21:32:47.311884+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.733556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.733556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.680676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":325,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-11T21:32:45.315836+0000","last_change":"2024-03-11T21:31:37.397343+0000","last_active":"2024-03-11T21:32:45.315836+0000","last_peered":"2024-03-11T21:32:45.315836+0000","last_clean":"2024-03-11T21:32:45.315836+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:32:45.315836+0000","last_undegraded":"2024-03-11T21:32:45.315836+0000","last_fullsized":"2024-03-11T21:32:45.315836+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:21.708539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.708539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.966512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":635,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.335197+0000","last_change":"2024-03-11T21:29:09.487728+0000","last_active":"2024-03-11T21:32:47.335197+0000","last_peered":"2024-03-11T21:32:47.335197+0000","last_clean":"2024-03-11T21:32:47.335197+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:32:47.335197+0000","last_undegraded":"2024-03-11T21:32:47.335197+0000","last_fullsized":"2024-03-11T21:32:47.335197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:29:09.487592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:09.487592+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.742154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":636,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-11T21:32:48.326377+0000","last_change":"2024-03-11T21:30:16.988987+0000","last_active":"2024-03-11T21:32:48.326377+0000","last_peered":"2024-03-11T21:32:48.326377+0000","last_clean":"2024-03-11T21:32:48.326377+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:32:48.326377+0000","last_undegraded":"2024-03-11T21:32:48.326377+0000","last_fullsized":"2024-03-11T21:32:48.326377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:30:16.988898+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:30:16.988898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.985703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":633,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.311873+0000","last_change":"2024-03-11T21:31:58.673043+0000","last_active":"2024-03-11T21:32:47.311873+0000","last_peered":"2024-03-11T21:32:47.311873+0000","last_clean":"2024-03-11T21:32:47.311873+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:32:47.311873+0000","last_undegraded":"2024-03-11T21:32:47.311873+0000","last_fullsized":"2024-03-11T21:32:47.311873+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.672796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.672796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.012613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":611,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:32:47.328046+0000","last_change":"2024-03-11T21:32:10.686985+0000","last_active":"2024-03-11T21:32:47.328046+0000","last_peered":"2024-03-11T21:32:47.328046+0000","last_clean":"2024-03-11T21:32:47.328046+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:32:47.328046+0000","last_undegraded":"2024-03-11T21:32:47.328046+0000","last_fullsized":"2024-03-11T21:32:47.328046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.686767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.686767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.081264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":262,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":7},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":4195516,"num_objects":123,"num_object_clones":2,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":550,"num_write_kb":8735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14069760,"data_stored":12586548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1804,"ondisk_log_size":1804,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":15,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":6400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":45,"acting":45,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":564,"num_read_kb":559,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841664,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73896,"kb_used_data":10760,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93671256,"statfs":{"total":95995035648,"available":95919366144,"internally_reserved":0,"allocated":11018240,"data_stored":8504333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168713,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77996,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93667156,"statfs":{"total":95995035648,"available":95915167744,"internally_reserved":0,"allocated":6684672,"data_stored":4306975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352335,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42148,"kb_used_data":5904,"kb_used_omap":27,"kb_used_meta":36196,"kb_avail":93703004,"statfs":{"total":95995035648,"available":95951876096,"internally_reserved":0,"allocated":6045696,"data_stored":3682249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":37065033},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73768,"kb_used_data":10632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93671384,"statfs":{"total":95995035648,"available":95919497216,"internally_reserved":0,"allocated":10887168,"data_stored":8502882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378803,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81644,"kb_used_data":6028,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663508,"statfs":{"total":95995035648,"available":95911432192,"internally_reserved":0,"allocated":6172672,"data_stored":3717609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117159,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81624,"kb_used_data":6008,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663528,"statfs":{"total":95995035648,"available":95911452672,"internally_reserved":0,"allocated":6152192,"data_stored":3717196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855517,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69208,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93675944,"statfs":{"total":95995035648,"available":95924166656,"internally_reserved":0,"allocated":6217728,"data_stored":3717764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495763,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85600,"kb_used_data":9996,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659552,"statfs":{"total":95995035648,"available":95907381248,"internally_reserved":0,"allocated":10235904,"data_stored":7898695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4325376,"data_stored":4194641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4341760,"data_stored":4194682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4194856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":262,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:32:50.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:32:50.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:50.720 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:50.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: cluster 2024-03-11T21:32:49.333362+0000 mgr.y (mgr.14550) 393 : cluster 0 pgmap v540: 275 pgs: 1 clean+premerge+peered, 4 unknown, 270 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:50.825 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:50.845 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:50.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:50.974 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: cluster 2024-03-11T21:32:49.333362+0000 mgr.y (mgr.14550) 393 : cluster 0 pgmap v540: 275 pgs: 1 clean+premerge+peered, 4 unknown, 270 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:49.352287+0000 mgr.y (mgr.14550) 394 : audit 0 from='client.30270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.079400+0000 mon.c (mon.2) 772 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: cluster 2024-03-11T21:32:50.317400+0000 mon.a (mon.0) 2336 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.325280+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]': finished 2024-03-11T21:32:51.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.325529+0000 mon.a (mon.0) 2338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: cluster 2024-03-11T21:32:50.332033+0000 mon.a (mon.0) 2339 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:32:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.337408+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.344345+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.501426+0000 mon.c (mon.2) 774 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:50 smithi203 bash[23654]: audit 2024-03-11T21:32:50.501861+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:51.104 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:49.352287+0000 mgr.y (mgr.14550) 394 : audit 0 from='client.30270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.079400+0000 mon.c (mon.2) 772 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: cluster 2024-03-11T21:32:50.317400+0000 mon.a (mon.0) 2336 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.325280+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]': finished 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.325529+0000 mon.a (mon.0) 2338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: cluster 2024-03-11T21:32:50.332033+0000 mon.a (mon.0) 2339 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.337408+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.344345+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.501426+0000 mon.c (mon.2) 774 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[21512]: audit 2024-03-11T21:32:50.501861+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: cluster 2024-03-11T21:32:49.333362+0000 mgr.y (mgr.14550) 393 : cluster 0 pgmap v540: 275 pgs: 1 clean+premerge+peered, 4 unknown, 270 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:49.352287+0000 mgr.y (mgr.14550) 394 : audit 0 from='client.30270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.079400+0000 mon.c (mon.2) 772 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: cluster 2024-03-11T21:32:50.317400+0000 mon.a (mon.0) 2336 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.325280+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-39", "mode": "writeback"}]': finished 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.325529+0000 mon.a (mon.0) 2338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi143-71604-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: cluster 2024-03-11T21:32:50.332033+0000 mon.a (mon.0) 2339 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.337408+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.344345+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.501426+0000 mon.c (mon.2) 774 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:50 smithi143 bash[29284]: audit 2024-03-11T21:32:50.501861+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:32:51.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:51.233 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:51.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:51.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.487 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:51.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:51.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:51.745 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:51.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:51.875 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.081033+0000 mon.c (mon.2) 775 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:51.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.173111+0000 mon.c (mon.2) 776 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.285764+0000 mon.a (mon.0) 2342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.302062+0000 mon.c (mon.2) 777 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.334926+0000 mon.c (mon.2) 778 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: cluster 2024-03-11T21:32:51.756075+0000 mon.a (mon.0) 2343 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.757381+0000 mon.a (mon.0) 2344 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[21512]: audit 2024-03-11T21:32:51.764923+0000 mon.a (mon.0) 2345 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.081033+0000 mon.c (mon.2) 775 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.173111+0000 mon.c (mon.2) 776 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.285764+0000 mon.a (mon.0) 2342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:51.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.302062+0000 mon.c (mon.2) 777 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.334926+0000 mon.c (mon.2) 778 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: cluster 2024-03-11T21:32:51.756075+0000 mon.a (mon.0) 2343 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:32:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.757381+0000 mon.a (mon.0) 2344 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:51.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:51 smithi143 bash[29284]: audit 2024-03-11T21:32:51.764923+0000 mon.a (mon.0) 2345 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:51.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.081033+0000 mon.c (mon.2) 775 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.173111+0000 mon.c (mon.2) 776 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.285764+0000 mon.a (mon.0) 2342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.302062+0000 mon.c (mon.2) 777 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.334926+0000 mon.c (mon.2) 778 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: cluster 2024-03-11T21:32:51.756075+0000 mon.a (mon.0) 2343 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.757381+0000 mon.a (mon.0) 2344 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]: dispatch 2024-03-11T21:32:52.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:51 smithi203 bash[23654]: audit 2024-03-11T21:32:51.764923+0000 mon.a (mon.0) 2345 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T21:32:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:52.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:52.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.268 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:52.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.383 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:32:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:52.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:52.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:52.529 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:52.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:52.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:52.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: cluster 2024-03-11T21:32:51.334183+0000 mgr.y (mgr.14550) 395 : cluster 0 pgmap v543: 274 pgs: 1 peering, 273 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: audit 2024-03-11T21:32:52.082343+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: cluster 2024-03-11T21:32:52.287340+0000 mon.a (mon.0) 2346 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: audit 2024-03-11T21:32:52.295777+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: audit 2024-03-11T21:32:52.296028+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: audit 2024-03-11T21:32:52.296178+0000 mon.a (mon.0) 2349 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T21:32:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:52 smithi203 bash[23654]: cluster 2024-03-11T21:32:52.302887+0000 mon.a (mon.0) 2350 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:32:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:52.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:52.917 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:53.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:53.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: cluster 2024-03-11T21:32:51.334183+0000 mgr.y (mgr.14550) 395 : cluster 0 pgmap v543: 274 pgs: 1 peering, 273 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:53.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: audit 2024-03-11T21:32:52.082343+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:53.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: cluster 2024-03-11T21:32:52.287340+0000 mon.a (mon.0) 2346 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: audit 2024-03-11T21:32:52.295777+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: audit 2024-03-11T21:32:52.296028+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: audit 2024-03-11T21:32:52.296178+0000 mon.a (mon.0) 2349 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[21512]: cluster 2024-03-11T21:32:52.302887+0000 mon.a (mon.0) 2350 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: cluster 2024-03-11T21:32:51.334183+0000 mgr.y (mgr.14550) 395 : cluster 0 pgmap v543: 274 pgs: 1 peering, 273 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: audit 2024-03-11T21:32:52.082343+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: cluster 2024-03-11T21:32:52.287340+0000 mon.a (mon.0) 2346 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: audit 2024-03-11T21:32:52.295777+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi143-71604-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: audit 2024-03-11T21:32:52.296028+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-39"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: audit 2024-03-11T21:32:52.296178+0000 mon.a (mon.0) 2349 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T21:32:53.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:52 smithi143 bash[29284]: cluster 2024-03-11T21:32:52.302887+0000 mon.a (mon.0) 2350 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:32:53.046 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:53.125 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:53.125 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:53.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:53.177 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:53.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:53.290 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-11T21:32:53.290 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-11T21:32:53.290 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: seed 71773 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (278 ms) 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (35696 ms) 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (116 ms) 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (36090 ms total) 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (11170 ms) 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-11T21:32:53.291 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (10059 ms) 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10743 ms) 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [3] 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [4,3] 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting some heads 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-11T21:32:53.292 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrubs... 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47671 ms) 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10003 ms) 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8046 ms) 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7994 ms) 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8016 ms) 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-11T21:32:53.293 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10464 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8952 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9570 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13026 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7989 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8003 ms) 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-11T21:32:53.294 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13461 ms) 2024-03-11T21:32:53.295 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-11T21:32:53.295 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7542 ms) 2024-03-11T21:32:53.295 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-11T21:32:53.295 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8007 ms) 2024-03-11T21:32:53.323 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:53.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.453 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:53.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:53.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:53.707 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:53.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:53.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:53.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:53 smithi143 bash[21512]: audit 2024-03-11T21:32:53.084090+0000 mon.c (mon.2) 780 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:53.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:53 smithi143 bash[21512]: cluster 2024-03-11T21:32:53.305927+0000 mon.a (mon.0) 2351 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:32:53.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:53 smithi143 bash[29284]: audit 2024-03-11T21:32:53.084090+0000 mon.c (mon.2) 780 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:53.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:53 smithi143 bash[29284]: cluster 2024-03-11T21:32:53.305927+0000 mon.a (mon.0) 2351 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:32:53.962 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:54.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:53 smithi203 bash[23654]: audit 2024-03-11T21:32:53.084090+0000 mon.c (mon.2) 780 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:54.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:53 smithi203 bash[23654]: cluster 2024-03-11T21:32:53.305927+0000 mon.a (mon.0) 2351 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:32:54.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:54.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:54.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:54.279 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:54.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:54.406 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:54.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:54.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:54.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:54.661 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:54.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:54.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:54.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: cluster 2024-03-11T21:32:53.335255+0000 mgr.y (mgr.14550) 396 : cluster 0 pgmap v546: 250 pgs: 7 creating+peering, 1 unknown, 3 peering, 239 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: cluster 2024-03-11T21:32:53.348133+0000 osd.6 (osd.6) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: cluster 2024-03-11T21:32:53.348592+0000 osd.6 (osd.6) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: audit 2024-03-11T21:32:54.085255+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: cluster 2024-03-11T21:32:54.298807+0000 mon.a (mon.0) 2352 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: cluster 2024-03-11T21:32:54.319919+0000 mon.a (mon.0) 2353 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:32:54.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: audit 2024-03-11T21:32:54.324485+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:54.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: audit 2024-03-11T21:32:54.329767+0000 mon.c (mon.2) 783 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:54.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: audit 2024-03-11T21:32:54.335339+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:54.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:54 smithi203 bash[23654]: audit 2024-03-11T21:32:54.336270+0000 mon.a (mon.0) 2355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:54.921 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:55.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.053 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:55.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:55.165 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:55.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: cluster 2024-03-11T21:32:53.335255+0000 mgr.y (mgr.14550) 396 : cluster 0 pgmap v546: 250 pgs: 7 creating+peering, 1 unknown, 3 peering, 239 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: cluster 2024-03-11T21:32:53.348133+0000 osd.6 (osd.6) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: cluster 2024-03-11T21:32:53.348592+0000 osd.6 (osd.6) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: audit 2024-03-11T21:32:54.085255+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: cluster 2024-03-11T21:32:54.298807+0000 mon.a (mon.0) 2352 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: cluster 2024-03-11T21:32:54.319919+0000 mon.a (mon.0) 2353 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: audit 2024-03-11T21:32:54.324485+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: audit 2024-03-11T21:32:54.329767+0000 mon.c (mon.2) 783 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: audit 2024-03-11T21:32:54.335339+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[21512]: audit 2024-03-11T21:32:54.336270+0000 mon.a (mon.0) 2355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: cluster 2024-03-11T21:32:53.335255+0000 mgr.y (mgr.14550) 396 : cluster 0 pgmap v546: 250 pgs: 7 creating+peering, 1 unknown, 3 peering, 239 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: cluster 2024-03-11T21:32:53.348133+0000 osd.6 (osd.6) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: cluster 2024-03-11T21:32:53.348592+0000 osd.6 (osd.6) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: audit 2024-03-11T21:32:54.085255+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: cluster 2024-03-11T21:32:54.298807+0000 mon.a (mon.0) 2352 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: cluster 2024-03-11T21:32:54.319919+0000 mon.a (mon.0) 2353 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: audit 2024-03-11T21:32:54.324485+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: audit 2024-03-11T21:32:54.329767+0000 mon.c (mon.2) 783 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: audit 2024-03-11T21:32:54.335339+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:32:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:54 smithi143 bash[29284]: audit 2024-03-11T21:32:54.336270+0000 mon.a (mon.0) 2355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:55.321 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:55.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.451 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:55.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:55.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:55.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:55.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.842 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:55.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.086922+0000 mon.c (mon.2) 784 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:55.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.312029+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:55.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.312321+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:55.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: cluster 2024-03-11T21:32:55.324127+0000 mon.a (mon.0) 2358 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.326094+0000 mon.c (mon.2) 785 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.328601+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.329148+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.330664+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.338500+0000 mon.c (mon.2) 787 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[29284]: audit 2024-03-11T21:32:55.341971+0000 mon.a (mon.0) 2361 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.086922+0000 mon.c (mon.2) 784 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.312029+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.312321+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: cluster 2024-03-11T21:32:55.324127+0000 mon.a (mon.0) 2358 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.326094+0000 mon.c (mon.2) 785 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.328601+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:55.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.329148+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:55.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.330664+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:55.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.338500+0000 mon.c (mon.2) 787 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:55.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:55 smithi143 bash[21512]: audit 2024-03-11T21:32:55.341971+0000 mon.a (mon.0) 2361 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:55.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:55.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.086922+0000 mon.c (mon.2) 784 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.312029+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.312321+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: cluster 2024-03-11T21:32:55.324127+0000 mon.a (mon.0) 2358 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.326094+0000 mon.c (mon.2) 785 : audit 1 from='client.? 172.21.15.143:0/488028299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.328601+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.329148+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.330664+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.338500+0000 mon.c (mon.2) 787 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:56.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:55 smithi203 bash[23654]: audit 2024-03-11T21:32:55.341971+0000 mon.a (mon.0) 2361 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T21:32:56.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.092 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:56.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:56.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:56.354 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:56.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.588 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:56.612 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:56.629 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:56.629 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:56.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.747 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:56.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: cluster 2024-03-11T21:32:55.337679+0000 mgr.y (mgr.14550) 397 : cluster 0 pgmap v549: 273 pgs: 32 unknown, 2 peering, 239 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:56.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.088408+0000 mon.c (mon.2) 788 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:56.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.321793+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:56.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.321931+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:56.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.322101+0000 mon.a (mon.0) 2364 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: cluster 2024-03-11T21:32:56.329253+0000 mon.a (mon.0) 2365 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.333627+0000 mon.c (mon.2) 789 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.336190+0000 mon.a (mon.0) 2366 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.375488+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.375863+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.376847+0000 mon.c (mon.2) 791 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.377142+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.377683+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:56.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:56 smithi203 bash[23654]: audit 2024-03-11T21:32:56.377961+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:56.878 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:56.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: cluster 2024-03-11T21:32:55.337679+0000 mgr.y (mgr.14550) 397 : cluster 0 pgmap v549: 273 pgs: 32 unknown, 2 peering, 239 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:56.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.088408+0000 mon.c (mon.2) 788 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:56.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.321793+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.321931+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.322101+0000 mon.a (mon.0) 2364 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: cluster 2024-03-11T21:32:56.329253+0000 mon.a (mon.0) 2365 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.333627+0000 mon.c (mon.2) 789 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.336190+0000 mon.a (mon.0) 2366 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.375488+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.375863+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.376847+0000 mon.c (mon.2) 791 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.377142+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.377683+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:56.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[29284]: audit 2024-03-11T21:32:56.377961+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: cluster 2024-03-11T21:32:55.337679+0000 mgr.y (mgr.14550) 397 : cluster 0 pgmap v549: 273 pgs: 32 unknown, 2 peering, 239 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.088408+0000 mon.c (mon.2) 788 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.321793+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi143-71604-56"}]': finished 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.321931+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.322101+0000 mon.a (mon.0) 2364 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: cluster 2024-03-11T21:32:56.329253+0000 mon.a (mon.0) 2365 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:32:56.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.333627+0000 mon.c (mon.2) 789 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.336190+0000 mon.a (mon.0) 2366 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.375488+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.375863+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.376847+0000 mon.c (mon.2) 791 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.377142+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.377683+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:56.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:56 smithi143 bash[21512]: audit 2024-03-11T21:32:56.377961+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:32:57.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:57.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:57.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:57.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:57.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:57.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:57.402 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:57.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:32:57.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:57.532 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:57.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:57.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:57.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:57.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.089655+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.324681+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.324955+0000 mon.a (mon.0) 2371 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: cluster 2024-03-11T21:32:57.333470+0000 mon.a (mon.0) 2372 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.342448+0000 mon.c (mon.2) 794 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.343363+0000 mon.c (mon.2) 795 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.343784+0000 mon.a (mon.0) 2373 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: audit 2024-03-11T21:32:57.348519+0000 mon.a (mon.0) 2374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: cluster 2024-03-11T21:32:57.394196+0000 osd.2 (osd.2) 181 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:58.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[21512]: cluster 2024-03-11T21:32:57.394680+0000 osd.2 (osd.2) 182 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.089655+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.324681+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.324955+0000 mon.a (mon.0) 2371 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: cluster 2024-03-11T21:32:57.333470+0000 mon.a (mon.0) 2372 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.342448+0000 mon.c (mon.2) 794 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.343363+0000 mon.c (mon.2) 795 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.343784+0000 mon.a (mon.0) 2373 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: audit 2024-03-11T21:32:57.348519+0000 mon.a (mon.0) 2374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: cluster 2024-03-11T21:32:57.394196+0000 osd.2 (osd.2) 181 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:58.011 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:57 smithi143 bash[29284]: cluster 2024-03-11T21:32:57.394680+0000 osd.2 (osd.2) 182 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.089655+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.324681+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:32:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.324955+0000 mon.a (mon.0) 2371 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi143-71604-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:32:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: cluster 2024-03-11T21:32:57.333470+0000 mon.a (mon.0) 2372 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:32:58.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.342448+0000 mon.c (mon.2) 794 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.343363+0000 mon.c (mon.2) 795 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.343784+0000 mon.a (mon.0) 2373 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:32:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: audit 2024-03-11T21:32:57.348519+0000 mon.a (mon.0) 2374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:32:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: cluster 2024-03-11T21:32:57.394196+0000 osd.2 (osd.2) 181 : cluster 0 purged_snaps scrub starts 2024-03-11T21:32:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:57 smithi203 bash[23654]: cluster 2024-03-11T21:32:57.394680+0000 osd.2 (osd.2) 182 : cluster 0 purged_snaps scrub ok 2024-03-11T21:32:58.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:58.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:58.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:58.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:58.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:58.316 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:58.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:58.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:58.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:32:58.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:58.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:58.835 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: cluster 2024-03-11T21:32:57.339595+0000 mgr.y (mgr.14550) 398 : cluster 0 pgmap v552: 273 pgs: 29 unknown, 1 peering, 243 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: audit 2024-03-11T21:32:58.090655+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: audit 2024-03-11T21:32:58.336579+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: audit 2024-03-11T21:32:58.349789+0000 mon.c (mon.2) 797 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: cluster 2024-03-11T21:32:58.783304+0000 mon.a (mon.0) 2376 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:32:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[21512]: audit 2024-03-11T21:32:58.784649+0000 mon.a (mon.0) 2377 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: cluster 2024-03-11T21:32:57.339595+0000 mgr.y (mgr.14550) 398 : cluster 0 pgmap v552: 273 pgs: 29 unknown, 1 peering, 243 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: audit 2024-03-11T21:32:58.090655+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: audit 2024-03-11T21:32:58.336579+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: audit 2024-03-11T21:32:58.349789+0000 mon.c (mon.2) 797 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: cluster 2024-03-11T21:32:58.783304+0000 mon.a (mon.0) 2376 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:32:58.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:58 smithi143 bash[29284]: audit 2024-03-11T21:32:58.784649+0000 mon.a (mon.0) 2377 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: cluster 2024-03-11T21:32:57.339595+0000 mgr.y (mgr.14550) 398 : cluster 0 pgmap v552: 273 pgs: 29 unknown, 1 peering, 243 active+clean; 8.4 MiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: audit 2024-03-11T21:32:58.090655+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: audit 2024-03-11T21:32:58.336579+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: audit 2024-03-11T21:32:58.349789+0000 mon.c (mon.2) 797 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: cluster 2024-03-11T21:32:58.783304+0000 mon.a (mon.0) 2376 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:32:58.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:58 smithi203 bash[23654]: audit 2024-03-11T21:32:58.784649+0000 mon.a (mon.0) 2377 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:32:58.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:59.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:59.099 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:59.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:59.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:59.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:59.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:59.476 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:59.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:59.606 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:59.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:59.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:59.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:59.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:00.005 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:58.553150+0000 mgr.y (mgr.14550) 399 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:59.091859+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:00.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:59.339072+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:59.339193+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:33:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: cluster 2024-03-11T21:32:59.342643+0000 mon.a (mon.0) 2380 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:33:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:59.343293+0000 mon.c (mon.2) 799 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:32:59 smithi203 bash[23654]: audit 2024-03-11T21:32:59.346320+0000 mon.a (mon.0) 2381 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:00.134 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:00.151 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:58.553150+0000 mgr.y (mgr.14550) 399 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:00.151 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:59.091859+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:59.339072+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:59.339193+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: cluster 2024-03-11T21:32:59.342643+0000 mon.a (mon.0) 2380 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:59.343293+0000 mon.c (mon.2) 799 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[29284]: audit 2024-03-11T21:32:59.346320+0000 mon.a (mon.0) 2381 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:58.553150+0000 mgr.y (mgr.14550) 399 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:59.091859+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:00.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:59.339072+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi143-71604-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:00.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:59.339193+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:33:00.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: cluster 2024-03-11T21:32:59.342643+0000 mon.a (mon.0) 2380 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:33:00.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:59.343293+0000 mon.c (mon.2) 799 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:32:59 smithi143 bash[21512]: audit 2024-03-11T21:32:59.346320+0000 mon.a (mon.0) 2381 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:33:00.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:00.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:00.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:00.391 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:00.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.522 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:00.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:00.651 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:00.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:00.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:00.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:01.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: cluster 2024-03-11T21:32:59.340615+0000 mgr.y (mgr.14550) 400 : cluster 0 pgmap v555: 280 pgs: 8 unknown, 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.093534+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: cluster 2024-03-11T21:33:00.339839+0000 mon.a (mon.0) 2382 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.341710+0000 mon.a (mon.0) 2383 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: cluster 2024-03-11T21:33:00.346743+0000 mon.a (mon.0) 2384 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.405085+0000 mon.c (mon.2) 801 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.405441+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.406087+0000 mon.c (mon.2) 802 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:00 smithi203 bash[23654]: audit 2024-03-11T21:33:00.406392+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:01.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.160 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: cluster 2024-03-11T21:32:59.340615+0000 mgr.y (mgr.14550) 400 : cluster 0 pgmap v555: 280 pgs: 8 unknown, 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.093534+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: cluster 2024-03-11T21:33:00.339839+0000 mon.a (mon.0) 2382 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.341710+0000 mon.a (mon.0) 2383 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: cluster 2024-03-11T21:33:00.346743+0000 mon.a (mon.0) 2384 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.405085+0000 mon.c (mon.2) 801 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.405441+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.406087+0000 mon.c (mon.2) 802 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[29284]: audit 2024-03-11T21:33:00.406392+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: cluster 2024-03-11T21:32:59.340615+0000 mgr.y (mgr.14550) 400 : cluster 0 pgmap v555: 280 pgs: 8 unknown, 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:01.161 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.093534+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: cluster 2024-03-11T21:33:00.339839+0000 mon.a (mon.0) 2382 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.341710+0000 mon.a (mon.0) 2383 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi143-71636-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: cluster 2024-03-11T21:33:00.346743+0000 mon.a (mon.0) 2384 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.405085+0000 mon.c (mon.2) 801 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.405441+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.406087+0000 mon.c (mon.2) 802 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:00 smithi143 bash[21512]: audit 2024-03-11T21:33:00.406392+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]: dispatch 2024-03-11T21:33:01.175 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:01.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:01.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.437 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:01.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:01.566 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:01.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:01.799 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:01.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:01.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[29284]: audit 2024-03-11T21:33:01.095036+0000 mon.c (mon.2) 803 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[29284]: audit 2024-03-11T21:33:01.290996+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]': finished 2024-03-11T21:33:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[29284]: cluster 2024-03-11T21:33:01.302063+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:33:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[29284]: audit 2024-03-11T21:33:01.308029+0000 mon.c (mon.2) 804 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[29284]: audit 2024-03-11T21:33:01.309184+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[21512]: audit 2024-03-11T21:33:01.095036+0000 mon.c (mon.2) 803 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[21512]: audit 2024-03-11T21:33:01.290996+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]': finished 2024-03-11T21:33:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[21512]: cluster 2024-03-11T21:33:01.302063+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:33:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[21512]: audit 2024-03-11T21:33:01.308029+0000 mon.c (mon.2) 804 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:01 smithi143 bash[21512]: audit 2024-03-11T21:33:01.309184+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:01.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:01 smithi203 bash[23654]: audit 2024-03-11T21:33:01.095036+0000 mon.c (mon.2) 803 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:01 smithi203 bash[23654]: audit 2024-03-11T21:33:01.290996+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-41"}]': finished 2024-03-11T21:33:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:01 smithi203 bash[23654]: cluster 2024-03-11T21:33:01.302063+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:33:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:01 smithi203 bash[23654]: audit 2024-03-11T21:33:01.308029+0000 mon.c (mon.2) 804 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:01 smithi203 bash[23654]: audit 2024-03-11T21:33:01.309184+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:02.085 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:02.186 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:02.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:02.213 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:02.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:02.327 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:02.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:02.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.468 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:02.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.599 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:02.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:02.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:02.829 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:02.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:02.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: cluster 2024-03-11T21:33:01.341387+0000 mgr.y (mgr.14550) 401 : cluster 0 pgmap v558: 272 pgs: 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: audit 2024-03-11T21:33:02.096828+0000 mon.c (mon.2) 805 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: audit 2024-03-11T21:33:02.293552+0000 mon.a (mon.0) 2390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: cluster 2024-03-11T21:33:02.300946+0000 mon.a (mon.0) 2391 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: audit 2024-03-11T21:33:02.307711+0000 mon.c (mon.2) 806 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[21512]: audit 2024-03-11T21:33:02.308664+0000 mon.a (mon.0) 2392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: cluster 2024-03-11T21:33:01.341387+0000 mgr.y (mgr.14550) 401 : cluster 0 pgmap v558: 272 pgs: 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:02.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: audit 2024-03-11T21:33:02.096828+0000 mon.c (mon.2) 805 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:02.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: audit 2024-03-11T21:33:02.293552+0000 mon.a (mon.0) 2390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:02.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: cluster 2024-03-11T21:33:02.300946+0000 mon.a (mon.0) 2391 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:33:02.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: audit 2024-03-11T21:33:02.307711+0000 mon.c (mon.2) 806 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:02 smithi143 bash[29284]: audit 2024-03-11T21:33:02.308664+0000 mon.a (mon.0) 2392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:02.980 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:03.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:03.095 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: cluster 2024-03-11T21:33:01.341387+0000 mgr.y (mgr.14550) 401 : cluster 0 pgmap v558: 272 pgs: 1 clean+premerge+peered, 271 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 127 B/s, 1 objects/s recovering 2024-03-11T21:33:03.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: audit 2024-03-11T21:33:02.096828+0000 mon.c (mon.2) 805 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:03.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: audit 2024-03-11T21:33:02.293552+0000 mon.a (mon.0) 2390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:03.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: cluster 2024-03-11T21:33:02.300946+0000 mon.a (mon.0) 2391 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:33:03.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: audit 2024-03-11T21:33:02.307711+0000 mon.c (mon.2) 806 : audit 1 from='client.? 172.21.15.143:0/2019422484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:03.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:02 smithi203 bash[23654]: audit 2024-03-11T21:33:02.308664+0000 mon.a (mon.0) 2392 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]: dispatch 2024-03-11T21:33:03.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:03.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:03.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:03.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:03.365 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:03.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:03.494 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:03.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:03.624 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:03.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.693+0000 7f14ac3cb700 1 -- 172.21.15.143:0/2937438950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 msgr2=0x7f14a406aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:03.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.693+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/2937438950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a406aac0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f14940099c0 tx=0x7f1494009cd0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/2937438950 shutdown_connections 2024-03-11T21:33:03.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/2937438950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a41071b0 0x7f14a410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/2937438950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a406aac0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/2937438950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a406b090 0x7f14a4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/2937438950 >> 172.21.15.143:0/2937438950 conn(0x7f14a4077a40 msgr2=0x7f14a4077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/2937438950 shutdown_connections 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/2937438950 wait complete. 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 librados: starting msgr at 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 librados: starting objecter 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 Processor -- start 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- start start 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 librados: setting wanted keys 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 librados: calling monclient init 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a41071b0 0x7f14a4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:03.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a4111650 con 0x7f14a406b090 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a41114d0 con 0x7f14a41071b0 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a41117d0 con 0x7f14a406a6e0 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a9966700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a9966700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33042/0 (socket says 172.21.15.143:33042) 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a9966700 1 -- 172.21.15.143:0/140007826 learned_addr learned my addr 172.21.15.143:0/140007826 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:03.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa968700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a41071b0 0x7f14a4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:03.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 -- 172.21.15.143:0/140007826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a41071b0 msgr2=0x7f14a4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:03.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a41071b0 0x7f14a4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 -- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 msgr2=0x7f14a419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:03.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 -- 172.21.15.143:0/140007826 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a41981b0 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a9966700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14aa167700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a419d890 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f14940099c0 tx=0x7f149400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 10 client.?.objecter ms_handle_connect 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1494009e60 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a4198440 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f149400fd70 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1494003800 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 10 client.41492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a4101000 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 librados: init done 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a404faa0 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.697+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f14940050d0 con 0x7f14a406a6e0 2024-03-11T21:33:03.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 0x7f148c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(382..382 src has 87..382) v4 ==== 9779+0+0 (secure 0 0 0) 0x7f149408f900 con 0x7f14a406a6e0 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 10 client.41492.objecter ms_dispatch 0x7f14a4001550 osd_map(382..382 src has 87..382) v4 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 3 client.41492.objecter handle_osd_map got epochs [382,382] > 0 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 3 client.41492.objecter handle_osd_map decoding full epoch 382 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a9966700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 0x7f148c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 20 client.41492.objecter dump_active .. 0 homeless 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a37fe700 10 client.41492.objecter ms_handle_connect 0x7f148c06f820 2024-03-11T21:33:03.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.701+0000 7f14a9966700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 0x7f148c071ce0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f14a4108690 tx=0x7f1498009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:03.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.705+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1494013070 con 0x7f14a406a6e0 2024-03-11T21:33:03.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:03.745 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:03.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:03.859 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.098499+0000 mon.c (mon.2) 807 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:03.859 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.297715+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:03.859 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: cluster 2024-03-11T21:33:03.308715+0000 mon.a (mon.0) 2394 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:33:03.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.310000+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:03.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.315452+0000 mon.a (mon.0) 2395 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:03.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.331810+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:03.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.333331+0000 mon.a (mon.0) 2397 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:03.860 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:03 smithi203 bash[23654]: audit 2024-03-11T21:33:03.333706+0000 mon.a (mon.0) 2398 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:03.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:03.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:03.925+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f14a4101320 con 0x7f14a406a6e0 2024-03-11T21:33:03.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:04.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:04.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3804385679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 msgr2=0x7fc9f01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3804385679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01011c0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00098f0 tx=0x7fc9e0009c00 comp rx=0 tx=0).stop 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3804385679 shutdown_connections 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3804385679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9f0106dc0 0x7fc9f01071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3804385679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9f0101700 0x7fc9f010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3804385679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01011c0 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3804385679 >> 172.21.15.143:0/3804385679 conn(0x7fc9f0076d80 msgr2=0x7fc9f0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3804385679 shutdown_connections 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3804385679 wait complete. 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 librados: starting msgr at 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 librados: starting objecter 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 Processor -- start 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- start start 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 librados: setting wanted keys 2024-03-11T21:33:04.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 librados: calling monclient init 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9f0101700 0x7fc9f0102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9f0106dc0 0x7fc9f0103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0110520 con 0x7fc9f0106dc0 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01103a0 con 0x7fc9f0101700 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01106a0 con 0x7fc9f0100d40 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:53388/0 (socket says 172.21.15.143:53388) 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 -- 172.21.15.143:0/3581361462 learned_addr learned my addr 172.21.15.143:0/3581361462 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9f0106dc0 0x7fc9f0103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9f0101700 0x7fc9f0102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9f0101700 msgr2=0x7fc9f0102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9f0101700 0x7fc9f0102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9f0106dc0 msgr2=0x7fc9f0103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9f0106dc0 0x7fc9f0103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f01a6130 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01029c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400ec10 tx=0x7fc9e400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9df7fe700 10 client.?.objecter ms_handle_connect 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9df7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400cd40 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc9e40044a0 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f01a6420 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4010430 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 10 client.41498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f01a69b0 con 0x7fc9f0100d40 2024-03-11T21:33:04.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 librados: init done 2024-03-11T21:33:04.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.121+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc9f004faa0 con 0x7fc9f0100d40 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc9e4004000 con 0x7fc9f0100d40 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc9d806f820 0x7fc9d8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(382..382 src has 87..382) v4 ==== 9779+0+0 (secure 0 0 0) 0x7fc9e408f1e0 con 0x7fc9f0100d40 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 osd_map(382..382 src has 87..382) v4 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 3 client.41498.objecter handle_osd_map got epochs [382,382] > 0 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 3 client.41498.objecter handle_osd_map decoding full epoch 382 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc9d806f820 0x7fc9d8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 20 client.41498.objecter dump_active .. 0 homeless 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v382) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc9e408f630 con 0x7fc9f0100d40 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 1 2024-03-11T21:33:04.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9d806f820 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc9d806f820 0x7fc9d8071ce0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0005bf0 tx=0x7fc9e0005b40 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc9f00629f0 0x7fc9f0062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 1 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 0 1 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 4 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 1 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01a9e00 con 0x7fc9f00629f0 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc9f00629f0 0x7fc9f0062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f00629f0 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc9f00629f0 0x7fc9f0062dd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01a9e00 con 0x7fc9f00629f0 2024-03-11T21:33:04.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:33:04.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 1 = system:0 2024-03-11T21:33:04.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 4 2024-03-11T21:33:04.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.125+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 0 1 2024-03-11T21:33:04.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:04.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 2 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 2 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f0103ef0 osd=0 3 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 0 2 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 4 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 2 2024-03-11T21:33:04.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f0062470 con 0x7fc9f00629f0 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f0062470 con 0x7fc9f00629f0 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 4 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 0 2 2024-03-11T21:33:04.122 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:33:04.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 3 2024-03-11T21:33:04.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc9f002c6b0 0x7fc9f002cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 1 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 4 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 3 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f010a840 con 0x7fc9f002c6b0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc9f002c6b0 0x7fc9f002cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f002c6b0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc9f002c6b0 0x7fc9f002cad0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f010a840 con 0x7fc9f002c6b0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 3 = system:0 2024-03-11T21:33:04.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 4 2024-03-11T21:33:04.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.145+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 1 3 2024-03-11T21:33:04.133 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 4 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 4 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f010a1c0 osd=1 3 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 1 4 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 4 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 4 2024-03-11T21:33:04.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f0109c40 con 0x7fc9f002c6b0 2024-03-11T21:33:04.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f0109c40 con 0x7fc9f002c6b0 2024-03-11T21:33:04.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 4 2024-03-11T21:33:04.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 1 4 2024-03-11T21:33:04.142 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 5 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc9f01abfc0 0x7fc9f01ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 5 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 2 5 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 4 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 5 2024-03-11T21:33:04.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01b1210 con 0x7fc9f01abfc0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc9f01abfc0 0x7fc9f01ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01abfc0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc9f01abfc0 0x7fc9f01ae3b0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01b1210 con 0x7fc9f01abfc0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 5 = system:0 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 4 2024-03-11T21:33:04.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.165+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 2 5 2024-03-11T21:33:04.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 6 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 6 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f010a040 osd=2 3 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 2 6 2024-03-11T21:33:04.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 4 2024-03-11T21:33:04.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 6 2024-03-11T21:33:04.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f002d260 con 0x7fc9f01abfc0 2024-03-11T21:33:04.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f002d260 con 0x7fc9f01abfc0 2024-03-11T21:33:04.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 4 2024-03-11T21:33:04.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 2 6 2024-03-11T21:33:04.164 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:33:04.164 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.164 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 7 2024-03-11T21:33:04.164 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc9f01a9960 0x7fc9f01afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 7 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 3 7 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 4 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 7 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f002d260 con 0x7fc9f01a9960 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc9f01a9960 0x7fc9f01afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc9f01a9960 0x7fc9f01afa70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01a9960 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f002d260 con 0x7fc9f01a9960 2024-03-11T21:33:04.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:33:04.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 7 = system:0 2024-03-11T21:33:04.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 4 2024-03-11T21:33:04.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.185+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 3 7 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 8 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 8 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f002dcf0 osd=3 3 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 3 8 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 4 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 8 2024-03-11T21:33:04.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f002d260 con 0x7fc9f01a9960 2024-03-11T21:33:04.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f002d260 con 0x7fc9f01a9960 2024-03-11T21:33:04.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 4 2024-03-11T21:33:04.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 3 8 2024-03-11T21:33:04.183 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:33:04.184 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.185 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 9 2024-03-11T21:33:04.185 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc9f01af5d0 0x7fc9f01b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.185 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 9 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 4 9 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 4 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 9 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01a9120 con 0x7fc9f01af5d0 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc9f01af5d0 0x7fc9f01b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc9f01af5d0 0x7fc9f01b53e0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.205+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01af5d0 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.098499+0000 mon.c (mon.2) 807 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.297715+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: cluster 2024-03-11T21:33:03.308715+0000 mon.a (mon.0) 2394 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.310000+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.315452+0000 mon.a (mon.0) 2395 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:04.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.331810+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.333331+0000 mon.a (mon.0) 2397 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[21512]: audit 2024-03-11T21:33:03.333706+0000 mon.a (mon.0) 2398 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.098499+0000 mon.c (mon.2) 807 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.297715+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi143-71604-57"}]': finished 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: cluster 2024-03-11T21:33:03.308715+0000 mon.a (mon.0) 2394 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.310000+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.315452+0000 mon.a (mon.0) 2395 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:04.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.331810+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:04.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.333331+0000 mon.a (mon.0) 2397 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:04.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:03 smithi143 bash[29284]: audit 2024-03-11T21:33:03.333706+0000 mon.a (mon.0) 2398 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:04.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.209+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01a9120 con 0x7fc9f01af5d0 2024-03-11T21:33:04.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.209+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:33:04.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.209+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 9 = system:0 2024-03-11T21:33:04.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.209+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 4 2024-03-11T21:33:04.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.209+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 4 9 2024-03-11T21:33:04.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 10 2024-03-11T21:33:04.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 10 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f002d260 osd=4 3 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 4 10 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 4 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 10 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.225+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f01a9120 con 0x7fc9f01af5d0 2024-03-11T21:33:04.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f01a9120 con 0x7fc9f01af5d0 2024-03-11T21:33:04.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 4 2024-03-11T21:33:04.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 4 10 2024-03-11T21:33:04.207 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:33:04.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 11 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc9f01b5e30 0x7fc9f01b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 11 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 5 11 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 4 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 11 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01bb220 con 0x7fc9f01b5e30 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc9f01b5e30 0x7fc9f01b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01b5e30 2024-03-11T21:33:04.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.229+0000 7fc9edd9b700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc9f01b5e30 0x7fc9f01b8220 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.233+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01bb220 con 0x7fc9f01b5e30 2024-03-11T21:33:04.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.233+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:33:04.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.233+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 11 = system:0 2024-03-11T21:33:04.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.233+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 4 2024-03-11T21:33:04.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.233+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 5 11 2024-03-11T21:33:04.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 12 2024-03-11T21:33:04.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 12 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01a9120 osd=5 3 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 5 12 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 4 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 12 2024-03-11T21:33:04.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.249+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f01b9fb0 con 0x7fc9f01b5e30 2024-03-11T21:33:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f01b9fb0 con 0x7fc9f01b5e30 2024-03-11T21:33:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 4 2024-03-11T21:33:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 5 12 2024-03-11T21:33:04.232 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:33:04.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 13 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc9f01ba6d0 0x7fc9f01bab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 13 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 6 13 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 4 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 13 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01b9a70 con 0x7fc9f01ba6d0 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.253+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc9f01ba6d0 0x7fc9f01bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01ba6d0 2024-03-11T21:33:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9eed9d700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc9f01ba6d0 0x7fc9f01bab30 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01b9a70 con 0x7fc9f01ba6d0 2024-03-11T21:33:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:33:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 13 = system:0 2024-03-11T21:33:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 4 2024-03-11T21:33:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.257+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 6 13 2024-03-11T21:33:04.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:04.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 14 2024-03-11T21:33:04.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 14 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01b9fb0 osd=6 3 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 6 14 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 4 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 14 2024-03-11T21:33:04.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.273+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f01b9a70 con 0x7fc9f01ba6d0 2024-03-11T21:33:04.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f01b9a70 con 0x7fc9f01ba6d0 2024-03-11T21:33:04.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 4 2024-03-11T21:33:04.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 6 14 2024-03-11T21:33:04.253 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 15 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc9f01b8d70 0x7fc9f01b91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 15 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 7 15 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 4 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 15 2024-03-11T21:33:04.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f01b9710 con 0x7fc9f01b8d70 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc9f01b8d70 0x7fc9f01b91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9ee59c700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc9f01b8d70 0x7fc9f01b91d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 10 client.41498.objecter ms_handle_connect 0x7fc9f01b8d70 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9f01b9710 con 0x7fc9f01b8d70 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:33:04.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 15 = system:0 2024-03-11T21:33:04.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 4 2024-03-11T21:33:04.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.277+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 7 15 2024-03-11T21:33:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:04.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 10 client.41498.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign -1 16 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 20 client.41498.objecter _get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_remove -1 16 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 20 client.41498.objecter get_session s=0x7fc9f01b9a70 osd=7 3 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 15 client.41498.objecter _session_command_op_assign 7 16 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 4 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 10 client.41498.objecter _send_command 16 2024-03-11T21:33:04.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.297+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9f01b9710 con 0x7fc9f01b8d70 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9df7fe700 1 -- 172.21.15.143:0/3581361462 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc9f01b9710 con 0x7fc9f01b8d70 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9df7fe700 10 client.41498.objecter ms_dispatch 0x7fc9f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9df7fe700 10 client.41498.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9df7fe700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 4 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9df7fe700 15 client.41498.objecter _session_command_op_remove 7 16 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:33:04.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 librados: watch_flush enter 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 librados: watch_flush exit 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.0 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc9f00629f0 msgr2=0x7fc9f0062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc9f00629f0 0x7fc9f0062dd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f0103ef0 osd=0 1 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.1 2024-03-11T21:33:04.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc9f002c6b0 msgr2=0x7fc9f002cad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc9f002c6b0 0x7fc9f002cad0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a1c0 osd=1 1 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.2 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc9f01abfc0 msgr2=0x7fc9f01ae3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc9f01abfc0 0x7fc9f01ae3b0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f010a040 osd=2 1 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.3 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc9f01a9960 msgr2=0x7fc9f01afa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc9f01a9960 0x7fc9f01afa70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002dcf0 osd=3 1 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.4 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc9f01af5d0 msgr2=0x7fc9f01b53e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc9f01af5d0 0x7fc9f01b53e0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f002d260 osd=4 1 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.5 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9eed9d700 1 -- 172.21.15.143:0/3581361462 reap_dead start 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc9f01b5e30 msgr2=0x7fc9f01b8220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc9f01b5e30 0x7fc9f01b8220 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9eed9d700 1 -- 172.21.15.143:0/3581361462 reap_dead start 2024-03-11T21:33:04.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9eed9d700 1 -- 172.21.15.143:0/3581361462 reap_dead start 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01a9120 osd=5 1 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.6 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc9f01ba6d0 msgr2=0x7fc9f01bab30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc9f01ba6d0 0x7fc9f01bab30 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9fb0 osd=6 1 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter close_session for osd.7 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc9f01b8d70 msgr2=0x7fc9f01b91d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc9f01b8d70 0x7fc9f01b91d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter put_session s=0x7fc9f01b9a70 osd=7 1 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 20 client.41498.objecter shutdown clearing up homeless session... 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 10 client.41498.objecter successfully canceled tick 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc9d806f820 msgr2=0x7fc9d8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc9d806f820 0x7fc9d8071ce0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0005bf0 tx=0x7fc9e0005b40 comp rx=0 tx=0).stop 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 msgr2=0x7fc9f01029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 --2- 172.21.15.143:0/3581361462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9f0100d40 0x7fc9f01029c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400ec10 tx=0x7fc9e400ef20 comp rx=0 tx=0).stop 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9eed9d700 1 -- 172.21.15.143:0/3581361462 reap_dead start 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 shutdown_connections 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 >> 172.21.15.143:0/3581361462 conn(0x7fc9f0076d80 msgr2=0x7fc9f0069b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 shutdown_connections 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 -- 172.21.15.143:0/3581361462 wait complete. 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.301+0000 7fc9f4bd7700 1 librados: shutdown 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:04.282 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:04.283 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:04.283 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:04.283 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:04.283 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:04.283 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:04.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.309+0000 7f14a37fe700 1 -- 172.21.15.143:0/140007826 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v383) v1 ==== 91+0+0 (secure 0 0 0) 0x7f1494060e20 con 0x7f14a406a6e0 2024-03-11T21:33:04.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 10 librados: watch_flush enter 2024-03-11T21:33:04.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 10 librados: watch_flush exit 2024-03-11T21:33:04.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 20 client.41492.objecter shutdown clearing up homeless session... 2024-03-11T21:33:04.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 10 client.41492.objecter successfully canceled tick 2024-03-11T21:33:04.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 msgr2=0x7f148c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 0x7f148c071ce0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f14a4108690 tx=0x7f1498009260 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 msgr2=0x7f14a419d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a419d890 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f14940099c0 tx=0x7f149400f690 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 shutdown_connections 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148c06f820 0x7f148c071ce0 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a41071b0 0x7f14a4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14a406a6e0 0x7f14a419d890 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 --2- 172.21.15.143:0/140007826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14a406b090 0x7f14a419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 >> 172.21.15.143:0/140007826 conn(0x7f14a4077a40 msgr2=0x7f14a40fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 shutdown_connections 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 -- 172.21.15.143:0/140007826 wait complete. 2024-03-11T21:33:04.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:04.317+0000 7f14ac3cb700 1 librados: shutdown 2024-03-11T21:33:04.297 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:33:04.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:04.383 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:04.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:04.512 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:04.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:04.643 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:04.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:04.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:04.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:04.899 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:05.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:05.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: cluster 2024-03-11T21:33:03.341853+0000 mgr.y (mgr.14550) 402 : cluster 0 pgmap v561: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:05.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:03.932054+0000 mon.c (mon.2) 809 : audit 1 from='client.? 172.21.15.143:0/140007826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:03.932420+0000 mon.a (mon.0) 2399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.100025+0000 mon.c (mon.2) 810 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.128138+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.143:0/3581361462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.298549+0000 mon.a (mon.0) 2400 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.305569+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.305677+0000 mon.a (mon.0) 2402 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.305750+0000 mon.a (mon.0) 2403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.312274+0000 mon.a (mon.0) 2404 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.314674+0000 mon.a (mon.0) 2405 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.316328+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.317298+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.014 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[29284]: audit 2024-03-11T21:33:04.318062+0000 mon.a (mon.0) 2406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: cluster 2024-03-11T21:33:03.341853+0000 mgr.y (mgr.14550) 402 : cluster 0 pgmap v561: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:03.932054+0000 mon.c (mon.2) 809 : audit 1 from='client.? 172.21.15.143:0/140007826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:03.932420+0000 mon.a (mon.0) 2399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.100025+0000 mon.c (mon.2) 810 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.128138+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.143:0/3581361462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.298549+0000 mon.a (mon.0) 2400 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.305569+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.305677+0000 mon.a (mon.0) 2402 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.305750+0000 mon.a (mon.0) 2403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.312274+0000 mon.a (mon.0) 2404 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.314674+0000 mon.a (mon.0) 2405 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:05.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.316328+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:05.016 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.317298+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.016 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:04 smithi143 bash[21512]: audit 2024-03-11T21:33:04.318062+0000 mon.a (mon.0) 2406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: cluster 2024-03-11T21:33:03.341853+0000 mgr.y (mgr.14550) 402 : cluster 0 pgmap v561: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:05.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:03.932054+0000 mon.c (mon.2) 809 : audit 1 from='client.? 172.21.15.143:0/140007826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:03.932420+0000 mon.a (mon.0) 2399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.100025+0000 mon.c (mon.2) 810 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.128138+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.143:0/3581361462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.298549+0000 mon.a (mon.0) 2400 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.305569+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.305677+0000 mon.a (mon.0) 2402 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi143-71604-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.305750+0000 mon.a (mon.0) 2403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.312274+0000 mon.a (mon.0) 2404 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.314674+0000 mon.a (mon.0) 2405 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.316328+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.317298+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:04 smithi203 bash[23654]: audit 2024-03-11T21:33:04.318062+0000 mon.a (mon.0) 2406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:05.030 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:05.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:05.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:05.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:05.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:05.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:05.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:05.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:05.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:05.736 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:33:05.737 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:33:05.737 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:33:05.737 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:33:05.773 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.798 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:05.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.658342+0000 osd.6 (osd.6) 173 : cluster 0 2.7 scrub starts 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.660200+0000 osd.6 (osd.6) 174 : cluster 0 2.7 scrub ok 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: audit 2024-03-11T21:33:05.101459+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: audit 2024-03-11T21:33:05.310907+0000 mon.a (mon.0) 2407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: audit 2024-03-11T21:33:05.320681+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.780259+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:33:05.912 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:05 smithi203 bash[23654]: audit 2024-03-11T21:33:05.780968+0000 mon.a (mon.0) 2409 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:05.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:05.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.658342+0000 osd.6 (osd.6) 173 : cluster 0 2.7 scrub starts 2024-03-11T21:33:05.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.660200+0000 osd.6 (osd.6) 174 : cluster 0 2.7 scrub ok 2024-03-11T21:33:05.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: audit 2024-03-11T21:33:05.101459+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: audit 2024-03-11T21:33:05.310907+0000 mon.a (mon.0) 2407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:05.987 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: audit 2024-03-11T21:33:05.320681+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.780259+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[21512]: audit 2024-03-11T21:33:05.780968+0000 mon.a (mon.0) 2409 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.658342+0000 osd.6 (osd.6) 173 : cluster 0 2.7 scrub starts 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.660200+0000 osd.6 (osd.6) 174 : cluster 0 2.7 scrub ok 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: audit 2024-03-11T21:33:05.101459+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: audit 2024-03-11T21:33:05.310907+0000 mon.a (mon.0) 2407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: audit 2024-03-11T21:33:05.320681+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.780259+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:33:05.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:05 smithi143 bash[29284]: audit 2024-03-11T21:33:05.780968+0000 mon.a (mon.0) 2409 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T21:33:06.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:06.057 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:06.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:06.187 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:06.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:06.323 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:06.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:06.451 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:06.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:33:06.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:06.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:06.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:06.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:06.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:06.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.946507+0000 osd.0 (osd.0) 219 : cluster 0 118.1a scrub starts 2024-03-11T21:33:06.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:04.948729+0000 osd.0 (osd.0) 220 : cluster 0 118.1a scrub ok 2024-03-11T21:33:06.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.342466+0000 mgr.y (mgr.14550) 403 : cluster 0 pgmap v564: 271 pgs: 4 creating+peering, 28 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:06.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.873721+0000 osd.1 (osd.1) 35 : cluster 0 6.1d scrub starts 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.875429+0000 osd.1 (osd.1) 36 : cluster 0 6.1d scrub ok 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.924672+0000 osd.0 (osd.0) 221 : cluster 0 118.19 scrub starts 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:05.927934+0000 osd.0 (osd.0) 222 : cluster 0 118.19 scrub ok 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.102721+0000 mon.c (mon.2) 816 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.215794+0000 mon.a (mon.0) 2410 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.218879+0000 mon.c (mon.2) 817 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:06.310900+0000 mon.a (mon.0) 2411 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.321076+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.321285+0000 mon.a (mon.0) 2413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: cluster 2024-03-11T21:33:06.329106+0000 mon.a (mon.0) 2414 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.331442+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:06.819 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[21512]: audit 2024-03-11T21:33:06.332323+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.946507+0000 osd.0 (osd.0) 219 : cluster 0 118.1a scrub starts 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:04.948729+0000 osd.0 (osd.0) 220 : cluster 0 118.1a scrub ok 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.342466+0000 mgr.y (mgr.14550) 403 : cluster 0 pgmap v564: 271 pgs: 4 creating+peering, 28 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.873721+0000 osd.1 (osd.1) 35 : cluster 0 6.1d scrub starts 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.875429+0000 osd.1 (osd.1) 36 : cluster 0 6.1d scrub ok 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.924672+0000 osd.0 (osd.0) 221 : cluster 0 118.19 scrub starts 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:05.927934+0000 osd.0 (osd.0) 222 : cluster 0 118.19 scrub ok 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.102721+0000 mon.c (mon.2) 816 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.215794+0000 mon.a (mon.0) 2410 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.218879+0000 mon.c (mon.2) 817 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:06.310900+0000 mon.a (mon.0) 2411 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.321076+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.321285+0000 mon.a (mon.0) 2413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T21:33:06.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: cluster 2024-03-11T21:33:06.329106+0000 mon.a (mon.0) 2414 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:33:06.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.331442+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:06.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:06 smithi143 bash[29284]: audit 2024-03-11T21:33:06.332323+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:06.831 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:06.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:06.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:07.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 -- 172.21.15.143:0/382494222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 msgr2=0x7f9de810db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:07.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/382494222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810db40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009fc0 tx=0x7f9de000b330 comp rx=0 tx=0).stop 2024-03-11T21:33:07.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 -- 172.21.15.143:0/382494222 shutdown_connections 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/382494222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810db40 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/382494222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de81072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/382494222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 0x7f9de8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 -- 172.21.15.143:0/382494222 >> 172.21.15.143:0/382494222 conn(0x7f9de8077a40 msgr2=0x7f9de8077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.033+0000 7f9dedbf5700 1 -- 172.21.15.143:0/382494222 shutdown_connections 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- 172.21.15.143:0/382494222 wait complete. 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 librados: starting msgr at 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 librados: starting objecter 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 Processor -- start 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- start start 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 librados: setting wanted keys 2024-03-11T21:33:07.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 librados: calling monclient init 2024-03-11T21:33:07.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 0x7f9de810e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81116e0 con 0x7f9de81019c0 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8111560 con 0x7f9de8101000 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8111860 con 0x7f9de8106ef0 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:53408/0 (socket says 172.21.15.143:53408) 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33084/0 (socket says 172.21.15.143:33084) 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 -- 172.21.15.143:0/2265773219 learned_addr learned my addr 172.21.15.143:0/2265773219 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de77fe700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 0x7f9de810e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 -- 172.21.15.143:0/2265773219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 msgr2=0x7f9de810e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:07.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 0x7f9de810e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 -- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 msgr2=0x7f9de810ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 -- 172.21.15.143:0/2265773219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de81a60a0 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de6ffd700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de7fff700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cce0 tx=0x7f9de0009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 10 client.?.objecter ms_handle_connect 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0021070 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de81a6330 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9de001d450 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00052a0 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 10 client.41504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de81a67e0 con 0x7f9de8106ef0 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 librados: init done 2024-03-11T21:33:07.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de804faa0 con 0x7f9de8106ef0 2024-03-11T21:33:07.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9de0024070 con 0x7f9de8106ef0 2024-03-11T21:33:07.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.037+0000 7f9de4ff9700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 0x7f9dd0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(385..385 src has 87..385) v4 ==== 10647+0+0 (secure 0 0 0) 0x7f9de000f020 con 0x7f9de8106ef0 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 10 client.41504.objecter ms_dispatch 0x7f9de8001550 osd_map(385..385 src has 87..385) v4 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 3 client.41504.objecter handle_osd_map got epochs [385,385] > 0 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 3 client.41504.objecter handle_osd_map decoding full epoch 385 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de77fe700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 0x7f9dd0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 20 client.41504.objecter dump_active .. 0 homeless 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de4ff9700 10 client.41504.objecter ms_handle_connect 0x7f9dd006f820 2024-03-11T21:33:07.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.041+0000 7f9de77fe700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 0x7f9dd0071ce0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc005fd0 tx=0x7f9ddc009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.946507+0000 osd.0 (osd.0) 219 : cluster 0 118.1a scrub starts 2024-03-11T21:33:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:04.948729+0000 osd.0 (osd.0) 220 : cluster 0 118.1a scrub ok 2024-03-11T21:33:07.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.342466+0000 mgr.y (mgr.14550) 403 : cluster 0 pgmap v564: 271 pgs: 4 creating+peering, 28 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.873721+0000 osd.1 (osd.1) 35 : cluster 0 6.1d scrub starts 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.875429+0000 osd.1 (osd.1) 36 : cluster 0 6.1d scrub ok 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.924672+0000 osd.0 (osd.0) 221 : cluster 0 118.19 scrub starts 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:05.927934+0000 osd.0 (osd.0) 222 : cluster 0 118.19 scrub ok 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.102721+0000 mon.c (mon.2) 816 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.215794+0000 mon.a (mon.0) 2410 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.218879+0000 mon.c (mon.2) 817 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:06.310900+0000 mon.a (mon.0) 2411 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.321076+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi143-71604-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.321285+0000 mon.a (mon.0) 2413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: cluster 2024-03-11T21:33:06.329106+0000 mon.a (mon.0) 2414 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:33:07.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.331442+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:07.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:06 smithi203 bash[23654]: audit 2024-03-11T21:33:06.332323+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:07.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.045+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9de002b020 con 0x7f9de8106ef0 2024-03-11T21:33:07.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:07.079 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:07.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:07.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:07.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.257+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9de810fbc0 con 0x7f9dd006f820 2024-03-11T21:33:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.285+0000 7f9de4ff9700 1 -- 172.21.15.143:0/2265773219 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+711087 (secure 0 0 0) 0x7f9de810fbc0 con 0x7f9dd006f820 2024-03-11T21:33:07.268 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:33:07.276 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 10 librados: watch_flush enter 2024-03-11T21:33:07.276 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 10 librados: watch_flush exit 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 20 client.41504.objecter shutdown clearing up homeless session... 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 10 client.41504.objecter successfully canceled tick 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 msgr2=0x7f9dd0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 0x7f9dd0071ce0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc005fd0 tx=0x7f9ddc009310 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 msgr2=0x7f9de810f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cce0 tx=0x7f9de0009bd0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 shutdown_connections 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9dd006f820 0x7f9dd0071ce0 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9de8106ef0 0x7f9de810f110 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9de8101000 0x7f9de810e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 --2- 172.21.15.143:0/2265773219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9de81019c0 0x7f9de810ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 >> 172.21.15.143:0/2265773219 conn(0x7f9de8077a40 msgr2=0x7f9de80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 shutdown_connections 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 -- 172.21.15.143:0/2265773219 wait complete. 2024-03-11T21:33:07.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:07.289+0000 7f9dedbf5700 1 librados: shutdown 2024-03-11T21:33:07.278 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:33:07.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:07.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.470 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:07.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:07.599 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:07.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:07.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:07.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: cluster 2024-03-11T21:33:06.707528+0000 osd.3 (osd.3) 229 : cluster 0 2.19 scrub starts 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: cluster 2024-03-11T21:33:06.709979+0000 osd.3 (osd.3) 230 : cluster 0 2.19 scrub ok 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: audit 2024-03-11T21:33:07.103633+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: audit 2024-03-11T21:33:07.323306+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.328184+0000 mon.a (mon.0) 2417 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: audit 2024-03-11T21:33:07.329524+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:07.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:07 smithi203 bash[23654]: audit 2024-03-11T21:33:07.330524+0000 mon.a (mon.0) 2418 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:07.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:07.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:07.986 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:07.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:08.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:08.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: cluster 2024-03-11T21:33:06.707528+0000 osd.3 (osd.3) 229 : cluster 0 2.19 scrub starts 2024-03-11T21:33:08.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: cluster 2024-03-11T21:33:06.709979+0000 osd.3 (osd.3) 230 : cluster 0 2.19 scrub ok 2024-03-11T21:33:08.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: audit 2024-03-11T21:33:07.103633+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:08.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: audit 2024-03-11T21:33:07.323306+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.328184+0000 mon.a (mon.0) 2417 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: audit 2024-03-11T21:33:07.329524+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[21512]: audit 2024-03-11T21:33:07.330524+0000 mon.a (mon.0) 2418 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: cluster 2024-03-11T21:33:06.707528+0000 osd.3 (osd.3) 229 : cluster 0 2.19 scrub starts 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: cluster 2024-03-11T21:33:06.709979+0000 osd.3 (osd.3) 230 : cluster 0 2.19 scrub ok 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: audit 2024-03-11T21:33:07.103633+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: audit 2024-03-11T21:33:07.323306+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.328184+0000 mon.a (mon.0) 2417 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: audit 2024-03-11T21:33:07.329524+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:08.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:07 smithi143 bash[29284]: audit 2024-03-11T21:33:07.330524+0000 mon.a (mon.0) 2418 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T21:33:08.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:08.247 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:08.288 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-03-11T21:33:06.323457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8793969,"num_objects":339,"num_object_clones":2,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":1633,"num_read_kb":1384,"num_write":1305,"num_write_kb":15184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1772808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2520,"ondisk_log_size":2520,"up":741,"acting":741,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":773,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":599704,"kb_used_data":75748,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749361512,"statfs":{"total":767960285184,"available":767346188288,"internally_reserved":0,"allocated":77565952,"data_stored":57463602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027976"},"pg_stats":[{"pgid":"266.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.1c","version":"0'0","reported_seq":13,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.319507+0000","last_change":"2024-03-11T21:33:04.329177+0000","last_active":"2024-03-11T21:33:05.319507+0000","last_peered":"2024-03-11T21:33:05.319507+0000","last_clean":"2024-03-11T21:33:05.319507+0000","last_became_active":"2024-03-11T21:33:04.328875+0000","last_became_peered":"2024-03-11T21:33:04.328875+0000","last_unstale":"2024-03-11T21:33:05.319507+0000","last_undegraded":"2024-03-11T21:33:05.319507+0000","last_fullsized":"2024-03-11T21:33:05.319507+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"266.1b","version":"0'0","reported_seq":13,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.317760+0000","last_change":"2024-03-11T21:33:04.337054+0000","last_active":"2024-03-11T21:33:05.317760+0000","last_peered":"2024-03-11T21:33:05.317760+0000","last_clean":"2024-03-11T21:33:05.317760+0000","last_became_active":"2024-03-11T21:33:04.336733+0000","last_became_peered":"2024-03-11T21:33:04.336733+0000","last_unstale":"2024-03-11T21:33:05.317760+0000","last_undegraded":"2024-03-11T21:33:05.317760+0000","last_fullsized":"2024-03-11T21:33:05.317760+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.868406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"266.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.19","version":"0'0","reported_seq":1,"reported_epoch":382,"state":"creating+peering","last_fresh":"2024-03-11T21:33:03.317608+0000","last_change":"2024-03-11T21:33:03.317608+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.317608+0000","last_undegraded":"2024-03-11T21:33:03.317608+0000","last_fullsized":"2024-03-11T21:33:03.317608+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"266.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.17","version":"0'0","reported_seq":1,"reported_epoch":382,"state":"creating+peering","last_fresh":"2024-03-11T21:33:03.316818+0000","last_change":"2024-03-11T21:33:03.316818+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.316818+0000","last_undegraded":"2024-03-11T21:33:03.316818+0000","last_fullsized":"2024-03-11T21:33:03.316818+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"266.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.11","version":"0'0","reported_seq":1,"reported_epoch":382,"state":"creating+peering","last_fresh":"2024-03-11T21:33:03.317110+0000","last_change":"2024-03-11T21:33:03.317110+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.317110+0000","last_undegraded":"2024-03-11T21:33:03.317110+0000","last_fullsized":"2024-03-11T21:33:03.317110+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"266.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.e","version":"0'0","reported_seq":13,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.324033+0000","last_change":"2024-03-11T21:33:04.338691+0000","last_active":"2024-03-11T21:33:05.324033+0000","last_peered":"2024-03-11T21:33:05.324033+0000","last_clean":"2024-03-11T21:33:05.324033+0000","last_became_active":"2024-03-11T21:33:04.338382+0000","last_became_peered":"2024-03-11T21:33:04.338382+0000","last_unstale":"2024-03-11T21:33:05.324033+0000","last_undegraded":"2024-03-11T21:33:05.324033+0000","last_fullsized":"2024-03-11T21:33:05.324033+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:08.507823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"266.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.7","version":"0'0","reported_seq":13,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.324472+0000","last_change":"2024-03-11T21:33:04.325585+0000","last_active":"2024-03-11T21:33:05.324472+0000","last_peered":"2024-03-11T21:33:05.324472+0000","last_clean":"2024-03-11T21:33:05.324472+0000","last_became_active":"2024-03-11T21:33:04.325357+0000","last_became_peered":"2024-03-11T21:33:04.325357+0000","last_unstale":"2024-03-11T21:33:05.324472+0000","last_undegraded":"2024-03-11T21:33:05.324472+0000","last_fullsized":"2024-03-11T21:33:05.324472+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:17.278745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"267.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.5","version":"0'0","reported_seq":1,"reported_epoch":382,"state":"creating+peering","last_fresh":"2024-03-11T21:33:03.317456+0000","last_change":"2024-03-11T21:33:03.317456+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.317456+0000","last_undegraded":"2024-03-11T21:33:03.317456+0000","last_fullsized":"2024-03-11T21:33:03.317456+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"267.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:06.311512+0000","last_change":"2024-03-11T21:33:06.311512+0000","last_active":"2024-03-11T21:33:06.311512+0000","last_peered":"2024-03-11T21:33:06.311512+0000","last_clean":"2024-03-11T21:33:06.311512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:06.311512+0000","last_undegraded":"2024-03-11T21:33:06.311512+0000","last_fullsized":"2024-03-11T21:33:06.311512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:06.311512+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.311512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"266.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:33:03.293568+0000","last_change":"2024-03-11T21:33:03.293568+0000","last_active":"2024-03-11T21:33:03.293568+0000","last_peered":"2024-03-11T21:33:03.293568+0000","last_clean":"2024-03-11T21:33:03.293568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:33:03.293568+0000","last_undegraded":"2024-03-11T21:33:03.293568+0000","last_fullsized":"2024-03-11T21:33:03.293568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:03.293568+0000","last_clean_scrub_stamp":"2024-03-11T21:33:03.293568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.9","version":"245'17","reported_seq":601,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299315+0000","last_change":"2024-03-11T21:32:34.298539+0000","last_active":"2024-03-11T21:33:02.299315+0000","last_peered":"2024-03-11T21:33:02.299315+0000","last_clean":"2024-03-11T21:33:02.299315+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:33:02.299315+0000","last_undegraded":"2024-03-11T21:33:02.299315+0000","last_fullsized":"2024-03-11T21:33:02.299315+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-03-11T21:32:02.895303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.895303+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:32.424863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001660573,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"332'18","reported_seq":593,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.762341+0000","last_change":"2024-03-11T21:32:34.299549+0000","last_active":"2024-03-11T21:33:02.762341+0000","last_peered":"2024-03-11T21:33:02.762341+0000","last_clean":"2024-03-11T21:33:02.762341+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:33:02.762341+0000","last_undegraded":"2024-03-11T21:33:02.762341+0000","last_fullsized":"2024-03-11T21:33:02.762341+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'18","last_scrub_stamp":"2024-03-11T21:32:22.035738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.035738+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.153674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076326579999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"296'34","reported_seq":606,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.332670+0000","last_change":"2024-03-11T21:32:34.335857+0000","last_active":"2024-03-11T21:33:02.332670+0000","last_peered":"2024-03-11T21:33:02.332670+0000","last_clean":"2024-03-11T21:33:02.332670+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:33:02.332670+0000","last_undegraded":"2024-03-11T21:33:02.332670+0000","last_fullsized":"2024-03-11T21:33:02.332670+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024159609999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"310'52","reported_seq":630,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297862+0000","last_change":"2024-03-11T21:32:34.308763+0000","last_active":"2024-03-11T21:33:02.297862+0000","last_peered":"2024-03-11T21:33:02.297862+0000","last_clean":"2024-03-11T21:33:02.297862+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:33:02.297862+0000","last_undegraded":"2024-03-11T21:33:02.297862+0000","last_fullsized":"2024-03-11T21:33:02.297862+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043833079999999998,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"245'51","reported_seq":627,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.314815+0000","last_change":"2024-03-11T21:32:34.306547+0000","last_active":"2024-03-11T21:33:02.314815+0000","last_peered":"2024-03-11T21:33:02.314815+0000","last_clean":"2024-03-11T21:33:02.314815+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:33:02.314815+0000","last_undegraded":"2024-03-11T21:33:02.314815+0000","last_fullsized":"2024-03-11T21:33:02.314815+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:31:05.766455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.766455+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022023799999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":597,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299602+0000","last_change":"2024-03-11T21:32:34.323770+0000","last_active":"2024-03-11T21:33:02.299602+0000","last_peered":"2024-03-11T21:33:02.299602+0000","last_clean":"2024-03-11T21:33:02.299602+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:33:02.299602+0000","last_undegraded":"2024-03-11T21:33:02.299602+0000","last_fullsized":"2024-03-11T21:33:02.299602+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036499739999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"353'183","reported_seq":723,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.304859+0000","last_change":"2024-03-11T21:32:36.339723+0000","last_active":"2024-03-11T21:33:02.304859+0000","last_peered":"2024-03-11T21:33:02.304859+0000","last_clean":"2024-03-11T21:33:02.304859+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:33:02.304859+0000","last_undegraded":"2024-03-11T21:33:02.304859+0000","last_fullsized":"2024-03-11T21:33:02.304859+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029727594,"stat_sum":{"num_bytes":117,"num_objects":13,"num_object_clones":1,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":55,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":666,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.772879+0000","last_change":"2024-03-11T21:32:34.312125+0000","last_active":"2024-03-11T21:33:02.772879+0000","last_peered":"2024-03-11T21:33:02.772879+0000","last_clean":"2024-03-11T21:33:02.772879+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:33:02.772879+0000","last_undegraded":"2024-03-11T21:33:02.772879+0000","last_fullsized":"2024-03-11T21:33:02.772879+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'102","last_scrub_stamp":"2024-03-11T21:31:02.926776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.926776+0000","objects_scrubbed":6,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013258202,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":588,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.313036+0000","last_change":"2024-03-11T21:32:34.353151+0000","last_active":"2024-03-11T21:33:03.313036+0000","last_peered":"2024-03-11T21:33:03.313036+0000","last_clean":"2024-03-11T21:33:03.313036+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:33:03.313036+0000","last_undegraded":"2024-03-11T21:33:03.313036+0000","last_fullsized":"2024-03-11T21:33:03.313036+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077514410000000004,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":626,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.311838+0000","last_change":"2024-03-11T21:32:34.352438+0000","last_active":"2024-03-11T21:33:03.311838+0000","last_peered":"2024-03-11T21:33:03.311838+0000","last_clean":"2024-03-11T21:33:03.311838+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:33:03.311838+0000","last_undegraded":"2024-03-11T21:33:03.311838+0000","last_fullsized":"2024-03-11T21:33:03.311838+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076282429999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"247'119","reported_seq":666,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.333709+0000","last_change":"2024-03-11T21:32:34.336215+0000","last_active":"2024-03-11T21:33:02.333709+0000","last_peered":"2024-03-11T21:33:02.333709+0000","last_clean":"2024-03-11T21:33:02.333709+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:33:02.333709+0000","last_undegraded":"2024-03-11T21:33:02.333709+0000","last_fullsized":"2024-03-11T21:33:02.333709+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026596910000000001,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"306'21","reported_seq":605,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299399+0000","last_change":"2024-03-11T21:32:34.309416+0000","last_active":"2024-03-11T21:33:02.299399+0000","last_peered":"2024-03-11T21:33:02.299399+0000","last_clean":"2024-03-11T21:33:02.299399+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:33:02.299399+0000","last_undegraded":"2024-03-11T21:33:02.299399+0000","last_fullsized":"2024-03-11T21:33:02.299399+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041327550000000001,"stat_sum":{"num_bytes":26,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"0'0","reported_seq":575,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.315090+0000","last_change":"2024-03-11T21:32:34.307693+0000","last_active":"2024-03-11T21:33:02.315090+0000","last_peered":"2024-03-11T21:33:02.315090+0000","last_clean":"2024-03-11T21:33:02.315090+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:33:02.315090+0000","last_undegraded":"2024-03-11T21:33:02.315090+0000","last_fullsized":"2024-03-11T21:33:02.315090+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.903210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.903210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.699206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002624487,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":605,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319253+0000","last_change":"2024-03-11T21:32:34.354250+0000","last_active":"2024-03-11T21:33:03.319253+0000","last_peered":"2024-03-11T21:33:03.319253+0000","last_clean":"2024-03-11T21:33:03.319253+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:33:03.319253+0000","last_undegraded":"2024-03-11T21:33:03.319253+0000","last_fullsized":"2024-03-11T21:33:03.319253+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080892789999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"311'39","reported_seq":599,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.349826+0000","last_change":"2024-03-11T21:32:34.320170+0000","last_active":"2024-03-11T21:33:02.349826+0000","last_peered":"2024-03-11T21:33:02.349826+0000","last_clean":"2024-03-11T21:33:02.349826+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:33:02.349826+0000","last_undegraded":"2024-03-11T21:33:02.349826+0000","last_fullsized":"2024-03-11T21:33:02.349826+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001055535,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":643,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.347350+0000","last_change":"2024-03-11T21:32:34.321070+0000","last_active":"2024-03-11T21:33:02.347350+0000","last_peered":"2024-03-11T21:33:02.347350+0000","last_clean":"2024-03-11T21:33:02.347350+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:33:02.347350+0000","last_undegraded":"2024-03-11T21:33:02.347350+0000","last_fullsized":"2024-03-11T21:33:02.347350+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789856,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":585,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.312723+0000","last_change":"2024-03-11T21:32:34.352506+0000","last_active":"2024-03-11T21:33:03.312723+0000","last_peered":"2024-03-11T21:33:03.312723+0000","last_clean":"2024-03-11T21:33:03.312723+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:33:03.312723+0000","last_undegraded":"2024-03-11T21:33:03.312723+0000","last_fullsized":"2024-03-11T21:33:03.312723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078384049999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"292'107","reported_seq":656,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.311437+0000","last_change":"2024-03-11T21:32:34.350993+0000","last_active":"2024-03-11T21:33:03.311437+0000","last_peered":"2024-03-11T21:33:03.311437+0000","last_clean":"2024-03-11T21:33:03.311437+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:33:03.311437+0000","last_undegraded":"2024-03-11T21:33:03.311437+0000","last_fullsized":"2024-03-11T21:33:03.311437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067123959999999998,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":476,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.317826+0000","last_change":"2024-03-11T21:32:34.301841+0000","last_active":"2024-03-11T21:33:05.317826+0000","last_peered":"2024-03-11T21:33:05.317826+0000","last_clean":"2024-03-11T21:33:05.317826+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:33:05.317826+0000","last_undegraded":"2024-03-11T21:33:05.317826+0000","last_fullsized":"2024-03-11T21:33:05.317826+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005553105,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":621,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.315449+0000","last_change":"2024-03-11T21:32:34.351742+0000","last_active":"2024-03-11T21:33:03.315449+0000","last_peered":"2024-03-11T21:33:03.315449+0000","last_clean":"2024-03-11T21:33:03.315449+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:33:03.315449+0000","last_undegraded":"2024-03-11T21:33:03.315449+0000","last_fullsized":"2024-03-11T21:33:03.315449+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071306190000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":659,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.768239+0000","last_change":"2024-03-11T21:32:34.326566+0000","last_active":"2024-03-11T21:33:02.768239+0000","last_peered":"2024-03-11T21:33:02.768239+0000","last_clean":"2024-03-11T21:33:02.768239+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:33:02.768239+0000","last_undegraded":"2024-03-11T21:33:02.768239+0000","last_fullsized":"2024-03-11T21:33:02.768239+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:32:07.554683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.554683+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:24.382804+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096827370000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"311'49","reported_seq":631,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.332663+0000","last_change":"2024-03-11T21:32:34.335870+0000","last_active":"2024-03-11T21:33:02.332663+0000","last_peered":"2024-03-11T21:33:02.332663+0000","last_clean":"2024-03-11T21:33:02.332663+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:33:02.332663+0000","last_undegraded":"2024-03-11T21:33:02.332663+0000","last_fullsized":"2024-03-11T21:33:02.332663+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023272950000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"296'34","reported_seq":488,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.324245+0000","last_change":"2024-03-11T21:32:34.311124+0000","last_active":"2024-03-11T21:33:05.324245+0000","last_peered":"2024-03-11T21:33:05.324245+0000","last_clean":"2024-03-11T21:33:05.324245+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:33:05.324245+0000","last_undegraded":"2024-03-11T21:33:05.324245+0000","last_fullsized":"2024-03-11T21:33:05.324245+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011782635,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"361'209","reported_seq":737,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.337787+0000","last_change":"2024-03-11T21:32:34.338193+0000","last_active":"2024-03-11T21:33:02.337787+0000","last_peered":"2024-03-11T21:33:02.337787+0000","last_clean":"2024-03-11T21:33:02.337787+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:33:02.337787+0000","last_undegraded":"2024-03-11T21:33:02.337787+0000","last_fullsized":"2024-03-11T21:33:02.337787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.575204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829950000000001,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"296'34","reported_seq":597,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.331313+0000","last_change":"2024-03-11T21:32:34.335400+0000","last_active":"2024-03-11T21:33:02.331313+0000","last_peered":"2024-03-11T21:33:02.331313+0000","last_clean":"2024-03-11T21:33:02.331313+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:33:02.331313+0000","last_undegraded":"2024-03-11T21:33:02.331313+0000","last_fullsized":"2024-03-11T21:33:02.331313+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024394040000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"324'14","reported_seq":640,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.762339+0000","last_change":"2024-03-11T21:32:34.298557+0000","last_active":"2024-03-11T21:33:02.762339+0000","last_peered":"2024-03-11T21:33:02.762339+0000","last_clean":"2024-03-11T21:33:02.762339+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:33:02.762339+0000","last_undegraded":"2024-03-11T21:33:02.762339+0000","last_fullsized":"2024-03-11T21:33:02.762339+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'1","last_scrub_stamp":"2024-03-11T21:31:05.888507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.888507+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0069698629999999998,"stat_sum":{"num_bytes":4194312,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"247'85","reported_seq":644,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.771140+0000","last_change":"2024-03-11T21:32:34.312524+0000","last_active":"2024-03-11T21:33:02.771140+0000","last_peered":"2024-03-11T21:33:02.771140+0000","last_clean":"2024-03-11T21:33:02.771140+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:33:02.771140+0000","last_undegraded":"2024-03-11T21:33:02.771140+0000","last_fullsized":"2024-03-11T21:33:02.771140+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'85","last_scrub_stamp":"2024-03-11T21:32:14.839185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.839185+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:24.613040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018288136999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"298'64","reported_seq":629,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.332194+0000","last_change":"2024-03-11T21:32:34.332276+0000","last_active":"2024-03-11T21:33:02.332194+0000","last_peered":"2024-03-11T21:33:02.332194+0000","last_clean":"2024-03-11T21:33:02.332194+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:33:02.332194+0000","last_undegraded":"2024-03-11T21:33:02.332194+0000","last_fullsized":"2024-03-11T21:33:02.332194+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016228219999999999,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.314510+0000","last_change":"2024-03-11T21:32:34.307434+0000","last_active":"2024-03-11T21:33:02.314510+0000","last_peered":"2024-03-11T21:33:02.314510+0000","last_clean":"2024-03-11T21:33:02.314510+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:33:02.314510+0000","last_undegraded":"2024-03-11T21:33:02.314510+0000","last_fullsized":"2024-03-11T21:33:02.314510+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.819204+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029485240000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":623,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299278+0000","last_change":"2024-03-11T21:32:34.315488+0000","last_active":"2024-03-11T21:33:02.299278+0000","last_peered":"2024-03-11T21:33:02.299278+0000","last_clean":"2024-03-11T21:33:02.299278+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:33:02.299278+0000","last_undegraded":"2024-03-11T21:33:02.299278+0000","last_fullsized":"2024-03-11T21:33:02.299278+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001496713,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"245'51","reported_seq":625,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.319742+0000","last_change":"2024-03-11T21:32:34.308970+0000","last_active":"2024-03-11T21:33:02.319742+0000","last_peered":"2024-03-11T21:33:02.319742+0000","last_clean":"2024-03-11T21:33:02.319742+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:33:02.319742+0000","last_undegraded":"2024-03-11T21:33:02.319742+0000","last_fullsized":"2024-03-11T21:33:02.319742+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:32:01.854464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.854464+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.086073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001987904,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"282'2","reported_seq":580,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.337895+0000","last_change":"2024-03-11T21:32:34.338218+0000","last_active":"2024-03-11T21:33:02.337895+0000","last_peered":"2024-03-11T21:33:02.337895+0000","last_clean":"2024-03-11T21:33:02.337895+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:33:02.337895+0000","last_undegraded":"2024-03-11T21:33:02.337895+0000","last_fullsized":"2024-03-11T21:33:02.337895+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011248230000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.341313+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:33:02.341313+0000","last_peered":"2024-03-11T21:33:02.341313+0000","last_clean":"2024-03-11T21:33:02.341313+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:33:02.341313+0000","last_undegraded":"2024-03-11T21:33:02.341313+0000","last_fullsized":"2024-03-11T21:33:02.341313+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.119355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.762085+0000","last_change":"2024-03-11T21:31:05.576659+0000","last_active":"2024-03-11T21:33:02.762085+0000","last_peered":"2024-03-11T21:33:02.762085+0000","last_clean":"2024-03-11T21:33:02.762085+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:33:02.762085+0000","last_undegraded":"2024-03-11T21:33:02.762085+0000","last_fullsized":"2024-03-11T21:33:02.762085+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.576558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.576558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":493,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.771395+0000","last_change":"2024-03-11T21:30:50.966683+0000","last_active":"2024-03-11T21:33:02.771395+0000","last_peered":"2024-03-11T21:33:02.771395+0000","last_clean":"2024-03-11T21:33:02.771395+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:33:02.771395+0000","last_undegraded":"2024-03-11T21:33:02.771395+0000","last_fullsized":"2024-03-11T21:33:02.771395+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:50.966582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:50.966582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":483,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.343422+0000","last_change":"2024-03-11T21:28:57.522457+0000","last_active":"2024-03-11T21:33:02.343422+0000","last_peered":"2024-03-11T21:33:02.343422+0000","last_clean":"2024-03-11T21:33:02.343422+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:33:02.343422+0000","last_undegraded":"2024-03-11T21:33:02.343422+0000","last_fullsized":"2024-03-11T21:33:02.343422+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.319776+0000","last_change":"2024-03-11T21:31:37.402407+0000","last_active":"2024-03-11T21:33:05.319776+0000","last_peered":"2024-03-11T21:33:05.319776+0000","last_clean":"2024-03-11T21:33:05.319776+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:33:05.319776+0000","last_undegraded":"2024-03-11T21:33:05.319776+0000","last_fullsized":"2024-03-11T21:33:05.319776+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.897147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.897147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.769919+0000","last_change":"2024-03-11T21:30:58.478329+0000","last_active":"2024-03-11T21:33:02.769919+0000","last_peered":"2024-03-11T21:33:02.769919+0000","last_clean":"2024-03-11T21:33:02.769919+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:33:02.769919+0000","last_undegraded":"2024-03-11T21:33:02.769919+0000","last_fullsized":"2024-03-11T21:33:02.769919+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.478183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.478183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":495,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.768236+0000","last_change":"2024-03-11T21:30:59.448512+0000","last_active":"2024-03-11T21:33:02.768236+0000","last_peered":"2024-03-11T21:33:02.768236+0000","last_clean":"2024-03-11T21:33:02.768236+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:33:02.768236+0000","last_undegraded":"2024-03-11T21:33:02.768236+0000","last_fullsized":"2024-03-11T21:33:02.768236+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.448405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":503,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.320259+0000","last_change":"2024-03-11T21:32:15.794130+0000","last_active":"2024-03-11T21:33:02.320259+0000","last_peered":"2024-03-11T21:33:02.320259+0000","last_clean":"2024-03-11T21:33:02.320259+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:33:02.320259+0000","last_undegraded":"2024-03-11T21:33:02.320259+0000","last_fullsized":"2024-03-11T21:33:02.320259+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.793910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.793910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.911787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":495,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299352+0000","last_change":"2024-03-11T21:30:54.963682+0000","last_active":"2024-03-11T21:33:02.299352+0000","last_peered":"2024-03-11T21:33:02.299352+0000","last_clean":"2024-03-11T21:33:02.299352+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:33:02.299352+0000","last_undegraded":"2024-03-11T21:33:02.299352+0000","last_fullsized":"2024-03-11T21:33:02.299352+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.963569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.963569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":491,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.341568+0000","last_change":"2024-03-11T21:31:01.656496+0000","last_active":"2024-03-11T21:33:02.341568+0000","last_peered":"2024-03-11T21:33:02.341568+0000","last_clean":"2024-03-11T21:33:02.341568+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:33:02.341568+0000","last_undegraded":"2024-03-11T21:33:02.341568+0000","last_fullsized":"2024-03-11T21:33:02.341568+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.656422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.656422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":496,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.318731+0000","last_change":"2024-03-11T21:30:51.770965+0000","last_active":"2024-03-11T21:33:03.318731+0000","last_peered":"2024-03-11T21:33:03.318731+0000","last_clean":"2024-03-11T21:33:03.318731+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:33:03.318731+0000","last_undegraded":"2024-03-11T21:33:03.318731+0000","last_fullsized":"2024-03-11T21:33:03.318731+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.770705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.770705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":493,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338506+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:33:02.338506+0000","last_peered":"2024-03-11T21:33:02.338506+0000","last_clean":"2024-03-11T21:33:02.338506+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:33:02.338506+0000","last_undegraded":"2024-03-11T21:33:02.338506+0000","last_fullsized":"2024-03-11T21:33:02.338506+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.448006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":496,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319146+0000","last_change":"2024-03-11T21:31:02.849078+0000","last_active":"2024-03-11T21:33:03.319146+0000","last_peered":"2024-03-11T21:33:03.319146+0000","last_clean":"2024-03-11T21:33:03.319146+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:33:03.319146+0000","last_undegraded":"2024-03-11T21:33:03.319146+0000","last_fullsized":"2024-03-11T21:33:03.319146+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.848978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.848978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":487,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.301097+0000","last_change":"2024-03-11T21:28:53.826473+0000","last_active":"2024-03-11T21:33:02.301097+0000","last_peered":"2024-03-11T21:33:02.301097+0000","last_clean":"2024-03-11T21:33:02.301097+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:33:02.301097+0000","last_undegraded":"2024-03-11T21:33:02.301097+0000","last_fullsized":"2024-03-11T21:33:02.301097+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.312690+0000","last_change":"2024-03-11T21:31:00.856017+0000","last_active":"2024-03-11T21:33:02.312690+0000","last_peered":"2024-03-11T21:33:02.312690+0000","last_clean":"2024-03-11T21:33:02.312690+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:33:02.312690+0000","last_undegraded":"2024-03-11T21:33:02.312690+0000","last_fullsized":"2024-03-11T21:33:02.312690+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.855899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.855899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133629+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":491,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.350040+0000","last_change":"2024-03-11T21:32:02.604095+0000","last_active":"2024-03-11T21:33:02.350040+0000","last_peered":"2024-03-11T21:33:02.350040+0000","last_clean":"2024-03-11T21:33:02.350040+0000","last_became_active":"2024-03-11T21:28:57.548665+0000","last_became_peered":"2024-03-11T21:28:57.548665+0000","last_unstale":"2024-03-11T21:33:02.350040+0000","last_undegraded":"2024-03-11T21:33:02.350040+0000","last_fullsized":"2024-03-11T21:33:02.350040+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.603946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.603946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.775791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.763290+0000","last_change":"2024-03-11T21:30:57.522881+0000","last_active":"2024-03-11T21:33:02.763290+0000","last_peered":"2024-03-11T21:33:02.763290+0000","last_clean":"2024-03-11T21:33:02.763290+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:33:02.763290+0000","last_undegraded":"2024-03-11T21:33:02.763290+0000","last_fullsized":"2024-03-11T21:33:02.763290+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:57.522689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.522689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":493,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330609+0000","last_change":"2024-03-11T21:32:04.563626+0000","last_active":"2024-03-11T21:33:02.330609+0000","last_peered":"2024-03-11T21:33:02.330609+0000","last_clean":"2024-03-11T21:33:02.330609+0000","last_became_active":"2024-03-11T21:28:53.852502+0000","last_became_peered":"2024-03-11T21:28:53.852502+0000","last_unstale":"2024-03-11T21:33:02.330609+0000","last_undegraded":"2024-03-11T21:33:02.330609+0000","last_fullsized":"2024-03-11T21:33:02.330609+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:34.350718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":483,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342672+0000","last_change":"2024-03-11T21:28:57.495142+0000","last_active":"2024-03-11T21:33:02.342672+0000","last_peered":"2024-03-11T21:33:02.342672+0000","last_clean":"2024-03-11T21:33:02.342672+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:33:02.342672+0000","last_undegraded":"2024-03-11T21:33:02.342672+0000","last_fullsized":"2024-03-11T21:33:02.342672+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":495,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.298191+0000","last_change":"2024-03-11T21:32:14.693680+0000","last_active":"2024-03-11T21:33:02.298191+0000","last_peered":"2024-03-11T21:33:02.298191+0000","last_clean":"2024-03-11T21:33:02.298191+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:33:02.298191+0000","last_undegraded":"2024-03-11T21:33:02.298191+0000","last_fullsized":"2024-03-11T21:33:02.298191+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.693586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.693586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.646372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":374,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.320411+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:33:05.320411+0000","last_peered":"2024-03-11T21:33:05.320411+0000","last_clean":"2024-03-11T21:33:05.320411+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:33:05.320411+0000","last_undegraded":"2024-03-11T21:33:05.320411+0000","last_fullsized":"2024-03-11T21:33:05.320411+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.546676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":491,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.345102+0000","last_change":"2024-03-11T21:32:07.649024+0000","last_active":"2024-03-11T21:33:02.345102+0000","last_peered":"2024-03-11T21:33:02.345102+0000","last_clean":"2024-03-11T21:33:02.345102+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:33:02.345102+0000","last_undegraded":"2024-03-11T21:33:02.345102+0000","last_fullsized":"2024-03-11T21:33:02.345102+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.648800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.648800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.940650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":491,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.347650+0000","last_change":"2024-03-11T21:32:15.722549+0000","last_active":"2024-03-11T21:33:02.347650+0000","last_peered":"2024-03-11T21:33:02.347650+0000","last_clean":"2024-03-11T21:33:02.347650+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:33:02.347650+0000","last_undegraded":"2024-03-11T21:33:02.347650+0000","last_fullsized":"2024-03-11T21:33:02.347650+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.722443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.722443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.727090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":496,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320434+0000","last_change":"2024-03-11T21:32:14.976068+0000","last_active":"2024-03-11T21:33:03.320434+0000","last_peered":"2024-03-11T21:33:03.320434+0000","last_clean":"2024-03-11T21:33:03.320434+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:33:03.320434+0000","last_undegraded":"2024-03-11T21:33:03.320434+0000","last_fullsized":"2024-03-11T21:33:03.320434+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.975966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.975966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:40.129592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.763418+0000","last_change":"2024-03-11T21:30:56.975013+0000","last_active":"2024-03-11T21:33:02.763418+0000","last_peered":"2024-03-11T21:33:02.763418+0000","last_clean":"2024-03-11T21:33:02.763418+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:33:02.763418+0000","last_undegraded":"2024-03-11T21:33:02.763418+0000","last_fullsized":"2024-03-11T21:33:02.763418+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:56.974794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.974794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":486,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.331007+0000","last_change":"2024-03-11T21:28:53.852772+0000","last_active":"2024-03-11T21:33:02.331007+0000","last_peered":"2024-03-11T21:33:02.331007+0000","last_clean":"2024-03-11T21:33:02.331007+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:33:02.331007+0000","last_undegraded":"2024-03-11T21:33:02.331007+0000","last_fullsized":"2024-03-11T21:33:02.331007+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.574377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":492,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.345844+0000","last_change":"2024-03-11T21:30:55.794089+0000","last_active":"2024-03-11T21:33:02.345844+0000","last_peered":"2024-03-11T21:33:02.345844+0000","last_clean":"2024-03-11T21:33:02.345844+0000","last_became_active":"2024-03-11T21:28:57.523449+0000","last_became_peered":"2024-03-11T21:28:57.523449+0000","last_unstale":"2024-03-11T21:33:02.345844+0000","last_undegraded":"2024-03-11T21:33:02.345844+0000","last_fullsized":"2024-03-11T21:33:02.345844+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.793796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.793796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":488,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.298414+0000","last_change":"2024-03-11T21:28:53.827705+0000","last_active":"2024-03-11T21:33:02.298414+0000","last_peered":"2024-03-11T21:33:02.298414+0000","last_clean":"2024-03-11T21:33:02.298414+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:33:02.298414+0000","last_undegraded":"2024-03-11T21:33:02.298414+0000","last_fullsized":"2024-03-11T21:33:02.298414+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:28:49.756221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":497,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319421+0000","last_change":"2024-03-11T21:30:59.897121+0000","last_active":"2024-03-11T21:33:03.319421+0000","last_peered":"2024-03-11T21:33:03.319421+0000","last_clean":"2024-03-11T21:33:03.319421+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:33:03.319421+0000","last_undegraded":"2024-03-11T21:33:03.319421+0000","last_fullsized":"2024-03-11T21:33:03.319421+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.897032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.897032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.335360+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:33:02.335360+0000","last_peered":"2024-03-11T21:33:02.335360+0000","last_clean":"2024-03-11T21:33:02.335360+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:33:02.335360+0000","last_undegraded":"2024-03-11T21:33:02.335360+0000","last_fullsized":"2024-03-11T21:33:02.335360+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:45.699273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.313074+0000","last_change":"2024-03-11T21:30:24.759417+0000","last_active":"2024-03-11T21:33:02.313074+0000","last_peered":"2024-03-11T21:33:02.313074+0000","last_clean":"2024-03-11T21:33:02.313074+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:33:02.313074+0000","last_undegraded":"2024-03-11T21:33:02.313074+0000","last_fullsized":"2024-03-11T21:33:02.313074+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.759285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.759285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":374,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.324603+0000","last_change":"2024-03-11T21:32:28.978945+0000","last_active":"2024-03-11T21:33:05.324603+0000","last_peered":"2024-03-11T21:33:05.324603+0000","last_clean":"2024-03-11T21:33:05.324603+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:33:05.324603+0000","last_undegraded":"2024-03-11T21:33:05.324603+0000","last_fullsized":"2024-03-11T21:33:05.324603+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:31.470081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"259'7","reported_seq":562,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.298234+0000","last_change":"2024-03-11T21:32:43.306060+0000","last_active":"2024-03-11T21:33:02.298234+0000","last_peered":"2024-03-11T21:33:02.298234+0000","last_clean":"2024-03-11T21:33:02.298234+0000","last_became_active":"2024-03-11T21:32:43.305002+0000","last_became_peered":"2024-03-11T21:32:43.305002+0000","last_unstale":"2024-03-11T21:33:02.298234+0000","last_undegraded":"2024-03-11T21:33:02.298234+0000","last_fullsized":"2024-03-11T21:33:02.298234+0000","mapping_epoch":139,"log_start":"259'7","ondisk_log_start":"259'7","created":139,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:47.897070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.897070+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":7,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":503,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320288+0000","last_change":"2024-03-11T21:30:42.847714+0000","last_active":"2024-03-11T21:33:03.320288+0000","last_peered":"2024-03-11T21:33:03.320288+0000","last_clean":"2024-03-11T21:33:03.320288+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:33:03.320288+0000","last_undegraded":"2024-03-11T21:33:03.320288+0000","last_fullsized":"2024-03-11T21:33:03.320288+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.847476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.847476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":775,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.331951+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:33:02.331951+0000","last_peered":"2024-03-11T21:33:02.331951+0000","last_clean":"2024-03-11T21:33:02.331951+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:33:02.331951+0000","last_undegraded":"2024-03-11T21:33:02.331951+0000","last_fullsized":"2024-03-11T21:33:02.331951+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":674,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342689+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:33:02.342689+0000","last_peered":"2024-03-11T21:33:02.342689+0000","last_clean":"2024-03-11T21:33:02.342689+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:33:02.342689+0000","last_undegraded":"2024-03-11T21:33:02.342689+0000","last_fullsized":"2024-03-11T21:33:02.342689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.313586+0000","last_change":"2024-03-11T21:30:55.823529+0000","last_active":"2024-03-11T21:33:02.313586+0000","last_peered":"2024-03-11T21:33:02.313586+0000","last_clean":"2024-03-11T21:33:02.313586+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:33:02.313586+0000","last_undegraded":"2024-03-11T21:33:02.313586+0000","last_fullsized":"2024-03-11T21:33:02.313586+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.823180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.823180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":650,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.772131+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:33:02.772131+0000","last_peered":"2024-03-11T21:33:02.772131+0000","last_clean":"2024-03-11T21:33:02.772131+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:33:02.772131+0000","last_undegraded":"2024-03-11T21:33:02.772131+0000","last_fullsized":"2024-03-11T21:33:02.772131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.496671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"259'16","reported_seq":573,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.335635+0000","last_change":"2024-03-11T21:33:03.335635+0000","last_active":"2024-03-11T21:33:03.335635+0000","last_peered":"2024-03-11T21:33:03.335635+0000","last_clean":"2024-03-11T21:33:03.335635+0000","last_became_active":"2024-03-11T21:33:03.335341+0000","last_became_peered":"2024-03-11T21:33:03.335341+0000","last_unstale":"2024-03-11T21:33:03.335635+0000","last_undegraded":"2024-03-11T21:33:03.335635+0000","last_fullsized":"2024-03-11T21:33:03.335635+0000","mapping_epoch":139,"log_start":"259'16","ondisk_log_start":"259'16","created":139,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:39.934432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:39.934432+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":36,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":509,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.777613+0000","last_change":"2024-03-11T21:31:51.834491+0000","last_active":"2024-03-11T21:33:02.777613+0000","last_peered":"2024-03-11T21:33:02.777613+0000","last_clean":"2024-03-11T21:33:02.777613+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:33:02.777613+0000","last_undegraded":"2024-03-11T21:33:02.777613+0000","last_fullsized":"2024-03-11T21:33:02.777613+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.834251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.834251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.072172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":673,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773400+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:33:02.773400+0000","last_peered":"2024-03-11T21:33:02.773400+0000","last_clean":"2024-03-11T21:33:02.773400+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:33:02.773400+0000","last_undegraded":"2024-03-11T21:33:02.773400+0000","last_fullsized":"2024-03-11T21:33:02.773400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.006264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":556,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.322632+0000","last_change":"2024-03-11T21:32:06.996760+0000","last_active":"2024-03-11T21:33:05.322632+0000","last_peered":"2024-03-11T21:33:05.322632+0000","last_clean":"2024-03-11T21:33:05.322632+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:33:05.322632+0000","last_undegraded":"2024-03-11T21:33:05.322632+0000","last_fullsized":"2024-03-11T21:33:05.322632+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.996518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.996518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.387438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.304942+0000","last_change":"2024-03-11T21:32:28.781453+0000","last_active":"2024-03-11T21:33:02.304942+0000","last_peered":"2024-03-11T21:33:02.304942+0000","last_clean":"2024-03-11T21:33:02.304942+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:33:02.304942+0000","last_undegraded":"2024-03-11T21:33:02.304942+0000","last_fullsized":"2024-03-11T21:33:02.304942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.324762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":653,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.317337+0000","last_change":"2024-03-11T21:32:25.048822+0000","last_active":"2024-03-11T21:33:03.317337+0000","last_peered":"2024-03-11T21:33:03.317337+0000","last_clean":"2024-03-11T21:33:03.317337+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:33:03.317337+0000","last_undegraded":"2024-03-11T21:33:03.317337+0000","last_fullsized":"2024-03-11T21:33:03.317337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.048696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.048696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:46.961096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":555,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300611+0000","last_change":"2024-03-11T21:32:35.401156+0000","last_active":"2024-03-11T21:33:02.300611+0000","last_peered":"2024-03-11T21:33:02.300611+0000","last_clean":"2024-03-11T21:33:02.300611+0000","last_became_active":"2024-03-11T21:32:35.400680+0000","last_became_peered":"2024-03-11T21:32:35.400680+0000","last_unstale":"2024-03-11T21:33:02.300611+0000","last_undegraded":"2024-03-11T21:33:02.300611+0000","last_fullsized":"2024-03-11T21:33:02.300611+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.918563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.918563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.779718+0000","last_change":"2024-03-11T21:30:51.971724+0000","last_active":"2024-03-11T21:33:02.779718+0000","last_peered":"2024-03-11T21:33:02.779718+0000","last_clean":"2024-03-11T21:33:02.779718+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:33:02.779718+0000","last_undegraded":"2024-03-11T21:33:02.779718+0000","last_fullsized":"2024-03-11T21:33:02.779718+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.971619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.971619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"106'25","reported_seq":685,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338474+0000","last_change":"2024-03-11T21:29:08.366566+0000","last_active":"2024-03-11T21:33:02.338474+0000","last_peered":"2024-03-11T21:33:02.338474+0000","last_clean":"2024-03-11T21:33:02.338474+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:33:02.338474+0000","last_undegraded":"2024-03-11T21:33:02.338474+0000","last_fullsized":"2024-03-11T21:33:02.338474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'25","last_scrub_stamp":"2024-03-11T21:29:08.366434+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:29:08.366434+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.312842+0000","last_change":"2024-03-11T21:30:50.697950+0000","last_active":"2024-03-11T21:33:02.312842+0000","last_peered":"2024-03-11T21:33:02.312842+0000","last_clean":"2024-03-11T21:33:02.312842+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:33:02.312842+0000","last_undegraded":"2024-03-11T21:33:02.312842+0000","last_fullsized":"2024-03-11T21:33:02.312842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:30:50.697855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.311803+0000","last_change":"2024-03-11T21:32:16.782474+0000","last_active":"2024-03-11T21:33:02.311803+0000","last_peered":"2024-03-11T21:33:02.311803+0000","last_clean":"2024-03-11T21:33:02.311803+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:33:02.311803+0000","last_undegraded":"2024-03-11T21:33:02.311803+0000","last_fullsized":"2024-03-11T21:33:02.311803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.782400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.782400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.897094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330343+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:33:02.330343+0000","last_peered":"2024-03-11T21:33:02.330343+0000","last_clean":"2024-03-11T21:33:02.330343+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:33:02.330343+0000","last_undegraded":"2024-03-11T21:33:02.330343+0000","last_fullsized":"2024-03-11T21:33:02.330343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.5","version":"259'9","reported_seq":582,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.768747+0000","last_change":"2024-03-11T21:32:55.359693+0000","last_active":"2024-03-11T21:33:02.768747+0000","last_peered":"2024-03-11T21:33:02.768747+0000","last_clean":"2024-03-11T21:33:02.768747+0000","last_became_active":"2024-03-11T21:32:55.358543+0000","last_became_peered":"2024-03-11T21:32:55.358543+0000","last_unstale":"2024-03-11T21:33:02.768747+0000","last_undegraded":"2024-03-11T21:33:02.768747+0000","last_fullsized":"2024-03-11T21:33:02.768747+0000","mapping_epoch":295,"log_start":"259'9","ondisk_log_start":"259'9","created":139,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.705939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:42.546466+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.705939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:45.141926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":375,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318089+0000","last_change":"2024-03-11T21:31:55.095325+0000","last_active":"2024-03-11T21:33:05.318089+0000","last_peered":"2024-03-11T21:33:05.318089+0000","last_clean":"2024-03-11T21:33:05.318089+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:33:05.318089+0000","last_undegraded":"2024-03-11T21:33:05.318089+0000","last_fullsized":"2024-03-11T21:33:05.318089+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.095111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.095111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"104'28","reported_seq":707,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.765181+0000","last_change":"2024-03-11T21:31:54.594364+0000","last_active":"2024-03-11T21:33:02.765181+0000","last_peered":"2024-03-11T21:33:02.765181+0000","last_clean":"2024-03-11T21:33:02.765181+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:33:02.765181+0000","last_undegraded":"2024-03-11T21:33:02.765181+0000","last_fullsized":"2024-03-11T21:33:02.765181+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:31:54.594116+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.594116+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334912+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:33:02.334912+0000","last_peered":"2024-03-11T21:33:02.334912+0000","last_clean":"2024-03-11T21:33:02.334912+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:33:02.334912+0000","last_undegraded":"2024-03-11T21:33:02.334912+0000","last_fullsized":"2024-03-11T21:33:02.334912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":554,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.320194+0000","last_change":"2024-03-11T21:31:37.399161+0000","last_active":"2024-03-11T21:33:05.320194+0000","last_peered":"2024-03-11T21:33:05.320194+0000","last_clean":"2024-03-11T21:33:05.320194+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:33:05.320194+0000","last_undegraded":"2024-03-11T21:33:05.320194+0000","last_fullsized":"2024-03-11T21:33:05.320194+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:14.307354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.307354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":658,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.320733+0000","last_change":"2024-03-11T21:32:05.876383+0000","last_active":"2024-03-11T21:33:02.320733+0000","last_peered":"2024-03-11T21:33:02.320733+0000","last_clean":"2024-03-11T21:33:02.320733+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:33:02.320733+0000","last_undegraded":"2024-03-11T21:33:02.320733+0000","last_fullsized":"2024-03-11T21:33:02.320733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.876018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.876018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.769278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"142'2","reported_seq":573,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.766582+0000","last_change":"2024-03-11T21:32:59.350452+0000","last_active":"2024-03-11T21:33:02.766582+0000","last_peered":"2024-03-11T21:33:02.766582+0000","last_clean":"2024-03-11T21:33:02.766582+0000","last_became_active":"2024-03-11T21:32:59.349766+0000","last_became_peered":"2024-03-11T21:32:59.349766+0000","last_unstale":"2024-03-11T21:33:02.766582+0000","last_undegraded":"2024-03-11T21:33:02.766582+0000","last_fullsized":"2024-03-11T21:33:02.766582+0000","mapping_epoch":295,"log_start":"142'2","ondisk_log_start":"142'2","created":139,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.545525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.545525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":2,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.639679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":515,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342090+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:33:02.342090+0000","last_peered":"2024-03-11T21:33:02.342090+0000","last_clean":"2024-03-11T21:33:02.342090+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:33:02.342090+0000","last_undegraded":"2024-03-11T21:33:02.342090+0000","last_fullsized":"2024-03-11T21:33:02.342090+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.202895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":674,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.344234+0000","last_change":"2024-03-11T21:32:28.904947+0000","last_active":"2024-03-11T21:33:02.344234+0000","last_peered":"2024-03-11T21:33:02.344234+0000","last_clean":"2024-03-11T21:33:02.344234+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:33:02.344234+0000","last_undegraded":"2024-03-11T21:33:02.344234+0000","last_fullsized":"2024-03-11T21:33:02.344234+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:32:28.904821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.904821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.529402+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":522,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.319333+0000","last_change":"2024-03-11T21:32:04.059244+0000","last_active":"2024-03-11T21:33:05.319333+0000","last_peered":"2024-03-11T21:33:05.319333+0000","last_clean":"2024-03-11T21:33:05.319333+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:33:05.319333+0000","last_undegraded":"2024-03-11T21:33:05.319333+0000","last_fullsized":"2024-03-11T21:33:05.319333+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.059137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:17.202132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":793,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.315701+0000","last_change":"2024-03-11T21:32:07.803946+0000","last_active":"2024-03-11T21:33:02.315701+0000","last_peered":"2024-03-11T21:33:02.315701+0000","last_clean":"2024-03-11T21:33:02.315701+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:33:02.315701+0000","last_undegraded":"2024-03-11T21:33:02.315701+0000","last_fullsized":"2024-03-11T21:33:02.315701+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:32:07.803828+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.803828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.048925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":660,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.303944+0000","last_change":"2024-03-11T21:32:08.744653+0000","last_active":"2024-03-11T21:33:02.303944+0000","last_peered":"2024-03-11T21:33:02.303944+0000","last_clean":"2024-03-11T21:33:02.303944+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:33:02.303944+0000","last_undegraded":"2024-03-11T21:33:02.303944+0000","last_fullsized":"2024-03-11T21:33:02.303944+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.744422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.744422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.054417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"259'7","reported_seq":471,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.323745+0000","last_change":"2024-03-11T21:32:47.336412+0000","last_active":"2024-03-11T21:33:05.323745+0000","last_peered":"2024-03-11T21:33:05.323745+0000","last_clean":"2024-03-11T21:33:05.323745+0000","last_became_active":"2024-03-11T21:32:47.335861+0000","last_became_peered":"2024-03-11T21:32:47.335861+0000","last_unstale":"2024-03-11T21:33:05.323745+0000","last_undegraded":"2024-03-11T21:33:05.323745+0000","last_fullsized":"2024-03-11T21:33:05.323745+0000","mapping_epoch":295,"log_start":"259'7","ondisk_log_start":"259'7","created":139,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.866571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:42.866571+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.866571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":14,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":510,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.346156+0000","last_change":"2024-03-11T21:31:37.390219+0000","last_active":"2024-03-11T21:33:02.346156+0000","last_peered":"2024-03-11T21:33:02.346156+0000","last_clean":"2024-03-11T21:33:02.346156+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:33:02.346156+0000","last_undegraded":"2024-03-11T21:33:02.346156+0000","last_fullsized":"2024-03-11T21:33:02.346156+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.818577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.818577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":684,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.341928+0000","last_change":"2024-03-11T21:31:37.390418+0000","last_active":"2024-03-11T21:33:02.341928+0000","last_peered":"2024-03-11T21:33:02.341928+0000","last_clean":"2024-03-11T21:33:02.341928+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:33:02.341928+0000","last_undegraded":"2024-03-11T21:33:02.341928+0000","last_fullsized":"2024-03-11T21:33:02.341928+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:13.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.487366+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":550,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318425+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:33:05.318425+0000","last_peered":"2024-03-11T21:33:05.318425+0000","last_clean":"2024-03-11T21:33:05.318425+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:33:05.318425+0000","last_undegraded":"2024-03-11T21:33:05.318425+0000","last_fullsized":"2024-03-11T21:33:05.318425+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:48.923389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":777,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.303711+0000","last_change":"2024-03-11T21:28:54.069307+0000","last_active":"2024-03-11T21:33:02.303711+0000","last_peered":"2024-03-11T21:33:02.303711+0000","last_clean":"2024-03-11T21:33:02.303711+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:33:02.303711+0000","last_undegraded":"2024-03-11T21:33:02.303711+0000","last_fullsized":"2024-03-11T21:33:02.303711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:54.069179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.069179+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":659,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773466+0000","last_change":"2024-03-11T21:32:05.838146+0000","last_active":"2024-03-11T21:33:02.773466+0000","last_peered":"2024-03-11T21:33:02.773466+0000","last_clean":"2024-03-11T21:33:02.773466+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:33:02.773466+0000","last_undegraded":"2024-03-11T21:33:02.773466+0000","last_fullsized":"2024-03-11T21:33:02.773466+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.838032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.838032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.307236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"259'11","reported_seq":456,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.323657+0000","last_change":"2024-03-11T21:32:51.310728+0000","last_active":"2024-03-11T21:33:05.323657+0000","last_peered":"2024-03-11T21:33:05.323657+0000","last_clean":"2024-03-11T21:33:05.323657+0000","last_became_active":"2024-03-11T21:32:51.309686+0000","last_became_peered":"2024-03-11T21:32:51.309686+0000","last_unstale":"2024-03-11T21:33:05.323657+0000","last_undegraded":"2024-03-11T21:33:05.323657+0000","last_fullsized":"2024-03-11T21:33:05.323657+0000","mapping_epoch":295,"log_start":"259'11","ondisk_log_start":"259'11","created":139,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'9","last_scrub_stamp":"2024-03-11T21:30:29.722893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:29.722893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":508,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334348+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:33:02.334348+0000","last_peered":"2024-03-11T21:33:02.334348+0000","last_clean":"2024-03-11T21:33:02.334348+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:33:02.334348+0000","last_undegraded":"2024-03-11T21:33:02.334348+0000","last_fullsized":"2024-03-11T21:33:02.334348+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.054249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"107'17","reported_seq":679,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297279+0000","last_change":"2024-03-11T21:30:29.035776+0000","last_active":"2024-03-11T21:33:02.297279+0000","last_peered":"2024-03-11T21:33:02.297279+0000","last_clean":"2024-03-11T21:33:02.297279+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:33:02.297279+0000","last_undegraded":"2024-03-11T21:33:02.297279+0000","last_fullsized":"2024-03-11T21:33:02.297279+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'17","last_scrub_stamp":"2024-03-11T21:30:29.035470+0000","last_deep_scrub":"107'17","last_deep_scrub_stamp":"2024-03-11T21:24:49.278162+0000","last_clean_scrub_stamp":"2024-03-11T21:30:29.035470+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330504+0000","last_change":"2024-03-11T21:31:37.402980+0000","last_active":"2024-03-11T21:33:02.330504+0000","last_peered":"2024-03-11T21:33:02.330504+0000","last_clean":"2024-03-11T21:33:02.330504+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:33:02.330504+0000","last_undegraded":"2024-03-11T21:33:02.330504+0000","last_fullsized":"2024-03-11T21:33:02.330504+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:05.302619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:29:05.302619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":654,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.780465+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:33:02.780465+0000","last_peered":"2024-03-11T21:33:02.780465+0000","last_clean":"2024-03-11T21:33:02.780465+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:33:02.780465+0000","last_undegraded":"2024-03-11T21:33:02.780465+0000","last_fullsized":"2024-03-11T21:33:02.780465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:49.570886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.346556+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:33:02.346556+0000","last_peered":"2024-03-11T21:33:02.346556+0000","last_clean":"2024-03-11T21:33:02.346556+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:33:02.346556+0000","last_undegraded":"2024-03-11T21:33:02.346556+0000","last_fullsized":"2024-03-11T21:33:02.346556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:34.315898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":564,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.299856+0000","last_change":"2024-03-11T21:32:23.731916+0000","last_active":"2024-03-11T21:33:02.299856+0000","last_peered":"2024-03-11T21:33:02.299856+0000","last_clean":"2024-03-11T21:33:02.299856+0000","last_became_active":"2024-03-11T21:32:09.326360+0000","last_became_peered":"2024-03-11T21:32:09.326360+0000","last_unstale":"2024-03-11T21:33:02.299856+0000","last_undegraded":"2024-03-11T21:33:02.299856+0000","last_fullsized":"2024-03-11T21:33:02.299856+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.731769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.731769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.404768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.346859+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:33:02.346859+0000","last_peered":"2024-03-11T21:33:02.346859+0000","last_clean":"2024-03-11T21:33:02.346859+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:33:02.346859+0000","last_undegraded":"2024-03-11T21:33:02.346859+0000","last_fullsized":"2024-03-11T21:33:02.346859+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:38.667213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":786,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342570+0000","last_change":"2024-03-11T21:31:37.390682+0000","last_active":"2024-03-11T21:33:02.342570+0000","last_peered":"2024-03-11T21:33:02.342570+0000","last_clean":"2024-03-11T21:33:02.342570+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:33:02.342570+0000","last_undegraded":"2024-03-11T21:33:02.342570+0000","last_fullsized":"2024-03-11T21:33:02.342570+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:28:53.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:53.325007+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":664,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297290+0000","last_change":"2024-03-11T21:30:52.898564+0000","last_active":"2024-03-11T21:33:02.297290+0000","last_peered":"2024-03-11T21:33:02.297290+0000","last_clean":"2024-03-11T21:33:02.297290+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:33:02.297290+0000","last_undegraded":"2024-03-11T21:33:02.297290+0000","last_fullsized":"2024-03-11T21:33:02.297290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.898357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":688,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319477+0000","last_change":"2024-03-11T21:31:58.977677+0000","last_active":"2024-03-11T21:33:03.319477+0000","last_peered":"2024-03-11T21:33:03.319477+0000","last_clean":"2024-03-11T21:33:03.319477+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:33:03.319477+0000","last_undegraded":"2024-03-11T21:33:03.319477+0000","last_fullsized":"2024-03-11T21:33:03.319477+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:31:58.977541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.977541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.700906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":671,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.766675+0000","last_change":"2024-03-11T21:30:31.534053+0000","last_active":"2024-03-11T21:33:02.766675+0000","last_peered":"2024-03-11T21:33:02.766675+0000","last_clean":"2024-03-11T21:33:02.766675+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:33:02.766675+0000","last_undegraded":"2024-03-11T21:33:02.766675+0000","last_fullsized":"2024-03-11T21:33:02.766675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:30:31.533807+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:30:31.533807+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334597+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:33:02.334597+0000","last_peered":"2024-03-11T21:33:02.334597+0000","last_clean":"2024-03-11T21:33:02.334597+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:33:02.334597+0000","last_undegraded":"2024-03-11T21:33:02.334597+0000","last_fullsized":"2024-03-11T21:33:02.334597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"142'2","reported_seq":557,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342479+0000","last_change":"2024-03-11T21:32:13.306779+0000","last_active":"2024-03-11T21:33:02.342479+0000","last_peered":"2024-03-11T21:33:02.342479+0000","last_clean":"2024-03-11T21:33:02.342479+0000","last_became_active":"2024-03-11T21:32:13.305739+0000","last_became_peered":"2024-03-11T21:32:13.305739+0000","last_unstale":"2024-03-11T21:33:02.342479+0000","last_undegraded":"2024-03-11T21:33:02.342479+0000","last_fullsized":"2024-03-11T21:33:02.342479+0000","mapping_epoch":139,"log_start":"142'2","ondisk_log_start":"142'2","created":139,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'2","last_scrub_stamp":"2024-03-11T21:30:35.816643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.816643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":507,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.335495+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:33:02.335495+0000","last_peered":"2024-03-11T21:33:02.335495+0000","last_clean":"2024-03-11T21:33:02.335495+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:33:02.335495+0000","last_undegraded":"2024-03-11T21:33:02.335495+0000","last_fullsized":"2024-03-11T21:33:02.335495+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.151214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297073+0000","last_change":"2024-03-11T21:31:02.847904+0000","last_active":"2024-03-11T21:33:02.297073+0000","last_peered":"2024-03-11T21:33:02.297073+0000","last_clean":"2024-03-11T21:33:02.297073+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:33:02.297073+0000","last_undegraded":"2024-03-11T21:33:02.297073+0000","last_fullsized":"2024-03-11T21:33:02.297073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.847692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.847692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.581115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1237,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342156+0000","last_change":"2024-03-11T21:30:17.907380+0000","last_active":"2024-03-11T21:33:02.342156+0000","last_peered":"2024-03-11T21:33:02.342156+0000","last_clean":"2024-03-11T21:33:02.342156+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:33:02.342156+0000","last_undegraded":"2024-03-11T21:33:02.342156+0000","last_fullsized":"2024-03-11T21:33:02.342156+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:30:17.907168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.907168+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":579,"num_read_kb":574,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":677,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.350297+0000","last_change":"2024-03-11T21:30:47.779423+0000","last_active":"2024-03-11T21:33:02.350297+0000","last_peered":"2024-03-11T21:33:02.350297+0000","last_clean":"2024-03-11T21:33:02.350297+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:33:02.350297+0000","last_undegraded":"2024-03-11T21:33:02.350297+0000","last_fullsized":"2024-03-11T21:33:02.350297+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:30:47.779298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.779298+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":567,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.322913+0000","last_change":"2024-03-11T21:31:37.408692+0000","last_active":"2024-03-11T21:33:05.322913+0000","last_peered":"2024-03-11T21:33:05.322913+0000","last_clean":"2024-03-11T21:33:05.322913+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:33:05.322913+0000","last_undegraded":"2024-03-11T21:33:05.322913+0000","last_fullsized":"2024-03-11T21:33:05.322913+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:29:16.233739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.233739+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300860+0000","last_change":"2024-03-11T21:30:58.912377+0000","last_active":"2024-03-11T21:33:02.300860+0000","last_peered":"2024-03-11T21:33:02.300860+0000","last_clean":"2024-03-11T21:33:02.300860+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:33:02.300860+0000","last_undegraded":"2024-03-11T21:33:02.300860+0000","last_fullsized":"2024-03-11T21:33:02.300860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.912133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":507,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338617+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:33:02.338617+0000","last_peered":"2024-03-11T21:33:02.338617+0000","last_clean":"2024-03-11T21:33:02.338617+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:33:02.338617+0000","last_undegraded":"2024-03-11T21:33:02.338617+0000","last_fullsized":"2024-03-11T21:33:02.338617+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":646,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.764903+0000","last_change":"2024-03-11T21:30:25.450518+0000","last_active":"2024-03-11T21:33:02.764903+0000","last_peered":"2024-03-11T21:33:02.764903+0000","last_clean":"2024-03-11T21:33:02.764903+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:33:02.764903+0000","last_undegraded":"2024-03-11T21:33:02.764903+0000","last_fullsized":"2024-03-11T21:33:02.764903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.450301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.450301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"297'32","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.321769+0000","last_change":"2024-03-11T21:32:05.077652+0000","last_active":"2024-03-11T21:33:05.321769+0000","last_peered":"2024-03-11T21:33:05.321769+0000","last_clean":"2024-03-11T21:33:05.321769+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:33:05.321769+0000","last_undegraded":"2024-03-11T21:33:05.321769+0000","last_fullsized":"2024-03-11T21:33:05.321769+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'32","last_scrub_stamp":"2024-03-11T21:32:05.077406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.077406+0000","objects_scrubbed":10,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.334401+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338350+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:33:02.338350+0000","last_peered":"2024-03-11T21:33:02.338350+0000","last_clean":"2024-03-11T21:33:02.338350+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:33:02.338350+0000","last_undegraded":"2024-03-11T21:33:02.338350+0000","last_fullsized":"2024-03-11T21:33:02.338350+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.436858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":514,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.317801+0000","last_change":"2024-03-11T21:32:07.975839+0000","last_active":"2024-03-11T21:33:05.317801+0000","last_peered":"2024-03-11T21:33:05.317801+0000","last_clean":"2024-03-11T21:33:05.317801+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:33:05.317801+0000","last_undegraded":"2024-03-11T21:33:05.317801+0000","last_fullsized":"2024-03-11T21:33:05.317801+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.975732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.975732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:13.100368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":502,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319895+0000","last_change":"2024-03-11T21:30:48.777877+0000","last_active":"2024-03-11T21:33:03.319895+0000","last_peered":"2024-03-11T21:33:03.319895+0000","last_clean":"2024-03-11T21:33:03.319895+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:33:03.319895+0000","last_undegraded":"2024-03-11T21:33:03.319895+0000","last_fullsized":"2024-03-11T21:33:03.319895+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.777772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.777772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":663,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.780387+0000","last_change":"2024-03-11T21:32:07.813703+0000","last_active":"2024-03-11T21:33:02.780387+0000","last_peered":"2024-03-11T21:33:02.780387+0000","last_clean":"2024-03-11T21:33:02.780387+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:33:02.780387+0000","last_undegraded":"2024-03-11T21:33:02.780387+0000","last_fullsized":"2024-03-11T21:33:02.780387+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.813604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.813604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.697988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":663,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330436+0000","last_change":"2024-03-11T21:31:37.403229+0000","last_active":"2024-03-11T21:33:02.330436+0000","last_peered":"2024-03-11T21:33:02.330436+0000","last_clean":"2024-03-11T21:33:02.330436+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:33:02.330436+0000","last_undegraded":"2024-03-11T21:33:02.330436+0000","last_fullsized":"2024-03-11T21:33:02.330436+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:48.364816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.364816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.574279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"84'14","reported_seq":674,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.315452+0000","last_change":"2024-03-11T21:30:30.000308+0000","last_active":"2024-03-11T21:33:03.315452+0000","last_peered":"2024-03-11T21:33:03.315452+0000","last_clean":"2024-03-11T21:33:03.315452+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:33:03.315452+0000","last_undegraded":"2024-03-11T21:33:03.315452+0000","last_fullsized":"2024-03-11T21:33:03.315452+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'14","last_scrub_stamp":"2024-03-11T21:30:30.000054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.000054+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":524,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.320666+0000","last_change":"2024-03-11T21:32:17.944438+0000","last_active":"2024-03-11T21:33:05.320666+0000","last_peered":"2024-03-11T21:33:05.320666+0000","last_clean":"2024-03-11T21:33:05.320666+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:33:05.320666+0000","last_undegraded":"2024-03-11T21:33:05.320666+0000","last_fullsized":"2024-03-11T21:33:05.320666+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.944141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.944141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.417735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.341440+0000","last_change":"2024-03-11T21:32:16.760994+0000","last_active":"2024-03-11T21:33:02.341440+0000","last_peered":"2024-03-11T21:33:02.341440+0000","last_clean":"2024-03-11T21:33:02.341440+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:33:02.341440+0000","last_undegraded":"2024-03-11T21:33:02.341440+0000","last_fullsized":"2024-03-11T21:33:02.341440+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.760914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.760914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.177261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":645,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334280+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:33:02.334280+0000","last_peered":"2024-03-11T21:33:02.334280+0000","last_clean":"2024-03-11T21:33:02.334280+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:33:02.334280+0000","last_undegraded":"2024-03-11T21:33:02.334280+0000","last_fullsized":"2024-03-11T21:33:02.334280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":917,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330471+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:33:02.330471+0000","last_peered":"2024-03-11T21:33:02.330471+0000","last_clean":"2024-03-11T21:33:02.330471+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:33:02.330471+0000","last_undegraded":"2024-03-11T21:33:02.330471+0000","last_fullsized":"2024-03-11T21:33:02.330471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.772184+0000","last_change":"2024-03-11T21:30:35.064347+0000","last_active":"2024-03-11T21:33:02.772184+0000","last_peered":"2024-03-11T21:33:02.772184+0000","last_clean":"2024-03-11T21:33:02.772184+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:33:02.772184+0000","last_undegraded":"2024-03-11T21:33:02.772184+0000","last_fullsized":"2024-03-11T21:33:02.772184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:35.064136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.064136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":665,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.345237+0000","last_change":"2024-03-11T21:30:32.862005+0000","last_active":"2024-03-11T21:33:02.345237+0000","last_peered":"2024-03-11T21:33:02.345237+0000","last_clean":"2024-03-11T21:33:02.345237+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:33:02.345237+0000","last_undegraded":"2024-03-11T21:33:02.345237+0000","last_fullsized":"2024-03-11T21:33:02.345237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:32.861910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.861910+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":650,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.780639+0000","last_change":"2024-03-11T21:32:26.159690+0000","last_active":"2024-03-11T21:33:02.780639+0000","last_peered":"2024-03-11T21:33:02.780639+0000","last_clean":"2024-03-11T21:33:02.780639+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:33:02.780639+0000","last_undegraded":"2024-03-11T21:33:02.780639+0000","last_fullsized":"2024-03-11T21:33:02.780639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:26.159472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:26.159472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.482009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":509,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.771647+0000","last_change":"2024-03-11T21:31:57.949982+0000","last_active":"2024-03-11T21:33:02.771647+0000","last_peered":"2024-03-11T21:33:02.771647+0000","last_clean":"2024-03-11T21:33:02.771647+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:33:02.771647+0000","last_undegraded":"2024-03-11T21:33:02.771647+0000","last_fullsized":"2024-03-11T21:33:02.771647+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.949771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.949771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.314081+0000","last_change":"2024-03-11T21:32:11.769286+0000","last_active":"2024-03-11T21:33:02.314081+0000","last_peered":"2024-03-11T21:33:02.314081+0000","last_clean":"2024-03-11T21:33:02.314081+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:33:02.314081+0000","last_undegraded":"2024-03-11T21:33:02.314081+0000","last_fullsized":"2024-03-11T21:33:02.314081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.769136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.769136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.591393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.304441+0000","last_change":"2024-03-11T21:30:59.840809+0000","last_active":"2024-03-11T21:33:02.304441+0000","last_peered":"2024-03-11T21:33:02.304441+0000","last_clean":"2024-03-11T21:33:02.304441+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:33:02.304441+0000","last_undegraded":"2024-03-11T21:33:02.304441+0000","last_fullsized":"2024-03-11T21:33:02.304441+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.840718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.840718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"149'26","reported_seq":588,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318797+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:33:05.318797+0000","last_peered":"2024-03-11T21:33:05.318797+0000","last_clean":"2024-03-11T21:33:05.318797+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:33:05.318797+0000","last_undegraded":"2024-03-11T21:33:05.318797+0000","last_fullsized":"2024-03-11T21:33:05.318797+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.837252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338091+0000","last_change":"2024-03-11T21:29:10.347962+0000","last_active":"2024-03-11T21:33:02.338091+0000","last_peered":"2024-03-11T21:33:02.338091+0000","last_clean":"2024-03-11T21:33:02.338091+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:33:02.338091+0000","last_undegraded":"2024-03-11T21:33:02.338091+0000","last_fullsized":"2024-03-11T21:33:02.338091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:10.347717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:29:10.347717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":502,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.315049+0000","last_change":"2024-03-11T21:30:53.861799+0000","last_active":"2024-03-11T21:33:03.315049+0000","last_peered":"2024-03-11T21:33:03.315049+0000","last_clean":"2024-03-11T21:33:03.315049+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:33:03.315049+0000","last_undegraded":"2024-03-11T21:33:03.315049+0000","last_fullsized":"2024-03-11T21:33:03.315049+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:53.861701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:53.861701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":658,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320225+0000","last_change":"2024-03-11T21:31:37.429746+0000","last_active":"2024-03-11T21:33:03.320225+0000","last_peered":"2024-03-11T21:33:03.320225+0000","last_clean":"2024-03-11T21:33:03.320225+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:33:03.320225+0000","last_undegraded":"2024-03-11T21:33:03.320225+0000","last_fullsized":"2024-03-11T21:33:03.320225+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:45.832418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:45.832418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"113'27","reported_seq":703,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297878+0000","last_change":"2024-03-11T21:30:48.891794+0000","last_active":"2024-03-11T21:33:02.297878+0000","last_peered":"2024-03-11T21:33:02.297878+0000","last_clean":"2024-03-11T21:33:02.297878+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:33:02.297878+0000","last_undegraded":"2024-03-11T21:33:02.297878+0000","last_fullsized":"2024-03-11T21:33:02.297878+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:30:48.891648+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.891648+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.581017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":663,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773094+0000","last_change":"2024-03-11T21:31:37.388733+0000","last_active":"2024-03-11T21:33:02.773094+0000","last_peered":"2024-03-11T21:33:02.773094+0000","last_clean":"2024-03-11T21:33:02.773094+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:33:02.773094+0000","last_undegraded":"2024-03-11T21:33:02.773094+0000","last_fullsized":"2024-03-11T21:33:02.773094+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.991582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.991582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.347854+0000","last_change":"2024-03-11T21:30:33.825558+0000","last_active":"2024-03-11T21:33:02.347854+0000","last_peered":"2024-03-11T21:33:02.347854+0000","last_clean":"2024-03-11T21:33:02.347854+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:33:02.347854+0000","last_undegraded":"2024-03-11T21:33:02.347854+0000","last_fullsized":"2024-03-11T21:33:02.347854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:33.825475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:33.825475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"259'14","reported_seq":570,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.304659+0000","last_change":"2024-03-11T21:32:05.389256+0000","last_active":"2024-03-11T21:33:02.304659+0000","last_peered":"2024-03-11T21:33:02.304659+0000","last_clean":"2024-03-11T21:33:02.304659+0000","last_became_active":"2024-03-11T21:32:05.388874+0000","last_became_peered":"2024-03-11T21:32:05.388874+0000","last_unstale":"2024-03-11T21:33:02.304659+0000","last_undegraded":"2024-03-11T21:33:02.304659+0000","last_fullsized":"2024-03-11T21:33:02.304659+0000","mapping_epoch":139,"log_start":"259'14","ondisk_log_start":"259'14","created":139,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'10","last_scrub_stamp":"2024-03-11T21:30:42.647195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.647195+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":22,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":508,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.779765+0000","last_change":"2024-03-11T21:31:55.943647+0000","last_active":"2024-03-11T21:33:02.779765+0000","last_peered":"2024-03-11T21:33:02.779765+0000","last_clean":"2024-03-11T21:33:02.779765+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:33:02.779765+0000","last_undegraded":"2024-03-11T21:33:02.779765+0000","last_fullsized":"2024-03-11T21:33:02.779765+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.943487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.943487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.360467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":792,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320153+0000","last_change":"2024-03-11T21:32:15.949343+0000","last_active":"2024-03-11T21:33:03.320153+0000","last_peered":"2024-03-11T21:33:03.320153+0000","last_clean":"2024-03-11T21:33:03.320153+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:33:03.320153+0000","last_undegraded":"2024-03-11T21:33:03.320153+0000","last_fullsized":"2024-03-11T21:33:03.320153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:32:15.949107+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.949107+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.139215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":551,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318002+0000","last_change":"2024-03-11T21:31:37.399181+0000","last_active":"2024-03-11T21:33:05.318002+0000","last_peered":"2024-03-11T21:33:05.318002+0000","last_clean":"2024-03-11T21:33:05.318002+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:33:05.318002+0000","last_undegraded":"2024-03-11T21:33:05.318002+0000","last_fullsized":"2024-03-11T21:33:05.318002+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:30:35.849630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.849630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":800,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773571+0000","last_change":"2024-03-11T21:30:54.944552+0000","last_active":"2024-03-11T21:33:02.773571+0000","last_peered":"2024-03-11T21:33:02.773571+0000","last_clean":"2024-03-11T21:33:02.773571+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:33:02.773571+0000","last_undegraded":"2024-03-11T21:33:02.773571+0000","last_fullsized":"2024-03-11T21:33:02.773571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T21:30:54.944305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.944305+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":522,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.320439+0000","last_change":"2024-03-11T21:31:37.399025+0000","last_active":"2024-03-11T21:33:05.320439+0000","last_peered":"2024-03-11T21:33:05.320439+0000","last_clean":"2024-03-11T21:33:05.320439+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:33:05.320439+0000","last_undegraded":"2024-03-11T21:33:05.320439+0000","last_fullsized":"2024-03-11T21:33:05.320439+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.716495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.716495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":653,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.312636+0000","last_change":"2024-03-11T21:32:16.907647+0000","last_active":"2024-03-11T21:33:03.312636+0000","last_peered":"2024-03-11T21:33:03.312636+0000","last_clean":"2024-03-11T21:33:03.312636+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:33:03.312636+0000","last_undegraded":"2024-03-11T21:33:03.312636+0000","last_fullsized":"2024-03-11T21:33:03.312636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.907562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.907562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:37.130510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"259'9","reported_seq":562,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.763406+0000","last_change":"2024-03-11T21:32:39.363387+0000","last_active":"2024-03-11T21:33:02.763406+0000","last_peered":"2024-03-11T21:33:02.763406+0000","last_clean":"2024-03-11T21:33:02.763406+0000","last_became_active":"2024-03-11T21:32:39.359239+0000","last_became_peered":"2024-03-11T21:32:39.359239+0000","last_unstale":"2024-03-11T21:33:02.763406+0000","last_undegraded":"2024-03-11T21:33:02.763406+0000","last_fullsized":"2024-03-11T21:33:02.763406+0000","mapping_epoch":139,"log_start":"259'9","ondisk_log_start":"259'9","created":139,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5","last_scrub_stamp":"2024-03-11T21:30:48.625569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.625569+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":375,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.320591+0000","last_change":"2024-03-11T21:31:58.092962+0000","last_active":"2024-03-11T21:33:05.320591+0000","last_peered":"2024-03-11T21:33:05.320591+0000","last_clean":"2024-03-11T21:33:05.320591+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:33:05.320591+0000","last_undegraded":"2024-03-11T21:33:05.320591+0000","last_fullsized":"2024-03-11T21:33:05.320591+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.092744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.092744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.013717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":671,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.298242+0000","last_change":"2024-03-11T21:32:12.801345+0000","last_active":"2024-03-11T21:33:02.298242+0000","last_peered":"2024-03-11T21:33:02.298242+0000","last_clean":"2024-03-11T21:33:02.298242+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:33:02.298242+0000","last_undegraded":"2024-03-11T21:33:02.298242+0000","last_fullsized":"2024-03-11T21:33:02.298242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.497490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"104'16","reported_seq":682,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338995+0000","last_change":"2024-03-11T21:31:37.405413+0000","last_active":"2024-03-11T21:33:02.338995+0000","last_peered":"2024-03-11T21:33:02.338995+0000","last_clean":"2024-03-11T21:33:02.338995+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:33:02.338995+0000","last_undegraded":"2024-03-11T21:33:02.338995+0000","last_fullsized":"2024-03-11T21:33:02.338995+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16","last_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_deep_scrub":"104'16","last_deep_scrub_stamp":"2024-03-11T21:28:59.327656+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.327656+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":681,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.305557+0000","last_change":"2024-03-11T21:31:05.819709+0000","last_active":"2024-03-11T21:33:02.305557+0000","last_peered":"2024-03-11T21:33:02.305557+0000","last_clean":"2024-03-11T21:33:02.305557+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:33:02.305557+0000","last_undegraded":"2024-03-11T21:33:02.305557+0000","last_fullsized":"2024-03-11T21:33:02.305557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:31:05.819557+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.819557+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.344706+0000","last_change":"2024-03-11T21:31:37.406030+0000","last_active":"2024-03-11T21:33:02.344706+0000","last_peered":"2024-03-11T21:33:02.344706+0000","last_clean":"2024-03-11T21:33:02.344706+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:33:02.344706+0000","last_undegraded":"2024-03-11T21:33:02.344706+0000","last_fullsized":"2024-03-11T21:33:02.344706+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:54.362432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:54.362432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.345078+0000","last_change":"2024-03-11T21:30:40.750808+0000","last_active":"2024-03-11T21:33:02.345078+0000","last_peered":"2024-03-11T21:33:02.345078+0000","last_clean":"2024-03-11T21:33:02.345078+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:33:02.345078+0000","last_undegraded":"2024-03-11T21:33:02.345078+0000","last_fullsized":"2024-03-11T21:33:02.345078+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.750735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.750735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":668,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.331732+0000","last_change":"2024-03-11T21:29:19.356816+0000","last_active":"2024-03-11T21:33:02.331732+0000","last_peered":"2024-03-11T21:33:02.331732+0000","last_clean":"2024-03-11T21:33:02.331732+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:33:02.331732+0000","last_undegraded":"2024-03-11T21:33:02.331732+0000","last_fullsized":"2024-03-11T21:33:02.331732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:29:19.356702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.356702+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":671,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297970+0000","last_change":"2024-03-11T21:31:37.405137+0000","last_active":"2024-03-11T21:33:02.297970+0000","last_peered":"2024-03-11T21:33:02.297970+0000","last_clean":"2024-03-11T21:33:02.297970+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:33:02.297970+0000","last_undegraded":"2024-03-11T21:33:02.297970+0000","last_fullsized":"2024-03-11T21:33:02.297970+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.738299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.738299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.133247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.350480+0000","last_change":"2024-03-11T21:31:37.406910+0000","last_active":"2024-03-11T21:33:02.350480+0000","last_peered":"2024-03-11T21:33:02.350480+0000","last_clean":"2024-03-11T21:33:02.350480+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:33:02.350480+0000","last_undegraded":"2024-03-11T21:33:02.350480+0000","last_fullsized":"2024-03-11T21:33:02.350480+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.881361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.881361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":658,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.315053+0000","last_change":"2024-03-11T21:32:04.847327+0000","last_active":"2024-03-11T21:33:02.315053+0000","last_peered":"2024-03-11T21:33:02.315053+0000","last_clean":"2024-03-11T21:33:02.315053+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:33:02.315053+0000","last_undegraded":"2024-03-11T21:33:02.315053+0000","last_fullsized":"2024-03-11T21:33:02.315053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.847094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.847094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.764379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.346184+0000","last_change":"2024-03-11T21:30:48.791220+0000","last_active":"2024-03-11T21:33:02.346184+0000","last_peered":"2024-03-11T21:33:02.346184+0000","last_clean":"2024-03-11T21:33:02.346184+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:33:02.346184+0000","last_undegraded":"2024-03-11T21:33:02.346184+0000","last_fullsized":"2024-03-11T21:33:02.346184+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.791036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"115'6","reported_seq":674,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.766602+0000","last_change":"2024-03-11T21:32:15.567675+0000","last_active":"2024-03-11T21:33:02.766602+0000","last_peered":"2024-03-11T21:33:02.766602+0000","last_clean":"2024-03-11T21:33:02.766602+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:33:02.766602+0000","last_undegraded":"2024-03-11T21:33:02.766602+0000","last_fullsized":"2024-03-11T21:33:02.766602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:32:15.567443+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.567443+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.706289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":671,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.764228+0000","last_change":"2024-03-11T21:31:59.593413+0000","last_active":"2024-03-11T21:33:02.764228+0000","last_peered":"2024-03-11T21:33:02.764228+0000","last_clean":"2024-03-11T21:33:02.764228+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:33:02.764228+0000","last_undegraded":"2024-03-11T21:33:02.764228+0000","last_fullsized":"2024-03-11T21:33:02.764228+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.593037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.593037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.709416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":663,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.778202+0000","last_change":"2024-03-11T21:32:06.800643+0000","last_active":"2024-03-11T21:33:02.778202+0000","last_peered":"2024-03-11T21:33:02.778202+0000","last_clean":"2024-03-11T21:33:02.778202+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:33:02.778202+0000","last_undegraded":"2024-03-11T21:33:02.778202+0000","last_fullsized":"2024-03-11T21:33:02.778202+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.800436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.800436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.924396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.337807+0000","last_change":"2024-03-11T21:28:45.377770+0000","last_active":"2024-03-11T21:33:02.337807+0000","last_peered":"2024-03-11T21:33:02.337807+0000","last_clean":"2024-03-11T21:33:02.337807+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:33:02.337807+0000","last_undegraded":"2024-03-11T21:33:02.337807+0000","last_fullsized":"2024-03-11T21:33:02.337807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:45.377668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:45.377668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.343545+0000","last_change":"2024-03-11T21:31:57.633633+0000","last_active":"2024-03-11T21:33:02.343545+0000","last_peered":"2024-03-11T21:33:02.343545+0000","last_clean":"2024-03-11T21:33:02.343545+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:33:02.343545+0000","last_undegraded":"2024-03-11T21:33:02.343545+0000","last_fullsized":"2024-03-11T21:33:02.343545+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.633438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.633438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.657983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":676,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.335063+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:33:02.335063+0000","last_peered":"2024-03-11T21:33:02.335063+0000","last_clean":"2024-03-11T21:33:02.335063+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:33:02.335063+0000","last_undegraded":"2024-03-11T21:33:02.335063+0000","last_fullsized":"2024-03-11T21:33:02.335063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.772433+0000","last_change":"2024-03-11T21:30:46.912846+0000","last_active":"2024-03-11T21:33:02.772433+0000","last_peered":"2024-03-11T21:33:02.772433+0000","last_clean":"2024-03-11T21:33:02.772433+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:33:02.772433+0000","last_undegraded":"2024-03-11T21:33:02.772433+0000","last_fullsized":"2024-03-11T21:33:02.772433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:46.912728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.912728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":794,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773946+0000","last_change":"2024-03-11T21:32:15.855148+0000","last_active":"2024-03-11T21:33:02.773946+0000","last_peered":"2024-03-11T21:33:02.773946+0000","last_clean":"2024-03-11T21:33:02.773946+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:33:02.773946+0000","last_undegraded":"2024-03-11T21:33:02.773946+0000","last_fullsized":"2024-03-11T21:33:02.773946+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:32:15.854918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.854918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.754470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":653,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320093+0000","last_change":"2024-03-11T21:30:49.769958+0000","last_active":"2024-03-11T21:33:03.320093+0000","last_peered":"2024-03-11T21:33:03.320093+0000","last_clean":"2024-03-11T21:33:03.320093+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:33:03.320093+0000","last_undegraded":"2024-03-11T21:33:03.320093+0000","last_fullsized":"2024-03-11T21:33:03.320093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:30:49.769830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.768089+0000","last_change":"2024-03-11T21:30:51.554762+0000","last_active":"2024-03-11T21:33:02.768089+0000","last_peered":"2024-03-11T21:33:02.768089+0000","last_clean":"2024-03-11T21:33:02.768089+0000","last_became_active":"2024-03-11T21:28:51.357900+0000","last_became_peered":"2024-03-11T21:28:51.357900+0000","last_unstale":"2024-03-11T21:33:02.768089+0000","last_undegraded":"2024-03-11T21:33:02.768089+0000","last_fullsized":"2024-03-11T21:33:02.768089+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.554539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.554539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"116'13","reported_seq":681,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.318906+0000","last_change":"2024-03-11T21:30:52.813167+0000","last_active":"2024-03-11T21:33:03.318906+0000","last_peered":"2024-03-11T21:33:03.318906+0000","last_clean":"2024-03-11T21:33:03.318906+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:33:03.318906+0000","last_undegraded":"2024-03-11T21:33:03.318906+0000","last_fullsized":"2024-03-11T21:33:03.318906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:30:52.812936+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.812936+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":665,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319445+0000","last_change":"2024-03-11T21:30:58.882981+0000","last_active":"2024-03-11T21:33:03.319445+0000","last_peered":"2024-03-11T21:33:03.319445+0000","last_clean":"2024-03-11T21:33:03.319445+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:33:03.319445+0000","last_undegraded":"2024-03-11T21:33:03.319445+0000","last_fullsized":"2024-03-11T21:33:03.319445+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.882820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.882820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":777,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.346344+0000","last_change":"2024-03-11T21:29:14.510232+0000","last_active":"2024-03-11T21:33:02.346344+0000","last_peered":"2024-03-11T21:33:02.346344+0000","last_clean":"2024-03-11T21:33:02.346344+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:33:02.346344+0000","last_undegraded":"2024-03-11T21:33:02.346344+0000","last_fullsized":"2024-03-11T21:33:02.346344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:29:14.510132+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.510132+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.306381+0000","last_change":"2024-03-11T21:30:31.977271+0000","last_active":"2024-03-11T21:33:02.306381+0000","last_peered":"2024-03-11T21:33:02.306381+0000","last_clean":"2024-03-11T21:33:02.306381+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:33:02.306381+0000","last_undegraded":"2024-03-11T21:33:02.306381+0000","last_fullsized":"2024-03-11T21:33:02.306381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:31.977047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:31.977047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580691+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":371,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.325052+0000","last_change":"2024-03-11T21:31:37.401438+0000","last_active":"2024-03-11T21:33:05.325052+0000","last_peered":"2024-03-11T21:33:05.325052+0000","last_clean":"2024-03-11T21:33:05.325052+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:33:05.325052+0000","last_undegraded":"2024-03-11T21:33:05.325052+0000","last_fullsized":"2024-03-11T21:33:05.325052+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.696902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.696902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"100'13","reported_seq":677,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338805+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:33:02.338805+0000","last_peered":"2024-03-11T21:33:02.338805+0000","last_clean":"2024-03-11T21:33:02.338805+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:33:02.338805+0000","last_undegraded":"2024-03-11T21:33:02.338805+0000","last_fullsized":"2024-03-11T21:33:02.338805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.634197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":670,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.767620+0000","last_change":"2024-03-11T21:32:09.571388+0000","last_active":"2024-03-11T21:33:02.767620+0000","last_peered":"2024-03-11T21:33:02.767620+0000","last_clean":"2024-03-11T21:33:02.767620+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:33:02.767620+0000","last_undegraded":"2024-03-11T21:33:02.767620+0000","last_fullsized":"2024-03-11T21:33:02.767620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.571257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.571257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.587765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.344544+0000","last_change":"2024-03-11T21:31:37.402794+0000","last_active":"2024-03-11T21:33:02.344544+0000","last_peered":"2024-03-11T21:33:02.344544+0000","last_clean":"2024-03-11T21:33:02.344544+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:33:02.344544+0000","last_undegraded":"2024-03-11T21:33:02.344544+0000","last_fullsized":"2024-03-11T21:33:02.344544+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:28.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:28.810562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":662,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.312566+0000","last_change":"2024-03-11T21:32:06.989987+0000","last_active":"2024-03-11T21:33:03.312566+0000","last_peered":"2024-03-11T21:33:03.312566+0000","last_clean":"2024-03-11T21:33:03.312566+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:33:03.312566+0000","last_undegraded":"2024-03-11T21:33:03.312566+0000","last_fullsized":"2024-03-11T21:33:03.312566+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.989724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.989724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.368810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":503,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319651+0000","last_change":"2024-03-11T21:30:36.943381+0000","last_active":"2024-03-11T21:33:03.319651+0000","last_peered":"2024-03-11T21:33:03.319651+0000","last_clean":"2024-03-11T21:33:03.319651+0000","last_became_active":"2024-03-11T21:28:51.388727+0000","last_became_peered":"2024-03-11T21:28:51.388727+0000","last_unstale":"2024-03-11T21:33:03.319651+0000","last_undegraded":"2024-03-11T21:33:03.319651+0000","last_fullsized":"2024-03-11T21:33:03.319651+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.943172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":674,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300370+0000","last_change":"2024-03-11T21:30:38.863143+0000","last_active":"2024-03-11T21:33:02.300370+0000","last_peered":"2024-03-11T21:33:02.300370+0000","last_clean":"2024-03-11T21:33:02.300370+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:33:02.300370+0000","last_undegraded":"2024-03-11T21:33:02.300370+0000","last_fullsized":"2024-03-11T21:33:02.300370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:38.862999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.862999+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":566,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.323009+0000","last_change":"2024-03-11T21:32:11.929832+0000","last_active":"2024-03-11T21:33:05.323009+0000","last_peered":"2024-03-11T21:33:05.323009+0000","last_clean":"2024-03-11T21:33:05.323009+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:33:05.323009+0000","last_undegraded":"2024-03-11T21:33:05.323009+0000","last_fullsized":"2024-03-11T21:33:05.323009+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.929628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.929628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.408544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":526,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.317543+0000","last_change":"2024-03-11T21:31:37.400845+0000","last_active":"2024-03-11T21:33:05.317543+0000","last_peered":"2024-03-11T21:33:05.317543+0000","last_clean":"2024-03-11T21:33:05.317543+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:33:05.317543+0000","last_undegraded":"2024-03-11T21:33:05.317543+0000","last_fullsized":"2024-03-11T21:33:05.317543+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:59.189248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:28:59.189248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342355+0000","last_change":"2024-03-11T21:30:27.784291+0000","last_active":"2024-03-11T21:33:02.342355+0000","last_peered":"2024-03-11T21:33:02.342355+0000","last_clean":"2024-03-11T21:33:02.342355+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:33:02.342355+0000","last_undegraded":"2024-03-11T21:33:02.342355+0000","last_fullsized":"2024-03-11T21:33:02.342355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:27.783939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.783939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":502,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319694+0000","last_change":"2024-03-11T21:32:18.866826+0000","last_active":"2024-03-11T21:33:03.319694+0000","last_peered":"2024-03-11T21:33:03.319694+0000","last_clean":"2024-03-11T21:33:03.319694+0000","last_became_active":"2024-03-11T21:28:51.388792+0000","last_became_peered":"2024-03-11T21:28:51.388792+0000","last_unstale":"2024-03-11T21:33:03.319694+0000","last_undegraded":"2024-03-11T21:33:03.319694+0000","last_fullsized":"2024-03-11T21:33:03.319694+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.866701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.866701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:37.142390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":670,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320588+0000","last_change":"2024-03-11T21:30:27.030104+0000","last_active":"2024-03-11T21:33:03.320588+0000","last_peered":"2024-03-11T21:33:03.320588+0000","last_clean":"2024-03-11T21:33:03.320588+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:33:03.320588+0000","last_undegraded":"2024-03-11T21:33:03.320588+0000","last_fullsized":"2024-03-11T21:33:03.320588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T21:30:27.029873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.029873+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.778744+0000","last_change":"2024-03-11T21:31:00.965260+0000","last_active":"2024-03-11T21:33:02.778744+0000","last_peered":"2024-03-11T21:33:02.778744+0000","last_clean":"2024-03-11T21:33:02.778744+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:33:02.778744+0000","last_undegraded":"2024-03-11T21:33:02.778744+0000","last_fullsized":"2024-03-11T21:33:02.778744+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.964917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.964917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.764140+0000","last_change":"2024-03-11T21:32:12.550833+0000","last_active":"2024-03-11T21:33:02.764140+0000","last_peered":"2024-03-11T21:33:02.764140+0000","last_clean":"2024-03-11T21:33:02.764140+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:33:02.764140+0000","last_undegraded":"2024-03-11T21:33:02.764140+0000","last_fullsized":"2024-03-11T21:33:02.764140+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.550625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.550625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.339924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.333663+0000","last_change":"2024-03-11T21:29:11.346981+0000","last_active":"2024-03-11T21:33:02.333663+0000","last_peered":"2024-03-11T21:33:02.333663+0000","last_clean":"2024-03-11T21:33:02.333663+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:33:02.333663+0000","last_undegraded":"2024-03-11T21:33:02.333663+0000","last_fullsized":"2024-03-11T21:33:02.333663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.346840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.346840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":511,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319675+0000","last_change":"2024-03-11T21:32:12.998779+0000","last_active":"2024-03-11T21:33:03.319675+0000","last_peered":"2024-03-11T21:33:03.319675+0000","last_clean":"2024-03-11T21:33:03.319675+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:33:03.319675+0000","last_undegraded":"2024-03-11T21:33:03.319675+0000","last_fullsized":"2024-03-11T21:33:03.319675+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.998644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.998644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:14.669863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":683,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.777923+0000","last_change":"2024-03-11T21:31:59.996470+0000","last_active":"2024-03-11T21:33:02.777923+0000","last_peered":"2024-03-11T21:33:02.777923+0000","last_clean":"2024-03-11T21:33:02.777923+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:33:02.777923+0000","last_undegraded":"2024-03-11T21:33:02.777923+0000","last_fullsized":"2024-03-11T21:33:02.777923+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:59.996246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.996246+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.079844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":670,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.766569+0000","last_change":"2024-03-11T21:32:28.777744+0000","last_active":"2024-03-11T21:33:02.766569+0000","last_peered":"2024-03-11T21:33:02.766569+0000","last_clean":"2024-03-11T21:33:02.766569+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:33:02.766569+0000","last_undegraded":"2024-03-11T21:33:02.766569+0000","last_fullsized":"2024-03-11T21:33:02.766569+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.777629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.777629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.837185+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":645,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.330643+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:33:02.330643+0000","last_peered":"2024-03-11T21:33:02.330643+0000","last_clean":"2024-03-11T21:33:02.330643+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:33:02.330643+0000","last_undegraded":"2024-03-11T21:33:02.330643+0000","last_fullsized":"2024-03-11T21:33:02.330643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.344899+0000","last_change":"2024-03-11T21:30:52.794455+0000","last_active":"2024-03-11T21:33:02.344899+0000","last_peered":"2024-03-11T21:33:02.344899+0000","last_clean":"2024-03-11T21:33:02.344899+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:33:02.344899+0000","last_undegraded":"2024-03-11T21:33:02.344899+0000","last_fullsized":"2024-03-11T21:33:02.344899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.794209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.794209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":498,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334711+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:33:02.334711+0000","last_peered":"2024-03-11T21:33:02.334711+0000","last_clean":"2024-03-11T21:33:02.334711+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:33:02.334711+0000","last_undegraded":"2024-03-11T21:33:02.334711+0000","last_fullsized":"2024-03-11T21:33:02.334711+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":594,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318274+0000","last_change":"2024-03-11T21:31:37.386896+0000","last_active":"2024-03-11T21:33:05.318274+0000","last_peered":"2024-03-11T21:33:05.318274+0000","last_clean":"2024-03-11T21:33:05.318274+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:33:05.318274+0000","last_undegraded":"2024-03-11T21:33:05.318274+0000","last_fullsized":"2024-03-11T21:33:05.318274+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:29:20.448110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.448110+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":668,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.763002+0000","last_change":"2024-03-11T21:32:29.159917+0000","last_active":"2024-03-11T21:33:02.763002+0000","last_peered":"2024-03-11T21:33:02.763002+0000","last_clean":"2024-03-11T21:33:02.763002+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:33:02.763002+0000","last_undegraded":"2024-03-11T21:33:02.763002+0000","last_fullsized":"2024-03-11T21:33:02.763002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:29.159716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:29.159716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.410503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":787,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.766857+0000","last_change":"2024-03-11T21:30:56.549284+0000","last_active":"2024-03-11T21:33:02.766857+0000","last_peered":"2024-03-11T21:33:02.766857+0000","last_clean":"2024-03-11T21:33:02.766857+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:33:02.766857+0000","last_undegraded":"2024-03-11T21:33:02.766857+0000","last_fullsized":"2024-03-11T21:33:02.766857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:56.549048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.549048+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.334380+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:33:02.334380+0000","last_peered":"2024-03-11T21:33:02.334380+0000","last_clean":"2024-03-11T21:33:02.334380+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:33:02.334380+0000","last_undegraded":"2024-03-11T21:33:02.334380+0000","last_fullsized":"2024-03-11T21:33:02.334380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":508,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.771726+0000","last_change":"2024-03-11T21:31:37.387585+0000","last_active":"2024-03-11T21:33:02.771726+0000","last_peered":"2024-03-11T21:33:02.771726+0000","last_clean":"2024-03-11T21:33:02.771726+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:33:02.771726+0000","last_undegraded":"2024-03-11T21:33:02.771726+0000","last_fullsized":"2024-03-11T21:33:02.771726+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:39.006804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:39.006804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":683,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.341591+0000","last_change":"2024-03-11T21:31:54.548395+0000","last_active":"2024-03-11T21:33:02.341591+0000","last_peered":"2024-03-11T21:33:02.341591+0000","last_clean":"2024-03-11T21:33:02.341591+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:33:02.341591+0000","last_undegraded":"2024-03-11T21:33:02.341591+0000","last_fullsized":"2024-03-11T21:33:02.341591+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:54.548275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.548275+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:22.947459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342797+0000","last_change":"2024-03-11T21:28:48.305255+0000","last_active":"2024-03-11T21:33:02.342797+0000","last_peered":"2024-03-11T21:33:02.342797+0000","last_clean":"2024-03-11T21:33:02.342797+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:33:02.342797+0000","last_undegraded":"2024-03-11T21:33:02.342797+0000","last_fullsized":"2024-03-11T21:33:02.342797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:28:48.305156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:28:48.305156+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":657,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319827+0000","last_change":"2024-03-11T21:32:23.014073+0000","last_active":"2024-03-11T21:33:03.319827+0000","last_peered":"2024-03-11T21:33:03.319827+0000","last_clean":"2024-03-11T21:33:03.319827+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:33:03.319827+0000","last_undegraded":"2024-03-11T21:33:03.319827+0000","last_fullsized":"2024-03-11T21:33:03.319827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.013984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.013984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.212286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":517,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.322841+0000","last_change":"2024-03-11T21:31:37.396975+0000","last_active":"2024-03-11T21:33:05.322841+0000","last_peered":"2024-03-11T21:33:05.322841+0000","last_clean":"2024-03-11T21:33:05.322841+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:33:05.322841+0000","last_undegraded":"2024-03-11T21:33:05.322841+0000","last_fullsized":"2024-03-11T21:33:05.322841+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:56.181111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:28:56.181111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":498,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.331558+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:33:02.331558+0000","last_peered":"2024-03-11T21:33:02.331558+0000","last_clean":"2024-03-11T21:33:02.331558+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:33:02.331558+0000","last_undegraded":"2024-03-11T21:33:02.331558+0000","last_fullsized":"2024-03-11T21:33:02.331558+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"118'7","reported_seq":664,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.319597+0000","last_change":"2024-03-11T21:30:28.077530+0000","last_active":"2024-03-11T21:33:03.319597+0000","last_peered":"2024-03-11T21:33:03.319597+0000","last_clean":"2024-03-11T21:33:03.319597+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:33:03.319597+0000","last_undegraded":"2024-03-11T21:33:03.319597+0000","last_fullsized":"2024-03-11T21:33:03.319597+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7","last_scrub_stamp":"2024-03-11T21:30:28.077400+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:30:28.077400+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":670,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.764570+0000","last_change":"2024-03-11T21:32:10.547894+0000","last_active":"2024-03-11T21:33:02.764570+0000","last_peered":"2024-03-11T21:33:02.764570+0000","last_clean":"2024-03-11T21:33:02.764570+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:33:02.764570+0000","last_undegraded":"2024-03-11T21:33:02.764570+0000","last_fullsized":"2024-03-11T21:33:02.764570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.547682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.547682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.675268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":511,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.323394+0000","last_change":"2024-03-11T21:32:01.016344+0000","last_active":"2024-03-11T21:33:05.323394+0000","last_peered":"2024-03-11T21:33:05.323394+0000","last_clean":"2024-03-11T21:33:05.323394+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:33:05.323394+0000","last_undegraded":"2024-03-11T21:33:05.323394+0000","last_fullsized":"2024-03-11T21:33:05.323394+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.016240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.016240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.304037+0000","last_change":"2024-03-11T21:30:40.909457+0000","last_active":"2024-03-11T21:33:02.304037+0000","last_peered":"2024-03-11T21:33:02.304037+0000","last_clean":"2024-03-11T21:33:02.304037+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:33:02.304037+0000","last_undegraded":"2024-03-11T21:33:02.304037+0000","last_fullsized":"2024-03-11T21:33:02.304037+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.909356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.909356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.297886+0000","last_change":"2024-03-11T21:32:25.655891+0000","last_active":"2024-03-11T21:33:02.297886+0000","last_peered":"2024-03-11T21:33:02.297886+0000","last_clean":"2024-03-11T21:33:02.297886+0000","last_became_active":"2024-03-11T21:28:51.388058+0000","last_became_peered":"2024-03-11T21:28:51.388058+0000","last_unstale":"2024-03-11T21:33:02.297886+0000","last_undegraded":"2024-03-11T21:33:02.297886+0000","last_fullsized":"2024-03-11T21:33:02.297886+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.655683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:53.974261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":684,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300445+0000","last_change":"2024-03-11T21:31:37.378913+0000","last_active":"2024-03-11T21:33:02.300445+0000","last_peered":"2024-03-11T21:33:02.300445+0000","last_clean":"2024-03-11T21:33:02.300445+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:33:02.300445+0000","last_undegraded":"2024-03-11T21:33:02.300445+0000","last_fullsized":"2024-03-11T21:33:02.300445+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:30:24.997673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.997673+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.580500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":662,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.344116+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:33:02.344116+0000","last_peered":"2024-03-11T21:33:02.344116+0000","last_clean":"2024-03-11T21:33:02.344116+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:33:02.344116+0000","last_undegraded":"2024-03-11T21:33:02.344116+0000","last_fullsized":"2024-03-11T21:33:02.344116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:50.107199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":654,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300637+0000","last_change":"2024-03-11T21:32:09.768728+0000","last_active":"2024-03-11T21:33:02.300637+0000","last_peered":"2024-03-11T21:33:02.300637+0000","last_clean":"2024-03-11T21:33:02.300637+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:33:02.300637+0000","last_undegraded":"2024-03-11T21:33:02.300637+0000","last_fullsized":"2024-03-11T21:33:02.300637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.768608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.768608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.029403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.350316+0000","last_change":"2024-03-11T21:30:36.777188+0000","last_active":"2024-03-11T21:33:02.350316+0000","last_peered":"2024-03-11T21:33:02.350316+0000","last_clean":"2024-03-11T21:33:02.350316+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:33:02.350316+0000","last_undegraded":"2024-03-11T21:33:02.350316+0000","last_fullsized":"2024-03-11T21:33:02.350316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.777107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.777107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":507,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.342392+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:33:02.342392+0000","last_peered":"2024-03-11T21:33:02.342392+0000","last_clean":"2024-03-11T21:33:02.342392+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:33:02.342392+0000","last_undegraded":"2024-03-11T21:33:02.342392+0000","last_fullsized":"2024-03-11T21:33:02.342392+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:05.583739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"84'12","reported_seq":680,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.773971+0000","last_change":"2024-03-11T21:32:09.759891+0000","last_active":"2024-03-11T21:33:02.773971+0000","last_peered":"2024-03-11T21:33:02.773971+0000","last_clean":"2024-03-11T21:33:02.773971+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:33:02.773971+0000","last_undegraded":"2024-03-11T21:33:02.773971+0000","last_fullsized":"2024-03-11T21:33:02.773971+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:32:09.759634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.759634+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.521604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.333610+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:33:02.333610+0000","last_peered":"2024-03-11T21:33:02.333610+0000","last_clean":"2024-03-11T21:33:02.333610+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:33:02.333610+0000","last_undegraded":"2024-03-11T21:33:02.333610+0000","last_fullsized":"2024-03-11T21:33:02.333610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.467463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":645,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.771878+0000","last_change":"2024-03-11T21:30:33.024945+0000","last_active":"2024-03-11T21:33:02.771878+0000","last_peered":"2024-03-11T21:33:02.771878+0000","last_clean":"2024-03-11T21:33:02.771878+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:33:02.771878+0000","last_undegraded":"2024-03-11T21:33:02.771878+0000","last_fullsized":"2024-03-11T21:33:02.771878+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:33.024687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:33.024687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":662,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.312683+0000","last_change":"2024-03-11T21:32:01.014273+0000","last_active":"2024-03-11T21:33:03.312683+0000","last_peered":"2024-03-11T21:33:03.312683+0000","last_clean":"2024-03-11T21:33:03.312683+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:33:03.312683+0000","last_undegraded":"2024-03-11T21:33:03.312683+0000","last_fullsized":"2024-03-11T21:33:03.312683+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.014168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.014168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:13.372767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":509,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.303567+0000","last_change":"2024-03-11T21:32:15.652648+0000","last_active":"2024-03-11T21:33:02.303567+0000","last_peered":"2024-03-11T21:33:02.303567+0000","last_clean":"2024-03-11T21:33:02.303567+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:33:02.303567+0000","last_undegraded":"2024-03-11T21:33:02.303567+0000","last_fullsized":"2024-03-11T21:33:02.303567+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.652323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.652323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:22.960005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"120'22","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.774019+0000","last_change":"2024-03-11T21:31:37.388352+0000","last_active":"2024-03-11T21:33:02.774019+0000","last_peered":"2024-03-11T21:33:02.774019+0000","last_clean":"2024-03-11T21:33:02.774019+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:33:02.774019+0000","last_undegraded":"2024-03-11T21:33:02.774019+0000","last_fullsized":"2024-03-11T21:33:02.774019+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'22","last_scrub_stamp":"2024-03-11T21:30:42.940256+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.940256+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.338638+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:33:02.338638+0000","last_peered":"2024-03-11T21:33:02.338638+0000","last_clean":"2024-03-11T21:33:02.338638+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:33:02.338638+0000","last_undegraded":"2024-03-11T21:33:02.338638+0000","last_fullsized":"2024-03-11T21:33:02.338638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.575015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":653,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.332666+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:33:02.332666+0000","last_peered":"2024-03-11T21:33:02.332666+0000","last_clean":"2024-03-11T21:33:02.332666+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:33:02.332666+0000","last_undegraded":"2024-03-11T21:33:02.332666+0000","last_fullsized":"2024-03-11T21:33:02.332666+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":651,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.767956+0000","last_change":"2024-03-11T21:31:37.387295+0000","last_active":"2024-03-11T21:33:02.767956+0000","last_peered":"2024-03-11T21:33:02.767956+0000","last_clean":"2024-03-11T21:33:02.767956+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:33:02.767956+0000","last_undegraded":"2024-03-11T21:33:02.767956+0000","last_fullsized":"2024-03-11T21:33:02.767956+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.009609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.009609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":507,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.335412+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:33:02.335412+0000","last_peered":"2024-03-11T21:33:02.335412+0000","last_clean":"2024-03-11T21:33:02.335412+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:33:02.335412+0000","last_undegraded":"2024-03-11T21:33:02.335412+0000","last_fullsized":"2024-03-11T21:33:02.335412+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"121'16","reported_seq":685,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320729+0000","last_change":"2024-03-11T21:30:35.925775+0000","last_active":"2024-03-11T21:33:03.320729+0000","last_peered":"2024-03-11T21:33:03.320729+0000","last_clean":"2024-03-11T21:33:03.320729+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:33:03.320729+0000","last_undegraded":"2024-03-11T21:33:03.320729+0000","last_fullsized":"2024-03-11T21:33:03.320729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:30:35.925542+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.925542+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.347901+0000","last_change":"2024-03-11T21:30:30.892128+0000","last_active":"2024-03-11T21:33:02.347901+0000","last_peered":"2024-03-11T21:33:02.347901+0000","last_clean":"2024-03-11T21:33:02.347901+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:33:02.347901+0000","last_undegraded":"2024-03-11T21:33:02.347901+0000","last_fullsized":"2024-03-11T21:33:02.347901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:30.891767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.891767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":515,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.318573+0000","last_change":"2024-03-11T21:32:02.053080+0000","last_active":"2024-03-11T21:33:05.318573+0000","last_peered":"2024-03-11T21:33:05.318573+0000","last_clean":"2024-03-11T21:33:05.318573+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:33:05.318573+0000","last_undegraded":"2024-03-11T21:33:05.318573+0000","last_fullsized":"2024-03-11T21:33:05.318573+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.052865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.052865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.499264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":659,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.300299+0000","last_change":"2024-03-11T21:32:07.733765+0000","last_active":"2024-03-11T21:33:02.300299+0000","last_peered":"2024-03-11T21:33:02.300299+0000","last_clean":"2024-03-11T21:33:02.300299+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:33:02.300299+0000","last_undegraded":"2024-03-11T21:33:02.300299+0000","last_fullsized":"2024-03-11T21:33:02.300299+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.733556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.733556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.680676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":365,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:33:05.321070+0000","last_change":"2024-03-11T21:31:37.397343+0000","last_active":"2024-03-11T21:33:05.321070+0000","last_peered":"2024-03-11T21:33:05.321070+0000","last_clean":"2024-03-11T21:33:05.321070+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:33:05.321070+0000","last_undegraded":"2024-03-11T21:33:05.321070+0000","last_fullsized":"2024-03-11T21:33:05.321070+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:21.708539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.708539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":665,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.348002+0000","last_change":"2024-03-11T21:29:09.487728+0000","last_active":"2024-03-11T21:33:02.348002+0000","last_peered":"2024-03-11T21:33:02.348002+0000","last_clean":"2024-03-11T21:33:02.348002+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:33:02.348002+0000","last_undegraded":"2024-03-11T21:33:02.348002+0000","last_fullsized":"2024-03-11T21:33:02.348002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:29:09.487592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:29:09.487592+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":666,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:33:03.320734+0000","last_change":"2024-03-11T21:30:16.988987+0000","last_active":"2024-03-11T21:33:03.320734+0000","last_peered":"2024-03-11T21:33:03.320734+0000","last_clean":"2024-03-11T21:33:03.320734+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:33:03.320734+0000","last_undegraded":"2024-03-11T21:33:03.320734+0000","last_fullsized":"2024-03-11T21:33:03.320734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:30:16.988898+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:30:16.988898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":664,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.298163+0000","last_change":"2024-03-11T21:31:58.673043+0000","last_active":"2024-03-11T21:33:02.298163+0000","last_peered":"2024-03-11T21:33:02.298163+0000","last_clean":"2024-03-11T21:33:02.298163+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:33:02.298163+0000","last_undegraded":"2024-03-11T21:33:02.298163+0000","last_fullsized":"2024-03-11T21:33:02.298163+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.672796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.672796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.012613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-03-11T21:33:02.343759+0000","last_change":"2024-03-11T21:32:10.686985+0000","last_active":"2024-03-11T21:33:02.343759+0000","last_peered":"2024-03-11T21:33:02.343759+0000","last_clean":"2024-03-11T21:33:02.343759+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:33:02.343759+0000","last_undegraded":"2024-03-11T21:33:02.343759+0000","last_fullsized":"2024-03-11T21:33:02.343759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.686767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.686767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.081264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":267,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":266,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":2},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195524,"num_objects":124,"num_object_clones":2,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26075136,"data_stored":24586572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1805,"ondisk_log_size":1805,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":6400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":579,"num_read_kb":574,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841667,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74152,"kb_used_data":11016,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93671000,"statfs":{"total":95995035648,"available":95919104000,"internally_reserved":0,"allocated":11280384,"data_stored":8676716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168716,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82128,"kb_used_data":10672,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93663024,"statfs":{"total":95995035648,"available":95910936576,"internally_reserved":0,"allocated":10928128,"data_stored":8468874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352339,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42516,"kb_used_data":6248,"kb_used_omap":27,"kb_used_meta":36196,"kb_avail":93702636,"statfs":{"total":95995035648,"available":95951499264,"internally_reserved":0,"allocated":6397952,"data_stored":3911408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":37065033},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77932,"kb_used_data":14796,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93667220,"statfs":{"total":95995035648,"available":95915233280,"internally_reserved":0,"allocated":15151104,"data_stored":12675273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378806,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81868,"kb_used_data":6264,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663284,"statfs":{"total":95995035648,"available":95911202816,"internally_reserved":0,"allocated":6414336,"data_stored":3879500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117162,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81880,"kb_used_data":6264,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663272,"statfs":{"total":95995035648,"available":95911190528,"internally_reserved":0,"allocated":6414336,"data_stored":3889579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855520,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73372,"kb_used_data":10236,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93671780,"statfs":{"total":95995035648,"available":95919902720,"internally_reserved":0,"allocated":10481664,"data_stored":7890155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495766,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85856,"kb_used_data":10252,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659296,"statfs":{"total":95995035648,"available":95907119104,"internally_reserved":0,"allocated":10498048,"data_stored":8072097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8327168,"data_stored":8194649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4000633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4341760,"data_stored":4194682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4000366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4194856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":266,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":266,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:33:08.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:33:08.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:08.372 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:08.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:08.502 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:08.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:08.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: ok, hit_set contains 264:602f83fe:::foo:head 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8991 ms) 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg_num = 32 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 0 ls 1710192789,0 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 1 ls 1710192789,0 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 2 ls 1710192789,0 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 3 ls 1710192789,0 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 4 ls 1710192789,0 2024-03-11T21:33:08.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 5 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 6 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 7 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 8 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 9 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 10 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 11 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 12 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 13 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 14 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 15 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 16 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 17 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 18 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 19 ls 1710192789,0 2024-03-11T21:33:08.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 20 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 21 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 22 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 23 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 24 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 25 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 26 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 27 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 28 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 29 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 30 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 31 ls 1710192789,0 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg_num = 32 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6cac518f:::0:head 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:02547ec2:::1:head 2024-03-11T21:33:08.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f905c69b:::2:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:cfc208b3:::3:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d83876eb:::4:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b29083e3:::5:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c4fdafeb:::6:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:5c6b0b28:::7:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:bd63b0f1:::8:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e960b815:::9:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:52ea6a34:::10:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:89d3ae78:::11:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:de5d7c5f:::12:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:566253c9:::13:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:62a1935d:::14:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:863748b0:::15:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3958e169:::16:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4d4dabf9:::17:head 2024-03-11T21:33:08.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:8391935d:::18:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:28883081:::19:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:69259c59:::20:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4bdb80b7:::21:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a11c5d71:::22:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:271af37b:::23:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:95b121be:::24:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:58d1031b:::25:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:0a050783:::26:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c709704c:::27:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:cbe56eaf:::28:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:86b4b162:::29:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:70d89383:::30:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:dd450c7c:::31:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6d5729b1:::32:head 2024-03-11T21:33:08.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c388f3fb:::33:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:56cfea31:::34:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9dbc1bf7:::35:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:40b74ccd:::36:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4d5aaf42:::37:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:920f362c:::38:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6cc53222:::39:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9cad833f:::40:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:1ea84d41:::41:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c4480ef6:::42:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a694361e:::43:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d1bd33e9:::44:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ddc2cd5d:::45:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:2b782207:::46:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7b187fca:::47:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:90ecdf6f:::48:head 2024-03-11T21:33:08.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a5ed95fe:::49:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ea0eaa55:::50:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f33ef17b:::51:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a0d1b2f6:::52:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:60c5229e:::53:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:edcbc575:::54:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:102cf253:::55:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:efb7fb0b:::56:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:50d0a326:::57:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d4dc5daf:::58:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3a130462:::59:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ec87ed71:::60:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d5bc9454:::61:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3ddfe313:::62:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7c2816b9:::63:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:47e00e4d:::64:head 2024-03-11T21:33:08.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c6410c18:::65:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b48ed237:::66:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:cd63ad31:::67:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b179e92b:::68:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:0d9f741a:::69:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6d3352ae:::70:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c6d5c19e:::71:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:bc4729c3:::72:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:77e930b9:::73:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:0abeecfd:::74:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b7c37e15:::75:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b6378398:::76:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:02bd68de:::77:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:cc795d2d:::78:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:630d4fea:::79:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e0d29ef5:::80:head 2024-03-11T21:33:08.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:fd6f13d2:::81:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:606461d5:::82:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:eadbdc43:::83:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:8761d0bb:::84:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9ef0186f:::85:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e0d41294:::86:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:961de695:::87:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:1423148f:::88:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:633a8fa2:::89:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a8653809:::90:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3dac8b33:::91:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:35aad435:::92:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f6dcc343:::93:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:dbbdad87:::94:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:1cb48ce0:::95:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:03cd461c:::96:head 2024-03-11T21:33:08.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:17a4ea99:::97:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9993c9a7:::98:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6394211c:::99:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:94c7ae57:::100:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6fdee5bb:::101:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9a477fd1:::102:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:eb850916:::103:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:affc56b9:::104:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b42dc814:::105:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f319f8f0:::106:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9a40b9de:::107:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:8b524f28:::108:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e3de589f:::109:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:90f90a5b:::110:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a7b4f1d7:::111:head 2024-03-11T21:33:08.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:af51766e:::112:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b6f90bd1:::113:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e0261208:::114:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c9569ef7:::115:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:61bebe50:::116:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:fe93412b:::117:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d3d38bee:::118:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3100ba0c:::119:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d0560ada:::120:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f0ea8b35:::121:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:766f231a:::122:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a07a2582:::123:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:bd7c6b3a:::124:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:fb2ddaff:::125:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4408e1fe:::126:head 2024-03-11T21:33:08.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ee1df7a7:::127:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c3002909:::128:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4f48ffa9:::129:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:edf38733:::130:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c08425c0:::131:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:5f902d98:::132:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:41ea2c93:::133:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:813cee13:::134:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:0131818d:::135:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:26ba5a85:::136:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:381b8a5a:::137:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:28797e47:::138:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:bfca7f22:::139:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:36807075:::140:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:80b03975:::141:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:5c15709b:::142:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f39ea15e:::143:head 2024-03-11T21:33:08.684 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ea992956:::144:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:48887b1c:::145:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:9f24a9dd:::146:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:987f100b:::147:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d2dd3581:::148:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7fed1808:::149:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c80b70e9:::150:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:85ed90f9:::151:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:36428b24:::152:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d044c34a:::153:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7c18bf58:::154:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d1c21232:::155:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a7a3c575:::156:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:87da0633:::157:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d5ac3822:::158:head 2024-03-11T21:33:08.685 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3f20522d:::159:head 2024-03-11T21:33:08.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6ca26563:::160:head 2024-03-11T21:33:08.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:532ce135:::161:head 2024-03-11T21:33:08.686 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:c78863e6:::162:head 2024-03-11T21:33:08.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:08.755 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:08.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:08.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:33:08.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: audit 2024-03-11T21:33:07.263254+0000 mgr.y (mgr.14550) 404 : audit 0 from='client.41504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.342965+0000 mgr.y (mgr.14550) 405 : cluster 0 pgmap v567: 279 pgs: 4 creating+peering, 32 unknown, 243 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.918047+0000 osd.1 (osd.1) 37 : cluster 0 5.12 scrub starts 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.920495+0000 osd.1 (osd.1) 38 : cluster 0 5.12 scrub ok 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.931164+0000 osd.0 (osd.0) 223 : cluster 0 118.4 scrub starts 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:07.934629+0000 osd.0 (osd.0) 224 : cluster 0 118.4 scrub ok 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: audit 2024-03-11T21:33:08.104793+0000 mon.c (mon.2) 821 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.184263+0000 osd.2 (osd.2) 183 : cluster 0 2.10 scrub starts 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.186722+0000 osd.2 (osd.2) 184 : cluster 0 2.10 scrub ok 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.260970+0000 osd.5 (osd.5) 237 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.261339+0000 osd.5 (osd.5) 238 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: audit 2024-03-11T21:33:08.334483+0000 mon.a (mon.0) 2419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T21:33:08.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.343138+0000 mon.a (mon.0) 2420 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: audit 2024-03-11T21:33:08.345984+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.582867+0000 osd.4 (osd.4) 199 : cluster 0 39.3 scrub starts 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.584158+0000 osd.4 (osd.4) 200 : cluster 0 39.3 scrub ok 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[21512]: audit 2024-03-11T21:33:08.699942+0000 mon.c (mon.2) 822 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: audit 2024-03-11T21:33:07.263254+0000 mgr.y (mgr.14550) 404 : audit 0 from='client.41504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.342965+0000 mgr.y (mgr.14550) 405 : cluster 0 pgmap v567: 279 pgs: 4 creating+peering, 32 unknown, 243 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.918047+0000 osd.1 (osd.1) 37 : cluster 0 5.12 scrub starts 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.920495+0000 osd.1 (osd.1) 38 : cluster 0 5.12 scrub ok 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.931164+0000 osd.0 (osd.0) 223 : cluster 0 118.4 scrub starts 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:07.934629+0000 osd.0 (osd.0) 224 : cluster 0 118.4 scrub ok 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: audit 2024-03-11T21:33:08.104793+0000 mon.c (mon.2) 821 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.184263+0000 osd.2 (osd.2) 183 : cluster 0 2.10 scrub starts 2024-03-11T21:33:08.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.186722+0000 osd.2 (osd.2) 184 : cluster 0 2.10 scrub ok 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.260970+0000 osd.5 (osd.5) 237 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.261339+0000 osd.5 (osd.5) 238 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: audit 2024-03-11T21:33:08.334483+0000 mon.a (mon.0) 2419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.343138+0000 mon.a (mon.0) 2420 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: audit 2024-03-11T21:33:08.345984+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.582867+0000 osd.4 (osd.4) 199 : cluster 0 39.3 scrub starts 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.584158+0000 osd.4 (osd.4) 200 : cluster 0 39.3 scrub ok 2024-03-11T21:33:08.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:08 smithi143 bash[29284]: audit 2024-03-11T21:33:08.699942+0000 mon.c (mon.2) 822 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:08.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:08.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:33:08.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:09.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:09.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.147 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: audit 2024-03-11T21:33:07.263254+0000 mgr.y (mgr.14550) 404 : audit 0 from='client.41504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:09.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.342965+0000 mgr.y (mgr.14550) 405 : cluster 0 pgmap v567: 279 pgs: 4 creating+peering, 32 unknown, 243 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-03-11T21:33:09.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.918047+0000 osd.1 (osd.1) 37 : cluster 0 5.12 scrub starts 2024-03-11T21:33:09.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.920495+0000 osd.1 (osd.1) 38 : cluster 0 5.12 scrub ok 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.931164+0000 osd.0 (osd.0) 223 : cluster 0 118.4 scrub starts 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:07.934629+0000 osd.0 (osd.0) 224 : cluster 0 118.4 scrub ok 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: audit 2024-03-11T21:33:08.104793+0000 mon.c (mon.2) 821 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.184263+0000 osd.2 (osd.2) 183 : cluster 0 2.10 scrub starts 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.186722+0000 osd.2 (osd.2) 184 : cluster 0 2.10 scrub ok 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.260970+0000 osd.5 (osd.5) 237 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.261339+0000 osd.5 (osd.5) 238 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: audit 2024-03-11T21:33:08.334483+0000 mon.a (mon.0) 2419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.343138+0000 mon.a (mon.0) 2420 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: audit 2024-03-11T21:33:08.345984+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.582867+0000 osd.4 (osd.4) 199 : cluster 0 39.3 scrub starts 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.584158+0000 osd.4 (osd.4) 200 : cluster 0 39.3 scrub ok 2024-03-11T21:33:09.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:08 smithi203 bash[23654]: audit 2024-03-11T21:33:08.699942+0000 mon.c (mon.2) 822 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi143-71773-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T21:33:09.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:09.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:09.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:09.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.659 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.789 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:10.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:10.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:10.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:08.561057+0000 mgr.y (mgr.14550) 406 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:10.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:08.885716+0000 mon.c (mon.2) 823 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:08.886250+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:08.887101+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.056 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:08.887512+0000 mon.a (mon.0) 2423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.888766+0000 osd.0 (osd.0) 225 : cluster 0 118.1 scrub starts 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: cluster 2024-03-11T21:33:08.891057+0000 osd.0 (osd.0) 226 : cluster 0 118.1 scrub ok 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:09.106271+0000 mon.c (mon.2) 825 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:09.336637+0000 mon.a (mon.0) 2424 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:09.336718+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]': finished 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.340655+0000 mon.a (mon.0) 2426 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: audit 2024-03-11T21:33:09.344087+0000 mon.a (mon.0) 2427 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:10.057 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.414946+0000 osd.5 (osd.5) 239 : cluster 0 2.2 scrub starts 2024-03-11T21:33:10.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:09 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.416862+0000 osd.5 (osd.5) 240 : cluster 0 2.2 scrub ok 2024-03-11T21:33:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:10.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:08.561057+0000 mgr.y (mgr.14550) 406 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:08.885716+0000 mon.c (mon.2) 823 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:08.886250+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:08.887101+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:08.887512+0000 mon.a (mon.0) 2423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.888766+0000 osd.0 (osd.0) 225 : cluster 0 118.1 scrub starts 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: cluster 2024-03-11T21:33:08.891057+0000 osd.0 (osd.0) 226 : cluster 0 118.1 scrub ok 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:09.106271+0000 mon.c (mon.2) 825 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:09.336637+0000 mon.a (mon.0) 2424 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:09.336718+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]': finished 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.340655+0000 mon.a (mon.0) 2426 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: audit 2024-03-11T21:33:09.344087+0000 mon.a (mon.0) 2427 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.414946+0000 osd.5 (osd.5) 239 : cluster 0 2.2 scrub starts 2024-03-11T21:33:10.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.416862+0000 osd.5 (osd.5) 240 : cluster 0 2.2 scrub ok 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:08.561057+0000 mgr.y (mgr.14550) 406 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:08.885716+0000 mon.c (mon.2) 823 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:08.886250+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:08.887101+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:08.887512+0000 mon.a (mon.0) 2423 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.888766+0000 osd.0 (osd.0) 225 : cluster 0 118.1 scrub starts 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: cluster 2024-03-11T21:33:08.891057+0000 osd.0 (osd.0) 226 : cluster 0 118.1 scrub ok 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:09.106271+0000 mon.c (mon.2) 825 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:09.336637+0000 mon.a (mon.0) 2424 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:09.336718+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-43"}]': finished 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.340655+0000 mon.a (mon.0) 2426 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:33:10.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: audit 2024-03-11T21:33:09.344087+0000 mon.a (mon.0) 2427 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]: dispatch 2024-03-11T21:33:10.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.414946+0000 osd.5 (osd.5) 239 : cluster 0 2.2 scrub starts 2024-03-11T21:33:10.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:09 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.416862+0000 osd.5 (osd.5) 240 : cluster 0 2.2 scrub ok 2024-03-11T21:33:10.189 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:10.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:10.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:10.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:10.453 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:10.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:10.583 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:10.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:10.711 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:10.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:10.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:10.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:10.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:11.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:11.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.343518+0000 mgr.y (mgr.14550) 407 : cluster 0 pgmap v570: 271 pgs: 271 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:33:11.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.648476+0000 osd.6 (osd.6) 175 : cluster 0 5.2 scrub starts 2024-03-11T21:33:11.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.651087+0000 osd.6 (osd.6) 176 : cluster 0 5.2 scrub ok 2024-03-11T21:33:11.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.673315+0000 osd.7 (osd.7) 257 : cluster 0 6.15 scrub starts 2024-03-11T21:33:11.079 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.675704+0000 osd.7 (osd.7) 258 : cluster 0 6.15 scrub ok 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.909697+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:09.912876+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.107490+0000 mon.c (mon.2) 826 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.160386+0000 osd.2 (osd.2) 185 : cluster 0 39.6 scrub starts 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.162816+0000 osd.2 (osd.2) 186 : cluster 0 39.6 scrub ok 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.348111+0000 mon.a (mon.0) 2428 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.355078+0000 mon.a (mon.0) 2429 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.392721+0000 mon.c (mon.2) 827 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.602329+0000 osd.4 (osd.4) 201 : cluster 0 39.15 scrub starts 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.604152+0000 osd.4 (osd.4) 202 : cluster 0 39.15 scrub ok 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.813144+0000 mon.a (mon.0) 2430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.813761+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.814055+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.814556+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:10 smithi203 bash[23654]: audit 2024-03-11T21:33:10.814862+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.343518+0000 mgr.y (mgr.14550) 407 : cluster 0 pgmap v570: 271 pgs: 271 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.648476+0000 osd.6 (osd.6) 175 : cluster 0 5.2 scrub starts 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.651087+0000 osd.6 (osd.6) 176 : cluster 0 5.2 scrub ok 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.673315+0000 osd.7 (osd.7) 257 : cluster 0 6.15 scrub starts 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.675704+0000 osd.7 (osd.7) 258 : cluster 0 6.15 scrub ok 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.909697+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T21:33:11.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:09.912876+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.107490+0000 mon.c (mon.2) 826 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.160386+0000 osd.2 (osd.2) 185 : cluster 0 39.6 scrub starts 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.162816+0000 osd.2 (osd.2) 186 : cluster 0 39.6 scrub ok 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.348111+0000 mon.a (mon.0) 2428 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.355078+0000 mon.a (mon.0) 2429 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.392721+0000 mon.c (mon.2) 827 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.602329+0000 osd.4 (osd.4) 201 : cluster 0 39.15 scrub starts 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.604152+0000 osd.4 (osd.4) 202 : cluster 0 39.15 scrub ok 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.813144+0000 mon.a (mon.0) 2430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.813761+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.814055+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.814556+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[21512]: audit 2024-03-11T21:33:10.814862+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.343518+0000 mgr.y (mgr.14550) 407 : cluster 0 pgmap v570: 271 pgs: 271 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.648476+0000 osd.6 (osd.6) 175 : cluster 0 5.2 scrub starts 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.651087+0000 osd.6 (osd.6) 176 : cluster 0 5.2 scrub ok 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.673315+0000 osd.7 (osd.7) 257 : cluster 0 6.15 scrub starts 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.675704+0000 osd.7 (osd.7) 258 : cluster 0 6.15 scrub ok 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.909697+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:09.912876+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.107490+0000 mon.c (mon.2) 826 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.160386+0000 osd.2 (osd.2) 185 : cluster 0 39.6 scrub starts 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.162816+0000 osd.2 (osd.2) 186 : cluster 0 39.6 scrub ok 2024-03-11T21:33:11.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.348111+0000 mon.a (mon.0) 2428 : audit 1 from='client.? 172.21.15.143:0/3042075408' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi143-71604-58"}]': finished 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.355078+0000 mon.a (mon.0) 2429 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.392721+0000 mon.c (mon.2) 827 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.602329+0000 osd.4 (osd.4) 201 : cluster 0 39.15 scrub starts 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.604152+0000 osd.4 (osd.4) 202 : cluster 0 39.15 scrub ok 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.813144+0000 mon.a (mon.0) 2430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.813761+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.814055+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.814556+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:10 smithi143 bash[29284]: audit 2024-03-11T21:33:10.814862+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:11.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:11.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:11.273 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:11.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:11.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:11.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.473 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:11.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.602 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:11.705 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:11.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:11.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:11.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.198630+0000 osd.3 (osd.3) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:11.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.198894+0000 osd.3 (osd.3) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:11.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.434762+0000 osd.5 (osd.5) 241 : cluster 0 2.12 scrub starts 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.436022+0000 osd.5 (osd.5) 242 : cluster 0 2.12 scrub ok 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.604801+0000 osd.6 (osd.6) 177 : cluster 0 5.1f scrub starts 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.606352+0000 osd.6 (osd.6) 178 : cluster 0 5.1f scrub ok 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.704646+0000 osd.7 (osd.7) 259 : cluster 0 2.0 scrub starts 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.706015+0000 osd.3 (osd.3) 233 : cluster 0 5.0 scrub starts 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.707080+0000 osd.7 (osd.7) 260 : cluster 0 2.0 scrub ok 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.708710+0000 osd.3 (osd.3) 234 : cluster 0 5.0 scrub ok 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.108854+0000 mon.c (mon.2) 830 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.109393+0000 mon.c (mon.2) 831 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.109687+0000 mon.a (mon.0) 2433 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.296469+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.296742+0000 mon.a (mon.0) 2435 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: cluster 2024-03-11T21:33:11.304320+0000 mon.a (mon.0) 2436 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:33:11.849 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.308436+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.308830+0000 mon.c (mon.2) 833 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.318634+0000 mon.c (mon.2) 834 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.319545+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.319701+0000 mon.c (mon.2) 836 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.324275+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.325833+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:11.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:11 smithi203 bash[23654]: audit 2024-03-11T21:33:11.326382+0000 mon.a (mon.0) 2439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:11.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:11.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.994 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:12.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.198630+0000 osd.3 (osd.3) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.198894+0000 osd.3 (osd.3) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.434762+0000 osd.5 (osd.5) 241 : cluster 0 2.12 scrub starts 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.436022+0000 osd.5 (osd.5) 242 : cluster 0 2.12 scrub ok 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.604801+0000 osd.6 (osd.6) 177 : cluster 0 5.1f scrub starts 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.606352+0000 osd.6 (osd.6) 178 : cluster 0 5.1f scrub ok 2024-03-11T21:33:12.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.704646+0000 osd.7 (osd.7) 259 : cluster 0 2.0 scrub starts 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.706015+0000 osd.3 (osd.3) 233 : cluster 0 5.0 scrub starts 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.707080+0000 osd.7 (osd.7) 260 : cluster 0 2.0 scrub ok 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.708710+0000 osd.3 (osd.3) 234 : cluster 0 5.0 scrub ok 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.108854+0000 mon.c (mon.2) 830 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.109393+0000 mon.c (mon.2) 831 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.109687+0000 mon.a (mon.0) 2433 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.296469+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.296742+0000 mon.a (mon.0) 2435 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:33:12.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: cluster 2024-03-11T21:33:11.304320+0000 mon.a (mon.0) 2436 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:33:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.308436+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.308830+0000 mon.c (mon.2) 833 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.318634+0000 mon.c (mon.2) 834 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.319545+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:12.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.198630+0000 osd.3 (osd.3) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.198894+0000 osd.3 (osd.3) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.434762+0000 osd.5 (osd.5) 241 : cluster 0 2.12 scrub starts 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.436022+0000 osd.5 (osd.5) 242 : cluster 0 2.12 scrub ok 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.604801+0000 osd.6 (osd.6) 177 : cluster 0 5.1f scrub starts 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.606352+0000 osd.6 (osd.6) 178 : cluster 0 5.1f scrub ok 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.704646+0000 osd.7 (osd.7) 259 : cluster 0 2.0 scrub starts 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.706015+0000 osd.3 (osd.3) 233 : cluster 0 5.0 scrub starts 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.707080+0000 osd.7 (osd.7) 260 : cluster 0 2.0 scrub ok 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.708710+0000 osd.3 (osd.3) 234 : cluster 0 5.0 scrub ok 2024-03-11T21:33:12.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.108854+0000 mon.c (mon.2) 830 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.109393+0000 mon.c (mon.2) 831 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.109687+0000 mon.a (mon.0) 2433 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.296469+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.296742+0000 mon.a (mon.0) 2435 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pg_num","val":"11"}]': finished 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: cluster 2024-03-11T21:33:11.304320+0000 mon.a (mon.0) 2436 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.308436+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.308830+0000 mon.c (mon.2) 833 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.318634+0000 mon.c (mon.2) 834 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.319545+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:12.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.319701+0000 mon.c (mon.2) 836 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:12.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.324275+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:12.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.325833+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:12.086 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[21512]: audit 2024-03-11T21:33:11.326382+0000 mon.a (mon.0) 2439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:12.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.319701+0000 mon.c (mon.2) 836 : audit 1 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:12.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.324275+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:12.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.325833+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:12.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:11 smithi143 bash[29284]: audit 2024-03-11T21:33:11.326382+0000 mon.a (mon.0) 2439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T21:33:12.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.126 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:12.229 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:12.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:12.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:12.391 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:12.436 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:12.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:12.522 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:12.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:12.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:12.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:12.882 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:12.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.949495+0000 osd.0 (osd.0) 227 : cluster 0 118.18 scrub starts 2024-03-11T21:33:12.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:10.951938+0000 osd.0 (osd.0) 228 : cluster 0 118.18 scrub ok 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:11.344079+0000 mgr.y (mgr.14550) 408 : cluster 0 pgmap v573: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.196910+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.199220+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: audit 2024-03-11T21:33:12.298809+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: audit 2024-03-11T21:33:12.298951+0000 mon.a (mon.0) 2441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: audit 2024-03-11T21:33:12.303694+0000 mon.c (mon.2) 837 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: audit 2024-03-11T21:33:12.306434+0000 mon.c (mon.2) 838 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.671284+0000 osd.4 (osd.4) 203 : cluster 0 5.e scrub starts 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.673920+0000 osd.4 (osd.4) 204 : cluster 0 5.e scrub ok 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.755886+0000 mon.a (mon.0) 2442 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.756216+0000 mon.a (mon.0) 2443 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:12.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[29284]: audit 2024-03-11T21:33:12.757436+0000 mon.a (mon.0) 2444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.949495+0000 osd.0 (osd.0) 227 : cluster 0 118.18 scrub starts 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:10.951938+0000 osd.0 (osd.0) 228 : cluster 0 118.18 scrub ok 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:11.344079+0000 mgr.y (mgr.14550) 408 : cluster 0 pgmap v573: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.196910+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.199220+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: audit 2024-03-11T21:33:12.298809+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: audit 2024-03-11T21:33:12.298951+0000 mon.a (mon.0) 2441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: audit 2024-03-11T21:33:12.303694+0000 mon.c (mon.2) 837 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: audit 2024-03-11T21:33:12.306434+0000 mon.c (mon.2) 838 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.671284+0000 osd.4 (osd.4) 203 : cluster 0 5.e scrub starts 2024-03-11T21:33:12.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.673920+0000 osd.4 (osd.4) 204 : cluster 0 5.e scrub ok 2024-03-11T21:33:12.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.755886+0000 mon.a (mon.0) 2442 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:33:12.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.756216+0000 mon.a (mon.0) 2443 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:12.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:12 smithi143 bash[21512]: audit 2024-03-11T21:33:12.757436+0000 mon.a (mon.0) 2444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:12.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:13.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.949495+0000 osd.0 (osd.0) 227 : cluster 0 118.18 scrub starts 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:10.951938+0000 osd.0 (osd.0) 228 : cluster 0 118.18 scrub ok 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:11.344079+0000 mgr.y (mgr.14550) 408 : cluster 0 pgmap v573: 271 pgs: 32 unknown, 239 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.196910+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.199220+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: audit 2024-03-11T21:33:12.298809+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: audit 2024-03-11T21:33:12.298951+0000 mon.a (mon.0) 2441 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi143-71636-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T21:33:13.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: audit 2024-03-11T21:33:12.303694+0000 mon.c (mon.2) 837 : audit 0 from='client.? 172.21.15.143:0/3890427852' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: audit 2024-03-11T21:33:12.306434+0000 mon.c (mon.2) 838 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.671284+0000 osd.4 (osd.4) 203 : cluster 0 5.e scrub starts 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.673920+0000 osd.4 (osd.4) 204 : cluster 0 5.e scrub ok 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.755886+0000 mon.a (mon.0) 2442 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.756216+0000 mon.a (mon.0) 2443 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:13.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:12 smithi203 bash[23654]: audit 2024-03-11T21:33:12.757436+0000 mon.a (mon.0) 2444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:13.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:13.088 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:13.089 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:13.089 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:13.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.156 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:13.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:13.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:13.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:13.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:13.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:13.553 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:13.656 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:13.683 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:13.786 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:13.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:13.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:13.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.436436+0000 osd.5 (osd.5) 243 : cluster 0 9.12 scrub starts 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.438017+0000 osd.5 (osd.5) 244 : cluster 0 9.12 scrub ok 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.693829+0000 osd.3 (osd.3) 235 : cluster 0 39.a scrub starts 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:12.694963+0000 osd.3 (osd.3) 236 : cluster 0 39.a scrub ok 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.184458+0000 osd.2 (osd.2) 189 : cluster 0 12.a scrub starts 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.186135+0000 osd.2 (osd.2) 190 : cluster 0 12.a scrub ok 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: audit 2024-03-11T21:33:13.302207+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: audit 2024-03-11T21:33:13.302383+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.310476+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: audit 2024-03-11T21:33:13.312555+0000 mon.c (mon.2) 839 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:13.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[21512]: audit 2024-03-11T21:33:13.313377+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.436436+0000 osd.5 (osd.5) 243 : cluster 0 9.12 scrub starts 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.438017+0000 osd.5 (osd.5) 244 : cluster 0 9.12 scrub ok 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.693829+0000 osd.3 (osd.3) 235 : cluster 0 39.a scrub starts 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:12.694963+0000 osd.3 (osd.3) 236 : cluster 0 39.a scrub ok 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.184458+0000 osd.2 (osd.2) 189 : cluster 0 12.a scrub starts 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.186135+0000 osd.2 (osd.2) 190 : cluster 0 12.a scrub ok 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: audit 2024-03-11T21:33:13.302207+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: audit 2024-03-11T21:33:13.302383+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.310476+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: audit 2024-03-11T21:33:13.312555+0000 mon.c (mon.2) 839 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:13.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:13 smithi143 bash[29284]: audit 2024-03-11T21:33:13.313377+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:13.940 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:14.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:14.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.436436+0000 osd.5 (osd.5) 243 : cluster 0 9.12 scrub starts 2024-03-11T21:33:14.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.438017+0000 osd.5 (osd.5) 244 : cluster 0 9.12 scrub ok 2024-03-11T21:33:14.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.693829+0000 osd.3 (osd.3) 235 : cluster 0 39.a scrub starts 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:12.694963+0000 osd.3 (osd.3) 236 : cluster 0 39.a scrub ok 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.184458+0000 osd.2 (osd.2) 189 : cluster 0 12.a scrub starts 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.186135+0000 osd.2 (osd.2) 190 : cluster 0 12.a scrub ok 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: audit 2024-03-11T21:33:13.302207+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi143-71604-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: audit 2024-03-11T21:33:13.302383+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.310476+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: audit 2024-03-11T21:33:13.312555+0000 mon.c (mon.2) 839 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:14.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:13 smithi203 bash[23654]: audit 2024-03-11T21:33:13.313377+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:33:14.067 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:14.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:14.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:14.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:14.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:14.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:14.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:14.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:14.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:14.720 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:14.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:14.850 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:14.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.344897+0000 mgr.y (mgr.14550) 409 : cluster 0 pgmap v576: 279 pgs: 15 unknown, 264 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.458866+0000 osd.5 (osd.5) 245 : cluster 0 2.3 scrub starts 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.461201+0000 osd.5 (osd.5) 246 : cluster 0 2.3 scrub ok 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.642737+0000 osd.6 (osd.6) 179 : cluster 0 12.1 scrub starts 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.648112+0000 osd.6 (osd.6) 180 : cluster 0 12.1 scrub ok 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.711470+0000 osd.7 (osd.7) 261 : cluster 0 3.d scrub starts 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.716486+0000 osd.7 (osd.7) 262 : cluster 0 3.d scrub ok 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.733736+0000 osd.3 (osd.3) 237 : cluster 0 39.1b scrub starts 2024-03-11T21:33:14.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.735311+0000 osd.3 (osd.3) 238 : cluster 0 39.1b scrub ok 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.939768+0000 osd.1 (osd.1) 41 : cluster 0 5.c scrub starts 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:13.942144+0000 osd.1 (osd.1) 42 : cluster 0 5.c scrub ok 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: audit 2024-03-11T21:33:14.311353+0000 mon.a (mon.0) 2449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.319227+0000 mon.a (mon.0) 2450 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: audit 2024-03-11T21:33:14.322401+0000 mon.c (mon.2) 840 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:14.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:14 smithi203 bash[23654]: audit 2024-03-11T21:33:14.325403+0000 mon.a (mon.0) 2451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:14.983 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:15.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.344897+0000 mgr.y (mgr.14550) 409 : cluster 0 pgmap v576: 279 pgs: 15 unknown, 264 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.458866+0000 osd.5 (osd.5) 245 : cluster 0 2.3 scrub starts 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.461201+0000 osd.5 (osd.5) 246 : cluster 0 2.3 scrub ok 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.642737+0000 osd.6 (osd.6) 179 : cluster 0 12.1 scrub starts 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.648112+0000 osd.6 (osd.6) 180 : cluster 0 12.1 scrub ok 2024-03-11T21:33:15.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.711470+0000 osd.7 (osd.7) 261 : cluster 0 3.d scrub starts 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.716486+0000 osd.7 (osd.7) 262 : cluster 0 3.d scrub ok 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.733736+0000 osd.3 (osd.3) 237 : cluster 0 39.1b scrub starts 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.735311+0000 osd.3 (osd.3) 238 : cluster 0 39.1b scrub ok 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.939768+0000 osd.1 (osd.1) 41 : cluster 0 5.c scrub starts 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:13.942144+0000 osd.1 (osd.1) 42 : cluster 0 5.c scrub ok 2024-03-11T21:33:15.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: audit 2024-03-11T21:33:14.311353+0000 mon.a (mon.0) 2449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.319227+0000 mon.a (mon.0) 2450 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: audit 2024-03-11T21:33:14.322401+0000 mon.c (mon.2) 840 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[29284]: audit 2024-03-11T21:33:14.325403+0000 mon.a (mon.0) 2451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.344897+0000 mgr.y (mgr.14550) 409 : cluster 0 pgmap v576: 279 pgs: 15 unknown, 264 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.458866+0000 osd.5 (osd.5) 245 : cluster 0 2.3 scrub starts 2024-03-11T21:33:15.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.461201+0000 osd.5 (osd.5) 246 : cluster 0 2.3 scrub ok 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.642737+0000 osd.6 (osd.6) 179 : cluster 0 12.1 scrub starts 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.648112+0000 osd.6 (osd.6) 180 : cluster 0 12.1 scrub ok 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.711470+0000 osd.7 (osd.7) 261 : cluster 0 3.d scrub starts 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.716486+0000 osd.7 (osd.7) 262 : cluster 0 3.d scrub ok 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.733736+0000 osd.3 (osd.3) 237 : cluster 0 39.1b scrub starts 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.735311+0000 osd.3 (osd.3) 238 : cluster 0 39.1b scrub ok 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.939768+0000 osd.1 (osd.1) 41 : cluster 0 5.c scrub starts 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:13.942144+0000 osd.1 (osd.1) 42 : cluster 0 5.c scrub ok 2024-03-11T21:33:15.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: audit 2024-03-11T21:33:14.311353+0000 mon.a (mon.0) 2449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:33:15.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.319227+0000 mon.a (mon.0) 2450 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:33:15.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: audit 2024-03-11T21:33:14.322401+0000 mon.c (mon.2) 840 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:15.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:14 smithi143 bash[21512]: audit 2024-03-11T21:33:14.325403+0000 mon.a (mon.0) 2451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:33:15.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:15.248 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:15.322 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:33:15.322 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:33:15.322 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:33:15.322 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:33:15.323 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:33:15.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:15.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:15.503 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:15.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.635 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:15.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.764 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:15.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:15.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.690858+0000 osd.3 (osd.3) 239 : cluster 0 6.b scrub starts 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.693306+0000 osd.3 (osd.3) 240 : cluster 0 6.b scrub ok 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.698819+0000 osd.4 (osd.4) 205 : cluster 0 118.f scrub starts 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.703279+0000 osd.4 (osd.4) 206 : cluster 0 118.f scrub ok 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.934528+0000 osd.1 (osd.1) 43 : cluster 0 118.1f scrub starts 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.938562+0000 osd.1 (osd.1) 44 : cluster 0 118.1f scrub ok 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: audit 2024-03-11T21:33:15.315760+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.324344+0000 mon.a (mon.0) 2453 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: audit 2024-03-11T21:33:15.331420+0000 mon.c (mon.2) 841 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: audit 2024-03-11T21:33:15.331865+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: audit 2024-03-11T21:33:15.332682+0000 mon.c (mon.2) 842 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[21512]: audit 2024-03-11T21:33:15.333028+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.690858+0000 osd.3 (osd.3) 239 : cluster 0 6.b scrub starts 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.693306+0000 osd.3 (osd.3) 240 : cluster 0 6.b scrub ok 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.698819+0000 osd.4 (osd.4) 205 : cluster 0 118.f scrub starts 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.703279+0000 osd.4 (osd.4) 206 : cluster 0 118.f scrub ok 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.934528+0000 osd.1 (osd.1) 43 : cluster 0 118.1f scrub starts 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.938562+0000 osd.1 (osd.1) 44 : cluster 0 118.1f scrub ok 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: audit 2024-03-11T21:33:15.315760+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.324344+0000 mon.a (mon.0) 2453 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: audit 2024-03-11T21:33:15.331420+0000 mon.c (mon.2) 841 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: audit 2024-03-11T21:33:15.331865+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: audit 2024-03-11T21:33:15.332682+0000 mon.c (mon.2) 842 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.003 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:15 smithi143 bash[29284]: audit 2024-03-11T21:33:15.333028+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.014 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.690858+0000 osd.3 (osd.3) 239 : cluster 0 6.b scrub starts 2024-03-11T21:33:16.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.693306+0000 osd.3 (osd.3) 240 : cluster 0 6.b scrub ok 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.698819+0000 osd.4 (osd.4) 205 : cluster 0 118.f scrub starts 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.703279+0000 osd.4 (osd.4) 206 : cluster 0 118.f scrub ok 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.934528+0000 osd.1 (osd.1) 43 : cluster 0 118.1f scrub starts 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.938562+0000 osd.1 (osd.1) 44 : cluster 0 118.1f scrub ok 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: audit 2024-03-11T21:33:15.315760+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.324344+0000 mon.a (mon.0) 2453 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: audit 2024-03-11T21:33:15.331420+0000 mon.c (mon.2) 841 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: audit 2024-03-11T21:33:15.331865+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: audit 2024-03-11T21:33:15.332682+0000 mon.c (mon.2) 842 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:15 smithi203 bash[23654]: audit 2024-03-11T21:33:15.333028+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:16.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:16.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:33:16.144 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:16.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:16.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:16.375 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:16.404 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:16.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:16.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:16.527 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:16.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:16.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.777 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:16.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.891 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.961736+0000 osd.0 (osd.0) 229 : cluster 0 118.0 scrub starts 2024-03-11T21:33:16.891 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:14.964346+0000 osd.0 (osd.0) 230 : cluster 0 118.0 scrub ok 2024-03-11T21:33:16.891 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.345662+0000 mgr.y (mgr.14550) 410 : cluster 0 pgmap v579: 271 pgs: 2 unknown, 269 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.596697+0000 osd.6 (osd.6) 181 : cluster 0 39.8 scrub starts 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.600015+0000 osd.6 (osd.6) 182 : cluster 0 39.8 scrub ok 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.706989+0000 osd.4 (osd.4) 207 : cluster 0 118.c scrub starts 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:15.709868+0000 osd.4 (osd.4) 208 : cluster 0 118.c scrub ok 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.324608+0000 mon.a (mon.0) 2456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.324773+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: cluster 2024-03-11T21:33:16.331431+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.334799+0000 mon.c (mon.2) 843 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:16.892 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.336738+0000 mon.c (mon.2) 844 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.337890+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[21512]: audit 2024-03-11T21:33:16.339336+0000 mon.a (mon.0) 2460 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.961736+0000 osd.0 (osd.0) 229 : cluster 0 118.0 scrub starts 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:14.964346+0000 osd.0 (osd.0) 230 : cluster 0 118.0 scrub ok 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.345662+0000 mgr.y (mgr.14550) 410 : cluster 0 pgmap v579: 271 pgs: 2 unknown, 269 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.596697+0000 osd.6 (osd.6) 181 : cluster 0 39.8 scrub starts 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.600015+0000 osd.6 (osd.6) 182 : cluster 0 39.8 scrub ok 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.706989+0000 osd.4 (osd.4) 207 : cluster 0 118.c scrub starts 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:15.709868+0000 osd.4 (osd.4) 208 : cluster 0 118.c scrub ok 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.324608+0000 mon.a (mon.0) 2456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:16.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.324773+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:16.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: cluster 2024-03-11T21:33:16.331431+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:33:16.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.334799+0000 mon.c (mon.2) 843 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:16.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.336738+0000 mon.c (mon.2) 844 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.337890+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:16.894 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:16 smithi143 bash[29284]: audit 2024-03-11T21:33:16.339336+0000 mon.a (mon.0) 2460 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:16.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:17.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.034 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:17.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.961736+0000 osd.0 (osd.0) 229 : cluster 0 118.0 scrub starts 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:14.964346+0000 osd.0 (osd.0) 230 : cluster 0 118.0 scrub ok 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.345662+0000 mgr.y (mgr.14550) 410 : cluster 0 pgmap v579: 271 pgs: 2 unknown, 269 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.596697+0000 osd.6 (osd.6) 181 : cluster 0 39.8 scrub starts 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.600015+0000 osd.6 (osd.6) 182 : cluster 0 39.8 scrub ok 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.706989+0000 osd.4 (osd.4) 207 : cluster 0 118.c scrub starts 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:15.709868+0000 osd.4 (osd.4) 208 : cluster 0 118.c scrub ok 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.324608+0000 mon.a (mon.0) 2456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.324773+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: cluster 2024-03-11T21:33:16.331431+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:33:17.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.334799+0000 mon.c (mon.2) 843 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:17.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.336738+0000 mon.c (mon.2) 844 : audit 1 from='client.? 172.21.15.143:0/1451317607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:17.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.337890+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:33:17.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:16 smithi203 bash[23654]: audit 2024-03-11T21:33:16.339336+0000 mon.a (mon.0) 2460 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]: dispatch 2024-03-11T21:33:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:17.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:17.299 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:17.320 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7462ddf6io.RoundTripAppendPP (3481 ms) 2024-03-11T21:33:17.320 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2559 ms) 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3019 ms) 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2962 ms) 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-11T21:33:17.321 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: setting pool EIO 2024-03-11T21:33:17.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: max_success 103, min_failed 104 2024-03-11T21:33:17.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4020 ms) 2024-03-11T21:33:17.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-11T21:33:17.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3028 ms) 2024-03-11T21:33:17.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (154723 ms total) 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2959 ms) 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9471 ms) 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14616 ms) 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-11T21:33:17.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3020 ms) 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30066 ms total) 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3367 ms) 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3367 ms total) 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13547 ms) 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-11T21:33:17.324 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7021 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6994 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7040 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2954 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7064 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6959 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6994 ms) 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-11T21:33:17.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7020 ms) 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7016 ms) 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6970 ms) 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7051 ms) 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-11T21:33:17.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6982 ms) 2024-03-11T21:33:17.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.430 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:17.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:17.558 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:17.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:17.686 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:17.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:17.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:17.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:18.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:18.073 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:18.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:18.205 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: cluster 2024-03-11T21:33:16.011623+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub starts 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: cluster 2024-03-11T21:33:16.013747+0000 osd.0 (osd.0) 232 : cluster 0 2.1f scrub ok 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: audit 2024-03-11T21:33:17.328247+0000 mon.a (mon.0) 2461 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: audit 2024-03-11T21:33:17.328483+0000 mon.a (mon.0) 2462 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.336058+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.665322+0000 osd.4 (osd.4) 209 : cluster 0 118.2 scrub starts 2024-03-11T21:33:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:17 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.671651+0000 osd.4 (osd.4) 210 : cluster 0 118.2 scrub ok 2024-03-11T21:33:18.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:18.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:16.011623+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub starts 2024-03-11T21:33:18.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:16.013747+0000 osd.0 (osd.0) 232 : cluster 0 2.1f scrub ok 2024-03-11T21:33:18.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: audit 2024-03-11T21:33:17.328247+0000 mon.a (mon.0) 2461 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:33:18.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: audit 2024-03-11T21:33:17.328483+0000 mon.a (mon.0) 2462 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:18.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.336058+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.665322+0000 osd.4 (osd.4) 209 : cluster 0 118.2 scrub starts 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.671651+0000 osd.4 (osd.4) 210 : cluster 0 118.2 scrub ok 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:16.011623+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub starts 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:16.013747+0000 osd.0 (osd.0) 232 : cluster 0 2.1f scrub ok 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: audit 2024-03-11T21:33:17.328247+0000 mon.a (mon.0) 2461 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:33:18.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: audit 2024-03-11T21:33:17.328483+0000 mon.a (mon.0) 2462 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi143-71604-59"}]': finished 2024-03-11T21:33:18.325 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.336058+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:33:18.325 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.665322+0000 osd.4 (osd.4) 209 : cluster 0 118.2 scrub starts 2024-03-11T21:33:18.325 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.671651+0000 osd.4 (osd.4) 210 : cluster 0 118.2 scrub ok 2024-03-11T21:33:18.342 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:18.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:18.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:18.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:18.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:18.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:18.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:18.839 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:18.869 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:18.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:18.983 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.011442+0000 osd.0 (osd.0) 233 : cluster 0 3.1 scrub starts 2024-03-11T21:33:18.983 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.014173+0000 osd.0 (osd.0) 234 : cluster 0 3.1 scrub ok 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.346941+0000 mgr.y (mgr.14550) 411 : cluster 0 pgmap v582: 271 pgs: 271 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.528105+0000 osd.5 (osd.5) 247 : cluster 0 5.1b scrub starts 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.530288+0000 osd.5 (osd.5) 248 : cluster 0 5.1b scrub ok 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.730442+0000 osd.3 (osd.3) 241 : cluster 0 3.1e scrub starts 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.733063+0000 osd.3 (osd.3) 242 : cluster 0 3.1e scrub ok 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.960817+0000 osd.1 (osd.1) 45 : cluster 0 118.1b scrub starts 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:17.964222+0000 osd.1 (osd.1) 46 : cluster 0 118.1b scrub ok 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: cluster 2024-03-11T21:33:18.345834+0000 mon.a (mon.0) 2464 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[29284]: audit 2024-03-11T21:33:18.348970+0000 mon.a (mon.0) 2465 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.011442+0000 osd.0 (osd.0) 233 : cluster 0 3.1 scrub starts 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.014173+0000 osd.0 (osd.0) 234 : cluster 0 3.1 scrub ok 2024-03-11T21:33:18.984 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.346941+0000 mgr.y (mgr.14550) 411 : cluster 0 pgmap v582: 271 pgs: 271 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.528105+0000 osd.5 (osd.5) 247 : cluster 0 5.1b scrub starts 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.530288+0000 osd.5 (osd.5) 248 : cluster 0 5.1b scrub ok 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.730442+0000 osd.3 (osd.3) 241 : cluster 0 3.1e scrub starts 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.733063+0000 osd.3 (osd.3) 242 : cluster 0 3.1e scrub ok 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.960817+0000 osd.1 (osd.1) 45 : cluster 0 118.1b scrub starts 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:17.964222+0000 osd.1 (osd.1) 46 : cluster 0 118.1b scrub ok 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: cluster 2024-03-11T21:33:18.345834+0000 mon.a (mon.0) 2464 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:33:18.985 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:18 smithi143 bash[21512]: audit 2024-03-11T21:33:18.348970+0000 mon.a (mon.0) 2465 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:18.997 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:19.100 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:19.121 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:19.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:19.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.011442+0000 osd.0 (osd.0) 233 : cluster 0 3.1 scrub starts 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.014173+0000 osd.0 (osd.0) 234 : cluster 0 3.1 scrub ok 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.346941+0000 mgr.y (mgr.14550) 411 : cluster 0 pgmap v582: 271 pgs: 271 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.528105+0000 osd.5 (osd.5) 247 : cluster 0 5.1b scrub starts 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.530288+0000 osd.5 (osd.5) 248 : cluster 0 5.1b scrub ok 2024-03-11T21:33:19.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.730442+0000 osd.3 (osd.3) 241 : cluster 0 3.1e scrub starts 2024-03-11T21:33:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.733063+0000 osd.3 (osd.3) 242 : cluster 0 3.1e scrub ok 2024-03-11T21:33:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.960817+0000 osd.1 (osd.1) 45 : cluster 0 118.1b scrub starts 2024-03-11T21:33:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:17.964222+0000 osd.1 (osd.1) 46 : cluster 0 118.1b scrub ok 2024-03-11T21:33:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: cluster 2024-03-11T21:33:18.345834+0000 mon.a (mon.0) 2464 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:33:19.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:18 smithi203 bash[23654]: audit 2024-03-11T21:33:18.348970+0000 mon.a (mon.0) 2465 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:19.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:19.377 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:19.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:19.508 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:19.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:19.639 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:19.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:19.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:19.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:19.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[21512]: audit 2024-03-11T21:33:18.572279+0000 mgr.y (mgr.14550) 412 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:19.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[21512]: audit 2024-03-11T21:33:19.340137+0000 mon.a (mon.0) 2466 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:19.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.345903+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:33:19.888 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[29284]: audit 2024-03-11T21:33:18.572279+0000 mgr.y (mgr.14550) 412 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:19.888 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[29284]: audit 2024-03-11T21:33:19.340137+0000 mon.a (mon.0) 2466 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:19.888 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:19 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.345903+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:33:19.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:20.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:20.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:19 smithi203 bash[23654]: audit 2024-03-11T21:33:18.572279+0000 mgr.y (mgr.14550) 412 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:20.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:19 smithi203 bash[23654]: audit 2024-03-11T21:33:19.340137+0000 mon.a (mon.0) 2466 : audit 1 from='client.? 172.21.15.143:0/1999396226' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi143-71604-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:20.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:19 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.345903+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:33:20.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:20.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:20.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:20.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:20.297 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:20.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:20.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:20.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:20.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:20.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:20.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:20.679 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:20.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:20.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:20.908 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:20.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.014939+0000 osd.0 (osd.0) 235 : cluster 0 3.1b scrub starts 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.017443+0000 osd.0 (osd.0) 236 : cluster 0 3.1b scrub ok 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.354164+0000 mgr.y (mgr.14550) 413 : cluster 0 pgmap v585: 303 pgs: 3 creating+peering, 29 unknown, 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 510 B/s wr, 24 op/s 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.694865+0000 osd.3 (osd.3) 243 : cluster 0 5.14 scrub starts 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:19.697565+0000 osd.3 (osd.3) 244 : cluster 0 5.14 scrub ok 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.351161+0000 mon.a (mon.0) 2468 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:33:20.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: audit 2024-03-11T21:33:20.388358+0000 mon.a (mon.0) 2469 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: audit 2024-03-11T21:33:20.389444+0000 mon.a (mon.0) 2470 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: audit 2024-03-11T21:33:20.389897+0000 mon.a (mon.0) 2471 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.619219+0000 osd.4 (osd.4) 211 : cluster 0 118.a scrub starts 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.623302+0000 osd.4 (osd.4) 212 : cluster 0 118.a scrub ok 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.014939+0000 osd.0 (osd.0) 235 : cluster 0 3.1b scrub starts 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.017443+0000 osd.0 (osd.0) 236 : cluster 0 3.1b scrub ok 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.354164+0000 mgr.y (mgr.14550) 413 : cluster 0 pgmap v585: 303 pgs: 3 creating+peering, 29 unknown, 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 510 B/s wr, 24 op/s 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.694865+0000 osd.3 (osd.3) 243 : cluster 0 5.14 scrub starts 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:19.697565+0000 osd.3 (osd.3) 244 : cluster 0 5.14 scrub ok 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.351161+0000 mon.a (mon.0) 2468 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:33:20.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: audit 2024-03-11T21:33:20.388358+0000 mon.a (mon.0) 2469 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:20.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: audit 2024-03-11T21:33:20.389444+0000 mon.a (mon.0) 2470 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:20.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: audit 2024-03-11T21:33:20.389897+0000 mon.a (mon.0) 2471 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:20.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.619219+0000 osd.4 (osd.4) 211 : cluster 0 118.a scrub starts 2024-03-11T21:33:20.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:20 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.623302+0000 osd.4 (osd.4) 212 : cluster 0 118.a scrub ok 2024-03-11T21:33:20.937 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:21.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:21.052 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.014939+0000 osd.0 (osd.0) 235 : cluster 0 3.1b scrub starts 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.017443+0000 osd.0 (osd.0) 236 : cluster 0 3.1b scrub ok 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.354164+0000 mgr.y (mgr.14550) 413 : cluster 0 pgmap v585: 303 pgs: 3 creating+peering, 29 unknown, 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 510 B/s wr, 24 op/s 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.694865+0000 osd.3 (osd.3) 243 : cluster 0 5.14 scrub starts 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:19.697565+0000 osd.3 (osd.3) 244 : cluster 0 5.14 scrub ok 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.351161+0000 mon.a (mon.0) 2468 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:33:21.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: audit 2024-03-11T21:33:20.388358+0000 mon.a (mon.0) 2469 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:21.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: audit 2024-03-11T21:33:20.389444+0000 mon.a (mon.0) 2470 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:21.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: audit 2024-03-11T21:33:20.389897+0000 mon.a (mon.0) 2471 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:21.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.619219+0000 osd.4 (osd.4) 211 : cluster 0 118.a scrub starts 2024-03-11T21:33:21.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:20 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.623302+0000 osd.4 (osd.4) 212 : cluster 0 118.a scrub ok 2024-03-11T21:33:21.066 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:21.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:21.197 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:21.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:21.327 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:21.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:21.458 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:21.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:21.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:21.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:21.719 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:21.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:21.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:21.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.537536+0000 osd.5 (osd.5) 249 : cluster 0 3.1c scrub starts 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.541986+0000 osd.5 (osd.5) 250 : cluster 0 3.1c scrub ok 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.692088+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.697233+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.217669+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.218825+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: audit 2024-03-11T21:33:21.253459+0000 mon.c (mon.2) 845 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.297552+0000 mon.a (mon.0) 2472 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: audit 2024-03-11T21:33:21.301440+0000 mon.a (mon.0) 2473 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.310550+0000 mon.a (mon.0) 2474 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:33:21.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:21 smithi203 bash[23654]: audit 2024-03-11T21:33:21.312834+0000 mon.a (mon.0) 2475 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:21.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.537536+0000 osd.5 (osd.5) 249 : cluster 0 3.1c scrub starts 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.541986+0000 osd.5 (osd.5) 250 : cluster 0 3.1c scrub ok 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.692088+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.697233+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.217669+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.218825+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: audit 2024-03-11T21:33:21.253459+0000 mon.c (mon.2) 845 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.297552+0000 mon.a (mon.0) 2472 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:22.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: audit 2024-03-11T21:33:21.301440+0000 mon.a (mon.0) 2473 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.310550+0000 mon.a (mon.0) 2474 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[21512]: audit 2024-03-11T21:33:21.312834+0000 mon.a (mon.0) 2475 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.537536+0000 osd.5 (osd.5) 249 : cluster 0 3.1c scrub starts 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.541986+0000 osd.5 (osd.5) 250 : cluster 0 3.1c scrub ok 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.692088+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.697233+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.217669+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.218825+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: audit 2024-03-11T21:33:21.253459+0000 mon.c (mon.2) 845 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.297552+0000 mon.a (mon.0) 2472 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:22.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: audit 2024-03-11T21:33:21.301440+0000 mon.a (mon.0) 2473 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi143-71604-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:22.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.310550+0000 mon.a (mon.0) 2474 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:33:22.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:21 smithi143 bash[29284]: audit 2024-03-11T21:33:21.312834+0000 mon.a (mon.0) 2475 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:22.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:22.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:22.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:22.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:22.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:22.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.367 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:22.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:22.494 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:22.597 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:22.621 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:22.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.751 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:22.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.951561+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:20.954016+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.114346+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.114943+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.354932+0000 mgr.y (mgr.14550) 414 : cluster 0 pgmap v588: 271 pgs: 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-03-11T21:33:22.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.964761+0000 osd.1 (osd.1) 49 : cluster 0 3.19 scrub starts 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:21.967752+0000 osd.1 (osd.1) 50 : cluster 0 3.19 scrub ok 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.751825+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.951561+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:20.954016+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.114346+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.114943+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.354932+0000 mgr.y (mgr.14550) 414 : cluster 0 pgmap v588: 271 pgs: 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.964761+0000 osd.1 (osd.1) 49 : cluster 0 3.19 scrub starts 2024-03-11T21:33:22.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:21.967752+0000 osd.1 (osd.1) 50 : cluster 0 3.19 scrub ok 2024-03-11T21:33:22.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:22 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.751825+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:33:22.988 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:23.014 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.951561+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-03-11T21:33:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:20.954016+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-03-11T21:33:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.114346+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:23.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.114943+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:23.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.354932+0000 mgr.y (mgr.14550) 414 : cluster 0 pgmap v588: 271 pgs: 271 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-03-11T21:33:23.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.964761+0000 osd.1 (osd.1) 49 : cluster 0 3.19 scrub starts 2024-03-11T21:33:23.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:21.967752+0000 osd.1 (osd.1) 50 : cluster 0 3.19 scrub ok 2024-03-11T21:33:23.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:22 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.751825+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:33:23.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:23.141 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:23.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.271 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:23.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/2966437684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c1020c0 msgr2=0x7f2b9c1024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- 172.21.15.143:0/2966437684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c1024a0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009920 tx=0x7f2b98009c30 comp rx=0 tx=0).stop 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/2966437684 shutdown_connections 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- 172.21.15.143:0/2966437684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c1024a0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- 172.21.15.143:0/2966437684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c107140 0x7f2b9c10ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- 172.21.15.143:0/2966437684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c102a70 0x7f2b9c106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/2966437684 >> 172.21.15.143:0/2966437684 conn(0x7f2b9c078f00 msgr2=0x7f2b9c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/2966437684 shutdown_connections 2024-03-11T21:33:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/2966437684 wait complete. 2024-03-11T21:33:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 librados: starting msgr at 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 librados: starting objecter 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 Processor -- start 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- start start 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 librados: setting wanted keys 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 librados: calling monclient init 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c19d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 0x7f2b9c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c1115e0 con 0x7f2b9c102a70 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c111460 con 0x7f2b9c1020c0 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c111760 con 0x7f2b9c107140 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba273b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 0x7f2b9c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35148/0 (socket says 172.21.15.143:35148) 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 -- 172.21.15.143:0/1505538315 learned_addr learned my addr 172.21.15.143:0/1505538315 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba273b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 0x7f2b9c197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44088/0 (socket says 172.21.15.143:44088) 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1f3a700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c19d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 -- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 msgr2=0x7f2b9c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 0x7f2b9c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 -- 172.21.15.143:0/1505538315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c1020c0 msgr2=0x7f2b9c19d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c19d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 -- 172.21.15.143:0/1505538315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c1981f0 con 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba1739700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00ec70 tx=0x7f2b8c00ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2b92ffd700 10 client.?.objecter ms_handle_connect 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2b92ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00cd40 con 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2b8c0044a0 con 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c1984e0 con 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c010430 con 0x7f2b9c102a70 2024-03-11T21:33:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 10 client.30327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:23.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c1a47a0 con 0x7f2b9c102a70 2024-03-11T21:33:23.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.481+0000 7f2ba419e700 1 librados: init done 2024-03-11T21:33:23.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c04faa0 con 0x7f2b9c102a70 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2b8c003f80 con 0x7f2b9c102a70 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 0x7f2b88071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(402..402 src has 87..402) v4 ==== 10673+0+0 (secure 0 0 0) 0x7f2b8c08f460 con 0x7f2b9c102a70 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 10 client.30327.objecter ms_dispatch 0x7f2b9c001550 osd_map(402..402 src has 87..402) v4 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 3 client.30327.objecter handle_osd_map got epochs [402,402] > 0 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 3 client.30327.objecter handle_osd_map decoding full epoch 402 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2ba1f3a700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 0x7f2b88071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 20 client.30327.objecter dump_active .. 0 homeless 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2b92ffd700 10 client.30327.objecter ms_handle_connect 0x7f2b8806f820 2024-03-11T21:33:23.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.485+0000 7f2ba1f3a700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 0x7f2b88071ce0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98005b40 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.489+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2b8c05bd10 con 0x7f2b9c102a70 2024-03-11T21:33:23.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:23.529 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:23.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/3320482675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18106dc0 msgr2=0x7f8d181071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- 172.21.15.143:0/3320482675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18106dc0 0x7f8d181071a0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009990 tx=0x7f8d14009ca0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/3320482675 shutdown_connections 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- 172.21.15.143:0/3320482675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- 172.21.15.143:0/3320482675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18106dc0 0x7f8d181071a0 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- 172.21.15.143:0/3320482675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d18100d40 0x7f8d181011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/3320482675 >> 172.21.15.143:0/3320482675 conn(0x7f8d18076d80 msgr2=0x7f8d18077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/3320482675 shutdown_connections 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/3320482675 wait complete. 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 librados: starting msgr at 2024-03-11T21:33:23.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 librados: starting objecter 2024-03-11T21:33:23.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 Processor -- start 2024-03-11T21:33:23.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- start start 2024-03-11T21:33:23.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 librados: setting wanted keys 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 librados: calling monclient init 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18100d40 0x7f8d1819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d18106dc0 0x7f8d181979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18110500 con 0x7f8d18101700 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18110380 con 0x7f8d18106dc0 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18110680 con 0x7f8d18100d40 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35156/0 (socket says 172.21.15.143:35156) 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 -- 172.21.15.143:0/423405495 learned_addr learned my addr 172.21.15.143:0/423405495 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18100d40 0x7f8d1819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d18106dc0 0x7f8d181979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18100d40 msgr2=0x7f8d1819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d18100d40 0x7f8d1819d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d18106dc0 msgr2=0x7f8d181979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d18106dc0 0x7f8d181979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18198140 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800d980 tx=0x7f8d0800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d0e7fc700 10 client.?.objecter ms_handle_connect 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d0e7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08017070 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d18198430 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8d08005020 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.645+0000 7f8d1f530700 10 client.30333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08009990 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d181028f0 con 0x7f8d18101700 2024-03-11T21:33:23.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 1 librados: init done 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8d18102400 con 0x7f8d18101700 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8d08004a30 con 0x7f8d18101700 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 0x7f8d04071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(402..402 src has 87..402) v4 ==== 10673+0+0 (secure 0 0 0) 0x7f8d0808e9a0 con 0x7f8d18101700 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 osd_map(402..402 src has 87..402) v4 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 3 client.30333.objecter handle_osd_map got epochs [402,402] > 0 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 3 client.30333.objecter handle_osd_map decoding full epoch 402 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 20 client.30333.objecter dump_active .. 0 homeless 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v402) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8d08020030 con 0x7f8d18101700 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 0x7f8d04071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 10 client.30333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 1 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8d180629f0 0x7f8d18062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 1 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 0 1 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 4 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 10 client.30333.objecter _send_command 1 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d18198f10 con 0x7f8d180629f0 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8d180629f0 0x7f8d18062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d0406f6f0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 0x7f8d04071bb0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400b7f0 tx=0x7f8d1401a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d180629f0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.649+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8d180629f0 0x7f8d18062dd0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.653+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d18198f10 con 0x7f8d180629f0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.653+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.653+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 1 = system:0 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.653+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 4 2024-03-11T21:33:23.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.653+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 0 1 2024-03-11T21:33:23.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 10 client.30333.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 2 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 2 2024-03-11T21:33:23.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1804faa0 osd=0 3 2024-03-11T21:33:23.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 0 2 2024-03-11T21:33:23.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 4 2024-03-11T21:33:23.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 10 client.30333.objecter _send_command 2 2024-03-11T21:33:23.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.669+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d18062470 con 0x7f8d180629f0 2024-03-11T21:33:23.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d18062470 con 0x7f8d180629f0 2024-03-11T21:33:23.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 4 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 0 2 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 10 client.30333.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8d1810a7a0 0x7f8d18102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 1 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 4 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 10 client.30333.objecter _send_command 3 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d18104800 con 0x7f8d1810a7a0 2024-03-11T21:33:23.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8d1810a7a0 0x7f8d18102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d1810a7a0 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8d1810a7a0 0x7f8d18102b20 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d18104800 con 0x7f8d1810a7a0 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 3 = system:0 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 4 2024-03-11T21:33:23.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.673+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 1 3 2024-03-11T21:33:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 10 client.30333.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 4 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 4 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1802d830 osd=1 3 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 1 4 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 4 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 10 client.30333.objecter _send_command 4 2024-03-11T21:33:23.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d1802dbf0 con 0x7f8d1810a7a0 2024-03-11T21:33:23.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d1802dbf0 con 0x7f8d1810a7a0 2024-03-11T21:33:23.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 4 2024-03-11T21:33:23.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.693+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 1 4 2024-03-11T21:33:23.672 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 10 client.30333.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 5 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8d18109e80 0x7f8d1802ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 5 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 2 5 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 4 2024-03-11T21:33:23.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 10 client.30333.objecter _send_command 5 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d1802d3c0 con 0x7f8d18109e80 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8d18109e80 0x7f8d1802ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8d18109e80 0x7f8d1802ce80 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d18109e80 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d1802d3c0 con 0x7f8d18109e80 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 5 = system:0 2024-03-11T21:33:23.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 4 2024-03-11T21:33:23.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.697+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 2 5 2024-03-11T21:33:23.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 10 client.30333.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 6 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 6 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d18109d10 osd=2 3 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 2 6 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 4 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 10 client.30333.objecter _send_command 6 2024-03-11T21:33:23.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.713+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d181aeb30 con 0x7f8d18109e80 2024-03-11T21:33:23.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d181aeb30 con 0x7f8d18109e80 2024-03-11T21:33:23.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 4 2024-03-11T21:33:23.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 2 6 2024-03-11T21:33:23.694 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:33:23.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 10 client.30333.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 7 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8d1802ca70 0x7f8d181b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 7 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 3 7 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 4 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 10 client.30333.objecter _send_command 7 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d18103d40 con 0x7f8d1802ca70 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8d1802ca70 0x7f8d181b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d1802ca70 2024-03-11T21:33:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8d1802ca70 0x7f8d181b1e70 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d18103d40 con 0x7f8d1802ca70 2024-03-11T21:33:23.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:33:23.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 7 = system:0 2024-03-11T21:33:23.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 4 2024-03-11T21:33:23.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.717+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 3 7 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 10 client.30333.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 8 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 8 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d1802c900 osd=3 3 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 3 8 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 4 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 10 client.30333.objecter _send_command 8 2024-03-11T21:33:23.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d18103d40 con 0x7f8d1802ca70 2024-03-11T21:33:23.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d18103d40 con 0x7f8d1802ca70 2024-03-11T21:33:23.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 4 2024-03-11T21:33:23.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 3 8 2024-03-11T21:33:23.715 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:33:23.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 10 client.30333.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 9 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8d181b40f0 0x7f8d181b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 9 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 4 9 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 4 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 10 client.30333.objecter _send_command 9 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d181b7f40 con 0x7f8d181b40f0 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.737+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8d181b40f0 0x7f8d181b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8d181b40f0 0x7f8d181b4550 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d181b40f0 2024-03-11T21:33:23.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d181b7f40 con 0x7f8d181b40f0 2024-03-11T21:33:23.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:33:23.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 9 = system:0 2024-03-11T21:33:23.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 4 2024-03-11T21:33:23.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.741+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 4 9 2024-03-11T21:33:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 10 client.30333.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 10 2024-03-11T21:33:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 10 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d18103d40 osd=4 3 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 4 10 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 4 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 10 client.30333.objecter _send_command 10 2024-03-11T21:33:23.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.757+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d181b2410 con 0x7f8d181b40f0 2024-03-11T21:33:23.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d181b2410 con 0x7f8d181b40f0 2024-03-11T21:33:23.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 4 2024-03-11T21:33:23.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 4 10 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 10 client.30333.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 11 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8d181b2b30 0x7f8d181b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 11 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 5 11 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 4 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 10 client.30333.objecter _send_command 11 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.761+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d181b3650 con 0x7f8d181b2b30 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8d181b2b30 0x7f8d181b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d181b2b30 2024-03-11T21:33:23.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d1cacb700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8d181b2b30 0x7f8d181b2f70 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d181b3650 con 0x7f8d181b2b30 2024-03-11T21:33:23.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:33:23.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 11 = system:0 2024-03-11T21:33:23.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 4 2024-03-11T21:33:23.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.765+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 5 11 2024-03-11T21:33:23.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.781+0000 7f8d1f530700 10 client.30333.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 12 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 12 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181b2410 osd=5 3 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 5 12 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 4 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 10 client.30333.objecter _send_command 12 2024-03-11T21:33:23.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.785+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d181b3860 con 0x7f8d181b2b30 2024-03-11T21:33:23.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d181b3860 con 0x7f8d181b2b30 2024-03-11T21:33:23.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 4 2024-03-11T21:33:23.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 5 12 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 10 client.30333.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 13 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8d181b92e0 0x7f8d181bb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 13 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 6 13 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 4 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 10 client.30333.objecter _send_command 13 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d181be530 con 0x7f8d181b92e0 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8d181b92e0 0x7f8d181bb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d181b92e0 2024-03-11T21:33:23.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d1dacd700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8d181b92e0 0x7f8d181bb6d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d181be530 con 0x7f8d181b92e0 2024-03-11T21:33:23.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:33:23.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 13 = system:0 2024-03-11T21:33:23.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 4 2024-03-11T21:33:23.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.789+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 6 13 2024-03-11T21:33:23.773 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:23.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.801+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2b9c198fc0 con 0x7f2b9c102a70 2024-03-11T21:33:23.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.801+0000 7f2b92ffd700 1 -- 172.21.15.143:0/1505538315 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2476 (secure 0 0 0) 0x7f2b9c198fc0 con 0x7f2b9c102a70 2024-03-11T21:33:23.779 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:33:23.779 INFO:teuthology.orchestra.run.smithi143.stdout:{"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1091,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":402,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192696,"num_in_osds":8,"osd_in_since":1710192115,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":271}],"num_pgs":271,"num_pools":11,"num_objects":340,"data_bytes":8793970,"bytes_used":633208832,"bytes_avail":767327076352,"bytes_total":767960285184,"read_bytes_sec":1277,"write_bytes_sec":510,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.143:8443/","prometheus":"http://172.21.15.143:9283/"}},"servicemap":{"epoch":24,"modified":"2024-03-11T21:32:37.329241+0000","services":{"rgw":{"daemons":{"summary":"","14427":{"start_epoch":13,"start_stamp":"2024-03-11T21:23:10.349156+0000","gid":14427,"addr":"172.21.15.143:0/3842484601","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi143","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi143","id":"foo.a","kernel_description":"#110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024","kernel_version":"5.15.0-100-generic","mem_swap_kb":"2097148","mem_total_kb":"32746924","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"25063d57-28bf-41f2-b5d5-290103490a0a","zone_name":"default","zonegroup_id":"4aab72bc-9c1b-4496-a7a5-2f18f6205dcf","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:33:23.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 10 librados: watch_flush enter 2024-03-11T21:33:23.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 10 librados: watch_flush exit 2024-03-11T21:33:23.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 20 client.30327.objecter shutdown clearing up homeless session... 2024-03-11T21:33:23.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 10 client.30327.objecter successfully canceled tick 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 msgr2=0x7f2b88071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 0x7f2b88071ce0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98005b40 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 msgr2=0x7f2b9c19de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00ec70 tx=0x7f2b8c00ef80 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 shutdown_connections 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b8806f820 0x7f2b88071ce0 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b9c107140 0x7f2b9c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b9c1020c0 0x7f2b9c19d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 --2- 172.21.15.143:0/1505538315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b9c102a70 0x7f2b9c19de10 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 >> 172.21.15.143:0/1505538315 conn(0x7f2b9c078f00 msgr2=0x7f2b9c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:23.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 shutdown_connections 2024-03-11T21:33:23.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 -- 172.21.15.143:0/1505538315 wait complete. 2024-03-11T21:33:23.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.805+0000 7f2ba419e700 1 librados: shutdown 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/1731484907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4109340 msgr2=0x7f2aa4110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/1731484907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4109340 0x7f2aa4110f10 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b0d0 tx=0x7f2a9c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/1731484907 shutdown_connections 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/1731484907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4109340 0x7f2aa4110f10 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/1731484907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 0x7f2aa4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/1731484907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4104c10 0x7f2aa4108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/1731484907 >> 172.21.15.143:0/1731484907 conn(0x7f2aa40ffaf0 msgr2=0x7f2aa4101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:23.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/1731484907 shutdown_connections 2024-03-11T21:33:23.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/1731484907 wait complete. 2024-03-11T21:33:23.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 librados: starting msgr at 2024-03-11T21:33:23.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 librados: starting objecter 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 Processor -- start 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- start start 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 librados: setting wanted keys 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 librados: calling monclient init 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 0x7f2aa4199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 0x7f2aa419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4113840 con 0x7f2aa4104c10 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41136c0 con 0x7f2aa4104260 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41139c0 con 0x7f2aa4109340 2024-03-11T21:33:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 0x7f2aa419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44124/0 (socket says 172.21.15.143:44124) 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa2d9d700 1 -- 172.21.15.143:0/3260656505 learned_addr learned my addr 172.21.15.143:0/3260656505 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa259c700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 0x7f2aa4199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 -- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 msgr2=0x7f2aa419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 -- 172.21.15.143:0/3260656505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 msgr2=0x7f2aa4199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 0x7f2aa4199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 -- 172.21.15.143:0/3260656505 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa419ed80 con 0x7f2aa4104c10 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa2d9d700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:33:23.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa1d9b700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 0x7f2aa419a2b0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800e9c0 tx=0x7f2a9800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2a937fe700 10 client.?.objecter ms_handle_connect 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2a937fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800cb70 con 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2a980044a0 con 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa419f070 con 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98003c80 con 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 10 client.30339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa41a68f0 con 0x7f2aa4104c10 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.809+0000 7f2aa8baf700 1 librados: init done 2024-03-11T21:33:23.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa404faa0 con 0x7f2aa4104c10 2024-03-11T21:33:23.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2a98003e40 con 0x7f2aa4104c10 2024-03-11T21:33:23.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 0x7f2a8c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(402..402 src has 87..402) v4 ==== 10673+0+0 (secure 0 0 0) 0x7f2a9808f230 con 0x7f2aa4104c10 2024-03-11T21:33:23.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 10 client.30339.objecter ms_dispatch 0x7f2aa4001550 osd_map(402..402 src has 87..402) v4 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 3 client.30339.objecter handle_osd_map got epochs [402,402] > 0 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 3 client.30339.objecter handle_osd_map decoding full epoch 402 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2aa259c700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 0x7f2a8c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 20 client.30339.objecter dump_active .. 0 homeless 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 10 client.30333.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 14 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 14 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181b3650 osd=6 3 2024-03-11T21:33:23.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 6 14 2024-03-11T21:33:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 4 2024-03-11T21:33:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 10 client.30333.objecter _send_command 14 2024-03-11T21:33:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d181bbc10 con 0x7f8d181b92e0 2024-03-11T21:33:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2aa259c700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 0x7f2a8c071ce0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f2a94005fd0 tx=0x7f2a9400a540 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.813+0000 7f2a937fe700 10 client.30339.objecter ms_handle_connect 0x7f2a8c06f820 2024-03-11T21:33:23.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2a9805bae0 con 0x7f2aa4104c10 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d181bbc10 con 0x7f8d181b92e0 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 4 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 6 14 2024-03-11T21:33:23.795 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 10 client.30333.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 15 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8d181bc290 0x7f8d181be680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 15 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 7 15 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 4 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 10 client.30333.objecter _send_command 15 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d181c16d0 con 0x7f8d181bc290 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.817+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8d181bc290 0x7f8d181be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:23.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 10 client.30333.objecter ms_handle_connect 0x7f8d181bc290 2024-03-11T21:33:23.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d1d2cc700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8d181bc290 0x7f8d181be680 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:23.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d181c16d0 con 0x7f8d181bc290 2024-03-11T21:33:23.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:33:23.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 15 = system:0 2024-03-11T21:33:23.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 4 2024-03-11T21:33:23.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.821+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 7 15 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 10 client.30333.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign -1 16 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 20 client.30333.objecter _get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_remove -1 16 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 20 client.30333.objecter get_session s=0x7f8d181bbc10 osd=7 3 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 15 client.30333.objecter _session_command_op_assign 7 16 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 4 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 10 client.30333.objecter _send_command 16 2024-03-11T21:33:23.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.841+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8d181c04a0 con 0x7f8d181bc290 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d0e7fc700 1 -- 172.21.15.143:0/423405495 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8d181c04a0 con 0x7f8d181bc290 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d0e7fc700 10 client.30333.objecter ms_dispatch 0x7f8d18001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d0e7fc700 10 client.30333.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d0e7fc700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 4 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d0e7fc700 15 client.30333.objecter _session_command_op_remove 7 16 2024-03-11T21:33:23.826 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 librados: watch_flush enter 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 librados: watch_flush exit 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.0 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8d180629f0 msgr2=0x7f8d18062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8d180629f0 0x7f8d18062dd0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1804faa0 osd=0 1 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.1 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8d1810a7a0 msgr2=0x7f8d18102b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8d1810a7a0 0x7f8d18102b20 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802d830 osd=1 1 2024-03-11T21:33:23.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.2 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8d18109e80 msgr2=0x7f8d1802ce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8d18109e80 0x7f8d1802ce80 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18109d10 osd=2 1 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1dacd700 1 -- 172.21.15.143:0/423405495 reap_dead start 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.3 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8d1802ca70 msgr2=0x7f8d181b1e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8d1802ca70 0x7f8d181b1e70 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d1802c900 osd=3 1 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.4 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8d181b40f0 msgr2=0x7f8d181b4550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8d181b40f0 0x7f8d181b4550 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d18103d40 osd=4 1 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.5 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8d181b2b30 msgr2=0x7f8d181b2f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8d181b2b30 0x7f8d181b2f70 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b2410 osd=5 1 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.6 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8d181b92e0 msgr2=0x7f8d181bb6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8d181b92e0 0x7f8d181bb6d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181b3650 osd=6 1 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter close_session for osd.7 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8d181bc290 msgr2=0x7f8d181be680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8d181bc290 0x7f8d181be680 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1dacd700 1 -- 172.21.15.143:0/423405495 reap_dead start 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter put_session s=0x7f8d181bbc10 osd=7 1 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 20 client.30333.objecter shutdown clearing up homeless session... 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 10 client.30333.objecter successfully canceled tick 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 msgr2=0x7f8d04071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 0x7f8d04071bb0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400b7f0 tx=0x7f8d1401a040 comp rx=0 tx=0).stop 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 msgr2=0x7f8d1819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800d980 tx=0x7f8d0800dc90 comp rx=0 tx=0).stop 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 shutdown_connections 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d18101700 0x7f8d1819dd60 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 --2- 172.21.15.143:0/423405495 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8d0406f6f0 0x7f8d04071bb0 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 >> 172.21.15.143:0/423405495 conn(0x7f8d18076d80 msgr2=0x7f8d180ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:23.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 shutdown_connections 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 -- 172.21.15.143:0/423405495 wait complete. 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:23.849+0000 7f8d1f530700 1 librados: shutdown 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:23.833 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:23.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.895 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.264920+0000 osd.7 (osd.7) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:23.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.265185+0000 osd.7 (osd.7) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:23.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.546657+0000 osd.5 (osd.5) 251 : cluster 0 9.7 scrub starts 2024-03-11T21:33:23.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.548702+0000 osd.5 (osd.5) 252 : cluster 0 9.7 scrub ok 2024-03-11T21:33:23.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.996380+0000 osd.1 (osd.1) 51 : cluster 0 9.14 scrub starts 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:22.998554+0000 osd.1 (osd.1) 52 : cluster 0 9.14 scrub ok 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: audit 2024-03-11T21:33:23.306435+0000 mon.a (mon.0) 2477 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.319544+0000 mon.a (mon.0) 2478 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: audit 2024-03-11T21:33:23.654240+0000 mon.a (mon.0) 2479 : audit 0 from='client.? 172.21.15.143:0/423405495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[29284]: audit 2024-03-11T21:33:23.806760+0000 mon.a (mon.0) 2480 : audit 0 from='client.? 172.21.15.143:0/1505538315' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.264920+0000 osd.7 (osd.7) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.265185+0000 osd.7 (osd.7) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.546657+0000 osd.5 (osd.5) 251 : cluster 0 9.7 scrub starts 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.548702+0000 osd.5 (osd.5) 252 : cluster 0 9.7 scrub ok 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.996380+0000 osd.1 (osd.1) 51 : cluster 0 9.14 scrub starts 2024-03-11T21:33:23.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:22.998554+0000 osd.1 (osd.1) 52 : cluster 0 9.14 scrub ok 2024-03-11T21:33:23.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: audit 2024-03-11T21:33:23.306435+0000 mon.a (mon.0) 2477 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:23.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.319544+0000 mon.a (mon.0) 2478 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:33:23.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: audit 2024-03-11T21:33:23.654240+0000 mon.a (mon.0) 2479 : audit 0 from='client.? 172.21.15.143:0/423405495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:23.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:23 smithi143 bash[21512]: audit 2024-03-11T21:33:23.806760+0000 mon.a (mon.0) 2480 : audit 0 from='client.? 172.21.15.143:0/1505538315' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:33:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:24.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:24.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.041+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2aa41a6b20 con 0x7f2aa4104c10 2024-03-11T21:33:24.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:24.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.264920+0000 osd.7 (osd.7) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:24.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.265185+0000 osd.7 (osd.7) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:24.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.546657+0000 osd.5 (osd.5) 251 : cluster 0 9.7 scrub starts 2024-03-11T21:33:24.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.548702+0000 osd.5 (osd.5) 252 : cluster 0 9.7 scrub ok 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.996380+0000 osd.1 (osd.1) 51 : cluster 0 9.14 scrub starts 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:22.998554+0000 osd.1 (osd.1) 52 : cluster 0 9.14 scrub ok 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: audit 2024-03-11T21:33:23.306435+0000 mon.a (mon.0) 2477 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi143-71604-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.319544+0000 mon.a (mon.0) 2478 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: audit 2024-03-11T21:33:23.654240+0000 mon.a (mon.0) 2479 : audit 0 from='client.? 172.21.15.143:0/423405495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:24.091 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:23 smithi203 bash[23654]: audit 2024-03-11T21:33:23.806760+0000 mon.a (mon.0) 2480 : audit 0 from='client.? 172.21.15.143:0/1505538315' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:33:24.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:24.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:24.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:24.259 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:24.260 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:24.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.309+0000 7f2a937fe700 1 -- 172.21.15.143:0/3260656505 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v403) v1 ==== 101+0+0 (secure 0 0 0) 0x7f2a9805f530 con 0x7f2aa4104c10 2024-03-11T21:33:24.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.309+0000 7f2aa8baf700 10 librados: watch_flush enter 2024-03-11T21:33:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.309+0000 7f2aa8baf700 10 librados: watch_flush exit 2024-03-11T21:33:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.309+0000 7f2aa8baf700 20 client.30339.objecter shutdown clearing up homeless session... 2024-03-11T21:33:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.309+0000 7f2aa8baf700 10 client.30339.objecter successfully canceled tick 2024-03-11T21:33:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 msgr2=0x7f2a8c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 0x7f2a8c071ce0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f2a94005fd0 tx=0x7f2a9400a540 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 msgr2=0x7f2aa419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 0x7f2aa419a2b0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800e9c0 tx=0x7f2a9800ecd0 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 shutdown_connections 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2a8c06f820 0x7f2a8c071ce0 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2aa4109340 0x7f2aa419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa4104260 0x7f2aa4199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 --2- 172.21.15.143:0/3260656505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2aa4104c10 0x7f2aa419a2b0 secure :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800e9c0 tx=0x7f2a9800ecd0 comp rx=0 tx=0).stop 2024-03-11T21:33:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 >> 172.21.15.143:0/3260656505 conn(0x7f2aa40ffaf0 msgr2=0x7f2aa4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:24.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 shutdown_connections 2024-03-11T21:33:24.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 -- 172.21.15.143:0/3260656505 wait complete. 2024-03-11T21:33:24.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:24.313+0000 7f2aa8baf700 1 librados: shutdown 2024-03-11T21:33:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:24.297 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:33:24.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:24.424 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:24.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:24.554 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:24.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:24.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:24.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:24.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:24.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:24.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.355463+0000 mgr.y (mgr.14550) 415 : cluster 0 pgmap v591: 279 pgs: 8 unknown, 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:24.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.548523+0000 osd.5 (osd.5) 253 : cluster 0 6.19 scrub starts 2024-03-11T21:33:24.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.549763+0000 osd.5 (osd.5) 254 : cluster 0 6.19 scrub ok 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.651639+0000 osd.3 (osd.3) 247 : cluster 0 118.6 scrub starts 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.654346+0000 osd.3 (osd.3) 248 : cluster 0 118.6 scrub ok 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.991849+0000 osd.1 (osd.1) 53 : cluster 0 2.4 scrub starts 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:23.993237+0000 osd.1 (osd.1) 54 : cluster 0 2.4 scrub ok 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: audit 2024-03-11T21:33:24.047117+0000 mon.a (mon.0) 2481 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:24.307393+0000 mon.a (mon.0) 2482 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: audit 2024-03-11T21:33:24.312076+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:24.927 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:24 smithi203 bash[23654]: cluster 2024-03-11T21:33:24.316594+0000 mon.a (mon.0) 2484 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:33:24.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:25.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:25.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:25.101 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.355463+0000 mgr.y (mgr.14550) 415 : cluster 0 pgmap v591: 279 pgs: 8 unknown, 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.548523+0000 osd.5 (osd.5) 253 : cluster 0 6.19 scrub starts 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.549763+0000 osd.5 (osd.5) 254 : cluster 0 6.19 scrub ok 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.651639+0000 osd.3 (osd.3) 247 : cluster 0 118.6 scrub starts 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.654346+0000 osd.3 (osd.3) 248 : cluster 0 118.6 scrub ok 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.991849+0000 osd.1 (osd.1) 53 : cluster 0 2.4 scrub starts 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:23.993237+0000 osd.1 (osd.1) 54 : cluster 0 2.4 scrub ok 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: audit 2024-03-11T21:33:24.047117+0000 mon.a (mon.0) 2481 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:24.307393+0000 mon.a (mon.0) 2482 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: audit 2024-03-11T21:33:24.312076+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:25.102 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[21512]: cluster 2024-03-11T21:33:24.316594+0000 mon.a (mon.0) 2484 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.355463+0000 mgr.y (mgr.14550) 415 : cluster 0 pgmap v591: 279 pgs: 8 unknown, 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.548523+0000 osd.5 (osd.5) 253 : cluster 0 6.19 scrub starts 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.549763+0000 osd.5 (osd.5) 254 : cluster 0 6.19 scrub ok 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.651639+0000 osd.3 (osd.3) 247 : cluster 0 118.6 scrub starts 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.654346+0000 osd.3 (osd.3) 248 : cluster 0 118.6 scrub ok 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.991849+0000 osd.1 (osd.1) 53 : cluster 0 2.4 scrub starts 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:23.993237+0000 osd.1 (osd.1) 54 : cluster 0 2.4 scrub ok 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: audit 2024-03-11T21:33:24.047117+0000 mon.a (mon.0) 2481 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:24.307393+0000 mon.a (mon.0) 2482 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: audit 2024-03-11T21:33:24.312076+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.143:0/3260656505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:25.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:24 smithi143 bash[29284]: cluster 2024-03-11T21:33:24.316594+0000 mon.a (mon.0) 2484 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:33:25.151 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:33:25.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:25.220 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:25.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:25.352 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:25.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:25.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:25.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:25.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:25.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:25.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:25.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:25.885 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:25.988 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.017 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:26.036 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-11T21:33:26.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:33:26.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:26.151 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:26.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.276 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:26.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.275407+0000 osd.2 (osd.2) 193 : cluster 0 2.e scrub starts 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.276727+0000 osd.2 (osd.2) 194 : cluster 0 2.e scrub ok 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.323340+0000 mon.a (mon.0) 2485 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: audit 2024-03-11T21:33:25.328766+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.356351+0000 mgr.y (mgr.14550) 416 : cluster 0 pgmap v594: 271 pgs: 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:33:26.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.575065+0000 osd.6 (osd.6) 183 : cluster 0 39.1e scrub starts 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.576249+0000 osd.6 (osd.6) 184 : cluster 0 39.1e scrub ok 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.588147+0000 osd.7 (osd.7) 265 : cluster 0 39.1 scrub starts 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.591943+0000 osd.7 (osd.7) 266 : cluster 0 39.1 scrub ok 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.681008+0000 osd.4 (osd.4) 213 : cluster 0 118.14 scrub starts 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.682764+0000 osd.4 (osd.4) 214 : cluster 0 118.14 scrub ok 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.689322+0000 osd.3 (osd.3) 249 : cluster 0 118.7 scrub starts 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.692502+0000 osd.3 (osd.3) 250 : cluster 0 118.7 scrub ok 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.720139+0000 osd.0 (osd.0) 239 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.720488+0000 osd.0 (osd.0) 240 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:26.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:26.032002+0000 osd.1 (osd.1) 55 : cluster 0 9.1f scrub starts 2024-03-11T21:33:26.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:26.034366+0000 osd.1 (osd.1) 56 : cluster 0 9.1f scrub ok 2024-03-11T21:33:26.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:26 smithi203 bash[23654]: cluster 2024-03-11T21:33:26.300636+0000 mon.a (mon.0) 2487 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:26.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:26.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.520 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.275407+0000 osd.2 (osd.2) 193 : cluster 0 2.e scrub starts 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.276727+0000 osd.2 (osd.2) 194 : cluster 0 2.e scrub ok 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.323340+0000 mon.a (mon.0) 2485 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: audit 2024-03-11T21:33:25.328766+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.356351+0000 mgr.y (mgr.14550) 416 : cluster 0 pgmap v594: 271 pgs: 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.575065+0000 osd.6 (osd.6) 183 : cluster 0 39.1e scrub starts 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.576249+0000 osd.6 (osd.6) 184 : cluster 0 39.1e scrub ok 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.588147+0000 osd.7 (osd.7) 265 : cluster 0 39.1 scrub starts 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.591943+0000 osd.7 (osd.7) 266 : cluster 0 39.1 scrub ok 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.681008+0000 osd.4 (osd.4) 213 : cluster 0 118.14 scrub starts 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.682764+0000 osd.4 (osd.4) 214 : cluster 0 118.14 scrub ok 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.689322+0000 osd.3 (osd.3) 249 : cluster 0 118.7 scrub starts 2024-03-11T21:33:26.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.692502+0000 osd.3 (osd.3) 250 : cluster 0 118.7 scrub ok 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.720139+0000 osd.0 (osd.0) 239 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.720488+0000 osd.0 (osd.0) 240 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:26.032002+0000 osd.1 (osd.1) 55 : cluster 0 9.1f scrub starts 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:26.034366+0000 osd.1 (osd.1) 56 : cluster 0 9.1f scrub ok 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[29284]: cluster 2024-03-11T21:33:26.300636+0000 mon.a (mon.0) 2487 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.275407+0000 osd.2 (osd.2) 193 : cluster 0 2.e scrub starts 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.276727+0000 osd.2 (osd.2) 194 : cluster 0 2.e scrub ok 2024-03-11T21:33:26.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.323340+0000 mon.a (mon.0) 2485 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: audit 2024-03-11T21:33:25.328766+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.356351+0000 mgr.y (mgr.14550) 416 : cluster 0 pgmap v594: 271 pgs: 271 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.575065+0000 osd.6 (osd.6) 183 : cluster 0 39.1e scrub starts 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.576249+0000 osd.6 (osd.6) 184 : cluster 0 39.1e scrub ok 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.588147+0000 osd.7 (osd.7) 265 : cluster 0 39.1 scrub starts 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.591943+0000 osd.7 (osd.7) 266 : cluster 0 39.1 scrub ok 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.681008+0000 osd.4 (osd.4) 213 : cluster 0 118.14 scrub starts 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.682764+0000 osd.4 (osd.4) 214 : cluster 0 118.14 scrub ok 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.689322+0000 osd.3 (osd.3) 249 : cluster 0 118.7 scrub starts 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.692502+0000 osd.3 (osd.3) 250 : cluster 0 118.7 scrub ok 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.720139+0000 osd.0 (osd.0) 239 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:26.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.720488+0000 osd.0 (osd.0) 240 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:26.524 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:26.032002+0000 osd.1 (osd.1) 55 : cluster 0 9.1f scrub starts 2024-03-11T21:33:26.524 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:26.034366+0000 osd.1 (osd.1) 56 : cluster 0 9.1f scrub ok 2024-03-11T21:33:26.524 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:26 smithi143 bash[21512]: cluster 2024-03-11T21:33:26.300636+0000 mon.a (mon.0) 2487 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:26.535 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:26.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.667 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:26.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:26.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:26.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:26.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:27.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:27.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:27.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:27.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:27.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:27.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.550638+0000 osd.5 (osd.5) 255 : cluster 0 39.1a scrub starts 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: cluster 2024-03-11T21:33:25.552959+0000 osd.5 (osd.5) 256 : cluster 0 39.1a scrub ok 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.316013+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: cluster 2024-03-11T21:33:26.320210+0000 mon.a (mon.0) 2489 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.323984+0000 mon.a (mon.0) 2490 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.345631+0000 mon.c (mon.2) 846 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.345978+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.347022+0000 mon.c (mon.2) 847 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.347329+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.347788+0000 mon.c (mon.2) 848 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:26.348121+0000 mon.a (mon.0) 2493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[21512]: audit 2024-03-11T21:33:27.319863+0000 mon.a (mon.0) 2494 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.550638+0000 osd.5 (osd.5) 255 : cluster 0 39.1a scrub starts 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: cluster 2024-03-11T21:33:25.552959+0000 osd.5 (osd.5) 256 : cluster 0 39.1a scrub ok 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.316013+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: cluster 2024-03-11T21:33:26.320210+0000 mon.a (mon.0) 2489 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.323984+0000 mon.a (mon.0) 2490 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.345631+0000 mon.c (mon.2) 846 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.345978+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.347022+0000 mon.c (mon.2) 847 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.347329+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.347788+0000 mon.c (mon.2) 848 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:26.348121+0000 mon.a (mon.0) 2493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:27 smithi143 bash[29284]: audit 2024-03-11T21:33:27.319863+0000 mon.a (mon.0) 2494 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.455 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:27.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:27.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.550638+0000 osd.5 (osd.5) 255 : cluster 0 39.1a scrub starts 2024-03-11T21:33:27.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: cluster 2024-03-11T21:33:25.552959+0000 osd.5 (osd.5) 256 : cluster 0 39.1a scrub ok 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.316013+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: cluster 2024-03-11T21:33:26.320210+0000 mon.a (mon.0) 2489 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.323984+0000 mon.a (mon.0) 2490 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.345631+0000 mon.c (mon.2) 846 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.345978+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.347022+0000 mon.c (mon.2) 847 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.347329+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.347788+0000 mon.c (mon.2) 848 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:26.348121+0000 mon.a (mon.0) 2493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:27.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:27 smithi203 bash[23654]: audit 2024-03-11T21:33:27.319863+0000 mon.a (mon.0) 2494 : audit 1 from='client.? 172.21.15.143:0/3442035717' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi143-71604-61"}]': finished 2024-03-11T21:33:27.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:27.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:27.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:27.847 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:27.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:27.978 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:28.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:33:28.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:33:28.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:28.117 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:28.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:28.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:28.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:28.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:28.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:28.502 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:28.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:28.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:28.633 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:28.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:28.741 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.226004+0000 osd.2 (osd.2) 195 : cluster 0 39.16 scrub starts 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.227841+0000 osd.2 (osd.2) 196 : cluster 0 39.16 scrub ok 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.320090+0000 mon.a (mon.0) 2495 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.327762+0000 mon.a (mon.0) 2496 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.329713+0000 mon.c (mon.2) 849 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.333713+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.356917+0000 mgr.y (mgr.14550) 417 : cluster 0 pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.371211+0000 mon.c (mon.2) 850 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.371606+0000 mon.a (mon.0) 2498 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.372450+0000 mon.c (mon.2) 851 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.226004+0000 osd.2 (osd.2) 195 : cluster 0 39.16 scrub starts 2024-03-11T21:33:28.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.227841+0000 osd.2 (osd.2) 196 : cluster 0 39.16 scrub ok 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.320090+0000 mon.a (mon.0) 2495 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.327762+0000 mon.a (mon.0) 2496 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.329713+0000 mon.c (mon.2) 849 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.333713+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.356917+0000 mgr.y (mgr.14550) 417 : cluster 0 pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.371211+0000 mon.c (mon.2) 850 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:28.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.371606+0000 mon.a (mon.0) 2498 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:28.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:28.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:28.890 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:28.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.226004+0000 osd.2 (osd.2) 195 : cluster 0 39.16 scrub starts 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.227841+0000 osd.2 (osd.2) 196 : cluster 0 39.16 scrub ok 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.320090+0000 mon.a (mon.0) 2495 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi143-71636-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.327762+0000 mon.a (mon.0) 2496 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.329713+0000 mon.c (mon.2) 849 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:29.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.333713+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.356917+0000 mgr.y (mgr.14550) 417 : cluster 0 pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.371211+0000 mon.c (mon.2) 850 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.371606+0000 mon.a (mon.0) 2498 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.372450+0000 mon.c (mon.2) 851 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.372757+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.373309+0000 mon.c (mon.2) 852 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:27.373603+0000 mon.a (mon.0) 2500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.683724+0000 osd.4 (osd.4) 215 : cluster 0 3.1a scrub starts 2024-03-11T21:33:29.005 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:28 smithi203 bash[23654]: audit 2024-03-11T21:33:28.321971+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:29.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:29.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.372450+0000 mon.c (mon.2) 851 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.372757+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.373309+0000 mon.c (mon.2) 852 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:27.373603+0000 mon.a (mon.0) 2500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.683724+0000 osd.4 (osd.4) 215 : cluster 0 3.1a scrub starts 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[21512]: audit 2024-03-11T21:33:28.321971+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.372757+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.373309+0000 mon.c (mon.2) 852 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.135 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:27.373603+0000 mon.a (mon.0) 2500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:29.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.683724+0000 osd.4 (osd.4) 215 : cluster 0 3.1a scrub starts 2024-03-11T21:33:29.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:28 smithi143 bash[29284]: audit 2024-03-11T21:33:28.321971+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:29.148 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:29.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:29.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:29.407 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:29.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:29.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:29.661 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:29.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:29.794 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:29.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.581470+0000 osd.5 (osd.5) 257 : cluster 0 118.e scrub starts 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.586315+0000 osd.5 (osd.5) 258 : cluster 0 118.e scrub ok 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:27.686777+0000 osd.4 (osd.4) 216 : cluster 0 3.1a scrub ok 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:28.218214+0000 osd.2 (osd.2) 197 : cluster 0 118.d scrub starts 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:28.221072+0000 osd.2 (osd.2) 198 : cluster 0 118.d scrub ok 2024-03-11T21:33:29.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: audit 2024-03-11T21:33:28.330402+0000 mon.c (mon.2) 853 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:28.569296+0000 osd.6 (osd.6) 185 : cluster 0 39.19 deep-scrub starts 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:28.577660+0000 osd.6 (osd.6) 186 : cluster 0 39.19 deep-scrub ok 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: audit 2024-03-11T21:33:28.581431+0000 mgr.y (mgr.14550) 418 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: cluster 2024-03-11T21:33:28.758347+0000 mon.a (mon.0) 2502 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: audit 2024-03-11T21:33:28.760786+0000 mon.a (mon.0) 2503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[29284]: audit 2024-03-11T21:33:29.324837+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.581470+0000 osd.5 (osd.5) 257 : cluster 0 118.e scrub starts 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.586315+0000 osd.5 (osd.5) 258 : cluster 0 118.e scrub ok 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:27.686777+0000 osd.4 (osd.4) 216 : cluster 0 3.1a scrub ok 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:28.218214+0000 osd.2 (osd.2) 197 : cluster 0 118.d scrub starts 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:28.221072+0000 osd.2 (osd.2) 198 : cluster 0 118.d scrub ok 2024-03-11T21:33:29.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: audit 2024-03-11T21:33:28.330402+0000 mon.c (mon.2) 853 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:28.569296+0000 osd.6 (osd.6) 185 : cluster 0 39.19 deep-scrub starts 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:28.577660+0000 osd.6 (osd.6) 186 : cluster 0 39.19 deep-scrub ok 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: audit 2024-03-11T21:33:28.581431+0000 mgr.y (mgr.14550) 418 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: cluster 2024-03-11T21:33:28.758347+0000 mon.a (mon.0) 2502 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: audit 2024-03-11T21:33:28.760786+0000 mon.a (mon.0) 2503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:29.910 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:29 smithi143 bash[21512]: audit 2024-03-11T21:33:29.324837+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:29.924 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:29.925 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.926 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:30.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.581470+0000 osd.5 (osd.5) 257 : cluster 0 118.e scrub starts 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.586315+0000 osd.5 (osd.5) 258 : cluster 0 118.e scrub ok 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:27.686777+0000 osd.4 (osd.4) 216 : cluster 0 3.1a scrub ok 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:28.218214+0000 osd.2 (osd.2) 197 : cluster 0 118.d scrub starts 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:28.221072+0000 osd.2 (osd.2) 198 : cluster 0 118.d scrub ok 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: audit 2024-03-11T21:33:28.330402+0000 mon.c (mon.2) 853 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:28.569296+0000 osd.6 (osd.6) 185 : cluster 0 39.19 deep-scrub starts 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:28.577660+0000 osd.6 (osd.6) 186 : cluster 0 39.19 deep-scrub ok 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: audit 2024-03-11T21:33:28.581431+0000 mgr.y (mgr.14550) 418 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: cluster 2024-03-11T21:33:28.758347+0000 mon.a (mon.0) 2502 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: audit 2024-03-11T21:33:28.760786+0000 mon.a (mon.0) 2503 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:30.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:29 smithi203 bash[23654]: audit 2024-03-11T21:33:29.324837+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi143-71636-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:30.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:30.058 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:30.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.188 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:30.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.323 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:30.404 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullP expected=8 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:b29083e3:::5:head expected=12:b29083e3:::5:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=5 expected=5 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:89d3ae78:::11:head expected=12:89d3ae78:::11:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=11 expected=11 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:566253c9:::13:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:566253c9:::13:head expected=12:566253c9:::13:head 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-03-11T21:33:30.405 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=13 expected=13 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:863748b0:::15:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:863748b0:::15:head expected=12:863748b0:::15:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=15 expected=15 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:6cac518f:::0:head expected=12:6cac518f:::0:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=0 expected=0 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:62a1935d:::14:head expected=12:62a1935d:::14:head 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-03-11T21:33:30.406 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=14 expected=14 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:5c6b0b28:::7:head expected=12:5c6b0b28:::7:head 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=7 expected=7 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=12:02547ec2:::1:head expected=12:02547ec2:::1:head 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=1 expected=1 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (283 ms) 2024-03-11T21:33:30.407 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (128778 ms) 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 0/5 -> MIN 12:33333333::::head 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 1/5 -> 12:33333333::::head 12:66666666::::head 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 2/5 -> 12:66666666::::head 12:99999999::::head 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 3/5 -> 12:99999999::::head 12:cccccccc::::head 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 4/5 -> 12:cccccccc::::head MAX 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (145035 ms) 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 7 tests from LibRadosList (274762 ms total) 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1051 ms) 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-11T21:33:30.408 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo2,foo3 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo2 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo3 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo4,foo5 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo4 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo5 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo6,foo7 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo7 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo6 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo4 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo5 2024-03-11T21:33:30.409 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo7 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo6 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo1 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo1 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo2 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo3 2024-03-11T21:33:30.410 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (49 ms) 2024-03-11T21:33:30.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:30.455 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:30.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:30.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:30.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:30.715 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:30.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.055848+0000 osd.0 (osd.0) 241 : cluster 0 9.1e scrub starts 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.057955+0000 osd.0 (osd.0) 242 : cluster 0 9.1e scrub ok 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.217081+0000 osd.2 (osd.2) 199 : cluster 0 118.15 scrub starts 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.221518+0000 osd.2 (osd.2) 200 : cluster 0 118.15 scrub ok 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.357443+0000 mgr.y (mgr.14550) 419 : cluster 0 pgmap v600: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:29.566381+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:30.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.668122+0000 osd.7 (osd.7) 267 : cluster 0 3.11 scrub starts 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.672093+0000 osd.7 (osd.7) 268 : cluster 0 3.11 scrub ok 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:29.763371+0000 mon.a (mon.0) 2505 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:29.773683+0000 mon.a (mon.0) 2506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:29.775728+0000 mon.c (mon.2) 855 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:29.776174+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:30.336707+0000 mon.a (mon.0) 2508 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: audit 2024-03-11T21:33:30.336940+0000 mon.a (mon.0) 2509 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]': finished 2024-03-11T21:33:30.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:30 smithi203 bash[23654]: cluster 2024-03-11T21:33:30.344325+0000 mon.a (mon.0) 2510 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:33:30.846 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:30.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:31.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.055848+0000 osd.0 (osd.0) 241 : cluster 0 9.1e scrub starts 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.057955+0000 osd.0 (osd.0) 242 : cluster 0 9.1e scrub ok 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.217081+0000 osd.2 (osd.2) 199 : cluster 0 118.15 scrub starts 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.221518+0000 osd.2 (osd.2) 200 : cluster 0 118.15 scrub ok 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.357443+0000 mgr.y (mgr.14550) 419 : cluster 0 pgmap v600: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:29.566381+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:31.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.668122+0000 osd.7 (osd.7) 267 : cluster 0 3.11 scrub starts 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.672093+0000 osd.7 (osd.7) 268 : cluster 0 3.11 scrub ok 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:29.763371+0000 mon.a (mon.0) 2505 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:29.773683+0000 mon.a (mon.0) 2506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:29.775728+0000 mon.c (mon.2) 855 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:29.776174+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:30.336707+0000 mon.a (mon.0) 2508 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: audit 2024-03-11T21:33:30.336940+0000 mon.a (mon.0) 2509 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]': finished 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[21512]: cluster 2024-03-11T21:33:30.344325+0000 mon.a (mon.0) 2510 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.055848+0000 osd.0 (osd.0) 241 : cluster 0 9.1e scrub starts 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.057955+0000 osd.0 (osd.0) 242 : cluster 0 9.1e scrub ok 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.217081+0000 osd.2 (osd.2) 199 : cluster 0 118.15 scrub starts 2024-03-11T21:33:31.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.221518+0000 osd.2 (osd.2) 200 : cluster 0 118.15 scrub ok 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.357443+0000 mgr.y (mgr.14550) 419 : cluster 0 pgmap v600: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:29.566381+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.668122+0000 osd.7 (osd.7) 267 : cluster 0 3.11 scrub starts 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.672093+0000 osd.7 (osd.7) 268 : cluster 0 3.11 scrub ok 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:29.763371+0000 mon.a (mon.0) 2505 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:29.773683+0000 mon.a (mon.0) 2506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:29.775728+0000 mon.c (mon.2) 855 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:29.776174+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]: dispatch 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:30.336707+0000 mon.a (mon.0) 2508 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi143-71604-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: audit 2024-03-11T21:33:30.336940+0000 mon.a (mon.0) 2509 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-45"}]': finished 2024-03-11T21:33:31.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:30 smithi143 bash[29284]: cluster 2024-03-11T21:33:30.344325+0000 mon.a (mon.0) 2510 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:33:31.106 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:31.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:31.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:31.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:31.361 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:31.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.486 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:31.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:31.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:31.736 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:31.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:31.866 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:31.955 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:31.956 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:31.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:31.978 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.044666+0000 osd.1 (osd.1) 57 : cluster 0 4.2 scrub starts 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.047433+0000 osd.1 (osd.1) 58 : cluster 0 4.2 scrub ok 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.315166+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[21512]: audit 2024-03-11T21:33:31.321184+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.044666+0000 osd.1 (osd.1) 57 : cluster 0 4.2 scrub starts 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.047433+0000 osd.1 (osd.1) 58 : cluster 0 4.2 scrub ok 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.315166+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:33:31.979 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:31 smithi143 bash[29284]: audit 2024-03-11T21:33:31.321184+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:31.990 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:32.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:32.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:31 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.044666+0000 osd.1 (osd.1) 57 : cluster 0 4.2 scrub starts 2024-03-11T21:33:32.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:31 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.047433+0000 osd.1 (osd.1) 58 : cluster 0 4.2 scrub ok 2024-03-11T21:33:32.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:31 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.315166+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:33:32.106 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:31 smithi203 bash[23654]: audit 2024-03-11T21:33:31.321184+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:32.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.251 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:32.291 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:32.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:32.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:32.502 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:32.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:32.634 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:32.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.766 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:32.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.358094+0000 mgr.y (mgr.14550) 420 : cluster 0 pgmap v603: 236 pgs: 1 creating+peering, 7 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.561952+0000 osd.5 (osd.5) 259 : cluster 0 118.10 scrub starts 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.565184+0000 osd.5 (osd.5) 260 : cluster 0 118.10 scrub ok 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.678252+0000 osd.7 (osd.7) 269 : cluster 0 3.e deep-scrub starts 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.683204+0000 osd.7 (osd.7) 270 : cluster 0 3.e deep-scrub ok 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:31.812032+0000 mon.a (mon.0) 2512 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:32.233426+0000 osd.2 (osd.2) 201 : cluster 0 118.5 scrub starts 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:32.235762+0000 osd.2 (osd.2) 202 : cluster 0 118.5 scrub ok 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.341840+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: cluster 2024-03-11T21:33:32.346073+0000 mon.a (mon.0) 2514 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.348758+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.881 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.349693+0000 mon.a (mon.0) 2515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.351053+0000 mon.c (mon.2) 858 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.351401+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.352682+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[21512]: audit 2024-03-11T21:33:32.352985+0000 mon.a (mon.0) 2517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.358094+0000 mgr.y (mgr.14550) 420 : cluster 0 pgmap v603: 236 pgs: 1 creating+peering, 7 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.561952+0000 osd.5 (osd.5) 259 : cluster 0 118.10 scrub starts 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.565184+0000 osd.5 (osd.5) 260 : cluster 0 118.10 scrub ok 2024-03-11T21:33:32.882 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.678252+0000 osd.7 (osd.7) 269 : cluster 0 3.e deep-scrub starts 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.683204+0000 osd.7 (osd.7) 270 : cluster 0 3.e deep-scrub ok 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:31.812032+0000 mon.a (mon.0) 2512 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:32.233426+0000 osd.2 (osd.2) 201 : cluster 0 118.5 scrub starts 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:32.235762+0000 osd.2 (osd.2) 202 : cluster 0 118.5 scrub ok 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.341840+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: cluster 2024-03-11T21:33:32.346073+0000 mon.a (mon.0) 2514 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.348758+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.349693+0000 mon.a (mon.0) 2515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.351053+0000 mon.c (mon.2) 858 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.351401+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.352682+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:32.883 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:32 smithi143 bash[29284]: audit 2024-03-11T21:33:32.352985+0000 mon.a (mon.0) 2517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:32.895 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:32.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.358094+0000 mgr.y (mgr.14550) 420 : cluster 0 pgmap v603: 236 pgs: 1 creating+peering, 7 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.561952+0000 osd.5 (osd.5) 259 : cluster 0 118.10 scrub starts 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.565184+0000 osd.5 (osd.5) 260 : cluster 0 118.10 scrub ok 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.678252+0000 osd.7 (osd.7) 269 : cluster 0 3.e deep-scrub starts 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.683204+0000 osd.7 (osd.7) 270 : cluster 0 3.e deep-scrub ok 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:31.812032+0000 mon.a (mon.0) 2512 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:32.233426+0000 osd.2 (osd.2) 201 : cluster 0 118.5 scrub starts 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:32.235762+0000 osd.2 (osd.2) 202 : cluster 0 118.5 scrub ok 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.341840+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: cluster 2024-03-11T21:33:32.346073+0000 mon.a (mon.0) 2514 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:33:33.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.348758+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.143:0/2289697665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:33.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.349693+0000 mon.a (mon.0) 2515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]: dispatch 2024-03-11T21:33:33.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.351053+0000 mon.c (mon.2) 858 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:33.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.351401+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:33.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.352682+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:33.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:32 smithi203 bash[23654]: audit 2024-03-11T21:33:32.352985+0000 mon.a (mon.0) 2517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:33.024 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:33.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:33.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:33.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:33.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:33.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:33.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:33.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:33.670 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:33.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:33.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:33.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:33.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.963039+0000 osd.0 (osd.0) 243 : cluster 0 9.5 scrub starts 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: cluster 2024-03-11T21:33:31.964597+0000 osd.0 (osd.0) 244 : cluster 0 9.5 scrub ok 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: cluster 2024-03-11T21:33:32.574094+0000 osd.3 (osd.3) 251 : cluster 0 5.17 scrub starts 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: cluster 2024-03-11T21:33:32.575647+0000 osd.3 (osd.3) 252 : cluster 0 5.17 scrub ok 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: audit 2024-03-11T21:33:33.346273+0000 mon.a (mon.0) 2518 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: audit 2024-03-11T21:33:33.346512+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: audit 2024-03-11T21:33:33.346735+0000 mon.a (mon.0) 2520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: cluster 2024-03-11T21:33:33.355445+0000 mon.a (mon.0) 2521 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:33:34.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: audit 2024-03-11T21:33:33.361247+0000 mon.c (mon.2) 860 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:33 smithi203 bash[23654]: audit 2024-03-11T21:33:33.382906+0000 mon.a (mon.0) 2522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:34.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.963039+0000 osd.0 (osd.0) 243 : cluster 0 9.5 scrub starts 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: cluster 2024-03-11T21:33:31.964597+0000 osd.0 (osd.0) 244 : cluster 0 9.5 scrub ok 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: cluster 2024-03-11T21:33:32.574094+0000 osd.3 (osd.3) 251 : cluster 0 5.17 scrub starts 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: cluster 2024-03-11T21:33:32.575647+0000 osd.3 (osd.3) 252 : cluster 0 5.17 scrub ok 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: audit 2024-03-11T21:33:33.346273+0000 mon.a (mon.0) 2518 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: audit 2024-03-11T21:33:33.346512+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: audit 2024-03-11T21:33:33.346735+0000 mon.a (mon.0) 2520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:34.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: cluster 2024-03-11T21:33:33.355445+0000 mon.a (mon.0) 2521 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: audit 2024-03-11T21:33:33.361247+0000 mon.c (mon.2) 860 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[21512]: audit 2024-03-11T21:33:33.382906+0000 mon.a (mon.0) 2522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.963039+0000 osd.0 (osd.0) 243 : cluster 0 9.5 scrub starts 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: cluster 2024-03-11T21:33:31.964597+0000 osd.0 (osd.0) 244 : cluster 0 9.5 scrub ok 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: cluster 2024-03-11T21:33:32.574094+0000 osd.3 (osd.3) 251 : cluster 0 5.17 scrub starts 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: cluster 2024-03-11T21:33:32.575647+0000 osd.3 (osd.3) 252 : cluster 0 5.17 scrub ok 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: audit 2024-03-11T21:33:33.346273+0000 mon.a (mon.0) 2518 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi143-71636-2"}]': finished 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: audit 2024-03-11T21:33:33.346512+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:34.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: audit 2024-03-11T21:33:33.346735+0000 mon.a (mon.0) 2520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:34.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: cluster 2024-03-11T21:33:33.355445+0000 mon.a (mon.0) 2521 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:33:34.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: audit 2024-03-11T21:33:33.361247+0000 mon.c (mon.2) 860 : audit 1 from='client.? 172.21.15.143:0/3787268790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:33 smithi143 bash[29284]: audit 2024-03-11T21:33:33.382906+0000 mon.a (mon.0) 2522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]: dispatch 2024-03-11T21:33:34.187 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:34.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:34.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:34.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.447 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:34.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:34.578 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:34.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:34.710 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:34.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:34.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: cluster 2024-03-11T21:33:33.359258+0000 mgr.y (mgr.14550) 421 : cluster 0 pgmap v606: 260 pgs: 19 creating+peering, 13 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:33.450991+0000 mon.c (mon.2) 861 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: cluster 2024-03-11T21:33:33.608515+0000 osd.3 (osd.3) 253 : cluster 0 6.6 scrub starts 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: cluster 2024-03-11T21:33:33.610066+0000 osd.3 (osd.3) 254 : cluster 0 6.6 scrub ok 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:33.869637+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: cluster 2024-03-11T21:33:34.346136+0000 mon.a (mon.0) 2524 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.356382+0000 mon.a (mon.0) 2525 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.356607+0000 mon.a (mon.0) 2526 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.366484+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:34.954 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: cluster 2024-03-11T21:33:34.366523+0000 mon.a (mon.0) 2527 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.372696+0000 mon.c (mon.2) 863 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.372840+0000 mon.a (mon.0) 2528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.374522+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.397511+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.398536+0000 mon.a (mon.0) 2531 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[21512]: audit 2024-03-11T21:33:34.398946+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: cluster 2024-03-11T21:33:33.359258+0000 mgr.y (mgr.14550) 421 : cluster 0 pgmap v606: 260 pgs: 19 creating+peering, 13 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:33.450991+0000 mon.c (mon.2) 861 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: cluster 2024-03-11T21:33:33.608515+0000 osd.3 (osd.3) 253 : cluster 0 6.6 scrub starts 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: cluster 2024-03-11T21:33:33.610066+0000 osd.3 (osd.3) 254 : cluster 0 6.6 scrub ok 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:33.869637+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:34.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: cluster 2024-03-11T21:33:34.346136+0000 mon.a (mon.0) 2524 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.356382+0000 mon.a (mon.0) 2525 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.356607+0000 mon.a (mon.0) 2526 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.366484+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: cluster 2024-03-11T21:33:34.366523+0000 mon.a (mon.0) 2527 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.372696+0000 mon.c (mon.2) 863 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.372840+0000 mon.a (mon.0) 2528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.374522+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.397511+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.398536+0000 mon.a (mon.0) 2531 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:34.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:34 smithi143 bash[29284]: audit 2024-03-11T21:33:34.398946+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:34.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:35.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:35.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:35.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: cluster 2024-03-11T21:33:33.359258+0000 mgr.y (mgr.14550) 421 : cluster 0 pgmap v606: 260 pgs: 19 creating+peering, 13 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:33:35.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:33.450991+0000 mon.c (mon.2) 861 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:35.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: cluster 2024-03-11T21:33:33.608515+0000 osd.3 (osd.3) 253 : cluster 0 6.6 scrub starts 2024-03-11T21:33:35.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: cluster 2024-03-11T21:33:33.610066+0000 osd.3 (osd.3) 254 : cluster 0 6.6 scrub ok 2024-03-11T21:33:35.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:33.869637+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: cluster 2024-03-11T21:33:34.346136+0000 mon.a (mon.0) 2524 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.356382+0000 mon.a (mon.0) 2525 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi143-71604-62"}]': finished 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.356607+0000 mon.a (mon.0) 2526 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.366484+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: cluster 2024-03-11T21:33:34.366523+0000 mon.a (mon.0) 2527 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.372696+0000 mon.c (mon.2) 863 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.372840+0000 mon.a (mon.0) 2528 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.374522+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.397511+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:35.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.398536+0000 mon.a (mon.0) 2531 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:35.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:34 smithi203 bash[23654]: audit 2024-03-11T21:33:34.398946+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:35.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.372 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:35.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:35.499 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:35.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:35.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:35.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.877 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:34.100992+0000 osd.1 (osd.1) 59 : cluster 0 39.1c scrub starts 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:34.103001+0000 osd.1 (osd.1) 60 : cluster 0 39.1c scrub ok 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.359870+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.360128+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.360415+0000 mon.a (mon.0) 2535 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.371110+0000 mon.a (mon.0) 2536 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.373044+0000 mon.c (mon.2) 864 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.381565+0000 mon.a (mon.0) 2537 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.383771+0000 mon.a (mon.0) 2538 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.413703+0000 mon.c (mon.2) 865 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:35.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: audit 2024-03-11T21:33:35.414030+0000 mon.a (mon.0) 2539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.540182+0000 osd.3 (osd.3) 255 : cluster 0 6.11 scrub starts 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.541727+0000 osd.3 (osd.3) 256 : cluster 0 6.11 scrub ok 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.630400+0000 osd.4 (osd.4) 217 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.635242+0000 osd.4 (osd.4) 218 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:34.100992+0000 osd.1 (osd.1) 59 : cluster 0 39.1c scrub starts 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:34.103001+0000 osd.1 (osd.1) 60 : cluster 0 39.1c scrub ok 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.359870+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.360128+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:35.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.360415+0000 mon.a (mon.0) 2535 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.371110+0000 mon.a (mon.0) 2536 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.373044+0000 mon.c (mon.2) 864 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.381565+0000 mon.a (mon.0) 2537 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.383771+0000 mon.a (mon.0) 2538 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.413703+0000 mon.c (mon.2) 865 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: audit 2024-03-11T21:33:35.414030+0000 mon.a (mon.0) 2539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.540182+0000 osd.3 (osd.3) 255 : cluster 0 6.11 scrub starts 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.541727+0000 osd.3 (osd.3) 256 : cluster 0 6.11 scrub ok 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.630400+0000 osd.4 (osd.4) 217 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:33:35.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:35 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.635242+0000 osd.4 (osd.4) 218 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:33:35.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:35.996 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:36.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:34.100992+0000 osd.1 (osd.1) 59 : cluster 0 39.1c scrub starts 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:34.103001+0000 osd.1 (osd.1) 60 : cluster 0 39.1c scrub ok 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.359870+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.360128+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71636-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.360415+0000 mon.a (mon.0) 2535 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi143-71604-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.371110+0000 mon.a (mon.0) 2536 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:33:36.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.373044+0000 mon.c (mon.2) 864 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.381565+0000 mon.a (mon.0) 2537 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.383771+0000 mon.a (mon.0) 2538 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]: dispatch 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.413703+0000 mon.c (mon.2) 865 : audit 1 from='client.? 172.21.15.143:0/1289133983' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: audit 2024-03-11T21:33:35.414030+0000 mon.a (mon.0) 2539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.540182+0000 osd.3 (osd.3) 255 : cluster 0 6.11 scrub starts 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.541727+0000 osd.3 (osd.3) 256 : cluster 0 6.11 scrub ok 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.630400+0000 osd.4 (osd.4) 217 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:33:36.140 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:35 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.635242+0000 osd.4 (osd.4) 218 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:33:36.153 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:36.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:36.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:36.368 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-11T21:33:36.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 1 0 2024-03-11T21:33:36.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 10 0 2024-03-11T21:33:36.369 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 13 0 2024-03-11T21:33:36.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 7 0 2024-03-11T21:33:36.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 14 0 2024-03-11T21:33:36.370 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 0 0 2024-03-11T21:33:36.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 15 0 2024-03-11T21:33:36.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 11 0 2024-03-11T21:33:36.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 5 0 2024-03-11T21:33:36.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 8 0 2024-03-11T21:33:36.371 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 6 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 3 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 4 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 12 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 9 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2 0 2024-03-11T21:33:36.372 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (70 ms) 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 3 tests from LibRadosListEC (1170 ms total) 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-11T21:33:36.373 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3024 ms) 2024-03-11T21:33:36.374 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 1 test from LibRadosListNP (3024 ms total) 2024-03-11T21:33:36.374 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-03-11T21:33:36.374 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] Global test environment tear-down 2024-03-11T21:33:36.374 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [==========] 11 tests from 3 test suites ran. (300781 ms total) 2024-03-11T21:33:36.374 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ PASSED ] 11 tests. 2024-03-11T21:33:36.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:36.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.542 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:36.645 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.670 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:36.773 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.796 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:36.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.928 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:37.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:37.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.360391+0000 mgr.y (mgr.14550) 422 : cluster 0 pgmap v608: 292 pgs: 19 creating+peering, 41 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.433030+0000 osd.6 (osd.6) 187 : cluster 0 9.18 scrub starts 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.434625+0000 osd.6 (osd.6) 188 : cluster 0 9.18 scrub ok 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.827430+0000 osd.4 (osd.4) 219 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.827888+0000 osd.4 (osd.4) 220 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:37.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.277185+0000 mon.a (mon.0) 2540 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.280893+0000 mon.c (mon.2) 866 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.370872+0000 mon.a (mon.0) 2541 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.379216+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]': finished 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.379283+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.385293+0000 mon.c (mon.2) 867 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.820229+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[21512]: audit 2024-03-11T21:33:36.821057+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.360391+0000 mgr.y (mgr.14550) 422 : cluster 0 pgmap v608: 292 pgs: 19 creating+peering, 41 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:33:37.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.433030+0000 osd.6 (osd.6) 187 : cluster 0 9.18 scrub starts 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.434625+0000 osd.6 (osd.6) 188 : cluster 0 9.18 scrub ok 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.827430+0000 osd.4 (osd.4) 219 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.827888+0000 osd.4 (osd.4) 220 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.277185+0000 mon.a (mon.0) 2540 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.280893+0000 mon.c (mon.2) 866 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.370872+0000 mon.a (mon.0) 2541 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.379216+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]': finished 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.379283+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.385293+0000 mon.c (mon.2) 867 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.820229+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:33:37.112 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:36 smithi143 bash[29284]: audit 2024-03-11T21:33:36.821057+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.181 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.360391+0000 mgr.y (mgr.14550) 422 : cluster 0 pgmap v608: 292 pgs: 19 creating+peering, 41 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.433030+0000 osd.6 (osd.6) 187 : cluster 0 9.18 scrub starts 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.434625+0000 osd.6 (osd.6) 188 : cluster 0 9.18 scrub ok 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.827430+0000 osd.4 (osd.4) 219 : cluster 0 purged_snaps scrub starts 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.827888+0000 osd.4 (osd.4) 220 : cluster 0 purged_snaps scrub ok 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.277185+0000 mon.a (mon.0) 2540 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.280893+0000 mon.c (mon.2) 866 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.370872+0000 mon.a (mon.0) 2541 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.379216+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-47", "mode": "writeback"}]': finished 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.379283+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi143-71636-3","pool2":"test-rados-api-smithi143-71636-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.385293+0000 mon.c (mon.2) 867 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.820229+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:33:37.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:36 smithi203 bash[23654]: audit 2024-03-11T21:33:36.821057+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:33:37.198 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:37.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.327 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:37.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:37.454 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:37.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:37.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:37.717 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:37.820 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:37.847 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:37.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:38.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.975103+0000 osd.0 (osd.0) 245 : cluster 0 5.5 scrub starts 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:35.976713+0000 osd.0 (osd.0) 246 : cluster 0 5.5 scrub ok 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.118454+0000 osd.1 (osd.1) 61 : cluster 0 5.1d scrub starts 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.120592+0000 osd.1 (osd.1) 62 : cluster 0 5.1d scrub ok 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.587316+0000 osd.7 (osd.7) 271 : cluster 0 2.8 scrub starts 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:36.592084+0000 osd.7 (osd.7) 272 : cluster 0 2.8 scrub ok 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: audit 2024-03-11T21:33:37.383033+0000 mon.a (mon.0) 2546 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:38.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: audit 2024-03-11T21:33:37.383312+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.390115+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: audit 2024-03-11T21:33:37.400121+0000 mon.c (mon.2) 868 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[29284]: audit 2024-03-11T21:33:37.401042+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.975103+0000 osd.0 (osd.0) 245 : cluster 0 5.5 scrub starts 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:35.976713+0000 osd.0 (osd.0) 246 : cluster 0 5.5 scrub ok 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.118454+0000 osd.1 (osd.1) 61 : cluster 0 5.1d scrub starts 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.120592+0000 osd.1 (osd.1) 62 : cluster 0 5.1d scrub ok 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.587316+0000 osd.7 (osd.7) 271 : cluster 0 2.8 scrub starts 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:36.592084+0000 osd.7 (osd.7) 272 : cluster 0 2.8 scrub ok 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: audit 2024-03-11T21:33:37.383033+0000 mon.a (mon.0) 2546 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:38.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: audit 2024-03-11T21:33:37.383312+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:33:38.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.390115+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:33:38.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: audit 2024-03-11T21:33:37.400121+0000 mon.c (mon.2) 868 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:37 smithi143 bash[21512]: audit 2024-03-11T21:33:37.401042+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:38.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.975103+0000 osd.0 (osd.0) 245 : cluster 0 5.5 scrub starts 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:35.976713+0000 osd.0 (osd.0) 246 : cluster 0 5.5 scrub ok 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.118454+0000 osd.1 (osd.1) 61 : cluster 0 5.1d scrub starts 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.120592+0000 osd.1 (osd.1) 62 : cluster 0 5.1d scrub ok 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.587316+0000 osd.7 (osd.7) 271 : cluster 0 2.8 scrub starts 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:36.592084+0000 osd.7 (osd.7) 272 : cluster 0 2.8 scrub ok 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: audit 2024-03-11T21:33:37.383033+0000 mon.a (mon.0) 2546 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi143-71604-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: audit 2024-03-11T21:33:37.383312+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.390115+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:33:38.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: audit 2024-03-11T21:33:37.400121+0000 mon.c (mon.2) 868 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:37 smithi203 bash[23654]: audit 2024-03-11T21:33:37.401042+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:33:38.239 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:38.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:38.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:38.497 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:38.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:38.626 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:38.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.755 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:38.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:38.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:38.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.361304+0000 mgr.y (mgr.14550) 423 : cluster 0 pgmap v611: 260 pgs: 19 creating+peering, 241 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 510 B/s wr, 0 op/s 2024-03-11T21:33:38.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.459909+0000 osd.5 (osd.5) 261 : cluster 0 3.5 scrub starts 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.464480+0000 osd.5 (osd.5) 262 : cluster 0 3.5 scrub ok 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.695549+0000 osd.4 (osd.4) 221 : cluster 0 4.1 scrub starts 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:37.698085+0000 osd.4 (osd.4) 222 : cluster 0 4.1 scrub ok 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: audit 2024-03-11T21:33:38.384514+0000 mon.a (mon.0) 2550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: cluster 2024-03-11T21:33:38.390735+0000 mon.a (mon.0) 2551 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: audit 2024-03-11T21:33:38.396416+0000 mon.c (mon.2) 869 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: audit 2024-03-11T21:33:38.397436+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:38.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:38 smithi203 bash[23654]: audit 2024-03-11T21:33:38.618797+0000 mon.c (mon.2) 870 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:38.881 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:38.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.004 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:39.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.361304+0000 mgr.y (mgr.14550) 423 : cluster 0 pgmap v611: 260 pgs: 19 creating+peering, 241 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 510 B/s wr, 0 op/s 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.459909+0000 osd.5 (osd.5) 261 : cluster 0 3.5 scrub starts 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.464480+0000 osd.5 (osd.5) 262 : cluster 0 3.5 scrub ok 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.695549+0000 osd.4 (osd.4) 221 : cluster 0 4.1 scrub starts 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:37.698085+0000 osd.4 (osd.4) 222 : cluster 0 4.1 scrub ok 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: audit 2024-03-11T21:33:38.384514+0000 mon.a (mon.0) 2550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: cluster 2024-03-11T21:33:38.390735+0000 mon.a (mon.0) 2551 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: audit 2024-03-11T21:33:38.396416+0000 mon.c (mon.2) 869 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: audit 2024-03-11T21:33:38.397436+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:39.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[21512]: audit 2024-03-11T21:33:38.618797+0000 mon.c (mon.2) 870 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.361304+0000 mgr.y (mgr.14550) 423 : cluster 0 pgmap v611: 260 pgs: 19 creating+peering, 241 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 510 B/s wr, 0 op/s 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.459909+0000 osd.5 (osd.5) 261 : cluster 0 3.5 scrub starts 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.464480+0000 osd.5 (osd.5) 262 : cluster 0 3.5 scrub ok 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.695549+0000 osd.4 (osd.4) 221 : cluster 0 4.1 scrub starts 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:37.698085+0000 osd.4 (osd.4) 222 : cluster 0 4.1 scrub ok 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: audit 2024-03-11T21:33:38.384514+0000 mon.a (mon.0) 2550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: cluster 2024-03-11T21:33:38.390735+0000 mon.a (mon.0) 2551 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: audit 2024-03-11T21:33:38.396416+0000 mon.c (mon.2) 869 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: audit 2024-03-11T21:33:38.397436+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:33:39.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:38 smithi143 bash[29284]: audit 2024-03-11T21:33:38.618797+0000 mon.c (mon.2) 870 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:39.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:39.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.257 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:39.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:39.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:39.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:39.520 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:39.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:39.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:39.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:39.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:38.585509+0000 mgr.y (mgr.14550) 424 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:38.726740+0000 osd.4 (osd.4) 223 : cluster 0 3.14 scrub starts 2024-03-11T21:33:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:38.728908+0000 osd.4 (osd.4) 224 : cluster 0 3.14 scrub ok 2024-03-11T21:33:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.121726+0000 osd.2 (osd.2) 203 : cluster 0 118.9 scrub starts 2024-03-11T21:33:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.123375+0000 osd.2 (osd.2) 204 : cluster 0 118.9 scrub ok 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.301082+0000 mon.c (mon.2) 871 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.302335+0000 mon.c (mon.2) 872 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.310699+0000 mon.a (mon.0) 2553 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.385056+0000 mon.a (mon.0) 2554 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.387133+0000 mon.a (mon.0) 2555 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.394532+0000 mon.a (mon.0) 2556 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.399896+0000 mon.a (mon.0) 2557 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.401140+0000 mon.c (mon.2) 873 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: audit 2024-03-11T21:33:39.402379+0000 mon.a (mon.0) 2558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:39.896 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.594185+0000 osd.3 (osd.3) 257 : cluster 0 9.2 scrub starts 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.596773+0000 osd.3 (osd.3) 258 : cluster 0 9.2 scrub ok 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:38.585509+0000 mgr.y (mgr.14550) 424 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:38.726740+0000 osd.4 (osd.4) 223 : cluster 0 3.14 scrub starts 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:38.728908+0000 osd.4 (osd.4) 224 : cluster 0 3.14 scrub ok 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.121726+0000 osd.2 (osd.2) 203 : cluster 0 118.9 scrub starts 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.123375+0000 osd.2 (osd.2) 204 : cluster 0 118.9 scrub ok 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.301082+0000 mon.c (mon.2) 871 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.302335+0000 mon.c (mon.2) 872 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:39.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.310699+0000 mon.a (mon.0) 2553 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.385056+0000 mon.a (mon.0) 2554 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.387133+0000 mon.a (mon.0) 2555 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.394532+0000 mon.a (mon.0) 2556 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.399896+0000 mon.a (mon.0) 2557 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.401140+0000 mon.c (mon.2) 873 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: audit 2024-03-11T21:33:39.402379+0000 mon.a (mon.0) 2558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.594185+0000 osd.3 (osd.3) 257 : cluster 0 9.2 scrub starts 2024-03-11T21:33:39.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:39 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.596773+0000 osd.3 (osd.3) 258 : cluster 0 9.2 scrub ok 2024-03-11T21:33:39.908 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:40.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:40.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:38.585509+0000 mgr.y (mgr.14550) 424 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:38.726740+0000 osd.4 (osd.4) 223 : cluster 0 3.14 scrub starts 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:38.728908+0000 osd.4 (osd.4) 224 : cluster 0 3.14 scrub ok 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.121726+0000 osd.2 (osd.2) 203 : cluster 0 118.9 scrub starts 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.123375+0000 osd.2 (osd.2) 204 : cluster 0 118.9 scrub ok 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.301082+0000 mon.c (mon.2) 871 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.302335+0000 mon.c (mon.2) 872 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.310699+0000 mon.a (mon.0) 2553 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.385056+0000 mon.a (mon.0) 2554 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.387133+0000 mon.a (mon.0) 2555 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.394532+0000 mon.a (mon.0) 2556 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.399896+0000 mon.a (mon.0) 2557 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:40.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.401140+0000 mon.c (mon.2) 873 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: audit 2024-03-11T21:33:39.402379+0000 mon.a (mon.0) 2558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:33:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.594185+0000 osd.3 (osd.3) 257 : cluster 0 9.2 scrub starts 2024-03-11T21:33:40.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:39 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.596773+0000 osd.3 (osd.3) 258 : cluster 0 9.2 scrub ok 2024-03-11T21:33:40.038 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:40.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:40.170 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:40.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:40.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:40.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:40.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:40.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:40.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:40.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:40.691 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:40.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:40.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:40.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:40.954 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:41.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:41.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:41.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.130335+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub starts 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.131506+0000 osd.1 (osd.1) 64 : cluster 0 9.0 scrub ok 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.362082+0000 mgr.y (mgr.14550) 425 : cluster 0 pgmap v614: 268 pgs: 1 creating+activating, 3 creating+peering, 1 unknown, 263 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.671034+0000 osd.7 (osd.7) 273 : cluster 0 6.5 scrub starts 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.685847+0000 osd.7 (osd.7) 274 : cluster 0 6.5 scrub ok 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.306810+0000 mon.a (mon.0) 2559 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: audit 2024-03-11T21:33:40.389683+0000 mon.a (mon.0) 2560 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: audit 2024-03-11T21:33:40.389849+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.397132+0000 mon.a (mon.0) 2562 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: audit 2024-03-11T21:33:40.400229+0000 mon.c (mon.2) 874 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[29284]: audit 2024-03-11T21:33:40.400562+0000 mon.a (mon.0) 2563 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:40.401708+0000 mon.a (mon.0) 2564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.546963+0000 osd.3 (osd.3) 259 : cluster 0 4.0 scrub starts 2024-03-11T21:33:41.127 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.550100+0000 osd.3 (osd.3) 260 : cluster 0 4.0 scrub ok 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.130335+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub starts 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.131506+0000 osd.1 (osd.1) 64 : cluster 0 9.0 scrub ok 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.362082+0000 mgr.y (mgr.14550) 425 : cluster 0 pgmap v614: 268 pgs: 1 creating+activating, 3 creating+peering, 1 unknown, 263 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.671034+0000 osd.7 (osd.7) 273 : cluster 0 6.5 scrub starts 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.685847+0000 osd.7 (osd.7) 274 : cluster 0 6.5 scrub ok 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.306810+0000 mon.a (mon.0) 2559 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: audit 2024-03-11T21:33:40.389683+0000 mon.a (mon.0) 2560 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: audit 2024-03-11T21:33:40.389849+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.397132+0000 mon.a (mon.0) 2562 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: audit 2024-03-11T21:33:40.400229+0000 mon.c (mon.2) 874 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.128 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:40 smithi143 bash[21512]: audit 2024-03-11T21:33:40.400562+0000 mon.a (mon.0) 2563 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:41.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:40.401708+0000 mon.a (mon.0) 2564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.546963+0000 osd.3 (osd.3) 259 : cluster 0 4.0 scrub starts 2024-03-11T21:33:41.129 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.550100+0000 osd.3 (osd.3) 260 : cluster 0 4.0 scrub ok 2024-03-11T21:33:41.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.130335+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub starts 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.131506+0000 osd.1 (osd.1) 64 : cluster 0 9.0 scrub ok 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.362082+0000 mgr.y (mgr.14550) 425 : cluster 0 pgmap v614: 268 pgs: 1 creating+activating, 3 creating+peering, 1 unknown, 263 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.671034+0000 osd.7 (osd.7) 273 : cluster 0 6.5 scrub starts 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.685847+0000 osd.7 (osd.7) 274 : cluster 0 6.5 scrub ok 2024-03-11T21:33:41.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.306810+0000 mon.a (mon.0) 2559 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: audit 2024-03-11T21:33:40.389683+0000 mon.a (mon.0) 2560 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: audit 2024-03-11T21:33:40.389849+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.397132+0000 mon.a (mon.0) 2562 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: audit 2024-03-11T21:33:40.400229+0000 mon.c (mon.2) 874 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: audit 2024-03-11T21:33:40.400562+0000 mon.a (mon.0) 2563 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]: dispatch 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: audit 2024-03-11T21:33:40.401708+0000 mon.a (mon.0) 2564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.546963+0000 osd.3 (osd.3) 259 : cluster 0 4.0 scrub starts 2024-03-11T21:33:41.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:40 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.550100+0000 osd.3 (osd.3) 260 : cluster 0 4.0 scrub ok 2024-03-11T21:33:41.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:41.302 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:41.302 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:41.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:41.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:41.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- 172.21.15.143:0/1188682591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468102a10 msgr2=0x7f9468106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:41.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- 172.21.15.143:0/1188682591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468102a10 0x7f9468106a60 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f94580099c0 tx=0x7f9458009cd0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- 172.21.15.143:0/1188682591 shutdown_connections 2024-03-11T21:33:41.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- 172.21.15.143:0/1188682591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468107140 0x7f946810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- 172.21.15.143:0/1188682591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102060 0x7f9468102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- 172.21.15.143:0/1188682591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468102a10 0x7f9468106a60 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- 172.21.15.143:0/1188682591 >> 172.21.15.143:0/1188682591 conn(0x7f94680fd8b0 msgr2=0x7f94680ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- 172.21.15.143:0/1188682591 shutdown_connections 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- 172.21.15.143:0/1188682591 wait complete. 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 librados: starting msgr at 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 librados: starting objecter 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 Processor -- start 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- start start 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 librados: setting wanted keys 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 librados: calling monclient init 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102a10 0x7f946819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468107140 0x7f9468195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:41.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468075e00 con 0x7f9468107140 2024-03-11T21:33:41.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468075c80 con 0x7f9468102a10 2024-03-11T21:33:41.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946ff75700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9468075f80 con 0x7f9468102060 2024-03-11T21:33:41.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48820/0 (socket says 172.21.15.143:48820) 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 -- 172.21.15.143:0/1918488756 learned_addr learned my addr 172.21.15.143:0/1918488756 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946e512700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468107140 0x7f9468195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946d510700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102a10 0x7f946819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 -- 172.21.15.143:0/1918488756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102a10 msgr2=0x7f946819bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102a10 0x7f946819bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 -- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468107140 msgr2=0x7f9468195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468107140 0x7f9468195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.432+0000 7f946dd11700 1 -- 172.21.15.143:0/1918488756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468196010 con 0x7f9468102060 2024-03-11T21:33:41.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946dd11700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f945c00ada0 tx=0x7f945c00c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:41.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 10 client.?.objecter ms_handle_connect 0x7f9468102060 2024-03-11T21:33:41.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:41.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c017020 con 0x7f9468102060 2024-03-11T21:33:41.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f945c00ce00 con 0x7f9468102060 2024-03-11T21:33:41.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9468196300 con 0x7f9468102060 2024-03-11T21:33:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0052a0 con 0x7f9468102060 2024-03-11T21:33:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946ff75700 10 client.41576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94681a2510 con 0x7f9468102060 2024-03-11T21:33:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946ff75700 1 librados: init done 2024-03-11T21:33:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9468103540 con 0x7f9468102060 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f945c00c980 con 0x7f9468102060 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 0x7f9450071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(420..420 src has 87..420) v4 ==== 9444+0+0 (secure 0 0 0) 0x7f945c08e310 con 0x7f9468102060 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 10 client.41576.objecter ms_dispatch 0x7f9468001550 osd_map(420..420 src has 87..420) v4 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 3 client.41576.objecter handle_osd_map got epochs [420,420] > 0 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 3 client.41576.objecter handle_osd_map decoding full epoch 420 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946d510700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 0x7f9450071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 20 client.41576.objecter dump_active .. 0 homeless 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f946d510700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 0x7f9450071ce0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9458009990 tx=0x7f9458005040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.436+0000 7f9466ffd700 10 client.41576.objecter ms_handle_connect 0x7f945006f820 2024-03-11T21:33:41.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.440+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f945c021020 con 0x7f9468102060 2024-03-11T21:33:41.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:41.479 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:41.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:41.605 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:41.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.652+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f946804faa0 con 0x7f945006f820 2024-03-11T21:33:41.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.672+0000 7f9466ffd700 1 -- 172.21.15.143:0/1918488756 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+666304 (secure 0 0 0) 0x7f946804faa0 con 0x7f945006f820 2024-03-11T21:33:41.660 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:33:41.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 10 librados: watch_flush enter 2024-03-11T21:33:41.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 10 librados: watch_flush exit 2024-03-11T21:33:41.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 20 client.41576.objecter shutdown clearing up homeless session... 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 10 client.41576.objecter successfully canceled tick 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 msgr2=0x7f9450071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 0x7f9450071ce0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9458009990 tx=0x7f9458005040 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 msgr2=0x7f946819b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f945c00ada0 tx=0x7f945c00c330 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 shutdown_connections 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f945006f820 0x7f9450071ce0 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9468107140 0x7f9468195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9468102060 0x7f946819b6b0 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 --2- 172.21.15.143:0/1918488756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9468102a10 0x7f946819bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 >> 172.21.15.143:0/1918488756 conn(0x7f94680fd8b0 msgr2=0x7f94681062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 shutdown_connections 2024-03-11T21:33:41.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 -- 172.21.15.143:0/1918488756 wait complete. 2024-03-11T21:33:41.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:41.676+0000 7f946ff75700 1 librados: shutdown 2024-03-11T21:33:41.671 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:33:41.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:41.738 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:41.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:41.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:41.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.967446+0000 osd.0 (osd.0) 247 : cluster 0 3.1f scrub starts 2024-03-11T21:33:41.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:39.970164+0000 osd.0 (osd.0) 248 : cluster 0 3.1f scrub ok 2024-03-11T21:33:41.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.432672+0000 osd.6 (osd.6) 189 : cluster 0 39.1d scrub starts 2024-03-11T21:33:41.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.435627+0000 osd.6 (osd.6) 190 : cluster 0 39.1d scrub ok 2024-03-11T21:33:41.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.724456+0000 osd.4 (osd.4) 225 : cluster 0 6.17 scrub starts 2024-03-11T21:33:41.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.729848+0000 osd.4 (osd.4) 226 : cluster 0 6.17 scrub ok 2024-03-11T21:33:41.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.203315+0000 osd.2 (osd.2) 205 : cluster 0 5.d scrub starts 2024-03-11T21:33:41.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.205239+0000 osd.2 (osd.2) 206 : cluster 0 5.d scrub ok 2024-03-11T21:33:41.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.310838+0000 mon.a (mon.0) 2565 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.310930+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:33:41.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.313657+0000 mon.a (mon.0) 2567 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:33:41.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.322858+0000 mon.c (mon.2) 875 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.330407+0000 mon.a (mon.0) 2568 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.348623+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.349100+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.349810+0000 mon.c (mon.2) 877 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.350131+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.350759+0000 mon.c (mon.2) 878 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[29284]: audit 2024-03-11T21:33:41.351058+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.967446+0000 osd.0 (osd.0) 247 : cluster 0 3.1f scrub starts 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:39.970164+0000 osd.0 (osd.0) 248 : cluster 0 3.1f scrub ok 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.432672+0000 osd.6 (osd.6) 189 : cluster 0 39.1d scrub starts 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.435627+0000 osd.6 (osd.6) 190 : cluster 0 39.1d scrub ok 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.724456+0000 osd.4 (osd.4) 225 : cluster 0 6.17 scrub starts 2024-03-11T21:33:41.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.729848+0000 osd.4 (osd.4) 226 : cluster 0 6.17 scrub ok 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.203315+0000 osd.2 (osd.2) 205 : cluster 0 5.d scrub starts 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.205239+0000 osd.2 (osd.2) 206 : cluster 0 5.d scrub ok 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.310838+0000 mon.a (mon.0) 2565 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.310930+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.313657+0000 mon.a (mon.0) 2567 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.322858+0000 mon.c (mon.2) 875 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.330407+0000 mon.a (mon.0) 2568 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.348623+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.349100+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.349810+0000 mon.c (mon.2) 877 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.350131+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.350759+0000 mon.c (mon.2) 878 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:41.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:41 smithi143 bash[21512]: audit 2024-03-11T21:33:41.351058+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:41.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.967446+0000 osd.0 (osd.0) 247 : cluster 0 3.1f scrub starts 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:39.970164+0000 osd.0 (osd.0) 248 : cluster 0 3.1f scrub ok 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.432672+0000 osd.6 (osd.6) 189 : cluster 0 39.1d scrub starts 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.435627+0000 osd.6 (osd.6) 190 : cluster 0 39.1d scrub ok 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.724456+0000 osd.4 (osd.4) 225 : cluster 0 6.17 scrub starts 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.729848+0000 osd.4 (osd.4) 226 : cluster 0 6.17 scrub ok 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.203315+0000 osd.2 (osd.2) 205 : cluster 0 5.d scrub starts 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.205239+0000 osd.2 (osd.2) 206 : cluster 0 5.d scrub ok 2024-03-11T21:33:41.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.310838+0000 mon.a (mon.0) 2565 : audit 1 from='client.? 172.21.15.143:0/2163146794' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi143-71604-63"}]': finished 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.310930+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.313657+0000 mon.a (mon.0) 2567 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.322858+0000 mon.c (mon.2) 875 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.330407+0000 mon.a (mon.0) 2568 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.348623+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.349100+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.349810+0000 mon.c (mon.2) 877 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.350131+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.350759+0000 mon.c (mon.2) 878 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:41.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:41 smithi203 bash[23654]: audit 2024-03-11T21:33:41.351058+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:42.007 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:42.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.126 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:42.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:42.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:42.273 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:42.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.404 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:42.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:42.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:42.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:42.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.789 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:42.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:42.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.986026+0000 osd.0 (osd.0) 249 : cluster 0 9.9 scrub starts 2024-03-11T21:33:42.897 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:40.988249+0000 osd.0 (osd.0) 250 : cluster 0 9.9 scrub ok 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.362554+0000 mgr.y (mgr.14550) 426 : cluster 0 pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.552811+0000 osd.5 (osd.5) 263 : cluster 0 39.0 scrub starts 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.560379+0000 osd.5 (osd.5) 264 : cluster 0 39.0 scrub ok 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:41.657516+0000 mgr.y (mgr.14550) 427 : audit 0 from='client.41576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.773317+0000 osd.4 (osd.4) 227 : cluster 0 6.1e scrub starts 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.774693+0000 osd.4 (osd.4) 228 : cluster 0 6.1e scrub ok 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.164450+0000 osd.2 (osd.2) 207 : cluster 0 118.17 scrub starts 2024-03-11T21:33:42.898 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.167690+0000 osd.2 (osd.2) 208 : cluster 0 118.17 scrub ok 2024-03-11T21:33:42.919 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:43.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.049 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:43.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:43.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.986026+0000 osd.0 (osd.0) 249 : cluster 0 9.9 scrub starts 2024-03-11T21:33:43.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:40.988249+0000 osd.0 (osd.0) 250 : cluster 0 9.9 scrub ok 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.362554+0000 mgr.y (mgr.14550) 426 : cluster 0 pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.552811+0000 osd.5 (osd.5) 263 : cluster 0 39.0 scrub starts 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.560379+0000 osd.5 (osd.5) 264 : cluster 0 39.0 scrub ok 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:41.657516+0000 mgr.y (mgr.14550) 427 : audit 0 from='client.41576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.773317+0000 osd.4 (osd.4) 227 : cluster 0 6.1e scrub starts 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.774693+0000 osd.4 (osd.4) 228 : cluster 0 6.1e scrub ok 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.164450+0000 osd.2 (osd.2) 207 : cluster 0 118.17 scrub starts 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.167690+0000 osd.2 (osd.2) 208 : cluster 0 118.17 scrub ok 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.332473+0000 mon.a (mon.0) 2572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.332656+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:43.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.339533+0000 mon.a (mon.0) 2574 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.340819+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.342092+0000 mon.a (mon.0) 2575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.422289+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: audit 2024-03-11T21:33:42.422938+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.589143+0000 osd.3 (osd.3) 261 : cluster 0 9.1b scrub starts 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.590959+0000 osd.3 (osd.3) 262 : cluster 0 9.1b scrub ok 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.332473+0000 mon.a (mon.0) 2572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.332656+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:43.166 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.339533+0000 mon.a (mon.0) 2574 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.340819+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.342092+0000 mon.a (mon.0) 2575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.422289+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: audit 2024-03-11T21:33:42.422938+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.589143+0000 osd.3 (osd.3) 261 : cluster 0 9.1b scrub starts 2024-03-11T21:33:43.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:42 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.590959+0000 osd.3 (osd.3) 262 : cluster 0 9.1b scrub ok 2024-03-11T21:33:43.179 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:43.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.986026+0000 osd.0 (osd.0) 249 : cluster 0 9.9 scrub starts 2024-03-11T21:33:43.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:40.988249+0000 osd.0 (osd.0) 250 : cluster 0 9.9 scrub ok 2024-03-11T21:33:43.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.362554+0000 mgr.y (mgr.14550) 426 : cluster 0 pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-11T21:33:43.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.552811+0000 osd.5 (osd.5) 263 : cluster 0 39.0 scrub starts 2024-03-11T21:33:43.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.560379+0000 osd.5 (osd.5) 264 : cluster 0 39.0 scrub ok 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:41.657516+0000 mgr.y (mgr.14550) 427 : audit 0 from='client.41576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.773317+0000 osd.4 (osd.4) 227 : cluster 0 6.1e scrub starts 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.774693+0000 osd.4 (osd.4) 228 : cluster 0 6.1e scrub ok 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.164450+0000 osd.2 (osd.2) 207 : cluster 0 118.17 scrub starts 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.167690+0000 osd.2 (osd.2) 208 : cluster 0 118.17 scrub ok 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.332473+0000 mon.a (mon.0) 2572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:33:43.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.332656+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi143-71604-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.339533+0000 mon.a (mon.0) 2574 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.340819+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.342092+0000 mon.a (mon.0) 2575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.422289+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: audit 2024-03-11T21:33:42.422938+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.589143+0000 osd.3 (osd.3) 261 : cluster 0 9.1b scrub starts 2024-03-11T21:33:43.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:42 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.590959+0000 osd.3 (osd.3) 262 : cluster 0 9.1b scrub ok 2024-03-11T21:33:43.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:43.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:43.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:43.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:43.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:43.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:43.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:43.828 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:43.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:43.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.972058+0000 osd.0 (osd.0) 251 : cluster 0 39.12 scrub starts 2024-03-11T21:33:43.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: cluster 2024-03-11T21:33:41.973903+0000 osd.0 (osd.0) 252 : cluster 0 39.12 scrub ok 2024-03-11T21:33:43.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.144785+0000 osd.2 (osd.2) 209 : cluster 0 5.6 scrub starts 2024-03-11T21:33:43.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.146843+0000 osd.2 (osd.2) 210 : cluster 0 5.6 scrub ok 2024-03-11T21:33:43.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: audit 2024-03-11T21:33:43.335874+0000 mon.a (mon.0) 2577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:33:43.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: audit 2024-03-11T21:33:43.345549+0000 mon.c (mon.2) 881 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:43.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.798786+0000 mon.a (mon.0) 2578 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:33:43.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:43 smithi203 bash[23654]: audit 2024-03-11T21:33:43.800343+0000 mon.a (mon.0) 2579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:43.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:44.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.972058+0000 osd.0 (osd.0) 251 : cluster 0 39.12 scrub starts 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: cluster 2024-03-11T21:33:41.973903+0000 osd.0 (osd.0) 252 : cluster 0 39.12 scrub ok 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.144785+0000 osd.2 (osd.2) 209 : cluster 0 5.6 scrub starts 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.146843+0000 osd.2 (osd.2) 210 : cluster 0 5.6 scrub ok 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: audit 2024-03-11T21:33:43.335874+0000 mon.a (mon.0) 2577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: audit 2024-03-11T21:33:43.345549+0000 mon.c (mon.2) 881 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.798786+0000 mon.a (mon.0) 2578 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:33:44.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[29284]: audit 2024-03-11T21:33:43.800343+0000 mon.a (mon.0) 2579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.972058+0000 osd.0 (osd.0) 251 : cluster 0 39.12 scrub starts 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: cluster 2024-03-11T21:33:41.973903+0000 osd.0 (osd.0) 252 : cluster 0 39.12 scrub ok 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.144785+0000 osd.2 (osd.2) 209 : cluster 0 5.6 scrub starts 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.146843+0000 osd.2 (osd.2) 210 : cluster 0 5.6 scrub ok 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: audit 2024-03-11T21:33:43.335874+0000 mon.a (mon.0) 2577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: audit 2024-03-11T21:33:43.345549+0000 mon.c (mon.2) 881 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.798786+0000 mon.a (mon.0) 2578 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:33:44.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:43 smithi143 bash[21512]: audit 2024-03-11T21:33:43.800343+0000 mon.a (mon.0) 2579 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:44.089 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:44.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:44.220 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:44.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:44.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:44.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:44.484 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:44.587 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:44.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:44.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:44.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:44.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:44.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:44.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:45.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:45.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:45.135 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:45.136 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.964436+0000 osd.0 (osd.0) 253 : cluster 0 3.12 scrub starts 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:42.968495+0000 osd.0 (osd.0) 254 : cluster 0 3.12 scrub ok 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.131231+0000 osd.1 (osd.1) 65 : cluster 0 9.4 scrub starts 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.133475+0000 osd.1 (osd.1) 66 : cluster 0 9.4 scrub ok 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:43.363310+0000 mgr.y (mgr.14550) 428 : cluster 0 pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.346411+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.346654+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: cluster 2024-03-11T21:33:44.355074+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.405724+0000 mon.c (mon.2) 882 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.406391+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.407303+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:44 smithi203 bash[23654]: audit 2024-03-11T21:33:44.407828+0000 mon.a (mon.0) 2584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:45.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:45.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:45.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.393 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.964436+0000 osd.0 (osd.0) 253 : cluster 0 3.12 scrub starts 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:42.968495+0000 osd.0 (osd.0) 254 : cluster 0 3.12 scrub ok 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.131231+0000 osd.1 (osd.1) 65 : cluster 0 9.4 scrub starts 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.133475+0000 osd.1 (osd.1) 66 : cluster 0 9.4 scrub ok 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:43.363310+0000 mgr.y (mgr.14550) 428 : cluster 0 pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.346411+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.346654+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: cluster 2024-03-11T21:33:44.355074+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.405724+0000 mon.c (mon.2) 882 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.406391+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.407303+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[29284]: audit 2024-03-11T21:33:44.407828+0000 mon.a (mon.0) 2584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.964436+0000 osd.0 (osd.0) 253 : cluster 0 3.12 scrub starts 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:42.968495+0000 osd.0 (osd.0) 254 : cluster 0 3.12 scrub ok 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.131231+0000 osd.1 (osd.1) 65 : cluster 0 9.4 scrub starts 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.133475+0000 osd.1 (osd.1) 66 : cluster 0 9.4 scrub ok 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:43.363310+0000 mgr.y (mgr.14550) 428 : cluster 0 pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.346411+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi143-71604-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.346654+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]': finished 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: cluster 2024-03-11T21:33:44.355074+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.405724+0000 mon.c (mon.2) 882 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.406391+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.407303+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:45 smithi143 bash[21512]: audit 2024-03-11T21:33:44.407828+0000 mon.a (mon.0) 2584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-47"}]: dispatch 2024-03-11T21:33:45.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:33:45.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:45.552 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:45.626 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-03-11T21:33:41.362266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792081,"num_objects":343,"num_object_clones":2,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":2097,"num_read_kb":1848,"num_write":1479,"num_write_kb":15130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2826,"ondisk_log_size":2826,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":801,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":617060,"kb_used_data":80660,"kb_used_omap":38,"kb_used_meta":536153,"kb_avail":749344156,"statfs":{"total":767960285184,"available":767328415744,"internally_reserved":0,"allocated":82595840,"data_stored":60673893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":549021109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":144,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.977618"},"pg_stats":[{"pgid":"274.1f","version":"0'0","reported_seq":25,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.393629+0000","last_change":"2024-03-11T21:33:33.401033+0000","last_active":"2024-03-11T21:33:38.393629+0000","last_peered":"2024-03-11T21:33:38.393629+0000","last_clean":"2024-03-11T21:33:38.393629+0000","last_became_active":"2024-03-11T21:33:33.400519+0000","last_became_peered":"2024-03-11T21:33:33.400519+0000","last_unstale":"2024-03-11T21:33:38.393629+0000","last_undegraded":"2024-03-11T21:33:38.393629+0000","last_fullsized":"2024-03-11T21:33:38.393629+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.737329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.1e","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388145+0000","last_change":"2024-03-11T21:33:33.388423+0000","last_active":"2024-03-11T21:33:37.388145+0000","last_peered":"2024-03-11T21:33:37.388145+0000","last_clean":"2024-03-11T21:33:37.388145+0000","last_became_active":"2024-03-11T21:33:33.388030+0000","last_became_peered":"2024-03-11T21:33:33.388030+0000","last_unstale":"2024-03-11T21:33:37.388145+0000","last_undegraded":"2024-03-11T21:33:37.388145+0000","last_fullsized":"2024-03-11T21:33:37.388145+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:36.461932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"274.1d","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399150+0000","last_change":"2024-03-11T21:33:33.391451+0000","last_active":"2024-03-11T21:33:37.399150+0000","last_peered":"2024-03-11T21:33:37.399150+0000","last_clean":"2024-03-11T21:33:37.399150+0000","last_became_active":"2024-03-11T21:33:33.390432+0000","last_became_peered":"2024-03-11T21:33:33.390432+0000","last_unstale":"2024-03-11T21:33:37.399150+0000","last_undegraded":"2024-03-11T21:33:37.399150+0000","last_fullsized":"2024-03-11T21:33:37.399150+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:46.392435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.1c","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387481+0000","last_change":"2024-03-11T21:33:33.388142+0000","last_active":"2024-03-11T21:33:37.387481+0000","last_peered":"2024-03-11T21:33:37.387481+0000","last_clean":"2024-03-11T21:33:37.387481+0000","last_became_active":"2024-03-11T21:33:33.387556+0000","last_became_peered":"2024-03-11T21:33:33.387556+0000","last_unstale":"2024-03-11T21:33:37.387481+0000","last_undegraded":"2024-03-11T21:33:37.387481+0000","last_fullsized":"2024-03-11T21:33:37.387481+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.996007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"274.1b","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.391773+0000","last_change":"2024-03-11T21:33:33.382074+0000","last_active":"2024-03-11T21:33:37.391773+0000","last_peered":"2024-03-11T21:33:37.391773+0000","last_clean":"2024-03-11T21:33:37.391773+0000","last_became_active":"2024-03-11T21:33:33.381166+0000","last_became_peered":"2024-03-11T21:33:33.381166+0000","last_unstale":"2024-03-11T21:33:37.391773+0000","last_undegraded":"2024-03-11T21:33:37.391773+0000","last_fullsized":"2024-03-11T21:33:37.391773+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.255332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.1a","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395584+0000","last_change":"2024-03-11T21:33:33.398013+0000","last_active":"2024-03-11T21:33:37.395584+0000","last_peered":"2024-03-11T21:33:37.395584+0000","last_clean":"2024-03-11T21:33:37.395584+0000","last_became_active":"2024-03-11T21:33:33.396943+0000","last_became_peered":"2024-03-11T21:33:33.396943+0000","last_unstale":"2024-03-11T21:33:37.395584+0000","last_undegraded":"2024-03-11T21:33:37.395584+0000","last_fullsized":"2024-03-11T21:33:37.395584+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.531041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.19","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.395393+0000","last_change":"2024-03-11T21:33:33.399465+0000","last_active":"2024-03-11T21:33:40.395393+0000","last_peered":"2024-03-11T21:33:40.395393+0000","last_clean":"2024-03-11T21:33:40.395393+0000","last_became_active":"2024-03-11T21:33:33.398182+0000","last_became_peered":"2024-03-11T21:33:33.398182+0000","last_unstale":"2024-03-11T21:33:40.395393+0000","last_undegraded":"2024-03-11T21:33:40.395393+0000","last_fullsized":"2024-03-11T21:33:40.395393+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:39.658176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.18","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382165+0000","last_change":"2024-03-11T21:33:33.391893+0000","last_active":"2024-03-11T21:33:37.382165+0000","last_peered":"2024-03-11T21:33:37.382165+0000","last_clean":"2024-03-11T21:33:37.382165+0000","last_became_active":"2024-03-11T21:33:33.391523+0000","last_became_peered":"2024-03-11T21:33:33.391523+0000","last_unstale":"2024-03-11T21:33:37.382165+0000","last_undegraded":"2024-03-11T21:33:37.382165+0000","last_fullsized":"2024-03-11T21:33:37.382165+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.822930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"274.17","version":"0'0","reported_seq":25,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.393830+0000","last_change":"2024-03-11T21:33:33.402227+0000","last_active":"2024-03-11T21:33:38.393830+0000","last_peered":"2024-03-11T21:33:38.393830+0000","last_clean":"2024-03-11T21:33:38.393830+0000","last_became_active":"2024-03-11T21:33:33.401714+0000","last_became_peered":"2024-03-11T21:33:33.401714+0000","last_unstale":"2024-03-11T21:33:38.393830+0000","last_undegraded":"2024-03-11T21:33:38.393830+0000","last_fullsized":"2024-03-11T21:33:38.393830+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.269730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.16","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.407723+0000","last_change":"2024-03-11T21:33:33.408410+0000","last_active":"2024-03-11T21:33:40.407723+0000","last_peered":"2024-03-11T21:33:40.407723+0000","last_clean":"2024-03-11T21:33:40.407723+0000","last_became_active":"2024-03-11T21:33:33.407807+0000","last_became_peered":"2024-03-11T21:33:33.407807+0000","last_unstale":"2024-03-11T21:33:40.407723+0000","last_undegraded":"2024-03-11T21:33:40.407723+0000","last_fullsized":"2024-03-11T21:33:40.407723+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.224545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.15","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396133+0000","last_change":"2024-03-11T21:33:33.363443+0000","last_active":"2024-03-11T21:33:37.396133+0000","last_peered":"2024-03-11T21:33:37.396133+0000","last_clean":"2024-03-11T21:33:37.396133+0000","last_became_active":"2024-03-11T21:33:33.363093+0000","last_became_peered":"2024-03-11T21:33:33.363093+0000","last_unstale":"2024-03-11T21:33:37.396133+0000","last_undegraded":"2024-03-11T21:33:37.396133+0000","last_fullsized":"2024-03-11T21:33:37.396133+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.833410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.14","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397443+0000","last_change":"2024-03-11T21:33:33.385380+0000","last_active":"2024-03-11T21:33:37.397443+0000","last_peered":"2024-03-11T21:33:37.397443+0000","last_clean":"2024-03-11T21:33:37.397443+0000","last_became_active":"2024-03-11T21:33:33.385014+0000","last_became_peered":"2024-03-11T21:33:33.385014+0000","last_unstale":"2024-03-11T21:33:37.397443+0000","last_undegraded":"2024-03-11T21:33:37.397443+0000","last_fullsized":"2024-03-11T21:33:37.397443+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.258309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.13","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.402723+0000","last_change":"2024-03-11T21:33:33.408074+0000","last_active":"2024-03-11T21:33:40.402723+0000","last_peered":"2024-03-11T21:33:40.402723+0000","last_clean":"2024-03-11T21:33:40.402723+0000","last_became_active":"2024-03-11T21:33:33.407596+0000","last_became_peered":"2024-03-11T21:33:33.407596+0000","last_unstale":"2024-03-11T21:33:40.402723+0000","last_undegraded":"2024-03-11T21:33:40.402723+0000","last_fullsized":"2024-03-11T21:33:40.402723+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.859756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.12","version":"0'0","reported_seq":25,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390270+0000","last_change":"2024-03-11T21:33:33.406896+0000","last_active":"2024-03-11T21:33:38.390270+0000","last_peered":"2024-03-11T21:33:38.390270+0000","last_clean":"2024-03-11T21:33:38.390270+0000","last_became_active":"2024-03-11T21:33:33.406211+0000","last_became_peered":"2024-03-11T21:33:33.406211+0000","last_unstale":"2024-03-11T21:33:38.390270+0000","last_undegraded":"2024-03-11T21:33:38.390270+0000","last_fullsized":"2024-03-11T21:33:38.390270+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.197211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.11","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.396451+0000","last_change":"2024-03-11T21:33:33.380906+0000","last_active":"2024-03-11T21:33:40.396451+0000","last_peered":"2024-03-11T21:33:40.396451+0000","last_clean":"2024-03-11T21:33:40.396451+0000","last_became_active":"2024-03-11T21:33:33.380300+0000","last_became_peered":"2024-03-11T21:33:33.380300+0000","last_unstale":"2024-03-11T21:33:40.396451+0000","last_undegraded":"2024-03-11T21:33:40.396451+0000","last_fullsized":"2024-03-11T21:33:40.396451+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:46.437542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.10","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.413791+0000","last_change":"2024-03-11T21:33:33.396803+0000","last_active":"2024-03-11T21:33:37.413791+0000","last_peered":"2024-03-11T21:33:37.413791+0000","last_clean":"2024-03-11T21:33:37.413791+0000","last_became_active":"2024-03-11T21:33:33.396289+0000","last_became_peered":"2024-03-11T21:33:33.396289+0000","last_unstale":"2024-03-11T21:33:37.413791+0000","last_undegraded":"2024-03-11T21:33:37.413791+0000","last_fullsized":"2024-03-11T21:33:37.413791+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.545943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.f","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383938+0000","last_change":"2024-03-11T21:33:33.380325+0000","last_active":"2024-03-11T21:33:37.383938+0000","last_peered":"2024-03-11T21:33:37.383938+0000","last_clean":"2024-03-11T21:33:37.383938+0000","last_became_active":"2024-03-11T21:33:33.380007+0000","last_became_peered":"2024-03-11T21:33:33.380007+0000","last_unstale":"2024-03-11T21:33:37.383938+0000","last_undegraded":"2024-03-11T21:33:37.383938+0000","last_fullsized":"2024-03-11T21:33:37.383938+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:58.152721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"274.e","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396725+0000","last_change":"2024-03-11T21:33:33.366336+0000","last_active":"2024-03-11T21:33:37.396725+0000","last_peered":"2024-03-11T21:33:37.396725+0000","last_clean":"2024-03-11T21:33:37.396725+0000","last_became_active":"2024-03-11T21:33:33.365955+0000","last_became_peered":"2024-03-11T21:33:33.365955+0000","last_unstale":"2024-03-11T21:33:37.396725+0000","last_undegraded":"2024-03-11T21:33:37.396725+0000","last_fullsized":"2024-03-11T21:33:37.396725+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.328767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.d","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417351+0000","last_change":"2024-03-11T21:33:33.389455+0000","last_active":"2024-03-11T21:33:37.417351+0000","last_peered":"2024-03-11T21:33:37.417351+0000","last_clean":"2024-03-11T21:33:37.417351+0000","last_became_active":"2024-03-11T21:33:33.388722+0000","last_became_peered":"2024-03-11T21:33:33.388722+0000","last_unstale":"2024-03-11T21:33:37.417351+0000","last_undegraded":"2024-03-11T21:33:37.417351+0000","last_fullsized":"2024-03-11T21:33:37.417351+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:42.250416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.c","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383020+0000","last_change":"2024-03-11T21:33:33.381293+0000","last_active":"2024-03-11T21:33:37.383020+0000","last_peered":"2024-03-11T21:33:37.383020+0000","last_clean":"2024-03-11T21:33:37.383020+0000","last_became_active":"2024-03-11T21:33:33.381028+0000","last_became_peered":"2024-03-11T21:33:33.381028+0000","last_unstale":"2024-03-11T21:33:37.383020+0000","last_undegraded":"2024-03-11T21:33:37.383020+0000","last_fullsized":"2024-03-11T21:33:37.383020+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:42.795956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"274.b","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.403651+0000","last_change":"2024-03-11T21:33:33.399296+0000","last_active":"2024-03-11T21:33:40.403651+0000","last_peered":"2024-03-11T21:33:40.403651+0000","last_clean":"2024-03-11T21:33:40.403651+0000","last_became_active":"2024-03-11T21:33:33.398735+0000","last_became_peered":"2024-03-11T21:33:33.398735+0000","last_unstale":"2024-03-11T21:33:40.403651+0000","last_undegraded":"2024-03-11T21:33:40.403651+0000","last_fullsized":"2024-03-11T21:33:40.403651+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:52.467876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.a","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395829+0000","last_change":"2024-03-11T21:33:33.398395+0000","last_active":"2024-03-11T21:33:37.395829+0000","last_peered":"2024-03-11T21:33:37.395829+0000","last_clean":"2024-03-11T21:33:37.395829+0000","last_became_active":"2024-03-11T21:33:33.397545+0000","last_became_peered":"2024-03-11T21:33:33.397545+0000","last_unstale":"2024-03-11T21:33:37.395829+0000","last_undegraded":"2024-03-11T21:33:37.395829+0000","last_fullsized":"2024-03-11T21:33:37.395829+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.631519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.9","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394325+0000","last_change":"2024-03-11T21:33:33.390790+0000","last_active":"2024-03-11T21:33:37.394325+0000","last_peered":"2024-03-11T21:33:37.394325+0000","last_clean":"2024-03-11T21:33:37.394325+0000","last_became_active":"2024-03-11T21:33:33.390451+0000","last_became_peered":"2024-03-11T21:33:33.390451+0000","last_unstale":"2024-03-11T21:33:37.394325+0000","last_undegraded":"2024-03-11T21:33:37.394325+0000","last_fullsized":"2024-03-11T21:33:37.394325+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:51.403621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"274.8","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.395025+0000","last_change":"2024-03-11T21:33:33.382506+0000","last_active":"2024-03-11T21:33:40.395025+0000","last_peered":"2024-03-11T21:33:40.395025+0000","last_clean":"2024-03-11T21:33:40.395025+0000","last_became_active":"2024-03-11T21:33:33.381958+0000","last_became_peered":"2024-03-11T21:33:33.381958+0000","last_unstale":"2024-03-11T21:33:40.395025+0000","last_undegraded":"2024-03-11T21:33:40.395025+0000","last_fullsized":"2024-03-11T21:33:40.395025+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.818307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.1","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.396890+0000","last_change":"2024-03-11T21:33:33.387081+0000","last_active":"2024-03-11T21:33:40.396890+0000","last_peered":"2024-03-11T21:33:40.396890+0000","last_clean":"2024-03-11T21:33:40.396890+0000","last_became_active":"2024-03-11T21:33:33.386444+0000","last_became_peered":"2024-03-11T21:33:33.386444+0000","last_unstale":"2024-03-11T21:33:40.396890+0000","last_undegraded":"2024-03-11T21:33:40.396890+0000","last_fullsized":"2024-03-11T21:33:40.396890+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:57.094412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.0","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382050+0000","last_change":"2024-03-11T21:33:33.378244+0000","last_active":"2024-03-11T21:33:37.382050+0000","last_peered":"2024-03-11T21:33:37.382050+0000","last_clean":"2024-03-11T21:33:37.382050+0000","last_became_active":"2024-03-11T21:33:33.377905+0000","last_became_peered":"2024-03-11T21:33:33.377905+0000","last_unstale":"2024-03-11T21:33:37.382050+0000","last_undegraded":"2024-03-11T21:33:37.382050+0000","last_fullsized":"2024-03-11T21:33:37.382050+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:58.300967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"274.3","version":"0'0","reported_seq":25,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390533+0000","last_change":"2024-03-11T21:33:33.393146+0000","last_active":"2024-03-11T21:33:38.390533+0000","last_peered":"2024-03-11T21:33:38.390533+0000","last_clean":"2024-03-11T21:33:38.390533+0000","last_became_active":"2024-03-11T21:33:33.389933+0000","last_became_peered":"2024-03-11T21:33:33.389933+0000","last_unstale":"2024-03-11T21:33:38.390533+0000","last_undegraded":"2024-03-11T21:33:38.390533+0000","last_fullsized":"2024-03-11T21:33:38.390533+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:49.394143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"274.2","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.403143+0000","last_change":"2024-03-11T21:33:33.406673+0000","last_active":"2024-03-11T21:33:40.403143+0000","last_peered":"2024-03-11T21:33:40.403143+0000","last_clean":"2024-03-11T21:33:40.403143+0000","last_became_active":"2024-03-11T21:33:33.403861+0000","last_became_peered":"2024-03-11T21:33:33.403861+0000","last_unstale":"2024-03-11T21:33:40.403143+0000","last_undegraded":"2024-03-11T21:33:40.403143+0000","last_fullsized":"2024-03-11T21:33:40.403143+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.439289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"274.5","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397687+0000","last_change":"2024-03-11T21:33:33.389976+0000","last_active":"2024-03-11T21:33:37.397687+0000","last_peered":"2024-03-11T21:33:37.397687+0000","last_clean":"2024-03-11T21:33:37.397687+0000","last_became_active":"2024-03-11T21:33:33.389394+0000","last_became_peered":"2024-03-11T21:33:33.389394+0000","last_unstale":"2024-03-11T21:33:37.397687+0000","last_undegraded":"2024-03-11T21:33:37.397687+0000","last_fullsized":"2024-03-11T21:33:37.397687+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.835189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"274.4","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394221+0000","last_change":"2024-03-11T21:33:33.393400+0000","last_active":"2024-03-11T21:33:37.394221+0000","last_peered":"2024-03-11T21:33:37.394221+0000","last_clean":"2024-03-11T21:33:37.394221+0000","last_became_active":"2024-03-11T21:33:33.392566+0000","last_became_peered":"2024-03-11T21:33:33.392566+0000","last_unstale":"2024-03-11T21:33:37.394221+0000","last_undegraded":"2024-03-11T21:33:37.394221+0000","last_fullsized":"2024-03-11T21:33:37.394221+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:56.791416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"274.7","version":"0'0","reported_seq":22,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420877+0000","last_change":"2024-03-11T21:33:33.387420+0000","last_active":"2024-03-11T21:33:37.420877+0000","last_peered":"2024-03-11T21:33:37.420877+0000","last_clean":"2024-03-11T21:33:37.420877+0000","last_became_active":"2024-03-11T21:33:33.386743+0000","last_became_peered":"2024-03-11T21:33:33.386743+0000","last_unstale":"2024-03-11T21:33:37.420877+0000","last_undegraded":"2024-03-11T21:33:37.420877+0000","last_fullsized":"2024-03-11T21:33:37.420877+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.967519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"274.6","version":"0'0","reported_seq":31,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405605+0000","last_change":"2024-03-11T21:33:33.408543+0000","last_active":"2024-03-11T21:33:40.405605+0000","last_peered":"2024-03-11T21:33:40.405605+0000","last_clean":"2024-03-11T21:33:40.405605+0000","last_became_active":"2024-03-11T21:33:33.407767+0000","last_became_peered":"2024-03-11T21:33:33.407767+0000","last_unstale":"2024-03-11T21:33:40.405605+0000","last_undegraded":"2024-03-11T21:33:40.405605+0000","last_fullsized":"2024-03-11T21:33:40.405605+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:32.334187+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.334187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.973559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.9","version":"245'17","reported_seq":677,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.391032+0000","last_change":"2024-03-11T21:32:34.298539+0000","last_active":"2024-03-11T21:33:37.391032+0000","last_peered":"2024-03-11T21:33:37.391032+0000","last_clean":"2024-03-11T21:33:37.391032+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:33:37.391032+0000","last_undegraded":"2024-03-11T21:33:37.391032+0000","last_fullsized":"2024-03-11T21:33:37.391032+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-03-11T21:32:02.895303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.895303+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001660573,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"412'20","reported_seq":671,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382721+0000","last_change":"2024-03-11T21:32:34.299549+0000","last_active":"2024-03-11T21:33:37.382721+0000","last_peered":"2024-03-11T21:33:37.382721+0000","last_clean":"2024-03-11T21:33:37.382721+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:33:37.382721+0000","last_undegraded":"2024-03-11T21:33:37.382721+0000","last_fullsized":"2024-03-11T21:33:37.382721+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'18","last_scrub_stamp":"2024-03-11T21:32:22.035738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.035738+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076326579999999996,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"296'34","reported_seq":682,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397115+0000","last_change":"2024-03-11T21:32:34.335857+0000","last_active":"2024-03-11T21:33:37.397115+0000","last_peered":"2024-03-11T21:33:37.397115+0000","last_clean":"2024-03-11T21:33:37.397115+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:33:37.397115+0000","last_undegraded":"2024-03-11T21:33:37.397115+0000","last_fullsized":"2024-03-11T21:33:37.397115+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024159609999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"412'54","reported_seq":716,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386307+0000","last_change":"2024-03-11T21:33:20.623614+0000","last_active":"2024-03-11T21:33:37.386307+0000","last_peered":"2024-03-11T21:33:37.386307+0000","last_clean":"2024-03-11T21:33:37.386307+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:33:37.386307+0000","last_undegraded":"2024-03-11T21:33:37.386307+0000","last_fullsized":"2024-03-11T21:33:37.386307+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'52","last_scrub_stamp":"2024-03-11T21:33:20.623315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.623315+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:46.393384+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0043833079999999998,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"412'52","reported_seq":712,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394288+0000","last_change":"2024-03-11T21:33:28.221239+0000","last_active":"2024-03-11T21:33:37.394288+0000","last_peered":"2024-03-11T21:33:37.394288+0000","last_clean":"2024-03-11T21:33:37.394288+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:33:37.394288+0000","last_undegraded":"2024-03-11T21:33:37.394288+0000","last_fullsized":"2024-03-11T21:33:37.394288+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:33:28.221082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.221082+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.733926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022023799999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":681,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387716+0000","last_change":"2024-03-11T21:33:15.710053+0000","last_active":"2024-03-11T21:33:37.387716+0000","last_peered":"2024-03-11T21:33:37.387716+0000","last_clean":"2024-03-11T21:33:37.387716+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:33:37.387716+0000","last_undegraded":"2024-03-11T21:33:37.387716+0000","last_fullsized":"2024-03-11T21:33:37.387716+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'34","last_scrub_stamp":"2024-03-11T21:33:15.709878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.709878+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.662560+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0036499739999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"353'183","reported_seq":807,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388011+0000","last_change":"2024-03-11T21:33:14.703400+0000","last_active":"2024-03-11T21:33:37.388011+0000","last_peered":"2024-03-11T21:33:37.388011+0000","last_clean":"2024-03-11T21:33:37.388011+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:33:37.388011+0000","last_undegraded":"2024-03-11T21:33:37.388011+0000","last_fullsized":"2024-03-11T21:33:37.388011+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'183","last_scrub_stamp":"2024-03-11T21:33:14.703284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.703284+0000","objects_scrubbed":13,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.288266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.029727594,"stat_sum":{"num_bytes":117,"num_objects":13,"num_object_clones":1,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":55,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":750,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383623+0000","last_change":"2024-03-11T21:33:27.586701+0000","last_active":"2024-03-11T21:33:37.383623+0000","last_peered":"2024-03-11T21:33:37.383623+0000","last_clean":"2024-03-11T21:33:37.383623+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:33:37.383623+0000","last_undegraded":"2024-03-11T21:33:37.383623+0000","last_fullsized":"2024-03-11T21:33:37.383623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:33:27.586328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.586328+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.293749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.013258202,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":672,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389241+0000","last_change":"2024-03-11T21:33:08.891202+0000","last_active":"2024-03-11T21:33:38.389241+0000","last_peered":"2024-03-11T21:33:38.389241+0000","last_clean":"2024-03-11T21:33:38.389241+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:33:38.389241+0000","last_undegraded":"2024-03-11T21:33:38.389241+0000","last_fullsized":"2024-03-11T21:33:38.389241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:33:08.891062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.891062+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:24.334197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0077514410000000004,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":711,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389027+0000","last_change":"2024-03-11T21:33:14.964478+0000","last_active":"2024-03-11T21:33:38.389027+0000","last_peered":"2024-03-11T21:33:38.389027+0000","last_clean":"2024-03-11T21:33:38.389027+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:33:38.389027+0000","last_undegraded":"2024-03-11T21:33:38.389027+0000","last_fullsized":"2024-03-11T21:33:38.389027+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:33:14.964352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.964352+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:36.933637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076282429999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"412'120","reported_seq":743,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397247+0000","last_change":"2024-03-11T21:32:34.336215+0000","last_active":"2024-03-11T21:33:37.397247+0000","last_peered":"2024-03-11T21:33:37.397247+0000","last_clean":"2024-03-11T21:33:37.397247+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:33:37.397247+0000","last_undegraded":"2024-03-11T21:33:37.397247+0000","last_fullsized":"2024-03-11T21:33:37.397247+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026596910000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":690,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386856+0000","last_change":"2024-03-11T21:33:17.671892+0000","last_active":"2024-03-11T21:33:37.386856+0000","last_peered":"2024-03-11T21:33:37.386856+0000","last_clean":"2024-03-11T21:33:37.386856+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:33:37.386856+0000","last_undegraded":"2024-03-11T21:33:37.386856+0000","last_fullsized":"2024-03-11T21:33:37.386856+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'21","last_scrub_stamp":"2024-03-11T21:33:17.671661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.671661+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.480112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0041327550000000001,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"0'0","reported_seq":659,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394793+0000","last_change":"2024-03-11T21:33:32.236109+0000","last_active":"2024-03-11T21:33:37.394793+0000","last_peered":"2024-03-11T21:33:37.394793+0000","last_clean":"2024-03-11T21:33:37.394793+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:33:37.394793+0000","last_undegraded":"2024-03-11T21:33:37.394793+0000","last_fullsized":"2024-03-11T21:33:37.394793+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.235772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.235772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.808379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002624487,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":689,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390792+0000","last_change":"2024-03-11T21:33:07.935033+0000","last_active":"2024-03-11T21:33:38.390792+0000","last_peered":"2024-03-11T21:33:38.390792+0000","last_clean":"2024-03-11T21:33:38.390792+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:33:38.390792+0000","last_undegraded":"2024-03-11T21:33:38.390792+0000","last_fullsized":"2024-03-11T21:33:38.390792+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:07.934653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.934653+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.483763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0080892789999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"412'41","reported_seq":685,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420843+0000","last_change":"2024-03-11T21:33:25.692636+0000","last_active":"2024-03-11T21:33:37.420843+0000","last_peered":"2024-03-11T21:33:37.420843+0000","last_clean":"2024-03-11T21:33:37.420843+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:33:37.420843+0000","last_undegraded":"2024-03-11T21:33:37.420843+0000","last_fullsized":"2024-03-11T21:33:37.420843+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'39","last_scrub_stamp":"2024-03-11T21:33:25.692508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.692508+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.803905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001055535,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":727,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.414170+0000","last_change":"2024-03-11T21:33:23.654493+0000","last_active":"2024-03-11T21:33:37.414170+0000","last_peered":"2024-03-11T21:33:37.414170+0000","last_clean":"2024-03-11T21:33:37.414170+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:33:37.414170+0000","last_undegraded":"2024-03-11T21:33:37.414170+0000","last_fullsized":"2024-03-11T21:33:37.414170+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:33:23.654355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.654355+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.399409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001789856,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":669,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389102+0000","last_change":"2024-03-11T21:33:05.928221+0000","last_active":"2024-03-11T21:33:38.389102+0000","last_peered":"2024-03-11T21:33:38.389102+0000","last_clean":"2024-03-11T21:33:38.389102+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:33:38.389102+0000","last_undegraded":"2024-03-11T21:33:38.389102+0000","last_fullsized":"2024-03-11T21:33:38.389102+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:33:05.927971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.927971+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:09.976095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0078384049999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":741,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.388978+0000","last_change":"2024-03-11T21:33:10.952092+0000","last_active":"2024-03-11T21:33:38.388978+0000","last_peered":"2024-03-11T21:33:38.388978+0000","last_clean":"2024-03-11T21:33:38.388978+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:33:38.388978+0000","last_undegraded":"2024-03-11T21:33:38.388978+0000","last_fullsized":"2024-03-11T21:33:38.388978+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'107","last_scrub_stamp":"2024-03-11T21:33:10.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.951942+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.958654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067123959999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":559,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.403885+0000","last_change":"2024-03-11T21:33:17.964447+0000","last_active":"2024-03-11T21:33:40.403885+0000","last_peered":"2024-03-11T21:33:40.403885+0000","last_clean":"2024-03-11T21:33:40.403885+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:33:40.403885+0000","last_undegraded":"2024-03-11T21:33:40.403885+0000","last_fullsized":"2024-03-11T21:33:40.403885+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:33:17.964235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.964235+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:44.293228+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.005553105,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":705,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390379+0000","last_change":"2024-03-11T21:33:04.948889+0000","last_active":"2024-03-11T21:33:38.390379+0000","last_peered":"2024-03-11T21:33:38.390379+0000","last_clean":"2024-03-11T21:33:38.390379+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:33:38.390379+0000","last_undegraded":"2024-03-11T21:33:38.390379+0000","last_fullsized":"2024-03-11T21:33:38.390379+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:33:04.948734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.948734+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:12.668946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0071306190000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":735,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396223+0000","last_change":"2024-03-11T21:32:34.326566+0000","last_active":"2024-03-11T21:33:37.396223+0000","last_peered":"2024-03-11T21:33:37.396223+0000","last_clean":"2024-03-11T21:33:37.396223+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:33:37.396223+0000","last_undegraded":"2024-03-11T21:33:37.396223+0000","last_fullsized":"2024-03-11T21:33:37.396223+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:32:07.554683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.554683+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096827370000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"412'50","reported_seq":708,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397617+0000","last_change":"2024-03-11T21:32:34.335870+0000","last_active":"2024-03-11T21:33:37.397617+0000","last_peered":"2024-03-11T21:33:37.397617+0000","last_clean":"2024-03-11T21:33:37.397617+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:33:37.397617+0000","last_undegraded":"2024-03-11T21:33:37.397617+0000","last_fullsized":"2024-03-11T21:33:37.397617+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023272950000000001,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"412'35","reported_seq":573,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405697+0000","last_change":"2024-03-11T21:33:14.938757+0000","last_active":"2024-03-11T21:33:40.405697+0000","last_peered":"2024-03-11T21:33:40.405697+0000","last_clean":"2024-03-11T21:33:40.405697+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:33:40.405697+0000","last_undegraded":"2024-03-11T21:33:40.405697+0000","last_fullsized":"2024-03-11T21:33:40.405697+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:14.938568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.938568+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:14.956180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011782635,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"361'209","reported_seq":814,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397723+0000","last_change":"2024-03-11T21:32:34.338193+0000","last_active":"2024-03-11T21:33:37.397723+0000","last_peered":"2024-03-11T21:33:37.397723+0000","last_clean":"2024-03-11T21:33:37.397723+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:33:37.397723+0000","last_undegraded":"2024-03-11T21:33:37.397723+0000","last_fullsized":"2024-03-11T21:33:37.397723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829950000000001,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":674,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396012+0000","last_change":"2024-03-11T21:32:34.335400+0000","last_active":"2024-03-11T21:33:37.396012+0000","last_peered":"2024-03-11T21:33:37.396012+0000","last_clean":"2024-03-11T21:33:37.396012+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:33:37.396012+0000","last_undegraded":"2024-03-11T21:33:37.396012+0000","last_fullsized":"2024-03-11T21:33:37.396012+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024394040000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"412'15","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382207+0000","last_change":"2024-03-11T21:33:31.565426+0000","last_active":"2024-03-11T21:33:37.382207+0000","last_peered":"2024-03-11T21:33:37.382207+0000","last_clean":"2024-03-11T21:33:37.382207+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:33:37.382207+0000","last_undegraded":"2024-03-11T21:33:37.382207+0000","last_fullsized":"2024-03-11T21:33:37.382207+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'14","last_scrub_stamp":"2024-03-11T21:33:31.565199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.565199+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:36.243407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0069698629999999998,"stat_sum":{"num_bytes":4194320,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"412'87","reported_seq":722,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382868+0000","last_change":"2024-03-11T21:32:34.312524+0000","last_active":"2024-03-11T21:33:37.382868+0000","last_peered":"2024-03-11T21:33:37.382868+0000","last_clean":"2024-03-11T21:33:37.382868+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:33:37.382868+0000","last_undegraded":"2024-03-11T21:33:37.382868+0000","last_fullsized":"2024-03-11T21:33:37.382868+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'85","last_scrub_stamp":"2024-03-11T21:32:14.839185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.839185+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:38.170857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018288136999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":706,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396129+0000","last_change":"2024-03-11T21:32:34.332276+0000","last_active":"2024-03-11T21:33:37.396129+0000","last_peered":"2024-03-11T21:33:37.396129+0000","last_clean":"2024-03-11T21:33:37.396129+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:33:37.396129+0000","last_undegraded":"2024-03-11T21:33:37.396129+0000","last_fullsized":"2024-03-11T21:33:37.396129+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016228219999999999,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394027+0000","last_change":"2024-03-11T21:33:29.221706+0000","last_active":"2024-03-11T21:33:37.394027+0000","last_peered":"2024-03-11T21:33:37.394027+0000","last_clean":"2024-03-11T21:33:37.394027+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:33:37.394027+0000","last_undegraded":"2024-03-11T21:33:37.394027+0000","last_fullsized":"2024-03-11T21:33:37.394027+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:33:29.221527+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.221527+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.874926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029485240000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":707,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387559+0000","last_change":"2024-03-11T21:33:25.682885+0000","last_active":"2024-03-11T21:33:37.387559+0000","last_peered":"2024-03-11T21:33:37.387559+0000","last_clean":"2024-03-11T21:33:37.387559+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:33:37.387559+0000","last_undegraded":"2024-03-11T21:33:37.387559+0000","last_fullsized":"2024-03-11T21:33:37.387559+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'51","last_scrub_stamp":"2024-03-11T21:33:25.682768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.682768+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.523241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001496713,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"412'53","reported_seq":703,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.400146+0000","last_change":"2024-03-11T21:32:34.308970+0000","last_active":"2024-03-11T21:33:37.400146+0000","last_peered":"2024-03-11T21:33:37.400146+0000","last_clean":"2024-03-11T21:33:37.400146+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:33:37.400146+0000","last_undegraded":"2024-03-11T21:33:37.400146+0000","last_fullsized":"2024-03-11T21:33:37.400146+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:32:01.854464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.854464+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001987904,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"412'3","reported_seq":657,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397998+0000","last_change":"2024-03-11T21:32:34.338218+0000","last_active":"2024-03-11T21:33:37.397998+0000","last_peered":"2024-03-11T21:33:37.397998+0000","last_clean":"2024-03-11T21:33:37.397998+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:33:37.397998+0000","last_undegraded":"2024-03-11T21:33:37.397998+0000","last_fullsized":"2024-03-11T21:33:37.397998+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011248230000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":570,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.412448+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:33:37.412448+0000","last_peered":"2024-03-11T21:33:37.412448+0000","last_clean":"2024-03-11T21:33:37.412448+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:33:37.412448+0000","last_undegraded":"2024-03-11T21:33:37.412448+0000","last_fullsized":"2024-03-11T21:33:37.412448+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.119355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":572,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394424+0000","last_change":"2024-03-11T21:33:28.579904+0000","last_active":"2024-03-11T21:33:37.394424+0000","last_peered":"2024-03-11T21:33:37.394424+0000","last_clean":"2024-03-11T21:33:37.394424+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:33:37.394424+0000","last_undegraded":"2024-03-11T21:33:37.394424+0000","last_fullsized":"2024-03-11T21:33:37.394424+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.577678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:46.193354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":571,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382406+0000","last_change":"2024-03-11T21:33:25.553124+0000","last_active":"2024-03-11T21:33:37.382406+0000","last_peered":"2024-03-11T21:33:37.382406+0000","last_clean":"2024-03-11T21:33:37.382406+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:33:37.382406+0000","last_undegraded":"2024-03-11T21:33:37.382406+0000","last_fullsized":"2024-03-11T21:33:37.382406+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.552965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.552965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.705964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":561,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417531+0000","last_change":"2024-03-11T21:33:13.735432+0000","last_active":"2024-03-11T21:33:37.417531+0000","last_peered":"2024-03-11T21:33:37.417531+0000","last_clean":"2024-03-11T21:33:37.417531+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:33:37.417531+0000","last_undegraded":"2024-03-11T21:33:37.417531+0000","last_fullsized":"2024-03-11T21:33:37.417531+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.735317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.735317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.675782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":476,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.403625+0000","last_change":"2024-03-11T21:33:34.103121+0000","last_active":"2024-03-11T21:33:40.403625+0000","last_peered":"2024-03-11T21:33:40.403625+0000","last_clean":"2024-03-11T21:33:40.403625+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:33:40.403625+0000","last_undegraded":"2024-03-11T21:33:40.403625+0000","last_fullsized":"2024-03-11T21:33:40.403625+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:34.103006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:34.103006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.132610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396621+0000","last_change":"2024-03-11T21:30:58.478329+0000","last_active":"2024-03-11T21:33:37.396621+0000","last_peered":"2024-03-11T21:33:37.396621+0000","last_clean":"2024-03-11T21:33:37.396621+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:33:37.396621+0000","last_undegraded":"2024-03-11T21:33:37.396621+0000","last_fullsized":"2024-03-11T21:33:37.396621+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.478183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.478183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":573,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396880+0000","last_change":"2024-03-11T21:33:25.576354+0000","last_active":"2024-03-11T21:33:37.396880+0000","last_peered":"2024-03-11T21:33:37.396880+0000","last_clean":"2024-03-11T21:33:37.396880+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:33:37.396880+0000","last_undegraded":"2024-03-11T21:33:37.396880+0000","last_fullsized":"2024-03-11T21:33:37.396880+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.576253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.576253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:27.330979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":573,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.400511+0000","last_change":"2024-03-11T21:32:15.794130+0000","last_active":"2024-03-11T21:33:37.400511+0000","last_peered":"2024-03-11T21:33:37.400511+0000","last_clean":"2024-03-11T21:33:37.400511+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:33:37.400511+0000","last_undegraded":"2024-03-11T21:33:37.400511+0000","last_fullsized":"2024-03-11T21:33:37.400511+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.793910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.793910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":566,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386869+0000","last_change":"2024-03-11T21:30:54.963682+0000","last_active":"2024-03-11T21:33:37.386869+0000","last_peered":"2024-03-11T21:33:37.386869+0000","last_clean":"2024-03-11T21:33:37.386869+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:33:37.386869+0000","last_undegraded":"2024-03-11T21:33:37.386869+0000","last_fullsized":"2024-03-11T21:33:37.386869+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.963569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.963569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":562,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.413827+0000","last_change":"2024-03-11T21:31:01.656496+0000","last_active":"2024-03-11T21:33:37.413827+0000","last_peered":"2024-03-11T21:33:37.413827+0000","last_clean":"2024-03-11T21:33:37.413827+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:33:37.413827+0000","last_undegraded":"2024-03-11T21:33:37.413827+0000","last_fullsized":"2024-03-11T21:33:37.413827+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.656422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.656422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":567,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390730+0000","last_change":"2024-03-11T21:30:51.770965+0000","last_active":"2024-03-11T21:33:38.390730+0000","last_peered":"2024-03-11T21:33:38.390730+0000","last_clean":"2024-03-11T21:33:38.390730+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:33:38.390730+0000","last_undegraded":"2024-03-11T21:33:38.390730+0000","last_fullsized":"2024-03-11T21:33:38.390730+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.770705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.770705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399754+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:33:37.399754+0000","last_peered":"2024-03-11T21:33:37.399754+0000","last_clean":"2024-03-11T21:33:37.399754+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:33:37.399754+0000","last_undegraded":"2024-03-11T21:33:37.399754+0000","last_fullsized":"2024-03-11T21:33:37.399754+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":567,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390675+0000","last_change":"2024-03-11T21:31:02.849078+0000","last_active":"2024-03-11T21:33:38.390675+0000","last_peered":"2024-03-11T21:33:38.390675+0000","last_clean":"2024-03-11T21:33:38.390675+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:33:38.390675+0000","last_undegraded":"2024-03-11T21:33:38.390675+0000","last_fullsized":"2024-03-11T21:33:38.390675+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.848978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.848978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":565,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388305+0000","last_change":"2024-03-11T21:33:10.604343+0000","last_active":"2024-03-11T21:33:37.388305+0000","last_peered":"2024-03-11T21:33:37.388305+0000","last_clean":"2024-03-11T21:33:37.388305+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:33:37.388305+0000","last_undegraded":"2024-03-11T21:33:37.388305+0000","last_fullsized":"2024-03-11T21:33:37.388305+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.604160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.604160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.058178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":572,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.401033+0000","last_change":"2024-03-11T21:33:27.227939+0000","last_active":"2024-03-11T21:33:37.401033+0000","last_peered":"2024-03-11T21:33:37.401033+0000","last_clean":"2024-03-11T21:33:37.401033+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:33:37.401033+0000","last_undegraded":"2024-03-11T21:33:37.401033+0000","last_fullsized":"2024-03-11T21:33:37.401033+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:27.227845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.227845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.639218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":561,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421238+0000","last_change":"2024-03-11T21:32:02.604095+0000","last_active":"2024-03-11T21:33:37.421238+0000","last_peered":"2024-03-11T21:33:37.421238+0000","last_clean":"2024-03-11T21:33:37.421238+0000","last_became_active":"2024-03-11T21:28:57.548665+0000","last_became_peered":"2024-03-11T21:28:57.548665+0000","last_unstale":"2024-03-11T21:33:37.421238+0000","last_undegraded":"2024-03-11T21:33:37.421238+0000","last_fullsized":"2024-03-11T21:33:37.421238+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.603946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.603946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":573,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395069+0000","last_change":"2024-03-11T21:33:15.600135+0000","last_active":"2024-03-11T21:33:37.395069+0000","last_peered":"2024-03-11T21:33:37.395069+0000","last_clean":"2024-03-11T21:33:37.395069+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:33:37.395069+0000","last_undegraded":"2024-03-11T21:33:37.395069+0000","last_fullsized":"2024-03-11T21:33:37.395069+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:15.600023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.600023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.478828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395641+0000","last_change":"2024-03-11T21:32:04.563626+0000","last_active":"2024-03-11T21:33:37.395641+0000","last_peered":"2024-03-11T21:33:37.395641+0000","last_clean":"2024-03-11T21:33:37.395641+0000","last_became_active":"2024-03-11T21:28:53.852502+0000","last_became_peered":"2024-03-11T21:28:53.852502+0000","last_unstale":"2024-03-11T21:33:37.395641+0000","last_undegraded":"2024-03-11T21:33:37.395641+0000","last_fullsized":"2024-03-11T21:33:37.395641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":562,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.416516+0000","last_change":"2024-03-11T21:33:12.695054+0000","last_active":"2024-03-11T21:33:37.416516+0000","last_peered":"2024-03-11T21:33:37.416516+0000","last_clean":"2024-03-11T21:33:37.416516+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:33:37.416516+0000","last_undegraded":"2024-03-11T21:33:37.416516+0000","last_fullsized":"2024-03-11T21:33:37.416516+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.694966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.694966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:36.431830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":566,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387571+0000","last_change":"2024-03-11T21:32:14.693680+0000","last_active":"2024-03-11T21:33:37.387571+0000","last_peered":"2024-03-11T21:33:37.387571+0000","last_clean":"2024-03-11T21:33:37.387571+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:33:37.387571+0000","last_undegraded":"2024-03-11T21:33:37.387571+0000","last_fullsized":"2024-03-11T21:33:37.387571+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.693586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.693586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":444,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.404176+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:33:40.404176+0000","last_peered":"2024-03-11T21:33:40.404176+0000","last_clean":"2024-03-11T21:33:40.404176+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:33:40.404176+0000","last_undegraded":"2024-03-11T21:33:40.404176+0000","last_fullsized":"2024-03-11T21:33:40.404176+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.546676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":562,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417759+0000","last_change":"2024-03-11T21:32:07.649024+0000","last_active":"2024-03-11T21:33:37.417759+0000","last_peered":"2024-03-11T21:33:37.417759+0000","last_clean":"2024-03-11T21:33:37.417759+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:33:37.417759+0000","last_undegraded":"2024-03-11T21:33:37.417759+0000","last_fullsized":"2024-03-11T21:33:37.417759+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.648800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.648800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":562,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417692+0000","last_change":"2024-03-11T21:32:15.722549+0000","last_active":"2024-03-11T21:33:37.417692+0000","last_peered":"2024-03-11T21:33:37.417692+0000","last_clean":"2024-03-11T21:33:37.417692+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:33:37.417692+0000","last_undegraded":"2024-03-11T21:33:37.417692+0000","last_fullsized":"2024-03-11T21:33:37.417692+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.722443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.722443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":567,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.393950+0000","last_change":"2024-03-11T21:32:14.976068+0000","last_active":"2024-03-11T21:33:38.393950+0000","last_peered":"2024-03-11T21:33:38.393950+0000","last_clean":"2024-03-11T21:33:38.393950+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:33:38.393950+0000","last_undegraded":"2024-03-11T21:33:38.393950+0000","last_fullsized":"2024-03-11T21:33:38.393950+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.975966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.975966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:40.129592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382739+0000","last_change":"2024-03-11T21:30:56.975013+0000","last_active":"2024-03-11T21:33:37.382739+0000","last_peered":"2024-03-11T21:33:37.382739+0000","last_clean":"2024-03-11T21:33:37.382739+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:33:37.382739+0000","last_undegraded":"2024-03-11T21:33:37.382739+0000","last_fullsized":"2024-03-11T21:33:37.382739+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:56.974794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.974794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395811+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:33:37.395811+0000","last_peered":"2024-03-11T21:33:37.395811+0000","last_clean":"2024-03-11T21:33:37.395811+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:33:37.395811+0000","last_undegraded":"2024-03-11T21:33:37.395811+0000","last_fullsized":"2024-03-11T21:33:37.395811+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.532268+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":562,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.416553+0000","last_change":"2024-03-11T21:30:55.794089+0000","last_active":"2024-03-11T21:33:37.416553+0000","last_peered":"2024-03-11T21:33:37.416553+0000","last_clean":"2024-03-11T21:33:37.416553+0000","last_became_active":"2024-03-11T21:28:57.523449+0000","last_became_peered":"2024-03-11T21:28:57.523449+0000","last_unstale":"2024-03-11T21:33:37.416553+0000","last_undegraded":"2024-03-11T21:33:37.416553+0000","last_fullsized":"2024-03-11T21:33:37.416553+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.793796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.793796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":566,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387433+0000","last_change":"2024-03-11T21:33:08.584273+0000","last_active":"2024-03-11T21:33:37.387433+0000","last_peered":"2024-03-11T21:33:37.387433+0000","last_clean":"2024-03-11T21:33:37.387433+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:33:37.387433+0000","last_undegraded":"2024-03-11T21:33:37.387433+0000","last_fullsized":"2024-03-11T21:33:37.387433+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.584163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.584163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.416698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":567,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.390907+0000","last_change":"2024-03-11T21:30:59.897121+0000","last_active":"2024-03-11T21:33:38.390907+0000","last_peered":"2024-03-11T21:33:38.390907+0000","last_clean":"2024-03-11T21:33:38.390907+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:33:38.390907+0000","last_undegraded":"2024-03-11T21:33:38.390907+0000","last_fullsized":"2024-03-11T21:33:38.390907+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.897032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.897032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":564,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.398211+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:33:37.398211+0000","last_peered":"2024-03-11T21:33:37.398211+0000","last_clean":"2024-03-11T21:33:37.398211+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:33:37.398211+0000","last_undegraded":"2024-03-11T21:33:37.398211+0000","last_fullsized":"2024-03-11T21:33:37.398211+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:45.699273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":573,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.403802+0000","last_change":"2024-03-11T21:33:10.163119+0000","last_active":"2024-03-11T21:33:37.403802+0000","last_peered":"2024-03-11T21:33:37.403802+0000","last_clean":"2024-03-11T21:33:37.403802+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:33:37.403802+0000","last_undegraded":"2024-03-11T21:33:37.403802+0000","last_fullsized":"2024-03-11T21:33:37.403802+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.162829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.162829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.878881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":444,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.406157+0000","last_change":"2024-03-11T21:32:28.978945+0000","last_active":"2024-03-11T21:33:40.406157+0000","last_peered":"2024-03-11T21:33:40.406157+0000","last_clean":"2024-03-11T21:33:40.406157+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:33:40.406157+0000","last_undegraded":"2024-03-11T21:33:40.406157+0000","last_fullsized":"2024-03-11T21:33:40.406157+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":581,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392205+0000","last_change":"2024-03-11T21:33:31.964693+0000","last_active":"2024-03-11T21:33:38.392205+0000","last_peered":"2024-03-11T21:33:38.392205+0000","last_clean":"2024-03-11T21:33:38.392205+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:33:38.392205+0000","last_undegraded":"2024-03-11T21:33:38.392205+0000","last_fullsized":"2024-03-11T21:33:38.392205+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:31.964602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.964602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.956887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":852,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396360+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:33:37.396360+0000","last_peered":"2024-03-11T21:33:37.396360+0000","last_clean":"2024-03-11T21:33:37.396360+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:33:37.396360+0000","last_undegraded":"2024-03-11T21:33:37.396360+0000","last_fullsized":"2024-03-11T21:33:37.396360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":762,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399633+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:33:37.399633+0000","last_peered":"2024-03-11T21:33:37.399633+0000","last_clean":"2024-03-11T21:33:37.399633+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:33:37.399633+0000","last_undegraded":"2024-03-11T21:33:37.399633+0000","last_fullsized":"2024-03-11T21:33:37.399633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.404202+0000","last_change":"2024-03-11T21:33:25.276824+0000","last_active":"2024-03-11T21:33:37.404202+0000","last_peered":"2024-03-11T21:33:37.404202+0000","last_clean":"2024-03-11T21:33:37.404202+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:33:37.404202+0000","last_undegraded":"2024-03-11T21:33:37.404202+0000","last_fullsized":"2024-03-11T21:33:37.404202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.276731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.276731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.729396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":720,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382579+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:33:37.382579+0000","last_peered":"2024-03-11T21:33:37.382579+0000","last_clean":"2024-03-11T21:33:37.382579+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:33:37.382579+0000","last_undegraded":"2024-03-11T21:33:37.382579+0000","last_fullsized":"2024-03-11T21:33:37.382579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.496671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":579,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384250+0000","last_change":"2024-03-11T21:31:51.834491+0000","last_active":"2024-03-11T21:33:37.384250+0000","last_peered":"2024-03-11T21:33:37.384250+0000","last_clean":"2024-03-11T21:33:37.384250+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:33:37.384250+0000","last_undegraded":"2024-03-11T21:33:37.384250+0000","last_fullsized":"2024-03-11T21:33:37.384250+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.834251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.834251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":758,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383749+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:33:37.383749+0000","last_peered":"2024-03-11T21:33:37.383749+0000","last_clean":"2024-03-11T21:33:37.383749+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:33:37.383749+0000","last_undegraded":"2024-03-11T21:33:37.383749+0000","last_fullsized":"2024-03-11T21:33:37.383749+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.006264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":626,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.404507+0000","last_change":"2024-03-11T21:32:06.996760+0000","last_active":"2024-03-11T21:33:40.404507+0000","last_peered":"2024-03-11T21:33:40.404507+0000","last_clean":"2024-03-11T21:33:40.404507+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:33:40.404507+0000","last_undegraded":"2024-03-11T21:33:40.404507+0000","last_fullsized":"2024-03-11T21:33:40.404507+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.996518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.996518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.390566+0000","last_change":"2024-03-11T21:32:28.781453+0000","last_active":"2024-03-11T21:33:37.390566+0000","last_peered":"2024-03-11T21:33:37.390566+0000","last_clean":"2024-03-11T21:33:37.390566+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:33:37.390566+0000","last_undegraded":"2024-03-11T21:33:37.390566+0000","last_fullsized":"2024-03-11T21:33:37.390566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.324762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":723,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389879+0000","last_change":"2024-03-11T21:32:25.048822+0000","last_active":"2024-03-11T21:33:38.389879+0000","last_peered":"2024-03-11T21:33:38.389879+0000","last_clean":"2024-03-11T21:33:38.389879+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:33:38.389879+0000","last_undegraded":"2024-03-11T21:33:38.389879+0000","last_fullsized":"2024-03-11T21:33:38.389879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.048696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.048696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:46.961096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384128+0000","last_change":"2024-03-11T21:33:22.548838+0000","last_active":"2024-03-11T21:33:37.384128+0000","last_peered":"2024-03-11T21:33:37.384128+0000","last_clean":"2024-03-11T21:33:37.384128+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:33:37.384128+0000","last_undegraded":"2024-03-11T21:33:37.384128+0000","last_fullsized":"2024-03-11T21:33:37.384128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.548708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.548708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.314884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":784,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399680+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:33:37.399680+0000","last_peered":"2024-03-11T21:33:37.399680+0000","last_clean":"2024-03-11T21:33:37.399680+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:33:37.399680+0000","last_undegraded":"2024-03-11T21:33:37.399680+0000","last_fullsized":"2024-03-11T21:33:37.399680+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.799876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399593+0000","last_change":"2024-03-11T21:33:12.199443+0000","last_active":"2024-03-11T21:33:37.399593+0000","last_peered":"2024-03-11T21:33:37.399593+0000","last_clean":"2024-03-11T21:33:37.399593+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:33:37.399593+0000","last_undegraded":"2024-03-11T21:33:37.399593+0000","last_fullsized":"2024-03-11T21:33:37.399593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.199234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.199234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:28.102665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.393480+0000","last_change":"2024-03-11T21:32:16.782474+0000","last_active":"2024-03-11T21:33:37.393480+0000","last_peered":"2024-03-11T21:33:37.393480+0000","last_clean":"2024-03-11T21:33:37.393480+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:33:37.393480+0000","last_undegraded":"2024-03-11T21:33:37.393480+0000","last_fullsized":"2024-03-11T21:33:37.393480+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.782400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.782400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.897094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":711,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395325+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:33:37.395325+0000","last_peered":"2024-03-11T21:33:37.395325+0000","last_clean":"2024-03-11T21:33:37.395325+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:33:37.395325+0000","last_undegraded":"2024-03-11T21:33:37.395325+0000","last_fullsized":"2024-03-11T21:33:37.395325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":454,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.395660+0000","last_change":"2024-03-11T21:33:39.131584+0000","last_active":"2024-03-11T21:33:40.395660+0000","last_peered":"2024-03-11T21:33:40.395660+0000","last_clean":"2024-03-11T21:33:40.395660+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:33:40.395660+0000","last_undegraded":"2024-03-11T21:33:40.395660+0000","last_fullsized":"2024-03-11T21:33:40.395660+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.131510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.131510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.433688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":801,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394649+0000","last_change":"2024-03-11T21:31:54.594364+0000","last_active":"2024-03-11T21:33:37.394649+0000","last_peered":"2024-03-11T21:33:37.394649+0000","last_clean":"2024-03-11T21:33:37.394649+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:33:37.394649+0000","last_undegraded":"2024-03-11T21:33:37.394649+0000","last_fullsized":"2024-03-11T21:33:37.394649+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'28","last_scrub_stamp":"2024-03-11T21:31:54.594116+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.594116+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":731,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397875+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:33:37.397875+0000","last_peered":"2024-03-11T21:33:37.397875+0000","last_clean":"2024-03-11T21:33:37.397875+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:33:37.397875+0000","last_undegraded":"2024-03-11T21:33:37.397875+0000","last_fullsized":"2024-03-11T21:33:37.397875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":632,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.403883+0000","last_change":"2024-03-11T21:33:13.942454+0000","last_active":"2024-03-11T21:33:40.403883+0000","last_peered":"2024-03-11T21:33:40.403883+0000","last_clean":"2024-03-11T21:33:40.403883+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:33:40.403883+0000","last_undegraded":"2024-03-11T21:33:40.403883+0000","last_fullsized":"2024-03-11T21:33:40.403883+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.942158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.942158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.543654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":728,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.400997+0000","last_change":"2024-03-11T21:32:05.876383+0000","last_active":"2024-03-11T21:33:37.400997+0000","last_peered":"2024-03-11T21:33:37.400997+0000","last_clean":"2024-03-11T21:33:37.400997+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:33:37.400997+0000","last_undegraded":"2024-03-11T21:33:37.400997+0000","last_fullsized":"2024-03-11T21:33:37.400997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.876018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.876018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":585,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.413992+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:33:37.413992+0000","last_peered":"2024-03-11T21:33:37.413992+0000","last_clean":"2024-03-11T21:33:37.413992+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:33:37.413992+0000","last_undegraded":"2024-03-11T21:33:37.413992+0000","last_fullsized":"2024-03-11T21:33:37.413992+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.202895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":753,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421130+0000","last_change":"2024-03-11T21:32:28.904947+0000","last_active":"2024-03-11T21:33:37.421130+0000","last_peered":"2024-03-11T21:33:37.421130+0000","last_clean":"2024-03-11T21:33:37.421130+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:33:37.421130+0000","last_undegraded":"2024-03-11T21:33:37.421130+0000","last_fullsized":"2024-03-11T21:33:37.421130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:32:28.904821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.904821+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":592,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.401968+0000","last_change":"2024-03-11T21:32:04.059244+0000","last_active":"2024-03-11T21:33:40.401968+0000","last_peered":"2024-03-11T21:33:40.401968+0000","last_clean":"2024-03-11T21:33:40.401968+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:33:40.401968+0000","last_undegraded":"2024-03-11T21:33:40.401968+0000","last_fullsized":"2024-03-11T21:33:40.401968+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.059137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":870,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399419+0000","last_change":"2024-03-11T21:32:07.803946+0000","last_active":"2024-03-11T21:33:37.399419+0000","last_peered":"2024-03-11T21:33:37.399419+0000","last_clean":"2024-03-11T21:33:37.399419+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:33:37.399419+0000","last_undegraded":"2024-03-11T21:33:37.399419+0000","last_fullsized":"2024-03-11T21:33:37.399419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:32:07.803828+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.803828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":730,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388713+0000","last_change":"2024-03-11T21:32:08.744653+0000","last_active":"2024-03-11T21:33:37.388713+0000","last_peered":"2024-03-11T21:33:37.388713+0000","last_clean":"2024-03-11T21:33:37.388713+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:33:37.388713+0000","last_undegraded":"2024-03-11T21:33:37.388713+0000","last_fullsized":"2024-03-11T21:33:37.388713+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.744422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.744422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":580,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.412193+0000","last_change":"2024-03-11T21:31:37.390219+0000","last_active":"2024-03-11T21:33:37.412193+0000","last_peered":"2024-03-11T21:33:37.412193+0000","last_clean":"2024-03-11T21:33:37.412193+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:33:37.412193+0000","last_undegraded":"2024-03-11T21:33:37.412193+0000","last_fullsized":"2024-03-11T21:33:37.412193+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:42.818577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:42.818577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":780,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.412117+0000","last_change":"2024-03-11T21:33:20.697629+0000","last_active":"2024-03-11T21:33:37.412117+0000","last_peered":"2024-03-11T21:33:37.412117+0000","last_clean":"2024-03-11T21:33:37.412117+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:33:37.412117+0000","last_undegraded":"2024-03-11T21:33:37.412117+0000","last_fullsized":"2024-03-11T21:33:37.412117+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:20.697247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.697247+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:39.581547+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":620,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.397545+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:33:40.397545+0000","last_peered":"2024-03-11T21:33:40.397545+0000","last_clean":"2024-03-11T21:33:40.397545+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:33:40.397545+0000","last_undegraded":"2024-03-11T21:33:40.397545+0000","last_fullsized":"2024-03-11T21:33:40.397545+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:48.923389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":862,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388590+0000","last_change":"2024-03-11T21:33:12.674157+0000","last_active":"2024-03-11T21:33:37.388590+0000","last_peered":"2024-03-11T21:33:37.388590+0000","last_clean":"2024-03-11T21:33:37.388590+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:33:37.388590+0000","last_undegraded":"2024-03-11T21:33:37.388590+0000","last_fullsized":"2024-03-11T21:33:37.388590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:12.673934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.673934+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:21.805791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":729,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383656+0000","last_change":"2024-03-11T21:32:05.838146+0000","last_active":"2024-03-11T21:33:37.383656+0000","last_peered":"2024-03-11T21:33:37.383656+0000","last_clean":"2024-03-11T21:33:37.383656+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:33:37.383656+0000","last_undegraded":"2024-03-11T21:33:37.383656+0000","last_fullsized":"2024-03-11T21:33:37.383656+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.838032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.838032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397417+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:33:37.397417+0000","last_peered":"2024-03-11T21:33:37.397417+0000","last_clean":"2024-03-11T21:33:37.397417+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:33:37.397417+0000","last_undegraded":"2024-03-11T21:33:37.397417+0000","last_fullsized":"2024-03-11T21:33:37.397417+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":766,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386987+0000","last_change":"2024-03-11T21:33:35.635520+0000","last_active":"2024-03-11T21:33:37.386987+0000","last_peered":"2024-03-11T21:33:37.386987+0000","last_clean":"2024-03-11T21:33:37.386987+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:33:37.386987+0000","last_undegraded":"2024-03-11T21:33:37.386987+0000","last_fullsized":"2024-03-11T21:33:37.386987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.635256+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.891508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395637+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:33:37.395637+0000","last_peered":"2024-03-11T21:33:37.395637+0000","last_clean":"2024-03-11T21:33:37.395637+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:33:37.395637+0000","last_undegraded":"2024-03-11T21:33:37.395637+0000","last_fullsized":"2024-03-11T21:33:37.395637+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.709131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":724,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384563+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:33:37.384563+0000","last_peered":"2024-03-11T21:33:37.384563+0000","last_clean":"2024-03-11T21:33:37.384563+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:33:37.384563+0000","last_undegraded":"2024-03-11T21:33:37.384563+0000","last_fullsized":"2024-03-11T21:33:37.384563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:49.570886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":712,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421677+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:33:37.421677+0000","last_peered":"2024-03-11T21:33:37.421677+0000","last_clean":"2024-03-11T21:33:37.421677+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:33:37.421677+0000","last_undegraded":"2024-03-11T21:33:37.421677+0000","last_fullsized":"2024-03-11T21:33:37.421677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":571,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421698+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:33:37.421698+0000","last_peered":"2024-03-11T21:33:37.421698+0000","last_clean":"2024-03-11T21:33:37.421698+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:33:37.421698+0000","last_undegraded":"2024-03-11T21:33:37.421698+0000","last_fullsized":"2024-03-11T21:33:37.421698+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:38.667213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":872,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.411852+0000","last_change":"2024-03-11T21:33:10.709056+0000","last_active":"2024-03-11T21:33:37.411852+0000","last_peered":"2024-03-11T21:33:37.411852+0000","last_clean":"2024-03-11T21:33:37.411852+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:33:37.411852+0000","last_undegraded":"2024-03-11T21:33:37.411852+0000","last_fullsized":"2024-03-11T21:33:37.411852+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:33:10.708723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.708723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:28.526439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":743,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.698245+0000","last_change":"2024-03-11T21:33:37.698300+0000","last_active":"2024-03-11T21:33:37.698245+0000","last_peered":"2024-03-11T21:33:37.698245+0000","last_clean":"2024-03-11T21:33:37.698245+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:33:37.698245+0000","last_undegraded":"2024-03-11T21:33:37.698245+0000","last_fullsized":"2024-03-11T21:33:37.698245+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:37.698095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.698095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.205420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":776,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392474+0000","last_change":"2024-03-11T21:31:58.977677+0000","last_active":"2024-03-11T21:33:38.392474+0000","last_peered":"2024-03-11T21:33:38.392474+0000","last_clean":"2024-03-11T21:33:38.392474+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:33:38.392474+0000","last_undegraded":"2024-03-11T21:33:38.392474+0000","last_fullsized":"2024-03-11T21:33:38.392474+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:31:58.977541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.977541+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":749,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397169+0000","last_change":"2024-03-11T21:33:04.660322+0000","last_active":"2024-03-11T21:33:37.397169+0000","last_peered":"2024-03-11T21:33:37.397169+0000","last_clean":"2024-03-11T21:33:37.397169+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:33:37.397169+0000","last_undegraded":"2024-03-11T21:33:37.397169+0000","last_fullsized":"2024-03-11T21:33:37.397169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:33:04.660206+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.660206+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:27.665988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":722,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397701+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:33:37.397701+0000","last_peered":"2024-03-11T21:33:37.397701+0000","last_clean":"2024-03-11T21:33:37.397701+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:33:37.397701+0000","last_undegraded":"2024-03-11T21:33:37.397701+0000","last_fullsized":"2024-03-11T21:33:37.397701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399319+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:33:37.399319+0000","last_peered":"2024-03-11T21:33:37.399319+0000","last_clean":"2024-03-11T21:33:37.399319+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:33:37.399319+0000","last_undegraded":"2024-03-11T21:33:37.399319+0000","last_fullsized":"2024-03-11T21:33:37.399319+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":726,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386915+0000","last_change":"2024-03-11T21:31:02.847904+0000","last_active":"2024-03-11T21:33:37.386915+0000","last_peered":"2024-03-11T21:33:37.386915+0000","last_clean":"2024-03-11T21:33:37.386915+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:33:37.386915+0000","last_undegraded":"2024-03-11T21:33:37.386915+0000","last_fullsized":"2024-03-11T21:33:37.386915+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.847692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.847692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1343,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.416819+0000","last_change":"2024-03-11T21:30:17.907380+0000","last_active":"2024-03-11T21:33:37.416819+0000","last_peered":"2024-03-11T21:33:37.416819+0000","last_clean":"2024-03-11T21:33:37.416819+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:33:37.416819+0000","last_undegraded":"2024-03-11T21:33:37.416819+0000","last_fullsized":"2024-03-11T21:33:37.416819+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:30:17.907168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.907168+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":753,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421329+0000","last_change":"2024-03-11T21:30:47.779423+0000","last_active":"2024-03-11T21:33:37.421329+0000","last_peered":"2024-03-11T21:33:37.421329+0000","last_clean":"2024-03-11T21:33:37.421329+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:33:37.421329+0000","last_undegraded":"2024-03-11T21:33:37.421329+0000","last_fullsized":"2024-03-11T21:33:37.421329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:30:47.779298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.779298+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":645,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.406110+0000","last_change":"2024-03-11T21:33:09.913175+0000","last_active":"2024-03-11T21:33:40.406110+0000","last_peered":"2024-03-11T21:33:40.406110+0000","last_clean":"2024-03-11T21:33:40.406110+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:33:40.406110+0000","last_undegraded":"2024-03-11T21:33:40.406110+0000","last_fullsized":"2024-03-11T21:33:40.406110+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:09.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.912886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:20.681554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":722,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387028+0000","last_change":"2024-03-11T21:30:58.912377+0000","last_active":"2024-03-11T21:33:37.387028+0000","last_peered":"2024-03-11T21:33:37.387028+0000","last_clean":"2024-03-11T21:33:37.387028+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:33:37.387028+0000","last_undegraded":"2024-03-11T21:33:37.387028+0000","last_fullsized":"2024-03-11T21:33:37.387028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.912133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.398263+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:33:37.398263+0000","last_peered":"2024-03-11T21:33:37.398263+0000","last_clean":"2024-03-11T21:33:37.398263+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:33:37.398263+0000","last_undegraded":"2024-03-11T21:33:37.398263+0000","last_fullsized":"2024-03-11T21:33:37.398263+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394580+0000","last_change":"2024-03-11T21:33:09.651325+0000","last_active":"2024-03-11T21:33:37.394580+0000","last_peered":"2024-03-11T21:33:37.394580+0000","last_clean":"2024-03-11T21:33:37.394580+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:33:37.394580+0000","last_undegraded":"2024-03-11T21:33:37.394580+0000","last_fullsized":"2024-03-11T21:33:37.394580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.651101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.651101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:18.894956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"389'44","reported_seq":800,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.406764+0000","last_change":"2024-03-11T21:32:05.077652+0000","last_active":"2024-03-11T21:33:40.406764+0000","last_peered":"2024-03-11T21:33:40.406764+0000","last_clean":"2024-03-11T21:33:40.406764+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:33:40.406764+0000","last_undegraded":"2024-03-11T21:33:40.406764+0000","last_fullsized":"2024-03-11T21:33:40.406764+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'32","last_scrub_stamp":"2024-03-11T21:32:05.077406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.077406+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399571+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:33:37.399571+0000","last_peered":"2024-03-11T21:33:37.399571+0000","last_clean":"2024-03-11T21:33:37.399571+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:33:37.399571+0000","last_undegraded":"2024-03-11T21:33:37.399571+0000","last_fullsized":"2024-03-11T21:33:37.399571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":585,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.397158+0000","last_change":"2024-03-11T21:32:07.975839+0000","last_active":"2024-03-11T21:33:40.397158+0000","last_peered":"2024-03-11T21:33:40.397158+0000","last_clean":"2024-03-11T21:33:40.397158+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:33:40.397158+0000","last_undegraded":"2024-03-11T21:33:40.397158+0000","last_fullsized":"2024-03-11T21:33:40.397158+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.975732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.975732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":573,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392769+0000","last_change":"2024-03-11T21:30:48.777877+0000","last_active":"2024-03-11T21:33:38.392769+0000","last_peered":"2024-03-11T21:33:38.392769+0000","last_clean":"2024-03-11T21:33:38.392769+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:33:38.392769+0000","last_undegraded":"2024-03-11T21:33:38.392769+0000","last_fullsized":"2024-03-11T21:33:38.392769+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.777772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.777772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":733,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384495+0000","last_change":"2024-03-11T21:32:07.813703+0000","last_active":"2024-03-11T21:33:37.384495+0000","last_peered":"2024-03-11T21:33:37.384495+0000","last_clean":"2024-03-11T21:33:37.384495+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:33:37.384495+0000","last_undegraded":"2024-03-11T21:33:37.384495+0000","last_fullsized":"2024-03-11T21:33:37.384495+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.813604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.813604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395440+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:33:37.395440+0000","last_peered":"2024-03-11T21:33:37.395440+0000","last_clean":"2024-03-11T21:33:37.395440+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:33:37.395440+0000","last_undegraded":"2024-03-11T21:33:37.395440+0000","last_fullsized":"2024-03-11T21:33:37.395440+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:11.903723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":774,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389686+0000","last_change":"2024-03-11T21:33:17.014343+0000","last_active":"2024-03-11T21:33:38.389686+0000","last_peered":"2024-03-11T21:33:38.389686+0000","last_clean":"2024-03-11T21:33:38.389686+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:33:38.389686+0000","last_undegraded":"2024-03-11T21:33:38.389686+0000","last_fullsized":"2024-03-11T21:33:38.389686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:17.014179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.014179+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.241673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":595,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405071+0000","last_change":"2024-03-11T21:32:17.944438+0000","last_active":"2024-03-11T21:33:40.405071+0000","last_peered":"2024-03-11T21:33:40.405071+0000","last_clean":"2024-03-11T21:33:40.405071+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:33:40.405071+0000","last_undegraded":"2024-03-11T21:33:40.405071+0000","last_fullsized":"2024-03-11T21:33:40.405071+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.944141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.944141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.417735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":571,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.416622+0000","last_change":"2024-03-11T21:32:16.760994+0000","last_active":"2024-03-11T21:33:37.416622+0000","last_peered":"2024-03-11T21:33:37.416622+0000","last_clean":"2024-03-11T21:33:37.416622+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:33:37.416622+0000","last_undegraded":"2024-03-11T21:33:37.416622+0000","last_fullsized":"2024-03-11T21:33:37.416622+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.760914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.760914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":716,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397907+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:33:37.397907+0000","last_peered":"2024-03-11T21:33:37.397907+0000","last_clean":"2024-03-11T21:33:37.397907+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:33:37.397907+0000","last_undegraded":"2024-03-11T21:33:37.397907+0000","last_fullsized":"2024-03-11T21:33:37.397907+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":987,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395534+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:33:37.395534+0000","last_peered":"2024-03-11T21:33:37.395534+0000","last_clean":"2024-03-11T21:33:37.395534+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:33:37.395534+0000","last_undegraded":"2024-03-11T21:33:37.395534+0000","last_fullsized":"2024-03-11T21:33:37.395534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383289+0000","last_change":"2024-03-11T21:33:13.461458+0000","last_active":"2024-03-11T21:33:37.383289+0000","last_peered":"2024-03-11T21:33:37.383289+0000","last_clean":"2024-03-11T21:33:37.383289+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:33:37.383289+0000","last_undegraded":"2024-03-11T21:33:37.383289+0000","last_fullsized":"2024-03-11T21:33:37.383289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.461214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.461214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.231159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":751,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417064+0000","last_change":"2024-03-11T21:30:32.862005+0000","last_active":"2024-03-11T21:33:37.417064+0000","last_peered":"2024-03-11T21:33:37.417064+0000","last_clean":"2024-03-11T21:33:37.417064+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:33:37.417064+0000","last_undegraded":"2024-03-11T21:33:37.417064+0000","last_fullsized":"2024-03-11T21:33:37.417064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:32.861910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.861910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":720,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384646+0000","last_change":"2024-03-11T21:32:26.159690+0000","last_active":"2024-03-11T21:33:37.384646+0000","last_peered":"2024-03-11T21:33:37.384646+0000","last_clean":"2024-03-11T21:33:37.384646+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:33:37.384646+0000","last_undegraded":"2024-03-11T21:33:37.384646+0000","last_fullsized":"2024-03-11T21:33:37.384646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:26.159472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:26.159472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":579,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382437+0000","last_change":"2024-03-11T21:31:57.949982+0000","last_active":"2024-03-11T21:33:37.382437+0000","last_peered":"2024-03-11T21:33:37.382437+0000","last_clean":"2024-03-11T21:33:37.382437+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:33:37.382437+0000","last_undegraded":"2024-03-11T21:33:37.382437+0000","last_fullsized":"2024-03-11T21:33:37.382437+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.949771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.949771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.404490+0000","last_change":"2024-03-11T21:32:11.769286+0000","last_active":"2024-03-11T21:33:37.404490+0000","last_peered":"2024-03-11T21:33:37.404490+0000","last_clean":"2024-03-11T21:33:37.404490+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:33:37.404490+0000","last_undegraded":"2024-03-11T21:33:37.404490+0000","last_fullsized":"2024-03-11T21:33:37.404490+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.769136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.769136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.393163+0000","last_change":"2024-03-11T21:33:21.218946+0000","last_active":"2024-03-11T21:33:37.393163+0000","last_peered":"2024-03-11T21:33:37.393163+0000","last_clean":"2024-03-11T21:33:37.393163+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:33:37.393163+0000","last_undegraded":"2024-03-11T21:33:37.393163+0000","last_fullsized":"2024-03-11T21:33:37.393163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:21.218831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.218831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.137900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":677,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.396120+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:33:40.396120+0000","last_peered":"2024-03-11T21:33:40.396120+0000","last_clean":"2024-03-11T21:33:40.396120+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:33:40.396120+0000","last_undegraded":"2024-03-11T21:33:40.396120+0000","last_fullsized":"2024-03-11T21:33:40.396120+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.837252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":712,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399511+0000","last_change":"2024-03-11T21:29:10.347962+0000","last_active":"2024-03-11T21:33:37.399511+0000","last_peered":"2024-03-11T21:33:37.399511+0000","last_clean":"2024-03-11T21:33:37.399511+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:33:37.399511+0000","last_undegraded":"2024-03-11T21:33:37.399511+0000","last_fullsized":"2024-03-11T21:33:37.399511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:10.347717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:29:10.347717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":573,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389425+0000","last_change":"2024-03-11T21:30:53.861799+0000","last_active":"2024-03-11T21:33:38.389425+0000","last_peered":"2024-03-11T21:33:38.389425+0000","last_clean":"2024-03-11T21:33:38.389425+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:33:38.389425+0000","last_undegraded":"2024-03-11T21:33:38.389425+0000","last_fullsized":"2024-03-11T21:33:38.389425+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:53.861701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:53.861701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":737,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392139+0000","last_change":"2024-03-11T21:33:35.976814+0000","last_active":"2024-03-11T21:33:38.392139+0000","last_peered":"2024-03-11T21:33:38.392139+0000","last_clean":"2024-03-11T21:33:38.392139+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:33:38.392139+0000","last_undegraded":"2024-03-11T21:33:38.392139+0000","last_fullsized":"2024-03-11T21:33:38.392139+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.976718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.976718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.438197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":789,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.387332+0000","last_change":"2024-03-11T21:30:48.891794+0000","last_active":"2024-03-11T21:33:37.387332+0000","last_peered":"2024-03-11T21:33:37.387332+0000","last_clean":"2024-03-11T21:33:37.387332+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:33:37.387332+0000","last_undegraded":"2024-03-11T21:33:37.387332+0000","last_fullsized":"2024-03-11T21:33:37.387332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:30:48.891648+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.891648+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":742,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382723+0000","last_change":"2024-03-11T21:33:09.417056+0000","last_active":"2024-03-11T21:33:37.382723+0000","last_peered":"2024-03-11T21:33:37.382723+0000","last_clean":"2024-03-11T21:33:37.382723+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:33:37.382723+0000","last_undegraded":"2024-03-11T21:33:37.382723+0000","last_fullsized":"2024-03-11T21:33:37.382723+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.416868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.416868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:09.520332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":720,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420794+0000","last_change":"2024-03-11T21:33:33.610158+0000","last_active":"2024-03-11T21:33:37.420794+0000","last_peered":"2024-03-11T21:33:37.420794+0000","last_clean":"2024-03-11T21:33:37.420794+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:33:37.420794+0000","last_undegraded":"2024-03-11T21:33:37.420794+0000","last_fullsized":"2024-03-11T21:33:37.420794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:33.610071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:33.610071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.428900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":579,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384141+0000","last_change":"2024-03-11T21:31:55.943647+0000","last_active":"2024-03-11T21:33:37.384141+0000","last_peered":"2024-03-11T21:33:37.384141+0000","last_clean":"2024-03-11T21:33:37.384141+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:33:37.384141+0000","last_undegraded":"2024-03-11T21:33:37.384141+0000","last_fullsized":"2024-03-11T21:33:37.384141+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.943487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.943487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":869,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392877+0000","last_change":"2024-03-11T21:32:15.949343+0000","last_active":"2024-03-11T21:33:38.392877+0000","last_peered":"2024-03-11T21:33:38.392877+0000","last_clean":"2024-03-11T21:33:38.392877+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:33:38.392877+0000","last_undegraded":"2024-03-11T21:33:38.392877+0000","last_fullsized":"2024-03-11T21:33:38.392877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:32:15.949107+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.949107+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":630,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.398589+0000","last_change":"2024-03-11T21:33:31.047676+0000","last_active":"2024-03-11T21:33:40.398589+0000","last_peered":"2024-03-11T21:33:40.398589+0000","last_clean":"2024-03-11T21:33:40.398589+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:33:40.398589+0000","last_undegraded":"2024-03-11T21:33:40.398589+0000","last_fullsized":"2024-03-11T21:33:40.398589+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:33:31.047447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.047447+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.521117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":904,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.464569+0000","last_change":"2024-03-11T21:33:37.464613+0000","last_active":"2024-03-11T21:33:37.464569+0000","last_peered":"2024-03-11T21:33:37.464569+0000","last_clean":"2024-03-11T21:33:37.464569+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:33:37.464569+0000","last_undegraded":"2024-03-11T21:33:37.464569+0000","last_fullsized":"2024-03-11T21:33:37.464569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:37.464485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.464485+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.411033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":601,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.404542+0000","last_change":"2024-03-11T21:33:23.993323+0000","last_active":"2024-03-11T21:33:40.404542+0000","last_peered":"2024-03-11T21:33:40.404542+0000","last_clean":"2024-03-11T21:33:40.404542+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:33:40.404542+0000","last_undegraded":"2024-03-11T21:33:40.404542+0000","last_fullsized":"2024-03-11T21:33:40.404542+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.993242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.993242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.886605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":724,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389498+0000","last_change":"2024-03-11T21:32:16.907647+0000","last_active":"2024-03-11T21:33:38.389498+0000","last_peered":"2024-03-11T21:33:38.389498+0000","last_clean":"2024-03-11T21:33:38.389498+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:33:38.389498+0000","last_undegraded":"2024-03-11T21:33:38.389498+0000","last_fullsized":"2024-03-11T21:33:38.389498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.907562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.907562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":446,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.404791+0000","last_change":"2024-03-11T21:31:58.092962+0000","last_active":"2024-03-11T21:33:40.404791+0000","last_peered":"2024-03-11T21:33:40.404791+0000","last_clean":"2024-03-11T21:33:40.404791+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:33:40.404791+0000","last_undegraded":"2024-03-11T21:33:40.404791+0000","last_fullsized":"2024-03-11T21:33:40.404791+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.092744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.092744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388158+0000","last_change":"2024-03-11T21:32:12.801345+0000","last_active":"2024-03-11T21:33:37.388158+0000","last_peered":"2024-03-11T21:33:37.388158+0000","last_clean":"2024-03-11T21:33:37.388158+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:33:37.388158+0000","last_undegraded":"2024-03-11T21:33:37.388158+0000","last_fullsized":"2024-03-11T21:33:37.388158+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":772,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.398397+0000","last_change":"2024-03-11T21:33:31.683462+0000","last_active":"2024-03-11T21:33:37.398397+0000","last_peered":"2024-03-11T21:33:37.398397+0000","last_clean":"2024-03-11T21:33:37.398397+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:33:37.398397+0000","last_undegraded":"2024-03-11T21:33:37.398397+0000","last_fullsized":"2024-03-11T21:33:37.398397+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.683216+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.041563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":751,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388363+0000","last_change":"2024-03-11T21:31:05.819709+0000","last_active":"2024-03-11T21:33:37.388363+0000","last_peered":"2024-03-11T21:33:37.388363+0000","last_clean":"2024-03-11T21:33:37.388363+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:33:37.388363+0000","last_undegraded":"2024-03-11T21:33:37.388363+0000","last_fullsized":"2024-03-11T21:33:37.388363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:31:05.819557+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.819557+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":729,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.413090+0000","last_change":"2024-03-11T21:33:14.693542+0000","last_active":"2024-03-11T21:33:37.413090+0000","last_peered":"2024-03-11T21:33:37.413090+0000","last_clean":"2024-03-11T21:33:37.413090+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:33:37.413090+0000","last_undegraded":"2024-03-11T21:33:37.413090+0000","last_fullsized":"2024-03-11T21:33:37.413090+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:14.693316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.693316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.736549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":570,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421531+0000","last_change":"2024-03-11T21:30:40.750808+0000","last_active":"2024-03-11T21:33:37.421531+0000","last_peered":"2024-03-11T21:33:37.421531+0000","last_clean":"2024-03-11T21:33:37.421531+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:33:37.421531+0000","last_undegraded":"2024-03-11T21:33:37.421531+0000","last_fullsized":"2024-03-11T21:33:37.421531+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.750735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.750735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":758,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396227+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:33:37.396227+0000","last_peered":"2024-03-11T21:33:37.396227+0000","last_clean":"2024-03-11T21:33:37.396227+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:33:37.396227+0000","last_undegraded":"2024-03-11T21:33:37.396227+0000","last_fullsized":"2024-03-11T21:33:37.396227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.123454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":749,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388065+0000","last_change":"2024-03-11T21:33:08.186982+0000","last_active":"2024-03-11T21:33:37.388065+0000","last_peered":"2024-03-11T21:33:37.388065+0000","last_clean":"2024-03-11T21:33:37.388065+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:33:37.388065+0000","last_undegraded":"2024-03-11T21:33:37.388065+0000","last_fullsized":"2024-03-11T21:33:37.388065+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.186732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.186732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.992162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":733,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.413823+0000","last_change":"2024-03-11T21:33:32.575760+0000","last_active":"2024-03-11T21:33:37.413823+0000","last_peered":"2024-03-11T21:33:37.413823+0000","last_clean":"2024-03-11T21:33:37.413823+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:33:37.413823+0000","last_undegraded":"2024-03-11T21:33:37.413823+0000","last_fullsized":"2024-03-11T21:33:37.413823+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.575652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.575652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.490302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":728,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.400235+0000","last_change":"2024-03-11T21:32:04.847327+0000","last_active":"2024-03-11T21:33:37.400235+0000","last_peered":"2024-03-11T21:33:37.400235+0000","last_clean":"2024-03-11T21:33:37.400235+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:33:37.400235+0000","last_undegraded":"2024-03-11T21:33:37.400235+0000","last_fullsized":"2024-03-11T21:33:37.400235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.847094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.847094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":570,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.417353+0000","last_change":"2024-03-11T21:30:48.791220+0000","last_active":"2024-03-11T21:33:37.417353+0000","last_peered":"2024-03-11T21:33:37.417353+0000","last_clean":"2024-03-11T21:33:37.417353+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:33:37.417353+0000","last_undegraded":"2024-03-11T21:33:37.417353+0000","last_fullsized":"2024-03-11T21:33:37.417353+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.791036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":750,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394178+0000","last_change":"2024-03-11T21:32:15.567675+0000","last_active":"2024-03-11T21:33:37.394178+0000","last_peered":"2024-03-11T21:33:37.394178+0000","last_clean":"2024-03-11T21:33:37.394178+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:33:37.394178+0000","last_undegraded":"2024-03-11T21:33:37.394178+0000","last_fullsized":"2024-03-11T21:33:37.394178+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:32:15.567443+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.567443+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.706289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394979+0000","last_change":"2024-03-11T21:31:59.593413+0000","last_active":"2024-03-11T21:33:37.394979+0000","last_peered":"2024-03-11T21:33:37.394979+0000","last_clean":"2024-03-11T21:33:37.394979+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:33:37.394979+0000","last_undegraded":"2024-03-11T21:33:37.394979+0000","last_fullsized":"2024-03-11T21:33:37.394979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.593037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.593037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":733,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384656+0000","last_change":"2024-03-11T21:32:06.800643+0000","last_active":"2024-03-11T21:33:37.384656+0000","last_peered":"2024-03-11T21:33:37.384656+0000","last_clean":"2024-03-11T21:33:37.384656+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:33:37.384656+0000","last_undegraded":"2024-03-11T21:33:37.384656+0000","last_fullsized":"2024-03-11T21:33:37.384656+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.800436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.800436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":719,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399425+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:33:37.399425+0000","last_peered":"2024-03-11T21:33:37.399425+0000","last_clean":"2024-03-11T21:33:37.399425+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:33:37.399425+0000","last_undegraded":"2024-03-11T21:33:37.399425+0000","last_fullsized":"2024-03-11T21:33:37.399425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.346560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":571,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.416001+0000","last_change":"2024-03-11T21:31:57.633633+0000","last_active":"2024-03-11T21:33:37.416001+0000","last_peered":"2024-03-11T21:33:37.416001+0000","last_clean":"2024-03-11T21:33:37.416001+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:33:37.416001+0000","last_undegraded":"2024-03-11T21:33:37.416001+0000","last_fullsized":"2024-03-11T21:33:37.416001+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.633438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.633438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":758,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397983+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:33:37.397983+0000","last_peered":"2024-03-11T21:33:37.397983+0000","last_clean":"2024-03-11T21:33:37.397983+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:33:37.397983+0000","last_undegraded":"2024-03-11T21:33:37.397983+0000","last_fullsized":"2024-03-11T21:33:37.397983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382629+0000","last_change":"2024-03-11T21:33:10.436145+0000","last_active":"2024-03-11T21:33:37.382629+0000","last_peered":"2024-03-11T21:33:37.382629+0000","last_clean":"2024-03-11T21:33:37.382629+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:33:37.382629+0000","last_undegraded":"2024-03-11T21:33:37.382629+0000","last_fullsized":"2024-03-11T21:33:37.382629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.436027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.436027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.260422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":871,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384043+0000","last_change":"2024-03-11T21:32:15.855148+0000","last_active":"2024-03-11T21:33:37.384043+0000","last_peered":"2024-03-11T21:33:37.384043+0000","last_clean":"2024-03-11T21:33:37.384043+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:33:37.384043+0000","last_undegraded":"2024-03-11T21:33:37.384043+0000","last_fullsized":"2024-03-11T21:33:37.384043+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:32:15.854918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.854918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":723,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392984+0000","last_change":"2024-03-11T21:30:49.769958+0000","last_active":"2024-03-11T21:33:38.392984+0000","last_peered":"2024-03-11T21:33:38.392984+0000","last_clean":"2024-03-11T21:33:38.392984+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:33:38.392984+0000","last_undegraded":"2024-03-11T21:33:38.392984+0000","last_fullsized":"2024-03-11T21:33:38.392984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:30:49.769830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":579,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394577+0000","last_change":"2024-03-11T21:33:35.434748+0000","last_active":"2024-03-11T21:33:37.394577+0000","last_peered":"2024-03-11T21:33:37.394577+0000","last_clean":"2024-03-11T21:33:37.394577+0000","last_became_active":"2024-03-11T21:28:51.357900+0000","last_became_peered":"2024-03-11T21:28:51.357900+0000","last_unstale":"2024-03-11T21:33:37.394577+0000","last_undegraded":"2024-03-11T21:33:37.394577+0000","last_fullsized":"2024-03-11T21:33:37.394577+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.434631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.434631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.368575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":760,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.391301+0000","last_change":"2024-03-11T21:30:52.813167+0000","last_active":"2024-03-11T21:33:38.391301+0000","last_peered":"2024-03-11T21:33:38.391301+0000","last_clean":"2024-03-11T21:33:38.391301+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:33:38.391301+0000","last_undegraded":"2024-03-11T21:33:38.391301+0000","last_fullsized":"2024-03-11T21:33:38.391301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'13","last_scrub_stamp":"2024-03-11T21:30:52.812936+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.812936+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":735,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.391741+0000","last_change":"2024-03-11T21:30:58.882981+0000","last_active":"2024-03-11T21:33:38.391741+0000","last_peered":"2024-03-11T21:33:38.391741+0000","last_clean":"2024-03-11T21:33:38.391741+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:33:38.391741+0000","last_undegraded":"2024-03-11T21:33:38.391741+0000","last_fullsized":"2024-03-11T21:33:38.391741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.882820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.882820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":862,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421538+0000","last_change":"2024-03-11T21:33:19.697889+0000","last_active":"2024-03-11T21:33:37.421538+0000","last_peered":"2024-03-11T21:33:37.421538+0000","last_clean":"2024-03-11T21:33:37.421538+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:33:37.421538+0000","last_undegraded":"2024-03-11T21:33:37.421538+0000","last_fullsized":"2024-03-11T21:33:37.421538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:33:19.697575+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.697575+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.072080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":721,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388585+0000","last_change":"2024-03-11T21:30:31.977271+0000","last_active":"2024-03-11T21:33:37.388585+0000","last_peered":"2024-03-11T21:33:37.388585+0000","last_clean":"2024-03-11T21:33:37.388585+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:33:37.388585+0000","last_undegraded":"2024-03-11T21:33:37.388585+0000","last_fullsized":"2024-03-11T21:33:37.388585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:31.977047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:31.977047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":450,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.407222+0000","last_change":"2024-03-11T21:33:26.034503+0000","last_active":"2024-03-11T21:33:40.407222+0000","last_peered":"2024-03-11T21:33:40.407222+0000","last_clean":"2024-03-11T21:33:40.407222+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:33:40.407222+0000","last_undegraded":"2024-03-11T21:33:40.407222+0000","last_fullsized":"2024-03-11T21:33:40.407222+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:26.034377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:26.034377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.744337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":756,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399803+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:33:37.399803+0000","last_peered":"2024-03-11T21:33:37.399803+0000","last_clean":"2024-03-11T21:33:37.399803+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:33:37.399803+0000","last_undegraded":"2024-03-11T21:33:37.399803+0000","last_fullsized":"2024-03-11T21:33:37.399803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396127+0000","last_change":"2024-03-11T21:32:09.571388+0000","last_active":"2024-03-11T21:33:37.396127+0000","last_peered":"2024-03-11T21:33:37.396127+0000","last_clean":"2024-03-11T21:33:37.396127+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:33:37.396127+0000","last_undegraded":"2024-03-11T21:33:37.396127+0000","last_fullsized":"2024-03-11T21:33:37.396127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.571257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.571257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.412284+0000","last_change":"2024-03-11T21:31:37.402794+0000","last_active":"2024-03-11T21:33:37.412284+0000","last_peered":"2024-03-11T21:33:37.412284+0000","last_clean":"2024-03-11T21:33:37.412284+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:33:37.412284+0000","last_undegraded":"2024-03-11T21:33:37.412284+0000","last_fullsized":"2024-03-11T21:33:37.412284+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:28.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:28.810562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":732,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389321+0000","last_change":"2024-03-11T21:32:06.989987+0000","last_active":"2024-03-11T21:33:38.389321+0000","last_peered":"2024-03-11T21:33:38.389321+0000","last_clean":"2024-03-11T21:33:38.389321+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:33:38.389321+0000","last_undegraded":"2024-03-11T21:33:38.389321+0000","last_fullsized":"2024-03-11T21:33:38.389321+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.989724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.989724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":581,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392375+0000","last_change":"2024-03-11T21:33:29.058203+0000","last_active":"2024-03-11T21:33:38.392375+0000","last_peered":"2024-03-11T21:33:38.392375+0000","last_clean":"2024-03-11T21:33:38.392375+0000","last_became_active":"2024-03-11T21:28:51.388727+0000","last_became_peered":"2024-03-11T21:28:51.388727+0000","last_unstale":"2024-03-11T21:33:38.392375+0000","last_undegraded":"2024-03-11T21:33:38.392375+0000","last_fullsized":"2024-03-11T21:33:38.392375+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:29.057966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.057966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:56.614499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":759,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388008+0000","last_change":"2024-03-11T21:30:38.863143+0000","last_active":"2024-03-11T21:33:37.388008+0000","last_peered":"2024-03-11T21:33:37.388008+0000","last_clean":"2024-03-11T21:33:37.388008+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:33:37.388008+0000","last_undegraded":"2024-03-11T21:33:37.388008+0000","last_fullsized":"2024-03-11T21:33:37.388008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:38.862999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.862999+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":636,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405681+0000","last_change":"2024-03-11T21:32:11.929832+0000","last_active":"2024-03-11T21:33:40.405681+0000","last_peered":"2024-03-11T21:33:40.405681+0000","last_clean":"2024-03-11T21:33:40.405681+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:33:40.405681+0000","last_undegraded":"2024-03-11T21:33:40.405681+0000","last_fullsized":"2024-03-11T21:33:40.405681+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.929628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.929628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":604,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.398214+0000","last_change":"2024-03-11T21:33:07.920722+0000","last_active":"2024-03-11T21:33:40.398214+0000","last_peered":"2024-03-11T21:33:40.398214+0000","last_clean":"2024-03-11T21:33:40.398214+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:33:40.398214+0000","last_undegraded":"2024-03-11T21:33:40.398214+0000","last_fullsized":"2024-03-11T21:33:40.398214+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:07.920509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.920509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.434359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":720,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.414331+0000","last_change":"2024-03-11T21:33:35.541846+0000","last_active":"2024-03-11T21:33:37.414331+0000","last_peered":"2024-03-11T21:33:37.414331+0000","last_clean":"2024-03-11T21:33:37.414331+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:33:37.414331+0000","last_undegraded":"2024-03-11T21:33:37.414331+0000","last_fullsized":"2024-03-11T21:33:37.414331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.541731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.541731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.532157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":572,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.391494+0000","last_change":"2024-03-11T21:32:18.866826+0000","last_active":"2024-03-11T21:33:38.391494+0000","last_peered":"2024-03-11T21:33:38.391494+0000","last_clean":"2024-03-11T21:33:38.391494+0000","last_became_active":"2024-03-11T21:28:51.388792+0000","last_became_peered":"2024-03-11T21:28:51.388792+0000","last_unstale":"2024-03-11T21:33:38.391494+0000","last_undegraded":"2024-03-11T21:33:38.391494+0000","last_fullsized":"2024-03-11T21:33:38.391494+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.866701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.866701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":757,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.394277+0000","last_change":"2024-03-11T21:33:20.954157+0000","last_active":"2024-03-11T21:33:38.394277+0000","last_peered":"2024-03-11T21:33:38.394277+0000","last_clean":"2024-03-11T21:33:38.394277+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:33:38.394277+0000","last_undegraded":"2024-03-11T21:33:38.394277+0000","last_fullsized":"2024-03-11T21:33:38.394277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:33:20.954022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.954022+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.519773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384933+0000","last_change":"2024-03-11T21:31:00.965260+0000","last_active":"2024-03-11T21:33:37.384933+0000","last_peered":"2024-03-11T21:33:37.384933+0000","last_clean":"2024-03-11T21:33:37.384933+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:33:37.384933+0000","last_undegraded":"2024-03-11T21:33:37.384933+0000","last_fullsized":"2024-03-11T21:33:37.384933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.964917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.964917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394916+0000","last_change":"2024-03-11T21:32:12.550833+0000","last_active":"2024-03-11T21:33:37.394916+0000","last_peered":"2024-03-11T21:33:37.394916+0000","last_clean":"2024-03-11T21:33:37.394916+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:33:37.394916+0000","last_undegraded":"2024-03-11T21:33:37.394916+0000","last_fullsized":"2024-03-11T21:33:37.394916+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.550625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.550625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":711,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396998+0000","last_change":"2024-03-11T21:29:11.346981+0000","last_active":"2024-03-11T21:33:37.396998+0000","last_peered":"2024-03-11T21:33:37.396998+0000","last_clean":"2024-03-11T21:33:37.396998+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:33:37.396998+0000","last_undegraded":"2024-03-11T21:33:37.396998+0000","last_fullsized":"2024-03-11T21:33:37.396998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.346840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.346840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":581,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.391196+0000","last_change":"2024-03-11T21:32:12.998779+0000","last_active":"2024-03-11T21:33:38.391196+0000","last_peered":"2024-03-11T21:33:38.391196+0000","last_clean":"2024-03-11T21:33:38.391196+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:33:38.391196+0000","last_undegraded":"2024-03-11T21:33:38.391196+0000","last_fullsized":"2024-03-11T21:33:38.391196+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.998644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.998644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":762,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.384494+0000","last_change":"2024-03-11T21:31:59.996470+0000","last_active":"2024-03-11T21:33:37.384494+0000","last_peered":"2024-03-11T21:33:37.384494+0000","last_clean":"2024-03-11T21:33:37.384494+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:33:37.384494+0000","last_undegraded":"2024-03-11T21:33:37.384494+0000","last_fullsized":"2024-03-11T21:33:37.384494+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:59.996246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.996246+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397063+0000","last_change":"2024-03-11T21:32:28.777744+0000","last_active":"2024-03-11T21:33:37.397063+0000","last_peered":"2024-03-11T21:33:37.397063+0000","last_clean":"2024-03-11T21:33:37.397063+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:33:37.397063+0000","last_undegraded":"2024-03-11T21:33:37.397063+0000","last_fullsized":"2024-03-11T21:33:37.397063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.777629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.777629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":715,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.395898+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:33:37.395898+0000","last_peered":"2024-03-11T21:33:37.395898+0000","last_clean":"2024-03-11T21:33:37.395898+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:33:37.395898+0000","last_undegraded":"2024-03-11T21:33:37.395898+0000","last_fullsized":"2024-03-11T21:33:37.395898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":712,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421443+0000","last_change":"2024-03-11T21:30:52.794455+0000","last_active":"2024-03-11T21:33:37.421443+0000","last_peered":"2024-03-11T21:33:37.421443+0000","last_clean":"2024-03-11T21:33:37.421443+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:33:37.421443+0000","last_undegraded":"2024-03-11T21:33:37.421443+0000","last_fullsized":"2024-03-11T21:33:37.421443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.794209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.794209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":569,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397786+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:33:37.397786+0000","last_peered":"2024-03-11T21:33:37.397786+0000","last_clean":"2024-03-11T21:33:37.397786+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:33:37.397786+0000","last_undegraded":"2024-03-11T21:33:37.397786+0000","last_fullsized":"2024-03-11T21:33:37.397786+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"389'27","reported_seq":690,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.397275+0000","last_change":"2024-03-11T21:33:21.967867+0000","last_active":"2024-03-11T21:33:40.397275+0000","last_peered":"2024-03-11T21:33:40.397275+0000","last_clean":"2024-03-11T21:33:40.397275+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:33:40.397275+0000","last_undegraded":"2024-03-11T21:33:40.397275+0000","last_fullsized":"2024-03-11T21:33:40.397275+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:21.967756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.967756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:50.206133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":738,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382540+0000","last_change":"2024-03-11T21:32:29.159917+0000","last_active":"2024-03-11T21:33:37.382540+0000","last_peered":"2024-03-11T21:33:37.382540+0000","last_clean":"2024-03-11T21:33:37.382540+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:33:37.382540+0000","last_undegraded":"2024-03-11T21:33:37.382540+0000","last_fullsized":"2024-03-11T21:33:37.382540+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:29.159716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:29.159716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:41.410503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":873,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397270+0000","last_change":"2024-03-11T21:33:10.606465+0000","last_active":"2024-03-11T21:33:37.397270+0000","last_peered":"2024-03-11T21:33:37.397270+0000","last_clean":"2024-03-11T21:33:37.397270+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:33:37.397270+0000","last_undegraded":"2024-03-11T21:33:37.397270+0000","last_fullsized":"2024-03-11T21:33:37.397270+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:10.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.606357+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:15.257340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":713,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.398070+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:33:37.398070+0000","last_peered":"2024-03-11T21:33:37.398070+0000","last_clean":"2024-03-11T21:33:37.398070+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:33:37.398070+0000","last_undegraded":"2024-03-11T21:33:37.398070+0000","last_fullsized":"2024-03-11T21:33:37.398070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":587,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382519+0000","last_change":"2024-03-11T21:33:12.438198+0000","last_active":"2024-03-11T21:33:37.382519+0000","last_peered":"2024-03-11T21:33:37.382519+0000","last_clean":"2024-03-11T21:33:37.382519+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:33:37.382519+0000","last_undegraded":"2024-03-11T21:33:37.382519+0000","last_fullsized":"2024-03-11T21:33:37.382519+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.438028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.438028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.544269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":762,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.412273+0000","last_change":"2024-03-11T21:31:54.548395+0000","last_active":"2024-03-11T21:33:37.412273+0000","last_peered":"2024-03-11T21:33:37.412273+0000","last_clean":"2024-03-11T21:33:37.412273+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:33:37.412273+0000","last_undegraded":"2024-03-11T21:33:37.412273+0000","last_fullsized":"2024-03-11T21:33:37.412273+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:54.548275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.548275+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":733,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.414331+0000","last_change":"2024-03-11T21:33:06.710223+0000","last_active":"2024-03-11T21:33:37.414331+0000","last_peered":"2024-03-11T21:33:37.414331+0000","last_clean":"2024-03-11T21:33:37.414331+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:33:37.414331+0000","last_undegraded":"2024-03-11T21:33:37.414331+0000","last_fullsized":"2024-03-11T21:33:37.414331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:06.709989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.709989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.785762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":728,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.392925+0000","last_change":"2024-03-11T21:32:23.014073+0000","last_active":"2024-03-11T21:33:38.392925+0000","last_peered":"2024-03-11T21:33:38.392925+0000","last_clean":"2024-03-11T21:33:38.392925+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:33:38.392925+0000","last_undegraded":"2024-03-11T21:33:38.392925+0000","last_fullsized":"2024-03-11T21:33:38.392925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.013984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.013984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.212286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":595,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405249+0000","last_change":"2024-03-11T21:33:05.875547+0000","last_active":"2024-03-11T21:33:40.405249+0000","last_peered":"2024-03-11T21:33:40.405249+0000","last_clean":"2024-03-11T21:33:40.405249+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:33:40.405249+0000","last_undegraded":"2024-03-11T21:33:40.405249+0000","last_fullsized":"2024-03-11T21:33:40.405249+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:05.875434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.875434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.539249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":569,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396086+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:33:37.396086+0000","last_peered":"2024-03-11T21:33:37.396086+0000","last_clean":"2024-03-11T21:33:37.396086+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:33:37.396086+0000","last_undegraded":"2024-03-11T21:33:37.396086+0000","last_fullsized":"2024-03-11T21:33:37.396086+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":746,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.391740+0000","last_change":"2024-03-11T21:33:19.017659+0000","last_active":"2024-03-11T21:33:38.391740+0000","last_peered":"2024-03-11T21:33:38.391740+0000","last_clean":"2024-03-11T21:33:38.391740+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:33:38.391740+0000","last_undegraded":"2024-03-11T21:33:38.391740+0000","last_fullsized":"2024-03-11T21:33:38.391740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:33:19.017449+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.017449+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.631706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.394543+0000","last_change":"2024-03-11T21:32:10.547894+0000","last_active":"2024-03-11T21:33:37.394543+0000","last_peered":"2024-03-11T21:33:37.394543+0000","last_clean":"2024-03-11T21:33:37.394543+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:33:37.394543+0000","last_undegraded":"2024-03-11T21:33:37.394543+0000","last_fullsized":"2024-03-11T21:33:37.394543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.547682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.547682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":589,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.406407+0000","last_change":"2024-03-11T21:33:36.120790+0000","last_active":"2024-03-11T21:33:40.406407+0000","last_peered":"2024-03-11T21:33:40.406407+0000","last_clean":"2024-03-11T21:33:40.406407+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:33:40.406407+0000","last_undegraded":"2024-03-11T21:33:40.406407+0000","last_fullsized":"2024-03-11T21:33:40.406407+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.120602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.120602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.298821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":722,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388725+0000","last_change":"2024-03-11T21:30:40.909457+0000","last_active":"2024-03-11T21:33:37.388725+0000","last_peered":"2024-03-11T21:33:37.388725+0000","last_clean":"2024-03-11T21:33:37.388725+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:33:37.388725+0000","last_undegraded":"2024-03-11T21:33:37.388725+0000","last_fullsized":"2024-03-11T21:33:37.388725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:40.909356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:40.909356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":571,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386562+0000","last_change":"2024-03-11T21:32:25.655891+0000","last_active":"2024-03-11T21:33:37.386562+0000","last_peered":"2024-03-11T21:33:37.386562+0000","last_clean":"2024-03-11T21:33:37.386562+0000","last_became_active":"2024-03-11T21:28:51.388058+0000","last_became_peered":"2024-03-11T21:28:51.388058+0000","last_unstale":"2024-03-11T21:33:37.386562+0000","last_undegraded":"2024-03-11T21:33:37.386562+0000","last_fullsized":"2024-03-11T21:33:37.386562+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.655683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:53.974261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":771,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386211+0000","last_change":"2024-03-11T21:33:27.686917+0000","last_active":"2024-03-11T21:33:37.386211+0000","last_peered":"2024-03-11T21:33:37.386211+0000","last_clean":"2024-03-11T21:33:37.386211+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:33:37.386211+0000","last_undegraded":"2024-03-11T21:33:37.386211+0000","last_fullsized":"2024-03-11T21:33:37.386211+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:33:27.686783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.686783+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.924138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":732,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420922+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:33:37.420922+0000","last_peered":"2024-03-11T21:33:37.420922+0000","last_clean":"2024-03-11T21:33:37.420922+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:33:37.420922+0000","last_undegraded":"2024-03-11T21:33:37.420922+0000","last_fullsized":"2024-03-11T21:33:37.420922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:50.107199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388168+0000","last_change":"2024-03-11T21:32:09.768728+0000","last_active":"2024-03-11T21:33:37.388168+0000","last_peered":"2024-03-11T21:33:37.388168+0000","last_clean":"2024-03-11T21:33:37.388168+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:33:37.388168+0000","last_undegraded":"2024-03-11T21:33:37.388168+0000","last_fullsized":"2024-03-11T21:33:37.388168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.768608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.768608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":713,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421519+0000","last_change":"2024-03-11T21:30:36.777188+0000","last_active":"2024-03-11T21:33:37.421519+0000","last_peered":"2024-03-11T21:33:37.421519+0000","last_clean":"2024-03-11T21:33:37.421519+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:33:37.421519+0000","last_undegraded":"2024-03-11T21:33:37.421519+0000","last_fullsized":"2024-03-11T21:33:37.421519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.777107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.777107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399499+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:33:37.399499+0000","last_peered":"2024-03-11T21:33:37.399499+0000","last_clean":"2024-03-11T21:33:37.399499+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:33:37.399499+0000","last_undegraded":"2024-03-11T21:33:37.399499+0000","last_fullsized":"2024-03-11T21:33:37.399499+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":760,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383960+0000","last_change":"2024-03-11T21:32:09.759891+0000","last_active":"2024-03-11T21:33:37.383960+0000","last_peered":"2024-03-11T21:33:37.383960+0000","last_clean":"2024-03-11T21:33:37.383960+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:33:37.383960+0000","last_undegraded":"2024-03-11T21:33:37.383960+0000","last_fullsized":"2024-03-11T21:33:37.383960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:32:09.759634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.759634+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":731,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.397709+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:33:37.397709+0000","last_peered":"2024-03-11T21:33:37.397709+0000","last_clean":"2024-03-11T21:33:37.397709+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:33:37.397709+0000","last_undegraded":"2024-03-11T21:33:37.397709+0000","last_fullsized":"2024-03-11T21:33:37.397709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":724,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383174+0000","last_change":"2024-03-11T21:33:17.530575+0000","last_active":"2024-03-11T21:33:37.383174+0000","last_peered":"2024-03-11T21:33:37.383174+0000","last_clean":"2024-03-11T21:33:37.383174+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:33:37.383174+0000","last_undegraded":"2024-03-11T21:33:37.383174+0000","last_fullsized":"2024-03-11T21:33:37.383174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:17.530299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.530299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.486176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":732,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.389456+0000","last_change":"2024-03-11T21:32:01.014273+0000","last_active":"2024-03-11T21:33:38.389456+0000","last_peered":"2024-03-11T21:33:38.389456+0000","last_clean":"2024-03-11T21:33:38.389456+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:33:38.389456+0000","last_undegraded":"2024-03-11T21:33:38.389456+0000","last_fullsized":"2024-03-11T21:33:38.389456+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.014168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.014168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":580,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.388285+0000","last_change":"2024-03-11T21:32:15.652648+0000","last_active":"2024-03-11T21:33:37.388285+0000","last_peered":"2024-03-11T21:33:37.388285+0000","last_clean":"2024-03-11T21:33:37.388285+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:33:37.388285+0000","last_undegraded":"2024-03-11T21:33:37.388285+0000","last_fullsized":"2024-03-11T21:33:37.388285+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.652323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.652323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":795,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.383996+0000","last_change":"2024-03-11T21:33:20.542248+0000","last_active":"2024-03-11T21:33:37.383996+0000","last_peered":"2024-03-11T21:33:37.383996+0000","last_clean":"2024-03-11T21:33:37.383996+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:33:37.383996+0000","last_undegraded":"2024-03-11T21:33:37.383996+0000","last_fullsized":"2024-03-11T21:33:37.383996+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:33:20.542001+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.542001+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.858604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":731,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.399737+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:33:37.399737+0000","last_peered":"2024-03-11T21:33:37.399737+0000","last_clean":"2024-03-11T21:33:37.399737+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:33:37.399737+0000","last_undegraded":"2024-03-11T21:33:37.399737+0000","last_fullsized":"2024-03-11T21:33:37.399737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":724,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.396579+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:33:37.396579+0000","last_peered":"2024-03-11T21:33:37.396579+0000","last_clean":"2024-03-11T21:33:37.396579+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:33:37.396579+0000","last_undegraded":"2024-03-11T21:33:37.396579+0000","last_fullsized":"2024-03-11T21:33:37.396579+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":729,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.382262+0000","last_change":"2024-03-11T21:33:23.549947+0000","last_active":"2024-03-11T21:33:37.382262+0000","last_peered":"2024-03-11T21:33:37.382262+0000","last_clean":"2024-03-11T21:33:37.382262+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:33:37.382262+0000","last_undegraded":"2024-03-11T21:33:37.382262+0000","last_fullsized":"2024-03-11T21:33:37.382262+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.549773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.549773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:34.175330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":578,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.398383+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:33:37.398383+0000","last_peered":"2024-03-11T21:33:37.398383+0000","last_clean":"2024-03-11T21:33:37.398383+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:33:37.398383+0000","last_undegraded":"2024-03-11T21:33:37.398383+0000","last_fullsized":"2024-03-11T21:33:37.398383+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":768,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.394460+0000","last_change":"2024-03-11T21:30:35.925775+0000","last_active":"2024-03-11T21:33:38.394460+0000","last_peered":"2024-03-11T21:33:38.394460+0000","last_clean":"2024-03-11T21:33:38.394460+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:33:38.394460+0000","last_undegraded":"2024-03-11T21:33:38.394460+0000","last_fullsized":"2024-03-11T21:33:38.394460+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-03-11T21:30:35.925542+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:35.925542+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420585+0000","last_change":"2024-03-11T21:30:30.892128+0000","last_active":"2024-03-11T21:33:37.420585+0000","last_peered":"2024-03-11T21:33:37.420585+0000","last_clean":"2024-03-11T21:33:37.420585+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:33:37.420585+0000","last_undegraded":"2024-03-11T21:33:37.420585+0000","last_fullsized":"2024-03-11T21:33:37.420585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:30.891767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.891767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":585,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.397684+0000","last_change":"2024-03-11T21:32:02.053080+0000","last_active":"2024-03-11T21:33:40.397684+0000","last_peered":"2024-03-11T21:33:40.397684+0000","last_clean":"2024-03-11T21:33:40.397684+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:33:40.397684+0000","last_undegraded":"2024-03-11T21:33:40.397684+0000","last_fullsized":"2024-03-11T21:33:40.397684+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.052865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.052865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":730,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386226+0000","last_change":"2024-03-11T21:32:07.733765+0000","last_active":"2024-03-11T21:33:37.386226+0000","last_peered":"2024-03-11T21:33:37.386226+0000","last_clean":"2024-03-11T21:33:37.386226+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:33:37.386226+0000","last_undegraded":"2024-03-11T21:33:37.386226+0000","last_fullsized":"2024-03-11T21:33:37.386226+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.733556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.733556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":443,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-11T21:33:40.405958+0000","last_change":"2024-03-11T21:33:22.998783+0000","last_active":"2024-03-11T21:33:40.405958+0000","last_peered":"2024-03-11T21:33:40.405958+0000","last_clean":"2024-03-11T21:33:40.405958+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:33:40.405958+0000","last_undegraded":"2024-03-11T21:33:40.405958+0000","last_fullsized":"2024-03-11T21:33:40.405958+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.468087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":759,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.420801+0000","last_change":"2024-03-11T21:33:17.733152+0000","last_active":"2024-03-11T21:33:37.420801+0000","last_peered":"2024-03-11T21:33:37.420801+0000","last_clean":"2024-03-11T21:33:37.420801+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:33:37.420801+0000","last_undegraded":"2024-03-11T21:33:37.420801+0000","last_fullsized":"2024-03-11T21:33:37.420801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:17.733068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.733068+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.427374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":745,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-11T21:33:38.394568+0000","last_change":"2024-03-11T21:33:16.013881+0000","last_active":"2024-03-11T21:33:38.394568+0000","last_peered":"2024-03-11T21:33:38.394568+0000","last_clean":"2024-03-11T21:33:38.394568+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:33:38.394568+0000","last_undegraded":"2024-03-11T21:33:38.394568+0000","last_fullsized":"2024-03-11T21:33:38.394568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:33:16.013752+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:33:16.013752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.774437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":734,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.386623+0000","last_change":"2024-03-11T21:31:58.673043+0000","last_active":"2024-03-11T21:33:37.386623+0000","last_peered":"2024-03-11T21:33:37.386623+0000","last_clean":"2024-03-11T21:33:37.386623+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:33:37.386623+0000","last_undegraded":"2024-03-11T21:33:37.386623+0000","last_fullsized":"2024-03-11T21:33:37.386623+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.672796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.672796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":712,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-11T21:33:37.421211+0000","last_change":"2024-03-11T21:32:10.686985+0000","last_active":"2024-03-11T21:33:37.421211+0000","last_peered":"2024-03-11T21:33:37.421211+0000","last_clean":"2024-03-11T21:33:37.421211+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:33:37.421211+0000","last_undegraded":"2024-03-11T21:33:37.421211+0000","last_fullsized":"2024-03-11T21:33:37.421211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.686767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.686767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":274,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195684,"num_objects":144,"num_object_clones":2,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":571,"num_write_kb":12662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26320896,"data_stored":24587052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1825,"ondisk_log_size":1825,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841674,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78908,"kb_used_data":11624,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666244,"statfs":{"total":95995035648,"available":95914233856,"internally_reserved":0,"allocated":11902976,"data_stored":9073645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168723,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86900,"kb_used_data":11284,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658252,"statfs":{"total":95995035648,"available":95906050048,"internally_reserved":0,"allocated":11554816,"data_stored":8876863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352346,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47272,"kb_used_data":6856,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93697880,"statfs":{"total":95995035648,"available":95946629120,"internally_reserved":0,"allocated":7020544,"data_stored":4308384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82668,"kb_used_data":15384,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93662484,"statfs":{"total":95995035648,"available":95910383616,"internally_reserved":0,"allocated":15753216,"data_stored":13072282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378813,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70024,"kb_used_data":6888,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93675128,"statfs":{"total":95995035648,"available":95923331072,"internally_reserved":0,"allocated":7053312,"data_stored":4287153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86632,"kb_used_data":6868,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93658520,"statfs":{"total":95995035648,"available":95906324480,"internally_reserved":0,"allocated":7032832,"data_stored":4286620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855527,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78156,"kb_used_data":10860,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666996,"statfs":{"total":95995035648,"available":95915003904,"internally_reserved":0,"allocated":11120640,"data_stored":8288196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495773,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86500,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658652,"statfs":{"total":95995035648,"available":95906459648,"internally_reserved":0,"allocated":11157504,"data_stored":8480750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":8194697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4000689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4366336,"data_stored":4194730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4161536,"data_stored":4000390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4476928,"data_stored":4194952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":274,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:33:45.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:45.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:45.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.813 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:45.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:45.936 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 -- 172.21.15.143:0/70564351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f0106fd0 msgr2=0x7f21f01073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 --2- 172.21.15.143:0/70564351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f0106fd0 0x7f21f01073b0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f21e800b0a0 tx=0x7f21e800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 -- 172.21.15.143:0/70564351 shutdown_connections 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 --2- 172.21.15.143:0/70564351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0102270 0x7f21f010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 --2- 172.21.15.143:0/70564351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f0106fd0 0x7f21f01073b0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 --2- 172.21.15.143:0/70564351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f01018b0 0x7f21f0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 -- 172.21.15.143:0/70564351 >> 172.21.15.143:0/70564351 conn(0x7f21f0076000 msgr2=0x7f21f0076410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 -- 172.21.15.143:0/70564351 shutdown_connections 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.056+0000 7f21f5c98700 1 -- 172.21.15.143:0/70564351 wait complete. 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 librados: starting msgr at 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 librados: starting objecter 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 Processor -- start 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- start start 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 librados: setting wanted keys 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 librados: calling monclient init 2024-03-11T21:33:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f0102270 0x7f21f0102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0106fd0 0x7f21f01034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0110680 con 0x7f21f0106fd0 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0110500 con 0x7f21f0102270 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0110800 con 0x7f21f01018b0 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58344/0 (socket says 172.21.15.143:58344) 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 -- 172.21.15.143:0/799702545 learned_addr learned my addr 172.21.15.143:0/799702545 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:46.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0106fd0 0x7f21f01034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f0102270 msgr2=0x7f21f0102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f0102270 0x7f21f0102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f0102270 0x7f21f0102f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0106fd0 msgr2=0x7f21f01034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0106fd0 0x7f21f01034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f01a61c0 con 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f0106fd0 0x7f21f01034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f21e800b070 tx=0x7f21e8004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 10 client.?.objecter ms_handle_connect 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e8021070 con 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f21e8003c00 con 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f01a6450 con 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e80046c0 con 0x7f21f01018b0 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 10 client.41582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f01a6900 con 0x7f21f01018b0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 librados: init done 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f21f004faa0 con 0x7f21f01018b0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f21e8007980 con 0x7f21f01018b0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 0x7f21dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(424..424 src has 87..424) v4 ==== 9880+0+0 (secure 0 0 0) 0x7f21e800f020 con 0x7f21f01018b0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 osd_map(424..424 src has 87..424) v4 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 3 client.41582.objecter handle_osd_map got epochs [424,424] > 0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21ecff9700 3 client.41582.objecter handle_osd_map decoding full epoch 424 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.060+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 0x7f21dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 20 client.41582.objecter dump_active .. 0 homeless 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v424) v1 ==== 54+0+16 (secure 0 0 0) 0x7f21e802a030 con 0x7f21f01018b0 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 10 client.41582.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 1 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f21f01093a0 0x7f21f0109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 1 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 0 1 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 4 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 10 client.41582.objecter _send_command 1 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01a9db0 con 0x7f21f01093a0 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21dc06f820 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 0x7f21dc071ce0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f21f01057e0 tx=0x7f21d8005cf0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f21f01093a0 0x7f21f0109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01093a0 2024-03-11T21:33:46.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f21f01093a0 0x7f21f0109780 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:46.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01a9db0 con 0x7f21f01093a0 2024-03-11T21:33:46.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:33:46.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 10 client.41582.objecter _finish_command 1 = system:0 2024-03-11T21:33:46.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 4 2024-03-11T21:33:46.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.064+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 0 1 2024-03-11T21:33:46.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 10 client.41582.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 2 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 2 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f0103f60 osd=0 3 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 0 2 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 4 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 10 client.41582.objecter _send_command 2 2024-03-11T21:33:46.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.084+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f0108e20 con 0x7f21f01093a0 2024-03-11T21:33:46.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f0108e20 con 0x7f21f01093a0 2024-03-11T21:33:46.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 10 client.41582.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 4 2024-03-11T21:33:46.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 0 2 2024-03-11T21:33:46.065 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 10 client.41582.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f21f002c6b0 0x7f21f002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f21f002c6b0 0x7f21f002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 1 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 4 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 10 client.41582.objecter _send_command 3 2024-03-11T21:33:46.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f0063070 con 0x7f21f002c6b0 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f002c6b0 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f21f002c6b0 0x7f21f002ca90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f0063070 con 0x7f21f002c6b0 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 10 client.41582.objecter _finish_command 3 = system:0 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 4 2024-03-11T21:33:46.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.088+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 1 3 2024-03-11T21:33:46.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 10 client.41582.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 4 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 4 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f00629f0 osd=1 3 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 1 4 2024-03-11T21:33:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 4 2024-03-11T21:33:46.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 10 client.41582.objecter _send_command 4 2024-03-11T21:33:46.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.104+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f0062470 con 0x7f21f002c6b0 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f0062470 con 0x7f21f002c6b0 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 10 client.41582.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 4 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 1 4 2024-03-11T21:33:46.085 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 10 client.41582.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 5 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f21f01abf70 0x7f21f01ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 5 2024-03-11T21:33:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 2 5 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 4 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 10 client.41582.objecter _send_command 5 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01b11c0 con 0x7f21f01abf70 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f21f01abf70 0x7f21f01ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01abf70 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f21f01abf70 0x7f21f01ae360 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01b11c0 con 0x7f21f01abf70 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 10 client.41582.objecter _finish_command 5 = system:0 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 4 2024-03-11T21:33:46.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.108+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 2 5 2024-03-11T21:33:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 10 client.41582.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 6 2024-03-11T21:33:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 6 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f0062870 osd=2 3 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 2 6 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 4 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 10 client.41582.objecter _send_command 6 2024-03-11T21:33:46.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f002d220 con 0x7f21f01abf70 2024-03-11T21:33:46.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f002d220 con 0x7f21f01abf70 2024-03-11T21:33:46.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 10 client.41582.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 4 2024-03-11T21:33:46.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 2 6 2024-03-11T21:33:46.107 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 10 client.41582.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 7 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f21f01a9910 0x7f21f01afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 7 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 3 7 2024-03-11T21:33:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 4 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 10 client.41582.objecter _send_command 7 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01b4b20 con 0x7f21f01a9910 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f21f01a9910 0x7f21f01afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01a9910 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.128+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f21f01a9910 0x7f21f01afa20 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.132+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01b4b20 con 0x7f21f01a9910 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.132+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.132+0000 7f21ecff9700 10 client.41582.objecter _finish_command 7 = system:0 2024-03-11T21:33:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.132+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 4 2024-03-11T21:33:46.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.132+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 3 7 2024-03-11T21:33:46.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 10 client.41582.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 8 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 8 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f002dcb0 osd=3 3 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 3 8 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 4 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 10 client.41582.objecter _send_command 8 2024-03-11T21:33:46.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.148+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f01a8e80 con 0x7f21f01a9910 2024-03-11T21:33:46.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f01a8e80 con 0x7f21f01a9910 2024-03-11T21:33:46.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 10 client.41582.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 4 2024-03-11T21:33:46.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 3 8 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 10 client.41582.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 9 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f21f01af580 0x7f21f01b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 9 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 4 9 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 4 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 10 client.41582.objecter _send_command 9 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01a8df0 con 0x7f21f01af580 2024-03-11T21:33:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f21f01af580 0x7f21f01b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01af580 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f21f01af580 0x7f21f01b3380 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01a8df0 con 0x7f21f01af580 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 10 client.41582.objecter _finish_command 9 = system:0 2024-03-11T21:33:46.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 4 2024-03-11T21:33:46.132 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.152+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 4 9 2024-03-11T21:33:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 10 client.41582.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 10 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 10 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01a8c30 osd=4 3 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 4 10 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 4 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 10 client.41582.objecter _send_command 10 2024-03-11T21:33:46.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.172+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f01a8df0 con 0x7f21f01af580 2024-03-11T21:33:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f01a8df0 con 0x7f21f01af580 2024-03-11T21:33:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 10 client.41582.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 4 2024-03-11T21:33:46.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 4 10 2024-03-11T21:33:46.153 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:33:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 10 client.41582.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 11 2024-03-11T21:33:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f21f01b3dd0 0x7f21f01b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 11 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 5 11 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 4 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 10 client.41582.objecter _send_command 11 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01bb660 con 0x7f21f01b3dd0 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f21f01b3dd0 0x7f21f01b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01b3dd0 2024-03-11T21:33:46.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.176+0000 7f21eeffd700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f21f01b3dd0 0x7f21f01b4230 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.180+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01bb660 con 0x7f21f01b3dd0 2024-03-11T21:33:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.180+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:33:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.180+0000 7f21ecff9700 10 client.41582.objecter _finish_command 11 = system:0 2024-03-11T21:33:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.180+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 4 2024-03-11T21:33:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.180+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 5 11 2024-03-11T21:33:46.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 10 client.41582.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 12 2024-03-11T21:33:46.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 12 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01a8df0 osd=5 3 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 5 12 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 4 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 10 client.41582.objecter _send_command 12 2024-03-11T21:33:46.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.196+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f01b6ce0 con 0x7f21f01b3dd0 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f01b6ce0 con 0x7f21f01b3dd0 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 10 client.41582.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 4 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 5 12 2024-03-11T21:33:46.177 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 10 client.41582.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 13 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 0x7f21f01b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 13 2024-03-11T21:33:46.178 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 6 13 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 4 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 10 client.41582.objecter _send_command 13 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01b7e50 con 0x7f21f01b7400 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 0x7f21f01b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01b7400 2024-03-11T21:33:46.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.200+0000 7f21effff700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 0x7f21f01b7860 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.204+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01b7e50 con 0x7f21f01b7400 2024-03-11T21:33:46.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.204+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:33:46.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.204+0000 7f21ecff9700 10 client.41582.objecter _finish_command 13 = system:0 2024-03-11T21:33:46.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.204+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 4 2024-03-11T21:33:46.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.204+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 6 13 2024-03-11T21:33:46.189 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:46.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 10 client.41582.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 14 2024-03-11T21:33:46.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 14 2024-03-11T21:33:46.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01b6ce0 osd=6 3 2024-03-11T21:33:46.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 6 14 2024-03-11T21:33:46.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 4 2024-03-11T21:33:46.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 10 client.41582.objecter _send_command 14 2024-03-11T21:33:46.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.220+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f01b7e50 con 0x7f21f01b7400 2024-03-11T21:33:46.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f01b7e50 con 0x7f21f01b7400 2024-03-11T21:33:46.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ecff9700 10 client.41582.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 4 2024-03-11T21:33:46.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 6 14 2024-03-11T21:33:46.203 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:33:46.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 10 client.41582.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:33:46.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 15 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 0x7f21f01bea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 15 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 7 15 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 4 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 10 client.41582.objecter _send_command 15 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f01c1ad0 con 0x7f21f01b8060 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.224+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 0x7f21f01bea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 10 client.41582.objecter ms_handle_connect 0x7f21f01b8060 2024-03-11T21:33:46.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ef7fe700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 0x7f21f01bea50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f01c1ad0 con 0x7f21f01b8060 2024-03-11T21:33:46.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:33:46.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 10 client.41582.objecter _finish_command 15 = system:0 2024-03-11T21:33:46.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 4 2024-03-11T21:33:46.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.228+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 7 15 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/2021387878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 msgr2=0x7f6368108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- 172.21.15.143:0/2021387878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f6368108310 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f635c009990 tx=0x7f635c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/2021387878 shutdown_connections 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- 172.21.15.143:0/2021387878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f6368108310 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- 172.21.15.143:0/2021387878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368102880 0x7f636810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- 172.21.15.143:0/2021387878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368101ec0 0x7f6368102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/2021387878 >> 172.21.15.143:0/2021387878 conn(0x7f6368078eb0 msgr2=0x7f63680792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/2021387878 shutdown_connections 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/2021387878 wait complete. 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 librados: starting msgr at 2024-03-11T21:33:46.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 librados: starting objecter 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 Processor -- start 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- start start 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 librados: setting wanted keys 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 librados: calling monclient init 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368101ec0 0x7f6368199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f636819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63681116e0 con 0x7f6368107f30 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6368111560 con 0x7f6368101ec0 2024-03-11T21:33:46.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6368111860 con 0x7f6368102880 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f636819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636d28b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636d28b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58356/0 (socket says 172.21.15.143:58356) 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636d28b700 1 -- 172.21.15.143:0/700324215 learned_addr learned my addr 172.21.15.143:0/700324215 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636da8c700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368101ec0 0x7f6368199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 -- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 msgr2=0x7f636819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 -- 172.21.15.143:0/700324215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368101ec0 msgr2=0x7f6368199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368101ec0 0x7f6368199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 -- 172.21.15.143:0/700324215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636819eeb0 con 0x7f6368107f30 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636d28b700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636e28d700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f636819e770 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f636000be20 tx=0x7f6360009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f635affd700 10 client.?.objecter ms_handle_connect 0x7f6368107f30 2024-03-11T21:33:46.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f635affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6360019070 con 0x7f6368107f30 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636819f1a0 con 0x7f6368107f30 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f636000cc10 con 0x7f6368107f30 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63600125a0 con 0x7f6368107f30 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 10 client.30387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63681a6a20 con 0x7f6368107f30 2024-03-11T21:33:46.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.236+0000 7f636fcf0700 1 librados: init done 2024-03-11T21:33:46.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636804faa0 con 0x7f6368107f30 2024-03-11T21:33:46.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f636001c070 con 0x7f6368107f30 2024-03-11T21:33:46.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 0x7f6354071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(424..424 src has 87..424) v4 ==== 9880+0+0 (secure 0 0 0) 0x7f636008e4d0 con 0x7f6368107f30 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f636da8c700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 0x7f6354071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 10 client.30387.objecter ms_dispatch 0x7f6368001550 osd_map(424..424 src has 87..424) v4 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 3 client.30387.objecter handle_osd_map got epochs [424,424] > 0 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 3 client.30387.objecter handle_osd_map decoding full epoch 424 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 20 client.30387.objecter dump_active .. 0 homeless 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f635affd700 10 client.30387.objecter ms_handle_connect 0x7f635406f820 2024-03-11T21:33:46.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.240+0000 7f636da8c700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 0x7f6354071ce0 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b2e0 tx=0x7f635c000bc0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:33:46.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.244+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f636005b020 con 0x7f6368107f30 2024-03-11T21:33:46.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 10 client.41582.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:33:46.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign -1 16 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 20 client.41582.objecter _get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_remove -1 16 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 20 client.41582.objecter get_session s=0x7f21f01b7e50 osd=7 3 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 15 client.41582.objecter _session_command_op_assign 7 16 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 4 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 10 client.41582.objecter _send_command 16 2024-03-11T21:33:46.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.248+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21f01c1b30 con 0x7f21f01b8060 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21ecff9700 1 -- 172.21.15.143:0/799702545 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21f01c1b30 con 0x7f21f01b8060 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21ecff9700 10 client.41582.objecter ms_dispatch 0x7f21f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21ecff9700 10 client.41582.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21ecff9700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 4 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21ecff9700 15 client.41582.objecter _session_command_op_remove 7 16 2024-03-11T21:33:46.232 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:33:46.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 librados: watch_flush enter 2024-03-11T21:33:46.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 librados: watch_flush exit 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.0 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f21f01093a0 msgr2=0x7f21f0109780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f21f01093a0 0x7f21f0109780 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0103f60 osd=0 1 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.1 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f21f002c6b0 msgr2=0x7f21f002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f21f002c6b0 0x7f21f002ca90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f00629f0 osd=1 1 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.2 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f21f01abf70 msgr2=0x7f21f01ae360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f21f01abf70 0x7f21f01ae360 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f0062870 osd=2 1 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.3 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f21f01a9910 msgr2=0x7f21f01afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f21f01a9910 0x7f21f01afa20 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f002dcb0 osd=3 1 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.4 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f21f01af580 msgr2=0x7f21f01b3380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f21f01af580 0x7f21f01b3380 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8c30 osd=4 1 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.5 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21effff700 1 -- 172.21.15.143:0/799702545 reap_dead start 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f21f01b3dd0 msgr2=0x7f21f01b4230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f21f01b3dd0 0x7f21f01b4230 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21effff700 1 -- 172.21.15.143:0/799702545 reap_dead start 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01a8df0 osd=5 1 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.6 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 msgr2=0x7f21f01b7860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21effff700 1 -- 172.21.15.143:0/799702545 reap_dead start 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 0x7f21f01b7860 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b6ce0 osd=6 1 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter close_session for osd.7 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 msgr2=0x7f21f01bea50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 0x7f21f01bea50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter put_session s=0x7f21f01b7e50 osd=7 1 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 20 client.41582.objecter shutdown clearing up homeless session... 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 10 client.41582.objecter successfully canceled tick 2024-03-11T21:33:46.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 msgr2=0x7f21dc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 0x7f21dc071ce0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f21f01057e0 tx=0x7f21d8005cf0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 msgr2=0x7f21f0102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f21e800b070 tx=0x7f21e8004ce0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 shutdown_connections 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f21f01b8060 0x7f21f01bea50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f21f01b7400 0x7f21f01b7860 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21f01018b0 0x7f21f0102a30 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 --2- 172.21.15.143:0/799702545 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f21dc06f820 0x7f21dc071ce0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 >> 172.21.15.143:0/799702545 conn(0x7f21f0076000 msgr2=0x7f21f0100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 shutdown_connections 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 -- 172.21.15.143:0/799702545 wait complete. 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.256+0000 7f21f5c98700 1 librados: shutdown 2024-03-11T21:33:46.237 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:46.238 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:46.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:46.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:46.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:46.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.354923+0000 mon.a (mon.0) 2585 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.364409+0000 mgr.y (mgr.14550) 429 : cluster 0 pgmap v624: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.519343+0000 osd.3 (osd.3) 263 : cluster 0 5.13 scrub starts 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.520620+0000 osd.3 (osd.3) 264 : cluster 0 5.13 scrub ok 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.918715+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:45.919914+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: audit 2024-03-11T21:33:46.066823+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.143:0/799702545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.310441+0000 mon.a (mon.0) 2586 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.354923+0000 mon.a (mon.0) 2585 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.364409+0000 mgr.y (mgr.14550) 429 : cluster 0 pgmap v624: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.519343+0000 osd.3 (osd.3) 263 : cluster 0 5.13 scrub starts 2024-03-11T21:33:46.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.520620+0000 osd.3 (osd.3) 264 : cluster 0 5.13 scrub ok 2024-03-11T21:33:46.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.918715+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:33:46.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:45.919914+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:33:46.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: audit 2024-03-11T21:33:46.066823+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.143:0/799702545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:46.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:46 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.310441+0000 mon.a (mon.0) 2586 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:46.439 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:46.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:46.488+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f63681a6c50 con 0x7f6368107f30 2024-03-11T21:33:46.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.563 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:46.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:46.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.354923+0000 mon.a (mon.0) 2585 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.364409+0000 mgr.y (mgr.14550) 429 : cluster 0 pgmap v624: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.519343+0000 osd.3 (osd.3) 263 : cluster 0 5.13 scrub starts 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.520620+0000 osd.3 (osd.3) 264 : cluster 0 5.13 scrub ok 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.918715+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:45.919914+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: audit 2024-03-11T21:33:46.066823+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.143:0/799702545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:46.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:46 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.310441+0000 mon.a (mon.0) 2586 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:46.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:46.825 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:46.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:47.058 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:47.086 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:47.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:47.217 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:47.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:47.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.356+0000 7f635affd700 1 -- 172.21.15.143:0/700324215 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v426) v1 ==== 87+0+0 (secure 0 0 0) 0x7f636005ea70 con 0x7f6368107f30 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 10 librados: watch_flush enter 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 10 librados: watch_flush exit 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 20 client.30387.objecter shutdown clearing up homeless session... 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 10 client.30387.objecter successfully canceled tick 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 msgr2=0x7f6354071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 0x7f6354071ce0 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b2e0 tx=0x7f635c000bc0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 msgr2=0x7f636819e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f636819e770 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f636000be20 tx=0x7f6360009ed0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 shutdown_connections 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f635406f820 0x7f6354071ce0 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6368107f30 0x7f636819e770 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6368101ec0 0x7f6368199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 --2- 172.21.15.143:0/700324215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6368102880 0x7f636819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 >> 172.21.15.143:0/700324215 conn(0x7f6368078eb0 msgr2=0x7f6368074010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 shutdown_connections 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 -- 172.21.15.143:0/700324215 wait complete. 2024-03-11T21:33:47.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:33:47.360+0000 7f636fcf0700 1 librados: shutdown 2024-03-11T21:33:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.364 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:47.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:47.366 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:33:47.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.057174+0000 osd.2 (osd.2) 211 : cluster 0 6.14 scrub starts 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.058734+0000 osd.2 (osd.2) 212 : cluster 0 6.14 scrub ok 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.135822+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.137670+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.362269+0000 mon.a (mon.0) 2587 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: audit 2024-03-11T21:33:46.364091+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: audit 2024-03-11T21:33:46.365474+0000 mon.c (mon.2) 886 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: audit 2024-03-11T21:33:46.374359+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: audit 2024-03-11T21:33:46.377421+0000 mon.a (mon.0) 2589 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.386263+0000 osd.6 (osd.6) 191 : cluster 0 3.a scrub starts 2024-03-11T21:33:47.474 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.395660+0000 osd.6 (osd.6) 192 : cluster 0 3.a scrub ok 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[21512]: audit 2024-03-11T21:33:46.495713+0000 mon.a (mon.0) 2590 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.057174+0000 osd.2 (osd.2) 211 : cluster 0 6.14 scrub starts 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.058734+0000 osd.2 (osd.2) 212 : cluster 0 6.14 scrub ok 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.135822+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.137670+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.362269+0000 mon.a (mon.0) 2587 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: audit 2024-03-11T21:33:46.364091+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: audit 2024-03-11T21:33:46.365474+0000 mon.c (mon.2) 886 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: audit 2024-03-11T21:33:46.374359+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: audit 2024-03-11T21:33:46.377421+0000 mon.a (mon.0) 2589 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.386263+0000 osd.6 (osd.6) 191 : cluster 0 3.a scrub starts 2024-03-11T21:33:47.475 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.395660+0000 osd.6 (osd.6) 192 : cluster 0 3.a scrub ok 2024-03-11T21:33:47.476 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:47 smithi143 bash[29284]: audit 2024-03-11T21:33:46.495713+0000 mon.a (mon.0) 2590 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:47.484 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:47.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:47.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.057174+0000 osd.2 (osd.2) 211 : cluster 0 6.14 scrub starts 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.058734+0000 osd.2 (osd.2) 212 : cluster 0 6.14 scrub ok 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.135822+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.137670+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.362269+0000 mon.a (mon.0) 2587 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:33:47.616 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: audit 2024-03-11T21:33:46.364091+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: audit 2024-03-11T21:33:46.365474+0000 mon.c (mon.2) 886 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: audit 2024-03-11T21:33:46.374359+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: audit 2024-03-11T21:33:46.377421+0000 mon.a (mon.0) 2589 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.386263+0000 osd.6 (osd.6) 191 : cluster 0 3.a scrub starts 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.395660+0000 osd.6 (osd.6) 192 : cluster 0 3.a scrub ok 2024-03-11T21:33:47.617 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:47 smithi203 bash[23654]: audit 2024-03-11T21:33:46.495713+0000 mon.a (mon.0) 2590 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:47.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:47.745 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:47.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:47.877 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:47.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:48.008 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:48.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.138 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:48.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.269 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:48.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.467507+0000 osd.5 (osd.5) 265 : cluster 0 2.16 scrub starts 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: cluster 2024-03-11T21:33:46.468643+0000 osd.5 (osd.5) 266 : cluster 0 2.16 scrub ok 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: cluster 2024-03-11T21:33:47.358334+0000 mon.a (mon.0) 2591 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.360624+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.360710+0000 mon.a (mon.0) 2593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.360766+0000 mon.a (mon.0) 2594 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: cluster 2024-03-11T21:33:47.365051+0000 mgr.y (mgr.14550) 430 : cluster 0 pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: cluster 2024-03-11T21:33:47.367221+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:33:48.383 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.376421+0000 mon.c (mon.2) 887 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.389392+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.415736+0000 mon.c (mon.2) 888 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[29284]: audit 2024-03-11T21:33:47.416346+0000 mon.a (mon.0) 2597 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.467507+0000 osd.5 (osd.5) 265 : cluster 0 2.16 scrub starts 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: cluster 2024-03-11T21:33:46.468643+0000 osd.5 (osd.5) 266 : cluster 0 2.16 scrub ok 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: cluster 2024-03-11T21:33:47.358334+0000 mon.a (mon.0) 2591 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:48.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.360624+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.360710+0000 mon.a (mon.0) 2593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.360766+0000 mon.a (mon.0) 2594 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: cluster 2024-03-11T21:33:47.365051+0000 mgr.y (mgr.14550) 430 : cluster 0 pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: cluster 2024-03-11T21:33:47.367221+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.376421+0000 mon.c (mon.2) 887 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.389392+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.415736+0000 mon.c (mon.2) 888 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:48 smithi143 bash[21512]: audit 2024-03-11T21:33:47.416346+0000 mon.a (mon.0) 2597 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:48.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:48.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:48.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:48.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.467507+0000 osd.5 (osd.5) 265 : cluster 0 2.16 scrub starts 2024-03-11T21:33:48.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: cluster 2024-03-11T21:33:46.468643+0000 osd.5 (osd.5) 266 : cluster 0 2.16 scrub ok 2024-03-11T21:33:48.635 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: cluster 2024-03-11T21:33:47.358334+0000 mon.a (mon.0) 2591 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.360624+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.360710+0000 mon.a (mon.0) 2593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.360766+0000 mon.a (mon.0) 2594 : audit 1 from='client.? 172.21.15.143:0/700324215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: cluster 2024-03-11T21:33:47.365051+0000 mgr.y (mgr.14550) 430 : cluster 0 pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: cluster 2024-03-11T21:33:47.367221+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.376421+0000 mon.c (mon.2) 887 : audit 1 from='client.? 172.21.15.143:0/217687934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.389392+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]: dispatch 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.415736+0000 mon.c (mon.2) 888 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.636 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:48 smithi203 bash[23654]: audit 2024-03-11T21:33:47.416346+0000 mon.a (mon.0) 2597 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:33:48.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:33:48.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:48.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:48.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.900 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:49.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:49.028 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:49.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.160 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:49.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:49.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:49.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:33:49.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.388175+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:49.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.388255+0000 mon.a (mon.0) 2599 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: cluster 2024-03-11T21:33:48.394728+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.395935+0000 mon.c (mon.2) 889 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.399378+0000 mon.a (mon.0) 2601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.431128+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.431525+0000 mon.a (mon.0) 2602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.432240+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.432549+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.433062+0000 mon.c (mon.2) 892 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.433387+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.407 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:49 smithi203 bash[23654]: audit 2024-03-11T21:33:48.588475+0000 mgr.y (mgr.14550) 431 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:49.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:49.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.388175+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.388255+0000 mon.a (mon.0) 2599 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: cluster 2024-03-11T21:33:48.394728+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.395935+0000 mon.c (mon.2) 889 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.399378+0000 mon.a (mon.0) 2601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.431128+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.431525+0000 mon.a (mon.0) 2602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.432240+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.432549+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.433062+0000 mon.c (mon.2) 892 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.433387+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[21512]: audit 2024-03-11T21:33:48.588475+0000 mgr.y (mgr.14550) 431 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.388175+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi143-71604-64"}]': finished 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.388255+0000 mon.a (mon.0) 2599 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: cluster 2024-03-11T21:33:48.394728+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.395935+0000 mon.c (mon.2) 889 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.537 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.399378+0000 mon.a (mon.0) 2601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.431128+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.431525+0000 mon.a (mon.0) 2602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.432240+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.432549+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.433062+0000 mon.c (mon.2) 892 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.433387+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:49.538 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:49 smithi143 bash[29284]: audit 2024-03-11T21:33:48.588475+0000 mgr.y (mgr.14550) 431 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:49.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:49.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:49.678 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:49.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.809 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:49.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:49.941 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:50.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.069 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:50.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:50.199 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:50.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:33:50.331 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:50.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.445 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: cluster 2024-03-11T21:33:49.365842+0000 mgr.y (mgr.14550) 432 : cluster 0 pgmap v629: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.391166+0000 mon.a (mon.0) 2605 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.391359+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: cluster 2024-03-11T21:33:49.401350+0000 mon.a (mon.0) 2607 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.403379+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.403944+0000 mon.c (mon.2) 894 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.415621+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.446 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:50 smithi203 bash[23654]: audit 2024-03-11T21:33:49.416824+0000 mon.a (mon.0) 2609 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.462 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:50.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.573 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: cluster 2024-03-11T21:33:49.365842+0000 mgr.y (mgr.14550) 432 : cluster 0 pgmap v629: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:50.573 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.391166+0000 mon.a (mon.0) 2605 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:33:50.573 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.391359+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:50.573 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: cluster 2024-03-11T21:33:49.401350+0000 mon.a (mon.0) 2607 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.403379+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.403944+0000 mon.c (mon.2) 894 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.415621+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[21512]: audit 2024-03-11T21:33:49.416824+0000 mon.a (mon.0) 2609 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: cluster 2024-03-11T21:33:49.365842+0000 mgr.y (mgr.14550) 432 : cluster 0 pgmap v629: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.391166+0000 mon.a (mon.0) 2605 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.391359+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi143-71604-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:50.574 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: cluster 2024-03-11T21:33:49.401350+0000 mon.a (mon.0) 2607 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:33:50.575 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.403379+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.575 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.403944+0000 mon.c (mon.2) 894 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.575 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.415621+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]: dispatch 2024-03-11T21:33:50.575 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:50 smithi143 bash[29284]: audit 2024-03-11T21:33:49.416824+0000 mon.a (mon.0) 2609 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:50.584 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:50.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.707 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:50.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.837 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:50.930 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:50.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:50.967 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:51.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:51.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:51.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:51.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:51.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:51.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: cluster 2024-03-11T21:33:50.407892+0000 mon.a (mon.0) 2610 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:51.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: audit 2024-03-11T21:33:50.419138+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]': finished 2024-03-11T21:33:51.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: cluster 2024-03-11T21:33:50.866751+0000 mon.a (mon.0) 2612 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:33:51.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: audit 2024-03-11T21:33:51.317012+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:51.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: cluster 2024-03-11T21:33:51.329958+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:33:51.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: audit 2024-03-11T21:33:51.360077+0000 mon.a (mon.0) 2615 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:51.471 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:51 smithi203 bash[23654]: audit 2024-03-11T21:33:51.363299+0000 mon.c (mon.2) 895 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:51.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:51.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:51.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: cluster 2024-03-11T21:33:50.407892+0000 mon.a (mon.0) 2610 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: audit 2024-03-11T21:33:50.419138+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]': finished 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: cluster 2024-03-11T21:33:50.866751+0000 mon.a (mon.0) 2612 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: audit 2024-03-11T21:33:51.317012+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: cluster 2024-03-11T21:33:51.329958+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: audit 2024-03-11T21:33:51.360077+0000 mon.a (mon.0) 2615 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:51.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[21512]: audit 2024-03-11T21:33:51.363299+0000 mon.c (mon.2) 895 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: cluster 2024-03-11T21:33:50.407892+0000 mon.a (mon.0) 2610 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: audit 2024-03-11T21:33:50.419138+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-49", "mode": "readproxy"}]': finished 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: cluster 2024-03-11T21:33:50.866751+0000 mon.a (mon.0) 2612 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: audit 2024-03-11T21:33:51.317012+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi143-71604-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: cluster 2024-03-11T21:33:51.329958+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: audit 2024-03-11T21:33:51.360077+0000 mon.a (mon.0) 2615 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:33:51.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:51 smithi143 bash[29284]: audit 2024-03-11T21:33:51.363299+0000 mon.c (mon.2) 895 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:51.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.744 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:51.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:51.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:51.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:52.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:52.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:52.134 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:52.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:52.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:52.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:52.380 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:33:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:52.397 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:52.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:52.525 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:52.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:52.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[29284]: cluster 2024-03-11T21:33:51.366487+0000 mgr.y (mgr.14550) 433 : cluster 0 pgmap v633: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[29284]: cluster 2024-03-11T21:33:52.327731+0000 mon.a (mon.0) 2616 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[29284]: cluster 2024-03-11T21:33:52.355443+0000 mon.a (mon.0) 2617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[21512]: cluster 2024-03-11T21:33:51.366487+0000 mgr.y (mgr.14550) 433 : cluster 0 pgmap v633: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[21512]: cluster 2024-03-11T21:33:52.327731+0000 mon.a (mon.0) 2616 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:33:52.637 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:52 smithi143 bash[21512]: cluster 2024-03-11T21:33:52.355443+0000 mon.a (mon.0) 2617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:52.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:52.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:52 smithi203 bash[23654]: cluster 2024-03-11T21:33:51.366487+0000 mgr.y (mgr.14550) 433 : cluster 0 pgmap v633: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:33:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:52 smithi203 bash[23654]: cluster 2024-03-11T21:33:52.327731+0000 mon.a (mon.0) 2616 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:33:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:52 smithi203 bash[23654]: cluster 2024-03-11T21:33:52.355443+0000 mon.a (mon.0) 2617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:33:52.773 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:52.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:52.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:52.966 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:53.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:53.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:53.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:53.156 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:53.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:53.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:53.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:53.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:53.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:53.680 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:53.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:53.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:53.942 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:54.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.073 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:54.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:54.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:54.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:54 smithi203 bash[23654]: cluster 2024-03-11T21:33:53.332178+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:33:54.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:54 smithi203 bash[23654]: audit 2024-03-11T21:33:53.335122+0000 mon.c (mon.2) 896 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:54 smithi203 bash[23654]: audit 2024-03-11T21:33:53.336143+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:54 smithi203 bash[23654]: cluster 2024-03-11T21:33:53.372961+0000 mgr.y (mgr.14550) 434 : cluster 0 pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:54.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:54.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:54.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:54.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:54.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:54.677 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:54.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[21512]: cluster 2024-03-11T21:33:53.332178+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:33:54.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[21512]: audit 2024-03-11T21:33:53.335122+0000 mon.c (mon.2) 896 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[21512]: audit 2024-03-11T21:33:53.336143+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[21512]: cluster 2024-03-11T21:33:53.372961+0000 mgr.y (mgr.14550) 434 : cluster 0 pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[29284]: cluster 2024-03-11T21:33:53.332178+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[29284]: audit 2024-03-11T21:33:53.335122+0000 mon.c (mon.2) 896 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[29284]: audit 2024-03-11T21:33:53.336143+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:54.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:54 smithi143 bash[29284]: cluster 2024-03-11T21:33:53.372961+0000 mgr.y (mgr.14550) 434 : cluster 0 pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:54.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.732 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:54.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:55.024 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:55.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:55.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:55.245 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:55.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[21512]: audit 2024-03-11T21:33:54.331259+0000 mon.a (mon.0) 2620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[21512]: cluster 2024-03-11T21:33:54.338954+0000 mon.a (mon.0) 2621 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[21512]: audit 2024-03-11T21:33:54.339044+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[21512]: audit 2024-03-11T21:33:54.340065+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[29284]: audit 2024-03-11T21:33:54.331259+0000 mon.a (mon.0) 2620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[29284]: cluster 2024-03-11T21:33:54.338954+0000 mon.a (mon.0) 2621 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:33:55.357 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[29284]: audit 2024-03-11T21:33:54.339044+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.358 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:55 smithi143 bash[29284]: audit 2024-03-11T21:33:54.340065+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.369 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:55.472 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:55.495 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:55.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:55 smithi203 bash[23654]: audit 2024-03-11T21:33:54.331259+0000 mon.a (mon.0) 2620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:55.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:55 smithi203 bash[23654]: cluster 2024-03-11T21:33:54.338954+0000 mon.a (mon.0) 2621 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:33:55.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:55 smithi203 bash[23654]: audit 2024-03-11T21:33:54.339044+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.143:0/3881119509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:55 smithi203 bash[23654]: audit 2024-03-11T21:33:54.340065+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]: dispatch 2024-03-11T21:33:55.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:55.626 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:55.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:55.758 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:55.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.890 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:55.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:56.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.149 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:56.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:56.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:56.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:56.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.341604+0000 mon.a (mon.0) 2623 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: cluster 2024-03-11T21:33:55.346855+0000 mon.a (mon.0) 2624 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.372092+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.372470+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.373302+0000 mon.c (mon.2) 899 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: cluster 2024-03-11T21:33:55.373546+0000 mgr.y (mgr.14550) 435 : cluster 0 pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.373675+0000 mon.a (mon.0) 2626 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.374178+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[29284]: audit 2024-03-11T21:33:55.374812+0000 mon.a (mon.0) 2627 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.341604+0000 mon.a (mon.0) 2623 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: cluster 2024-03-11T21:33:55.346855+0000 mon.a (mon.0) 2624 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.372092+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.372470+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.373302+0000 mon.c (mon.2) 899 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: cluster 2024-03-11T21:33:55.373546+0000 mgr.y (mgr.14550) 435 : cluster 0 pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.373675+0000 mon.a (mon.0) 2626 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.374178+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:56 smithi143 bash[21512]: audit 2024-03-11T21:33:55.374812+0000 mon.a (mon.0) 2627 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:56.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:56.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.666 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:56.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:56.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.341604+0000 mon.a (mon.0) 2623 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi143-71604-65"}]': finished 2024-03-11T21:33:56.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: cluster 2024-03-11T21:33:55.346855+0000 mon.a (mon.0) 2624 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:33:56.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.372092+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.372470+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.373302+0000 mon.c (mon.2) 899 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: cluster 2024-03-11T21:33:55.373546+0000 mgr.y (mgr.14550) 435 : cluster 0 pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.373675+0000 mon.a (mon.0) 2626 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.374178+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:56 smithi203 bash[23654]: audit 2024-03-11T21:33:55.374812+0000 mon.a (mon.0) 2627 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:33:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:56.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:57.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:57.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:57.294 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:57.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:57.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:57.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:57.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:57.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:57.805 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:57.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[29284]: audit 2024-03-11T21:33:56.350083+0000 mon.a (mon.0) 2628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[29284]: cluster 2024-03-11T21:33:56.354814+0000 mon.a (mon.0) 2629 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[29284]: audit 2024-03-11T21:33:56.356364+0000 mon.c (mon.2) 901 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[29284]: audit 2024-03-11T21:33:56.356768+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[21512]: audit 2024-03-11T21:33:56.350083+0000 mon.a (mon.0) 2628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[21512]: cluster 2024-03-11T21:33:56.354814+0000 mon.a (mon.0) 2629 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[21512]: audit 2024-03-11T21:33:56.356364+0000 mon.c (mon.2) 901 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:57.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:57 smithi143 bash[21512]: audit 2024-03-11T21:33:56.356768+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:57.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.962 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:57 smithi203 bash[23654]: audit 2024-03-11T21:33:56.350083+0000 mon.a (mon.0) 2628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi143-71604-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:33:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:57 smithi203 bash[23654]: cluster 2024-03-11T21:33:56.354814+0000 mon.a (mon.0) 2629 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:33:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:57 smithi203 bash[23654]: audit 2024-03-11T21:33:56.356364+0000 mon.c (mon.2) 901 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:57 smithi203 bash[23654]: audit 2024-03-11T21:33:56.356768+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:33:58.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:58.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:58.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:58.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:58.362 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:58.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:58.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:58.627 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:58.639 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:33:58.639 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:33:58.639 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:33:58.639 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:33:58.640 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:33:58.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:58.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:58.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:33:58.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: cluster 2024-03-11T21:33:57.374005+0000 mgr.y (mgr.14550) 436 : cluster 0 pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: cluster 2024-03-11T21:33:57.784599+0000 mon.a (mon.0) 2631 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.022348+0000 mon.c (mon.2) 902 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.022637+0000 mon.c (mon.2) 903 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.022847+0000 mon.c (mon.2) 904 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.022916+0000 mon.a (mon.0) 2632 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.023010+0000 mon.c (mon.2) 905 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.023175+0000 mon.c (mon.2) 906 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.023357+0000 mon.c (mon.2) 907 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.023541+0000 mon.c (mon.2) 908 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.023680+0000 mon.a (mon.0) 2633 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.024166+0000 mon.a (mon.0) 2634 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.024492+0000 mon.a (mon.0) 2635 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.024809+0000 mon.a (mon.0) 2636 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.025131+0000 mon.a (mon.0) 2637 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.025449+0000 mon.a (mon.0) 2638 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.362548+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.362771+0000 mon.a (mon.0) 2640 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.362956+0000 mon.a (mon.0) 2641 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.363137+0000 mon.a (mon.0) 2642 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.363313+0000 mon.a (mon.0) 2643 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.363505+0000 mon.a (mon.0) 2644 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.363693+0000 mon.a (mon.0) 2645 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: audit 2024-03-11T21:33:58.363891+0000 mon.a (mon.0) 2646 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-11T21:33:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:58 smithi203 bash[23654]: cluster 2024-03-11T21:33:58.371467+0000 mon.a (mon.0) 2647 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:33:58.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: cluster 2024-03-11T21:33:57.374005+0000 mgr.y (mgr.14550) 436 : cluster 0 pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: cluster 2024-03-11T21:33:57.784599+0000 mon.a (mon.0) 2631 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.022348+0000 mon.c (mon.2) 902 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.022637+0000 mon.c (mon.2) 903 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.022847+0000 mon.c (mon.2) 904 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.022916+0000 mon.a (mon.0) 2632 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.023010+0000 mon.c (mon.2) 905 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.023175+0000 mon.c (mon.2) 906 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.023357+0000 mon.c (mon.2) 907 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.023541+0000 mon.c (mon.2) 908 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.023680+0000 mon.a (mon.0) 2633 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.024166+0000 mon.a (mon.0) 2634 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.024492+0000 mon.a (mon.0) 2635 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.024809+0000 mon.a (mon.0) 2636 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.025131+0000 mon.a (mon.0) 2637 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.025449+0000 mon.a (mon.0) 2638 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: cluster 2024-03-11T21:33:57.374005+0000 mgr.y (mgr.14550) 436 : cluster 0 pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: cluster 2024-03-11T21:33:57.784599+0000 mon.a (mon.0) 2631 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.022348+0000 mon.c (mon.2) 902 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.022637+0000 mon.c (mon.2) 903 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.022847+0000 mon.c (mon.2) 904 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.022916+0000 mon.a (mon.0) 2632 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.023010+0000 mon.c (mon.2) 905 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.023175+0000 mon.c (mon.2) 906 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.023357+0000 mon.c (mon.2) 907 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.023541+0000 mon.c (mon.2) 908 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.023680+0000 mon.a (mon.0) 2633 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.024166+0000 mon.a (mon.0) 2634 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]: dispatch 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.024492+0000 mon.a (mon.0) 2635 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]: dispatch 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.024809+0000 mon.a (mon.0) 2636 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.025131+0000 mon.a (mon.0) 2637 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.025449+0000 mon.a (mon.0) 2638 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]: dispatch 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.362548+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.362771+0000 mon.a (mon.0) 2640 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.362956+0000 mon.a (mon.0) 2641 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]': finished 2024-03-11T21:33:58.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.363137+0000 mon.a (mon.0) 2642 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.363313+0000 mon.a (mon.0) 2643 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.363505+0000 mon.a (mon.0) 2644 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.363693+0000 mon.a (mon.0) 2645 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: audit 2024-03-11T21:33:58.363891+0000 mon.a (mon.0) 2646 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[21512]: cluster 2024-03-11T21:33:58.371467+0000 mon.a (mon.0) 2647 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.362548+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi143-71604-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.362771+0000 mon.a (mon.0) 2640 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.362956+0000 mon.a (mon.0) 2641 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.18", "id": [7, 2]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.363137+0000 mon.a (mon.0) 2642 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1d", "id": [7, 2]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.363313+0000 mon.a (mon.0) 2643 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1e", "id": [7, 4]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.363505+0000 mon.a (mon.0) 2644 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-11T21:33:58.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.363693+0000 mon.a (mon.0) 2645 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-11T21:33:58.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: audit 2024-03-11T21:33:58.363891+0000 mon.a (mon.0) 2646 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.17", "id": [3, 1]}]': finished 2024-03-11T21:33:58.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:58 smithi143 bash[29284]: cluster 2024-03-11T21:33:58.371467+0000 mon.a (mon.0) 2647 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:33:58.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:59.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:59.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:59.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.278 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:59.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:59.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:59.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:59.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:59.657 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:59.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:59.785 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:59.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:59.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:00.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:59 smithi203 bash[23654]: audit 2024-03-11T21:33:58.599843+0000 mgr.y (mgr.14550) 437 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:00.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:33:59 smithi203 bash[23654]: cluster 2024-03-11T21:33:59.381672+0000 mon.a (mon.0) 2648 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:34:00.048 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:00.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:00.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:00.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:59 smithi143 bash[29284]: audit 2024-03-11T21:33:58.599843+0000 mgr.y (mgr.14550) 437 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:00.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:33:59 smithi143 bash[29284]: cluster 2024-03-11T21:33:59.381672+0000 mon.a (mon.0) 2648 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:34:00.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:59 smithi143 bash[21512]: audit 2024-03-11T21:33:58.599843+0000 mgr.y (mgr.14550) 437 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:00.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:33:59 smithi143 bash[21512]: cluster 2024-03-11T21:33:59.381672+0000 mon.a (mon.0) 2648 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:34:00.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:00.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:00.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:00.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:00.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.573 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:00.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:00.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:00.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[21512]: cluster 2024-03-11T21:33:59.375432+0000 mgr.y (mgr.14550) 438 : cluster 0 pgmap v645: 268 pgs: 2 peering, 2 creating+peering, 6 unknown, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:00.365382+0000 mon.a (mon.0) 2649 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:00.382619+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[21512]: audit 2024-03-11T21:34:00.387110+0000 mon.c (mon.2) 909 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[21512]: audit 2024-03-11T21:34:00.389454+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[29284]: cluster 2024-03-11T21:33:59.375432+0000 mgr.y (mgr.14550) 438 : cluster 0 pgmap v645: 268 pgs: 2 peering, 2 creating+peering, 6 unknown, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:00.365382+0000 mon.a (mon.0) 2649 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:00.382619+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:34:00.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[29284]: audit 2024-03-11T21:34:00.387110+0000 mon.c (mon.2) 909 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:00.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:00 smithi143 bash[29284]: audit 2024-03-11T21:34:00.389454+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:00.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:00.968 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:00 smithi203 bash[23654]: cluster 2024-03-11T21:33:59.375432+0000 mgr.y (mgr.14550) 438 : cluster 0 pgmap v645: 268 pgs: 2 peering, 2 creating+peering, 6 unknown, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:00.365382+0000 mon.a (mon.0) 2649 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:00.382619+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:34:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:00 smithi203 bash[23654]: audit 2024-03-11T21:34:00.387110+0000 mon.c (mon.2) 909 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:00 smithi203 bash[23654]: audit 2024-03-11T21:34:00.389454+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:01.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:01.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.227 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:01.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:01.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:01.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:01.484 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:01.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:01.615 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:01.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.746 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:01.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.877 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:00.946778+0000 mon.c (mon.2) 910 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:00.947538+0000 mon.a (mon.0) 2652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.320533+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.320701+0000 mon.a (mon.0) 2654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: cluster 2024-03-11T21:34:01.327328+0000 mon.a (mon.0) 2655 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.329433+0000 mon.c (mon.2) 911 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.331801+0000 mon.a (mon.0) 2656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.331854+0000 mon.c (mon.2) 912 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[29284]: audit 2024-03-11T21:34:01.332929+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:00.946778+0000 mon.c (mon.2) 910 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:00.947538+0000 mon.a (mon.0) 2652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.320533+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.320701+0000 mon.a (mon.0) 2654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: cluster 2024-03-11T21:34:01.327328+0000 mon.a (mon.0) 2655 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.329433+0000 mon.c (mon.2) 911 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.331801+0000 mon.a (mon.0) 2656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.331854+0000 mon.c (mon.2) 912 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:01.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:01 smithi143 bash[21512]: audit 2024-03-11T21:34:01.332929+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:01.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:02.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:00.946778+0000 mon.c (mon.2) 910 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:02.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:00.947538+0000 mon.a (mon.0) 2652 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.320533+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.320701+0000 mon.a (mon.0) 2654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: cluster 2024-03-11T21:34:01.327328+0000 mon.a (mon.0) 2655 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.329433+0000 mon.c (mon.2) 911 : audit 1 from='client.? 172.21.15.143:0/2843625864' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.331801+0000 mon.a (mon.0) 2656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]: dispatch 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.331854+0000 mon.c (mon.2) 912 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:02.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:01 smithi203 bash[23654]: audit 2024-03-11T21:34:01.332929+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:02.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:02.139 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:02.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:02.268 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout:P2 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3016 ms) 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6978 ms) 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7037 ms) 2024-03-11T21:34:02.322 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6943 ms) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7084 ms) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6953 ms) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6989 ms) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (138612 ms total) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (326768 ms total) 2024-03-11T21:34:02.323 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-11T21:34:02.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.381 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:02.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:02.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:02.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:02.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:02.655 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:02.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:02.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:02.888 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.915 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:03.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:34:03.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:03.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: cluster 2024-03-11T21:34:01.376250+0000 mgr.y (mgr.14550) 439 : cluster 0 pgmap v648: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:03.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: cluster 2024-03-11T21:34:02.321627+0000 mon.a (mon.0) 2658 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.323812+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.323996+0000 mon.a (mon.0) 2660 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: cluster 2024-03-11T21:34:02.331144+0000 mon.a (mon.0) 2661 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.417705+0000 mon.c (mon.2) 913 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.418140+0000 mon.a (mon.0) 2662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.418822+0000 mon.c (mon.2) 914 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:02 smithi203 bash[23654]: audit 2024-03-11T21:34:02.419300+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.070 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:03.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: cluster 2024-03-11T21:34:01.376250+0000 mgr.y (mgr.14550) 439 : cluster 0 pgmap v648: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: cluster 2024-03-11T21:34:02.321627+0000 mon.a (mon.0) 2658 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.323812+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.323996+0000 mon.a (mon.0) 2660 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: cluster 2024-03-11T21:34:02.331144+0000 mon.a (mon.0) 2661 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.417705+0000 mon.c (mon.2) 913 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.418140+0000 mon.a (mon.0) 2662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.418822+0000 mon.c (mon.2) 914 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[21512]: audit 2024-03-11T21:34:02.419300+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: cluster 2024-03-11T21:34:01.376250+0000 mgr.y (mgr.14550) 439 : cluster 0 pgmap v648: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: cluster 2024-03-11T21:34:02.321627+0000 mon.a (mon.0) 2658 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.323812+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi143-71604-66"}]': finished 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.323996+0000 mon.a (mon.0) 2660 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]': finished 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: cluster 2024-03-11T21:34:02.331144+0000 mon.a (mon.0) 2661 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.417705+0000 mon.c (mon.2) 913 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.418140+0000 mon.a (mon.0) 2662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.418822+0000 mon.c (mon.2) 914 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:02 smithi143 bash[29284]: audit 2024-03-11T21:34:02.419300+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-49"}]: dispatch 2024-03-11T21:34:03.201 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout::::163:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:5d165639:::164:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f43765fc:::165:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b4c720e9:::166:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e694b040:::167:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:afa38db2:::168:head 2024-03-11T21:34:03.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:77ba9f53:::169:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:87495034:::170:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7c96bf0e:::171:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:dbe346cc:::172:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:e943ec24:::173:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f97a9c0c:::174:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:6f26e74d:::175:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4f95e106:::176:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:0e6f2f8f:::177:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:05db05f1:::178:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:38a78d66:::179:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:d095610b:::180:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:a1a9d709:::181:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:1e5d39db:::182:head 2024-03-11T21:34:03.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:f7df4fb9:::183:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:03a7f161:::184:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ba70721e:::185:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:28e5662d:::186:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:973d52de:::187:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:4303eb1c:::188:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:b990b48e:::189:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:29b8165b:::190:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:3547f197:::191:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:7e260936:::192:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:1abec7b1:::193:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:10fdda93:::194:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:15817eea:::195:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:770bab57:::196:head 2024-03-11T21:34:03.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:ed9e13e7:::197:head 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:71471a8f:::198:head 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 266:10fb1d02:::199:head 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8049 ms) 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first is 1710192798 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,0 2024-03-11T21:34:03.333 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,1710192806,1710192807,0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,1710192806,1710192807,0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192798,1710192800,1710192801,1710192803,1710192804,1710192806,1710192807,0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710192801,1710192803,1710192804,1710192806,1710192807,1710192809,1710192810,0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first now 1710192801, trimmed 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20960 ms) 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: foo0 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14041 ms) 2024-03-11T21:34:03.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-11T21:34:03.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:03.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:03.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:03.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:03.735 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:03.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.864 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:03.993 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:04.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.123 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:04.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.254 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:04.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:04.367 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:04 smithi203 bash[23654]: cluster 2024-03-11T21:34:03.345895+0000 mon.a (mon.0) 2664 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:34:04.368 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:04 smithi203 bash[23654]: cluster 2024-03-11T21:34:03.377331+0000 mgr.y (mgr.14550) 440 : cluster 0 pgmap v651: 228 pgs: 2 peering, 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:04.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:04.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:04.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:04.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- 172.21.15.143:0/4162563904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac108dd0 msgr2=0x7f0fac1152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:04.602 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- 172.21.15.143:0/4162563904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac108dd0 0x7f0fac1152e0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b0a0 tx=0x7f0fa400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- 172.21.15.143:0/4162563904 shutdown_connections 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- 172.21.15.143:0/4162563904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac10e4a0 0x7f0fac10e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- 172.21.15.143:0/4162563904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac108dd0 0x7f0fac1152e0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- 172.21.15.143:0/4162563904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108410 0x7f0fac108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- 172.21.15.143:0/4162563904 >> 172.21.15.143:0/4162563904 conn(0x7f0fac0755b0 msgr2=0x7f0fac0779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- 172.21.15.143:0/4162563904 shutdown_connections 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- 172.21.15.143:0/4162563904 wait complete. 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 librados: starting msgr at 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 librados: starting objecter 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 Processor -- start 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- start start 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 librados: setting wanted keys 2024-03-11T21:34:04.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 librados: calling monclient init 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 0x7f0fac19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108dd0 0x7f0fac19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac117bf0 con 0x7f0fac108410 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac117a70 con 0x7f0fac108dd0 2024-03-11T21:34:04.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb457f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac117d70 con 0x7f0fac10e4a0 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb231b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 0x7f0fac19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43778/0 (socket says 172.21.15.143:43778) 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb231b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 0x7f0fac19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35498/0 (socket says 172.21.15.143:35498) 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 -- 172.21.15.143:0/3901793263 learned_addr learned my addr 172.21.15.143:0/3901793263 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:04.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb1b1a700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108dd0 0x7f0fac19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 -- 172.21.15.143:0/3901793263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108dd0 msgr2=0x7f0fac19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108dd0 0x7f0fac19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 -- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 msgr2=0x7f0fac19e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 0x7f0fac19e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 -- 172.21.15.143:0/3901793263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac1a3150 con 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fb2b1c700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b070 tx=0x7f0fa4007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fa37fe700 10 client.?.objecter ms_handle_connect 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fa37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.624+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4021020 con 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0fa4004c50 con 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac1a33e0 con 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40052b0 con 0x7f0fac10e4a0 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb457f700 10 client.41606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:04.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac1aacc0 con 0x7f0fac10e4a0 2024-03-11T21:34:04.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb457f700 1 librados: init done 2024-03-11T21:34:04.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac04faa0 con 0x7f0fac10e4a0 2024-03-11T21:34:04.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0fa4024020 con 0x7f0fac10e4a0 2024-03-11T21:34:04.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 0x7f0f98071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:04.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(443..443 src has 87..443) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f0fa400f020 con 0x7f0fac10e4a0 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb231b700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 0x7f0f98071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 10 client.41606.objecter ms_dispatch 0x7f0fac001550 osd_map(443..443 src has 87..443) v4 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 3 client.41606.objecter handle_osd_map got epochs [443,443] > 0 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 3 client.41606.objecter handle_osd_map decoding full epoch 443 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 20 client.41606.objecter dump_active .. 0 homeless 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fa37fe700 10 client.41606.objecter ms_handle_connect 0x7f0f9806f820 2024-03-11T21:34:04.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.628+0000 7f0fb231b700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 0x7f0f98071ce0 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009be0 tx=0x7f0fa8009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:04.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.632+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0fa405c920 con 0x7f0fac10e4a0 2024-03-11T21:34:04.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:04.638 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:04.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:04 smithi143 bash[29284]: cluster 2024-03-11T21:34:03.345895+0000 mon.a (mon.0) 2664 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:34:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:04 smithi143 bash[29284]: cluster 2024-03-11T21:34:03.377331+0000 mgr.y (mgr.14550) 440 : cluster 0 pgmap v651: 228 pgs: 2 peering, 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:04 smithi143 bash[21512]: cluster 2024-03-11T21:34:03.345895+0000 mon.a (mon.0) 2664 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:34:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:04 smithi143 bash[21512]: cluster 2024-03-11T21:34:03.377331+0000 mgr.y (mgr.14550) 440 : cluster 0 pgmap v651: 228 pgs: 2 peering, 226 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:04.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.768 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:04.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.844+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0fac0760a0 con 0x7f0f9806f820 2024-03-11T21:34:04.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.864+0000 7f0fa37fe700 1 -- 172.21.15.143:0/3901793263 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+662706 (secure 0 0 0) 0x7f0fac0760a0 con 0x7f0f9806f820 2024-03-11T21:34:04.850 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:34:04.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 10 librados: watch_flush enter 2024-03-11T21:34:04.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 10 librados: watch_flush exit 2024-03-11T21:34:04.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 20 client.41606.objecter shutdown clearing up homeless session... 2024-03-11T21:34:04.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 10 client.41606.objecter successfully canceled tick 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 msgr2=0x7f0f98071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 0x7f0f98071ce0 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009be0 tx=0x7f0fa8009260 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 msgr2=0x7f0fac1a2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b070 tx=0x7f0fa4007a20 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 shutdown_connections 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f9806f820 0x7f0f98071ce0 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fac108410 0x7f0fac19e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fac10e4a0 0x7f0fac1a2a10 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 --2- 172.21.15.143:0/3901793263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0fac108dd0 0x7f0fac19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 >> 172.21.15.143:0/3901793263 conn(0x7f0fac0755b0 msgr2=0x7f0fac075990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 shutdown_connections 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 -- 172.21.15.143:0/3901793263 wait complete. 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:04.872+0000 7f0fb457f700 1 librados: shutdown 2024-03-11T21:34:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:34:04.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:04.890 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:04.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.018 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:05.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:05.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:05.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:05.388 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[21512]: cluster 2024-03-11T21:34:04.363095+0000 mon.a (mon.0) 2665 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:34:05.388 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[21512]: audit 2024-03-11T21:34:04.369891+0000 mon.c (mon.2) 915 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[21512]: audit 2024-03-11T21:34:04.371142+0000 mon.a (mon.0) 2666 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[21512]: audit 2024-03-11T21:34:04.849522+0000 mgr.y (mgr.14550) 441 : audit 0 from='client.41606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[29284]: cluster 2024-03-11T21:34:04.363095+0000 mon.a (mon.0) 2665 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[29284]: audit 2024-03-11T21:34:04.369891+0000 mon.c (mon.2) 915 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[29284]: audit 2024-03-11T21:34:04.371142+0000 mon.a (mon.0) 2666 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.389 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:05 smithi143 bash[29284]: audit 2024-03-11T21:34:04.849522+0000 mgr.y (mgr.14550) 441 : audit 0 from='client.41606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:05.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:05.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:05.530 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:05.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:05.659 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:05.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:05 smithi203 bash[23654]: cluster 2024-03-11T21:34:04.363095+0000 mon.a (mon.0) 2665 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:34:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:05 smithi203 bash[23654]: audit 2024-03-11T21:34:04.369891+0000 mon.c (mon.2) 915 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:05 smithi203 bash[23654]: audit 2024-03-11T21:34:04.371142+0000 mon.a (mon.0) 2666 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:05 smithi203 bash[23654]: audit 2024-03-11T21:34:04.849522+0000 mgr.y (mgr.14550) 441 : audit 0 from='client.41606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:05.790 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:05.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:05.919 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:06.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:06.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:06.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:06.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:06.319 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:06.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: audit 2024-03-11T21:34:05.354928+0000 mon.a (mon.0) 2667 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: cluster 2024-03-11T21:34:05.364280+0000 mon.a (mon.0) 2668 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: cluster 2024-03-11T21:34:05.378495+0000 mgr.y (mgr.14550) 442 : cluster 0 pgmap v654: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: audit 2024-03-11T21:34:05.421584+0000 mon.c (mon.2) 916 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: audit 2024-03-11T21:34:05.421963+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:06 smithi203 bash[23654]: cluster 2024-03-11T21:34:06.319731+0000 mon.a (mon.0) 2670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:06.450 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:06.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:06.583 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:06.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: audit 2024-03-11T21:34:05.354928+0000 mon.a (mon.0) 2667 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: cluster 2024-03-11T21:34:05.364280+0000 mon.a (mon.0) 2668 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: cluster 2024-03-11T21:34:05.378495+0000 mgr.y (mgr.14550) 442 : cluster 0 pgmap v654: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: audit 2024-03-11T21:34:05.421584+0000 mon.c (mon.2) 916 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: audit 2024-03-11T21:34:05.421963+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[21512]: cluster 2024-03-11T21:34:06.319731+0000 mon.a (mon.0) 2670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: audit 2024-03-11T21:34:05.354928+0000 mon.a (mon.0) 2667 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: cluster 2024-03-11T21:34:05.364280+0000 mon.a (mon.0) 2668 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: cluster 2024-03-11T21:34:05.378495+0000 mgr.y (mgr.14550) 442 : cluster 0 pgmap v654: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:06.605 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: audit 2024-03-11T21:34:05.421584+0000 mon.c (mon.2) 916 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: audit 2024-03-11T21:34:05.421963+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:34:06.606 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:06 smithi143 bash[29284]: cluster 2024-03-11T21:34:06.319731+0000 mon.a (mon.0) 2670 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:06.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.713 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:06.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:06.841 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:06.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:06.973 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:07.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:07.100 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:07.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:07.231 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:07.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3319911191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 msgr2=0x7f7f6010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3319911191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f6010db70 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800b0a0 tx=0x7f7f5800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3319911191 shutdown_connections 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3319911191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60106dc0 0x7f7f601071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3319911191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f6010db70 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3319911191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60100d40 0x7f7f601011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3319911191 >> 172.21.15.143:0/3319911191 conn(0x7f7f60076d80 msgr2=0x7f7f60077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3319911191 shutdown_connections 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3319911191 wait complete. 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 librados: starting msgr at 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 librados: starting objecter 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 Processor -- start 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- start start 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 librados: setting wanted keys 2024-03-11T21:34:07.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 librados: calling monclient init 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f60198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60106dc0 0x7f7f601a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60110530 con 0x7f7f60100d40 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f601103b0 con 0x7f7f60106dc0 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f601106b0 con 0x7f7f60101700 2024-03-11T21:34:07.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f65d8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f60198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35518/0 (socket says 172.21.15.143:35518) 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f65d8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f60198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43798/0 (socket says 172.21.15.143:43798) 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 -- 172.21.15.143:0/3437733835 learned_addr learned my addr 172.21.15.143:0/3437733835 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60106dc0 0x7f7f601a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 msgr2=0x7f7f60198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f60198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60106dc0 msgr2=0x7f7f601a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60106dc0 0x7f7f601a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f601a3f00 con 0x7f7f60100d40 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f60101700 0x7f7f60198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60106dc0 0x7f7f601a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9a0 tx=0x7f7f5c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f577fe700 10 client.?.objecter ms_handle_connect 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f577fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00cbd0 con 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f601a4130 con 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7f5c00cd70 con 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c010e60 con 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 10 client.30411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f601a46c0 con 0x7f7f60100d40 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.260+0000 7f7f687f3700 1 librados: init done 2024-03-11T21:34:07.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7f6004faa0 con 0x7f7f60100d40 2024-03-11T21:34:07.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7f5c010430 con 0x7f7f60100d40 2024-03-11T21:34:07.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 0x7f7f4c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(445..445 src has 87..445) v4 ==== 9621+0+0 (secure 0 0 0) 0x7f7f5c08f4a0 con 0x7f7f60100d40 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 osd_map(445..445 src has 87..445) v4 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 3 client.30411.objecter handle_osd_map got epochs [445,445] > 0 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 3 client.30411.objecter handle_osd_map decoding full epoch 445 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 0x7f7f4c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 20 client.30411.objecter dump_active .. 0 homeless 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v445) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7f5c08f8f0 con 0x7f7f60100d40 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f4c06f820 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 0x7f7f4c071ce0 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f7f60106870 tx=0x7f7f5000b3c0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 10 client.30411.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 1 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7f600629f0 0x7f7f60062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 1 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 0 1 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 4 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 10 client.30411.objecter _send_command 1 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601a7ab0 con 0x7f7f600629f0 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7f600629f0 0x7f7f60062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7f600629f0 0x7f7f60062dd0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f600629f0 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601a7ab0 con 0x7f7f600629f0 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:34:07.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 10 client.30411.objecter _finish_command 1 = system:0 2024-03-11T21:34:07.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 4 2024-03-11T21:34:07.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.264+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 0 1 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 10 client.30411.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 2 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 2 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6006a120 osd=0 3 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 0 2 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 4 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 10 client.30411.objecter _send_command 2 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f60062470 con 0x7f7f600629f0 2024-03-11T21:34:07.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f60062470 con 0x7f7f600629f0 2024-03-11T21:34:07.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 10 client.30411.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 4 2024-03-11T21:34:07.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 0 2 2024-03-11T21:34:07.263 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 10 client.30411.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7f6002c6b0 0x7f7f6002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 1 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 4 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 10 client.30411.objecter _send_command 3 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601087b0 con 0x7f7f6002c6b0 2024-03-11T21:34:07.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7f6002c6b0 0x7f7f6002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f6002c6b0 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.284+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7f6002c6b0 0x7f7f6002ca90 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.288+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601087b0 con 0x7f7f6002c6b0 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.288+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.288+0000 7f7f577fe700 10 client.30411.objecter _finish_command 3 = system:0 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.288+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 4 2024-03-11T21:34:07.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.288+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 1 3 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 10 client.30411.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 4 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 4 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f60108130 osd=1 3 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 1 4 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 4 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 10 client.30411.objecter _send_command 4 2024-03-11T21:34:07.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f60107bb0 con 0x7f7f6002c6b0 2024-03-11T21:34:07.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f60107bb0 con 0x7f7f6002c6b0 2024-03-11T21:34:07.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f577fe700 10 client.30411.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 4 2024-03-11T21:34:07.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.304+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 1 4 2024-03-11T21:34:07.284 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:34:07.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 10 client.30411.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 5 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7f60196730 0x7f7f60196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 5 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 2 5 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 4 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 10 client.30411.objecter _send_command 5 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601b1210 con 0x7f7f60196730 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7f60196730 0x7f7f60196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f60196730 2024-03-11T21:34:07.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7f60196730 0x7f7f60196b70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601b1210 con 0x7f7f60196730 2024-03-11T21:34:07.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:34:07.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 10 client.30411.objecter _finish_command 5 = system:0 2024-03-11T21:34:07.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 4 2024-03-11T21:34:07.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.308+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 2 5 2024-03-11T21:34:07.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 10 client.30411.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 6 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 6 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6002d7a0 osd=2 3 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 2 6 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 4 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 10 client.30411.objecter _send_command 6 2024-03-11T21:34:07.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.324+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f6002d220 con 0x7f7f60196730 2024-03-11T21:34:07.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f6002d220 con 0x7f7f60196730 2024-03-11T21:34:07.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 10 client.30411.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 4 2024-03-11T21:34:07.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 2 6 2024-03-11T21:34:07.305 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:34:07.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 10 client.30411.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 7 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7f601a7610 0x7f7f601970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 7 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 3 7 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 4 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 10 client.30411.objecter _send_command 7 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601b4b70 con 0x7f7f601a7610 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7f601a7610 0x7f7f601970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f601a7610 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7f601a7610 0x7f7f601970b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601b4b70 con 0x7f7f601a7610 2024-03-11T21:34:07.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:34:07.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 10 client.30411.objecter _finish_command 7 = system:0 2024-03-11T21:34:07.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 4 2024-03-11T21:34:07.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.328+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 3 7 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 10 client.30411.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 8 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 8 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f6002d620 osd=3 3 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 3 8 2024-03-11T21:34:07.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 4 2024-03-11T21:34:07.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 10 client.30411.objecter _send_command 8 2024-03-11T21:34:07.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f601a6b80 con 0x7f7f601a7610 2024-03-11T21:34:07.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f601a6b80 con 0x7f7f601a7610 2024-03-11T21:34:07.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 10 client.30411.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 4 2024-03-11T21:34:07.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 3 8 2024-03-11T21:34:07.327 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:34:07.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 10 client.30411.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 9 2024-03-11T21:34:07.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7f601b0d70 0x7f7f601b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 9 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 4 9 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 4 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 10 client.30411.objecter _send_command 9 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601a6af0 con 0x7f7f601b0d70 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7f601b0d70 0x7f7f601b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7f601b0d70 0x7f7f601b53e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.348+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f601b0d70 2024-03-11T21:34:07.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.352+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601a6af0 con 0x7f7f601b0d70 2024-03-11T21:34:07.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.352+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:34:07.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.352+0000 7f7f577fe700 10 client.30411.objecter _finish_command 9 = system:0 2024-03-11T21:34:07.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.352+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 4 2024-03-11T21:34:07.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.352+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 4 9 2024-03-11T21:34:07.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:07.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 10 client.30411.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 10 2024-03-11T21:34:07.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 10 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6930 osd=4 3 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 4 10 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 4 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 10 client.30411.objecter _send_command 10 2024-03-11T21:34:07.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.368+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f601a6af0 con 0x7f7f601b0d70 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f601a6af0 con 0x7f7f601b0d70 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 10 client.30411.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 4 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 4 10 2024-03-11T21:34:07.350 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 10 client.30411.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 11 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7f601b5e30 0x7f7f601b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 11 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 5 11 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 4 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 10 client.30411.objecter _send_command 11 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601bb240 con 0x7f7f601b5e30 2024-03-11T21:34:07.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7f601b5e30 0x7f7f601b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f601b5e30 2024-03-11T21:34:07.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.372+0000 7f7f65d8e700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7f601b5e30 0x7f7f601b8220 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.376+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601bb240 con 0x7f7f601b5e30 2024-03-11T21:34:07.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.376+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:34:07.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.376+0000 7f7f577fe700 10 client.30411.objecter _finish_command 11 = system:0 2024-03-11T21:34:07.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.376+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 4 2024-03-11T21:34:07.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.376+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 5 11 2024-03-11T21:34:07.354 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:07.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 10 client.30411.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 12 2024-03-11T21:34:07.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 12 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6d40 osd=5 3 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 5 12 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 4 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 10 client.30411.objecter _send_command 12 2024-03-11T21:34:07.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.396+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f601b9fd0 con 0x7f7f601b5e30 2024-03-11T21:34:07.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f601b9fd0 con 0x7f7f601b5e30 2024-03-11T21:34:07.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f577fe700 10 client.30411.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 4 2024-03-11T21:34:07.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 5 12 2024-03-11T21:34:07.382 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:34:07.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 10 client.30411.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 13 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7f601ba6f0 0x7f7f601bab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 13 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 6 13 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 4 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 10 client.30411.objecter _send_command 13 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.404+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601be650 con 0x7f7f601ba6f0 2024-03-11T21:34:07.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.408+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7f601ba6f0 0x7f7f601bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.408+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f601ba6f0 2024-03-11T21:34:07.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.408+0000 7f7f66d90700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7f601ba6f0 0x7f7f601bab30 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.412+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601be650 con 0x7f7f601ba6f0 2024-03-11T21:34:07.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.412+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:34:07.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.412+0000 7f7f577fe700 10 client.30411.objecter _finish_command 13 = system:0 2024-03-11T21:34:07.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.412+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 4 2024-03-11T21:34:07.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.412+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 6 13 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 10 client.30411.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 14 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 14 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601a6bb0 osd=6 3 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 6 14 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 4 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 10 client.30411.objecter _send_command 14 2024-03-11T21:34:07.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.432+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f601bd490 con 0x7f7f601ba6f0 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f601bd490 con 0x7f7f601ba6f0 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 10 client.30411.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 4 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 6 14 2024-03-11T21:34:07.413 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:34:07.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 10 client.30411.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:07.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 15 2024-03-11T21:34:07.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7f601bdbb0 0x7f7f601bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 15 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 7 15 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 4 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 10 client.30411.objecter _send_command 15 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f601c19f0 con 0x7f7f601bdbb0 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7f601bdbb0 0x7f7f601bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f577fe700 10 client.30411.objecter ms_handle_connect 0x7f7f601bdbb0 2024-03-11T21:34:07.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.436+0000 7f7f6658f700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7f601bdbb0 0x7f7f601bdfd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.440+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7f601c19f0 con 0x7f7f601bdbb0 2024-03-11T21:34:07.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.440+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:34:07.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.440+0000 7f7f577fe700 10 client.30411.objecter _finish_command 15 = system:0 2024-03-11T21:34:07.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.440+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 4 2024-03-11T21:34:07.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.440+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 7 15 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 10 client.30411.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign -1 16 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 20 client.30411.objecter _get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_remove -1 16 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 20 client.30411.objecter get_session s=0x7f7f601bd490 osd=7 3 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 15 client.30411.objecter _session_command_op_assign 7 16 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 4 2024-03-11T21:34:07.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 10 client.30411.objecter _send_command 16 2024-03-11T21:34:07.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.456+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7f601b8800 con 0x7f7f601bdbb0 2024-03-11T21:34:07.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f577fe700 1 -- 172.21.15.143:0/3437733835 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7f601b8800 con 0x7f7f601bdbb0 2024-03-11T21:34:07.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f577fe700 10 client.30411.objecter ms_dispatch 0x7f7f60001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:07.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f577fe700 10 client.30411.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:07.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f577fe700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 4 2024-03-11T21:34:07.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f577fe700 15 client.30411.objecter _session_command_op_remove 7 16 2024-03-11T21:34:07.455 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:34:07.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 librados: watch_flush enter 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 librados: watch_flush exit 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.0 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7f600629f0 msgr2=0x7f7f60062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7f600629f0 0x7f7f60062dd0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6006a120 osd=0 1 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.1 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7f6002c6b0 msgr2=0x7f7f6002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7f6002c6b0 0x7f7f6002ca90 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f60108130 osd=1 1 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.2 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7f60196730 msgr2=0x7f7f60196b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7f60196730 0x7f7f60196b70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d7a0 osd=2 1 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.3 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7f601a7610 msgr2=0x7f7f601970b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7f601a7610 0x7f7f601970b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f6002d620 osd=3 1 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.4 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7f601b0d70 msgr2=0x7f7f601b53e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7f601b0d70 0x7f7f601b53e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6930 osd=4 1 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.5 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7f601b5e30 msgr2=0x7f7f601b8220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7f601b5e30 0x7f7f601b8220 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6d40 osd=5 1 2024-03-11T21:34:07.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.6 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7f601ba6f0 msgr2=0x7f7f601bab30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7f601ba6f0 0x7f7f601bab30 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601a6bb0 osd=6 1 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter close_session for osd.7 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7f601bdbb0 msgr2=0x7f7f601bdfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7f601bdbb0 0x7f7f601bdfd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f66d90700 1 -- 172.21.15.143:0/3437733835 reap_dead start 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter put_session s=0x7f7f601bd490 osd=7 1 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 20 client.30411.objecter shutdown clearing up homeless session... 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 10 client.30411.objecter successfully canceled tick 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f66d90700 1 -- 172.21.15.143:0/3437733835 reap_dead start 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f66d90700 1 -- 172.21.15.143:0/3437733835 reap_dead start 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f66d90700 1 -- 172.21.15.143:0/3437733835 reap_dead start 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f66d90700 1 -- 172.21.15.143:0/3437733835 reap_dead start 2024-03-11T21:34:07.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 msgr2=0x7f7f4c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 0x7f7f4c071ce0 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f7f60106870 tx=0x7f7f5000b3c0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 msgr2=0x7f7f60198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9a0 tx=0x7f7f5c00ecb0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 shutdown_connections 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f60100d40 0x7f7f60198940 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 --2- 172.21.15.143:0/3437733835 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7f4c06f820 0x7f7f4c071ce0 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.476+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 >> 172.21.15.143:0/3437733835 conn(0x7f7f60076d80 msgr2=0x7f7f600ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.480+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 shutdown_connections 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.480+0000 7f7f687f3700 1 -- 172.21.15.143:0/3437733835 wait complete. 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.480+0000 7f7f687f3700 1 librados: shutdown 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:07.460 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:07.461 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:07.461 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:07.461 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:07.461 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:07.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:07.462 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-03-11T21:34:04.356049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792089,"num_objects":344,"num_object_clones":2,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":2126,"num_read_kb":1876,"num_write":1480,"num_write_kb":15131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2827,"ondisk_log_size":2827,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":774,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":607908,"kb_used_data":83928,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749353308,"statfs":{"total":767960285184,"available":767337787392,"internally_reserved":0,"allocated":85942272,"data_stored":62780878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.979472"},"pg_stats":[{"pgid":"281.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"281.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:34:04.338127+0000","last_change":"2024-03-11T21:34:04.338127+0000","last_active":"2024-03-11T21:34:04.338127+0000","last_peered":"2024-03-11T21:34:04.338127+0000","last_clean":"2024-03-11T21:34:04.338127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:04.338127+0000","last_undegraded":"2024-03-11T21:34:04.338127+0000","last_fullsized":"2024-03-11T21:34:04.338127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:04.338127+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.338127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.9","version":"245'17","reported_seq":742,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328219+0000","last_change":"2024-03-11T21:33:39.123510+0000","last_active":"2024-03-11T21:34:02.328219+0000","last_peered":"2024-03-11T21:34:02.328219+0000","last_clean":"2024-03-11T21:34:02.328219+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:34:02.328219+0000","last_undegraded":"2024-03-11T21:34:02.328219+0000","last_fullsized":"2024-03-11T21:34:02.328219+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-03-11T21:33:39.123380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.123380+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:50.187380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001660573,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"412'20","reported_seq":726,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.400316+0000","last_change":"2024-03-11T21:32:34.299549+0000","last_active":"2024-03-11T21:34:02.400316+0000","last_peered":"2024-03-11T21:34:02.400316+0000","last_clean":"2024-03-11T21:34:02.400316+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:34:02.400316+0000","last_undegraded":"2024-03-11T21:34:02.400316+0000","last_fullsized":"2024-03-11T21:34:02.400316+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'18","last_scrub_stamp":"2024-03-11T21:32:22.035738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.035738+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076326579999999996,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"296'34","reported_seq":737,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389651+0000","last_change":"2024-03-11T21:32:34.335857+0000","last_active":"2024-03-11T21:34:02.389651+0000","last_peered":"2024-03-11T21:34:02.389651+0000","last_clean":"2024-03-11T21:34:02.389651+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:34:02.389651+0000","last_undegraded":"2024-03-11T21:34:02.389651+0000","last_fullsized":"2024-03-11T21:34:02.389651+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024159609999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"412'54","reported_seq":771,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326269+0000","last_change":"2024-03-11T21:33:20.623614+0000","last_active":"2024-03-11T21:34:02.326269+0000","last_peered":"2024-03-11T21:34:02.326269+0000","last_clean":"2024-03-11T21:34:02.326269+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:34:02.326269+0000","last_undegraded":"2024-03-11T21:34:02.326269+0000","last_fullsized":"2024-03-11T21:34:02.326269+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'52","last_scrub_stamp":"2024-03-11T21:33:20.623315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.623315+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.384819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0043833079999999998,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"412'52","reported_seq":769,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.367096+0000","last_change":"2024-03-11T21:33:28.221239+0000","last_active":"2024-03-11T21:34:02.367096+0000","last_peered":"2024-03-11T21:34:02.367096+0000","last_clean":"2024-03-11T21:34:02.367096+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:34:02.367096+0000","last_undegraded":"2024-03-11T21:34:02.367096+0000","last_fullsized":"2024-03-11T21:34:02.367096+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:33:28.221082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.221082+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.462634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022023799999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":736,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327702+0000","last_change":"2024-03-11T21:33:15.710053+0000","last_active":"2024-03-11T21:34:02.327702+0000","last_peered":"2024-03-11T21:34:02.327702+0000","last_clean":"2024-03-11T21:34:02.327702+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:34:02.327702+0000","last_undegraded":"2024-03-11T21:34:02.327702+0000","last_fullsized":"2024-03-11T21:34:02.327702+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'34","last_scrub_stamp":"2024-03-11T21:33:15.709878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.709878+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.249755+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0036499739999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"353'183","reported_seq":862,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328416+0000","last_change":"2024-03-11T21:33:14.703400+0000","last_active":"2024-03-11T21:34:02.328416+0000","last_peered":"2024-03-11T21:34:02.328416+0000","last_clean":"2024-03-11T21:34:02.328416+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:34:02.328416+0000","last_undegraded":"2024-03-11T21:34:02.328416+0000","last_fullsized":"2024-03-11T21:34:02.328416+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'183","last_scrub_stamp":"2024-03-11T21:33:14.703284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.703284+0000","objects_scrubbed":13,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:17.422601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.029727594,"stat_sum":{"num_bytes":117,"num_objects":13,"num_object_clones":1,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":55,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":805,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405919+0000","last_change":"2024-03-11T21:33:27.586701+0000","last_active":"2024-03-11T21:34:02.405919+0000","last_peered":"2024-03-11T21:34:02.405919+0000","last_clean":"2024-03-11T21:34:02.405919+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:34:02.405919+0000","last_undegraded":"2024-03-11T21:34:02.405919+0000","last_fullsized":"2024-03-11T21:34:02.405919+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:33:27.586328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.586328+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:49.263277+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.013258202,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":727,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.349515+0000","last_change":"2024-03-11T21:33:08.891202+0000","last_active":"2024-03-11T21:34:03.349515+0000","last_peered":"2024-03-11T21:34:03.349515+0000","last_clean":"2024-03-11T21:34:03.349515+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:34:03.349515+0000","last_undegraded":"2024-03-11T21:34:03.349515+0000","last_fullsized":"2024-03-11T21:34:03.349515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:33:08.891062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.891062+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.610002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0077514410000000004,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":765,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.347866+0000","last_change":"2024-03-11T21:33:14.964478+0000","last_active":"2024-03-11T21:34:03.347866+0000","last_peered":"2024-03-11T21:34:03.347866+0000","last_clean":"2024-03-11T21:34:03.347866+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:34:03.347866+0000","last_undegraded":"2024-03-11T21:34:03.347866+0000","last_fullsized":"2024-03-11T21:34:03.347866+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:33:14.964352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.964352+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.802464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076282429999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"412'120","reported_seq":798,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390412+0000","last_change":"2024-03-11T21:32:34.336215+0000","last_active":"2024-03-11T21:34:02.390412+0000","last_peered":"2024-03-11T21:34:02.390412+0000","last_clean":"2024-03-11T21:34:02.390412+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:34:02.390412+0000","last_undegraded":"2024-03-11T21:34:02.390412+0000","last_fullsized":"2024-03-11T21:34:02.390412+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026596910000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":745,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326595+0000","last_change":"2024-03-11T21:33:17.671892+0000","last_active":"2024-03-11T21:34:02.326595+0000","last_peered":"2024-03-11T21:34:02.326595+0000","last_clean":"2024-03-11T21:34:02.326595+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:34:02.326595+0000","last_undegraded":"2024-03-11T21:34:02.326595+0000","last_fullsized":"2024-03-11T21:34:02.326595+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'21","last_scrub_stamp":"2024-03-11T21:33:17.671661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.671661+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.774478+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0041327550000000001,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"426'1","reported_seq":718,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.366429+0000","last_change":"2024-03-11T21:33:32.236109+0000","last_active":"2024-03-11T21:34:02.366429+0000","last_peered":"2024-03-11T21:34:02.366429+0000","last_clean":"2024-03-11T21:34:02.366429+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:34:02.366429+0000","last_undegraded":"2024-03-11T21:34:02.366429+0000","last_fullsized":"2024-03-11T21:34:02.366429+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.235772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.235772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.638379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002624487,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":744,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.352129+0000","last_change":"2024-03-11T21:33:07.935033+0000","last_active":"2024-03-11T21:34:03.352129+0000","last_peered":"2024-03-11T21:34:03.352129+0000","last_clean":"2024-03-11T21:34:03.352129+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:34:03.352129+0000","last_undegraded":"2024-03-11T21:34:03.352129+0000","last_fullsized":"2024-03-11T21:34:03.352129+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:07.934653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.934653+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:14.435004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0080892789999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"412'41","reported_seq":738,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.382529+0000","last_change":"2024-03-11T21:33:25.692636+0000","last_active":"2024-03-11T21:34:02.382529+0000","last_peered":"2024-03-11T21:34:02.382529+0000","last_clean":"2024-03-11T21:34:02.382529+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:34:02.382529+0000","last_undegraded":"2024-03-11T21:34:02.382529+0000","last_fullsized":"2024-03-11T21:34:02.382529+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'39","last_scrub_stamp":"2024-03-11T21:33:25.692508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.692508+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.464747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001055535,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":780,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380361+0000","last_change":"2024-03-11T21:33:23.654493+0000","last_active":"2024-03-11T21:34:02.380361+0000","last_peered":"2024-03-11T21:34:02.380361+0000","last_clean":"2024-03-11T21:34:02.380361+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:34:02.380361+0000","last_undegraded":"2024-03-11T21:34:02.380361+0000","last_fullsized":"2024-03-11T21:34:02.380361+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:33:23.654355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.654355+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.325109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001789856,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":723,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.349029+0000","last_change":"2024-03-11T21:33:05.928221+0000","last_active":"2024-03-11T21:34:03.349029+0000","last_peered":"2024-03-11T21:34:03.349029+0000","last_clean":"2024-03-11T21:34:03.349029+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:34:03.349029+0000","last_undegraded":"2024-03-11T21:34:03.349029+0000","last_fullsized":"2024-03-11T21:34:03.349029+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:33:05.927971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.927971+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.190078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0078384049999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":795,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.347537+0000","last_change":"2024-03-11T21:33:10.952092+0000","last_active":"2024-03-11T21:34:03.347537+0000","last_peered":"2024-03-11T21:34:03.347537+0000","last_clean":"2024-03-11T21:34:03.347537+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:34:03.347537+0000","last_undegraded":"2024-03-11T21:34:03.347537+0000","last_fullsized":"2024-03-11T21:34:03.347537+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'107","last_scrub_stamp":"2024-03-11T21:33:10.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.951942+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.297855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067123959999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":603,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.380966+0000","last_change":"2024-03-11T21:33:17.964447+0000","last_active":"2024-03-11T21:34:00.380966+0000","last_peered":"2024-03-11T21:34:00.380966+0000","last_clean":"2024-03-11T21:34:00.380966+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:34:00.380966+0000","last_undegraded":"2024-03-11T21:34:00.380966+0000","last_fullsized":"2024-03-11T21:34:00.380966+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:33:17.964235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.964235+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.502834+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.005553105,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":759,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.352560+0000","last_change":"2024-03-11T21:33:04.948889+0000","last_active":"2024-03-11T21:34:03.352560+0000","last_peered":"2024-03-11T21:34:03.352560+0000","last_clean":"2024-03-11T21:34:03.352560+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:34:03.352560+0000","last_undegraded":"2024-03-11T21:34:03.352560+0000","last_fullsized":"2024-03-11T21:34:03.352560+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:33:04.948734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.948734+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.348007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0071306190000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378898+0000","last_change":"2024-03-11T21:32:34.326566+0000","last_active":"2024-03-11T21:34:02.378898+0000","last_peered":"2024-03-11T21:34:02.378898+0000","last_clean":"2024-03-11T21:34:02.378898+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:34:02.378898+0000","last_undegraded":"2024-03-11T21:34:02.378898+0000","last_fullsized":"2024-03-11T21:34:02.378898+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:32:07.554683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.554683+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096827370000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"412'50","reported_seq":766,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390125+0000","last_change":"2024-03-11T21:32:34.335870+0000","last_active":"2024-03-11T21:34:02.390125+0000","last_peered":"2024-03-11T21:34:02.390125+0000","last_clean":"2024-03-11T21:34:02.390125+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:34:02.390125+0000","last_undegraded":"2024-03-11T21:34:02.390125+0000","last_fullsized":"2024-03-11T21:34:02.390125+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023272950000000001,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"412'35","reported_seq":617,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390449+0000","last_change":"2024-03-11T21:33:14.938757+0000","last_active":"2024-03-11T21:34:00.390449+0000","last_peered":"2024-03-11T21:34:00.390449+0000","last_clean":"2024-03-11T21:34:00.390449+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:34:00.390449+0000","last_undegraded":"2024-03-11T21:34:00.390449+0000","last_fullsized":"2024-03-11T21:34:00.390449+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:14.938568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.938568+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:34.701797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011782635,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"361'209","reported_seq":868,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392151+0000","last_change":"2024-03-11T21:32:34.338193+0000","last_active":"2024-03-11T21:34:02.392151+0000","last_peered":"2024-03-11T21:34:02.392151+0000","last_clean":"2024-03-11T21:34:02.392151+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:34:02.392151+0000","last_undegraded":"2024-03-11T21:34:02.392151+0000","last_fullsized":"2024-03-11T21:34:02.392151+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829950000000001,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":728,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389514+0000","last_change":"2024-03-11T21:32:34.335400+0000","last_active":"2024-03-11T21:34:02.389514+0000","last_peered":"2024-03-11T21:34:02.389514+0000","last_clean":"2024-03-11T21:34:02.389514+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:34:02.389514+0000","last_undegraded":"2024-03-11T21:34:02.389514+0000","last_fullsized":"2024-03-11T21:34:02.389514+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024394040000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"412'15","reported_seq":779,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.400467+0000","last_change":"2024-03-11T21:33:31.565426+0000","last_active":"2024-03-11T21:34:02.400467+0000","last_peered":"2024-03-11T21:34:02.400467+0000","last_clean":"2024-03-11T21:34:02.400467+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:34:02.400467+0000","last_undegraded":"2024-03-11T21:34:02.400467+0000","last_fullsized":"2024-03-11T21:34:02.400467+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'14","last_scrub_stamp":"2024-03-11T21:33:31.565199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.565199+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:42.542855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0069698629999999998,"stat_sum":{"num_bytes":4194320,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"412'87","reported_seq":776,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403552+0000","last_change":"2024-03-11T21:32:34.312524+0000","last_active":"2024-03-11T21:34:02.403552+0000","last_peered":"2024-03-11T21:34:02.403552+0000","last_clean":"2024-03-11T21:34:02.403552+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:34:02.403552+0000","last_undegraded":"2024-03-11T21:34:02.403552+0000","last_fullsized":"2024-03-11T21:34:02.403552+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'85","last_scrub_stamp":"2024-03-11T21:32:14.839185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.839185+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018288136999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389204+0000","last_change":"2024-03-11T21:32:34.332276+0000","last_active":"2024-03-11T21:34:02.389204+0000","last_peered":"2024-03-11T21:34:02.389204+0000","last_clean":"2024-03-11T21:34:02.389204+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:34:02.389204+0000","last_undegraded":"2024-03-11T21:34:02.389204+0000","last_fullsized":"2024-03-11T21:34:02.389204+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016228219999999999,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.365704+0000","last_change":"2024-03-11T21:33:29.221706+0000","last_active":"2024-03-11T21:34:02.365704+0000","last_peered":"2024-03-11T21:34:02.365704+0000","last_clean":"2024-03-11T21:34:02.365704+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:34:02.365704+0000","last_undegraded":"2024-03-11T21:34:02.365704+0000","last_fullsized":"2024-03-11T21:34:02.365704+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:33:29.221527+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.221527+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:42.381652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029485240000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":761,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327131+0000","last_change":"2024-03-11T21:33:25.682885+0000","last_active":"2024-03-11T21:34:02.327131+0000","last_peered":"2024-03-11T21:34:02.327131+0000","last_clean":"2024-03-11T21:34:02.327131+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:34:02.327131+0000","last_undegraded":"2024-03-11T21:34:02.327131+0000","last_fullsized":"2024-03-11T21:34:02.327131+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'51","last_scrub_stamp":"2024-03-11T21:33:25.682768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.682768+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.367806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001496713,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"412'53","reported_seq":767,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.370609+0000","last_change":"2024-03-11T21:33:42.167828+0000","last_active":"2024-03-11T21:34:02.370609+0000","last_peered":"2024-03-11T21:34:02.370609+0000","last_clean":"2024-03-11T21:34:02.370609+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:34:02.370609+0000","last_undegraded":"2024-03-11T21:34:02.370609+0000","last_fullsized":"2024-03-11T21:34:02.370609+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'53","last_scrub_stamp":"2024-03-11T21:33:42.167697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.167697+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:03.235363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001987904,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"412'3","reported_seq":711,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392275+0000","last_change":"2024-03-11T21:32:34.338218+0000","last_active":"2024-03-11T21:34:02.392275+0000","last_peered":"2024-03-11T21:34:02.392275+0000","last_clean":"2024-03-11T21:34:02.392275+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:34:02.392275+0000","last_undegraded":"2024-03-11T21:34:02.392275+0000","last_fullsized":"2024-03-11T21:34:02.392275+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011248230000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377820+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:34:02.377820+0000","last_peered":"2024-03-11T21:34:02.377820+0000","last_clean":"2024-03-11T21:34:02.377820+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:34:02.377820+0000","last_undegraded":"2024-03-11T21:34:02.377820+0000","last_fullsized":"2024-03-11T21:34:02.377820+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377480+0000","last_change":"2024-03-11T21:33:28.579904+0000","last_active":"2024-03-11T21:34:02.377480+0000","last_peered":"2024-03-11T21:34:02.377480+0000","last_clean":"2024-03-11T21:34:02.377480+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:34:02.377480+0000","last_undegraded":"2024-03-11T21:34:02.377480+0000","last_fullsized":"2024-03-11T21:34:02.377480+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.577678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:46.193354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":621,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401121+0000","last_change":"2024-03-11T21:33:25.553124+0000","last_active":"2024-03-11T21:34:02.401121+0000","last_peered":"2024-03-11T21:34:02.401121+0000","last_clean":"2024-03-11T21:34:02.401121+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:34:02.401121+0000","last_undegraded":"2024-03-11T21:34:02.401121+0000","last_fullsized":"2024-03-11T21:34:02.401121+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.552965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.552965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.705964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":610,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378776+0000","last_change":"2024-03-11T21:33:13.735432+0000","last_active":"2024-03-11T21:34:02.378776+0000","last_peered":"2024-03-11T21:34:02.378776+0000","last_clean":"2024-03-11T21:34:02.378776+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:34:02.378776+0000","last_undegraded":"2024-03-11T21:34:02.378776+0000","last_fullsized":"2024-03-11T21:34:02.378776+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.735317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.735317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.675782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":516,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393079+0000","last_change":"2024-03-11T21:33:34.103121+0000","last_active":"2024-03-11T21:34:00.393079+0000","last_peered":"2024-03-11T21:34:00.393079+0000","last_clean":"2024-03-11T21:34:00.393079+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:34:00.393079+0000","last_undegraded":"2024-03-11T21:34:00.393079+0000","last_fullsized":"2024-03-11T21:34:00.393079+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:34.103006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:34.103006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.132610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379685+0000","last_change":"2024-03-11T21:33:40.436006+0000","last_active":"2024-03-11T21:34:02.379685+0000","last_peered":"2024-03-11T21:34:02.379685+0000","last_clean":"2024-03-11T21:34:02.379685+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:34:02.379685+0000","last_undegraded":"2024-03-11T21:34:02.379685+0000","last_fullsized":"2024-03-11T21:34:02.379685+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.435648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.435648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:05.273758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379607+0000","last_change":"2024-03-11T21:33:25.576354+0000","last_active":"2024-03-11T21:34:02.379607+0000","last_peered":"2024-03-11T21:34:02.379607+0000","last_clean":"2024-03-11T21:34:02.379607+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:34:02.379607+0000","last_undegraded":"2024-03-11T21:34:02.379607+0000","last_fullsized":"2024-03-11T21:34:02.379607+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.576253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.576253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:27.330979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.371088+0000","last_change":"2024-03-11T21:32:15.794130+0000","last_active":"2024-03-11T21:34:02.371088+0000","last_peered":"2024-03-11T21:34:02.371088+0000","last_clean":"2024-03-11T21:34:02.371088+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:34:02.371088+0000","last_undegraded":"2024-03-11T21:34:02.371088+0000","last_fullsized":"2024-03-11T21:34:02.371088+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.793910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.793910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":615,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326305+0000","last_change":"2024-03-11T21:30:54.963682+0000","last_active":"2024-03-11T21:34:02.326305+0000","last_peered":"2024-03-11T21:34:02.326305+0000","last_clean":"2024-03-11T21:34:02.326305+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:34:02.326305+0000","last_undegraded":"2024-03-11T21:34:02.326305+0000","last_fullsized":"2024-03-11T21:34:02.326305+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.963569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.963569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":610,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378042+0000","last_change":"2024-03-11T21:31:01.656496+0000","last_active":"2024-03-11T21:34:02.378042+0000","last_peered":"2024-03-11T21:34:02.378042+0000","last_clean":"2024-03-11T21:34:02.378042+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:34:02.378042+0000","last_undegraded":"2024-03-11T21:34:02.378042+0000","last_fullsized":"2024-03-11T21:34:02.378042+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.656422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.656422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":624,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.353294+0000","last_change":"2024-03-11T21:33:41.974005+0000","last_active":"2024-03-11T21:34:03.353294+0000","last_peered":"2024-03-11T21:34:03.353294+0000","last_clean":"2024-03-11T21:34:03.353294+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:34:03.353294+0000","last_undegraded":"2024-03-11T21:34:03.353294+0000","last_fullsized":"2024-03-11T21:34:03.353294+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.973907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:49.188951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":613,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392269+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:34:02.392269+0000","last_peered":"2024-03-11T21:34:02.392269+0000","last_clean":"2024-03-11T21:34:02.392269+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:34:02.392269+0000","last_undegraded":"2024-03-11T21:34:02.392269+0000","last_fullsized":"2024-03-11T21:34:02.392269+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":616,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.351790+0000","last_change":"2024-03-11T21:31:02.849078+0000","last_active":"2024-03-11T21:34:03.351790+0000","last_peered":"2024-03-11T21:34:03.351790+0000","last_clean":"2024-03-11T21:34:03.351790+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:34:03.351790+0000","last_undegraded":"2024-03-11T21:34:03.351790+0000","last_fullsized":"2024-03-11T21:34:03.351790+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.848978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.848978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":615,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328017+0000","last_change":"2024-03-11T21:33:10.604343+0000","last_active":"2024-03-11T21:34:02.328017+0000","last_peered":"2024-03-11T21:34:02.328017+0000","last_clean":"2024-03-11T21:34:02.328017+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:34:02.328017+0000","last_undegraded":"2024-03-11T21:34:02.328017+0000","last_fullsized":"2024-03-11T21:34:02.328017+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.604160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.604160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.058178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.368075+0000","last_change":"2024-03-11T21:33:27.227939+0000","last_active":"2024-03-11T21:34:02.368075+0000","last_peered":"2024-03-11T21:34:02.368075+0000","last_clean":"2024-03-11T21:34:02.368075+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:34:02.368075+0000","last_undegraded":"2024-03-11T21:34:02.368075+0000","last_fullsized":"2024-03-11T21:34:02.368075+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:27.227845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.227845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.639218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":13,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390271+0000","last_change":"2024-03-11T21:33:59.394929+0000","last_active":"2024-03-11T21:34:00.390271+0000","last_peered":"2024-03-11T21:34:00.390271+0000","last_clean":"2024-03-11T21:34:00.390271+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:34:00.390271+0000","last_undegraded":"2024-03-11T21:34:00.390271+0000","last_fullsized":"2024-03-11T21:34:00.390271+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.603946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.603946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377523+0000","last_change":"2024-03-11T21:33:15.600135+0000","last_active":"2024-03-11T21:34:02.377523+0000","last_peered":"2024-03-11T21:34:02.377523+0000","last_clean":"2024-03-11T21:34:02.377523+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:34:02.377523+0000","last_undegraded":"2024-03-11T21:34:02.377523+0000","last_fullsized":"2024-03-11T21:34:02.377523+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:15.600023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.600023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:16.478828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388704+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:34:02.388704+0000","last_peered":"2024-03-11T21:34:02.388704+0000","last_clean":"2024-03-11T21:34:02.388704+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:34:02.388704+0000","last_undegraded":"2024-03-11T21:34:02.388704+0000","last_fullsized":"2024-03-11T21:34:02.388704+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":610,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378608+0000","last_change":"2024-03-11T21:33:12.695054+0000","last_active":"2024-03-11T21:34:02.378608+0000","last_peered":"2024-03-11T21:34:02.378608+0000","last_clean":"2024-03-11T21:34:02.378608+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:34:02.378608+0000","last_undegraded":"2024-03-11T21:34:02.378608+0000","last_fullsized":"2024-03-11T21:34:02.378608+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.694966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.694966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:36.431830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":615,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327580+0000","last_change":"2024-03-11T21:32:14.693680+0000","last_active":"2024-03-11T21:34:02.327580+0000","last_peered":"2024-03-11T21:34:02.327580+0000","last_clean":"2024-03-11T21:34:02.327580+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:34:02.327580+0000","last_undegraded":"2024-03-11T21:34:02.327580+0000","last_fullsized":"2024-03-11T21:34:02.327580+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.693586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.693586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":485,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393407+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:34:00.393407+0000","last_peered":"2024-03-11T21:34:00.393407+0000","last_clean":"2024-03-11T21:34:00.393407+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:34:00.393407+0000","last_undegraded":"2024-03-11T21:34:00.393407+0000","last_fullsized":"2024-03-11T21:34:00.393407+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":610,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379874+0000","last_change":"2024-03-11T21:32:07.649024+0000","last_active":"2024-03-11T21:34:02.379874+0000","last_peered":"2024-03-11T21:34:02.379874+0000","last_clean":"2024-03-11T21:34:02.379874+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:34:02.379874+0000","last_undegraded":"2024-03-11T21:34:02.379874+0000","last_fullsized":"2024-03-11T21:34:02.379874+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.648800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.648800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":610,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380351+0000","last_change":"2024-03-11T21:32:15.722549+0000","last_active":"2024-03-11T21:34:02.380351+0000","last_peered":"2024-03-11T21:34:02.380351+0000","last_clean":"2024-03-11T21:34:02.380351+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:34:02.380351+0000","last_undegraded":"2024-03-11T21:34:02.380351+0000","last_fullsized":"2024-03-11T21:34:02.380351+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.722443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.722443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":616,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.356929+0000","last_change":"2024-03-11T21:32:14.976068+0000","last_active":"2024-03-11T21:34:03.356929+0000","last_peered":"2024-03-11T21:34:03.356929+0000","last_clean":"2024-03-11T21:34:03.356929+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:34:03.356929+0000","last_undegraded":"2024-03-11T21:34:03.356929+0000","last_fullsized":"2024-03-11T21:34:03.356929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.975966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.975966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":621,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401214+0000","last_change":"2024-03-11T21:33:41.560584+0000","last_active":"2024-03-11T21:34:02.401214+0000","last_peered":"2024-03-11T21:34:02.401214+0000","last_clean":"2024-03-11T21:34:02.401214+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:34:02.401214+0000","last_undegraded":"2024-03-11T21:34:02.401214+0000","last_fullsized":"2024-03-11T21:34:02.401214+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.560389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.560389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:44.840448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":613,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388809+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:34:02.388809+0000","last_peered":"2024-03-11T21:34:02.388809+0000","last_clean":"2024-03-11T21:34:02.388809+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:34:02.388809+0000","last_undegraded":"2024-03-11T21:34:02.388809+0000","last_fullsized":"2024-03-11T21:34:02.388809+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.532268+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":13,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390743+0000","last_change":"2024-03-11T21:33:59.395800+0000","last_active":"2024-03-11T21:34:00.390743+0000","last_peered":"2024-03-11T21:34:00.390743+0000","last_clean":"2024-03-11T21:34:00.390743+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:34:00.390743+0000","last_undegraded":"2024-03-11T21:34:00.390743+0000","last_fullsized":"2024-03-11T21:34:00.390743+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.793796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.793796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":615,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327454+0000","last_change":"2024-03-11T21:33:08.584273+0000","last_active":"2024-03-11T21:34:02.327454+0000","last_peered":"2024-03-11T21:34:02.327454+0000","last_clean":"2024-03-11T21:34:02.327454+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:34:02.327454+0000","last_undegraded":"2024-03-11T21:34:02.327454+0000","last_fullsized":"2024-03-11T21:34:02.327454+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.584163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.584163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.416698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":616,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.352437+0000","last_change":"2024-03-11T21:30:59.897121+0000","last_active":"2024-03-11T21:34:03.352437+0000","last_peered":"2024-03-11T21:34:03.352437+0000","last_clean":"2024-03-11T21:34:03.352437+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:34:03.352437+0000","last_undegraded":"2024-03-11T21:34:03.352437+0000","last_fullsized":"2024-03-11T21:34:03.352437+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.897032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.897032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":613,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391453+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:34:02.391453+0000","last_peered":"2024-03-11T21:34:02.391453+0000","last_clean":"2024-03-11T21:34:02.391453+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:34:02.391453+0000","last_undegraded":"2024-03-11T21:34:02.391453+0000","last_fullsized":"2024-03-11T21:34:02.391453+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":623,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.368458+0000","last_change":"2024-03-11T21:33:10.163119+0000","last_active":"2024-03-11T21:34:02.368458+0000","last_peered":"2024-03-11T21:34:02.368458+0000","last_clean":"2024-03-11T21:34:02.368458+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:34:02.368458+0000","last_undegraded":"2024-03-11T21:34:02.368458+0000","last_fullsized":"2024-03-11T21:34:02.368458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.162829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.162829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.878881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":485,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390679+0000","last_change":"2024-03-11T21:32:28.978945+0000","last_active":"2024-03-11T21:34:00.390679+0000","last_peered":"2024-03-11T21:34:00.390679+0000","last_clean":"2024-03-11T21:34:00.390679+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:34:00.390679+0000","last_undegraded":"2024-03-11T21:34:00.390679+0000","last_fullsized":"2024-03-11T21:34:00.390679+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":630,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.356529+0000","last_change":"2024-03-11T21:33:31.964693+0000","last_active":"2024-03-11T21:34:03.356529+0000","last_peered":"2024-03-11T21:34:03.356529+0000","last_clean":"2024-03-11T21:34:03.356529+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:34:03.356529+0000","last_undegraded":"2024-03-11T21:34:03.356529+0000","last_fullsized":"2024-03-11T21:34:03.356529+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:31.964602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.964602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.956887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":907,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390327+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:34:02.390327+0000","last_peered":"2024-03-11T21:34:02.390327+0000","last_clean":"2024-03-11T21:34:02.390327+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:34:02.390327+0000","last_undegraded":"2024-03-11T21:34:02.390327+0000","last_fullsized":"2024-03-11T21:34:02.390327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":811,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.395374+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:34:02.395374+0000","last_peered":"2024-03-11T21:34:02.395374+0000","last_clean":"2024-03-11T21:34:02.395374+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:34:02.395374+0000","last_undegraded":"2024-03-11T21:34:02.395374+0000","last_fullsized":"2024-03-11T21:34:02.395374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.368839+0000","last_change":"2024-03-11T21:33:25.276824+0000","last_active":"2024-03-11T21:34:02.368839+0000","last_peered":"2024-03-11T21:34:02.368839+0000","last_clean":"2024-03-11T21:34:02.368839+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:34:02.368839+0000","last_undegraded":"2024-03-11T21:34:02.368839+0000","last_fullsized":"2024-03-11T21:34:02.368839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.276731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.276731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.729396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":770,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403607+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:34:02.403607+0000","last_peered":"2024-03-11T21:34:02.403607+0000","last_clean":"2024-03-11T21:34:02.403607+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:34:02.403607+0000","last_undegraded":"2024-03-11T21:34:02.403607+0000","last_fullsized":"2024-03-11T21:34:02.403607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:07.609739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":628,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.404251+0000","last_change":"2024-03-11T21:31:51.834491+0000","last_active":"2024-03-11T21:34:02.404251+0000","last_peered":"2024-03-11T21:34:02.404251+0000","last_clean":"2024-03-11T21:34:02.404251+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:34:02.404251+0000","last_undegraded":"2024-03-11T21:34:02.404251+0000","last_fullsized":"2024-03-11T21:34:02.404251+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.834251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.834251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":807,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.404338+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:34:02.404338+0000","last_peered":"2024-03-11T21:34:02.404338+0000","last_clean":"2024-03-11T21:34:02.404338+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:34:02.404338+0000","last_undegraded":"2024-03-11T21:34:02.404338+0000","last_fullsized":"2024-03-11T21:34:02.404338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":667,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.383867+0000","last_change":"2024-03-11T21:32:06.996760+0000","last_active":"2024-03-11T21:34:00.383867+0000","last_peered":"2024-03-11T21:34:00.383867+0000","last_clean":"2024-03-11T21:34:00.383867+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:34:00.383867+0000","last_undegraded":"2024-03-11T21:34:00.383867+0000","last_fullsized":"2024-03-11T21:34:00.383867+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.996518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.996518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":783,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.329325+0000","last_change":"2024-03-11T21:32:28.781453+0000","last_active":"2024-03-11T21:34:02.329325+0000","last_peered":"2024-03-11T21:34:02.329325+0000","last_clean":"2024-03-11T21:34:02.329325+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:34:02.329325+0000","last_undegraded":"2024-03-11T21:34:02.329325+0000","last_fullsized":"2024-03-11T21:34:02.329325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":773,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.351350+0000","last_change":"2024-03-11T21:32:25.048822+0000","last_active":"2024-03-11T21:34:03.351350+0000","last_peered":"2024-03-11T21:34:03.351350+0000","last_clean":"2024-03-11T21:34:03.351350+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:34:03.351350+0000","last_undegraded":"2024-03-11T21:34:03.351350+0000","last_fullsized":"2024-03-11T21:34:03.351350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.048696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.048696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403538+0000","last_change":"2024-03-11T21:33:22.548838+0000","last_active":"2024-03-11T21:34:02.403538+0000","last_peered":"2024-03-11T21:34:02.403538+0000","last_clean":"2024-03-11T21:34:02.403538+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:34:02.403538+0000","last_undegraded":"2024-03-11T21:34:02.403538+0000","last_fullsized":"2024-03-11T21:34:02.403538+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.548708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.548708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.314884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":833,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392233+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:34:02.392233+0000","last_peered":"2024-03-11T21:34:02.392233+0000","last_clean":"2024-03-11T21:34:02.392233+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:34:02.392233+0000","last_undegraded":"2024-03-11T21:34:02.392233+0000","last_fullsized":"2024-03-11T21:34:02.392233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.799876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.367532+0000","last_change":"2024-03-11T21:33:12.199443+0000","last_active":"2024-03-11T21:34:02.367532+0000","last_peered":"2024-03-11T21:34:02.367532+0000","last_clean":"2024-03-11T21:34:02.367532+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:34:02.367532+0000","last_undegraded":"2024-03-11T21:34:02.367532+0000","last_fullsized":"2024-03-11T21:34:02.367532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.199234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.199234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:28.102665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":782,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.363702+0000","last_change":"2024-03-11T21:32:16.782474+0000","last_active":"2024-03-11T21:34:02.363702+0000","last_peered":"2024-03-11T21:34:02.363702+0000","last_clean":"2024-03-11T21:34:02.363702+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:34:02.363702+0000","last_undegraded":"2024-03-11T21:34:02.363702+0000","last_fullsized":"2024-03-11T21:34:02.363702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.782400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.782400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":761,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388560+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:34:02.388560+0000","last_peered":"2024-03-11T21:34:02.388560+0000","last_clean":"2024-03-11T21:34:02.388560+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:34:02.388560+0000","last_undegraded":"2024-03-11T21:34:02.388560+0000","last_fullsized":"2024-03-11T21:34:02.388560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":494,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.385759+0000","last_change":"2024-03-11T21:33:39.131584+0000","last_active":"2024-03-11T21:34:00.385759+0000","last_peered":"2024-03-11T21:34:00.385759+0000","last_clean":"2024-03-11T21:34:00.385759+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:34:00.385759+0000","last_undegraded":"2024-03-11T21:34:00.385759+0000","last_fullsized":"2024-03-11T21:34:00.385759+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.131510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.131510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.433688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":860,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378221+0000","last_change":"2024-03-11T21:33:46.396054+0000","last_active":"2024-03-11T21:34:02.378221+0000","last_peered":"2024-03-11T21:34:02.378221+0000","last_clean":"2024-03-11T21:34:02.378221+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:34:02.378221+0000","last_undegraded":"2024-03-11T21:34:02.378221+0000","last_fullsized":"2024-03-11T21:34:02.378221+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:33:46.395671+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.395671+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.312098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":780,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391343+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:34:02.391343+0000","last_peered":"2024-03-11T21:34:02.391343+0000","last_clean":"2024-03-11T21:34:02.391343+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:34:02.391343+0000","last_undegraded":"2024-03-11T21:34:02.391343+0000","last_fullsized":"2024-03-11T21:34:02.391343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":673,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393211+0000","last_change":"2024-03-11T21:33:13.942454+0000","last_active":"2024-03-11T21:34:00.393211+0000","last_peered":"2024-03-11T21:34:00.393211+0000","last_clean":"2024-03-11T21:34:00.393211+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:34:00.393211+0000","last_undegraded":"2024-03-11T21:34:00.393211+0000","last_fullsized":"2024-03-11T21:34:00.393211+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.942158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.942158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.543654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":778,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.371499+0000","last_change":"2024-03-11T21:32:05.876383+0000","last_active":"2024-03-11T21:34:02.371499+0000","last_peered":"2024-03-11T21:34:02.371499+0000","last_clean":"2024-03-11T21:34:02.371499+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:34:02.371499+0000","last_undegraded":"2024-03-11T21:34:02.371499+0000","last_fullsized":"2024-03-11T21:34:02.371499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.876018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.876018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":633,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378282+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:34:02.378282+0000","last_peered":"2024-03-11T21:34:02.378282+0000","last_clean":"2024-03-11T21:34:02.378282+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:34:02.378282+0000","last_undegraded":"2024-03-11T21:34:02.378282+0000","last_fullsized":"2024-03-11T21:34:02.378282+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":801,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379126+0000","last_change":"2024-03-11T21:32:28.904947+0000","last_active":"2024-03-11T21:34:02.379126+0000","last_peered":"2024-03-11T21:34:02.379126+0000","last_clean":"2024-03-11T21:34:02.379126+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:34:02.379126+0000","last_undegraded":"2024-03-11T21:34:02.379126+0000","last_fullsized":"2024-03-11T21:34:02.379126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:32:28.904821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.904821+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":633,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390316+0000","last_change":"2024-03-11T21:32:04.059244+0000","last_active":"2024-03-11T21:34:00.390316+0000","last_peered":"2024-03-11T21:34:00.390316+0000","last_clean":"2024-03-11T21:34:00.390316+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:34:00.390316+0000","last_undegraded":"2024-03-11T21:34:00.390316+0000","last_fullsized":"2024-03-11T21:34:00.390316+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.059137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":933,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.367059+0000","last_change":"2024-03-11T21:33:41.205378+0000","last_active":"2024-03-11T21:34:02.367059+0000","last_peered":"2024-03-11T21:34:02.367059+0000","last_clean":"2024-03-11T21:34:02.367059+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:34:02.367059+0000","last_undegraded":"2024-03-11T21:34:02.367059+0000","last_fullsized":"2024-03-11T21:34:02.367059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:33:41.205243+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.205243+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:07.172685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":779,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328296+0000","last_change":"2024-03-11T21:32:08.744653+0000","last_active":"2024-03-11T21:34:02.328296+0000","last_peered":"2024-03-11T21:34:02.328296+0000","last_clean":"2024-03-11T21:34:02.328296+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:34:02.328296+0000","last_undegraded":"2024-03-11T21:34:02.328296+0000","last_fullsized":"2024-03-11T21:34:02.328296+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.744422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.744422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":636,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380037+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:34:02.380037+0000","last_peered":"2024-03-11T21:34:02.380037+0000","last_clean":"2024-03-11T21:34:02.380037+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:34:02.380037+0000","last_undegraded":"2024-03-11T21:34:02.380037+0000","last_fullsized":"2024-03-11T21:34:02.380037+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.099107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":829,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378046+0000","last_change":"2024-03-11T21:33:20.697629+0000","last_active":"2024-03-11T21:34:02.378046+0000","last_peered":"2024-03-11T21:34:02.378046+0000","last_clean":"2024-03-11T21:34:02.378046+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:34:02.378046+0000","last_undegraded":"2024-03-11T21:34:02.378046+0000","last_fullsized":"2024-03-11T21:34:02.378046+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:20.697247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.697247+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:39.581547+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":661,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.382347+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:34:00.382347+0000","last_peered":"2024-03-11T21:34:00.382347+0000","last_clean":"2024-03-11T21:34:00.382347+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:34:00.382347+0000","last_undegraded":"2024-03-11T21:34:00.382347+0000","last_fullsized":"2024-03-11T21:34:00.382347+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.261219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":916,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328125+0000","last_change":"2024-03-11T21:33:12.674157+0000","last_active":"2024-03-11T21:34:02.328125+0000","last_peered":"2024-03-11T21:34:02.328125+0000","last_clean":"2024-03-11T21:34:02.328125+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:34:02.328125+0000","last_undegraded":"2024-03-11T21:34:02.328125+0000","last_fullsized":"2024-03-11T21:34:02.328125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:12.673934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.673934+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:21.805791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":778,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.402420+0000","last_change":"2024-03-11T21:32:05.838146+0000","last_active":"2024-03-11T21:34:02.402420+0000","last_peered":"2024-03-11T21:34:02.402420+0000","last_clean":"2024-03-11T21:34:02.402420+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:34:02.402420+0000","last_undegraded":"2024-03-11T21:34:02.402420+0000","last_fullsized":"2024-03-11T21:34:02.402420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.838032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.838032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390030+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:34:02.390030+0000","last_peered":"2024-03-11T21:34:02.390030+0000","last_clean":"2024-03-11T21:34:02.390030+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:34:02.390030+0000","last_undegraded":"2024-03-11T21:34:02.390030+0000","last_fullsized":"2024-03-11T21:34:02.390030+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":816,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326367+0000","last_change":"2024-03-11T21:33:35.635520+0000","last_active":"2024-03-11T21:34:02.326367+0000","last_peered":"2024-03-11T21:34:02.326367+0000","last_clean":"2024-03-11T21:34:02.326367+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:34:02.326367+0000","last_undegraded":"2024-03-11T21:34:02.326367+0000","last_fullsized":"2024-03-11T21:34:02.326367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.635256+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.891508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388911+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:34:02.388911+0000","last_peered":"2024-03-11T21:34:02.388911+0000","last_clean":"2024-03-11T21:34:02.388911+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:34:02.388911+0000","last_undegraded":"2024-03-11T21:34:02.388911+0000","last_fullsized":"2024-03-11T21:34:02.388911+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.709131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":773,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405527+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:34:02.405527+0000","last_peered":"2024-03-11T21:34:02.405527+0000","last_clean":"2024-03-11T21:34:02.405527+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:34:02.405527+0000","last_undegraded":"2024-03-11T21:34:02.405527+0000","last_fullsized":"2024-03-11T21:34:02.405527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379625+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:34:02.379625+0000","last_peered":"2024-03-11T21:34:02.379625+0000","last_clean":"2024-03-11T21:34:02.379625+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:34:02.379625+0000","last_undegraded":"2024-03-11T21:34:02.379625+0000","last_fullsized":"2024-03-11T21:34:02.379625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379763+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:34:02.379763+0000","last_peered":"2024-03-11T21:34:02.379763+0000","last_clean":"2024-03-11T21:34:02.379763+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:34:02.379763+0000","last_undegraded":"2024-03-11T21:34:02.379763+0000","last_fullsized":"2024-03-11T21:34:02.379763+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":925,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378201+0000","last_change":"2024-03-11T21:33:10.709056+0000","last_active":"2024-03-11T21:34:02.378201+0000","last_peered":"2024-03-11T21:34:02.378201+0000","last_clean":"2024-03-11T21:34:02.378201+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:34:02.378201+0000","last_undegraded":"2024-03-11T21:34:02.378201+0000","last_fullsized":"2024-03-11T21:34:02.378201+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:33:10.708723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.708723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:28.526439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":792,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326426+0000","last_change":"2024-03-11T21:33:37.698300+0000","last_active":"2024-03-11T21:34:02.326426+0000","last_peered":"2024-03-11T21:34:02.326426+0000","last_clean":"2024-03-11T21:34:02.326426+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:34:02.326426+0000","last_undegraded":"2024-03-11T21:34:02.326426+0000","last_fullsized":"2024-03-11T21:34:02.326426+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:37.698095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.698095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.205420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":826,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.358520+0000","last_change":"2024-03-11T21:31:58.977677+0000","last_active":"2024-03-11T21:34:03.358520+0000","last_peered":"2024-03-11T21:34:03.358520+0000","last_clean":"2024-03-11T21:34:03.358520+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:34:03.358520+0000","last_undegraded":"2024-03-11T21:34:03.358520+0000","last_fullsized":"2024-03-11T21:34:03.358520+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:31:58.977541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.977541+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":800,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379907+0000","last_change":"2024-03-11T21:33:04.660322+0000","last_active":"2024-03-11T21:34:02.379907+0000","last_peered":"2024-03-11T21:34:02.379907+0000","last_clean":"2024-03-11T21:34:02.379907+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:34:02.379907+0000","last_undegraded":"2024-03-11T21:34:02.379907+0000","last_fullsized":"2024-03-11T21:34:02.379907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:33:04.660206+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.660206+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:27.665988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":771,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390528+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:34:02.390528+0000","last_peered":"2024-03-11T21:34:02.390528+0000","last_clean":"2024-03-11T21:34:02.390528+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:34:02.390528+0000","last_undegraded":"2024-03-11T21:34:02.390528+0000","last_fullsized":"2024-03-11T21:34:02.390528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391702+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:34:02.391702+0000","last_peered":"2024-03-11T21:34:02.391702+0000","last_clean":"2024-03-11T21:34:02.391702+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:34:02.391702+0000","last_undegraded":"2024-03-11T21:34:02.391702+0000","last_fullsized":"2024-03-11T21:34:02.391702+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":775,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326305+0000","last_change":"2024-03-11T21:31:02.847904+0000","last_active":"2024-03-11T21:34:02.326305+0000","last_peered":"2024-03-11T21:34:02.326305+0000","last_clean":"2024-03-11T21:34:02.326305+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:34:02.326305+0000","last_undegraded":"2024-03-11T21:34:02.326305+0000","last_fullsized":"2024-03-11T21:34:02.326305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:02.847692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.847692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378139+0000","last_change":"2024-03-11T21:33:40.550313+0000","last_active":"2024-03-11T21:34:02.378139+0000","last_peered":"2024-03-11T21:34:02.378139+0000","last_clean":"2024-03-11T21:34:02.378139+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:34:02.378139+0000","last_undegraded":"2024-03-11T21:34:02.378139+0000","last_fullsized":"2024-03-11T21:34:02.378139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:33:40.550110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.550110+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.511780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":639,"num_read_kb":634,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":802,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.382707+0000","last_change":"2024-03-11T21:30:47.779423+0000","last_active":"2024-03-11T21:34:02.382707+0000","last_peered":"2024-03-11T21:34:02.382707+0000","last_clean":"2024-03-11T21:34:02.382707+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:34:02.382707+0000","last_undegraded":"2024-03-11T21:34:02.382707+0000","last_fullsized":"2024-03-11T21:34:02.382707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T21:30:47.779298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:47.779298+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":686,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390089+0000","last_change":"2024-03-11T21:33:09.913175+0000","last_active":"2024-03-11T21:34:00.390089+0000","last_peered":"2024-03-11T21:34:00.390089+0000","last_clean":"2024-03-11T21:34:00.390089+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:34:00.390089+0000","last_undegraded":"2024-03-11T21:34:00.390089+0000","last_fullsized":"2024-03-11T21:34:00.390089+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:09.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.912886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:20.681554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":771,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326932+0000","last_change":"2024-03-11T21:30:58.912377+0000","last_active":"2024-03-11T21:34:02.326932+0000","last_peered":"2024-03-11T21:34:02.326932+0000","last_clean":"2024-03-11T21:34:02.326932+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:34:02.326932+0000","last_undegraded":"2024-03-11T21:34:02.326932+0000","last_fullsized":"2024-03-11T21:34:02.326932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.912133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392650+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:34:02.392650+0000","last_peered":"2024-03-11T21:34:02.392650+0000","last_clean":"2024-03-11T21:34:02.392650+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:34:02.392650+0000","last_undegraded":"2024-03-11T21:34:02.392650+0000","last_fullsized":"2024-03-11T21:34:02.392650+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":775,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377890+0000","last_change":"2024-03-11T21:33:09.651325+0000","last_active":"2024-03-11T21:34:02.377890+0000","last_peered":"2024-03-11T21:34:02.377890+0000","last_clean":"2024-03-11T21:34:02.377890+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:34:02.377890+0000","last_undegraded":"2024-03-11T21:34:02.377890+0000","last_fullsized":"2024-03-11T21:34:02.377890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.651101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.651101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:18.894956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"389'44","reported_seq":844,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.395089+0000","last_change":"2024-03-11T21:32:05.077652+0000","last_active":"2024-03-11T21:34:00.395089+0000","last_peered":"2024-03-11T21:34:00.395089+0000","last_clean":"2024-03-11T21:34:00.395089+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:34:00.395089+0000","last_undegraded":"2024-03-11T21:34:00.395089+0000","last_fullsized":"2024-03-11T21:34:00.395089+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'32","last_scrub_stamp":"2024-03-11T21:32:05.077406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.077406+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392059+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:34:02.392059+0000","last_peered":"2024-03-11T21:34:02.392059+0000","last_clean":"2024-03-11T21:34:02.392059+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:34:02.392059+0000","last_undegraded":"2024-03-11T21:34:02.392059+0000","last_fullsized":"2024-03-11T21:34:02.392059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":633,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.381390+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:34:00.381390+0000","last_peered":"2024-03-11T21:34:00.381390+0000","last_clean":"2024-03-11T21:34:00.381390+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:34:00.381390+0000","last_undegraded":"2024-03-11T21:34:00.381390+0000","last_fullsized":"2024-03-11T21:34:00.381390+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:15.222087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":622,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.355934+0000","last_change":"2024-03-11T21:30:48.777877+0000","last_active":"2024-03-11T21:34:03.355934+0000","last_peered":"2024-03-11T21:34:03.355934+0000","last_clean":"2024-03-11T21:34:03.355934+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:34:03.355934+0000","last_undegraded":"2024-03-11T21:34:03.355934+0000","last_fullsized":"2024-03-11T21:34:03.355934+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.777772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.777772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":783,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405344+0000","last_change":"2024-03-11T21:32:07.813703+0000","last_active":"2024-03-11T21:34:02.405344+0000","last_peered":"2024-03-11T21:34:02.405344+0000","last_clean":"2024-03-11T21:34:02.405344+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:34:02.405344+0000","last_undegraded":"2024-03-11T21:34:02.405344+0000","last_fullsized":"2024-03-11T21:34:02.405344+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.813604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.813604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388559+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:34:02.388559+0000","last_peered":"2024-03-11T21:34:02.388559+0000","last_clean":"2024-03-11T21:34:02.388559+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:34:02.388559+0000","last_undegraded":"2024-03-11T21:34:02.388559+0000","last_fullsized":"2024-03-11T21:34:02.388559+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:11.903723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":823,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.350550+0000","last_change":"2024-03-11T21:33:17.014343+0000","last_active":"2024-03-11T21:34:03.350550+0000","last_peered":"2024-03-11T21:34:03.350550+0000","last_clean":"2024-03-11T21:34:03.350550+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:34:03.350550+0000","last_undegraded":"2024-03-11T21:34:03.350550+0000","last_fullsized":"2024-03-11T21:34:03.350550+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:17.014179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.014179+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.241673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":635,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393910+0000","last_change":"2024-03-11T21:32:17.944438+0000","last_active":"2024-03-11T21:34:00.393910+0000","last_peered":"2024-03-11T21:34:00.393910+0000","last_clean":"2024-03-11T21:34:00.393910+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:34:00.393910+0000","last_undegraded":"2024-03-11T21:34:00.393910+0000","last_fullsized":"2024-03-11T21:34:00.393910+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.944141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.944141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377922+0000","last_change":"2024-03-11T21:32:16.760994+0000","last_active":"2024-03-11T21:34:02.377922+0000","last_peered":"2024-03-11T21:34:02.377922+0000","last_clean":"2024-03-11T21:34:02.377922+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:34:02.377922+0000","last_undegraded":"2024-03-11T21:34:02.377922+0000","last_fullsized":"2024-03-11T21:34:02.377922+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.760914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.760914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":765,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.390739+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:34:02.390739+0000","last_peered":"2024-03-11T21:34:02.390739+0000","last_clean":"2024-03-11T21:34:02.390739+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:34:02.390739+0000","last_undegraded":"2024-03-11T21:34:02.390739+0000","last_fullsized":"2024-03-11T21:34:02.390739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1036,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.388756+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:34:02.388756+0000","last_peered":"2024-03-11T21:34:02.388756+0000","last_clean":"2024-03-11T21:34:02.388756+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:34:02.388756+0000","last_undegraded":"2024-03-11T21:34:02.388756+0000","last_fullsized":"2024-03-11T21:34:02.388756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403871+0000","last_change":"2024-03-11T21:33:13.461458+0000","last_active":"2024-03-11T21:34:02.403871+0000","last_peered":"2024-03-11T21:34:02.403871+0000","last_clean":"2024-03-11T21:34:02.403871+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:34:02.403871+0000","last_undegraded":"2024-03-11T21:34:02.403871+0000","last_fullsized":"2024-03-11T21:34:02.403871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.461214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.461214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.231159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":799,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379240+0000","last_change":"2024-03-11T21:30:32.862005+0000","last_active":"2024-03-11T21:34:02.379240+0000","last_peered":"2024-03-11T21:34:02.379240+0000","last_clean":"2024-03-11T21:34:02.379240+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:34:02.379240+0000","last_undegraded":"2024-03-11T21:34:02.379240+0000","last_fullsized":"2024-03-11T21:34:02.379240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:30:32.861910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.861910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":770,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405636+0000","last_change":"2024-03-11T21:32:26.159690+0000","last_active":"2024-03-11T21:34:02.405636+0000","last_peered":"2024-03-11T21:34:02.405636+0000","last_clean":"2024-03-11T21:34:02.405636+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:34:02.405636+0000","last_undegraded":"2024-03-11T21:34:02.405636+0000","last_fullsized":"2024-03-11T21:34:02.405636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:26.159472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:26.159472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":628,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401297+0000","last_change":"2024-03-11T21:31:57.949982+0000","last_active":"2024-03-11T21:34:02.401297+0000","last_peered":"2024-03-11T21:34:02.401297+0000","last_clean":"2024-03-11T21:34:02.401297+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:34:02.401297+0000","last_undegraded":"2024-03-11T21:34:02.401297+0000","last_fullsized":"2024-03-11T21:34:02.401297+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.949771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.949771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.370322+0000","last_change":"2024-03-11T21:33:43.147050+0000","last_active":"2024-03-11T21:34:02.370322+0000","last_peered":"2024-03-11T21:34:02.370322+0000","last_clean":"2024-03-11T21:34:02.370322+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:34:02.370322+0000","last_undegraded":"2024-03-11T21:34:02.370322+0000","last_fullsized":"2024-03-11T21:34:02.370322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.146852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.146852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.907189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328372+0000","last_change":"2024-03-11T21:33:21.218946+0000","last_active":"2024-03-11T21:34:02.328372+0000","last_peered":"2024-03-11T21:34:02.328372+0000","last_clean":"2024-03-11T21:34:02.328372+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:34:02.328372+0000","last_undegraded":"2024-03-11T21:34:02.328372+0000","last_fullsized":"2024-03-11T21:34:02.328372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:21.218831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.218831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:23.137900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":717,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.386455+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:34:00.386455+0000","last_peered":"2024-03-11T21:34:00.386455+0000","last_clean":"2024-03-11T21:34:00.386455+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:34:00.386455+0000","last_undegraded":"2024-03-11T21:34:00.386455+0000","last_fullsized":"2024-03-11T21:34:00.386455+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.261123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":769,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391944+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:34:02.391944+0000","last_peered":"2024-03-11T21:34:02.391944+0000","last_clean":"2024-03-11T21:34:02.391944+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:34:02.391944+0000","last_undegraded":"2024-03-11T21:34:02.391944+0000","last_fullsized":"2024-03-11T21:34:02.391944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:51.335664+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":630,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.349965+0000","last_change":"2024-03-11T21:33:40.988530+0000","last_active":"2024-03-11T21:34:03.349965+0000","last_peered":"2024-03-11T21:34:03.349965+0000","last_clean":"2024-03-11T21:34:03.349965+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:34:03.349965+0000","last_undegraded":"2024-03-11T21:34:03.349965+0000","last_fullsized":"2024-03-11T21:34:03.349965+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.988258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.988258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.542071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":786,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.356638+0000","last_change":"2024-03-11T21:33:35.976814+0000","last_active":"2024-03-11T21:34:03.356638+0000","last_peered":"2024-03-11T21:34:03.356638+0000","last_clean":"2024-03-11T21:34:03.356638+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:34:03.356638+0000","last_undegraded":"2024-03-11T21:34:03.356638+0000","last_fullsized":"2024-03-11T21:34:03.356638+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.976718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.976718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.438197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":838,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327302+0000","last_change":"2024-03-11T21:30:48.891794+0000","last_active":"2024-03-11T21:34:02.327302+0000","last_peered":"2024-03-11T21:34:02.327302+0000","last_clean":"2024-03-11T21:34:02.327302+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:34:02.327302+0000","last_undegraded":"2024-03-11T21:34:02.327302+0000","last_fullsized":"2024-03-11T21:34:02.327302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'27","last_scrub_stamp":"2024-03-11T21:30:48.891648+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.891648+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.402090+0000","last_change":"2024-03-11T21:33:09.417056+0000","last_active":"2024-03-11T21:34:02.402090+0000","last_peered":"2024-03-11T21:34:02.402090+0000","last_clean":"2024-03-11T21:34:02.402090+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:34:02.402090+0000","last_undegraded":"2024-03-11T21:34:02.402090+0000","last_fullsized":"2024-03-11T21:34:02.402090+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.416868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.416868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:09.520332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":769,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380562+0000","last_change":"2024-03-11T21:33:33.610158+0000","last_active":"2024-03-11T21:34:02.380562+0000","last_peered":"2024-03-11T21:34:02.380562+0000","last_clean":"2024-03-11T21:34:02.380562+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:34:02.380562+0000","last_undegraded":"2024-03-11T21:34:02.380562+0000","last_fullsized":"2024-03-11T21:34:02.380562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:33.610071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:33.610071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.428900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":628,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.404019+0000","last_change":"2024-03-11T21:31:55.943647+0000","last_active":"2024-03-11T21:34:02.404019+0000","last_peered":"2024-03-11T21:34:02.404019+0000","last_clean":"2024-03-11T21:34:02.404019+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:34:02.404019+0000","last_undegraded":"2024-03-11T21:34:02.404019+0000","last_fullsized":"2024-03-11T21:34:02.404019+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.943487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.943487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":923,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.355654+0000","last_change":"2024-03-11T21:32:15.949343+0000","last_active":"2024-03-11T21:34:03.355654+0000","last_peered":"2024-03-11T21:34:03.355654+0000","last_clean":"2024-03-11T21:34:03.355654+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:34:03.355654+0000","last_undegraded":"2024-03-11T21:34:03.355654+0000","last_fullsized":"2024-03-11T21:34:03.355654+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:32:15.949107+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.949107+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":670,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.385895+0000","last_change":"2024-03-11T21:33:31.047676+0000","last_active":"2024-03-11T21:34:00.385895+0000","last_peered":"2024-03-11T21:34:00.385895+0000","last_clean":"2024-03-11T21:34:00.385895+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:34:00.385895+0000","last_undegraded":"2024-03-11T21:34:00.385895+0000","last_fullsized":"2024-03-11T21:34:00.385895+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:33:31.047447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.047447+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.521117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":958,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.402601+0000","last_change":"2024-03-11T21:33:37.464613+0000","last_active":"2024-03-11T21:34:02.402601+0000","last_peered":"2024-03-11T21:34:02.402601+0000","last_clean":"2024-03-11T21:34:02.402601+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:34:02.402601+0000","last_undegraded":"2024-03-11T21:34:02.402601+0000","last_fullsized":"2024-03-11T21:34:02.402601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:37.464485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.464485+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:38.411033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":641,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393657+0000","last_change":"2024-03-11T21:33:23.993323+0000","last_active":"2024-03-11T21:34:00.393657+0000","last_peered":"2024-03-11T21:34:00.393657+0000","last_clean":"2024-03-11T21:34:00.393657+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:34:00.393657+0000","last_undegraded":"2024-03-11T21:34:00.393657+0000","last_fullsized":"2024-03-11T21:34:00.393657+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.993242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.993242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.886605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":773,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.352881+0000","last_change":"2024-03-11T21:32:16.907647+0000","last_active":"2024-03-11T21:34:03.352881+0000","last_peered":"2024-03-11T21:34:03.352881+0000","last_clean":"2024-03-11T21:34:03.352881+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:34:03.352881+0000","last_undegraded":"2024-03-11T21:34:03.352881+0000","last_fullsized":"2024-03-11T21:34:03.352881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.907562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.907562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":494,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.393824+0000","last_change":"2024-03-11T21:33:43.133706+0000","last_active":"2024-03-11T21:34:00.393824+0000","last_peered":"2024-03-11T21:34:00.393824+0000","last_clean":"2024-03-11T21:34:00.393824+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:34:00.393824+0000","last_undegraded":"2024-03-11T21:34:00.393824+0000","last_fullsized":"2024-03-11T21:34:00.393824+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.133485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.133485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.234400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":792,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328605+0000","last_change":"2024-03-11T21:32:12.801345+0000","last_active":"2024-03-11T21:34:02.328605+0000","last_peered":"2024-03-11T21:34:02.328605+0000","last_clean":"2024-03-11T21:34:02.328605+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:34:02.328605+0000","last_undegraded":"2024-03-11T21:34:02.328605+0000","last_fullsized":"2024-03-11T21:34:02.328605+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":821,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392980+0000","last_change":"2024-03-11T21:33:31.683462+0000","last_active":"2024-03-11T21:34:02.392980+0000","last_peered":"2024-03-11T21:34:02.392980+0000","last_clean":"2024-03-11T21:34:02.392980+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:34:02.392980+0000","last_undegraded":"2024-03-11T21:34:02.392980+0000","last_fullsized":"2024-03-11T21:34:02.392980+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.683216+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.041563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":800,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328480+0000","last_change":"2024-03-11T21:31:05.819709+0000","last_active":"2024-03-11T21:34:02.328480+0000","last_peered":"2024-03-11T21:34:02.328480+0000","last_clean":"2024-03-11T21:34:02.328480+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:34:02.328480+0000","last_undegraded":"2024-03-11T21:34:02.328480+0000","last_fullsized":"2024-03-11T21:34:02.328480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:31:05.819557+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.819557+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":777,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379310+0000","last_change":"2024-03-11T21:33:14.693542+0000","last_active":"2024-03-11T21:34:02.379310+0000","last_peered":"2024-03-11T21:34:02.379310+0000","last_clean":"2024-03-11T21:34:02.379310+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:34:02.379310+0000","last_undegraded":"2024-03-11T21:34:02.379310+0000","last_fullsized":"2024-03-11T21:34:02.379310+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:14.693316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.693316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.736549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379457+0000","last_change":"2024-03-11T21:33:42.591082+0000","last_active":"2024-03-11T21:34:02.379457+0000","last_peered":"2024-03-11T21:34:02.379457+0000","last_clean":"2024-03-11T21:34:02.379457+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:34:02.379457+0000","last_undegraded":"2024-03-11T21:34:02.379457+0000","last_fullsized":"2024-03-11T21:34:02.379457+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:42.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.590963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.601528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":807,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389910+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:34:02.389910+0000","last_peered":"2024-03-11T21:34:02.389910+0000","last_clean":"2024-03-11T21:34:02.389910+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:34:02.389910+0000","last_undegraded":"2024-03-11T21:34:02.389910+0000","last_fullsized":"2024-03-11T21:34:02.389910+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.123454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":799,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328363+0000","last_change":"2024-03-11T21:33:08.186982+0000","last_active":"2024-03-11T21:34:02.328363+0000","last_peered":"2024-03-11T21:34:02.328363+0000","last_clean":"2024-03-11T21:34:02.328363+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:34:02.328363+0000","last_undegraded":"2024-03-11T21:34:02.328363+0000","last_fullsized":"2024-03-11T21:34:02.328363+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.186732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.186732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.992162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.382848+0000","last_change":"2024-03-11T21:33:32.575760+0000","last_active":"2024-03-11T21:34:02.382848+0000","last_peered":"2024-03-11T21:34:02.382848+0000","last_clean":"2024-03-11T21:34:02.382848+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:34:02.382848+0000","last_undegraded":"2024-03-11T21:34:02.382848+0000","last_fullsized":"2024-03-11T21:34:02.382848+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.575652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.575652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:40.490302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":786,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.368391+0000","last_change":"2024-03-11T21:33:46.058840+0000","last_active":"2024-03-11T21:34:02.368391+0000","last_peered":"2024-03-11T21:34:02.368391+0000","last_clean":"2024-03-11T21:34:02.368391+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:34:02.368391+0000","last_undegraded":"2024-03-11T21:34:02.368391+0000","last_fullsized":"2024-03-11T21:34:02.368391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.058738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.058738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.334303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380299+0000","last_change":"2024-03-11T21:30:48.791220+0000","last_active":"2024-03-11T21:34:02.380299+0000","last_peered":"2024-03-11T21:34:02.380299+0000","last_clean":"2024-03-11T21:34:02.380299+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:34:02.380299+0000","last_undegraded":"2024-03-11T21:34:02.380299+0000","last_fullsized":"2024-03-11T21:34:02.380299+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:48.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.791036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":800,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377765+0000","last_change":"2024-03-11T21:32:15.567675+0000","last_active":"2024-03-11T21:34:02.377765+0000","last_peered":"2024-03-11T21:34:02.377765+0000","last_clean":"2024-03-11T21:34:02.377765+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:34:02.377765+0000","last_undegraded":"2024-03-11T21:34:02.377765+0000","last_fullsized":"2024-03-11T21:34:02.377765+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-03-11T21:32:15.567443+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.567443+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378520+0000","last_change":"2024-03-11T21:31:59.593413+0000","last_active":"2024-03-11T21:34:02.378520+0000","last_peered":"2024-03-11T21:34:02.378520+0000","last_clean":"2024-03-11T21:34:02.378520+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:34:02.378520+0000","last_undegraded":"2024-03-11T21:34:02.378520+0000","last_fullsized":"2024-03-11T21:34:02.378520+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.593037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.593037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":782,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405214+0000","last_change":"2024-03-11T21:32:06.800643+0000","last_active":"2024-03-11T21:34:02.405214+0000","last_peered":"2024-03-11T21:34:02.405214+0000","last_clean":"2024-03-11T21:34:02.405214+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:34:02.405214+0000","last_undegraded":"2024-03-11T21:34:02.405214+0000","last_fullsized":"2024-03-11T21:34:02.405214+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.800436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.800436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":768,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391788+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:34:02.391788+0000","last_peered":"2024-03-11T21:34:02.391788+0000","last_clean":"2024-03-11T21:34:02.391788+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:34:02.391788+0000","last_undegraded":"2024-03-11T21:34:02.391788+0000","last_fullsized":"2024-03-11T21:34:02.391788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:26.346560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":619,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380683+0000","last_change":"2024-03-11T21:31:57.633633+0000","last_active":"2024-03-11T21:34:02.380683+0000","last_peered":"2024-03-11T21:34:02.380683+0000","last_clean":"2024-03-11T21:34:02.380683+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:34:02.380683+0000","last_undegraded":"2024-03-11T21:34:02.380683+0000","last_fullsized":"2024-03-11T21:34:02.380683+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.633438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.633438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":807,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391603+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:34:02.391603+0000","last_peered":"2024-03-11T21:34:02.391603+0000","last_clean":"2024-03-11T21:34:02.391603+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:34:02.391603+0000","last_undegraded":"2024-03-11T21:34:02.391603+0000","last_fullsized":"2024-03-11T21:34:02.391603+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":789,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401795+0000","last_change":"2024-03-11T21:33:10.436145+0000","last_active":"2024-03-11T21:34:02.401795+0000","last_peered":"2024-03-11T21:34:02.401795+0000","last_clean":"2024-03-11T21:34:02.401795+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:34:02.401795+0000","last_undegraded":"2024-03-11T21:34:02.401795+0000","last_fullsized":"2024-03-11T21:34:02.401795+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.436027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.436027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.260422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":925,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403727+0000","last_change":"2024-03-11T21:32:15.855148+0000","last_active":"2024-03-11T21:34:02.403727+0000","last_peered":"2024-03-11T21:34:02.403727+0000","last_clean":"2024-03-11T21:34:02.403727+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:34:02.403727+0000","last_undegraded":"2024-03-11T21:34:02.403727+0000","last_fullsized":"2024-03-11T21:34:02.403727+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:32:15.854918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.854918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":780,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.359680+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:34:03.359680+0000","last_peered":"2024-03-11T21:34:03.359680+0000","last_clean":"2024-03-11T21:34:03.359680+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:34:03.359680+0000","last_undegraded":"2024-03-11T21:34:03.359680+0000","last_fullsized":"2024-03-11T21:34:03.359680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:09.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":635,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377875+0000","last_change":"2024-03-11T21:33:59.482824+0000","last_active":"2024-03-11T21:34:02.377875+0000","last_peered":"2024-03-11T21:34:02.377875+0000","last_clean":"2024-03-11T21:34:02.377875+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:34:02.377875+0000","last_undegraded":"2024-03-11T21:34:02.377875+0000","last_fullsized":"2024-03-11T21:34:02.377875+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.434631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.434631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:39.990245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":817,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.353938+0000","last_change":"2024-03-11T21:33:42.968747+0000","last_active":"2024-03-11T21:34:03.353938+0000","last_peered":"2024-03-11T21:34:03.353938+0000","last_clean":"2024-03-11T21:34:03.353938+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:34:03.353938+0000","last_undegraded":"2024-03-11T21:34:03.353938+0000","last_fullsized":"2024-03-11T21:34:03.353938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.000934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":784,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.354870+0000","last_change":"2024-03-11T21:30:58.882981+0000","last_active":"2024-03-11T21:34:03.354870+0000","last_peered":"2024-03-11T21:34:03.354870+0000","last_clean":"2024-03-11T21:34:03.354870+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:34:03.354870+0000","last_undegraded":"2024-03-11T21:34:03.354870+0000","last_fullsized":"2024-03-11T21:34:03.354870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.882820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.882820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":915,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379587+0000","last_change":"2024-03-11T21:33:19.697889+0000","last_active":"2024-03-11T21:34:02.379587+0000","last_peered":"2024-03-11T21:34:02.379587+0000","last_clean":"2024-03-11T21:34:02.379587+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:34:02.379587+0000","last_undegraded":"2024-03-11T21:34:02.379587+0000","last_fullsized":"2024-03-11T21:34:02.379587+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:33:19.697575+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.697575+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:35.072080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":778,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328668+0000","last_change":"2024-03-11T21:33:40.730218+0000","last_active":"2024-03-11T21:34:02.328668+0000","last_peered":"2024-03-11T21:34:02.328668+0000","last_clean":"2024-03-11T21:34:02.328668+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:34:02.328668+0000","last_undegraded":"2024-03-11T21:34:02.328668+0000","last_fullsized":"2024-03-11T21:34:02.328668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.729858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.729858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.347472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":490,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.392869+0000","last_change":"2024-03-11T21:33:26.034503+0000","last_active":"2024-03-11T21:34:00.392869+0000","last_peered":"2024-03-11T21:34:00.392869+0000","last_clean":"2024-03-11T21:34:00.392869+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:34:00.392869+0000","last_undegraded":"2024-03-11T21:34:00.392869+0000","last_fullsized":"2024-03-11T21:34:00.392869+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:26.034377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:26.034377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.744337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":805,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392654+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:34:02.392654+0000","last_peered":"2024-03-11T21:34:02.392654+0000","last_clean":"2024-03-11T21:34:02.392654+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:34:02.392654+0000","last_undegraded":"2024-03-11T21:34:02.392654+0000","last_fullsized":"2024-03-11T21:34:02.392654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378980+0000","last_change":"2024-03-11T21:32:09.571388+0000","last_active":"2024-03-11T21:34:02.378980+0000","last_peered":"2024-03-11T21:34:02.378980+0000","last_clean":"2024-03-11T21:34:02.378980+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:34:02.378980+0000","last_undegraded":"2024-03-11T21:34:02.378980+0000","last_fullsized":"2024-03-11T21:34:02.378980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.571257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.571257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379319+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:34:02.379319+0000","last_peered":"2024-03-11T21:34:02.379319+0000","last_clean":"2024-03-11T21:34:02.379319+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:34:02.379319+0000","last_undegraded":"2024-03-11T21:34:02.379319+0000","last_fullsized":"2024-03-11T21:34:02.379319+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.579186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":781,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.352580+0000","last_change":"2024-03-11T21:32:06.989987+0000","last_active":"2024-03-11T21:34:03.352580+0000","last_peered":"2024-03-11T21:34:03.352580+0000","last_clean":"2024-03-11T21:34:03.352580+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:34:03.352580+0000","last_undegraded":"2024-03-11T21:34:03.352580+0000","last_fullsized":"2024-03-11T21:34:03.352580+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.989724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.989724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":636,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.358242+0000","last_change":"2024-03-11T21:33:59.486720+0000","last_active":"2024-03-11T21:34:03.358242+0000","last_peered":"2024-03-11T21:34:03.358242+0000","last_clean":"2024-03-11T21:34:03.358242+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:34:03.358242+0000","last_undegraded":"2024-03-11T21:34:03.358242+0000","last_fullsized":"2024-03-11T21:34:03.358242+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:29.057966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.057966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.682186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":816,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328133+0000","last_change":"2024-03-11T21:33:38.729069+0000","last_active":"2024-03-11T21:34:02.328133+0000","last_peered":"2024-03-11T21:34:02.328133+0000","last_clean":"2024-03-11T21:34:02.328133+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:34:02.328133+0000","last_undegraded":"2024-03-11T21:34:02.328133+0000","last_fullsized":"2024-03-11T21:34:02.328133+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:38.728913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:38.728913+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.169128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":677,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390665+0000","last_change":"2024-03-11T21:32:11.929832+0000","last_active":"2024-03-11T21:34:00.390665+0000","last_peered":"2024-03-11T21:34:00.390665+0000","last_clean":"2024-03-11T21:34:00.390665+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:34:00.390665+0000","last_undegraded":"2024-03-11T21:34:00.390665+0000","last_fullsized":"2024-03-11T21:34:00.390665+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.929628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.929628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":645,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.385535+0000","last_change":"2024-03-11T21:33:07.920722+0000","last_active":"2024-03-11T21:34:00.385535+0000","last_peered":"2024-03-11T21:34:00.385535+0000","last_clean":"2024-03-11T21:34:00.385535+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:34:00.385535+0000","last_undegraded":"2024-03-11T21:34:00.385535+0000","last_fullsized":"2024-03-11T21:34:00.385535+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:07.920509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.920509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.434359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":768,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379124+0000","last_change":"2024-03-11T21:33:35.541846+0000","last_active":"2024-03-11T21:34:02.379124+0000","last_peered":"2024-03-11T21:34:02.379124+0000","last_clean":"2024-03-11T21:34:02.379124+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:34:02.379124+0000","last_undegraded":"2024-03-11T21:34:02.379124+0000","last_fullsized":"2024-03-11T21:34:02.379124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.541731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.541731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.532157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":628,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.355605+0000","last_change":"2024-03-11T21:33:59.486095+0000","last_active":"2024-03-11T21:34:03.355605+0000","last_peered":"2024-03-11T21:34:03.355605+0000","last_clean":"2024-03-11T21:34:03.355605+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:34:03.355605+0000","last_undegraded":"2024-03-11T21:34:03.355605+0000","last_fullsized":"2024-03-11T21:34:03.355605+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.866701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.866701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.851934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":806,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.359887+0000","last_change":"2024-03-11T21:33:20.954157+0000","last_active":"2024-03-11T21:34:03.359887+0000","last_peered":"2024-03-11T21:34:03.359887+0000","last_clean":"2024-03-11T21:34:03.359887+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:34:03.359887+0000","last_undegraded":"2024-03-11T21:34:03.359887+0000","last_fullsized":"2024-03-11T21:34:03.359887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:33:20.954022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.954022+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.519773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":789,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.405677+0000","last_change":"2024-03-11T21:33:46.468750+0000","last_active":"2024-03-11T21:34:02.405677+0000","last_peered":"2024-03-11T21:34:02.405677+0000","last_clean":"2024-03-11T21:34:02.405677+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:34:02.405677+0000","last_undegraded":"2024-03-11T21:34:02.405677+0000","last_fullsized":"2024-03-11T21:34:02.405677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.468649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.468649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.584649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":782,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378307+0000","last_change":"2024-03-11T21:32:12.550833+0000","last_active":"2024-03-11T21:34:02.378307+0000","last_peered":"2024-03-11T21:34:02.378307+0000","last_clean":"2024-03-11T21:34:02.378307+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:34:02.378307+0000","last_undegraded":"2024-03-11T21:34:02.378307+0000","last_fullsized":"2024-03-11T21:34:02.378307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.550625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.550625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389889+0000","last_change":"2024-03-11T21:29:11.346981+0000","last_active":"2024-03-11T21:34:02.389889+0000","last_peered":"2024-03-11T21:34:02.389889+0000","last_clean":"2024-03-11T21:34:02.389889+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:34:02.389889+0000","last_undegraded":"2024-03-11T21:34:02.389889+0000","last_fullsized":"2024-03-11T21:34:02.389889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.346840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.346840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":631,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.353201+0000","last_change":"2024-03-11T21:32:12.998779+0000","last_active":"2024-03-11T21:34:03.353201+0000","last_peered":"2024-03-11T21:34:03.353201+0000","last_clean":"2024-03-11T21:34:03.353201+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:34:03.353201+0000","last_undegraded":"2024-03-11T21:34:03.353201+0000","last_fullsized":"2024-03-11T21:34:03.353201+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.998644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.998644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":811,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.404706+0000","last_change":"2024-03-11T21:31:59.996470+0000","last_active":"2024-03-11T21:34:02.404706+0000","last_peered":"2024-03-11T21:34:02.404706+0000","last_clean":"2024-03-11T21:34:02.404706+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:34:02.404706+0000","last_undegraded":"2024-03-11T21:34:02.404706+0000","last_fullsized":"2024-03-11T21:34:02.404706+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:59.996246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.996246+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379862+0000","last_change":"2024-03-11T21:32:28.777744+0000","last_active":"2024-03-11T21:34:02.379862+0000","last_peered":"2024-03-11T21:34:02.379862+0000","last_clean":"2024-03-11T21:34:02.379862+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:34:02.379862+0000","last_undegraded":"2024-03-11T21:34:02.379862+0000","last_fullsized":"2024-03-11T21:34:02.379862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.777629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.777629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397425+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":764,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389195+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:34:02.389195+0000","last_peered":"2024-03-11T21:34:02.389195+0000","last_clean":"2024-03-11T21:34:02.389195+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:34:02.389195+0000","last_undegraded":"2024-03-11T21:34:02.389195+0000","last_fullsized":"2024-03-11T21:34:02.389195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379449+0000","last_change":"2024-03-11T21:30:52.794455+0000","last_active":"2024-03-11T21:34:02.379449+0000","last_peered":"2024-03-11T21:34:02.379449+0000","last_clean":"2024-03-11T21:34:02.379449+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:34:02.379449+0000","last_undegraded":"2024-03-11T21:34:02.379449+0000","last_fullsized":"2024-03-11T21:34:02.379449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:52.794209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:30:52.794209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":618,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391253+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:34:02.391253+0000","last_peered":"2024-03-11T21:34:02.391253+0000","last_clean":"2024-03-11T21:34:02.391253+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:34:02.391253+0000","last_undegraded":"2024-03-11T21:34:02.391253+0000","last_fullsized":"2024-03-11T21:34:02.391253+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"389'27","reported_seq":731,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.381824+0000","last_change":"2024-03-11T21:33:21.967867+0000","last_active":"2024-03-11T21:34:00.381824+0000","last_peered":"2024-03-11T21:34:00.381824+0000","last_clean":"2024-03-11T21:34:00.381824+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:34:00.381824+0000","last_undegraded":"2024-03-11T21:34:00.381824+0000","last_fullsized":"2024-03-11T21:34:00.381824+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:21.967756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.967756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:50.206133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":787,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.400889+0000","last_change":"2024-03-11T21:32:29.159917+0000","last_active":"2024-03-11T21:34:02.400889+0000","last_peered":"2024-03-11T21:34:02.400889+0000","last_clean":"2024-03-11T21:34:02.400889+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:34:02.400889+0000","last_undegraded":"2024-03-11T21:34:02.400889+0000","last_fullsized":"2024-03-11T21:34:02.400889+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:29.159716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:29.159716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":928,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379977+0000","last_change":"2024-03-11T21:33:10.606465+0000","last_active":"2024-03-11T21:34:02.379977+0000","last_peered":"2024-03-11T21:34:02.379977+0000","last_clean":"2024-03-11T21:34:02.379977+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:34:02.379977+0000","last_undegraded":"2024-03-11T21:34:02.379977+0000","last_fullsized":"2024-03-11T21:34:02.379977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:10.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.606357+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:15.257340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":762,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391498+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:34:02.391498+0000","last_peered":"2024-03-11T21:34:02.391498+0000","last_clean":"2024-03-11T21:34:02.391498+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:34:02.391498+0000","last_undegraded":"2024-03-11T21:34:02.391498+0000","last_fullsized":"2024-03-11T21:34:02.391498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":636,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401412+0000","last_change":"2024-03-11T21:33:12.438198+0000","last_active":"2024-03-11T21:34:02.401412+0000","last_peered":"2024-03-11T21:34:02.401412+0000","last_clean":"2024-03-11T21:34:02.401412+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:34:02.401412+0000","last_undegraded":"2024-03-11T21:34:02.401412+0000","last_fullsized":"2024-03-11T21:34:02.401412+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.438028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.438028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:37.544269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":810,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377922+0000","last_change":"2024-03-11T21:31:54.548395+0000","last_active":"2024-03-11T21:34:02.377922+0000","last_peered":"2024-03-11T21:34:02.377922+0000","last_clean":"2024-03-11T21:34:02.377922+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:34:02.377922+0000","last_undegraded":"2024-03-11T21:34:02.377922+0000","last_fullsized":"2024-03-11T21:34:02.377922+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:54.548275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.548275+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379247+0000","last_change":"2024-03-11T21:33:06.710223+0000","last_active":"2024-03-11T21:34:02.379247+0000","last_peered":"2024-03-11T21:34:02.379247+0000","last_clean":"2024-03-11T21:34:02.379247+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:34:02.379247+0000","last_undegraded":"2024-03-11T21:34:02.379247+0000","last_fullsized":"2024-03-11T21:34:02.379247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:06.709989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.709989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:32.785762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":777,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.359257+0000","last_change":"2024-03-11T21:32:23.014073+0000","last_active":"2024-03-11T21:34:03.359257+0000","last_peered":"2024-03-11T21:34:03.359257+0000","last_clean":"2024-03-11T21:34:03.359257+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:34:03.359257+0000","last_undegraded":"2024-03-11T21:34:03.359257+0000","last_fullsized":"2024-03-11T21:34:03.359257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.013984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.013984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":635,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.384796+0000","last_change":"2024-03-11T21:33:05.875547+0000","last_active":"2024-03-11T21:34:00.384796+0000","last_peered":"2024-03-11T21:34:00.384796+0000","last_clean":"2024-03-11T21:34:00.384796+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:34:00.384796+0000","last_undegraded":"2024-03-11T21:34:00.384796+0000","last_fullsized":"2024-03-11T21:34:00.384796+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:05.875434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.875434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.539249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":618,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389302+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:34:02.389302+0000","last_peered":"2024-03-11T21:34:02.389302+0000","last_clean":"2024-03-11T21:34:02.389302+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:34:02.389302+0000","last_undegraded":"2024-03-11T21:34:02.389302+0000","last_fullsized":"2024-03-11T21:34:02.389302+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":795,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.355173+0000","last_change":"2024-03-11T21:33:19.017659+0000","last_active":"2024-03-11T21:34:03.355173+0000","last_peered":"2024-03-11T21:34:03.355173+0000","last_clean":"2024-03-11T21:34:03.355173+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:34:03.355173+0000","last_undegraded":"2024-03-11T21:34:03.355173+0000","last_fullsized":"2024-03-11T21:34:03.355173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:33:19.017449+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.017449+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.631706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":790,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.377912+0000","last_change":"2024-03-11T21:32:10.547894+0000","last_active":"2024-03-11T21:34:02.377912+0000","last_peered":"2024-03-11T21:34:02.377912+0000","last_clean":"2024-03-11T21:34:02.377912+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:34:02.377912+0000","last_undegraded":"2024-03-11T21:34:02.377912+0000","last_fullsized":"2024-03-11T21:34:02.377912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.547682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.547682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:05.397375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":629,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.390539+0000","last_change":"2024-03-11T21:33:36.120790+0000","last_active":"2024-03-11T21:34:00.390539+0000","last_peered":"2024-03-11T21:34:00.390539+0000","last_clean":"2024-03-11T21:34:00.390539+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:34:00.390539+0000","last_undegraded":"2024-03-11T21:34:00.390539+0000","last_fullsized":"2024-03-11T21:34:00.390539+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.120602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.120602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.298821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":779,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328318+0000","last_change":"2024-03-11T21:33:41.774829+0000","last_active":"2024-03-11T21:34:02.328318+0000","last_peered":"2024-03-11T21:34:02.328318+0000","last_clean":"2024-03-11T21:34:02.328318+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:34:02.328318+0000","last_undegraded":"2024-03-11T21:34:02.328318+0000","last_fullsized":"2024-03-11T21:34:02.328318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.774699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.774699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.521322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":626,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.325787+0000","last_change":"2024-03-11T21:33:59.391439+0000","last_active":"2024-03-11T21:34:02.325787+0000","last_peered":"2024-03-11T21:34:02.325787+0000","last_clean":"2024-03-11T21:34:02.325787+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:34:02.325787+0000","last_undegraded":"2024-03-11T21:34:02.325787+0000","last_fullsized":"2024-03-11T21:34:02.325787+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.655683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":820,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327082+0000","last_change":"2024-03-11T21:33:27.686917+0000","last_active":"2024-03-11T21:34:02.327082+0000","last_peered":"2024-03-11T21:34:02.327082+0000","last_clean":"2024-03-11T21:34:02.327082+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:34:02.327082+0000","last_undegraded":"2024-03-11T21:34:02.327082+0000","last_fullsized":"2024-03-11T21:34:02.327082+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:33:27.686783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.686783+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:30.924138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":780,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.379080+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:34:02.379080+0000","last_peered":"2024-03-11T21:34:02.379080+0000","last_clean":"2024-03-11T21:34:02.379080+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:34:02.379080+0000","last_undegraded":"2024-03-11T21:34:02.379080+0000","last_fullsized":"2024-03-11T21:34:02.379080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":774,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.328313+0000","last_change":"2024-03-11T21:32:09.768728+0000","last_active":"2024-03-11T21:34:02.328313+0000","last_peered":"2024-03-11T21:34:02.328313+0000","last_clean":"2024-03-11T21:34:02.328313+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:34:02.328313+0000","last_undegraded":"2024-03-11T21:34:02.328313+0000","last_fullsized":"2024-03-11T21:34:02.328313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.768608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.768608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":761,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.382718+0000","last_change":"2024-03-11T21:30:36.777188+0000","last_active":"2024-03-11T21:34:02.382718+0000","last_peered":"2024-03-11T21:34:02.382718+0000","last_clean":"2024-03-11T21:34:02.382718+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:34:02.382718+0000","last_undegraded":"2024-03-11T21:34:02.382718+0000","last_fullsized":"2024-03-11T21:34:02.382718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:36.777107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:30:36.777107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.519966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.393260+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:34:02.393260+0000","last_peered":"2024-03-11T21:34:02.393260+0000","last_clean":"2024-03-11T21:34:02.393260+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:34:02.393260+0000","last_undegraded":"2024-03-11T21:34:02.393260+0000","last_fullsized":"2024-03-11T21:34:02.393260+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":809,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.402904+0000","last_change":"2024-03-11T21:32:09.759891+0000","last_active":"2024-03-11T21:34:02.402904+0000","last_peered":"2024-03-11T21:34:02.402904+0000","last_clean":"2024-03-11T21:34:02.402904+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:34:02.402904+0000","last_undegraded":"2024-03-11T21:34:02.402904+0000","last_fullsized":"2024-03-11T21:34:02.402904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:32:09.759634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.759634+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":780,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389985+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:34:02.389985+0000","last_peered":"2024-03-11T21:34:02.389985+0000","last_clean":"2024-03-11T21:34:02.389985+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:34:02.389985+0000","last_undegraded":"2024-03-11T21:34:02.389985+0000","last_fullsized":"2024-03-11T21:34:02.389985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":773,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.403804+0000","last_change":"2024-03-11T21:33:17.530575+0000","last_active":"2024-03-11T21:34:02.403804+0000","last_peered":"2024-03-11T21:34:02.403804+0000","last_clean":"2024-03-11T21:34:02.403804+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:34:02.403804+0000","last_undegraded":"2024-03-11T21:34:02.403804+0000","last_fullsized":"2024-03-11T21:34:02.403804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:17.530299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.530299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:29.486176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":781,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.353161+0000","last_change":"2024-03-11T21:32:01.014273+0000","last_active":"2024-03-11T21:34:03.353161+0000","last_peered":"2024-03-11T21:34:03.353161+0000","last_clean":"2024-03-11T21:34:03.353161+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:34:03.353161+0000","last_undegraded":"2024-03-11T21:34:03.353161+0000","last_fullsized":"2024-03-11T21:34:03.353161+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.014168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.014168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":629,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.327938+0000","last_change":"2024-03-11T21:32:15.652648+0000","last_active":"2024-03-11T21:34:02.327938+0000","last_peered":"2024-03-11T21:34:02.327938+0000","last_clean":"2024-03-11T21:34:02.327938+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:34:02.327938+0000","last_undegraded":"2024-03-11T21:34:02.327938+0000","last_fullsized":"2024-03-11T21:34:02.327938+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.652323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.652323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":844,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.404781+0000","last_change":"2024-03-11T21:33:20.542248+0000","last_active":"2024-03-11T21:34:02.404781+0000","last_peered":"2024-03-11T21:34:02.404781+0000","last_clean":"2024-03-11T21:34:02.404781+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:34:02.404781+0000","last_undegraded":"2024-03-11T21:34:02.404781+0000","last_fullsized":"2024-03-11T21:34:02.404781+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:33:20.542001+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.542001+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.858604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":780,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.392355+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:34:02.392355+0000","last_peered":"2024-03-11T21:34:02.392355+0000","last_clean":"2024-03-11T21:34:02.392355+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:34:02.392355+0000","last_undegraded":"2024-03-11T21:34:02.392355+0000","last_fullsized":"2024-03-11T21:34:02.392355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":773,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.389589+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:34:02.389589+0000","last_peered":"2024-03-11T21:34:02.389589+0000","last_clean":"2024-03-11T21:34:02.389589+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:34:02.389589+0000","last_undegraded":"2024-03-11T21:34:02.389589+0000","last_fullsized":"2024-03-11T21:34:02.389589+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":778,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.401802+0000","last_change":"2024-03-11T21:33:23.549947+0000","last_active":"2024-03-11T21:34:02.401802+0000","last_peered":"2024-03-11T21:34:02.401802+0000","last_clean":"2024-03-11T21:34:02.401802+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:34:02.401802+0000","last_undegraded":"2024-03-11T21:34:02.401802+0000","last_fullsized":"2024-03-11T21:34:02.401802+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.549773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.549773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:34.175330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":627,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.391497+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:34:02.391497+0000","last_peered":"2024-03-11T21:34:02.391497+0000","last_clean":"2024-03-11T21:34:02.391497+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:34:02.391497+0000","last_undegraded":"2024-03-11T21:34:02.391497+0000","last_fullsized":"2024-03-11T21:34:02.391497+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":825,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.361627+0000","last_change":"2024-03-11T21:33:39.970299+0000","last_active":"2024-03-11T21:34:03.361627+0000","last_peered":"2024-03-11T21:34:03.361627+0000","last_clean":"2024-03-11T21:34:03.361627+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:34:03.361627+0000","last_undegraded":"2024-03-11T21:34:03.361627+0000","last_fullsized":"2024-03-11T21:34:03.361627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:39.970169+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.970169+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:41.161902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":773,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380524+0000","last_change":"2024-03-11T21:30:30.892128+0000","last_active":"2024-03-11T21:34:02.380524+0000","last_peered":"2024-03-11T21:34:02.380524+0000","last_clean":"2024-03-11T21:34:02.380524+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:34:02.380524+0000","last_undegraded":"2024-03-11T21:34:02.380524+0000","last_fullsized":"2024-03-11T21:34:02.380524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:30.891767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:30.891767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":626,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.382236+0000","last_change":"2024-03-11T21:32:02.053080+0000","last_active":"2024-03-11T21:34:00.382236+0000","last_peered":"2024-03-11T21:34:00.382236+0000","last_clean":"2024-03-11T21:34:00.382236+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:34:00.382236+0000","last_undegraded":"2024-03-11T21:34:00.382236+0000","last_fullsized":"2024-03-11T21:34:00.382236+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.052865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.052865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:03.260444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":779,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.326889+0000","last_change":"2024-03-11T21:32:07.733765+0000","last_active":"2024-03-11T21:34:02.326889+0000","last_peered":"2024-03-11T21:34:02.326889+0000","last_clean":"2024-03-11T21:34:02.326889+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:34:02.326889+0000","last_undegraded":"2024-03-11T21:34:02.326889+0000","last_fullsized":"2024-03-11T21:34:02.326889+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.733556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.733556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":484,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-11T21:34:00.394490+0000","last_change":"2024-03-11T21:33:22.998783+0000","last_active":"2024-03-11T21:34:00.394490+0000","last_peered":"2024-03-11T21:34:00.394490+0000","last_clean":"2024-03-11T21:34:00.394490+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:34:00.394490+0000","last_undegraded":"2024-03-11T21:34:00.394490+0000","last_fullsized":"2024-03-11T21:34:00.394490+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:33.468087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":807,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.380653+0000","last_change":"2024-03-11T21:33:17.733152+0000","last_active":"2024-03-11T21:34:02.380653+0000","last_peered":"2024-03-11T21:34:02.380653+0000","last_clean":"2024-03-11T21:34:02.380653+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:34:02.380653+0000","last_undegraded":"2024-03-11T21:34:02.380653+0000","last_fullsized":"2024-03-11T21:34:02.380653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:17.733068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.733068+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:25.427374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":794,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-03-11T21:34:03.361926+0000","last_change":"2024-03-11T21:33:16.013881+0000","last_active":"2024-03-11T21:34:03.361926+0000","last_peered":"2024-03-11T21:34:03.361926+0000","last_clean":"2024-03-11T21:34:03.361926+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:34:03.361926+0000","last_undegraded":"2024-03-11T21:34:03.361926+0000","last_fullsized":"2024-03-11T21:34:03.361926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:33:16.013752+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:33:16.013752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:31.774437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":783,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.325961+0000","last_change":"2024-03-11T21:31:58.673043+0000","last_active":"2024-03-11T21:34:02.325961+0000","last_peered":"2024-03-11T21:34:02.325961+0000","last_clean":"2024-03-11T21:34:02.325961+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:34:02.325961+0000","last_undegraded":"2024-03-11T21:34:02.325961+0000","last_fullsized":"2024-03-11T21:34:02.325961+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.672796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.672796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.841428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-03-11T21:34:02.378938+0000","last_change":"2024-03-11T21:32:10.686985+0000","last_active":"2024-03-11T21:34:02.378938+0000","last_peered":"2024-03-11T21:34:02.378938+0000","last_clean":"2024-03-11T21:34:02.378938+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:34:02.378938+0000","last_undegraded":"2024-03-11T21:34:02.378938+0000","last_fullsized":"2024-03-11T21:34:02.378938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.686767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.686767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:04.520380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":281,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195692,"num_objects":145,"num_object_clones":2,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":103,"num_read_kb":7,"num_write":572,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26333184,"data_stored":24587076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1826,"ondisk_log_size":1826,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":639,"num_read_kb":634,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841679,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79340,"kb_used_data":12044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665812,"statfs":{"total":95995035648,"available":95913791488,"internally_reserved":0,"allocated":12333056,"data_stored":9345331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168728,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87308,"kb_used_data":11692,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657844,"statfs":{"total":95995035648,"available":95905632256,"internally_reserved":0,"allocated":11972608,"data_stored":9137090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352350,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47592,"kb_used_data":7188,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93697560,"statfs":{"total":95995035648,"available":95946301440,"internally_reserved":0,"allocated":7360512,"data_stored":4525010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83100,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93662052,"statfs":{"total":95995035648,"available":95909941248,"internally_reserved":0,"allocated":16183296,"data_stored":13343968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378818,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70460,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674692,"statfs":{"total":95995035648,"available":95922884608,"internally_reserved":0,"allocated":7487488,"data_stored":4558847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117174,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70424,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674728,"statfs":{"total":95995035648,"available":95922921472,"internally_reserved":0,"allocated":7462912,"data_stored":4558306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855532,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78576,"kb_used_data":11280,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666576,"statfs":{"total":95995035648,"available":95914573824,"internally_reserved":0,"allocated":11550720,"data_stored":8559882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495778,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91108,"kb_used_data":11320,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93654044,"statfs":{"total":95995035648,"available":95901741056,"internally_reserved":0,"allocated":11591680,"data_stored":8752444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":8194697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4000689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4370432,"data_stored":4194738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4000398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4476928,"data_stored":4194952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:34:07.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:34:07.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:06.390714+0000 mon.a (mon.0) 2671 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:34:07.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: cluster 2024-03-11T21:34:06.394160+0000 mon.a (mon.0) 2672 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:34:07.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:06.396110+0000 mon.c (mon.2) 917 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:06.402077+0000 mon.a (mon.0) 2673 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:07.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:06.403200+0000 mon.a (mon.0) 2674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:06.403871+0000 mon.c (mon.2) 918 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:07.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:07 smithi203 bash[23654]: audit 2024-03-11T21:34:07.268675+0000 mon.a (mon.0) 2675 : audit 0 from='client.? 172.21.15.143:0/3437733835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:07.521 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- 172.21.15.143:0/3631408788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b01005b0 msgr2=0x7f04b0100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/3631408788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b01005b0 0x7f04b0100990 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f04a00099c0 tx=0x7f04a0009cd0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- 172.21.15.143:0/3631408788 shutdown_connections 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/3631408788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01071d0 0x7f04b010ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/3631408788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b01005b0 0x7f04b0100990 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/3631408788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b0100f60 0x7f04b0106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- 172.21.15.143:0/3631408788 >> 172.21.15.143:0/3631408788 conn(0x7f04b00772e0 msgr2=0x7f04b00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- 172.21.15.143:0/3631408788 shutdown_connections 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- 172.21.15.143:0/3631408788 wait complete. 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 librados: starting msgr at 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 librados: starting objecter 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 Processor -- start 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- start start 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 librados: setting wanted keys 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 librados: calling monclient init 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 0x7f04b0101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 0x7f04b0101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0111650 con 0x7f04b01005b0 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01114d0 con 0x7f04b01071d0 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04b5dd5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01117d0 con 0x7f04b0100f60 2024-03-11T21:34:07.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35536/0 (socket says 172.21.15.143:35536) 2024-03-11T21:34:07.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 -- 172.21.15.143:0/2697655698 learned_addr learned my addr 172.21.15.143:0/2697655698 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:07.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04aeffd700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 0x7f04b0101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04affff700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 0x7f04b0101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 -- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 msgr2=0x7f04b0101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 0x7f04b0101a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 -- 172.21.15.143:0/2697655698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 msgr2=0x7f04b0101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:07.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 0x7f04b0101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:07.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04af7fe700 1 -- 172.21.15.143:0/2697655698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b01a6170 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04aeffd700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 0x7f04b0101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.600+0000 7f04affff700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 0x7f04b0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04af7fe700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005910 tx=0x7f04a000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 10 client.?.objecter ms_handle_connect 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0027070 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01a6400 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f04a000fc90 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0022c90 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04b5dd5700 10 client.30417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01a68b0 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04b5dd5700 1 librados: init done 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b004faa0 con 0x7f04b01005b0 2024-03-11T21:34:07.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f04a002a070 con 0x7f04b01005b0 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 0x7f0498071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(446..446 src has 87..446) v4 ==== 9621+0+0 (secure 0 0 0) 0x7f04a008f880 con 0x7f04b01005b0 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 10 client.30417.objecter ms_dispatch 0x7f04b0001550 osd_map(446..446 src has 87..446) v4 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 3 client.30417.objecter handle_osd_map got epochs [446,446] > 0 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 3 client.30417.objecter handle_osd_map decoding full epoch 446 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04aeffd700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 0x7f0498071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 20 client.30417.objecter dump_active .. 0 homeless 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04acff9700 10 client.30417.objecter ms_handle_connect 0x7f049806f820 2024-03-11T21:34:07.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.604+0000 7f04aeffd700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 0x7f0498071ce0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f04b00ffce0 tx=0x7f04a400a300 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:07.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.608+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f04a0013070 con 0x7f04b01005b0 2024-03-11T21:34:07.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:07.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:06.390714+0000 mon.a (mon.0) 2671 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:34:07.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: cluster 2024-03-11T21:34:06.394160+0000 mon.a (mon.0) 2672 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:34:07.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:06.396110+0000 mon.c (mon.2) 917 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.633 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:06.402077+0000 mon.a (mon.0) 2673 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:06.403200+0000 mon.a (mon.0) 2674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:06.403871+0000 mon.c (mon.2) 918 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[21512]: audit 2024-03-11T21:34:07.268675+0000 mon.a (mon.0) 2675 : audit 0 from='client.? 172.21.15.143:0/3437733835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:06.390714+0000 mon.a (mon.0) 2671 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: cluster 2024-03-11T21:34:06.394160+0000 mon.a (mon.0) 2672 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:06.396110+0000 mon.c (mon.2) 917 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:06.402077+0000 mon.a (mon.0) 2673 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:06.403200+0000 mon.a (mon.0) 2674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:06.403871+0000 mon.c (mon.2) 918 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:07.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:07 smithi143 bash[29284]: audit 2024-03-11T21:34:07.268675+0000 mon.a (mon.0) 2675 : audit 0 from='client.? 172.21.15.143:0/3437733835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:07.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:07.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:07.771 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:07.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:07.832+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f04b01a6bd0 con 0x7f04b01005b0 2024-03-11T21:34:07.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:07.899 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:08.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:08.028 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:08.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:08.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:08.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:08.292 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:08.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.408+0000 7f04acff9700 1 -- 172.21.15.143:0/2697655698 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v447) v1 ==== 97+0+0 (secure 0 0 0) 0x7f04a005d2f0 con 0x7f04b01005b0 2024-03-11T21:34:08.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.412+0000 7f04b5dd5700 10 librados: watch_flush enter 2024-03-11T21:34:08.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.412+0000 7f04b5dd5700 10 librados: watch_flush exit 2024-03-11T21:34:08.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.412+0000 7f04b5dd5700 20 client.30417.objecter shutdown clearing up homeless session... 2024-03-11T21:34:08.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.412+0000 7f04b5dd5700 10 client.30417.objecter successfully canceled tick 2024-03-11T21:34:08.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.416+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 msgr2=0x7f0498071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:08.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.416+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 0x7f0498071ce0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f04b00ffce0 tx=0x7f04a400a300 comp rx=0 tx=0).stop 2024-03-11T21:34:08.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.416+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 msgr2=0x7f04b0101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:08.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.416+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005910 tx=0x7f04a000fa00 comp rx=0 tx=0).stop 2024-03-11T21:34:08.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:08.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 shutdown_connections 2024-03-11T21:34:08.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f049806f820 0x7f0498071ce0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b01005b0 0x7f04b0101530 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04b01071d0 0x7f04b0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 --2- 172.21.15.143:0/2697655698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04b0100f60 0x7f04b0101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 >> 172.21.15.143:0/2697655698 conn(0x7f04b00772e0 msgr2=0x7f04b0104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 shutdown_connections 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.420+0000 7f04b5dd5700 1 -- 172.21.15.143:0/2697655698 wait complete. 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:08.424+0000 7f04b5dd5700 1 librados: shutdown 2024-03-11T21:34:08.400 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: cluster 2024-03-11T21:34:07.378987+0000 mgr.y (mgr.14550) 443 : cluster 0 pgmap v656: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s wr, 0 op/s 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: audit 2024-03-11T21:34:07.402207+0000 mon.a (mon.0) 2676 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: cluster 2024-03-11T21:34:07.407484+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: audit 2024-03-11T21:34:07.411918+0000 mon.c (mon.2) 919 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: audit 2024-03-11T21:34:07.417610+0000 mon.a (mon.0) 2678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[29284]: audit 2024-03-11T21:34:07.839123+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: cluster 2024-03-11T21:34:07.378987+0000 mgr.y (mgr.14550) 443 : cluster 0 pgmap v656: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s wr, 0 op/s 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: audit 2024-03-11T21:34:07.402207+0000 mon.a (mon.0) 2676 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: cluster 2024-03-11T21:34:07.407484+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: audit 2024-03-11T21:34:07.411918+0000 mon.c (mon.2) 919 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: audit 2024-03-11T21:34:07.417610+0000 mon.a (mon.0) 2678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:08 smithi143 bash[21512]: audit 2024-03-11T21:34:07.839123+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:08.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:08.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: cluster 2024-03-11T21:34:07.378987+0000 mgr.y (mgr.14550) 443 : cluster 0 pgmap v656: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s wr, 0 op/s 2024-03-11T21:34:08.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: audit 2024-03-11T21:34:07.402207+0000 mon.a (mon.0) 2676 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:08.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: cluster 2024-03-11T21:34:07.407484+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:34:08.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: audit 2024-03-11T21:34:07.411918+0000 mon.c (mon.2) 919 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: audit 2024-03-11T21:34:07.417610+0000 mon.a (mon.0) 2678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]: dispatch 2024-03-11T21:34:08.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[23654]: audit 2024-03-11T21:34:07.839123+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:08.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:08.553 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:08.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:08.667 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:08.683 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:08.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:08.813 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:08.916 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:08.945 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:09.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:09.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:09.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.206 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:09.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:09.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:09.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: cluster 2024-03-11T21:34:08.412933+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: cluster 2024-03-11T21:34:08.412992+0000 mon.a (mon.0) 2681 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: audit 2024-03-11T21:34:08.414942+0000 mon.a (mon.0) 2682 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]': finished 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: audit 2024-03-11T21:34:08.415209+0000 mon.a (mon.0) 2683 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: cluster 2024-03-11T21:34:08.424659+0000 mon.a (mon.0) 2684 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: audit 2024-03-11T21:34:08.496973+0000 mon.c (mon.2) 920 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: audit 2024-03-11T21:34:08.497309+0000 mon.a (mon.0) 2685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[21512]: audit 2024-03-11T21:34:08.609809+0000 mgr.y (mgr.14550) 444 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: cluster 2024-03-11T21:34:08.412933+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:34:09.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: cluster 2024-03-11T21:34:08.412992+0000 mon.a (mon.0) 2681 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: audit 2024-03-11T21:34:08.414942+0000 mon.a (mon.0) 2682 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]': finished 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: audit 2024-03-11T21:34:08.415209+0000 mon.a (mon.0) 2683 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: cluster 2024-03-11T21:34:08.424659+0000 mon.a (mon.0) 2684 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: audit 2024-03-11T21:34:08.496973+0000 mon.c (mon.2) 920 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: audit 2024-03-11T21:34:08.497309+0000 mon.a (mon.0) 2685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:09 smithi143 bash[29284]: audit 2024-03-11T21:34:08.609809+0000 mgr.y (mgr.14550) 444 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:09.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:09.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:09.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:09.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:09.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: cluster 2024-03-11T21:34:08.412933+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: cluster 2024-03-11T21:34:08.412992+0000 mon.a (mon.0) 2681 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: audit 2024-03-11T21:34:08.414942+0000 mon.a (mon.0) 2682 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-51", "mode": "writeback"}]': finished 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: audit 2024-03-11T21:34:08.415209+0000 mon.a (mon.0) 2683 : audit 1 from='client.? 172.21.15.143:0/2697655698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: cluster 2024-03-11T21:34:08.424659+0000 mon.a (mon.0) 2684 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: audit 2024-03-11T21:34:08.496973+0000 mon.c (mon.2) 920 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: audit 2024-03-11T21:34:08.497309+0000 mon.a (mon.0) 2685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:34:09.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:09 smithi203 bash[23654]: audit 2024-03-11T21:34:08.609809+0000 mgr.y (mgr.14550) 444 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:09.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.859 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:09.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:09.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:10.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.118 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:10.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:10.245 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:10.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:10.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:10.507 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:10.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:10 smithi203 bash[23654]: cluster 2024-03-11T21:34:09.380202+0000 mgr.y (mgr.14550) 445 : cluster 0 pgmap v659: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 763 B/s wr, 2 op/s 2024-03-11T21:34:10.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:10 smithi203 bash[23654]: audit 2024-03-11T21:34:09.418837+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:34:10.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:10 smithi203 bash[23654]: cluster 2024-03-11T21:34:09.426682+0000 mon.a (mon.0) 2687 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:34:10.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:10 smithi203 bash[23654]: audit 2024-03-11T21:34:09.435736+0000 mon.c (mon.2) 921 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:10 smithi203 bash[23654]: audit 2024-03-11T21:34:09.452581+0000 mon.a (mon.0) 2688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:10.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:10.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[21512]: cluster 2024-03-11T21:34:09.380202+0000 mgr.y (mgr.14550) 445 : cluster 0 pgmap v659: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 763 B/s wr, 2 op/s 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[21512]: audit 2024-03-11T21:34:09.418837+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[21512]: cluster 2024-03-11T21:34:09.426682+0000 mon.a (mon.0) 2687 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[21512]: audit 2024-03-11T21:34:09.435736+0000 mon.c (mon.2) 921 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[21512]: audit 2024-03-11T21:34:09.452581+0000 mon.a (mon.0) 2688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[29284]: cluster 2024-03-11T21:34:09.380202+0000 mgr.y (mgr.14550) 445 : cluster 0 pgmap v659: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 763 B/s wr, 2 op/s 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[29284]: audit 2024-03-11T21:34:09.418837+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[29284]: cluster 2024-03-11T21:34:09.426682+0000 mon.a (mon.0) 2687 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[29284]: audit 2024-03-11T21:34:09.435736+0000 mon.c (mon.2) 921 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:10 smithi143 bash[29284]: audit 2024-03-11T21:34:09.452581+0000 mon.a (mon.0) 2688 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:34:10.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:10.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:34:10.769 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:10.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:10.899 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:11.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:11.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:11.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.284 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:11.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.414 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:11.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:11.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:34:11.546 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:11.556 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:10.440173+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:34:11.556 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: cluster 2024-03-11T21:34:10.449594+0000 mon.a (mon.0) 2690 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:34:11.556 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:10.451619+0000 mon.c (mon.2) 922 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.556 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:10.453834+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: cluster 2024-03-11T21:34:11.310416+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: cluster 2024-03-11T21:34:11.310758+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: cluster 2024-03-11T21:34:11.322810+0000 mon.a (mon.0) 2692 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:11.324629+0000 mon.a (mon.0) 2693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: cluster 2024-03-11T21:34:11.331833+0000 mon.a (mon.0) 2694 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:11.334205+0000 mon.c (mon.2) 923 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[29284]: audit 2024-03-11T21:34:11.340932+0000 mon.a (mon.0) 2695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:10.440173+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: cluster 2024-03-11T21:34:10.449594+0000 mon.a (mon.0) 2690 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:10.451619+0000 mon.c (mon.2) 922 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.557 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:10.453834+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: cluster 2024-03-11T21:34:11.310416+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: cluster 2024-03-11T21:34:11.310758+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: cluster 2024-03-11T21:34:11.322810+0000 mon.a (mon.0) 2692 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:11.324629+0000 mon.a (mon.0) 2693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: cluster 2024-03-11T21:34:11.331833+0000 mon.a (mon.0) 2694 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:11.334205+0000 mon.c (mon.2) 923 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.558 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:11 smithi143 bash[21512]: audit 2024-03-11T21:34:11.340932+0000 mon.a (mon.0) 2695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:11.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:10.440173+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:34:11.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: cluster 2024-03-11T21:34:10.449594+0000 mon.a (mon.0) 2690 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:34:11.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:10.451619+0000 mon.c (mon.2) 922 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:10.453834+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:34:11.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: cluster 2024-03-11T21:34:11.310416+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: cluster 2024-03-11T21:34:11.310758+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: cluster 2024-03-11T21:34:11.322810+0000 mon.a (mon.0) 2692 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:11.324629+0000 mon.a (mon.0) 2693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: cluster 2024-03-11T21:34:11.331833+0000 mon.a (mon.0) 2694 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:11.334205+0000 mon.c (mon.2) 923 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:11 smithi203 bash[23654]: audit 2024-03-11T21:34:11.340932+0000 mon.a (mon.0) 2695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:34:11.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:11.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:11.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:11.935 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:12.025 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:12.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:12.067 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:12.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:12.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:12.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:12.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:12.331 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:12.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:12.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:12.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:12.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21512]: cluster 2024-03-11T21:34:11.380955+0000 mgr.y (mgr.14550) 446 : cluster 0 pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 773 B/s wr, 3 op/s 2024-03-11T21:34:12.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21512]: audit 2024-03-11T21:34:12.338826+0000 mon.a (mon.0) 2696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:34:12.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21512]: cluster 2024-03-11T21:34:12.349460+0000 mon.a (mon.0) 2697 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:34:12.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21512]: audit 2024-03-11T21:34:12.353591+0000 mon.c (mon.2) 924 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.577 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[21512]: audit 2024-03-11T21:34:12.358282+0000 mon.a (mon.0) 2698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.578 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[29284]: cluster 2024-03-11T21:34:11.380955+0000 mgr.y (mgr.14550) 446 : cluster 0 pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 773 B/s wr, 3 op/s 2024-03-11T21:34:12.578 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[29284]: audit 2024-03-11T21:34:12.338826+0000 mon.a (mon.0) 2696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:34:12.578 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[29284]: cluster 2024-03-11T21:34:12.349460+0000 mon.a (mon.0) 2697 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:34:12.578 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[29284]: audit 2024-03-11T21:34:12.353591+0000 mon.c (mon.2) 924 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.578 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:12 smithi143 bash[29284]: audit 2024-03-11T21:34:12.358282+0000 mon.a (mon.0) 2698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:12.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:12.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:12 smithi203 bash[23654]: cluster 2024-03-11T21:34:11.380955+0000 mgr.y (mgr.14550) 446 : cluster 0 pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 773 B/s wr, 3 op/s 2024-03-11T21:34:12.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:12 smithi203 bash[23654]: audit 2024-03-11T21:34:12.338826+0000 mon.a (mon.0) 2696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:34:12.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:12 smithi203 bash[23654]: cluster 2024-03-11T21:34:12.349460+0000 mon.a (mon.0) 2697 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:34:12.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:12 smithi203 bash[23654]: audit 2024-03-11T21:34:12.353591+0000 mon.c (mon.2) 924 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:12 smithi203 bash[23654]: audit 2024-03-11T21:34:12.358282+0000 mon.a (mon.0) 2698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:34:12.722 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:12.825 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:12.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:12.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:12.989 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:13.041 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:13.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:13.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.258 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:13.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:13.389 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:13.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:13.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:13.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.651 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:13.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:13.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:13.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:13.913 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:14.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:14.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:14.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:14.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:14.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:14.306 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:14.344 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[29284]: cluster 2024-03-11T21:34:13.259445+0000 osd.3 (osd.3) 265 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:14.344 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[29284]: cluster 2024-03-11T21:34:13.259733+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:14.344 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[29284]: audit 2024-03-11T21:34:13.357541+0000 mon.a (mon.0) 2699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:34:14.344 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[29284]: cluster 2024-03-11T21:34:13.365304+0000 mon.a (mon.0) 2700 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:34:14.345 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[29284]: cluster 2024-03-11T21:34:13.381887+0000 mgr.y (mgr.14550) 447 : cluster 0 pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 259 B/s wr, 2 op/s 2024-03-11T21:34:14.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:14.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:14.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:14 smithi203 bash[23654]: cluster 2024-03-11T21:34:13.259445+0000 osd.3 (osd.3) 265 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:14.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:14 smithi203 bash[23654]: cluster 2024-03-11T21:34:13.259733+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:14.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:14 smithi203 bash[23654]: audit 2024-03-11T21:34:13.357541+0000 mon.a (mon.0) 2699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:34:14.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:14 smithi203 bash[23654]: cluster 2024-03-11T21:34:13.365304+0000 mon.a (mon.0) 2700 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:34:14.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:14 smithi203 bash[23654]: cluster 2024-03-11T21:34:13.381887+0000 mgr.y (mgr.14550) 447 : cluster 0 pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 259 B/s wr, 2 op/s 2024-03-11T21:34:14.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:14.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:14.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[21512]: cluster 2024-03-11T21:34:13.259445+0000 osd.3 (osd.3) 265 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:14.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[21512]: cluster 2024-03-11T21:34:13.259733+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:14.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[21512]: audit 2024-03-11T21:34:13.357541+0000 mon.a (mon.0) 2699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:34:14.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[21512]: cluster 2024-03-11T21:34:13.365304+0000 mon.a (mon.0) 2700 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:34:14.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:14 smithi143 bash[21512]: cluster 2024-03-11T21:34:13.381887+0000 mgr.y (mgr.14550) 447 : cluster 0 pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 259 B/s wr, 2 op/s 2024-03-11T21:34:14.670 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:14.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:14.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:14.826 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:14.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:14.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:15.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:15.072 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:15.081 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:15.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:15.214 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:15.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:15.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:15.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:15.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:15.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:15 smithi203 bash[23654]: cluster 2024-03-11T21:34:14.357537+0000 mon.a (mon.0) 2701 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:34:15.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:15 smithi143 bash[29284]: cluster 2024-03-11T21:34:14.357537+0000 mon.a (mon.0) 2701 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:34:15.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:15 smithi143 bash[21512]: cluster 2024-03-11T21:34:14.357537+0000 mon.a (mon.0) 2701 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:34:15.608 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:15.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.738 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:15.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.869 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:15.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:16.000 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:16.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:16.128 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:16.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.261 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:16.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:16.391 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:16.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.506 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:16 smithi143 bash[21512]: cluster 2024-03-11T21:34:15.383081+0000 mgr.y (mgr.14550) 448 : cluster 0 pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:16.506 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:16 smithi143 bash[29284]: cluster 2024-03-11T21:34:15.383081+0000 mgr.y (mgr.14550) 448 : cluster 0 pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:16.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:16.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:16.647 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:16.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:16.761 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:16 smithi203 bash[23654]: cluster 2024-03-11T21:34:15.383081+0000 mgr.y (mgr.14550) 448 : cluster 0 pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:16.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:16.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:16.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:17.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:17.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:17.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:17.173 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:17.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:17.302 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:17.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:17.428 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:17.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:17.559 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:17.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:17.689 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:17.785 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:17.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:17.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:17.920 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:17.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:18.052 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:18.080 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:18.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:18.211 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:18.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:18.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:18.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.455 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:18 smithi203 bash[23654]: cluster 2024-03-11T21:34:17.383748+0000 mgr.y (mgr.14550) 449 : cluster 0 pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:18.470 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:18.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:18.601 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:18.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:18 smithi143 bash[29284]: cluster 2024-03-11T21:34:17.383748+0000 mgr.y (mgr.14550) 449 : cluster 0 pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:18.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:18 smithi143 bash[21512]: cluster 2024-03-11T21:34:17.383748+0000 mgr.y (mgr.14550) 449 : cluster 0 pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:18.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:18.731 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:18.770 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:18.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:18.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:18.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:18.987 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:19.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:19.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:19.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.244 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:19.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:19.374 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:19.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:19.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:19.633 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:19.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:19 smithi143 bash[21512]: audit 2024-03-11T21:34:18.617948+0000 mgr.y (mgr.14550) 450 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:19.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:19 smithi143 bash[29284]: audit 2024-03-11T21:34:18.617948+0000 mgr.y (mgr.14550) 450 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:19.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:19 smithi203 bash[23654]: audit 2024-03-11T21:34:18.617948+0000 mgr.y (mgr.14550) 450 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:19.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:19.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:19.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:20.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:20.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:20.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:20.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:20.284 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:20.387 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:20.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:20.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:20.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:20 smithi203 bash[23654]: cluster 2024-03-11T21:34:19.386143+0000 mgr.y (mgr.14550) 451 : cluster 0 pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:20.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:20.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:20.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:20 smithi143 bash[29284]: cluster 2024-03-11T21:34:19.386143+0000 mgr.y (mgr.14550) 451 : cluster 0 pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:20.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:20 smithi143 bash[21512]: cluster 2024-03-11T21:34:19.386143+0000 mgr.y (mgr.14550) 451 : cluster 0 pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:34:20.672 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:20.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.801 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:20.904 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.933 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:21.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:21.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:21.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:21.191 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:21.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.323 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:21.426 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:21.448 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:21.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:21.576 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:21.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.708 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:21.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.839 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:21.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.970 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:22.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:22.100 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:22.203 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:22.230 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:22.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:22.345 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:22.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:22.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:22.491 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:22.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:22.621 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:22.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:22.734 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:22 smithi203 bash[23654]: cluster 2024-03-11T21:34:21.386920+0000 mgr.y (mgr.14550) 452 : cluster 0 pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:22.734 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:22 smithi203 bash[23654]: audit 2024-03-11T21:34:21.424332+0000 mon.a (mon.0) 2702 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:22.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:22 smithi203 bash[23654]: audit 2024-03-11T21:34:21.425796+0000 mon.c (mon.2) 925 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:22.751 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:22.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[29284]: cluster 2024-03-11T21:34:21.386920+0000 mgr.y (mgr.14550) 452 : cluster 0 pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:22.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[29284]: audit 2024-03-11T21:34:21.424332+0000 mon.a (mon.0) 2702 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:22.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[29284]: audit 2024-03-11T21:34:21.425796+0000 mon.c (mon.2) 925 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:22.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[21512]: cluster 2024-03-11T21:34:21.386920+0000 mgr.y (mgr.14550) 452 : cluster 0 pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:22.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[21512]: audit 2024-03-11T21:34:21.424332+0000 mon.a (mon.0) 2702 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:22.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:22 smithi143 bash[21512]: audit 2024-03-11T21:34:21.425796+0000 mon.c (mon.2) 925 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:22.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:22.884 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:22.988 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:23.017 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:23.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:23.149 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:23.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:23.278 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:23.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:23.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:23.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:23.537 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:23.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:23.651 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:23 smithi203 bash[23654]: cluster 2024-03-11T21:34:22.397986+0000 osd.6 (osd.6) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:23.651 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:23 smithi203 bash[23654]: cluster 2024-03-11T21:34:22.398291+0000 osd.6 (osd.6) 194 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:23.651 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:23 smithi203 bash[23654]: cluster 2024-03-11T21:34:23.339269+0000 mon.a (mon.0) 2703 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:34:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:23.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:23.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[21512]: cluster 2024-03-11T21:34:22.397986+0000 osd.6 (osd.6) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:23.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[21512]: cluster 2024-03-11T21:34:22.398291+0000 osd.6 (osd.6) 194 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:23.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[21512]: cluster 2024-03-11T21:34:23.339269+0000 mon.a (mon.0) 2703 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:34:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[29284]: cluster 2024-03-11T21:34:22.397986+0000 osd.6 (osd.6) 193 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[29284]: cluster 2024-03-11T21:34:22.398291+0000 osd.6 (osd.6) 194 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:23 smithi143 bash[29284]: cluster 2024-03-11T21:34:23.339269+0000 mon.a (mon.0) 2703 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:34:23.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:23.795 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:23.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:23.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:24.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:24.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:24.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:24.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.287+0000 7f170f688700 1 -- 172.21.15.143:0/1723210975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17081071c0 msgr2=0x7f170810ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.287+0000 7f170f688700 1 --2- 172.21.15.143:0/1723210975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17081071c0 0x7f170810ed40 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f170000b0d0 tx=0x7f170000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/1723210975 shutdown_connections 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- 172.21.15.143:0/1723210975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170806a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- 172.21.15.143:0/1723210975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17081071c0 0x7f170810ed40 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- 172.21.15.143:0/1723210975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f170806a720 0x7f1708106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/1723210975 >> 172.21.15.143:0/1723210975 conn(0x7f1708076d80 msgr2=0x7f1708077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/1723210975 shutdown_connections 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/1723210975 wait complete. 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 librados: starting msgr at 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 librados: starting objecter 2024-03-11T21:34:24.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 Processor -- start 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- start start 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 librados: setting wanted keys 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 librados: calling monclient init 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 0x7f170819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17081071c0 0x7f17081979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708111680 con 0x7f1708069d70 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708111500 con 0x7f17081071c0 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1708111800 con 0x7f170806a720 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170cc23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 0x7f170819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54506/0 (socket says 172.21.15.143:54506) 2024-03-11T21:34:24.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 -- 172.21.15.143:0/4040224664 learned_addr learned my addr 172.21.15.143:0/4040224664 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:24.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170dc25700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17081071c0 0x7f17081979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:24.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 -- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 msgr2=0x7f170819dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:24.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 0x7f170819dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 -- 172.21.15.143:0/4040224664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17081071c0 msgr2=0x7f17081979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17081071c0 0x7f17081979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 -- 172.21.15.143:0/4040224664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708198140 con 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170cc23700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 0x7f170819dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170d424700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f170400eb30 tx=0x7f170400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f16fe7fc700 10 client.?.objecter ms_handle_connect 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f16fe7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170400cc90 con 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1708198430 con 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f170400ce30 con 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 10 client.30423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704003df0 con 0x7f1708069d70 2024-03-11T21:34:24.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1708100d40 con 0x7f1708069d70 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 librados: init done 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.291+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1708103570 con 0x7f1708069d70 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1704004000 con 0x7f1708069d70 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 0x7f16f4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(453..453 src has 87..453) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f170408ec70 con 0x7f1708069d70 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 10 client.30423.objecter ms_dispatch 0x7f1708001550 osd_map(453..453 src has 87..453) v4 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 3 client.30423.objecter handle_osd_map got epochs [453,453] > 0 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 3 client.30423.objecter handle_osd_map decoding full epoch 453 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f170cc23700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 0x7f16f4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 20 client.30423.objecter dump_active .. 0 homeless 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f170cc23700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 0x7f16f4071bb0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f17081085f0 tx=0x7f16f800a300 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:24.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.295+0000 7f16fe7fc700 10 client.30423.objecter ms_handle_connect 0x7f16f406f6f0 2024-03-11T21:34:24.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.299+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f170405b930 con 0x7f1708069d70 2024-03-11T21:34:24.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:24.324 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:24.364 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:34:24.364 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:34:24.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:24.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:24.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:24.576 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:24.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.611+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1708198f10 con 0x7f1708069d70 2024-03-11T21:34:24.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.611+0000 7f16fe7fc700 1 -- 172.21.15.143:0/4040224664 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2687 (secure 0 0 0) 0x7f170405f380 con 0x7f1708069d70 2024-03-11T21:34:24.587 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:34:24.587 INFO:teuthology.orchestra.run.smithi143.stdout:{"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"CACHE_POOL_NEAR_FULL":{"severity":"HEALTH_WARN","summary":{"message":"1 cache pools at or near target size","count":1},"muted":false},"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1151,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":454,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192696,"num_in_osds":8,"osd_in_since":1710192115,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":260}],"num_pgs":260,"num_pools":10,"num_objects":350,"data_bytes":8792137,"bytes_used":624054272,"bytes_avail":767336230912,"bytes_total":767960285184,"read_bytes_sec":1023,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0,"evict_bytes_sec":204,"num_flush_mode_high":2,"num_evict_mode_some":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.143:8443/","prometheus":"http://172.21.15.143:9283/"}},"servicemap":{"epoch":24,"modified":"2024-03-11T21:32:37.329241+0000","services":{"rgw":{"daemons":{"summary":"","14427":{"start_epoch":13,"start_stamp":"2024-03-11T21:23:10.349156+0000","gid":14427,"addr":"172.21.15.143:0/3842484601","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi143","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi143","id":"foo.a","kernel_description":"#110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024","kernel_version":"5.15.0-100-generic","mem_swap_kb":"2097148","mem_total_kb":"32746924","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"25063d57-28bf-41f2-b5d5-290103490a0a","zone_name":"default","zonegroup_id":"4aab72bc-9c1b-4496-a7a5-2f18f6205dcf","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 10 librados: watch_flush enter 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 10 librados: watch_flush exit 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 20 client.30423.objecter shutdown clearing up homeless session... 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 10 client.30423.objecter successfully canceled tick 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 msgr2=0x7f16f4071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 0x7f16f4071bb0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f17081085f0 tx=0x7f16f800a300 comp rx=0 tx=0).stop 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 msgr2=0x7f170819d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:24.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f170400eb30 tx=0x7f170400ee40 comp rx=0 tx=0).stop 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 shutdown_connections 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f16f406f6f0 0x7f16f4071bb0 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1708069d70 0x7f170819d820 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17081071c0 0x7f17081979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 --2- 172.21.15.143:0/4040224664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170806a720 0x7f170819dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 >> 172.21.15.143:0/4040224664 conn(0x7f1708076d80 msgr2=0x7f17080febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 shutdown_connections 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 -- 172.21.15.143:0/4040224664 wait complete. 2024-03-11T21:34:24.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:24.615+0000 7f170f688700 1 librados: shutdown 2024-03-11T21:34:24.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: cluster 2024-03-11T21:34:23.387629+0000 mgr.y (mgr.14550) 453 : cluster 0 pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: audit 2024-03-11T21:34:23.422367+0000 mon.c (mon.2) 926 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: audit 2024-03-11T21:34:23.423265+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: audit 2024-03-11T21:34:24.341014+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: cluster 2024-03-11T21:34:24.345569+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: audit 2024-03-11T21:34:24.351188+0000 mon.c (mon.2) 927 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[21512]: audit 2024-03-11T21:34:24.351562+0000 mon.a (mon.0) 2707 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: cluster 2024-03-11T21:34:23.387629+0000 mgr.y (mgr.14550) 453 : cluster 0 pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: audit 2024-03-11T21:34:23.422367+0000 mon.c (mon.2) 926 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: audit 2024-03-11T21:34:23.423265+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: audit 2024-03-11T21:34:24.341014+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: cluster 2024-03-11T21:34:24.345569+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: audit 2024-03-11T21:34:24.351188+0000 mon.c (mon.2) 927 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:24 smithi143 bash[29284]: audit 2024-03-11T21:34:24.351562+0000 mon.a (mon.0) 2707 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:24.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: cluster 2024-03-11T21:34:23.387629+0000 mgr.y (mgr.14550) 453 : cluster 0 pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: audit 2024-03-11T21:34:23.422367+0000 mon.c (mon.2) 926 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: audit 2024-03-11T21:34:23.423265+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: audit 2024-03-11T21:34:24.341014+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: cluster 2024-03-11T21:34:24.345569+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: audit 2024-03-11T21:34:24.351188+0000 mon.c (mon.2) 927 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:24 smithi203 bash[23654]: audit 2024-03-11T21:34:24.351562+0000 mon.a (mon.0) 2707 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:24.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:24.826 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:24.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:24.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:25.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:25.088 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:25.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:25.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:25.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:25.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:25.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:25.480 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 -- 172.21.15.143:0/2939276969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac106d80 msgr2=0x7f19ac107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/2939276969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac106d80 0x7f19ac107160 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f199c009990 tx=0x7f199c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 -- 172.21.15.143:0/2939276969 shutdown_connections 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/2939276969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac106d80 0x7f19ac107160 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/2939276969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19ac1016d0 0x7f19ac10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/2939276969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac100d10 0x7f19ac101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.547+0000 7f19b1e1c700 1 -- 172.21.15.143:0/2939276969 >> 172.21.15.143:0/2939276969 conn(0x7f19ac077cb0 msgr2=0x7f19ac0780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- 172.21.15.143:0/2939276969 shutdown_connections 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- 172.21.15.143:0/2939276969 wait complete. 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 librados: starting msgr at 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 librados: starting objecter 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 Processor -- start 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- start start 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 librados: setting wanted keys 2024-03-11T21:34:25.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 librados: calling monclient init 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19ac100d10 0x7f19ac19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 0x7f19ac19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac106d80 0x7f19ac197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac111680 con 0x7f19ac1016d0 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac111500 con 0x7f19ac100d10 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac111800 con 0x7f19ac106d80 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 0x7f19ac19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19abfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac106d80 0x7f19ac197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 0x7f19ac19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54530/0 (socket says 172.21.15.143:54530) 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 -- 172.21.15.143:0/1820219729 learned_addr learned my addr 172.21.15.143:0/1820219729 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac106d80 msgr2=0x7f19ac197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac106d80 0x7f19ac197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19ac100d10 msgr2=0x7f19ac19d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19ac100d10 0x7f19ac19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac1981e0 con 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19ac106d80 0x7f19ac197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 0x7f19ac19de00 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f19a000eb30 tx=0x7f19a000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 10 client.?.objecter ms_handle_connect 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a000cc90 con 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f19a000ce30 con 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac1984d0 con 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0003df0 con 0x7f19ac1016d0 2024-03-11T21:34:25.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 10 client.30429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac1a4790 con 0x7f19ac1016d0 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 librados: init done 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f19ac102430 con 0x7f19ac1016d0 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f19a0004000 con 0x7f19ac1016d0 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f199406f820 0x7f1994071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.551+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(455..455 src has 87..455) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f19a008ed20 con 0x7f19ac1016d0 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 osd_map(455..455 src has 87..455) v4 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 3 client.30429.objecter handle_osd_map got epochs [455,455] > 0 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 3 client.30429.objecter handle_osd_map decoding full epoch 455 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f199406f820 0x7f1994071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 20 client.30429.objecter dump_active .. 0 homeless 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v455) v1 ==== 54+0+16 (secure 0 0 0) 0x7f19a0094370 con 0x7f19ac1016d0 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 1 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f19ac0629f0 0x7f19ac062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 1 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 0 1 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 4 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 10 client.30429.objecter _send_command 1 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac198fb0 con 0x7f19ac0629f0 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f199406f820 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f199406f820 0x7f1994071ce0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f199c00b7f0 tx=0x7f199c005fb0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f19ac0629f0 0x7f19ac062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f19ac0629f0 0x7f19ac062dd0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac0629f0 2024-03-11T21:34:25.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac198fb0 con 0x7f19ac0629f0 2024-03-11T21:34:25.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:34:25.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 1 = system:0 2024-03-11T21:34:25.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 4 2024-03-11T21:34:25.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.555+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 0 1 2024-03-11T21:34:25.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 2 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 2 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac04faa0 osd=0 3 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 0 2 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 4 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 10 client.30429.objecter _send_command 2 2024-03-11T21:34:25.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.571+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac062470 con 0x7f19ac0629f0 2024-03-11T21:34:25.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac062470 con 0x7f19ac0629f0 2024-03-11T21:34:25.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 4 2024-03-11T21:34:25.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 0 2 2024-03-11T21:34:25.549 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f19ac1087e0 0x7f19ac1a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 1 3 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 4 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 10 client.30429.objecter _send_command 3 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac02d9a0 con 0x7f19ac1087e0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f19ac1087e0 0x7f19ac1a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac1087e0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f19ac1087e0 0x7f19ac1a8a40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac02d9a0 con 0x7f19ac1087e0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 3 = system:0 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 4 2024-03-11T21:34:25.551 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.575+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 1 3 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 4 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 4 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac02d830 osd=1 3 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 1 4 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 4 2024-03-11T21:34:25.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 10 client.30429.objecter _send_command 4 2024-03-11T21:34:25.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.591+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1a8f80 con 0x7f19ac1087e0 2024-03-11T21:34:25.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1a8f80 con 0x7f19ac1087e0 2024-03-11T21:34:25.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 4 2024-03-11T21:34:25.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 1 4 2024-03-11T21:34:25.569 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 5 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f19ac107ec0 0x7f19ac1ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 5 2024-03-11T21:34:25.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 2 5 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 4 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 10 client.30429.objecter _send_command 5 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac02d470 con 0x7f19ac107ec0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f19ac107ec0 0x7f19ac1ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac107ec0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f19ac107ec0 0x7f19ac1ac320 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac02d470 con 0x7f19ac107ec0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 5 = system:0 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 4 2024-03-11T21:34:25.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.595+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 2 5 2024-03-11T21:34:25.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:25.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 6 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 6 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac107d50 osd=2 3 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 2 6 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 4 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 10 client.30429.objecter _send_command 6 2024-03-11T21:34:25.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac02c900 con 0x7f19ac107ec0 2024-03-11T21:34:25.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac02c900 con 0x7f19ac107ec0 2024-03-11T21:34:25.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 4 2024-03-11T21:34:25.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 2 6 2024-03-11T21:34:25.590 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:34:25.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 7 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f19ac10e580 0x7f19ac10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 7 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 3 7 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 4 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 10 client.30429.objecter _send_command 7 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac1a9fe0 con 0x7f19ac10e580 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f19ac10e580 0x7f19ac10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac10e580 2024-03-11T21:34:25.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f19ac10e580 0x7f19ac10e9a0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac1a9fe0 con 0x7f19ac10e580 2024-03-11T21:34:25.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:34:25.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 7 = system:0 2024-03-11T21:34:25.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 4 2024-03-11T21:34:25.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.615+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 3 7 2024-03-11T21:34:25.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 8 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 8 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9960 osd=3 3 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 3 8 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 4 2024-03-11T21:34:25.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 10 client.30429.objecter _send_command 8 2024-03-11T21:34:25.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1a93e0 con 0x7f19ac10e580 2024-03-11T21:34:25.610 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:34:25.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1a93e0 con 0x7f19ac10e580 2024-03-11T21:34:25.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 4 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 3 8 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 9 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f19ac1ad030 0x7f19ac1ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 9 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 4 9 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 4 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 10 client.30429.objecter _send_command 9 2024-03-11T21:34:25.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac1a9300 con 0x7f19ac1ad030 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f19ac1ad030 0x7f19ac1ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac1ad030 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.635+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f19ac1ad030 0x7f19ac1ad470 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.639+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac1a9300 con 0x7f19ac1ad030 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.639+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.639+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 9 = system:0 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.639+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 4 2024-03-11T21:34:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.639+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 4 9 2024-03-11T21:34:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:25.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 10 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 10 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9190 osd=4 3 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 4 10 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 4 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 10 client.30429.objecter _send_command 10 2024-03-11T21:34:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.655+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1a9300 con 0x7f19ac1ad030 2024-03-11T21:34:25.632 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:34:25.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1a9300 con 0x7f19ac1ad030 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 4 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 4 10 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 11 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f19ac1b6320 0x7f19ac1b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 11 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 5 11 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 4 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 10 client.30429.objecter _send_command 11 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac1bb660 con 0x7f19ac1b6320 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f19ac1b6320 0x7f19ac1b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac1b6320 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19aaffd700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f19ac1b6320 0x7f19ac1b8710 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac1bb660 con 0x7f19ac1b6320 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 11 = system:0 2024-03-11T21:34:25.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 4 2024-03-11T21:34:25.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.659+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 5 11 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 12 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 12 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1a9550 osd=5 3 2024-03-11T21:34:25.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 5 12 2024-03-11T21:34:25.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 4 2024-03-11T21:34:25.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 10 client.30429.objecter _send_command 12 2024-03-11T21:34:25.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.675+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1b8c50 con 0x7f19ac1b6320 2024-03-11T21:34:25.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1b8c50 con 0x7f19ac1b6320 2024-03-11T21:34:25.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 4 2024-03-11T21:34:25.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 5 12 2024-03-11T21:34:25.654 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 13 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f19ac1b9370 0x7f19ac1b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 13 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 6 13 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 4 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 10 client.30429.objecter _send_command 13 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac1bea30 con 0x7f19ac1b9370 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f19ac1b9370 0x7f19ac1b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19abfff700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f19ac1b9370 0x7f19ac1b97d0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.679+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac1b9370 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.683+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac1bea30 con 0x7f19ac1b9370 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.683+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:34:25.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.683+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 13 = system:0 2024-03-11T21:34:25.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.683+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 4 2024-03-11T21:34:25.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.683+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 6 13 2024-03-11T21:34:25.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 14 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 14 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1b8c50 osd=6 3 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 6 14 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 4 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 10 client.30429.objecter _send_command 14 2024-03-11T21:34:25.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1bd8b0 con 0x7f19ac1b9370 2024-03-11T21:34:25.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1bd8b0 con 0x7f19ac1b9370 2024-03-11T21:34:25.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 4 2024-03-11T21:34:25.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 6 14 2024-03-11T21:34:25.675 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 15 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f19ac1bdfd0 0x7f19ac1be410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 15 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 7 15 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 4 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 10 client.30429.objecter _send_command 15 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.699+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ac1c1da0 con 0x7f19ac1bdfd0 2024-03-11T21:34:25.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f19ac1bdfd0 0x7f19ac1be410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19ab7fe700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f19ac1bdfd0 0x7f19ac1be410 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 10 client.30429.objecter ms_handle_connect 0x7f19ac1bdfd0 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19ac1c1da0 con 0x7f19ac1bdfd0 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 15 = system:0 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 4 2024-03-11T21:34:25.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.703+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 7 15 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:24.614944+0000 mon.a (mon.0) 2708 : audit 0 from='client.? 172.21.15.143:0/4040224664' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:25.343594+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: cluster 2024-03-11T21:34:25.351252+0000 mon.a (mon.0) 2710 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:25.411894+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:25.412458+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:25.413357+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[21512]: audit 2024-03-11T21:34:25.413754+0000 mon.a (mon.0) 2712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:24.614944+0000 mon.a (mon.0) 2708 : audit 0 from='client.? 172.21.15.143:0/4040224664' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:25.343594+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: cluster 2024-03-11T21:34:25.351252+0000 mon.a (mon.0) 2710 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:25.411894+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:25.412458+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:25.413357+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:25 smithi143 bash[29284]: audit 2024-03-11T21:34:25.413754+0000 mon.a (mon.0) 2712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 10 client.30429.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:34:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign -1 16 2024-03-11T21:34:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 20 client.30429.objecter _get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_remove -1 16 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 20 client.30429.objecter get_session s=0x7f19ac1bd8b0 osd=7 3 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 15 client.30429.objecter _session_command_op_assign 7 16 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 4 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 10 client.30429.objecter _send_command 16 2024-03-11T21:34:25.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.719+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19ac1b9d10 con 0x7f19ac1bdfd0 2024-03-11T21:34:25.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19a8ff9700 1 -- 172.21.15.143:0/1820219729 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19ac1b9d10 con 0x7f19ac1bdfd0 2024-03-11T21:34:25.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19a8ff9700 10 client.30429.objecter ms_dispatch 0x7f19ac001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:25.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19a8ff9700 10 client.30429.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:25.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19a8ff9700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 4 2024-03-11T21:34:25.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19a8ff9700 15 client.30429.objecter _session_command_op_remove 7 16 2024-03-11T21:34:25.699 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:34:25.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 librados: watch_flush enter 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 librados: watch_flush exit 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.0 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f19ac0629f0 msgr2=0x7f19ac062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f19ac0629f0 0x7f19ac062dd0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac04faa0 osd=0 1 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.1 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f19ac1087e0 msgr2=0x7f19ac1a8a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f19ac1087e0 0x7f19ac1a8a40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac02d830 osd=1 1 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.2 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f19ac107ec0 msgr2=0x7f19ac1ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f19ac107ec0 0x7f19ac1ac320 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac107d50 osd=2 1 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.3 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f19ac10e580 msgr2=0x7f19ac10e9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f19ac10e580 0x7f19ac10e9a0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9960 osd=3 1 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.4 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f19ac1ad030 msgr2=0x7f19ac1ad470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f19ac1ad030 0x7f19ac1ad470 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9190 osd=4 1 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.5 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f19ac1b6320 msgr2=0x7f19ac1b8710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f19ac1b6320 0x7f19ac1b8710 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1a9550 osd=5 1 2024-03-11T21:34:25.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.6 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f19ac1b9370 msgr2=0x7f19ac1b97d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f19ac1b9370 0x7f19ac1b97d0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1b8c50 osd=6 1 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter close_session for osd.7 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f19ac1bdfd0 msgr2=0x7f19ac1be410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f19ac1bdfd0 0x7f19ac1be410 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter put_session s=0x7f19ac1bd8b0 osd=7 1 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 20 client.30429.objecter shutdown clearing up homeless session... 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 10 client.30429.objecter successfully canceled tick 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19abfff700 1 -- 172.21.15.143:0/1820219729 reap_dead start 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19abfff700 1 -- 172.21.15.143:0/1820219729 reap_dead start 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f199406f820 msgr2=0x7f1994071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f199406f820 0x7f1994071ce0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f199c00b7f0 tx=0x7f199c005fb0 comp rx=0 tx=0).stop 2024-03-11T21:34:25.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 msgr2=0x7f19ac19de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 --2- 172.21.15.143:0/1820219729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19ac1016d0 0x7f19ac19de00 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f19a000eb30 tx=0x7f19a000ee40 comp rx=0 tx=0).stop 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19abfff700 1 -- 172.21.15.143:0/1820219729 reap_dead start 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19abfff700 1 -- 172.21.15.143:0/1820219729 reap_dead start 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19abfff700 1 -- 172.21.15.143:0/1820219729 reap_dead start 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 shutdown_connections 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 >> 172.21.15.143:0/1820219729 conn(0x7f19ac077cb0 msgr2=0x7f19ac0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 shutdown_connections 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 -- 172.21.15.143:0/1820219729 wait complete. 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:25.723+0000 7f19b1e1c700 1 librados: shutdown 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:25.704 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:25.704 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:25.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:25.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:25.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:24.614944+0000 mon.a (mon.0) 2708 : audit 0 from='client.? 172.21.15.143:0/4040224664' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:25.343594+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]': finished 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: cluster 2024-03-11T21:34:25.351252+0000 mon.a (mon.0) 2710 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:25.411894+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:25.412458+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:25.413357+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:25 smithi203 bash[23654]: audit 2024-03-11T21:34:25.413754+0000 mon.a (mon.0) 2712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-51"}]: dispatch 2024-03-11T21:34:25.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:25.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:25.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:25.991 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:26.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.123 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:26.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:26.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:26.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.377 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:26.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:26.507 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:26.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:26.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:26.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:26.767 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:26.843 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[21512]: cluster 2024-03-11T21:34:25.388157+0000 mgr.y (mgr.14550) 454 : cluster 0 pgmap v675: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:34:26.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[21512]: audit 2024-03-11T21:34:25.555573+0000 mon.a (mon.0) 2713 : audit 0 from='client.? 172.21.15.143:0/1820219729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:26.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[21512]: cluster 2024-03-11T21:34:26.334254+0000 mon.a (mon.0) 2714 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:34:26.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[29284]: cluster 2024-03-11T21:34:25.388157+0000 mgr.y (mgr.14550) 454 : cluster 0 pgmap v675: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:34:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[29284]: audit 2024-03-11T21:34:25.555573+0000 mon.a (mon.0) 2713 : audit 0 from='client.? 172.21.15.143:0/1820219729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:26 smithi143 bash[29284]: cluster 2024-03-11T21:34:26.334254+0000 mon.a (mon.0) 2714 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:34:27.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:27.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:26 smithi203 bash[23654]: cluster 2024-03-11T21:34:25.388157+0000 mgr.y (mgr.14550) 454 : cluster 0 pgmap v675: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:34:27.010 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:26 smithi203 bash[23654]: audit 2024-03-11T21:34:25.555573+0000 mon.a (mon.0) 2713 : audit 0 from='client.? 172.21.15.143:0/1820219729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:27.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:26 smithi203 bash[23654]: cluster 2024-03-11T21:34:26.334254+0000 mon.a (mon.0) 2714 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:34:27.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:27.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:27.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:27.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:27.421 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:27.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:27.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:27.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:27.680 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:27.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:34:27.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:27.795 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: cluster 2024-03-11T21:34:26.727112+0000 osd.2 (osd.2) 213 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:27.795 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: cluster 2024-03-11T21:34:26.727381+0000 osd.2 (osd.2) 214 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:27.795 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: cluster 2024-03-11T21:34:26.785128+0000 mon.a (mon.0) 2715 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:34:27.795 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: cluster 2024-03-11T21:34:27.343102+0000 mon.a (mon.0) 2716 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:34:27.796 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: audit 2024-03-11T21:34:27.357447+0000 mon.c (mon.2) 930 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:27.796 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:27 smithi203 bash[23654]: audit 2024-03-11T21:34:27.359481+0000 mon.a (mon.0) 2717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:27.809 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:27.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: cluster 2024-03-11T21:34:26.727112+0000 osd.2 (osd.2) 213 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: cluster 2024-03-11T21:34:26.727381+0000 osd.2 (osd.2) 214 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: cluster 2024-03-11T21:34:26.785128+0000 mon.a (mon.0) 2715 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: cluster 2024-03-11T21:34:27.343102+0000 mon.a (mon.0) 2716 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: audit 2024-03-11T21:34:27.357447+0000 mon.c (mon.2) 930 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[29284]: audit 2024-03-11T21:34:27.359481+0000 mon.a (mon.0) 2717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: cluster 2024-03-11T21:34:26.727112+0000 osd.2 (osd.2) 213 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:28.804 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: cluster 2024-03-11T21:34:26.727381+0000 osd.2 (osd.2) 214 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:28.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: cluster 2024-03-11T21:34:26.785128+0000 mon.a (mon.0) 2715 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:34:28.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: cluster 2024-03-11T21:34:27.343102+0000 mon.a (mon.0) 2716 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:34:28.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: audit 2024-03-11T21:34:27.357447+0000 mon.c (mon.2) 930 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:28.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:27 smithi143 bash[21512]: audit 2024-03-11T21:34:27.359481+0000 mon.a (mon.0) 2717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:28.806 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-11T21:34:28.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:34:28.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: cluster 2024-03-11T21:34:27.388935+0000 mgr.y (mgr.14550) 455 : cluster 0 pgmap v678: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: cluster 2024-03-11T21:34:27.905595+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: cluster 2024-03-11T21:34:27.905965+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: cluster 2024-03-11T21:34:28.334114+0000 mon.a (mon.0) 2718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: audit 2024-03-11T21:34:28.344879+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: cluster 2024-03-11T21:34:28.357440+0000 mon.a (mon.0) 2720 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: audit 2024-03-11T21:34:28.431719+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: audit 2024-03-11T21:34:28.432129+0000 mon.a (mon.0) 2721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: audit 2024-03-11T21:34:28.433018+0000 mon.c (mon.2) 932 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[21512]: audit 2024-03-11T21:34:28.433526+0000 mon.a (mon.0) 2722 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: cluster 2024-03-11T21:34:27.388935+0000 mgr.y (mgr.14550) 455 : cluster 0 pgmap v678: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:28.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: cluster 2024-03-11T21:34:27.905595+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: cluster 2024-03-11T21:34:27.905965+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: cluster 2024-03-11T21:34:28.334114+0000 mon.a (mon.0) 2718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: audit 2024-03-11T21:34:28.344879+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: cluster 2024-03-11T21:34:28.357440+0000 mon.a (mon.0) 2720 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: audit 2024-03-11T21:34:28.431719+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: audit 2024-03-11T21:34:28.432129+0000 mon.a (mon.0) 2721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: audit 2024-03-11T21:34:28.433018+0000 mon.c (mon.2) 932 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:28.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:28 smithi143 bash[29284]: audit 2024-03-11T21:34:28.433526+0000 mon.a (mon.0) 2722 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:28.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:29.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:29.020 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: cluster 2024-03-11T21:34:27.388935+0000 mgr.y (mgr.14550) 455 : cluster 0 pgmap v678: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:29.020 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: cluster 2024-03-11T21:34:27.905595+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: cluster 2024-03-11T21:34:27.905965+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: cluster 2024-03-11T21:34:28.334114+0000 mon.a (mon.0) 2718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: audit 2024-03-11T21:34:28.344879+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: cluster 2024-03-11T21:34:28.357440+0000 mon.a (mon.0) 2720 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: audit 2024-03-11T21:34:28.431719+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: audit 2024-03-11T21:34:28.432129+0000 mon.a (mon.0) 2721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: audit 2024-03-11T21:34:28.433018+0000 mon.c (mon.2) 932 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:29.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:28 smithi203 bash[23654]: audit 2024-03-11T21:34:28.433526+0000 mon.a (mon.0) 2722 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-53"}]: dispatch 2024-03-11T21:34:29.102 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:29.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:29.256 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:29.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:29.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:29.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.513 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:29.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:29.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:29.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:30.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:30.034 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:30.037 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:29 smithi143 bash[21512]: audit 2024-03-11T21:34:28.619599+0000 mgr.y (mgr.14550) 456 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:30.037 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:29 smithi143 bash[21512]: cluster 2024-03-11T21:34:29.354473+0000 mon.a (mon.0) 2723 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:34:30.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:29 smithi143 bash[29284]: audit 2024-03-11T21:34:28.619599+0000 mgr.y (mgr.14550) 456 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:30.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:29 smithi143 bash[29284]: cluster 2024-03-11T21:34:29.354473+0000 mon.a (mon.0) 2723 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:34:30.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.160 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/73737766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac06a4d0 msgr2=0x7f1eac10ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/73737766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac06a4d0 0x7f1eac10ec90 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009990 tx=0x7f1ea4009ca0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/73737766 shutdown_connections 2024-03-11T21:34:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/73737766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac107e70 0x7f1eac108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/73737766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac06a4d0 0x7f1eac10ec90 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/73737766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac069f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/73737766 >> 172.21.15.143:0/73737766 conn(0x7f1eac076d80 msgr2=0x7f1eac077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:30.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/73737766 shutdown_connections 2024-03-11T21:34:30.183 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/73737766 wait complete. 2024-03-11T21:34:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 librados: starting msgr at 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 librados: starting objecter 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 Processor -- start 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- start start 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 librados: setting wanted keys 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 librados: calling monclient init 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 0x7f1eac19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac111670 con 0x7f1eac06a4d0 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1114f0 con 0x7f1eac069b10 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb2a0b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1117f0 con 0x7f1eac107e70 2024-03-11T21:34:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb0fa8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eab7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 0x7f1eac19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb0fa8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37760/0 (socket says 172.21.15.143:37760) 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.207+0000 7f1eb0fa8700 1 -- 172.21.15.143:0/1938619462 learned_addr learned my addr 172.21.15.143:0/1938619462 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eabfff700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 -- 172.21.15.143:0/1938619462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 msgr2=0x7f1eac19d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac19d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 -- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 msgr2=0x7f1eac19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 0x7f1eac19dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 -- 172.21.15.143:0/1938619462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac1980f0 con 0x7f1eac107e70 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eabfff700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac19d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eab7fe700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 0x7f1eac19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb0fa8700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40047b0 tx=0x7f1ea4009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 10 client.?.objecter ms_handle_connect 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4027070 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1ea4022470 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac198380 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb2a0b700 10 client.41636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4017650 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac100d40 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb2a0b700 1 librados: init done 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac103520 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1ea402a070 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 0x7f1e94071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(459..459 src has 87..459) v4 ==== 9228+0+0 (secure 0 0 0) 0x7f1ea408f8e0 con 0x7f1eac107e70 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 10 client.41636.objecter ms_dispatch 0x7f1eac001550 osd_map(459..459 src has 87..459) v4 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 3 client.41636.objecter handle_osd_map got epochs [459,459] > 0 2024-03-11T21:34:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 3 client.41636.objecter handle_osd_map decoding full epoch 459 2024-03-11T21:34:30.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eabfff700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 0x7f1e94071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:30.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 20 client.41636.objecter dump_active .. 0 homeless 2024-03-11T21:34:30.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1ea97fa700 10 client.41636.objecter ms_handle_connect 0x7f1e9406f6f0 2024-03-11T21:34:30.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.211+0000 7f1eabfff700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 0x7f1e94071bb0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f1eac198ad0 tx=0x7f1ea000b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:30.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.215+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1ea4013070 con 0x7f1eac107e70 2024-03-11T21:34:30.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:29 smithi203 bash[23654]: audit 2024-03-11T21:34:28.619599+0000 mgr.y (mgr.14550) 456 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:29 smithi203 bash[23654]: cluster 2024-03-11T21:34:29.354473+0000 mon.a (mon.0) 2723 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:34:30.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:30.391 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:30.426 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:30.451+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f1eac04faa0 con 0x7f1eac107e70 2024-03-11T21:34:30.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:30.538 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:30.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:30.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:30.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:30.796 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:30.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:34:30.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: cluster 2024-03-11T21:34:29.389803+0000 mgr.y (mgr.14550) 457 : cluster 0 pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: cluster 2024-03-11T21:34:30.362384+0000 mon.a (mon.0) 2724 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: audit 2024-03-11T21:34:30.369653+0000 mon.c (mon.2) 933 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: audit 2024-03-11T21:34:30.370012+0000 mon.a (mon.0) 2725 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: audit 2024-03-11T21:34:30.455443+0000 mon.c (mon.2) 934 : audit 1 from='client.? 172.21.15.143:0/1938619462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:30.913 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:30 smithi203 bash[23654]: audit 2024-03-11T21:34:30.455764+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:30.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:31.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: cluster 2024-03-11T21:34:29.389803+0000 mgr.y (mgr.14550) 457 : cluster 0 pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: cluster 2024-03-11T21:34:30.362384+0000 mon.a (mon.0) 2724 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: audit 2024-03-11T21:34:30.369653+0000 mon.c (mon.2) 933 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: audit 2024-03-11T21:34:30.370012+0000 mon.a (mon.0) 2725 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: audit 2024-03-11T21:34:30.455443+0000 mon.c (mon.2) 934 : audit 1 from='client.? 172.21.15.143:0/1938619462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[29284]: audit 2024-03-11T21:34:30.455764+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: cluster 2024-03-11T21:34:29.389803+0000 mgr.y (mgr.14550) 457 : cluster 0 pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: cluster 2024-03-11T21:34:30.362384+0000 mon.a (mon.0) 2724 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: audit 2024-03-11T21:34:30.369653+0000 mon.c (mon.2) 933 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:31.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: audit 2024-03-11T21:34:30.370012+0000 mon.a (mon.0) 2725 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: audit 2024-03-11T21:34:30.455443+0000 mon.c (mon.2) 934 : audit 1 from='client.? 172.21.15.143:0/1938619462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:31.044 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:30 smithi143 bash[21512]: audit 2024-03-11T21:34:30.455764+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:34:31.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:31.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:31.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:31.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:31.313 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:31.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1ea97fa700 1 -- 172.21.15.143:0/1938619462 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v461) v1 ==== 91+0+0 (secure 0 0 0) 0x7f1ea405c6c0 con 0x7f1eac107e70 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 10 librados: watch_flush enter 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 10 librados: watch_flush exit 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 20 client.41636.objecter shutdown clearing up homeless session... 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 10 client.41636.objecter successfully canceled tick 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 msgr2=0x7f1e94071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 0x7f1e94071bb0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f1eac198ad0 tx=0x7f1ea000b040 comp rx=0 tx=0).stop 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 msgr2=0x7f1eac197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40047b0 tx=0x7f1ea4009510 comp rx=0 tx=0).stop 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 shutdown_connections 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e9406f6f0 0x7f1e94071bb0 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eac107e70 0x7f1eac197990 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:31.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eac069b10 0x7f1eac19d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 --2- 172.21.15.143:0/1938619462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eac06a4d0 0x7f1eac19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 >> 172.21.15.143:0/1938619462 conn(0x7f1eac076d80 msgr2=0x7f1eac108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 shutdown_connections 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 -- 172.21.15.143:0/1938619462 wait complete. 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:31.367+0000 7f1eb2a0b700 1 librados: shutdown 2024-03-11T21:34:31.344 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:34:31.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:31.442 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:31.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:31.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:31.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:31.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:31.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:32.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:32.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:32.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.226 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:32.361 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:32.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.355877+0000 mon.a (mon.0) 2727 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.358155+0000 mon.a (mon.0) 2728 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.358279+0000 mon.a (mon.0) 2729 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.363742+0000 mon.a (mon.0) 2730 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.368999+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.390418+0000 mgr.y (mgr.14550) 458 : cluster 0 pgmap v684: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.409059+0000 osd.6 (osd.6) 195 : cluster 0 2.11 scrub starts 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.417981+0000 osd.6 (osd.6) 196 : cluster 0 2.11 scrub ok 2024-03-11T21:34:32.435 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.455181+0000 mon.c (mon.2) 936 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.455517+0000 mon.a (mon.0) 2731 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.455944+0000 mon.c (mon.2) 937 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: audit 2024-03-11T21:34:31.456178+0000 mon.a (mon.0) 2732 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.518569+0000 osd.3 (osd.3) 267 : cluster 0 6.1f scrub starts 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.520034+0000 osd.3 (osd.3) 268 : cluster 0 6.1f scrub ok 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.355877+0000 mon.a (mon.0) 2727 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.358155+0000 mon.a (mon.0) 2728 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.358279+0000 mon.a (mon.0) 2729 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.363742+0000 mon.a (mon.0) 2730 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.368999+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.390418+0000 mgr.y (mgr.14550) 458 : cluster 0 pgmap v684: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.409059+0000 osd.6 (osd.6) 195 : cluster 0 2.11 scrub starts 2024-03-11T21:34:32.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.417981+0000 osd.6 (osd.6) 196 : cluster 0 2.11 scrub ok 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.455181+0000 mon.c (mon.2) 936 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.455517+0000 mon.a (mon.0) 2731 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.455944+0000 mon.c (mon.2) 937 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: audit 2024-03-11T21:34:31.456178+0000 mon.a (mon.0) 2732 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.518569+0000 osd.3 (osd.3) 267 : cluster 0 6.1f scrub starts 2024-03-11T21:34:32.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:32 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.520034+0000 osd.3 (osd.3) 268 : cluster 0 6.1f scrub ok 2024-03-11T21:34:32.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:32.496 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:32.598 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.622 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.355877+0000 mon.a (mon.0) 2727 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.358155+0000 mon.a (mon.0) 2728 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.358279+0000 mon.a (mon.0) 2729 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.363742+0000 mon.a (mon.0) 2730 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.368999+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.390418+0000 mgr.y (mgr.14550) 458 : cluster 0 pgmap v684: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.409059+0000 osd.6 (osd.6) 195 : cluster 0 2.11 scrub starts 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.417981+0000 osd.6 (osd.6) 196 : cluster 0 2.11 scrub ok 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.455181+0000 mon.c (mon.2) 936 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.455517+0000 mon.a (mon.0) 2731 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.455944+0000 mon.c (mon.2) 937 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: audit 2024-03-11T21:34:31.456178+0000 mon.a (mon.0) 2732 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-55"}]: dispatch 2024-03-11T21:34:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.518569+0000 osd.3 (osd.3) 267 : cluster 0 6.1f scrub starts 2024-03-11T21:34:32.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:32 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.520034+0000 osd.3 (osd.3) 268 : cluster 0 6.1f scrub ok 2024-03-11T21:34:32.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:32.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:32.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.004 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:33.010 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:33.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.139 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:33.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.270 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:33.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:33.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:33.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:33.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.779127+0000 osd.5 (osd.5) 269 : cluster 0 2.12 scrub starts 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.781323+0000 osd.5 (osd.5) 270 : cluster 0 2.12 scrub ok 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.807559+0000 osd.4 (osd.4) 229 : cluster 0 6.2 scrub starts 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:31.810003+0000 osd.4 (osd.4) 230 : cluster 0 6.2 scrub ok 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.238411+0000 osd.1 (osd.1) 69 : cluster 0 5.12 scrub starts 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.241149+0000 osd.1 (osd.1) 70 : cluster 0 5.12 scrub ok 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.350993+0000 osd.2 (osd.2) 215 : cluster 0 39.1f scrub starts 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.354010+0000 osd.2 (osd.2) 216 : cluster 0 39.1f scrub ok 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.370859+0000 mon.a (mon.0) 2733 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.963308+0000 osd.7 (osd.7) 275 : cluster 0 3.e scrub starts 2024-03-11T21:34:33.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:33 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.970245+0000 osd.7 (osd.7) 276 : cluster 0 3.e scrub ok 2024-03-11T21:34:33.537 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:33.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.779127+0000 osd.5 (osd.5) 269 : cluster 0 2.12 scrub starts 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.781323+0000 osd.5 (osd.5) 270 : cluster 0 2.12 scrub ok 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.807559+0000 osd.4 (osd.4) 229 : cluster 0 6.2 scrub starts 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:31.810003+0000 osd.4 (osd.4) 230 : cluster 0 6.2 scrub ok 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.238411+0000 osd.1 (osd.1) 69 : cluster 0 5.12 scrub starts 2024-03-11T21:34:33.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.241149+0000 osd.1 (osd.1) 70 : cluster 0 5.12 scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.350993+0000 osd.2 (osd.2) 215 : cluster 0 39.1f scrub starts 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.354010+0000 osd.2 (osd.2) 216 : cluster 0 39.1f scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.370859+0000 mon.a (mon.0) 2733 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.963308+0000 osd.7 (osd.7) 275 : cluster 0 3.e scrub starts 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.970245+0000 osd.7 (osd.7) 276 : cluster 0 3.e scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.779127+0000 osd.5 (osd.5) 269 : cluster 0 2.12 scrub starts 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.781323+0000 osd.5 (osd.5) 270 : cluster 0 2.12 scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.807559+0000 osd.4 (osd.4) 229 : cluster 0 6.2 scrub starts 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:31.810003+0000 osd.4 (osd.4) 230 : cluster 0 6.2 scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.238411+0000 osd.1 (osd.1) 69 : cluster 0 5.12 scrub starts 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.241149+0000 osd.1 (osd.1) 70 : cluster 0 5.12 scrub ok 2024-03-11T21:34:33.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.350993+0000 osd.2 (osd.2) 215 : cluster 0 39.1f scrub starts 2024-03-11T21:34:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.354010+0000 osd.2 (osd.2) 216 : cluster 0 39.1f scrub ok 2024-03-11T21:34:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.370859+0000 mon.a (mon.0) 2733 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:34:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.963308+0000 osd.7 (osd.7) 275 : cluster 0 3.e scrub starts 2024-03-11T21:34:33.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:33 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.970245+0000 osd.7 (osd.7) 276 : cluster 0 3.e scrub ok 2024-03-11T21:34:33.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:33.801 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:33.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.931 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:34.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:34.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:34.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:34.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:34.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:34.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:34.443 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:34.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:34.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:34:34.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.602 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.806669+0000 osd.5 (osd.5) 271 : cluster 0 118.13 scrub starts 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:32.811703+0000 osd.5 (osd.5) 272 : cluster 0 118.13 scrub ok 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.336959+0000 osd.2 (osd.2) 217 : cluster 0 6.f scrub starts 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.339221+0000 osd.2 (osd.2) 218 : cluster 0 6.f scrub ok 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.391390+0000 mgr.y (mgr.14550) 459 : cluster 0 pgmap v687: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.398547+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: audit 2024-03-11T21:34:33.401556+0000 mon.c (mon.2) 938 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: audit 2024-03-11T21:34:33.405553+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.532602+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[21512]: cluster 2024-03-11T21:34:33.535744+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.806669+0000 osd.5 (osd.5) 271 : cluster 0 118.13 scrub starts 2024-03-11T21:34:34.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:32.811703+0000 osd.5 (osd.5) 272 : cluster 0 118.13 scrub ok 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.336959+0000 osd.2 (osd.2) 217 : cluster 0 6.f scrub starts 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.339221+0000 osd.2 (osd.2) 218 : cluster 0 6.f scrub ok 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.391390+0000 mgr.y (mgr.14550) 459 : cluster 0 pgmap v687: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.398547+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: audit 2024-03-11T21:34:33.401556+0000 mon.c (mon.2) 938 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: audit 2024-03-11T21:34:33.405553+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.532602+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-03-11T21:34:34.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:34 smithi143 bash[29284]: cluster 2024-03-11T21:34:33.535744+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-03-11T21:34:34.705 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.806669+0000 osd.5 (osd.5) 271 : cluster 0 118.13 scrub starts 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:32.811703+0000 osd.5 (osd.5) 272 : cluster 0 118.13 scrub ok 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.336959+0000 osd.2 (osd.2) 217 : cluster 0 6.f scrub starts 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.339221+0000 osd.2 (osd.2) 218 : cluster 0 6.f scrub ok 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.391390+0000 mgr.y (mgr.14550) 459 : cluster 0 pgmap v687: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.398547+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: audit 2024-03-11T21:34:33.401556+0000 mon.c (mon.2) 938 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: audit 2024-03-11T21:34:33.405553+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.532602+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-03-11T21:34:34.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:34 smithi203 bash[23654]: cluster 2024-03-11T21:34:33.535744+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-03-11T21:34:34.828 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:34.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:34.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.980 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:35.039 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:35.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:35.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:35.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:35.239 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:35.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:35.365 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:35.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:35.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:35.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:35.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:35.607 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: cluster 2024-03-11T21:34:34.386823+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: audit 2024-03-11T21:34:34.388725+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: cluster 2024-03-11T21:34:34.396705+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: audit 2024-03-11T21:34:34.473993+0000 mon.c (mon.2) 939 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: audit 2024-03-11T21:34:34.474448+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: audit 2024-03-11T21:34:34.475234+0000 mon.c (mon.2) 940 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: audit 2024-03-11T21:34:34.475583+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: cluster 2024-03-11T21:34:34.958809+0000 osd.7 (osd.7) 277 : cluster 0 6.12 scrub starts 2024-03-11T21:34:35.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:35 smithi203 bash[23654]: cluster 2024-03-11T21:34:34.961321+0000 osd.7 (osd.7) 278 : cluster 0 6.12 scrub ok 2024-03-11T21:34:35.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: cluster 2024-03-11T21:34:34.386823+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: audit 2024-03-11T21:34:34.388725+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: cluster 2024-03-11T21:34:34.396705+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: audit 2024-03-11T21:34:34.473993+0000 mon.c (mon.2) 939 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: audit 2024-03-11T21:34:34.474448+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: audit 2024-03-11T21:34:34.475234+0000 mon.c (mon.2) 940 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: audit 2024-03-11T21:34:34.475583+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: cluster 2024-03-11T21:34:34.958809+0000 osd.7 (osd.7) 277 : cluster 0 6.12 scrub starts 2024-03-11T21:34:35.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[21512]: cluster 2024-03-11T21:34:34.961321+0000 osd.7 (osd.7) 278 : cluster 0 6.12 scrub ok 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: cluster 2024-03-11T21:34:34.386823+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: audit 2024-03-11T21:34:34.388725+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: cluster 2024-03-11T21:34:34.396705+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: audit 2024-03-11T21:34:34.473993+0000 mon.c (mon.2) 939 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: audit 2024-03-11T21:34:34.474448+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: audit 2024-03-11T21:34:34.475234+0000 mon.c (mon.2) 940 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: audit 2024-03-11T21:34:34.475583+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-57"}]: dispatch 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: cluster 2024-03-11T21:34:34.958809+0000 osd.7 (osd.7) 277 : cluster 0 6.12 scrub starts 2024-03-11T21:34:35.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:35 smithi143 bash[29284]: cluster 2024-03-11T21:34:34.961321+0000 osd.7 (osd.7) 278 : cluster 0 6.12 scrub ok 2024-03-11T21:34:35.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:35.754 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:35.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:35.880 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:35.984 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:36.011 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:36.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:36.142 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:36.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.273 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:36.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:36.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.103167+0000 osd.0 (osd.0) 259 : cluster 0 9.b scrub starts 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.105708+0000 osd.0 (osd.0) 260 : cluster 0 9.b scrub ok 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.348791+0000 osd.6 (osd.6) 197 : cluster 0 2.1a scrub starts 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.350114+0000 osd.6 (osd.6) 198 : cluster 0 2.1a scrub ok 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.392273+0000 mgr.y (mgr.14550) 460 : cluster 0 pgmap v689: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.428742+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.591067+0000 osd.3 (osd.3) 271 : cluster 0 2.1e scrub starts 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.592778+0000 osd.3 (osd.3) 272 : cluster 0 2.1e scrub ok 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.342238+0000 mon.a (mon.0) 2742 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:34:36.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: audit 2024-03-11T21:34:36.351225+0000 mon.c (mon.2) 941 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[29284]: audit 2024-03-11T21:34:36.360754+0000 mon.a (mon.0) 2743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.646 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:36.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.103167+0000 osd.0 (osd.0) 259 : cluster 0 9.b scrub starts 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.105708+0000 osd.0 (osd.0) 260 : cluster 0 9.b scrub ok 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.348791+0000 osd.6 (osd.6) 197 : cluster 0 2.1a scrub starts 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.350114+0000 osd.6 (osd.6) 198 : cluster 0 2.1a scrub ok 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.392273+0000 mgr.y (mgr.14550) 460 : cluster 0 pgmap v689: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.428742+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.591067+0000 osd.3 (osd.3) 271 : cluster 0 2.1e scrub starts 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.592778+0000 osd.3 (osd.3) 272 : cluster 0 2.1e scrub ok 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.342238+0000 mon.a (mon.0) 2742 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: audit 2024-03-11T21:34:36.351225+0000 mon.c (mon.2) 941 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:36 smithi203 bash[23654]: audit 2024-03-11T21:34:36.360754+0000 mon.a (mon.0) 2743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.773 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:36.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.888 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.103167+0000 osd.0 (osd.0) 259 : cluster 0 9.b scrub starts 2024-03-11T21:34:36.888 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.105708+0000 osd.0 (osd.0) 260 : cluster 0 9.b scrub ok 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.348791+0000 osd.6 (osd.6) 197 : cluster 0 2.1a scrub starts 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.350114+0000 osd.6 (osd.6) 198 : cluster 0 2.1a scrub ok 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.392273+0000 mgr.y (mgr.14550) 460 : cluster 0 pgmap v689: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.428742+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.591067+0000 osd.3 (osd.3) 271 : cluster 0 2.1e scrub starts 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.592778+0000 osd.3 (osd.3) 272 : cluster 0 2.1e scrub ok 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.342238+0000 mon.a (mon.0) 2742 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: audit 2024-03-11T21:34:36.351225+0000 mon.c (mon.2) 941 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:36 smithi143 bash[21512]: audit 2024-03-11T21:34:36.360754+0000 mon.a (mon.0) 2743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:36.905 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:37.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:37.024 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:37.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:37.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:37.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:37.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:37.292 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:37.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:37.418 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:37.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:37.533 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.913137+0000 osd.4 (osd.4) 231 : cluster 0 39.10 scrub starts 2024-03-11T21:34:37.533 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:35.917817+0000 osd.4 (osd.4) 232 : cluster 0 39.10 scrub ok 2024-03-11T21:34:37.533 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.106177+0000 osd.0 (osd.0) 261 : cluster 0 39.4 scrub starts 2024-03-11T21:34:37.533 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.108241+0000 osd.0 (osd.0) 262 : cluster 0 39.4 scrub ok 2024-03-11T21:34:37.533 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.322352+0000 osd.2 (osd.2) 219 : cluster 0 5.8 scrub starts 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.323612+0000 osd.2 (osd.2) 220 : cluster 0 5.8 scrub ok 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:36.444718+0000 mon.a (mon.0) 2744 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:36.449099+0000 mon.c (mon.2) 942 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.633199+0000 osd.7 (osd.7) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.633535+0000 osd.7 (osd.7) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.834636+0000 osd.5 (osd.5) 273 : cluster 0 9.a scrub starts 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.837067+0000 osd.5 (osd.5) 274 : cluster 0 9.a scrub ok 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.342193+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.349195+0000 mon.a (mon.0) 2746 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.354990+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.440160+0000 mon.c (mon.2) 944 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.440603+0000 mon.a (mon.0) 2747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.534 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.441246+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[21512]: audit 2024-03-11T21:34:37.441547+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.913137+0000 osd.4 (osd.4) 231 : cluster 0 39.10 scrub starts 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:35.917817+0000 osd.4 (osd.4) 232 : cluster 0 39.10 scrub ok 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.106177+0000 osd.0 (osd.0) 261 : cluster 0 39.4 scrub starts 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.108241+0000 osd.0 (osd.0) 262 : cluster 0 39.4 scrub ok 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.322352+0000 osd.2 (osd.2) 219 : cluster 0 5.8 scrub starts 2024-03-11T21:34:37.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.323612+0000 osd.2 (osd.2) 220 : cluster 0 5.8 scrub ok 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:36.444718+0000 mon.a (mon.0) 2744 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:36.449099+0000 mon.c (mon.2) 942 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.633199+0000 osd.7 (osd.7) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.633535+0000 osd.7 (osd.7) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.834636+0000 osd.5 (osd.5) 273 : cluster 0 9.a scrub starts 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.837067+0000 osd.5 (osd.5) 274 : cluster 0 9.a scrub ok 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.342193+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.349195+0000 mon.a (mon.0) 2746 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.354990+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.440160+0000 mon.c (mon.2) 944 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.440603+0000 mon.a (mon.0) 2747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.441246+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:37 smithi143 bash[29284]: audit 2024-03-11T21:34:37.441547+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:37.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.913137+0000 osd.4 (osd.4) 231 : cluster 0 39.10 scrub starts 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:35.917817+0000 osd.4 (osd.4) 232 : cluster 0 39.10 scrub ok 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.106177+0000 osd.0 (osd.0) 261 : cluster 0 39.4 scrub starts 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.108241+0000 osd.0 (osd.0) 262 : cluster 0 39.4 scrub ok 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.322352+0000 osd.2 (osd.2) 219 : cluster 0 5.8 scrub starts 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.323612+0000 osd.2 (osd.2) 220 : cluster 0 5.8 scrub ok 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:36.444718+0000 mon.a (mon.0) 2744 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:36.449099+0000 mon.c (mon.2) 942 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.633199+0000 osd.7 (osd.7) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.633535+0000 osd.7 (osd.7) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.834636+0000 osd.5 (osd.5) 273 : cluster 0 9.a scrub starts 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.837067+0000 osd.5 (osd.5) 274 : cluster 0 9.a scrub ok 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.342193+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.349195+0000 mon.a (mon.0) 2746 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.354990+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.440160+0000 mon.c (mon.2) 944 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.440603+0000 mon.a (mon.0) 2747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.441246+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:37 smithi203 bash[23654]: audit 2024-03-11T21:34:37.441547+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-59"}]: dispatch 2024-03-11T21:34:37.676 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:37.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:37.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:37.807 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:37.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:37.938 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:38.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:38.070 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:38.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:38.198 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:38.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:38.328 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:38.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:38.458 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:38.561 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:38.587 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:38.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:38.712 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:38.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:38.848 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:38.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:38.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.852612+0000 osd.4 (osd.4) 233 : cluster 0 2.f scrub starts 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:36.855615+0000 osd.4 (osd.4) 234 : cluster 0 2.f scrub ok 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.077679+0000 osd.0 (osd.0) 263 : cluster 0 2.13 scrub starts 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.079353+0000 osd.0 (osd.0) 264 : cluster 0 2.13 scrub ok 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.330479+0000 osd.6 (osd.6) 199 : cluster 0 2.14 scrub starts 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.332567+0000 osd.6 (osd.6) 200 : cluster 0 2.14 scrub ok 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:37.392849+0000 mgr.y (mgr.14550) 461 : cluster 0 pgmap v693: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[21512]: cluster 2024-03-11T21:34:38.357123+0000 mon.a (mon.0) 2749 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.852612+0000 osd.4 (osd.4) 233 : cluster 0 2.f scrub starts 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:36.855615+0000 osd.4 (osd.4) 234 : cluster 0 2.f scrub ok 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.077679+0000 osd.0 (osd.0) 263 : cluster 0 2.13 scrub starts 2024-03-11T21:34:38.962 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.079353+0000 osd.0 (osd.0) 264 : cluster 0 2.13 scrub ok 2024-03-11T21:34:38.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.330479+0000 osd.6 (osd.6) 199 : cluster 0 2.14 scrub starts 2024-03-11T21:34:38.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.332567+0000 osd.6 (osd.6) 200 : cluster 0 2.14 scrub ok 2024-03-11T21:34:38.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:37.392849+0000 mgr.y (mgr.14550) 461 : cluster 0 pgmap v693: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:38.963 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:38 smithi143 bash[29284]: cluster 2024-03-11T21:34:38.357123+0000 mon.a (mon.0) 2749 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:34:38.974 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:39.020 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.852612+0000 osd.4 (osd.4) 233 : cluster 0 2.f scrub starts 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:36.855615+0000 osd.4 (osd.4) 234 : cluster 0 2.f scrub ok 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.077679+0000 osd.0 (osd.0) 263 : cluster 0 2.13 scrub starts 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.079353+0000 osd.0 (osd.0) 264 : cluster 0 2.13 scrub ok 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.330479+0000 osd.6 (osd.6) 199 : cluster 0 2.14 scrub starts 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.332567+0000 osd.6 (osd.6) 200 : cluster 0 2.14 scrub ok 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:37.392849+0000 mgr.y (mgr.14550) 461 : cluster 0 pgmap v693: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:39.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:38 smithi203 bash[23654]: cluster 2024-03-11T21:34:38.357123+0000 mon.a (mon.0) 2749 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:34:39.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:39.091 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:39.106 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:39.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:39.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:39.372 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:39.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:39.498 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:39.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.627 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:39.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:39.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:39.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.870 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: cluster 2024-03-11T21:34:38.593647+0000 osd.3 (osd.3) 273 : cluster 0 3.2 scrub starts 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: cluster 2024-03-11T21:34:38.596726+0000 osd.3 (osd.3) 274 : cluster 0 3.2 scrub ok 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: audit 2024-03-11T21:34:38.629879+0000 mgr.y (mgr.14550) 462 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: cluster 2024-03-11T21:34:38.845929+0000 osd.5 (osd.5) 275 : cluster 0 9.6 scrub starts 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: cluster 2024-03-11T21:34:38.848607+0000 osd.5 (osd.5) 276 : cluster 0 9.6 scrub ok 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.364702+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: audit 2024-03-11T21:34:39.374051+0000 mon.c (mon.2) 946 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: audit 2024-03-11T21:34:39.375509+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:39.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:39 smithi203 bash[23654]: audit 2024-03-11T21:34:39.404877+0000 mon.c (mon.2) 947 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:39.883 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:39.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:40.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:40.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.140 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:40.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: cluster 2024-03-11T21:34:38.593647+0000 osd.3 (osd.3) 273 : cluster 0 3.2 scrub starts 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: cluster 2024-03-11T21:34:38.596726+0000 osd.3 (osd.3) 274 : cluster 0 3.2 scrub ok 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: audit 2024-03-11T21:34:38.629879+0000 mgr.y (mgr.14550) 462 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: cluster 2024-03-11T21:34:38.845929+0000 osd.5 (osd.5) 275 : cluster 0 9.6 scrub starts 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: cluster 2024-03-11T21:34:38.848607+0000 osd.5 (osd.5) 276 : cluster 0 9.6 scrub ok 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.364702+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: audit 2024-03-11T21:34:39.374051+0000 mon.c (mon.2) 946 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: audit 2024-03-11T21:34:39.375509+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[29284]: audit 2024-03-11T21:34:39.404877+0000 mon.c (mon.2) 947 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: cluster 2024-03-11T21:34:38.593647+0000 osd.3 (osd.3) 273 : cluster 0 3.2 scrub starts 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: cluster 2024-03-11T21:34:38.596726+0000 osd.3 (osd.3) 274 : cluster 0 3.2 scrub ok 2024-03-11T21:34:40.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: audit 2024-03-11T21:34:38.629879+0000 mgr.y (mgr.14550) 462 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: cluster 2024-03-11T21:34:38.845929+0000 osd.5 (osd.5) 275 : cluster 0 9.6 scrub starts 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: cluster 2024-03-11T21:34:38.848607+0000 osd.5 (osd.5) 276 : cluster 0 9.6 scrub ok 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.364702+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: audit 2024-03-11T21:34:39.374051+0000 mon.c (mon.2) 946 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: audit 2024-03-11T21:34:39.375509+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:40.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:39 smithi143 bash[21512]: audit 2024-03-11T21:34:39.404877+0000 mon.c (mon.2) 947 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:40.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:40.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:40.370 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:40.398 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:40.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.529 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:40.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.659 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:40.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:40.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:40.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:40.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:41.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.049 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.163341+0000 osd.1 (osd.1) 71 : cluster 0 39.2 scrub starts 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.164417+0000 osd.1 (osd.1) 72 : cluster 0 39.2 scrub ok 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.393511+0000 mgr.y (mgr.14550) 463 : cluster 0 pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.586007+0000 osd.3 (osd.3) 275 : cluster 0 9.1a scrub starts 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.587521+0000 osd.3 (osd.3) 276 : cluster 0 9.1a scrub ok 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:39.934508+0000 mon.a (mon.0) 2752 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.943980+0000 osd.4 (osd.4) 235 : cluster 0 3.3 scrub starts 2024-03-11T21:34:41.051 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:39.945897+0000 mon.a (mon.0) 2753 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:39.945938+0000 osd.4 (osd.4) 236 : cluster 0 3.3 scrub ok 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.026850+0000 mon.c (mon.2) 948 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.027466+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.029281+0000 mon.c (mon.2) 949 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.029712+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.030710+0000 mon.c (mon.2) 950 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.031051+0000 mon.a (mon.0) 2756 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.031935+0000 mon.c (mon.2) 951 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.032289+0000 mon.a (mon.0) 2757 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cephadm 2024-03-11T21:34:40.032999+0000 mgr.y (mgr.14550) 464 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3292M 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.037957+0000 mon.a (mon.0) 2758 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.358890+0000 mon.a (mon.0) 2759 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:41.052 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.367057+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.163341+0000 osd.1 (osd.1) 71 : cluster 0 39.2 scrub starts 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.164417+0000 osd.1 (osd.1) 72 : cluster 0 39.2 scrub ok 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.393511+0000 mgr.y (mgr.14550) 463 : cluster 0 pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.586007+0000 osd.3 (osd.3) 275 : cluster 0 9.1a scrub starts 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.587521+0000 osd.3 (osd.3) 276 : cluster 0 9.1a scrub ok 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:39.934508+0000 mon.a (mon.0) 2752 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.943980+0000 osd.4 (osd.4) 235 : cluster 0 3.3 scrub starts 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:39.945897+0000 mon.a (mon.0) 2753 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:39.945938+0000 osd.4 (osd.4) 236 : cluster 0 3.3 scrub ok 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.026850+0000 mon.c (mon.2) 948 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.027466+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.053 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.029281+0000 mon.c (mon.2) 949 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.029712+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.030710+0000 mon.c (mon.2) 950 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.031051+0000 mon.a (mon.0) 2756 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.031935+0000 mon.c (mon.2) 951 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.032289+0000 mon.a (mon.0) 2757 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cephadm 2024-03-11T21:34:40.032999+0000 mgr.y (mgr.14550) 464 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3292M 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.037957+0000 mon.a (mon.0) 2758 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.358890+0000 mon.a (mon.0) 2759 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:41.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.367057+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:41.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.369566+0000 mon.a (mon.0) 2761 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:34:41.055 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.373543+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.376710+0000 mon.c (mon.2) 953 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.376985+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.418950+0000 osd.2 (osd.2) 221 : cluster 0 5.a scrub starts 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.424439+0000 osd.2 (osd.2) 222 : cluster 0 5.a scrub ok 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.589378+0000 mon.c (mon.2) 954 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.590148+0000 mon.a (mon.0) 2763 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.592062+0000 mon.c (mon.2) 955 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.592585+0000 mon.a (mon.0) 2764 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.594156+0000 mon.c (mon.2) 956 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.594694+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.596340+0000 mon.c (mon.2) 957 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.597158+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.609098+0000 mon.a (mon.0) 2767 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.056 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.615654+0000 mon.c (mon.2) 958 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.617044+0000 mon.c (mon.2) 959 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[21512]: audit 2024-03-11T21:34:40.631365+0000 mon.a (mon.0) 2768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.369566+0000 mon.a (mon.0) 2761 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.373543+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.376710+0000 mon.c (mon.2) 953 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.376985+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.418950+0000 osd.2 (osd.2) 221 : cluster 0 5.a scrub starts 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.424439+0000 osd.2 (osd.2) 222 : cluster 0 5.a scrub ok 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.589378+0000 mon.c (mon.2) 954 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.590148+0000 mon.a (mon.0) 2763 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.057 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.592062+0000 mon.c (mon.2) 955 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.592585+0000 mon.a (mon.0) 2764 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.594156+0000 mon.c (mon.2) 956 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.594694+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.596340+0000 mon.c (mon.2) 957 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.597158+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.609098+0000 mon.a (mon.0) 2767 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.615654+0000 mon.c (mon.2) 958 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.617044+0000 mon.c (mon.2) 959 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:41.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:40 smithi143 bash[29284]: audit 2024-03-11T21:34:40.631365+0000 mon.a (mon.0) 2768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:41.180 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.163341+0000 osd.1 (osd.1) 71 : cluster 0 39.2 scrub starts 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.164417+0000 osd.1 (osd.1) 72 : cluster 0 39.2 scrub ok 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.393511+0000 mgr.y (mgr.14550) 463 : cluster 0 pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.586007+0000 osd.3 (osd.3) 275 : cluster 0 9.1a scrub starts 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.587521+0000 osd.3 (osd.3) 276 : cluster 0 9.1a scrub ok 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:39.934508+0000 mon.a (mon.0) 2752 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.943980+0000 osd.4 (osd.4) 235 : cluster 0 3.3 scrub starts 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:39.945897+0000 mon.a (mon.0) 2753 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:39.945938+0000 osd.4 (osd.4) 236 : cluster 0 3.3 scrub ok 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.026850+0000 mon.c (mon.2) 948 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.027466+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.029281+0000 mon.c (mon.2) 949 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.029712+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.030710+0000 mon.c (mon.2) 950 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.031051+0000 mon.a (mon.0) 2756 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.031935+0000 mon.c (mon.2) 951 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.032289+0000 mon.a (mon.0) 2757 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cephadm 2024-03-11T21:34:40.032999+0000 mgr.y (mgr.14550) 464 : cephadm 1 Adjusting osd_memory_target on smithi203 to 3292M 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.037957+0000 mon.a (mon.0) 2758 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.358890+0000 mon.a (mon.0) 2759 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.367057+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.369566+0000 mon.a (mon.0) 2761 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.373543+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:41.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.376710+0000 mon.c (mon.2) 953 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.376985+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.418950+0000 osd.2 (osd.2) 221 : cluster 0 5.a scrub starts 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.424439+0000 osd.2 (osd.2) 222 : cluster 0 5.a scrub ok 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.589378+0000 mon.c (mon.2) 954 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.590148+0000 mon.a (mon.0) 2763 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.592062+0000 mon.c (mon.2) 955 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.592585+0000 mon.a (mon.0) 2764 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.594156+0000 mon.c (mon.2) 956 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.594694+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.596340+0000 mon.c (mon.2) 957 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.597158+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.609098+0000 mon.a (mon.0) 2767 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.615654+0000 mon.c (mon.2) 958 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.617044+0000 mon.c (mon.2) 959 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:41.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:40 smithi203 bash[23654]: audit 2024-03-11T21:34:40.631365+0000 mon.a (mon.0) 2768 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:34:41.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:41.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:41.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:41.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:41.541 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:41.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:41.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.694 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:41.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.035613+0000 osd.0 (osd.0) 265 : cluster 0 39.14 scrub starts 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: cluster 2024-03-11T21:34:40.037157+0000 osd.0 (osd.0) 266 : cluster 0 39.14 scrub ok 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: cephadm 2024-03-11T21:34:40.599037+0000 mgr.y (mgr.14550) 465 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3292M 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: audit 2024-03-11T21:34:41.371483+0000 mon.a (mon.0) 2769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.379468+0000 mon.a (mon.0) 2770 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:34:41.803 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: audit 2024-03-11T21:34:41.453112+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: audit 2024-03-11T21:34:41.453578+0000 mon.a (mon.0) 2771 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.804 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: audit 2024-03-11T21:34:41.454336+0000 mon.c (mon.2) 961 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:41.819 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:41.880 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:34:41.880 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:34:41.880 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:34:41.880 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:34:41.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.035613+0000 osd.0 (osd.0) 265 : cluster 0 39.14 scrub starts 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: cluster 2024-03-11T21:34:40.037157+0000 osd.0 (osd.0) 266 : cluster 0 39.14 scrub ok 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: cephadm 2024-03-11T21:34:40.599037+0000 mgr.y (mgr.14550) 465 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3292M 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: audit 2024-03-11T21:34:41.371483+0000 mon.a (mon.0) 2769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.379468+0000 mon.a (mon.0) 2770 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: audit 2024-03-11T21:34:41.453112+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: audit 2024-03-11T21:34:41.453578+0000 mon.a (mon.0) 2771 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.931 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: audit 2024-03-11T21:34:41.454336+0000 mon.c (mon.2) 961 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[21512]: audit 2024-03-11T21:34:41.454683+0000 mon.a (mon.0) 2772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.035613+0000 osd.0 (osd.0) 265 : cluster 0 39.14 scrub starts 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: cluster 2024-03-11T21:34:40.037157+0000 osd.0 (osd.0) 266 : cluster 0 39.14 scrub ok 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: cephadm 2024-03-11T21:34:40.599037+0000 mgr.y (mgr.14550) 465 : cephadm 1 Adjusting osd_memory_target on smithi143 to 3292M 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: audit 2024-03-11T21:34:41.371483+0000 mon.a (mon.0) 2769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.379468+0000 mon.a (mon.0) 2770 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: audit 2024-03-11T21:34:41.453112+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: audit 2024-03-11T21:34:41.453578+0000 mon.a (mon.0) 2771 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: audit 2024-03-11T21:34:41.454336+0000 mon.c (mon.2) 961 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:41.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:41 smithi143 bash[29284]: audit 2024-03-11T21:34:41.454683+0000 mon.a (mon.0) 2772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:41.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:42.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.075 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:42.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:42.190 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:41 smithi203 bash[23654]: audit 2024-03-11T21:34:41.454683+0000 mon.a (mon.0) 2772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-61"}]: dispatch 2024-03-11T21:34:42.206 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:42.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:42.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:42.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:42.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:42.695 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.716 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:42.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.067508+0000 osd.0 (osd.0) 267 : cluster 0 6.10 scrub starts 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.069425+0000 osd.0 (osd.0) 268 : cluster 0 6.10 scrub ok 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.171418+0000 osd.1 (osd.1) 73 : cluster 0 2.a scrub starts 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.173615+0000 osd.1 (osd.1) 74 : cluster 0 2.a scrub ok 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:41.394568+0000 mgr.y (mgr.14550) 466 : cluster 0 pgmap v699: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.238748+0000 osd.6 (osd.6) 201 : cluster 0 118.1d scrub starts 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.241595+0000 osd.6 (osd.6) 202 : cluster 0 118.1d scrub ok 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.383701+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.067508+0000 osd.0 (osd.0) 267 : cluster 0 6.10 scrub starts 2024-03-11T21:34:42.837 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.069425+0000 osd.0 (osd.0) 268 : cluster 0 6.10 scrub ok 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.171418+0000 osd.1 (osd.1) 73 : cluster 0 2.a scrub starts 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.173615+0000 osd.1 (osd.1) 74 : cluster 0 2.a scrub ok 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:41.394568+0000 mgr.y (mgr.14550) 466 : cluster 0 pgmap v699: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.238748+0000 osd.6 (osd.6) 201 : cluster 0 118.1d scrub starts 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.241595+0000 osd.6 (osd.6) 202 : cluster 0 118.1d scrub ok 2024-03-11T21:34:42.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:42 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.383701+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:34:42.850 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:42.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.982 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:43.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:43.112 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:43.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:43.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.067508+0000 osd.0 (osd.0) 267 : cluster 0 6.10 scrub starts 2024-03-11T21:34:43.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.069425+0000 osd.0 (osd.0) 268 : cluster 0 6.10 scrub ok 2024-03-11T21:34:43.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.171418+0000 osd.1 (osd.1) 73 : cluster 0 2.a scrub starts 2024-03-11T21:34:43.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.173615+0000 osd.1 (osd.1) 74 : cluster 0 2.a scrub ok 2024-03-11T21:34:43.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:41.394568+0000 mgr.y (mgr.14550) 466 : cluster 0 pgmap v699: 260 pgs: 28 unknown, 232 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:34:43.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.238748+0000 osd.6 (osd.6) 201 : cluster 0 118.1d scrub starts 2024-03-11T21:34:43.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.241595+0000 osd.6 (osd.6) 202 : cluster 0 118.1d scrub ok 2024-03-11T21:34:43.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:42 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.383701+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:34:43.243 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:43.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.382 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:43.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.512 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:43.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:43.644 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:43.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:43.774 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:43.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.089735+0000 osd.0 (osd.0) 269 : cluster 0 6.18 scrub starts 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.092331+0000 osd.0 (osd.0) 270 : cluster 0 6.18 scrub ok 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.855174+0000 osd.5 (osd.5) 277 : cluster 0 6.d scrub starts 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.856572+0000 osd.3 (osd.3) 277 : cluster 0 6.13 scrub starts 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.857438+0000 osd.5 (osd.5) 278 : cluster 0 6.d scrub ok 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:42.859441+0000 osd.3 (osd.3) 278 : cluster 0 6.13 scrub ok 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.033827+0000 osd.7 (osd.7) 281 : cluster 0 118.3 scrub starts 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.036575+0000 osd.7 (osd.7) 282 : cluster 0 118.3 scrub ok 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.410738+0000 mon.a (mon.0) 2774 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: audit 2024-03-11T21:34:43.414208+0000 mon.c (mon.2) 962 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:43.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: audit 2024-03-11T21:34:43.415932+0000 mon.a (mon.0) 2775 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:43.890 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.481853+0000 osd.2 (osd.2) 223 : cluster 0 5.b scrub starts 2024-03-11T21:34:43.890 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:43 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.484114+0000 osd.2 (osd.2) 224 : cluster 0 5.b scrub ok 2024-03-11T21:34:43.904 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:44.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:44.034 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:44.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.089735+0000 osd.0 (osd.0) 269 : cluster 0 6.18 scrub starts 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.092331+0000 osd.0 (osd.0) 270 : cluster 0 6.18 scrub ok 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.855174+0000 osd.5 (osd.5) 277 : cluster 0 6.d scrub starts 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.856572+0000 osd.3 (osd.3) 277 : cluster 0 6.13 scrub starts 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.857438+0000 osd.5 (osd.5) 278 : cluster 0 6.d scrub ok 2024-03-11T21:34:44.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:42.859441+0000 osd.3 (osd.3) 278 : cluster 0 6.13 scrub ok 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.033827+0000 osd.7 (osd.7) 281 : cluster 0 118.3 scrub starts 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.036575+0000 osd.7 (osd.7) 282 : cluster 0 118.3 scrub ok 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.410738+0000 mon.a (mon.0) 2774 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: audit 2024-03-11T21:34:43.414208+0000 mon.c (mon.2) 962 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: audit 2024-03-11T21:34:43.415932+0000 mon.a (mon.0) 2775 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.481853+0000 osd.2 (osd.2) 223 : cluster 0 5.b scrub starts 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.484114+0000 osd.2 (osd.2) 224 : cluster 0 5.b scrub ok 2024-03-11T21:34:44.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.089735+0000 osd.0 (osd.0) 269 : cluster 0 6.18 scrub starts 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.092331+0000 osd.0 (osd.0) 270 : cluster 0 6.18 scrub ok 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.855174+0000 osd.5 (osd.5) 277 : cluster 0 6.d scrub starts 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.856572+0000 osd.3 (osd.3) 277 : cluster 0 6.13 scrub starts 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.857438+0000 osd.5 (osd.5) 278 : cluster 0 6.d scrub ok 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:42.859441+0000 osd.3 (osd.3) 278 : cluster 0 6.13 scrub ok 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.033827+0000 osd.7 (osd.7) 281 : cluster 0 118.3 scrub starts 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.036575+0000 osd.7 (osd.7) 282 : cluster 0 118.3 scrub ok 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.410738+0000 mon.a (mon.0) 2774 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: audit 2024-03-11T21:34:43.414208+0000 mon.c (mon.2) 962 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: audit 2024-03-11T21:34:43.415932+0000 mon.a (mon.0) 2775 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.481853+0000 osd.2 (osd.2) 223 : cluster 0 5.b scrub starts 2024-03-11T21:34:44.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:43 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.484114+0000 osd.2 (osd.2) 224 : cluster 0 5.b scrub ok 2024-03-11T21:34:44.165 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:44.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.296 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:44.399 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:44.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:44.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:44.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:44.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:44.693 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:44.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:44.821 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:44.828 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.225314+0000 osd.1 (osd.1) 75 : cluster 0 5.19 scrub starts 2024-03-11T21:34:44.828 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.227848+0000 osd.1 (osd.1) 76 : cluster 0 5.19 scrub ok 2024-03-11T21:34:44.828 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.395602+0000 mgr.y (mgr.14550) 467 : cluster 0 pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:34:44.828 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.906745+0000 osd.4 (osd.4) 237 : cluster 0 5.1 scrub starts 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:43.908339+0000 osd.4 (osd.4) 238 : cluster 0 5.1 scrub ok 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.230066+0000 osd.6 (osd.6) 203 : cluster 0 2.17 scrub starts 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.232382+0000 osd.6 (osd.6) 204 : cluster 0 2.17 scrub ok 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: audit 2024-03-11T21:34:44.397224+0000 mon.a (mon.0) 2776 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.407064+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: audit 2024-03-11T21:34:44.419827+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: audit 2024-03-11T21:34:44.451009+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: audit 2024-03-11T21:34:44.451928+0000 mon.a (mon.0) 2778 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.455009+0000 osd.2 (osd.2) 225 : cluster 0 2.1 scrub starts 2024-03-11T21:34:44.829 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:44 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.457154+0000 osd.2 (osd.2) 226 : cluster 0 2.1 scrub ok 2024-03-11T21:34:44.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.952 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.225314+0000 osd.1 (osd.1) 75 : cluster 0 5.19 scrub starts 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.227848+0000 osd.1 (osd.1) 76 : cluster 0 5.19 scrub ok 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.395602+0000 mgr.y (mgr.14550) 467 : cluster 0 pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.906745+0000 osd.4 (osd.4) 237 : cluster 0 5.1 scrub starts 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:43.908339+0000 osd.4 (osd.4) 238 : cluster 0 5.1 scrub ok 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.230066+0000 osd.6 (osd.6) 203 : cluster 0 2.17 scrub starts 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.232382+0000 osd.6 (osd.6) 204 : cluster 0 2.17 scrub ok 2024-03-11T21:34:45.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: audit 2024-03-11T21:34:44.397224+0000 mon.a (mon.0) 2776 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.407064+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: audit 2024-03-11T21:34:44.419827+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: audit 2024-03-11T21:34:44.451009+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: audit 2024-03-11T21:34:44.451928+0000 mon.a (mon.0) 2778 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.455009+0000 osd.2 (osd.2) 225 : cluster 0 2.1 scrub starts 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.457154+0000 osd.2 (osd.2) 226 : cluster 0 2.1 scrub ok 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.225314+0000 osd.1 (osd.1) 75 : cluster 0 5.19 scrub starts 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.227848+0000 osd.1 (osd.1) 76 : cluster 0 5.19 scrub ok 2024-03-11T21:34:45.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.395602+0000 mgr.y (mgr.14550) 467 : cluster 0 pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.906745+0000 osd.4 (osd.4) 237 : cluster 0 5.1 scrub starts 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:43.908339+0000 osd.4 (osd.4) 238 : cluster 0 5.1 scrub ok 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.230066+0000 osd.6 (osd.6) 203 : cluster 0 2.17 scrub starts 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.232382+0000 osd.6 (osd.6) 204 : cluster 0 2.17 scrub ok 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: audit 2024-03-11T21:34:44.397224+0000 mon.a (mon.0) 2776 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.407064+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: audit 2024-03-11T21:34:44.419827+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: audit 2024-03-11T21:34:44.451009+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: audit 2024-03-11T21:34:44.451928+0000 mon.a (mon.0) 2778 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.455009+0000 osd.2 (osd.2) 225 : cluster 0 2.1 scrub starts 2024-03-11T21:34:45.003 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:44 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.457154+0000 osd.2 (osd.2) 226 : cluster 0 2.1 scrub ok 2024-03-11T21:34:45.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:45.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:45.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:45.211 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:45.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:45.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:45.446 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:45.472 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:45.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:45.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:45.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:45.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:45.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.189669+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.191426+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.834754+0000 osd.5 (osd.5) 279 : cluster 0 3.16 scrub starts 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:44.838417+0000 osd.5 (osd.5) 280 : cluster 0 3.16 scrub ok 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.183146+0000 osd.6 (osd.6) 205 : cluster 0 5.11 scrub starts 2024-03-11T21:34:45.853 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.185580+0000 osd.6 (osd.6) 206 : cluster 0 5.11 scrub ok 2024-03-11T21:34:45.854 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: audit 2024-03-11T21:34:45.426406+0000 mon.a (mon.0) 2779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:45.854 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:45 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.434007+0000 mon.a (mon.0) 2780 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:34:45.870 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:45.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:46.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:46.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.189669+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.191426+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.834754+0000 osd.5 (osd.5) 279 : cluster 0 3.16 scrub starts 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:44.838417+0000 osd.5 (osd.5) 280 : cluster 0 3.16 scrub ok 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.183146+0000 osd.6 (osd.6) 205 : cluster 0 5.11 scrub starts 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.185580+0000 osd.6 (osd.6) 206 : cluster 0 5.11 scrub ok 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: audit 2024-03-11T21:34:45.426406+0000 mon.a (mon.0) 2779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:46.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.434007+0000 mon.a (mon.0) 2780 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.189669+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.191426+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.834754+0000 osd.5 (osd.5) 279 : cluster 0 3.16 scrub starts 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:44.838417+0000 osd.5 (osd.5) 280 : cluster 0 3.16 scrub ok 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.183146+0000 osd.6 (osd.6) 205 : cluster 0 5.11 scrub starts 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.185580+0000 osd.6 (osd.6) 206 : cluster 0 5.11 scrub ok 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: audit 2024-03-11T21:34:45.426406+0000 mon.a (mon.0) 2779 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:34:46.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:45 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.434007+0000 mon.a (mon.0) 2780 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:34:46.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:46.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:46.262 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:46.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:46.392 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:46.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:46.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:46.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:46.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:34:46.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.778 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:46.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:47.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:47.031 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.396553+0000 mgr.y (mgr.14550) 468 : cluster 0 pgmap v704: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.787372+0000 osd.5 (osd.5) 281 : cluster 0 9.f scrub starts 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.789999+0000 osd.5 (osd.5) 282 : cluster 0 9.f scrub ok 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.932461+0000 osd.4 (osd.4) 239 : cluster 0 6.1a scrub starts 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:45.933651+0000 osd.4 (osd.4) 240 : cluster 0 6.1a scrub ok 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.187049+0000 osd.6 (osd.6) 207 : cluster 0 3.10 scrub starts 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.190289+0000 osd.6 (osd.6) 208 : cluster 0 3.10 scrub ok 2024-03-11T21:34:47.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.336989+0000 mon.a (mon.0) 2781 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.347769+0000 mon.a (mon.0) 2782 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.462060+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[21512]: cluster 2024-03-11T21:34:46.463485+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.396553+0000 mgr.y (mgr.14550) 468 : cluster 0 pgmap v704: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.787372+0000 osd.5 (osd.5) 281 : cluster 0 9.f scrub starts 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.789999+0000 osd.5 (osd.5) 282 : cluster 0 9.f scrub ok 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.932461+0000 osd.4 (osd.4) 239 : cluster 0 6.1a scrub starts 2024-03-11T21:34:47.034 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:45.933651+0000 osd.4 (osd.4) 240 : cluster 0 6.1a scrub ok 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.187049+0000 osd.6 (osd.6) 207 : cluster 0 3.10 scrub starts 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.190289+0000 osd.6 (osd.6) 208 : cluster 0 3.10 scrub ok 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.336989+0000 mon.a (mon.0) 2781 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.347769+0000 mon.a (mon.0) 2782 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.462060+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-03-11T21:34:47.035 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:46 smithi143 bash[29284]: cluster 2024-03-11T21:34:46.463485+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-03-11T21:34:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:47.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:47.165 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:47.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:47.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.396553+0000 mgr.y (mgr.14550) 468 : cluster 0 pgmap v704: 260 pgs: 3 creating+peering, 29 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.787372+0000 osd.5 (osd.5) 281 : cluster 0 9.f scrub starts 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.789999+0000 osd.5 (osd.5) 282 : cluster 0 9.f scrub ok 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.932461+0000 osd.4 (osd.4) 239 : cluster 0 6.1a scrub starts 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:45.933651+0000 osd.4 (osd.4) 240 : cluster 0 6.1a scrub ok 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.187049+0000 osd.6 (osd.6) 207 : cluster 0 3.10 scrub starts 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.190289+0000 osd.6 (osd.6) 208 : cluster 0 3.10 scrub ok 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.336989+0000 mon.a (mon.0) 2781 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.347769+0000 mon.a (mon.0) 2782 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.462060+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-03-11T21:34:47.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:46 smithi203 bash[23654]: cluster 2024-03-11T21:34:46.463485+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-03-11T21:34:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:47.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:47.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:47.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:47.555 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:47.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:47.686 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:47.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:47.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:47.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:47.948 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:48.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:48.078 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:48.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:48.208 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:48.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:48.340 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:48.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:48.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:48.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:48.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.362487+0000 mon.a (mon.0) 2783 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:34:48.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.397521+0000 mgr.y (mgr.14550) 469 : cluster 0 pgmap v708: 260 pgs: 3 creating+peering, 26 unknown, 231 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:48.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.884330+0000 osd.1 (osd.1) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:48.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.884794+0000 osd.1 (osd.1) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:48.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.062325+0000 osd.1 (osd.1) 81 : cluster 0 2.d scrub starts 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.064609+0000 osd.1 (osd.1) 82 : cluster 0 2.d scrub ok 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.362487+0000 mon.a (mon.0) 2783 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.397521+0000 mgr.y (mgr.14550) 469 : cluster 0 pgmap v708: 260 pgs: 3 creating+peering, 26 unknown, 231 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.884330+0000 osd.1 (osd.1) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.884794+0000 osd.1 (osd.1) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.062325+0000 osd.1 (osd.1) 81 : cluster 0 2.d scrub starts 2024-03-11T21:34:48.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:48 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.064609+0000 osd.1 (osd.1) 82 : cluster 0 2.d scrub ok 2024-03-11T21:34:48.607 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.362487+0000 mon.a (mon.0) 2783 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.397521+0000 mgr.y (mgr.14550) 469 : cluster 0 pgmap v708: 260 pgs: 3 creating+peering, 26 unknown, 231 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.884330+0000 osd.1 (osd.1) 79 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.884794+0000 osd.1 (osd.1) 80 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.062325+0000 osd.1 (osd.1) 81 : cluster 0 2.d scrub starts 2024-03-11T21:34:48.614 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.064609+0000 osd.1 (osd.1) 82 : cluster 0 2.d scrub ok 2024-03-11T21:34:48.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:48.734 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:48.750 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/64428702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98103b00 msgr2=0x7fbb98103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- 172.21.15.143:0/64428702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98103b00 0x7fbb98103f80 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009990 tx=0x7fbb88009ca0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/64428702 shutdown_connections 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- 172.21.15.143:0/64428702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb9810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- 172.21.15.143:0/64428702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9806a7f0 0x7fbb9806abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- 172.21.15.143:0/64428702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98103b00 0x7fbb98103f80 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/64428702 >> 172.21.15.143:0/64428702 conn(0x7fbb98076d80 msgr2=0x7fbb98077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/64428702 shutdown_connections 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/64428702 wait complete. 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 librados: starting msgr at 2024-03-11T21:34:48.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 librados: starting objecter 2024-03-11T21:34:48.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 Processor -- start 2024-03-11T21:34:48.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- start start 2024-03-11T21:34:48.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 librados: setting wanted keys 2024-03-11T21:34:48.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 librados: calling monclient init 2024-03-11T21:34:48.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:48.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 0x7fbb981013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:48.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:48.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb981117a0 con 0x7fbb9806a7f0 2024-03-11T21:34:48.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98111620 con 0x7fbb98103b00 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98111920 con 0x7fbb981044c0 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37770/0 (socket says 172.21.15.143:37770) 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33514/0 (socket says 172.21.15.143:33514) 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 -- 172.21.15.143:0/1239666205 learned_addr learned my addr 172.21.15.143:0/1239666205 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb977fe700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 0x7fbb981013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:48.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 -- 172.21.15.143:0/1239666205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 msgr2=0x7fbb981013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 0x7fbb981013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 -- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 msgr2=0x7fbb98100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 -- 172.21.15.143:0/1239666205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb981a6260 con 0x7fbb981044c0 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb97fff700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb977fe700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 0x7fbb981013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9c800700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000ef30 tx=0x7fbb9000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb957fa700 10 client.?.objecter ms_handle_connect 0x7fbb981044c0 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb957fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:48.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900092e0 con 0x7fbb981044c0 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb981a64f0 con 0x7fbb981044c0 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fbb90004bb0 con 0x7fbb981044c0 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90004020 con 0x7fbb981044c0 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 10 client.41642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb981a6a30 con 0x7fbb981044c0 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.799+0000 7fbb9e263700 1 librados: init done 2024-03-11T21:34:48.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9804faa0 con 0x7fbb981044c0 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbb9001b030 con 0x7fbb981044c0 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 0x7fbb80071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(478..478 src has 87..478) v4 ==== 9629+0+0 (secure 0 0 0) 0x7fbb9008ef90 con 0x7fbb981044c0 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 10 client.41642.objecter ms_dispatch 0x7fbb98001550 osd_map(478..478 src has 87..478) v4 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb97fff700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 0x7fbb80071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 3 client.41642.objecter handle_osd_map got epochs [478,478] > 0 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 3 client.41642.objecter handle_osd_map decoding full epoch 478 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 20 client.41642.objecter dump_active .. 0 homeless 2024-03-11T21:34:48.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb957fa700 10 client.41642.objecter ms_handle_connect 0x7fbb8006f820 2024-03-11T21:34:48.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.803+0000 7fbb97fff700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 0x7fbb80071ce0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c005950 tx=0x7fbb8c00b410 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:48.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:48.807+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fbb90022020 con 0x7fbb981044c0 2024-03-11T21:34:48.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:48.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.015+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb98108340 con 0x7fbb8006f820 2024-03-11T21:34:48.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.025 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:49.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:49.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.051+0000 7fbb957fa700 1 -- 172.21.15.143:0/1239666205 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+665197 (secure 0 0 0) 0x7fbb98108340 con 0x7fbb8006f820 2024-03-11T21:34:49.029 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:34:49.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 10 librados: watch_flush enter 2024-03-11T21:34:49.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 10 librados: watch_flush exit 2024-03-11T21:34:49.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 20 client.41642.objecter shutdown clearing up homeless session... 2024-03-11T21:34:49.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 10 client.41642.objecter successfully canceled tick 2024-03-11T21:34:49.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 msgr2=0x7fbb80071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 0x7fbb80071ce0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c005950 tx=0x7fbb8c00b410 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 msgr2=0x7fbb98101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000ef30 tx=0x7fbb9000c3a0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 shutdown_connections 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbb8006f820 0x7fbb80071ce0 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb9806a7f0 0x7fbb98100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb981044c0 0x7fbb98101930 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 --2- 172.21.15.143:0/1239666205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb98103b00 0x7fbb981013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 >> 172.21.15.143:0/1239666205 conn(0x7fbb98076d80 msgr2=0x7fbb98106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 shutdown_connections 2024-03-11T21:34:49.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 -- 172.21.15.143:0/1239666205 wait complete. 2024-03-11T21:34:49.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.059+0000 7fbb9e263700 1 librados: shutdown 2024-03-11T21:34:49.044 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:34:49.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:49.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:49.282 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:49.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:49.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:49.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.957175+0000 osd.4 (osd.4) 241 : cluster 0 9.16 scrub starts 2024-03-11T21:34:49.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: cluster 2024-03-11T21:34:47.959711+0000 osd.4 (osd.4) 242 : cluster 0 9.16 scrub ok 2024-03-11T21:34:49.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.371112+0000 mon.a (mon.0) 2784 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:34:49.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.408737+0000 osd.2 (osd.2) 229 : cluster 0 39.6 scrub starts 2024-03-11T21:34:49.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.410119+0000 osd.2 (osd.2) 230 : cluster 0 39.6 scrub ok 2024-03-11T21:34:49.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: audit 2024-03-11T21:34:48.641207+0000 mgr.y (mgr.14550) 470 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:49.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:49 smithi203 bash[23654]: audit 2024-03-11T21:34:49.018082+0000 mgr.y (mgr.14550) 471 : audit 0 from='client.41642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:49.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:49.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.957175+0000 osd.4 (osd.4) 241 : cluster 0 9.16 scrub starts 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: cluster 2024-03-11T21:34:47.959711+0000 osd.4 (osd.4) 242 : cluster 0 9.16 scrub ok 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.371112+0000 mon.a (mon.0) 2784 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.408737+0000 osd.2 (osd.2) 229 : cluster 0 39.6 scrub starts 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.410119+0000 osd.2 (osd.2) 230 : cluster 0 39.6 scrub ok 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: audit 2024-03-11T21:34:48.641207+0000 mgr.y (mgr.14550) 470 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:49.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[29284]: audit 2024-03-11T21:34:49.018082+0000 mgr.y (mgr.14550) 471 : audit 0 from='client.41642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.957175+0000 osd.4 (osd.4) 241 : cluster 0 9.16 scrub starts 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: cluster 2024-03-11T21:34:47.959711+0000 osd.4 (osd.4) 242 : cluster 0 9.16 scrub ok 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.371112+0000 mon.a (mon.0) 2784 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.408737+0000 osd.2 (osd.2) 229 : cluster 0 39.6 scrub starts 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.410119+0000 osd.2 (osd.2) 230 : cluster 0 39.6 scrub ok 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: audit 2024-03-11T21:34:48.641207+0000 mgr.y (mgr.14550) 470 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:49.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:49 smithi143 bash[21512]: audit 2024-03-11T21:34:49.018082+0000 mgr.y (mgr.14550) 471 : audit 0 from='client.41642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:49.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:49.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:49.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:49.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- 172.21.15.143:0/2937369709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 msgr2=0x7fe108106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- 172.21.15.143:0/2937369709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 0x7fe108106aa0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00099c0 tx=0x7fe0f0009cd0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- 172.21.15.143:0/2937369709 shutdown_connections 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- 172.21.15.143:0/2937369709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- 172.21.15.143:0/2937369709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1081071d0 0x7fe10810ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- 172.21.15.143:0/2937369709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 0x7fe108106aa0 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- 172.21.15.143:0/2937369709 >> 172.21.15.143:0/2937369709 conn(0x7fe1080772e0 msgr2=0x7fe1080fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- 172.21.15.143:0/2937369709 shutdown_connections 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- 172.21.15.143:0/2937369709 wait complete. 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 librados: starting msgr at 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 librados: starting objecter 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 Processor -- start 2024-03-11T21:34:49.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- start start 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 librados: setting wanted keys 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 librados: calling monclient init 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 0x7fe108101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1081071d0 0x7fe108101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1081116e0 con 0x7fe1081005b0 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe108111560 con 0x7fe1081071d0 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe10e231700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe108111860 con 0x7fe108100f60 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33522/0 (socket says 172.21.15.143:33522) 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe106ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 0x7fe108101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.979+0000 7fe1077fe700 1 -- 172.21.15.143:0/2235092745 learned_addr learned my addr 172.21.15.143:0/2235092745 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1081071d0 0x7fe108101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 msgr2=0x7fe108101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:49.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe108100f60 0x7fe108101a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1081071d0 msgr2=0x7fe108101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1081071d0 0x7fe108101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1081a61b0 con 0x7fe1081005b0 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fe10000c370 tx=0x7fe10000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 10 client.?.objecter ms_handle_connect 0x7fe1081005b0 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000cea0 con 0x7fe1081005b0 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1081a6440 con 0x7fe1081005b0 2024-03-11T21:34:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe1000090d0 con 0x7fe1081005b0 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100004660 con 0x7fe1081005b0 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 10 client.30447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1081a69a0 con 0x7fe1081005b0 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 librados: init done 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe10804faa0 con 0x7fe1081005b0 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe10001a020 con 0x7fe1081005b0 2024-03-11T21:34:49.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 0x7fe0f4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(479..479 src has 87..479) v4 ==== 9685+0+0 (secure 0 0 0) 0x7fe10008f000 con 0x7fe1081005b0 2024-03-11T21:34:49.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 osd_map(479..479 src has 87..479) v4 2024-03-11T21:34:49.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 3 client.30447.objecter handle_osd_map got epochs [479,479] > 0 2024-03-11T21:34:49.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 3 client.30447.objecter handle_osd_map decoding full epoch 479 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 0x7fe0f4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 20 client.30447.objecter dump_active .. 0 homeless 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v479) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe100010040 con 0x7fe1081005b0 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 10 client.30447.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 1 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe1080629f0 0x7fe108062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 1 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 0 1 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 4 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 10 client.30447.objecter _send_command 1 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081a9d60 con 0x7fe1080629f0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe0f406f820 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 0x7fe0f4071ce0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00098e0 tx=0x7fe0f0005a90 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe1080629f0 0x7fe108062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe1080629f0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.983+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe1080629f0 0x7fe108062dd0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.987+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081a9d60 con 0x7fe1080629f0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.987+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.987+0000 7fe104ff9700 10 client.30447.objecter _finish_command 1 = system:0 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.987+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 4 2024-03-11T21:34:49.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:49.987+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 0 1 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 10 client.30447.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 2 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 2 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe108102a60 osd=0 3 2024-03-11T21:34:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 0 2 2024-03-11T21:34:49.985 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 4 2024-03-11T21:34:49.985 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 10 client.30447.objecter _send_command 2 2024-03-11T21:34:49.985 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe108062470 con 0x7fe1080629f0 2024-03-11T21:34:49.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe108062470 con 0x7fe1080629f0 2024-03-11T21:34:49.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:49.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe104ff9700 10 client.30447.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:49.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 4 2024-03-11T21:34:49.986 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.007+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 0 2 2024-03-11T21:34:49.987 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:34:49.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 10 client.30447.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:49.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 3 2024-03-11T21:34:49.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe10802c6b0 0x7fe10802ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 1 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 4 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 10 client.30447.objecter _send_command 3 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081a9a30 con 0x7fe10802c6b0 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe10802c6b0 0x7fe10802ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe10802c6b0 0x7fe10802ca90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe10802c6b0 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081a9a30 con 0x7fe10802c6b0 2024-03-11T21:34:49.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:34:49.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 10 client.30447.objecter _finish_command 3 = system:0 2024-03-11T21:34:49.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 4 2024-03-11T21:34:49.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.011+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 1 3 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 10 client.30447.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 4 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 4 2024-03-11T21:34:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081a93b0 osd=1 3 2024-03-11T21:34:50.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 1 4 2024-03-11T21:34:50.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 4 2024-03-11T21:34:50.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 10 client.30447.objecter _send_command 4 2024-03-11T21:34:50.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.027+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe1081a8e30 con 0x7fe10802c6b0 2024-03-11T21:34:50.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1081a8e30 con 0x7fe10802c6b0 2024-03-11T21:34:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 10 client.30447.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 4 2024-03-11T21:34:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 1 4 2024-03-11T21:34:50.004 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 10 client.30447.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 5 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe1081abf20 0x7fe1081ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 5 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 2 5 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 4 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 10 client.30447.objecter _send_command 5 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe10802dcb0 con 0x7fe1081abf20 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe1081abf20 0x7fe1081ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe1081abf20 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe1081abf20 0x7fe1081ae310 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe10802dcb0 con 0x7fe1081abf20 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 10 client.30447.objecter _finish_command 5 = system:0 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 4 2024-03-11T21:34:50.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.031+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 2 5 2024-03-11T21:34:50.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:50.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 10 client.30447.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 6 2024-03-11T21:34:50.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 6 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081a9230 osd=2 3 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 2 6 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 4 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 10 client.30447.objecter _send_command 6 2024-03-11T21:34:50.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe10802dcb0 con 0x7fe1081abf20 2024-03-11T21:34:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe10802dcb0 con 0x7fe1081abf20 2024-03-11T21:34:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe104ff9700 10 client.30447.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 4 2024-03-11T21:34:50.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 2 6 2024-03-11T21:34:50.043 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:34:50.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 10 client.30447.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 7 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe108109950 0x7fe1081b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 7 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 3 7 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 4 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 10 client.30447.objecter _send_command 7 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081b4840 con 0x7fe108109950 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.067+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe108109950 0x7fe1081b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe108109950 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe108109950 0x7fe1081b19e0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081b4840 con 0x7fe108109950 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 10 client.30447.objecter _finish_command 7 = system:0 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 4 2024-03-11T21:34:50.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.071+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 3 7 2024-03-11T21:34:50.057 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 10 client.30447.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 8 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 8 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe10802dcb0 osd=3 3 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 3 8 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 4 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 10 client.30447.objecter _send_command 8 2024-03-11T21:34:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe108108ec0 con 0x7fe108109950 2024-03-11T21:34:50.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.103+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe108108ec0 con 0x7fe108109950 2024-03-11T21:34:50.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 10 client.30447.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 4 2024-03-11T21:34:50.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 3 8 2024-03-11T21:34:50.080 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:34:50.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 10 client.30447.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 9 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe108109030 0x7fe1081b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 9 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 4 9 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 4 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 10 client.30447.objecter _send_command 9 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081af730 con 0x7fe108109030 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe108109030 0x7fe1081b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe108109030 2024-03-11T21:34:50.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe108109030 0x7fe1081b50b0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081af730 con 0x7fe108109030 2024-03-11T21:34:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:34:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 10 client.30447.objecter _finish_command 9 = system:0 2024-03-11T21:34:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 4 2024-03-11T21:34:50.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.107+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 4 9 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 10 client.30447.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 10 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 10 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe108108ec0 osd=4 3 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 4 10 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 4 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 10 client.30447.objecter _send_command 10 2024-03-11T21:34:50.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.123+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe1081af730 con 0x7fe108109030 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1081af730 con 0x7fe108109030 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 10 client.30447.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 4 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 4 10 2024-03-11T21:34:50.101 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 10 client.30447.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 11 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe1081b5b00 0x7fe1081b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 11 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 5 11 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 4 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 10 client.30447.objecter _send_command 11 2024-03-11T21:34:50.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081bb0f0 con 0x7fe1081b5b00 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe1081b5b00 0x7fe1081b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe1081b5b00 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe106ffd700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe1081b5b00 0x7fe1081b5f40 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081bb0f0 con 0x7fe1081b5b00 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 10 client.30447.objecter _finish_command 11 = system:0 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 4 2024-03-11T21:34:50.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.127+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 5 11 2024-03-11T21:34:50.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 10 client.30447.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 12 2024-03-11T21:34:50.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 12 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081af730 osd=5 3 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 5 12 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 4 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 10 client.30447.objecter _send_command 12 2024-03-11T21:34:50.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe1081b6590 con 0x7fe1081b5b00 2024-03-11T21:34:50.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1081b6590 con 0x7fe1081b5b00 2024-03-11T21:34:50.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe104ff9700 10 client.30447.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 4 2024-03-11T21:34:50.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 5 12 2024-03-11T21:34:50.123 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 10 client.30447.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 13 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe1081b6cb0 0x7fe1081b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 13 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 6 13 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 4 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 10 client.30447.objecter _send_command 13 2024-03-11T21:34:50.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.147+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081b77a0 con 0x7fe1081b6cb0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe1081b6cb0 0x7fe1081b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe1081b6cb0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe107fff700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe1081b6cb0 0x7fe1081b70f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081b77a0 con 0x7fe1081b6cb0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 10 client.30447.objecter _finish_command 13 = system:0 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 4 2024-03-11T21:34:50.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.151+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 6 13 2024-03-11T21:34:50.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 10 client.30447.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 14 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 14 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081b6590 osd=6 3 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 6 14 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 4 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 10 client.30447.objecter _send_command 14 2024-03-11T21:34:50.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.167+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe1081b77a0 con 0x7fe1081b6cb0 2024-03-11T21:34:50.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1081b77a0 con 0x7fe1081b6cb0 2024-03-11T21:34:50.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 10 client.30447.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 4 2024-03-11T21:34:50.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 6 14 2024-03-11T21:34:50.146 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:34:50.146 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 10 client.30447.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 15 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe1081b7910 0x7fe1081be4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 15 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 7 15 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 4 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 10 client.30447.objecter _send_command 15 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1081c1540 con 0x7fe1081b7910 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe1081b7910 0x7fe1081be4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe1077fe700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe1081b7910 0x7fe1081be4c0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.171+0000 7fe104ff9700 10 client.30447.objecter ms_handle_connect 0x7fe1081b7910 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.175+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1081c1540 con 0x7fe1081b7910 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.175+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.175+0000 7fe104ff9700 10 client.30447.objecter _finish_command 15 = system:0 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.175+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 4 2024-03-11T21:34:50.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.175+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 7 15 2024-03-11T21:34:50.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 10 client.30447.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign -1 16 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 20 client.30447.objecter _get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 15 client.30447.objecter _session_command_op_remove -1 16 2024-03-11T21:34:50.165 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 20 client.30447.objecter get_session s=0x7fe1081b77a0 osd=7 3 2024-03-11T21:34:50.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 15 client.30447.objecter _session_command_op_assign 7 16 2024-03-11T21:34:50.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 4 2024-03-11T21:34:50.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 10 client.30447.objecter _send_command 16 2024-03-11T21:34:50.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.191+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe1081c15a0 con 0x7fe1081b7910 2024-03-11T21:34:50.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe104ff9700 1 -- 172.21.15.143:0/2235092745 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1081c15a0 con 0x7fe1081b7910 2024-03-11T21:34:50.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe104ff9700 10 client.30447.objecter ms_dispatch 0x7fe108001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:34:50.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe104ff9700 10 client.30447.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:34:50.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe104ff9700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 4 2024-03-11T21:34:50.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe104ff9700 15 client.30447.objecter _session_command_op_remove 7 16 2024-03-11T21:34:50.170 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:34:50.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 librados: watch_flush enter 2024-03-11T21:34:50.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 librados: watch_flush exit 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.0 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe1080629f0 msgr2=0x7fe108062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe1080629f0 0x7fe108062dd0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108102a60 osd=0 1 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.1 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe10802c6b0 msgr2=0x7fe10802ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe10802c6b0 0x7fe10802ca90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a93b0 osd=1 1 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.2 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe1081abf20 msgr2=0x7fe1081ae310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe1081abf20 0x7fe1081ae310 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081a9230 osd=2 1 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.3 2024-03-11T21:34:50.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe108109950 msgr2=0x7fe1081b19e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe107fff700 1 -- 172.21.15.143:0/2235092745 reap_dead start 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe108109950 0x7fe1081b19e0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe10802dcb0 osd=3 1 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.4 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe108109030 msgr2=0x7fe1081b50b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe108109030 0x7fe1081b50b0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe108108ec0 osd=4 1 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.5 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe1081b5b00 msgr2=0x7fe1081b5f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe1081b5b00 0x7fe1081b5f40 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081af730 osd=5 1 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.6 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe1081b6cb0 msgr2=0x7fe1081b70f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe1081b6cb0 0x7fe1081b70f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b6590 osd=6 1 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter close_session for osd.7 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe1081b7910 msgr2=0x7fe1081be4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe1081b7910 0x7fe1081be4c0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter put_session s=0x7fe1081b77a0 osd=7 1 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 20 client.30447.objecter shutdown clearing up homeless session... 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 10 client.30447.objecter successfully canceled tick 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe107fff700 1 -- 172.21.15.143:0/2235092745 reap_dead start 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 msgr2=0x7fe0f4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 0x7fe0f4071ce0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00098e0 tx=0x7fe0f0005a90 comp rx=0 tx=0).stop 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 msgr2=0x7fe108101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fe10000c370 tx=0x7fe10000c680 comp rx=0 tx=0).stop 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 shutdown_connections 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1081005b0 0x7fe108101530 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:50.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 --2- 172.21.15.143:0/2235092745 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe0f406f820 0x7fe0f4071ce0 secure :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00098e0 tx=0x7fe0f0005a90 comp rx=0 tx=0).stop 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 >> 172.21.15.143:0/2235092745 conn(0x7fe1080772e0 msgr2=0x7fe108104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 shutdown_connections 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 -- 172.21.15.143:0/2235092745 wait complete. 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:50.195+0000 7fe10e231700 1 librados: shutdown 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:50.175 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:50.176 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:50.176 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:50.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:50.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.304 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:50.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.741252+0000 osd.4 (osd.4) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:48.741620+0000 osd.4 (osd.4) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:49.387276+0000 mon.a (mon.0) 2785 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:49.398667+0000 mgr.y (mgr.14550) 472 : cluster 0 pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:49.556152+0000 osd.3 (osd.3) 279 : cluster 0 39.11 scrub starts 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:49.557484+0000 osd.3 (osd.3) 280 : cluster 0 39.11 scrub ok 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: audit 2024-03-11T21:34:49.985786+0000 mon.a (mon.0) 2786 : audit 0 from='client.? 172.21.15.143:0/2235092745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.110892+0000 osd.7 (osd.7) 283 : cluster 0 118.16 scrub starts 2024-03-11T21:34:50.511 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.112520+0000 osd.7 (osd.7) 284 : cluster 0 118.16 scrub ok 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.741252+0000 osd.4 (osd.4) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:48.741620+0000 osd.4 (osd.4) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:49.387276+0000 mon.a (mon.0) 2785 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:49.398667+0000 mgr.y (mgr.14550) 472 : cluster 0 pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:49.556152+0000 osd.3 (osd.3) 279 : cluster 0 39.11 scrub starts 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:49.557484+0000 osd.3 (osd.3) 280 : cluster 0 39.11 scrub ok 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: audit 2024-03-11T21:34:49.985786+0000 mon.a (mon.0) 2786 : audit 0 from='client.? 172.21.15.143:0/2235092745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.110892+0000 osd.7 (osd.7) 283 : cluster 0 118.16 scrub starts 2024-03-11T21:34:50.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:50 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.112520+0000 osd.7 (osd.7) 284 : cluster 0 118.16 scrub ok 2024-03-11T21:34:50.535 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.562 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:50.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:50.692 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.741252+0000 osd.4 (osd.4) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:48.741620+0000 osd.4 (osd.4) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:49.387276+0000 mon.a (mon.0) 2785 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:49.398667+0000 mgr.y (mgr.14550) 472 : cluster 0 pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:49.556152+0000 osd.3 (osd.3) 279 : cluster 0 39.11 scrub starts 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:49.557484+0000 osd.3 (osd.3) 280 : cluster 0 39.11 scrub ok 2024-03-11T21:34:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: audit 2024-03-11T21:34:49.985786+0000 mon.a (mon.0) 2786 : audit 0 from='client.? 172.21.15.143:0/2235092745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:50.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.110892+0000 osd.7 (osd.7) 283 : cluster 0 118.16 scrub starts 2024-03-11T21:34:50.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:50 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.112520+0000 osd.7 (osd.7) 284 : cluster 0 118.16 scrub ok 2024-03-11T21:34:50.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:50.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:50.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.953 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:51.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:51.085 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:51.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:51.214 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:51.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:51.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:51.479 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:51.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:51.607 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:51.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.365895+0000 osd.2 (osd.2) 231 : cluster 0 2.10 scrub starts 2024-03-11T21:34:51.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.366801+0000 osd.2 (osd.2) 232 : cluster 0 2.10 scrub ok 2024-03-11T21:34:51.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[21512]: cluster 2024-03-11T21:34:51.348843+0000 mon.a (mon.0) 2787 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:34:51.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.365895+0000 osd.2 (osd.2) 231 : cluster 0 2.10 scrub starts 2024-03-11T21:34:51.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.366801+0000 osd.2 (osd.2) 232 : cluster 0 2.10 scrub ok 2024-03-11T21:34:51.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:51 smithi143 bash[29284]: cluster 2024-03-11T21:34:51.348843+0000 mon.a (mon.0) 2787 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:34:51.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:51.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:51 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.365895+0000 osd.2 (osd.2) 231 : cluster 0 2.10 scrub starts 2024-03-11T21:34:51.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:51 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.366801+0000 osd.2 (osd.2) 232 : cluster 0 2.10 scrub ok 2024-03-11T21:34:51.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:51 smithi203 bash[23654]: cluster 2024-03-11T21:34:51.348843+0000 mon.a (mon.0) 2787 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:34:51.738 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:51.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:51.869 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:51.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.000 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:52.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:52.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:52.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:52.260 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:52.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.371 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:34:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.789330+0000 osd.5 (osd.5) 283 : cluster 0 3.1d scrub starts 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: cluster 2024-03-11T21:34:50.793803+0000 osd.5 (osd.5) 284 : cluster 0 3.1d scrub ok 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: cluster 2024-03-11T21:34:51.088148+0000 osd.0 (osd.0) 271 : cluster 0 9.1c scrub starts 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: cluster 2024-03-11T21:34:51.090578+0000 osd.0 (osd.0) 272 : cluster 0 9.1c scrub ok 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: cluster 2024-03-11T21:34:51.399224+0000 mgr.y (mgr.14550) 473 : cluster 0 pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:34:52.373 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[29284]: audit 2024-03-11T21:34:51.464038+0000 mon.c (mon.2) 965 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.382 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-03-11T21:34:48.371531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792196,"num_objects":363,"num_object_clones":4,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":2183,"num_read_kb":1926,"num_write":1512,"num_write_kb":15157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2873,"ondisk_log_size":2873,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":764,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":612824,"kb_used_data":88856,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749348392,"statfs":{"total":767960285184,"available":767332753408,"internally_reserved":0,"allocated":90988544,"data_stored":65664616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":8,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.971444"},"pg_stats":[{"pgid":"287.1f","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376328+0000","last_change":"2024-03-11T21:34:44.429500+0000","last_active":"2024-03-11T21:34:47.376328+0000","last_peered":"2024-03-11T21:34:47.376328+0000","last_clean":"2024-03-11T21:34:47.376328+0000","last_became_active":"2024-03-11T21:34:44.428622+0000","last_became_peered":"2024-03-11T21:34:44.428622+0000","last_unstale":"2024-03-11T21:34:47.376328+0000","last_undegraded":"2024-03-11T21:34:47.376328+0000","last_fullsized":"2024-03-11T21:34:47.376328+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.223561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.1e","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368235+0000","last_change":"2024-03-11T21:34:44.441386+0000","last_active":"2024-03-11T21:34:47.368235+0000","last_peered":"2024-03-11T21:34:47.368235+0000","last_clean":"2024-03-11T21:34:47.368235+0000","last_became_active":"2024-03-11T21:34:44.440753+0000","last_became_peered":"2024-03-11T21:34:44.440753+0000","last_unstale":"2024-03-11T21:34:47.368235+0000","last_undegraded":"2024-03-11T21:34:47.368235+0000","last_fullsized":"2024-03-11T21:34:47.368235+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.139095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.1d","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376047+0000","last_change":"2024-03-11T21:34:44.430478+0000","last_active":"2024-03-11T21:34:47.376047+0000","last_peered":"2024-03-11T21:34:47.376047+0000","last_clean":"2024-03-11T21:34:47.376047+0000","last_became_active":"2024-03-11T21:34:44.429897+0000","last_became_peered":"2024-03-11T21:34:44.429897+0000","last_unstale":"2024-03-11T21:34:47.376047+0000","last_undegraded":"2024-03-11T21:34:47.376047+0000","last_fullsized":"2024-03-11T21:34:47.376047+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.403818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.1c","version":"475'2","reported_seq":19,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391171+0000","last_change":"2024-03-11T21:34:44.456144+0000","last_active":"2024-03-11T21:34:47.391171+0000","last_peered":"2024-03-11T21:34:47.391171+0000","last_clean":"2024-03-11T21:34:47.391171+0000","last_became_active":"2024-03-11T21:34:44.455801+0000","last_became_peered":"2024-03-11T21:34:44.455801+0000","last_unstale":"2024-03-11T21:34:47.391171+0000","last_undegraded":"2024-03-11T21:34:47.391171+0000","last_fullsized":"2024-03-11T21:34:47.391171+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.1b","version":"0'0","reported_seq":1,"reported_epoch":473,"state":"creating+peering","last_fresh":"2024-03-11T21:34:43.414474+0000","last_change":"2024-03-11T21:34:43.414474+0000","last_active":"2024-03-11T21:34:43.390268+0000","last_peered":"2024-03-11T21:34:43.390268+0000","last_clean":"2024-03-11T21:34:43.390268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:43.414474+0000","last_undegraded":"2024-03-11T21:34:43.414474+0000","last_fullsized":"2024-03-11T21:34:43.414474+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.1a","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375407+0000","last_change":"2024-03-11T21:34:44.435062+0000","last_active":"2024-03-11T21:34:47.375407+0000","last_peered":"2024-03-11T21:34:47.375407+0000","last_clean":"2024-03-11T21:34:47.375407+0000","last_became_active":"2024-03-11T21:34:44.434403+0000","last_became_peered":"2024-03-11T21:34:44.434403+0000","last_unstale":"2024-03-11T21:34:47.375407+0000","last_undegraded":"2024-03-11T21:34:47.375407+0000","last_fullsized":"2024-03-11T21:34:47.375407+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.19","version":"477'3","reported_seq":22,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376011+0000","last_change":"2024-03-11T21:34:44.432658+0000","last_active":"2024-03-11T21:34:47.376011+0000","last_peered":"2024-03-11T21:34:47.376011+0000","last_clean":"2024-03-11T21:34:47.376011+0000","last_became_active":"2024-03-11T21:34:44.432212+0000","last_became_peered":"2024-03-11T21:34:44.432212+0000","last_unstale":"2024-03-11T21:34:47.376011+0000","last_undegraded":"2024-03-11T21:34:47.376011+0000","last_fullsized":"2024-03-11T21:34:47.376011+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.930357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.18","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390287+0000","last_change":"2024-03-11T21:34:44.435887+0000","last_active":"2024-03-11T21:34:47.390287+0000","last_peered":"2024-03-11T21:34:47.390287+0000","last_clean":"2024-03-11T21:34:47.390287+0000","last_became_active":"2024-03-11T21:34:44.435001+0000","last_became_peered":"2024-03-11T21:34:44.435001+0000","last_unstale":"2024-03-11T21:34:47.390287+0000","last_undegraded":"2024-03-11T21:34:47.390287+0000","last_fullsized":"2024-03-11T21:34:47.390287+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:13.019156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.17","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368757+0000","last_change":"2024-03-11T21:34:44.443881+0000","last_active":"2024-03-11T21:34:47.368757+0000","last_peered":"2024-03-11T21:34:47.368757+0000","last_clean":"2024-03-11T21:34:47.368757+0000","last_became_active":"2024-03-11T21:34:44.443342+0000","last_became_peered":"2024-03-11T21:34:44.443342+0000","last_unstale":"2024-03-11T21:34:47.368757+0000","last_undegraded":"2024-03-11T21:34:47.368757+0000","last_fullsized":"2024-03-11T21:34:47.368757+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.619722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.16","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368307+0000","last_change":"2024-03-11T21:34:44.445594+0000","last_active":"2024-03-11T21:34:47.368307+0000","last_peered":"2024-03-11T21:34:47.368307+0000","last_clean":"2024-03-11T21:34:47.368307+0000","last_became_active":"2024-03-11T21:34:44.445102+0000","last_became_peered":"2024-03-11T21:34:44.445102+0000","last_unstale":"2024-03-11T21:34:47.368307+0000","last_undegraded":"2024-03-11T21:34:47.368307+0000","last_fullsized":"2024-03-11T21:34:47.368307+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.813542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.15","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393085+0000","last_change":"2024-03-11T21:34:44.447613+0000","last_active":"2024-03-11T21:34:47.393085+0000","last_peered":"2024-03-11T21:34:47.393085+0000","last_clean":"2024-03-11T21:34:47.393085+0000","last_became_active":"2024-03-11T21:34:44.447247+0000","last_became_peered":"2024-03-11T21:34:44.447247+0000","last_unstale":"2024-03-11T21:34:47.393085+0000","last_undegraded":"2024-03-11T21:34:47.393085+0000","last_fullsized":"2024-03-11T21:34:47.393085+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:01.421146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.14","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391576+0000","last_change":"2024-03-11T21:34:44.443084+0000","last_active":"2024-03-11T21:34:47.391576+0000","last_peered":"2024-03-11T21:34:47.391576+0000","last_clean":"2024-03-11T21:34:47.391576+0000","last_became_active":"2024-03-11T21:34:44.442710+0000","last_became_peered":"2024-03-11T21:34:44.442710+0000","last_unstale":"2024-03-11T21:34:47.391576+0000","last_undegraded":"2024-03-11T21:34:47.391576+0000","last_fullsized":"2024-03-11T21:34:47.391576+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.235460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.13","version":"0'0","reported_seq":10,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.454689+0000","last_change":"2024-03-11T21:34:44.435890+0000","last_active":"2024-03-11T21:34:44.454689+0000","last_peered":"2024-03-11T21:34:44.454689+0000","last_clean":"2024-03-11T21:34:44.454689+0000","last_became_active":"2024-03-11T21:34:44.435083+0000","last_became_peered":"2024-03-11T21:34:44.435083+0000","last_unstale":"2024-03-11T21:34:44.454689+0000","last_undegraded":"2024-03-11T21:34:44.454689+0000","last_fullsized":"2024-03-11T21:34:44.454689+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.332535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.12","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356927+0000","last_change":"2024-03-11T21:34:44.430273+0000","last_active":"2024-03-11T21:34:47.356927+0000","last_peered":"2024-03-11T21:34:47.356927+0000","last_clean":"2024-03-11T21:34:47.356927+0000","last_became_active":"2024-03-11T21:34:44.429778+0000","last_became_peered":"2024-03-11T21:34:44.429778+0000","last_unstale":"2024-03-11T21:34:47.356927+0000","last_undegraded":"2024-03-11T21:34:47.356927+0000","last_fullsized":"2024-03-11T21:34:47.356927+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.11","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.374893+0000","last_change":"2024-03-11T21:34:44.444152+0000","last_active":"2024-03-11T21:34:47.374893+0000","last_peered":"2024-03-11T21:34:47.374893+0000","last_clean":"2024-03-11T21:34:47.374893+0000","last_became_active":"2024-03-11T21:34:44.443675+0000","last_became_peered":"2024-03-11T21:34:44.443675+0000","last_unstale":"2024-03-11T21:34:47.374893+0000","last_undegraded":"2024-03-11T21:34:47.374893+0000","last_fullsized":"2024-03-11T21:34:47.374893+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.783644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.10","version":"476'2","reported_seq":19,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356431+0000","last_change":"2024-03-11T21:34:44.441652+0000","last_active":"2024-03-11T21:34:47.356431+0000","last_peered":"2024-03-11T21:34:47.356431+0000","last_clean":"2024-03-11T21:34:47.356431+0000","last_became_active":"2024-03-11T21:34:44.440931+0000","last_became_peered":"2024-03-11T21:34:44.440931+0000","last_unstale":"2024-03-11T21:34:47.356431+0000","last_undegraded":"2024-03-11T21:34:47.356431+0000","last_fullsized":"2024-03-11T21:34:47.356431+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.893202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.f","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358262+0000","last_change":"2024-03-11T21:34:44.441155+0000","last_active":"2024-03-11T21:34:47.358262+0000","last_peered":"2024-03-11T21:34:47.358262+0000","last_clean":"2024-03-11T21:34:47.358262+0000","last_became_active":"2024-03-11T21:34:44.440255+0000","last_became_peered":"2024-03-11T21:34:44.440255+0000","last_unstale":"2024-03-11T21:34:47.358262+0000","last_undegraded":"2024-03-11T21:34:47.358262+0000","last_fullsized":"2024-03-11T21:34:47.358262+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.799860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.e","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385001+0000","last_change":"2024-03-11T21:34:44.438672+0000","last_active":"2024-03-11T21:34:47.385001+0000","last_peered":"2024-03-11T21:34:47.385001+0000","last_clean":"2024-03-11T21:34:47.385001+0000","last_became_active":"2024-03-11T21:34:44.438068+0000","last_became_peered":"2024-03-11T21:34:44.438068+0000","last_unstale":"2024-03-11T21:34:47.385001+0000","last_undegraded":"2024-03-11T21:34:47.385001+0000","last_fullsized":"2024-03-11T21:34:47.385001+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.422291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"287.d","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391427+0000","last_change":"2024-03-11T21:34:44.455859+0000","last_active":"2024-03-11T21:34:47.391427+0000","last_peered":"2024-03-11T21:34:47.391427+0000","last_clean":"2024-03-11T21:34:47.391427+0000","last_became_active":"2024-03-11T21:34:44.455539+0000","last_became_peered":"2024-03-11T21:34:44.455539+0000","last_unstale":"2024-03-11T21:34:47.391427+0000","last_undegraded":"2024-03-11T21:34:47.391427+0000","last_fullsized":"2024-03-11T21:34:47.391427+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.121411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.c","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367352+0000","last_change":"2024-03-11T21:34:44.439565+0000","last_active":"2024-03-11T21:34:47.367352+0000","last_peered":"2024-03-11T21:34:47.367352+0000","last_clean":"2024-03-11T21:34:47.367352+0000","last_became_active":"2024-03-11T21:34:44.438909+0000","last_became_peered":"2024-03-11T21:34:44.438909+0000","last_unstale":"2024-03-11T21:34:47.367352+0000","last_undegraded":"2024-03-11T21:34:47.367352+0000","last_fullsized":"2024-03-11T21:34:47.367352+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:10.187968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.b","version":"0'0","reported_seq":1,"reported_epoch":473,"state":"creating+peering","last_fresh":"2024-03-11T21:34:43.414048+0000","last_change":"2024-03-11T21:34:43.414048+0000","last_active":"2024-03-11T21:34:43.390268+0000","last_peered":"2024-03-11T21:34:43.390268+0000","last_clean":"2024-03-11T21:34:43.390268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:43.414048+0000","last_undegraded":"2024-03-11T21:34:43.414048+0000","last_fullsized":"2024-03-11T21:34:43.414048+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.a","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366522+0000","last_change":"2024-03-11T21:34:44.438449+0000","last_active":"2024-03-11T21:34:47.366522+0000","last_peered":"2024-03-11T21:34:47.366522+0000","last_clean":"2024-03-11T21:34:47.366522+0000","last_became_active":"2024-03-11T21:34:44.437827+0000","last_became_peered":"2024-03-11T21:34:44.437827+0000","last_unstale":"2024-03-11T21:34:47.366522+0000","last_undegraded":"2024-03-11T21:34:47.366522+0000","last_fullsized":"2024-03-11T21:34:47.366522+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.405554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.9","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366309+0000","last_change":"2024-03-11T21:34:44.443718+0000","last_active":"2024-03-11T21:34:47.366309+0000","last_peered":"2024-03-11T21:34:47.366309+0000","last_clean":"2024-03-11T21:34:47.366309+0000","last_became_active":"2024-03-11T21:34:44.443025+0000","last_became_peered":"2024-03-11T21:34:44.443025+0000","last_unstale":"2024-03-11T21:34:47.366309+0000","last_undegraded":"2024-03-11T21:34:47.366309+0000","last_fullsized":"2024-03-11T21:34:47.366309+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:55.361921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.8","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.374704+0000","last_change":"2024-03-11T21:34:44.430411+0000","last_active":"2024-03-11T21:34:47.374704+0000","last_peered":"2024-03-11T21:34:47.374704+0000","last_clean":"2024-03-11T21:34:47.374704+0000","last_became_active":"2024-03-11T21:34:44.429530+0000","last_became_peered":"2024-03-11T21:34:44.429530+0000","last_unstale":"2024-03-11T21:34:47.374704+0000","last_undegraded":"2024-03-11T21:34:47.374704+0000","last_fullsized":"2024-03-11T21:34:47.374704+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.837048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.7","version":"0'0","reported_seq":1,"reported_epoch":473,"state":"creating+peering","last_fresh":"2024-03-11T21:34:43.413204+0000","last_change":"2024-03-11T21:34:43.413204+0000","last_active":"2024-03-11T21:34:43.390268+0000","last_peered":"2024-03-11T21:34:43.390268+0000","last_clean":"2024-03-11T21:34:43.390268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:34:43.413204+0000","last_undegraded":"2024-03-11T21:34:43.413204+0000","last_fullsized":"2024-03-11T21:34:43.413204+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.6","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368364+0000","last_change":"2024-03-11T21:34:44.444673+0000","last_active":"2024-03-11T21:34:47.368364+0000","last_peered":"2024-03-11T21:34:47.368364+0000","last_clean":"2024-03-11T21:34:47.368364+0000","last_became_active":"2024-03-11T21:34:44.442788+0000","last_became_peered":"2024-03-11T21:34:44.442788+0000","last_unstale":"2024-03-11T21:34:47.368364+0000","last_undegraded":"2024-03-11T21:34:47.368364+0000","last_fullsized":"2024-03-11T21:34:47.368364+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.046083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.5","version":"0'0","reported_seq":10,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.454632+0000","last_change":"2024-03-11T21:34:44.438016+0000","last_active":"2024-03-11T21:34:44.454632+0000","last_peered":"2024-03-11T21:34:44.454632+0000","last_clean":"2024-03-11T21:34:44.454632+0000","last_became_active":"2024-03-11T21:34:44.437215+0000","last_became_peered":"2024-03-11T21:34:44.437215+0000","last_unstale":"2024-03-11T21:34:44.454632+0000","last_undegraded":"2024-03-11T21:34:44.454632+0000","last_fullsized":"2024-03-11T21:34:44.454632+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:49.960073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.4","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375536+0000","last_change":"2024-03-11T21:34:44.431713+0000","last_active":"2024-03-11T21:34:47.375536+0000","last_peered":"2024-03-11T21:34:47.375536+0000","last_clean":"2024-03-11T21:34:47.375536+0000","last_became_active":"2024-03-11T21:34:44.431191+0000","last_became_peered":"2024-03-11T21:34:44.431191+0000","last_unstale":"2024-03-11T21:34:47.375536+0000","last_undegraded":"2024-03-11T21:34:47.375536+0000","last_fullsized":"2024-03-11T21:34:47.375536+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.744204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.3","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384890+0000","last_change":"2024-03-11T21:34:44.437273+0000","last_active":"2024-03-11T21:34:47.384890+0000","last_peered":"2024-03-11T21:34:47.384890+0000","last_clean":"2024-03-11T21:34:47.384890+0000","last_became_active":"2024-03-11T21:34:44.436574+0000","last_became_peered":"2024-03-11T21:34:44.436574+0000","last_unstale":"2024-03-11T21:34:47.384890+0000","last_undegraded":"2024-03-11T21:34:47.384890+0000","last_fullsized":"2024-03-11T21:34:47.384890+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.659963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"287.2","version":"0'0","reported_seq":10,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.454595+0000","last_change":"2024-03-11T21:34:44.440013+0000","last_active":"2024-03-11T21:34:44.454595+0000","last_peered":"2024-03-11T21:34:44.454595+0000","last_clean":"2024-03-11T21:34:44.454595+0000","last_became_active":"2024-03-11T21:34:44.439626+0000","last_became_peered":"2024-03-11T21:34:44.439626+0000","last_unstale":"2024-03-11T21:34:44.454595+0000","last_undegraded":"2024-03-11T21:34:44.454595+0000","last_fullsized":"2024-03-11T21:34:44.454595+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.545574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.1","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392818+0000","last_change":"2024-03-11T21:34:44.453166+0000","last_active":"2024-03-11T21:34:47.392818+0000","last_peered":"2024-03-11T21:34:47.392818+0000","last_clean":"2024-03-11T21:34:47.392818+0000","last_became_active":"2024-03-11T21:34:44.452589+0000","last_became_peered":"2024-03-11T21:34:44.452589+0000","last_unstale":"2024-03-11T21:34:47.392818+0000","last_undegraded":"2024-03-11T21:34:47.392818+0000","last_fullsized":"2024-03-11T21:34:47.392818+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:06.018483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.0","version":"0'0","reported_seq":17,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384074+0000","last_change":"2024-03-11T21:34:44.443969+0000","last_active":"2024-03-11T21:34:47.384074+0000","last_peered":"2024-03-11T21:34:47.384074+0000","last_clean":"2024-03-11T21:34:47.384074+0000","last_became_active":"2024-03-11T21:34:44.443534+0000","last_became_peered":"2024-03-11T21:34:44.443534+0000","last_unstale":"2024-03-11T21:34:47.384074+0000","last_undegraded":"2024-03-11T21:34:47.384074+0000","last_fullsized":"2024-03-11T21:34:47.384074+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.780508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.9","version":"245'17","reported_seq":822,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390034+0000","last_change":"2024-03-11T21:33:39.123510+0000","last_active":"2024-03-11T21:34:47.390034+0000","last_peered":"2024-03-11T21:34:47.390034+0000","last_clean":"2024-03-11T21:34:47.390034+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:34:47.390034+0000","last_undegraded":"2024-03-11T21:34:47.390034+0000","last_fullsized":"2024-03-11T21:34:47.390034+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-03-11T21:33:39.123380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.123380+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001660573,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"461'24","reported_seq":809,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384285+0000","last_change":"2024-03-11T21:32:34.299549+0000","last_active":"2024-03-11T21:34:47.384285+0000","last_peered":"2024-03-11T21:34:47.384285+0000","last_clean":"2024-03-11T21:34:47.384285+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:34:47.384285+0000","last_undegraded":"2024-03-11T21:34:47.384285+0000","last_fullsized":"2024-03-11T21:34:47.384285+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'18","last_scrub_stamp":"2024-03-11T21:32:22.035738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.035738+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076326579999999996,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"464'40","reported_seq":823,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366915+0000","last_change":"2024-03-11T21:32:34.335857+0000","last_active":"2024-03-11T21:34:47.366915+0000","last_peered":"2024-03-11T21:34:47.366915+0000","last_clean":"2024-03-11T21:34:47.366915+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:34:47.366915+0000","last_undegraded":"2024-03-11T21:34:47.366915+0000","last_fullsized":"2024-03-11T21:34:47.366915+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024159609999999999,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"444'55","reported_seq":854,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357026+0000","last_change":"2024-03-11T21:33:20.623614+0000","last_active":"2024-03-11T21:34:47.357026+0000","last_peered":"2024-03-11T21:34:47.357026+0000","last_clean":"2024-03-11T21:34:47.357026+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:34:47.357026+0000","last_undegraded":"2024-03-11T21:34:47.357026+0000","last_fullsized":"2024-03-11T21:34:47.357026+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'52","last_scrub_stamp":"2024-03-11T21:33:20.623315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.623315+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0043833079999999998,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"412'52","reported_seq":849,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391644+0000","last_change":"2024-03-11T21:33:28.221239+0000","last_active":"2024-03-11T21:34:47.391644+0000","last_peered":"2024-03-11T21:34:47.391644+0000","last_clean":"2024-03-11T21:34:47.391644+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:34:47.391644+0000","last_undegraded":"2024-03-11T21:34:47.391644+0000","last_fullsized":"2024-03-11T21:34:47.391644+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'51","last_scrub_stamp":"2024-03-11T21:33:28.221082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.221082+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.994628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022023799999999999,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":816,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358129+0000","last_change":"2024-03-11T21:33:15.710053+0000","last_active":"2024-03-11T21:34:47.358129+0000","last_peered":"2024-03-11T21:34:47.358129+0000","last_clean":"2024-03-11T21:34:47.358129+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:34:47.358129+0000","last_undegraded":"2024-03-11T21:34:47.358129+0000","last_fullsized":"2024-03-11T21:34:47.358129+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'34","last_scrub_stamp":"2024-03-11T21:33:15.709878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.709878+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0036499739999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"444'184","reported_seq":945,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358539+0000","last_change":"2024-03-11T21:33:14.703400+0000","last_active":"2024-03-11T21:34:47.358539+0000","last_peered":"2024-03-11T21:34:47.358539+0000","last_clean":"2024-03-11T21:34:47.358539+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:34:47.358539+0000","last_undegraded":"2024-03-11T21:34:47.358539+0000","last_fullsized":"2024-03-11T21:34:47.358539+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'183","last_scrub_stamp":"2024-03-11T21:33:14.703284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.703284+0000","objects_scrubbed":13,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.029727594,"stat_sum":{"num_bytes":125,"num_objects":14,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":56,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":883,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385115+0000","last_change":"2024-03-11T21:33:27.586701+0000","last_active":"2024-03-11T21:34:47.385115+0000","last_peered":"2024-03-11T21:34:47.385115+0000","last_clean":"2024-03-11T21:34:47.385115+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:34:47.385115+0000","last_undegraded":"2024-03-11T21:34:47.385115+0000","last_fullsized":"2024-03-11T21:34:47.385115+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:33:27.586328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.586328+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:56.498917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.013258202,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":794,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406146+0000","last_change":"2024-03-11T21:33:08.891202+0000","last_active":"2024-03-11T21:34:43.406146+0000","last_peered":"2024-03-11T21:34:43.406146+0000","last_clean":"2024-03-11T21:34:43.406146+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:34:43.406146+0000","last_undegraded":"2024-03-11T21:34:43.406146+0000","last_fullsized":"2024-03-11T21:34:43.406146+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:33:08.891062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.891062+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0077514410000000004,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":832,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.404771+0000","last_change":"2024-03-11T21:33:14.964478+0000","last_active":"2024-03-11T21:34:43.404771+0000","last_peered":"2024-03-11T21:34:43.404771+0000","last_clean":"2024-03-11T21:34:43.404771+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:34:43.404771+0000","last_undegraded":"2024-03-11T21:34:43.404771+0000","last_fullsized":"2024-03-11T21:34:43.404771+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:33:14.964352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.964352+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076282429999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"412'120","reported_seq":885,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367074+0000","last_change":"2024-03-11T21:34:43.036701+0000","last_active":"2024-03-11T21:34:47.367074+0000","last_peered":"2024-03-11T21:34:47.367074+0000","last_clean":"2024-03-11T21:34:47.367074+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:34:47.367074+0000","last_undegraded":"2024-03-11T21:34:47.367074+0000","last_fullsized":"2024-03-11T21:34:47.367074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.183080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0026596910000000001,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":825,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357120+0000","last_change":"2024-03-11T21:33:17.671892+0000","last_active":"2024-03-11T21:34:47.357120+0000","last_peered":"2024-03-11T21:34:47.357120+0000","last_clean":"2024-03-11T21:34:47.357120+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:34:47.357120+0000","last_undegraded":"2024-03-11T21:34:47.357120+0000","last_fullsized":"2024-03-11T21:34:47.357120+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'21","last_scrub_stamp":"2024-03-11T21:33:17.671661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.671661+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0041327550000000001,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"458'3","reported_seq":800,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391899+0000","last_change":"2024-03-11T21:33:32.236109+0000","last_active":"2024-03-11T21:34:47.391899+0000","last_peered":"2024-03-11T21:34:47.391899+0000","last_clean":"2024-03-11T21:34:47.391899+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:34:47.391899+0000","last_undegraded":"2024-03-11T21:34:47.391899+0000","last_fullsized":"2024-03-11T21:34:47.391899+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.235772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.235772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002624487,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":811,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406641+0000","last_change":"2024-03-11T21:33:07.935033+0000","last_active":"2024-03-11T21:34:43.406641+0000","last_peered":"2024-03-11T21:34:43.406641+0000","last_clean":"2024-03-11T21:34:43.406641+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:34:43.406641+0000","last_undegraded":"2024-03-11T21:34:43.406641+0000","last_fullsized":"2024-03-11T21:34:43.406641+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:07.934653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.934653+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0080892789999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"412'41","reported_seq":816,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394868+0000","last_change":"2024-03-11T21:33:25.692636+0000","last_active":"2024-03-11T21:34:47.394868+0000","last_peered":"2024-03-11T21:34:47.394868+0000","last_clean":"2024-03-11T21:34:47.394868+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:34:47.394868+0000","last_undegraded":"2024-03-11T21:34:47.394868+0000","last_fullsized":"2024-03-11T21:34:47.394868+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'39","last_scrub_stamp":"2024-03-11T21:33:25.692508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.692508+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001055535,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":858,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394713+0000","last_change":"2024-03-11T21:33:23.654493+0000","last_active":"2024-03-11T21:34:47.394713+0000","last_peered":"2024-03-11T21:34:47.394713+0000","last_clean":"2024-03-11T21:34:47.394713+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:34:47.394713+0000","last_undegraded":"2024-03-11T21:34:47.394713+0000","last_fullsized":"2024-03-11T21:34:47.394713+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:33:23.654355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.654355+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001789856,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":791,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406064+0000","last_change":"2024-03-11T21:33:05.928221+0000","last_active":"2024-03-11T21:34:43.406064+0000","last_peered":"2024-03-11T21:34:43.406064+0000","last_clean":"2024-03-11T21:34:43.406064+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:34:43.406064+0000","last_undegraded":"2024-03-11T21:34:43.406064+0000","last_fullsized":"2024-03-11T21:34:43.406064+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:33:05.927971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.927971+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0078384049999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":863,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.404620+0000","last_change":"2024-03-11T21:33:10.952092+0000","last_active":"2024-03-11T21:34:43.404620+0000","last_peered":"2024-03-11T21:34:43.404620+0000","last_clean":"2024-03-11T21:34:43.404620+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:34:43.404620+0000","last_undegraded":"2024-03-11T21:34:43.404620+0000","last_fullsized":"2024-03-11T21:34:43.404620+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'107","last_scrub_stamp":"2024-03-11T21:33:10.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.951942+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067123959999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":681,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.404053+0000","last_change":"2024-03-11T21:33:17.964447+0000","last_active":"2024-03-11T21:34:44.404053+0000","last_peered":"2024-03-11T21:34:44.404053+0000","last_clean":"2024-03-11T21:34:44.404053+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:34:44.404053+0000","last_undegraded":"2024-03-11T21:34:44.404053+0000","last_fullsized":"2024-03-11T21:34:44.404053+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:33:17.964235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.964235+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.005553105,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":827,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.404501+0000","last_change":"2024-03-11T21:33:04.948889+0000","last_active":"2024-03-11T21:34:43.404501+0000","last_peered":"2024-03-11T21:34:43.404501+0000","last_clean":"2024-03-11T21:34:43.404501+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:34:43.404501+0000","last_undegraded":"2024-03-11T21:34:43.404501+0000","last_fullsized":"2024-03-11T21:34:43.404501+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:33:04.948734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.948734+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0071306190000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":878,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376174+0000","last_change":"2024-03-11T21:34:42.241731+0000","last_active":"2024-03-11T21:34:47.376174+0000","last_peered":"2024-03-11T21:34:47.376174+0000","last_clean":"2024-03-11T21:34:47.376174+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:34:47.376174+0000","last_undegraded":"2024-03-11T21:34:47.376174+0000","last_fullsized":"2024-03-11T21:34:47.376174+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:34:42.241601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.241601+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.236313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096827370000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"477'63","reported_seq":857,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.371653+0000","last_change":"2024-03-11T21:32:34.335870+0000","last_active":"2024-03-11T21:34:47.371653+0000","last_peered":"2024-03-11T21:34:47.371653+0000","last_clean":"2024-03-11T21:34:47.371653+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:34:47.371653+0000","last_undegraded":"2024-03-11T21:34:47.371653+0000","last_fullsized":"2024-03-11T21:34:47.371653+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023272950000000001,"stat_sum":{"num_bytes":66,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"444'37","reported_seq":700,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.405218+0000","last_change":"2024-03-11T21:33:14.938757+0000","last_active":"2024-03-11T21:34:44.405218+0000","last_peered":"2024-03-11T21:34:44.405218+0000","last_clean":"2024-03-11T21:34:44.405218+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:34:44.405218+0000","last_undegraded":"2024-03-11T21:34:44.405218+0000","last_fullsized":"2024-03-11T21:34:44.405218+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:14.938568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.938568+0000","objects_scrubbed":2,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011782635,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"361'209","reported_seq":947,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368457+0000","last_change":"2024-03-11T21:32:34.338193+0000","last_active":"2024-03-11T21:34:47.368457+0000","last_peered":"2024-03-11T21:34:47.368457+0000","last_clean":"2024-03-11T21:34:47.368457+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:34:47.368457+0000","last_undegraded":"2024-03-11T21:34:47.368457+0000","last_fullsized":"2024-03-11T21:34:47.368457+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829950000000001,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":808,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366971+0000","last_change":"2024-03-11T21:32:34.335400+0000","last_active":"2024-03-11T21:34:47.366971+0000","last_peered":"2024-03-11T21:34:47.366971+0000","last_clean":"2024-03-11T21:34:47.366971+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:34:47.366971+0000","last_undegraded":"2024-03-11T21:34:47.366971+0000","last_fullsized":"2024-03-11T21:34:47.366971+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024394040000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"464'17","reported_seq":860,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384103+0000","last_change":"2024-03-11T21:33:31.565426+0000","last_active":"2024-03-11T21:34:47.384103+0000","last_peered":"2024-03-11T21:34:47.384103+0000","last_clean":"2024-03-11T21:34:47.384103+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:34:47.384103+0000","last_undegraded":"2024-03-11T21:34:47.384103+0000","last_fullsized":"2024-03-11T21:34:47.384103+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'14","last_scrub_stamp":"2024-03-11T21:33:31.565199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.565199+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0069698629999999998,"stat_sum":{"num_bytes":4194330,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":865,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385083+0000","last_change":"2024-03-11T21:34:32.811969+0000","last_active":"2024-03-11T21:34:47.385083+0000","last_peered":"2024-03-11T21:34:47.385083+0000","last_clean":"2024-03-11T21:34:47.385083+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:34:47.385083+0000","last_undegraded":"2024-03-11T21:34:47.385083+0000","last_fullsized":"2024-03-11T21:34:47.385083+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'87","last_scrub_stamp":"2024-03-11T21:34:32.811720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.811720+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.300363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018288136999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366765+0000","last_change":"2024-03-11T21:32:34.332276+0000","last_active":"2024-03-11T21:34:47.366765+0000","last_peered":"2024-03-11T21:34:47.366765+0000","last_clean":"2024-03-11T21:34:47.366765+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:34:47.366765+0000","last_undegraded":"2024-03-11T21:34:47.366765+0000","last_fullsized":"2024-03-11T21:34:47.366765+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016228219999999999,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":862,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391634+0000","last_change":"2024-03-11T21:33:29.221706+0000","last_active":"2024-03-11T21:34:47.391634+0000","last_peered":"2024-03-11T21:34:47.391634+0000","last_clean":"2024-03-11T21:34:47.391634+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:34:47.391634+0000","last_undegraded":"2024-03-11T21:34:47.391634+0000","last_fullsized":"2024-03-11T21:34:47.391634+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:33:29.221527+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:31:02.819204+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.221527+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:51.490446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029485240000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":842,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357720+0000","last_change":"2024-03-11T21:33:25.682885+0000","last_active":"2024-03-11T21:34:47.357720+0000","last_peered":"2024-03-11T21:34:47.357720+0000","last_clean":"2024-03-11T21:34:47.357720+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:34:47.357720+0000","last_undegraded":"2024-03-11T21:34:47.357720+0000","last_fullsized":"2024-03-11T21:34:47.357720+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'51","last_scrub_stamp":"2024-03-11T21:33:25.682768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.682768+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001496713,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"467'56","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392338+0000","last_change":"2024-03-11T21:33:42.167828+0000","last_active":"2024-03-11T21:34:47.392338+0000","last_peered":"2024-03-11T21:34:47.392338+0000","last_clean":"2024-03-11T21:34:47.392338+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:34:47.392338+0000","last_undegraded":"2024-03-11T21:34:47.392338+0000","last_fullsized":"2024-03-11T21:34:47.392338+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'53","last_scrub_stamp":"2024-03-11T21:33:42.167697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.167697+0000","objects_scrubbed":5,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:56.393818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001987904,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"467'6","reported_seq":794,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368498+0000","last_change":"2024-03-11T21:32:34.338218+0000","last_active":"2024-03-11T21:34:47.368498+0000","last_peered":"2024-03-11T21:34:47.368498+0000","last_clean":"2024-03-11T21:34:47.368498+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:34:47.368498+0000","last_undegraded":"2024-03-11T21:34:47.368498+0000","last_fullsized":"2024-03-11T21:34:47.368498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.236818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011248230000000001,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":690,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392103+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:34:47.392103+0000","last_peered":"2024-03-11T21:34:47.392103+0000","last_clean":"2024-03-11T21:34:47.392103+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:34:47.392103+0000","last_undegraded":"2024-03-11T21:34:47.392103+0000","last_fullsized":"2024-03-11T21:34:47.392103+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":693,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.374986+0000","last_change":"2024-03-11T21:33:28.579904+0000","last_active":"2024-03-11T21:34:47.374986+0000","last_peered":"2024-03-11T21:34:47.374986+0000","last_clean":"2024-03-11T21:34:47.374986+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:34:47.374986+0000","last_undegraded":"2024-03-11T21:34:47.374986+0000","last_fullsized":"2024-03-11T21:34:47.374986+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:33:28.577678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":691,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384167+0000","last_change":"2024-03-11T21:33:25.553124+0000","last_active":"2024-03-11T21:34:47.384167+0000","last_peered":"2024-03-11T21:34:47.384167+0000","last_clean":"2024-03-11T21:34:47.384167+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:34:47.384167+0000","last_undegraded":"2024-03-11T21:34:47.384167+0000","last_fullsized":"2024-03-11T21:34:47.384167+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.552965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.552965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.705964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392712+0000","last_change":"2024-03-11T21:33:13.735432+0000","last_active":"2024-03-11T21:34:47.392712+0000","last_peered":"2024-03-11T21:34:47.392712+0000","last_clean":"2024-03-11T21:34:47.392712+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:34:47.392712+0000","last_undegraded":"2024-03-11T21:34:47.392712+0000","last_fullsized":"2024-03-11T21:34:47.392712+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.735317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.735317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":587,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.412087+0000","last_change":"2024-03-11T21:33:34.103121+0000","last_active":"2024-03-11T21:34:44.412087+0000","last_peered":"2024-03-11T21:34:44.412087+0000","last_clean":"2024-03-11T21:34:44.412087+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:34:44.412087+0000","last_undegraded":"2024-03-11T21:34:44.412087+0000","last_fullsized":"2024-03-11T21:34:44.412087+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:34.103006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:34.103006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.132610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":693,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376503+0000","last_change":"2024-03-11T21:33:40.436006+0000","last_active":"2024-03-11T21:34:47.376503+0000","last_peered":"2024-03-11T21:34:47.376503+0000","last_clean":"2024-03-11T21:34:47.376503+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:34:47.376503+0000","last_undegraded":"2024-03-11T21:34:47.376503+0000","last_fullsized":"2024-03-11T21:34:47.376503+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.435648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.435648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:05.273758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":693,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376282+0000","last_change":"2024-03-11T21:33:25.576354+0000","last_active":"2024-03-11T21:34:47.376282+0000","last_peered":"2024-03-11T21:34:47.376282+0000","last_clean":"2024-03-11T21:34:47.376282+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:34:47.376282+0000","last_undegraded":"2024-03-11T21:34:47.376282+0000","last_fullsized":"2024-03-11T21:34:47.376282+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.576253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:59.448405+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.576253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":703,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392419+0000","last_change":"2024-03-11T21:34:32.354266+0000","last_active":"2024-03-11T21:34:47.392419+0000","last_peered":"2024-03-11T21:34:47.392419+0000","last_clean":"2024-03-11T21:34:47.392419+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:34:47.392419+0000","last_undegraded":"2024-03-11T21:34:47.392419+0000","last_fullsized":"2024-03-11T21:34:47.392419+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.354023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.354023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.133336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":695,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356897+0000","last_change":"2024-03-11T21:34:35.918096+0000","last_active":"2024-03-11T21:34:47.356897+0000","last_peered":"2024-03-11T21:34:47.356897+0000","last_clean":"2024-03-11T21:34:47.356897+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:34:47.356897+0000","last_undegraded":"2024-03-11T21:34:47.356897+0000","last_fullsized":"2024-03-11T21:34:47.356897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.917835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.917835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.696652+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393030+0000","last_change":"2024-03-11T21:31:01.656496+0000","last_active":"2024-03-11T21:34:47.393030+0000","last_peered":"2024-03-11T21:34:47.393030+0000","last_clean":"2024-03-11T21:34:47.393030+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:34:47.393030+0000","last_undegraded":"2024-03-11T21:34:47.393030+0000","last_fullsized":"2024-03-11T21:34:47.393030+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.656422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.656422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":686,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.404897+0000","last_change":"2024-03-11T21:33:41.974005+0000","last_active":"2024-03-11T21:34:43.404897+0000","last_peered":"2024-03-11T21:34:43.404897+0000","last_clean":"2024-03-11T21:34:43.404897+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:34:43.404897+0000","last_undegraded":"2024-03-11T21:34:43.404897+0000","last_fullsized":"2024-03-11T21:34:43.404897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.973907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:49.188951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":684,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367998+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:34:47.367998+0000","last_peered":"2024-03-11T21:34:47.367998+0000","last_clean":"2024-03-11T21:34:47.367998+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:34:47.367998+0000","last_undegraded":"2024-03-11T21:34:47.367998+0000","last_fullsized":"2024-03-11T21:34:47.367998+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":686,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406556+0000","last_change":"2024-03-11T21:34:40.037312+0000","last_active":"2024-03-11T21:34:43.406556+0000","last_peered":"2024-03-11T21:34:43.406556+0000","last_clean":"2024-03-11T21:34:43.406556+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:34:43.406556+0000","last_undegraded":"2024-03-11T21:34:43.406556+0000","last_fullsized":"2024-03-11T21:34:43.406556+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.037161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.037161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.123966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":687,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358724+0000","last_change":"2024-03-11T21:33:10.604343+0000","last_active":"2024-03-11T21:34:47.358724+0000","last_peered":"2024-03-11T21:34:47.358724+0000","last_clean":"2024-03-11T21:34:47.358724+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:34:47.358724+0000","last_undegraded":"2024-03-11T21:34:47.358724+0000","last_fullsized":"2024-03-11T21:34:47.358724+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.604160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.604160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":695,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392617+0000","last_change":"2024-03-11T21:33:27.227939+0000","last_active":"2024-03-11T21:34:47.392617+0000","last_peered":"2024-03-11T21:34:47.392617+0000","last_clean":"2024-03-11T21:34:47.392617+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:34:47.392617+0000","last_undegraded":"2024-03-11T21:34:47.392617+0000","last_fullsized":"2024-03-11T21:34:47.392617+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:27.227845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.227845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":83,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.404941+0000","last_change":"2024-03-11T21:33:59.394929+0000","last_active":"2024-03-11T21:34:44.404941+0000","last_peered":"2024-03-11T21:34:44.404941+0000","last_clean":"2024-03-11T21:34:44.404941+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:34:44.404941+0000","last_undegraded":"2024-03-11T21:34:44.404941+0000","last_fullsized":"2024-03-11T21:34:44.404941+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.603946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.603946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":693,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.374799+0000","last_change":"2024-03-11T21:33:15.600135+0000","last_active":"2024-03-11T21:34:47.374799+0000","last_peered":"2024-03-11T21:34:47.374799+0000","last_clean":"2024-03-11T21:34:47.374799+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:34:47.374799+0000","last_undegraded":"2024-03-11T21:34:47.374799+0000","last_fullsized":"2024-03-11T21:34:47.374799+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:15.600023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.600023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":690,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366623+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:34:47.366623+0000","last_peered":"2024-03-11T21:34:47.366623+0000","last_clean":"2024-03-11T21:34:47.366623+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:34:47.366623+0000","last_undegraded":"2024-03-11T21:34:47.366623+0000","last_fullsized":"2024-03-11T21:34:47.366623+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394281+0000","last_change":"2024-03-11T21:33:12.695054+0000","last_active":"2024-03-11T21:34:47.394281+0000","last_peered":"2024-03-11T21:34:47.394281+0000","last_clean":"2024-03-11T21:34:47.394281+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:34:47.394281+0000","last_undegraded":"2024-03-11T21:34:47.394281+0000","last_fullsized":"2024-03-11T21:34:47.394281+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.694966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.694966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":687,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358341+0000","last_change":"2024-03-11T21:32:14.693680+0000","last_active":"2024-03-11T21:34:47.358341+0000","last_peered":"2024-03-11T21:34:47.358341+0000","last_clean":"2024-03-11T21:34:47.358341+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:34:47.358341+0000","last_undegraded":"2024-03-11T21:34:47.358341+0000","last_fullsized":"2024-03-11T21:34:47.358341+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.693586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.693586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":555,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.411273+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:34:44.411273+0000","last_peered":"2024-03-11T21:34:44.411273+0000","last_clean":"2024-03-11T21:34:44.411273+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:34:44.411273+0000","last_undegraded":"2024-03-11T21:34:44.411273+0000","last_fullsized":"2024-03-11T21:34:44.411273+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394155+0000","last_change":"2024-03-11T21:32:07.649024+0000","last_active":"2024-03-11T21:34:47.394155+0000","last_peered":"2024-03-11T21:34:47.394155+0000","last_clean":"2024-03-11T21:34:47.394155+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:34:47.394155+0000","last_undegraded":"2024-03-11T21:34:47.394155+0000","last_fullsized":"2024-03-11T21:34:47.394155+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.648800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.648800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":681,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394805+0000","last_change":"2024-03-11T21:32:15.722549+0000","last_active":"2024-03-11T21:34:47.394805+0000","last_peered":"2024-03-11T21:34:47.394805+0000","last_clean":"2024-03-11T21:34:47.394805+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:34:47.394805+0000","last_undegraded":"2024-03-11T21:34:47.394805+0000","last_fullsized":"2024-03-11T21:34:47.394805+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:15.722443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.722443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":678,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407440+0000","last_change":"2024-03-11T21:32:14.976068+0000","last_active":"2024-03-11T21:34:43.407440+0000","last_peered":"2024-03-11T21:34:43.407440+0000","last_clean":"2024-03-11T21:34:43.407440+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:34:43.407440+0000","last_undegraded":"2024-03-11T21:34:43.407440+0000","last_fullsized":"2024-03-11T21:34:43.407440+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.975966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.975966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":691,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384784+0000","last_change":"2024-03-11T21:33:41.560584+0000","last_active":"2024-03-11T21:34:47.384784+0000","last_peered":"2024-03-11T21:34:47.384784+0000","last_clean":"2024-03-11T21:34:47.384784+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:34:47.384784+0000","last_undegraded":"2024-03-11T21:34:47.384784+0000","last_fullsized":"2024-03-11T21:34:47.384784+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.560389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.560389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":684,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366552+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:34:47.366552+0000","last_peered":"2024-03-11T21:34:47.366552+0000","last_clean":"2024-03-11T21:34:47.366552+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:34:47.366552+0000","last_undegraded":"2024-03-11T21:34:47.366552+0000","last_fullsized":"2024-03-11T21:34:47.366552+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.532268+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":91,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.412109+0000","last_change":"2024-03-11T21:34:39.164537+0000","last_active":"2024-03-11T21:34:44.412109+0000","last_peered":"2024-03-11T21:34:44.412109+0000","last_clean":"2024-03-11T21:34:44.412109+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:34:44.412109+0000","last_undegraded":"2024-03-11T21:34:44.412109+0000","last_fullsized":"2024-03-11T21:34:44.412109+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.164421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.164421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.867286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":687,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358224+0000","last_change":"2024-03-11T21:33:08.584273+0000","last_active":"2024-03-11T21:34:47.358224+0000","last_peered":"2024-03-11T21:34:47.358224+0000","last_clean":"2024-03-11T21:34:47.358224+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:34:47.358224+0000","last_undegraded":"2024-03-11T21:34:47.358224+0000","last_fullsized":"2024-03-11T21:34:47.358224+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.584163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.584163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":686,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406713+0000","last_change":"2024-03-11T21:34:36.108350+0000","last_active":"2024-03-11T21:34:43.406713+0000","last_peered":"2024-03-11T21:34:43.406713+0000","last_clean":"2024-03-11T21:34:43.406713+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:34:43.406713+0000","last_undegraded":"2024-03-11T21:34:43.406713+0000","last_fullsized":"2024-03-11T21:34:43.406713+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.108246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.108246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.604525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":684,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367993+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:34:47.367993+0000","last_peered":"2024-03-11T21:34:47.367993+0000","last_clean":"2024-03-11T21:34:47.367993+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:34:47.367993+0000","last_undegraded":"2024-03-11T21:34:47.367993+0000","last_fullsized":"2024-03-11T21:34:47.367993+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":695,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392149+0000","last_change":"2024-03-11T21:33:10.163119+0000","last_active":"2024-03-11T21:34:47.392149+0000","last_peered":"2024-03-11T21:34:47.392149+0000","last_clean":"2024-03-11T21:34:47.392149+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:34:47.392149+0000","last_undegraded":"2024-03-11T21:34:47.392149+0000","last_fullsized":"2024-03-11T21:34:47.392149+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.162829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.162829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":555,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.405470+0000","last_change":"2024-03-11T21:32:28.978945+0000","last_active":"2024-03-11T21:34:44.405470+0000","last_peered":"2024-03-11T21:34:44.405470+0000","last_clean":"2024-03-11T21:34:44.405470+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:34:44.405470+0000","last_undegraded":"2024-03-11T21:34:44.405470+0000","last_fullsized":"2024-03-11T21:34:44.405470+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:28.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":692,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406317+0000","last_change":"2024-03-11T21:33:31.964693+0000","last_active":"2024-03-11T21:34:43.406317+0000","last_peered":"2024-03-11T21:34:43.406317+0000","last_clean":"2024-03-11T21:34:43.406317+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:34:43.406317+0000","last_undegraded":"2024-03-11T21:34:43.406317+0000","last_fullsized":"2024-03-11T21:34:43.406317+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:31.964602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.964602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.956887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":987,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367499+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:34:47.367499+0000","last_peered":"2024-03-11T21:34:47.367499+0000","last_clean":"2024-03-11T21:34:47.367499+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:34:47.367499+0000","last_undegraded":"2024-03-11T21:34:47.367499+0000","last_fullsized":"2024-03-11T21:34:47.367499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":883,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.369899+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:34:47.369899+0000","last_peered":"2024-03-11T21:34:47.369899+0000","last_clean":"2024-03-11T21:34:47.369899+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:34:47.369899+0000","last_undegraded":"2024-03-11T21:34:47.369899+0000","last_fullsized":"2024-03-11T21:34:47.369899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392454+0000","last_change":"2024-03-11T21:33:25.276824+0000","last_active":"2024-03-11T21:34:47.392454+0000","last_peered":"2024-03-11T21:34:47.392454+0000","last_clean":"2024-03-11T21:34:47.392454+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:34:47.392454+0000","last_undegraded":"2024-03-11T21:34:47.392454+0000","last_fullsized":"2024-03-11T21:34:47.392454+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.276731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.276731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.729396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384557+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:34:47.384557+0000","last_peered":"2024-03-11T21:34:47.384557+0000","last_clean":"2024-03-11T21:34:47.384557+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:34:47.384557+0000","last_undegraded":"2024-03-11T21:34:47.384557+0000","last_fullsized":"2024-03-11T21:34:47.384557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":706,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386337+0000","last_change":"2024-03-11T21:34:38.848849+0000","last_active":"2024-03-11T21:34:47.386337+0000","last_peered":"2024-03-11T21:34:47.386337+0000","last_clean":"2024-03-11T21:34:47.386337+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:34:47.386337+0000","last_undegraded":"2024-03-11T21:34:47.386337+0000","last_fullsized":"2024-03-11T21:34:47.386337+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:38.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:40.304698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":878,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385086+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:34:47.385086+0000","last_peered":"2024-03-11T21:34:47.385086+0000","last_clean":"2024-03-11T21:34:47.385086+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:34:47.385086+0000","last_undegraded":"2024-03-11T21:34:47.385086+0000","last_fullsized":"2024-03-11T21:34:47.385086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":737,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.409353+0000","last_change":"2024-03-11T21:32:06.996760+0000","last_active":"2024-03-11T21:34:44.409353+0000","last_peered":"2024-03-11T21:34:44.409353+0000","last_clean":"2024-03-11T21:34:44.409353+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:34:44.409353+0000","last_undegraded":"2024-03-11T21:34:44.409353+0000","last_fullsized":"2024-03-11T21:34:44.409353+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.996518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.996518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390459+0000","last_change":"2024-03-11T21:34:40.424576+0000","last_active":"2024-03-11T21:34:47.390459+0000","last_peered":"2024-03-11T21:34:47.390459+0000","last_clean":"2024-03-11T21:34:47.390459+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:34:47.390459+0000","last_undegraded":"2024-03-11T21:34:47.390459+0000","last_fullsized":"2024-03-11T21:34:47.390459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.424446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.424446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.552331+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":835,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.409871+0000","last_change":"2024-03-11T21:32:25.048822+0000","last_active":"2024-03-11T21:34:43.409871+0000","last_peered":"2024-03-11T21:34:43.409871+0000","last_clean":"2024-03-11T21:34:43.409871+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:34:43.409871+0000","last_undegraded":"2024-03-11T21:34:43.409871+0000","last_fullsized":"2024-03-11T21:34:43.409871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.048696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.048696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":697,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385684+0000","last_change":"2024-03-11T21:33:22.548838+0000","last_active":"2024-03-11T21:34:47.385684+0000","last_peered":"2024-03-11T21:34:47.385684+0000","last_clean":"2024-03-11T21:34:47.385684+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:34:47.385684+0000","last_undegraded":"2024-03-11T21:34:47.385684+0000","last_fullsized":"2024-03-11T21:34:47.385684+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.548708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.548708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":905,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368460+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:34:47.368460+0000","last_peered":"2024-03-11T21:34:47.368460+0000","last_clean":"2024-03-11T21:34:47.368460+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:34:47.368460+0000","last_undegraded":"2024-03-11T21:34:47.368460+0000","last_fullsized":"2024-03-11T21:34:47.368460+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":871,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391623+0000","last_change":"2024-03-11T21:34:46.463590+0000","last_active":"2024-03-11T21:34:47.391623+0000","last_peered":"2024-03-11T21:34:47.391623+0000","last_clean":"2024-03-11T21:34:47.391623+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:34:47.391623+0000","last_undegraded":"2024-03-11T21:34:47.391623+0000","last_fullsized":"2024-03-11T21:34:47.391623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:46.463490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:34:46.463490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.211498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391031+0000","last_change":"2024-03-11T21:34:43.484238+0000","last_active":"2024-03-11T21:34:47.391031+0000","last_peered":"2024-03-11T21:34:47.391031+0000","last_clean":"2024-03-11T21:34:47.391031+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:34:47.391031+0000","last_undegraded":"2024-03-11T21:34:47.391031+0000","last_fullsized":"2024-03-11T21:34:47.391031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.484121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.484121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:07.084724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":832,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366439+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:34:47.366439+0000","last_peered":"2024-03-11T21:34:47.366439+0000","last_clean":"2024-03-11T21:34:47.366439+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:34:47.366439+0000","last_undegraded":"2024-03-11T21:34:47.366439+0000","last_fullsized":"2024-03-11T21:34:47.366439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":564,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.404878+0000","last_change":"2024-03-11T21:33:39.131584+0000","last_active":"2024-03-11T21:34:44.404878+0000","last_peered":"2024-03-11T21:34:44.404878+0000","last_clean":"2024-03-11T21:34:44.404878+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:34:44.404878+0000","last_undegraded":"2024-03-11T21:34:44.404878+0000","last_fullsized":"2024-03-11T21:34:44.404878+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.131510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.131510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.433688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":930,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375577+0000","last_change":"2024-03-11T21:33:46.396054+0000","last_active":"2024-03-11T21:34:47.375577+0000","last_peered":"2024-03-11T21:34:47.375577+0000","last_clean":"2024-03-11T21:34:47.375577+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:34:47.375577+0000","last_undegraded":"2024-03-11T21:34:47.375577+0000","last_fullsized":"2024-03-11T21:34:47.375577+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:33:46.395671+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.395671+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.312098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367648+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:34:47.367648+0000","last_peered":"2024-03-11T21:34:47.367648+0000","last_clean":"2024-03-11T21:34:47.367648+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:34:47.367648+0000","last_undegraded":"2024-03-11T21:34:47.367648+0000","last_fullsized":"2024-03-11T21:34:47.367648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":743,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.410786+0000","last_change":"2024-03-11T21:33:13.942454+0000","last_active":"2024-03-11T21:34:44.410786+0000","last_peered":"2024-03-11T21:34:44.410786+0000","last_clean":"2024-03-11T21:34:44.410786+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:34:44.410786+0000","last_undegraded":"2024-03-11T21:34:44.410786+0000","last_fullsized":"2024-03-11T21:34:44.410786+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.942158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.942158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":859,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392943+0000","last_change":"2024-03-11T21:34:33.339535+0000","last_active":"2024-03-11T21:34:47.392943+0000","last_peered":"2024-03-11T21:34:47.392943+0000","last_clean":"2024-03-11T21:34:47.392943+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:34:47.392943+0000","last_undegraded":"2024-03-11T21:34:47.392943+0000","last_fullsized":"2024-03-11T21:34:47.392943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.339234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.339234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:00.073588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":704,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393482+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:34:47.393482+0000","last_peered":"2024-03-11T21:34:47.393482+0000","last_clean":"2024-03-11T21:34:47.393482+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:34:47.393482+0000","last_undegraded":"2024-03-11T21:34:47.393482+0000","last_fullsized":"2024-03-11T21:34:47.393482+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":873,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.396723+0000","last_change":"2024-03-11T21:32:28.904947+0000","last_active":"2024-03-11T21:34:47.396723+0000","last_peered":"2024-03-11T21:34:47.396723+0000","last_clean":"2024-03-11T21:34:47.396723+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:34:47.396723+0000","last_undegraded":"2024-03-11T21:34:47.396723+0000","last_fullsized":"2024-03-11T21:34:47.396723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:32:28.904821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:28.904821+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":711,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.411823+0000","last_change":"2024-03-11T21:34:41.173845+0000","last_active":"2024-03-11T21:34:44.411823+0000","last_peered":"2024-03-11T21:34:44.411823+0000","last_clean":"2024-03-11T21:34:44.411823+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:34:44.411823+0000","last_undegraded":"2024-03-11T21:34:44.411823+0000","last_fullsized":"2024-03-11T21:34:44.411823+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:41.416478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1015,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392147+0000","last_change":"2024-03-11T21:33:41.205378+0000","last_active":"2024-03-11T21:34:47.392147+0000","last_peered":"2024-03-11T21:34:47.392147+0000","last_clean":"2024-03-11T21:34:47.392147+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:34:47.392147+0000","last_undegraded":"2024-03-11T21:34:47.392147+0000","last_fullsized":"2024-03-11T21:34:47.392147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:33:41.205243+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.205243+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:07.172685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358757+0000","last_change":"2024-03-11T21:32:08.744653+0000","last_active":"2024-03-11T21:34:47.358757+0000","last_peered":"2024-03-11T21:34:47.358757+0000","last_clean":"2024-03-11T21:34:47.358757+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:34:47.358757+0000","last_undegraded":"2024-03-11T21:34:47.358757+0000","last_fullsized":"2024-03-11T21:34:47.358757+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.744422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.744422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":707,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395144+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:34:47.395144+0000","last_peered":"2024-03-11T21:34:47.395144+0000","last_clean":"2024-03-11T21:34:47.395144+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:34:47.395144+0000","last_undegraded":"2024-03-11T21:34:47.395144+0000","last_fullsized":"2024-03-11T21:34:47.395144+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.099107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":900,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392955+0000","last_change":"2024-03-11T21:33:20.697629+0000","last_active":"2024-03-11T21:34:47.392955+0000","last_peered":"2024-03-11T21:34:47.392955+0000","last_clean":"2024-03-11T21:34:47.392955+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:34:47.392955+0000","last_undegraded":"2024-03-11T21:34:47.392955+0000","last_fullsized":"2024-03-11T21:34:47.392955+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:20.697247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.697247+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":731,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.413968+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:34:44.413968+0000","last_peered":"2024-03-11T21:34:44.413968+0000","last_clean":"2024-03-11T21:34:44.413968+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:34:44.413968+0000","last_undegraded":"2024-03-11T21:34:44.413968+0000","last_fullsized":"2024-03-11T21:34:44.413968+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":998,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358521+0000","last_change":"2024-03-11T21:33:12.674157+0000","last_active":"2024-03-11T21:34:47.358521+0000","last_peered":"2024-03-11T21:34:47.358521+0000","last_clean":"2024-03-11T21:34:47.358521+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:34:47.358521+0000","last_undegraded":"2024-03-11T21:34:47.358521+0000","last_fullsized":"2024-03-11T21:34:47.358521+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:12.673934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.673934+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":857,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385922+0000","last_change":"2024-03-11T21:34:42.857662+0000","last_active":"2024-03-11T21:34:47.385922+0000","last_peered":"2024-03-11T21:34:47.385922+0000","last_clean":"2024-03-11T21:34:47.385922+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:34:47.385922+0000","last_undegraded":"2024-03-11T21:34:47.385922+0000","last_fullsized":"2024-03-11T21:34:47.385922+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.857448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.857448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.706469+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367334+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:34:47.367334+0000","last_peered":"2024-03-11T21:34:47.367334+0000","last_clean":"2024-03-11T21:34:47.367334+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:34:47.367334+0000","last_undegraded":"2024-03-11T21:34:47.367334+0000","last_fullsized":"2024-03-11T21:34:47.367334+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":888,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356917+0000","last_change":"2024-03-11T21:33:35.635520+0000","last_active":"2024-03-11T21:34:47.356917+0000","last_peered":"2024-03-11T21:34:47.356917+0000","last_clean":"2024-03-11T21:34:47.356917+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:34:47.356917+0000","last_undegraded":"2024-03-11T21:34:47.356917+0000","last_fullsized":"2024-03-11T21:34:47.356917+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.635256+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":861,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366662+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:34:47.366662+0000","last_peered":"2024-03-11T21:34:47.366662+0000","last_clean":"2024-03-11T21:34:47.366662+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:34:47.366662+0000","last_undegraded":"2024-03-11T21:34:47.366662+0000","last_fullsized":"2024-03-11T21:34:47.366662+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.709131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":844,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386217+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:34:47.386217+0000","last_peered":"2024-03-11T21:34:47.386217+0000","last_clean":"2024-03-11T21:34:47.386217+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:34:47.386217+0000","last_undegraded":"2024-03-11T21:34:47.386217+0000","last_fullsized":"2024-03-11T21:34:47.386217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":832,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393547+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:34:47.393547+0000","last_peered":"2024-03-11T21:34:47.393547+0000","last_clean":"2024-03-11T21:34:47.393547+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:34:47.393547+0000","last_undegraded":"2024-03-11T21:34:47.393547+0000","last_fullsized":"2024-03-11T21:34:47.393547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":690,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393676+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:34:47.393676+0000","last_peered":"2024-03-11T21:34:47.393676+0000","last_clean":"2024-03-11T21:34:47.393676+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:34:47.393676+0000","last_undegraded":"2024-03-11T21:34:47.393676+0000","last_fullsized":"2024-03-11T21:34:47.393676+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1005,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393442+0000","last_change":"2024-03-11T21:33:10.709056+0000","last_active":"2024-03-11T21:34:47.393442+0000","last_peered":"2024-03-11T21:34:47.393442+0000","last_clean":"2024-03-11T21:34:47.393442+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:34:47.393442+0000","last_undegraded":"2024-03-11T21:34:47.393442+0000","last_fullsized":"2024-03-11T21:34:47.393442+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:33:10.708723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.708723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":864,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357048+0000","last_change":"2024-03-11T21:33:37.698300+0000","last_active":"2024-03-11T21:34:47.357048+0000","last_peered":"2024-03-11T21:34:47.357048+0000","last_clean":"2024-03-11T21:34:47.357048+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:34:47.357048+0000","last_undegraded":"2024-03-11T21:34:47.357048+0000","last_fullsized":"2024-03-11T21:34:47.357048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:37.698095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.698095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.205420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":888,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407370+0000","last_change":"2024-03-11T21:31:58.977677+0000","last_active":"2024-03-11T21:34:43.407370+0000","last_peered":"2024-03-11T21:34:43.407370+0000","last_clean":"2024-03-11T21:34:43.407370+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:34:43.407370+0000","last_undegraded":"2024-03-11T21:34:43.407370+0000","last_fullsized":"2024-03-11T21:34:43.407370+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:31:58.977541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.977541+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":870,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376663+0000","last_change":"2024-03-11T21:33:04.660322+0000","last_active":"2024-03-11T21:34:47.376663+0000","last_peered":"2024-03-11T21:34:47.376663+0000","last_clean":"2024-03-11T21:34:47.376663+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:34:47.376663+0000","last_undegraded":"2024-03-11T21:34:47.376663+0000","last_fullsized":"2024-03-11T21:34:47.376663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:33:04.660206+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:27:12.893784+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.660206+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":842,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367396+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:34:47.367396+0000","last_peered":"2024-03-11T21:34:47.367396+0000","last_clean":"2024-03-11T21:34:47.367396+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:34:47.367396+0000","last_undegraded":"2024-03-11T21:34:47.367396+0000","last_fullsized":"2024-03-11T21:34:47.367396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368159+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:34:47.368159+0000","last_peered":"2024-03-11T21:34:47.368159+0000","last_clean":"2024-03-11T21:34:47.368159+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:34:47.368159+0000","last_undegraded":"2024-03-11T21:34:47.368159+0000","last_fullsized":"2024-03-11T21:34:47.368159+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":855,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356983+0000","last_change":"2024-03-11T21:34:43.908469+0000","last_active":"2024-03-11T21:34:47.356983+0000","last_peered":"2024-03-11T21:34:47.356983+0000","last_clean":"2024-03-11T21:34:47.356983+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:34:47.356983+0000","last_undegraded":"2024-03-11T21:34:47.356983+0000","last_fullsized":"2024-03-11T21:34:47.356983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.908344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.908344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:06.367719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1540,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393307+0000","last_change":"2024-03-11T21:33:40.550313+0000","last_active":"2024-03-11T21:34:47.393307+0000","last_peered":"2024-03-11T21:34:47.393307+0000","last_clean":"2024-03-11T21:34:47.393307+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:34:47.393307+0000","last_undegraded":"2024-03-11T21:34:47.393307+0000","last_fullsized":"2024-03-11T21:34:47.393307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:33:40.550110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.550110+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.511780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":684,"num_read_kb":679,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":881,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395183+0000","last_change":"2024-03-11T21:34:33.535935+0000","last_active":"2024-03-11T21:34:47.395183+0000","last_peered":"2024-03-11T21:34:47.395183+0000","last_clean":"2024-03-11T21:34:47.395183+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:34:47.395183+0000","last_undegraded":"2024-03-11T21:34:47.395183+0000","last_fullsized":"2024-03-11T21:34:47.395183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:34:33.535753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.535753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.050650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":756,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.404964+0000","last_change":"2024-03-11T21:33:09.913175+0000","last_active":"2024-03-11T21:34:44.404964+0000","last_peered":"2024-03-11T21:34:44.404964+0000","last_clean":"2024-03-11T21:34:44.404964+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:34:44.404964+0000","last_undegraded":"2024-03-11T21:34:44.404964+0000","last_fullsized":"2024-03-11T21:34:44.404964+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:09.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.912886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357457+0000","last_change":"2024-03-11T21:34:31.810307+0000","last_active":"2024-03-11T21:34:47.357457+0000","last_peered":"2024-03-11T21:34:47.357457+0000","last_clean":"2024-03-11T21:34:47.357457+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:34:47.357457+0000","last_undegraded":"2024-03-11T21:34:47.357457+0000","last_fullsized":"2024-03-11T21:34:47.357457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.810016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.810016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.863820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368707+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:34:47.368707+0000","last_peered":"2024-03-11T21:34:47.368707+0000","last_clean":"2024-03-11T21:34:47.368707+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:34:47.368707+0000","last_undegraded":"2024-03-11T21:34:47.368707+0000","last_fullsized":"2024-03-11T21:34:47.368707+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":845,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375511+0000","last_change":"2024-03-11T21:33:09.651325+0000","last_active":"2024-03-11T21:34:47.375511+0000","last_peered":"2024-03-11T21:34:47.375511+0000","last_clean":"2024-03-11T21:34:47.375511+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:34:47.375511+0000","last_undegraded":"2024-03-11T21:34:47.375511+0000","last_fullsized":"2024-03-11T21:34:47.375511+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.651101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.651101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"389'44","reported_seq":923,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.413180+0000","last_change":"2024-03-11T21:32:05.077652+0000","last_active":"2024-03-11T21:34:44.413180+0000","last_peered":"2024-03-11T21:34:44.413180+0000","last_clean":"2024-03-11T21:34:44.413180+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:34:44.413180+0000","last_undegraded":"2024-03-11T21:34:44.413180+0000","last_fullsized":"2024-03-11T21:34:44.413180+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'32","last_scrub_stamp":"2024-03-11T21:32:05.077406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.077406+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368367+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:34:47.368367+0000","last_peered":"2024-03-11T21:34:47.368367+0000","last_clean":"2024-03-11T21:34:47.368367+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:34:47.368367+0000","last_undegraded":"2024-03-11T21:34:47.368367+0000","last_fullsized":"2024-03-11T21:34:47.368367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":703,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.413268+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:34:44.413268+0000","last_peered":"2024-03-11T21:34:44.413268+0000","last_clean":"2024-03-11T21:34:44.413268+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:34:44.413268+0000","last_undegraded":"2024-03-11T21:34:44.413268+0000","last_fullsized":"2024-03-11T21:34:44.413268+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:15.222087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":692,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.409115+0000","last_change":"2024-03-11T21:34:35.105891+0000","last_active":"2024-03-11T21:34:43.409115+0000","last_peered":"2024-03-11T21:34:43.409115+0000","last_clean":"2024-03-11T21:34:43.409115+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:34:43.409115+0000","last_undegraded":"2024-03-11T21:34:43.409115+0000","last_fullsized":"2024-03-11T21:34:43.409115+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.105714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.105714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.841002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":853,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386191+0000","last_change":"2024-03-11T21:32:07.813703+0000","last_active":"2024-03-11T21:34:47.386191+0000","last_peered":"2024-03-11T21:34:47.386191+0000","last_clean":"2024-03-11T21:34:47.386191+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:34:47.386191+0000","last_undegraded":"2024-03-11T21:34:47.386191+0000","last_fullsized":"2024-03-11T21:34:47.386191+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.813604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.813604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":861,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366449+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:34:47.366449+0000","last_peered":"2024-03-11T21:34:47.366449+0000","last_clean":"2024-03-11T21:34:47.366449+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:34:47.366449+0000","last_undegraded":"2024-03-11T21:34:47.366449+0000","last_fullsized":"2024-03-11T21:34:47.366449+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":885,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.409654+0000","last_change":"2024-03-11T21:33:17.014343+0000","last_active":"2024-03-11T21:34:43.409654+0000","last_peered":"2024-03-11T21:34:43.409654+0000","last_clean":"2024-03-11T21:34:43.409654+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:34:43.409654+0000","last_undegraded":"2024-03-11T21:34:43.409654+0000","last_fullsized":"2024-03-11T21:34:43.409654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:17.014179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.014179+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":705,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.412052+0000","last_change":"2024-03-11T21:32:17.944438+0000","last_active":"2024-03-11T21:34:44.412052+0000","last_peered":"2024-03-11T21:34:44.412052+0000","last_clean":"2024-03-11T21:34:44.412052+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:34:44.412052+0000","last_undegraded":"2024-03-11T21:34:44.412052+0000","last_fullsized":"2024-03-11T21:34:44.412052+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.944141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.944141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":690,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392320+0000","last_change":"2024-03-11T21:32:16.760994+0000","last_active":"2024-03-11T21:34:47.392320+0000","last_peered":"2024-03-11T21:34:47.392320+0000","last_clean":"2024-03-11T21:34:47.392320+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:34:47.392320+0000","last_undegraded":"2024-03-11T21:34:47.392320+0000","last_fullsized":"2024-03-11T21:34:47.392320+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.760914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.760914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":836,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367811+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:34:47.367811+0000","last_peered":"2024-03-11T21:34:47.367811+0000","last_clean":"2024-03-11T21:34:47.367811+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:34:47.367811+0000","last_undegraded":"2024-03-11T21:34:47.367811+0000","last_fullsized":"2024-03-11T21:34:47.367811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1108,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366664+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:34:47.366664+0000","last_peered":"2024-03-11T21:34:47.366664+0000","last_clean":"2024-03-11T21:34:47.366664+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:34:47.366664+0000","last_undegraded":"2024-03-11T21:34:47.366664+0000","last_fullsized":"2024-03-11T21:34:47.366664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":860,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385964+0000","last_change":"2024-03-11T21:33:13.461458+0000","last_active":"2024-03-11T21:34:47.385964+0000","last_peered":"2024-03-11T21:34:47.385964+0000","last_clean":"2024-03-11T21:34:47.385964+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:34:47.385964+0000","last_undegraded":"2024-03-11T21:34:47.385964+0000","last_fullsized":"2024-03-11T21:34:47.385964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.461214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.461214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":878,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394823+0000","last_change":"2024-03-11T21:34:38.596842+0000","last_active":"2024-03-11T21:34:47.394823+0000","last_peered":"2024-03-11T21:34:47.394823+0000","last_clean":"2024-03-11T21:34:47.394823+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:34:47.394823+0000","last_undegraded":"2024-03-11T21:34:47.394823+0000","last_fullsized":"2024-03-11T21:34:47.394823+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:34:38.596731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.596731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.505236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386314+0000","last_change":"2024-03-11T21:32:26.159690+0000","last_active":"2024-03-11T21:34:47.386314+0000","last_peered":"2024-03-11T21:34:47.386314+0000","last_clean":"2024-03-11T21:34:47.386314+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:34:47.386314+0000","last_undegraded":"2024-03-11T21:34:47.386314+0000","last_fullsized":"2024-03-11T21:34:47.386314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:26.159472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:26.159472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":706,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384189+0000","last_change":"2024-03-11T21:34:36.837312+0000","last_active":"2024-03-11T21:34:47.384189+0000","last_peered":"2024-03-11T21:34:47.384189+0000","last_clean":"2024-03-11T21:34:47.384189+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:34:47.384189+0000","last_undegraded":"2024-03-11T21:34:47.384189+0000","last_fullsized":"2024-03-11T21:34:47.384189+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.837081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.837081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.917829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392857+0000","last_change":"2024-03-11T21:33:43.147050+0000","last_active":"2024-03-11T21:34:47.392857+0000","last_peered":"2024-03-11T21:34:47.392857+0000","last_clean":"2024-03-11T21:34:47.392857+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:34:47.392857+0000","last_undegraded":"2024-03-11T21:34:47.392857+0000","last_fullsized":"2024-03-11T21:34:47.392857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.146852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.146852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.907189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":871,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390119+0000","last_change":"2024-03-11T21:34:44.457291+0000","last_active":"2024-03-11T21:34:47.390119+0000","last_peered":"2024-03-11T21:34:47.390119+0000","last_clean":"2024-03-11T21:34:47.390119+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:34:47.390119+0000","last_undegraded":"2024-03-11T21:34:47.390119+0000","last_fullsized":"2024-03-11T21:34:47.390119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.457161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.457161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.611516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":787,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.412570+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:34:44.412570+0000","last_peered":"2024-03-11T21:34:44.412570+0000","last_clean":"2024-03-11T21:34:44.412570+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:34:44.412570+0000","last_undegraded":"2024-03-11T21:34:44.412570+0000","last_fullsized":"2024-03-11T21:34:44.412570+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368338+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:34:47.368338+0000","last_peered":"2024-03-11T21:34:47.368338+0000","last_clean":"2024-03-11T21:34:47.368338+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:34:47.368338+0000","last_undegraded":"2024-03-11T21:34:47.368338+0000","last_fullsized":"2024-03-11T21:34:47.368338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:51.335664+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":692,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.409384+0000","last_change":"2024-03-11T21:33:40.988530+0000","last_active":"2024-03-11T21:34:43.409384+0000","last_peered":"2024-03-11T21:34:43.409384+0000","last_clean":"2024-03-11T21:34:43.409384+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:34:43.409384+0000","last_undegraded":"2024-03-11T21:34:43.409384+0000","last_fullsized":"2024-03-11T21:34:43.409384+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.988258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.988258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.542071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":848,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407442+0000","last_change":"2024-03-11T21:33:35.976814+0000","last_active":"2024-03-11T21:34:43.407442+0000","last_peered":"2024-03-11T21:34:43.407442+0000","last_clean":"2024-03-11T21:34:43.407442+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:34:43.407442+0000","last_undegraded":"2024-03-11T21:34:43.407442+0000","last_fullsized":"2024-03-11T21:34:43.407442+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.976718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.976718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:47.438197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":918,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357805+0000","last_change":"2024-03-11T21:34:39.946045+0000","last_active":"2024-03-11T21:34:47.357805+0000","last_peered":"2024-03-11T21:34:47.357805+0000","last_clean":"2024-03-11T21:34:47.357805+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:34:47.357805+0000","last_undegraded":"2024-03-11T21:34:47.357805+0000","last_fullsized":"2024-03-11T21:34:47.357805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:34:39.945944+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.945944+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:01.612882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":861,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384696+0000","last_change":"2024-03-11T21:33:09.417056+0000","last_active":"2024-03-11T21:34:47.384696+0000","last_peered":"2024-03-11T21:34:47.384696+0000","last_clean":"2024-03-11T21:34:47.384696+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:34:47.384696+0000","last_undegraded":"2024-03-11T21:34:47.384696+0000","last_fullsized":"2024-03-11T21:34:47.384696+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.416868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.416868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395237+0000","last_change":"2024-03-11T21:33:33.610158+0000","last_active":"2024-03-11T21:34:47.395237+0000","last_peered":"2024-03-11T21:34:47.395237+0000","last_clean":"2024-03-11T21:34:47.395237+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:34:47.395237+0000","last_undegraded":"2024-03-11T21:34:47.395237+0000","last_fullsized":"2024-03-11T21:34:47.395237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:33.610071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:33.610071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":706,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385752+0000","last_change":"2024-03-11T21:34:45.790219+0000","last_active":"2024-03-11T21:34:47.385752+0000","last_peered":"2024-03-11T21:34:47.385752+0000","last_clean":"2024-03-11T21:34:47.385752+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:34:47.385752+0000","last_undegraded":"2024-03-11T21:34:47.385752+0000","last_fullsized":"2024-03-11T21:34:47.385752+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.790012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.790012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.607134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":993,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.408985+0000","last_change":"2024-03-11T21:32:15.949343+0000","last_active":"2024-03-11T21:34:43.408985+0000","last_peered":"2024-03-11T21:34:43.408985+0000","last_clean":"2024-03-11T21:34:43.408985+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:34:43.408985+0000","last_undegraded":"2024-03-11T21:34:43.408985+0000","last_fullsized":"2024-03-11T21:34:43.408985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:32:15.949107+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.949107+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":740,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.405773+0000","last_change":"2024-03-11T21:33:31.047676+0000","last_active":"2024-03-11T21:34:44.405773+0000","last_peered":"2024-03-11T21:34:44.405773+0000","last_clean":"2024-03-11T21:34:44.405773+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:34:44.405773+0000","last_undegraded":"2024-03-11T21:34:44.405773+0000","last_fullsized":"2024-03-11T21:34:44.405773+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:33:31.047447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.047447+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1037,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385735+0000","last_change":"2024-03-11T21:33:37.464613+0000","last_active":"2024-03-11T21:34:47.385735+0000","last_peered":"2024-03-11T21:34:47.385735+0000","last_clean":"2024-03-11T21:34:47.385735+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:34:47.385735+0000","last_undegraded":"2024-03-11T21:34:47.385735+0000","last_fullsized":"2024-03-11T21:34:47.385735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:37.464485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.464485+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":711,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.411531+0000","last_change":"2024-03-11T21:33:23.993323+0000","last_active":"2024-03-11T21:34:44.411531+0000","last_peered":"2024-03-11T21:34:44.411531+0000","last_clean":"2024-03-11T21:34:44.411531+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:34:44.411531+0000","last_undegraded":"2024-03-11T21:34:44.411531+0000","last_fullsized":"2024-03-11T21:34:44.411531+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.993242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.993242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.886605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":834,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.405387+0000","last_change":"2024-03-11T21:32:16.907647+0000","last_active":"2024-03-11T21:34:43.405387+0000","last_peered":"2024-03-11T21:34:43.405387+0000","last_clean":"2024-03-11T21:34:43.405387+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:34:43.405387+0000","last_undegraded":"2024-03-11T21:34:43.405387+0000","last_fullsized":"2024-03-11T21:34:43.405387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.907562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.907562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":564,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.411786+0000","last_change":"2024-03-11T21:33:43.133706+0000","last_active":"2024-03-11T21:34:44.411786+0000","last_peered":"2024-03-11T21:34:44.411786+0000","last_clean":"2024-03-11T21:34:44.411786+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:34:44.411786+0000","last_undegraded":"2024-03-11T21:34:44.411786+0000","last_fullsized":"2024-03-11T21:34:44.411786+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.133485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.133485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:45.234400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":872,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390462+0000","last_change":"2024-03-11T21:34:36.323730+0000","last_active":"2024-03-11T21:34:47.390462+0000","last_peered":"2024-03-11T21:34:47.390462+0000","last_clean":"2024-03-11T21:34:47.390462+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:34:47.390462+0000","last_undegraded":"2024-03-11T21:34:47.390462+0000","last_fullsized":"2024-03-11T21:34:47.390462+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.323616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.323616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.807642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":901,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368851+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:34:47.368851+0000","last_peered":"2024-03-11T21:34:47.368851+0000","last_clean":"2024-03-11T21:34:47.368851+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:34:47.368851+0000","last_undegraded":"2024-03-11T21:34:47.368851+0000","last_fullsized":"2024-03-11T21:34:47.368851+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.849278+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":881,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358741+0000","last_change":"2024-03-11T21:34:36.855849+0000","last_active":"2024-03-11T21:34:47.358741+0000","last_peered":"2024-03-11T21:34:47.358741+0000","last_clean":"2024-03-11T21:34:47.358741+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:34:47.358741+0000","last_undegraded":"2024-03-11T21:34:47.358741+0000","last_fullsized":"2024-03-11T21:34:47.358741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:34:36.855625+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.855625+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.347567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":849,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.396950+0000","last_change":"2024-03-11T21:33:14.693542+0000","last_active":"2024-03-11T21:34:47.396950+0000","last_peered":"2024-03-11T21:34:47.396950+0000","last_clean":"2024-03-11T21:34:47.396950+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:34:47.396950+0000","last_undegraded":"2024-03-11T21:34:47.396950+0000","last_fullsized":"2024-03-11T21:34:47.396950+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:14.693316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.693316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.397156+0000","last_change":"2024-03-11T21:33:42.591082+0000","last_active":"2024-03-11T21:34:47.397156+0000","last_peered":"2024-03-11T21:34:47.397156+0000","last_clean":"2024-03-11T21:34:47.397156+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:34:47.397156+0000","last_undegraded":"2024-03-11T21:34:47.397156+0000","last_fullsized":"2024-03-11T21:34:47.397156+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:42.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.590963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":879,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367414+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:34:47.367414+0000","last_peered":"2024-03-11T21:34:47.367414+0000","last_clean":"2024-03-11T21:34:47.367414+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:34:47.367414+0000","last_undegraded":"2024-03-11T21:34:47.367414+0000","last_fullsized":"2024-03-11T21:34:47.367414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":872,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.390090+0000","last_change":"2024-03-11T21:33:08.186982+0000","last_active":"2024-03-11T21:34:47.390090+0000","last_peered":"2024-03-11T21:34:47.390090+0000","last_clean":"2024-03-11T21:34:47.390090+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:34:47.390090+0000","last_undegraded":"2024-03-11T21:34:47.390090+0000","last_fullsized":"2024-03-11T21:34:47.390090+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.186732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.186732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395455+0000","last_change":"2024-03-11T21:33:32.575760+0000","last_active":"2024-03-11T21:34:47.395455+0000","last_peered":"2024-03-11T21:34:47.395455+0000","last_clean":"2024-03-11T21:34:47.395455+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:34:47.395455+0000","last_undegraded":"2024-03-11T21:34:47.395455+0000","last_fullsized":"2024-03-11T21:34:47.395455+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.575652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.575652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":858,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.391941+0000","last_change":"2024-03-11T21:33:46.058840+0000","last_active":"2024-03-11T21:34:47.391941+0000","last_peered":"2024-03-11T21:34:47.391941+0000","last_clean":"2024-03-11T21:34:47.391941+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:34:47.391941+0000","last_undegraded":"2024-03-11T21:34:47.391941+0000","last_fullsized":"2024-03-11T21:34:47.391941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.058738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.058738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.334303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395310+0000","last_change":"2024-03-11T21:34:39.587604+0000","last_active":"2024-03-11T21:34:47.395310+0000","last_peered":"2024-03-11T21:34:47.395310+0000","last_clean":"2024-03-11T21:34:47.395310+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:34:47.395310+0000","last_undegraded":"2024-03-11T21:34:47.395310+0000","last_fullsized":"2024-03-11T21:34:47.395310+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.587525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.587525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.756876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":879,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.374945+0000","last_change":"2024-03-11T21:34:46.190536+0000","last_active":"2024-03-11T21:34:47.374945+0000","last_peered":"2024-03-11T21:34:47.374945+0000","last_clean":"2024-03-11T21:34:47.374945+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:34:47.374945+0000","last_undegraded":"2024-03-11T21:34:47.374945+0000","last_fullsized":"2024-03-11T21:34:47.374945+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:34:46.190301+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:34:46.190301+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.316002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":870,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375889+0000","last_change":"2024-03-11T21:34:31.418213+0000","last_active":"2024-03-11T21:34:47.375889+0000","last_peered":"2024-03-11T21:34:47.375889+0000","last_clean":"2024-03-11T21:34:47.375889+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:34:47.375889+0000","last_undegraded":"2024-03-11T21:34:47.375889+0000","last_fullsized":"2024-03-11T21:34:47.375889+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.417992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.417992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.992829+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386712+0000","last_change":"2024-03-11T21:32:06.800643+0000","last_active":"2024-03-11T21:34:47.386712+0000","last_peered":"2024-03-11T21:34:47.386712+0000","last_clean":"2024-03-11T21:34:47.386712+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:34:47.386712+0000","last_undegraded":"2024-03-11T21:34:47.386712+0000","last_fullsized":"2024-03-11T21:34:47.386712+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.800436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:08.954967+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.800436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":839,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368172+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:34:47.368172+0000","last_peered":"2024-03-11T21:34:47.368172+0000","last_clean":"2024-03-11T21:34:47.368172+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:34:47.368172+0000","last_undegraded":"2024-03-11T21:34:47.368172+0000","last_fullsized":"2024-03-11T21:34:47.368172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":690,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394677+0000","last_change":"2024-03-11T21:31:57.633633+0000","last_active":"2024-03-11T21:34:47.394677+0000","last_peered":"2024-03-11T21:34:47.394677+0000","last_clean":"2024-03-11T21:34:47.394677+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:34:47.394677+0000","last_undegraded":"2024-03-11T21:34:47.394677+0000","last_fullsized":"2024-03-11T21:34:47.394677+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.633438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.633438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":879,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367817+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:34:47.367817+0000","last_peered":"2024-03-11T21:34:47.367817+0000","last_clean":"2024-03-11T21:34:47.367817+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:34:47.367817+0000","last_undegraded":"2024-03-11T21:34:47.367817+0000","last_fullsized":"2024-03-11T21:34:47.367817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":868,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385135+0000","last_change":"2024-03-11T21:34:31.781546+0000","last_active":"2024-03-11T21:34:47.385135+0000","last_peered":"2024-03-11T21:34:47.385135+0000","last_clean":"2024-03-11T21:34:47.385135+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:34:47.385135+0000","last_undegraded":"2024-03-11T21:34:47.385135+0000","last_fullsized":"2024-03-11T21:34:47.385135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:50.330053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1004,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386040+0000","last_change":"2024-03-11T21:32:15.855148+0000","last_active":"2024-03-11T21:34:47.386040+0000","last_peered":"2024-03-11T21:34:47.386040+0000","last_clean":"2024-03-11T21:34:47.386040+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:34:47.386040+0000","last_undegraded":"2024-03-11T21:34:47.386040+0000","last_fullsized":"2024-03-11T21:34:47.386040+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:32:15.854918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.854918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":842,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407662+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:34:43.407662+0000","last_peered":"2024-03-11T21:34:43.407662+0000","last_clean":"2024-03-11T21:34:43.407662+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:34:43.407662+0000","last_undegraded":"2024-03-11T21:34:43.407662+0000","last_fullsized":"2024-03-11T21:34:43.407662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:09.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":705,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375192+0000","last_change":"2024-03-11T21:33:59.482824+0000","last_active":"2024-03-11T21:34:47.375192+0000","last_peered":"2024-03-11T21:34:47.375192+0000","last_clean":"2024-03-11T21:34:47.375192+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:34:47.375192+0000","last_undegraded":"2024-03-11T21:34:47.375192+0000","last_fullsized":"2024-03-11T21:34:47.375192+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.434631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.434631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":879,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.405197+0000","last_change":"2024-03-11T21:33:42.968747+0000","last_active":"2024-03-11T21:34:43.405197+0000","last_peered":"2024-03-11T21:34:43.405197+0000","last_clean":"2024-03-11T21:34:43.405197+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:34:43.405197+0000","last_undegraded":"2024-03-11T21:34:43.405197+0000","last_fullsized":"2024-03-11T21:34:43.405197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.000934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":854,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.408661+0000","last_change":"2024-03-11T21:34:37.079514+0000","last_active":"2024-03-11T21:34:43.408661+0000","last_peered":"2024-03-11T21:34:43.408661+0000","last_clean":"2024-03-11T21:34:43.408661+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:34:43.408661+0000","last_undegraded":"2024-03-11T21:34:43.408661+0000","last_fullsized":"2024-03-11T21:34:43.408661+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.079361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.079361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.776119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":996,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393523+0000","last_change":"2024-03-11T21:33:19.697889+0000","last_active":"2024-03-11T21:34:47.393523+0000","last_peered":"2024-03-11T21:34:47.393523+0000","last_clean":"2024-03-11T21:34:47.393523+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:34:47.393523+0000","last_undegraded":"2024-03-11T21:34:47.393523+0000","last_fullsized":"2024-03-11T21:34:47.393523+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:33:19.697575+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.697575+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":850,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.359101+0000","last_change":"2024-03-11T21:33:40.730218+0000","last_active":"2024-03-11T21:34:47.359101+0000","last_peered":"2024-03-11T21:34:47.359101+0000","last_clean":"2024-03-11T21:34:47.359101+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:34:47.359101+0000","last_undegraded":"2024-03-11T21:34:47.359101+0000","last_fullsized":"2024-03-11T21:34:47.359101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.729858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.729858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.347472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":560,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.405818+0000","last_change":"2024-03-11T21:33:26.034503+0000","last_active":"2024-03-11T21:34:44.405818+0000","last_peered":"2024-03-11T21:34:44.405818+0000","last_clean":"2024-03-11T21:34:44.405818+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:34:44.405818+0000","last_undegraded":"2024-03-11T21:34:44.405818+0000","last_fullsized":"2024-03-11T21:34:44.405818+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:26.034377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:26.034377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:54.744337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":876,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368608+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:34:47.368608+0000","last_peered":"2024-03-11T21:34:47.368608+0000","last_clean":"2024-03-11T21:34:47.368608+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:34:47.368608+0000","last_undegraded":"2024-03-11T21:34:47.368608+0000","last_fullsized":"2024-03-11T21:34:47.368608+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":869,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375854+0000","last_change":"2024-03-11T21:34:37.332784+0000","last_active":"2024-03-11T21:34:47.375854+0000","last_peered":"2024-03-11T21:34:47.375854+0000","last_clean":"2024-03-11T21:34:47.375854+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:34:47.375854+0000","last_undegraded":"2024-03-11T21:34:47.375854+0000","last_fullsized":"2024-03-11T21:34:47.375854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.332577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.332577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.761682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":853,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.396947+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:34:47.396947+0000","last_peered":"2024-03-11T21:34:47.396947+0000","last_clean":"2024-03-11T21:34:47.396947+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:34:47.396947+0000","last_undegraded":"2024-03-11T21:34:47.396947+0000","last_fullsized":"2024-03-11T21:34:47.396947+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.579186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":850,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.405247+0000","last_change":"2024-03-11T21:34:41.069563+0000","last_active":"2024-03-11T21:34:43.405247+0000","last_peered":"2024-03-11T21:34:43.405247+0000","last_clean":"2024-03-11T21:34:43.405247+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:34:43.405247+0000","last_undegraded":"2024-03-11T21:34:43.405247+0000","last_fullsized":"2024-03-11T21:34:43.405247+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.069432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.069432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:55.125039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":698,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407355+0000","last_change":"2024-03-11T21:33:59.486720+0000","last_active":"2024-03-11T21:34:43.407355+0000","last_peered":"2024-03-11T21:34:43.407355+0000","last_clean":"2024-03-11T21:34:43.407355+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:34:43.407355+0000","last_undegraded":"2024-03-11T21:34:43.407355+0000","last_fullsized":"2024-03-11T21:34:43.407355+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:29.057966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.057966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":889,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358846+0000","last_change":"2024-03-11T21:33:38.729069+0000","last_active":"2024-03-11T21:34:47.358846+0000","last_peered":"2024-03-11T21:34:47.358846+0000","last_clean":"2024-03-11T21:34:47.358846+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:34:47.358846+0000","last_undegraded":"2024-03-11T21:34:47.358846+0000","last_fullsized":"2024-03-11T21:34:47.358846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:38.728913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:38.728913+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":755,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.422268+0000","last_change":"2024-03-11T21:34:44.191646+0000","last_active":"2024-03-11T21:34:44.422268+0000","last_peered":"2024-03-11T21:34:44.422268+0000","last_clean":"2024-03-11T21:34:44.422268+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:34:44.422268+0000","last_undegraded":"2024-03-11T21:34:44.422268+0000","last_fullsized":"2024-03-11T21:34:44.422268+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.191432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.191432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.662638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":723,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.405195+0000","last_change":"2024-03-11T21:34:32.241415+0000","last_active":"2024-03-11T21:34:44.405195+0000","last_peered":"2024-03-11T21:34:44.405195+0000","last_clean":"2024-03-11T21:34:44.405195+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:34:44.405195+0000","last_undegraded":"2024-03-11T21:34:44.405195+0000","last_fullsized":"2024-03-11T21:34:44.405195+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.241159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.241159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.885858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":839,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.393818+0000","last_change":"2024-03-11T21:33:35.541846+0000","last_active":"2024-03-11T21:34:47.393818+0000","last_peered":"2024-03-11T21:34:47.393818+0000","last_clean":"2024-03-11T21:34:47.393818+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:34:47.393818+0000","last_undegraded":"2024-03-11T21:34:47.393818+0000","last_fullsized":"2024-03-11T21:34:47.393818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.541731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.541731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:00.532157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":690,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406884+0000","last_change":"2024-03-11T21:33:59.486095+0000","last_active":"2024-03-11T21:34:43.406884+0000","last_peered":"2024-03-11T21:34:43.406884+0000","last_clean":"2024-03-11T21:34:43.406884+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:34:43.406884+0000","last_undegraded":"2024-03-11T21:34:43.406884+0000","last_fullsized":"2024-03-11T21:34:43.406884+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.866701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.866701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":868,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407933+0000","last_change":"2024-03-11T21:33:20.954157+0000","last_active":"2024-03-11T21:34:43.407933+0000","last_peered":"2024-03-11T21:34:43.407933+0000","last_clean":"2024-03-11T21:34:43.407933+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:34:43.407933+0000","last_undegraded":"2024-03-11T21:34:43.407933+0000","last_fullsized":"2024-03-11T21:34:43.407933+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:33:20.954022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.954022+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:48.519773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":859,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.387077+0000","last_change":"2024-03-11T21:33:46.468750+0000","last_active":"2024-03-11T21:34:47.387077+0000","last_peered":"2024-03-11T21:34:47.387077+0000","last_clean":"2024-03-11T21:34:47.387077+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:34:47.387077+0000","last_undegraded":"2024-03-11T21:34:47.387077+0000","last_fullsized":"2024-03-11T21:34:47.387077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.468649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.468649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:59.584649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":861,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375835+0000","last_change":"2024-03-11T21:34:45.185932+0000","last_active":"2024-03-11T21:34:47.375835+0000","last_peered":"2024-03-11T21:34:47.375835+0000","last_clean":"2024-03-11T21:34:47.375835+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:34:47.375835+0000","last_undegraded":"2024-03-11T21:34:47.375835+0000","last_fullsized":"2024-03-11T21:34:47.375835+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.185599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.185599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.803583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":839,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367164+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:34:47.367164+0000","last_peered":"2024-03-11T21:34:47.367164+0000","last_clean":"2024-03-11T21:34:47.367164+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:34:47.367164+0000","last_undegraded":"2024-03-11T21:34:47.367164+0000","last_fullsized":"2024-03-11T21:34:47.367164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.815187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":693,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.406925+0000","last_change":"2024-03-11T21:32:12.998779+0000","last_active":"2024-03-11T21:34:43.406925+0000","last_peered":"2024-03-11T21:34:43.406925+0000","last_clean":"2024-03-11T21:34:43.406925+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:34:43.406925+0000","last_undegraded":"2024-03-11T21:34:43.406925+0000","last_fullsized":"2024-03-11T21:34:43.406925+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.998644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.998644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":889,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.386562+0000","last_change":"2024-03-11T21:34:44.838684+0000","last_active":"2024-03-11T21:34:47.386562+0000","last_peered":"2024-03-11T21:34:47.386562+0000","last_clean":"2024-03-11T21:34:47.386562+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:34:47.386562+0000","last_undegraded":"2024-03-11T21:34:47.386562+0000","last_fullsized":"2024-03-11T21:34:47.386562+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:44.838430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.838430+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:11.067144+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":868,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376600+0000","last_change":"2024-03-11T21:34:44.232623+0000","last_active":"2024-03-11T21:34:47.376600+0000","last_peered":"2024-03-11T21:34:47.376600+0000","last_clean":"2024-03-11T21:34:47.376600+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:34:47.376600+0000","last_undegraded":"2024-03-11T21:34:47.376600+0000","last_fullsized":"2024-03-11T21:34:47.376600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.232392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.232392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.623885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":836,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366983+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:34:47.366983+0000","last_peered":"2024-03-11T21:34:47.366983+0000","last_clean":"2024-03-11T21:34:47.366983+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:34:47.366983+0000","last_undegraded":"2024-03-11T21:34:47.366983+0000","last_fullsized":"2024-03-11T21:34:47.366983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":839,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.397020+0000","last_change":"2024-03-11T21:34:42.859611+0000","last_active":"2024-03-11T21:34:47.397020+0000","last_peered":"2024-03-11T21:34:47.397020+0000","last_clean":"2024-03-11T21:34:47.397020+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:34:47.397020+0000","last_undegraded":"2024-03-11T21:34:47.397020+0000","last_fullsized":"2024-03-11T21:34:47.397020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.859451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.859451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.405214+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":689,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367484+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:34:47.367484+0000","last_peered":"2024-03-11T21:34:47.367484+0000","last_clean":"2024-03-11T21:34:47.367484+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:34:47.367484+0000","last_undegraded":"2024-03-11T21:34:47.367484+0000","last_fullsized":"2024-03-11T21:34:47.367484+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"389'27","reported_seq":801,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.413691+0000","last_change":"2024-03-11T21:33:21.967867+0000","last_active":"2024-03-11T21:34:44.413691+0000","last_peered":"2024-03-11T21:34:44.413691+0000","last_clean":"2024-03-11T21:34:44.413691+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:34:44.413691+0000","last_undegraded":"2024-03-11T21:34:44.413691+0000","last_fullsized":"2024-03-11T21:34:44.413691+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:21.967756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.967756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:50.206133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":857,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384637+0000","last_change":"2024-03-11T21:32:29.159917+0000","last_active":"2024-03-11T21:34:47.384637+0000","last_peered":"2024-03-11T21:34:47.384637+0000","last_clean":"2024-03-11T21:34:47.384637+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:34:47.384637+0000","last_undegraded":"2024-03-11T21:34:47.384637+0000","last_fullsized":"2024-03-11T21:34:47.384637+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:29.159716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:29.159716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1007,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.376706+0000","last_change":"2024-03-11T21:33:10.606465+0000","last_active":"2024-03-11T21:34:47.376706+0000","last_peered":"2024-03-11T21:34:47.376706+0000","last_clean":"2024-03-11T21:34:47.376706+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:34:47.376706+0000","last_undegraded":"2024-03-11T21:34:47.376706+0000","last_fullsized":"2024-03-11T21:34:47.376706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:33:10.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.606357+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":833,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367892+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:34:47.367892+0000","last_peered":"2024-03-11T21:34:47.367892+0000","last_clean":"2024-03-11T21:34:47.367892+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:34:47.367892+0000","last_undegraded":"2024-03-11T21:34:47.367892+0000","last_fullsized":"2024-03-11T21:34:47.367892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":706,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384345+0000","last_change":"2024-03-11T21:33:12.438198+0000","last_active":"2024-03-11T21:34:47.384345+0000","last_peered":"2024-03-11T21:34:47.384345+0000","last_clean":"2024-03-11T21:34:47.384345+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:34:47.384345+0000","last_undegraded":"2024-03-11T21:34:47.384345+0000","last_fullsized":"2024-03-11T21:34:47.384345+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.438028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.438028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":881,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.392560+0000","last_change":"2024-03-11T21:31:54.548395+0000","last_active":"2024-03-11T21:34:47.392560+0000","last_peered":"2024-03-11T21:34:47.392560+0000","last_clean":"2024-03-11T21:34:47.392560+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:34:47.392560+0000","last_undegraded":"2024-03-11T21:34:47.392560+0000","last_fullsized":"2024-03-11T21:34:47.392560+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T21:31:54.548275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.548275+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394026+0000","last_change":"2024-03-11T21:33:06.710223+0000","last_active":"2024-03-11T21:34:47.394026+0000","last_peered":"2024-03-11T21:34:47.394026+0000","last_clean":"2024-03-11T21:34:47.394026+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:34:47.394026+0000","last_undegraded":"2024-03-11T21:34:47.394026+0000","last_fullsized":"2024-03-11T21:34:47.394026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:06.709989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.709989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":839,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.407583+0000","last_change":"2024-03-11T21:32:23.014073+0000","last_active":"2024-03-11T21:34:43.407583+0000","last_peered":"2024-03-11T21:34:43.407583+0000","last_clean":"2024-03-11T21:34:43.407583+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:34:43.407583+0000","last_undegraded":"2024-03-11T21:34:43.407583+0000","last_fullsized":"2024-03-11T21:34:43.407583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.013984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.013984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":706,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.409948+0000","last_change":"2024-03-11T21:33:05.875547+0000","last_active":"2024-03-11T21:34:44.409948+0000","last_peered":"2024-03-11T21:34:44.409948+0000","last_clean":"2024-03-11T21:34:44.409948+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:34:44.409948+0000","last_undegraded":"2024-03-11T21:34:44.409948+0000","last_fullsized":"2024-03-11T21:34:44.409948+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:05.875434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.875434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":689,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.366929+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:34:47.366929+0000","last_peered":"2024-03-11T21:34:47.366929+0000","last_clean":"2024-03-11T21:34:47.366929+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:34:47.366929+0000","last_undegraded":"2024-03-11T21:34:47.366929+0000","last_fullsized":"2024-03-11T21:34:47.366929+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":857,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.408774+0000","last_change":"2024-03-11T21:33:19.017659+0000","last_active":"2024-03-11T21:34:43.408774+0000","last_peered":"2024-03-11T21:34:43.408774+0000","last_clean":"2024-03-11T21:34:43.408774+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:34:43.408774+0000","last_undegraded":"2024-03-11T21:34:43.408774+0000","last_fullsized":"2024-03-11T21:34:43.408774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:33:19.017449+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.017449+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:43.631706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":868,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.375393+0000","last_change":"2024-03-11T21:34:35.350319+0000","last_active":"2024-03-11T21:34:47.375393+0000","last_peered":"2024-03-11T21:34:47.375393+0000","last_clean":"2024-03-11T21:34:47.375393+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:34:47.375393+0000","last_undegraded":"2024-03-11T21:34:47.375393+0000","last_fullsized":"2024-03-11T21:34:47.375393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.350122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.350122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.565430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":700,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.422553+0000","last_change":"2024-03-11T21:33:36.120790+0000","last_active":"2024-03-11T21:34:44.422553+0000","last_peered":"2024-03-11T21:34:44.422553+0000","last_clean":"2024-03-11T21:34:44.422553+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:34:44.422553+0000","last_undegraded":"2024-03-11T21:34:44.422553+0000","last_fullsized":"2024-03-11T21:34:44.422553+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.120602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.120602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358773+0000","last_change":"2024-03-11T21:33:41.774829+0000","last_active":"2024-03-11T21:34:47.358773+0000","last_peered":"2024-03-11T21:34:47.358773+0000","last_clean":"2024-03-11T21:34:47.358773+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:34:47.358773+0000","last_undegraded":"2024-03-11T21:34:47.358773+0000","last_fullsized":"2024-03-11T21:34:47.358773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.774699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.774699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.521322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356544+0000","last_change":"2024-03-11T21:33:59.391439+0000","last_active":"2024-03-11T21:34:47.356544+0000","last_peered":"2024-03-11T21:34:47.356544+0000","last_clean":"2024-03-11T21:34:47.356544+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:34:47.356544+0000","last_undegraded":"2024-03-11T21:34:47.356544+0000","last_fullsized":"2024-03-11T21:34:47.356544+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.655683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":892,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357465+0000","last_change":"2024-03-11T21:33:27.686917+0000","last_active":"2024-03-11T21:34:47.357465+0000","last_peered":"2024-03-11T21:34:47.357465+0000","last_clean":"2024-03-11T21:34:47.357465+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:34:47.357465+0000","last_undegraded":"2024-03-11T21:34:47.357465+0000","last_fullsized":"2024-03-11T21:34:47.357465+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:33:27.686783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.686783+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.396570+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:34:47.396570+0000","last_peered":"2024-03-11T21:34:47.396570+0000","last_clean":"2024-03-11T21:34:47.396570+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:34:47.396570+0000","last_undegraded":"2024-03-11T21:34:47.396570+0000","last_fullsized":"2024-03-11T21:34:47.396570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":846,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.358932+0000","last_change":"2024-03-11T21:32:09.768728+0000","last_active":"2024-03-11T21:34:47.358932+0000","last_peered":"2024-03-11T21:34:47.358932+0000","last_clean":"2024-03-11T21:34:47.358932+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:34:47.358932+0000","last_undegraded":"2024-03-11T21:34:47.358932+0000","last_fullsized":"2024-03-11T21:34:47.358932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.768608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.768608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":840,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395232+0000","last_change":"2024-03-11T21:34:31.520200+0000","last_active":"2024-03-11T21:34:47.395232+0000","last_peered":"2024-03-11T21:34:47.395232+0000","last_clean":"2024-03-11T21:34:47.395232+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:34:47.395232+0000","last_undegraded":"2024-03-11T21:34:47.395232+0000","last_fullsized":"2024-03-11T21:34:47.395232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.520041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.520041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.369680+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:34:47.369680+0000","last_peered":"2024-03-11T21:34:47.369680+0000","last_clean":"2024-03-11T21:34:47.369680+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:34:47.369680+0000","last_undegraded":"2024-03-11T21:34:47.369680+0000","last_fullsized":"2024-03-11T21:34:47.369680+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":879,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385933+0000","last_change":"2024-03-11T21:32:09.759891+0000","last_active":"2024-03-11T21:34:47.385933+0000","last_peered":"2024-03-11T21:34:47.385933+0000","last_clean":"2024-03-11T21:34:47.385933+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:34:47.385933+0000","last_undegraded":"2024-03-11T21:34:47.385933+0000","last_fullsized":"2024-03-11T21:34:47.385933+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T21:32:09.759634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.759634+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367454+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:34:47.367454+0000","last_peered":"2024-03-11T21:34:47.367454+0000","last_clean":"2024-03-11T21:34:47.367454+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:34:47.367454+0000","last_undegraded":"2024-03-11T21:34:47.367454+0000","last_fullsized":"2024-03-11T21:34:47.367454+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":843,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385835+0000","last_change":"2024-03-11T21:33:17.530575+0000","last_active":"2024-03-11T21:34:47.385835+0000","last_peered":"2024-03-11T21:34:47.385835+0000","last_clean":"2024-03-11T21:34:47.385835+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:34:47.385835+0000","last_undegraded":"2024-03-11T21:34:47.385835+0000","last_fullsized":"2024-03-11T21:34:47.385835+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:17.530299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.530299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":850,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.405485+0000","last_change":"2024-03-11T21:34:42.092702+0000","last_active":"2024-03-11T21:34:43.405485+0000","last_peered":"2024-03-11T21:34:43.405485+0000","last_clean":"2024-03-11T21:34:43.405485+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:34:43.405485+0000","last_undegraded":"2024-03-11T21:34:43.405485+0000","last_fullsized":"2024-03-11T21:34:43.405485+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.092348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.092348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.061520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":709,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.959897+0000","last_change":"2024-03-11T21:34:47.959956+0000","last_active":"2024-03-11T21:34:47.959897+0000","last_peered":"2024-03-11T21:34:47.959897+0000","last_clean":"2024-03-11T21:34:47.959897+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:34:47.959897+0000","last_undegraded":"2024-03-11T21:34:47.959897+0000","last_fullsized":"2024-03-11T21:34:47.959897+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:47.959724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:47.959724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.149816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":914,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.385895+0000","last_change":"2024-03-11T21:33:20.542248+0000","last_active":"2024-03-11T21:34:47.385895+0000","last_peered":"2024-03-11T21:34:47.385895+0000","last_clean":"2024-03-11T21:34:47.385895+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:34:47.385895+0000","last_undegraded":"2024-03-11T21:34:47.385895+0000","last_fullsized":"2024-03-11T21:34:47.385895+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:33:20.542001+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.542001+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":851,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.368516+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:34:47.368516+0000","last_peered":"2024-03-11T21:34:47.368516+0000","last_clean":"2024-03-11T21:34:47.368516+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:34:47.368516+0000","last_undegraded":"2024-03-11T21:34:47.368516+0000","last_fullsized":"2024-03-11T21:34:47.368516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":844,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367008+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:34:47.367008+0000","last_peered":"2024-03-11T21:34:47.367008+0000","last_clean":"2024-03-11T21:34:47.367008+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:34:47.367008+0000","last_undegraded":"2024-03-11T21:34:47.367008+0000","last_fullsized":"2024-03-11T21:34:47.367008+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":848,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.384365+0000","last_change":"2024-03-11T21:33:23.549947+0000","last_active":"2024-03-11T21:34:47.384365+0000","last_peered":"2024-03-11T21:34:47.384365+0000","last_clean":"2024-03-11T21:34:47.384365+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:34:47.384365+0000","last_undegraded":"2024-03-11T21:34:47.384365+0000","last_fullsized":"2024-03-11T21:34:47.384365+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.549773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.549773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":698,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.367994+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:34:47.367994+0000","last_peered":"2024-03-11T21:34:47.367994+0000","last_clean":"2024-03-11T21:34:47.367994+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:34:47.367994+0000","last_undegraded":"2024-03-11T21:34:47.367994+0000","last_fullsized":"2024-03-11T21:34:47.367994+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":887,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.408135+0000","last_change":"2024-03-11T21:33:39.970299+0000","last_active":"2024-03-11T21:34:43.408135+0000","last_peered":"2024-03-11T21:34:43.408135+0000","last_clean":"2024-03-11T21:34:43.408135+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:34:43.408135+0000","last_undegraded":"2024-03-11T21:34:43.408135+0000","last_fullsized":"2024-03-11T21:34:43.408135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:39.970169+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.970169+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":852,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395107+0000","last_change":"2024-03-11T21:34:35.593537+0000","last_active":"2024-03-11T21:34:47.395107+0000","last_peered":"2024-03-11T21:34:47.395107+0000","last_clean":"2024-03-11T21:34:47.395107+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:34:47.395107+0000","last_undegraded":"2024-03-11T21:34:47.395107+0000","last_fullsized":"2024-03-11T21:34:47.395107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.592784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.592784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:50.366116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":704,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.414214+0000","last_change":"2024-03-11T21:34:43.228208+0000","last_active":"2024-03-11T21:34:44.414214+0000","last_peered":"2024-03-11T21:34:44.414214+0000","last_clean":"2024-03-11T21:34:44.414214+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:34:44.414214+0000","last_undegraded":"2024-03-11T21:34:44.414214+0000","last_fullsized":"2024-03-11T21:34:44.414214+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.227861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.227861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.622588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":859,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.357341+0000","last_change":"2024-03-11T21:34:45.933759+0000","last_active":"2024-03-11T21:34:47.357341+0000","last_peered":"2024-03-11T21:34:47.357341+0000","last_clean":"2024-03-11T21:34:47.357341+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:34:47.357341+0000","last_undegraded":"2024-03-11T21:34:47.357341+0000","last_fullsized":"2024-03-11T21:34:47.357341+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.933659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.933659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.191761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":554,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-11T21:34:44.412553+0000","last_change":"2024-03-11T21:33:22.998783+0000","last_active":"2024-03-11T21:34:44.412553+0000","last_peered":"2024-03-11T21:34:44.412553+0000","last_clean":"2024-03-11T21:34:44.412553+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:34:44.412553+0000","last_undegraded":"2024-03-11T21:34:44.412553+0000","last_fullsized":"2024-03-11T21:34:44.412553+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":878,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.395280+0000","last_change":"2024-03-11T21:33:17.733152+0000","last_active":"2024-03-11T21:34:47.395280+0000","last_peered":"2024-03-11T21:34:47.395280+0000","last_clean":"2024-03-11T21:34:47.395280+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:34:47.395280+0000","last_undegraded":"2024-03-11T21:34:47.395280+0000","last_fullsized":"2024-03-11T21:34:47.395280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:17.733068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.733068+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":856,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-11T21:34:43.408261+0000","last_change":"2024-03-11T21:33:16.013881+0000","last_active":"2024-03-11T21:34:43.408261+0000","last_peered":"2024-03-11T21:34:43.408261+0000","last_clean":"2024-03-11T21:34:43.408261+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:34:43.408261+0000","last_undegraded":"2024-03-11T21:34:43.408261+0000","last_fullsized":"2024-03-11T21:34:43.408261+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:33:16.013752+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:33:16.013752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":855,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.356614+0000","last_change":"2024-03-11T21:31:58.673043+0000","last_active":"2024-03-11T21:34:47.356614+0000","last_peered":"2024-03-11T21:34:47.356614+0000","last_clean":"2024-03-11T21:34:47.356614+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:34:47.356614+0000","last_undegraded":"2024-03-11T21:34:47.356614+0000","last_fullsized":"2024-03-11T21:34:47.356614+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.672796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.672796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":831,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-11T21:34:47.394518+0000","last_change":"2024-03-11T21:32:10.686985+0000","last_active":"2024-03-11T21:34:47.394518+0000","last_peered":"2024-03-11T21:34:47.394518+0000","last_clean":"2024-03-11T21:34:47.394518+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:34:47.394518+0000","last_undegraded":"2024-03-11T21:34:47.394518+0000","last_fullsized":"2024-03-11T21:34:47.394518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.686767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.686767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":287,"num_pg":32,"stat_sum":{"num_bytes":6,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195793,"num_objects":161,"num_object_clones":4,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":597,"num_write_kb":12686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26480640,"data_stored":24587373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1865,"ondisk_log_size":1865,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":684,"num_read_kb":679,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841688,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84140,"kb_used_data":12684,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661012,"statfs":{"total":95995035648,"available":95908876288,"internally_reserved":0,"allocated":12988416,"data_stored":9711811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168737,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87908,"kb_used_data":12304,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657244,"statfs":{"total":95995035648,"available":95905017856,"internally_reserved":0,"allocated":12599296,"data_stored":9504793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352359,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48248,"kb_used_data":7820,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93696904,"statfs":{"total":95995035648,"available":95945629696,"internally_reserved":0,"allocated":8007680,"data_stored":4893655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87792,"kb_used_data":16336,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93657360,"statfs":{"total":95995035648,"available":95905136640,"internally_reserved":0,"allocated":16728064,"data_stored":13659061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378827,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75236,"kb_used_data":7940,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669916,"statfs":{"total":95995035648,"available":95917993984,"internally_reserved":0,"allocated":8130560,"data_stored":4925296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117183,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71048,"kb_used_data":7912,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674104,"statfs":{"total":95995035648,"available":95922282496,"internally_reserved":0,"allocated":8101888,"data_stored":4924751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855541,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83372,"kb_used_data":11916,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661780,"statfs":{"total":95995035648,"available":95909662720,"internally_reserved":0,"allocated":12201984,"data_stored":8926353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495787,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75080,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670072,"statfs":{"total":95995035648,"available":95918153728,"internally_reserved":0,"allocated":12230656,"data_stored":9118896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8359936,"data_stored":8194715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4000747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4386816,"data_stored":4194771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4000406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4195019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:34:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.432 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:52.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:52.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.789330+0000 osd.5 (osd.5) 283 : cluster 0 3.1d scrub starts 2024-03-11T21:34:52.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: cluster 2024-03-11T21:34:50.793803+0000 osd.5 (osd.5) 284 : cluster 0 3.1d scrub ok 2024-03-11T21:34:52.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: cluster 2024-03-11T21:34:51.088148+0000 osd.0 (osd.0) 271 : cluster 0 9.1c scrub starts 2024-03-11T21:34:52.538 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: cluster 2024-03-11T21:34:51.090578+0000 osd.0 (osd.0) 272 : cluster 0 9.1c scrub ok 2024-03-11T21:34:52.539 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: cluster 2024-03-11T21:34:51.399224+0000 mgr.y (mgr.14550) 473 : cluster 0 pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:34:52.539 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:52 smithi143 bash[21512]: audit 2024-03-11T21:34:51.464038+0000 mon.c (mon.2) 965 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.561 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:52.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:52.685 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:52.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.789330+0000 osd.5 (osd.5) 283 : cluster 0 3.1d scrub starts 2024-03-11T21:34:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: cluster 2024-03-11T21:34:50.793803+0000 osd.5 (osd.5) 284 : cluster 0 3.1d scrub ok 2024-03-11T21:34:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: cluster 2024-03-11T21:34:51.088148+0000 osd.0 (osd.0) 271 : cluster 0 9.1c scrub starts 2024-03-11T21:34:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: cluster 2024-03-11T21:34:51.090578+0000 osd.0 (osd.0) 272 : cluster 0 9.1c scrub ok 2024-03-11T21:34:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: cluster 2024-03-11T21:34:51.399224+0000 mgr.y (mgr.14550) 473 : cluster 0 pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:34:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:52 smithi203 bash[23654]: audit 2024-03-11T21:34:51.464038+0000 mon.c (mon.2) 965 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- 172.21.15.143:0/37260922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3806a050 msgr2=0x7f1a3806a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- 172.21.15.143:0/37260922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3806a050 0x7f1a3806a4d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00b0a0 tx=0x7f1a2c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- 172.21.15.143:0/37260922 shutdown_connections 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- 172.21.15.143:0/37260922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a38104880 0x7f1a38104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- 172.21.15.143:0/37260922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38106d60 0x7f1a38069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- 172.21.15.143:0/37260922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3806a050 0x7f1a3806a4d0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- 172.21.15.143:0/37260922 >> 172.21.15.143:0/37260922 conn(0x7f1a38076d80 msgr2=0x7f1a38077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- 172.21.15.143:0/37260922 shutdown_connections 2024-03-11T21:34:52.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- 172.21.15.143:0/37260922 wait complete. 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 librados: starting msgr at 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 librados: starting objecter 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 Processor -- start 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- start start 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 librados: setting wanted keys 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 librados: calling monclient init 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 0x7f1a3819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 0x7f1a381a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38117bf0 con 0x7f1a38106d60 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38117a70 con 0x7f1a38104880 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a40291700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38117d70 con 0x7f1a3806a050 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a3e02d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a3e82e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 0x7f1a381a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:52.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a3e02d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37810/0 (socket says 172.21.15.143:37810) 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.811+0000 7f1a3e02d700 1 -- 172.21.15.143:0/467053396 learned_addr learned my addr 172.21.15.143:0/467053396 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3d82c700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 0x7f1a3819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 -- 172.21.15.143:0/467053396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 msgr2=0x7f1a3819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 0x7f1a3819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 -- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 msgr2=0x7f1a381a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 0x7f1a381a2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 -- 172.21.15.143:0/467053396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a381a3220 con 0x7f1a3806a050 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3d82c700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 0x7f1a3819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e82e700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 0x7f1a381a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3e02d700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000d950 tx=0x7f1a3000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 10 client.?.objecter ms_handle_connect 0x7f1a3806a050 2024-03-11T21:34:52.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30016070 con 0x7f1a3806a050 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1a30009dc0 con 0x7f1a3806a050 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a381a3510 con 0x7f1a3806a050 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3000f3c0 con 0x7f1a3806a050 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a40291700 10 client.41654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a381aad90 con 0x7f1a3806a050 2024-03-11T21:34:52.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a40291700 1 librados: init done 2024-03-11T21:34:52.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:52.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3804faa0 con 0x7f1a3806a050 2024-03-11T21:34:52.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1a30019070 con 0x7f1a3806a050 2024-03-11T21:34:52.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 0x7f1a20071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(480..480 src has 87..480) v4 ==== 9657+0+0 (secure 0 0 0) 0x7f1a3008e650 con 0x7f1a3806a050 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 10 client.41654.objecter ms_dispatch 0x7f1a38001550 osd_map(480..480 src has 87..480) v4 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 3 client.41654.objecter handle_osd_map got epochs [480,480] > 0 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 3 client.41654.objecter handle_osd_map decoding full epoch 480 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3d82c700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 0x7f1a20071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 20 client.41654.objecter dump_active .. 0 homeless 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a377fe700 10 client.41654.objecter ms_handle_connect 0x7f1a2006f820 2024-03-11T21:34:52.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.815+0000 7f1a3d82c700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 0x7f1a20071ce0 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009e60 tx=0x7f1a2c00b700 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:34:52.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:52.819+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1a3005b280 con 0x7f1a3806a050 2024-03-11T21:34:52.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:52.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:52.950 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:53.002 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:53.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.047+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1a381aaf60 con 0x7f1a3806a050 2024-03-11T21:34:53.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.073 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:53.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:53.199 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:53.302 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.328 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:53.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.403+0000 7f1a377fe700 1 -- 172.21.15.143:0/467053396 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v481) v1 ==== 101+0+0 (secure 0 0 0) 0x7f1a3005ecd0 con 0x7f1a3806a050 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 10 librados: watch_flush enter 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 10 librados: watch_flush exit 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 20 client.41654.objecter shutdown clearing up homeless session... 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 10 client.41654.objecter successfully canceled tick 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 msgr2=0x7f1a20071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.407+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 0x7f1a20071ce0 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009e60 tx=0x7f1a2c00b700 comp rx=0 tx=0).stop 2024-03-11T21:34:53.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 msgr2=0x7f1a3819e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000d950 tx=0x7f1a3000dc60 comp rx=0 tx=0).stop 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 shutdown_connections 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1a2006f820 0x7f1a20071ce0 secure :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009e60 tx=0x7f1a2c00b700 comp rx=0 tx=0).stop 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a38106d60 0x7f1a381a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3806a050 0x7f1a3819e210 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:53.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 --2- 172.21.15.143:0/467053396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a38104880 0x7f1a3819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:34:53.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 >> 172.21.15.143:0/467053396 conn(0x7f1a38076d80 msgr2=0x7f1a38103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:34:53.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 shutdown_connections 2024-03-11T21:34:53.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 -- 172.21.15.143:0/467053396 wait complete. 2024-03-11T21:34:53.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:34:53.411+0000 7f1a40291700 1 librados: shutdown 2024-03-11T21:34:53.386 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:34:53.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:53.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:53.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.043648+0000 osd.0 (osd.0) 273 : cluster 0 5.3 scrub starts 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.046339+0000 osd.0 (osd.0) 274 : cluster 0 5.3 scrub ok 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.097054+0000 osd.1 (osd.1) 83 : cluster 0 39.17 scrub starts 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.102248+0000 osd.1 (osd.1) 84 : cluster 0 39.17 scrub ok 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.294502+0000 osd.6 (osd.6) 209 : cluster 0 5.1f scrub starts 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: cluster 2024-03-11T21:34:52.297767+0000 osd.6 (osd.6) 210 : cluster 0 5.1f scrub ok 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: audit 2024-03-11T21:34:53.051843+0000 mon.c (mon.2) 966 : audit 1 from='client.? 172.21.15.143:0/467053396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.588 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[21512]: audit 2024-03-11T21:34:53.052221+0000 mon.a (mon.0) 2788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.043648+0000 osd.0 (osd.0) 273 : cluster 0 5.3 scrub starts 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.046339+0000 osd.0 (osd.0) 274 : cluster 0 5.3 scrub ok 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.097054+0000 osd.1 (osd.1) 83 : cluster 0 39.17 scrub starts 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.102248+0000 osd.1 (osd.1) 84 : cluster 0 39.17 scrub ok 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.294502+0000 osd.6 (osd.6) 209 : cluster 0 5.1f scrub starts 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: cluster 2024-03-11T21:34:52.297767+0000 osd.6 (osd.6) 210 : cluster 0 5.1f scrub ok 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: audit 2024-03-11T21:34:53.051843+0000 mon.c (mon.2) 966 : audit 1 from='client.? 172.21.15.143:0/467053396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.589 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:53 smithi143 bash[29284]: audit 2024-03-11T21:34:53.052221+0000 mon.a (mon.0) 2788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:53.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:53.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.043648+0000 osd.0 (osd.0) 273 : cluster 0 5.3 scrub starts 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.046339+0000 osd.0 (osd.0) 274 : cluster 0 5.3 scrub ok 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.097054+0000 osd.1 (osd.1) 83 : cluster 0 39.17 scrub starts 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.102248+0000 osd.1 (osd.1) 84 : cluster 0 39.17 scrub ok 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.294502+0000 osd.6 (osd.6) 209 : cluster 0 5.1f scrub starts 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: cluster 2024-03-11T21:34:52.297767+0000 osd.6 (osd.6) 210 : cluster 0 5.1f scrub ok 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: audit 2024-03-11T21:34:53.051843+0000 mon.c (mon.2) 966 : audit 1 from='client.? 172.21.15.143:0/467053396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:53 smithi203 bash[23654]: audit 2024-03-11T21:34:53.052221+0000 mon.a (mon.0) 2788 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:53.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:53.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:53.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:53.961 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:54.064 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:54.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:54.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:54.221 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:54.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:54.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:54.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.087566+0000 osd.0 (osd.0) 275 : cluster 0 3.6 scrub starts 2024-03-11T21:34:54.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.090143+0000 osd.0 (osd.0) 276 : cluster 0 3.6 scrub ok 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.119619+0000 osd.4 (osd.4) 245 : cluster 0 5.18 scrub starts 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.121685+0000 osd.4 (osd.4) 246 : cluster 0 5.18 scrub ok 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.399528+0000 mon.a (mon.0) 2789 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.400432+0000 mgr.y (mgr.14550) 474 : cluster 0 pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: audit 2024-03-11T21:34:53.401407+0000 mon.a (mon.0) 2790 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.405177+0000 mon.a (mon.0) 2791 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.462074+0000 osd.2 (osd.2) 233 : cluster 0 118.5 scrub starts 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.464416+0000 osd.2 (osd.2) 234 : cluster 0 118.5 scrub ok 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.518757+0000 osd.3 (osd.3) 281 : cluster 0 9.19 scrub starts 2024-03-11T21:34:54.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:54 smithi203 bash[23654]: cluster 2024-03-11T21:34:53.520338+0000 osd.3 (osd.3) 282 : cluster 0 9.19 scrub ok 2024-03-11T21:34:54.483 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:54.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:34:54.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:54.597 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.087566+0000 osd.0 (osd.0) 275 : cluster 0 3.6 scrub starts 2024-03-11T21:34:54.597 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.090143+0000 osd.0 (osd.0) 276 : cluster 0 3.6 scrub ok 2024-03-11T21:34:54.597 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.119619+0000 osd.4 (osd.4) 245 : cluster 0 5.18 scrub starts 2024-03-11T21:34:54.597 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.121685+0000 osd.4 (osd.4) 246 : cluster 0 5.18 scrub ok 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.399528+0000 mon.a (mon.0) 2789 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.400432+0000 mgr.y (mgr.14550) 474 : cluster 0 pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: audit 2024-03-11T21:34:53.401407+0000 mon.a (mon.0) 2790 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.405177+0000 mon.a (mon.0) 2791 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.462074+0000 osd.2 (osd.2) 233 : cluster 0 118.5 scrub starts 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.464416+0000 osd.2 (osd.2) 234 : cluster 0 118.5 scrub ok 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.518757+0000 osd.3 (osd.3) 281 : cluster 0 9.19 scrub starts 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[21512]: cluster 2024-03-11T21:34:53.520338+0000 osd.3 (osd.3) 282 : cluster 0 9.19 scrub ok 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.087566+0000 osd.0 (osd.0) 275 : cluster 0 3.6 scrub starts 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.090143+0000 osd.0 (osd.0) 276 : cluster 0 3.6 scrub ok 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.119619+0000 osd.4 (osd.4) 245 : cluster 0 5.18 scrub starts 2024-03-11T21:34:54.598 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.121685+0000 osd.4 (osd.4) 246 : cluster 0 5.18 scrub ok 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.399528+0000 mon.a (mon.0) 2789 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.400432+0000 mgr.y (mgr.14550) 474 : cluster 0 pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: audit 2024-03-11T21:34:53.401407+0000 mon.a (mon.0) 2790 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.405177+0000 mon.a (mon.0) 2791 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.462074+0000 osd.2 (osd.2) 233 : cluster 0 118.5 scrub starts 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.464416+0000 osd.2 (osd.2) 234 : cluster 0 118.5 scrub ok 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.518757+0000 osd.3 (osd.3) 281 : cluster 0 9.19 scrub starts 2024-03-11T21:34:54.599 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:54 smithi143 bash[29284]: cluster 2024-03-11T21:34:53.520338+0000 osd.3 (osd.3) 282 : cluster 0 9.19 scrub ok 2024-03-11T21:34:54.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:54.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:54.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:54.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:54.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:54.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:55.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:55.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:55.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:55.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:34:55.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:55.516 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:55.542 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.318457+0000 osd.6 (osd.6) 211 : cluster 0 39.8 scrub starts 2024-03-11T21:34:55.542 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.320994+0000 osd.6 (osd.6) 212 : cluster 0 39.8 scrub ok 2024-03-11T21:34:55.542 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.498780+0000 osd.3 (osd.3) 283 : cluster 0 9.8 scrub starts 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.501174+0000 osd.3 (osd.3) 284 : cluster 0 9.8 scrub ok 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.076412+0000 osd.7 (osd.7) 285 : cluster 0 118.b scrub starts 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.080469+0000 osd.7 (osd.7) 286 : cluster 0 118.b scrub ok 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.318457+0000 osd.6 (osd.6) 211 : cluster 0 39.8 scrub starts 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.320994+0000 osd.6 (osd.6) 212 : cluster 0 39.8 scrub ok 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.498780+0000 osd.3 (osd.3) 283 : cluster 0 9.8 scrub starts 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.501174+0000 osd.3 (osd.3) 284 : cluster 0 9.8 scrub ok 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.076412+0000 osd.7 (osd.7) 285 : cluster 0 118.b scrub starts 2024-03-11T21:34:55.543 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:55 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.080469+0000 osd.7 (osd.7) 286 : cluster 0 118.b scrub ok 2024-03-11T21:34:55.619 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.759 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.318457+0000 osd.6 (osd.6) 211 : cluster 0 39.8 scrub starts 2024-03-11T21:34:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.320994+0000 osd.6 (osd.6) 212 : cluster 0 39.8 scrub ok 2024-03-11T21:34:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.498780+0000 osd.3 (osd.3) 283 : cluster 0 9.8 scrub starts 2024-03-11T21:34:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.501174+0000 osd.3 (osd.3) 284 : cluster 0 9.8 scrub ok 2024-03-11T21:34:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.076412+0000 osd.7 (osd.7) 285 : cluster 0 118.b scrub starts 2024-03-11T21:34:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:55 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.080469+0000 osd.7 (osd.7) 286 : cluster 0 118.b scrub ok 2024-03-11T21:34:55.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:55.892 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:55.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:56.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:56.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:56.150 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:56.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:56.279 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:56.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.413 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:56.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:56.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:56.657 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.759049+0000 osd.5 (osd.5) 285 : cluster 0 118.8 scrub starts 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:54.761320+0000 osd.5 (osd.5) 286 : cluster 0 118.8 scrub ok 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.128285+0000 osd.1 (osd.1) 85 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.143596+0000 osd.1 (osd.1) 86 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.401124+0000 mgr.y (mgr.14550) 475 : cluster 0 pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 510 B/s wr, 2 op/s 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[21512]: cluster 2024-03-11T21:34:56.352246+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.759049+0000 osd.5 (osd.5) 285 : cluster 0 118.8 scrub starts 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:54.761320+0000 osd.5 (osd.5) 286 : cluster 0 118.8 scrub ok 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.128285+0000 osd.1 (osd.1) 85 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.143596+0000 osd.1 (osd.1) 86 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:34:56.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.401124+0000 mgr.y (mgr.14550) 475 : cluster 0 pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 510 B/s wr, 2 op/s 2024-03-11T21:34:56.659 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:56 smithi143 bash[29284]: cluster 2024-03-11T21:34:56.352246+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:34:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.675 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:56.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.759049+0000 osd.5 (osd.5) 285 : cluster 0 118.8 scrub starts 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:54.761320+0000 osd.5 (osd.5) 286 : cluster 0 118.8 scrub ok 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.128285+0000 osd.1 (osd.1) 85 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.143596+0000 osd.1 (osd.1) 86 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.401124+0000 mgr.y (mgr.14550) 475 : cluster 0 pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 510 B/s wr, 2 op/s 2024-03-11T21:34:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:56 smithi203 bash[23654]: cluster 2024-03-11T21:34:56.352246+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:34:56.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:56.805 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:56.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:57.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.065 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:57.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:57.195 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:57.299 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.327 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:57.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.454 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:57.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.711609+0000 osd.5 (osd.5) 287 : cluster 0 5.15 scrub starts 2024-03-11T21:34:57.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:55.714200+0000 osd.5 (osd.5) 288 : cluster 0 5.15 scrub ok 2024-03-11T21:34:57.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:56.436002+0000 osd.2 (osd.2) 235 : cluster 0 118.9 deep-scrub starts 2024-03-11T21:34:57.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:56.441717+0000 osd.2 (osd.2) 236 : cluster 0 118.9 deep-scrub ok 2024-03-11T21:34:57.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:56.552312+0000 osd.3 (osd.3) 285 : cluster 0 3.18 scrub starts 2024-03-11T21:34:57.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:57 smithi203 bash[23654]: cluster 2024-03-11T21:34:56.555994+0000 osd.3 (osd.3) 286 : cluster 0 3.18 scrub ok 2024-03-11T21:34:57.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:34:57.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.600 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.711609+0000 osd.5 (osd.5) 287 : cluster 0 5.15 scrub starts 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:55.714200+0000 osd.5 (osd.5) 288 : cluster 0 5.15 scrub ok 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:56.436002+0000 osd.2 (osd.2) 235 : cluster 0 118.9 deep-scrub starts 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:56.441717+0000 osd.2 (osd.2) 236 : cluster 0 118.9 deep-scrub ok 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:56.552312+0000 osd.3 (osd.3) 285 : cluster 0 3.18 scrub starts 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[29284]: cluster 2024-03-11T21:34:56.555994+0000 osd.3 (osd.3) 286 : cluster 0 3.18 scrub ok 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.711609+0000 osd.5 (osd.5) 287 : cluster 0 5.15 scrub starts 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:55.714200+0000 osd.5 (osd.5) 288 : cluster 0 5.15 scrub ok 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:56.436002+0000 osd.2 (osd.2) 235 : cluster 0 118.9 deep-scrub starts 2024-03-11T21:34:57.601 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:56.441717+0000 osd.2 (osd.2) 236 : cluster 0 118.9 deep-scrub ok 2024-03-11T21:34:57.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:56.552312+0000 osd.3 (osd.3) 285 : cluster 0 3.18 scrub starts 2024-03-11T21:34:57.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:57 smithi143 bash[21512]: cluster 2024-03-11T21:34:56.555994+0000 osd.3 (osd.3) 286 : cluster 0 3.18 scrub ok 2024-03-11T21:34:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:57.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.739 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:57.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.867 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:57.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.997 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:58.100 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:58.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:58.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:58.258 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:58.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:58.390 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:58.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:58.521 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:58.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.128020+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub starts 2024-03-11T21:34:58.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.129292+0000 osd.0 (osd.0) 278 : cluster 0 6.0 scrub ok 2024-03-11T21:34:58.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.183824+0000 osd.4 (osd.4) 247 : cluster 0 5.1c scrub starts 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.186499+0000 osd.4 (osd.4) 248 : cluster 0 5.1c scrub ok 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.361828+0000 osd.6 (osd.6) 213 : cluster 0 5.2 scrub starts 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.364044+0000 osd.6 (osd.6) 214 : cluster 0 5.2 scrub ok 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.401721+0000 mgr.y (mgr.14550) 476 : cluster 0 pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.035892+0000 osd.7 (osd.7) 287 : cluster 0 118.1c scrub starts 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.041759+0000 osd.7 (osd.7) 288 : cluster 0 118.1c scrub ok 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: audit 2024-03-11T21:34:58.406092+0000 mon.c (mon.2) 967 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[23654]: audit 2024-03-11T21:34:58.406887+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:58.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:58.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.128020+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub starts 2024-03-11T21:34:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.129292+0000 osd.0 (osd.0) 278 : cluster 0 6.0 scrub ok 2024-03-11T21:34:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.183824+0000 osd.4 (osd.4) 247 : cluster 0 5.1c scrub starts 2024-03-11T21:34:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.186499+0000 osd.4 (osd.4) 248 : cluster 0 5.1c scrub ok 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.361828+0000 osd.6 (osd.6) 213 : cluster 0 5.2 scrub starts 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.364044+0000 osd.6 (osd.6) 214 : cluster 0 5.2 scrub ok 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.401721+0000 mgr.y (mgr.14550) 476 : cluster 0 pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.035892+0000 osd.7 (osd.7) 287 : cluster 0 118.1c scrub starts 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.041759+0000 osd.7 (osd.7) 288 : cluster 0 118.1c scrub ok 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: audit 2024-03-11T21:34:58.406092+0000 mon.c (mon.2) 967 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[21512]: audit 2024-03-11T21:34:58.406887+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.128020+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub starts 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.129292+0000 osd.0 (osd.0) 278 : cluster 0 6.0 scrub ok 2024-03-11T21:34:58.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.183824+0000 osd.4 (osd.4) 247 : cluster 0 5.1c scrub starts 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.186499+0000 osd.4 (osd.4) 248 : cluster 0 5.1c scrub ok 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.361828+0000 osd.6 (osd.6) 213 : cluster 0 5.2 scrub starts 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.364044+0000 osd.6 (osd.6) 214 : cluster 0 5.2 scrub ok 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.401721+0000 mgr.y (mgr.14550) 476 : cluster 0 pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.035892+0000 osd.7 (osd.7) 287 : cluster 0 118.1c scrub starts 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.041759+0000 osd.7 (osd.7) 288 : cluster 0 118.1c scrub ok 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: audit 2024-03-11T21:34:58.406092+0000 mon.c (mon.2) 967 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:58 smithi143 bash[29284]: audit 2024-03-11T21:34:58.406887+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-11T21:34:58.705 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:34:58.705 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:34:58.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:58.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:34:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:34:58.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:58.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:58.907 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:59.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:59.038 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:59.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:59.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:59.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:59.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:59.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:59.681 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:59.682 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.749276+0000 osd.5 (osd.5) 289 : cluster 0 6.7 deep-scrub starts 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:57.751642+0000 osd.5 (osd.5) 290 : cluster 0 6.7 deep-scrub ok 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.089052+0000 osd.0 (osd.0) 279 : cluster 0 39.f scrub starts 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.091657+0000 osd.0 (osd.0) 280 : cluster 0 39.f scrub ok 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.317782+0000 osd.6 (osd.6) 215 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.319452+0000 osd.6 (osd.6) 216 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: audit 2024-03-11T21:34:58.432375+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.439780+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.483399+0000 osd.2 (osd.2) 237 : cluster 0 39.16 scrub starts 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.485045+0000 osd.2 (osd.2) 238 : cluster 0 39.16 scrub ok 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[29284]: audit 2024-03-11T21:34:58.652592+0000 mgr.y (mgr.14550) 477 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:59.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.749276+0000 osd.5 (osd.5) 289 : cluster 0 6.7 deep-scrub starts 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:57.751642+0000 osd.5 (osd.5) 290 : cluster 0 6.7 deep-scrub ok 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.089052+0000 osd.0 (osd.0) 279 : cluster 0 39.f scrub starts 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.091657+0000 osd.0 (osd.0) 280 : cluster 0 39.f scrub ok 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.317782+0000 osd.6 (osd.6) 215 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.319452+0000 osd.6 (osd.6) 216 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: audit 2024-03-11T21:34:58.432375+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.439780+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.483399+0000 osd.2 (osd.2) 237 : cluster 0 39.16 scrub starts 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.485045+0000 osd.2 (osd.2) 238 : cluster 0 39.16 scrub ok 2024-03-11T21:34:59.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:34:59 smithi143 bash[21512]: audit 2024-03-11T21:34:58.652592+0000 mgr.y (mgr.14550) 477 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:59.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.749276+0000 osd.5 (osd.5) 289 : cluster 0 6.7 deep-scrub starts 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:57.751642+0000 osd.5 (osd.5) 290 : cluster 0 6.7 deep-scrub ok 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.089052+0000 osd.0 (osd.0) 279 : cluster 0 39.f scrub starts 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.091657+0000 osd.0 (osd.0) 280 : cluster 0 39.f scrub ok 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.317782+0000 osd.6 (osd.6) 215 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.319452+0000 osd.6 (osd.6) 216 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: audit 2024-03-11T21:34:58.432375+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.439780+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:34:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.483399+0000 osd.2 (osd.2) 237 : cluster 0 39.16 scrub starts 2024-03-11T21:34:59.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.485045+0000 osd.2 (osd.2) 238 : cluster 0 39.16 scrub ok 2024-03-11T21:34:59.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:34:59 smithi203 bash[23654]: audit 2024-03-11T21:34:58.652592+0000 mgr.y (mgr.14550) 477 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:59.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.813 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:59.916 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:59.935 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:00.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:00.065 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:00.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:00.195 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:00.218 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:00.219 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:00.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:00.320 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:00.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.787241+0000 osd.5 (osd.5) 291 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:58.788743+0000 osd.5 (osd.5) 292 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:59.402932+0000 mgr.y (mgr.14550) 478 : cluster 0 pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:59.469867+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:59.517708+0000 osd.3 (osd.3) 287 : cluster 0 39.d scrub starts 2024-03-11T21:35:00.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:00 smithi203 bash[23654]: cluster 2024-03-11T21:34:59.524782+0000 osd.3 (osd.3) 288 : cluster 0 39.d scrub ok 2024-03-11T21:35:00.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:00.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:00.564 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.787241+0000 osd.5 (osd.5) 291 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:58.788743+0000 osd.5 (osd.5) 292 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:59.402932+0000 mgr.y (mgr.14550) 478 : cluster 0 pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:59.469867+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:59.517708+0000 osd.3 (osd.3) 287 : cluster 0 39.d scrub starts 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[29284]: cluster 2024-03-11T21:34:59.524782+0000 osd.3 (osd.3) 288 : cluster 0 39.d scrub ok 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.787241+0000 osd.5 (osd.5) 291 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:58.788743+0000 osd.5 (osd.5) 292 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:59.402932+0000 mgr.y (mgr.14550) 478 : cluster 0 pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:00.565 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:59.469867+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:35:00.566 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:59.517708+0000 osd.3 (osd.3) 287 : cluster 0 39.d scrub starts 2024-03-11T21:35:00.566 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:00 smithi143 bash[21512]: cluster 2024-03-11T21:34:59.524782+0000 osd.3 (osd.3) 288 : cluster 0 39.d scrub ok 2024-03-11T21:35:00.580 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:00.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:00.710 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:00.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:00.838 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:00.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:00.969 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:01.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.098 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:01.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:01.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:01.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:01.354 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:01.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:01.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.405234+0000 osd.6 (osd.6) 217 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:35:01.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.408799+0000 osd.6 (osd.6) 218 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:35:01.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.478473+0000 osd.2 (osd.2) 239 : cluster 0 118.d scrub starts 2024-03-11T21:35:01.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.480342+0000 osd.2 (osd.2) 240 : cluster 0 118.d scrub ok 2024-03-11T21:35:01.468 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.522237+0000 osd.3 (osd.3) 289 : cluster 0 6.1b scrub starts 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[29284]: cluster 2024-03-11T21:35:00.524420+0000 osd.3 (osd.3) 290 : cluster 0 6.1b scrub ok 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.405234+0000 osd.6 (osd.6) 217 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.408799+0000 osd.6 (osd.6) 218 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.478473+0000 osd.2 (osd.2) 239 : cluster 0 118.d scrub starts 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.480342+0000 osd.2 (osd.2) 240 : cluster 0 118.d scrub ok 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.522237+0000 osd.3 (osd.3) 289 : cluster 0 6.1b scrub starts 2024-03-11T21:35:01.469 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:01 smithi143 bash[21512]: cluster 2024-03-11T21:35:00.524420+0000 osd.3 (osd.3) 290 : cluster 0 6.1b scrub ok 2024-03-11T21:35:01.481 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:01.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:01.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.405234+0000 osd.6 (osd.6) 217 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:35:01.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.408799+0000 osd.6 (osd.6) 218 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:35:01.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.478473+0000 osd.2 (osd.2) 239 : cluster 0 118.d scrub starts 2024-03-11T21:35:01.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.480342+0000 osd.2 (osd.2) 240 : cluster 0 118.d scrub ok 2024-03-11T21:35:01.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.522237+0000 osd.3 (osd.3) 289 : cluster 0 6.1b scrub starts 2024-03-11T21:35:01.597 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:01 smithi203 bash[23654]: cluster 2024-03-11T21:35:00.524420+0000 osd.3 (osd.3) 290 : cluster 0 6.1b scrub ok 2024-03-11T21:35:01.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:01.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:01.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.871 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:01.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:02.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:02.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.116 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:02.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:02.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.253 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:02.259 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:02.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:02.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:02.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:02.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.023156+0000 osd.0 (osd.0) 281 : cluster 0 5.1e scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.025587+0000 osd.0 (osd.0) 282 : cluster 0 5.1e scrub ok 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.151318+0000 osd.1 (osd.1) 87 : cluster 0 39.7 scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.154489+0000 osd.1 (osd.1) 88 : cluster 0 39.7 scrub ok 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.193296+0000 osd.4 (osd.4) 249 : cluster 0 6.e scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.196305+0000 osd.4 (osd.4) 250 : cluster 0 6.e scrub ok 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.403631+0000 mgr.y (mgr.14550) 479 : cluster 0 pgmap v722: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.434986+0000 osd.2 (osd.2) 241 : cluster 0 2.e scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.437557+0000 osd.2 (osd.2) 242 : cluster 0 2.e scrub ok 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.531782+0000 osd.3 (osd.3) 291 : cluster 0 3.b scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[21512]: cluster 2024-03-11T21:35:01.534126+0000 osd.3 (osd.3) 292 : cluster 0 3.b scrub ok 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.023156+0000 osd.0 (osd.0) 281 : cluster 0 5.1e scrub starts 2024-03-11T21:35:02.635 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.025587+0000 osd.0 (osd.0) 282 : cluster 0 5.1e scrub ok 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.151318+0000 osd.1 (osd.1) 87 : cluster 0 39.7 scrub starts 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.154489+0000 osd.1 (osd.1) 88 : cluster 0 39.7 scrub ok 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.193296+0000 osd.4 (osd.4) 249 : cluster 0 6.e scrub starts 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.196305+0000 osd.4 (osd.4) 250 : cluster 0 6.e scrub ok 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.403631+0000 mgr.y (mgr.14550) 479 : cluster 0 pgmap v722: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.434986+0000 osd.2 (osd.2) 241 : cluster 0 2.e scrub starts 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.437557+0000 osd.2 (osd.2) 242 : cluster 0 2.e scrub ok 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.531782+0000 osd.3 (osd.3) 291 : cluster 0 3.b scrub starts 2024-03-11T21:35:02.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:02 smithi143 bash[29284]: cluster 2024-03-11T21:35:01.534126+0000 osd.3 (osd.3) 292 : cluster 0 3.b scrub ok 2024-03-11T21:35:02.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:02.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:02.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.023156+0000 osd.0 (osd.0) 281 : cluster 0 5.1e scrub starts 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.025587+0000 osd.0 (osd.0) 282 : cluster 0 5.1e scrub ok 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.151318+0000 osd.1 (osd.1) 87 : cluster 0 39.7 scrub starts 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.154489+0000 osd.1 (osd.1) 88 : cluster 0 39.7 scrub ok 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.193296+0000 osd.4 (osd.4) 249 : cluster 0 6.e scrub starts 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.196305+0000 osd.4 (osd.4) 250 : cluster 0 6.e scrub ok 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.403631+0000 mgr.y (mgr.14550) 479 : cluster 0 pgmap v722: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.434986+0000 osd.2 (osd.2) 241 : cluster 0 2.e scrub starts 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.437557+0000 osd.2 (osd.2) 242 : cluster 0 2.e scrub ok 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.531782+0000 osd.3 (osd.3) 291 : cluster 0 3.b scrub starts 2024-03-11T21:35:02.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:02 smithi203 bash[23654]: cluster 2024-03-11T21:35:01.534126+0000 osd.3 (osd.3) 292 : cluster 0 3.b scrub ok 2024-03-11T21:35:02.780 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:02.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:02.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:03.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:03.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:03.172 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:03.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.305 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:03.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.439 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:03.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.566 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:03.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[21512]: cluster 2024-03-11T21:35:02.010303+0000 osd.0 (osd.0) 283 : cluster 0 6.9 scrub starts 2024-03-11T21:35:03.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[21512]: cluster 2024-03-11T21:35:02.011977+0000 osd.0 (osd.0) 284 : cluster 0 6.9 scrub ok 2024-03-11T21:35:03.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[21512]: cluster 2024-03-11T21:35:02.145415+0000 osd.4 (osd.4) 251 : cluster 0 39.b scrub starts 2024-03-11T21:35:03.655 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[21512]: cluster 2024-03-11T21:35:02.148334+0000 osd.4 (osd.4) 252 : cluster 0 39.b scrub ok 2024-03-11T21:35:03.656 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[29284]: cluster 2024-03-11T21:35:02.010303+0000 osd.0 (osd.0) 283 : cluster 0 6.9 scrub starts 2024-03-11T21:35:03.656 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[29284]: cluster 2024-03-11T21:35:02.011977+0000 osd.0 (osd.0) 284 : cluster 0 6.9 scrub ok 2024-03-11T21:35:03.656 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[29284]: cluster 2024-03-11T21:35:02.145415+0000 osd.4 (osd.4) 251 : cluster 0 39.b scrub starts 2024-03-11T21:35:03.656 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:03 smithi143 bash[29284]: cluster 2024-03-11T21:35:02.148334+0000 osd.4 (osd.4) 252 : cluster 0 39.b scrub ok 2024-03-11T21:35:03.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:03.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:03 smithi203 bash[23654]: cluster 2024-03-11T21:35:02.010303+0000 osd.0 (osd.0) 283 : cluster 0 6.9 scrub starts 2024-03-11T21:35:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:03 smithi203 bash[23654]: cluster 2024-03-11T21:35:02.011977+0000 osd.0 (osd.0) 284 : cluster 0 6.9 scrub ok 2024-03-11T21:35:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:03 smithi203 bash[23654]: cluster 2024-03-11T21:35:02.145415+0000 osd.4 (osd.4) 251 : cluster 0 39.b scrub starts 2024-03-11T21:35:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:03 smithi203 bash[23654]: cluster 2024-03-11T21:35:02.148334+0000 osd.4 (osd.4) 252 : cluster 0 39.b scrub ok 2024-03-11T21:35:03.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:03.827 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:03.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:04.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.089 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:04.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:04.220 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:04.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:04.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:04.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.479 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:04.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:04.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.405716+0000 mgr.y (mgr.14550) 480 : cluster 0 pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.463444+0000 osd.6 (osd.6) 219 : cluster 0 9.18 scrub starts 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.464690+0000 osd.6 (osd.6) 220 : cluster 0 9.18 scrub ok 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.557509+0000 osd.3 (osd.3) 293 : cluster 0 39.e scrub starts 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.560196+0000 osd.3 (osd.3) 294 : cluster 0 39.e scrub ok 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.772131+0000 osd.5 (osd.5) 293 : cluster 0 5.7 scrub starts 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:03.774339+0000 osd.5 (osd.5) 294 : cluster 0 5.7 scrub ok 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.075117+0000 osd.1 (osd.1) 89 : cluster 0 6.4 scrub starts 2024-03-11T21:35:04.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.405716+0000 mgr.y (mgr.14550) 480 : cluster 0 pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.463444+0000 osd.6 (osd.6) 219 : cluster 0 9.18 scrub starts 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.464690+0000 osd.6 (osd.6) 220 : cluster 0 9.18 scrub ok 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.557509+0000 osd.3 (osd.3) 293 : cluster 0 39.e scrub starts 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.560196+0000 osd.3 (osd.3) 294 : cluster 0 39.e scrub ok 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.772131+0000 osd.5 (osd.5) 293 : cluster 0 5.7 scrub starts 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:03.774339+0000 osd.5 (osd.5) 294 : cluster 0 5.7 scrub ok 2024-03-11T21:35:04.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:04 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.075117+0000 osd.1 (osd.1) 89 : cluster 0 6.4 scrub starts 2024-03-11T21:35:04.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.405716+0000 mgr.y (mgr.14550) 480 : cluster 0 pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.463444+0000 osd.6 (osd.6) 219 : cluster 0 9.18 scrub starts 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.464690+0000 osd.6 (osd.6) 220 : cluster 0 9.18 scrub ok 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.557509+0000 osd.3 (osd.3) 293 : cluster 0 39.e scrub starts 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.560196+0000 osd.3 (osd.3) 294 : cluster 0 39.e scrub ok 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.772131+0000 osd.5 (osd.5) 293 : cluster 0 5.7 scrub starts 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:03.774339+0000 osd.5 (osd.5) 294 : cluster 0 5.7 scrub ok 2024-03-11T21:35:04.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:04 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.075117+0000 osd.1 (osd.1) 89 : cluster 0 6.4 scrub starts 2024-03-11T21:35:04.743 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:04.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:04.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:04.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:04.985 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:05.002 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:05.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:05.133 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:05.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.262 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:05.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:05.398 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:05.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:05.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:05.660 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:05.762 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.076785+0000 osd.1 (osd.1) 90 : cluster 0 6.4 scrub ok 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.181166+0000 osd.4 (osd.4) 253 : cluster 0 9.10 scrub starts 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.183134+0000 osd.4 (osd.4) 254 : cluster 0 9.10 scrub ok 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.442532+0000 osd.2 (osd.2) 243 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[29284]: cluster 2024-03-11T21:35:04.446149+0000 osd.2 (osd.2) 244 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.076785+0000 osd.1 (osd.1) 90 : cluster 0 6.4 scrub ok 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.181166+0000 osd.4 (osd.4) 253 : cluster 0 9.10 scrub starts 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.183134+0000 osd.4 (osd.4) 254 : cluster 0 9.10 scrub ok 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.442532+0000 osd.2 (osd.2) 243 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:35:05.763 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:05 smithi143 bash[21512]: cluster 2024-03-11T21:35:04.446149+0000 osd.2 (osd.2) 244 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:35:05.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:05 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.076785+0000 osd.1 (osd.1) 90 : cluster 0 6.4 scrub ok 2024-03-11T21:35:05.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:05 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.181166+0000 osd.4 (osd.4) 253 : cluster 0 9.10 scrub starts 2024-03-11T21:35:05.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:05 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.183134+0000 osd.4 (osd.4) 254 : cluster 0 9.10 scrub ok 2024-03-11T21:35:05.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:05 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.442532+0000 osd.2 (osd.2) 243 : cluster 0 118.15 deep-scrub starts 2024-03-11T21:35:05.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:05 smithi203 bash[23654]: cluster 2024-03-11T21:35:04.446149+0000 osd.2 (osd.2) 244 : cluster 0 118.15 deep-scrub ok 2024-03-11T21:35:05.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:05.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:05.924 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:06.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:06.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:06.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:06.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:06.412 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:06.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.209244+0000 osd.4 (osd.4) 255 : cluster 0 39.15 deep-scrub starts 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.211624+0000 osd.4 (osd.4) 256 : cluster 0 39.15 deep-scrub ok 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.406535+0000 mgr.y (mgr.14550) 481 : cluster 0 pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.447344+0000 osd.6 (osd.6) 221 : cluster 0 39.19 scrub starts 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.450306+0000 osd.6 (osd.6) 222 : cluster 0 39.19 scrub ok 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.454272+0000 osd.2 (osd.2) 245 : cluster 0 6.14 scrub starts 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.456046+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub ok 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.739838+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:05.741818+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-03-11T21:35:06.551 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[21512]: cluster 2024-03-11T21:35:06.357217+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.209244+0000 osd.4 (osd.4) 255 : cluster 0 39.15 deep-scrub starts 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.211624+0000 osd.4 (osd.4) 256 : cluster 0 39.15 deep-scrub ok 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.406535+0000 mgr.y (mgr.14550) 481 : cluster 0 pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.447344+0000 osd.6 (osd.6) 221 : cluster 0 39.19 scrub starts 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.450306+0000 osd.6 (osd.6) 222 : cluster 0 39.19 scrub ok 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.454272+0000 osd.2 (osd.2) 245 : cluster 0 6.14 scrub starts 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.456046+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub ok 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.739838+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:05.741818+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-03-11T21:35:06.552 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:06 smithi143 bash[29284]: cluster 2024-03-11T21:35:06.357217+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:35:06.563 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:06.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:06.694 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:06.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.209244+0000 osd.4 (osd.4) 255 : cluster 0 39.15 deep-scrub starts 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.211624+0000 osd.4 (osd.4) 256 : cluster 0 39.15 deep-scrub ok 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.406535+0000 mgr.y (mgr.14550) 481 : cluster 0 pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.447344+0000 osd.6 (osd.6) 221 : cluster 0 39.19 scrub starts 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.450306+0000 osd.6 (osd.6) 222 : cluster 0 39.19 scrub ok 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.454272+0000 osd.2 (osd.2) 245 : cluster 0 6.14 scrub starts 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.456046+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub ok 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.739838+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:05.741818+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-03-11T21:35:06.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:06 smithi203 bash[23654]: cluster 2024-03-11T21:35:06.357217+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:35:06.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:06.825 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:06.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:07.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:07.086 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:07.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:07.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:07.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:07.347 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:07.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:07.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:07.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:07.592 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: cluster 2024-03-11T21:35:06.469593+0000 osd.6 (osd.6) 223 : cluster 0 39.1d scrub starts 2024-03-11T21:35:07.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: cluster 2024-03-11T21:35:06.473512+0000 osd.6 (osd.6) 224 : cluster 0 39.1d scrub ok 2024-03-11T21:35:07.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: audit 2024-03-11T21:35:06.500857+0000 mon.a (mon.0) 2798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:07.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: audit 2024-03-11T21:35:06.502359+0000 mon.c (mon.2) 968 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: cluster 2024-03-11T21:35:07.135960+0000 osd.7 (osd.7) 289 : cluster 0 118.12 scrub starts 2024-03-11T21:35:07.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:07 smithi203 bash[23654]: cluster 2024-03-11T21:35:07.139118+0000 osd.7 (osd.7) 290 : cluster 0 118.12 scrub ok 2024-03-11T21:35:07.610 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:07.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:07.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:07.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: cluster 2024-03-11T21:35:06.469593+0000 osd.6 (osd.6) 223 : cluster 0 39.1d scrub starts 2024-03-11T21:35:07.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: cluster 2024-03-11T21:35:06.473512+0000 osd.6 (osd.6) 224 : cluster 0 39.1d scrub ok 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: audit 2024-03-11T21:35:06.500857+0000 mon.a (mon.0) 2798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: audit 2024-03-11T21:35:06.502359+0000 mon.c (mon.2) 968 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: cluster 2024-03-11T21:35:07.135960+0000 osd.7 (osd.7) 289 : cluster 0 118.12 scrub starts 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[21512]: cluster 2024-03-11T21:35:07.139118+0000 osd.7 (osd.7) 290 : cluster 0 118.12 scrub ok 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: cluster 2024-03-11T21:35:06.469593+0000 osd.6 (osd.6) 223 : cluster 0 39.1d scrub starts 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: cluster 2024-03-11T21:35:06.473512+0000 osd.6 (osd.6) 224 : cluster 0 39.1d scrub ok 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: audit 2024-03-11T21:35:06.500857+0000 mon.a (mon.0) 2798 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: audit 2024-03-11T21:35:06.502359+0000 mon.c (mon.2) 968 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: cluster 2024-03-11T21:35:07.135960+0000 osd.7 (osd.7) 289 : cluster 0 118.12 scrub starts 2024-03-11T21:35:07.833 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:07 smithi143 bash[29284]: cluster 2024-03-11T21:35:07.139118+0000 osd.7 (osd.7) 290 : cluster 0 118.12 scrub ok 2024-03-11T21:35:07.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:07.869 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:07.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:08.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:08.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:08.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.267 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:08.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.398 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:08.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.512 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[23654]: cluster 2024-03-11T21:35:07.188332+0000 osd.4 (osd.4) 257 : cluster 0 5.e scrub starts 2024-03-11T21:35:08.512 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[23654]: cluster 2024-03-11T21:35:07.191757+0000 osd.4 (osd.4) 258 : cluster 0 5.e scrub ok 2024-03-11T21:35:08.512 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[23654]: cluster 2024-03-11T21:35:07.407399+0000 mgr.y (mgr.14550) 482 : cluster 0 pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:08.512 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[23654]: cluster 2024-03-11T21:35:08.141541+0000 osd.7 (osd.7) 291 : cluster 0 118.11 scrub starts 2024-03-11T21:35:08.512 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[23654]: cluster 2024-03-11T21:35:08.144904+0000 osd.7 (osd.7) 292 : cluster 0 118.11 scrub ok 2024-03-11T21:35:08.527 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:08.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.656 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:08.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:08.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:08.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- 172.21.15.143:0/86236448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 msgr2=0x7f3004110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- 172.21.15.143:0/86236448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 0x7f3004110ea0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000009ca0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- 172.21.15.143:0/86236448 shutdown_connections 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- 172.21.15.143:0/86236448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 0x7f3004110ea0 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- 172.21.15.143:0/86236448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 0x7f3004100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- 172.21.15.143:0/86236448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f3004100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- 172.21.15.143:0/86236448 >> 172.21.15.143:0/86236448 conn(0x7f30040772e0 msgr2=0x7f30040fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- 172.21.15.143:0/86236448 shutdown_connections 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- 172.21.15.143:0/86236448 wait complete. 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 librados: starting msgr at 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 librados: starting objecter 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 Processor -- start 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- start start 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 librados: setting wanted keys 2024-03-11T21:35:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 librados: calling monclient init 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 0x7f300419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 0x7f300419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004113a50 con 0x7f3004100870 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30041138d0 con 0x7f3004102f20 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.827+0000 7f300b51a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3004113bd0 con 0x7f300410eab0 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47666/0 (socket says 172.21.15.143:47666) 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3009ab7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 0x7f300419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 -- 172.21.15.143:0/2691067693 learned_addr learned my addr 172.21.15.143:0/2691067693 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 0x7f300419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 msgr2=0x7f300419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300410eab0 0x7f300419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 msgr2=0x7f300419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 0x7f300419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300419f080 con 0x7f3004100870 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3004102f20 0x7f300419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00ada0 tx=0x7f2ffc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 10 client.?.objecter ms_handle_connect 0x7f3004100870 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc017020 con 0x7f3004100870 2024-03-11T21:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300419f370 con 0x7f3004100870 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2ffc00ce00 con 0x7f3004100870 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc005280 con 0x7f3004100870 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 10 client.30459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30041a6bf0 con 0x7f3004100870 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 librados: init done 2024-03-11T21:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2fe8005100 con 0x7f3004100870 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2ffc00c980 con 0x7f3004100870 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 0x7f2ff0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(485..485 src has 87..485) v4 ==== 9686+0+0 (secure 0 0 0) 0x7f2ffc08ec50 con 0x7f3004100870 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 osd_map(485..485 src has 87..485) v4 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 3 client.30459.objecter handle_osd_map got epochs [485,485] > 0 2024-03-11T21:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 3 client.30459.objecter handle_osd_map decoding full epoch 485 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 0x7f2ff0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 20 client.30459.objecter dump_active .. 0 homeless 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v485) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2ffc021020 con 0x7f3004100870 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 0x7f2ff0071ce0 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f30040ffce0 tx=0x7f2ff4009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2ff006f820 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 10 client.30459.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 1 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f2fe8004730 0x7f2fe8004b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 1 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 0 1 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 4 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 10 client.30459.objecter _send_command 1 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe8008b70 con 0x7f2fe8004730 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.831+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f2fe8004730 0x7f2fe8004b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f2fe8004730 0x7f2fe8004b50 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe8004730 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe8008b70 con 0x7f2fe8004730 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 1 = system:0 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 4 2024-03-11T21:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.835+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 0 1 2024-03-11T21:35:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 10 client.30459.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 2 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 2 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80053e0 osd=0 3 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 0 2 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 4 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 10 client.30459.objecter _send_command 2 2024-03-11T21:35:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.851+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe80041e0 con 0x7f2fe8004730 2024-03-11T21:35:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe80041e0 con 0x7f2fe8004730 2024-03-11T21:35:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 4 2024-03-11T21:35:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 0 2 2024-03-11T21:35:08.828 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:35:08.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 10 client.30459.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f2fe8002750 0x7f2fe8002bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 1 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 4 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 10 client.30459.objecter _send_command 3 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe80089a0 con 0x7f2fe8002750 2024-03-11T21:35:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f2fe8002750 0x7f2fe8002bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f2fe8002750 0x7f2fe8002bc0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe8002750 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe80089a0 con 0x7f2fe8002750 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 3 = system:0 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 4 2024-03-11T21:35:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.855+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 1 3 2024-03-11T21:35:08.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 10 client.30459.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 4 2024-03-11T21:35:08.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 4 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80081c0 osd=1 3 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 1 4 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 4 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 10 client.30459.objecter _send_command 4 2024-03-11T21:35:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.871+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8007ca0 con 0x7f2fe8002750 2024-03-11T21:35:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8007ca0 con 0x7f2fe8002750 2024-03-11T21:35:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 4 2024-03-11T21:35:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 1 4 2024-03-11T21:35:08.849 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:35:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 10 client.30459.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 5 2024-03-11T21:35:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2fe800b240 0x7f2fe800d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 5 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 2 5 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 4 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 10 client.30459.objecter _send_command 5 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe8010490 con 0x7f2fe800b240 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2fe800b240 0x7f2fe800d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe800b240 2024-03-11T21:35:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2fe800b240 0x7f2fe800d630 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe8010490 con 0x7f2fe800b240 2024-03-11T21:35:08.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:35:08.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 5 = system:0 2024-03-11T21:35:08.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 4 2024-03-11T21:35:08.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.875+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 2 5 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 10 client.30459.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 6 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 6 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8003980 osd=2 3 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 2 6 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 4 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 10 client.30459.objecter _send_command 6 2024-03-11T21:35:08.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.891+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8003400 con 0x7f2fe800b240 2024-03-11T21:35:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8003400 con 0x7f2fe800b240 2024-03-11T21:35:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 4 2024-03-11T21:35:08.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 2 6 2024-03-11T21:35:08.869 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 10 client.30459.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 7 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2fe800ecf0 0x7f2fe800f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 7 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 3 7 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 4 2024-03-11T21:35:08.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 10 client.30459.objecter _send_command 7 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe800f6c0 con 0x7f2fe800ecf0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2fe800ecf0 0x7f2fe800f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe800ecf0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2fe800ecf0 0x7f2fe800f180 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe800f6c0 con 0x7f2fe800ecf0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 7 = system:0 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 4 2024-03-11T21:35:08.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.895+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 3 7 2024-03-11T21:35:08.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 10 client.30459.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 8 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 8 2024-03-11T21:35:08.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8001b50 osd=3 3 2024-03-11T21:35:08.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 3 8 2024-03-11T21:35:08.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 4 2024-03-11T21:35:08.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 10 client.30459.objecter _send_command 8 2024-03-11T21:35:08.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe800f6c0 con 0x7f2fe800ecf0 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe800f6c0 con 0x7f2fe800ecf0 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 4 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 3 8 2024-03-11T21:35:08.890 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 10 client.30459.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 9 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f2fe8001740 0x7f2fe8014690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 9 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 4 9 2024-03-11T21:35:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 4 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 10 client.30459.objecter _send_command 9 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe8017620 con 0x7f2fe8001740 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f2fe8001740 0x7f2fe8014690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f2fe8001740 0x7f2fe8014690 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.915+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe8001740 2024-03-11T21:35:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.919+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe8017620 con 0x7f2fe8001740 2024-03-11T21:35:08.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.919+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:35:08.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.919+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 9 = system:0 2024-03-11T21:35:08.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.919+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 4 2024-03-11T21:35:08.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.919+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 4 9 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 10 client.30459.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 10 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 10 2024-03-11T21:35:08.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe80015d0 osd=4 3 2024-03-11T21:35:08.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 4 10 2024-03-11T21:35:08.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 4 2024-03-11T21:35:08.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 10 client.30459.objecter _send_command 10 2024-03-11T21:35:08.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.935+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8017620 con 0x7f2fe8001740 2024-03-11T21:35:08.911 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8017620 con 0x7f2fe8001740 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 4 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 4 10 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 10 client.30459.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 11 2024-03-11T21:35:08.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f2fe8016a00 0x7f2fe8016e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 11 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 5 11 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 4 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 10 client.30459.objecter _send_command 11 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe801a9f0 con 0x7f2fe8016a00 2024-03-11T21:35:08.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f2fe8016a00 0x7f2fe8016e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f3008ab5700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f2fe8016a00 0x7f2fe8016e60 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe8016a00 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe801a9f0 con 0x7f2fe8016a00 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 11 = system:0 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 4 2024-03-11T21:35:08.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.939+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 5 11 2024-03-11T21:35:08.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 10 client.30459.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 12 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 12 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8017620 osd=5 3 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 5 12 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 4 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 10 client.30459.objecter _send_command 12 2024-03-11T21:35:08.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.955+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8019870 con 0x7f2fe8016a00 2024-03-11T21:35:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8019870 con 0x7f2fe8016a00 2024-03-11T21:35:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 4 2024-03-11T21:35:08.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 5 12 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 10 client.30459.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 13 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f2fe801a010 0x7f2fe801a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 13 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 6 13 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 4 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 10 client.30459.objecter _send_command 13 2024-03-11T21:35:08.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe801dd60 con 0x7f2fe801a010 2024-03-11T21:35:08.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[29284]: cluster 2024-03-11T21:35:07.188332+0000 osd.4 (osd.4) 257 : cluster 0 5.e scrub starts 2024-03-11T21:35:08.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[29284]: cluster 2024-03-11T21:35:07.191757+0000 osd.4 (osd.4) 258 : cluster 0 5.e scrub ok 2024-03-11T21:35:08.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[29284]: cluster 2024-03-11T21:35:07.407399+0000 mgr.y (mgr.14550) 482 : cluster 0 pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:08.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[29284]: cluster 2024-03-11T21:35:08.141541+0000 osd.7 (osd.7) 291 : cluster 0 118.11 scrub starts 2024-03-11T21:35:08.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[29284]: cluster 2024-03-11T21:35:08.144904+0000 osd.7 (osd.7) 292 : cluster 0 118.11 scrub ok 2024-03-11T21:35:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[21512]: cluster 2024-03-11T21:35:07.188332+0000 osd.4 (osd.4) 257 : cluster 0 5.e scrub starts 2024-03-11T21:35:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[21512]: cluster 2024-03-11T21:35:07.191757+0000 osd.4 (osd.4) 258 : cluster 0 5.e scrub ok 2024-03-11T21:35:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[21512]: cluster 2024-03-11T21:35:07.407399+0000 mgr.y (mgr.14550) 482 : cluster 0 pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[21512]: cluster 2024-03-11T21:35:08.141541+0000 osd.7 (osd.7) 291 : cluster 0 118.11 scrub starts 2024-03-11T21:35:08.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:08 smithi143 bash[21512]: cluster 2024-03-11T21:35:08.144904+0000 osd.7 (osd.7) 292 : cluster 0 118.11 scrub ok 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f2fe801a010 0x7f2fe801a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe801a010 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.959+0000 7f3009ab7700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f2fe801a010 0x7f2fe801a430 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.963+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe801dd60 con 0x7f2fe801a010 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.963+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.963+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 13 = system:0 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.963+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 4 2024-03-11T21:35:08.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.963+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 6 13 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 10 client.30459.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 14 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 14 2024-03-11T21:35:08.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8019870 osd=6 3 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 6 14 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 4 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 10 client.30459.objecter _send_command 14 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8014c70 con 0x7f2fe801a010 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8014c70 con 0x7f2fe801a010 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 4 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 6 14 2024-03-11T21:35:08.956 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 10 client.30459.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 15 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f2fe8015390 0x7f2fe80157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 15 2024-03-11T21:35:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 7 15 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 4 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 10 client.30459.objecter _send_command 15 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.979+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fe8016050 con 0x7f2fe8015390 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f2fe8015390 0x7f2fe80157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f30092b6700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f2fe8015390 0x7f2fe80157d0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 10 client.30459.objecter ms_handle_connect 0x7f2fe8015390 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2fe8016050 con 0x7f2fe8015390 2024-03-11T21:35:08.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:35:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 15 = system:0 2024-03-11T21:35:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 4 2024-03-11T21:35:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.983+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 7 15 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 10 client.30459.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign -1 16 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 20 client.30459.objecter _get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 15 client.30459.objecter _session_command_op_remove -1 16 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 20 client.30459.objecter get_session s=0x7f2fe8014c70 osd=7 3 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 15 client.30459.objecter _session_command_op_assign 7 16 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 4 2024-03-11T21:35:08.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 10 client.30459.objecter _send_command 16 2024-03-11T21:35:08.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:08.999+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2fe8016260 con 0x7f2fe8015390 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f2ffa7fc700 1 -- 172.21.15.143:0/2691067693 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2fe8016260 con 0x7f2fe8015390 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f2ffa7fc700 10 client.30459.objecter ms_dispatch 0x7f3004001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f2ffa7fc700 10 client.30459.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f2ffa7fc700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 4 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f2ffa7fc700 15 client.30459.objecter _session_command_op_remove 7 16 2024-03-11T21:35:08.977 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 librados: watch_flush enter 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 librados: watch_flush exit 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.0 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f2fe8004730 msgr2=0x7f2fe8004b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f2fe8004730 0x7f2fe8004b50 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80053e0 osd=0 1 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.1 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f2fe8002750 msgr2=0x7f2fe8002bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f2fe8002750 0x7f2fe8002bc0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80081c0 osd=1 1 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.2 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2fe800b240 msgr2=0x7f2fe800d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f2fe800b240 0x7f2fe800d630 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8003980 osd=2 1 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.3 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2fe800ecf0 msgr2=0x7f2fe800f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f3009ab7700 1 -- 172.21.15.143:0/2691067693 reap_dead start 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2fe800ecf0 0x7f2fe800f180 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8001b50 osd=3 1 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.4 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f2fe8001740 msgr2=0x7f2fe8014690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f2fe8001740 0x7f2fe8014690 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe80015d0 osd=4 1 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.5 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f2fe8016a00 msgr2=0x7f2fe8016e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f2fe8016a00 0x7f2fe8016e60 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8017620 osd=5 1 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.6 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f2fe801a010 msgr2=0x7f2fe801a430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f2fe801a010 0x7f2fe801a430 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8019870 osd=6 1 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter close_session for osd.7 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f2fe8015390 msgr2=0x7f2fe80157d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f2fe8015390 0x7f2fe80157d0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter put_session s=0x7f2fe8014c70 osd=7 1 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 20 client.30459.objecter shutdown clearing up homeless session... 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f3009ab7700 1 -- 172.21.15.143:0/2691067693 reap_dead start 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 10 client.30459.objecter successfully canceled tick 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 msgr2=0x7f2ff0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 0x7f2ff0071ce0 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f30040ffce0 tx=0x7f2ff4009260 comp rx=0 tx=0).stop 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 msgr2=0x7f300419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00ada0 tx=0x7f2ffc00c330 comp rx=0 tx=0).stop 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 shutdown_connections 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3004100870 0x7f300419a0c0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 --2- 172.21.15.143:0/2691067693 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2ff006f820 0x7f2ff0071ce0 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 >> 172.21.15.143:0/2691067693 conn(0x7f30040772e0 msgr2=0x7f30040fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:08.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 shutdown_connections 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 -- 172.21.15.143:0/2691067693 wait complete. 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:09.003+0000 7f300b51a700 1 librados: shutdown 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:08.983 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:08.984 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:09.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:09.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:09.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:09.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:09.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:09.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:09.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:09.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:09.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:09.562 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:09.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:09.681 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:09 smithi203 bash[23654]: cluster 2024-03-11T21:35:08.020490+0000 osd.0 (osd.0) 285 : cluster 0 9.1d scrub starts 2024-03-11T21:35:09.681 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:09 smithi203 bash[23654]: cluster 2024-03-11T21:35:08.022531+0000 osd.0 (osd.0) 286 : cluster 0 9.1d scrub ok 2024-03-11T21:35:09.681 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:09 smithi203 bash[23654]: audit 2024-03-11T21:35:08.662591+0000 mgr.y (mgr.14550) 483 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:09.682 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:09 smithi203 bash[23654]: audit 2024-03-11T21:35:08.833964+0000 mon.a (mon.0) 2799 : audit 0 from='client.? 172.21.15.143:0/2691067693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:09.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:09.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:09.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[29284]: cluster 2024-03-11T21:35:08.020490+0000 osd.0 (osd.0) 285 : cluster 0 9.1d scrub starts 2024-03-11T21:35:09.815 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[29284]: cluster 2024-03-11T21:35:08.022531+0000 osd.0 (osd.0) 286 : cluster 0 9.1d scrub ok 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[29284]: audit 2024-03-11T21:35:08.662591+0000 mgr.y (mgr.14550) 483 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[29284]: audit 2024-03-11T21:35:08.833964+0000 mon.a (mon.0) 2799 : audit 0 from='client.? 172.21.15.143:0/2691067693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[21512]: cluster 2024-03-11T21:35:08.020490+0000 osd.0 (osd.0) 285 : cluster 0 9.1d scrub starts 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[21512]: cluster 2024-03-11T21:35:08.022531+0000 osd.0 (osd.0) 286 : cluster 0 9.1d scrub ok 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[21512]: audit 2024-03-11T21:35:08.662591+0000 mgr.y (mgr.14550) 483 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:09.816 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:09 smithi143 bash[21512]: audit 2024-03-11T21:35:08.833964+0000 mon.a (mon.0) 2799 : audit 0 from='client.? 172.21.15.143:0/2691067693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:09.833 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:09.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:09.964 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:10.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:10.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:10.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:10.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:10.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:10.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:10.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:10.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:10.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:10.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:10.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:10.726 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.042015+0000 osd.0 (osd.0) 287 : cluster 0 118.18 scrub starts 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.044560+0000 osd.0 (osd.0) 288 : cluster 0 118.18 scrub ok 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.196211+0000 osd.4 (osd.4) 259 : cluster 0 118.c deep-scrub starts 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.199414+0000 osd.4 (osd.4) 260 : cluster 0 118.c deep-scrub ok 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.409794+0000 mgr.y (mgr.14550) 484 : cluster 0 pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[21512]: cluster 2024-03-11T21:35:09.531143+0000 mon.a (mon.0) 2800 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.042015+0000 osd.0 (osd.0) 287 : cluster 0 118.18 scrub starts 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.044560+0000 osd.0 (osd.0) 288 : cluster 0 118.18 scrub ok 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.196211+0000 osd.4 (osd.4) 259 : cluster 0 118.c deep-scrub starts 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.199414+0000 osd.4 (osd.4) 260 : cluster 0 118.c deep-scrub ok 2024-03-11T21:35:10.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.409794+0000 mgr.y (mgr.14550) 484 : cluster 0 pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:10.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:10 smithi143 bash[29284]: cluster 2024-03-11T21:35:09.531143+0000 mon.a (mon.0) 2800 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:35:10.742 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:10.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.042015+0000 osd.0 (osd.0) 287 : cluster 0 118.18 scrub starts 2024-03-11T21:35:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.044560+0000 osd.0 (osd.0) 288 : cluster 0 118.18 scrub ok 2024-03-11T21:35:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.196211+0000 osd.4 (osd.4) 259 : cluster 0 118.c deep-scrub starts 2024-03-11T21:35:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.199414+0000 osd.4 (osd.4) 260 : cluster 0 118.c deep-scrub ok 2024-03-11T21:35:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.409794+0000 mgr.y (mgr.14550) 484 : cluster 0 pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:10 smithi203 bash[23654]: cluster 2024-03-11T21:35:09.531143+0000 mon.a (mon.0) 2800 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:35:10.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:10.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:10.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:11.003 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:11.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:11.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:11.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:11.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:11.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- 172.21.15.143:0/4084438930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498100d40 msgr2=0x7f14981011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 --2- 172.21.15.143:0/4084438930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498100d40 0x7f14981011c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f14880098f0 tx=0x7f1488009c00 comp rx=0 tx=0).stop 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- 172.21.15.143:0/4084438930 shutdown_connections 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 --2- 172.21.15.143:0/4084438930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498101700 0x7f149810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 --2- 172.21.15.143:0/4084438930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498106dc0 0x7f14981071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 --2- 172.21.15.143:0/4084438930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498100d40 0x7f14981011c0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- 172.21.15.143:0/4084438930 >> 172.21.15.143:0/4084438930 conn(0x7f1498076d80 msgr2=0x7f1498077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- 172.21.15.143:0/4084438930 shutdown_connections 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- 172.21.15.143:0/4084438930 wait complete. 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 librados: starting msgr at 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 librados: starting objecter 2024-03-11T21:35:11.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 Processor -- start 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 -- start start 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 librados: setting wanted keys 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.390+0000 7f149f650700 1 librados: calling monclient init 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 0x7f1498199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 0x7f149819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14981105c0 con 0x7f1498106dc0 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1498110440 con 0x7f1498100d40 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1498110740 con 0x7f1498101700 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:11.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149dbed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 0x7f149819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:57474/0 (socket says 172.21.15.143:57474) 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 -- 172.21.15.143:0/905745908 learned_addr learned my addr 172.21.15.143:0/905745908 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 -- 172.21.15.143:0/905745908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 msgr2=0x7f1498199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149d3ec700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 0x7f1498199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 0x7f1498199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 -- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 msgr2=0x7f149819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 0x7f149819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 -- 172.21.15.143:0/905745908 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149819eed0 con 0x7f1498101700 2024-03-11T21:35:11.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149d3ec700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 0x7f1498199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149dbed700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 0x7f149819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149cbeb700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f1488005950 tx=0x7f148800fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 10 client.?.objecter ms_handle_connect 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148800b7d0 con 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f14880054c0 con 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488003aa0 con 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149819f160 con 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 10 client.41666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14981a6a40 con 0x7f1498101700 2024-03-11T21:35:11.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 librados: init done 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1498102400 con 0x7f1498101700 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f148800b970 con 0x7f1498101700 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 0x7f1484071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(486..486 src has 87..486) v4 ==== 9714+0+0 (secure 0 0 0) 0x7f148808fc40 con 0x7f1498101700 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 10 client.41666.objecter ms_dispatch 0x7f1498001550 osd_map(486..486 src has 87..486) v4 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 3 client.41666.objecter handle_osd_map got epochs [486,486] > 0 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 3 client.41666.objecter handle_osd_map decoding full epoch 486 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f149d3ec700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 0x7f1484071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.394+0000 7f148e7fc700 20 client.41666.objecter dump_active .. 0 homeless 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.398+0000 7f148e7fc700 10 client.41666.objecter ms_handle_connect 0x7f148406f820 2024-03-11T21:35:11.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.398+0000 7f149d3ec700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 0x7f1484071ce0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f1490009fd0 tx=0x7f149000b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:11.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.402+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1488013030 con 0x7f1498101700 2024-03-11T21:35:11.379 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:11.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.501 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:11.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.610+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14981001e0 con 0x7f148406f820 2024-03-11T21:35:11.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.646+0000 7f148e7fc700 1 -- 172.21.15.143:0/905745908 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+666567 (secure 0 0 0) 0x7f14981001e0 con 0x7f148406f820 2024-03-11T21:35:11.630 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:35:11.635 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 10 librados: watch_flush enter 2024-03-11T21:35:11.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 10 librados: watch_flush exit 2024-03-11T21:35:11.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 20 client.41666.objecter shutdown clearing up homeless session... 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 10 client.41666.objecter successfully canceled tick 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 msgr2=0x7f1484071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 0x7f1484071ce0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f1490009fd0 tx=0x7f149000b040 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 msgr2=0x7f149819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f1488005950 tx=0x7f148800fb40 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 shutdown_connections 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f148406f820 0x7f1484071ce0 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1498106dc0 0x7f149819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1498100d40 0x7f1498199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 --2- 172.21.15.143:0/905745908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1498101700 0x7f149819a400 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 >> 172.21.15.143:0/905745908 conn(0x7f1498076d80 msgr2=0x7f14980ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 shutdown_connections 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 -- 172.21.15.143:0/905745908 wait complete. 2024-03-11T21:35:11.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:11.654+0000 7f149f650700 1 librados: shutdown 2024-03-11T21:35:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:10.389033+0000 osd.2 (osd.2) 247 : cluster 0 5.6 scrub starts 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:10.390794+0000 osd.2 (osd.2) 248 : cluster 0 5.6 scrub ok 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:10.693802+0000 osd.5 (osd.5) 297 : cluster 0 6.a scrub starts 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:10.696080+0000 osd.5 (osd.5) 298 : cluster 0 6.a scrub ok 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.137610+0000 osd.7 (osd.7) 293 : cluster 0 6.8 scrub starts 2024-03-11T21:35:11.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.139462+0000 osd.7 (osd.7) 294 : cluster 0 6.8 scrub ok 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:10.389033+0000 osd.2 (osd.2) 247 : cluster 0 5.6 scrub starts 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:10.390794+0000 osd.2 (osd.2) 248 : cluster 0 5.6 scrub ok 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:10.693802+0000 osd.5 (osd.5) 297 : cluster 0 6.a scrub starts 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:10.696080+0000 osd.5 (osd.5) 298 : cluster 0 6.a scrub ok 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.137610+0000 osd.7 (osd.7) 293 : cluster 0 6.8 scrub starts 2024-03-11T21:35:11.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:11 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.139462+0000 osd.7 (osd.7) 294 : cluster 0 6.8 scrub ok 2024-03-11T21:35:11.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:11.760 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:10.389033+0000 osd.2 (osd.2) 247 : cluster 0 5.6 scrub starts 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:10.390794+0000 osd.2 (osd.2) 248 : cluster 0 5.6 scrub ok 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:10.693802+0000 osd.5 (osd.5) 297 : cluster 0 6.a scrub starts 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:10.696080+0000 osd.5 (osd.5) 298 : cluster 0 6.a scrub ok 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.137610+0000 osd.7 (osd.7) 293 : cluster 0 6.8 scrub starts 2024-03-11T21:35:11.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:11 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.139462+0000 osd.7 (osd.7) 294 : cluster 0 6.8 scrub ok 2024-03-11T21:35:11.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:11.891 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:11.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 -- 172.21.15.143:0/3230305469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958101700 msgr2=0x7fa95810db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/3230305469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958101700 0x7fa95810db70 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fa95000b0a0 tx=0x7fa95000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 -- 172.21.15.143:0/3230305469 shutdown_connections 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/3230305469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958101700 0x7fa95810db70 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/3230305469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958106dc0 0x7fa9581071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/3230305469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 -- 172.21.15.143:0/3230305469 >> 172.21.15.143:0/3230305469 conn(0x7fa958076d80 msgr2=0x7fa958077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.090+0000 7fa95d1f4700 1 -- 172.21.15.143:0/3230305469 shutdown_connections 2024-03-11T21:35:12.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- 172.21.15.143:0/3230305469 wait complete. 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 librados: starting msgr at 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 librados: starting objecter 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 Processor -- start 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- start start 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 librados: setting wanted keys 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 librados: calling monclient init 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958101700 0x7fa958102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 0x7fa958103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:12.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa958110520 con 0x7fa958100d40 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9581103a0 con 0x7fa958101700 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9581106a0 con 0x7fa958106dc0 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa956d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 0x7fa958103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa956d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47714/0 (socket says 172.21.15.143:47714) 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa956d9d700 1 -- 172.21.15.143:0/1390023962 learned_addr learned my addr 172.21.15.143:0/1390023962 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95659c700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958101700 0x7fa958102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 -- 172.21.15.143:0/1390023962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958101700 msgr2=0x7fa958102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958101700 0x7fa958102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 -- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 msgr2=0x7fa9581029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 -- 172.21.15.143:0/1390023962 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9581a5fe0 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95759e700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 0x7fa958103440 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa95000bfd0 tx=0x7fa950007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 10 client.?.objecter ms_handle_connect 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa950021020 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9581a6270 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa950004c50 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa950005050 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 10 client.41672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9581a6720 con 0x7fa958106dc0 2024-03-11T21:35:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 librados: init done 2024-03-11T21:35:12.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95804faa0 con 0x7fa958106dc0 2024-03-11T21:35:12.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa950024020 con 0x7fa958106dc0 2024-03-11T21:35:12.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.094+0000 7fa93ffff700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 0x7fa940071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(486..486 src has 87..486) v4 ==== 9714+0+0 (secure 0 0 0) 0x7fa95000f020 con 0x7fa958106dc0 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 10 client.41672.objecter ms_dispatch 0x7fa958001550 osd_map(486..486 src has 87..486) v4 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 3 client.41672.objecter handle_osd_map got epochs [486,486] > 0 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 3 client.41672.objecter handle_osd_map decoding full epoch 486 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa956d9d700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 0x7fa940071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 20 client.41672.objecter dump_active .. 0 homeless 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa956d9d700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 0x7fa940071ce0 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7fa94800aca0 tx=0x7fa948005f90 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:12.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.098+0000 7fa93ffff700 10 client.41672.objecter ms_handle_connect 0x7fa94006f820 2024-03-11T21:35:12.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.102+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa95005c8f0 con 0x7fa958106dc0 2024-03-11T21:35:12.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:12.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:12.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:12.263 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:12.276 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:12.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.322+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa9581a6a40 con 0x7fa958106dc0 2024-03-11T21:35:12.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:12.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:12.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:12.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.534+0000 7fa93ffff700 1 -- 172.21.15.143:0/1390023962 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v487) v1 ==== 87+0+0 (secure 0 0 0) 0x7fa95007c0e0 con 0x7fa958106dc0 2024-03-11T21:35:12.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 10 librados: watch_flush enter 2024-03-11T21:35:12.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 10 librados: watch_flush exit 2024-03-11T21:35:12.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 20 client.41672.objecter shutdown clearing up homeless session... 2024-03-11T21:35:12.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 10 client.41672.objecter successfully canceled tick 2024-03-11T21:35:12.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 msgr2=0x7fa940071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:12.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 0x7fa940071ce0 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7fa94800aca0 tx=0x7fa948005f90 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 msgr2=0x7fa958103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 0x7fa958103440 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa95000bfd0 tx=0x7fa950007a20 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 shutdown_connections 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa94006f820 0x7fa940071ce0 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa958100d40 0x7fa9581029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa958106dc0 0x7fa958103440 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 --2- 172.21.15.143:0/1390023962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa958101700 0x7fa958102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 >> 172.21.15.143:0/1390023962 conn(0x7fa958076d80 msgr2=0x7fa958069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:12.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.538+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 shutdown_connections 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.128382+0000 osd.1 (osd.1) 91 : cluster 0 39.c scrub starts 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.129974+0000 osd.1 (osd.1) 92 : cluster 0 39.c scrub ok 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.410496+0000 mgr.y (mgr.14550) 485 : cluster 0 pgmap v729: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: audit 2024-03-11T21:35:11.615134+0000 mgr.y (mgr.14550) 486 : audit 0 from='client.41666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.668290+0000 osd.5 (osd.5) 299 : cluster 0 3.c scrub starts 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: cluster 2024-03-11T21:35:11.672306+0000 osd.5 (osd.5) 300 : cluster 0 3.c scrub ok 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: audit 2024-03-11T21:35:12.327584+0000 mon.c (mon.2) 969 : audit 1 from='client.? 172.21.15.143:0/1390023962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.514 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[21512]: audit 2024-03-11T21:35:12.327893+0000 mon.a (mon.0) 2801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.542+0000 7fa95d1f4700 1 -- 172.21.15.143:0/1390023962 wait complete. 2024-03-11T21:35:12.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:12.542+0000 7fa95d1f4700 1 librados: shutdown 2024-03-11T21:35:12.517 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:35:12.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:12.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:12.663 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:12.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:12.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.128382+0000 osd.1 (osd.1) 91 : cluster 0 39.c scrub starts 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.129974+0000 osd.1 (osd.1) 92 : cluster 0 39.c scrub ok 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.410496+0000 mgr.y (mgr.14550) 485 : cluster 0 pgmap v729: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: audit 2024-03-11T21:35:11.615134+0000 mgr.y (mgr.14550) 486 : audit 0 from='client.41666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.668290+0000 osd.5 (osd.5) 299 : cluster 0 3.c scrub starts 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: cluster 2024-03-11T21:35:11.672306+0000 osd.5 (osd.5) 300 : cluster 0 3.c scrub ok 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: audit 2024-03-11T21:35:12.327584+0000 mon.c (mon.2) 969 : audit 1 from='client.? 172.21.15.143:0/1390023962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:12 smithi203 bash[23654]: audit 2024-03-11T21:35:12.327893+0000 mon.a (mon.0) 2801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.128382+0000 osd.1 (osd.1) 91 : cluster 0 39.c scrub starts 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.129974+0000 osd.1 (osd.1) 92 : cluster 0 39.c scrub ok 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.410496+0000 mgr.y (mgr.14550) 485 : cluster 0 pgmap v729: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: audit 2024-03-11T21:35:11.615134+0000 mgr.y (mgr.14550) 486 : audit 0 from='client.41666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.668290+0000 osd.5 (osd.5) 299 : cluster 0 3.c scrub starts 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: cluster 2024-03-11T21:35:11.672306+0000 osd.5 (osd.5) 300 : cluster 0 3.c scrub ok 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: audit 2024-03-11T21:35:12.327584+0000 mon.c (mon.2) 969 : audit 1 from='client.? 172.21.15.143:0/1390023962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:12 smithi143 bash[29284]: audit 2024-03-11T21:35:12.327893+0000 mon.a (mon.0) 2801 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:12.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:12.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:12.921 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:13.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:13.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:13.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:13.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:13.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:13.314 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:13.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:13.445 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:13.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:35:13.516 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.209701+0000 osd.4 (osd.4) 261 : cluster 0 3.1a scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.212153+0000 osd.4 (osd.4) 262 : cluster 0 3.1a scrub ok 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.338732+0000 osd.6 (osd.6) 225 : cluster 0 3.a scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.343612+0000 osd.6 (osd.6) 226 : cluster 0 3.a scrub ok 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.527549+0000 mon.a (mon.0) 2802 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: audit 2024-03-11T21:35:12.529017+0000 mon.a (mon.0) 2803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.535749+0000 mon.a (mon.0) 2804 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.700893+0000 osd.5 (osd.5) 301 : cluster 0 5.f scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:12.702294+0000 osd.5 (osd.5) 302 : cluster 0 5.f scrub ok 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[29284]: cluster 2024-03-11T21:35:13.099293+0000 osd.7 (osd.7) 295 : cluster 0 2.1d scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.209701+0000 osd.4 (osd.4) 261 : cluster 0 3.1a scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.212153+0000 osd.4 (osd.4) 262 : cluster 0 3.1a scrub ok 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.338732+0000 osd.6 (osd.6) 225 : cluster 0 3.a scrub starts 2024-03-11T21:35:13.517 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.343612+0000 osd.6 (osd.6) 226 : cluster 0 3.a scrub ok 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.527549+0000 mon.a (mon.0) 2802 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: audit 2024-03-11T21:35:12.529017+0000 mon.a (mon.0) 2803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.535749+0000 mon.a (mon.0) 2804 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.700893+0000 osd.5 (osd.5) 301 : cluster 0 5.f scrub starts 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:12.702294+0000 osd.5 (osd.5) 302 : cluster 0 5.f scrub ok 2024-03-11T21:35:13.518 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:13 smithi143 bash[21512]: cluster 2024-03-11T21:35:13.099293+0000 osd.7 (osd.7) 295 : cluster 0 2.1d scrub starts 2024-03-11T21:35:13.524 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-03-11T21:35:11.410049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792197,"num_objects":362,"num_object_clones":3,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":2210,"num_read_kb":1953,"num_write":1519,"num_write_kb":15160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2897,"ondisk_log_size":2897,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":780,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":613896,"kb_used_data":90060,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749347320,"statfs":{"total":767960285184,"available":767331655680,"internally_reserved":0,"allocated":92221440,"data_stored":66409426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006210"},"pg_stats":[{"pgid":"287.1f","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.475163+0000","last_change":"2024-03-11T21:34:44.429500+0000","last_active":"2024-03-11T21:35:06.475163+0000","last_peered":"2024-03-11T21:35:06.475163+0000","last_clean":"2024-03-11T21:35:06.475163+0000","last_became_active":"2024-03-11T21:34:44.428622+0000","last_became_peered":"2024-03-11T21:34:44.428622+0000","last_unstale":"2024-03-11T21:35:06.475163+0000","last_undegraded":"2024-03-11T21:35:06.475163+0000","last_fullsized":"2024-03-11T21:35:06.475163+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.223561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.1e","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972396+0000","last_change":"2024-03-11T21:34:44.441386+0000","last_active":"2024-03-11T21:35:06.972396+0000","last_peered":"2024-03-11T21:35:06.972396+0000","last_clean":"2024-03-11T21:35:06.972396+0000","last_became_active":"2024-03-11T21:34:44.440753+0000","last_became_peered":"2024-03-11T21:34:44.440753+0000","last_unstale":"2024-03-11T21:35:06.972396+0000","last_undegraded":"2024-03-11T21:35:06.972396+0000","last_fullsized":"2024-03-11T21:35:06.972396+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.139095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.1d","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473394+0000","last_change":"2024-03-11T21:34:44.430478+0000","last_active":"2024-03-11T21:35:06.473394+0000","last_peered":"2024-03-11T21:35:06.473394+0000","last_clean":"2024-03-11T21:35:06.473394+0000","last_became_active":"2024-03-11T21:34:44.429897+0000","last_became_peered":"2024-03-11T21:34:44.429897+0000","last_unstale":"2024-03-11T21:35:06.473394+0000","last_undegraded":"2024-03-11T21:35:06.473394+0000","last_fullsized":"2024-03-11T21:35:06.473394+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.403818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.1c","version":"475'2","reported_seq":37,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354726+0000","last_change":"2024-03-11T21:34:44.456144+0000","last_active":"2024-03-11T21:35:06.354726+0000","last_peered":"2024-03-11T21:35:06.354726+0000","last_clean":"2024-03-11T21:35:06.354726+0000","last_became_active":"2024-03-11T21:34:44.455801+0000","last_became_peered":"2024-03-11T21:34:44.455801+0000","last_unstale":"2024-03-11T21:35:06.354726+0000","last_undegraded":"2024-03-11T21:35:06.354726+0000","last_fullsized":"2024-03-11T21:35:06.354726+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.1b","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974222+0000","last_change":"2024-03-11T21:34:44.443332+0000","last_active":"2024-03-11T21:35:06.974222+0000","last_peered":"2024-03-11T21:35:06.974222+0000","last_clean":"2024-03-11T21:35:06.974222+0000","last_became_active":"2024-03-11T21:34:44.442420+0000","last_became_peered":"2024-03-11T21:34:44.442420+0000","last_unstale":"2024-03-11T21:35:06.974222+0000","last_undegraded":"2024-03-11T21:35:06.974222+0000","last_fullsized":"2024-03-11T21:35:06.974222+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.280554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.1a","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473499+0000","last_change":"2024-03-11T21:34:44.435062+0000","last_active":"2024-03-11T21:35:06.473499+0000","last_peered":"2024-03-11T21:35:06.473499+0000","last_clean":"2024-03-11T21:35:06.473499+0000","last_became_active":"2024-03-11T21:34:44.434403+0000","last_became_peered":"2024-03-11T21:34:44.434403+0000","last_unstale":"2024-03-11T21:35:06.473499+0000","last_undegraded":"2024-03-11T21:35:06.473499+0000","last_fullsized":"2024-03-11T21:35:06.473499+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.19","version":"484'5","reported_seq":41,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968077+0000","last_change":"2024-03-11T21:34:44.432658+0000","last_active":"2024-03-11T21:35:06.968077+0000","last_peered":"2024-03-11T21:35:06.968077+0000","last_clean":"2024-03-11T21:35:06.968077+0000","last_became_active":"2024-03-11T21:34:44.432212+0000","last_became_peered":"2024-03-11T21:34:44.432212+0000","last_unstale":"2024-03-11T21:35:06.968077+0000","last_undegraded":"2024-03-11T21:35:06.968077+0000","last_fullsized":"2024-03-11T21:35:06.968077+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.930357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.18","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477345+0000","last_change":"2024-03-11T21:34:44.435887+0000","last_active":"2024-03-11T21:35:06.477345+0000","last_peered":"2024-03-11T21:35:06.477345+0000","last_clean":"2024-03-11T21:35:06.477345+0000","last_became_active":"2024-03-11T21:34:44.435001+0000","last_became_peered":"2024-03-11T21:34:44.435001+0000","last_unstale":"2024-03-11T21:35:06.477345+0000","last_undegraded":"2024-03-11T21:35:06.477345+0000","last_fullsized":"2024-03-11T21:35:06.477345+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:13.019156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.17","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971616+0000","last_change":"2024-03-11T21:34:44.443881+0000","last_active":"2024-03-11T21:35:06.971616+0000","last_peered":"2024-03-11T21:35:06.971616+0000","last_clean":"2024-03-11T21:35:06.971616+0000","last_became_active":"2024-03-11T21:34:44.443342+0000","last_became_peered":"2024-03-11T21:34:44.443342+0000","last_unstale":"2024-03-11T21:35:06.971616+0000","last_undegraded":"2024-03-11T21:35:06.971616+0000","last_fullsized":"2024-03-11T21:35:06.971616+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.619722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.16","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972593+0000","last_change":"2024-03-11T21:34:44.445594+0000","last_active":"2024-03-11T21:35:06.972593+0000","last_peered":"2024-03-11T21:35:06.972593+0000","last_clean":"2024-03-11T21:35:06.972593+0000","last_became_active":"2024-03-11T21:34:44.445102+0000","last_became_peered":"2024-03-11T21:34:44.445102+0000","last_unstale":"2024-03-11T21:35:06.972593+0000","last_undegraded":"2024-03-11T21:35:06.972593+0000","last_fullsized":"2024-03-11T21:35:06.972593+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.813542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.15","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479038+0000","last_change":"2024-03-11T21:34:44.447613+0000","last_active":"2024-03-11T21:35:06.479038+0000","last_peered":"2024-03-11T21:35:06.479038+0000","last_clean":"2024-03-11T21:35:06.479038+0000","last_became_active":"2024-03-11T21:34:44.447247+0000","last_became_peered":"2024-03-11T21:34:44.447247+0000","last_unstale":"2024-03-11T21:35:06.479038+0000","last_undegraded":"2024-03-11T21:35:06.479038+0000","last_fullsized":"2024-03-11T21:35:06.479038+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:01.421146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.14","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355239+0000","last_change":"2024-03-11T21:34:44.443084+0000","last_active":"2024-03-11T21:35:06.355239+0000","last_peered":"2024-03-11T21:35:06.355239+0000","last_clean":"2024-03-11T21:35:06.355239+0000","last_became_active":"2024-03-11T21:34:44.442710+0000","last_became_peered":"2024-03-11T21:34:44.442710+0000","last_unstale":"2024-03-11T21:35:06.355239+0000","last_undegraded":"2024-03-11T21:35:06.355239+0000","last_fullsized":"2024-03-11T21:35:06.355239+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.235460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.13","version":"475'1","reported_seq":36,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.537709+0000","last_change":"2024-03-11T21:34:44.435890+0000","last_active":"2024-03-11T21:35:09.537709+0000","last_peered":"2024-03-11T21:35:09.537709+0000","last_clean":"2024-03-11T21:35:09.537709+0000","last_became_active":"2024-03-11T21:34:44.435083+0000","last_became_peered":"2024-03-11T21:34:44.435083+0000","last_unstale":"2024-03-11T21:35:09.537709+0000","last_undegraded":"2024-03-11T21:35:09.537709+0000","last_fullsized":"2024-03-11T21:35:09.537709+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:44.604772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.12","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349353+0000","last_change":"2024-03-11T21:34:44.430273+0000","last_active":"2024-03-11T21:35:06.349353+0000","last_peered":"2024-03-11T21:35:06.349353+0000","last_clean":"2024-03-11T21:35:06.349353+0000","last_became_active":"2024-03-11T21:34:44.429778+0000","last_became_peered":"2024-03-11T21:34:44.429778+0000","last_unstale":"2024-03-11T21:35:06.349353+0000","last_undegraded":"2024-03-11T21:35:06.349353+0000","last_fullsized":"2024-03-11T21:35:06.349353+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.11","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472505+0000","last_change":"2024-03-11T21:34:44.444152+0000","last_active":"2024-03-11T21:35:06.472505+0000","last_peered":"2024-03-11T21:35:06.472505+0000","last_clean":"2024-03-11T21:35:06.472505+0000","last_became_active":"2024-03-11T21:34:44.443675+0000","last_became_peered":"2024-03-11T21:34:44.443675+0000","last_unstale":"2024-03-11T21:35:06.472505+0000","last_undegraded":"2024-03-11T21:35:06.472505+0000","last_fullsized":"2024-03-11T21:35:06.472505+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.783644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.10","version":"484'4","reported_seq":37,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.348640+0000","last_change":"2024-03-11T21:34:44.441652+0000","last_active":"2024-03-11T21:35:06.348640+0000","last_peered":"2024-03-11T21:35:06.348640+0000","last_clean":"2024-03-11T21:35:06.348640+0000","last_became_active":"2024-03-11T21:34:44.440931+0000","last_became_peered":"2024-03-11T21:34:44.440931+0000","last_unstale":"2024-03-11T21:35:06.348640+0000","last_undegraded":"2024-03-11T21:35:06.348640+0000","last_fullsized":"2024-03-11T21:35:06.348640+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.893202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.f","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354068+0000","last_change":"2024-03-11T21:34:44.441155+0000","last_active":"2024-03-11T21:35:06.354068+0000","last_peered":"2024-03-11T21:35:06.354068+0000","last_clean":"2024-03-11T21:35:06.354068+0000","last_became_active":"2024-03-11T21:34:44.440255+0000","last_became_peered":"2024-03-11T21:34:44.440255+0000","last_unstale":"2024-03-11T21:35:06.354068+0000","last_undegraded":"2024-03-11T21:35:06.354068+0000","last_fullsized":"2024-03-11T21:35:06.354068+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.799860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"287.e","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746876+0000","last_change":"2024-03-11T21:34:44.438672+0000","last_active":"2024-03-11T21:35:06.746876+0000","last_peered":"2024-03-11T21:35:06.746876+0000","last_clean":"2024-03-11T21:35:06.746876+0000","last_became_active":"2024-03-11T21:34:44.438068+0000","last_became_peered":"2024-03-11T21:34:44.438068+0000","last_unstale":"2024-03-11T21:35:06.746876+0000","last_undegraded":"2024-03-11T21:35:06.746876+0000","last_fullsized":"2024-03-11T21:35:06.746876+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.422291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"287.d","version":"0'0","reported_seq":39,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356949+0000","last_change":"2024-03-11T21:34:59.527262+0000","last_active":"2024-03-11T21:35:06.356949+0000","last_peered":"2024-03-11T21:35:06.356949+0000","last_clean":"2024-03-11T21:35:06.356949+0000","last_became_active":"2024-03-11T21:34:59.526178+0000","last_became_peered":"2024-03-11T21:34:59.526178+0000","last_unstale":"2024-03-11T21:35:06.356949+0000","last_undegraded":"2024-03-11T21:35:06.356949+0000","last_fullsized":"2024-03-11T21:35:06.356949+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.674554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"287.c","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.970876+0000","last_change":"2024-03-11T21:34:44.439565+0000","last_active":"2024-03-11T21:35:06.970876+0000","last_peered":"2024-03-11T21:35:06.970876+0000","last_clean":"2024-03-11T21:35:06.970876+0000","last_became_active":"2024-03-11T21:34:44.438909+0000","last_became_peered":"2024-03-11T21:34:44.438909+0000","last_unstale":"2024-03-11T21:35:06.970876+0000","last_undegraded":"2024-03-11T21:35:06.970876+0000","last_fullsized":"2024-03-11T21:35:06.970876+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:10.187968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.b","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974439+0000","last_change":"2024-03-11T21:34:44.455821+0000","last_active":"2024-03-11T21:35:06.974439+0000","last_peered":"2024-03-11T21:35:06.974439+0000","last_clean":"2024-03-11T21:35:06.974439+0000","last_became_active":"2024-03-11T21:34:44.455426+0000","last_became_peered":"2024-03-11T21:34:44.455426+0000","last_unstale":"2024-03-11T21:35:06.974439+0000","last_undegraded":"2024-03-11T21:35:06.974439+0000","last_fullsized":"2024-03-11T21:35:06.974439+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.955338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.a","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.969283+0000","last_change":"2024-03-11T21:34:44.438449+0000","last_active":"2024-03-11T21:35:06.969283+0000","last_peered":"2024-03-11T21:35:06.969283+0000","last_clean":"2024-03-11T21:35:06.969283+0000","last_became_active":"2024-03-11T21:34:44.437827+0000","last_became_peered":"2024-03-11T21:34:44.437827+0000","last_unstale":"2024-03-11T21:35:06.969283+0000","last_undegraded":"2024-03-11T21:35:06.969283+0000","last_fullsized":"2024-03-11T21:35:06.969283+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.405554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.9","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968344+0000","last_change":"2024-03-11T21:34:44.443718+0000","last_active":"2024-03-11T21:35:06.968344+0000","last_peered":"2024-03-11T21:35:06.968344+0000","last_clean":"2024-03-11T21:35:06.968344+0000","last_became_active":"2024-03-11T21:34:44.443025+0000","last_became_peered":"2024-03-11T21:34:44.443025+0000","last_unstale":"2024-03-11T21:35:06.968344+0000","last_undegraded":"2024-03-11T21:35:06.968344+0000","last_fullsized":"2024-03-11T21:35:06.968344+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:55.361921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.8","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472259+0000","last_change":"2024-03-11T21:34:44.430411+0000","last_active":"2024-03-11T21:35:06.472259+0000","last_peered":"2024-03-11T21:35:06.472259+0000","last_clean":"2024-03-11T21:35:06.472259+0000","last_became_active":"2024-03-11T21:34:44.429530+0000","last_became_peered":"2024-03-11T21:34:44.429530+0000","last_unstale":"2024-03-11T21:35:06.472259+0000","last_undegraded":"2024-03-11T21:35:06.472259+0000","last_fullsized":"2024-03-11T21:35:06.472259+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.837048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.7","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976521+0000","last_change":"2024-03-11T21:34:44.445228+0000","last_active":"2024-03-11T21:35:06.976521+0000","last_peered":"2024-03-11T21:35:06.976521+0000","last_clean":"2024-03-11T21:35:06.976521+0000","last_became_active":"2024-03-11T21:34:44.444851+0000","last_became_peered":"2024-03-11T21:34:44.444851+0000","last_unstale":"2024-03-11T21:35:06.976521+0000","last_undegraded":"2024-03-11T21:35:06.976521+0000","last_fullsized":"2024-03-11T21:35:06.976521+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.991080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"287.6","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972890+0000","last_change":"2024-03-11T21:34:44.444673+0000","last_active":"2024-03-11T21:35:06.972890+0000","last_peered":"2024-03-11T21:35:06.972890+0000","last_clean":"2024-03-11T21:35:06.972890+0000","last_became_active":"2024-03-11T21:34:44.442788+0000","last_became_peered":"2024-03-11T21:34:44.442788+0000","last_unstale":"2024-03-11T21:35:06.972890+0000","last_undegraded":"2024-03-11T21:35:06.972890+0000","last_fullsized":"2024-03-11T21:35:06.972890+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.046083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"287.5","version":"0'0","reported_seq":35,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538796+0000","last_change":"2024-03-11T21:34:44.438016+0000","last_active":"2024-03-11T21:35:09.538796+0000","last_peered":"2024-03-11T21:35:09.538796+0000","last_clean":"2024-03-11T21:35:09.538796+0000","last_became_active":"2024-03-11T21:34:44.437215+0000","last_became_peered":"2024-03-11T21:34:44.437215+0000","last_unstale":"2024-03-11T21:35:09.538796+0000","last_undegraded":"2024-03-11T21:35:09.538796+0000","last_fullsized":"2024-03-11T21:35:09.538796+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.638667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.4","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472869+0000","last_change":"2024-03-11T21:34:44.431713+0000","last_active":"2024-03-11T21:35:06.472869+0000","last_peered":"2024-03-11T21:35:06.472869+0000","last_clean":"2024-03-11T21:35:06.472869+0000","last_became_active":"2024-03-11T21:34:44.431191+0000","last_became_peered":"2024-03-11T21:34:44.431191+0000","last_unstale":"2024-03-11T21:35:06.472869+0000","last_undegraded":"2024-03-11T21:35:06.472869+0000","last_fullsized":"2024-03-11T21:35:06.472869+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.744204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"287.3","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745880+0000","last_change":"2024-03-11T21:34:44.437273+0000","last_active":"2024-03-11T21:35:06.745880+0000","last_peered":"2024-03-11T21:35:06.745880+0000","last_clean":"2024-03-11T21:35:06.745880+0000","last_became_active":"2024-03-11T21:34:44.436574+0000","last_became_peered":"2024-03-11T21:34:44.436574+0000","last_unstale":"2024-03-11T21:35:06.745880+0000","last_undegraded":"2024-03-11T21:35:06.745880+0000","last_fullsized":"2024-03-11T21:35:06.745880+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:51.659963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"287.2","version":"0'0","reported_seq":35,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.532812+0000","last_change":"2024-03-11T21:34:44.440013+0000","last_active":"2024-03-11T21:35:09.532812+0000","last_peered":"2024-03-11T21:35:09.532812+0000","last_clean":"2024-03-11T21:35:09.532812+0000","last_became_active":"2024-03-11T21:34:44.439626+0000","last_became_peered":"2024-03-11T21:34:44.439626+0000","last_unstale":"2024-03-11T21:35:09.532812+0000","last_undegraded":"2024-03-11T21:35:09.532812+0000","last_fullsized":"2024-03-11T21:35:09.532812+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.728708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"287.1","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477370+0000","last_change":"2024-03-11T21:34:44.453166+0000","last_active":"2024-03-11T21:35:06.477370+0000","last_peered":"2024-03-11T21:35:06.477370+0000","last_clean":"2024-03-11T21:35:06.477370+0000","last_became_active":"2024-03-11T21:34:44.452589+0000","last_became_peered":"2024-03-11T21:34:44.452589+0000","last_unstale":"2024-03-11T21:35:06.477370+0000","last_undegraded":"2024-03-11T21:35:06.477370+0000","last_fullsized":"2024-03-11T21:35:06.477370+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:06.018483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"287.0","version":"0'0","reported_seq":33,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.744749+0000","last_change":"2024-03-11T21:34:44.443969+0000","last_active":"2024-03-11T21:35:06.744749+0000","last_peered":"2024-03-11T21:35:06.744749+0000","last_clean":"2024-03-11T21:35:06.744749+0000","last_became_active":"2024-03-11T21:34:44.443534+0000","last_became_peered":"2024-03-11T21:34:44.443534+0000","last_unstale":"2024-03-11T21:35:06.744749+0000","last_undegraded":"2024-03-11T21:35:06.744749+0000","last_fullsized":"2024-03-11T21:35:06.744749+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:43.390268+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.390268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.780508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.9","version":"482'18","reported_seq":858,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356110+0000","last_change":"2024-03-11T21:34:59.470363+0000","last_active":"2024-03-11T21:35:06.356110+0000","last_peered":"2024-03-11T21:35:06.356110+0000","last_clean":"2024-03-11T21:35:06.356110+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:35:06.356110+0000","last_undegraded":"2024-03-11T21:35:06.356110+0000","last_fullsized":"2024-03-11T21:35:06.356110+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'18","last_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_deep_scrub":"482'18","last_deep_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.441729+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:03.606679+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013249049999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"461'24","reported_seq":844,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745074+0000","last_change":"2024-03-11T21:34:59.518849+0000","last_active":"2024-03-11T21:35:06.745074+0000","last_peered":"2024-03-11T21:35:06.745074+0000","last_clean":"2024-03-11T21:35:06.745074+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:35:06.745074+0000","last_undegraded":"2024-03-11T21:35:06.745074+0000","last_fullsized":"2024-03-11T21:35:06.745074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'24","last_scrub_stamp":"2024-03-11T21:34:54.761329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.761329+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:15.865054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001174565,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"464'40","reported_seq":858,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.969462+0000","last_change":"2024-03-11T21:34:59.786358+0000","last_active":"2024-03-11T21:35:06.969462+0000","last_peered":"2024-03-11T21:35:06.969462+0000","last_clean":"2024-03-11T21:35:06.969462+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:35:06.969462+0000","last_undegraded":"2024-03-11T21:35:06.969462+0000","last_fullsized":"2024-03-11T21:35:06.969462+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'40","last_scrub_stamp":"2024-03-11T21:34:55.080483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.080483+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:16.784905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041559939999999997,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"444'55","reported_seq":881,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349695+0000","last_change":"2024-03-11T21:34:59.457077+0000","last_active":"2024-03-11T21:35:06.349695+0000","last_peered":"2024-03-11T21:35:06.349695+0000","last_clean":"2024-03-11T21:35:06.349695+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:35:06.349695+0000","last_undegraded":"2024-03-11T21:35:06.349695+0000","last_fullsized":"2024-03-11T21:35:06.349695+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'52","last_scrub_stamp":"2024-03-11T21:33:20.623315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.623315+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020298180000000001,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"412'52","reported_seq":884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357137+0000","last_change":"2024-03-11T21:35:00.480497+0000","last_active":"2024-03-11T21:35:06.357137+0000","last_peered":"2024-03-11T21:35:06.357137+0000","last_clean":"2024-03-11T21:35:06.357137+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:35:06.357137+0000","last_undegraded":"2024-03-11T21:35:06.357137+0000","last_fullsized":"2024-03-11T21:35:06.357137+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'52","last_scrub_stamp":"2024-03-11T21:35:00.480347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.480347+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.336248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020713099999999998,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"255'34","reported_seq":843,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.351886+0000","last_change":"2024-03-11T21:34:59.457988+0000","last_active":"2024-03-11T21:35:06.351886+0000","last_peered":"2024-03-11T21:35:06.351886+0000","last_clean":"2024-03-11T21:35:06.351886+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:35:06.351886+0000","last_undegraded":"2024-03-11T21:35:06.351886+0000","last_fullsized":"2024-03-11T21:35:06.351886+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'34","last_scrub_stamp":"2024-03-11T21:33:15.709878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:15.709878+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0021175669999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"444'184","reported_seq":972,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354256+0000","last_change":"2024-03-11T21:34:59.458150+0000","last_active":"2024-03-11T21:35:06.354256+0000","last_peered":"2024-03-11T21:35:06.354256+0000","last_clean":"2024-03-11T21:35:06.354256+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:35:06.354256+0000","last_undegraded":"2024-03-11T21:35:06.354256+0000","last_fullsized":"2024-03-11T21:35:06.354256+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'183","last_scrub_stamp":"2024-03-11T21:33:14.703284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.703284+0000","objects_scrubbed":13,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001101749,"stat_sum":{"num_bytes":125,"num_objects":14,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":56,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":910,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746942+0000","last_change":"2024-03-11T21:34:59.524740+0000","last_active":"2024-03-11T21:35:06.746942+0000","last_peered":"2024-03-11T21:35:06.746942+0000","last_clean":"2024-03-11T21:35:06.746942+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:35:06.746942+0000","last_undegraded":"2024-03-11T21:35:06.746942+0000","last_fullsized":"2024-03-11T21:35:06.746942+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:33:27.586328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.586328+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0042559970000000001,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":832,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971348+0000","last_change":"2024-03-11T21:34:59.781193+0000","last_active":"2024-03-11T21:35:06.971348+0000","last_peered":"2024-03-11T21:35:06.971348+0000","last_clean":"2024-03-11T21:35:06.971348+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:35:06.971348+0000","last_undegraded":"2024-03-11T21:35:06.971348+0000","last_fullsized":"2024-03-11T21:35:06.971348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:33:08.891062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.891062+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025221639999999999,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":870,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971280+0000","last_change":"2024-03-11T21:34:59.780207+0000","last_active":"2024-03-11T21:35:06.971280+0000","last_peered":"2024-03-11T21:35:06.971280+0000","last_clean":"2024-03-11T21:35:06.971280+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:35:06.971280+0000","last_undegraded":"2024-03-11T21:35:06.971280+0000","last_fullsized":"2024-03-11T21:35:06.971280+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:33:14.964352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.964352+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00177477,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"412'120","reported_seq":912,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.969791+0000","last_change":"2024-03-11T21:34:59.790665+0000","last_active":"2024-03-11T21:35:06.969791+0000","last_peered":"2024-03-11T21:35:06.969791+0000","last_clean":"2024-03-11T21:35:06.969791+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:35:06.969791+0000","last_undegraded":"2024-03-11T21:35:06.969791+0000","last_fullsized":"2024-03-11T21:35:06.969791+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.000415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003513955,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":852,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349920+0000","last_change":"2024-03-11T21:34:59.456696+0000","last_active":"2024-03-11T21:35:06.349920+0000","last_peered":"2024-03-11T21:35:06.349920+0000","last_clean":"2024-03-11T21:35:06.349920+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:35:06.349920+0000","last_undegraded":"2024-03-11T21:35:06.349920+0000","last_fullsized":"2024-03-11T21:35:06.349920+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'21","last_scrub_stamp":"2024-03-11T21:33:17.671661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.671661+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0016765370000000001,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"458'3","reported_seq":835,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357345+0000","last_change":"2024-03-11T21:34:59.474996+0000","last_active":"2024-03-11T21:35:06.357345+0000","last_peered":"2024-03-11T21:35:06.357345+0000","last_clean":"2024-03-11T21:35:06.357345+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:35:06.357345+0000","last_undegraded":"2024-03-11T21:35:06.357345+0000","last_fullsized":"2024-03-11T21:35:06.357345+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'3","last_scrub_stamp":"2024-03-11T21:34:53.464423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.464423+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:10.965832+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.002090162,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":849,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974416+0000","last_change":"2024-03-11T21:34:59.781684+0000","last_active":"2024-03-11T21:35:06.974416+0000","last_peered":"2024-03-11T21:35:06.974416+0000","last_clean":"2024-03-11T21:35:06.974416+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:35:06.974416+0000","last_undegraded":"2024-03-11T21:35:06.974416+0000","last_fullsized":"2024-03-11T21:35:06.974416+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:07.934653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:07.934653+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021076559999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"412'41","reported_seq":843,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479679+0000","last_change":"2024-03-11T21:34:59.526008+0000","last_active":"2024-03-11T21:35:06.479679+0000","last_peered":"2024-03-11T21:35:06.479679+0000","last_clean":"2024-03-11T21:35:06.479679+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:35:06.479679+0000","last_undegraded":"2024-03-11T21:35:06.479679+0000","last_fullsized":"2024-03-11T21:35:06.479679+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'39","last_scrub_stamp":"2024-03-11T21:33:25.692508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.692508+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014636619999999999,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":885,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479285+0000","last_change":"2024-03-11T21:34:59.526116+0000","last_active":"2024-03-11T21:35:06.479285+0000","last_peered":"2024-03-11T21:35:06.479285+0000","last_clean":"2024-03-11T21:35:06.479285+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:35:06.479285+0000","last_undegraded":"2024-03-11T21:35:06.479285+0000","last_fullsized":"2024-03-11T21:35:06.479285+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:33:23.654355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.654355+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023751990000000001,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":829,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971330+0000","last_change":"2024-03-11T21:34:59.780644+0000","last_active":"2024-03-11T21:35:06.971330+0000","last_peered":"2024-03-11T21:35:06.971330+0000","last_clean":"2024-03-11T21:35:06.971330+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:35:06.971330+0000","last_undegraded":"2024-03-11T21:35:06.971330+0000","last_fullsized":"2024-03-11T21:35:06.971330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:33:05.927971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.927971+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020122569999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":901,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971208+0000","last_change":"2024-03-11T21:34:59.780112+0000","last_active":"2024-03-11T21:35:06.971208+0000","last_peered":"2024-03-11T21:35:06.971208+0000","last_clean":"2024-03-11T21:35:06.971208+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:35:06.971208+0000","last_undegraded":"2024-03-11T21:35:06.971208+0000","last_fullsized":"2024-03-11T21:35:06.971208+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'107","last_scrub_stamp":"2024-03-11T21:33:10.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.951942+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017313070000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":724,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538085+0000","last_change":"2024-03-11T21:35:09.541073+0000","last_active":"2024-03-11T21:35:09.538085+0000","last_peered":"2024-03-11T21:35:09.538085+0000","last_clean":"2024-03-11T21:35:09.538085+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:35:09.538085+0000","last_undegraded":"2024-03-11T21:35:09.538085+0000","last_fullsized":"2024-03-11T21:35:09.538085+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:33:17.964235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.964235+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0036993310000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"118.1a","version":"247'56","reported_seq":865,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973476+0000","last_change":"2024-03-11T21:34:59.781506+0000","last_active":"2024-03-11T21:35:06.973476+0000","last_peered":"2024-03-11T21:35:06.973476+0000","last_clean":"2024-03-11T21:35:06.973476+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:35:06.973476+0000","last_undegraded":"2024-03-11T21:35:06.973476+0000","last_fullsized":"2024-03-11T21:35:06.973476+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:33:04.948734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:04.948734+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027682850000000001,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"314'85","reported_seq":905,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473710+0000","last_change":"2024-03-11T21:34:59.788135+0000","last_active":"2024-03-11T21:35:06.473710+0000","last_peered":"2024-03-11T21:35:06.473710+0000","last_clean":"2024-03-11T21:35:06.473710+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:35:06.473710+0000","last_undegraded":"2024-03-11T21:35:06.473710+0000","last_fullsized":"2024-03-11T21:35:06.473710+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'85","last_scrub_stamp":"2024-03-11T21:34:42.241601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.241601+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:10.113505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043381729999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"484'71","reported_seq":901,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971337+0000","last_change":"2024-03-11T21:34:59.792527+0000","last_active":"2024-03-11T21:35:06.971337+0000","last_peered":"2024-03-11T21:35:06.971337+0000","last_clean":"2024-03-11T21:35:06.971337+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:35:06.971337+0000","last_undegraded":"2024-03-11T21:35:06.971337+0000","last_fullsized":"2024-03-11T21:35:06.971337+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'69","last_scrub_stamp":"2024-03-11T21:34:58.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.041773+0000","objects_scrubbed":9,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:16.272473+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0085999360000000007,"stat_sum":{"num_bytes":61,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"444'37","reported_seq":743,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.540943+0000","last_change":"2024-03-11T21:35:09.542091+0000","last_active":"2024-03-11T21:35:09.540943+0000","last_peered":"2024-03-11T21:35:09.540943+0000","last_clean":"2024-03-11T21:35:09.540943+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:35:09.540943+0000","last_undegraded":"2024-03-11T21:35:09.540943+0000","last_fullsized":"2024-03-11T21:35:09.540943+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:33:14.938568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.938568+0000","objects_scrubbed":2,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016893019999999999,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"}]},{"pgid":"118.1e","version":"361'209","reported_seq":974,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973306+0000","last_change":"2024-03-11T21:34:59.791688+0000","last_active":"2024-03-11T21:35:06.973306+0000","last_peered":"2024-03-11T21:35:06.973306+0000","last_clean":"2024-03-11T21:35:06.973306+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:35:06.973306+0000","last_undegraded":"2024-03-11T21:35:06.973306+0000","last_fullsized":"2024-03-11T21:35:06.973306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002103499,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":843,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:08.145092+0000","last_change":"2024-03-11T21:35:08.145164+0000","last_active":"2024-03-11T21:35:08.145092+0000","last_peered":"2024-03-11T21:35:08.145092+0000","last_clean":"2024-03-11T21:35:08.145092+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:35:08.145092+0000","last_undegraded":"2024-03-11T21:35:08.145092+0000","last_fullsized":"2024-03-11T21:35:08.145092+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'35","last_scrub_stamp":"2024-03-11T21:35:08.144914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.144914+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.408072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024053049999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"464'17","reported_seq":887,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.744846+0000","last_change":"2024-03-11T21:34:59.518866+0000","last_active":"2024-03-11T21:35:06.744846+0000","last_peered":"2024-03-11T21:35:06.744846+0000","last_clean":"2024-03-11T21:35:06.744846+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:35:06.744846+0000","last_undegraded":"2024-03-11T21:35:06.744846+0000","last_fullsized":"2024-03-11T21:35:06.744846+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'14","last_scrub_stamp":"2024-03-11T21:33:31.565199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.565199+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001232085,"stat_sum":{"num_bytes":4194330,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":892,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746160+0000","last_change":"2024-03-11T21:34:59.521603+0000","last_active":"2024-03-11T21:35:06.746160+0000","last_peered":"2024-03-11T21:35:06.746160+0000","last_clean":"2024-03-11T21:35:06.746160+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:35:06.746160+0000","last_undegraded":"2024-03-11T21:35:06.746160+0000","last_fullsized":"2024-03-11T21:35:06.746160+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'87","last_scrub_stamp":"2024-03-11T21:34:32.811720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.811720+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.631990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001731319,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":875,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.139200+0000","last_change":"2024-03-11T21:35:07.139237+0000","last_active":"2024-03-11T21:35:07.139200+0000","last_peered":"2024-03-11T21:35:07.139200+0000","last_clean":"2024-03-11T21:35:07.139200+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:35:07.139200+0000","last_undegraded":"2024-03-11T21:35:07.139200+0000","last_fullsized":"2024-03-11T21:35:07.139200+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'65","last_scrub_stamp":"2024-03-11T21:35:07.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.139123+0000","objects_scrubbed":8,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:35.810664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002219478,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":897,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356910+0000","last_change":"2024-03-11T21:35:04.446304+0000","last_active":"2024-03-11T21:35:06.356910+0000","last_peered":"2024-03-11T21:35:06.356910+0000","last_clean":"2024-03-11T21:35:06.356910+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:35:06.356910+0000","last_undegraded":"2024-03-11T21:35:06.356910+0000","last_fullsized":"2024-03-11T21:35:06.356910+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.446157+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:07.937277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018741809999999999,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"310'51","reported_seq":869,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.351337+0000","last_change":"2024-03-11T21:34:59.457751+0000","last_active":"2024-03-11T21:35:06.351337+0000","last_peered":"2024-03-11T21:35:06.351337+0000","last_clean":"2024-03-11T21:35:06.351337+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:35:06.351337+0000","last_undegraded":"2024-03-11T21:35:06.351337+0000","last_fullsized":"2024-03-11T21:35:06.351337+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'51","last_scrub_stamp":"2024-03-11T21:33:25.682768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.682768+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020127560000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"467'56","reported_seq":878,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356674+0000","last_change":"2024-03-11T21:34:59.477203+0000","last_active":"2024-03-11T21:35:06.356674+0000","last_peered":"2024-03-11T21:35:06.356674+0000","last_clean":"2024-03-11T21:35:06.356674+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:35:06.356674+0000","last_undegraded":"2024-03-11T21:35:06.356674+0000","last_fullsized":"2024-03-11T21:35:06.356674+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'53","last_scrub_stamp":"2024-03-11T21:33:42.167697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.167697+0000","objects_scrubbed":5,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076561900000000004,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"467'6","reported_seq":829,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973682+0000","last_change":"2024-03-11T21:34:59.792569+0000","last_active":"2024-03-11T21:35:06.973682+0000","last_peered":"2024-03-11T21:35:06.973682+0000","last_clean":"2024-03-11T21:35:06.973682+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:35:06.973682+0000","last_undegraded":"2024-03-11T21:35:06.973682+0000","last_fullsized":"2024-03-11T21:35:06.973682+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'6","last_scrub_stamp":"2024-03-11T21:34:50.112525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.112525+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:07.967843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028533600000000001,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":706,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478227+0000","last_change":"2024-03-11T21:32:27.923539+0000","last_active":"2024-03-11T21:35:06.478227+0000","last_peered":"2024-03-11T21:35:06.478227+0000","last_clean":"2024-03-11T21:35:06.478227+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:35:06.478227+0000","last_undegraded":"2024-03-11T21:35:06.478227+0000","last_fullsized":"2024-03-11T21:35:06.478227+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:27.923325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:27.923325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":717,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472631+0000","last_change":"2024-03-11T21:35:05.450716+0000","last_active":"2024-03-11T21:35:06.472631+0000","last_peered":"2024-03-11T21:35:06.472631+0000","last_clean":"2024-03-11T21:35:06.472631+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:35:06.472631+0000","last_undegraded":"2024-03-11T21:35:06.472631+0000","last_fullsized":"2024-03-11T21:35:06.472631+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.450323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.450323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:19.425575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":707,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745837+0000","last_change":"2024-03-11T21:33:25.553124+0000","last_active":"2024-03-11T21:35:06.745837+0000","last_peered":"2024-03-11T21:35:06.745837+0000","last_clean":"2024-03-11T21:35:06.745837+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:35:06.745837+0000","last_undegraded":"2024-03-11T21:35:06.745837+0000","last_fullsized":"2024-03-11T21:35:06.745837+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.552965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.552965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":697,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477797+0000","last_change":"2024-03-11T21:33:13.735432+0000","last_active":"2024-03-11T21:35:06.477797+0000","last_peered":"2024-03-11T21:35:06.477797+0000","last_clean":"2024-03-11T21:35:06.477797+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:35:06.477797+0000","last_undegraded":"2024-03-11T21:35:06.477797+0000","last_fullsized":"2024-03-11T21:35:06.477797+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.735317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.735317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":611,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538298+0000","last_change":"2024-03-11T21:33:34.103121+0000","last_active":"2024-03-11T21:35:09.538298+0000","last_peered":"2024-03-11T21:35:09.538298+0000","last_clean":"2024-03-11T21:35:09.538298+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:35:09.538298+0000","last_undegraded":"2024-03-11T21:35:09.538298+0000","last_fullsized":"2024-03-11T21:35:09.538298+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:34.103006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:34.103006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":717,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.474498+0000","last_change":"2024-03-11T21:35:06.473871+0000","last_active":"2024-03-11T21:35:06.474498+0000","last_peered":"2024-03-11T21:35:06.474498+0000","last_clean":"2024-03-11T21:35:06.474498+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:35:06.474498+0000","last_undegraded":"2024-03-11T21:35:06.474498+0000","last_fullsized":"2024-03-11T21:35:06.474498+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.473532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.473532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:14.324669+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":718,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.474622+0000","last_change":"2024-03-11T21:34:58.319554+0000","last_active":"2024-03-11T21:35:06.474622+0000","last_peered":"2024-03-11T21:35:06.474622+0000","last_clean":"2024-03-11T21:35:06.474622+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:35:06.474622+0000","last_undegraded":"2024-03-11T21:35:06.474622+0000","last_fullsized":"2024-03-11T21:35:06.474622+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.319457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:21.657447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":720,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357001+0000","last_change":"2024-03-11T21:34:32.354266+0000","last_active":"2024-03-11T21:35:06.357001+0000","last_peered":"2024-03-11T21:35:06.357001+0000","last_clean":"2024-03-11T21:35:06.357001+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:35:06.357001+0000","last_undegraded":"2024-03-11T21:35:06.357001+0000","last_fullsized":"2024-03-11T21:35:06.357001+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.354023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.354023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.133336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":711,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349864+0000","last_change":"2024-03-11T21:34:35.918096+0000","last_active":"2024-03-11T21:35:06.349864+0000","last_peered":"2024-03-11T21:35:06.349864+0000","last_clean":"2024-03-11T21:35:06.349864+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:35:06.349864+0000","last_undegraded":"2024-03-11T21:35:06.349864+0000","last_fullsized":"2024-03-11T21:35:06.349864+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.917835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.917835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.696652+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":705,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477488+0000","last_change":"2024-03-11T21:34:49.557575+0000","last_active":"2024-03-11T21:35:06.477488+0000","last_peered":"2024-03-11T21:35:06.477488+0000","last_clean":"2024-03-11T21:35:06.477488+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:35:06.477488+0000","last_undegraded":"2024-03-11T21:35:06.477488+0000","last_fullsized":"2024-03-11T21:35:06.477488+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:49.557488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:49.557488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.936444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":710,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972529+0000","last_change":"2024-03-11T21:33:41.974005+0000","last_active":"2024-03-11T21:35:06.972529+0000","last_peered":"2024-03-11T21:35:06.972529+0000","last_clean":"2024-03-11T21:35:06.972529+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:35:06.972529+0000","last_undegraded":"2024-03-11T21:35:06.972529+0000","last_fullsized":"2024-03-11T21:35:06.972529+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.973907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":700,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972936+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:35:06.972936+0000","last_peered":"2024-03-11T21:35:06.972936+0000","last_clean":"2024-03-11T21:35:06.972936+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:35:06.972936+0000","last_undegraded":"2024-03-11T21:35:06.972936+0000","last_fullsized":"2024-03-11T21:35:06.972936+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":710,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974191+0000","last_change":"2024-03-11T21:34:40.037312+0000","last_active":"2024-03-11T21:35:06.974191+0000","last_peered":"2024-03-11T21:35:06.974191+0000","last_clean":"2024-03-11T21:35:06.974191+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:35:06.974191+0000","last_undegraded":"2024-03-11T21:35:06.974191+0000","last_fullsized":"2024-03-11T21:35:06.974191+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.037161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.037161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.123966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":711,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.353142+0000","last_change":"2024-03-11T21:35:05.211872+0000","last_active":"2024-03-11T21:35:06.353142+0000","last_peered":"2024-03-11T21:35:06.353142+0000","last_clean":"2024-03-11T21:35:06.353142+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:35:06.353142+0000","last_undegraded":"2024-03-11T21:35:06.353142+0000","last_fullsized":"2024-03-11T21:35:06.353142+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.211634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:32.162818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":719,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355312+0000","last_change":"2024-03-11T21:34:58.485169+0000","last_active":"2024-03-11T21:35:06.355312+0000","last_peered":"2024-03-11T21:35:06.355312+0000","last_clean":"2024-03-11T21:35:06.355312+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:35:06.355312+0000","last_undegraded":"2024-03-11T21:35:06.355312+0000","last_fullsized":"2024-03-11T21:35:06.355312+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.485052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.485052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:03.536510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":115,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.539873+0000","last_change":"2024-03-11T21:34:52.102628+0000","last_active":"2024-03-11T21:35:09.539873+0000","last_peered":"2024-03-11T21:35:09.539873+0000","last_clean":"2024-03-11T21:35:09.539873+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:35:09.539873+0000","last_undegraded":"2024-03-11T21:35:09.539873+0000","last_fullsized":"2024-03-11T21:35:09.539873+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:52.102271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.102271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.080698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":717,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472576+0000","last_change":"2024-03-11T21:34:54.321311+0000","last_active":"2024-03-11T21:35:06.472576+0000","last_peered":"2024-03-11T21:35:06.472576+0000","last_clean":"2024-03-11T21:35:06.472576+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:35:06.472576+0000","last_undegraded":"2024-03-11T21:35:06.472576+0000","last_fullsized":"2024-03-11T21:35:06.472576+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.321007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.321007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:12.720518+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":706,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968654+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:35:06.968654+0000","last_peered":"2024-03-11T21:35:06.968654+0000","last_clean":"2024-03-11T21:35:06.968654+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:35:06.968654+0000","last_undegraded":"2024-03-11T21:35:06.968654+0000","last_fullsized":"2024-03-11T21:35:06.968654+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":697,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477676+0000","last_change":"2024-03-11T21:33:12.695054+0000","last_active":"2024-03-11T21:35:06.477676+0000","last_peered":"2024-03-11T21:35:06.477676+0000","last_clean":"2024-03-11T21:35:06.477676+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:35:06.477676+0000","last_undegraded":"2024-03-11T21:35:06.477676+0000","last_fullsized":"2024-03-11T21:35:06.477676+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.694966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.694966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":711,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350981+0000","last_change":"2024-03-11T21:35:02.148627+0000","last_active":"2024-03-11T21:35:06.350981+0000","last_peered":"2024-03-11T21:35:06.350981+0000","last_clean":"2024-03-11T21:35:06.350981+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:35:06.350981+0000","last_undegraded":"2024-03-11T21:35:06.350981+0000","last_fullsized":"2024-03-11T21:35:06.350981+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.148344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.148344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.606703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":579,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538855+0000","last_change":"2024-03-11T21:32:22.987833+0000","last_active":"2024-03-11T21:35:09.538855+0000","last_peered":"2024-03-11T21:35:09.538855+0000","last_clean":"2024-03-11T21:35:09.538855+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:35:09.538855+0000","last_undegraded":"2024-03-11T21:35:09.538855+0000","last_fullsized":"2024-03-11T21:35:09.538855+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.987710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.987710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":705,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479139+0000","last_change":"2024-03-11T21:34:59.525103+0000","last_active":"2024-03-11T21:35:06.479139+0000","last_peered":"2024-03-11T21:35:06.479139+0000","last_clean":"2024-03-11T21:35:06.479139+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:35:06.479139+0000","last_undegraded":"2024-03-11T21:35:06.479139+0000","last_fullsized":"2024-03-11T21:35:06.479139+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:59.524800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:59.524800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:15.354697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":705,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479366+0000","last_change":"2024-03-11T21:35:03.560512+0000","last_active":"2024-03-11T21:35:06.479366+0000","last_peered":"2024-03-11T21:35:06.479366+0000","last_clean":"2024-03-11T21:35:06.479366+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:35:06.479366+0000","last_undegraded":"2024-03-11T21:35:06.479366+0000","last_fullsized":"2024-03-11T21:35:06.479366+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.560208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.560208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.367119+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":710,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976674+0000","last_change":"2024-03-11T21:34:58.091810+0000","last_active":"2024-03-11T21:35:06.976674+0000","last_peered":"2024-03-11T21:35:06.976674+0000","last_clean":"2024-03-11T21:35:06.976674+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:35:06.976674+0000","last_undegraded":"2024-03-11T21:35:06.976674+0000","last_fullsized":"2024-03-11T21:35:06.976674+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.091664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.091664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.019039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":708,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745446+0000","last_change":"2024-03-11T21:33:41.560584+0000","last_active":"2024-03-11T21:35:06.745446+0000","last_peered":"2024-03-11T21:35:06.745446+0000","last_clean":"2024-03-11T21:35:06.745446+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:35:06.745446+0000","last_undegraded":"2024-03-11T21:35:06.745446+0000","last_fullsized":"2024-03-11T21:35:06.745446+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.560389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.560389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":701,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968772+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:35:06.968772+0000","last_peered":"2024-03-11T21:35:06.968772+0000","last_clean":"2024-03-11T21:35:06.968772+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:35:06.968772+0000","last_undegraded":"2024-03-11T21:35:06.968772+0000","last_fullsized":"2024-03-11T21:35:06.968772+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":115,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.534969+0000","last_change":"2024-03-11T21:34:39.164537+0000","last_active":"2024-03-11T21:35:09.534969+0000","last_peered":"2024-03-11T21:35:09.534969+0000","last_clean":"2024-03-11T21:35:09.534969+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:35:09.534969+0000","last_undegraded":"2024-03-11T21:35:09.534969+0000","last_fullsized":"2024-03-11T21:35:09.534969+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.164421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.164421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.867286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":704,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350838+0000","last_change":"2024-03-11T21:33:08.584273+0000","last_active":"2024-03-11T21:35:06.350838+0000","last_peered":"2024-03-11T21:35:06.350838+0000","last_clean":"2024-03-11T21:35:06.350838+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:35:06.350838+0000","last_undegraded":"2024-03-11T21:35:06.350838+0000","last_fullsized":"2024-03-11T21:35:06.350838+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:08.584163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:08.584163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":711,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974811+0000","last_change":"2024-03-11T21:34:36.108350+0000","last_active":"2024-03-11T21:35:06.974811+0000","last_peered":"2024-03-11T21:35:06.974811+0000","last_clean":"2024-03-11T21:35:06.974811+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:35:06.974811+0000","last_undegraded":"2024-03-11T21:35:06.974811+0000","last_fullsized":"2024-03-11T21:35:06.974811+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.108246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.108246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.604525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":701,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972520+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:35:06.972520+0000","last_peered":"2024-03-11T21:35:06.972520+0000","last_clean":"2024-03-11T21:35:06.972520+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:35:06.972520+0000","last_undegraded":"2024-03-11T21:35:06.972520+0000","last_fullsized":"2024-03-11T21:35:06.972520+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":719,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355442+0000","last_change":"2024-03-11T21:34:48.410224+0000","last_active":"2024-03-11T21:35:06.355442+0000","last_peered":"2024-03-11T21:35:06.355442+0000","last_clean":"2024-03-11T21:35:06.355442+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:35:06.355442+0000","last_undegraded":"2024-03-11T21:35:06.355442+0000","last_fullsized":"2024-03-11T21:35:06.355442+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.410124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.410124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:06.214610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":587,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.540387+0000","last_change":"2024-03-11T21:35:01.154707+0000","last_active":"2024-03-11T21:35:09.540387+0000","last_peered":"2024-03-11T21:35:09.540387+0000","last_clean":"2024-03-11T21:35:09.540387+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:35:09.540387+0000","last_undegraded":"2024-03-11T21:35:09.540387+0000","last_fullsized":"2024-03-11T21:35:09.540387+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.154502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.154502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.515763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":717,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976184+0000","last_change":"2024-03-11T21:33:31.964693+0000","last_active":"2024-03-11T21:35:06.976184+0000","last_peered":"2024-03-11T21:35:06.976184+0000","last_clean":"2024-03-11T21:35:06.976184+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:35:06.976184+0000","last_undegraded":"2024-03-11T21:35:06.976184+0000","last_fullsized":"2024-03-11T21:35:06.976184+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:31.964602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.964602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1007,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.970115+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:35:06.970115+0000","last_peered":"2024-03-11T21:35:06.970115+0000","last_clean":"2024-03-11T21:35:06.970115+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:35:06.970115+0000","last_undegraded":"2024-03-11T21:35:06.970115+0000","last_fullsized":"2024-03-11T21:35:06.970115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":899,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973509+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:35:06.973509+0000","last_peered":"2024-03-11T21:35:06.973509+0000","last_clean":"2024-03-11T21:35:06.973509+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:35:06.973509+0000","last_undegraded":"2024-03-11T21:35:06.973509+0000","last_fullsized":"2024-03-11T21:35:06.973509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":887,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355688+0000","last_change":"2024-03-11T21:35:01.437775+0000","last_active":"2024-03-11T21:35:06.355688+0000","last_peered":"2024-03-11T21:35:06.355688+0000","last_clean":"2024-03-11T21:35:06.355688+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:35:06.355688+0000","last_undegraded":"2024-03-11T21:35:06.355688+0000","last_fullsized":"2024-03-11T21:35:06.355688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.437566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.437566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:03.108458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746176+0000","last_change":"2024-03-11T21:32:24.092785+0000","last_active":"2024-03-11T21:35:06.746176+0000","last_peered":"2024-03-11T21:35:06.746176+0000","last_clean":"2024-03-11T21:35:06.746176+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:35:06.746176+0000","last_undegraded":"2024-03-11T21:35:06.746176+0000","last_fullsized":"2024-03-11T21:35:06.746176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.092567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.092567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747758+0000","last_change":"2024-03-11T21:34:38.848849+0000","last_active":"2024-03-11T21:35:06.747758+0000","last_peered":"2024-03-11T21:35:06.747758+0000","last_clean":"2024-03-11T21:35:06.747758+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:35:06.747758+0000","last_undegraded":"2024-03-11T21:35:06.747758+0000","last_fullsized":"2024-03-11T21:35:06.747758+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:38.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:40.304698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":894,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746937+0000","last_change":"2024-03-11T21:32:23.043898+0000","last_active":"2024-03-11T21:35:06.746937+0000","last_peered":"2024-03-11T21:35:06.746937+0000","last_clean":"2024-03-11T21:35:06.746937+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:35:06.746937+0000","last_undegraded":"2024-03-11T21:35:06.746937+0000","last_fullsized":"2024-03-11T21:35:06.746937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:32:23.043658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.043658+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":769,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.537518+0000","last_change":"2024-03-11T21:34:48.064829+0000","last_active":"2024-03-11T21:35:09.537518+0000","last_peered":"2024-03-11T21:35:09.537518+0000","last_clean":"2024-03-11T21:35:09.537518+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:35:09.537518+0000","last_undegraded":"2024-03-11T21:35:09.537518+0000","last_fullsized":"2024-03-11T21:35:09.537518+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.064622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.064622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.307857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":879,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354481+0000","last_change":"2024-03-11T21:34:40.424576+0000","last_active":"2024-03-11T21:35:06.354481+0000","last_peered":"2024-03-11T21:35:06.354481+0000","last_clean":"2024-03-11T21:35:06.354481+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:35:06.354481+0000","last_undegraded":"2024-03-11T21:35:06.354481+0000","last_fullsized":"2024-03-11T21:35:06.354481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.424446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.424446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.552331+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971978+0000","last_change":"2024-03-11T21:35:02.012107+0000","last_active":"2024-03-11T21:35:06.971978+0000","last_peered":"2024-03-11T21:35:06.971978+0000","last_clean":"2024-03-11T21:35:06.971978+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:35:06.971978+0000","last_undegraded":"2024-03-11T21:35:06.971978+0000","last_fullsized":"2024-03-11T21:35:06.971978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.011982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.011982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:27.342539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":713,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747179+0000","last_change":"2024-03-11T21:33:22.548838+0000","last_active":"2024-03-11T21:35:06.747179+0000","last_peered":"2024-03-11T21:35:06.747179+0000","last_clean":"2024-03-11T21:35:06.747179+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:35:06.747179+0000","last_undegraded":"2024-03-11T21:35:06.747179+0000","last_fullsized":"2024-03-11T21:35:06.747179+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.548708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.548708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":921,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974367+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:35:06.974367+0000","last_peered":"2024-03-11T21:35:06.974367+0000","last_clean":"2024-03-11T21:35:06.974367+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:35:06.974367+0000","last_undegraded":"2024-03-11T21:35:06.974367+0000","last_fullsized":"2024-03-11T21:35:06.974367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":887,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355408+0000","last_change":"2024-03-11T21:34:46.463590+0000","last_active":"2024-03-11T21:35:06.355408+0000","last_peered":"2024-03-11T21:35:06.355408+0000","last_clean":"2024-03-11T21:35:06.355408+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:35:06.355408+0000","last_undegraded":"2024-03-11T21:35:06.355408+0000","last_fullsized":"2024-03-11T21:35:06.355408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:46.463490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:34:46.463490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.211498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":879,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354730+0000","last_change":"2024-03-11T21:34:43.484238+0000","last_active":"2024-03-11T21:35:06.354730+0000","last_peered":"2024-03-11T21:35:06.354730+0000","last_clean":"2024-03-11T21:35:06.354730+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:35:06.354730+0000","last_undegraded":"2024-03-11T21:35:06.354730+0000","last_fullsized":"2024-03-11T21:35:06.354730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.484121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.484121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:07.084724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":848,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.967525+0000","last_change":"2024-03-11T21:30:24.467913+0000","last_active":"2024-03-11T21:35:06.967525+0000","last_peered":"2024-03-11T21:35:06.967525+0000","last_clean":"2024-03-11T21:35:06.967525+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:35:06.967525+0000","last_undegraded":"2024-03-11T21:35:06.967525+0000","last_fullsized":"2024-03-11T21:35:06.967525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:24.467781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.467781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":588,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.533400+0000","last_change":"2024-03-11T21:33:39.131584+0000","last_active":"2024-03-11T21:35:09.533400+0000","last_peered":"2024-03-11T21:35:09.533400+0000","last_clean":"2024-03-11T21:35:09.533400+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:35:09.533400+0000","last_undegraded":"2024-03-11T21:35:09.533400+0000","last_fullsized":"2024-03-11T21:35:09.533400+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.131510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.131510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":946,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473427+0000","last_change":"2024-03-11T21:33:46.396054+0000","last_active":"2024-03-11T21:35:06.473427+0000","last_peered":"2024-03-11T21:35:06.473427+0000","last_clean":"2024-03-11T21:35:06.473427+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:35:06.473427+0000","last_undegraded":"2024-03-11T21:35:06.473427+0000","last_fullsized":"2024-03-11T21:35:06.473427+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:33:46.395671+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.395671+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.312098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":868,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971405+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:35:06.971405+0000","last_peered":"2024-03-11T21:35:06.971405+0000","last_clean":"2024-03-11T21:35:06.971405+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:35:06.971405+0000","last_undegraded":"2024-03-11T21:35:06.971405+0000","last_fullsized":"2024-03-11T21:35:06.971405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":767,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538579+0000","last_change":"2024-03-11T21:33:13.942454+0000","last_active":"2024-03-11T21:35:09.538579+0000","last_peered":"2024-03-11T21:35:09.538579+0000","last_clean":"2024-03-11T21:35:09.538579+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:35:09.538579+0000","last_undegraded":"2024-03-11T21:35:09.538579+0000","last_fullsized":"2024-03-11T21:35:09.538579+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.942158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.942158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":875,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357324+0000","last_change":"2024-03-11T21:34:33.339535+0000","last_active":"2024-03-11T21:35:06.357324+0000","last_peered":"2024-03-11T21:35:06.357324+0000","last_clean":"2024-03-11T21:35:06.357324+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:35:06.357324+0000","last_undegraded":"2024-03-11T21:35:06.357324+0000","last_fullsized":"2024-03-11T21:35:06.357324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.339234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.339234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:00.073588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":721,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477648+0000","last_change":"2024-03-11T21:32:14.697394+0000","last_active":"2024-03-11T21:35:06.477648+0000","last_peered":"2024-03-11T21:35:06.477648+0000","last_clean":"2024-03-11T21:35:06.477648+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:35:06.477648+0000","last_undegraded":"2024-03-11T21:35:06.477648+0000","last_fullsized":"2024-03-11T21:35:06.477648+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.697115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":897,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478064+0000","last_change":"2024-03-11T21:35:01.534298+0000","last_active":"2024-03-11T21:35:06.478064+0000","last_peered":"2024-03-11T21:35:06.478064+0000","last_clean":"2024-03-11T21:35:06.478064+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:35:06.478064+0000","last_undegraded":"2024-03-11T21:35:06.478064+0000","last_fullsized":"2024-03-11T21:35:06.478064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:01.534131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.534131+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:15.229592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":735,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.534550+0000","last_change":"2024-03-11T21:34:41.173845+0000","last_active":"2024-03-11T21:35:09.534550+0000","last_peered":"2024-03-11T21:35:09.534550+0000","last_clean":"2024-03-11T21:35:09.534550+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:35:09.534550+0000","last_undegraded":"2024-03-11T21:35:09.534550+0000","last_fullsized":"2024-03-11T21:35:09.534550+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:41.416478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1035,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357645+0000","last_change":"2024-03-11T21:33:41.205378+0000","last_active":"2024-03-11T21:35:06.357645+0000","last_peered":"2024-03-11T21:35:06.357645+0000","last_clean":"2024-03-11T21:35:06.357645+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:35:06.357645+0000","last_undegraded":"2024-03-11T21:35:06.357645+0000","last_fullsized":"2024-03-11T21:35:06.357645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:33:41.205243+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.205243+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:07.172685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":876,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354187+0000","last_change":"2024-03-11T21:35:01.196442+0000","last_active":"2024-03-11T21:35:06.354187+0000","last_peered":"2024-03-11T21:35:06.354187+0000","last_clean":"2024-03-11T21:35:06.354187+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:35:06.354187+0000","last_undegraded":"2024-03-11T21:35:06.354187+0000","last_fullsized":"2024-03-11T21:35:06.354187+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.196310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.196310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:12.718777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":724,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478720+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:35:06.478720+0000","last_peered":"2024-03-11T21:35:06.478720+0000","last_clean":"2024-03-11T21:35:06.478720+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:35:06.478720+0000","last_undegraded":"2024-03-11T21:35:06.478720+0000","last_fullsized":"2024-03-11T21:35:06.478720+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":916,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478721+0000","last_change":"2024-03-11T21:33:20.697629+0000","last_active":"2024-03-11T21:35:06.478721+0000","last_peered":"2024-03-11T21:35:06.478721+0000","last_clean":"2024-03-11T21:35:06.478721+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:35:06.478721+0000","last_undegraded":"2024-03-11T21:35:06.478721+0000","last_fullsized":"2024-03-11T21:35:06.478721+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:20.697247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.697247+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":755,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.536523+0000","last_change":"2024-03-11T21:32:18.957178+0000","last_active":"2024-03-11T21:35:09.536523+0000","last_peered":"2024-03-11T21:35:09.536523+0000","last_clean":"2024-03-11T21:35:09.536523+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:35:09.536523+0000","last_undegraded":"2024-03-11T21:35:09.536523+0000","last_fullsized":"2024-03-11T21:35:09.536523+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.957068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.957068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1026,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.191921+0000","last_change":"2024-03-11T21:35:07.191988+0000","last_active":"2024-03-11T21:35:07.191921+0000","last_peered":"2024-03-11T21:35:07.191921+0000","last_clean":"2024-03-11T21:35:07.191921+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:35:07.191921+0000","last_undegraded":"2024-03-11T21:35:07.191921+0000","last_fullsized":"2024-03-11T21:35:07.191921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:35:07.191770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.191770+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:35.791017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":873,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746884+0000","last_change":"2024-03-11T21:34:42.857662+0000","last_active":"2024-03-11T21:35:06.746884+0000","last_peered":"2024-03-11T21:35:06.746884+0000","last_clean":"2024-03-11T21:35:06.746884+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:35:06.746884+0000","last_undegraded":"2024-03-11T21:35:06.746884+0000","last_fullsized":"2024-03-11T21:35:06.746884+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.857448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.857448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.706469+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":715,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.970388+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:35:06.970388+0000","last_peered":"2024-03-11T21:35:06.970388+0000","last_clean":"2024-03-11T21:35:06.970388+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:35:06.970388+0000","last_undegraded":"2024-03-11T21:35:06.970388+0000","last_fullsized":"2024-03-11T21:35:06.970388+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":904,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349472+0000","last_change":"2024-03-11T21:33:35.635520+0000","last_active":"2024-03-11T21:35:06.349472+0000","last_peered":"2024-03-11T21:35:06.349472+0000","last_clean":"2024-03-11T21:35:06.349472+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:35:06.349472+0000","last_undegraded":"2024-03-11T21:35:06.349472+0000","last_fullsized":"2024-03-11T21:35:06.349472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.635256+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":877,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.967914+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:35:06.967914+0000","last_peered":"2024-03-11T21:35:06.967914+0000","last_clean":"2024-03-11T21:35:06.967914+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:35:06.967914+0000","last_undegraded":"2024-03-11T21:35:06.967914+0000","last_fullsized":"2024-03-11T21:35:06.967914+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":860,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747728+0000","last_change":"2024-03-11T21:32:19.961583+0000","last_active":"2024-03-11T21:35:06.747728+0000","last_peered":"2024-03-11T21:35:06.747728+0000","last_clean":"2024-03-11T21:35:06.747728+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:35:06.747728+0000","last_undegraded":"2024-03-11T21:35:06.747728+0000","last_fullsized":"2024-03-11T21:35:06.747728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.961475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.961475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":848,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478429+0000","last_change":"2024-03-11T21:32:22.807836+0000","last_active":"2024-03-11T21:35:06.478429+0000","last_peered":"2024-03-11T21:35:06.478429+0000","last_clean":"2024-03-11T21:35:06.478429+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:35:06.478429+0000","last_undegraded":"2024-03-11T21:35:06.478429+0000","last_fullsized":"2024-03-11T21:35:06.478429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.807727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.807727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":706,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478687+0000","last_change":"2024-03-11T21:32:11.663654+0000","last_active":"2024-03-11T21:35:06.478687+0000","last_peered":"2024-03-11T21:35:06.478687+0000","last_clean":"2024-03-11T21:35:06.478687+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:35:06.478687+0000","last_undegraded":"2024-03-11T21:35:06.478687+0000","last_fullsized":"2024-03-11T21:35:06.478687+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:11.663532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:32:11.663532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1026,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478948+0000","last_change":"2024-03-11T21:33:10.709056+0000","last_active":"2024-03-11T21:35:06.478948+0000","last_peered":"2024-03-11T21:35:06.478948+0000","last_clean":"2024-03-11T21:35:06.478948+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:35:06.478948+0000","last_undegraded":"2024-03-11T21:35:06.478948+0000","last_fullsized":"2024-03-11T21:35:06.478948+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:33:10.708723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.708723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349702+0000","last_change":"2024-03-11T21:33:37.698300+0000","last_active":"2024-03-11T21:35:06.349702+0000","last_peered":"2024-03-11T21:35:06.349702+0000","last_clean":"2024-03-11T21:35:06.349702+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:35:06.349702+0000","last_undegraded":"2024-03-11T21:35:06.349702+0000","last_fullsized":"2024-03-11T21:35:06.349702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:37.698095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.698095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":920,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976361+0000","last_change":"2024-03-11T21:34:53.090278+0000","last_active":"2024-03-11T21:35:06.976361+0000","last_peered":"2024-03-11T21:35:06.976361+0000","last_clean":"2024-03-11T21:35:06.976361+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:35:06.976361+0000","last_undegraded":"2024-03-11T21:35:06.976361+0000","last_fullsized":"2024-03-11T21:35:06.976361+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:53.090147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.090147+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:09.943805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":894,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.475750+0000","last_change":"2024-03-11T21:35:00.409061+0000","last_active":"2024-03-11T21:35:06.475750+0000","last_peered":"2024-03-11T21:35:06.475750+0000","last_clean":"2024-03-11T21:35:06.475750+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:35:06.475750+0000","last_undegraded":"2024-03-11T21:35:06.475750+0000","last_fullsized":"2024-03-11T21:35:06.475750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.408810+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.799625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":858,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.970835+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:35:06.970835+0000","last_peered":"2024-03-11T21:35:06.970835+0000","last_clean":"2024-03-11T21:35:06.970835+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:35:06.970835+0000","last_undegraded":"2024-03-11T21:35:06.970835+0000","last_fullsized":"2024-03-11T21:35:06.970835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973857+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:35:06.973857+0000","last_peered":"2024-03-11T21:35:06.973857+0000","last_clean":"2024-03-11T21:35:06.973857+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:35:06.973857+0000","last_undegraded":"2024-03-11T21:35:06.973857+0000","last_fullsized":"2024-03-11T21:35:06.973857+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":871,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349408+0000","last_change":"2024-03-11T21:34:43.908469+0000","last_active":"2024-03-11T21:35:06.349408+0000","last_peered":"2024-03-11T21:35:06.349408+0000","last_clean":"2024-03-11T21:35:06.349408+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:35:06.349408+0000","last_undegraded":"2024-03-11T21:35:06.349408+0000","last_fullsized":"2024-03-11T21:35:06.349408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.908344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.908344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:06.367719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1577,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.489359+0000","last_change":"2024-03-11T21:33:40.550313+0000","last_active":"2024-03-11T21:35:07.489359+0000","last_peered":"2024-03-11T21:35:07.489359+0000","last_clean":"2024-03-11T21:35:07.489359+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:35:07.489359+0000","last_undegraded":"2024-03-11T21:35:07.489359+0000","last_fullsized":"2024-03-11T21:35:07.489359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:33:40.550110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.550110+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":897,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.480468+0000","last_change":"2024-03-11T21:34:33.535935+0000","last_active":"2024-03-11T21:35:06.480468+0000","last_peered":"2024-03-11T21:35:06.480468+0000","last_clean":"2024-03-11T21:35:06.480468+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:35:06.480468+0000","last_undegraded":"2024-03-11T21:35:06.480468+0000","last_fullsized":"2024-03-11T21:35:06.480468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:34:33.535753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.535753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.050650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":780,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.540123+0000","last_change":"2024-03-11T21:33:09.913175+0000","last_active":"2024-03-11T21:35:09.540123+0000","last_peered":"2024-03-11T21:35:09.540123+0000","last_clean":"2024-03-11T21:35:09.540123+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:35:09.540123+0000","last_undegraded":"2024-03-11T21:35:09.540123+0000","last_fullsized":"2024-03-11T21:35:09.540123+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:09.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.912886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350424+0000","last_change":"2024-03-11T21:34:31.810307+0000","last_active":"2024-03-11T21:35:06.350424+0000","last_peered":"2024-03-11T21:35:06.350424+0000","last_clean":"2024-03-11T21:35:06.350424+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:35:06.350424+0000","last_undegraded":"2024-03-11T21:35:06.350424+0000","last_fullsized":"2024-03-11T21:35:06.350424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.810016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.810016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.863820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974044+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:35:06.974044+0000","last_peered":"2024-03-11T21:35:06.974044+0000","last_clean":"2024-03-11T21:35:06.974044+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:35:06.974044+0000","last_undegraded":"2024-03-11T21:35:06.974044+0000","last_fullsized":"2024-03-11T21:35:06.974044+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":869,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473178+0000","last_change":"2024-03-11T21:34:57.364338+0000","last_active":"2024-03-11T21:35:06.473178+0000","last_peered":"2024-03-11T21:35:06.473178+0000","last_clean":"2024-03-11T21:35:06.473178+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:35:06.473178+0000","last_undegraded":"2024-03-11T21:35:06.473178+0000","last_fullsized":"2024-03-11T21:35:06.473178+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.364057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.364057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:27.348356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":970,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.541021+0000","last_change":"2024-03-11T21:34:55.143888+0000","last_active":"2024-03-11T21:35:09.541021+0000","last_peered":"2024-03-11T21:35:09.541021+0000","last_clean":"2024-03-11T21:35:09.541021+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:35:09.541021+0000","last_undegraded":"2024-03-11T21:35:09.541021+0000","last_fullsized":"2024-03-11T21:35:09.541021+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.143614+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.728703+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":868,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974183+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:35:06.974183+0000","last_peered":"2024-03-11T21:35:06.974183+0000","last_clean":"2024-03-11T21:35:06.974183+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:35:06.974183+0000","last_undegraded":"2024-03-11T21:35:06.974183+0000","last_fullsized":"2024-03-11T21:35:06.974183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":727,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.535623+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:35:09.535623+0000","last_peered":"2024-03-11T21:35:09.535623+0000","last_clean":"2024-03-11T21:35:09.535623+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:35:09.535623+0000","last_undegraded":"2024-03-11T21:35:09.535623+0000","last_fullsized":"2024-03-11T21:35:09.535623+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:15.222087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":716,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976685+0000","last_change":"2024-03-11T21:34:35.105891+0000","last_active":"2024-03-11T21:35:06.976685+0000","last_peered":"2024-03-11T21:35:06.976685+0000","last_clean":"2024-03-11T21:35:06.976685+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:35:06.976685+0000","last_undegraded":"2024-03-11T21:35:06.976685+0000","last_fullsized":"2024-03-11T21:35:06.976685+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.105714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.105714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.841002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":877,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747687+0000","last_change":"2024-03-11T21:35:03.774565+0000","last_active":"2024-03-11T21:35:06.747687+0000","last_peered":"2024-03-11T21:35:06.747687+0000","last_clean":"2024-03-11T21:35:06.747687+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:35:06.747687+0000","last_undegraded":"2024-03-11T21:35:06.747687+0000","last_fullsized":"2024-03-11T21:35:06.747687+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.774354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.774354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:09.117476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":878,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.967589+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:35:06.967589+0000","last_peered":"2024-03-11T21:35:06.967589+0000","last_clean":"2024-03-11T21:35:06.967589+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:35:06.967589+0000","last_undegraded":"2024-03-11T21:35:06.967589+0000","last_fullsized":"2024-03-11T21:35:06.967589+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":909,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971797+0000","last_change":"2024-03-11T21:33:17.014343+0000","last_active":"2024-03-11T21:35:06.971797+0000","last_peered":"2024-03-11T21:35:06.971797+0000","last_clean":"2024-03-11T21:35:06.971797+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:35:06.971797+0000","last_undegraded":"2024-03-11T21:35:06.971797+0000","last_fullsized":"2024-03-11T21:35:06.971797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:17.014179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.014179+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":737,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.539663+0000","last_change":"2024-03-11T21:35:04.076881+0000","last_active":"2024-03-11T21:35:09.539663+0000","last_peered":"2024-03-11T21:35:09.539663+0000","last_clean":"2024-03-11T21:35:09.539663+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:35:09.539663+0000","last_undegraded":"2024-03-11T21:35:09.539663+0000","last_fullsized":"2024-03-11T21:35:09.539663+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.076789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.076789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.593356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478330+0000","last_change":"2024-03-11T21:34:54.501393+0000","last_active":"2024-03-11T21:35:06.478330+0000","last_peered":"2024-03-11T21:35:06.478330+0000","last_clean":"2024-03-11T21:35:06.478330+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:35:06.478330+0000","last_undegraded":"2024-03-11T21:35:06.478330+0000","last_fullsized":"2024-03-11T21:35:06.478330+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.501183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.501183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.241950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":852,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972413+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:35:06.972413+0000","last_peered":"2024-03-11T21:35:06.972413+0000","last_clean":"2024-03-11T21:35:06.972413+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:35:06.972413+0000","last_undegraded":"2024-03-11T21:35:06.972413+0000","last_fullsized":"2024-03-11T21:35:06.972413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1124,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.967866+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:35:06.967866+0000","last_peered":"2024-03-11T21:35:06.967866+0000","last_clean":"2024-03-11T21:35:06.967866+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:35:06.967866+0000","last_undegraded":"2024-03-11T21:35:06.967866+0000","last_fullsized":"2024-03-11T21:35:06.967866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":876,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746479+0000","last_change":"2024-03-11T21:33:13.461458+0000","last_active":"2024-03-11T21:35:06.746479+0000","last_peered":"2024-03-11T21:35:06.746479+0000","last_clean":"2024-03-11T21:35:06.746479+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:35:06.746479+0000","last_undegraded":"2024-03-11T21:35:06.746479+0000","last_fullsized":"2024-03-11T21:35:06.746479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:13.461214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.461214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":894,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478088+0000","last_change":"2024-03-11T21:34:38.596842+0000","last_active":"2024-03-11T21:35:06.478088+0000","last_peered":"2024-03-11T21:35:06.478088+0000","last_clean":"2024-03-11T21:35:06.478088+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:35:06.478088+0000","last_undegraded":"2024-03-11T21:35:06.478088+0000","last_fullsized":"2024-03-11T21:35:06.478088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:34:38.596731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.596731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.505236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":864,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747759+0000","last_change":"2024-03-11T21:34:57.751893+0000","last_active":"2024-03-11T21:35:06.747759+0000","last_peered":"2024-03-11T21:35:06.747759+0000","last_clean":"2024-03-11T21:35:06.747759+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:35:06.747759+0000","last_undegraded":"2024-03-11T21:35:06.747759+0000","last_fullsized":"2024-03-11T21:35:06.747759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.751652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.870223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745789+0000","last_change":"2024-03-11T21:34:36.837312+0000","last_active":"2024-03-11T21:35:06.745789+0000","last_peered":"2024-03-11T21:35:06.745789+0000","last_clean":"2024-03-11T21:35:06.745789+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:35:06.745789+0000","last_undegraded":"2024-03-11T21:35:06.745789+0000","last_fullsized":"2024-03-11T21:35:06.745789+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.837081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.837081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.917829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":879,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356065+0000","last_change":"2024-03-11T21:33:43.147050+0000","last_active":"2024-03-11T21:35:06.356065+0000","last_peered":"2024-03-11T21:35:06.356065+0000","last_clean":"2024-03-11T21:35:06.356065+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:35:06.356065+0000","last_undegraded":"2024-03-11T21:35:06.356065+0000","last_fullsized":"2024-03-11T21:35:06.356065+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.146852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.146852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":887,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356435+0000","last_change":"2024-03-11T21:34:44.457291+0000","last_active":"2024-03-11T21:35:06.356435+0000","last_peered":"2024-03-11T21:35:06.356435+0000","last_clean":"2024-03-11T21:35:06.356435+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:35:06.356435+0000","last_undegraded":"2024-03-11T21:35:06.356435+0000","last_fullsized":"2024-03-11T21:35:06.356435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.457161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.457161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.611516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":811,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.534181+0000","last_change":"2024-03-11T21:32:20.982556+0000","last_active":"2024-03-11T21:35:09.534181+0000","last_peered":"2024-03-11T21:35:09.534181+0000","last_clean":"2024-03-11T21:35:09.534181+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:35:09.534181+0000","last_undegraded":"2024-03-11T21:35:09.534181+0000","last_fullsized":"2024-03-11T21:35:09.534181+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'26","last_scrub_stamp":"2024-03-11T21:32:20.982418+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.982418+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973998+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:35:06.973998+0000","last_peered":"2024-03-11T21:35:06.973998+0000","last_clean":"2024-03-11T21:35:06.973998+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:35:06.973998+0000","last_undegraded":"2024-03-11T21:35:06.973998+0000","last_fullsized":"2024-03-11T21:35:06.973998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":716,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971646+0000","last_change":"2024-03-11T21:33:40.988530+0000","last_active":"2024-03-11T21:35:06.971646+0000","last_peered":"2024-03-11T21:35:06.971646+0000","last_clean":"2024-03-11T21:35:06.971646+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:35:06.971646+0000","last_undegraded":"2024-03-11T21:35:06.971646+0000","last_fullsized":"2024-03-11T21:35:06.971646+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.988258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.988258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":872,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976190+0000","last_change":"2024-03-11T21:33:35.976814+0000","last_active":"2024-03-11T21:35:06.976190+0000","last_peered":"2024-03-11T21:35:06.976190+0000","last_clean":"2024-03-11T21:35:06.976190+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:35:06.976190+0000","last_undegraded":"2024-03-11T21:35:06.976190+0000","last_fullsized":"2024-03-11T21:35:06.976190+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.976718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.976718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350229+0000","last_change":"2024-03-11T21:34:39.946045+0000","last_active":"2024-03-11T21:35:06.350229+0000","last_peered":"2024-03-11T21:35:06.350229+0000","last_clean":"2024-03-11T21:35:06.350229+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:35:06.350229+0000","last_undegraded":"2024-03-11T21:35:06.350229+0000","last_fullsized":"2024-03-11T21:35:06.350229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:34:39.945944+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.945944+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:01.612882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":877,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746317+0000","last_change":"2024-03-11T21:33:09.417056+0000","last_active":"2024-03-11T21:35:06.746317+0000","last_peered":"2024-03-11T21:35:06.746317+0000","last_clean":"2024-03-11T21:35:06.746317+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:35:06.746317+0000","last_undegraded":"2024-03-11T21:35:06.746317+0000","last_fullsized":"2024-03-11T21:35:06.746317+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.416868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.416868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479570+0000","last_change":"2024-03-11T21:33:33.610158+0000","last_active":"2024-03-11T21:35:06.479570+0000","last_peered":"2024-03-11T21:35:06.479570+0000","last_clean":"2024-03-11T21:35:06.479570+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:35:06.479570+0000","last_undegraded":"2024-03-11T21:35:06.479570+0000","last_fullsized":"2024-03-11T21:35:06.479570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:33.610071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:33.610071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747303+0000","last_change":"2024-03-11T21:34:45.790219+0000","last_active":"2024-03-11T21:35:06.747303+0000","last_peered":"2024-03-11T21:35:06.747303+0000","last_clean":"2024-03-11T21:35:06.747303+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:35:06.747303+0000","last_undegraded":"2024-03-11T21:35:06.747303+0000","last_fullsized":"2024-03-11T21:35:06.747303+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.790012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.790012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.607134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1030,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976565+0000","last_change":"2024-03-11T21:34:52.046581+0000","last_active":"2024-03-11T21:35:06.976565+0000","last_peered":"2024-03-11T21:35:06.976565+0000","last_clean":"2024-03-11T21:35:06.976565+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:35:06.976565+0000","last_undegraded":"2024-03-11T21:35:06.976565+0000","last_fullsized":"2024-03-11T21:35:06.976565+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:34:52.046352+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.046352+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:05.374888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":764,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.533676+0000","last_change":"2024-03-11T21:33:31.047676+0000","last_active":"2024-03-11T21:35:09.533676+0000","last_peered":"2024-03-11T21:35:09.533676+0000","last_clean":"2024-03-11T21:35:09.533676+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:35:09.533676+0000","last_undegraded":"2024-03-11T21:35:09.533676+0000","last_fullsized":"2024-03-11T21:35:09.533676+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:33:31.047447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.047447+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1057,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746955+0000","last_change":"2024-03-11T21:33:37.464613+0000","last_active":"2024-03-11T21:35:06.746955+0000","last_peered":"2024-03-11T21:35:06.746955+0000","last_clean":"2024-03-11T21:35:06.746955+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:35:06.746955+0000","last_undegraded":"2024-03-11T21:35:06.746955+0000","last_fullsized":"2024-03-11T21:35:06.746955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:33:37.464485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:37.464485+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":735,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.539127+0000","last_change":"2024-03-11T21:33:23.993323+0000","last_active":"2024-03-11T21:35:09.539127+0000","last_peered":"2024-03-11T21:35:09.539127+0000","last_clean":"2024-03-11T21:35:09.539127+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:35:09.539127+0000","last_undegraded":"2024-03-11T21:35:09.539127+0000","last_fullsized":"2024-03-11T21:35:09.539127+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.993242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.993242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":866,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972575+0000","last_change":"2024-03-11T21:34:57.129410+0000","last_active":"2024-03-11T21:35:06.972575+0000","last_peered":"2024-03-11T21:35:06.972575+0000","last_clean":"2024-03-11T21:35:06.972575+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:35:06.972575+0000","last_undegraded":"2024-03-11T21:35:06.972575+0000","last_fullsized":"2024-03-11T21:35:06.972575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.129296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.129296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.365908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":588,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.539401+0000","last_change":"2024-03-11T21:33:43.133706+0000","last_active":"2024-03-11T21:35:09.539401+0000","last_peered":"2024-03-11T21:35:09.539401+0000","last_clean":"2024-03-11T21:35:09.539401+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:35:09.539401+0000","last_undegraded":"2024-03-11T21:35:09.539401+0000","last_fullsized":"2024-03-11T21:35:09.539401+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.133485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.133485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":888,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.357556+0000","last_change":"2024-03-11T21:34:36.323730+0000","last_active":"2024-03-11T21:35:06.357556+0000","last_peered":"2024-03-11T21:35:06.357556+0000","last_clean":"2024-03-11T21:35:06.357556+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:35:06.357556+0000","last_undegraded":"2024-03-11T21:35:06.357556+0000","last_fullsized":"2024-03-11T21:35:06.357556+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.323616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.323616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.807642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":917,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974464+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:35:06.974464+0000","last_peered":"2024-03-11T21:35:06.974464+0000","last_clean":"2024-03-11T21:35:06.974464+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:35:06.974464+0000","last_undegraded":"2024-03-11T21:35:06.974464+0000","last_fullsized":"2024-03-11T21:35:06.974464+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.849278+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":897,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354668+0000","last_change":"2024-03-11T21:34:36.855849+0000","last_active":"2024-03-11T21:35:06.354668+0000","last_peered":"2024-03-11T21:35:06.354668+0000","last_clean":"2024-03-11T21:35:06.354668+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:35:06.354668+0000","last_undegraded":"2024-03-11T21:35:06.354668+0000","last_fullsized":"2024-03-11T21:35:06.354668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:34:36.855625+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.855625+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.347567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":865,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478274+0000","last_change":"2024-03-11T21:33:14.693542+0000","last_active":"2024-03-11T21:35:06.478274+0000","last_peered":"2024-03-11T21:35:06.478274+0000","last_clean":"2024-03-11T21:35:06.478274+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:35:06.478274+0000","last_undegraded":"2024-03-11T21:35:06.478274+0000","last_fullsized":"2024-03-11T21:35:06.478274+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:14.693316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.693316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478436+0000","last_change":"2024-03-11T21:33:42.591082+0000","last_active":"2024-03-11T21:35:06.478436+0000","last_peered":"2024-03-11T21:35:06.478436+0000","last_clean":"2024-03-11T21:35:06.478436+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:35:06.478436+0000","last_undegraded":"2024-03-11T21:35:06.478436+0000","last_fullsized":"2024-03-11T21:35:06.478436+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:42.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.590963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":895,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.969522+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:35:06.969522+0000","last_peered":"2024-03-11T21:35:06.969522+0000","last_clean":"2024-03-11T21:35:06.969522+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:35:06.969522+0000","last_undegraded":"2024-03-11T21:35:06.969522+0000","last_fullsized":"2024-03-11T21:35:06.969522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":896,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.356090+0000","last_change":"2024-03-11T21:34:50.366890+0000","last_active":"2024-03-11T21:35:06.356090+0000","last_peered":"2024-03-11T21:35:06.356090+0000","last_clean":"2024-03-11T21:35:06.356090+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:35:06.356090+0000","last_undegraded":"2024-03-11T21:35:06.356090+0000","last_fullsized":"2024-03-11T21:35:06.356090+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:50.366805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.366805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.421230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":868,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.480285+0000","last_change":"2024-03-11T21:33:32.575760+0000","last_active":"2024-03-11T21:35:06.480285+0000","last_peered":"2024-03-11T21:35:06.480285+0000","last_clean":"2024-03-11T21:35:06.480285+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:35:06.480285+0000","last_undegraded":"2024-03-11T21:35:06.480285+0000","last_fullsized":"2024-03-11T21:35:06.480285+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.575652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.575652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":883,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355773+0000","last_change":"2024-03-11T21:35:05.456150+0000","last_active":"2024-03-11T21:35:06.355773+0000","last_peered":"2024-03-11T21:35:06.355773+0000","last_clean":"2024-03-11T21:35:06.355773+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:35:06.355773+0000","last_undegraded":"2024-03-11T21:35:06.355773+0000","last_fullsized":"2024-03-11T21:35:06.355773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.183993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478780+0000","last_change":"2024-03-11T21:34:39.587604+0000","last_active":"2024-03-11T21:35:06.478780+0000","last_peered":"2024-03-11T21:35:06.478780+0000","last_clean":"2024-03-11T21:35:06.478780+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:35:06.478780+0000","last_undegraded":"2024-03-11T21:35:06.478780+0000","last_fullsized":"2024-03-11T21:35:06.478780+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.587525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.587525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.756876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":895,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.472865+0000","last_change":"2024-03-11T21:34:46.190536+0000","last_active":"2024-03-11T21:35:06.472865+0000","last_peered":"2024-03-11T21:35:06.472865+0000","last_clean":"2024-03-11T21:35:06.472865+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:35:06.472865+0000","last_undegraded":"2024-03-11T21:35:06.472865+0000","last_fullsized":"2024-03-11T21:35:06.472865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:34:46.190301+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:34:46.190301+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.316002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":886,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.474606+0000","last_change":"2024-03-11T21:34:31.418213+0000","last_active":"2024-03-11T21:35:06.474606+0000","last_peered":"2024-03-11T21:35:06.474606+0000","last_clean":"2024-03-11T21:35:06.474606+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:35:06.474606+0000","last_undegraded":"2024-03-11T21:35:06.474606+0000","last_fullsized":"2024-03-11T21:35:06.474606+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.417992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.417992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.992829+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":876,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.748098+0000","last_change":"2024-03-11T21:34:58.788887+0000","last_active":"2024-03-11T21:35:06.748098+0000","last_peered":"2024-03-11T21:35:06.748098+0000","last_clean":"2024-03-11T21:35:06.748098+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:35:06.748098+0000","last_undegraded":"2024-03-11T21:35:06.748098+0000","last_fullsized":"2024-03-11T21:35:06.748098+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.788749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.968662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":855,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973744+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:35:06.973744+0000","last_peered":"2024-03-11T21:35:06.973744+0000","last_clean":"2024-03-11T21:35:06.973744+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:35:06.973744+0000","last_undegraded":"2024-03-11T21:35:06.973744+0000","last_fullsized":"2024-03-11T21:35:06.973744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478555+0000","last_change":"2024-03-11T21:34:53.520431+0000","last_active":"2024-03-11T21:35:06.478555+0000","last_peered":"2024-03-11T21:35:06.478555+0000","last_clean":"2024-03-11T21:35:06.478555+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:35:06.478555+0000","last_undegraded":"2024-03-11T21:35:06.478555+0000","last_fullsized":"2024-03-11T21:35:06.478555+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.520342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.520342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:00.564247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":895,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971749+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:35:06.971749+0000","last_peered":"2024-03-11T21:35:06.971749+0000","last_clean":"2024-03-11T21:35:06.971749+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:35:06.971749+0000","last_undegraded":"2024-03-11T21:35:06.971749+0000","last_fullsized":"2024-03-11T21:35:06.971749+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746133+0000","last_change":"2024-03-11T21:34:31.781546+0000","last_active":"2024-03-11T21:35:06.746133+0000","last_peered":"2024-03-11T21:35:06.746133+0000","last_clean":"2024-03-11T21:35:06.746133+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:35:06.746133+0000","last_undegraded":"2024-03-11T21:35:06.746133+0000","last_fullsized":"2024-03-11T21:35:06.746133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:50.330053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1032,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747264+0000","last_change":"2024-03-11T21:34:55.714432+0000","last_active":"2024-03-11T21:35:06.747264+0000","last_peered":"2024-03-11T21:35:06.747264+0000","last_clean":"2024-03-11T21:35:06.747264+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:35:06.747264+0000","last_undegraded":"2024-03-11T21:35:06.747264+0000","last_fullsized":"2024-03-11T21:35:06.747264+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:34:55.714209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.714209+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:21.968789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":866,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976669+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:35:06.976669+0000","last_peered":"2024-03-11T21:35:06.976669+0000","last_clean":"2024-03-11T21:35:06.976669+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:35:06.976669+0000","last_undegraded":"2024-03-11T21:35:06.976669+0000","last_fullsized":"2024-03-11T21:35:06.976669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:09.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":729,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473330+0000","last_change":"2024-03-11T21:35:03.464789+0000","last_active":"2024-03-11T21:35:06.473330+0000","last_peered":"2024-03-11T21:35:06.473330+0000","last_clean":"2024-03-11T21:35:06.473330+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:35:06.473330+0000","last_undegraded":"2024-03-11T21:35:06.473330+0000","last_fullsized":"2024-03-11T21:35:06.473330+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.464695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.464695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.993166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":904,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973051+0000","last_change":"2024-03-11T21:33:42.968747+0000","last_active":"2024-03-11T21:35:06.973051+0000","last_peered":"2024-03-11T21:35:06.973051+0000","last_clean":"2024-03-11T21:35:06.973051+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:35:06.973051+0000","last_undegraded":"2024-03-11T21:35:06.973051+0000","last_fullsized":"2024-03-11T21:35:06.973051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":879,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976038+0000","last_change":"2024-03-11T21:34:37.079514+0000","last_active":"2024-03-11T21:35:06.976038+0000","last_peered":"2024-03-11T21:35:06.976038+0000","last_clean":"2024-03-11T21:35:06.976038+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:35:06.976038+0000","last_undegraded":"2024-03-11T21:35:06.976038+0000","last_fullsized":"2024-03-11T21:35:06.976038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.079361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.079361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.776119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1016,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478289+0000","last_change":"2024-03-11T21:33:19.697889+0000","last_active":"2024-03-11T21:35:06.478289+0000","last_peered":"2024-03-11T21:35:06.478289+0000","last_clean":"2024-03-11T21:35:06.478289+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:35:06.478289+0000","last_undegraded":"2024-03-11T21:35:06.478289+0000","last_fullsized":"2024-03-11T21:35:06.478289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:33:19.697575+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.697575+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":866,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.355121+0000","last_change":"2024-03-11T21:33:40.730218+0000","last_active":"2024-03-11T21:35:06.355121+0000","last_peered":"2024-03-11T21:35:06.355121+0000","last_clean":"2024-03-11T21:35:06.355121+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:35:06.355121+0000","last_undegraded":"2024-03-11T21:35:06.355121+0000","last_fullsized":"2024-03-11T21:35:06.355121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.729858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.729858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":584,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.541221+0000","last_change":"2024-03-11T21:33:26.034503+0000","last_active":"2024-03-11T21:35:09.541221+0000","last_peered":"2024-03-11T21:35:09.541221+0000","last_clean":"2024-03-11T21:35:09.541221+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:35:09.541221+0000","last_undegraded":"2024-03-11T21:35:09.541221+0000","last_fullsized":"2024-03-11T21:35:09.541221+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:26.034377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:26.034377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":892,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974679+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:35:06.974679+0000","last_peered":"2024-03-11T21:35:06.974679+0000","last_clean":"2024-03-11T21:35:06.974679+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:35:06.974679+0000","last_undegraded":"2024-03-11T21:35:06.974679+0000","last_fullsized":"2024-03-11T21:35:06.974679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":885,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473298+0000","last_change":"2024-03-11T21:34:37.332784+0000","last_active":"2024-03-11T21:35:06.473298+0000","last_peered":"2024-03-11T21:35:06.473298+0000","last_clean":"2024-03-11T21:35:06.473298+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:35:06.473298+0000","last_undegraded":"2024-03-11T21:35:06.473298+0000","last_fullsized":"2024-03-11T21:35:06.473298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.332577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.332577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.761682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":869,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478298+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:35:06.478298+0000","last_peered":"2024-03-11T21:35:06.478298+0000","last_clean":"2024-03-11T21:35:06.478298+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:35:06.478298+0000","last_undegraded":"2024-03-11T21:35:06.478298+0000","last_fullsized":"2024-03-11T21:35:06.478298+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.579186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":875,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972314+0000","last_change":"2024-03-11T21:34:41.069563+0000","last_active":"2024-03-11T21:35:06.972314+0000","last_peered":"2024-03-11T21:35:06.972314+0000","last_clean":"2024-03-11T21:35:06.972314+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:35:06.972314+0000","last_undegraded":"2024-03-11T21:35:06.972314+0000","last_fullsized":"2024-03-11T21:35:06.972314+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.069432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.069432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:55.125039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":723,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976352+0000","last_change":"2024-03-11T21:33:59.486720+0000","last_active":"2024-03-11T21:35:06.976352+0000","last_peered":"2024-03-11T21:35:06.976352+0000","last_clean":"2024-03-11T21:35:06.976352+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:35:06.976352+0000","last_undegraded":"2024-03-11T21:35:06.976352+0000","last_fullsized":"2024-03-11T21:35:06.976352+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:29.057966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.057966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":905,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.352395+0000","last_change":"2024-03-11T21:33:38.729069+0000","last_active":"2024-03-11T21:35:06.352395+0000","last_peered":"2024-03-11T21:35:06.352395+0000","last_clean":"2024-03-11T21:35:06.352395+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:35:06.352395+0000","last_undegraded":"2024-03-11T21:35:06.352395+0000","last_fullsized":"2024-03-11T21:35:06.352395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:38.728913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:38.728913+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":779,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538905+0000","last_change":"2024-03-11T21:34:44.191646+0000","last_active":"2024-03-11T21:35:09.538905+0000","last_peered":"2024-03-11T21:35:09.538905+0000","last_clean":"2024-03-11T21:35:09.538905+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:35:09.538905+0000","last_undegraded":"2024-03-11T21:35:09.538905+0000","last_fullsized":"2024-03-11T21:35:09.538905+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.191432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.191432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.662638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":747,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.533285+0000","last_change":"2024-03-11T21:34:32.241415+0000","last_active":"2024-03-11T21:35:09.533285+0000","last_peered":"2024-03-11T21:35:09.533285+0000","last_clean":"2024-03-11T21:35:09.533285+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:35:09.533285+0000","last_undegraded":"2024-03-11T21:35:09.533285+0000","last_fullsized":"2024-03-11T21:35:09.533285+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.241159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.241159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.885858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477769+0000","last_change":"2024-03-11T21:33:35.541846+0000","last_active":"2024-03-11T21:35:06.477769+0000","last_peered":"2024-03-11T21:35:06.477769+0000","last_clean":"2024-03-11T21:35:06.477769+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:35:06.477769+0000","last_undegraded":"2024-03-11T21:35:06.477769+0000","last_fullsized":"2024-03-11T21:35:06.477769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.541731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.541731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:08.022661+0000","last_change":"2024-03-11T21:35:08.022701+0000","last_active":"2024-03-11T21:35:08.022661+0000","last_peered":"2024-03-11T21:35:08.022661+0000","last_clean":"2024-03-11T21:35:08.022661+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:35:08.022661+0000","last_undegraded":"2024-03-11T21:35:08.022661+0000","last_fullsized":"2024-03-11T21:35:08.022661+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.022536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.022536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:34.267410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":892,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.977918+0000","last_change":"2024-03-11T21:33:20.954157+0000","last_active":"2024-03-11T21:35:06.977918+0000","last_peered":"2024-03-11T21:35:06.977918+0000","last_clean":"2024-03-11T21:35:06.977918+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:35:06.977918+0000","last_undegraded":"2024-03-11T21:35:06.977918+0000","last_fullsized":"2024-03-11T21:35:06.977918+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:33:20.954022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.954022+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":876,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.748293+0000","last_change":"2024-03-11T21:33:46.468750+0000","last_active":"2024-03-11T21:35:06.748293+0000","last_peered":"2024-03-11T21:35:06.748293+0000","last_clean":"2024-03-11T21:35:06.748293+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:35:06.748293+0000","last_undegraded":"2024-03-11T21:35:06.748293+0000","last_fullsized":"2024-03-11T21:35:06.748293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.468649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.468649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":877,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.474313+0000","last_change":"2024-03-11T21:34:45.185932+0000","last_active":"2024-03-11T21:35:06.474313+0000","last_peered":"2024-03-11T21:35:06.474313+0000","last_clean":"2024-03-11T21:35:06.474313+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:35:06.474313+0000","last_undegraded":"2024-03-11T21:35:06.474313+0000","last_fullsized":"2024-03-11T21:35:06.474313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.185599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.185599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.803583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972785+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:35:06.972785+0000","last_peered":"2024-03-11T21:35:06.972785+0000","last_clean":"2024-03-11T21:35:06.972785+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:35:06.972785+0000","last_undegraded":"2024-03-11T21:35:06.972785+0000","last_fullsized":"2024-03-11T21:35:06.972785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.815187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":725,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974925+0000","last_change":"2024-03-11T21:34:51.090840+0000","last_active":"2024-03-11T21:35:06.974925+0000","last_peered":"2024-03-11T21:35:06.974925+0000","last_clean":"2024-03-11T21:35:06.974925+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:35:06.974925+0000","last_undegraded":"2024-03-11T21:35:06.974925+0000","last_fullsized":"2024-03-11T21:35:06.974925+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:51.090589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:51.090589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.312379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":906,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.748097+0000","last_change":"2024-03-11T21:34:44.838684+0000","last_active":"2024-03-11T21:35:06.748097+0000","last_peered":"2024-03-11T21:35:06.748097+0000","last_clean":"2024-03-11T21:35:06.748097+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:35:06.748097+0000","last_undegraded":"2024-03-11T21:35:06.748097+0000","last_fullsized":"2024-03-11T21:35:06.748097+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:44.838430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.838430+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:11.067144+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.475623+0000","last_change":"2024-03-11T21:34:44.232623+0000","last_active":"2024-03-11T21:35:06.475623+0000","last_peered":"2024-03-11T21:35:06.475623+0000","last_clean":"2024-03-11T21:35:06.475623+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:35:06.475623+0000","last_undegraded":"2024-03-11T21:35:06.475623+0000","last_fullsized":"2024-03-11T21:35:06.475623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.232392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.232392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:08.623885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":852,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968340+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:35:06.968340+0000","last_peered":"2024-03-11T21:35:06.968340+0000","last_clean":"2024-03-11T21:35:06.968340+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:35:06.968340+0000","last_undegraded":"2024-03-11T21:35:06.968340+0000","last_fullsized":"2024-03-11T21:35:06.968340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478511+0000","last_change":"2024-03-11T21:34:42.859611+0000","last_active":"2024-03-11T21:35:06.478511+0000","last_peered":"2024-03-11T21:35:06.478511+0000","last_clean":"2024-03-11T21:35:06.478511+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:35:06.478511+0000","last_undegraded":"2024-03-11T21:35:06.478511+0000","last_fullsized":"2024-03-11T21:35:06.478511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.859451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.859451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.405214+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":705,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971142+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:35:06.971142+0000","last_peered":"2024-03-11T21:35:06.971142+0000","last_clean":"2024-03-11T21:35:06.971142+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:35:06.971142+0000","last_undegraded":"2024-03-11T21:35:06.971142+0000","last_fullsized":"2024-03-11T21:35:06.971142+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"389'27","reported_seq":825,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.536233+0000","last_change":"2024-03-11T21:33:21.967867+0000","last_active":"2024-03-11T21:35:09.536233+0000","last_peered":"2024-03-11T21:35:09.536233+0000","last_clean":"2024-03-11T21:35:09.536233+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:35:09.536233+0000","last_undegraded":"2024-03-11T21:35:09.536233+0000","last_fullsized":"2024-03-11T21:35:09.536233+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:21.967756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.967756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":881,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745254+0000","last_change":"2024-03-11T21:35:05.742077+0000","last_active":"2024-03-11T21:35:06.745254+0000","last_peered":"2024-03-11T21:35:06.745254+0000","last_clean":"2024-03-11T21:35:06.745254+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:35:06.745254+0000","last_undegraded":"2024-03-11T21:35:06.745254+0000","last_fullsized":"2024-03-11T21:35:06.745254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.741828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.741828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:09.997539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1035,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.475891+0000","last_change":"2024-03-11T21:34:52.298009+0000","last_active":"2024-03-11T21:35:06.475891+0000","last_peered":"2024-03-11T21:35:06.475891+0000","last_clean":"2024-03-11T21:35:06.475891+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:35:06.475891+0000","last_undegraded":"2024-03-11T21:35:06.475891+0000","last_fullsized":"2024-03-11T21:35:06.475891+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:34:52.297778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.297778+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:55.751909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":849,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972917+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:35:06.972917+0000","last_peered":"2024-03-11T21:35:06.972917+0000","last_clean":"2024-03-11T21:35:06.972917+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:35:06.972917+0000","last_undegraded":"2024-03-11T21:35:06.972917+0000","last_fullsized":"2024-03-11T21:35:06.972917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745883+0000","last_change":"2024-03-11T21:33:12.438198+0000","last_active":"2024-03-11T21:35:06.745883+0000","last_peered":"2024-03-11T21:35:06.745883+0000","last_clean":"2024-03-11T21:35:06.745883+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:35:06.745883+0000","last_undegraded":"2024-03-11T21:35:06.745883+0000","last_fullsized":"2024-03-11T21:35:06.745883+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.438028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.438028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":905,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478585+0000","last_change":"2024-03-11T21:34:56.556273+0000","last_active":"2024-03-11T21:35:06.478585+0000","last_peered":"2024-03-11T21:35:06.478585+0000","last_clean":"2024-03-11T21:35:06.478585+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:35:06.478585+0000","last_undegraded":"2024-03-11T21:35:06.478585+0000","last_fullsized":"2024-03-11T21:35:06.478585+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:56.556007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.556007+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:23.524059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":869,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478201+0000","last_change":"2024-03-11T21:33:06.710223+0000","last_active":"2024-03-11T21:35:06.478201+0000","last_peered":"2024-03-11T21:35:06.478201+0000","last_clean":"2024-03-11T21:35:06.478201+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:35:06.478201+0000","last_undegraded":"2024-03-11T21:35:06.478201+0000","last_fullsized":"2024-03-11T21:35:06.478201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:06.709989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.709989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":872,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976678+0000","last_change":"2024-03-11T21:35:01.025842+0000","last_active":"2024-03-11T21:35:06.976678+0000","last_peered":"2024-03-11T21:35:06.976678+0000","last_clean":"2024-03-11T21:35:06.976678+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:35:06.976678+0000","last_undegraded":"2024-03-11T21:35:06.976678+0000","last_fullsized":"2024-03-11T21:35:06.976678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.025597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.025597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.762958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":730,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.538473+0000","last_change":"2024-03-11T21:33:05.875547+0000","last_active":"2024-03-11T21:35:09.538473+0000","last_peered":"2024-03-11T21:35:09.538473+0000","last_clean":"2024-03-11T21:35:09.538473+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:35:09.538473+0000","last_undegraded":"2024-03-11T21:35:09.538473+0000","last_fullsized":"2024-03-11T21:35:09.538473+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:05.875434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:05.875434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":705,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968954+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:35:06.968954+0000","last_peered":"2024-03-11T21:35:06.968954+0000","last_clean":"2024-03-11T21:35:06.968954+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:35:06.968954+0000","last_undegraded":"2024-03-11T21:35:06.968954+0000","last_fullsized":"2024-03-11T21:35:06.968954+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":881,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.976185+0000","last_change":"2024-03-11T21:33:19.017659+0000","last_active":"2024-03-11T21:35:06.976185+0000","last_peered":"2024-03-11T21:35:06.976185+0000","last_clean":"2024-03-11T21:35:06.976185+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:35:06.976185+0000","last_undegraded":"2024-03-11T21:35:06.976185+0000","last_fullsized":"2024-03-11T21:35:06.976185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:33:19.017449+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.017449+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.473026+0000","last_change":"2024-03-11T21:34:35.350319+0000","last_active":"2024-03-11T21:35:06.473026+0000","last_peered":"2024-03-11T21:35:06.473026+0000","last_clean":"2024-03-11T21:35:06.473026+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:35:06.473026+0000","last_undegraded":"2024-03-11T21:35:06.473026+0000","last_fullsized":"2024-03-11T21:35:06.473026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.350122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.350122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.565430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":724,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.539364+0000","last_change":"2024-03-11T21:33:36.120790+0000","last_active":"2024-03-11T21:35:09.539364+0000","last_peered":"2024-03-11T21:35:09.539364+0000","last_clean":"2024-03-11T21:35:09.539364+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:35:09.539364+0000","last_undegraded":"2024-03-11T21:35:09.539364+0000","last_fullsized":"2024-03-11T21:35:09.539364+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.120602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.120602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.354197+0000","last_change":"2024-03-11T21:33:41.774829+0000","last_active":"2024-03-11T21:35:06.354197+0000","last_peered":"2024-03-11T21:35:06.354197+0000","last_clean":"2024-03-11T21:35:06.354197+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:35:06.354197+0000","last_undegraded":"2024-03-11T21:35:06.354197+0000","last_fullsized":"2024-03-11T21:35:06.354197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.774699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.774699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.521322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":722,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349117+0000","last_change":"2024-03-11T21:35:04.183348+0000","last_active":"2024-03-11T21:35:06.349117+0000","last_peered":"2024-03-11T21:35:06.349117+0000","last_clean":"2024-03-11T21:35:06.349117+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:35:06.349117+0000","last_undegraded":"2024-03-11T21:35:06.349117+0000","last_fullsized":"2024-03-11T21:35:06.349117+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.183144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.183144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:31.008582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":908,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350626+0000","last_change":"2024-03-11T21:33:27.686917+0000","last_active":"2024-03-11T21:35:06.350626+0000","last_peered":"2024-03-11T21:35:06.350626+0000","last_clean":"2024-03-11T21:35:06.350626+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:35:06.350626+0000","last_undegraded":"2024-03-11T21:35:06.350626+0000","last_fullsized":"2024-03-11T21:35:06.350626+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:33:27.686783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:27.686783+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.478039+0000","last_change":"2024-03-11T21:32:24.831734+0000","last_active":"2024-03-11T21:35:06.478039+0000","last_peered":"2024-03-11T21:35:06.478039+0000","last_clean":"2024-03-11T21:35:06.478039+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:35:06.478039+0000","last_undegraded":"2024-03-11T21:35:06.478039+0000","last_fullsized":"2024-03-11T21:35:06.478039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:24.831533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.831533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":870,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.352677+0000","last_change":"2024-03-11T21:34:57.186736+0000","last_active":"2024-03-11T21:35:06.352677+0000","last_peered":"2024-03-11T21:35:06.352677+0000","last_clean":"2024-03-11T21:35:06.352677+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:35:06.352677+0000","last_undegraded":"2024-03-11T21:35:06.352677+0000","last_fullsized":"2024-03-11T21:35:06.352677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.186509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.186509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:25.347327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.480090+0000","last_change":"2024-03-11T21:34:31.520200+0000","last_active":"2024-03-11T21:35:06.480090+0000","last_peered":"2024-03-11T21:35:06.480090+0000","last_clean":"2024-03-11T21:35:06.480090+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:35:06.480090+0000","last_undegraded":"2024-03-11T21:35:06.480090+0000","last_fullsized":"2024-03-11T21:35:06.480090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.520041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.520041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:58.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972601+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:35:06.972601+0000","last_peered":"2024-03-11T21:35:06.972601+0000","last_clean":"2024-03-11T21:35:06.972601+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:35:06.972601+0000","last_undegraded":"2024-03-11T21:35:06.972601+0000","last_fullsized":"2024-03-11T21:35:06.972601+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":903,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747130+0000","last_change":"2024-03-11T21:34:50.794115+0000","last_active":"2024-03-11T21:35:06.747130+0000","last_peered":"2024-03-11T21:35:06.747130+0000","last_clean":"2024-03-11T21:35:06.747130+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:35:06.747130+0000","last_undegraded":"2024-03-11T21:35:06.747130+0000","last_fullsized":"2024-03-11T21:35:06.747130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:34:50.793817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.793817+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.790662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.971905+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:35:06.971905+0000","last_peered":"2024-03-11T21:35:06.971905+0000","last_clean":"2024-03-11T21:35:06.971905+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:35:06.971905+0000","last_undegraded":"2024-03-11T21:35:06.971905+0000","last_fullsized":"2024-03-11T21:35:06.971905+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":859,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.746446+0000","last_change":"2024-03-11T21:33:17.530575+0000","last_active":"2024-03-11T21:35:06.746446+0000","last_peered":"2024-03-11T21:35:06.746446+0000","last_clean":"2024-03-11T21:35:06.746446+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:35:06.746446+0000","last_undegraded":"2024-03-11T21:35:06.746446+0000","last_fullsized":"2024-03-11T21:35:06.746446+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:17.530299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.530299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":874,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972808+0000","last_change":"2024-03-11T21:34:42.092702+0000","last_active":"2024-03-11T21:35:06.972808+0000","last_peered":"2024-03-11T21:35:06.972808+0000","last_clean":"2024-03-11T21:35:06.972808+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:35:06.972808+0000","last_undegraded":"2024-03-11T21:35:06.972808+0000","last_fullsized":"2024-03-11T21:35:06.972808+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.092348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.092348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.061520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":725,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.353596+0000","last_change":"2024-03-11T21:34:47.959956+0000","last_active":"2024-03-11T21:35:06.353596+0000","last_peered":"2024-03-11T21:35:06.353596+0000","last_clean":"2024-03-11T21:35:06.353596+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:35:06.353596+0000","last_undegraded":"2024-03-11T21:35:06.353596+0000","last_fullsized":"2024-03-11T21:35:06.353596+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:47.959724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:47.959724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.149816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":930,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.747261+0000","last_change":"2024-03-11T21:33:20.542248+0000","last_active":"2024-03-11T21:35:06.747261+0000","last_peered":"2024-03-11T21:35:06.747261+0000","last_clean":"2024-03-11T21:35:06.747261+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:35:06.747261+0000","last_undegraded":"2024-03-11T21:35:06.747261+0000","last_fullsized":"2024-03-11T21:35:06.747261+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:33:20.542001+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.542001+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":867,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.974458+0000","last_change":"2024-03-11T21:30:25.479269+0000","last_active":"2024-03-11T21:35:06.974458+0000","last_peered":"2024-03-11T21:35:06.974458+0000","last_clean":"2024-03-11T21:35:06.974458+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:35:06.974458+0000","last_undegraded":"2024-03-11T21:35:06.974458+0000","last_fullsized":"2024-03-11T21:35:06.974458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.479173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.479173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":860,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.972054+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:35:06.972054+0000","last_peered":"2024-03-11T21:35:06.972054+0000","last_clean":"2024-03-11T21:35:06.972054+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:35:06.972054+0000","last_undegraded":"2024-03-11T21:35:06.972054+0000","last_fullsized":"2024-03-11T21:35:06.972054+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":864,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.745698+0000","last_change":"2024-03-11T21:33:23.549947+0000","last_active":"2024-03-11T21:35:06.745698+0000","last_peered":"2024-03-11T21:35:06.745698+0000","last_clean":"2024-03-11T21:35:06.745698+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:35:06.745698+0000","last_undegraded":"2024-03-11T21:35:06.745698+0000","last_fullsized":"2024-03-11T21:35:06.745698+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.549773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.549773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":714,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973484+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:35:06.973484+0000","last_peered":"2024-03-11T21:35:06.973484+0000","last_clean":"2024-03-11T21:35:06.973484+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:35:06.973484+0000","last_undegraded":"2024-03-11T21:35:06.973484+0000","last_fullsized":"2024-03-11T21:35:06.973484+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":911,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.977993+0000","last_change":"2024-03-11T21:33:39.970299+0000","last_active":"2024-03-11T21:35:06.977993+0000","last_peered":"2024-03-11T21:35:06.977993+0000","last_clean":"2024-03-11T21:35:06.977993+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:35:06.977993+0000","last_undegraded":"2024-03-11T21:35:06.977993+0000","last_fullsized":"2024-03-11T21:35:06.977993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:39.970169+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.970169+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":869,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479527+0000","last_change":"2024-03-11T21:34:35.593537+0000","last_active":"2024-03-11T21:35:06.479527+0000","last_peered":"2024-03-11T21:35:06.479527+0000","last_clean":"2024-03-11T21:35:06.479527+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:35:06.479527+0000","last_undegraded":"2024-03-11T21:35:06.479527+0000","last_fullsized":"2024-03-11T21:35:06.479527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.592784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.592784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:50.366116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":728,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.536723+0000","last_change":"2024-03-11T21:34:43.228208+0000","last_active":"2024-03-11T21:35:09.536723+0000","last_peered":"2024-03-11T21:35:09.536723+0000","last_clean":"2024-03-11T21:35:09.536723+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:35:09.536723+0000","last_undegraded":"2024-03-11T21:35:09.536723+0000","last_fullsized":"2024-03-11T21:35:09.536723+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.227861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.227861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.622588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":875,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.350339+0000","last_change":"2024-03-11T21:34:45.933759+0000","last_active":"2024-03-11T21:35:06.350339+0000","last_peered":"2024-03-11T21:35:06.350339+0000","last_clean":"2024-03-11T21:35:06.350339+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:35:06.350339+0000","last_undegraded":"2024-03-11T21:35:06.350339+0000","last_fullsized":"2024-03-11T21:35:06.350339+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.933659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.933659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:52.191761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":578,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-11T21:35:09.540170+0000","last_change":"2024-03-11T21:33:22.998783+0000","last_active":"2024-03-11T21:35:09.540170+0000","last_peered":"2024-03-11T21:35:09.540170+0000","last_clean":"2024-03-11T21:35:09.540170+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:35:09.540170+0000","last_undegraded":"2024-03-11T21:35:09.540170+0000","last_fullsized":"2024-03-11T21:35:09.540170+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":895,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.479712+0000","last_change":"2024-03-11T21:33:17.733152+0000","last_active":"2024-03-11T21:35:06.479712+0000","last_peered":"2024-03-11T21:35:06.479712+0000","last_clean":"2024-03-11T21:35:06.479712+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:35:06.479712+0000","last_undegraded":"2024-03-11T21:35:06.479712+0000","last_fullsized":"2024-03-11T21:35:06.479712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:33:17.733068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:17.733068+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.978019+0000","last_change":"2024-03-11T21:33:16.013881+0000","last_active":"2024-03-11T21:35:06.978019+0000","last_peered":"2024-03-11T21:35:06.978019+0000","last_clean":"2024-03-11T21:35:06.978019+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:35:06.978019+0000","last_undegraded":"2024-03-11T21:35:06.978019+0000","last_fullsized":"2024-03-11T21:35:06.978019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:33:16.013752+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:33:16.013752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":879,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.349142+0000","last_change":"2024-03-11T21:34:53.121915+0000","last_active":"2024-03-11T21:35:06.349142+0000","last_peered":"2024-03-11T21:35:06.349142+0000","last_clean":"2024-03-11T21:35:06.349142+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:35:06.349142+0000","last_undegraded":"2024-03-11T21:35:06.349142+0000","last_fullsized":"2024-03-11T21:35:06.349142+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.121696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.121696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:59.002073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":855,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.477944+0000","last_change":"2024-03-11T21:35:00.524627+0000","last_active":"2024-03-11T21:35:06.477944+0000","last_peered":"2024-03-11T21:35:06.477944+0000","last_clean":"2024-03-11T21:35:06.477944+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:35:06.477944+0000","last_undegraded":"2024-03-11T21:35:06.477944+0000","last_fullsized":"2024-03-11T21:35:06.477944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:00.524430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.524430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:04.786899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":287,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":10,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195788,"num_objects":160,"num_object_clones":3,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26488832,"data_stored":24587365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1874,"ondisk_log_size":1874,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841692,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84252,"kb_used_data":12808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93660900,"statfs":{"total":95995035648,"available":95908761600,"internally_reserved":0,"allocated":13115392,"data_stored":9792061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168741,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88036,"kb_used_data":12444,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657116,"statfs":{"total":95995035648,"available":95904886784,"internally_reserved":0,"allocated":12742656,"data_stored":9595236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352364,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48436,"kb_used_data":8032,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93696716,"statfs":{"total":95995035648,"available":95945437184,"internally_reserved":0,"allocated":8224768,"data_stored":5015134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87996,"kb_used_data":16552,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93657156,"statfs":{"total":95995035648,"available":95904927744,"internally_reserved":0,"allocated":16949248,"data_stored":13790667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378831,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75340,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669812,"statfs":{"total":95995035648,"available":95917887488,"internally_reserved":0,"allocated":8261632,"data_stored":5005556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117187,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71164,"kb_used_data":8040,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673988,"statfs":{"total":95995035648,"available":95922163712,"internally_reserved":0,"allocated":8232960,"data_stored":5005011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855545,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83488,"kb_used_data":12044,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661664,"statfs":{"total":95995035648,"available":95909543936,"internally_reserved":0,"allocated":12333056,"data_stored":9006605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495791,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75184,"kb_used_data":12072,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669968,"statfs":{"total":95995035648,"available":95918047232,"internally_reserved":0,"allocated":12361728,"data_stored":9199156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8359936,"data_stored":8194715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4000739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4386816,"data_stored":4194771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4000406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4195011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":287,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:35:13.534 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:35:13.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:35:13.534 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:35:13.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:35:13.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.209701+0000 osd.4 (osd.4) 261 : cluster 0 3.1a scrub starts 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.212153+0000 osd.4 (osd.4) 262 : cluster 0 3.1a scrub ok 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.338732+0000 osd.6 (osd.6) 225 : cluster 0 3.a scrub starts 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.343612+0000 osd.6 (osd.6) 226 : cluster 0 3.a scrub ok 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.527549+0000 mon.a (mon.0) 2802 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: audit 2024-03-11T21:35:12.529017+0000 mon.a (mon.0) 2803 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.535749+0000 mon.a (mon.0) 2804 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.700893+0000 osd.5 (osd.5) 301 : cluster 0 5.f scrub starts 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:12.702294+0000 osd.5 (osd.5) 302 : cluster 0 5.f scrub ok 2024-03-11T21:35:13.560 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:13 smithi203 bash[23654]: cluster 2024-03-11T21:35:13.099293+0000 osd.7 (osd.7) 295 : cluster 0 2.1d scrub starts 2024-03-11T21:35:13.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:13.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:13.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:13.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:13.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:13.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:13.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:14.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:14.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:14.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:14.217 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:14.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:14.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:14.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:14.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:14 smithi203 bash[23654]: cluster 2024-03-11T21:35:13.411974+0000 mgr.y (mgr.14550) 487 : cluster 0 pgmap v731: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:35:14.610 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:14.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:14.739 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:14.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.872 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:14.890 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:14 smithi143 bash[29284]: cluster 2024-03-11T21:35:13.411974+0000 mgr.y (mgr.14550) 487 : cluster 0 pgmap v731: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:35:14.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:14 smithi143 bash[21512]: cluster 2024-03-11T21:35:13.411974+0000 mgr.y (mgr.14550) 487 : cluster 0 pgmap v731: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:35:14.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:15.008 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:15.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:15.140 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:15.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:15.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:15.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:15.398 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:15.431 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:35:15.431 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:35:15.431 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:35:15.431 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:35:15.431 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1154 118.a e296/295 118:506a4333:test-rados-api-smithi143-71773-14::foo73:3 v 296'36, mlcod=0'0)", 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.404563+0000", 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.053092322, 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.027731412, 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404563+0000", 2024-03-11T21:35:15.432 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404563+0000", 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404564+0000", 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9200000000000002e-07 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.433 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404568+0000", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5589999999999999e-06 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404574+0000", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2600000000000002e-06 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404582+0000", 2024-03-11T21:35:15.434 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1450000000000002e-06 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404872+0000", 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029007399999999997 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404896+0000", 2024-03-11T21:35:15.435 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3995999999999999e-05 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.432252+0000", 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.027356407999999999 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.432295+0000", 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2678e-05 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.436 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1155 118.a e296/295 118:56cb8e2d:test-rados-api-smithi143-71773-14::foo9:3 v 296'38, mlcod=0'0)", 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.404604+0000", 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.053051469, 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.02794746, 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.437 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404604+0000", 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404604+0000", 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.438 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404605+0000", 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3099999999999998e-07 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404613+0000", 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1340000000000004e-06 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404622+0000", 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9160000000000007e-06 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.439 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.404628+0000", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3790000000000003e-06 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421609+0000", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.016981353000000001 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421631+0000", 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1171000000000001e-05 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.440 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.432507+0000", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.010876471 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.432552+0000", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4604999999999997e-05 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDPGPush(118.2 296/295 [PushOp(118:40282ed0:test-rados-api-smithi143-71773-20::foo:head, version: 274'15, data_included: [0~3], data_size: 3, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(118:40282ed0:test-rados-api-smithi143-71773-20::foo:head@274'15, size: 3, copy_subset: [(0, 3)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.405897+0000", 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.051758315, 2024-03-11T21:35:15.441 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.015663522999999999, 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.405897+0000", 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.405897+0000", 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.442 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.405898+0000", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2300000000000001e-07 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.405914+0000", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5914e-05 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.406294+0000", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00037996999999999998 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.443 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.406314+0000", 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9876000000000001e-05 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421371+0000", 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.015057216999999999 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421385+0000", 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4294000000000001e-05 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.444 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421561+0000", 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017572900000000001 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.3.0:727 118.6 e296/295 118:60bb15d0:test-rados-api-smithi143-71773-14::foo54:3 v 296'51, mlcod=0'0)", 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.420737+0000", 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.036918802, 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014798699, 2024-03-11T21:35:15.445 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420737+0000", 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420737+0000", 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420737+0000", 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8300000000000001e-07 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420741+0000", 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0439999999999997e-06 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.447 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420745+0000", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4599999999999999e-06 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.420817+0000", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2616999999999999e-05 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421931+0000", 2024-03-11T21:35:15.448 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001113558 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421942+0000", 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1022000000000001e-05 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.435446+0000", 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013504260000000001 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.449 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.435536+0000", 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9555000000000006e-05 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.3.0:728 118.6 e296/295 118:607be112:test-rados-api-smithi143-71773-14::foo83:3 v 296'53, mlcod=0'0)", 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.421833+0000", 2024-03-11T21:35:15.450 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.03582286400001, 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.015822072, 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421833+0000", 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421833+0000", 2024-03-11T21:35:15.451 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421833+0000", 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8999999999999998e-07 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421844+0000", 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0893999999999999e-05 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.452 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421851+0000", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8029999999999999e-06 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.421858+0000", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7490000000000001e-06 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.434857+0000", 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012999642 2024-03-11T21:35:15.453 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.434879+0000", 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1617e-05 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437584+0000", 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002704887 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.454 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437655+0000", 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1190000000000001e-05 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.0.0:1974 118.4 e296/295 118:25485e61:test-rados-api-smithi143-71773-14::foo36:3 v 296'32, mlcod=0'0)", 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.422957+0000", 2024-03-11T21:35:15.455 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.034698695, 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014588225999999999, 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422957+0000", 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422957+0000", 2024-03-11T21:35:15.456 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422957+0000", 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8400000000000001e-07 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422960+0000", 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9469999999999999e-06 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.457 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422976+0000", 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6235999999999998e-05 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422982+0000", 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4580000000000001e-06 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.458 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.423105+0000", 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000123399 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.423123+0000", 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8011000000000001e-05 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.459 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437527+0000", 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014403423 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437545+0000", 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8567999999999999e-05 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.460 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.0.0:1975 118.4 e296/295 118:274646bc:test-rados-api-smithi143-71773-14::foo12:3 v 296'34, mlcod=0'0)", 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:37.422988+0000", 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 218.034667568, 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014659109, 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422988+0000", 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.461 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422988+0000", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.422988+0000", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.36e-07 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.423096+0000", 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000107946 2024-03-11T21:35:15.462 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.423101+0000", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7249999999999997e-06 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.423105+0000", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.631e-06 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437313+0000", 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014208502 2024-03-11T21:35:15.463 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437327+0000", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3444e-05 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437620+0000", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029300000000000002 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:37.437647+0000", 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7625000000000001e-05 2024-03-11T21:35:15.464 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:2048 236.12 236:4bb5dbef:test-rados-api-smithi143-71773-26::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e301)", 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:42.288683+0000", 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 213.168972943, 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.019663201000000002, 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2048 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.465 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288683+0000", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288683+0000", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288685+0000", 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0200000000000001e-06 2024-03-11T21:35:15.466 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288692+0000", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5870000000000004e-06 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288703+0000", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0679000000000001e-05 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.288716+0000", 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3114e-05 2024-03-11T21:35:15.467 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.291298+0000", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002581847 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.291590+0000", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000291873 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.301363+0000", 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0097727109999999999 2024-03-11T21:35:15.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.301491+0000", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012840400000000001 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.308059+0000", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0065684920000000004 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.308127+0000", 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7568000000000001e-05 2024-03-11T21:35:15.469 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:42.308346+0000", 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00021890599999999999 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.7.0:1261 118.1c e311/295 118:3df71cb3:test-rados-api-smithi143-71773-28::foo:a v 311'49, mlcod=306'45)", 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:52.292702+0000", 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 203.16495348399999, 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.021876828000000001, 2024-03-11T21:35:15.470 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292702+0000", 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292702+0000", 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.471 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292704+0000", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.686e-06 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292720+0000", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6344000000000001e-05 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292742+0000", 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1614999999999999e-05 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.472 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292757+0000", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5656999999999999e-05 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292857+0000", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9504000000000001e-05 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.292916+0000", 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8692e-05 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.473 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.314501+0000", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.021585033999999999 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:52.314579+0000", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8295999999999997e-05 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:979 244.10 e321/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 321'2, mlcod=0'0)", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:02.302666+0000", 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 193.15498932099999, 2024-03-11T21:35:15.474 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013139088, 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.302666+0000", 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.302666+0000", 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.475 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.302668+0000", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.711e-06 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.306231+0000", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.003562792 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.306248+0000", 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6997999999999999e-05 2024-03-11T21:35:15.476 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.306258+0000", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.022e-05 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.306329+0000", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0732e-05 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.306350+0000", 2024-03-11T21:35:15.477 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1489e-05 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.315769+0000", 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0094182999999999992 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.315805+0000", 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6845999999999999e-05 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.478 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1063 244.10 e321/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 321'86, mlcod=321'84)", 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:02.960316+0000", 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 192.497339668, 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012882708999999999, 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.479 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.960316+0000", 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.960316+0000", 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.480 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.960318+0000", 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.877e-06 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.963159+0000", 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0028407570000000002 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.963193+0000", 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3948000000000001e-05 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.481 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.963245+0000", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2635e-05 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.963417+0000", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017130699999999999 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.963480+0000", 2024-03-11T21:35:15.482 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3752999999999998e-05 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.973131+0000", 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0096503539999999999 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:02.973199+0000", 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8077999999999994e-05 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.483 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1075 244.10 e321/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 321'98, mlcod=321'96)", 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:03.064487+0000", 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 192.393168927, 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012759189000000001, 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.484 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.064487+0000", 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.064487+0000", 2024-03-11T21:35:15.486 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.064489+0000", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7199999999999998e-06 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.070469+0000", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0059793190000000003 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.070502+0000", 2024-03-11T21:35:15.487 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3036e-05 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.070527+0000", 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4851e-05 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.070604+0000", 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6885999999999995e-05 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.488 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.070622+0000", 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8632999999999999e-05 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.077206+0000", 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0065841620000000002 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.077246+0000", 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.9582000000000002e-05 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1096 244.10 e321/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 321'119, mlcod=321'117)", 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:03.232810+0000", 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 192.22484606500001, 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013500267999999999, 2024-03-11T21:35:15.490 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.232810+0000", 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.232810+0000", 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.491 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.232811+0000", 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.032e-06 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.235470+0000", 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0026590060000000002 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.492 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.235493+0000", 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3193e-05 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.235507+0000", 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3861e-05 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.235586+0000", 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9808999999999994e-05 2024-03-11T21:35:15.493 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.235608+0000", 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1987000000000001e-05 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.246209+0000", 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.010600426 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.494 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.246310+0000", 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000100954 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1105 244.10 e322/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 322'128, mlcod=322'126)", 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:03.425116+0000", 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 192.032539939, 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014389608, 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.495 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.425116+0000", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.425116+0000", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.496 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.425118+0000", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.683e-06 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.429482+0000", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0043634090000000004 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.429520+0000", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8019000000000003e-05 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.429541+0000", 2024-03-11T21:35:15.497 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0789999999999999e-05 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.429663+0000", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012214999999999999 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.429708+0000", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5584999999999998e-05 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.439424+0000", 2024-03-11T21:35:15.498 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0097155079999999994 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.439505+0000", 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1464000000000005e-05 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1111 244.10 e322/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 322'134, mlcod=322'132)", 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:03.481078+0000", 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 191.97657719700001, 2024-03-11T21:35:15.499 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012335629000000001, 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.481078+0000", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.481078+0000", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.500 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.481080+0000", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6360000000000001e-06 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.484985+0000", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0039048360000000001 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.485024+0000", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.9016e-05 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.485043+0000", 2024-03-11T21:35:15.501 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9272000000000001e-05 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.485171+0000", 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012774999999999999 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.485215+0000", 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4486000000000001e-05 2024-03-11T21:35:15.502 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.493294+0000", 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0080787129999999995 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.493414+0000", 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011992 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.2.0:1114 244.10 e322/315 244:08623bad:test-rados-api-smithi143-71773-30::foo:c v 322'137, mlcod=322'135)", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:03.508857+0000", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 191.948798676, 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012861391, 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.508857+0000", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.504 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.508857+0000", 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.508858+0000", 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4720000000000001e-06 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.513430+0000", 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0045711140000000003 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.505 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.513460+0000", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0540000000000002e-05 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.513478+0000", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7711e-05 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.513611+0000", 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013322799999999999 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.506 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.513657+0000", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6205999999999998e-05 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.521615+0000", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0079579290000000007 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:03.521718+0000", 2024-03-11T21:35:15.507 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000103191 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:2421 244.0 244:00000000:%01:::head [pgnls start_epoch 324] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e324)", 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:05.293060+0000", 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 190.16459536599999, 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.097321374000000002, 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:35:15.508 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2421 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.293060+0000", 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.509 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.293060+0000", 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.293063+0000", 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0960000000000001e-06 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.293075+0000", 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1646e-05 2024-03-11T21:35:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.373977+0000", 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.080902128000000004 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.374011+0000", 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4292999999999998e-05 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.511 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.390045+0000", 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.016033756 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.390087+0000", 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2162000000000001e-05 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.390382+0000", 2024-03-11T21:35:15.512 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000294293 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:2455 244.0 244:00000000::::head [pgnls start_epoch 326] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e326)", 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:07.303328+0000", 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 188.15432787399999, 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.092773260999999996, 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:35:15.513 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2455 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.303328+0000", 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.514 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.303328+0000", 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.303331+0000", 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.145e-06 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.303349+0000", 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7949e-05 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.378081+0000", 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.074732456000000003 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.378124+0000", 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2605000000000003e-05 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.516 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.395207+0000", 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.017083329000000001 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.395314+0000", 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000107161 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:07.396101+0000", 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00078661599999999996 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:3239 260.1e 260:7e31988a:test-rados-api-smithi143-71773-38::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:42.302735+0000", 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 153.15492092599999, 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.018444197999999998, 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.518 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3239 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302735+0000", 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.519 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302735+0000", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302737+0000", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7159999999999999e-06 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302750+0000", 2024-03-11T21:35:15.520 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2298e-05 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302767+0000", 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7455000000000001e-05 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.302783+0000", 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5418e-05 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.521 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.303176+0000", 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00039386999999999999 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.303478+0000", 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000301592 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.522 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.319766+0000", 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.016288395000000001 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.319850+0000", 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3134999999999996e-05 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.320018+0000", 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016805600000000001 2024-03-11T21:35:15.523 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.320893+0000", 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00087568200000000002 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.321044+0000", 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000150428 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.321108+0000", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3989e-05 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.321148+0000", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0394e-05 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:42.321179+0000", 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0769999999999998e-05 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.525 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:5419 278.0 278:00000000:%01:::head [pgnls start_epoch 441] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e441)", 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:34:02.334298+0000", 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 73.123357511999998, 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.028969069, 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5419 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.526 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.334298+0000", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.334298+0000", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.334302+0000", 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.5499999999999999e-06 2024-03-11T21:35:15.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.334321+0000", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8960000000000001e-05 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.362784+0000", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.028462893 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.362809+0000", 2024-03-11T21:35:15.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5774000000000001e-05 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.362923+0000", 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000113653 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.362997+0000", 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4282999999999993e-05 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:35:15.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:02.363267+0000", 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026995600000000002 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:15.530 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:35:15.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:15.552 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:15.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:15.683 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:15.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:15.814 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:15.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:15.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:16.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.076 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:16.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:16.208 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:16.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:16.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:35:16.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:16.471 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:16.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:16 smithi143 bash[29284]: cluster 2024-03-11T21:35:15.412826+0000 mgr.y (mgr.14550) 488 : cluster 0 pgmap v732: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:16.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:16 smithi143 bash[29284]: cluster 2024-03-11T21:35:16.364529+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:35:16.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:16 smithi143 bash[21512]: cluster 2024-03-11T21:35:15.412826+0000 mgr.y (mgr.14550) 488 : cluster 0 pgmap v732: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:16.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:16 smithi143 bash[21512]: cluster 2024-03-11T21:35:16.364529+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:35:16.599 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:16.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:16.728 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:16.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:16 smithi203 bash[23654]: cluster 2024-03-11T21:35:15.412826+0000 mgr.y (mgr.14550) 488 : cluster 0 pgmap v732: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:16.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:16 smithi203 bash[23654]: cluster 2024-03-11T21:35:16.364529+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:35:16.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:16.859 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:16.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:35:16.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:17.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:17.104 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:17.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:17.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:17.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.370 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:17.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.500 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:17.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:17.628 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:17.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.759 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:17.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:17.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:17.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.020 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:18.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.151 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:18.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.282 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:18.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:18.412 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:18.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:18.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.678 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:18.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:18.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:18.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:18 smithi143 bash[21512]: cluster 2024-03-11T21:35:17.413657+0000 mgr.y (mgr.14550) 489 : cluster 0 pgmap v734: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-03-11T21:35:18.793 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:18 smithi143 bash[29284]: cluster 2024-03-11T21:35:17.413657+0000 mgr.y (mgr.14550) 489 : cluster 0 pgmap v734: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-03-11T21:35:18.809 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:18.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:18.941 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:18.946 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:18.948 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:19.021 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:19.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:18 smithi203 bash[23654]: cluster 2024-03-11T21:35:17.413657+0000 mgr.y (mgr.14550) 489 : cluster 0 pgmap v734: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-03-11T21:35:19.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:19.067 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:19.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.198 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:19.302 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:19.329 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:19.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:19.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.589 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:19.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:19 smithi143 bash[21512]: audit 2024-03-11T21:35:18.670964+0000 mgr.y (mgr.14550) 490 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:19 smithi143 bash[29284]: audit 2024-03-11T21:35:18.670964+0000 mgr.y (mgr.14550) 490 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:19.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:19.717 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:19.770 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:19 smithi203 bash[23654]: audit 2024-03-11T21:35:18.670964+0000 mgr.y (mgr.14550) 490 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:19.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:19.846 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:19.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.975 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:20.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:20.107 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:20.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:20.237 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:20.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:20.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:20.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:20.497 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:20.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:20.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:20.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[29284]: cluster 2024-03-11T21:35:19.415903+0000 mgr.y (mgr.14550) 491 : cluster 0 pgmap v735: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[29284]: audit 2024-03-11T21:35:19.577469+0000 mon.c (mon.2) 970 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[29284]: audit 2024-03-11T21:35:19.578179+0000 mon.a (mon.0) 2806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[29284]: audit 2024-03-11T21:35:19.579335+0000 mon.c (mon.2) 971 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[29284]: audit 2024-03-11T21:35:19.580122+0000 mon.a (mon.0) 2807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[21512]: cluster 2024-03-11T21:35:19.415903+0000 mgr.y (mgr.14550) 491 : cluster 0 pgmap v735: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[21512]: audit 2024-03-11T21:35:19.577469+0000 mon.c (mon.2) 970 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[21512]: audit 2024-03-11T21:35:19.578179+0000 mon.a (mon.0) 2806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.744 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[21512]: audit 2024-03-11T21:35:19.579335+0000 mon.c (mon.2) 971 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.745 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:20 smithi143 bash[21512]: audit 2024-03-11T21:35:19.580122+0000 mon.a (mon.0) 2807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.756 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:20.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:20 smithi203 bash[23654]: cluster 2024-03-11T21:35:19.415903+0000 mgr.y (mgr.14550) 491 : cluster 0 pgmap v735: 260 pgs: 260 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:35:20.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:20 smithi203 bash[23654]: audit 2024-03-11T21:35:19.577469+0000 mon.c (mon.2) 970 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:20 smithi203 bash[23654]: audit 2024-03-11T21:35:19.578179+0000 mon.a (mon.0) 2806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:20.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:20 smithi203 bash[23654]: audit 2024-03-11T21:35:19.579335+0000 mon.c (mon.2) 971 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:20 smithi203 bash[23654]: audit 2024-03-11T21:35:19.580122+0000 mon.a (mon.0) 2807 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-63"}]: dispatch 2024-03-11T21:35:20.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:20.881 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:20.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:20.994 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:20.995 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:21.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:21.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:21.143 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:21.246 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:21.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:21.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:21.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:21.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:21.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:21.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:21.659 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:21.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: cluster 2024-03-11T21:35:20.594349+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:35:21.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: cluster 2024-03-11T21:35:21.367852+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:35:21.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: audit 2024-03-11T21:35:21.374338+0000 mon.c (mon.2) 972 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:21.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: audit 2024-03-11T21:35:21.376741+0000 mon.a (mon.0) 2810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:21.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: audit 2024-03-11T21:35:21.526394+0000 mon.a (mon.0) 2811 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:21.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[21512]: audit 2024-03-11T21:35:21.529640+0000 mon.c (mon.2) 973 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:21.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:21.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:21.875 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: cluster 2024-03-11T21:35:20.594349+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:35:21.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: cluster 2024-03-11T21:35:21.367852+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:35:21.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: audit 2024-03-11T21:35:21.374338+0000 mon.c (mon.2) 972 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:21.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: audit 2024-03-11T21:35:21.376741+0000 mon.a (mon.0) 2810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:21.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: audit 2024-03-11T21:35:21.526394+0000 mon.a (mon.0) 2811 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:21.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:21 smithi143 bash[29284]: audit 2024-03-11T21:35:21.529640+0000 mon.c (mon.2) 973 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:21.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:21.915 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:22.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: cluster 2024-03-11T21:35:20.594349+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: cluster 2024-03-11T21:35:21.367852+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: audit 2024-03-11T21:35:21.374338+0000 mon.c (mon.2) 972 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: audit 2024-03-11T21:35:21.376741+0000 mon.a (mon.0) 2810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: audit 2024-03-11T21:35:21.526394+0000 mon.a (mon.0) 2811 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:22.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:21 smithi203 bash[23654]: audit 2024-03-11T21:35:21.529640+0000 mon.c (mon.2) 973 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:22.047 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:22.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:22.161 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:22.177 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:22.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:22.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:22.435 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:22.538 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:22.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:22.697 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:22.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: cluster 2024-03-11T21:35:21.416749+0000 mgr.y (mgr.14550) 492 : cluster 0 pgmap v738: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:22.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: cluster 2024-03-11T21:35:22.357221+0000 mon.a (mon.0) 2812 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:22.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: audit 2024-03-11T21:35:22.365720+0000 mon.a (mon.0) 2813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:22.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: cluster 2024-03-11T21:35:22.372094+0000 mon.a (mon.0) 2814 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:35:22.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: audit 2024-03-11T21:35:22.381088+0000 mon.c (mon.2) 974 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:22.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: audit 2024-03-11T21:35:22.387099+0000 mon.c (mon.2) 975 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:22 smithi203 bash[23654]: audit 2024-03-11T21:35:22.388420+0000 mon.a (mon.0) 2815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:22.803 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: cluster 2024-03-11T21:35:21.416749+0000 mgr.y (mgr.14550) 492 : cluster 0 pgmap v738: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: cluster 2024-03-11T21:35:22.357221+0000 mon.a (mon.0) 2812 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: audit 2024-03-11T21:35:22.365720+0000 mon.a (mon.0) 2813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: cluster 2024-03-11T21:35:22.372094+0000 mon.a (mon.0) 2814 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: audit 2024-03-11T21:35:22.381088+0000 mon.c (mon.2) 974 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: audit 2024-03-11T21:35:22.387099+0000 mon.c (mon.2) 975 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[29284]: audit 2024-03-11T21:35:22.388420+0000 mon.a (mon.0) 2815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: cluster 2024-03-11T21:35:21.416749+0000 mgr.y (mgr.14550) 492 : cluster 0 pgmap v738: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:22.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: cluster 2024-03-11T21:35:22.357221+0000 mon.a (mon.0) 2812 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:22.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: audit 2024-03-11T21:35:22.365720+0000 mon.a (mon.0) 2813 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:22.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: cluster 2024-03-11T21:35:22.372094+0000 mon.a (mon.0) 2814 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:35:22.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: audit 2024-03-11T21:35:22.381088+0000 mon.c (mon.2) 974 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:22.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: audit 2024-03-11T21:35:22.387099+0000 mon.c (mon.2) 975 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:22 smithi143 bash[21512]: audit 2024-03-11T21:35:22.388420+0000 mon.a (mon.0) 2815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:22.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:22.956 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:23.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:23.088 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:23.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:23.217 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:23.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:23.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:23.477 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:23.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:23.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:23.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:23.739 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:23.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.870 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:23.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.001 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:24.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:24.131 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:24.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.261 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:24.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.391 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:24.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[29284]: audit 2024-03-11T21:35:23.368905+0000 mon.a (mon.0) 2816 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[29284]: cluster 2024-03-11T21:35:23.380069+0000 mon.a (mon.0) 2817 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[29284]: cluster 2024-03-11T21:35:23.417767+0000 mgr.y (mgr.14550) 493 : cluster 0 pgmap v741: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[21512]: audit 2024-03-11T21:35:23.368905+0000 mon.a (mon.0) 2816 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[21512]: cluster 2024-03-11T21:35:23.380069+0000 mon.a (mon.0) 2817 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:35:24.505 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:24 smithi143 bash[21512]: cluster 2024-03-11T21:35:23.417767+0000 mgr.y (mgr.14550) 493 : cluster 0 pgmap v741: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:24.520 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:24 smithi203 bash[23654]: audit 2024-03-11T21:35:23.368905+0000 mon.a (mon.0) 2816 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:24 smithi203 bash[23654]: cluster 2024-03-11T21:35:23.380069+0000 mon.a (mon.0) 2817 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:35:24.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:24 smithi203 bash[23654]: cluster 2024-03-11T21:35:23.417767+0000 mgr.y (mgr.14550) 493 : cluster 0 pgmap v741: 260 pgs: 6 unknown, 254 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:24.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:24.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:24.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:24.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:24.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:24.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:25.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:25.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:25.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:25.176 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:25.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:25.307 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:25.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:25.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:25.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:25 smithi203 bash[23654]: cluster 2024-03-11T21:35:24.380939+0000 mon.a (mon.0) 2818 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:35:25.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:25.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:25 smithi143 bash[21512]: cluster 2024-03-11T21:35:24.380939+0000 mon.a (mon.0) 2818 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:35:25.553 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:25 smithi143 bash[29284]: cluster 2024-03-11T21:35:24.380939+0000 mon.a (mon.0) 2818 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:35:25.569 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:25.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:25.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:25.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:25.828 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:25.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:25.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:26.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:26.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:26.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:26.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:26.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:26.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:26 smithi203 bash[23654]: cluster 2024-03-11T21:35:25.401551+0000 mon.a (mon.0) 2819 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:35:26.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:26 smithi203 bash[23654]: cluster 2024-03-11T21:35:25.418413+0000 mgr.y (mgr.14550) 494 : cluster 0 pgmap v744: 260 pgs: 3 unknown, 257 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:26.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:26.591 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:26 smithi143 bash[21512]: cluster 2024-03-11T21:35:25.401551+0000 mon.a (mon.0) 2819 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:35:26.592 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:26 smithi143 bash[21512]: cluster 2024-03-11T21:35:25.418413+0000 mgr.y (mgr.14550) 494 : cluster 0 pgmap v744: 260 pgs: 3 unknown, 257 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:26.592 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:26 smithi143 bash[29284]: cluster 2024-03-11T21:35:25.401551+0000 mon.a (mon.0) 2819 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:35:26.592 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:26 smithi143 bash[29284]: cluster 2024-03-11T21:35:25.418413+0000 mgr.y (mgr.14550) 494 : cluster 0 pgmap v744: 260 pgs: 3 unknown, 257 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:26.608 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:26.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:26.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:26.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:26.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:27.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:27.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.125 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:27.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:27.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:27.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:27.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:27.514 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:27.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:27 smithi203 bash[23654]: cluster 2024-03-11T21:35:26.434486+0000 mon.a (mon.0) 2820 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:35:27.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:27.636 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:27 smithi143 bash[21512]: cluster 2024-03-11T21:35:26.434486+0000 mon.a (mon.0) 2820 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:35:27.636 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:27 smithi143 bash[29284]: cluster 2024-03-11T21:35:26.434486+0000 mon.a (mon.0) 2820 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:35:27.642 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:27.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:27.772 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:27.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:28.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.030 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:28.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:28.162 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:28.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.290 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:28.393 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:28.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:28 smithi203 bash[23654]: cluster 2024-03-11T21:35:27.419057+0000 mgr.y (mgr.14550) 495 : cluster 0 pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 253 B/s wr, 0 op/s 2024-03-11T21:35:28.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:28.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:28 smithi143 bash[21512]: cluster 2024-03-11T21:35:27.419057+0000 mgr.y (mgr.14550) 495 : cluster 0 pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 253 B/s wr, 0 op/s 2024-03-11T21:35:28.666 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:28 smithi143 bash[29284]: cluster 2024-03-11T21:35:27.419057+0000 mgr.y (mgr.14550) 495 : cluster 0 pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 253 B/s wr, 0 op/s 2024-03-11T21:35:28.681 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:28.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:28.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:28.813 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:28.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:28.943 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:29.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:29.205 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:29.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:29.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:29.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:29.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:29.496 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:29 smithi143 bash[21512]: audit 2024-03-11T21:35:28.677318+0000 mgr.y (mgr.14550) 496 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:29.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.580 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:29 smithi203 bash[23654]: audit 2024-03-11T21:35:28.677318+0000 mgr.y (mgr.14550) 496 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:29.595 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.736 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:29 smithi143 bash[29284]: audit 2024-03-11T21:35:28.677318+0000 mgr.y (mgr.14550) 496 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:29.829 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:29.857 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:29.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:29.988 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:30.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:30.400 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:30.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:30.516 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:30 smithi143 bash[21512]: cluster 2024-03-11T21:35:29.421741+0000 mgr.y (mgr.14550) 497 : cluster 0 pgmap v747: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T21:35:30.517 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:30 smithi143 bash[29284]: cluster 2024-03-11T21:35:29.421741+0000 mgr.y (mgr.14550) 497 : cluster 0 pgmap v747: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T21:35:30.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:30.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:30.649 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:30 smithi203 bash[23654]: cluster 2024-03-11T21:35:29.421741+0000 mgr.y (mgr.14550) 497 : cluster 0 pgmap v747: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T21:35:30.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:30.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:30.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:30.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:30.928 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:31.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:31.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:31.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.320 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:31.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:31.447 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:31.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:31.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:31 smithi203 bash[23654]: cluster 2024-03-11T21:35:29.648103+0000 osd.5 (osd.5) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:31.562 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:31 smithi203 bash[23654]: cluster 2024-03-11T21:35:29.648332+0000 osd.5 (osd.5) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:31.562 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:31 smithi203 bash[23654]: cluster 2024-03-11T21:35:30.464477+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:35:31.562 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:31 smithi203 bash[23654]: cluster 2024-03-11T21:35:31.358469+0000 mon.a (mon.0) 2822 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:31.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:31.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:31.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[21512]: cluster 2024-03-11T21:35:29.648103+0000 osd.5 (osd.5) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[21512]: cluster 2024-03-11T21:35:29.648332+0000 osd.5 (osd.5) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[21512]: cluster 2024-03-11T21:35:30.464477+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:35:31.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[21512]: cluster 2024-03-11T21:35:31.358469+0000 mon.a (mon.0) 2822 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:31.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[29284]: cluster 2024-03-11T21:35:29.648103+0000 osd.5 (osd.5) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:31.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[29284]: cluster 2024-03-11T21:35:29.648332+0000 osd.5 (osd.5) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:31.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[29284]: cluster 2024-03-11T21:35:30.464477+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:35:31.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:31 smithi143 bash[29284]: cluster 2024-03-11T21:35:31.358469+0000 mon.a (mon.0) 2822 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:31.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:31.963 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:32.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:32.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:32.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:32.224 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:32.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:32.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:32.452 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:32.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:32.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:32.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:32.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:32.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:32.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[21512]: cluster 2024-03-11T21:35:31.125393+0000 osd.3 (osd.3) 295 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:32.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[21512]: cluster 2024-03-11T21:35:31.125673+0000 osd.3 (osd.3) 296 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:32.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[21512]: cluster 2024-03-11T21:35:31.422513+0000 mgr.y (mgr.14550) 498 : cluster 0 pgmap v749: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T21:35:32.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[21512]: cluster 2024-03-11T21:35:31.487770+0000 mon.a (mon.0) 2823 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:35:32.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[29284]: cluster 2024-03-11T21:35:31.125393+0000 osd.3 (osd.3) 295 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:32.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[29284]: cluster 2024-03-11T21:35:31.125673+0000 osd.3 (osd.3) 296 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:32.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[29284]: cluster 2024-03-11T21:35:31.422513+0000 mgr.y (mgr.14550) 498 : cluster 0 pgmap v749: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T21:35:32.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:32 smithi143 bash[29284]: cluster 2024-03-11T21:35:31.487770+0000 mon.a (mon.0) 2823 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:35:32.743 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:32 smithi203 bash[23654]: cluster 2024-03-11T21:35:31.125393+0000 osd.3 (osd.3) 295 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:32 smithi203 bash[23654]: cluster 2024-03-11T21:35:31.125673+0000 osd.3 (osd.3) 296 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:32 smithi203 bash[23654]: cluster 2024-03-11T21:35:31.422513+0000 mgr.y (mgr.14550) 498 : cluster 0 pgmap v749: 260 pgs: 1 active+clean+snaptrim, 259 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T21:35:32.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:32 smithi203 bash[23654]: cluster 2024-03-11T21:35:31.487770+0000 mon.a (mon.0) 2823 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:35:32.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:32.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:32.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:33.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:33.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:33.135 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:33.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 -- 172.21.15.143:0/287955132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c06a4b0 msgr2=0x7f7d2c10ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 --2- 172.21.15.143:0/287955132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c06a4b0 0x7f7d2c10ecd0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009f90 tx=0x7f7d2400b330 comp rx=0 tx=0).stop 2024-03-11T21:35:33.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 -- 172.21.15.143:0/287955132 shutdown_connections 2024-03-11T21:35:33.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 --2- 172.21.15.143:0/287955132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c06a4b0 0x7f7d2c10ecd0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 --2- 172.21.15.143:0/287955132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c107e90 0x7f7d2c108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 --2- 172.21.15.143:0/287955132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d2c069b10 0x7f7d2c069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 -- 172.21.15.143:0/287955132 >> 172.21.15.143:0/287955132 conn(0x7f7d2c076d80 msgr2=0x7f7d2c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:33.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 -- 172.21.15.143:0/287955132 shutdown_connections 2024-03-11T21:35:33.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 -- 172.21.15.143:0/287955132 wait complete. 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 librados: starting msgr at 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.190+0000 7f7d31965700 1 librados: starting objecter 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 Processor -- start 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- start start 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 librados: setting wanted keys 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 librados: calling monclient init 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d2c069b10 0x7f7d2c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 0x7f7d2c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 0x7f7d2c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 0x7f7d2c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 0x7f7d2c101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:32802/0 (socket says 172.21.15.143:32802) 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 0x7f7d2c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2a7fc700 1 -- 172.21.15.143:0/3680999184 learned_addr learned my addr 172.21.15.143:0/3680999184 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1115f0 con 0x7f7d2c107e90 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c111470 con 0x7f7d2c069b10 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c111770 con 0x7f7d2c06a4b0 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 msgr2=0x7f7d2c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 0x7f7d2c101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d2c069b10 msgr2=0x7f7d2c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d2c069b10 0x7f7d2c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c1a60f0 con 0x7f7d2c107e90 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d2c06a4b0 0x7f7d2c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 0x7f7d2c101890 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009790 tx=0x7f7d2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d30963700 10 client.?.objecter ms_handle_connect 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d30963700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24021070 con 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7d2401d450 con 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240052a0 con 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c1a6380 con 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 10 client.30477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c1a67e0 con 0x7f7d2c107e90 2024-03-11T21:35:33.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d31965700 1 librados: init done 2024-03-11T21:35:33.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.194+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7d0c005100 con 0x7f7d2c107e90 2024-03-11T21:35:33.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7d24024070 con 0x7f7d2c107e90 2024-03-11T21:35:33.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 0x7f7d14071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(497..497 src has 87..497) v4 ==== 9657+0+0 (secure 0 0 0) 0x7f7d2400f020 con 0x7f7d2c107e90 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 osd_map(497..497 src has 87..497) v4 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 3 client.30477.objecter handle_osd_map got epochs [497,497] > 0 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 3 client.30477.objecter handle_osd_map decoding full epoch 497 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 0x7f7d14071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 20 client.30477.objecter dump_active .. 0 homeless 2024-03-11T21:35:33.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v497) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7d24094030 con 0x7f7d2c107e90 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 0x7f7d14071ce0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c005fd0 tx=0x7f7d1c0095b0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d1406f820 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 10 client.30477.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 1 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7d0c004730 0x7f7d0c004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 1 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 0 1 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 4 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 10 client.30477.objecter _send_command 1 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c008b70 con 0x7f7d0c004730 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.198+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7d0c004730 0x7f7d0c004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c004730 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7d0c004730 0x7f7d0c004b70 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c008b70 con 0x7f7d0c004730 2024-03-11T21:35:33.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:35:33.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 10 client.30477.objecter _finish_command 1 = system:0 2024-03-11T21:35:33.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 4 2024-03-11T21:35:33.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.202+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 0 1 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 10 client.30477.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 2 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 2 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c0053e0 osd=0 3 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 0 2 2024-03-11T21:35:33.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 4 2024-03-11T21:35:33.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 10 client.30477.objecter _send_command 2 2024-03-11T21:35:33.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.218+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c0042d0 con 0x7f7d0c004730 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c0042d0 con 0x7f7d0c004730 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 10 client.30477.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 4 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 0 2 2024-03-11T21:35:33.196 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 10 client.30477.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 3 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7d0c002750 0x7f7d0c002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 3 2024-03-11T21:35:33.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 1 3 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 4 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 10 client.30477.objecter _send_command 3 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c003110 con 0x7f7d0c002750 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7d0c002750 0x7f7d0c002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7d0c002750 0x7f7d0c002bd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.198 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c002750 2024-03-11T21:35:33.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c003110 con 0x7f7d0c002750 2024-03-11T21:35:33.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:35:33.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 10 client.30477.objecter _finish_command 3 = system:0 2024-03-11T21:35:33.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 4 2024-03-11T21:35:33.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.222+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 1 3 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 10 client.30477.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 4 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 4 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c004560 osd=1 3 2024-03-11T21:35:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 1 4 2024-03-11T21:35:33.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 4 2024-03-11T21:35:33.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 10 client.30477.objecter _send_command 4 2024-03-11T21:35:33.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c003b40 con 0x7f7d0c002750 2024-03-11T21:35:33.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c003b40 con 0x7f7d0c002750 2024-03-11T21:35:33.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 10 client.30477.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 4 2024-03-11T21:35:33.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 1 4 2024-03-11T21:35:33.218 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 10 client.30477.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 5 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7d0c00b240 0x7f7d0c00d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 5 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 2 5 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 4 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 10 client.30477.objecter _send_command 5 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c010620 con 0x7f7d0c00b240 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7d0c00b240 0x7f7d0c00d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c00b240 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.242+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7d0c00b240 0x7f7d0c00d630 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.246+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c010620 con 0x7f7d0c00b240 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.246+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:35:33.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.246+0000 7f7d30963700 10 client.30477.objecter _finish_command 5 = system:0 2024-03-11T21:35:33.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.246+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 4 2024-03-11T21:35:33.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.246+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 2 5 2024-03-11T21:35:33.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.270+0000 7f7d127fc700 10 client.30477.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 6 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 6 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c001b30 osd=2 3 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 2 6 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 4 2024-03-11T21:35:33.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 10 client.30477.objecter _send_command 6 2024-03-11T21:35:33.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c0035c0 con 0x7f7d0c00b240 2024-03-11T21:35:33.249 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:35:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c0035c0 con 0x7f7d0c00b240 2024-03-11T21:35:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 10 client.30477.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 4 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 2 6 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 10 client.30477.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 7 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7d0c008720 0x7f7d0c00ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 7 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 3 7 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 4 2024-03-11T21:35:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 10 client.30477.objecter _send_command 7 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c00f520 con 0x7f7d0c008720 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7d0c008720 0x7f7d0c00ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c008720 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7d0c008720 0x7f7d0c00ed80 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c00f520 con 0x7f7d0c008720 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 10 client.30477.objecter _finish_command 7 = system:0 2024-03-11T21:35:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 4 2024-03-11T21:35:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.274+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 3 7 2024-03-11T21:35:33.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 10 client.30477.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 8 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 8 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c0039c0 osd=3 3 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 3 8 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 4 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 10 client.30477.objecter _send_command 8 2024-03-11T21:35:33.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c007c40 con 0x7f7d0c008720 2024-03-11T21:35:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:33.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c007c40 con 0x7f7d0c008720 2024-03-11T21:35:33.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d30963700 10 client.30477.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 4 2024-03-11T21:35:33.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.294+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 3 8 2024-03-11T21:35:33.271 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 10 client.30477.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 9 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7d0c007db0 0x7f7d0c0147f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 9 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 4 9 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 4 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 10 client.30477.objecter _send_command 9 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c0177c0 con 0x7f7d0c007db0 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7d0c007db0 0x7f7d0c0147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c007db0 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7d0c007db0 0x7f7d0c0147f0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c0177c0 con 0x7f7d0c007db0 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 10 client.30477.objecter _finish_command 9 = system:0 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 4 2024-03-11T21:35:33.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.298+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 4 9 2024-03-11T21:35:33.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 10 client.30477.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 10 2024-03-11T21:35:33.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 10 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c007c40 osd=4 3 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 4 10 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 4 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 10 client.30477.objecter _send_command 10 2024-03-11T21:35:33.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.314+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c016550 con 0x7f7d0c007db0 2024-03-11T21:35:33.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c016550 con 0x7f7d0c007db0 2024-03-11T21:35:33.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 10 client.30477.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 4 2024-03-11T21:35:33.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 4 10 2024-03-11T21:35:33.293 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 10 client.30477.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 11 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7d0c016c70 0x7f7d0c0170d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 11 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 5 11 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 4 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 10 client.30477.objecter _send_command 11 2024-03-11T21:35:33.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c01ab90 con 0x7f7d0c016c70 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7d0c016c70 0x7f7d0c0170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c016c70 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.318+0000 7f7d2a7fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7d0c016c70 0x7f7d0c0170d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.322+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c01ab90 con 0x7f7d0c016c70 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.322+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.322+0000 7f7d30963700 10 client.30477.objecter _finish_command 11 = system:0 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.322+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 4 2024-03-11T21:35:33.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.322+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 5 11 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 10 client.30477.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 12 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 12 2024-03-11T21:35:33.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c016550 osd=5 3 2024-03-11T21:35:33.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 5 12 2024-03-11T21:35:33.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 4 2024-03-11T21:35:33.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 10 client.30477.objecter _send_command 12 2024-03-11T21:35:33.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.338+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c019a10 con 0x7f7d0c016c70 2024-03-11T21:35:33.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c019a10 con 0x7f7d0c016c70 2024-03-11T21:35:33.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 10 client.30477.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 4 2024-03-11T21:35:33.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 5 12 2024-03-11T21:35:33.316 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 10 client.30477.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 13 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7d0c01a130 0x7f7d0c01a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 13 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 6 13 2024-03-11T21:35:33.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 4 2024-03-11T21:35:33.318 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 10 client.30477.objecter _send_command 13 2024-03-11T21:35:33.318 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c01df30 con 0x7f7d0c01a130 2024-03-11T21:35:33.318 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7d0c01a130 0x7f7d0c01a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.318 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c01a130 2024-03-11T21:35:33.318 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d2b7fe700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7d0c01a130 0x7f7d0c01a570 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- 172.21.15.143:0/1608113119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c069b10 msgr2=0x7fc61c069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- 172.21.15.143:0/1608113119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c069b10 0x7fc61c069ef0 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7fc618009990 tx=0x7fc618009ca0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- 172.21.15.143:0/1608113119 shutdown_connections 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- 172.21.15.143:0/1608113119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c10e8f0 0x7fc61c110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- 172.21.15.143:0/1608113119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c069b10 0x7fc61c069ef0 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- 172.21.15.143:0/1608113119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 0x7fc61c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- 172.21.15.143:0/1608113119 >> 172.21.15.143:0/1608113119 conn(0x7fc61c076d80 msgr2=0x7fc61c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- 172.21.15.143:0/1608113119 shutdown_connections 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- 172.21.15.143:0/1608113119 wait complete. 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 librados: starting msgr at 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 librados: starting objecter 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 Processor -- start 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- start start 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 librados: setting wanted keys 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 librados: calling monclient init 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 0x7fc61c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1137e0 con 0x7fc61c069b10 2024-03-11T21:35:33.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c113660 con 0x7fc61c06a430 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc6247b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c113960 con 0x7fc61c10e8f0 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c01df30 con 0x7f7d0c01a130 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 10 client.30477.objecter _finish_command 13 = system:0 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 4 2024-03-11T21:35:33.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 6 13 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc62254e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc622d4f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.342+0000 7fc622d4f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49854/0 (socket says 172.21.15.143:49854) 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33948/0 (socket says 172.21.15.143:33948) 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc622d4f700 1 -- 172.21.15.143:0/2402642413 learned_addr learned my addr 172.21.15.143:0/2402642413 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc621d4d700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 0x7fc61c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 -- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 msgr2=0x7fc61c19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 -- 172.21.15.143:0/2402642413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 msgr2=0x7fc61c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 0x7fc61c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 -- 172.21.15.143:0/2402642413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c19ed30 con 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc622d4f700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc62254e700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fc6180059d0 tx=0x7fc6180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 10 client.?.objecter ms_handle_connect 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618027070 con 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc618022470 con 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c19efc0 con 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618017650 con 0x7fc61c069b10 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6247b2700 10 client.30483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:33.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c1a68a0 con 0x7fc61c069b10 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6247b2700 1 librados: init done 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c101320 con 0x7fc61c069b10 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc61802a070 con 0x7fc61c069b10 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 0x7fc608071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(497..497 src has 87..497) v4 ==== 9657+0+0 (secure 0 0 0) 0x7fc61808f9f0 con 0x7fc61c069b10 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 10 client.30483.objecter ms_dispatch 0x7fc61c001550 osd_map(497..497 src has 87..497) v4 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 3 client.30483.objecter handle_osd_map got epochs [497,497] > 0 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc6137fe700 3 client.30483.objecter handle_osd_map decoding full epoch 497 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.346+0000 7fc621d4d700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 0x7fc608071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.350+0000 7fc6137fe700 20 client.30483.objecter dump_active .. 0 homeless 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.350+0000 7fc6137fe700 10 client.30483.objecter ms_handle_connect 0x7fc60806f820 2024-03-11T21:35:33.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.350+0000 7fc621d4d700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 0x7fc608071ce0 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0775f0 tx=0x7fc60c00a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.354+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc618013070 con 0x7fc61c069b10 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 10 client.30477.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 14 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 14 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c019a10 osd=6 3 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 6 14 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 4 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 10 client.30477.objecter _send_command 14 2024-03-11T21:35:33.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.362+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c014d80 con 0x7f7d0c01a130 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c014d80 con 0x7f7d0c01a130 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d30963700 10 client.30477.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 4 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 6 14 2024-03-11T21:35:33.342 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:35:33.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 10 client.30477.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:33.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 15 2024-03-11T21:35:33.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7d0c0154a0 0x7f7d0c0158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:33.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 15 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 7 15 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 4 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 10 client.30477.objecter _send_command 15 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d0c015f90 con 0x7f7d0c0154a0 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.366+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7d0c0154a0 0x7f7d0c0158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 10 client.30477.objecter ms_handle_connect 0x7f7d0c0154a0 2024-03-11T21:35:33.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d2affd700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7d0c0154a0 0x7f7d0c0158e0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:33.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d0c015f90 con 0x7f7d0c0154a0 2024-03-11T21:35:33.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:35:33.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 10 client.30477.objecter _finish_command 15 = system:0 2024-03-11T21:35:33.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 4 2024-03-11T21:35:33.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.370+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 7 15 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 10 client.30477.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign -1 16 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 20 client.30477.objecter _get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_remove -1 16 2024-03-11T21:35:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 20 client.30477.objecter get_session s=0x7f7d0c014d80 osd=7 3 2024-03-11T21:35:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 15 client.30477.objecter _session_command_op_assign 7 16 2024-03-11T21:35:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 4 2024-03-11T21:35:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 10 client.30477.objecter _send_command 16 2024-03-11T21:35:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.386+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d0c015f90 con 0x7f7d0c0154a0 2024-03-11T21:35:33.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.390+0000 7f7d30963700 1 -- 172.21.15.143:0/3680999184 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0c015f90 con 0x7f7d0c0154a0 2024-03-11T21:35:33.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.390+0000 7f7d30963700 10 client.30477.objecter ms_dispatch 0x7f7d2c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:33.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.390+0000 7f7d30963700 10 client.30477.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:33.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.390+0000 7f7d30963700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 4 2024-03-11T21:35:33.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.390+0000 7f7d30963700 15 client.30477.objecter _session_command_op_remove 7 16 2024-03-11T21:35:33.367 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 librados: watch_flush enter 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 librados: watch_flush exit 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.0 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7d0c004730 msgr2=0x7f7d0c004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f7d0c004730 0x7f7d0c004b70 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0053e0 osd=0 1 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.1 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7d0c002750 msgr2=0x7f7d0c002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f7d0c002750 0x7f7d0c002bd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c004560 osd=1 1 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.2 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7d0c00b240 msgr2=0x7f7d0c00d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7d0c00b240 0x7f7d0c00d630 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c001b30 osd=2 1 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.3 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7d0c008720 msgr2=0x7f7d0c00ed80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f7d0c008720 0x7f7d0c00ed80 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c0039c0 osd=3 1 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.4 2024-03-11T21:35:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7d0c007db0 msgr2=0x7f7d0c0147f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f7d0c007db0 0x7f7d0c0147f0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c007c40 osd=4 1 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.5 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7d0c016c70 msgr2=0x7f7d0c0170d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f7d0c016c70 0x7f7d0c0170d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c016550 osd=5 1 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.6 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7d0c01a130 msgr2=0x7f7d0c01a570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f7d0c01a130 0x7f7d0c01a570 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c019a10 osd=6 1 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter close_session for osd.7 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7d0c0154a0 msgr2=0x7f7d0c0158e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.370 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f7d0c0154a0 0x7f7d0c0158e0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter put_session s=0x7f7d0c014d80 osd=7 1 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 20 client.30477.objecter shutdown clearing up homeless session... 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 10 client.30477.objecter successfully canceled tick 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 reap_dead start 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 reap_dead start 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 reap_dead start 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 reap_dead start 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d2b7fe700 1 -- 172.21.15.143:0/3680999184 reap_dead start 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 msgr2=0x7f7d14071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 0x7f7d14071ce0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c005fd0 tx=0x7f7d1c0095b0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 msgr2=0x7f7d2c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 0x7f7d2c101890 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009790 tx=0x7f7d2400ccb0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 shutdown_connections 2024-03-11T21:35:33.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2c107e90 0x7f7d2c101890 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 --2- 172.21.15.143:0/3680999184 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7d1406f820 0x7f7d14071ce0 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 >> 172.21.15.143:0/3680999184 conn(0x7f7d2c076d80 msgr2=0x7f7d2c108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:33.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 shutdown_connections 2024-03-11T21:35:33.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 -- 172.21.15.143:0/3680999184 wait complete. 2024-03-11T21:35:33.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.394+0000 7f7d127fc700 1 librados: shutdown 2024-03-11T21:35:33.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:33.373 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:33.495 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:33 smithi143 bash[21512]: audit 2024-03-11T21:35:33.200521+0000 mon.a (mon.0) 2824 : audit 0 from='client.? 172.21.15.143:0/3680999184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:33.495 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:33 smithi143 bash[29284]: audit 2024-03-11T21:35:33.200521+0000 mon.a (mon.0) 2824 : audit 0 from='client.? 172.21.15.143:0/3680999184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:33.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:33.513 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:33.520 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:33 smithi203 bash[23654]: audit 2024-03-11T21:35:33.200521+0000 mon.a (mon.0) 2824 : audit 0 from='client.? 172.21.15.143:0/3680999184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:33.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.570+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc61c19b080 con 0x7fc61c069b10 2024-03-11T21:35:33.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.570+0000 7fc6137fe700 1 -- 172.21.15.143:0/2402642413 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v497) v1 ==== 81+0+60993 (secure 0 0 0) 0x7fc61805c6a0 con 0x7fc61c069b10 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: "epoch": 497, 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: "created": "2024-03-11T21:12:47.384719+0000", 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: "modified": "2024-03-11T21:35:31.470581+0000", 2024-03-11T21:35:33.547 INFO:teuthology.orchestra.run.smithi143.stdout: "last_up_change": "2024-03-11T21:31:36.320182+0000", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "last_in_change": "2024-03-11T21:21:55.456354+0000", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_num": 5801984, 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_set": [ 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "noscrub", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "pglog_hardlimit", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "purged_snapdirs", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_deletes", 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "sortbitwise" 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_version": 111, 2024-03-11T21:35:33.548 INFO:teuthology.orchestra.run.smithi143.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_snapshot": "", 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_max": 288, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "max_osd": 8, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "min_compat_client": "luminous", 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "require_osd_release": "squid", 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "allow_crimson": false, 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: "pools": [ 2024-03-11T21:35:33.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 1, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".mgr", 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:18:08.036690+0000", 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.550 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 1, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 1, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 1, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 1, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 1, 2024-03-11T21:35:33.551 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "141", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.553 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.555 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_max": 32, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_min": 1 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr": {} 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 7.8899998664855957, 2024-03-11T21:35:33.556 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 7.8899998664855957, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 3, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 3, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.557 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 2, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".rgw.root", 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:01.432928+0000", 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.558 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "78", 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.560 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.563 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.564 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 3, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:03.441476+0000", 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.565 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.566 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "80", 2024-03-11T21:35:33.567 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.568 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.569 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.570 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.571 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 4, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "datapool", 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:04.441432+0000", 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.572 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 3, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 3, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 3, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 3, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 3, 2024-03-11T21:35:33.573 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "87", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 2, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 87, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.575 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.576 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd": {} 2024-03-11T21:35:33.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2.6500000953674316, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2.6500000953674316, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.578 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 5, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:05.388543+0000", 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.579 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.580 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "82", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.581 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.582 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.583 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.584 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.25, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.25, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.25, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.25, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.585 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 6, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:07.642696+0000", 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.586 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.587 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "84", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.588 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.589 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.590 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_bias": 4 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:35:33.591 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.592 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 9, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:36.978109+0000", 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.593 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.594 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.597 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.598 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 39, 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "SnapListsmithi143-72399-1", 2024-03-11T21:35:33.599 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:49.753089+0000", 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "153", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.602 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.603 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.604 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.605 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 118, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-6", 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:30.235634+0000", 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.606 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.607 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "495", 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "454", 2024-03-11T21:35:33.608 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "454", 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "454", 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 19, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 495, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:35:33.611 INFO:teuthology.orchestra.run.smithi143.stdout: "fingerprint_algorithm": "sha1" 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.612 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 288, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-65", 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:35:21.032439+0000", 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:35:33.613 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:35:33.614 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "492", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:35:33.615 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:35:33.616 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:35:33.617 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.618 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2.25, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2.25, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2.25, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2.25, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.619 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "osds": [ 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d", 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 8, 2024-03-11T21:35:33.620 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6802", 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6803", 2024-03-11T21:35:33.621 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6804", 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.622 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6805", 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.623 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.624 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6808", 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6809", 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.625 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6806", 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6807", 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:35:33.626 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6803/2903432023", 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6805/2903432023", 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6809/2903432023", 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6807/2903432023", 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.627 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02", 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 13, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 246, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 295, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 247, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.628 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6810", 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6811", 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.629 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6812", 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.630 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6813", 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6816", 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.631 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6817", 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.632 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6814", 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6815", 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6811/4235040845", 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6813/4235040845", 2024-03-11T21:35:33.633 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6817/4235040845", 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6815/4235040845", 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d", 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.634 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 19, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.635 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6818", 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6819", 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.636 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6820", 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6821", 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.637 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6824", 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6825", 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.638 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6822", 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6823", 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.639 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6819/2592131625", 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6821/2592131625", 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6825/2592131625", 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6823/2592131625", 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.640 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc", 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 27, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6826", 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6827", 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.642 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6828", 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6829", 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.643 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6832", 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.644 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6833", 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6830", 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.645 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6831", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6827/2146571980", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6829/2146571980", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6833/2146571980", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6831/2146571980", 2024-03-11T21:35:33.646 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "d7b9e088-d377-4335-8863-11d581828f7e", 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.647 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 35, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6800", 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.648 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6801", 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.649 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6802", 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6803", 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.650 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6806", 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6807", 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.651 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6804", 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6805", 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.652 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6801/2090835691", 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6803/2090835691", 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6807/2090835691", 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6805/2090835691", 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.653 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01", 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 46, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.654 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6808", 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6809", 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.655 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6810", 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6811", 2024-03-11T21:35:33.656 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6814", 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.657 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6815", 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6812", 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.658 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6813", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6809/2432038345", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6811/2432038345", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2432038345", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6813/2432038345", 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.659 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241", 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.660 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 56, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6816", 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6817", 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6818", 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.662 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6819", 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6822", 2024-03-11T21:35:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6823", 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6820", 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6821", 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6817/513748834", 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6819/513748834", 2024-03-11T21:35:33.665 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6823/513748834", 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6821/513748834", 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "88b53171-69d2-4922-a787-e63d44d01c60", 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:35:33.666 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 66, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 490, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.667 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6824", 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6825", 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.668 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6826", 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6827", 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6830", 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6831", 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.670 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6828", 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6829", 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:35:33.671 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6825/1251497952", 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6827/1251497952", 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6831/1251497952", 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6829/1251497952", 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.672 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_xinfo": [ 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:27.905971+0000", 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.673 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:47.884800+0000", 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 247 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:35:33.674 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:26.727386+0000", 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.675 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:35:31.125705+0000", 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.676 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:48.741625+0000", 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.677 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:35:29.648335+0000", 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:22.398299+0000", 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:34:36.633540+0000", 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.679 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap": [], 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_items": [ 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "2.18", 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.680 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.10", 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.18", 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.1d", 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.682 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.1e", 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 4 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.683 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.2", 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.9", 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.684 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.17", 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.685 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_primaries": [], 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_temp": [], 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_temp": [], 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "blocklist": { 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3638739149": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4274199357": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3217951344": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3912416566": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/906834629": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2791822486": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:35:33.686 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2190104670": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4171894415": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2845155916": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2913593434": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/143435387": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2930358289": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3638739149": "2024-03-12T21:23:57.022184+0000" 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "range_blocklist": {}, 2024-03-11T21:35:33.687 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profiles": { 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "default": { 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van" 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps_queue": [ 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 28, 2024-03-11T21:35:33.688 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 5, 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 35, 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 1, 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:35:33.689 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 89, 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 4, 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 7, 2024-03-11T21:35:33.690 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 118, 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 19, 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.691 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 137, 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 3 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.693 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 156, 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 165, 2024-03-11T21:35:33.694 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 173, 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:35:33.695 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 178, 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.696 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 185, 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 198, 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 4, 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:35:33.697 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 210, 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 7, 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 230, 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 8, 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 241, 2024-03-11T21:35:33.699 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 10, 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 244, 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 11, 2024-03-11T21:35:33.700 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 256, 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 14, 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:35:33.701 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "new_removed_snaps": [], 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "new_purged_snaps": [], 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_node_flags": {}, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "device_class_flags": {}, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode": { 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_enabled": false, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_bucket_count": 0, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_bucket": 0 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:35:33.702 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:35:33.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 10 librados: watch_flush enter 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 10 librados: watch_flush exit 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 20 client.30483.objecter shutdown clearing up homeless session... 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 10 client.30483.objecter successfully canceled tick 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 msgr2=0x7fc608071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 0x7fc608071ce0 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0775f0 tx=0x7fc60c00a380 comp rx=0 tx=0).stop 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 msgr2=0x7fc61c199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fc6180059d0 tx=0x7fc6180047b0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 shutdown_connections 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc60806f820 0x7fc608071ce0 unknown :-1 s=CLOSED pgs=3379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc61c069b10 0x7fc61c199d70 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc61c10e8f0 0x7fc61c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 --2- 172.21.15.143:0/2402642413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61c06a430 0x7fc61c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 >> 172.21.15.143:0/2402642413 conn(0x7fc61c076d80 msgr2=0x7fc61c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 shutdown_connections 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 -- 172.21.15.143:0/2402642413 wait complete. 2024-03-11T21:35:33.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:33.574+0000 7fc6247b2700 1 librados: shutdown 2024-03-11T21:35:33.728 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:33.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:33.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:33.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:33.987 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:33.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:34.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:34.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:34.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:34.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:34.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:34.376 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:34.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:34.493 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:34 smithi203 bash[23654]: cluster 2024-03-11T21:35:33.423575+0000 mgr.y (mgr.14550) 499 : cluster 0 pgmap v751: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T21:35:34.493 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:34 smithi203 bash[23654]: cluster 2024-03-11T21:35:33.517460+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:34.493 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:34 smithi203 bash[23654]: cluster 2024-03-11T21:35:33.517853+0000 osd.6 (osd.6) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:34.494 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:34 smithi203 bash[23654]: audit 2024-03-11T21:35:33.573222+0000 mon.a (mon.0) 2825 : audit 0 from='client.? 172.21.15.143:0/2402642413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:35:34.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:34.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:34.637 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:34.726 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[29284]: cluster 2024-03-11T21:35:33.423575+0000 mgr.y (mgr.14550) 499 : cluster 0 pgmap v751: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[29284]: cluster 2024-03-11T21:35:33.517460+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[29284]: cluster 2024-03-11T21:35:33.517853+0000 osd.6 (osd.6) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[29284]: audit 2024-03-11T21:35:33.573222+0000 mon.a (mon.0) 2825 : audit 0 from='client.? 172.21.15.143:0/2402642413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[21512]: cluster 2024-03-11T21:35:33.423575+0000 mgr.y (mgr.14550) 499 : cluster 0 pgmap v751: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[21512]: cluster 2024-03-11T21:35:33.517460+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[21512]: cluster 2024-03-11T21:35:33.517853+0000 osd.6 (osd.6) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:34.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:34 smithi143 bash[21512]: audit 2024-03-11T21:35:33.573222+0000 mon.a (mon.0) 2825 : audit 0 from='client.? 172.21.15.143:0/2402642413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:35:34.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:34.768 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:34.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:34.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:35.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:35.034 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:35.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:35.165 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:35.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:35.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- 172.21.15.143:0/2621377392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 msgr2=0x7f677c106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- 172.21.15.143:0/2621377392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c106b40 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f6764009990 tx=0x7f6764009ca0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- 172.21.15.143:0/2621377392 shutdown_connections 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- 172.21.15.143:0/2621377392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c107220 0x7f677c10edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- 172.21.15.143:0/2621377392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c06a6a0 0x7f677c06aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- 172.21.15.143:0/2621377392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c106b40 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- 172.21.15.143:0/2621377392 >> 172.21.15.143:0/2621377392 conn(0x7f677c077de0 msgr2=0x7f677c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- 172.21.15.143:0/2621377392 shutdown_connections 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- 172.21.15.143:0/2621377392 wait complete. 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 librados: starting msgr at 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 librados: starting objecter 2024-03-11T21:35:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 Processor -- start 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- start start 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 librados: setting wanted keys 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 librados: calling monclient init 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c107220 0x7f677c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c111640 con 0x7f677c06b050 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c1114c0 con 0x7f677c107220 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.370+0000 7f67812aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c1117c0 con 0x7f677c06a6a0 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677a7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49872/0 (socket says 172.21.15.143:49872) 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 -- 172.21.15.143:0/3313258219 learned_addr learned my addr 172.21.15.143:0/3313258219 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:35.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677b7fe700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c107220 0x7f677c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 -- 172.21.15.143:0/3313258219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c107220 msgr2=0x7f677c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c107220 0x7f677c101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 -- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 msgr2=0x7f677c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 -- 172.21.15.143:0/3313258219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c1a6170 con 0x7f677c06a6a0 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677a7fc700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:35:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677affd700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f676c00d980 tx=0x7f676c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 10 client.?.objecter ms_handle_connect 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c016070 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c1a6460 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f676c004830 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c005440 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f67812aa700 10 client.41690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c1a69c0 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f67812aa700 1 librados: init done 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c04faa0 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f676c0055e0 con 0x7f677c06a6a0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 0x7f6768071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677a7fc700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 0x7f6768071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:35.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(498..498 src has 87..498) v4 ==== 9657+0+0 (secure 0 0 0) 0x7f676c08e820 con 0x7f677c06a6a0 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 10 client.41690.objecter ms_dispatch 0x7f677c001550 osd_map(498..498 src has 87..498) v4 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 3 client.41690.objecter handle_osd_map got epochs [498,498] > 0 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 3 client.41690.objecter handle_osd_map decoding full epoch 498 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 20 client.41690.objecter dump_active .. 0 homeless 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f677a7fc700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 0x7f6768071ce0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f676400b5c0 tx=0x7f6764005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:35.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.374+0000 7f6773fff700 10 client.41690.objecter ms_handle_connect 0x7f676806f820 2024-03-11T21:35:35.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.378+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f676c01f020 con 0x7f677c06a6a0 2024-03-11T21:35:35.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:35.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:35.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:35.540 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:35 smithi203 bash[23654]: cluster 2024-03-11T21:35:34.512082+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:35:35.556 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:35.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:35.602+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f677c1a6bf0 con 0x7f677c06a6a0 2024-03-11T21:35:35.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:35.687 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:35.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:35.801 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:35 smithi143 bash[29284]: cluster 2024-03-11T21:35:34.512082+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:35:35.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:35 smithi143 bash[21512]: cluster 2024-03-11T21:35:34.512082+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:35:35.817 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:35.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:35.947 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:36.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:36.078 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:36.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.208 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:36.282 INFO:tasks.thrashosds.thrasher:j is {'epoch': 497, 'fsid': 'eebe2e36-dfeb-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:12:47.384719+0000', 'modified': '2024-03-11T21:35:31.470581+0000', 'last_up_change': '2024-03-11T21:31:36.320182+0000', 'last_in_change': '2024-03-11T21:21:55.456354+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 111, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 288, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:18:08.036690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:23:01.432928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:23:03.441476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:23:04.441432+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:23:05.388543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:23:07.642696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-11T21:28:36.978109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 39, 'pool_name': 'SnapListsmithi143-72399-1', 'create_time': '2024-03-11T21:28:49.753089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 118, 'pool_name': 'test-rados-api-smithi143-71773-6', 'create_time': '2024-03-11T21:29:30.235634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '495', 'last_force_op_resend': '454', 'last_force_op_resend_prenautilus': '454', 'last_force_op_resend_preluminous': '454', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 495, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 288, 'pool_name': 'test-rados-api-smithi143-71773-65', 'create_time': '2024-03-11T21:35:21.032439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '492', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6802', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6803', 'nonce': 2903432023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6804', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6805', 'nonce': 2903432023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6808', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6809', 'nonce': 2903432023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6806', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6807', 'nonce': 2903432023}]}, 'public_addr': '172.21.15.143:6803/2903432023', 'cluster_addr': '172.21.15.143:6805/2903432023', 'heartbeat_back_addr': '172.21.15.143:6809/2903432023', 'heartbeat_front_addr': '172.21.15.143:6807/2903432023', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2948dc44-68d3-4b3d-92bf-2b0c46374f02', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 246, 'up_from': 295, 'up_thru': 490, 'down_at': 247, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6810', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6811', 'nonce': 4235040845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6812', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6813', 'nonce': 4235040845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6816', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6817', 'nonce': 4235040845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6814', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6815', 'nonce': 4235040845}]}, 'public_addr': '172.21.15.143:6811/4235040845', 'cluster_addr': '172.21.15.143:6813/4235040845', 'heartbeat_back_addr': '172.21.15.143:6817/4235040845', 'heartbeat_front_addr': '172.21.15.143:6815/4235040845', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7217589a-5835-407b-91a7-f5c055a48f3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6818', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6819', 'nonce': 2592131625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6820', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6821', 'nonce': 2592131625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6824', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6825', 'nonce': 2592131625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6822', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6823', 'nonce': 2592131625}]}, 'public_addr': '172.21.15.143:6819/2592131625', 'cluster_addr': '172.21.15.143:6821/2592131625', 'heartbeat_back_addr': '172.21.15.143:6825/2592131625', 'heartbeat_front_addr': '172.21.15.143:6823/2592131625', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a3e7177f-a5a0-44ad-a65d-9aeac3a680cc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6826', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6827', 'nonce': 2146571980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6828', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6829', 'nonce': 2146571980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2146571980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6830', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6831', 'nonce': 2146571980}]}, 'public_addr': '172.21.15.143:6827/2146571980', 'cluster_addr': '172.21.15.143:6829/2146571980', 'heartbeat_back_addr': '172.21.15.143:6833/2146571980', 'heartbeat_front_addr': '172.21.15.143:6831/2146571980', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd7b9e088-d377-4335-8863-11d581828f7e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6800', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 2090835691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6802', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 2090835691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6806', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 2090835691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6804', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 2090835691}]}, 'public_addr': '172.21.15.203:6801/2090835691', 'cluster_addr': '172.21.15.203:6803/2090835691', 'heartbeat_back_addr': '172.21.15.203:6807/2090835691', 'heartbeat_front_addr': '172.21.15.203:6805/2090835691', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b3b2a75c-68a5-4e00-8068-622e1ebbaa01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6808', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 2432038345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6810', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 2432038345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6814', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2432038345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6812', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2432038345}]}, 'public_addr': '172.21.15.203:6809/2432038345', 'cluster_addr': '172.21.15.203:6811/2432038345', 'heartbeat_back_addr': '172.21.15.203:6815/2432038345', 'heartbeat_front_addr': '172.21.15.203:6813/2432038345', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ceffb234-db0c-4ffa-a142-9b201df9b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6816', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6817', 'nonce': 513748834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6818', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6819', 'nonce': 513748834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6822', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6823', 'nonce': 513748834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6820', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6821', 'nonce': 513748834}]}, 'public_addr': '172.21.15.203:6817/513748834', 'cluster_addr': '172.21.15.203:6819/513748834', 'heartbeat_back_addr': '172.21.15.203:6823/513748834', 'heartbeat_front_addr': '172.21.15.203:6821/513748834', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '88b53171-69d2-4922-a787-e63d44d01c60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6824', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6825', 'nonce': 1251497952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6826', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6827', 'nonce': 1251497952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6830', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6831', 'nonce': 1251497952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6828', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6829', 'nonce': 1251497952}]}, 'public_addr': '172.21.15.203:6825/1251497952', 'cluster_addr': '172.21.15.203:6827/1251497952', 'heartbeat_back_addr': '172.21.15.203:6831/1251497952', 'heartbeat_front_addr': '172.21.15.203:6829/1251497952', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:27.905971+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:47.884800+0000', 'dead_epoch': 247}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:26.727386+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:35:31.125705+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:48.741625+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:35:29.648335+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:22.398299+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:34:36.633540+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.10', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.1e', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '39.2', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '39.9', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '39.17', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.143:6801/3638739149': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4274199357': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/3217951344': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/3912416566': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/906834629': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2791822486': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2190104670': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4171894415': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2845155916': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2913593434': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/143435387': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2930358289': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6800/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3638739149': '2024-03-12T21:23:57.022184+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 28, 'snaps': [{'begin': 5, 'length': 2}]}, {'pool': 35, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 89, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 118, 'snaps': [{'begin': 19, 'length': 1}]}, {'pool': 137, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 178, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 185, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 198, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 210, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 230, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 244, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 256, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:35:36.282 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-11T21:35:36.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.339 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.374+0000 7f6773fff700 1 -- 172.21.15.143:0/3313258219 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v499) v1 ==== 97+0+0 (secure 0 0 0) 0x7f676c022350 con 0x7f677c06a6a0 2024-03-11T21:35:36.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 10 librados: watch_flush enter 2024-03-11T21:35:36.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 10 librados: watch_flush exit 2024-03-11T21:35:36.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 20 client.41690.objecter shutdown clearing up homeless session... 2024-03-11T21:35:36.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 10 client.41690.objecter successfully canceled tick 2024-03-11T21:35:36.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 msgr2=0x7f6768071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:36.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 0x7f6768071ce0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f676400b5c0 tx=0x7f6764005960 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 msgr2=0x7f677c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f676c00d980 tx=0x7f676c00dc90 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 shutdown_connections 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f676806f820 0x7f6768071ce0 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f677c107220 0x7f677c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677c06a6a0 0x7f677c100d90 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 --2- 172.21.15.143:0/3313258219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f677c06b050 0x7f677c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 >> 172.21.15.143:0/3313258219 conn(0x7f677c077de0 msgr2=0x7f677c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:36.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 shutdown_connections 2024-03-11T21:35:36.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 -- 172.21.15.143:0/3313258219 wait complete. 2024-03-11T21:35:36.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:36.378+0000 7f67812aa700 1 librados: shutdown 2024-03-11T21:35:36.356 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:35:36.359 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:36.359 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:36.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.471 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:36.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:36.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: cluster 2024-03-11T21:35:35.424112+0000 mgr.y (mgr.14550) 500 : cluster 0 pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:35.608067+0000 mon.c (mon.2) 976 : audit 1 from='client.? 172.21.15.143:0/3313258219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:35.608600+0000 mon.a (mon.0) 2827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: cluster 2024-03-11T21:35:36.362284+0000 mon.a (mon.0) 2828 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:36.365978+0000 mon.a (mon.0) 2829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: cluster 2024-03-11T21:35:36.368538+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:36.478390+0000 mon.c (mon.2) 977 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:36.478685+0000 mon.a (mon.0) 2831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:36.479108+0000 mon.c (mon.2) 978 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[21512]: audit 2024-03-11T21:35:36.479352+0000 mon.a (mon.0) 2832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: cluster 2024-03-11T21:35:35.424112+0000 mgr.y (mgr.14550) 500 : cluster 0 pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:35.608067+0000 mon.c (mon.2) 976 : audit 1 from='client.? 172.21.15.143:0/3313258219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:35.608600+0000 mon.a (mon.0) 2827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: cluster 2024-03-11T21:35:36.362284+0000 mon.a (mon.0) 2828 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:36.365978+0000 mon.a (mon.0) 2829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: cluster 2024-03-11T21:35:36.368538+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:36.478390+0000 mon.c (mon.2) 977 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:36.478685+0000 mon.a (mon.0) 2831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:36.479108+0000 mon.c (mon.2) 978 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:36 smithi143 bash[29284]: audit 2024-03-11T21:35:36.479352+0000 mon.a (mon.0) 2832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:36.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: cluster 2024-03-11T21:35:35.424112+0000 mgr.y (mgr.14550) 500 : cluster 0 pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:35.608067+0000 mon.c (mon.2) 976 : audit 1 from='client.? 172.21.15.143:0/3313258219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:35.608600+0000 mon.a (mon.0) 2827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: cluster 2024-03-11T21:35:36.362284+0000 mon.a (mon.0) 2828 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:36.365978+0000 mon.a (mon.0) 2829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: cluster 2024-03-11T21:35:36.368538+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:35:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:36.478390+0000 mon.c (mon.2) 977 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:36.478685+0000 mon.a (mon.0) 2831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:36.479108+0000 mon.c (mon.2) 978 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:36 smithi203 bash[23654]: audit 2024-03-11T21:35:36.479352+0000 mon.a (mon.0) 2832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-65"}]: dispatch 2024-03-11T21:35:36.728 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:36.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:36.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:36.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:36.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:37.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:37.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:37.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:37.252 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:37.356 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:37.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:37.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:37.517 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:37.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:37.646 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:37.749 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:37.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:37.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:35:37.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:37.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:37 smithi203 bash[23654]: audit 2024-03-11T21:35:36.561933+0000 mon.c (mon.2) 979 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:37.890 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:37 smithi203 bash[23654]: cluster 2024-03-11T21:35:37.393775+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:35:37.905 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:38.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:38.036 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:38.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:38.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:38.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:37 smithi143 bash[21512]: audit 2024-03-11T21:35:36.561933+0000 mon.c (mon.2) 979 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:38.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:37 smithi143 bash[21512]: cluster 2024-03-11T21:35:37.393775+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:35:38.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:37 smithi143 bash[29284]: audit 2024-03-11T21:35:36.561933+0000 mon.c (mon.2) 979 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:38.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:37 smithi143 bash[29284]: cluster 2024-03-11T21:35:37.393775+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:35:38.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:35:38.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.325 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:38.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.451 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:38.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:38.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:38.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:38.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:38.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:38.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:38.950 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: cluster 2024-03-11T21:35:37.424563+0000 mgr.y (mgr.14550) 501 : cluster 0 pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: cluster 2024-03-11T21:35:37.762218+0000 osd.2 (osd.2) 249 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: cluster 2024-03-11T21:35:37.762465+0000 osd.2 (osd.2) 250 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: cluster 2024-03-11T21:35:38.395877+0000 mon.a (mon.0) 2834 : cluster 0 osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: audit 2024-03-11T21:35:38.405619+0000 mon.c (mon.2) 980 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:38.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:38 smithi203 bash[23654]: audit 2024-03-11T21:35:38.406033+0000 mon.a (mon.0) 2835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:38.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:39.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: cluster 2024-03-11T21:35:37.424563+0000 mgr.y (mgr.14550) 501 : cluster 0 pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: cluster 2024-03-11T21:35:37.762218+0000 osd.2 (osd.2) 249 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: cluster 2024-03-11T21:35:37.762465+0000 osd.2 (osd.2) 250 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: cluster 2024-03-11T21:35:38.395877+0000 mon.a (mon.0) 2834 : cluster 0 osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: audit 2024-03-11T21:35:38.405619+0000 mon.c (mon.2) 980 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[21512]: audit 2024-03-11T21:35:38.406033+0000 mon.a (mon.0) 2835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: cluster 2024-03-11T21:35:37.424563+0000 mgr.y (mgr.14550) 501 : cluster 0 pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:39.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: cluster 2024-03-11T21:35:37.762218+0000 osd.2 (osd.2) 249 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:39.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: cluster 2024-03-11T21:35:37.762465+0000 osd.2 (osd.2) 250 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:39.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: cluster 2024-03-11T21:35:38.395877+0000 mon.a (mon.0) 2834 : cluster 0 osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:35:39.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: audit 2024-03-11T21:35:38.405619+0000 mon.c (mon.2) 980 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:39.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:38 smithi143 bash[29284]: audit 2024-03-11T21:35:38.406033+0000 mon.a (mon.0) 2835 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:39.098 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:39.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:39.229 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:39.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:39.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:39.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:39.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:39.617 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:39.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:39.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:35:39.861 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[21512]: audit 2024-03-11T21:35:38.689017+0000 mgr.y (mgr.14550) 502 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:39.861 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[21512]: audit 2024-03-11T21:35:39.394476+0000 mon.a (mon.0) 2836 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:39.861 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[21512]: cluster 2024-03-11T21:35:39.398909+0000 mon.a (mon.0) 2837 : cluster 0 osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:35:39.861 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[21512]: audit 2024-03-11T21:35:39.405027+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:39.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[29284]: audit 2024-03-11T21:35:38.689017+0000 mgr.y (mgr.14550) 502 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:39.862 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[29284]: audit 2024-03-11T21:35:39.394476+0000 mon.a (mon.0) 2836 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:39.862 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[29284]: cluster 2024-03-11T21:35:39.398909+0000 mon.a (mon.0) 2837 : cluster 0 osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:35:39.862 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:39 smithi143 bash[29284]: audit 2024-03-11T21:35:39.405027+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:39.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:39.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:39.990 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:39 smithi203 bash[23654]: audit 2024-03-11T21:35:38.689017+0000 mgr.y (mgr.14550) 502 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:39.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:39 smithi203 bash[23654]: audit 2024-03-11T21:35:39.394476+0000 mon.a (mon.0) 2836 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:39.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:39 smithi203 bash[23654]: cluster 2024-03-11T21:35:39.398909+0000 mon.a (mon.0) 2837 : cluster 0 osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:35:39.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:39 smithi203 bash[23654]: audit 2024-03-11T21:35:39.405027+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:40.006 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:40.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.134 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:40.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:40.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:40.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:40.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:40.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:40.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:40.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:40.657 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:40.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:40.785 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:40.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:40.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:41.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:41.041 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:41.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: cluster 2024-03-11T21:35:39.425003+0000 mgr.y (mgr.14550) 503 : cluster 0 pgmap v759: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: cluster 2024-03-11T21:35:40.410706+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: audit 2024-03-11T21:35:40.472298+0000 mon.c (mon.2) 982 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: audit 2024-03-11T21:35:40.473053+0000 mon.a (mon.0) 2839 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: audit 2024-03-11T21:35:40.474314+0000 mon.c (mon.2) 983 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: audit 2024-03-11T21:35:40.474971+0000 mon.a (mon.0) 2840 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.154 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:40 smithi203 bash[23654]: audit 2024-03-11T21:35:40.720738+0000 mon.c (mon.2) 984 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: cluster 2024-03-11T21:35:39.425003+0000 mgr.y (mgr.14550) 503 : cluster 0 pgmap v759: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: cluster 2024-03-11T21:35:40.410706+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: audit 2024-03-11T21:35:40.472298+0000 mon.c (mon.2) 982 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: audit 2024-03-11T21:35:40.473053+0000 mon.a (mon.0) 2839 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: audit 2024-03-11T21:35:40.474314+0000 mon.c (mon.2) 983 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: audit 2024-03-11T21:35:40.474971+0000 mon.a (mon.0) 2840 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[21512]: audit 2024-03-11T21:35:40.720738+0000 mon.c (mon.2) 984 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: cluster 2024-03-11T21:35:39.425003+0000 mgr.y (mgr.14550) 503 : cluster 0 pgmap v759: 260 pgs: 2 creating+peering, 30 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: cluster 2024-03-11T21:35:40.410706+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: audit 2024-03-11T21:35:40.472298+0000 mon.c (mon.2) 982 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: audit 2024-03-11T21:35:40.473053+0000 mon.a (mon.0) 2839 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: audit 2024-03-11T21:35:40.474314+0000 mon.c (mon.2) 983 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: audit 2024-03-11T21:35:40.474971+0000 mon.a (mon.0) 2840 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-67"}]: dispatch 2024-03-11T21:35:41.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:40 smithi143 bash[29284]: audit 2024-03-11T21:35:40.720738+0000 mon.c (mon.2) 984 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:41.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:41.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:35:41.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:35:41.283 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-11T21:35:41.283 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:35:41.283 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:35:41.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-11T21:35:41.294 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:41.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:41.419 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:41.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.546 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:41.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:41.673 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:41.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.797 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:41.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:41.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:42.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:42.054 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:42.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:42.183 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:42.184 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:42.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: audit 2024-03-11T21:35:41.246789+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: audit 2024-03-11T21:35:41.258756+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: cluster 2024-03-11T21:35:41.375157+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: cluster 2024-03-11T21:35:41.425469+0000 mgr.y (mgr.14550) 504 : cluster 0 pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: audit 2024-03-11T21:35:41.940667+0000 mon.c (mon.2) 985 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: audit 2024-03-11T21:35:41.942456+0000 mon.c (mon.2) 986 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:42.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[23654]: audit 2024-03-11T21:35:41.950825+0000 mon.a (mon.0) 2844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.315 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:42.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: audit 2024-03-11T21:35:41.246789+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: audit 2024-03-11T21:35:41.258756+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: cluster 2024-03-11T21:35:41.375157+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: cluster 2024-03-11T21:35:41.425469+0000 mgr.y (mgr.14550) 504 : cluster 0 pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: audit 2024-03-11T21:35:41.940667+0000 mon.c (mon.2) 985 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: audit 2024-03-11T21:35:41.942456+0000 mon.c (mon.2) 986 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:42.429 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[21512]: audit 2024-03-11T21:35:41.950825+0000 mon.a (mon.0) 2844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: audit 2024-03-11T21:35:41.246789+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: audit 2024-03-11T21:35:41.258756+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: cluster 2024-03-11T21:35:41.375157+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: cluster 2024-03-11T21:35:41.425469+0000 mgr.y (mgr.14550) 504 : cluster 0 pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: audit 2024-03-11T21:35:41.940667+0000 mon.c (mon.2) 985 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: audit 2024-03-11T21:35:41.942456+0000 mon.c (mon.2) 986 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:42.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:42 smithi143 bash[29284]: audit 2024-03-11T21:35:41.950825+0000 mon.a (mon.0) 2844 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:35:42.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:42.521 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:35:42 smithi203 bash[51372]: logger=cleanup t=2024-03-11T21:35:42.335699379Z level=info msg="Completed cleanup jobs" duration=332.413417ms 2024-03-11T21:35:42.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:42.572 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:42.592 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:42.593 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:42.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:42.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:42.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:42.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:42.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:42.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:43.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:43.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:43.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:43.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.498 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:43.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:43.607 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:43.629 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:43.630 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:43 smithi203 bash[23654]: cluster 2024-03-11T21:35:42.388047+0000 mon.a (mon.0) 2845 : cluster 0 osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:35:43.630 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:43 smithi203 bash[23654]: audit 2024-03-11T21:35:42.392494+0000 mon.c (mon.2) 987 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:43 smithi203 bash[23654]: audit 2024-03-11T21:35:42.393361+0000 mon.a (mon.0) 2846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:43.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.760 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.760 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[29284]: cluster 2024-03-11T21:35:42.388047+0000 mon.a (mon.0) 2845 : cluster 0 osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:35:43.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[29284]: audit 2024-03-11T21:35:42.392494+0000 mon.c (mon.2) 987 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.761 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[29284]: audit 2024-03-11T21:35:42.393361+0000 mon.a (mon.0) 2846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[21512]: cluster 2024-03-11T21:35:42.388047+0000 mon.a (mon.0) 2845 : cluster 0 osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:35:43.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[21512]: audit 2024-03-11T21:35:42.392494+0000 mon.c (mon.2) 987 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.761 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:43 smithi143 bash[21512]: audit 2024-03-11T21:35:42.393361+0000 mon.a (mon.0) 2846 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:43.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:43.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:43.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:44.019 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:44.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:44.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:44.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:44.277 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:44.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:44.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.380871+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:44.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: cluster 2024-03-11T21:35:43.387507+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:35:44.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.424738+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:44.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: cluster 2024-03-11T21:35:43.425957+0000 mgr.y (mgr.14550) 505 : cluster 0 pgmap v765: 260 pgs: 26 creating+peering, 6 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:44.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.471939+0000 mon.c (mon.2) 989 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.472357+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.472899+0000 mon.c (mon.2) 990 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.392 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:44 smithi203 bash[23654]: audit 2024-03-11T21:35:43.473187+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.403 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:44.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:44.529 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:44.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.380871+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: cluster 2024-03-11T21:35:43.387507+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.424738+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: cluster 2024-03-11T21:35:43.425957+0000 mgr.y (mgr.14550) 505 : cluster 0 pgmap v765: 260 pgs: 26 creating+peering, 6 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.471939+0000 mon.c (mon.2) 989 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.472357+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.472899+0000 mon.c (mon.2) 990 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.642 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[29284]: audit 2024-03-11T21:35:43.473187+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.380871+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: cluster 2024-03-11T21:35:43.387507+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.424738+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: cluster 2024-03-11T21:35:43.425957+0000 mgr.y (mgr.14550) 505 : cluster 0 pgmap v765: 260 pgs: 26 creating+peering, 6 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.471939+0000 mon.c (mon.2) 989 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.472357+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.472899+0000 mon.c (mon.2) 990 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.643 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:44 smithi143 bash[21512]: audit 2024-03-11T21:35:43.473187+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-69"}]: dispatch 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:44.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:44.783 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:44.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:44.911 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:45.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:45.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:45.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:45.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:45.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.304 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:45.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:45.437 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:45.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:45 smithi203 bash[23654]: cluster 2024-03-11T21:35:44.413108+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:35:45.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:45.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:45.651 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:45.652 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:45 smithi143 bash[29284]: cluster 2024-03-11T21:35:44.413108+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:35:45.652 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:45 smithi143 bash[21512]: cluster 2024-03-11T21:35:44.413108+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:35:45.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.699 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:45.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:45.826 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:45.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:46.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:46.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:46.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:46.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.340 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:46.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:46.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:46.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.579 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:46 smithi203 bash[23654]: cluster 2024-03-11T21:35:45.426986+0000 mgr.y (mgr.14550) 506 : cluster 0 pgmap v768: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:46.579 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:46 smithi203 bash[23654]: cluster 2024-03-11T21:35:45.431257+0000 mon.a (mon.0) 2852 : cluster 0 osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:35:46.580 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:46 smithi203 bash[23654]: audit 2024-03-11T21:35:45.442086+0000 mon.c (mon.2) 991 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.580 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:46 smithi203 bash[23654]: audit 2024-03-11T21:35:45.444228+0000 mon.a (mon.0) 2853 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.594 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[29284]: cluster 2024-03-11T21:35:45.426986+0000 mgr.y (mgr.14550) 506 : cluster 0 pgmap v768: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[29284]: cluster 2024-03-11T21:35:45.431257+0000 mon.a (mon.0) 2852 : cluster 0 osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[29284]: audit 2024-03-11T21:35:45.442086+0000 mon.c (mon.2) 991 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[29284]: audit 2024-03-11T21:35:45.444228+0000 mon.a (mon.0) 2853 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[21512]: cluster 2024-03-11T21:35:45.426986+0000 mgr.y (mgr.14550) 506 : cluster 0 pgmap v768: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[21512]: cluster 2024-03-11T21:35:45.431257+0000 mon.a (mon.0) 2852 : cluster 0 osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[21512]: audit 2024-03-11T21:35:45.442086+0000 mon.c (mon.2) 991 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:46 smithi143 bash[21512]: audit 2024-03-11T21:35:45.444228+0000 mon.a (mon.0) 2853 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:46.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.724 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:46.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:46.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:46.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.983 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:47.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:47.113 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:47.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.243 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:47.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:47.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:47.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:47.505 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:47.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.634 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:47.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: cluster 2024-03-11T21:35:46.421844+0000 mon.a (mon.0) 2854 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: audit 2024-03-11T21:35:46.423678+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: cluster 2024-03-11T21:35:46.435340+0000 mon.a (mon.0) 2856 : cluster 0 osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: audit 2024-03-11T21:35:46.437154+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: audit 2024-03-11T21:35:46.453910+0000 mon.c (mon.2) 993 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: audit 2024-03-11T21:35:46.478953+0000 mon.a (mon.0) 2857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: audit 2024-03-11T21:35:47.425767+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[29284]: cluster 2024-03-11T21:35:47.430589+0000 mon.a (mon.0) 2859 : cluster 0 osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: cluster 2024-03-11T21:35:46.421844+0000 mon.a (mon.0) 2854 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: audit 2024-03-11T21:35:46.423678+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:47.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: cluster 2024-03-11T21:35:46.435340+0000 mon.a (mon.0) 2856 : cluster 0 osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:35:47.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: audit 2024-03-11T21:35:46.437154+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:47.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: audit 2024-03-11T21:35:46.453910+0000 mon.c (mon.2) 993 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: audit 2024-03-11T21:35:46.478953+0000 mon.a (mon.0) 2857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: audit 2024-03-11T21:35:47.425767+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:47.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:47 smithi143 bash[21512]: cluster 2024-03-11T21:35:47.430589+0000 mon.a (mon.0) 2859 : cluster 0 osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:35:47.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: cluster 2024-03-11T21:35:46.421844+0000 mon.a (mon.0) 2854 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: audit 2024-03-11T21:35:46.423678+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: cluster 2024-03-11T21:35:46.435340+0000 mon.a (mon.0) 2856 : cluster 0 osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: audit 2024-03-11T21:35:46.437154+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: audit 2024-03-11T21:35:46.453910+0000 mon.c (mon.2) 993 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: audit 2024-03-11T21:35:46.478953+0000 mon.a (mon.0) 2857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: audit 2024-03-11T21:35:47.425767+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:35:47.749 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:47 smithi203 bash[23654]: cluster 2024-03-11T21:35:47.430589+0000 mon.a (mon.0) 2859 : cluster 0 osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:35:47.763 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:47.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:47.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:48.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:48.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:48.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.280 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:48.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.412 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:48.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:48.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.658 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:48 smithi143 bash[29284]: cluster 2024-03-11T21:35:47.427513+0000 mgr.y (mgr.14550) 507 : cluster 0 pgmap v771: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:48.658 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:48 smithi143 bash[21512]: cluster 2024-03-11T21:35:47.427513+0000 mgr.y (mgr.14550) 507 : cluster 0 pgmap v771: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:48.669 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:48 smithi203 bash[23654]: cluster 2024-03-11T21:35:47.427513+0000 mgr.y (mgr.14550) 507 : cluster 0 pgmap v771: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:35:48.674 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:48.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:48.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:48.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:48.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.934 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:49.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:49.063 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:49.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:49.195 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:49.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:49.319 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:49.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:49.448 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:49.551 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:49.563 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:49 smithi143 bash[29284]: cluster 2024-03-11T21:35:48.471794+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:35:49.563 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:49 smithi143 bash[29284]: audit 2024-03-11T21:35:48.697109+0000 mgr.y (mgr.14550) 508 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:49.563 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:49 smithi143 bash[21512]: cluster 2024-03-11T21:35:48.471794+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:35:49.563 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:49 smithi143 bash[21512]: audit 2024-03-11T21:35:48.697109+0000 mgr.y (mgr.14550) 508 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:49.579 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:49.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:49.709 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:49.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:49 smithi203 bash[23654]: cluster 2024-03-11T21:35:48.471794+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:35:49.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:49 smithi203 bash[23654]: audit 2024-03-11T21:35:48.697109+0000 mgr.y (mgr.14550) 508 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:49.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:49.840 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:49.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:49.973 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:50.100 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:50.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:50.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.476 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:50 smithi143 bash[21512]: cluster 2024-03-11T21:35:49.428801+0000 mgr.y (mgr.14550) 509 : cluster 0 pgmap v773: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-11T21:35:50.476 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:50 smithi143 bash[29284]: cluster 2024-03-11T21:35:49.428801+0000 mgr.y (mgr.14550) 509 : cluster 0 pgmap v773: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-11T21:35:50.492 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:50.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:50.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:50.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:50 smithi203 bash[23654]: cluster 2024-03-11T21:35:49.428801+0000 mgr.y (mgr.14550) 509 : cluster 0 pgmap v773: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-11T21:35:50.754 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:50.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:50.885 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:50.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:51.011 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:51.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:51.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:51.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.403 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:51.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:51.512 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[21512]: cluster 2024-03-11T21:35:50.680483+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:51.512 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[21512]: cluster 2024-03-11T21:35:50.680897+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:51.512 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[21512]: cluster 2024-03-11T21:35:51.377991+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:35:51.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[29284]: cluster 2024-03-11T21:35:50.680483+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:51.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[29284]: cluster 2024-03-11T21:35:50.680897+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:51.513 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:51 smithi143 bash[29284]: cluster 2024-03-11T21:35:51.377991+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:35:51.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:51.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:51.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:51 smithi203 bash[23654]: cluster 2024-03-11T21:35:50.680483+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:51.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:51 smithi203 bash[23654]: cluster 2024-03-11T21:35:50.680897+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:51.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:51 smithi203 bash[23654]: cluster 2024-03-11T21:35:51.377991+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:35:51.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:51.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:51.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:52.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:52.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.183 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:52.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.298 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:35:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:35:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:52.313 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:52.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:52.444 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:52.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:52.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:52.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:52 smithi143 bash[21512]: cluster 2024-03-11T21:35:51.429665+0000 mgr.y (mgr.14550) 510 : cluster 0 pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-11T21:35:52.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:52 smithi143 bash[21512]: audit 2024-03-11T21:35:51.575158+0000 mon.c (mon.2) 994 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:52.677 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:52 smithi143 bash[29284]: cluster 2024-03-11T21:35:51.429665+0000 mgr.y (mgr.14550) 510 : cluster 0 pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-11T21:35:52.677 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:52 smithi143 bash[29284]: audit 2024-03-11T21:35:51.575158+0000 mon.c (mon.2) 994 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:52.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:52 smithi203 bash[23654]: cluster 2024-03-11T21:35:51.429665+0000 mgr.y (mgr.14550) 510 : cluster 0 pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-11T21:35:52.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:52 smithi203 bash[23654]: audit 2024-03-11T21:35:51.575158+0000 mon.c (mon.2) 994 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:52.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.840 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:52.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:52.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:53.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.100 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:53.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:53.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:53.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.364 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:53.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:53.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:53.617 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:53.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.744 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:53.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:53.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.004 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:54.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:54.136 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:54.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:54.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:54.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:54.394 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:54.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:54.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.655 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[29284]: cluster 2024-03-11T21:35:53.430859+0000 mgr.y (mgr.14550) 511 : cluster 0 pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[29284]: cluster 2024-03-11T21:35:53.670617+0000 osd.0 (osd.0) 289 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[29284]: cluster 2024-03-11T21:35:53.670820+0000 osd.0 (osd.0) 290 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[21512]: cluster 2024-03-11T21:35:53.430859+0000 mgr.y (mgr.14550) 511 : cluster 0 pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[21512]: cluster 2024-03-11T21:35:53.670617+0000 osd.0 (osd.0) 289 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:54.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:54 smithi143 bash[21512]: cluster 2024-03-11T21:35:53.670820+0000 osd.0 (osd.0) 290 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:54.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:54.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:54 smithi203 bash[23654]: cluster 2024-03-11T21:35:53.430859+0000 mgr.y (mgr.14550) 511 : cluster 0 pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-11T21:35:54.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:54 smithi203 bash[23654]: cluster 2024-03-11T21:35:53.670617+0000 osd.0 (osd.0) 289 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:54.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:54 smithi203 bash[23654]: cluster 2024-03-11T21:35:53.670820+0000 osd.0 (osd.0) 290 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:54.780 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:54.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:54.911 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:55.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:55.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:55.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:55.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:55.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:55.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:55.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:55.432 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:55.535 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:55.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.677 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:55 smithi143 bash[29284]: cluster 2024-03-11T21:35:54.512051+0000 mon.a (mon.0) 2862 : cluster 0 osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:35:55.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:55 smithi143 bash[21512]: cluster 2024-03-11T21:35:54.512051+0000 mon.a (mon.0) 2862 : cluster 0 osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:35:55.692 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:55.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:55 smithi203 bash[23654]: cluster 2024-03-11T21:35:54.512051+0000 mon.a (mon.0) 2862 : cluster 0 osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:35:55.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:55.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:55.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:56.031 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:56.031 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:56.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.070 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:56.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:56.202 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:56.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:56.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:56.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:56.469 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:56.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[21512]: cluster 2024-03-11T21:35:55.431871+0000 mgr.y (mgr.14550) 512 : cluster 0 pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 735 B/s wr, 2 op/s 2024-03-11T21:35:56.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[21512]: cluster 2024-03-11T21:35:56.370678+0000 mon.a (mon.0) 2863 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:56.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[21512]: cluster 2024-03-11T21:35:56.379261+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:35:56.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[29284]: cluster 2024-03-11T21:35:55.431871+0000 mgr.y (mgr.14550) 512 : cluster 0 pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 735 B/s wr, 2 op/s 2024-03-11T21:35:56.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[29284]: cluster 2024-03-11T21:35:56.370678+0000 mon.a (mon.0) 2863 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:56.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:56 smithi143 bash[29284]: cluster 2024-03-11T21:35:56.379261+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:35:56.605 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:56.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.738 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:56 smithi203 bash[23654]: cluster 2024-03-11T21:35:55.431871+0000 mgr.y (mgr.14550) 512 : cluster 0 pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 735 B/s wr, 2 op/s 2024-03-11T21:35:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:56 smithi203 bash[23654]: cluster 2024-03-11T21:35:56.370678+0000 mon.a (mon.0) 2863 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:56.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:56 smithi203 bash[23654]: cluster 2024-03-11T21:35:56.379261+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:35:56.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:56.872 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:56.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:57.004 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:57.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.134 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:57.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.265 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:57.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:57.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:57.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:57.530 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 -- 172.21.15.143:0/2792595159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc100d40 msgr2=0x7fadcc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 --2- 172.21.15.143:0/2792595159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc100d40 0x7fadcc1011c0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009990 tx=0x7fadbc009ca0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 -- 172.21.15.143:0/2792595159 shutdown_connections 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 --2- 172.21.15.143:0/2792595159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc106dc0 0x7fadcc1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 --2- 172.21.15.143:0/2792595159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc101700 0x7fadcc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 --2- 172.21.15.143:0/2792595159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc100d40 0x7fadcc1011c0 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 -- 172.21.15.143:0/2792595159 >> 172.21.15.143:0/2792595159 conn(0x7fadcc076d80 msgr2=0x7fadcc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 -- 172.21.15.143:0/2792595159 shutdown_connections 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 -- 172.21.15.143:0/2792595159 wait complete. 2024-03-11T21:35:57.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.594+0000 7fadd32f7700 1 librados: starting msgr at 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 librados: starting objecter 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 Processor -- start 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- start start 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 librados: setting wanted keys 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 librados: calling monclient init 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc100d40 0x7fadcc19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc106dc0 0x7fadcc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc110560 con 0x7fadcc106dc0 2024-03-11T21:35:57.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1103e0 con 0x7fadcc100d40 2024-03-11T21:35:57.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc1106e0 con 0x7fadcc101700 2024-03-11T21:35:57.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd1894700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc106dc0 0x7fadcc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33188/0 (socket says 172.21.15.143:33188) 2024-03-11T21:35:57.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 -- 172.21.15.143:0/1261105033 learned_addr learned my addr 172.21.15.143:0/1261105033 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc100d40 0x7fadcc19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc100d40 msgr2=0x7fadcc19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc100d40 0x7fadcc19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc106dc0 msgr2=0x7fadcc1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc106dc0 0x7fadcc1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc198130 con 0x7fadcc101700 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadcc100d40 0x7fadcc19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fadbc000c00 tx=0x7fadbc0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 10 client.?.objecter ms_handle_connect 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc027070 con 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc1983c0 con 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fadbc022470 con 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc017650 con 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 10 client.41696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc1a46e0 con 0x7fadcc101700 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 librados: init done 2024-03-11T21:35:57.578 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fadcc102400 con 0x7fadcc101700 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fadbc02a070 con 0x7fadcc101700 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 0x7fadb8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(515..515 src has 87..515) v4 ==== 9629+0+0 (secure 0 0 0) 0x7fadbc08ff20 con 0x7fadcc101700 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 osd_map(515..515 src has 87..515) v4 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 3 client.41696.objecter handle_osd_map got epochs [515,515] > 0 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 3 client.41696.objecter handle_osd_map decoding full epoch 515 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 0x7fadb8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 20 client.41696.objecter dump_active .. 0 homeless 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.598+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v515) v1 ==== 54+0+16 (secure 0 0 0) 0x7fadbc031020 con 0x7fadcc101700 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 10 client.41696.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 1 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fadcc0629f0 0x7fadcc062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 1 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 0 1 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 4 2024-03-11T21:35:57.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 10 client.41696.objecter _send_command 1 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc04faa0 con 0x7fadcc0629f0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 0x7fadb8071ce0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fadcc198b10 tx=0x7fadc8009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadb806f820 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fadcc0629f0 0x7fadcc062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc0629f0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fadcc0629f0 0x7fadcc062dd0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc04faa0 con 0x7fadcc0629f0 2024-03-11T21:35:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:35:57.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 10 client.41696.objecter _finish_command 1 = system:0 2024-03-11T21:35:57.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 4 2024-03-11T21:35:57.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.602+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 0 1 2024-03-11T21:35:57.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 10 client.41696.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 2 2024-03-11T21:35:57.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 2 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc198f00 osd=0 3 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 0 2 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 4 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 10 client.41696.objecter _send_command 2 2024-03-11T21:35:57.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.618+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc062470 con 0x7fadcc0629f0 2024-03-11T21:35:57.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc062470 con 0x7fadcc0629f0 2024-03-11T21:35:57.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 10 client.41696.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 4 2024-03-11T21:35:57.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 0 2 2024-03-11T21:35:57.596 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:35:57.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 10 client.41696.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 3 2024-03-11T21:35:57.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fadcc10a7a0 0x7fadcc1a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 1 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 4 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 10 client.41696.objecter _send_command 3 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc02d9a0 con 0x7fadcc10a7a0 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fadcc10a7a0 0x7fadcc1a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc10a7a0 2024-03-11T21:35:57.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fadcc10a7a0 0x7fadcc1a88d0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc02d9a0 con 0x7fadcc10a7a0 2024-03-11T21:35:57.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:35:57.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 10 client.41696.objecter _finish_command 3 = system:0 2024-03-11T21:35:57.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 4 2024-03-11T21:35:57.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.622+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 1 3 2024-03-11T21:35:57.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 10 client.41696.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 4 2024-03-11T21:35:57.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 4 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc02d830 osd=1 3 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 1 4 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 4 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 10 client.41696.objecter _send_command 4 2024-03-11T21:35:57.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.638+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc1a8e10 con 0x7fadcc10a7a0 2024-03-11T21:35:57.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc1a8e10 con 0x7fadcc10a7a0 2024-03-11T21:35:57.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 10 client.41696.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 4 2024-03-11T21:35:57.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 1 4 2024-03-11T21:35:57.617 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:35:57.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 10 client.41696.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 5 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fadcc109e80 0x7fadcc1ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 5 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 2 5 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 4 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 10 client.41696.objecter _send_command 5 2024-03-11T21:35:57.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc02d420 con 0x7fadcc109e80 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fadcc109e80 0x7fadcc1ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc109e80 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fadcc109e80 0x7fadcc1ac1b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc02d420 con 0x7fadcc109e80 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 10 client.41696.objecter _finish_command 5 = system:0 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 4 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.642+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 2 5 2024-03-11T21:35:57.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 10 client.41696.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 6 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 6 2024-03-11T21:35:57.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc109d10 osd=2 3 2024-03-11T21:35:57.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 2 6 2024-03-11T21:35:57.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 4 2024-03-11T21:35:57.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 10 client.41696.objecter _send_command 6 2024-03-11T21:35:57.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc02c900 con 0x7fadcc109e80 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc02c900 con 0x7fadcc109e80 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 10 client.41696.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 4 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 2 6 2024-03-11T21:35:57.637 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:35:57.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 10 client.41696.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 7 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fadcc1a9ca0 0x7fadcc1af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 7 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 3 7 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 4 2024-03-11T21:35:57.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 10 client.41696.objecter _send_command 7 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc1b26e0 con 0x7fadcc1a9ca0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fadcc1a9ca0 0x7fadcc1af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc1a9ca0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.662+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fadcc1a9ca0 0x7fadcc1af880 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.666+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc1b26e0 con 0x7fadcc1a9ca0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.666+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.666+0000 7fadc27fc700 10 client.41696.objecter _finish_command 7 = system:0 2024-03-11T21:35:57.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.666+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 4 2024-03-11T21:35:57.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.666+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 3 7 2024-03-11T21:35:57.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:57.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 10 client.41696.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 8 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 8 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc02cb50 osd=3 3 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 3 8 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 4 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 10 client.41696.objecter _send_command 8 2024-03-11T21:35:57.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.682+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc1a92d0 con 0x7fadcc1a9ca0 2024-03-11T21:35:57.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc1a92d0 con 0x7fadcc1a9ca0 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 10 client.41696.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 4 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 3 8 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:35:57.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 10 client.41696.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 9 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fadcc1ad3d0 0x7fadcc103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 9 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 4 9 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 4 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 10 client.41696.objecter _send_command 9 2024-03-11T21:35:57.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc1b3570 con 0x7fadcc1ad3d0 2024-03-11T21:35:57.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fadcc1ad3d0 0x7fadcc103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc1ad3d0 2024-03-11T21:35:57.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.686+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fadcc1ad3d0 0x7fadcc103a70 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.690+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc1b3570 con 0x7fadcc1ad3d0 2024-03-11T21:35:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.690+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:35:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.690+0000 7fadc27fc700 10 client.41696.objecter _finish_command 9 = system:0 2024-03-11T21:35:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.690+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 4 2024-03-11T21:35:57.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.690+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 4 9 2024-03-11T21:35:57.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 10 client.41696.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 10 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 10 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1a92d0 osd=4 3 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 4 10 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 4 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 10 client.41696.objecter _send_command 10 2024-03-11T21:35:57.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.706+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc103fb0 con 0x7fadcc1ad3d0 2024-03-11T21:35:57.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc103fb0 con 0x7fadcc1ad3d0 2024-03-11T21:35:57.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 10 client.41696.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 4 2024-03-11T21:35:57.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 4 10 2024-03-11T21:35:57.685 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 10 client.41696.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 11 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fadcc1046d0 0x7fadcc1b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 11 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 5 11 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 4 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 10 client.41696.objecter _send_command 11 2024-03-11T21:35:57.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc1b3490 con 0x7fadcc1046d0 2024-03-11T21:35:57.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fadcc1046d0 0x7fadcc1b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadd0892700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fadcc1046d0 0x7fadcc1b1dd0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.710+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc1046d0 2024-03-11T21:35:57.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.714+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc1b3490 con 0x7fadcc1046d0 2024-03-11T21:35:57.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.714+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:35:57.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.714+0000 7fadc27fc700 10 client.41696.objecter _finish_command 11 = system:0 2024-03-11T21:35:57.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.714+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 4 2024-03-11T21:35:57.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.714+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 5 11 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 10 client.41696.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 12 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 12 2024-03-11T21:35:57.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc103fb0 osd=5 3 2024-03-11T21:35:57.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 5 12 2024-03-11T21:35:57.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 4 2024-03-11T21:35:57.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 10 client.41696.objecter _send_command 12 2024-03-11T21:35:57.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.730+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc1b2310 con 0x7fadcc1046d0 2024-03-11T21:35:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc1b2310 con 0x7fadcc1046d0 2024-03-11T21:35:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadc27fc700 10 client.41696.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 4 2024-03-11T21:35:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 5 12 2024-03-11T21:35:57.710 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:35:57.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 10 client.41696.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 13 2024-03-11T21:35:57.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fadcc1b2a30 0x7fadcc1b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 13 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 6 13 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 4 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 10 client.41696.objecter _send_command 13 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc1be5b0 con 0x7fadcc1b2a30 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.734+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fadcc1b2a30 0x7fadcc1b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc1b2a30 2024-03-11T21:35:57.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadd1894700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fadcc1b2a30 0x7fadcc1b2e70 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc1be5b0 con 0x7fadcc1b2a30 2024-03-11T21:35:57.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:35:57.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 10 client.41696.objecter _finish_command 13 = system:0 2024-03-11T21:35:57.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 4 2024-03-11T21:35:57.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.738+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 6 13 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 10 client.41696.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 14 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 14 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1b2310 osd=6 3 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 6 14 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 4 2024-03-11T21:35:57.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 10 client.41696.objecter _send_command 14 2024-03-11T21:35:57.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.754+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc1bbc90 con 0x7fadcc1b2a30 2024-03-11T21:35:57.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc1bbc90 con 0x7fadcc1b2a30 2024-03-11T21:35:57.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadc27fc700 10 client.41696.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 4 2024-03-11T21:35:57.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 6 14 2024-03-11T21:35:57.735 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 10 client.41696.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 15 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fadcc1bc310 0x7fadcc1be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 15 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 7 15 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 4 2024-03-11T21:35:57.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 10 client.41696.objecter _send_command 15 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.758+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadcc1c17a0 con 0x7fadcc1bc310 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fadcc1bc310 0x7fadcc1be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 10 client.41696.objecter ms_handle_connect 0x7fadcc1bc310 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadd1093700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fadcc1bc310 0x7fadcc1be720 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadcc1c17a0 con 0x7fadcc1bc310 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 10 client.41696.objecter _finish_command 15 = system:0 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 4 2024-03-11T21:35:57.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.762+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 7 15 2024-03-11T21:35:57.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 10 client.41696.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign -1 16 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 20 client.41696.objecter _get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_remove -1 16 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 20 client.41696.objecter get_session s=0x7fadcc1bbc90 osd=7 3 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 15 client.41696.objecter _session_command_op_assign 7 16 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 4 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 10 client.41696.objecter _send_command 16 2024-03-11T21:35:57.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.786+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadcc1c17a0 con 0x7fadcc1bc310 2024-03-11T21:35:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadc27fc700 1 -- 172.21.15.143:0/1261105033 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fadcc1c17a0 con 0x7fadcc1bc310 2024-03-11T21:35:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadc27fc700 10 client.41696.objecter ms_dispatch 0x7fadcc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:35:57.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadc27fc700 10 client.41696.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:35:57.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadc27fc700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 4 2024-03-11T21:35:57.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadc27fc700 15 client.41696.objecter _session_command_op_remove 7 16 2024-03-11T21:35:57.774 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:35:57.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 librados: watch_flush enter 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 librados: watch_flush exit 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.0 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fadcc0629f0 msgr2=0x7fadcc062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fadcc0629f0 0x7fadcc062dd0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc198f00 osd=0 1 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.1 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fadcc10a7a0 msgr2=0x7fadcc1a88d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fadcc10a7a0 0x7fadcc1a88d0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02d830 osd=1 1 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.2 2024-03-11T21:35:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fadcc109e80 msgr2=0x7fadcc1ac1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fadcc109e80 0x7fadcc1ac1b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc109d10 osd=2 1 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.3 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fadcc1a9ca0 msgr2=0x7fadcc1af880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fadcc1a9ca0 0x7fadcc1af880 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc02cb50 osd=3 1 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.4 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fadcc1ad3d0 msgr2=0x7fadcc103a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fadcc1ad3d0 0x7fadcc103a70 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd1894700 1 -- 172.21.15.143:0/1261105033 reap_dead start 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1a92d0 osd=4 1 2024-03-11T21:35:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.5 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fadcc1046d0 msgr2=0x7fadcc1b1dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fadcc1046d0 0x7fadcc1b1dd0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd1894700 1 -- 172.21.15.143:0/1261105033 reap_dead start 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc103fb0 osd=5 1 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.6 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fadcc1b2a30 msgr2=0x7fadcc1b2e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fadcc1b2a30 0x7fadcc1b2e70 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1b2310 osd=6 1 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter close_session for osd.7 2024-03-11T21:35:57.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fadcc1bc310 msgr2=0x7fadcc1be720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fadcc1bc310 0x7fadcc1be720 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter put_session s=0x7fadcc1bbc90 osd=7 1 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd1894700 1 -- 172.21.15.143:0/1261105033 reap_dead start 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 20 client.41696.objecter shutdown clearing up homeless session... 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 10 client.41696.objecter successfully canceled tick 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 msgr2=0x7fadb8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 0x7fadb8071ce0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fadcc198b10 tx=0x7fadc8009260 comp rx=0 tx=0).stop 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 msgr2=0x7fadcc19dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fadbc000c00 tx=0x7fadbc0057c0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 shutdown_connections 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc101700 0x7fadcc19dd50 secure :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fadbc000c00 tx=0x7fadbc0057c0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 --2- 172.21.15.143:0/1261105033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fadb806f820 0x7fadb8071ce0 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.798+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 >> 172.21.15.143:0/1261105033 conn(0x7fadcc076d80 msgr2=0x7fadcc0ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.802+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 shutdown_connections 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.802+0000 7fadd32f7700 1 -- 172.21.15.143:0/1261105033 wait complete. 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:35:57.802+0000 7fadd32f7700 1 librados: shutdown 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:57.888 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:57.913 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:58.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:58.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:58.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.175 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:58.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:58.305 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:58.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:58.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:58.566 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:58.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.679 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[23654]: cluster 2024-03-11T21:35:57.432477+0000 mgr.y (mgr.14550) 513 : cluster 0 pgmap v780: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 676 B/s wr, 1 op/s 2024-03-11T21:35:58.680 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[23654]: cluster 2024-03-11T21:35:57.494403+0000 osd.7 (osd.7) 296 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:58.680 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[23654]: cluster 2024-03-11T21:35:57.494732+0000 osd.7 (osd.7) 297 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:58.680 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[23654]: cluster 2024-03-11T21:35:57.531140+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:35:58.680 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[23654]: audit 2024-03-11T21:35:57.602551+0000 mon.c (mon.2) 995 : audit 0 from='client.? 172.21.15.143:0/1261105033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:58.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[29284]: cluster 2024-03-11T21:35:57.432477+0000 mgr.y (mgr.14550) 513 : cluster 0 pgmap v780: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 676 B/s wr, 1 op/s 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[29284]: cluster 2024-03-11T21:35:57.494403+0000 osd.7 (osd.7) 296 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[29284]: cluster 2024-03-11T21:35:57.494732+0000 osd.7 (osd.7) 297 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[29284]: cluster 2024-03-11T21:35:57.531140+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[29284]: audit 2024-03-11T21:35:57.602551+0000 mon.c (mon.2) 995 : audit 0 from='client.? 172.21.15.143:0/1261105033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[21512]: cluster 2024-03-11T21:35:57.432477+0000 mgr.y (mgr.14550) 513 : cluster 0 pgmap v780: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 676 B/s wr, 1 op/s 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[21512]: cluster 2024-03-11T21:35:57.494403+0000 osd.7 (osd.7) 296 : cluster 0 purged_snaps scrub starts 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[21512]: cluster 2024-03-11T21:35:57.494732+0000 osd.7 (osd.7) 297 : cluster 0 purged_snaps scrub ok 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[21512]: cluster 2024-03-11T21:35:57.531140+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:35:58.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:58 smithi143 bash[21512]: audit 2024-03-11T21:35:57.602551+0000 mon.c (mon.2) 995 : audit 0 from='client.? 172.21.15.143:0/1261105033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:58.696 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:58.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:35:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:35:58.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:58.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:58.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.955 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:59.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:59.084 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:59.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:59.215 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:59.318 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:59.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:59.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:35:59.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:59.607 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:59.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.719 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:35:59 smithi143 bash[29284]: audit 2024-03-11T21:35:58.707473+0000 mgr.y (mgr.14550) 514 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:59.719 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:35:59 smithi143 bash[21512]: audit 2024-03-11T21:35:58.707473+0000 mgr.y (mgr.14550) 514 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:59.730 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:59.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:35:59 smithi203 bash[23654]: audit 2024-03-11T21:35:58.707473+0000 mgr.y (mgr.14550) 514 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:59.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:59.860 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:59.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:59.991 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:00.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:00.124 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:00.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:00.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:00.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:00.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:00.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:00.631 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:00 smithi143 bash[21512]: cluster 2024-03-11T21:35:59.433847+0000 mgr.y (mgr.14550) 515 : cluster 0 pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:00.631 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:00 smithi143 bash[29284]: cluster 2024-03-11T21:35:59.433847+0000 mgr.y (mgr.14550) 515 : cluster 0 pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:00.646 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3723380085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 msgr2=0x7f7184102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- 172.21.15.143:0/3723380085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f7184102310 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f7178009920 tx=0x7f7178009c30 comp rx=0 tx=0).stop 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3723380085 shutdown_connections 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- 172.21.15.143:0/3723380085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184102850 0x7f718410ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- 172.21.15.143:0/3723380085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184107ea0 0x7f7184108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- 172.21.15.143:0/3723380085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f7184102310 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:00.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3723380085 >> 172.21.15.143:0/3723380085 conn(0x7f7184078f40 msgr2=0x7f7184079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:00.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3723380085 shutdown_connections 2024-03-11T21:36:00.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3723380085 wait complete. 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 librados: starting msgr at 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 librados: starting objecter 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 Processor -- start 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- start start 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 librados: setting wanted keys 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 librados: calling monclient init 2024-03-11T21:36:00.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 0x7f7184074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184107ea0 0x7f7184074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7184111620 con 0x7f7184101e90 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71841114a0 con 0x7f7184107ea0 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71841117a0 con 0x7f7184102850 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:37896/0 (socket says 172.21.15.143:37896) 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 -- 172.21.15.143:0/3107227112 learned_addr learned my addr 172.21.15.143:0/3107227112 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7181d9b700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 0x7f7184074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7182d9d700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184107ea0 0x7f7184074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 -- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 msgr2=0x7f7184074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 0x7f7184074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 -- 172.21.15.143:0/3107227112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184107ea0 msgr2=0x7f7184074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:00.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184107ea0 0x7f7184074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 -- 172.21.15.143:0/3107227112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71841a6200 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7181d9b700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 0x7f7184074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f718259c700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f717c00e3c0 tx=0x7f717c00e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f71737fe700 10 client.?.objecter ms_handle_connect 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f71737fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0090d0 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f717c009270 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71841a6490 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c004530 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 10 client.30501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71841a69a0 con 0x7f7184101e90 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 librados: init done 2024-03-11T21:36:00.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.718+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7184103550 con 0x7f7184101e90 2024-03-11T21:36:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f717c01b030 con 0x7f7184101e90 2024-03-11T21:36:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 0x7f716c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(515..515 src has 87..515) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f717c08e550 con 0x7f7184101e90 2024-03-11T21:36:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 10 client.30501.objecter ms_dispatch 0x7f7184001550 osd_map(515..515 src has 87..515) v4 2024-03-11T21:36:00.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 3 client.30501.objecter handle_osd_map got epochs [515,515] > 0 2024-03-11T21:36:00.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 3 client.30501.objecter handle_osd_map decoding full epoch 515 2024-03-11T21:36:00.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f7181d9b700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 0x7f716c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:00.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 20 client.30501.objecter dump_active .. 0 homeless 2024-03-11T21:36:00.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f71737fe700 10 client.30501.objecter ms_handle_connect 0x7f716c06f820 2024-03-11T21:36:00.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.722+0000 7f7181d9b700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 0x7f716c071ce0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f71780098f0 tx=0x7f7178019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.726+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f717c010040 con 0x7f7184101e90 2024-03-11T21:36:00.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.761 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:00 smithi203 bash[23654]: cluster 2024-03-11T21:35:59.433847+0000 mgr.y (mgr.14550) 515 : cluster 0 pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:00.763 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:36:00.763 INFO:teuthology.orchestra.run.smithi143.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:36:00.763 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:36:00.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:00.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:00.909 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:00.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:00.958+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f7184075610 con 0x7f7184101e90 2024-03-11T21:36:01.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:01.038 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:01.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:01.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:01.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.374+0000 7f71737fe700 1 -- 172.21.15.143:0/3107227112 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v516) v1 ==== 91+0+0 (secure 0 0 0) 0x7f717c05b220 con 0x7f7184101e90 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.374+0000 7f7188b24700 10 librados: watch_flush enter 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.374+0000 7f7188b24700 10 librados: watch_flush exit 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.374+0000 7f7188b24700 20 client.30501.objecter shutdown clearing up homeless session... 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 10 client.30501.objecter successfully canceled tick 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 msgr2=0x7f716c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 0x7f716c071ce0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f71780098f0 tx=0x7f7178019040 comp rx=0 tx=0).stop 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 msgr2=0x7f71840740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f717c00e3c0 tx=0x7f717c00e6d0 comp rx=0 tx=0).stop 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 shutdown_connections 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f716c06f820 0x7f716c071ce0 secure :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f71780098f0 tx=0x7f7178019040 comp rx=0 tx=0).stop 2024-03-11T21:36:01.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7184101e90 0x7f71840740e0 secure :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f717c00e3c0 tx=0x7f717c00e6d0 comp rx=0 tx=0).stop 2024-03-11T21:36:01.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7184107ea0 0x7f7184074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:01.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 --2- 172.21.15.143:0/3107227112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7184102850 0x7f7184074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:01.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.378+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 >> 172.21.15.143:0/3107227112 conn(0x7f7184078f40 msgr2=0x7f7184108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:01.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.382+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 shutdown_connections 2024-03-11T21:36:01.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.382+0000 7f7188b24700 1 -- 172.21.15.143:0/3107227112 wait complete. 2024-03-11T21:36:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:01.382+0000 7f7188b24700 1 librados: shutdown 2024-03-11T21:36:01.359 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:36:01.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.431 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:01.535 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:01.560 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:01.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:01.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:01 smithi203 bash[23654]: audit 2024-03-11T21:36:00.961123+0000 mon.a (mon.0) 2866 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:01.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:01 smithi203 bash[23654]: cluster 2024-03-11T21:36:01.373449+0000 mon.a (mon.0) 2867 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:01.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:01 smithi203 bash[23654]: audit 2024-03-11T21:36:01.375862+0000 mon.a (mon.0) 2868 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:01.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:01 smithi203 bash[23654]: cluster 2024-03-11T21:36:01.380913+0000 mon.a (mon.0) 2869 : cluster 0 osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:36:01.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[29284]: audit 2024-03-11T21:36:00.961123+0000 mon.a (mon.0) 2866 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:01.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[29284]: cluster 2024-03-11T21:36:01.373449+0000 mon.a (mon.0) 2867 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:01.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[29284]: audit 2024-03-11T21:36:01.375862+0000 mon.a (mon.0) 2868 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:01.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[29284]: cluster 2024-03-11T21:36:01.380913+0000 mon.a (mon.0) 2869 : cluster 0 osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:36:01.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[21512]: audit 2024-03-11T21:36:00.961123+0000 mon.a (mon.0) 2866 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:01.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[21512]: cluster 2024-03-11T21:36:01.373449+0000 mon.a (mon.0) 2867 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:01.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[21512]: audit 2024-03-11T21:36:01.375862+0000 mon.a (mon.0) 2868 : audit 1 from='client.? 172.21.15.143:0/3107227112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:01.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:01 smithi143 bash[21512]: cluster 2024-03-11T21:36:01.380913+0000 mon.a (mon.0) 2869 : cluster 0 osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:36:01.691 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:01.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:01.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:01.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:01.950 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:02.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.081 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:02.183 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:02.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.212 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:02.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:02.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:02.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:02.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:36:02.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.585 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:02 smithi203 bash[23654]: cluster 2024-03-11T21:36:01.434371+0000 mgr.y (mgr.14550) 516 : cluster 0 pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:02.585 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:02 smithi203 bash[23654]: cluster 2024-03-11T21:36:01.807581+0000 osd.2 (osd.2) 251 : cluster 0 6.f scrub starts 2024-03-11T21:36:02.585 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:02 smithi203 bash[23654]: cluster 2024-03-11T21:36:01.810160+0000 osd.2 (osd.2) 252 : cluster 0 6.f scrub ok 2024-03-11T21:36:02.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:02.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.728 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:02.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:02.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[21512]: cluster 2024-03-11T21:36:01.434371+0000 mgr.y (mgr.14550) 516 : cluster 0 pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:02.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[21512]: cluster 2024-03-11T21:36:01.807581+0000 osd.2 (osd.2) 251 : cluster 0 6.f scrub starts 2024-03-11T21:36:02.842 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[21512]: cluster 2024-03-11T21:36:01.810160+0000 osd.2 (osd.2) 252 : cluster 0 6.f scrub ok 2024-03-11T21:36:02.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[29284]: cluster 2024-03-11T21:36:01.434371+0000 mgr.y (mgr.14550) 516 : cluster 0 pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:02.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[29284]: cluster 2024-03-11T21:36:01.807581+0000 osd.2 (osd.2) 251 : cluster 0 6.f scrub starts 2024-03-11T21:36:02.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:02 smithi143 bash[29284]: cluster 2024-03-11T21:36:01.810160+0000 osd.2 (osd.2) 252 : cluster 0 6.f scrub ok 2024-03-11T21:36:02.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:02.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:02.981 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:03.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:03.114 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:03.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:03.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:03.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.376 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:03.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:03.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:03.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:03.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[29284]: cluster 2024-03-11T21:36:02.263116+0000 osd.4 (osd.4) 263 : cluster 0 118.2 scrub starts 2024-03-11T21:36:03.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[29284]: cluster 2024-03-11T21:36:02.266504+0000 osd.4 (osd.4) 264 : cluster 0 118.2 scrub ok 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[29284]: cluster 2024-03-11T21:36:02.739152+0000 osd.5 (osd.5) 305 : cluster 0 2.2 scrub starts 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[29284]: cluster 2024-03-11T21:36:02.740735+0000 osd.5 (osd.5) 306 : cluster 0 2.2 scrub ok 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[21512]: cluster 2024-03-11T21:36:02.263116+0000 osd.4 (osd.4) 263 : cluster 0 118.2 scrub starts 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[21512]: cluster 2024-03-11T21:36:02.266504+0000 osd.4 (osd.4) 264 : cluster 0 118.2 scrub ok 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[21512]: cluster 2024-03-11T21:36:02.739152+0000 osd.5 (osd.5) 305 : cluster 0 2.2 scrub starts 2024-03-11T21:36:03.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:03 smithi143 bash[21512]: cluster 2024-03-11T21:36:02.740735+0000 osd.5 (osd.5) 306 : cluster 0 2.2 scrub ok 2024-03-11T21:36:03.640 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:03.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:03 smithi203 bash[23654]: cluster 2024-03-11T21:36:02.263116+0000 osd.4 (osd.4) 263 : cluster 0 118.2 scrub starts 2024-03-11T21:36:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:03 smithi203 bash[23654]: cluster 2024-03-11T21:36:02.266504+0000 osd.4 (osd.4) 264 : cluster 0 118.2 scrub ok 2024-03-11T21:36:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:03 smithi203 bash[23654]: cluster 2024-03-11T21:36:02.739152+0000 osd.5 (osd.5) 305 : cluster 0 2.2 scrub starts 2024-03-11T21:36:03.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:03 smithi203 bash[23654]: cluster 2024-03-11T21:36:02.740735+0000 osd.5 (osd.5) 306 : cluster 0 2.2 scrub ok 2024-03-11T21:36:03.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:03.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:03.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:04.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:04.036 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:04.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:04.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:04.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:04.427 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:04.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:36:04.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:04.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:04.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.678 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:04.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.217374+0000 osd.0 (osd.0) 291 : cluster 0 3.1 scrub starts 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.221870+0000 osd.0 (osd.0) 292 : cluster 0 3.1 scrub ok 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.282764+0000 osd.4 (osd.4) 265 : cluster 0 39.3 scrub starts 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.284526+0000 osd.4 (osd.4) 266 : cluster 0 39.3 scrub ok 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.382486+0000 osd.6 (osd.6) 229 : cluster 0 3.10 scrub starts 2024-03-11T21:36:04.792 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.386387+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub ok 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[29284]: cluster 2024-03-11T21:36:03.435597+0000 mgr.y (mgr.14550) 517 : cluster 0 pgmap v785: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 435 B/s wr, 3 op/s 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.217374+0000 osd.0 (osd.0) 291 : cluster 0 3.1 scrub starts 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.221870+0000 osd.0 (osd.0) 292 : cluster 0 3.1 scrub ok 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.282764+0000 osd.4 (osd.4) 265 : cluster 0 39.3 scrub starts 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.284526+0000 osd.4 (osd.4) 266 : cluster 0 39.3 scrub ok 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.382486+0000 osd.6 (osd.6) 229 : cluster 0 3.10 scrub starts 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.386387+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub ok 2024-03-11T21:36:04.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:04 smithi143 bash[21512]: cluster 2024-03-11T21:36:03.435597+0000 mgr.y (mgr.14550) 517 : cluster 0 pgmap v785: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 435 B/s wr, 3 op/s 2024-03-11T21:36:04.808 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:04.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:04.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:04.949 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.217374+0000 osd.0 (osd.0) 291 : cluster 0 3.1 scrub starts 2024-03-11T21:36:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.221870+0000 osd.0 (osd.0) 292 : cluster 0 3.1 scrub ok 2024-03-11T21:36:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.282764+0000 osd.4 (osd.4) 265 : cluster 0 39.3 scrub starts 2024-03-11T21:36:05.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.284526+0000 osd.4 (osd.4) 266 : cluster 0 39.3 scrub ok 2024-03-11T21:36:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.382486+0000 osd.6 (osd.6) 229 : cluster 0 3.10 scrub starts 2024-03-11T21:36:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.386387+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub ok 2024-03-11T21:36:05.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:04 smithi203 bash[23654]: cluster 2024-03-11T21:36:03.435597+0000 mgr.y (mgr.14550) 517 : cluster 0 pgmap v785: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 435 B/s wr, 3 op/s 2024-03-11T21:36:05.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:05.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:05.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:05.193 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:05.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.453 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:05.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:05.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.696 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.111903+0000 osd.1 (osd.1) 95 : cluster 0 3.0 scrub starts 2024-03-11T21:36:05.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.116706+0000 osd.1 (osd.1) 96 : cluster 0 3.0 scrub ok 2024-03-11T21:36:05.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.293678+0000 osd.4 (osd.4) 267 : cluster 0 118.f scrub starts 2024-03-11T21:36:05.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.302363+0000 osd.4 (osd.4) 268 : cluster 0 118.f scrub ok 2024-03-11T21:36:05.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.659829+0000 osd.3 (osd.3) 297 : cluster 0 39.e scrub starts 2024-03-11T21:36:05.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:05 smithi203 bash[23654]: cluster 2024-03-11T21:36:04.661491+0000 osd.3 (osd.3) 298 : cluster 0 39.e scrub ok 2024-03-11T21:36:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.843 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.111903+0000 osd.1 (osd.1) 95 : cluster 0 3.0 scrub starts 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.116706+0000 osd.1 (osd.1) 96 : cluster 0 3.0 scrub ok 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.293678+0000 osd.4 (osd.4) 267 : cluster 0 118.f scrub starts 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.302363+0000 osd.4 (osd.4) 268 : cluster 0 118.f scrub ok 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.659829+0000 osd.3 (osd.3) 297 : cluster 0 39.e scrub starts 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[29284]: cluster 2024-03-11T21:36:04.661491+0000 osd.3 (osd.3) 298 : cluster 0 39.e scrub ok 2024-03-11T21:36:05.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.111903+0000 osd.1 (osd.1) 95 : cluster 0 3.0 scrub starts 2024-03-11T21:36:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.116706+0000 osd.1 (osd.1) 96 : cluster 0 3.0 scrub ok 2024-03-11T21:36:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.293678+0000 osd.4 (osd.4) 267 : cluster 0 118.f scrub starts 2024-03-11T21:36:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.302363+0000 osd.4 (osd.4) 268 : cluster 0 118.f scrub ok 2024-03-11T21:36:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.659829+0000 osd.3 (osd.3) 297 : cluster 0 39.e scrub starts 2024-03-11T21:36:05.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:05 smithi143 bash[21512]: cluster 2024-03-11T21:36:04.661491+0000 osd.3 (osd.3) 298 : cluster 0 39.e scrub ok 2024-03-11T21:36:05.945 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:05.972 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:06.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:06.103 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:06.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:06.232 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:06.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:06.358 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:06.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:06.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:06.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:06.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[29284]: cluster 2024-03-11T21:36:05.134123+0000 osd.1 (osd.1) 97 : cluster 0 2.9 scrub starts 2024-03-11T21:36:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[29284]: cluster 2024-03-11T21:36:05.135745+0000 osd.1 (osd.1) 98 : cluster 0 2.9 scrub ok 2024-03-11T21:36:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[29284]: cluster 2024-03-11T21:36:05.436760+0000 mgr.y (mgr.14550) 518 : cluster 0 pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-11T21:36:06.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[21512]: cluster 2024-03-11T21:36:05.134123+0000 osd.1 (osd.1) 97 : cluster 0 2.9 scrub starts 2024-03-11T21:36:06.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[21512]: cluster 2024-03-11T21:36:05.135745+0000 osd.1 (osd.1) 98 : cluster 0 2.9 scrub ok 2024-03-11T21:36:06.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:06 smithi143 bash[21512]: cluster 2024-03-11T21:36:05.436760+0000 mgr.y (mgr.14550) 518 : cluster 0 pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-11T21:36:06.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:06.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:06.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:06 smithi203 bash[23654]: cluster 2024-03-11T21:36:05.134123+0000 osd.1 (osd.1) 97 : cluster 0 2.9 scrub starts 2024-03-11T21:36:06.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:06 smithi203 bash[23654]: cluster 2024-03-11T21:36:05.135745+0000 osd.1 (osd.1) 98 : cluster 0 2.9 scrub ok 2024-03-11T21:36:06.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:06 smithi203 bash[23654]: cluster 2024-03-11T21:36:05.436760+0000 mgr.y (mgr.14550) 518 : cluster 0 pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-11T21:36:06.870 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:06.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:36:06.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:36:06.915 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:36:06.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:36:06.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:06.991 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:07.002 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:07.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:07.124 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:07.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:07.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:07.380 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:07.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:07.511 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:07.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:07.639 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:07.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.770 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:07 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.263638+0000 osd.4 (osd.4) 269 : cluster 0 118.a scrub starts 2024-03-11T21:36:07.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:07 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.267040+0000 osd.4 (osd.4) 270 : cluster 0 118.a scrub ok 2024-03-11T21:36:07.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:07 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.396404+0000 osd.6 (osd.6) 231 : cluster 0 2.14 scrub starts 2024-03-11T21:36:07.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:07 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.398140+0000 osd.6 (osd.6) 232 : cluster 0 2.14 scrub ok 2024-03-11T21:36:07.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:07 smithi203 bash[23654]: audit 2024-03-11T21:36:06.590640+0000 mon.c (mon.2) 996 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:07.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:07.900 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:07.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.263638+0000 osd.4 (osd.4) 269 : cluster 0 118.a scrub starts 2024-03-11T21:36:07.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.267040+0000 osd.4 (osd.4) 270 : cluster 0 118.a scrub ok 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.396404+0000 osd.6 (osd.6) 231 : cluster 0 2.14 scrub starts 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.398140+0000 osd.6 (osd.6) 232 : cluster 0 2.14 scrub ok 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[29284]: audit 2024-03-11T21:36:06.590640+0000 mon.c (mon.2) 996 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.263638+0000 osd.4 (osd.4) 269 : cluster 0 118.a scrub starts 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.267040+0000 osd.4 (osd.4) 270 : cluster 0 118.a scrub ok 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.396404+0000 osd.6 (osd.6) 231 : cluster 0 2.14 scrub starts 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.398140+0000 osd.6 (osd.6) 232 : cluster 0 2.14 scrub ok 2024-03-11T21:36:07.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:07 smithi143 bash[21512]: audit 2024-03-11T21:36:06.590640+0000 mon.c (mon.2) 996 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:08.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:08.030 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:08.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:08.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:08.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:08.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.418 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:08.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:08.548 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:08.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.931523+0000 osd.2 (osd.2) 253 : cluster 0 118.17 scrub starts 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:06.936611+0000 osd.2 (osd.2) 254 : cluster 0 118.17 scrub ok 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.264631+0000 osd.4 (osd.4) 271 : cluster 0 3.9 scrub starts 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.268551+0000 osd.4 (osd.4) 272 : cluster 0 3.9 scrub ok 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.437525+0000 mgr.y (mgr.14550) 519 : cluster 0 pgmap v787: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 310 B/s wr, 2 op/s 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.695760+0000 osd.3 (osd.3) 299 : cluster 0 6.c scrub starts 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.698053+0000 osd.3 (osd.3) 300 : cluster 0 6.c scrub ok 2024-03-11T21:36:08.663 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: audit 2024-03-11T21:36:08.577032+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: audit 2024-03-11T21:36:08.577528+0000 mon.a (mon.0) 2870 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: audit 2024-03-11T21:36:08.578295+0000 mon.c (mon.2) 998 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.664 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[23654]: audit 2024-03-11T21:36:08.578582+0000 mon.a (mon.0) 2871 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.681 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:08.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.811 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:08.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:08.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.931523+0000 osd.2 (osd.2) 253 : cluster 0 118.17 scrub starts 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:06.936611+0000 osd.2 (osd.2) 254 : cluster 0 118.17 scrub ok 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.264631+0000 osd.4 (osd.4) 271 : cluster 0 3.9 scrub starts 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.268551+0000 osd.4 (osd.4) 272 : cluster 0 3.9 scrub ok 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.437525+0000 mgr.y (mgr.14550) 519 : cluster 0 pgmap v787: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 310 B/s wr, 2 op/s 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.695760+0000 osd.3 (osd.3) 299 : cluster 0 6.c scrub starts 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.698053+0000 osd.3 (osd.3) 300 : cluster 0 6.c scrub ok 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: audit 2024-03-11T21:36:08.577032+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: audit 2024-03-11T21:36:08.577528+0000 mon.a (mon.0) 2870 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: audit 2024-03-11T21:36:08.578295+0000 mon.c (mon.2) 998 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[29284]: audit 2024-03-11T21:36:08.578582+0000 mon.a (mon.0) 2871 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.931523+0000 osd.2 (osd.2) 253 : cluster 0 118.17 scrub starts 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:06.936611+0000 osd.2 (osd.2) 254 : cluster 0 118.17 scrub ok 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.264631+0000 osd.4 (osd.4) 271 : cluster 0 3.9 scrub starts 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.268551+0000 osd.4 (osd.4) 272 : cluster 0 3.9 scrub ok 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.437525+0000 mgr.y (mgr.14550) 519 : cluster 0 pgmap v787: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 310 B/s wr, 2 op/s 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.695760+0000 osd.3 (osd.3) 299 : cluster 0 6.c scrub starts 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.698053+0000 osd.3 (osd.3) 300 : cluster 0 6.c scrub ok 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: audit 2024-03-11T21:36:08.577032+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: audit 2024-03-11T21:36:08.577528+0000 mon.a (mon.0) 2870 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: audit 2024-03-11T21:36:08.578295+0000 mon.c (mon.2) 998 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:08 smithi143 bash[21512]: audit 2024-03-11T21:36:08.578582+0000 mon.a (mon.0) 2871 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-71"}]: dispatch 2024-03-11T21:36:08.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:09.036 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:09.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:09.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:09.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.204 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:09.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.333 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:09.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:09.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.593 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:09.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.708 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.901892+0000 osd.2 (osd.2) 255 : cluster 0 5.d scrub starts 2024-03-11T21:36:09.708 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:07.904860+0000 osd.2 (osd.2) 256 : cluster 0 5.d scrub ok 2024-03-11T21:36:09.708 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.291537+0000 osd.0 (osd.0) 293 : cluster 0 118.19 scrub starts 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.294284+0000 osd.0 (osd.0) 294 : cluster 0 118.19 scrub ok 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.641800+0000 osd.5 (osd.5) 307 : cluster 0 9.7 scrub starts 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.644679+0000 osd.5 (osd.5) 308 : cluster 0 9.7 scrub ok 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.681967+0000 osd.3 (osd.3) 301 : cluster 0 9.c scrub starts 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.684097+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub ok 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: audit 2024-03-11T21:36:08.715776+0000 mgr.y (mgr.14550) 520 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.041224+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.043128+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:36:09.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.901892+0000 osd.2 (osd.2) 255 : cluster 0 5.d scrub starts 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:07.904860+0000 osd.2 (osd.2) 256 : cluster 0 5.d scrub ok 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.291537+0000 osd.0 (osd.0) 293 : cluster 0 118.19 scrub starts 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.294284+0000 osd.0 (osd.0) 294 : cluster 0 118.19 scrub ok 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.641800+0000 osd.5 (osd.5) 307 : cluster 0 9.7 scrub starts 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.644679+0000 osd.5 (osd.5) 308 : cluster 0 9.7 scrub ok 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.681967+0000 osd.3 (osd.3) 301 : cluster 0 9.c scrub starts 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.684097+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub ok 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: audit 2024-03-11T21:36:08.715776+0000 mgr.y (mgr.14550) 520 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.041224+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:36:09.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:09 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.043128+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:36:09.719 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:09.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:09.834 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.901892+0000 osd.2 (osd.2) 255 : cluster 0 5.d scrub starts 2024-03-11T21:36:09.834 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:07.904860+0000 osd.2 (osd.2) 256 : cluster 0 5.d scrub ok 2024-03-11T21:36:09.834 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.291537+0000 osd.0 (osd.0) 293 : cluster 0 118.19 scrub starts 2024-03-11T21:36:09.834 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.294284+0000 osd.0 (osd.0) 294 : cluster 0 118.19 scrub ok 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.641800+0000 osd.5 (osd.5) 307 : cluster 0 9.7 scrub starts 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.644679+0000 osd.5 (osd.5) 308 : cluster 0 9.7 scrub ok 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.681967+0000 osd.3 (osd.3) 301 : cluster 0 9.c scrub starts 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.684097+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub ok 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: audit 2024-03-11T21:36:08.715776+0000 mgr.y (mgr.14550) 520 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.041224+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-03-11T21:36:09.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:09 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.043128+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-03-11T21:36:09.850 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:09.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.983 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:10.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:10.115 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:10.218 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:10.244 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:10.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:10.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:10.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:10.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.866888+0000 osd.2 (osd.2) 257 : cluster 0 39.1f scrub starts 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:08.869663+0000 osd.2 (osd.2) 258 : cluster 0 39.1f scrub ok 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.440062+0000 mgr.y (mgr.14550) 521 : cluster 0 pgmap v788: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.602242+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.646023+0000 osd.5 (osd.5) 309 : cluster 0 5.1b scrub starts 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.648309+0000 osd.5 (osd.5) 310 : cluster 0 5.1b scrub ok 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.675597+0000 osd.3 (osd.3) 303 : cluster 0 39.18 scrub starts 2024-03-11T21:36:10.578 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[21512]: cluster 2024-03-11T21:36:09.680342+0000 osd.3 (osd.3) 304 : cluster 0 39.18 scrub ok 2024-03-11T21:36:10.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:10.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:10.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:10.759 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.866888+0000 osd.2 (osd.2) 257 : cluster 0 39.1f scrub starts 2024-03-11T21:36:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:08.869663+0000 osd.2 (osd.2) 258 : cluster 0 39.1f scrub ok 2024-03-11T21:36:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.440062+0000 mgr.y (mgr.14550) 521 : cluster 0 pgmap v788: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-11T21:36:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.602242+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:36:10.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.646023+0000 osd.5 (osd.5) 309 : cluster 0 5.1b scrub starts 2024-03-11T21:36:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.648309+0000 osd.5 (osd.5) 310 : cluster 0 5.1b scrub ok 2024-03-11T21:36:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.675597+0000 osd.3 (osd.3) 303 : cluster 0 39.18 scrub starts 2024-03-11T21:36:10.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:10 smithi203 bash[23654]: cluster 2024-03-11T21:36:09.680342+0000 osd.3 (osd.3) 304 : cluster 0 39.18 scrub ok 2024-03-11T21:36:10.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:10.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.866888+0000 osd.2 (osd.2) 257 : cluster 0 39.1f scrub starts 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:08.869663+0000 osd.2 (osd.2) 258 : cluster 0 39.1f scrub ok 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.440062+0000 mgr.y (mgr.14550) 521 : cluster 0 pgmap v788: 260 pgs: 260 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.602242+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.646023+0000 osd.5 (osd.5) 309 : cluster 0 5.1b scrub starts 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.648309+0000 osd.5 (osd.5) 310 : cluster 0 5.1b scrub ok 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.675597+0000 osd.3 (osd.3) 303 : cluster 0 39.18 scrub starts 2024-03-11T21:36:10.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:10 smithi143 bash[29284]: cluster 2024-03-11T21:36:09.680342+0000 osd.3 (osd.3) 304 : cluster 0 39.18 scrub ok 2024-03-11T21:36:10.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:11.020 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:11.063 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:11.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.150 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:11.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:11.278 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:11.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:11.408 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:11.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:11.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:11.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.646 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: cluster 2024-03-11T21:36:10.239622+0000 osd.4 (osd.4) 273 : cluster 0 3.14 scrub starts 2024-03-11T21:36:11.646 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: cluster 2024-03-11T21:36:10.243634+0000 osd.4 (osd.4) 274 : cluster 0 3.14 scrub ok 2024-03-11T21:36:11.646 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: cluster 2024-03-11T21:36:10.460401+0000 osd.6 (osd.6) 233 : cluster 0 5.11 scrub starts 2024-03-11T21:36:11.646 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: cluster 2024-03-11T21:36:10.461717+0000 osd.6 (osd.6) 234 : cluster 0 5.11 scrub ok 2024-03-11T21:36:11.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: cluster 2024-03-11T21:36:10.597465+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:36:11.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: audit 2024-03-11T21:36:10.601701+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:11.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:11 smithi203 bash[23654]: audit 2024-03-11T21:36:10.609588+0000 mon.a (mon.0) 2874 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:11.659 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:11.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.790 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:11.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:11.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: cluster 2024-03-11T21:36:10.239622+0000 osd.4 (osd.4) 273 : cluster 0 3.14 scrub starts 2024-03-11T21:36:11.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: cluster 2024-03-11T21:36:10.243634+0000 osd.4 (osd.4) 274 : cluster 0 3.14 scrub ok 2024-03-11T21:36:11.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: cluster 2024-03-11T21:36:10.460401+0000 osd.6 (osd.6) 233 : cluster 0 5.11 scrub starts 2024-03-11T21:36:11.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: cluster 2024-03-11T21:36:10.461717+0000 osd.6 (osd.6) 234 : cluster 0 5.11 scrub ok 2024-03-11T21:36:11.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: cluster 2024-03-11T21:36:10.597465+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: audit 2024-03-11T21:36:10.601701+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[29284]: audit 2024-03-11T21:36:10.609588+0000 mon.a (mon.0) 2874 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: cluster 2024-03-11T21:36:10.239622+0000 osd.4 (osd.4) 273 : cluster 0 3.14 scrub starts 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: cluster 2024-03-11T21:36:10.243634+0000 osd.4 (osd.4) 274 : cluster 0 3.14 scrub ok 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: cluster 2024-03-11T21:36:10.460401+0000 osd.6 (osd.6) 233 : cluster 0 5.11 scrub starts 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: cluster 2024-03-11T21:36:10.461717+0000 osd.6 (osd.6) 234 : cluster 0 5.11 scrub ok 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: cluster 2024-03-11T21:36:10.597465+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: audit 2024-03-11T21:36:10.601701+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:11.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:11 smithi143 bash[21512]: audit 2024-03-11T21:36:10.609588+0000 mon.a (mon.0) 2874 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:12.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:12.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:12.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:12.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:12.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:12.326 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:12.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:12.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:12.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:12.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:12.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:12.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:12.723 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:12.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:12.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.201230+0000 osd.4 (osd.4) 275 : cluster 0 118.14 scrub starts 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.203400+0000 osd.4 (osd.4) 276 : cluster 0 118.14 scrub ok 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.353856+0000 osd.0 (osd.0) 295 : cluster 0 118.1 scrub starts 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.356098+0000 osd.0 (osd.0) 296 : cluster 0 118.1 scrub ok 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.440809+0000 mgr.y (mgr.14550) 522 : cluster 0 pgmap v791: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.542444+0000 osd.7 (osd.7) 298 : cluster 0 2.1d scrub starts 2024-03-11T21:36:12.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.544693+0000 osd.7 (osd.7) 299 : cluster 0 2.1d scrub ok 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: audit 2024-03-11T21:36:11.604891+0000 mon.a (mon.0) 2875 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.611850+0000 mon.a (mon.0) 2876 : cluster 0 osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.615554+0000 osd.5 (osd.5) 311 : cluster 0 3.1c scrub starts 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: audit 2024-03-11T21:36:11.625123+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.636449+0000 osd.5 (osd.5) 312 : cluster 0 3.1c scrub ok 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.655121+0000 osd.3 (osd.3) 305 : cluster 0 9.1 scrub starts 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.656605+0000 osd.3 (osd.3) 306 : cluster 0 9.1 scrub ok 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.795470+0000 osd.2 (osd.2) 259 : cluster 0 2.c scrub starts 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:11.797948+0000 osd.2 (osd.2) 260 : cluster 0 2.c scrub ok 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.030658+0000 osd.1 (osd.1) 101 : cluster 0 6.1d scrub starts 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.033249+0000 osd.1 (osd.1) 102 : cluster 0 6.1d scrub ok 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.380232+0000 mon.a (mon.0) 2877 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:12.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.615789+0000 mon.a (mon.0) 2878 : cluster 0 osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.201230+0000 osd.4 (osd.4) 275 : cluster 0 118.14 scrub starts 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.203400+0000 osd.4 (osd.4) 276 : cluster 0 118.14 scrub ok 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.353856+0000 osd.0 (osd.0) 295 : cluster 0 118.1 scrub starts 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.356098+0000 osd.0 (osd.0) 296 : cluster 0 118.1 scrub ok 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.440809+0000 mgr.y (mgr.14550) 522 : cluster 0 pgmap v791: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.542444+0000 osd.7 (osd.7) 298 : cluster 0 2.1d scrub starts 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.544693+0000 osd.7 (osd.7) 299 : cluster 0 2.1d scrub ok 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: audit 2024-03-11T21:36:11.604891+0000 mon.a (mon.0) 2875 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.611850+0000 mon.a (mon.0) 2876 : cluster 0 osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.615554+0000 osd.5 (osd.5) 311 : cluster 0 3.1c scrub starts 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: audit 2024-03-11T21:36:11.625123+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:12.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.636449+0000 osd.5 (osd.5) 312 : cluster 0 3.1c scrub ok 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.655121+0000 osd.3 (osd.3) 305 : cluster 0 9.1 scrub starts 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.656605+0000 osd.3 (osd.3) 306 : cluster 0 9.1 scrub ok 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.795470+0000 osd.2 (osd.2) 259 : cluster 0 2.c scrub starts 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:11.797948+0000 osd.2 (osd.2) 260 : cluster 0 2.c scrub ok 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.030658+0000 osd.1 (osd.1) 101 : cluster 0 6.1d scrub starts 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.033249+0000 osd.1 (osd.1) 102 : cluster 0 6.1d scrub ok 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.380232+0000 mon.a (mon.0) 2877 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:12.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:12 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.615789+0000 mon.a (mon.0) 2878 : cluster 0 osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:36:12.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.201230+0000 osd.4 (osd.4) 275 : cluster 0 118.14 scrub starts 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.203400+0000 osd.4 (osd.4) 276 : cluster 0 118.14 scrub ok 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.353856+0000 osd.0 (osd.0) 295 : cluster 0 118.1 scrub starts 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.356098+0000 osd.0 (osd.0) 296 : cluster 0 118.1 scrub ok 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.440809+0000 mgr.y (mgr.14550) 522 : cluster 0 pgmap v791: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.542444+0000 osd.7 (osd.7) 298 : cluster 0 2.1d scrub starts 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.544693+0000 osd.7 (osd.7) 299 : cluster 0 2.1d scrub ok 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: audit 2024-03-11T21:36:11.604891+0000 mon.a (mon.0) 2875 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.611850+0000 mon.a (mon.0) 2876 : cluster 0 osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.615554+0000 osd.5 (osd.5) 311 : cluster 0 3.1c scrub starts 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: audit 2024-03-11T21:36:11.625123+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.636449+0000 osd.5 (osd.5) 312 : cluster 0 3.1c scrub ok 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.655121+0000 osd.3 (osd.3) 305 : cluster 0 9.1 scrub starts 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.656605+0000 osd.3 (osd.3) 306 : cluster 0 9.1 scrub ok 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.795470+0000 osd.2 (osd.2) 259 : cluster 0 2.c scrub starts 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:11.797948+0000 osd.2 (osd.2) 260 : cluster 0 2.c scrub ok 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.030658+0000 osd.1 (osd.1) 101 : cluster 0 6.1d scrub starts 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.033249+0000 osd.1 (osd.1) 102 : cluster 0 6.1d scrub ok 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.380232+0000 mon.a (mon.0) 2877 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:12.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:12 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.615789+0000 mon.a (mon.0) 2878 : cluster 0 osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:36:12.983 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:13.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:13.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:13.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:13.245 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:13.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:13.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.503 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:13.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:13.673 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:36:13.674 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:36:13.674 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:36:13.674 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:36:13.674 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.674 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24836.0:2 25.1 25:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 446.89298257500002, 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6763401199999999, 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.675 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24836", 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/184328842", 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.676 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807042+0000", 2024-03-11T21:36:13.677 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.678 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.678 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.678 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807044+0000", 2024-03-11T21:36:13.678 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.843e-06 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807048+0000", 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7269999999999998e-06 2024-03-11T21:36:13.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807058+0000", 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7999999999999993e-06 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:46.807067+0000", 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9590000000000001e-06 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.681 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:49.771853+0000", 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9647853340000001 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.682 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:49.771874+0000", 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1265999999999999e-05 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.358550+0000", 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5866757090000001 2024-03-11T21:36:13.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.358659+0000", 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000109337 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.371175+0000", 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012516042999999999 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.685 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:36:13.686 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.378988+0000", 2024-03-11T21:36:13.686 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0078134309999999992 2024-03-11T21:36:13.686 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.686 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.686 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.423413+0000", 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.044424287 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.687 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483276+0000", 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.059863422999999999 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483335+0000", 2024-03-11T21:36:13.688 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8535000000000001e-05 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:51.483382+0000", 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7426000000000003e-05 2024-03-11T21:36:13.689 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.24730.0:2 36.1c 36:39ce3343:test-rados-api-smithi143-71918-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-11T21:36:13.690 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 441.376666564, 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.577233369, 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.24730", 2024-03-11T21:36:13.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1999332124", 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.692 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323358+0000", 2024-03-11T21:36:13.693 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323359+0000", 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5039999999999999e-06 2024-03-11T21:36:13.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.323366+0000", 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2489999999999998e-06 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.325811+0000", 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024457099999999998 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.325825+0000", 2024-03-11T21:36:13.696 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3283e-05 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.326088+0000", 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026380000000000002 2024-03-11T21:36:13.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.698 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.698 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:52.326098+0000", 2024-03-11T21:36:13.698 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1339999999999993e-06 2024-03-11T21:36:13.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885403+0000", 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5593055150000001 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.699 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885537+0000", 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000134141 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:36:13.700 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.885675+0000", 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000137236 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.891300+0000", 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056256520000000001 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.701 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.891791+0000", 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00049041999999999998 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900495+0000", 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0087042160000000007 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.702 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900549+0000", 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4632000000000001e-05 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:54.900591+0000", 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1876999999999999e-05 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.25139.0:2 51.15 51:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 437.64571739500002, 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8283110329999999, 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.25139", 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2889742841", 2024-03-11T21:36:13.704 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054307+0000", 2024-03-11T21:36:13.705 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054308+0000", 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.850000000000001e-07 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.706 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054313+0000", 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2109999999999997e-06 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054328+0000", 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4766999999999999e-05 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.707 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.054337+0000", 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.02e-06 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.088502+0000", 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.034165244999999997 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.708 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:56.088513+0000", 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0225e-05 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.869905+0000", 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7813922409999998 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.709 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870009+0000", 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000104211 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870101+0000", 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2275000000000007e-05 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.710 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.870700+0000", 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00059873000000000001 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.871403+0000", 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00070267699999999997 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882550+0000", 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011147527000000001 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882585+0000", 2024-03-11T21:36:13.712 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.5185000000000003e-05 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:28:58.882618+0000", 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2734000000000002e-05 2024-03-11T21:36:13.713 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23010.0:2 121.15 121:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-03-11T21:36:13.714 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 399.96055705100002, 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8154935549999998, 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.715 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23010", 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2727152565", 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.716 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739467+0000", 2024-03-11T21:36:13.717 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.718 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.718 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.718 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.718 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739470+0000", 2024-03-11T21:36:13.718 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0810000000000001e-06 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739478+0000", 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1980000000000001e-06 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.719 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739495+0000", 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7447000000000001e-05 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.720 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739508+0000", 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.271e-05 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.721 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739585+0000", 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7406999999999999e-05 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:33.739596+0000", 2024-03-11T21:36:13.722 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0583e-05 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.529742+0000", 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7901457499999998 2024-03-11T21:36:13.723 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.529867+0000", 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000125258 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.724 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.530003+0000", 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013603000000000001 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.725 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.551591+0000", 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.021587688000000001 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.726 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.552517+0000", 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00092659700000000003 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554865+0000", 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002348232 2024-03-11T21:36:13.727 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554920+0000", 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4293999999999999e-05 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.728 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.729 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.729 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.554961+0000", 2024-03-11T21:36:13.729 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1279999999999998e-05 2024-03-11T21:36:13.729 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.729 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:1 123.6s0 123:653d3f2f:::000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 398.20224283900001, 2024-03-11T21:36:13.730 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0403799309999999, 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:36:13.731 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.733 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.733 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.733 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497782+0000", 2024-03-11T21:36:13.733 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.733 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497784+0000", 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.88e-06 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.734 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497789+0000", 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6980000000000003e-06 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.735 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497818+0000", 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8280999999999999e-05 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497829+0000", 2024-03-11T21:36:13.736 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1199999999999999e-05 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497903+0000", 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4468999999999995e-05 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.737 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:35.497908+0000", 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4460000000000003e-06 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.738 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493194+0000", 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.99528657200000004 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493271+0000", 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7151999999999996e-05 2024-03-11T21:36:13.739 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.493536+0000", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026473799999999999 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.499652+0000", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0061159090000000001 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.538093+0000", 2024-03-11T21:36:13.740 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.038441365999999998 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:36.538162+0000", 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8219999999999994e-05 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.e epoch 178/176 rep_tid 597 entries 178'7 (0'0) error 126:748fcfa5:::test_252:head by client.23697.0:259 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 177'5)", 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 395.09492916300002, 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15239376199999999, 2024-03-11T21:36:13.741 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605095+0000", 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605096+0000", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.13e-06 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605108+0000", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1088e-05 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605129+0000", 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1834000000000001e-05 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.743 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605144+0000", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5088e-05 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605297+0000", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000152299 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.757489+0000", 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15219232299999999 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.744 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:268 123.6s0 123:680f0a60:::267:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 395.09449307699998, 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.153234183, 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:36:13.745 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 268 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605531+0000", 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.746 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605533+0000", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5710000000000001e-06 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605540+0000", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1250000000000004e-06 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605554+0000", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.379e-05 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.747 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.605563+0000", 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.8170000000000005e-06 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606212+0000", 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00064885599999999996 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606345+0000", 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013343599999999999 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.606687+0000", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000342381 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758441+0000", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.15175385299999999 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758507+0000", 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6124000000000006e-05 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:38.758766+0000", 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025823000000000003 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.22002.0:413 123.6s0 123:7760f31e:::412:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 394.15731800700001, 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.183137513, 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.22002", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 413 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542706+0000", 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.751 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542708+0000", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7570000000000001e-06 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542716+0000", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9230000000000002e-06 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542732+0000", 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5624000000000002e-05 2024-03-11T21:36:13.752 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542740+0000", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8699999999999992e-06 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542795+0000", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4852000000000003e-05 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542888+0000", 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.3542000000000007e-05 2024-03-11T21:36:13.753 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543159+0000", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027125599999999999 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725710+0000", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18255065400000001 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725798+0000", 2024-03-11T21:36:13.754 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.7621999999999996e-05 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725844+0000", 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6412999999999998e-05 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.14 epoch 179/176 rep_tid 673 entries 179'13 (0'0) error 126:2884fdb0:::test_383:head by client.23697.0:390 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 179'11)", 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 394.157053357, 2024-03-11T21:36:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.182524724, 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542971+0000", 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.756 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542972+0000", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0890000000000001e-06 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.542984+0000", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2224e-05 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543008+0000", 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3124000000000001e-05 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.757 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543022+0000", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4521e-05 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543225+0000", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020285100000000001 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725496+0000", 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18227091500000001 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.758 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 179/175 ECSubWriteReply(tid=485, last_complete=179'50, committed=1, applied=1))", 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 394.15625311299999, 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18223889700000001, 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.759 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543771+0000", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543772+0000", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6300000000000005e-07 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.543776+0000", 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0799999999999999e-06 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.760 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725718+0000", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18194215899999999 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725726+0000", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5889999999999997e-06 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725756+0000", 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0587999999999999e-05 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.761 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.726010+0000", 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000253818 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWriteReply(123.6s0 179/175 ECSubWriteReply(tid=485, last_complete=179'50, committed=1, applied=1))", 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 394.15511971000001, 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18037215600000001, 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.762 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544905+0000", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544906+0000", 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7699999999999992e-07 2024-03-11T21:36:13.763 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.544910+0000", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3409999999999996e-06 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725101+0000", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.18019127800000001 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725118+0000", 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6705e-05 2024-03-11T21:36:13.764 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725263+0000", 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014521099999999999 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:39.725277+0000", 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3644e-05 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.765 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23010.0:428 121.14 121:2a92f4ca:::test_421:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 393.68456614600001, 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.294164496, 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23010", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2727152565", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 428 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.766 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015458+0000", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015459+0000", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0759999999999999e-06 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.767 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015465+0000", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.054e-06 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015477+0000", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1257000000000001e-05 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015485+0000", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.79e-06 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.768 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015740+0000", 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025537800000000001 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.015907+0000", 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016721800000000001 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.309623+0000", 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.29371572299999998 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.769 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing_reply(121.14 epoch 179/174 rep_tid 537 lcod 179'15)", 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 393.68284115199998, 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19906863499999999, 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.770 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017183+0000", 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017184+0000", 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3300000000000002e-07 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.771 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017187+0000", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.27e-06 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017198+0000", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1039999999999999e-05 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017206+0000", 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2849999999999997e-06 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.772 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.017273+0000", 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.745e-05 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216252+0000", 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19897895700000001 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.773 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWrite(123.4s2 179/175 ECSubWrite(tid=447, reqid=client.22002.0:483, at_version=0'0, trim_to=0'0, roll_forward_to=179'52))", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 393.67989843800001, 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19683077800000001, 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.774 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020126+0000", 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020127+0000", 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.387e-06 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.020144+0000", 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6331999999999999e-05 2024-03-11T21:36:13.775 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216496+0000", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.196352472 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216507+0000", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0541e-05 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216557+0000", 2024-03-11T21:36:13.776 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.0590999999999997e-05 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216567+0000", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.448e-06 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216928+0000", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00036090599999999998 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216957+0000", 2024-03-11T21:36:13.777 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9101000000000001e-05 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(126.1f epoch 179/176 rep_tid 708 entries 179'12 (0'0) error 126:fdf52ff4:::test_450:head by client.23697.0:457 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 179'10)", 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 393.67692807999998, 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19388946500000001, 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023096+0000", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023098+0000", 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8250000000000001e-06 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.023115+0000", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6483000000000001e-05 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216307+0000", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.19319249399999999 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216335+0000", 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7846000000000001e-05 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216434+0000", 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9415999999999998e-05 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:40.216986+0000", 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00055140099999999997 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.29976.0:1 219.6s0 219:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:09.284476+0000", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 304.41554838000002, 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1062499649999999, 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.29976", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2899448680", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284476+0000", 2024-03-11T21:36:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284476+0000", 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284478+0000", 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.584e-06 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284493+0000", 2024-03-11T21:36:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5379e-05 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284510+0000", 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6670000000000001e-05 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.284519+0000", 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9220000000000003e-06 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.289492+0000", 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0049731599999999999 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:09.289493+0000", 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3599999999999999e-06 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.386264+0000", 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.096771097 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.386415+0000", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000150815 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.386820+0000", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00040453199999999998 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.387368+0000", 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00054820300000000004 2024-03-11T21:36:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.390653+0000", 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0032848180000000001 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:10.390726+0000", 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3424999999999996e-05 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:1673 220.2 220:40282ed0:test-rados-api-smithi143-71773-20::foo:head [writefull 0~3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:15.266157+0000", 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 298.43386776900002, 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.45459530599999998, 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1673 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266157+0000", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266157+0000", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266159+0000", 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5050000000000002e-06 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266173+0000", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.331e-05 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266202+0000", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9379000000000001e-05 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266217+0000", 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4612e-05 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266602+0000", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00038569 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.266609+0000", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1609999999999997e-06 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.712040+0000", 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.44543061099999998 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.712254+0000", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000213742 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.719044+0000", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0067905960000000003 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.719144+0000", 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9288000000000002e-05 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.719277+0000", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013293699999999999 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.720240+0000", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00096301999999999998 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.720632+0000", 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000392159 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.720680+0000", 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7886999999999997e-05 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.720752+0000", 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2409000000000004e-05 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41336.0:1 243.7s0 243:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-03-11T21:36:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:53.274422+0000", 2024-03-11T21:36:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 260.42560211799997, 2024-03-11T21:36:13.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.047278691, 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41336", 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1928853474", 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:36:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274422+0000", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274422+0000", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274425+0000", 2024-03-11T21:36:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5560000000000002e-06 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274438+0000", 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3040000000000001e-05 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274468+0000", 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0525000000000003e-05 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.274489+0000", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0338000000000001e-05 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.277129+0000", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0026397600000000001 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:53.277133+0000", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.527e-06 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.305975+0000", 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0288418720000001 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.306274+0000", 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029861599999999997 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.310080+0000", 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0038060920000000001 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.318215+0000", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0081350169999999996 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.321086+0000", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0028708140000000002 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:54.321701+0000", 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00061553400000000005 2024-03-11T21:36:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41444.0:1 259.6s0 259:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:38.330054+0000", 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 215.369970473, 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.037758811, 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41444", 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3006824523", 2024-03-11T21:36:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330054+0000", 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330054+0000", 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330056+0000", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7579999999999999e-06 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330066+0000", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0407999999999999e-05 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330083+0000", 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6342999999999999e-05 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.330098+0000", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5291999999999999e-05 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.334292+0000", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0041938069999999999 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:38.334296+0000", 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.51e-06 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.361183+0000", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0268865549999999 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.361274+0000", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1644000000000006e-05 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.361552+0000", 2024-03-11T21:36:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027777699999999999 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.361974+0000", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00042185200000000001 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.367673+0000", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056987460000000002 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:39.367813+0000", 2024-03-11T21:36:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014011899999999999 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41600.0:1 280.7s0 280:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:33:58.481225+0000", 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 135.21879962099999, 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.928825448, 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41600", 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2843625864", 2024-03-11T21:36:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481225+0000", 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481225+0000", 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:36:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481227+0000", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3659999999999999e-06 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481237+0000", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2730000000000008e-06 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481251+0000", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4504999999999999e-05 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:36:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481267+0000", 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6234000000000001e-05 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481360+0000", 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2608000000000005e-05 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:58.481366+0000", 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.896e-06 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.404124+0000", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.92275799700000005 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.404420+0000", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029577 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.405493+0000", 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001073655 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.406889+0000", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001395828 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.409974+0000", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0030852229999999998 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:59.410050+0000", 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6093000000000005e-05 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:36:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:36:13.817 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:36:13.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.652552+0000 osd.3 (osd.3) 307 : cluster 0 2.1b scrub starts 2024-03-11T21:36:13.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[29284]: cluster 2024-03-11T21:36:12.655597+0000 osd.3 (osd.3) 308 : cluster 0 2.1b scrub ok 2024-03-11T21:36:13.817 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.614665+0000 mon.a (mon.0) 2879 : cluster 0 osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:36:13.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.652552+0000 osd.3 (osd.3) 307 : cluster 0 2.1b scrub starts 2024-03-11T21:36:13.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[21512]: cluster 2024-03-11T21:36:12.655597+0000 osd.3 (osd.3) 308 : cluster 0 2.1b scrub ok 2024-03-11T21:36:13.818 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:13 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.614665+0000 mon.a (mon.0) 2879 : cluster 0 osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:36:13.845 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:13.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:13.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:13 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.652552+0000 osd.3 (osd.3) 307 : cluster 0 2.1b scrub starts 2024-03-11T21:36:13.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:13 smithi203 bash[23654]: cluster 2024-03-11T21:36:12.655597+0000 osd.3 (osd.3) 308 : cluster 0 2.1b scrub ok 2024-03-11T21:36:13.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:13 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.614665+0000 mon.a (mon.0) 2879 : cluster 0 osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:36:13.975 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:14.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:14.106 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:14.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:14.237 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:14.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:14.369 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:14.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:14.498 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:14.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:14.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:14.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:14.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.201416+0000 osd.4 (osd.4) 277 : cluster 0 4.1 scrub starts 2024-03-11T21:36:14.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.203831+0000 osd.4 (osd.4) 278 : cluster 0 4.1 scrub ok 2024-03-11T21:36:14.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.355124+0000 osd.0 (osd.0) 297 : cluster 0 118.0 scrub starts 2024-03-11T21:36:14.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.358858+0000 osd.0 (osd.0) 298 : cluster 0 118.0 scrub ok 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[21512]: cluster 2024-03-11T21:36:13.442844+0000 mgr.y (mgr.14550) 523 : cluster 0 pgmap v794: 260 pgs: 5 unknown, 255 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.201416+0000 osd.4 (osd.4) 277 : cluster 0 4.1 scrub starts 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.203831+0000 osd.4 (osd.4) 278 : cluster 0 4.1 scrub ok 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.355124+0000 osd.0 (osd.0) 297 : cluster 0 118.0 scrub starts 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.358858+0000 osd.0 (osd.0) 298 : cluster 0 118.0 scrub ok 2024-03-11T21:36:14.743 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:14 smithi143 bash[29284]: cluster 2024-03-11T21:36:13.442844+0000 mgr.y (mgr.14550) 523 : cluster 0 pgmap v794: 260 pgs: 5 unknown, 255 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:14.758 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:14.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:14.873 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:14 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.201416+0000 osd.4 (osd.4) 277 : cluster 0 4.1 scrub starts 2024-03-11T21:36:14.873 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:14 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.203831+0000 osd.4 (osd.4) 278 : cluster 0 4.1 scrub ok 2024-03-11T21:36:14.873 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:14 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.355124+0000 osd.0 (osd.0) 297 : cluster 0 118.0 scrub starts 2024-03-11T21:36:14.873 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:14 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.358858+0000 osd.0 (osd.0) 298 : cluster 0 118.0 scrub ok 2024-03-11T21:36:14.874 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:14 smithi203 bash[23654]: cluster 2024-03-11T21:36:13.442844+0000 mgr.y (mgr.14550) 523 : cluster 0 pgmap v794: 260 pgs: 5 unknown, 255 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:14.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:14.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:15.020 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:15.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.151 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:15.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:15.273 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:15.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:15.402 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:15.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.532 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:15.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.252605+0000 osd.4 (osd.4) 279 : cluster 0 6.17 scrub starts 2024-03-11T21:36:15.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.254836+0000 osd.4 (osd.4) 280 : cluster 0 6.17 scrub ok 2024-03-11T21:36:15.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.625622+0000 osd.3 (osd.3) 309 : cluster 0 39.1b scrub starts 2024-03-11T21:36:15.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.628355+0000 osd.3 (osd.3) 310 : cluster 0 39.1b scrub ok 2024-03-11T21:36:15.647 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.672801+0000 osd.5 (osd.5) 313 : cluster 0 6.19 scrub starts 2024-03-11T21:36:15.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.674490+0000 osd.5 (osd.5) 314 : cluster 0 6.19 scrub ok 2024-03-11T21:36:15.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.785770+0000 osd.2 (osd.2) 261 : cluster 0 2.1 scrub starts 2024-03-11T21:36:15.648 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:15 smithi203 bash[23654]: cluster 2024-03-11T21:36:14.786957+0000 osd.2 (osd.2) 262 : cluster 0 2.1 scrub ok 2024-03-11T21:36:15.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:15.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:15.791 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:15.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:15.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.252605+0000 osd.4 (osd.4) 279 : cluster 0 6.17 scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.254836+0000 osd.4 (osd.4) 280 : cluster 0 6.17 scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.625622+0000 osd.3 (osd.3) 309 : cluster 0 39.1b scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.628355+0000 osd.3 (osd.3) 310 : cluster 0 39.1b scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.672801+0000 osd.5 (osd.5) 313 : cluster 0 6.19 scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.674490+0000 osd.5 (osd.5) 314 : cluster 0 6.19 scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.785770+0000 osd.2 (osd.2) 261 : cluster 0 2.1 scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[21512]: cluster 2024-03-11T21:36:14.786957+0000 osd.2 (osd.2) 262 : cluster 0 2.1 scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.252605+0000 osd.4 (osd.4) 279 : cluster 0 6.17 scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.254836+0000 osd.4 (osd.4) 280 : cluster 0 6.17 scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.625622+0000 osd.3 (osd.3) 309 : cluster 0 39.1b scrub starts 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.628355+0000 osd.3 (osd.3) 310 : cluster 0 39.1b scrub ok 2024-03-11T21:36:15.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.672801+0000 osd.5 (osd.5) 313 : cluster 0 6.19 scrub starts 2024-03-11T21:36:15.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.674490+0000 osd.5 (osd.5) 314 : cluster 0 6.19 scrub ok 2024-03-11T21:36:15.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.785770+0000 osd.2 (osd.2) 261 : cluster 0 2.1 scrub starts 2024-03-11T21:36:15.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:15 smithi143 bash[29284]: cluster 2024-03-11T21:36:14.786957+0000 osd.2 (osd.2) 262 : cluster 0 2.1 scrub ok 2024-03-11T21:36:15.921 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:16.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:16.052 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:16.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:16.184 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:16.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:16.315 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:16.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:16.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:16.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:16.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:16.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:15.443993+0000 mgr.y (mgr.14550) 524 : cluster 0 pgmap v796: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:15.675479+0000 osd.3 (osd.3) 311 : cluster 0 3.1e scrub starts 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:15.678022+0000 osd.3 (osd.3) 312 : cluster 0 3.1e scrub ok 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:16.050525+0000 osd.1 (osd.1) 103 : cluster 0 118.1f scrub starts 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:16.054848+0000 osd.1 (osd.1) 104 : cluster 0 118.1f scrub ok 2024-03-11T21:36:16.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[21512]: cluster 2024-03-11T21:36:16.385762+0000 mon.a (mon.0) 2880 : cluster 0 osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:15.443993+0000 mgr.y (mgr.14550) 524 : cluster 0 pgmap v796: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:15.675479+0000 osd.3 (osd.3) 311 : cluster 0 3.1e scrub starts 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:15.678022+0000 osd.3 (osd.3) 312 : cluster 0 3.1e scrub ok 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:16.050525+0000 osd.1 (osd.1) 103 : cluster 0 118.1f scrub starts 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:16.054848+0000 osd.1 (osd.1) 104 : cluster 0 118.1f scrub ok 2024-03-11T21:36:16.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:16 smithi143 bash[29284]: cluster 2024-03-11T21:36:16.385762+0000 mon.a (mon.0) 2880 : cluster 0 osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:36:16.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:16.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:16.831 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:16.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:15.443993+0000 mgr.y (mgr.14550) 524 : cluster 0 pgmap v796: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:15.675479+0000 osd.3 (osd.3) 311 : cluster 0 3.1e scrub starts 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:15.678022+0000 osd.3 (osd.3) 312 : cluster 0 3.1e scrub ok 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:16.050525+0000 osd.1 (osd.1) 103 : cluster 0 118.1f scrub starts 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:16.054848+0000 osd.1 (osd.1) 104 : cluster 0 118.1f scrub ok 2024-03-11T21:36:16.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:16 smithi203 bash[23654]: cluster 2024-03-11T21:36:16.385762+0000 mon.a (mon.0) 2880 : cluster 0 osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:36:16.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:17.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:17.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:17.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:17.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:17.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:17.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:17.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:17.616 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:17.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:17.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:17.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:17.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.010 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:18.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.140 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:18.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:18.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:18.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:18.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:18.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:36:18.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:18.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:18.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.287191+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.289715+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.327730+0000 osd.0 (osd.0) 299 : cluster 0 118.1a scrub starts 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.330724+0000 osd.0 (osd.0) 300 : cluster 0 118.1a scrub ok 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.444835+0000 mgr.y (mgr.14550) 525 : cluster 0 pgmap v798: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.766366+0000 osd.5 (osd.5) 315 : cluster 0 118.e scrub starts 2024-03-11T21:36:18.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.771130+0000 osd.5 (osd.5) 316 : cluster 0 118.e scrub ok 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.287191+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.289715+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.327730+0000 osd.0 (osd.0) 299 : cluster 0 118.1a scrub starts 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.330724+0000 osd.0 (osd.0) 300 : cluster 0 118.1a scrub ok 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.444835+0000 mgr.y (mgr.14550) 525 : cluster 0 pgmap v798: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.766366+0000 osd.5 (osd.5) 315 : cluster 0 118.e scrub starts 2024-03-11T21:36:18.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:18 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.771130+0000 osd.5 (osd.5) 316 : cluster 0 118.e scrub ok 2024-03-11T21:36:18.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:18.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:18.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.287191+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-03-11T21:36:18.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.289715+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-03-11T21:36:18.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.327730+0000 osd.0 (osd.0) 299 : cluster 0 118.1a scrub starts 2024-03-11T21:36:18.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.330724+0000 osd.0 (osd.0) 300 : cluster 0 118.1a scrub ok 2024-03-11T21:36:18.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.444835+0000 mgr.y (mgr.14550) 525 : cluster 0 pgmap v798: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-11T21:36:18.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.766366+0000 osd.5 (osd.5) 315 : cluster 0 118.e scrub starts 2024-03-11T21:36:18.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:18 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.771130+0000 osd.5 (osd.5) 316 : cluster 0 118.e scrub ok 2024-03-11T21:36:18.795 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:18.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:18.921 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:19.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:19.046 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:19.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:19.176 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:19.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:19.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:19.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:19.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:19.541 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:19.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:19.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.823644+0000 osd.2 (osd.2) 263 : cluster 0 118.5 scrub starts 2024-03-11T21:36:19.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:17.826335+0000 osd.2 (osd.2) 264 : cluster 0 118.5 scrub ok 2024-03-11T21:36:19.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.209308+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:19.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.209591+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.313875+0000 osd.0 (osd.0) 301 : cluster 0 118.4 scrub starts 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.317345+0000 osd.0 (osd.0) 302 : cluster 0 118.4 scrub ok 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.446604+0000 osd.6 (osd.6) 235 : cluster 0 118.1d scrub starts 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.450829+0000 osd.6 (osd.6) 236 : cluster 0 118.1d scrub ok 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: cluster 2024-03-11T21:36:18.676530+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[21512]: audit 2024-03-11T21:36:18.725061+0000 mgr.y (mgr.14550) 526 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.823644+0000 osd.2 (osd.2) 263 : cluster 0 118.5 scrub starts 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:17.826335+0000 osd.2 (osd.2) 264 : cluster 0 118.5 scrub ok 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.209308+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:19.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.209591+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.313875+0000 osd.0 (osd.0) 301 : cluster 0 118.4 scrub starts 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.317345+0000 osd.0 (osd.0) 302 : cluster 0 118.4 scrub ok 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.446604+0000 osd.6 (osd.6) 235 : cluster 0 118.1d scrub starts 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.450829+0000 osd.6 (osd.6) 236 : cluster 0 118.1d scrub ok 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: cluster 2024-03-11T21:36:18.676530+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:36:19.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:19 smithi143 bash[29284]: audit 2024-03-11T21:36:18.725061+0000 mgr.y (mgr.14550) 526 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:19.697 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:19.800 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:19.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:19.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:19.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:20.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.823644+0000 osd.2 (osd.2) 263 : cluster 0 118.5 scrub starts 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:17.826335+0000 osd.2 (osd.2) 264 : cluster 0 118.5 scrub ok 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.209308+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.209591+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.313875+0000 osd.0 (osd.0) 301 : cluster 0 118.4 scrub starts 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.317345+0000 osd.0 (osd.0) 302 : cluster 0 118.4 scrub ok 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.446604+0000 osd.6 (osd.6) 235 : cluster 0 118.1d scrub starts 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.450829+0000 osd.6 (osd.6) 236 : cluster 0 118.1d scrub ok 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: cluster 2024-03-11T21:36:18.676530+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:36:20.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:19 smithi203 bash[23654]: audit 2024-03-11T21:36:18.725061+0000 mgr.y (mgr.14550) 526 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:20.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:20.088 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:20.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:20.218 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:20.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:20.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:20.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:20.476 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:20.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.600 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:20.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:20.730 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:20.780 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.120970+0000 osd.1 (osd.1) 105 : cluster 0 118.1b scrub starts 2024-03-11T21:36:20.780 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.125145+0000 osd.1 (osd.1) 106 : cluster 0 118.1b scrub ok 2024-03-11T21:36:20.780 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.276847+0000 osd.4 (osd.4) 285 : cluster 0 2.f scrub starts 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.279366+0000 osd.4 (osd.4) 286 : cluster 0 2.f scrub ok 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.445668+0000 mgr.y (mgr.14550) 527 : cluster 0 pgmap v800: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.740302+0000 osd.3 (osd.3) 313 : cluster 0 5.0 scrub starts 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.742115+0000 osd.3 (osd.3) 314 : cluster 0 5.0 scrub ok 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.800806+0000 osd.5 (osd.5) 317 : cluster 0 9.12 scrub starts 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.803268+0000 osd.5 (osd.5) 318 : cluster 0 9.12 scrub ok 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.120970+0000 osd.1 (osd.1) 105 : cluster 0 118.1b scrub starts 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.125145+0000 osd.1 (osd.1) 106 : cluster 0 118.1b scrub ok 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.276847+0000 osd.4 (osd.4) 285 : cluster 0 2.f scrub starts 2024-03-11T21:36:20.781 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.279366+0000 osd.4 (osd.4) 286 : cluster 0 2.f scrub ok 2024-03-11T21:36:20.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.445668+0000 mgr.y (mgr.14550) 527 : cluster 0 pgmap v800: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T21:36:20.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.740302+0000 osd.3 (osd.3) 313 : cluster 0 5.0 scrub starts 2024-03-11T21:36:20.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.742115+0000 osd.3 (osd.3) 314 : cluster 0 5.0 scrub ok 2024-03-11T21:36:20.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.800806+0000 osd.5 (osd.5) 317 : cluster 0 9.12 scrub starts 2024-03-11T21:36:20.782 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:20 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.803268+0000 osd.5 (osd.5) 318 : cluster 0 9.12 scrub ok 2024-03-11T21:36:20.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- 172.21.15.143:0/2103312266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674101ec0 msgr2=0x7f3674102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 --2- 172.21.15.143:0/2103312266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674101ec0 0x7f3674102340 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f365c009990 tx=0x7f365c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- 172.21.15.143:0/2103312266 shutdown_connections 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 --2- 172.21.15.143:0/2103312266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674107f30 0x7f3674108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 --2- 172.21.15.143:0/2103312266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674102880 0x7f367410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 --2- 172.21.15.143:0/2103312266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674101ec0 0x7f3674102340 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- 172.21.15.143:0/2103312266 >> 172.21.15.143:0/2103312266 conn(0x7f3674078eb0 msgr2=0x7f36740792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- 172.21.15.143:0/2103312266 shutdown_connections 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- 172.21.15.143:0/2103312266 wait complete. 2024-03-11T21:36:20.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 librados: starting msgr at 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 librados: starting objecter 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 Processor -- start 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 -- start start 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.977+0000 7f367904f700 1 librados: setting wanted keys 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 librados: calling monclient init 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 0x7f367419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674107f30 0x7f3674197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36741116f0 con 0x7f3674101ec0 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674111570 con 0x7f3674107f30 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3674111870 con 0x7f3674102880 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 0x7f367419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 0x7f367419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50924/0 (socket says 172.21.15.143:50924) 2024-03-11T21:36:20.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58734/0 (socket says 172.21.15.143:58734) 2024-03-11T21:36:20.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 -- 172.21.15.143:0/345486029 learned_addr learned my addr 172.21.15.143:0/345486029 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:20.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674107f30 0x7f3674197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674107f30 msgr2=0x7f3674197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674107f30 0x7f3674197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 msgr2=0x7f367419d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 0x7f367419d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36741981c0 con 0x7f3674102880 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3674107f30 0x7f3674197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3674101ec0 0x7f367419d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f365c006010 tx=0x7f365c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 10 client.?.objecter ms_handle_connect 0x7f3674102880 2024-03-11T21:36:20.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c027070 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f365c022470 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3674198450 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c017650 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 10 client.41708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36741a4770 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 librados: init done 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f367404faa0 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f365c02a070 con 0x7f3674102880 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 0x7f3660071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(523..523 src has 87..523) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f365c08f710 con 0x7f3674102880 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 osd_map(523..523 src has 87..523) v4 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 3 client.41708.objecter handle_osd_map got epochs [523,523] > 0 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 3 client.41708.objecter handle_osd_map decoding full epoch 523 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 0x7f3660071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 20 client.41708.objecter dump_active .. 0 homeless 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v523) v1 ==== 54+0+16 (secure 0 0 0) 0x7f365c013070 con 0x7f3674102880 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 10 client.41708.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 1 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f367410a420 0x7f367410a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 0x7f3660071ce0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f3664009c10 tx=0x7f366400b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 1 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 0 1 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 4 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 10 client.41708.objecter _send_command 1 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f366006f820 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.981+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741a7ad0 con 0x7f367410a420 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f367410a420 0x7f367410a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f367410a420 0x7f367410a800 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:20.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f367410a420 2024-03-11T21:36:20.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741a7ad0 con 0x7f367410a420 2024-03-11T21:36:20.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:36:20.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 10 client.41708.objecter _finish_command 1 = system:0 2024-03-11T21:36:20.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 4 2024-03-11T21:36:20.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:20.985+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 0 1 2024-03-11T21:36:20.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 10 client.41708.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 2 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 2 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f3674198f90 osd=0 3 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 0 2 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 4 2024-03-11T21:36:20.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 10 client.41708.objecter _send_command 2 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.001+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3674109ea0 con 0x7f367410a420 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3674109ea0 con 0x7f367410a420 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 10 client.41708.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 4 2024-03-11T21:36:20.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 0 2 2024-03-11T21:36:20.980 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:36:20.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 10 client.41708.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:20.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f367402c6b0 0x7f367402ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 1 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 4 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 10 client.41708.objecter _send_command 3 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3674063070 con 0x7f367402c6b0 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f367402c6b0 0x7f367402ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f367402c6b0 2024-03-11T21:36:20.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f367402c6b0 0x7f367402ca90 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3674063070 con 0x7f367402c6b0 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 10 client.41708.objecter _finish_command 3 = system:0 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 4 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.005+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 1 3 2024-03-11T21:36:20.982 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 10 client.41708.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 4 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:20.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 4 2024-03-11T21:36:21.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36740629f0 osd=1 3 2024-03-11T21:36:21.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 1 4 2024-03-11T21:36:21.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 4 2024-03-11T21:36:21.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 10 client.41708.objecter _send_command 4 2024-03-11T21:36:21.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3674062470 con 0x7f367402c6b0 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3674062470 con 0x7f367402c6b0 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 10 client.41708.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 4 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 1 4 2024-03-11T21:36:21.001 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 10 client.41708.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 5 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f36741a9c90 0x7f36741ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 5 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 2 5 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 4 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 10 client.41708.objecter _send_command 5 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741aeee0 con 0x7f36741a9c90 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f36741a9c90 0x7f36741ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f36741a9c90 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.025+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f36741a9c90 0x7f36741ac080 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.029+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741aeee0 con 0x7f36741a9c90 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.029+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.029+0000 7f366bfff700 10 client.41708.objecter _finish_command 5 = system:0 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.029+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 4 2024-03-11T21:36:21.003 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.029+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 2 5 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 10 client.41708.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 6 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 6 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f3674062870 osd=2 3 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 2 6 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 4 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 10 client.41708.objecter _send_command 6 2024-03-11T21:36:21.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.045+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f367402d220 con 0x7f36741a9c90 2024-03-11T21:36:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.120970+0000 osd.1 (osd.1) 105 : cluster 0 118.1b scrub starts 2024-03-11T21:36:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.125145+0000 osd.1 (osd.1) 106 : cluster 0 118.1b scrub ok 2024-03-11T21:36:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.276847+0000 osd.4 (osd.4) 285 : cluster 0 2.f scrub starts 2024-03-11T21:36:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.279366+0000 osd.4 (osd.4) 286 : cluster 0 2.f scrub ok 2024-03-11T21:36:21.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.445668+0000 mgr.y (mgr.14550) 527 : cluster 0 pgmap v800: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T21:36:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.740302+0000 osd.3 (osd.3) 313 : cluster 0 5.0 scrub starts 2024-03-11T21:36:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.742115+0000 osd.3 (osd.3) 314 : cluster 0 5.0 scrub ok 2024-03-11T21:36:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.800806+0000 osd.5 (osd.5) 317 : cluster 0 9.12 scrub starts 2024-03-11T21:36:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:20 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.803268+0000 osd.5 (osd.5) 318 : cluster 0 9.12 scrub ok 2024-03-11T21:36:21.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f367402d220 con 0x7f36741a9c90 2024-03-11T21:36:21.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 10 client.41708.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 4 2024-03-11T21:36:21.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 2 6 2024-03-11T21:36:21.024 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:36:21.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 10 client.41708.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 7 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3674075190 0x7f36741ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 7 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 3 7 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 4 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 10 client.41708.objecter _send_command 7 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741a7830 con 0x7f3674075190 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3674075190 0x7f36741ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3674075190 0x7f36741ae5d0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f3674075190 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741a7830 con 0x7f3674075190 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 10 client.41708.objecter _finish_command 7 = system:0 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 4 2024-03-11T21:36:21.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.049+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 3 7 2024-03-11T21:36:21.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 10 client.41708.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 8 2024-03-11T21:36:21.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 8 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a7120 osd=3 3 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 3 8 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 4 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 10 client.41708.objecter _send_command 8 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36740755d0 con 0x7f3674075190 2024-03-11T21:36:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f36740755d0 con 0x7f3674075190 2024-03-11T21:36:21.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 10 client.41708.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 4 2024-03-11T21:36:21.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 3 8 2024-03-11T21:36:21.045 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 10 client.41708.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 9 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3674075ae0 0x7f36741b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 9 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 4 9 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 4 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 10 client.41708.objecter _send_command 9 2024-03-11T21:36:21.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3674076060 con 0x7f3674075ae0 2024-03-11T21:36:21.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3674075ae0 0x7f36741b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f3674075ae0 2024-03-11T21:36:21.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.069+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3674075ae0 0x7f36741b54b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.073+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f366401e040 con 0x7f3674075ae0 2024-03-11T21:36:21.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.073+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:36:21.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.073+0000 7f366bfff700 10 client.41708.objecter _finish_command 9 = system:0 2024-03-11T21:36:21.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.073+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 4 2024-03-11T21:36:21.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.073+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 4 9 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 10 client.41708.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 10 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.065 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 10 2024-03-11T21:36:21.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741aed60 osd=4 3 2024-03-11T21:36:21.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 4 10 2024-03-11T21:36:21.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 4 2024-03-11T21:36:21.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 10 client.41708.objecter _send_command 10 2024-03-11T21:36:21.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.089+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36741aeb70 con 0x7f3674075ae0 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f366401e020 con 0x7f3674075ae0 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f366bfff700 10 client.41708.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 4 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 4 10 2024-03-11T21:36:21.070 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 10 client.41708.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 11 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3674074cf0 0x7f36741b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 11 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 5 11 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 4 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 10 client.41708.objecter _send_command 11 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.093+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741a6ea0 con 0x7f3674074cf0 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3674074cf0 0x7f36741b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f367259c700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3674074cf0 0x7f36741b6b70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f3674074cf0 2024-03-11T21:36:21.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741a6ea0 con 0x7f3674074cf0 2024-03-11T21:36:21.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:36:21.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 10 client.41708.objecter _finish_command 11 = system:0 2024-03-11T21:36:21.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 4 2024-03-11T21:36:21.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.097+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 5 11 2024-03-11T21:36:21.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 10 client.41708.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 12 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 12 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a6950 osd=5 3 2024-03-11T21:36:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 5 12 2024-03-11T21:36:21.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 4 2024-03-11T21:36:21.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 10 client.41708.objecter _send_command 12 2024-03-11T21:36:21.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.113+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36741a6ea0 con 0x7f3674074cf0 2024-03-11T21:36:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.117+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f36741a6ea0 con 0x7f3674074cf0 2024-03-11T21:36:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.117+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.117+0000 7f366bfff700 10 client.41708.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.117+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 4 2024-03-11T21:36:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.117+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 5 12 2024-03-11T21:36:21.095 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:36:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 10 client.41708.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 13 2024-03-11T21:36:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f36741b75c0 0x7f36741b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 13 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 6 13 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 4 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 10 client.41708.objecter _send_command 13 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741bece0 con 0x7f36741b75c0 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f36741b75c0 0x7f36741b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f367359e700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f36741b75c0 0x7f36741b7a20 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f36741b75c0 2024-03-11T21:36:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741bece0 con 0x7f36741b75c0 2024-03-11T21:36:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:36:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 10 client.41708.objecter _finish_command 13 = system:0 2024-03-11T21:36:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 4 2024-03-11T21:36:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.121+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 6 13 2024-03-11T21:36:21.112 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 10 client.41708.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 14 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 14 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741a6ea0 osd=6 3 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 6 14 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 4 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 10 client.41708.objecter _send_command 14 2024-03-11T21:36:21.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36741bc3c0 con 0x7f36741b75c0 2024-03-11T21:36:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f36741bc3c0 con 0x7f36741b75c0 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 10 client.41708.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 4 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 6 14 2024-03-11T21:36:21.117 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:36:21.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 10 client.41708.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:21.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 15 2024-03-11T21:36:21.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f36741bca40 0x7f36741bee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:21.118 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 15 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 7 15 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 4 2024-03-11T21:36:21.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 10 client.41708.objecter _send_command 15 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36741c1d70 con 0x7f36741bca40 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f36741bca40 0x7f36741bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f366bfff700 10 client.41708.objecter ms_handle_connect 0x7f36741bca40 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.141+0000 7f3672d9d700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f36741bca40 0x7f36741bee30 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.145+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36741c1d70 con 0x7f36741bca40 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.145+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.145+0000 7f366bfff700 10 client.41708.objecter _finish_command 15 = system:0 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.145+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 4 2024-03-11T21:36:21.122 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.145+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 7 15 2024-03-11T21:36:21.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign -1 16 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter _get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 15 client.41708.objecter _session_command_op_remove -1 16 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter get_session s=0x7f36741bc3c0 osd=7 3 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 15 client.41708.objecter _session_command_op_assign 7 16 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 4 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter _send_command 16 2024-03-11T21:36:21.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36741c0b40 con 0x7f36741bca40 2024-03-11T21:36:21.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f366bfff700 1 -- 172.21.15.143:0/345486029 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f36741c0b40 con 0x7f36741bca40 2024-03-11T21:36:21.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f366bfff700 10 client.41708.objecter ms_dispatch 0x7f3674001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:21.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f366bfff700 10 client.41708.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:21.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f366bfff700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 4 2024-03-11T21:36:21.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f366bfff700 15 client.41708.objecter _session_command_op_remove 7 16 2024-03-11T21:36:21.138 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:36:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 librados: watch_flush enter 2024-03-11T21:36:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 librados: watch_flush exit 2024-03-11T21:36:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.0 2024-03-11T21:36:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f367410a420 msgr2=0x7f367410a800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f367410a420 0x7f367410a800 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674198f90 osd=0 1 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.1 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f367402c6b0 msgr2=0x7f367402ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f367402c6b0 0x7f367402ca90 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36740629f0 osd=1 1 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.2 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f36741a9c90 msgr2=0x7f36741ac080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f36741a9c90 0x7f36741ac080 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f3674062870 osd=2 1 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.3 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3674075190 msgr2=0x7f36741ae5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3674075190 0x7f36741ae5d0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a7120 osd=3 1 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.4 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3674075ae0 msgr2=0x7f36741b54b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3674075ae0 0x7f36741b54b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741aed60 osd=4 1 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.5 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3674074cf0 msgr2=0x7f36741b6b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3674074cf0 0x7f36741b6b70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6950 osd=5 1 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.6 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f36741b75c0 msgr2=0x7f36741b7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f36741b75c0 0x7f36741b7a20 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741a6ea0 osd=6 1 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter close_session for osd.7 2024-03-11T21:36:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f36741bca40 msgr2=0x7f36741bee30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f36741bca40 0x7f36741bee30 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter put_session s=0x7f36741bc3c0 osd=7 1 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 20 client.41708.objecter shutdown clearing up homeless session... 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 10 client.41708.objecter successfully canceled tick 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367359e700 1 -- 172.21.15.143:0/345486029 reap_dead start 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367359e700 1 -- 172.21.15.143:0/345486029 reap_dead start 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367359e700 1 -- 172.21.15.143:0/345486029 reap_dead start 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367359e700 1 -- 172.21.15.143:0/345486029 reap_dead start 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367359e700 1 -- 172.21.15.143:0/345486029 reap_dead start 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 msgr2=0x7f3660071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 0x7f3660071ce0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f3664009c10 tx=0x7f366400b040 comp rx=0 tx=0).stop 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 msgr2=0x7f367419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.161+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f365c006010 tx=0x7f365c0057c0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 shutdown_connections 2024-03-11T21:36:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3674102880 0x7f367419dde0 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 --2- 172.21.15.143:0/345486029 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f366006f820 0x7f3660071ce0 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 >> 172.21.15.143:0/345486029 conn(0x7f3674078eb0 msgr2=0x7f3674108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 shutdown_connections 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 -- 172.21.15.143:0/345486029 wait complete. 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:21.165+0000 7f367904f700 1 librados: shutdown 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:21.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:21.242 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:21.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.372 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:21.475 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:21.503 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:21.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:21.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:21.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.763 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:21.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:21.878 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.820686+0000 osd.2 (osd.2) 265 : cluster 0 5.8 scrub starts 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:19.822215+0000 osd.2 (osd.2) 266 : cluster 0 5.8 scrub ok 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:20.113879+0000 osd.1 (osd.1) 107 : cluster 0 5.c scrub starts 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:20.116300+0000 osd.1 (osd.1) 108 : cluster 0 5.c scrub ok 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:20.828579+0000 osd.5 (osd.5) 319 : cluster 0 3.5 scrub starts 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:20.831444+0000 osd.5 (osd.5) 320 : cluster 0 3.5 scrub ok 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: audit 2024-03-11T21:36:20.986945+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.143:0/345486029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.381331+0000 mon.a (mon.0) 2882 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.388433+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[21512]: audit 2024-03-11T21:36:21.606283+0000 mon.c (mon.2) 1002 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.820686+0000 osd.2 (osd.2) 265 : cluster 0 5.8 scrub starts 2024-03-11T21:36:21.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:19.822215+0000 osd.2 (osd.2) 266 : cluster 0 5.8 scrub ok 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:20.113879+0000 osd.1 (osd.1) 107 : cluster 0 5.c scrub starts 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:20.116300+0000 osd.1 (osd.1) 108 : cluster 0 5.c scrub ok 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:20.828579+0000 osd.5 (osd.5) 319 : cluster 0 3.5 scrub starts 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:20.831444+0000 osd.5 (osd.5) 320 : cluster 0 3.5 scrub ok 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: audit 2024-03-11T21:36:20.986945+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.143:0/345486029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.381331+0000 mon.a (mon.0) 2882 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.388433+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:36:21.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:21 smithi143 bash[29284]: audit 2024-03-11T21:36:21.606283+0000 mon.c (mon.2) 1002 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:21.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:21.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.820686+0000 osd.2 (osd.2) 265 : cluster 0 5.8 scrub starts 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:19.822215+0000 osd.2 (osd.2) 266 : cluster 0 5.8 scrub ok 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:20.113879+0000 osd.1 (osd.1) 107 : cluster 0 5.c scrub starts 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:20.116300+0000 osd.1 (osd.1) 108 : cluster 0 5.c scrub ok 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:20.828579+0000 osd.5 (osd.5) 319 : cluster 0 3.5 scrub starts 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:20.831444+0000 osd.5 (osd.5) 320 : cluster 0 3.5 scrub ok 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: audit 2024-03-11T21:36:20.986945+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.143:0/345486029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.381331+0000 mon.a (mon.0) 2882 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.388433+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:36:22.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:21 smithi203 bash[23654]: audit 2024-03-11T21:36:21.606283+0000 mon.c (mon.2) 1002 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:22.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:22.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:22.138 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:22.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:22.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.284 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:22.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:22.428 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:22.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:22.564 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:22.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.696 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:22.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:22 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.235888+0000 osd.4 (osd.4) 287 : cluster 0 6.1a scrub starts 2024-03-11T21:36:22.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:22 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.238264+0000 osd.4 (osd.4) 288 : cluster 0 6.1a scrub ok 2024-03-11T21:36:22.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:22 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.446340+0000 mgr.y (mgr.14550) 528 : cluster 0 pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-11T21:36:22.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:22 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.703441+0000 osd.3 (osd.3) 315 : cluster 0 118.7 scrub starts 2024-03-11T21:36:22.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:22 smithi203 bash[23654]: cluster 2024-03-11T21:36:21.706809+0000 osd.3 (osd.3) 316 : cluster 0 118.7 scrub ok 2024-03-11T21:36:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:22.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.235888+0000 osd.4 (osd.4) 287 : cluster 0 6.1a scrub starts 2024-03-11T21:36:22.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.238264+0000 osd.4 (osd.4) 288 : cluster 0 6.1a scrub ok 2024-03-11T21:36:22.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.446340+0000 mgr.y (mgr.14550) 528 : cluster 0 pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.703441+0000 osd.3 (osd.3) 315 : cluster 0 118.7 scrub starts 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[21512]: cluster 2024-03-11T21:36:21.706809+0000 osd.3 (osd.3) 316 : cluster 0 118.7 scrub ok 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.235888+0000 osd.4 (osd.4) 287 : cluster 0 6.1a scrub starts 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.238264+0000 osd.4 (osd.4) 288 : cluster 0 6.1a scrub ok 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.446340+0000 mgr.y (mgr.14550) 528 : cluster 0 pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.703441+0000 osd.3 (osd.3) 315 : cluster 0 118.7 scrub starts 2024-03-11T21:36:22.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:22 smithi143 bash[29284]: cluster 2024-03-11T21:36:21.706809+0000 osd.3 (osd.3) 316 : cluster 0 118.7 scrub ok 2024-03-11T21:36:22.962 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:23.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:23.093 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:23.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:23.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:23.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:23.481 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:23.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:23.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:23.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.154987+0000 osd.1 (osd.1) 109 : cluster 0 9.14 scrub starts 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.157401+0000 osd.1 (osd.1) 110 : cluster 0 9.14 scrub ok 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.426490+0000 osd.6 (osd.6) 237 : cluster 0 5.1f scrub starts 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.429559+0000 osd.6 (osd.6) 238 : cluster 0 5.1f scrub ok 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.896290+0000 osd.5 (osd.5) 321 : cluster 0 2.3 scrub starts 2024-03-11T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:23 smithi203 bash[23654]: cluster 2024-03-11T21:36:22.897999+0000 osd.5 (osd.5) 322 : cluster 0 2.3 scrub ok 2024-03-11T21:36:23.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:23.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:23.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.154987+0000 osd.1 (osd.1) 109 : cluster 0 9.14 scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.157401+0000 osd.1 (osd.1) 110 : cluster 0 9.14 scrub ok 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.426490+0000 osd.6 (osd.6) 237 : cluster 0 5.1f scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.429559+0000 osd.6 (osd.6) 238 : cluster 0 5.1f scrub ok 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.896290+0000 osd.5 (osd.5) 321 : cluster 0 2.3 scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[21512]: cluster 2024-03-11T21:36:22.897999+0000 osd.5 (osd.5) 322 : cluster 0 2.3 scrub ok 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.154987+0000 osd.1 (osd.1) 109 : cluster 0 9.14 scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.157401+0000 osd.1 (osd.1) 110 : cluster 0 9.14 scrub ok 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.426490+0000 osd.6 (osd.6) 237 : cluster 0 5.1f scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.429559+0000 osd.6 (osd.6) 238 : cluster 0 5.1f scrub ok 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.896290+0000 osd.5 (osd.5) 321 : cluster 0 2.3 scrub starts 2024-03-11T21:36:23.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:23 smithi143 bash[29284]: cluster 2024-03-11T21:36:22.897999+0000 osd.5 (osd.5) 322 : cluster 0 2.3 scrub ok 2024-03-11T21:36:23.872 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:23.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:24.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- 172.21.15.143:0/1060254793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc100d40 msgr2=0x7f4afc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- 172.21.15.143:0/1060254793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc100d40 0x7f4afc1011c0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0098f0 tx=0x7f4aec009c00 comp rx=0 tx=0).stop 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- 172.21.15.143:0/1060254793 shutdown_connections 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- 172.21.15.143:0/1060254793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc101700 0x7f4afc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- 172.21.15.143:0/1060254793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc106dc0 0x7f4afc1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- 172.21.15.143:0/1060254793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc100d40 0x7f4afc1011c0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- 172.21.15.143:0/1060254793 >> 172.21.15.143:0/1060254793 conn(0x7f4afc076d80 msgr2=0x7f4afc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- 172.21.15.143:0/1060254793 shutdown_connections 2024-03-11T21:36:24.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- 172.21.15.143:0/1060254793 wait complete. 2024-03-11T21:36:24.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 librados: starting msgr at 2024-03-11T21:36:24.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 librados: starting objecter 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 Processor -- start 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- start start 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 librados: setting wanted keys 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 librados: calling monclient init 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 0x7f4afc19d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc101700 0x7f4afc19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc110570 con 0x7f4afc101700 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc1103f0 con 0x7f4afc100d40 2024-03-11T21:36:24.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b03265700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc1106f0 con 0x7f4afc106dc0 2024-03-11T21:36:24.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b00800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc101700 0x7f4afc19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37918/0 (socket says 172.21.15.143:37918) 2024-03-11T21:36:24.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 -- 172.21.15.143:0/4069736529 learned_addr learned my addr 172.21.15.143:0/4069736529 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:24.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01001700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 0x7f4afc19d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 -- 172.21.15.143:0/4069736529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 msgr2=0x7f4afc19d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 0x7f4afc19d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 -- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc101700 msgr2=0x7f4afc19dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc101700 0x7f4afc19dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.057+0000 7f4b01802700 1 -- 172.21.15.143:0/4069736529 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc198100 con 0x7f4afc106dc0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b01001700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 0x7f4afc19d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b01802700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f4af400e3c0 tx=0x7f4af400e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 10 client.?.objecter ms_handle_connect 0x7f4afc106dc0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af40092d0 con 0x7f4afc106dc0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4af400ee70 con 0x7f4afc106dc0 2024-03-11T21:36:24.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc1983f0 con 0x7f4afc106dc0 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4007d40 con 0x7f4afc106dc0 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b03265700 10 client.41714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc1a46b0 con 0x7f4afc106dc0 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b03265700 1 librados: init done 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4afc04faa0 con 0x7f4afc106dc0 2024-03-11T21:36:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4af4012050 con 0x7f4afc106dc0 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 0x7f4ae8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(524..524 src has 87..524) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f4af408e610 con 0x7f4afc106dc0 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 10 client.41714.objecter ms_dispatch 0x7f4afc001550 osd_map(524..524 src has 87..524) v4 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 3 client.41714.objecter handle_osd_map got epochs [524,524] > 0 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 3 client.41714.objecter handle_osd_map decoding full epoch 524 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4b01001700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 0x7f4ae8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.061+0000 7f4af27fc700 20 client.41714.objecter dump_active .. 0 homeless 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.065+0000 7f4af27fc700 10 client.41714.objecter ms_handle_connect 0x7f4ae806f820 2024-03-11T21:36:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.065+0000 7f4b01001700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 0x7f4ae8071ce0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005950 tx=0x7f4af80058e0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.069+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4af4022060 con 0x7f4afc106dc0 2024-03-11T21:36:24.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.126 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/1834848337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04101700 msgr2=0x7fcd0410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/1834848337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04101700 0x7fcd0410db70 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00b0a0 tx=0x7fccfc00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/1834848337 shutdown_connections 2024-03-11T21:36:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/1834848337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04101700 0x7fcd0410db70 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/1834848337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 0x7fcd041071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/1834848337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04100d40 0x7fcd041011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/1834848337 >> 172.21.15.143:0/1834848337 conn(0x7fcd04076d80 msgr2=0x7fcd04077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/1834848337 shutdown_connections 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/1834848337 wait complete. 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 librados: starting msgr at 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 librados: starting objecter 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 Processor -- start 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- start start 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 librados: setting wanted keys 2024-03-11T21:36:24.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 librados: calling monclient init 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 0x7fcd0419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 0x7fcd0419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 0x7fcd04197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04110570 con 0x7fcd04101700 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd041103f0 con 0x7fcd04106dc0 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd041106f0 con 0x7fcd04100d40 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd03fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 0x7fcd0419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 0x7fcd0419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd03fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 0x7fcd0419d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37940/0 (socket says 172.21.15.143:37940) 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd03fff700 1 -- 172.21.15.143:0/247092816 learned_addr learned my addr 172.21.15.143:0/247092816 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd08a90700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 0x7fcd04197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 -- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 msgr2=0x7fcd0419d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.140 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 0x7fcd0419d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 -- 172.21.15.143:0/247092816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 msgr2=0x7fcd04197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 0x7fcd04197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 -- 172.21.15.143:0/247092816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd04198060 con 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd037fe700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 0x7fcd0419dc80 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd017fa700 10 client.?.objecter ms_handle_connect 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd017fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf800cb70 con 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fccf80044a0 con 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd04198350 con 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8003cb0 con 0x7fcd04101700 2024-03-11T21:36:24.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 10 client.30519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:24.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd041a4610 con 0x7fcd04101700 2024-03-11T21:36:24.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.161+0000 7fcd0a4f3700 1 librados: init done 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0404faa0 con 0x7fcd04101700 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fccf8003e70 con 0x7fcd04101700 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 0x7fccec071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(524..524 src has 87..524) v4 ==== 9629+0+0 (secure 0 0 0) 0x7fccf808ed30 con 0x7fcd04101700 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 10 client.30519.objecter ms_dispatch 0x7fcd04001550 osd_map(524..524 src has 87..524) v4 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 3 client.30519.objecter handle_osd_map got epochs [524,524] > 0 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 3 client.30519.objecter handle_osd_map decoding full epoch 524 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd03fff700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 0x7fccec071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:24.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 20 client.30519.objecter dump_active .. 0 homeless 2024-03-11T21:36:24.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd017fa700 10 client.30519.objecter ms_handle_connect 0x7fccec06f820 2024-03-11T21:36:24.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.165+0000 7fcd03fff700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 0x7fccec071ce0 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7fcd04198a40 tx=0x7fccf400b500 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:24.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.169+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fccf805b980 con 0x7fcd04101700 2024-03-11T21:36:24.154 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:24.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:24.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.313+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f4afc1a48e0 con 0x7f4afc106dc0 2024-03-11T21:36:24.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.389+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd04105b10 con 0x7fccec06f820 2024-03-11T21:36:24.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:24.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.417+0000 7fcd017fa700 1 -- 172.21.15.143:0/247092816 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+665403 (secure 0 0 0) 0x7fcd04105b10 con 0x7fccec06f820 2024-03-11T21:36:24.407 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:36:24.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 10 librados: watch_flush enter 2024-03-11T21:36:24.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 10 librados: watch_flush exit 2024-03-11T21:36:24.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 20 client.30519.objecter shutdown clearing up homeless session... 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 10 client.30519.objecter successfully canceled tick 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 msgr2=0x7fccec071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 0x7fccec071ce0 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7fcd04198a40 tx=0x7fccf400b500 comp rx=0 tx=0).stop 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 msgr2=0x7fcd0419dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 0x7fcd0419dc80 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800ed00 comp rx=0 tx=0).stop 2024-03-11T21:36:24.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 shutdown_connections 2024-03-11T21:36:24.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fccec06f820 0x7fccec071ce0 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd04106dc0 0x7fcd04197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd04100d40 0x7fcd0419d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 --2- 172.21.15.143:0/247092816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd04101700 0x7fcd0419dc80 secure :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800ed00 comp rx=0 tx=0).stop 2024-03-11T21:36:24.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 >> 172.21.15.143:0/247092816 conn(0x7fcd04076d80 msgr2=0x7fcd04069dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:24.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 shutdown_connections 2024-03-11T21:36:24.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 -- 172.21.15.143:0/247092816 wait complete. 2024-03-11T21:36:24.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.421+0000 7fcd0a4f3700 1 librados: shutdown 2024-03-11T21:36:24.420 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:36:24.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:24.510 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:24.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:24.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.709+0000 7f4af27fc700 1 -- 172.21.15.143:0/4069736529 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v525) v1 ==== 101+0+0 (secure 0 0 0) 0x7f4af4010040 con 0x7f4afc106dc0 2024-03-11T21:36:24.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 10 librados: watch_flush enter 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 10 librados: watch_flush exit 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 20 client.41714.objecter shutdown clearing up homeless session... 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 10 client.41714.objecter successfully canceled tick 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 msgr2=0x7f4ae8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 0x7f4ae8071ce0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005950 tx=0x7f4af80058e0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 msgr2=0x7f4afc1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f4af400e3c0 tx=0x7f4af400e6d0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 shutdown_connections 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ae806f820 0x7f4ae8071ce0 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4afc106dc0 0x7f4afc1979a0 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4afc100d40 0x7f4afc19d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 --2- 172.21.15.143:0/4069736529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4afc101700 0x7f4afc19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:24.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 >> 172.21.15.143:0/4069736529 conn(0x7f4afc076d80 msgr2=0x7f4afc069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:24.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 shutdown_connections 2024-03-11T21:36:24.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 -- 172.21.15.143:0/4069736529 wait complete. 2024-03-11T21:36:24.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:24.713+0000 7f4b03265700 1 librados: shutdown 2024-03-11T21:36:24.689 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:36:24.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:24.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: cluster 2024-03-11T21:36:23.448463+0000 mgr.y (mgr.14550) 529 : cluster 0 pgmap v803: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 144 B/s wr, 3 op/s 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:23.700048+0000 mon.c (mon.2) 1003 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:23.700395+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:23.700985+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:23.701263+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: cluster 2024-03-11T21:36:23.723998+0000 osd.3 (osd.3) 317 : cluster 0 2.19 scrub starts 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: cluster 2024-03-11T21:36:23.725646+0000 osd.3 (osd.3) 318 : cluster 0 2.19 scrub ok 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:24.317229+0000 mon.c (mon.2) 1005 : audit 1 from='client.? 172.21.15.143:0/4069736529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:24 smithi203 bash[23654]: audit 2024-03-11T21:36:24.317595+0000 mon.a (mon.0) 2886 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:24.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:24.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: cluster 2024-03-11T21:36:23.448463+0000 mgr.y (mgr.14550) 529 : cluster 0 pgmap v803: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 144 B/s wr, 3 op/s 2024-03-11T21:36:24.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:23.700048+0000 mon.c (mon.2) 1003 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:23.700395+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:23.700985+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:23.701263+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: cluster 2024-03-11T21:36:23.723998+0000 osd.3 (osd.3) 317 : cluster 0 2.19 scrub starts 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: cluster 2024-03-11T21:36:23.725646+0000 osd.3 (osd.3) 318 : cluster 0 2.19 scrub ok 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:24.317229+0000 mon.c (mon.2) 1005 : audit 1 from='client.? 172.21.15.143:0/4069736529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[29284]: audit 2024-03-11T21:36:24.317595+0000 mon.a (mon.0) 2886 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: cluster 2024-03-11T21:36:23.448463+0000 mgr.y (mgr.14550) 529 : cluster 0 pgmap v803: 260 pgs: 260 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 144 B/s wr, 3 op/s 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:23.700048+0000 mon.c (mon.2) 1003 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:23.700395+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:23.700985+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:23.701263+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-73"}]: dispatch 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: cluster 2024-03-11T21:36:23.723998+0000 osd.3 (osd.3) 317 : cluster 0 2.19 scrub starts 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: cluster 2024-03-11T21:36:23.725646+0000 osd.3 (osd.3) 318 : cluster 0 2.19 scrub ok 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:24.317229+0000 mon.c (mon.2) 1005 : audit 1 from='client.? 172.21.15.143:0/4069736529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:24 smithi143 bash[21512]: audit 2024-03-11T21:36:24.317595+0000 mon.a (mon.0) 2886 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:24.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:25.019 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:25.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:25.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:25.248 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.276 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:25.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:25.408 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:25.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:25.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:25.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:25.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:36:25.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.179691+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.181482+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.218194+0000 osd.0 (osd.0) 303 : cluster 0 2.1f scrub starts 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.220017+0000 osd.0 (osd.0) 304 : cluster 0 2.1f scrub ok 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: audit 2024-03-11T21:36:24.393751+0000 mgr.y (mgr.14550) 530 : audit 0 from='client.30519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.695720+0000 mon.a (mon.0) 2887 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:25.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: audit 2024-03-11T21:36:24.704414+0000 mon.a (mon.0) 2888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:25.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.708141+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:36:25.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.743927+0000 osd.3 (osd.3) 319 : cluster 0 5.14 scrub starts 2024-03-11T21:36:25.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.749200+0000 osd.3 (osd.3) 320 : cluster 0 5.14 scrub ok 2024-03-11T21:36:25.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:25 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.811487+0000 osd.2 (osd.2) 267 : cluster 0 5.a scrub starts 2024-03-11T21:36:25.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:25.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.179691+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:36:25.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.181482+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:36:25.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.218194+0000 osd.0 (osd.0) 303 : cluster 0 2.1f scrub starts 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.220017+0000 osd.0 (osd.0) 304 : cluster 0 2.1f scrub ok 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: audit 2024-03-11T21:36:24.393751+0000 mgr.y (mgr.14550) 530 : audit 0 from='client.30519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.695720+0000 mon.a (mon.0) 2887 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: audit 2024-03-11T21:36:24.704414+0000 mon.a (mon.0) 2888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.708141+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.743927+0000 osd.3 (osd.3) 319 : cluster 0 5.14 scrub starts 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.749200+0000 osd.3 (osd.3) 320 : cluster 0 5.14 scrub ok 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.811487+0000 osd.2 (osd.2) 267 : cluster 0 5.a scrub starts 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.179691+0000 osd.1 (osd.1) 111 : cluster 0 4.2 scrub starts 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.181482+0000 osd.1 (osd.1) 112 : cluster 0 4.2 scrub ok 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.218194+0000 osd.0 (osd.0) 303 : cluster 0 2.1f scrub starts 2024-03-11T21:36:25.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.220017+0000 osd.0 (osd.0) 304 : cluster 0 2.1f scrub ok 2024-03-11T21:36:25.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: audit 2024-03-11T21:36:24.393751+0000 mgr.y (mgr.14550) 530 : audit 0 from='client.30519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:36:25.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:26.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:26.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:26.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:26.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.695720+0000 mon.a (mon.0) 2887 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:26.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: audit 2024-03-11T21:36:24.704414+0000 mon.a (mon.0) 2888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:26.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.708141+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:36:26.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.743927+0000 osd.3 (osd.3) 319 : cluster 0 5.14 scrub starts 2024-03-11T21:36:26.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.749200+0000 osd.3 (osd.3) 320 : cluster 0 5.14 scrub ok 2024-03-11T21:36:26.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:25 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.811487+0000 osd.2 (osd.2) 267 : cluster 0 5.a scrub starts 2024-03-11T21:36:26.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:26.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:26.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:26.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:26.451 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:26.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.583 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:26.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:26.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.847 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:26.863 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-03-11T21:36:23.446606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792309,"num_objects":372,"num_object_clones":7,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":2283,"num_read_kb":2024,"num_write":1538,"num_write_kb":15181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2955,"ondisk_log_size":2955,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":780,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":602512,"kb_used_data":95268,"kb_used_omap":38,"kb_used_meta":507033,"kb_avail":749358704,"statfs":{"total":767960285184,"available":767343312896,"internally_reserved":0,"allocated":97554432,"data_stored":69514703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":519202229},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.062792"},"pg_stats":[{"pgid":"292.1f","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417174+0000","last_change":"2024-03-11T21:36:11.626523+0000","last_active":"2024-03-11T21:36:21.417174+0000","last_peered":"2024-03-11T21:36:21.417174+0000","last_clean":"2024-03-11T21:36:21.417174+0000","last_became_active":"2024-03-11T21:36:11.626091+0000","last_became_peered":"2024-03-11T21:36:11.626091+0000","last_unstale":"2024-03-11T21:36:21.417174+0000","last_undegraded":"2024-03-11T21:36:21.417174+0000","last_fullsized":"2024-03-11T21:36:21.417174+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:11.013378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"292.1e","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.384930+0000","last_change":"2024-03-11T21:36:11.638045+0000","last_active":"2024-03-11T21:36:21.384930+0000","last_peered":"2024-03-11T21:36:21.384930+0000","last_clean":"2024-03-11T21:36:21.384930+0000","last_became_active":"2024-03-11T21:36:11.637478+0000","last_became_peered":"2024-03-11T21:36:11.637478+0000","last_unstale":"2024-03-11T21:36:21.384930+0000","last_undegraded":"2024-03-11T21:36:21.384930+0000","last_fullsized":"2024-03-11T21:36:21.384930+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.968151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"292.1d","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418614+0000","last_change":"2024-03-11T21:36:11.640212+0000","last_active":"2024-03-11T21:36:21.418614+0000","last_peered":"2024-03-11T21:36:21.418614+0000","last_clean":"2024-03-11T21:36:21.418614+0000","last_became_active":"2024-03-11T21:36:11.639909+0000","last_became_peered":"2024-03-11T21:36:11.639909+0000","last_unstale":"2024-03-11T21:36:21.418614+0000","last_undegraded":"2024-03-11T21:36:21.418614+0000","last_fullsized":"2024-03-11T21:36:21.418614+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:11.632006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"292.1c","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415756+0000","last_change":"2024-03-11T21:36:11.640915+0000","last_active":"2024-03-11T21:36:21.415756+0000","last_peered":"2024-03-11T21:36:21.415756+0000","last_clean":"2024-03-11T21:36:21.415756+0000","last_became_active":"2024-03-11T21:36:11.640438+0000","last_became_peered":"2024-03-11T21:36:11.640438+0000","last_unstale":"2024-03-11T21:36:21.415756+0000","last_undegraded":"2024-03-11T21:36:21.415756+0000","last_fullsized":"2024-03-11T21:36:21.415756+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:19.955424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"292.1b","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.382969+0000","last_change":"2024-03-11T21:36:11.634177+0000","last_active":"2024-03-11T21:36:21.382969+0000","last_peered":"2024-03-11T21:36:21.382969+0000","last_clean":"2024-03-11T21:36:21.382969+0000","last_became_active":"2024-03-11T21:36:11.632781+0000","last_became_peered":"2024-03-11T21:36:11.632781+0000","last_unstale":"2024-03-11T21:36:21.382969+0000","last_undegraded":"2024-03-11T21:36:21.382969+0000","last_fullsized":"2024-03-11T21:36:21.382969+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.732262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"292.1a","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420782+0000","last_change":"2024-03-11T21:36:11.631175+0000","last_active":"2024-03-11T21:36:21.420782+0000","last_peered":"2024-03-11T21:36:21.420782+0000","last_clean":"2024-03-11T21:36:21.420782+0000","last_became_active":"2024-03-11T21:36:11.630845+0000","last_became_peered":"2024-03-11T21:36:11.630845+0000","last_unstale":"2024-03-11T21:36:21.420782+0000","last_undegraded":"2024-03-11T21:36:21.420782+0000","last_fullsized":"2024-03-11T21:36:21.420782+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.565528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"292.19","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414119+0000","last_change":"2024-03-11T21:36:11.635502+0000","last_active":"2024-03-11T21:36:21.414119+0000","last_peered":"2024-03-11T21:36:21.414119+0000","last_clean":"2024-03-11T21:36:21.414119+0000","last_became_active":"2024-03-11T21:36:11.634436+0000","last_became_peered":"2024-03-11T21:36:11.634436+0000","last_unstale":"2024-03-11T21:36:21.414119+0000","last_undegraded":"2024-03-11T21:36:21.414119+0000","last_fullsized":"2024-03-11T21:36:21.414119+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.735153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"292.18","version":"521'2","reported_seq":20,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608063+0000","last_change":"2024-03-11T21:36:11.641452+0000","last_active":"2024-03-11T21:36:16.608063+0000","last_peered":"2024-03-11T21:36:16.608063+0000","last_clean":"2024-03-11T21:36:16.608063+0000","last_became_active":"2024-03-11T21:36:11.641113+0000","last_became_peered":"2024-03-11T21:36:11.641113+0000","last_unstale":"2024-03-11T21:36:16.608063+0000","last_undegraded":"2024-03-11T21:36:16.608063+0000","last_fullsized":"2024-03-11T21:36:16.608063+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.852038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"292.17","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418946+0000","last_change":"2024-03-11T21:36:11.633243+0000","last_active":"2024-03-11T21:36:21.418946+0000","last_peered":"2024-03-11T21:36:21.418946+0000","last_clean":"2024-03-11T21:36:21.418946+0000","last_became_active":"2024-03-11T21:36:11.632928+0000","last_became_peered":"2024-03-11T21:36:11.632928+0000","last_unstale":"2024-03-11T21:36:21.418946+0000","last_undegraded":"2024-03-11T21:36:21.418946+0000","last_fullsized":"2024-03-11T21:36:21.418946+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.081222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"292.16","version":"0'0","reported_seq":17,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610000+0000","last_change":"2024-03-11T21:36:11.635816+0000","last_active":"2024-03-11T21:36:16.610000+0000","last_peered":"2024-03-11T21:36:16.610000+0000","last_clean":"2024-03-11T21:36:16.610000+0000","last_became_active":"2024-03-11T21:36:11.635505+0000","last_became_peered":"2024-03-11T21:36:11.635505+0000","last_unstale":"2024-03-11T21:36:16.610000+0000","last_undegraded":"2024-03-11T21:36:16.610000+0000","last_fullsized":"2024-03-11T21:36:16.610000+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.285863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"292.15","version":"521'2","reported_seq":24,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414279+0000","last_change":"2024-03-11T21:36:11.618443+0000","last_active":"2024-03-11T21:36:21.414279+0000","last_peered":"2024-03-11T21:36:21.414279+0000","last_clean":"2024-03-11T21:36:21.414279+0000","last_became_active":"2024-03-11T21:36:11.618056+0000","last_became_peered":"2024-03-11T21:36:11.618056+0000","last_unstale":"2024-03-11T21:36:21.414279+0000","last_undegraded":"2024-03-11T21:36:21.414279+0000","last_fullsized":"2024-03-11T21:36:21.414279+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.806695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"292.14","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417979+0000","last_change":"2024-03-11T21:36:11.630904+0000","last_active":"2024-03-11T21:36:21.417979+0000","last_peered":"2024-03-11T21:36:21.417979+0000","last_clean":"2024-03-11T21:36:21.417979+0000","last_became_active":"2024-03-11T21:36:11.630600+0000","last_became_peered":"2024-03-11T21:36:11.630600+0000","last_unstale":"2024-03-11T21:36:21.417979+0000","last_undegraded":"2024-03-11T21:36:21.417979+0000","last_fullsized":"2024-03-11T21:36:21.417979+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.288419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"292.13","version":"0'0","reported_seq":19,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672307+0000","last_change":"2024-03-11T21:36:11.630852+0000","last_active":"2024-03-11T21:36:18.672307+0000","last_peered":"2024-03-11T21:36:18.672307+0000","last_clean":"2024-03-11T21:36:18.672307+0000","last_became_active":"2024-03-11T21:36:11.630486+0000","last_became_peered":"2024-03-11T21:36:11.630486+0000","last_unstale":"2024-03-11T21:36:18.672307+0000","last_undegraded":"2024-03-11T21:36:18.672307+0000","last_fullsized":"2024-03-11T21:36:18.672307+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.735253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"292.12","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421062+0000","last_change":"2024-03-11T21:36:11.631248+0000","last_active":"2024-03-11T21:36:21.421062+0000","last_peered":"2024-03-11T21:36:21.421062+0000","last_clean":"2024-03-11T21:36:21.421062+0000","last_became_active":"2024-03-11T21:36:11.630166+0000","last_became_peered":"2024-03-11T21:36:11.630166+0000","last_unstale":"2024-03-11T21:36:21.421062+0000","last_undegraded":"2024-03-11T21:36:21.421062+0000","last_fullsized":"2024-03-11T21:36:21.421062+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:19.256213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"292.11","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419942+0000","last_change":"2024-03-11T21:36:11.625138+0000","last_active":"2024-03-11T21:36:21.419942+0000","last_peered":"2024-03-11T21:36:21.419942+0000","last_clean":"2024-03-11T21:36:21.419942+0000","last_became_active":"2024-03-11T21:36:11.623599+0000","last_became_peered":"2024-03-11T21:36:11.623599+0000","last_unstale":"2024-03-11T21:36:21.419942+0000","last_undegraded":"2024-03-11T21:36:21.419942+0000","last_fullsized":"2024-03-11T21:36:21.419942+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.295535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.10","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419840+0000","last_change":"2024-03-11T21:36:11.629111+0000","last_active":"2024-03-11T21:36:21.419840+0000","last_peered":"2024-03-11T21:36:21.419840+0000","last_clean":"2024-03-11T21:36:21.419840+0000","last_became_active":"2024-03-11T21:36:11.628759+0000","last_became_peered":"2024-03-11T21:36:11.628759+0000","last_unstale":"2024-03-11T21:36:21.419840+0000","last_undegraded":"2024-03-11T21:36:21.419840+0000","last_fullsized":"2024-03-11T21:36:21.419840+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.430311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.f","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421035+0000","last_change":"2024-03-11T21:36:11.628958+0000","last_active":"2024-03-11T21:36:21.421035+0000","last_peered":"2024-03-11T21:36:21.421035+0000","last_clean":"2024-03-11T21:36:21.421035+0000","last_became_active":"2024-03-11T21:36:11.628520+0000","last_became_peered":"2024-03-11T21:36:11.628520+0000","last_unstale":"2024-03-11T21:36:21.421035+0000","last_undegraded":"2024-03-11T21:36:21.421035+0000","last_fullsized":"2024-03-11T21:36:21.421035+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.011344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.e","version":"0'0","reported_seq":17,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610208+0000","last_change":"2024-03-11T21:36:11.641479+0000","last_active":"2024-03-11T21:36:16.610208+0000","last_peered":"2024-03-11T21:36:16.610208+0000","last_clean":"2024-03-11T21:36:16.610208+0000","last_became_active":"2024-03-11T21:36:11.641142+0000","last_became_peered":"2024-03-11T21:36:11.641142+0000","last_unstale":"2024-03-11T21:36:16.610208+0000","last_undegraded":"2024-03-11T21:36:16.610208+0000","last_fullsized":"2024-03-11T21:36:16.610208+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.171960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"292.d","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414444+0000","last_change":"2024-03-11T21:36:11.632403+0000","last_active":"2024-03-11T21:36:21.414444+0000","last_peered":"2024-03-11T21:36:21.414444+0000","last_clean":"2024-03-11T21:36:21.414444+0000","last_became_active":"2024-03-11T21:36:11.632069+0000","last_became_peered":"2024-03-11T21:36:11.632069+0000","last_unstale":"2024-03-11T21:36:21.414444+0000","last_undegraded":"2024-03-11T21:36:21.414444+0000","last_fullsized":"2024-03-11T21:36:21.414444+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:39.733061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"292.c","version":"521'2","reported_seq":24,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420663+0000","last_change":"2024-03-11T21:36:11.627180+0000","last_active":"2024-03-11T21:36:21.420663+0000","last_peered":"2024-03-11T21:36:21.420663+0000","last_clean":"2024-03-11T21:36:21.420663+0000","last_became_active":"2024-03-11T21:36:11.626862+0000","last_became_peered":"2024-03-11T21:36:11.626862+0000","last_unstale":"2024-03-11T21:36:21.420663+0000","last_undegraded":"2024-03-11T21:36:21.420663+0000","last_fullsized":"2024-03-11T21:36:21.420663+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.904655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.b","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415419+0000","last_change":"2024-03-11T21:36:11.634969+0000","last_active":"2024-03-11T21:36:21.415419+0000","last_peered":"2024-03-11T21:36:21.415419+0000","last_clean":"2024-03-11T21:36:21.415419+0000","last_became_active":"2024-03-11T21:36:11.634322+0000","last_became_peered":"2024-03-11T21:36:11.634322+0000","last_unstale":"2024-03-11T21:36:21.415419+0000","last_undegraded":"2024-03-11T21:36:21.415419+0000","last_fullsized":"2024-03-11T21:36:21.415419+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.598779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"292.a","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414736+0000","last_change":"2024-03-11T21:36:11.639374+0000","last_active":"2024-03-11T21:36:21.414736+0000","last_peered":"2024-03-11T21:36:21.414736+0000","last_clean":"2024-03-11T21:36:21.414736+0000","last_became_active":"2024-03-11T21:36:11.639070+0000","last_became_peered":"2024-03-11T21:36:11.639070+0000","last_unstale":"2024-03-11T21:36:21.414736+0000","last_undegraded":"2024-03-11T21:36:21.414736+0000","last_fullsized":"2024-03-11T21:36:21.414736+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:28.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"292.9","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.413783+0000","last_change":"2024-03-11T21:36:11.618444+0000","last_active":"2024-03-11T21:36:21.413783+0000","last_peered":"2024-03-11T21:36:21.413783+0000","last_clean":"2024-03-11T21:36:21.413783+0000","last_became_active":"2024-03-11T21:36:11.618191+0000","last_became_peered":"2024-03-11T21:36:11.618191+0000","last_unstale":"2024-03-11T21:36:21.413783+0000","last_undegraded":"2024-03-11T21:36:21.413783+0000","last_fullsized":"2024-03-11T21:36:21.413783+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"292.8","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420424+0000","last_change":"2024-03-11T21:36:11.625960+0000","last_active":"2024-03-11T21:36:21.420424+0000","last_peered":"2024-03-11T21:36:21.420424+0000","last_clean":"2024-03-11T21:36:21.420424+0000","last_became_active":"2024-03-11T21:36:11.625652+0000","last_became_peered":"2024-03-11T21:36:11.625652+0000","last_unstale":"2024-03-11T21:36:21.420424+0000","last_undegraded":"2024-03-11T21:36:21.420424+0000","last_fullsized":"2024-03-11T21:36:21.420424+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:12.655565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"292.7","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419023+0000","last_change":"2024-03-11T21:36:11.633872+0000","last_active":"2024-03-11T21:36:21.419023+0000","last_peered":"2024-03-11T21:36:21.419023+0000","last_clean":"2024-03-11T21:36:21.419023+0000","last_became_active":"2024-03-11T21:36:11.633067+0000","last_became_peered":"2024-03-11T21:36:11.633067+0000","last_unstale":"2024-03-11T21:36:21.419023+0000","last_undegraded":"2024-03-11T21:36:21.419023+0000","last_fullsized":"2024-03-11T21:36:21.419023+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:28.092585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"292.6","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421097+0000","last_change":"2024-03-11T21:36:11.634655+0000","last_active":"2024-03-11T21:36:21.421097+0000","last_peered":"2024-03-11T21:36:21.421097+0000","last_clean":"2024-03-11T21:36:21.421097+0000","last_became_active":"2024-03-11T21:36:11.634306+0000","last_became_peered":"2024-03-11T21:36:11.634306+0000","last_unstale":"2024-03-11T21:36:21.421097+0000","last_undegraded":"2024-03-11T21:36:21.421097+0000","last_fullsized":"2024-03-11T21:36:21.421097+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:15.331073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.5","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421393+0000","last_change":"2024-03-11T21:36:11.639236+0000","last_active":"2024-03-11T21:36:21.421393+0000","last_peered":"2024-03-11T21:36:21.421393+0000","last_clean":"2024-03-11T21:36:21.421393+0000","last_became_active":"2024-03-11T21:36:11.638882+0000","last_became_peered":"2024-03-11T21:36:11.638882+0000","last_unstale":"2024-03-11T21:36:21.421393+0000","last_undegraded":"2024-03-11T21:36:21.421393+0000","last_fullsized":"2024-03-11T21:36:21.421393+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.424409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.4","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415935+0000","last_change":"2024-03-11T21:36:11.638008+0000","last_active":"2024-03-11T21:36:21.415935+0000","last_peered":"2024-03-11T21:36:21.415935+0000","last_clean":"2024-03-11T21:36:21.415935+0000","last_became_active":"2024-03-11T21:36:11.637653+0000","last_became_peered":"2024-03-11T21:36:11.637653+0000","last_unstale":"2024-03-11T21:36:21.415935+0000","last_undegraded":"2024-03-11T21:36:21.415935+0000","last_fullsized":"2024-03-11T21:36:21.415935+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.455955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"292.3","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421920+0000","last_change":"2024-03-11T21:36:11.629279+0000","last_active":"2024-03-11T21:36:21.421920+0000","last_peered":"2024-03-11T21:36:21.421920+0000","last_clean":"2024-03-11T21:36:21.421920+0000","last_became_active":"2024-03-11T21:36:11.628842+0000","last_became_peered":"2024-03-11T21:36:11.628842+0000","last_unstale":"2024-03-11T21:36:21.421920+0000","last_undegraded":"2024-03-11T21:36:21.421920+0000","last_fullsized":"2024-03-11T21:36:21.421920+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.070467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"292.2","version":"0'0","reported_seq":21,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420081+0000","last_change":"2024-03-11T21:36:11.628183+0000","last_active":"2024-03-11T21:36:21.420081+0000","last_peered":"2024-03-11T21:36:21.420081+0000","last_clean":"2024-03-11T21:36:21.420081+0000","last_became_active":"2024-03-11T21:36:11.627837+0000","last_became_peered":"2024-03-11T21:36:11.627837+0000","last_unstale":"2024-03-11T21:36:21.420081+0000","last_undegraded":"2024-03-11T21:36:21.420081+0000","last_fullsized":"2024-03-11T21:36:21.420081+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.050509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.1","version":"521'4","reported_seq":25,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420050+0000","last_change":"2024-03-11T21:36:11.628346+0000","last_active":"2024-03-11T21:36:21.420050+0000","last_peered":"2024-03-11T21:36:21.420050+0000","last_clean":"2024-03-11T21:36:21.420050+0000","last_became_active":"2024-03-11T21:36:11.627915+0000","last_became_peered":"2024-03-11T21:36:11.627915+0000","last_unstale":"2024-03-11T21:36:21.420050+0000","last_undegraded":"2024-03-11T21:36:21.420050+0000","last_fullsized":"2024-03-11T21:36:21.420050+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:31.166295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"292.0","version":"0'0","reported_seq":19,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.671185+0000","last_change":"2024-03-11T21:36:11.628257+0000","last_active":"2024-03-11T21:36:18.671185+0000","last_peered":"2024-03-11T21:36:18.671185+0000","last_clean":"2024-03-11T21:36:18.671185+0000","last_became_active":"2024-03-11T21:36:11.627932+0000","last_became_peered":"2024-03-11T21:36:11.627932+0000","last_unstale":"2024-03-11T21:36:18.671185+0000","last_undegraded":"2024-03-11T21:36:18.671185+0000","last_fullsized":"2024-03-11T21:36:18.671185+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":518,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:10.589924+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.589924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.141621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.9","version":"482'18","reported_seq":955,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420089+0000","last_change":"2024-03-11T21:35:26.443255+0000","last_active":"2024-03-11T21:36:21.420089+0000","last_peered":"2024-03-11T21:36:21.420089+0000","last_clean":"2024-03-11T21:36:21.420089+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:36:21.420089+0000","last_undegraded":"2024-03-11T21:36:21.420089+0000","last_fullsized":"2024-03-11T21:36:21.420089+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'18","last_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_deep_scrub":"482'18","last_deep_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.441729+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.817126+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0010370220000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"461'24","reported_seq":938,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.382036+0000","last_change":"2024-03-11T21:35:26.450518+0000","last_active":"2024-03-11T21:36:21.382036+0000","last_peered":"2024-03-11T21:36:21.382036+0000","last_clean":"2024-03-11T21:36:21.382036+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:36:21.382036+0000","last_undegraded":"2024-03-11T21:36:21.382036+0000","last_fullsized":"2024-03-11T21:36:21.382036+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'24","last_scrub_stamp":"2024-03-11T21:34:54.761329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.761329+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041612760000000002,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"464'40","reported_seq":955,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417823+0000","last_change":"2024-03-11T21:35:26.453000+0000","last_active":"2024-03-11T21:36:21.417823+0000","last_peered":"2024-03-11T21:36:21.417823+0000","last_clean":"2024-03-11T21:36:21.417823+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:36:21.417823+0000","last_undegraded":"2024-03-11T21:36:21.417823+0000","last_fullsized":"2024-03-11T21:36:21.417823+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'40","last_scrub_stamp":"2024-03-11T21:34:55.080483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.080483+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014521345999999999,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"444'55","reported_seq":986,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416244+0000","last_change":"2024-03-11T21:36:06.267240+0000","last_active":"2024-03-11T21:36:21.416244+0000","last_peered":"2024-03-11T21:36:21.416244+0000","last_clean":"2024-03-11T21:36:21.416244+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:36:21.416244+0000","last_undegraded":"2024-03-11T21:36:21.416244+0000","last_fullsized":"2024-03-11T21:36:21.416244+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'55","last_scrub_stamp":"2024-03-11T21:36:06.267048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.267048+0000","objects_scrubbed":7,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:20.014883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014549960000000001,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"412'52","reported_seq":981,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421757+0000","last_change":"2024-03-11T21:35:26.444186+0000","last_active":"2024-03-11T21:36:21.421757+0000","last_peered":"2024-03-11T21:36:21.421757+0000","last_clean":"2024-03-11T21:36:21.421757+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:36:21.421757+0000","last_undegraded":"2024-03-11T21:36:21.421757+0000","last_fullsized":"2024-03-11T21:36:21.421757+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'52","last_scrub_stamp":"2024-03-11T21:35:00.480347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.480347+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.817828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082181799999999998,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"485'36","reported_seq":950,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415372+0000","last_change":"2024-03-11T21:35:26.433372+0000","last_active":"2024-03-11T21:36:21.415372+0000","last_peered":"2024-03-11T21:36:21.415372+0000","last_clean":"2024-03-11T21:36:21.415372+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:36:21.415372+0000","last_undegraded":"2024-03-11T21:36:21.415372+0000","last_fullsized":"2024-03-11T21:36:21.415372+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'36","last_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_deep_scrub":"485'36","last_deep_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.199420+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.779318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023323459999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"444'184","reported_seq":1077,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417245+0000","last_change":"2024-03-11T21:36:04.302637+0000","last_active":"2024-03-11T21:36:21.417245+0000","last_peered":"2024-03-11T21:36:21.417245+0000","last_clean":"2024-03-11T21:36:21.417245+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:36:21.417245+0000","last_undegraded":"2024-03-11T21:36:21.417245+0000","last_fullsized":"2024-03-11T21:36:21.417245+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'184","last_scrub_stamp":"2024-03-11T21:36:04.302373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.302373+0000","objects_scrubbed":14,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.870306+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001783946,"stat_sum":{"num_bytes":125,"num_objects":14,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":56,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":1012,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385065+0000","last_change":"2024-03-11T21:36:17.771393+0000","last_active":"2024-03-11T21:36:21.385065+0000","last_peered":"2024-03-11T21:36:21.385065+0000","last_clean":"2024-03-11T21:36:21.385065+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:36:21.385065+0000","last_undegraded":"2024-03-11T21:36:21.385065+0000","last_fullsized":"2024-03-11T21:36:21.385065+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:36:17.771145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.771145+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.169197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0030091919999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":933,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608443+0000","last_change":"2024-03-11T21:36:11.356246+0000","last_active":"2024-03-11T21:36:16.608443+0000","last_peered":"2024-03-11T21:36:16.608443+0000","last_clean":"2024-03-11T21:36:16.608443+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:36:16.608443+0000","last_undegraded":"2024-03-11T21:36:16.608443+0000","last_fullsized":"2024-03-11T21:36:16.608443+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:36:11.356102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.356102+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:28.012965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001800236,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"296'56","reported_seq":971,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608331+0000","last_change":"2024-03-11T21:36:13.359143+0000","last_active":"2024-03-11T21:36:16.608331+0000","last_peered":"2024-03-11T21:36:16.608331+0000","last_clean":"2024-03-11T21:36:16.608331+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:36:16.608331+0000","last_undegraded":"2024-03-11T21:36:16.608331+0000","last_fullsized":"2024-03-11T21:36:16.608331+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:36:13.358869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.358869+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:32.254273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028501400000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"521'132","reported_seq":1019,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417908+0000","last_change":"2024-03-11T21:35:26.453765+0000","last_active":"2024-03-11T21:36:21.417908+0000","last_peered":"2024-03-11T21:36:21.417908+0000","last_clean":"2024-03-11T21:36:21.417908+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:36:21.417908+0000","last_undegraded":"2024-03-11T21:36:21.417908+0000","last_fullsized":"2024-03-11T21:36:21.417908+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015353044,"stat_sum":{"num_bytes":115,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":39,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":957,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416360+0000","last_change":"2024-03-11T21:36:02.266701+0000","last_active":"2024-03-11T21:36:21.416360+0000","last_peered":"2024-03-11T21:36:21.416360+0000","last_clean":"2024-03-11T21:36:21.416360+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:36:21.416360+0000","last_undegraded":"2024-03-11T21:36:21.416360+0000","last_fullsized":"2024-03-11T21:36:21.416360+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'22","last_scrub_stamp":"2024-03-11T21:36:02.266514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.266514+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:09.997323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001443038,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"458'3","reported_seq":940,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421877+0000","last_change":"2024-03-11T21:36:17.826472+0000","last_active":"2024-03-11T21:36:21.421877+0000","last_peered":"2024-03-11T21:36:21.421877+0000","last_clean":"2024-03-11T21:36:21.421877+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:36:21.421877+0000","last_undegraded":"2024-03-11T21:36:21.421877+0000","last_fullsized":"2024-03-11T21:36:21.421877+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'3","last_scrub_stamp":"2024-03-11T21:36:17.826340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.826340+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.775926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079279899999999996,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":950,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.317525+0000","last_change":"2024-03-11T21:36:18.317604+0000","last_active":"2024-03-11T21:36:18.317525+0000","last_peered":"2024-03-11T21:36:18.317525+0000","last_clean":"2024-03-11T21:36:18.317525+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:36:18.317525+0000","last_undegraded":"2024-03-11T21:36:18.317525+0000","last_fullsized":"2024-03-11T21:36:18.317525+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:36:18.317358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.317358+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.642703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022017640000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"506'46","reported_seq":953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.706894+0000","last_change":"2024-03-11T21:36:21.706930+0000","last_active":"2024-03-11T21:36:21.706894+0000","last_peered":"2024-03-11T21:36:21.706894+0000","last_clean":"2024-03-11T21:36:21.706894+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:36:21.706894+0000","last_undegraded":"2024-03-11T21:36:21.706894+0000","last_fullsized":"2024-03-11T21:36:21.706894+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'46","last_scrub_stamp":"2024-03-11T21:36:21.706814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.706814+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:47.773546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028979269999999998,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":981,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422693+0000","last_change":"2024-03-11T21:35:26.597681+0000","last_active":"2024-03-11T21:36:21.422693+0000","last_peered":"2024-03-11T21:36:21.422693+0000","last_clean":"2024-03-11T21:36:21.422693+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:36:21.422693+0000","last_undegraded":"2024-03-11T21:36:21.422693+0000","last_fullsized":"2024-03-11T21:36:21.422693+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:33:23.654355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.654355+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000857495,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":929,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608338+0000","last_change":"2024-03-11T21:36:08.294434+0000","last_active":"2024-03-11T21:36:16.608338+0000","last_peered":"2024-03-11T21:36:16.608338+0000","last_clean":"2024-03-11T21:36:16.608338+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:36:16.608338+0000","last_undegraded":"2024-03-11T21:36:16.608338+0000","last_fullsized":"2024-03-11T21:36:16.608338+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:36:08.294289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.294289+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.501293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00159348,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":1001,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608327+0000","last_change":"2024-03-11T21:35:26.437723+0000","last_active":"2024-03-11T21:36:16.608327+0000","last_peered":"2024-03-11T21:36:16.608327+0000","last_clean":"2024-03-11T21:36:16.608327+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:36:16.608327+0000","last_undegraded":"2024-03-11T21:36:16.608327+0000","last_fullsized":"2024-03-11T21:36:16.608327+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'108","last_scrub_stamp":"2024-03-11T21:35:09.044564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.044564+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032700849999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":819,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:19.125256+0000","last_change":"2024-03-11T21:36:19.125306+0000","last_active":"2024-03-11T21:36:19.125256+0000","last_peered":"2024-03-11T21:36:19.125256+0000","last_clean":"2024-03-11T21:36:19.125256+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:36:19.125256+0000","last_undegraded":"2024-03-11T21:36:19.125256+0000","last_fullsized":"2024-03-11T21:36:19.125256+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:36:19.125151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.125151+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.106731+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00224048,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":965,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:17.330918+0000","last_change":"2024-03-11T21:36:17.331041+0000","last_active":"2024-03-11T21:36:17.330918+0000","last_peered":"2024-03-11T21:36:17.330918+0000","last_clean":"2024-03-11T21:36:17.330918+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:36:17.330918+0000","last_undegraded":"2024-03-11T21:36:17.330918+0000","last_fullsized":"2024-03-11T21:36:17.330918+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:36:17.330735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.330735+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.092844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072179299999999998,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"515'102","reported_seq":1025,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416050+0000","last_change":"2024-03-11T21:36:18.451092+0000","last_active":"2024-03-11T21:36:21.416050+0000","last_peered":"2024-03-11T21:36:21.416050+0000","last_clean":"2024-03-11T21:36:21.416050+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:36:21.416050+0000","last_undegraded":"2024-03-11T21:36:21.416050+0000","last_fullsized":"2024-03-11T21:36:21.416050+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'102","last_scrub_stamp":"2024-03-11T21:36:18.450840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.450840+0000","objects_scrubbed":7,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:28.366841+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0026391510000000002,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"486'73","reported_seq":1002,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418240+0000","last_change":"2024-03-11T21:35:26.454329+0000","last_active":"2024-03-11T21:36:21.418240+0000","last_peered":"2024-03-11T21:36:21.418240+0000","last_clean":"2024-03-11T21:36:21.418240+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:36:21.418240+0000","last_undegraded":"2024-03-11T21:36:21.418240+0000","last_fullsized":"2024-03-11T21:36:21.418240+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'69","last_scrub_stamp":"2024-03-11T21:34:58.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.041773+0000","objects_scrubbed":9,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.011639752999999999,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"444'37","reported_seq":839,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673614+0000","last_change":"2024-03-11T21:36:16.054966+0000","last_active":"2024-03-11T21:36:18.673614+0000","last_peered":"2024-03-11T21:36:18.673614+0000","last_clean":"2024-03-11T21:36:18.673614+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:36:18.673614+0000","last_undegraded":"2024-03-11T21:36:18.673614+0000","last_fullsized":"2024-03-11T21:36:18.673614+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'37","last_scrub_stamp":"2024-03-11T21:36:16.054853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.054853+0000","objects_scrubbed":5,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.434049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012079040000000001,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"503'215","reported_seq":1077,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418887+0000","last_change":"2024-03-11T21:35:26.455346+0000","last_active":"2024-03-11T21:36:21.418887+0000","last_peered":"2024-03-11T21:36:21.418887+0000","last_clean":"2024-03-11T21:36:21.418887+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:36:21.418887+0000","last_undegraded":"2024-03-11T21:36:21.418887+0000","last_fullsized":"2024-03-11T21:36:21.418887+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0095356580000000007,"stat_sum":{"num_bytes":151,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":67,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":939,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415270+0000","last_change":"2024-03-11T21:35:26.453569+0000","last_active":"2024-03-11T21:36:21.415270+0000","last_peered":"2024-03-11T21:36:21.415270+0000","last_clean":"2024-03-11T21:36:21.415270+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:36:21.415270+0000","last_undegraded":"2024-03-11T21:36:21.415270+0000","last_fullsized":"2024-03-11T21:36:21.415270+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'35","last_scrub_stamp":"2024-03-11T21:35:08.144914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.144914+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016015273999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"464'17","reported_seq":980,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.381981+0000","last_change":"2024-03-11T21:35:26.449618+0000","last_active":"2024-03-11T21:36:21.381981+0000","last_peered":"2024-03-11T21:36:21.381981+0000","last_clean":"2024-03-11T21:36:21.381981+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:36:21.381981+0000","last_undegraded":"2024-03-11T21:36:21.381981+0000","last_fullsized":"2024-03-11T21:36:21.381981+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'14","last_scrub_stamp":"2024-03-11T21:33:31.565199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.565199+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0045876709999999998,"stat_sum":{"num_bytes":4194330,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":985,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383588+0000","last_change":"2024-03-11T21:35:26.451133+0000","last_active":"2024-03-11T21:36:21.383588+0000","last_peered":"2024-03-11T21:36:21.383588+0000","last_clean":"2024-03-11T21:36:21.383588+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:36:21.383588+0000","last_undegraded":"2024-03-11T21:36:21.383588+0000","last_fullsized":"2024-03-11T21:36:21.383588+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'87","last_scrub_stamp":"2024-03-11T21:34:32.811720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.811720+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032466769999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":971,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417777+0000","last_change":"2024-03-11T21:35:26.453300+0000","last_active":"2024-03-11T21:36:21.417777+0000","last_peered":"2024-03-11T21:36:21.417777+0000","last_clean":"2024-03-11T21:36:21.417777+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:36:21.417777+0000","last_undegraded":"2024-03-11T21:36:21.417777+0000","last_fullsized":"2024-03-11T21:36:21.417777+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'65","last_scrub_stamp":"2024-03-11T21:35:07.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.139123+0000","objects_scrubbed":8,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:35.157679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014996763,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":993,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421510+0000","last_change":"2024-03-11T21:35:26.444064+0000","last_active":"2024-03-11T21:36:21.421510+0000","last_peered":"2024-03-11T21:36:21.421510+0000","last_clean":"2024-03-11T21:36:21.421510+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:36:21.421510+0000","last_undegraded":"2024-03-11T21:36:21.421510+0000","last_fullsized":"2024-03-11T21:36:21.421510+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.446157+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00098680300000000007,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"495'67","reported_seq":987,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415264+0000","last_change":"2024-03-11T21:36:11.203520+0000","last_active":"2024-03-11T21:36:21.415264+0000","last_peered":"2024-03-11T21:36:21.415264+0000","last_clean":"2024-03-11T21:36:21.415264+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:36:21.415264+0000","last_undegraded":"2024-03-11T21:36:21.415264+0000","last_fullsized":"2024-03-11T21:36:21.415264+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'67","last_scrub_stamp":"2024-03-11T21:36:11.203406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.203406+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:12.833033+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.0043048899999998,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"467'56","reported_seq":982,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422622+0000","last_change":"2024-03-11T21:36:06.936868+0000","last_active":"2024-03-11T21:36:21.422622+0000","last_peered":"2024-03-11T21:36:21.422622+0000","last_clean":"2024-03-11T21:36:21.422622+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:36:21.422622+0000","last_undegraded":"2024-03-11T21:36:21.422622+0000","last_fullsized":"2024-03-11T21:36:21.422622+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'56","last_scrub_stamp":"2024-03-11T21:36:06.936622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.936622+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:14.805000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056505100000000001,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"467'6","reported_seq":925,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418890+0000","last_change":"2024-03-11T21:35:26.455653+0000","last_active":"2024-03-11T21:36:21.418890+0000","last_peered":"2024-03-11T21:36:21.418890+0000","last_clean":"2024-03-11T21:36:21.418890+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:36:21.418890+0000","last_undegraded":"2024-03-11T21:36:21.418890+0000","last_fullsized":"2024-03-11T21:36:21.418890+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'6","last_scrub_stamp":"2024-03-11T21:34:50.112525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.112525+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177331,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420515+0000","last_change":"2024-03-11T21:36:09.680554+0000","last_active":"2024-03-11T21:36:21.420515+0000","last_peered":"2024-03-11T21:36:21.420515+0000","last_clean":"2024-03-11T21:36:21.420515+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:36:21.420515+0000","last_undegraded":"2024-03-11T21:36:21.420515+0000","last_fullsized":"2024-03-11T21:36:21.420515+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.680346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.680346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.025150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":796,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415078+0000","last_change":"2024-03-11T21:35:05.450716+0000","last_active":"2024-03-11T21:36:21.415078+0000","last_peered":"2024-03-11T21:36:21.415078+0000","last_clean":"2024-03-11T21:36:21.415078+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:36:21.415078+0000","last_undegraded":"2024-03-11T21:36:21.415078+0000","last_fullsized":"2024-03-11T21:36:21.415078+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.450323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.450323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":783,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383120+0000","last_change":"2024-03-11T21:33:25.553124+0000","last_active":"2024-03-11T21:36:21.383120+0000","last_peered":"2024-03-11T21:36:21.383120+0000","last_clean":"2024-03-11T21:36:21.383120+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:36:21.383120+0000","last_undegraded":"2024-03-11T21:36:21.383120+0000","last_fullsized":"2024-03-11T21:36:21.383120+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.552965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.552965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":783,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422243+0000","last_change":"2024-03-11T21:36:14.628566+0000","last_active":"2024-03-11T21:36:21.422243+0000","last_peered":"2024-03-11T21:36:21.422243+0000","last_clean":"2024-03-11T21:36:21.422243+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:36:21.422243+0000","last_undegraded":"2024-03-11T21:36:21.422243+0000","last_fullsized":"2024-03-11T21:36:21.422243+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.259727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":685,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673006+0000","last_change":"2024-03-11T21:33:34.103121+0000","last_active":"2024-03-11T21:36:18.673006+0000","last_peered":"2024-03-11T21:36:18.673006+0000","last_clean":"2024-03-11T21:36:18.673006+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:36:18.673006+0000","last_undegraded":"2024-03-11T21:36:18.673006+0000","last_fullsized":"2024-03-11T21:36:18.673006+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:34.103006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:34.103006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":796,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416417+0000","last_change":"2024-03-11T21:35:06.473871+0000","last_active":"2024-03-11T21:36:21.416417+0000","last_peered":"2024-03-11T21:36:21.416417+0000","last_clean":"2024-03-11T21:36:21.416417+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:36:21.416417+0000","last_undegraded":"2024-03-11T21:36:21.416417+0000","last_fullsized":"2024-03-11T21:36:21.416417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.473532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.473532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":796,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416845+0000","last_change":"2024-03-11T21:34:58.319554+0000","last_active":"2024-03-11T21:36:21.416845+0000","last_peered":"2024-03-11T21:36:21.416845+0000","last_clean":"2024-03-11T21:36:21.416845+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:36:21.416845+0000","last_undegraded":"2024-03-11T21:36:21.416845+0000","last_fullsized":"2024-03-11T21:36:21.416845+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.319457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:21.657447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":806,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422795+0000","last_change":"2024-03-11T21:36:08.869894+0000","last_active":"2024-03-11T21:36:21.422795+0000","last_peered":"2024-03-11T21:36:21.422795+0000","last_clean":"2024-03-11T21:36:21.422795+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:36:21.422795+0000","last_undegraded":"2024-03-11T21:36:21.422795+0000","last_fullsized":"2024-03-11T21:36:21.422795+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.869673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.869673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.060598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":790,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414151+0000","last_change":"2024-03-11T21:34:35.918096+0000","last_active":"2024-03-11T21:36:21.414151+0000","last_peered":"2024-03-11T21:36:21.414151+0000","last_clean":"2024-03-11T21:36:21.414151+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:36:21.414151+0000","last_undegraded":"2024-03-11T21:36:21.414151+0000","last_fullsized":"2024-03-11T21:36:21.414151+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.917835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.917835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":783,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420635+0000","last_change":"2024-03-11T21:34:49.557575+0000","last_active":"2024-03-11T21:36:21.420635+0000","last_peered":"2024-03-11T21:36:21.420635+0000","last_clean":"2024-03-11T21:36:21.420635+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:36:21.420635+0000","last_undegraded":"2024-03-11T21:36:21.420635+0000","last_fullsized":"2024-03-11T21:36:21.420635+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:49.557488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:49.557488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609224+0000","last_change":"2024-03-11T21:33:41.974005+0000","last_active":"2024-03-11T21:36:16.609224+0000","last_peered":"2024-03-11T21:36:16.609224+0000","last_clean":"2024-03-11T21:36:16.609224+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:36:16.609224+0000","last_undegraded":"2024-03-11T21:36:16.609224+0000","last_fullsized":"2024-03-11T21:36:16.609224+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.973907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":779,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418782+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:36:21.418782+0000","last_peered":"2024-03-11T21:36:21.418782+0000","last_clean":"2024-03-11T21:36:21.418782+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:36:21.418782+0000","last_undegraded":"2024-03-11T21:36:21.418782+0000","last_fullsized":"2024-03-11T21:36:21.418782+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609314+0000","last_change":"2024-03-11T21:34:40.037312+0000","last_active":"2024-03-11T21:36:16.609314+0000","last_peered":"2024-03-11T21:36:16.609314+0000","last_clean":"2024-03-11T21:36:16.609314+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:36:16.609314+0000","last_undegraded":"2024-03-11T21:36:16.609314+0000","last_fullsized":"2024-03-11T21:36:16.609314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.037161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.037161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":790,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416971+0000","last_change":"2024-03-11T21:35:05.211872+0000","last_active":"2024-03-11T21:36:21.416971+0000","last_peered":"2024-03-11T21:36:21.416971+0000","last_clean":"2024-03-11T21:36:21.416971+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:36:21.416971+0000","last_undegraded":"2024-03-11T21:36:21.416971+0000","last_fullsized":"2024-03-11T21:36:21.416971+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.211634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:32.162818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":798,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421240+0000","last_change":"2024-03-11T21:34:58.485169+0000","last_active":"2024-03-11T21:36:21.421240+0000","last_peered":"2024-03-11T21:36:21.421240+0000","last_clean":"2024-03-11T21:36:21.421240+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:36:21.421240+0000","last_undegraded":"2024-03-11T21:36:21.421240+0000","last_fullsized":"2024-03-11T21:36:21.421240+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.485052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.485052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":189,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673409+0000","last_change":"2024-03-11T21:34:52.102628+0000","last_active":"2024-03-11T21:36:18.673409+0000","last_peered":"2024-03-11T21:36:18.673409+0000","last_clean":"2024-03-11T21:36:18.673409+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:36:18.673409+0000","last_undegraded":"2024-03-11T21:36:18.673409+0000","last_fullsized":"2024-03-11T21:36:18.673409+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:52.102271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.102271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.080698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":796,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.413542+0000","last_change":"2024-03-11T21:34:54.321311+0000","last_active":"2024-03-11T21:36:21.413542+0000","last_peered":"2024-03-11T21:36:21.413542+0000","last_clean":"2024-03-11T21:36:21.413542+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:36:21.413542+0000","last_undegraded":"2024-03-11T21:36:21.413542+0000","last_fullsized":"2024-03-11T21:36:21.413542+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.321007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.321007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":785,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414998+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:36:21.414998+0000","last_peered":"2024-03-11T21:36:21.414998+0000","last_clean":"2024-03-11T21:36:21.414998+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:36:21.414998+0000","last_undegraded":"2024-03-11T21:36:21.414998+0000","last_fullsized":"2024-03-11T21:36:21.414998+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":775,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421604+0000","last_change":"2024-03-11T21:33:12.695054+0000","last_active":"2024-03-11T21:36:21.421604+0000","last_peered":"2024-03-11T21:36:21.421604+0000","last_clean":"2024-03-11T21:36:21.421604+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:36:21.421604+0000","last_undegraded":"2024-03-11T21:36:21.421604+0000","last_fullsized":"2024-03-11T21:36:21.421604+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:12.694966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:12.694966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":790,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414619+0000","last_change":"2024-03-11T21:35:02.148627+0000","last_active":"2024-03-11T21:36:21.414619+0000","last_peered":"2024-03-11T21:36:21.414619+0000","last_clean":"2024-03-11T21:36:21.414619+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:36:21.414619+0000","last_undegraded":"2024-03-11T21:36:21.414619+0000","last_fullsized":"2024-03-11T21:36:21.414619+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.148344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.148344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.606703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":662,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673209+0000","last_change":"2024-03-11T21:35:11.130108+0000","last_active":"2024-03-11T21:36:18.673209+0000","last_peered":"2024-03-11T21:36:18.673209+0000","last_clean":"2024-03-11T21:36:18.673209+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:36:18.673209+0000","last_undegraded":"2024-03-11T21:36:18.673209+0000","last_fullsized":"2024-03-11T21:36:18.673209+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.129979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.129979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:28.297504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":783,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422665+0000","last_change":"2024-03-11T21:34:59.525103+0000","last_active":"2024-03-11T21:36:21.422665+0000","last_peered":"2024-03-11T21:36:21.422665+0000","last_clean":"2024-03-11T21:36:21.422665+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:36:21.422665+0000","last_undegraded":"2024-03-11T21:36:21.422665+0000","last_fullsized":"2024-03-11T21:36:21.422665+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:59.524800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:59.524800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":791,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422839+0000","last_change":"2024-03-11T21:36:04.661608+0000","last_active":"2024-03-11T21:36:21.422839+0000","last_peered":"2024-03-11T21:36:21.422839+0000","last_clean":"2024-03-11T21:36:21.422839+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:36:21.422839+0000","last_undegraded":"2024-03-11T21:36:21.422839+0000","last_fullsized":"2024-03-11T21:36:21.422839+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.661495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.661495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.634867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610344+0000","last_change":"2024-03-11T21:34:58.091810+0000","last_active":"2024-03-11T21:36:16.610344+0000","last_peered":"2024-03-11T21:36:16.610344+0000","last_clean":"2024-03-11T21:36:16.610344+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:36:16.610344+0000","last_undegraded":"2024-03-11T21:36:16.610344+0000","last_fullsized":"2024-03-11T21:36:16.610344+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.091664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.091664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":783,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.382482+0000","last_change":"2024-03-11T21:33:41.560584+0000","last_active":"2024-03-11T21:36:21.382482+0000","last_peered":"2024-03-11T21:36:21.382482+0000","last_clean":"2024-03-11T21:36:21.382482+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:36:21.382482+0000","last_undegraded":"2024-03-11T21:36:21.382482+0000","last_fullsized":"2024-03-11T21:36:21.382482+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:41.560389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:41.560389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":779,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415105+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:36:21.415105+0000","last_peered":"2024-03-11T21:36:21.415105+0000","last_clean":"2024-03-11T21:36:21.415105+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:36:21.415105+0000","last_undegraded":"2024-03-11T21:36:21.415105+0000","last_fullsized":"2024-03-11T21:36:21.415105+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":190,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672903+0000","last_change":"2024-03-11T21:34:39.164537+0000","last_active":"2024-03-11T21:36:18.672903+0000","last_peered":"2024-03-11T21:36:18.672903+0000","last_clean":"2024-03-11T21:36:18.672903+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:36:18.672903+0000","last_undegraded":"2024-03-11T21:36:18.672903+0000","last_fullsized":"2024-03-11T21:36:18.672903+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.164421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.164421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":790,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414575+0000","last_change":"2024-03-11T21:36:03.284628+0000","last_active":"2024-03-11T21:36:21.414575+0000","last_peered":"2024-03-11T21:36:21.414575+0000","last_clean":"2024-03-11T21:36:21.414575+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:36:21.414575+0000","last_undegraded":"2024-03-11T21:36:21.414575+0000","last_fullsized":"2024-03-11T21:36:21.414575+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:03.284530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.284530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:15.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609693+0000","last_change":"2024-03-11T21:34:36.108350+0000","last_active":"2024-03-11T21:36:16.609693+0000","last_peered":"2024-03-11T21:36:16.609693+0000","last_clean":"2024-03-11T21:36:16.609693+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:36:16.609693+0000","last_undegraded":"2024-03-11T21:36:16.609693+0000","last_fullsized":"2024-03-11T21:36:16.609693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.108246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.108246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":779,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419110+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:36:21.419110+0000","last_peered":"2024-03-11T21:36:21.419110+0000","last_clean":"2024-03-11T21:36:21.419110+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:36:21.419110+0000","last_undegraded":"2024-03-11T21:36:21.419110+0000","last_fullsized":"2024-03-11T21:36:21.419110+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":798,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421338+0000","last_change":"2024-03-11T21:34:48.410224+0000","last_active":"2024-03-11T21:36:21.421338+0000","last_peered":"2024-03-11T21:36:21.421338+0000","last_clean":"2024-03-11T21:36:21.421338+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:36:21.421338+0000","last_undegraded":"2024-03-11T21:36:21.421338+0000","last_fullsized":"2024-03-11T21:36:21.421338+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.410124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.410124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":662,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673917+0000","last_change":"2024-03-11T21:35:01.154707+0000","last_active":"2024-03-11T21:36:18.673917+0000","last_peered":"2024-03-11T21:36:18.673917+0000","last_clean":"2024-03-11T21:36:18.673917+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:36:18.673917+0000","last_undegraded":"2024-03-11T21:36:18.673917+0000","last_fullsized":"2024-03-11T21:36:18.673917+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.154502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.154502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.515763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":791,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610301+0000","last_change":"2024-03-11T21:33:31.964693+0000","last_active":"2024-03-11T21:36:16.610301+0000","last_peered":"2024-03-11T21:36:16.610301+0000","last_clean":"2024-03-11T21:36:16.610301+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:36:16.610301+0000","last_undegraded":"2024-03-11T21:36:16.610301+0000","last_fullsized":"2024-03-11T21:36:16.610301+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:31.964602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.964602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1101,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415792+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:36:21.415792+0000","last_peered":"2024-03-11T21:36:21.415792+0000","last_clean":"2024-03-11T21:36:21.415792+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:36:21.415792+0000","last_undegraded":"2024-03-11T21:36:21.415792+0000","last_fullsized":"2024-03-11T21:36:21.415792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":977,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419538+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:36:21.419538+0000","last_peered":"2024-03-11T21:36:21.419538+0000","last_clean":"2024-03-11T21:36:21.419538+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:36:21.419538+0000","last_undegraded":"2024-03-11T21:36:21.419538+0000","last_fullsized":"2024-03-11T21:36:21.419538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":965,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421874+0000","last_change":"2024-03-11T21:35:01.437775+0000","last_active":"2024-03-11T21:36:21.421874+0000","last_peered":"2024-03-11T21:36:21.421874+0000","last_clean":"2024-03-11T21:36:21.421874+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:36:21.421874+0000","last_undegraded":"2024-03-11T21:36:21.421874+0000","last_fullsized":"2024-03-11T21:36:21.421874+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.437566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.437566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":940,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383407+0000","last_change":"2024-03-11T21:35:10.696295+0000","last_active":"2024-03-11T21:36:21.383407+0000","last_peered":"2024-03-11T21:36:21.383407+0000","last_clean":"2024-03-11T21:36:21.383407+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:36:21.383407+0000","last_undegraded":"2024-03-11T21:36:21.383407+0000","last_fullsized":"2024-03-11T21:36:21.383407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.696090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.696090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":798,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385485+0000","last_change":"2024-03-11T21:34:38.848849+0000","last_active":"2024-03-11T21:36:21.385485+0000","last_peered":"2024-03-11T21:36:21.385485+0000","last_clean":"2024-03-11T21:36:21.385485+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:36:21.385485+0000","last_undegraded":"2024-03-11T21:36:21.385485+0000","last_fullsized":"2024-03-11T21:36:21.385485+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:38.848616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.848616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":977,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.384318+0000","last_change":"2024-03-11T21:35:11.672580+0000","last_active":"2024-03-11T21:36:21.384318+0000","last_peered":"2024-03-11T21:36:21.384318+0000","last_clean":"2024-03-11T21:36:21.384318+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:36:21.384318+0000","last_undegraded":"2024-03-11T21:36:21.384318+0000","last_fullsized":"2024-03-11T21:36:21.384318+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:35:11.672324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.672324+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":844,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672468+0000","last_change":"2024-03-11T21:34:48.064829+0000","last_active":"2024-03-11T21:36:18.672468+0000","last_peered":"2024-03-11T21:36:18.672468+0000","last_clean":"2024-03-11T21:36:18.672468+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:36:18.672468+0000","last_undegraded":"2024-03-11T21:36:18.672468+0000","last_fullsized":"2024-03-11T21:36:18.672468+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.064622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.064622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":958,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420303+0000","last_change":"2024-03-11T21:34:40.424576+0000","last_active":"2024-03-11T21:36:21.420303+0000","last_peered":"2024-03-11T21:36:21.420303+0000","last_clean":"2024-03-11T21:36:21.420303+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:36:21.420303+0000","last_undegraded":"2024-03-11T21:36:21.420303+0000","last_fullsized":"2024-03-11T21:36:21.420303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.424446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.424446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":942,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608954+0000","last_change":"2024-03-11T21:35:02.012107+0000","last_active":"2024-03-11T21:36:16.608954+0000","last_peered":"2024-03-11T21:36:16.608954+0000","last_clean":"2024-03-11T21:36:16.608954+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:36:16.608954+0000","last_undegraded":"2024-03-11T21:36:16.608954+0000","last_fullsized":"2024-03-11T21:36:16.608954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.011982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.011982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:27.342539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":797,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385383+0000","last_change":"2024-03-11T21:36:08.644929+0000","last_active":"2024-03-11T21:36:21.385383+0000","last_peered":"2024-03-11T21:36:21.385383+0000","last_clean":"2024-03-11T21:36:21.385383+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:36:21.385383+0000","last_undegraded":"2024-03-11T21:36:21.385383+0000","last_fullsized":"2024-03-11T21:36:21.385383+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.644692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.644692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.731782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":999,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419541+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:36:21.419541+0000","last_peered":"2024-03-11T21:36:21.419541+0000","last_clean":"2024-03-11T21:36:21.419541+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:36:21.419541+0000","last_undegraded":"2024-03-11T21:36:21.419541+0000","last_fullsized":"2024-03-11T21:36:21.419541+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":973,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421272+0000","last_change":"2024-03-11T21:36:11.798174+0000","last_active":"2024-03-11T21:36:21.421272+0000","last_peered":"2024-03-11T21:36:21.421272+0000","last_clean":"2024-03-11T21:36:21.421272+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:36:21.421272+0000","last_undegraded":"2024-03-11T21:36:21.421272+0000","last_fullsized":"2024-03-11T21:36:21.421272+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:17.280025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":957,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421621+0000","last_change":"2024-03-11T21:34:43.484238+0000","last_active":"2024-03-11T21:36:21.421621+0000","last_peered":"2024-03-11T21:36:21.421621+0000","last_clean":"2024-03-11T21:36:21.421621+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:36:21.421621+0000","last_undegraded":"2024-03-11T21:36:21.421621+0000","last_fullsized":"2024-03-11T21:36:21.421621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.484121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.484121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":935,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414561+0000","last_change":"2024-03-11T21:35:11.139678+0000","last_active":"2024-03-11T21:36:21.414561+0000","last_peered":"2024-03-11T21:36:21.414561+0000","last_clean":"2024-03-11T21:36:21.414561+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:36:21.414561+0000","last_undegraded":"2024-03-11T21:36:21.414561+0000","last_fullsized":"2024-03-11T21:36:21.414561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.139473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.139473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:30.148493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":663,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672121+0000","last_change":"2024-03-11T21:33:39.131584+0000","last_active":"2024-03-11T21:36:18.672121+0000","last_peered":"2024-03-11T21:36:18.672121+0000","last_clean":"2024-03-11T21:36:18.672121+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:36:18.672121+0000","last_undegraded":"2024-03-11T21:36:18.672121+0000","last_fullsized":"2024-03-11T21:36:18.672121+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.131510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.131510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1033,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415444+0000","last_change":"2024-03-11T21:35:12.343880+0000","last_active":"2024-03-11T21:36:21.415444+0000","last_peered":"2024-03-11T21:36:21.415444+0000","last_clean":"2024-03-11T21:36:21.415444+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:36:21.415444+0000","last_undegraded":"2024-03-11T21:36:21.415444+0000","last_fullsized":"2024-03-11T21:36:21.415444+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:35:12.343626+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.343626+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:39.202539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418282+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:36:21.418282+0000","last_peered":"2024-03-11T21:36:21.418282+0000","last_clean":"2024-03-11T21:36:21.418282+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:36:21.418282+0000","last_undegraded":"2024-03-11T21:36:21.418282+0000","last_fullsized":"2024-03-11T21:36:21.418282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":850,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:20.116477+0000","last_change":"2024-03-11T21:36:20.116589+0000","last_active":"2024-03-11T21:36:20.116477+0000","last_peered":"2024-03-11T21:36:20.116477+0000","last_clean":"2024-03-11T21:36:20.116477+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:36:20.116477+0000","last_undegraded":"2024-03-11T21:36:20.116477+0000","last_fullsized":"2024-03-11T21:36:20.116477+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:20.116311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.116311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.628264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":961,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423076+0000","last_change":"2024-03-11T21:36:01.810406+0000","last_active":"2024-03-11T21:36:21.423076+0000","last_peered":"2024-03-11T21:36:21.423076+0000","last_clean":"2024-03-11T21:36:21.423076+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:36:21.423076+0000","last_undegraded":"2024-03-11T21:36:21.423076+0000","last_fullsized":"2024-03-11T21:36:21.423076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:01.810170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:01.810170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:18.600258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":806,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420918+0000","last_change":"2024-03-11T21:36:11.656705+0000","last_active":"2024-03-11T21:36:21.420918+0000","last_peered":"2024-03-11T21:36:21.420918+0000","last_clean":"2024-03-11T21:36:21.420918+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:36:21.420918+0000","last_undegraded":"2024-03-11T21:36:21.420918+0000","last_fullsized":"2024-03-11T21:36:21.420918+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:36:11.656610+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.656610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.369092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":974,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422445+0000","last_change":"2024-03-11T21:35:01.534298+0000","last_active":"2024-03-11T21:36:21.422445+0000","last_peered":"2024-03-11T21:36:21.422445+0000","last_clean":"2024-03-11T21:36:21.422445+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:36:21.422445+0000","last_undegraded":"2024-03-11T21:36:21.422445+0000","last_fullsized":"2024-03-11T21:36:21.422445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:01.534131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.534131+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":810,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672863+0000","last_change":"2024-03-11T21:34:41.173845+0000","last_active":"2024-03-11T21:36:18.672863+0000","last_peered":"2024-03-11T21:36:18.672863+0000","last_clean":"2024-03-11T21:36:18.672863+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:36:18.672863+0000","last_undegraded":"2024-03-11T21:36:18.672863+0000","last_fullsized":"2024-03-11T21:36:18.672863+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1136,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422125+0000","last_change":"2024-03-11T21:36:07.905103+0000","last_active":"2024-03-11T21:36:21.422125+0000","last_peered":"2024-03-11T21:36:21.422125+0000","last_clean":"2024-03-11T21:36:21.422125+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:36:21.422125+0000","last_undegraded":"2024-03-11T21:36:21.422125+0000","last_fullsized":"2024-03-11T21:36:21.422125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:36:07.904875+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.904875+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.866007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":954,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417543+0000","last_change":"2024-03-11T21:35:01.196442+0000","last_active":"2024-03-11T21:36:21.417543+0000","last_peered":"2024-03-11T21:36:21.417543+0000","last_clean":"2024-03-11T21:36:21.417543+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:36:21.417543+0000","last_undegraded":"2024-03-11T21:36:21.417543+0000","last_fullsized":"2024-03-11T21:36:21.417543+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.196310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.196310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":801,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422092+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:36:21.422092+0000","last_peered":"2024-03-11T21:36:21.422092+0000","last_clean":"2024-03-11T21:36:21.422092+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:36:21.422092+0000","last_undegraded":"2024-03-11T21:36:21.422092+0000","last_fullsized":"2024-03-11T21:36:21.422092+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":994,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420732+0000","last_change":"2024-03-11T21:33:20.697629+0000","last_active":"2024-03-11T21:36:21.420732+0000","last_peered":"2024-03-11T21:36:21.420732+0000","last_clean":"2024-03-11T21:36:21.420732+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:36:21.420732+0000","last_undegraded":"2024-03-11T21:36:21.420732+0000","last_fullsized":"2024-03-11T21:36:21.420732+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:20.697247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.697247+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":838,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672567+0000","last_change":"2024-03-11T21:36:05.135845+0000","last_active":"2024-03-11T21:36:18.672567+0000","last_peered":"2024-03-11T21:36:18.672567+0000","last_clean":"2024-03-11T21:36:18.672567+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:36:18.672567+0000","last_undegraded":"2024-03-11T21:36:18.672567+0000","last_fullsized":"2024-03-11T21:36:18.672567+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:05.135750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:05.135750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.810637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1119,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417413+0000","last_change":"2024-03-11T21:35:07.191988+0000","last_active":"2024-03-11T21:36:21.417413+0000","last_peered":"2024-03-11T21:36:21.417413+0000","last_clean":"2024-03-11T21:36:21.417413+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:36:21.417413+0000","last_undegraded":"2024-03-11T21:36:21.417413+0000","last_fullsized":"2024-03-11T21:36:21.417413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:35:07.191770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.191770+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:35.791017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":948,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385901+0000","last_change":"2024-03-11T21:34:42.857662+0000","last_active":"2024-03-11T21:36:21.385901+0000","last_peered":"2024-03-11T21:36:21.385901+0000","last_clean":"2024-03-11T21:36:21.385901+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:36:21.385901+0000","last_undegraded":"2024-03-11T21:36:21.385901+0000","last_fullsized":"2024-03-11T21:36:21.385901+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.857448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.857448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":793,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418047+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:36:21.418047+0000","last_peered":"2024-03-11T21:36:21.418047+0000","last_clean":"2024-03-11T21:36:21.418047+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:36:21.418047+0000","last_undegraded":"2024-03-11T21:36:21.418047+0000","last_fullsized":"2024-03-11T21:36:21.418047+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":991,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414251+0000","last_change":"2024-03-11T21:36:07.268802+0000","last_active":"2024-03-11T21:36:21.414251+0000","last_peered":"2024-03-11T21:36:21.414251+0000","last_clean":"2024-03-11T21:36:21.414251+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:36:21.414251+0000","last_undegraded":"2024-03-11T21:36:21.414251+0000","last_fullsized":"2024-03-11T21:36:21.414251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:36:07.268562+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.268562+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:10.428038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":956,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414821+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:36:21.414821+0000","last_peered":"2024-03-11T21:36:21.414821+0000","last_clean":"2024-03-11T21:36:21.414821+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:36:21.414821+0000","last_undegraded":"2024-03-11T21:36:21.414821+0000","last_fullsized":"2024-03-11T21:36:21.414821+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":943,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386212+0000","last_change":"2024-03-11T21:35:12.702568+0000","last_active":"2024-03-11T21:36:21.386212+0000","last_peered":"2024-03-11T21:36:21.386212+0000","last_clean":"2024-03-11T21:36:21.386212+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:36:21.386212+0000","last_undegraded":"2024-03-11T21:36:21.386212+0000","last_fullsized":"2024-03-11T21:36:21.386212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.702301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.702301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:27.160392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":933,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422655+0000","last_change":"2024-03-11T21:36:07.698299+0000","last_active":"2024-03-11T21:36:21.422655+0000","last_peered":"2024-03-11T21:36:21.422655+0000","last_clean":"2024-03-11T21:36:21.422655+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:36:21.422655+0000","last_undegraded":"2024-03-11T21:36:21.422655+0000","last_fullsized":"2024-03-11T21:36:21.422655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.698066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.698066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.708798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422836+0000","last_change":"2024-03-11T21:36:08.684202+0000","last_active":"2024-03-11T21:36:21.422836+0000","last_peered":"2024-03-11T21:36:21.422836+0000","last_clean":"2024-03-11T21:36:21.422836+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:36:21.422836+0000","last_undegraded":"2024-03-11T21:36:21.422836+0000","last_fullsized":"2024-03-11T21:36:21.422836+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.684101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.684101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.559643+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1126,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420869+0000","last_change":"2024-03-11T21:36:19.742269+0000","last_active":"2024-03-11T21:36:21.420869+0000","last_peered":"2024-03-11T21:36:21.420869+0000","last_clean":"2024-03-11T21:36:21.420869+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:36:21.420869+0000","last_undegraded":"2024-03-11T21:36:21.420869+0000","last_fullsized":"2024-03-11T21:36:21.420869+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:36:19.742119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.742119+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:31.824078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":967,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414292+0000","last_change":"2024-03-11T21:36:13.204077+0000","last_active":"2024-03-11T21:36:21.414292+0000","last_peered":"2024-03-11T21:36:21.414292+0000","last_clean":"2024-03-11T21:36:21.414292+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:36:21.414292+0000","last_undegraded":"2024-03-11T21:36:21.414292+0000","last_fullsized":"2024-03-11T21:36:21.414292+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:13.203841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.203841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.395164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":995,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610556+0000","last_change":"2024-03-11T21:34:53.090278+0000","last_active":"2024-03-11T21:36:16.610556+0000","last_peered":"2024-03-11T21:36:16.610556+0000","last_clean":"2024-03-11T21:36:16.610556+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:36:16.610556+0000","last_undegraded":"2024-03-11T21:36:16.610556+0000","last_fullsized":"2024-03-11T21:36:16.610556+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:53.090147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.090147+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":973,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416955+0000","last_change":"2024-03-11T21:35:00.409061+0000","last_active":"2024-03-11T21:36:21.416955+0000","last_peered":"2024-03-11T21:36:21.416955+0000","last_clean":"2024-03-11T21:36:21.416955+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:36:21.416955+0000","last_undegraded":"2024-03-11T21:36:21.416955+0000","last_fullsized":"2024-03-11T21:36:21.416955+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.408810+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":937,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418151+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:36:21.418151+0000","last_peered":"2024-03-11T21:36:21.418151+0000","last_clean":"2024-03-11T21:36:21.418151+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:36:21.418151+0000","last_undegraded":"2024-03-11T21:36:21.418151+0000","last_fullsized":"2024-03-11T21:36:21.418151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":793,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418880+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:36:21.418880+0000","last_peered":"2024-03-11T21:36:21.418880+0000","last_clean":"2024-03-11T21:36:21.418880+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:36:21.418880+0000","last_undegraded":"2024-03-11T21:36:21.418880+0000","last_fullsized":"2024-03-11T21:36:21.418880+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":950,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414084+0000","last_change":"2024-03-11T21:34:43.908469+0000","last_active":"2024-03-11T21:36:21.414084+0000","last_peered":"2024-03-11T21:36:21.414084+0000","last_clean":"2024-03-11T21:36:21.414084+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:36:21.414084+0000","last_undegraded":"2024-03-11T21:36:21.414084+0000","last_fullsized":"2024-03-11T21:36:21.414084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.908344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.908344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1729,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.651458+0000","last_change":"2024-03-11T21:33:40.550313+0000","last_active":"2024-03-11T21:36:21.651458+0000","last_peered":"2024-03-11T21:36:21.651458+0000","last_clean":"2024-03-11T21:36:21.651458+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:36:21.651458+0000","last_undegraded":"2024-03-11T21:36:21.651458+0000","last_fullsized":"2024-03-11T21:36:21.651458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:33:40.550110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.550110+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":975,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423780+0000","last_change":"2024-03-11T21:34:33.535935+0000","last_active":"2024-03-11T21:36:21.423780+0000","last_peered":"2024-03-11T21:36:21.423780+0000","last_clean":"2024-03-11T21:36:21.423780+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:36:21.423780+0000","last_undegraded":"2024-03-11T21:36:21.423780+0000","last_fullsized":"2024-03-11T21:36:21.423780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:34:33.535753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.535753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":863,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672873+0000","last_change":"2024-03-11T21:36:09.043237+0000","last_active":"2024-03-11T21:36:18.672873+0000","last_peered":"2024-03-11T21:36:18.672873+0000","last_clean":"2024-03-11T21:36:18.672873+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:36:18.672873+0000","last_undegraded":"2024-03-11T21:36:18.672873+0000","last_fullsized":"2024-03-11T21:36:18.672873+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:09.043133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.043133+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.706316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416584+0000","last_change":"2024-03-11T21:34:31.810307+0000","last_active":"2024-03-11T21:36:21.416584+0000","last_peered":"2024-03-11T21:36:21.416584+0000","last_clean":"2024-03-11T21:36:21.416584+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:36:21.416584+0000","last_undegraded":"2024-03-11T21:36:21.416584+0000","last_fullsized":"2024-03-11T21:36:21.416584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.810016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.810016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":793,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419082+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:36:21.419082+0000","last_peered":"2024-03-11T21:36:21.419082+0000","last_clean":"2024-03-11T21:36:21.419082+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:36:21.419082+0000","last_undegraded":"2024-03-11T21:36:21.419082+0000","last_fullsized":"2024-03-11T21:36:21.419082+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":948,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415364+0000","last_change":"2024-03-11T21:34:57.364338+0000","last_active":"2024-03-11T21:36:21.415364+0000","last_peered":"2024-03-11T21:36:21.415364+0000","last_clean":"2024-03-11T21:36:21.415364+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:36:21.415364+0000","last_undegraded":"2024-03-11T21:36:21.415364+0000","last_fullsized":"2024-03-11T21:36:21.415364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.364057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.364057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:27.348356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1059,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673936+0000","last_change":"2024-03-11T21:34:55.143888+0000","last_active":"2024-03-11T21:36:18.673936+0000","last_peered":"2024-03-11T21:36:18.673936+0000","last_clean":"2024-03-11T21:36:18.673936+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:36:18.673936+0000","last_undegraded":"2024-03-11T21:36:18.673936+0000","last_fullsized":"2024-03-11T21:36:18.673936+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.143614+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":947,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419450+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:36:21.419450+0000","last_peered":"2024-03-11T21:36:21.419450+0000","last_clean":"2024-03-11T21:36:21.419450+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:36:21.419450+0000","last_undegraded":"2024-03-11T21:36:21.419450+0000","last_fullsized":"2024-03-11T21:36:21.419450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":802,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672143+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:36:18.672143+0000","last_peered":"2024-03-11T21:36:18.672143+0000","last_clean":"2024-03-11T21:36:18.672143+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:36:18.672143+0000","last_undegraded":"2024-03-11T21:36:18.672143+0000","last_fullsized":"2024-03-11T21:36:18.672143+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":791,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610159+0000","last_change":"2024-03-11T21:34:35.105891+0000","last_active":"2024-03-11T21:36:16.610159+0000","last_peered":"2024-03-11T21:36:16.610159+0000","last_clean":"2024-03-11T21:36:16.610159+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:36:16.610159+0000","last_undegraded":"2024-03-11T21:36:16.610159+0000","last_fullsized":"2024-03-11T21:36:16.610159+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.105714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.105714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386141+0000","last_change":"2024-03-11T21:35:03.774565+0000","last_active":"2024-03-11T21:36:21.386141+0000","last_peered":"2024-03-11T21:36:21.386141+0000","last_clean":"2024-03-11T21:36:21.386141+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:36:21.386141+0000","last_undegraded":"2024-03-11T21:36:21.386141+0000","last_fullsized":"2024-03-11T21:36:21.386141+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.774354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.774354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":956,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414690+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:36:21.414690+0000","last_peered":"2024-03-11T21:36:21.414690+0000","last_clean":"2024-03-11T21:36:21.414690+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:36:21.414690+0000","last_undegraded":"2024-03-11T21:36:21.414690+0000","last_fullsized":"2024-03-11T21:36:21.414690+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":992,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608811+0000","last_change":"2024-03-11T21:36:03.222156+0000","last_active":"2024-03-11T21:36:16.608811+0000","last_peered":"2024-03-11T21:36:16.608811+0000","last_clean":"2024-03-11T21:36:16.608811+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:36:16.608811+0000","last_undegraded":"2024-03-11T21:36:16.608811+0000","last_fullsized":"2024-03-11T21:36:16.608811+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:03.221883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.221883+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.237647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":812,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673542+0000","last_change":"2024-03-11T21:35:04.076881+0000","last_active":"2024-03-11T21:36:18.673542+0000","last_peered":"2024-03-11T21:36:18.673542+0000","last_clean":"2024-03-11T21:36:18.673542+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:36:18.673542+0000","last_undegraded":"2024-03-11T21:36:18.673542+0000","last_fullsized":"2024-03-11T21:36:18.673542+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.076789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.076789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.593356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420541+0000","last_change":"2024-03-11T21:34:54.501393+0000","last_active":"2024-03-11T21:36:21.420541+0000","last_peered":"2024-03-11T21:36:21.420541+0000","last_clean":"2024-03-11T21:36:21.420541+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:36:21.420541+0000","last_undegraded":"2024-03-11T21:36:21.420541+0000","last_fullsized":"2024-03-11T21:36:21.420541+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.501183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.501183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":931,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418700+0000","last_change":"2024-03-11T21:30:37.346294+0000","last_active":"2024-03-11T21:36:21.418700+0000","last_peered":"2024-03-11T21:36:21.418700+0000","last_clean":"2024-03-11T21:36:21.418700+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:36:21.418700+0000","last_undegraded":"2024-03-11T21:36:21.418700+0000","last_fullsized":"2024-03-11T21:36:21.418700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:37.346208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:37.346208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1202,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414787+0000","last_change":"2024-03-11T21:30:48.243597+0000","last_active":"2024-03-11T21:36:21.414787+0000","last_peered":"2024-03-11T21:36:21.414787+0000","last_clean":"2024-03-11T21:36:21.414787+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:36:21.414787+0000","last_undegraded":"2024-03-11T21:36:21.414787+0000","last_fullsized":"2024-03-11T21:36:21.414787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:30:48.243369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:30:48.243369+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":960,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:22.898089+0000","last_change":"2024-03-11T21:36:22.898119+0000","last_active":"2024-03-11T21:36:22.898089+0000","last_peered":"2024-03-11T21:36:22.898089+0000","last_clean":"2024-03-11T21:36:22.898089+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:36:22.898089+0000","last_undegraded":"2024-03-11T21:36:22.898089+0000","last_fullsized":"2024-03-11T21:36:22.898089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.898004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.898004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:51.002189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":972,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422376+0000","last_change":"2024-03-11T21:34:38.596842+0000","last_active":"2024-03-11T21:36:21.422376+0000","last_peered":"2024-03-11T21:36:21.422376+0000","last_clean":"2024-03-11T21:36:21.422376+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:36:21.422376+0000","last_undegraded":"2024-03-11T21:36:21.422376+0000","last_fullsized":"2024-03-11T21:36:21.422376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:34:38.596731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.596731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":940,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386413+0000","last_change":"2024-03-11T21:34:57.751893+0000","last_active":"2024-03-11T21:36:21.386413+0000","last_peered":"2024-03-11T21:36:21.386413+0000","last_clean":"2024-03-11T21:36:21.386413+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:36:21.386413+0000","last_undegraded":"2024-03-11T21:36:21.386413+0000","last_fullsized":"2024-03-11T21:36:21.386413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.751652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":798,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383242+0000","last_change":"2024-03-11T21:34:36.837312+0000","last_active":"2024-03-11T21:36:21.383242+0000","last_peered":"2024-03-11T21:36:21.383242+0000","last_clean":"2024-03-11T21:36:21.383242+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:36:21.383242+0000","last_undegraded":"2024-03-11T21:36:21.383242+0000","last_fullsized":"2024-03-11T21:36:21.383242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.837081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.837081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":966,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422176+0000","last_change":"2024-03-11T21:35:10.391001+0000","last_active":"2024-03-11T21:36:21.422176+0000","last_peered":"2024-03-11T21:36:21.422176+0000","last_clean":"2024-03-11T21:36:21.422176+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:36:21.422176+0000","last_undegraded":"2024-03-11T21:36:21.422176+0000","last_fullsized":"2024-03-11T21:36:21.422176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.390804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.390804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:24.332372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":974,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420295+0000","last_change":"2024-03-11T21:36:14.787093+0000","last_active":"2024-03-11T21:36:21.420295+0000","last_peered":"2024-03-11T21:36:21.420295+0000","last_clean":"2024-03-11T21:36:21.420295+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:36:21.420295+0000","last_undegraded":"2024-03-11T21:36:21.420295+0000","last_fullsized":"2024-03-11T21:36:21.420295+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.786963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.786963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":894,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672406+0000","last_change":"2024-03-11T21:36:04.117071+0000","last_active":"2024-03-11T21:36:18.672406+0000","last_peered":"2024-03-11T21:36:18.672406+0000","last_clean":"2024-03-11T21:36:18.672406+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:36:18.672406+0000","last_undegraded":"2024-03-11T21:36:18.672406+0000","last_fullsized":"2024-03-11T21:36:18.672406+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:36:04.116720+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.116720+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.016493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":935,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419327+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:36:21.419327+0000","last_peered":"2024-03-11T21:36:21.419327+0000","last_clean":"2024-03-11T21:36:21.419327+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:36:21.419327+0000","last_undegraded":"2024-03-11T21:36:21.419327+0000","last_fullsized":"2024-03-11T21:36:21.419327+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":791,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608580+0000","last_change":"2024-03-11T21:33:40.988530+0000","last_active":"2024-03-11T21:36:16.608580+0000","last_peered":"2024-03-11T21:36:16.608580+0000","last_clean":"2024-03-11T21:36:16.608580+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:36:16.608580+0000","last_undegraded":"2024-03-11T21:36:16.608580+0000","last_fullsized":"2024-03-11T21:36:16.608580+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:40.988258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:40.988258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":947,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610242+0000","last_change":"2024-03-11T21:33:35.976814+0000","last_active":"2024-03-11T21:36:16.610242+0000","last_peered":"2024-03-11T21:36:16.610242+0000","last_clean":"2024-03-11T21:36:16.610242+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:36:16.610242+0000","last_undegraded":"2024-03-11T21:36:16.610242+0000","last_fullsized":"2024-03-11T21:36:16.610242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.976718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.976718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1013,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414497+0000","last_change":"2024-03-11T21:34:39.946045+0000","last_active":"2024-03-11T21:36:21.414497+0000","last_peered":"2024-03-11T21:36:21.414497+0000","last_clean":"2024-03-11T21:36:21.414497+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:36:21.414497+0000","last_undegraded":"2024-03-11T21:36:21.414497+0000","last_fullsized":"2024-03-11T21:36:21.414497+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:34:39.945944+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.945944+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":961,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383582+0000","last_change":"2024-03-11T21:36:02.740887+0000","last_active":"2024-03-11T21:36:21.383582+0000","last_peered":"2024-03-11T21:36:21.383582+0000","last_clean":"2024-03-11T21:36:21.383582+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:36:21.383582+0000","last_undegraded":"2024-03-11T21:36:21.383582+0000","last_fullsized":"2024-03-11T21:36:21.383582+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:02.740740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.740740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:08.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":934,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423097+0000","last_change":"2024-03-11T21:33:33.610158+0000","last_active":"2024-03-11T21:36:21.423097+0000","last_peered":"2024-03-11T21:36:21.423097+0000","last_clean":"2024-03-11T21:36:21.423097+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:36:21.423097+0000","last_undegraded":"2024-03-11T21:36:21.423097+0000","last_fullsized":"2024-03-11T21:36:21.423097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:33.610071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:33.610071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":798,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385497+0000","last_change":"2024-03-11T21:34:45.790219+0000","last_active":"2024-03-11T21:36:21.385497+0000","last_peered":"2024-03-11T21:36:21.385497+0000","last_clean":"2024-03-11T21:36:21.385497+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:36:21.385497+0000","last_undegraded":"2024-03-11T21:36:21.385497+0000","last_fullsized":"2024-03-11T21:36:21.385497+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.790012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.790012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1119,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609929+0000","last_change":"2024-03-11T21:34:52.046581+0000","last_active":"2024-03-11T21:36:16.609929+0000","last_peered":"2024-03-11T21:36:16.609929+0000","last_clean":"2024-03-11T21:36:16.609929+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:36:16.609929+0000","last_undegraded":"2024-03-11T21:36:16.609929+0000","last_fullsized":"2024-03-11T21:36:16.609929+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:34:52.046352+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.046352+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":839,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672458+0000","last_change":"2024-03-11T21:33:31.047676+0000","last_active":"2024-03-11T21:36:18.672458+0000","last_peered":"2024-03-11T21:36:18.672458+0000","last_clean":"2024-03-11T21:36:18.672458+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:36:18.672458+0000","last_undegraded":"2024-03-11T21:36:18.672458+0000","last_fullsized":"2024-03-11T21:36:18.672458+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:33:31.047447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:33:31.047447+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1156,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385914+0000","last_change":"2024-03-11T21:36:20.831599+0000","last_active":"2024-03-11T21:36:21.385914+0000","last_peered":"2024-03-11T21:36:21.385914+0000","last_clean":"2024-03-11T21:36:21.385914+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:36:21.385914+0000","last_undegraded":"2024-03-11T21:36:21.385914+0000","last_fullsized":"2024-03-11T21:36:21.385914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:20.831450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.831450+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.836710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":810,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673335+0000","last_change":"2024-03-11T21:33:23.993323+0000","last_active":"2024-03-11T21:36:18.673335+0000","last_peered":"2024-03-11T21:36:18.673335+0000","last_clean":"2024-03-11T21:36:18.673335+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:36:18.673335+0000","last_undegraded":"2024-03-11T21:36:18.673335+0000","last_fullsized":"2024-03-11T21:36:18.673335+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:23.993242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:23.993242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":941,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608868+0000","last_change":"2024-03-11T21:34:57.129410+0000","last_active":"2024-03-11T21:36:16.608868+0000","last_peered":"2024-03-11T21:36:16.608868+0000","last_clean":"2024-03-11T21:36:16.608868+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:36:16.608868+0000","last_undegraded":"2024-03-11T21:36:16.608868+0000","last_fullsized":"2024-03-11T21:36:16.608868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.129296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.129296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":663,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673439+0000","last_change":"2024-03-11T21:33:43.133706+0000","last_active":"2024-03-11T21:36:18.673439+0000","last_peered":"2024-03-11T21:36:18.673439+0000","last_clean":"2024-03-11T21:36:18.673439+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:36:18.673439+0000","last_undegraded":"2024-03-11T21:36:18.673439+0000","last_fullsized":"2024-03-11T21:36:18.673439+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:43.133485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:43.133485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":975,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420132+0000","last_change":"2024-03-11T21:36:19.822311+0000","last_active":"2024-03-11T21:36:21.420132+0000","last_peered":"2024-03-11T21:36:21.420132+0000","last_clean":"2024-03-11T21:36:21.420132+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:36:21.420132+0000","last_undegraded":"2024-03-11T21:36:21.420132+0000","last_fullsized":"2024-03-11T21:36:21.420132+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.822219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.822219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.316423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":995,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419188+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:36:21.419188+0000","last_peered":"2024-03-11T21:36:21.419188+0000","last_clean":"2024-03-11T21:36:21.419188+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:36:21.419188+0000","last_undegraded":"2024-03-11T21:36:21.419188+0000","last_fullsized":"2024-03-11T21:36:21.419188+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":983,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417399+0000","last_change":"2024-03-11T21:36:19.279496+0000","last_active":"2024-03-11T21:36:21.417399+0000","last_peered":"2024-03-11T21:36:21.417399+0000","last_clean":"2024-03-11T21:36:21.417399+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:36:21.417399+0000","last_undegraded":"2024-03-11T21:36:21.417399+0000","last_fullsized":"2024-03-11T21:36:21.417399+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:36:19.279370+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.279370+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.704439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":942,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422664+0000","last_change":"2024-03-11T21:33:14.693542+0000","last_active":"2024-03-11T21:36:21.422664+0000","last_peered":"2024-03-11T21:36:21.422664+0000","last_clean":"2024-03-11T21:36:21.422664+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:36:21.422664+0000","last_undegraded":"2024-03-11T21:36:21.422664+0000","last_fullsized":"2024-03-11T21:36:21.422664+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:14.693316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:14.693316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422930+0000","last_change":"2024-03-11T21:33:42.591082+0000","last_active":"2024-03-11T21:36:21.422930+0000","last_peered":"2024-03-11T21:36:21.422930+0000","last_clean":"2024-03-11T21:36:21.422930+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:36:21.422930+0000","last_undegraded":"2024-03-11T21:36:21.422930+0000","last_fullsized":"2024-03-11T21:36:21.422930+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:42.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.590963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":973,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415572+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:36:21.415572+0000","last_peered":"2024-03-11T21:36:21.415572+0000","last_clean":"2024-03-11T21:36:21.415572+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:36:21.415572+0000","last_undegraded":"2024-03-11T21:36:21.415572+0000","last_fullsized":"2024-03-11T21:36:21.415572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":974,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420507+0000","last_change":"2024-03-11T21:34:50.366890+0000","last_active":"2024-03-11T21:36:21.420507+0000","last_peered":"2024-03-11T21:36:21.420507+0000","last_clean":"2024-03-11T21:36:21.420507+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:36:21.420507+0000","last_undegraded":"2024-03-11T21:36:21.420507+0000","last_fullsized":"2024-03-11T21:36:21.420507+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:50.366805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:45.975124+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.366805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423889+0000","last_change":"2024-03-11T21:33:32.575760+0000","last_active":"2024-03-11T21:36:21.423889+0000","last_peered":"2024-03-11T21:36:21.423889+0000","last_clean":"2024-03-11T21:36:21.423889+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:36:21.423889+0000","last_undegraded":"2024-03-11T21:36:21.423889+0000","last_fullsized":"2024-03-11T21:36:21.423889+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:32.575652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:32.575652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":961,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421559+0000","last_change":"2024-03-11T21:35:05.456150+0000","last_active":"2024-03-11T21:36:21.421559+0000","last_peered":"2024-03-11T21:36:21.421559+0000","last_clean":"2024-03-11T21:36:21.421559+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:36:21.421559+0000","last_undegraded":"2024-03-11T21:36:21.421559+0000","last_fullsized":"2024-03-11T21:36:21.421559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.183993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422262+0000","last_change":"2024-03-11T21:34:39.587604+0000","last_active":"2024-03-11T21:36:21.422262+0000","last_peered":"2024-03-11T21:36:21.422262+0000","last_clean":"2024-03-11T21:36:21.422262+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:36:21.422262+0000","last_undegraded":"2024-03-11T21:36:21.422262+0000","last_fullsized":"2024-03-11T21:36:21.422262+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.587525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.587525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":981,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416448+0000","last_change":"2024-03-11T21:36:03.386633+0000","last_active":"2024-03-11T21:36:21.416448+0000","last_peered":"2024-03-11T21:36:21.416448+0000","last_clean":"2024-03-11T21:36:21.416448+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:36:21.416448+0000","last_undegraded":"2024-03-11T21:36:21.416448+0000","last_fullsized":"2024-03-11T21:36:21.416448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:36:03.386397+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.386397+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.033266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":964,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415494+0000","last_change":"2024-03-11T21:34:31.418213+0000","last_active":"2024-03-11T21:36:21.415494+0000","last_peered":"2024-03-11T21:36:21.415494+0000","last_clean":"2024-03-11T21:36:21.415494+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:36:21.415494+0000","last_undegraded":"2024-03-11T21:36:21.415494+0000","last_fullsized":"2024-03-11T21:36:21.415494+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.417992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.417992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":952,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385683+0000","last_change":"2024-03-11T21:34:58.788887+0000","last_active":"2024-03-11T21:36:21.385683+0000","last_peered":"2024-03-11T21:36:21.385683+0000","last_clean":"2024-03-11T21:36:21.385683+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:36:21.385683+0000","last_undegraded":"2024-03-11T21:36:21.385683+0000","last_fullsized":"2024-03-11T21:36:21.385683+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.788749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":934,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419216+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:36:21.419216+0000","last_peered":"2024-03-11T21:36:21.419216+0000","last_clean":"2024-03-11T21:36:21.419216+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:36:21.419216+0000","last_undegraded":"2024-03-11T21:36:21.419216+0000","last_fullsized":"2024-03-11T21:36:21.419216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":792,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421493+0000","last_change":"2024-03-11T21:34:53.520431+0000","last_active":"2024-03-11T21:36:21.421493+0000","last_peered":"2024-03-11T21:36:21.421493+0000","last_clean":"2024-03-11T21:36:21.421493+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:36:21.421493+0000","last_undegraded":"2024-03-11T21:36:21.421493+0000","last_fullsized":"2024-03-11T21:36:21.421493+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.520342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.520342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":973,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418311+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:36:21.418311+0000","last_peered":"2024-03-11T21:36:21.418311+0000","last_clean":"2024-03-11T21:36:21.418311+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:36:21.418311+0000","last_undegraded":"2024-03-11T21:36:21.418311+0000","last_fullsized":"2024-03-11T21:36:21.418311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":959,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383463+0000","last_change":"2024-03-11T21:34:31.781546+0000","last_active":"2024-03-11T21:36:21.383463+0000","last_peered":"2024-03-11T21:36:21.383463+0000","last_clean":"2024-03-11T21:36:21.383463+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:36:21.383463+0000","last_undegraded":"2024-03-11T21:36:21.383463+0000","last_fullsized":"2024-03-11T21:36:21.383463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1123,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386392+0000","last_change":"2024-03-11T21:34:55.714432+0000","last_active":"2024-03-11T21:36:21.386392+0000","last_peered":"2024-03-11T21:36:21.386392+0000","last_clean":"2024-03-11T21:36:21.386392+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:36:21.386392+0000","last_undegraded":"2024-03-11T21:36:21.386392+0000","last_fullsized":"2024-03-11T21:36:21.386392+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:34:55.714209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.714209+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:21.968789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":941,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610789+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:36:16.610789+0000","last_peered":"2024-03-11T21:36:16.610789+0000","last_clean":"2024-03-11T21:36:16.610789+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:36:16.610789+0000","last_undegraded":"2024-03-11T21:36:16.610789+0000","last_fullsized":"2024-03-11T21:36:16.610789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":808,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.413964+0000","last_change":"2024-03-11T21:35:03.464789+0000","last_active":"2024-03-11T21:36:21.413964+0000","last_peered":"2024-03-11T21:36:21.413964+0000","last_clean":"2024-03-11T21:36:21.413964+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:36:21.413964+0000","last_undegraded":"2024-03-11T21:36:21.413964+0000","last_fullsized":"2024-03-11T21:36:21.413964+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.464695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.464695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":978,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609674+0000","last_change":"2024-03-11T21:33:42.968747+0000","last_active":"2024-03-11T21:36:16.609674+0000","last_peered":"2024-03-11T21:36:16.609674+0000","last_clean":"2024-03-11T21:36:16.609674+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:36:16.609674+0000","last_undegraded":"2024-03-11T21:36:16.609674+0000","last_fullsized":"2024-03-11T21:36:16.609674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":953,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609537+0000","last_change":"2024-03-11T21:34:37.079514+0000","last_active":"2024-03-11T21:36:16.609537+0000","last_peered":"2024-03-11T21:36:16.609537+0000","last_clean":"2024-03-11T21:36:16.609537+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:36:16.609537+0000","last_undegraded":"2024-03-11T21:36:16.609537+0000","last_fullsized":"2024-03-11T21:36:16.609537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.079361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.079361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1108,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422432+0000","last_change":"2024-03-11T21:33:19.697889+0000","last_active":"2024-03-11T21:36:21.422432+0000","last_peered":"2024-03-11T21:36:21.422432+0000","last_clean":"2024-03-11T21:36:21.422432+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:36:21.422432+0000","last_undegraded":"2024-03-11T21:36:21.422432+0000","last_fullsized":"2024-03-11T21:36:21.422432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:33:19.697575+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.697575+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417684+0000","last_change":"2024-03-11T21:36:14.255051+0000","last_active":"2024-03-11T21:36:21.417684+0000","last_peered":"2024-03-11T21:36:21.417684+0000","last_clean":"2024-03-11T21:36:21.417684+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:36:21.417684+0000","last_undegraded":"2024-03-11T21:36:21.417684+0000","last_fullsized":"2024-03-11T21:36:21.417684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.254847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.254847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.154175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":659,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.674037+0000","last_change":"2024-03-11T21:33:26.034503+0000","last_active":"2024-03-11T21:36:18.674037+0000","last_peered":"2024-03-11T21:36:18.674037+0000","last_clean":"2024-03-11T21:36:18.674037+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:36:18.674037+0000","last_undegraded":"2024-03-11T21:36:18.674037+0000","last_fullsized":"2024-03-11T21:36:18.674037+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:26.034377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:26.034377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":971,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419615+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:36:21.419615+0000","last_peered":"2024-03-11T21:36:21.419615+0000","last_clean":"2024-03-11T21:36:21.419615+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:36:21.419615+0000","last_undegraded":"2024-03-11T21:36:21.419615+0000","last_fullsized":"2024-03-11T21:36:21.419615+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":971,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416795+0000","last_change":"2024-03-11T21:36:06.398266+0000","last_active":"2024-03-11T21:36:21.416795+0000","last_peered":"2024-03-11T21:36:21.416795+0000","last_clean":"2024-03-11T21:36:21.416795+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:36:21.416795+0000","last_undegraded":"2024-03-11T21:36:21.416795+0000","last_fullsized":"2024-03-11T21:36:21.416795+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.398145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.398145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.611267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423361+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:36:21.423361+0000","last_peered":"2024-03-11T21:36:21.423361+0000","last_clean":"2024-03-11T21:36:21.423361+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:36:21.423361+0000","last_undegraded":"2024-03-11T21:36:21.423361+0000","last_fullsized":"2024-03-11T21:36:21.423361+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":949,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608750+0000","last_change":"2024-03-11T21:34:41.069563+0000","last_active":"2024-03-11T21:36:16.608750+0000","last_peered":"2024-03-11T21:36:16.608750+0000","last_clean":"2024-03-11T21:36:16.608750+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:36:16.608750+0000","last_undegraded":"2024-03-11T21:36:16.608750+0000","last_fullsized":"2024-03-11T21:36:16.608750+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.069432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.069432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":797,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610367+0000","last_change":"2024-03-11T21:33:59.486720+0000","last_active":"2024-03-11T21:36:16.610367+0000","last_peered":"2024-03-11T21:36:16.610367+0000","last_clean":"2024-03-11T21:36:16.610367+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:36:16.610367+0000","last_undegraded":"2024-03-11T21:36:16.610367+0000","last_fullsized":"2024-03-11T21:36:16.610367+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:29.057966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.057966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":991,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415777+0000","last_change":"2024-03-11T21:36:10.243928+0000","last_active":"2024-03-11T21:36:21.415777+0000","last_peered":"2024-03-11T21:36:21.415777+0000","last_clean":"2024-03-11T21:36:21.415777+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:36:21.415777+0000","last_undegraded":"2024-03-11T21:36:21.415777+0000","last_fullsized":"2024-03-11T21:36:21.415777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:10.243644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.243644+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.746160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":853,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672733+0000","last_change":"2024-03-11T21:34:44.191646+0000","last_active":"2024-03-11T21:36:18.672733+0000","last_peered":"2024-03-11T21:36:18.672733+0000","last_clean":"2024-03-11T21:36:18.672733+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:36:18.672733+0000","last_undegraded":"2024-03-11T21:36:18.672733+0000","last_fullsized":"2024-03-11T21:36:18.672733+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.191432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.191432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":822,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672347+0000","last_change":"2024-03-11T21:34:32.241415+0000","last_active":"2024-03-11T21:36:18.672347+0000","last_peered":"2024-03-11T21:36:18.672347+0000","last_clean":"2024-03-11T21:36:18.672347+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:36:18.672347+0000","last_undegraded":"2024-03-11T21:36:18.672347+0000","last_fullsized":"2024-03-11T21:36:18.672347+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.241159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.241159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":933,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421052+0000","last_change":"2024-03-11T21:33:35.541846+0000","last_active":"2024-03-11T21:36:21.421052+0000","last_peered":"2024-03-11T21:36:21.421052+0000","last_clean":"2024-03-11T21:36:21.421052+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:36:21.421052+0000","last_undegraded":"2024-03-11T21:36:21.421052+0000","last_fullsized":"2024-03-11T21:36:21.421052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:35.541731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:33:35.541731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":797,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609684+0000","last_change":"2024-03-11T21:35:08.022701+0000","last_active":"2024-03-11T21:36:16.609684+0000","last_peered":"2024-03-11T21:36:16.609684+0000","last_clean":"2024-03-11T21:36:16.609684+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:36:16.609684+0000","last_undegraded":"2024-03-11T21:36:16.609684+0000","last_fullsized":"2024-03-11T21:36:16.609684+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.022536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.022536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:34.267410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":967,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610685+0000","last_change":"2024-03-11T21:33:20.954157+0000","last_active":"2024-03-11T21:36:16.610685+0000","last_peered":"2024-03-11T21:36:16.610685+0000","last_clean":"2024-03-11T21:36:16.610685+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:36:16.610685+0000","last_undegraded":"2024-03-11T21:36:16.610685+0000","last_fullsized":"2024-03-11T21:36:16.610685+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:33:20.954022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:20.954022+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":951,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386548+0000","last_change":"2024-03-11T21:33:46.468750+0000","last_active":"2024-03-11T21:36:21.386548+0000","last_peered":"2024-03-11T21:36:21.386548+0000","last_clean":"2024-03-11T21:36:21.386548+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:36:21.386548+0000","last_undegraded":"2024-03-11T21:36:21.386548+0000","last_fullsized":"2024-03-11T21:36:21.386548+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.468649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.468649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":963,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415388+0000","last_change":"2024-03-11T21:36:10.461846+0000","last_active":"2024-03-11T21:36:21.415388+0000","last_peered":"2024-03-11T21:36:21.415388+0000","last_clean":"2024-03-11T21:36:21.415388+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:36:21.415388+0000","last_undegraded":"2024-03-11T21:36:21.415388+0000","last_fullsized":"2024-03-11T21:36:21.415388+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.461724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.461724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.175158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":934,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418237+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:36:21.418237+0000","last_peered":"2024-03-11T21:36:21.418237+0000","last_clean":"2024-03-11T21:36:21.418237+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:36:21.418237+0000","last_undegraded":"2024-03-11T21:36:21.418237+0000","last_fullsized":"2024-03-11T21:36:21.418237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":800,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610105+0000","last_change":"2024-03-11T21:34:51.090840+0000","last_active":"2024-03-11T21:36:16.610105+0000","last_peered":"2024-03-11T21:36:16.610105+0000","last_clean":"2024-03-11T21:36:16.610105+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:36:16.610105+0000","last_undegraded":"2024-03-11T21:36:16.610105+0000","last_fullsized":"2024-03-11T21:36:16.610105+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:51.090589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:51.090589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":981,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.385593+0000","last_change":"2024-03-11T21:34:44.838684+0000","last_active":"2024-03-11T21:36:21.385593+0000","last_peered":"2024-03-11T21:36:21.385593+0000","last_clean":"2024-03-11T21:36:21.385593+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:36:21.385593+0000","last_undegraded":"2024-03-11T21:36:21.385593+0000","last_fullsized":"2024-03-11T21:36:21.385593+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:44.838430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.838430+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":963,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416802+0000","last_change":"2024-03-11T21:34:44.232623+0000","last_active":"2024-03-11T21:36:21.416802+0000","last_peered":"2024-03-11T21:36:21.416802+0000","last_clean":"2024-03-11T21:36:21.416802+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:36:21.416802+0000","last_undegraded":"2024-03-11T21:36:21.416802+0000","last_fullsized":"2024-03-11T21:36:21.416802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.232392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:40.946307+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.232392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":930,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.414929+0000","last_change":"2024-03-11T21:30:38.390610+0000","last_active":"2024-03-11T21:36:21.414929+0000","last_peered":"2024-03-11T21:36:21.414929+0000","last_clean":"2024-03-11T21:36:21.414929+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:36:21.414929+0000","last_undegraded":"2024-03-11T21:36:21.414929+0000","last_fullsized":"2024-03-11T21:36:21.414929+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:38.390407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:38.390407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":933,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422793+0000","last_change":"2024-03-11T21:34:42.859611+0000","last_active":"2024-03-11T21:36:21.422793+0000","last_peered":"2024-03-11T21:36:21.422793+0000","last_clean":"2024-03-11T21:36:21.422793+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:36:21.422793+0000","last_undegraded":"2024-03-11T21:36:21.422793+0000","last_fullsized":"2024-03-11T21:36:21.422793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.859451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.859451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":784,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418157+0000","last_change":"2024-03-11T21:30:54.336453+0000","last_active":"2024-03-11T21:36:21.418157+0000","last_peered":"2024-03-11T21:36:21.418157+0000","last_clean":"2024-03-11T21:36:21.418157+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:36:21.418157+0000","last_undegraded":"2024-03-11T21:36:21.418157+0000","last_fullsized":"2024-03-11T21:36:21.418157+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:54.336314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:54.336314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"389'27","reported_seq":899,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672350+0000","last_change":"2024-03-11T21:33:21.967867+0000","last_active":"2024-03-11T21:36:18.672350+0000","last_peered":"2024-03-11T21:36:18.672350+0000","last_clean":"2024-03-11T21:36:18.672350+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:36:18.672350+0000","last_undegraded":"2024-03-11T21:36:18.672350+0000","last_fullsized":"2024-03-11T21:36:18.672350+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:33:21.967756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:21.967756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":957,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.382381+0000","last_change":"2024-03-11T21:35:05.742077+0000","last_active":"2024-03-11T21:36:21.382381+0000","last_peered":"2024-03-11T21:36:21.382381+0000","last_clean":"2024-03-11T21:36:21.382381+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:36:21.382381+0000","last_undegraded":"2024-03-11T21:36:21.382381+0000","last_fullsized":"2024-03-11T21:36:21.382381+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.741828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.741828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1137,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:22.429742+0000","last_change":"2024-03-11T21:36:22.429814+0000","last_active":"2024-03-11T21:36:22.429742+0000","last_peered":"2024-03-11T21:36:22.429742+0000","last_clean":"2024-03-11T21:36:22.429742+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:36:22.429742+0000","last_undegraded":"2024-03-11T21:36:22.429742+0000","last_fullsized":"2024-03-11T21:36:22.429742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:36:22.429570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.429570+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.430380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":928,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418957+0000","last_change":"2024-03-11T21:30:27.499771+0000","last_active":"2024-03-11T21:36:21.418957+0000","last_peered":"2024-03-11T21:36:21.418957+0000","last_clean":"2024-03-11T21:36:21.418957+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:36:21.418957+0000","last_undegraded":"2024-03-11T21:36:21.418957+0000","last_fullsized":"2024-03-11T21:36:21.418957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:30:27.499431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:30:27.499431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":806,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.383244+0000","last_change":"2024-03-11T21:36:19.803577+0000","last_active":"2024-03-11T21:36:21.383244+0000","last_peered":"2024-03-11T21:36:21.383244+0000","last_clean":"2024-03-11T21:36:21.383244+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:36:21.383244+0000","last_undegraded":"2024-03-11T21:36:21.383244+0000","last_fullsized":"2024-03-11T21:36:21.383244+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.803281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.803281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.105969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":983,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.420684+0000","last_change":"2024-03-11T21:34:56.556273+0000","last_active":"2024-03-11T21:36:21.420684+0000","last_peered":"2024-03-11T21:36:21.420684+0000","last_clean":"2024-03-11T21:36:21.420684+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:36:21.420684+0000","last_undegraded":"2024-03-11T21:36:21.420684+0000","last_fullsized":"2024-03-11T21:36:21.420684+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:56.556007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.556007+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:23.524059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.421088+0000","last_change":"2024-03-11T21:33:06.710223+0000","last_active":"2024-03-11T21:36:21.421088+0000","last_peered":"2024-03-11T21:36:21.421088+0000","last_clean":"2024-03-11T21:36:21.421088+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:36:21.421088+0000","last_undegraded":"2024-03-11T21:36:21.421088+0000","last_fullsized":"2024-03-11T21:36:21.421088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:33:06.709989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:06.709989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":946,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610719+0000","last_change":"2024-03-11T21:35:01.025842+0000","last_active":"2024-03-11T21:36:16.610719+0000","last_peered":"2024-03-11T21:36:16.610719+0000","last_clean":"2024-03-11T21:36:16.610719+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:36:16.610719+0000","last_undegraded":"2024-03-11T21:36:16.610719+0000","last_fullsized":"2024-03-11T21:36:16.610719+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.025597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.025597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:22.762958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":812,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672587+0000","last_change":"2024-03-11T21:36:12.033537+0000","last_active":"2024-03-11T21:36:18.672587+0000","last_peered":"2024-03-11T21:36:18.672587+0000","last_clean":"2024-03-11T21:36:18.672587+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:36:18.672587+0000","last_undegraded":"2024-03-11T21:36:18.672587+0000","last_fullsized":"2024-03-11T21:36:18.672587+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.033259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.033259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.444912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":784,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415377+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:36:21.415377+0000","last_peered":"2024-03-11T21:36:21.415377+0000","last_clean":"2024-03-11T21:36:21.415377+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:36:21.415377+0000","last_undegraded":"2024-03-11T21:36:21.415377+0000","last_fullsized":"2024-03-11T21:36:21.415377+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":956,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.609755+0000","last_change":"2024-03-11T21:33:19.017659+0000","last_active":"2024-03-11T21:36:16.609755+0000","last_peered":"2024-03-11T21:36:16.609755+0000","last_clean":"2024-03-11T21:36:16.609755+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:36:16.609755+0000","last_undegraded":"2024-03-11T21:36:16.609755+0000","last_fullsized":"2024-03-11T21:36:16.609755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:33:19.017449+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:33:19.017449+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":963,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415331+0000","last_change":"2024-03-11T21:34:35.350319+0000","last_active":"2024-03-11T21:36:21.415331+0000","last_peered":"2024-03-11T21:36:21.415331+0000","last_clean":"2024-03-11T21:36:21.415331+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:36:21.415331+0000","last_undegraded":"2024-03-11T21:36:21.415331+0000","last_fullsized":"2024-03-11T21:36:21.415331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.350122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:46.552643+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.350122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":798,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672934+0000","last_change":"2024-03-11T21:33:36.120790+0000","last_active":"2024-03-11T21:36:18.672934+0000","last_peered":"2024-03-11T21:36:18.672934+0000","last_clean":"2024-03-11T21:36:18.672934+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:36:18.672934+0000","last_undegraded":"2024-03-11T21:36:18.672934+0000","last_fullsized":"2024-03-11T21:36:18.672934+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.120602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.120602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":954,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417618+0000","last_change":"2024-03-11T21:36:17.289970+0000","last_active":"2024-03-11T21:36:21.417618+0000","last_peered":"2024-03-11T21:36:21.417618+0000","last_clean":"2024-03-11T21:36:21.417618+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:36:21.417618+0000","last_undegraded":"2024-03-11T21:36:21.417618+0000","last_fullsized":"2024-03-11T21:36:21.417618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:17.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.289725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.559372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":801,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.413742+0000","last_change":"2024-03-11T21:35:04.183348+0000","last_active":"2024-03-11T21:36:21.413742+0000","last_peered":"2024-03-11T21:36:21.413742+0000","last_clean":"2024-03-11T21:36:21.413742+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:36:21.413742+0000","last_undegraded":"2024-03-11T21:36:21.413742+0000","last_fullsized":"2024-03-11T21:36:21.413742+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.183144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.183144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:31.008582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":995,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416613+0000","last_change":"2024-03-11T21:35:12.212273+0000","last_active":"2024-03-11T21:36:21.416613+0000","last_peered":"2024-03-11T21:36:21.416613+0000","last_clean":"2024-03-11T21:36:21.416613+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:36:21.416613+0000","last_undegraded":"2024-03-11T21:36:21.416613+0000","last_fullsized":"2024-03-11T21:36:21.416613+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:12.212159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.212159+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:25.995357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422436+0000","last_change":"2024-03-11T21:36:12.656033+0000","last_active":"2024-03-11T21:36:21.422436+0000","last_peered":"2024-03-11T21:36:21.422436+0000","last_clean":"2024-03-11T21:36:21.422436+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:36:21.422436+0000","last_undegraded":"2024-03-11T21:36:21.422436+0000","last_fullsized":"2024-03-11T21:36:21.422436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.655615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.655615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.899187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":949,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.415946+0000","last_change":"2024-03-11T21:34:57.186736+0000","last_active":"2024-03-11T21:36:21.415946+0000","last_peered":"2024-03-11T21:36:21.415946+0000","last_clean":"2024-03-11T21:36:21.415946+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:36:21.415946+0000","last_undegraded":"2024-03-11T21:36:21.415946+0000","last_fullsized":"2024-03-11T21:36:21.415946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.186509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.186509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:25.347327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":934,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423810+0000","last_change":"2024-03-11T21:34:31.520200+0000","last_active":"2024-03-11T21:36:21.423810+0000","last_peered":"2024-03-11T21:36:21.423810+0000","last_clean":"2024-03-11T21:36:21.423810+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:36:21.423810+0000","last_undegraded":"2024-03-11T21:36:21.423810+0000","last_fullsized":"2024-03-11T21:36:21.423810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.520041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.520041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":793,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419399+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:36:21.419399+0000","last_peered":"2024-03-11T21:36:21.419399+0000","last_clean":"2024-03-11T21:36:21.419399+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:36:21.419399+0000","last_undegraded":"2024-03-11T21:36:21.419399+0000","last_fullsized":"2024-03-11T21:36:21.419399+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":979,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.386215+0000","last_change":"2024-03-11T21:34:50.794115+0000","last_active":"2024-03-11T21:36:21.386215+0000","last_peered":"2024-03-11T21:36:21.386215+0000","last_clean":"2024-03-11T21:36:21.386215+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:36:21.386215+0000","last_undegraded":"2024-03-11T21:36:21.386215+0000","last_fullsized":"2024-03-11T21:36:21.386215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:34:50.793817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.793817+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418423+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:36:21.418423+0000","last_peered":"2024-03-11T21:36:21.418423+0000","last_clean":"2024-03-11T21:36:21.418423+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:36:21.418423+0000","last_undegraded":"2024-03-11T21:36:21.418423+0000","last_fullsized":"2024-03-11T21:36:21.418423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":943,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.384072+0000","last_change":"2024-03-11T21:36:09.648618+0000","last_active":"2024-03-11T21:36:21.384072+0000","last_peered":"2024-03-11T21:36:21.384072+0000","last_clean":"2024-03-11T21:36:21.384072+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:36:21.384072+0000","last_undegraded":"2024-03-11T21:36:21.384072+0000","last_fullsized":"2024-03-11T21:36:21.384072+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.648319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.648319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.702874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":949,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.608927+0000","last_change":"2024-03-11T21:34:42.092702+0000","last_active":"2024-03-11T21:36:16.608927+0000","last_peered":"2024-03-11T21:36:16.608927+0000","last_clean":"2024-03-11T21:36:16.608927+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:36:16.608927+0000","last_undegraded":"2024-03-11T21:36:16.608927+0000","last_fullsized":"2024-03-11T21:36:16.608927+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.092348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.092348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":804,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.417267+0000","last_change":"2024-03-11T21:34:47.959956+0000","last_active":"2024-03-11T21:36:21.417267+0000","last_peered":"2024-03-11T21:36:21.417267+0000","last_clean":"2024-03-11T21:36:21.417267+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:36:21.417267+0000","last_undegraded":"2024-03-11T21:36:21.417267+0000","last_fullsized":"2024-03-11T21:36:21.417267+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:47.959724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:47.959724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1014,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.384583+0000","last_change":"2024-03-11T21:36:11.636803+0000","last_active":"2024-03-11T21:36:21.384583+0000","last_peered":"2024-03-11T21:36:21.384583+0000","last_clean":"2024-03-11T21:36:21.384583+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:36:21.384583+0000","last_undegraded":"2024-03-11T21:36:21.384583+0000","last_fullsized":"2024-03-11T21:36:21.384583+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:36:11.636460+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.636460+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.421627+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":957,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.419553+0000","last_change":"2024-03-11T21:36:11.544839+0000","last_active":"2024-03-11T21:36:21.419553+0000","last_peered":"2024-03-11T21:36:21.419553+0000","last_clean":"2024-03-11T21:36:21.419553+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:36:21.419553+0000","last_undegraded":"2024-03-11T21:36:21.419553+0000","last_fullsized":"2024-03-11T21:36:21.419553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.544702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.544702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.699059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":939,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418158+0000","last_change":"2024-03-11T21:31:37.401666+0000","last_active":"2024-03-11T21:36:21.418158+0000","last_peered":"2024-03-11T21:36:21.418158+0000","last_clean":"2024-03-11T21:36:21.418158+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:36:21.418158+0000","last_undegraded":"2024-03-11T21:36:21.418158+0000","last_fullsized":"2024-03-11T21:36:21.418158+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.563585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.563585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":948,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.382798+0000","last_change":"2024-03-11T21:36:14.674630+0000","last_active":"2024-03-11T21:36:21.382798+0000","last_peered":"2024-03-11T21:36:21.382798+0000","last_clean":"2024-03-11T21:36:21.382798+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:36:21.382798+0000","last_undegraded":"2024-03-11T21:36:21.382798+0000","last_fullsized":"2024-03-11T21:36:21.382798+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.674494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.674494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:18.870558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":793,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.418767+0000","last_change":"2024-03-11T21:31:37.416701+0000","last_active":"2024-03-11T21:36:21.418767+0000","last_peered":"2024-03-11T21:36:21.418767+0000","last_clean":"2024-03-11T21:36:21.418767+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:36:21.418767+0000","last_undegraded":"2024-03-11T21:36:21.418767+0000","last_fullsized":"2024-03-11T21:36:21.418767+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:32.448219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:32.448219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":986,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610789+0000","last_change":"2024-03-11T21:33:39.970299+0000","last_active":"2024-03-11T21:36:16.610789+0000","last_peered":"2024-03-11T21:36:16.610789+0000","last_clean":"2024-03-11T21:36:16.610789+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:36:16.610789+0000","last_undegraded":"2024-03-11T21:36:16.610789+0000","last_fullsized":"2024-03-11T21:36:16.610789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:33:39.970169+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.970169+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":946,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423037+0000","last_change":"2024-03-11T21:34:35.593537+0000","last_active":"2024-03-11T21:36:21.423037+0000","last_peered":"2024-03-11T21:36:21.423037+0000","last_clean":"2024-03-11T21:36:21.423037+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:36:21.423037+0000","last_undegraded":"2024-03-11T21:36:21.423037+0000","last_fullsized":"2024-03-11T21:36:21.423037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.592784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.592784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":802,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.672664+0000","last_change":"2024-03-11T21:34:43.228208+0000","last_active":"2024-03-11T21:36:18.672664+0000","last_peered":"2024-03-11T21:36:18.672664+0000","last_clean":"2024-03-11T21:36:18.672664+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:36:18.672664+0000","last_undegraded":"2024-03-11T21:36:18.672664+0000","last_fullsized":"2024-03-11T21:36:18.672664+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.227861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.227861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":962,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.416550+0000","last_change":"2024-03-11T21:36:21.238514+0000","last_active":"2024-03-11T21:36:21.416550+0000","last_peered":"2024-03-11T21:36:21.416550+0000","last_clean":"2024-03-11T21:36:21.416550+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:36:21.416550+0000","last_undegraded":"2024-03-11T21:36:21.416550+0000","last_fullsized":"2024-03-11T21:36:21.416550+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:21.238275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.238275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.708937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":653,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:36:18.673742+0000","last_change":"2024-03-11T21:33:22.998783+0000","last_active":"2024-03-11T21:36:18.673742+0000","last_peered":"2024-03-11T21:36:18.673742+0000","last_clean":"2024-03-11T21:36:18.673742+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:36:18.673742+0000","last_undegraded":"2024-03-11T21:36:18.673742+0000","last_fullsized":"2024-03-11T21:36:18.673742+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:22.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:22.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":980,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.423200+0000","last_change":"2024-03-11T21:36:15.678133+0000","last_active":"2024-03-11T21:36:21.423200+0000","last_peered":"2024-03-11T21:36:21.423200+0000","last_clean":"2024-03-11T21:36:21.423200+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:36:21.423200+0000","last_undegraded":"2024-03-11T21:36:21.423200+0000","last_fullsized":"2024-03-11T21:36:21.423200+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:15.678027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.678027+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.603669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":955,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:36:16.610927+0000","last_change":"2024-03-11T21:33:16.013881+0000","last_active":"2024-03-11T21:36:16.610927+0000","last_peered":"2024-03-11T21:36:16.610927+0000","last_clean":"2024-03-11T21:36:16.610927+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:36:16.610927+0000","last_undegraded":"2024-03-11T21:36:16.610927+0000","last_fullsized":"2024-03-11T21:36:16.610927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:33:16.013752+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:33:16.013752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":958,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.413856+0000","last_change":"2024-03-11T21:34:53.121915+0000","last_active":"2024-03-11T21:36:21.413856+0000","last_peered":"2024-03-11T21:36:21.413856+0000","last_clean":"2024-03-11T21:36:21.413856+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:36:21.413856+0000","last_undegraded":"2024-03-11T21:36:21.413856+0000","last_fullsized":"2024-03-11T21:36:21.413856+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.121696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.121696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":933,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:36:21.422323+0000","last_change":"2024-03-11T21:35:00.524627+0000","last_active":"2024-03-11T21:36:21.422323+0000","last_peered":"2024-03-11T21:36:21.422323+0000","last_clean":"2024-03-11T21:36:21.422323+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:36:21.422323+0000","last_undegraded":"2024-03-11T21:36:21.422323+0000","last_fullsized":"2024-03-11T21:36:21.422323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:00.524430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.524430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":292,"num_pg":32,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8195872,"num_objects":169,"num_object_clones":7,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":111,"num_read_kb":10,"num_write":620,"num_write_kb":12709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":24587649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1934,"ondisk_log_size":1934,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841707,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84932,"kb_used_data":13488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93660220,"statfs":{"total":95995035648,"available":95908065280,"internally_reserved":0,"allocated":13811712,"data_stored":10185247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168756,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76232,"kb_used_data":13096,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93668920,"statfs":{"total":95995035648,"available":95916974080,"internally_reserved":0,"allocated":13410304,"data_stored":9988350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352378,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36572,"kb_used_data":8648,"kb_used_omap":27,"kb_used_meta":27876,"kb_avail":93708580,"statfs":{"total":95995035648,"available":95957585920,"internally_reserved":0,"allocated":8855552,"data_stored":5388123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":28545353},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88604,"kb_used_data":17172,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93656548,"statfs":{"total":95995035648,"available":95904305152,"internally_reserved":0,"allocated":17584128,"data_stored":14164083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378846,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76012,"kb_used_data":8728,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669140,"statfs":{"total":95995035648,"available":95917199360,"internally_reserved":0,"allocated":8937472,"data_stored":5398706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117202,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75984,"kb_used_data":8700,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669168,"statfs":{"total":95995035648,"available":95917228032,"internally_reserved":0,"allocated":8908800,"data_stored":5398153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855560,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84152,"kb_used_data":12708,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661000,"statfs":{"total":95995035648,"available":95908864000,"internally_reserved":0,"allocated":13012992,"data_stored":9399759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495806,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80024,"kb_used_data":12728,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665128,"statfs":{"total":95995035648,"available":95913091072,"internally_reserved":0,"allocated":13033472,"data_stored":9592282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8368128,"data_stored":8194735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4329472,"data_stored":4000771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":4194801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4177920,"data_stored":4000426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4538368,"data_stored":4195083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":292,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:36:26.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-11T21:36:26.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:24.816019+0000 osd.2 (osd.2) 268 : cluster 0 5.a scrub ok 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.201520+0000 osd.0 (osd.0) 305 : cluster 0 9.1e scrub starts 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.203460+0000 osd.0 (osd.0) 306 : cluster 0 9.1e scrub ok 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.252733+0000 osd.4 (osd.4) 289 : cluster 0 9.16 scrub starts 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.255831+0000 osd.4 (osd.4) 290 : cluster 0 9.16 scrub ok 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.449680+0000 mgr.y (mgr.14550) 531 : cluster 0 pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.711228+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.713392+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.733725+0000 mon.a (mon.0) 2890 : cluster 0 osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:36:26.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: audit 2024-03-11T21:36:25.739510+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:26.982 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:26 smithi203 bash[23654]: audit 2024-03-11T21:36:25.740509+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:24.816019+0000 osd.2 (osd.2) 268 : cluster 0 5.a scrub ok 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.201520+0000 osd.0 (osd.0) 305 : cluster 0 9.1e scrub starts 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.203460+0000 osd.0 (osd.0) 306 : cluster 0 9.1e scrub ok 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.252733+0000 osd.4 (osd.4) 289 : cluster 0 9.16 scrub starts 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.255831+0000 osd.4 (osd.4) 290 : cluster 0 9.16 scrub ok 2024-03-11T21:36:26.987 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.449680+0000 mgr.y (mgr.14550) 531 : cluster 0 pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.711228+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.713392+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.733725+0000 mon.a (mon.0) 2890 : cluster 0 osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: audit 2024-03-11T21:36:25.739510+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[29284]: audit 2024-03-11T21:36:25.740509+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:24.816019+0000 osd.2 (osd.2) 268 : cluster 0 5.a scrub ok 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.201520+0000 osd.0 (osd.0) 305 : cluster 0 9.1e scrub starts 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.203460+0000 osd.0 (osd.0) 306 : cluster 0 9.1e scrub ok 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.252733+0000 osd.4 (osd.4) 289 : cluster 0 9.16 scrub starts 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.255831+0000 osd.4 (osd.4) 290 : cluster 0 9.16 scrub ok 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.449680+0000 mgr.y (mgr.14550) 531 : cluster 0 pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:26.988 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.711228+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-03-11T21:36:26.989 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.713392+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-03-11T21:36:26.989 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.733725+0000 mon.a (mon.0) 2890 : cluster 0 osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:36:26.989 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: audit 2024-03-11T21:36:25.739510+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:26.989 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:26 smithi143 bash[21512]: audit 2024-03-11T21:36:25.740509+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:27.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:27.102 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:27.205 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:27.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:27.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:27.359 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:27.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:27.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:27.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:27.619 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:27.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:27.753 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:27.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:27.867 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.848942+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:36:27.867 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:25.851088+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:36:27.867 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.173958+0000 osd.0 (osd.0) 307 : cluster 0 3.1f scrub starts 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.177862+0000 osd.0 (osd.0) 308 : cluster 0 3.1f scrub ok 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.442119+0000 osd.7 (osd.7) 300 : cluster 0 5.1a scrub starts 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.444352+0000 osd.7 (osd.7) 301 : cluster 0 5.1a scrub ok 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: audit 2024-03-11T21:36:26.726223+0000 mon.a (mon.0) 2892 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.737890+0000 mon.a (mon.0) 2893 : cluster 0 osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.749050+0000 osd.3 (osd.3) 323 : cluster 0 39.a scrub starts 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: audit 2024-03-11T21:36:26.749321+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:27.868 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:27 smithi203 bash[23654]: cluster 2024-03-11T21:36:26.767121+0000 osd.3 (osd.3) 324 : cluster 0 39.a scrub ok 2024-03-11T21:36:27.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:27.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.848942+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:36:27.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:25.851088+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.173958+0000 osd.0 (osd.0) 307 : cluster 0 3.1f scrub starts 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.177862+0000 osd.0 (osd.0) 308 : cluster 0 3.1f scrub ok 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.442119+0000 osd.7 (osd.7) 300 : cluster 0 5.1a scrub starts 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.444352+0000 osd.7 (osd.7) 301 : cluster 0 5.1a scrub ok 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: audit 2024-03-11T21:36:26.726223+0000 mon.a (mon.0) 2892 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.737890+0000 mon.a (mon.0) 2893 : cluster 0 osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.749050+0000 osd.3 (osd.3) 323 : cluster 0 39.a scrub starts 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: audit 2024-03-11T21:36:26.749321+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[21512]: cluster 2024-03-11T21:36:26.767121+0000 osd.3 (osd.3) 324 : cluster 0 39.a scrub ok 2024-03-11T21:36:27.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.848942+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:25.851088+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.173958+0000 osd.0 (osd.0) 307 : cluster 0 3.1f scrub starts 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.177862+0000 osd.0 (osd.0) 308 : cluster 0 3.1f scrub ok 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.442119+0000 osd.7 (osd.7) 300 : cluster 0 5.1a scrub starts 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.444352+0000 osd.7 (osd.7) 301 : cluster 0 5.1a scrub ok 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: audit 2024-03-11T21:36:26.726223+0000 mon.a (mon.0) 2892 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.737890+0000 mon.a (mon.0) 2893 : cluster 0 osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.749050+0000 osd.3 (osd.3) 323 : cluster 0 39.a scrub starts 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: audit 2024-03-11T21:36:26.749321+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:27.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:27 smithi143 bash[29284]: cluster 2024-03-11T21:36:26.767121+0000 osd.3 (osd.3) 324 : cluster 0 39.a scrub ok 2024-03-11T21:36:27.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:28.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:28.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:28.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:28.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:28.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:28.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:28.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:28.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:28.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:28.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:28.665 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:28.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:28.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.193561+0000 osd.0 (osd.0) 309 : cluster 0 3.1b scrub starts 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.196377+0000 osd.0 (osd.0) 310 : cluster 0 3.1b scrub ok 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.450477+0000 mgr.y (mgr.14550) 532 : cluster 0 pgmap v808: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.465992+0000 osd.7 (osd.7) 302 : cluster 0 9.15 scrub starts 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.467484+0000 osd.7 (osd.7) 303 : cluster 0 9.15 scrub ok 2024-03-11T21:36:28.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:28 smithi203 bash[23654]: cluster 2024-03-11T21:36:27.766312+0000 mon.a (mon.0) 2894 : cluster 0 osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:36:28.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:28.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:28.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:28.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.193561+0000 osd.0 (osd.0) 309 : cluster 0 3.1b scrub starts 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.196377+0000 osd.0 (osd.0) 310 : cluster 0 3.1b scrub ok 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.450477+0000 mgr.y (mgr.14550) 532 : cluster 0 pgmap v808: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.465992+0000 osd.7 (osd.7) 302 : cluster 0 9.15 scrub starts 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.467484+0000 osd.7 (osd.7) 303 : cluster 0 9.15 scrub ok 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[29284]: cluster 2024-03-11T21:36:27.766312+0000 mon.a (mon.0) 2894 : cluster 0 osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.193561+0000 osd.0 (osd.0) 309 : cluster 0 3.1b scrub starts 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.196377+0000 osd.0 (osd.0) 310 : cluster 0 3.1b scrub ok 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.450477+0000 mgr.y (mgr.14550) 532 : cluster 0 pgmap v808: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.465992+0000 osd.7 (osd.7) 302 : cluster 0 9.15 scrub starts 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.467484+0000 osd.7 (osd.7) 303 : cluster 0 9.15 scrub ok 2024-03-11T21:36:28.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:28 smithi143 bash[21512]: cluster 2024-03-11T21:36:27.766312+0000 mon.a (mon.0) 2894 : cluster 0 osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:36:29.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:29.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:29.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:29.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:29.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:29.307 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:29.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:29.439 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:29.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:29.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:29.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:36:29.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:29.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:29.797 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:29.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:29.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:29.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.146075+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:36:29.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.148846+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.180025+0000 osd.0 (osd.0) 311 : cluster 0 5.5 scrub starts 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.181558+0000 osd.0 (osd.0) 312 : cluster 0 5.5 scrub ok 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: audit 2024-03-11T21:36:28.736437+0000 mgr.y (mgr.14550) 533 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.737946+0000 osd.3 (osd.3) 325 : cluster 0 118.6 scrub starts 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.741014+0000 osd.3 (osd.3) 326 : cluster 0 118.6 scrub ok 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: cluster 2024-03-11T21:36:28.774860+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: audit 2024-03-11T21:36:28.870461+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: audit 2024-03-11T21:36:28.870873+0000 mon.a (mon.0) 2896 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: audit 2024-03-11T21:36:28.871421+0000 mon.c (mon.2) 1009 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:29.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[21512]: audit 2024-03-11T21:36:28.871704+0000 mon.a (mon.0) 2897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.146075+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.148846+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.180025+0000 osd.0 (osd.0) 311 : cluster 0 5.5 scrub starts 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.181558+0000 osd.0 (osd.0) 312 : cluster 0 5.5 scrub ok 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: audit 2024-03-11T21:36:28.736437+0000 mgr.y (mgr.14550) 533 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.737946+0000 osd.3 (osd.3) 325 : cluster 0 118.6 scrub starts 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.741014+0000 osd.3 (osd.3) 326 : cluster 0 118.6 scrub ok 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: cluster 2024-03-11T21:36:28.774860+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: audit 2024-03-11T21:36:28.870461+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: audit 2024-03-11T21:36:28.870873+0000 mon.a (mon.0) 2896 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: audit 2024-03-11T21:36:28.871421+0000 mon.c (mon.2) 1009 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:29.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:29 smithi143 bash[29284]: audit 2024-03-11T21:36:28.871704+0000 mon.a (mon.0) 2897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:29.946 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:30.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.146075+0000 osd.1 (osd.1) 113 : cluster 0 5.1d scrub starts 2024-03-11T21:36:30.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.148846+0000 osd.1 (osd.1) 114 : cluster 0 5.1d scrub ok 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.180025+0000 osd.0 (osd.0) 311 : cluster 0 5.5 scrub starts 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.181558+0000 osd.0 (osd.0) 312 : cluster 0 5.5 scrub ok 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: audit 2024-03-11T21:36:28.736437+0000 mgr.y (mgr.14550) 533 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.737946+0000 osd.3 (osd.3) 325 : cluster 0 118.6 scrub starts 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.741014+0000 osd.3 (osd.3) 326 : cluster 0 118.6 scrub ok 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: cluster 2024-03-11T21:36:28.774860+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: audit 2024-03-11T21:36:28.870461+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: audit 2024-03-11T21:36:28.870873+0000 mon.a (mon.0) 2896 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: audit 2024-03-11T21:36:28.871421+0000 mon.c (mon.2) 1009 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:30.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:29 smithi203 bash[23654]: audit 2024-03-11T21:36:28.871704+0000 mon.a (mon.0) 2897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-75"}]: dispatch 2024-03-11T21:36:30.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:30.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.209 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:30.226 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:36:30.226 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:36:30.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:30.331 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:30.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:30.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:30.590 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:30.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:30.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:30.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:30.850 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:30.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.451334+0000 mgr.y (mgr.14550) 534 : cluster 0 pgmap v811: 260 pgs: 4 unknown, 256 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.508519+0000 osd.7 (osd.7) 304 : cluster 0 5.10 scrub starts 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.509831+0000 osd.7 (osd.7) 305 : cluster 0 5.10 scrub ok 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.786747+0000 mon.a (mon.0) 2898 : cluster 0 osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.032576+0000 osd.5 (osd.5) 323 : cluster 0 118.10 scrub starts 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.037338+0000 osd.5 (osd.5) 324 : cluster 0 118.10 scrub ok 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.226273+0000 osd.4 (osd.4) 291 : cluster 0 5.18 scrub starts 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.228450+0000 osd.4 (osd.4) 292 : cluster 0 5.18 scrub ok 2024-03-11T21:36:30.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.451334+0000 mgr.y (mgr.14550) 534 : cluster 0 pgmap v811: 260 pgs: 4 unknown, 256 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.508519+0000 osd.7 (osd.7) 304 : cluster 0 5.10 scrub starts 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.509831+0000 osd.7 (osd.7) 305 : cluster 0 5.10 scrub ok 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.786747+0000 mon.a (mon.0) 2898 : cluster 0 osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.032576+0000 osd.5 (osd.5) 323 : cluster 0 118.10 scrub starts 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.037338+0000 osd.5 (osd.5) 324 : cluster 0 118.10 scrub ok 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.226273+0000 osd.4 (osd.4) 291 : cluster 0 5.18 scrub starts 2024-03-11T21:36:30.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:30 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.228450+0000 osd.4 (osd.4) 292 : cluster 0 5.18 scrub ok 2024-03-11T21:36:30.953 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:30.980 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:31.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.451334+0000 mgr.y (mgr.14550) 534 : cluster 0 pgmap v811: 260 pgs: 4 unknown, 256 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:36:31.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.508519+0000 osd.7 (osd.7) 304 : cluster 0 5.10 scrub starts 2024-03-11T21:36:31.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.509831+0000 osd.7 (osd.7) 305 : cluster 0 5.10 scrub ok 2024-03-11T21:36:31.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.786747+0000 mon.a (mon.0) 2898 : cluster 0 osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:36:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.032576+0000 osd.5 (osd.5) 323 : cluster 0 118.10 scrub starts 2024-03-11T21:36:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.037338+0000 osd.5 (osd.5) 324 : cluster 0 118.10 scrub ok 2024-03-11T21:36:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.226273+0000 osd.4 (osd.4) 291 : cluster 0 5.18 scrub starts 2024-03-11T21:36:31.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:30 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.228450+0000 osd.4 (osd.4) 292 : cluster 0 5.18 scrub ok 2024-03-11T21:36:31.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:31.110 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:31.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:36:31.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:31.243 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:31.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:31.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:31.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:31.504 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:31.560 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:31.560 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:31.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.635 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.763 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:31.878 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.865999+0000 osd.2 (osd.2) 271 : cluster 0 118.9 scrub starts 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:29.868169+0000 osd.2 (osd.2) 272 : cluster 0 118.9 scrub ok 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.691163+0000 osd.3 (osd.3) 327 : cluster 0 6.b scrub starts 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.692911+0000 osd.3 (osd.3) 328 : cluster 0 6.b scrub ok 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.796193+0000 mon.a (mon.0) 2899 : cluster 0 osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: audit 2024-03-11T21:36:30.800967+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: audit 2024-03-11T21:36:30.802383+0000 mon.a (mon.0) 2900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: audit 2024-03-11T21:36:31.387192+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.391343+0000 mon.a (mon.0) 2902 : cluster 0 osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[29284]: audit 2024-03-11T21:36:31.402765+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:31.879 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.865999+0000 osd.2 (osd.2) 271 : cluster 0 118.9 scrub starts 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:29.868169+0000 osd.2 (osd.2) 272 : cluster 0 118.9 scrub ok 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.691163+0000 osd.3 (osd.3) 327 : cluster 0 6.b scrub starts 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.692911+0000 osd.3 (osd.3) 328 : cluster 0 6.b scrub ok 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.796193+0000 mon.a (mon.0) 2899 : cluster 0 osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: audit 2024-03-11T21:36:30.800967+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: audit 2024-03-11T21:36:30.802383+0000 mon.a (mon.0) 2900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: audit 2024-03-11T21:36:31.387192+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.391343+0000 mon.a (mon.0) 2902 : cluster 0 osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:36:31.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:31 smithi143 bash[21512]: audit 2024-03-11T21:36:31.402765+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:31.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.865999+0000 osd.2 (osd.2) 271 : cluster 0 118.9 scrub starts 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:29.868169+0000 osd.2 (osd.2) 272 : cluster 0 118.9 scrub ok 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.691163+0000 osd.3 (osd.3) 327 : cluster 0 6.b scrub starts 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.692911+0000 osd.3 (osd.3) 328 : cluster 0 6.b scrub ok 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.796193+0000 mon.a (mon.0) 2899 : cluster 0 osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: audit 2024-03-11T21:36:30.800967+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: audit 2024-03-11T21:36:30.802383+0000 mon.a (mon.0) 2900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: audit 2024-03-11T21:36:31.387192+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.391343+0000 mon.a (mon.0) 2902 : cluster 0 osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:36:32.011 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:31 smithi203 bash[23654]: audit 2024-03-11T21:36:31.402765+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:32.027 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:32.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:32.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:32.183 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:32.259 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:32.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:32.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:32.419 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:32.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:32.549 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:32.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:32.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:32.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:32.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.826348+0000 osd.2 (osd.2) 273 : cluster 0 39.16 scrub starts 2024-03-11T21:36:32.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:30.828319+0000 osd.2 (osd.2) 274 : cluster 0 39.16 scrub ok 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.172331+0000 osd.0 (osd.0) 313 : cluster 0 9.9 scrub starts 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.174188+0000 osd.0 (osd.0) 314 : cluster 0 9.9 scrub ok 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.451806+0000 mgr.y (mgr.14550) 535 : cluster 0 pgmap v815: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 555 B/s wr, 1 op/s 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.675004+0000 osd.3 (osd.3) 329 : cluster 0 3.8 scrub starts 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:31.676943+0000 osd.3 (osd.3) 330 : cluster 0 3.8 scrub ok 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:32.025636+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:32.028115+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: cluster 2024-03-11T21:36:32.415211+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: audit 2024-03-11T21:36:32.481216+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: audit 2024-03-11T21:36:32.481977+0000 mon.a (mon.0) 2904 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: audit 2024-03-11T21:36:32.483112+0000 mon.c (mon.2) 1013 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[21512]: audit 2024-03-11T21:36:32.483708+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.826348+0000 osd.2 (osd.2) 273 : cluster 0 39.16 scrub starts 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:30.828319+0000 osd.2 (osd.2) 274 : cluster 0 39.16 scrub ok 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.172331+0000 osd.0 (osd.0) 313 : cluster 0 9.9 scrub starts 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.174188+0000 osd.0 (osd.0) 314 : cluster 0 9.9 scrub ok 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.451806+0000 mgr.y (mgr.14550) 535 : cluster 0 pgmap v815: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 555 B/s wr, 1 op/s 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.675004+0000 osd.3 (osd.3) 329 : cluster 0 3.8 scrub starts 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:31.676943+0000 osd.3 (osd.3) 330 : cluster 0 3.8 scrub ok 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:32.025636+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:32.028115+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: cluster 2024-03-11T21:36:32.415211+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: audit 2024-03-11T21:36:32.481216+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: audit 2024-03-11T21:36:32.481977+0000 mon.a (mon.0) 2904 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: audit 2024-03-11T21:36:32.483112+0000 mon.c (mon.2) 1013 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:32 smithi143 bash[29284]: audit 2024-03-11T21:36:32.483708+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.809 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:32.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.826348+0000 osd.2 (osd.2) 273 : cluster 0 39.16 scrub starts 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:30.828319+0000 osd.2 (osd.2) 274 : cluster 0 39.16 scrub ok 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.172331+0000 osd.0 (osd.0) 313 : cluster 0 9.9 scrub starts 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.174188+0000 osd.0 (osd.0) 314 : cluster 0 9.9 scrub ok 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.451806+0000 mgr.y (mgr.14550) 535 : cluster 0 pgmap v815: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 555 B/s wr, 1 op/s 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.675004+0000 osd.3 (osd.3) 329 : cluster 0 3.8 scrub starts 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:31.676943+0000 osd.3 (osd.3) 330 : cluster 0 3.8 scrub ok 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:32.025636+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:32.028115+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: cluster 2024-03-11T21:36:32.415211+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: audit 2024-03-11T21:36:32.481216+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.944 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: audit 2024-03-11T21:36:32.481977+0000 mon.a (mon.0) 2904 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:32.945 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: audit 2024-03-11T21:36:32.483112+0000 mon.c (mon.2) 1013 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.945 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:32 smithi203 bash[23654]: audit 2024-03-11T21:36:32.483708+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-77"}]: dispatch 2024-03-11T21:36:32.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:33.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:33.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:33.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:33.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:33.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:33.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:33.475 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:33.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:33.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:33.650 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:33.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:33.734 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:33.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:33.864 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:33.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.992 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:34.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.119 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:34.222 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:34.251 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:34.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:34.379 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:34.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:34.507 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:34.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:34.623 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.042727+0000 osd.5 (osd.5) 325 : cluster 0 39.0 scrub starts 2024-03-11T21:36:34.624 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.044467+0000 osd.5 (osd.5) 326 : cluster 0 39.0 scrub ok 2024-03-11T21:36:34.624 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.413738+0000 mon.a (mon.0) 2906 : cluster 0 osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:36:34.624 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.452440+0000 mgr.y (mgr.14550) 536 : cluster 0 pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 279 B/s wr, 2 op/s 2024-03-11T21:36:34.624 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.484355+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub starts 2024-03-11T21:36:34.624 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:34 smithi203 bash[23654]: cluster 2024-03-11T21:36:33.500142+0000 osd.7 (osd.7) 307 : cluster 0 6.1c scrub ok 2024-03-11T21:36:34.641 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.042727+0000 osd.5 (osd.5) 325 : cluster 0 39.0 scrub starts 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.044467+0000 osd.5 (osd.5) 326 : cluster 0 39.0 scrub ok 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.413738+0000 mon.a (mon.0) 2906 : cluster 0 osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.452440+0000 mgr.y (mgr.14550) 536 : cluster 0 pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 279 B/s wr, 2 op/s 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.484355+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub starts 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[29284]: cluster 2024-03-11T21:36:33.500142+0000 osd.7 (osd.7) 307 : cluster 0 6.1c scrub ok 2024-03-11T21:36:34.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.042727+0000 osd.5 (osd.5) 325 : cluster 0 39.0 scrub starts 2024-03-11T21:36:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.044467+0000 osd.5 (osd.5) 326 : cluster 0 39.0 scrub ok 2024-03-11T21:36:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.413738+0000 mon.a (mon.0) 2906 : cluster 0 osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:36:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.452440+0000 mgr.y (mgr.14550) 536 : cluster 0 pgmap v818: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 279 B/s wr, 2 op/s 2024-03-11T21:36:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.484355+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub starts 2024-03-11T21:36:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:34 smithi143 bash[21512]: cluster 2024-03-11T21:36:33.500142+0000 osd.7 (osd.7) 307 : cluster 0 6.1c scrub ok 2024-03-11T21:36:34.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.771 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:34.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.902 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:35.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:35.033 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:35.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:35.163 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:35.267 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:35.292 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:35.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.432 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:35.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:35.564 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:35.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.015322+0000 osd.5 (osd.5) 327 : cluster 0 39.1a scrub starts 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.020602+0000 osd.5 (osd.5) 328 : cluster 0 39.1a scrub ok 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.072780+0000 osd.0 (osd.0) 315 : cluster 0 3.17 scrub starts 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.076350+0000 osd.0 (osd.0) 316 : cluster 0 3.17 scrub ok 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.431508+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:36:35.677 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: audit 2024-03-11T21:36:34.437641+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: audit 2024-03-11T21:36:34.439336+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.523789+0000 osd.7 (osd.7) 308 : cluster 0 1.0 scrub starts 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.526955+0000 osd.7 (osd.7) 309 : cluster 0 1.0 scrub ok 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.616907+0000 osd.6 (osd.6) 239 : cluster 0 2.11 scrub starts 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.619127+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub ok 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.742835+0000 osd.3 (osd.3) 331 : cluster 0 5.17 scrub starts 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.744989+0000 osd.3 (osd.3) 332 : cluster 0 5.17 scrub ok 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.791101+0000 osd.2 (osd.2) 275 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.793523+0000 osd.2 (osd.2) 276 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.995640+0000 osd.1 (osd.1) 117 : cluster 0 2.4 scrub starts 2024-03-11T21:36:35.678 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[21512]: cluster 2024-03-11T21:36:34.997831+0000 osd.1 (osd.1) 118 : cluster 0 2.4 scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.015322+0000 osd.5 (osd.5) 327 : cluster 0 39.1a scrub starts 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.020602+0000 osd.5 (osd.5) 328 : cluster 0 39.1a scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.072780+0000 osd.0 (osd.0) 315 : cluster 0 3.17 scrub starts 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.076350+0000 osd.0 (osd.0) 316 : cluster 0 3.17 scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.431508+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: audit 2024-03-11T21:36:34.437641+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: audit 2024-03-11T21:36:34.439336+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.523789+0000 osd.7 (osd.7) 308 : cluster 0 1.0 scrub starts 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.526955+0000 osd.7 (osd.7) 309 : cluster 0 1.0 scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.616907+0000 osd.6 (osd.6) 239 : cluster 0 2.11 scrub starts 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.619127+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.742835+0000 osd.3 (osd.3) 331 : cluster 0 5.17 scrub starts 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.744989+0000 osd.3 (osd.3) 332 : cluster 0 5.17 scrub ok 2024-03-11T21:36:35.679 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.791101+0000 osd.2 (osd.2) 275 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:36:35.680 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.793523+0000 osd.2 (osd.2) 276 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:36:35.680 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.995640+0000 osd.1 (osd.1) 117 : cluster 0 2.4 scrub starts 2024-03-11T21:36:35.680 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:35 smithi143 bash[29284]: cluster 2024-03-11T21:36:34.997831+0000 osd.1 (osd.1) 118 : cluster 0 2.4 scrub ok 2024-03-11T21:36:35.690 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.015322+0000 osd.5 (osd.5) 327 : cluster 0 39.1a scrub starts 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.020602+0000 osd.5 (osd.5) 328 : cluster 0 39.1a scrub ok 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.072780+0000 osd.0 (osd.0) 315 : cluster 0 3.17 scrub starts 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.076350+0000 osd.0 (osd.0) 316 : cluster 0 3.17 scrub ok 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.431508+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: audit 2024-03-11T21:36:34.437641+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: audit 2024-03-11T21:36:34.439336+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.523789+0000 osd.7 (osd.7) 308 : cluster 0 1.0 scrub starts 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.526955+0000 osd.7 (osd.7) 309 : cluster 0 1.0 scrub ok 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.616907+0000 osd.6 (osd.6) 239 : cluster 0 2.11 scrub starts 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.619127+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub ok 2024-03-11T21:36:35.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.742835+0000 osd.3 (osd.3) 331 : cluster 0 5.17 scrub starts 2024-03-11T21:36:35.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.744989+0000 osd.3 (osd.3) 332 : cluster 0 5.17 scrub ok 2024-03-11T21:36:35.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.791101+0000 osd.2 (osd.2) 275 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:36:35.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.793523+0000 osd.2 (osd.2) 276 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:36:35.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.995640+0000 osd.1 (osd.1) 117 : cluster 0 2.4 scrub starts 2024-03-11T21:36:35.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:35 smithi203 bash[23654]: cluster 2024-03-11T21:36:34.997831+0000 osd.1 (osd.1) 118 : cluster 0 2.4 scrub ok 2024-03-11T21:36:35.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:35.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:35.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:35.949 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:36.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:36.078 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:36.170 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:36.181 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:36.204 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:36.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.334 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:36.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:36.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: audit 2024-03-11T21:36:35.440258+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:36.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: cluster 2024-03-11T21:36:35.452217+0000 mon.a (mon.0) 2910 : cluster 0 osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: cluster 2024-03-11T21:36:35.453053+0000 mgr.y (mgr.14550) 537 : cluster 0 pgmap v821: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: audit 2024-03-11T21:36:35.458808+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: cluster 2024-03-11T21:36:35.781135+0000 osd.2 (osd.2) 277 : cluster 0 39.6 deep-scrub starts 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[21512]: cluster 2024-03-11T21:36:35.782792+0000 osd.2 (osd.2) 278 : cluster 0 39.6 deep-scrub ok 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: audit 2024-03-11T21:36:35.440258+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: cluster 2024-03-11T21:36:35.452217+0000 mon.a (mon.0) 2910 : cluster 0 osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: cluster 2024-03-11T21:36:35.453053+0000 mgr.y (mgr.14550) 537 : cluster 0 pgmap v821: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: audit 2024-03-11T21:36:35.458808+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: cluster 2024-03-11T21:36:35.781135+0000 osd.2 (osd.2) 277 : cluster 0 39.6 deep-scrub starts 2024-03-11T21:36:36.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:36 smithi143 bash[29284]: cluster 2024-03-11T21:36:35.782792+0000 osd.2 (osd.2) 278 : cluster 0 39.6 deep-scrub ok 2024-03-11T21:36:36.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:36.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:36.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:36.695 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.723 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: audit 2024-03-11T21:36:35.440258+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:36:36.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: cluster 2024-03-11T21:36:35.452217+0000 mon.a (mon.0) 2910 : cluster 0 osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:36:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: cluster 2024-03-11T21:36:35.453053+0000 mgr.y (mgr.14550) 537 : cluster 0 pgmap v821: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:36:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: audit 2024-03-11T21:36:35.458808+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:36:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: cluster 2024-03-11T21:36:35.781135+0000 osd.2 (osd.2) 277 : cluster 0 39.6 deep-scrub starts 2024-03-11T21:36:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:36 smithi203 bash[23654]: cluster 2024-03-11T21:36:35.782792+0000 osd.2 (osd.2) 278 : cluster 0 39.6 deep-scrub ok 2024-03-11T21:36:36.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:36.853 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:36.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:37.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:37.109 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:37.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:37.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:37.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:37.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:37.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.495 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:37.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.044101+0000 osd.5 (osd.5) 329 : cluster 0 2.16 scrub starts 2024-03-11T21:36:37.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.045948+0000 osd.5 (osd.5) 330 : cluster 0 2.16 scrub ok 2024-03-11T21:36:37.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.106215+0000 osd.4 (osd.4) 293 : cluster 0 39.10 scrub starts 2024-03-11T21:36:37.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.107797+0000 osd.4 (osd.4) 294 : cluster 0 39.10 scrub ok 2024-03-11T21:36:37.593 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.452290+0000 mon.a (mon.0) 2911 : cluster 0 osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: audit 2024-03-11T21:36:36.650523+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: audit 2024-03-11T21:36:36.653467+0000 mon.c (mon.2) 1016 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.678943+0000 osd.3 (osd.3) 333 : cluster 0 9.1b scrub starts 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[21512]: cluster 2024-03-11T21:36:36.681017+0000 osd.3 (osd.3) 334 : cluster 0 9.1b scrub ok 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.044101+0000 osd.5 (osd.5) 329 : cluster 0 2.16 scrub starts 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.045948+0000 osd.5 (osd.5) 330 : cluster 0 2.16 scrub ok 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.106215+0000 osd.4 (osd.4) 293 : cluster 0 39.10 scrub starts 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.107797+0000 osd.4 (osd.4) 294 : cluster 0 39.10 scrub ok 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.452290+0000 mon.a (mon.0) 2911 : cluster 0 osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: audit 2024-03-11T21:36:36.650523+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: audit 2024-03-11T21:36:36.653467+0000 mon.c (mon.2) 1016 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:37.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.678943+0000 osd.3 (osd.3) 333 : cluster 0 9.1b scrub starts 2024-03-11T21:36:37.595 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:37 smithi143 bash[29284]: cluster 2024-03-11T21:36:36.681017+0000 osd.3 (osd.3) 334 : cluster 0 9.1b scrub ok 2024-03-11T21:36:37.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.624 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:37.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.044101+0000 osd.5 (osd.5) 329 : cluster 0 2.16 scrub starts 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.045948+0000 osd.5 (osd.5) 330 : cluster 0 2.16 scrub ok 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.106215+0000 osd.4 (osd.4) 293 : cluster 0 39.10 scrub starts 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.107797+0000 osd.4 (osd.4) 294 : cluster 0 39.10 scrub ok 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.452290+0000 mon.a (mon.0) 2911 : cluster 0 osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: audit 2024-03-11T21:36:36.650523+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: audit 2024-03-11T21:36:36.653467+0000 mon.c (mon.2) 1016 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.678943+0000 osd.3 (osd.3) 333 : cluster 0 9.1b scrub starts 2024-03-11T21:36:37.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:37 smithi203 bash[23654]: cluster 2024-03-11T21:36:36.681017+0000 osd.3 (osd.3) 334 : cluster 0 9.1b scrub ok 2024-03-11T21:36:37.754 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:37.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:37.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:37.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:38.021 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:38.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:38.159 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:38.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:38.290 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:38.393 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.423 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:38.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.012843+0000 osd.5 (osd.5) 331 : cluster 0 9.6 scrub starts 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.014738+0000 osd.5 (osd.5) 332 : cluster 0 9.6 scrub ok 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.138882+0000 osd.4 (osd.4) 295 : cluster 0 6.2 scrub starts 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.140936+0000 osd.4 (osd.4) 296 : cluster 0 6.2 scrub ok 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.453919+0000 mgr.y (mgr.14550) 538 : cluster 0 pgmap v823: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 506 B/s wr, 1 op/s 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.467472+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: audit 2024-03-11T21:36:37.509282+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: audit 2024-03-11T21:36:37.509586+0000 mgr.y (mgr.14550) 539 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.685852+0000 osd.6 (osd.6) 241 : cluster 0 295.1 deep-scrub starts 2024-03-11T21:36:38.539 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[29284]: cluster 2024-03-11T21:36:37.688546+0000 osd.6 (osd.6) 242 : cluster 0 295.1 deep-scrub ok 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.012843+0000 osd.5 (osd.5) 331 : cluster 0 9.6 scrub starts 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.014738+0000 osd.5 (osd.5) 332 : cluster 0 9.6 scrub ok 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.138882+0000 osd.4 (osd.4) 295 : cluster 0 6.2 scrub starts 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.140936+0000 osd.4 (osd.4) 296 : cluster 0 6.2 scrub ok 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.453919+0000 mgr.y (mgr.14550) 538 : cluster 0 pgmap v823: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 506 B/s wr, 1 op/s 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.467472+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: audit 2024-03-11T21:36:37.509282+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: audit 2024-03-11T21:36:37.509586+0000 mgr.y (mgr.14550) 539 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.540 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.685852+0000 osd.6 (osd.6) 241 : cluster 0 295.1 deep-scrub starts 2024-03-11T21:36:38.541 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:38 smithi143 bash[21512]: cluster 2024-03-11T21:36:37.688546+0000 osd.6 (osd.6) 242 : cluster 0 295.1 deep-scrub ok 2024-03-11T21:36:38.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:38.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.012843+0000 osd.5 (osd.5) 331 : cluster 0 9.6 scrub starts 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.014738+0000 osd.5 (osd.5) 332 : cluster 0 9.6 scrub ok 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.138882+0000 osd.4 (osd.4) 295 : cluster 0 6.2 scrub starts 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.140936+0000 osd.4 (osd.4) 296 : cluster 0 6.2 scrub ok 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.453919+0000 mgr.y (mgr.14550) 538 : cluster 0 pgmap v823: 260 pgs: 5 creating+peering, 27 unknown, 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 506 B/s wr, 1 op/s 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.467472+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: audit 2024-03-11T21:36:37.509282+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.670 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: audit 2024-03-11T21:36:37.509586+0000 mgr.y (mgr.14550) 539 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "295.1"}]: dispatch 2024-03-11T21:36:38.671 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.685852+0000 osd.6 (osd.6) 241 : cluster 0 295.1 deep-scrub starts 2024-03-11T21:36:38.671 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[23654]: cluster 2024-03-11T21:36:37.688546+0000 osd.6 (osd.6) 242 : cluster 0 295.1 deep-scrub ok 2024-03-11T21:36:38.687 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:38.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.815 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:38.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:38.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:38.946 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:39.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:39.076 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:39.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:39.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:39.336 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:39.440 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:39.468 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:39.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.599 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:39.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[21512]: cluster 2024-03-11T21:36:38.631231+0000 osd.3 (osd.3) 335 : cluster 0 4.0 scrub starts 2024-03-11T21:36:39.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[21512]: cluster 2024-03-11T21:36:38.633047+0000 osd.3 (osd.3) 336 : cluster 0 4.0 scrub ok 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[21512]: audit 2024-03-11T21:36:38.742908+0000 mgr.y (mgr.14550) 540 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[21512]: cluster 2024-03-11T21:36:38.998116+0000 osd.1 (osd.1) 119 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[29284]: cluster 2024-03-11T21:36:38.631231+0000 osd.3 (osd.3) 335 : cluster 0 4.0 scrub starts 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[29284]: cluster 2024-03-11T21:36:38.633047+0000 osd.3 (osd.3) 336 : cluster 0 4.0 scrub ok 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[29284]: audit 2024-03-11T21:36:38.742908+0000 mgr.y (mgr.14550) 540 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:39.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:39 smithi143 bash[29284]: cluster 2024-03-11T21:36:38.998116+0000 osd.1 (osd.1) 119 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:36:39.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:39 smithi203 bash[23654]: cluster 2024-03-11T21:36:38.631231+0000 osd.3 (osd.3) 335 : cluster 0 4.0 scrub starts 2024-03-11T21:36:39.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:39 smithi203 bash[23654]: cluster 2024-03-11T21:36:38.633047+0000 osd.3 (osd.3) 336 : cluster 0 4.0 scrub ok 2024-03-11T21:36:39.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:39 smithi203 bash[23654]: audit 2024-03-11T21:36:38.742908+0000 mgr.y (mgr.14550) 540 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:39.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:39 smithi203 bash[23654]: cluster 2024-03-11T21:36:38.998116+0000 osd.1 (osd.1) 119 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:36:39.730 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:39.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.859 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:39.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.987 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:40.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:40.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:40.377 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:40.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.502 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:40.632 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:38.859063+0000 osd.5 (osd.5) 333 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:38.859354+0000 osd.5 (osd.5) 334 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:39.151323+0000 osd.1 (osd.1) 120 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:39.455508+0000 mgr.y (mgr.14550) 541 : cluster 0 pgmap v825: 260 pgs: 5 creating+peering, 255 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 1018 B/s wr, 2 op/s 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:39.524809+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:39.988803+0000 osd.1 (osd.1) 121 : cluster 0 9.1f scrub starts 2024-03-11T21:36:40.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[29284]: cluster 2024-03-11T21:36:39.992212+0000 osd.1 (osd.1) 122 : cluster 0 9.1f scrub ok 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:38.859063+0000 osd.5 (osd.5) 333 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:38.859354+0000 osd.5 (osd.5) 334 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:39.151323+0000 osd.1 (osd.1) 120 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:39.455508+0000 mgr.y (mgr.14550) 541 : cluster 0 pgmap v825: 260 pgs: 5 creating+peering, 255 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 1018 B/s wr, 2 op/s 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:39.524809+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:39.988803+0000 osd.1 (osd.1) 121 : cluster 0 9.1f scrub starts 2024-03-11T21:36:40.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:40 smithi143 bash[21512]: cluster 2024-03-11T21:36:39.992212+0000 osd.1 (osd.1) 122 : cluster 0 9.1f scrub ok 2024-03-11T21:36:40.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:38.859063+0000 osd.5 (osd.5) 333 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:38.859354+0000 osd.5 (osd.5) 334 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:39.151323+0000 osd.1 (osd.1) 120 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:39.455508+0000 mgr.y (mgr.14550) 541 : cluster 0 pgmap v825: 260 pgs: 5 creating+peering, 255 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 1018 B/s wr, 2 op/s 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:39.524809+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:39.988803+0000 osd.1 (osd.1) 121 : cluster 0 9.1f scrub starts 2024-03-11T21:36:40.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:40 smithi203 bash[23654]: cluster 2024-03-11T21:36:39.992212+0000 osd.1 (osd.1) 122 : cluster 0 9.1f scrub ok 2024-03-11T21:36:40.761 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:40.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:40.892 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:40.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:41.023 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:41.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:41.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:41.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:41.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:41.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:41.416 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:41.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:41.547 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:41.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:41.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:41.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.201758+0000 osd.4 (osd.4) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:36:41.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.206498+0000 osd.4 (osd.4) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:36:41.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.556720+0000 osd.3 (osd.3) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:41.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.556914+0000 osd.3 (osd.3) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:41.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.659245+0000 osd.3 (osd.3) 339 : cluster 0 6.11 scrub starts 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.661032+0000 osd.3 (osd.3) 340 : cluster 0 6.11 scrub ok 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.709691+0000 osd.6 (osd.6) 243 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:40.710901+0000 osd.6 (osd.6) 244 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.398354+0000 mon.a (mon.0) 2915 : cluster 0 osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.201758+0000 osd.4 (osd.4) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.206498+0000 osd.4 (osd.4) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:36:41.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.556720+0000 osd.3 (osd.3) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.556914+0000 osd.3 (osd.3) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.659245+0000 osd.3 (osd.3) 339 : cluster 0 6.11 scrub starts 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.661032+0000 osd.3 (osd.3) 340 : cluster 0 6.11 scrub ok 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.709691+0000 osd.6 (osd.6) 243 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:40.710901+0000 osd.6 (osd.6) 244 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:36:41.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:41 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.398354+0000 mon.a (mon.0) 2915 : cluster 0 osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:36:41.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.201758+0000 osd.4 (osd.4) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:36:41.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.206498+0000 osd.4 (osd.4) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.556720+0000 osd.3 (osd.3) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.556914+0000 osd.3 (osd.3) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.659245+0000 osd.3 (osd.3) 339 : cluster 0 6.11 scrub starts 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.661032+0000 osd.3 (osd.3) 340 : cluster 0 6.11 scrub ok 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.709691+0000 osd.6 (osd.6) 243 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:40.710901+0000 osd.6 (osd.6) 244 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:36:41.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:41 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.398354+0000 mon.a (mon.0) 2915 : cluster 0 osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:36:41.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:41.811 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:41.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:41.940 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:42.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/3205277219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a9006a4b0 msgr2=0x7f3a9010ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- 172.21.15.143:0/3205277219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a9006a4b0 0x7f3a9010ecd0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800b0a0 tx=0x7f3a8800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/3205277219 shutdown_connections 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- 172.21.15.143:0/3205277219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a9006a4b0 0x7f3a9010ecd0 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- 172.21.15.143:0/3205277219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90107e90 0x7f3a90108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- 172.21.15.143:0/3205277219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a90069b10 0x7f3a90069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/3205277219 >> 172.21.15.143:0/3205277219 conn(0x7f3a90076d80 msgr2=0x7f3a90077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:42.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/3205277219 shutdown_connections 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/3205277219 wait complete. 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 librados: starting msgr at 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 librados: starting objecter 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 Processor -- start 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- start start 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 librados: setting wanted keys 2024-03-11T21:36:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 librados: calling monclient init 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90069b10 0x7f3a90100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a9006a4b0 0x7f3a90101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 0x7f3a90101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90111630 con 0x7f3a9006a4b0 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a901114b0 con 0x7f3a90069b10 2024-03-11T21:36:42.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a901117b0 con 0x7f3a90107e90 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 0x7f3a90101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a9006a4b0 0x7f3a90101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 0x7f3a90101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:41446/0 (socket says 172.21.15.143:41446) 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 learned_addr learned my addr 172.21.15.143:0/972413340 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90069b10 0x7f3a90100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 msgr2=0x7f3a90101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 0x7f3a90101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90069b10 msgr2=0x7f3a90100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90069b10 0x7f3a90100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a901a60b0 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a90107e90 0x7f3a90101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a9006a4b0 0x7f3a90101350 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400e9f0 tx=0x7f3a8400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a77fff700 10 client.?.objecter ms_handle_connect 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a77fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8400cb70 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a901a63a0 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3a8400ee90 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 10 client.30525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84003bf0 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a901a6930 con 0x7f3a9006a4b0 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 librados: init done 2024-03-11T21:36:42.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.085+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3a90103540 con 0x7f3a9006a4b0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3a84003e90 con 0x7f3a9006a4b0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3a7806f6f0 0x7f3a78071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(540..540 src has 87..540) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f3a8408ebe0 con 0x7f3a9006a4b0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 osd_map(540..540 src has 87..540) v4 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 3 client.30525.objecter handle_osd_map got epochs [540,540] > 0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 3 client.30525.objecter handle_osd_map decoding full epoch 540 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3a7806f6f0 0x7f3a78071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 20 client.30525.objecter dump_active .. 0 homeless 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v540) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3a84094230 con 0x7f3a9006a4b0 2024-03-11T21:36:42.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 10 client.30525.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 1 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3a900629f0 0x7f3a90062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 1 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 0 1 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 4 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 10 client.30525.objecter _send_command 1 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a90102340 con 0x7f3a900629f0 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a7806f6f0 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3a7806f6f0 0x7f3a78071bb0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f3a800045b0 tx=0x7f3a8000b3c0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3a900629f0 0x7f3a90062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3a900629f0 0x7f3a90062dd0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a900629f0 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a90102340 con 0x7f3a900629f0 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 10 client.30525.objecter _finish_command 1 = system:0 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 4 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.089+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 0 1 2024-03-11T21:36:42.068 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 10 client.30525.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 2 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 2 2024-03-11T21:36:42.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9004faa0 osd=0 3 2024-03-11T21:36:42.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 0 2 2024-03-11T21:36:42.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 4 2024-03-11T21:36:42.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 10 client.30525.objecter _send_command 2 2024-03-11T21:36:42.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a90062470 con 0x7f3a900629f0 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a90062470 con 0x7f3a900629f0 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 10 client.30525.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 4 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 0 2 2024-03-11T21:36:42.085 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:36:42.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 10 client.30525.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 3 2024-03-11T21:36:42.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3a9010aa00 0x7f3a901aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 1 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 4 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 10 client.30525.objecter _send_command 3 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a9002d9a0 con 0x7f3a9010aa00 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3a9010aa00 0x7f3a901aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a9010aa00 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.109+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3a9010aa00 0x7f3a901aabe0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.113+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a9002d9a0 con 0x7f3a9010aa00 2024-03-11T21:36:42.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.113+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:36:42.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.113+0000 7f3a77fff700 10 client.30525.objecter _finish_command 3 = system:0 2024-03-11T21:36:42.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.113+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 4 2024-03-11T21:36:42.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.113+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 1 3 2024-03-11T21:36:42.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 10 client.30525.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 4 2024-03-11T21:36:42.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 4 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9002d830 osd=1 3 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 1 4 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 4 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 10 client.30525.objecter _send_command 4 2024-03-11T21:36:42.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.129+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901ab120 con 0x7f3a9010aa00 2024-03-11T21:36:42.107 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901ab120 con 0x7f3a9010aa00 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 10 client.30525.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 4 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 1 4 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 10 client.30525.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 5 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3a9010a0e0 0x7f3a901ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 5 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 2 5 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 4 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 10 client.30525.objecter _send_command 5 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a9002d390 con 0x7f3a9010a0e0 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3a9010a0e0 0x7f3a901ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a9010a0e0 2024-03-11T21:36:42.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3a9010a0e0 0x7f3a901ae4c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a9002d390 con 0x7f3a9010a0e0 2024-03-11T21:36:42.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:36:42.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 10 client.30525.objecter _finish_command 5 = system:0 2024-03-11T21:36:42.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 4 2024-03-11T21:36:42.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.133+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 2 5 2024-03-11T21:36:42.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 10 client.30525.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 6 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 6 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a90109f70 osd=2 3 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 2 6 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 4 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 10 client.30525.objecter _send_command 6 2024-03-11T21:36:42.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901aea00 con 0x7f3a9010a0e0 2024-03-11T21:36:42.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901aea00 con 0x7f3a9010a0e0 2024-03-11T21:36:42.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 10 client.30525.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 4 2024-03-11T21:36:42.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 2 6 2024-03-11T21:36:42.129 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 10 client.30525.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 7 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3a9002ca70 0x7f3a901b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 7 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 3 7 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 4 2024-03-11T21:36:42.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 10 client.30525.objecter _send_command 7 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a901ac070 con 0x7f3a9002ca70 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3a9002ca70 0x7f3a901b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a9002ca70 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.153+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3a9002ca70 0x7f3a901b1b90 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.157+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a901ac070 con 0x7f3a9002ca70 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.157+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.157+0000 7f3a77fff700 10 client.30525.objecter _finish_command 7 = system:0 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.157+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 4 2024-03-11T21:36:42.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.157+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 3 7 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 10 client.30525.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 8 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 8 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a9002c900 osd=3 3 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 3 8 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 4 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 10 client.30525.objecter _send_command 8 2024-03-11T21:36:42.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901ac070 con 0x7f3a9002ca70 2024-03-11T21:36:42.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901ac070 con 0x7f3a9002ca70 2024-03-11T21:36:42.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a77fff700 10 client.30525.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 4 2024-03-11T21:36:42.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 3 8 2024-03-11T21:36:42.150 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 10 client.30525.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 9 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3a901af1d0 0x7f3a901af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 9 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 4 9 2024-03-11T21:36:42.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 4 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.173+0000 7f3a95410700 10 client.30525.objecter _send_command 9 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a901b7c20 con 0x7f3a901af1d0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3a901af1d0 0x7f3a901af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a901af1d0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3a901af1d0 0x7f3a901af610 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a901b7c20 con 0x7f3a901af1d0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 10 client.30525.objecter _finish_command 9 = system:0 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 4 2024-03-11T21:36:42.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.177+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 4 9 2024-03-11T21:36:42.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:42.180 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 10 client.30525.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 10 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 10 2024-03-11T21:36:42.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901ac070 osd=4 3 2024-03-11T21:36:42.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 4 10 2024-03-11T21:36:42.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 4 2024-03-11T21:36:42.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 10 client.30525.objecter _send_command 10 2024-03-11T21:36:42.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.213+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901b5300 con 0x7f3a901af1d0 2024-03-11T21:36:42.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:42.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901b5300 con 0x7f3a901af1d0 2024-03-11T21:36:42.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a77fff700 10 client.30525.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 4 2024-03-11T21:36:42.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 4 10 2024-03-11T21:36:42.194 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:36:42.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 10 client.30525.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 11 2024-03-11T21:36:42.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3a901b7240 0x7f3a901b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 11 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 5 11 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 4 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 10 client.30525.objecter _send_command 11 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.217+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a901bafc0 con 0x7f3a901b7240 2024-03-11T21:36:42.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3a901b7240 0x7f3a901b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:42.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a901b7240 2024-03-11T21:36:42.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a8e7fc700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3a901b7240 0x7f3a901b7680 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a901bafc0 con 0x7f3a901b7240 2024-03-11T21:36:42.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:36:42.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 10 client.30525.objecter _finish_command 11 = system:0 2024-03-11T21:36:42.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 4 2024-03-11T21:36:42.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.221+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 5 11 2024-03-11T21:36:42.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 10 client.30525.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 12 2024-03-11T21:36:42.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 12 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b5300 osd=5 3 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 5 12 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 4 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 10 client.30525.objecter _send_command 12 2024-03-11T21:36:42.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.241+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901b9e40 con 0x7f3a901b7240 2024-03-11T21:36:42.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901b9e40 con 0x7f3a901b7240 2024-03-11T21:36:42.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 10 client.30525.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 4 2024-03-11T21:36:42.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 5 12 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 10 client.30525.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 13 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3a901ba5c0 0x7f3a901baa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 13 2024-03-11T21:36:42.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 6 13 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 4 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 10 client.30525.objecter _send_command 13 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a901be360 con 0x7f3a901ba5c0 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3a901ba5c0 0x7f3a901baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a901ba5c0 2024-03-11T21:36:42.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a8f7fe700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3a901ba5c0 0x7f3a901baa20 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a901be360 con 0x7f3a901ba5c0 2024-03-11T21:36:42.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:36:42.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 10 client.30525.objecter _finish_command 13 = system:0 2024-03-11T21:36:42.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 4 2024-03-11T21:36:42.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.245+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 6 13 2024-03-11T21:36:42.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 10 client.30525.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 14 2024-03-11T21:36:42.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 14 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b9e40 osd=6 3 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 6 14 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 4 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 10 client.30525.objecter _send_command 14 2024-03-11T21:36:42.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.265+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901b5470 con 0x7f3a901ba5c0 2024-03-11T21:36:42.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901b5470 con 0x7f3a901ba5c0 2024-03-11T21:36:42.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 10 client.30525.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 4 2024-03-11T21:36:42.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 6 14 2024-03-11T21:36:42.244 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:36:42.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 10 client.30525.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:42.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 15 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3a901b6850 0x7f3a901b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 15 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 7 15 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 4 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 10 client.30525.objecter _send_command 15 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a901b6210 con 0x7f3a901b6850 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3a901b6850 0x7f3a901b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a77fff700 10 client.30525.objecter ms_handle_connect 0x7f3a901b6850 2024-03-11T21:36:42.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.269+0000 7f3a8effd700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3a901b6850 0x7f3a901b5b90 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:42.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.273+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a901b6210 con 0x7f3a901b6850 2024-03-11T21:36:42.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.273+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:36:42.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.273+0000 7f3a77fff700 10 client.30525.objecter _finish_command 15 = system:0 2024-03-11T21:36:42.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.273+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 4 2024-03-11T21:36:42.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.273+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 7 15 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 10 client.30525.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign -1 16 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 20 client.30525.objecter _get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 15 client.30525.objecter _session_command_op_remove -1 16 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 20 client.30525.objecter get_session s=0x7f3a901b5470 osd=7 3 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 15 client.30525.objecter _session_command_op_assign 7 16 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 4 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 10 client.30525.objecter _send_command 16 2024-03-11T21:36:42.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.289+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3a901b6210 con 0x7f3a901b6850 2024-03-11T21:36:42.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a77fff700 1 -- 172.21.15.143:0/972413340 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3a901b6210 con 0x7f3a901b6850 2024-03-11T21:36:42.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a77fff700 10 client.30525.objecter ms_dispatch 0x7f3a90001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:42.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a77fff700 10 client.30525.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:42.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a77fff700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 4 2024-03-11T21:36:42.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a77fff700 15 client.30525.objecter _session_command_op_remove 7 16 2024-03-11T21:36:42.270 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 librados: watch_flush enter 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 librados: watch_flush exit 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.0 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3a900629f0 msgr2=0x7f3a90062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3a900629f0 0x7f3a90062dd0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9004faa0 osd=0 1 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.1 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3a9010aa00 msgr2=0x7f3a901aabe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3a9010aa00 0x7f3a901aabe0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002d830 osd=1 1 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.2 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3a9010a0e0 msgr2=0x7f3a901ae4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3a9010a0e0 0x7f3a901ae4c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a90109f70 osd=2 1 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.3 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3a9002ca70 msgr2=0x7f3a901b1b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3a9002ca70 0x7f3a901b1b90 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a9002c900 osd=3 1 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.4 2024-03-11T21:36:42.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3a901af1d0 msgr2=0x7f3a901af610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3a901af1d0 0x7f3a901af610 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901ac070 osd=4 1 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.5 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3a901b7240 msgr2=0x7f3a901b7680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3a901b7240 0x7f3a901b7680 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5300 osd=5 1 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.6 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3a901ba5c0 msgr2=0x7f3a901baa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3a901ba5c0 0x7f3a901baa20 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b9e40 osd=6 1 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter close_session for osd.7 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3a901b6850 msgr2=0x7f3a901b5b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3a901b6850 0x7f3a901b5b90 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter put_session s=0x7f3a901b5470 osd=7 1 2024-03-11T21:36:42.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 20 client.30525.objecter shutdown clearing up homeless session... 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 10 client.30525.objecter successfully canceled tick 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 reap_dead start 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3a7806f6f0 msgr2=0x7f3a78071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 reap_dead start 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3a7806f6f0 0x7f3a78071bb0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f3a800045b0 tx=0x7f3a8000b3c0 comp rx=0 tx=0).stop 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a9006a4b0 msgr2=0x7f3a90101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 --2- 172.21.15.143:0/972413340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a9006a4b0 0x7f3a90101350 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400e9f0 tx=0x7f3a8400ed00 comp rx=0 tx=0).stop 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 reap_dead start 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 reap_dead start 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a8f7fe700 1 -- 172.21.15.143:0/972413340 reap_dead start 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 shutdown_connections 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.293+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 >> 172.21.15.143:0/972413340 conn(0x7f3a90076d80 msgr2=0x7f3a90108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.297+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 shutdown_connections 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.297+0000 7f3a95410700 1 -- 172.21.15.143:0/972413340 wait complete. 2024-03-11T21:36:42.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:42.297+0000 7f3a95410700 1 librados: shutdown 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:42.275 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:42.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:42.321 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:42.424 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:42.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:42.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.244373+0000 osd.4 (osd.4) 299 : cluster 0 5.1 scrub starts 2024-03-11T21:36:42.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.246768+0000 osd.4 (osd.4) 300 : cluster 0 5.1 scrub ok 2024-03-11T21:36:42.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.456144+0000 mgr.y (mgr.14550) 542 : cluster 0 pgmap v828: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 1022 B/s wr, 2 op/s 2024-03-11T21:36:42.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.697888+0000 osd.6 (osd.6) 245 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:36:42.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: cluster 2024-03-11T21:36:41.700566+0000 osd.6 (osd.6) 246 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:36:42.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: audit 2024-03-11T21:36:41.995748+0000 mon.c (mon.2) 1018 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:42.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:42 smithi203 bash[23654]: audit 2024-03-11T21:36:42.092532+0000 mon.a (mon.0) 2916 : audit 0 from='client.? 172.21.15.143:0/972413340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:42.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:42.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:42.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.244373+0000 osd.4 (osd.4) 299 : cluster 0 5.1 scrub starts 2024-03-11T21:36:42.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.246768+0000 osd.4 (osd.4) 300 : cluster 0 5.1 scrub ok 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.456144+0000 mgr.y (mgr.14550) 542 : cluster 0 pgmap v828: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 1022 B/s wr, 2 op/s 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.697888+0000 osd.6 (osd.6) 245 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: cluster 2024-03-11T21:36:41.700566+0000 osd.6 (osd.6) 246 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: audit 2024-03-11T21:36:41.995748+0000 mon.c (mon.2) 1018 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[21512]: audit 2024-03-11T21:36:42.092532+0000 mon.a (mon.0) 2916 : audit 0 from='client.? 172.21.15.143:0/972413340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.244373+0000 osd.4 (osd.4) 299 : cluster 0 5.1 scrub starts 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.246768+0000 osd.4 (osd.4) 300 : cluster 0 5.1 scrub ok 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.456144+0000 mgr.y (mgr.14550) 542 : cluster 0 pgmap v828: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 1022 B/s wr, 2 op/s 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.697888+0000 osd.6 (osd.6) 245 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: cluster 2024-03-11T21:36:41.700566+0000 osd.6 (osd.6) 246 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: audit 2024-03-11T21:36:41.995748+0000 mon.c (mon.2) 1018 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:42.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:42 smithi143 bash[29284]: audit 2024-03-11T21:36:42.092532+0000 mon.a (mon.0) 2916 : audit 0 from='client.? 172.21.15.143:0/972413340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:42.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:42.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:42.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:42.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:42.953 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/1289513732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 msgr2=0x7f2bf0108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/1289513732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf0108c70 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0099c0 tx=0x7f2bdc009cd0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/1289513732 shutdown_connections 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/1289513732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0109350 0x7f2bf0110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/1289513732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0104270 0x7f2bf0104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/1289513732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf0108c70 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/1289513732 >> 172.21.15.143:0/1289513732 conn(0x7f2bf00ffb00 msgr2=0x7f2bf0101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:43.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/1289513732 shutdown_connections 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/1289513732 wait complete. 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 librados: starting msgr at 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 librados: starting objecter 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 Processor -- start 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- start start 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 librados: setting wanted keys 2024-03-11T21:36:43.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 librados: calling monclient init 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 0x7f2bf019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0113850 con 0x7f2bf0104c20 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf01136d0 con 0x7f2bf0109350 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf01139d0 con 0x7f2bf0104270 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:43.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:41456/0 (socket says 172.21.15.143:41456) 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 -- 172.21.15.143:0/3026121857 learned_addr learned my addr 172.21.15.143:0/3026121857 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2beffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf527f700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 0x7f2bf019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 -- 172.21.15.143:0/3026121857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 msgr2=0x7f2bf019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 0x7f2bf019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 -- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 msgr2=0x7f2bf019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 -- 172.21.15.143:0/3026121857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf019ee90 con 0x7f2bf0104270 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf527f700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 0x7f2bf019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2beffff700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf4a7e700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f2be400b530 tx=0x7f2be400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bedffb700 10 client.?.objecter ms_handle_connect 0x7f2bf0104270 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bedffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4015020 con 0x7f2bf0104270 2024-03-11T21:36:43.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf019f180 con 0x7f2bf0104270 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2be4011cb0 con 0x7f2bf0104270 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4010d90 con 0x7f2bf0104270 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 10 client.41732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf01a6a00 con 0x7f2bf0104270 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.033+0000 7f2bf6ce2700 1 librados: init done 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf0105750 con 0x7f2bf0104270 2024-03-11T21:36:43.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2be4018020 con 0x7f2bf0104270 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 0x7f2be0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(540..540 src has 87..540) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f2be408e430 con 0x7f2bf0104270 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 10 client.41732.objecter ms_dispatch 0x7f2bf0001550 osd_map(540..540 src has 87..540) v4 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 3 client.41732.objecter handle_osd_map got epochs [540,540] > 0 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 3 client.41732.objecter handle_osd_map decoding full epoch 540 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2beffff700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 0x7f2be0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 20 client.41732.objecter dump_active .. 0 homeless 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2bedffb700 10 client.41732.objecter ms_handle_connect 0x7f2be006f820 2024-03-11T21:36:43.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.037+0000 7f2beffff700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 0x7f2be0071ce0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc000c00 tx=0x7f2bdc019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:43.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.041+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2be401f020 con 0x7f2bf0104270 2024-03-11T21:36:43.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:43.077 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:43.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.206 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:43.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.253+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7f2bf019b1e0 con 0x7f2bf0104270 2024-03-11T21:36:43.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:43.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:43.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:43.466 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:43.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.497+0000 7f2bedffb700 1 -- 172.21.15.143:0/3026121857 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v541) v1 ==== 129+0+0 (secure 0 0 0) 0x7f2be405ead0 con 0x7f2bf0104270 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 10 librados: watch_flush enter 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 10 librados: watch_flush exit 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 20 client.41732.objecter shutdown clearing up homeless session... 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 10 client.41732.objecter successfully canceled tick 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 msgr2=0x7f2be0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 0x7f2be0071ce0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc000c00 tx=0x7f2bdc019040 comp rx=0 tx=0).stop 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 msgr2=0x7f2bf0199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:43.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f2be400b530 tx=0x7f2be400b840 comp rx=0 tx=0).stop 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 shutdown_connections 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2be006f820 0x7f2be0071ce0 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bf0109350 0x7f2bf019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf0104270 0x7f2bf0199e80 secure :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f2be400b530 tx=0x7f2be400b840 comp rx=0 tx=0).stop 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 --2- 172.21.15.143:0/3026121857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf0104c20 0x7f2bf019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 >> 172.21.15.143:0/3026121857 conn(0x7f2bf00ffb00 msgr2=0x7f2bf0100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 shutdown_connections 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 -- 172.21.15.143:0/3026121857 wait complete. 2024-03-11T21:36:43.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:43.501+0000 7f2bf6ce2700 1 librados: shutdown 2024-03-11T21:36:43.478 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pgp_num to 1 2024-03-11T21:36:43.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:43.595 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:43.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:43.708 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:42.715915+0000 osd.6 (osd.6) 247 : cluster 0 2.7 scrub starts 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:42.718304+0000 osd.6 (osd.6) 248 : cluster 0 2.7 scrub ok 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:42.730702+0000 osd.7 (osd.7) 310 : cluster 0 5.4 scrub starts 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:42.732348+0000 osd.7 (osd.7) 311 : cluster 0 5.4 scrub ok 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.077676+0000 osd.1 (osd.1) 123 : cluster 0 39.1c scrub starts 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.079409+0000 osd.1 (osd.1) 124 : cluster 0 39.1c scrub ok 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: audit 2024-03-11T21:36:43.257109+0000 mon.c (mon.2) 1019 : audit 1 from='client.? 172.21.15.143:0/3026121857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[21512]: audit 2024-03-11T21:36:43.257442+0000 mon.a (mon.0) 2917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:42.715915+0000 osd.6 (osd.6) 247 : cluster 0 2.7 scrub starts 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:42.718304+0000 osd.6 (osd.6) 248 : cluster 0 2.7 scrub ok 2024-03-11T21:36:43.709 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:42.730702+0000 osd.7 (osd.7) 310 : cluster 0 5.4 scrub starts 2024-03-11T21:36:43.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:42.732348+0000 osd.7 (osd.7) 311 : cluster 0 5.4 scrub ok 2024-03-11T21:36:43.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.077676+0000 osd.1 (osd.1) 123 : cluster 0 39.1c scrub starts 2024-03-11T21:36:43.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.079409+0000 osd.1 (osd.1) 124 : cluster 0 39.1c scrub ok 2024-03-11T21:36:43.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: audit 2024-03-11T21:36:43.257109+0000 mon.c (mon.2) 1019 : audit 1 from='client.? 172.21.15.143:0/3026121857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.710 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:43 smithi143 bash[29284]: audit 2024-03-11T21:36:43.257442+0000 mon.a (mon.0) 2917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.721 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:43.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:42.715915+0000 osd.6 (osd.6) 247 : cluster 0 2.7 scrub starts 2024-03-11T21:36:43.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:42.718304+0000 osd.6 (osd.6) 248 : cluster 0 2.7 scrub ok 2024-03-11T21:36:43.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:42.730702+0000 osd.7 (osd.7) 310 : cluster 0 5.4 scrub starts 2024-03-11T21:36:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:42.732348+0000 osd.7 (osd.7) 311 : cluster 0 5.4 scrub ok 2024-03-11T21:36:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.077676+0000 osd.1 (osd.1) 123 : cluster 0 39.1c scrub starts 2024-03-11T21:36:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.079409+0000 osd.1 (osd.1) 124 : cluster 0 39.1c scrub ok 2024-03-11T21:36:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: audit 2024-03-11T21:36:43.257109+0000 mon.c (mon.2) 1019 : audit 1 from='client.? 172.21.15.143:0/3026121857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:43 smithi203 bash[23654]: audit 2024-03-11T21:36:43.257442+0000 mon.a (mon.0) 2917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:36:43.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:43.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:43.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.983 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:44.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:44.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:44.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:44.242 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:44.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:44.370 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:44.474 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:44.499 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:44.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:44.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:44.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:44.745 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.143582+0000 osd.0 (osd.0) 317 : cluster 0 39.12 scrub starts 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.145132+0000 osd.0 (osd.0) 318 : cluster 0 39.12 scrub ok 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.458092+0000 mgr.y (mgr.14550) 543 : cluster 0 pgmap v829: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 852 B/s wr, 3 op/s 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[29284]: audit 2024-03-11T21:36:43.496533+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.505275+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.143582+0000 osd.0 (osd.0) 317 : cluster 0 39.12 scrub starts 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.145132+0000 osd.0 (osd.0) 318 : cluster 0 39.12 scrub ok 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.458092+0000 mgr.y (mgr.14550) 543 : cluster 0 pgmap v829: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 852 B/s wr, 3 op/s 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[21512]: audit 2024-03-11T21:36:43.496533+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:36:44.746 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:44 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.505275+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:36:44.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:44.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:44 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.143582+0000 osd.0 (osd.0) 317 : cluster 0 39.12 scrub starts 2024-03-11T21:36:44.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:44 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.145132+0000 osd.0 (osd.0) 318 : cluster 0 39.12 scrub ok 2024-03-11T21:36:44.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:44 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.458092+0000 mgr.y (mgr.14550) 543 : cluster 0 pgmap v829: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 852 B/s wr, 3 op/s 2024-03-11T21:36:44.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:44 smithi203 bash[23654]: audit 2024-03-11T21:36:43.496533+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:36:44.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:44 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.505275+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:36:44.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:44.892 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:44.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:45.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.147 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:45.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:45.276 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:45.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:45.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:45.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:45.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:45.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- 172.21.15.143:0/2403102086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c101740 msgr2=0x7f1e4c101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 --2- 172.21.15.143:0/2403102086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c101740 0x7f1e4c101b20 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00b0d0 tx=0x7f1e3c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- 172.21.15.143:0/2403102086 shutdown_connections 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 --2- 172.21.15.143:0/2403102086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c106300 0x7f1e4c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 --2- 172.21.15.143:0/2403102086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c101740 0x7f1e4c101b20 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 --2- 172.21.15.143:0/2403102086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- 172.21.15.143:0/2403102086 >> 172.21.15.143:0/2403102086 conn(0x7f1e4c077380 msgr2=0x7f1e4c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- 172.21.15.143:0/2403102086 shutdown_connections 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- 172.21.15.143:0/2403102086 wait complete. 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 librados: starting msgr at 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 librados: starting objecter 2024-03-11T21:36:45.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 Processor -- start 2024-03-11T21:36:45.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 -- start start 2024-03-11T21:36:45.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 librados: setting wanted keys 2024-03-11T21:36:45.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.585+0000 7f1e50879700 1 librados: calling monclient init 2024-03-11T21:36:45.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c101740 0x7f1e4c103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 0x7f1e4c104970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c110820 con 0x7f1e4c1020f0 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c1106a0 con 0x7f1e4c101740 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c1109a0 con 0x7f1e4c106300 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33050/0 (socket says 172.21.15.143:33050) 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 -- 172.21.15.143:0/1821646898 learned_addr learned my addr 172.21.15.143:0/1821646898 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e4ad9d700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 0x7f1e4c104970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e4a59c700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c101740 0x7f1e4c103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 -- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 msgr2=0x7f1e4c104970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 0x7f1e4c104970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 -- 172.21.15.143:0/1821646898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c101740 msgr2=0x7f1e4c103ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c101740 0x7f1e4c103ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 -- 172.21.15.143:0/1821646898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c1a6420 con 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e4ad9d700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 0x7f1e4c104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e49d9b700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400ea30 tx=0x7f1e3400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 10 client.?.objecter ms_handle_connect 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400cbd0 con 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f1e340044a0 con 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c1a6710 con 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e34003d30 con 0x7f1e4c1020f0 2024-03-11T21:36:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 10 client.30537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c1a6ca0 con 0x7f1e4c1020f0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 librados: init done 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c0ffc00 con 0x7f1e4c1020f0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1e34003ef0 con 0x7f1e4c1020f0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 0x7f1e38071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(541..541 src has 87..541) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f1e3408ed20 con 0x7f1e4c1020f0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 10 client.30537.objecter ms_dispatch 0x7f1e4c001550 osd_map(541..541 src has 87..541) v4 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 3 client.30537.objecter handle_osd_map got epochs [541,541] > 0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 3 client.30537.objecter handle_osd_map decoding full epoch 541 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e4a59c700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 0x7f1e38071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:45.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.589+0000 7f1e437fe700 20 client.30537.objecter dump_active .. 0 homeless 2024-03-11T21:36:45.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.593+0000 7f1e437fe700 10 client.30537.objecter ms_handle_connect 0x7f1e3806f820 2024-03-11T21:36:45.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.593+0000 7f1e4a59c700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 0x7f1e38071ce0 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009e60 tx=0x7f1e3c009780 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:45.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.597+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1e34016330 con 0x7f1e4c1020f0 2024-03-11T21:36:45.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.656 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:45.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.847339+0000 osd.2 (osd.2) 279 : cluster 0 5.b scrub starts 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:43.849573+0000 osd.2 (osd.2) 280 : cluster 0 5.b scrub ok 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.621397+0000 osd.6 (osd.6) 249 : cluster 0 9.18 scrub starts 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.624677+0000 osd.6 (osd.6) 250 : cluster 0 9.18 scrub ok 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.811702+0000 osd.2 (osd.2) 281 : cluster 0 118.15 scrub starts 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.814348+0000 osd.2 (osd.2) 282 : cluster 0 118.15 scrub ok 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.951041+0000 osd.5 (osd.5) 335 : cluster 0 9.a scrub starts 2024-03-11T21:36:45.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:44.952519+0000 osd.5 (osd.5) 336 : cluster 0 9.a scrub ok 2024-03-11T21:36:45.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.052685+0000 osd.1 (osd.1) 125 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:36:45.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:45 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.054806+0000 osd.1 (osd.1) 126 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:36:45.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:45.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:45.813+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f1e4c105420 con 0x7f1e4c1020f0 2024-03-11T21:36:45.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.847339+0000 osd.2 (osd.2) 279 : cluster 0 5.b scrub starts 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:43.849573+0000 osd.2 (osd.2) 280 : cluster 0 5.b scrub ok 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.621397+0000 osd.6 (osd.6) 249 : cluster 0 9.18 scrub starts 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.624677+0000 osd.6 (osd.6) 250 : cluster 0 9.18 scrub ok 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.811702+0000 osd.2 (osd.2) 281 : cluster 0 118.15 scrub starts 2024-03-11T21:36:45.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.814348+0000 osd.2 (osd.2) 282 : cluster 0 118.15 scrub ok 2024-03-11T21:36:45.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.951041+0000 osd.5 (osd.5) 335 : cluster 0 9.a scrub starts 2024-03-11T21:36:45.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:44.952519+0000 osd.5 (osd.5) 336 : cluster 0 9.a scrub ok 2024-03-11T21:36:45.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.052685+0000 osd.1 (osd.1) 125 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:36:45.902 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.054806+0000 osd.1 (osd.1) 126 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:36:45.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.847339+0000 osd.2 (osd.2) 279 : cluster 0 5.b scrub starts 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:43.849573+0000 osd.2 (osd.2) 280 : cluster 0 5.b scrub ok 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.621397+0000 osd.6 (osd.6) 249 : cluster 0 9.18 scrub starts 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.624677+0000 osd.6 (osd.6) 250 : cluster 0 9.18 scrub ok 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.811702+0000 osd.2 (osd.2) 281 : cluster 0 118.15 scrub starts 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.814348+0000 osd.2 (osd.2) 282 : cluster 0 118.15 scrub ok 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.951041+0000 osd.5 (osd.5) 335 : cluster 0 9.a scrub starts 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:44.952519+0000 osd.5 (osd.5) 336 : cluster 0 9.a scrub ok 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.052685+0000 osd.1 (osd.1) 125 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:36:45.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:45 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.054806+0000 osd.1 (osd.1) 126 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:36:45.913 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:46.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:46.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:46.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:46.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:46.303 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:46.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.389+0000 7f1e437fe700 1 -- 172.21.15.143:0/1821646898 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v542) v1 ==== 87+0+0 (secure 0 0 0) 0x7f1e3405b9f0 con 0x7f1e4c1020f0 2024-03-11T21:36:46.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 10 librados: watch_flush enter 2024-03-11T21:36:46.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 10 librados: watch_flush exit 2024-03-11T21:36:46.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 20 client.30537.objecter shutdown clearing up homeless session... 2024-03-11T21:36:46.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 10 client.30537.objecter successfully canceled tick 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 msgr2=0x7f1e38071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 0x7f1e38071ce0 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009e60 tx=0x7f1e3c009780 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 msgr2=0x7f1e4c104430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400ea30 tx=0x7f1e3400ed40 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 shutdown_connections 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f1e3806f820 0x7f1e38071ce0 unknown :-1 s=CLOSED pgs=3388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e4c106300 0x7f1e4c104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e4c101740 0x7f1e4c103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 --2- 172.21.15.143:0/1821646898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e4c1020f0 0x7f1e4c104430 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:46.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 >> 172.21.15.143:0/1821646898 conn(0x7f1e4c077380 msgr2=0x7f1e4c0fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:46.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 shutdown_connections 2024-03-11T21:36:46.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.393+0000 7f1e50879700 1 -- 172.21.15.143:0/1821646898 wait complete. 2024-03-11T21:36:46.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:46.397+0000 7f1e50879700 1 librados: shutdown 2024-03-11T21:36:46.372 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:36:46.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:46.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:46.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:46.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:46.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.218233+0000 osd.0 (osd.0) 319 : cluster 0 9.5 scrub starts 2024-03-11T21:36:46.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.219979+0000 osd.0 (osd.0) 320 : cluster 0 9.5 scrub ok 2024-03-11T21:36:46.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.222645+0000 osd.4 (osd.4) 301 : cluster 0 6.e deep-scrub starts 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.224246+0000 osd.4 (osd.4) 302 : cluster 0 6.e deep-scrub ok 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.458954+0000 mgr.y (mgr.14550) 544 : cluster 0 pgmap v831: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.771701+0000 osd.2 (osd.2) 283 : cluster 0 118.d deep-scrub starts 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:45.778388+0000 osd.2 (osd.2) 284 : cluster 0 118.d deep-scrub ok 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: audit 2024-03-11T21:36:45.819258+0000 mon.a (mon.0) 2920 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:46.391268+0000 mon.a (mon.0) 2921 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: audit 2024-03-11T21:36:46.393335+0000 mon.a (mon.0) 2922 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:46.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:46 smithi203 bash[23654]: cluster 2024-03-11T21:36:46.400448+0000 mon.a (mon.0) 2923 : cluster 0 osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:36:46.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.218233+0000 osd.0 (osd.0) 319 : cluster 0 9.5 scrub starts 2024-03-11T21:36:46.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.219979+0000 osd.0 (osd.0) 320 : cluster 0 9.5 scrub ok 2024-03-11T21:36:46.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.222645+0000 osd.4 (osd.4) 301 : cluster 0 6.e deep-scrub starts 2024-03-11T21:36:46.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.224246+0000 osd.4 (osd.4) 302 : cluster 0 6.e deep-scrub ok 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.458954+0000 mgr.y (mgr.14550) 544 : cluster 0 pgmap v831: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.771701+0000 osd.2 (osd.2) 283 : cluster 0 118.d deep-scrub starts 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:45.778388+0000 osd.2 (osd.2) 284 : cluster 0 118.d deep-scrub ok 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: audit 2024-03-11T21:36:45.819258+0000 mon.a (mon.0) 2920 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:46.391268+0000 mon.a (mon.0) 2921 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: audit 2024-03-11T21:36:46.393335+0000 mon.a (mon.0) 2922 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[21512]: cluster 2024-03-11T21:36:46.400448+0000 mon.a (mon.0) 2923 : cluster 0 osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.218233+0000 osd.0 (osd.0) 319 : cluster 0 9.5 scrub starts 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.219979+0000 osd.0 (osd.0) 320 : cluster 0 9.5 scrub ok 2024-03-11T21:36:46.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.222645+0000 osd.4 (osd.4) 301 : cluster 0 6.e deep-scrub starts 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.224246+0000 osd.4 (osd.4) 302 : cluster 0 6.e deep-scrub ok 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.458954+0000 mgr.y (mgr.14550) 544 : cluster 0 pgmap v831: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.771701+0000 osd.2 (osd.2) 283 : cluster 0 118.d deep-scrub starts 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:45.778388+0000 osd.2 (osd.2) 284 : cluster 0 118.d deep-scrub ok 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: audit 2024-03-11T21:36:45.819258+0000 mon.a (mon.0) 2920 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:46.391268+0000 mon.a (mon.0) 2921 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: audit 2024-03-11T21:36:46.393335+0000 mon.a (mon.0) 2922 : audit 1 from='client.? 172.21.15.143:0/1821646898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:46.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:46 smithi143 bash[29284]: cluster 2024-03-11T21:36:46.400448+0000 mon.a (mon.0) 2923 : cluster 0 osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:36:46.690 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:46.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:46.819 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:46.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:46.952 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:47.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:47.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:47.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:36:47.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.201 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:47.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:47.322 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:47.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.454 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:47.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:47.584 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:47.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:47.695 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:47 smithi203 bash[23654]: cluster 2024-03-11T21:36:46.796751+0000 osd.7 (osd.7) 312 : cluster 0 9.13 deep-scrub starts 2024-03-11T21:36:47.695 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:47 smithi203 bash[23654]: cluster 2024-03-11T21:36:46.798820+0000 osd.7 (osd.7) 313 : cluster 0 9.13 deep-scrub ok 2024-03-11T21:36:47.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:47.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:47.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:47.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:47 smithi143 bash[21512]: cluster 2024-03-11T21:36:46.796751+0000 osd.7 (osd.7) 312 : cluster 0 9.13 deep-scrub starts 2024-03-11T21:36:47.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:47 smithi143 bash[21512]: cluster 2024-03-11T21:36:46.798820+0000 osd.7 (osd.7) 313 : cluster 0 9.13 deep-scrub ok 2024-03-11T21:36:47.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:47 smithi143 bash[29284]: cluster 2024-03-11T21:36:46.796751+0000 osd.7 (osd.7) 312 : cluster 0 9.13 deep-scrub starts 2024-03-11T21:36:47.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:47 smithi143 bash[29284]: cluster 2024-03-11T21:36:46.798820+0000 osd.7 (osd.7) 313 : cluster 0 9.13 deep-scrub ok 2024-03-11T21:36:47.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:47.968 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:48.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:48.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:48.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.220 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:48.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:48.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:48.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:48.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.591 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:48 smithi203 bash[23654]: cluster 2024-03-11T21:36:47.186988+0000 osd.0 (osd.0) 321 : cluster 0 3.12 scrub starts 2024-03-11T21:36:48.591 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:48 smithi203 bash[23654]: cluster 2024-03-11T21:36:47.459680+0000 mgr.y (mgr.14550) 545 : cluster 0 pgmap v833: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-11T21:36:48.591 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:48 smithi203 bash[23654]: audit 2024-03-11T21:36:48.031481+0000 mon.a (mon.0) 2924 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.591 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:48 smithi203 bash[23654]: audit 2024-03-11T21:36:48.039831+0000 mon.a (mon.0) 2925 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.605 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:48.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.726 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[29284]: cluster 2024-03-11T21:36:47.186988+0000 osd.0 (osd.0) 321 : cluster 0 3.12 scrub starts 2024-03-11T21:36:48.726 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[29284]: cluster 2024-03-11T21:36:47.459680+0000 mgr.y (mgr.14550) 545 : cluster 0 pgmap v833: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-11T21:36:48.726 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[29284]: audit 2024-03-11T21:36:48.031481+0000 mon.a (mon.0) 2924 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.727 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[29284]: audit 2024-03-11T21:36:48.039831+0000 mon.a (mon.0) 2925 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[21512]: cluster 2024-03-11T21:36:47.186988+0000 osd.0 (osd.0) 321 : cluster 0 3.12 scrub starts 2024-03-11T21:36:48.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[21512]: cluster 2024-03-11T21:36:47.459680+0000 mgr.y (mgr.14550) 545 : cluster 0 pgmap v833: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-11T21:36:48.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[21512]: audit 2024-03-11T21:36:48.031481+0000 mon.a (mon.0) 2924 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:48 smithi143 bash[21512]: audit 2024-03-11T21:36:48.039831+0000 mon.a (mon.0) 2925 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:48.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:48.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:48.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:48.858 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:48.926 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:36:48.927 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:36:48.927 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:36:48.927 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:36:48.927 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:36:48.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.986 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:49.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:49.115 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:49.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:49.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.381 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:49.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.510 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:49.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:49.640 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:49.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:49 smithi143 bash[29284]: audit 2024-03-11T21:36:48.752203+0000 mgr.y (mgr.14550) 546 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:49.683 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:49 smithi143 bash[21512]: audit 2024-03-11T21:36:48.752203+0000 mgr.y (mgr.14550) 546 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:49.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:49.754 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:49 smithi203 bash[23654]: audit 2024-03-11T21:36:48.752203+0000 mgr.y (mgr.14550) 546 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:49.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.898 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:50.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.029 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:50.052 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:36:50.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:50.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:50.383 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.409 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:50.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:50.523 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:50 smithi143 bash[29284]: cluster 2024-03-11T21:36:49.460979+0000 mgr.y (mgr.14550) 547 : cluster 0 pgmap v834: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:50.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:50 smithi143 bash[21512]: cluster 2024-03-11T21:36:49.460979+0000 mgr.y (mgr.14550) 547 : cluster 0 pgmap v834: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:50.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:36:50.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:50.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.771 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:50 smithi203 bash[23654]: cluster 2024-03-11T21:36:49.460979+0000 mgr.y (mgr.14550) 547 : cluster 0 pgmap v834: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:50.792 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:50.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:50.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:51.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:51.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:51.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:51.315 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:51.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.448 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:51.551 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.576 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:51.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:51.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:51.828 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:51.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:51.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:52.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:52.086 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:52.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:52.216 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:52.242 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:36:52.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:36:52.243 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-11T21:36:52.243 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-11T21:36:52.243 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:36:52.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:52.329 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:36:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:52.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:52.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:52.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:52.562 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:52.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:52.596 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:52.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:52.718 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:52.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:52.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: cluster 2024-03-11T21:36:51.462540+0000 mgr.y (mgr.14550) 548 : cluster 0 pgmap v835: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:36:52.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: cluster 2024-03-11T21:36:51.478500+0000 osd.1 (osd.1) 127 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:52.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: cluster 2024-03-11T21:36:51.478815+0000 osd.1 (osd.1) 128 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:52.832 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:51.693912+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:51.695292+0000 mon.c (mon.2) 1020 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:52.142808+0000 mon.a (mon.0) 2927 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:52.149298+0000 mon.a (mon.0) 2928 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: cluster 2024-03-11T21:36:52.410282+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:52.671282+0000 mon.c (mon.2) 1021 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:52.671959+0000 mon.c (mon.2) 1022 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:52.833 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:52 smithi203 bash[23654]: audit 2024-03-11T21:36:52.677503+0000 mon.a (mon.0) 2930 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.846 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:52.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: cluster 2024-03-11T21:36:51.462540+0000 mgr.y (mgr.14550) 548 : cluster 0 pgmap v835: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:36:52.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: cluster 2024-03-11T21:36:51.478500+0000 osd.1 (osd.1) 127 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: cluster 2024-03-11T21:36:51.478815+0000 osd.1 (osd.1) 128 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:51.693912+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:51.695292+0000 mon.c (mon.2) 1020 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:52.142808+0000 mon.a (mon.0) 2927 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:52.149298+0000 mon.a (mon.0) 2928 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: cluster 2024-03-11T21:36:52.410282+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:52.671282+0000 mon.c (mon.2) 1021 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:52.671959+0000 mon.c (mon.2) 1022 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[29284]: audit 2024-03-11T21:36:52.677503+0000 mon.a (mon.0) 2930 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: cluster 2024-03-11T21:36:51.462540+0000 mgr.y (mgr.14550) 548 : cluster 0 pgmap v835: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: cluster 2024-03-11T21:36:51.478500+0000 osd.1 (osd.1) 127 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: cluster 2024-03-11T21:36:51.478815+0000 osd.1 (osd.1) 128 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:51.693912+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:51.695292+0000 mon.c (mon.2) 1020 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:52.142808+0000 mon.a (mon.0) 2927 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:52.149298+0000 mon.a (mon.0) 2928 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: cluster 2024-03-11T21:36:52.410282+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:52.671282+0000 mon.c (mon.2) 1021 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:52.671959+0000 mon.c (mon.2) 1022 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:52.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:52 smithi143 bash[21512]: audit 2024-03-11T21:36:52.677503+0000 mon.a (mon.0) 2930 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:36:52.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:52.976 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:53.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.107 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:53.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:53.238 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:53.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.368 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:53.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:53.499 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:53.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:53.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.763 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:53.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.877 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:53 smithi203 bash[23654]: cluster 2024-03-11T21:36:52.471239+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:53.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:53 smithi203 bash[23654]: cluster 2024-03-11T21:36:52.471580+0000 osd.6 (osd.6) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:53.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:53 smithi203 bash[23654]: cluster 2024-03-11T21:36:53.415814+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:36:53.895 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:53.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[21512]: cluster 2024-03-11T21:36:52.471239+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:53.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[21512]: cluster 2024-03-11T21:36:52.471580+0000 osd.6 (osd.6) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:53.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[21512]: cluster 2024-03-11T21:36:53.415814+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:36:53.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[29284]: cluster 2024-03-11T21:36:52.471239+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:53.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[29284]: cluster 2024-03-11T21:36:52.471580+0000 osd.6 (osd.6) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:53.934 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:53 smithi143 bash[29284]: cluster 2024-03-11T21:36:53.415814+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:36:53.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:54.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:54.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:54.145 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:54.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:54.270 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:54.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.401 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:54.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.532 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:54.604 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:54.605 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:54.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:54.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:54.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.787 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:54.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:54.919 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:54.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:54 smithi203 bash[23654]: cluster 2024-03-11T21:36:53.463467+0000 mgr.y (mgr.14550) 549 : cluster 0 pgmap v838: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:36:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:54.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:54 smithi143 bash[29284]: cluster 2024-03-11T21:36:53.463467+0000 mgr.y (mgr.14550) 549 : cluster 0 pgmap v838: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:36:54.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:54 smithi143 bash[21512]: cluster 2024-03-11T21:36:53.463467+0000 mgr.y (mgr.14550) 549 : cluster 0 pgmap v838: 260 pgs: 1 active+clean+scrubbing, 259 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:36:55.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:55.048 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:55.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:55.177 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:55.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:55.308 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:55.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:55.440 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:55.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:55.572 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:55.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:55.702 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:55.805 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:55.825 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:55.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:55.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:56.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:56.086 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:56.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:56.215 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:56.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:56.344 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:56.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:56.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.603 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:56.648 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:56.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:56 smithi203 bash[23654]: cluster 2024-03-11T21:36:55.464444+0000 mgr.y (mgr.14550) 550 : cluster 0 pgmap v839: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:56.734 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:56.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:56.864 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:56.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:56 smithi143 bash[29284]: cluster 2024-03-11T21:36:55.464444+0000 mgr.y (mgr.14550) 550 : cluster 0 pgmap v839: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:56.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:56 smithi143 bash[21512]: cluster 2024-03-11T21:36:55.464444+0000 mgr.y (mgr.14550) 550 : cluster 0 pgmap v839: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:36:56.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:56.994 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:57.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:57.125 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:57.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:57.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:57.390 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:57.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:57.519 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:57.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:57.633 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[38255]: debug 2024-03-11T21:36:57.590+0000 7f027fb40700 -1 snap_mapper.add_oid found existing snaps mapped on 295:858e15ba:test-rados-api-smithi143-71773-80::foo:2, removing 2024-03-11T21:36:57.651 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:57.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.765 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[150181]: debug 2024-03-11T21:36:57.596+0000 7f3a3338e700 -1 snap_mapper.add_oid found existing snaps mapped on 295:858e15ba:test-rados-api-smithi143-71773-80::foo:2, removing 2024-03-11T21:36:57.765 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[49102]: debug 2024-03-11T21:36:57.596+0000 7f5575dbb700 -1 snap_mapper.add_oid found existing snaps mapped on 295:858e15ba:test-rados-api-smithi143-71773-80::foo:2, removing 2024-03-11T21:36:57.765 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: cluster 2024-03-11T21:36:56.561023+0000 osd.2 (osd.2) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: cluster 2024-03-11T21:36:56.561367+0000 osd.2 (osd.2) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: cluster 2024-03-11T21:36:57.416367+0000 mon.a (mon.0) 2932 : cluster 0 osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: audit 2024-03-11T21:36:57.605879+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: audit 2024-03-11T21:36:57.606582+0000 mon.a (mon.0) 2933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: audit 2024-03-11T21:36:57.607548+0000 mon.c (mon.2) 1024 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[29284]: audit 2024-03-11T21:36:57.607968+0000 mon.a (mon.0) 2934 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: cluster 2024-03-11T21:36:56.561023+0000 osd.2 (osd.2) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: cluster 2024-03-11T21:36:56.561367+0000 osd.2 (osd.2) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: cluster 2024-03-11T21:36:57.416367+0000 mon.a (mon.0) 2932 : cluster 0 osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: audit 2024-03-11T21:36:57.605879+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:57.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: audit 2024-03-11T21:36:57.606582+0000 mon.a (mon.0) 2933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:57.767 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: audit 2024-03-11T21:36:57.607548+0000 mon.c (mon.2) 1024 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:57.767 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:57 smithi143 bash[21512]: audit 2024-03-11T21:36:57.607968+0000 mon.a (mon.0) 2934 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:57.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:58.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:58.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: cluster 2024-03-11T21:36:56.561023+0000 osd.2 (osd.2) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: cluster 2024-03-11T21:36:56.561367+0000 osd.2 (osd.2) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: cluster 2024-03-11T21:36:57.416367+0000 mon.a (mon.0) 2932 : cluster 0 osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: audit 2024-03-11T21:36:57.605879+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: audit 2024-03-11T21:36:57.606582+0000 mon.a (mon.0) 2933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: audit 2024-03-11T21:36:57.607548+0000 mon.c (mon.2) 1024 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:58.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:57 smithi203 bash[23654]: audit 2024-03-11T21:36:57.607968+0000 mon.a (mon.0) 2934 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-79"}]: dispatch 2024-03-11T21:36:58.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:58.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:58.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:58.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.304 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:58.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:58.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:58.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:58.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.692 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:58.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.807 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:36:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:36:58.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:58.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:59.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:58 smithi203 bash[23654]: cluster 2024-03-11T21:36:57.197066+0000 osd.0 (osd.0) 322 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:58 smithi203 bash[23654]: cluster 2024-03-11T21:36:57.197403+0000 osd.0 (osd.0) 323 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:58 smithi203 bash[23654]: cluster 2024-03-11T21:36:57.465236+0000 mgr.y (mgr.14550) 551 : cluster 0 pgmap v841: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:59.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:58 smithi203 bash[23654]: cluster 2024-03-11T21:36:58.422174+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:36:59.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.069 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[21512]: cluster 2024-03-11T21:36:57.197066+0000 osd.0 (osd.0) 322 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:59.069 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[21512]: cluster 2024-03-11T21:36:57.197403+0000 osd.0 (osd.0) 323 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:59.069 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[21512]: cluster 2024-03-11T21:36:57.465236+0000 mgr.y (mgr.14550) 551 : cluster 0 pgmap v841: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:59.070 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[21512]: cluster 2024-03-11T21:36:58.422174+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:36:59.070 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[29284]: cluster 2024-03-11T21:36:57.197066+0000 osd.0 (osd.0) 322 : cluster 0 purged_snaps scrub starts 2024-03-11T21:36:59.070 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[29284]: cluster 2024-03-11T21:36:57.197403+0000 osd.0 (osd.0) 323 : cluster 0 purged_snaps scrub ok 2024-03-11T21:36:59.070 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[29284]: cluster 2024-03-11T21:36:57.465236+0000 mgr.y (mgr.14550) 551 : cluster 0 pgmap v841: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:36:59.070 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:58 smithi143 bash[29284]: cluster 2024-03-11T21:36:58.422174+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:36:59.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:59.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:59.211 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:59.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- 172.21.15.143:0/3292546069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c109370 msgr2=0x7fdc8c110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3292546069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c109370 0x7fdc8c110f40 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009fc0 tx=0x7fdc8400b330 comp rx=0 tx=0).stop 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- 172.21.15.143:0/3292546069 shutdown_connections 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3292546069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c109370 0x7fdc8c110f40 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3292546069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c104290 0x7fdc8c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3292546069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104c40 0x7fdc8c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- 172.21.15.143:0/3292546069 >> 172.21.15.143:0/3292546069 conn(0x7fdc8c0ffb40 msgr2=0x7fdc8c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- 172.21.15.143:0/3292546069 shutdown_connections 2024-03-11T21:36:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- 172.21.15.143:0/3292546069 wait complete. 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 librados: starting msgr at 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 librados: starting objecter 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 Processor -- start 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- start start 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 librados: setting wanted keys 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 librados: calling monclient init 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104290 0x7fdc8c199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 0x7fdc8c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c109370 0x7fdc8c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c113870 con 0x7fdc8c104c40 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1136f0 con 0x7fdc8c104290 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc91abf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1139f0 con 0x7fdc8c109370 2024-03-11T21:36:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc8affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 0x7fdc8c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc8affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 0x7fdc8c19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42230/0 (socket says 172.21.15.143:42230) 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc8affd700 1 -- 172.21.15.143:0/3860154396 learned_addr learned my addr 172.21.15.143:0/3860154396 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.340+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c109370 0x7fdc8c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104290 0x7fdc8c199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c109370 msgr2=0x7fdc8c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c109370 0x7fdc8c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104290 msgr2=0x7fdc8c199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104290 0x7fdc8c199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c19eda0 con 0x7fdc8c104c40 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc8c109370 0x7fdc8c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc8c104290 0x7fdc8c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 0x7fdc8c19a2d0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00c8f0 tx=0x7fdc7c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 10 client.?.objecter ms_handle_connect 0x7fdc8c104c40 2024-03-11T21:36:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c007940 con 0x7fdc8c104c40 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fdc7c00f450 con 0x7fdc8c104c40 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c19f090 con 0x7fdc8c104c40 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c018660 con 0x7fdc8c104c40 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc91abf700 10 client.30543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1a6910 con 0x7fdc8c104c40 2024-03-11T21:36:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc91abf700 1 librados: init done 2024-03-11T21:36:59.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdc8c04faa0 con 0x7fdc8c104c40 2024-03-11T21:36:59.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fdc7c007ae0 con 0x7fdc8c104c40 2024-03-11T21:36:59.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc7406f820 0x7fdc74071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(546..546 src has 87..546) v4 ==== 9244+0+0 (secure 0 0 0) 0x7fdc7c08ea90 con 0x7fdc8c104c40 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 osd_map(546..546 src has 87..546) v4 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 3 client.30543.objecter handle_osd_map got epochs [546,546] > 0 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 3 client.30543.objecter handle_osd_map decoding full epoch 546 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc7406f820 0x7fdc74071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 20 client.30543.objecter dump_active .. 0 homeless 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.344+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v546) v1 ==== 54+0+16 (secure 0 0 0) 0x7fdc7c014030 con 0x7fdc8c104c40 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 10 client.30543.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 1 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fdc8c0629f0 0x7fdc8c062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 1 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 0 1 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 4 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 10 client.30543.objecter _send_command 1 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1a9d30 con 0x7fdc8c0629f0 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc7406f820 0x7fdc74071ce0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009750 tx=0x7fdc80009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc7406f820 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fdc8c0629f0 0x7fdc8c062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fdc8c0629f0 0x7fdc8c062dd0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c0629f0 2024-03-11T21:36:59.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1a9d30 con 0x7fdc8c0629f0 2024-03-11T21:36:59.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:36:59.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 1 = system:0 2024-03-11T21:36:59.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 4 2024-03-11T21:36:59.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.348+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 0 1 2024-03-11T21:36:59.336 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 10 client.30543.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 2 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 2 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c19b0f0 osd=0 3 2024-03-11T21:36:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 0 2 2024-03-11T21:36:59.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 4 2024-03-11T21:36:59.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 10 client.30543.objecter _send_command 2 2024-03-11T21:36:59.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c062470 con 0x7fdc8c0629f0 2024-03-11T21:36:59.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c062470 con 0x7fdc8c0629f0 2024-03-11T21:36:59.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 4 2024-03-11T21:36:59.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 0 2 2024-03-11T21:36:59.345 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:36:59.346 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 10 client.30543.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.346 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 3 2024-03-11T21:36:59.346 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fdc8c02c6b0 0x7fdc8c02cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 1 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 4 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 10 client.30543.objecter _send_command 3 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c10bce0 con 0x7fdc8c02c6b0 2024-03-11T21:36:59.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fdc8c02c6b0 0x7fdc8c02cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c02c6b0 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fdc8c02c6b0 0x7fdc8c02cab0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c10bce0 con 0x7fdc8c02c6b0 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 3 = system:0 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 4 2024-03-11T21:36:59.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.368+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 1 3 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 10 client.30543.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 4 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 4 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c10b660 osd=1 3 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 1 4 2024-03-11T21:36:59.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 4 2024-03-11T21:36:59.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 10 client.30543.objecter _send_command 4 2024-03-11T21:36:59.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c10b0e0 con 0x7fdc8c02c6b0 2024-03-11T21:36:59.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c10b0e0 con 0x7fdc8c02c6b0 2024-03-11T21:36:59.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 4 2024-03-11T21:36:59.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 1 4 2024-03-11T21:36:59.366 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 10 client.30543.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 5 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdc8c1abef0 0x7fdc8c1ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 5 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 2 5 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 4 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 10 client.30543.objecter _send_command 5 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c02dcd0 con 0x7fdc8c1abef0 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdc8c1abef0 0x7fdc8c1ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1abef0 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.388+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdc8c1abef0 0x7fdc8c1ae2e0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.392+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c02dcd0 con 0x7fdc8c1abef0 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.392+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:36:59.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.392+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 5 = system:0 2024-03-11T21:36:59.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.392+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 4 2024-03-11T21:36:59.369 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.392+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 2 5 2024-03-11T21:36:59.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 10 client.30543.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 6 2024-03-11T21:36:59.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 6 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c10b4e0 osd=2 3 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 2 6 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 4 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 10 client.30543.objecter _send_command 6 2024-03-11T21:36:59.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c02d2a0 con 0x7fdc8c1abef0 2024-03-11T21:36:59.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c02d2a0 con 0x7fdc8c1abef0 2024-03-11T21:36:59.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 4 2024-03-11T21:36:59.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 2 6 2024-03-11T21:36:59.387 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:36:59.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 10 client.30543.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 7 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fdc8c1af9a0 0x7fdc8c1b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 7 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 3 7 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 4 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 10 client.30543.objecter _send_command 7 2024-03-11T21:36:59.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1a9ae0 con 0x7fdc8c1af9a0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.408+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fdc8c1af9a0 0x7fdc8c1b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1af9a0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fdc8c1af9a0 0x7fdc8c1b1d90 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1a9ae0 con 0x7fdc8c1af9a0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 7 = system:0 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 4 2024-03-11T21:36:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.412+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 3 7 2024-03-11T21:36:59.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 10 client.30543.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 8 2024-03-11T21:36:59.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 8 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1a9380 osd=3 3 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 3 8 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 4 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 10 client.30543.objecter _send_command 8 2024-03-11T21:36:59.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c1a8f20 con 0x7fdc8c1af9a0 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c1a8f20 con 0x7fdc8c1af9a0 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 4 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 3 8 2024-03-11T21:36:59.417 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 10 client.30543.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 9 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fdc8c1af500 0x7fdc8c1b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 9 2024-03-11T21:36:59.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 4 9 2024-03-11T21:36:59.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 4 2024-03-11T21:36:59.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 10 client.30543.objecter _send_command 9 2024-03-11T21:36:59.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.440+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1b82c0 con 0x7fdc8c1af500 2024-03-11T21:36:59.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.444+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fdc8c1af500 0x7fdc8c1b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1af500 2024-03-11T21:36:59.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fdc8c1af500 0x7fdc8c1b5460 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1b82c0 con 0x7fdc8c1af500 2024-03-11T21:36:59.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:36:59.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 9 = system:0 2024-03-11T21:36:59.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 4 2024-03-11T21:36:59.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.448+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 4 9 2024-03-11T21:36:59.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:59.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 10 client.30543.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 10 2024-03-11T21:36:59.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 10 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1a8ec0 osd=4 3 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 4 10 2024-03-11T21:36:59.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 4 2024-03-11T21:36:59.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 10 client.30543.objecter _send_command 10 2024-03-11T21:36:59.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.468+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c1b59a0 con 0x7fdc8c1af500 2024-03-11T21:36:59.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c1b59a0 con 0x7fdc8c1af500 2024-03-11T21:36:59.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 4 2024-03-11T21:36:59.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 4 10 2024-03-11T21:36:59.451 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:36:59.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 10 client.30543.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 11 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fdc8c1b6020 0x7fdc8c1b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 11 2024-03-11T21:36:59.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 5 11 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 4 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 10 client.30543.objecter _send_command 11 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1bb490 con 0x7fdc8c1b6020 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.472+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fdc8c1b6020 0x7fdc8c1b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1b6020 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc8affd700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fdc8c1b6020 0x7fdc8c1b8410 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1bb490 con 0x7fdc8c1b6020 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 11 = system:0 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 4 2024-03-11T21:36:59.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.476+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 5 11 2024-03-11T21:36:59.459 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 10 client.30543.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 12 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 12 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1b59a0 osd=5 3 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 5 12 2024-03-11T21:36:59.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 4 2024-03-11T21:36:59.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 10 client.30543.objecter _send_command 12 2024-03-11T21:36:59.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.492+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c1ba260 con 0x7fdc8c1b6020 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c1ba260 con 0x7fdc8c1b6020 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 4 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 5 12 2024-03-11T21:36:59.472 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:36:59.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 10 client.30543.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 13 2024-03-11T21:36:59.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fdc8c1ba980 0x7fdc8c1bade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 13 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 6 13 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 4 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 10 client.30543.objecter _send_command 13 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1be8b0 con 0x7fdc8c1ba980 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fdc8c1ba980 0x7fdc8c1bade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1ba980 2024-03-11T21:36:59.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.496+0000 7fdc8bfff700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fdc8c1ba980 0x7fdc8c1bade0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.500+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1be8b0 con 0x7fdc8c1ba980 2024-03-11T21:36:59.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.500+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:36:59.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.500+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 13 = system:0 2024-03-11T21:36:59.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.500+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 4 2024-03-11T21:36:59.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.500+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 6 13 2024-03-11T21:36:59.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 10 client.30543.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 14 2024-03-11T21:36:59.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 14 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1ba260 osd=6 3 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 6 14 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 4 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 10 client.30543.objecter _send_command 14 2024-03-11T21:36:59.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.516+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c1be8b0 con 0x7fdc8c1ba980 2024-03-11T21:36:59.495 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c1be8b0 con 0x7fdc8c1ba980 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 4 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 6 14 2024-03-11T21:36:59.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 10 client.30543.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 15 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fdc8c1bdbf0 0x7fdc8c1be050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 15 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 7 15 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 4 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 10 client.30543.objecter _send_command 15 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c1c1c70 con 0x7fdc8c1bdbf0 2024-03-11T21:36:59.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fdc8c1bdbf0 0x7fdc8c1be050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc8b7fe700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fdc8c1bdbf0 0x7fdc8c1be050 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.520+0000 7fdc88ff9700 10 client.30543.objecter ms_handle_connect 0x7fdc8c1bdbf0 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.524+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc8c1c1c70 con 0x7fdc8c1bdbf0 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.524+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.524+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 15 = system:0 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.524+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 4 2024-03-11T21:36:59.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.524+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 7 15 2024-03-11T21:36:59.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 10 client.30543.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign -1 16 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 20 client.30543.objecter _get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_remove -1 16 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 20 client.30543.objecter get_session s=0x7fdc8c1be8b0 osd=7 3 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 15 client.30543.objecter _session_command_op_assign 7 16 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 4 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 10 client.30543.objecter _send_command 16 2024-03-11T21:36:59.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.540+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc8c1b8a90 con 0x7fdc8c1bdbf0 2024-03-11T21:36:59.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc88ff9700 1 -- 172.21.15.143:0/3860154396 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdc8c1b8a90 con 0x7fdc8c1bdbf0 2024-03-11T21:36:59.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc88ff9700 10 client.30543.objecter ms_dispatch 0x7fdc8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:36:59.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc88ff9700 10 client.30543.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:36:59.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc88ff9700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 4 2024-03-11T21:36:59.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc88ff9700 15 client.30543.objecter _session_command_op_remove 7 16 2024-03-11T21:36:59.521 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:36:59.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 librados: watch_flush enter 2024-03-11T21:36:59.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 librados: watch_flush exit 2024-03-11T21:36:59.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.0 2024-03-11T21:36:59.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fdc8c0629f0 msgr2=0x7fdc8c062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fdc8c0629f0 0x7fdc8c062dd0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c19b0f0 osd=0 1 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.1 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fdc8c02c6b0 msgr2=0x7fdc8c02cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fdc8c02c6b0 0x7fdc8c02cab0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b660 osd=1 1 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.2 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdc8c1abef0 msgr2=0x7fdc8c1ae2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdc8c1abef0 0x7fdc8c1ae2e0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c10b4e0 osd=2 1 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.3 2024-03-11T21:36:59.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fdc8c1af9a0 msgr2=0x7fdc8c1b1d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fdc8c1af9a0 0x7fdc8c1b1d90 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc8bfff700 1 -- 172.21.15.143:0/3860154396 reap_dead start 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a9380 osd=3 1 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.4 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fdc8c1af500 msgr2=0x7fdc8c1b5460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fdc8c1af500 0x7fdc8c1b5460 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1a8ec0 osd=4 1 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.5 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc8bfff700 1 -- 172.21.15.143:0/3860154396 reap_dead start 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fdc8c1b6020 msgr2=0x7fdc8c1b8410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fdc8c1b6020 0x7fdc8c1b8410 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1b59a0 osd=5 1 2024-03-11T21:36:59.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.6 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fdc8c1ba980 msgr2=0x7fdc8c1bade0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fdc8c1ba980 0x7fdc8c1bade0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1ba260 osd=6 1 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter close_session for osd.7 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fdc8c1bdbf0 msgr2=0x7fdc8c1be050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fdc8c1bdbf0 0x7fdc8c1be050 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter put_session s=0x7fdc8c1be8b0 osd=7 1 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 20 client.30543.objecter shutdown clearing up homeless session... 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 10 client.30543.objecter successfully canceled tick 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc7406f820 msgr2=0x7fdc74071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc7406f820 0x7fdc74071ce0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009750 tx=0x7fdc80009310 comp rx=0 tx=0).stop 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 msgr2=0x7fdc8c19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:36:59.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 --2- 172.21.15.143:0/3860154396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc8c104c40 0x7fdc8c19a2d0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00c8f0 tx=0x7fdc7c00cc00 comp rx=0 tx=0).stop 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc8bfff700 1 -- 172.21.15.143:0/3860154396 reap_dead start 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 shutdown_connections 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 >> 172.21.15.143:0/3860154396 conn(0x7fdc8c0ffb40 msgr2=0x7fdc8c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 shutdown_connections 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 -- 172.21.15.143:0/3860154396 wait complete. 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:36:59.544+0000 7fdc91abf700 1 librados: shutdown 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:59.526 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:59.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:59.581 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:59.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:59.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:59.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.831 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:59.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:00.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:00.091 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[29284]: audit 2024-03-11T21:36:58.763007+0000 mgr.y (mgr.14550) 552 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[29284]: audit 2024-03-11T21:36:59.350434+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.143:0/3860154396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[29284]: cluster 2024-03-11T21:36:59.421073+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[29284]: audit 2024-03-11T21:36:59.423188+0000 mon.c (mon.2) 1025 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[29284]: audit 2024-03-11T21:36:59.423662+0000 mon.a (mon.0) 2938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[21512]: audit 2024-03-11T21:36:58.763007+0000 mgr.y (mgr.14550) 552 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[21512]: audit 2024-03-11T21:36:59.350434+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.143:0/3860154396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:00.184 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[21512]: cluster 2024-03-11T21:36:59.421073+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:37:00.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[21512]: audit 2024-03-11T21:36:59.423188+0000 mon.c (mon.2) 1025 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:36:59 smithi143 bash[21512]: audit 2024-03-11T21:36:59.423662+0000 mon.a (mon.0) 2938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.194 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:00.221 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:00.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:59 smithi203 bash[23654]: audit 2024-03-11T21:36:58.763007+0000 mgr.y (mgr.14550) 552 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:00.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:59 smithi203 bash[23654]: audit 2024-03-11T21:36:59.350434+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.143:0/3860154396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:00.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:59 smithi203 bash[23654]: cluster 2024-03-11T21:36:59.421073+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:37:00.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:59 smithi203 bash[23654]: audit 2024-03-11T21:36:59.423188+0000 mon.c (mon.2) 1025 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:36:59 smithi203 bash[23654]: audit 2024-03-11T21:36:59.423662+0000 mon.a (mon.0) 2938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:00.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:00.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:00.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:00.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:00.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:00.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:00.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:00.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:01.021 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: cluster 2024-03-11T21:36:59.465827+0000 mgr.y (mgr.14550) 553 : cluster 0 pgmap v844: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: cluster 2024-03-11T21:37:00.416766+0000 mon.a (mon.0) 2939 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: audit 2024-03-11T21:37:00.424728+0000 mon.a (mon.0) 2940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: cluster 2024-03-11T21:37:00.431408+0000 mon.a (mon.0) 2941 : cluster 0 osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: audit 2024-03-11T21:37:00.440738+0000 mon.c (mon.2) 1026 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: audit 2024-03-11T21:37:00.446919+0000 mon.c (mon.2) 1027 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:00 smithi203 bash[23654]: audit 2024-03-11T21:37:00.447420+0000 mon.a (mon.0) 2942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.131 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: cluster 2024-03-11T21:36:59.465827+0000 mgr.y (mgr.14550) 553 : cluster 0 pgmap v844: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: cluster 2024-03-11T21:37:00.416766+0000 mon.a (mon.0) 2939 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: audit 2024-03-11T21:37:00.424728+0000 mon.a (mon.0) 2940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: cluster 2024-03-11T21:37:00.431408+0000 mon.a (mon.0) 2941 : cluster 0 osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: audit 2024-03-11T21:37:00.440738+0000 mon.c (mon.2) 1026 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: audit 2024-03-11T21:37:00.446919+0000 mon.c (mon.2) 1027 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[29284]: audit 2024-03-11T21:37:00.447420+0000 mon.a (mon.0) 2942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: cluster 2024-03-11T21:36:59.465827+0000 mgr.y (mgr.14550) 553 : cluster 0 pgmap v844: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: cluster 2024-03-11T21:37:00.416766+0000 mon.a (mon.0) 2939 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:01.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: audit 2024-03-11T21:37:00.424728+0000 mon.a (mon.0) 2940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:01.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: cluster 2024-03-11T21:37:00.431408+0000 mon.a (mon.0) 2941 : cluster 0 osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:37:01.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: audit 2024-03-11T21:37:00.440738+0000 mon.c (mon.2) 1026 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:01.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: audit 2024-03-11T21:37:00.446919+0000 mon.c (mon.2) 1027 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:00 smithi143 bash[21512]: audit 2024-03-11T21:37:00.447420+0000 mon.a (mon.0) 2942 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:01.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.263 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:01.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:01.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:01.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:01.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:01.671 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:01.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.803 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:01.906 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.933 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:02.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:02.063 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:02.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.183 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:02.192 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:02.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:02.322 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:02.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.452 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:02.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.575 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21512]: audit 2024-03-11T21:37:01.426955+0000 mon.a (mon.0) 2943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:02.575 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21512]: cluster 2024-03-11T21:37:01.433072+0000 mon.a (mon.0) 2944 : cluster 0 osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:37:02.575 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21512]: audit 2024-03-11T21:37:01.437491+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.575 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21512]: audit 2024-03-11T21:37:01.438064+0000 mon.a (mon.0) 2945 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.575 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[21512]: cluster 2024-03-11T21:37:01.466292+0000 mgr.y (mgr.14550) 554 : cluster 0 pgmap v847: 260 pgs: 3 creating+activating, 29 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-11T21:37:02.576 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[29284]: audit 2024-03-11T21:37:01.426955+0000 mon.a (mon.0) 2943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:02.576 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[29284]: cluster 2024-03-11T21:37:01.433072+0000 mon.a (mon.0) 2944 : cluster 0 osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:37:02.576 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[29284]: audit 2024-03-11T21:37:01.437491+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.576 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[29284]: audit 2024-03-11T21:37:01.438064+0000 mon.a (mon.0) 2945 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.576 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:02 smithi143 bash[29284]: cluster 2024-03-11T21:37:01.466292+0000 mgr.y (mgr.14550) 554 : cluster 0 pgmap v847: 260 pgs: 3 creating+activating, 29 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-11T21:37:02.583 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:02.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:02 smithi203 bash[23654]: audit 2024-03-11T21:37:01.426955+0000 mon.a (mon.0) 2943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:02.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:02 smithi203 bash[23654]: cluster 2024-03-11T21:37:01.433072+0000 mon.a (mon.0) 2944 : cluster 0 osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:37:02.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:02 smithi203 bash[23654]: audit 2024-03-11T21:37:01.437491+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.699 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:02 smithi203 bash[23654]: audit 2024-03-11T21:37:01.438064+0000 mon.a (mon.0) 2945 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:02.699 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:02 smithi203 bash[23654]: cluster 2024-03-11T21:37:01.466292+0000 mgr.y (mgr.14550) 554 : cluster 0 pgmap v847: 260 pgs: 3 creating+activating, 29 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-11T21:37:02.715 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:02.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.847 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:02.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:02.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:03.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:03.106 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:03.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:03.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:03.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.365 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:03.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:03.479 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:03 smithi203 bash[23654]: audit 2024-03-11T21:37:02.438788+0000 mon.a (mon.0) 2946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:03.479 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:03 smithi203 bash[23654]: cluster 2024-03-11T21:37:02.445975+0000 mon.a (mon.0) 2947 : cluster 0 osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:37:03.479 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:03 smithi203 bash[23654]: audit 2024-03-11T21:37:02.450122+0000 mon.c (mon.2) 1029 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.479 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:03 smithi203 bash[23654]: audit 2024-03-11T21:37:02.450993+0000 mon.a (mon.0) 2948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.501 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:03.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:03.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[21512]: audit 2024-03-11T21:37:02.438788+0000 mon.a (mon.0) 2946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:03.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[21512]: cluster 2024-03-11T21:37:02.445975+0000 mon.a (mon.0) 2947 : cluster 0 osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:37:03.616 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[21512]: audit 2024-03-11T21:37:02.450122+0000 mon.c (mon.2) 1029 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.617 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[21512]: audit 2024-03-11T21:37:02.450993+0000 mon.a (mon.0) 2948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.617 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[29284]: audit 2024-03-11T21:37:02.438788+0000 mon.a (mon.0) 2946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:03.617 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[29284]: cluster 2024-03-11T21:37:02.445975+0000 mon.a (mon.0) 2947 : cluster 0 osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:37:03.617 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[29284]: audit 2024-03-11T21:37:02.450122+0000 mon.c (mon.2) 1029 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.617 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:03 smithi143 bash[29284]: audit 2024-03-11T21:37:02.450993+0000 mon.a (mon.0) 2948 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:03.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:03.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:03.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:03.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:04.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:04.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.153 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:04.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:04.283 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:04.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:37:04.387 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:04.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:04.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:04.521 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:04 smithi203 bash[23654]: audit 2024-03-11T21:37:03.441209+0000 mon.a (mon.0) 2949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:04 smithi203 bash[23654]: cluster 2024-03-11T21:37:03.447339+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:37:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:04 smithi203 bash[23654]: audit 2024-03-11T21:37:03.453468+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:04 smithi203 bash[23654]: cluster 2024-03-11T21:37:03.466820+0000 mgr.y (mgr.14550) 555 : cluster 0 pgmap v850: 260 pgs: 3 creating+activating, 2 unknown, 255 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:04.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:04 smithi203 bash[23654]: audit 2024-03-11T21:37:03.467574+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.535 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:04.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[21512]: audit 2024-03-11T21:37:03.441209+0000 mon.a (mon.0) 2949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:04.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[21512]: cluster 2024-03-11T21:37:03.447339+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:37:04.650 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[21512]: audit 2024-03-11T21:37:03.453468+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[21512]: cluster 2024-03-11T21:37:03.466820+0000 mgr.y (mgr.14550) 555 : cluster 0 pgmap v850: 260 pgs: 3 creating+activating, 2 unknown, 255 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[21512]: audit 2024-03-11T21:37:03.467574+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[29284]: audit 2024-03-11T21:37:03.441209+0000 mon.a (mon.0) 2949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[29284]: cluster 2024-03-11T21:37:03.447339+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[29284]: audit 2024-03-11T21:37:03.453468+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[29284]: cluster 2024-03-11T21:37:03.466820+0000 mgr.y (mgr.14550) 555 : cluster 0 pgmap v850: 260 pgs: 3 creating+activating, 2 unknown, 255 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:04.651 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:04 smithi143 bash[29284]: audit 2024-03-11T21:37:03.467574+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:04.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:04.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:04.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:04.923 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:05.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.053 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:05.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:05.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:05.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.318 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:05.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.445 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:05.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:05.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- 172.21.15.143:0/1809085071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340101ec0 msgr2=0x7f4340102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 --2- 172.21.15.143:0/1809085071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340101ec0 0x7f4340102340 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f4334009990 tx=0x7f4334009ca0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- 172.21.15.143:0/1809085071 shutdown_connections 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 --2- 172.21.15.143:0/1809085071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340107f30 0x7f4340108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 --2- 172.21.15.143:0/1809085071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340102880 0x7f434010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 --2- 172.21.15.143:0/1809085071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340101ec0 0x7f4340102340 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- 172.21.15.143:0/1809085071 >> 172.21.15.143:0/1809085071 conn(0x7f4340078eb0 msgr2=0x7f43400792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- 172.21.15.143:0/1809085071 shutdown_connections 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- 172.21.15.143:0/1809085071 wait complete. 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 librados: starting msgr at 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 librados: starting objecter 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 Processor -- start 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.592+0000 7f43454da700 1 -- start start 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 librados: setting wanted keys 2024-03-11T21:37:05.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 librados: calling monclient init 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340101ec0 0x7f43400740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 0x7f4340074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4340111660 con 0x7f4340101ec0 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43401114e0 con 0x7f4340107f30 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43401117e0 con 0x7f4340102880 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:52562/0 (socket says 172.21.15.143:52562) 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 -- 172.21.15.143:0/1979907560 learned_addr learned my addr 172.21.15.143:0/1979907560 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433effd700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340101ec0 0x7f43400740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433f7fe700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 0x7f4340074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 -- 172.21.15.143:0/1979907560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 msgr2=0x7f4340074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 0x7f4340074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 -- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340101ec0 msgr2=0x7f43400740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340101ec0 0x7f43400740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 -- 172.21.15.143:0/1979907560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43401a60f0 con 0x7f4340102880 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433f7fe700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 0x7f4340074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433e7fc700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f43340047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 10 client.?.objecter ms_handle_connect 0x7f4340102880 2024-03-11T21:37:05.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:05.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334027070 con 0x7f4340102880 2024-03-11T21:37:05.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4334022470 con 0x7f4340102880 2024-03-11T21:37:05.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43401a6380 con 0x7f4340102880 2024-03-11T21:37:05.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334017650 con 0x7f4340102880 2024-03-11T21:37:05.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 10 client.41750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43401a67d0 con 0x7f4340102880 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 librados: init done 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434004faa0 con 0x7f4340102880 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f433402a070 con 0x7f4340102880 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 0x7f4328071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(553..553 src has 87..553) v4 ==== 9696+0+0 (secure 0 0 0) 0x7f433408fa90 con 0x7f4340102880 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 10 client.41750.objecter ms_dispatch 0x7f4340001550 osd_map(553..553 src has 87..553) v4 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 3 client.41750.objecter handle_osd_map got epochs [553,553] > 0 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 3 client.41750.objecter handle_osd_map decoding full epoch 553 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f433effd700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 0x7f4328071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.596+0000 7f4327fff700 20 client.41750.objecter dump_active .. 0 homeless 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.600+0000 7f4327fff700 10 client.41750.objecter ms_handle_connect 0x7f432806f820 2024-03-11T21:37:05.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.600+0000 7f433effd700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 0x7f4328071ce0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f433800f540 tx=0x7f433800b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.577 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:05.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:05.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.604+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4334013070 con 0x7f4340102880 2024-03-11T21:37:05.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:05.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[29284]: audit 2024-03-11T21:37:04.462186+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:05.683 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[29284]: cluster 2024-03-11T21:37:04.471233+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[29284]: audit 2024-03-11T21:37:04.511315+0000 mon.c (mon.2) 1031 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[29284]: audit 2024-03-11T21:37:04.511682+0000 mon.a (mon.0) 2954 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[21512]: audit 2024-03-11T21:37:04.462186+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[21512]: cluster 2024-03-11T21:37:04.471233+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[21512]: audit 2024-03-11T21:37:04.511315+0000 mon.c (mon.2) 1031 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:05 smithi143 bash[21512]: audit 2024-03-11T21:37:04.511682+0000 mon.a (mon.0) 2954 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.689 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:05 smithi203 bash[23654]: audit 2024-03-11T21:37:04.462186+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:05.689 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:05 smithi203 bash[23654]: cluster 2024-03-11T21:37:04.471233+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:37:05.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:05 smithi203 bash[23654]: audit 2024-03-11T21:37:04.511315+0000 mon.c (mon.2) 1031 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:05 smithi203 bash[23654]: audit 2024-03-11T21:37:04.511682+0000 mon.a (mon.0) 2954 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:37:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:05.778 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:05.778 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:05.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:05.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.844+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f43401a6af0 con 0x7f4340102880 2024-03-11T21:37:05.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/920589775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 msgr2=0x7f463410ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- 172.21.15.143:0/920589775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f463410ed40 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f462c00b0d0 tx=0x7f462c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/920589775 shutdown_connections 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- 172.21.15.143:0/920589775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f463410ed40 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- 172.21.15.143:0/920589775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463406a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- 172.21.15.143:0/920589775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f4634106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/920589775 >> 172.21.15.143:0/920589775 conn(0x7f4634076d80 msgr2=0x7f4634077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/920589775 shutdown_connections 2024-03-11T21:37:05.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/920589775 wait complete. 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 librados: starting msgr at 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 librados: starting objecter 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 Processor -- start 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- start start 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 librados: setting wanted keys 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 librados: calling monclient init 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f46341979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634111680 con 0x7f463406a720 2024-03-11T21:37:05.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634111500 con 0x7f4634069d70 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4634111800 con 0x7f46341071c0 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:60506/0 (socket says 172.21.15.143:60506) 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 -- 172.21.15.143:0/2227192875 learned_addr learned my addr 172.21.15.143:0/2227192875 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4633fff700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f46341979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f46337fe700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 -- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 msgr2=0x7f46341979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f46341979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 -- 172.21.15.143:0/2227192875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 msgr2=0x7f463419d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463419d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 -- 172.21.15.143:0/2227192875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634198140 con 0x7f463406a720 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4633fff700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f46341979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:37:05.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f46337fe700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4632ffd700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f461c009fd0 tx=0x7f461c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4630ff9700 10 client.?.objecter ms_handle_connect 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4630ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00cbd0 con 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f461c0044a0 con 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4634198430 con 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c003d00 con 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 10 client.30555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46341a46f0 con 0x7f463406a720 2024-03-11T21:37:05.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.852+0000 7f4639e2d700 1 librados: init done 2024-03-11T21:37:05.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463404faa0 con 0x7f463406a720 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f461c003fa0 con 0x7f463406a720 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 0x7f4620071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(553..553 src has 87..553) v4 ==== 9696+0+0 (secure 0 0 0) 0x7f461c08ed40 con 0x7f463406a720 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 10 client.30555.objecter ms_dispatch 0x7f4634001550 osd_map(553..553 src has 87..553) v4 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 3 client.30555.objecter handle_osd_map got epochs [553,553] > 0 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 3 client.30555.objecter handle_osd_map decoding full epoch 553 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f46337fe700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 0x7f4620071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 20 client.30555.objecter dump_active .. 0 homeless 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f4630ff9700 10 client.30555.objecter ms_handle_connect 0x7f462006f820 2024-03-11T21:37:05.835 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.856+0000 7f46337fe700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 0x7f4620071ce0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f4634198b20 tx=0x7f462400b2c0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:05.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:05.860+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f461c016330 con 0x7f463406a720 2024-03-11T21:37:05.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:05.962 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:06.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.056+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46340ff2c0 con 0x7f462006f820 2024-03-11T21:37:06.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:06.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.100+0000 7f4630ff9700 1 -- 172.21.15.143:0/2227192875 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+670252 (secure 0 0 0) 0x7f46340ff2c0 con 0x7f462006f820 2024-03-11T21:37:06.085 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 10 librados: watch_flush enter 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 10 librados: watch_flush exit 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 20 client.30555.objecter shutdown clearing up homeless session... 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 10 client.30555.objecter successfully canceled tick 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 msgr2=0x7f4620071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 0x7f4620071ce0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f4634198b20 tx=0x7f462400b2c0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 msgr2=0x7f463419dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:06.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.104+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f461c009fd0 tx=0x7f461c00ed10 comp rx=0 tx=0).stop 2024-03-11T21:37:06.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 shutdown_connections 2024-03-11T21:37:06.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f462006f820 0x7f4620071ce0 unknown :-1 s=CLOSED pgs=3391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46341071c0 0x7f46341979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4634069d70 0x7f463419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 --2- 172.21.15.143:0/2227192875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463406a720 0x7f463419dd60 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 >> 172.21.15.143:0/2227192875 conn(0x7f4634076d80 msgr2=0x7f46340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 shutdown_connections 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 -- 172.21.15.143:0/2227192875 wait complete. 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.108+0000 7f4639e2d700 1 librados: shutdown 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:37:06.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:06.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.224 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:06.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:06.354 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:06.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.408+0000 7f4327fff700 1 -- 172.21.15.143:0/1979907560 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v554) v1 ==== 97+0+0 (secure 0 0 0) 0x7f433405c6a0 con 0x7f4340102880 2024-03-11T21:37:06.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.412+0000 7f43454da700 10 librados: watch_flush enter 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 10 librados: watch_flush exit 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 20 client.41750.objecter shutdown clearing up homeless session... 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 10 client.41750.objecter successfully canceled tick 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 msgr2=0x7f4328071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 0x7f4328071ce0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f433800f540 tx=0x7f433800b040 comp rx=0 tx=0).stop 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 msgr2=0x7f4340074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f43340047b0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 shutdown_connections 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f432806f820 0x7f4328071ce0 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4340101ec0 0x7f43400740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4340107f30 0x7f4340074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 --2- 172.21.15.143:0/1979907560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4340102880 0x7f4340074620 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 >> 172.21.15.143:0/1979907560 conn(0x7f4340078eb0 msgr2=0x7f4340108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 shutdown_connections 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 -- 172.21.15.143:0/1979907560 wait complete. 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:06.416+0000 7f43454da700 1 librados: shutdown 2024-03-11T21:37:06.396 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:37:06.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:06.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:06.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: cluster 2024-03-11T21:37:05.467444+0000 mgr.y (mgr.14550) 556 : cluster 0 pgmap v852: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 253 B/s wr, 2 op/s 2024-03-11T21:37:06.593 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:05.475351+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: cluster 2024-03-11T21:37:05.479515+0000 mon.a (mon.0) 2956 : cluster 0 osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:05.518753+0000 mon.c (mon.2) 1032 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:05.519140+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:05.851459+0000 mon.c (mon.2) 1033 : audit 1 from='client.? 172.21.15.143:0/1979907560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:05.851790+0000 mon.a (mon.0) 2958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:06.061799+0000 mgr.y (mgr.14550) 557 : audit 0 from='client.30555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: cluster 2024-03-11T21:37:06.398750+0000 mon.a (mon.0) 2959 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: cluster 2024-03-11T21:37:06.400030+0000 mon.a (mon.0) 2960 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:06.402231+0000 mon.a (mon.0) 2961 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:06.402419+0000 mon.a (mon.0) 2962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:06.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: cluster 2024-03-11T21:37:06.410170+0000 mon.a (mon.0) 2963 : cluster 0 osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:37:06.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:06.424707+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:06 smithi203 bash[23654]: audit 2024-03-11T21:37:06.425281+0000 mon.a (mon.0) 2964 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: cluster 2024-03-11T21:37:05.467444+0000 mgr.y (mgr.14550) 556 : cluster 0 pgmap v852: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 253 B/s wr, 2 op/s 2024-03-11T21:37:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:05.475351+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:37:06.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: cluster 2024-03-11T21:37:05.479515+0000 mon.a (mon.0) 2956 : cluster 0 osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:05.518753+0000 mon.c (mon.2) 1032 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:05.519140+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:05.851459+0000 mon.c (mon.2) 1033 : audit 1 from='client.? 172.21.15.143:0/1979907560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:05.851790+0000 mon.a (mon.0) 2958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:06.061799+0000 mgr.y (mgr.14550) 557 : audit 0 from='client.30555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: cluster 2024-03-11T21:37:06.398750+0000 mon.a (mon.0) 2959 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: cluster 2024-03-11T21:37:06.400030+0000 mon.a (mon.0) 2960 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:06.402231+0000 mon.a (mon.0) 2961 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:06.402419+0000 mon.a (mon.0) 2962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: cluster 2024-03-11T21:37:06.410170+0000 mon.a (mon.0) 2963 : cluster 0 osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:06.424707+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[29284]: audit 2024-03-11T21:37:06.425281+0000 mon.a (mon.0) 2964 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: cluster 2024-03-11T21:37:05.467444+0000 mgr.y (mgr.14550) 556 : cluster 0 pgmap v852: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 253 B/s wr, 2 op/s 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:05.475351+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: cluster 2024-03-11T21:37:05.479515+0000 mon.a (mon.0) 2956 : cluster 0 osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:05.518753+0000 mon.c (mon.2) 1032 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:05.519140+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:05.851459+0000 mon.c (mon.2) 1033 : audit 1 from='client.? 172.21.15.143:0/1979907560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:05.851790+0000 mon.a (mon.0) 2958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:06.061799+0000 mgr.y (mgr.14550) 557 : audit 0 from='client.30555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: cluster 2024-03-11T21:37:06.398750+0000 mon.a (mon.0) 2959 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: cluster 2024-03-11T21:37:06.400030+0000 mon.a (mon.0) 2960 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:06.402231+0000 mon.a (mon.0) 2961 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:06.402419+0000 mon.a (mon.0) 2962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:06.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: cluster 2024-03-11T21:37:06.410170+0000 mon.a (mon.0) 2963 : cluster 0 osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:37:06.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:06.424707+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:06 smithi143 bash[21512]: audit 2024-03-11T21:37:06.425281+0000 mon.a (mon.0) 2964 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:06.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:06.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:06.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:06.996 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:07.102 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:07.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:07.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:07.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:07.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:07.394 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:07.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:07.523 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:07.627 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:06.710362+0000 mon.c (mon.2) 1035 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:07.404789+0000 mon.a (mon.0) 2965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: cluster 2024-03-11T21:37:07.413475+0000 mon.a (mon.0) 2966 : cluster 0 osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:07.471030+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:07.471342+0000 mon.a (mon.0) 2967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:07.471975+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[21512]: audit 2024-03-11T21:37:07.472262+0000 mon.a (mon.0) 2968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:06.710362+0000 mon.c (mon.2) 1035 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:07.404789+0000 mon.a (mon.0) 2965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: cluster 2024-03-11T21:37:07.413475+0000 mon.a (mon.0) 2966 : cluster 0 osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:07.471030+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:07.471342+0000 mon.a (mon.0) 2967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:07.471975+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.639 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:07 smithi143 bash[29284]: audit 2024-03-11T21:37:07.472262+0000 mon.a (mon.0) 2968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:37:07.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:07.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:07.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:06.710362+0000 mon.c (mon.2) 1035 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:07.404789+0000 mon.a (mon.0) 2965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:07.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: cluster 2024-03-11T21:37:07.413475+0000 mon.a (mon.0) 2966 : cluster 0 osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:37:07.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:07.471030+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:07.471342+0000 mon.a (mon.0) 2967 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:07.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:07.471975+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:07 smithi203 bash[23654]: audit 2024-03-11T21:37:07.472262+0000 mon.a (mon.0) 2968 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-81"}]: dispatch 2024-03-11T21:37:07.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:07.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:07.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:08.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.040 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:08.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:08.167 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:08.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:08.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:08.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:08.431 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:08.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:08.547 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:08 smithi203 bash[23654]: cluster 2024-03-11T21:37:07.467904+0000 mgr.y (mgr.14550) 558 : cluster 0 pgmap v856: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:08.562 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:08.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:08.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:08.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:08.829 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:08 smithi143 bash[29284]: cluster 2024-03-11T21:37:07.467904+0000 mgr.y (mgr.14550) 558 : cluster 0 pgmap v856: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:08.830 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:08 smithi143 bash[21512]: cluster 2024-03-11T21:37:07.467904+0000 mgr.y (mgr.14550) 558 : cluster 0 pgmap v856: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-03-11T21:37:05.476362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8793024,"num_objects":375,"num_object_clones":10,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":2325,"num_read_kb":2064,"num_write":1543,"num_write_kb":15185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2984,"ondisk_log_size":2984,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":748,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":622700,"kb_used_data":98744,"kb_used_omap":38,"kb_used_meta":523673,"kb_avail":749338516,"statfs":{"total":767960285184,"available":767322640384,"internally_reserved":0,"allocated":101113856,"data_stored":71616410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":536241589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":700,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010360"},"pg_stats":[{"pgid":"296.1f","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101365+0000","last_change":"2024-03-11T21:37:00.452111+0000","last_active":"2024-03-11T21:37:02.101365+0000","last_peered":"2024-03-11T21:37:02.101365+0000","last_clean":"2024-03-11T21:37:02.101365+0000","last_became_active":"2024-03-11T21:37:00.451092+0000","last_became_peered":"2024-03-11T21:37:00.451092+0000","last_unstale":"2024-03-11T21:37:02.101365+0000","last_undegraded":"2024-03-11T21:37:02.101365+0000","last_fullsized":"2024-03-11T21:37:02.101365+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:14.807897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"296.1e","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.441093+0000","last_change":"2024-03-11T21:37:00.447096+0000","last_active":"2024-03-11T21:37:02.441093+0000","last_peered":"2024-03-11T21:37:02.441093+0000","last_clean":"2024-03-11T21:37:02.441093+0000","last_became_active":"2024-03-11T21:37:00.446050+0000","last_became_peered":"2024-03-11T21:37:00.446050+0000","last_unstale":"2024-03-11T21:37:02.441093+0000","last_undegraded":"2024-03-11T21:37:02.441093+0000","last_fullsized":"2024-03-11T21:37:02.441093+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:11.495465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.1d","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.111857+0000","last_change":"2024-03-11T21:37:00.443508+0000","last_active":"2024-03-11T21:37:02.111857+0000","last_peered":"2024-03-11T21:37:02.111857+0000","last_clean":"2024-03-11T21:37:02.111857+0000","last_became_active":"2024-03-11T21:37:00.443171+0000","last_became_peered":"2024-03-11T21:37:00.443171+0000","last_unstale":"2024-03-11T21:37:02.111857+0000","last_undegraded":"2024-03-11T21:37:02.111857+0000","last_fullsized":"2024-03-11T21:37:02.111857+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.294238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.1c","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.439386+0000","last_change":"2024-03-11T21:37:00.444994+0000","last_active":"2024-03-11T21:37:02.439386+0000","last_peered":"2024-03-11T21:37:02.439386+0000","last_clean":"2024-03-11T21:37:02.439386+0000","last_became_active":"2024-03-11T21:37:00.444134+0000","last_became_peered":"2024-03-11T21:37:00.444134+0000","last_unstale":"2024-03-11T21:37:02.439386+0000","last_undegraded":"2024-03-11T21:37:02.439386+0000","last_fullsized":"2024-03-11T21:37:02.439386+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:03.813776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.1b","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.108124+0000","last_change":"2024-03-11T21:37:00.443545+0000","last_active":"2024-03-11T21:37:02.108124+0000","last_peered":"2024-03-11T21:37:02.108124+0000","last_clean":"2024-03-11T21:37:02.108124+0000","last_became_active":"2024-03-11T21:37:00.442927+0000","last_became_peered":"2024-03-11T21:37:00.442927+0000","last_unstale":"2024-03-11T21:37:02.108124+0000","last_undegraded":"2024-03-11T21:37:02.108124+0000","last_fullsized":"2024-03-11T21:37:02.108124+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:16.122809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.1a","version":"0'0","reported_seq":19,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517842+0000","last_change":"2024-03-11T21:37:00.450449+0000","last_active":"2024-03-11T21:37:03.517842+0000","last_peered":"2024-03-11T21:37:03.517842+0000","last_clean":"2024-03-11T21:37:03.517842+0000","last_became_active":"2024-03-11T21:37:00.450129+0000","last_became_peered":"2024-03-11T21:37:00.450129+0000","last_unstale":"2024-03-11T21:37:03.517842+0000","last_undegraded":"2024-03-11T21:37:03.517842+0000","last_fullsized":"2024-03-11T21:37:03.517842+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:17.319348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"296.19","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101691+0000","last_change":"2024-03-11T21:37:00.474216+0000","last_active":"2024-03-11T21:37:02.101691+0000","last_peered":"2024-03-11T21:37:02.101691+0000","last_clean":"2024-03-11T21:37:02.101691+0000","last_became_active":"2024-03-11T21:37:00.472691+0000","last_became_peered":"2024-03-11T21:37:00.472691+0000","last_unstale":"2024-03-11T21:37:02.101691+0000","last_undegraded":"2024-03-11T21:37:02.101691+0000","last_fullsized":"2024-03-11T21:37:02.101691+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:03.165495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"296.18","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.108113+0000","last_change":"2024-03-11T21:37:00.450462+0000","last_active":"2024-03-11T21:37:02.108113+0000","last_peered":"2024-03-11T21:37:02.108113+0000","last_clean":"2024-03-11T21:37:02.108113+0000","last_became_active":"2024-03-11T21:37:00.450081+0000","last_became_peered":"2024-03-11T21:37:00.450081+0000","last_unstale":"2024-03-11T21:37:02.108113+0000","last_undegraded":"2024-03-11T21:37:02.108113+0000","last_fullsized":"2024-03-11T21:37:02.108113+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:17.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.17","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.440622+0000","last_change":"2024-03-11T21:37:00.455279+0000","last_active":"2024-03-11T21:37:02.440622+0000","last_peered":"2024-03-11T21:37:02.440622+0000","last_clean":"2024-03-11T21:37:02.440622+0000","last_became_active":"2024-03-11T21:37:00.454402+0000","last_became_peered":"2024-03-11T21:37:00.454402+0000","last_unstale":"2024-03-11T21:37:02.440622+0000","last_undegraded":"2024-03-11T21:37:02.440622+0000","last_fullsized":"2024-03-11T21:37:02.440622+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:12.639696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.16","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.105144+0000","last_change":"2024-03-11T21:37:00.480134+0000","last_active":"2024-03-11T21:37:02.105144+0000","last_peered":"2024-03-11T21:37:02.105144+0000","last_clean":"2024-03-11T21:37:02.105144+0000","last_became_active":"2024-03-11T21:37:00.478309+0000","last_became_peered":"2024-03-11T21:37:00.478309+0000","last_unstale":"2024-03-11T21:37:02.105144+0000","last_undegraded":"2024-03-11T21:37:02.105144+0000","last_fullsized":"2024-03-11T21:37:02.105144+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:16.037595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"296.15","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104156+0000","last_change":"2024-03-11T21:37:00.478940+0000","last_active":"2024-03-11T21:37:02.104156+0000","last_peered":"2024-03-11T21:37:02.104156+0000","last_clean":"2024-03-11T21:37:02.104156+0000","last_became_active":"2024-03-11T21:37:00.478444+0000","last_became_peered":"2024-03-11T21:37:00.478444+0000","last_unstale":"2024-03-11T21:37:02.104156+0000","last_undegraded":"2024-03-11T21:37:02.104156+0000","last_fullsized":"2024-03-11T21:37:02.104156+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:16.188601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"296.14","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.108853+0000","last_change":"2024-03-11T21:37:00.448219+0000","last_active":"2024-03-11T21:37:02.108853+0000","last_peered":"2024-03-11T21:37:02.108853+0000","last_clean":"2024-03-11T21:37:02.108853+0000","last_became_active":"2024-03-11T21:37:00.447828+0000","last_became_peered":"2024-03-11T21:37:00.447828+0000","last_unstale":"2024-03-11T21:37:02.108853+0000","last_undegraded":"2024-03-11T21:37:02.108853+0000","last_fullsized":"2024-03-11T21:37:02.108853+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:24.371498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.13","version":"0'0","reported_seq":19,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519107+0000","last_change":"2024-03-11T21:37:00.478346+0000","last_active":"2024-03-11T21:37:03.519107+0000","last_peered":"2024-03-11T21:37:03.519107+0000","last_clean":"2024-03-11T21:37:03.519107+0000","last_became_active":"2024-03-11T21:37:00.478002+0000","last_became_peered":"2024-03-11T21:37:00.478002+0000","last_unstale":"2024-03-11T21:37:03.519107+0000","last_undegraded":"2024-03-11T21:37:03.519107+0000","last_fullsized":"2024-03-11T21:37:03.519107+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:05.013921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"296.12","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432056+0000","last_change":"2024-03-11T21:37:00.469334+0000","last_active":"2024-03-11T21:37:01.432056+0000","last_peered":"2024-03-11T21:37:01.432056+0000","last_clean":"2024-03-11T21:37:01.432056+0000","last_became_active":"2024-03-11T21:37:00.468823+0000","last_became_peered":"2024-03-11T21:37:00.468823+0000","last_unstale":"2024-03-11T21:37:01.432056+0000","last_undegraded":"2024-03-11T21:37:01.432056+0000","last_fullsized":"2024-03-11T21:37:01.432056+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:06.555808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"296.11","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201073+0000","last_change":"2024-03-11T21:37:00.458937+0000","last_active":"2024-03-11T21:37:03.201073+0000","last_peered":"2024-03-11T21:37:03.201073+0000","last_clean":"2024-03-11T21:37:03.201073+0000","last_became_active":"2024-03-11T21:37:00.458052+0000","last_became_peered":"2024-03-11T21:37:00.458052+0000","last_unstale":"2024-03-11T21:37:03.201073+0000","last_undegraded":"2024-03-11T21:37:03.201073+0000","last_fullsized":"2024-03-11T21:37:03.201073+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:00.330270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"296.10","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.106619+0000","last_change":"2024-03-11T21:37:00.460287+0000","last_active":"2024-03-11T21:37:02.106619+0000","last_peered":"2024-03-11T21:37:02.106619+0000","last_clean":"2024-03-11T21:37:02.106619+0000","last_became_active":"2024-03-11T21:37:00.459982+0000","last_became_peered":"2024-03-11T21:37:00.459982+0000","last_unstale":"2024-03-11T21:37:02.106619+0000","last_undegraded":"2024-03-11T21:37:02.106619+0000","last_fullsized":"2024-03-11T21:37:02.106619+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:29.156889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.f","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204287+0000","last_change":"2024-03-11T21:37:00.445703+0000","last_active":"2024-03-11T21:37:03.204287+0000","last_peered":"2024-03-11T21:37:03.204287+0000","last_clean":"2024-03-11T21:37:03.204287+0000","last_became_active":"2024-03-11T21:37:00.444797+0000","last_became_peered":"2024-03-11T21:37:00.444797+0000","last_unstale":"2024-03-11T21:37:03.204287+0000","last_undegraded":"2024-03-11T21:37:03.204287+0000","last_fullsized":"2024-03-11T21:37:03.204287+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:22.712621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"296.e","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113268+0000","last_change":"2024-03-11T21:37:00.448273+0000","last_active":"2024-03-11T21:37:02.113268+0000","last_peered":"2024-03-11T21:37:02.113268+0000","last_clean":"2024-03-11T21:37:02.113268+0000","last_became_active":"2024-03-11T21:37:00.447944+0000","last_became_peered":"2024-03-11T21:37:00.447944+0000","last_unstale":"2024-03-11T21:37:02.113268+0000","last_undegraded":"2024-03-11T21:37:02.113268+0000","last_fullsized":"2024-03-11T21:37:02.113268+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:22.079398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"296.d","version":"0'0","reported_seq":22,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474399+0000","last_change":"2024-03-11T21:37:00.471204+0000","last_active":"2024-03-11T21:37:04.474399+0000","last_peered":"2024-03-11T21:37:04.474399+0000","last_clean":"2024-03-11T21:37:04.474399+0000","last_became_active":"2024-03-11T21:37:00.470852+0000","last_became_peered":"2024-03-11T21:37:00.470852+0000","last_unstale":"2024-03-11T21:37:04.474399+0000","last_undegraded":"2024-03-11T21:37:04.474399+0000","last_fullsized":"2024-03-11T21:37:04.474399+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.162145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"296.c","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.203897+0000","last_change":"2024-03-11T21:37:00.449429+0000","last_active":"2024-03-11T21:37:03.203897+0000","last_peered":"2024-03-11T21:37:03.203897+0000","last_clean":"2024-03-11T21:37:03.203897+0000","last_became_active":"2024-03-11T21:37:00.448556+0000","last_became_peered":"2024-03-11T21:37:00.448556+0000","last_unstale":"2024-03-11T21:37:03.203897+0000","last_undegraded":"2024-03-11T21:37:03.203897+0000","last_fullsized":"2024-03-11T21:37:03.203897+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:17.465719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"296.b","version":"0'0","reported_seq":22,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.470896+0000","last_change":"2024-03-11T21:37:00.458935+0000","last_active":"2024-03-11T21:37:04.470896+0000","last_peered":"2024-03-11T21:37:04.470896+0000","last_clean":"2024-03-11T21:37:04.470896+0000","last_became_active":"2024-03-11T21:37:00.458449+0000","last_became_peered":"2024-03-11T21:37:00.458449+0000","last_unstale":"2024-03-11T21:37:04.470896+0000","last_undegraded":"2024-03-11T21:37:04.470896+0000","last_fullsized":"2024-03-11T21:37:04.470896+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:01.197536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"296.a","version":"0'0","reported_seq":22,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.476915+0000","last_change":"2024-03-11T21:37:00.458539+0000","last_active":"2024-03-11T21:37:04.476915+0000","last_peered":"2024-03-11T21:37:04.476915+0000","last_clean":"2024-03-11T21:37:04.476915+0000","last_became_active":"2024-03-11T21:37:00.458196+0000","last_became_peered":"2024-03-11T21:37:00.458196+0000","last_unstale":"2024-03-11T21:37:04.476915+0000","last_undegraded":"2024-03-11T21:37:04.476915+0000","last_fullsized":"2024-03-11T21:37:04.476915+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:09.235577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"296.9","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437920+0000","last_change":"2024-03-11T21:37:00.465301+0000","last_active":"2024-03-11T21:37:02.437920+0000","last_peered":"2024-03-11T21:37:02.437920+0000","last_clean":"2024-03-11T21:37:02.437920+0000","last_became_active":"2024-03-11T21:37:00.464627+0000","last_became_peered":"2024-03-11T21:37:00.464627+0000","last_unstale":"2024-03-11T21:37:02.437920+0000","last_undegraded":"2024-03-11T21:37:02.437920+0000","last_fullsized":"2024-03-11T21:37:02.437920+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:12.319438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.8","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437467+0000","last_change":"2024-03-11T21:37:00.454240+0000","last_active":"2024-03-11T21:37:02.437467+0000","last_peered":"2024-03-11T21:37:02.437467+0000","last_clean":"2024-03-11T21:37:02.437467+0000","last_became_active":"2024-03-11T21:37:00.453271+0000","last_became_peered":"2024-03-11T21:37:00.453271+0000","last_unstale":"2024-03-11T21:37:02.437467+0000","last_undegraded":"2024-03-11T21:37:02.437467+0000","last_fullsized":"2024-03-11T21:37:02.437467+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:20.559131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.7","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204276+0000","last_change":"2024-03-11T21:37:00.466532+0000","last_active":"2024-03-11T21:37:03.204276+0000","last_peered":"2024-03-11T21:37:03.204276+0000","last_clean":"2024-03-11T21:37:03.204276+0000","last_became_active":"2024-03-11T21:37:00.465795+0000","last_became_peered":"2024-03-11T21:37:00.465795+0000","last_unstale":"2024-03-11T21:37:03.204276+0000","last_undegraded":"2024-03-11T21:37:03.204276+0000","last_fullsized":"2024-03-11T21:37:03.204276+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:25.367453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"296.6","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104369+0000","last_change":"2024-03-11T21:37:00.479065+0000","last_active":"2024-03-11T21:37:02.104369+0000","last_peered":"2024-03-11T21:37:02.104369+0000","last_clean":"2024-03-11T21:37:02.104369+0000","last_became_active":"2024-03-11T21:37:00.478403+0000","last_became_peered":"2024-03-11T21:37:00.478403+0000","last_unstale":"2024-03-11T21:37:02.104369+0000","last_undegraded":"2024-03-11T21:37:02.104369+0000","last_fullsized":"2024-03-11T21:37:02.104369+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:08.375538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"296.5","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.439448+0000","last_change":"2024-03-11T21:37:00.450659+0000","last_active":"2024-03-11T21:37:02.439448+0000","last_peered":"2024-03-11T21:37:02.439448+0000","last_clean":"2024-03-11T21:37:02.439448+0000","last_became_active":"2024-03-11T21:37:00.449776+0000","last_became_peered":"2024-03-11T21:37:00.449776+0000","last_unstale":"2024-03-11T21:37:02.439448+0000","last_undegraded":"2024-03-11T21:37:02.439448+0000","last_fullsized":"2024-03-11T21:37:02.439448+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:19.603933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"296.4","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.434865+0000","last_change":"2024-03-11T21:37:00.478150+0000","last_active":"2024-03-11T21:37:01.434865+0000","last_peered":"2024-03-11T21:37:01.434865+0000","last_clean":"2024-03-11T21:37:01.434865+0000","last_became_active":"2024-03-11T21:37:00.477827+0000","last_became_peered":"2024-03-11T21:37:00.477827+0000","last_unstale":"2024-03-11T21:37:01.434865+0000","last_undegraded":"2024-03-11T21:37:01.434865+0000","last_fullsized":"2024-03-11T21:37:01.434865+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:22.730480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"296.3","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435001+0000","last_change":"2024-03-11T21:37:00.457949+0000","last_active":"2024-03-11T21:37:01.435001+0000","last_peered":"2024-03-11T21:37:01.435001+0000","last_clean":"2024-03-11T21:37:01.435001+0000","last_became_active":"2024-03-11T21:37:00.457622+0000","last_became_peered":"2024-03-11T21:37:00.457622+0000","last_unstale":"2024-03-11T21:37:01.435001+0000","last_undegraded":"2024-03-11T21:37:01.435001+0000","last_fullsized":"2024-03-11T21:37:01.435001+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:07.670184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"296.2","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201698+0000","last_change":"2024-03-11T21:37:00.445182+0000","last_active":"2024-03-11T21:37:03.201698+0000","last_peered":"2024-03-11T21:37:03.201698+0000","last_clean":"2024-03-11T21:37:03.201698+0000","last_became_active":"2024-03-11T21:37:00.444307+0000","last_became_peered":"2024-03-11T21:37:00.444307+0000","last_unstale":"2024-03-11T21:37:03.201698+0000","last_undegraded":"2024-03-11T21:37:03.201698+0000","last_fullsized":"2024-03-11T21:37:03.201698+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:27.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"296.1","version":"0'0","reported_seq":13,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101887+0000","last_change":"2024-03-11T21:37:00.475800+0000","last_active":"2024-03-11T21:37:02.101887+0000","last_peered":"2024-03-11T21:37:02.101887+0000","last_clean":"2024-03-11T21:37:02.101887+0000","last_became_active":"2024-03-11T21:37:00.475030+0000","last_became_peered":"2024-03-11T21:37:00.475030+0000","last_unstale":"2024-03-11T21:37:02.101887+0000","last_undegraded":"2024-03-11T21:37:02.101887+0000","last_fullsized":"2024-03-11T21:37:02.101887+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:07.001706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"296.0","version":"0'0","reported_seq":16,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.200913+0000","last_change":"2024-03-11T21:37:00.450322+0000","last_active":"2024-03-11T21:37:03.200913+0000","last_peered":"2024-03-11T21:37:03.200913+0000","last_clean":"2024-03-11T21:37:03.200913+0000","last_became_active":"2024-03-11T21:37:00.449447+0000","last_became_peered":"2024-03-11T21:37:00.449447+0000","last_unstale":"2024-03-11T21:37:03.200913+0000","last_undegraded":"2024-03-11T21:37:03.200913+0000","last_fullsized":"2024-03-11T21:37:03.200913+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:59.414573+0000","last_clean_scrub_stamp":"2024-03-11T21:36:59.414573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:24.120062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.9","version":"482'18","reported_seq":1015,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.106691+0000","last_change":"2024-03-11T21:36:29.868300+0000","last_active":"2024-03-11T21:37:02.106691+0000","last_peered":"2024-03-11T21:37:02.106691+0000","last_clean":"2024-03-11T21:37:02.106691+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:37:02.106691+0000","last_undegraded":"2024-03-11T21:37:02.106691+0000","last_fullsized":"2024-03-11T21:37:02.106691+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'18","last_scrub_stamp":"2024-03-11T21:36:29.868174+0000","last_deep_scrub":"482'18","last_deep_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_clean_scrub_stamp":"2024-03-11T21:36:29.868174+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.069584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010370220000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"545'27","reported_seq":993,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.099730+0000","last_change":"2024-03-11T21:35:26.450518+0000","last_active":"2024-03-11T21:37:02.099730+0000","last_peered":"2024-03-11T21:37:02.099730+0000","last_clean":"2024-03-11T21:37:02.099730+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:37:02.099730+0000","last_undegraded":"2024-03-11T21:37:02.099730+0000","last_fullsized":"2024-03-11T21:37:02.099730+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'24","last_scrub_stamp":"2024-03-11T21:34:54.761329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.761329+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041612760000000002,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"464'40","reported_seq":1005,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103617+0000","last_change":"2024-03-11T21:35:26.453000+0000","last_active":"2024-03-11T21:37:02.103617+0000","last_peered":"2024-03-11T21:37:02.103617+0000","last_clean":"2024-03-11T21:37:02.103617+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:37:02.103617+0000","last_undegraded":"2024-03-11T21:37:02.103617+0000","last_fullsized":"2024-03-11T21:37:02.103617+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'40","last_scrub_stamp":"2024-03-11T21:34:55.080483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.080483+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014521345999999999,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"529'70","reported_seq":1056,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201478+0000","last_change":"2024-03-11T21:36:06.267240+0000","last_active":"2024-03-11T21:37:03.201478+0000","last_peered":"2024-03-11T21:37:03.201478+0000","last_clean":"2024-03-11T21:37:03.201478+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:37:03.201478+0000","last_undegraded":"2024-03-11T21:37:03.201478+0000","last_fullsized":"2024-03-11T21:37:03.201478+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'55","last_scrub_stamp":"2024-03-11T21:36:06.267048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.267048+0000","objects_scrubbed":7,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:17.478300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014549960000000001,"stat_sum":{"num_bytes":87,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"541'56","reported_seq":1045,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.112569+0000","last_change":"2024-03-11T21:36:45.778540+0000","last_active":"2024-03-11T21:37:02.112569+0000","last_peered":"2024-03-11T21:37:02.112569+0000","last_clean":"2024-03-11T21:37:02.112569+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:37:02.112569+0000","last_undegraded":"2024-03-11T21:37:02.112569+0000","last_fullsized":"2024-03-11T21:37:02.112569+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'56","last_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_deep_scrub":"541'56","last_deep_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.778396+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:47.275324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00082181799999999998,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"485'36","reported_seq":1005,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202860+0000","last_change":"2024-03-11T21:35:26.433372+0000","last_active":"2024-03-11T21:37:03.202860+0000","last_peered":"2024-03-11T21:37:03.202860+0000","last_clean":"2024-03-11T21:37:03.202860+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:37:03.202860+0000","last_undegraded":"2024-03-11T21:37:03.202860+0000","last_fullsized":"2024-03-11T21:37:03.202860+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'36","last_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_deep_scrub":"485'36","last_deep_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.199420+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.385940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023323459999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"444'184","reported_seq":1132,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204466+0000","last_change":"2024-03-11T21:36:04.302637+0000","last_active":"2024-03-11T21:37:03.204466+0000","last_peered":"2024-03-11T21:37:03.204466+0000","last_clean":"2024-03-11T21:37:03.204466+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:37:03.204466+0000","last_undegraded":"2024-03-11T21:37:03.204466+0000","last_fullsized":"2024-03-11T21:37:03.204466+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'184","last_scrub_stamp":"2024-03-11T21:36:04.302373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.302373+0000","objects_scrubbed":14,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.026005+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001783946,"stat_sum":{"num_bytes":125,"num_objects":14,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":56,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"311'107","reported_seq":1064,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100900+0000","last_change":"2024-03-11T21:36:17.771393+0000","last_active":"2024-03-11T21:37:02.100900+0000","last_peered":"2024-03-11T21:37:02.100900+0000","last_clean":"2024-03-11T21:37:02.100900+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:37:02.100900+0000","last_undegraded":"2024-03-11T21:37:02.100900+0000","last_fullsized":"2024-03-11T21:37:02.100900+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:36:17.771145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.771145+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:19.003690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0030091919999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"369'18","reported_seq":994,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517306+0000","last_change":"2024-03-11T21:36:11.356246+0000","last_active":"2024-03-11T21:37:03.517306+0000","last_peered":"2024-03-11T21:37:03.517306+0000","last_clean":"2024-03-11T21:37:03.517306+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:37:03.517306+0000","last_undegraded":"2024-03-11T21:37:03.517306+0000","last_fullsized":"2024-03-11T21:37:03.517306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:36:11.356102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.356102+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:19.568439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001800236,"stat_sum":{"num_bytes":4000019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"533'63","reported_seq":1040,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517055+0000","last_change":"2024-03-11T21:36:13.359143+0000","last_active":"2024-03-11T21:37:03.517055+0000","last_peered":"2024-03-11T21:37:03.517055+0000","last_clean":"2024-03-11T21:37:03.517055+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:37:03.517055+0000","last_undegraded":"2024-03-11T21:37:03.517055+0000","last_fullsized":"2024-03-11T21:37:03.517055+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:36:13.358869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.358869+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.972060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028501400000000001,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":20,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"538'134","reported_seq":1074,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103621+0000","last_change":"2024-03-11T21:35:26.453765+0000","last_active":"2024-03-11T21:37:02.103621+0000","last_peered":"2024-03-11T21:37:02.103621+0000","last_clean":"2024-03-11T21:37:02.103621+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:37:02.103621+0000","last_undegraded":"2024-03-11T21:37:02.103621+0000","last_fullsized":"2024-03-11T21:37:02.103621+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015353044,"stat_sum":{"num_bytes":125,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":41,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":1012,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201665+0000","last_change":"2024-03-11T21:36:02.266701+0000","last_active":"2024-03-11T21:37:03.201665+0000","last_peered":"2024-03-11T21:37:03.201665+0000","last_clean":"2024-03-11T21:37:03.201665+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:37:03.201665+0000","last_undegraded":"2024-03-11T21:37:03.201665+0000","last_fullsized":"2024-03-11T21:37:03.201665+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'22","last_scrub_stamp":"2024-03-11T21:36:02.266514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.266514+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:17.012345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001443038,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"458'3","reported_seq":992,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113016+0000","last_change":"2024-03-11T21:36:17.826472+0000","last_active":"2024-03-11T21:37:02.113016+0000","last_peered":"2024-03-11T21:37:02.113016+0000","last_clean":"2024-03-11T21:37:02.113016+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:37:02.113016+0000","last_undegraded":"2024-03-11T21:37:02.113016+0000","last_fullsized":"2024-03-11T21:37:02.113016+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'3","last_scrub_stamp":"2024-03-11T21:36:17.826340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.826340+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.080628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079279899999999996,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":1011,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518980+0000","last_change":"2024-03-11T21:36:18.317604+0000","last_active":"2024-03-11T21:37:03.518980+0000","last_peered":"2024-03-11T21:37:03.518980+0000","last_clean":"2024-03-11T21:37:03.518980+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:37:03.518980+0000","last_undegraded":"2024-03-11T21:37:03.518980+0000","last_fullsized":"2024-03-11T21:37:03.518980+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:36:18.317358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.317358+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.707208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022017640000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"506'46","reported_seq":1006,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435823+0000","last_change":"2024-03-11T21:36:21.706930+0000","last_active":"2024-03-11T21:37:01.435823+0000","last_peered":"2024-03-11T21:37:01.435823+0000","last_clean":"2024-03-11T21:37:01.435823+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:37:01.435823+0000","last_undegraded":"2024-03-11T21:37:01.435823+0000","last_fullsized":"2024-03-11T21:37:01.435823+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'46","last_scrub_stamp":"2024-03-11T21:36:21.706814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.706814+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.386403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028979269999999998,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":1042,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435251+0000","last_change":"2024-03-11T21:36:28.741175+0000","last_active":"2024-03-11T21:37:01.435251+0000","last_peered":"2024-03-11T21:37:01.435251+0000","last_clean":"2024-03-11T21:37:01.435251+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:37:01.435251+0000","last_undegraded":"2024-03-11T21:37:01.435251+0000","last_fullsized":"2024-03-11T21:37:01.435251+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:36:28.741022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.741022+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.020272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000857495,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"247'17","reported_seq":991,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516975+0000","last_change":"2024-03-11T21:36:08.294434+0000","last_active":"2024-03-11T21:37:03.516975+0000","last_peered":"2024-03-11T21:37:03.516975+0000","last_clean":"2024-03-11T21:37:03.516975+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:37:03.516975+0000","last_undegraded":"2024-03-11T21:37:03.516975+0000","last_fullsized":"2024-03-11T21:37:03.516975+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:36:08.294289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.294289+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.951123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00159348,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":1063,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516947+0000","last_change":"2024-03-11T21:35:26.437723+0000","last_active":"2024-03-11T21:37:03.516947+0000","last_peered":"2024-03-11T21:37:03.516947+0000","last_clean":"2024-03-11T21:37:03.516947+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:37:03.516947+0000","last_undegraded":"2024-03-11T21:37:03.516947+0000","last_fullsized":"2024-03-11T21:37:03.516947+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'108","last_scrub_stamp":"2024-03-11T21:35:09.044564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.044564+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032700849999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"296'17","reported_seq":881,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.470958+0000","last_change":"2024-03-11T21:36:19.125306+0000","last_active":"2024-03-11T21:37:04.470958+0000","last_peered":"2024-03-11T21:37:04.470958+0000","last_clean":"2024-03-11T21:37:04.470958+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:37:04.470958+0000","last_undegraded":"2024-03-11T21:37:04.470958+0000","last_fullsized":"2024-03-11T21:37:04.470958+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:36:19.125151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.125151+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:48.732140+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00224048,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":1027,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517980+0000","last_change":"2024-03-11T21:36:17.331041+0000","last_active":"2024-03-11T21:37:03.517980+0000","last_peered":"2024-03-11T21:37:03.517980+0000","last_clean":"2024-03-11T21:37:03.517980+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:37:03.517980+0000","last_undegraded":"2024-03-11T21:37:03.517980+0000","last_fullsized":"2024-03-11T21:37:03.517980+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:36:17.330735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.330735+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.410618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072179299999999998,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"515'102","reported_seq":1079,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.439829+0000","last_change":"2024-03-11T21:36:18.451092+0000","last_active":"2024-03-11T21:37:02.439829+0000","last_peered":"2024-03-11T21:37:02.439829+0000","last_clean":"2024-03-11T21:37:02.439829+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:37:02.439829+0000","last_undegraded":"2024-03-11T21:37:02.439829+0000","last_fullsized":"2024-03-11T21:37:02.439829+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'102","last_scrub_stamp":"2024-03-11T21:36:18.450840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.450840+0000","objects_scrubbed":7,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:21.015218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0026391510000000002,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"486'73","reported_seq":1053,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104292+0000","last_change":"2024-03-11T21:35:26.454329+0000","last_active":"2024-03-11T21:37:02.104292+0000","last_peered":"2024-03-11T21:37:02.104292+0000","last_clean":"2024-03-11T21:37:02.104292+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:37:02.104292+0000","last_undegraded":"2024-03-11T21:37:02.104292+0000","last_fullsized":"2024-03-11T21:37:02.104292+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'69","last_scrub_stamp":"2024-03-11T21:34:58.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.041773+0000","objects_scrubbed":9,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:07.673061+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.011639752999999999,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"552'38","reported_seq":902,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.503427+0000","last_change":"2024-03-11T21:36:16.054966+0000","last_active":"2024-03-11T21:37:04.503427+0000","last_peered":"2024-03-11T21:37:04.503427+0000","last_clean":"2024-03-11T21:37:04.503427+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:37:04.503427+0000","last_undegraded":"2024-03-11T21:37:04.503427+0000","last_fullsized":"2024-03-11T21:37:04.503427+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'37","last_scrub_stamp":"2024-03-11T21:36:16.054853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.054853+0000","objects_scrubbed":5,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.306489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012079040000000001,"stat_sum":{"num_bytes":746,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"503'215","reported_seq":1127,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104448+0000","last_change":"2024-03-11T21:35:26.455346+0000","last_active":"2024-03-11T21:37:02.104448+0000","last_peered":"2024-03-11T21:37:02.104448+0000","last_clean":"2024-03-11T21:37:02.104448+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:37:02.104448+0000","last_undegraded":"2024-03-11T21:37:02.104448+0000","last_fullsized":"2024-03-11T21:37:02.104448+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0095356580000000007,"stat_sum":{"num_bytes":151,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":67,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":990,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102305+0000","last_change":"2024-03-11T21:35:26.453569+0000","last_active":"2024-03-11T21:37:02.102305+0000","last_peered":"2024-03-11T21:37:02.102305+0000","last_clean":"2024-03-11T21:37:02.102305+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:37:02.102305+0000","last_undegraded":"2024-03-11T21:37:02.102305+0000","last_fullsized":"2024-03-11T21:37:02.102305+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'35","last_scrub_stamp":"2024-03-11T21:35:08.144914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.144914+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016015273999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"464'17","reported_seq":1041,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.099610+0000","last_change":"2024-03-11T21:36:30.037490+0000","last_active":"2024-03-11T21:37:02.099610+0000","last_peered":"2024-03-11T21:37:02.099610+0000","last_clean":"2024-03-11T21:37:02.099610+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:37:02.099610+0000","last_undegraded":"2024-03-11T21:37:02.099610+0000","last_fullsized":"2024-03-11T21:37:02.099610+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'17","last_scrub_stamp":"2024-03-11T21:36:30.037348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.037348+0000","objects_scrubbed":4,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.250514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0045876709999999998,"stat_sum":{"num_bytes":4194330,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":1038,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101856+0000","last_change":"2024-03-11T21:35:26.451133+0000","last_active":"2024-03-11T21:37:02.101856+0000","last_peered":"2024-03-11T21:37:02.101856+0000","last_clean":"2024-03-11T21:37:02.101856+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:37:02.101856+0000","last_undegraded":"2024-03-11T21:37:02.101856+0000","last_fullsized":"2024-03-11T21:37:02.101856+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'87","last_scrub_stamp":"2024-03-11T21:34:32.811720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.811720+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032466769999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":1022,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103512+0000","last_change":"2024-03-11T21:35:26.453300+0000","last_active":"2024-03-11T21:37:02.103512+0000","last_peered":"2024-03-11T21:37:02.103512+0000","last_clean":"2024-03-11T21:37:02.103512+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:37:02.103512+0000","last_undegraded":"2024-03-11T21:37:02.103512+0000","last_fullsized":"2024-03-11T21:37:02.103512+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'65","last_scrub_stamp":"2024-03-11T21:35:07.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.139123+0000","objects_scrubbed":8,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014996763,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"261'72","reported_seq":1054,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.112246+0000","last_change":"2024-03-11T21:36:44.814463+0000","last_active":"2024-03-11T21:37:02.112246+0000","last_peered":"2024-03-11T21:37:02.112246+0000","last_clean":"2024-03-11T21:37:02.112246+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:37:02.112246+0000","last_undegraded":"2024-03-11T21:37:02.112246+0000","last_fullsized":"2024-03-11T21:37:02.112246+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:36:44.814352+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.814352+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.466414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00098680300000000007,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"495'67","reported_seq":1043,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202614+0000","last_change":"2024-03-11T21:36:11.203520+0000","last_active":"2024-03-11T21:37:03.202614+0000","last_peered":"2024-03-11T21:37:03.202614+0000","last_clean":"2024-03-11T21:37:03.202614+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:37:03.202614+0000","last_undegraded":"2024-03-11T21:37:03.202614+0000","last_fullsized":"2024-03-11T21:37:03.202614+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'67","last_scrub_stamp":"2024-03-11T21:36:11.203406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.203406+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:39.788908+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.0043048899999998,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"467'56","reported_seq":1035,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.112351+0000","last_change":"2024-03-11T21:36:06.936868+0000","last_active":"2024-03-11T21:37:02.112351+0000","last_peered":"2024-03-11T21:37:02.112351+0000","last_clean":"2024-03-11T21:37:02.112351+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:37:02.112351+0000","last_undegraded":"2024-03-11T21:37:02.112351+0000","last_fullsized":"2024-03-11T21:37:02.112351+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'56","last_scrub_stamp":"2024-03-11T21:36:06.936622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.936622+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.189358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056505100000000001,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"467'6","reported_seq":976,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104583+0000","last_change":"2024-03-11T21:35:26.455653+0000","last_active":"2024-03-11T21:37:02.104583+0000","last_peered":"2024-03-11T21:37:02.104583+0000","last_clean":"2024-03-11T21:37:02.104583+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:37:02.104583+0000","last_undegraded":"2024-03-11T21:37:02.104583+0000","last_fullsized":"2024-03-11T21:37:02.104583+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'6","last_scrub_stamp":"2024-03-11T21:34:50.112525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.112525+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177331,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.431642+0000","last_change":"2024-03-11T21:36:09.680554+0000","last_active":"2024-03-11T21:37:01.431642+0000","last_peered":"2024-03-11T21:37:01.431642+0000","last_clean":"2024-03-11T21:37:01.431642+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:37:01.431642+0000","last_undegraded":"2024-03-11T21:37:01.431642+0000","last_fullsized":"2024-03-11T21:37:01.431642+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.680346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.680346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.025150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":847,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437784+0000","last_change":"2024-03-11T21:35:05.450716+0000","last_active":"2024-03-11T21:37:02.437784+0000","last_peered":"2024-03-11T21:37:02.437784+0000","last_clean":"2024-03-11T21:37:02.437784+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:37:02.437784+0000","last_undegraded":"2024-03-11T21:37:02.437784+0000","last_fullsized":"2024-03-11T21:37:02.437784+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.450323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.450323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":840,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100701+0000","last_change":"2024-03-11T21:36:34.020742+0000","last_active":"2024-03-11T21:37:02.100701+0000","last_peered":"2024-03-11T21:37:02.100701+0000","last_clean":"2024-03-11T21:37:02.100701+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:37:02.100701+0000","last_undegraded":"2024-03-11T21:37:02.100701+0000","last_fullsized":"2024-03-11T21:37:02.100701+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.020607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.020607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.450801+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":833,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435061+0000","last_change":"2024-03-11T21:36:14.628566+0000","last_active":"2024-03-11T21:37:01.435061+0000","last_peered":"2024-03-11T21:37:01.435061+0000","last_clean":"2024-03-11T21:37:01.435061+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:37:01.435061+0000","last_undegraded":"2024-03-11T21:37:01.435061+0000","last_fullsized":"2024-03-11T21:37:01.435061+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.259727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":752,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.473936+0000","last_change":"2024-03-11T21:36:43.079499+0000","last_active":"2024-03-11T21:37:04.473936+0000","last_peered":"2024-03-11T21:37:04.473936+0000","last_clean":"2024-03-11T21:37:04.473936+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:37:04.473936+0000","last_undegraded":"2024-03-11T21:37:04.473936+0000","last_fullsized":"2024-03-11T21:37:04.473936+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.079413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.079413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:47.032407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":847,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.440491+0000","last_change":"2024-03-11T21:35:06.473871+0000","last_active":"2024-03-11T21:37:02.440491+0000","last_peered":"2024-03-11T21:37:02.440491+0000","last_clean":"2024-03-11T21:37:02.440491+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:37:02.440491+0000","last_undegraded":"2024-03-11T21:37:02.440491+0000","last_fullsized":"2024-03-11T21:37:02.440491+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.473532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.473532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":848,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.440763+0000","last_change":"2024-03-11T21:34:58.319554+0000","last_active":"2024-03-11T21:37:02.440763+0000","last_peered":"2024-03-11T21:37:02.440763+0000","last_clean":"2024-03-11T21:37:02.440763+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:37:02.440763+0000","last_undegraded":"2024-03-11T21:37:02.440763+0000","last_fullsized":"2024-03-11T21:37:02.440763+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.319457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113288+0000","last_change":"2024-03-11T21:36:08.869894+0000","last_active":"2024-03-11T21:37:02.113288+0000","last_peered":"2024-03-11T21:37:02.113288+0000","last_clean":"2024-03-11T21:37:02.113288+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:37:02.113288+0000","last_undegraded":"2024-03-11T21:37:02.113288+0000","last_fullsized":"2024-03-11T21:37:02.113288+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.869673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.869673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.060598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":850,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201690+0000","last_change":"2024-03-11T21:36:36.107906+0000","last_active":"2024-03-11T21:37:03.201690+0000","last_peered":"2024-03-11T21:37:03.201690+0000","last_clean":"2024-03-11T21:37:03.201690+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:37:03.201690+0000","last_undegraded":"2024-03-11T21:37:03.201690+0000","last_fullsized":"2024-03-11T21:37:03.201690+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.107801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.107801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.759983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":833,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.433349+0000","last_change":"2024-03-11T21:34:49.557575+0000","last_active":"2024-03-11T21:37:01.433349+0000","last_peered":"2024-03-11T21:37:01.433349+0000","last_clean":"2024-03-11T21:37:01.433349+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:37:01.433349+0000","last_undegraded":"2024-03-11T21:37:01.433349+0000","last_fullsized":"2024-03-11T21:37:01.433349+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:49.557488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:49.557488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":851,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518477+0000","last_change":"2024-03-11T21:36:43.145228+0000","last_active":"2024-03-11T21:37:03.518477+0000","last_peered":"2024-03-11T21:37:03.518477+0000","last_clean":"2024-03-11T21:37:03.518477+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:37:03.518477+0000","last_undegraded":"2024-03-11T21:37:03.518477+0000","last_fullsized":"2024-03-11T21:37:03.518477+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.145137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.145137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:09.186494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":827,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098020+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:37:02.098020+0000","last_peered":"2024-03-11T21:37:02.098020+0000","last_clean":"2024-03-11T21:37:02.098020+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:37:02.098020+0000","last_undegraded":"2024-03-11T21:37:02.098020+0000","last_fullsized":"2024-03-11T21:37:02.098020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":843,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516352+0000","last_change":"2024-03-11T21:34:40.037312+0000","last_active":"2024-03-11T21:37:03.516352+0000","last_peered":"2024-03-11T21:37:03.516352+0000","last_clean":"2024-03-11T21:37:03.516352+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:37:03.516352+0000","last_undegraded":"2024-03-11T21:37:03.516352+0000","last_fullsized":"2024-03-11T21:37:03.516352+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.037161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.037161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":842,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204756+0000","last_change":"2024-03-11T21:35:05.211872+0000","last_active":"2024-03-11T21:37:03.204756+0000","last_peered":"2024-03-11T21:37:03.204756+0000","last_clean":"2024-03-11T21:37:03.204756+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:37:03.204756+0000","last_undegraded":"2024-03-11T21:37:03.204756+0000","last_fullsized":"2024-03-11T21:37:03.204756+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.211634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.386126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113588+0000","last_change":"2024-03-11T21:36:30.828411+0000","last_active":"2024-03-11T21:37:02.113588+0000","last_peered":"2024-03-11T21:37:02.113588+0000","last_clean":"2024-03-11T21:37:02.113588+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:37:02.113588+0000","last_undegraded":"2024-03-11T21:37:02.113588+0000","last_fullsized":"2024-03-11T21:37:02.113588+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.828323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.828323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.963712+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":248,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.477002+0000","last_change":"2024-03-11T21:34:52.102628+0000","last_active":"2024-03-11T21:37:04.477002+0000","last_peered":"2024-03-11T21:37:04.477002+0000","last_clean":"2024-03-11T21:37:04.477002+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:37:04.477002+0000","last_undegraded":"2024-03-11T21:37:04.477002+0000","last_fullsized":"2024-03-11T21:37:04.477002+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:52.102271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.102271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.980105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":847,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437447+0000","last_change":"2024-03-11T21:34:54.321311+0000","last_active":"2024-03-11T21:37:02.437447+0000","last_peered":"2024-03-11T21:37:02.437447+0000","last_clean":"2024-03-11T21:37:02.437447+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:37:02.437447+0000","last_undegraded":"2024-03-11T21:37:02.437447+0000","last_fullsized":"2024-03-11T21:37:02.437447+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.321007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.321007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":833,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102124+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:37:02.102124+0000","last_peered":"2024-03-11T21:37:02.102124+0000","last_clean":"2024-03-11T21:37:02.102124+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:37:02.102124+0000","last_undegraded":"2024-03-11T21:37:02.102124+0000","last_fullsized":"2024-03-11T21:37:02.102124+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":833,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432374+0000","last_change":"2024-03-11T21:36:26.767684+0000","last_active":"2024-03-11T21:37:01.432374+0000","last_peered":"2024-03-11T21:37:01.432374+0000","last_clean":"2024-03-11T21:37:01.432374+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:37:01.432374+0000","last_undegraded":"2024-03-11T21:37:01.432374+0000","last_fullsized":"2024-03-11T21:37:01.432374+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.767129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.767129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.431972+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":842,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202875+0000","last_change":"2024-03-11T21:35:02.148627+0000","last_active":"2024-03-11T21:37:03.202875+0000","last_peered":"2024-03-11T21:37:03.202875+0000","last_clean":"2024-03-11T21:37:03.202875+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:37:03.202875+0000","last_undegraded":"2024-03-11T21:37:03.202875+0000","last_fullsized":"2024-03-11T21:37:03.202875+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.148344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.148344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.385612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":720,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474658+0000","last_change":"2024-03-11T21:35:11.130108+0000","last_active":"2024-03-11T21:37:04.474658+0000","last_peered":"2024-03-11T21:37:04.474658+0000","last_clean":"2024-03-11T21:37:04.474658+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:37:04.474658+0000","last_undegraded":"2024-03-11T21:37:04.474658+0000","last_fullsized":"2024-03-11T21:37:04.474658+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.129979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.129979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.980254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":833,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435433+0000","last_change":"2024-03-11T21:34:59.525103+0000","last_active":"2024-03-11T21:37:01.435433+0000","last_peered":"2024-03-11T21:37:01.435433+0000","last_clean":"2024-03-11T21:37:01.435433+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:37:01.435433+0000","last_undegraded":"2024-03-11T21:37:01.435433+0000","last_fullsized":"2024-03-11T21:37:01.435433+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:59.524800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:59.524800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.581147+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":841,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.436717+0000","last_change":"2024-03-11T21:36:04.661608+0000","last_active":"2024-03-11T21:37:01.436717+0000","last_peered":"2024-03-11T21:37:01.436717+0000","last_clean":"2024-03-11T21:37:01.436717+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:37:01.436717+0000","last_undegraded":"2024-03-11T21:37:01.436717+0000","last_fullsized":"2024-03-11T21:37:01.436717+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.661495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.661495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.634867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":843,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521136+0000","last_change":"2024-03-11T21:34:58.091810+0000","last_active":"2024-03-11T21:37:03.521136+0000","last_peered":"2024-03-11T21:37:03.521136+0000","last_clean":"2024-03-11T21:37:03.521136+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:37:03.521136+0000","last_undegraded":"2024-03-11T21:37:03.521136+0000","last_fullsized":"2024-03-11T21:37:03.521136+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.091664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.091664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":841,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100159+0000","last_change":"2024-03-11T21:36:33.044584+0000","last_active":"2024-03-11T21:37:02.100159+0000","last_peered":"2024-03-11T21:37:02.100159+0000","last_clean":"2024-03-11T21:37:02.100159+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:37:02.100159+0000","last_undegraded":"2024-03-11T21:37:02.100159+0000","last_fullsized":"2024-03-11T21:37:02.100159+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:33.044471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.044471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.788507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":827,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098013+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:37:02.098013+0000","last_peered":"2024-03-11T21:37:02.098013+0000","last_clean":"2024-03-11T21:37:02.098013+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:37:02.098013+0000","last_undegraded":"2024-03-11T21:37:02.098013+0000","last_fullsized":"2024-03-11T21:37:02.098013+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":248,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474750+0000","last_change":"2024-03-11T21:34:39.164537+0000","last_active":"2024-03-11T21:37:04.474750+0000","last_peered":"2024-03-11T21:37:04.474750+0000","last_clean":"2024-03-11T21:37:04.474750+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:37:04.474750+0000","last_undegraded":"2024-03-11T21:37:04.474750+0000","last_fullsized":"2024-03-11T21:37:04.474750+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.164421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.164421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":842,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202818+0000","last_change":"2024-03-11T21:36:03.284628+0000","last_active":"2024-03-11T21:37:03.202818+0000","last_peered":"2024-03-11T21:37:03.202818+0000","last_clean":"2024-03-11T21:37:03.202818+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:37:03.202818+0000","last_undegraded":"2024-03-11T21:37:03.202818+0000","last_fullsized":"2024-03-11T21:37:03.202818+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:03.284530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.284530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:15.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":843,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519061+0000","last_change":"2024-03-11T21:34:36.108350+0000","last_active":"2024-03-11T21:37:03.519061+0000","last_peered":"2024-03-11T21:37:03.519061+0000","last_clean":"2024-03-11T21:37:03.519061+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:37:03.519061+0000","last_undegraded":"2024-03-11T21:37:03.519061+0000","last_fullsized":"2024-03-11T21:37:03.519061+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.108246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.108246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":827,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098061+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:37:02.098061+0000","last_peered":"2024-03-11T21:37:02.098061+0000","last_clean":"2024-03-11T21:37:02.098061+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:37:02.098061+0000","last_undegraded":"2024-03-11T21:37:02.098061+0000","last_fullsized":"2024-03-11T21:37:02.098061+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113675+0000","last_change":"2024-03-11T21:36:35.782923+0000","last_active":"2024-03-11T21:37:02.113675+0000","last_peered":"2024-03-11T21:37:02.113675+0000","last_clean":"2024-03-11T21:37:02.113675+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:37:02.113675+0000","last_undegraded":"2024-03-11T21:37:02.113675+0000","last_fullsized":"2024-03-11T21:37:02.113675+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:36:35.782796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.504664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":720,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.477226+0000","last_change":"2024-03-11T21:35:01.154707+0000","last_active":"2024-03-11T21:37:04.477226+0000","last_peered":"2024-03-11T21:37:04.477226+0000","last_clean":"2024-03-11T21:37:04.477226+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:37:04.477226+0000","last_undegraded":"2024-03-11T21:37:04.477226+0000","last_fullsized":"2024-03-11T21:37:04.477226+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.154502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.154502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.980205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":857,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521560+0000","last_change":"2024-03-11T21:36:45.220132+0000","last_active":"2024-03-11T21:37:03.521560+0000","last_peered":"2024-03-11T21:37:03.521560+0000","last_clean":"2024-03-11T21:37:03.521560+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:37:03.521560+0000","last_undegraded":"2024-03-11T21:37:03.521560+0000","last_fullsized":"2024-03-11T21:37:03.521560+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.219989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.219989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.466225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1157,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102976+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:37:02.102976+0000","last_peered":"2024-03-11T21:37:02.102976+0000","last_clean":"2024-03-11T21:37:02.102976+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:37:02.102976+0000","last_undegraded":"2024-03-11T21:37:02.102976+0000","last_fullsized":"2024-03-11T21:37:02.102976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1026,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.105093+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:37:02.105093+0000","last_peered":"2024-03-11T21:37:02.105093+0000","last_clean":"2024-03-11T21:37:02.105093+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:37:02.105093+0000","last_undegraded":"2024-03-11T21:37:02.105093+0000","last_fullsized":"2024-03-11T21:37:02.105093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1023,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113962+0000","last_change":"2024-03-11T21:36:25.851234+0000","last_active":"2024-03-11T21:37:02.113962+0000","last_peered":"2024-03-11T21:37:02.113962+0000","last_clean":"2024-03-11T21:37:02.113962+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:37:02.113962+0000","last_undegraded":"2024-03-11T21:37:02.113962+0000","last_fullsized":"2024-03-11T21:37:02.113962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.851095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.851095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:39.358983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":989,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101189+0000","last_change":"2024-03-11T21:35:10.696295+0000","last_active":"2024-03-11T21:37:02.101189+0000","last_peered":"2024-03-11T21:37:02.101189+0000","last_clean":"2024-03-11T21:37:02.101189+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:37:02.101189+0000","last_undegraded":"2024-03-11T21:37:02.101189+0000","last_fullsized":"2024-03-11T21:37:02.101189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.696090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.696090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.845039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102017+0000","last_change":"2024-03-11T21:36:37.014962+0000","last_active":"2024-03-11T21:37:02.102017+0000","last_peered":"2024-03-11T21:37:02.102017+0000","last_clean":"2024-03-11T21:37:02.102017+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:37:02.102017+0000","last_undegraded":"2024-03-11T21:37:02.102017+0000","last_fullsized":"2024-03-11T21:37:02.102017+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.014751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.014751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.818693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1027,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101448+0000","last_change":"2024-03-11T21:35:11.672580+0000","last_active":"2024-03-11T21:37:02.101448+0000","last_peered":"2024-03-11T21:37:02.101448+0000","last_clean":"2024-03-11T21:37:02.101448+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:37:02.101448+0000","last_undegraded":"2024-03-11T21:37:02.101448+0000","last_fullsized":"2024-03-11T21:37:02.101448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:35:11.672324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.672324+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.845018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":902,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474562+0000","last_change":"2024-03-11T21:34:48.064829+0000","last_active":"2024-03-11T21:37:04.474562+0000","last_peered":"2024-03-11T21:37:04.474562+0000","last_clean":"2024-03-11T21:37:04.474562+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:37:04.474562+0000","last_undegraded":"2024-03-11T21:37:04.474562+0000","last_fullsized":"2024-03-11T21:37:04.474562+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.064622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.064622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1015,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.107338+0000","last_change":"2024-03-11T21:36:24.816126+0000","last_active":"2024-03-11T21:37:02.107338+0000","last_peered":"2024-03-11T21:37:02.107338+0000","last_clean":"2024-03-11T21:37:02.107338+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:37:02.107338+0000","last_undegraded":"2024-03-11T21:37:02.107338+0000","last_fullsized":"2024-03-11T21:37:02.107338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:24.816023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.816023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:32.230362+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1000,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518662+0000","last_change":"2024-03-11T21:35:02.012107+0000","last_active":"2024-03-11T21:37:03.518662+0000","last_peered":"2024-03-11T21:37:03.518662+0000","last_clean":"2024-03-11T21:37:03.518662+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:37:03.518662+0000","last_undegraded":"2024-03-11T21:37:03.518662+0000","last_fullsized":"2024-03-11T21:37:03.518662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.011982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.011982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":846,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101506+0000","last_change":"2024-03-11T21:36:08.644929+0000","last_active":"2024-03-11T21:37:02.101506+0000","last_peered":"2024-03-11T21:37:02.101506+0000","last_clean":"2024-03-11T21:37:02.101506+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:37:02.101506+0000","last_undegraded":"2024-03-11T21:37:02.101506+0000","last_fullsized":"2024-03-11T21:37:02.101506+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.644692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.644692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.731782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1048,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104742+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:37:02.104742+0000","last_peered":"2024-03-11T21:37:02.104742+0000","last_clean":"2024-03-11T21:37:02.104742+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:37:02.104742+0000","last_undegraded":"2024-03-11T21:37:02.104742+0000","last_fullsized":"2024-03-11T21:37:02.104742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1023,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.110757+0000","last_change":"2024-03-11T21:36:11.798174+0000","last_active":"2024-03-11T21:37:02.110757+0000","last_peered":"2024-03-11T21:37:02.110757+0000","last_clean":"2024-03-11T21:37:02.110757+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:37:02.110757+0000","last_undegraded":"2024-03-11T21:37:02.110757+0000","last_fullsized":"2024-03-11T21:37:02.110757+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:17.280025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1015,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.108622+0000","last_change":"2024-03-11T21:36:43.849743+0000","last_active":"2024-03-11T21:37:02.108622+0000","last_peered":"2024-03-11T21:37:02.108622+0000","last_clean":"2024-03-11T21:37:02.108622+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:37:02.108622+0000","last_undegraded":"2024-03-11T21:37:02.108622+0000","last_fullsized":"2024-03-11T21:37:02.108622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.849578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.849578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.597341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":983,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.099067+0000","last_change":"2024-03-11T21:35:11.139678+0000","last_active":"2024-03-11T21:37:02.099067+0000","last_peered":"2024-03-11T21:37:02.099067+0000","last_clean":"2024-03-11T21:37:02.099067+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:37:02.099067+0000","last_undegraded":"2024-03-11T21:37:02.099067+0000","last_fullsized":"2024-03-11T21:37:02.099067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.139473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.139473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":729,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.470615+0000","last_change":"2024-03-11T21:36:45.055123+0000","last_active":"2024-03-11T21:37:04.470615+0000","last_peered":"2024-03-11T21:37:04.470615+0000","last_clean":"2024-03-11T21:37:04.470615+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:37:04.470615+0000","last_undegraded":"2024-03-11T21:37:04.470615+0000","last_fullsized":"2024-03-11T21:37:04.470615+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.054820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.210724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1084,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.438220+0000","last_change":"2024-03-11T21:35:12.343880+0000","last_active":"2024-03-11T21:37:02.438220+0000","last_peered":"2024-03-11T21:37:02.438220+0000","last_clean":"2024-03-11T21:37:02.438220+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:37:02.438220+0000","last_undegraded":"2024-03-11T21:37:02.438220+0000","last_fullsized":"2024-03-11T21:37:02.438220+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:35:12.343626+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.343626+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":995,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104017+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:37:02.104017+0000","last_peered":"2024-03-11T21:37:02.104017+0000","last_clean":"2024-03-11T21:37:02.104017+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:37:02.104017+0000","last_undegraded":"2024-03-11T21:37:02.104017+0000","last_fullsized":"2024-03-11T21:37:02.104017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":908,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474061+0000","last_change":"2024-03-11T21:36:20.116589+0000","last_active":"2024-03-11T21:37:04.474061+0000","last_peered":"2024-03-11T21:37:04.474061+0000","last_clean":"2024-03-11T21:37:04.474061+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:37:04.474061+0000","last_undegraded":"2024-03-11T21:37:04.474061+0000","last_fullsized":"2024-03-11T21:37:04.474061+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:20.116311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.116311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.628264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1011,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113052+0000","last_change":"2024-03-11T21:36:01.810406+0000","last_active":"2024-03-11T21:37:02.113052+0000","last_peered":"2024-03-11T21:37:02.113052+0000","last_clean":"2024-03-11T21:37:02.113052+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:37:02.113052+0000","last_undegraded":"2024-03-11T21:37:02.113052+0000","last_fullsized":"2024-03-11T21:37:02.113052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:01.810170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:01.810170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:18.600258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":856,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432257+0000","last_change":"2024-03-11T21:36:11.656705+0000","last_active":"2024-03-11T21:37:01.432257+0000","last_peered":"2024-03-11T21:37:01.432257+0000","last_clean":"2024-03-11T21:37:01.432257+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:37:01.432257+0000","last_undegraded":"2024-03-11T21:37:01.432257+0000","last_fullsized":"2024-03-11T21:37:01.432257+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:36:11.656610+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.656610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.369092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1025,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435568+0000","last_change":"2024-03-11T21:35:01.534298+0000","last_active":"2024-03-11T21:37:01.435568+0000","last_peered":"2024-03-11T21:37:01.435568+0000","last_clean":"2024-03-11T21:37:01.435568+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:37:01.435568+0000","last_undegraded":"2024-03-11T21:37:01.435568+0000","last_fullsized":"2024-03-11T21:37:01.435568+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:01.534131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.534131+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.581104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":868,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474563+0000","last_change":"2024-03-11T21:34:41.173845+0000","last_active":"2024-03-11T21:37:04.474563+0000","last_peered":"2024-03-11T21:37:04.474563+0000","last_clean":"2024-03-11T21:37:04.474563+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:37:04.474563+0000","last_undegraded":"2024-03-11T21:37:04.474563+0000","last_fullsized":"2024-03-11T21:37:04.474563+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1194,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.113289+0000","last_change":"2024-03-11T21:36:07.905103+0000","last_active":"2024-03-11T21:37:02.113289+0000","last_peered":"2024-03-11T21:37:02.113289+0000","last_clean":"2024-03-11T21:37:02.113289+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:37:02.113289+0000","last_undegraded":"2024-03-11T21:37:02.113289+0000","last_fullsized":"2024-03-11T21:37:02.113289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:36:07.904875+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.904875+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.866007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1015,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204551+0000","last_change":"2024-03-11T21:36:45.224370+0000","last_active":"2024-03-11T21:37:03.204551+0000","last_peered":"2024-03-11T21:37:03.204551+0000","last_clean":"2024-03-11T21:37:03.204551+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:37:03.204551+0000","last_undegraded":"2024-03-11T21:37:03.204551+0000","last_fullsized":"2024-03-11T21:37:03.204551+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.224251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:13.711133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":851,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.433543+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:37:01.433543+0000","last_peered":"2024-03-11T21:37:01.433543+0000","last_clean":"2024-03-11T21:37:01.433543+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:37:01.433543+0000","last_undegraded":"2024-03-11T21:37:01.433543+0000","last_fullsized":"2024-03-11T21:37:01.433543+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1052,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432039+0000","last_change":"2024-03-11T21:36:31.677071+0000","last_active":"2024-03-11T21:37:01.432039+0000","last_peered":"2024-03-11T21:37:01.432039+0000","last_clean":"2024-03-11T21:37:01.432039+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:37:01.432039+0000","last_undegraded":"2024-03-11T21:37:01.432039+0000","last_fullsized":"2024-03-11T21:37:01.432039+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:36:31.676948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.676948+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.706279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":896,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474328+0000","last_change":"2024-03-11T21:36:05.135845+0000","last_active":"2024-03-11T21:37:04.474328+0000","last_peered":"2024-03-11T21:37:04.474328+0000","last_clean":"2024-03-11T21:37:04.474328+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:37:04.474328+0000","last_undegraded":"2024-03-11T21:37:04.474328+0000","last_fullsized":"2024-03-11T21:37:04.474328+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:05.135750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:05.135750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.810637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1180,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204226+0000","last_change":"2024-03-11T21:35:07.191988+0000","last_active":"2024-03-11T21:37:03.204226+0000","last_peered":"2024-03-11T21:37:03.204226+0000","last_clean":"2024-03-11T21:37:03.204226+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:37:03.204226+0000","last_undegraded":"2024-03-11T21:37:03.204226+0000","last_fullsized":"2024-03-11T21:37:03.204226+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:35:07.191770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.191770+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.386237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":998,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103846+0000","last_change":"2024-03-11T21:34:42.857662+0000","last_active":"2024-03-11T21:37:02.103846+0000","last_peered":"2024-03-11T21:37:02.103846+0000","last_clean":"2024-03-11T21:37:02.103846+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:37:02.103846+0000","last_undegraded":"2024-03-11T21:37:02.103846+0000","last_fullsized":"2024-03-11T21:37:02.103846+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.857448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.857448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844867+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103850+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:37:02.103850+0000","last_peered":"2024-03-11T21:37:02.103850+0000","last_clean":"2024-03-11T21:37:02.103850+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:37:02.103850+0000","last_undegraded":"2024-03-11T21:37:02.103850+0000","last_fullsized":"2024-03-11T21:37:02.103850+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:07.672318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1043,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201780+0000","last_change":"2024-03-11T21:36:07.268802+0000","last_active":"2024-03-11T21:37:03.201780+0000","last_peered":"2024-03-11T21:37:03.201780+0000","last_clean":"2024-03-11T21:37:03.201780+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:37:03.201780+0000","last_undegraded":"2024-03-11T21:37:03.201780+0000","last_fullsized":"2024-03-11T21:37:03.201780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:36:07.268562+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.268562+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:10.428038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1004,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098968+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:37:02.098968+0000","last_peered":"2024-03-11T21:37:02.098968+0000","last_clean":"2024-03-11T21:37:02.098968+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:37:02.098968+0000","last_undegraded":"2024-03-11T21:37:02.098968+0000","last_fullsized":"2024-03-11T21:37:02.098968+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":993,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102076+0000","last_change":"2024-03-11T21:35:12.702568+0000","last_active":"2024-03-11T21:37:02.102076+0000","last_peered":"2024-03-11T21:37:02.102076+0000","last_clean":"2024-03-11T21:37:02.102076+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:37:02.102076+0000","last_undegraded":"2024-03-11T21:37:02.102076+0000","last_fullsized":"2024-03-11T21:37:02.102076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.702301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.702301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.845144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":984,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435122+0000","last_change":"2024-03-11T21:36:07.698299+0000","last_active":"2024-03-11T21:37:01.435122+0000","last_peered":"2024-03-11T21:37:01.435122+0000","last_clean":"2024-03-11T21:37:01.435122+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:37:01.435122+0000","last_undegraded":"2024-03-11T21:37:01.435122+0000","last_fullsized":"2024-03-11T21:37:01.435122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.698066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.698066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.708798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435307+0000","last_change":"2024-03-11T21:36:08.684202+0000","last_active":"2024-03-11T21:37:01.435307+0000","last_peered":"2024-03-11T21:37:01.435307+0000","last_clean":"2024-03-11T21:37:01.435307+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:37:01.435307+0000","last_undegraded":"2024-03-11T21:37:01.435307+0000","last_fullsized":"2024-03-11T21:37:01.435307+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.684101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.684101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.559643+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1184,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432248+0000","last_change":"2024-03-11T21:36:19.742269+0000","last_active":"2024-03-11T21:37:01.432248+0000","last_peered":"2024-03-11T21:37:01.432248+0000","last_clean":"2024-03-11T21:37:01.432248+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:37:01.432248+0000","last_undegraded":"2024-03-11T21:37:01.432248+0000","last_fullsized":"2024-03-11T21:37:01.432248+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:36:19.742119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.742119+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:31.824078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1019,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201953+0000","last_change":"2024-03-11T21:36:13.204077+0000","last_active":"2024-03-11T21:37:03.201953+0000","last_peered":"2024-03-11T21:37:03.201953+0000","last_clean":"2024-03-11T21:37:03.201953+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:37:03.201953+0000","last_undegraded":"2024-03-11T21:37:03.201953+0000","last_fullsized":"2024-03-11T21:37:03.201953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:13.203841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.203841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.395164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1053,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519585+0000","last_change":"2024-03-11T21:34:53.090278+0000","last_active":"2024-03-11T21:37:03.519585+0000","last_peered":"2024-03-11T21:37:03.519585+0000","last_clean":"2024-03-11T21:37:03.519585+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:37:03.519585+0000","last_undegraded":"2024-03-11T21:37:03.519585+0000","last_fullsized":"2024-03-11T21:37:03.519585+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:53.090147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.090147+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1032,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.441013+0000","last_change":"2024-03-11T21:36:42.718457+0000","last_active":"2024-03-11T21:37:02.441013+0000","last_peered":"2024-03-11T21:37:02.441013+0000","last_clean":"2024-03-11T21:37:02.441013+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:37:02.441013+0000","last_undegraded":"2024-03-11T21:37:02.441013+0000","last_fullsized":"2024-03-11T21:37:02.441013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:36:42.718312+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.718312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:06.767644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":985,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103883+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:37:02.103883+0000","last_peered":"2024-03-11T21:37:02.103883+0000","last_clean":"2024-03-11T21:37:02.103883+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:37:02.103883+0000","last_undegraded":"2024-03-11T21:37:02.103883+0000","last_fullsized":"2024-03-11T21:37:02.103883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":841,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104473+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:37:02.104473+0000","last_peered":"2024-03-11T21:37:02.104473+0000","last_clean":"2024-03-11T21:37:02.104473+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:37:02.104473+0000","last_undegraded":"2024-03-11T21:37:02.104473+0000","last_fullsized":"2024-03-11T21:37:02.104473+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1010,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201642+0000","last_change":"2024-03-11T21:36:41.246983+0000","last_active":"2024-03-11T21:37:03.201642+0000","last_peered":"2024-03-11T21:37:03.201642+0000","last_clean":"2024-03-11T21:37:03.201642+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:37:03.201642+0000","last_undegraded":"2024-03-11T21:37:03.201642+0000","last_fullsized":"2024-03-11T21:37:03.201642+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:41.246781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:41.246781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.546575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1827,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.736390+0000","last_change":"2024-03-11T21:36:38.633175+0000","last_active":"2024-03-11T21:37:01.736390+0000","last_peered":"2024-03-11T21:37:01.736390+0000","last_clean":"2024-03-11T21:37:01.736390+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:37:01.736390+0000","last_undegraded":"2024-03-11T21:37:01.736390+0000","last_fullsized":"2024-03-11T21:37:01.736390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:36:38.633053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:38.633053+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.674424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":819,"num_read_kb":814,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1025,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.437707+0000","last_change":"2024-03-11T21:34:33.535935+0000","last_active":"2024-03-11T21:37:01.437707+0000","last_peered":"2024-03-11T21:37:01.437707+0000","last_clean":"2024-03-11T21:37:01.437707+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:37:01.437707+0000","last_undegraded":"2024-03-11T21:37:01.437707+0000","last_fullsized":"2024-03-11T21:37:01.437707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:34:33.535753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.535753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":921,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474374+0000","last_change":"2024-03-11T21:36:09.043237+0000","last_active":"2024-03-11T21:37:04.474374+0000","last_peered":"2024-03-11T21:37:04.474374+0000","last_clean":"2024-03-11T21:37:04.474374+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:37:04.474374+0000","last_undegraded":"2024-03-11T21:37:04.474374+0000","last_fullsized":"2024-03-11T21:37:04.474374+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:09.043133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.043133+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.706316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1006,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202146+0000","last_change":"2024-03-11T21:36:37.141040+0000","last_active":"2024-03-11T21:37:03.202146+0000","last_peered":"2024-03-11T21:37:03.202146+0000","last_clean":"2024-03-11T21:37:03.202146+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:37:03.202146+0000","last_undegraded":"2024-03-11T21:37:03.202146+0000","last_fullsized":"2024-03-11T21:37:03.202146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.140941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.140941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.411940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":841,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104805+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:37:02.104805+0000","last_peered":"2024-03-11T21:37:02.104805+0000","last_clean":"2024-03-11T21:37:02.104805+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:37:02.104805+0000","last_undegraded":"2024-03-11T21:37:02.104805+0000","last_fullsized":"2024-03-11T21:37:02.104805+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":999,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.438067+0000","last_change":"2024-03-11T21:34:57.364338+0000","last_active":"2024-03-11T21:37:02.438067+0000","last_peered":"2024-03-11T21:37:02.438067+0000","last_clean":"2024-03-11T21:37:02.438067+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:37:02.438067+0000","last_undegraded":"2024-03-11T21:37:02.438067+0000","last_fullsized":"2024-03-11T21:37:02.438067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.364057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.364057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:02.677258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1126,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.477035+0000","last_change":"2024-03-11T21:34:55.143888+0000","last_active":"2024-03-11T21:37:04.477035+0000","last_peered":"2024-03-11T21:37:04.477035+0000","last_clean":"2024-03-11T21:37:04.477035+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:37:04.477035+0000","last_undegraded":"2024-03-11T21:37:04.477035+0000","last_fullsized":"2024-03-11T21:37:04.477035+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.143614+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979999+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":995,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104614+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:37:02.104614+0000","last_peered":"2024-03-11T21:37:02.104614+0000","last_clean":"2024-03-11T21:37:02.104614+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:37:02.104614+0000","last_undegraded":"2024-03-11T21:37:02.104614+0000","last_fullsized":"2024-03-11T21:37:02.104614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":860,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474029+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:37:04.474029+0000","last_peered":"2024-03-11T21:37:04.474029+0000","last_clean":"2024-03-11T21:37:04.474029+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:37:04.474029+0000","last_undegraded":"2024-03-11T21:37:04.474029+0000","last_fullsized":"2024-03-11T21:37:04.474029+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":849,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521430+0000","last_change":"2024-03-11T21:34:35.105891+0000","last_active":"2024-03-11T21:37:03.521430+0000","last_peered":"2024-03-11T21:37:03.521430+0000","last_clean":"2024-03-11T21:37:03.521430+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:37:03.521430+0000","last_undegraded":"2024-03-11T21:37:03.521430+0000","last_fullsized":"2024-03-11T21:37:03.521430+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.105714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.105714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1002,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102071+0000","last_change":"2024-03-11T21:35:03.774565+0000","last_active":"2024-03-11T21:37:02.102071+0000","last_peered":"2024-03-11T21:37:02.102071+0000","last_clean":"2024-03-11T21:37:02.102071+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:37:02.102071+0000","last_undegraded":"2024-03-11T21:37:02.102071+0000","last_fullsized":"2024-03-11T21:37:02.102071+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.774354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.774354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1005,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098773+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:37:02.098773+0000","last_peered":"2024-03-11T21:37:02.098773+0000","last_clean":"2024-03-11T21:37:02.098773+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:37:02.098773+0000","last_undegraded":"2024-03-11T21:37:02.098773+0000","last_fullsized":"2024-03-11T21:37:02.098773+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:07.672501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1050,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518443+0000","last_change":"2024-03-11T21:36:03.222156+0000","last_active":"2024-03-11T21:37:03.518443+0000","last_peered":"2024-03-11T21:37:03.518443+0000","last_clean":"2024-03-11T21:37:03.518443+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:37:03.518443+0000","last_undegraded":"2024-03-11T21:37:03.518443+0000","last_fullsized":"2024-03-11T21:37:03.518443+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:03.221883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.221883+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.237647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":870,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474497+0000","last_change":"2024-03-11T21:35:04.076881+0000","last_active":"2024-03-11T21:37:04.474497+0000","last_peered":"2024-03-11T21:37:04.474497+0000","last_clean":"2024-03-11T21:37:04.474497+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:37:04.474497+0000","last_undegraded":"2024-03-11T21:37:04.474497+0000","last_fullsized":"2024-03-11T21:37:04.474497+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.076789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.076789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.980154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.431772+0000","last_change":"2024-03-11T21:34:54.501393+0000","last_active":"2024-03-11T21:37:01.431772+0000","last_peered":"2024-03-11T21:37:01.431772+0000","last_clean":"2024-03-11T21:37:01.431772+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:37:01.431772+0000","last_undegraded":"2024-03-11T21:37:01.431772+0000","last_fullsized":"2024-03-11T21:37:01.431772+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.501183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.501183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":987,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104547+0000","last_change":"2024-03-11T21:36:42.732454+0000","last_active":"2024-03-11T21:37:02.104547+0000","last_peered":"2024-03-11T21:37:02.104547+0000","last_clean":"2024-03-11T21:37:02.104547+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:37:02.104547+0000","last_undegraded":"2024-03-11T21:37:02.104547+0000","last_fullsized":"2024-03-11T21:37:02.104547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:42.732354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.732354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:57.600353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1260,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.098842+0000","last_change":"2024-03-11T21:36:34.527211+0000","last_active":"2024-03-11T21:37:02.098842+0000","last_peered":"2024-03-11T21:37:02.098842+0000","last_clean":"2024-03-11T21:37:02.098842+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:37:02.098842+0000","last_undegraded":"2024-03-11T21:37:02.098842+0000","last_fullsized":"2024-03-11T21:37:02.098842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:36:34.526970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.526970+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.043725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1009,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102383+0000","last_change":"2024-03-11T21:36:22.898119+0000","last_active":"2024-03-11T21:37:02.102383+0000","last_peered":"2024-03-11T21:37:02.102383+0000","last_clean":"2024-03-11T21:37:02.102383+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:37:02.102383+0000","last_undegraded":"2024-03-11T21:37:02.102383+0000","last_fullsized":"2024-03-11T21:37:02.102383+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.898004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.898004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:51.002189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1022,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432844+0000","last_change":"2024-03-11T21:34:38.596842+0000","last_active":"2024-03-11T21:37:01.432844+0000","last_peered":"2024-03-11T21:37:01.432844+0000","last_clean":"2024-03-11T21:37:01.432844+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:37:01.432844+0000","last_undegraded":"2024-03-11T21:37:01.432844+0000","last_fullsized":"2024-03-11T21:37:01.432844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:34:38.596731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.596731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":989,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102188+0000","last_change":"2024-03-11T21:34:57.751893+0000","last_active":"2024-03-11T21:37:02.102188+0000","last_peered":"2024-03-11T21:37:02.102188+0000","last_clean":"2024-03-11T21:37:02.102188+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:37:02.102188+0000","last_undegraded":"2024-03-11T21:37:02.102188+0000","last_fullsized":"2024-03-11T21:37:02.102188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.751652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100867+0000","last_change":"2024-03-11T21:36:44.952662+0000","last_active":"2024-03-11T21:37:02.100867+0000","last_peered":"2024-03-11T21:37:02.100867+0000","last_clean":"2024-03-11T21:37:02.100867+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:37:02.100867+0000","last_undegraded":"2024-03-11T21:37:02.100867+0000","last_fullsized":"2024-03-11T21:37:02.100867+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.952524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.952524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.790806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1015,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.114461+0000","last_change":"2024-03-11T21:35:10.391001+0000","last_active":"2024-03-11T21:37:02.114461+0000","last_peered":"2024-03-11T21:37:02.114461+0000","last_clean":"2024-03-11T21:37:02.114461+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:37:02.114461+0000","last_undegraded":"2024-03-11T21:37:02.114461+0000","last_fullsized":"2024-03-11T21:37:02.114461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.390804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.390804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.652712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1023,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.107131+0000","last_change":"2024-03-11T21:36:14.787093+0000","last_active":"2024-03-11T21:37:02.107131+0000","last_peered":"2024-03-11T21:37:02.107131+0000","last_clean":"2024-03-11T21:37:02.107131+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:37:02.107131+0000","last_undegraded":"2024-03-11T21:37:02.107131+0000","last_fullsized":"2024-03-11T21:37:02.107131+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.786963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.786963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":952,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.470801+0000","last_change":"2024-03-11T21:36:04.117071+0000","last_active":"2024-03-11T21:37:04.470801+0000","last_peered":"2024-03-11T21:37:04.470801+0000","last_clean":"2024-03-11T21:37:04.470801+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:37:04.470801+0000","last_undegraded":"2024-03-11T21:37:04.470801+0000","last_fullsized":"2024-03-11T21:37:04.470801+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:36:04.116720+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.116720+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.016493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":983,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104605+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:37:02.104605+0000","last_peered":"2024-03-11T21:37:02.104605+0000","last_clean":"2024-03-11T21:37:02.104605+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:37:02.104605+0000","last_undegraded":"2024-03-11T21:37:02.104605+0000","last_fullsized":"2024-03-11T21:37:02.104605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":857,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517827+0000","last_change":"2024-03-11T21:36:31.174299+0000","last_active":"2024-03-11T21:37:03.517827+0000","last_peered":"2024-03-11T21:37:03.517827+0000","last_clean":"2024-03-11T21:37:03.517827+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:37:03.517827+0000","last_undegraded":"2024-03-11T21:37:03.517827+0000","last_fullsized":"2024-03-11T21:37:03.517827+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:31.174194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.174194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.037963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1013,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521580+0000","last_change":"2024-03-11T21:36:28.181725+0000","last_active":"2024-03-11T21:37:03.521580+0000","last_peered":"2024-03-11T21:37:03.521580+0000","last_clean":"2024-03-11T21:37:03.521580+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:37:03.521580+0000","last_undegraded":"2024-03-11T21:37:03.521580+0000","last_fullsized":"2024-03-11T21:37:03.521580+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.217593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1073,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202585+0000","last_change":"2024-03-11T21:36:40.206751+0000","last_active":"2024-03-11T21:37:03.202585+0000","last_peered":"2024-03-11T21:37:03.202585+0000","last_clean":"2024-03-11T21:37:03.202585+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:37:03.202585+0000","last_undegraded":"2024-03-11T21:37:03.202585+0000","last_fullsized":"2024-03-11T21:37:03.202585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:36:40.206510+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.206510+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.588861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1010,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101301+0000","last_change":"2024-03-11T21:36:02.740887+0000","last_active":"2024-03-11T21:37:02.101301+0000","last_peered":"2024-03-11T21:37:02.101301+0000","last_clean":"2024-03-11T21:37:02.101301+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:37:02.101301+0000","last_undegraded":"2024-03-11T21:37:02.101301+0000","last_fullsized":"2024-03-11T21:37:02.101301+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:02.740740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.740740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:08.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":992,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.437011+0000","last_change":"2024-03-11T21:36:25.713628+0000","last_active":"2024-03-11T21:37:01.437011+0000","last_peered":"2024-03-11T21:37:01.437011+0000","last_clean":"2024-03-11T21:37:01.437011+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:37:01.437011+0000","last_undegraded":"2024-03-11T21:37:01.437011+0000","last_fullsized":"2024-03-11T21:37:01.437011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.713405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.713405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.494128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":847,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101616+0000","last_change":"2024-03-11T21:34:45.790219+0000","last_active":"2024-03-11T21:37:02.101616+0000","last_peered":"2024-03-11T21:37:02.101616+0000","last_clean":"2024-03-11T21:37:02.101616+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:37:02.101616+0000","last_undegraded":"2024-03-11T21:37:02.101616+0000","last_fullsized":"2024-03-11T21:37:02.101616+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.790012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.790012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1186,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521354+0000","last_change":"2024-03-11T21:34:52.046581+0000","last_active":"2024-03-11T21:37:03.521354+0000","last_peered":"2024-03-11T21:37:03.521354+0000","last_clean":"2024-03-11T21:37:03.521354+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:37:03.521354+0000","last_undegraded":"2024-03-11T21:37:03.521354+0000","last_fullsized":"2024-03-11T21:37:03.521354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:34:52.046352+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.046352+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":905,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.476828+0000","last_change":"2024-03-11T21:36:24.181588+0000","last_active":"2024-03-11T21:37:04.476828+0000","last_peered":"2024-03-11T21:37:04.476828+0000","last_clean":"2024-03-11T21:37:04.476828+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:37:04.476828+0000","last_undegraded":"2024-03-11T21:37:04.476828+0000","last_fullsized":"2024-03-11T21:37:04.476828+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:36:24.181487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.181487+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.095936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1213,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103958+0000","last_change":"2024-03-11T21:36:20.831599+0000","last_active":"2024-03-11T21:37:02.103958+0000","last_peered":"2024-03-11T21:37:02.103958+0000","last_clean":"2024-03-11T21:37:02.103958+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:37:02.103958+0000","last_undegraded":"2024-03-11T21:37:02.103958+0000","last_fullsized":"2024-03-11T21:37:02.103958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:20.831450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.831450+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.836710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":876,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474270+0000","last_change":"2024-03-11T21:36:34.998047+0000","last_active":"2024-03-11T21:37:04.474270+0000","last_peered":"2024-03-11T21:37:04.474270+0000","last_clean":"2024-03-11T21:37:04.474270+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:37:04.474270+0000","last_undegraded":"2024-03-11T21:37:04.474270+0000","last_fullsized":"2024-03-11T21:37:04.474270+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.997840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.997840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:00.667483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":999,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517278+0000","last_change":"2024-03-11T21:34:57.129410+0000","last_active":"2024-03-11T21:37:03.517278+0000","last_peered":"2024-03-11T21:37:03.517278+0000","last_clean":"2024-03-11T21:37:03.517278+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:37:03.517278+0000","last_undegraded":"2024-03-11T21:37:03.517278+0000","last_fullsized":"2024-03-11T21:37:03.517278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.129296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.129296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":729,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474389+0000","last_change":"2024-03-11T21:36:32.028349+0000","last_active":"2024-03-11T21:37:04.474389+0000","last_peered":"2024-03-11T21:37:04.474389+0000","last_clean":"2024-03-11T21:37:04.474389+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:37:04.474389+0000","last_undegraded":"2024-03-11T21:37:04.474389+0000","last_fullsized":"2024-03-11T21:37:04.474389+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:32.028125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:32.028125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:38.254526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1024,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.107434+0000","last_change":"2024-03-11T21:36:19.822311+0000","last_active":"2024-03-11T21:37:02.107434+0000","last_peered":"2024-03-11T21:37:02.107434+0000","last_clean":"2024-03-11T21:37:02.107434+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:37:02.107434+0000","last_undegraded":"2024-03-11T21:37:02.107434+0000","last_fullsized":"2024-03-11T21:37:02.107434+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.822219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.822219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.316423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1044,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.105091+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:37:02.105091+0000","last_peered":"2024-03-11T21:37:02.105091+0000","last_clean":"2024-03-11T21:37:02.105091+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:37:02.105091+0000","last_undegraded":"2024-03-11T21:37:02.105091+0000","last_fullsized":"2024-03-11T21:37:02.105091+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1036,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.200178+0000","last_change":"2024-03-11T21:36:19.279496+0000","last_active":"2024-03-11T21:37:03.200178+0000","last_peered":"2024-03-11T21:37:03.200178+0000","last_clean":"2024-03-11T21:37:03.200178+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:37:03.200178+0000","last_undegraded":"2024-03-11T21:37:03.200178+0000","last_fullsized":"2024-03-11T21:37:03.200178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:36:19.279370+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.279370+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.704439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1001,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.436506+0000","last_change":"2024-03-11T21:36:30.693038+0000","last_active":"2024-03-11T21:37:01.436506+0000","last_peered":"2024-03-11T21:37:01.436506+0000","last_clean":"2024-03-11T21:37:01.436506+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:37:01.436506+0000","last_undegraded":"2024-03-11T21:37:01.436506+0000","last_fullsized":"2024-03-11T21:37:01.436506+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.692917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.692917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.071020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":850,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.436718+0000","last_change":"2024-03-11T21:36:36.681111+0000","last_active":"2024-03-11T21:37:01.436718+0000","last_peered":"2024-03-11T21:37:01.436718+0000","last_clean":"2024-03-11T21:37:01.436718+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:37:01.436718+0000","last_undegraded":"2024-03-11T21:37:01.436718+0000","last_fullsized":"2024-03-11T21:37:01.436718+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.681022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.681022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:51.283896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1021,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102682+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:37:02.102682+0000","last_peered":"2024-03-11T21:37:02.102682+0000","last_clean":"2024-03-11T21:37:02.102682+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:37:02.102682+0000","last_undegraded":"2024-03-11T21:37:02.102682+0000","last_fullsized":"2024-03-11T21:37:02.102682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1032,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.107307+0000","last_change":"2024-03-11T21:36:34.793765+0000","last_active":"2024-03-11T21:37:02.107307+0000","last_peered":"2024-03-11T21:37:02.107307+0000","last_clean":"2024-03-11T21:37:02.107307+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:37:02.107307+0000","last_undegraded":"2024-03-11T21:37:02.107307+0000","last_fullsized":"2024-03-11T21:37:02.107307+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.793536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.132621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1004,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.438024+0000","last_change":"2024-03-11T21:36:34.745217+0000","last_active":"2024-03-11T21:37:01.438024+0000","last_peered":"2024-03-11T21:37:01.438024+0000","last_clean":"2024-03-11T21:37:01.438024+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:37:01.438024+0000","last_undegraded":"2024-03-11T21:37:01.438024+0000","last_fullsized":"2024-03-11T21:37:01.438024+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.744999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.744999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:56.618971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1010,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.111832+0000","last_change":"2024-03-11T21:35:05.456150+0000","last_active":"2024-03-11T21:37:02.111832+0000","last_peered":"2024-03-11T21:37:02.111832+0000","last_clean":"2024-03-11T21:37:02.111832+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:37:02.111832+0000","last_undegraded":"2024-03-11T21:37:02.111832+0000","last_fullsized":"2024-03-11T21:37:02.111832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.652694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.433684+0000","last_change":"2024-03-11T21:34:39.587604+0000","last_active":"2024-03-11T21:37:01.433684+0000","last_peered":"2024-03-11T21:37:01.433684+0000","last_clean":"2024-03-11T21:37:01.433684+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:37:01.433684+0000","last_undegraded":"2024-03-11T21:37:01.433684+0000","last_fullsized":"2024-03-11T21:37:01.433684+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.587525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.587525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1033,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437960+0000","last_change":"2024-03-11T21:36:03.386633+0000","last_active":"2024-03-11T21:37:02.437960+0000","last_peered":"2024-03-11T21:37:02.437960+0000","last_clean":"2024-03-11T21:37:02.437960+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:37:02.437960+0000","last_undegraded":"2024-03-11T21:37:02.437960+0000","last_fullsized":"2024-03-11T21:37:02.437960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:36:03.386397+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.386397+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:16.033266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1024,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.438733+0000","last_change":"2024-03-11T21:36:34.619421+0000","last_active":"2024-03-11T21:37:02.438733+0000","last_peered":"2024-03-11T21:37:02.438733+0000","last_clean":"2024-03-11T21:37:02.438733+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:37:02.438733+0000","last_undegraded":"2024-03-11T21:37:02.438733+0000","last_fullsized":"2024-03-11T21:37:02.438733+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.619136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.619136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.642811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1001,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102312+0000","last_change":"2024-03-11T21:34:58.788887+0000","last_active":"2024-03-11T21:37:02.102312+0000","last_peered":"2024-03-11T21:37:02.102312+0000","last_clean":"2024-03-11T21:37:02.102312+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:37:02.102312+0000","last_undegraded":"2024-03-11T21:37:02.102312+0000","last_fullsized":"2024-03-11T21:37:02.102312+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.788749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.845059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":982,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104478+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:37:02.104478+0000","last_peered":"2024-03-11T21:37:02.104478+0000","last_clean":"2024-03-11T21:37:02.104478+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:37:02.104478+0000","last_undegraded":"2024-03-11T21:37:02.104478+0000","last_fullsized":"2024-03-11T21:37:02.104478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":842,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.433108+0000","last_change":"2024-03-11T21:34:53.520431+0000","last_active":"2024-03-11T21:37:01.433108+0000","last_peered":"2024-03-11T21:37:01.433108+0000","last_clean":"2024-03-11T21:37:01.433108+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:37:01.433108+0000","last_undegraded":"2024-03-11T21:37:01.433108+0000","last_fullsized":"2024-03-11T21:37:01.433108+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.520342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.520342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.581009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1021,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104039+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:37:02.104039+0000","last_peered":"2024-03-11T21:37:02.104039+0000","last_clean":"2024-03-11T21:37:02.104039+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:37:02.104039+0000","last_undegraded":"2024-03-11T21:37:02.104039+0000","last_fullsized":"2024-03-11T21:37:02.104039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1008,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101266+0000","last_change":"2024-03-11T21:34:31.781546+0000","last_active":"2024-03-11T21:37:02.101266+0000","last_peered":"2024-03-11T21:37:02.101266+0000","last_clean":"2024-03-11T21:37:02.101266+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:37:02.101266+0000","last_undegraded":"2024-03-11T21:37:02.101266+0000","last_fullsized":"2024-03-11T21:37:02.101266+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1180,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104252+0000","last_change":"2024-03-11T21:34:55.714432+0000","last_active":"2024-03-11T21:37:02.104252+0000","last_peered":"2024-03-11T21:37:02.104252+0000","last_clean":"2024-03-11T21:37:02.104252+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:37:02.104252+0000","last_undegraded":"2024-03-11T21:37:02.104252+0000","last_fullsized":"2024-03-11T21:37:02.104252+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:34:55.714209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.714209+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.845112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":999,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519769+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:37:03.519769+0000","last_peered":"2024-03-11T21:37:03.519769+0000","last_clean":"2024-03-11T21:37:03.519769+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:37:03.519769+0000","last_undegraded":"2024-03-11T21:37:03.519769+0000","last_fullsized":"2024-03-11T21:37:03.519769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":867,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.438172+0000","last_change":"2024-03-11T21:36:44.625010+0000","last_active":"2024-03-11T21:37:02.438172+0000","last_peered":"2024-03-11T21:37:02.438172+0000","last_clean":"2024-03-11T21:37:02.438172+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:37:02.438172+0000","last_undegraded":"2024-03-11T21:37:02.438172+0000","last_fullsized":"2024-03-11T21:37:02.438172+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.624688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.624688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:07.822451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1039,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518749+0000","last_change":"2024-03-11T21:36:52.718981+0000","last_active":"2024-03-11T21:37:03.518749+0000","last_peered":"2024-03-11T21:37:03.518749+0000","last_clean":"2024-03-11T21:37:03.518749+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:37:03.518749+0000","last_undegraded":"2024-03-11T21:37:03.518749+0000","last_fullsized":"2024-03-11T21:37:03.518749+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1011,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516780+0000","last_change":"2024-03-11T21:34:37.079514+0000","last_active":"2024-03-11T21:37:03.516780+0000","last_peered":"2024-03-11T21:37:03.516780+0000","last_clean":"2024-03-11T21:37:03.516780+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:37:03.516780+0000","last_undegraded":"2024-03-11T21:37:03.516780+0000","last_fullsized":"2024-03-11T21:37:03.516780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.079361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.079361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1174,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435118+0000","last_change":"2024-03-11T21:36:24.749355+0000","last_active":"2024-03-11T21:37:01.435118+0000","last_peered":"2024-03-11T21:37:01.435118+0000","last_clean":"2024-03-11T21:37:01.435118+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:37:01.435118+0000","last_undegraded":"2024-03-11T21:37:01.435118+0000","last_fullsized":"2024-03-11T21:37:01.435118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:36:24.749208+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.749208+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.423022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1005,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.205161+0000","last_change":"2024-03-11T21:36:14.255051+0000","last_active":"2024-03-11T21:37:03.205161+0000","last_peered":"2024-03-11T21:37:03.205161+0000","last_clean":"2024-03-11T21:37:03.205161+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:37:03.205161+0000","last_undegraded":"2024-03-11T21:37:03.205161+0000","last_fullsized":"2024-03-11T21:37:03.205161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.254847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.254847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.154175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":725,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.477333+0000","last_change":"2024-03-11T21:36:39.992513+0000","last_active":"2024-03-11T21:37:04.477333+0000","last_peered":"2024-03-11T21:37:04.477333+0000","last_clean":"2024-03-11T21:37:04.477333+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:37:04.477333+0000","last_undegraded":"2024-03-11T21:37:04.477333+0000","last_fullsized":"2024-03-11T21:37:04.477333+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:39.992222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.992222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:44.338226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1019,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104889+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:37:02.104889+0000","last_peered":"2024-03-11T21:37:02.104889+0000","last_clean":"2024-03-11T21:37:02.104889+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:37:02.104889+0000","last_undegraded":"2024-03-11T21:37:02.104889+0000","last_fullsized":"2024-03-11T21:37:02.104889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1022,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.439997+0000","last_change":"2024-03-11T21:36:06.398266+0000","last_active":"2024-03-11T21:37:02.439997+0000","last_peered":"2024-03-11T21:37:02.439997+0000","last_clean":"2024-03-11T21:37:02.439997+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:37:02.439997+0000","last_undegraded":"2024-03-11T21:37:02.439997+0000","last_fullsized":"2024-03-11T21:37:02.439997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.398145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.398145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.611267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":996,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435788+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:37:01.435788+0000","last_peered":"2024-03-11T21:37:01.435788+0000","last_clean":"2024-03-11T21:37:01.435788+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:37:01.435788+0000","last_undegraded":"2024-03-11T21:37:01.435788+0000","last_fullsized":"2024-03-11T21:37:01.435788+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1007,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517234+0000","last_change":"2024-03-11T21:34:41.069563+0000","last_active":"2024-03-11T21:37:03.517234+0000","last_peered":"2024-03-11T21:37:03.517234+0000","last_clean":"2024-03-11T21:37:03.517234+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:37:03.517234+0000","last_undegraded":"2024-03-11T21:37:03.517234+0000","last_fullsized":"2024-03-11T21:37:03.517234+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.069432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.069432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":864,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516556+0000","last_change":"2024-03-11T21:36:25.203562+0000","last_active":"2024-03-11T21:37:03.516556+0000","last_peered":"2024-03-11T21:37:03.516556+0000","last_clean":"2024-03-11T21:37:03.516556+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:37:03.516556+0000","last_undegraded":"2024-03-11T21:37:03.516556+0000","last_fullsized":"2024-03-11T21:37:03.516556+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.203464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.203464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.544573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1043,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204022+0000","last_change":"2024-03-11T21:36:10.243928+0000","last_active":"2024-03-11T21:37:03.204022+0000","last_peered":"2024-03-11T21:37:03.204022+0000","last_clean":"2024-03-11T21:37:03.204022+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:37:03.204022+0000","last_undegraded":"2024-03-11T21:37:03.204022+0000","last_fullsized":"2024-03-11T21:37:03.204022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:10.243644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.243644+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:13.746160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":912,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.476932+0000","last_change":"2024-03-11T21:34:44.191646+0000","last_active":"2024-03-11T21:37:04.476932+0000","last_peered":"2024-03-11T21:37:04.476932+0000","last_clean":"2024-03-11T21:37:04.476932+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:37:04.476932+0000","last_undegraded":"2024-03-11T21:37:04.476932+0000","last_fullsized":"2024-03-11T21:37:04.476932+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.191432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.191432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.980051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":880,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474067+0000","last_change":"2024-03-11T21:34:32.241415+0000","last_active":"2024-03-11T21:37:04.474067+0000","last_peered":"2024-03-11T21:37:04.474067+0000","last_clean":"2024-03-11T21:37:04.474067+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:37:04.474067+0000","last_undegraded":"2024-03-11T21:37:04.474067+0000","last_fullsized":"2024-03-11T21:37:04.474067+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.241159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.241159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":991,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432492+0000","last_change":"2024-03-11T21:36:40.661182+0000","last_active":"2024-03-11T21:37:01.432492+0000","last_peered":"2024-03-11T21:37:01.432492+0000","last_clean":"2024-03-11T21:37:01.432492+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:37:01.432492+0000","last_undegraded":"2024-03-11T21:37:01.432492+0000","last_fullsized":"2024-03-11T21:37:01.432492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.904990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":855,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.516463+0000","last_change":"2024-03-11T21:35:08.022701+0000","last_active":"2024-03-11T21:37:03.516463+0000","last_peered":"2024-03-11T21:37:03.516463+0000","last_clean":"2024-03-11T21:37:03.516463+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:37:03.516463+0000","last_undegraded":"2024-03-11T21:37:03.516463+0000","last_fullsized":"2024-03-11T21:37:03.516463+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.022536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.022536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1033,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521581+0000","last_change":"2024-03-11T21:36:34.076596+0000","last_active":"2024-03-11T21:37:03.521581+0000","last_peered":"2024-03-11T21:37:03.521581+0000","last_clean":"2024-03-11T21:37:03.521581+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:37:03.521581+0000","last_undegraded":"2024-03-11T21:37:03.521581+0000","last_fullsized":"2024-03-11T21:37:03.521581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:36:34.076364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.076364+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.807798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1008,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102551+0000","last_change":"2024-03-11T21:36:36.046087+0000","last_active":"2024-03-11T21:37:02.102551+0000","last_peered":"2024-03-11T21:37:02.102551+0000","last_clean":"2024-03-11T21:37:02.102551+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:37:02.102551+0000","last_undegraded":"2024-03-11T21:37:02.102551+0000","last_fullsized":"2024-03-11T21:37:02.102551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.045956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.045956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:57.013408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1014,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.438529+0000","last_change":"2024-03-11T21:36:10.461846+0000","last_active":"2024-03-11T21:37:02.438529+0000","last_peered":"2024-03-11T21:37:02.438529+0000","last_clean":"2024-03-11T21:37:02.438529+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:37:02.438529+0000","last_undegraded":"2024-03-11T21:37:02.438529+0000","last_fullsized":"2024-03-11T21:37:02.438529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.461724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.461724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.175158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":982,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103847+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:37:02.103847+0000","last_peered":"2024-03-11T21:37:02.103847+0000","last_clean":"2024-03-11T21:37:02.103847+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:37:02.103847+0000","last_undegraded":"2024-03-11T21:37:02.103847+0000","last_fullsized":"2024-03-11T21:37:02.103847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":858,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519236+0000","last_change":"2024-03-11T21:34:51.090840+0000","last_active":"2024-03-11T21:37:03.519236+0000","last_peered":"2024-03-11T21:37:03.519236+0000","last_clean":"2024-03-11T21:37:03.519236+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:37:03.519236+0000","last_undegraded":"2024-03-11T21:37:03.519236+0000","last_fullsized":"2024-03-11T21:37:03.519236+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:51.090589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:51.090589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1030,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102216+0000","last_change":"2024-03-11T21:34:44.838684+0000","last_active":"2024-03-11T21:37:02.102216+0000","last_peered":"2024-03-11T21:37:02.102216+0000","last_clean":"2024-03-11T21:37:02.102216+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:37:02.102216+0000","last_undegraded":"2024-03-11T21:37:02.102216+0000","last_fullsized":"2024-03-11T21:37:02.102216+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:44.838430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.838430+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844975+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1022,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.440818+0000","last_change":"2024-03-11T21:36:41.700743+0000","last_active":"2024-03-11T21:37:02.440818+0000","last_peered":"2024-03-11T21:37:02.440818+0000","last_clean":"2024-03-11T21:37:02.440818+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:37:02.440818+0000","last_undegraded":"2024-03-11T21:37:02.440818+0000","last_fullsized":"2024-03-11T21:37:02.440818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:36:41.700576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.382265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":986,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100954+0000","last_change":"2024-03-11T21:36:29.509965+0000","last_active":"2024-03-11T21:37:02.100954+0000","last_peered":"2024-03-11T21:37:02.100954+0000","last_clean":"2024-03-11T21:37:02.100954+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:37:02.100954+0000","last_undegraded":"2024-03-11T21:37:02.100954+0000","last_fullsized":"2024-03-11T21:37:02.100954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:29.509836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:29.509836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.692029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":983,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.436617+0000","last_change":"2024-03-11T21:34:42.859611+0000","last_active":"2024-03-11T21:37:01.436617+0000","last_peered":"2024-03-11T21:37:01.436617+0000","last_clean":"2024-03-11T21:37:01.436617+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:37:01.436617+0000","last_undegraded":"2024-03-11T21:37:01.436617+0000","last_fullsized":"2024-03-11T21:37:01.436617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.859451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.859451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580740+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":840,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103911+0000","last_change":"2024-03-11T21:36:46.798923+0000","last_active":"2024-03-11T21:37:02.103911+0000","last_peered":"2024-03-11T21:37:02.103911+0000","last_clean":"2024-03-11T21:37:02.103911+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:37:02.103911+0000","last_undegraded":"2024-03-11T21:37:02.103911+0000","last_fullsized":"2024-03-11T21:37:02.103911+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:36:46.798825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:52.800246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":973,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474208+0000","last_change":"2024-03-11T21:36:39.151588+0000","last_active":"2024-03-11T21:37:04.474208+0000","last_peered":"2024-03-11T21:37:04.474208+0000","last_clean":"2024-03-11T21:37:04.474208+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:37:04.474208+0000","last_undegraded":"2024-03-11T21:37:04.474208+0000","last_fullsized":"2024-03-11T21:37:04.474208+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.151333+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.270569+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1006,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.099974+0000","last_change":"2024-03-11T21:35:05.742077+0000","last_active":"2024-03-11T21:37:02.099974+0000","last_peered":"2024-03-11T21:37:02.099974+0000","last_clean":"2024-03-11T21:37:02.099974+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:37:02.099974+0000","last_undegraded":"2024-03-11T21:37:02.099974+0000","last_fullsized":"2024-03-11T21:37:02.099974+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.741828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.741828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1196,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.441111+0000","last_change":"2024-03-11T21:36:22.429814+0000","last_active":"2024-03-11T21:37:02.441111+0000","last_peered":"2024-03-11T21:37:02.441111+0000","last_clean":"2024-03-11T21:37:02.441111+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:37:02.441111+0000","last_undegraded":"2024-03-11T21:37:02.441111+0000","last_fullsized":"2024-03-11T21:37:02.441111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:36:22.429570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.429570+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.430380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":984,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104783+0000","last_change":"2024-03-11T21:36:33.500511+0000","last_active":"2024-03-11T21:37:02.104783+0000","last_peered":"2024-03-11T21:37:02.104783+0000","last_clean":"2024-03-11T21:37:02.104783+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:37:02.104783+0000","last_undegraded":"2024-03-11T21:37:02.104783+0000","last_fullsized":"2024-03-11T21:37:02.104783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:36:33.500161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.500161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.966184+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":855,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100995+0000","last_change":"2024-03-11T21:36:19.803577+0000","last_active":"2024-03-11T21:37:02.100995+0000","last_peered":"2024-03-11T21:37:02.100995+0000","last_clean":"2024-03-11T21:37:02.100995+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:37:02.100995+0000","last_undegraded":"2024-03-11T21:37:02.100995+0000","last_fullsized":"2024-03-11T21:37:02.100995+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.803281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.803281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.105969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1033,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.431868+0000","last_change":"2024-03-11T21:34:56.556273+0000","last_active":"2024-03-11T21:37:01.431868+0000","last_peered":"2024-03-11T21:37:01.431868+0000","last_clean":"2024-03-11T21:37:01.431868+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:37:01.431868+0000","last_undegraded":"2024-03-11T21:37:01.431868+0000","last_fullsized":"2024-03-11T21:37:01.431868+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:56.556007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.556007+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.581200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1004,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.432620+0000","last_change":"2024-03-11T21:36:23.725753+0000","last_active":"2024-03-11T21:37:01.432620+0000","last_peered":"2024-03-11T21:37:01.432620+0000","last_clean":"2024-03-11T21:37:01.432620+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:37:01.432620+0000","last_undegraded":"2024-03-11T21:37:01.432620+0000","last_fullsized":"2024-03-11T21:37:01.432620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:23.725651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:23.725651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.581569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1004,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.519725+0000","last_change":"2024-03-11T21:35:01.025842+0000","last_active":"2024-03-11T21:37:03.519725+0000","last_peered":"2024-03-11T21:37:03.519725+0000","last_clean":"2024-03-11T21:37:03.519725+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:37:03.519725+0000","last_undegraded":"2024-03-11T21:37:03.519725+0000","last_fullsized":"2024-03-11T21:37:03.519725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.025597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.025597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.158166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":871,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474811+0000","last_change":"2024-03-11T21:36:12.033537+0000","last_active":"2024-03-11T21:37:04.474811+0000","last_peered":"2024-03-11T21:37:04.474811+0000","last_clean":"2024-03-11T21:37:04.474811+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:37:04.474811+0000","last_undegraded":"2024-03-11T21:37:04.474811+0000","last_fullsized":"2024-03-11T21:37:04.474811+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.033259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.033259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.444912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":832,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102421+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:37:02.102421+0000","last_peered":"2024-03-11T21:37:02.102421+0000","last_clean":"2024-03-11T21:37:02.102421+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:37:02.102421+0000","last_undegraded":"2024-03-11T21:37:02.102421+0000","last_fullsized":"2024-03-11T21:37:02.102421+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1022,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521255+0000","last_change":"2024-03-11T21:36:27.196610+0000","last_active":"2024-03-11T21:37:03.521255+0000","last_peered":"2024-03-11T21:37:03.521255+0000","last_clean":"2024-03-11T21:37:03.521255+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:37:03.521255+0000","last_undegraded":"2024-03-11T21:37:03.521255+0000","last_fullsized":"2024-03-11T21:37:03.521255+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:36:27.196386+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.196386+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:52.764389+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1022,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.437912+0000","last_change":"2024-03-11T21:36:40.711020+0000","last_active":"2024-03-11T21:37:02.437912+0000","last_peered":"2024-03-11T21:37:02.437912+0000","last_clean":"2024-03-11T21:37:02.437912+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:37:02.437912+0000","last_undegraded":"2024-03-11T21:37:02.437912+0000","last_fullsized":"2024-03-11T21:37:02.437912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.710906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:09.755708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":865,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.477103+0000","last_change":"2024-03-11T21:36:28.149075+0000","last_active":"2024-03-11T21:37:04.477103+0000","last_peered":"2024-03-11T21:37:04.477103+0000","last_clean":"2024-03-11T21:37:04.477103+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:37:04.477103+0000","last_undegraded":"2024-03-11T21:37:04.477103+0000","last_fullsized":"2024-03-11T21:37:04.477103+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.148859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.148859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.073735+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1006,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204632+0000","last_change":"2024-03-11T21:36:17.289970+0000","last_active":"2024-03-11T21:37:03.204632+0000","last_peered":"2024-03-11T21:37:03.204632+0000","last_clean":"2024-03-11T21:37:03.204632+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:37:03.204632+0000","last_undegraded":"2024-03-11T21:37:03.204632+0000","last_fullsized":"2024-03-11T21:37:03.204632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:17.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.289725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.559372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":853,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201093+0000","last_change":"2024-03-11T21:35:04.183348+0000","last_active":"2024-03-11T21:37:03.201093+0000","last_peered":"2024-03-11T21:37:03.201093+0000","last_clean":"2024-03-11T21:37:03.201093+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:37:03.201093+0000","last_undegraded":"2024-03-11T21:37:03.201093+0000","last_fullsized":"2024-03-11T21:37:03.201093+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.183144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.183144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.386036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":1047,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.202251+0000","last_change":"2024-03-11T21:35:12.212273+0000","last_active":"2024-03-11T21:37:03.202251+0000","last_peered":"2024-03-11T21:37:03.202251+0000","last_clean":"2024-03-11T21:37:03.202251+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:37:03.202251+0000","last_undegraded":"2024-03-11T21:37:03.202251+0000","last_fullsized":"2024-03-11T21:37:03.202251+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:12.212159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.212159+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.385830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1003,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435435+0000","last_change":"2024-03-11T21:36:12.656033+0000","last_active":"2024-03-11T21:37:01.435435+0000","last_peered":"2024-03-11T21:37:01.435435+0000","last_clean":"2024-03-11T21:37:01.435435+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:37:01.435435+0000","last_undegraded":"2024-03-11T21:37:01.435435+0000","last_fullsized":"2024-03-11T21:37:01.435435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.655615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.655615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.899187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1001,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.204255+0000","last_change":"2024-03-11T21:34:57.186736+0000","last_active":"2024-03-11T21:37:03.204255+0000","last_peered":"2024-03-11T21:37:03.204255+0000","last_clean":"2024-03-11T21:37:03.204255+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:37:03.204255+0000","last_undegraded":"2024-03-11T21:37:03.204255+0000","last_fullsized":"2024-03-11T21:37:03.204255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.186509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.186509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.385716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":984,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.437847+0000","last_change":"2024-03-11T21:34:31.520200+0000","last_active":"2024-03-11T21:37:01.437847+0000","last_peered":"2024-03-11T21:37:01.437847+0000","last_clean":"2024-03-11T21:37:01.437847+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:37:01.437847+0000","last_undegraded":"2024-03-11T21:37:01.437847+0000","last_fullsized":"2024-03-11T21:37:01.437847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.520041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.520041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":841,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104842+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:37:02.104842+0000","last_peered":"2024-03-11T21:37:02.104842+0000","last_clean":"2024-03-11T21:37:02.104842+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:37:02.104842+0000","last_undegraded":"2024-03-11T21:37:02.104842+0000","last_fullsized":"2024-03-11T21:37:02.104842+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1028,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104137+0000","last_change":"2024-03-11T21:34:50.794115+0000","last_active":"2024-03-11T21:37:02.104137+0000","last_peered":"2024-03-11T21:37:02.104137+0000","last_clean":"2024-03-11T21:37:02.104137+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:37:02.104137+0000","last_undegraded":"2024-03-11T21:37:02.104137+0000","last_fullsized":"2024-03-11T21:37:02.104137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:34:50.793817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.793817+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.844758+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":994,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104235+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:37:02.104235+0000","last_peered":"2024-03-11T21:37:02.104235+0000","last_clean":"2024-03-11T21:37:02.104235+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:37:02.104235+0000","last_undegraded":"2024-03-11T21:37:02.104235+0000","last_fullsized":"2024-03-11T21:37:02.104235+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":992,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.102265+0000","last_change":"2024-03-11T21:36:09.648618+0000","last_active":"2024-03-11T21:37:02.102265+0000","last_peered":"2024-03-11T21:37:02.102265+0000","last_clean":"2024-03-11T21:37:02.102265+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:37:02.102265+0000","last_undegraded":"2024-03-11T21:37:02.102265+0000","last_fullsized":"2024-03-11T21:37:02.102265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.648319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.648319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.702874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1007,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.517345+0000","last_change":"2024-03-11T21:34:42.092702+0000","last_active":"2024-03-11T21:37:03.517345+0000","last_peered":"2024-03-11T21:37:03.517345+0000","last_clean":"2024-03-11T21:37:03.517345+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:37:03.517345+0000","last_undegraded":"2024-03-11T21:37:03.517345+0000","last_fullsized":"2024-03-11T21:37:03.517345+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.092348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.092348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:05.157831+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":864,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.203859+0000","last_change":"2024-03-11T21:36:25.256048+0000","last_active":"2024-03-11T21:37:03.203859+0000","last_peered":"2024-03-11T21:37:03.203859+0000","last_clean":"2024-03-11T21:37:03.203859+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:37:03.203859+0000","last_undegraded":"2024-03-11T21:37:03.203859+0000","last_fullsized":"2024-03-11T21:37:03.203859+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.255841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.255841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.942368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1063,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.101713+0000","last_change":"2024-03-11T21:36:11.636803+0000","last_active":"2024-03-11T21:37:02.101713+0000","last_peered":"2024-03-11T21:37:02.101713+0000","last_clean":"2024-03-11T21:37:02.101713+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:37:02.101713+0000","last_undegraded":"2024-03-11T21:37:02.101713+0000","last_fullsized":"2024-03-11T21:37:02.101713+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:36:11.636460+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.636460+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.421627+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1005,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104748+0000","last_change":"2024-03-11T21:36:11.544839+0000","last_active":"2024-03-11T21:37:02.104748+0000","last_peered":"2024-03-11T21:37:02.104748+0000","last_clean":"2024-03-11T21:37:02.104748+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:37:02.104748+0000","last_undegraded":"2024-03-11T21:37:02.104748+0000","last_fullsized":"2024-03-11T21:37:02.104748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.544702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.544702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.699059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":995,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.103728+0000","last_change":"2024-03-11T21:36:26.444581+0000","last_active":"2024-03-11T21:37:02.103728+0000","last_peered":"2024-03-11T21:37:02.103728+0000","last_clean":"2024-03-11T21:37:02.103728+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:37:02.103728+0000","last_undegraded":"2024-03-11T21:37:02.103728+0000","last_fullsized":"2024-03-11T21:37:02.103728+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.444362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.444362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.863127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":997,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.100235+0000","last_change":"2024-03-11T21:36:14.674630+0000","last_active":"2024-03-11T21:37:02.100235+0000","last_peered":"2024-03-11T21:37:02.100235+0000","last_clean":"2024-03-11T21:37:02.100235+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:37:02.100235+0000","last_undegraded":"2024-03-11T21:37:02.100235+0000","last_fullsized":"2024-03-11T21:37:02.100235+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.674494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.674494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:18.870558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":849,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:02.104268+0000","last_change":"2024-03-11T21:36:27.467589+0000","last_active":"2024-03-11T21:37:02.104268+0000","last_peered":"2024-03-11T21:37:02.104268+0000","last_clean":"2024-03-11T21:37:02.104268+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:37:02.104268+0000","last_undegraded":"2024-03-11T21:37:02.104268+0000","last_fullsized":"2024-03-11T21:37:02.104268+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:27.467488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.467488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.587538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1052,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.521849+0000","last_change":"2024-03-11T21:36:26.178132+0000","last_active":"2024-03-11T21:37:03.521849+0000","last_peered":"2024-03-11T21:37:03.521849+0000","last_clean":"2024-03-11T21:37:03.521849+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:37:03.521849+0000","last_undegraded":"2024-03-11T21:37:03.521849+0000","last_fullsized":"2024-03-11T21:37:03.521849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:36:26.177873+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.177873+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.726625+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":997,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.436950+0000","last_change":"2024-03-11T21:34:35.593537+0000","last_active":"2024-03-11T21:37:01.436950+0000","last_peered":"2024-03-11T21:37:01.436950+0000","last_clean":"2024-03-11T21:37:01.436950+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:37:01.436950+0000","last_undegraded":"2024-03-11T21:37:01.436950+0000","last_fullsized":"2024-03-11T21:37:01.436950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.592784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.592784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.580689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":861,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474429+0000","last_change":"2024-03-11T21:34:43.228208+0000","last_active":"2024-03-11T21:37:04.474429+0000","last_peered":"2024-03-11T21:37:04.474429+0000","last_clean":"2024-03-11T21:37:04.474429+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:37:04.474429+0000","last_undegraded":"2024-03-11T21:37:04.474429+0000","last_fullsized":"2024-03-11T21:37:04.474429+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.227861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.227861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:04.979921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1014,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201937+0000","last_change":"2024-03-11T21:36:21.238514+0000","last_active":"2024-03-11T21:37:03.201937+0000","last_peered":"2024-03-11T21:37:03.201937+0000","last_clean":"2024-03-11T21:37:03.201937+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:37:03.201937+0000","last_undegraded":"2024-03-11T21:37:03.201937+0000","last_fullsized":"2024-03-11T21:37:03.201937+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:21.238275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.238275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.708937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":719,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-11T21:37:04.474768+0000","last_change":"2024-03-11T21:36:22.157617+0000","last_active":"2024-03-11T21:37:04.474768+0000","last_peered":"2024-03-11T21:37:04.474768+0000","last_clean":"2024-03-11T21:37:04.474768+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:37:04.474768+0000","last_undegraded":"2024-03-11T21:37:04.474768+0000","last_fullsized":"2024-03-11T21:37:04.474768+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.157410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.157410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.753331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1030,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.437130+0000","last_change":"2024-03-11T21:36:15.678133+0000","last_active":"2024-03-11T21:37:01.437130+0000","last_peered":"2024-03-11T21:37:01.437130+0000","last_clean":"2024-03-11T21:37:01.437130+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:37:01.437130+0000","last_undegraded":"2024-03-11T21:37:01.437130+0000","last_fullsized":"2024-03-11T21:37:01.437130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:15.678027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.678027+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.603669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1021,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.518342+0000","last_change":"2024-03-11T21:36:24.220137+0000","last_active":"2024-03-11T21:37:03.518342+0000","last_peered":"2024-03-11T21:37:03.518342+0000","last_clean":"2024-03-11T21:37:03.518342+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:37:03.518342+0000","last_undegraded":"2024-03-11T21:37:03.518342+0000","last_fullsized":"2024-03-11T21:37:03.518342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:36:24.220021+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.220021+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.512238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1018,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-11T21:37:03.201195+0000","last_change":"2024-03-11T21:36:30.228605+0000","last_active":"2024-03-11T21:37:03.201195+0000","last_peered":"2024-03-11T21:37:03.201195+0000","last_clean":"2024-03-11T21:37:03.201195+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:37:03.201195+0000","last_undegraded":"2024-03-11T21:37:03.201195+0000","last_fullsized":"2024-03-11T21:37:03.201195+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.228458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.228458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.430097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":983,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-11T21:37:01.435214+0000","last_change":"2024-03-11T21:35:00.524627+0000","last_active":"2024-03-11T21:37:01.435214+0000","last_peered":"2024-03-11T21:37:01.435214+0000","last_clean":"2024-03-11T21:37:01.435214+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:37:01.435214+0000","last_undegraded":"2024-03-11T21:37:01.435214+0000","last_fullsized":"2024-03-11T21:37:01.435214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:00.524430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.524430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:03.581054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":296,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8196627,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":122,"num_read_kb":15,"num_write":635,"num_write_kb":12717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26673152,"data_stored":24589794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1966,"ondisk_log_size":1966,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":819,"num_read_kb":814,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841715,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89504,"kb_used_data":13900,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655648,"statfs":{"total":95995035648,"available":95903383552,"internally_reserved":0,"allocated":14233600,"data_stored":10437596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168764,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76664,"kb_used_data":13528,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93668488,"statfs":{"total":95995035648,"available":95916531712,"internally_reserved":0,"allocated":13852672,"data_stored":10250942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352387,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37088,"kb_used_data":9152,"kb_used_omap":27,"kb_used_meta":27876,"kb_avail":93708064,"statfs":{"total":95995035648,"available":95957057536,"internally_reserved":0,"allocated":9371648,"data_stored":5683181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":28545353},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738619,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93252,"kb_used_data":17636,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93651900,"statfs":{"total":95995035648,"available":95899545600,"internally_reserved":0,"allocated":18059264,"data_stored":14446373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378854,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80584,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664568,"statfs":{"total":95995035648,"available":95912517632,"internally_reserved":0,"allocated":9359360,"data_stored":5651055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117210,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76424,"kb_used_data":9116,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93668728,"statfs":{"total":95995035648,"available":95916777472,"internally_reserved":0,"allocated":9334784,"data_stored":5650508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855568,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88756,"kb_used_data":13128,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93656396,"statfs":{"total":95995035648,"available":95904149504,"internally_reserved":0,"allocated":13443072,"data_stored":9652114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495814,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80428,"kb_used_data":13144,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93664724,"statfs":{"total":95995035648,"available":95912677376,"internally_reserved":0,"allocated":13459456,"data_stored":9844641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8376320,"data_stored":8194745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":2582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4345856,"data_stored":4000797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4177920,"data_stored":4000426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4542464,"data_stored":4195093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":296,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:37:08.843 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-11T21:37:08.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:08.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:09.051 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:09.076 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:09.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:09.205 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:09.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:09.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:09.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:09.465 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:09.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:09.580 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:09 smithi203 bash[23654]: cluster 2024-03-11T21:37:08.502630+0000 mon.a (mon.0) 2969 : cluster 0 osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:37:09.580 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:09 smithi203 bash[23654]: audit 2024-03-11T21:37:08.772292+0000 mgr.y (mgr.14550) 559 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:09.595 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:09.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:09.727 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:09.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:09.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:09 smithi143 bash[21512]: cluster 2024-03-11T21:37:08.502630+0000 mon.a (mon.0) 2969 : cluster 0 osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:37:09.842 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:09 smithi143 bash[21512]: audit 2024-03-11T21:37:08.772292+0000 mgr.y (mgr.14550) 559 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:09.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:09 smithi143 bash[29284]: cluster 2024-03-11T21:37:08.502630+0000 mon.a (mon.0) 2969 : cluster 0 osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:37:09.842 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:09 smithi143 bash[29284]: audit 2024-03-11T21:37:08.772292+0000 mgr.y (mgr.14550) 559 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:09.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:09.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:09.991 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:10.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:10.223 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.252 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:10.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:10.380 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:10.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:10.523 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:10.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:10.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:10.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:10.767 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:10 smithi203 bash[23654]: cluster 2024-03-11T21:37:09.469009+0000 mgr.y (mgr.14550) 560 : cluster 0 pgmap v858: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 34 op/s 2024-03-11T21:37:10.768 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:10 smithi203 bash[23654]: cluster 2024-03-11T21:37:09.511250+0000 mon.a (mon.0) 2970 : cluster 0 osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:37:10.768 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:10 smithi203 bash[23654]: audit 2024-03-11T21:37:09.516606+0000 mon.c (mon.2) 1038 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.768 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:10 smithi203 bash[23654]: audit 2024-03-11T21:37:09.517089+0000 mon.a (mon.0) 2971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:10.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:37:10.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[21512]: cluster 2024-03-11T21:37:09.469009+0000 mgr.y (mgr.14550) 560 : cluster 0 pgmap v858: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 34 op/s 2024-03-11T21:37:10.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[21512]: cluster 2024-03-11T21:37:09.511250+0000 mon.a (mon.0) 2970 : cluster 0 osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[21512]: audit 2024-03-11T21:37:09.516606+0000 mon.c (mon.2) 1038 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[21512]: audit 2024-03-11T21:37:09.517089+0000 mon.a (mon.0) 2971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[29284]: cluster 2024-03-11T21:37:09.469009+0000 mgr.y (mgr.14550) 560 : cluster 0 pgmap v858: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 34 op/s 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[29284]: cluster 2024-03-11T21:37:09.511250+0000 mon.a (mon.0) 2970 : cluster 0 osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[29284]: audit 2024-03-11T21:37:09.516606+0000 mon.c (mon.2) 1038 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.886 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:10 smithi143 bash[29284]: audit 2024-03-11T21:37:09.517089+0000 mon.a (mon.0) 2971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:10.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.910 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:11.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:11.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:11.168 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:11.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.300 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:11.386 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:11.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:11.429 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:11.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:11.559 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:11.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:11.673 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[21512]: audit 2024-03-11T21:37:10.510093+0000 mon.a (mon.0) 2972 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[21512]: cluster 2024-03-11T21:37:10.518050+0000 mon.a (mon.0) 2973 : cluster 0 osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[21512]: audit 2024-03-11T21:37:10.548634+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[21512]: audit 2024-03-11T21:37:10.554026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[21512]: audit 2024-03-11T21:37:10.555184+0000 mon.a (mon.0) 2974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[29284]: audit 2024-03-11T21:37:10.510093+0000 mon.a (mon.0) 2972 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[29284]: cluster 2024-03-11T21:37:10.518050+0000 mon.a (mon.0) 2973 : cluster 0 osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[29284]: audit 2024-03-11T21:37:10.548634+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[29284]: audit 2024-03-11T21:37:10.554026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:11 smithi143 bash[29284]: audit 2024-03-11T21:37:10.555184+0000 mon.a (mon.0) 2974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.687 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:11 smithi203 bash[23654]: audit 2024-03-11T21:37:10.510093+0000 mon.a (mon.0) 2972 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:11 smithi203 bash[23654]: cluster 2024-03-11T21:37:10.518050+0000 mon.a (mon.0) 2973 : cluster 0 osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:37:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:11 smithi203 bash[23654]: audit 2024-03-11T21:37:10.548634+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:11 smithi203 bash[23654]: audit 2024-03-11T21:37:10.554026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:11 smithi203 bash[23654]: audit 2024-03-11T21:37:10.555184+0000 mon.a (mon.0) 2974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:11.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.817 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:11.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:11.947 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:12.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:12.078 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:12.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:12.209 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:12.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:12.323 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:12.340 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:12.409 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:12.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:12.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:12.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:12.597 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:12.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21512]: cluster 2024-03-11T21:37:11.469563+0000 mgr.y (mgr.14550) 561 : cluster 0 pgmap v861: 260 pgs: 4 creating+peering, 28 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-03-11T21:37:12.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21512]: audit 2024-03-11T21:37:11.529573+0000 mon.a (mon.0) 2975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:12.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21512]: cluster 2024-03-11T21:37:11.535072+0000 mon.a (mon.0) 2976 : cluster 0 osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21512]: audit 2024-03-11T21:37:11.540616+0000 mon.c (mon.2) 1041 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[21512]: audit 2024-03-11T21:37:11.656328+0000 mon.a (mon.0) 2977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[29284]: cluster 2024-03-11T21:37:11.469563+0000 mgr.y (mgr.14550) 561 : cluster 0 pgmap v861: 260 pgs: 4 creating+peering, 28 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[29284]: audit 2024-03-11T21:37:11.529573+0000 mon.a (mon.0) 2975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[29284]: cluster 2024-03-11T21:37:11.535072+0000 mon.a (mon.0) 2976 : cluster 0 osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[29284]: audit 2024-03-11T21:37:11.540616+0000 mon.c (mon.2) 1041 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:12 smithi143 bash[29284]: audit 2024-03-11T21:37:11.656328+0000 mon.a (mon.0) 2977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:12.728 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:12 smithi203 bash[23654]: cluster 2024-03-11T21:37:11.469563+0000 mgr.y (mgr.14550) 561 : cluster 0 pgmap v861: 260 pgs: 4 creating+peering, 28 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-03-11T21:37:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:12 smithi203 bash[23654]: audit 2024-03-11T21:37:11.529573+0000 mon.a (mon.0) 2975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:12 smithi203 bash[23654]: cluster 2024-03-11T21:37:11.535072+0000 mon.a (mon.0) 2976 : cluster 0 osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:37:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:12 smithi203 bash[23654]: audit 2024-03-11T21:37:11.540616+0000 mon.c (mon.2) 1041 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:12 smithi203 bash[23654]: audit 2024-03-11T21:37:11.656328+0000 mon.a (mon.0) 2977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:12.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:12.859 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:12.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:12.989 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:13.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.119 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:13.222 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.252 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:13.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:13.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:13.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:13.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:12.542954+0000 mon.a (mon.0) 2978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:12.548147+0000 mon.c (mon.2) 1042 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: cluster 2024-03-11T21:37:12.548742+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:12.550297+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:13.544500+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: cluster 2024-03-11T21:37:13.549119+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:13.549526+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.741 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:13 smithi203 bash[23654]: audit 2024-03-11T21:37:13.550555+0000 mon.a (mon.0) 2983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:12.542954+0000 mon.a (mon.0) 2978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:12.548147+0000 mon.c (mon.2) 1042 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: cluster 2024-03-11T21:37:12.548742+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:12.550297+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:13.544500+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: cluster 2024-03-11T21:37:13.549119+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:13.549526+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[29284]: audit 2024-03-11T21:37:13.550555+0000 mon.a (mon.0) 2983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:12.542954+0000 mon.a (mon.0) 2978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:12.548147+0000 mon.c (mon.2) 1042 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: cluster 2024-03-11T21:37:12.548742+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:37:13.768 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:12.550297+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:13.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:13.544500+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:13.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: cluster 2024-03-11T21:37:13.549119+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:37:13.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:13.549526+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.769 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:13 smithi143 bash[21512]: audit 2024-03-11T21:37:13.550555+0000 mon.a (mon.0) 2983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:13.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:13.844 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:37:13.844 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:37:13.844 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T21:37:13.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:37:13.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:13.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:14.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:14.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:14.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:14.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:14.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:14.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:14.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:14.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:14.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:14.561 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:14.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:14.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:14 smithi143 bash[29284]: cluster 2024-03-11T21:37:13.470769+0000 mgr.y (mgr.14550) 562 : cluster 0 pgmap v864: 260 pgs: 4 creating+peering, 5 unknown, 251 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-11T21:37:14.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:14 smithi143 bash[29284]: audit 2024-03-11T21:37:14.549265+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:14.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:14 smithi143 bash[21512]: cluster 2024-03-11T21:37:13.470769+0000 mgr.y (mgr.14550) 562 : cluster 0 pgmap v864: 260 pgs: 4 creating+peering, 5 unknown, 251 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-11T21:37:14.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:14 smithi143 bash[21512]: audit 2024-03-11T21:37:14.549265+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:14.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:14.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:14 smithi203 bash[23654]: cluster 2024-03-11T21:37:13.470769+0000 mgr.y (mgr.14550) 562 : cluster 0 pgmap v864: 260 pgs: 4 creating+peering, 5 unknown, 251 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-11T21:37:14.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:14 smithi203 bash[23654]: audit 2024-03-11T21:37:14.549265+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:14.816 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:14.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:14.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:15.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.078 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:15.181 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:15.209 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:15.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:15.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:15.454 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:15.472 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:15.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:15.600 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:15.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.728 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.729 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:15 smithi143 bash[29284]: cluster 2024-03-11T21:37:14.559564+0000 mon.a (mon.0) 2985 : cluster 0 osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:37:15.730 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:15 smithi143 bash[21512]: cluster 2024-03-11T21:37:14.559564+0000 mon.a (mon.0) 2985 : cluster 0 osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:37:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:15.749 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:15.750 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:15.760 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:15 smithi203 bash[23654]: cluster 2024-03-11T21:37:14.559564+0000 mon.a (mon.0) 2985 : cluster 0 osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:37:15.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:15.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:15.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:15.982 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:16.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:16.113 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:16.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:16.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:16.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:16.371 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:16.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:16.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:37:16.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:16.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:16.625 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:16.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:16.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: cluster 2024-03-11T21:37:15.471968+0000 mgr.y (mgr.14550) 563 : cluster 0 pgmap v867: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: cluster 2024-03-11T21:37:15.584439+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: cluster 2024-03-11T21:37:16.415269+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: audit 2024-03-11T21:37:16.515502+0000 mon.c (mon.2) 1044 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: audit 2024-03-11T21:37:16.515911+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: audit 2024-03-11T21:37:16.516647+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:16 smithi203 bash[23654]: audit 2024-03-11T21:37:16.517022+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:16.883 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:16.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 -- 172.21.15.143:0/2590842009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 msgr2=0x7f6a0c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 --2- 172.21.15.143:0/2590842009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c102340 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009990 tx=0x7f69f4009ca0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 -- 172.21.15.143:0/2590842009 shutdown_connections 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 --2- 172.21.15.143:0/2590842009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 --2- 172.21.15.143:0/2590842009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 0x7f6a0c108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 --2- 172.21.15.143:0/2590842009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c102340 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 -- 172.21.15.143:0/2590842009 >> 172.21.15.143:0/2590842009 conn(0x7f6a0c078eb0 msgr2=0x7f6a0c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 -- 172.21.15.143:0/2590842009 shutdown_connections 2024-03-11T21:37:16.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 -- 172.21.15.143:0/2590842009 wait complete. 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 librados: starting msgr at 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.915+0000 7f6a12181700 1 librados: starting objecter 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 Processor -- start 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- start start 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 librados: setting wanted keys 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 librados: calling monclient init 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 0x7f6a0c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c111660 con 0x7f6a0c101ec0 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c1114e0 con 0x7f6a0c107f30 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c1117e0 con 0x7f6a0c102880 2024-03-11T21:37:16.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48800/0 (socket says 172.21.15.143:48800) 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0affd700 1 -- 172.21.15.143:0/1407060566 learned_addr learned my addr 172.21.15.143:0/1407060566 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 0x7f6a0c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 msgr2=0x7f6a0c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 msgr2=0x7f6a0c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 0x7f6a0c074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c1a6080 con 0x7f6a0c101ec0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a0c107f30 0x7f6a0c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c102880 0x7f6a0c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c0740e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c970 tx=0x7f69fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 10 client.?.objecter ms_handle_connect 0x7f6a0c101ec0 2024-03-11T21:37:16.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:16.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc014070 con 0x7f6a0c101ec0 2024-03-11T21:37:16.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f69fc004440 con 0x7f6a0c101ec0 2024-03-11T21:37:16.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0c1a6370 con 0x7f6a0c101ec0 2024-03-11T21:37:16.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0039f0 con 0x7f6a0c101ec0 2024-03-11T21:37:16.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 10 client.30561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:16.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0c1a6900 con 0x7f6a0c101ec0 2024-03-11T21:37:16.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 librados: init done 2024-03-11T21:37:16.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6a0c04faa0 con 0x7f6a0c101ec0 2024-03-11T21:37:16.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f69fc017070 con 0x7f6a0c101ec0 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f69f806f820 0x7f69f8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(564..564 src has 87..564) v4 ==== 9696+0+0 (secure 0 0 0) 0x7f69fc08e710 con 0x7f6a0c101ec0 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 osd_map(564..564 src has 87..564) v4 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 3 client.30561.objecter handle_osd_map got epochs [564,564] > 0 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a08ff9700 3 client.30561.objecter handle_osd_map decoding full epoch 564 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.919+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f69f806f820 0x7f69f8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 20 client.30561.objecter dump_active .. 0 homeless 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v564) v1 ==== 54+0+16 (secure 0 0 0) 0x7f69fc01e030 con 0x7f6a0c101ec0 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 10 client.30561.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 1 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f6a0c109380 0x7f6a0c109760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 1 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 0 1 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 4 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 10 client.30561.objecter _send_command 1 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1a9d50 con 0x7f6a0c109380 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f6a0c109380 0x7f6a0c109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f69f806f820 0x7f69f8071ce0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c079640 tx=0x7f69f401a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f69f806f820 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c109380 2024-03-11T21:37:16.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f6a0c109380 0x7f6a0c109760 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1a9d50 con 0x7f6a0c109380 2024-03-11T21:37:16.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:37:16.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 1 = system:0 2024-03-11T21:37:16.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 4 2024-03-11T21:37:16.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.923+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 0 1 2024-03-11T21:37:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 10 client.30561.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 2 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 2 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c075610 osd=0 3 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 0 2 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 4 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 10 client.30561.objecter _send_command 2 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.939+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c108e00 con 0x7f6a0c109380 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c108e00 con 0x7f6a0c109380 2024-03-11T21:37:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:16.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:16.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 4 2024-03-11T21:37:16.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 0 2 2024-03-11T21:37:16.920 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:37:16.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 10 client.30561.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:16.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 3 2024-03-11T21:37:16.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f6a0c02c6b0 0x7f6a0c02cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 1 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 4 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 10 client.30561.objecter _send_command 3 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c063070 con 0x7f6a0c02c6b0 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f6a0c02c6b0 0x7f6a0c02cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c02c6b0 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f6a0c02c6b0 0x7f6a0c02cab0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c063070 con 0x7f6a0c02c6b0 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:37:16.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 3 = system:0 2024-03-11T21:37:16.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 4 2024-03-11T21:37:16.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.943+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 1 3 2024-03-11T21:37:16.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: cluster 2024-03-11T21:37:15.471968+0000 mgr.y (mgr.14550) 563 : cluster 0 pgmap v867: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-11T21:37:16.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: cluster 2024-03-11T21:37:15.584439+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:37:16.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: cluster 2024-03-11T21:37:16.415269+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:37:16.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: audit 2024-03-11T21:37:16.515502+0000 mon.c (mon.2) 1044 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: audit 2024-03-11T21:37:16.515911+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: audit 2024-03-11T21:37:16.516647+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[21512]: audit 2024-03-11T21:37:16.517022+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: cluster 2024-03-11T21:37:15.471968+0000 mgr.y (mgr.14550) 563 : cluster 0 pgmap v867: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: cluster 2024-03-11T21:37:15.584439+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: cluster 2024-03-11T21:37:16.415269+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: audit 2024-03-11T21:37:16.515502+0000 mon.c (mon.2) 1044 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: audit 2024-03-11T21:37:16.515911+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: audit 2024-03-11T21:37:16.516647+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:16 smithi143 bash[29284]: audit 2024-03-11T21:37:16.517022+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-83"}]: dispatch 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 10 client.30561.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 4 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 4 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c0629f0 osd=1 3 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 1 4 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 4 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 10 client.30561.objecter _send_command 4 2024-03-11T21:37:16.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c062470 con 0x7f6a0c02c6b0 2024-03-11T21:37:16.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c062470 con 0x7f6a0c02c6b0 2024-03-11T21:37:16.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:16.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:16.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 4 2024-03-11T21:37:16.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 1 4 2024-03-11T21:37:16.938 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:37:16.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 10 client.30561.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 5 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f6a0c1abf10 0x7f6a0c1ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 5 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 2 5 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 4 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 10 client.30561.objecter _send_command 5 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1b1160 con 0x7f6a0c1abf10 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f6a0c1abf10 0x7f6a0c1ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1abf10 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f6a0c1abf10 0x7f6a0c1ae300 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1b1160 con 0x7f6a0c1abf10 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 5 = system:0 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 4 2024-03-11T21:37:16.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.963+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 2 5 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 10 client.30561.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 6 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 6 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c062870 osd=2 3 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 2 6 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 4 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 10 client.30561.objecter _send_command 6 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c02d240 con 0x7f6a0c1abf10 2024-03-11T21:37:16.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c02d240 con 0x7f6a0c1abf10 2024-03-11T21:37:16.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:16.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:16.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 4 2024-03-11T21:37:16.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 2 6 2024-03-11T21:37:16.958 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 10 client.30561.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 7 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f6a0c1a98b0 0x7f6a0c1af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 7 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 3 7 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 4 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 10 client.30561.objecter _send_command 7 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1b4ac0 con 0x7f6a0c1a98b0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f6a0c1a98b0 0x7f6a0c1af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1a98b0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f6a0c1a98b0 0x7f6a0c1af9c0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1b4ac0 con 0x7f6a0c1a98b0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 7 = system:0 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 4 2024-03-11T21:37:16.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:16.983+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 3 7 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 10 client.30561.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 8 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 8 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c02dcd0 osd=3 3 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 3 8 2024-03-11T21:37:16.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 4 2024-03-11T21:37:16.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 10 client.30561.objecter _send_command 8 2024-03-11T21:37:16.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c1a8e80 con 0x7f6a0c1a98b0 2024-03-11T21:37:16.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c1a8e80 con 0x7f6a0c1a98b0 2024-03-11T21:37:16.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:16.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:16.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 4 2024-03-11T21:37:16.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 3 8 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.003+0000 7f6a12181700 10 client.30561.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 9 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f6a0c1af520 0x7f6a0c1b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 9 2024-03-11T21:37:16.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 4 9 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 4 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 10 client.30561.objecter _send_command 9 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1b84b0 con 0x7f6a0c1af520 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f6a0c1af520 0x7f6a0c1b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1af520 2024-03-11T21:37:16.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f6a0c1af520 0x7f6a0c1b3320 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1b84b0 con 0x7f6a0c1af520 2024-03-11T21:37:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:37:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 9 = system:0 2024-03-11T21:37:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 4 2024-03-11T21:37:16.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.007+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 4 9 2024-03-11T21:37:16.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 10 client.30561.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 10 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 10 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1a8c30 osd=4 3 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 4 10 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 4 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 10 client.30561.objecter _send_command 10 2024-03-11T21:37:17.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.027+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c1b3860 con 0x7f6a0c1af520 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c1b3860 con 0x7f6a0c1af520 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 4 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 4 10 2024-03-11T21:37:17.006 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:37:17.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 10 client.30561.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:17.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 11 2024-03-11T21:37:17.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f6a0c1b3d70 0x7f6a0c1b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:17.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 11 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 5 11 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 4 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 10 client.30561.objecter _send_command 11 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1bb640 con 0x7f6a0c1b3d70 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.031+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f6a0c1b3d70 0x7f6a0c1b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:17.009 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:17.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1b3d70 2024-03-11T21:37:17.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a0affd700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f6a0c1b3d70 0x7f6a0c1b41d0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1bb640 con 0x7f6a0c1b3d70 2024-03-11T21:37:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:37:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 11 = system:0 2024-03-11T21:37:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 4 2024-03-11T21:37:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.035+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 5 11 2024-03-11T21:37:17.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 10 client.30561.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 12 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 12 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1a8eb0 osd=5 3 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 5 12 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 4 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 10 client.30561.objecter _send_command 12 2024-03-11T21:37:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.051+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c1b8d20 con 0x7f6a0c1b3d70 2024-03-11T21:37:17.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.055+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c1b8d20 con 0x7f6a0c1b3d70 2024-03-11T21:37:17.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.055+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:17.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.055+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:17.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.055+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 4 2024-03-11T21:37:17.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.055+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 5 12 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 10 client.30561.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 13 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f6a0c1b93a0 0x7f6a0c1bb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 13 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 6 13 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 4 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 10 client.30561.objecter _send_command 13 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1be810 con 0x7f6a0c1b93a0 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f6a0c1b93a0 0x7f6a0c1bb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1b93a0 2024-03-11T21:37:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a0bfff700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f6a0c1b93a0 0x7f6a0c1bb790 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:17.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1be810 con 0x7f6a0c1b93a0 2024-03-11T21:37:17.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:37:17.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 13 = system:0 2024-03-11T21:37:17.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 4 2024-03-11T21:37:17.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.059+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 6 13 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 10 client.30561.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 14 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 14 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c1b8d20 osd=6 3 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 6 14 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 4 2024-03-11T21:37:17.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 10 client.30561.objecter _send_command 14 2024-03-11T21:37:17.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.079+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c000930 con 0x7f6a0c1b93a0 2024-03-11T21:37:17.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c000930 con 0x7f6a0c1b93a0 2024-03-11T21:37:17.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:17.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:17.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 4 2024-03-11T21:37:17.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 6 14 2024-03-11T21:37:17.057 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:37:17.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 10 client.30561.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:17.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 15 2024-03-11T21:37:17.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f6a0c1bdb80 0x7f6a0c1bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 15 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 7 15 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 4 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 10 client.30561.objecter _send_command 15 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0c1c1be0 con 0x7f6a0c1bdb80 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f6a0c1bdb80 0x7f6a0c1bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a08ff9700 10 client.30561.objecter ms_handle_connect 0x7f6a0c1bdb80 2024-03-11T21:37:17.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.083+0000 7f6a0b7fe700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f6a0c1bdb80 0x7f6a0c1bdfc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:17.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.087+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a0c1c1be0 con 0x7f6a0c1bdb80 2024-03-11T21:37:17.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.087+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:37:17.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.087+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 15 = system:0 2024-03-11T21:37:17.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.087+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 4 2024-03-11T21:37:17.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.087+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 7 15 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 10 client.30561.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign -1 16 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 20 client.30561.objecter _get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 15 client.30561.objecter _session_command_op_remove -1 16 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 20 client.30561.objecter get_session s=0x7f6a0c000930 osd=7 3 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 15 client.30561.objecter _session_command_op_assign 7 16 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 4 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 10 client.30561.objecter _send_command 16 2024-03-11T21:37:17.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.103+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a0c1c0a60 con 0x7f6a0c1bdb80 2024-03-11T21:37:17.081 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:37:17.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a08ff9700 1 -- 172.21.15.143:0/1407060566 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6a0c1c0a60 con 0x7f6a0c1bdb80 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a08ff9700 10 client.30561.objecter ms_dispatch 0x7f6a0c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a08ff9700 10 client.30561.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a08ff9700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 4 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a08ff9700 15 client.30561.objecter _session_command_op_remove 7 16 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 librados: watch_flush enter 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 librados: watch_flush exit 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.0 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f6a0c109380 msgr2=0x7f6a0c109760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f6a0c109380 0x7f6a0c109760 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c075610 osd=0 1 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.1 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f6a0c02c6b0 msgr2=0x7f6a0c02cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f6a0c02c6b0 0x7f6a0c02cab0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c0629f0 osd=1 1 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.2 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f6a0c1abf10 msgr2=0x7f6a0c1ae300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f6a0c1abf10 0x7f6a0c1ae300 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c062870 osd=2 1 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.3 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f6a0c1a98b0 msgr2=0x7f6a0c1af9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f6a0c1a98b0 0x7f6a0c1af9c0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a0bfff700 1 -- 172.21.15.143:0/1407060566 reap_dead start 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c02dcd0 osd=3 1 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.4 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f6a0c1af520 msgr2=0x7f6a0c1b3320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f6a0c1af520 0x7f6a0c1b3320 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a0bfff700 1 -- 172.21.15.143:0/1407060566 reap_dead start 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8c30 osd=4 1 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.5 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f6a0c1b3d70 msgr2=0x7f6a0c1b41d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f6a0c1b3d70 0x7f6a0c1b41d0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1a8eb0 osd=5 1 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.6 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f6a0c1b93a0 msgr2=0x7f6a0c1bb790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f6a0c1b93a0 0x7f6a0c1bb790 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c1b8d20 osd=6 1 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter close_session for osd.7 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f6a0c1bdb80 msgr2=0x7f6a0c1bdfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f6a0c1bdb80 0x7f6a0c1bdfc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter put_session s=0x7f6a0c000930 osd=7 1 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 20 client.30561.objecter shutdown clearing up homeless session... 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 10 client.30561.objecter successfully canceled tick 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f69f806f820 msgr2=0x7f69f8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f69f806f820 0x7f69f8071ce0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c079640 tx=0x7f69f401a040 comp rx=0 tx=0).stop 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 msgr2=0x7f6a0c0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 --2- 172.21.15.143:0/1407060566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c101ec0 0x7f6a0c0740e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c970 tx=0x7f69fc00cc80 comp rx=0 tx=0).stop 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a0bfff700 1 -- 172.21.15.143:0/1407060566 reap_dead start 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 shutdown_connections 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 >> 172.21.15.143:0/1407060566 conn(0x7f6a0c078eb0 msgr2=0x7f6a0c105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 shutdown_connections 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 -- 172.21.15.143:0/1407060566 wait complete. 2024-03-11T21:37:17.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:17.107+0000 7f6a12181700 1 librados: shutdown 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:17.087 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:17.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:17.133 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:17.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:17.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:17.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:17.388 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:17.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:17.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:17.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:17.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:17.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:17.773 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:17.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:17.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:17 smithi143 bash[29284]: audit 2024-03-11T21:37:16.922770+0000 mon.a (mon.0) 2990 : audit 0 from='client.? 172.21.15.143:0/1407060566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:17.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:17 smithi143 bash[29284]: cluster 2024-03-11T21:37:17.422797+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:37:17.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:17 smithi143 bash[21512]: audit 2024-03-11T21:37:16.922770+0000 mon.a (mon.0) 2990 : audit 0 from='client.? 172.21.15.143:0/1407060566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:17.884 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:17 smithi143 bash[21512]: cluster 2024-03-11T21:37:17.422797+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:37:17.895 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:17.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:18.022 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:18.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.136 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:17 smithi203 bash[23654]: audit 2024-03-11T21:37:16.922770+0000 mon.a (mon.0) 2990 : audit 0 from='client.? 172.21.15.143:0/1407060566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:18.137 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:17 smithi203 bash[23654]: cluster 2024-03-11T21:37:17.422797+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:37:18.150 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:18.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.278 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:18.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:18.411 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:18.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:18.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:18.645 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:18.673 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:18.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:18.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:18.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:18.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[21512]: cluster 2024-03-11T21:37:17.472511+0000 mgr.y (mgr.14550) 564 : cluster 0 pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:18.913 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[21512]: cluster 2024-03-11T21:37:18.439761+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[21512]: audit 2024-03-11T21:37:18.451349+0000 mon.c (mon.2) 1046 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[21512]: audit 2024-03-11T21:37:18.451867+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[29284]: cluster 2024-03-11T21:37:17.472511+0000 mgr.y (mgr.14550) 564 : cluster 0 pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[29284]: cluster 2024-03-11T21:37:18.439761+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[29284]: audit 2024-03-11T21:37:18.451349+0000 mon.c (mon.2) 1046 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:18.914 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:18 smithi143 bash[29284]: audit 2024-03-11T21:37:18.451867+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:18.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:19.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:18 smithi203 bash[23654]: cluster 2024-03-11T21:37:17.472511+0000 mgr.y (mgr.14550) 564 : cluster 0 pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:19.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:18 smithi203 bash[23654]: cluster 2024-03-11T21:37:18.439761+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:37:19.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:18 smithi203 bash[23654]: audit 2024-03-11T21:37:18.451349+0000 mon.c (mon.2) 1046 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:19.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:18 smithi203 bash[23654]: audit 2024-03-11T21:37:18.451867+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:19.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:19.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:19.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.317 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:19.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:19.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:19.467 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:19.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:19.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:19.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:19.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:19.858 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:19.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:19.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:20.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: cluster 2024-03-11T21:37:18.310550+0000 osd.7 (osd.7) 314 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: cluster 2024-03-11T21:37:18.310772+0000 osd.7 (osd.7) 315 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: audit 2024-03-11T21:37:18.779662+0000 mgr.y (mgr.14550) 565 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: audit 2024-03-11T21:37:19.434587+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: cluster 2024-03-11T21:37:19.450586+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: audit 2024-03-11T21:37:19.466102+0000 mon.c (mon.2) 1047 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: audit 2024-03-11T21:37:19.482591+0000 mon.c (mon.2) 1048 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[21512]: audit 2024-03-11T21:37:19.483191+0000 mon.a (mon.0) 2996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: cluster 2024-03-11T21:37:18.310550+0000 osd.7 (osd.7) 314 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: cluster 2024-03-11T21:37:18.310772+0000 osd.7 (osd.7) 315 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: audit 2024-03-11T21:37:18.779662+0000 mgr.y (mgr.14550) 565 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: audit 2024-03-11T21:37:19.434587+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: cluster 2024-03-11T21:37:19.450586+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: audit 2024-03-11T21:37:19.466102+0000 mon.c (mon.2) 1047 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: audit 2024-03-11T21:37:19.482591+0000 mon.c (mon.2) 1048 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:19 smithi143 bash[29284]: audit 2024-03-11T21:37:19.483191+0000 mon.a (mon.0) 2996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:20.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:20.247 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:20.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: cluster 2024-03-11T21:37:18.310550+0000 osd.7 (osd.7) 314 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:20.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: cluster 2024-03-11T21:37:18.310772+0000 osd.7 (osd.7) 315 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:20.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: audit 2024-03-11T21:37:18.779662+0000 mgr.y (mgr.14550) 565 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: audit 2024-03-11T21:37:19.434587+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: cluster 2024-03-11T21:37:19.450586+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:37:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: audit 2024-03-11T21:37:19.466102+0000 mon.c (mon.2) 1047 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: audit 2024-03-11T21:37:19.482591+0000 mon.c (mon.2) 1048 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:19 smithi203 bash[23654]: audit 2024-03-11T21:37:19.483191+0000 mon.a (mon.0) 2996 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:20.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:20.377 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:20.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:20.508 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:20.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:20.640 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:20.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:20.770 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:20.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:37:20.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:20.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[21512]: cluster 2024-03-11T21:37:19.473128+0000 mgr.y (mgr.14550) 566 : cluster 0 pgmap v874: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[21512]: audit 2024-03-11T21:37:20.445676+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[21512]: cluster 2024-03-11T21:37:20.454563+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[21512]: audit 2024-03-11T21:37:20.459617+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[21512]: audit 2024-03-11T21:37:20.460685+0000 mon.a (mon.0) 2999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[29284]: cluster 2024-03-11T21:37:19.473128+0000 mgr.y (mgr.14550) 566 : cluster 0 pgmap v874: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[29284]: audit 2024-03-11T21:37:20.445676+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[29284]: cluster 2024-03-11T21:37:20.454563+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[29284]: audit 2024-03-11T21:37:20.459617+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:20.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:20 smithi143 bash[29284]: audit 2024-03-11T21:37:20.460685+0000 mon.a (mon.0) 2999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:20.921 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:20 smithi203 bash[23654]: cluster 2024-03-11T21:37:19.473128+0000 mgr.y (mgr.14550) 566 : cluster 0 pgmap v874: 260 pgs: 6 creating+peering, 26 unknown, 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T21:37:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:20 smithi203 bash[23654]: audit 2024-03-11T21:37:20.445676+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:20 smithi203 bash[23654]: cluster 2024-03-11T21:37:20.454563+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:37:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:20 smithi203 bash[23654]: audit 2024-03-11T21:37:20.459617+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:21.022 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:20 smithi203 bash[23654]: audit 2024-03-11T21:37:20.460685+0000 mon.a (mon.0) 2999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:21.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:21.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:21.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:21.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:21.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:21.279 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:21.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:21.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:21.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:21.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:21.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:21.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:21.695 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:21.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:21.825 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:21.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:21.963 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:22.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:22.092 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:22.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.223 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:22.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:22.337 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:22.353 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:22.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:22.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:22.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:22.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: audit 2024-03-11T21:37:21.448386+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:22.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: cluster 2024-03-11T21:37:21.455943+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:37:22.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: audit 2024-03-11T21:37:21.457872+0000 mon.c (mon.2) 1050 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.593 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: audit 2024-03-11T21:37:21.459097+0000 mon.a (mon.0) 3002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: cluster 2024-03-11T21:37:21.473969+0000 mgr.y (mgr.14550) 567 : cluster 0 pgmap v877: 260 pgs: 6 creating+peering, 22 unknown, 232 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[29284]: audit 2024-03-11T21:37:21.752143+0000 mon.c (mon.2) 1051 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: audit 2024-03-11T21:37:21.448386+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: cluster 2024-03-11T21:37:21.455943+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: audit 2024-03-11T21:37:21.457872+0000 mon.c (mon.2) 1050 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: audit 2024-03-11T21:37:21.459097+0000 mon.a (mon.0) 3002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: cluster 2024-03-11T21:37:21.473969+0000 mgr.y (mgr.14550) 567 : cluster 0 pgmap v877: 260 pgs: 6 creating+peering, 22 unknown, 232 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-11T21:37:22.594 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:22 smithi143 bash[21512]: audit 2024-03-11T21:37:21.752143+0000 mon.c (mon.2) 1051 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:22.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: audit 2024-03-11T21:37:21.448386+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_tier","val": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:37:22.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: cluster 2024-03-11T21:37:21.455943+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:37:22.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: audit 2024-03-11T21:37:21.457872+0000 mon.c (mon.2) 1050 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: audit 2024-03-11T21:37:21.459097+0000 mon.a (mon.0) 3002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:22.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: cluster 2024-03-11T21:37:21.473969+0000 mgr.y (mgr.14550) 567 : cluster 0 pgmap v877: 260 pgs: 6 creating+peering, 22 unknown, 232 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-11T21:37:22.678 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:22 smithi203 bash[23654]: audit 2024-03-11T21:37:21.752143+0000 mon.c (mon.2) 1051 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:22.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:22.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:22.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.870 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:22.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:23.000 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:23.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:23.131 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:23.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:23.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 -- 172.21.15.143:0/3832667037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406a4e0 msgr2=0x7f063406a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 --2- 172.21.15.143:0/3832667037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406a4e0 0x7f063406a940 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f061c009990 tx=0x7f061c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 -- 172.21.15.143:0/3832667037 shutdown_connections 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 --2- 172.21.15.143:0/3832667037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 --2- 172.21.15.143:0/3832667037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406ae80 0x7f063410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 --2- 172.21.15.143:0/3832667037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406a4e0 0x7f063406a940 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 -- 172.21.15.143:0/3832667037 >> 172.21.15.143:0/3832667037 conn(0x7f0634077c80 msgr2=0x7f0634078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 -- 172.21.15.143:0/3832667037 shutdown_connections 2024-03-11T21:37:23.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 -- 172.21.15.143:0/3832667037 wait complete. 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 librados: starting msgr at 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 librados: starting objecter 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.479+0000 7f063a432700 1 Processor -- start 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- start start 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 librados: setting wanted keys 2024-03-11T21:37:23.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 librados: calling monclient init 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406a4e0 0x7f0634100db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406ae80 0x7f06341012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06341115d0 con 0x7f0634107e90 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634111450 con 0x7f063406a4e0 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0634111750 con 0x7f063406ae80 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06337fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406ae80 0x7f06341012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:53112/0 (socket says 172.21.15.143:53112) 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 -- 172.21.15.143:0/197251468 learned_addr learned my addr 172.21.15.143:0/197251468 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f0633fff700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406a4e0 0x7f0634100db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 -- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406ae80 msgr2=0x7f06341012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406ae80 0x7f06341012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 -- 172.21.15.143:0/197251468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406a4e0 msgr2=0x7f0634100db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406a4e0 0x7f0634100db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 -- 172.21.15.143:0/197251468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06341a60c0 con 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06389cf700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f062400c900 tx=0x7f062400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 10 client.?.objecter ms_handle_connect 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624007b00 con 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06341a63b0 con 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 10 client.30567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341a68c0 con 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 librados: init done 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f062400ce50 con 0x7f0634107e90 2024-03-11T21:37:23.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624018400 con 0x7f0634107e90 2024-03-11T21:37:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063404faa0 con 0x7f0634107e90 2024-03-11T21:37:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f062400f450 con 0x7f0634107e90 2024-03-11T21:37:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 0x7f0620071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:23.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f0633fff700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 0x7f0620071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(571..571 src has 87..571) v4 ==== 9696+0+0 (secure 0 0 0) 0x7f062408ea80 con 0x7f0634107e90 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 10 client.30567.objecter ms_dispatch 0x7f0634001550 osd_map(571..571 src has 87..571) v4 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 3 client.30567.objecter handle_osd_map got epochs [571,571] > 0 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.483+0000 7f06317fa700 3 client.30567.objecter handle_osd_map decoding full epoch 571 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.487+0000 7f06317fa700 20 client.30567.objecter dump_active .. 0 homeless 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.487+0000 7f06317fa700 10 client.30567.objecter ms_handle_connect 0x7f062006f820 2024-03-11T21:37:23.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.487+0000 7f0633fff700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 0x7f0620071ce0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f062c009da0 tx=0x7f062c00b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:23.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.491+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0624014030 con 0x7f0634107e90 2024-03-11T21:37:23.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.501 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:23.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:23 smithi203 bash[23654]: audit 2024-03-11T21:37:22.457496+0000 mon.a (mon.0) 3003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:23.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:23 smithi203 bash[23654]: cluster 2024-03-11T21:37:22.464186+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:37:23.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:23 smithi203 bash[23654]: audit 2024-03-11T21:37:22.469557+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.522 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:23 smithi203 bash[23654]: audit 2024-03-11T21:37:22.471685+0000 mon.a (mon.0) 3005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:23.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[21512]: audit 2024-03-11T21:37:22.457496+0000 mon.a (mon.0) 3003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[21512]: cluster 2024-03-11T21:37:22.464186+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[21512]: audit 2024-03-11T21:37:22.469557+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[21512]: audit 2024-03-11T21:37:22.471685+0000 mon.a (mon.0) 3005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[29284]: audit 2024-03-11T21:37:22.457496+0000 mon.a (mon.0) 3003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[29284]: cluster 2024-03-11T21:37:22.464186+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:37:23.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[29284]: audit 2024-03-11T21:37:22.469557+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:23 smithi143 bash[29284]: audit 2024-03-11T21:37:22.471685+0000 mon.a (mon.0) 3005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:23.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:23.719+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f06341a6ba0 con 0x7f0634107e90 2024-03-11T21:37:23.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:23.761 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:23.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:23.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:23.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:24.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:24.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:24.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:24.276 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:24.303 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:37:24.303 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:37:24.303 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:37:24.303 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:37:24.303 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:37:24.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:24.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:24.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.475+0000 7f06317fa700 1 -- 172.21.15.143:0/197251468 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v572) v1 ==== 91+0+0 (secure 0 0 0) 0x7f062405b710 con 0x7f0634107e90 2024-03-11T21:37:24.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 10 librados: watch_flush enter 2024-03-11T21:37:24.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 10 librados: watch_flush exit 2024-03-11T21:37:24.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 20 client.30567.objecter shutdown clearing up homeless session... 2024-03-11T21:37:24.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 10 client.30567.objecter successfully canceled tick 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 msgr2=0x7f0620071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 0x7f0620071ce0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f062c009da0 tx=0x7f062c00b040 comp rx=0 tx=0).stop 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 msgr2=0x7f0634101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f062400c900 tx=0x7f062400cc10 comp rx=0 tx=0).stop 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 shutdown_connections 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f062006f820 0x7f0620071ce0 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0634107e90 0x7f0634101830 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:24.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f063406a4e0 0x7f0634100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 --2- 172.21.15.143:0/197251468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063406ae80 0x7f06341012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.479+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 >> 172.21.15.143:0/197251468 conn(0x7f0634077c80 msgr2=0x7f0634105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.487+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 shutdown_connections 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.487+0000 7f063a432700 1 -- 172.21.15.143:0/197251468 wait complete. 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:24.487+0000 7f063a432700 1 librados: shutdown 2024-03-11T21:37:24.462 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:37:24.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:24.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[21512]: audit 2024-03-11T21:37:23.459581+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[21512]: cluster 2024-03-11T21:37:23.462952+0000 mon.a (mon.0) 3007 : cluster 0 osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[21512]: cluster 2024-03-11T21:37:23.474540+0000 mgr.y (mgr.14550) 568 : cluster 0 pgmap v880: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[21512]: audit 2024-03-11T21:37:23.723914+0000 mon.a (mon.0) 3008 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[29284]: audit 2024-03-11T21:37:23.459581+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:24.523 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[29284]: cluster 2024-03-11T21:37:23.462952+0000 mon.a (mon.0) 3007 : cluster 0 osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:37:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[29284]: cluster 2024-03-11T21:37:23.474540+0000 mgr.y (mgr.14550) 568 : cluster 0 pgmap v880: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:37:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:24 smithi143 bash[29284]: audit 2024-03-11T21:37:23.723914+0000 mon.a (mon.0) 3008 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:24.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:24.652 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:24.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:24 smithi203 bash[23654]: audit 2024-03-11T21:37:23.459581+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:24 smithi203 bash[23654]: cluster 2024-03-11T21:37:23.462952+0000 mon.a (mon.0) 3007 : cluster 0 osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:37:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:24 smithi203 bash[23654]: cluster 2024-03-11T21:37:23.474540+0000 mgr.y (mgr.14550) 568 : cluster 0 pgmap v880: 260 pgs: 6 creating+peering, 254 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:37:24.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:24 smithi203 bash[23654]: audit 2024-03-11T21:37:23.723914+0000 mon.a (mon.0) 3008 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:24.783 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:24.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.914 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:25.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:37:25.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:25.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:25.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:25.173 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:25.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:25.298 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:25.401 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:25.429 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:25.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:25.542 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:25 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.468654+0000 mon.a (mon.0) 3009 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:25.542 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:25 smithi203 bash[23654]: audit 2024-03-11T21:37:24.476771+0000 mon.a (mon.0) 3010 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:25.542 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:25 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.482747+0000 mon.a (mon.0) 3011 : cluster 0 osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:37:25.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:25.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:25.672 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.468654+0000 mon.a (mon.0) 3009 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:25.672 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[21512]: audit 2024-03-11T21:37:24.476771+0000 mon.a (mon.0) 3010 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:25.672 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.482747+0000 mon.a (mon.0) 3011 : cluster 0 osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:37:25.672 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.468654+0000 mon.a (mon.0) 3009 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:25.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[29284]: audit 2024-03-11T21:37:24.476771+0000 mon.a (mon.0) 3010 : audit 1 from='client.? 172.21.15.143:0/197251468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:25.673 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:25 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.482747+0000 mon.a (mon.0) 3011 : cluster 0 osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:37:25.686 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:25.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:25.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:25.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:25.948 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:26.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:26.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.210 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:26.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:26.340 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:26.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:26.470 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:26.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:26.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:26.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:26.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:26.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:26.856 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:26.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:26.966 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.649444+0000 osd.5 (osd.5) 337 : cluster 0 2.12 scrub starts 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.653977+0000 osd.5 (osd.5) 338 : cluster 0 2.12 scrub ok 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.962396+0000 osd.4 (osd.4) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:24.962601+0000 osd.4 (osd.4) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.341012+0000 osd.3 (osd.3) 341 : cluster 0 6.c scrub starts 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.342907+0000 osd.3 (osd.3) 342 : cluster 0 6.c scrub ok 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.475268+0000 mgr.y (mgr.14550) 569 : cluster 0 pgmap v882: 260 pgs: 260 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.482731+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: audit 2024-03-11T21:37:25.549057+0000 mon.c (mon.2) 1053 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: audit 2024-03-11T21:37:25.549803+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: audit 2024-03-11T21:37:25.550705+0000 mon.c (mon.2) 1054 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: audit 2024-03-11T21:37:25.551149+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:26.967 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.787919+0000 osd.2 (osd.2) 287 : cluster 0 39.1f scrub starts 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:25.791042+0000 osd.2 (osd.2) 288 : cluster 0 39.1f scrub ok 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.119660+0000 osd.1 (osd.1) 129 : cluster 0 2.6 scrub starts 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.122479+0000 osd.1 (osd.1) 130 : cluster 0 2.6 scrub ok 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.418554+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.649444+0000 osd.5 (osd.5) 337 : cluster 0 2.12 scrub starts 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.653977+0000 osd.5 (osd.5) 338 : cluster 0 2.12 scrub ok 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.962396+0000 osd.4 (osd.4) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:24.962601+0000 osd.4 (osd.4) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.341012+0000 osd.3 (osd.3) 341 : cluster 0 6.c scrub starts 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.342907+0000 osd.3 (osd.3) 342 : cluster 0 6.c scrub ok 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.475268+0000 mgr.y (mgr.14550) 569 : cluster 0 pgmap v882: 260 pgs: 260 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.482731+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:37:26.968 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: audit 2024-03-11T21:37:25.549057+0000 mon.c (mon.2) 1053 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: audit 2024-03-11T21:37:25.549803+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: audit 2024-03-11T21:37:25.550705+0000 mon.c (mon.2) 1054 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: audit 2024-03-11T21:37:25.551149+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.787919+0000 osd.2 (osd.2) 287 : cluster 0 39.1f scrub starts 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:25.791042+0000 osd.2 (osd.2) 288 : cluster 0 39.1f scrub ok 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.119660+0000 osd.1 (osd.1) 129 : cluster 0 2.6 scrub starts 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.122479+0000 osd.1 (osd.1) 130 : cluster 0 2.6 scrub ok 2024-03-11T21:37:26.969 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:26 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.418554+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:37:26.977 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:27.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:27.103 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/1502637345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798102880 msgr2=0x7f879810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- 172.21.15.143:0/1502637345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798102880 0x7f879810ec70 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f879000b0a0 tx=0x7f879000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/1502637345 shutdown_connections 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- 172.21.15.143:0/1502637345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798102880 0x7f879810ec70 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- 172.21.15.143:0/1502637345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798107f30 0x7f8798108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- 172.21.15.143:0/1502637345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 0x7f8798102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/1502637345 >> 172.21.15.143:0/1502637345 conn(0x7f8798078eb0 msgr2=0x7f87980792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/1502637345 shutdown_connections 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/1502637345 wait complete. 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 librados: starting msgr at 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 librados: starting objecter 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 Processor -- start 2024-03-11T21:37:27.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- start start 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 librados: setting wanted keys 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 librados: calling monclient init 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 0x7f87980740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798102880 0x7f8798074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87981116a0 con 0x7f8798101ec0 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8798111520 con 0x7f8798102880 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8798111820 con 0x7f8798107f30 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40880/0 (socket says 172.21.15.143:40880) 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 -- 172.21.15.143:0/3077317645 learned_addr learned my addr 172.21.15.143:0/3077317645 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f8796ffd700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 0x7f87980740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87967fc700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798102880 0x7f8798074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:27.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 -- 172.21.15.143:0/3077317645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798102880 msgr2=0x7f8798074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798102880 0x7f8798074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 -- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 msgr2=0x7f87980740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 0x7f87980740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 -- 172.21.15.143:0/3077317645 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87981a6140 con 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f87977fe700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f879000b070 tx=0x7f8790007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879c869700 10 client.?.objecter ms_handle_connect 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879c869700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790021020 con 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8790004c50 con 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87981a63d0 con 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87900052b0 con 0x7f8798107f30 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 10 client.41774.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:27.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87981a6880 con 0x7f8798107f30 2024-03-11T21:37:27.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.130+0000 7f879d86b700 1 librados: init done 2024-03-11T21:37:27.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879804faa0 con 0x7f8798107f30 2024-03-11T21:37:27.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8790024020 con 0x7f8798107f30 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 0x7f8780071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(574..574 src has 87..574) v4 ==== 9244+0+0 (secure 0 0 0) 0x7f879000f020 con 0x7f8798107f30 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 10 client.41774.objecter ms_dispatch 0x7f8798001550 osd_map(574..574 src has 87..574) v4 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f8796ffd700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 0x7f8780071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 3 client.41774.objecter handle_osd_map got epochs [574,574] > 0 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 3 client.41774.objecter handle_osd_map decoding full epoch 574 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 20 client.41774.objecter dump_active .. 0 homeless 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f879c869700 10 client.41774.objecter ms_handle_connect 0x7f878006f820 2024-03-11T21:37:27.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.134+0000 7f8796ffd700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 0x7f8780071ce0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7f8788009a10 tx=0x7f8788009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:27.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.138+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f879005c920 con 0x7f8798107f30 2024-03-11T21:37:27.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.649444+0000 osd.5 (osd.5) 337 : cluster 0 2.12 scrub starts 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.653977+0000 osd.5 (osd.5) 338 : cluster 0 2.12 scrub ok 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.962396+0000 osd.4 (osd.4) 303 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:24.962601+0000 osd.4 (osd.4) 304 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.341012+0000 osd.3 (osd.3) 341 : cluster 0 6.c scrub starts 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.342907+0000 osd.3 (osd.3) 342 : cluster 0 6.c scrub ok 2024-03-11T21:37:27.218 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.475268+0000 mgr.y (mgr.14550) 569 : cluster 0 pgmap v882: 260 pgs: 260 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.482731+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: audit 2024-03-11T21:37:25.549057+0000 mon.c (mon.2) 1053 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: audit 2024-03-11T21:37:25.549803+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: audit 2024-03-11T21:37:25.550705+0000 mon.c (mon.2) 1054 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: audit 2024-03-11T21:37:25.551149+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-85"}]: dispatch 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.787919+0000 osd.2 (osd.2) 287 : cluster 0 39.1f scrub starts 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:25.791042+0000 osd.2 (osd.2) 288 : cluster 0 39.1f scrub ok 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.119660+0000 osd.1 (osd.1) 129 : cluster 0 2.6 scrub starts 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.122479+0000 osd.1 (osd.1) 130 : cluster 0 2.6 scrub ok 2024-03-11T21:37:27.219 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:26 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.418554+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:37:27.228 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:27.234 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:27.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.346+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8798107170 con 0x7f878006f820 2024-03-11T21:37:27.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:27.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.378+0000 7f879c869700 1 -- 172.21.15.143:0/3077317645 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+589452 (secure 0 0 0) 0x7f8798107170 con 0x7f878006f820 2024-03-11T21:37:27.359 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:37:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 10 librados: watch_flush enter 2024-03-11T21:37:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 10 librados: watch_flush exit 2024-03-11T21:37:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 20 client.41774.objecter shutdown clearing up homeless session... 2024-03-11T21:37:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 10 client.41774.objecter successfully canceled tick 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 msgr2=0x7f8780071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 0x7f8780071ce0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7f8788009a10 tx=0x7f8788009190 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 msgr2=0x7f8798074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f879000b070 tx=0x7f8790007a20 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 shutdown_connections 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f878006f820 0x7f8780071ce0 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798101ec0 0x7f87980740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8798107f30 0x7f8798074b60 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 --2- 172.21.15.143:0/3077317645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8798102880 0x7f8798074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 >> 172.21.15.143:0/3077317645 conn(0x7f8798078eb0 msgr2=0x7f8798105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 shutdown_connections 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 -- 172.21.15.143:0/3077317645 wait complete. 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:27.382+0000 7f879d86b700 1 librados: shutdown 2024-03-11T21:37:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:37:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.366 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:27.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:27.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:27.493 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:27.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:27.624 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:27.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:27.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:27.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:27.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:27.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.209423+0000 osd.4 (osd.4) 305 : cluster 0 39.b scrub starts 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.212269+0000 osd.4 (osd.4) 306 : cluster 0 39.b scrub ok 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.666248+0000 osd.5 (osd.5) 339 : cluster 0 118.13 scrub starts 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:26.671036+0000 osd.5 (osd.5) 340 : cluster 0 118.13 scrub ok 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.110520+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:37:27.999 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.114943+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.200712+0000 osd.4 (osd.4) 307 : cluster 0 5.1c scrub starts 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.202973+0000 osd.4 (osd.4) 308 : cluster 0 5.1c scrub ok 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.416577+0000 mon.a (mon.0) 3016 : cluster 0 osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: audit 2024-03-11T21:37:27.419104+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: audit 2024-03-11T21:37:27.421022+0000 mon.a (mon.0) 3017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.739171+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.741675+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:37:28.000 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.209423+0000 osd.4 (osd.4) 305 : cluster 0 39.b scrub starts 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.212269+0000 osd.4 (osd.4) 306 : cluster 0 39.b scrub ok 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.666248+0000 osd.5 (osd.5) 339 : cluster 0 118.13 scrub starts 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:26.671036+0000 osd.5 (osd.5) 340 : cluster 0 118.13 scrub ok 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.110520+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.114943+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.200712+0000 osd.4 (osd.4) 307 : cluster 0 5.1c scrub starts 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.202973+0000 osd.4 (osd.4) 308 : cluster 0 5.1c scrub ok 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.416577+0000 mon.a (mon.0) 3016 : cluster 0 osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: audit 2024-03-11T21:37:27.419104+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: audit 2024-03-11T21:37:27.421022+0000 mon.a (mon.0) 3017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.739171+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:37:28.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:27 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.741675+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:37:28.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:28.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.209423+0000 osd.4 (osd.4) 305 : cluster 0 39.b scrub starts 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.212269+0000 osd.4 (osd.4) 306 : cluster 0 39.b scrub ok 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.666248+0000 osd.5 (osd.5) 339 : cluster 0 118.13 scrub starts 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:26.671036+0000 osd.5 (osd.5) 340 : cluster 0 118.13 scrub ok 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.110520+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:37:28.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.114943+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.200712+0000 osd.4 (osd.4) 307 : cluster 0 5.1c scrub starts 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.202973+0000 osd.4 (osd.4) 308 : cluster 0 5.1c scrub ok 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.416577+0000 mon.a (mon.0) 3016 : cluster 0 osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: audit 2024-03-11T21:37:27.419104+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: audit 2024-03-11T21:37:27.421022+0000 mon.a (mon.0) 3017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.739171+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:37:28.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:27 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.741675+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:37:28.145 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:28.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:28.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.413 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (17999 ms) 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22981 ms) 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3019 ms) 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3024 ms) 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-11T21:37:28.430 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3053 ms) 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (2932 ms) 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4022 ms) 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38206 ms) 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16806 ms) 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.431 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3976 ms) 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3037 ms) 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25189 ms) 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15109 ms) 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5080 ms) 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3623 ms) 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrubs... 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25010 ms) 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10081 ms) 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8919 ms) 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8995 ms) 2024-03-11T21:37:28.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-11T21:37:28.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:28.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:28.675 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:28.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:37:28.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.813 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:28.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: audit 2024-03-11T21:37:27.351704+0000 mgr.y (mgr.14550) 570 : audit 0 from='client.41774 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.475856+0000 mgr.y (mgr.14550) 571 : cluster 0 pgmap v886: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.938888+0000 osd.0 (osd.0) 324 : cluster 0 3.12 scrub starts 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:27.942413+0000 osd.0 (osd.0) 325 : cluster 0 3.12 scrub ok 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.072398+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:37:28.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.074082+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:37:28.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.413338+0000 mon.a (mon.0) 3018 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:28.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: audit 2024-03-11T21:37:28.425768+0000 mon.a (mon.0) 3019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:28.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.432901+0000 mon.a (mon.0) 3020 : cluster 0 osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:37:28.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:28 smithi203 bash[23654]: audit 2024-03-11T21:37:28.439172+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:29.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:29.076 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:29.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:29.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: audit 2024-03-11T21:37:27.351704+0000 mgr.y (mgr.14550) 570 : audit 0 from='client.41774 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:29.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.475856+0000 mgr.y (mgr.14550) 571 : cluster 0 pgmap v886: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:29.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.938888+0000 osd.0 (osd.0) 324 : cluster 0 3.12 scrub starts 2024-03-11T21:37:29.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:27.942413+0000 osd.0 (osd.0) 325 : cluster 0 3.12 scrub ok 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.072398+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.074082+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.413338+0000 mon.a (mon.0) 3018 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: audit 2024-03-11T21:37:28.425768+0000 mon.a (mon.0) 3019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.432901+0000 mon.a (mon.0) 3020 : cluster 0 osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[21512]: audit 2024-03-11T21:37:28.439172+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: audit 2024-03-11T21:37:27.351704+0000 mgr.y (mgr.14550) 570 : audit 0 from='client.41774 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.475856+0000 mgr.y (mgr.14550) 571 : cluster 0 pgmap v886: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.938888+0000 osd.0 (osd.0) 324 : cluster 0 3.12 scrub starts 2024-03-11T21:37:29.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:27.942413+0000 osd.0 (osd.0) 325 : cluster 0 3.12 scrub ok 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.072398+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.074082+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.413338+0000 mon.a (mon.0) 3018 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: audit 2024-03-11T21:37:28.425768+0000 mon.a (mon.0) 3019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.432901+0000 mon.a (mon.0) 3020 : cluster 0 osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:37:29.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:28 smithi143 bash[29284]: audit 2024-03-11T21:37:28.439172+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:29.207 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:29.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.338 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:29.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:29.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:29.593 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:29.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:29.724 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:29.756 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-03-11T21:37:26.414070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8801731,"num_objects":382,"num_object_clones":10,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":2422,"num_read_kb":2123,"num_write":1654,"num_write_kb":15250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3060,"ondisk_log_size":3060,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":761,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":629608,"kb_used_data":101504,"kb_used_omap":38,"kb_used_meta":527833,"kb_avail":749331608,"statfs":{"total":767960285184,"available":767315566592,"internally_reserved":0,"allocated":103940096,"data_stored":73309585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":540501429},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.954678"},"pg_stats":[{"pgid":"118.9","version":"564'26","reported_seq":1064,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.466537+0000","last_change":"2024-03-11T21:36:29.868300+0000","last_active":"2024-03-11T21:37:22.466537+0000","last_peered":"2024-03-11T21:37:22.466537+0000","last_clean":"2024-03-11T21:37:22.466537+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:37:22.466537+0000","last_undegraded":"2024-03-11T21:37:22.466537+0000","last_fullsized":"2024-03-11T21:37:22.466537+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'18","last_scrub_stamp":"2024-03-11T21:36:29.868174+0000","last_deep_scrub":"482'18","last_deep_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_clean_scrub_stamp":"2024-03-11T21:36:29.868174+0000","objects_scrubbed":1,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.069584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010370220000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":17,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"555'29","reported_seq":1037,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545440+0000","last_change":"2024-03-11T21:35:26.450518+0000","last_active":"2024-03-11T21:37:22.545440+0000","last_peered":"2024-03-11T21:37:22.545440+0000","last_clean":"2024-03-11T21:37:22.545440+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:37:22.545440+0000","last_undegraded":"2024-03-11T21:37:22.545440+0000","last_fullsized":"2024-03-11T21:37:22.545440+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'24","last_scrub_stamp":"2024-03-11T21:34:54.761329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.761329+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041612760000000002,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"564'45","reported_seq":1051,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455146+0000","last_change":"2024-03-11T21:35:26.453000+0000","last_active":"2024-03-11T21:37:22.455146+0000","last_peered":"2024-03-11T21:37:22.455146+0000","last_clean":"2024-03-11T21:37:22.455146+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:37:22.455146+0000","last_undegraded":"2024-03-11T21:37:22.455146+0000","last_fullsized":"2024-03-11T21:37:22.455146+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'40","last_scrub_stamp":"2024-03-11T21:34:55.080483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.080483+0000","objects_scrubbed":4,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014521345999999999,"stat_sum":{"num_bytes":2337,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"564'75","reported_seq":1102,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539132+0000","last_change":"2024-03-11T21:36:06.267240+0000","last_active":"2024-03-11T21:37:22.539132+0000","last_peered":"2024-03-11T21:37:22.539132+0000","last_clean":"2024-03-11T21:37:22.539132+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:37:22.539132+0000","last_undegraded":"2024-03-11T21:37:22.539132+0000","last_fullsized":"2024-03-11T21:37:22.539132+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'55","last_scrub_stamp":"2024-03-11T21:36:06.267048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.267048+0000","objects_scrubbed":7,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.312975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014549960000000001,"stat_sum":{"num_bytes":654,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":32,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"564'68","reported_seq":1098,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468370+0000","last_change":"2024-03-11T21:36:45.778540+0000","last_active":"2024-03-11T21:37:22.468370+0000","last_peered":"2024-03-11T21:37:22.468370+0000","last_clean":"2024-03-11T21:37:22.468370+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:37:22.468370+0000","last_undegraded":"2024-03-11T21:37:22.468370+0000","last_fullsized":"2024-03-11T21:37:22.468370+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'56","last_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_deep_scrub":"541'56","last_deep_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.778396+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:47.275324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00082181799999999998,"stat_sum":{"num_bytes":1378,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6,"num_write":32,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"485'36","reported_seq":1045,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541370+0000","last_change":"2024-03-11T21:35:26.433372+0000","last_active":"2024-03-11T21:37:22.541370+0000","last_peered":"2024-03-11T21:37:22.541370+0000","last_clean":"2024-03-11T21:37:22.541370+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:37:22.541370+0000","last_undegraded":"2024-03-11T21:37:22.541370+0000","last_fullsized":"2024-03-11T21:37:22.541370+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'36","last_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_deep_scrub":"485'36","last_deep_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.199420+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023323459999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"444'184","reported_seq":1172,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541959+0000","last_change":"2024-03-11T21:36:04.302637+0000","last_active":"2024-03-11T21:37:22.541959+0000","last_peered":"2024-03-11T21:37:22.541959+0000","last_clean":"2024-03-11T21:37:22.541959+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:37:22.541959+0000","last_undegraded":"2024-03-11T21:37:22.541959+0000","last_fullsized":"2024-03-11T21:37:22.541959+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'184","last_scrub_stamp":"2024-03-11T21:36:04.302373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.302373+0000","objects_scrubbed":14,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.026005+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001783946,"stat_sum":{"num_bytes":125,"num_objects":14,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":56,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"555'111","reported_seq":1110,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547156+0000","last_change":"2024-03-11T21:36:17.771393+0000","last_active":"2024-03-11T21:37:22.547156+0000","last_peered":"2024-03-11T21:37:22.547156+0000","last_clean":"2024-03-11T21:37:22.547156+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:37:22.547156+0000","last_undegraded":"2024-03-11T21:37:22.547156+0000","last_fullsized":"2024-03-11T21:37:22.547156+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'107","last_scrub_stamp":"2024-03-11T21:36:17.771145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.771145+0000","objects_scrubbed":7,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0030091919999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":38,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"564'21","reported_seq":1038,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468526+0000","last_change":"2024-03-11T21:36:11.356246+0000","last_active":"2024-03-11T21:37:23.468526+0000","last_peered":"2024-03-11T21:37:23.468526+0000","last_clean":"2024-03-11T21:37:23.468526+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:37:23.468526+0000","last_undegraded":"2024-03-11T21:37:23.468526+0000","last_fullsized":"2024-03-11T21:37:23.468526+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:36:11.356102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.356102+0000","objects_scrubbed":2,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001800236,"stat_sum":{"num_bytes":4001234,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":11,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"533'63","reported_seq":1080,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.467762+0000","last_change":"2024-03-11T21:36:13.359143+0000","last_active":"2024-03-11T21:37:23.467762+0000","last_peered":"2024-03-11T21:37:23.467762+0000","last_clean":"2024-03-11T21:37:23.467762+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:37:23.467762+0000","last_undegraded":"2024-03-11T21:37:23.467762+0000","last_fullsized":"2024-03-11T21:37:23.467762+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:36:13.358869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.358869+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.972060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028501400000000001,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":20,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"555'144","reported_seq":1125,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455271+0000","last_change":"2024-03-11T21:35:26.453765+0000","last_active":"2024-03-11T21:37:22.455271+0000","last_peered":"2024-03-11T21:37:22.455271+0000","last_clean":"2024-03-11T21:37:22.455271+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:37:22.455271+0000","last_undegraded":"2024-03-11T21:37:22.455271+0000","last_fullsized":"2024-03-11T21:37:22.455271+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687167+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015353044,"stat_sum":{"num_bytes":125,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":17,"num_read_kb":6,"num_write":53,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":1052,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540210+0000","last_change":"2024-03-11T21:36:02.266701+0000","last_active":"2024-03-11T21:37:22.540210+0000","last_peered":"2024-03-11T21:37:22.540210+0000","last_clean":"2024-03-11T21:37:22.540210+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:37:22.540210+0000","last_undegraded":"2024-03-11T21:37:22.540210+0000","last_fullsized":"2024-03-11T21:37:22.540210+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'22","last_scrub_stamp":"2024-03-11T21:36:02.266514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.266514+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001443038,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"564'6","reported_seq":1036,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468386+0000","last_change":"2024-03-11T21:36:17.826472+0000","last_active":"2024-03-11T21:37:22.468386+0000","last_peered":"2024-03-11T21:37:22.468386+0000","last_clean":"2024-03-11T21:37:22.468386+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:37:22.468386+0000","last_undegraded":"2024-03-11T21:37:22.468386+0000","last_fullsized":"2024-03-11T21:37:22.468386+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'3","last_scrub_stamp":"2024-03-11T21:36:17.826340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.826340+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.080628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079279899999999996,"stat_sum":{"num_bytes":529,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":1051,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470807+0000","last_change":"2024-03-11T21:36:18.317604+0000","last_active":"2024-03-11T21:37:23.470807+0000","last_peered":"2024-03-11T21:37:23.470807+0000","last_clean":"2024-03-11T21:37:23.470807+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:37:23.470807+0000","last_undegraded":"2024-03-11T21:37:23.470807+0000","last_fullsized":"2024-03-11T21:37:23.470807+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:36:18.317358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.317358+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.707208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022017640000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"506'46","reported_seq":1047,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.556934+0000","last_change":"2024-03-11T21:36:21.706930+0000","last_active":"2024-03-11T21:37:22.556934+0000","last_peered":"2024-03-11T21:37:22.556934+0000","last_clean":"2024-03-11T21:37:22.556934+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:37:22.556934+0000","last_undegraded":"2024-03-11T21:37:22.556934+0000","last_fullsized":"2024-03-11T21:37:22.556934+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'46","last_scrub_stamp":"2024-03-11T21:36:21.706814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.706814+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.386403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028979269999999998,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"340'75","reported_seq":1083,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557064+0000","last_change":"2024-03-11T21:36:28.741175+0000","last_active":"2024-03-11T21:37:22.557064+0000","last_peered":"2024-03-11T21:37:22.557064+0000","last_clean":"2024-03-11T21:37:22.557064+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:37:22.557064+0000","last_undegraded":"2024-03-11T21:37:22.557064+0000","last_fullsized":"2024-03-11T21:37:22.557064+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:36:28.741022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.741022+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.020272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000857495,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"564'18","reported_seq":1032,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468486+0000","last_change":"2024-03-11T21:36:08.294434+0000","last_active":"2024-03-11T21:37:23.468486+0000","last_peered":"2024-03-11T21:37:23.468486+0000","last_clean":"2024-03-11T21:37:23.468486+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:37:23.468486+0000","last_undegraded":"2024-03-11T21:37:23.468486+0000","last_fullsized":"2024-03-11T21:37:23.468486+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:36:08.294289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.294289+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.951123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00159348,"stat_sum":{"num_bytes":1226,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"412'108","reported_seq":1103,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.463118+0000","last_change":"2024-03-11T21:35:26.437723+0000","last_active":"2024-03-11T21:37:23.463118+0000","last_peered":"2024-03-11T21:37:23.463118+0000","last_clean":"2024-03-11T21:37:23.463118+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:37:23.463118+0000","last_undegraded":"2024-03-11T21:37:23.463118+0000","last_fullsized":"2024-03-11T21:37:23.463118+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'108","last_scrub_stamp":"2024-03-11T21:35:09.044564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.044564+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032700849999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"564'20","reported_seq":925,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.481443+0000","last_change":"2024-03-11T21:36:19.125306+0000","last_active":"2024-03-11T21:37:24.481443+0000","last_peered":"2024-03-11T21:37:24.481443+0000","last_clean":"2024-03-11T21:37:24.481443+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:37:24.481443+0000","last_undegraded":"2024-03-11T21:37:24.481443+0000","last_fullsized":"2024-03-11T21:37:24.481443+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:36:19.125151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.125151+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:48.732140+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00224048,"stat_sum":{"num_bytes":711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"247'56","reported_seq":1067,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.469126+0000","last_change":"2024-03-11T21:36:17.331041+0000","last_active":"2024-03-11T21:37:23.469126+0000","last_peered":"2024-03-11T21:37:23.469126+0000","last_clean":"2024-03-11T21:37:23.469126+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:37:23.469126+0000","last_undegraded":"2024-03-11T21:37:23.469126+0000","last_fullsized":"2024-03-11T21:37:23.469126+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:36:17.330735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.330735+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072179299999999998,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"515'102","reported_seq":1119,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542751+0000","last_change":"2024-03-11T21:36:18.451092+0000","last_active":"2024-03-11T21:37:22.542751+0000","last_peered":"2024-03-11T21:37:22.542751+0000","last_clean":"2024-03-11T21:37:22.542751+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:37:22.542751+0000","last_undegraded":"2024-03-11T21:37:22.542751+0000","last_fullsized":"2024-03-11T21:37:22.542751+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'102","last_scrub_stamp":"2024-03-11T21:36:18.450840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.450840+0000","objects_scrubbed":7,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643729+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0026391510000000002,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"554'75","reported_seq":1096,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455265+0000","last_change":"2024-03-11T21:35:26.454329+0000","last_active":"2024-03-11T21:37:22.455265+0000","last_peered":"2024-03-11T21:37:22.455265+0000","last_clean":"2024-03-11T21:37:22.455265+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:37:22.455265+0000","last_undegraded":"2024-03-11T21:37:22.455265+0000","last_fullsized":"2024-03-11T21:37:22.455265+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'69","last_scrub_stamp":"2024-03-11T21:34:58.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.041773+0000","objects_scrubbed":9,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.011639752999999999,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":28,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"555'43","reported_seq":948,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485170+0000","last_change":"2024-03-11T21:36:16.054966+0000","last_active":"2024-03-11T21:37:24.485170+0000","last_peered":"2024-03-11T21:37:24.485170+0000","last_clean":"2024-03-11T21:37:24.485170+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:37:24.485170+0000","last_undegraded":"2024-03-11T21:37:24.485170+0000","last_fullsized":"2024-03-11T21:37:24.485170+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'37","last_scrub_stamp":"2024-03-11T21:36:16.054853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.054853+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.306489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012079040000000001,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":21,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"554'219","reported_seq":1172,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456701+0000","last_change":"2024-03-11T21:35:26.455346+0000","last_active":"2024-03-11T21:37:22.456701+0000","last_peered":"2024-03-11T21:37:22.456701+0000","last_clean":"2024-03-11T21:37:22.456701+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:37:22.456701+0000","last_undegraded":"2024-03-11T21:37:22.456701+0000","last_fullsized":"2024-03-11T21:37:22.456701+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'204","last_scrub_stamp":"2024-03-11T21:31:51.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.388400+0000","objects_scrubbed":12,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0095356580000000007,"stat_sum":{"num_bytes":151,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":73,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"412'35","reported_seq":1031,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455152+0000","last_change":"2024-03-11T21:35:26.453569+0000","last_active":"2024-03-11T21:37:22.455152+0000","last_peered":"2024-03-11T21:37:22.455152+0000","last_clean":"2024-03-11T21:37:22.455152+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:37:22.455152+0000","last_undegraded":"2024-03-11T21:37:22.455152+0000","last_fullsized":"2024-03-11T21:37:22.455152+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'35","last_scrub_stamp":"2024-03-11T21:35:08.144914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.144914+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016015273999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"464'17","reported_seq":1083,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545394+0000","last_change":"2024-03-11T21:36:30.037490+0000","last_active":"2024-03-11T21:37:22.545394+0000","last_peered":"2024-03-11T21:37:22.545394+0000","last_clean":"2024-03-11T21:37:22.545394+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:37:22.545394+0000","last_undegraded":"2024-03-11T21:37:22.545394+0000","last_fullsized":"2024-03-11T21:37:22.545394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'17","last_scrub_stamp":"2024-03-11T21:36:30.037348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.037348+0000","objects_scrubbed":4,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.250514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0045876709999999998,"stat_sum":{"num_bytes":4194330,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":1080,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546677+0000","last_change":"2024-03-11T21:35:26.451133+0000","last_active":"2024-03-11T21:37:22.546677+0000","last_peered":"2024-03-11T21:37:22.546677+0000","last_clean":"2024-03-11T21:37:22.546677+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:37:22.546677+0000","last_undegraded":"2024-03-11T21:37:22.546677+0000","last_fullsized":"2024-03-11T21:37:22.546677+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'87","last_scrub_stamp":"2024-03-11T21:34:32.811720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.811720+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032466769999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":1063,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454959+0000","last_change":"2024-03-11T21:35:26.453300+0000","last_active":"2024-03-11T21:37:22.454959+0000","last_peered":"2024-03-11T21:37:22.454959+0000","last_clean":"2024-03-11T21:37:22.454959+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:37:22.454959+0000","last_undegraded":"2024-03-11T21:37:22.454959+0000","last_fullsized":"2024-03-11T21:37:22.454959+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'65","last_scrub_stamp":"2024-03-11T21:35:07.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.139123+0000","objects_scrubbed":8,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014996763,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"564'75","reported_seq":1098,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468218+0000","last_change":"2024-03-11T21:36:44.814463+0000","last_active":"2024-03-11T21:37:22.468218+0000","last_peered":"2024-03-11T21:37:22.468218+0000","last_clean":"2024-03-11T21:37:22.468218+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:37:22.468218+0000","last_undegraded":"2024-03-11T21:37:22.468218+0000","last_fullsized":"2024-03-11T21:37:22.468218+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:36:44.814352+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.814352+0000","objects_scrubbed":4,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.466414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00098680300000000007,"stat_sum":{"num_bytes":1586,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"495'67","reported_seq":1083,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541140+0000","last_change":"2024-03-11T21:36:11.203520+0000","last_active":"2024-03-11T21:37:22.541140+0000","last_peered":"2024-03-11T21:37:22.541140+0000","last_clean":"2024-03-11T21:37:22.541140+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:37:22.541140+0000","last_undegraded":"2024-03-11T21:37:22.541140+0000","last_fullsized":"2024-03-11T21:37:22.541140+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'67","last_scrub_stamp":"2024-03-11T21:36:11.203406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.203406+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:39.788908+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.0043048899999998,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"467'56","reported_seq":1076,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.469751+0000","last_change":"2024-03-11T21:36:06.936868+0000","last_active":"2024-03-11T21:37:22.469751+0000","last_peered":"2024-03-11T21:37:22.469751+0000","last_clean":"2024-03-11T21:37:22.469751+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:37:22.469751+0000","last_undegraded":"2024-03-11T21:37:22.469751+0000","last_fullsized":"2024-03-11T21:37:22.469751+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'56","last_scrub_stamp":"2024-03-11T21:36:06.936622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.936622+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.189358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056505100000000001,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"555'12","reported_seq":1023,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456748+0000","last_change":"2024-03-11T21:35:26.455653+0000","last_active":"2024-03-11T21:37:22.456748+0000","last_peered":"2024-03-11T21:37:22.456748+0000","last_clean":"2024-03-11T21:37:22.456748+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:37:22.456748+0000","last_undegraded":"2024-03-11T21:37:22.456748+0000","last_fullsized":"2024-03-11T21:37:22.456748+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'6","last_scrub_stamp":"2024-03-11T21:34:50.112525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.112525+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001177331,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546648+0000","last_change":"2024-03-11T21:36:09.680554+0000","last_active":"2024-03-11T21:37:22.546648+0000","last_peered":"2024-03-11T21:37:22.546648+0000","last_clean":"2024-03-11T21:37:22.546648+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:37:22.546648+0000","last_undegraded":"2024-03-11T21:37:22.546648+0000","last_fullsized":"2024-03-11T21:37:22.546648+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.680346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.680346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.345201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":887,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539763+0000","last_change":"2024-03-11T21:35:05.450716+0000","last_active":"2024-03-11T21:37:22.539763+0000","last_peered":"2024-03-11T21:37:22.539763+0000","last_clean":"2024-03-11T21:37:22.539763+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:37:22.539763+0000","last_undegraded":"2024-03-11T21:37:22.539763+0000","last_fullsized":"2024-03-11T21:37:22.539763+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.450323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.450323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545677+0000","last_change":"2024-03-11T21:36:34.020742+0000","last_active":"2024-03-11T21:37:22.545677+0000","last_peered":"2024-03-11T21:37:22.545677+0000","last_clean":"2024-03-11T21:37:22.545677+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:37:22.545677+0000","last_undegraded":"2024-03-11T21:37:22.545677+0000","last_fullsized":"2024-03-11T21:37:22.545677+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.020607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.020607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.450801+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549107+0000","last_change":"2024-03-11T21:36:14.628566+0000","last_active":"2024-03-11T21:37:22.549107+0000","last_peered":"2024-03-11T21:37:22.549107+0000","last_clean":"2024-03-11T21:37:22.549107+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:37:22.549107+0000","last_undegraded":"2024-03-11T21:37:22.549107+0000","last_fullsized":"2024-03-11T21:37:22.549107+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.259727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":792,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482084+0000","last_change":"2024-03-11T21:36:43.079499+0000","last_active":"2024-03-11T21:37:24.482084+0000","last_peered":"2024-03-11T21:37:24.482084+0000","last_clean":"2024-03-11T21:37:24.482084+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:37:24.482084+0000","last_undegraded":"2024-03-11T21:37:24.482084+0000","last_fullsized":"2024-03-11T21:37:24.482084+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.079413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.079413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:47.032407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":887,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.543213+0000","last_change":"2024-03-11T21:35:06.473871+0000","last_active":"2024-03-11T21:37:22.543213+0000","last_peered":"2024-03-11T21:37:22.543213+0000","last_clean":"2024-03-11T21:37:22.543213+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:37:22.543213+0000","last_undegraded":"2024-03-11T21:37:22.543213+0000","last_fullsized":"2024-03-11T21:37:22.543213+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.473532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.473532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":888,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542933+0000","last_change":"2024-03-11T21:34:58.319554+0000","last_active":"2024-03-11T21:37:22.542933+0000","last_peered":"2024-03-11T21:37:22.542933+0000","last_clean":"2024-03-11T21:37:22.542933+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:37:22.542933+0000","last_undegraded":"2024-03-11T21:37:22.542933+0000","last_fullsized":"2024-03-11T21:37:22.542933+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.319457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":897,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.469813+0000","last_change":"2024-03-11T21:36:08.869894+0000","last_active":"2024-03-11T21:37:22.469813+0000","last_peered":"2024-03-11T21:37:22.469813+0000","last_clean":"2024-03-11T21:37:22.469813+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:37:22.469813+0000","last_undegraded":"2024-03-11T21:37:22.469813+0000","last_fullsized":"2024-03-11T21:37:22.469813+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.869673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.869673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.060598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":890,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539743+0000","last_change":"2024-03-11T21:36:36.107906+0000","last_active":"2024-03-11T21:37:22.539743+0000","last_peered":"2024-03-11T21:37:22.539743+0000","last_clean":"2024-03-11T21:37:22.539743+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:37:22.539743+0000","last_undegraded":"2024-03-11T21:37:22.539743+0000","last_fullsized":"2024-03-11T21:37:22.539743+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.107801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.107801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.759983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546680+0000","last_change":"2024-03-11T21:34:49.557575+0000","last_active":"2024-03-11T21:37:22.546680+0000","last_peered":"2024-03-11T21:37:22.546680+0000","last_clean":"2024-03-11T21:37:22.546680+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:37:22.546680+0000","last_undegraded":"2024-03-11T21:37:22.546680+0000","last_fullsized":"2024-03-11T21:37:22.546680+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:49.557488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:49.557488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":891,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.469334+0000","last_change":"2024-03-11T21:36:43.145228+0000","last_active":"2024-03-11T21:37:23.469334+0000","last_peered":"2024-03-11T21:37:23.469334+0000","last_clean":"2024-03-11T21:37:23.469334+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:37:23.469334+0000","last_undegraded":"2024-03-11T21:37:23.469334+0000","last_fullsized":"2024-03-11T21:37:23.469334+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.145137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.145137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:09.186494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":868,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456772+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:37:22.456772+0000","last_peered":"2024-03-11T21:37:22.456772+0000","last_clean":"2024-03-11T21:37:22.456772+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:37:22.456772+0000","last_undegraded":"2024-03-11T21:37:22.456772+0000","last_fullsized":"2024-03-11T21:37:22.456772+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":883,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470686+0000","last_change":"2024-03-11T21:34:40.037312+0000","last_active":"2024-03-11T21:37:23.470686+0000","last_peered":"2024-03-11T21:37:23.470686+0000","last_clean":"2024-03-11T21:37:23.470686+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:37:23.470686+0000","last_undegraded":"2024-03-11T21:37:23.470686+0000","last_fullsized":"2024-03-11T21:37:23.470686+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:40.037161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:40.037161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542991+0000","last_change":"2024-03-11T21:35:05.211872+0000","last_active":"2024-03-11T21:37:22.542991+0000","last_peered":"2024-03-11T21:37:22.542991+0000","last_clean":"2024-03-11T21:37:22.542991+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:37:22.542991+0000","last_undegraded":"2024-03-11T21:37:22.542991+0000","last_fullsized":"2024-03-11T21:37:22.542991+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.211634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":896,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468287+0000","last_change":"2024-03-11T21:36:30.828411+0000","last_active":"2024-03-11T21:37:22.468287+0000","last_peered":"2024-03-11T21:37:22.468287+0000","last_clean":"2024-03-11T21:37:22.468287+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:37:22.468287+0000","last_undegraded":"2024-03-11T21:37:22.468287+0000","last_fullsized":"2024-03-11T21:37:22.468287+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.828323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.828323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.963712+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":288,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.484953+0000","last_change":"2024-03-11T21:34:52.102628+0000","last_active":"2024-03-11T21:37:24.484953+0000","last_peered":"2024-03-11T21:37:24.484953+0000","last_clean":"2024-03-11T21:37:24.484953+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:37:24.484953+0000","last_undegraded":"2024-03-11T21:37:24.484953+0000","last_fullsized":"2024-03-11T21:37:24.484953+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:52.102271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.102271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":887,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539805+0000","last_change":"2024-03-11T21:34:54.321311+0000","last_active":"2024-03-11T21:37:22.539805+0000","last_peered":"2024-03-11T21:37:22.539805+0000","last_clean":"2024-03-11T21:37:22.539805+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:37:22.539805+0000","last_undegraded":"2024-03-11T21:37:22.539805+0000","last_fullsized":"2024-03-11T21:37:22.539805+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.321007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.321007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643343+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454710+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:37:22.454710+0000","last_peered":"2024-03-11T21:37:22.454710+0000","last_clean":"2024-03-11T21:37:22.454710+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:37:22.454710+0000","last_undegraded":"2024-03-11T21:37:22.454710+0000","last_fullsized":"2024-03-11T21:37:22.454710+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548638+0000","last_change":"2024-03-11T21:36:26.767684+0000","last_active":"2024-03-11T21:37:22.548638+0000","last_peered":"2024-03-11T21:37:22.548638+0000","last_clean":"2024-03-11T21:37:22.548638+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:37:22.548638+0000","last_undegraded":"2024-03-11T21:37:22.548638+0000","last_fullsized":"2024-03-11T21:37:22.548638+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.767129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.767129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.431972+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541324+0000","last_change":"2024-03-11T21:35:02.148627+0000","last_active":"2024-03-11T21:37:22.541324+0000","last_peered":"2024-03-11T21:37:22.541324+0000","last_clean":"2024-03-11T21:37:22.541324+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:37:22.541324+0000","last_undegraded":"2024-03-11T21:37:22.541324+0000","last_fullsized":"2024-03-11T21:37:22.541324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.148344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.148344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":760,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482773+0000","last_change":"2024-03-11T21:35:11.130108+0000","last_active":"2024-03-11T21:37:24.482773+0000","last_peered":"2024-03-11T21:37:24.482773+0000","last_clean":"2024-03-11T21:37:24.482773+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:37:24.482773+0000","last_undegraded":"2024-03-11T21:37:24.482773+0000","last_fullsized":"2024-03-11T21:37:24.482773+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.129979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.129979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.555541+0000","last_change":"2024-03-11T21:34:59.525103+0000","last_active":"2024-03-11T21:37:22.555541+0000","last_peered":"2024-03-11T21:37:22.555541+0000","last_clean":"2024-03-11T21:37:22.555541+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:37:22.555541+0000","last_undegraded":"2024-03-11T21:37:22.555541+0000","last_fullsized":"2024-03-11T21:37:22.555541+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:59.524800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:59.524800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.345106+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.556508+0000","last_change":"2024-03-11T21:36:04.661608+0000","last_active":"2024-03-11T21:37:22.556508+0000","last_peered":"2024-03-11T21:37:22.556508+0000","last_clean":"2024-03-11T21:37:22.556508+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:37:22.556508+0000","last_undegraded":"2024-03-11T21:37:22.556508+0000","last_fullsized":"2024-03-11T21:37:22.556508+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.661495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.661495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.634867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":883,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472402+0000","last_change":"2024-03-11T21:34:58.091810+0000","last_active":"2024-03-11T21:37:23.472402+0000","last_peered":"2024-03-11T21:37:23.472402+0000","last_clean":"2024-03-11T21:37:23.472402+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:37:23.472402+0000","last_undegraded":"2024-03-11T21:37:23.472402+0000","last_fullsized":"2024-03-11T21:37:23.472402+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.091664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.091664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545856+0000","last_change":"2024-03-11T21:36:33.044584+0000","last_active":"2024-03-11T21:37:22.545856+0000","last_peered":"2024-03-11T21:37:22.545856+0000","last_clean":"2024-03-11T21:37:22.545856+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:37:22.545856+0000","last_undegraded":"2024-03-11T21:37:22.545856+0000","last_fullsized":"2024-03-11T21:37:22.545856+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:33.044471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.044471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.788507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":869,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454829+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:37:22.454829+0000","last_peered":"2024-03-11T21:37:22.454829+0000","last_clean":"2024-03-11T21:37:22.454829+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:37:22.454829+0000","last_undegraded":"2024-03-11T21:37:22.454829+0000","last_fullsized":"2024-03-11T21:37:22.454829+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686898+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":288,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485753+0000","last_change":"2024-03-11T21:34:39.164537+0000","last_active":"2024-03-11T21:37:24.485753+0000","last_peered":"2024-03-11T21:37:24.485753+0000","last_clean":"2024-03-11T21:37:24.485753+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:37:24.485753+0000","last_undegraded":"2024-03-11T21:37:24.485753+0000","last_fullsized":"2024-03-11T21:37:24.485753+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.164421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.164421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542292+0000","last_change":"2024-03-11T21:36:03.284628+0000","last_active":"2024-03-11T21:37:22.542292+0000","last_peered":"2024-03-11T21:37:22.542292+0000","last_clean":"2024-03-11T21:37:22.542292+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:37:22.542292+0000","last_undegraded":"2024-03-11T21:37:22.542292+0000","last_fullsized":"2024-03-11T21:37:22.542292+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:03.284530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.284530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":884,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470927+0000","last_change":"2024-03-11T21:34:36.108350+0000","last_active":"2024-03-11T21:37:23.470927+0000","last_peered":"2024-03-11T21:37:23.470927+0000","last_clean":"2024-03-11T21:37:23.470927+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:37:23.470927+0000","last_undegraded":"2024-03-11T21:37:23.470927+0000","last_fullsized":"2024-03-11T21:37:23.470927+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:36.108246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:36.108246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":869,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455930+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:37:22.455930+0000","last_peered":"2024-03-11T21:37:22.455930+0000","last_clean":"2024-03-11T21:37:22.455930+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:37:22.455930+0000","last_undegraded":"2024-03-11T21:37:22.455930+0000","last_fullsized":"2024-03-11T21:37:22.455930+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":896,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468533+0000","last_change":"2024-03-11T21:36:35.782923+0000","last_active":"2024-03-11T21:37:22.468533+0000","last_peered":"2024-03-11T21:37:22.468533+0000","last_clean":"2024-03-11T21:37:22.468533+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:37:22.468533+0000","last_undegraded":"2024-03-11T21:37:22.468533+0000","last_fullsized":"2024-03-11T21:37:22.468533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:36:35.782796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.504664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":760,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485297+0000","last_change":"2024-03-11T21:35:01.154707+0000","last_active":"2024-03-11T21:37:24.485297+0000","last_peered":"2024-03-11T21:37:24.485297+0000","last_clean":"2024-03-11T21:37:24.485297+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:37:24.485297+0000","last_undegraded":"2024-03-11T21:37:24.485297+0000","last_fullsized":"2024-03-11T21:37:24.485297+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.154502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.154502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":898,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472174+0000","last_change":"2024-03-11T21:36:45.220132+0000","last_active":"2024-03-11T21:37:23.472174+0000","last_peered":"2024-03-11T21:37:23.472174+0000","last_clean":"2024-03-11T21:37:23.472174+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:37:23.472174+0000","last_undegraded":"2024-03-11T21:37:23.472174+0000","last_fullsized":"2024-03-11T21:37:23.472174+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.219989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.219989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.466225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1202,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456176+0000","last_change":"2024-03-11T21:31:05.363156+0000","last_active":"2024-03-11T21:37:22.456176+0000","last_peered":"2024-03-11T21:37:22.456176+0000","last_clean":"2024-03-11T21:37:22.456176+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:37:22.456176+0000","last_undegraded":"2024-03-11T21:37:22.456176+0000","last_fullsized":"2024-03-11T21:37:22.456176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:31:05.363052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.363052+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1067,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.458061+0000","last_change":"2024-03-11T21:30:44.296718+0000","last_active":"2024-03-11T21:37:22.458061+0000","last_peered":"2024-03-11T21:37:22.458061+0000","last_clean":"2024-03-11T21:37:22.458061+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:37:22.458061+0000","last_undegraded":"2024-03-11T21:37:22.458061+0000","last_fullsized":"2024-03-11T21:37:22.458061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T21:30:44.296431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:44.296431+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685735+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1064,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468722+0000","last_change":"2024-03-11T21:36:25.851234+0000","last_active":"2024-03-11T21:37:22.468722+0000","last_peered":"2024-03-11T21:37:22.468722+0000","last_clean":"2024-03-11T21:37:22.468722+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:37:22.468722+0000","last_undegraded":"2024-03-11T21:37:22.468722+0000","last_fullsized":"2024-03-11T21:37:22.468722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.851095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.851095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:39.358983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1031,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546027+0000","last_change":"2024-03-11T21:35:10.696295+0000","last_active":"2024-03-11T21:37:22.546027+0000","last_peered":"2024-03-11T21:37:22.546027+0000","last_clean":"2024-03-11T21:37:22.546027+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:37:22.546027+0000","last_undegraded":"2024-03-11T21:37:22.546027+0000","last_fullsized":"2024-03-11T21:37:22.546027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.696090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.696090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":898,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547624+0000","last_change":"2024-03-11T21:36:37.014962+0000","last_active":"2024-03-11T21:37:22.547624+0000","last_peered":"2024-03-11T21:37:22.547624+0000","last_clean":"2024-03-11T21:37:22.547624+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:37:22.547624+0000","last_undegraded":"2024-03-11T21:37:22.547624+0000","last_fullsized":"2024-03-11T21:37:22.547624+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.014751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.014751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:04.818693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1069,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547016+0000","last_change":"2024-03-11T21:35:11.672580+0000","last_active":"2024-03-11T21:37:22.547016+0000","last_peered":"2024-03-11T21:37:22.547016+0000","last_clean":"2024-03-11T21:37:22.547016+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:37:22.547016+0000","last_undegraded":"2024-03-11T21:37:22.547016+0000","last_fullsized":"2024-03-11T21:37:22.547016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:35:11.672324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.672324+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":942,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482259+0000","last_change":"2024-03-11T21:34:48.064829+0000","last_active":"2024-03-11T21:37:24.482259+0000","last_peered":"2024-03-11T21:37:24.482259+0000","last_clean":"2024-03-11T21:37:24.482259+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:37:24.482259+0000","last_undegraded":"2024-03-11T21:37:24.482259+0000","last_fullsized":"2024-03-11T21:37:24.482259+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:48.064622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:34:48.064622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1056,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.467297+0000","last_change":"2024-03-11T21:36:24.816126+0000","last_active":"2024-03-11T21:37:22.467297+0000","last_peered":"2024-03-11T21:37:22.467297+0000","last_clean":"2024-03-11T21:37:22.467297+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:37:22.467297+0000","last_undegraded":"2024-03-11T21:37:22.467297+0000","last_fullsized":"2024-03-11T21:37:22.467297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:24.816023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.816023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:32.230362+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1040,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468876+0000","last_change":"2024-03-11T21:35:02.012107+0000","last_active":"2024-03-11T21:37:23.468876+0000","last_peered":"2024-03-11T21:37:23.468876+0000","last_clean":"2024-03-11T21:37:23.468876+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:37:23.468876+0000","last_undegraded":"2024-03-11T21:37:23.468876+0000","last_fullsized":"2024-03-11T21:37:23.468876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.011982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.011982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":889,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547894+0000","last_change":"2024-03-11T21:36:08.644929+0000","last_active":"2024-03-11T21:37:22.547894+0000","last_peered":"2024-03-11T21:37:22.547894+0000","last_clean":"2024-03-11T21:37:22.547894+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:37:22.547894+0000","last_undegraded":"2024-03-11T21:37:22.547894+0000","last_fullsized":"2024-03-11T21:37:22.547894+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.644692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.644692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.731782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1089,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456481+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:37:22.456481+0000","last_peered":"2024-03-11T21:37:22.456481+0000","last_clean":"2024-03-11T21:37:22.456481+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:37:22.456481+0000","last_undegraded":"2024-03-11T21:37:22.456481+0000","last_fullsized":"2024-03-11T21:37:22.456481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1064,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.467925+0000","last_change":"2024-03-11T21:36:11.798174+0000","last_active":"2024-03-11T21:37:22.467925+0000","last_peered":"2024-03-11T21:37:22.467925+0000","last_clean":"2024-03-11T21:37:22.467925+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:37:22.467925+0000","last_undegraded":"2024-03-11T21:37:22.467925+0000","last_fullsized":"2024-03-11T21:37:22.467925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.701349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1056,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468030+0000","last_change":"2024-03-11T21:36:43.849743+0000","last_active":"2024-03-11T21:37:22.468030+0000","last_peered":"2024-03-11T21:37:22.468030+0000","last_clean":"2024-03-11T21:37:22.468030+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:37:22.468030+0000","last_undegraded":"2024-03-11T21:37:22.468030+0000","last_fullsized":"2024-03-11T21:37:22.468030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.849578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.849578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.597341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1024,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454606+0000","last_change":"2024-03-11T21:35:11.139678+0000","last_active":"2024-03-11T21:37:22.454606+0000","last_peered":"2024-03-11T21:37:22.454606+0000","last_clean":"2024-03-11T21:37:22.454606+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:37:22.454606+0000","last_undegraded":"2024-03-11T21:37:22.454606+0000","last_fullsized":"2024-03-11T21:37:22.454606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.139473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.139473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":769,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.481265+0000","last_change":"2024-03-11T21:36:45.055123+0000","last_active":"2024-03-11T21:37:24.481265+0000","last_peered":"2024-03-11T21:37:24.481265+0000","last_clean":"2024-03-11T21:37:24.481265+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:37:24.481265+0000","last_undegraded":"2024-03-11T21:37:24.481265+0000","last_fullsized":"2024-03-11T21:37:24.481265+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.054820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.210724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1124,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540571+0000","last_change":"2024-03-11T21:35:12.343880+0000","last_active":"2024-03-11T21:37:22.540571+0000","last_peered":"2024-03-11T21:37:22.540571+0000","last_clean":"2024-03-11T21:37:22.540571+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:37:22.540571+0000","last_undegraded":"2024-03-11T21:37:22.540571+0000","last_fullsized":"2024-03-11T21:37:22.540571+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:35:12.343626+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.343626+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1036,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455966+0000","last_change":"2024-03-11T21:30:51.296552+0000","last_active":"2024-03-11T21:37:22.455966+0000","last_peered":"2024-03-11T21:37:22.455966+0000","last_clean":"2024-03-11T21:37:22.455966+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:37:22.455966+0000","last_undegraded":"2024-03-11T21:37:22.455966+0000","last_fullsized":"2024-03-11T21:37:22.455966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:51.296264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:30:51.296264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685898+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":948,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482392+0000","last_change":"2024-03-11T21:36:20.116589+0000","last_active":"2024-03-11T21:37:24.482392+0000","last_peered":"2024-03-11T21:37:24.482392+0000","last_clean":"2024-03-11T21:37:24.482392+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:37:24.482392+0000","last_undegraded":"2024-03-11T21:37:24.482392+0000","last_fullsized":"2024-03-11T21:37:24.482392+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:20.116311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.116311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.628264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1052,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.470027+0000","last_change":"2024-03-11T21:36:01.810406+0000","last_active":"2024-03-11T21:37:22.470027+0000","last_peered":"2024-03-11T21:37:22.470027+0000","last_clean":"2024-03-11T21:37:22.470027+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:37:22.470027+0000","last_undegraded":"2024-03-11T21:37:22.470027+0000","last_fullsized":"2024-03-11T21:37:22.470027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:01.810170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:01.810170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":898,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547243+0000","last_change":"2024-03-11T21:36:11.656705+0000","last_active":"2024-03-11T21:37:22.547243+0000","last_peered":"2024-03-11T21:37:22.547243+0000","last_clean":"2024-03-11T21:37:22.547243+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:37:22.547243+0000","last_undegraded":"2024-03-11T21:37:22.547243+0000","last_fullsized":"2024-03-11T21:37:22.547243+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:36:11.656610+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.656610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.369092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1066,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549678+0000","last_change":"2024-03-11T21:35:01.534298+0000","last_active":"2024-03-11T21:37:22.549678+0000","last_peered":"2024-03-11T21:37:22.549678+0000","last_clean":"2024-03-11T21:37:22.549678+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:37:22.549678+0000","last_undegraded":"2024-03-11T21:37:22.549678+0000","last_fullsized":"2024-03-11T21:37:22.549678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:01.534131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.534131+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.345064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":908,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485684+0000","last_change":"2024-03-11T21:34:41.173845+0000","last_active":"2024-03-11T21:37:24.485684+0000","last_peered":"2024-03-11T21:37:24.485684+0000","last_clean":"2024-03-11T21:37:24.485684+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:37:24.485684+0000","last_undegraded":"2024-03-11T21:37:24.485684+0000","last_fullsized":"2024-03-11T21:37:24.485684+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1239,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468597+0000","last_change":"2024-03-11T21:36:07.905103+0000","last_active":"2024-03-11T21:37:22.468597+0000","last_peered":"2024-03-11T21:37:22.468597+0000","last_clean":"2024-03-11T21:37:22.468597+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:37:22.468597+0000","last_undegraded":"2024-03-11T21:37:22.468597+0000","last_fullsized":"2024-03-11T21:37:22.468597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:36:07.904875+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.904875+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:22.701299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1055,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542139+0000","last_change":"2024-03-11T21:36:45.224370+0000","last_active":"2024-03-11T21:37:22.542139+0000","last_peered":"2024-03-11T21:37:22.542139+0000","last_clean":"2024-03-11T21:37:22.542139+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:37:22.542139+0000","last_undegraded":"2024-03-11T21:37:22.542139+0000","last_fullsized":"2024-03-11T21:37:22.542139+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.224251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:13.711133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":893,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549782+0000","last_change":"2024-03-11T21:33:39.596996+0000","last_active":"2024-03-11T21:37:22.549782+0000","last_peered":"2024-03-11T21:37:22.549782+0000","last_clean":"2024-03-11T21:37:22.549782+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:37:22.549782+0000","last_undegraded":"2024-03-11T21:37:22.549782+0000","last_fullsized":"2024-03-11T21:37:22.549782+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.596784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.596784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1093,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547358+0000","last_change":"2024-03-11T21:36:31.677071+0000","last_active":"2024-03-11T21:37:22.547358+0000","last_peered":"2024-03-11T21:37:22.547358+0000","last_clean":"2024-03-11T21:37:22.547358+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:37:22.547358+0000","last_undegraded":"2024-03-11T21:37:22.547358+0000","last_fullsized":"2024-03-11T21:37:22.547358+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:36:31.676948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.676948+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.706279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":936,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.484764+0000","last_change":"2024-03-11T21:36:05.135845+0000","last_active":"2024-03-11T21:37:24.484764+0000","last_peered":"2024-03-11T21:37:24.484764+0000","last_clean":"2024-03-11T21:37:24.484764+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:37:24.484764+0000","last_undegraded":"2024-03-11T21:37:24.484764+0000","last_fullsized":"2024-03-11T21:37:24.484764+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:05.135750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:05.135750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:24.810637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1224,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541884+0000","last_change":"2024-03-11T21:35:07.191988+0000","last_active":"2024-03-11T21:37:22.541884+0000","last_peered":"2024-03-11T21:37:22.541884+0000","last_clean":"2024-03-11T21:37:22.541884+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:37:22.541884+0000","last_undegraded":"2024-03-11T21:37:22.541884+0000","last_fullsized":"2024-03-11T21:37:22.541884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:35:07.191770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.191770+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1040,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547062+0000","last_change":"2024-03-11T21:34:42.857662+0000","last_active":"2024-03-11T21:37:22.547062+0000","last_peered":"2024-03-11T21:37:22.547062+0000","last_clean":"2024-03-11T21:37:22.547062+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:37:22.547062+0000","last_undegraded":"2024-03-11T21:37:22.547062+0000","last_fullsized":"2024-03-11T21:37:22.547062+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.857448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.857448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455394+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:37:22.455394+0000","last_peered":"2024-03-11T21:37:22.455394+0000","last_clean":"2024-03-11T21:37:22.455394+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:37:22.455394+0000","last_undegraded":"2024-03-11T21:37:22.455394+0000","last_fullsized":"2024-03-11T21:37:22.455394+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1083,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539805+0000","last_change":"2024-03-11T21:36:07.268802+0000","last_active":"2024-03-11T21:37:22.539805+0000","last_peered":"2024-03-11T21:37:22.539805+0000","last_clean":"2024-03-11T21:37:22.539805+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:37:22.539805+0000","last_undegraded":"2024-03-11T21:37:22.539805+0000","last_fullsized":"2024-03-11T21:37:22.539805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:36:07.268562+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.268562+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1045,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454842+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:37:22.454842+0000","last_peered":"2024-03-11T21:37:22.454842+0000","last_clean":"2024-03-11T21:37:22.454842+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:37:22.454842+0000","last_undegraded":"2024-03-11T21:37:22.454842+0000","last_fullsized":"2024-03-11T21:37:22.454842+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1035,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548205+0000","last_change":"2024-03-11T21:35:12.702568+0000","last_active":"2024-03-11T21:37:22.548205+0000","last_peered":"2024-03-11T21:37:22.548205+0000","last_clean":"2024-03-11T21:37:22.548205+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:37:22.548205+0000","last_undegraded":"2024-03-11T21:37:22.548205+0000","last_fullsized":"2024-03-11T21:37:22.548205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.702301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.702301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1025,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.554667+0000","last_change":"2024-03-11T21:36:07.698299+0000","last_active":"2024-03-11T21:37:22.554667+0000","last_peered":"2024-03-11T21:37:22.554667+0000","last_clean":"2024-03-11T21:37:22.554667+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:37:22.554667+0000","last_undegraded":"2024-03-11T21:37:22.554667+0000","last_fullsized":"2024-03-11T21:37:22.554667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.698066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.698066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.708798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.554922+0000","last_change":"2024-03-11T21:36:08.684202+0000","last_active":"2024-03-11T21:37:22.554922+0000","last_peered":"2024-03-11T21:37:22.554922+0000","last_clean":"2024-03-11T21:37:22.554922+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:37:22.554922+0000","last_undegraded":"2024-03-11T21:37:22.554922+0000","last_fullsized":"2024-03-11T21:37:22.554922+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.684101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.684101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.559643+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1230,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547775+0000","last_change":"2024-03-11T21:36:19.742269+0000","last_active":"2024-03-11T21:37:22.547775+0000","last_peered":"2024-03-11T21:37:22.547775+0000","last_clean":"2024-03-11T21:37:22.547775+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:37:22.547775+0000","last_undegraded":"2024-03-11T21:37:22.547775+0000","last_fullsized":"2024-03-11T21:37:22.547775+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:36:19.742119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.742119+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:31.824078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1059,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539912+0000","last_change":"2024-03-11T21:36:13.204077+0000","last_active":"2024-03-11T21:37:22.539912+0000","last_peered":"2024-03-11T21:37:22.539912+0000","last_clean":"2024-03-11T21:37:22.539912+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:37:22.539912+0000","last_undegraded":"2024-03-11T21:37:22.539912+0000","last_fullsized":"2024-03-11T21:37:22.539912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:13.203841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.203841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.395164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1093,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472420+0000","last_change":"2024-03-11T21:34:53.090278+0000","last_active":"2024-03-11T21:37:23.472420+0000","last_peered":"2024-03-11T21:37:23.472420+0000","last_clean":"2024-03-11T21:37:23.472420+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:37:23.472420+0000","last_undegraded":"2024-03-11T21:37:23.472420+0000","last_fullsized":"2024-03-11T21:37:23.472420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:53.090147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.090147+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1072,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.543003+0000","last_change":"2024-03-11T21:36:42.718457+0000","last_active":"2024-03-11T21:37:22.543003+0000","last_peered":"2024-03-11T21:37:22.543003+0000","last_clean":"2024-03-11T21:37:22.543003+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:37:22.543003+0000","last_undegraded":"2024-03-11T21:37:22.543003+0000","last_fullsized":"2024-03-11T21:37:22.543003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:36:42.718312+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.718312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:06.767644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1026,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455692+0000","last_change":"2024-03-11T21:31:04.355073+0000","last_active":"2024-03-11T21:37:22.455692+0000","last_peered":"2024-03-11T21:37:22.455692+0000","last_clean":"2024-03-11T21:37:22.455692+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:37:22.455692+0000","last_undegraded":"2024-03-11T21:37:22.455692+0000","last_fullsized":"2024-03-11T21:37:22.455692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:31:04.354966+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.354966+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456088+0000","last_change":"2024-03-11T21:31:54.451439+0000","last_active":"2024-03-11T21:37:22.456088+0000","last_peered":"2024-03-11T21:37:22.456088+0000","last_clean":"2024-03-11T21:37:22.456088+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:37:22.456088+0000","last_undegraded":"2024-03-11T21:37:22.456088+0000","last_fullsized":"2024-03-11T21:37:22.456088+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.451227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.451227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1050,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539700+0000","last_change":"2024-03-11T21:36:41.246983+0000","last_active":"2024-03-11T21:37:22.539700+0000","last_peered":"2024-03-11T21:37:22.539700+0000","last_clean":"2024-03-11T21:37:22.539700+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:37:22.539700+0000","last_undegraded":"2024-03-11T21:37:22.539700+0000","last_fullsized":"2024-03-11T21:37:22.539700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:41.246781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:41.246781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.546575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1889,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547746+0000","last_change":"2024-03-11T21:36:38.633175+0000","last_active":"2024-03-11T21:37:22.547746+0000","last_peered":"2024-03-11T21:37:22.547746+0000","last_clean":"2024-03-11T21:37:22.547746+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:37:22.547746+0000","last_undegraded":"2024-03-11T21:37:22.547746+0000","last_fullsized":"2024-03-11T21:37:22.547746+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:36:38.633053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:38.633053+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.674424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":839,"num_read_kb":834,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1066,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557383+0000","last_change":"2024-03-11T21:34:33.535935+0000","last_active":"2024-03-11T21:37:22.557383+0000","last_peered":"2024-03-11T21:37:22.557383+0000","last_clean":"2024-03-11T21:37:22.557383+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:37:22.557383+0000","last_undegraded":"2024-03-11T21:37:22.557383+0000","last_fullsized":"2024-03-11T21:37:22.557383+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:34:33.535753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.535753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":961,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482320+0000","last_change":"2024-03-11T21:36:09.043237+0000","last_active":"2024-03-11T21:37:24.482320+0000","last_peered":"2024-03-11T21:37:24.482320+0000","last_clean":"2024-03-11T21:37:24.482320+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:37:24.482320+0000","last_undegraded":"2024-03-11T21:37:24.482320+0000","last_fullsized":"2024-03-11T21:37:24.482320+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:09.043133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.043133+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1046,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540556+0000","last_change":"2024-03-11T21:36:37.141040+0000","last_active":"2024-03-11T21:37:22.540556+0000","last_peered":"2024-03-11T21:37:22.540556+0000","last_clean":"2024-03-11T21:37:22.540556+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:37:22.540556+0000","last_undegraded":"2024-03-11T21:37:22.540556+0000","last_fullsized":"2024-03-11T21:37:22.540556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.140941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.140941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.411940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456925+0000","last_change":"2024-03-11T21:31:53.404454+0000","last_active":"2024-03-11T21:37:22.456925+0000","last_peered":"2024-03-11T21:37:22.456925+0000","last_clean":"2024-03-11T21:37:22.456925+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:37:22.456925+0000","last_undegraded":"2024-03-11T21:37:22.456925+0000","last_fullsized":"2024-03-11T21:37:22.456925+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:53.404326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:53.404326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1039,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540504+0000","last_change":"2024-03-11T21:34:57.364338+0000","last_active":"2024-03-11T21:37:22.540504+0000","last_peered":"2024-03-11T21:37:22.540504+0000","last_clean":"2024-03-11T21:37:22.540504+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:37:22.540504+0000","last_undegraded":"2024-03-11T21:37:22.540504+0000","last_fullsized":"2024-03-11T21:37:22.540504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.364057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.364057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1170,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485900+0000","last_change":"2024-03-11T21:34:55.143888+0000","last_active":"2024-03-11T21:37:24.485900+0000","last_peered":"2024-03-11T21:37:24.485900+0000","last_clean":"2024-03-11T21:37:24.485900+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:37:24.485900+0000","last_undegraded":"2024-03-11T21:37:24.485900+0000","last_fullsized":"2024-03-11T21:37:24.485900+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.143614+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104618+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1036,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456451+0000","last_change":"2024-03-11T21:32:10.544142+0000","last_active":"2024-03-11T21:37:22.456451+0000","last_peered":"2024-03-11T21:37:22.456451+0000","last_clean":"2024-03-11T21:37:22.456451+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:37:22.456451+0000","last_undegraded":"2024-03-11T21:37:22.456451+0000","last_fullsized":"2024-03-11T21:37:22.456451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.543732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.543732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":900,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482299+0000","last_change":"2024-03-11T21:33:46.137826+0000","last_active":"2024-03-11T21:37:24.482299+0000","last_peered":"2024-03-11T21:37:24.482299+0000","last_clean":"2024-03-11T21:37:24.482299+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:37:24.482299+0000","last_undegraded":"2024-03-11T21:37:24.482299+0000","last_fullsized":"2024-03-11T21:37:24.482299+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:46.137679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:46.137679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":889,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470472+0000","last_change":"2024-03-11T21:34:35.105891+0000","last_active":"2024-03-11T21:37:23.470472+0000","last_peered":"2024-03-11T21:37:23.470472+0000","last_clean":"2024-03-11T21:37:23.470472+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:37:23.470472+0000","last_undegraded":"2024-03-11T21:37:23.470472+0000","last_fullsized":"2024-03-11T21:37:23.470472+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.105714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.105714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1044,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548134+0000","last_change":"2024-03-11T21:35:03.774565+0000","last_active":"2024-03-11T21:37:22.548134+0000","last_peered":"2024-03-11T21:37:22.548134+0000","last_clean":"2024-03-11T21:37:22.548134+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:37:22.548134+0000","last_undegraded":"2024-03-11T21:37:22.548134+0000","last_fullsized":"2024-03-11T21:37:22.548134+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:03.774354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:03.774354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1046,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454678+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:37:22.454678+0000","last_peered":"2024-03-11T21:37:22.454678+0000","last_clean":"2024-03-11T21:37:22.454678+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:37:22.454678+0000","last_undegraded":"2024-03-11T21:37:22.454678+0000","last_fullsized":"2024-03-11T21:37:22.454678+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1090,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468775+0000","last_change":"2024-03-11T21:36:03.222156+0000","last_active":"2024-03-11T21:37:23.468775+0000","last_peered":"2024-03-11T21:37:23.468775+0000","last_clean":"2024-03-11T21:37:23.468775+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:37:23.468775+0000","last_undegraded":"2024-03-11T21:37:23.468775+0000","last_fullsized":"2024-03-11T21:37:23.468775+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:03.221883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.221883+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.237647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":910,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485675+0000","last_change":"2024-03-11T21:35:04.076881+0000","last_active":"2024-03-11T21:37:24.485675+0000","last_peered":"2024-03-11T21:37:24.485675+0000","last_clean":"2024-03-11T21:37:24.485675+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:37:24.485675+0000","last_undegraded":"2024-03-11T21:37:24.485675+0000","last_fullsized":"2024-03-11T21:37:24.485675+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.076789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.076789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546755+0000","last_change":"2024-03-11T21:34:54.501393+0000","last_active":"2024-03-11T21:37:22.546755+0000","last_peered":"2024-03-11T21:37:22.546755+0000","last_clean":"2024-03-11T21:37:22.546755+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:37:22.546755+0000","last_undegraded":"2024-03-11T21:37:22.546755+0000","last_fullsized":"2024-03-11T21:37:22.546755+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:54.501183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:54.501183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1028,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455541+0000","last_change":"2024-03-11T21:36:42.732454+0000","last_active":"2024-03-11T21:37:22.455541+0000","last_peered":"2024-03-11T21:37:22.455541+0000","last_clean":"2024-03-11T21:37:22.455541+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:37:22.455541+0000","last_undegraded":"2024-03-11T21:37:22.455541+0000","last_fullsized":"2024-03-11T21:37:22.455541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:42.732354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.732354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:57.600353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1301,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454844+0000","last_change":"2024-03-11T21:36:34.527211+0000","last_active":"2024-03-11T21:37:22.454844+0000","last_peered":"2024-03-11T21:37:22.454844+0000","last_clean":"2024-03-11T21:37:22.454844+0000","last_became_active":"2024-03-11T21:22:24.517506+0000","last_became_peered":"2024-03-11T21:22:24.517506+0000","last_unstale":"2024-03-11T21:37:22.454844+0000","last_undegraded":"2024-03-11T21:37:22.454844+0000","last_fullsized":"2024-03-11T21:37:22.454844+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:36:34.526970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.526970+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.043725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1051,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547106+0000","last_change":"2024-03-11T21:36:22.898119+0000","last_active":"2024-03-11T21:37:22.547106+0000","last_peered":"2024-03-11T21:37:22.547106+0000","last_clean":"2024-03-11T21:37:22.547106+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:37:22.547106+0000","last_undegraded":"2024-03-11T21:37:22.547106+0000","last_fullsized":"2024-03-11T21:37:22.547106+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.898004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.898004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:51.002189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1063,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549472+0000","last_change":"2024-03-11T21:34:38.596842+0000","last_active":"2024-03-11T21:37:22.549472+0000","last_peered":"2024-03-11T21:37:22.549472+0000","last_clean":"2024-03-11T21:37:22.549472+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:37:22.549472+0000","last_undegraded":"2024-03-11T21:37:22.549472+0000","last_fullsized":"2024-03-11T21:37:22.549472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:34:38.596731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:38.596731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1031,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548202+0000","last_change":"2024-03-11T21:34:57.751893+0000","last_active":"2024-03-11T21:37:22.548202+0000","last_peered":"2024-03-11T21:37:22.548202+0000","last_clean":"2024-03-11T21:37:22.548202+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:37:22.548202+0000","last_undegraded":"2024-03-11T21:37:22.548202+0000","last_fullsized":"2024-03-11T21:37:22.548202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.751652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":898,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545733+0000","last_change":"2024-03-11T21:36:44.952662+0000","last_active":"2024-03-11T21:37:22.545733+0000","last_peered":"2024-03-11T21:37:22.545733+0000","last_clean":"2024-03-11T21:37:22.545733+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:37:22.545733+0000","last_undegraded":"2024-03-11T21:37:22.545733+0000","last_fullsized":"2024-03-11T21:37:22.545733+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.952524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.952524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:10.790806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1056,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468999+0000","last_change":"2024-03-11T21:35:10.391001+0000","last_active":"2024-03-11T21:37:22.468999+0000","last_peered":"2024-03-11T21:37:22.468999+0000","last_clean":"2024-03-11T21:37:22.468999+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:37:22.468999+0000","last_undegraded":"2024-03-11T21:37:22.468999+0000","last_fullsized":"2024-03-11T21:37:22.468999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.390804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.390804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1064,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.466871+0000","last_change":"2024-03-11T21:36:14.787093+0000","last_active":"2024-03-11T21:37:22.466871+0000","last_peered":"2024-03-11T21:37:22.466871+0000","last_clean":"2024-03-11T21:37:22.466871+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:37:22.466871+0000","last_undegraded":"2024-03-11T21:37:22.466871+0000","last_fullsized":"2024-03-11T21:37:22.466871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.786963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.786963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":992,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.481448+0000","last_change":"2024-03-11T21:36:04.117071+0000","last_active":"2024-03-11T21:37:24.481448+0000","last_peered":"2024-03-11T21:37:24.481448+0000","last_clean":"2024-03-11T21:37:24.481448+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:37:24.481448+0000","last_undegraded":"2024-03-11T21:37:24.481448+0000","last_fullsized":"2024-03-11T21:37:24.481448+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:36:04.116720+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.116720+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.016493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1024,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456384+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:37:22.456384+0000","last_peered":"2024-03-11T21:37:22.456384+0000","last_clean":"2024-03-11T21:37:22.456384+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:37:22.456384+0000","last_undegraded":"2024-03-11T21:37:22.456384+0000","last_fullsized":"2024-03-11T21:37:22.456384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686834+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":897,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468671+0000","last_change":"2024-03-11T21:36:31.174299+0000","last_active":"2024-03-11T21:37:23.468671+0000","last_peered":"2024-03-11T21:37:23.468671+0000","last_clean":"2024-03-11T21:37:23.468671+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:37:23.468671+0000","last_undegraded":"2024-03-11T21:37:23.468671+0000","last_fullsized":"2024-03-11T21:37:23.468671+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:31.174194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.174194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:42.037963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1053,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472080+0000","last_change":"2024-03-11T21:36:28.181725+0000","last_active":"2024-03-11T21:37:23.472080+0000","last_peered":"2024-03-11T21:37:23.472080+0000","last_clean":"2024-03-11T21:37:23.472080+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:37:23.472080+0000","last_undegraded":"2024-03-11T21:37:23.472080+0000","last_fullsized":"2024-03-11T21:37:23.472080+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.217593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1113,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541119+0000","last_change":"2024-03-11T21:36:40.206751+0000","last_active":"2024-03-11T21:37:22.541119+0000","last_peered":"2024-03-11T21:37:22.541119+0000","last_clean":"2024-03-11T21:37:22.541119+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:37:22.541119+0000","last_undegraded":"2024-03-11T21:37:22.541119+0000","last_fullsized":"2024-03-11T21:37:22.541119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:36:40.206510+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.206510+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.588861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1052,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546172+0000","last_change":"2024-03-11T21:36:02.740887+0000","last_active":"2024-03-11T21:37:22.546172+0000","last_peered":"2024-03-11T21:37:22.546172+0000","last_clean":"2024-03-11T21:37:22.546172+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:37:22.546172+0000","last_undegraded":"2024-03-11T21:37:22.546172+0000","last_fullsized":"2024-03-11T21:37:22.546172+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:02.740740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.740740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1033,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557150+0000","last_change":"2024-03-11T21:36:25.713628+0000","last_active":"2024-03-11T21:37:22.557150+0000","last_peered":"2024-03-11T21:37:22.557150+0000","last_clean":"2024-03-11T21:37:22.557150+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:37:22.557150+0000","last_undegraded":"2024-03-11T21:37:22.557150+0000","last_fullsized":"2024-03-11T21:37:22.557150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.713405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.713405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.494128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":889,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547972+0000","last_change":"2024-03-11T21:34:45.790219+0000","last_active":"2024-03-11T21:37:22.547972+0000","last_peered":"2024-03-11T21:37:22.547972+0000","last_clean":"2024-03-11T21:37:22.547972+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:37:22.547972+0000","last_undegraded":"2024-03-11T21:37:22.547972+0000","last_fullsized":"2024-03-11T21:37:22.547972+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:45.790012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:45.790012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1230,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470466+0000","last_change":"2024-03-11T21:34:52.046581+0000","last_active":"2024-03-11T21:37:23.470466+0000","last_peered":"2024-03-11T21:37:23.470466+0000","last_clean":"2024-03-11T21:37:23.470466+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:37:23.470466+0000","last_undegraded":"2024-03-11T21:37:23.470466+0000","last_fullsized":"2024-03-11T21:37:23.470466+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:34:52.046352+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:34:52.046352+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":945,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485375+0000","last_change":"2024-03-11T21:36:24.181588+0000","last_active":"2024-03-11T21:37:24.485375+0000","last_peered":"2024-03-11T21:37:24.485375+0000","last_clean":"2024-03-11T21:37:24.485375+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:37:24.485375+0000","last_undegraded":"2024-03-11T21:37:24.485375+0000","last_fullsized":"2024-03-11T21:37:24.485375+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:36:24.181487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.181487+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.095936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1259,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547176+0000","last_change":"2024-03-11T21:36:20.831599+0000","last_active":"2024-03-11T21:37:22.547176+0000","last_peered":"2024-03-11T21:37:22.547176+0000","last_clean":"2024-03-11T21:37:22.547176+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:37:22.547176+0000","last_undegraded":"2024-03-11T21:37:22.547176+0000","last_fullsized":"2024-03-11T21:37:22.547176+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:20.831450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.831450+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:46.836710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":916,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482912+0000","last_change":"2024-03-11T21:36:34.998047+0000","last_active":"2024-03-11T21:37:24.482912+0000","last_peered":"2024-03-11T21:37:24.482912+0000","last_clean":"2024-03-11T21:37:24.482912+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:37:24.482912+0000","last_undegraded":"2024-03-11T21:37:24.482912+0000","last_fullsized":"2024-03-11T21:37:24.482912+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.997840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.997840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:00.667483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1040,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468250+0000","last_change":"2024-03-11T21:34:57.129410+0000","last_active":"2024-03-11T21:37:23.468250+0000","last_peered":"2024-03-11T21:37:23.468250+0000","last_clean":"2024-03-11T21:37:23.468250+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:37:23.468250+0000","last_undegraded":"2024-03-11T21:37:23.468250+0000","last_fullsized":"2024-03-11T21:37:23.468250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.129296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.129296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":769,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485577+0000","last_change":"2024-03-11T21:36:32.028349+0000","last_active":"2024-03-11T21:37:24.485577+0000","last_peered":"2024-03-11T21:37:24.485577+0000","last_clean":"2024-03-11T21:37:24.485577+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:37:24.485577+0000","last_undegraded":"2024-03-11T21:37:24.485577+0000","last_fullsized":"2024-03-11T21:37:24.485577+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:32.028125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:32.028125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:38.254526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1065,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.466426+0000","last_change":"2024-03-11T21:36:19.822311+0000","last_active":"2024-03-11T21:37:22.466426+0000","last_peered":"2024-03-11T21:37:22.466426+0000","last_clean":"2024-03-11T21:37:22.466426+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:37:22.466426+0000","last_undegraded":"2024-03-11T21:37:22.466426+0000","last_fullsized":"2024-03-11T21:37:22.466426+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.822219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.822219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.316423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1085,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.457775+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:37:22.457775+0000","last_peered":"2024-03-11T21:37:22.457775+0000","last_clean":"2024-03-11T21:37:22.457775+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:37:22.457775+0000","last_undegraded":"2024-03-11T21:37:22.457775+0000","last_fullsized":"2024-03-11T21:37:22.457775+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1076,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542385+0000","last_change":"2024-03-11T21:36:19.279496+0000","last_active":"2024-03-11T21:37:22.542385+0000","last_peered":"2024-03-11T21:37:22.542385+0000","last_clean":"2024-03-11T21:37:22.542385+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:37:22.542385+0000","last_undegraded":"2024-03-11T21:37:22.542385+0000","last_fullsized":"2024-03-11T21:37:22.542385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:36:19.279370+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.279370+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:25.704439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1042,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.550599+0000","last_change":"2024-03-11T21:36:30.693038+0000","last_active":"2024-03-11T21:37:22.550599+0000","last_peered":"2024-03-11T21:37:22.550599+0000","last_clean":"2024-03-11T21:37:22.550599+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:37:22.550599+0000","last_undegraded":"2024-03-11T21:37:22.550599+0000","last_fullsized":"2024-03-11T21:37:22.550599+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.692917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.692917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.071020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":891,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.551297+0000","last_change":"2024-03-11T21:36:36.681111+0000","last_active":"2024-03-11T21:37:22.551297+0000","last_peered":"2024-03-11T21:37:22.551297+0000","last_clean":"2024-03-11T21:37:22.551297+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:37:22.551297+0000","last_undegraded":"2024-03-11T21:37:22.551297+0000","last_fullsized":"2024-03-11T21:37:22.551297+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.681022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.681022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:51.283896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1063,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455948+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:37:22.455948+0000","last_peered":"2024-03-11T21:37:22.455948+0000","last_clean":"2024-03-11T21:37:22.455948+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:37:22.455948+0000","last_undegraded":"2024-03-11T21:37:22.455948+0000","last_fullsized":"2024-03-11T21:37:22.455948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1073,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.466130+0000","last_change":"2024-03-11T21:36:34.793765+0000","last_active":"2024-03-11T21:37:22.466130+0000","last_peered":"2024-03-11T21:37:22.466130+0000","last_clean":"2024-03-11T21:37:22.466130+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:37:22.466130+0000","last_undegraded":"2024-03-11T21:37:22.466130+0000","last_fullsized":"2024-03-11T21:37:22.466130+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.793536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.132621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1045,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557729+0000","last_change":"2024-03-11T21:36:34.745217+0000","last_active":"2024-03-11T21:37:22.557729+0000","last_peered":"2024-03-11T21:37:22.557729+0000","last_clean":"2024-03-11T21:37:22.557729+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:37:22.557729+0000","last_undegraded":"2024-03-11T21:37:22.557729+0000","last_fullsized":"2024-03-11T21:37:22.557729+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.744999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.744999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:56.618971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1052,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.468215+0000","last_change":"2024-03-11T21:35:05.456150+0000","last_active":"2024-03-11T21:37:22.468215+0000","last_peered":"2024-03-11T21:37:22.468215+0000","last_clean":"2024-03-11T21:37:22.468215+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:37:22.468215+0000","last_undegraded":"2024-03-11T21:37:22.468215+0000","last_fullsized":"2024-03-11T21:37:22.468215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549906+0000","last_change":"2024-03-11T21:34:39.587604+0000","last_active":"2024-03-11T21:37:22.549906+0000","last_peered":"2024-03-11T21:37:22.549906+0000","last_clean":"2024-03-11T21:37:22.549906+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:37:22.549906+0000","last_undegraded":"2024-03-11T21:37:22.549906+0000","last_fullsized":"2024-03-11T21:37:22.549906+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:39.587525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:39.587525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1073,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539960+0000","last_change":"2024-03-11T21:36:03.386633+0000","last_active":"2024-03-11T21:37:22.539960+0000","last_peered":"2024-03-11T21:37:22.539960+0000","last_clean":"2024-03-11T21:37:22.539960+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:37:22.539960+0000","last_undegraded":"2024-03-11T21:37:22.539960+0000","last_fullsized":"2024-03-11T21:37:22.539960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:36:03.386397+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.386397+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643676+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1064,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540100+0000","last_change":"2024-03-11T21:36:34.619421+0000","last_active":"2024-03-11T21:37:22.540100+0000","last_peered":"2024-03-11T21:37:22.540100+0000","last_clean":"2024-03-11T21:37:22.540100+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:37:22.540100+0000","last_undegraded":"2024-03-11T21:37:22.540100+0000","last_fullsized":"2024-03-11T21:37:22.540100+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.619136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.619136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.642811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1043,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547728+0000","last_change":"2024-03-11T21:34:58.788887+0000","last_active":"2024-03-11T21:37:22.547728+0000","last_peered":"2024-03-11T21:37:22.547728+0000","last_clean":"2024-03-11T21:37:22.547728+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:37:22.547728+0000","last_undegraded":"2024-03-11T21:37:22.547728+0000","last_fullsized":"2024-03-11T21:37:22.547728+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.788749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1023,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456250+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:37:22.456250+0000","last_peered":"2024-03-11T21:37:22.456250+0000","last_clean":"2024-03-11T21:37:22.456250+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:37:22.456250+0000","last_undegraded":"2024-03-11T21:37:22.456250+0000","last_fullsized":"2024-03-11T21:37:22.456250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":883,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548867+0000","last_change":"2024-03-11T21:34:53.520431+0000","last_active":"2024-03-11T21:37:22.548867+0000","last_peered":"2024-03-11T21:37:22.548867+0000","last_clean":"2024-03-11T21:37:22.548867+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:37:22.548867+0000","last_undegraded":"2024-03-11T21:37:22.548867+0000","last_fullsized":"2024-03-11T21:37:22.548867+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.520342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.520342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1063,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456020+0000","last_change":"2024-03-11T21:30:46.325972+0000","last_active":"2024-03-11T21:37:22.456020+0000","last_peered":"2024-03-11T21:37:22.456020+0000","last_clean":"2024-03-11T21:37:22.456020+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:37:22.456020+0000","last_undegraded":"2024-03-11T21:37:22.456020+0000","last_fullsized":"2024-03-11T21:37:22.456020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:30:46.325866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:46.325866+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1051,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546015+0000","last_change":"2024-03-11T21:34:31.781546+0000","last_active":"2024-03-11T21:37:22.546015+0000","last_peered":"2024-03-11T21:37:22.546015+0000","last_clean":"2024-03-11T21:37:22.546015+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:37:22.546015+0000","last_undegraded":"2024-03-11T21:37:22.546015+0000","last_fullsized":"2024-03-11T21:37:22.546015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.781332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.781332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1226,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547435+0000","last_change":"2024-03-11T21:34:55.714432+0000","last_active":"2024-03-11T21:37:22.547435+0000","last_peered":"2024-03-11T21:37:22.547435+0000","last_clean":"2024-03-11T21:37:22.547435+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:37:22.547435+0000","last_undegraded":"2024-03-11T21:37:22.547435+0000","last_fullsized":"2024-03-11T21:37:22.547435+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:34:55.714209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.714209+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1039,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472550+0000","last_change":"2024-03-11T21:33:45.920045+0000","last_active":"2024-03-11T21:37:23.472550+0000","last_peered":"2024-03-11T21:37:23.472550+0000","last_clean":"2024-03-11T21:37:23.472550+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:37:23.472550+0000","last_undegraded":"2024-03-11T21:37:23.472550+0000","last_fullsized":"2024-03-11T21:37:23.472550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":907,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540135+0000","last_change":"2024-03-11T21:36:44.625010+0000","last_active":"2024-03-11T21:37:22.540135+0000","last_peered":"2024-03-11T21:37:22.540135+0000","last_clean":"2024-03-11T21:37:22.540135+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:37:22.540135+0000","last_undegraded":"2024-03-11T21:37:22.540135+0000","last_fullsized":"2024-03-11T21:37:22.540135+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.624688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.624688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:07.822451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1080,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.469441+0000","last_change":"2024-03-11T21:36:52.718981+0000","last_active":"2024-03-11T21:37:23.469441+0000","last_peered":"2024-03-11T21:37:23.469441+0000","last_clean":"2024-03-11T21:37:23.469441+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:37:23.469441+0000","last_undegraded":"2024-03-11T21:37:23.469441+0000","last_fullsized":"2024-03-11T21:37:23.469441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:42.968507+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:33:42.968507+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1052,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470089+0000","last_change":"2024-03-11T21:34:37.079514+0000","last_active":"2024-03-11T21:37:23.470089+0000","last_peered":"2024-03-11T21:37:23.470089+0000","last_clean":"2024-03-11T21:37:23.470089+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:37:23.470089+0000","last_undegraded":"2024-03-11T21:37:23.470089+0000","last_fullsized":"2024-03-11T21:37:23.470089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:37.079361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:34:37.079361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1220,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.554548+0000","last_change":"2024-03-11T21:36:24.749355+0000","last_active":"2024-03-11T21:37:22.554548+0000","last_peered":"2024-03-11T21:37:22.554548+0000","last_clean":"2024-03-11T21:37:22.554548+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:37:22.554548+0000","last_undegraded":"2024-03-11T21:37:22.554548+0000","last_fullsized":"2024-03-11T21:37:22.554548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:36:24.749208+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.749208+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.423022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1045,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542618+0000","last_change":"2024-03-11T21:36:14.255051+0000","last_active":"2024-03-11T21:37:22.542618+0000","last_peered":"2024-03-11T21:37:22.542618+0000","last_clean":"2024-03-11T21:37:22.542618+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:37:22.542618+0000","last_undegraded":"2024-03-11T21:37:22.542618+0000","last_fullsized":"2024-03-11T21:37:22.542618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.254847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.254847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.154175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":765,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485447+0000","last_change":"2024-03-11T21:36:39.992513+0000","last_active":"2024-03-11T21:37:24.485447+0000","last_peered":"2024-03-11T21:37:24.485447+0000","last_clean":"2024-03-11T21:37:24.485447+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:37:24.485447+0000","last_undegraded":"2024-03-11T21:37:24.485447+0000","last_fullsized":"2024-03-11T21:37:24.485447+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:39.992222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.992222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:44.338226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1060,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456552+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:37:22.456552+0000","last_peered":"2024-03-11T21:37:22.456552+0000","last_clean":"2024-03-11T21:37:22.456552+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:37:22.456552+0000","last_undegraded":"2024-03-11T21:37:22.456552+0000","last_fullsized":"2024-03-11T21:37:22.456552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1063,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542457+0000","last_change":"2024-03-11T21:36:06.398266+0000","last_active":"2024-03-11T21:37:22.542457+0000","last_peered":"2024-03-11T21:37:22.542457+0000","last_clean":"2024-03-11T21:37:22.542457+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:37:22.542457+0000","last_undegraded":"2024-03-11T21:37:22.542457+0000","last_fullsized":"2024-03-11T21:37:22.542457+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.398145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.398145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.643626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1038,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.550279+0000","last_change":"2024-03-11T21:33:45.520748+0000","last_active":"2024-03-11T21:37:22.550279+0000","last_peered":"2024-03-11T21:37:22.550279+0000","last_clean":"2024-03-11T21:37:22.550279+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:37:22.550279+0000","last_undegraded":"2024-03-11T21:37:22.550279+0000","last_fullsized":"2024-03-11T21:37:22.550279+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:45.520626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:33:45.520626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1048,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468125+0000","last_change":"2024-03-11T21:34:41.069563+0000","last_active":"2024-03-11T21:37:23.468125+0000","last_peered":"2024-03-11T21:37:23.468125+0000","last_clean":"2024-03-11T21:37:23.468125+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:37:23.468125+0000","last_undegraded":"2024-03-11T21:37:23.468125+0000","last_fullsized":"2024-03-11T21:37:23.468125+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:41.069432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:34:41.069432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":904,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472347+0000","last_change":"2024-03-11T21:36:25.203562+0000","last_active":"2024-03-11T21:37:23.472347+0000","last_peered":"2024-03-11T21:37:23.472347+0000","last_clean":"2024-03-11T21:37:23.472347+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:37:23.472347+0000","last_undegraded":"2024-03-11T21:37:23.472347+0000","last_fullsized":"2024-03-11T21:37:23.472347+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.203464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.203464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.544573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1084,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541755+0000","last_change":"2024-03-11T21:36:10.243928+0000","last_active":"2024-03-11T21:37:22.541755+0000","last_peered":"2024-03-11T21:37:22.541755+0000","last_clean":"2024-03-11T21:37:22.541755+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:37:22.541755+0000","last_undegraded":"2024-03-11T21:37:22.541755+0000","last_fullsized":"2024-03-11T21:37:22.541755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:10.243644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.243644+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":952,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482698+0000","last_change":"2024-03-11T21:34:44.191646+0000","last_active":"2024-03-11T21:37:24.482698+0000","last_peered":"2024-03-11T21:37:24.482698+0000","last_clean":"2024-03-11T21:37:24.482698+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:37:24.482698+0000","last_undegraded":"2024-03-11T21:37:24.482698+0000","last_fullsized":"2024-03-11T21:37:24.482698+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:44.191432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.191432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":920,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485275+0000","last_change":"2024-03-11T21:34:32.241415+0000","last_active":"2024-03-11T21:37:24.485275+0000","last_peered":"2024-03-11T21:37:24.485275+0000","last_clean":"2024-03-11T21:37:24.485275+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:37:24.485275+0000","last_undegraded":"2024-03-11T21:37:24.485275+0000","last_fullsized":"2024-03-11T21:37:24.485275+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:32.241159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.241159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1033,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547666+0000","last_change":"2024-03-11T21:36:40.661182+0000","last_active":"2024-03-11T21:37:22.547666+0000","last_peered":"2024-03-11T21:37:22.547666+0000","last_clean":"2024-03-11T21:37:22.547666+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:37:22.547666+0000","last_undegraded":"2024-03-11T21:37:22.547666+0000","last_fullsized":"2024-03-11T21:37:22.547666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:59.904990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":895,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.471592+0000","last_change":"2024-03-11T21:35:08.022701+0000","last_active":"2024-03-11T21:37:23.471592+0000","last_peered":"2024-03-11T21:37:23.471592+0000","last_clean":"2024-03-11T21:37:23.471592+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:37:23.471592+0000","last_undegraded":"2024-03-11T21:37:23.471592+0000","last_fullsized":"2024-03-11T21:37:23.471592+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.022536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.022536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1073,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472837+0000","last_change":"2024-03-11T21:36:34.076596+0000","last_active":"2024-03-11T21:37:23.472837+0000","last_peered":"2024-03-11T21:37:23.472837+0000","last_clean":"2024-03-11T21:37:23.472837+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:37:23.472837+0000","last_undegraded":"2024-03-11T21:37:23.472837+0000","last_fullsized":"2024-03-11T21:37:23.472837+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:36:34.076364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.076364+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:02.807798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1050,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547894+0000","last_change":"2024-03-11T21:36:36.046087+0000","last_active":"2024-03-11T21:37:22.547894+0000","last_peered":"2024-03-11T21:37:22.547894+0000","last_clean":"2024-03-11T21:37:22.547894+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:37:22.547894+0000","last_undegraded":"2024-03-11T21:37:22.547894+0000","last_fullsized":"2024-03-11T21:37:22.547894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.045956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.045956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:57.013408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1055,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540029+0000","last_change":"2024-03-11T21:36:10.461846+0000","last_active":"2024-03-11T21:37:22.540029+0000","last_peered":"2024-03-11T21:37:22.540029+0000","last_clean":"2024-03-11T21:37:22.540029+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:37:22.540029+0000","last_undegraded":"2024-03-11T21:37:22.540029+0000","last_fullsized":"2024-03-11T21:37:22.540029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.461724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.461724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:37.175158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1024,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455643+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:37:22.455643+0000","last_peered":"2024-03-11T21:37:22.455643+0000","last_clean":"2024-03-11T21:37:22.455643+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:37:22.455643+0000","last_undegraded":"2024-03-11T21:37:22.455643+0000","last_fullsized":"2024-03-11T21:37:22.455643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.687004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":898,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.471257+0000","last_change":"2024-03-11T21:34:51.090840+0000","last_active":"2024-03-11T21:37:23.471257+0000","last_peered":"2024-03-11T21:37:23.471257+0000","last_clean":"2024-03-11T21:37:23.471257+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:37:23.471257+0000","last_undegraded":"2024-03-11T21:37:23.471257+0000","last_fullsized":"2024-03-11T21:37:23.471257+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:51.090589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:51.090589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1072,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547689+0000","last_change":"2024-03-11T21:34:44.838684+0000","last_active":"2024-03-11T21:37:22.547689+0000","last_peered":"2024-03-11T21:37:22.547689+0000","last_clean":"2024-03-11T21:37:22.547689+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:37:22.547689+0000","last_undegraded":"2024-03-11T21:37:22.547689+0000","last_fullsized":"2024-03-11T21:37:22.547689+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:44.838430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:44.838430+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1062,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542899+0000","last_change":"2024-03-11T21:36:41.700743+0000","last_active":"2024-03-11T21:37:22.542899+0000","last_peered":"2024-03-11T21:37:22.542899+0000","last_clean":"2024-03-11T21:37:22.542899+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:37:22.542899+0000","last_undegraded":"2024-03-11T21:37:22.542899+0000","last_fullsized":"2024-03-11T21:37:22.542899+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:36:41.700576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.382265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1028,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.454987+0000","last_change":"2024-03-11T21:36:29.509965+0000","last_active":"2024-03-11T21:37:22.454987+0000","last_peered":"2024-03-11T21:37:22.454987+0000","last_clean":"2024-03-11T21:37:22.454987+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:37:22.454987+0000","last_undegraded":"2024-03-11T21:37:22.454987+0000","last_fullsized":"2024-03-11T21:37:22.454987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:29.509836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:29.509836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:40.692029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1025,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.550937+0000","last_change":"2024-03-11T21:34:42.859611+0000","last_active":"2024-03-11T21:37:22.550937+0000","last_peered":"2024-03-11T21:37:22.550937+0000","last_clean":"2024-03-11T21:37:22.550937+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:37:22.550937+0000","last_undegraded":"2024-03-11T21:37:22.550937+0000","last_fullsized":"2024-03-11T21:37:22.550937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.859451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.859451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344700+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":881,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455899+0000","last_change":"2024-03-11T21:36:46.798923+0000","last_active":"2024-03-11T21:37:22.455899+0000","last_peered":"2024-03-11T21:37:22.455899+0000","last_clean":"2024-03-11T21:37:22.455899+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:37:22.455899+0000","last_undegraded":"2024-03-11T21:37:22.455899+0000","last_fullsized":"2024-03-11T21:37:22.455899+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:36:46.798825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:52.800246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":1013,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.484363+0000","last_change":"2024-03-11T21:36:39.151588+0000","last_active":"2024-03-11T21:37:24.484363+0000","last_peered":"2024-03-11T21:37:24.484363+0000","last_clean":"2024-03-11T21:37:24.484363+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:37:24.484363+0000","last_undegraded":"2024-03-11T21:37:24.484363+0000","last_fullsized":"2024-03-11T21:37:24.484363+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.151333+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:49.270569+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1048,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545769+0000","last_change":"2024-03-11T21:35:05.742077+0000","last_active":"2024-03-11T21:37:22.545769+0000","last_peered":"2024-03-11T21:37:22.545769+0000","last_clean":"2024-03-11T21:37:22.545769+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:37:22.545769+0000","last_undegraded":"2024-03-11T21:37:22.545769+0000","last_fullsized":"2024-03-11T21:37:22.545769+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:05.741828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.741828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1240,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.543008+0000","last_change":"2024-03-11T21:36:22.429814+0000","last_active":"2024-03-11T21:37:22.543008+0000","last_peered":"2024-03-11T21:37:22.543008+0000","last_clean":"2024-03-11T21:37:22.543008+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:37:22.543008+0000","last_undegraded":"2024-03-11T21:37:22.543008+0000","last_fullsized":"2024-03-11T21:37:22.543008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:36:22.429570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.429570+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:41.430380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1025,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455713+0000","last_change":"2024-03-11T21:36:33.500511+0000","last_active":"2024-03-11T21:37:22.455713+0000","last_peered":"2024-03-11T21:37:22.455713+0000","last_clean":"2024-03-11T21:37:22.455713+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:37:22.455713+0000","last_undegraded":"2024-03-11T21:37:22.455713+0000","last_fullsized":"2024-03-11T21:37:22.455713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:36:33.500161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.500161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.966184+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":897,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545846+0000","last_change":"2024-03-11T21:36:19.803577+0000","last_active":"2024-03-11T21:37:22.545846+0000","last_peered":"2024-03-11T21:37:22.545846+0000","last_clean":"2024-03-11T21:37:22.545846+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:37:22.545846+0000","last_undegraded":"2024-03-11T21:37:22.545846+0000","last_fullsized":"2024-03-11T21:37:22.545846+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.803281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.803281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:29.105969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1074,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546924+0000","last_change":"2024-03-11T21:34:56.556273+0000","last_active":"2024-03-11T21:37:22.546924+0000","last_peered":"2024-03-11T21:37:22.546924+0000","last_clean":"2024-03-11T21:37:22.546924+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:37:22.546924+0000","last_undegraded":"2024-03-11T21:37:22.546924+0000","last_fullsized":"2024-03-11T21:37:22.546924+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:56.556007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.556007+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.345157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1045,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.548001+0000","last_change":"2024-03-11T21:36:23.725753+0000","last_active":"2024-03-11T21:37:22.548001+0000","last_peered":"2024-03-11T21:37:22.548001+0000","last_clean":"2024-03-11T21:37:22.548001+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:37:22.548001+0000","last_undegraded":"2024-03-11T21:37:22.548001+0000","last_fullsized":"2024-03-11T21:37:22.548001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:23.725651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:23.725651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.581569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1045,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472545+0000","last_change":"2024-03-11T21:35:01.025842+0000","last_active":"2024-03-11T21:37:23.472545+0000","last_peered":"2024-03-11T21:37:23.472545+0000","last_clean":"2024-03-11T21:37:23.472545+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:37:23.472545+0000","last_undegraded":"2024-03-11T21:37:23.472545+0000","last_fullsized":"2024-03-11T21:37:23.472545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.025597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.025597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":911,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482460+0000","last_change":"2024-03-11T21:36:12.033537+0000","last_active":"2024-03-11T21:37:24.482460+0000","last_peered":"2024-03-11T21:37:24.482460+0000","last_clean":"2024-03-11T21:37:24.482460+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:37:24.482460+0000","last_undegraded":"2024-03-11T21:37:24.482460+0000","last_fullsized":"2024-03-11T21:37:24.482460+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.033259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.033259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.444912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":873,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455252+0000","last_change":"2024-03-11T21:30:59.344594+0000","last_active":"2024-03-11T21:37:22.455252+0000","last_peered":"2024-03-11T21:37:22.455252+0000","last_clean":"2024-03-11T21:37:22.455252+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:37:22.455252+0000","last_undegraded":"2024-03-11T21:37:22.455252+0000","last_fullsized":"2024-03-11T21:37:22.455252+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.344476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.344476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.685997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1062,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.470176+0000","last_change":"2024-03-11T21:36:27.196610+0000","last_active":"2024-03-11T21:37:23.470176+0000","last_peered":"2024-03-11T21:37:23.470176+0000","last_clean":"2024-03-11T21:37:23.470176+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:37:23.470176+0000","last_undegraded":"2024-03-11T21:37:23.470176+0000","last_fullsized":"2024-03-11T21:37:23.470176+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:36:27.196386+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.196386+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:52.764389+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1062,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540464+0000","last_change":"2024-03-11T21:36:40.711020+0000","last_active":"2024-03-11T21:37:22.540464+0000","last_peered":"2024-03-11T21:37:22.540464+0000","last_clean":"2024-03-11T21:37:22.540464+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:37:22.540464+0000","last_undegraded":"2024-03-11T21:37:22.540464+0000","last_fullsized":"2024-03-11T21:37:22.540464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.710906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:09.755708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":905,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.482833+0000","last_change":"2024-03-11T21:36:28.149075+0000","last_active":"2024-03-11T21:37:24.482833+0000","last_peered":"2024-03-11T21:37:24.482833+0000","last_clean":"2024-03-11T21:37:24.482833+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:37:24.482833+0000","last_undegraded":"2024-03-11T21:37:24.482833+0000","last_fullsized":"2024-03-11T21:37:24.482833+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.148859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.148859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.073735+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1047,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.542251+0000","last_change":"2024-03-11T21:36:17.289970+0000","last_active":"2024-03-11T21:37:22.542251+0000","last_peered":"2024-03-11T21:37:22.542251+0000","last_clean":"2024-03-11T21:37:22.542251+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:37:22.542251+0000","last_undegraded":"2024-03-11T21:37:22.542251+0000","last_fullsized":"2024-03-11T21:37:22.542251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:17.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.289725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.559372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":893,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539428+0000","last_change":"2024-03-11T21:35:04.183348+0000","last_active":"2024-03-11T21:37:22.539428+0000","last_peered":"2024-03-11T21:37:22.539428+0000","last_clean":"2024-03-11T21:37:22.539428+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:37:22.539428+0000","last_undegraded":"2024-03-11T21:37:22.539428+0000","last_fullsized":"2024-03-11T21:37:22.539428+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.183144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.183144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":1087,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540729+0000","last_change":"2024-03-11T21:35:12.212273+0000","last_active":"2024-03-11T21:37:22.540729+0000","last_peered":"2024-03-11T21:37:22.540729+0000","last_clean":"2024-03-11T21:37:22.540729+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:37:22.540729+0000","last_undegraded":"2024-03-11T21:37:22.540729+0000","last_fullsized":"2024-03-11T21:37:22.540729+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:12.212159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.212159+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1044,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.551226+0000","last_change":"2024-03-11T21:36:12.656033+0000","last_active":"2024-03-11T21:37:22.551226+0000","last_peered":"2024-03-11T21:37:22.551226+0000","last_clean":"2024-03-11T21:37:22.551226+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:37:22.551226+0000","last_undegraded":"2024-03-11T21:37:22.551226+0000","last_fullsized":"2024-03-11T21:37:22.551226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.655615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.655615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.899187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1041,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541940+0000","last_change":"2024-03-11T21:34:57.186736+0000","last_active":"2024-03-11T21:37:22.541940+0000","last_peered":"2024-03-11T21:37:22.541940+0000","last_clean":"2024-03-11T21:37:22.541940+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:37:22.541940+0000","last_undegraded":"2024-03-11T21:37:22.541940+0000","last_fullsized":"2024-03-11T21:37:22.541940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:57.186509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:34:57.186509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1025,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557390+0000","last_change":"2024-03-11T21:34:31.520200+0000","last_active":"2024-03-11T21:37:22.557390+0000","last_peered":"2024-03-11T21:37:22.557390+0000","last_clean":"2024-03-11T21:37:22.557390+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:37:22.557390+0000","last_undegraded":"2024-03-11T21:37:22.557390+0000","last_fullsized":"2024-03-11T21:37:22.557390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.520041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:20.277140+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.520041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":882,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.457943+0000","last_change":"2024-03-11T21:31:59.451645+0000","last_active":"2024-03-11T21:37:22.457943+0000","last_peered":"2024-03-11T21:37:22.457943+0000","last_clean":"2024-03-11T21:37:22.457943+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:37:22.457943+0000","last_undegraded":"2024-03-11T21:37:22.457943+0000","last_fullsized":"2024-03-11T21:37:22.457943+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.451354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.451354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1070,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547400+0000","last_change":"2024-03-11T21:34:50.794115+0000","last_active":"2024-03-11T21:37:22.547400+0000","last_peered":"2024-03-11T21:37:22.547400+0000","last_clean":"2024-03-11T21:37:22.547400+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:37:22.547400+0000","last_undegraded":"2024-03-11T21:37:22.547400+0000","last_fullsized":"2024-03-11T21:37:22.547400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:34:50.793817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.793817+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1035,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455339+0000","last_change":"2024-03-11T21:32:13.461368+0000","last_active":"2024-03-11T21:37:22.455339+0000","last_peered":"2024-03-11T21:37:22.455339+0000","last_clean":"2024-03-11T21:37:22.455339+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:37:22.455339+0000","last_undegraded":"2024-03-11T21:37:22.455339+0000","last_fullsized":"2024-03-11T21:37:22.455339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.461151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.686258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1034,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.546982+0000","last_change":"2024-03-11T21:36:09.648618+0000","last_active":"2024-03-11T21:37:22.546982+0000","last_peered":"2024-03-11T21:37:22.546982+0000","last_clean":"2024-03-11T21:37:22.546982+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:37:22.546982+0000","last_undegraded":"2024-03-11T21:37:22.546982+0000","last_fullsized":"2024-03-11T21:37:22.546982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.648319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.648319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:23.702874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1047,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.468377+0000","last_change":"2024-03-11T21:34:42.092702+0000","last_active":"2024-03-11T21:37:23.468377+0000","last_peered":"2024-03-11T21:37:23.468377+0000","last_clean":"2024-03-11T21:37:23.468377+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:37:23.468377+0000","last_undegraded":"2024-03-11T21:37:23.468377+0000","last_fullsized":"2024-03-11T21:37:23.468377+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:42.092348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:34:42.092348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:26.058402+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":904,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.541647+0000","last_change":"2024-03-11T21:36:25.256048+0000","last_active":"2024-03-11T21:37:22.541647+0000","last_peered":"2024-03-11T21:37:22.541647+0000","last_clean":"2024-03-11T21:37:22.541647+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:37:22.541647+0000","last_undegraded":"2024-03-11T21:37:22.541647+0000","last_fullsized":"2024-03-11T21:37:22.541647+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.255841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.255841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:35.942368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1105,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.547375+0000","last_change":"2024-03-11T21:36:11.636803+0000","last_active":"2024-03-11T21:37:22.547375+0000","last_peered":"2024-03-11T21:37:22.547375+0000","last_clean":"2024-03-11T21:37:22.547375+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:37:22.547375+0000","last_undegraded":"2024-03-11T21:37:22.547375+0000","last_fullsized":"2024-03-11T21:37:22.547375+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:36:11.636460+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.636460+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:30.421627+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1046,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456520+0000","last_change":"2024-03-11T21:36:11.544839+0000","last_active":"2024-03-11T21:37:22.456520+0000","last_peered":"2024-03-11T21:37:22.456520+0000","last_clean":"2024-03-11T21:37:22.456520+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:37:22.456520+0000","last_undegraded":"2024-03-11T21:37:22.456520+0000","last_fullsized":"2024-03-11T21:37:22.456520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.544702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.544702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:36.699059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1036,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.455466+0000","last_change":"2024-03-11T21:36:26.444581+0000","last_active":"2024-03-11T21:37:22.455466+0000","last_peered":"2024-03-11T21:37:22.455466+0000","last_clean":"2024-03-11T21:37:22.455466+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:37:22.455466+0000","last_undegraded":"2024-03-11T21:37:22.455466+0000","last_fullsized":"2024-03-11T21:37:22.455466+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.444362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.444362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.863127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1039,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.545782+0000","last_change":"2024-03-11T21:36:14.674630+0000","last_active":"2024-03-11T21:37:22.545782+0000","last_peered":"2024-03-11T21:37:22.545782+0000","last_clean":"2024-03-11T21:37:22.545782+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:37:22.545782+0000","last_undegraded":"2024-03-11T21:37:22.545782+0000","last_fullsized":"2024-03-11T21:37:22.545782+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.674494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.674494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":890,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.456053+0000","last_change":"2024-03-11T21:36:27.467589+0000","last_active":"2024-03-11T21:37:22.456053+0000","last_peered":"2024-03-11T21:37:22.456053+0000","last_clean":"2024-03-11T21:37:22.456053+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:37:22.456053+0000","last_undegraded":"2024-03-11T21:37:22.456053+0000","last_fullsized":"2024-03-11T21:37:22.456053+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:27.467488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.467488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:33.587538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1092,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472945+0000","last_change":"2024-03-11T21:36:26.178132+0000","last_active":"2024-03-11T21:37:23.472945+0000","last_peered":"2024-03-11T21:37:23.472945+0000","last_clean":"2024-03-11T21:37:23.472945+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:37:23.472945+0000","last_undegraded":"2024-03-11T21:37:23.472945+0000","last_fullsized":"2024-03-11T21:37:23.472945+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:36:26.177873+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.177873+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:54.726625+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1038,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.556903+0000","last_change":"2024-03-11T21:34:35.593537+0000","last_active":"2024-03-11T21:37:22.556903+0000","last_peered":"2024-03-11T21:37:22.556903+0000","last_clean":"2024-03-11T21:37:22.556903+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:37:22.556903+0000","last_undegraded":"2024-03-11T21:37:22.556903+0000","last_fullsized":"2024-03-11T21:37:22.556903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:35.592784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:34:35.592784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.344658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":901,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.484987+0000","last_change":"2024-03-11T21:34:43.228208+0000","last_active":"2024-03-11T21:37:24.484987+0000","last_peered":"2024-03-11T21:37:24.484987+0000","last_clean":"2024-03-11T21:37:24.484987+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:37:24.484987+0000","last_undegraded":"2024-03-11T21:37:24.484987+0000","last_fullsized":"2024-03-11T21:37:24.484987+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:43.227861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.227861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.104560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1054,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.540431+0000","last_change":"2024-03-11T21:36:21.238514+0000","last_active":"2024-03-11T21:37:22.540431+0000","last_peered":"2024-03-11T21:37:22.540431+0000","last_clean":"2024-03-11T21:37:22.540431+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:37:22.540431+0000","last_undegraded":"2024-03-11T21:37:22.540431+0000","last_fullsized":"2024-03-11T21:37:22.540431+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:21.238275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.238275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.708937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":759,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-11T21:37:24.485724+0000","last_change":"2024-03-11T21:36:22.157617+0000","last_active":"2024-03-11T21:37:24.485724+0000","last_peered":"2024-03-11T21:37:24.485724+0000","last_clean":"2024-03-11T21:37:24.485724+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:37:24.485724+0000","last_undegraded":"2024-03-11T21:37:24.485724+0000","last_fullsized":"2024-03-11T21:37:24.485724+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.157410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.157410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:43.753331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1072,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.557384+0000","last_change":"2024-03-11T21:36:15.678133+0000","last_active":"2024-03-11T21:37:22.557384+0000","last_peered":"2024-03-11T21:37:22.557384+0000","last_clean":"2024-03-11T21:37:22.557384+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:37:22.557384+0000","last_undegraded":"2024-03-11T21:37:22.557384+0000","last_fullsized":"2024-03-11T21:37:22.557384+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:15.678027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.678027+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:34.603669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1061,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-11T21:37:23.472965+0000","last_change":"2024-03-11T21:36:24.220137+0000","last_active":"2024-03-11T21:37:23.472965+0000","last_peered":"2024-03-11T21:37:23.472965+0000","last_clean":"2024-03-11T21:37:23.472965+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:37:23.472965+0000","last_undegraded":"2024-03-11T21:37:23.472965+0000","last_fullsized":"2024-03-11T21:37:23.472965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:36:24.220021+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.220021+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:45.512238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1058,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.539419+0000","last_change":"2024-03-11T21:36:30.228605+0000","last_active":"2024-03-11T21:37:22.539419+0000","last_peered":"2024-03-11T21:37:22.539419+0000","last_clean":"2024-03-11T21:37:22.539419+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:37:22.539419+0000","last_undegraded":"2024-03-11T21:37:22.539419+0000","last_fullsized":"2024-03-11T21:37:22.539419+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.228458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.228458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:55.430097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1024,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-11T21:37:22.549314+0000","last_change":"2024-03-11T21:35:00.524627+0000","last_active":"2024-03-11T21:37:22.549314+0000","last_peered":"2024-03-11T21:37:22.549314+0000","last_clean":"2024-03-11T21:37:22.549314+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:37:22.549314+0000","last_undegraded":"2024-03-11T21:37:22.549314+0000","last_fullsized":"2024-03-11T21:37:22.549314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:00.524430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.524430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:37:27.345015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8205334,"num_objects":183,"num_object_clones":10,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":199,"num_read_kb":54,"num_write":746,"num_write_kb":12782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26767360,"data_stored":24615988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2042,"ondisk_log_size":2042,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":839,"num_read_kb":834,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841719,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89880,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655272,"statfs":{"total":95995035648,"available":95902998528,"internally_reserved":0,"allocated":14594048,"data_stored":10656018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168768,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76988,"kb_used_data":13864,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93668164,"statfs":{"total":95995035648,"available":95916199936,"internally_reserved":0,"allocated":14196736,"data_stored":10457220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352391,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41568,"kb_used_data":9484,"kb_used_omap":27,"kb_used_meta":32036,"kb_avail":93703584,"statfs":{"total":95995035648,"available":95952470016,"internally_reserved":0,"allocated":9711616,"data_stored":5889295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":32805193},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93600,"kb_used_data":17972,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93651552,"statfs":{"total":95995035648,"available":95899189248,"internally_reserved":0,"allocated":18403328,"data_stored":14653345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378858,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80944,"kb_used_data":9500,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664208,"statfs":{"total":95995035648,"available":95912148992,"internally_reserved":0,"allocated":9728000,"data_stored":5869930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117214,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76744,"kb_used_data":9448,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93668408,"statfs":{"total":95995035648,"available":95916449792,"internally_reserved":0,"allocated":9674752,"data_stored":5855579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855572,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89092,"kb_used_data":13476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93656060,"statfs":{"total":95995035648,"available":95903805440,"internally_reserved":0,"allocated":13799424,"data_stored":9862494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495818,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80792,"kb_used_data":13508,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93664360,"statfs":{"total":95995035648,"available":95912304640,"internally_reserved":0,"allocated":13832192,"data_stored":10065704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8197185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":4122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":1222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4366336,"data_stored":4006673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":4201099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4186112,"data_stored":4002162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":4198740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:37:29.765 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:37:29.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-11T21:37:29.780 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:29.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:29.851 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:29.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.481333+0000 osd.3 (osd.3) 343 : cluster 0 9.2 scrub starts 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.488505+0000 osd.3 (osd.3) 344 : cluster 0 9.2 scrub ok 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.730788+0000 osd.7 (osd.7) 316 : cluster 0 3.f scrub starts 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.732818+0000 osd.6 (osd.6) 253 : cluster 0 39.8 scrub starts 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.733555+0000 osd.7 (osd.7) 317 : cluster 0 3.f scrub ok 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.734650+0000 osd.6 (osd.6) 254 : cluster 0 39.8 scrub ok 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: audit 2024-03-11T21:37:28.783420+0000 mgr.y (mgr.14550) 572 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.952641+0000 osd.0 (osd.0) 326 : cluster 0 6.16 scrub starts 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:28.954156+0000 osd.0 (osd.0) 327 : cluster 0 6.16 scrub ok 2024-03-11T21:37:29.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.204337+0000 osd.4 (osd.4) 309 : cluster 0 3.1a scrub starts 2024-03-11T21:37:29.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.208420+0000 osd.4 (osd.4) 310 : cluster 0 3.1a scrub ok 2024-03-11T21:37:29.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.441550+0000 mon.a (mon.0) 3021 : cluster 0 osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:37:29.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: audit 2024-03-11T21:37:29.443221+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:29.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:29 smithi203 bash[23654]: audit 2024-03-11T21:37:29.445222+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:29.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:30.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:30.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.481333+0000 osd.3 (osd.3) 343 : cluster 0 9.2 scrub starts 2024-03-11T21:37:30.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.488505+0000 osd.3 (osd.3) 344 : cluster 0 9.2 scrub ok 2024-03-11T21:37:30.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.730788+0000 osd.7 (osd.7) 316 : cluster 0 3.f scrub starts 2024-03-11T21:37:30.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.732818+0000 osd.6 (osd.6) 253 : cluster 0 39.8 scrub starts 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.733555+0000 osd.7 (osd.7) 317 : cluster 0 3.f scrub ok 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.734650+0000 osd.6 (osd.6) 254 : cluster 0 39.8 scrub ok 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: audit 2024-03-11T21:37:28.783420+0000 mgr.y (mgr.14550) 572 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.952641+0000 osd.0 (osd.0) 326 : cluster 0 6.16 scrub starts 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:28.954156+0000 osd.0 (osd.0) 327 : cluster 0 6.16 scrub ok 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.204337+0000 osd.4 (osd.4) 309 : cluster 0 3.1a scrub starts 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.208420+0000 osd.4 (osd.4) 310 : cluster 0 3.1a scrub ok 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.441550+0000 mon.a (mon.0) 3021 : cluster 0 osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: audit 2024-03-11T21:37:29.443221+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[21512]: audit 2024-03-11T21:37:29.445222+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.481333+0000 osd.3 (osd.3) 343 : cluster 0 9.2 scrub starts 2024-03-11T21:37:30.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.488505+0000 osd.3 (osd.3) 344 : cluster 0 9.2 scrub ok 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.730788+0000 osd.7 (osd.7) 316 : cluster 0 3.f scrub starts 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.732818+0000 osd.6 (osd.6) 253 : cluster 0 39.8 scrub starts 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.733555+0000 osd.7 (osd.7) 317 : cluster 0 3.f scrub ok 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.734650+0000 osd.6 (osd.6) 254 : cluster 0 39.8 scrub ok 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: audit 2024-03-11T21:37:28.783420+0000 mgr.y (mgr.14550) 572 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.952641+0000 osd.0 (osd.0) 326 : cluster 0 6.16 scrub starts 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:28.954156+0000 osd.0 (osd.0) 327 : cluster 0 6.16 scrub ok 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.204337+0000 osd.4 (osd.4) 309 : cluster 0 3.1a scrub starts 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.208420+0000 osd.4 (osd.4) 310 : cluster 0 3.1a scrub ok 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.441550+0000 mon.a (mon.0) 3021 : cluster 0 osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: audit 2024-03-11T21:37:29.443221+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:30.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:29 smithi143 bash[29284]: audit 2024-03-11T21:37:29.445222+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:30.113 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:30.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.245 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:30.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.381 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:30.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.508 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:30.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:30.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:30.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:30.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:30.905 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:31.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:31.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:31.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:31.148 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.476576+0000 mgr.y (mgr.14550) 573 : cluster 0 pgmap v889: 292 pgs: 25 creating+peering, 36 unknown, 231 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2024-03-11T21:37:31.148 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.736755+0000 osd.7 (osd.7) 318 : cluster 0 5.9 scrub starts 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:29.738481+0000 osd.7 (osd.7) 319 : cluster 0 5.9 scrub ok 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.123232+0000 osd.1 (osd.1) 135 : cluster 0 2.a scrub starts 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.124763+0000 osd.1 (osd.1) 136 : cluster 0 2.a scrub ok 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.215830+0000 osd.4 (osd.4) 311 : cluster 0 118.c scrub starts 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.218231+0000 osd.4 (osd.4) 312 : cluster 0 118.c scrub ok 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: audit 2024-03-11T21:37:30.438383+0000 mon.a (mon.0) 3023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: audit 2024-03-11T21:37:30.443284+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.445719+0000 mon.a (mon.0) 3024 : cluster 0 osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: audit 2024-03-11T21:37:30.448599+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.465401+0000 osd.3 (osd.3) 345 : cluster 0 5.13 scrub starts 2024-03-11T21:37:31.150 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.467624+0000 osd.3 (osd.3) 346 : cluster 0 5.13 scrub ok 2024-03-11T21:37:31.150 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.779148+0000 osd.2 (osd.2) 291 : cluster 0 5.6 scrub starts 2024-03-11T21:37:31.150 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:30 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.781364+0000 osd.2 (osd.2) 292 : cluster 0 5.6 scrub ok 2024-03-11T21:37:31.160 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.476576+0000 mgr.y (mgr.14550) 573 : cluster 0 pgmap v889: 292 pgs: 25 creating+peering, 36 unknown, 231 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.736755+0000 osd.7 (osd.7) 318 : cluster 0 5.9 scrub starts 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:29.738481+0000 osd.7 (osd.7) 319 : cluster 0 5.9 scrub ok 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.123232+0000 osd.1 (osd.1) 135 : cluster 0 2.a scrub starts 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.124763+0000 osd.1 (osd.1) 136 : cluster 0 2.a scrub ok 2024-03-11T21:37:31.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.215830+0000 osd.4 (osd.4) 311 : cluster 0 118.c scrub starts 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.218231+0000 osd.4 (osd.4) 312 : cluster 0 118.c scrub ok 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: audit 2024-03-11T21:37:30.438383+0000 mon.a (mon.0) 3023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: audit 2024-03-11T21:37:30.443284+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.445719+0000 mon.a (mon.0) 3024 : cluster 0 osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: audit 2024-03-11T21:37:30.448599+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.465401+0000 osd.3 (osd.3) 345 : cluster 0 5.13 scrub starts 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.467624+0000 osd.3 (osd.3) 346 : cluster 0 5.13 scrub ok 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.779148+0000 osd.2 (osd.2) 291 : cluster 0 5.6 scrub starts 2024-03-11T21:37:31.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.781364+0000 osd.2 (osd.2) 292 : cluster 0 5.6 scrub ok 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.476576+0000 mgr.y (mgr.14550) 573 : cluster 0 pgmap v889: 292 pgs: 25 creating+peering, 36 unknown, 231 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.736755+0000 osd.7 (osd.7) 318 : cluster 0 5.9 scrub starts 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:29.738481+0000 osd.7 (osd.7) 319 : cluster 0 5.9 scrub ok 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.123232+0000 osd.1 (osd.1) 135 : cluster 0 2.a scrub starts 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.124763+0000 osd.1 (osd.1) 136 : cluster 0 2.a scrub ok 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.215830+0000 osd.4 (osd.4) 311 : cluster 0 118.c scrub starts 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.218231+0000 osd.4 (osd.4) 312 : cluster 0 118.c scrub ok 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: audit 2024-03-11T21:37:30.438383+0000 mon.a (mon.0) 3023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: audit 2024-03-11T21:37:30.443284+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.445719+0000 mon.a (mon.0) 3024 : cluster 0 osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: audit 2024-03-11T21:37:30.448599+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]: dispatch 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.465401+0000 osd.3 (osd.3) 345 : cluster 0 5.13 scrub starts 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.467624+0000 osd.3 (osd.3) 346 : cluster 0 5.13 scrub ok 2024-03-11T21:37:31.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.779148+0000 osd.2 (osd.2) 291 : cluster 0 5.6 scrub starts 2024-03-11T21:37:31.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:30 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.781364+0000 osd.2 (osd.2) 292 : cluster 0 5.6 scrub ok 2024-03-11T21:37:31.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:31.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:31.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:31.550 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:31.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:31.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:31.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:31.812 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:31.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.731914+0000 osd.6 (osd.6) 255 : cluster 0 39.1d scrub starts 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:30.734460+0000 osd.6 (osd.6) 256 : cluster 0 39.1d scrub ok 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.173134+0000 osd.4 (osd.4) 313 : cluster 0 9.10 scrub starts 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.174556+0000 osd.4 (osd.4) 314 : cluster 0 9.10 scrub ok 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: audit 2024-03-11T21:37:31.442412+0000 mon.a (mon.0) 3026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]': finished 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.451923+0000 mon.a (mon.0) 3027 : cluster 0 osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: audit 2024-03-11T21:37:31.456737+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: audit 2024-03-11T21:37:31.458805+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.806440+0000 osd.2 (osd.2) 293 : cluster 0 5.d scrub starts 2024-03-11T21:37:31.926 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:31 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.809625+0000 osd.2 (osd.2) 294 : cluster 0 5.d scrub ok 2024-03-11T21:37:31.941 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:32.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:32.071 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:32.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.185 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:32.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.731914+0000 osd.6 (osd.6) 255 : cluster 0 39.1d scrub starts 2024-03-11T21:37:32.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:30.734460+0000 osd.6 (osd.6) 256 : cluster 0 39.1d scrub ok 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.173134+0000 osd.4 (osd.4) 313 : cluster 0 9.10 scrub starts 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.174556+0000 osd.4 (osd.4) 314 : cluster 0 9.10 scrub ok 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: audit 2024-03-11T21:37:31.442412+0000 mon.a (mon.0) 3026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]': finished 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.451923+0000 mon.a (mon.0) 3027 : cluster 0 osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: audit 2024-03-11T21:37:31.456737+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: audit 2024-03-11T21:37:31.458805+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.806440+0000 osd.2 (osd.2) 293 : cluster 0 5.d scrub starts 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.809625+0000 osd.2 (osd.2) 294 : cluster 0 5.d scrub ok 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.731914+0000 osd.6 (osd.6) 255 : cluster 0 39.1d scrub starts 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:30.734460+0000 osd.6 (osd.6) 256 : cluster 0 39.1d scrub ok 2024-03-11T21:37:32.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.173134+0000 osd.4 (osd.4) 313 : cluster 0 9.10 scrub starts 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.174556+0000 osd.4 (osd.4) 314 : cluster 0 9.10 scrub ok 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: audit 2024-03-11T21:37:31.442412+0000 mon.a (mon.0) 3026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_tier","val": "test-rados-api-smithi143-71773-89-test-flush"}]': finished 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.451923+0000 mon.a (mon.0) 3027 : cluster 0 osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: audit 2024-03-11T21:37:31.456737+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: audit 2024-03-11T21:37:31.458805+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.806440+0000 osd.2 (osd.2) 293 : cluster 0 5.d scrub starts 2024-03-11T21:37:32.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:31 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.809625+0000 osd.2 (osd.2) 294 : cluster 0 5.d scrub ok 2024-03-11T21:37:32.199 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:32.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:32.329 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:32.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.462 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:32.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:32.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:32.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- 172.21.15.143:0/1502217931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68106ef0 msgr2=0x7faf681072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/1502217931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68106ef0 0x7faf681072d0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7faf580099c0 tx=0x7faf58009cd0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- 172.21.15.143:0/1502217931 shutdown_connections 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/1502217931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf681019c0 0x7faf6810db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/1502217931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68106ef0 0x7faf681072d0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/1502217931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf68101000 0x7faf68101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- 172.21.15.143:0/1502217931 >> 172.21.15.143:0/1502217931 conn(0x7faf68077a40 msgr2=0x7faf68077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- 172.21.15.143:0/1502217931 shutdown_connections 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- 172.21.15.143:0/1502217931 wait complete. 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 librados: starting msgr at 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 librados: starting objecter 2024-03-11T21:37:32.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 Processor -- start 2024-03-11T21:37:32.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- start start 2024-03-11T21:37:32.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 librados: setting wanted keys 2024-03-11T21:37:32.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 librados: calling monclient init 2024-03-11T21:37:32.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 0x7faf68198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf681019c0 0x7faf68198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf68106ef0 0x7faf681a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681105d0 con 0x7faf681019c0 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf68110450 con 0x7faf68106ef0 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf6e0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf68110750 con 0x7faf68101000 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 0x7faf68198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf66ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf681019c0 0x7faf68198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 0x7faf68198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40902/0 (socket says 172.21.15.143:40902) 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 -- 172.21.15.143:0/2682637761 learned_addr learned my addr 172.21.15.143:0/2682637761 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf68106ef0 0x7faf681a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf68106ef0 msgr2=0x7faf681a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf68106ef0 0x7faf681a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf681019c0 msgr2=0x7faf68198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf681019c0 0x7faf68198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf681a3ff0 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf681019c0 0x7faf68198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 0x7faf68198a30 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf64ff9700 10 client.?.objecter ms_handle_connect 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf64ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.686+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58009e60 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf681a41c0 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7faf5800f460 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58017650 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 10 client.41780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf68196820 con 0x7faf68101000 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 librados: init done 2024-03-11T21:37:32.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faf68109270 con 0x7faf68101000 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7faf5800f960 con 0x7faf68101000 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf5006f820 0x7faf50071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(580..580 src has 87..580) v4 ==== 10063+0+0 (secure 0 0 0) 0x7faf5808fc30 con 0x7faf68101000 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 osd_map(580..580 src has 87..580) v4 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 3 client.41780.objecter handle_osd_map got epochs [580,580] > 0 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 3 client.41780.objecter handle_osd_map decoding full epoch 580 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf5006f820 0x7faf50071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 20 client.41780.objecter dump_active .. 0 homeless 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v580) v1 ==== 54+0+16 (secure 0 0 0) 0x7faf58013070 con 0x7faf68101000 2024-03-11T21:37:32.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 1 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7faf68062c40 0x7faf6802c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 1 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 0 1 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 4 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 10 client.41780.objecter _send_command 1 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf6804faa0 con 0x7faf68062c40 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf5006f820 0x7faf50071ce0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009680 tx=0x7faf5c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf5006f820 2024-03-11T21:37:32.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7faf68062c40 0x7faf6802c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7faf68062c40 0x7faf6802c6b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.690+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf68062c40 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.694+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf6804faa0 con 0x7faf68062c40 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.694+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.694+0000 7faf64ff9700 10 client.41780.objecter _finish_command 1 = system:0 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.694+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 4 2024-03-11T21:37:32.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.694+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 0 1 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 2 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 2 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf6806aab0 osd=0 3 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 0 2 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 4 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 10 client.41780.objecter _send_command 2 2024-03-11T21:37:32.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf6802cbf0 con 0x7faf68062c40 2024-03-11T21:37:32.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf6802cbf0 con 0x7faf68062c40 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf64ff9700 10 client.41780.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 4 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 0 2 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:37:32.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7faf68062830 0x7faf68196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 1 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 4 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 10 client.41780.objecter _send_command 3 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.710+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf68198760 con 0x7faf68062830 2024-03-11T21:37:32.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7faf68062830 0x7faf68196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf68062830 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7faf68062830 0x7faf68196a80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf68198760 con 0x7faf68062830 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 10 client.41780.objecter _finish_command 3 = system:0 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 4 2024-03-11T21:37:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.714+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 1 3 2024-03-11T21:37:32.695 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 4 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 4 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf680626c0 osd=1 3 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 1 4 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 4 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 10 client.41780.objecter _send_command 4 2024-03-11T21:37:32.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.730+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf68196fc0 con 0x7faf68062830 2024-03-11T21:37:32.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf68196fc0 con 0x7faf68062830 2024-03-11T21:37:32.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 10 client.41780.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 4 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 1 4 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 5 2024-03-11T21:37:32.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7faf68198350 0x7faf681082a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 5 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 2 5 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 4 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 10 client.41780.objecter _send_command 5 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681087e0 con 0x7faf68198350 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7faf68198350 0x7faf681082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf68198350 2024-03-11T21:37:32.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.734+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7faf68198350 0x7faf681082a0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.738+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681087e0 con 0x7faf68198350 2024-03-11T21:37:32.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.738+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:37:32.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.738+0000 7faf64ff9700 10 client.41780.objecter _finish_command 5 = system:0 2024-03-11T21:37:32.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.738+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 4 2024-03-11T21:37:32.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.738+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 2 5 2024-03-11T21:37:32.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 6 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 6 2024-03-11T21:37:32.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf6802dcd0 osd=2 3 2024-03-11T21:37:32.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 2 6 2024-03-11T21:37:32.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 4 2024-03-11T21:37:32.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 10 client.41780.objecter _send_command 6 2024-03-11T21:37:32.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.754+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf681aeab0 con 0x7faf68198350 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf681aeab0 con 0x7faf68198350 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 10 client.41780.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 4 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 2 6 2024-03-11T21:37:32.733 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:37:32.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 7 2024-03-11T21:37:32.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7faf68107e90 0x7faf681b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 7 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 3 7 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 4 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 10 client.41780.objecter _send_command 7 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681980a0 con 0x7faf68107e90 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7faf68107e90 0x7faf681b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf68107e90 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7faf68107e90 0x7faf681b1c40 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681980a0 con 0x7faf68107e90 2024-03-11T21:37:32.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:37:32.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 10 client.41780.objecter _finish_command 7 = system:0 2024-03-11T21:37:32.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 4 2024-03-11T21:37:32.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.758+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 3 7 2024-03-11T21:37:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 8 2024-03-11T21:37:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 8 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf68107d20 osd=3 3 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 3 8 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 4 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 10 client.41780.objecter _send_command 8 2024-03-11T21:37:32.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf681973c0 con 0x7faf68107e90 2024-03-11T21:37:32.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf681973c0 con 0x7faf68107e90 2024-03-11T21:37:32.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 10 client.41780.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 4 2024-03-11T21:37:32.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 3 8 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 9 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7faf681af790 0x7faf681b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 9 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 4 9 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 4 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 10 client.41780.objecter _send_command 9 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681b8170 con 0x7faf681af790 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7faf681af790 0x7faf681b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf681af790 2024-03-11T21:37:32.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.778+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7faf681af790 0x7faf681b5310 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.782+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681b8170 con 0x7faf681af790 2024-03-11T21:37:32.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.782+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:37:32.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.782+0000 7faf64ff9700 10 client.41780.objecter _finish_command 9 = system:0 2024-03-11T21:37:32.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.782+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 4 2024-03-11T21:37:32.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.782+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 4 9 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 10 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 10 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681973c0 osd=4 3 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 4 10 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 4 2024-03-11T21:37:32.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 10 client.41780.objecter _send_command 10 2024-03-11T21:37:32.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.798+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf68197630 con 0x7faf681af790 2024-03-11T21:37:32.779 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf68197630 con 0x7faf681af790 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf64ff9700 10 client.41780.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 4 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 4 10 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 11 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf681b2e60 0x7faf681b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 11 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 5 11 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 4 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 10 client.41780.objecter _send_command 11 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681aedb0 con 0x7faf681b2e60 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.802+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf681b2e60 0x7faf681b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf681b2e60 2024-03-11T21:37:32.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf66ffd700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf681b2e60 0x7faf681b89e0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681aedb0 con 0x7faf681b2e60 2024-03-11T21:37:32.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:37:32.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 10 client.41780.objecter _finish_command 11 = system:0 2024-03-11T21:37:32.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 4 2024-03-11T21:37:32.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.806+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 5 11 2024-03-11T21:37:32.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 12 2024-03-11T21:37:32.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 12 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681aeab0 osd=5 3 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 5 12 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 4 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 10 client.41780.objecter _send_command 12 2024-03-11T21:37:32.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.822+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf680008d0 con 0x7faf681b2e60 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf680008d0 con 0x7faf681b2e60 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf64ff9700 10 client.41780.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 4 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 5 12 2024-03-11T21:37:32.804 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 13 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.826+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faf681b9430 0x7faf681bb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 13 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 6 13 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 4 2024-03-11T21:37:32.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 10 client.41780.objecter _send_command 13 2024-03-11T21:37:32.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681be680 con 0x7faf681b9430 2024-03-11T21:37:32.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faf681b9430 0x7faf681bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf681b9430 2024-03-11T21:37:32.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf67fff700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faf681b9430 0x7faf681bb820 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681be680 con 0x7faf681b9430 2024-03-11T21:37:32.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:37:32.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 10 client.41780.objecter _finish_command 13 = system:0 2024-03-11T21:37:32.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 4 2024-03-11T21:37:32.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.830+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 6 13 2024-03-11T21:37:32.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 14 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 14 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf681af0a0 osd=6 3 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 6 14 2024-03-11T21:37:32.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 4 2024-03-11T21:37:32.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 10 client.41780.objecter _send_command 14 2024-03-11T21:37:32.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.850+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf681b23d0 con 0x7faf681b9430 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf681b23d0 con 0x7faf681b9430 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 10 client.41780.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 4 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 6 14 2024-03-11T21:37:32.829 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 15 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7faf681b2180 0x7faf681bcee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 15 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 7 15 2024-03-11T21:37:32.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 4 2024-03-11T21:37:32.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 10 client.41780.objecter _send_command 15 2024-03-11T21:37:32.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf681be650 con 0x7faf681b2180 2024-03-11T21:37:32.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7faf681b2180 0x7faf681bcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:32.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf64ff9700 10 client.41780.objecter ms_handle_connect 0x7faf681b2180 2024-03-11T21:37:32.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.854+0000 7faf677fe700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7faf681b2180 0x7faf681bcee0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:32.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.858+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf681be650 con 0x7faf681b2180 2024-03-11T21:37:32.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.858+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:37:32.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.858+0000 7faf64ff9700 10 client.41780.objecter _finish_command 15 = system:0 2024-03-11T21:37:32.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.858+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 4 2024-03-11T21:37:32.833 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.858+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 7 15 2024-03-11T21:37:32.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 10 client.41780.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign -1 16 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 20 client.41780.objecter _get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_remove -1 16 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 20 client.41780.objecter get_session s=0x7faf68197690 osd=7 3 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 15 client.41780.objecter _session_command_op_assign 7 16 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 4 2024-03-11T21:37:32.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 10 client.41780.objecter _send_command 16 2024-03-11T21:37:32.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.874+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf681be650 con 0x7faf681b2180 2024-03-11T21:37:32.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf64ff9700 1 -- 172.21.15.143:0/2682637761 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faf681be650 con 0x7faf681b2180 2024-03-11T21:37:32.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf64ff9700 10 client.41780.objecter ms_dispatch 0x7faf68001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:32.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf64ff9700 10 client.41780.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:32.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf64ff9700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 4 2024-03-11T21:37:32.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf64ff9700 15 client.41780.objecter _session_command_op_remove 7 16 2024-03-11T21:37:32.853 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 librados: watch_flush enter 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 librados: watch_flush exit 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.0 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7faf68062c40 msgr2=0x7faf6802c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7faf68062c40 0x7faf6802c6b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6806aab0 osd=0 1 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.1 2024-03-11T21:37:32.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7faf68062830 msgr2=0x7faf68196a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7faf68062830 0x7faf68196a80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf680626c0 osd=1 1 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.2 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7faf68198350 msgr2=0x7faf681082a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7faf68198350 0x7faf681082a0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf6802dcd0 osd=2 1 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.3 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7faf68107e90 msgr2=0x7faf681b1c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7faf68107e90 0x7faf681b1c40 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf67fff700 1 -- 172.21.15.143:0/2682637761 reap_dead start 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf67fff700 1 -- 172.21.15.143:0/2682637761 reap_dead start 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68107d20 osd=3 1 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.4 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7faf681af790 msgr2=0x7faf681b5310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7faf681af790 0x7faf681b5310 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681973c0 osd=4 1 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.5 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf681b2e60 msgr2=0x7faf681b89e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7faf681b2e60 0x7faf681b89e0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681aeab0 osd=5 1 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.6 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faf681b9430 msgr2=0x7faf681bb820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faf681b9430 0x7faf681bb820 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf681af0a0 osd=6 1 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter close_session for osd.7 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7faf681b2180 msgr2=0x7faf681bcee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7faf681b2180 0x7faf681bcee0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter put_session s=0x7faf68197690 osd=7 1 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 20 client.41780.objecter shutdown clearing up homeless session... 2024-03-11T21:37:32.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 10 client.41780.objecter successfully canceled tick 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf5006f820 msgr2=0x7faf50071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faf5006f820 0x7faf50071ce0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009680 tx=0x7faf5c009260 comp rx=0 tx=0).stop 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 msgr2=0x7faf68198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 --2- 172.21.15.143:0/2682637761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf68101000 0x7faf68198a30 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58003da0 comp rx=0 tx=0).stop 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf67fff700 1 -- 172.21.15.143:0/2682637761 reap_dead start 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf67fff700 1 -- 172.21.15.143:0/2682637761 reap_dead start 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 shutdown_connections 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 >> 172.21.15.143:0/2682637761 conn(0x7faf68077a40 msgr2=0x7faf680fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 shutdown_connections 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 -- 172.21.15.143:0/2682637761 wait complete. 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:32.878+0000 7faf6e0f0700 1 librados: shutdown 2024-03-11T21:37:32.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:32.858 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:32.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.477298+0000 mgr.y (mgr.14550) 574 : cluster 0 pgmap v892: 292 pgs: 2 creating+activating, 25 creating+peering, 24 unknown, 241 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.730366+0000 osd.7 (osd.7) 320 : cluster 0 3.13 scrub starts 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.732222+0000 osd.7 (osd.7) 321 : cluster 0 3.13 scrub ok 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.778307+0000 osd.6 (osd.6) 257 : cluster 0 39.19 scrub starts 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:31.780096+0000 osd.6 (osd.6) 258 : cluster 0 39.19 scrub ok 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: audit 2024-03-11T21:37:32.453419+0000 mon.a (mon.0) 3029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:32.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.458383+0000 mon.a (mon.0) 3030 : cluster 0 osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: audit 2024-03-11T21:37:32.465591+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: audit 2024-03-11T21:37:32.466529+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.468886+0000 osd.3 (osd.3) 347 : cluster 0 3.2 scrub starts 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.471462+0000 osd.3 (osd.3) 348 : cluster 0 3.2 scrub ok 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[29284]: audit 2024-03-11T21:37:32.693176+0000 mon.c (mon.2) 1061 : audit 0 from='client.? 172.21.15.143:0/2682637761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.477298+0000 mgr.y (mgr.14550) 574 : cluster 0 pgmap v892: 292 pgs: 2 creating+activating, 25 creating+peering, 24 unknown, 241 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.730366+0000 osd.7 (osd.7) 320 : cluster 0 3.13 scrub starts 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.732222+0000 osd.7 (osd.7) 321 : cluster 0 3.13 scrub ok 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.778307+0000 osd.6 (osd.6) 257 : cluster 0 39.19 scrub starts 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:31.780096+0000 osd.6 (osd.6) 258 : cluster 0 39.19 scrub ok 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: audit 2024-03-11T21:37:32.453419+0000 mon.a (mon.0) 3029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:32.956 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.458383+0000 mon.a (mon.0) 3030 : cluster 0 osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:37:32.957 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: audit 2024-03-11T21:37:32.465591+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:32.957 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: audit 2024-03-11T21:37:32.466529+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:32.957 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.468886+0000 osd.3 (osd.3) 347 : cluster 0 3.2 scrub starts 2024-03-11T21:37:32.957 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.471462+0000 osd.3 (osd.3) 348 : cluster 0 3.2 scrub ok 2024-03-11T21:37:32.957 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:32 smithi143 bash[21512]: audit 2024-03-11T21:37:32.693176+0000 mon.c (mon.2) 1061 : audit 0 from='client.? 172.21.15.143:0/2682637761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:32.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:33.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:33.084 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.477298+0000 mgr.y (mgr.14550) 574 : cluster 0 pgmap v892: 292 pgs: 2 creating+activating, 25 creating+peering, 24 unknown, 241 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-11T21:37:33.084 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.730366+0000 osd.7 (osd.7) 320 : cluster 0 3.13 scrub starts 2024-03-11T21:37:33.084 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.732222+0000 osd.7 (osd.7) 321 : cluster 0 3.13 scrub ok 2024-03-11T21:37:33.084 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.778307+0000 osd.6 (osd.6) 257 : cluster 0 39.19 scrub starts 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:31.780096+0000 osd.6 (osd.6) 258 : cluster 0 39.19 scrub ok 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: audit 2024-03-11T21:37:32.453419+0000 mon.a (mon.0) 3029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.458383+0000 mon.a (mon.0) 3030 : cluster 0 osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: audit 2024-03-11T21:37:32.465591+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: audit 2024-03-11T21:37:32.466529+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.468886+0000 osd.3 (osd.3) 347 : cluster 0 3.2 scrub starts 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.471462+0000 osd.3 (osd.3) 348 : cluster 0 3.2 scrub ok 2024-03-11T21:37:33.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:32 smithi203 bash[23654]: audit 2024-03-11T21:37:32.693176+0000 mon.c (mon.2) 1061 : audit 0 from='client.? 172.21.15.143:0/2682637761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:33.092 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:33.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:33.222 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:33.233 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:33.233 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:33.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:33.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:33.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:33.476 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:33.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:33.605 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:33.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:33.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:33.867 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:33.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:33.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:34.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:37:34.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.714600+0000 osd.5 (osd.5) 341 : cluster 0 3.1d scrub starts 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.718695+0000 osd.5 (osd.5) 342 : cluster 0 3.1d scrub ok 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.736628+0000 osd.7 (osd.7) 322 : cluster 0 2.b scrub starts 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.738950+0000 osd.7 (osd.7) 323 : cluster 0 2.b scrub ok 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:33.217660+0000 osd.4 (osd.4) 315 : cluster 0 39.15 scrub starts 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:33.219148+0000 osd.4 (osd.4) 316 : cluster 0 39.15 scrub ok 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: audit 2024-03-11T21:37:33.456247+0000 mon.a (mon.0) 3032 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:34.112 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[21512]: cluster 2024-03-11T21:37:33.464801+0000 mon.a (mon.0) 3033 : cluster 0 osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.714600+0000 osd.5 (osd.5) 341 : cluster 0 3.1d scrub starts 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.718695+0000 osd.5 (osd.5) 342 : cluster 0 3.1d scrub ok 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.736628+0000 osd.7 (osd.7) 322 : cluster 0 2.b scrub starts 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.738950+0000 osd.7 (osd.7) 323 : cluster 0 2.b scrub ok 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:33.217660+0000 osd.4 (osd.4) 315 : cluster 0 39.15 scrub starts 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:33.219148+0000 osd.4 (osd.4) 316 : cluster 0 39.15 scrub ok 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: audit 2024-03-11T21:37:33.456247+0000 mon.a (mon.0) 3032 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:34.113 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:33 smithi143 bash[29284]: cluster 2024-03-11T21:37:33.464801+0000 mon.a (mon.0) 3033 : cluster 0 osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:37:34.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:34.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:34.244 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:34.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:34.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.714600+0000 osd.5 (osd.5) 341 : cluster 0 3.1d scrub starts 2024-03-11T21:37:34.272 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.718695+0000 osd.5 (osd.5) 342 : cluster 0 3.1d scrub ok 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.736628+0000 osd.7 (osd.7) 322 : cluster 0 2.b scrub starts 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.738950+0000 osd.7 (osd.7) 323 : cluster 0 2.b scrub ok 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:33.217660+0000 osd.4 (osd.4) 315 : cluster 0 39.15 scrub starts 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:33.219148+0000 osd.4 (osd.4) 316 : cluster 0 39.15 scrub ok 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: audit 2024-03-11T21:37:33.456247+0000 mon.a (mon.0) 3032 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:37:34.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:33 smithi203 bash[23654]: cluster 2024-03-11T21:37:33.464801+0000 mon.a (mon.0) 3033 : cluster 0 osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:37:34.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:34.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:34.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:34.513 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:34.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:34.642 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:34.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:34.770 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:34.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:35.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:35.032 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:35.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.975846+0000 osd.0 (osd.0) 328 : cluster 0 39.14 scrub starts 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:32.977357+0000 osd.0 (osd.0) 329 : cluster 0 39.14 scrub ok 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:33.478358+0000 mgr.y (mgr.14550) 575 : cluster 0 pgmap v895: 292 pgs: 2 creating+activating, 2 unknown, 288 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:34.269226+0000 osd.4 (osd.4) 317 : cluster 0 5.e scrub starts 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:34.271624+0000 osd.4 (osd.4) 318 : cluster 0 5.e scrub ok 2024-03-11T21:37:35.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: cluster 2024-03-11T21:37:34.463561+0000 mon.a (mon.0) 3034 : cluster 0 osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: audit 2024-03-11T21:37:34.574813+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: audit 2024-03-11T21:37:34.575736+0000 mon.a (mon.0) 3035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: audit 2024-03-11T21:37:34.577134+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[21512]: audit 2024-03-11T21:37:34.577809+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.975846+0000 osd.0 (osd.0) 328 : cluster 0 39.14 scrub starts 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:32.977357+0000 osd.0 (osd.0) 329 : cluster 0 39.14 scrub ok 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:33.478358+0000 mgr.y (mgr.14550) 575 : cluster 0 pgmap v895: 292 pgs: 2 creating+activating, 2 unknown, 288 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:34.269226+0000 osd.4 (osd.4) 317 : cluster 0 5.e scrub starts 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:34.271624+0000 osd.4 (osd.4) 318 : cluster 0 5.e scrub ok 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: cluster 2024-03-11T21:37:34.463561+0000 mon.a (mon.0) 3034 : cluster 0 osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: audit 2024-03-11T21:37:34.574813+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: audit 2024-03-11T21:37:34.575736+0000 mon.a (mon.0) 3035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: audit 2024-03-11T21:37:34.577134+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:34 smithi143 bash[29284]: audit 2024-03-11T21:37:34.577809+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.161 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:35.262 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:35.264 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.975846+0000 osd.0 (osd.0) 328 : cluster 0 39.14 scrub starts 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:32.977357+0000 osd.0 (osd.0) 329 : cluster 0 39.14 scrub ok 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:33.478358+0000 mgr.y (mgr.14550) 575 : cluster 0 pgmap v895: 292 pgs: 2 creating+activating, 2 unknown, 288 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:34.269226+0000 osd.4 (osd.4) 317 : cluster 0 5.e scrub starts 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:34.271624+0000 osd.4 (osd.4) 318 : cluster 0 5.e scrub ok 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: cluster 2024-03-11T21:37:34.463561+0000 mon.a (mon.0) 3034 : cluster 0 osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:37:35.315 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: audit 2024-03-11T21:37:34.574813+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: audit 2024-03-11T21:37:34.575736+0000 mon.a (mon.0) 3035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:35.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: audit 2024-03-11T21:37:34.577134+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.316 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:34 smithi203 bash[23654]: audit 2024-03-11T21:37:34.577809+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-87"}]: dispatch 2024-03-11T21:37:35.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:35.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:35.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:35.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:35.607 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:35.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:35.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:35.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:37:35.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:35.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:36.017 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:36.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:36.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:36.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.413 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:36.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.544 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:36.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:36.674 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:34.987092+0000 osd.0 (osd.0) 330 : cluster 0 9.b scrub starts 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:34.989705+0000 osd.0 (osd.0) 331 : cluster 0 9.b scrub ok 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.227950+0000 osd.4 (osd.4) 319 : cluster 0 3.9 scrub starts 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.232320+0000 osd.4 (osd.4) 320 : cluster 0 3.9 scrub ok 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.477839+0000 mon.a (mon.0) 3037 : cluster 0 osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.479447+0000 mgr.y (mgr.14550) 576 : cluster 0 pgmap v898: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.751942+0000 osd.7 (osd.7) 324 : cluster 0 6.3 scrub starts 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:35.753986+0000 osd.7 (osd.7) 325 : cluster 0 6.3 scrub ok 2024-03-11T21:37:36.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: cluster 2024-03-11T21:37:36.427027+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: audit 2024-03-11T21:37:36.440176+0000 mon.c (mon.2) 1064 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[21512]: audit 2024-03-11T21:37:36.440937+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:34.987092+0000 osd.0 (osd.0) 330 : cluster 0 9.b scrub starts 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:34.989705+0000 osd.0 (osd.0) 331 : cluster 0 9.b scrub ok 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.227950+0000 osd.4 (osd.4) 319 : cluster 0 3.9 scrub starts 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.232320+0000 osd.4 (osd.4) 320 : cluster 0 3.9 scrub ok 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.477839+0000 mon.a (mon.0) 3037 : cluster 0 osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.479447+0000 mgr.y (mgr.14550) 576 : cluster 0 pgmap v898: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.751942+0000 osd.7 (osd.7) 324 : cluster 0 6.3 scrub starts 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:35.753986+0000 osd.7 (osd.7) 325 : cluster 0 6.3 scrub ok 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: cluster 2024-03-11T21:37:36.427027+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: audit 2024-03-11T21:37:36.440176+0000 mon.c (mon.2) 1064 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:36 smithi143 bash[29284]: audit 2024-03-11T21:37:36.440937+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:34.987092+0000 osd.0 (osd.0) 330 : cluster 0 9.b scrub starts 2024-03-11T21:37:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:34.989705+0000 osd.0 (osd.0) 331 : cluster 0 9.b scrub ok 2024-03-11T21:37:36.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.227950+0000 osd.4 (osd.4) 319 : cluster 0 3.9 scrub starts 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.232320+0000 osd.4 (osd.4) 320 : cluster 0 3.9 scrub ok 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.477839+0000 mon.a (mon.0) 3037 : cluster 0 osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.479447+0000 mgr.y (mgr.14550) 576 : cluster 0 pgmap v898: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.751942+0000 osd.7 (osd.7) 324 : cluster 0 6.3 scrub starts 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:35.753986+0000 osd.7 (osd.7) 325 : cluster 0 6.3 scrub ok 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: cluster 2024-03-11T21:37:36.427027+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: audit 2024-03-11T21:37:36.440176+0000 mon.c (mon.2) 1064 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:36 smithi203 bash[23654]: audit 2024-03-11T21:37:36.440937+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:36.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:36.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.928 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:37.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:37.055 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:37.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:37.179 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:37.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:37.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:37.437 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:37.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:37.552 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:36.471443+0000 mon.a (mon.0) 3040 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:36.778420+0000 osd.7 (osd.7) 326 : cluster 0 9.11 scrub starts 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:36.780029+0000 osd.7 (osd.7) 327 : cluster 0 9.11 scrub ok 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: audit 2024-03-11T21:37:36.785868+0000 mon.c (mon.2) 1065 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.069437+0000 osd.1 (osd.1) 137 : cluster 0 39.2 scrub starts 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.071163+0000 osd.1 (osd.1) 138 : cluster 0 39.2 scrub ok 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: audit 2024-03-11T21:37:37.420335+0000 mon.a (mon.0) 3041 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.427319+0000 mon.a (mon.0) 3042 : cluster 0 osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: audit 2024-03-11T21:37:37.432185+0000 mon.c (mon.2) 1066 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: audit 2024-03-11T21:37:37.438226+0000 mon.c (mon.2) 1067 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[21512]: audit 2024-03-11T21:37:37.438957+0000 mon.a (mon.0) 3043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:36.471443+0000 mon.a (mon.0) 3040 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:37.553 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:36.778420+0000 osd.7 (osd.7) 326 : cluster 0 9.11 scrub starts 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:36.780029+0000 osd.7 (osd.7) 327 : cluster 0 9.11 scrub ok 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: audit 2024-03-11T21:37:36.785868+0000 mon.c (mon.2) 1065 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.069437+0000 osd.1 (osd.1) 137 : cluster 0 39.2 scrub starts 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.071163+0000 osd.1 (osd.1) 138 : cluster 0 39.2 scrub ok 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: audit 2024-03-11T21:37:37.420335+0000 mon.a (mon.0) 3041 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.427319+0000 mon.a (mon.0) 3042 : cluster 0 osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: audit 2024-03-11T21:37:37.432185+0000 mon.c (mon.2) 1066 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: audit 2024-03-11T21:37:37.438226+0000 mon.c (mon.2) 1067 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.554 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:37 smithi143 bash[29284]: audit 2024-03-11T21:37:37.438957+0000 mon.a (mon.0) 3043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.568 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:37.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:37.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:36.471443+0000 mon.a (mon.0) 3040 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:36.778420+0000 osd.7 (osd.7) 326 : cluster 0 9.11 scrub starts 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:36.780029+0000 osd.7 (osd.7) 327 : cluster 0 9.11 scrub ok 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: audit 2024-03-11T21:37:36.785868+0000 mon.c (mon.2) 1065 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.069437+0000 osd.1 (osd.1) 137 : cluster 0 39.2 scrub starts 2024-03-11T21:37:37.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.071163+0000 osd.1 (osd.1) 138 : cluster 0 39.2 scrub ok 2024-03-11T21:37:37.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: audit 2024-03-11T21:37:37.420335+0000 mon.a (mon.0) 3041 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:37.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.427319+0000 mon.a (mon.0) 3042 : cluster 0 osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:37:37.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: audit 2024-03-11T21:37:37.432185+0000 mon.c (mon.2) 1066 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:37.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: audit 2024-03-11T21:37:37.438226+0000 mon.c (mon.2) 1067 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:37 smithi203 bash[23654]: audit 2024-03-11T21:37:37.438957+0000 mon.a (mon.0) 3043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:37:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:37.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:37.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:38.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:38.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.222 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:38.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:38.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:38.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:38.479 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:38.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.047044+0000 osd.0 (osd.0) 332 : cluster 0 9.1c scrub starts 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.049603+0000 osd.0 (osd.0) 333 : cluster 0 9.1c scrub ok 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.154582+0000 osd.4 (osd.4) 321 : cluster 0 3.14 scrub starts 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.158645+0000 osd.4 (osd.4) 322 : cluster 0 3.14 scrub ok 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.480188+0000 mgr.y (mgr.14550) 577 : cluster 0 pgmap v901: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: audit 2024-03-11T21:37:38.422358+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:38.594 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[29284]: cluster 2024-03-11T21:37:38.427292+0000 mon.a (mon.0) 3045 : cluster 0 osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.047044+0000 osd.0 (osd.0) 332 : cluster 0 9.1c scrub starts 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.049603+0000 osd.0 (osd.0) 333 : cluster 0 9.1c scrub ok 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.154582+0000 osd.4 (osd.4) 321 : cluster 0 3.14 scrub starts 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.158645+0000 osd.4 (osd.4) 322 : cluster 0 3.14 scrub ok 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.480188+0000 mgr.y (mgr.14550) 577 : cluster 0 pgmap v901: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: audit 2024-03-11T21:37:38.422358+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:38.595 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:38 smithi143 bash[21512]: cluster 2024-03-11T21:37:38.427292+0000 mon.a (mon.0) 3045 : cluster 0 osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:37:38.608 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:38.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.047044+0000 osd.0 (osd.0) 332 : cluster 0 9.1c scrub starts 2024-03-11T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.049603+0000 osd.0 (osd.0) 333 : cluster 0 9.1c scrub ok 2024-03-11T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.154582+0000 osd.4 (osd.4) 321 : cluster 0 3.14 scrub starts 2024-03-11T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.158645+0000 osd.4 (osd.4) 322 : cluster 0 3.14 scrub ok 2024-03-11T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.480188+0000 mgr.y (mgr.14550) 577 : cluster 0 pgmap v901: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T21:37:38.724 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: audit 2024-03-11T21:37:38.422358+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:37:38.724 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[23654]: cluster 2024-03-11T21:37:38.427292+0000 mon.a (mon.0) 3045 : cluster 0 osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:37:38.739 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:38.787 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:38.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:38.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:38.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:39.022 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:39.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:39.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:39.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:39.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:39.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:39.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:39.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:39.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:39.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:39 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.609629+0000 osd.5 (osd.5) 343 : cluster 0 118.8 scrub starts 2024-03-11T21:37:39.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:39 smithi203 bash[23654]: cluster 2024-03-11T21:37:37.613666+0000 osd.5 (osd.5) 344 : cluster 0 118.8 scrub ok 2024-03-11T21:37:39.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:39 smithi203 bash[23654]: cluster 2024-03-11T21:37:38.782742+0000 osd.7 (osd.7) 328 : cluster 0 9.e scrub starts 2024-03-11T21:37:39.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:39 smithi203 bash[23654]: cluster 2024-03-11T21:37:38.784316+0000 osd.7 (osd.7) 329 : cluster 0 9.e scrub ok 2024-03-11T21:37:39.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:39 smithi203 bash[23654]: audit 2024-03-11T21:37:38.793730+0000 mgr.y (mgr.14550) 578 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:39.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:39.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.609629+0000 osd.5 (osd.5) 343 : cluster 0 118.8 scrub starts 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[29284]: cluster 2024-03-11T21:37:37.613666+0000 osd.5 (osd.5) 344 : cluster 0 118.8 scrub ok 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[29284]: cluster 2024-03-11T21:37:38.782742+0000 osd.7 (osd.7) 328 : cluster 0 9.e scrub starts 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[29284]: cluster 2024-03-11T21:37:38.784316+0000 osd.7 (osd.7) 329 : cluster 0 9.e scrub ok 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[29284]: audit 2024-03-11T21:37:38.793730+0000 mgr.y (mgr.14550) 578 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:39.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.609629+0000 osd.5 (osd.5) 343 : cluster 0 118.8 scrub starts 2024-03-11T21:37:39.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[21512]: cluster 2024-03-11T21:37:37.613666+0000 osd.5 (osd.5) 344 : cluster 0 118.8 scrub ok 2024-03-11T21:37:39.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[21512]: cluster 2024-03-11T21:37:38.782742+0000 osd.7 (osd.7) 328 : cluster 0 9.e scrub starts 2024-03-11T21:37:39.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[21512]: cluster 2024-03-11T21:37:38.784316+0000 osd.7 (osd.7) 329 : cluster 0 9.e scrub ok 2024-03-11T21:37:39.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:39 smithi143 bash[21512]: audit 2024-03-11T21:37:38.793730+0000 mgr.y (mgr.14550) 578 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:39.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:40.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:40.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:40.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:40.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.296 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:40.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:40.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:40.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:40.569 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:40.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:40.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:38.660420+0000 osd.5 (osd.5) 345 : cluster 0 9.f scrub starts 2024-03-11T21:37:40.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:38.663288+0000 osd.5 (osd.5) 346 : cluster 0 9.f scrub ok 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.481253+0000 mgr.y (mgr.14550) 579 : cluster 0 pgmap v903: 260 pgs: 2 unknown, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.502808+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.784095+0000 osd.7 (osd.7) 330 : cluster 0 9.17 scrub starts 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.787455+0000 osd.7 (osd.7) 331 : cluster 0 9.17 scrub ok 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.818886+0000 osd.2 (osd.2) 295 : cluster 0 2.c scrub starts 2024-03-11T21:37:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:40 smithi203 bash[23654]: cluster 2024-03-11T21:37:39.820423+0000 osd.2 (osd.2) 296 : cluster 0 2.c scrub ok 2024-03-11T21:37:40.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:40.765 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:38.660420+0000 osd.5 (osd.5) 345 : cluster 0 9.f scrub starts 2024-03-11T21:37:40.765 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:38.663288+0000 osd.5 (osd.5) 346 : cluster 0 9.f scrub ok 2024-03-11T21:37:40.765 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.481253+0000 mgr.y (mgr.14550) 579 : cluster 0 pgmap v903: 260 pgs: 2 unknown, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:37:40.765 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.502808+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.784095+0000 osd.7 (osd.7) 330 : cluster 0 9.17 scrub starts 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.787455+0000 osd.7 (osd.7) 331 : cluster 0 9.17 scrub ok 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.818886+0000 osd.2 (osd.2) 295 : cluster 0 2.c scrub starts 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[29284]: cluster 2024-03-11T21:37:39.820423+0000 osd.2 (osd.2) 296 : cluster 0 2.c scrub ok 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:38.660420+0000 osd.5 (osd.5) 345 : cluster 0 9.f scrub starts 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:38.663288+0000 osd.5 (osd.5) 346 : cluster 0 9.f scrub ok 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.481253+0000 mgr.y (mgr.14550) 579 : cluster 0 pgmap v903: 260 pgs: 2 unknown, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.502808+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.784095+0000 osd.7 (osd.7) 330 : cluster 0 9.17 scrub starts 2024-03-11T21:37:40.766 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.787455+0000 osd.7 (osd.7) 331 : cluster 0 9.17 scrub ok 2024-03-11T21:37:40.767 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.818886+0000 osd.2 (osd.2) 295 : cluster 0 2.c scrub starts 2024-03-11T21:37:40.767 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:40 smithi143 bash[21512]: cluster 2024-03-11T21:37:39.820423+0000 osd.2 (osd.2) 296 : cluster 0 2.c scrub ok 2024-03-11T21:37:40.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:40.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:40.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:41.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:41.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:41.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:41.218 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:41.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:41.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:41.479 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:41.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:40.513984+0000 mon.a (mon.0) 3047 : cluster 0 osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:40.559573+0000 osd.3 (osd.3) 349 : cluster 0 2.1e scrub starts 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:40.561992+0000 osd.3 (osd.3) 350 : cluster 0 2.1e scrub ok 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: audit 2024-03-11T21:37:40.589288+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: audit 2024-03-11T21:37:40.589983+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: audit 2024-03-11T21:37:40.591139+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: audit 2024-03-11T21:37:40.591732+0000 mon.a (mon.0) 3049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:40.802820+0000 osd.2 (osd.2) 297 : cluster 0 118.17 scrub starts 2024-03-11T21:37:41.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:40.805250+0000 osd.2 (osd.2) 298 : cluster 0 118.17 scrub ok 2024-03-11T21:37:41.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.045388+0000 osd.1 (osd.1) 139 : cluster 0 2.d scrub starts 2024-03-11T21:37:41.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:41 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.046962+0000 osd.1 (osd.1) 140 : cluster 0 2.d scrub ok 2024-03-11T21:37:41.609 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:41.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:41.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.865 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:41.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:40.513984+0000 mon.a (mon.0) 3047 : cluster 0 osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:40.559573+0000 osd.3 (osd.3) 349 : cluster 0 2.1e scrub starts 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:40.561992+0000 osd.3 (osd.3) 350 : cluster 0 2.1e scrub ok 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: audit 2024-03-11T21:37:40.589288+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: audit 2024-03-11T21:37:40.589983+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: audit 2024-03-11T21:37:40.591139+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: audit 2024-03-11T21:37:40.591732+0000 mon.a (mon.0) 3049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:40.802820+0000 osd.2 (osd.2) 297 : cluster 0 118.17 scrub starts 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:40.805250+0000 osd.2 (osd.2) 298 : cluster 0 118.17 scrub ok 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.045388+0000 osd.1 (osd.1) 139 : cluster 0 2.d scrub starts 2024-03-11T21:37:41.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.046962+0000 osd.1 (osd.1) 140 : cluster 0 2.d scrub ok 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:40.513984+0000 mon.a (mon.0) 3047 : cluster 0 osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:40.559573+0000 osd.3 (osd.3) 349 : cluster 0 2.1e scrub starts 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:40.561992+0000 osd.3 (osd.3) 350 : cluster 0 2.1e scrub ok 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: audit 2024-03-11T21:37:40.589288+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: audit 2024-03-11T21:37:40.589983+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: audit 2024-03-11T21:37:40.591139+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: audit 2024-03-11T21:37:40.591732+0000 mon.a (mon.0) 3049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-90"}]: dispatch 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:40.802820+0000 osd.2 (osd.2) 297 : cluster 0 118.17 scrub starts 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:40.805250+0000 osd.2 (osd.2) 298 : cluster 0 118.17 scrub ok 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.045388+0000 osd.1 (osd.1) 139 : cluster 0 2.d scrub starts 2024-03-11T21:37:41.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:41 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.046962+0000 osd.1 (osd.1) 140 : cluster 0 2.d scrub ok 2024-03-11T21:37:41.939 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:41.939 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:41.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.987 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:42.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:42.117 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:42.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:42.247 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:42.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:42.361 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:42.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:42.511 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:42.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:42.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:42.685 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.482183+0000 mgr.y (mgr.14550) 580 : cluster 0 pgmap v906: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 252 B/s wr, 2 op/s 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.521078+0000 mon.a (mon.0) 3050 : cluster 0 osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.782952+0000 osd.2 (osd.2) 299 : cluster 0 6.f scrub starts 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.786838+0000 osd.2 (osd.2) 300 : cluster 0 6.f scrub ok 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.902313+0000 osd.0 (osd.0) 334 : cluster 0 2.13 scrub starts 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[29284]: cluster 2024-03-11T21:37:41.904477+0000 osd.0 (osd.0) 335 : cluster 0 2.13 scrub ok 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.482183+0000 mgr.y (mgr.14550) 580 : cluster 0 pgmap v906: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 252 B/s wr, 2 op/s 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.521078+0000 mon.a (mon.0) 3050 : cluster 0 osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.782952+0000 osd.2 (osd.2) 299 : cluster 0 6.f scrub starts 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.786838+0000 osd.2 (osd.2) 300 : cluster 0 6.f scrub ok 2024-03-11T21:37:42.686 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.902313+0000 osd.0 (osd.0) 334 : cluster 0 2.13 scrub starts 2024-03-11T21:37:42.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:42 smithi143 bash[21512]: cluster 2024-03-11T21:37:41.904477+0000 osd.0 (osd.0) 335 : cluster 0 2.13 scrub ok 2024-03-11T21:37:42.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/1590449138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4104c20 msgr2=0x7fc3e4108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/1590449138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4104c20 0x7fc3e4108c70 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80099c0 tx=0x7fc3d8009cd0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/1590449138 shutdown_connections 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/1590449138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4109350 0x7fc3e4110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/1590449138 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104270 0x7fc3e4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/1590449138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4104c20 0x7fc3e4108c70 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/1590449138 >> 172.21.15.143:0/1590449138 conn(0x7fc3e40ffb00 msgr2=0x7fc3e4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/1590449138 shutdown_connections 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/1590449138 wait complete. 2024-03-11T21:37:42.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 librados: starting msgr at 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 librados: starting objecter 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 Processor -- start 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 -- start start 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 librados: setting wanted keys 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.710+0000 7fc3ea7cf700 1 librados: calling monclient init 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4104270 0x7fc3e4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 0x7fc3e419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4113810 con 0x7fc3e4109350 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4113690 con 0x7fc3e4104270 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4113990 con 0x7fc3e4104c20 2024-03-11T21:37:42.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 0x7fc3e419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41046/0 (socket says 172.21.15.143:41046) 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 -- 172.21.15.143:0/81785983 learned_addr learned my addr 172.21.15.143:0/81785983 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e3fff700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4104270 0x7fc3e4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 -- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 msgr2=0x7fc3e419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 0x7fc3e419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 -- 172.21.15.143:0/81785983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4104270 msgr2=0x7fc3e4199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4104270 0x7fc3e4199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 -- 172.21.15.143:0/81785983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e419edb0 con 0x7fc3e4109350 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e37fe700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 0x7fc3e419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e8d6c700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc007ee0 tx=0x7fc3dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 10 client.?.objecter ms_handle_connect 0x7fc3e4109350 2024-03-11T21:37:42.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00dbd0 con 0x7fc3e4109350 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e419f0a0 con 0x7fc3e4109350 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc3dc00dd70 con 0x7fc3e4109350 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc005840 con 0x7fc3e4109350 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 10 client.30585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e41a6920 con 0x7fc3e4109350 2024-03-11T21:37:42.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 librados: init done 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e4105750 con 0x7fc3e4109350 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc3dc01a020 con 0x7fc3e4109350 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 0x7fc3cc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(590..590 src has 87..590) v4 ==== 9616+0+0 (secure 0 0 0) 0x7fc3dc08e8d0 con 0x7fc3e4109350 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 10 client.30585.objecter ms_dispatch 0x7fc3e4001550 osd_map(590..590 src has 87..590) v4 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 3 client.30585.objecter handle_osd_map got epochs [590,590] > 0 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 3 client.30585.objecter handle_osd_map decoding full epoch 590 2024-03-11T21:37:42.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e3fff700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 0x7fc3cc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 20 client.30585.objecter dump_active .. 0 homeless 2024-03-11T21:37:42.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e17fa700 10 client.30585.objecter ms_handle_connect 0x7fc3cc06f820 2024-03-11T21:37:42.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.714+0000 7fc3e3fff700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 0x7fc3cc071ce0 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4005950 tx=0x7fc3d40058e0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:42.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.718+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc3dc010070 con 0x7fc3e4109350 2024-03-11T21:37:42.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:37:42.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:42.768 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.482183+0000 mgr.y (mgr.14550) 580 : cluster 0 pgmap v906: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 252 B/s wr, 2 op/s 2024-03-11T21:37:42.772 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.521078+0000 mon.a (mon.0) 3050 : cluster 0 osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:37:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.782952+0000 osd.2 (osd.2) 299 : cluster 0 6.f scrub starts 2024-03-11T21:37:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.786838+0000 osd.2 (osd.2) 300 : cluster 0 6.f scrub ok 2024-03-11T21:37:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.902313+0000 osd.0 (osd.0) 334 : cluster 0 2.13 scrub starts 2024-03-11T21:37:42.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:42 smithi203 bash[23654]: cluster 2024-03-11T21:37:41.904477+0000 osd.0 (osd.0) 335 : cluster 0 2.13 scrub ok 2024-03-11T21:37:42.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:42.890 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- 172.21.15.143:0/4019861860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 msgr2=0x7f61101059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- 172.21.15.143:0/4019861860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f61101059c0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f61040099c0 tx=0x7f6104009cd0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- 172.21.15.143:0/4019861860 shutdown_connections 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- 172.21.15.143:0/4019861860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61101060a0 0x7f611010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- 172.21.15.143:0/4019861860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6110100fc0 0x7f61101013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- 172.21.15.143:0/4019861860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f61101059c0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- 172.21.15.143:0/4019861860 >> 172.21.15.143:0/4019861860 conn(0x7f6110077de0 msgr2=0x7f61100781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- 172.21.15.143:0/4019861860 shutdown_connections 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- 172.21.15.143:0/4019861860 wait complete. 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 librados: starting msgr at 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 librados: starting objecter 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 Processor -- start 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- start start 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 librados: setting wanted keys 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 librados: calling monclient init 2024-03-11T21:37:42.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6110100fc0 0x7f611019d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6110110560 con 0x7f6110101970 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101103e0 con 0x7f6110100fc0 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.930+0000 7f6117f4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61101106e0 con 0x7f61101060a0 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61164e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61164e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:45518/0 (socket says 172.21.15.143:45518) 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41064/0 (socket says 172.21.15.143:41064) 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61164e7700 1 -- 172.21.15.143:0/1750500677 learned_addr learned my addr 172.21.15.143:0/1750500677 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:42.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6115ce6700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6110100fc0 0x7f611019d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 -- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 msgr2=0x7f6110197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 -- 172.21.15.143:0/1750500677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6110100fc0 msgr2=0x7f611019d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6110100fc0 0x7f611019d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 -- 172.21.15.143:0/1750500677 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61101981f0 con 0x7f6110101970 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61164e7700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f61154e5700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f611006b6e0 tx=0x7f610400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 10 client.?.objecter ms_handle_connect 0x7f6110101970 2024-03-11T21:37:42.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104009e60 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f6104005550 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6110198480 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6117f4a700 10 client.30591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400bea0 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61101a47a0 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6117f4a700 1 librados: init done 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f61040050d0 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61101024a0 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 0x7f60f8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(590..590 src has 87..590) v4 ==== 9616+0+0 (secure 0 0 0) 0x7f610408f8d0 con 0x7f6110101970 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 10 client.30591.objecter ms_dispatch 0x7f6110001550 osd_map(590..590 src has 87..590) v4 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 3 client.30591.objecter handle_osd_map got epochs [590,590] > 0 2024-03-11T21:37:42.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 3 client.30591.objecter handle_osd_map decoding full epoch 590 2024-03-11T21:37:42.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6115ce6700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 0x7f60f8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:42.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 20 client.30591.objecter dump_active .. 0 homeless 2024-03-11T21:37:42.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f610effd700 10 client.30591.objecter ms_handle_connect 0x7f60f806f6f0 2024-03-11T21:37:42.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.934+0000 7f6115ce6700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 0x7f60f8071bb0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7f6110198bd0 tx=0x7f610000b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:42.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.942+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6104013070 con 0x7f6110101970 2024-03-11T21:37:42.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:42.954+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc3e419b100 con 0x7fc3e4109350 2024-03-11T21:37:42.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:43.018 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:43.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.150+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) v1 -- 0x7f611004faa0 con 0x7f6110101970 2024-03-11T21:37:43.145 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:43.247 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:43.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:43.400 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:43.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:43.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.538+0000 7fc3e17fa700 1 -- 172.21.15.143:0/81785983 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v591) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc3dc05b5b0 con 0x7fc3e4109350 2024-03-11T21:37:43.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.538+0000 7f610effd700 1 -- 172.21.15.143:0/1750500677 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v591) v1 ==== 129+0+0 (secure 0 0 0) 0x7f610405c530 con 0x7f6110101970 2024-03-11T21:37:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 10 librados: watch_flush enter 2024-03-11T21:37:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 10 librados: watch_flush exit 2024-03-11T21:37:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 20 client.30585.objecter shutdown clearing up homeless session... 2024-03-11T21:37:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 10 client.30585.objecter successfully canceled tick 2024-03-11T21:37:43.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 msgr2=0x7fc3cc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 0x7fc3cc071ce0 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4005950 tx=0x7fc3d40058e0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 msgr2=0x7fc3e419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc007ee0 tx=0x7fc3dc00d3b0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 shutdown_connections 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc3cc06f820 0x7fc3cc071ce0 secure :-1 s=CLOSED pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4005950 tx=0x7fc3d40058e0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4109350 0x7fc3e419e670 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4104270 0x7fc3e4199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 --2- 172.21.15.143:0/81785983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4104c20 0x7fc3e419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.542+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 >> 172.21.15.143:0/81785983 conn(0x7fc3e40ffb00 msgr2=0x7fc3e41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:43.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.546+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 shutdown_connections 2024-03-11T21:37:43.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7fc3ea7cf700 1 -- 172.21.15.143:0/81785983 wait complete. 2024-03-11T21:37:43.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7fc3ea7cf700 1 librados: shutdown 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.546+0000 7f6117f4a700 10 librados: watch_flush enter 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.546+0000 7f6117f4a700 10 librados: watch_flush exit 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.546+0000 7f6117f4a700 20 client.30591.objecter shutdown clearing up homeless session... 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.546+0000 7f6117f4a700 10 client.30591.objecter successfully canceled tick 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 msgr2=0x7f60f8071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 0x7f60f8071bb0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7f6110198bd0 tx=0x7f610000b040 comp rx=0 tx=0).stop 2024-03-11T21:37:43.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 msgr2=0x7f611019de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:43.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f611006b6e0 tx=0x7f610400fa00 comp rx=0 tx=0).stop 2024-03-11T21:37:43.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 shutdown_connections 2024-03-11T21:37:43.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f60f806f6f0 0x7f60f8071bb0 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61101060a0 0x7f6110197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6110100fc0 0x7f611019d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 --2- 172.21.15.143:0/1750500677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6110101970 0x7f611019de10 secure :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f611006b6e0 tx=0x7f610400fa00 comp rx=0 tx=0).stop 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 >> 172.21.15.143:0/1750500677 conn(0x7f6110077de0 msgr2=0x7f611006a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.550+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 shutdown_connections 2024-03-11T21:37:43.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.554+0000 7f6117f4a700 1 -- 172.21.15.143:0/1750500677 wait complete. 2024-03-11T21:37:43.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:43.554+0000 7f6117f4a700 1 librados: shutdown 2024-03-11T21:37:43.536 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pg_num to 11 2024-03-11T21:37:43.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:43.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.541795+0000 mon.a (mon.0) 3051 : cluster 0 osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: audit 2024-03-11T21:37:42.548405+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: audit 2024-03-11T21:37:42.550594+0000 mon.a (mon.0) 3052 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.627944+0000 osd.3 (osd.3) 351 : cluster 0 6.13 scrub starts 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.629412+0000 osd.3 (osd.3) 352 : cluster 0 6.13 scrub ok 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.677999+0000 osd.5 (osd.5) 347 : cluster 0 6.d scrub starts 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.679447+0000 osd.5 (osd.5) 348 : cluster 0 6.d scrub ok 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.760648+0000 osd.2 (osd.2) 301 : cluster 0 2.1 scrub starts 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.761601+0000 osd.2 (osd.2) 302 : cluster 0 2.1 scrub ok 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.891639+0000 osd.0 (osd.0) 336 : cluster 0 6.18 scrub starts 2024-03-11T21:37:43.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:42.892882+0000 osd.0 (osd.0) 337 : cluster 0 6.18 scrub ok 2024-03-11T21:37:43.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: audit 2024-03-11T21:37:42.958819+0000 mon.a (mon.0) 3053 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:43.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.025336+0000 osd.1 (osd.1) 141 : cluster 0 5.12 scrub starts 2024-03-11T21:37:43.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.026598+0000 osd.1 (osd.1) 142 : cluster 0 5.12 scrub ok 2024-03-11T21:37:43.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:43 smithi203 bash[23654]: audit 2024-03-11T21:37:43.156148+0000 mon.a (mon.0) 3054 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:37:43.671 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:43.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:43.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.541795+0000 mon.a (mon.0) 3051 : cluster 0 osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: audit 2024-03-11T21:37:42.548405+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: audit 2024-03-11T21:37:42.550594+0000 mon.a (mon.0) 3052 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.627944+0000 osd.3 (osd.3) 351 : cluster 0 6.13 scrub starts 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.629412+0000 osd.3 (osd.3) 352 : cluster 0 6.13 scrub ok 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.677999+0000 osd.5 (osd.5) 347 : cluster 0 6.d scrub starts 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.679447+0000 osd.5 (osd.5) 348 : cluster 0 6.d scrub ok 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.760648+0000 osd.2 (osd.2) 301 : cluster 0 2.1 scrub starts 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.761601+0000 osd.2 (osd.2) 302 : cluster 0 2.1 scrub ok 2024-03-11T21:37:43.866 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.891639+0000 osd.0 (osd.0) 336 : cluster 0 6.18 scrub starts 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:42.892882+0000 osd.0 (osd.0) 337 : cluster 0 6.18 scrub ok 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: audit 2024-03-11T21:37:42.958819+0000 mon.a (mon.0) 3053 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.025336+0000 osd.1 (osd.1) 141 : cluster 0 5.12 scrub starts 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.026598+0000 osd.1 (osd.1) 142 : cluster 0 5.12 scrub ok 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[29284]: audit 2024-03-11T21:37:43.156148+0000 mon.a (mon.0) 3054 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.541795+0000 mon.a (mon.0) 3051 : cluster 0 osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: audit 2024-03-11T21:37:42.548405+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: audit 2024-03-11T21:37:42.550594+0000 mon.a (mon.0) 3052 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.627944+0000 osd.3 (osd.3) 351 : cluster 0 6.13 scrub starts 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.629412+0000 osd.3 (osd.3) 352 : cluster 0 6.13 scrub ok 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.677999+0000 osd.5 (osd.5) 347 : cluster 0 6.d scrub starts 2024-03-11T21:37:43.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.679447+0000 osd.5 (osd.5) 348 : cluster 0 6.d scrub ok 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.760648+0000 osd.2 (osd.2) 301 : cluster 0 2.1 scrub starts 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.761601+0000 osd.2 (osd.2) 302 : cluster 0 2.1 scrub ok 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.891639+0000 osd.0 (osd.0) 336 : cluster 0 6.18 scrub starts 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:42.892882+0000 osd.0 (osd.0) 337 : cluster 0 6.18 scrub ok 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: audit 2024-03-11T21:37:42.958819+0000 mon.a (mon.0) 3053 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.025336+0000 osd.1 (osd.1) 141 : cluster 0 5.12 scrub starts 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.026598+0000 osd.1 (osd.1) 142 : cluster 0 5.12 scrub ok 2024-03-11T21:37:43.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:43 smithi143 bash[21512]: audit 2024-03-11T21:37:43.156148+0000 mon.a (mon.0) 3054 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:37:43.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:43.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:44.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:44.065 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:44.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:44.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.328 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:44.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.458 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:44.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.589 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.484271+0000 mgr.y (mgr.14550) 581 : cluster 0 pgmap v909: 260 pgs: 24 creating+peering, 8 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.529441+0000 mon.a (mon.0) 3055 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.531088+0000 mon.a (mon.0) 3056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: audit 2024-03-11T21:37:43.541430+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: audit 2024-03-11T21:37:43.541678+0000 mon.a (mon.0) 3058 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: audit 2024-03-11T21:37:43.541849+0000 mon.a (mon.0) 3059 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:37:44.686 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.549591+0000 mon.a (mon.0) 3060 : cluster 0 osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: audit 2024-03-11T21:37:43.553839+0000 mon.c (mon.2) 1071 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.716633+0000 osd.6 (osd.6) 259 : cluster 0 39.1e scrub starts 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[29284]: cluster 2024-03-11T21:37:43.718288+0000 osd.6 (osd.6) 260 : cluster 0 39.1e scrub ok 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.484271+0000 mgr.y (mgr.14550) 581 : cluster 0 pgmap v909: 260 pgs: 24 creating+peering, 8 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.529441+0000 mon.a (mon.0) 3055 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.531088+0000 mon.a (mon.0) 3056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: audit 2024-03-11T21:37:43.541430+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: audit 2024-03-11T21:37:43.541678+0000 mon.a (mon.0) 3058 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: audit 2024-03-11T21:37:43.541849+0000 mon.a (mon.0) 3059 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.549591+0000 mon.a (mon.0) 3060 : cluster 0 osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: audit 2024-03-11T21:37:43.553839+0000 mon.c (mon.2) 1071 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:44.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.716633+0000 osd.6 (osd.6) 259 : cluster 0 39.1e scrub starts 2024-03-11T21:37:44.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:44 smithi143 bash[21512]: cluster 2024-03-11T21:37:43.718288+0000 osd.6 (osd.6) 260 : cluster 0 39.1e scrub ok 2024-03-11T21:37:44.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.484271+0000 mgr.y (mgr.14550) 581 : cluster 0 pgmap v909: 260 pgs: 24 creating+peering, 8 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:37:44.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.529441+0000 mon.a (mon.0) 3055 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:44.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.531088+0000 mon.a (mon.0) 3056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: audit 2024-03-11T21:37:43.541430+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: audit 2024-03-11T21:37:43.541678+0000 mon.a (mon.0) 3058 : audit 1 from='client.? 172.21.15.143:0/81785983' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: audit 2024-03-11T21:37:43.541849+0000 mon.a (mon.0) 3059 : audit 1 from='client.? 172.21.15.143:0/1750500677' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.549591+0000 mon.a (mon.0) 3060 : cluster 0 osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: audit 2024-03-11T21:37:43.553839+0000 mon.c (mon.2) 1071 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.716633+0000 osd.6 (osd.6) 259 : cluster 0 39.1e scrub starts 2024-03-11T21:37:44.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:44 smithi203 bash[23654]: cluster 2024-03-11T21:37:43.718288+0000 osd.6 (osd.6) 260 : cluster 0 39.1e scrub ok 2024-03-11T21:37:44.719 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:44.822 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.850 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:44.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.980 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:45.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:45.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:45.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:45.241 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:45.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:45.371 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:45.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:45.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:45.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: cluster 2024-03-11T21:37:44.560423+0000 mon.a (mon.0) 3061 : cluster 0 osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: cluster 2024-03-11T21:37:44.668008+0000 osd.6 (osd.6) 261 : cluster 0 5.2 scrub starts 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: cluster 2024-03-11T21:37:44.669654+0000 osd.6 (osd.6) 262 : cluster 0 5.2 scrub ok 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: cluster 2024-03-11T21:37:44.990600+0000 osd.1 (osd.1) 143 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: cluster 2024-03-11T21:37:44.992896+0000 osd.1 (osd.1) 144 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: audit 2024-03-11T21:37:45.485368+0000 mon.c (mon.2) 1072 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.614 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[29284]: audit 2024-03-11T21:37:45.486635+0000 mon.a (mon.0) 3062 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: cluster 2024-03-11T21:37:44.560423+0000 mon.a (mon.0) 3061 : cluster 0 osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: cluster 2024-03-11T21:37:44.668008+0000 osd.6 (osd.6) 261 : cluster 0 5.2 scrub starts 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: cluster 2024-03-11T21:37:44.669654+0000 osd.6 (osd.6) 262 : cluster 0 5.2 scrub ok 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: cluster 2024-03-11T21:37:44.990600+0000 osd.1 (osd.1) 143 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: cluster 2024-03-11T21:37:44.992896+0000 osd.1 (osd.1) 144 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: audit 2024-03-11T21:37:45.485368+0000 mon.c (mon.2) 1072 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.615 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:45 smithi143 bash[21512]: audit 2024-03-11T21:37:45.486635+0000 mon.a (mon.0) 3062 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.625 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:45.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: cluster 2024-03-11T21:37:44.560423+0000 mon.a (mon.0) 3061 : cluster 0 osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: cluster 2024-03-11T21:37:44.668008+0000 osd.6 (osd.6) 261 : cluster 0 5.2 scrub starts 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: cluster 2024-03-11T21:37:44.669654+0000 osd.6 (osd.6) 262 : cluster 0 5.2 scrub ok 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: cluster 2024-03-11T21:37:44.990600+0000 osd.1 (osd.1) 143 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: cluster 2024-03-11T21:37:44.992896+0000 osd.1 (osd.1) 144 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: audit 2024-03-11T21:37:45.485368+0000 mon.c (mon.2) 1072 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:45 smithi203 bash[23654]: audit 2024-03-11T21:37:45.486635+0000 mon.a (mon.0) 3062 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:45.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:45.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:45.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:45.976 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:45.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:46.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:46.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:46.145 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:46.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:46.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:46.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:46.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:46.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:46.536 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:46.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:46.664 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:46.767 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:46.791 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:46.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.215622+0000 osd.4 (osd.4) 323 : cluster 0 118.f scrub starts 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.224481+0000 osd.4 (osd.4) 324 : cluster 0 118.f scrub ok 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.484939+0000 mgr.y (mgr.14550) 582 : cluster 0 pgmap v912: 260 pgs: 28 creating+peering, 232 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: audit 2024-03-11T21:37:45.562477+0000 mon.a (mon.0) 3063 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.568247+0000 mon.a (mon.0) 3064 : cluster 0 osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: audit 2024-03-11T21:37:45.657320+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: audit 2024-03-11T21:37:45.657824+0000 mon.a (mon.0) 3065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:46.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: audit 2024-03-11T21:37:45.658572+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: audit 2024-03-11T21:37:45.658890+0000 mon.a (mon.0) 3066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.689902+0000 osd.3 (osd.3) 353 : cluster 0 9.1a scrub starts 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.691195+0000 osd.3 (osd.3) 354 : cluster 0 9.1a scrub ok 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.764145+0000 osd.7 (osd.7) 332 : cluster 0 118.1e deep-scrub starts 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.781125+0000 osd.7 (osd.7) 333 : cluster 0 118.1e deep-scrub ok 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.429844+0000 mon.a (mon.0) 3067 : cluster 0 osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.215622+0000 osd.4 (osd.4) 323 : cluster 0 118.f scrub starts 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.224481+0000 osd.4 (osd.4) 324 : cluster 0 118.f scrub ok 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.484939+0000 mgr.y (mgr.14550) 582 : cluster 0 pgmap v912: 260 pgs: 28 creating+peering, 232 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:37:46.907 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: audit 2024-03-11T21:37:45.562477+0000 mon.a (mon.0) 3063 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.568247+0000 mon.a (mon.0) 3064 : cluster 0 osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: audit 2024-03-11T21:37:45.657320+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: audit 2024-03-11T21:37:45.657824+0000 mon.a (mon.0) 3065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: audit 2024-03-11T21:37:45.658572+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: audit 2024-03-11T21:37:45.658890+0000 mon.a (mon.0) 3066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.689902+0000 osd.3 (osd.3) 353 : cluster 0 9.1a scrub starts 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.691195+0000 osd.3 (osd.3) 354 : cluster 0 9.1a scrub ok 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.764145+0000 osd.7 (osd.7) 332 : cluster 0 118.1e deep-scrub starts 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.781125+0000 osd.7 (osd.7) 333 : cluster 0 118.1e deep-scrub ok 2024-03-11T21:37:46.908 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:46 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.429844+0000 mon.a (mon.0) 3067 : cluster 0 osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:37:46.917 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:47.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:47.045 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:47.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:47.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.215622+0000 osd.4 (osd.4) 323 : cluster 0 118.f scrub starts 2024-03-11T21:37:47.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.224481+0000 osd.4 (osd.4) 324 : cluster 0 118.f scrub ok 2024-03-11T21:37:47.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.484939+0000 mgr.y (mgr.14550) 582 : cluster 0 pgmap v912: 260 pgs: 28 creating+peering, 232 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:37:47.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: audit 2024-03-11T21:37:45.562477+0000 mon.a (mon.0) 3063 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:37:47.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.568247+0000 mon.a (mon.0) 3064 : cluster 0 osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: audit 2024-03-11T21:37:45.657320+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: audit 2024-03-11T21:37:45.657824+0000 mon.a (mon.0) 3065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: audit 2024-03-11T21:37:45.658572+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: audit 2024-03-11T21:37:45.658890+0000 mon.a (mon.0) 3066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-92"}]: dispatch 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.689902+0000 osd.3 (osd.3) 353 : cluster 0 9.1a scrub starts 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.691195+0000 osd.3 (osd.3) 354 : cluster 0 9.1a scrub ok 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.764145+0000 osd.7 (osd.7) 332 : cluster 0 118.1e deep-scrub starts 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.781125+0000 osd.7 (osd.7) 333 : cluster 0 118.1e deep-scrub ok 2024-03-11T21:37:47.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:46 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.429844+0000 mon.a (mon.0) 3067 : cluster 0 osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:37:47.175 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:47.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:47.305 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:47.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:47.443 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:47.505 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:47.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:47.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:47.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:37:47.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:47.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:47.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:47.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:47.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:47.957 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:48.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.669147+0000 osd.5 (osd.5) 349 : cluster 0 6.7 scrub starts 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:45.670197+0000 osd.5 (osd.5) 350 : cluster 0 6.7 scrub ok 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.189696+0000 osd.4 (osd.4) 325 : cluster 0 39.3 scrub starts 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.192243+0000 osd.4 (osd.4) 326 : cluster 0 39.3 scrub ok 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.718355+0000 osd.7 (osd.7) 334 : cluster 0 1.0 scrub starts 2024-03-11T21:37:48.072 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.720887+0000 osd.7 (osd.7) 335 : cluster 0 1.0 scrub ok 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.913348+0000 osd.3 (osd.3) 355 : cluster 0 39.11 scrub starts 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:46.916528+0000 osd.3 (osd.3) 356 : cluster 0 39.11 scrub ok 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.450354+0000 mon.a (mon.0) 3068 : cluster 0 osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: audit 2024-03-11T21:37:47.467821+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: audit 2024-03-11T21:37:47.468485+0000 mon.a (mon.0) 3069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: audit 2024-03-11T21:37:47.485797+0000 mon.c (mon.2) 1076 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[21512]: audit 2024-03-11T21:37:47.486843+0000 mon.a (mon.0) 3070 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.669147+0000 osd.5 (osd.5) 349 : cluster 0 6.7 scrub starts 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:45.670197+0000 osd.5 (osd.5) 350 : cluster 0 6.7 scrub ok 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.189696+0000 osd.4 (osd.4) 325 : cluster 0 39.3 scrub starts 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.192243+0000 osd.4 (osd.4) 326 : cluster 0 39.3 scrub ok 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.718355+0000 osd.7 (osd.7) 334 : cluster 0 1.0 scrub starts 2024-03-11T21:37:48.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.720887+0000 osd.7 (osd.7) 335 : cluster 0 1.0 scrub ok 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.913348+0000 osd.3 (osd.3) 355 : cluster 0 39.11 scrub starts 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:46.916528+0000 osd.3 (osd.3) 356 : cluster 0 39.11 scrub ok 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.450354+0000 mon.a (mon.0) 3068 : cluster 0 osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: audit 2024-03-11T21:37:47.467821+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: audit 2024-03-11T21:37:47.468485+0000 mon.a (mon.0) 3069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: audit 2024-03-11T21:37:47.485797+0000 mon.c (mon.2) 1076 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.074 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:47 smithi143 bash[29284]: audit 2024-03-11T21:37:47.486843+0000 mon.a (mon.0) 3070 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.088 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:48.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.669147+0000 osd.5 (osd.5) 349 : cluster 0 6.7 scrub starts 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:45.670197+0000 osd.5 (osd.5) 350 : cluster 0 6.7 scrub ok 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.189696+0000 osd.4 (osd.4) 325 : cluster 0 39.3 scrub starts 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.192243+0000 osd.4 (osd.4) 326 : cluster 0 39.3 scrub ok 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.718355+0000 osd.7 (osd.7) 334 : cluster 0 1.0 scrub starts 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.720887+0000 osd.7 (osd.7) 335 : cluster 0 1.0 scrub ok 2024-03-11T21:37:48.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.913348+0000 osd.3 (osd.3) 355 : cluster 0 39.11 scrub starts 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:46.916528+0000 osd.3 (osd.3) 356 : cluster 0 39.11 scrub ok 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.450354+0000 mon.a (mon.0) 3068 : cluster 0 osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: audit 2024-03-11T21:37:47.467821+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: audit 2024-03-11T21:37:47.468485+0000 mon.a (mon.0) 3069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: audit 2024-03-11T21:37:47.485797+0000 mon.c (mon.2) 1076 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:47 smithi203 bash[23654]: audit 2024-03-11T21:37:47.486843+0000 mon.a (mon.0) 3070 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:37:48.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:48.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:48.346 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:48.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:48.492 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:48.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:48.619 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:48.722 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:48.749 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:48.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:48.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:48.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.485460+0000 mgr.y (mgr.14550) 583 : cluster 0 pgmap v916: 270 pgs: 42 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.666841+0000 osd.5 (osd.5) 351 : cluster 0 5.7 scrub starts 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.668374+0000 osd.5 (osd.5) 352 : cluster 0 5.7 scrub ok 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.670988+0000 osd.7 (osd.7) 336 : cluster 0 1.1 scrub starts 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.672778+0000 osd.7 (osd.7) 337 : cluster 0 1.1 scrub ok 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.890712+0000 osd.0 (osd.0) 338 : cluster 0 6.10 scrub starts 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:47.892061+0000 osd.0 (osd.0) 339 : cluster 0 6.10 scrub ok 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: audit 2024-03-11T21:37:48.458659+0000 mon.a (mon.0) 3071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:48.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: audit 2024-03-11T21:37:48.458819+0000 mon.a (mon.0) 3072 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:48.466218+0000 mon.a (mon.0) 3073 : cluster 0 osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: audit 2024-03-11T21:37:48.480466+0000 mon.c (mon.2) 1077 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:48.768215+0000 osd.2 (osd.2) 303 : cluster 0 5.a scrub starts 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[21512]: cluster 2024-03-11T21:37:48.769827+0000 osd.2 (osd.2) 304 : cluster 0 5.a scrub ok 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.485460+0000 mgr.y (mgr.14550) 583 : cluster 0 pgmap v916: 270 pgs: 42 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.666841+0000 osd.5 (osd.5) 351 : cluster 0 5.7 scrub starts 2024-03-11T21:37:48.995 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.668374+0000 osd.5 (osd.5) 352 : cluster 0 5.7 scrub ok 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.670988+0000 osd.7 (osd.7) 336 : cluster 0 1.1 scrub starts 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.672778+0000 osd.7 (osd.7) 337 : cluster 0 1.1 scrub ok 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.890712+0000 osd.0 (osd.0) 338 : cluster 0 6.10 scrub starts 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:47.892061+0000 osd.0 (osd.0) 339 : cluster 0 6.10 scrub ok 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: audit 2024-03-11T21:37:48.458659+0000 mon.a (mon.0) 3071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: audit 2024-03-11T21:37:48.458819+0000 mon.a (mon.0) 3072 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:48.466218+0000 mon.a (mon.0) 3073 : cluster 0 osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: audit 2024-03-11T21:37:48.480466+0000 mon.c (mon.2) 1077 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:48.768215+0000 osd.2 (osd.2) 303 : cluster 0 5.a scrub starts 2024-03-11T21:37:48.996 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:48 smithi143 bash[29284]: cluster 2024-03-11T21:37:48.769827+0000 osd.2 (osd.2) 304 : cluster 0 5.a scrub ok 2024-03-11T21:37:49.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:49.028 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:49.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.485460+0000 mgr.y (mgr.14550) 583 : cluster 0 pgmap v916: 270 pgs: 42 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:49.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.666841+0000 osd.5 (osd.5) 351 : cluster 0 5.7 scrub starts 2024-03-11T21:37:49.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.668374+0000 osd.5 (osd.5) 352 : cluster 0 5.7 scrub ok 2024-03-11T21:37:49.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.670988+0000 osd.7 (osd.7) 336 : cluster 0 1.1 scrub starts 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.672778+0000 osd.7 (osd.7) 337 : cluster 0 1.1 scrub ok 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.890712+0000 osd.0 (osd.0) 338 : cluster 0 6.10 scrub starts 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:47.892061+0000 osd.0 (osd.0) 339 : cluster 0 6.10 scrub ok 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: audit 2024-03-11T21:37:48.458659+0000 mon.a (mon.0) 3071 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: audit 2024-03-11T21:37:48.458819+0000 mon.a (mon.0) 3072 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:48.466218+0000 mon.a (mon.0) 3073 : cluster 0 osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: audit 2024-03-11T21:37:48.480466+0000 mon.c (mon.2) 1077 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:48.768215+0000 osd.2 (osd.2) 303 : cluster 0 5.a scrub starts 2024-03-11T21:37:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:48 smithi203 bash[23654]: cluster 2024-03-11T21:37:48.769827+0000 osd.2 (osd.2) 304 : cluster 0 5.a scrub ok 2024-03-11T21:37:49.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:49.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:49.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:49.263 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:49.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:49.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:49.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:49.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:49.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:49.757 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:49.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:49.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:49 smithi143 bash[21512]: audit 2024-03-11T21:37:48.802865+0000 mgr.y (mgr.14550) 584 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:49.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:49 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.468111+0000 mon.a (mon.0) 3074 : cluster 0 osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:37:49.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:49 smithi143 bash[29284]: audit 2024-03-11T21:37:48.802865+0000 mgr.y (mgr.14550) 584 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:49.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:49 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.468111+0000 mon.a (mon.0) 3074 : cluster 0 osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:37:50.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:50.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:50.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:49 smithi203 bash[23654]: audit 2024-03-11T21:37:48.802865+0000 mgr.y (mgr.14550) 584 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:50.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:49 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.468111+0000 mon.a (mon.0) 3074 : cluster 0 osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:37:50.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:50.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- 172.21.15.143:0/1559883333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2758069d70 msgr2=0x7f275806a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- 172.21.15.143:0/1559883333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2758069d70 0x7f275806a150 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f27480099c0 tx=0x7f2748009cd0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- 172.21.15.143:0/1559883333 shutdown_connections 2024-03-11T21:37:50.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- 172.21.15.143:0/1559883333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27581071c0 0x7f275810ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- 172.21.15.143:0/1559883333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2758069d70 0x7f275806a150 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- 172.21.15.143:0/1559883333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f275806a720 0x7f2758106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- 172.21.15.143:0/1559883333 >> 172.21.15.143:0/1559883333 conn(0x7f2758076d80 msgr2=0x7f2758077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- 172.21.15.143:0/1559883333 shutdown_connections 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- 172.21.15.143:0/1559883333 wait complete. 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 librados: starting msgr at 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 librados: starting objecter 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 Processor -- start 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- start start 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 librados: setting wanted keys 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 librados: calling monclient init 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2758069d70 0x7f275819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f275806a720 0x7f275819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27581071c0 0x7f27581979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758111630 con 0x7f2758069d70 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27581114b0 con 0x7f275806a720 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f275d9bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27581117b0 con 0x7f27581071c0 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f2756ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2758069d70 0x7f275819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f27577fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27581071c0 0x7f27581979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f2756ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2758069d70 0x7f275819d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35332/0 (socket says 172.21.15.143:35332) 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f2756ffd700 1 -- 172.21.15.143:0/3607825033 learned_addr learned my addr 172.21.15.143:0/3607825033 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.225+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f275806a720 0x7f275819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f275806a720 msgr2=0x7f275819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f275806a720 0x7f275819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2758069d70 msgr2=0x7f275819d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2758069d70 0x7f275819d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2758198120 con 0x7f27581071c0 2024-03-11T21:37:50.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f275806a720 0x7f275819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:37:50.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27581071c0 0x7f27581979c0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f2750007f10 tx=0x7f275000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 10 client.?.objecter ms_handle_connect 0x7f27581071c0 2024-03-11T21:37:50.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:50.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275000ddd0 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2758198410 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2750009e20 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750015400 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 10 client.41798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2758100d40 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 librados: init done 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f275804faa0 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2750004ad0 con 0x7f27581071c0 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f274006f820 0x7f2740071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(597..597 src has 87..597) v4 ==== 9919+0+0 (secure 0 0 0) 0x7f275008e540 con 0x7f27581071c0 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 osd_map(597..597 src has 87..597) v4 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 3 client.41798.objecter handle_osd_map got epochs [597,597] > 0 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 3 client.41798.objecter handle_osd_map decoding full epoch 597 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f274006f820 0x7f2740071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 20 client.41798.objecter dump_active .. 0 homeless 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v597) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2750010070 con 0x7f27581071c0 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 10 client.41798.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 1 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f27580ff680 0x7f275802c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 1 2024-03-11T21:37:50.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 0 1 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 4 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 10 client.41798.objecter _send_command 1 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2758062f00 con 0x7f27580ff680 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f274006f820 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f274006f820 0x7f2740071ce0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f27480037a0 tx=0x7f2748019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f27580ff680 0x7f275802c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27580ff680 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.229+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f27580ff680 0x7f275802c6b0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.233+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2758062f00 con 0x7f27580ff680 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.233+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.233+0000 7f275c9b9700 10 client.41798.objecter _finish_command 1 = system:0 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.233+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 4 2024-03-11T21:37:50.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.233+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 0 1 2024-03-11T21:37:50.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 10 client.41798.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 2 2024-03-11T21:37:50.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 2 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27580ff510 osd=0 3 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 0 2 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 4 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 10 client.41798.objecter _send_command 2 2024-03-11T21:37:50.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f275802cbf0 con 0x7f27580ff680 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f275802cbf0 con 0x7f27580ff680 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275c9b9700 10 client.41798.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 4 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.249+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 0 2 2024-03-11T21:37:50.229 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 10 client.41798.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f27580625e0 0x7f2758101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 1 3 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 4 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 10 client.41798.objecter _send_command 3 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2758102d40 con 0x7f27580625e0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f27580625e0 0x7f2758101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27580625e0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f27580625e0 0x7f2758101060 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2758102d40 con 0x7f27580625e0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 10 client.41798.objecter _finish_command 3 = system:0 2024-03-11T21:37:50.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 4 2024-03-11T21:37:50.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.253+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 1 3 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 10 client.41798.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 4 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 4 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758062470 osd=1 3 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 1 4 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 4 2024-03-11T21:37:50.248 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 10 client.41798.objecter _send_command 4 2024-03-11T21:37:50.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.269+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f27581015a0 con 0x7f27580625e0 2024-03-11T21:37:50.249 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f27581015a0 con 0x7f27580625e0 2024-03-11T21:37:50.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 10 client.41798.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 4 2024-03-11T21:37:50.250 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 1 4 2024-03-11T21:37:50.250 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:37:50.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 10 client.41798.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 5 2024-03-11T21:37:50.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f27581093a0 0x7f27581097c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.251 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 5 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 2 5 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 4 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 10 client.41798.objecter _send_command 5 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2758102aa0 con 0x7f27581093a0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f27581093a0 0x7f27581097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27581093a0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f27581093a0 0x7f27581097c0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2758102aa0 con 0x7f27581093a0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 10 client.41798.objecter _finish_command 5 = system:0 2024-03-11T21:37:50.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 4 2024-03-11T21:37:50.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.273+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 2 5 2024-03-11T21:37:50.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 10 client.41798.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 6 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 6 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758102930 osd=2 3 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 2 6 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 4 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 10 client.41798.objecter _send_command 6 2024-03-11T21:37:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.289+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2758108e20 con 0x7f27581093a0 2024-03-11T21:37:50.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2758108e20 con 0x7f27581093a0 2024-03-11T21:37:50.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 10 client.41798.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 4 2024-03-11T21:37:50.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 2 6 2024-03-11T21:37:50.271 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 10 client.41798.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 7 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2758102430 0x7f27581b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 7 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 3 7 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 4 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 10 client.41798.objecter _send_command 7 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27581b4b00 con 0x7f2758102430 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2758102430 0x7f27581b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f2758102430 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2758102430 0x7f27581b1ca0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27581b4b00 con 0x7f2758102430 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 10 client.41798.objecter _finish_command 7 = system:0 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 4 2024-03-11T21:37:50.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.293+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 3 7 2024-03-11T21:37:50.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 10 client.41798.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 8 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 8 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758109220 osd=3 3 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 3 8 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 4 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 10 client.41798.objecter _send_command 8 2024-03-11T21:37:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2758101a60 con 0x7f2758102430 2024-03-11T21:37:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2758101a60 con 0x7f2758102430 2024-03-11T21:37:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 10 client.41798.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 4 2024-03-11T21:37:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 3 8 2024-03-11T21:37:50.293 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 10 client.41798.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 9 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f27581af7f0 0x7f27581b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 9 2024-03-11T21:37:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 4 9 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 4 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 10 client.41798.objecter _send_command 9 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2758101a20 con 0x7f27581af7f0 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f27581af7f0 0x7f27581b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27581af7f0 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.313+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f27581af7f0 0x7f27581b3360 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.317+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2758101a20 con 0x7f27581af7f0 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.317+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:37:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.317+0000 7f275c9b9700 10 client.41798.objecter _finish_command 9 = system:0 2024-03-11T21:37:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.317+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 4 2024-03-11T21:37:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.317+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 4 9 2024-03-11T21:37:50.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:50.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 10 client.41798.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 10 2024-03-11T21:37:50.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 10 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f2758101810 osd=4 3 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 4 10 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 4 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 10 client.41798.objecter _send_command 10 2024-03-11T21:37:50.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f27581b38a0 con 0x7f27581af7f0 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f27581b38a0 con 0x7f27581af7f0 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275c9b9700 10 client.41798.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 4 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.333+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 4 10 2024-03-11T21:37:50.313 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:37:50.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 10 client.41798.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 11 2024-03-11T21:37:50.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f27581b3fc0 0x7f27581b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 11 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 5 11 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 4 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 10 client.41798.objecter _send_command 11 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27581bb5f0 con 0x7f27581b3fc0 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f27581b3fc0 0x7f27581b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f27567fc700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f27581b3fc0 0x7f27581b4420 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27581b3fc0 2024-03-11T21:37:50.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27581bb5f0 con 0x7f27581b3fc0 2024-03-11T21:37:50.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:37:50.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 10 client.41798.objecter _finish_command 11 = system:0 2024-03-11T21:37:50.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 4 2024-03-11T21:37:50.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.337+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 5 11 2024-03-11T21:37:50.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 10 client.41798.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 12 2024-03-11T21:37:50.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 12 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b38a0 osd=5 3 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 5 12 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 4 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 10 client.41798.objecter _send_command 12 2024-03-11T21:37:50.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.353+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f27581b8cd0 con 0x7f27581b3fc0 2024-03-11T21:37:50.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f27581b8cd0 con 0x7f27581b3fc0 2024-03-11T21:37:50.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 10 client.41798.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 4 2024-03-11T21:37:50.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 5 12 2024-03-11T21:37:50.335 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:37:50.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 10 client.41798.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 13 2024-03-11T21:37:50.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f27581b93f0 0x7f27581b9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 13 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 6 13 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 4 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 10 client.41798.objecter _send_command 13 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27581be990 con 0x7f27581b93f0 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f27581b93f0 0x7f27581b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f27577fe700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f27581b93f0 0x7f27581b9830 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27581b93f0 2024-03-11T21:37:50.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27581be990 con 0x7f27581b93f0 2024-03-11T21:37:50.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:37:50.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 10 client.41798.objecter _finish_command 13 = system:0 2024-03-11T21:37:50.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 4 2024-03-11T21:37:50.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.357+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 6 13 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 10 client.41798.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 14 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 14 2024-03-11T21:37:50.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b8cd0 osd=6 3 2024-03-11T21:37:50.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 6 14 2024-03-11T21:37:50.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 4 2024-03-11T21:37:50.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 10 client.41798.objecter _send_command 14 2024-03-11T21:37:50.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.377+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f27581b9e80 con 0x7f27581b93f0 2024-03-11T21:37:50.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f27581b9e80 con 0x7f27581b93f0 2024-03-11T21:37:50.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 10 client.41798.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 4 2024-03-11T21:37:50.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 6 14 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 10 client.41798.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 15 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f27581ba5a0 0x7f27581baa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 15 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 7 15 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 4 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 10 client.41798.objecter _send_command 15 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27581bb100 con 0x7f27581ba5a0 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f27581ba5a0 0x7f27581baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f275c9b9700 10 client.41798.objecter ms_handle_connect 0x7f27581ba5a0 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.381+0000 7f2756ffd700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f27581ba5a0 0x7f27581baa20 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:50.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.385+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27581bb100 con 0x7f27581ba5a0 2024-03-11T21:37:50.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.385+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:37:50.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.385+0000 7f275c9b9700 10 client.41798.objecter _finish_command 15 = system:0 2024-03-11T21:37:50.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.385+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 4 2024-03-11T21:37:50.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.385+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 7 15 2024-03-11T21:37:50.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 10 client.41798.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:37:50.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign -1 16 2024-03-11T21:37:50.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 20 client.41798.objecter _get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_remove -1 16 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 20 client.41798.objecter get_session s=0x7f27581b9e80 osd=7 3 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 15 client.41798.objecter _session_command_op_assign 7 16 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 4 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 10 client.41798.objecter _send_command 16 2024-03-11T21:37:50.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.401+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f27581bb100 con 0x7f27581ba5a0 2024-03-11T21:37:50.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275c9b9700 1 -- 172.21.15.143:0/3607825033 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f27581bb100 con 0x7f27581ba5a0 2024-03-11T21:37:50.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275c9b9700 10 client.41798.objecter ms_dispatch 0x7f2758001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:37:50.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275c9b9700 10 client.41798.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:37:50.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275c9b9700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 4 2024-03-11T21:37:50.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275c9b9700 15 client.41798.objecter _session_command_op_remove 7 16 2024-03-11T21:37:50.384 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:37:50.385 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 librados: watch_flush enter 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 librados: watch_flush exit 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.0 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f27580ff680 msgr2=0x7f275802c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f27580ff680 0x7f275802c6b0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27580ff510 osd=0 1 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.1 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f27580625e0 msgr2=0x7f2758101060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f27580625e0 0x7f2758101060 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758062470 osd=1 1 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.2 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f27581093a0 msgr2=0x7f27581097c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f27581093a0 0x7f27581097c0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758102930 osd=2 1 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.3 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2758102430 msgr2=0x7f27581b1ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f2758102430 0x7f27581b1ca0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758109220 osd=3 1 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 reap_dead start 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.4 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f27581af7f0 msgr2=0x7f27581b3360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f27581af7f0 0x7f27581b3360 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 reap_dead start 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f2758101810 osd=4 1 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.5 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f27581b3fc0 msgr2=0x7f27581b4420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f27581b3fc0 0x7f27581b4420 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b38a0 osd=5 1 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.6 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f27581b93f0 msgr2=0x7f27581b9830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f27581b93f0 0x7f27581b9830 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b8cd0 osd=6 1 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter close_session for osd.7 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f27581ba5a0 msgr2=0x7f27581baa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f27581ba5a0 0x7f27581baa20 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter put_session s=0x7f27581b9e80 osd=7 1 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 20 client.41798.objecter shutdown clearing up homeless session... 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 10 client.41798.objecter successfully canceled tick 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f274006f820 msgr2=0x7f2740071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f274006f820 0x7f2740071ce0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f27480037a0 tx=0x7f2748019040 comp rx=0 tx=0).stop 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27581071c0 msgr2=0x7f27581979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 --2- 172.21.15.143:0/3607825033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27581071c0 0x7f27581979c0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f2750007f10 tx=0x7f275000d3b0 comp rx=0 tx=0).stop 2024-03-11T21:37:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 reap_dead start 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f27577fe700 1 -- 172.21.15.143:0/3607825033 reap_dead start 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 shutdown_connections 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 >> 172.21.15.143:0/3607825033 conn(0x7f2758076d80 msgr2=0x7f27580febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 shutdown_connections 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 -- 172.21.15.143:0/3607825033 wait complete. 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:50.405+0000 7f275d9bb700 1 librados: shutdown 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:50.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.429 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:50.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:50.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:37:50.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:37:50.618 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:37:50.618 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-03-11T21:37:50.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-11T21:37:50.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:50.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:50.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:50.819 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:50.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.485966+0000 mgr.y (mgr.14550) 585 : cluster 0 pgmap v919: 270 pgs: 23 creating+peering, 7 unknown, 240 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.831222+0000 osd.0 (osd.0) 340 : cluster 0 39.4 scrub starts 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.833161+0000 osd.0 (osd.0) 341 : cluster 0 39.4 scrub ok 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.933127+0000 osd.1 (osd.1) 145 : cluster 0 3.4 scrub starts 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:49.936205+0000 osd.1 (osd.1) 146 : cluster 0 3.4 scrub ok 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: audit 2024-03-11T21:37:50.234305+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.143:0/3607825033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.461188+0000 mon.a (mon.0) 3075 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.472331+0000 mon.a (mon.0) 3076 : cluster 0 osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:37:50.936 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.485966+0000 mgr.y (mgr.14550) 585 : cluster 0 pgmap v919: 270 pgs: 23 creating+peering, 7 unknown, 240 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.831222+0000 osd.0 (osd.0) 340 : cluster 0 39.4 scrub starts 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.833161+0000 osd.0 (osd.0) 341 : cluster 0 39.4 scrub ok 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.933127+0000 osd.1 (osd.1) 145 : cluster 0 3.4 scrub starts 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:49.936205+0000 osd.1 (osd.1) 146 : cluster 0 3.4 scrub ok 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: audit 2024-03-11T21:37:50.234305+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.143:0/3607825033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.461188+0000 mon.a (mon.0) 3075 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:50.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:50 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.472331+0000 mon.a (mon.0) 3076 : cluster 0 osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:37:50.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:50.997 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:37:50.997 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:37:50.997 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:37:50.997 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:37:50.998 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:37:51.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.485966+0000 mgr.y (mgr.14550) 585 : cluster 0 pgmap v919: 270 pgs: 23 creating+peering, 7 unknown, 240 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.831222+0000 osd.0 (osd.0) 340 : cluster 0 39.4 scrub starts 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.833161+0000 osd.0 (osd.0) 341 : cluster 0 39.4 scrub ok 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.933127+0000 osd.1 (osd.1) 145 : cluster 0 3.4 scrub starts 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:49.936205+0000 osd.1 (osd.1) 146 : cluster 0 3.4 scrub ok 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: audit 2024-03-11T21:37:50.234305+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.143:0/3607825033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.461188+0000 mon.a (mon.0) 3075 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:37:51.055 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:50 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.472331+0000 mon.a (mon.0) 3076 : cluster 0 osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:37:51.066 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:51.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:51.196 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:51.299 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.325 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:51.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.452 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:51.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:51.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:51.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.714 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:51.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:37:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.848 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:51.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:51.975 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:52.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.698468+0000 osd.5 (osd.5) 353 : cluster 0 2.18 scrub starts 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.701800+0000 osd.5 (osd.5) 354 : cluster 0 2.18 scrub ok 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.873893+0000 osd.0 (osd.0) 342 : cluster 0 39.f scrub starts 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:50.875598+0000 osd.0 (osd.0) 343 : cluster 0 39.f scrub ok 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:51.435039+0000 mon.a (mon.0) 3077 : cluster 0 osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:51.816710+0000 osd.2 (osd.2) 305 : cluster 0 5.8 scrub starts 2024-03-11T21:37:52.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: cluster 2024-03-11T21:37:51.818599+0000 osd.2 (osd.2) 306 : cluster 0 5.8 scrub ok 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: audit 2024-03-11T21:37:51.825937+0000 mon.a (mon.0) 3078 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[29284]: audit 2024-03-11T21:37:51.829670+0000 mon.c (mon.2) 1079 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.698468+0000 osd.5 (osd.5) 353 : cluster 0 2.18 scrub starts 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.701800+0000 osd.5 (osd.5) 354 : cluster 0 2.18 scrub ok 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.873893+0000 osd.0 (osd.0) 342 : cluster 0 39.f scrub starts 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:50.875598+0000 osd.0 (osd.0) 343 : cluster 0 39.f scrub ok 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:51.435039+0000 mon.a (mon.0) 3077 : cluster 0 osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:51.816710+0000 osd.2 (osd.2) 305 : cluster 0 5.8 scrub starts 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: cluster 2024-03-11T21:37:51.818599+0000 osd.2 (osd.2) 306 : cluster 0 5.8 scrub ok 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: audit 2024-03-11T21:37:51.825937+0000 mon.a (mon.0) 3078 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:51 smithi143 bash[21512]: audit 2024-03-11T21:37:51.829670+0000 mon.c (mon.2) 1079 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:52.086 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.698468+0000 osd.5 (osd.5) 353 : cluster 0 2.18 scrub starts 2024-03-11T21:37:52.086 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.701800+0000 osd.5 (osd.5) 354 : cluster 0 2.18 scrub ok 2024-03-11T21:37:52.086 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.873893+0000 osd.0 (osd.0) 342 : cluster 0 39.f scrub starts 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:50.875598+0000 osd.0 (osd.0) 343 : cluster 0 39.f scrub ok 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:51.435039+0000 mon.a (mon.0) 3077 : cluster 0 osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:51.816710+0000 osd.2 (osd.2) 305 : cluster 0 5.8 scrub starts 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: cluster 2024-03-11T21:37:51.818599+0000 osd.2 (osd.2) 306 : cluster 0 5.8 scrub ok 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: audit 2024-03-11T21:37:51.825937+0000 mon.a (mon.0) 3078 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:52.087 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:51 smithi203 bash[23654]: audit 2024-03-11T21:37:51.829670+0000 mon.c (mon.2) 1079 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:52.097 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:52.186 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:37:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:52.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.228 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:52.331 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.359 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:52.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:52.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:52.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:52.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:52.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:52.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:52.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:52.997 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:53.009 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:51.486421+0000 mgr.y (mgr.14550) 586 : cluster 0 pgmap v922: 270 pgs: 23 creating+peering, 247 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.161720+0000 osd.4 (osd.4) 327 : cluster 0 118.2 scrub starts 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.164046+0000 osd.4 (osd.4) 328 : cluster 0 118.2 scrub ok 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.435522+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.693849+0000 osd.3 (osd.3) 357 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.699577+0000 osd.3 (osd.3) 358 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:37:53.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:52 smithi203 bash[23654]: audit 2024-03-11T21:37:52.723190+0000 mon.c (mon.2) 1080 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:53.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.117 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:51.486421+0000 mgr.y (mgr.14550) 586 : cluster 0 pgmap v922: 270 pgs: 23 creating+peering, 247 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.161720+0000 osd.4 (osd.4) 327 : cluster 0 118.2 scrub starts 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.164046+0000 osd.4 (osd.4) 328 : cluster 0 118.2 scrub ok 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.435522+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.693849+0000 osd.3 (osd.3) 357 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:37:53.186 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.699577+0000 osd.3 (osd.3) 358 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[21512]: audit 2024-03-11T21:37:52.723190+0000 mon.c (mon.2) 1080 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:51.486421+0000 mgr.y (mgr.14550) 586 : cluster 0 pgmap v922: 270 pgs: 23 creating+peering, 247 active+clean; 8.3 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.161720+0000 osd.4 (osd.4) 327 : cluster 0 118.2 scrub starts 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.164046+0000 osd.4 (osd.4) 328 : cluster 0 118.2 scrub ok 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.435522+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.693849+0000 osd.3 (osd.3) 357 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.699577+0000 osd.3 (osd.3) 358 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:37:53.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:52 smithi143 bash[29284]: audit 2024-03-11T21:37:52.723190+0000 mon.c (mon.2) 1080 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:53.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:53.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:53.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:53.497 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:53.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:37:53.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:53.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:53.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:53.746 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:53.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:53.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:53.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.675323+0000 osd.5 (osd.5) 355 : cluster 0 1.8 scrub starts 2024-03-11T21:37:53.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.677813+0000 osd.5 (osd.5) 356 : cluster 0 1.8 scrub ok 2024-03-11T21:37:53.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.717720+0000 osd.7 (osd.7) 338 : cluster 0 1.9 scrub starts 2024-03-11T21:37:53.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.719902+0000 osd.7 (osd.7) 339 : cluster 0 1.9 scrub ok 2024-03-11T21:37:53.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.831857+0000 osd.6 (osd.6) 263 : cluster 0 1.6 scrub starts 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.833491+0000 osd.6 (osd.6) 264 : cluster 0 1.6 scrub ok 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.934017+0000 osd.1 (osd.1) 147 : cluster 0 1.3 scrub starts 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:52.935676+0000 osd.1 (osd.1) 148 : cluster 0 1.3 scrub ok 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.191537+0000 osd.4 (osd.4) 329 : cluster 0 1.5 scrub starts 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.192746+0000 osd.4 (osd.4) 330 : cluster 0 1.5 scrub ok 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: audit 2024-03-11T21:37:53.348567+0000 mon.c (mon.2) 1081 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: audit 2024-03-11T21:37:53.350155+0000 mon.c (mon.2) 1082 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:53.896 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[21512]: audit 2024-03-11T21:37:53.358183+0000 mon.a (mon.0) 3080 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:53.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:53.991 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.675323+0000 osd.5 (osd.5) 355 : cluster 0 1.8 scrub starts 2024-03-11T21:37:53.991 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.677813+0000 osd.5 (osd.5) 356 : cluster 0 1.8 scrub ok 2024-03-11T21:37:53.991 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.717720+0000 osd.7 (osd.7) 338 : cluster 0 1.9 scrub starts 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.719902+0000 osd.7 (osd.7) 339 : cluster 0 1.9 scrub ok 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.831857+0000 osd.6 (osd.6) 263 : cluster 0 1.6 scrub starts 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.833491+0000 osd.6 (osd.6) 264 : cluster 0 1.6 scrub ok 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.934017+0000 osd.1 (osd.1) 147 : cluster 0 1.3 scrub starts 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:52.935676+0000 osd.1 (osd.1) 148 : cluster 0 1.3 scrub ok 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.191537+0000 osd.4 (osd.4) 329 : cluster 0 1.5 scrub starts 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.192746+0000 osd.4 (osd.4) 330 : cluster 0 1.5 scrub ok 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: audit 2024-03-11T21:37:53.348567+0000 mon.c (mon.2) 1081 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: audit 2024-03-11T21:37:53.350155+0000 mon.c (mon.2) 1082 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:53.992 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:53 smithi143 bash[29284]: audit 2024-03-11T21:37:53.358183+0000 mon.a (mon.0) 3080 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:54.007 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:54.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:54.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:54.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:54.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.675323+0000 osd.5 (osd.5) 355 : cluster 0 1.8 scrub starts 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.677813+0000 osd.5 (osd.5) 356 : cluster 0 1.8 scrub ok 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.717720+0000 osd.7 (osd.7) 338 : cluster 0 1.9 scrub starts 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.719902+0000 osd.7 (osd.7) 339 : cluster 0 1.9 scrub ok 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.831857+0000 osd.6 (osd.6) 263 : cluster 0 1.6 scrub starts 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.833491+0000 osd.6 (osd.6) 264 : cluster 0 1.6 scrub ok 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.934017+0000 osd.1 (osd.1) 147 : cluster 0 1.3 scrub starts 2024-03-11T21:37:54.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:52.935676+0000 osd.1 (osd.1) 148 : cluster 0 1.3 scrub ok 2024-03-11T21:37:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.191537+0000 osd.4 (osd.4) 329 : cluster 0 1.5 scrub starts 2024-03-11T21:37:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.192746+0000 osd.4 (osd.4) 330 : cluster 0 1.5 scrub ok 2024-03-11T21:37:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: audit 2024-03-11T21:37:53.348567+0000 mon.c (mon.2) 1081 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: audit 2024-03-11T21:37:53.350155+0000 mon.c (mon.2) 1082 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:54.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:53 smithi203 bash[23654]: audit 2024-03-11T21:37:53.358183+0000 mon.a (mon.0) 3080 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:37:54.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:54.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:54.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:54.521 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:54.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:54.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:54.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:54.782 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:54.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:54.914 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:55.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:55.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.159 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.487587+0000 mgr.y (mgr.14550) 587 : cluster 0 pgmap v924: 270 pgs: 270 active+clean; 8.7 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.679738+0000 osd.7 (osd.7) 340 : cluster 0 1.2 scrub starts 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.682372+0000 osd.7 (osd.7) 341 : cluster 0 1.2 scrub ok 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.722427+0000 osd.5 (osd.5) 357 : cluster 0 1.a deep-scrub starts 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.724757+0000 osd.5 (osd.5) 358 : cluster 0 1.a deep-scrub ok 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.783734+0000 osd.6 (osd.6) 265 : cluster 0 1.7 scrub starts 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.785928+0000 osd.6 (osd.6) 266 : cluster 0 1.7 scrub ok 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.927557+0000 osd.1 (osd.1) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:53.927765+0000 osd.1 (osd.1) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.209640+0000 osd.4 (osd.4) 331 : cluster 0 118.a scrub starts 2024-03-11T21:37:55.160 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:54 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.213917+0000 osd.4 (osd.4) 332 : cluster 0 118.a scrub ok 2024-03-11T21:37:55.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.487587+0000 mgr.y (mgr.14550) 587 : cluster 0 pgmap v924: 270 pgs: 270 active+clean; 8.7 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.679738+0000 osd.7 (osd.7) 340 : cluster 0 1.2 scrub starts 2024-03-11T21:37:55.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.682372+0000 osd.7 (osd.7) 341 : cluster 0 1.2 scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.722427+0000 osd.5 (osd.5) 357 : cluster 0 1.a deep-scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.724757+0000 osd.5 (osd.5) 358 : cluster 0 1.a deep-scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.783734+0000 osd.6 (osd.6) 265 : cluster 0 1.7 scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.785928+0000 osd.6 (osd.6) 266 : cluster 0 1.7 scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.927557+0000 osd.1 (osd.1) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:53.927765+0000 osd.1 (osd.1) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.209640+0000 osd.4 (osd.4) 331 : cluster 0 118.a scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.213917+0000 osd.4 (osd.4) 332 : cluster 0 118.a scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.487587+0000 mgr.y (mgr.14550) 587 : cluster 0 pgmap v924: 270 pgs: 270 active+clean; 8.7 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.679738+0000 osd.7 (osd.7) 340 : cluster 0 1.2 scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.682372+0000 osd.7 (osd.7) 341 : cluster 0 1.2 scrub ok 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.722427+0000 osd.5 (osd.5) 357 : cluster 0 1.a deep-scrub starts 2024-03-11T21:37:55.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.724757+0000 osd.5 (osd.5) 358 : cluster 0 1.a deep-scrub ok 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.783734+0000 osd.6 (osd.6) 265 : cluster 0 1.7 scrub starts 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.785928+0000 osd.6 (osd.6) 266 : cluster 0 1.7 scrub ok 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.927557+0000 osd.1 (osd.1) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:53.927765+0000 osd.1 (osd.1) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.209640+0000 osd.4 (osd.4) 331 : cluster 0 118.a scrub starts 2024-03-11T21:37:55.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:54 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.213917+0000 osd.4 (osd.4) 332 : cluster 0 118.a scrub ok 2024-03-11T21:37:55.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:55.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:55.436 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:55.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:55.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.694 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:55.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:55.825 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:55.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.738318+0000 osd.5 (osd.5) 359 : cluster 0 1.1 scrub starts 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.740279+0000 osd.5 (osd.5) 360 : cluster 0 1.1 scrub ok 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.755040+0000 osd.5 (osd.5) 361 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.755317+0000 osd.5 (osd.5) 362 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.911430+0000 osd.0 (osd.0) 344 : cluster 0 5.3 scrub starts 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.914528+0000 osd.0 (osd.0) 345 : cluster 0 5.3 scrub ok 2024-03-11T21:37:55.939 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:55 smithi203 bash[23654]: cluster 2024-03-11T21:37:54.931885+0000 mon.a (mon.0) 3081 : cluster 0 osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:37:55.954 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:55.990 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:56.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:56.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.738318+0000 osd.5 (osd.5) 359 : cluster 0 1.1 scrub starts 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.740279+0000 osd.5 (osd.5) 360 : cluster 0 1.1 scrub ok 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.755040+0000 osd.5 (osd.5) 361 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.755317+0000 osd.5 (osd.5) 362 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.911430+0000 osd.0 (osd.0) 344 : cluster 0 5.3 scrub starts 2024-03-11T21:37:56.187 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.914528+0000 osd.0 (osd.0) 345 : cluster 0 5.3 scrub ok 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[21512]: cluster 2024-03-11T21:37:54.931885+0000 mon.a (mon.0) 3081 : cluster 0 osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.738318+0000 osd.5 (osd.5) 359 : cluster 0 1.1 scrub starts 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.740279+0000 osd.5 (osd.5) 360 : cluster 0 1.1 scrub ok 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.755040+0000 osd.5 (osd.5) 361 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.755317+0000 osd.5 (osd.5) 362 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.911430+0000 osd.0 (osd.0) 344 : cluster 0 5.3 scrub starts 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.914528+0000 osd.0 (osd.0) 345 : cluster 0 5.3 scrub ok 2024-03-11T21:37:56.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:55 smithi143 bash[29284]: cluster 2024-03-11T21:37:54.931885+0000 mon.a (mon.0) 3081 : cluster 0 osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:37:56.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:56.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:56.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.353 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:56.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:56.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:56.619 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:56.673 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:56.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:56.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:56.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:56.879 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:56.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:57.008 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.488514+0000 mgr.y (mgr.14550) 588 : cluster 0 pgmap v926: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.749320+0000 osd.5 (osd.5) 363 : cluster 0 3.16 scrub starts 2024-03-11T21:37:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.751616+0000 osd.5 (osd.5) 364 : cluster 0 3.16 scrub ok 2024-03-11T21:37:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.838542+0000 osd.6 (osd.6) 267 : cluster 0 3.a scrub starts 2024-03-11T21:37:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.841284+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub ok 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.864043+0000 osd.0 (osd.0) 346 : cluster 0 6.0 scrub starts 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.866571+0000 osd.0 (osd.0) 347 : cluster 0 6.0 scrub ok 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.909574+0000 osd.2 (osd.2) 307 : cluster 0 39.16 scrub starts 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:55.911417+0000 osd.2 (osd.2) 308 : cluster 0 39.16 scrub ok 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.216154+0000 osd.4 (osd.4) 333 : cluster 0 118.14 scrub starts 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.219892+0000 osd.4 (osd.4) 334 : cluster 0 118.14 scrub ok 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.435483+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.580072+0000 osd.3 (osd.3) 359 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:56 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.580285+0000 osd.3 (osd.3) 360 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:57.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:57.141 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.488514+0000 mgr.y (mgr.14550) 588 : cluster 0 pgmap v926: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.749320+0000 osd.5 (osd.5) 363 : cluster 0 3.16 scrub starts 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.751616+0000 osd.5 (osd.5) 364 : cluster 0 3.16 scrub ok 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.838542+0000 osd.6 (osd.6) 267 : cluster 0 3.a scrub starts 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.841284+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub ok 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.864043+0000 osd.0 (osd.0) 346 : cluster 0 6.0 scrub starts 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.866571+0000 osd.0 (osd.0) 347 : cluster 0 6.0 scrub ok 2024-03-11T21:37:57.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.909574+0000 osd.2 (osd.2) 307 : cluster 0 39.16 scrub starts 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:55.911417+0000 osd.2 (osd.2) 308 : cluster 0 39.16 scrub ok 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.216154+0000 osd.4 (osd.4) 333 : cluster 0 118.14 scrub starts 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.219892+0000 osd.4 (osd.4) 334 : cluster 0 118.14 scrub ok 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.435483+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.580072+0000 osd.3 (osd.3) 359 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.580285+0000 osd.3 (osd.3) 360 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.488514+0000 mgr.y (mgr.14550) 588 : cluster 0 pgmap v926: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.749320+0000 osd.5 (osd.5) 363 : cluster 0 3.16 scrub starts 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.751616+0000 osd.5 (osd.5) 364 : cluster 0 3.16 scrub ok 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.838542+0000 osd.6 (osd.6) 267 : cluster 0 3.a scrub starts 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.841284+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub ok 2024-03-11T21:37:57.188 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.864043+0000 osd.0 (osd.0) 346 : cluster 0 6.0 scrub starts 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.866571+0000 osd.0 (osd.0) 347 : cluster 0 6.0 scrub ok 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.909574+0000 osd.2 (osd.2) 307 : cluster 0 39.16 scrub starts 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:55.911417+0000 osd.2 (osd.2) 308 : cluster 0 39.16 scrub ok 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.216154+0000 osd.4 (osd.4) 333 : cluster 0 118.14 scrub starts 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.219892+0000 osd.4 (osd.4) 334 : cluster 0 118.14 scrub ok 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.435483+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.580072+0000 osd.3 (osd.3) 359 : cluster 0 purged_snaps scrub starts 2024-03-11T21:37:57.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:56 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.580285+0000 osd.3 (osd.3) 360 : cluster 0 purged_snaps scrub ok 2024-03-11T21:37:57.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:57.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:57.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:57.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:57.535 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:57.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:57.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:57.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:57.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.928 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:58.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.702402+0000 osd.7 (osd.7) 342 : cluster 0 9.d scrub starts 2024-03-11T21:37:58.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.704338+0000 osd.7 (osd.7) 343 : cluster 0 9.d scrub ok 2024-03-11T21:37:58.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.711725+0000 osd.5 (osd.5) 365 : cluster 0 3.c scrub starts 2024-03-11T21:37:58.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.714014+0000 osd.5 (osd.5) 366 : cluster 0 3.c scrub ok 2024-03-11T21:37:58.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.808249+0000 osd.6 (osd.6) 269 : cluster 0 2.14 scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:56.809693+0000 osd.6 (osd.6) 270 : cluster 0 2.14 scrub ok 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.210275+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.212648+0000 osd.4 (osd.4) 336 : cluster 0 2.f scrub ok 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.450212+0000 mon.a (mon.0) 3083 : cluster 0 osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.569148+0000 osd.3 (osd.3) 361 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.570882+0000 osd.3 (osd.3) 362 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.702402+0000 osd.7 (osd.7) 342 : cluster 0 9.d scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.704338+0000 osd.7 (osd.7) 343 : cluster 0 9.d scrub ok 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.711725+0000 osd.5 (osd.5) 365 : cluster 0 3.c scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.714014+0000 osd.5 (osd.5) 366 : cluster 0 3.c scrub ok 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.808249+0000 osd.6 (osd.6) 269 : cluster 0 2.14 scrub starts 2024-03-11T21:37:58.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:56.809693+0000 osd.6 (osd.6) 270 : cluster 0 2.14 scrub ok 2024-03-11T21:37:58.040 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.210275+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub starts 2024-03-11T21:37:58.040 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.212648+0000 osd.4 (osd.4) 336 : cluster 0 2.f scrub ok 2024-03-11T21:37:58.040 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.450212+0000 mon.a (mon.0) 3083 : cluster 0 osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:37:58.040 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.569148+0000 osd.3 (osd.3) 361 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:37:58.040 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:57 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.570882+0000 osd.3 (osd.3) 362 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:37:58.059 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:58.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- 172.21.15.143:0/1184843633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006aea0 msgr2=0x7f119010ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:58.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- 172.21.15.143:0/1184843633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006aea0 0x7f119010ecb0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f118800b0a0 tx=0x7f118800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- 172.21.15.143:0/1184843633 shutdown_connections 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- 172.21.15.143:0/1184843633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006aea0 0x7f119010ecb0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- 172.21.15.143:0/1184843633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 0x7f1190108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- 172.21.15.143:0/1184843633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006a4e0 0x7f119006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- 172.21.15.143:0/1184843633 >> 172.21.15.143:0/1184843633 conn(0x7f1190077bd0 msgr2=0x7f1190077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- 172.21.15.143:0/1184843633 shutdown_connections 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- 172.21.15.143:0/1184843633 wait complete. 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 librados: starting msgr at 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 librados: starting objecter 2024-03-11T21:37:58.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 Processor -- start 2024-03-11T21:37:58.098 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:58.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- start start 2024-03-11T21:37:58.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 librados: setting wanted keys 2024-03-11T21:37:58.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 librados: calling monclient init 2024-03-11T21:37:58.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 0x7f1190101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:58.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 0x7f11901015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 0x7f1190101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11901116b0 con 0x7f119006aea0 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1190111530 con 0x7f1190107e90 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.117+0000 7f1194b16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1190111830 con 0x7f119006a4e0 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118dd9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 0x7f11901015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 0x7f1190101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118dd9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 0x7f11901015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43516/0 (socket says 172.21.15.143:43516) 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118dd9b700 1 -- 172.21.15.143:0/3189494668 learned_addr learned my addr 172.21.15.143:0/3189494668 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118ed9d700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 0x7f1190101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 -- 172.21.15.143:0/3189494668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 msgr2=0x7f1190101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 0x7f1190101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 -- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 msgr2=0x7f11901015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 0x7f11901015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 -- 172.21.15.143:0/3189494668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11901a6150 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118e59c700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 0x7f1190101090 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f118000d750 tx=0x7f118000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 10 client.?.objecter ms_handle_connect 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180014030 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f11800044a0 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180005280 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11901a6440 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f1194b16700 10 client.41804.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11901a6930 con 0x7f119006a4e0 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f1194b16700 1 librados: init done 2024-03-11T21:37:58.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1190103520 con 0x7f119006a4e0 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f1180004a30 con 0x7f119006a4e0 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 0x7f1178071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f118dd9b700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 0x7f1178071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(603..603 src has 87..603) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f118008e390 con 0x7f119006a4e0 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 10 client.41804.objecter ms_dispatch 0x7f1190001550 osd_map(603..603 src has 87..603) v4 2024-03-11T21:37:58.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 3 client.41804.objecter handle_osd_map got epochs [603,603] > 0 2024-03-11T21:37:58.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 3 client.41804.objecter handle_osd_map decoding full epoch 603 2024-03-11T21:37:58.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.121+0000 7f117f7fe700 20 client.41804.objecter dump_active .. 0 homeless 2024-03-11T21:37:58.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.125+0000 7f117f7fe700 10 client.41804.objecter ms_handle_connect 0x7f117806f820 2024-03-11T21:37:58.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.125+0000 7f118dd9b700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 0x7f1178071ce0 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7f1190078360 tx=0x7f1184009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:37:58.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.129+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f118005afe0 con 0x7f119006a4e0 2024-03-11T21:37:58.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.702402+0000 osd.7 (osd.7) 342 : cluster 0 9.d scrub starts 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.704338+0000 osd.7 (osd.7) 343 : cluster 0 9.d scrub ok 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.711725+0000 osd.5 (osd.5) 365 : cluster 0 3.c scrub starts 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.714014+0000 osd.5 (osd.5) 366 : cluster 0 3.c scrub ok 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.808249+0000 osd.6 (osd.6) 269 : cluster 0 2.14 scrub starts 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:56.809693+0000 osd.6 (osd.6) 270 : cluster 0 2.14 scrub ok 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.210275+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub starts 2024-03-11T21:37:58.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.212648+0000 osd.4 (osd.4) 336 : cluster 0 2.f scrub ok 2024-03-11T21:37:58.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.450212+0000 mon.a (mon.0) 3083 : cluster 0 osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:37:58.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.569148+0000 osd.3 (osd.3) 361 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:37:58.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:57 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.570882+0000 osd.3 (osd.3) 362 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:37:58.184 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:58.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.315 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:58.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.353+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f11901a6b90 con 0x7f119006a4e0 2024-03-11T21:37:58.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:58.445 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:58.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:58.576 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:58.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:58.709 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:58.812 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:58.839 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:58.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:37:58.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:58.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.969+0000 7f117f7fe700 1 -- 172.21.15.143:0/3189494668 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v604) v1 ==== 87+0+0 (secure 0 0 0) 0x7f118005ea30 con 0x7f119006a4e0 2024-03-11T21:37:58.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 10 librados: watch_flush enter 2024-03-11T21:37:58.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 10 librados: watch_flush exit 2024-03-11T21:37:58.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 20 client.41804.objecter shutdown clearing up homeless session... 2024-03-11T21:37:58.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 10 client.41804.objecter successfully canceled tick 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 msgr2=0x7f1178071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 0x7f1178071ce0 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7f1190078360 tx=0x7f1184009290 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 msgr2=0x7f1190101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 0x7f1190101090 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f118000d750 tx=0x7f118000da60 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 shutdown_connections 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f117806f820 0x7f1178071ce0 unknown :-1 s=CLOSED pgs=3399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1190107e90 0x7f1190101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f119006a4e0 0x7f1190101090 secure :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f118000d750 tx=0x7f118000da60 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 --2- 172.21.15.143:0/3189494668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f119006aea0 0x7f11901015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:37:58.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 >> 172.21.15.143:0/3189494668 conn(0x7f1190077bd0 msgr2=0x7f1190108bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:37:58.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 shutdown_connections 2024-03-11T21:37:58.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 -- 172.21.15.143:0/3189494668 wait complete. 2024-03-11T21:37:58.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:37:58.973+0000 7f1194b16700 1 librados: shutdown 2024-03-11T21:37:58.952 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:37:58.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.489001+0000 mgr.y (mgr.14550) 589 : cluster 0 pgmap v929: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.984020+0000 osd.1 (osd.1) 151 : cluster 0 2.15 scrub starts 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: cluster 2024-03-11T21:37:57.985304+0000 osd.1 (osd.1) 152 : cluster 0 2.15 scrub ok 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.253757+0000 osd.4 (osd.4) 337 : cluster 0 4.1 scrub starts 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.255317+0000 osd.4 (osd.4) 338 : cluster 0 4.1 scrub ok 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.357904+0000 mon.c (mon.2) 1083 : audit 1 from='client.? 172.21.15.143:0/3189494668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.358399+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.644091+0000 mon.c (mon.2) 1084 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.009 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.644541+0000 mon.c (mon.2) 1085 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.644850+0000 mon.a (mon.0) 3085 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:58 smithi143 bash[21512]: audit 2024-03-11T21:37:58.645598+0000 mon.a (mon.0) 3086 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:59.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.489001+0000 mgr.y (mgr.14550) 589 : cluster 0 pgmap v929: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.984020+0000 osd.1 (osd.1) 151 : cluster 0 2.15 scrub starts 2024-03-11T21:37:59.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:57.985304+0000 osd.1 (osd.1) 152 : cluster 0 2.15 scrub ok 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.253757+0000 osd.4 (osd.4) 337 : cluster 0 4.1 scrub starts 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.255317+0000 osd.4 (osd.4) 338 : cluster 0 4.1 scrub ok 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.357904+0000 mon.c (mon.2) 1083 : audit 1 from='client.? 172.21.15.143:0/3189494668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.358399+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.644091+0000 mon.c (mon.2) 1084 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.644541+0000 mon.c (mon.2) 1085 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.644850+0000 mon.a (mon.0) 3085 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.645598+0000 mon.a (mon.0) 3086 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.489001+0000 mgr.y (mgr.14550) 589 : cluster 0 pgmap v929: 270 pgs: 270 active+clean; 8.6 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.984020+0000 osd.1 (osd.1) 151 : cluster 0 2.15 scrub starts 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: cluster 2024-03-11T21:37:57.985304+0000 osd.1 (osd.1) 152 : cluster 0 2.15 scrub ok 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.253757+0000 osd.4 (osd.4) 337 : cluster 0 4.1 scrub starts 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.255317+0000 osd.4 (osd.4) 338 : cluster 0 4.1 scrub ok 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.357904+0000 mon.c (mon.2) 1083 : audit 1 from='client.? 172.21.15.143:0/3189494668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.358399+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.644091+0000 mon.c (mon.2) 1084 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.644541+0000 mon.c (mon.2) 1085 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.644850+0000 mon.a (mon.0) 3085 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]: dispatch 2024-03-11T21:37:59.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:58 smithi203 bash[23654]: audit 2024-03-11T21:37:58.645598+0000 mon.a (mon.0) 3086 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]: dispatch 2024-03-11T21:37:59.224 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:59.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:59.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:59.486 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:59.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:59.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:59.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.745 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:59.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:59.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:59.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:00.003 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:00.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:00.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.685724+0000 osd.7 (osd.7) 344 : cluster 0 2.5 scrub starts 2024-03-11T21:38:00.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.687248+0000 osd.7 (osd.7) 345 : cluster 0 2.5 scrub ok 2024-03-11T21:38:00.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.814032+0000 mgr.y (mgr.14550) 590 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.940555+0000 osd.1 (osd.1) 153 : cluster 0 39.17 scrub starts 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.943263+0000 osd.1 (osd.1) 154 : cluster 0 39.17 scrub ok 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.956457+0000 mon.a (mon.0) 3087 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.966046+0000 mon.a (mon.0) 3088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.966124+0000 mon.a (mon.0) 3089 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]': finished 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: audit 2024-03-11T21:37:58.966215+0000 mon.a (mon.0) 3090 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]': finished 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[29284]: cluster 2024-03-11T21:37:58.969649+0000 mon.a (mon.0) 3091 : cluster 0 osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.685724+0000 osd.7 (osd.7) 344 : cluster 0 2.5 scrub starts 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.687248+0000 osd.7 (osd.7) 345 : cluster 0 2.5 scrub ok 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: audit 2024-03-11T21:37:58.814032+0000 mgr.y (mgr.14550) 590 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:00.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.940555+0000 osd.1 (osd.1) 153 : cluster 0 39.17 scrub starts 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.943263+0000 osd.1 (osd.1) 154 : cluster 0 39.17 scrub ok 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.956457+0000 mon.a (mon.0) 3087 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: audit 2024-03-11T21:37:58.966046+0000 mon.a (mon.0) 3088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: audit 2024-03-11T21:37:58.966124+0000 mon.a (mon.0) 3089 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]': finished 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: audit 2024-03-11T21:37:58.966215+0000 mon.a (mon.0) 3090 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]': finished 2024-03-11T21:38:00.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:37:59 smithi143 bash[21512]: cluster 2024-03-11T21:37:58.969649+0000 mon.a (mon.0) 3091 : cluster 0 osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24881.0:2 39.0 39.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:38:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 550.28921345200001, 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.450910398, 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24881", 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2563940939", 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845571+0000", 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845574+0000", 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0280000000000001e-06 2024-03-11T21:38:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845587+0000", 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2883e-05 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845617+0000", 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9745000000000001e-05 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.845640+0000", 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3057e-05 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.296469+0000", 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4508295920000001 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.296481+0000", 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2092999999999999e-05 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24704.0:2 8.d 8:b26ffda6:test-rados-api-smithi143-71620-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 550.27582321099999, 2024-03-11T21:38:00.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079922434000000001, 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24704", 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2524399149", 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858961+0000", 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858964+0000", 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7319999999999998e-06 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.858979+0000", 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.483e-05 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.859005+0000", 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5877e-05 2024-03-11T21:38:00.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.859027+0000", 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1950000000000002e-05 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.881850+0000", 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022822952000000001 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.881861+0000", 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1335000000000001e-05 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927087+0000", 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.045226508999999998 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927217+0000", 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000129985 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927327+0000", 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109405 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927718+0000", 2024-03-11T21:38:00.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000391419 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.927945+0000", 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022644399999999999 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938814+0000", 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010869838 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938852+0000", 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7960000000000002e-05 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.938884+0000", 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1198000000000002e-05 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24610.0:2 18.b 18:d5d10dbd:test-rados-api-smithi143-71773-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 548.88459452699999, 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17578144600000001, 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24610", 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2450754466", 2024-03-11T21:38:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250190+0000", 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250192+0000", 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9530000000000002e-06 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.250208+0000", 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6571000000000001e-05 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.252972+0000", 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027639729999999999 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.252988+0000", 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5387e-05 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.301037+0000", 2024-03-11T21:38:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.048049341000000002 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.301047+0000", 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6290000000000005e-06 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.391847+0000", 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.090799885999999996 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.392130+0000", 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028327700000000002 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.425971+0000", 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033841428999999999 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24706.0:1 44.6s0 44:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 547.81753002599999, 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1487131640000001, 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24706", 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1682110848", 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317254+0000", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317256+0000", 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7039999999999999e-06 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317263+0000", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1570000000000003e-06 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317277+0000", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4049000000000001e-05 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317293+0000", 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5897e-05 2024-03-11T21:38:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317535+0000", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000241609 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.317539+0000", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2599999999999999e-06 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.459236+0000", 2024-03-11T21:38:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1416968569999995 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.459367+0000", 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130815 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.462593+0000", 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003226591 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:00.145 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.463579+0000", 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00098520500000000007 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.465848+0000", 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022699159999999999 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.465968+0000", 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119104 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.146 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25037.0:3 41.14 41:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 546.29917110700001, 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.639917509, 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25037", 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2135306702", 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835613+0000", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835615+0000", 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5820000000000001e-06 2024-03-11T21:38:00.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835621+0000", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1959999999999996e-06 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835638+0000", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7345000000000001e-05 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835650+0000", 2024-03-11T21:38:00.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1102e-05 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835724+0000", 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4890999999999998e-05 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835734+0000", 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9119999999999993e-06 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.475447+0000", 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6397126469999996 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.475531+0000", 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3833999999999997e-05 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.151 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24743.0:2061 13.b 13:d04d64e0:test-rados-api-smithi143-71756-16::big.copy:head [copy-from ver 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 546.14993127800005, 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.099396153000000001, 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24743", 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/4179355585", 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2061 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.152 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984853+0000", 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984855+0000", 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0320000000000002e-06 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984867+0000", 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1814000000000001e-05 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984897+0000", 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9697e-05 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984911+0000", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4501999999999999e-05 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.984977+0000", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6087999999999995e-05 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.985060+0000", 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2614000000000006e-05 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.058952+0000", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073892422999999999 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.061358+0000", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024060510000000002 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084061+0000", 2024-03-11T21:38:00.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022702994000000001 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084189+0000", 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012778 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084222+0000", 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3139000000000002e-05 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.084249+0000", 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7019e-05 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25046.0:4 42.1f 42:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 541.27242770199996, 2024-03-11T21:38:00.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6107376059999998, 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25046", 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/232344682", 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:38:00.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862357+0000", 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862359+0000", 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3020000000000002e-06 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.862365+0000", 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4350000000000002e-06 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.865086+0000", 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002720311 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.865103+0000", 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6869000000000001e-05 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.874460+0000", 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0093571869999999994 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.874473+0000", 2024-03-11T21:38:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3203999999999999e-05 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.472983+0000", 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.598509559 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.473094+0000", 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111739 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.163 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:11 99.a 99:5634a9cf:test-rados-api-smithi143-71838-9::foo:head [write 0~128] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 511.67534332899999, 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069531287999999997, 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:38:00.164 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 11 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459441+0000", 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459443+0000", 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1289999999999999e-06 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459453+0000", 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9820000000000003e-06 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459471+0000", 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7869000000000001e-05 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.459483+0000", 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2266000000000001e-05 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.460708+0000", 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012245190000000001 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.461176+0000", 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000467863 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.461353+0000", 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017721999999999999 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.475346+0000", 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013993017999999999 2024-03-11T21:38:00.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.485203+0000", 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098573140000000007 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528895+0000", 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043691299000000003 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528936+0000", 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1437000000000003e-05 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:28.528972+0000", 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6371999999999997e-05 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:16 99.a 99:5634a9cf:test-rados-api-smithi143-71838-9::foo:head [write 128~128] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 511.037639748, 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.081470465000000006, 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 16 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097145+0000", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097148+0000", 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9730000000000002e-06 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097161+0000", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3337e-05 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.097189+0000", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8345999999999999e-05 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.098332+0000", 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011424009999999999 2024-03-11T21:38:00.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.099046+0000", 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000714626 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.099059+0000", 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3112000000000001e-05 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148091+0000", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.049031673999999997 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148268+0000", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017690800000000001 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.148619+0000", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035063 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.153402+0000", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047830279999999999 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.156195+0000", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027934739999999998 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178525+0000", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022330236999999999 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178573+0000", 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7401000000000002e-05 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:29.178615+0000", 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2317999999999997e-05 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34496.0:2 122.0 122:0384ab8e:::oid:head [write 0~0,setxattr key1 (5) in=4b,omap-clear] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 507.518204545, 2024-03-11T21:38:00.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8526834349999999, 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34496", 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1432393101", 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616580+0000", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616582+0000", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8080000000000001e-06 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616590+0000", 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4689999999999997e-06 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616604+0000", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4267000000000001e-05 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616616+0000", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2012e-05 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616820+0000", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020306999999999999 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:32.616839+0000", 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9046e-05 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463381+0000", 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.846542758 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463537+0000", 2024-03-11T21:38:00.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015549499999999999 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.463700+0000", 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016336899999999999 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.464262+0000", 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00056225600000000002 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.468013+0000", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037509230000000002 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469133+0000", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001119854 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469209+0000", 2024-03-11T21:38:00.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5422999999999999e-05 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:35.469263+0000", 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4685000000000002e-05 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.185 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23730.0:2 127.6 127:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 503.70414962500001, 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15423601100000001, 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23730", 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/484288227", 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430635+0000", 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430637+0000", 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.843e-06 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.430645+0000", 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6219999999999998e-06 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.431014+0000", 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00036898900000000001 2024-03-11T21:38:00.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.431746+0000", 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00073208599999999996 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.450995+0000", 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019249097999999999 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.451006+0000", 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1021e-05 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565207+0000", 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11420063699999999 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565343+0000", 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000135456 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565497+0000", 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000154384 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.565997+0000", 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049974800000000005 2024-03-11T21:38:00.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.566648+0000", 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00065155900000000001 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584801+0000", 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018152435000000001 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584843+0000", 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2166000000000003e-05 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.584871+0000", 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7966999999999998e-05 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:259 126.e 126:748fcfa5:::test_252:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e178)", 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 501.53541261300001, 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15545046300000001, 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:38:00.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 259 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599372+0000", 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599373+0000", 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.497e-06 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599382+0000", 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5019999999999992e-06 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599397+0000", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5291e-05 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599404+0000", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9829999999999999e-06 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599454+0000", 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0043999999999997e-05 2024-03-11T21:38:00.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.599552+0000", 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7511999999999997e-05 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:38.754822+0000", 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15527063399999999 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.198 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:390 126.14 126:2884fdb0:::test_383:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.59669119799997, 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.184484805, 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:38:00.199 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 390 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538093+0000", 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538094+0000", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0049999999999999e-06 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538101+0000", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5400000000000001e-06 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538115+0000", 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3743e-05 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538122+0000", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7219999999999999e-06 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538186+0000", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3812999999999994e-05 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.538309+0000", 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012344299999999999 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:39.722578+0000", 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18426853900000001 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.23697.0:457 126.1f 126:fdf52ff4:::test_450:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e179)", 2024-03-11T21:38:00.203 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.115407334, 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.22404676100000001, 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.23697", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 457 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019377+0000", 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019378+0000", 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0500000000000003e-07 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019381+0000", 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2119999999999999e-06 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019388+0000", 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019393+0000", 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.135e-06 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019441+0000", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8386000000000001e-05 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.019513+0000", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1167000000000005e-05 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.243424+0000", 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.22391118400000001 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.207 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25487.0:53 99.12 99:4cecbdbb:test-rados-api-smithi143-71838-11::foo:head [stat,delete] snapc a=[a,9] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.00892044599999, 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088322633999999997, 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25487", 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2823279075", 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 53 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.208 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125864+0000", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125867+0000", 2024-03-11T21:38:00.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7369999999999998e-06 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125879+0000", 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2309e-05 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125908+0000", 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9062999999999999e-05 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.125939+0000", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1127000000000002e-05 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.126101+0000", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000161924 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.126122+0000", 2024-03-11T21:38:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0843e-05 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181570+0000", 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.055448099000000001 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181708+0000", 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013806499999999999 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.181990+0000", 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028172199999999998 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.184022+0000", 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002032491 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.196112+0000", 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012089177 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214070+0000", 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017958096 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214132+0000", 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2129000000000001e-05 2024-03-11T21:38:00.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.214187+0000", 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4852000000000003e-05 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:109 118.10 118:0e6d2b0c:test-rados-api-smithi143-71773-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 493.55024693199999, 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.103370712, 2024-03-11T21:38:00.215 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584537+0000", 2024-03-11T21:38:00.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584540+0000", 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4229999999999999e-06 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584554+0000", 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4113e-05 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584582+0000", 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7679e-05 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584607+0000", 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5684e-05 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.584990+0000", 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000382471 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.585007+0000", 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7200000000000001e-05 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.647849+0000", 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.062841852000000004 2024-03-11T21:38:00.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.648005+0000", 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015635700000000001 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.648161+0000", 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015576500000000001 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.676335+0000", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.028173650000000001 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.680739+0000", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004404771 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687805+0000", 2024-03-11T21:38:00.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070651589999999997 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687863+0000", 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8025000000000001e-05 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.687908+0000", 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5562999999999997e-05 2024-03-11T21:38:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:773 118.13 118:ca30d8bc:test-rados-api-smithi143-71773-14::foo0:head [writefull 0~11] snapc 6=[6,5,4,3] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 469.503747154, 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100009098, 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.223 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 773 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631037+0000", 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631041+0000", 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6899999999999998e-06 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631060+0000", 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8953e-05 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631093+0000", 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2598999999999997e-05 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631120+0000", 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7223e-05 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631322+0000", 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000202394 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.631330+0000", 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2700000000000004e-06 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.652699+0000", 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021368809999999999 2024-03-11T21:38:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.652966+0000", 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000266419 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.653366+0000", 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00040020600000000001 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.656073+0000", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002707344 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.724089+0000", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.068015961999999999 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.730923+0000", 2024-03-11T21:38:00.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0068343570000000001 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.730997+0000", 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3328e-05 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:10.731046+0000", 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9543000000000003e-05 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.230 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40991.0:1 191.4s0 191:3d1c1d8f:test-rados-api-smithi143-71756-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 456.88346285599999, 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.011248849, 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40991", 2024-03-11T21:38:00.231 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3284700860", 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251322+0000", 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251324+0000", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2060000000000001e-06 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251335+0000", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1275e-05 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251355+0000", 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0013999999999998e-05 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251371+0000", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5851000000000001e-05 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251472+0000", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101586 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:23.251477+0000", 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8509999999999998e-06 2024-03-11T21:38:00.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247183+0000", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99570569600000003 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247325+0000", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000141528 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.247806+0000", 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00048166799999999998 2024-03-11T21:38:00.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.252955+0000", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051488669999999997 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.262484+0000", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0095285509999999997 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:24.262570+0000", 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6755999999999994e-05 2024-03-11T21:38:00.236 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41141.0:1 215.6s0 215:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 417.88675751300002, 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.060282137, 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41141", 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2586124918", 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.237 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248027+0000", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248030+0000", 2024-03-11T21:38:00.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9699999999999999e-06 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248046+0000", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6478999999999999e-05 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248091+0000", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4422000000000003e-05 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.248118+0000", 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7543e-05 2024-03-11T21:38:00.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.264982+0000", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016863974 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:02.264990+0000", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.306e-06 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.299012+0000", 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0340223479999999 2024-03-11T21:38:00.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.299309+0000", 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000297416 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.300363+0000", 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001053153 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:00.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.306158+0000", 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057951310000000002 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.308200+0000", 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020427129999999998 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:03.308309+0000", 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108682 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.29997.0:1 223.6s0 223:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 403.85193892799998, 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0641188610000001, 2024-03-11T21:38:00.243 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.29997", 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2134772814", 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:38:00.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282845+0000", 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282847+0000", 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9659999999999999e-06 2024-03-11T21:38:00.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282854+0000", 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.99e-06 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282867+0000", 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2959000000000001e-05 2024-03-11T21:38:00.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.282965+0000", 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7283999999999999e-05 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.283039+0000", 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4071000000000003e-05 2024-03-11T21:38:00.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:16.283043+0000", 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4510000000000002e-06 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.340931+0000", 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0578876260000001 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.341105+0000", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017386499999999999 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.342266+0000", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011612949999999999 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.344603+0000", 2024-03-11T21:38:00.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023371720000000002 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.346859+0000", 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022555769999999999 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:17.346964+0000", 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105605 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.250 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.251 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.251 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:00.251 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:00.251 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.685724+0000 osd.7 (osd.7) 344 : cluster 0 2.5 scrub starts 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.687248+0000 osd.7 (osd.7) 345 : cluster 0 2.5 scrub ok 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: audit 2024-03-11T21:37:58.814032+0000 mgr.y (mgr.14550) 590 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.940555+0000 osd.1 (osd.1) 153 : cluster 0 39.17 scrub starts 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.943263+0000 osd.1 (osd.1) 154 : cluster 0 39.17 scrub ok 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.956457+0000 mon.a (mon.0) 3087 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: audit 2024-03-11T21:37:58.966046+0000 mon.a (mon.0) 3088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:00.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: audit 2024-03-11T21:37:58.966124+0000 mon.a (mon.0) 3089 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.6", "id": [0, 5]}]': finished 2024-03-11T21:38:00.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: audit 2024-03-11T21:37:58.966215+0000 mon.a (mon.0) 3090 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "303.19", "id": [0, 4]}]': finished 2024-03-11T21:38:00.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:37:59 smithi203 bash[23654]: cluster 2024-03-11T21:37:58.969649+0000 mon.a (mon.0) 3091 : cluster 0 osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:38:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.253 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:00.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:00.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:00.385 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:00.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.516 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:00.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:00.644 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:00.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.774 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:00.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:38:00.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.926 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:01.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:01.058 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:01.161 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:01.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:01.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:00 smithi143 bash[29284]: cluster 2024-03-11T21:37:59.489827+0000 mgr.y (mgr.14550) 591 : cluster 0 pgmap v931: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:01.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:00 smithi143 bash[29284]: cluster 2024-03-11T21:37:59.971070+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:38:01.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:00 smithi143 bash[21512]: cluster 2024-03-11T21:37:59.489827+0000 mgr.y (mgr.14550) 591 : cluster 0 pgmap v931: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:01.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:00 smithi143 bash[21512]: cluster 2024-03-11T21:37:59.971070+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:38:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:00 smithi203 bash[23654]: cluster 2024-03-11T21:37:59.489827+0000 mgr.y (mgr.14550) 591 : cluster 0 pgmap v931: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:01.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:00 smithi203 bash[23654]: cluster 2024-03-11T21:37:59.971070+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:38:01.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:01.313 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:01.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:01.444 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:01.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:01.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:01.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:01.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:01.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:01.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:01.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:01.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:01.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:02.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:02.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:02.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:02.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:02.229 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:01 smithi143 bash[21512]: cluster 2024-03-11T21:38:00.988949+0000 mon.a (mon.0) 3093 : cluster 0 osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:38:02.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:01 smithi143 bash[29284]: cluster 2024-03-11T21:38:00.988949+0000 mon.a (mon.0) 3093 : cluster 0 osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:38:02.229 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:02.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:01 smithi203 bash[23654]: cluster 2024-03-11T21:38:00.988949+0000 mon.a (mon.0) 3093 : cluster 0 osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:38:02.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:02.357 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:02.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:02.489 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:02.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:02.618 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:02.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:02.750 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:02.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:02.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:02.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:02.997 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:02 smithi143 bash[29284]: cluster 2024-03-11T21:38:01.490402+0000 mgr.y (mgr.14550) 592 : cluster 0 pgmap v934: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:02.997 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:02 smithi143 bash[21512]: cluster 2024-03-11T21:38:01.490402+0000 mgr.y (mgr.14550) 592 : cluster 0 pgmap v934: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:03.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:03.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:03.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:03.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:03.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:38:03.261 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:02 smithi203 bash[23654]: cluster 2024-03-11T21:38:01.490402+0000 mgr.y (mgr.14550) 592 : cluster 0 pgmap v934: 270 pgs: 270 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:03.277 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:03.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:03.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:03.537 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:03.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.661 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:03.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:03.788 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:03.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:04.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:04.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:04.178 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/3497806325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074930 msgr2=0x7fe9cc10e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/3497806325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074930 0x7fe9cc10e330 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009990 tx=0x7fe9c0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/3497806325 shutdown_connections 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/3497806325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074010 0x7fe9cc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/3497806325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/3497806325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074930 0x7fe9cc10e330 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/3497806325 >> 172.21.15.143:0/3497806325 conn(0x7fe9cc078eb0 msgr2=0x7fe9cc0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.249+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/3497806325 shutdown_connections 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/3497806325 wait complete. 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 librados: starting msgr at 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 librados: starting objecter 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 Processor -- start 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- start start 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 librados: setting wanted keys 2024-03-11T21:38:04.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 librados: calling monclient init 2024-03-11T21:38:04.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 0x7fe9cc19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc113800 con 0x7fe9cc074930 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc113680 con 0x7fe9cc10e870 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc113980 con 0x7fe9cc074010 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d0d16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d0d16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43150/0 (socket says 172.21.15.143:43150) 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d0d16700 1 -- 172.21.15.143:0/1378669871 learned_addr learned my addr 172.21.15.143:0/1378669871 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 0x7fe9cc19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d1517700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 -- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 msgr2=0x7fe9cc199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 -- 172.21.15.143:0/1378669871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 msgr2=0x7fe9cc19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 -- 172.21.15.143:0/1378669871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc19ee70 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d0d16700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d1517700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9cbfff700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 0x7fe9cc19a3a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00059d0 tx=0x7fe9c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 10 client.?.objecter ms_handle_connect 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0027070 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc19f100 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe9c0022470 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0017650 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 10 client.30606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1a69e0 con 0x7fe9cc074930 2024-03-11T21:38:04.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 librados: init done 2024-03-11T21:38:04.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc0756a0 con 0x7fe9cc074930 2024-03-11T21:38:04.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.253+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe9c002a070 con 0x7fe9cc074930 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 0x7fe9bc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(606..606 src has 87..606) v4 ==== 9743+0+0 (secure 0 0 0) 0x7fe9c008fb80 con 0x7fe9cc074930 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 10 client.30606.objecter ms_dispatch 0x7fe9cc001550 osd_map(606..606 src has 87..606) v4 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9d0d16700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 0x7fe9bc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 3 client.30606.objecter handle_osd_map got epochs [606,606] > 0 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 3 client.30606.objecter handle_osd_map decoding full epoch 606 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 20 client.30606.objecter dump_active .. 0 homeless 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9c9ffb700 10 client.30606.objecter ms_handle_connect 0x7fe9bc06f820 2024-03-11T21:38:04.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.257+0000 7fe9d0d16700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 0x7fe9bc071ce0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8011550 tx=0x7fe9b8009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:04.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.261+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe9c0013070 con 0x7fe9cc074930 2024-03-11T21:38:04.281 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:04.299 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:04.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:04.429 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:04.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.485+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fe9cc19b1c0 con 0x7fe9cc074930 2024-03-11T21:38:04.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:04.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:04.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:04.687 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:04.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:04.820 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:04.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:04.950 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:04.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.993+0000 7fe9c9ffb700 1 -- 172.21.15.143:0/1378669871 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v607) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe9c005c760 con 0x7fe9cc074930 2024-03-11T21:38:04.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 10 librados: watch_flush enter 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 10 librados: watch_flush exit 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 20 client.30606.objecter shutdown clearing up homeless session... 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 10 client.30606.objecter successfully canceled tick 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 msgr2=0x7fe9bc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 0x7fe9bc071ce0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8011550 tx=0x7fe9b8009260 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 msgr2=0x7fe9cc19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 0x7fe9cc19a3a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00059d0 tx=0x7fe9c00047b0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 shutdown_connections 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe9bc06f820 0x7fe9bc071ce0 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9cc10e870 0x7fe9cc19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9cc074010 0x7fe9cc199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 --2- 172.21.15.143:0/1378669871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9cc074930 0x7fe9cc19a3a0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:04.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 >> 172.21.15.143:0/1378669871 conn(0x7fe9cc078eb0 msgr2=0x7fe9cc1062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:04.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 shutdown_connections 2024-03-11T21:38:04.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 -- 172.21.15.143:0/1378669871 wait complete. 2024-03-11T21:38:04.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:04.997+0000 7fe9d2f7a700 1 librados: shutdown 2024-03-11T21:38:04.976 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-11T21:38:05.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:05.061 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:05 smithi143 bash[29284]: cluster 2024-03-11T21:38:03.491722+0000 mgr.y (mgr.14550) 593 : cluster 0 pgmap v935: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:05.061 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:05 smithi143 bash[29284]: audit 2024-03-11T21:38:04.489976+0000 mon.a (mon.0) 3094 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.061 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:05 smithi143 bash[21512]: cluster 2024-03-11T21:38:03.491722+0000 mgr.y (mgr.14550) 593 : cluster 0 pgmap v935: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:05.061 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:05 smithi143 bash[21512]: audit 2024-03-11T21:38:04.489976+0000 mon.a (mon.0) 3094 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:05.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.202 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:05.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:05 smithi203 bash[23654]: cluster 2024-03-11T21:38:03.491722+0000 mgr.y (mgr.14550) 593 : cluster 0 pgmap v935: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:05.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:05 smithi203 bash[23654]: audit 2024-03-11T21:38:04.489976+0000 mon.a (mon.0) 3094 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.306 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.334 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:05.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:05.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:05.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:05.594 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:05.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:05.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.851 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:05.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:05.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:05.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:06.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:06 smithi143 bash[21512]: audit 2024-03-11T21:38:04.996369+0000 mon.a (mon.0) 3095 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:38:06.094 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:06 smithi143 bash[21512]: cluster 2024-03-11T21:38:05.012647+0000 mon.a (mon.0) 3096 : cluster 0 osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:38:06.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:06 smithi143 bash[29284]: audit 2024-03-11T21:38:04.996369+0000 mon.a (mon.0) 3095 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:38:06.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:06 smithi143 bash[29284]: cluster 2024-03-11T21:38:05.012647+0000 mon.a (mon.0) 3096 : cluster 0 osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:38:06.110 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:06.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:06.237 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:06.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:06 smithi203 bash[23654]: audit 2024-03-11T21:38:04.996369+0000 mon.a (mon.0) 3095 : audit 1 from='client.? 172.21.15.143:0/1378669871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:38:06.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:06 smithi203 bash[23654]: cluster 2024-03-11T21:38:05.012647+0000 mon.a (mon.0) 3096 : cluster 0 osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:38:06.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:06.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:06.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:06.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:06.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 -- 172.21.15.143:0/3298047404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 msgr2=0x7f0d501059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 --2- 172.21.15.143:0/3298047404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 0x7f0d501059c0 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f0d400099c0 tx=0x7f0d40009cd0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 -- 172.21.15.143:0/3298047404 shutdown_connections 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 --2- 172.21.15.143:0/3298047404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d501060a0 0x7f0d5010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 --2- 172.21.15.143:0/3298047404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d50100fc0 0x7f0d501013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 --2- 172.21.15.143:0/3298047404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 0x7f0d501059c0 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 -- 172.21.15.143:0/3298047404 >> 172.21.15.143:0/3298047404 conn(0x7f0d50077de0 msgr2=0x7f0d500781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 -- 172.21.15.143:0/3298047404 shutdown_connections 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 -- 172.21.15.143:0/3298047404 wait complete. 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.712+0000 7f0d57723700 1 librados: starting msgr at 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 librados: starting objecter 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 Processor -- start 2024-03-11T21:38:06.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- start start 2024-03-11T21:38:06.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 librados: setting wanted keys 2024-03-11T21:38:06.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 librados: calling monclient init 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 0x7f0d50102a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 0x7f0d50102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d501060a0 0x7f0d501034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50110560 con 0x7f0d50100fc0 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d501103e0 con 0x7f0d501060a0 2024-03-11T21:38:06.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d501106e0 con 0x7f0d50101970 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 0x7f0d50102a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 0x7f0d50102a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41306/0 (socket says 172.21.15.143:41306) 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 -- 172.21.15.143:0/1579341961 learned_addr learned my addr 172.21.15.143:0/1579341961 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 0x7f0d50102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d501060a0 0x7f0d501034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 msgr2=0x7f0d50102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d50101970 0x7f0d50102fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d501060a0 msgr2=0x7f0d501034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d501060a0 0x7f0d501034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d501a61d0 con 0x7f0d50100fc0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d501060a0 0x7f0d501034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 0x7f0d50102a60 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c970 tx=0x7f0d4c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 10 client.?.objecter ms_handle_connect 0x7f0d50100fc0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c007aa0 con 0x7f0d50100fc0 2024-03-11T21:38:06.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d501a64c0 con 0x7f0d50100fc0 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0d4c004cb0 con 0x7f0d50100fc0 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c005470 con 0x7f0d50100fc0 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 10 client.30612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d501a6980 con 0x7f0d50100fc0 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 librados: init done 2024-03-11T21:38:06.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0d5004faa0 con 0x7f0d50100fc0 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0d4c004750 con 0x7f0d50100fc0 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0d3c06f820 0x7f0d3c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(608..608 src has 87..608) v4 ==== 9747+0+0 (secure 0 0 0) 0x7f0d4c08ef90 con 0x7f0d50100fc0 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0d3c06f820 0x7f0d3c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 osd_map(608..608 src has 87..608) v4 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 3 client.30612.objecter handle_osd_map got epochs [608,608] > 0 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 3 client.30612.objecter handle_osd_map decoding full epoch 608 2024-03-11T21:38:06.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 20 client.30612.objecter dump_active .. 0 homeless 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.716+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v608) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0d4c025020 con 0x7f0d50100fc0 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 10 client.30612.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 1 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0d500629f0 0x7f0d50062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 1 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 0 1 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 4 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 10 client.30612.objecter _send_command 1 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501a9da0 con 0x7f0d500629f0 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d3c06f820 2024-03-11T21:38:06.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0d3c06f820 0x7f0d3c071ce0 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f0d40005b40 tx=0x7f0d40005a90 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0d500629f0 0x7f0d50062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d500629f0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0d500629f0 0x7f0d50062dd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501a9da0 con 0x7f0d500629f0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 10 client.30612.objecter _finish_command 1 = system:0 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 4 2024-03-11T21:38:06.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.720+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 0 1 2024-03-11T21:38:06.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 10 client.30612.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 2 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 2 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50103f90 osd=0 3 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 0 2 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 4 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 10 client.30612.objecter _send_command 2 2024-03-11T21:38:06.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.736+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d50062470 con 0x7f0d500629f0 2024-03-11T21:38:06.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d50062470 con 0x7f0d500629f0 2024-03-11T21:38:06.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 10 client.30612.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 4 2024-03-11T21:38:06.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 0 2 2024-03-11T21:38:06.719 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 10 client.30612.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f0d5002c6b0 0x7f0d5002caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 1 3 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 4 2024-03-11T21:38:06.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 10 client.30612.objecter _send_command 3 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d50109a20 con 0x7f0d5002c6b0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f0d5002c6b0 0x7f0d5002caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f0d5002c6b0 0x7f0d5002caf0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d5002c6b0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d50109a20 con 0x7f0d5002c6b0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 10 client.30612.objecter _finish_command 3 = system:0 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 4 2024-03-11T21:38:06.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.740+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 1 3 2024-03-11T21:38:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 10 client.30612.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 4 2024-03-11T21:38:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 4 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501093a0 osd=1 3 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 1 4 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 4 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 10 client.30612.objecter _send_command 4 2024-03-11T21:38:06.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.756+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d50108e20 con 0x7f0d5002c6b0 2024-03-11T21:38:06.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d50108e20 con 0x7f0d5002c6b0 2024-03-11T21:38:06.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 10 client.30612.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 4 2024-03-11T21:38:06.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 1 4 2024-03-11T21:38:06.740 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:38:06.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 10 client.30612.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 5 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f0d501abf60 0x7f0d501ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 5 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 2 5 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 4 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 10 client.30612.objecter _send_command 5 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501b11b0 con 0x7f0d501abf60 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f0d501abf60 0x7f0d501ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501abf60 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f0d501abf60 0x7f0d501ae350 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501b11b0 con 0x7f0d501abf60 2024-03-11T21:38:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:38:06.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 10 client.30612.objecter _finish_command 5 = system:0 2024-03-11T21:38:06.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 4 2024-03-11T21:38:06.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.760+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 2 5 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 10 client.30612.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 6 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 6 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50109bd0 osd=2 3 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 2 6 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 4 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 10 client.30612.objecter _send_command 6 2024-03-11T21:38:06.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d5002d280 con 0x7f0d501abf60 2024-03-11T21:38:06.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:06.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d5002d280 con 0x7f0d501abf60 2024-03-11T21:38:06.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 10 client.30612.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 4 2024-03-11T21:38:06.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 2 6 2024-03-11T21:38:06.760 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 10 client.30612.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 7 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0d501a9900 0x7f0d501afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 7 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 3 7 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 4 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 10 client.30612.objecter _send_command 7 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501b4b10 con 0x7f0d501a9900 2024-03-11T21:38:06.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0d501a9900 0x7f0d501afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501a9900 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0d501a9900 0x7f0d501afa10 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501b4b10 con 0x7f0d501a9900 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 10 client.30612.objecter _finish_command 7 = system:0 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 4 2024-03-11T21:38:06.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.780+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 3 7 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 10 client.30612.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 8 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 8 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d50109220 osd=3 3 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 3 8 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 4 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 10 client.30612.objecter _send_command 8 2024-03-11T21:38:06.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d501a8ed0 con 0x7f0d501a9900 2024-03-11T21:38:06.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d501a8ed0 con 0x7f0d501a9900 2024-03-11T21:38:06.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 10 client.30612.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 4 2024-03-11T21:38:06.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 3 8 2024-03-11T21:38:06.781 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 10 client.30612.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 9 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f0d501af570 0x7f0d501b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 9 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 4 9 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 4 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 10 client.30612.objecter _send_command 9 2024-03-11T21:38:06.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501a8eb0 con 0x7f0d501af570 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f0d501af570 0x7f0d501b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501af570 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.800+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f0d501af570 0x7f0d501b5380 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.804+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501a8eb0 con 0x7f0d501af570 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.804+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.804+0000 7f0d467fc700 10 client.30612.objecter _finish_command 9 = system:0 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.804+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 4 2024-03-11T21:38:06.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.804+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 4 9 2024-03-11T21:38:06.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 10 client.30612.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 10 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 10 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501a8c80 osd=4 3 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 4 10 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 4 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 10 client.30612.objecter _send_command 10 2024-03-11T21:38:06.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.820+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d501a8eb0 con 0x7f0d501af570 2024-03-11T21:38:06.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d501a8eb0 con 0x7f0d501af570 2024-03-11T21:38:06.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 10 client.30612.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 4 2024-03-11T21:38:06.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 4 10 2024-03-11T21:38:06.805 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 10 client.30612.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 11 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f0d501b7670 0x7f0d501b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 11 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 5 11 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 4 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 10 client.30612.objecter _send_command 11 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501bb3a0 con 0x7f0d501b7670 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f0d501b7670 0x7f0d501b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501b7670 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.824+0000 7f0d54cbe700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f0d501b7670 0x7f0d501b7a90 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.828+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501bb3a0 con 0x7f0d501b7670 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.828+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.828+0000 7f0d467fc700 10 client.30612.objecter _finish_command 11 = system:0 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.828+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 4 2024-03-11T21:38:06.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.828+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 5 11 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 10 client.30612.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 12 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 12 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501a8eb0 osd=5 3 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 5 12 2024-03-11T21:38:06.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 4 2024-03-11T21:38:06.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 10 client.30612.objecter _send_command 12 2024-03-11T21:38:06.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.844+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d501b5920 con 0x7f0d501b7670 2024-03-11T21:38:06.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d501b5920 con 0x7f0d501b7670 2024-03-11T21:38:06.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 10 client.30612.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 4 2024-03-11T21:38:06.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 5 12 2024-03-11T21:38:06.827 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:38:06.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 10 client.30612.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 13 2024-03-11T21:38:06.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f0d501b6040 0x7f0d501b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 13 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 6 13 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 4 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 10 client.30612.objecter _send_command 13 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501b6bb0 con 0x7f0d501b6040 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f0d501b6040 0x7f0d501b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501b6040 2024-03-11T21:38:06.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d55cc0700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f0d501b6040 0x7f0d501b6480 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501b6bb0 con 0x7f0d501b6040 2024-03-11T21:38:06.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:38:06.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 10 client.30612.objecter _finish_command 13 = system:0 2024-03-11T21:38:06.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 4 2024-03-11T21:38:06.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.848+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 6 13 2024-03-11T21:38:06.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 10 client.30612.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 14 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 14 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501b5920 osd=6 3 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 6 14 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 4 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 10 client.30612.objecter _send_command 14 2024-03-11T21:38:06.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d501b6bb0 con 0x7f0d501b6040 2024-03-11T21:38:06.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d501b6bb0 con 0x7f0d501b6040 2024-03-11T21:38:06.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d467fc700 10 client.30612.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 4 2024-03-11T21:38:06.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 6 14 2024-03-11T21:38:06.849 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 10 client.30612.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 15 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f0d501bc740 0x7f0d501beb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 15 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 7 15 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 4 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 10 client.30612.objecter _send_command 15 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d501c1990 con 0x7f0d501bc740 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.868+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f0d501bc740 0x7f0d501beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 10 client.30612.objecter ms_handle_connect 0x7f0d501bc740 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d554bf700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f0d501bc740 0x7f0d501beb30 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d501c1990 con 0x7f0d501bc740 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 10 client.30612.objecter _finish_command 15 = system:0 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 4 2024-03-11T21:38:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.872+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 7 15 2024-03-11T21:38:06.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 10 client.30612.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign -1 16 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 20 client.30612.objecter _get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 15 client.30612.objecter _session_command_op_remove -1 16 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 20 client.30612.objecter get_session s=0x7f0d501b6bb0 osd=7 3 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 15 client.30612.objecter _session_command_op_assign 7 16 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 4 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 10 client.30612.objecter _send_command 16 2024-03-11T21:38:06.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.888+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0d501bf070 con 0x7f0d501bc740 2024-03-11T21:38:06.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d467fc700 1 -- 172.21.15.143:0/1579341961 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0d501bf070 con 0x7f0d501bc740 2024-03-11T21:38:06.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d467fc700 10 client.30612.objecter ms_dispatch 0x7f0d50001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:06.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d467fc700 10 client.30612.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:06.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d467fc700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 4 2024-03-11T21:38:06.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d467fc700 15 client.30612.objecter _session_command_op_remove 7 16 2024-03-11T21:38:06.876 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:38:06.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 librados: watch_flush enter 2024-03-11T21:38:06.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 librados: watch_flush exit 2024-03-11T21:38:06.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.0 2024-03-11T21:38:06.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0d500629f0 msgr2=0x7f0d50062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f0d500629f0 0x7f0d50062dd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50103f90 osd=0 1 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.1 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f0d5002c6b0 msgr2=0x7f0d5002caf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f0d5002c6b0 0x7f0d5002caf0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501093a0 osd=1 1 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.2 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f0d501abf60 msgr2=0x7f0d501ae350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f0d501abf60 0x7f0d501ae350 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109bd0 osd=2 1 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.3 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0d501a9900 msgr2=0x7f0d501afa10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f0d501a9900 0x7f0d501afa10 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d50109220 osd=3 1 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.4 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f0d501af570 msgr2=0x7f0d501b5380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f0d501af570 0x7f0d501b5380 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8c80 osd=4 1 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.5 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f0d501b7670 msgr2=0x7f0d501b7a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f0d501b7670 0x7f0d501b7a90 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501a8eb0 osd=5 1 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.6 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f0d501b6040 msgr2=0x7f0d501b6480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f0d501b6040 0x7f0d501b6480 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b5920 osd=6 1 2024-03-11T21:38:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter close_session for osd.7 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f0d501bc740 msgr2=0x7f0d501beb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f0d501bc740 0x7f0d501beb30 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter put_session s=0x7f0d501b6bb0 osd=7 1 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 20 client.30612.objecter shutdown clearing up homeless session... 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 10 client.30612.objecter successfully canceled tick 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d55cc0700 1 -- 172.21.15.143:0/1579341961 reap_dead start 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0d3c06f820 msgr2=0x7f0d3c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0d3c06f820 0x7f0d3c071ce0 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f0d40005b40 tx=0x7f0d40005a90 comp rx=0 tx=0).stop 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d55cc0700 1 -- 172.21.15.143:0/1579341961 reap_dead start 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 msgr2=0x7f0d50102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 --2- 172.21.15.143:0/1579341961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d50100fc0 0x7f0d50102a60 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c970 tx=0x7f0d4c00cc80 comp rx=0 tx=0).stop 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d55cc0700 1 -- 172.21.15.143:0/1579341961 reap_dead start 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d55cc0700 1 -- 172.21.15.143:0/1579341961 reap_dead start 2024-03-11T21:38:06.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d55cc0700 1 -- 172.21.15.143:0/1579341961 reap_dead start 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 shutdown_connections 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 >> 172.21.15.143:0/1579341961 conn(0x7f0d50077de0 msgr2=0x7f0d5006a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 shutdown_connections 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 -- 172.21.15.143:0/1579341961 wait complete. 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:06.892+0000 7f0d57723700 1 librados: shutdown 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:06.881 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:06.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:06.913 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:07.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:07 smithi203 bash[23654]: cluster 2024-03-11T21:38:05.492449+0000 mgr.y (mgr.14550) 594 : cluster 0 pgmap v937: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:07.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:07 smithi203 bash[23654]: cluster 2024-03-11T21:38:06.442209+0000 mon.a (mon.0) 3097 : cluster 0 osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:38:07.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:07 smithi203 bash[23654]: audit 2024-03-11T21:38:06.722571+0000 mon.a (mon.0) 3098 : audit 0 from='client.? 172.21.15.143:0/1579341961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:07.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:07 smithi203 bash[23654]: audit 2024-03-11T21:38:06.854238+0000 mon.a (mon.0) 3099 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:07.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:07 smithi203 bash[23654]: audit 2024-03-11T21:38:06.857727+0000 mon.c (mon.2) 1086 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:07.037 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:07.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[21512]: cluster 2024-03-11T21:38:05.492449+0000 mgr.y (mgr.14550) 594 : cluster 0 pgmap v937: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[21512]: cluster 2024-03-11T21:38:06.442209+0000 mon.a (mon.0) 3097 : cluster 0 osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[21512]: audit 2024-03-11T21:38:06.722571+0000 mon.a (mon.0) 3098 : audit 0 from='client.? 172.21.15.143:0/1579341961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[21512]: audit 2024-03-11T21:38:06.854238+0000 mon.a (mon.0) 3099 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[21512]: audit 2024-03-11T21:38:06.857727+0000 mon.c (mon.2) 1086 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[29284]: cluster 2024-03-11T21:38:05.492449+0000 mgr.y (mgr.14550) 594 : cluster 0 pgmap v937: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[29284]: cluster 2024-03-11T21:38:06.442209+0000 mon.a (mon.0) 3097 : cluster 0 osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[29284]: audit 2024-03-11T21:38:06.722571+0000 mon.a (mon.0) 3098 : audit 0 from='client.? 172.21.15.143:0/1579341961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:07.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[29284]: audit 2024-03-11T21:38:06.854238+0000 mon.a (mon.0) 3099 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:07.151 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:07 smithi143 bash[29284]: audit 2024-03-11T21:38:06.857727+0000 mon.c (mon.2) 1086 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:07.162 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:07.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:07.292 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:07.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:07.423 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:07.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:07.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.685 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:07.788 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:07.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:07.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:07.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:08.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:08.074 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:08.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.204 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:08.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:08.335 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:08.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:08.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:08.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:08.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:08.734 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:08.734 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:08.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.829 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:08.841 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:08.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:08.971 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:09.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:09.101 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:09.118 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:09.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:09.229 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:09.231 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:09 smithi203 bash[23654]: cluster 2024-03-11T21:38:07.493306+0000 mgr.y (mgr.14550) 595 : cluster 0 pgmap v939: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:09.231 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:09 smithi203 bash[23654]: cluster 2024-03-11T21:38:08.418132+0000 osd.0 (osd.0) 348 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:09.231 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:09 smithi203 bash[23654]: cluster 2024-03-11T21:38:08.418455+0000 osd.0 (osd.0) 349 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:09.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:09.360 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:09.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[21512]: cluster 2024-03-11T21:38:07.493306+0000 mgr.y (mgr.14550) 595 : cluster 0 pgmap v939: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:09.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[21512]: cluster 2024-03-11T21:38:08.418132+0000 osd.0 (osd.0) 348 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:09.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[21512]: cluster 2024-03-11T21:38:08.418455+0000 osd.0 (osd.0) 349 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[29284]: cluster 2024-03-11T21:38:07.493306+0000 mgr.y (mgr.14550) 595 : cluster 0 pgmap v939: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:38:09.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[29284]: cluster 2024-03-11T21:38:08.418132+0000 osd.0 (osd.0) 348 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:09.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:09 smithi143 bash[29284]: cluster 2024-03-11T21:38:08.418455+0000 osd.0 (osd.0) 349 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:09.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:38:09.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:09.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:09.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:09.623 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:09.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:09.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:09.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:09.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:09.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:38:09.985 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:10.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:10.125 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:10 smithi203 bash[23654]: audit 2024-03-11T21:38:08.822592+0000 mgr.y (mgr.14550) 596 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:10.125 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:10 smithi203 bash[23654]: cluster 2024-03-11T21:38:09.057955+0000 mon.a (mon.0) 3100 : cluster 0 osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:38:10.142 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:10.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:10.270 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:10.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:10.385 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:10 smithi143 bash[21512]: audit 2024-03-11T21:38:08.822592+0000 mgr.y (mgr.14550) 596 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:10.386 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:10 smithi143 bash[21512]: cluster 2024-03-11T21:38:09.057955+0000 mon.a (mon.0) 3100 : cluster 0 osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:38:10.386 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:10 smithi143 bash[29284]: audit 2024-03-11T21:38:08.822592+0000 mgr.y (mgr.14550) 596 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:10.386 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:10 smithi143 bash[29284]: cluster 2024-03-11T21:38:09.057955+0000 mon.a (mon.0) 3100 : cluster 0 osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:38:10.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:10.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.526 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:10.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:10.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:10.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:10.788 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:10.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:11.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:11.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:11.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[21512]: cluster 2024-03-11T21:38:09.493966+0000 mgr.y (mgr.14550) 597 : cluster 0 pgmap v941: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[21512]: audit 2024-03-11T21:38:11.035905+0000 mon.c (mon.2) 1087 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[21512]: audit 2024-03-11T21:38:11.036907+0000 mon.a (mon.0) 3101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[21512]: audit 2024-03-11T21:38:11.038281+0000 mon.c (mon.2) 1088 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[21512]: audit 2024-03-11T21:38:11.039008+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[29284]: cluster 2024-03-11T21:38:09.493966+0000 mgr.y (mgr.14550) 597 : cluster 0 pgmap v941: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[29284]: audit 2024-03-11T21:38:11.035905+0000 mon.c (mon.2) 1087 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[29284]: audit 2024-03-11T21:38:11.036907+0000 mon.a (mon.0) 3101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[29284]: audit 2024-03-11T21:38:11.038281+0000 mon.c (mon.2) 1088 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:11 smithi143 bash[29284]: audit 2024-03-11T21:38:11.039008+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:11.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:11.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:11.299 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:11 smithi203 bash[23654]: cluster 2024-03-11T21:38:09.493966+0000 mgr.y (mgr.14550) 597 : cluster 0 pgmap v941: 270 pgs: 270 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:11.299 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:11 smithi203 bash[23654]: audit 2024-03-11T21:38:11.035905+0000 mon.c (mon.2) 1087 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.299 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:11 smithi203 bash[23654]: audit 2024-03-11T21:38:11.036907+0000 mon.a (mon.0) 3101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:11.299 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:11 smithi203 bash[23654]: audit 2024-03-11T21:38:11.038281+0000 mon.c (mon.2) 1088 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.299 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:11 smithi203 bash[23654]: audit 2024-03-11T21:38:11.039008+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-94"}]: dispatch 2024-03-11T21:38:11.314 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:11.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:11.457 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:11.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:11.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:11.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:38:11.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:38:11.603 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:38:11.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:38:11.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:11.712 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:11.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:11.836 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:11.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:11.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:12.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:12.099 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:12.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:12.226 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:12.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:12.361 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:12.413 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:12.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:12.489 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:12.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:12.617 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:12.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:12 smithi143 bash[21512]: cluster 2024-03-11T21:38:11.442957+0000 mon.a (mon.0) 3103 : cluster 0 osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:38:12.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:12 smithi143 bash[21512]: cluster 2024-03-11T21:38:11.494563+0000 mgr.y (mgr.14550) 598 : cluster 0 pgmap v943: 238 pgs: 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:12 smithi143 bash[29284]: cluster 2024-03-11T21:38:11.442957+0000 mon.a (mon.0) 3103 : cluster 0 osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:38:12.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:12 smithi143 bash[29284]: cluster 2024-03-11T21:38:11.494563+0000 mgr.y (mgr.14550) 598 : cluster 0 pgmap v943: 238 pgs: 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:12.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:12.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:12.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:12 smithi203 bash[23654]: cluster 2024-03-11T21:38:11.442957+0000 mon.a (mon.0) 3103 : cluster 0 osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:38:12.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:12 smithi203 bash[23654]: cluster 2024-03-11T21:38:11.494563+0000 mgr.y (mgr.14550) 598 : cluster 0 pgmap v943: 238 pgs: 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:12.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:12.873 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:12.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/1614570674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 msgr2=0x7f8fec110f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:12.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- 172.21.15.143:0/1614570674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec110f20 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400b0d0 tx=0x7f8fe400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/1614570674 shutdown_connections 2024-03-11T21:38:12.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- 172.21.15.143:0/1614570674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec110f20 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- 172.21.15.143:0/1614570674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104270 0x7f8fec104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- 172.21.15.143:0/1614570674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104c20 0x7f8fec108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/1614570674 >> 172.21.15.143:0/1614570674 conn(0x7f8fec0ffb00 msgr2=0x7f8fec101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/1614570674 shutdown_connections 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/1614570674 wait complete. 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 librados: starting msgr at 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 librados: starting objecter 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 Processor -- start 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- start start 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 librados: setting wanted keys 2024-03-11T21:38:12.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 librados: calling monclient init 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104270 0x7f8fec199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104c20 0x7f8fec19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec113850 con 0x7f8fec104270 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1136d0 con 0x7f8fec104c20 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1139d0 con 0x7f8fec109350 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff0ff6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104270 0x7f8fec199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:43174/0 (socket says 172.21.15.143:43174) 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 -- 172.21.15.143:0/3000207851 learned_addr learned my addr 172.21.15.143:0/3000207851 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8febfff700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104c20 0x7f8fec19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 -- 172.21.15.143:0/3000207851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104c20 msgr2=0x7f8fec19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104c20 0x7f8fec19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 -- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104270 msgr2=0x7f8fec199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104270 0x7f8fec199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 -- 172.21.15.143:0/3000207851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec19ed80 con 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff17f7700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4006070 tx=0x7f8fe4007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8fe9ffb700 10 client.?.objecter ms_handle_connect 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8fe9ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4021020 con 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8fe4004c50 con 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec19f010 con 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4005050 con 0x7f8fec109350 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 10 client.41822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec1a68f0 con 0x7f8fec109350 2024-03-11T21:38:12.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.932+0000 7f8ff325a700 1 librados: init done 2024-03-11T21:38:12.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec04faa0 con 0x7f8fec109350 2024-03-11T21:38:12.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8fe4024020 con 0x7f8fec109350 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 0x7f8fd4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(611..611 src has 87..611) v4 ==== 9648+0+0 (secure 0 0 0) 0x7f8fe400f020 con 0x7f8fec109350 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 10 client.41822.objecter ms_dispatch 0x7f8fec001550 osd_map(611..611 src has 87..611) v4 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 3 client.41822.objecter handle_osd_map got epochs [611,611] > 0 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 3 client.41822.objecter handle_osd_map decoding full epoch 611 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8ff0ff6700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 0x7f8fd4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 20 client.41822.objecter dump_active .. 0 homeless 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8fe9ffb700 10 client.41822.objecter ms_handle_connect 0x7f8fd406f820 2024-03-11T21:38:12.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.936+0000 7f8ff0ff6700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 0x7f8fd4071ce0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0005950 tx=0x7f8fe000b410 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:12.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:12.940+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8fe405cc20 con 0x7f8fec109350 2024-03-11T21:38:12.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:13.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:13.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:13.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.160+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8fec19b0d0 con 0x7f8fec109350 2024-03-11T21:38:13.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.253 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:13.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.484+0000 7f8fe9ffb700 1 -- 172.21.15.143:0/3000207851 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v612) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8fec19b0d0 con 0x7f8fec109350 2024-03-11T21:38:13.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 10 librados: watch_flush enter 2024-03-11T21:38:13.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 10 librados: watch_flush exit 2024-03-11T21:38:13.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 20 client.41822.objecter shutdown clearing up homeless session... 2024-03-11T21:38:13.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 10 client.41822.objecter successfully canceled tick 2024-03-11T21:38:13.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 msgr2=0x7f8fd4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:13.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 0x7f8fd4071ce0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0005950 tx=0x7f8fe000b410 comp rx=0 tx=0).stop 2024-03-11T21:38:13.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 msgr2=0x7f8fec19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:13.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.496+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4006070 tx=0x7f8fe4007a20 comp rx=0 tx=0).stop 2024-03-11T21:38:13.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 shutdown_connections 2024-03-11T21:38:13.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8fd406f820 0x7f8fd4071ce0 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fec104270 0x7f8fec199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fec109350 0x7f8fec19e640 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 --2- 172.21.15.143:0/3000207851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fec104c20 0x7f8fec19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 >> 172.21.15.143:0/3000207851 conn(0x7f8fec0ffb00 msgr2=0x7f8fec1001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 shutdown_connections 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.500+0000 7f8ff325a700 1 -- 172.21.15.143:0/3000207851 wait complete. 2024-03-11T21:38:13.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:13.504+0000 7f8ff325a700 1 librados: shutdown 2024-03-11T21:38:13.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:13.484 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[29284]: cluster 2024-03-11T21:38:12.464273+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[29284]: audit 2024-03-11T21:38:12.467685+0000 mon.c (mon.2) 1089 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[29284]: audit 2024-03-11T21:38:12.468378+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[29284]: audit 2024-03-11T21:38:13.167044+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.143:0/3000207851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[29284]: audit 2024-03-11T21:38:13.167669+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[21512]: cluster 2024-03-11T21:38:12.464273+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:38:13.497 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[21512]: audit 2024-03-11T21:38:12.467685+0000 mon.c (mon.2) 1089 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.498 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[21512]: audit 2024-03-11T21:38:12.468378+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.498 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[21512]: audit 2024-03-11T21:38:13.167044+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.143:0/3000207851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.498 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:13 smithi143 bash[21512]: audit 2024-03-11T21:38:13.167669+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.513 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:13.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.750 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:13 smithi203 bash[23654]: cluster 2024-03-11T21:38:12.464273+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:38:13.750 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:13 smithi203 bash[23654]: audit 2024-03-11T21:38:12.467685+0000 mon.c (mon.2) 1089 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.750 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:13 smithi203 bash[23654]: audit 2024-03-11T21:38:12.468378+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:13.751 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:13 smithi203 bash[23654]: audit 2024-03-11T21:38:13.167044+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.143:0/3000207851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.751 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:13 smithi203 bash[23654]: audit 2024-03-11T21:38:13.167669+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:13.764 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:13.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:14.027 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:14.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:14.158 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:14.228 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:14.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:14.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:14.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:14.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:14.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: cluster 2024-03-11T21:38:13.460113+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: audit 2024-03-11T21:38:13.463056+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: audit 2024-03-11T21:38:13.463290+0000 mon.a (mon.0) 3109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: cluster 2024-03-11T21:38:13.471220+0000 mon.a (mon.0) 3110 : cluster 0 osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: audit 2024-03-11T21:38:13.477545+0000 mon.c (mon.2) 1091 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:14.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:14 smithi203 bash[23654]: cluster 2024-03-11T21:38:13.508933+0000 mgr.y (mgr.14550) 599 : cluster 0 pgmap v946: 270 pgs: 26 creating+peering, 6 unknown, 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:14.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: cluster 2024-03-11T21:38:13.460113+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:14.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: audit 2024-03-11T21:38:13.463056+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:14.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: audit 2024-03-11T21:38:13.463290+0000 mon.a (mon.0) 3109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:14.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: cluster 2024-03-11T21:38:13.471220+0000 mon.a (mon.0) 3110 : cluster 0 osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:38:14.535 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: audit 2024-03-11T21:38:13.477545+0000 mon.c (mon.2) 1091 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[29284]: cluster 2024-03-11T21:38:13.508933+0000 mgr.y (mgr.14550) 599 : cluster 0 pgmap v946: 270 pgs: 26 creating+peering, 6 unknown, 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: cluster 2024-03-11T21:38:13.460113+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: audit 2024-03-11T21:38:13.463056+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: audit 2024-03-11T21:38:13.463290+0000 mon.a (mon.0) 3109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: cluster 2024-03-11T21:38:13.471220+0000 mon.a (mon.0) 3110 : cluster 0 osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: audit 2024-03-11T21:38:13.477545+0000 mon.c (mon.2) 1091 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:14.536 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:14 smithi143 bash[21512]: cluster 2024-03-11T21:38:13.508933+0000 mgr.y (mgr.14550) 599 : cluster 0 pgmap v946: 270 pgs: 26 creating+peering, 6 unknown, 238 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:14.550 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:14.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:14.679 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:14.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:14.809 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:14.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:14.940 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:15.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.069 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:15.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:15.177 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:15.199 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:15.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:15.326 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:15.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:15.457 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:15.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:15.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:15 smithi203 bash[23654]: cluster 2024-03-11T21:38:14.508142+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:38:15.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:15.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.713 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:15.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:15.845 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:15.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:15 smithi143 bash[21512]: cluster 2024-03-11T21:38:14.508142+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:38:15.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:15 smithi143 bash[29284]: cluster 2024-03-11T21:38:14.508142+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:38:15.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:15.977 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:16.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:16.109 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:16.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:16.238 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:16.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:16.368 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:16.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:16.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:16.623 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:16.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:38:16.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:16.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:16 smithi143 bash[29284]: cluster 2024-03-11T21:38:15.509832+0000 mgr.y (mgr.14550) 600 : cluster 0 pgmap v948: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 251 B/s wr, 1 op/s 2024-03-11T21:38:16.752 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:16 smithi143 bash[29284]: cluster 2024-03-11T21:38:16.441270+0000 mon.a (mon.0) 3112 : cluster 0 osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:38:16.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:16 smithi143 bash[21512]: cluster 2024-03-11T21:38:15.509832+0000 mgr.y (mgr.14550) 600 : cluster 0 pgmap v948: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 251 B/s wr, 1 op/s 2024-03-11T21:38:16.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:16 smithi143 bash[21512]: cluster 2024-03-11T21:38:16.441270+0000 mon.a (mon.0) 3112 : cluster 0 osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:38:16.769 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:16.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:16 smithi203 bash[23654]: cluster 2024-03-11T21:38:15.509832+0000 mgr.y (mgr.14550) 600 : cluster 0 pgmap v948: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 251 B/s wr, 1 op/s 2024-03-11T21:38:16.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:16 smithi203 bash[23654]: cluster 2024-03-11T21:38:16.441270+0000 mon.a (mon.0) 3112 : cluster 0 osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:38:16.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:17.000 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:17.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:17.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:17.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:17.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:17.300 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:17.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:17.432 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:17.491 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:17.491 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:17.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:17.547 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:17 smithi143 bash[29284]: cluster 2024-03-11T21:38:16.709709+0000 osd.6 (osd.6) 271 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:17.547 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:17 smithi143 bash[29284]: cluster 2024-03-11T21:38:16.710028+0000 osd.6 (osd.6) 272 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:17.547 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:17 smithi143 bash[21512]: cluster 2024-03-11T21:38:16.709709+0000 osd.6 (osd.6) 271 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:17.547 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:17 smithi143 bash[21512]: cluster 2024-03-11T21:38:16.710028+0000 osd.6 (osd.6) 272 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:17.562 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:17.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:17.690 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:17.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:17 smithi203 bash[23654]: cluster 2024-03-11T21:38:16.709709+0000 osd.6 (osd.6) 271 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:17.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:17 smithi203 bash[23654]: cluster 2024-03-11T21:38:16.710028+0000 osd.6 (osd.6) 272 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:17.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.822 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:17.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.949 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:18.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:18.076 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:18.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:18.206 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:18.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:38:18.307 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:18.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:18.333 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:18.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:18.458 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:18.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:18.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:18.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:18 smithi143 bash[21512]: cluster 2024-03-11T21:38:17.510701+0000 mgr.y (mgr.14550) 601 : cluster 0 pgmap v950: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:38:18.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:18 smithi143 bash[21512]: cluster 2024-03-11T21:38:17.534497+0000 mon.a (mon.0) 3113 : cluster 0 osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:38:18.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:18 smithi143 bash[29284]: cluster 2024-03-11T21:38:17.510701+0000 mgr.y (mgr.14550) 601 : cluster 0 pgmap v950: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:38:18.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:18 smithi143 bash[29284]: cluster 2024-03-11T21:38:17.534497+0000 mon.a (mon.0) 3113 : cluster 0 osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:38:18.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:18.700 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:18 smithi203 bash[23654]: cluster 2024-03-11T21:38:17.510701+0000 mgr.y (mgr.14550) 601 : cluster 0 pgmap v950: 270 pgs: 26 creating+peering, 244 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:38:18.700 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:18 smithi203 bash[23654]: cluster 2024-03-11T21:38:17.534497+0000 mon.a (mon.0) 3113 : cluster 0 osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:38:18.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:18.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:18.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:18.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:18.975 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:19.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:19.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:19.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:19.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:19.235 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:19.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:19.367 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:19.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:19.500 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:19.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:19.615 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:19 smithi203 bash[23654]: audit 2024-03-11T21:38:18.830661+0000 mgr.y (mgr.14550) 602 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:19.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:19.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:19.762 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:19.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:19.876 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:19 smithi143 bash[29284]: audit 2024-03-11T21:38:18.830661+0000 mgr.y (mgr.14550) 602 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:19.876 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:19 smithi143 bash[21512]: audit 2024-03-11T21:38:18.830661+0000 mgr.y (mgr.14550) 602 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:19.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:19.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:20.026 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:20.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:20.156 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:20.259 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:20.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:20.417 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:20.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:20.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:20.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- 172.21.15.143:0/21715719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c109370 msgr2=0x7fe19c110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- 172.21.15.143:0/21715719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c109370 0x7fe19c110f40 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7fe19400b0d0 tx=0x7fe19400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- 172.21.15.143:0/21715719 shutdown_connections 2024-03-11T21:38:20.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- 172.21.15.143:0/21715719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c109370 0x7fe19c110f40 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- 172.21.15.143:0/21715719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c104290 0x7fe19c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- 172.21.15.143:0/21715719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe19c104c40 0x7fe19c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- 172.21.15.143:0/21715719 >> 172.21.15.143:0/21715719 conn(0x7fe19c0ffb40 msgr2=0x7fe19c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- 172.21.15.143:0/21715719 shutdown_connections 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- 172.21.15.143:0/21715719 wait complete. 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 librados: starting msgr at 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 librados: starting objecter 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 Processor -- start 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- start start 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 librados: setting wanted keys 2024-03-11T21:38:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 librados: calling monclient init 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe19c104290 0x7fe19c199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c104c40 0x7fe19c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 0x7fe19c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1138d0 con 0x7fe19c104c40 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c113750 con 0x7fe19c104290 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe1a1344700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c113a50 con 0x7fe19c109370 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe193fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c104c40 0x7fe19c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe19b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 0x7fe19c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe19b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 0x7fe19c19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:59360/0 (socket says 172.21.15.143:59360) 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 learned_addr learned my addr 172.21.15.143:0/346405503 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:20.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.636+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe19c104290 0x7fe19c199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe19c104290 msgr2=0x7fe19c199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe19c104290 0x7fe19c199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c104c40 msgr2=0x7fe19c19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c104c40 0x7fe19c19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c19ee90 con 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe19c104c40 0x7fe19c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 0x7fe19c19e750 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fe194006070 tx=0x7fe194007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 10 client.?.objecter ms_handle_connect 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194021020 con 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe194004c50 con 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c19f120 con 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194005050 con 0x7fe19c109370 2024-03-11T21:38:20.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe1a1344700 10 client.41828.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:20.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c1a6a00 con 0x7fe19c109370 2024-03-11T21:38:20.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe1a1344700 1 librados: init done 2024-03-11T21:38:20.622 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe19c04faa0 con 0x7fe19c109370 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe194024020 con 0x7fe19c109370 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe18806f820 0x7fe188071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(616..616 src has 87..616) v4 ==== 9661+0+0 (secure 0 0 0) 0x7fe19400f020 con 0x7fe19c109370 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 osd_map(616..616 src has 87..616) v4 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 3 client.41828.objecter handle_osd_map got epochs [616,616] > 0 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 3 client.41828.objecter handle_osd_map decoding full epoch 616 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe18806f820 0x7fe188071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 20 client.41828.objecter dump_active .. 0 homeless 2024-03-11T21:38:20.623 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v616) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe1940901a0 con 0x7fe19c109370 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe18806f820 0x7fe188071ce0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009bf0 tx=0x7fe18c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.640+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe18806f820 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 10 client.41828.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 1 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe19c0629f0 0x7fe19c062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.624 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 1 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 0 1 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 4 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 10 client.41828.objecter _send_command 1 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1a9d60 con 0x7fe19c0629f0 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe19c0629f0 0x7fe19c062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe19c0629f0 0x7fe19c062dd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c0629f0 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1a9d60 con 0x7fe19c0629f0 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:38:20.625 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 10 client.41828.objecter _finish_command 1 = system:0 2024-03-11T21:38:20.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 4 2024-03-11T21:38:20.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.644+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 0 1 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 10 client.41828.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 2 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 2 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c19b1e0 osd=0 3 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 0 2 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 4 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 10 client.41828.objecter _send_command 2 2024-03-11T21:38:20.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.664+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c062470 con 0x7fe19c0629f0 2024-03-11T21:38:20.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c062470 con 0x7fe19c0629f0 2024-03-11T21:38:20.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 10 client.41828.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 4 2024-03-11T21:38:20.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 0 2 2024-03-11T21:38:20.647 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:38:20.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 10 client.41828.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe19c02c6b0 0x7fe19c02ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 1 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 4 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 10 client.41828.objecter _send_command 3 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c10bce0 con 0x7fe19c02c6b0 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe19c02c6b0 0x7fe19c02ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c02c6b0 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe19c02c6b0 0x7fe19c02ca90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c10bce0 con 0x7fe19c02c6b0 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 10 client.41828.objecter _finish_command 3 = system:0 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 4 2024-03-11T21:38:20.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.668+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 1 3 2024-03-11T21:38:20.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 10 client.41828.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 4 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 4 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c10b660 osd=1 3 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 1 4 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 4 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 10 client.41828.objecter _send_command 4 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c10b0e0 con 0x7fe19c02c6b0 2024-03-11T21:38:20.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c10b0e0 con 0x7fe19c02c6b0 2024-03-11T21:38:20.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 10 client.41828.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 4 2024-03-11T21:38:20.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 1 4 2024-03-11T21:38:20.669 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:38:20.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 10 client.41828.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 5 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe19c1abf20 0x7fe19c1ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 5 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 2 5 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 4 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 10 client.41828.objecter _send_command 5 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c02dcb0 con 0x7fe19c1abf20 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe19c1abf20 0x7fe19c1ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1abf20 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe19c1abf20 0x7fe19c1ae310 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c02dcb0 con 0x7fe19c1abf20 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 10 client.41828.objecter _finish_command 5 = system:0 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 4 2024-03-11T21:38:20.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.688+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 2 5 2024-03-11T21:38:20.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:20.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:20 smithi143 bash[29284]: cluster 2024-03-11T21:38:19.513238+0000 mgr.y (mgr.14550) 603 : cluster 0 pgmap v952: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:20.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:20 smithi143 bash[21512]: cluster 2024-03-11T21:38:19.513238+0000 mgr.y (mgr.14550) 603 : cluster 0 pgmap v952: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 10 client.41828.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 6 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 6 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c10b4e0 osd=2 3 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 2 6 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 4 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 10 client.41828.objecter _send_command 6 2024-03-11T21:38:20.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.708+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c02d280 con 0x7fe19c1abf20 2024-03-11T21:38:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c02d280 con 0x7fe19c1abf20 2024-03-11T21:38:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 10 client.41828.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 4 2024-03-11T21:38:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 2 6 2024-03-11T21:38:20.692 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 10 client.41828.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 7 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe19c1af9d0 0x7fe19c1b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 7 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 3 7 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 4 2024-03-11T21:38:20.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 10 client.41828.objecter _send_command 7 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1a9b70 con 0x7fe19c1af9d0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe19c1af9d0 0x7fe19c1b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1af9d0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe19c1af9d0 0x7fe19c1b1dc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1a9b70 con 0x7fe19c1af9d0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 10 client.41828.objecter _finish_command 7 = system:0 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 4 2024-03-11T21:38:20.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.712+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 3 7 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 10 client.41828.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 8 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 8 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a93b0 osd=3 3 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 3 8 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 4 2024-03-11T21:38:20.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 10 client.41828.objecter _send_command 8 2024-03-11T21:38:20.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.732+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c1a8e30 con 0x7fe19c1af9d0 2024-03-11T21:38:20.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c1a8e30 con 0x7fe19c1af9d0 2024-03-11T21:38:20.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 10 client.41828.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 4 2024-03-11T21:38:20.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 3 8 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 10 client.41828.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 9 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe19c1af530 0x7fe19c1b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 9 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 4 9 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 4 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 10 client.41828.objecter _send_command 9 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1a8d50 con 0x7fe19c1af530 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe19c1af530 0x7fe19c1b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1af530 2024-03-11T21:38:20.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.736+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe19c1af530 0x7fe19c1b3480 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.740+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1a8d50 con 0x7fe19c1af530 2024-03-11T21:38:20.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.740+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:38:20.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.740+0000 7fe198ff9700 10 client.41828.objecter _finish_command 9 = system:0 2024-03-11T21:38:20.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.740+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 4 2024-03-11T21:38:20.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.740+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 4 9 2024-03-11T21:38:20.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 10 client.41828.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 10 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 10 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a8be0 osd=4 3 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 4 10 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 4 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 10 client.41828.objecter _send_command 10 2024-03-11T21:38:20.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.756+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c1a8d50 con 0x7fe19c1af530 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c1a8d50 con 0x7fe19c1af530 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 10 client.41828.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 4 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 4 10 2024-03-11T21:38:20.740 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 10 client.41828.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 11 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe19c1b3f30 0x7fe19c1b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 11 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 5 11 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 4 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 10 client.41828.objecter _send_command 11 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1bb5c0 con 0x7fe19c1b3f30 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe19c1b3f30 0x7fe19c1b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1b3f30 2024-03-11T21:38:20.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.760+0000 7fe193fff700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe19c1b3f30 0x7fe19c1b4350 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.764+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1bb5c0 con 0x7fe19c1b3f30 2024-03-11T21:38:20.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.764+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:38:20.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.764+0000 7fe198ff9700 10 client.41828.objecter _finish_command 11 = system:0 2024-03-11T21:38:20.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.764+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 4 2024-03-11T21:38:20.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.764+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 5 11 2024-03-11T21:38:20.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 10 client.41828.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 12 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 12 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1a8d50 osd=5 3 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 5 12 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 4 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 10 client.41828.objecter _send_command 12 2024-03-11T21:38:20.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.780+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c1b8ca0 con 0x7fe19c1b3f30 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c1b8ca0 con 0x7fe19c1b3f30 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe198ff9700 10 client.41828.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 4 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 5 12 2024-03-11T21:38:20.765 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 10 client.41828.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 13 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe19c1b9320 0x7fe19c1bb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 13 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 6 13 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 4 2024-03-11T21:38:20.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 10 client.41828.objecter _send_command 13 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1be780 con 0x7fe19c1b9320 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.784+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe19c1b9320 0x7fe19c1bb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1b9320 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe19b7fe700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe19c1b9320 0x7fe19c1bb710 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1be780 con 0x7fe19c1b9320 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:38:20.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 10 client.41828.objecter _finish_command 13 = system:0 2024-03-11T21:38:20.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 4 2024-03-11T21:38:20.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.788+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 6 13 2024-03-11T21:38:20.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:20.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 10 client.41828.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 14 2024-03-11T21:38:20.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 14 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1b8ca0 osd=6 3 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 6 14 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 4 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 10 client.41828.objecter _send_command 14 2024-03-11T21:38:20.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.808+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c1bd510 con 0x7fe19c1b9320 2024-03-11T21:38:20.791 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:20 smithi203 bash[23654]: cluster 2024-03-11T21:38:19.513238+0000 mgr.y (mgr.14550) 603 : cluster 0 pgmap v952: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c1bd510 con 0x7fe19c1b9320 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 10 client.41828.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 4 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 6 14 2024-03-11T21:38:20.793 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 10 client.41828.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 15 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe19c1bdc80 0x7fe19c1be0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 15 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 7 15 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 4 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 10 client.41828.objecter _send_command 15 2024-03-11T21:38:20.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe19c1bd060 con 0x7fe19c1bdc80 2024-03-11T21:38:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe19c1bdc80 0x7fe19c1be0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe198ff9700 10 client.41828.objecter ms_handle_connect 0x7fe19c1bdc80 2024-03-11T21:38:20.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.812+0000 7fe19affd700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe19c1bdc80 0x7fe19c1be0e0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.816+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe19c1bd060 con 0x7fe19c1bdc80 2024-03-11T21:38:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.816+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:38:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.816+0000 7fe198ff9700 10 client.41828.objecter _finish_command 15 = system:0 2024-03-11T21:38:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.816+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 4 2024-03-11T21:38:20.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.816+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 7 15 2024-03-11T21:38:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign -1 16 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter _get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_remove -1 16 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter get_session s=0x7fe19c1bd510 osd=7 3 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 15 client.41828.objecter _session_command_op_assign 7 16 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 4 2024-03-11T21:38:20.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter _send_command 16 2024-03-11T21:38:20.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe19c1bd020 con 0x7fe19c1bdc80 2024-03-11T21:38:20.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe198ff9700 1 -- 172.21.15.143:0/346405503 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe19c1bd020 con 0x7fe19c1bdc80 2024-03-11T21:38:20.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe198ff9700 10 client.41828.objecter ms_dispatch 0x7fe19c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:20.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe198ff9700 10 client.41828.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:20.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe198ff9700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 4 2024-03-11T21:38:20.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe198ff9700 15 client.41828.objecter _session_command_op_remove 7 16 2024-03-11T21:38:20.818 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 librados: watch_flush enter 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 librados: watch_flush exit 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.0 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe19c0629f0 msgr2=0x7fe19c062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fe19c0629f0 0x7fe19c062dd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c19b1e0 osd=0 1 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.1 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe19c02c6b0 msgr2=0x7fe19c02ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fe19c02c6b0 0x7fe19c02ca90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b660 osd=1 1 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.2 2024-03-11T21:38:20.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe19c1abf20 msgr2=0x7fe19c1ae310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fe19c1abf20 0x7fe19c1ae310 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c10b4e0 osd=2 1 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.3 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe19c1af9d0 msgr2=0x7fe19c1b1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fe19c1af9d0 0x7fe19c1b1dc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a93b0 osd=3 1 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.4 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 reap_dead start 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe19c1af530 msgr2=0x7fe19c1b3480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fe19c1af530 0x7fe19c1b3480 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8be0 osd=4 1 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.5 2024-03-11T21:38:20.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 reap_dead start 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe19c1b3f30 msgr2=0x7fe19c1b4350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fe19c1b3f30 0x7fe19c1b4350 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1a8d50 osd=5 1 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.6 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe19c1b9320 msgr2=0x7fe19c1bb710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fe19c1b9320 0x7fe19c1bb710 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1b8ca0 osd=6 1 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter close_session for osd.7 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe19c1bdc80 msgr2=0x7fe19c1be0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fe19c1bdc80 0x7fe19c1be0e0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter put_session s=0x7fe19c1bd510 osd=7 1 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 20 client.41828.objecter shutdown clearing up homeless session... 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 10 client.41828.objecter successfully canceled tick 2024-03-11T21:38:20.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe18806f820 msgr2=0x7fe188071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe18806f820 0x7fe188071ce0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009bf0 tx=0x7fe18c009260 comp rx=0 tx=0).stop 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 msgr2=0x7fe19c19e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 --2- 172.21.15.143:0/346405503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe19c109370 0x7fe19c19e750 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fe194006070 tx=0x7fe194007a20 comp rx=0 tx=0).stop 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe19b7fe700 1 -- 172.21.15.143:0/346405503 reap_dead start 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 shutdown_connections 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 >> 172.21.15.143:0/346405503 conn(0x7fe19c0ffb40 msgr2=0x7fe19c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 shutdown_connections 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 -- 172.21.15.143:0/346405503 wait complete. 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:20.836+0000 7fe1a1344700 1 librados: shutdown 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:20.822 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:20.823 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:20.823 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:20.823 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:20.823 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:20.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:21.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:21.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:21.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:21.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:21.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:21.318 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:21.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:21.449 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:21.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:21.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:21.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:21.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[21512]: cluster 2024-03-11T21:38:20.482831+0000 osd.2 (osd.2) 309 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:21.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[21512]: cluster 2024-03-11T21:38:20.483076+0000 osd.2 (osd.2) 310 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:21.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[21512]: cluster 2024-03-11T21:38:20.548356+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:38:21.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[21512]: audit 2024-03-11T21:38:20.645893+0000 mon.c (mon.2) 1092 : audit 0 from='client.? 172.21.15.143:0/346405503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:21.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[29284]: cluster 2024-03-11T21:38:20.482831+0000 osd.2 (osd.2) 309 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:21.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[29284]: cluster 2024-03-11T21:38:20.483076+0000 osd.2 (osd.2) 310 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:21.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[29284]: cluster 2024-03-11T21:38:20.548356+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:38:21.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:21 smithi143 bash[29284]: audit 2024-03-11T21:38:20.645893+0000 mon.c (mon.2) 1092 : audit 0 from='client.? 172.21.15.143:0/346405503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:21.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:21.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:21.824 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:21 smithi203 bash[23654]: cluster 2024-03-11T21:38:20.482831+0000 osd.2 (osd.2) 309 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:21.824 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:21 smithi203 bash[23654]: cluster 2024-03-11T21:38:20.483076+0000 osd.2 (osd.2) 310 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:21.824 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:21 smithi203 bash[23654]: cluster 2024-03-11T21:38:20.548356+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:38:21.824 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:21 smithi203 bash[23654]: audit 2024-03-11T21:38:20.645893+0000 mon.c (mon.2) 1092 : audit 0 from='client.? 172.21.15.143:0/346405503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:21.837 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:21.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:21.973 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:22.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:22.103 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:22.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:22.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.347 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:22.364 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:22.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:22.495 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:22.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:22.626 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:22.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:22 smithi143 bash[29284]: cluster 2024-03-11T21:38:21.513824+0000 mgr.y (mgr.14550) 604 : cluster 0 pgmap v954: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:22.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:22 smithi143 bash[29284]: audit 2024-03-11T21:38:21.873555+0000 mon.c (mon.2) 1093 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.687 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:22 smithi143 bash[21512]: cluster 2024-03-11T21:38:21.513824+0000 mgr.y (mgr.14550) 604 : cluster 0 pgmap v954: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:22.688 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:22 smithi143 bash[21512]: audit 2024-03-11T21:38:21.873555+0000 mon.c (mon.2) 1093 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:22.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:22 smithi203 bash[23654]: cluster 2024-03-11T21:38:21.513824+0000 mgr.y (mgr.14550) 604 : cluster 0 pgmap v954: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-03-11T21:38:22.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:22 smithi203 bash[23654]: audit 2024-03-11T21:38:21.873555+0000 mon.c (mon.2) 1093 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.755 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:22.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.886 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:22.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:23.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.140 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:23.243 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:23.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:23.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:23.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:23.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:23.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:23.663 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:23.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:23.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:23 smithi143 bash[29284]: cluster 2024-03-11T21:38:22.986380+0000 osd.7 (osd.7) 346 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:23.777 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:23 smithi143 bash[29284]: cluster 2024-03-11T21:38:22.986731+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:23.778 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:23 smithi143 bash[21512]: cluster 2024-03-11T21:38:22.986380+0000 osd.7 (osd.7) 346 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:23.778 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:23 smithi143 bash[21512]: cluster 2024-03-11T21:38:22.986731+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:23.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:23.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:23 smithi203 bash[23654]: cluster 2024-03-11T21:38:22.986380+0000 osd.7 (osd.7) 346 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:23.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:23 smithi203 bash[23654]: cluster 2024-03-11T21:38:22.986731+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:23.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:24.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:24.054 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:24.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:24.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:24.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:24.310 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:24.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:24.442 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:24.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:38:24.546 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[150181]: debug 2024-03-11T21:38:24.560+0000 7f3a2cb81700 -1 snap_mapper.add_oid found existing snaps mapped on 118:6d6b5e77:test-rados-api-smithi143-71773-97::foo:21, removing 2024-03-11T21:38:24.546 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[43348]: debug 2024-03-11T21:38:24.560+0000 7f54a8603700 -1 snap_mapper.add_oid found existing snaps mapped on 118:6d6b5e77:test-rados-api-smithi143-71773-97::foo:21, removing 2024-03-11T21:38:24.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:24.572 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:24.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:24.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:24.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/2344280180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4107f30 msgr2=0x7ff3f4108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:24.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/2344280180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4107f30 0x7ff3f4108310 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80098f0 tx=0x7ff3e8009c00 comp rx=0 tx=0).stop 2024-03-11T21:38:24.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/2344280180 shutdown_connections 2024-03-11T21:38:24.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/2344280180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4102880 0x7ff3f410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/2344280180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4107f30 0x7ff3f4108310 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/2344280180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4101ec0 0x7ff3f4102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/2344280180 >> 172.21.15.143:0/2344280180 conn(0x7ff3f4078eb0 msgr2=0x7ff3f40792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/2344280180 shutdown_connections 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/2344280180 wait complete. 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 librados: starting msgr at 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 librados: starting objecter 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 Processor -- start 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- start start 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 librados: setting wanted keys 2024-03-11T21:38:24.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 librados: calling monclient init 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4101ec0 0x7ff3f40740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4107f30 0x7ff3f4074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41116a0 con 0x7ff3f4107f30 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4111520 con 0x7ff3f4101ec0 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4111820 con 0x7ff3f4102880 2024-03-11T21:38:24.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47276/0 (socket says 172.21.15.143:47276) 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 -- 172.21.15.143:0/3033689903 learned_addr learned my addr 172.21.15.143:0/3033689903 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f2d9d700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4107f30 0x7ff3f4074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f259c700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4101ec0 0x7ff3f40740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 -- 172.21.15.143:0/3033689903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4101ec0 msgr2=0x7ff3f40740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4101ec0 0x7ff3f40740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 -- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4107f30 msgr2=0x7ff3f4074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4107f30 0x7ff3f4074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 -- 172.21.15.143:0/3033689903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f41a6190 con 0x7ff3f4102880 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f1d9b700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400ec10 tx=0x7ff3e400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3e37fe700 10 client.?.objecter ms_handle_connect 0x7ff3f4102880 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3e37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:24.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400cc90 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f41a6480 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff3e400ce30 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4010430 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 10 client.41834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f41a69c0 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.768+0000 7ff3f8b3a700 1 librados: init done 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f404faa0 con 0x7ff3f4102880 2024-03-11T21:38:24.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff3e4003ef0 con 0x7ff3f4102880 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 0x7ff3dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(617..617 src has 87..617) v4 ==== 9661+0+0 (secure 0 0 0) 0x7ff3e408edc0 con 0x7ff3f4102880 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 10 client.41834.objecter ms_dispatch 0x7ff3f4001550 osd_map(617..617 src has 87..617) v4 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 3 client.41834.objecter handle_osd_map got epochs [617,617] > 0 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 3 client.41834.objecter handle_osd_map decoding full epoch 617 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3f259c700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 0x7ff3dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 20 client.41834.objecter dump_active .. 0 homeless 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3e37fe700 10 client.41834.objecter ms_handle_connect 0x7ff3dc06f820 2024-03-11T21:38:24.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.772+0000 7ff3f259c700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 0x7ff3dc071ce0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009fd0 tx=0x7ff3e8019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.776+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff3e405b9f0 con 0x7ff3f4102880 2024-03-11T21:38:24.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:24 smithi203 bash[23654]: cluster 2024-03-11T21:38:23.516167+0000 mgr.y (mgr.14550) 605 : cluster 0 pgmap v955: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 723 B/s wr, 2 op/s 2024-03-11T21:38:24.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:24 smithi203 bash[23654]: cluster 2024-03-11T21:38:23.575544+0000 mon.a (mon.0) 3115 : cluster 0 osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:38:24.774 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:38:24 smithi203 bash[43837]: debug 2024-03-11T21:38:24.555+0000 7feb03132700 -1 snap_mapper.add_oid found existing snaps mapped on 118:6d6b5e77:test-rados-api-smithi143-71773-97::foo:21, removing 2024-03-11T21:38:24.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:24.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[29284]: cluster 2024-03-11T21:38:23.516167+0000 mgr.y (mgr.14550) 605 : cluster 0 pgmap v955: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 723 B/s wr, 2 op/s 2024-03-11T21:38:24.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[29284]: cluster 2024-03-11T21:38:23.575544+0000 mon.a (mon.0) 3115 : cluster 0 osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:38:24.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[21512]: cluster 2024-03-11T21:38:23.516167+0000 mgr.y (mgr.14550) 605 : cluster 0 pgmap v955: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 723 B/s wr, 2 op/s 2024-03-11T21:38:24.806 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:24 smithi143 bash[21512]: cluster 2024-03-11T21:38:23.575544+0000 mon.a (mon.0) 3115 : cluster 0 osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:38:24.817 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:24.920 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:24.940 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:24.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:24.996+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff3f41a6c80 con 0x7ff3f4102880 2024-03-11T21:38:24.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.000+0000 7ff3e37fe700 1 -- 172.21.15.143:0/3033689903 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v617) v1 ==== 81+0+60857 (secure 0 0 0) 0x7ff3e405f440 con 0x7ff3f4102880 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "epoch": 617, 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "fsid": "eebe2e36-dfeb-11ee-95c8-87774f69a715", 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "created": "2024-03-11T21:12:47.384719+0000", 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "modified": "2024-03-11T21:38:23.559944+0000", 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "last_up_change": "2024-03-11T21:31:36.320182+0000", 2024-03-11T21:38:24.982 INFO:teuthology.orchestra.run.smithi143.stdout: "last_in_change": "2024-03-11T21:21:55.456354+0000", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_num": 5806080, 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_set": [ 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "nodeep-scrub", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "noscrub", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "pglog_hardlimit", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "purged_snapdirs", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_deletes", 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "sortbitwise" 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_version": 111, 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:38:24.983 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_snapshot": "", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_max": 304, 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "max_osd": 8, 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "min_compat_client": "luminous", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "require_osd_release": "squid", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "allow_crimson": false, 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "pools": [ 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 1, 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".mgr", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:18:08.036690+0000", 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:24.984 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 11, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 11, 2024-03-11T21:38:24.985 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 11, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 11, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 11, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "596", 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:24.986 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "593", 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:24.987 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:24.988 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_max": 32, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_min": 1 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.989 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "mgr": {} 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2.1800000667572021, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2.1800000667572021, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:24.990 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 2, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": ".rgw.root", 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:01.432928+0000", 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:24.991 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:24.992 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "78", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:24.993 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:24.994 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:24.995 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:24.996 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 3, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:03.441476+0000", 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:24.997 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:24.998 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "80", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:24.999 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.000 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.001 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.002 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 4, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "datapool", 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:04.441432+0000", 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.003 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 3, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 3, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 3, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 3, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 3, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.004 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "87", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 2, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 87, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.005 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.006 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.007 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "rbd": {} 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2.6500000953674316, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2.6500000953674316, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.008 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 5, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:05.388543+0000", 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.009 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.010 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "82", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.011 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.012 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.013 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.25, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.25, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.25, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.25, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.014 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 6, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:23:07.642696+0000", 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.015 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.016 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "84", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.017 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.018 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_bias": 4 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.019 INFO:teuthology.orchestra.run.smithi143.stdout: "rgw": {} 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.020 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 9, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:36.978109+0000", 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.021 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.022 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "148", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.023 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.024 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.025 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.026 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 39, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "SnapListsmithi143-72399-1", 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:28:49.753089+0000", 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.027 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.028 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "153", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.029 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.030 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": {}, 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.031 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.5, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.5, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.5, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.5, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 118, 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-6", 2024-03-11T21:38:25.032 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:29:30.235634+0000", 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 8193, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.033 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "613", 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "454", 2024-03-11T21:38:25.034 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "454", 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "454", 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 33, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 613, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.035 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.036 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "options": { 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "fingerprint_algorithm": "sha1" 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.037 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 1.75, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 1.75, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 1.75, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 1.75, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.038 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 304, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_name": "test-rados-api-smithi143-71773-96", 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "create_time": "2024-03-11T21:38:11.919746+0000", 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "flags": 1, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "flags_names": "hashpspool", 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "type": 1, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 3, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "min_size": 2, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_rule": 0, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "object_hash": 2, 2024-03-11T21:38:25.039 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num": 32, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num": 32, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_placement_num_target": 32, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_target": 32, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_num_pending": 32, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "last_pg_merge_meta": { 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "source_pgid": "0.0", 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "ready_epoch": 0, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_started": 0, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "last_epoch_clean": 0, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "source_version": "0'0", 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "target_version": "0'0" 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.040 INFO:teuthology.orchestra.run.smithi143.stdout: "last_change": "613", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend": "0", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "auid": 0, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_seq": 0, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "snap_epoch": 0, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "pool_snaps": [], 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps": "[]", 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_bytes": 0, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "quota_max_objects": 0, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "tiers": [], 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "tier_of": -1, 2024-03-11T21:38:25.041 INFO:teuthology.orchestra.run.smithi143.stdout: "read_tier": -1, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "write_tier": -1, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_mode": "none", 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_bytes": 0, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "target_max_objects": 0, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_flush_age": 0, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "cache_min_evict_age": 0, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profile": "", 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_params": { 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "none" 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_period": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_count": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "use_gmt_hitset": true, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "grade_table": [], 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "stripe_width": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "expected_num_objects": 0, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "fast_read": false, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "options": {}, 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "application_metadata": { 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: "rados": {} 2024-03-11T21:38:25.043 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "read_balance": { 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "score_type": "Fair distribution", 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "score_acting": 2, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "score_stable": 2, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "optimal_score": 1, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_acting": 2, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "raw_score_stable": 2, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity": 1, 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:25.044 INFO:teuthology.orchestra.run.smithi143.stdout: "osds": [ 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d", 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 8, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.045 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6802", 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6803", 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.046 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6804", 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6805", 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.047 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6808", 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6809", 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.048 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6806", 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6807", 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2903432023 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.049 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6803/2903432023", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6805/2903432023", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6809/2903432023", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6807/2903432023", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "2948dc44-68d3-4b3d-92bf-2b0c46374f02", 2024-03-11T21:38:25.050 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 13, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 246, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 295, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 247, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.051 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6810", 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6811", 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.052 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6812", 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6813", 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.053 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6816", 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6817", 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.054 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6814", 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6815", 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 4235040845 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6811/4235040845", 2024-03-11T21:38:25.055 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6813/4235040845", 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6817/4235040845", 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6815/4235040845", 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "7217589a-5835-407b-91a7-f5c055a48f3d", 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.056 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 19, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6818", 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.057 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6819", 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6820", 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.058 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6821", 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6824", 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.059 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6825", 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6822", 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.060 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6823", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2592131625 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6819/2592131625", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6821/2592131625", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6825/2592131625", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6823/2592131625", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.061 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "a3e7177f-a5a0-44ad-a65d-9aeac3a680cc", 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 27, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.062 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6826", 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6827", 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.063 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6828", 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6829", 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.064 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6832", 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6833", 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.065 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6830", 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.143:6831", 2024-03-11T21:38:25.066 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2146571980 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.143:6827/2146571980", 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.143:6829/2146571980", 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.143:6833/2146571980", 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.143:6831/2146571980", 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.067 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "d7b9e088-d377-4335-8863-11d581828f7e", 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 35, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.068 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6800", 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6801", 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.069 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6802", 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6803", 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.070 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6806", 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6807", 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.071 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6804", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6805", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2090835691 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6801/2090835691", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6803/2090835691", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6807/2090835691", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6805/2090835691", 2024-03-11T21:38:25.072 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "b3b2a75c-68a5-4e00-8068-622e1ebbaa01", 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 46, 2024-03-11T21:38:25.073 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6808", 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6809", 2024-03-11T21:38:25.074 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6810", 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6811", 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.075 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6814", 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6815", 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.076 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6812", 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6813", 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 2432038345 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.077 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6809/2432038345", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6811/2432038345", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2432038345", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6813/2432038345", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "ceffb234-db0c-4ffa-a142-9b201df9b241", 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.078 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 56, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6816", 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.079 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6817", 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6818", 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.080 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6819", 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6822", 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.081 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6823", 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6820", 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6821", 2024-03-11T21:38:25.082 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 513748834 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6817/513748834", 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6819/513748834", 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6823/513748834", 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6821/513748834", 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.083 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "uuid": "88b53171-69d2-4922-a787-e63d44d01c60", 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "up": 1, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "in": 1, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "weight": 1, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_affinity": 1, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_begin": 0, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "last_clean_end": 0, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "up_from": 66, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "up_thru": 611, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "down_at": 0, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "lost_at": 0, 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addrs": { 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.084 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6824", 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6825", 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addrs": { 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.085 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6826", 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6827", 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.086 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addrs": { 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6830", 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6831", 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.087 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addrs": { 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "addrvec": [ 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v2", 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6828", 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "172.21.15.203:6829", 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1251497952 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.088 INFO:teuthology.orchestra.run.smithi143.stdout: "public_addr": "172.21.15.203:6825/1251497952", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "cluster_addr": "172.21.15.203:6827/1251497952", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_back_addr": "172.21.15.203:6831/1251497952", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "heartbeat_front_addr": "172.21.15.203:6829/1251497952", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "state": [ 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "exists", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "up" 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_xinfo": [ 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 0, 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.089 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:38:08.418460+0000", 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 1, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.090 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:37:53.927771+0000", 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 247 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 2, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:38:20.483099+0000", 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.091 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 3, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:37:56.580288+0000", 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 4, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.092 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:37:24.962604+0000", 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 5, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:37:54.755322+0000", 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.093 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 6, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:38:16.710036+0000", 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "osd": 7, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "down_stamp": "0.000000", 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_probability": 0, 2024-03-11T21:38:25.094 INFO:teuthology.orchestra.run.smithi143.stdout: "laggy_interval": 0, 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "features": 4540701547738038271, 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "old_weight": 0, 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "last_purged_snaps_scrub": "2024-03-11T21:38:22.986738+0000", 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "dead_epoch": 0 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap": [], 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_items": [ 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "2.18", 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:38:25.095 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.10", 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.18", 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.096 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.1d", 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 2 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.097 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "9.1e", 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 7, 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 4 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.2", 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.098 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.9", 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "pgid": "39.17", 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: "mappings": [ 2024-03-11T21:38:25.099 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "from": 3, 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "to": 1 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_upmap_primaries": [], 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "pg_temp": [], 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "primary_temp": [], 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "blocklist": { 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3638739149": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.100 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4274199357": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3217951344": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/3912416566": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/906834629": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2791822486": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2190104670": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/4171894415": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/3787195463": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2845155916": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6801/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2913593434": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/143435387": "2024-03-12T21:13:53.401579+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:0/2930358289": "2024-03-12T21:23:57.022184+0000", 2024-03-11T21:38:25.101 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/1839244747": "2024-03-12T21:13:19.169870+0000", 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "172.21.15.143:6800/3638739149": "2024-03-12T21:23:57.022184+0000" 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "range_blocklist": {}, 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "erasure_code_profiles": { 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "default": { 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "crush-failure-domain": "osd", 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "k": "2", 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "m": "1", 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "plugin": "jerasure", 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "technique": "reed_sol_van" 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: "removed_snaps_queue": [ 2024-03-11T21:38:25.102 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 28, 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 5, 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 35, 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 1, 2024-03-11T21:38:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 89, 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 4, 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 7, 2024-03-11T21:38:25.104 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 137, 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 3 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 156, 2024-03-11T21:38:25.105 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 165, 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.106 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 173, 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 178, 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.107 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 2, 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 185, 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 3, 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.108 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 198, 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 4, 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 4 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 210, 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 7, 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.109 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 230, 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 8, 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.110 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 241, 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 10, 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 244, 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 11, 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 2 2024-03-11T21:38:25.111 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "pool": 256, 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "snaps": [ 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "begin": 14, 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "length": 1 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: ], 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "new_removed_snaps": [], 2024-03-11T21:38:25.112 INFO:teuthology.orchestra.run.smithi143.stdout: "new_purged_snaps": [], 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "crush_node_flags": {}, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "device_class_flags": {}, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode": { 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_enabled": false, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_bucket_count": 0, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: "stretch_mode_bucket": 0 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:38:25.113 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:38:25.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 10 librados: watch_flush enter 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 10 librados: watch_flush exit 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 20 client.41834.objecter shutdown clearing up homeless session... 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 10 client.41834.objecter successfully canceled tick 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 msgr2=0x7ff3dc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 0x7ff3dc071ce0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009fd0 tx=0x7ff3e8019040 comp rx=0 tx=0).stop 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 msgr2=0x7ff3f4074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400ec10 tx=0x7ff3e400ef20 comp rx=0 tx=0).stop 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 shutdown_connections 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff3dc06f820 0x7ff3dc071ce0 unknown :-1 s=CLOSED pgs=3404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3f4107f30 0x7ff3f4074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f4101ec0 0x7ff3f40740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 --2- 172.21.15.143:0/3033689903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3f4102880 0x7ff3f4074620 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 >> 172.21.15.143:0/3033689903 conn(0x7ff3f4078eb0 msgr2=0x7ff3f4108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 shutdown_connections 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 -- 172.21.15.143:0/3033689903 wait complete. 2024-03-11T21:38:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:25.004+0000 7ff3f8b3a700 1 librados: shutdown 2024-03-11T21:38:25.139 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:25.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:25.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:25.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:25.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:25.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:25.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:25.661 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:25.764 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:25.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:25 smithi203 bash[23654]: audit 2024-03-11T21:38:24.605001+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:25 smithi203 bash[23654]: audit 2024-03-11T21:38:24.605391+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:25 smithi203 bash[23654]: audit 2024-03-11T21:38:24.605985+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:25 smithi203 bash[23654]: audit 2024-03-11T21:38:24.606283+0000 mon.a (mon.0) 3117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:25 smithi203 bash[23654]: audit 2024-03-11T21:38:25.004085+0000 mon.c (mon.2) 1096 : audit 0 from='client.? 172.21.15.143:0/3033689903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:38:25.785 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:25.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[29284]: audit 2024-03-11T21:38:24.605001+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[29284]: audit 2024-03-11T21:38:24.605391+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[29284]: audit 2024-03-11T21:38:24.605985+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[29284]: audit 2024-03-11T21:38:24.606283+0000 mon.a (mon.0) 3117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[29284]: audit 2024-03-11T21:38:25.004085+0000 mon.c (mon.2) 1096 : audit 0 from='client.? 172.21.15.143:0/3033689903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:38:25.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[21512]: audit 2024-03-11T21:38:24.605001+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[21512]: audit 2024-03-11T21:38:24.605391+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[21512]: audit 2024-03-11T21:38:24.605985+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[21512]: audit 2024-03-11T21:38:24.606283+0000 mon.a (mon.0) 3117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-96"}]: dispatch 2024-03-11T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:25 smithi143 bash[21512]: audit 2024-03-11T21:38:25.004085+0000 mon.c (mon.2) 1096 : audit 0 from='client.? 172.21.15.143:0/3033689903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:38:25.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:26.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:26.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24670.0:2 20.1 20:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:41.679241+0000", 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 584.45615189600005, 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7793374689999997, 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.121 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24670", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/503308991", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679241+0000", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679241+0000", 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679242+0000", 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.57e-06 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679248+0000", 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5319999999999997e-06 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679260+0000", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2305e-05 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679269+0000", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8459999999999997e-06 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679325+0000", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5720999999999997e-05 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:41.679332+0000", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.34e-06 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.437566+0000", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.75823368 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.437688+0000", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012233900000000001 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:38:26.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.437822+0000", 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013405000000000001 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.444914+0000", 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070921339999999999 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.447195+0000", 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022807320000000002 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458513+0000", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011318376999999999 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458550+0000", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6968999999999998e-05 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458578+0000", 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7874000000000001e-05 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.127 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14952.0:2 31.15 31:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:49.786150+0000", 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 576.34924251999996, 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6567776080000001, 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14952", 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/820536278", 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.128 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786150+0000", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786150+0000", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786153+0000", 2024-03-11T21:38:26.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0800000000000002e-06 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786170+0000", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7183000000000001e-05 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786207+0000", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7132999999999998e-05 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786230+0000", 2024-03-11T21:38:26.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3045999999999999e-05 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786319+0000", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8120000000000003e-05 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:49.786330+0000", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1365e-05 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.442738+0000", 2024-03-11T21:38:26.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6564083059999999 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.442928+0000", 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018937500000000001 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24598.0:2 16.d 16:b25ed315:test-rados-api-smithi143-71641-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:51.325720+0000", 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 574.80967292599996, 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.087458847000000006, 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.132 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24598", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1041041831", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325720+0000", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325720+0000", 2024-03-11T21:38:26.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325722+0000", 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2230000000000001e-06 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325729+0000", 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3819999999999998e-06 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325743+0000", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3542e-05 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325755+0000", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2e-05 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325824+0000", 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9646000000000004e-05 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.325834+0000", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0043e-05 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.399966+0000", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.074131931999999998 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.400114+0000", 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014783600000000001 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.400335+0000", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220485 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.402535+0000", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022006590000000002 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.411805+0000", 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0092701020000000005 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.413068+0000", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001262717 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.413134+0000", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5821999999999994e-05 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.413178+0000", 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4458e-05 2024-03-11T21:38:26.138 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14952.0:3 31.15 31:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:51.458582+0000", 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 574.67681038499995, 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.052883004999999997, 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14952", 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/820536278", 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:38:26.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458582+0000", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458582+0000", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458583+0000", 2024-03-11T21:38:26.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8899999999999998e-07 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458586+0000", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9960000000000002e-06 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458592+0000", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2759999999999997e-06 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458597+0000", 2024-03-11T21:38:26.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9259999999999999e-06 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458634+0000", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6702999999999997e-05 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458709+0000", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5541000000000001e-05 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.458801+0000", 2024-03-11T21:38:26.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1880999999999997e-05 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.459394+0000", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00059241700000000003 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.459795+0000", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00040133299999999997 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.511363+0000", 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051567901999999999 2024-03-11T21:38:26.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.511407+0000", 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3819999999999997e-05 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:51.511465+0000", 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8320999999999999e-05 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.144 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14904.0:52 30.1e 30:7d23f061:test-rados-api-smithi143-72044-7::testobj:head [read 0~8] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:52.271874+0000", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 573.86351840999998, 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046548762, 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14904", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2694737065", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 52 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.271874+0000", 2024-03-11T21:38:26.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.271874+0000", 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.271876+0000", 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9709999999999998e-06 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.271884+0000", 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6520000000000006e-06 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.272061+0000", 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017735400000000001 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.272077+0000", 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5999e-05 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.318226+0000", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046148953999999999 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.318322+0000", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5829999999999996e-05 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:52.318423+0000", 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010100200000000001 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.148 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25046.0:3 42.14 42:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:53.835421+0000", 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 572.299971196, 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0188583429999998, 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25046", 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/232344682", 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835421+0000", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835421+0000", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835423+0000", 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1720000000000001e-06 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835429+0000", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7710000000000003e-06 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835440+0000", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.043e-05 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835450+0000", 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0638999999999999e-05 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835563+0000", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011254199999999999 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:53.835573+0000", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8570000000000007e-06 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.854163+0000", 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0185902310000001 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:58.854280+0000", 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011670100000000001 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24743.0:2069 13.f 13:f50fc370:test-rados-api-smithi143-71756-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.307752+0000", 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 571.82764059800002, 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10221892, 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.153 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24743", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/4179355585", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2069 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.307752+0000", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.307752+0000", 2024-03-11T21:38:26.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.307754+0000", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7829999999999999e-06 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342343+0000", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.034589667999999997 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342382+0000", 2024-03-11T21:38:26.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8365000000000003e-05 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342400+0000", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8354000000000001e-05 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342479+0000", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9450999999999995e-05 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342596+0000", 2024-03-11T21:38:26.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116537 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.342702+0000", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010603300000000001 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.404419+0000", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.061717159000000001 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.406028+0000", 2024-03-11T21:38:26.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016085920000000001 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.409870+0000", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038423910000000001 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.409924+0000", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3746999999999999e-05 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.409971+0000", 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6839999999999999e-05 2024-03-11T21:38:26.158 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24743.0:2070 13.1c 13:3b97e158:test-rados-api-smithi143-71756-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.410346+0000", 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 571.72504605100005, 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.122620834, 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24743", 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/4179355585", 2024-03-11T21:38:26.159 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2070 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.410346+0000", 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.410346+0000", 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.410348+0000", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9549999999999999e-06 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.439516+0000", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.029167391000000001 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.439555+0000", 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9187999999999998e-05 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.439580+0000", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5310999999999999e-05 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.439690+0000", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109873 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.439910+0000", 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022003300000000001 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.440157+0000", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000246956 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.473201+0000", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033043701000000002 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.532733+0000", 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.059532333999999999 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.532775+0000", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1443000000000003e-05 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.532900+0000", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012491999999999999 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.532967+0000", 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7729000000000002e-05 2024-03-11T21:38:26.164 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.24743.0:2074 13.1e e144/139 13:7a94f9ba:test-rados-api-smithi143-71756-17::big3:head v 144'1, mlcod=0'0)", 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.584792+0000", 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 571.55060056000002, 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.065732517000000004, 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584792+0000", 2024-03-11T21:38:26.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584792+0000", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.584793+0000", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5099999999999999e-07 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.599015+0000", 2024-03-11T21:38:26.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014222629000000001 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.599055+0000", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9406000000000002e-05 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.599074+0000", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8876999999999999e-05 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.599182+0000", 2024-03-11T21:38:26.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108152 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.599209+0000", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7733e-05 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650490+0000", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051280410999999998 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650524+0000", 2024-03-11T21:38:26.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4557999999999997e-05 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.25169.0:32 21.c 21:37619e84:::7.obj3:head [write 0~135] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:28:54.605105+0000", 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 571.53028720199995, 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.045273426999999998, 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.25169", 2024-03-11T21:38:26.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/574308987", 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 32 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605105+0000", 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605105+0000", 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605108+0000", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4260000000000002e-06 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605120+0000", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2525e-05 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605146+0000", 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6012000000000001e-05 2024-03-11T21:38:26.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605160+0000", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4012999999999999e-05 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605255+0000", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4798999999999998e-05 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605434+0000", 2024-03-11T21:38:26.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000179001 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.605629+0000", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019460300000000001 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.641143+0000", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.035514413000000002 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650053+0000", 2024-03-11T21:38:26.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089095779999999996 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650292+0000", 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023963599999999999 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650332+0000", 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0101000000000001e-05 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:28:54.650379+0000", 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6319999999999997e-05 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14976.0:5 35.12 35:4aed5205:test-rados-api-smithi143-71818-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:01.393221+0000", 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 564.74217185700002, 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.074744766000000004, 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14976", 2024-03-11T21:38:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1384281194", 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 5 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393221+0000", 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393221+0000", 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393223+0000", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2000000000000001e-06 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393236+0000", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.272e-05 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393260+0000", 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4311000000000002e-05 2024-03-11T21:38:26.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393276+0000", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6572999999999999e-05 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393399+0000", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012232700000000001 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.393414+0000", 2024-03-11T21:38:26.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5268999999999999e-05 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.423063+0000", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.029648942000000001 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.423319+0000", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025652900000000002 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.423632+0000", 2024-03-11T21:38:26.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000312985 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.426835+0000", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003202791 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.460329+0000", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033493842000000003 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.467869+0000", 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075403459999999999 2024-03-11T21:38:26.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.467917+0000", 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7941000000000001e-05 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:01.467965+0000", 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.799e-05 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.37106.0:2 128.1b 128:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:36.454989+0000", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 529.68040378000001, 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.398788052, 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.37106", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1762262377", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.454989+0000", 2024-03-11T21:38:26.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.454989+0000", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.454991+0000", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2740000000000002e-06 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.454999+0000", 2024-03-11T21:38:26.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2439999999999993e-06 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.455012+0000", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3162999999999999e-05 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.455024+0000", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1708e-05 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.455092+0000", 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8108999999999995e-05 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:36.455103+0000", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0360000000000001e-05 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.799337+0000", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.344234395 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.799440+0000", 2024-03-11T21:38:26.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103193 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.799549+0000", 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108502 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.799921+0000", 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037265600000000002 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.814070+0000", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01414842 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.853701+0000", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.039630842999999999 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.853743+0000", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2716000000000002e-05 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:37.853777+0000", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3469000000000001e-05 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.22002.0:484 123.4s0 123:34b887cc:::483:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:40.014901+0000", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 526.12049099800004, 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29097255300000002, 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.22002", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2361743075", 2024-03-11T21:38:26.188 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 484 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014901+0000", 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014901+0000", 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014902+0000", 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1900000000000002e-07 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014906+0000", 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5949999999999999e-06 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014914+0000", 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.875e-06 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014918+0000", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.775e-06 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.014949+0000", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0938000000000001e-05 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.015016+0000", 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6999e-05 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.015200+0000", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000183995 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.015419+0000", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000218519 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.305807+0000", 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29038780400000003 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.305874+0000", 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7334000000000003e-05 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40852.0:2 134.1b 134:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:40.218593+0000", 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 525.91679992000002, 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1218916939999999, 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.193 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40852", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1763403163", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218593+0000", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218593+0000", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218595+0000", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9570000000000001e-06 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218601+0000", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.99e-06 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218613+0000", 2024-03-11T21:38:26.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1765e-05 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.218623+0000", 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0074999999999999e-05 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.224290+0000", 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056663920000000001 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:40.224299+0000", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1819999999999995e-06 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.288194+0000", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.063895335 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.288318+0000", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000123941 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.288457+0000", 2024-03-11T21:38:26.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000139264 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.289011+0000", 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055376999999999998 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.289142+0000", 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130951 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.340417+0000", 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051274880000000002 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.340451+0000", 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4136e-05 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:41.340484+0000", 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3056000000000003e-05 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.28353.0:1 137.0s0 137:070175dc:test-rados-api-smithi143-71818-11::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:45.089408+0000", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 521.04598490900003, 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.56800159400000005, 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.28353", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/202579008", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089408+0000", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089408+0000", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089411+0000", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3670000000000001e-06 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089427+0000", 2024-03-11T21:38:26.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5843000000000001e-05 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089463+0000", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6696000000000002e-05 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089497+0000", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3287000000000002e-05 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089637+0000", 2024-03-11T21:38:26.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014050099999999999 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.089644+0000", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6869999999999997e-06 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.640681+0000", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.55103672699999995 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.640897+0000", 2024-03-11T21:38:26.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000216603 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.642362+0000", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014647530000000001 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.645618+0000", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032556130000000001 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.657307+0000", 2024-03-11T21:38:26.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011689425999999999 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.657409+0000", 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102091 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40658.0:2 140.1b 140:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:45.171209+0000", 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 520.96418353199999, 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.546016463, 2024-03-11T21:38:26.205 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40658", 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/709334961", 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171209+0000", 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171209+0000", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171212+0000", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.689e-06 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171221+0000", 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7960000000000007e-06 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171240+0000", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8547999999999999e-05 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171255+0000", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4598e-05 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171351+0000", 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6032000000000005e-05 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:45.171364+0000", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3467e-05 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.689533+0000", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5181685310000002 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.689672+0000", 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013963499999999999 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.689812+0000", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000139305 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.696849+0000", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070369910000000003 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.698533+0000", 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00168435 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.717122+0000", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018588963999999999 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.717181+0000", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8798999999999997e-05 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.717225+0000", 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4758000000000001e-05 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.211 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.40664.0:2 141.6 141:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:29:46.586931+0000", 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 519.548461729, 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.119684043, 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.40664", 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2472074103", 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.212 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.586931+0000", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.586931+0000", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.586934+0000", 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.36e-06 2024-03-11T21:38:26.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.586948+0000", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3616e-05 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.586979+0000", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1114000000000001e-05 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.587004+0000", 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5486e-05 2024-03-11T21:38:26.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.594678+0000", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076741800000000001 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.594699+0000", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.014e-05 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.595915+0000", 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001216106 2024-03-11T21:38:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.595936+0000", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1382999999999998e-05 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.684014+0000", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088078109000000002 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.684137+0000", 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012286299999999999 2024-03-11T21:38:26.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.684293+0000", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000155494 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.688299+0000", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0040061799999999998 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.696781+0000", 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0084818559999999994 2024-03-11T21:38:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.706522+0000", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0097412660000000002 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.706562+0000", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9734000000000003e-05 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:29:46.706615+0000", 2024-03-11T21:38:26.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3155999999999997e-05 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.32849.0:1073 173.d 173:b0874ef8:test-rados-api-smithi143-71773-14::foo68:3 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e212)", 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:30:14.224774+0000", 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 491.910618263, 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.091129525000000003, 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.32849", 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1073 2024-03-11T21:38:26.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224774+0000", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224774+0000", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224777+0000", 2024-03-11T21:38:26.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.526e-06 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224791+0000", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.381e-05 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224822+0000", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1050000000000003e-05 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.224875+0000", 2024-03-11T21:38:26.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3192999999999997e-05 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.225074+0000", 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019910300000000001 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.225880+0000", 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080623000000000001 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.303446+0000", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077565481000000006 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.303587+0000", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014174200000000001 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.315527+0000", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011940002999999999 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.315628+0000", 2024-03-11T21:38:26.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0001011 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:30:14.315904+0000", 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027528699999999998 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41219.0:1 227.6s0 227:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:31:23.285505+0000", 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 422.849887599, 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0481005750000001, 2024-03-11T21:38:26.224 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41219", 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1445291562", 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285505+0000", 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285505+0000", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285508+0000", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.264e-06 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285523+0000", 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4708999999999999e-05 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285552+0000", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9385000000000001e-05 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.285578+0000", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6245999999999999e-05 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.289847+0000", 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042681180000000004 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:23.289853+0000", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5200000000000003e-06 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.312446+0000", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.022592948 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.312585+0000", 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013925299999999999 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.313078+0000", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049310100000000002 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.314313+0000", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012344879999999999 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.333513+0000", 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019200136 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:31:24.333605+0000", 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2406999999999998e-05 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.30288.0:1 267.1s0 267:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:33:06.330350+0000", 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 319.80504227500001, 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0172374200000001, 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:38:26.230 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.30288", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3042075408", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330350+0000", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330350+0000", 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330353+0000", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.745e-06 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330364+0000", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1029000000000001e-05 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330389+0000", 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5380000000000001e-05 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.330406+0000", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6923999999999999e-05 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.339101+0000", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086946550000000008 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:06.339106+0000", 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9960000000000001e-06 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.342550+0000", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0034441599999999 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.342691+0000", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000141228 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.343042+0000", 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000351078 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.343569+0000", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0005264 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.347414+0000", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003845003 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:33:07.347588+0000", 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017382199999999999 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.235 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.236 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.236 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:38:26.236 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:38:26.236 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:38:26.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:26.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:26.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:26.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:26.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:26.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:26.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:26.637 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:26 smithi203 bash[23654]: cluster 2024-03-11T21:38:25.517269+0000 mgr.y (mgr.14550) 606 : cluster 0 pgmap v957: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 641 B/s wr, 2 op/s 2024-03-11T21:38:26.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:26 smithi203 bash[23654]: cluster 2024-03-11T21:38:25.580301+0000 mon.a (mon.0) 3118 : cluster 0 osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:38:26.652 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:26.668 INFO:tasks.thrashosds.thrasher:j is {'epoch': 617, 'fsid': 'eebe2e36-dfeb-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:12:47.384719+0000', 'modified': '2024-03-11T21:38:23.559944+0000', 'last_up_change': '2024-03-11T21:31:36.320182+0000', 'last_in_change': '2024-03-11T21:21:55.456354+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 111, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 304, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:18:08.036690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '596', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '593', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:23:01.432928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:23:03.441476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:23:04.441432+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:23:05.388543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:23:07.642696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-11T21:28:36.978109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 39, 'pool_name': 'SnapListsmithi143-72399-1', 'create_time': '2024-03-11T21:28:49.753089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 118, 'pool_name': 'test-rados-api-smithi143-71773-6', 'create_time': '2024-03-11T21:29:30.235634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '613', 'last_force_op_resend': '454', 'last_force_op_resend_prenautilus': '454', 'last_force_op_resend_preluminous': '454', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 613, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 304, 'pool_name': 'test-rados-api-smithi143-71773-96', 'create_time': '2024-03-11T21:38:11.919746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '613', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6802', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6803', 'nonce': 2903432023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6804', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6805', 'nonce': 2903432023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6808', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6809', 'nonce': 2903432023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6806', 'nonce': 2903432023}, {'type': 'v1', 'addr': '172.21.15.143:6807', 'nonce': 2903432023}]}, 'public_addr': '172.21.15.143:6803/2903432023', 'cluster_addr': '172.21.15.143:6805/2903432023', 'heartbeat_back_addr': '172.21.15.143:6809/2903432023', 'heartbeat_front_addr': '172.21.15.143:6807/2903432023', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2948dc44-68d3-4b3d-92bf-2b0c46374f02', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 246, 'up_from': 295, 'up_thru': 611, 'down_at': 247, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6810', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6811', 'nonce': 4235040845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6812', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6813', 'nonce': 4235040845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6816', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6817', 'nonce': 4235040845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6814', 'nonce': 4235040845}, {'type': 'v1', 'addr': '172.21.15.143:6815', 'nonce': 4235040845}]}, 'public_addr': '172.21.15.143:6811/4235040845', 'cluster_addr': '172.21.15.143:6813/4235040845', 'heartbeat_back_addr': '172.21.15.143:6817/4235040845', 'heartbeat_front_addr': '172.21.15.143:6815/4235040845', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7217589a-5835-407b-91a7-f5c055a48f3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6818', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6819', 'nonce': 2592131625}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6820', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6821', 'nonce': 2592131625}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6824', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6825', 'nonce': 2592131625}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6822', 'nonce': 2592131625}, {'type': 'v1', 'addr': '172.21.15.143:6823', 'nonce': 2592131625}]}, 'public_addr': '172.21.15.143:6819/2592131625', 'cluster_addr': '172.21.15.143:6821/2592131625', 'heartbeat_back_addr': '172.21.15.143:6825/2592131625', 'heartbeat_front_addr': '172.21.15.143:6823/2592131625', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a3e7177f-a5a0-44ad-a65d-9aeac3a680cc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6826', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6827', 'nonce': 2146571980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6828', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6829', 'nonce': 2146571980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2146571980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6830', 'nonce': 2146571980}, {'type': 'v1', 'addr': '172.21.15.143:6831', 'nonce': 2146571980}]}, 'public_addr': '172.21.15.143:6827/2146571980', 'cluster_addr': '172.21.15.143:6829/2146571980', 'heartbeat_back_addr': '172.21.15.143:6833/2146571980', 'heartbeat_front_addr': '172.21.15.143:6831/2146571980', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd7b9e088-d377-4335-8863-11d581828f7e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6800', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 2090835691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6802', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 2090835691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6806', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 2090835691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6804', 'nonce': 2090835691}, {'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 2090835691}]}, 'public_addr': '172.21.15.203:6801/2090835691', 'cluster_addr': '172.21.15.203:6803/2090835691', 'heartbeat_back_addr': '172.21.15.203:6807/2090835691', 'heartbeat_front_addr': '172.21.15.203:6805/2090835691', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b3b2a75c-68a5-4e00-8068-622e1ebbaa01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6808', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 2432038345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6810', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 2432038345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6814', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2432038345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6812', 'nonce': 2432038345}, {'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2432038345}]}, 'public_addr': '172.21.15.203:6809/2432038345', 'cluster_addr': '172.21.15.203:6811/2432038345', 'heartbeat_back_addr': '172.21.15.203:6815/2432038345', 'heartbeat_front_addr': '172.21.15.203:6813/2432038345', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ceffb234-db0c-4ffa-a142-9b201df9b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6816', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6817', 'nonce': 513748834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6818', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6819', 'nonce': 513748834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6822', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6823', 'nonce': 513748834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6820', 'nonce': 513748834}, {'type': 'v1', 'addr': '172.21.15.203:6821', 'nonce': 513748834}]}, 'public_addr': '172.21.15.203:6817/513748834', 'cluster_addr': '172.21.15.203:6819/513748834', 'heartbeat_back_addr': '172.21.15.203:6823/513748834', 'heartbeat_front_addr': '172.21.15.203:6821/513748834', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '88b53171-69d2-4922-a787-e63d44d01c60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6824', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6825', 'nonce': 1251497952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6826', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6827', 'nonce': 1251497952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6830', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6831', 'nonce': 1251497952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6828', 'nonce': 1251497952}, {'type': 'v1', 'addr': '172.21.15.203:6829', 'nonce': 1251497952}]}, 'public_addr': '172.21.15.203:6825/1251497952', 'cluster_addr': '172.21.15.203:6827/1251497952', 'heartbeat_back_addr': '172.21.15.203:6831/1251497952', 'heartbeat_front_addr': '172.21.15.203:6829/1251497952', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:38:08.418460+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:37:53.927771+0000', 'dead_epoch': 247}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:38:20.483099+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:37:56.580288+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:37:24.962604+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:37:54.755322+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:38:16.710036+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:38:22.986738+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.10', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '9.1e', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '39.2', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '39.9', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '39.17', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.143:6801/3638739149': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4274199357': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/3217951344': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/3912416566': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/906834629': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2791822486': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2190104670': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:0/4171894415': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/3787195463': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2845155916': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6801/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:0/2913593434': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/143435387': '2024-03-12T21:13:53.401579+0000', '172.21.15.143:0/2930358289': '2024-03-12T21:23:57.022184+0000', '172.21.15.143:6800/1839244747': '2024-03-12T21:13:19.169870+0000', '172.21.15.143:6800/3638739149': '2024-03-12T21:23:57.022184+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 28, 'snaps': [{'begin': 5, 'length': 2}]}, {'pool': 35, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 89, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 156, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 178, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 185, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 198, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 210, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 230, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 244, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 256, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:38:26.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:38:26.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:26.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:26.885 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:26.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:26 smithi143 bash[21512]: cluster 2024-03-11T21:38:25.517269+0000 mgr.y (mgr.14550) 606 : cluster 0 pgmap v957: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 641 B/s wr, 2 op/s 2024-03-11T21:38:26.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:26 smithi143 bash[21512]: cluster 2024-03-11T21:38:25.580301+0000 mon.a (mon.0) 3118 : cluster 0 osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:38:26.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:26 smithi143 bash[29284]: cluster 2024-03-11T21:38:25.517269+0000 mgr.y (mgr.14550) 606 : cluster 0 pgmap v957: 270 pgs: 270 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 641 B/s wr, 2 op/s 2024-03-11T21:38:26.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:26 smithi143 bash[29284]: cluster 2024-03-11T21:38:25.580301+0000 mon.a (mon.0) 3118 : cluster 0 osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:38:26.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:26.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:38:27.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:27.040 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:27.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.161 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.421 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:27.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:27.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.673 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:27 smithi203 bash[23654]: cluster 2024-03-11T21:38:26.615593+0000 mon.a (mon.0) 3119 : cluster 0 osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:38:27.673 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:27 smithi203 bash[23654]: audit 2024-03-11T21:38:26.622117+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.673 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:27 smithi203 bash[23654]: audit 2024-03-11T21:38:26.623435+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:27.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[29284]: cluster 2024-03-11T21:38:26.615593+0000 mon.a (mon.0) 3119 : cluster 0 osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:38:27.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[29284]: audit 2024-03-11T21:38:26.622117+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.804 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[29284]: audit 2024-03-11T21:38:26.623435+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[21512]: cluster 2024-03-11T21:38:26.615593+0000 mon.a (mon.0) 3119 : cluster 0 osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:38:27.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[21512]: audit 2024-03-11T21:38:26.622117+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.805 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:27 smithi143 bash[21512]: audit 2024-03-11T21:38:26.623435+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:27.819 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:27.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:28.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.082 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:28.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:28.211 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/3425445482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 msgr2=0x7f3eb806aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/3425445482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb806aea0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009990 tx=0x7f3eac009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/3425445482 shutdown_connections 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/3425445482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb8110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/3425445482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 0x7f3eb806a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/3425445482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb806aea0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/3425445482 >> 172.21.15.143:0/3425445482 conn(0x7f3eb8077a40 msgr2=0x7f3eb8077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:28.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/3425445482 shutdown_connections 2024-03-11T21:38:28.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/3425445482 wait complete. 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 librados: starting msgr at 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 librados: starting objecter 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 Processor -- start 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- start start 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 librados: setting wanted keys 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 librados: calling monclient init 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 0x7f3eb819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8113830 con 0x7f3eb810e670 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81136b0 con 0x7f3eb8102d00 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81139b0 con 0x7f3eb806aa20 2024-03-11T21:38:28.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47302/0 (socket says 172.21.15.143:47302) 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 -- 172.21.15.143:0/1715611027 learned_addr learned my addr 172.21.15.143:0/1715611027 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 -- 172.21.15.143:0/1715611027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 msgr2=0x7f3eb819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb67fc700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 0x7f3eb819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 0x7f3eb819a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 -- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 msgr2=0x7f3eb819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb819e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 -- 172.21.15.143:0/1715611027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb819eeb0 con 0x7f3eb806aa20 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb77fe700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb819e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3eb6ffd700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800d980 tx=0x7f3ea800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3e9ffff700 10 client.?.objecter ms_handle_connect 0x7f3eb806aa20 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3e9ffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8017070 con 0x7f3eb806aa20 2024-03-11T21:38:28.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3ea8005020 con 0x7f3eb806aa20 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb819f1a0 con 0x7f3eb806aa20 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8009ad0 con 0x7f3eb806aa20 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 10 client.41840.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb81a6a20 con 0x7f3eb806aa20 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.239+0000 7f3ebd5bf700 1 librados: init done 2024-03-11T21:38:28.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb804faa0 con 0x7f3eb806aa20 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3ea8009c70 con 0x7f3eb806aa20 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 0x7f3ea0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(620..620 src has 87..620) v4 ==== 9661+0+0 (secure 0 0 0) 0x7f3ea808e9e0 con 0x7f3eb806aa20 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 10 client.41840.objecter ms_dispatch 0x7f3eb8001550 osd_map(620..620 src has 87..620) v4 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 3 client.41840.objecter handle_osd_map got epochs [620,620] > 0 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 3 client.41840.objecter handle_osd_map decoding full epoch 620 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3eb67fc700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 0x7f3ea0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 20 client.41840.objecter dump_active .. 0 homeless 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3e9ffff700 10 client.41840.objecter ms_handle_connect 0x7f3ea006f820 2024-03-11T21:38:28.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.243+0000 7f3eb67fc700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 0x7f3ea0071ce0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b2e0 tx=0x7f3eac005e80 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:28.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.247+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3ea8020020 con 0x7f3eb806aa20 2024-03-11T21:38:28.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:28.342 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:28.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.471+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f3eb81a6bf0 con 0x7f3eb806aa20 2024-03-11T21:38:28.475 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:28.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.623+0000 7f3e9ffff700 1 -- 172.21.15.143:0/1715611027 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v621) v1 ==== 91+0+0 (secure 0 0 0) 0x7f3ea805b610 con 0x7f3eb806aa20 2024-03-11T21:38:28.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 10 librados: watch_flush enter 2024-03-11T21:38:28.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 10 librados: watch_flush exit 2024-03-11T21:38:28.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 20 client.41840.objecter shutdown clearing up homeless session... 2024-03-11T21:38:28.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 10 client.41840.objecter successfully canceled tick 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 msgr2=0x7f3ea0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 0x7f3ea0071ce0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b2e0 tx=0x7f3eac005e80 comp rx=0 tx=0).stop 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 msgr2=0x7f3eb8199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800d980 tx=0x7f3ea800dc90 comp rx=0 tx=0).stop 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 shutdown_connections 2024-03-11T21:38:28.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3ea006f820 0x7f3ea0071ce0 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb810e670 0x7f3eb819e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb806aa20 0x7f3eb8199ea0 secure :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800d980 tx=0x7f3ea800dc90 comp rx=0 tx=0).stop 2024-03-11T21:38:28.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 --2- 172.21.15.143:0/1715611027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3eb8102d00 0x7f3eb819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:28.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.627+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 >> 172.21.15.143:0/1715611027 conn(0x7f3eb8077a40 msgr2=0x7f3eb80ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.631+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 shutdown_connections 2024-03-11T21:38:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.631+0000 7f3ebd5bf700 1 -- 172.21.15.143:0/1715611027 wait complete. 2024-03-11T21:38:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:28.631+0000 7f3ebd5bf700 1 librados: shutdown 2024-03-11T21:38:28.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:28.613 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:38:28.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:28.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: cluster 2024-03-11T21:38:27.518085+0000 mgr.y (mgr.14550) 607 : cluster 0 pgmap v960: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:28.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: cluster 2024-03-11T21:38:27.605524+0000 mon.a (mon.0) 3121 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:28.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: audit 2024-03-11T21:38:27.613654+0000 mon.a (mon.0) 3122 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:28.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: cluster 2024-03-11T21:38:27.618602+0000 mon.a (mon.0) 3123 : cluster 0 osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:38:28.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: audit 2024-03-11T21:38:27.643610+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: audit 2024-03-11T21:38:27.645116+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: audit 2024-03-11T21:38:28.476897+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.143:0/1715611027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[23654]: audit 2024-03-11T21:38:28.477423+0000 mon.a (mon.0) 3125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:28.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:28.871 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:28.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:28.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: cluster 2024-03-11T21:38:27.518085+0000 mgr.y (mgr.14550) 607 : cluster 0 pgmap v960: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:28.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: cluster 2024-03-11T21:38:27.605524+0000 mon.a (mon.0) 3121 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:28.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: audit 2024-03-11T21:38:27.613654+0000 mon.a (mon.0) 3122 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:28.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: cluster 2024-03-11T21:38:27.618602+0000 mon.a (mon.0) 3123 : cluster 0 osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: audit 2024-03-11T21:38:27.643610+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: audit 2024-03-11T21:38:27.645116+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: audit 2024-03-11T21:38:28.476897+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.143:0/1715611027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[29284]: audit 2024-03-11T21:38:28.477423+0000 mon.a (mon.0) 3125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: cluster 2024-03-11T21:38:27.518085+0000 mgr.y (mgr.14550) 607 : cluster 0 pgmap v960: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: cluster 2024-03-11T21:38:27.605524+0000 mon.a (mon.0) 3121 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: audit 2024-03-11T21:38:27.613654+0000 mon.a (mon.0) 3122 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: cluster 2024-03-11T21:38:27.618602+0000 mon.a (mon.0) 3123 : cluster 0 osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: audit 2024-03-11T21:38:27.643610+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.927 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: audit 2024-03-11T21:38:27.645116+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:28.928 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: audit 2024-03-11T21:38:28.476897+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.143:0/1715611027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.928 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:28 smithi143 bash[21512]: audit 2024-03-11T21:38:28.477423+0000 mon.a (mon.0) 3125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:28.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:29.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:29.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:29.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:29.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:29.260 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:29.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:29.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:29.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:29.520 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:29.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:29.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: cluster 2024-03-11T21:38:28.614217+0000 mon.a (mon.0) 3126 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:29.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: audit 2024-03-11T21:38:28.616416+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:29.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: audit 2024-03-11T21:38:28.616609+0000 mon.a (mon.0) 3128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:29.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: cluster 2024-03-11T21:38:28.623580+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:38:29.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: audit 2024-03-11T21:38:28.629008+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:29.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:29.760 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: cluster 2024-03-11T21:38:28.614217+0000 mon.a (mon.0) 3126 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:29.760 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: audit 2024-03-11T21:38:28.616416+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:29.760 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: audit 2024-03-11T21:38:28.616609+0000 mon.a (mon.0) 3128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:29.760 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: cluster 2024-03-11T21:38:28.623580+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:38:29.761 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: audit 2024-03-11T21:38:28.629008+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.761 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: audit 2024-03-11T21:38:28.640899+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.761 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:29 smithi203 bash[23654]: audit 2024-03-11T21:38:28.838542+0000 mgr.y (mgr.14550) 608 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:29.773 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:29.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:29.901 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: audit 2024-03-11T21:38:28.640899+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[21512]: audit 2024-03-11T21:38:28.838542+0000 mgr.y (mgr.14550) 608 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:29.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: cluster 2024-03-11T21:38:28.614217+0000 mon.a (mon.0) 3126 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:29.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: audit 2024-03-11T21:38:28.616416+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:29.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: audit 2024-03-11T21:38:28.616609+0000 mon.a (mon.0) 3128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: cluster 2024-03-11T21:38:28.623580+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:38:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: audit 2024-03-11T21:38:28.629008+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: audit 2024-03-11T21:38:28.640899+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]: dispatch 2024-03-11T21:38:29.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:29 smithi143 bash[29284]: audit 2024-03-11T21:38:28.838542+0000 mgr.y (mgr.14550) 608 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:30.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:30.033 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:30.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:30.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:30.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.293 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:30.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:30.423 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:30.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:30.554 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.518920+0000 mgr.y (mgr.14550) 609 : cluster 0 pgmap v963: 270 pgs: 5 unknown, 265 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.630788+0000 mon.a (mon.0) 3131 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: audit 2024-03-11T21:38:29.633108+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]': finished 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.639646+0000 mon.a (mon.0) 3133 : cluster 0 osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: audit 2024-03-11T21:38:29.644197+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.627 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: audit 2024-03-11T21:38:29.653033+0000 mon.a (mon.0) 3134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.749818+0000 osd.3 (osd.3) 363 : cluster 0 9.8 scrub starts 2024-03-11T21:38:30.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.752431+0000 osd.3 (osd.3) 364 : cluster 0 9.8 scrub ok 2024-03-11T21:38:30.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.772602+0000 osd.5 (osd.5) 367 : cluster 0 118.e scrub starts 2024-03-11T21:38:30.628 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[21512]: cluster 2024-03-11T21:38:29.774927+0000 osd.5 (osd.5) 368 : cluster 0 118.e scrub ok 2024-03-11T21:38:30.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:30.667 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.518920+0000 mgr.y (mgr.14550) 609 : cluster 0 pgmap v963: 270 pgs: 5 unknown, 265 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.630788+0000 mon.a (mon.0) 3131 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: audit 2024-03-11T21:38:29.633108+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]': finished 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.639646+0000 mon.a (mon.0) 3133 : cluster 0 osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: audit 2024-03-11T21:38:29.644197+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: audit 2024-03-11T21:38:29.653033+0000 mon.a (mon.0) 3134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.749818+0000 osd.3 (osd.3) 363 : cluster 0 9.8 scrub starts 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.752431+0000 osd.3 (osd.3) 364 : cluster 0 9.8 scrub ok 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.772602+0000 osd.5 (osd.5) 367 : cluster 0 118.e scrub starts 2024-03-11T21:38:30.668 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:30 smithi203 bash[23654]: cluster 2024-03-11T21:38:29.774927+0000 osd.5 (osd.5) 368 : cluster 0 118.e scrub ok 2024-03-11T21:38:30.681 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:30.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:30.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.518920+0000 mgr.y (mgr.14550) 609 : cluster 0 pgmap v963: 270 pgs: 5 unknown, 265 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.630788+0000 mon.a (mon.0) 3131 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: audit 2024-03-11T21:38:29.633108+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-98", "mode": "writeback"}]': finished 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.639646+0000 mon.a (mon.0) 3133 : cluster 0 osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: audit 2024-03-11T21:38:29.644197+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: audit 2024-03-11T21:38:29.653033+0000 mon.a (mon.0) 3134 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.749818+0000 osd.3 (osd.3) 363 : cluster 0 9.8 scrub starts 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.752431+0000 osd.3 (osd.3) 364 : cluster 0 9.8 scrub ok 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.772602+0000 osd.5 (osd.5) 367 : cluster 0 118.e scrub starts 2024-03-11T21:38:30.927 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:30 smithi143 bash[29284]: cluster 2024-03-11T21:38:29.774927+0000 osd.5 (osd.5) 368 : cluster 0 118.e scrub ok 2024-03-11T21:38:30.944 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:31.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:31.132 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:31.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:31.205 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:31.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:31.337 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:31.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:31.467 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:31.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:31.597 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:31.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:30.640750+0000 osd.6 (osd.6) 273 : cluster 0 3.10 scrub starts 2024-03-11T21:38:31.638 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:30.645588+0000 osd.6 (osd.6) 274 : cluster 0 3.10 scrub ok 2024-03-11T21:38:31.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:38:31.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:31.713 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:31.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:31.828 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:30.640750+0000 osd.6 (osd.6) 273 : cluster 0 3.10 scrub starts 2024-03-11T21:38:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:30.645588+0000 osd.6 (osd.6) 274 : cluster 0 3.10 scrub ok 2024-03-11T21:38:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:30.656570+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:30.661695+0000 mon.a (mon.0) 3136 : cluster 0 osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:38:31.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:30.669189+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:30.670773+0000 mon.a (mon.0) 3137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:30.793934+0000 osd.3 (osd.3) 365 : cluster 0 3.7 scrub starts 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:30.797849+0000 osd.3 (osd.3) 366 : cluster 0 3.7 scrub ok 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.440472+0000 mon.a (mon.0) 3138 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:31.442561+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.451726+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:31.453972+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[21512]: audit 2024-03-11T21:38:31.454634+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:31.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:30.656570+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:30.661695+0000 mon.a (mon.0) 3136 : cluster 0 osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:30.669189+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:30.670773+0000 mon.a (mon.0) 3137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:30.793934+0000 osd.3 (osd.3) 365 : cluster 0 3.7 scrub starts 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:30.797849+0000 osd.3 (osd.3) 366 : cluster 0 3.7 scrub ok 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.440472+0000 mon.a (mon.0) 3138 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:31.442561+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.451726+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:31.453972+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:31.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:31 smithi143 bash[29284]: audit 2024-03-11T21:38:31.454634+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:31.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:31.975 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:30.640750+0000 osd.6 (osd.6) 273 : cluster 0 3.10 scrub starts 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:30.645588+0000 osd.6 (osd.6) 274 : cluster 0 3.10 scrub ok 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:30.656570+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:30.661695+0000 mon.a (mon.0) 3136 : cluster 0 osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:30.669189+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:30.670773+0000 mon.a (mon.0) 3137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:30.793934+0000 osd.3 (osd.3) 365 : cluster 0 3.7 scrub starts 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:30.797849+0000 osd.3 (osd.3) 366 : cluster 0 3.7 scrub ok 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.440472+0000 mon.a (mon.0) 3138 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:32.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:31.442561+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:38:32.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.451726+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:38:32.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:31.453972+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:32.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:31 smithi203 bash[23654]: audit 2024-03-11T21:38:31.454634+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:38:32.078 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:32.105 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:32.156 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:32.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:32.218 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:32.231 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:32.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:32.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:32.491 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:32.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:32.620 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:32.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:32.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.519536+0000 mgr.y (mgr.14550) 610 : cluster 0 pgmap v967: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 524 B/s wr, 2 op/s 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.647364+0000 osd.6 (osd.6) 275 : cluster 0 5.11 scrub starts 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.651328+0000 osd.6 (osd.6) 276 : cluster 0 5.11 scrub ok 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.701203+0000 osd.5 (osd.5) 369 : cluster 0 6.a scrub starts 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.702636+0000 osd.5 (osd.5) 370 : cluster 0 6.a scrub ok 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.829694+0000 osd.2 (osd.2) 311 : cluster 0 118.5 scrub starts 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.832238+0000 osd.2 (osd.2) 312 : cluster 0 118.5 scrub ok 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.927182+0000 osd.0 (osd.0) 350 : cluster 0 118.18 scrub starts 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:31.938702+0000 osd.0 (osd.0) 351 : cluster 0 118.18 scrub ok 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: audit 2024-03-11T21:38:32.444834+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: audit 2024-03-11T21:38:32.451839+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: cluster 2024-03-11T21:38:32.453744+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:38:32.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:32 smithi203 bash[23654]: audit 2024-03-11T21:38:32.457217+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:32.881 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:32.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.519536+0000 mgr.y (mgr.14550) 610 : cluster 0 pgmap v967: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 524 B/s wr, 2 op/s 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.647364+0000 osd.6 (osd.6) 275 : cluster 0 5.11 scrub starts 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.651328+0000 osd.6 (osd.6) 276 : cluster 0 5.11 scrub ok 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.701203+0000 osd.5 (osd.5) 369 : cluster 0 6.a scrub starts 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.702636+0000 osd.5 (osd.5) 370 : cluster 0 6.a scrub ok 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.829694+0000 osd.2 (osd.2) 311 : cluster 0 118.5 scrub starts 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.832238+0000 osd.2 (osd.2) 312 : cluster 0 118.5 scrub ok 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.927182+0000 osd.0 (osd.0) 350 : cluster 0 118.18 scrub starts 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:31.938702+0000 osd.0 (osd.0) 351 : cluster 0 118.18 scrub ok 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: audit 2024-03-11T21:38:32.444834+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: audit 2024-03-11T21:38:32.451839+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: cluster 2024-03-11T21:38:32.453744+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:38:32.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[29284]: audit 2024-03-11T21:38:32.457217+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.519536+0000 mgr.y (mgr.14550) 610 : cluster 0 pgmap v967: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 524 B/s wr, 2 op/s 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.647364+0000 osd.6 (osd.6) 275 : cluster 0 5.11 scrub starts 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.651328+0000 osd.6 (osd.6) 276 : cluster 0 5.11 scrub ok 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.701203+0000 osd.5 (osd.5) 369 : cluster 0 6.a scrub starts 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.702636+0000 osd.5 (osd.5) 370 : cluster 0 6.a scrub ok 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.829694+0000 osd.2 (osd.2) 311 : cluster 0 118.5 scrub starts 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.832238+0000 osd.2 (osd.2) 312 : cluster 0 118.5 scrub ok 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.927182+0000 osd.0 (osd.0) 350 : cluster 0 118.18 scrub starts 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:31.938702+0000 osd.0 (osd.0) 351 : cluster 0 118.18 scrub ok 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: audit 2024-03-11T21:38:32.444834+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: audit 2024-03-11T21:38:32.451839+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: cluster 2024-03-11T21:38:32.453744+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:38:32.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:32 smithi143 bash[21512]: audit 2024-03-11T21:38:32.457217+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:32.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:33.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:33.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:33.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:33.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.389 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:33.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:33.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: cluster 2024-03-11T21:38:32.515620+0000 osd.4 (osd.4) 339 : cluster 0 6.17 scrub starts 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: cluster 2024-03-11T21:38:32.517278+0000 osd.4 (osd.4) 340 : cluster 0 6.17 scrub ok 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: cluster 2024-03-11T21:38:32.834729+0000 osd.2 (osd.2) 313 : cluster 0 118.9 scrub starts 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: cluster 2024-03-11T21:38:32.837175+0000 osd.2 (osd.2) 314 : cluster 0 118.9 scrub ok 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: audit 2024-03-11T21:38:33.447135+0000 mon.a (mon.0) 3145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: audit 2024-03-11T21:38:33.453847+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.455190+0000 mon.a (mon.0) 3146 : cluster 0 osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:38:33.763 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:33 smithi203 bash[23654]: audit 2024-03-11T21:38:33.457029+0000 mon.a (mon.0) 3147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:33.778 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:33.906 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: cluster 2024-03-11T21:38:32.515620+0000 osd.4 (osd.4) 339 : cluster 0 6.17 scrub starts 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: cluster 2024-03-11T21:38:32.517278+0000 osd.4 (osd.4) 340 : cluster 0 6.17 scrub ok 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: cluster 2024-03-11T21:38:32.834729+0000 osd.2 (osd.2) 313 : cluster 0 118.9 scrub starts 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: cluster 2024-03-11T21:38:32.837175+0000 osd.2 (osd.2) 314 : cluster 0 118.9 scrub ok 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: audit 2024-03-11T21:38:33.447135+0000 mon.a (mon.0) 3145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: audit 2024-03-11T21:38:33.453847+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:33.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.455190+0000 mon.a (mon.0) 3146 : cluster 0 osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[29284]: audit 2024-03-11T21:38:33.457029+0000 mon.a (mon.0) 3147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: cluster 2024-03-11T21:38:32.515620+0000 osd.4 (osd.4) 339 : cluster 0 6.17 scrub starts 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: cluster 2024-03-11T21:38:32.517278+0000 osd.4 (osd.4) 340 : cluster 0 6.17 scrub ok 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: cluster 2024-03-11T21:38:32.834729+0000 osd.2 (osd.2) 313 : cluster 0 118.9 scrub starts 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: cluster 2024-03-11T21:38:32.837175+0000 osd.2 (osd.2) 314 : cluster 0 118.9 scrub ok 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: audit 2024-03-11T21:38:33.447135+0000 mon.a (mon.0) 3145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: audit 2024-03-11T21:38:33.453847+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.455190+0000 mon.a (mon.0) 3146 : cluster 0 osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:38:33.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:33 smithi143 bash[21512]: audit 2024-03-11T21:38:33.457029+0000 mon.a (mon.0) 3147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:38:34.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:34.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:34.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:34.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:34.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:34.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:34.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:34.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:34.688 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:34.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.520374+0000 mgr.y (mgr.14550) 611 : cluster 0 pgmap v970: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:34.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.597962+0000 osd.6 (osd.6) 277 : cluster 0 2.11 scrub starts 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.599334+0000 osd.6 (osd.6) 278 : cluster 0 2.11 scrub ok 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.686157+0000 osd.5 (osd.5) 371 : cluster 0 5.16 scrub starts 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.688595+0000 osd.5 (osd.5) 372 : cluster 0 5.16 scrub ok 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.759885+0000 osd.7 (osd.7) 348 : cluster 0 2.1c scrub starts 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:33.762290+0000 osd.7 (osd.7) 349 : cluster 0 2.1c scrub ok 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: audit 2024-03-11T21:38:34.449682+0000 mon.a (mon.0) 3148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: cluster 2024-03-11T21:38:34.457472+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: audit 2024-03-11T21:38:34.531166+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:34 smithi203 bash[23654]: audit 2024-03-11T21:38:34.532002+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:34.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.520374+0000 mgr.y (mgr.14550) 611 : cluster 0 pgmap v970: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:34.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.597962+0000 osd.6 (osd.6) 277 : cluster 0 2.11 scrub starts 2024-03-11T21:38:34.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.599334+0000 osd.6 (osd.6) 278 : cluster 0 2.11 scrub ok 2024-03-11T21:38:34.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.686157+0000 osd.5 (osd.5) 371 : cluster 0 5.16 scrub starts 2024-03-11T21:38:34.793 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.688595+0000 osd.5 (osd.5) 372 : cluster 0 5.16 scrub ok 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.759885+0000 osd.7 (osd.7) 348 : cluster 0 2.1c scrub starts 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:33.762290+0000 osd.7 (osd.7) 349 : cluster 0 2.1c scrub ok 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: audit 2024-03-11T21:38:34.449682+0000 mon.a (mon.0) 3148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: cluster 2024-03-11T21:38:34.457472+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: audit 2024-03-11T21:38:34.531166+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[21512]: audit 2024-03-11T21:38:34.532002+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.520374+0000 mgr.y (mgr.14550) 611 : cluster 0 pgmap v970: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.597962+0000 osd.6 (osd.6) 277 : cluster 0 2.11 scrub starts 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.599334+0000 osd.6 (osd.6) 278 : cluster 0 2.11 scrub ok 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.686157+0000 osd.5 (osd.5) 371 : cluster 0 5.16 scrub starts 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.688595+0000 osd.5 (osd.5) 372 : cluster 0 5.16 scrub ok 2024-03-11T21:38:34.794 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.759885+0000 osd.7 (osd.7) 348 : cluster 0 2.1c scrub starts 2024-03-11T21:38:34.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:33.762290+0000 osd.7 (osd.7) 349 : cluster 0 2.1c scrub ok 2024-03-11T21:38:34.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: audit 2024-03-11T21:38:34.449682+0000 mon.a (mon.0) 3148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:38:34.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: cluster 2024-03-11T21:38:34.457472+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:38:34.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: audit 2024-03-11T21:38:34.531166+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:34 smithi143 bash[29284]: audit 2024-03-11T21:38:34.532002+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:34.817 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:34.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.944 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:35.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.076 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.204 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.223 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:38:35.223 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:38:35.223 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:38:35.223 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:38:35.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:35.333 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:35.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.463 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:35.594 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:35.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:35.725 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:35.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:35.774 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: cluster 2024-03-11T21:38:34.912562+0000 osd.0 (osd.0) 352 : cluster 0 3.6 scrub starts 2024-03-11T21:38:35.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: cluster 2024-03-11T21:38:34.915146+0000 osd.0 (osd.0) 353 : cluster 0 3.6 scrub ok 2024-03-11T21:38:35.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: audit 2024-03-11T21:38:35.468341+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:35.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.474252+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:38:35.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: audit 2024-03-11T21:38:35.474371+0000 mon.c (mon.2) 1107 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:35 smithi203 bash[23654]: audit 2024-03-11T21:38:35.475564+0000 mon.a (mon.0) 3153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:35.853 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: cluster 2024-03-11T21:38:34.912562+0000 osd.0 (osd.0) 352 : cluster 0 3.6 scrub starts 2024-03-11T21:38:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: cluster 2024-03-11T21:38:34.915146+0000 osd.0 (osd.0) 353 : cluster 0 3.6 scrub ok 2024-03-11T21:38:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: audit 2024-03-11T21:38:35.468341+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.474252+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:38:35.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: audit 2024-03-11T21:38:35.474371+0000 mon.c (mon.2) 1107 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[29284]: audit 2024-03-11T21:38:35.475564+0000 mon.a (mon.0) 3153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: cluster 2024-03-11T21:38:34.912562+0000 osd.0 (osd.0) 352 : cluster 0 3.6 scrub starts 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: cluster 2024-03-11T21:38:34.915146+0000 osd.0 (osd.0) 353 : cluster 0 3.6 scrub ok 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: audit 2024-03-11T21:38:35.468341+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.474252+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: audit 2024-03-11T21:38:35.474371+0000 mon.c (mon.2) 1107 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:35 smithi143 bash[21512]: audit 2024-03-11T21:38:35.475564+0000 mon.a (mon.0) 3153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]: dispatch 2024-03-11T21:38:35.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:35.981 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:36.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:38:36.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:36.112 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:36.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:36.245 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:36.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:36.377 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:36.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:36.506 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:36.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.637 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:36.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.450986+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.453180+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.521044+0000 mgr.y (mgr.14550) 612 : cluster 0 pgmap v973: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.854443+0000 osd.2 (osd.2) 315 : cluster 0 2.e scrub starts 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:35.856548+0000 osd.2 (osd.2) 316 : cluster 0 2.e scrub ok 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.441454+0000 mon.a (mon.0) 3154 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: audit 2024-03-11T21:38:36.514122+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:36.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:36 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.524937+0000 mon.a (mon.0) 3156 : cluster 0 osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:38:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:36.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:36.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.450986+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.453180+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.521044+0000 mgr.y (mgr.14550) 612 : cluster 0 pgmap v973: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.854443+0000 osd.2 (osd.2) 315 : cluster 0 2.e scrub starts 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:35.856548+0000 osd.2 (osd.2) 316 : cluster 0 2.e scrub ok 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.441454+0000 mon.a (mon.0) 3154 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: audit 2024-03-11T21:38:36.514122+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:36.938 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.524937+0000 mon.a (mon.0) 3156 : cluster 0 osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.450986+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.453180+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.521044+0000 mgr.y (mgr.14550) 612 : cluster 0 pgmap v973: 270 pgs: 270 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.854443+0000 osd.2 (osd.2) 315 : cluster 0 2.e scrub starts 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:35.856548+0000 osd.2 (osd.2) 316 : cluster 0 2.e scrub ok 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.441454+0000 mon.a (mon.0) 3154 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: audit 2024-03-11T21:38:36.514122+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-98"}]': finished 2024-03-11T21:38:36.939 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:36 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.524937+0000 mon.a (mon.0) 3156 : cluster 0 osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:38:36.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:37.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:37.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:37.153 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:37.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:37.284 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:37.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:37.416 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:37.493 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:37.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:37.546 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:37.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:37.676 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:37.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:37.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:38.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:38.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:38.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:38.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.434586+0000 osd.4 (osd.4) 343 : cluster 0 9.16 scrub starts 2024-03-11T21:38:38.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.439800+0000 osd.4 (osd.4) 344 : cluster 0 9.16 scrub ok 2024-03-11T21:38:38.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.657335+0000 osd.5 (osd.5) 373 : cluster 0 5.15 scrub starts 2024-03-11T21:38:38.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.659765+0000 osd.5 (osd.5) 374 : cluster 0 5.15 scrub ok 2024-03-11T21:38:38.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.790409+0000 osd.1 (osd.1) 155 : cluster 0 6.4 scrub starts 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.792586+0000 osd.1 (osd.1) 156 : cluster 0 6.4 scrub ok 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.878075+0000 osd.2 (osd.2) 317 : cluster 0 2.10 scrub starts 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:36.880553+0000 osd.2 (osd.2) 318 : cluster 0 2.10 scrub ok 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: audit 2024-03-11T21:38:36.912885+0000 mon.c (mon.2) 1108 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[29284]: cluster 2024-03-11T21:38:37.520350+0000 mon.a (mon.0) 3157 : cluster 0 osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.434586+0000 osd.4 (osd.4) 343 : cluster 0 9.16 scrub starts 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.439800+0000 osd.4 (osd.4) 344 : cluster 0 9.16 scrub ok 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.657335+0000 osd.5 (osd.5) 373 : cluster 0 5.15 scrub starts 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.659765+0000 osd.5 (osd.5) 374 : cluster 0 5.15 scrub ok 2024-03-11T21:38:38.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.790409+0000 osd.1 (osd.1) 155 : cluster 0 6.4 scrub starts 2024-03-11T21:38:38.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.792586+0000 osd.1 (osd.1) 156 : cluster 0 6.4 scrub ok 2024-03-11T21:38:38.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.878075+0000 osd.2 (osd.2) 317 : cluster 0 2.10 scrub starts 2024-03-11T21:38:38.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:36.880553+0000 osd.2 (osd.2) 318 : cluster 0 2.10 scrub ok 2024-03-11T21:38:38.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: audit 2024-03-11T21:38:36.912885+0000 mon.c (mon.2) 1108 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:38.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:37 smithi143 bash[21512]: cluster 2024-03-11T21:38:37.520350+0000 mon.a (mon.0) 3157 : cluster 0 osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:38:38.170 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:38.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.434586+0000 osd.4 (osd.4) 343 : cluster 0 9.16 scrub starts 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.439800+0000 osd.4 (osd.4) 344 : cluster 0 9.16 scrub ok 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.657335+0000 osd.5 (osd.5) 373 : cluster 0 5.15 scrub starts 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.659765+0000 osd.5 (osd.5) 374 : cluster 0 5.15 scrub ok 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.790409+0000 osd.1 (osd.1) 155 : cluster 0 6.4 scrub starts 2024-03-11T21:38:38.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.792586+0000 osd.1 (osd.1) 156 : cluster 0 6.4 scrub ok 2024-03-11T21:38:38.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.878075+0000 osd.2 (osd.2) 317 : cluster 0 2.10 scrub starts 2024-03-11T21:38:38.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:36.880553+0000 osd.2 (osd.2) 318 : cluster 0 2.10 scrub ok 2024-03-11T21:38:38.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: audit 2024-03-11T21:38:36.912885+0000 mon.c (mon.2) 1108 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:38.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:37 smithi203 bash[23654]: cluster 2024-03-11T21:38:37.520350+0000 mon.a (mon.0) 3157 : cluster 0 osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:38:38.301 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:38.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:38.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:38.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:38.562 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:38.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:38.690 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:38.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:38.821 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:38.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:38.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:39.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:37.457804+0000 osd.4 (osd.4) 345 : cluster 0 5.1 scrub starts 2024-03-11T21:38:39.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:37.460127+0000 osd.4 (osd.4) 346 : cluster 0 5.1 scrub ok 2024-03-11T21:38:39.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:37.521789+0000 mgr.y (mgr.14550) 613 : cluster 0 pgmap v976: 238 pgs: 238 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:39.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:38.529440+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: audit 2024-03-11T21:38:38.548270+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: audit 2024-03-11T21:38:38.552447+0000 mon.a (mon.0) 3159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:38.736060+0000 osd.3 (osd.3) 367 : cluster 0 9.19 scrub starts 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[29284]: cluster 2024-03-11T21:38:38.738764+0000 osd.3 (osd.3) 368 : cluster 0 9.19 scrub ok 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:37.457804+0000 osd.4 (osd.4) 345 : cluster 0 5.1 scrub starts 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:37.460127+0000 osd.4 (osd.4) 346 : cluster 0 5.1 scrub ok 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:37.521789+0000 mgr.y (mgr.14550) 613 : cluster 0 pgmap v976: 238 pgs: 238 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:38.529440+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: audit 2024-03-11T21:38:38.548270+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: audit 2024-03-11T21:38:38.552447+0000 mon.a (mon.0) 3159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:38.736060+0000 osd.3 (osd.3) 367 : cluster 0 9.19 scrub starts 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:38 smithi143 bash[21512]: cluster 2024-03-11T21:38:38.738764+0000 osd.3 (osd.3) 368 : cluster 0 9.19 scrub ok 2024-03-11T21:38:39.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.084 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:37.457804+0000 osd.4 (osd.4) 345 : cluster 0 5.1 scrub starts 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:37.460127+0000 osd.4 (osd.4) 346 : cluster 0 5.1 scrub ok 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:37.521789+0000 mgr.y (mgr.14550) 613 : cluster 0 pgmap v976: 238 pgs: 238 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:38.529440+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: audit 2024-03-11T21:38:38.548270+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: audit 2024-03-11T21:38:38.552447+0000 mon.a (mon.0) 3159 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:38.736060+0000 osd.3 (osd.3) 367 : cluster 0 9.19 scrub starts 2024-03-11T21:38:39.085 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:38 smithi203 bash[23654]: cluster 2024-03-11T21:38:38.738764+0000 osd.3 (osd.3) 368 : cluster 0 9.19 scrub ok 2024-03-11T21:38:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:39.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.214 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:39.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 -- 172.21.15.143:0/1110273197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844106dc0 msgr2=0x7f28441071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 --2- 172.21.15.143:0/1110273197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844106dc0 0x7f28441071a0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f2834009990 tx=0x7f2834009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 -- 172.21.15.143:0/1110273197 shutdown_connections 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 --2- 172.21.15.143:0/1110273197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844101700 0x7f284410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 --2- 172.21.15.143:0/1110273197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844106dc0 0x7f28441071a0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 --2- 172.21.15.143:0/1110273197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2844100d40 0x7f28441011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.239+0000 7f284a9c8700 1 -- 172.21.15.143:0/1110273197 >> 172.21.15.143:0/1110273197 conn(0x7f2844076d80 msgr2=0x7f2844077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- 172.21.15.143:0/1110273197 shutdown_connections 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- 172.21.15.143:0/1110273197 wait complete. 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 librados: starting msgr at 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 librados: starting objecter 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 Processor -- start 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- start start 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 librados: setting wanted keys 2024-03-11T21:38:39.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 librados: calling monclient init 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 0x7f2844199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2844101700 0x7f284419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844106dc0 0x7f284419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844110520 con 0x7f2844106dc0 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28441103a0 con 0x7f2844101700 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28441106a0 con 0x7f2844100d40 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 0x7f2844199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 0x7f2844199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48172/0 (socket says 172.21.15.143:48172) 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 -- 172.21.15.143:0/354807447 learned_addr learned my addr 172.21.15.143:0/354807447 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844106dc0 0x7f284419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2844101700 0x7f284419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2844101700 msgr2=0x7f284419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2844101700 0x7f284419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844106dc0 msgr2=0x7f284419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844106dc0 0x7f284419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284419edc0 con 0x7f2844100d40 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 0x7f2844199db0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f28340057f0 tx=0x7f283400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 10 client.?.objecter ms_handle_connect 0x7f2844100d40 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834026070 con 0x7f2844100d40 2024-03-11T21:38:39.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2834021410 con 0x7f2844100d40 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284419f050 con 0x7f2844100d40 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 10 client.41846.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28441a6930 con 0x7f2844100d40 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 librados: init done 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834017650 con 0x7f2844100d40 2024-03-11T21:38:39.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f284404faa0 con 0x7f2844100d40 2024-03-11T21:38:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.243+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2834029070 con 0x7f2844100d40 2024-03-11T21:38:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f282c06f820 0x7f282c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(631..631 src has 87..631) v4 ==== 9649+0+0 (secure 0 0 0) 0x7f283408fa20 con 0x7f2844100d40 2024-03-11T21:38:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 osd_map(631..631 src has 87..631) v4 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 3 client.41846.objecter handle_osd_map got epochs [631,631] > 0 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 3 client.41846.objecter handle_osd_map decoding full epoch 631 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f282c06f820 0x7f282c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 20 client.41846.objecter dump_active .. 0 homeless 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v631) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2834013070 con 0x7f2844100d40 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f282c06f820 0x7f282c071ce0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f28440773f0 tx=0x7f283800b410 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f282c06f820 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 10 client.41846.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 1 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f284406a5e0 0x7f284402c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 1 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 0 1 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 4 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 10 client.41846.objecter _send_command 1 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2844062f00 con 0x7f284406a5e0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f284406a5e0 0x7f284402c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f284406a5e0 0x7f284402c6b0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f284406a5e0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2844062f00 con 0x7f284406a5e0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 10 client.41846.objecter _finish_command 1 = system:0 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 4 2024-03-11T21:38:39.227 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.247+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 0 1 2024-03-11T21:38:39.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 10 client.41846.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 2 2024-03-11T21:38:39.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.242 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 2 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f284406a470 osd=0 3 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 0 2 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 4 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 10 client.41846.objecter _send_command 2 2024-03-11T21:38:39.243 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.263+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f284402cbf0 con 0x7f284406a5e0 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f284402cbf0 con 0x7f284406a5e0 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 10 client.41846.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 4 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 0 2 2024-03-11T21:38:39.244 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:38:39.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 10 client.41846.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 3 2024-03-11T21:38:39.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f28440625e0 0x7f28441aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.245 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 1 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 4 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 10 client.41846.objecter _send_command 3 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f284402dae0 con 0x7f28440625e0 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f28440625e0 0x7f28441aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28440625e0 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f28440625e0 0x7f28441aab20 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f284402dae0 con 0x7f28440625e0 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:38:39.246 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 10 client.41846.objecter _finish_command 3 = system:0 2024-03-11T21:38:39.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 4 2024-03-11T21:38:39.247 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.267+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 1 3 2024-03-11T21:38:39.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 10 client.41846.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 4 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 4 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f2844062470 osd=1 3 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 1 4 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 4 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 10 client.41846.objecter _send_command 4 2024-03-11T21:38:39.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f284402d050 con 0x7f28440625e0 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f284402d050 con 0x7f28440625e0 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 10 client.41846.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 4 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 1 4 2024-03-11T21:38:39.265 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 10 client.41846.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 5 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f28441ac1e0 0x7f28441ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 5 2024-03-11T21:38:39.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 2 5 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 4 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 10 client.41846.objecter _send_command 5 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2844104750 con 0x7f28441ac1e0 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f28441ac1e0 0x7f28441ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441ac1e0 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f28441ac1e0 0x7f28441ae5d0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2844104750 con 0x7f28441ac1e0 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:38:39.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 10 client.41846.objecter _finish_command 5 = system:0 2024-03-11T21:38:39.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 4 2024-03-11T21:38:39.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.287+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 2 5 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 10 client.41846.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 6 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 6 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441040d0 osd=2 3 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 2 6 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 4 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 10 client.41846.objecter _send_command 6 2024-03-11T21:38:39.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2844103b50 con 0x7f28441ac1e0 2024-03-11T21:38:39.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2844103b50 con 0x7f28441ac1e0 2024-03-11T21:38:39.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 10 client.41846.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 4 2024-03-11T21:38:39.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 2 6 2024-03-11T21:38:39.286 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 10 client.41846.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 7 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f28441abd40 0x7f28441b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 7 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 3 7 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 4 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 10 client.41846.objecter _send_command 7 2024-03-11T21:38:39.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2844103f10 con 0x7f28441abd40 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f28441abd40 0x7f28441b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441abd40 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f28441abd40 0x7f28441b1ca0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f283c009040 con 0x7f28441abd40 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 10 client.41846.objecter _finish_command 7 = system:0 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 4 2024-03-11T21:38:39.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.307+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 3 7 2024-03-11T21:38:39.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 10 client.41846.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 8 2024-03-11T21:38:39.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 8 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f2844103da0 osd=3 3 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 3 8 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 4 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 10 client.41846.objecter _send_command 8 2024-03-11T21:38:39.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28441ab3d0 con 0x7f28441abd40 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f28441ab3d0 con 0x7f28441abd40 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f28417fa700 10 client.41846.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 4 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 3 8 2024-03-11T21:38:39.307 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 10 client.41846.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 9 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f28441af7f0 0x7f28441b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 9 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 4 9 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 4 2024-03-11T21:38:39.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 10 client.41846.objecter _send_command 9 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28441b8340 con 0x7f28441af7f0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.327+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f28441af7f0 0x7f28441b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441af7f0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f28441af7f0 0x7f28441b3360 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28441b8340 con 0x7f28441af7f0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 10 client.41846.objecter _finish_command 9 = system:0 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 4 2024-03-11T21:38:39.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.331+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 4 9 2024-03-11T21:38:39.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 10 client.41846.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 10 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 10 2024-03-11T21:38:39.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441ab3d0 osd=4 3 2024-03-11T21:38:39.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 4 10 2024-03-11T21:38:39.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 4 2024-03-11T21:38:39.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 10 client.41846.objecter _send_command 10 2024-03-11T21:38:39.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.347+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28441b38a0 con 0x7f28441af7f0 2024-03-11T21:38:39.330 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f28441b38a0 con 0x7f28441af7f0 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f28417fa700 10 client.41846.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 4 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 4 10 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 10 client.41846.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 11 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f28441b3fc0 0x7f28441b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 11 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 5 11 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 4 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 10 client.41846.objecter _send_command 11 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.351+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28441bb4d0 con 0x7f28441b3fc0 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f28441b3fc0 0x7f28441b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441b3fc0 2024-03-11T21:38:39.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28437fe700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f28441b3fc0 0x7f28441b4400 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28441bb4d0 con 0x7f28441b3fc0 2024-03-11T21:38:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:38:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 10 client.41846.objecter _finish_command 11 = system:0 2024-03-11T21:38:39.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 4 2024-03-11T21:38:39.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.355+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 5 11 2024-03-11T21:38:39.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:39.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 10 client.41846.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 12 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 12 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b38a0 osd=5 3 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 5 12 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 4 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 10 client.41846.objecter _send_command 12 2024-03-11T21:38:39.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.371+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28441b8bb0 con 0x7f28441b3fc0 2024-03-11T21:38:39.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f28441b8bb0 con 0x7f28441b3fc0 2024-03-11T21:38:39.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 10 client.41846.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 4 2024-03-11T21:38:39.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 5 12 2024-03-11T21:38:39.354 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:38:39.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 10 client.41846.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 13 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f28441b92d0 0x7f28441b9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 13 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 6 13 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 4 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 10 client.41846.objecter _send_command 13 2024-03-11T21:38:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28441be8a0 con 0x7f28441b92d0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f28441b92d0 0x7f28441b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441b92d0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.375+0000 7f2848f65700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f28441b92d0 0x7f28441b9750 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.379+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28441be8a0 con 0x7f28441b92d0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.379+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.379+0000 7f28417fa700 10 client.41846.objecter _finish_command 13 = system:0 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.379+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 4 2024-03-11T21:38:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.379+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 6 13 2024-03-11T21:38:39.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 10 client.41846.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 14 2024-03-11T21:38:39.372 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 14 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b8bb0 osd=6 3 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 6 14 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 4 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 10 client.41846.objecter _send_command 14 2024-03-11T21:38:39.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.395+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28441b9da0 con 0x7f28441b92d0 2024-03-11T21:38:39.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f28441b9da0 con 0x7f28441b92d0 2024-03-11T21:38:39.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 10 client.41846.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 4 2024-03-11T21:38:39.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 6 14 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 10 client.41846.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 15 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f28441ba4c0 0x7f28441ba920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 15 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 7 15 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 4 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 10 client.41846.objecter _send_command 15 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28441baf70 con 0x7f28441ba4c0 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f28441ba4c0 0x7f28441ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f2843fff700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f28441ba4c0 0x7f28441ba920 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 10 client.41846.objecter ms_handle_connect 0x7f28441ba4c0 2024-03-11T21:38:39.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28441baf70 con 0x7f28441ba4c0 2024-03-11T21:38:39.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:38:39.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 10 client.41846.objecter _finish_command 15 = system:0 2024-03-11T21:38:39.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 4 2024-03-11T21:38:39.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.399+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 7 15 2024-03-11T21:38:39.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:38:39.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign -1 16 2024-03-11T21:38:39.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter _get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_remove -1 16 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter get_session s=0x7f28441b9da0 osd=7 3 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 15 client.41846.objecter _session_command_op_assign 7 16 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 4 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter _send_command 16 2024-03-11T21:38:39.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28441baf70 con 0x7f28441ba4c0 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f28417fa700 1 -- 172.21.15.143:0/354807447 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f28441baf70 con 0x7f28441ba4c0 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f28417fa700 10 client.41846.objecter ms_dispatch 0x7f2844001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f28417fa700 10 client.41846.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f28417fa700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 4 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f28417fa700 15 client.41846.objecter _session_command_op_remove 7 16 2024-03-11T21:38:39.398 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:38:39.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 librados: watch_flush enter 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 librados: watch_flush exit 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.0 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f284406a5e0 msgr2=0x7f284402c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f284406a5e0 0x7f284402c6b0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f284406a470 osd=0 1 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.1 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f28440625e0 msgr2=0x7f28441aab20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f28440625e0 0x7f28441aab20 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844062470 osd=1 1 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.2 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f28441ac1e0 msgr2=0x7f28441ae5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f28441ac1e0 0x7f28441ae5d0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441040d0 osd=2 1 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.3 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f28441abd40 msgr2=0x7f28441b1ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f28441abd40 0x7f28441b1ca0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f2844103da0 osd=3 1 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.4 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f28441af7f0 msgr2=0x7f28441b3360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f28441af7f0 0x7f28441b3360 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441ab3d0 osd=4 1 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.5 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f28441b3fc0 msgr2=0x7f28441b4400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f28441b3fc0 0x7f28441b4400 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f2848f65700 1 -- 172.21.15.143:0/354807447 reap_dead start 2024-03-11T21:38:39.401 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f2848f65700 1 -- 172.21.15.143:0/354807447 reap_dead start 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f2848f65700 1 -- 172.21.15.143:0/354807447 reap_dead start 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b38a0 osd=5 1 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.6 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f28441b92d0 msgr2=0x7f28441b9750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f28441b92d0 0x7f28441b9750 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b8bb0 osd=6 1 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter close_session for osd.7 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f28441ba4c0 msgr2=0x7f28441ba920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f28441ba4c0 0x7f28441ba920 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter put_session s=0x7f28441b9da0 osd=7 1 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 20 client.41846.objecter shutdown clearing up homeless session... 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 10 client.41846.objecter successfully canceled tick 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f282c06f820 msgr2=0x7f282c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.402 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f282c06f820 0x7f282c071ce0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f28440773f0 tx=0x7f283800b410 comp rx=0 tx=0).stop 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 msgr2=0x7f2844199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 --2- 172.21.15.143:0/354807447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2844100d40 0x7f2844199db0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f28340057f0 tx=0x7f283400bfa0 comp rx=0 tx=0).stop 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f2848f65700 1 -- 172.21.15.143:0/354807447 reap_dead start 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 shutdown_connections 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 >> 172.21.15.143:0/354807447 conn(0x7f2844076d80 msgr2=0x7f2844069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.419+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 shutdown_connections 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.423+0000 7f284a9c8700 1 -- 172.21.15.143:0/354807447 wait complete. 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:39.423+0000 7f284a9c8700 1 librados: shutdown 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:39.403 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:39.404 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:39.404 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:39.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.472 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:39.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:39.595 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:39.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:39.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:39.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:39.829 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:39.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:39.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:39.984 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: cluster 2024-03-11T21:38:38.513575+0000 osd.6 (osd.6) 279 : cluster 0 5.1f scrub starts 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: cluster 2024-03-11T21:38:38.516652+0000 osd.6 (osd.6) 280 : cluster 0 5.1f scrub ok 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: audit 2024-03-11T21:38:38.846347+0000 mgr.y (mgr.14550) 614 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: audit 2024-03-11T21:38:39.247780+0000 mon.c (mon.2) 1110 : audit 0 from='client.? 172.21.15.143:0/354807447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: audit 2024-03-11T21:38:39.527718+0000 mon.a (mon.0) 3160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: cluster 2024-03-11T21:38:39.535084+0000 mon.a (mon.0) 3161 : cluster 0 osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: audit 2024-03-11T21:38:39.536712+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: audit 2024-03-11T21:38:39.537933+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: cluster 2024-03-11T21:38:39.755114+0000 osd.3 (osd.3) 369 : cluster 0 6.1b scrub starts 2024-03-11T21:38:40.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:39 smithi203 bash[23654]: cluster 2024-03-11T21:38:39.757303+0000 osd.3 (osd.3) 370 : cluster 0 6.1b scrub ok 2024-03-11T21:38:40.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:40.112 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:40.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:40.227 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: cluster 2024-03-11T21:38:38.513575+0000 osd.6 (osd.6) 279 : cluster 0 5.1f scrub starts 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: cluster 2024-03-11T21:38:38.516652+0000 osd.6 (osd.6) 280 : cluster 0 5.1f scrub ok 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: audit 2024-03-11T21:38:38.846347+0000 mgr.y (mgr.14550) 614 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: audit 2024-03-11T21:38:39.247780+0000 mon.c (mon.2) 1110 : audit 0 from='client.? 172.21.15.143:0/354807447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: audit 2024-03-11T21:38:39.527718+0000 mon.a (mon.0) 3160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: cluster 2024-03-11T21:38:39.535084+0000 mon.a (mon.0) 3161 : cluster 0 osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: audit 2024-03-11T21:38:39.536712+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: audit 2024-03-11T21:38:39.537933+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: cluster 2024-03-11T21:38:39.755114+0000 osd.3 (osd.3) 369 : cluster 0 6.1b scrub starts 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[21512]: cluster 2024-03-11T21:38:39.757303+0000 osd.3 (osd.3) 370 : cluster 0 6.1b scrub ok 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: cluster 2024-03-11T21:38:38.513575+0000 osd.6 (osd.6) 279 : cluster 0 5.1f scrub starts 2024-03-11T21:38:40.228 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: cluster 2024-03-11T21:38:38.516652+0000 osd.6 (osd.6) 280 : cluster 0 5.1f scrub ok 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: audit 2024-03-11T21:38:38.846347+0000 mgr.y (mgr.14550) 614 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: audit 2024-03-11T21:38:39.247780+0000 mon.c (mon.2) 1110 : audit 0 from='client.? 172.21.15.143:0/354807447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: audit 2024-03-11T21:38:39.527718+0000 mon.a (mon.0) 3160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: cluster 2024-03-11T21:38:39.535084+0000 mon.a (mon.0) 3161 : cluster 0 osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: audit 2024-03-11T21:38:39.536712+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: audit 2024-03-11T21:38:39.537933+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: cluster 2024-03-11T21:38:39.755114+0000 osd.3 (osd.3) 369 : cluster 0 6.1b scrub starts 2024-03-11T21:38:40.229 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:39 smithi143 bash[29284]: cluster 2024-03-11T21:38:39.757303+0000 osd.3 (osd.3) 370 : cluster 0 6.1b scrub ok 2024-03-11T21:38:40.244 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:40.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:40.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:40.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:40.505 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:40.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:40.638 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:40.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:40.768 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:40.818 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:40.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:40.895 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:40.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:41.009 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[29284]: cluster 2024-03-11T21:38:39.522965+0000 mgr.y (mgr.14550) 615 : cluster 0 pgmap v978: 270 pgs: 5 creating+peering, 27 unknown, 238 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:41.009 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[29284]: audit 2024-03-11T21:38:40.530391+0000 mon.a (mon.0) 3163 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[29284]: cluster 2024-03-11T21:38:40.534745+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[29284]: audit 2024-03-11T21:38:40.536679+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[29284]: audit 2024-03-11T21:38:40.537346+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[21512]: cluster 2024-03-11T21:38:39.522965+0000 mgr.y (mgr.14550) 615 : cluster 0 pgmap v978: 270 pgs: 5 creating+peering, 27 unknown, 238 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[21512]: audit 2024-03-11T21:38:40.530391+0000 mon.a (mon.0) 3163 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[21512]: cluster 2024-03-11T21:38:40.534745+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[21512]: audit 2024-03-11T21:38:40.536679+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.010 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:40 smithi143 bash[21512]: audit 2024-03-11T21:38:40.537346+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:41.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:41.153 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:41.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:41.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:40 smithi203 bash[23654]: cluster 2024-03-11T21:38:39.522965+0000 mgr.y (mgr.14550) 615 : cluster 0 pgmap v978: 270 pgs: 5 creating+peering, 27 unknown, 238 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:41.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:40 smithi203 bash[23654]: audit 2024-03-11T21:38:40.530391+0000 mon.a (mon.0) 3163 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:38:41.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:40 smithi203 bash[23654]: cluster 2024-03-11T21:38:40.534745+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:38:41.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:40 smithi203 bash[23654]: audit 2024-03-11T21:38:40.536679+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:40 smithi203 bash[23654]: audit 2024-03-11T21:38:40.537346+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:41.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:41.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:41.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:41.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:41.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:41.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:41.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:41.805 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:41.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.935 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:42.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:42.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:40.780713+0000 osd.7 (osd.7) 350 : cluster 0 39.9 scrub starts 2024-03-11T21:38:42.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:40.784788+0000 osd.7 (osd.7) 351 : cluster 0 39.9 scrub ok 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.443299+0000 mon.a (mon.0) 3166 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: audit 2024-03-11T21:38:41.446100+0000 mon.a (mon.0) 3167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.453569+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: audit 2024-03-11T21:38:41.456312+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: audit 2024-03-11T21:38:41.457091+0000 mon.a (mon.0) 3169 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.526723+0000 osd.4 (osd.4) 347 : cluster 0 6.1a scrub starts 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.530397+0000 osd.4 (osd.4) 348 : cluster 0 6.1a scrub ok 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:40.780713+0000 osd.7 (osd.7) 350 : cluster 0 39.9 scrub starts 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:40.784788+0000 osd.7 (osd.7) 351 : cluster 0 39.9 scrub ok 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.443299+0000 mon.a (mon.0) 3166 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: audit 2024-03-11T21:38:41.446100+0000 mon.a (mon.0) 3167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:42.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.453569+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:38:42.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: audit 2024-03-11T21:38:41.456312+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: audit 2024-03-11T21:38:41.457091+0000 mon.a (mon.0) 3169 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.526723+0000 osd.4 (osd.4) 347 : cluster 0 6.1a scrub starts 2024-03-11T21:38:42.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:41 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.530397+0000 osd.4 (osd.4) 348 : cluster 0 6.1a scrub ok 2024-03-11T21:38:42.060 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:42.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/4285957737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440100d40 msgr2=0x7f04401011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- 172.21.15.143:0/4285957737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440100d40 0x7f04401011c0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f0430009990 tx=0x7f0430009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/4285957737 shutdown_connections 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- 172.21.15.143:0/4285957737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 0x7f04401071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- 172.21.15.143:0/4285957737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440101700 0x7f044010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- 172.21.15.143:0/4285957737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440100d40 0x7f04401011c0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/4285957737 >> 172.21.15.143:0/4285957737 conn(0x7f0440076d80 msgr2=0x7f0440077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/4285957737 shutdown_connections 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/4285957737 wait complete. 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 librados: starting msgr at 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 librados: starting objecter 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 Processor -- start 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- start start 2024-03-11T21:38:42.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 librados: setting wanted keys 2024-03-11T21:38:42.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 librados: calling monclient init 2024-03-11T21:38:42.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 0x7f04401989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:42.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:42.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 0x7f04401a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440110570 con 0x7f0440106dc0 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04401103f0 con 0x7f0440100d40 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04401106f0 con 0x7f0440101700 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0444bc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 0x7f04401a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48196/0 (socket says 172.21.15.143:48196) 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 -- 172.21.15.143:0/3774759128 learned_addr learned my addr 172.21.15.143:0/3774759128 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043ffff700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 0x7f04401989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:42.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 -- 172.21.15.143:0/3774759128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 msgr2=0x7f04401989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 0x7f04401989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 -- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 msgr2=0x7f04401a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 0x7f04401a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 -- 172.21.15.143:0/3774759128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04401a3f60 con 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043ffff700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 0x7f04401989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043f7fe700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f0430000c00 tx=0x7f04300047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043d7fa700 10 client.?.objecter ms_handle_connect 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430027070 con 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0430022470 con 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04401a4190 con 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430017650 con 0x7f0440101700 2024-03-11T21:38:42.158 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 10 client.41852.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:42.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04401a45e0 con 0x7f0440101700 2024-03-11T21:38:42.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.175+0000 7f0446626700 1 librados: init done 2024-03-11T21:38:42.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044004faa0 con 0x7f0440101700 2024-03-11T21:38:42.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f043002a070 con 0x7f0440101700 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 0x7f042c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(634..634 src has 87..634) v4 ==== 9670+0+0 (secure 0 0 0) 0x7f043008fa80 con 0x7f0440101700 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 10 client.41852.objecter ms_dispatch 0x7f0440001550 osd_map(634..634 src has 87..634) v4 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 3 client.41852.objecter handle_osd_map got epochs [634,634] > 0 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 3 client.41852.objecter handle_osd_map decoding full epoch 634 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043ffff700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 0x7f042c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 20 client.41852.objecter dump_active .. 0 homeless 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043d7fa700 10 client.41852.objecter ms_handle_connect 0x7f042c06f820 2024-03-11T21:38:42.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.179+0000 7f043ffff700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 0x7f042c071ce0 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f0428009cc0 tx=0x7f0428009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:42.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.183+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0430013070 con 0x7f0440101700 2024-03-11T21:38:42.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:42.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:42.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:42.207 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:42.249 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:40.780713+0000 osd.7 (osd.7) 350 : cluster 0 39.9 scrub starts 2024-03-11T21:38:42.249 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:40.784788+0000 osd.7 (osd.7) 351 : cluster 0 39.9 scrub ok 2024-03-11T21:38:42.249 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.443299+0000 mon.a (mon.0) 3166 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: audit 2024-03-11T21:38:41.446100+0000 mon.a (mon.0) 3167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-6", "overlaypool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.453569+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: audit 2024-03-11T21:38:41.456312+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: audit 2024-03-11T21:38:41.457091+0000 mon.a (mon.0) 3169 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]: dispatch 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.526723+0000 osd.4 (osd.4) 347 : cluster 0 6.1a scrub starts 2024-03-11T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:41 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.530397+0000 osd.4 (osd.4) 348 : cluster 0 6.1a scrub ok 2024-03-11T21:38:42.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:42.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.399+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04401081c0 con 0x7f042c06f820 2024-03-11T21:38:42.405 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.427+0000 7f043d7fa700 1 -- 172.21.15.143:0/3774759128 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+687356 (secure 0 0 0) 0x7f04401081c0 con 0x7f042c06f820 2024-03-11T21:38:42.408 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 10 librados: watch_flush enter 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 10 librados: watch_flush exit 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 20 client.41852.objecter shutdown clearing up homeless session... 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 10 client.41852.objecter successfully canceled tick 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 msgr2=0x7f042c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 0x7f042c071ce0 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f0428009cc0 tx=0x7f0428009260 comp rx=0 tx=0).stop 2024-03-11T21:38:42.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 msgr2=0x7f0440198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f0430000c00 tx=0x7f04300047b0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 shutdown_connections 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f042c06f820 0x7f042c071ce0 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0440106dc0 0x7f04401a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440100d40 0x7f04401989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 --2- 172.21.15.143:0/3774759128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0440101700 0x7f0440198ee0 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 >> 172.21.15.143:0/3774759128 conn(0x7f0440076d80 msgr2=0x7f0440107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 shutdown_connections 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 -- 172.21.15.143:0/3774759128 wait complete. 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:42.431+0000 7f0446626700 1 librados: shutdown 2024-03-11T21:38:42.421 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:38:42.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:42.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:42.606 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:42.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:42.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:42.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.868 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:42.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:42.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.523830+0000 mgr.y (mgr.14550) 616 : cluster 0 pgmap v982: 270 pgs: 5 creating+peering, 22 unknown, 243 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.580358+0000 osd.5 (osd.5) 375 : cluster 0 5.f scrub starts 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.581745+0000 osd.5 (osd.5) 376 : cluster 0 5.f scrub ok 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.751655+0000 osd.1 (osd.1) 157 : cluster 0 39.7 scrub starts 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.757306+0000 osd.1 (osd.1) 158 : cluster 0 39.7 scrub ok 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.778021+0000 osd.7 (osd.7) 352 : cluster 0 9.3 scrub starts 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:41.783965+0000 osd.7 (osd.7) 353 : cluster 0 9.3 scrub ok 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.446820+0000 mon.a (mon.0) 3170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: audit 2024-03-11T21:38:42.451945+0000 mon.a (mon.0) 3171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]': finished 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.455828+0000 mon.a (mon.0) 3172 : cluster 0 osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:38:43.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: audit 2024-03-11T21:38:42.457874+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: audit 2024-03-11T21:38:42.459009+0000 mon.a (mon.0) 3173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.522788+0000 osd.4 (osd.4) 349 : cluster 0 6.2 scrub starts 2024-03-11T21:38:43.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.524501+0000 osd.4 (osd.4) 350 : cluster 0 6.2 scrub ok 2024-03-11T21:38:43.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.805831+0000 osd.3 (osd.3) 371 : cluster 0 3.b scrub starts 2024-03-11T21:38:43.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:42 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.809665+0000 osd.3 (osd.3) 372 : cluster 0 3.b scrub ok 2024-03-11T21:38:43.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.523830+0000 mgr.y (mgr.14550) 616 : cluster 0 pgmap v982: 270 pgs: 5 creating+peering, 22 unknown, 243 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.580358+0000 osd.5 (osd.5) 375 : cluster 0 5.f scrub starts 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.581745+0000 osd.5 (osd.5) 376 : cluster 0 5.f scrub ok 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.751655+0000 osd.1 (osd.1) 157 : cluster 0 39.7 scrub starts 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.757306+0000 osd.1 (osd.1) 158 : cluster 0 39.7 scrub ok 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.778021+0000 osd.7 (osd.7) 352 : cluster 0 9.3 scrub starts 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:41.783965+0000 osd.7 (osd.7) 353 : cluster 0 9.3 scrub ok 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.446820+0000 mon.a (mon.0) 3170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: audit 2024-03-11T21:38:42.451945+0000 mon.a (mon.0) 3171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]': finished 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.455828+0000 mon.a (mon.0) 3172 : cluster 0 osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:38:43.113 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: audit 2024-03-11T21:38:42.457874+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: audit 2024-03-11T21:38:42.459009+0000 mon.a (mon.0) 3173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.522788+0000 osd.4 (osd.4) 349 : cluster 0 6.2 scrub starts 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.524501+0000 osd.4 (osd.4) 350 : cluster 0 6.2 scrub ok 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.805831+0000 osd.3 (osd.3) 371 : cluster 0 3.b scrub starts 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.809665+0000 osd.3 (osd.3) 372 : cluster 0 3.b scrub ok 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.523830+0000 mgr.y (mgr.14550) 616 : cluster 0 pgmap v982: 270 pgs: 5 creating+peering, 22 unknown, 243 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.580358+0000 osd.5 (osd.5) 375 : cluster 0 5.f scrub starts 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.581745+0000 osd.5 (osd.5) 376 : cluster 0 5.f scrub ok 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.751655+0000 osd.1 (osd.1) 157 : cluster 0 39.7 scrub starts 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.757306+0000 osd.1 (osd.1) 158 : cluster 0 39.7 scrub ok 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.778021+0000 osd.7 (osd.7) 352 : cluster 0 9.3 scrub starts 2024-03-11T21:38:43.114 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:41.783965+0000 osd.7 (osd.7) 353 : cluster 0 9.3 scrub ok 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.446820+0000 mon.a (mon.0) 3170 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: audit 2024-03-11T21:38:42.451945+0000 mon.a (mon.0) 3171 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-100", "mode": "writeback"}]': finished 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.455828+0000 mon.a (mon.0) 3172 : cluster 0 osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: audit 2024-03-11T21:38:42.457874+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: audit 2024-03-11T21:38:42.459009+0000 mon.a (mon.0) 3173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.522788+0000 osd.4 (osd.4) 349 : cluster 0 6.2 scrub starts 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.524501+0000 osd.4 (osd.4) 350 : cluster 0 6.2 scrub ok 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.805831+0000 osd.3 (osd.3) 371 : cluster 0 3.b scrub starts 2024-03-11T21:38:43.115 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:42 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.809665+0000 osd.3 (osd.3) 372 : cluster 0 3.b scrub ok 2024-03-11T21:38:43.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:43.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:43.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:43.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:43.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:43.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:43.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:43.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:43.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:43.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:43.781 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:43.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:43.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:43.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:38:44.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: audit 2024-03-11T21:38:42.401128+0000 mgr.y (mgr.14550) 617 : audit 0 from='client.41852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:44.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.768825+0000 osd.0 (osd.0) 354 : cluster 0 5.1e scrub starts 2024-03-11T21:38:44.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.770935+0000 osd.0 (osd.0) 355 : cluster 0 5.1e scrub ok 2024-03-11T21:38:44.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.802046+0000 osd.7 (osd.7) 354 : cluster 0 3.15 scrub starts 2024-03-11T21:38:44.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: cluster 2024-03-11T21:38:42.804513+0000 osd.7 (osd.7) 355 : cluster 0 3.15 scrub ok 2024-03-11T21:38:44.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: audit 2024-03-11T21:38:43.454087+0000 mon.a (mon.0) 3174 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:38:44.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: cluster 2024-03-11T21:38:43.459736+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:38:44.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: audit 2024-03-11T21:38:43.463414+0000 mon.c (mon.2) 1115 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:43 smithi203 bash[23654]: audit 2024-03-11T21:38:43.464080+0000 mon.a (mon.0) 3176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.042 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:44.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.169 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:44.241 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:44.242 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:44.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: audit 2024-03-11T21:38:42.401128+0000 mgr.y (mgr.14550) 617 : audit 0 from='client.41852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.768825+0000 osd.0 (osd.0) 354 : cluster 0 5.1e scrub starts 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.770935+0000 osd.0 (osd.0) 355 : cluster 0 5.1e scrub ok 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.802046+0000 osd.7 (osd.7) 354 : cluster 0 3.15 scrub starts 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: cluster 2024-03-11T21:38:42.804513+0000 osd.7 (osd.7) 355 : cluster 0 3.15 scrub ok 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: audit 2024-03-11T21:38:43.454087+0000 mon.a (mon.0) 3174 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: cluster 2024-03-11T21:38:43.459736+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: audit 2024-03-11T21:38:43.463414+0000 mon.c (mon.2) 1115 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.282 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[21512]: audit 2024-03-11T21:38:43.464080+0000 mon.a (mon.0) 3176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: audit 2024-03-11T21:38:42.401128+0000 mgr.y (mgr.14550) 617 : audit 0 from='client.41852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.768825+0000 osd.0 (osd.0) 354 : cluster 0 5.1e scrub starts 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.770935+0000 osd.0 (osd.0) 355 : cluster 0 5.1e scrub ok 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.802046+0000 osd.7 (osd.7) 354 : cluster 0 3.15 scrub starts 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: cluster 2024-03-11T21:38:42.804513+0000 osd.7 (osd.7) 355 : cluster 0 3.15 scrub ok 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: audit 2024-03-11T21:38:43.454087+0000 mon.a (mon.0) 3174 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: cluster 2024-03-11T21:38:43.459736+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: audit 2024-03-11T21:38:43.463414+0000 mon.c (mon.2) 1115 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:43 smithi143 bash[29284]: audit 2024-03-11T21:38:43.464080+0000 mon.a (mon.0) 3176 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:38:44.285 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":982,"stamp":"2024-03-11T21:38:41.523271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811221,"num_objects":401,"num_object_clones":16,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":2307,"num_read_kb":1901,"num_write":1521,"num_write_kb":12871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4423,"ondisk_log_size":4423,"up":744,"acting":744,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":654392,"kb_used_data":109624,"kb_used_omap":38,"kb_used_meta":544473,"kb_avail":749306824,"statfs":{"total":767960285184,"available":767290187776,"internally_reserved":0,"allocated":112254976,"data_stored":78257172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":557540789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002016"},"pg_stats":[{"pgid":"306.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1d","version":"0'0","reported_seq":10,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.563171+0000","last_change":"2024-03-11T21:38:39.559433+0000","last_active":"2024-03-11T21:38:39.563171+0000","last_peered":"2024-03-11T21:38:39.563171+0000","last_clean":"2024-03-11T21:38:39.563171+0000","last_became_active":"2024-03-11T21:38:39.556630+0000","last_became_peered":"2024-03-11T21:38:39.556630+0000","last_unstale":"2024-03-11T21:38:39.563171+0000","last_undegraded":"2024-03-11T21:38:39.563171+0000","last_fullsized":"2024-03-11T21:38:39.563171+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:56.556828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"306.1c","version":"0'0","reported_seq":1,"reported_epoch":631,"state":"creating+peering","last_fresh":"2024-03-11T21:38:38.539563+0000","last_change":"2024-03-11T21:38:38.539563+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.539563+0000","last_undegraded":"2024-03-11T21:38:38.539563+0000","last_fullsized":"2024-03-11T21:38:38.539563+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"306.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.17","version":"0'0","reported_seq":1,"reported_epoch":631,"state":"creating+peering","last_fresh":"2024-03-11T21:38:38.539939+0000","last_change":"2024-03-11T21:38:38.539939+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.539939+0000","last_undegraded":"2024-03-11T21:38:38.539939+0000","last_fullsized":"2024-03-11T21:38:38.539939+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"306.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.13","version":"0'0","reported_seq":1,"reported_epoch":631,"state":"creating+peering","last_fresh":"2024-03-11T21:38:38.541021+0000","last_change":"2024-03-11T21:38:38.541021+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.541021+0000","last_undegraded":"2024-03-11T21:38:38.541021+0000","last_fullsized":"2024-03-11T21:38:38.541021+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"306.12","version":"0'0","reported_seq":10,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.564910+0000","last_change":"2024-03-11T21:38:39.564266+0000","last_active":"2024-03-11T21:38:39.564910+0000","last_peered":"2024-03-11T21:38:39.564910+0000","last_clean":"2024-03-11T21:38:39.564910+0000","last_became_active":"2024-03-11T21:38:39.563937+0000","last_became_peered":"2024-03-11T21:38:39.563937+0000","last_unstale":"2024-03-11T21:38:39.564910+0000","last_undegraded":"2024-03-11T21:38:39.564910+0000","last_fullsized":"2024-03-11T21:38:39.564910+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:08.337406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"306.11","version":"0'0","reported_seq":10,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.563590+0000","last_change":"2024-03-11T21:38:39.562932+0000","last_active":"2024-03-11T21:38:39.563590+0000","last_peered":"2024-03-11T21:38:39.563590+0000","last_clean":"2024-03-11T21:38:39.563590+0000","last_became_active":"2024-03-11T21:38:39.562469+0000","last_became_peered":"2024-03-11T21:38:39.562469+0000","last_unstale":"2024-03-11T21:38:39.563590+0000","last_undegraded":"2024-03-11T21:38:39.563590+0000","last_fullsized":"2024-03-11T21:38:39.563590+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:43.727063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"306.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.5","version":"0'0","reported_seq":10,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.563144+0000","last_change":"2024-03-11T21:38:39.559423+0000","last_active":"2024-03-11T21:38:39.563144+0000","last_peered":"2024-03-11T21:38:39.563144+0000","last_clean":"2024-03-11T21:38:39.563144+0000","last_became_active":"2024-03-11T21:38:39.556325+0000","last_became_peered":"2024-03-11T21:38:39.556325+0000","last_unstale":"2024-03-11T21:38:39.563144+0000","last_undegraded":"2024-03-11T21:38:39.563144+0000","last_fullsized":"2024-03-11T21:38:39.563144+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:46.157289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"306.4","version":"0'0","reported_seq":1,"reported_epoch":631,"state":"creating+peering","last_fresh":"2024-03-11T21:38:38.538966+0000","last_change":"2024-03-11T21:38:38.538966+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.538966+0000","last_undegraded":"2024-03-11T21:38:38.538966+0000","last_fullsized":"2024-03-11T21:38:38.538966+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"306.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.2","version":"0'0","reported_seq":1,"reported_epoch":631,"state":"creating+peering","last_fresh":"2024-03-11T21:38:38.538008+0000","last_change":"2024-03-11T21:38:38.538008+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.538008+0000","last_undegraded":"2024-03-11T21:38:38.538008+0000","last_fullsized":"2024-03-11T21:38:38.538008+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"306.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:38:38.516528+0000","last_change":"2024-03-11T21:38:38.516528+0000","last_active":"2024-03-11T21:38:38.516528+0000","last_peered":"2024-03-11T21:38:38.516528+0000","last_clean":"2024-03-11T21:38:38.516528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:38:38.516528+0000","last_undegraded":"2024-03-11T21:38:38.516528+0000","last_fullsized":"2024-03-11T21:38:38.516528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.0","version":"0'0","reported_seq":10,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.563131+0000","last_change":"2024-03-11T21:38:39.558362+0000","last_active":"2024-03-11T21:38:39.563131+0000","last_peered":"2024-03-11T21:38:39.563131+0000","last_clean":"2024-03-11T21:38:39.563131+0000","last_became_active":"2024-03-11T21:38:39.556484+0000","last_became_peered":"2024-03-11T21:38:39.556484+0000","last_unstale":"2024-03-11T21:38:39.563131+0000","last_undegraded":"2024-03-11T21:38:39.563131+0000","last_fullsized":"2024-03-11T21:38:39.563131+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":631,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:38.516528+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.516528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:55.893805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.9","version":"573'28","reported_seq":1212,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.764003+0000","last_change":"2024-03-11T21:38:32.837289+0000","last_active":"2024-03-11T21:38:37.764003+0000","last_peered":"2024-03-11T21:38:37.764003+0000","last_clean":"2024-03-11T21:38:37.764003+0000","last_became_active":"2024-03-11T21:29:32.624776+0000","last_became_peered":"2024-03-11T21:29:32.624776+0000","last_unstale":"2024-03-11T21:38:37.764003+0000","last_undegraded":"2024-03-11T21:38:37.764003+0000","last_fullsized":"2024-03-11T21:38:37.764003+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'28","last_scrub_stamp":"2024-03-11T21:38:32.837180+0000","last_deep_scrub":"482'18","last_deep_scrub_stamp":"2024-03-11T21:34:56.441729+0000","last_clean_scrub_stamp":"2024-03-11T21:38:32.837180+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:45.581333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021853670000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":20,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.8","version":"555'29","reported_seq":1182,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517609+0000","last_change":"2024-03-11T21:38:01.734168+0000","last_active":"2024-03-11T21:38:37.517609+0000","last_peered":"2024-03-11T21:38:37.517609+0000","last_clean":"2024-03-11T21:38:37.517609+0000","last_became_active":"2024-03-11T21:29:32.605455+0000","last_became_peered":"2024-03-11T21:29:32.605455+0000","last_unstale":"2024-03-11T21:38:37.517609+0000","last_undegraded":"2024-03-11T21:38:37.517609+0000","last_fullsized":"2024-03-11T21:38:37.517609+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'29","last_scrub_stamp":"2024-03-11T21:37:37.613678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:37.613678+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:00.747931+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001006765,"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.b","version":"573'48","reported_seq":1192,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761182+0000","last_change":"2024-03-11T21:38:01.662915+0000","last_active":"2024-03-11T21:38:37.761182+0000","last_peered":"2024-03-11T21:38:37.761182+0000","last_clean":"2024-03-11T21:38:37.761182+0000","last_became_active":"2024-03-11T21:31:37.403616+0000","last_became_peered":"2024-03-11T21:31:37.403616+0000","last_unstale":"2024-03-11T21:38:37.761182+0000","last_undegraded":"2024-03-11T21:38:37.761182+0000","last_fullsized":"2024-03-11T21:38:37.761182+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'40","last_scrub_stamp":"2024-03-11T21:34:55.080483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:55.080483+0000","objects_scrubbed":4,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037028590000000002,"stat_sum":{"num_bytes":2904,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":27,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.a","version":"627'77","reported_seq":1252,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750189+0000","last_change":"2024-03-11T21:38:00.979157+0000","last_active":"2024-03-11T21:38:37.750189+0000","last_peered":"2024-03-11T21:38:37.750189+0000","last_clean":"2024-03-11T21:38:37.750189+0000","last_became_active":"2024-03-11T21:31:37.379265+0000","last_became_peered":"2024-03-11T21:31:37.379265+0000","last_unstale":"2024-03-11T21:38:37.750189+0000","last_undegraded":"2024-03-11T21:38:37.750189+0000","last_fullsized":"2024-03-11T21:38:37.750189+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'75","last_scrub_stamp":"2024-03-11T21:37:54.213923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:54.213923+0000","objects_scrubbed":11,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:20.459283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00096690399999999998,"stat_sum":{"num_bytes":654,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":33,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.d","version":"573'70","reported_seq":1238,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772445+0000","last_change":"2024-03-11T21:38:00.997703+0000","last_active":"2024-03-11T21:38:37.772445+0000","last_peered":"2024-03-11T21:38:37.772445+0000","last_clean":"2024-03-11T21:38:37.772445+0000","last_became_active":"2024-03-11T21:29:32.624236+0000","last_became_peered":"2024-03-11T21:29:32.624236+0000","last_unstale":"2024-03-11T21:38:37.772445+0000","last_undegraded":"2024-03-11T21:38:37.772445+0000","last_fullsized":"2024-03-11T21:38:37.772445+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'56","last_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_deep_scrub":"541'56","last_deep_scrub_stamp":"2024-03-11T21:36:45.778396+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.778396+0000","objects_scrubbed":4,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001585202,"stat_sum":{"num_bytes":5025,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":6,"num_write":36,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.c","version":"588'50","reported_seq":1207,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753153+0000","last_change":"2024-03-11T21:38:00.980328+0000","last_active":"2024-03-11T21:38:37.753153+0000","last_peered":"2024-03-11T21:38:37.753153+0000","last_clean":"2024-03-11T21:38:37.753153+0000","last_became_active":"2024-03-11T21:29:32.636175+0000","last_became_peered":"2024-03-11T21:29:32.636175+0000","last_unstale":"2024-03-11T21:38:37.753153+0000","last_undegraded":"2024-03-11T21:38:37.753153+0000","last_fullsized":"2024-03-11T21:38:37.753153+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'36","last_scrub_stamp":"2024-03-11T21:37:30.218236+0000","last_deep_scrub":"485'36","last_deep_scrub_stamp":"2024-03-11T21:35:09.199420+0000","last_clean_scrub_stamp":"2024-03-11T21:37:30.218236+0000","objects_scrubbed":2,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:51.491843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010934359999999999,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.f","version":"573'185","reported_seq":1319,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754969+0000","last_change":"2024-03-11T21:38:00.981012+0000","last_active":"2024-03-11T21:38:37.754969+0000","last_peered":"2024-03-11T21:38:37.754969+0000","last_clean":"2024-03-11T21:38:37.754969+0000","last_became_active":"2024-03-11T21:29:32.601898+0000","last_became_peered":"2024-03-11T21:29:32.601898+0000","last_unstale":"2024-03-11T21:38:37.754969+0000","last_undegraded":"2024-03-11T21:38:37.754969+0000","last_fullsized":"2024-03-11T21:38:37.754969+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'185","last_scrub_stamp":"2024-03-11T21:37:45.224492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:45.224492+0000","objects_scrubbed":15,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:14.004056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000836803,"stat_sum":{"num_bytes":646,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":58,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.e","version":"555'111","reported_seq":1255,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.523221+0000","last_change":"2024-03-11T21:38:29.775061+0000","last_active":"2024-03-11T21:38:37.523221+0000","last_peered":"2024-03-11T21:38:37.523221+0000","last_clean":"2024-03-11T21:38:37.523221+0000","last_became_active":"2024-03-11T21:29:32.606373+0000","last_became_peered":"2024-03-11T21:29:32.606373+0000","last_unstale":"2024-03-11T21:38:37.523221+0000","last_undegraded":"2024-03-11T21:38:37.523221+0000","last_fullsized":"2024-03-11T21:38:37.523221+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'111","last_scrub_stamp":"2024-03-11T21:38:29.774933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:38:29.774933+0000","objects_scrubbed":7,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:39.439287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00068933799999999995,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":38,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.1","version":"564'21","reported_seq":1176,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.543127+0000","last_change":"2024-03-11T21:38:01.332492+0000","last_active":"2024-03-11T21:38:38.543127+0000","last_peered":"2024-03-11T21:38:38.543127+0000","last_clean":"2024-03-11T21:38:38.543127+0000","last_became_active":"2024-03-11T21:29:32.626193+0000","last_became_peered":"2024-03-11T21:29:32.626193+0000","last_unstale":"2024-03-11T21:38:38.543127+0000","last_undegraded":"2024-03-11T21:38:38.543127+0000","last_fullsized":"2024-03-11T21:38:38.543127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'18","last_scrub_stamp":"2024-03-11T21:36:11.356102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.356102+0000","objects_scrubbed":2,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0028826979999999999,"stat_sum":{"num_bytes":4001234,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":11,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.0","version":"573'64","reported_seq":1220,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.543234+0000","last_change":"2024-03-11T21:38:01.332544+0000","last_active":"2024-03-11T21:38:38.543234+0000","last_peered":"2024-03-11T21:38:38.543234+0000","last_clean":"2024-03-11T21:38:38.543234+0000","last_became_active":"2024-03-11T21:31:37.407745+0000","last_became_peered":"2024-03-11T21:31:37.407745+0000","last_unstale":"2024-03-11T21:38:38.543234+0000","last_undegraded":"2024-03-11T21:38:38.543234+0000","last_fullsized":"2024-03-11T21:38:38.543234+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'56","last_scrub_stamp":"2024-03-11T21:36:13.358869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:13.358869+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0029555279999999998,"stat_sum":{"num_bytes":1263,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":22,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.3","version":"555'144","reported_seq":1263,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762084+0000","last_change":"2024-03-11T21:38:01.664099+0000","last_active":"2024-03-11T21:38:37.762084+0000","last_peered":"2024-03-11T21:38:37.762084+0000","last_clean":"2024-03-11T21:38:37.762084+0000","last_became_active":"2024-03-11T21:29:32.649185+0000","last_became_peered":"2024-03-11T21:29:32.649185+0000","last_unstale":"2024-03-11T21:38:37.762084+0000","last_undegraded":"2024-03-11T21:38:37.762084+0000","last_fullsized":"2024-03-11T21:38:37.762084+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'120","last_scrub_stamp":"2024-03-11T21:34:43.036580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:43.036580+0000","objects_scrubbed":8,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029553679999999999,"stat_sum":{"num_bytes":125,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":17,"num_read_kb":6,"num_write":53,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.2","version":"412'22","reported_seq":1198,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750332+0000","last_change":"2024-03-11T21:38:00.979150+0000","last_active":"2024-03-11T21:38:37.750332+0000","last_peered":"2024-03-11T21:38:37.750332+0000","last_clean":"2024-03-11T21:38:37.750332+0000","last_became_active":"2024-03-11T21:31:37.377914+0000","last_became_peered":"2024-03-11T21:31:37.377914+0000","last_unstale":"2024-03-11T21:38:37.750332+0000","last_undegraded":"2024-03-11T21:38:37.750332+0000","last_fullsized":"2024-03-11T21:38:37.750332+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'22","last_scrub_stamp":"2024-03-11T21:37:52.164051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.164051+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:07.186622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00088586399999999999,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.5","version":"564'6","reported_seq":1182,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772718+0000","last_change":"2024-03-11T21:38:31.832346+0000","last_active":"2024-03-11T21:38:37.772718+0000","last_peered":"2024-03-11T21:38:37.772718+0000","last_clean":"2024-03-11T21:38:37.772718+0000","last_became_active":"2024-03-11T21:29:32.610041+0000","last_became_peered":"2024-03-11T21:29:32.610041+0000","last_unstale":"2024-03-11T21:38:37.772718+0000","last_undegraded":"2024-03-11T21:38:37.772718+0000","last_fullsized":"2024-03-11T21:38:37.772718+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'6","last_scrub_stamp":"2024-03-11T21:38:31.832243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:38:31.832243+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:39.267308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018858010000000001,"stat_sum":{"num_bytes":529,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"296'34","reported_seq":1189,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.546813+0000","last_change":"2024-03-11T21:38:01.333027+0000","last_active":"2024-03-11T21:38:38.546813+0000","last_peered":"2024-03-11T21:38:38.546813+0000","last_clean":"2024-03-11T21:38:38.546813+0000","last_became_active":"2024-03-11T21:31:37.412176+0000","last_became_peered":"2024-03-11T21:31:37.412176+0000","last_unstale":"2024-03-11T21:38:38.546813+0000","last_undegraded":"2024-03-11T21:38:38.546813+0000","last_fullsized":"2024-03-11T21:38:38.546813+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'34","last_scrub_stamp":"2024-03-11T21:36:18.317358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.317358+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025099760000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.7","version":"506'46","reported_seq":1185,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.585928+0000","last_change":"2024-03-11T21:38:01.545519+0000","last_active":"2024-03-11T21:38:37.585928+0000","last_peered":"2024-03-11T21:38:37.585928+0000","last_clean":"2024-03-11T21:38:37.585928+0000","last_became_active":"2024-03-11T21:29:32.639781+0000","last_became_peered":"2024-03-11T21:29:32.639781+0000","last_unstale":"2024-03-11T21:38:37.585928+0000","last_undegraded":"2024-03-11T21:38:37.585928+0000","last_fullsized":"2024-03-11T21:38:37.585928+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'46","last_scrub_stamp":"2024-03-11T21:36:21.706814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.706814+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00071004100000000001,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.6","version":"573'76","reported_seq":1222,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.584574+0000","last_change":"2024-03-11T21:38:01.541317+0000","last_active":"2024-03-11T21:38:37.584574+0000","last_peered":"2024-03-11T21:38:37.584574+0000","last_clean":"2024-03-11T21:38:37.584574+0000","last_became_active":"2024-03-11T21:31:37.407245+0000","last_became_peered":"2024-03-11T21:31:37.407245+0000","last_unstale":"2024-03-11T21:38:37.584574+0000","last_undegraded":"2024-03-11T21:38:37.584574+0000","last_fullsized":"2024-03-11T21:38:37.584574+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'75","last_scrub_stamp":"2024-03-11T21:36:28.741022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.741022+0000","objects_scrubbed":5,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028743649999999998,"stat_sum":{"num_bytes":1591,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.19","version":"564'18","reported_seq":1169,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.542835+0000","last_change":"2024-03-11T21:38:01.331335+0000","last_active":"2024-03-11T21:38:38.542835+0000","last_peered":"2024-03-11T21:38:38.542835+0000","last_clean":"2024-03-11T21:38:38.542835+0000","last_became_active":"2024-03-11T21:29:32.629078+0000","last_became_peered":"2024-03-11T21:29:32.629078+0000","last_unstale":"2024-03-11T21:38:38.542835+0000","last_undegraded":"2024-03-11T21:38:38.542835+0000","last_fullsized":"2024-03-11T21:38:38.542835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'17","last_scrub_stamp":"2024-03-11T21:36:08.294289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.294289+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017101200000000001,"stat_sum":{"num_bytes":1226,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.18","version":"593'118","reported_seq":1258,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.542886+0000","last_change":"2024-03-11T21:38:31.938918+0000","last_active":"2024-03-11T21:38:38.542886+0000","last_peered":"2024-03-11T21:38:38.542886+0000","last_clean":"2024-03-11T21:38:38.542886+0000","last_became_active":"2024-03-11T21:29:32.642741+0000","last_became_peered":"2024-03-11T21:29:32.642741+0000","last_unstale":"2024-03-11T21:38:38.542886+0000","last_undegraded":"2024-03-11T21:38:38.542886+0000","last_fullsized":"2024-03-11T21:38:38.542886+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"593'118","last_scrub_stamp":"2024-03-11T21:38:31.938711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:38:31.938711+0000","objects_scrubbed":11,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:33.920072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001935724,"stat_sum":{"num_bytes":109,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1b","version":"564'20","reported_seq":1062,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.534761+0000","last_change":"2024-03-11T21:38:00.994546+0000","last_active":"2024-03-11T21:38:39.534761+0000","last_peered":"2024-03-11T21:38:39.534761+0000","last_clean":"2024-03-11T21:38:39.534761+0000","last_became_active":"2024-03-11T21:31:37.406481+0000","last_became_peered":"2024-03-11T21:31:37.406481+0000","last_unstale":"2024-03-11T21:38:39.534761+0000","last_undegraded":"2024-03-11T21:38:39.534761+0000","last_fullsized":"2024-03-11T21:38:39.534761+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'17","last_scrub_stamp":"2024-03-11T21:36:19.125151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.125151+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001340033,"stat_sum":{"num_bytes":711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1a","version":"582'59","reported_seq":1207,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.544635+0000","last_change":"2024-03-11T21:38:01.332970+0000","last_active":"2024-03-11T21:38:38.544635+0000","last_peered":"2024-03-11T21:38:38.544635+0000","last_clean":"2024-03-11T21:38:38.544635+0000","last_became_active":"2024-03-11T21:29:32.654268+0000","last_became_peered":"2024-03-11T21:29:32.654268+0000","last_unstale":"2024-03-11T21:38:38.544635+0000","last_undegraded":"2024-03-11T21:38:38.544635+0000","last_fullsized":"2024-03-11T21:38:38.544635+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'56","last_scrub_stamp":"2024-03-11T21:36:17.330735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:17.330735+0000","objects_scrubbed":4,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022098040000000001,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":20,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"118.1d","version":"515'102","reported_seq":1253,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750776+0000","last_change":"2024-03-11T21:38:01.785170+0000","last_active":"2024-03-11T21:38:37.750776+0000","last_peered":"2024-03-11T21:38:37.750776+0000","last_clean":"2024-03-11T21:38:37.750776+0000","last_became_active":"2024-03-11T21:31:37.418042+0000","last_became_peered":"2024-03-11T21:31:37.418042+0000","last_unstale":"2024-03-11T21:38:37.750776+0000","last_undegraded":"2024-03-11T21:38:37.750776+0000","last_fullsized":"2024-03-11T21:38:37.750776+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'102","last_scrub_stamp":"2024-03-11T21:36:18.450840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.450840+0000","objects_scrubbed":7,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0019142129999999999,"stat_sum":{"num_bytes":75,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.1c","version":"573'76","reported_seq":1234,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.760713+0000","last_change":"2024-03-11T21:38:01.660739+0000","last_active":"2024-03-11T21:38:37.760713+0000","last_peered":"2024-03-11T21:38:37.760713+0000","last_clean":"2024-03-11T21:38:37.760713+0000","last_became_active":"2024-03-11T21:31:37.404868+0000","last_became_peered":"2024-03-11T21:31:37.404868+0000","last_unstale":"2024-03-11T21:38:37.760713+0000","last_undegraded":"2024-03-11T21:38:37.760713+0000","last_fullsized":"2024-03-11T21:38:37.760713+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'69","last_scrub_stamp":"2024-03-11T21:34:58.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:58.041773+0000","objects_scrubbed":9,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001721605,"stat_sum":{"num_bytes":1271,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.1f","version":"555'43","reported_seq":1086,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.538135+0000","last_change":"2024-03-11T21:38:00.999906+0000","last_active":"2024-03-11T21:38:39.538135+0000","last_peered":"2024-03-11T21:38:39.538135+0000","last_clean":"2024-03-11T21:38:39.538135+0000","last_became_active":"2024-03-11T21:31:37.413538+0000","last_became_peered":"2024-03-11T21:31:37.413538+0000","last_unstale":"2024-03-11T21:38:39.538135+0000","last_undegraded":"2024-03-11T21:38:39.538135+0000","last_fullsized":"2024-03-11T21:38:39.538135+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'37","last_scrub_stamp":"2024-03-11T21:36:16.054853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.054853+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001133486,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":21,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.1e","version":"593'234","reported_seq":1333,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762781+0000","last_change":"2024-03-11T21:38:01.664340+0000","last_active":"2024-03-11T21:38:37.762781+0000","last_peered":"2024-03-11T21:38:37.762781+0000","last_clean":"2024-03-11T21:38:37.762781+0000","last_became_active":"2024-03-11T21:29:32.618223+0000","last_became_peered":"2024-03-11T21:29:32.618223+0000","last_unstale":"2024-03-11T21:38:37.762781+0000","last_undegraded":"2024-03-11T21:38:37.762781+0000","last_fullsized":"2024-03-11T21:38:37.762781+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"593'234","last_scrub_stamp":"2024-03-11T21:37:45.781130+0000","last_deep_scrub":"593'234","last_deep_scrub_stamp":"2024-03-11T21:37:45.781130+0000","last_clean_scrub_stamp":"2024-03-11T21:37:45.781130+0000","objects_scrubbed":15,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:46.483940+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.002121454,"stat_sum":{"num_bytes":151,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":73,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.11","version":"573'36","reported_seq":1170,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759481+0000","last_change":"2024-03-11T21:38:01.662129+0000","last_active":"2024-03-11T21:38:37.759481+0000","last_peered":"2024-03-11T21:38:37.759481+0000","last_clean":"2024-03-11T21:38:37.759481+0000","last_became_active":"2024-03-11T21:31:37.401383+0000","last_became_peered":"2024-03-11T21:31:37.401383+0000","last_unstale":"2024-03-11T21:38:37.759481+0000","last_undegraded":"2024-03-11T21:38:37.759481+0000","last_fullsized":"2024-03-11T21:38:37.759481+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'35","last_scrub_stamp":"2024-03-11T21:35:08.144914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.144914+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030129570000000001,"stat_sum":{"num_bytes":730,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.10","version":"609'31","reported_seq":1234,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517247+0000","last_change":"2024-03-11T21:38:01.736722+0000","last_active":"2024-03-11T21:38:37.517247+0000","last_peered":"2024-03-11T21:38:37.517247+0000","last_clean":"2024-03-11T21:38:37.517247+0000","last_became_active":"2024-03-11T21:29:32.623330+0000","last_became_peered":"2024-03-11T21:29:32.623330+0000","last_unstale":"2024-03-11T21:38:37.517247+0000","last_undegraded":"2024-03-11T21:38:37.517247+0000","last_fullsized":"2024-03-11T21:38:37.517247+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'17","last_scrub_stamp":"2024-03-11T21:36:30.037348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.037348+0000","objects_scrubbed":4,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0038636690000000001,"stat_sum":{"num_bytes":4194350,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":8201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.13","version":"471'89","reported_seq":1224,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517523+0000","last_change":"2024-03-11T21:38:01.734568+0000","last_active":"2024-03-11T21:38:37.517523+0000","last_peered":"2024-03-11T21:38:37.517523+0000","last_clean":"2024-03-11T21:38:37.517523+0000","last_became_active":"2024-03-11T21:29:32.636642+0000","last_became_peered":"2024-03-11T21:29:32.636642+0000","last_unstale":"2024-03-11T21:38:37.517523+0000","last_undegraded":"2024-03-11T21:38:37.517523+0000","last_fullsized":"2024-03-11T21:38:37.517523+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'89","last_scrub_stamp":"2024-03-11T21:37:26.671049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:26.671049+0000","objects_scrubbed":8,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:46.406496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086092199999999997,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.12","version":"412'65","reported_seq":1200,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759511+0000","last_change":"2024-03-11T21:38:01.663901+0000","last_active":"2024-03-11T21:38:37.759511+0000","last_peered":"2024-03-11T21:38:37.759511+0000","last_clean":"2024-03-11T21:38:37.759511+0000","last_became_active":"2024-03-11T21:29:32.636184+0000","last_became_peered":"2024-03-11T21:29:32.636184+0000","last_unstale":"2024-03-11T21:38:37.759511+0000","last_undegraded":"2024-03-11T21:38:37.759511+0000","last_fullsized":"2024-03-11T21:38:37.759511+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'65","last_scrub_stamp":"2024-03-11T21:35:07.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.139123+0000","objects_scrubbed":8,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033599039999999999,"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"118.15","version":"564'75","reported_seq":1235,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772165+0000","last_change":"2024-03-11T21:38:00.997612+0000","last_active":"2024-03-11T21:38:37.772165+0000","last_peered":"2024-03-11T21:38:37.772165+0000","last_clean":"2024-03-11T21:38:37.772165+0000","last_became_active":"2024-03-11T21:29:32.609648+0000","last_became_peered":"2024-03-11T21:29:32.609648+0000","last_unstale":"2024-03-11T21:38:37.772165+0000","last_undegraded":"2024-03-11T21:38:37.772165+0000","last_fullsized":"2024-03-11T21:38:37.772165+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-11T21:36:44.814352+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-11T21:35:04.446157+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.814352+0000","objects_scrubbed":4,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00169505,"stat_sum":{"num_bytes":1586,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.14","version":"495'67","reported_seq":1228,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753011+0000","last_change":"2024-03-11T21:38:00.980639+0000","last_active":"2024-03-11T21:38:37.753011+0000","last_peered":"2024-03-11T21:38:37.753011+0000","last_clean":"2024-03-11T21:38:37.753011+0000","last_became_active":"2024-03-11T21:31:37.378072+0000","last_became_peered":"2024-03-11T21:31:37.378072+0000","last_unstale":"2024-03-11T21:38:37.753011+0000","last_undegraded":"2024-03-11T21:38:37.753011+0000","last_fullsized":"2024-03-11T21:38:37.753011+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'67","last_scrub_stamp":"2024-03-11T21:37:56.219900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:56.219900+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:14.323046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013946379999999999,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.17","version":"573'59","reported_seq":1224,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.771666+0000","last_change":"2024-03-11T21:38:01.001471+0000","last_active":"2024-03-11T21:38:37.771666+0000","last_peered":"2024-03-11T21:38:37.771666+0000","last_clean":"2024-03-11T21:38:37.771666+0000","last_became_active":"2024-03-11T21:29:32.646878+0000","last_became_peered":"2024-03-11T21:29:32.646878+0000","last_unstale":"2024-03-11T21:38:37.771666+0000","last_undegraded":"2024-03-11T21:38:37.771666+0000","last_fullsized":"2024-03-11T21:38:37.771666+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'59","last_scrub_stamp":"2024-03-11T21:37:40.805255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:37:40.805255+0000","objects_scrubbed":6,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:44.259938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018778239999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.16","version":"616'21","reported_seq":1172,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762802+0000","last_change":"2024-03-11T21:38:01.663262+0000","last_active":"2024-03-11T21:38:37.762802+0000","last_peered":"2024-03-11T21:38:37.762802+0000","last_clean":"2024-03-11T21:38:37.762802+0000","last_became_active":"2024-03-11T21:31:37.401375+0000","last_became_peered":"2024-03-11T21:31:37.401375+0000","last_unstale":"2024-03-11T21:38:37.762802+0000","last_undegraded":"2024-03-11T21:38:37.762802+0000","last_fullsized":"2024-03-11T21:38:37.762802+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'6","last_scrub_stamp":"2024-03-11T21:34:50.112525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.236818+0000","last_clean_scrub_stamp":"2024-03-11T21:34:50.112525+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003413887,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":1004,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.520366+0000","last_change":"2024-03-11T21:36:09.680554+0000","last_active":"2024-03-11T21:38:37.520366+0000","last_peered":"2024-03-11T21:38:37.520366+0000","last_clean":"2024-03-11T21:38:37.520366+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:38:37.520366+0000","last_undegraded":"2024-03-11T21:38:37.520366+0000","last_fullsized":"2024-03-11T21:38:37.520366+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.680346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.680346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1014,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750474+0000","last_change":"2024-03-11T21:37:31.780203+0000","last_active":"2024-03-11T21:38:37.750474+0000","last_peered":"2024-03-11T21:38:37.750474+0000","last_clean":"2024-03-11T21:38:37.750474+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:38:37.750474+0000","last_undegraded":"2024-03-11T21:38:37.750474+0000","last_fullsized":"2024-03-11T21:38:37.750474+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:31.780100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:37:31.780100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:54.404633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1001,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.516411+0000","last_change":"2024-03-11T21:36:34.020742+0000","last_active":"2024-03-11T21:38:37.516411+0000","last_peered":"2024-03-11T21:38:37.516411+0000","last_clean":"2024-03-11T21:38:37.516411+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:38:37.516411+0000","last_undegraded":"2024-03-11T21:38:37.516411+0000","last_fullsized":"2024-03-11T21:38:37.516411+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.020607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.020607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":994,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576879+0000","last_change":"2024-03-11T21:36:14.628566+0000","last_active":"2024-03-11T21:38:37.576879+0000","last_peered":"2024-03-11T21:38:37.576879+0000","last_clean":"2024-03-11T21:38:37.576879+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:38:37.576879+0000","last_undegraded":"2024-03-11T21:38:37.576879+0000","last_fullsized":"2024-03-11T21:38:37.576879+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":912,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.535767+0000","last_change":"2024-03-11T21:36:43.079499+0000","last_active":"2024-03-11T21:38:39.535767+0000","last_peered":"2024-03-11T21:38:39.535767+0000","last_clean":"2024-03-11T21:38:39.535767+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:38:39.535767+0000","last_undegraded":"2024-03-11T21:38:39.535767+0000","last_fullsized":"2024-03-11T21:38:39.535767+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.079413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.079413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1013,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.751247+0000","last_change":"2024-03-11T21:37:30.734855+0000","last_active":"2024-03-11T21:38:37.751247+0000","last_peered":"2024-03-11T21:38:37.751247+0000","last_clean":"2024-03-11T21:38:37.751247+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:38:37.751247+0000","last_undegraded":"2024-03-11T21:38:37.751247+0000","last_fullsized":"2024-03-11T21:38:37.751247+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:30.734479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:30.734479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:47.636365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1014,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754901+0000","last_change":"2024-03-11T21:37:43.718436+0000","last_active":"2024-03-11T21:38:37.754901+0000","last_peered":"2024-03-11T21:38:37.754901+0000","last_clean":"2024-03-11T21:38:37.754901+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:38:37.754901+0000","last_undegraded":"2024-03-11T21:38:37.754901+0000","last_fullsized":"2024-03-11T21:38:37.754901+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:43.718293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.319457+0000","last_clean_scrub_stamp":"2024-03-11T21:37:43.718293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:12.731445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1025,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.771900+0000","last_change":"2024-03-11T21:37:25.791320+0000","last_active":"2024-03-11T21:38:37.771900+0000","last_peered":"2024-03-11T21:38:37.771900+0000","last_clean":"2024-03-11T21:38:37.771900+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:38:37.771900+0000","last_undegraded":"2024-03-11T21:38:37.771900+0000","last_fullsized":"2024-03-11T21:38:37.771900+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:25.791048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:25.791048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750715+0000","last_change":"2024-03-11T21:36:36.107906+0000","last_active":"2024-03-11T21:38:37.750715+0000","last_peered":"2024-03-11T21:38:37.750715+0000","last_clean":"2024-03-11T21:38:37.750715+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:38:37.750715+0000","last_undegraded":"2024-03-11T21:38:37.750715+0000","last_fullsized":"2024-03-11T21:38:37.750715+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.107801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.107801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1003,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.519323+0000","last_change":"2024-03-11T21:37:46.916669+0000","last_active":"2024-03-11T21:38:37.519323+0000","last_peered":"2024-03-11T21:38:37.519323+0000","last_clean":"2024-03-11T21:38:37.519323+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:38:37.519323+0000","last_undegraded":"2024-03-11T21:38:37.519323+0000","last_fullsized":"2024-03-11T21:38:37.519323+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:46.916535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:46.916535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:01.285317+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1012,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.545005+0000","last_change":"2024-03-11T21:36:43.145228+0000","last_active":"2024-03-11T21:38:38.545005+0000","last_peered":"2024-03-11T21:38:38.545005+0000","last_clean":"2024-03-11T21:38:38.545005+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:38:38.545005+0000","last_undegraded":"2024-03-11T21:38:38.545005+0000","last_fullsized":"2024-03-11T21:38:38.545005+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.145137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.145137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":989,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762423+0000","last_change":"2024-03-11T21:32:25.434869+0000","last_active":"2024-03-11T21:38:37.762423+0000","last_peered":"2024-03-11T21:38:37.762423+0000","last_clean":"2024-03-11T21:38:37.762423+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:38:37.762423+0000","last_undegraded":"2024-03-11T21:38:37.762423+0000","last_fullsized":"2024-03-11T21:38:37.762423+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:25.434777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:25.434777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1012,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.546542+0000","last_change":"2024-03-11T21:37:32.977458+0000","last_active":"2024-03-11T21:38:38.546542+0000","last_peered":"2024-03-11T21:38:38.546542+0000","last_clean":"2024-03-11T21:38:38.546542+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:38:38.546542+0000","last_undegraded":"2024-03-11T21:38:38.546542+0000","last_fullsized":"2024-03-11T21:38:38.546542+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:32.977361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:32.977361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:55.348126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.757416+0000","last_change":"2024-03-11T21:37:33.219238+0000","last_active":"2024-03-11T21:38:37.757416+0000","last_peered":"2024-03-11T21:38:37.757416+0000","last_clean":"2024-03-11T21:38:37.757416+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:38:37.757416+0000","last_undegraded":"2024-03-11T21:38:37.757416+0000","last_fullsized":"2024-03-11T21:38:37.757416+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:33.219152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:37:33.219152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1025,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.771796+0000","last_change":"2024-03-11T21:37:55.911576+0000","last_active":"2024-03-11T21:38:37.771796+0000","last_peered":"2024-03-11T21:38:37.771796+0000","last_clean":"2024-03-11T21:38:37.771796+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:38:37.771796+0000","last_undegraded":"2024-03-11T21:38:37.771796+0000","last_fullsized":"2024-03-11T21:38:37.771796+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:55.911423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:55.911423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:25.045975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":416,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.537825+0000","last_change":"2024-03-11T21:37:58.943366+0000","last_active":"2024-03-11T21:38:39.537825+0000","last_peered":"2024-03-11T21:38:39.537825+0000","last_clean":"2024-03-11T21:38:39.537825+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:38:39.537825+0000","last_undegraded":"2024-03-11T21:38:39.537825+0000","last_fullsized":"2024-03-11T21:38:39.537825+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:58.943267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:58.943267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:26.324712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1014,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.748751+0000","last_change":"2024-03-11T21:37:28.734794+0000","last_active":"2024-03-11T21:38:37.748751+0000","last_peered":"2024-03-11T21:38:37.748751+0000","last_clean":"2024-03-11T21:38:37.748751+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:38:37.748751+0000","last_undegraded":"2024-03-11T21:38:37.748751+0000","last_fullsized":"2024-03-11T21:38:37.748751+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:28.734660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:28.734660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:50.146330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":995,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759276+0000","last_change":"2024-03-11T21:33:59.484063+0000","last_active":"2024-03-11T21:38:37.759276+0000","last_peered":"2024-03-11T21:38:37.759276+0000","last_clean":"2024-03-11T21:38:37.759276+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:38:37.759276+0000","last_undegraded":"2024-03-11T21:38:37.759276+0000","last_fullsized":"2024-03-11T21:38:37.759276+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.563498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.563498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":995,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.575359+0000","last_change":"2024-03-11T21:36:26.767684+0000","last_active":"2024-03-11T21:38:37.575359+0000","last_peered":"2024-03-11T21:38:37.575359+0000","last_clean":"2024-03-11T21:38:37.575359+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:38:37.575359+0000","last_undegraded":"2024-03-11T21:38:37.575359+0000","last_fullsized":"2024-03-11T21:38:37.575359+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.767129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.767129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753207+0000","last_change":"2024-03-11T21:37:26.212560+0000","last_active":"2024-03-11T21:38:37.753207+0000","last_peered":"2024-03-11T21:38:37.753207+0000","last_clean":"2024-03-11T21:38:37.753207+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:38:37.753207+0000","last_undegraded":"2024-03-11T21:38:37.753207+0000","last_fullsized":"2024-03-11T21:38:37.753207+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:26.212282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:26.212282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":881,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.535981+0000","last_change":"2024-03-11T21:35:11.130108+0000","last_active":"2024-03-11T21:38:39.535981+0000","last_peered":"2024-03-11T21:38:39.535981+0000","last_clean":"2024-03-11T21:38:39.535981+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:38:39.535981+0000","last_undegraded":"2024-03-11T21:38:39.535981+0000","last_fullsized":"2024-03-11T21:38:39.535981+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.129979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.129979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":995,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.583094+0000","last_change":"2024-03-11T21:34:59.525103+0000","last_active":"2024-03-11T21:38:37.583094+0000","last_peered":"2024-03-11T21:38:37.583094+0000","last_clean":"2024-03-11T21:38:37.583094+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:38:37.583094+0000","last_undegraded":"2024-03-11T21:38:37.583094+0000","last_fullsized":"2024-03-11T21:38:37.583094+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:59.524800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:34:59.524800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1003,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.584758+0000","last_change":"2024-03-11T21:36:04.661608+0000","last_active":"2024-03-11T21:38:37.584758+0000","last_peered":"2024-03-11T21:38:37.584758+0000","last_clean":"2024-03-11T21:38:37.584758+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:38:37.584758+0000","last_undegraded":"2024-03-11T21:38:37.584758+0000","last_fullsized":"2024-03-11T21:38:37.584758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.661495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.661495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1012,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.548195+0000","last_change":"2024-03-11T21:37:50.875699+0000","last_active":"2024-03-11T21:38:38.548195+0000","last_peered":"2024-03-11T21:38:38.548195+0000","last_clean":"2024-03-11T21:38:38.548195+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:38:38.548195+0000","last_undegraded":"2024-03-11T21:38:38.548195+0000","last_fullsized":"2024-03-11T21:38:38.548195+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:50.875603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:50.875603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:52.890533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1002,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.519079+0000","last_change":"2024-03-11T21:36:33.044584+0000","last_active":"2024-03-11T21:38:37.519079+0000","last_peered":"2024-03-11T21:38:37.519079+0000","last_clean":"2024-03-11T21:38:37.519079+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:38:37.519079+0000","last_undegraded":"2024-03-11T21:38:37.519079+0000","last_fullsized":"2024-03-11T21:38:37.519079+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:33.044471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.044471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":989,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759475+0000","last_change":"2024-03-11T21:33:25.592066+0000","last_active":"2024-03-11T21:38:37.759475+0000","last_peered":"2024-03-11T21:38:37.759475+0000","last_clean":"2024-03-11T21:38:37.759475+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:38:37.759475+0000","last_undegraded":"2024-03-11T21:38:37.759475+0000","last_fullsized":"2024-03-11T21:38:37.759475+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:25.591948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:33:25.591948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":417,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.552356+0000","last_change":"2024-03-11T21:37:37.071365+0000","last_active":"2024-03-11T21:38:39.552356+0000","last_peered":"2024-03-11T21:38:39.552356+0000","last_clean":"2024-03-11T21:38:39.552356+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:38:39.552356+0000","last_undegraded":"2024-03-11T21:38:39.552356+0000","last_fullsized":"2024-03-11T21:38:39.552356+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:37.071174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:37.071174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:03.172520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753076+0000","last_change":"2024-03-11T21:37:46.192519+0000","last_active":"2024-03-11T21:38:37.753076+0000","last_peered":"2024-03-11T21:38:37.753076+0000","last_clean":"2024-03-11T21:38:37.753076+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:38:37.753076+0000","last_undegraded":"2024-03-11T21:38:37.753076+0000","last_fullsized":"2024-03-11T21:38:37.753076+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:46.192253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:46.192253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:51.501543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1012,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.547048+0000","last_change":"2024-03-11T21:37:49.833347+0000","last_active":"2024-03-11T21:38:38.547048+0000","last_peered":"2024-03-11T21:38:38.547048+0000","last_clean":"2024-03-11T21:38:38.547048+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:38:38.547048+0000","last_undegraded":"2024-03-11T21:38:38.547048+0000","last_fullsized":"2024-03-11T21:38:38.547048+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:49.833168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:37:49.833168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:58.155046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":989,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761828+0000","last_change":"2024-03-11T21:32:19.435695+0000","last_active":"2024-03-11T21:38:37.761828+0000","last_peered":"2024-03-11T21:38:37.761828+0000","last_clean":"2024-03-11T21:38:37.761828+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:38:37.761828+0000","last_undegraded":"2024-03-11T21:38:37.761828+0000","last_fullsized":"2024-03-11T21:38:37.761828+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.435397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1017,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.771813+0000","last_change":"2024-03-11T21:36:35.782923+0000","last_active":"2024-03-11T21:38:37.771813+0000","last_peered":"2024-03-11T21:38:37.771813+0000","last_clean":"2024-03-11T21:38:37.771813+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:38:37.771813+0000","last_undegraded":"2024-03-11T21:38:37.771813+0000","last_fullsized":"2024-03-11T21:38:37.771813+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:36:35.782796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":881,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.538355+0000","last_change":"2024-03-11T21:35:01.154707+0000","last_active":"2024-03-11T21:38:39.538355+0000","last_peered":"2024-03-11T21:38:39.538355+0000","last_clean":"2024-03-11T21:38:39.538355+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:38:39.538355+0000","last_undegraded":"2024-03-11T21:38:39.538355+0000","last_fullsized":"2024-03-11T21:38:39.538355+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.154502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.154502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1018,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.549944+0000","last_change":"2024-03-11T21:36:45.220132+0000","last_active":"2024-03-11T21:38:38.549944+0000","last_peered":"2024-03-11T21:38:38.549944+0000","last_clean":"2024-03-11T21:38:38.549944+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:38:38.549944+0000","last_undegraded":"2024-03-11T21:38:38.549944+0000","last_fullsized":"2024-03-11T21:38:38.549944+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.219989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.219989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1346,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759875+0000","last_change":"2024-03-11T21:37:29.738596+0000","last_active":"2024-03-11T21:38:37.759875+0000","last_peered":"2024-03-11T21:38:37.759875+0000","last_clean":"2024-03-11T21:38:37.759875+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:38:37.759875+0000","last_undegraded":"2024-03-11T21:38:37.759875+0000","last_fullsized":"2024-03-11T21:38:37.759875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:37:29.738486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:29.738486+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:47.324082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1195,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.764070+0000","last_change":"2024-03-11T21:37:28.733714+0000","last_active":"2024-03-11T21:38:37.764070+0000","last_peered":"2024-03-11T21:38:37.764070+0000","last_clean":"2024-03-11T21:38:37.764070+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:38:37.764070+0000","last_undegraded":"2024-03-11T21:38:37.764070+0000","last_fullsized":"2024-03-11T21:38:37.764070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:37:28.733562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:28.733562+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:50.892062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1192,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772287+0000","last_change":"2024-03-11T21:38:35.856758+0000","last_active":"2024-03-11T21:38:37.772287+0000","last_peered":"2024-03-11T21:38:37.772287+0000","last_clean":"2024-03-11T21:38:37.772287+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:38:37.772287+0000","last_undegraded":"2024-03-11T21:38:37.772287+0000","last_fullsized":"2024-03-11T21:38:37.772287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:35.856558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:38:35.856558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:00.159413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1159,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517249+0000","last_change":"2024-03-11T21:38:31.702800+0000","last_active":"2024-03-11T21:38:37.517249+0000","last_peered":"2024-03-11T21:38:37.517249+0000","last_clean":"2024-03-11T21:38:37.517249+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:38:37.517249+0000","last_undegraded":"2024-03-11T21:38:37.517249+0000","last_fullsized":"2024-03-11T21:38:37.517249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:31.702640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:38:31.702640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:51.069487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1017,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.524047+0000","last_change":"2024-03-11T21:36:37.014962+0000","last_active":"2024-03-11T21:38:37.524047+0000","last_peered":"2024-03-11T21:38:37.524047+0000","last_clean":"2024-03-11T21:38:37.524047+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:38:37.524047+0000","last_undegraded":"2024-03-11T21:38:37.524047+0000","last_fullsized":"2024-03-11T21:38:37.524047+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.014751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.014751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1196,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.520527+0000","last_change":"2024-03-11T21:37:56.714159+0000","last_active":"2024-03-11T21:38:37.520527+0000","last_peered":"2024-03-11T21:38:37.520527+0000","last_clean":"2024-03-11T21:38:37.520527+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:38:37.520527+0000","last_undegraded":"2024-03-11T21:38:37.520527+0000","last_fullsized":"2024-03-11T21:38:37.520527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:37:56.714020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:56.714020+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:21.783105+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1071,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.537402+0000","last_change":"2024-03-11T21:37:41.047070+0000","last_active":"2024-03-11T21:38:39.537402+0000","last_peered":"2024-03-11T21:38:39.537402+0000","last_clean":"2024-03-11T21:38:39.537402+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:38:39.537402+0000","last_undegraded":"2024-03-11T21:38:39.537402+0000","last_fullsized":"2024-03-11T21:38:39.537402+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:41.046966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:37:41.046966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:07.569499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1185,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.768310+0000","last_change":"2024-03-11T21:37:48.769942+0000","last_active":"2024-03-11T21:38:37.768310+0000","last_peered":"2024-03-11T21:38:37.768310+0000","last_clean":"2024-03-11T21:38:37.768310+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:38:37.768310+0000","last_undegraded":"2024-03-11T21:38:37.768310+0000","last_fullsized":"2024-03-11T21:38:37.768310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:48.769834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:48.769834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:00.111107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1161,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.544607+0000","last_change":"2024-03-11T21:35:02.012107+0000","last_active":"2024-03-11T21:38:38.544607+0000","last_peered":"2024-03-11T21:38:38.544607+0000","last_clean":"2024-03-11T21:38:38.544607+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:38:38.544607+0000","last_undegraded":"2024-03-11T21:38:38.544607+0000","last_fullsized":"2024-03-11T21:38:38.544607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:02.011982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:35:02.011982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1008,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.524175+0000","last_change":"2024-03-11T21:36:08.644929+0000","last_active":"2024-03-11T21:38:37.524175+0000","last_peered":"2024-03-11T21:38:37.524175+0000","last_clean":"2024-03-11T21:38:37.524175+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:38:37.524175+0000","last_undegraded":"2024-03-11T21:38:37.524175+0000","last_fullsized":"2024-03-11T21:38:37.524175+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.644692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.644692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1209,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762380+0000","last_change":"2024-03-11T21:33:13.716731+0000","last_active":"2024-03-11T21:38:37.762380+0000","last_peered":"2024-03-11T21:38:37.762380+0000","last_clean":"2024-03-11T21:38:37.762380+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:38:37.762380+0000","last_undegraded":"2024-03-11T21:38:37.762380+0000","last_fullsized":"2024-03-11T21:38:37.762380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:33:13.716497+0000","last_deep_scrub":"106'25","last_deep_scrub_stamp":"2024-03-11T21:24:41.172690+0000","last_clean_scrub_stamp":"2024-03-11T21:33:13.716497+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1192,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.767732+0000","last_change":"2024-03-11T21:37:39.820563+0000","last_active":"2024-03-11T21:38:37.767732+0000","last_peered":"2024-03-11T21:38:37.767732+0000","last_clean":"2024-03-11T21:38:37.767732+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:38:37.767732+0000","last_undegraded":"2024-03-11T21:38:37.767732+0000","last_fullsized":"2024-03-11T21:38:37.767732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:39.820428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:37:39.820428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:40.577300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1176,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.767767+0000","last_change":"2024-03-11T21:36:43.849743+0000","last_active":"2024-03-11T21:38:37.767767+0000","last_peered":"2024-03-11T21:38:37.767767+0000","last_clean":"2024-03-11T21:38:37.767767+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:38:37.767767+0000","last_undegraded":"2024-03-11T21:38:37.767767+0000","last_fullsized":"2024-03-11T21:38:37.767767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:43.849578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:36:43.849578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1145,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.758241+0000","last_change":"2024-03-11T21:35:11.139678+0000","last_active":"2024-03-11T21:38:37.758241+0000","last_peered":"2024-03-11T21:38:37.758241+0000","last_clean":"2024-03-11T21:38:37.758241+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:38:37.758241+0000","last_undegraded":"2024-03-11T21:38:37.758241+0000","last_fullsized":"2024-03-11T21:38:37.758241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.139473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.139473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":395,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.516862+0000","last_change":"2024-03-11T21:37:52.678051+0000","last_active":"2024-03-11T21:38:37.516862+0000","last_peered":"2024-03-11T21:38:37.516862+0000","last_clean":"2024-03-11T21:38:37.516862+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:38:37.516862+0000","last_undegraded":"2024-03-11T21:38:37.516862+0000","last_fullsized":"2024-03-11T21:38:37.516862+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:52.677826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.677826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:15.262090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":890,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.537346+0000","last_change":"2024-03-11T21:36:45.055123+0000","last_active":"2024-03-11T21:38:39.537346+0000","last_peered":"2024-03-11T21:38:39.537346+0000","last_clean":"2024-03-11T21:38:39.537346+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:38:39.537346+0000","last_undegraded":"2024-03-11T21:38:39.537346+0000","last_fullsized":"2024-03-11T21:38:39.537346+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.054820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1251,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754908+0000","last_change":"2024-03-11T21:37:55.841463+0000","last_active":"2024-03-11T21:38:37.754908+0000","last_peered":"2024-03-11T21:38:37.754908+0000","last_clean":"2024-03-11T21:38:37.754908+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:38:37.754908+0000","last_undegraded":"2024-03-11T21:38:37.754908+0000","last_fullsized":"2024-03-11T21:38:37.754908+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:37:55.841289+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:37:55.841289+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:12.053757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1164,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761793+0000","last_change":"2024-03-11T21:37:32.739162+0000","last_active":"2024-03-11T21:38:37.761793+0000","last_peered":"2024-03-11T21:38:37.761793+0000","last_clean":"2024-03-11T21:38:37.761793+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:38:37.761793+0000","last_undegraded":"2024-03-11T21:38:37.761793+0000","last_fullsized":"2024-03-11T21:38:37.761793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:32.738960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:32.738960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:55.167280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1069,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.535830+0000","last_change":"2024-03-11T21:36:20.116589+0000","last_active":"2024-03-11T21:38:39.535830+0000","last_peered":"2024-03-11T21:38:39.535830+0000","last_clean":"2024-03-11T21:38:39.535830+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:38:39.535830+0000","last_undegraded":"2024-03-11T21:38:39.535830+0000","last_fullsized":"2024-03-11T21:38:39.535830+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:20.116311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.116311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1180,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772337+0000","last_change":"2024-03-11T21:37:41.787057+0000","last_active":"2024-03-11T21:38:37.772337+0000","last_peered":"2024-03-11T21:38:37.772337+0000","last_clean":"2024-03-11T21:38:37.772337+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:38:37.772337+0000","last_undegraded":"2024-03-11T21:38:37.772337+0000","last_fullsized":"2024-03-11T21:38:37.772337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:41.786848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:37:41.786848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:44.183024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1446,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.760075+0000","last_change":"2024-03-11T21:37:52.720219+0000","last_active":"2024-03-11T21:38:37.760075+0000","last_peered":"2024-03-11T21:38:37.760075+0000","last_clean":"2024-03-11T21:38:37.760075+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:38:37.760075+0000","last_undegraded":"2024-03-11T21:38:37.760075+0000","last_fullsized":"2024-03-11T21:38:37.760075+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:52.719912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.719912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:56.357814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1018,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.519560+0000","last_change":"2024-03-11T21:36:11.656705+0000","last_active":"2024-03-11T21:38:37.519560+0000","last_peered":"2024-03-11T21:38:37.519560+0000","last_clean":"2024-03-11T21:38:37.519560+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:38:37.519560+0000","last_undegraded":"2024-03-11T21:38:37.519560+0000","last_fullsized":"2024-03-11T21:38:37.519560+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:36:11.656610+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.656610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1186,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.580891+0000","last_change":"2024-03-11T21:35:01.534298+0000","last_active":"2024-03-11T21:38:37.580891+0000","last_peered":"2024-03-11T21:38:37.580891+0000","last_clean":"2024-03-11T21:38:37.580891+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:38:37.580891+0000","last_undegraded":"2024-03-11T21:38:37.580891+0000","last_fullsized":"2024-03-11T21:38:37.580891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:35:01.534131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.534131+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1037,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.552270+0000","last_change":"2024-03-11T21:37:30.124870+0000","last_active":"2024-03-11T21:38:39.552270+0000","last_peered":"2024-03-11T21:38:39.552270+0000","last_clean":"2024-03-11T21:38:39.552270+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:38:39.552270+0000","last_undegraded":"2024-03-11T21:38:39.552270+0000","last_fullsized":"2024-03-11T21:38:39.552270+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:30.124767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:37:30.124767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1382,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.773303+0000","last_change":"2024-03-11T21:37:31.809855+0000","last_active":"2024-03-11T21:38:37.773303+0000","last_peered":"2024-03-11T21:38:37.773303+0000","last_clean":"2024-03-11T21:38:37.773303+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:38:37.773303+0000","last_undegraded":"2024-03-11T21:38:37.773303+0000","last_fullsized":"2024-03-11T21:38:37.773303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:37:31.809639+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:37:31.809639+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:56.030005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1175,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754735+0000","last_change":"2024-03-11T21:36:45.224370+0000","last_active":"2024-03-11T21:38:37.754735+0000","last_peered":"2024-03-11T21:38:37.754735+0000","last_clean":"2024-03-11T21:38:37.754735+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:38:37.754735+0000","last_undegraded":"2024-03-11T21:38:37.754735+0000","last_fullsized":"2024-03-11T21:38:37.754735+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:36:45.224251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":395,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.515894+0000","last_change":"2024-03-11T21:37:53.725035+0000","last_active":"2024-03-11T21:38:37.515894+0000","last_peered":"2024-03-11T21:38:37.515894+0000","last_clean":"2024-03-11T21:38:37.515894+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:38:37.515894+0000","last_undegraded":"2024-03-11T21:38:37.515894+0000","last_fullsized":"2024-03-11T21:38:37.515894+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:53.724768+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:37:53.724768+0000","last_clean_scrub_stamp":"2024-03-11T21:37:53.724768+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:02.846048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1021,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576069+0000","last_change":"2024-03-11T21:37:28.488638+0000","last_active":"2024-03-11T21:38:37.576069+0000","last_peered":"2024-03-11T21:38:37.576069+0000","last_clean":"2024-03-11T21:38:37.576069+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:38:37.576069+0000","last_undegraded":"2024-03-11T21:38:37.576069+0000","last_fullsized":"2024-03-11T21:38:37.576069+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:28.488511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:28.488511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1214,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576174+0000","last_change":"2024-03-11T21:36:31.677071+0000","last_active":"2024-03-11T21:38:37.576174+0000","last_peered":"2024-03-11T21:38:37.576174+0000","last_clean":"2024-03-11T21:38:37.576174+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:38:37.576174+0000","last_undegraded":"2024-03-11T21:38:37.576174+0000","last_fullsized":"2024-03-11T21:38:37.576174+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:36:31.676948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.676948+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1065,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.534219+0000","last_change":"2024-03-11T21:37:27.115092+0000","last_active":"2024-03-11T21:38:39.534219+0000","last_peered":"2024-03-11T21:38:39.534219+0000","last_clean":"2024-03-11T21:38:39.534219+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:38:39.534219+0000","last_undegraded":"2024-03-11T21:38:39.534219+0000","last_fullsized":"2024-03-11T21:38:39.534219+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:27.114950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:27.114950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1367,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754469+0000","last_change":"2024-03-11T21:37:34.271852+0000","last_active":"2024-03-11T21:38:37.754469+0000","last_peered":"2024-03-11T21:38:37.754469+0000","last_clean":"2024-03-11T21:38:37.754469+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:38:37.754469+0000","last_undegraded":"2024-03-11T21:38:37.754469+0000","last_fullsized":"2024-03-11T21:38:37.754469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:37:34.271636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:34.271636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:43.748555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1167,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.523848+0000","last_change":"2024-03-11T21:37:42.679560+0000","last_active":"2024-03-11T21:38:37.523848+0000","last_peered":"2024-03-11T21:38:37.523848+0000","last_clean":"2024-03-11T21:38:37.523848+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:38:37.523848+0000","last_undegraded":"2024-03-11T21:38:37.523848+0000","last_fullsized":"2024-03-11T21:38:37.523848+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:42.679452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:37:42.679452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:05.558292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1003,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761384+0000","last_change":"2024-03-11T21:31:56.391104+0000","last_active":"2024-03-11T21:38:37.761384+0000","last_peered":"2024-03-11T21:38:37.761384+0000","last_clean":"2024-03-11T21:38:37.761384+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:38:37.761384+0000","last_undegraded":"2024-03-11T21:38:37.761384+0000","last_fullsized":"2024-03-11T21:38:37.761384+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.390965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1212,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750039+0000","last_change":"2024-03-11T21:37:35.232729+0000","last_active":"2024-03-11T21:38:37.750039+0000","last_peered":"2024-03-11T21:38:37.750039+0000","last_clean":"2024-03-11T21:38:37.750039+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:38:37.750039+0000","last_undegraded":"2024-03-11T21:38:37.750039+0000","last_fullsized":"2024-03-11T21:38:37.750039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:37:35.232342+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:37:35.232342+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:58.984716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.758677+0000","last_change":"2024-03-11T21:33:36.592449+0000","last_active":"2024-03-11T21:38:37.758677+0000","last_peered":"2024-03-11T21:38:37.758677+0000","last_clean":"2024-03-11T21:38:37.758677+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:38:37.758677+0000","last_undegraded":"2024-03-11T21:38:37.758677+0000","last_fullsized":"2024-03-11T21:38:37.758677+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:36.592101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:33:36.592101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1154,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.525312+0000","last_change":"2024-03-11T21:35:12.702568+0000","last_active":"2024-03-11T21:38:37.525312+0000","last_peered":"2024-03-11T21:38:37.525312+0000","last_clean":"2024-03-11T21:38:37.525312+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:38:37.525312+0000","last_undegraded":"2024-03-11T21:38:37.525312+0000","last_fullsized":"2024-03-11T21:38:37.525312+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.702301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.702301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1153,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.580833+0000","last_change":"2024-03-11T21:37:25.343129+0000","last_active":"2024-03-11T21:38:37.580833+0000","last_peered":"2024-03-11T21:38:37.580833+0000","last_clean":"2024-03-11T21:38:37.580833+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:38:37.580833+0000","last_undegraded":"2024-03-11T21:38:37.580833+0000","last_fullsized":"2024-03-11T21:38:37.580833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:25.342917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:37:25.342917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:47.294497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":396,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.581545+0000","last_change":"2024-03-11T21:37:52.700064+0000","last_active":"2024-03-11T21:38:37.581545+0000","last_peered":"2024-03-11T21:38:37.581545+0000","last_clean":"2024-03-11T21:38:37.581545+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:38:37.581545+0000","last_undegraded":"2024-03-11T21:38:37.581545+0000","last_fullsized":"2024-03-11T21:38:37.581545+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.699591+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:15.467811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1004,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.581667+0000","last_change":"2024-03-11T21:36:08.684202+0000","last_active":"2024-03-11T21:38:37.581667+0000","last_peered":"2024-03-11T21:38:37.581667+0000","last_clean":"2024-03-11T21:38:37.581667+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:38:37.581667+0000","last_undegraded":"2024-03-11T21:38:37.581667+0000","last_fullsized":"2024-03-11T21:38:37.581667+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.684101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.684101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1365,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576398+0000","last_change":"2024-03-11T21:36:19.742269+0000","last_active":"2024-03-11T21:38:37.576398+0000","last_peered":"2024-03-11T21:38:37.576398+0000","last_clean":"2024-03-11T21:38:37.576398+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:38:37.576398+0000","last_undegraded":"2024-03-11T21:38:37.576398+0000","last_fullsized":"2024-03-11T21:38:37.576398+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:36:19.742119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.742119+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1188,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750170+0000","last_change":"2024-03-11T21:37:58.255437+0000","last_active":"2024-03-11T21:38:37.750170+0000","last_peered":"2024-03-11T21:38:37.750170+0000","last_clean":"2024-03-11T21:38:37.750170+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:38:37.750170+0000","last_undegraded":"2024-03-11T21:38:37.750170+0000","last_fullsized":"2024-03-11T21:38:37.750170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:58.255322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:37:58.255322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:00.394815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1222,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.548893+0000","last_change":"2024-03-11T21:38:34.915316+0000","last_active":"2024-03-11T21:38:38.548893+0000","last_peered":"2024-03-11T21:38:38.548893+0000","last_clean":"2024-03-11T21:38:38.548893+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:38:38.548893+0000","last_undegraded":"2024-03-11T21:38:38.548893+0000","last_fullsized":"2024-03-11T21:38:38.548893+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:38:34.915151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:34.915151+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:58.647082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1191,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.757073+0000","last_change":"2024-03-11T21:36:42.718457+0000","last_active":"2024-03-11T21:38:37.757073+0000","last_peered":"2024-03-11T21:38:37.757073+0000","last_clean":"2024-03-11T21:38:37.757073+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:38:37.757073+0000","last_undegraded":"2024-03-11T21:38:37.757073+0000","last_fullsized":"2024-03-11T21:38:37.757073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:36:42.718312+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.718312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1155,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761574+0000","last_change":"2024-03-11T21:37:35.754246+0000","last_active":"2024-03-11T21:38:37.761574+0000","last_peered":"2024-03-11T21:38:37.761574+0000","last_clean":"2024-03-11T21:38:37.761574+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:38:37.761574+0000","last_undegraded":"2024-03-11T21:38:37.761574+0000","last_fullsized":"2024-03-11T21:38:37.761574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:37:35.753995+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:37:35.753995+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:39.501144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":395,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753908+0000","last_change":"2024-03-11T21:37:53.192844+0000","last_active":"2024-03-11T21:38:37.753908+0000","last_peered":"2024-03-11T21:38:37.753908+0000","last_clean":"2024-03-11T21:38:37.753908+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:38:37.753908+0000","last_undegraded":"2024-03-11T21:38:37.753908+0000","last_fullsized":"2024-03-11T21:38:37.753908+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:53.192751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:53.192751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:02.282164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762033+0000","last_change":"2024-03-11T21:37:56.704443+0000","last_active":"2024-03-11T21:38:37.762033+0000","last_peered":"2024-03-11T21:38:37.762033+0000","last_clean":"2024-03-11T21:38:37.762033+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:38:37.762033+0000","last_undegraded":"2024-03-11T21:38:37.762033+0000","last_fullsized":"2024-03-11T21:38:37.762033+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:56.704343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:56.704343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:19.545425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1179,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.749914+0000","last_change":"2024-03-11T21:38:37.460349+0000","last_active":"2024-03-11T21:38:37.749914+0000","last_peered":"2024-03-11T21:38:37.749914+0000","last_clean":"2024-03-11T21:38:37.749914+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:38:37.749914+0000","last_undegraded":"2024-03-11T21:38:37.749914+0000","last_fullsized":"2024-03-11T21:38:37.749914+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:37.460138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:38:37.460138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:49.367690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2084,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576337+0000","last_change":"2024-03-11T21:36:38.633175+0000","last_active":"2024-03-11T21:38:37.576337+0000","last_peered":"2024-03-11T21:38:37.576337+0000","last_clean":"2024-03-11T21:38:37.576337+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:38:37.576337+0000","last_undegraded":"2024-03-11T21:38:37.576337+0000","last_fullsized":"2024-03-11T21:38:37.576337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:36:38.633053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:38.633053+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":914,"num_read_kb":909,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1195,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.586183+0000","last_change":"2024-03-11T21:38:30.798109+0000","last_active":"2024-03-11T21:38:37.586183+0000","last_peered":"2024-03-11T21:38:37.586183+0000","last_clean":"2024-03-11T21:38:37.586183+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:38:37.586183+0000","last_undegraded":"2024-03-11T21:38:37.586183+0000","last_fullsized":"2024-03-11T21:38:37.586183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:38:30.797863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:30.797863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:47.802182+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1090,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.538140+0000","last_change":"2024-03-11T21:37:26.122849+0000","last_active":"2024-03-11T21:38:39.538140+0000","last_peered":"2024-03-11T21:38:39.538140+0000","last_clean":"2024-03-11T21:38:39.538140+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:38:39.538140+0000","last_undegraded":"2024-03-11T21:38:39.538140+0000","last_fullsized":"2024-03-11T21:38:39.538140+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:37:26.122498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:26.122498+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1167,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.751581+0000","last_change":"2024-03-11T21:36:37.141040+0000","last_active":"2024-03-11T21:38:37.751581+0000","last_peered":"2024-03-11T21:38:37.751581+0000","last_clean":"2024-03-11T21:38:37.751581+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:38:37.751581+0000","last_undegraded":"2024-03-11T21:38:37.751581+0000","last_fullsized":"2024-03-11T21:38:37.751581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:37.140941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:36:37.140941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":400,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754479+0000","last_change":"2024-03-11T21:37:52.833647+0000","last_active":"2024-03-11T21:38:37.754479+0000","last_peered":"2024-03-11T21:38:37.754479+0000","last_clean":"2024-03-11T21:38:37.754479+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:38:37.754479+0000","last_undegraded":"2024-03-11T21:38:37.754479+0000","last_fullsized":"2024-03-11T21:38:37.754479+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:52.833501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.833501+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:01.308318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.763549+0000","last_change":"2024-03-11T21:37:38.784462+0000","last_active":"2024-03-11T21:38:37.763549+0000","last_peered":"2024-03-11T21:38:37.763549+0000","last_clean":"2024-03-11T21:38:37.763549+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:38:37.763549+0000","last_undegraded":"2024-03-11T21:38:37.763549+0000","last_fullsized":"2024-03-11T21:38:37.763549+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:38.784321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:38.784321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:08.490269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754796+0000","last_change":"2024-03-11T21:37:44.669777+0000","last_active":"2024-03-11T21:38:37.754796+0000","last_peered":"2024-03-11T21:38:37.754796+0000","last_clean":"2024-03-11T21:38:37.754796+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:38:37.754796+0000","last_undegraded":"2024-03-11T21:38:37.754796+0000","last_fullsized":"2024-03-11T21:38:37.754796+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:44.669662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:44.669662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:57.502094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1314,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.537044+0000","last_change":"2024-03-11T21:37:49.936347+0000","last_active":"2024-03-11T21:38:39.537044+0000","last_peered":"2024-03-11T21:38:39.537044+0000","last_clean":"2024-03-11T21:38:39.537044+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:38:39.537044+0000","last_undegraded":"2024-03-11T21:38:39.537044+0000","last_fullsized":"2024-03-11T21:38:39.537044+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:37:49.936210+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:37:49.936210+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:11.376257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1165,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762291+0000","last_change":"2024-03-11T21:37:58.687404+0000","last_active":"2024-03-11T21:38:37.762291+0000","last_peered":"2024-03-11T21:38:37.762291+0000","last_clean":"2024-03-11T21:38:37.762291+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:38:37.762291+0000","last_undegraded":"2024-03-11T21:38:37.762291+0000","last_fullsized":"2024-03-11T21:38:37.762291+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:58.687257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:58.687257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:22.469325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1029,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.534023+0000","last_change":"2024-03-11T21:37:28.074221+0000","last_active":"2024-03-11T21:38:39.534023+0000","last_peered":"2024-03-11T21:38:39.534023+0000","last_clean":"2024-03-11T21:38:39.534023+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:38:39.534023+0000","last_undegraded":"2024-03-11T21:38:39.534023+0000","last_fullsized":"2024-03-11T21:38:39.534023+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:28.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:37:28.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:45.548401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":399,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.533779+0000","last_change":"2024-03-11T21:37:52.935805+0000","last_active":"2024-03-11T21:38:39.533779+0000","last_peered":"2024-03-11T21:38:39.533779+0000","last_clean":"2024-03-11T21:38:39.533779+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:38:39.533779+0000","last_undegraded":"2024-03-11T21:38:39.533779+0000","last_fullsized":"2024-03-11T21:38:39.533779+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:52.935681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:52.935681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:09.654706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1018,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.556375+0000","last_change":"2024-03-11T21:37:34.989860+0000","last_active":"2024-03-11T21:38:38.556375+0000","last_peered":"2024-03-11T21:38:38.556375+0000","last_clean":"2024-03-11T21:38:38.556375+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:38:38.556375+0000","last_undegraded":"2024-03-11T21:38:38.556375+0000","last_fullsized":"2024-03-11T21:38:38.556375+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:34.989711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:34.989711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:39.670750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1172,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.525281+0000","last_change":"2024-03-11T21:37:47.668506+0000","last_active":"2024-03-11T21:38:37.525281+0000","last_peered":"2024-03-11T21:38:37.525281+0000","last_clean":"2024-03-11T21:38:37.525281+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:38:37.525281+0000","last_undegraded":"2024-03-11T21:38:37.525281+0000","last_fullsized":"2024-03-11T21:38:37.525281+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:47.668378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:47.668378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:03.313867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.758417+0000","last_change":"2024-03-11T21:33:10.707295+0000","last_active":"2024-03-11T21:38:37.758417+0000","last_peered":"2024-03-11T21:38:37.758417+0000","last_clean":"2024-03-11T21:38:37.758417+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:38:37.758417+0000","last_undegraded":"2024-03-11T21:38:37.758417+0000","last_fullsized":"2024-03-11T21:38:37.758417+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:10.707090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:33:10.707090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1211,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.543839+0000","last_change":"2024-03-11T21:36:03.222156+0000","last_active":"2024-03-11T21:38:38.543839+0000","last_peered":"2024-03-11T21:38:38.543839+0000","last_clean":"2024-03-11T21:38:38.543839+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:38:38.543839+0000","last_undegraded":"2024-03-11T21:38:38.543839+0000","last_fullsized":"2024-03-11T21:38:38.543839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:03.221883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.221883+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1039,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.536443+0000","last_change":"2024-03-11T21:38:36.792863+0000","last_active":"2024-03-11T21:38:39.536443+0000","last_peered":"2024-03-11T21:38:39.536443+0000","last_clean":"2024-03-11T21:38:39.536443+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:38:39.536443+0000","last_undegraded":"2024-03-11T21:38:39.536443+0000","last_fullsized":"2024-03-11T21:38:39.536443+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:36.792597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:38:36.792597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:58.845400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1012,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.520821+0000","last_change":"2024-03-11T21:38:29.752594+0000","last_active":"2024-03-11T21:38:37.520821+0000","last_peered":"2024-03-11T21:38:37.520821+0000","last_clean":"2024-03-11T21:38:37.520821+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:38:37.520821+0000","last_undegraded":"2024-03-11T21:38:37.520821+0000","last_fullsized":"2024-03-11T21:38:37.520821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:29.752437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:29.752437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:36.849609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1149,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761200+0000","last_change":"2024-03-11T21:36:42.732454+0000","last_active":"2024-03-11T21:38:37.761200+0000","last_peered":"2024-03-11T21:38:37.761200+0000","last_clean":"2024-03-11T21:38:37.761200+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:38:37.761200+0000","last_undegraded":"2024-03-11T21:38:37.761200+0000","last_fullsized":"2024-03-11T21:38:37.761200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:42.732354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:42.732354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1439,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.758539+0000","last_change":"2024-03-11T21:37:46.721125+0000","last_active":"2024-03-11T21:38:37.758539+0000","last_peered":"2024-03-11T21:38:37.758539+0000","last_clean":"2024-03-11T21:38:37.758539+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:38:37.758539+0000","last_undegraded":"2024-03-11T21:38:37.758539+0000","last_fullsized":"2024-03-11T21:38:37.758539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:46.720898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:46.720898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:13.025161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1171,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.518867+0000","last_change":"2024-03-11T21:36:22.898119+0000","last_active":"2024-03-11T21:38:37.518867+0000","last_peered":"2024-03-11T21:38:37.518867+0000","last_clean":"2024-03-11T21:38:37.518867+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:38:37.518867+0000","last_undegraded":"2024-03-11T21:38:37.518867+0000","last_fullsized":"2024-03-11T21:38:37.518867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.898004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.898004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1192,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.575736+0000","last_change":"2024-03-11T21:37:32.471611+0000","last_active":"2024-03-11T21:38:37.575736+0000","last_peered":"2024-03-11T21:38:37.575736+0000","last_clean":"2024-03-11T21:38:37.575736+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:38:37.575736+0000","last_undegraded":"2024-03-11T21:38:37.575736+0000","last_fullsized":"2024-03-11T21:38:37.575736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:37:32.471467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:32.471467+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:50.436603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1159,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.525556+0000","last_change":"2024-03-11T21:37:45.670317+0000","last_active":"2024-03-11T21:38:37.525556+0000","last_peered":"2024-03-11T21:38:37.525556+0000","last_clean":"2024-03-11T21:38:37.525556+0000","last_became_active":"2024-03-11T21:23:09.159231+0000","last_became_peered":"2024-03-11T21:23:09.159231+0000","last_unstale":"2024-03-11T21:38:37.525556+0000","last_undegraded":"2024-03-11T21:38:37.525556+0000","last_fullsized":"2024-03-11T21:38:37.525556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:45.670202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:37:45.670202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:03.390395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1446,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.760174+0000","last_change":"2024-03-11T21:37:53.682714+0000","last_active":"2024-03-11T21:38:37.760174+0000","last_peered":"2024-03-11T21:38:37.760174+0000","last_clean":"2024-03-11T21:38:37.760174+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:38:37.760174+0000","last_undegraded":"2024-03-11T21:38:37.760174+0000","last_fullsized":"2024-03-11T21:38:37.760174+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:53.682383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:53.682383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:12.289088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1017,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.516663+0000","last_change":"2024-03-11T21:36:44.952662+0000","last_active":"2024-03-11T21:38:37.516663+0000","last_peered":"2024-03-11T21:38:37.516663+0000","last_clean":"2024-03-11T21:38:37.516663+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:38:37.516663+0000","last_undegraded":"2024-03-11T21:38:37.516663+0000","last_fullsized":"2024-03-11T21:38:37.516663+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.952524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.952524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1185,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.772743+0000","last_change":"2024-03-11T21:37:30.781612+0000","last_active":"2024-03-11T21:38:37.772743+0000","last_peered":"2024-03-11T21:38:37.772743+0000","last_clean":"2024-03-11T21:38:37.772743+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:38:37.772743+0000","last_undegraded":"2024-03-11T21:38:37.772743+0000","last_fullsized":"2024-03-11T21:38:37.772743+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:30.781374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:30.781374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1193,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.765057+0000","last_change":"2024-03-11T21:37:42.761696+0000","last_active":"2024-03-11T21:38:37.765057+0000","last_peered":"2024-03-11T21:38:37.765057+0000","last_clean":"2024-03-11T21:38:37.765057+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:38:37.765057+0000","last_undegraded":"2024-03-11T21:38:37.765057+0000","last_fullsized":"2024-03-11T21:38:37.765057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:42.761605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:42.761605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:00.380344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":1113,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.537712+0000","last_change":"2024-03-11T21:36:04.117071+0000","last_active":"2024-03-11T21:38:39.537712+0000","last_peered":"2024-03-11T21:38:39.537712+0000","last_clean":"2024-03-11T21:38:39.537712+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:38:39.537712+0000","last_undegraded":"2024-03-11T21:38:39.537712+0000","last_fullsized":"2024-03-11T21:38:39.537712+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:36:04.116720+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.116720+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1145,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762141+0000","last_change":"2024-03-11T21:33:39.686158+0000","last_active":"2024-03-11T21:38:37.762141+0000","last_peered":"2024-03-11T21:38:37.762141+0000","last_clean":"2024-03-11T21:38:37.762141+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:38:37.762141+0000","last_undegraded":"2024-03-11T21:38:37.762141+0000","last_fullsized":"2024-03-11T21:38:37.762141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:39.685858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:39.685858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":395,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.516206+0000","last_change":"2024-03-11T21:37:54.740519+0000","last_active":"2024-03-11T21:38:37.516206+0000","last_peered":"2024-03-11T21:38:37.516206+0000","last_clean":"2024-03-11T21:38:37.516206+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:38:37.516206+0000","last_undegraded":"2024-03-11T21:38:37.516206+0000","last_fullsized":"2024-03-11T21:38:37.516206+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:54.740292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:54.740292+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:11.103895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1018,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.543537+0000","last_change":"2024-03-11T21:36:31.174299+0000","last_active":"2024-03-11T21:38:38.543537+0000","last_peered":"2024-03-11T21:38:38.543537+0000","last_clean":"2024-03-11T21:38:38.543537+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:38:38.543537+0000","last_undegraded":"2024-03-11T21:38:38.543537+0000","last_fullsized":"2024-03-11T21:38:38.543537+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:31.174194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:31.174194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1174,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.549691+0000","last_change":"2024-03-11T21:36:28.181725+0000","last_active":"2024-03-11T21:38:38.549691+0000","last_peered":"2024-03-11T21:38:38.549691+0000","last_clean":"2024-03-11T21:38:38.549691+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:38:38.549691+0000","last_undegraded":"2024-03-11T21:38:38.549691+0000","last_fullsized":"2024-03-11T21:38:38.549691+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1234,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.752762+0000","last_change":"2024-03-11T21:36:40.206751+0000","last_active":"2024-03-11T21:38:37.752762+0000","last_peered":"2024-03-11T21:38:37.752762+0000","last_clean":"2024-03-11T21:38:37.752762+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:38:37.752762+0000","last_undegraded":"2024-03-11T21:38:37.752762+0000","last_fullsized":"2024-03-11T21:38:37.752762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:36:40.206510+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.206510+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1172,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517603+0000","last_change":"2024-03-11T21:36:02.740887+0000","last_active":"2024-03-11T21:38:37.517603+0000","last_peered":"2024-03-11T21:38:37.517603+0000","last_clean":"2024-03-11T21:38:37.517603+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:38:37.517603+0000","last_undegraded":"2024-03-11T21:38:37.517603+0000","last_fullsized":"2024-03-11T21:38:37.517603+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:02.740740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:02.740740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1154,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.584893+0000","last_change":"2024-03-11T21:36:25.713628+0000","last_active":"2024-03-11T21:38:37.584893+0000","last_peered":"2024-03-11T21:38:37.584893+0000","last_clean":"2024-03-11T21:38:37.584893+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:38:37.584893+0000","last_undegraded":"2024-03-11T21:38:37.584893+0000","last_fullsized":"2024-03-11T21:38:37.584893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.713405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.713405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":400,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.756711+0000","last_change":"2024-03-11T21:37:53.786172+0000","last_active":"2024-03-11T21:38:37.756711+0000","last_peered":"2024-03-11T21:38:37.756711+0000","last_clean":"2024-03-11T21:38:37.756711+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:38:37.756711+0000","last_undegraded":"2024-03-11T21:38:37.756711+0000","last_fullsized":"2024-03-11T21:38:37.756711+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:37:53.785939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:37:53.785939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:59.794919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1017,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.524452+0000","last_change":"2024-03-11T21:37:38.663511+0000","last_active":"2024-03-11T21:38:37.524452+0000","last_peered":"2024-03-11T21:38:37.524452+0000","last_clean":"2024-03-11T21:38:37.524452+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:38:37.524452+0000","last_undegraded":"2024-03-11T21:38:37.524452+0000","last_fullsized":"2024-03-11T21:38:37.524452+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:38.663297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:38.663297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:38.925863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1374,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.555921+0000","last_change":"2024-03-11T21:37:54.914763+0000","last_active":"2024-03-11T21:38:38.555921+0000","last_peered":"2024-03-11T21:38:38.555921+0000","last_clean":"2024-03-11T21:38:38.555921+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:38:38.555921+0000","last_undegraded":"2024-03-11T21:38:38.555921+0000","last_fullsized":"2024-03-11T21:38:38.555921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:37:54.914541+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:37:54.914541+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:17.649559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1066,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551875+0000","last_change":"2024-03-11T21:36:24.181588+0000","last_active":"2024-03-11T21:38:39.551875+0000","last_peered":"2024-03-11T21:38:39.551875+0000","last_clean":"2024-03-11T21:38:39.551875+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:38:39.551875+0000","last_undegraded":"2024-03-11T21:38:39.551875+0000","last_fullsized":"2024-03-11T21:38:39.551875+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:36:24.181487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.181487+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1394,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.522966+0000","last_change":"2024-03-11T21:36:20.831599+0000","last_active":"2024-03-11T21:38:37.522966+0000","last_peered":"2024-03-11T21:38:37.522966+0000","last_clean":"2024-03-11T21:38:37.522966+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:38:37.522966+0000","last_undegraded":"2024-03-11T21:38:37.522966+0000","last_fullsized":"2024-03-11T21:38:37.522966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:36:20.831450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:20.831450+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1037,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.536120+0000","last_change":"2024-03-11T21:36:34.998047+0000","last_active":"2024-03-11T21:38:39.536120+0000","last_peered":"2024-03-11T21:38:39.536120+0000","last_clean":"2024-03-11T21:38:39.536120+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:38:39.536120+0000","last_undegraded":"2024-03-11T21:38:39.536120+0000","last_fullsized":"2024-03-11T21:38:39.536120+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.997840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.997840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1168,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.544504+0000","last_change":"2024-03-11T21:37:55.866795+0000","last_active":"2024-03-11T21:38:38.544504+0000","last_peered":"2024-03-11T21:38:38.544504+0000","last_clean":"2024-03-11T21:38:38.544504+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:38:38.544504+0000","last_undegraded":"2024-03-11T21:38:38.544504+0000","last_fullsized":"2024-03-11T21:38:38.544504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:55.866581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:37:55.866581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:02.734299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":890,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.536261+0000","last_change":"2024-03-11T21:36:32.028349+0000","last_active":"2024-03-11T21:38:39.536261+0000","last_peered":"2024-03-11T21:38:39.536261+0000","last_clean":"2024-03-11T21:38:39.536261+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:38:39.536261+0000","last_undegraded":"2024-03-11T21:38:39.536261+0000","last_fullsized":"2024-03-11T21:38:39.536261+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:32.028125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:32.028125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1194,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.756252+0000","last_change":"2024-03-11T21:37:51.818738+0000","last_active":"2024-03-11T21:38:37.756252+0000","last_peered":"2024-03-11T21:38:37.756252+0000","last_clean":"2024-03-11T21:38:37.756252+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:38:37.756252+0000","last_undegraded":"2024-03-11T21:38:37.756252+0000","last_fullsized":"2024-03-11T21:38:37.756252+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:51.818604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:37:51.818604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:05.036778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1205,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.763702+0000","last_change":"2024-03-11T21:34:32.970508+0000","last_active":"2024-03-11T21:38:37.763702+0000","last_peered":"2024-03-11T21:38:37.763702+0000","last_clean":"2024-03-11T21:38:37.763702+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:38:37.763702+0000","last_undegraded":"2024-03-11T21:38:37.763702+0000","last_fullsized":"2024-03-11T21:38:37.763702+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:34:32.970256+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.970256+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1204,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.755470+0000","last_change":"2024-03-11T21:37:57.212815+0000","last_active":"2024-03-11T21:38:37.755470+0000","last_peered":"2024-03-11T21:38:37.755470+0000","last_clean":"2024-03-11T21:38:37.755470+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:38:37.755470+0000","last_undegraded":"2024-03-11T21:38:37.755470+0000","last_fullsized":"2024-03-11T21:38:37.755470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:37:57.212655+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:37:57.212655+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:15.542039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1162,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.582113+0000","last_change":"2024-03-11T21:36:30.693038+0000","last_active":"2024-03-11T21:38:37.582113+0000","last_peered":"2024-03-11T21:38:37.582113+0000","last_clean":"2024-03-11T21:38:37.582113+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:38:37.582113+0000","last_undegraded":"2024-03-11T21:38:37.582113+0000","last_fullsized":"2024-03-11T21:38:37.582113+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.692917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.692917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.582258+0000","last_change":"2024-03-11T21:36:36.681111+0000","last_active":"2024-03-11T21:38:37.582258+0000","last_peered":"2024-03-11T21:38:37.582258+0000","last_clean":"2024-03-11T21:38:37.582258+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:38:37.582258+0000","last_undegraded":"2024-03-11T21:38:37.582258+0000","last_fullsized":"2024-03-11T21:38:37.582258+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.681022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.681022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1183,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759915+0000","last_change":"2024-03-11T21:33:29.672365+0000","last_active":"2024-03-11T21:38:37.759915+0000","last_peered":"2024-03-11T21:38:37.759915+0000","last_clean":"2024-03-11T21:38:37.759915+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:38:37.759915+0000","last_undegraded":"2024-03-11T21:38:37.759915+0000","last_fullsized":"2024-03-11T21:38:37.759915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:33:29.672103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:33:29.672103+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1201,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.757359+0000","last_change":"2024-03-11T21:38:36.880861+0000","last_active":"2024-03-11T21:38:37.757359+0000","last_peered":"2024-03-11T21:38:37.757359+0000","last_clean":"2024-03-11T21:38:37.757359+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:38:37.757359+0000","last_undegraded":"2024-03-11T21:38:37.757359+0000","last_fullsized":"2024-03-11T21:38:37.757359+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:36.880563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:38:36.880563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:05.731041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.586477+0000","last_change":"2024-03-11T21:36:34.745217+0000","last_active":"2024-03-11T21:38:37.586477+0000","last_peered":"2024-03-11T21:38:37.586477+0000","last_clean":"2024-03-11T21:38:37.586477+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:38:37.586477+0000","last_undegraded":"2024-03-11T21:38:37.586477+0000","last_fullsized":"2024-03-11T21:38:37.586477+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:34.744999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.744999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1180,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.769184+0000","last_change":"2024-03-11T21:37:27.741946+0000","last_active":"2024-03-11T21:38:37.769184+0000","last_peered":"2024-03-11T21:38:37.769184+0000","last_clean":"2024-03-11T21:38:37.769184+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:38:37.769184+0000","last_undegraded":"2024-03-11T21:38:37.769184+0000","last_fullsized":"2024-03-11T21:38:37.769184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:27.741685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:37:27.741685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.576238+0000","last_change":"2024-03-11T21:37:45.691293+0000","last_active":"2024-03-11T21:38:37.576238+0000","last_peered":"2024-03-11T21:38:37.576238+0000","last_clean":"2024-03-11T21:38:37.576238+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:38:37.576238+0000","last_undegraded":"2024-03-11T21:38:37.576238+0000","last_fullsized":"2024-03-11T21:38:37.576238+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:45.691199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:45.691199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:56.653094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1199,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.749166+0000","last_change":"2024-03-11T21:38:30.645950+0000","last_active":"2024-03-11T21:38:37.749166+0000","last_peered":"2024-03-11T21:38:37.749166+0000","last_clean":"2024-03-11T21:38:37.749166+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:38:37.749166+0000","last_undegraded":"2024-03-11T21:38:37.749166+0000","last_fullsized":"2024-03-11T21:38:37.749166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:38:30.645599+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:38:30.645599+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:54.857989+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1190,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.751104+0000","last_change":"2024-03-11T21:38:33.599460+0000","last_active":"2024-03-11T21:38:37.751104+0000","last_peered":"2024-03-11T21:38:37.751104+0000","last_clean":"2024-03-11T21:38:37.751104+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:38:37.751104+0000","last_undegraded":"2024-03-11T21:38:37.751104+0000","last_fullsized":"2024-03-11T21:38:37.751104+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:33.599338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:38:33.599338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:41.788488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1171,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.524548+0000","last_change":"2024-03-11T21:38:33.688791+0000","last_active":"2024-03-11T21:38:37.524548+0000","last_peered":"2024-03-11T21:38:37.524548+0000","last_clean":"2024-03-11T21:38:37.524548+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:38:37.524548+0000","last_undegraded":"2024-03-11T21:38:37.524548+0000","last_fullsized":"2024-03-11T21:38:37.524548+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:33.688611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:58.788749+0000","last_clean_scrub_stamp":"2024-03-11T21:38:33.688611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:47.640672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1144,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762229+0000","last_change":"2024-03-11T21:33:09.675932+0000","last_active":"2024-03-11T21:38:37.762229+0000","last_peered":"2024-03-11T21:38:37.762229+0000","last_clean":"2024-03-11T21:38:37.762229+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:38:37.762229+0000","last_undegraded":"2024-03-11T21:38:37.762229+0000","last_fullsized":"2024-03-11T21:38:37.762229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:33:09.675714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:33:09.675714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1004,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.520836+0000","last_change":"2024-03-11T21:34:53.520431+0000","last_active":"2024-03-11T21:38:37.520836+0000","last_peered":"2024-03-11T21:38:37.520836+0000","last_clean":"2024-03-11T21:38:37.520836+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:38:37.520836+0000","last_undegraded":"2024-03-11T21:38:37.520836+0000","last_fullsized":"2024-03-11T21:38:37.520836+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:53.520342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:34:53.520342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1191,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761897+0000","last_change":"2024-03-11T21:37:31.732406+0000","last_active":"2024-03-11T21:38:37.761897+0000","last_peered":"2024-03-11T21:38:37.761897+0000","last_clean":"2024-03-11T21:38:37.761897+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:38:37.761897+0000","last_undegraded":"2024-03-11T21:38:37.761897+0000","last_fullsized":"2024-03-11T21:38:37.761897+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:37:31.732231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:31.732231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:46.191316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1178,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517325+0000","last_change":"2024-03-11T21:37:24.654198+0000","last_active":"2024-03-11T21:38:37.517325+0000","last_peered":"2024-03-11T21:38:37.517325+0000","last_clean":"2024-03-11T21:38:37.517325+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:38:37.517325+0000","last_undegraded":"2024-03-11T21:38:37.517325+0000","last_fullsized":"2024-03-11T21:38:37.517325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:24.653987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:24.653987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1369,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.523723+0000","last_change":"2024-03-11T21:38:36.659890+0000","last_active":"2024-03-11T21:38:37.523723+0000","last_peered":"2024-03-11T21:38:37.523723+0000","last_clean":"2024-03-11T21:38:37.523723+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:38:37.523723+0000","last_undegraded":"2024-03-11T21:38:37.523723+0000","last_fullsized":"2024-03-11T21:38:37.523723+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:38:36.659770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:38:36.659770+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:58.861049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1168,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.549810+0000","last_change":"2024-03-11T21:37:28.954262+0000","last_active":"2024-03-11T21:38:38.549810+0000","last_peered":"2024-03-11T21:38:38.549810+0000","last_clean":"2024-03-11T21:38:38.549810+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:38:38.549810+0000","last_undegraded":"2024-03-11T21:38:38.549810+0000","last_fullsized":"2024-03-11T21:38:38.549810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:28.954160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:37:28.954160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:48.970883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1026,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.749848+0000","last_change":"2024-03-11T21:36:44.625010+0000","last_active":"2024-03-11T21:38:37.749848+0000","last_peered":"2024-03-11T21:38:37.749848+0000","last_clean":"2024-03-11T21:38:37.749848+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:38:37.749848+0000","last_undegraded":"2024-03-11T21:38:37.749848+0000","last_fullsized":"2024-03-11T21:38:37.749848+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:44.624688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:44.624688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1208,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.545330+0000","last_change":"2024-03-11T21:37:27.942660+0000","last_active":"2024-03-11T21:38:38.545330+0000","last_peered":"2024-03-11T21:38:38.545330+0000","last_clean":"2024-03-11T21:38:38.545330+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:38:38.545330+0000","last_undegraded":"2024-03-11T21:38:38.545330+0000","last_fullsized":"2024-03-11T21:38:38.545330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:37:27.942424+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:37:27.942424+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:45.569969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1180,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.555465+0000","last_change":"2024-03-11T21:37:41.904678+0000","last_active":"2024-03-11T21:38:38.555465+0000","last_peered":"2024-03-11T21:38:38.555465+0000","last_clean":"2024-03-11T21:38:38.555465+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:38:38.555465+0000","last_undegraded":"2024-03-11T21:38:38.555465+0000","last_fullsized":"2024-03-11T21:38:38.555465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:41.904487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:37:41.904487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:03.936715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1355,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.579544+0000","last_change":"2024-03-11T21:36:24.749355+0000","last_active":"2024-03-11T21:38:37.579544+0000","last_peered":"2024-03-11T21:38:37.579544+0000","last_clean":"2024-03-11T21:38:37.579544+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:38:37.579544+0000","last_undegraded":"2024-03-11T21:38:37.579544+0000","last_fullsized":"2024-03-11T21:38:37.579544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:36:24.749208+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.749208+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1174,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.755902+0000","last_change":"2024-03-11T21:38:32.517376+0000","last_active":"2024-03-11T21:38:37.755902+0000","last_peered":"2024-03-11T21:38:37.755902+0000","last_clean":"2024-03-11T21:38:37.755902+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:38:37.755902+0000","last_undegraded":"2024-03-11T21:38:37.755902+0000","last_fullsized":"2024-03-11T21:38:37.755902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:32.517284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:38:32.517284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:34.284404+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":886,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551153+0000","last_change":"2024-03-11T21:36:39.992513+0000","last_active":"2024-03-11T21:38:39.551153+0000","last_peered":"2024-03-11T21:38:39.551153+0000","last_clean":"2024-03-11T21:38:39.551153+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:38:39.551153+0000","last_undegraded":"2024-03-11T21:38:39.551153+0000","last_fullsized":"2024-03-11T21:38:39.551153+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:39.992222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.992222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1181,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762517+0000","last_change":"2024-03-11T21:32:22.373672+0000","last_active":"2024-03-11T21:38:37.762517+0000","last_peered":"2024-03-11T21:38:37.762517+0000","last_clean":"2024-03-11T21:38:37.762517+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:38:37.762517+0000","last_undegraded":"2024-03-11T21:38:37.762517+0000","last_fullsized":"2024-03-11T21:38:37.762517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'13","last_scrub_stamp":"2024-03-11T21:32:22.373456+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.373456+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1189,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.752244+0000","last_change":"2024-03-11T21:37:56.809836+0000","last_active":"2024-03-11T21:38:37.752244+0000","last_peered":"2024-03-11T21:38:37.752244+0000","last_clean":"2024-03-11T21:38:37.752244+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:38:37.752244+0000","last_undegraded":"2024-03-11T21:38:37.752244+0000","last_fullsized":"2024-03-11T21:38:37.752244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:56.809697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:37:56.809697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:25.854597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.581979+0000","last_change":"2024-03-11T21:37:30.467794+0000","last_active":"2024-03-11T21:38:37.581979+0000","last_peered":"2024-03-11T21:38:37.581979+0000","last_clean":"2024-03-11T21:38:37.581979+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:38:37.581979+0000","last_undegraded":"2024-03-11T21:38:37.581979+0000","last_fullsized":"2024-03-11T21:38:37.581979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:30.467633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:30.467633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:49.130642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1176,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.544221+0000","last_change":"2024-03-11T21:37:47.892153+0000","last_active":"2024-03-11T21:38:38.544221+0000","last_peered":"2024-03-11T21:38:38.544221+0000","last_clean":"2024-03-11T21:38:38.544221+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:38:38.544221+0000","last_undegraded":"2024-03-11T21:38:38.544221+0000","last_fullsized":"2024-03-11T21:38:38.544221+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:47.892065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:37:47.892065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:04.556336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1024,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.548606+0000","last_change":"2024-03-11T21:36:25.203562+0000","last_active":"2024-03-11T21:38:38.548606+0000","last_peered":"2024-03-11T21:38:38.548606+0000","last_clean":"2024-03-11T21:38:38.548606+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:38:38.548606+0000","last_undegraded":"2024-03-11T21:38:38.548606+0000","last_fullsized":"2024-03-11T21:38:38.548606+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:25.203464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:25.203464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1212,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753515+0000","last_change":"2024-03-11T21:37:37.158919+0000","last_active":"2024-03-11T21:38:37.753515+0000","last_peered":"2024-03-11T21:38:37.753515+0000","last_clean":"2024-03-11T21:38:37.753515+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:38:37.753515+0000","last_undegraded":"2024-03-11T21:38:37.753515+0000","last_fullsized":"2024-03-11T21:38:37.753515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:37:37.158659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:37.158659+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:53.947505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1080,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551272+0000","last_change":"2024-03-11T21:37:57.985458+0000","last_active":"2024-03-11T21:38:39.551272+0000","last_peered":"2024-03-11T21:38:39.551272+0000","last_clean":"2024-03-11T21:38:39.551272+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:38:39.551272+0000","last_undegraded":"2024-03-11T21:38:39.551272+0000","last_fullsized":"2024-03-11T21:38:39.551272+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:57.985309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:37:57.985309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:26.864620+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1049,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551788+0000","last_change":"2024-03-11T21:37:43.026692+0000","last_active":"2024-03-11T21:38:39.551788+0000","last_peered":"2024-03-11T21:38:39.551788+0000","last_clean":"2024-03-11T21:38:39.551788+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:38:39.551788+0000","last_undegraded":"2024-03-11T21:38:39.551788+0000","last_fullsized":"2024-03-11T21:38:39.551788+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:43.026602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:37:43.026602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:47.672837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1153,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.519683+0000","last_change":"2024-03-11T21:36:40.661182+0000","last_active":"2024-03-11T21:38:37.519683+0000","last_peered":"2024-03-11T21:38:37.519683+0000","last_clean":"2024-03-11T21:38:37.519683+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:38:37.519683+0000","last_undegraded":"2024-03-11T21:38:37.519683+0000","last_fullsized":"2024-03-11T21:38:37.519683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1015,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.546812+0000","last_change":"2024-03-11T21:35:08.022701+0000","last_active":"2024-03-11T21:38:38.546812+0000","last_peered":"2024-03-11T21:38:38.546812+0000","last_clean":"2024-03-11T21:38:38.546812+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:38:38.546812+0000","last_undegraded":"2024-03-11T21:38:38.546812+0000","last_fullsized":"2024-03-11T21:38:38.546812+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.022536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.022536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1194,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.553119+0000","last_change":"2024-03-11T21:36:34.076596+0000","last_active":"2024-03-11T21:38:38.553119+0000","last_peered":"2024-03-11T21:38:38.553119+0000","last_clean":"2024-03-11T21:38:38.553119+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:38:38.553119+0000","last_undegraded":"2024-03-11T21:38:38.553119+0000","last_fullsized":"2024-03-11T21:38:38.553119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:36:34.076364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:34.076364+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1170,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.525014+0000","last_change":"2024-03-11T21:36:36.046087+0000","last_active":"2024-03-11T21:38:37.525014+0000","last_peered":"2024-03-11T21:38:37.525014+0000","last_clean":"2024-03-11T21:38:37.525014+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:38:37.525014+0000","last_undegraded":"2024-03-11T21:38:37.525014+0000","last_fullsized":"2024-03-11T21:38:37.525014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:36.045956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:36.045956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1181,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750981+0000","last_change":"2024-03-11T21:38:31.651449+0000","last_active":"2024-03-11T21:38:37.750981+0000","last_peered":"2024-03-11T21:38:37.750981+0000","last_clean":"2024-03-11T21:38:37.750981+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:38:37.750981+0000","last_undegraded":"2024-03-11T21:38:37.750981+0000","last_fullsized":"2024-03-11T21:38:37.750981+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:31.651335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:38:31.651335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:38.701317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1144,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.760040+0000","last_change":"2024-03-11T21:34:34.961557+0000","last_active":"2024-03-11T21:38:37.760040+0000","last_peered":"2024-03-11T21:38:37.760040+0000","last_clean":"2024-03-11T21:38:37.760040+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:38:37.760040+0000","last_undegraded":"2024-03-11T21:38:37.760040+0000","last_fullsized":"2024-03-11T21:38:37.760040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:34.961331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:34:34.961331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1026,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.547727+0000","last_change":"2024-03-11T21:37:37.049846+0000","last_active":"2024-03-11T21:38:38.547727+0000","last_peered":"2024-03-11T21:38:38.547727+0000","last_clean":"2024-03-11T21:38:38.547727+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:38:38.547727+0000","last_undegraded":"2024-03-11T21:38:38.547727+0000","last_fullsized":"2024-03-11T21:38:38.547727+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:37.049613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:37.049613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:42.502846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1200,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.524389+0000","last_change":"2024-03-11T21:37:55.751784+0000","last_active":"2024-03-11T21:38:37.524389+0000","last_peered":"2024-03-11T21:38:37.524389+0000","last_clean":"2024-03-11T21:38:37.524389+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:38:37.524389+0000","last_undegraded":"2024-03-11T21:38:37.524389+0000","last_fullsized":"2024-03-11T21:38:37.524389+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:37:55.751621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:55.751621+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:58.691919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1181,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.757014+0000","last_change":"2024-03-11T21:36:41.700743+0000","last_active":"2024-03-11T21:38:37.757014+0000","last_peered":"2024-03-11T21:38:37.757014+0000","last_clean":"2024-03-11T21:38:37.757014+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:38:37.757014+0000","last_undegraded":"2024-03-11T21:38:37.757014+0000","last_fullsized":"2024-03-11T21:38:37.757014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:36:41.700576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1148,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.758875+0000","last_change":"2024-03-11T21:36:29.509965+0000","last_active":"2024-03-11T21:38:37.758875+0000","last_peered":"2024-03-11T21:38:37.758875+0000","last_clean":"2024-03-11T21:38:37.758875+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:38:37.758875+0000","last_undegraded":"2024-03-11T21:38:37.758875+0000","last_fullsized":"2024-03-11T21:38:37.758875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:29.509836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:29.509836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1153,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.582246+0000","last_change":"2024-03-11T21:37:42.629513+0000","last_active":"2024-03-11T21:38:37.582246+0000","last_peered":"2024-03-11T21:38:37.582246+0000","last_clean":"2024-03-11T21:38:37.582246+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:38:37.582246+0000","last_undegraded":"2024-03-11T21:38:37.582246+0000","last_fullsized":"2024-03-11T21:38:37.582246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:42.629417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:28.313489+0000","last_clean_scrub_stamp":"2024-03-11T21:37:42.629417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:01.791148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1002,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761689+0000","last_change":"2024-03-11T21:36:46.798923+0000","last_active":"2024-03-11T21:38:37.761689+0000","last_peered":"2024-03-11T21:38:37.761689+0000","last_clean":"2024-03-11T21:38:37.761689+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:38:37.761689+0000","last_undegraded":"2024-03-11T21:38:37.761689+0000","last_fullsized":"2024-03-11T21:38:37.761689+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:36:46.798825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":1133,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.534169+0000","last_change":"2024-03-11T21:36:39.151588+0000","last_active":"2024-03-11T21:38:39.534169+0000","last_peered":"2024-03-11T21:38:39.534169+0000","last_clean":"2024-03-11T21:38:39.534169+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:38:39.534169+0000","last_undegraded":"2024-03-11T21:38:39.534169+0000","last_fullsized":"2024-03-11T21:38:39.534169+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:36:39.151333+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1176,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.518324+0000","last_change":"2024-03-11T21:37:50.702017+0000","last_active":"2024-03-11T21:38:37.518324+0000","last_peered":"2024-03-11T21:38:37.518324+0000","last_clean":"2024-03-11T21:38:37.518324+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:38:37.518324+0000","last_undegraded":"2024-03-11T21:38:37.518324+0000","last_fullsized":"2024-03-11T21:38:37.518324+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:50.701811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:50.701811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:54.609249+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1374,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.757130+0000","last_change":"2024-03-11T21:36:22.429814+0000","last_active":"2024-03-11T21:38:37.757130+0000","last_peered":"2024-03-11T21:38:37.757130+0000","last_clean":"2024-03-11T21:38:37.757130+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:38:37.757130+0000","last_undegraded":"2024-03-11T21:38:37.757130+0000","last_fullsized":"2024-03-11T21:38:37.757130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:36:22.429570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.429570+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1146,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761801+0000","last_change":"2024-03-11T21:36:33.500511+0000","last_active":"2024-03-11T21:38:37.761801+0000","last_peered":"2024-03-11T21:38:37.761801+0000","last_clean":"2024-03-11T21:38:37.761801+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:38:37.761801+0000","last_undegraded":"2024-03-11T21:38:37.761801+0000","last_fullsized":"2024-03-11T21:38:37.761801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:36:33.500161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:33.500161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1017,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517010+0000","last_change":"2024-03-11T21:36:19.803577+0000","last_active":"2024-03-11T21:38:37.517010+0000","last_peered":"2024-03-11T21:38:37.517010+0000","last_clean":"2024-03-11T21:38:37.517010+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:38:37.517010+0000","last_undegraded":"2024-03-11T21:38:37.517010+0000","last_fullsized":"2024-03-11T21:38:37.517010+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:19.803281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:19.803281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1195,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.575991+0000","last_change":"2024-03-11T21:34:56.556273+0000","last_active":"2024-03-11T21:38:37.575991+0000","last_peered":"2024-03-11T21:38:37.575991+0000","last_clean":"2024-03-11T21:38:37.575991+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:38:37.575991+0000","last_undegraded":"2024-03-11T21:38:37.575991+0000","last_fullsized":"2024-03-11T21:38:37.575991+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:34:56.556007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:34:56.556007+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.519807+0000","last_change":"2024-03-11T21:36:23.725753+0000","last_active":"2024-03-11T21:38:37.519807+0000","last_peered":"2024-03-11T21:38:37.519807+0000","last_clean":"2024-03-11T21:38:37.519807+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:38:37.519807+0000","last_undegraded":"2024-03-11T21:38:37.519807+0000","last_fullsized":"2024-03-11T21:38:37.519807+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:36:23.725651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:23.725651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1165,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.549565+0000","last_change":"2024-03-11T21:35:01.025842+0000","last_active":"2024-03-11T21:38:38.549565+0000","last_peered":"2024-03-11T21:38:38.549565+0000","last_clean":"2024-03-11T21:38:38.549565+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:38:38.549565+0000","last_undegraded":"2024-03-11T21:38:38.549565+0000","last_fullsized":"2024-03-11T21:38:38.549565+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:01.025597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:39.439030+0000","last_clean_scrub_stamp":"2024-03-11T21:35:01.025597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1031,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551213+0000","last_change":"2024-03-11T21:36:12.033537+0000","last_active":"2024-03-11T21:38:39.551213+0000","last_peered":"2024-03-11T21:38:39.551213+0000","last_clean":"2024-03-11T21:38:39.551213+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:38:39.551213+0000","last_undegraded":"2024-03-11T21:38:39.551213+0000","last_fullsized":"2024-03-11T21:38:39.551213+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.033259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.033259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1002,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759570+0000","last_change":"2024-03-11T21:37:36.780195+0000","last_active":"2024-03-11T21:38:37.759570+0000","last_peered":"2024-03-11T21:38:37.759570+0000","last_clean":"2024-03-11T21:38:37.759570+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:38:37.759570+0000","last_undegraded":"2024-03-11T21:38:37.759570+0000","last_fullsized":"2024-03-11T21:38:37.759570+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:36.780037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:36.780037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:50.848747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1183,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.555759+0000","last_change":"2024-03-11T21:36:27.196610+0000","last_active":"2024-03-11T21:38:38.555759+0000","last_peered":"2024-03-11T21:38:38.555759+0000","last_clean":"2024-03-11T21:38:38.555759+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:38:38.555759+0000","last_undegraded":"2024-03-11T21:38:38.555759+0000","last_fullsized":"2024-03-11T21:38:38.555759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:36:27.196386+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.196386+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1181,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754704+0000","last_change":"2024-03-11T21:36:40.711020+0000","last_active":"2024-03-11T21:38:37.754704+0000","last_peered":"2024-03-11T21:38:37.754704+0000","last_clean":"2024-03-11T21:38:37.754704+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:38:37.754704+0000","last_undegraded":"2024-03-11T21:38:37.754704+0000","last_fullsized":"2024-03-11T21:38:37.754704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:36:40.710906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1025,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.551396+0000","last_change":"2024-03-11T21:36:28.149075+0000","last_active":"2024-03-11T21:38:39.551396+0000","last_peered":"2024-03-11T21:38:39.551396+0000","last_clean":"2024-03-11T21:38:39.551396+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:38:39.551396+0000","last_undegraded":"2024-03-11T21:38:39.551396+0000","last_fullsized":"2024-03-11T21:38:39.551396+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:28.148859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:36:28.148859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1175,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754856+0000","last_change":"2024-03-11T21:38:35.453424+0000","last_active":"2024-03-11T21:38:37.754856+0000","last_peered":"2024-03-11T21:38:37.754856+0000","last_clean":"2024-03-11T21:38:37.754856+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:38:37.754856+0000","last_undegraded":"2024-03-11T21:38:37.754856+0000","last_fullsized":"2024-03-11T21:38:37.754856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:35.453192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:38:35.453192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:37.142492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1022,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.749883+0000","last_change":"2024-03-11T21:37:31.174678+0000","last_active":"2024-03-11T21:38:37.749883+0000","last_peered":"2024-03-11T21:38:37.749883+0000","last_clean":"2024-03-11T21:38:37.749883+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:38:37.749883+0000","last_undegraded":"2024-03-11T21:38:37.749883+0000","last_fullsized":"2024-03-11T21:38:37.749883+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:31.174562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:25.655683+0000","last_clean_scrub_stamp":"2024-03-11T21:37:31.174562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:39.385178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"389'15","reported_seq":1216,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.751731+0000","last_change":"2024-03-11T21:37:29.208680+0000","last_active":"2024-03-11T21:38:37.751731+0000","last_peered":"2024-03-11T21:38:37.751731+0000","last_clean":"2024-03-11T21:38:37.751731+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:38:37.751731+0000","last_undegraded":"2024-03-11T21:38:37.751731+0000","last_fullsized":"2024-03-11T21:38:37.751731+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:37:29.208431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:29.208431+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:56.377664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1165,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.580747+0000","last_change":"2024-03-11T21:36:12.656033+0000","last_active":"2024-03-11T21:38:37.580747+0000","last_peered":"2024-03-11T21:38:37.580747+0000","last_clean":"2024-03-11T21:38:37.580747+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:38:37.580747+0000","last_undegraded":"2024-03-11T21:38:37.580747+0000","last_fullsized":"2024-03-11T21:38:37.580747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.655615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.655615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1170,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.753685+0000","last_change":"2024-03-11T21:37:27.203194+0000","last_active":"2024-03-11T21:38:37.753685+0000","last_peered":"2024-03-11T21:38:37.753685+0000","last_clean":"2024-03-11T21:38:37.753685+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:38:37.753685+0000","last_undegraded":"2024-03-11T21:38:37.753685+0000","last_fullsized":"2024-03-11T21:38:37.753685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:27.202986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:32.753964+0000","last_clean_scrub_stamp":"2024-03-11T21:37:27.202986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:52.632703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1154,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.586296+0000","last_change":"2024-03-11T21:37:57.570999+0000","last_active":"2024-03-11T21:38:37.586296+0000","last_peered":"2024-03-11T21:38:37.586296+0000","last_clean":"2024-03-11T21:38:37.586296+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:38:37.586296+0000","last_undegraded":"2024-03-11T21:38:37.586296+0000","last_fullsized":"2024-03-11T21:38:37.586296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:37:57.570887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:10.786553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.763859+0000","last_change":"2024-03-11T21:37:39.787759+0000","last_active":"2024-03-11T21:38:37.763859+0000","last_peered":"2024-03-11T21:38:37.763859+0000","last_clean":"2024-03-11T21:38:37.763859+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:38:37.763859+0000","last_undegraded":"2024-03-11T21:38:37.763859+0000","last_fullsized":"2024-03-11T21:38:37.763859+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:39.787465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:37:39.787465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:06.273303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1198,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.523427+0000","last_change":"2024-03-11T21:37:32.718945+0000","last_active":"2024-03-11T21:38:37.523427+0000","last_peered":"2024-03-11T21:38:37.523427+0000","last_clean":"2024-03-11T21:38:37.523427+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:38:37.523427+0000","last_undegraded":"2024-03-11T21:38:37.523427+0000","last_fullsized":"2024-03-11T21:38:37.523427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:37:32.718706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:37:32.718706+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:00.421383+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1164,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.760855+0000","last_change":"2024-03-11T21:38:33.762516+0000","last_active":"2024-03-11T21:38:37.760855+0000","last_peered":"2024-03-11T21:38:37.760855+0000","last_clean":"2024-03-11T21:38:37.760855+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:38:37.760855+0000","last_undegraded":"2024-03-11T21:38:37.760855+0000","last_fullsized":"2024-03-11T21:38:37.760855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:33.762300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:38:33.762300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:34.708216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1154,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.518781+0000","last_change":"2024-03-11T21:36:09.648618+0000","last_active":"2024-03-11T21:38:37.518781+0000","last_peered":"2024-03-11T21:38:37.518781+0000","last_clean":"2024-03-11T21:38:37.518781+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:38:37.518781+0000","last_undegraded":"2024-03-11T21:38:37.518781+0000","last_fullsized":"2024-03-11T21:38:37.518781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:09.648319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.648319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1176,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.544788+0000","last_change":"2024-03-11T21:37:42.892967+0000","last_active":"2024-03-11T21:38:38.544788+0000","last_peered":"2024-03-11T21:38:38.544788+0000","last_clean":"2024-03-11T21:38:38.544788+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:38:38.544788+0000","last_undegraded":"2024-03-11T21:38:38.544788+0000","last_fullsized":"2024-03-11T21:38:38.544788+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:42.892886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:37:42.892886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:49.283922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1033,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.754182+0000","last_change":"2024-03-11T21:38:36.440185+0000","last_active":"2024-03-11T21:38:37.754182+0000","last_peered":"2024-03-11T21:38:37.754182+0000","last_clean":"2024-03-11T21:38:37.754182+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:38:37.754182+0000","last_undegraded":"2024-03-11T21:38:37.754182+0000","last_fullsized":"2024-03-11T21:38:37.754182+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:36.439816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:36.439816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:52.570026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1225,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.520760+0000","last_change":"2024-03-11T21:36:11.636803+0000","last_active":"2024-03-11T21:38:37.520760+0000","last_peered":"2024-03-11T21:38:37.520760+0000","last_clean":"2024-03-11T21:38:37.520760+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:38:37.520760+0000","last_undegraded":"2024-03-11T21:38:37.520760+0000","last_fullsized":"2024-03-11T21:38:37.520760+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:36:11.636460+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.636460+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1167,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.762412+0000","last_change":"2024-03-11T21:36:11.544839+0000","last_active":"2024-03-11T21:38:37.762412+0000","last_peered":"2024-03-11T21:38:37.762412+0000","last_clean":"2024-03-11T21:38:37.762412+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:38:37.762412+0000","last_undegraded":"2024-03-11T21:38:37.762412+0000","last_fullsized":"2024-03-11T21:38:37.762412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:11.544702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.544702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1157,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.759887+0000","last_change":"2024-03-11T21:36:26.444581+0000","last_active":"2024-03-11T21:38:37.759887+0000","last_peered":"2024-03-11T21:38:37.759887+0000","last_clean":"2024-03-11T21:38:37.759887+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:38:37.759887+0000","last_undegraded":"2024-03-11T21:38:37.759887+0000","last_fullsized":"2024-03-11T21:38:37.759887+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:26.444362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.444362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1159,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.517234+0000","last_change":"2024-03-11T21:36:14.674630+0000","last_active":"2024-03-11T21:38:37.517234+0000","last_peered":"2024-03-11T21:38:37.517234+0000","last_clean":"2024-03-11T21:38:37.517234+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:38:37.517234+0000","last_undegraded":"2024-03-11T21:38:37.517234+0000","last_fullsized":"2024-03-11T21:38:37.517234+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.674494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.674494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1011,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.761935+0000","last_change":"2024-03-11T21:36:27.467589+0000","last_active":"2024-03-11T21:38:37.761935+0000","last_peered":"2024-03-11T21:38:37.761935+0000","last_clean":"2024-03-11T21:38:37.761935+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:38:37.761935+0000","last_undegraded":"2024-03-11T21:38:37.761935+0000","last_fullsized":"2024-03-11T21:38:37.761935+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:27.467488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:27.467488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1213,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.554797+0000","last_change":"2024-03-11T21:36:26.178132+0000","last_active":"2024-03-11T21:38:38.554797+0000","last_peered":"2024-03-11T21:38:38.554797+0000","last_clean":"2024-03-11T21:38:38.554797+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:38:38.554797+0000","last_undegraded":"2024-03-11T21:38:38.554797+0000","last_fullsized":"2024-03-11T21:38:38.554797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:36:26.177873+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:36:26.177873+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1166,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.584888+0000","last_change":"2024-03-11T21:37:40.562262+0000","last_active":"2024-03-11T21:38:37.584888+0000","last_peered":"2024-03-11T21:38:37.584888+0000","last_clean":"2024-03-11T21:38:37.584888+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:38:37.584888+0000","last_undegraded":"2024-03-11T21:38:37.584888+0000","last_fullsized":"2024-03-11T21:38:37.584888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:40.562002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:37:40.562002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:39:01.402035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1029,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.534310+0000","last_change":"2024-03-11T21:37:44.993205+0000","last_active":"2024-03-11T21:38:39.534310+0000","last_peered":"2024-03-11T21:38:39.534310+0000","last_clean":"2024-03-11T21:38:39.534310+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:38:39.534310+0000","last_undegraded":"2024-03-11T21:38:39.534310+0000","last_fullsized":"2024-03-11T21:38:39.534310+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:37:44.992906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:44.992906+0000","last_clean_scrub_stamp":"2024-03-11T21:37:44.992906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:57.317982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1175,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.751589+0000","last_change":"2024-03-11T21:36:21.238514+0000","last_active":"2024-03-11T21:38:37.751589+0000","last_peered":"2024-03-11T21:38:37.751589+0000","last_clean":"2024-03-11T21:38:37.751589+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:38:37.751589+0000","last_undegraded":"2024-03-11T21:38:37.751589+0000","last_fullsized":"2024-03-11T21:38:37.751589+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:21.238275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:36:21.238275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":880,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-03-11T21:38:39.536740+0000","last_change":"2024-03-11T21:36:22.157617+0000","last_active":"2024-03-11T21:38:39.536740+0000","last_peered":"2024-03-11T21:38:39.536740+0000","last_clean":"2024-03-11T21:38:39.536740+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:38:39.536740+0000","last_undegraded":"2024-03-11T21:38:39.536740+0000","last_fullsized":"2024-03-11T21:38:39.536740+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:22.157410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:36:22.157410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1192,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.585004+0000","last_change":"2024-03-11T21:36:15.678133+0000","last_active":"2024-03-11T21:38:37.585004+0000","last_peered":"2024-03-11T21:38:37.585004+0000","last_clean":"2024-03-11T21:38:37.585004+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:38:37.585004+0000","last_undegraded":"2024-03-11T21:38:37.585004+0000","last_fullsized":"2024-03-11T21:38:37.585004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:36:15.678027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.678027+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1182,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-03-11T21:38:38.554950+0000","last_change":"2024-03-11T21:36:24.220137+0000","last_active":"2024-03-11T21:38:38.554950+0000","last_peered":"2024-03-11T21:38:38.554950+0000","last_clean":"2024-03-11T21:38:38.554950+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:38:38.554950+0000","last_undegraded":"2024-03-11T21:38:38.554950+0000","last_fullsized":"2024-03-11T21:38:38.554950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:36:24.220021+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:36:24.220021+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1179,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.750000+0000","last_change":"2024-03-11T21:36:30.228605+0000","last_active":"2024-03-11T21:38:37.750000+0000","last_peered":"2024-03-11T21:38:37.750000+0000","last_clean":"2024-03-11T21:38:37.750000+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:38:37.750000+0000","last_undegraded":"2024-03-11T21:38:37.750000+0000","last_fullsized":"2024-03-11T21:38:37.750000+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:30.228458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:36:30.228458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1145,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-11T21:38:37.580534+0000","last_change":"2024-03-11T21:35:00.524627+0000","last_active":"2024-03-11T21:38:37.580534+0000","last_peered":"2024-03-11T21:38:37.580534+0000","last_clean":"2024-03-11T21:38:37.580534+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:38:37.580534+0000","last_undegraded":"2024-03-11T21:38:37.580534+0000","last_fullsized":"2024-03-11T21:38:37.580534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:00.524430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:35:00.524430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":306,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":30,"acting":30,"num_store_stats":2},{"poolid":118,"num_pg":32,"stat_sum":{"num_bytes":8214824,"num_objects":202,"num_object_clones":16,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":221,"num_read_kb":63,"num_write":792,"num_write_kb":12819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26959872,"data_stored":24644499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2124,"ondisk_log_size":2124,"up":96,"acting":96,"num_store_stats":8},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":914,"num_read_kb":909,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841734,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94804,"kb_used_data":15016,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93650348,"statfs":{"total":95995035648,"available":95897956352,"internally_reserved":0,"allocated":15376384,"data_stored":11014294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":81656266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168783,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81564,"kb_used_data":14268,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93663588,"statfs":{"total":95995035648,"available":95911514112,"internally_reserved":0,"allocated":14610432,"data_stored":10472329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352406,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42584,"kb_used_data":10500,"kb_used_omap":27,"kb_used_meta":32036,"kb_avail":93702568,"statfs":{"total":95995035648,"available":95951429632,"internally_reserved":0,"allocated":10752000,"data_stored":6509251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":32805193},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738638,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94032,"kb_used_data":18404,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93651120,"statfs":{"total":95995035648,"available":95898746880,"internally_reserved":0,"allocated":18845696,"data_stored":14679037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378873,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82224,"kb_used_data":10780,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662928,"statfs":{"total":95995035648,"available":95910838272,"internally_reserved":0,"allocated":11038720,"data_stored":6751631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117229,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82532,"kb_used_data":11064,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93662620,"statfs":{"total":95995035648,"available":95910522880,"internally_reserved":0,"allocated":11329536,"data_stored":7073315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855587,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90112,"kb_used_data":14496,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655040,"statfs":{"total":95995035648,"available":95902760960,"internally_reserved":0,"allocated":14843904,"data_stored":10484319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495833,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86540,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93658612,"statfs":{"total":95995035648,"available":95906418688,"internally_reserved":0,"allocated":15458304,"data_stored":11272996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8198440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":9003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":4013301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4440064,"data_stored":4203228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4002713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4583424,"data_stored":4204909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":306,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":306,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:38:44.295 INFO:tasks.thrashosds.thrasher:Setting 6.7 to [0, 4, 2] 2024-03-11T21:38:44.295 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.7', '0', '4', '2'] 2024-03-11T21:38:44.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 6.7 0 4 2 2024-03-11T21:38:44.295 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:44.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:44.422 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:44.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:44.544 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:44.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.675 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:44.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:44.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:44.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:44.932 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:44.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:38:45.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:45.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:45.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:45.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:45.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:43.524649+0000 mgr.y (mgr.14550) 618 : cluster 0 pgmap v985: 270 pgs: 5 creating+peering, 265 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:45.188 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:43.717942+0000 osd.1 (osd.1) 159 : cluster 0 39.c scrub starts 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:43.720354+0000 osd.1 (osd.1) 160 : cluster 0 39.c scrub ok 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.442481+0000 osd.4 (osd.4) 351 : cluster 0 5.18 scrub starts 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.445500+0000 osd.4 (osd.4) 352 : cluster 0 5.18 scrub ok 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: audit 2024-03-11T21:38:44.463601+0000 mon.a (mon.0) 3177 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.469356+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: audit 2024-03-11T21:38:44.471555+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[29284]: audit 2024-03-11T21:38:44.472809+0000 mon.a (mon.0) 3179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:43.524649+0000 mgr.y (mgr.14550) 618 : cluster 0 pgmap v985: 270 pgs: 5 creating+peering, 265 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:45.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:43.717942+0000 osd.1 (osd.1) 159 : cluster 0 39.c scrub starts 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:43.720354+0000 osd.1 (osd.1) 160 : cluster 0 39.c scrub ok 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.442481+0000 osd.4 (osd.4) 351 : cluster 0 5.18 scrub starts 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.445500+0000 osd.4 (osd.4) 352 : cluster 0 5.18 scrub ok 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: audit 2024-03-11T21:38:44.463601+0000 mon.a (mon.0) 3177 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.469356+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: audit 2024-03-11T21:38:44.471555+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:44 smithi143 bash[21512]: audit 2024-03-11T21:38:44.472809+0000 mon.a (mon.0) 3179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:45.274 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:43.524649+0000 mgr.y (mgr.14550) 618 : cluster 0 pgmap v985: 270 pgs: 5 creating+peering, 265 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:43.717942+0000 osd.1 (osd.1) 159 : cluster 0 39.c scrub starts 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:43.720354+0000 osd.1 (osd.1) 160 : cluster 0 39.c scrub ok 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.442481+0000 osd.4 (osd.4) 351 : cluster 0 5.18 scrub starts 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.445500+0000 osd.4 (osd.4) 352 : cluster 0 5.18 scrub ok 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: audit 2024-03-11T21:38:44.463601+0000 mon.a (mon.0) 3177 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.469356+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: audit 2024-03-11T21:38:44.471555+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.275 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:44 smithi203 bash[23654]: audit 2024-03-11T21:38:44.472809+0000 mon.a (mon.0) 3179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:38:45.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:45.318 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:45.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:45.451 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:45.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:45.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:45.715 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 -- 172.21.15.143:0/712737932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 msgr2=0x7f0f401082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 --2- 172.21.15.143:0/712737932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f401082b0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009990 tx=0x7f0f30009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 -- 172.21.15.143:0/712737932 shutdown_connections 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 --2- 172.21.15.143:0/712737932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40102830 0x7f0f4010ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 --2- 172.21.15.143:0/712737932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f401082b0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 --2- 172.21.15.143:0/712737932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40101e90 0x7f0f401022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 -- 172.21.15.143:0/712737932 >> 172.21.15.143:0/712737932 conn(0x7f0f40078f90 msgr2=0x7f0f400793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 -- 172.21.15.143:0/712737932 shutdown_connections 2024-03-11T21:38:45.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.767+0000 7f0f458f0700 1 -- 172.21.15.143:0/712737932 wait complete. 2024-03-11T21:38:45.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 librados: starting msgr at 2024-03-11T21:38:45.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 librados: starting objecter 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 Processor -- start 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- start start 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 librados: setting wanted keys 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 librados: calling monclient init 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40101e90 0x7f0f400740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f40074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f401115f0 con 0x7f0f40102830 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f40111470 con 0x7f0f40101e90 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f40111770 con 0x7f0f40107ed0 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3e7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f40074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:45.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3e7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57052/0 (socket says 172.21.15.143:57052) 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3e7fc700 1 -- 172.21.15.143:0/4137368833 learned_addr learned my addr 172.21.15.143:0/4137368833 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3effd700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40101e90 0x7f0f400740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 -- 172.21.15.143:0/4137368833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40101e90 msgr2=0x7f0f400740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40101e90 0x7f0f400740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 -- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 msgr2=0x7f0f40074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 -- 172.21.15.143:0/4137368833 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f401a6100 con 0x7f0f40107ed0 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3e7fc700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f3f7fe700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f40074b60 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800aa90 tx=0x7f0f3800ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:45.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 10 client.?.objecter ms_handle_connect 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f38017020 con 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0f38004bb0 con 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f401a63f0 con 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f38005020 con 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 10 client.41858.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f401a68e0 con 0x7f0f40107ed0 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 librados: init done 2024-03-11T21:38:45.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4004faa0 con 0x7f0f40107ed0 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0f3800c580 con 0x7f0f40107ed0 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.771+0000 7f0f448ee700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 0x7f0f28071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(638..638 src has 87..638) v4 ==== 9670+0+0 (secure 0 0 0) 0x7f0f3808df40 con 0x7f0f40107ed0 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 10 client.41858.objecter ms_dispatch 0x7f0f40001550 osd_map(638..638 src has 87..638) v4 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 3 client.41858.objecter handle_osd_map got epochs [638,638] > 0 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 3 client.41858.objecter handle_osd_map decoding full epoch 638 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f3effd700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 0x7f0f28071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:45.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 20 client.41858.objecter dump_active .. 0 homeless 2024-03-11T21:38:45.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f448ee700 10 client.41858.objecter ms_handle_connect 0x7f0f2806f820 2024-03-11T21:38:45.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.775+0000 7f0f3effd700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 0x7f0f28071ce0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000b2e0 tx=0x7f0f30005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:45.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:45.779+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0f3805abe0 con 0x7f0f40107ed0 2024-03-11T21:38:45.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:45.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:45.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:45.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.003+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0f40075610 con 0x7f0f40107ed0 2024-03-11T21:38:46.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:46.100 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.547230+0000 osd.5 (osd.5) 377 : cluster 0 2.2 scrub starts 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.549758+0000 osd.5 (osd.5) 378 : cluster 0 2.2 scrub ok 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.943870+0000 osd.2 (osd.2) 319 : cluster 0 118.15 scrub starts 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:44.947920+0000 osd.2 (osd.2) 320 : cluster 0 118.15 scrub ok 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.464140+0000 mon.a (mon.0) 3180 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: audit 2024-03-11T21:38:45.466130+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.470881+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: audit 2024-03-11T21:38:45.474678+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: audit 2024-03-11T21:38:45.475278+0000 mon.a (mon.0) 3183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.478633+0000 osd.4 (osd.4) 353 : cluster 0 3.3 scrub starts 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.485758+0000 osd.4 (osd.4) 354 : cluster 0 3.3 scrub ok 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.547230+0000 osd.5 (osd.5) 377 : cluster 0 2.2 scrub starts 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.549758+0000 osd.5 (osd.5) 378 : cluster 0 2.2 scrub ok 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.943870+0000 osd.2 (osd.2) 319 : cluster 0 118.15 scrub starts 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:44.947920+0000 osd.2 (osd.2) 320 : cluster 0 118.15 scrub ok 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.464140+0000 mon.a (mon.0) 3180 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: audit 2024-03-11T21:38:45.466130+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.470881+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: audit 2024-03-11T21:38:45.474678+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: audit 2024-03-11T21:38:45.475278+0000 mon.a (mon.0) 3183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.478633+0000 osd.4 (osd.4) 353 : cluster 0 3.3 scrub starts 2024-03-11T21:38:46.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:45 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.485758+0000 osd.4 (osd.4) 354 : cluster 0 3.3 scrub ok 2024-03-11T21:38:46.203 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.547230+0000 osd.5 (osd.5) 377 : cluster 0 2.2 scrub starts 2024-03-11T21:38:46.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.549758+0000 osd.5 (osd.5) 378 : cluster 0 2.2 scrub ok 2024-03-11T21:38:46.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.943870+0000 osd.2 (osd.2) 319 : cluster 0 118.15 scrub starts 2024-03-11T21:38:46.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:44.947920+0000 osd.2 (osd.2) 320 : cluster 0 118.15 scrub ok 2024-03-11T21:38:46.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.464140+0000 mon.a (mon.0) 3180 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: audit 2024-03-11T21:38:45.466130+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.470881+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: audit 2024-03-11T21:38:45.474678+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: audit 2024-03-11T21:38:45.475278+0000 mon.a (mon.0) 3183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.478633+0000 osd.4 (osd.4) 353 : cluster 0 3.3 scrub starts 2024-03-11T21:38:46.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:45 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.485758+0000 osd.4 (osd.4) 354 : cluster 0 3.3 scrub ok 2024-03-11T21:38:46.230 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:46.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.359 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:46.463 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:46.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.487+0000 7f0f448ee700 1 -- 172.21.15.143:0/4137368833 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v639) v1 ==== 101+0+0 (secure 0 0 0) 0x7f0f40075610 con 0x7f0f40107ed0 2024-03-11T21:38:46.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.495+0000 7f0f458f0700 10 librados: watch_flush enter 2024-03-11T21:38:46.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 10 librados: watch_flush exit 2024-03-11T21:38:46.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 20 client.41858.objecter shutdown clearing up homeless session... 2024-03-11T21:38:46.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 10 client.41858.objecter successfully canceled tick 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 msgr2=0x7f0f28071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 0x7f0f28071ce0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000b2e0 tx=0x7f0f30005960 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 msgr2=0x7f0f40074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f40074b60 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800aa90 tx=0x7f0f3800ada0 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 shutdown_connections 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f2806f820 0x7f0f28071ce0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f40107ed0 0x7f0f40074b60 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f40101e90 0x7f0f400740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 --2- 172.21.15.143:0/4137368833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f40102830 0x7f0f40074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:46.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 >> 172.21.15.143:0/4137368833 conn(0x7f0f40078f90 msgr2=0x7f0f40105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:46.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 shutdown_connections 2024-03-11T21:38:46.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 -- 172.21.15.143:0/4137368833 wait complete. 2024-03-11T21:38:46.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:46.499+0000 7f0f458f0700 1 librados: shutdown 2024-03-11T21:38:46.481 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:38:46.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:46.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:46.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:46.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:46.746 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:46.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:46.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:46.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.421096+0000 osd.6 (osd.6) 281 : cluster 0 118.1d scrub starts 2024-03-11T21:38:46.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.426375+0000 osd.6 (osd.6) 282 : cluster 0 118.1d scrub ok 2024-03-11T21:38:46.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.525799+0000 mgr.y (mgr.14550) 619 : cluster 0 pgmap v988: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:46.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.729836+0000 osd.1 (osd.1) 161 : cluster 0 118.1b scrub starts 2024-03-11T21:38:46.991 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.733754+0000 osd.1 (osd.1) 162 : cluster 0 118.1b scrub ok 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.794084+0000 osd.7 (osd.7) 356 : cluster 0 39.13 scrub starts 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.796029+0000 osd.7 (osd.7) 357 : cluster 0 39.13 scrub ok 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.915767+0000 osd.3 (osd.3) 373 : cluster 0 39.d scrub starts 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:45.918740+0000 osd.3 (osd.3) 374 : cluster 0 39.d scrub ok 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.006419+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.143:0/4137368833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.006720+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.445273+0000 mon.a (mon.0) 3185 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.449508+0000 osd.4 (osd.4) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.450016+0000 osd.4 (osd.4) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.467502+0000 mon.a (mon.0) 3186 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.475729+0000 mon.a (mon.0) 3187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.476016+0000 mon.a (mon.0) 3188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.483689+0000 mon.a (mon.0) 3189 : cluster 0 osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:38:46.992 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.568834+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:46.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:46 smithi203 bash[23654]: audit 2024-03-11T21:38:46.569170+0000 mon.a (mon.0) 3190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:47.007 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:47.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:47.134 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.421096+0000 osd.6 (osd.6) 281 : cluster 0 118.1d scrub starts 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.426375+0000 osd.6 (osd.6) 282 : cluster 0 118.1d scrub ok 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.525799+0000 mgr.y (mgr.14550) 619 : cluster 0 pgmap v988: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.729836+0000 osd.1 (osd.1) 161 : cluster 0 118.1b scrub starts 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.733754+0000 osd.1 (osd.1) 162 : cluster 0 118.1b scrub ok 2024-03-11T21:38:47.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.794084+0000 osd.7 (osd.7) 356 : cluster 0 39.13 scrub starts 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.796029+0000 osd.7 (osd.7) 357 : cluster 0 39.13 scrub ok 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.915767+0000 osd.3 (osd.3) 373 : cluster 0 39.d scrub starts 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:45.918740+0000 osd.3 (osd.3) 374 : cluster 0 39.d scrub ok 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.006419+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.143:0/4137368833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.006720+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.445273+0000 mon.a (mon.0) 3185 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.449508+0000 osd.4 (osd.4) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.450016+0000 osd.4 (osd.4) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.467502+0000 mon.a (mon.0) 3186 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.475729+0000 mon.a (mon.0) 3187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.476016+0000 mon.a (mon.0) 3188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:47.190 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.483689+0000 mon.a (mon.0) 3189 : cluster 0 osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.568834+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[21512]: audit 2024-03-11T21:38:46.569170+0000 mon.a (mon.0) 3190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.421096+0000 osd.6 (osd.6) 281 : cluster 0 118.1d scrub starts 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.426375+0000 osd.6 (osd.6) 282 : cluster 0 118.1d scrub ok 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.525799+0000 mgr.y (mgr.14550) 619 : cluster 0 pgmap v988: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:47.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.729836+0000 osd.1 (osd.1) 161 : cluster 0 118.1b scrub starts 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.733754+0000 osd.1 (osd.1) 162 : cluster 0 118.1b scrub ok 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.794084+0000 osd.7 (osd.7) 356 : cluster 0 39.13 scrub starts 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.796029+0000 osd.7 (osd.7) 357 : cluster 0 39.13 scrub ok 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.915767+0000 osd.3 (osd.3) 373 : cluster 0 39.d scrub starts 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:45.918740+0000 osd.3 (osd.3) 374 : cluster 0 39.d scrub ok 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.006419+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.143:0/4137368833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.006720+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.445273+0000 mon.a (mon.0) 3185 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.449508+0000 osd.4 (osd.4) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.450016+0000 osd.4 (osd.4) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.467502+0000 mon.a (mon.0) 3186 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.475729+0000 mon.a (mon.0) 3187 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.476016+0000 mon.a (mon.0) 3188 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:47.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.483689+0000 mon.a (mon.0) 3189 : cluster 0 osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:38:47.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.568834+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:47.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:46 smithi143 bash[29284]: audit 2024-03-11T21:38:46.569170+0000 mon.a (mon.0) 3190 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:47.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:47.263 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:47.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:47.393 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:47.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.526 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:47.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:47.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:47.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:47.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:47.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:47.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:47.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:48.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.048 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:48.151 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:48.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.562161+0000 osd.5 (osd.5) 379 : cluster 0 6.19 scrub starts 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.563318+0000 osd.5 (osd.5) 380 : cluster 0 6.19 scrub ok 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.712228+0000 osd.1 (osd.1) 163 : cluster 0 6.1d scrub starts 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.714449+0000 osd.1 (osd.1) 164 : cluster 0 6.1d scrub ok 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.789396+0000 osd.7 (osd.7) 358 : cluster 0 39.5 scrub starts 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:46.791173+0000 osd.7 (osd.7) 359 : cluster 0 39.5 scrub ok 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: audit 2024-03-11T21:38:47.479437+0000 mon.a (mon.0) 3191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.488832+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: audit 2024-03-11T21:38:47.493174+0000 mon.c (mon.2) 1120 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:47 smithi203 bash[23654]: audit 2024-03-11T21:38:47.496360+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:48.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:48.433 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:48.438 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.562161+0000 osd.5 (osd.5) 379 : cluster 0 6.19 scrub starts 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.563318+0000 osd.5 (osd.5) 380 : cluster 0 6.19 scrub ok 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.712228+0000 osd.1 (osd.1) 163 : cluster 0 6.1d scrub starts 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.714449+0000 osd.1 (osd.1) 164 : cluster 0 6.1d scrub ok 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.789396+0000 osd.7 (osd.7) 358 : cluster 0 39.5 scrub starts 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:46.791173+0000 osd.7 (osd.7) 359 : cluster 0 39.5 scrub ok 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: audit 2024-03-11T21:38:47.479437+0000 mon.a (mon.0) 3191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.488832+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: audit 2024-03-11T21:38:47.493174+0000 mon.c (mon.2) 1120 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[29284]: audit 2024-03-11T21:38:47.496360+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.562161+0000 osd.5 (osd.5) 379 : cluster 0 6.19 scrub starts 2024-03-11T21:38:48.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.563318+0000 osd.5 (osd.5) 380 : cluster 0 6.19 scrub ok 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.712228+0000 osd.1 (osd.1) 163 : cluster 0 6.1d scrub starts 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.714449+0000 osd.1 (osd.1) 164 : cluster 0 6.1d scrub ok 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.789396+0000 osd.7 (osd.7) 358 : cluster 0 39.5 scrub starts 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:46.791173+0000 osd.7 (osd.7) 359 : cluster 0 39.5 scrub ok 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: audit 2024-03-11T21:38:47.479437+0000 mon.a (mon.0) 3191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]': finished 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.488832+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: audit 2024-03-11T21:38:47.493174+0000 mon.c (mon.2) 1120 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.440 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:47 smithi143 bash[21512]: audit 2024-03-11T21:38:47.496360+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]: dispatch 2024-03-11T21:38:48.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:48.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:48.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:48.695 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:48.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:48.825 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:48.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:49.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:49.088 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.377712+0000 osd.6 (osd.6) 283 : cluster 0 2.17 scrub starts 2024-03-11T21:38:49.088 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.379981+0000 osd.6 (osd.6) 284 : cluster 0 2.17 scrub ok 2024-03-11T21:38:49.088 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.526623+0000 mgr.y (mgr.14550) 620 : cluster 0 pgmap v991: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.712591+0000 osd.1 (osd.1) 165 : cluster 0 3.0 scrub starts 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:47.716241+0000 osd.1 (osd.1) 166 : cluster 0 3.0 scrub ok 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.486125+0000 osd.5 (osd.5) 381 : cluster 0 5.1b scrub starts 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.488264+0000 osd.5 (osd.5) 382 : cluster 0 5.1b scrub ok 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: audit 2024-03-11T21:38:48.488632+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:49.089 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:48 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.497212+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:38:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.377712+0000 osd.6 (osd.6) 283 : cluster 0 2.17 scrub starts 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.379981+0000 osd.6 (osd.6) 284 : cluster 0 2.17 scrub ok 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.526623+0000 mgr.y (mgr.14550) 620 : cluster 0 pgmap v991: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.712591+0000 osd.1 (osd.1) 165 : cluster 0 3.0 scrub starts 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:47.716241+0000 osd.1 (osd.1) 166 : cluster 0 3.0 scrub ok 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.486125+0000 osd.5 (osd.5) 381 : cluster 0 5.1b scrub starts 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.488264+0000 osd.5 (osd.5) 382 : cluster 0 5.1b scrub ok 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: audit 2024-03-11T21:38:48.488632+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:49.189 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.497212+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.377712+0000 osd.6 (osd.6) 283 : cluster 0 2.17 scrub starts 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.379981+0000 osd.6 (osd.6) 284 : cluster 0 2.17 scrub ok 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.526623+0000 mgr.y (mgr.14550) 620 : cluster 0 pgmap v991: 270 pgs: 270 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.712591+0000 osd.1 (osd.1) 165 : cluster 0 3.0 scrub starts 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:47.716241+0000 osd.1 (osd.1) 166 : cluster 0 3.0 scrub ok 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.486125+0000 osd.5 (osd.5) 381 : cluster 0 5.1b scrub starts 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.488264+0000 osd.5 (osd.5) 382 : cluster 0 5.1b scrub ok 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: audit 2024-03-11T21:38:48.488632+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-100"}]': finished 2024-03-11T21:38:49.190 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:48 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.497212+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:38:49.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:49.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.349 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:49.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:49.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:49.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:49.616 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:49.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:38:49.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:49.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:49.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:50.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:50.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:50.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.403296+0000 osd.6 (osd.6) 285 : cluster 0 2.7 scrub starts 2024-03-11T21:38:50.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.406183+0000 osd.6 (osd.6) 286 : cluster 0 2.7 scrub ok 2024-03-11T21:38:50.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.703308+0000 osd.1 (osd.1) 167 : cluster 0 9.4 scrub starts 2024-03-11T21:38:50.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.705069+0000 osd.1 (osd.1) 168 : cluster 0 9.4 scrub ok 2024-03-11T21:38:50.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.791503+0000 osd.7 (osd.7) 360 : cluster 0 2.0 scrub starts 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:48.793396+0000 osd.7 (osd.7) 361 : cluster 0 2.0 scrub ok 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: audit 2024-03-11T21:38:48.854577+0000 mgr.y (mgr.14550) 621 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[29284]: cluster 2024-03-11T21:38:49.500359+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.403296+0000 osd.6 (osd.6) 285 : cluster 0 2.7 scrub starts 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.406183+0000 osd.6 (osd.6) 286 : cluster 0 2.7 scrub ok 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.703308+0000 osd.1 (osd.1) 167 : cluster 0 9.4 scrub starts 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.705069+0000 osd.1 (osd.1) 168 : cluster 0 9.4 scrub ok 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.791503+0000 osd.7 (osd.7) 360 : cluster 0 2.0 scrub starts 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:48.793396+0000 osd.7 (osd.7) 361 : cluster 0 2.0 scrub ok 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: audit 2024-03-11T21:38:48.854577+0000 mgr.y (mgr.14550) 621 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:50.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:49 smithi143 bash[21512]: cluster 2024-03-11T21:38:49.500359+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:38:50.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:50.189 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:50.189 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:50.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.403296+0000 osd.6 (osd.6) 285 : cluster 0 2.7 scrub starts 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.406183+0000 osd.6 (osd.6) 286 : cluster 0 2.7 scrub ok 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.703308+0000 osd.1 (osd.1) 167 : cluster 0 9.4 scrub starts 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.705069+0000 osd.1 (osd.1) 168 : cluster 0 9.4 scrub ok 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.791503+0000 osd.7 (osd.7) 360 : cluster 0 2.0 scrub starts 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:48.793396+0000 osd.7 (osd.7) 361 : cluster 0 2.0 scrub ok 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: audit 2024-03-11T21:38:48.854577+0000 mgr.y (mgr.14550) 621 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:50.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:49 smithi203 bash[23654]: cluster 2024-03-11T21:38:49.500359+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:38:50.263 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:50.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:50.394 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:50.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:50.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:50.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:50.655 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:50.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:50.786 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:50.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:50.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:51.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:49.527187+0000 mgr.y (mgr.14550) 622 : cluster 0 pgmap v994: 238 pgs: 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:49.927010+0000 osd.3 (osd.3) 375 : cluster 0 3.18 scrub starts 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:49.930658+0000 osd.3 (osd.3) 376 : cluster 0 3.18 scrub ok 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.408172+0000 osd.4 (osd.4) 357 : cluster 0 39.10 scrub starts 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.410038+0000 osd.4 (osd.4) 358 : cluster 0 39.10 scrub ok 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.509062+0000 mon.a (mon.0) 3197 : cluster 0 osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: audit 2024-03-11T21:38:50.524198+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: audit 2024-03-11T21:38:50.525839+0000 mon.a (mon.0) 3198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.689563+0000 osd.0 (osd.0) 356 : cluster 0 6.9 scrub starts 2024-03-11T21:38:51.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:50 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.691847+0000 osd.0 (osd.0) 357 : cluster 0 6.9 scrub ok 2024-03-11T21:38:51.045 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:51.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:51.168 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:51.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:51.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:51.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:49.527187+0000 mgr.y (mgr.14550) 622 : cluster 0 pgmap v994: 238 pgs: 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:51.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:49.927010+0000 osd.3 (osd.3) 375 : cluster 0 3.18 scrub starts 2024-03-11T21:38:51.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:49.930658+0000 osd.3 (osd.3) 376 : cluster 0 3.18 scrub ok 2024-03-11T21:38:51.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.408172+0000 osd.4 (osd.4) 357 : cluster 0 39.10 scrub starts 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.410038+0000 osd.4 (osd.4) 358 : cluster 0 39.10 scrub ok 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.509062+0000 mon.a (mon.0) 3197 : cluster 0 osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: audit 2024-03-11T21:38:50.524198+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: audit 2024-03-11T21:38:50.525839+0000 mon.a (mon.0) 3198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.689563+0000 osd.0 (osd.0) 356 : cluster 0 6.9 scrub starts 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.691847+0000 osd.0 (osd.0) 357 : cluster 0 6.9 scrub ok 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:49.527187+0000 mgr.y (mgr.14550) 622 : cluster 0 pgmap v994: 238 pgs: 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:49.927010+0000 osd.3 (osd.3) 375 : cluster 0 3.18 scrub starts 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:49.930658+0000 osd.3 (osd.3) 376 : cluster 0 3.18 scrub ok 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.408172+0000 osd.4 (osd.4) 357 : cluster 0 39.10 scrub starts 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.410038+0000 osd.4 (osd.4) 358 : cluster 0 39.10 scrub ok 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.509062+0000 mon.a (mon.0) 3197 : cluster 0 osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:38:51.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: audit 2024-03-11T21:38:50.524198+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: audit 2024-03-11T21:38:50.525839+0000 mon.a (mon.0) 3198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:38:51.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.689563+0000 osd.0 (osd.0) 356 : cluster 0 6.9 scrub starts 2024-03-11T21:38:51.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:50 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.691847+0000 osd.0 (osd.0) 357 : cluster 0 6.9 scrub ok 2024-03-11T21:38:51.427 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:51.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:51.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:51.658 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.686 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:51.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:51.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:51.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:51.940 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:52.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.070 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.392063+0000 osd.6 (osd.6) 287 : cluster 0 9.18 scrub starts 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.393856+0000 osd.6 (osd.6) 288 : cluster 0 9.18 scrub ok 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.513609+0000 osd.5 (osd.5) 383 : cluster 0 9.7 scrub starts 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: cluster 2024-03-11T21:38:50.523909+0000 osd.5 (osd.5) 384 : cluster 0 9.7 scrub ok 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: audit 2024-03-11T21:38:51.450083+0000 mon.a (mon.0) 3199 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:52.083 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:51 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.455627+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: audit 2024-03-11T21:38:51.463792+0000 mon.c (mon.2) 1122 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: audit 2024-03-11T21:38:51.481536+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: audit 2024-03-11T21:38:51.482792+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.489204+0000 osd.5 (osd.5) 385 : cluster 0 9.12 scrub starts 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.492581+0000 osd.5 (osd.5) 386 : cluster 0 9.12 scrub ok 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.703374+0000 osd.0 (osd.0) 358 : cluster 0 9.1d scrub starts 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.705197+0000 osd.0 (osd.0) 359 : cluster 0 9.1d scrub ok 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: audit 2024-03-11T21:38:51.933658+0000 mon.a (mon.0) 3202 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21512]: audit 2024-03-11T21:38:51.937763+0000 mon.c (mon.2) 1124 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.392063+0000 osd.6 (osd.6) 287 : cluster 0 9.18 scrub starts 2024-03-11T21:38:52.084 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.393856+0000 osd.6 (osd.6) 288 : cluster 0 9.18 scrub ok 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.513609+0000 osd.5 (osd.5) 383 : cluster 0 9.7 scrub starts 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:50.523909+0000 osd.5 (osd.5) 384 : cluster 0 9.7 scrub ok 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.450083+0000 mon.a (mon.0) 3199 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.455627+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.463792+0000 mon.c (mon.2) 1122 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.481536+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.482792+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.489204+0000 osd.5 (osd.5) 385 : cluster 0 9.12 scrub starts 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.492581+0000 osd.5 (osd.5) 386 : cluster 0 9.12 scrub ok 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.703374+0000 osd.0 (osd.0) 358 : cluster 0 9.1d scrub starts 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.705197+0000 osd.0 (osd.0) 359 : cluster 0 9.1d scrub ok 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.933658+0000 mon.a (mon.0) 3202 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:52.085 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[29284]: audit 2024-03-11T21:38:51.937763+0000 mon.c (mon.2) 1124 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:52.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.392063+0000 osd.6 (osd.6) 287 : cluster 0 9.18 scrub starts 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.393856+0000 osd.6 (osd.6) 288 : cluster 0 9.18 scrub ok 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.513609+0000 osd.5 (osd.5) 383 : cluster 0 9.7 scrub starts 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:50.523909+0000 osd.5 (osd.5) 384 : cluster 0 9.7 scrub ok 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.450083+0000 mon.a (mon.0) 3199 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:38:52.184 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.455627+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.463792+0000 mon.c (mon.2) 1122 : audit 0 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.481536+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.482792+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.489204+0000 osd.5 (osd.5) 385 : cluster 0 9.12 scrub starts 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.492581+0000 osd.5 (osd.5) 386 : cluster 0 9.12 scrub ok 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.703374+0000 osd.0 (osd.0) 358 : cluster 0 9.1d scrub starts 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.705197+0000 osd.0 (osd.0) 359 : cluster 0 9.1d scrub ok 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.933658+0000 mon.a (mon.0) 3202 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:52.185 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:51 smithi203 bash[23654]: audit 2024-03-11T21:38:51.937763+0000 mon.c (mon.2) 1124 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:52.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:52.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:52.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:52.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.438 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:38:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:38:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:52.458 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:52.560 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:52.587 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:52.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.717 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:52.820 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:52.849 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:52.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.977 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.433966+0000 osd.6 (osd.6) 289 : cluster 0 2.1a scrub starts 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.435740+0000 osd.6 (osd.6) 290 : cluster 0 2.1a scrub ok 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:51.527664+0000 mgr.y (mgr.14550) 623 : cluster 0 pgmap v997: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:52.367013+0000 osd.4 (osd.4) 359 : cluster 0 6.e scrub starts 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:52.368938+0000 osd.4 (osd.4) 360 : cluster 0 6.e scrub ok 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: audit 2024-03-11T21:38:52.454401+0000 mon.a (mon.0) 3203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:52.464004+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: audit 2024-03-11T21:38:52.467662+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: audit 2024-03-11T21:38:52.469438+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:52.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:52.711048+0000 osd.0 (osd.0) 360 : cluster 0 118.19 scrub starts 2024-03-11T21:38:52.981 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[29284]: cluster 2024-03-11T21:38:52.720484+0000 osd.0 (osd.0) 361 : cluster 0 118.19 scrub ok 2024-03-11T21:38:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:53.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:53.109 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:53.169 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:53.169 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.433966+0000 osd.6 (osd.6) 289 : cluster 0 2.1a scrub starts 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.435740+0000 osd.6 (osd.6) 290 : cluster 0 2.1a scrub ok 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:51.527664+0000 mgr.y (mgr.14550) 623 : cluster 0 pgmap v997: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:52.367013+0000 osd.4 (osd.4) 359 : cluster 0 6.e scrub starts 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:52.368938+0000 osd.4 (osd.4) 360 : cluster 0 6.e scrub ok 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: audit 2024-03-11T21:38:52.454401+0000 mon.a (mon.0) 3203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:52.464004+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:38:53.207 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: audit 2024-03-11T21:38:52.467662+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:53.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: audit 2024-03-11T21:38:52.469438+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:53.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:52.711048+0000 osd.0 (osd.0) 360 : cluster 0 118.19 scrub starts 2024-03-11T21:38:53.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:52 smithi203 bash[23654]: cluster 2024-03-11T21:38:52.720484+0000 osd.0 (osd.0) 361 : cluster 0 118.19 scrub ok 2024-03-11T21:38:53.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:53.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.433966+0000 osd.6 (osd.6) 289 : cluster 0 2.1a scrub starts 2024-03-11T21:38:53.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.435740+0000 osd.6 (osd.6) 290 : cluster 0 2.1a scrub ok 2024-03-11T21:38:53.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:51.527664+0000 mgr.y (mgr.14550) 623 : cluster 0 pgmap v997: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:52.367013+0000 osd.4 (osd.4) 359 : cluster 0 6.e scrub starts 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:52.368938+0000 osd.4 (osd.4) 360 : cluster 0 6.e scrub ok 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: audit 2024-03-11T21:38:52.454401+0000 mon.a (mon.0) 3203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:52.464004+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: audit 2024-03-11T21:38:52.467662+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: audit 2024-03-11T21:38:52.469438+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:52.711048+0000 osd.0 (osd.0) 360 : cluster 0 118.19 scrub starts 2024-03-11T21:38:53.224 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:53 smithi143 bash[21512]: cluster 2024-03-11T21:38:52.720484+0000 osd.0 (osd.0) 361 : cluster 0 118.19 scrub ok 2024-03-11T21:38:53.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:53.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:53.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:53.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:53.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:53.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:53.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:53.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:53.754 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:53.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:53.881 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:53.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:38:53.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.007 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:54.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.131 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:54.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: audit 2024-03-11T21:38:53.446561+0000 mon.c (mon.2) 1126 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: audit 2024-03-11T21:38:53.458558+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.466418+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: audit 2024-03-11T21:38:53.472663+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: audit 2024-03-11T21:38:53.473624+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.687210+0000 osd.0 (osd.0) 362 : cluster 0 118.1a scrub starts 2024-03-11T21:38:54.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:53 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.689438+0000 osd.0 (osd.0) 363 : cluster 0 118.1a scrub ok 2024-03-11T21:38:54.258 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:54.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:54.373 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: audit 2024-03-11T21:38:53.446561+0000 mon.c (mon.2) 1126 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: audit 2024-03-11T21:38:53.458558+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.466418+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: audit 2024-03-11T21:38:53.472663+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: audit 2024-03-11T21:38:53.473624+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.687210+0000 osd.0 (osd.0) 362 : cluster 0 118.1a scrub starts 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.689438+0000 osd.0 (osd.0) 363 : cluster 0 118.1a scrub ok 2024-03-11T21:38:54.374 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: audit 2024-03-11T21:38:53.446561+0000 mon.c (mon.2) 1126 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: audit 2024-03-11T21:38:53.458558+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.466418+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: audit 2024-03-11T21:38:53.472663+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: audit 2024-03-11T21:38:53.473624+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.687210+0000 osd.0 (osd.0) 362 : cluster 0 118.1a scrub starts 2024-03-11T21:38:54.375 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:54 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.689438+0000 osd.0 (osd.0) 363 : cluster 0 118.1a scrub ok 2024-03-11T21:38:54.388 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:54.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.510 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:54.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:54.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:54.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:54.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:54.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:54.882 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:54.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:55.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:55.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.529577+0000 mgr.y (mgr.14550) 624 : cluster 0 pgmap v1000: 270 pgs: 6 unknown, 264 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:55.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.637136+0000 osd.1 (osd.1) 169 : cluster 0 118.1f scrub starts 2024-03-11T21:38:55.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:53.641747+0000 osd.1 (osd.1) 170 : cluster 0 118.1f scrub ok 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.186005+0000 mon.c (mon.2) 1128 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.187529+0000 mon.c (mon.2) 1129 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.198018+0000 mon.a (mon.0) 3209 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:54.360514+0000 osd.4 (osd.4) 361 : cluster 0 39.b scrub starts 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:54.363396+0000 osd.4 (osd.4) 362 : cluster 0 39.b scrub ok 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.469371+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: cluster 2024-03-11T21:38:54.473230+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.545690+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.546167+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.546864+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:55 smithi203 bash[23654]: audit 2024-03-11T21:38:54.547258+0000 mon.a (mon.0) 3213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:55.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:55.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.529577+0000 mgr.y (mgr.14550) 624 : cluster 0 pgmap v1000: 270 pgs: 6 unknown, 264 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:55.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.637136+0000 osd.1 (osd.1) 169 : cluster 0 118.1f scrub starts 2024-03-11T21:38:55.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:53.641747+0000 osd.1 (osd.1) 170 : cluster 0 118.1f scrub ok 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.186005+0000 mon.c (mon.2) 1128 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.187529+0000 mon.c (mon.2) 1129 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.198018+0000 mon.a (mon.0) 3209 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:54.360514+0000 osd.4 (osd.4) 361 : cluster 0 39.b scrub starts 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:54.363396+0000 osd.4 (osd.4) 362 : cluster 0 39.b scrub ok 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.469371+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: cluster 2024-03-11T21:38:54.473230+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.545690+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.546167+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.546864+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.156 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[21512]: audit 2024-03-11T21:38:54.547258+0000 mon.a (mon.0) 3213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.529577+0000 mgr.y (mgr.14550) 624 : cluster 0 pgmap v1000: 270 pgs: 6 unknown, 264 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.637136+0000 osd.1 (osd.1) 169 : cluster 0 118.1f scrub starts 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:53.641747+0000 osd.1 (osd.1) 170 : cluster 0 118.1f scrub ok 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.186005+0000 mon.c (mon.2) 1128 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.187529+0000 mon.c (mon.2) 1129 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.198018+0000 mon.a (mon.0) 3209 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:54.360514+0000 osd.4 (osd.4) 361 : cluster 0 39.b scrub starts 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:54.363396+0000 osd.4 (osd.4) 362 : cluster 0 39.b scrub ok 2024-03-11T21:38:55.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.469371+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:38:55.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: cluster 2024-03-11T21:38:54.473230+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:38:55.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.545690+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.546167+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-6"}]: dispatch 2024-03-11T21:38:55.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.546864+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.143:0/1735581' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.158 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:55 smithi143 bash[29284]: audit 2024-03-11T21:38:54.547258+0000 mon.a (mon.0) 3213 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-6", "tierpool": "test-rados-api-smithi143-71773-102"}]: dispatch 2024-03-11T21:38:55.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:55.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:55.303 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:55.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:55.433 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:55.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:55.561 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:55.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:55.691 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:55.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:55.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:55.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:55.950 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:56.052 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.081 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:56.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:56.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:56.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 -- 172.21.15.143:0/2308554581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a720 msgr2=0x7f9df4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/2308554581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a720 0x7f9df4106a90 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f9df00099c0 tx=0x7f9df0009cd0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 -- 172.21.15.143:0/2308554581 shutdown_connections 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/2308554581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df4069d70 0x7f9df406a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/2308554581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df41071c0 0x7f9df410ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/2308554581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a720 0x7f9df4106a90 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 -- 172.21.15.143:0/2308554581 >> 172.21.15.143:0/2308554581 conn(0x7f9df4076d80 msgr2=0x7f9df4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 -- 172.21.15.143:0/2308554581 shutdown_connections 2024-03-11T21:38:56.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.354+0000 7f9dfb4de700 1 -- 172.21.15.143:0/2308554581 wait complete. 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 librados: starting msgr at 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 librados: starting objecter 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 Processor -- start 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- start start 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 librados: setting wanted keys 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 librados: calling monclient init 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4069d70 0x7f9df4100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 0x7f9df4101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4111630 con 0x7f9df406a720 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41114b0 con 0x7f9df4069d70 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41117b0 con 0x7f9df41071c0 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df9a7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 0x7f9df4101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34876/0 (socket says 172.21.15.143:34876) 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 -- 172.21.15.143:0/4158121331 learned_addr learned my addr 172.21.15.143:0/4158121331 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df927a700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4069d70 0x7f9df4100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 -- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 msgr2=0x7f9df4101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 0x7f9df4101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 -- 172.21.15.143:0/4158121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4069d70 msgr2=0x7f9df4100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4069d70 0x7f9df4100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 -- 172.21.15.143:0/4158121331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df41a6100 con 0x7f9df406a720 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df9a7b700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 0x7f9df4101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9df8a79700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f9df000fa00 tx=0x7f9df000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 10 client.?.objecter ms_handle_connect 0x7f9df406a720 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0009e60 con 0x7f9df406a720 2024-03-11T21:38:56.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df41a6390 con 0x7f9df406a720 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9df0005550 con 0x7f9df406a720 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df000bea0 con 0x7f9df406a720 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 10 client.30660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df41a67e0 con 0x7f9df406a720 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 librados: init done 2024-03-11T21:38:56.340 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:56.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df404faa0 con 0x7f9df406a720 2024-03-11T21:38:56.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9df00050d0 con 0x7f9df406a720 2024-03-11T21:38:56.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.358+0000 7f9dea7fc700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(648..648 src has 87..648) v4 ==== 9276+0+0 (secure 0 0 0) 0x7f9df008fa30 con 0x7f9df406a720 2024-03-11T21:38:56.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 10 client.30660.objecter ms_dispatch 0x7f9df4001550 osd_map(648..648 src has 87..648) v4 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 3 client.30660.objecter handle_osd_map got epochs [648,648] > 0 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 3 client.30660.objecter handle_osd_map decoding full epoch 648 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9df927a700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 20 client.30660.objecter dump_active .. 0 homeless 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9dea7fc700 10 client.30660.objecter ms_handle_connect 0x7f9de006f820 2024-03-11T21:38:56.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.362+0000 7f9df927a700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009680 tx=0x7f9de4009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:56.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.366+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9df0013070 con 0x7f9df406a720 2024-03-11T21:38:56.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:56.470 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:56.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:56.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.610+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]} v 0) v1 -- 0x7f9df41a6a40 con 0x7f9df406a720 2024-03-11T21:38:56.594 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 -- 172.21.15.143:0/2387398429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648106dc0 msgr2=0x7fd6481071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 --2- 172.21.15.143:0/2387398429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648106dc0 0x7fd6481071a0 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fd638009990 tx=0x7fd638009ca0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 -- 172.21.15.143:0/2387398429 shutdown_connections 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 --2- 172.21.15.143:0/2387398429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648101700 0x7fd64810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 --2- 172.21.15.143:0/2387398429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648106dc0 0x7fd6481071a0 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 --2- 172.21.15.143:0/2387398429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd648100d40 0x7fd6481011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.650+0000 7fd64cc6e700 1 -- 172.21.15.143:0/2387398429 >> 172.21.15.143:0/2387398429 conn(0x7fd648076d80 msgr2=0x7fd648077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- 172.21.15.143:0/2387398429 shutdown_connections 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- 172.21.15.143:0/2387398429 wait complete. 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 librados: starting msgr at 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 librados: starting objecter 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 Processor -- start 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- start start 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 librados: setting wanted keys 2024-03-11T21:38:56.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 librados: calling monclient init 2024-03-11T21:38:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648100d40 0x7fd6481029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd648101700 0x7fd648102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648110530 con 0x7fd648106dc0 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6481103b0 con 0x7fd648101700 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6481106b0 con 0x7fd648100d40 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648100d40 0x7fd6481029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34920/0 (socket says 172.21.15.143:34920) 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 learned_addr learned my addr 172.21.15.143:0/1901908568 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd648101700 0x7fd648102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648100d40 msgr2=0x7fd6481029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd648100d40 0x7fd6481029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd648101700 msgr2=0x7fd648102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.635 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd648101700 0x7fd648102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6481aa500 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fd64000c370 tx=0x7fd64000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 10 client.?.objecter ms_handle_connect 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64000ce20 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6481aa790 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd6400090d0 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400046b0 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 10 client.30672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6481aacd0 con 0x7fd648106dc0 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd64cc6e700 1 librados: init done 2024-03-11T21:38:56.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd62c005100 con 0x7fd648106dc0 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.654+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd64001a020 con 0x7fd648106dc0 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 0x7fd630071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(649..649 src has 87..649) v4 ==== 8876+0+0 (secure 0 0 0) 0x7fd64008e0d0 con 0x7fd648106dc0 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 osd_map(649..649 src has 87..649) v4 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 3 client.30672.objecter handle_osd_map got epochs [649,649] > 0 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 3 client.30672.objecter handle_osd_map decoding full epoch 649 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 0x7fd630071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 20 client.30672.objecter dump_active .. 0 homeless 2024-03-11T21:38:56.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v649) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd640010040 con 0x7fd648106dc0 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 10 client.30672.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 1 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd62c004730 0x7fd62c004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd62c004730 0x7fd62c004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 1 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd63006f820 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 0x7fd630071ce0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fd63800b2e0 tx=0x7fd638005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 0 1 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 4 2024-03-11T21:38:56.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 10 client.30672.objecter _send_command 1 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c008b70 con 0x7fd62c004730 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c004730 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd62c004730 0x7fd62c004b70 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c008b70 con 0x7fd62c004730 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 10 client.30672.objecter _finish_command 1 = system:0 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 4 2024-03-11T21:38:56.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.658+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 0 1 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 10 client.30672.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 2 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 2 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c0053e0 osd=0 3 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 0 2 2024-03-11T21:38:56.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 4 2024-03-11T21:38:56.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 10 client.30672.objecter _send_command 2 2024-03-11T21:38:56.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c0042a0 con 0x7fd62c004730 2024-03-11T21:38:56.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c0042a0 con 0x7fd62c004730 2024-03-11T21:38:56.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 10 client.30672.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 4 2024-03-11T21:38:56.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 0 2 2024-03-11T21:38:56.659 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 10 client.30672.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd62c002750 0x7fd62c002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 1 3 2024-03-11T21:38:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 4 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 10 client.30672.objecter _send_command 3 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c003110 con 0x7fd62c002750 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd62c002750 0x7fd62c002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c002750 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.678+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd62c002750 0x7fd62c002bd0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.682+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c003110 con 0x7fd62c002750 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.682+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.682+0000 7fd6377fe700 10 client.30672.objecter _finish_command 3 = system:0 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.682+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 4 2024-03-11T21:38:56.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.682+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 1 3 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 10 client.30672.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 4 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 4 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c004530 osd=1 3 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 1 4 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 4 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 10 client.30672.objecter _send_command 4 2024-03-11T21:38:56.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.698+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c003b20 con 0x7fd62c002750 2024-03-11T21:38:56.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c003b20 con 0x7fd62c002750 2024-03-11T21:38:56.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 10 client.30672.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 4 2024-03-11T21:38:56.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 1 4 2024-03-11T21:38:56.681 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 10 client.30672.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 5 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd62c00b240 0x7fd62c00d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 5 2024-03-11T21:38:56.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 2 5 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 4 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 10 client.30672.objecter _send_command 5 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c010520 con 0x7fd62c00b240 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd62c00b240 0x7fd62c00d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c00b240 2024-03-11T21:38:56.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd62c00b240 0x7fd62c00d630 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c010520 con 0x7fd62c00b240 2024-03-11T21:38:56.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:38:56.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 10 client.30672.objecter _finish_command 5 = system:0 2024-03-11T21:38:56.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 4 2024-03-11T21:38:56.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.702+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 2 5 2024-03-11T21:38:56.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:56.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 10 client.30672.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 6 2024-03-11T21:38:56.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 6 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c003d80 osd=2 3 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 2 6 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 4 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 10 client.30672.objecter _send_command 6 2024-03-11T21:38:56.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c003600 con 0x7fd62c00b240 2024-03-11T21:38:56.702 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c003600 con 0x7fd62c00b240 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 10 client.30672.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 4 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 2 6 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 10 client.30672.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 7 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd62c002090 0x7fd62c00ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 7 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 3 7 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 4 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 10 client.30672.objecter _send_command 7 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c013f20 con 0x7fd62c002090 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd62c002090 0x7fd62c00ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c002090 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd62c002090 0x7fd62c00ecf0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c013f20 con 0x7fd62c002090 2024-03-11T21:38:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:38:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 10 client.30672.objecter _finish_command 7 = system:0 2024-03-11T21:38:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 4 2024-03-11T21:38:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.722+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 3 7 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.433028+0000 osd.5 (osd.5) 387 : cluster 0 3.1c scrub starts 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.437611+0000 osd.5 (osd.5) 388 : cluster 0 3.1c scrub ok 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.477173+0000 mon.a (mon.0) 3214 : cluster 0 osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.530364+0000 mgr.y (mgr.14550) 625 : cluster 0 pgmap v1003: 238 pgs: 238 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.653943+0000 osd.1 (osd.1) 171 : cluster 0 9.14 scrub starts 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.658876+0000 osd.1 (osd.1) 172 : cluster 0 9.14 scrub ok 2024-03-11T21:38:56.705 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.796747+0000 osd.2 (osd.2) 321 : cluster 0 39.6 scrub starts 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[21512]: cluster 2024-03-11T21:38:55.800013+0000 osd.2 (osd.2) 322 : cluster 0 39.6 scrub ok 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.433028+0000 osd.5 (osd.5) 387 : cluster 0 3.1c scrub starts 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.437611+0000 osd.5 (osd.5) 388 : cluster 0 3.1c scrub ok 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.477173+0000 mon.a (mon.0) 3214 : cluster 0 osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.530364+0000 mgr.y (mgr.14550) 625 : cluster 0 pgmap v1003: 238 pgs: 238 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.653943+0000 osd.1 (osd.1) 171 : cluster 0 9.14 scrub starts 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.658876+0000 osd.1 (osd.1) 172 : cluster 0 9.14 scrub ok 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.796747+0000 osd.2 (osd.2) 321 : cluster 0 39.6 scrub starts 2024-03-11T21:38:56.706 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:56 smithi143 bash[29284]: cluster 2024-03-11T21:38:55.800013+0000 osd.2 (osd.2) 322 : cluster 0 39.6 scrub ok 2024-03-11T21:38:56.715 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 10 client.30672.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 8 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 8 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c003600 osd=3 3 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 3 8 2024-03-11T21:38:56.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 4 2024-03-11T21:38:56.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 10 client.30672.objecter _send_command 8 2024-03-11T21:38:56.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.742+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c001720 con 0x7fd62c002090 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c001720 con 0x7fd62c002090 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 10 client.30672.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 4 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 3 8 2024-03-11T21:38:56.724 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 10 client.30672.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 9 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd62c00e850 0x7fd62c012740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 9 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 4 9 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 4 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 10 client.30672.objecter _send_command 9 2024-03-11T21:38:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c013e00 con 0x7fd62c00e850 2024-03-11T21:38:56.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd62c00e850 0x7fd62c012740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c00e850 2024-03-11T21:38:56.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd62c00e850 0x7fd62c012740 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c013e00 con 0x7fd62c00e850 2024-03-11T21:38:56.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:38:56.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 10 client.30672.objecter _finish_command 9 = system:0 2024-03-11T21:38:56.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 4 2024-03-11T21:38:56.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.746+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 4 9 2024-03-11T21:38:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 10 client.30672.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 10 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 10 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c001720 osd=4 3 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 4 10 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 4 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 10 client.30672.objecter _send_command 10 2024-03-11T21:38:56.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.766+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c012c80 con 0x7fd62c00e850 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c012c80 con 0x7fd62c00e850 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 10 client.30672.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 4 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 4 10 2024-03-11T21:38:56.751 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 10 client.30672.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 11 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd62c0133a0 0x7fd62c013800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 11 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 5 11 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 4 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 10 client.30672.objecter _send_command 11 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c013dd0 con 0x7fd62c0133a0 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd62c0133a0 0x7fd62c013800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c0133a0 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.770+0000 7fd645d9b700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd62c0133a0 0x7fd62c013800 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.774+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c013dd0 con 0x7fd62c0133a0 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.774+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:38:56.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.774+0000 7fd6377fe700 10 client.30672.objecter _finish_command 11 = system:0 2024-03-11T21:38:56.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.774+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 4 2024-03-11T21:38:56.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.774+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 5 11 2024-03-11T21:38:56.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 10 client.30672.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 12 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 12 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c012c80 osd=5 3 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 5 12 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 4 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 10 client.30672.objecter _send_command 12 2024-03-11T21:38:56.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.794+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c013dd0 con 0x7fd62c0133a0 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.433028+0000 osd.5 (osd.5) 387 : cluster 0 3.1c scrub starts 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.437611+0000 osd.5 (osd.5) 388 : cluster 0 3.1c scrub ok 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.477173+0000 mon.a (mon.0) 3214 : cluster 0 osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.530364+0000 mgr.y (mgr.14550) 625 : cluster 0 pgmap v1003: 238 pgs: 238 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.653943+0000 osd.1 (osd.1) 171 : cluster 0 9.14 scrub starts 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.658876+0000 osd.1 (osd.1) 172 : cluster 0 9.14 scrub ok 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.796747+0000 osd.2 (osd.2) 321 : cluster 0 39.6 scrub starts 2024-03-11T21:38:56.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:56 smithi203 bash[23654]: cluster 2024-03-11T21:38:55.800013+0000 osd.2 (osd.2) 322 : cluster 0 39.6 scrub ok 2024-03-11T21:38:56.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c013dd0 con 0x7fd62c0133a0 2024-03-11T21:38:56.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 10 client.30672.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 4 2024-03-11T21:38:56.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 5 12 2024-03-11T21:38:56.778 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:38:56.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 10 client.30672.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 13 2024-03-11T21:38:56.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd62c0186f0 0x7fd62c01ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 13 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 6 13 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 4 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 10 client.30672.objecter _send_command 13 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c01db70 con 0x7fd62c0186f0 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd62c0186f0 0x7fd62c01ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c0186f0 2024-03-11T21:38:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.798+0000 7fd646d9d700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd62c0186f0 0x7fd62c01ab20 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.802+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c01db70 con 0x7fd62c0186f0 2024-03-11T21:38:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.802+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:38:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.802+0000 7fd6377fe700 10 client.30672.objecter _finish_command 13 = system:0 2024-03-11T21:38:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.802+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 4 2024-03-11T21:38:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.802+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 6 13 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 10 client.30672.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 14 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 14 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c013dd0 osd=6 3 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 6 14 2024-03-11T21:38:56.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 4 2024-03-11T21:38:56.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 10 client.30672.objecter _send_command 14 2024-03-11T21:38:56.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.818+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c01db70 con 0x7fd62c0186f0 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c01db70 con 0x7fd62c0186f0 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 10 client.30672.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 4 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 6 14 2024-03-11T21:38:56.802 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 10 client.30672.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 15 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd62c01cea0 0x7fd62c01d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 15 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 7 15 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 4 2024-03-11T21:38:56.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 10 client.30672.objecter _send_command 15 2024-03-11T21:38:56.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c020f10 con 0x7fd62c01cea0 2024-03-11T21:38:56.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd62c01cea0 0x7fd62c01d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:38:56.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd6377fe700 10 client.30672.objecter ms_handle_connect 0x7fd62c01cea0 2024-03-11T21:38:56.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.822+0000 7fd64659c700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd62c01cea0 0x7fd62c01d320 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:38:56.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.826+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd62c020f10 con 0x7fd62c01cea0 2024-03-11T21:38:56.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.826+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:38:56.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.826+0000 7fd6377fe700 10 client.30672.objecter _finish_command 15 = system:0 2024-03-11T21:38:56.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.826+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 4 2024-03-11T21:38:56.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.826+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 7 15 2024-03-11T21:38:56.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 10 client.30672.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign -1 16 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 20 client.30672.objecter _get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_remove -1 16 2024-03-11T21:38:56.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 20 client.30672.objecter get_session s=0x7fd62c01db70 osd=7 3 2024-03-11T21:38:56.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 15 client.30672.objecter _session_command_op_assign 7 16 2024-03-11T21:38:56.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 4 2024-03-11T21:38:56.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 10 client.30672.objecter _send_command 16 2024-03-11T21:38:56.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.842+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd62c01fd90 con 0x7fd62c01cea0 2024-03-11T21:38:56.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6377fe700 1 -- 172.21.15.143:0/1901908568 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd62c01fd90 con 0x7fd62c01cea0 2024-03-11T21:38:56.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6377fe700 10 client.30672.objecter ms_dispatch 0x7fd648001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:38:56.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6377fe700 10 client.30672.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:38:56.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6377fe700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 4 2024-03-11T21:38:56.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6377fe700 15 client.30672.objecter _session_command_op_remove 7 16 2024-03-11T21:38:56.827 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 librados: watch_flush enter 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 librados: watch_flush exit 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.0 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd62c004730 msgr2=0x7fd62c004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd62c004730 0x7fd62c004b70 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c0053e0 osd=0 1 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.1 2024-03-11T21:38:56.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd62c002750 msgr2=0x7fd62c002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd62c002750 0x7fd62c002bd0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c004530 osd=1 1 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.2 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd62c00b240 msgr2=0x7fd62c00d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd62c00b240 0x7fd62c00d630 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 reap_dead start 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003d80 osd=2 1 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.3 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd62c002090 msgr2=0x7fd62c00ecf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd62c002090 0x7fd62c00ecf0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c003600 osd=3 1 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.4 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd62c00e850 msgr2=0x7fd62c012740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd62c00e850 0x7fd62c012740 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c001720 osd=4 1 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.5 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd62c0133a0 msgr2=0x7fd62c013800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd62c0133a0 0x7fd62c013800 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c012c80 osd=5 1 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.6 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd62c0186f0 msgr2=0x7fd62c01ab20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd62c0186f0 0x7fd62c01ab20 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c013dd0 osd=6 1 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter close_session for osd.7 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd62c01cea0 msgr2=0x7fd62c01d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd62c01cea0 0x7fd62c01d320 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter put_session s=0x7fd62c01db70 osd=7 1 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 20 client.30672.objecter shutdown clearing up homeless session... 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 10 client.30672.objecter successfully canceled tick 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd646d9d700 1 -- 172.21.15.143:0/1901908568 reap_dead start 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 msgr2=0x7fd630071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 0x7fd630071ce0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fd63800b2e0 tx=0x7fd638005960 comp rx=0 tx=0).stop 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 msgr2=0x7fd648103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fd64000c370 tx=0x7fd64000c680 comp rx=0 tx=0).stop 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 shutdown_connections 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648106dc0 0x7fd648103440 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 --2- 172.21.15.143:0/1901908568 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd63006f820 0x7fd630071ce0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 >> 172.21.15.143:0/1901908568 conn(0x7fd648076d80 msgr2=0x7fd6480ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 shutdown_connections 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 -- 172.21.15.143:0/1901908568 wait complete. 2024-03-11T21:38:56.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:56.846+0000 7fd6357fa700 1 librados: shutdown 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:56.832 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:56.836 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:56.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:56.968 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:57.071 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:57.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:57.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.354 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:57.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:57.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.494+0000 7f9dea7fc700 1 -- 172.21.15.143:0/4158121331 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]=0 set 6.7 pg_upmap mapping to [0,4,2] v650) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9df005c860 con 0x7f9df406a720 2024-03-11T21:38:57.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.494+0000 7f9dfb4de700 10 librados: watch_flush enter 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 10 librados: watch_flush exit 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 20 client.30660.objecter shutdown clearing up homeless session... 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 10 client.30660.objecter successfully canceled tick 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 msgr2=0x7f9de0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009680 tx=0x7f9de4009260 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 msgr2=0x7f9df4101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f9df000fa00 tx=0x7f9df000fa30 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 shutdown_connections 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df41071c0 0x7f9df4101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4069d70 0x7f9df4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 --2- 172.21.15.143:0/4158121331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a720 0x7f9df4101350 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 >> 172.21.15.143:0/4158121331 conn(0x7f9df4076d80 msgr2=0x7f9df40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:38:57.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 shutdown_connections 2024-03-11T21:38:57.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 -- 172.21.15.143:0/4158121331 wait complete. 2024-03-11T21:38:57.481 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:38:57.498+0000 7f9dfb4de700 1 librados: shutdown 2024-03-11T21:38:57.481 INFO:teuthology.orchestra.run.smithi143.stderr:set 6.7 pg_upmap mapping to [0,4,2] 2024-03-11T21:38:57.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:57.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:57.621 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: cluster 2024-03-11T21:38:56.415217+0000 osd.5 (osd.5) 389 : cluster 0 118.10 deep-scrub starts 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: cluster 2024-03-11T21:38:56.435814+0000 osd.5 (osd.5) 390 : cluster 0 118.10 deep-scrub ok 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: cluster 2024-03-11T21:38:56.489437+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: audit 2024-03-11T21:38:56.520493+0000 mon.a (mon.0) 3216 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: audit 2024-03-11T21:38:56.522106+0000 mon.a (mon.0) 3217 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: audit 2024-03-11T21:38:56.522561+0000 mon.a (mon.0) 3218 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: audit 2024-03-11T21:38:56.615594+0000 mon.a (mon.0) 3219 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]: dispatch 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[29284]: audit 2024-03-11T21:38:56.659760+0000 mon.a (mon.0) 3220 : audit 0 from='client.? 172.21.15.143:0/1901908568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: cluster 2024-03-11T21:38:56.415217+0000 osd.5 (osd.5) 389 : cluster 0 118.10 deep-scrub starts 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: cluster 2024-03-11T21:38:56.435814+0000 osd.5 (osd.5) 390 : cluster 0 118.10 deep-scrub ok 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: cluster 2024-03-11T21:38:56.489437+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: audit 2024-03-11T21:38:56.520493+0000 mon.a (mon.0) 3216 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: audit 2024-03-11T21:38:56.522106+0000 mon.a (mon.0) 3217 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: audit 2024-03-11T21:38:56.522561+0000 mon.a (mon.0) 3218 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:38:57.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: audit 2024-03-11T21:38:56.615594+0000 mon.a (mon.0) 3219 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]: dispatch 2024-03-11T21:38:57.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:57 smithi143 bash[21512]: audit 2024-03-11T21:38:56.659760+0000 mon.a (mon.0) 3220 : audit 0 from='client.? 172.21.15.143:0/1901908568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:57.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:57.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: cluster 2024-03-11T21:38:56.415217+0000 osd.5 (osd.5) 389 : cluster 0 118.10 deep-scrub starts 2024-03-11T21:38:57.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: cluster 2024-03-11T21:38:56.435814+0000 osd.5 (osd.5) 390 : cluster 0 118.10 deep-scrub ok 2024-03-11T21:38:57.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: cluster 2024-03-11T21:38:56.489437+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:38:57.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: audit 2024-03-11T21:38:56.520493+0000 mon.a (mon.0) 3216 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: audit 2024-03-11T21:38:56.522106+0000 mon.a (mon.0) 3217 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:57.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: audit 2024-03-11T21:38:56.522561+0000 mon.a (mon.0) 3218 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:38:57.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: audit 2024-03-11T21:38:56.615594+0000 mon.a (mon.0) 3219 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]: dispatch 2024-03-11T21:38:57.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:57 smithi203 bash[23654]: audit 2024-03-11T21:38:56.659760+0000 mon.a (mon.0) 3220 : audit 0 from='client.? 172.21.15.143:0/1901908568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:57.751 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:57.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:57.882 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:57.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:58.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.143 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:58.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:58.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:58.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:58.405 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:58.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:58.520 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: audit 2024-03-11T21:38:57.495527+0000 mon.a (mon.0) 3221 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:38:58.520 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: audit 2024-03-11T21:38:57.495764+0000 mon.a (mon.0) 3222 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]': finished 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: cluster 2024-03-11T21:38:57.503178+0000 mon.a (mon.0) 3223 : cluster 0 osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: audit 2024-03-11T21:38:57.506408+0000 mon.a (mon.0) 3224 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: cluster 2024-03-11T21:38:57.530815+0000 mgr.y (mgr.14550) 626 : cluster 0 pgmap v1006: 206 pgs: 206 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: cluster 2024-03-11T21:38:57.854385+0000 osd.1 (osd.1) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[29284]: cluster 2024-03-11T21:38:57.854983+0000 osd.1 (osd.1) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: audit 2024-03-11T21:38:57.495527+0000 mon.a (mon.0) 3221 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: audit 2024-03-11T21:38:57.495764+0000 mon.a (mon.0) 3222 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]': finished 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: cluster 2024-03-11T21:38:57.503178+0000 mon.a (mon.0) 3223 : cluster 0 osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: audit 2024-03-11T21:38:57.506408+0000 mon.a (mon.0) 3224 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:58.521 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: cluster 2024-03-11T21:38:57.530815+0000 mgr.y (mgr.14550) 626 : cluster 0 pgmap v1006: 206 pgs: 206 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:58.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: cluster 2024-03-11T21:38:57.854385+0000 osd.1 (osd.1) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:58.522 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:58 smithi143 bash[21512]: cluster 2024-03-11T21:38:57.854983+0000 osd.1 (osd.1) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:58.533 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:58.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:58.663 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:58.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: audit 2024-03-11T21:38:57.495527+0000 mon.a (mon.0) 3221 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:38:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: audit 2024-03-11T21:38:57.495764+0000 mon.a (mon.0) 3222 : audit 1 from='client.? 172.21.15.143:0/4158121331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [0, 4, 2]}]': finished 2024-03-11T21:38:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: cluster 2024-03-11T21:38:57.503178+0000 mon.a (mon.0) 3223 : cluster 0 osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:38:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: audit 2024-03-11T21:38:57.506408+0000 mon.a (mon.0) 3224 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:38:58.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: cluster 2024-03-11T21:38:57.530815+0000 mgr.y (mgr.14550) 626 : cluster 0 pgmap v1006: 206 pgs: 206 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:38:58.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: cluster 2024-03-11T21:38:57.854385+0000 osd.1 (osd.1) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:38:58.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[23654]: cluster 2024-03-11T21:38:57.854983+0000 osd.1 (osd.1) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:38:58.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:58.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:38:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:38:58.915 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:59.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:59.047 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:59.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:59.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:59.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:59.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:59.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:59.449 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:59.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:59.573 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:59.605 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:59.676 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:59.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[29284]: cluster 2024-03-11T21:38:58.853776+0000 osd.3 (osd.3) 377 : cluster 0 39.18 scrub starts 2024-03-11T21:38:59.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[29284]: cluster 2024-03-11T21:38:58.856086+0000 osd.3 (osd.3) 378 : cluster 0 39.18 scrub ok 2024-03-11T21:38:59.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[29284]: audit 2024-03-11T21:38:58.860628+0000 mgr.y (mgr.14550) 627 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:59.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[29284]: cluster 2024-03-11T21:38:58.946745+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:38:59.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[21512]: cluster 2024-03-11T21:38:58.853776+0000 osd.3 (osd.3) 377 : cluster 0 39.18 scrub starts 2024-03-11T21:38:59.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[21512]: cluster 2024-03-11T21:38:58.856086+0000 osd.3 (osd.3) 378 : cluster 0 39.18 scrub ok 2024-03-11T21:38:59.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[21512]: audit 2024-03-11T21:38:58.860628+0000 mgr.y (mgr.14550) 627 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:59.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:38:59 smithi143 bash[21512]: cluster 2024-03-11T21:38:58.946745+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:38:59.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:59.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:59 smithi203 bash[23654]: cluster 2024-03-11T21:38:58.853776+0000 osd.3 (osd.3) 377 : cluster 0 39.18 scrub starts 2024-03-11T21:38:59.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:59 smithi203 bash[23654]: cluster 2024-03-11T21:38:58.856086+0000 osd.3 (osd.3) 378 : cluster 0 39.18 scrub ok 2024-03-11T21:38:59.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:59 smithi203 bash[23654]: audit 2024-03-11T21:38:58.860628+0000 mgr.y (mgr.14550) 627 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:59.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:38:59 smithi203 bash[23654]: cluster 2024-03-11T21:38:58.946745+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:38:59.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:59.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:59.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:00.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:00.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:00.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:00.224 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:00.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:00.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:00.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:00.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:00.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: audit 2024-03-11T21:38:59.514419+0000 mon.a (mon.0) 3226 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.518822+0000 mon.a (mon.0) 3227 : cluster 0 osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.531515+0000 mgr.y (mgr.14550) 628 : cluster 0 pgmap v1009: 214 pgs: 8 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.629836+0000 osd.1 (osd.1) 175 : cluster 0 9.1f scrub starts 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.631865+0000 osd.1 (osd.1) 176 : cluster 0 9.1f scrub ok 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.730574+0000 osd.2 (osd.2) 323 : cluster 0 5.b scrub starts 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.732163+0000 osd.2 (osd.2) 324 : cluster 0 5.b scrub ok 2024-03-11T21:39:00.600 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.763377+0000 osd.7 (osd.7) 362 : cluster 0 3.d deep-scrub starts 2024-03-11T21:39:00.601 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.768904+0000 osd.7 (osd.7) 363 : cluster 0 3.d deep-scrub ok 2024-03-11T21:39:00.601 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:00 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.898775+0000 osd.3 (osd.3) 379 : cluster 0 39.e scrub starts 2024-03-11T21:39:00.615 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:00.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: audit 2024-03-11T21:38:59.514419+0000 mon.a (mon.0) 3226 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.518822+0000 mon.a (mon.0) 3227 : cluster 0 osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.531515+0000 mgr.y (mgr.14550) 628 : cluster 0 pgmap v1009: 214 pgs: 8 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.629836+0000 osd.1 (osd.1) 175 : cluster 0 9.1f scrub starts 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.631865+0000 osd.1 (osd.1) 176 : cluster 0 9.1f scrub ok 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.730574+0000 osd.2 (osd.2) 323 : cluster 0 5.b scrub starts 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.732163+0000 osd.2 (osd.2) 324 : cluster 0 5.b scrub ok 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.763377+0000 osd.7 (osd.7) 362 : cluster 0 3.d deep-scrub starts 2024-03-11T21:39:00.727 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.768904+0000 osd.7 (osd.7) 363 : cluster 0 3.d deep-scrub ok 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.898775+0000 osd.3 (osd.3) 379 : cluster 0 39.e scrub starts 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: audit 2024-03-11T21:38:59.514419+0000 mon.a (mon.0) 3226 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi143-71773-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.518822+0000 mon.a (mon.0) 3227 : cluster 0 osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.531515+0000 mgr.y (mgr.14550) 628 : cluster 0 pgmap v1009: 214 pgs: 8 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.629836+0000 osd.1 (osd.1) 175 : cluster 0 9.1f scrub starts 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.631865+0000 osd.1 (osd.1) 176 : cluster 0 9.1f scrub ok 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.730574+0000 osd.2 (osd.2) 323 : cluster 0 5.b scrub starts 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.732163+0000 osd.2 (osd.2) 324 : cluster 0 5.b scrub ok 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.763377+0000 osd.7 (osd.7) 362 : cluster 0 3.d deep-scrub starts 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.768904+0000 osd.7 (osd.7) 363 : cluster 0 3.d deep-scrub ok 2024-03-11T21:39:00.728 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:00 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.898775+0000 osd.3 (osd.3) 379 : cluster 0 39.e scrub starts 2024-03-11T21:39:00.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:00.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:00.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:00.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:01.003 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:01.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:01.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:39:01.130 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:01.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:01.260 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:01.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:01.392 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:01.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:01.520 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:01.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:01.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:01.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:01.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:38:59.900742+0000 osd.3 (osd.3) 380 : cluster 0 39.e scrub ok 2024-03-11T21:39:01.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:00.419395+0000 osd.6 (osd.6) 291 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:00.422090+0000 osd.6 (osd.6) 292 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:00.514498+0000 mon.a (mon.0) 3228 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:00.514539+0000 mon.a (mon.0) 3229 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:00.528954+0000 mon.a (mon.0) 3230 : cluster 0 osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:01.481459+0000 mon.a (mon.0) 3231 : cluster 0 osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: audit 2024-03-11T21:39:01.483681+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:01 smithi203 bash[23654]: audit 2024-03-11T21:39:01.494455+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 -- 172.21.15.143:0/1952474841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914107f30 msgr2=0x7fd914108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1952474841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914107f30 0x7fd914108310 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fd904009990 tx=0x7fd904009ca0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 -- 172.21.15.143:0/1952474841 shutdown_connections 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1952474841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914102880 0x7fd91410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1952474841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914107f30 0x7fd914108310 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1952474841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914101ec0 0x7fd914102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.838+0000 7fd919dc1700 1 -- 172.21.15.143:0/1952474841 >> 172.21.15.143:0/1952474841 conn(0x7fd914078eb0 msgr2=0x7fd9140792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- 172.21.15.143:0/1952474841 shutdown_connections 2024-03-11T21:39:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- 172.21.15.143:0/1952474841 wait complete. 2024-03-11T21:39:01.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 librados: starting msgr at 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 librados: starting objecter 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 Processor -- start 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- start start 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 librados: setting wanted keys 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 librados: calling monclient init 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 0x7fd91419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914102880 0x7fd91419dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914111630 con 0x7fd914107f30 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9141114b0 con 0x7fd914101ec0 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9141117b0 con 0x7fd914102880 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:01.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34960/0 (socket says 172.21.15.143:34960) 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 -- 172.21.15.143:0/1574162995 learned_addr learned my addr 172.21.15.143:0/1574162995 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd9137fe700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 0x7fd91419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd912ffd700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914102880 0x7fd91419dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 -- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914102880 msgr2=0x7fd91419dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914102880 0x7fd91419dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 -- 172.21.15.143:0/1574162995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 msgr2=0x7fd91419d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 0x7fd91419d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 -- 172.21.15.143:0/1574162995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914198140 con 0x7fd914107f30 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd9137fe700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 0x7fd91419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd913fff700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fd90c007ee0 tx=0x7fd90c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 10 client.?.objecter ms_handle_connect 0x7fd914107f30 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c004bb0 con 0x7fd914107f30 2024-03-11T21:39:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd914198430 con 0x7fd914107f30 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd90c004d50 con 0x7fd914107f30 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c005830 con 0x7fd914107f30 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 10 client.30684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9141a46f0 con 0x7fd914107f30 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 librados: init done 2024-03-11T21:39:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91404faa0 con 0x7fd914107f30 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.842+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd90c01a020 con 0x7fd914107f30 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 0x7fd8fc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(654..654 src has 87..654) v4 ==== 10149+0+0 (secure 0 0 0) 0x7fd90c08e340 con 0x7fd914107f30 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 10 client.30684.objecter ms_dispatch 0x7fd914001550 osd_map(654..654 src has 87..654) v4 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 3 client.30684.objecter handle_osd_map got epochs [654,654] > 0 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 3 client.30684.objecter handle_osd_map decoding full epoch 654 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd9137fe700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 0x7fd8fc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 20 client.30684.objecter dump_active .. 0 homeless 2024-03-11T21:39:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd9137fe700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 0x7fd8fc071ce0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7fd90400b2e0 tx=0x7fd90401a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.846+0000 7fd910ff9700 10 client.30684.objecter ms_handle_connect 0x7fd8fc06f820 2024-03-11T21:39:01.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:01.850+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd90c010070 con 0x7fd914107f30 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:38:59.900742+0000 osd.3 (osd.3) 380 : cluster 0 39.e scrub ok 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:00.419395+0000 osd.6 (osd.6) 291 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:00.422090+0000 osd.6 (osd.6) 292 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:00.514498+0000 mon.a (mon.0) 3228 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:00.514539+0000 mon.a (mon.0) 3229 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:00.528954+0000 mon.a (mon.0) 3230 : cluster 0 osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:01.481459+0000 mon.a (mon.0) 3231 : cluster 0 osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: audit 2024-03-11T21:39:01.483681+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[21512]: audit 2024-03-11T21:39:01.494455+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:38:59.900742+0000 osd.3 (osd.3) 380 : cluster 0 39.e scrub ok 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:00.419395+0000 osd.6 (osd.6) 291 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:00.422090+0000 osd.6 (osd.6) 292 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:00.514498+0000 mon.a (mon.0) 3228 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:00.514539+0000 mon.a (mon.0) 3229 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:00.528954+0000 mon.a (mon.0) 3230 : cluster 0 osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:01.481459+0000 mon.a (mon.0) 3231 : cluster 0 osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: audit 2024-03-11T21:39:01.483681+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.870 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:01 smithi143 bash[29284]: audit 2024-03-11T21:39:01.494455+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:01.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:01.908 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:02.011 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.029 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:02.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.070+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fd9141a4920 con 0x7fd914107f30 2024-03-11T21:39:02.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:02.158 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:02.188 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:02.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.289 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:02.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:02.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:02.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.454+0000 7fd910ff9700 1 -- 172.21.15.143:0/1574162995 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v655) v1 ==== 87+0+0 (secure 0 0 0) 0x7fd90c05ad80 con 0x7fd914107f30 2024-03-11T21:39:02.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.458+0000 7fd919dc1700 10 librados: watch_flush enter 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 10 librados: watch_flush exit 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 20 client.30684.objecter shutdown clearing up homeless session... 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 10 client.30684.objecter successfully canceled tick 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 msgr2=0x7fd8fc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 0x7fd8fc071ce0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7fd90400b2e0 tx=0x7fd90401a040 comp rx=0 tx=0).stop 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 msgr2=0x7fd9141979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:02.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fd90c007ee0 tx=0x7fd90c00d3b0 comp rx=0 tx=0).stop 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 shutdown_connections 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd8fc06f820 0x7fd8fc071ce0 unknown :-1 s=CLOSED pgs=3424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd914107f30 0x7fd9141979e0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd914101ec0 0x7fd91419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 --2- 172.21.15.143:0/1574162995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd914102880 0x7fd91419dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 >> 172.21.15.143:0/1574162995 conn(0x7fd914078eb0 msgr2=0x7fd914108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.462+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 shutdown_connections 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.466+0000 7fd919dc1700 1 -- 172.21.15.143:0/1574162995 wait complete. 2024-03-11T21:39:02.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:02.466+0000 7fd919dc1700 1 librados: shutdown 2024-03-11T21:39:02.457 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:39:02.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:02.541 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:02.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:02.672 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:02.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.805 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:02.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:02.920 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: cluster 2024-03-11T21:39:01.532351+0000 mgr.y (mgr.14550) 629 : cluster 0 pgmap v1012: 246 pgs: 1 creating+peering, 39 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: cluster 2024-03-11T21:39:01.786488+0000 osd.7 (osd.7) 364 : cluster 0 6.15 scrub starts 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: cluster 2024-03-11T21:39:01.789057+0000 osd.7 (osd.7) 365 : cluster 0 6.15 scrub ok 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: audit 2024-03-11T21:39:02.072969+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: cluster 2024-03-11T21:39:02.456049+0000 mon.a (mon.0) 3234 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: audit 2024-03-11T21:39:02.458766+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: audit 2024-03-11T21:39:02.458970+0000 mon.a (mon.0) 3236 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:02.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:02 smithi203 bash[23654]: cluster 2024-03-11T21:39:02.466311+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:39:02.936 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:03.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:03.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: cluster 2024-03-11T21:39:01.532351+0000 mgr.y (mgr.14550) 629 : cluster 0 pgmap v1012: 246 pgs: 1 creating+peering, 39 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:03.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: cluster 2024-03-11T21:39:01.786488+0000 osd.7 (osd.7) 364 : cluster 0 6.15 scrub starts 2024-03-11T21:39:03.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: cluster 2024-03-11T21:39:01.789057+0000 osd.7 (osd.7) 365 : cluster 0 6.15 scrub ok 2024-03-11T21:39:03.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: audit 2024-03-11T21:39:02.072969+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:03.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: cluster 2024-03-11T21:39:02.456049+0000 mon.a (mon.0) 3234 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: audit 2024-03-11T21:39:02.458766+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: audit 2024-03-11T21:39:02.458970+0000 mon.a (mon.0) 3236 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[29284]: cluster 2024-03-11T21:39:02.466311+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: cluster 2024-03-11T21:39:01.532351+0000 mgr.y (mgr.14550) 629 : cluster 0 pgmap v1012: 246 pgs: 1 creating+peering, 39 unknown, 1 activating, 205 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: cluster 2024-03-11T21:39:01.786488+0000 osd.7 (osd.7) 364 : cluster 0 6.15 scrub starts 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: cluster 2024-03-11T21:39:01.789057+0000 osd.7 (osd.7) 365 : cluster 0 6.15 scrub ok 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: audit 2024-03-11T21:39:02.072969+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: cluster 2024-03-11T21:39:02.456049+0000 mon.a (mon.0) 3234 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: audit 2024-03-11T21:39:02.458766+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: audit 2024-03-11T21:39:02.458970+0000 mon.a (mon.0) 3236 : audit 1 from='client.? 172.21.15.143:0/1574162995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:03.178 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:02 smithi143 bash[21512]: cluster 2024-03-11T21:39:02.466311+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:39:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:03.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:03.326 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:03.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:03.569 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:03.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:03.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:03.727 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:03.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.857 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:03.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:03.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:04.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:04.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:04.129 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:39:04.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:39:04.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:04.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:04.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:04.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:04 smithi203 bash[23654]: cluster 2024-03-11T21:39:03.483026+0000 mon.a (mon.0) 3238 : cluster 0 osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:39:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:04 smithi203 bash[23654]: audit 2024-03-11T21:39:03.489706+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:04 smithi203 bash[23654]: audit 2024-03-11T21:39:03.490633+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:04 smithi203 bash[23654]: cluster 2024-03-11T21:39:03.533383+0000 mgr.y (mgr.14550) 630 : cluster 0 pgmap v1015: 278 pgs: 1 creating+peering, 41 unknown, 1 activating, 235 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:04.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:04.612 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:04.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:04.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[29284]: cluster 2024-03-11T21:39:03.483026+0000 mon.a (mon.0) 3238 : cluster 0 osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[29284]: audit 2024-03-11T21:39:03.489706+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[29284]: audit 2024-03-11T21:39:03.490633+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[29284]: cluster 2024-03-11T21:39:03.533383+0000 mgr.y (mgr.14550) 630 : cluster 0 pgmap v1015: 278 pgs: 1 creating+peering, 41 unknown, 1 activating, 235 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[21512]: cluster 2024-03-11T21:39:03.483026+0000 mon.a (mon.0) 3238 : cluster 0 osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[21512]: audit 2024-03-11T21:39:03.489706+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[21512]: audit 2024-03-11T21:39:03.490633+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:04.747 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:04 smithi143 bash[21512]: cluster 2024-03-11T21:39:03.533383+0000 mgr.y (mgr.14550) 630 : cluster 0 pgmap v1015: 278 pgs: 1 creating+peering, 41 unknown, 1 activating, 235 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:04.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.892 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:04.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.022 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:05.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.156 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:05.234 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:39:05.234 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:39:05.234 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:39:05.234 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:39:05.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.279 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:05.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:05.410 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:05.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:05.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[21512]: audit 2024-03-11T21:39:04.474325+0000 mon.a (mon.0) 3240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[21512]: cluster 2024-03-11T21:39:04.483798+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:39:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[21512]: audit 2024-03-11T21:39:04.488004+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[21512]: audit 2024-03-11T21:39:04.489765+0000 mon.a (mon.0) 3242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.689 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[29284]: audit 2024-03-11T21:39:04.474325+0000 mon.a (mon.0) 3240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:05.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[29284]: cluster 2024-03-11T21:39:04.483798+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:39:05.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[29284]: audit 2024-03-11T21:39:04.488004+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:05 smithi143 bash[29284]: audit 2024-03-11T21:39:04.489765+0000 mon.a (mon.0) 3242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:05 smithi203 bash[23654]: audit 2024-03-11T21:39:04.474325+0000 mon.a (mon.0) 3240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:05.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:05 smithi203 bash[23654]: cluster 2024-03-11T21:39:04.483798+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:39:05.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:05 smithi203 bash[23654]: audit 2024-03-11T21:39:04.488004+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:05 smithi203 bash[23654]: audit 2024-03-11T21:39:04.489765+0000 mon.a (mon.0) 3242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:05.798 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:05.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.927 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:06.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:06.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:06.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:06.187 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:06.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:06.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:06.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:39:06.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:06.447 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:06.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:06.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:06.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: audit 2024-03-11T21:39:05.476229+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: cluster 2024-03-11T21:39:05.479994+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: audit 2024-03-11T21:39:05.491670+0000 mon.c (mon.2) 1135 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: audit 2024-03-11T21:39:05.492048+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: cluster 2024-03-11T21:39:05.534170+0000 mgr.y (mgr.14550) 631 : cluster 0 pgmap v1018: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[29284]: cluster 2024-03-11T21:39:06.454760+0000 mon.a (mon.0) 3246 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: audit 2024-03-11T21:39:05.476229+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: cluster 2024-03-11T21:39:05.479994+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:39:06.602 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: audit 2024-03-11T21:39:05.491670+0000 mon.c (mon.2) 1135 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: audit 2024-03-11T21:39:05.492048+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: cluster 2024-03-11T21:39:05.534170+0000 mgr.y (mgr.14550) 631 : cluster 0 pgmap v1018: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:39:06.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:06 smithi143 bash[21512]: cluster 2024-03-11T21:39:06.454760+0000 mon.a (mon.0) 3246 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:06.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:06.706 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:06.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: audit 2024-03-11T21:39:05.476229+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: cluster 2024-03-11T21:39:05.479994+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:39:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: audit 2024-03-11T21:39:05.491670+0000 mon.c (mon.2) 1135 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: audit 2024-03-11T21:39:05.492048+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: cluster 2024-03-11T21:39:05.534170+0000 mgr.y (mgr.14550) 631 : cluster 0 pgmap v1018: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:39:06.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:06 smithi203 bash[23654]: cluster 2024-03-11T21:39:06.454760+0000 mon.a (mon.0) 3246 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:06.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:06.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:06.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:06.971 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:07.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:07.103 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:07.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:07.234 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:07.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:07.364 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:07.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:07.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:07.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:07.624 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:07.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:07.739 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: cluster 2024-03-11T21:39:06.485135+0000 mon.a (mon.0) 3247 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:39:07.739 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: audit 2024-03-11T21:39:06.495994+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:07.739 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: audit 2024-03-11T21:39:06.501679+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: cluster 2024-03-11T21:39:06.504444+0000 mon.a (mon.0) 3249 : cluster 0 osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: audit 2024-03-11T21:39:06.508471+0000 mon.a (mon.0) 3250 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: audit 2024-03-11T21:39:06.961458+0000 mon.a (mon.0) 3251 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[29284]: audit 2024-03-11T21:39:06.965341+0000 mon.c (mon.2) 1137 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: cluster 2024-03-11T21:39:06.485135+0000 mon.a (mon.0) 3247 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: audit 2024-03-11T21:39:06.495994+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: audit 2024-03-11T21:39:06.501679+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: cluster 2024-03-11T21:39:06.504444+0000 mon.a (mon.0) 3249 : cluster 0 osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: audit 2024-03-11T21:39:06.508471+0000 mon.a (mon.0) 3250 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: audit 2024-03-11T21:39:06.961458+0000 mon.a (mon.0) 3251 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:07.740 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:07 smithi143 bash[21512]: audit 2024-03-11T21:39:06.965341+0000 mon.c (mon.2) 1137 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:07.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:07.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: cluster 2024-03-11T21:39:06.485135+0000 mon.a (mon.0) 3247 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:39:07.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: audit 2024-03-11T21:39:06.495994+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-107", "overlaypool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:07.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: audit 2024-03-11T21:39:06.501679+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: cluster 2024-03-11T21:39:06.504444+0000 mon.a (mon.0) 3249 : cluster 0 osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:39:07.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: audit 2024-03-11T21:39:06.508471+0000 mon.a (mon.0) 3250 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:07.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: audit 2024-03-11T21:39:06.961458+0000 mon.a (mon.0) 3251 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:07.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:07 smithi203 bash[23654]: audit 2024-03-11T21:39:06.965341+0000 mon.c (mon.2) 1137 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:07.809 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:07.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:07.881 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:07.984 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:08.012 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:08.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:08.142 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:08.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:08.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:08.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.404 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:08.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: cluster 2024-03-11T21:39:06.950164+0000 osd.5 (osd.5) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:08.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: cluster 2024-03-11T21:39:06.950524+0000 osd.5 (osd.5) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:08.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: cluster 2024-03-11T21:39:07.497490+0000 mon.a (mon.0) 3252 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:08.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: audit 2024-03-11T21:39:07.509243+0000 mon.a (mon.0) 3253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:08.518 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: cluster 2024-03-11T21:39:07.516171+0000 mon.a (mon.0) 3254 : cluster 0 osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:39:08.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: cluster 2024-03-11T21:39:07.535072+0000 mgr.y (mgr.14550) 632 : cluster 0 pgmap v1021: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:39:08.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: audit 2024-03-11T21:39:07.549781+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:08.519 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[23654]: audit 2024-03-11T21:39:07.550277+0000 mon.a (mon.0) 3255 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:08.535 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:08.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.661 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:08.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.790 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:08.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.920 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: cluster 2024-03-11T21:39:06.950164+0000 osd.5 (osd.5) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: cluster 2024-03-11T21:39:06.950524+0000 osd.5 (osd.5) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: cluster 2024-03-11T21:39:07.497490+0000 mon.a (mon.0) 3252 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: audit 2024-03-11T21:39:07.509243+0000 mon.a (mon.0) 3253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: cluster 2024-03-11T21:39:07.516171+0000 mon.a (mon.0) 3254 : cluster 0 osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:39:08.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: cluster 2024-03-11T21:39:07.535072+0000 mgr.y (mgr.14550) 632 : cluster 0 pgmap v1021: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: audit 2024-03-11T21:39:07.549781+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[21512]: audit 2024-03-11T21:39:07.550277+0000 mon.a (mon.0) 3255 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: cluster 2024-03-11T21:39:06.950164+0000 osd.5 (osd.5) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: cluster 2024-03-11T21:39:06.950524+0000 osd.5 (osd.5) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: cluster 2024-03-11T21:39:07.497490+0000 mon.a (mon.0) 3252 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: audit 2024-03-11T21:39:07.509243+0000 mon.a (mon.0) 3253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: cluster 2024-03-11T21:39:07.516171+0000 mon.a (mon.0) 3254 : cluster 0 osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: cluster 2024-03-11T21:39:07.535072+0000 mgr.y (mgr.14550) 632 : cluster 0 pgmap v1021: 278 pgs: 2 creating+peering, 25 unknown, 251 active+clean; 582 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: audit 2024-03-11T21:39:07.549781+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:08.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:08 smithi143 bash[29284]: audit 2024-03-11T21:39:07.550277+0000 mon.a (mon.0) 3255 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]: dispatch 2024-03-11T21:39:09.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:09.046 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:09.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:09.168 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:09.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:09.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:09.299 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:09.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:09.431 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:09.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:09.559 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:09.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:09.673 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: audit 2024-03-11T21:39:08.512046+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]': finished 2024-03-11T21:39:09.673 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: cluster 2024-03-11T21:39:08.519995+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:39:09.673 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: audit 2024-03-11T21:39:08.526372+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: audit 2024-03-11T21:39:08.535767+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: cluster 2024-03-11T21:39:08.606511+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: cluster 2024-03-11T21:39:08.606930+0000 osd.3 (osd.3) 382 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[21512]: audit 2024-03-11T21:39:08.871820+0000 mgr.y (mgr.14550) 633 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: audit 2024-03-11T21:39:08.512046+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]': finished 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: cluster 2024-03-11T21:39:08.519995+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: audit 2024-03-11T21:39:08.526372+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: audit 2024-03-11T21:39:08.535767+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: cluster 2024-03-11T21:39:08.606511+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: cluster 2024-03-11T21:39:08.606930+0000 osd.3 (osd.3) 382 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:09.674 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:09 smithi143 bash[29284]: audit 2024-03-11T21:39:08.871820+0000 mgr.y (mgr.14550) 633 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:09.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:09.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: audit 2024-03-11T21:39:08.512046+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-107"}]': finished 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: cluster 2024-03-11T21:39:08.519995+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: audit 2024-03-11T21:39:08.526372+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.143:0/1724782028' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: audit 2024-03-11T21:39:08.535767+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]: dispatch 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: cluster 2024-03-11T21:39:08.606511+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: cluster 2024-03-11T21:39:08.606930+0000 osd.3 (osd.3) 382 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:09.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:09 smithi203 bash[23654]: audit 2024-03-11T21:39:08.871820+0000 mgr.y (mgr.14550) 633 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:09.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:09.819 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:09.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:09.949 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:10.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:10.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:10.206 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:10.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:10.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:10.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:10.582 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:10 smithi203 bash[23654]: cluster 2024-03-11T21:39:09.528018+0000 mon.a (mon.0) 3259 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:10.582 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:10 smithi203 bash[23654]: audit 2024-03-11T21:39:09.530211+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:10.582 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:10 smithi203 bash[23654]: cluster 2024-03-11T21:39:09.535757+0000 mgr.y (mgr.14550) 634 : cluster 0 pgmap v1024: 278 pgs: 278 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:10.583 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:10 smithi203 bash[23654]: cluster 2024-03-11T21:39:09.538537+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:39:10.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:10.624 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:10.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[21512]: cluster 2024-03-11T21:39:09.528018+0000 mon.a (mon.0) 3259 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[21512]: audit 2024-03-11T21:39:09.530211+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[21512]: cluster 2024-03-11T21:39:09.535757+0000 mgr.y (mgr.14550) 634 : cluster 0 pgmap v1024: 278 pgs: 278 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[21512]: cluster 2024-03-11T21:39:09.538537+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[29284]: cluster 2024-03-11T21:39:09.528018+0000 mon.a (mon.0) 3259 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:10.712 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[29284]: audit 2024-03-11T21:39:09.530211+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-107", "tierpool": "test-rados-api-smithi143-71773-107-cache"}]': finished 2024-03-11T21:39:10.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[29284]: cluster 2024-03-11T21:39:09.535757+0000 mgr.y (mgr.14550) 634 : cluster 0 pgmap v1024: 278 pgs: 278 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:10.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:10 smithi143 bash[29284]: cluster 2024-03-11T21:39:09.538537+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:39:10.727 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:10.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:10.858 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:10.878 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:10.879 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:10.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:11.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:11.120 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:11.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:11.251 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:11.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:11.382 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:11.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:11.506 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:11.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:11.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:11.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:11.763 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:11.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: cluster 2024-03-11T21:39:10.541247+0000 mon.a (mon.0) 3262 : cluster 0 osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: cluster 2024-03-11T21:39:11.456572+0000 mon.a (mon.0) 3263 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: cluster 2024-03-11T21:39:11.463902+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.492905+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.493350+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.494687+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.495058+0000 mon.a (mon.0) 3266 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.495702+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:11.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:11 smithi203 bash[23654]: audit 2024-03-11T21:39:11.496089+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:11.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:11.896 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:11.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:12.027 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: cluster 2024-03-11T21:39:10.541247+0000 mon.a (mon.0) 3262 : cluster 0 osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: cluster 2024-03-11T21:39:11.456572+0000 mon.a (mon.0) 3263 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: cluster 2024-03-11T21:39:11.463902+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.492905+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.493350+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.494687+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.495058+0000 mon.a (mon.0) 3266 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.495702+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:12.085 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[21512]: audit 2024-03-11T21:39:11.496089+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: cluster 2024-03-11T21:39:10.541247+0000 mon.a (mon.0) 3262 : cluster 0 osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: cluster 2024-03-11T21:39:11.456572+0000 mon.a (mon.0) 3263 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: cluster 2024-03-11T21:39:11.463902+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.492905+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.493350+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.494687+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.495058+0000 mon.a (mon.0) 3266 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.495702+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:12.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:11 smithi143 bash[29284]: audit 2024-03-11T21:39:11.496089+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:12.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:12.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:12.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:12.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:12.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:12.439 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:12.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:12.645 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:12.673 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:12.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:12 smithi203 bash[23654]: cluster 2024-03-11T21:39:11.536275+0000 mgr.y (mgr.14550) 635 : cluster 0 pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:12.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:12 smithi203 bash[23654]: audit 2024-03-11T21:39:12.473801+0000 mon.a (mon.0) 3268 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:12.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:12 smithi203 bash[23654]: cluster 2024-03-11T21:39:12.485396+0000 mon.a (mon.0) 3269 : cluster 0 osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:39:12.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:12 smithi203 bash[23654]: audit 2024-03-11T21:39:12.486843+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:12 smithi203 bash[23654]: audit 2024-03-11T21:39:12.487453+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:12.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:12.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.936 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[29284]: cluster 2024-03-11T21:39:11.536275+0000 mgr.y (mgr.14550) 635 : cluster 0 pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[29284]: audit 2024-03-11T21:39:12.473801+0000 mon.a (mon.0) 3268 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[29284]: cluster 2024-03-11T21:39:12.485396+0000 mon.a (mon.0) 3269 : cluster 0 osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[29284]: audit 2024-03-11T21:39:12.486843+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[29284]: audit 2024-03-11T21:39:12.487453+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21512]: cluster 2024-03-11T21:39:11.536275+0000 mgr.y (mgr.14550) 635 : cluster 0 pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:12.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21512]: audit 2024-03-11T21:39:12.473801+0000 mon.a (mon.0) 3268 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:12.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21512]: cluster 2024-03-11T21:39:12.485396+0000 mon.a (mon.0) 3269 : cluster 0 osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:39:12.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21512]: audit 2024-03-11T21:39:12.486843+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:12.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:12 smithi143 bash[21512]: audit 2024-03-11T21:39:12.487453+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:13.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.066 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:13.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.197 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:13.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:13.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:13.456 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:13.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- 172.21.15.143:0/4081353332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 msgr2=0x7f8df010ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- 172.21.15.143:0/4081353332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df010ecc0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f8de800b0d0 tx=0x7f8de800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- 172.21.15.143:0/4081353332 shutdown_connections 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- 172.21.15.143:0/4081353332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0106e00 0x7f8df01071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- 172.21.15.143:0/4081353332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df010ecc0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- 172.21.15.143:0/4081353332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0100330 0x7f8df01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- 172.21.15.143:0/4081353332 >> 172.21.15.143:0/4081353332 conn(0x7f8df00772e0 msgr2=0x7f8df00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:13.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- 172.21.15.143:0/4081353332 shutdown_connections 2024-03-11T21:39:13.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- 172.21.15.143:0/4081353332 wait complete. 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 librados: starting msgr at 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 librados: starting objecter 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 Processor -- start 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 -- start start 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 librados: setting wanted keys 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 librados: calling monclient init 2024-03-11T21:39:13.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0100330 0x7f8df019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.622+0000 7f8df647b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0106e00 0x7f8df0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8df647b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0111780 con 0x7f8df0100330 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8df647b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0111600 con 0x7f8df0106e00 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8df647b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0111900 con 0x7f8df0100cf0 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8def7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0100330 0x7f8df019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8def7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60698/0 (socket says 172.21.15.143:60698) 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8def7fe700 1 -- 172.21.15.143:0/2326911456 learned_addr learned my addr 172.21.15.143:0/2326911456 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 msgr2=0x7f8df019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8df0100cf0 0x7f8df019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0106e00 msgr2=0x7f8df0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:13.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0106e00 0x7f8df0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0106e00 0x7f8df0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df01981d0 con 0x7f8df0100330 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df0106e00 0x7f8df0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0100330 0x7f8df019d860 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e9f0 tx=0x7f8de000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8ded7fa700 10 client.?.objecter ms_handle_connect 0x7f8df0100330 2024-03-11T21:39:13.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.626+0000 7f8ded7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:13.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.719 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:13.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:13.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000cbd0 con 0x7f8df0100330 2024-03-11T21:39:13.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8de000cd70 con 0x7f8df0100330 2024-03-11T21:39:13.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0018680 con 0x7f8df0100330 2024-03-11T21:39:13.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df01984c0 con 0x7f8df0100330 2024-03-11T21:39:13.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8df647b700 10 client.30696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:13.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df01a4730 con 0x7f8df0100330 2024-03-11T21:39:13.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.918+0000 7f8df647b700 1 librados: init done 2024-03-11T21:39:13.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8df004faa0 con 0x7f8df0100330 2024-03-11T21:39:13.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8de0018820 con 0x7f8df0100330 2024-03-11T21:39:13.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8dd806f820 0x7f8dd8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(666..666 src has 87..666) v4 ==== 10216+0+0 (secure 0 0 0) 0x7f8de008ef40 con 0x7f8df0100330 2024-03-11T21:39:13.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 osd_map(666..666 src has 87..666) v4 2024-03-11T21:39:13.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 3 client.30696.objecter handle_osd_map got epochs [666,666] > 0 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 3 client.30696.objecter handle_osd_map decoding full epoch 666 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8dd806f820 0x7f8dd8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 20 client.30696.objecter dump_active .. 0 homeless 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v666) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8de0091500 con 0x7f8df0100330 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 10 client.30696.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 1 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8df00629f0 0x7f8df0062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 1 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 0 1 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 4 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 10 client.30696.objecter _send_command 1 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01a7af0 con 0x7f8df00629f0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8df00629f0 0x7f8df0062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8dd806f820 0x7f8dd8071ce0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f8df00ffb30 tx=0x7f8de4009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8dd806f820 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df00629f0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.922+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8df00629f0 0x7f8df0062dd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.926+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01a7af0 con 0x7f8df00629f0 2024-03-11T21:39:13.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.926+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:39:13.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.926+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 1 = system:0 2024-03-11T21:39:13.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.926+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 4 2024-03-11T21:39:13.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.926+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 0 1 2024-03-11T21:39:13.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 10 client.30696.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:13.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 2 2024-03-11T21:39:13.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 2 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df0198f50 osd=0 3 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 0 2 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 4 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 10 client.30696.objecter _send_command 2 2024-03-11T21:39:13.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df0062470 con 0x7f8df00629f0 2024-03-11T21:39:13.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df0062470 con 0x7f8df00629f0 2024-03-11T21:39:13.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:13.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:13.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 4 2024-03-11T21:39:13.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 0 2 2024-03-11T21:39:13.931 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:39:13.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 10 client.30696.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:13.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 3 2024-03-11T21:39:13.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8df002c6b0 0x7f8df002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 1 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 4 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 10 client.30696.objecter _send_command 3 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df0109a50 con 0x7f8df002c6b0 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8df002c6b0 0x7f8df002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8df002c6b0 0x7f8df002ca90 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df002c6b0 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df0109a50 con 0x7f8df002c6b0 2024-03-11T21:39:13.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:39:13.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 3 = system:0 2024-03-11T21:39:13.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 4 2024-03-11T21:39:13.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.950+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 1 3 2024-03-11T21:39:13.945 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 10 client.30696.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 4 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 4 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01093d0 osd=1 3 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 1 4 2024-03-11T21:39:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 4 2024-03-11T21:39:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 10 client.30696.objecter _send_command 4 2024-03-11T21:39:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df0108e50 con 0x7f8df002c6b0 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df0108e50 con 0x7f8df002c6b0 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 4 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 1 4 2024-03-11T21:39:13.951 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 10 client.30696.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 5 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8df0101440 0x7f8df01018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 5 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 2 5 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 4 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 10 client.30696.objecter _send_command 5 2024-03-11T21:39:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01b1250 con 0x7f8df0101440 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8df0101440 0x7f8df01018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8df0101440 0x7f8df01018a0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.970+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df0101440 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.974+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01b1250 con 0x7f8df0101440 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.974+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.974+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 5 = system:0 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.974+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 4 2024-03-11T21:39:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.974+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 2 5 2024-03-11T21:39:13.973 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 10 client.30696.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 6 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 6 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df002d7a0 osd=2 3 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 2 6 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 4 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 10 client.30696.objecter _send_command 6 2024-03-11T21:39:13.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df002d220 con 0x7f8df0101440 2024-03-11T21:39:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df002d220 con 0x7f8df0101440 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 4 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 2 6 2024-03-11T21:39:13.976 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 10 client.30696.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 7 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8df01a7650 0x7f8df0101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 7 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 3 7 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 4 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 10 client.30696.objecter _send_command 7 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df002d5e0 con 0x7f8df01a7650 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.994+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8df01a7650 0x7f8df0101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df01a7650 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8df01a7650 0x7f8df0101de0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8de800bd40 con 0x7f8df01a7650 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 7 = system:0 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 4 2024-03-11T21:39:13.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:13.998+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 3 7 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 10 client.30696.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 8 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 8 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df002d470 osd=3 3 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 3 8 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 4 2024-03-11T21:39:13.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 10 client.30696.objecter _send_command 8 2024-03-11T21:39:13.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df01a6c20 con 0x7f8df01a7650 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df01a6c20 con 0x7f8df01a7650 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 4 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 3 8 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:39:13.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 10 client.30696.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 9 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8df01b0db0 0x7f8df01b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 9 2024-03-11T21:39:14.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 4 9 2024-03-11T21:39:14.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 4 2024-03-11T21:39:14.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 10 client.30696.objecter _send_command 9 2024-03-11T21:39:14.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01a6be0 con 0x7f8df01b0db0 2024-03-11T21:39:14.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8df01b0db0 0x7f8df01b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df01b0db0 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.018+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8df01b0db0 0x7f8df01b3410 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.022+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01a6be0 con 0x7f8df01b0db0 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.022+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.022+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 9 = system:0 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.022+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 4 2024-03-11T21:39:14.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.022+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 4 9 2024-03-11T21:39:14.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 10 client.30696.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:14.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 10 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 10 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a69d0 osd=4 3 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 4 10 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 4 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 10 client.30696.objecter _send_command 10 2024-03-11T21:39:14.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.038+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df01a6be0 con 0x7f8df01b0db0 2024-03-11T21:39:14.021 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df01a6be0 con 0x7f8df01b0db0 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 4 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 4 10 2024-03-11T21:39:14.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 10 client.30696.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 11 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8df01b3e60 0x7f8df01b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 11 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 5 11 2024-03-11T21:39:14.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 4 2024-03-11T21:39:14.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 10 client.30696.objecter _send_command 11 2024-03-11T21:39:14.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01bb550 con 0x7f8df01b3e60 2024-03-11T21:39:14.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8df01b3e60 0x7f8df01b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:14.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df01b3e60 2024-03-11T21:39:14.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8def7fe700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8df01b3e60 0x7f8df01b42c0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01bb550 con 0x7f8df01b3e60 2024-03-11T21:39:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:39:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 11 = system:0 2024-03-11T21:39:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 4 2024-03-11T21:39:14.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.042+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 5 11 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 10 client.30696.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 12 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 12 2024-03-11T21:39:14.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a6be0 osd=5 3 2024-03-11T21:39:14.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 5 12 2024-03-11T21:39:14.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 4 2024-03-11T21:39:14.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 10 client.30696.objecter _send_command 12 2024-03-11T21:39:14.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.058+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df01b8c30 con 0x7f8df01b3e60 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df01b8c30 con 0x7f8df01b3e60 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 4 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 5 12 2024-03-11T21:39:14.043 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 10 client.30696.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 13 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8df01b92b0 0x7f8df01bb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 13 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 6 13 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 4 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 10 client.30696.objecter _send_command 13 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01be720 con 0x7f8df01b92b0 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8df01b92b0 0x7f8df01bb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df01b92b0 2024-03-11T21:39:14.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.062+0000 7f8df4a18700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8df01b92b0 0x7f8df01bb6a0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:14.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.066+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01be720 con 0x7f8df01b92b0 2024-03-11T21:39:14.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.066+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:39:14.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.066+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 13 = system:0 2024-03-11T21:39:14.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.066+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 4 2024-03-11T21:39:14.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.066+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 6 13 2024-03-11T21:39:14.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 10 client.30696.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:14.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 14 2024-03-11T21:39:14.063 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 14 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01b8c30 osd=6 3 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 6 14 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 4 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 10 client.30696.objecter _send_command 14 2024-03-11T21:39:14.064 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.082+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df01a6e00 con 0x7f8df01b92b0 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df01a6e00 con 0x7f8df01b92b0 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 4 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 6 14 2024-03-11T21:39:14.068 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:39:14.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 10 client.30696.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:14.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 15 2024-03-11T21:39:14.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8df01bda70 0x7f8df01bded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:14.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 15 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 7 15 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 4 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 10 client.30696.objecter _send_command 15 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.086+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df01c1af0 con 0x7f8df01bda70 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8df01bda70 0x7f8df01bded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8deffff700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8df01bda70 0x7f8df01bded0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:14.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 10 client.30696.objecter ms_handle_connect 0x7f8df01bda70 2024-03-11T21:39:14.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8df01c1af0 con 0x7f8df01bda70 2024-03-11T21:39:14.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:39:14.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 15 = system:0 2024-03-11T21:39:14.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 4 2024-03-11T21:39:14.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.090+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 7 15 2024-03-11T21:39:14.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 10 client.30696.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign -1 16 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 20 client.30696.objecter _get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 15 client.30696.objecter _session_command_op_remove -1 16 2024-03-11T21:39:14.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 20 client.30696.objecter get_session s=0x7f8df01a6e00 osd=7 3 2024-03-11T21:39:14.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 15 client.30696.objecter _session_command_op_assign 7 16 2024-03-11T21:39:14.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 4 2024-03-11T21:39:14.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 10 client.30696.objecter _send_command 16 2024-03-11T21:39:14.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.110+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8df01bbd00 con 0x7f8df01bda70 2024-03-11T21:39:14.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8ded7fa700 1 -- 172.21.15.143:0/2326911456 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8df01bbd00 con 0x7f8df01bda70 2024-03-11T21:39:14.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8ded7fa700 10 client.30696.objecter ms_dispatch 0x7f8df0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:14.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8ded7fa700 10 client.30696.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:14.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8ded7fa700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 4 2024-03-11T21:39:14.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8ded7fa700 15 client.30696.objecter _session_command_op_remove 7 16 2024-03-11T21:39:14.095 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:39:14.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 librados: watch_flush enter 2024-03-11T21:39:14.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 librados: watch_flush exit 2024-03-11T21:39:14.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.0 2024-03-11T21:39:14.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8df00629f0 msgr2=0x7f8df0062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f8df00629f0 0x7f8df0062dd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df0198f50 osd=0 1 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.1 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8df002c6b0 msgr2=0x7f8df002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f8df002c6b0 0x7f8df002ca90 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01093d0 osd=1 1 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.2 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8df0101440 msgr2=0x7f8df01018a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f8df0101440 0x7f8df01018a0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d7a0 osd=2 1 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.3 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8df01a7650 msgr2=0x7f8df0101de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8df01a7650 0x7f8df0101de0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df002d470 osd=3 1 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.4 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8df01b0db0 msgr2=0x7f8df01b3410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f8df01b0db0 0x7f8df01b3410 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a69d0 osd=4 1 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.5 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8df01b3e60 msgr2=0x7f8df01b42c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f8df01b3e60 0x7f8df01b42c0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6be0 osd=5 1 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.6 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8df01b92b0 msgr2=0x7f8df01bb6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f8df01b92b0 0x7f8df01bb6a0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01b8c30 osd=6 1 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter close_session for osd.7 2024-03-11T21:39:14.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8df01bda70 msgr2=0x7f8df01bded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f8df01bda70 0x7f8df01bded0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter put_session s=0x7f8df01a6e00 osd=7 1 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 20 client.30696.objecter shutdown clearing up homeless session... 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 10 client.30696.objecter successfully canceled tick 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8dd806f820 msgr2=0x7f8dd8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8dd806f820 0x7f8dd8071ce0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f8df00ffb30 tx=0x7f8de4009290 comp rx=0 tx=0).stop 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0100330 msgr2=0x7f8df019d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 --2- 172.21.15.143:0/2326911456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df0100330 0x7f8df019d860 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e9f0 tx=0x7f8de000ed00 comp rx=0 tx=0).stop 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df4a18700 1 -- 172.21.15.143:0/2326911456 reap_dead start 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df4a18700 1 -- 172.21.15.143:0/2326911456 reap_dead start 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df4a18700 1 -- 172.21.15.143:0/2326911456 reap_dead start 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df4a18700 1 -- 172.21.15.143:0/2326911456 reap_dead start 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df4a18700 1 -- 172.21.15.143:0/2326911456 reap_dead start 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 shutdown_connections 2024-03-11T21:39:14.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.114+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 >> 172.21.15.143:0/2326911456 conn(0x7f8df00772e0 msgr2=0x7f8df01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:14.100 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:14.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.118+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 shutdown_connections 2024-03-11T21:39:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:14.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.118+0000 7f8df647b700 1 -- 172.21.15.143:0/2326911456 wait complete. 2024-03-11T21:39:14.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:14.118+0000 7f8df647b700 1 librados: shutdown 2024-03-11T21:39:14.102 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:39:14.102 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:39:14.102 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:39:14.102 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:39:14.103 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:39:14.103 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:39:14.103 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:39:14.103 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:39:14.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:14.223 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:14.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:14.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:14.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:14.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:14.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:14.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[21512]: cluster 2024-03-11T21:39:13.537079+0000 mgr.y (mgr.14550) 636 : cluster 0 pgmap v1030: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:14.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[21512]: cluster 2024-03-11T21:39:13.922041+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:39:14.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[21512]: audit 2024-03-11T21:39:13.925953+0000 mon.a (mon.0) 3272 : audit 0 from='client.? 172.21.15.143:0/2326911456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:14.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[29284]: cluster 2024-03-11T21:39:13.537079+0000 mgr.y (mgr.14550) 636 : cluster 0 pgmap v1030: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:14.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[29284]: cluster 2024-03-11T21:39:13.922041+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:39:14.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:14 smithi143 bash[29284]: audit 2024-03-11T21:39:13.925953+0000 mon.a (mon.0) 3272 : audit 0 from='client.? 172.21.15.143:0/2326911456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:14.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:14.724 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:14 smithi203 bash[23654]: cluster 2024-03-11T21:39:13.537079+0000 mgr.y (mgr.14550) 636 : cluster 0 pgmap v1030: 214 pgs: 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:14.724 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:14 smithi203 bash[23654]: cluster 2024-03-11T21:39:13.922041+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:39:14.724 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:14 smithi203 bash[23654]: audit 2024-03-11T21:39:13.925953+0000 mon.a (mon.0) 3272 : audit 0 from='client.? 172.21.15.143:0/2326911456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:14.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:14.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:14.872 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:14.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:15.003 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:15.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:15.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.262 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:15.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:15.393 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:15.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:15 smithi203 bash[23654]: audit 2024-03-11T21:39:14.487066+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:15.509 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:15 smithi203 bash[23654]: cluster 2024-03-11T21:39:14.492882+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:39:15.521 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:15.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:15.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:15.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:15.784 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:15.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:15.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:15.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:15 smithi143 bash[21512]: audit 2024-03-11T21:39:14.487066+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:15.939 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:15 smithi143 bash[21512]: cluster 2024-03-11T21:39:14.492882+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:39:15.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:15 smithi143 bash[29284]: audit 2024-03-11T21:39:14.487066+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:15.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:15 smithi143 bash[29284]: cluster 2024-03-11T21:39:14.492882+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:39:16.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:16.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:16.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:16.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:16.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.307 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:16.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:16.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:16.554 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:16 smithi203 bash[23654]: cluster 2024-03-11T21:39:15.517272+0000 mon.a (mon.0) 3275 : cluster 0 osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:39:16.555 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:16 smithi203 bash[23654]: audit 2024-03-11T21:39:15.533939+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.555 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:16 smithi203 bash[23654]: audit 2024-03-11T21:39:15.536887+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.555 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:16 smithi203 bash[23654]: cluster 2024-03-11T21:39:15.537724+0000 mgr.y (mgr.14550) 637 : cluster 0 pgmap v1033: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:16.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:16.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:16.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:16.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:16.769 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[29284]: cluster 2024-03-11T21:39:15.517272+0000 mon.a (mon.0) 3275 : cluster 0 osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[29284]: audit 2024-03-11T21:39:15.533939+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[29284]: audit 2024-03-11T21:39:15.536887+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[29284]: cluster 2024-03-11T21:39:15.537724+0000 mgr.y (mgr.14550) 637 : cluster 0 pgmap v1033: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[21512]: cluster 2024-03-11T21:39:15.517272+0000 mon.a (mon.0) 3275 : cluster 0 osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[21512]: audit 2024-03-11T21:39:15.533939+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[21512]: audit 2024-03-11T21:39:15.536887+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:16.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:16 smithi143 bash[21512]: cluster 2024-03-11T21:39:15.537724+0000 mgr.y (mgr.14550) 637 : cluster 0 pgmap v1033: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:16.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.828 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:16.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:16.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:17.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:17.085 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:17.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:17.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:17.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:17.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:17.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.475 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:17.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.605 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:17.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.734 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: cluster 2024-03-11T21:39:16.237696+0000 osd.0 (osd.0) 364 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: cluster 2024-03-11T21:39:16.237913+0000 osd.0 (osd.0) 365 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: cluster 2024-03-11T21:39:16.502301+0000 mon.a (mon.0) 3277 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: audit 2024-03-11T21:39:16.536929+0000 mon.a (mon.0) 3278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: cluster 2024-03-11T21:39:16.564895+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: audit 2024-03-11T21:39:16.576153+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:17 smithi203 bash[23654]: audit 2024-03-11T21:39:16.577425+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: cluster 2024-03-11T21:39:16.237696+0000 osd.0 (osd.0) 364 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: cluster 2024-03-11T21:39:16.237913+0000 osd.0 (osd.0) 365 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: cluster 2024-03-11T21:39:16.502301+0000 mon.a (mon.0) 3277 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: audit 2024-03-11T21:39:16.536929+0000 mon.a (mon.0) 3278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: cluster 2024-03-11T21:39:16.564895+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: audit 2024-03-11T21:39:16.576153+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.850 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[29284]: audit 2024-03-11T21:39:16.577425+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.851 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: cluster 2024-03-11T21:39:16.237696+0000 osd.0 (osd.0) 364 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: cluster 2024-03-11T21:39:16.237913+0000 osd.0 (osd.0) 365 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: cluster 2024-03-11T21:39:16.502301+0000 mon.a (mon.0) 3277 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: audit 2024-03-11T21:39:16.536929+0000 mon.a (mon.0) 3278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: cluster 2024-03-11T21:39:16.564895+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: audit 2024-03-11T21:39:16.576153+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:17 smithi143 bash[21512]: audit 2024-03-11T21:39:16.577425+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:17.865 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:17.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:17.996 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:18.099 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:18.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:18.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:18.257 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:18.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:18.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:18.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:18.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:18.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:18.633 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[29284]: cluster 2024-03-11T21:39:17.538264+0000 mgr.y (mgr.14550) 638 : cluster 0 pgmap v1035: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:18.633 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[29284]: audit 2024-03-11T21:39:17.545768+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:18.633 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[29284]: audit 2024-03-11T21:39:17.551509+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.633 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[29284]: cluster 2024-03-11T21:39:17.554402+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:39:18.633 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[29284]: audit 2024-03-11T21:39:17.555734+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[21512]: cluster 2024-03-11T21:39:17.538264+0000 mgr.y (mgr.14550) 638 : cluster 0 pgmap v1035: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:18.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[21512]: audit 2024-03-11T21:39:17.545768+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:18.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[21512]: audit 2024-03-11T21:39:17.551509+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[21512]: cluster 2024-03-11T21:39:17.554402+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:39:18.634 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:18 smithi143 bash[21512]: audit 2024-03-11T21:39:17.555734+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:18.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:18.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[23654]: cluster 2024-03-11T21:39:17.538264+0000 mgr.y (mgr.14550) 638 : cluster 0 pgmap v1035: 254 pgs: 40 unknown, 214 active+clean; 582 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:18.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[23654]: audit 2024-03-11T21:39:17.545768+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:18.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[23654]: audit 2024-03-11T21:39:17.551509+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[23654]: cluster 2024-03-11T21:39:17.554402+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:39:18.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[23654]: audit 2024-03-11T21:39:17.555734+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:18.779 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:18.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:18.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:39:18.908 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:19.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:19.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:19.150 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:19.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.297 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:19.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:19.427 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:19.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.560 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:19.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:19.674 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:19 smithi203 bash[23654]: audit 2024-03-11T21:39:18.550197+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:19.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:19 smithi203 bash[23654]: cluster 2024-03-11T21:39:18.559658+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:39:19.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:19 smithi203 bash[23654]: audit 2024-03-11T21:39:18.561493+0000 mon.c (mon.2) 1147 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:19 smithi203 bash[23654]: audit 2024-03-11T21:39:18.569881+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.675 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:19 smithi203 bash[23654]: audit 2024-03-11T21:39:18.882345+0000 mgr.y (mgr.14550) 639 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:19.691 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:19.794 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[29284]: audit 2024-03-11T21:39:18.550197+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:19.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[29284]: cluster 2024-03-11T21:39:18.559658+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:39:19.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[29284]: audit 2024-03-11T21:39:18.561493+0000 mon.c (mon.2) 1147 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[29284]: audit 2024-03-11T21:39:18.569881+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.806 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[29284]: audit 2024-03-11T21:39:18.882345+0000 mgr.y (mgr.14550) 639 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:19.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[21512]: audit 2024-03-11T21:39:18.550197+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-109", "overlaypool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:19.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[21512]: cluster 2024-03-11T21:39:18.559658+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:39:19.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[21512]: audit 2024-03-11T21:39:18.561493+0000 mon.c (mon.2) 1147 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[21512]: audit 2024-03-11T21:39:18.569881+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:39:19.807 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:19 smithi143 bash[21512]: audit 2024-03-11T21:39:18.882345+0000 mgr.y (mgr.14550) 639 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:19.820 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:19.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.949 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:20.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:20.079 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:20.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:20.212 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:20.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:20.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:20.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:20.471 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:20.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: cluster 2024-03-11T21:39:19.539346+0000 mgr.y (mgr.14550) 640 : cluster 0 pgmap v1038: 254 pgs: 4 unknown, 250 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: cluster 2024-03-11T21:39:19.562934+0000 mon.a (mon.0) 3287 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: audit 2024-03-11T21:39:19.565324+0000 mon.a (mon.0) 3288 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: cluster 2024-03-11T21:39:19.570941+0000 mon.a (mon.0) 3289 : cluster 0 osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: audit 2024-03-11T21:39:19.576872+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[29284]: audit 2024-03-11T21:39:19.585275+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: cluster 2024-03-11T21:39:19.539346+0000 mgr.y (mgr.14550) 640 : cluster 0 pgmap v1038: 254 pgs: 4 unknown, 250 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: cluster 2024-03-11T21:39:19.562934+0000 mon.a (mon.0) 3287 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:20.586 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: audit 2024-03-11T21:39:19.565324+0000 mon.a (mon.0) 3288 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:20.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: cluster 2024-03-11T21:39:19.570941+0000 mon.a (mon.0) 3289 : cluster 0 osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:39:20.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: audit 2024-03-11T21:39:19.576872+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.587 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:20 smithi143 bash[21512]: audit 2024-03-11T21:39:19.585275+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.601 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:20.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: cluster 2024-03-11T21:39:19.539346+0000 mgr.y (mgr.14550) 640 : cluster 0 pgmap v1038: 254 pgs: 4 unknown, 250 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: cluster 2024-03-11T21:39:19.562934+0000 mon.a (mon.0) 3287 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: audit 2024-03-11T21:39:19.565324+0000 mon.a (mon.0) 3288 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: cluster 2024-03-11T21:39:19.570941+0000 mon.a (mon.0) 3289 : cluster 0 osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: audit 2024-03-11T21:39:19.576872+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:20 smithi203 bash[23654]: audit 2024-03-11T21:39:19.585275+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:39:20.732 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:20.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:20.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:20.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:20.992 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:21.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:21.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:21.256 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:21.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:21.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.515 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:21.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:21.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:20.579032+0000 mon.a (mon.0) 3291 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: cluster 2024-03-11T21:39:20.586676+0000 mon.a (mon.0) 3292 : cluster 0 osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:20.589957+0000 mon.c (mon.2) 1149 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:20.602219+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:21.466731+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: cluster 2024-03-11T21:39:21.478744+0000 mon.a (mon.0) 3295 : cluster 0 osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:21.480267+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.759 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:21 smithi203 bash[23654]: audit 2024-03-11T21:39:21.481190+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.775 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:20.579032+0000 mon.a (mon.0) 3291 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: cluster 2024-03-11T21:39:20.586676+0000 mon.a (mon.0) 3292 : cluster 0 osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:20.589957+0000 mon.c (mon.2) 1149 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:20.602219+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:21.466731+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: cluster 2024-03-11T21:39:21.478744+0000 mon.a (mon.0) 3295 : cluster 0 osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:21.480267+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.832 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[29284]: audit 2024-03-11T21:39:21.481190+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:20.579032+0000 mon.a (mon.0) 3291 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: cluster 2024-03-11T21:39:20.586676+0000 mon.a (mon.0) 3292 : cluster 0 osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:20.589957+0000 mon.c (mon.2) 1149 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:20.602219+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:21.466731+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: cluster 2024-03-11T21:39:21.478744+0000 mon.a (mon.0) 3295 : cluster 0 osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:21.480267+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.833 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:21 smithi143 bash[21512]: audit 2024-03-11T21:39:21.481190+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:39:21.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:21.908 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- 172.21.15.143:0/3891928245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494107ea0 msgr2=0x7fa494108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- 172.21.15.143:0/3891928245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494107ea0 0x7fa494108280 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fa490009990 tx=0x7fa490009ca0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- 172.21.15.143:0/3891928245 shutdown_connections 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- 172.21.15.143:0/3891928245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494107ea0 0x7fa494108280 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- 172.21.15.143:0/3891928245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494102830 0x7fa49410ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- 172.21.15.143:0/3891928245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494101e90 0x7fa4941022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- 172.21.15.143:0/3891928245 >> 172.21.15.143:0/3891928245 conn(0x7fa494078eb0 msgr2=0x7fa4940792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- 172.21.15.143:0/3891928245 shutdown_connections 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- 172.21.15.143:0/3891928245 wait complete. 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 librados: starting msgr at 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 librados: starting objecter 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 Processor -- start 2024-03-11T21:39:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- start start 2024-03-11T21:39:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 librados: setting wanted keys 2024-03-11T21:39:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 librados: calling monclient init 2024-03-11T21:39:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494102830 0x7fa49419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 0x7fa494197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa494111640 con 0x7fa494101e90 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941114c0 con 0x7fa494107ea0 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49bac4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941117c0 con 0x7fa494102830 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46378/0 (socket says 172.21.15.143:46378) 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 -- 172.21.15.143:0/454228596 learned_addr learned my addr 172.21.15.143:0/454228596 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49905f700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494102830 0x7fa49419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49a061700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 0x7fa494197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 -- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494102830 msgr2=0x7fa49419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494102830 0x7fa49419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 -- 172.21.15.143:0/454228596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 msgr2=0x7fa494197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 0x7fa494197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 -- 172.21.15.143:0/454228596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494198200 con 0x7fa494101e90 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa49a061700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 0x7fa494197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:39:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa499860700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fa4900057f0 tx=0x7fa490005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa48affd700 10 client.?.objecter ms_handle_connect 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa48affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490027070 con 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa490003f50 con 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.981+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490022b70 con 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa494198490 con 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49bac4700 10 client.30702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4941a47b0 con 0x7fa494101e90 2024-03-11T21:39:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49bac4700 1 librados: init done 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa494103550 con 0x7fa494101e90 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa490022d10 con 0x7fa494101e90 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 0x7fa480071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(674..674 src has 87..674) v4 ==== 11022+0+0 (secure 0 0 0) 0x7fa490090080 con 0x7fa494101e90 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 10 client.30702.objecter ms_dispatch 0x7fa494001550 osd_map(674..674 src has 87..674) v4 2024-03-11T21:39:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 3 client.30702.objecter handle_osd_map got epochs [674,674] > 0 2024-03-11T21:39:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 3 client.30702.objecter handle_osd_map decoding full epoch 674 2024-03-11T21:39:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49905f700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 0x7fa480071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 20 client.30702.objecter dump_active .. 0 homeless 2024-03-11T21:39:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa48affd700 10 client.30702.objecter ms_handle_connect 0x7fa48006f820 2024-03-11T21:39:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.985+0000 7fa49905f700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 0x7fa480071ce0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7fa494079840 tx=0x7fa484009190 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:21.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:21.993+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa490013070 con 0x7fa494101e90 2024-03-11T21:39:22.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:22.039 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:22.116 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:39:22.116 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:39:22.142 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:22.164 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:22.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.237+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fa494198fd0 con 0x7fa494101e90 2024-03-11T21:39:22.267 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:22.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/1934938962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 msgr2=0x7fc18810db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/1934938962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18810db80 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fc18000b0a0 tx=0x7fc18000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/1934938962 shutdown_connections 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/1934938962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18810db80 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/1934938962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188106dd0 0x7fc1881071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/1934938962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188100d20 0x7fc1881011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/1934938962 >> 172.21.15.143:0/1934938962 conn(0x7fc188077c80 msgr2=0x7fc188078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:22.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/1934938962 shutdown_connections 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/1934938962 wait complete. 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 librados: starting msgr at 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 librados: starting objecter 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 Processor -- start 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- start start 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 librados: setting wanted keys 2024-03-11T21:39:22.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 librados: calling monclient init 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18819ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188106dd0 0x7fc188199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1881104e0 con 0x7fc1881016e0 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188110360 con 0x7fc188106dd0 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc188110660 con 0x7fc188100d20 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:22.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc185d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18819ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60732/0 (socket says 172.21.15.143:60732) 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 -- 172.21.15.143:0/3299466557 learned_addr learned my addr 172.21.15.143:0/3299466557 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc186d9d700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188106dd0 0x7fc188199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 -- 172.21.15.143:0/3299466557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188106dd0 msgr2=0x7fc188199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188106dd0 0x7fc188199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 -- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 msgr2=0x7fc18819ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18819ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 -- 172.21.15.143:0/3299466557 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18819a410 con 0x7fc188100d20 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18659c700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fc17800eb30 tx=0x7fc17800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc17f7fe700 10 client.?.objecter ms_handle_connect 0x7fc188100d20 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc17f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17800cc90 con 0x7fc188100d20 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc17800ce30 con 0x7fc188100d20 2024-03-11T21:39:22.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178003df0 con 0x7fc188100d20 2024-03-11T21:39:22.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18819a700 con 0x7fc188100d20 2024-03-11T21:39:22.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 10 client.41909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:22.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1881a69e0 con 0x7fc188100d20 2024-03-11T21:39:22.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.325+0000 7fc18cbf6700 1 librados: init done 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc188102410 con 0x7fc188100d20 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc178003f90 con 0x7fc188100d20 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 0x7fc174071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(674..674 src has 87..674) v4 ==== 11022+0+0 (secure 0 0 0) 0x7fc17808f280 con 0x7fc188100d20 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 10 client.41909.objecter ms_dispatch 0x7fc188001550 osd_map(674..674 src has 87..674) v4 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 3 client.41909.objecter handle_osd_map got epochs [674,674] > 0 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 3 client.41909.objecter handle_osd_map decoding full epoch 674 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc185d9b700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 0x7fc174071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:22.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 20 client.41909.objecter dump_active .. 0 homeless 2024-03-11T21:39:22.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc17f7fe700 10 client.41909.objecter ms_handle_connect 0x7fc17406f820 2024-03-11T21:39:22.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.329+0000 7fc185d9b700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 0x7fc174071ce0 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fc18819b280 tx=0x7fc17000a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:22.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.333+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc17805b980 con 0x7fc188100d20 2024-03-11T21:39:22.391 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.400 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:22.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.469+0000 7fa48affd700 1 -- 172.21.15.143:0/454228596 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v675) v1 ==== 113+0+0 (secure 0 0 0) 0x7fa49005c760 con 0x7fa494101e90 2024-03-11T21:39:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 10 librados: watch_flush enter 2024-03-11T21:39:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 10 librados: watch_flush exit 2024-03-11T21:39:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 20 client.30702.objecter shutdown clearing up homeless session... 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 10 client.30702.objecter successfully canceled tick 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 msgr2=0x7fa480071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 0x7fa480071ce0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7fa494079840 tx=0x7fa484009190 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 msgr2=0x7fa49419d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fa4900057f0 tx=0x7fa490005e80 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 shutdown_connections 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa48006f820 0x7fa480071ce0 secure :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7fa494079840 tx=0x7fa484009190 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa494101e90 0x7fa49419d8e0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494107ea0 0x7fa494197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 --2- 172.21.15.143:0/454228596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa494102830 0x7fa49419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.473+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 >> 172.21.15.143:0/454228596 conn(0x7fa494078eb0 msgr2=0x7fa494105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.477+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 shutdown_connections 2024-03-11T21:39:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.477+0000 7fa49bac4700 1 -- 172.21.15.143:0/454228596 wait complete. 2024-03-11T21:39:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.477+0000 7fa49bac4700 1 librados: shutdown 2024-03-11T21:39:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:39:22.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:22.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:22.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:22.573+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc18819aff0 con 0x7fc188100d20 2024-03-11T21:39:22.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:22.665 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:22.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:22.796 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:22.899 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:23.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: cluster 2024-03-11T21:39:21.540322+0000 mgr.y (mgr.14550) 641 : cluster 0 pgmap v1042: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.004013+0000 mon.a (mon.0) 3297 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.006589+0000 mon.c (mon.2) 1151 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.241393+0000 mon.a (mon.0) 3298 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: cluster 2024-03-11T21:39:22.467359+0000 mon.a (mon.0) 3299 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.474305+0000 mon.a (mon.0) 3300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.474416+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: cluster 2024-03-11T21:39:22.478522+0000 mon.a (mon.0) 3302 : cluster 0 osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:39:23.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.479845+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.480280+0000 mon.a (mon.0) 3303 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.578601+0000 mon.c (mon.2) 1153 : audit 1 from='client.? 172.21.15.143:0/3299466557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.042 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:23 smithi203 bash[23654]: audit 2024-03-11T21:39:22.578899+0000 mon.a (mon.0) 3304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.056 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:23.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: cluster 2024-03-11T21:39:21.540322+0000 mgr.y (mgr.14550) 641 : cluster 0 pgmap v1042: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.004013+0000 mon.a (mon.0) 3297 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.006589+0000 mon.c (mon.2) 1151 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.241393+0000 mon.a (mon.0) 3298 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: cluster 2024-03-11T21:39:22.467359+0000 mon.a (mon.0) 3299 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.474305+0000 mon.a (mon.0) 3300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.474416+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: cluster 2024-03-11T21:39:22.478522+0000 mon.a (mon.0) 3302 : cluster 0 osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:39:23.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.479845+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.480280+0000 mon.a (mon.0) 3303 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.578601+0000 mon.c (mon.2) 1153 : audit 1 from='client.? 172.21.15.143:0/3299466557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[21512]: audit 2024-03-11T21:39:22.578899+0000 mon.a (mon.0) 3304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: cluster 2024-03-11T21:39:21.540322+0000 mgr.y (mgr.14550) 641 : cluster 0 pgmap v1042: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.004013+0000 mon.a (mon.0) 3297 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.006589+0000 mon.c (mon.2) 1151 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.241393+0000 mon.a (mon.0) 3298 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: cluster 2024-03-11T21:39:22.467359+0000 mon.a (mon.0) 3299 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:23.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.474305+0000 mon.a (mon.0) 3300 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.474416+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.143:0/454228596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: cluster 2024-03-11T21:39:22.478522+0000 mon.a (mon.0) 3302 : cluster 0 osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.479845+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.480280+0000 mon.a (mon.0) 3303 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.578601+0000 mon.c (mon.2) 1153 : audit 1 from='client.? 172.21.15.143:0/3299466557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.174 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:23 smithi143 bash[29284]: audit 2024-03-11T21:39:22.578899+0000 mon.a (mon.0) 3304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:23.187 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:23.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:23.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:23.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:23.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.485+0000 7fc17f7fe700 1 -- 172.21.15.143:0/3299466557 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v676) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc17805f3d0 con 0x7fc188100d20 2024-03-11T21:39:23.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 10 librados: watch_flush enter 2024-03-11T21:39:23.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 10 librados: watch_flush exit 2024-03-11T21:39:23.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 20 client.41909.objecter shutdown clearing up homeless session... 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 10 client.41909.objecter successfully canceled tick 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 msgr2=0x7fc174071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 0x7fc174071ce0 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fc18819b280 tx=0x7fc17000a380 comp rx=0 tx=0).stop 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 msgr2=0x7fc18819fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fc17800eb30 tx=0x7fc17800ee40 comp rx=0 tx=0).stop 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 shutdown_connections 2024-03-11T21:39:23.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc17406f820 0x7fc174071ce0 unknown :-1 s=CLOSED pgs=3428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:23.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188106dd0 0x7fc188199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:23.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc188100d20 0x7fc18819fa60 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:23.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 --2- 172.21.15.143:0/3299466557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1881016e0 0x7fc18819ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:23.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 >> 172.21.15.143:0/3299466557 conn(0x7fc188077c80 msgr2=0x7fc18806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:23.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 shutdown_connections 2024-03-11T21:39:23.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 -- 172.21.15.143:0/3299466557 wait complete. 2024-03-11T21:39:23.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:23.493+0000 7fc18cbf6700 1 librados: shutdown 2024-03-11T21:39:23.478 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:39:23.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:23.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:23.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:23.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:24.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.098 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:24.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:39:24.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: cluster 2024-03-11T21:39:23.475095+0000 mon.a (mon.0) 3305 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: audit 2024-03-11T21:39:23.478732+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: audit 2024-03-11T21:39:23.478956+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: cluster 2024-03-11T21:39:23.490431+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: audit 2024-03-11T21:39:23.551986+0000 mon.c (mon.2) 1154 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[21512]: audit 2024-03-11T21:39:23.552402+0000 mon.a (mon.0) 3309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: cluster 2024-03-11T21:39:23.475095+0000 mon.a (mon.0) 3305 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: audit 2024-03-11T21:39:23.478732+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: audit 2024-03-11T21:39:23.478956+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: cluster 2024-03-11T21:39:23.490431+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: audit 2024-03-11T21:39:23.551986+0000 mon.c (mon.2) 1154 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:24 smithi143 bash[29284]: audit 2024-03-11T21:39:23.552402+0000 mon.a (mon.0) 3309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: cluster 2024-03-11T21:39:23.475095+0000 mon.a (mon.0) 3305 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: audit 2024-03-11T21:39:23.478732+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: audit 2024-03-11T21:39:23.478956+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: cluster 2024-03-11T21:39:23.490431+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: audit 2024-03-11T21:39:23.551986+0000 mon.c (mon.2) 1154 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:24 smithi203 bash[23654]: audit 2024-03-11T21:39:23.552402+0000 mon.a (mon.0) 3309 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:24.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:24.357 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:24.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:24.489 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:24.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.618 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:24.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:24.750 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:24.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:24.879 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:24.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:25.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:25.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:25.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:25.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:25 smithi203 bash[23654]: cluster 2024-03-11T21:39:23.540873+0000 mgr.y (mgr.14550) 642 : cluster 0 pgmap v1045: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:25 smithi203 bash[23654]: audit 2024-03-11T21:39:24.487513+0000 mon.a (mon.0) 3310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:25 smithi203 bash[23654]: cluster 2024-03-11T21:39:24.494121+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:39:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:25 smithi203 bash[23654]: audit 2024-03-11T21:39:24.496260+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.257 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:25 smithi203 bash[23654]: audit 2024-03-11T21:39:24.497133+0000 mon.a (mon.0) 3312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:25.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:25.399 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:25.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[21512]: cluster 2024-03-11T21:39:23.540873+0000 mgr.y (mgr.14550) 642 : cluster 0 pgmap v1045: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:25.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[21512]: audit 2024-03-11T21:39:24.487513+0000 mon.a (mon.0) 3310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:25.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[21512]: cluster 2024-03-11T21:39:24.494121+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:39:25.436 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[21512]: audit 2024-03-11T21:39:24.496260+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[21512]: audit 2024-03-11T21:39:24.497133+0000 mon.a (mon.0) 3312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[29284]: cluster 2024-03-11T21:39:23.540873+0000 mgr.y (mgr.14550) 642 : cluster 0 pgmap v1045: 254 pgs: 254 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[29284]: audit 2024-03-11T21:39:24.487513+0000 mon.a (mon.0) 3310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[29284]: cluster 2024-03-11T21:39:24.494121+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[29284]: audit 2024-03-11T21:39:24.496260+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:25 smithi143 bash[29284]: audit 2024-03-11T21:39:24.497133+0000 mon.a (mon.0) 3312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]: dispatch 2024-03-11T21:39:25.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:39:25.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:25.523 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:25.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:25.645 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:25.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.776 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:25.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:25.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:25.955 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:26.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:26.033 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:26.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.164 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:26.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:26.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:26.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:26.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:26.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:26.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:26.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:26.671 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:26 smithi203 bash[23654]: audit 2024-03-11T21:39:25.491841+0000 mon.a (mon.0) 3313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:26.671 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:26 smithi203 bash[23654]: cluster 2024-03-11T21:39:25.495688+0000 mon.a (mon.0) 3314 : cluster 0 osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:39:26.672 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:26 smithi203 bash[23654]: cluster 2024-03-11T21:39:25.541488+0000 mgr.y (mgr.14550) 643 : cluster 0 pgmap v1048: 254 pgs: 254 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:26.672 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:26 smithi203 bash[23654]: cluster 2024-03-11T21:39:26.465605+0000 mon.a (mon.0) 3315 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:26.686 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:26.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.819 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:26.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:26.932 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[29284]: audit 2024-03-11T21:39:25.491841+0000 mon.a (mon.0) 3313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:26.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[29284]: cluster 2024-03-11T21:39:25.495688+0000 mon.a (mon.0) 3314 : cluster 0 osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:39:26.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[29284]: cluster 2024-03-11T21:39:25.541488+0000 mgr.y (mgr.14550) 643 : cluster 0 pgmap v1048: 254 pgs: 254 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:26.933 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[29284]: cluster 2024-03-11T21:39:26.465605+0000 mon.a (mon.0) 3315 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:26.933 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[21512]: audit 2024-03-11T21:39:25.491841+0000 mon.a (mon.0) 3313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-109", "tierpool": "test-rados-api-smithi143-71773-109-cache"}]': finished 2024-03-11T21:39:26.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[21512]: cluster 2024-03-11T21:39:25.495688+0000 mon.a (mon.0) 3314 : cluster 0 osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:39:26.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[21512]: cluster 2024-03-11T21:39:25.541488+0000 mgr.y (mgr.14550) 643 : cluster 0 pgmap v1048: 254 pgs: 254 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:26.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:26 smithi143 bash[21512]: cluster 2024-03-11T21:39:26.465605+0000 mon.a (mon.0) 3315 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:26.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:27.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:27.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:27.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:27.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:27.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:27.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:27.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:27.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:27.488 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:39:27.512 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:27 smithi143 bash[21512]: cluster 2024-03-11T21:39:26.516909+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:39:27.512 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:27 smithi143 bash[29284]: cluster 2024-03-11T21:39:26.516909+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:39:27.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:27.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:27.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:27.716 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:27.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:27 smithi203 bash[23654]: cluster 2024-03-11T21:39:26.516909+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:39:27.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:27.846 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:27.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:27.976 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:28.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:28.104 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:28.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:28.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:28.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:28.365 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:28.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:28.494 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:28.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:28.623 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[29284]: cluster 2024-03-11T21:39:27.513610+0000 mon.a (mon.0) 3317 : cluster 0 osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[29284]: audit 2024-03-11T21:39:27.519273+0000 mon.c (mon.2) 1156 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[29284]: audit 2024-03-11T21:39:27.530794+0000 mon.a (mon.0) 3318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[29284]: cluster 2024-03-11T21:39:27.541999+0000 mgr.y (mgr.14550) 644 : cluster 0 pgmap v1051: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[21512]: cluster 2024-03-11T21:39:27.513610+0000 mon.a (mon.0) 3317 : cluster 0 osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[21512]: audit 2024-03-11T21:39:27.519273+0000 mon.c (mon.2) 1156 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[21512]: audit 2024-03-11T21:39:27.530794+0000 mon.a (mon.0) 3318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:28 smithi143 bash[21512]: cluster 2024-03-11T21:39:27.541999+0000 mgr.y (mgr.14550) 644 : cluster 0 pgmap v1051: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:28.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:28.754 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:28.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:28 smithi203 bash[23654]: cluster 2024-03-11T21:39:27.513610+0000 mon.a (mon.0) 3317 : cluster 0 osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:39:28.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:28 smithi203 bash[23654]: audit 2024-03-11T21:39:27.519273+0000 mon.c (mon.2) 1156 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:28 smithi203 bash[23654]: audit 2024-03-11T21:39:27.530794+0000 mon.a (mon.0) 3318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:28.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:28 smithi203 bash[23654]: cluster 2024-03-11T21:39:27.541999+0000 mgr.y (mgr.14550) 644 : cluster 0 pgmap v1051: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:28.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:28.877 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:28.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:29.010 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:29.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:29.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:29.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:29.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:29.273 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:29.375 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.404 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:29.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:29.540 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:29.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:29.671 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[29284]: audit 2024-03-11T21:39:28.530808+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[29284]: cluster 2024-03-11T21:39:28.537180+0000 mon.a (mon.0) 3320 : cluster 0 osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[29284]: audit 2024-03-11T21:39:28.539814+0000 mon.c (mon.2) 1157 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[29284]: audit 2024-03-11T21:39:28.540185+0000 mon.a (mon.0) 3321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[29284]: audit 2024-03-11T21:39:28.893595+0000 mgr.y (mgr.14550) 645 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[21512]: audit 2024-03-11T21:39:28.530808+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[21512]: cluster 2024-03-11T21:39:28.537180+0000 mon.a (mon.0) 3320 : cluster 0 osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[21512]: audit 2024-03-11T21:39:28.539814+0000 mon.c (mon.2) 1157 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[21512]: audit 2024-03-11T21:39:28.540185+0000 mon.a (mon.0) 3321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:29 smithi143 bash[21512]: audit 2024-03-11T21:39:28.893595+0000 mgr.y (mgr.14550) 645 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:29.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:29.786 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:29 smithi203 bash[23654]: audit 2024-03-11T21:39:28.530808+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:29.786 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:29 smithi203 bash[23654]: cluster 2024-03-11T21:39:28.537180+0000 mon.a (mon.0) 3320 : cluster 0 osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:39:29.786 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:29 smithi203 bash[23654]: audit 2024-03-11T21:39:28.539814+0000 mon.c (mon.2) 1157 : audit 1 from='client.? 172.21.15.143:0/3838231862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.787 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:29 smithi203 bash[23654]: audit 2024-03-11T21:39:28.540185+0000 mon.a (mon.0) 3321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]: dispatch 2024-03-11T21:39:29.787 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:29 smithi203 bash[23654]: audit 2024-03-11T21:39:28.893595+0000 mgr.y (mgr.14550) 645 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:29.801 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:29.904 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:29.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:30.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.068 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:30.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:30.198 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:30.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:30.328 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:30.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:30.457 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:30.560 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:30.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:30.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:30.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[21512]: audit 2024-03-11T21:39:29.534632+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:30.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[21512]: cluster 2024-03-11T21:39:29.542823+0000 mgr.y (mgr.14550) 646 : cluster 0 pgmap v1054: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:30.697 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[21512]: cluster 2024-03-11T21:39:29.543789+0000 mon.a (mon.0) 3323 : cluster 0 osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:39:30.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[29284]: audit 2024-03-11T21:39:29.534632+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:30.697 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[29284]: cluster 2024-03-11T21:39:29.542823+0000 mgr.y (mgr.14550) 646 : cluster 0 pgmap v1054: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:30.698 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:30 smithi143 bash[29284]: cluster 2024-03-11T21:39:29.543789+0000 mon.a (mon.0) 3323 : cluster 0 osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:39:30.713 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:30.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:30 smithi203 bash[23654]: audit 2024-03-11T21:39:29.534632+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-109"}]': finished 2024-03-11T21:39:30.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:30 smithi203 bash[23654]: cluster 2024-03-11T21:39:29.542823+0000 mgr.y (mgr.14550) 646 : cluster 0 pgmap v1054: 214 pgs: 214 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:30.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:30 smithi203 bash[23654]: cluster 2024-03-11T21:39:29.543789+0000 mon.a (mon.0) 3323 : cluster 0 osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:39:30.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:30.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:30.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:31.068 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:31.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:31.102 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- 172.21.15.143:0/3815964651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 msgr2=0x7fa4b8106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/3815964651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b8106b40 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009990 tx=0x7fa4a0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- 172.21.15.143:0/3815964651 shutdown_connections 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/3815964651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b8107220 0x7fa4b810edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/3815964651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b806a6a0 0x7fa4b806aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/3815964651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b8106b40 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- 172.21.15.143:0/3815964651 >> 172.21.15.143:0/3815964651 conn(0x7fa4b8077de0 msgr2=0x7fa4b80781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:31.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- 172.21.15.143:0/3815964651 shutdown_connections 2024-03-11T21:39:31.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- 172.21.15.143:0/3815964651 wait complete. 2024-03-11T21:39:31.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 librados: starting msgr at 2024-03-11T21:39:31.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 librados: starting objecter 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 Processor -- start 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- start start 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 librados: setting wanted keys 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 librados: calling monclient init 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 0x7fa4b819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b8107220 0x7fa4b8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b81116d0 con 0x7fa4b806b050 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8111550 con 0x7fa4b8107220 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4bcb11700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8111850 con 0x7fa4b806a6a0 2024-03-11T21:39:31.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 0x7fa4b819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 0x7fa4b819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37680/0 (socket says 172.21.15.143:37680) 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54826/0 (socket says 172.21.15.143:54826) 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 -- 172.21.15.143:0/54198219 learned_addr learned my addr 172.21.15.143:0/54198219 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b8107220 0x7fa4b8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b8107220 msgr2=0x7fa4b8197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b8107220 0x7fa4b8197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 msgr2=0x7fa4b819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b8198410 con 0x7fa4b806a6a0 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b806b050 0x7fa4b819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 0x7fa4b819da60 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800b530 tx=0x7fa4a800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4af7fe700 10 client.?.objecter ms_handle_connect 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4af7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.141+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8015020 con 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa4a8011cb0 con 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8010d90 con 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b8198700 con 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 10 client.41915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b81a49e0 con 0x7fa4b806a6a0 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 librados: init done 2024-03-11T21:39:31.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa4b81035b0 con 0x7fa4b806a6a0 2024-03-11T21:39:31.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa4a8018020 con 0x7fa4b806a6a0 2024-03-11T21:39:31.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa4a406f820 0x7fa4a4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(683..683 src has 87..683) v4 ==== 9353+0+0 (secure 0 0 0) 0x7fa4a808e870 con 0x7fa4b806a6a0 2024-03-11T21:39:31.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 osd_map(683..683 src has 87..683) v4 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 3 client.41915.objecter handle_osd_map got epochs [683,683] > 0 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa4a406f820 0x7fa4a4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 3 client.41915.objecter handle_osd_map decoding full epoch 683 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 20 client.41915.objecter dump_active .. 0 homeless 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v683) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa4a801f020 con 0x7fa4b806a6a0 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 1 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa4b80629f0 0x7fa4b8062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa4a406f820 0x7fa4a4071ce0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8199310 tx=0x7fa4a001a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 1 2024-03-11T21:39:31.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 0 1 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 4 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 10 client.41915.objecter _send_command 1 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b804faa0 con 0x7fa4b80629f0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4a406f820 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa4b80629f0 0x7fa4b8062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b80629f0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.145+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa4b80629f0 0x7fa4b8062dd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.149+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b804faa0 con 0x7fa4b80629f0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.149+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.149+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 1 = system:0 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.149+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 4 2024-03-11T21:39:31.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.149+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 0 1 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 2 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 2 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b8198ff0 osd=0 3 2024-03-11T21:39:31.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 0 2 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 4 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 10 client.41915.objecter _send_command 2 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b8062470 con 0x7fa4b80629f0 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b8062470 con 0x7fa4b80629f0 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 4 2024-03-11T21:39:31.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 0 2 2024-03-11T21:39:31.149 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 3 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa4b8109960 0x7fa4b81a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 3 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 1 3 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 4 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 10 client.41915.objecter _send_command 3 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.165+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b8102cd0 con 0x7fa4b8109960 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa4b8109960 0x7fa4b81a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b8109960 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa4b8109960 0x7fa4b81a6bc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b8102cd0 con 0x7fa4b8109960 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 3 = system:0 2024-03-11T21:39:31.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 4 2024-03-11T21:39:31.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.169+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 1 3 2024-03-11T21:39:31.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 4 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 4 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b802d830 osd=1 3 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 1 4 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 4 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 10 client.41915.objecter _send_command 4 2024-03-11T21:39:31.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.185+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b802dbf0 con 0x7fa4b8109960 2024-03-11T21:39:31.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b802dbf0 con 0x7fa4b8109960 2024-03-11T21:39:31.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 4 2024-03-11T21:39:31.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 1 4 2024-03-11T21:39:31.170 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 5 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa4b8109040 0x7fa4b81aa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 5 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 2 5 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 4 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 10 client.41915.objecter _send_command 5 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b802d390 con 0x7fa4b8109040 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa4b8109040 0x7fa4b81aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b8109040 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa4b8109040 0x7fa4b81aa290 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b802d390 con 0x7fa4b8109040 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 5 = system:0 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 4 2024-03-11T21:39:31.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.189+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 2 5 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 6 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 6 2024-03-11T21:39:31.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b8108ed0 osd=2 3 2024-03-11T21:39:31.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 2 6 2024-03-11T21:39:31.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 4 2024-03-11T21:39:31.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 10 client.41915.objecter _send_command 6 2024-03-11T21:39:31.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.205+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81aa7d0 con 0x7fa4b8109040 2024-03-11T21:39:31.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81aa7d0 con 0x7fa4b8109040 2024-03-11T21:39:31.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 4 2024-03-11T21:39:31.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 2 6 2024-03-11T21:39:31.190 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:39:31.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 7 2024-03-11T21:39:31.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa4b81a7de0 0x7fa4b81abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 7 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 3 7 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 4 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 10 client.41915.objecter _send_command 7 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b802cbd0 con 0x7fa4b81a7de0 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa4b81a7de0 0x7fa4b81abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b81a7de0 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa4b81a7de0 0x7fa4b81abb60 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b802cbd0 con 0x7fa4b81a7de0 2024-03-11T21:39:31.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:39:31.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 7 = system:0 2024-03-11T21:39:31.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 4 2024-03-11T21:39:31.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.209+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 3 7 2024-03-11T21:39:31.205 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:31.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 8 2024-03-11T21:39:31.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 8 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b802c900 osd=3 3 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 3 8 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 4 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 10 client.41915.objecter _send_command 8 2024-03-11T21:39:31.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81ac0a0 con 0x7fa4b81a7de0 2024-03-11T21:39:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81ac0a0 con 0x7fa4b81a7de0 2024-03-11T21:39:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 4 2024-03-11T21:39:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 3 8 2024-03-11T21:39:31.212 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:39:31.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 9 2024-03-11T21:39:31.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa4b81ac7b0 0x7fa4b81acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 9 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 4 9 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 4 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 10 client.41915.objecter _send_command 9 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b81b3d70 con 0x7fa4b81ac7b0 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa4b81ac7b0 0x7fa4b81acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b81ac7b0 2024-03-11T21:39:31.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.229+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa4b81ac7b0 0x7fa4b81acbf0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.233+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b81b3d70 con 0x7fa4b81ac7b0 2024-03-11T21:39:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.233+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:39:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.233+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 9 = system:0 2024-03-11T21:39:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.233+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 4 2024-03-11T21:39:31.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.233+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 4 9 2024-03-11T21:39:31.224 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 10 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 10 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81ac0a0 osd=4 3 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 4 10 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 4 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 10 client.41915.objecter _send_command 10 2024-03-11T21:39:31.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.249+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81b13c0 con 0x7fa4b81ac7b0 2024-03-11T21:39:31.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81b13c0 con 0x7fa4b81ac7b0 2024-03-11T21:39:31.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 4 2024-03-11T21:39:31.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 4 10 2024-03-11T21:39:31.238 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:39:31.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 11 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa4b81b1a40 0x7fa4b81b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 11 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 5 11 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 4 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 10 client.41915.objecter _send_command 11 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b81b7110 con 0x7fa4b81b1a40 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa4b81b1a40 0x7fa4b81b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b81b1a40 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4affff700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa4b81b1a40 0x7fa4b81b1ec0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b81b7110 con 0x7fa4b81b1a40 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 11 = system:0 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 4 2024-03-11T21:39:31.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.257+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 5 11 2024-03-11T21:39:31.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 12 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 12 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b13c0 osd=5 3 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 5 12 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 4 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 10 client.41915.objecter _send_command 12 2024-03-11T21:39:31.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.277+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81b5f90 con 0x7fa4b81b1a40 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81b5f90 con 0x7fa4b81b1a40 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 4 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 5 12 2024-03-11T21:39:31.262 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 13 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa4b81b66a0 0x7fa4b81b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 13 2024-03-11T21:39:31.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 6 13 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 4 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 10 client.41915.objecter _send_command 13 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b81ba500 con 0x7fa4b81b66a0 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa4b81b66a0 0x7fa4b81b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4b6d9d700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa4b81b66a0 0x7fa4b81b6ae0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.281+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b81b66a0 2024-03-11T21:39:31.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.285+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b81ba500 con 0x7fa4b81b66a0 2024-03-11T21:39:31.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.285+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:39:31.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.285+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 13 = system:0 2024-03-11T21:39:31.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.285+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 4 2024-03-11T21:39:31.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.285+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 6 13 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 14 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 14 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b5f90 osd=6 3 2024-03-11T21:39:31.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 6 14 2024-03-11T21:39:31.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 4 2024-03-11T21:39:31.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 10 client.41915.objecter _send_command 14 2024-03-11T21:39:31.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.301+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81b2450 con 0x7fa4b81b66a0 2024-03-11T21:39:31.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81b2450 con 0x7fa4b81b66a0 2024-03-11T21:39:31.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 4 2024-03-11T21:39:31.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 6 14 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 15 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa4b81b2ad0 0x7fa4b81b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 15 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 7 15 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 4 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 10 client.41915.objecter _send_command 15 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b81b3620 con 0x7fa4b81b2ad0 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa4b81b2ad0 0x7fa4b81b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4b659c700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa4b81b2ad0 0x7fa4b81b2ef0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:31.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.305+0000 7fa4af7fe700 10 client.41915.objecter ms_handle_connect 0x7fa4b81b2ad0 2024-03-11T21:39:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.309+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa4b81b3620 con 0x7fa4b81b2ad0 2024-03-11T21:39:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.309+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:39:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.309+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 15 = system:0 2024-03-11T21:39:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.309+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 4 2024-03-11T21:39:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.309+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 7 15 2024-03-11T21:39:31.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 10 client.41915.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign -1 16 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 20 client.41915.objecter _get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_remove -1 16 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 20 client.41915.objecter get_session s=0x7fa4b81b2450 osd=7 3 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 15 client.41915.objecter _session_command_op_assign 7 16 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 4 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 10 client.41915.objecter _send_command 16 2024-03-11T21:39:31.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.325+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4b81b3830 con 0x7fa4b81b2ad0 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4af7fe700 1 -- 172.21.15.143:0/54198219 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa4b81b3830 con 0x7fa4b81b2ad0 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4af7fe700 10 client.41915.objecter ms_dispatch 0x7fa4b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4af7fe700 10 client.41915.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4af7fe700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 4 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4af7fe700 15 client.41915.objecter _session_command_op_remove 7 16 2024-03-11T21:39:31.312 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:39:31.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 librados: watch_flush enter 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 librados: watch_flush exit 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.0 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa4b80629f0 msgr2=0x7fa4b8062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa4b80629f0 0x7fa4b8062dd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8198ff0 osd=0 1 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.1 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa4b8109960 msgr2=0x7fa4b81a6bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa4b8109960 0x7fa4b81a6bc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802d830 osd=1 1 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.2 2024-03-11T21:39:31.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa4b8109040 msgr2=0x7fa4b81aa290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa4b8109040 0x7fa4b81aa290 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b8108ed0 osd=2 1 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.3 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa4b81a7de0 msgr2=0x7fa4b81abb60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa4b81a7de0 0x7fa4b81abb60 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b802c900 osd=3 1 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.4 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa4b81ac7b0 msgr2=0x7fa4b81acbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa4b81ac7b0 0x7fa4b81acbf0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81ac0a0 osd=4 1 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.5 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa4b81b1a40 msgr2=0x7fa4b81b1ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa4b81b1a40 0x7fa4b81b1ec0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b13c0 osd=5 1 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.6 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa4b81b66a0 msgr2=0x7fa4b81b6ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa4b81b66a0 0x7fa4b81b6ae0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b5f90 osd=6 1 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter close_session for osd.7 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa4b81b2ad0 msgr2=0x7fa4b81b2ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa4b81b2ad0 0x7fa4b81b2ef0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter put_session s=0x7fa4b81b2450 osd=7 1 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 20 client.41915.objecter shutdown clearing up homeless session... 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 10 client.41915.objecter successfully canceled tick 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4b6d9d700 1 -- 172.21.15.143:0/54198219 reap_dead start 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4b6d9d700 1 -- 172.21.15.143:0/54198219 reap_dead start 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa4a406f820 msgr2=0x7fa4a4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4b6d9d700 1 -- 172.21.15.143:0/54198219 reap_dead start 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa4a406f820 0x7fa4a4071ce0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8199310 tx=0x7fa4a001a040 comp rx=0 tx=0).stop 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4b6d9d700 1 -- 172.21.15.143:0/54198219 reap_dead start 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 msgr2=0x7fa4b819da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4bcb11700 1 --2- 172.21.15.143:0/54198219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4b806a6a0 0x7fa4b819da60 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800b530 tx=0x7fa4a800b840 comp rx=0 tx=0).stop 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.329+0000 7fa4b6d9d700 1 -- 172.21.15.143:0/54198219 reap_dead start 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.333+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 shutdown_connections 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.333+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 >> 172.21.15.143:0/54198219 conn(0x7fa4b8077de0 msgr2=0x7fa4b80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.333+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 shutdown_connections 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.333+0000 7fa4bcb11700 1 -- 172.21.15.143:0/54198219 wait complete. 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:31.333+0000 7fa4bcb11700 1 librados: shutdown 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:39:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:39:31.318 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:39:31.318 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:39:31.318 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:39:31.318 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:39:31.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.346 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:31.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.478 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:31.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: cluster 2024-03-11T21:39:30.560234+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:39:31.585 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: audit 2024-03-11T21:39:30.571325+0000 mon.a (mon.0) 3325 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.609 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: audit 2024-03-11T21:39:31.149191+0000 mon.c (mon.2) 1158 : audit 0 from='client.? 172.21.15.143:0/54198219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: cluster 2024-03-11T21:39:31.466950+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: audit 2024-03-11T21:39:31.470398+0000 mon.a (mon.0) 3327 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: cluster 2024-03-11T21:39:31.478756+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:39:31.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[21512]: audit 2024-03-11T21:39:31.479697+0000 mon.a (mon.0) 3329 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: cluster 2024-03-11T21:39:30.560234+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: audit 2024-03-11T21:39:30.571325+0000 mon.a (mon.0) 3325 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: audit 2024-03-11T21:39:31.149191+0000 mon.c (mon.2) 1158 : audit 0 from='client.? 172.21.15.143:0/54198219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: cluster 2024-03-11T21:39:31.466950+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: audit 2024-03-11T21:39:31.470398+0000 mon.a (mon.0) 3327 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: cluster 2024-03-11T21:39:31.478756+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:39:31.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:31 smithi143 bash[29284]: audit 2024-03-11T21:39:31.479697+0000 mon.a (mon.0) 3329 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.739 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: cluster 2024-03-11T21:39:30.560234+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: audit 2024-03-11T21:39:30.571325+0000 mon.a (mon.0) 3325 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: audit 2024-03-11T21:39:31.149191+0000 mon.c (mon.2) 1158 : audit 0 from='client.? 172.21.15.143:0/54198219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: cluster 2024-03-11T21:39:31.466950+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: audit 2024-03-11T21:39:31.470398+0000 mon.a (mon.0) 3327 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:31.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: cluster 2024-03-11T21:39:31.478756+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:39:31.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:31 smithi203 bash[23654]: audit 2024-03-11T21:39:31.479697+0000 mon.a (mon.0) 3329 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]: dispatch 2024-03-11T21:39:31.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.866 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:31.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:31.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:32.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:32.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:32.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.243 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:32.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:32.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:32.513 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:32.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:32.642 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:32.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: cluster 2024-03-11T21:39:31.543478+0000 mgr.y (mgr.14550) 647 : cluster 0 pgmap v1057: 206 pgs: 206 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:32.742 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.472731+0000 mon.a (mon.0) 3330 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:32.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:32.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: cluster 2024-03-11T21:39:31.543478+0000 mgr.y (mgr.14550) 647 : cluster 0 pgmap v1057: 206 pgs: 206 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:32.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.472731+0000 mon.a (mon.0) 3330 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:32.757 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: cluster 2024-03-11T21:39:32.479026+0000 mon.a (mon.0) 3331 : cluster 0 osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.499925+0000 mon.c (mon.2) 1159 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.500310+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.501358+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.501659+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.502150+0000 mon.c (mon.2) 1161 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:32.758 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:32 smithi203 bash[23654]: audit 2024-03-11T21:39:32.502454+0000 mon.a (mon.0) 3334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:32.773 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:32.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.902 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:33.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: cluster 2024-03-11T21:39:32.479026+0000 mon.a (mon.0) 3331 : cluster 0 osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.499925+0000 mon.c (mon.2) 1159 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.500310+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.501358+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.501659+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.502150+0000 mon.c (mon.2) 1161 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:33.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[21512]: audit 2024-03-11T21:39:32.502454+0000 mon.a (mon.0) 3334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: cluster 2024-03-11T21:39:31.543478+0000 mgr.y (mgr.14550) 647 : cluster 0 pgmap v1057: 206 pgs: 206 active+clean; 582 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.472731+0000 mon.a (mon.0) 3330 : audit 1 from='client.? 172.21.15.143:0/631890419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi143-71773-104"}]': finished 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: cluster 2024-03-11T21:39:32.479026+0000 mon.a (mon.0) 3331 : cluster 0 osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.499925+0000 mon.c (mon.2) 1159 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.500310+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.501358+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.501659+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.502150+0000 mon.c (mon.2) 1161 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:33.018 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:32 smithi143 bash[29284]: audit 2024-03-11T21:39:32.502454+0000 mon.a (mon.0) 3334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:39:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:33.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:33.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:33.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:33.285 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:33.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:33.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:33.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:33.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:33.677 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:33.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.805 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:33.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:33.938 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:34.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:34.070 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:34.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:34.201 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:34.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:34.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:34.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:34.466 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:34.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:34.597 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:34.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:34.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: audit 2024-03-11T21:39:33.485778+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:34.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: cluster 2024-03-11T21:39:33.489980+0000 mon.a (mon.0) 3336 : cluster 0 osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:39:34.710 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: audit 2024-03-11T21:39:33.498160+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: audit 2024-03-11T21:39:33.498885+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: cluster 2024-03-11T21:39:33.544499+0000 mgr.y (mgr.14550) 648 : cluster 0 pgmap v1060: 206 pgs: 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: cluster 2024-03-11T21:39:34.225158+0000 osd.6 (osd.6) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[21512]: cluster 2024-03-11T21:39:34.225422+0000 osd.6 (osd.6) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: audit 2024-03-11T21:39:33.485778+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: cluster 2024-03-11T21:39:33.489980+0000 mon.a (mon.0) 3336 : cluster 0 osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: audit 2024-03-11T21:39:33.498160+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: audit 2024-03-11T21:39:33.498885+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: cluster 2024-03-11T21:39:33.544499+0000 mgr.y (mgr.14550) 648 : cluster 0 pgmap v1060: 206 pgs: 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: cluster 2024-03-11T21:39:34.225158+0000 osd.6 (osd.6) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:34.711 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:34 smithi143 bash[29284]: cluster 2024-03-11T21:39:34.225422+0000 osd.6 (osd.6) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:34.722 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:34.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: audit 2024-03-11T21:39:33.485778+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi143-71773-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:39:34.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: cluster 2024-03-11T21:39:33.489980+0000 mon.a (mon.0) 3336 : cluster 0 osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:39:34.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: audit 2024-03-11T21:39:33.498160+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: audit 2024-03-11T21:39:33.498885+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:34.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: cluster 2024-03-11T21:39:33.544499+0000 mgr.y (mgr.14550) 648 : cluster 0 pgmap v1060: 206 pgs: 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:39:34.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: cluster 2024-03-11T21:39:34.225158+0000 osd.6 (osd.6) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:34.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:34 smithi203 bash[23654]: cluster 2024-03-11T21:39:34.225422+0000 osd.6 (osd.6) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:34.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:34.848 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:34.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:34.970 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:35.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.101 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:35.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:35.234 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:35.314 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:35.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.362 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:35.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:35.495 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:35.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.610 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:35 smithi203 bash[23654]: cluster 2024-03-11T21:39:34.945822+0000 mon.a (mon.0) 3338 : cluster 0 osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:39:35.619 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:35 smithi143 bash[21512]: cluster 2024-03-11T21:39:34.945822+0000 mon.a (mon.0) 3338 : cluster 0 osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:39:35.619 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:35 smithi143 bash[29284]: cluster 2024-03-11T21:39:34.945822+0000 mon.a (mon.0) 3338 : cluster 0 osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:39:35.625 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:35.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:39:35.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:35.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:35.885 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:35.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:36.010 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:36.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:36.136 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:36.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:36.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:36.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:36.396 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:36.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:36.536 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:36.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:36.666 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:36.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:36.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:36 smithi203 bash[23654]: audit 2024-03-11T21:39:35.510122+0000 mon.a (mon.0) 3339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:36.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:36 smithi203 bash[23654]: cluster 2024-03-11T21:39:35.518419+0000 mon.a (mon.0) 3340 : cluster 0 osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:39:36.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:36 smithi203 bash[23654]: cluster 2024-03-11T21:39:35.545065+0000 mgr.y (mgr.14550) 649 : cluster 0 pgmap v1063: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:36.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:36 smithi203 bash[23654]: cluster 2024-03-11T21:39:36.469287+0000 mon.a (mon.0) 3341 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[21512]: audit 2024-03-11T21:39:35.510122+0000 mon.a (mon.0) 3339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[21512]: cluster 2024-03-11T21:39:35.518419+0000 mon.a (mon.0) 3340 : cluster 0 osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[21512]: cluster 2024-03-11T21:39:35.545065+0000 mgr.y (mgr.14550) 649 : cluster 0 pgmap v1063: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[21512]: cluster 2024-03-11T21:39:36.469287+0000 mon.a (mon.0) 3341 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[29284]: audit 2024-03-11T21:39:35.510122+0000 mon.a (mon.0) 3339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi143-71773-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:36.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[29284]: cluster 2024-03-11T21:39:35.518419+0000 mon.a (mon.0) 3340 : cluster 0 osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:39:36.780 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[29284]: cluster 2024-03-11T21:39:35.545065+0000 mgr.y (mgr.14550) 649 : cluster 0 pgmap v1063: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:39:36.780 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:36 smithi143 bash[29284]: cluster 2024-03-11T21:39:36.469287+0000 mon.a (mon.0) 3341 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:36.794 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:36.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:36.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:37.028 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:37.054 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:37.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:37.184 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:37.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:37.315 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:37.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:37.444 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:37.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:37.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:37 smithi203 bash[23654]: cluster 2024-03-11T21:39:36.531821+0000 mon.a (mon.0) 3342 : cluster 0 osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:39:37.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:37 smithi203 bash[23654]: audit 2024-03-11T21:39:36.535993+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:37 smithi203 bash[23654]: audit 2024-03-11T21:39:36.537058+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:37 smithi203 bash[23654]: audit 2024-03-11T21:39:37.029051+0000 mon.a (mon.0) 3344 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:37.561 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:37 smithi203 bash[23654]: audit 2024-03-11T21:39:37.031213+0000 mon.c (mon.2) 1164 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:37.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:37.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:37.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:37.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:37.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:37.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[21512]: cluster 2024-03-11T21:39:36.531821+0000 mon.a (mon.0) 3342 : cluster 0 osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[21512]: audit 2024-03-11T21:39:36.535993+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[21512]: audit 2024-03-11T21:39:36.537058+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[21512]: audit 2024-03-11T21:39:37.029051+0000 mon.a (mon.0) 3344 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[21512]: audit 2024-03-11T21:39:37.031213+0000 mon.c (mon.2) 1164 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[29284]: cluster 2024-03-11T21:39:36.531821+0000 mon.a (mon.0) 3342 : cluster 0 osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:39:37.940 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[29284]: audit 2024-03-11T21:39:36.535993+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[29284]: audit 2024-03-11T21:39:36.537058+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:37.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[29284]: audit 2024-03-11T21:39:37.029051+0000 mon.a (mon.0) 3344 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:37.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:37 smithi143 bash[29284]: audit 2024-03-11T21:39:37.031213+0000 mon.c (mon.2) 1164 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:37.967 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:38.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:38.095 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:38.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:38.226 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:38.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:38.357 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:38.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:38.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:38.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:38.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:38.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:38.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:38.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:37.527184+0000 mon.a (mon.0) 3345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: cluster 2024-03-11T21:39:37.536160+0000 mon.a (mon.0) 3346 : cluster 0 osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: cluster 2024-03-11T21:39:37.546128+0000 mgr.y (mgr.14550) 650 : cluster 0 pgmap v1066: 246 pgs: 40 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:37.586007+0000 mon.c (mon.2) 1165 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:37.586408+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:38.529033+0000 mon.a (mon.0) 3348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: cluster 2024-03-11T21:39:38.532886+0000 mon.a (mon.0) 3349 : cluster 0 osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:38.535462+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.862 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[21512]: audit 2024-03-11T21:39:38.535862+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:37.527184+0000 mon.a (mon.0) 3345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: cluster 2024-03-11T21:39:37.536160+0000 mon.a (mon.0) 3346 : cluster 0 osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: cluster 2024-03-11T21:39:37.546128+0000 mgr.y (mgr.14550) 650 : cluster 0 pgmap v1066: 246 pgs: 40 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:37.586007+0000 mon.c (mon.2) 1165 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:37.586408+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:38.529033+0000 mon.a (mon.0) 3348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: cluster 2024-03-11T21:39:38.532886+0000 mon.a (mon.0) 3349 : cluster 0 osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:38.535462+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.863 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:38 smithi143 bash[29284]: audit 2024-03-11T21:39:38.535862+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.877 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:37.527184+0000 mon.a (mon.0) 3345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: cluster 2024-03-11T21:39:37.536160+0000 mon.a (mon.0) 3346 : cluster 0 osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: cluster 2024-03-11T21:39:37.546128+0000 mgr.y (mgr.14550) 650 : cluster 0 pgmap v1066: 246 pgs: 40 unknown, 206 active+clean; 582 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:37.586007+0000 mon.c (mon.2) 1165 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:37.586408+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:38.529033+0000 mon.a (mon.0) 3348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: cluster 2024-03-11T21:39:38.532886+0000 mon.a (mon.0) 3349 : cluster 0 osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:38.535462+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[23654]: audit 2024-03-11T21:39:38.535862+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:38.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:38.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:39.007 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:39.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:39.140 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:39.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:39.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:39.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:39.401 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:39.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.23697.0:657 126.8 126:10260cb2:::test_650:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e181)", 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:41.621003+0000", 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 597.91305086600005, 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.083230671000000006, 2024-03-11T21:39:39.524 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.23697", 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1545131348", 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 657 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621003+0000", 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.525 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621003+0000", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621005+0000", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.198e-06 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621015+0000", 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.009e-05 2024-03-11T21:39:39.526 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621035+0000", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9721e-05 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621047+0000", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2734e-05 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621122+0000", 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4729999999999998e-05 2024-03-11T21:39:39.527 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.621283+0000", 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000161213 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.704233+0000", 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.082949985000000004 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.528 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing_reply(126.8 epoch 181/176 rep_tid 1573 lcod 181'21)", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:41.622578+0000", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 597.91147556700002, 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.081664546000000005, 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622578+0000", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.529 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622578+0000", 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622579+0000", 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4799999999999997e-07 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622583+0000", 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4039999999999997e-06 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.530 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622594+0000", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1045e-05 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622601+0000", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.956e-06 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.622669+0000", 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7651000000000007e-05 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:41.704242+0000", 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.081573742000000005 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.29010.0:2 138.0 138:0384ab8e:::oid:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:44.650456+0000", 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 594.88359790699997, 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0031684030000001, 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:39.532 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.29010", 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2930328358", 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650456+0000", 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.533 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650456+0000", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650457+0000", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8080000000000001e-06 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650464+0000", 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6390000000000003e-06 2024-03-11T21:39:39.534 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650473+0000", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1069999999999994e-06 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650484+0000", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0438000000000001e-05 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650678+0000", 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000194253 2024-03-11T21:39:39.535 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.650688+0000", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9650000000000003e-06 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.621534+0000", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.97084614499999999 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.621665+0000", 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000130846 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.536 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.621815+0000", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000150533 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.624484+0000", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0026687590000000001 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.631050+0000", 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0065661019999999999 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.537 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.653514+0000", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.022463642999999998 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.653577+0000", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2825999999999998e-05 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.653624+0000", 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7339000000000002e-05 2024-03-11T21:39:39.538 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.29016.0:2 139.0 139:0384ab8e:::oid:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:29:44.651262+0000", 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 594.88279118000003, 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.01514606, 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.29016", 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1964770894", 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:39:39.539 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651262+0000", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651262+0000", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651263+0000", 2024-03-11T21:39:39.540 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.054e-06 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651269+0000", 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9449999999999998e-06 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651281+0000", 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1722000000000001e-05 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.541 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651289+0000", 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3410000000000003e-06 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651399+0000", 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000109586 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:44.651409+0000", 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0169999999999999e-05 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.542 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.661458+0000", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.010048864 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.661583+0000", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000125294 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.661719+0000", 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013593399999999999 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.543 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.662997+0000", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001277863 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.664428+0000", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0014310289999999999 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.666304+0000", 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001875397 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.544 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.666361+0000", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7119e-05 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:29:45.666408+0000", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7741999999999998e-05 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:972 173.f 173:f6be59e7:test-rados-api-smithi143-71773-14::foo19:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e212)", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:30:13.553810+0000", 2024-03-11T21:39:39.545 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 565.980243771, 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.086694532000000005, 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 972 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553810+0000", 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553810+0000", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553811+0000", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2800000000000005e-07 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553816+0000", 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2390000000000002e-06 2024-03-11T21:39:39.547 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553826+0000", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0321e-05 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553832+0000", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8080000000000001e-06 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.553874+0000", 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2147999999999997e-05 2024-03-11T21:39:39.548 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.554024+0000", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015022200000000001 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.555547+0000", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0015226669999999999 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.555671+0000", 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000123845 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.549 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.556627+0000", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00095585400000000001 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.556687+0000", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.0686000000000001e-05 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.557419+0000", 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00073156000000000002 2024-03-11T21:39:39.550 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.640060+0000", 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.082640823000000002 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:30:13.640504+0000", 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00044443100000000001 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.551 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41201.0:2 222.1e 222:7cdf4dc8:::oid_XattrsRoundTripPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:31:15.268452+0000", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 504.26560142400001, 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.44789408000000003, 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41201", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/4149958807", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268452+0000", 2024-03-11T21:39:39.552 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268452+0000", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268454+0000", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9769999999999999e-06 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268461+0000", 2024-03-11T21:39:39.553 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.81e-06 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268472+0000", 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1474000000000001e-05 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268484+0000", 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1476e-05 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.554 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268540+0000", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6360999999999998e-05 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.268547+0000", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7220000000000002e-06 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.712602+0000", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.44405506500000003 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.555 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.712703+0000", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000101412 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.712846+0000", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014213100000000001 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.716070+0000", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0032242870000000002 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.716215+0000", 2024-03-11T21:39:39.556 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014492800000000001 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.716269+0000", 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4642000000000003e-05 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:31:15.716346+0000", 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6794999999999999e-05 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.557 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41354.0:1 247.6s0 247:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:05.381345+0000", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 454.15270828500002, 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3910665899999999, 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41354", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/2500018052", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.558 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381345+0000", 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381345+0000", 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381347+0000", 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7999999999999999e-06 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.559 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381354+0000", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.306e-06 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381366+0000", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1508000000000001e-05 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.381379+0000", 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3534e-05 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.560 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.382745+0000", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0013656479999999999 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:05.382749+0000", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4680000000000003e-06 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.767788+0000", 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.385038816 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.561 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.768078+0000", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00028991899999999997 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.769188+0000", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0011102009999999999 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.771877+0000", 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0026881470000000001 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.562 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.772368+0000", 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00049098200000000005 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:06.772412+0000", 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4261000000000002e-05 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41435.0:1 258.2s0 258:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-03-11T21:39:39.563 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:32:31.301714+0000", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 428.23233960200002, 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0101941780000001, 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41435", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/126623859", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301714+0000", 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.564 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301714+0000", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301716+0000", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2450000000000001e-06 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301732+0000", 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6192999999999999e-05 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.565 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301746+0000", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3633e-05 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301761+0000", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.464e-05 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301852+0000", 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1216000000000002e-05 2024-03-11T21:39:39.566 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:31.301857+0000", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2240000000000003e-06 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.308254+0000", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.006396632 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.308480+0000", 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00022674899999999999 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.567 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.308960+0000", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00047987000000000003 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.309405+0000", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000444241 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.311828+0000", 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0024230639999999999 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.568 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:32:32.311908+0000", 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.0470999999999996e-05 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.32849.0:5101 278.8 278:10000000:test-rados-api-smithi143-71773-50:::head [pgnls start_epoch 429] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e429)", 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:33:50.446963+0000", 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 349.08709003299998, 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.23984044299999999, 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.569 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.32849", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1735581", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5101 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.446963+0000", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.446963+0000", 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.570 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.446966+0000", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.373e-06 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.446975+0000", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1390000000000001e-06 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.447001+0000", 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6288e-05 2024-03-11T21:39:39.571 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.447014+0000", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2255e-05 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.447105+0000", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1191000000000002e-05 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.447113+0000", 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9049999999999997e-06 2024-03-11T21:39:39.572 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.686396+0000", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.239283041 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.686487+0000", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1265000000000001e-05 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:33:50.686804+0000", 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000316986 2024-03-11T21:39:39.573 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(6.13,QREQUEST,e:472,from:3,reservation_nonce:10)", 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:34:42.539925+0000", 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 296.99412857300001, 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.313547509, 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539925+0000", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539925+0000", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539926+0000", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2309999999999999e-06 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.575 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539932+0000", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2219999999999999e-06 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539955+0000", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3023e-05 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.539965+0000", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6099999999999995e-06 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.540030+0000", 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.5159000000000004e-05 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.540040+0000", 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.4860000000000003e-06 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.853350+0000", 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.31331009500000001 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.577 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.853369+0000", 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8791e-05 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:34:42.853472+0000", 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000103892 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3255 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:38:58.732189+0000", 2024-03-11T21:39:39.578 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 40.801864973999997, 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0019685589999999999, 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3255 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732189+0000", 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.579 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732189+0000", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732192+0000", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.96e-06 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732205+0000", 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3852999999999999e-05 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.580 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732249+0000", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3328000000000003e-05 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.732278+0000", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9535000000000001e-05 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.733571+0000", 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001293177 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.581 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.733814+0000", 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00024213699999999999 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:58.734157+0000", 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000343569 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.582 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "pg_update_log_missing(308.7s2 epoch 653/652 rep_tid 242 entries 653'1 (0'0) error 308:e7e50e30:test-rados-api-smithi143-71773-105::foo:head by client.30666.0:1 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:00.654068+0000", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 38.879985662000003, 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0046723850000000003, 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654068+0000", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654068+0000", 2024-03-11T21:39:39.583 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654069+0000", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.08e-06 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654089+0000", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0366000000000001e-05 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654130+0000", 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0754000000000003e-05 2024-03-11T21:39:39.584 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654172+0000", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2162000000000001e-05 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.654554+0000", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00038218099999999999 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.658740+0000", 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0041858420000000004 2024-03-11T21:39:39.585 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3265 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e651)", 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:03.736329+0000", 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 35.797724183, 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0010248379999999999, 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3265 2024-03-11T21:39:39.586 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736329+0000", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736329+0000", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736334+0000", 2024-03-11T21:39:39.587 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4429999999999996e-06 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736358+0000", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4119000000000001e-05 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736426+0000", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8274000000000004e-05 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736459+0000", 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2882000000000001e-05 2024-03-11T21:39:39.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736750+0000", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000291005 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.736981+0000", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00023115700000000001 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:03.737354+0000", 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000372958 2024-03-11T21:39:39.589 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3285 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:13.744304+0000", 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 25.789749756999999, 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001179572, 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3285 2024-03-11T21:39:39.590 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744304+0000", 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744304+0000", 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.591 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744306+0000", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.165e-06 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744314+0000", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3340000000000002e-06 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744340+0000", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5935000000000002e-05 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744354+0000", 2024-03-11T21:39:39.592 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4188999999999999e-05 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.744915+0000", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00056066999999999998 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.745206+0000", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029137299999999999 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:13.745483+0000", 2024-03-11T21:39:39.593 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027690599999999999 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3326 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e681)", 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:33.760280+0000", 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 5.7737739880000003, 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00092449100000000005, 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:39:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3326 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760280+0000", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760280+0000", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.595 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760283+0000", 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3950000000000001e-06 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760296+0000", 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3236e-05 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760341+0000", 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5024000000000001e-05 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.596 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760357+0000", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6008e-05 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.760916+0000", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00055914599999999995 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.761050+0000", 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013325300000000001 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.597 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:33.761204+0000", 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015442900000000001 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWrite(313.2s1 690/688 ECSubWrite(tid=1550, reqid=client.41921.0:2, at_version=690'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:37.577761+0000", 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 1.956292178, 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0023283230000000002, 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:39:39.598 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577761+0000", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577761+0000", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577762+0000", 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.5300000000000003e-07 2024-03-11T21:39:39.599 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577937+0000", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017483099999999999 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577954+0000", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7455000000000001e-05 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.577971+0000", 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6226e-05 2024-03-11T21:39:39.600 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.579325+0000", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001354318 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.579349+0000", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4391000000000001e-05 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.580018+0000", 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000668628 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.601 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.580090+0000", 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1620999999999997e-05 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWrite(313.2s1 690/688 ECSubWrite(tid=1551, reqid=client.41921.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=690'1))", 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:37.582590+0000", 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 1.951463875, 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00052772099999999998, 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.602 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582590+0000", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582590+0000", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582591+0000", 2024-03-11T21:39:39.603 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.052e-06 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582606+0000", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5392000000000001e-05 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582631+0000", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4499e-05 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582644+0000", 2024-03-11T21:39:39.604 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.34e-05 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582740+0000", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6255999999999995e-05 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.582752+0000", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1603999999999999e-05 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.583050+0000", 2024-03-11T21:39:39.605 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029801399999999998 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:37.583117+0000", 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7503999999999996e-05 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3336 5.3 5:c609908c:::notify.5:head [watch ping cookie 93859898880384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:38.764321+0000", 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 0.76973210400000003, 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001180186, 2024-03-11T21:39:39.606 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3336 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764321+0000", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:39:39.607 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764321+0000", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764325+0000", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7730000000000001e-06 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764340+0000", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5167000000000001e-05 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:39:39.608 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764395+0000", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4747999999999998e-05 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764438+0000", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2660999999999997e-05 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.764802+0000", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00036454899999999998 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.765087+0000", 2024-03-11T21:39:39.609 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00028448699999999999 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:38.765502+0000", 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00041480100000000002 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:39:39.610 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:39:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.611 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:39.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:39.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:39.740 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:39.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[29284]: cluster 2024-03-11T21:39:38.169763+0000 osd.2 (osd.2) 325 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:39.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[29284]: cluster 2024-03-11T21:39:38.170271+0000 osd.2 (osd.2) 326 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:39.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[29284]: audit 2024-03-11T21:39:38.903204+0000 mgr.y (mgr.14550) 651 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:39.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[29284]: audit 2024-03-11T21:39:39.531412+0000 mon.a (mon.0) 3351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:39.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[29284]: cluster 2024-03-11T21:39:39.535332+0000 mon.a (mon.0) 3352 : cluster 0 osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:39:39.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[21512]: cluster 2024-03-11T21:39:38.169763+0000 osd.2 (osd.2) 325 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:39.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[21512]: cluster 2024-03-11T21:39:38.170271+0000 osd.2 (osd.2) 326 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:39.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[21512]: audit 2024-03-11T21:39:38.903204+0000 mgr.y (mgr.14550) 651 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:39.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[21512]: audit 2024-03-11T21:39:39.531412+0000 mon.a (mon.0) 3351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:39.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:39 smithi143 bash[21512]: cluster 2024-03-11T21:39:39.535332+0000 mon.a (mon.0) 3352 : cluster 0 osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:39:39.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:39.854 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:39 smithi203 bash[23654]: cluster 2024-03-11T21:39:38.169763+0000 osd.2 (osd.2) 325 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:39.854 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:39 smithi203 bash[23654]: cluster 2024-03-11T21:39:38.170271+0000 osd.2 (osd.2) 326 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:39.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:39 smithi203 bash[23654]: audit 2024-03-11T21:39:38.903204+0000 mgr.y (mgr.14550) 651 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:39.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:39 smithi203 bash[23654]: audit 2024-03-11T21:39:39.531412+0000 mon.a (mon.0) 3351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:39.855 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:39 smithi203 bash[23654]: cluster 2024-03-11T21:39:39.535332+0000 mon.a (mon.0) 3352 : cluster 0 osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:39:39.871 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:39.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:40.021 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:40.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.150 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:40.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:40.280 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:40.383 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.410 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:40.516 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:40.538 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:40.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:40.667 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[29284]: cluster 2024-03-11T21:39:39.546917+0000 mgr.y (mgr.14550) 652 : cluster 0 pgmap v1069: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[29284]: audit 2024-03-11T21:39:39.594682+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[29284]: audit 2024-03-11T21:39:39.595011+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[21512]: cluster 2024-03-11T21:39:39.546917+0000 mgr.y (mgr.14550) 652 : cluster 0 pgmap v1069: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[21512]: audit 2024-03-11T21:39:39.594682+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:40.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:40 smithi143 bash[21512]: audit 2024-03-11T21:39:39.595011+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:40.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:40.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:40.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:41.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:40 smithi203 bash[23654]: cluster 2024-03-11T21:39:39.546917+0000 mgr.y (mgr.14550) 652 : cluster 0 pgmap v1069: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:41.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:40 smithi203 bash[23654]: audit 2024-03-11T21:39:39.594682+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:41.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:40 smithi203 bash[23654]: audit 2024-03-11T21:39:39.595011+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:41.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:41.056 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:41.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:41.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:41.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:41.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.446 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:41.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:41.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:41.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: audit 2024-03-11T21:39:40.575682+0000 mon.a (mon.0) 3354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: cluster 2024-03-11T21:39:40.581119+0000 mon.a (mon.0) 3355 : cluster 0 osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: audit 2024-03-11T21:39:40.582235+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: audit 2024-03-11T21:39:40.583425+0000 mon.a (mon.0) 3356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: cluster 2024-03-11T21:39:41.471596+0000 mon.a (mon.0) 3357 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: audit 2024-03-11T21:39:41.474915+0000 mon.a (mon.0) 3358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:41.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:41 smithi203 bash[23654]: cluster 2024-03-11T21:39:41.482437+0000 mon.a (mon.0) 3359 : cluster 0 osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:39:41.833 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:41.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: audit 2024-03-11T21:39:40.575682+0000 mon.a (mon.0) 3354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:41.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: cluster 2024-03-11T21:39:40.581119+0000 mon.a (mon.0) 3355 : cluster 0 osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:39:41.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: audit 2024-03-11T21:39:40.582235+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: audit 2024-03-11T21:39:40.583425+0000 mon.a (mon.0) 3356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: cluster 2024-03-11T21:39:41.471596+0000 mon.a (mon.0) 3357 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: audit 2024-03-11T21:39:41.474915+0000 mon.a (mon.0) 3358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[21512]: cluster 2024-03-11T21:39:41.482437+0000 mon.a (mon.0) 3359 : cluster 0 osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: audit 2024-03-11T21:39:40.575682+0000 mon.a (mon.0) 3354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: cluster 2024-03-11T21:39:40.581119+0000 mon.a (mon.0) 3355 : cluster 0 osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: audit 2024-03-11T21:39:40.582235+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: audit 2024-03-11T21:39:40.583425+0000 mon.a (mon.0) 3356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]: dispatch 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: cluster 2024-03-11T21:39:41.471596+0000 mon.a (mon.0) 3357 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: audit 2024-03-11T21:39:41.474915+0000 mon.a (mon.0) 3358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-112"}]': finished 2024-03-11T21:39:41.864 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:41 smithi143 bash[29284]: cluster 2024-03-11T21:39:41.482437+0000 mon.a (mon.0) 3359 : cluster 0 osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:39:41.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:41.963 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:42.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.093 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:42.197 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:42.218 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:42.224 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:42.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:42.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:42.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:42.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:39:42.488 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:42.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.615 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:42.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:42.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:42.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:42.871 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- 172.21.15.143:0/2988323324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87201071c0 msgr2=0x7f872010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- 172.21.15.143:0/2988323324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87201071c0 0x7f872010ed40 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f871800b0d0 tx=0x7f871800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- 172.21.15.143:0/2988323324 shutdown_connections 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- 172.21.15.143:0/2988323324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8720069d70 0x7f872006a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- 172.21.15.143:0/2988323324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87201071c0 0x7f872010ed40 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- 172.21.15.143:0/2988323324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872006a720 0x7f8720106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- 172.21.15.143:0/2988323324 >> 172.21.15.143:0/2988323324 conn(0x7f8720076d80 msgr2=0x7f8720077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- 172.21.15.143:0/2988323324 shutdown_connections 2024-03-11T21:39:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- 172.21.15.143:0/2988323324 wait complete. 2024-03-11T21:39:42.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 librados: starting msgr at 2024-03-11T21:39:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 librados: starting objecter 2024-03-11T21:39:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 Processor -- start 2024-03-11T21:39:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- start start 2024-03-11T21:39:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 librados: setting wanted keys 2024-03-11T21:39:42.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 librados: calling monclient init 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720069d70 0x7f872010da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 0x7f872010df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 0x7f87201b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8720111680 con 0x7f87201071c0 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8720111500 con 0x7f8720069d70 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.972+0000 7f87284e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8720111800 con 0x7f872006a720 2024-03-11T21:39:42.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 0x7f872010df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8726a81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 0x7f87201b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8726a81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 0x7f87201b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:53650/0 (socket says 172.21.15.143:53650) 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8726a81700 1 -- 172.21.15.143:0/1642213849 learned_addr learned my addr 172.21.15.143:0/1642213849 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8726280700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720069d70 0x7f872010da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 -- 172.21.15.143:0/1642213849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720069d70 msgr2=0x7f872010da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720069d70 0x7f872010da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 -- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 msgr2=0x7f87201b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 0x7f87201b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 -- 172.21.15.143:0/1642213849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87201b2db0 con 0x7f872006a720 2024-03-11T21:39:42.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f8725a7f700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 0x7f872010df50 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f87201032c0 tx=0x7f871000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87177fe700 10 client.?.objecter ms_handle_connect 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87177fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871000e880 con 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f871000f450 con 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87201b2f80 con 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710018660 con 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87284e4700 10 client.41927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87201b3430 con 0x7f872006a720 2024-03-11T21:39:42.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87284e4700 1 librados: init done 2024-03-11T21:39:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.976+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8720103570 con 0x7f872006a720 2024-03-11T21:39:42.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f871000f950 con 0x7f872006a720 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 0x7f870c071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(695..695 src has 87..695) v4 ==== 9768+0+0 (secure 0 0 0) 0x7f871008e780 con 0x7f872006a720 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 10 client.41927.objecter ms_dispatch 0x7f8720001550 osd_map(695..695 src has 87..695) v4 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 3 client.41927.objecter handle_osd_map got epochs [695,695] > 0 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 3 client.41927.objecter handle_osd_map decoding full epoch 695 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f8726280700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 0x7f870c071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:42.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 20 client.41927.objecter dump_active .. 0 homeless 2024-03-11T21:39:42.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f8726280700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 0x7f870c071d30 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f871c00ac50 tx=0x7f871c00a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:42.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.980+0000 7f87177fe700 10 client.41927.objecter ms_handle_connect 0x7f870c06f870 2024-03-11T21:39:42.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:42.984+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8710014030 con 0x7f872006a720 2024-03-11T21:39:42.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:42.991 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:43.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:43.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:42 smithi143 bash[21512]: cluster 2024-03-11T21:39:41.547762+0000 mgr.y (mgr.14550) 653 : cluster 0 pgmap v1072: 246 pgs: 246 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:39:43.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:42 smithi143 bash[21512]: cluster 2024-03-11T21:39:42.488450+0000 mon.a (mon.0) 3360 : cluster 0 osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:39:43.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:42 smithi143 bash[29284]: cluster 2024-03-11T21:39:41.547762+0000 mgr.y (mgr.14550) 653 : cluster 0 pgmap v1072: 246 pgs: 246 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:39:43.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:42 smithi143 bash[29284]: cluster 2024-03-11T21:39:42.488450+0000 mon.a (mon.0) 3360 : cluster 0 osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:39:43.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:43.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.216+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f872010ef40 con 0x7f872006a720 2024-03-11T21:39:43.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.242 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:42 smithi203 bash[23654]: cluster 2024-03-11T21:39:41.547762+0000 mgr.y (mgr.14550) 653 : cluster 0 pgmap v1072: 246 pgs: 246 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:39:43.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:42 smithi203 bash[23654]: cluster 2024-03-11T21:39:42.488450+0000 mon.a (mon.0) 3360 : cluster 0 osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:39:43.324 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:43.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:43.364 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:43.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:43.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.496+0000 7f87177fe700 1 -- 172.21.15.143:0/1642213849 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v696) v1 ==== 113+0+0 (secure 0 0 0) 0x7f871005b440 con 0x7f872006a720 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 10 librados: watch_flush enter 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 10 librados: watch_flush exit 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 20 client.41927.objecter shutdown clearing up homeless session... 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 10 client.41927.objecter successfully canceled tick 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 msgr2=0x7f870c071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:43.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 0x7f870c071d30 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f871c00ac50 tx=0x7f871c00a380 comp rx=0 tx=0).stop 2024-03-11T21:39:43.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 msgr2=0x7f872010df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:43.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.500+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 0x7f872010df50 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f87201032c0 tx=0x7f871000cc20 comp rx=0 tx=0).stop 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 shutdown_connections 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f870c06f870 0x7f870c071d30 unknown :-1 s=CLOSED pgs=3431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87201071c0 0x7f87201b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720069d70 0x7f872010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 --2- 172.21.15.143:0/1642213849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f872006a720 0x7f872010df50 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 >> 172.21.15.143:0/1642213849 conn(0x7f8720076d80 msgr2=0x7f87200febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:43.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 shutdown_connections 2024-03-11T21:39:43.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.508+0000 7f87284e4700 1 -- 172.21.15.143:0/1642213849 wait complete. 2024-03-11T21:39:43.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:43.512+0000 7f87284e4700 1 librados: shutdown 2024-03-11T21:39:43.495 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:39:43.504 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:43.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:43.636 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:43.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:43.766 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:43.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:44.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: audit 2024-03-11T21:39:43.221997+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.143:0/1642213849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: audit 2024-03-11T21:39:43.222530+0000 mon.a (mon.0) 3361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: audit 2024-03-11T21:39:43.487395+0000 mon.a (mon.0) 3362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:39:44.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: cluster 2024-03-11T21:39:43.495135+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:39:44.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: audit 2024-03-11T21:39:43.507132+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:43 smithi203 bash[23654]: audit 2024-03-11T21:39:43.507887+0000 mon.a (mon.0) 3364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:44.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:44.141 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: audit 2024-03-11T21:39:43.221997+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.143:0/1642213849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.141 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: audit 2024-03-11T21:39:43.222530+0000 mon.a (mon.0) 3361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: audit 2024-03-11T21:39:43.487395+0000 mon.a (mon.0) 3362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: cluster 2024-03-11T21:39:43.495135+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: audit 2024-03-11T21:39:43.507132+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[29284]: audit 2024-03-11T21:39:43.507887+0000 mon.a (mon.0) 3364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:43 smithi143 bash[21512]: audit 2024-03-11T21:39:43.221997+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.143:0/1642213849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:43 smithi143 bash[21512]: audit 2024-03-11T21:39:43.222530+0000 mon.a (mon.0) 3361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:43 smithi143 bash[21512]: audit 2024-03-11T21:39:43.487395+0000 mon.a (mon.0) 3362 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:43 smithi143 bash[21512]: cluster 2024-03-11T21:39:43.495135+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:43 smithi143 bash[21512]: audit 2024-03-11T21:39:43.507132+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.142 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:44 smithi143 bash[21512]: audit 2024-03-11T21:39:43.507887+0000 mon.a (mon.0) 3364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:44.156 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:44.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:44.288 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:44.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.418 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:44.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:44.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:44.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.678 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:44.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.805 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:44.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:44.935 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:45.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:45.067 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:45.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:45.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:44 smithi203 bash[23654]: cluster 2024-03-11T21:39:43.548535+0000 mgr.y (mgr.14550) 654 : cluster 0 pgmap v1075: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:39:45.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:44 smithi203 bash[23654]: audit 2024-03-11T21:39:44.495821+0000 mon.a (mon.0) 3365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:45.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:44 smithi203 bash[23654]: cluster 2024-03-11T21:39:44.502674+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:39:45.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:44 smithi203 bash[23654]: audit 2024-03-11T21:39:44.534088+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:44 smithi203 bash[23654]: audit 2024-03-11T21:39:44.534632+0000 mon.a (mon.0) 3367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:45.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:45.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:45.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: cluster 2024-03-11T21:39:43.548535+0000 mgr.y (mgr.14550) 654 : cluster 0 pgmap v1075: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:44.495821+0000 mon.a (mon.0) 3365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: cluster 2024-03-11T21:39:44.502674+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:44.534088+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:44.534632+0000 mon.a (mon.0) 3367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.441 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: cluster 2024-03-11T21:39:43.548535+0000 mgr.y (mgr.14550) 654 : cluster 0 pgmap v1075: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:39:45.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:44.495821+0000 mon.a (mon.0) 3365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:45.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: cluster 2024-03-11T21:39:44.502674+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:39:45.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:44.534088+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:44.534632+0000 mon.a (mon.0) 3367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:45.472 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:45.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:45.605 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:45.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:45.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:45.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:45.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:45.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:46.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:46.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: cluster 2024-03-11T21:39:45.177341+0000 osd.7 (osd.7) 366 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: cluster 2024-03-11T21:39:45.177825+0000 osd.7 (osd.7) 367 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:45.498856+0000 mon.a (mon.0) 3368 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: cluster 2024-03-11T21:39:45.507466+0000 mon.a (mon.0) 3369 : cluster 0 osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:45.508710+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[29284]: audit 2024-03-11T21:39:45.510993+0000 mon.a (mon.0) 3370 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: cluster 2024-03-11T21:39:45.177341+0000 osd.7 (osd.7) 366 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: cluster 2024-03-11T21:39:45.177825+0000 osd.7 (osd.7) 367 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:45.498856+0000 mon.a (mon.0) 3368 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: cluster 2024-03-11T21:39:45.507466+0000 mon.a (mon.0) 3369 : cluster 0 osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:45.508710+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:45 smithi143 bash[21512]: audit 2024-03-11T21:39:45.510993+0000 mon.a (mon.0) 3370 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:46.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:46.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:46.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: cluster 2024-03-11T21:39:45.177341+0000 osd.7 (osd.7) 366 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:46.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: cluster 2024-03-11T21:39:45.177825+0000 osd.7 (osd.7) 367 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:46.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: audit 2024-03-11T21:39:45.498856+0000 mon.a (mon.0) 3368 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:46.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: cluster 2024-03-11T21:39:45.507466+0000 mon.a (mon.0) 3369 : cluster 0 osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:39:46.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: audit 2024-03-11T21:39:45.508710+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:45 smithi203 bash[23654]: audit 2024-03-11T21:39:45.510993+0000 mon.a (mon.0) 3370 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:46.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:39:46.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:46.396 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:46.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- 172.21.15.143:0/119973596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4100cf0 msgr2=0x7fd2e410ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:46.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- 172.21.15.143:0/119973596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4100cf0 0x7fd2e410ecc0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009fc0 tx=0x7fd2dc00b330 comp rx=0 tx=0).stop 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- 172.21.15.143:0/119973596 shutdown_connections 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- 172.21.15.143:0/119973596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4100cf0 0x7fd2e410ecc0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- 172.21.15.143:0/119973596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4106e00 0x7fd2e41071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- 172.21.15.143:0/119973596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100330 0x7fd2e41007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- 172.21.15.143:0/119973596 >> 172.21.15.143:0/119973596 conn(0x7fd2e40772e0 msgr2=0x7fd2e40fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- 172.21.15.143:0/119973596 shutdown_connections 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- 172.21.15.143:0/119973596 wait complete. 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 librados: starting msgr at 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 librados: starting objecter 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 Processor -- start 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- start start 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 librados: setting wanted keys 2024-03-11T21:39:46.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 librados: calling monclient init 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4100330 0x7fd2e419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 0x7fd2e4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e41116e0 con 0x7fd2e4100cf0 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e4111560 con 0x7fd2e4100330 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e975c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e4111860 con 0x7fd2e4106e00 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e27fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e27fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45544/0 (socket says 172.21.15.143:45544) 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e27fc700 1 -- 172.21.15.143:0/272397174 learned_addr learned my addr 172.21.15.143:0/272397174 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.452+0000 7fd2e37fe700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 0x7fd2e4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e2ffd700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4100330 0x7fd2e419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 -- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 msgr2=0x7fd2e4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 0x7fd2e4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:46.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 -- 172.21.15.143:0/272397174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4100330 msgr2=0x7fd2e419d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4100330 0x7fd2e419d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 -- 172.21.15.143:0/272397174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e4198070 con 0x7fd2e4100cf0 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e37fe700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 0x7fd2e4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e27fc700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400ec10 tx=0x7fd2d400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 10 client.?.objecter ms_handle_connect 0x7fd2e4100cf0 2024-03-11T21:39:46.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d400cd40 con 0x7fd2e4100cf0 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd2d40044a0 con 0x7fd2e4100cf0 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4010430 con 0x7fd2e4100cf0 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e4198360 con 0x7fd2e4100cf0 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e975c700 10 client.30729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:46.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e41a4620 con 0x7fd2e4100cf0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e975c700 1 librados: init done 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e404faa0 con 0x7fd2e4100cf0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd2d4004000 con 0x7fd2e4100cf0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 0x7fd2d0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(698..698 src has 87..698) v4 ==== 10149+0+0 (secure 0 0 0) 0x7fd2d408f1f0 con 0x7fd2e4100cf0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 10 client.30729.objecter ms_dispatch 0x7fd2e4001550 osd_map(698..698 src has 87..698) v4 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 3 client.30729.objecter handle_osd_map got epochs [698,698] > 0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 3 client.30729.objecter handle_osd_map decoding full epoch 698 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2e2ffd700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 0x7fd2d0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:46.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.456+0000 7fd2dbfff700 20 client.30729.objecter dump_active .. 0 homeless 2024-03-11T21:39:46.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.460+0000 7fd2dbfff700 10 client.30729.objecter ms_handle_connect 0x7fd2d006f820 2024-03-11T21:39:46.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.460+0000 7fd2e2ffd700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 0x7fd2d0071ce0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005ea0 tx=0x7fd2cc005e30 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:46.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.464+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd2d401f340 con 0x7fd2e4100cf0 2024-03-11T21:39:46.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.524 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:46.627 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.653 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:46.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:46.708+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fd2e4198e40 con 0x7fd2e4100cf0 2024-03-11T21:39:46.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:46.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:46.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.913 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:47.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:47.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:47.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: cluster 2024-03-11T21:39:45.549347+0000 mgr.y (mgr.14550) 655 : cluster 0 pgmap v1078: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: cluster 2024-03-11T21:39:46.474056+0000 mon.a (mon.0) 3371 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: audit 2024-03-11T21:39:46.529571+0000 mon.a (mon.0) 3372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: cluster 2024-03-11T21:39:46.535141+0000 mon.a (mon.0) 3373 : cluster 0 osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: audit 2024-03-11T21:39:46.548150+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: audit 2024-03-11T21:39:46.548745+0000 mon.a (mon.0) 3374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[29284]: audit 2024-03-11T21:39:46.712762+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: cluster 2024-03-11T21:39:45.549347+0000 mgr.y (mgr.14550) 655 : cluster 0 pgmap v1078: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:39:47.157 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: cluster 2024-03-11T21:39:46.474056+0000 mon.a (mon.0) 3371 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:47.158 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: audit 2024-03-11T21:39:46.529571+0000 mon.a (mon.0) 3372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:47.158 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: cluster 2024-03-11T21:39:46.535141+0000 mon.a (mon.0) 3373 : cluster 0 osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:39:47.158 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: audit 2024-03-11T21:39:46.548150+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.158 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: audit 2024-03-11T21:39:46.548745+0000 mon.a (mon.0) 3374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.158 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:46 smithi143 bash[21512]: audit 2024-03-11T21:39:46.712762+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:47.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:47.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:47.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: cluster 2024-03-11T21:39:45.549347+0000 mgr.y (mgr.14550) 655 : cluster 0 pgmap v1078: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:39:47.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: cluster 2024-03-11T21:39:46.474056+0000 mon.a (mon.0) 3371 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:47.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: audit 2024-03-11T21:39:46.529571+0000 mon.a (mon.0) 3372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:47.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: cluster 2024-03-11T21:39:46.535141+0000 mon.a (mon.0) 3373 : cluster 0 osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:39:47.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: audit 2024-03-11T21:39:46.548150+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: audit 2024-03-11T21:39:46.548745+0000 mon.a (mon.0) 3374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]: dispatch 2024-03-11T21:39:47.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:46 smithi203 bash[23654]: audit 2024-03-11T21:39:46.712762+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:47.302 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:47.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:47.432 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:47.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.528+0000 7fd2dbfff700 1 -- 172.21.15.143:0/272397174 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v700) v1 ==== 91+0+0 (secure 0 0 0) 0x7fd2e4198e40 con 0x7fd2e4100cf0 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 10 librados: watch_flush enter 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 10 librados: watch_flush exit 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 20 client.30729.objecter shutdown clearing up homeless session... 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 10 client.30729.objecter successfully canceled tick 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 msgr2=0x7fd2d0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:47.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 0x7fd2d0071ce0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005ea0 tx=0x7fd2cc005e30 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 msgr2=0x7fd2e419dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400ec10 tx=0x7fd2d400ef20 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 shutdown_connections 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd2d006f820 0x7fd2d0071ce0 unknown :-1 s=CLOSED pgs=3432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e4106e00 0x7fd2e4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2e4100330 0x7fd2e419d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 --2- 172.21.15.143:0/272397174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e4100cf0 0x7fd2e419dc90 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 >> 172.21.15.143:0/272397174 conn(0x7fd2e40772e0 msgr2=0x7fd2e41048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 shutdown_connections 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 -- 172.21.15.143:0/272397174 wait complete. 2024-03-11T21:39:47.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:47.532+0000 7fd2e975c700 1 librados: shutdown 2024-03-11T21:39:47.516 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:39:47.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:47.566 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:47.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:47.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:47.821 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:47.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:48.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:48.065 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: cluster 2024-03-11T21:39:47.530350+0000 mon.a (mon.0) 3376 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: cluster 2024-03-11T21:39:47.530415+0000 mon.a (mon.0) 3377 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: audit 2024-03-11T21:39:47.532765+0000 mon.a (mon.0) 3378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]': finished 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: audit 2024-03-11T21:39:47.532947+0000 mon.a (mon.0) 3379 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: cluster 2024-03-11T21:39:47.542835+0000 mon.a (mon.0) 3380 : cluster 0 osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: audit 2024-03-11T21:39:47.612402+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.066 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:47 smithi203 bash[23654]: audit 2024-03-11T21:39:47.612780+0000 mon.a (mon.0) 3381 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.081 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:48.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: cluster 2024-03-11T21:39:47.530350+0000 mon.a (mon.0) 3376 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: cluster 2024-03-11T21:39:47.530415+0000 mon.a (mon.0) 3377 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: audit 2024-03-11T21:39:47.532765+0000 mon.a (mon.0) 3378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]': finished 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: audit 2024-03-11T21:39:47.532947+0000 mon.a (mon.0) 3379 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: cluster 2024-03-11T21:39:47.542835+0000 mon.a (mon.0) 3380 : cluster 0 osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: audit 2024-03-11T21:39:47.612402+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[21512]: audit 2024-03-11T21:39:47.612780+0000 mon.a (mon.0) 3381 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: cluster 2024-03-11T21:39:47.530350+0000 mon.a (mon.0) 3376 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: cluster 2024-03-11T21:39:47.530415+0000 mon.a (mon.0) 3377 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:48.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: audit 2024-03-11T21:39:47.532765+0000 mon.a (mon.0) 3378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-114", "mode": "writeback"}]': finished 2024-03-11T21:39:48.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: audit 2024-03-11T21:39:47.532947+0000 mon.a (mon.0) 3379 : audit 1 from='client.? 172.21.15.143:0/272397174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:48.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: cluster 2024-03-11T21:39:47.542835+0000 mon.a (mon.0) 3380 : cluster 0 osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:39:48.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: audit 2024-03-11T21:39:47.612402+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:47 smithi143 bash[29284]: audit 2024-03-11T21:39:47.612780+0000 mon.a (mon.0) 3381 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:39:48.211 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:48.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.346 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:48.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:48.476 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:48.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:48.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:48.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:48.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:48.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.862 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:48.965 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:48.993 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:49.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: cluster 2024-03-11T21:39:47.549993+0000 mgr.y (mgr.14550) 656 : cluster 0 pgmap v1081: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: cluster 2024-03-11T21:39:48.013159+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: cluster 2024-03-11T21:39:48.014736+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: audit 2024-03-11T21:39:48.534963+0000 mon.a (mon.0) 3382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: cluster 2024-03-11T21:39:48.541337+0000 mon.a (mon.0) 3383 : cluster 0 osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: audit 2024-03-11T21:39:48.548775+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[29284]: audit 2024-03-11T21:39:48.549253+0000 mon.a (mon.0) 3384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: cluster 2024-03-11T21:39:47.549993+0000 mgr.y (mgr.14550) 656 : cluster 0 pgmap v1081: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:49.108 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: cluster 2024-03-11T21:39:48.013159+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-03-11T21:39:49.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: cluster 2024-03-11T21:39:48.014736+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-03-11T21:39:49.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: audit 2024-03-11T21:39:48.534963+0000 mon.a (mon.0) 3382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:49.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: cluster 2024-03-11T21:39:48.541337+0000 mon.a (mon.0) 3383 : cluster 0 osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:39:49.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: audit 2024-03-11T21:39:48.548775+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:48 smithi143 bash[21512]: audit 2024-03-11T21:39:48.549253+0000 mon.a (mon.0) 3384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.123 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:49.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:49.253 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:49.276 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:49.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: cluster 2024-03-11T21:39:47.549993+0000 mgr.y (mgr.14550) 656 : cluster 0 pgmap v1081: 246 pgs: 3 creating+peering, 25 unknown, 218 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: cluster 2024-03-11T21:39:48.013159+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: cluster 2024-03-11T21:39:48.014736+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: audit 2024-03-11T21:39:48.534963+0000 mon.a (mon.0) 3382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: cluster 2024-03-11T21:39:48.541337+0000 mon.a (mon.0) 3383 : cluster 0 osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: audit 2024-03-11T21:39:48.548775+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:48 smithi203 bash[23654]: audit 2024-03-11T21:39:48.549253+0000 mon.a (mon.0) 3384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]: dispatch 2024-03-11T21:39:49.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:49.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:49.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:49.512 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:49.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:49.641 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:49.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.880+0000 7ff0ebca1700 1 -- 172.21.15.143:0/3936665587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4101ec0 msgr2=0x7ff0e4102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/3936665587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4101ec0 0x7ff0e4102340 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009990 tx=0x7ff0d8009ca0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/3936665587 shutdown_connections 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/3936665587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e4102880 0x7ff0e410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/3936665587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4107f30 0x7ff0e4108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/3936665587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4101ec0 0x7ff0e4102340 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/3936665587 >> 172.21.15.143:0/3936665587 conn(0x7ff0e4078eb0 msgr2=0x7ff0e40792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/3936665587 shutdown_connections 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/3936665587 wait complete. 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 librados: starting msgr at 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 librados: starting objecter 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 Processor -- start 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- start start 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 librados: setting wanted keys 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 librados: calling monclient init 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e4102880 0x7ff0e419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4107f30 0x7ff0e4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e41116f0 con 0x7ff0e4107f30 2024-03-11T21:39:49.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4111570 con 0x7ff0e4102880 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4111870 con 0x7ff0e4101ec0 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ea23e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4107f30 0x7ff0e4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:35762/0 (socket says 172.21.15.143:35762) 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 -- 172.21.15.143:0/321155009 learned_addr learned my addr 172.21.15.143:0/321155009 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e4102880 0x7ff0e419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e4102880 msgr2=0x7ff0e419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e4102880 0x7ff0e419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4107f30 msgr2=0x7ff0e4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4107f30 0x7ff0e4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e41981c0 con 0x7ff0e4101ec0 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e4107f30 0x7ff0e4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00e9a0 tx=0x7ff0dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 10 client.?.objecter ms_handle_connect 0x7ff0e4101ec0 2024-03-11T21:39:49.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc00cbd0 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff0dc00cd70 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e41984b0 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc010e60 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 10 client.41939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e41a4770 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 librados: init done 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff0e404faa0 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.884+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff0dc010430 con 0x7ff0e4101ec0 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 0x7ff0cc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(702..702 src has 87..702) v4 ==== 10141+0+0 (secure 0 0 0) 0x7ff0dc08ef40 con 0x7ff0e4101ec0 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 osd_map(702..702 src has 87..702) v4 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 3 client.41939.objecter handle_osd_map got epochs [702,702] > 0 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 3 client.41939.objecter handle_osd_map decoding full epoch 702 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 0x7ff0cc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 20 client.41939.objecter dump_active .. 0 homeless 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v702) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff0dc091500 con 0x7ff0e4101ec0 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 1 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0e4109680 0x7ff0e402c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 1 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 0 1 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 4 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 10 client.41939.objecter _send_command 1 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e410a930 con 0x7ff0e4109680 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 0x7ff0cc071ce0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8006010 tx=0x7ff0d801a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0cc06f820 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0e4109680 0x7ff0e402c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e4109680 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0e4109680 0x7ff0e402c6b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e410a930 con 0x7ff0e4109680 2024-03-11T21:39:49.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:39:49.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 1 = system:0 2024-03-11T21:39:49.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 4 2024-03-11T21:39:49.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.888+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 0 1 2024-03-11T21:39:49.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 2 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 2 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e4109510 osd=0 3 2024-03-11T21:39:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 0 2 2024-03-11T21:39:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 4 2024-03-11T21:39:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 10 client.41939.objecter _send_command 2 2024-03-11T21:39:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.904+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e410aba0 con 0x7ff0e4109680 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e410aba0 con 0x7ff0e4109680 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 4 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 0 2 2024-03-11T21:39:49.891 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:39:49.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 3 2024-03-11T21:39:49.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0e4062f00 0x7ff0e41aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 1 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 4 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 10 client.41939.objecter _send_command 3 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e410a260 con 0x7ff0e4062f00 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0e4062f00 0x7ff0e41aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e4062f00 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0e4062f00 0x7ff0e41aae80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0dc05b990 con 0x7ff0e4062f00 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:39:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 3 = system:0 2024-03-11T21:39:49.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 4 2024-03-11T21:39:49.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.908+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 1 3 2024-03-11T21:39:49.901 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 4 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 4 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e410a0f0 osd=1 3 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 1 4 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 4 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 10 client.41939.objecter _send_command 4 2024-03-11T21:39:49.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e4076020 con 0x7ff0e4062f00 2024-03-11T21:39:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e4076020 con 0x7ff0e4062f00 2024-03-11T21:39:49.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 4 2024-03-11T21:39:49.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 1 4 2024-03-11T21:39:49.912 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 5 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0e402d8d0 0x7ff0e41ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 5 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 2 5 2024-03-11T21:39:49.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 4 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 10 client.41939.objecter _send_command 5 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e4062830 con 0x7ff0e402d8d0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0e402d8d0 0x7ff0e41ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0e402d8d0 0x7ff0e41ae550 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e402d8d0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0d800bdd0 con 0x7ff0e402d8d0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 5 = system:0 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 4 2024-03-11T21:39:49.914 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.928+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 2 5 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 6 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 6 2024-03-11T21:39:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e40626c0 osd=2 3 2024-03-11T21:39:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 2 6 2024-03-11T21:39:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 4 2024-03-11T21:39:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 10 client.41939.objecter _send_command 6 2024-03-11T21:39:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e402ce40 con 0x7ff0e402d8d0 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e402ce40 con 0x7ff0e402d8d0 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 4 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 2 6 2024-03-11T21:39:49.932 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 7 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0e41ac0a0 0x7ff0e41b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 7 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 3 7 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 4 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 10 client.41939.objecter _send_command 7 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e41b4a80 con 0x7ff0e41ac0a0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0e41ac0a0 0x7ff0e41b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e41ac0a0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.948+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0e41ac0a0 0x7ff0e41b1c20 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.952+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e41b4a80 con 0x7ff0e41ac0a0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.952+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.952+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 7 = system:0 2024-03-11T21:39:49.934 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.952+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 4 2024-03-11T21:39:49.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.952+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 3 7 2024-03-11T21:39:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 8 2024-03-11T21:39:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 8 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e402ce40 osd=3 3 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 3 8 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 4 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 10 client.41939.objecter _send_command 8 2024-03-11T21:39:49.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.968+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e41ab730 con 0x7ff0e41ac0a0 2024-03-11T21:39:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e41ab730 con 0x7ff0e41ac0a0 2024-03-11T21:39:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 4 2024-03-11T21:39:49.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 3 8 2024-03-11T21:39:49.957 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 9 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0e41af770 0x7ff0e41b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 9 2024-03-11T21:39:49.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 4 9 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 4 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 10 client.41939.objecter _send_command 9 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e41ab650 con 0x7ff0e41af770 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0e41af770 0x7ff0e41b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e41af770 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.972+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0e41af770 0x7ff0e41b32e0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.976+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e41ab650 con 0x7ff0e41af770 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.976+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.976+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 9 = system:0 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.976+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 4 2024-03-11T21:39:49.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.976+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 4 9 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 10 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 10 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41ab4e0 osd=4 3 2024-03-11T21:39:49.976 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 4 10 2024-03-11T21:39:49.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 4 2024-03-11T21:39:49.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 10 client.41939.objecter _send_command 10 2024-03-11T21:39:49.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.992+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e41ab650 con 0x7ff0e41af770 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e41ab650 con 0x7ff0e41af770 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 4 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 4 10 2024-03-11T21:39:49.980 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 11 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0e41b3d30 0x7ff0e41b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:49.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 11 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 5 11 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 4 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 10 client.41939.objecter _send_command 11 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e41b47d0 con 0x7ff0e41b3d30 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0e41b3d30 0x7ff0e41b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e41b3d30 2024-03-11T21:39:49.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:49.996+0000 7ff0e923c700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0e41b3d30 0x7ff0e41b4190 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:49.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.000+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e41b47d0 con 0x7ff0e41b3d30 2024-03-11T21:39:49.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.000+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:39:49.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.000+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 11 = system:0 2024-03-11T21:39:49.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.000+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 4 2024-03-11T21:39:49.983 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.000+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 5 11 2024-03-11T21:39:50.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 12 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 12 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41ab650 osd=5 3 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 5 12 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 4 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 10 client.41939.objecter _send_command 12 2024-03-11T21:39:50.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.020+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e41b8ce0 con 0x7ff0e41b3d30 2024-03-11T21:39:50.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.006 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e41b8ce0 con 0x7ff0e41b3d30 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 4 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 5 12 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 13 2024-03-11T21:39:50.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 0x7ff0e41b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 13 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 6 13 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 4 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 10 client.41939.objecter _send_command 13 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e41be7f0 con 0x7ff0e41b9360 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 0x7ff0e41b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e41b9360 2024-03-11T21:39:50.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0ea23e700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 0x7ff0e41b97c0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:50.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e41be7f0 con 0x7ff0e41b9360 2024-03-11T21:39:50.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:39:50.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 13 = system:0 2024-03-11T21:39:50.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 4 2024-03-11T21:39:50.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.024+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 6 13 2024-03-11T21:39:50.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:48.682047+0000 osd.6 (osd.6) 295 : cluster 0 39.8 scrub starts 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:48.684045+0000 osd.6 (osd.6) 296 : cluster 0 39.8 scrub ok 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: audit 2024-03-11T21:39:48.914535+0000 mgr.y (mgr.14550) 657 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.546293+0000 mon.a (mon.0) 3385 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: audit 2024-03-11T21:39:49.548134+0000 mon.a (mon.0) 3386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.557331+0000 mon.a (mon.0) 3387 : cluster 0 osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.735501+0000 osd.4 (osd.4) 363 : cluster 0 5.1 scrub starts 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.737869+0000 osd.4 (osd.4) 364 : cluster 0 5.1 scrub ok 2024-03-11T21:39:50.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:49 smithi203 bash[23654]: audit 2024-03-11T21:39:49.891924+0000 mon.c (mon.2) 1176 : audit 0 from='client.? 172.21.15.143:0/321155009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 14 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 14 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41b8ce0 osd=6 3 2024-03-11T21:39:50.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 6 14 2024-03-11T21:39:50.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 4 2024-03-11T21:39:50.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 10 client.41939.objecter _send_command 14 2024-03-11T21:39:50.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.060+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e41b9d00 con 0x7ff0e41b9360 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0e41b9d00 con 0x7ff0e41b9360 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 4 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 6 14 2024-03-11T21:39:50.047 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 15 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 0x7ff0e41ba830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 15 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 7 15 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 4 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 10 client.41939.objecter _send_command 15 2024-03-11T21:39:50.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e41bafa0 con 0x7ff0e41ba3d0 2024-03-11T21:39:50.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 0x7ff0e41ba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:39:50.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e2ffd700 10 client.41939.objecter ms_handle_connect 0x7ff0e41ba3d0 2024-03-11T21:39:50.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.064+0000 7ff0e9a3d700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 0x7ff0e41ba830 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:39:50.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.068+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0e41bafa0 con 0x7ff0e41ba3d0 2024-03-11T21:39:50.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.068+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:39:50.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.068+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 15 = system:0 2024-03-11T21:39:50.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.068+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 4 2024-03-11T21:39:50.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.068+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 7 15 2024-03-11T21:39:50.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 10 client.41939.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:39:50.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign -1 16 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 20 client.41939.objecter _get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_remove -1 16 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 20 client.41939.objecter get_session s=0x7ff0e41b9d00 osd=7 3 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 15 client.41939.objecter _session_command_op_assign 7 16 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 4 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 10 client.41939.objecter _send_command 16 2024-03-11T21:39:50.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.084+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0e41bafc0 con 0x7ff0e41ba3d0 2024-03-11T21:39:50.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0e2ffd700 1 -- 172.21.15.143:0/321155009 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0dc05f120 con 0x7ff0e41ba3d0 2024-03-11T21:39:50.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0e2ffd700 10 client.41939.objecter ms_dispatch 0x7ff0e4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:39:50.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0e2ffd700 10 client.41939.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:39:50.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0e2ffd700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 4 2024-03-11T21:39:50.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0e2ffd700 15 client.41939.objecter _session_command_op_remove 7 16 2024-03-11T21:39:50.073 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 librados: watch_flush enter 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 librados: watch_flush exit 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.0 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0e4109680 msgr2=0x7ff0e402c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0e4109680 0x7ff0e402c6b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e4109510 osd=0 1 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.1 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0e4062f00 msgr2=0x7ff0e41aae80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0e4062f00 0x7ff0e41aae80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e410a0f0 osd=1 1 2024-03-11T21:39:50.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.2 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0e402d8d0 msgr2=0x7ff0e41ae550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0e402d8d0 0x7ff0e41ae550 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e40626c0 osd=2 1 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.3 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0e41ac0a0 msgr2=0x7ff0e41b1c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0e41ac0a0 0x7ff0e41b1c20 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e402ce40 osd=3 1 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.4 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0e41af770 msgr2=0x7ff0e41b32e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0e41af770 0x7ff0e41b32e0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab4e0 osd=4 1 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.5 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0e41b3d30 msgr2=0x7ff0e41b4190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0e41b3d30 0x7ff0e41b4190 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41ab650 osd=5 1 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.6 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ea23e700 1 -- 172.21.15.143:0/321155009 reap_dead start 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ea23e700 1 -- 172.21.15.143:0/321155009 reap_dead start 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ea23e700 1 -- 172.21.15.143:0/321155009 reap_dead start 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ea23e700 1 -- 172.21.15.143:0/321155009 reap_dead start 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 msgr2=0x7ff0e41b97c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 0x7ff0e41b97c0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b8ce0 osd=6 1 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter close_session for osd.7 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 msgr2=0x7ff0e41ba830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 0x7ff0e41ba830 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter put_session s=0x7ff0e41b9d00 osd=7 1 2024-03-11T21:39:50.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 20 client.41939.objecter shutdown clearing up homeless session... 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 10 client.41939.objecter successfully canceled tick 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 msgr2=0x7ff0cc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 0x7ff0cc071ce0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8006010 tx=0x7ff0d801a040 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 msgr2=0x7ff0e419d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00e9a0 tx=0x7ff0dc00ecb0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 shutdown_connections 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0e41ba3d0 0x7ff0e41ba830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0e41b9360 0x7ff0e41b97c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e4101ec0 0x7ff0e419d8a0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 --2- 172.21.15.143:0/321155009 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0cc06f820 0x7ff0cc071ce0 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 >> 172.21.15.143:0/321155009 conn(0x7ff0e4078eb0 msgr2=0x7ff0e4108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 shutdown_connections 2024-03-11T21:39:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 -- 172.21.15.143:0/321155009 wait complete. 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:39:50.088+0000 7ff0ebca1700 1 librados: shutdown 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:39:50.078 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:39:50.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.145 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:50.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:48.682047+0000 osd.6 (osd.6) 295 : cluster 0 39.8 scrub starts 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:48.684045+0000 osd.6 (osd.6) 296 : cluster 0 39.8 scrub ok 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: audit 2024-03-11T21:39:48.914535+0000 mgr.y (mgr.14550) 657 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.546293+0000 mon.a (mon.0) 3385 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: audit 2024-03-11T21:39:49.548134+0000 mon.a (mon.0) 3386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.557331+0000 mon.a (mon.0) 3387 : cluster 0 osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.735501+0000 osd.4 (osd.4) 363 : cluster 0 5.1 scrub starts 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.737869+0000 osd.4 (osd.4) 364 : cluster 0 5.1 scrub ok 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[21512]: audit 2024-03-11T21:39:49.891924+0000 mon.c (mon.2) 1176 : audit 0 from='client.? 172.21.15.143:0/321155009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:48.682047+0000 osd.6 (osd.6) 295 : cluster 0 39.8 scrub starts 2024-03-11T21:39:50.260 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:48.684045+0000 osd.6 (osd.6) 296 : cluster 0 39.8 scrub ok 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: audit 2024-03-11T21:39:48.914535+0000 mgr.y (mgr.14550) 657 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.546293+0000 mon.a (mon.0) 3385 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: audit 2024-03-11T21:39:49.548134+0000 mon.a (mon.0) 3386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-114"}]': finished 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.557331+0000 mon.a (mon.0) 3387 : cluster 0 osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.735501+0000 osd.4 (osd.4) 363 : cluster 0 5.1 scrub starts 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.737869+0000 osd.4 (osd.4) 364 : cluster 0 5.1 scrub ok 2024-03-11T21:39:50.261 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:49 smithi143 bash[29284]: audit 2024-03-11T21:39:49.891924+0000 mon.c (mon.2) 1176 : audit 0 from='client.? 172.21.15.143:0/321155009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:50.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:50.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:50.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:50.539 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:50.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:50.670 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:50.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:50.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:50.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:50.929 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:51.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:51.061 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:51.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.176 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.546215+0000 osd.5 (osd.5) 393 : cluster 0 6.d scrub starts 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.547602+0000 osd.5 (osd.5) 394 : cluster 0 6.d scrub ok 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[21512]: cluster 2024-03-11T21:39:49.550661+0000 mgr.y (mgr.14550) 658 : cluster 0 pgmap v1084: 246 pgs: 246 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.554164+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.546215+0000 osd.5 (osd.5) 393 : cluster 0 6.d scrub starts 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.547602+0000 osd.5 (osd.5) 394 : cluster 0 6.d scrub ok 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[29284]: cluster 2024-03-11T21:39:49.550661+0000 mgr.y (mgr.14550) 658 : cluster 0 pgmap v1084: 246 pgs: 246 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:39:51.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:50 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.554164+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:39:51.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:51.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:50 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.546215+0000 osd.5 (osd.5) 393 : cluster 0 6.d scrub starts 2024-03-11T21:39:51.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:50 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.547602+0000 osd.5 (osd.5) 394 : cluster 0 6.d scrub ok 2024-03-11T21:39:51.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:50 smithi203 bash[23654]: cluster 2024-03-11T21:39:49.550661+0000 mgr.y (mgr.14550) 658 : cluster 0 pgmap v1084: 246 pgs: 246 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:39:51.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:50 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.554164+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:39:51.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:51.319 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:51.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:51.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:39:51.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.575 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:51.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:51.705 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:51.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:51.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:51.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:51.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:52.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:52.097 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:52.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:52.211 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:39:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.751343+0000 osd.6 (osd.6) 297 : cluster 0 39.19 scrub starts 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.753642+0000 osd.6 (osd.6) 298 : cluster 0 39.19 scrub ok 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.768401+0000 osd.4 (osd.4) 365 : cluster 0 39.15 scrub starts 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.771269+0000 osd.4 (osd.4) 366 : cluster 0 39.15 scrub ok 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.849991+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub starts 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:50.851946+0000 osd.7 (osd.7) 369 : cluster 0 3.11 scrub ok 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:51 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.482473+0000 mon.a (mon.0) 3389 : cluster 0 osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[21512]: audit 2024-03-11T21:39:51.487422+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[21512]: audit 2024-03-11T21:39:51.494632+0000 mon.a (mon.0) 3390 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.751343+0000 osd.6 (osd.6) 297 : cluster 0 39.19 scrub starts 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.753642+0000 osd.6 (osd.6) 298 : cluster 0 39.19 scrub ok 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.768401+0000 osd.4 (osd.4) 365 : cluster 0 39.15 scrub starts 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.771269+0000 osd.4 (osd.4) 366 : cluster 0 39.15 scrub ok 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.849991+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub starts 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:50.851946+0000 osd.7 (osd.7) 369 : cluster 0 3.11 scrub ok 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.482473+0000 mon.a (mon.0) 3389 : cluster 0 osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: audit 2024-03-11T21:39:51.487422+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:52 smithi143 bash[29284]: audit 2024-03-11T21:39:51.494632+0000 mon.a (mon.0) 3390 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.751343+0000 osd.6 (osd.6) 297 : cluster 0 39.19 scrub starts 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.753642+0000 osd.6 (osd.6) 298 : cluster 0 39.19 scrub ok 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.768401+0000 osd.4 (osd.4) 365 : cluster 0 39.15 scrub starts 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.771269+0000 osd.4 (osd.4) 366 : cluster 0 39.15 scrub ok 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.849991+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub starts 2024-03-11T21:39:52.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:50.851946+0000 osd.7 (osd.7) 369 : cluster 0 3.11 scrub ok 2024-03-11T21:39:52.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.482473+0000 mon.a (mon.0) 3389 : cluster 0 osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:39:52.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: audit 2024-03-11T21:39:51.487422+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:51 smithi203 bash[23654]: audit 2024-03-11T21:39:51.494632+0000 mon.a (mon.0) 3390 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:39:52.330 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.357 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:52.494 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:52.599 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:52.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:52.704 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:52.704 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:52.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:52.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:52.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:52.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:52.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.218441+0000 osd.1 (osd.1) 177 : cluster 0 4.2 scrub starts 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.220498+0000 osd.1 (osd.1) 178 : cluster 0 4.2 scrub ok 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.551123+0000 mgr.y (mgr.14550) 659 : cluster 0 pgmap v1087: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.885137+0000 osd.7 (osd.7) 370 : cluster 0 6.5 scrub starts 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:51.886832+0000 osd.7 (osd.7) 371 : cluster 0 6.5 scrub ok 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: audit 2024-03-11T21:39:52.050994+0000 mon.c (mon.2) 1178 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: audit 2024-03-11T21:39:52.481601+0000 mon.a (mon.0) 3391 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:53.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:52.490494+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:39:53.003 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:52.679435+0000 osd.2 (osd.2) 327 : cluster 0 6.14 scrub starts 2024-03-11T21:39:53.003 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:52 smithi203 bash[23654]: cluster 2024-03-11T21:39:52.681208+0000 osd.2 (osd.2) 328 : cluster 0 6.14 scrub ok 2024-03-11T21:39:53.016 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:53.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:53.146 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:53.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:39:53.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.218441+0000 osd.1 (osd.1) 177 : cluster 0 4.2 scrub starts 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.220498+0000 osd.1 (osd.1) 178 : cluster 0 4.2 scrub ok 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.551123+0000 mgr.y (mgr.14550) 659 : cluster 0 pgmap v1087: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.885137+0000 osd.7 (osd.7) 370 : cluster 0 6.5 scrub starts 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:51.886832+0000 osd.7 (osd.7) 371 : cluster 0 6.5 scrub ok 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: audit 2024-03-11T21:39:52.050994+0000 mon.c (mon.2) 1178 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:53.265 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: audit 2024-03-11T21:39:52.481601+0000 mon.a (mon.0) 3391 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:52.490494+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:52.679435+0000 osd.2 (osd.2) 327 : cluster 0 6.14 scrub starts 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[29284]: cluster 2024-03-11T21:39:52.681208+0000 osd.2 (osd.2) 328 : cluster 0 6.14 scrub ok 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.218441+0000 osd.1 (osd.1) 177 : cluster 0 4.2 scrub starts 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.220498+0000 osd.1 (osd.1) 178 : cluster 0 4.2 scrub ok 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.551123+0000 mgr.y (mgr.14550) 659 : cluster 0 pgmap v1087: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.885137+0000 osd.7 (osd.7) 370 : cluster 0 6.5 scrub starts 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:51.886832+0000 osd.7 (osd.7) 371 : cluster 0 6.5 scrub ok 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: audit 2024-03-11T21:39:52.050994+0000 mon.c (mon.2) 1178 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:53.266 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: audit 2024-03-11T21:39:52.481601+0000 mon.a (mon.0) 3391 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:39:53.267 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:52.490494+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:39:53.267 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:52.679435+0000 osd.2 (osd.2) 327 : cluster 0 6.14 scrub starts 2024-03-11T21:39:53.267 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:53 smithi143 bash[21512]: cluster 2024-03-11T21:39:52.681208+0000 osd.2 (osd.2) 328 : cluster 0 6.14 scrub ok 2024-03-11T21:39:53.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:53.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:53.408 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:53.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:53.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:53.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:53.668 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:53.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:53.793 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:53.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:53.925 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:54.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.057 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:54.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[21512]: cluster 2024-03-11T21:39:52.901746+0000 osd.7 (osd.7) 372 : cluster 0 39.1 scrub starts 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[21512]: cluster 2024-03-11T21:39:52.904282+0000 osd.7 (osd.7) 373 : cluster 0 39.1 scrub ok 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.513743+0000 mon.a (mon.0) 3393 : cluster 0 osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[21512]: audit 2024-03-11T21:39:53.557708+0000 mon.c (mon.2) 1179 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[21512]: audit 2024-03-11T21:39:53.559434+0000 mon.a (mon.0) 3394 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[29284]: cluster 2024-03-11T21:39:52.901746+0000 osd.7 (osd.7) 372 : cluster 0 39.1 scrub starts 2024-03-11T21:39:54.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[29284]: cluster 2024-03-11T21:39:52.904282+0000 osd.7 (osd.7) 373 : cluster 0 39.1 scrub ok 2024-03-11T21:39:54.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.513743+0000 mon.a (mon.0) 3393 : cluster 0 osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:39:54.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[29284]: audit 2024-03-11T21:39:53.557708+0000 mon.c (mon.2) 1179 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:54 smithi143 bash[29284]: audit 2024-03-11T21:39:53.559434+0000 mon.a (mon.0) 3394 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:54.276 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:53 smithi203 bash[23654]: cluster 2024-03-11T21:39:52.901746+0000 osd.7 (osd.7) 372 : cluster 0 39.1 scrub starts 2024-03-11T21:39:54.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:53 smithi203 bash[23654]: cluster 2024-03-11T21:39:52.904282+0000 osd.7 (osd.7) 373 : cluster 0 39.1 scrub ok 2024-03-11T21:39:54.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:53 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.513743+0000 mon.a (mon.0) 3393 : cluster 0 osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:39:54.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:54 smithi203 bash[23654]: audit 2024-03-11T21:39:53.557708+0000 mon.c (mon.2) 1179 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:54 smithi203 bash[23654]: audit 2024-03-11T21:39:53.559434+0000 mon.a (mon.0) 3394 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:39:54.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.316 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:54.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:54.443 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:54.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:54.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:54.702 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:54.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:54.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:54.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:54.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:55.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.552064+0000 mgr.y (mgr.14550) 660 : cluster 0 pgmap v1090: 246 pgs: 7 unknown, 239 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.667821+0000 osd.5 (osd.5) 395 : cluster 0 39.1a scrub starts 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.675486+0000 osd.5 (osd.5) 396 : cluster 0 39.1a scrub ok 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.855839+0000 osd.7 (osd.7) 374 : cluster 0 2.8 scrub starts 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:53.858263+0000 osd.7 (osd.7) 375 : cluster 0 2.8 scrub ok 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.258627+0000 mon.c (mon.2) 1180 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:55.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:54.507587+0000 mon.a (mon.0) 3395 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.511244+0000 mon.a (mon.0) 3396 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.519030+0000 mon.c (mon.2) 1181 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: cluster 2024-03-11T21:39:54.521182+0000 mon.a (mon.0) 3397 : cluster 0 osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.525049+0000 mon.a (mon.0) 3398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.938962+0000 mon.c (mon.2) 1182 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.940553+0000 mon.c (mon.2) 1183 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:55.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:55 smithi203 bash[23654]: audit 2024-03-11T21:39:54.951151+0000 mon.a (mon.0) 3399 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:55.085 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:55.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:55.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.552064+0000 mgr.y (mgr.14550) 660 : cluster 0 pgmap v1090: 246 pgs: 7 unknown, 239 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.667821+0000 osd.5 (osd.5) 395 : cluster 0 39.1a scrub starts 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.675486+0000 osd.5 (osd.5) 396 : cluster 0 39.1a scrub ok 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.855839+0000 osd.7 (osd.7) 374 : cluster 0 2.8 scrub starts 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:53.858263+0000 osd.7 (osd.7) 375 : cluster 0 2.8 scrub ok 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.258627+0000 mon.c (mon.2) 1180 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:54.507587+0000 mon.a (mon.0) 3395 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.511244+0000 mon.a (mon.0) 3396 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.519030+0000 mon.c (mon.2) 1181 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: cluster 2024-03-11T21:39:54.521182+0000 mon.a (mon.0) 3397 : cluster 0 osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.525049+0000 mon.a (mon.0) 3398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.220 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.938962+0000 mon.c (mon.2) 1182 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.940553+0000 mon.c (mon.2) 1183 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[29284]: audit 2024-03-11T21:39:54.951151+0000 mon.a (mon.0) 3399 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.552064+0000 mgr.y (mgr.14550) 660 : cluster 0 pgmap v1090: 246 pgs: 7 unknown, 239 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.667821+0000 osd.5 (osd.5) 395 : cluster 0 39.1a scrub starts 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.675486+0000 osd.5 (osd.5) 396 : cluster 0 39.1a scrub ok 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.855839+0000 osd.7 (osd.7) 374 : cluster 0 2.8 scrub starts 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:53.858263+0000 osd.7 (osd.7) 375 : cluster 0 2.8 scrub ok 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.258627+0000 mon.c (mon.2) 1180 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:54.507587+0000 mon.a (mon.0) 3395 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.511244+0000 mon.a (mon.0) 3396 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.519030+0000 mon.c (mon.2) 1181 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: cluster 2024-03-11T21:39:54.521182+0000 mon.a (mon.0) 3397 : cluster 0 osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:39:55.221 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.525049+0000 mon.a (mon.0) 3398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:39:55.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.938962+0000 mon.c (mon.2) 1182 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:55.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.940553+0000 mon.c (mon.2) 1183 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:55.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:55 smithi143 bash[21512]: audit 2024-03-11T21:39:54.951151+0000 mon.a (mon.0) 3399 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:39:55.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:55.345 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:55.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:55.470 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:55.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.607 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:55.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:55.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:55.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:55.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:55.877 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:55.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:56.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:56.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: cluster 2024-03-11T21:39:54.820878+0000 osd.7 (osd.7) 376 : cluster 0 6.12 scrub starts 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: cluster 2024-03-11T21:39:54.822052+0000 osd.7 (osd.7) 377 : cluster 0 6.12 scrub ok 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.443456+0000 osd.0 (osd.0) 366 : cluster 0 3.1 scrub starts 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.448134+0000 osd.0 (osd.0) 367 : cluster 0 3.1 scrub ok 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: audit 2024-03-11T21:39:55.513631+0000 mon.a (mon.0) 3400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.517847+0000 mon.a (mon.0) 3401 : cluster 0 osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: audit 2024-03-11T21:39:55.520800+0000 mon.c (mon.2) 1184 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:56 smithi203 bash[23654]: audit 2024-03-11T21:39:55.521576+0000 mon.a (mon.0) 3402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.143 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:56.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: cluster 2024-03-11T21:39:54.820878+0000 osd.7 (osd.7) 376 : cluster 0 6.12 scrub starts 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: cluster 2024-03-11T21:39:54.822052+0000 osd.7 (osd.7) 377 : cluster 0 6.12 scrub ok 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.443456+0000 osd.0 (osd.0) 366 : cluster 0 3.1 scrub starts 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.448134+0000 osd.0 (osd.0) 367 : cluster 0 3.1 scrub ok 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: audit 2024-03-11T21:39:55.513631+0000 mon.a (mon.0) 3400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.517847+0000 mon.a (mon.0) 3401 : cluster 0 osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: audit 2024-03-11T21:39:55.520800+0000 mon.c (mon.2) 1184 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[29284]: audit 2024-03-11T21:39:55.521576+0000 mon.a (mon.0) 3402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: cluster 2024-03-11T21:39:54.820878+0000 osd.7 (osd.7) 376 : cluster 0 6.12 scrub starts 2024-03-11T21:39:56.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: cluster 2024-03-11T21:39:54.822052+0000 osd.7 (osd.7) 377 : cluster 0 6.12 scrub ok 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.443456+0000 osd.0 (osd.0) 366 : cluster 0 3.1 scrub starts 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.448134+0000 osd.0 (osd.0) 367 : cluster 0 3.1 scrub ok 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: audit 2024-03-11T21:39:55.513631+0000 mon.a (mon.0) 3400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.517847+0000 mon.a (mon.0) 3401 : cluster 0 osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: audit 2024-03-11T21:39:55.520800+0000 mon.c (mon.2) 1184 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.260 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:56 smithi143 bash[21512]: audit 2024-03-11T21:39:55.521576+0000 mon.a (mon.0) 3402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]: dispatch 2024-03-11T21:39:56.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:56.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:56.403 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:56.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:56.536 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:56.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:56.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:56.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:56.901 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:57.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:57.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:57.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:57.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.259229+0000 osd.1 (osd.1) 179 : cluster 0 39.1c scrub starts 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.260744+0000 osd.1 (osd.1) 180 : cluster 0 39.1c scrub ok 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:55.552627+0000 mgr.y (mgr.14550) 661 : cluster 0 pgmap v1093: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.425173+0000 osd.0 (osd.0) 368 : cluster 0 5.5 scrub starts 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.427301+0000 osd.0 (osd.0) 369 : cluster 0 5.5 scrub ok 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.450652+0000 osd.4 (osd.4) 367 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.450955+0000 osd.4 (osd.4) 368 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.515075+0000 mon.a (mon.0) 3403 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: audit 2024-03-11T21:39:56.525667+0000 mon.a (mon.0) 3404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]': finished 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: cluster 2024-03-11T21:39:56.533046+0000 mon.a (mon.0) 3405 : cluster 0 osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:39:57.191 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[29284]: audit 2024-03-11T21:39:56.568464+0000 mon.c (mon.2) 1185 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.259229+0000 osd.1 (osd.1) 179 : cluster 0 39.1c scrub starts 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.260744+0000 osd.1 (osd.1) 180 : cluster 0 39.1c scrub ok 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:55.552627+0000 mgr.y (mgr.14550) 661 : cluster 0 pgmap v1093: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.425173+0000 osd.0 (osd.0) 368 : cluster 0 5.5 scrub starts 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.427301+0000 osd.0 (osd.0) 369 : cluster 0 5.5 scrub ok 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.450652+0000 osd.4 (osd.4) 367 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.450955+0000 osd.4 (osd.4) 368 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.515075+0000 mon.a (mon.0) 3403 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: audit 2024-03-11T21:39:56.525667+0000 mon.a (mon.0) 3404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]': finished 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: cluster 2024-03-11T21:39:56.533046+0000 mon.a (mon.0) 3405 : cluster 0 osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:39:57.192 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:57 smithi143 bash[21512]: audit 2024-03-11T21:39:56.568464+0000 mon.c (mon.2) 1185 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.259229+0000 osd.1 (osd.1) 179 : cluster 0 39.1c scrub starts 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.260744+0000 osd.1 (osd.1) 180 : cluster 0 39.1c scrub ok 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:55.552627+0000 mgr.y (mgr.14550) 661 : cluster 0 pgmap v1093: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.425173+0000 osd.0 (osd.0) 368 : cluster 0 5.5 scrub starts 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.427301+0000 osd.0 (osd.0) 369 : cluster 0 5.5 scrub ok 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.450652+0000 osd.4 (osd.4) 367 : cluster 0 purged_snaps scrub starts 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.450955+0000 osd.4 (osd.4) 368 : cluster 0 purged_snaps scrub ok 2024-03-11T21:39:57.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.515075+0000 mon.a (mon.0) 3403 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:39:57.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: audit 2024-03-11T21:39:56.525667+0000 mon.a (mon.0) 3404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-116", "mode": "writeback"}]': finished 2024-03-11T21:39:57.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: cluster 2024-03-11T21:39:56.533046+0000 mon.a (mon.0) 3405 : cluster 0 osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:39:57.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:57 smithi203 bash[23654]: audit 2024-03-11T21:39:56.568464+0000 mon.c (mon.2) 1185 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:57.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.310 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:57.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:57.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:57.565 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:57.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:57.694 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:57.756 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:57.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:57.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:57.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:57.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:58.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:58.085 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:58.189 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[21512]: audit 2024-03-11T21:39:56.568676+0000 mgr.y (mgr.14550) 662 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:58.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.686942+0000 osd.2 (osd.2) 329 : cluster 0 39.1f scrub starts 2024-03-11T21:39:58.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.692640+0000 osd.2 (osd.2) 330 : cluster 0 39.1f scrub ok 2024-03-11T21:39:58.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[29284]: audit 2024-03-11T21:39:56.568676+0000 mgr.y (mgr.14550) 662 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:58.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.686942+0000 osd.2 (osd.2) 329 : cluster 0 39.1f scrub starts 2024-03-11T21:39:58.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:58 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.692640+0000 osd.2 (osd.2) 330 : cluster 0 39.1f scrub ok 2024-03-11T21:39:58.220 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:58.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:58 smithi203 bash[23654]: audit 2024-03-11T21:39:56.568676+0000 mgr.y (mgr.14550) 662 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "316.4"}]: dispatch 2024-03-11T21:39:58.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:58 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.686942+0000 osd.2 (osd.2) 329 : cluster 0 39.1f scrub starts 2024-03-11T21:39:58.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:58 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.692640+0000 osd.2 (osd.2) 330 : cluster 0 39.1f scrub ok 2024-03-11T21:39:58.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:58.348 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:58.454 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.481 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:58.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:58.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:58.740 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:58.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.870 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:58.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:58.999 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:59.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:59.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:59.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:59.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:39:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:39:59.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.213860+0000 osd.1 (osd.1) 181 : cluster 0 316.4 scrub starts 2024-03-11T21:39:59.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.217604+0000 osd.1 (osd.1) 182 : cluster 0 316.4 scrub ok 2024-03-11T21:39:59.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.553451+0000 mgr.y (mgr.14550) 663 : cluster 0 pgmap v1095: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:59.244 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.730643+0000 osd.6 (osd.6) 299 : cluster 0 5.2 scrub starts 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.732709+0000 osd.6 (osd.6) 300 : cluster 0 5.2 scrub ok 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.776323+0000 osd.4 (osd.4) 369 : cluster 0 9.10 scrub starts 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:57.778294+0000 osd.4 (osd.4) 370 : cluster 0 9.10 scrub ok 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.003781+0000 osd.3 (osd.3) 385 : cluster 0 2.19 scrub starts 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.005741+0000 osd.3 (osd.3) 386 : cluster 0 2.19 scrub ok 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.425800+0000 osd.0 (osd.0) 370 : cluster 0 9.1e scrub starts 2024-03-11T21:39:59.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:39:59 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.428402+0000 osd.0 (osd.0) 371 : cluster 0 9.1e scrub ok 2024-03-11T21:39:59.260 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:59.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:59.375 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.213860+0000 osd.1 (osd.1) 181 : cluster 0 316.4 scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.217604+0000 osd.1 (osd.1) 182 : cluster 0 316.4 scrub ok 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.553451+0000 mgr.y (mgr.14550) 663 : cluster 0 pgmap v1095: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.730643+0000 osd.6 (osd.6) 299 : cluster 0 5.2 scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.732709+0000 osd.6 (osd.6) 300 : cluster 0 5.2 scrub ok 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.776323+0000 osd.4 (osd.4) 369 : cluster 0 9.10 scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:57.778294+0000 osd.4 (osd.4) 370 : cluster 0 9.10 scrub ok 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.003781+0000 osd.3 (osd.3) 385 : cluster 0 2.19 scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.005741+0000 osd.3 (osd.3) 386 : cluster 0 2.19 scrub ok 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.425800+0000 osd.0 (osd.0) 370 : cluster 0 9.1e scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.428402+0000 osd.0 (osd.0) 371 : cluster 0 9.1e scrub ok 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.213860+0000 osd.1 (osd.1) 181 : cluster 0 316.4 scrub starts 2024-03-11T21:39:59.376 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.217604+0000 osd.1 (osd.1) 182 : cluster 0 316.4 scrub ok 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.553451+0000 mgr.y (mgr.14550) 663 : cluster 0 pgmap v1095: 246 pgs: 246 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.730643+0000 osd.6 (osd.6) 299 : cluster 0 5.2 scrub starts 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.732709+0000 osd.6 (osd.6) 300 : cluster 0 5.2 scrub ok 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.776323+0000 osd.4 (osd.4) 369 : cluster 0 9.10 scrub starts 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:57.778294+0000 osd.4 (osd.4) 370 : cluster 0 9.10 scrub ok 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.003781+0000 osd.3 (osd.3) 385 : cluster 0 2.19 scrub starts 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.005741+0000 osd.3 (osd.3) 386 : cluster 0 2.19 scrub ok 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.425800+0000 osd.0 (osd.0) 370 : cluster 0 9.1e scrub starts 2024-03-11T21:39:59.377 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:39:59 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.428402+0000 osd.0 (osd.0) 371 : cluster 0 9.1e scrub ok 2024-03-11T21:39:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:59.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:59.521 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:59.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:59.653 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:59.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:59.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:59.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:59.917 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:00.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.769664+0000 osd.4 (osd.4) 371 : cluster 0 5.e scrub starts 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.772168+0000 osd.4 (osd.4) 372 : cluster 0 5.e scrub ok 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.840181+0000 osd.7 (osd.7) 378 : cluster 0 3.e scrub starts 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:58.843900+0000 osd.7 (osd.7) 379 : cluster 0 3.e scrub ok 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: audit 2024-03-11T21:39:58.925935+0000 mgr.y (mgr.14550) 664 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.032351+0000 osd.3 (osd.3) 387 : cluster 0 5.0 scrub starts 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.035138+0000 osd.3 (osd.3) 388 : cluster 0 5.0 scrub ok 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.658567+0000 osd.2 (osd.2) 331 : cluster 0 5.6 scrub starts 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.660926+0000 osd.2 (osd.2) 332 : cluster 0 5.6 scrub ok 2024-03-11T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000153+0000 mon.a (mon.0) 3406 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; nodeep-scrub flag(s) set; 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000210+0000 mon.a (mon.0) 3407 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000238+0000 mon.a (mon.0) 3408 : cluster 3 pool 'test-rados-api-smithi143-71773-116' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000267+0000 mon.a (mon.0) 3409 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000294+0000 mon.a (mon.0) 3410 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000320+0000 mon.a (mon.0) 3411 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000351+0000 mon.a (mon.0) 3412 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000375+0000 mon.a (mon.0) 3413 : cluster 3 application not enabled on pool 'test-rados-api-smithi143-71773-111' 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.000395+0000 mon.a (mon.0) 3414 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.769664+0000 osd.4 (osd.4) 371 : cluster 0 5.e scrub starts 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.772168+0000 osd.4 (osd.4) 372 : cluster 0 5.e scrub ok 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.840181+0000 osd.7 (osd.7) 378 : cluster 0 3.e scrub starts 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:58.843900+0000 osd.7 (osd.7) 379 : cluster 0 3.e scrub ok 2024-03-11T21:40:00.031 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: audit 2024-03-11T21:39:58.925935+0000 mgr.y (mgr.14550) 664 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.032351+0000 osd.3 (osd.3) 387 : cluster 0 5.0 scrub starts 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.035138+0000 osd.3 (osd.3) 388 : cluster 0 5.0 scrub ok 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.658567+0000 osd.2 (osd.2) 331 : cluster 0 5.6 scrub starts 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.660926+0000 osd.2 (osd.2) 332 : cluster 0 5.6 scrub ok 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000153+0000 mon.a (mon.0) 3406 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; nodeep-scrub flag(s) set; 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000210+0000 mon.a (mon.0) 3407 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000238+0000 mon.a (mon.0) 3408 : cluster 3 pool 'test-rados-api-smithi143-71773-116' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000267+0000 mon.a (mon.0) 3409 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000294+0000 mon.a (mon.0) 3410 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000320+0000 mon.a (mon.0) 3411 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000351+0000 mon.a (mon.0) 3412 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000375+0000 mon.a (mon.0) 3413 : cluster 3 application not enabled on pool 'test-rados-api-smithi143-71773-111' 2024-03-11T21:40:00.032 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.000395+0000 mon.a (mon.0) 3414 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- 172.21.15.143:0/488358520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 msgr2=0x7f617410ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- 172.21.15.143:0/488358520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617410ed10 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f616c00b0a0 tx=0x7f616c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- 172.21.15.143:0/488358520 shutdown_connections 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- 172.21.15.143:0/488358520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174107ed0 0x7f61741082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- 172.21.15.143:0/488358520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617410ed10 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- 172.21.15.143:0/488358520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174101e90 0x7f61741022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- 172.21.15.143:0/488358520 >> 172.21.15.143:0/488358520 conn(0x7f6174078f90 msgr2=0x7f61740793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- 172.21.15.143:0/488358520 shutdown_connections 2024-03-11T21:40:00.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- 172.21.15.143:0/488358520 wait complete. 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 librados: starting msgr at 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 librados: starting objecter 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 Processor -- start 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- start start 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 librados: setting wanted keys 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 librados: calling monclient init 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:00.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617419dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174107ed0 0x7f6174197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6174111650 con 0x7f6174101e90 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741114d0 con 0x7f6174107ed0 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617b8b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61741117d0 con 0x7f6174102830 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617964f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617964f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43128/0 (socket says 172.21.15.143:43128) 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617964f700 1 -- 172.21.15.143:0/2256665327 learned_addr learned my addr 172.21.15.143:0/2256665327 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f6178e4e700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617419dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f6179e50700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174107ed0 0x7f6174197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617964f700 1 -- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 msgr2=0x7f617419dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.052+0000 7f617964f700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617419dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617964f700 1 -- 172.21.15.143:0/2256665327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174107ed0 msgr2=0x7f6174197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617964f700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174107ed0 0x7f6174197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:00.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617964f700 1 -- 172.21.15.143:0/2256665327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174198080 con 0x7f6174101e90 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617964f700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f617000eb30 tx=0x7f617000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 10 client.?.objecter ms_handle_connect 0x7f6174101e90 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000cc90 con 0x7f6174101e90 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f617000ce30 con 0x7f6174101e90 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170003df0 con 0x7f6174101e90 2024-03-11T21:40:00.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174198370 con 0x7f6174101e90 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617b8b3700 10 client.30741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741a4630 con 0x7f6174101e90 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617b8b3700 1 librados: init done 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617404faa0 con 0x7f6174101e90 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f6170003f90 con 0x7f6174101e90 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 0x7f6160071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(709..709 src has 87..709) v4 ==== 10149+0+0 (secure 0 0 0) 0x7f617008eee0 con 0x7f6174101e90 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 10 client.30741.objecter ms_dispatch 0x7f6174001550 osd_map(709..709 src has 87..709) v4 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 3 client.30741.objecter handle_osd_map got epochs [709,709] > 0 2024-03-11T21:40:00.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 3 client.30741.objecter handle_osd_map decoding full epoch 709 2024-03-11T21:40:00.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f6178e4e700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 0x7f6160071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:00.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 20 client.30741.objecter dump_active .. 0 homeless 2024-03-11T21:40:00.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f6178e4e700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 0x7f6160071ce0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f6174079980 tx=0x7f616400a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:00.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.056+0000 7f616a7fc700 10 client.30741.objecter ms_handle_connect 0x7f616006f820 2024-03-11T21:40:00.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:00.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.064+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f617005b9a0 con 0x7f6174101e90 2024-03-11T21:40:00.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.769664+0000 osd.4 (osd.4) 371 : cluster 0 5.e scrub starts 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.772168+0000 osd.4 (osd.4) 372 : cluster 0 5.e scrub ok 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.840181+0000 osd.7 (osd.7) 378 : cluster 0 3.e scrub starts 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:58.843900+0000 osd.7 (osd.7) 379 : cluster 0 3.e scrub ok 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: audit 2024-03-11T21:39:58.925935+0000 mgr.y (mgr.14550) 664 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.032351+0000 osd.3 (osd.3) 387 : cluster 0 5.0 scrub starts 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.035138+0000 osd.3 (osd.3) 388 : cluster 0 5.0 scrub ok 2024-03-11T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.658567+0000 osd.2 (osd.2) 331 : cluster 0 5.6 scrub starts 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.660926+0000 osd.2 (osd.2) 332 : cluster 0 5.6 scrub ok 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000153+0000 mon.a (mon.0) 3406 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; nodeep-scrub flag(s) set; 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000210+0000 mon.a (mon.0) 3407 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000238+0000 mon.a (mon.0) 3408 : cluster 3 pool 'test-rados-api-smithi143-71773-116' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000267+0000 mon.a (mon.0) 3409 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000294+0000 mon.a (mon.0) 3410 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000320+0000 mon.a (mon.0) 3411 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000351+0000 mon.a (mon.0) 3412 : cluster 3 application not enabled on pool 'SnapListsmithi143-72399-1' 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000375+0000 mon.a (mon.0) 3413 : cluster 3 application not enabled on pool 'test-rados-api-smithi143-71773-111' 2024-03-11T21:40:00.158 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.000395+0000 mon.a (mon.0) 3414 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:40:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:00.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:00.276 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:00.292+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f6174198e50 con 0x7f6174101e90 2024-03-11T21:40:00.304 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:00.306 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:00.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:00.474 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:40:00.474 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:40:00.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:00.564 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:00.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:00.691 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:00.794 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:00.818 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:00.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:00.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:01.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.028+0000 7f616a7fc700 1 -- 172.21.15.143:0/2256665327 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v710) v1 ==== 113+0+0 (secure 0 0 0) 0x7f617005f3f0 con 0x7f6174101e90 2024-03-11T21:40:01.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 10 librados: watch_flush enter 2024-03-11T21:40:01.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 10 librados: watch_flush exit 2024-03-11T21:40:01.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 20 client.30741.objecter shutdown clearing up homeless session... 2024-03-11T21:40:01.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 10 client.30741.objecter successfully canceled tick 2024-03-11T21:40:01.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 msgr2=0x7f6160071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 0x7f6160071ce0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f6174079980 tx=0x7f616400a380 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 msgr2=0x7f617419d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f617000eb30 tx=0x7f617000ee40 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 shutdown_connections 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f616006f820 0x7f6160071ce0 unknown :-1 s=CLOSED pgs=3434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6174101e90 0x7f617419d760 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6174107ed0 0x7f6174197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 --2- 172.21.15.143:0/2256665327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6174102830 0x7f617419dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 >> 172.21.15.143:0/2256665327 conn(0x7f6174078f90 msgr2=0x7f6174105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 shutdown_connections 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 -- 172.21.15.143:0/2256665327 wait complete. 2024-03-11T21:40:01.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:01.032+0000 7f617b8b3700 1 librados: shutdown 2024-03-11T21:40:01.017 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:40:01.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:01.076 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:01.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.287783+0000 osd.1 (osd.1) 183 : cluster 0 3.19 scrub starts 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.292509+0000 osd.1 (osd.1) 184 : cluster 0 3.19 scrub ok 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.554693+0000 mgr.y (mgr.14550) 665 : cluster 0 pgmap v1096: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.801853+0000 osd.7 (osd.7) 380 : cluster 0 6.8 scrub starts 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:39:59.803810+0000 osd.7 (osd.7) 381 : cluster 0 6.8 scrub ok 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: audit 2024-03-11T21:40:00.298691+0000 mon.a (mon.0) 3415 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.676748+0000 osd.2 (osd.2) 333 : cluster 0 2.c scrub starts 2024-03-11T21:40:01.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.678195+0000 osd.2 (osd.2) 334 : cluster 0 2.c scrub ok 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.287783+0000 osd.1 (osd.1) 183 : cluster 0 3.19 scrub starts 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.292509+0000 osd.1 (osd.1) 184 : cluster 0 3.19 scrub ok 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.554693+0000 mgr.y (mgr.14550) 665 : cluster 0 pgmap v1096: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.801853+0000 osd.7 (osd.7) 380 : cluster 0 6.8 scrub starts 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:39:59.803810+0000 osd.7 (osd.7) 381 : cluster 0 6.8 scrub ok 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: audit 2024-03-11T21:40:00.298691+0000 mon.a (mon.0) 3415 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.676748+0000 osd.2 (osd.2) 333 : cluster 0 2.c scrub starts 2024-03-11T21:40:01.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:01 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.678195+0000 osd.2 (osd.2) 334 : cluster 0 2.c scrub ok 2024-03-11T21:40:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.287783+0000 osd.1 (osd.1) 183 : cluster 0 3.19 scrub starts 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.292509+0000 osd.1 (osd.1) 184 : cluster 0 3.19 scrub ok 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.554693+0000 mgr.y (mgr.14550) 665 : cluster 0 pgmap v1096: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.801853+0000 osd.7 (osd.7) 380 : cluster 0 6.8 scrub starts 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:39:59.803810+0000 osd.7 (osd.7) 381 : cluster 0 6.8 scrub ok 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: audit 2024-03-11T21:40:00.298691+0000 mon.a (mon.0) 3415 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:40:01.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.676748+0000 osd.2 (osd.2) 333 : cluster 0 2.c scrub starts 2024-03-11T21:40:01.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:01 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.678195+0000 osd.2 (osd.2) 334 : cluster 0 2.c scrub ok 2024-03-11T21:40:01.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:01.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:01.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:01.468 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:01.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:01.599 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:01.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:01.729 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:01.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.860 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:01.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:01.992 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.329696+0000 osd.1 (osd.1) 185 : cluster 0 5.c scrub starts 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.331843+0000 osd.1 (osd.1) 186 : cluster 0 5.c scrub ok 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.698967+0000 osd.6 (osd.6) 301 : cluster 0 1.7 scrub starts 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.701170+0000 osd.6 (osd.6) 302 : cluster 0 1.7 scrub ok 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.854191+0000 osd.4 (osd.4) 373 : cluster 0 39.3 scrub starts 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:00.858091+0000 osd.4 (osd.4) 374 : cluster 0 39.3 scrub ok 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: audit 2024-03-11T21:40:01.032125+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.035880+0000 mon.a (mon.0) 3417 : cluster 0 osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.477696+0000 osd.0 (osd.0) 372 : cluster 0 9.9 scrub starts 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.480167+0000 osd.0 (osd.0) 373 : cluster 0 9.9 scrub ok 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.329696+0000 osd.1 (osd.1) 185 : cluster 0 5.c scrub starts 2024-03-11T21:40:02.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.331843+0000 osd.1 (osd.1) 186 : cluster 0 5.c scrub ok 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.698967+0000 osd.6 (osd.6) 301 : cluster 0 1.7 scrub starts 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.701170+0000 osd.6 (osd.6) 302 : cluster 0 1.7 scrub ok 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.854191+0000 osd.4 (osd.4) 373 : cluster 0 39.3 scrub starts 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:00.858091+0000 osd.4 (osd.4) 374 : cluster 0 39.3 scrub ok 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: audit 2024-03-11T21:40:01.032125+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.035880+0000 mon.a (mon.0) 3417 : cluster 0 osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.477696+0000 osd.0 (osd.0) 372 : cluster 0 9.9 scrub starts 2024-03-11T21:40:02.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.480167+0000 osd.0 (osd.0) 373 : cluster 0 9.9 scrub ok 2024-03-11T21:40:02.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:02.119 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:02.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.252 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.329696+0000 osd.1 (osd.1) 185 : cluster 0 5.c scrub starts 2024-03-11T21:40:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.331843+0000 osd.1 (osd.1) 186 : cluster 0 5.c scrub ok 2024-03-11T21:40:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.698967+0000 osd.6 (osd.6) 301 : cluster 0 1.7 scrub starts 2024-03-11T21:40:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.701170+0000 osd.6 (osd.6) 302 : cluster 0 1.7 scrub ok 2024-03-11T21:40:02.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.854191+0000 osd.4 (osd.4) 373 : cluster 0 39.3 scrub starts 2024-03-11T21:40:02.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:00.858091+0000 osd.4 (osd.4) 374 : cluster 0 39.3 scrub ok 2024-03-11T21:40:02.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: audit 2024-03-11T21:40:01.032125+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.143:0/2256665327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:40:02.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.035880+0000 mon.a (mon.0) 3417 : cluster 0 osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:40:02.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.477696+0000 osd.0 (osd.0) 372 : cluster 0 9.9 scrub starts 2024-03-11T21:40:02.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:02 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.480167+0000 osd.0 (osd.0) 373 : cluster 0 9.9 scrub ok 2024-03-11T21:40:02.341 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:02.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:02.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:02.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.514 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:02.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:02.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:02.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:02.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:02.906 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:03.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:03.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:03.164 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:03.267 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.381000+0000 osd.1 (osd.1) 187 : cluster 0 5.1d scrub starts 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.383110+0000 osd.1 (osd.1) 188 : cluster 0 5.1d scrub ok 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.555496+0000 mgr.y (mgr.14550) 666 : cluster 0 pgmap v1098: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 508 B/s wr, 2 op/s 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.696005+0000 osd.6 (osd.6) 303 : cluster 0 1.6 scrub starts 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[29284]: cluster 2024-03-11T21:40:01.698572+0000 osd.6 (osd.6) 304 : cluster 0 1.6 scrub ok 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.381000+0000 osd.1 (osd.1) 187 : cluster 0 5.1d scrub starts 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.383110+0000 osd.1 (osd.1) 188 : cluster 0 5.1d scrub ok 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.555496+0000 mgr.y (mgr.14550) 666 : cluster 0 pgmap v1098: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 508 B/s wr, 2 op/s 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.696005+0000 osd.6 (osd.6) 303 : cluster 0 1.6 scrub starts 2024-03-11T21:40:03.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:03 smithi143 bash[21512]: cluster 2024-03-11T21:40:01.698572+0000 osd.6 (osd.6) 304 : cluster 0 1.6 scrub ok 2024-03-11T21:40:03.293 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:03.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.425 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:03.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:03 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.381000+0000 osd.1 (osd.1) 187 : cluster 0 5.1d scrub starts 2024-03-11T21:40:03.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:03 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.383110+0000 osd.1 (osd.1) 188 : cluster 0 5.1d scrub ok 2024-03-11T21:40:03.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:03 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.555496+0000 mgr.y (mgr.14550) 666 : cluster 0 pgmap v1098: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 508 B/s wr, 2 op/s 2024-03-11T21:40:03.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:03 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.696005+0000 osd.6 (osd.6) 303 : cluster 0 1.6 scrub starts 2024-03-11T21:40:03.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:03 smithi203 bash[23654]: cluster 2024-03-11T21:40:01.698572+0000 osd.6 (osd.6) 304 : cluster 0 1.6 scrub ok 2024-03-11T21:40:03.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.555 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:03.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:03.684 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:03.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:03.813 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:03.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:03.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:04.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:04.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.767108+0000 osd.5 (osd.5) 397 : cluster 0 39.0 scrub starts 2024-03-11T21:40:04.058 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.769618+0000 osd.5 (osd.5) 398 : cluster 0 39.0 scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.828294+0000 osd.7 (osd.7) 382 : cluster 0 9.15 scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.830716+0000 osd.7 (osd.7) 383 : cluster 0 9.15 scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.847274+0000 osd.4 (osd.4) 375 : cluster 0 5.1c scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:02.849243+0000 osd.4 (osd.4) 376 : cluster 0 5.1c scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:03.107537+0000 osd.3 (osd.3) 389 : cluster 0 9.c scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[29284]: cluster 2024-03-11T21:40:03.109510+0000 osd.3 (osd.3) 390 : cluster 0 9.c scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.767108+0000 osd.5 (osd.5) 397 : cluster 0 39.0 scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.769618+0000 osd.5 (osd.5) 398 : cluster 0 39.0 scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.828294+0000 osd.7 (osd.7) 382 : cluster 0 9.15 scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.830716+0000 osd.7 (osd.7) 383 : cluster 0 9.15 scrub ok 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.847274+0000 osd.4 (osd.4) 375 : cluster 0 5.1c scrub starts 2024-03-11T21:40:04.059 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:02.849243+0000 osd.4 (osd.4) 376 : cluster 0 5.1c scrub ok 2024-03-11T21:40:04.060 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:03.107537+0000 osd.3 (osd.3) 389 : cluster 0 9.c scrub starts 2024-03-11T21:40:04.060 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:04 smithi143 bash[21512]: cluster 2024-03-11T21:40:03.109510+0000 osd.3 (osd.3) 390 : cluster 0 9.c scrub ok 2024-03-11T21:40:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:04.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:04.187 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.767108+0000 osd.5 (osd.5) 397 : cluster 0 39.0 scrub starts 2024-03-11T21:40:04.187 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.769618+0000 osd.5 (osd.5) 398 : cluster 0 39.0 scrub ok 2024-03-11T21:40:04.187 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.828294+0000 osd.7 (osd.7) 382 : cluster 0 9.15 scrub starts 2024-03-11T21:40:04.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.830716+0000 osd.7 (osd.7) 383 : cluster 0 9.15 scrub ok 2024-03-11T21:40:04.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.847274+0000 osd.4 (osd.4) 375 : cluster 0 5.1c scrub starts 2024-03-11T21:40:04.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:02.849243+0000 osd.4 (osd.4) 376 : cluster 0 5.1c scrub ok 2024-03-11T21:40:04.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:03.107537+0000 osd.3 (osd.3) 389 : cluster 0 9.c scrub starts 2024-03-11T21:40:04.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:04 smithi203 bash[23654]: cluster 2024-03-11T21:40:03.109510+0000 osd.3 (osd.3) 390 : cluster 0 9.c scrub ok 2024-03-11T21:40:04.204 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:04.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:04.335 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:04.438 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:04.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:04.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:04.597 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:04.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.728 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:04.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.857 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:04.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:04.989 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:05.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:05.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:05.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:05.251 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:05.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:05 smithi203 bash[23654]: cluster 2024-03-11T21:40:03.404428+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:40:05.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:05 smithi203 bash[23654]: cluster 2024-03-11T21:40:03.406759+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:40:05.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:05 smithi203 bash[23654]: cluster 2024-03-11T21:40:03.557717+0000 mgr.y (mgr.14550) 667 : cluster 0 pgmap v1099: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-11T21:40:05.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:05 smithi203 bash[23654]: cluster 2024-03-11T21:40:04.474986+0000 osd.0 (osd.0) 374 : cluster 0 2.1f scrub starts 2024-03-11T21:40:05.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:05 smithi203 bash[23654]: cluster 2024-03-11T21:40:04.477906+0000 osd.0 (osd.0) 375 : cluster 0 2.1f scrub ok 2024-03-11T21:40:05.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.380 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:05.384 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:40:05.394 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[21512]: cluster 2024-03-11T21:40:03.404428+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[21512]: cluster 2024-03-11T21:40:03.406759+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[21512]: cluster 2024-03-11T21:40:03.557717+0000 mgr.y (mgr.14550) 667 : cluster 0 pgmap v1099: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[21512]: cluster 2024-03-11T21:40:04.474986+0000 osd.0 (osd.0) 374 : cluster 0 2.1f scrub starts 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[21512]: cluster 2024-03-11T21:40:04.477906+0000 osd.0 (osd.0) 375 : cluster 0 2.1f scrub ok 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[29284]: cluster 2024-03-11T21:40:03.404428+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[29284]: cluster 2024-03-11T21:40:03.406759+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[29284]: cluster 2024-03-11T21:40:03.557717+0000 mgr.y (mgr.14550) 667 : cluster 0 pgmap v1099: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[29284]: cluster 2024-03-11T21:40:04.474986+0000 osd.0 (osd.0) 374 : cluster 0 2.1f scrub starts 2024-03-11T21:40:05.395 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:05 smithi143 bash[29284]: cluster 2024-03-11T21:40:04.477906+0000 osd.0 (osd.0) 375 : cluster 0 2.1f scrub ok 2024-03-11T21:40:05.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:40:05.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:05.512 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:05.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:05.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:05.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:05.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:06.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:06.031 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:06.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:06.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:06.266 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:06.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:04.806722+0000 osd.7 (osd.7) 384 : cluster 0 5.1a scrub starts 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:04.809041+0000 osd.7 (osd.7) 385 : cluster 0 5.1a scrub ok 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.510152+0000 osd.0 (osd.0) 376 : cluster 0 3.1b scrub starts 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.512455+0000 osd.0 (osd.0) 377 : cluster 0 3.1b scrub ok 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.708112+0000 osd.2 (osd.2) 335 : cluster 0 6.f scrub starts 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.709348+0000 osd.2 (osd.2) 336 : cluster 0 6.f scrub ok 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:04.806722+0000 osd.7 (osd.7) 384 : cluster 0 5.1a scrub starts 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:04.809041+0000 osd.7 (osd.7) 385 : cluster 0 5.1a scrub ok 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.510152+0000 osd.0 (osd.0) 376 : cluster 0 3.1b scrub starts 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.512455+0000 osd.0 (osd.0) 377 : cluster 0 3.1b scrub ok 2024-03-11T21:40:06.297 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.708112+0000 osd.2 (osd.2) 335 : cluster 0 6.f scrub starts 2024-03-11T21:40:06.298 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:06 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.709348+0000 osd.2 (osd.2) 336 : cluster 0 6.f scrub ok 2024-03-11T21:40:06.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:06.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:04.806722+0000 osd.7 (osd.7) 384 : cluster 0 5.1a scrub starts 2024-03-11T21:40:06.408 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:04.809041+0000 osd.7 (osd.7) 385 : cluster 0 5.1a scrub ok 2024-03-11T21:40:06.409 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.510152+0000 osd.0 (osd.0) 376 : cluster 0 3.1b scrub starts 2024-03-11T21:40:06.409 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.512455+0000 osd.0 (osd.0) 377 : cluster 0 3.1b scrub ok 2024-03-11T21:40:06.409 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.708112+0000 osd.2 (osd.2) 335 : cluster 0 6.f scrub starts 2024-03-11T21:40:06.409 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:06 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.709348+0000 osd.2 (osd.2) 336 : cluster 0 6.f scrub ok 2024-03-11T21:40:06.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:06.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:06.551 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:06.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:06.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:06.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:06.806 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:06.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- 172.21.15.143:0/2766837520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 msgr2=0x7fc2f4110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:06.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- 172.21.15.143:0/2766837520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f4110f40 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00b0d0 tx=0x7fc2ec00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- 172.21.15.143:0/2766837520 shutdown_connections 2024-03-11T21:40:06.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- 172.21.15.143:0/2766837520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f4110f40 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- 172.21.15.143:0/2766837520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 0x7fc2f4104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- 172.21.15.143:0/2766837520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f4108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- 172.21.15.143:0/2766837520 >> 172.21.15.143:0/2766837520 conn(0x7fc2f40ffb40 msgr2=0x7fc2f4101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:06.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- 172.21.15.143:0/2766837520 shutdown_connections 2024-03-11T21:40:06.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- 172.21.15.143:0/2766837520 wait complete. 2024-03-11T21:40:06.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 librados: starting msgr at 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 librados: starting objecter 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 Processor -- start 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- start start 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 librados: setting wanted keys 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 librados: calling monclient init 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 0x7fc2f4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4113870 con 0x7fc2f4104c40 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f41136f0 con 0x7fc2f4104290 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2fa140700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f41139f0 con 0x7fc2f4109370 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:06.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37958/0 (socket says 172.21.15.143:37958) 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34254/0 (socket says 172.21.15.143:34254) 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 -- 172.21.15.143:0/1061051756 learned_addr learned my addr 172.21.15.143:0/1061051756 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f37fe700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 0x7fc2f4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 -- 172.21.15.143:0/1061051756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 msgr2=0x7fc2f4199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 0x7fc2f4199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 -- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 msgr2=0x7fc2f419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f3fff700 1 -- 172.21.15.143:0/1061051756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f419edb0 con 0x7fc2f4109370 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.915+0000 7fc2f2ffd700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f3fff700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec006070 tx=0x7fc2ec007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 10 client.?.objecter ms_handle_connect 0x7fc2f4109370 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec021020 con 0x7fc2f4109370 2024-03-11T21:40:06.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f419f040 con 0x7fc2f4109370 2024-03-11T21:40:06.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc2ec004c50 con 0x7fc2f4109370 2024-03-11T21:40:06.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec005050 con 0x7fc2f4109370 2024-03-11T21:40:06.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2fa140700 10 client.41951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:06.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f41a6920 con 0x7fc2f4109370 2024-03-11T21:40:06.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2fa140700 1 librados: init done 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f404faa0 con 0x7fc2f4109370 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc2ec024020 con 0x7fc2f4109370 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 0x7fc2e0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(710..710 src has 87..710) v4 ==== 10149+0+0 (secure 0 0 0) 0x7fc2ec00f020 con 0x7fc2f4109370 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 10 client.41951.objecter ms_dispatch 0x7fc2f4001550 osd_map(710..710 src has 87..710) v4 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 3 client.41951.objecter handle_osd_map got epochs [710,710] > 0 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 3 client.41951.objecter handle_osd_map decoding full epoch 710 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f37fe700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 0x7fc2e0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 20 client.41951.objecter dump_active .. 0 homeless 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f0ff9700 10 client.41951.objecter ms_handle_connect 0x7fc2e006f820 2024-03-11T21:40:06.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.919+0000 7fc2f37fe700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 0x7fc2e0071ce0 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005950 tx=0x7fc2e4005cf0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:06.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:06.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:06.923+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc2ec05c840 con 0x7fc2f4109370 2024-03-11T21:40:06.928 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:07.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:07.057 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:07.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:07.163+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc2f41a6af0 con 0x7fc2f4109370 2024-03-11T21:40:07.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.412710+0000 osd.1 (osd.1) 191 : cluster 0 9.0 scrub starts 2024-03-11T21:40:07.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.414365+0000 osd.1 (osd.1) 192 : cluster 0 9.0 scrub ok 2024-03-11T21:40:07.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.558667+0000 mgr.y (mgr.14550) 668 : cluster 0 pgmap v1100: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 340 B/s wr, 1 op/s 2024-03-11T21:40:07.171 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.824645+0000 osd.5 (osd.5) 399 : cluster 0 3.5 scrub starts 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:05.829084+0000 osd.5 (osd.5) 400 : cluster 0 3.5 scrub ok 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.156869+0000 osd.3 (osd.3) 391 : cluster 0 3.8 scrub starts 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.159818+0000 osd.3 (osd.3) 392 : cluster 0 3.8 scrub ok 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.479044+0000 osd.0 (osd.0) 378 : cluster 0 3.1f scrub starts 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.481090+0000 osd.0 (osd.0) 379 : cluster 0 3.1f scrub ok 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.660504+0000 osd.2 (osd.2) 337 : cluster 0 5.d scrub starts 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.663503+0000 osd.2 (osd.2) 338 : cluster 0 5.d scrub ok 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[21512]: audit 2024-03-11T21:40:07.072651+0000 mon.a (mon.0) 3418 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.412710+0000 osd.1 (osd.1) 191 : cluster 0 9.0 scrub starts 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.414365+0000 osd.1 (osd.1) 192 : cluster 0 9.0 scrub ok 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.558667+0000 mgr.y (mgr.14550) 668 : cluster 0 pgmap v1100: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 340 B/s wr, 1 op/s 2024-03-11T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.824645+0000 osd.5 (osd.5) 399 : cluster 0 3.5 scrub starts 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:05.829084+0000 osd.5 (osd.5) 400 : cluster 0 3.5 scrub ok 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.156869+0000 osd.3 (osd.3) 391 : cluster 0 3.8 scrub starts 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.159818+0000 osd.3 (osd.3) 392 : cluster 0 3.8 scrub ok 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.479044+0000 osd.0 (osd.0) 378 : cluster 0 3.1f scrub starts 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.481090+0000 osd.0 (osd.0) 379 : cluster 0 3.1f scrub ok 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.660504+0000 osd.2 (osd.2) 337 : cluster 0 5.d scrub starts 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.663503+0000 osd.2 (osd.2) 338 : cluster 0 5.d scrub ok 2024-03-11T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:07 smithi143 bash[29284]: audit 2024-03-11T21:40:07.072651+0000 mon.a (mon.0) 3418 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:07.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:07.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:07.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:07.412 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.438 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:07.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.412710+0000 osd.1 (osd.1) 191 : cluster 0 9.0 scrub starts 2024-03-11T21:40:07.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.414365+0000 osd.1 (osd.1) 192 : cluster 0 9.0 scrub ok 2024-03-11T21:40:07.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.558667+0000 mgr.y (mgr.14550) 668 : cluster 0 pgmap v1100: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 340 B/s wr, 1 op/s 2024-03-11T21:40:07.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.824645+0000 osd.5 (osd.5) 399 : cluster 0 3.5 scrub starts 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:05.829084+0000 osd.5 (osd.5) 400 : cluster 0 3.5 scrub ok 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.156869+0000 osd.3 (osd.3) 391 : cluster 0 3.8 scrub starts 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.159818+0000 osd.3 (osd.3) 392 : cluster 0 3.8 scrub ok 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.479044+0000 osd.0 (osd.0) 378 : cluster 0 3.1f scrub starts 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.481090+0000 osd.0 (osd.0) 379 : cluster 0 3.1f scrub ok 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.660504+0000 osd.2 (osd.2) 337 : cluster 0 5.d scrub starts 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.663503+0000 osd.2 (osd.2) 338 : cluster 0 5.d scrub ok 2024-03-11T21:40:07.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:07 smithi203 bash[23654]: audit 2024-03-11T21:40:07.072651+0000 mon.a (mon.0) 3418 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:07.541 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:07.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:07.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:07.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:07.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.836 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:07.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:07.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:08.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:08.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.435831+0000 osd.1 (osd.1) 193 : cluster 0 2.6 scrub starts 2024-03-11T21:40:08.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.437596+0000 osd.1 (osd.1) 194 : cluster 0 2.6 scrub ok 2024-03-11T21:40:08.075 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.584780+0000 osd.6 (osd.6) 305 : cluster 0 3.a scrub starts 2024-03-11T21:40:08.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.103+0000 7fc2f0ff9700 1 -- 172.21.15.143:0/1061051756 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v711) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc2ec062020 con 0x7fc2f4109370 2024-03-11T21:40:08.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 10 librados: watch_flush enter 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 10 librados: watch_flush exit 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 20 client.41951.objecter shutdown clearing up homeless session... 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 10 client.41951.objecter successfully canceled tick 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 msgr2=0x7fc2e0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 0x7fc2e0071ce0 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005950 tx=0x7fc2e4005cf0 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 msgr2=0x7fc2f419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec006070 tx=0x7fc2ec007a20 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 shutdown_connections 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc2e006f820 0x7fc2e0071ce0 unknown :-1 s=CLOSED pgs=3435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f4109370 0x7fc2f419e670 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f4104290 0x7fc2f4199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 --2- 172.21.15.143:0/1061051756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f4104c40 0x7fc2f419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:08.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 >> 172.21.15.143:0/1061051756 conn(0x7fc2f40ffb40 msgr2=0x7fc2f41000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:08.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 shutdown_connections 2024-03-11T21:40:08.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 -- 172.21.15.143:0/1061051756 wait complete. 2024-03-11T21:40:08.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:08.107+0000 7fc2fa140700 1 librados: shutdown 2024-03-11T21:40:08.091 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:40:08.101 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:08.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:08.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.435831+0000 osd.1 (osd.1) 193 : cluster 0 2.6 scrub starts 2024-03-11T21:40:08.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.437596+0000 osd.1 (osd.1) 194 : cluster 0 2.6 scrub ok 2024-03-11T21:40:08.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.584780+0000 osd.6 (osd.6) 305 : cluster 0 3.a scrub starts 2024-03-11T21:40:08.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.587397+0000 osd.6 (osd.6) 306 : cluster 0 3.a scrub ok 2024-03-11T21:40:08.217 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.834685+0000 osd.7 (osd.7) 386 : cluster 0 2.1d scrub starts 2024-03-11T21:40:08.217 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: cluster 2024-03-11T21:40:06.836859+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub ok 2024-03-11T21:40:08.217 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: audit 2024-03-11T21:40:07.083241+0000 mon.c (mon.2) 1186 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.217 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: audit 2024-03-11T21:40:07.170210+0000 mon.c (mon.2) 1187 : audit 1 from='client.? 172.21.15.143:0/1061051756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.217 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[23654]: audit 2024-03-11T21:40:07.170718+0000 mon.a (mon.0) 3419 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.233 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:08.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.587397+0000 osd.6 (osd.6) 306 : cluster 0 3.a scrub ok 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.834685+0000 osd.7 (osd.7) 386 : cluster 0 2.1d scrub starts 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: cluster 2024-03-11T21:40:06.836859+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub ok 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: audit 2024-03-11T21:40:07.083241+0000 mon.c (mon.2) 1186 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: audit 2024-03-11T21:40:07.170210+0000 mon.c (mon.2) 1187 : audit 1 from='client.? 172.21.15.143:0/1061051756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.348 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[21512]: audit 2024-03-11T21:40:07.170718+0000 mon.a (mon.0) 3419 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.435831+0000 osd.1 (osd.1) 193 : cluster 0 2.6 scrub starts 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.437596+0000 osd.1 (osd.1) 194 : cluster 0 2.6 scrub ok 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.584780+0000 osd.6 (osd.6) 305 : cluster 0 3.a scrub starts 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.587397+0000 osd.6 (osd.6) 306 : cluster 0 3.a scrub ok 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.834685+0000 osd.7 (osd.7) 386 : cluster 0 2.1d scrub starts 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: cluster 2024-03-11T21:40:06.836859+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub ok 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: audit 2024-03-11T21:40:07.083241+0000 mon.c (mon.2) 1186 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: audit 2024-03-11T21:40:07.170210+0000 mon.c (mon.2) 1187 : audit 1 from='client.? 172.21.15.143:0/1061051756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.349 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:08 smithi143 bash[29284]: audit 2024-03-11T21:40:07.170718+0000 mon.a (mon.0) 3419 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:08.363 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:08.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:08.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:08.594 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:08.621 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:08.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.752 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:08.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:08.882 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:08.985 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.012 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:09.027 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:09.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:09.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:09.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:07.559194+0000 mgr.y (mgr.14550) 669 : cluster 0 pgmap v1101: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 307 B/s wr, 1 op/s 2024-03-11T21:40:09.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.078801+0000 mon.a (mon.0) 3420 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:40:09.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: audit 2024-03-11T21:40:08.090069+0000 mon.a (mon.0) 3421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.099111+0000 mon.a (mon.0) 3422 : cluster 0 osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.199765+0000 osd.3 (osd.3) 393 : cluster 0 9.1 scrub starts 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.202866+0000 osd.3 (osd.3) 394 : cluster 0 9.1 scrub ok 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.473274+0000 osd.0 (osd.0) 380 : cluster 0 3.17 scrub starts 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.476633+0000 osd.0 (osd.0) 381 : cluster 0 3.17 scrub ok 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.646060+0000 osd.2 (osd.2) 339 : cluster 0 5.a scrub starts 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.647542+0000 osd.2 (osd.2) 340 : cluster 0 5.a scrub ok 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:07.559194+0000 mgr.y (mgr.14550) 669 : cluster 0 pgmap v1101: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 307 B/s wr, 1 op/s 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.078801+0000 mon.a (mon.0) 3420 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: audit 2024-03-11T21:40:08.090069+0000 mon.a (mon.0) 3421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.099111+0000 mon.a (mon.0) 3422 : cluster 0 osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:40:09.257 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.199765+0000 osd.3 (osd.3) 393 : cluster 0 9.1 scrub starts 2024-03-11T21:40:09.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.202866+0000 osd.3 (osd.3) 394 : cluster 0 9.1 scrub ok 2024-03-11T21:40:09.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.473274+0000 osd.0 (osd.0) 380 : cluster 0 3.17 scrub starts 2024-03-11T21:40:09.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.476633+0000 osd.0 (osd.0) 381 : cluster 0 3.17 scrub ok 2024-03-11T21:40:09.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.646060+0000 osd.2 (osd.2) 339 : cluster 0 5.a scrub starts 2024-03-11T21:40:09.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:09 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.647542+0000 osd.2 (osd.2) 340 : cluster 0 5.a scrub ok 2024-03-11T21:40:09.271 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:07.559194+0000 mgr.y (mgr.14550) 669 : cluster 0 pgmap v1101: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 307 B/s wr, 1 op/s 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.078801+0000 mon.a (mon.0) 3420 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: audit 2024-03-11T21:40:08.090069+0000 mon.a (mon.0) 3421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.099111+0000 mon.a (mon.0) 3422 : cluster 0 osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.199765+0000 osd.3 (osd.3) 393 : cluster 0 9.1 scrub starts 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.202866+0000 osd.3 (osd.3) 394 : cluster 0 9.1 scrub ok 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.473274+0000 osd.0 (osd.0) 380 : cluster 0 3.17 scrub starts 2024-03-11T21:40:09.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.476633+0000 osd.0 (osd.0) 381 : cluster 0 3.17 scrub ok 2024-03-11T21:40:09.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.646060+0000 osd.2 (osd.2) 339 : cluster 0 5.a scrub starts 2024-03-11T21:40:09.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:09 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.647542+0000 osd.2 (osd.2) 340 : cluster 0 5.a scrub ok 2024-03-11T21:40:09.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:09.401 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:09.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.533 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:09.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:09.663 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:09.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.794 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:09.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:09.926 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:10.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:10.057 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 -- 172.21.15.143:0/756990724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 msgr2=0x7fd1dc110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/756990724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc110c60 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400b0a0 tx=0x7fd1d400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 -- 172.21.15.143:0/756990724 shutdown_connections 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/756990724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc110c60 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/756990724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 0x7fd1dc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/756990724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1dc074930 0x7fd1dc10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 -- 172.21.15.143:0/756990724 >> 172.21.15.143:0/756990724 conn(0x7fd1dc078eb0 msgr2=0x7fd1dc0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 -- 172.21.15.143:0/756990724 shutdown_connections 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.159+0000 7fd1e1afd700 1 -- 172.21.15.143:0/756990724 wait complete. 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 librados: starting msgr at 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 librados: starting objecter 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 Processor -- start 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- start start 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 librados: setting wanted keys 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 librados: calling monclient init 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 0x7fd1dc19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1dc074930 0x7fd1dc19ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1137d0 con 0x7fd1dc10e870 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc113650 con 0x7fd1dc074930 2024-03-11T21:40:10.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc113950 con 0x7fd1dc074010 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 0x7fd1dc19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 0x7fd1dc19fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37978/0 (socket says 172.21.15.143:37978) 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1db7fe700 1 -- 172.21.15.143:0/2996729310 learned_addr learned my addr 172.21.15.143:0/2996729310 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:34264/0 (socket says 172.21.15.143:34264) 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1dc074930 0x7fd1dc19ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 msgr2=0x7fd1dc19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1dc074010 0x7fd1dc19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1dc074930 msgr2=0x7fd1dc19ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1dc074930 0x7fd1dc19ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc19a3f0 con 0x7fd1dc10e870 2024-03-11T21:40:10.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400bfd0 tx=0x7fd1d4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 10 client.?.objecter ms_handle_connect 0x7fd1dc10e870 2024-03-11T21:40:10.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:10.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4021020 con 0x7fd1dc10e870 2024-03-11T21:40:10.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd1d4004c50 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4005050 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc19a680 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 10 client.30753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc1a67f0 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 librados: init done 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd1dc0756a0 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.163+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd1d4024020 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 0x7fd1c8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(711..711 src has 87..711) v4 ==== 10149+0+0 (secure 0 0 0) 0x7fd1d400f070 con 0x7fd1dc10e870 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 osd_map(711..711 src has 87..711) v4 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 3 client.30753.objecter handle_osd_map got epochs [711,711] > 0 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 3 client.30753.objecter handle_osd_map decoding full epoch 711 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 0x7fd1c8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 20 client.30753.objecter dump_active .. 0 homeless 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v711) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd1d402b020 con 0x7fd1dc10e870 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 1 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd1dc062c40 0x7fd1dc1079f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 1 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 0 1 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 4 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 10 client.30753.objecter _send_command 1 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc04faa0 con 0x7fd1dc062c40 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1c806f820 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 0x7fd1c8071ce0 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009680 tx=0x7fd1cc009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.149 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd1dc062c40 0x7fd1dc1079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc062c40 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd1dc062c40 0x7fd1dc1079f0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc04faa0 con 0x7fd1dc062c40 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 1 = system:0 2024-03-11T21:40:10.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 4 2024-03-11T21:40:10.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.167+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 0 1 2024-03-11T21:40:10.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 2 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 2 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc19afd0 osd=0 3 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 0 2 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 4 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 10 client.30753.objecter _send_command 2 2024-03-11T21:40:10.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc107f30 con 0x7fd1dc062c40 2024-03-11T21:40:10.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc107f30 con 0x7fd1dc062c40 2024-03-11T21:40:10.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 4 2024-03-11T21:40:10.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 0 2 2024-03-11T21:40:10.169 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:40:10.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 3 2024-03-11T21:40:10.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd1dc062830 0x7fd1dc1aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 1 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 4 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 10 client.30753.objecter _send_command 3 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc02d390 con 0x7fd1dc062830 2024-03-11T21:40:10.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd1dc062830 0x7fd1dc1aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc062830 2024-03-11T21:40:10.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.187+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd1dc062830 0x7fd1dc1aa9e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.191+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc02d390 con 0x7fd1dc062830 2024-03-11T21:40:10.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.191+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:40:10.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.191+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 3 = system:0 2024-03-11T21:40:10.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.191+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 4 2024-03-11T21:40:10.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.191+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 1 3 2024-03-11T21:40:10.181 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 4 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 4 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc0626c0 osd=1 3 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 1 4 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 4 2024-03-11T21:40:10.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 10 client.30753.objecter _send_command 4 2024-03-11T21:40:10.190 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.207+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1aaf20 con 0x7fd1dc062830 2024-03-11T21:40:10.190 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:40:10.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1aaf20 con 0x7fd1dc062830 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 4 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 1 4 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 5 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd1dc02ca70 0x7fd1dc1ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 5 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 2 5 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 4 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 10 client.30753.objecter _send_command 5 2024-03-11T21:40:10.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc108f10 con 0x7fd1dc02ca70 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd1dc02ca70 0x7fd1dc1ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc02ca70 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd1dc02ca70 0x7fd1dc1ae2c0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc108f10 con 0x7fd1dc02ca70 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 5 = system:0 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 4 2024-03-11T21:40:10.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.211+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 2 5 2024-03-11T21:40:10.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 6 2024-03-11T21:40:10.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 6 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc02c900 osd=2 3 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 2 6 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 4 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 10 client.30753.objecter _send_command 6 2024-03-11T21:40:10.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.227+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc108330 con 0x7fd1dc02ca70 2024-03-11T21:40:10.211 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:40:10.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc108330 con 0x7fd1dc02ca70 2024-03-11T21:40:10.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 4 2024-03-11T21:40:10.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 2 6 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 7 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1dc1abdb0 0x7fd1dc1b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 7 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 3 7 2024-03-11T21:40:10.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 4 2024-03-11T21:40:10.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 10 client.30753.objecter _send_command 7 2024-03-11T21:40:10.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc1b47f0 con 0x7fd1dc1abdb0 2024-03-11T21:40:10.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1dc1abdb0 0x7fd1dc1b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc1abdb0 2024-03-11T21:40:10.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1dc1abdb0 0x7fd1dc1b1990 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc1b47f0 con 0x7fd1dc1abdb0 2024-03-11T21:40:10.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:40:10.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 7 = system:0 2024-03-11T21:40:10.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 4 2024-03-11T21:40:10.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.231+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 3 7 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 8 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 8 2024-03-11T21:40:10.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc108580 osd=3 3 2024-03-11T21:40:10.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 3 8 2024-03-11T21:40:10.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 4 2024-03-11T21:40:10.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 10 client.30753.objecter _send_command 8 2024-03-11T21:40:10.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1ab3e0 con 0x7fd1dc1abdb0 2024-03-11T21:40:10.231 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1ab3e0 con 0x7fd1dc1abdb0 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 4 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 3 8 2024-03-11T21:40:10.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 9 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd1dc1af4e0 0x7fd1dc1b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 9 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 4 9 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 4 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 10 client.30753.objecter _send_command 9 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc1b7fa0 con 0x7fd1dc1af4e0 2024-03-11T21:40:10.233 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd1dc1af4e0 0x7fd1dc1b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc1af4e0 2024-03-11T21:40:10.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.251+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd1dc1af4e0 0x7fd1dc1b5060 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.255+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc1b7fa0 con 0x7fd1dc1af4e0 2024-03-11T21:40:10.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.255+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:40:10.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.255+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 9 = system:0 2024-03-11T21:40:10.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.255+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 4 2024-03-11T21:40:10.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.255+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 4 9 2024-03-11T21:40:10.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 10 2024-03-11T21:40:10.252 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 10 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ab3e0 osd=4 3 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 4 10 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 4 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 10 client.30753.objecter _send_command 10 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.271+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1aea50 con 0x7fd1dc1af4e0 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1aea50 con 0x7fd1dc1af4e0 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 4 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 4 10 2024-03-11T21:40:10.257 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 11 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd1dc1b2bb0 0x7fd1dc1b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 11 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 5 11 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 4 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 10 client.30753.objecter _send_command 11 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc1ae970 con 0x7fd1dc1b2bb0 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.275+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd1dc1b2bb0 0x7fd1dc1b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc1b2bb0 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1daffd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd1dc1b2bb0 0x7fd1dc1b55a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc1ae970 con 0x7fd1dc1b2bb0 2024-03-11T21:40:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:40:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 11 = system:0 2024-03-11T21:40:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 4 2024-03-11T21:40:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.279+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 5 11 2024-03-11T21:40:10.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 12 2024-03-11T21:40:10.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 12 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ae800 osd=5 3 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 5 12 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 4 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 10 client.30753.objecter _send_command 12 2024-03-11T21:40:10.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.295+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1ae970 con 0x7fd1dc1b2bb0 2024-03-11T21:40:10.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1ae970 con 0x7fd1dc1b2bb0 2024-03-11T21:40:10.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 4 2024-03-11T21:40:10.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 5 12 2024-03-11T21:40:10.282 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 13 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd1dc1b5ff0 0x7fd1dc1b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.299+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 13 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 6 13 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 4 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 10 client.30753.objecter _send_command 13 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc1b6aa0 con 0x7fd1dc1b5ff0 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd1dc1b5ff0 0x7fd1dc1b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1dbfff700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd1dc1b5ff0 0x7fd1dc1b6450 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc1b5ff0 2024-03-11T21:40:10.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1d405c2b0 con 0x7fd1dc1b5ff0 2024-03-11T21:40:10.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:40:10.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 13 = system:0 2024-03-11T21:40:10.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 4 2024-03-11T21:40:10.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.303+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 6 13 2024-03-11T21:40:10.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.419572+0000 osd.1 (osd.1) 195 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:40:10.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.421509+0000 osd.1 (osd.1) 196 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:40:10.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.643003+0000 osd.6 (osd.6) 307 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:08.645310+0000 osd.6 (osd.6) 308 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: audit 2024-03-11T21:40:08.934283+0000 mgr.y (mgr.14550) 670 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.509701+0000 osd.0 (osd.0) 382 : cluster 0 9.5 scrub starts 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.512286+0000 osd.0 (osd.0) 383 : cluster 0 9.5 scrub ok 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.419572+0000 osd.1 (osd.1) 195 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.421509+0000 osd.1 (osd.1) 196 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.643003+0000 osd.6 (osd.6) 307 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:08.645310+0000 osd.6 (osd.6) 308 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: audit 2024-03-11T21:40:08.934283+0000 mgr.y (mgr.14550) 670 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.509701+0000 osd.0 (osd.0) 382 : cluster 0 9.5 scrub starts 2024-03-11T21:40:10.293 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:10 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.512286+0000 osd.0 (osd.0) 383 : cluster 0 9.5 scrub ok 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 14 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 14 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1ae970 osd=6 3 2024-03-11T21:40:10.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 6 14 2024-03-11T21:40:10.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 4 2024-03-11T21:40:10.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 10 client.30753.objecter _send_command 14 2024-03-11T21:40:10.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.323+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1bc200 con 0x7fd1dc1b5ff0 2024-03-11T21:40:10.304 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:10.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1bc200 con 0x7fd1dc1b5ff0 2024-03-11T21:40:10.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 4 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 6 14 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 15 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd1dc1bc920 0x7fd1dc1bcda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 15 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 7 15 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 4 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 10 client.30753.objecter _send_command 15 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1dc1be570 con 0x7fd1dc1bc920 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd1dc1bc920 0x7fd1dc1bcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1d8ff9700 10 client.30753.objecter ms_handle_connect 0x7fd1dc1bc920 2024-03-11T21:40:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.327+0000 7fd1db7fe700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd1dc1bc920 0x7fd1dc1bcda0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:10.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.331+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1dc1be570 con 0x7fd1dc1bc920 2024-03-11T21:40:10.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.331+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:40:10.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.331+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 15 = system:0 2024-03-11T21:40:10.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.331+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 4 2024-03-11T21:40:10.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.331+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 7 15 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 10 client.30753.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign -1 16 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 20 client.30753.objecter _get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_remove -1 16 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 20 client.30753.objecter get_session s=0x7fd1dc1bc200 osd=7 3 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 15 client.30753.objecter _session_command_op_assign 7 16 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 4 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 10 client.30753.objecter _send_command 16 2024-03-11T21:40:10.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.347+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd1dc1be570 con 0x7fd1dc1bc920 2024-03-11T21:40:10.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1d8ff9700 1 -- 172.21.15.143:0/2996729310 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd1dc1be570 con 0x7fd1dc1bc920 2024-03-11T21:40:10.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1d8ff9700 10 client.30753.objecter ms_dispatch 0x7fd1dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:10.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1d8ff9700 10 client.30753.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:10.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1d8ff9700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 4 2024-03-11T21:40:10.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1d8ff9700 15 client.30753.objecter _session_command_op_remove 7 16 2024-03-11T21:40:10.334 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:40:10.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 librados: watch_flush enter 2024-03-11T21:40:10.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 librados: watch_flush exit 2024-03-11T21:40:10.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.0 2024-03-11T21:40:10.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd1dc062c40 msgr2=0x7fd1dc1079f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd1dc062c40 0x7fd1dc1079f0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc19afd0 osd=0 1 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.1 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd1dc062830 msgr2=0x7fd1dc1aa9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd1dc062830 0x7fd1dc1aa9e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc0626c0 osd=1 1 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.2 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd1dc02ca70 msgr2=0x7fd1dc1ae2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd1dc02ca70 0x7fd1dc1ae2c0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc02c900 osd=2 1 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.3 2024-03-11T21:40:10.338 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1dc1abdb0 msgr2=0x7fd1dc1b1990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd1dc1abdb0 0x7fd1dc1b1990 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1dbfff700 1 -- 172.21.15.143:0/2996729310 reap_dead start 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc108580 osd=3 1 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.4 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd1dc1af4e0 msgr2=0x7fd1dc1b5060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd1dc1af4e0 0x7fd1dc1b5060 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ab3e0 osd=4 1 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.5 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd1dc1b2bb0 msgr2=0x7fd1dc1b55a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd1dc1b2bb0 0x7fd1dc1b55a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae800 osd=5 1 2024-03-11T21:40:10.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.6 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd1dc1b5ff0 msgr2=0x7fd1dc1b6450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd1dc1b5ff0 0x7fd1dc1b6450 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1ae970 osd=6 1 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter close_session for osd.7 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd1dc1bc920 msgr2=0x7fd1dc1bcda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd1dc1bc920 0x7fd1dc1bcda0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter put_session s=0x7fd1dc1bc200 osd=7 1 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 20 client.30753.objecter shutdown clearing up homeless session... 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 10 client.30753.objecter successfully canceled tick 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1dbfff700 1 -- 172.21.15.143:0/2996729310 reap_dead start 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 msgr2=0x7fd1c8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 0x7fd1c8071ce0 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009680 tx=0x7fd1cc009260 comp rx=0 tx=0).stop 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 msgr2=0x7fd1dc199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400bfd0 tx=0x7fd1d4007a20 comp rx=0 tx=0).stop 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 shutdown_connections 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1dc10e870 0x7fd1dc199b10 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 --2- 172.21.15.143:0/2996729310 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd1c806f820 0x7fd1c8071ce0 unknown :-1 s=CLOSED pgs=3436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:10.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 >> 172.21.15.143:0/2996729310 conn(0x7fd1dc078eb0 msgr2=0x7fd1dc1062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 shutdown_connections 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 -- 172.21.15.143:0/2996729310 wait complete. 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:10.351+0000 7fd1e1afd700 1 librados: shutdown 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:10.344 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:10.345 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:10.345 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:10.345 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:10.345 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:10.345 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:10.406 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.426 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:10.527 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.419572+0000 osd.1 (osd.1) 195 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.421509+0000 osd.1 (osd.1) 196 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.643003+0000 osd.6 (osd.6) 307 : cluster 0 39.1e deep-scrub starts 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:08.645310+0000 osd.6 (osd.6) 308 : cluster 0 39.1e deep-scrub ok 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: audit 2024-03-11T21:40:08.934283+0000 mgr.y (mgr.14550) 670 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.509701+0000 osd.0 (osd.0) 382 : cluster 0 9.5 scrub starts 2024-03-11T21:40:10.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:10 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.512286+0000 osd.0 (osd.0) 383 : cluster 0 9.5 scrub ok 2024-03-11T21:40:10.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:10.556 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:10.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:10.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:10.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:10.821 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:10.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:10.953 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:11.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.397049+0000 osd.1 (osd.1) 197 : cluster 0 2.a scrub starts 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.399365+0000 osd.1 (osd.1) 198 : cluster 0 2.a scrub ok 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.561563+0000 mgr.y (mgr.14550) 671 : cluster 0 pgmap v1103: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.936160+0000 osd.4 (osd.4) 377 : cluster 0 3.14 scrub starts 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: cluster 2024-03-11T21:40:09.938190+0000 osd.4 (osd.4) 378 : cluster 0 3.14 scrub ok 2024-03-11T21:40:11.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[21512]: audit 2024-03-11T21:40:10.168021+0000 mon.a (mon.0) 3423 : audit 0 from='client.? 172.21.15.143:0/2996729310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:11.085 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:11.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:11.202 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.397049+0000 osd.1 (osd.1) 197 : cluster 0 2.a scrub starts 2024-03-11T21:40:11.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.399365+0000 osd.1 (osd.1) 198 : cluster 0 2.a scrub ok 2024-03-11T21:40:11.320 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.561563+0000 mgr.y (mgr.14550) 671 : cluster 0 pgmap v1103: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:11.321 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.936160+0000 osd.4 (osd.4) 377 : cluster 0 3.14 scrub starts 2024-03-11T21:40:11.321 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: cluster 2024-03-11T21:40:09.938190+0000 osd.4 (osd.4) 378 : cluster 0 3.14 scrub ok 2024-03-11T21:40:11.321 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:11 smithi203 bash[23654]: audit 2024-03-11T21:40:10.168021+0000 mon.a (mon.0) 3423 : audit 0 from='client.? 172.21.15.143:0/2996729310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.323 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:11.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:11.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.397049+0000 osd.1 (osd.1) 197 : cluster 0 2.a scrub starts 2024-03-11T21:40:11.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.399365+0000 osd.1 (osd.1) 198 : cluster 0 2.a scrub ok 2024-03-11T21:40:11.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.561563+0000 mgr.y (mgr.14550) 671 : cluster 0 pgmap v1103: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:11.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.936160+0000 osd.4 (osd.4) 377 : cluster 0 3.14 scrub starts 2024-03-11T21:40:11.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: cluster 2024-03-11T21:40:09.938190+0000 osd.4 (osd.4) 378 : cluster 0 3.14 scrub ok 2024-03-11T21:40:11.436 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:11 smithi143 bash[29284]: audit 2024-03-11T21:40:10.168021+0000 mon.a (mon.0) 3423 : audit 0 from='client.? 172.21.15.143:0/2996729310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:11.452 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:11.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:11.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:11.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:11.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:11.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:11.849 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:11.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:11.981 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:12.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:40:12.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.113 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:12.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:10.610159+0000 osd.6 (osd.6) 309 : cluster 0 2.14 scrub starts 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:10.612444+0000 osd.6 (osd.6) 310 : cluster 0 2.14 scrub ok 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:10.824804+0000 osd.5 (osd.5) 401 : cluster 0 2.3 scrub starts 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:10.827172+0000 osd.5 (osd.5) 402 : cluster 0 2.3 scrub ok 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.490354+0000 mon.a (mon.0) 3424 : cluster 0 osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.681429+0000 osd.2 (osd.2) 341 : cluster 0 2.1 scrub starts 2024-03-11T21:40:12.227 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:12 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.683428+0000 osd.2 (osd.2) 342 : cluster 0 2.1 scrub ok 2024-03-11T21:40:12.241 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:12.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:12.373 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:12.381 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:12.382 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:12.405 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:10.610159+0000 osd.6 (osd.6) 309 : cluster 0 2.14 scrub starts 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:10.612444+0000 osd.6 (osd.6) 310 : cluster 0 2.14 scrub ok 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:10.824804+0000 osd.5 (osd.5) 401 : cluster 0 2.3 scrub starts 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:10.827172+0000 osd.5 (osd.5) 402 : cluster 0 2.3 scrub ok 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.490354+0000 mon.a (mon.0) 3424 : cluster 0 osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.681429+0000 osd.2 (osd.2) 341 : cluster 0 2.1 scrub starts 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.683428+0000 osd.2 (osd.2) 342 : cluster 0 2.1 scrub ok 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:10.610159+0000 osd.6 (osd.6) 309 : cluster 0 2.14 scrub starts 2024-03-11T21:40:12.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:10.612444+0000 osd.6 (osd.6) 310 : cluster 0 2.14 scrub ok 2024-03-11T21:40:12.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:10.824804+0000 osd.5 (osd.5) 401 : cluster 0 2.3 scrub starts 2024-03-11T21:40:12.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:10.827172+0000 osd.5 (osd.5) 402 : cluster 0 2.3 scrub ok 2024-03-11T21:40:12.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.490354+0000 mon.a (mon.0) 3424 : cluster 0 osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:40:12.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.681429+0000 osd.2 (osd.2) 341 : cluster 0 2.1 scrub starts 2024-03-11T21:40:12.407 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:12 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.683428+0000 osd.2 (osd.2) 342 : cluster 0 2.1 scrub ok 2024-03-11T21:40:12.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:12.510 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:12.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:12.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:12.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.770 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:12.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:12.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:13.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:13.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:13.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:13.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.562407+0000 mgr.y (mgr.14550) 672 : cluster 0 pgmap v1105: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.655330+0000 osd.6 (osd.6) 311 : cluster 0 5.11 scrub starts 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: cluster 2024-03-11T21:40:11.657673+0000 osd.6 (osd.6) 312 : cluster 0 5.11 scrub ok 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: cluster 2024-03-11T21:40:12.518995+0000 mon.a (mon.0) 3425 : cluster 0 osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: audit 2024-03-11T21:40:12.584101+0000 mon.c (mon.2) 1188 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.191 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[21512]: audit 2024-03-11T21:40:12.584616+0000 mon.a (mon.0) 3426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.562407+0000 mgr.y (mgr.14550) 672 : cluster 0 pgmap v1105: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.655330+0000 osd.6 (osd.6) 311 : cluster 0 5.11 scrub starts 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: cluster 2024-03-11T21:40:11.657673+0000 osd.6 (osd.6) 312 : cluster 0 5.11 scrub ok 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: cluster 2024-03-11T21:40:12.518995+0000 mon.a (mon.0) 3425 : cluster 0 osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: audit 2024-03-11T21:40:12.584101+0000 mon.c (mon.2) 1188 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.192 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:13 smithi143 bash[29284]: audit 2024-03-11T21:40:12.584616+0000 mon.a (mon.0) 3426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:13.296 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:13.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:40:13.399 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:13.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:13.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:13.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.562407+0000 mgr.y (mgr.14550) 672 : cluster 0 pgmap v1105: 246 pgs: 246 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:13.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.655330+0000 osd.6 (osd.6) 311 : cluster 0 5.11 scrub starts 2024-03-11T21:40:13.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: cluster 2024-03-11T21:40:11.657673+0000 osd.6 (osd.6) 312 : cluster 0 5.11 scrub ok 2024-03-11T21:40:13.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: cluster 2024-03-11T21:40:12.518995+0000 mon.a (mon.0) 3425 : cluster 0 osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:40:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: audit 2024-03-11T21:40:12.584101+0000 mon.c (mon.2) 1188 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:13 smithi203 bash[23654]: audit 2024-03-11T21:40:12.584616+0000 mon.a (mon.0) 3426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:13.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:13.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:13.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:13.805 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:13.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:13.933 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:14.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:14.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:14.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.193 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:14.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.324 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:14.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:14.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:14.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:14.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.712 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:14.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: audit 2024-03-11T21:40:13.507398+0000 mon.a (mon.0) 3427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.519744+0000 mon.a (mon.0) 3428 : cluster 0 osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: audit 2024-03-11T21:40:13.521772+0000 mon.c (mon.2) 1189 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: audit 2024-03-11T21:40:13.522264+0000 mon.a (mon.0) 3429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.530338+0000 osd.0 (osd.0) 384 : cluster 0 39.12 deep-scrub starts 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.531995+0000 osd.0 (osd.0) 385 : cluster 0 39.12 deep-scrub ok 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.563052+0000 mgr.y (mgr.14550) 673 : cluster 0 pgmap v1108: 246 pgs: 5 active+clean+snaptrim, 241 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.604009+0000 osd.6 (osd.6) 313 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.608125+0000 osd.6 (osd.6) 314 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.897153+0000 osd.7 (osd.7) 388 : cluster 0 6.1c scrub starts 2024-03-11T21:40:14.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:14 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.900655+0000 osd.7 (osd.7) 389 : cluster 0 6.1c scrub ok 2024-03-11T21:40:14.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.826 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: audit 2024-03-11T21:40:13.507398+0000 mon.a (mon.0) 3427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.519744+0000 mon.a (mon.0) 3428 : cluster 0 osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: audit 2024-03-11T21:40:13.521772+0000 mon.c (mon.2) 1189 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: audit 2024-03-11T21:40:13.522264+0000 mon.a (mon.0) 3429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.530338+0000 osd.0 (osd.0) 384 : cluster 0 39.12 deep-scrub starts 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.531995+0000 osd.0 (osd.0) 385 : cluster 0 39.12 deep-scrub ok 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.563052+0000 mgr.y (mgr.14550) 673 : cluster 0 pgmap v1108: 246 pgs: 5 active+clean+snaptrim, 241 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.604009+0000 osd.6 (osd.6) 313 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.608125+0000 osd.6 (osd.6) 314 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.897153+0000 osd.7 (osd.7) 388 : cluster 0 6.1c scrub starts 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.900655+0000 osd.7 (osd.7) 389 : cluster 0 6.1c scrub ok 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: audit 2024-03-11T21:40:13.507398+0000 mon.a (mon.0) 3427 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:14.827 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.519744+0000 mon.a (mon.0) 3428 : cluster 0 osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: audit 2024-03-11T21:40:13.521772+0000 mon.c (mon.2) 1189 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: audit 2024-03-11T21:40:13.522264+0000 mon.a (mon.0) 3429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]: dispatch 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.530338+0000 osd.0 (osd.0) 384 : cluster 0 39.12 deep-scrub starts 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.531995+0000 osd.0 (osd.0) 385 : cluster 0 39.12 deep-scrub ok 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.563052+0000 mgr.y (mgr.14550) 673 : cluster 0 pgmap v1108: 246 pgs: 5 active+clean+snaptrim, 241 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.604009+0000 osd.6 (osd.6) 313 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.608125+0000 osd.6 (osd.6) 314 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.897153+0000 osd.7 (osd.7) 388 : cluster 0 6.1c scrub starts 2024-03-11T21:40:14.828 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:14 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.900655+0000 osd.7 (osd.7) 389 : cluster 0 6.1c scrub ok 2024-03-11T21:40:14.839 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:14.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:15.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:15.101 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:15.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:15.232 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:15.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.363 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:15.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:15.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.775915+0000 osd.5 (osd.5) 403 : cluster 0 2.16 scrub starts 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.779401+0000 osd.5 (osd.5) 404 : cluster 0 2.16 scrub ok 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.882572+0000 osd.4 (osd.4) 379 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:13.889348+0000 osd.4 (osd.4) 380 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.508450+0000 mon.a (mon.0) 3430 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: audit 2024-03-11T21:40:14.518648+0000 mon.a (mon.0) 3431 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.524961+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.593629+0000 osd.2 (osd.2) 343 : cluster 0 5.8 scrub starts 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.594992+0000 osd.2 (osd.2) 344 : cluster 0 5.8 scrub ok 2024-03-11T21:40:15.603 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.600797+0000 osd.6 (osd.6) 315 : cluster 0 5.1f scrub starts 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.604043+0000 osd.6 (osd.6) 316 : cluster 0 5.1f scrub ok 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.775915+0000 osd.5 (osd.5) 403 : cluster 0 2.16 scrub starts 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.779401+0000 osd.5 (osd.5) 404 : cluster 0 2.16 scrub ok 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.882572+0000 osd.4 (osd.4) 379 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:13.889348+0000 osd.4 (osd.4) 380 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.508450+0000 mon.a (mon.0) 3430 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: audit 2024-03-11T21:40:14.518648+0000 mon.a (mon.0) 3431 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.524961+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.593629+0000 osd.2 (osd.2) 343 : cluster 0 5.8 scrub starts 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.594992+0000 osd.2 (osd.2) 344 : cluster 0 5.8 scrub ok 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.600797+0000 osd.6 (osd.6) 315 : cluster 0 5.1f scrub starts 2024-03-11T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:15 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.604043+0000 osd.6 (osd.6) 316 : cluster 0 5.1f scrub ok 2024-03-11T21:40:15.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:15.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:15.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.775915+0000 osd.5 (osd.5) 403 : cluster 0 2.16 scrub starts 2024-03-11T21:40:15.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.779401+0000 osd.5 (osd.5) 404 : cluster 0 2.16 scrub ok 2024-03-11T21:40:15.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.882572+0000 osd.4 (osd.4) 379 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:40:15.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:13.889348+0000 osd.4 (osd.4) 380 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.508450+0000 mon.a (mon.0) 3430 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: audit 2024-03-11T21:40:14.518648+0000 mon.a (mon.0) 3431 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-116"}]': finished 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.524961+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.593629+0000 osd.2 (osd.2) 343 : cluster 0 5.8 scrub starts 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.594992+0000 osd.2 (osd.2) 344 : cluster 0 5.8 scrub ok 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.600797+0000 osd.6 (osd.6) 315 : cluster 0 5.1f scrub starts 2024-03-11T21:40:15.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:15 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.604043+0000 osd.6 (osd.6) 316 : cluster 0 5.1f scrub ok 2024-03-11T21:40:15.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:15.871 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:15.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:15.994 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:16.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:16.126 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:16.229 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:16.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:16.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.385 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:16.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:16.515 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:16.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.755971+0000 osd.5 (osd.5) 405 : cluster 0 9.6 scrub starts 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:14.758073+0000 osd.5 (osd.5) 406 : cluster 0 9.6 scrub ok 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.375487+0000 osd.3 (osd.3) 395 : cluster 0 3.1e scrub starts 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.377630+0000 osd.3 (osd.3) 396 : cluster 0 3.1e scrub ok 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.533026+0000 mon.a (mon.0) 3433 : cluster 0 osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.563500+0000 mgr.y (mgr.14550) 674 : cluster 0 pgmap v1111: 214 pgs: 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:40:16.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.612140+0000 osd.2 (osd.2) 345 : cluster 0 39.16 scrub starts 2024-03-11T21:40:16.632 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.615186+0000 osd.2 (osd.2) 346 : cluster 0 39.16 scrub ok 2024-03-11T21:40:16.632 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.972985+0000 osd.6 (osd.6) 317 : cluster 0 3.10 scrub starts 2024-03-11T21:40:16.632 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:16 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.976927+0000 osd.6 (osd.6) 318 : cluster 0 3.10 scrub ok 2024-03-11T21:40:16.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:16.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:16.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.755971+0000 osd.5 (osd.5) 405 : cluster 0 9.6 scrub starts 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:14.758073+0000 osd.5 (osd.5) 406 : cluster 0 9.6 scrub ok 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.375487+0000 osd.3 (osd.3) 395 : cluster 0 3.1e scrub starts 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.377630+0000 osd.3 (osd.3) 396 : cluster 0 3.1e scrub ok 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.533026+0000 mon.a (mon.0) 3433 : cluster 0 osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:40:16.778 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.563500+0000 mgr.y (mgr.14550) 674 : cluster 0 pgmap v1111: 214 pgs: 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.612140+0000 osd.2 (osd.2) 345 : cluster 0 39.16 scrub starts 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.615186+0000 osd.2 (osd.2) 346 : cluster 0 39.16 scrub ok 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.972985+0000 osd.6 (osd.6) 317 : cluster 0 3.10 scrub starts 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.976927+0000 osd.6 (osd.6) 318 : cluster 0 3.10 scrub ok 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.755971+0000 osd.5 (osd.5) 405 : cluster 0 9.6 scrub starts 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:14.758073+0000 osd.5 (osd.5) 406 : cluster 0 9.6 scrub ok 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.375487+0000 osd.3 (osd.3) 395 : cluster 0 3.1e scrub starts 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.377630+0000 osd.3 (osd.3) 396 : cluster 0 3.1e scrub ok 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.533026+0000 mon.a (mon.0) 3433 : cluster 0 osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.563500+0000 mgr.y (mgr.14550) 674 : cluster 0 pgmap v1111: 214 pgs: 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:40:16.779 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.612140+0000 osd.2 (osd.2) 345 : cluster 0 39.16 scrub starts 2024-03-11T21:40:16.780 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.615186+0000 osd.2 (osd.2) 346 : cluster 0 39.16 scrub ok 2024-03-11T21:40:16.780 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.972985+0000 osd.6 (osd.6) 317 : cluster 0 3.10 scrub starts 2024-03-11T21:40:16.780 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:16 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.976927+0000 osd.6 (osd.6) 318 : cluster 0 3.10 scrub ok 2024-03-11T21:40:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:16.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:16.907 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:17.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.030 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:17.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:17.161 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:17.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.293 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:17.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:17.399 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:17.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:17.465 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:17.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.558 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:17.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.687 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:17.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.844426+0000 osd.4 (osd.4) 381 : cluster 0 3.9 scrub starts 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.854082+0000 osd.4 (osd.4) 382 : cluster 0 3.9 scrub ok 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.904214+0000 osd.5 (osd.5) 407 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:15.904560+0000 osd.5 (osd.5) 408 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:16.551195+0000 mon.a (mon.0) 3434 : cluster 0 osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: audit 2024-03-11T21:40:16.558567+0000 mon.c (mon.2) 1190 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: audit 2024-03-11T21:40:16.559882+0000 mon.a (mon.0) 3435 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:16.586522+0000 osd.2 (osd.2) 347 : cluster 0 39.6 scrub starts 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:16.588764+0000 osd.2 (osd.2) 348 : cluster 0 39.6 scrub ok 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:16.864229+0000 osd.7 (osd.7) 390 : cluster 0 5.10 scrub starts 2024-03-11T21:40:17.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:17 smithi203 bash[23654]: cluster 2024-03-11T21:40:16.866534+0000 osd.7 (osd.7) 391 : cluster 0 5.10 scrub ok 2024-03-11T21:40:17.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:17.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:17.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.844426+0000 osd.4 (osd.4) 381 : cluster 0 3.9 scrub starts 2024-03-11T21:40:17.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.854082+0000 osd.4 (osd.4) 382 : cluster 0 3.9 scrub ok 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.904214+0000 osd.5 (osd.5) 407 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:15.904560+0000 osd.5 (osd.5) 408 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:16.551195+0000 mon.a (mon.0) 3434 : cluster 0 osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: audit 2024-03-11T21:40:16.558567+0000 mon.c (mon.2) 1190 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: audit 2024-03-11T21:40:16.559882+0000 mon.a (mon.0) 3435 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:16.586522+0000 osd.2 (osd.2) 347 : cluster 0 39.6 scrub starts 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:16.588764+0000 osd.2 (osd.2) 348 : cluster 0 39.6 scrub ok 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:16.864229+0000 osd.7 (osd.7) 390 : cluster 0 5.10 scrub starts 2024-03-11T21:40:17.931 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[29284]: cluster 2024-03-11T21:40:16.866534+0000 osd.7 (osd.7) 391 : cluster 0 5.10 scrub ok 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.844426+0000 osd.4 (osd.4) 381 : cluster 0 3.9 scrub starts 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.854082+0000 osd.4 (osd.4) 382 : cluster 0 3.9 scrub ok 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.904214+0000 osd.5 (osd.5) 407 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:15.904560+0000 osd.5 (osd.5) 408 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:16.551195+0000 mon.a (mon.0) 3434 : cluster 0 osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: audit 2024-03-11T21:40:16.558567+0000 mon.c (mon.2) 1190 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: audit 2024-03-11T21:40:16.559882+0000 mon.a (mon.0) 3435 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:16.586522+0000 osd.2 (osd.2) 347 : cluster 0 39.6 scrub starts 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:16.588764+0000 osd.2 (osd.2) 348 : cluster 0 39.6 scrub ok 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:16.864229+0000 osd.7 (osd.7) 390 : cluster 0 5.10 scrub starts 2024-03-11T21:40:17.932 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:17 smithi143 bash[21512]: cluster 2024-03-11T21:40:16.866534+0000 osd.7 (osd.7) 391 : cluster 0 5.10 scrub ok 2024-03-11T21:40:17.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.051 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.077 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.337 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:18.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:18.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.363376+0000 osd.1 (osd.1) 199 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.364991+0000 osd.1 (osd.1) 200 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.497302+0000 osd.3 (osd.3) 397 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.497540+0000 osd.3 (osd.3) 398 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: audit 2024-03-11T21:40:17.546106+0000 mon.a (mon.0) 3436 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.550426+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.564120+0000 mgr.y (mgr.14550) 675 : cluster 0 pgmap v1114: 246 pgs: 32 unknown, 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.601760+0000 osd.2 (osd.2) 349 : cluster 0 2.e scrub starts 2024-03-11T21:40:18.712 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.604091+0000 osd.2 (osd.2) 350 : cluster 0 2.e scrub ok 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.363376+0000 osd.1 (osd.1) 199 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.364991+0000 osd.1 (osd.1) 200 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.497302+0000 osd.3 (osd.3) 397 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.497540+0000 osd.3 (osd.3) 398 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: audit 2024-03-11T21:40:17.546106+0000 mon.a (mon.0) 3436 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.550426+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.564120+0000 mgr.y (mgr.14550) 675 : cluster 0 pgmap v1114: 246 pgs: 32 unknown, 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.601760+0000 osd.2 (osd.2) 349 : cluster 0 2.e scrub starts 2024-03-11T21:40:18.713 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:18 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.604091+0000 osd.2 (osd.2) 350 : cluster 0 2.e scrub ok 2024-03-11T21:40:18.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:18.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.363376+0000 osd.1 (osd.1) 199 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:40:18.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.364991+0000 osd.1 (osd.1) 200 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:40:18.841 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.497302+0000 osd.3 (osd.3) 397 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.497540+0000 osd.3 (osd.3) 398 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: audit 2024-03-11T21:40:17.546106+0000 mon.a (mon.0) 3436 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.550426+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.564120+0000 mgr.y (mgr.14550) 675 : cluster 0 pgmap v1114: 246 pgs: 32 unknown, 3 active+clean+snaptrim, 211 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.601760+0000 osd.2 (osd.2) 349 : cluster 0 2.e scrub starts 2024-03-11T21:40:18.842 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.604091+0000 osd.2 (osd.2) 350 : cluster 0 2.e scrub ok 2024-03-11T21:40:18.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:18.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:18.988 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:19.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:19.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:19.119 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:19.223 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:19.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:19.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:19.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.917238+0000 osd.4 (osd.4) 383 : cluster 0 4.1 scrub starts 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:17.919564+0000 osd.4 (osd.4) 384 : cluster 0 4.1 scrub ok 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.381071+0000 osd.1 (osd.1) 201 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.382712+0000 osd.1 (osd.1) 202 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.393895+0000 osd.3 (osd.3) 399 : cluster 0 5.14 scrub starts 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.395960+0000 osd.3 (osd.3) 400 : cluster 0 5.14 scrub ok 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.557744+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: audit 2024-03-11T21:40:18.580202+0000 mon.c (mon.2) 1191 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: audit 2024-03-11T21:40:18.580723+0000 mon.a (mon.0) 3439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.627 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.627378+0000 osd.0 (osd.0) 386 : cluster 0 9.b scrub starts 2024-03-11T21:40:19.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.629731+0000 osd.0 (osd.0) 387 : cluster 0 9.b scrub ok 2024-03-11T21:40:19.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.839704+0000 osd.7 (osd.7) 392 : cluster 0 9.13 scrub starts 2024-03-11T21:40:19.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.842510+0000 osd.7 (osd.7) 393 : cluster 0 9.13 scrub ok 2024-03-11T21:40:19.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:19 smithi203 bash[23654]: audit 2024-03-11T21:40:18.942206+0000 mgr.y (mgr.14550) 676 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:19.644 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:19.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:19.776 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:19.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.905 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.917238+0000 osd.4 (osd.4) 383 : cluster 0 4.1 scrub starts 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:17.919564+0000 osd.4 (osd.4) 384 : cluster 0 4.1 scrub ok 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.381071+0000 osd.1 (osd.1) 201 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.382712+0000 osd.1 (osd.1) 202 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.393895+0000 osd.3 (osd.3) 399 : cluster 0 5.14 scrub starts 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.395960+0000 osd.3 (osd.3) 400 : cluster 0 5.14 scrub ok 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.557744+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: audit 2024-03-11T21:40:18.580202+0000 mon.c (mon.2) 1191 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: audit 2024-03-11T21:40:18.580723+0000 mon.a (mon.0) 3439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.627378+0000 osd.0 (osd.0) 386 : cluster 0 9.b scrub starts 2024-03-11T21:40:19.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.629731+0000 osd.0 (osd.0) 387 : cluster 0 9.b scrub ok 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.839704+0000 osd.7 (osd.7) 392 : cluster 0 9.13 scrub starts 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.842510+0000 osd.7 (osd.7) 393 : cluster 0 9.13 scrub ok 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[21512]: audit 2024-03-11T21:40:18.942206+0000 mgr.y (mgr.14550) 676 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.917238+0000 osd.4 (osd.4) 383 : cluster 0 4.1 scrub starts 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:17.919564+0000 osd.4 (osd.4) 384 : cluster 0 4.1 scrub ok 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.381071+0000 osd.1 (osd.1) 201 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.382712+0000 osd.1 (osd.1) 202 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.393895+0000 osd.3 (osd.3) 399 : cluster 0 5.14 scrub starts 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.395960+0000 osd.3 (osd.3) 400 : cluster 0 5.14 scrub ok 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.557744+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: audit 2024-03-11T21:40:18.580202+0000 mon.c (mon.2) 1191 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: audit 2024-03-11T21:40:18.580723+0000 mon.a (mon.0) 3439 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:19.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.627378+0000 osd.0 (osd.0) 386 : cluster 0 9.b scrub starts 2024-03-11T21:40:19.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.629731+0000 osd.0 (osd.0) 387 : cluster 0 9.b scrub ok 2024-03-11T21:40:19.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.839704+0000 osd.7 (osd.7) 392 : cluster 0 9.13 scrub starts 2024-03-11T21:40:19.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.842510+0000 osd.7 (osd.7) 393 : cluster 0 9.13 scrub ok 2024-03-11T21:40:19.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:19 smithi143 bash[29284]: audit 2024-03-11T21:40:18.942206+0000 mgr.y (mgr.14550) 676 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:20.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:20.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:20.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:20.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:20.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:20.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:20.394 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:20.420 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:20.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:20.553 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:20.656 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:20.684 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:20.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.888581+0000 osd.4 (osd.4) 385 : cluster 0 1.5 scrub starts 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:18.890349+0000 osd.4 (osd.4) 386 : cluster 0 1.5 scrub ok 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.426053+0000 osd.3 (osd.3) 401 : cluster 0 39.a scrub starts 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.428195+0000 osd.3 (osd.3) 402 : cluster 0 39.a scrub ok 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.520871+0000 osd.2 (osd.2) 351 : cluster 0 2.10 scrub starts 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.523386+0000 osd.2 (osd.2) 352 : cluster 0 2.10 scrub ok 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.564907+0000 mgr.y (mgr.14550) 677 : cluster 0 pgmap v1116: 246 pgs: 6 unknown, 240 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: audit 2024-03-11T21:40:19.569288+0000 mon.a (mon.0) 3440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.572583+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: audit 2024-03-11T21:40:19.578214+0000 mon.c (mon.2) 1192 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.799 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: audit 2024-03-11T21:40:19.579287+0000 mon.a (mon.0) 3442 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.597233+0000 osd.0 (osd.0) 388 : cluster 0 9.1c scrub starts 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.599101+0000 osd.0 (osd.0) 389 : cluster 0 9.1c scrub ok 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.741845+0000 osd.1 (osd.1) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.742438+0000 osd.1 (osd.1) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.759422+0000 osd.5 (osd.5) 409 : cluster 0 9.a scrub starts 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[29284]: cluster 2024-03-11T21:40:19.765990+0000 osd.5 (osd.5) 410 : cluster 0 9.a scrub ok 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.888581+0000 osd.4 (osd.4) 385 : cluster 0 1.5 scrub starts 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:18.890349+0000 osd.4 (osd.4) 386 : cluster 0 1.5 scrub ok 2024-03-11T21:40:20.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.426053+0000 osd.3 (osd.3) 401 : cluster 0 39.a scrub starts 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.428195+0000 osd.3 (osd.3) 402 : cluster 0 39.a scrub ok 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.520871+0000 osd.2 (osd.2) 351 : cluster 0 2.10 scrub starts 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.523386+0000 osd.2 (osd.2) 352 : cluster 0 2.10 scrub ok 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.564907+0000 mgr.y (mgr.14550) 677 : cluster 0 pgmap v1116: 246 pgs: 6 unknown, 240 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: audit 2024-03-11T21:40:19.569288+0000 mon.a (mon.0) 3440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.572583+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: audit 2024-03-11T21:40:19.578214+0000 mon.c (mon.2) 1192 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: audit 2024-03-11T21:40:19.579287+0000 mon.a (mon.0) 3442 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.597233+0000 osd.0 (osd.0) 388 : cluster 0 9.1c scrub starts 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.599101+0000 osd.0 (osd.0) 389 : cluster 0 9.1c scrub ok 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.741845+0000 osd.1 (osd.1) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.742438+0000 osd.1 (osd.1) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:20.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.759422+0000 osd.5 (osd.5) 409 : cluster 0 9.a scrub starts 2024-03-11T21:40:20.802 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:20 smithi143 bash[21512]: cluster 2024-03-11T21:40:19.765990+0000 osd.5 (osd.5) 410 : cluster 0 9.a scrub ok 2024-03-11T21:40:20.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:20.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.888581+0000 osd.4 (osd.4) 385 : cluster 0 1.5 scrub starts 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:18.890349+0000 osd.4 (osd.4) 386 : cluster 0 1.5 scrub ok 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.426053+0000 osd.3 (osd.3) 401 : cluster 0 39.a scrub starts 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.428195+0000 osd.3 (osd.3) 402 : cluster 0 39.a scrub ok 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.520871+0000 osd.2 (osd.2) 351 : cluster 0 2.10 scrub starts 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.523386+0000 osd.2 (osd.2) 352 : cluster 0 2.10 scrub ok 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.564907+0000 mgr.y (mgr.14550) 677 : cluster 0 pgmap v1116: 246 pgs: 6 unknown, 240 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: audit 2024-03-11T21:40:19.569288+0000 mon.a (mon.0) 3440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.572583+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: audit 2024-03-11T21:40:19.578214+0000 mon.c (mon.2) 1192 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: audit 2024-03-11T21:40:19.579287+0000 mon.a (mon.0) 3442 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:20.928 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.597233+0000 osd.0 (osd.0) 388 : cluster 0 9.1c scrub starts 2024-03-11T21:40:20.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.599101+0000 osd.0 (osd.0) 389 : cluster 0 9.1c scrub ok 2024-03-11T21:40:20.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.741845+0000 osd.1 (osd.1) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:20.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.742438+0000 osd.1 (osd.1) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:20.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.759422+0000 osd.5 (osd.5) 409 : cluster 0 9.a scrub starts 2024-03-11T21:40:20.929 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:20 smithi203 bash[23654]: cluster 2024-03-11T21:40:19.765990+0000 osd.5 (osd.5) 410 : cluster 0 9.a scrub ok 2024-03-11T21:40:20.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:21.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:21.071 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- 172.21.15.143:0/2118095565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4101700 msgr2=0x7febe410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- 172.21.15.143:0/2118095565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4101700 0x7febe410db70 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7febdc00b0a0 tx=0x7febdc00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- 172.21.15.143:0/2118095565 shutdown_connections 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- 172.21.15.143:0/2118095565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4106dc0 0x7febe41071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- 172.21.15.143:0/2118095565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4101700 0x7febe410db70 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- 172.21.15.143:0/2118095565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4100d40 0x7febe41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- 172.21.15.143:0/2118095565 >> 172.21.15.143:0/2118095565 conn(0x7febe4076d80 msgr2=0x7febe4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- 172.21.15.143:0/2118095565 shutdown_connections 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- 172.21.15.143:0/2118095565 wait complete. 2024-03-11T21:40:21.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 librados: starting msgr at 2024-03-11T21:40:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 librados: starting objecter 2024-03-11T21:40:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 Processor -- start 2024-03-11T21:40:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- start start 2024-03-11T21:40:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 librados: setting wanted keys 2024-03-11T21:40:21.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 librados: calling monclient init 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4101700 0x7febe419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 0x7febe41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4110570 con 0x7febe4100d40 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe41103f0 con 0x7febe4101700 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febea356700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe41106f0 con 0x7febe4106dc0 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33202/0 (socket says 172.21.15.143:33202) 2024-03-11T21:40:21.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 0x7febe41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe3fff700 1 -- 172.21.15.143:0/591938247 learned_addr learned my addr 172.21.15.143:0/591938247 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe37fe700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4101700 0x7febe419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 -- 172.21.15.143:0/591938247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4101700 msgr2=0x7febe419dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4101700 0x7febe419dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 -- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 msgr2=0x7febe419d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 -- 172.21.15.143:0/591938247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4198120 con 0x7febe4106dc0 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe3fff700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:40:21.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe88f3700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 0x7febe41979c0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7febdc00bfd0 tx=0x7febdc007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe17fa700 10 client.?.objecter ms_handle_connect 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.107+0000 7febe17fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc021020 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febea356700 1 -- 172.21.15.143:0/591938247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe41983b0 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7febdc004c50 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc005050 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febea356700 10 client.41963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febea356700 1 -- 172.21.15.143:0/591938247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe41028f0 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febea356700 1 librados: init done 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febea356700 1 -- 172.21.15.143:0/591938247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe404faa0 con 0x7febe4106dc0 2024-03-11T21:40:21.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7febdc024020 con 0x7febe4106dc0 2024-03-11T21:40:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 0x7febcc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(721..721 src has 87..721) v4 ==== 10205+0+0 (secure 0 0 0) 0x7febdc00f020 con 0x7febe4106dc0 2024-03-11T21:40:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 10 client.41963.objecter ms_dispatch 0x7febe4001550 osd_map(721..721 src has 87..721) v4 2024-03-11T21:40:21.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 3 client.41963.objecter handle_osd_map got epochs [721,721] > 0 2024-03-11T21:40:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 3 client.41963.objecter handle_osd_map decoding full epoch 721 2024-03-11T21:40:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe3fff700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 0x7febcc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 20 client.41963.objecter dump_active .. 0 homeless 2024-03-11T21:40:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe17fa700 10 client.41963.objecter ms_handle_connect 0x7febcc06f820 2024-03-11T21:40:21.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.111+0000 7febe3fff700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 0x7febcc071ce0 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7febe41073e0 tx=0x7febd8009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:21.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.115+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7febdc011e50 con 0x7febe4106dc0 2024-03-11T21:40:21.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.202 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:21.278 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:40:21.278 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:40:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:40:21.279 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:40:21.279 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:40:21.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.333 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:21.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.355+0000 7febea356700 1 -- 172.21.15.143:0/591938247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7febe4102ac0 con 0x7febe4106dc0 2024-03-11T21:40:21.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.464 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.499+0000 7febe17fa700 1 -- 172.21.15.143:0/591938247 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v722) v1 ==== 113+0+0 (secure 0 0 0) 0x7febdc05c8d0 con 0x7febe4106dc0 2024-03-11T21:40:21.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 10 librados: watch_flush enter 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 10 librados: watch_flush exit 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 20 client.41963.objecter shutdown clearing up homeless session... 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 10 client.41963.objecter successfully canceled tick 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 msgr2=0x7febcc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 0x7febcc071ce0 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7febe41073e0 tx=0x7febd8009260 comp rx=0 tx=0).stop 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 msgr2=0x7febe41979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:21.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 0x7febe41979c0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7febdc00bfd0 tx=0x7febdc007a20 comp rx=0 tx=0).stop 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 shutdown_connections 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7febcc06f820 0x7febcc071ce0 unknown :-1 s=CLOSED pgs=3437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febe4100d40 0x7febe419d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe4106dc0 0x7febe41979c0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 --2- 172.21.15.143:0/591938247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febe4101700 0x7febe419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 >> 172.21.15.143:0/591938247 conn(0x7febe4076d80 msgr2=0x7febe4069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 shutdown_connections 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.503+0000 7febea356700 1 -- 172.21.15.143:0/591938247 wait complete. 2024-03-11T21:40:21.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:21.507+0000 7febea356700 1 librados: shutdown 2024-03-11T21:40:21.488 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:40:21.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:21.588 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:21.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:21.702 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.073685+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.073971+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.393808+0000 osd.1 (osd.1) 205 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.395699+0000 osd.1 (osd.1) 206 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:20.577677+0000 mon.a (mon.0) 3443 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.583086+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:20.584358+0000 mon.c (mon.2) 1193 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:20.585556+0000 mon.a (mon.0) 3445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.803810+0000 osd.5 (osd.5) 411 : cluster 0 2.12 scrub starts 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.810183+0000 osd.5 (osd.5) 412 : cluster 0 2.12 scrub ok 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:21.357529+0000 mon.c (mon.2) 1194 : audit 1 from='client.? 172.21.15.143:0/591938247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:21.357918+0000 mon.a (mon.0) 3446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.703 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.489770+0000 mon.a (mon.0) 3447 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:21.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:21.492419+0000 mon.a (mon.0) 3448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]': finished 2024-03-11T21:40:21.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: audit 2024-03-11T21:40:21.492628+0000 mon.a (mon.0) 3449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:40:21.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:21 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.500264+0000 mon.a (mon.0) 3450 : cluster 0 osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:40:21.717 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:21.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:21.845 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.073685+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.073971+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.393808+0000 osd.1 (osd.1) 205 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.395699+0000 osd.1 (osd.1) 206 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:20.577677+0000 mon.a (mon.0) 3443 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.583086+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:20.584358+0000 mon.c (mon.2) 1193 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:20.585556+0000 mon.a (mon.0) 3445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.803810+0000 osd.5 (osd.5) 411 : cluster 0 2.12 scrub starts 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.810183+0000 osd.5 (osd.5) 412 : cluster 0 2.12 scrub ok 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:21.357529+0000 mon.c (mon.2) 1194 : audit 1 from='client.? 172.21.15.143:0/591938247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.892 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:21.357918+0000 mon.a (mon.0) 3446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.489770+0000 mon.a (mon.0) 3447 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:21.492419+0000 mon.a (mon.0) 3448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]': finished 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: audit 2024-03-11T21:40:21.492628+0000 mon.a (mon.0) 3449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.500264+0000 mon.a (mon.0) 3450 : cluster 0 osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.073685+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.073971+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.393808+0000 osd.1 (osd.1) 205 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.395699+0000 osd.1 (osd.1) 206 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:20.577677+0000 mon.a (mon.0) 3443 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.583086+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:20.584358+0000 mon.c (mon.2) 1193 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.893 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:20.585556+0000 mon.a (mon.0) 3445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]: dispatch 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.803810+0000 osd.5 (osd.5) 411 : cluster 0 2.12 scrub starts 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.810183+0000 osd.5 (osd.5) 412 : cluster 0 2.12 scrub ok 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:21.357529+0000 mon.c (mon.2) 1194 : audit 1 from='client.? 172.21.15.143:0/591938247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:21.357918+0000 mon.a (mon.0) 3446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.489770+0000 mon.a (mon.0) 3447 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:21.492419+0000 mon.a (mon.0) 3448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-118", "mode": "writeback"}]': finished 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: audit 2024-03-11T21:40:21.492628+0000 mon.a (mon.0) 3449 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:40:21.894 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:21 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.500264+0000 mon.a (mon.0) 3450 : cluster 0 osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:40:21.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:22.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:22.099 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:22.197 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:22.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:22.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:22.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:22.353 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:22.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:22.480 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:22.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:22.609 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:22.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.833089+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub starts 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:20.836402+0000 osd.4 (osd.4) 388 : cluster 0 2.f scrub ok 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.381869+0000 osd.1 (osd.1) 207 : cluster 0 39.2 deep-scrub starts 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.384751+0000 osd.1 (osd.1) 208 : cluster 0 39.2 deep-scrub ok 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.549701+0000 osd.0 (osd.0) 392 : cluster 0 3.12 scrub starts 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.551775+0000 osd.0 (osd.0) 393 : cluster 0 3.12 scrub ok 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.565492+0000 mgr.y (mgr.14550) 678 : cluster 0 pgmap v1120: 246 pgs: 246 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.833407+0000 osd.5 (osd.5) 413 : cluster 0 9.f scrub starts 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:21.836971+0000 osd.5 (osd.5) 414 : cluster 0 9.f scrub ok 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: audit 2024-03-11T21:40:22.109394+0000 mon.a (mon.0) 3451 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: audit 2024-03-11T21:40:22.111818+0000 mon.c (mon.2) 1195 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:22.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[21512]: cluster 2024-03-11T21:40:22.518982+0000 mon.a (mon.0) 3452 : cluster 0 osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.833089+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub starts 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:20.836402+0000 osd.4 (osd.4) 388 : cluster 0 2.f scrub ok 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.381869+0000 osd.1 (osd.1) 207 : cluster 0 39.2 deep-scrub starts 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.384751+0000 osd.1 (osd.1) 208 : cluster 0 39.2 deep-scrub ok 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.549701+0000 osd.0 (osd.0) 392 : cluster 0 3.12 scrub starts 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.551775+0000 osd.0 (osd.0) 393 : cluster 0 3.12 scrub ok 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.565492+0000 mgr.y (mgr.14550) 678 : cluster 0 pgmap v1120: 246 pgs: 246 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.833407+0000 osd.5 (osd.5) 413 : cluster 0 9.f scrub starts 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:21.836971+0000 osd.5 (osd.5) 414 : cluster 0 9.f scrub ok 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: audit 2024-03-11T21:40:22.109394+0000 mon.a (mon.0) 3451 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: audit 2024-03-11T21:40:22.111818+0000 mon.c (mon.2) 1195 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:22.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:22 smithi143 bash[29284]: cluster 2024-03-11T21:40:22.518982+0000 mon.a (mon.0) 3452 : cluster 0 osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:40:22.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:22.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:22.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:22.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:22.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:23.004 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.833089+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub starts 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:20.836402+0000 osd.4 (osd.4) 388 : cluster 0 2.f scrub ok 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.381869+0000 osd.1 (osd.1) 207 : cluster 0 39.2 deep-scrub starts 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.384751+0000 osd.1 (osd.1) 208 : cluster 0 39.2 deep-scrub ok 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.549701+0000 osd.0 (osd.0) 392 : cluster 0 3.12 scrub starts 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.551775+0000 osd.0 (osd.0) 393 : cluster 0 3.12 scrub ok 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.565492+0000 mgr.y (mgr.14550) 678 : cluster 0 pgmap v1120: 246 pgs: 246 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.833407+0000 osd.5 (osd.5) 413 : cluster 0 9.f scrub starts 2024-03-11T21:40:23.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:21.836971+0000 osd.5 (osd.5) 414 : cluster 0 9.f scrub ok 2024-03-11T21:40:23.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: audit 2024-03-11T21:40:22.109394+0000 mon.a (mon.0) 3451 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:23.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: audit 2024-03-11T21:40:22.111818+0000 mon.c (mon.2) 1195 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:23.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:22 smithi203 bash[23654]: cluster 2024-03-11T21:40:22.518982+0000 mon.a (mon.0) 3452 : cluster 0 osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:40:23.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:23.135 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:23.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:23.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.398 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:23.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:23.528 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:23.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:23.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: audit 2024-03-11T21:40:22.583629+0000 mon.c (mon.2) 1196 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: audit 2024-03-11T21:40:22.592185+0000 mon.a (mon.0) 3453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: cluster 2024-03-11T21:40:22.686446+0000 osd.6 (osd.6) 319 : cluster 0 2.17 scrub starts 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: cluster 2024-03-11T21:40:22.688021+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub ok 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: cluster 2024-03-11T21:40:22.762801+0000 osd.7 (osd.7) 394 : cluster 0 5.4 scrub starts 2024-03-11T21:40:23.773 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:23 smithi203 bash[23654]: cluster 2024-03-11T21:40:22.764234+0000 osd.7 (osd.7) 395 : cluster 0 5.4 scrub ok 2024-03-11T21:40:23.788 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:23.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: audit 2024-03-11T21:40:22.583629+0000 mon.c (mon.2) 1196 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: audit 2024-03-11T21:40:22.592185+0000 mon.a (mon.0) 3453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: cluster 2024-03-11T21:40:22.686446+0000 osd.6 (osd.6) 319 : cluster 0 2.17 scrub starts 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: cluster 2024-03-11T21:40:22.688021+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub ok 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: cluster 2024-03-11T21:40:22.762801+0000 osd.7 (osd.7) 394 : cluster 0 5.4 scrub starts 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[21512]: cluster 2024-03-11T21:40:22.764234+0000 osd.7 (osd.7) 395 : cluster 0 5.4 scrub ok 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: audit 2024-03-11T21:40:22.583629+0000 mon.c (mon.2) 1196 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: audit 2024-03-11T21:40:22.592185+0000 mon.a (mon.0) 3453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:23.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: cluster 2024-03-11T21:40:22.686446+0000 osd.6 (osd.6) 319 : cluster 0 2.17 scrub starts 2024-03-11T21:40:23.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: cluster 2024-03-11T21:40:22.688021+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub ok 2024-03-11T21:40:23.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: cluster 2024-03-11T21:40:22.762801+0000 osd.7 (osd.7) 394 : cluster 0 5.4 scrub starts 2024-03-11T21:40:23.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:23 smithi143 bash[29284]: cluster 2024-03-11T21:40:22.764234+0000 osd.7 (osd.7) 395 : cluster 0 5.4 scrub ok 2024-03-11T21:40:23.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:24.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:24.049 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:24.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:24.179 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:24.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.310 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:24.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:24.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:24.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.569 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:24.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:24.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:24.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:24.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.567717+0000 mgr.y (mgr.14550) 679 : cluster 0 pgmap v1122: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.585502+0000 osd.0 (osd.0) 394 : cluster 0 6.16 scrub starts 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.588103+0000 osd.0 (osd.0) 395 : cluster 0 6.16 scrub ok 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: audit 2024-03-11T21:40:23.598461+0000 mon.a (mon.0) 3454 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.606148+0000 mon.a (mon.0) 3455 : cluster 0 osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: audit 2024-03-11T21:40:23.608185+0000 mon.c (mon.2) 1197 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: audit 2024-03-11T21:40:23.610204+0000 mon.a (mon.0) 3456 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.746651+0000 osd.7 (osd.7) 396 : cluster 0 6.3 scrub starts 2024-03-11T21:40:24.831 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:24 smithi203 bash[23654]: cluster 2024-03-11T21:40:23.749763+0000 osd.7 (osd.7) 397 : cluster 0 6.3 scrub ok 2024-03-11T21:40:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:24.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.567717+0000 mgr.y (mgr.14550) 679 : cluster 0 pgmap v1122: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:24.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.585502+0000 osd.0 (osd.0) 394 : cluster 0 6.16 scrub starts 2024-03-11T21:40:24.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.588103+0000 osd.0 (osd.0) 395 : cluster 0 6.16 scrub ok 2024-03-11T21:40:24.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: audit 2024-03-11T21:40:23.598461+0000 mon.a (mon.0) 3454 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.606148+0000 mon.a (mon.0) 3455 : cluster 0 osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: audit 2024-03-11T21:40:23.608185+0000 mon.c (mon.2) 1197 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: audit 2024-03-11T21:40:23.610204+0000 mon.a (mon.0) 3456 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.746651+0000 osd.7 (osd.7) 396 : cluster 0 6.3 scrub starts 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[29284]: cluster 2024-03-11T21:40:23.749763+0000 osd.7 (osd.7) 397 : cluster 0 6.3 scrub ok 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.567717+0000 mgr.y (mgr.14550) 679 : cluster 0 pgmap v1122: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.585502+0000 osd.0 (osd.0) 394 : cluster 0 6.16 scrub starts 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.588103+0000 osd.0 (osd.0) 395 : cluster 0 6.16 scrub ok 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: audit 2024-03-11T21:40:23.598461+0000 mon.a (mon.0) 3454 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.606148+0000 mon.a (mon.0) 3455 : cluster 0 osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:40:24.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: audit 2024-03-11T21:40:23.608185+0000 mon.c (mon.2) 1197 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: audit 2024-03-11T21:40:23.610204+0000 mon.a (mon.0) 3456 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]: dispatch 2024-03-11T21:40:24.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.746651+0000 osd.7 (osd.7) 396 : cluster 0 6.3 scrub starts 2024-03-11T21:40:24.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:24 smithi143 bash[21512]: cluster 2024-03-11T21:40:23.749763+0000 osd.7 (osd.7) 397 : cluster 0 6.3 scrub ok 2024-03-11T21:40:24.959 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:25.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:25.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:25.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:40:25.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:40:25.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:25.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:25.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:25.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:25.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.494 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:25.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:25.610 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.345430+0000 osd.1 (osd.1) 209 : cluster 0 2.d scrub starts 2024-03-11T21:40:25.610 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.347028+0000 osd.1 (osd.1) 210 : cluster 0 2.d scrub ok 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.475777+0000 osd.3 (osd.3) 403 : cluster 0 6.b scrub starts 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.477241+0000 osd.3 (osd.3) 404 : cluster 0 6.b scrub ok 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.599037+0000 mon.a (mon.0) 3457 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: audit 2024-03-11T21:40:24.607754+0000 mon.a (mon.0) 3458 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.611268+0000 mon.a (mon.0) 3459 : cluster 0 osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.687233+0000 osd.6 (osd.6) 321 : cluster 0 2.7 scrub starts 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.690017+0000 osd.6 (osd.6) 322 : cluster 0 2.7 scrub ok 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.840087+0000 osd.5 (osd.5) 415 : cluster 0 2.18 scrub starts 2024-03-11T21:40:25.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:25 smithi203 bash[23654]: cluster 2024-03-11T21:40:24.841545+0000 osd.5 (osd.5) 416 : cluster 0 2.18 scrub ok 2024-03-11T21:40:25.632 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:25.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.345430+0000 osd.1 (osd.1) 209 : cluster 0 2.d scrub starts 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.347028+0000 osd.1 (osd.1) 210 : cluster 0 2.d scrub ok 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.475777+0000 osd.3 (osd.3) 403 : cluster 0 6.b scrub starts 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.477241+0000 osd.3 (osd.3) 404 : cluster 0 6.b scrub ok 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.599037+0000 mon.a (mon.0) 3457 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: audit 2024-03-11T21:40:24.607754+0000 mon.a (mon.0) 3458 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.611268+0000 mon.a (mon.0) 3459 : cluster 0 osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.687233+0000 osd.6 (osd.6) 321 : cluster 0 2.7 scrub starts 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.690017+0000 osd.6 (osd.6) 322 : cluster 0 2.7 scrub ok 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.840087+0000 osd.5 (osd.5) 415 : cluster 0 2.18 scrub starts 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[21512]: cluster 2024-03-11T21:40:24.841545+0000 osd.5 (osd.5) 416 : cluster 0 2.18 scrub ok 2024-03-11T21:40:25.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.345430+0000 osd.1 (osd.1) 209 : cluster 0 2.d scrub starts 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.347028+0000 osd.1 (osd.1) 210 : cluster 0 2.d scrub ok 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.475777+0000 osd.3 (osd.3) 403 : cluster 0 6.b scrub starts 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.477241+0000 osd.3 (osd.3) 404 : cluster 0 6.b scrub ok 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.599037+0000 mon.a (mon.0) 3457 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: audit 2024-03-11T21:40:24.607754+0000 mon.a (mon.0) 3458 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-118"}]': finished 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.611268+0000 mon.a (mon.0) 3459 : cluster 0 osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.687233+0000 osd.6 (osd.6) 321 : cluster 0 2.7 scrub starts 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.690017+0000 osd.6 (osd.6) 322 : cluster 0 2.7 scrub ok 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.840087+0000 osd.5 (osd.5) 415 : cluster 0 2.18 scrub starts 2024-03-11T21:40:25.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:25 smithi143 bash[29284]: cluster 2024-03-11T21:40:24.841545+0000 osd.5 (osd.5) 416 : cluster 0 2.18 scrub ok 2024-03-11T21:40:25.756 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:25.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:25.881 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:25.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:26.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:26.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:26.137 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:26.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:26.268 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:26.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:26.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.529 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:26.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:26.660 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:26.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.466599+0000 osd.3 (osd.3) 405 : cluster 0 39.1b scrub starts 2024-03-11T21:40:26.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.470223+0000 osd.3 (osd.3) 406 : cluster 0 39.1b scrub ok 2024-03-11T21:40:26.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.568664+0000 mgr.y (mgr.14550) 680 : cluster 0 pgmap v1125: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:26.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.588085+0000 osd.0 (osd.0) 396 : cluster 0 6.18 scrub starts 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.590242+0000 osd.0 (osd.0) 397 : cluster 0 6.18 scrub ok 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.608777+0000 mon.a (mon.0) 3460 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.616767+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.709134+0000 osd.6 (osd.6) 323 : cluster 0 2.1a scrub starts 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.710324+0000 osd.6 (osd.6) 324 : cluster 0 2.1a scrub ok 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.890003+0000 osd.5 (osd.5) 417 : cluster 0 3.16 scrub starts 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:25.893442+0000 osd.5 (osd.5) 418 : cluster 0 3.16 scrub ok 2024-03-11T21:40:26.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.510839+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:40:26.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: audit 2024-03-11T21:40:26.527898+0000 mon.c (mon.2) 1198 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[29284]: audit 2024-03-11T21:40:26.533093+0000 mon.a (mon.0) 3463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.466599+0000 osd.3 (osd.3) 405 : cluster 0 39.1b scrub starts 2024-03-11T21:40:26.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.470223+0000 osd.3 (osd.3) 406 : cluster 0 39.1b scrub ok 2024-03-11T21:40:26.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.568664+0000 mgr.y (mgr.14550) 680 : cluster 0 pgmap v1125: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.588085+0000 osd.0 (osd.0) 396 : cluster 0 6.18 scrub starts 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.590242+0000 osd.0 (osd.0) 397 : cluster 0 6.18 scrub ok 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.608777+0000 mon.a (mon.0) 3460 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.616767+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.709134+0000 osd.6 (osd.6) 323 : cluster 0 2.1a scrub starts 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.710324+0000 osd.6 (osd.6) 324 : cluster 0 2.1a scrub ok 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.890003+0000 osd.5 (osd.5) 417 : cluster 0 3.16 scrub starts 2024-03-11T21:40:26.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:25.893442+0000 osd.5 (osd.5) 418 : cluster 0 3.16 scrub ok 2024-03-11T21:40:26.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.510839+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:40:26.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: audit 2024-03-11T21:40:26.527898+0000 mon.c (mon.2) 1198 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.696 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:26 smithi143 bash[21512]: audit 2024-03-11T21:40:26.533093+0000 mon.a (mon.0) 3463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.764 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.791 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:26.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.466599+0000 osd.3 (osd.3) 405 : cluster 0 39.1b scrub starts 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.470223+0000 osd.3 (osd.3) 406 : cluster 0 39.1b scrub ok 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.568664+0000 mgr.y (mgr.14550) 680 : cluster 0 pgmap v1125: 246 pgs: 246 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.588085+0000 osd.0 (osd.0) 396 : cluster 0 6.18 scrub starts 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.590242+0000 osd.0 (osd.0) 397 : cluster 0 6.18 scrub ok 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.608777+0000 mon.a (mon.0) 3460 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.616767+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.709134+0000 osd.6 (osd.6) 323 : cluster 0 2.1a scrub starts 2024-03-11T21:40:26.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.710324+0000 osd.6 (osd.6) 324 : cluster 0 2.1a scrub ok 2024-03-11T21:40:26.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.890003+0000 osd.5 (osd.5) 417 : cluster 0 3.16 scrub starts 2024-03-11T21:40:26.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:25.893442+0000 osd.5 (osd.5) 418 : cluster 0 3.16 scrub ok 2024-03-11T21:40:26.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.510839+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:40:26.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: audit 2024-03-11T21:40:26.527898+0000 mon.c (mon.2) 1198 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:26 smithi203 bash[23654]: audit 2024-03-11T21:40:26.533093+0000 mon.a (mon.0) 3463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:26.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:27.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:27.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:27.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:27.181 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:27.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:27.306 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:27.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:27.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:27.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/1579896010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 msgr2=0x7f84801022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- 172.21.15.143:0/1579896010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84801022e0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f8470009990 tx=0x7f8470009ca0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/1579896010 shutdown_connections 2024-03-11T21:40:27.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- 172.21.15.143:0/1579896010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480102820 0x7f848010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- 172.21.15.143:0/1579896010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480107ee0 0x7f84801082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- 172.21.15.143:0/1579896010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84801022e0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/1579896010 >> 172.21.15.143:0/1579896010 conn(0x7f8480078f90 msgr2=0x7f84800793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/1579896010 shutdown_connections 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/1579896010 wait complete. 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 librados: starting msgr at 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 librados: starting objecter 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 Processor -- start 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- start start 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 librados: setting wanted keys 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 librados: calling monclient init 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 0x7f8480074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 0x7f8480074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84801115f0 con 0x7f8480101e60 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8480111470 con 0x7f8480107ee0 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8480111770 con 0x7f8480102820 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56914/0 (socket says 172.21.15.143:56914) 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 0x7f8480074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847ffff700 1 -- 172.21.15.143:0/3014812511 learned_addr learned my addr 172.21.15.143:0/3014812511 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 -- 172.21.15.143:0/3014812511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 msgr2=0x7f8480074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f8484c6c700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 0x7f8480074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 0x7f8480074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 -- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 msgr2=0x7f84800740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 -- 172.21.15.143:0/3014812511 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84801a60b0 con 0x7f8480102820 2024-03-11T21:40:27.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847ffff700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:40:27.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f8484c6c700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 0x7f8480074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:40:27.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847f7fe700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 0x7f8480074620 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f8470006010 tx=0x7f84700057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847d7fa700 10 client.?.objecter ms_handle_connect 0x7f8480102820 2024-03-11T21:40:27.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:27.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470027070 con 0x7f8480102820 2024-03-11T21:40:27.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84801a6340 con 0x7f8480102820 2024-03-11T21:40:27.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8470022470 con 0x7f8480102820 2024-03-11T21:40:27.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470017650 con 0x7f8480102820 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 10 client.41969.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84801a67f0 con 0x7f8480102820 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 librados: init done 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.487+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848004faa0 con 0x7f8480102820 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f847002a070 con 0x7f8480102820 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 0x7f8468071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(727..727 src has 87..727) v4 ==== 10225+0+0 (secure 0 0 0) 0x7f847008f9c0 con 0x7f8480102820 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 10 client.41969.objecter ms_dispatch 0x7f8480001550 osd_map(727..727 src has 87..727) v4 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 3 client.41969.objecter handle_osd_map got epochs [727,727] > 0 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 3 client.41969.objecter handle_osd_map decoding full epoch 727 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847ffff700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 0x7f8468071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 20 client.41969.objecter dump_active .. 0 homeless 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847ffff700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 0x7f8468071ce0 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f8474009680 tx=0x7f8474009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.491+0000 7f847d7fa700 10 client.41969.objecter ms_handle_connect 0x7f846806f820 2024-03-11T21:40:27.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.495+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8470091e20 con 0x7f8480102820 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- 172.21.15.143:0/3484664779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc06a700 msgr2=0x7ff9bc06aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- 172.21.15.143:0/3484664779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc06a700 0x7ff9bc06aae0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40099c0 tx=0x7ff9b4009cd0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- 172.21.15.143:0/3484664779 shutdown_connections 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- 172.21.15.143:0/3484664779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc06a700 0x7ff9bc06aae0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- 172.21.15.143:0/3484664779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc10ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- 172.21.15.143:0/3484664779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9bc06b0b0 0x7ff9bc106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- 172.21.15.143:0/3484664779 >> 172.21.15.143:0/3484664779 conn(0x7ff9bc077a40 msgr2=0x7ff9bc077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- 172.21.15.143:0/3484664779 shutdown_connections 2024-03-11T21:40:27.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- 172.21.15.143:0/3484664779 wait complete. 2024-03-11T21:40:27.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 librados: starting msgr at 2024-03-11T21:40:27.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 librados: starting objecter 2024-03-11T21:40:27.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 Processor -- start 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- start start 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 librados: setting wanted keys 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 librados: calling monclient init 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9bc06a700 0x7ff9bc19d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9bc06b0b0 0x7ff9bc19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc111630 con 0x7ff9bc1071b0 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc1114b0 con 0x7ff9bc06a700 2024-03-11T21:40:27.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c2491700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc1117b0 con 0x7ff9bc06b0b0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c0a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c0a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56934/0 (socket says 172.21.15.143:56934) 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 learned_addr learned my addr 172.21.15.143:0/119309831 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9bc06b0b0 0x7ff9bc19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.555+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9bc06a700 0x7ff9bc19d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9bc06b0b0 msgr2=0x7ff9bc19ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9bc06b0b0 0x7ff9bc19ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9bc06a700 msgr2=0x7ff9bc19d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9bc06a700 0x7ff9bc19d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc198190 con 0x7ff9bc1071b0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc197a30 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00ada0 tx=0x7ff9ac00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 10 client.?.objecter ms_handle_connect 0x7ff9bc1071b0 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:27.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac017020 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9bc198480 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff9ac00cd50 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac005280 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 10 client.30771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9bc101000 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 librados: init done 2024-03-11T21:40:27.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff9bc04faa0 con 0x7ff9bc1071b0 2024-03-11T21:40:27.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff9ac00c8d0 con 0x7ff9bc1071b0 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff9a806f820 0x7ff9a8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(728..728 src has 87..728) v4 ==== 10210+0+0 (secure 0 0 0) 0x7ff9ac08edd0 con 0x7ff9bc1071b0 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 osd_map(728..728 src has 87..728) v4 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 3 client.30771.objecter handle_osd_map got epochs [728,728] > 0 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 3 client.30771.objecter handle_osd_map decoding full epoch 728 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff9a806f820 0x7ff9a8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 20 client.30771.objecter dump_active .. 0 homeless 2024-03-11T21:40:27.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v728) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff9ac021020 con 0x7ff9bc1071b0 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 10 client.30771.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 1 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff9bc0629f0 0x7ff9bc062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 1 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 0 1 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 4 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 10 client.30771.objecter _send_command 1 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1024a0 con 0x7ff9bc0629f0 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff9bc0629f0 0x7ff9bc062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9a806f820 2024-03-11T21:40:27.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff9a806f820 0x7ff9a8071ce0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4000c00 tx=0x7ff9b4019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc0629f0 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.559+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff9bc0629f0 0x7ff9bc062dd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.563+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc1024a0 con 0x7ff9bc0629f0 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.563+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.563+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 1 = system:0 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.563+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 4 2024-03-11T21:40:27.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.563+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 0 1 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 10 client.30771.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 2 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 2 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc198f60 osd=0 3 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 0 2 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 4 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 10 client.30771.objecter _send_command 2 2024-03-11T21:40:27.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.579+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc102ee0 con 0x7ff9bc0629f0 2024-03-11T21:40:27.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc102ee0 con 0x7ff9bc0629f0 2024-03-11T21:40:27.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 4 2024-03-11T21:40:27.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 0 2 2024-03-11T21:40:27.563 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:40:27.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 10 client.30771.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff9bc101830 0x7ff9bc101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 1 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 4 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 10 client.30771.objecter _send_command 3 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc062830 con 0x7ff9bc101830 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff9bc101830 0x7ff9bc101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc101830 2024-03-11T21:40:27.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff9bc101830 0x7ff9bc101c50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9b400bad0 con 0x7ff9bc101830 2024-03-11T21:40:27.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:40:27.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 3 = system:0 2024-03-11T21:40:27.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 4 2024-03-11T21:40:27.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.583+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 1 3 2024-03-11T21:40:27.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:27.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 10 client.30771.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 4 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.599+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 4 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc0626c0 osd=1 3 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 1 4 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 4 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 10 client.30771.objecter _send_command 4 2024-03-11T21:40:27.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc02c6b0 con 0x7ff9bc101830 2024-03-11T21:40:27.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc02c6b0 con 0x7ff9bc101830 2024-03-11T21:40:27.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 4 2024-03-11T21:40:27.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 1 4 2024-03-11T21:40:27.586 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:40:27.586 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 10 client.30771.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 5 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff9bc1093b0 0x7ff9bc1097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 5 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 2 5 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 4 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 10 client.30771.objecter _send_command 5 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc02dbb0 con 0x7ff9bc1093b0 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff9bc1093b0 0x7ff9bc1097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc1093b0 2024-03-11T21:40:27.587 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.603+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff9bc1093b0 0x7ff9bc1097d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.607+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc02dbb0 con 0x7ff9bc1093b0 2024-03-11T21:40:27.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.607+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:40:27.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.607+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 5 = system:0 2024-03-11T21:40:27.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.607+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 4 2024-03-11T21:40:27.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.607+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 2 5 2024-03-11T21:40:27.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 10 client.30771.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 6 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 6 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc02da40 osd=2 3 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 2 6 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 4 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 10 client.30771.objecter _send_command 6 2024-03-11T21:40:27.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.623+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc108e30 con 0x7ff9bc1093b0 2024-03-11T21:40:27.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc108e30 con 0x7ff9bc1093b0 2024-03-11T21:40:27.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 4 2024-03-11T21:40:27.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 2 6 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 10 client.30771.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 7 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff9bc02d540 0x7ff9bc1ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 7 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 3 7 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 4 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 10 client.30771.objecter _send_command 7 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1091f0 con 0x7ff9bc02d540 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff9bc02d540 0x7ff9bc1ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc02d540 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff9bc02d540 0x7ff9bc1ad900 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9ac05b850 con 0x7ff9bc02d540 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 7 = system:0 2024-03-11T21:40:27.609 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 4 2024-03-11T21:40:27.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.627+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 3 7 2024-03-11T21:40:27.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 10 client.30771.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.626 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 8 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 8 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc109080 osd=3 3 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 3 8 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 4 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 10 client.30771.objecter _send_command 8 2024-03-11T21:40:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.643+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc02cab0 con 0x7ff9bc02d540 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc02cab0 con 0x7ff9bc02d540 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 4 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 3 8 2024-03-11T21:40:27.628 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 10 client.30771.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 9 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff9bc02cc20 0x7ff9bc1b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 9 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 4 9 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 4 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 10 client.30771.objecter _send_command 9 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1ab640 con 0x7ff9bc02cc20 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff9bc02cc20 0x7ff9bc1b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc02cc20 2024-03-11T21:40:27.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff9bc02cc20 0x7ff9bc1b0fd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc1ab640 con 0x7ff9bc02cc20 2024-03-11T21:40:27.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:40:27.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 9 = system:0 2024-03-11T21:40:27.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 4 2024-03-11T21:40:27.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.647+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 4 9 2024-03-11T21:40:27.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 10 client.30771.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 10 2024-03-11T21:40:27.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 10 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc02cab0 osd=4 3 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 4 10 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 4 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 10 client.30771.objecter _send_command 10 2024-03-11T21:40:27.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.667+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc1ab6a0 con 0x7ff9bc02cc20 2024-03-11T21:40:27.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc1ab6a0 con 0x7ff9bc02cc20 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 4 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 4 10 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:40:27.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 10 client.30771.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 11 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff9bc1b1a80 0x7ff9bc1b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 11 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 5 11 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 4 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 10 client.30771.objecter _send_command 11 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1b7070 con 0x7ff9bc1b1a80 2024-03-11T21:40:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff9bc1b1a80 0x7ff9bc1b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc1b1a80 2024-03-11T21:40:27.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9bb7fe700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff9bc1b1a80 0x7ff9bc1b1ec0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc1b7070 con 0x7ff9bc1b1a80 2024-03-11T21:40:27.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:40:27.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 11 = system:0 2024-03-11T21:40:27.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 4 2024-03-11T21:40:27.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.671+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 5 11 2024-03-11T21:40:27.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 10 client.30771.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 12 2024-03-11T21:40:27.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 12 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1ab6a0 osd=5 3 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 5 12 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 4 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 10 client.30771.objecter _send_command 12 2024-03-11T21:40:27.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc1b2510 con 0x7ff9bc1b1a80 2024-03-11T21:40:27.672 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc1b2510 con 0x7ff9bc1b1a80 2024-03-11T21:40:27.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 4 2024-03-11T21:40:27.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 5 12 2024-03-11T21:40:27.674 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 10 client.30771.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 13 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff9bc1b2c30 0x7ff9bc1b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 13 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 6 13 2024-03-11T21:40:27.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 4 2024-03-11T21:40:27.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 10 client.30771.objecter _send_command 13 2024-03-11T21:40:27.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1b3720 con 0x7ff9bc1b2c30 2024-03-11T21:40:27.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.691+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff9bc1b2c30 0x7ff9bc1b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc1b2c30 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9c0a2e700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff9bc1b2c30 0x7ff9bc1b3070 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc1b3720 con 0x7ff9bc1b2c30 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 13 = system:0 2024-03-11T21:40:27.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 4 2024-03-11T21:40:27.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.695+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 6 13 2024-03-11T21:40:27.695 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 10 client.30771.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 14 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 14 2024-03-11T21:40:27.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1b2510 osd=6 3 2024-03-11T21:40:27.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 6 14 2024-03-11T21:40:27.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 4 2024-03-11T21:40:27.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 10 client.30771.objecter _send_command 14 2024-03-11T21:40:27.701 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.719+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc1b3720 con 0x7ff9bc1b2c30 2024-03-11T21:40:27.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc1b3720 con 0x7ff9bc1b2c30 2024-03-11T21:40:27.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 4 2024-03-11T21:40:27.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 6 14 2024-03-11T21:40:27.704 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:40:27.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 10 client.30771.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 15 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff9bc1b3890 0x7ff9bc1ba420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 15 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 7 15 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 4 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 10 client.30771.objecter _send_command 15 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9bc1bd4a0 con 0x7ff9bc1b3890 2024-03-11T21:40:27.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff9bc1b3890 0x7ff9bc1ba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:27.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 10 client.30771.objecter ms_handle_connect 0x7ff9bc1b3890 2024-03-11T21:40:27.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9bbfff700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff9bc1b3890 0x7ff9bc1ba420 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:27.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9bc1bd4a0 con 0x7ff9bc1b3890 2024-03-11T21:40:27.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.723+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:40:27.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.727+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 15 = system:0 2024-03-11T21:40:27.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.727+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 4 2024-03-11T21:40:27.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.727+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 7 15 2024-03-11T21:40:27.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 10 client.30771.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:40:27.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign -1 16 2024-03-11T21:40:27.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 20 client.30771.objecter _get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_remove -1 16 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 20 client.30771.objecter get_session s=0x7ff9bc1b3720 osd=7 3 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 15 client.30771.objecter _session_command_op_assign 7 16 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 4 2024-03-11T21:40:27.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 10 client.30771.objecter _send_command 16 2024-03-11T21:40:27.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.743+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9bc1bc250 con 0x7ff9bc1b3890 2024-03-11T21:40:27.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9b97fa700 1 -- 172.21.15.143:0/119309831 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff9bc1bc250 con 0x7ff9bc1b3890 2024-03-11T21:40:27.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9b97fa700 10 client.30771.objecter ms_dispatch 0x7ff9bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:27.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9b97fa700 10 client.30771.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:27.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9b97fa700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 4 2024-03-11T21:40:27.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9b97fa700 15 client.30771.objecter _session_command_op_remove 7 16 2024-03-11T21:40:27.729 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:40:27.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 librados: watch_flush enter 2024-03-11T21:40:27.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 librados: watch_flush exit 2024-03-11T21:40:27.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.0 2024-03-11T21:40:27.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff9bc0629f0 msgr2=0x7ff9bc062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff9bc0629f0 0x7ff9bc062dd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc198f60 osd=0 1 2024-03-11T21:40:27.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.1 2024-03-11T21:40:27.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff9bc101830 msgr2=0x7ff9bc101c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff9bc101830 0x7ff9bc101c50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc0626c0 osd=1 1 2024-03-11T21:40:27.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.2 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff9bc1093b0 msgr2=0x7ff9bc1097d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff9bc1093b0 0x7ff9bc1097d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02da40 osd=2 1 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.3 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff9bc02d540 msgr2=0x7ff9bc1ad900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff9bc02d540 0x7ff9bc1ad900 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc109080 osd=3 1 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.4 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff9bc02cc20 msgr2=0x7ff9bc1b0fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff9bc02cc20 0x7ff9bc1b0fd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc02cab0 osd=4 1 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.5 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff9bc1b1a80 msgr2=0x7ff9bc1b1ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 reap_dead start 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff9bc1b1a80 0x7ff9bc1b1ec0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 reap_dead start 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 reap_dead start 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1ab6a0 osd=5 1 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.6 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff9bc1b2c30 msgr2=0x7ff9bc1b3070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff9bc1b2c30 0x7ff9bc1b3070 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b2510 osd=6 1 2024-03-11T21:40:27.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter close_session for osd.7 2024-03-11T21:40:27.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff9bc1b3890 msgr2=0x7ff9bc1ba420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff9bc1b3890 0x7ff9bc1ba420 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:27.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter put_session s=0x7ff9bc1b3720 osd=7 1 2024-03-11T21:40:27.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 20 client.30771.objecter shutdown clearing up homeless session... 2024-03-11T21:40:27.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 10 client.30771.objecter successfully canceled tick 2024-03-11T21:40:27.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff9a806f820 msgr2=0x7ff9a8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff9a806f820 0x7ff9a8071ce0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4000c00 tx=0x7ff9b4019040 comp rx=0 tx=0).stop 2024-03-11T21:40:27.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 msgr2=0x7ff9bc197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:27.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 --2- 172.21.15.143:0/119309831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc1071b0 0x7ff9bc197a30 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00ada0 tx=0x7ff9ac00c330 comp rx=0 tx=0).stop 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c0a2e700 1 -- 172.21.15.143:0/119309831 reap_dead start 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 shutdown_connections 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 >> 172.21.15.143:0/119309831 conn(0x7ff9bc077a40 msgr2=0x7ff9bc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 shutdown_connections 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 -- 172.21.15.143:0/119309831 wait complete. 2024-03-11T21:40:27.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.747+0000 7ff9c2491700 1 librados: shutdown 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:27.740 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:27.741 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:27.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:27.755+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f84801a6b10 con 0x7f8480102820 2024-03-11T21:40:27.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.553137+0000 osd.0 (osd.0) 398 : cluster 0 39.f scrub starts 2024-03-11T21:40:27.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.557123+0000 osd.0 (osd.0) 399 : cluster 0 39.f scrub ok 2024-03-11T21:40:27.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.565468+0000 osd.2 (osd.2) 353 : cluster 0 5.b scrub starts 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.566854+0000 osd.2 (osd.2) 354 : cluster 0 5.b scrub ok 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.720531+0000 osd.6 (osd.6) 325 : cluster 0 9.18 scrub starts 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.722117+0000 osd.6 (osd.6) 326 : cluster 0 9.18 scrub ok 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.726440+0000 osd.7 (osd.7) 398 : cluster 0 3.13 scrub starts 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.728506+0000 osd.7 (osd.7) 399 : cluster 0 3.13 scrub ok 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: audit 2024-03-11T21:40:27.498612+0000 mon.a (mon.0) 3464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:27.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.503320+0000 mon.a (mon.0) 3465 : cluster 0 osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:40:27.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: audit 2024-03-11T21:40:27.536123+0000 mon.c (mon.2) 1199 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: audit 2024-03-11T21:40:27.536497+0000 mon.a (mon.0) 3466 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:27 smithi203 bash[23654]: audit 2024-03-11T21:40:27.561748+0000 mon.a (mon.0) 3467 : audit 0 from='client.? 172.21.15.143:0/119309831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:27.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:27.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:27.932 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.553137+0000 osd.0 (osd.0) 398 : cluster 0 39.f scrub starts 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.557123+0000 osd.0 (osd.0) 399 : cluster 0 39.f scrub ok 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.565468+0000 osd.2 (osd.2) 353 : cluster 0 5.b scrub starts 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.566854+0000 osd.2 (osd.2) 354 : cluster 0 5.b scrub ok 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.720531+0000 osd.6 (osd.6) 325 : cluster 0 9.18 scrub starts 2024-03-11T21:40:27.941 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.722117+0000 osd.6 (osd.6) 326 : cluster 0 9.18 scrub ok 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.726440+0000 osd.7 (osd.7) 398 : cluster 0 3.13 scrub starts 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.728506+0000 osd.7 (osd.7) 399 : cluster 0 3.13 scrub ok 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: audit 2024-03-11T21:40:27.498612+0000 mon.a (mon.0) 3464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.503320+0000 mon.a (mon.0) 3465 : cluster 0 osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: audit 2024-03-11T21:40:27.536123+0000 mon.c (mon.2) 1199 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: audit 2024-03-11T21:40:27.536497+0000 mon.a (mon.0) 3466 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[21512]: audit 2024-03-11T21:40:27.561748+0000 mon.a (mon.0) 3467 : audit 0 from='client.? 172.21.15.143:0/119309831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.553137+0000 osd.0 (osd.0) 398 : cluster 0 39.f scrub starts 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.557123+0000 osd.0 (osd.0) 399 : cluster 0 39.f scrub ok 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.565468+0000 osd.2 (osd.2) 353 : cluster 0 5.b scrub starts 2024-03-11T21:40:27.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.566854+0000 osd.2 (osd.2) 354 : cluster 0 5.b scrub ok 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.720531+0000 osd.6 (osd.6) 325 : cluster 0 9.18 scrub starts 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.722117+0000 osd.6 (osd.6) 326 : cluster 0 9.18 scrub ok 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.726440+0000 osd.7 (osd.7) 398 : cluster 0 3.13 scrub starts 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.728506+0000 osd.7 (osd.7) 399 : cluster 0 3.13 scrub ok 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: audit 2024-03-11T21:40:27.498612+0000 mon.a (mon.0) 3464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.503320+0000 mon.a (mon.0) 3465 : cluster 0 osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: audit 2024-03-11T21:40:27.536123+0000 mon.c (mon.2) 1199 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: audit 2024-03-11T21:40:27.536497+0000 mon.a (mon.0) 3466 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:27.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:27 smithi143 bash[29284]: audit 2024-03-11T21:40:27.561748+0000 mon.a (mon.0) 3467 : audit 0 from='client.? 172.21.15.143:0/119309831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:27.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:28.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:28.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:28.213 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:28.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:28.342 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:28.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.474 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:28.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.519+0000 7f847d7fa700 1 -- 172.21.15.143:0/3014812511 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v729) v1 ==== 87+0+0 (secure 0 0 0) 0x7f8470013070 con 0x7f8480102820 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 10 librados: watch_flush enter 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 10 librados: watch_flush exit 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 20 client.41969.objecter shutdown clearing up homeless session... 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 10 client.41969.objecter successfully canceled tick 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 msgr2=0x7f8468071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 0x7f8468071ce0 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f8474009680 tx=0x7f8474009260 comp rx=0 tx=0).stop 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 msgr2=0x7f8480074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 0x7f8480074620 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f8470006010 tx=0x7f84700057c0 comp rx=0 tx=0).stop 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 shutdown_connections 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f846806f820 0x7f8468071ce0 unknown :-1 s=CLOSED pgs=3438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:28.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8480101e60 0x7f84800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8480107ee0 0x7f8480074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 --2- 172.21.15.143:0/3014812511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8480102820 0x7f8480074620 secure :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f8470006010 tx=0x7f84700057c0 comp rx=0 tx=0).stop 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 >> 172.21.15.143:0/3014812511 conn(0x7f8480078f90 msgr2=0x7f8480108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 shutdown_connections 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 -- 172.21.15.143:0/3014812511 wait complete. 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:28.523+0000 7f84866cf700 1 librados: shutdown 2024-03-11T21:40:28.507 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:40:28.577 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:28.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:28.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:28.729 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:28.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.893105+0000 osd.4 (osd.4) 389 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:26.895400+0000 osd.4 (osd.4) 390 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.430769+0000 osd.1 (osd.1) 211 : cluster 0 1.3 scrub starts 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.432559+0000 osd.1 (osd.1) 212 : cluster 0 1.3 scrub ok 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.526338+0000 osd.0 (osd.0) 400 : cluster 0 39.14 scrub starts 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.533770+0000 osd.0 (osd.0) 401 : cluster 0 39.14 scrub ok 2024-03-11T21:40:28.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:27.569112+0000 mgr.y (mgr.14550) 681 : cluster 0 pgmap v1129: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:27.757477+0000 mon.c (mon.2) 1200 : audit 1 from='client.? 172.21.15.143:0/3014812511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:27.757775+0000 mon.a (mon.0) 3468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.500016+0000 mon.a (mon.0) 3469 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:28.510431+0000 mon.a (mon.0) 3470 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:28.510594+0000 mon.a (mon.0) 3471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.516660+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:28.523385+0000 mon.c (mon.2) 1201 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[21512]: audit 2024-03-11T21:40:28.528370+0000 mon.a (mon.0) 3473 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.893105+0000 osd.4 (osd.4) 389 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:26.895400+0000 osd.4 (osd.4) 390 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.430769+0000 osd.1 (osd.1) 211 : cluster 0 1.3 scrub starts 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.432559+0000 osd.1 (osd.1) 212 : cluster 0 1.3 scrub ok 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.526338+0000 osd.0 (osd.0) 400 : cluster 0 39.14 scrub starts 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.533770+0000 osd.0 (osd.0) 401 : cluster 0 39.14 scrub ok 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:27.569112+0000 mgr.y (mgr.14550) 681 : cluster 0 pgmap v1129: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:27.757477+0000 mon.c (mon.2) 1200 : audit 1 from='client.? 172.21.15.143:0/3014812511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:27.757775+0000 mon.a (mon.0) 3468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.500016+0000 mon.a (mon.0) 3469 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:28.510431+0000 mon.a (mon.0) 3470 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:28.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:28.510594+0000 mon.a (mon.0) 3471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:28.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.516660+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:40:28.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:28.523385+0000 mon.c (mon.2) 1201 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.847 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:28 smithi143 bash[29284]: audit 2024-03-11T21:40:28.528370+0000 mon.a (mon.0) 3473 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.858 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:28.930 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.893105+0000 osd.4 (osd.4) 389 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:26.895400+0000 osd.4 (osd.4) 390 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.430769+0000 osd.1 (osd.1) 211 : cluster 0 1.3 scrub starts 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.432559+0000 osd.1 (osd.1) 212 : cluster 0 1.3 scrub ok 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.526338+0000 osd.0 (osd.0) 400 : cluster 0 39.14 scrub starts 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.533770+0000 osd.0 (osd.0) 401 : cluster 0 39.14 scrub ok 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:27.569112+0000 mgr.y (mgr.14550) 681 : cluster 0 pgmap v1129: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:27.757477+0000 mon.c (mon.2) 1200 : audit 1 from='client.? 172.21.15.143:0/3014812511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:27.757775+0000 mon.a (mon.0) 3468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.500016+0000 mon.a (mon.0) 3469 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:28.510431+0000 mon.a (mon.0) 3470 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:28.510594+0000 mon.a (mon.0) 3471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.516660+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:28.523385+0000 mon.c (mon.2) 1201 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[23654]: audit 2024-03-11T21:40:28.528370+0000 mon.a (mon.0) 3473 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:28.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:28.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:29.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:29.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:29.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:29.252 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:29.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.383 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.514 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.767578+0000 osd.6 (osd.6) 327 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:40:29.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.770663+0000 osd.6 (osd.6) 328 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:40:29.628 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.916866+0000 osd.5 (osd.5) 419 : cluster 0 3.1d scrub starts 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: cluster 2024-03-11T21:40:28.925256+0000 osd.5 (osd.5) 420 : cluster 0 3.1d scrub ok 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: audit 2024-03-11T21:40:28.950081+0000 mgr.y (mgr.14550) 682 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: audit 2024-03-11T21:40:29.513427+0000 mon.a (mon.0) 3474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: cluster 2024-03-11T21:40:29.521406+0000 mon.a (mon.0) 3475 : cluster 0 osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: audit 2024-03-11T21:40:29.522880+0000 mon.c (mon.2) 1202 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:29.629 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:29 smithi203 bash[23654]: audit 2024-03-11T21:40:29.524446+0000 mon.a (mon.0) 3476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:29.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:29.776 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:29.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.905 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.767578+0000 osd.6 (osd.6) 327 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.770663+0000 osd.6 (osd.6) 328 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.916866+0000 osd.5 (osd.5) 419 : cluster 0 3.1d scrub starts 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: cluster 2024-03-11T21:40:28.925256+0000 osd.5 (osd.5) 420 : cluster 0 3.1d scrub ok 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: audit 2024-03-11T21:40:28.950081+0000 mgr.y (mgr.14550) 682 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: audit 2024-03-11T21:40:29.513427+0000 mon.a (mon.0) 3474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: cluster 2024-03-11T21:40:29.521406+0000 mon.a (mon.0) 3475 : cluster 0 osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:40:29.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: audit 2024-03-11T21:40:29.522880+0000 mon.c (mon.2) 1202 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[21512]: audit 2024-03-11T21:40:29.524446+0000 mon.a (mon.0) 3476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.767578+0000 osd.6 (osd.6) 327 : cluster 0 39.1d deep-scrub starts 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.770663+0000 osd.6 (osd.6) 328 : cluster 0 39.1d deep-scrub ok 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.916866+0000 osd.5 (osd.5) 419 : cluster 0 3.1d scrub starts 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: cluster 2024-03-11T21:40:28.925256+0000 osd.5 (osd.5) 420 : cluster 0 3.1d scrub ok 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: audit 2024-03-11T21:40:28.950081+0000 mgr.y (mgr.14550) 682 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: audit 2024-03-11T21:40:29.513427+0000 mon.a (mon.0) 3474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: cluster 2024-03-11T21:40:29.521406+0000 mon.a (mon.0) 3475 : cluster 0 osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: audit 2024-03-11T21:40:29.522880+0000 mon.c (mon.2) 1202 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:29.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:29 smithi143 bash[29284]: audit 2024-03-11T21:40:29.524446+0000 mon.a (mon.0) 3476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]: dispatch 2024-03-11T21:40:30.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:30.031 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:30.078 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:30.080 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:30.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:30.157 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:30.260 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:30.287 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:30.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:30.415 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:30.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:30.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:30.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:30.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[29284]: cluster 2024-03-11T21:40:29.570216+0000 mgr.y (mgr.14550) 683 : cluster 0 pgmap v1132: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 258 B/s wr, 1 op/s 2024-03-11T21:40:30.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[29284]: cluster 2024-03-11T21:40:30.513653+0000 mon.a (mon.0) 3477 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:30.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[29284]: audit 2024-03-11T21:40:30.516057+0000 mon.a (mon.0) 3478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]': finished 2024-03-11T21:40:30.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[29284]: cluster 2024-03-11T21:40:30.523529+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:40:30.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[21512]: cluster 2024-03-11T21:40:29.570216+0000 mgr.y (mgr.14550) 683 : cluster 0 pgmap v1132: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 258 B/s wr, 1 op/s 2024-03-11T21:40:30.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[21512]: cluster 2024-03-11T21:40:30.513653+0000 mon.a (mon.0) 3477 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:30.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[21512]: audit 2024-03-11T21:40:30.516057+0000 mon.a (mon.0) 3478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]': finished 2024-03-11T21:40:30.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:30 smithi143 bash[21512]: cluster 2024-03-11T21:40:30.523529+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:40:30.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:30.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:30.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:40:30.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:30.934 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:30 smithi203 bash[23654]: cluster 2024-03-11T21:40:29.570216+0000 mgr.y (mgr.14550) 683 : cluster 0 pgmap v1132: 246 pgs: 5 unknown, 241 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 258 B/s wr, 1 op/s 2024-03-11T21:40:31.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:30 smithi203 bash[23654]: cluster 2024-03-11T21:40:30.513653+0000 mon.a (mon.0) 3477 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:31.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:30 smithi203 bash[23654]: audit 2024-03-11T21:40:30.516057+0000 mon.a (mon.0) 3478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-120", "mode": "writeback"}]': finished 2024-03-11T21:40:31.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:30 smithi203 bash[23654]: cluster 2024-03-11T21:40:30.523529+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:40:31.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:31.065 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:31.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:31.194 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:31.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.323 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:31.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:31.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.582 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:40:31.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:31.710 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:31.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: audit 2024-03-11T21:40:30.646369+0000 mon.c (mon.2) 1203 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: audit 2024-03-11T21:40:30.646969+0000 mon.a (mon.0) 3480 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: cluster 2024-03-11T21:40:31.496282+0000 mon.a (mon.0) 3481 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: audit 2024-03-11T21:40:31.500276+0000 mon.a (mon.0) 3482 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: cluster 2024-03-11T21:40:31.512870+0000 mon.a (mon.0) 3483 : cluster 0 osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: audit 2024-03-11T21:40:31.513038+0000 mon.c (mon.2) 1204 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:31 smithi203 bash[23654]: audit 2024-03-11T21:40:31.517416+0000 mon.a (mon.0) 3484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: audit 2024-03-11T21:40:30.646369+0000 mon.c (mon.2) 1203 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.820 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: audit 2024-03-11T21:40:30.646969+0000 mon.a (mon.0) 3480 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: cluster 2024-03-11T21:40:31.496282+0000 mon.a (mon.0) 3481 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: audit 2024-03-11T21:40:31.500276+0000 mon.a (mon.0) 3482 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: cluster 2024-03-11T21:40:31.512870+0000 mon.a (mon.0) 3483 : cluster 0 osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: audit 2024-03-11T21:40:31.513038+0000 mon.c (mon.2) 1204 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[29284]: audit 2024-03-11T21:40:31.517416+0000 mon.a (mon.0) 3484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: audit 2024-03-11T21:40:30.646369+0000 mon.c (mon.2) 1203 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: audit 2024-03-11T21:40:30.646969+0000 mon.a (mon.0) 3480 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: cluster 2024-03-11T21:40:31.496282+0000 mon.a (mon.0) 3481 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: audit 2024-03-11T21:40:31.500276+0000 mon.a (mon.0) 3482 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: cluster 2024-03-11T21:40:31.512870+0000 mon.a (mon.0) 3483 : cluster 0 osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: audit 2024-03-11T21:40:31.513038+0000 mon.c (mon.2) 1204 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.821 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:31 smithi143 bash[21512]: audit 2024-03-11T21:40:31.517416+0000 mon.a (mon.0) 3484 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]: dispatch 2024-03-11T21:40:31.831 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:31.961 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:32.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:32.095 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:32.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:32.219 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:32.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:32.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:32.434 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:32.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:32.481 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:32.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:32.613 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[21512]: cluster 2024-03-11T21:40:31.571183+0000 mgr.y (mgr.14550) 684 : cluster 0 pgmap v1135: 246 pgs: 246 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[21512]: cluster 2024-03-11T21:40:32.500395+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[21512]: audit 2024-03-11T21:40:32.501972+0000 mon.a (mon.0) 3486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[21512]: cluster 2024-03-11T21:40:32.508814+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[29284]: cluster 2024-03-11T21:40:31.571183+0000 mgr.y (mgr.14550) 684 : cluster 0 pgmap v1135: 246 pgs: 246 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[29284]: cluster 2024-03-11T21:40:32.500395+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[29284]: audit 2024-03-11T21:40:32.501972+0000 mon.a (mon.0) 3486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:32.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:32 smithi143 bash[29284]: cluster 2024-03-11T21:40:32.508814+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:40:32.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:32.744 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:32.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:32.858 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:32 smithi203 bash[23654]: cluster 2024-03-11T21:40:31.571183+0000 mgr.y (mgr.14550) 684 : cluster 0 pgmap v1135: 246 pgs: 246 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:32.858 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:32 smithi203 bash[23654]: cluster 2024-03-11T21:40:32.500395+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:32.858 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:32 smithi203 bash[23654]: audit 2024-03-11T21:40:32.501972+0000 mon.a (mon.0) 3486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-120"}]': finished 2024-03-11T21:40:32.858 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:32 smithi203 bash[23654]: cluster 2024-03-11T21:40:32.508814+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:40:32.874 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:32.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:33.004 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:33.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:33.135 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:33.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:33.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:33.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:33.394 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:33.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.526 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:33.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:33.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:33.790 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:33.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:33.918 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:34.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.043 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:34.146 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:34.178 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:34.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:34.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.440 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:34.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:34.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:34 smithi143 bash[21512]: cluster 2024-03-11T21:40:33.533939+0000 mon.a (mon.0) 3488 : cluster 0 osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:40:34.684 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:34 smithi143 bash[21512]: cluster 2024-03-11T21:40:33.572040+0000 mgr.y (mgr.14550) 685 : cluster 0 pgmap v1138: 214 pgs: 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:34.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:34 smithi143 bash[29284]: cluster 2024-03-11T21:40:33.533939+0000 mon.a (mon.0) 3488 : cluster 0 osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:40:34.684 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:34 smithi143 bash[29284]: cluster 2024-03-11T21:40:33.572040+0000 mgr.y (mgr.14550) 685 : cluster 0 pgmap v1138: 214 pgs: 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:34.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:34.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:34 smithi203 bash[23654]: cluster 2024-03-11T21:40:33.533939+0000 mon.a (mon.0) 3488 : cluster 0 osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:40:34.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:34 smithi203 bash[23654]: cluster 2024-03-11T21:40:33.572040+0000 mgr.y (mgr.14550) 685 : cluster 0 pgmap v1138: 214 pgs: 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:34.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:34.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:34.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:34.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:35.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:35.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:35.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.217 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:35.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:35.347 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:35.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:35.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:35.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:35.590 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[21512]: cluster 2024-03-11T21:40:34.552043+0000 mon.a (mon.0) 3489 : cluster 0 osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:40:35.590 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[21512]: audit 2024-03-11T21:40:34.560342+0000 mon.c (mon.2) 1205 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.590 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[21512]: audit 2024-03-11T21:40:34.562161+0000 mon.a (mon.0) 3490 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.590 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[29284]: cluster 2024-03-11T21:40:34.552043+0000 mon.a (mon.0) 3489 : cluster 0 osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:40:35.590 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[29284]: audit 2024-03-11T21:40:34.560342+0000 mon.c (mon.2) 1205 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.591 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:35 smithi143 bash[29284]: audit 2024-03-11T21:40:34.562161+0000 mon.a (mon.0) 3490 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.602 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:35.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:35.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:35 smithi203 bash[23654]: cluster 2024-03-11T21:40:34.552043+0000 mon.a (mon.0) 3489 : cluster 0 osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:40:35.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:35 smithi203 bash[23654]: audit 2024-03-11T21:40:34.560342+0000 mon.c (mon.2) 1205 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:35 smithi203 bash[23654]: audit 2024-03-11T21:40:34.562161+0000 mon.a (mon.0) 3490 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:35.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:35.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.862 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:35.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:35.993 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:36.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:36.124 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:36.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:36.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:36.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:36.381 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:36.459 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:40:36.459 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:40:36.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:36.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/1449038954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34107f30 msgr2=0x7f9c34108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- 172.21.15.143:0/1449038954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34107f30 0x7f9c34108310 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f9c240098f0 tx=0x7f9c24009c00 comp rx=0 tx=0).stop 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/1449038954 shutdown_connections 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- 172.21.15.143:0/1449038954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34102880 0x7f9c3410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- 172.21.15.143:0/1449038954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34107f30 0x7f9c34108310 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- 172.21.15.143:0/1449038954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34101ec0 0x7f9c34102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/1449038954 >> 172.21.15.143:0/1449038954 conn(0x7f9c34078eb0 msgr2=0x7f9c340792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/1449038954 shutdown_connections 2024-03-11T21:40:36.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/1449038954 wait complete. 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 librados: starting msgr at 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 librados: starting objecter 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 Processor -- start 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- start start 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 librados: setting wanted keys 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 librados: calling monclient init 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 0x7f9c340740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:36.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34107f30 0x7f9c34074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341116a0 con 0x7f9c34107f30 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34111520 con 0x7f9c34101ec0 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34111820 con 0x7f9c34102880 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58392/0 (socket says 172.21.15.143:58392) 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 -- 172.21.15.143:0/2075452817 learned_addr learned my addr 172.21.15.143:0/2075452817 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c337fe700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34107f30 0x7f9c34074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c32ffd700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 0x7f9c340740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 -- 172.21.15.143:0/2075452817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 msgr2=0x7f9c340740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 0x7f9c340740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 -- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34107f30 msgr2=0x7f9c34074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34107f30 0x7f9c34074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 -- 172.21.15.143:0/2075452817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c341a6230 con 0x7f9c34102880 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c32ffd700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 0x7f9c340740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:40:36.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c327fc700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800ec10 tx=0x7f9c2800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c1bfff700 10 client.?.objecter ms_handle_connect 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c1bfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800cd40 con 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9c280044a0 con 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28010430 con 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c341a6520 con 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 10 client.41981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c341a6a10 con 0x7f9c34102880 2024-03-11T21:40:36.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.514+0000 7f9c397b1700 1 librados: init done 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3404faa0 con 0x7f9c34102880 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9c28004000 con 0x7f9c34102880 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 0x7f9c1c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(736..736 src has 87..736) v4 ==== 10210+0+0 (secure 0 0 0) 0x7f9c2808f150 con 0x7f9c34102880 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 10 client.41981.objecter ms_dispatch 0x7f9c34001550 osd_map(736..736 src has 87..736) v4 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 3 client.41981.objecter handle_osd_map got epochs [736,736] > 0 2024-03-11T21:40:36.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 3 client.41981.objecter handle_osd_map decoding full epoch 736 2024-03-11T21:40:36.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c32ffd700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 0x7f9c1c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:36.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 20 client.41981.objecter dump_active .. 0 homeless 2024-03-11T21:40:36.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c1bfff700 10 client.41981.objecter ms_handle_connect 0x7f9c1c06f820 2024-03-11T21:40:36.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.518+0000 7f9c32ffd700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 0x7f9c1c071ce0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009230 tx=0x7f9c24005b40 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:36.504 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.522+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9c2805bb50 con 0x7f9c34102880 2024-03-11T21:40:36.510 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:36.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:35.542751+0000 mon.a (mon.0) 3491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: cluster 2024-03-11T21:40:35.552241+0000 mon.a (mon.0) 3492 : cluster 0 osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: cluster 2024-03-11T21:40:35.573197+0000 mgr.y (mgr.14550) 686 : cluster 0 pgmap v1141: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:35.587760+0000 mon.c (mon.2) 1206 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:35.594070+0000 mon.a (mon.0) 3493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:36.544191+0000 mon.a (mon.0) 3494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: cluster 2024-03-11T21:40:36.548414+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:36.549255+0000 mon.c (mon.2) 1207 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.624 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[29284]: audit 2024-03-11T21:40:36.550257+0000 mon.a (mon.0) 3496 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:35.542751+0000 mon.a (mon.0) 3491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: cluster 2024-03-11T21:40:35.552241+0000 mon.a (mon.0) 3492 : cluster 0 osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: cluster 2024-03-11T21:40:35.573197+0000 mgr.y (mgr.14550) 686 : cluster 0 pgmap v1141: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:35.587760+0000 mon.c (mon.2) 1206 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:35.594070+0000 mon.a (mon.0) 3493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:36.544191+0000 mon.a (mon.0) 3494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: cluster 2024-03-11T21:40:36.548414+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:36.549255+0000 mon.c (mon.2) 1207 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.625 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:36 smithi143 bash[21512]: audit 2024-03-11T21:40:36.550257+0000 mon.a (mon.0) 3496 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:36.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:36.750+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f9c341a6cd0 con 0x7f9c34102880 2024-03-11T21:40:36.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:36.761 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:35.542751+0000 mon.a (mon.0) 3491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: cluster 2024-03-11T21:40:35.552241+0000 mon.a (mon.0) 3492 : cluster 0 osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: cluster 2024-03-11T21:40:35.573197+0000 mgr.y (mgr.14550) 686 : cluster 0 pgmap v1141: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:35.587760+0000 mon.c (mon.2) 1206 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:35.594070+0000 mon.a (mon.0) 3493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:36.544191+0000 mon.a (mon.0) 3494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: cluster 2024-03-11T21:40:36.548414+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:36.549255+0000 mon.c (mon.2) 1207 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:36 smithi203 bash[23654]: audit 2024-03-11T21:40:36.550257+0000 mon.a (mon.0) 3496 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:36.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:36.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:36.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:37.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:37.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:37.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:37.279 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:37.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:37.410 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:37.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.554+0000 7f9c1bfff700 1 -- 172.21.15.143:0/2075452817 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v738) v1 ==== 113+0+0 (secure 0 0 0) 0x7f9c2805f5a0 con 0x7f9c34102880 2024-03-11T21:40:37.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 10 librados: watch_flush enter 2024-03-11T21:40:37.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 10 librados: watch_flush exit 2024-03-11T21:40:37.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 20 client.41981.objecter shutdown clearing up homeless session... 2024-03-11T21:40:37.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 10 client.41981.objecter successfully canceled tick 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 msgr2=0x7f9c1c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 0x7f9c1c071ce0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009230 tx=0x7f9c24005b40 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 msgr2=0x7f9c34074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800ec10 tx=0x7f9c2800ef20 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 shutdown_connections 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9c1c06f820 0x7f9c1c071ce0 unknown :-1 s=CLOSED pgs=3440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c34107f30 0x7f9c34074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c34101ec0 0x7f9c340740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 --2- 172.21.15.143:0/2075452817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34102880 0x7f9c34074620 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:37.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 >> 172.21.15.143:0/2075452817 conn(0x7f9c34078eb0 msgr2=0x7f9c341005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:37.543 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 shutdown_connections 2024-03-11T21:40:37.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 -- 172.21.15.143:0/2075452817 wait complete. 2024-03-11T21:40:37.544 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:37.558+0000 7f9c397b1700 1 librados: shutdown 2024-03-11T21:40:37.544 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:40:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:37.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:40:37.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.656 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:36.755725+0000 mon.c (mon.2) 1208 : audit 1 from='client.? 172.21.15.143:0/2075452817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:36.756392+0000 mon.a (mon.0) 3497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.137613+0000 mon.a (mon.0) 3498 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.140143+0000 mon.c (mon.2) 1209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.548045+0000 mon.a (mon.0) 3499 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.548285+0000 mon.a (mon.0) 3500 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: cluster 2024-03-11T21:40:37.555915+0000 mon.a (mon.0) 3501 : cluster 0 osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.558220+0000 mon.c (mon.2) 1210 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.657 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:37 smithi203 bash[23654]: audit 2024-03-11T21:40:37.560059+0000 mon.a (mon.0) 3502 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.671 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:37.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:37.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:37.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:36.755725+0000 mon.c (mon.2) 1208 : audit 1 from='client.? 172.21.15.143:0/2075452817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:36.756392+0000 mon.a (mon.0) 3497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.137613+0000 mon.a (mon.0) 3498 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.140143+0000 mon.c (mon.2) 1209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:37.910 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.548045+0000 mon.a (mon.0) 3499 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.548285+0000 mon.a (mon.0) 3500 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: cluster 2024-03-11T21:40:37.555915+0000 mon.a (mon.0) 3501 : cluster 0 osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.558220+0000 mon.c (mon.2) 1210 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[29284]: audit 2024-03-11T21:40:37.560059+0000 mon.a (mon.0) 3502 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:36.755725+0000 mon.c (mon.2) 1208 : audit 1 from='client.? 172.21.15.143:0/2075452817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:36.756392+0000 mon.a (mon.0) 3497 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.137613+0000 mon.a (mon.0) 3498 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.140143+0000 mon.c (mon.2) 1209 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.548045+0000 mon.a (mon.0) 3499 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.548285+0000 mon.a (mon.0) 3500 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: cluster 2024-03-11T21:40:37.555915+0000 mon.a (mon.0) 3501 : cluster 0 osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:40:37.911 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.558220+0000 mon.c (mon.2) 1210 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:37 smithi143 bash[21512]: audit 2024-03-11T21:40:37.560059+0000 mon.a (mon.0) 3502 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]: dispatch 2024-03-11T21:40:37.922 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:38.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:38.057 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:38.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.187 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:38.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.318 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:38.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:38.447 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:38.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:40:38.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:38.580 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:38.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:38.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:38.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:38.818 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:38 smithi203 bash[23654]: cluster 2024-03-11T21:40:37.573737+0000 mgr.y (mgr.14550) 687 : cluster 0 pgmap v1144: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:38.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:38 smithi203 bash[23654]: cluster 2024-03-11T21:40:38.549101+0000 mon.a (mon.0) 3503 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:38.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:38 smithi203 bash[23654]: audit 2024-03-11T21:40:38.556536+0000 mon.a (mon.0) 3504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]': finished 2024-03-11T21:40:38.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:38 smithi203 bash[23654]: cluster 2024-03-11T21:40:38.563655+0000 mon.a (mon.0) 3505 : cluster 0 osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:40:38.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:38.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.941 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[29284]: cluster 2024-03-11T21:40:37.573737+0000 mgr.y (mgr.14550) 687 : cluster 0 pgmap v1144: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[29284]: cluster 2024-03-11T21:40:38.549101+0000 mon.a (mon.0) 3503 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[29284]: audit 2024-03-11T21:40:38.556536+0000 mon.a (mon.0) 3504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]': finished 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[29284]: cluster 2024-03-11T21:40:38.563655+0000 mon.a (mon.0) 3505 : cluster 0 osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[21512]: cluster 2024-03-11T21:40:37.573737+0000 mgr.y (mgr.14550) 687 : cluster 0 pgmap v1144: 246 pgs: 5 creating+peering, 27 unknown, 214 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[21512]: cluster 2024-03-11T21:40:38.549101+0000 mon.a (mon.0) 3503 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[21512]: audit 2024-03-11T21:40:38.556536+0000 mon.a (mon.0) 3504 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-122", "mode": "writeback"}]': finished 2024-03-11T21:40:38.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:38 smithi143 bash[21512]: cluster 2024-03-11T21:40:38.563655+0000 mon.a (mon.0) 3505 : cluster 0 osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:40:38.959 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:38.960 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:38.963 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:39.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:38 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:39.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:39.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:39.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:39.353 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:39.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:39.481 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:39.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:39 smithi203 bash[23654]: audit 2024-03-11T21:40:38.663246+0000 mon.c (mon.2) 1211 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:39 smithi203 bash[23654]: audit 2024-03-11T21:40:38.663642+0000 mon.a (mon.0) 3506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.596 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:39 smithi203 bash[23654]: audit 2024-03-11T21:40:38.961417+0000 mgr.y (mgr.14550) 688 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:39.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:39.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:39.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.855 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[29284]: audit 2024-03-11T21:40:38.663246+0000 mon.c (mon.2) 1211 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.855 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[29284]: audit 2024-03-11T21:40:38.663642+0000 mon.a (mon.0) 3506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.856 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[29284]: audit 2024-03-11T21:40:38.961417+0000 mgr.y (mgr.14550) 688 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:39.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[21512]: audit 2024-03-11T21:40:38.663246+0000 mon.c (mon.2) 1211 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[21512]: audit 2024-03-11T21:40:38.663642+0000 mon.a (mon.0) 3506 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:39.856 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:39 smithi143 bash[21512]: audit 2024-03-11T21:40:38.961417+0000 mgr.y (mgr.14550) 688 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:39.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:39.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:40.004 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:40.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.136 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.370 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.396 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:40.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:40.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[29284]: cluster 2024-03-11T21:40:39.574858+0000 mgr.y (mgr.14550) 689 : cluster 0 pgmap v1146: 246 pgs: 5 creating+peering, 241 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-11T21:40:40.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[29284]: audit 2024-03-11T21:40:39.592624+0000 mon.a (mon.0) 3507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:40.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[29284]: cluster 2024-03-11T21:40:39.600399+0000 mon.a (mon.0) 3508 : cluster 0 osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:40:40.634 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[29284]: audit 2024-03-11T21:40:39.601569+0000 mon.c (mon.2) 1212 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[29284]: audit 2024-03-11T21:40:39.604130+0000 mon.a (mon.0) 3509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[21512]: cluster 2024-03-11T21:40:39.574858+0000 mgr.y (mgr.14550) 689 : cluster 0 pgmap v1146: 246 pgs: 5 creating+peering, 241 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[21512]: audit 2024-03-11T21:40:39.592624+0000 mon.a (mon.0) 3507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[21512]: cluster 2024-03-11T21:40:39.600399+0000 mon.a (mon.0) 3508 : cluster 0 osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[21512]: audit 2024-03-11T21:40:39.601569+0000 mon.c (mon.2) 1212 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:40.635 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:40 smithi143 bash[21512]: audit 2024-03-11T21:40:39.604130+0000 mon.a (mon.0) 3509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:40.771 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:40.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:41.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:41.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:41.028 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[51372]: logger=sqlstore.transactions t=2024-03-11T21:40:40.739526014Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:40:41.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[23654]: cluster 2024-03-11T21:40:39.574858+0000 mgr.y (mgr.14550) 689 : cluster 0 pgmap v1146: 246 pgs: 5 creating+peering, 241 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-11T21:40:41.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[23654]: audit 2024-03-11T21:40:39.592624+0000 mon.a (mon.0) 3507 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:41.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[23654]: cluster 2024-03-11T21:40:39.600399+0000 mon.a (mon.0) 3508 : cluster 0 osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:40:41.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[23654]: audit 2024-03-11T21:40:39.601569+0000 mon.c (mon.2) 1212 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:41.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:40 smithi203 bash[23654]: audit 2024-03-11T21:40:39.604130+0000 mon.a (mon.0) 3509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]: dispatch 2024-03-11T21:40:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:41.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.156 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:41.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:41.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:41.389 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.417 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:41.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:41.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:41.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:41.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:41.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.804 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:41.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:41.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:42.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:42.064 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:42.086 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[29284]: cluster 2024-03-11T21:40:40.593361+0000 mon.a (mon.0) 3510 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[29284]: audit 2024-03-11T21:40:40.602904+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[29284]: cluster 2024-03-11T21:40:40.605736+0000 mon.a (mon.0) 3512 : cluster 0 osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[29284]: cluster 2024-03-11T21:40:41.508010+0000 mon.a (mon.0) 3513 : cluster 0 osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[21512]: cluster 2024-03-11T21:40:40.593361+0000 mon.a (mon.0) 3510 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[21512]: audit 2024-03-11T21:40:40.602904+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[21512]: cluster 2024-03-11T21:40:40.605736+0000 mon.a (mon.0) 3512 : cluster 0 osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:40:42.087 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:41 smithi143 bash[21512]: cluster 2024-03-11T21:40:41.508010+0000 mon.a (mon.0) 3513 : cluster 0 osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:40:42.173 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.218 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:42.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:42.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:41 smithi203 bash[23654]: cluster 2024-03-11T21:40:40.593361+0000 mon.a (mon.0) 3510 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:42.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:41 smithi203 bash[23654]: audit 2024-03-11T21:40:40.602904+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-122"}]': finished 2024-03-11T21:40:42.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:41 smithi203 bash[23654]: cluster 2024-03-11T21:40:40.605736+0000 mon.a (mon.0) 3512 : cluster 0 osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:40:42.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:41 smithi203 bash[23654]: cluster 2024-03-11T21:40:41.508010+0000 mon.a (mon.0) 3513 : cluster 0 osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:40:42.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:42.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:42.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.503 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:42.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:42.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:42.769 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:42.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:42.896 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:42.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:43.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:42 smithi203 bash[23654]: cluster 2024-03-11T21:40:41.575432+0000 mgr.y (mgr.14550) 690 : cluster 0 pgmap v1150: 214 pgs: 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:43.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:42 smithi203 bash[23654]: cluster 2024-03-11T21:40:42.521285+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:40:43.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:42 smithi203 bash[23654]: audit 2024-03-11T21:40:42.522912+0000 mon.c (mon.2) 1213 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:42 smithi203 bash[23654]: audit 2024-03-11T21:40:42.524172+0000 mon.a (mon.0) 3515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.031 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:43.031 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:43.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:43.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:43.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.285 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:43.291 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[21512]: cluster 2024-03-11T21:40:41.575432+0000 mgr.y (mgr.14550) 690 : cluster 0 pgmap v1150: 214 pgs: 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:43.291 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[21512]: cluster 2024-03-11T21:40:42.521285+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:40:43.291 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[21512]: audit 2024-03-11T21:40:42.522912+0000 mon.c (mon.2) 1213 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.292 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[21512]: audit 2024-03-11T21:40:42.524172+0000 mon.a (mon.0) 3515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[29284]: cluster 2024-03-11T21:40:41.575432+0000 mgr.y (mgr.14550) 690 : cluster 0 pgmap v1150: 214 pgs: 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:40:43.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[29284]: cluster 2024-03-11T21:40:42.521285+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:40:43.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[29284]: audit 2024-03-11T21:40:42.522912+0000 mon.c (mon.2) 1213 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.292 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:42 smithi143 bash[29284]: audit 2024-03-11T21:40:42.524172+0000 mon.a (mon.0) 3515 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:43.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:43.415 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:43.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:43.554 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:43.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:43.686 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:43.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:43.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:44.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.075 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:44.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.205 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:44.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.342 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:44.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:44.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:44.468 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:44.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.597 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:44.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:44.777 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:44 smithi203 bash[23654]: audit 2024-03-11T21:40:43.516827+0000 mon.a (mon.0) 3516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:44.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:44 smithi203 bash[23654]: cluster 2024-03-11T21:40:43.520475+0000 mon.a (mon.0) 3517 : cluster 0 osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:40:44.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:44 smithi203 bash[23654]: cluster 2024-03-11T21:40:43.575992+0000 mgr.y (mgr.14550) 691 : cluster 0 pgmap v1153: 246 pgs: 20 creating+peering, 12 unknown, 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 257 B/s wr, 2 op/s 2024-03-11T21:40:44.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.856 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:44.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[21512]: audit 2024-03-11T21:40:43.516827+0000 mon.a (mon.0) 3516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:44.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[21512]: cluster 2024-03-11T21:40:43.520475+0000 mon.a (mon.0) 3517 : cluster 0 osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:40:44.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[21512]: cluster 2024-03-11T21:40:43.575992+0000 mgr.y (mgr.14550) 691 : cluster 0 pgmap v1153: 246 pgs: 20 creating+peering, 12 unknown, 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 257 B/s wr, 2 op/s 2024-03-11T21:40:44.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[29284]: audit 2024-03-11T21:40:43.516827+0000 mon.a (mon.0) 3516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:44.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[29284]: cluster 2024-03-11T21:40:43.520475+0000 mon.a (mon.0) 3517 : cluster 0 osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:40:44.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:44 smithi143 bash[29284]: cluster 2024-03-11T21:40:43.575992+0000 mgr.y (mgr.14550) 691 : cluster 0 pgmap v1153: 246 pgs: 20 creating+peering, 12 unknown, 214 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 257 B/s wr, 2 op/s 2024-03-11T21:40:44.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.986 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:45.089 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:45.117 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:45.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:45.243 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:45.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:45.373 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:45.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.506 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:45.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.637 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[21512]: cluster 2024-03-11T21:40:44.534687+0000 mon.a (mon.0) 3518 : cluster 0 osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[21512]: audit 2024-03-11T21:40:44.565941+0000 mon.c (mon.2) 1214 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[21512]: audit 2024-03-11T21:40:44.566470+0000 mon.a (mon.0) 3519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[29284]: cluster 2024-03-11T21:40:44.534687+0000 mon.a (mon.0) 3518 : cluster 0 osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[29284]: audit 2024-03-11T21:40:44.565941+0000 mon.c (mon.2) 1214 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:45 smithi143 bash[29284]: audit 2024-03-11T21:40:44.566470+0000 mon.a (mon.0) 3519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:45.752 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:45 smithi203 bash[23654]: cluster 2024-03-11T21:40:44.534687+0000 mon.a (mon.0) 3518 : cluster 0 osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:40:45.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:45 smithi203 bash[23654]: audit 2024-03-11T21:40:44.565941+0000 mon.c (mon.2) 1214 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:45 smithi203 bash[23654]: audit 2024-03-11T21:40:44.566470+0000 mon.a (mon.0) 3519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:45.769 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:45.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:45.898 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- 172.21.15.143:0/4039802159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 msgr2=0x7fb2f81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/4039802159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 0x7fb2f81011a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0098f0 tx=0x7fb2ec009c00 comp rx=0 tx=0).stop 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- 172.21.15.143:0/4039802159 shutdown_connections 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/4039802159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f81016e0 0x7fb2f810db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/4039802159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f8106dd0 0x7fb2f81071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:45.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/4039802159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 0x7fb2f81011a0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- 172.21.15.143:0/4039802159 >> 172.21.15.143:0/4039802159 conn(0x7fb2f8077d90 msgr2=0x7fb2f80781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- 172.21.15.143:0/4039802159 shutdown_connections 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- 172.21.15.143:0/4039802159 wait complete. 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 librados: starting msgr at 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 librados: starting objecter 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 Processor -- start 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- start start 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 librados: setting wanted keys 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 librados: calling monclient init 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 0x7fb2f8199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8106dd0 0x7fb2f819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f810f410 con 0x7fb2f8100d20 2024-03-11T21:40:45.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f810f290 con 0x7fb2f8106dd0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2fda7e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f810f590 con 0x7fb2f81016e0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:55900/0 (socket says 172.21.15.143:55900) 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 -- 172.21.15.143:0/3529877513 learned_addr learned my addr 172.21.15.143:0/3529877513 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 0x7fb2f8199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8106dd0 0x7fb2f819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8106dd0 msgr2=0x7fb2f819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f8106dd0 0x7fb2f819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 msgr2=0x7fb2f8199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f8100d20 0x7fb2f8199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.982+0000 7fb2f6ffd700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f819ee00 con 0x7fb2f81016e0 2024-03-11T21:40:45.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0059d0 tx=0x7fb2ec00fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 10 client.?.objecter ms_handle_connect 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec027070 con 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb2ec005040 con 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec004910 con 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f819f090 con 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 10 client.41987.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f81a6970 con 0x7fb2f81016e0 2024-03-11T21:40:45.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 librados: init done 2024-03-11T21:40:45.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb2f804faa0 con 0x7fb2f81016e0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb2ec02a070 con 0x7fb2f81016e0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 0x7fb2e0071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 0x7fb2e0071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(746..746 src has 87..746) v4 ==== 10205+0+0 (secure 0 0 0) 0x7fb2ec08f8a0 con 0x7fb2f81016e0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 osd_map(746..746 src has 87..746) v4 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 3 client.41987.objecter handle_osd_map got epochs [746,746] > 0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 3 client.41987.objecter handle_osd_map decoding full epoch 746 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 20 client.41987.objecter dump_active .. 0 homeless 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v746) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb2ec013030 con 0x7fb2f81016e0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2e006f810 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 0x7fb2e0071cd0 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8005fd0 tx=0x7fb2e8009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:45.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 1 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fb2f8062c40 0x7fb2f8104020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 1 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 0 1 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 4 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 10 client.41987.objecter _send_command 1 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81056e0 con 0x7fb2f8062c40 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fb2f8062c40 0x7fb2f8104020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f8062c40 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.986+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fb2f8062c40 0x7fb2f8104020 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.990+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f81056e0 con 0x7fb2f8062c40 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.990+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.990+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 1 = system:0 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.990+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 4 2024-03-11T21:40:45.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:45.990+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 0 1 2024-03-11T21:40:45.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:45.989 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 2 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 2 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f819b150 osd=0 3 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 0 2 2024-03-11T21:40:45.990 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 4 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 10 client.41987.objecter _send_command 2 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.006+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f8104560 con 0x7fb2f8062c40 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f8104560 con 0x7fb2f8062c40 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 4 2024-03-11T21:40:45.991 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 0 2 2024-03-11T21:40:45.992 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fb2f8062830 0x7fb2f81aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 1 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 4 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 10 client.41987.objecter _send_command 3 2024-03-11T21:40:45.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f8105450 con 0x7fb2f8062830 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fb2f8062830 0x7fb2f81aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f8062830 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fb2f8062830 0x7fb2f81aab60 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f8105450 con 0x7fb2f8062830 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 3 = system:0 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 4 2024-03-11T21:40:45.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.010+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 1 3 2024-03-11T21:40:46.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 4 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:46.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 4 2024-03-11T21:40:46.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f80626c0 osd=1 3 2024-03-11T21:40:46.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 1 4 2024-03-11T21:40:46.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 4 2024-03-11T21:40:46.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 10 client.41987.objecter _send_command 4 2024-03-11T21:40:46.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.026+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81049c0 con 0x7fb2f8062830 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81049c0 con 0x7fb2f8062830 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 4 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 1 4 2024-03-11T21:40:46.013 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 5 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fb2f81ac220 0x7fb2f81ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 5 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 2 5 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 4 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 10 client.41987.objecter _send_command 5 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f802d500 con 0x7fb2f81ac220 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fb2f81ac220 0x7fb2f81ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81ac220 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fb2f81ac220 0x7fb2f81ae610 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f802d500 con 0x7fb2f81ac220 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 5 = system:0 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 4 2024-03-11T21:40:46.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.030+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 2 5 2024-03-11T21:40:46.021 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:46.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 6 2024-03-11T21:40:46.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 6 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f802ce80 osd=2 3 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 2 6 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 4 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 10 client.41987.objecter _send_command 6 2024-03-11T21:40:46.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f802c900 con 0x7fb2f81ac220 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f802c900 con 0x7fb2f81ac220 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 4 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 2 6 2024-03-11T21:40:46.034 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 7 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fb2f81abd80 0x7fb2f81b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 7 2024-03-11T21:40:46.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 3 7 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 4 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 10 client.41987.objecter _send_command 7 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81b4b40 con 0x7fb2f81abd80 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.050+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fb2f81abd80 0x7fb2f81b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81abd80 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fb2f81abd80 0x7fb2f81b1ce0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f81b4b40 con 0x7fb2f81abd80 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:40:46.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 7 = system:0 2024-03-11T21:40:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 4 2024-03-11T21:40:46.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.054+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 3 7 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 8 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 8 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f802cd00 osd=3 3 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 3 8 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 4 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 10 client.41987.objecter _send_command 8 2024-03-11T21:40:46.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81ab3b0 con 0x7fb2f81abd80 2024-03-11T21:40:46.056 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81ab3b0 con 0x7fb2f81abd80 2024-03-11T21:40:46.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 4 2024-03-11T21:40:46.057 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 3 8 2024-03-11T21:40:46.057 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 9 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fb2f81af830 0x7fb2f81b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 9 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 4 9 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 4 2024-03-11T21:40:46.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 10 client.41987.objecter _send_command 9 2024-03-11T21:40:46.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81ab400 con 0x7fb2f81af830 2024-03-11T21:40:46.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fb2f81af830 0x7fb2f81b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81af830 2024-03-11T21:40:46.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.074+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fb2f81af830 0x7fb2f81b53b0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.078+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2e8011ba0 con 0x7fb2f81af830 2024-03-11T21:40:46.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.078+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:40:46.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.078+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 9 = system:0 2024-03-11T21:40:46.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.078+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 4 2024-03-11T21:40:46.062 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.078+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 4 9 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 10 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 10 2024-03-11T21:40:46.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81ab100 osd=4 3 2024-03-11T21:40:46.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 4 10 2024-03-11T21:40:46.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 4 2024-03-11T21:40:46.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 10 client.41987.objecter _send_command 10 2024-03-11T21:40:46.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.094+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81aee60 con 0x7fb2f81af830 2024-03-11T21:40:46.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81aee60 con 0x7fb2f81af830 2024-03-11T21:40:46.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 4 2024-03-11T21:40:46.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 4 10 2024-03-11T21:40:46.082 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:40:46.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 11 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fb2f81b2f00 0x7fb2f81b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 11 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 5 11 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 4 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 10 client.41987.objecter _send_command 11 2024-03-11T21:40:46.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81bba50 con 0x7fb2f81b2f00 2024-03-11T21:40:46.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.098+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fb2f81b2f00 0x7fb2f81b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f6ffd700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fb2f81b2f00 0x7fb2f81b6a70 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81b2f00 2024-03-11T21:40:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f81bba50 con 0x7fb2f81b2f00 2024-03-11T21:40:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:40:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 11 = system:0 2024-03-11T21:40:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 4 2024-03-11T21:40:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.102+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 5 11 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 12 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 12 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81aec10 osd=5 3 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 5 12 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 4 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 10 client.41987.objecter _send_command 12 2024-03-11T21:40:46.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.122+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81b2470 con 0x7fb2f81b2f00 2024-03-11T21:40:46.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81b2470 con 0x7fb2f81b2f00 2024-03-11T21:40:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 4 2024-03-11T21:40:46.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 5 12 2024-03-11T21:40:46.108 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 13 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fb2f81b65d0 0x7fb2f81bc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 13 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 6 13 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 4 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 10 client.41987.objecter _send_command 13 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81b2390 con 0x7fb2f81b65d0 2024-03-11T21:40:46.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fb2f81b65d0 0x7fb2f81bc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f7fff700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fb2f81b65d0 0x7fb2f81bc2c0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.126+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81b65d0 2024-03-11T21:40:46.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.130+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f81b2390 con 0x7fb2f81b65d0 2024-03-11T21:40:46.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.130+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:40:46.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.130+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 13 = system:0 2024-03-11T21:40:46.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.130+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 4 2024-03-11T21:40:46.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.130+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 6 13 2024-03-11T21:40:46.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 14 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 14 2024-03-11T21:40:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81b2220 osd=6 3 2024-03-11T21:40:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 6 14 2024-03-11T21:40:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 4 2024-03-11T21:40:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 10 client.41987.objecter _send_command 14 2024-03-11T21:40:46.130 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.146+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81b2390 con 0x7fb2f81b65d0 2024-03-11T21:40:46.133 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81b2390 con 0x7fb2f81b65d0 2024-03-11T21:40:46.133 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.134 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.134 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 4 2024-03-11T21:40:46.134 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 6 14 2024-03-11T21:40:46.134 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 15 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fb2f81bcd10 0x7fb2f81bf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 15 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 7 15 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 4 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 10 client.41987.objecter _send_command 15 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2f81c2090 con 0x7fb2f81bcd10 2024-03-11T21:40:46.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.150+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fb2f81bcd10 0x7fb2f81bf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 10 client.41987.objecter ms_handle_connect 0x7fb2f81bcd10 2024-03-11T21:40:46.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f77fe700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fb2f81bcd10 0x7fb2f81bf100 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2f81c2090 con 0x7fb2f81bcd10 2024-03-11T21:40:46.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:40:46.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 15 = system:0 2024-03-11T21:40:46.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 4 2024-03-11T21:40:46.137 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.154+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 7 15 2024-03-11T21:40:46.150 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 10 client.41987.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign -1 16 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 20 client.41987.objecter _get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_remove -1 16 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 20 client.41987.objecter get_session s=0x7fb2f81b25e0 osd=7 3 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 15 client.41987.objecter _session_command_op_assign 7 16 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 4 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 10 client.41987.objecter _send_command 16 2024-03-11T21:40:46.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.170+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2f81c0e60 con 0x7fb2f81bcd10 2024-03-11T21:40:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f4ff9700 1 -- 172.21.15.143:0/3529877513 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb2f81c0e60 con 0x7fb2f81bcd10 2024-03-11T21:40:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f4ff9700 10 client.41987.objecter ms_dispatch 0x7fb2f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:40:46.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f4ff9700 10 client.41987.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:40:46.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f4ff9700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 4 2024-03-11T21:40:46.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f4ff9700 15 client.41987.objecter _session_command_op_remove 7 16 2024-03-11T21:40:46.157 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 librados: watch_flush enter 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 librados: watch_flush exit 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.0 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fb2f8062c40 msgr2=0x7fb2f8104020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fb2f8062c40 0x7fb2f8104020 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f819b150 osd=0 1 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.1 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fb2f8062830 msgr2=0x7fb2f81aab60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fb2f8062830 0x7fb2f81aab60 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f80626c0 osd=1 1 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.2 2024-03-11T21:40:46.159 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fb2f81ac220 msgr2=0x7fb2f81ae610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fb2f81ac220 0x7fb2f81ae610 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802ce80 osd=2 1 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.3 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f7fff700 1 -- 172.21.15.143:0/3529877513 reap_dead start 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fb2f81abd80 msgr2=0x7fb2f81b1ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fb2f81abd80 0x7fb2f81b1ce0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f802cd00 osd=3 1 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.4 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fb2f81af830 msgr2=0x7fb2f81b53b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fb2f81af830 0x7fb2f81b53b0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81ab100 osd=4 1 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.5 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fb2f81b2f00 msgr2=0x7fb2f81b6a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fb2f81b2f00 0x7fb2f81b6a70 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81aec10 osd=5 1 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.6 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fb2f81b65d0 msgr2=0x7fb2f81bc2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fb2f81b65d0 0x7fb2f81bc2c0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b2220 osd=6 1 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter close_session for osd.7 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fb2f81bcd10 msgr2=0x7fb2f81bf100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fb2f81bcd10 0x7fb2f81bf100 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter put_session s=0x7fb2f81b25e0 osd=7 1 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 20 client.41987.objecter shutdown clearing up homeless session... 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 10 client.41987.objecter successfully canceled tick 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2f7fff700 1 -- 172.21.15.143:0/3529877513 reap_dead start 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 msgr2=0x7fb2e0071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 0x7fb2e0071cd0 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8005fd0 tx=0x7fb2e8009310 comp rx=0 tx=0).stop 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 msgr2=0x7fb2f819a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0059d0 tx=0x7fb2ec00fb20 comp rx=0 tx=0).stop 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 shutdown_connections 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f81016e0 0x7fb2f819a330 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 --2- 172.21.15.143:0/3529877513 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2e006f810 0x7fb2e0071cd0 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 >> 172.21.15.143:0/3529877513 conn(0x7fb2f8077d90 msgr2=0x7fb2f8102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 shutdown_connections 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 -- 172.21.15.143:0/3529877513 wait complete. 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.174+0000 7fb2fda7e700 1 librados: shutdown 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:46.162 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:46.163 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:46.163 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:46.163 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:46.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:46.279 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:46.381 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:46.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:46.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:46.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:46.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/1212235729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc107ea0 msgr2=0x7fc4dc108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/1212235729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc107ea0 0x7fc4dc108280 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009990 tx=0x7fc4cc009ca0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/1212235729 shutdown_connections 2024-03-11T21:40:46.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/1212235729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc102850 0x7fc4dc10ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/1212235729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc107ea0 0x7fc4dc108280 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/1212235729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc101e90 0x7fc4dc102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/1212235729 >> 172.21.15.143:0/1212235729 conn(0x7fc4dc078f40 msgr2=0x7fc4dc079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:46.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/1212235729 shutdown_connections 2024-03-11T21:40:46.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/1212235729 wait complete. 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 librados: starting msgr at 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.558+0000 7fc4e0ad0700 1 librados: starting objecter 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 Processor -- start 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- start start 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 librados: setting wanted keys 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 librados: calling monclient init 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 0x7fc4dc0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc111660 con 0x7fc4dc107ea0 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc1114e0 con 0x7fc4dc101e90 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc1117e0 con 0x7fc4dc102850 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4dad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4dad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35866/0 (socket says 172.21.15.143:35866) 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4dad9d700 1 -- 172.21.15.143:0/67265126 learned_addr learned my addr 172.21.15.143:0/67265126 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:55918/0 (socket says 172.21.15.143:55918) 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4da59c700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 0x7fc4dc0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 -- 172.21.15.143:0/67265126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 msgr2=0x7fc4dc0740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 0x7fc4dc0740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 -- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 msgr2=0x7fc4dc074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 -- 172.21.15.143:0/67265126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc1a61e0 con 0x7fc4dc102850 2024-03-11T21:40:46.546 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4da59c700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 0x7fc4dc0740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4dad9d700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4d9d9b700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000e9f0 tx=0x7fc4d000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 10 client.?.objecter ms_handle_connect 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000cb70 con 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc4d00044a0 con 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0003cb0 con 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc1a64d0 con 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 10 client.41993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc1a6a60 con 0x7fc4dc102850 2024-03-11T21:40:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 librados: init done 2024-03-11T21:40:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc04faa0 con 0x7fc4dc102850 2024-03-11T21:40:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.562+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc4d0003e90 con 0x7fc4dc102850 2024-03-11T21:40:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 0x7fc4c4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4da59c700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 0x7fc4c4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(746..746 src has 87..746) v4 ==== 10205+0+0 (secure 0 0 0) 0x7fc4d008ef00 con 0x7fc4dc102850 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 10 client.41993.objecter ms_dispatch 0x7fc4dc001550 osd_map(746..746 src has 87..746) v4 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 3 client.41993.objecter handle_osd_map got epochs [746,746] > 0 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 3 client.41993.objecter handle_osd_map decoding full epoch 746 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 20 client.41993.objecter dump_active .. 0 homeless 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4cb7fe700 10 client.41993.objecter ms_handle_connect 0x7fc4c406f820 2024-03-11T21:40:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.566+0000 7fc4da59c700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 0x7fc4c4071ce0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00b2e0 tx=0x7fc4cc01a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:46.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.570+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc4d005b990 con 0x7fc4dc102850 2024-03-11T21:40:46.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: audit 2024-03-11T21:40:45.529953+0000 mon.a (mon.0) 3520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: cluster 2024-03-11T21:40:45.538699+0000 mon.a (mon.0) 3521 : cluster 0 osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: audit 2024-03-11T21:40:45.545333+0000 mon.c (mon.2) 1215 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: audit 2024-03-11T21:40:45.555059+0000 mon.a (mon.0) 3522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: cluster 2024-03-11T21:40:45.576780+0000 mgr.y (mgr.14550) 692 : cluster 0 pgmap v1156: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[29284]: audit 2024-03-11T21:40:45.990187+0000 mon.c (mon.2) 1216 : audit 0 from='client.? 172.21.15.143:0/3529877513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:46.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: audit 2024-03-11T21:40:45.529953+0000 mon.a (mon.0) 3520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:46.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: cluster 2024-03-11T21:40:45.538699+0000 mon.a (mon.0) 3521 : cluster 0 osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:40:46.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: audit 2024-03-11T21:40:45.545333+0000 mon.c (mon.2) 1215 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: audit 2024-03-11T21:40:45.555059+0000 mon.a (mon.0) 3522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: cluster 2024-03-11T21:40:45.576780+0000 mgr.y (mgr.14550) 692 : cluster 0 pgmap v1156: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:40:46.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:46 smithi143 bash[21512]: audit 2024-03-11T21:40:45.990187+0000 mon.c (mon.2) 1216 : audit 0 from='client.? 172.21.15.143:0/3529877513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:46.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:46.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:46.794+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc4dc1a6d20 con 0x7fc4dc102850 2024-03-11T21:40:46.786 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:46.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: audit 2024-03-11T21:40:45.529953+0000 mon.a (mon.0) 3520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: cluster 2024-03-11T21:40:45.538699+0000 mon.a (mon.0) 3521 : cluster 0 osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: audit 2024-03-11T21:40:45.545333+0000 mon.c (mon.2) 1215 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: audit 2024-03-11T21:40:45.555059+0000 mon.a (mon.0) 3522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: cluster 2024-03-11T21:40:45.576780+0000 mgr.y (mgr.14550) 692 : cluster 0 pgmap v1156: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:40:46.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:46 smithi203 bash[23654]: audit 2024-03-11T21:40:45.990187+0000 mon.c (mon.2) 1216 : audit 0 from='client.? 172.21.15.143:0/3529877513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:46.917 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:47.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:47.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:47.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:47.173 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:47.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:47.303 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:47.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:47.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:47.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:47.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.598+0000 7fc4cb7fe700 1 -- 172.21.15.143:0/67265126 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v748) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc4d005f3e0 con 0x7fc4dc102850 2024-03-11T21:40:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 10 librados: watch_flush enter 2024-03-11T21:40:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 10 librados: watch_flush exit 2024-03-11T21:40:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 20 client.41993.objecter shutdown clearing up homeless session... 2024-03-11T21:40:47.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 10 client.41993.objecter successfully canceled tick 2024-03-11T21:40:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 msgr2=0x7fc4c4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 0x7fc4c4071ce0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00b2e0 tx=0x7fc4cc01a040 comp rx=0 tx=0).stop 2024-03-11T21:40:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 msgr2=0x7fc4dc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:47.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000e9f0 tx=0x7fc4d000ed00 comp rx=0 tx=0).stop 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 shutdown_connections 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc4c406f820 0x7fc4c4071ce0 unknown :-1 s=CLOSED pgs=3442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc107ea0 0x7fc4dc074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4dc101e90 0x7fc4dc0740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 --2- 172.21.15.143:0/67265126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc102850 0x7fc4dc074620 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 >> 172.21.15.143:0/67265126 conn(0x7fc4dc078f40 msgr2=0x7fc4dc105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 shutdown_connections 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.606+0000 7fc4e0ad0700 1 -- 172.21.15.143:0/67265126 wait complete. 2024-03-11T21:40:47.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:47.610+0000 7fc4e0ad0700 1 librados: shutdown 2024-03-11T21:40:47.593 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:40:47.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: audit 2024-03-11T21:40:46.578593+0000 mon.a (mon.0) 3523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: cluster 2024-03-11T21:40:46.582300+0000 mon.a (mon.0) 3524 : cluster 0 osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: audit 2024-03-11T21:40:46.586558+0000 mon.c (mon.2) 1217 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: audit 2024-03-11T21:40:46.596356+0000 mon.a (mon.0) 3525 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: audit 2024-03-11T21:40:46.797046+0000 mon.c (mon.2) 1218 : audit 1 from='client.? 172.21.15.143:0/67265126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[29284]: audit 2024-03-11T21:40:46.797560+0000 mon.a (mon.0) 3526 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: audit 2024-03-11T21:40:46.578593+0000 mon.a (mon.0) 3523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: cluster 2024-03-11T21:40:46.582300+0000 mon.a (mon.0) 3524 : cluster 0 osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: audit 2024-03-11T21:40:46.586558+0000 mon.c (mon.2) 1217 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: audit 2024-03-11T21:40:46.596356+0000 mon.a (mon.0) 3525 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: audit 2024-03-11T21:40:46.797046+0000 mon.c (mon.2) 1218 : audit 1 from='client.? 172.21.15.143:0/67265126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:47 smithi143 bash[21512]: audit 2024-03-11T21:40:46.797560+0000 mon.a (mon.0) 3526 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:47.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: audit 2024-03-11T21:40:46.578593+0000 mon.a (mon.0) 3523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: cluster 2024-03-11T21:40:46.582300+0000 mon.a (mon.0) 3524 : cluster 0 osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: audit 2024-03-11T21:40:46.586558+0000 mon.c (mon.2) 1217 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: audit 2024-03-11T21:40:46.596356+0000 mon.a (mon.0) 3525 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]: dispatch 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: audit 2024-03-11T21:40:46.797046+0000 mon.c (mon.2) 1218 : audit 1 from='client.? 172.21.15.143:0/67265126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.806 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:47 smithi203 bash[23654]: audit 2024-03-11T21:40:46.797560+0000 mon.a (mon.0) 3526 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:47.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:47.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:48.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:48.077 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:48.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:48.206 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:48.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:48.339 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:48.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:48.469 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:48.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:48.600 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:48.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: cluster 2024-03-11T21:40:47.577613+0000 mgr.y (mgr.14550) 693 : cluster 0 pgmap v1158: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 252 B/s wr, 0 op/s 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: cluster 2024-03-11T21:40:47.590537+0000 mon.a (mon.0) 3527 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: cluster 2024-03-11T21:40:47.590592+0000 mon.a (mon.0) 3528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: audit 2024-03-11T21:40:47.592718+0000 mon.a (mon.0) 3529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]': finished 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: audit 2024-03-11T21:40:47.592946+0000 mon.a (mon.0) 3530 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[29284]: cluster 2024-03-11T21:40:47.600983+0000 mon.a (mon.0) 3531 : cluster 0 osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:40:48.716 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: cluster 2024-03-11T21:40:47.577613+0000 mgr.y (mgr.14550) 693 : cluster 0 pgmap v1158: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 252 B/s wr, 0 op/s 2024-03-11T21:40:48.717 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: cluster 2024-03-11T21:40:47.590537+0000 mon.a (mon.0) 3527 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:48.717 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: cluster 2024-03-11T21:40:47.590592+0000 mon.a (mon.0) 3528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:48.717 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: audit 2024-03-11T21:40:47.592718+0000 mon.a (mon.0) 3529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]': finished 2024-03-11T21:40:48.717 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: audit 2024-03-11T21:40:47.592946+0000 mon.a (mon.0) 3530 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:48.717 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:48 smithi143 bash[21512]: cluster 2024-03-11T21:40:47.600983+0000 mon.a (mon.0) 3531 : cluster 0 osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:40:48.730 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:48.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: cluster 2024-03-11T21:40:47.577613+0000 mgr.y (mgr.14550) 693 : cluster 0 pgmap v1158: 246 pgs: 20 creating+peering, 226 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 252 B/s wr, 0 op/s 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: cluster 2024-03-11T21:40:47.590537+0000 mon.a (mon.0) 3527 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: cluster 2024-03-11T21:40:47.590592+0000 mon.a (mon.0) 3528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: audit 2024-03-11T21:40:47.592718+0000 mon.a (mon.0) 3529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-124", "mode": "writeback"}]': finished 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: audit 2024-03-11T21:40:47.592946+0000 mon.a (mon.0) 3530 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:48.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[23654]: cluster 2024-03-11T21:40:47.600983+0000 mon.a (mon.0) 3531 : cluster 0 osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:40:48.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:48.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:48.991 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:49.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:49.121 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:49.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:49.278 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:48 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:49.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:49.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.506 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:49.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.632 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:49.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.763 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:49.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[21512]: cluster 2024-03-11T21:40:48.624413+0000 mon.a (mon.0) 3532 : cluster 0 osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[21512]: audit 2024-03-11T21:40:48.733476+0000 mon.c (mon.2) 1219 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[21512]: audit 2024-03-11T21:40:48.734023+0000 mon.a (mon.0) 3533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[21512]: audit 2024-03-11T21:40:48.966763+0000 mgr.y (mgr.14550) 694 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[29284]: cluster 2024-03-11T21:40:48.624413+0000 mon.a (mon.0) 3532 : cluster 0 osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[29284]: audit 2024-03-11T21:40:48.733476+0000 mon.c (mon.2) 1219 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[29284]: audit 2024-03-11T21:40:48.734023+0000 mon.a (mon.0) 3533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:49.801 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:49 smithi143 bash[29284]: audit 2024-03-11T21:40:48.966763+0000 mgr.y (mgr.14550) 694 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:49.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.893 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:49.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.023 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:50.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:49 smithi203 bash[23654]: cluster 2024-03-11T21:40:48.624413+0000 mon.a (mon.0) 3532 : cluster 0 osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:40:50.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:49 smithi203 bash[23654]: audit 2024-03-11T21:40:48.733476+0000 mon.c (mon.2) 1219 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:50.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:49 smithi203 bash[23654]: audit 2024-03-11T21:40:48.734023+0000 mon.a (mon.0) 3533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:50.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:49 smithi203 bash[23654]: audit 2024-03-11T21:40:48.966763+0000 mgr.y (mgr.14550) 694 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:50.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.147 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:50.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:50.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:50.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:50.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:50.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.536 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:50.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.649 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[21512]: cluster 2024-03-11T21:40:49.578274+0000 mgr.y (mgr.14550) 695 : cluster 0 pgmap v1161: 246 pgs: 246 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:40:50.649 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[21512]: audit 2024-03-11T21:40:49.631052+0000 mon.a (mon.0) 3534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:50.649 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[21512]: cluster 2024-03-11T21:40:49.641343+0000 mon.a (mon.0) 3535 : cluster 0 osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:40:50.649 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[21512]: audit 2024-03-11T21:40:49.642276+0000 mon.c (mon.2) 1220 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.649 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[21512]: audit 2024-03-11T21:40:49.642820+0000 mon.a (mon.0) 3536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[29284]: cluster 2024-03-11T21:40:49.578274+0000 mgr.y (mgr.14550) 695 : cluster 0 pgmap v1161: 246 pgs: 246 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:40:50.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[29284]: audit 2024-03-11T21:40:49.631052+0000 mon.a (mon.0) 3534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:50.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[29284]: cluster 2024-03-11T21:40:49.641343+0000 mon.a (mon.0) 3535 : cluster 0 osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:40:50.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[29284]: audit 2024-03-11T21:40:49.642276+0000 mon.c (mon.2) 1220 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.650 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:50 smithi143 bash[29284]: audit 2024-03-11T21:40:49.642820+0000 mon.a (mon.0) 3536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:50.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:50.793 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:50.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:50.908 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:50 smithi203 bash[23654]: cluster 2024-03-11T21:40:49.578274+0000 mgr.y (mgr.14550) 695 : cluster 0 pgmap v1161: 246 pgs: 246 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:40:50.908 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:50 smithi203 bash[23654]: audit 2024-03-11T21:40:49.631052+0000 mon.a (mon.0) 3534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:50.908 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:50 smithi203 bash[23654]: cluster 2024-03-11T21:40:49.641343+0000 mon.a (mon.0) 3535 : cluster 0 osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:40:50.908 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:50 smithi203 bash[23654]: audit 2024-03-11T21:40:49.642276+0000 mon.c (mon.2) 1220 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.908 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:50 smithi203 bash[23654]: audit 2024-03-11T21:40:49.642820+0000 mon.a (mon.0) 3536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]: dispatch 2024-03-11T21:40:50.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:51.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:51.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:51.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.181 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:51.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:51.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:51.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.442 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:51.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:51.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:40:51.566 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:51.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:51.694 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:51.797 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:51.815 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:51.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:51.942 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:52.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:52.073 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:52.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.201 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.202 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:51 smithi203 bash[23654]: cluster 2024-03-11T21:40:50.631500+0000 mon.a (mon.0) 3537 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:52.202 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:51 smithi203 bash[23654]: audit 2024-03-11T21:40:50.633504+0000 mon.a (mon.0) 3538 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:52.202 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:51 smithi203 bash[23654]: cluster 2024-03-11T21:40:50.640098+0000 mon.a (mon.0) 3539 : cluster 0 osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:40:52.203 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:51 smithi203 bash[23654]: cluster 2024-03-11T21:40:51.509867+0000 mon.a (mon.0) 3540 : cluster 0 osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:40:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:52.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:52.331 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:52.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:40:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[21512]: cluster 2024-03-11T21:40:50.631500+0000 mon.a (mon.0) 3537 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[21512]: audit 2024-03-11T21:40:50.633504+0000 mon.a (mon.0) 3538 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[21512]: cluster 2024-03-11T21:40:50.640098+0000 mon.a (mon.0) 3539 : cluster 0 osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[21512]: cluster 2024-03-11T21:40:51.509867+0000 mon.a (mon.0) 3540 : cluster 0 osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[29284]: cluster 2024-03-11T21:40:50.631500+0000 mon.a (mon.0) 3537 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:52.442 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[29284]: audit 2024-03-11T21:40:50.633504+0000 mon.a (mon.0) 3538 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-124"}]': finished 2024-03-11T21:40:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[29284]: cluster 2024-03-11T21:40:50.640098+0000 mon.a (mon.0) 3539 : cluster 0 osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:40:52.443 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:51 smithi143 bash[29284]: cluster 2024-03-11T21:40:51.509867+0000 mon.a (mon.0) 3540 : cluster 0 osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:40:52.462 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:52.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:52.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:52.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:52.721 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:52.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.851 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:52.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:52.981 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:53.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:53.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:52 smithi203 bash[23654]: cluster 2024-03-11T21:40:51.578849+0000 mgr.y (mgr.14550) 696 : cluster 0 pgmap v1165: 214 pgs: 214 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:40:53.095 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:52 smithi203 bash[23654]: audit 2024-03-11T21:40:52.173724+0000 mon.c (mon.2) 1221 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.095 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:52 smithi203 bash[23654]: cluster 2024-03-11T21:40:52.537200+0000 mon.a (mon.0) 3541 : cluster 0 osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:40:53.095 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:52 smithi203 bash[23654]: audit 2024-03-11T21:40:52.543289+0000 mon.c (mon.2) 1222 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.095 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:52 smithi203 bash[23654]: audit 2024-03-11T21:40:52.548382+0000 mon.a (mon.0) 3542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.108 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:53.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:53.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:53.367 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:53.399 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:40:53.409 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21512]: cluster 2024-03-11T21:40:51.578849+0000 mgr.y (mgr.14550) 696 : cluster 0 pgmap v1165: 214 pgs: 214 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21512]: audit 2024-03-11T21:40:52.173724+0000 mon.c (mon.2) 1221 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21512]: cluster 2024-03-11T21:40:52.537200+0000 mon.a (mon.0) 3541 : cluster 0 osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21512]: audit 2024-03-11T21:40:52.543289+0000 mon.c (mon.2) 1222 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[21512]: audit 2024-03-11T21:40:52.548382+0000 mon.a (mon.0) 3542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[29284]: cluster 2024-03-11T21:40:51.578849+0000 mgr.y (mgr.14550) 696 : cluster 0 pgmap v1165: 214 pgs: 214 active+clean; 583 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[29284]: audit 2024-03-11T21:40:52.173724+0000 mon.c (mon.2) 1221 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[29284]: cluster 2024-03-11T21:40:52.537200+0000 mon.a (mon.0) 3541 : cluster 0 osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[29284]: audit 2024-03-11T21:40:52.543289+0000 mon.c (mon.2) 1222 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.410 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:52 smithi143 bash[29284]: audit 2024-03-11T21:40:52.548382+0000 mon.a (mon.0) 3542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:53.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:53.492 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:53.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.619 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:53.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.728 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:40:53.728 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:40:53.728 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:40:53.728 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:40:53.748 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:53.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:53.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:53.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:54.007 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:54.110 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:54.137 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:54.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.266 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:54.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/1322373545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4104c20 msgr2=0x7f5fe4108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:54.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/1322373545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4104c20 0x7f5fe4108c70 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099c0 tx=0x7f5fd8009cd0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/1322373545 shutdown_connections 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/1322373545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4109350 0x7f5fe4110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/1322373545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104270 0x7f5fe4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/1322373545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4104c20 0x7f5fe4108c70 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/1322373545 >> 172.21.15.143:0/1322373545 conn(0x7f5fe40ffac0 msgr2=0x7f5fe4101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/1322373545 shutdown_connections 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/1322373545 wait complete. 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 librados: starting msgr at 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 librados: starting objecter 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 Processor -- start 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- start start 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 librados: setting wanted keys 2024-03-11T21:40:54.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 librados: calling monclient init 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104c20 0x7f5fe419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4113810 con 0x7f5fe4109350 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4113690 con 0x7f5fe4104c20 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4113990 con 0x7f5fe4104270 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:54.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43500/0 (socket says 172.21.15.143:43500) 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51394/0 (socket says 172.21.15.143:51394) 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe3fff700 1 -- 172.21.15.143:0/2143792849 learned_addr learned my addr 172.21.15.143:0/2143792849 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe2ffd700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104c20 0x7f5fe419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 -- 172.21.15.143:0/2143792849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104c20 msgr2=0x7f5fe419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104c20 0x7f5fe419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 -- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 msgr2=0x7f5fe419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 -- 172.21.15.143:0/2143792849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe419ede0 con 0x7f5fe4104270 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe3fff700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe37fe700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400c970 tx=0x7f5fd400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe0ff9700 10 client.?.objecter ms_handle_connect 0x7f5fe4104270 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe0ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400ce60 con 0x7f5fe4104270 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe419f0d0 con 0x7f5fe4104270 2024-03-11T21:40:54.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5fd4004830 con 0x7f5fe4104270 2024-03-11T21:40:54.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 10 client.41999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:40:54.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4005310 con 0x7f5fe4104270 2024-03-11T21:40:54.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe41a6950 con 0x7f5fe4104270 2024-03-11T21:40:54.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.354+0000 7f5fe9ea1700 1 librados: init done 2024-03-11T21:40:54.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe404faa0 con 0x7f5fe4104270 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5fd4007b00 con 0x7f5fe4104270 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 0x7f5fcc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(754..754 src has 87..754) v4 ==== 10266+0+0 (secure 0 0 0) 0x7f5fd408e900 con 0x7f5fe4104270 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 10 client.41999.objecter ms_dispatch 0x7f5fe4001550 osd_map(754..754 src has 87..754) v4 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 3 client.41999.objecter handle_osd_map got epochs [754,754] > 0 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 3 client.41999.objecter handle_osd_map decoding full epoch 754 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe2ffd700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 0x7f5fcc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 20 client.41999.objecter dump_active .. 0 homeless 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe2ffd700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 0x7f5fcc071ce0 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8000c00 tx=0x7f5fd8005940 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:40:54.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.358+0000 7f5fe0ff9700 10 client.41999.objecter ms_handle_connect 0x7f5fcc06f820 2024-03-11T21:40:54.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.362+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5fd4014030 con 0x7f5fe4104270 2024-03-11T21:40:54.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:54.397 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:54.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:54.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:54.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:54.594+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f5fe41a6c40 con 0x7f5fe4104270 2024-03-11T21:40:54.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:54.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:54 smithi203 bash[23654]: audit 2024-03-11T21:40:53.531450+0000 mon.a (mon.0) 3543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:54.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:54 smithi203 bash[23654]: cluster 2024-03-11T21:40:53.535103+0000 mon.a (mon.0) 3544 : cluster 0 osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:40:54.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:54 smithi203 bash[23654]: audit 2024-03-11T21:40:53.538631+0000 mon.c (mon.2) 1223 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:54 smithi203 bash[23654]: audit 2024-03-11T21:40:53.539169+0000 mon.a (mon.0) 3545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:54 smithi203 bash[23654]: cluster 2024-03-11T21:40:53.579398+0000 mgr.y (mgr.14550) 697 : cluster 0 pgmap v1168: 246 pgs: 26 creating+peering, 6 unknown, 214 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 259 B/s rd, 0 op/s 2024-03-11T21:40:54.786 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:54.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[21512]: audit 2024-03-11T21:40:53.531450+0000 mon.a (mon.0) 3543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[21512]: cluster 2024-03-11T21:40:53.535103+0000 mon.a (mon.0) 3544 : cluster 0 osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[21512]: audit 2024-03-11T21:40:53.538631+0000 mon.c (mon.2) 1223 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[21512]: audit 2024-03-11T21:40:53.539169+0000 mon.a (mon.0) 3545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[21512]: cluster 2024-03-11T21:40:53.579398+0000 mgr.y (mgr.14550) 697 : cluster 0 pgmap v1168: 246 pgs: 26 creating+peering, 6 unknown, 214 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 259 B/s rd, 0 op/s 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[29284]: audit 2024-03-11T21:40:53.531450+0000 mon.a (mon.0) 3543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[29284]: cluster 2024-03-11T21:40:53.535103+0000 mon.a (mon.0) 3544 : cluster 0 osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:40:54.900 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[29284]: audit 2024-03-11T21:40:53.538631+0000 mon.c (mon.2) 1223 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[29284]: audit 2024-03-11T21:40:53.539169+0000 mon.a (mon.0) 3545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:40:54.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:54 smithi143 bash[29284]: cluster 2024-03-11T21:40:53.579398+0000 mgr.y (mgr.14550) 697 : cluster 0 pgmap v1168: 246 pgs: 26 creating+peering, 6 unknown, 214 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 259 B/s rd, 0 op/s 2024-03-11T21:40:54.915 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:54.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:40:55.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:55.041 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:55.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.167 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:55.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:55.294 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:55.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.421 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:55.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:55.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.546+0000 7f5fe0ff9700 1 -- 172.21.15.143:0/2143792849 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v756) v1 ==== 113+0+0 (secure 0 0 0) 0x7f5fd405b2d0 con 0x7f5fe4104270 2024-03-11T21:40:55.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 10 librados: watch_flush enter 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 10 librados: watch_flush exit 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 20 client.41999.objecter shutdown clearing up homeless session... 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 10 client.41999.objecter successfully canceled tick 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 msgr2=0x7f5fcc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 0x7f5fcc071ce0 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8000c00 tx=0x7f5fd8005940 comp rx=0 tx=0).stop 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 msgr2=0x7f5fe4199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:40:55.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.550+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400c970 tx=0x7f5fd400cc80 comp rx=0 tx=0).stop 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 shutdown_connections 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5fcc06f820 0x7f5fcc071ce0 unknown :-1 s=CLOSED pgs=3443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe4109350 0x7f5fe419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe4104270 0x7f5fe4199dd0 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 --2- 172.21.15.143:0/2143792849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe4104c20 0x7f5fe419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 >> 172.21.15.143:0/2143792849 conn(0x7f5fe40ffac0 msgr2=0x7f5fe4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 shutdown_connections 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 -- 172.21.15.143:0/2143792849 wait complete. 2024-03-11T21:40:55.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:40:55.554+0000 7f5fe9ea1700 1 librados: shutdown 2024-03-11T21:40:55.540 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:40:55.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:55.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:55.676 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:54.540025+0000 mon.a (mon.0) 3546 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: cluster 2024-03-11T21:40:54.546826+0000 mon.a (mon.0) 3547 : cluster 0 osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:54.549193+0000 mon.c (mon.2) 1224 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:54.550197+0000 mon.a (mon.0) 3548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:54.596822+0000 mon.c (mon.2) 1225 : audit 1 from='client.? 172.21.15.143:0/2143792849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:54.597181+0000 mon.a (mon.0) 3549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[21512]: audit 2024-03-11T21:40:55.035889+0000 mon.c (mon.2) 1226 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:54.540025+0000 mon.a (mon.0) 3546 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: cluster 2024-03-11T21:40:54.546826+0000 mon.a (mon.0) 3547 : cluster 0 osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:54.549193+0000 mon.c (mon.2) 1224 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:54.550197+0000 mon.a (mon.0) 3548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:54.596822+0000 mon.c (mon.2) 1225 : audit 1 from='client.? 172.21.15.143:0/2143792849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:54.597181+0000 mon.a (mon.0) 3549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:55 smithi143 bash[29284]: audit 2024-03-11T21:40:55.035889+0000 mon.c (mon.2) 1226 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:55.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:55.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:55.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:54.540025+0000 mon.a (mon.0) 3546 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: cluster 2024-03-11T21:40:54.546826+0000 mon.a (mon.0) 3547 : cluster 0 osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:54.549193+0000 mon.c (mon.2) 1224 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:54.550197+0000 mon.a (mon.0) 3548 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:54.596822+0000 mon.c (mon.2) 1225 : audit 1 from='client.? 172.21.15.143:0/2143792849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:54.597181+0000 mon.a (mon.0) 3549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:40:55.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:55 smithi203 bash[23654]: audit 2024-03-11T21:40:55.035889+0000 mon.c (mon.2) 1226 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:55.933 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:56.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.065 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:56.168 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.196 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:56.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:56.327 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:56.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:56.458 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:56.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:40:56.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:56.590 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:56.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:56.714 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:56.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.543119+0000 mon.a (mon.0) 3550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:56.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.543210+0000 mon.a (mon.0) 3551 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:40:56.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: cluster 2024-03-11T21:40:55.549877+0000 mon.a (mon.0) 3552 : cluster 0 osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.553619+0000 mon.c (mon.2) 1227 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.562481+0000 mon.a (mon.0) 3553 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: cluster 2024-03-11T21:40:55.579815+0000 mgr.y (mgr.14550) 698 : cluster 0 pgmap v1171: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.713539+0000 mon.c (mon.2) 1228 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.714565+0000 mon.c (mon.2) 1229 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:56 smithi203 bash[23654]: audit 2024-03-11T21:40:55.723073+0000 mon.a (mon.0) 3554 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:56.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:56.840 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:56.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.543119+0000 mon.a (mon.0) 3550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:56.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.543210+0000 mon.a (mon.0) 3551 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:40:56.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: cluster 2024-03-11T21:40:55.549877+0000 mon.a (mon.0) 3552 : cluster 0 osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.553619+0000 mon.c (mon.2) 1227 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.562481+0000 mon.a (mon.0) 3553 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: cluster 2024-03-11T21:40:55.579815+0000 mgr.y (mgr.14550) 698 : cluster 0 pgmap v1171: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.713539+0000 mon.c (mon.2) 1228 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.714565+0000 mon.c (mon.2) 1229 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[21512]: audit 2024-03-11T21:40:55.723073+0000 mon.a (mon.0) 3554 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.543119+0000 mon.a (mon.0) 3550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.543210+0000 mon.a (mon.0) 3551 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: cluster 2024-03-11T21:40:55.549877+0000 mon.a (mon.0) 3552 : cluster 0 osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.553619+0000 mon.c (mon.2) 1227 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.562481+0000 mon.a (mon.0) 3553 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]: dispatch 2024-03-11T21:40:56.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: cluster 2024-03-11T21:40:55.579815+0000 mgr.y (mgr.14550) 698 : cluster 0 pgmap v1171: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:40:56.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.713539+0000 mon.c (mon.2) 1228 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:56.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.714565+0000 mon.c (mon.2) 1229 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:56.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:56 smithi143 bash[29284]: audit 2024-03-11T21:40:55.723073+0000 mon.a (mon.0) 3554 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:40:56.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.965 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:57.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.090 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:57.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:57.221 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:57.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:57.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:57.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:57.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:57.612 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:57.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[29284]: cluster 2024-03-11T21:40:56.555123+0000 mon.a (mon.0) 3555 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:57.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[29284]: audit 2024-03-11T21:40:56.565237+0000 mon.a (mon.0) 3556 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]': finished 2024-03-11T21:40:57.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[29284]: cluster 2024-03-11T21:40:56.574690+0000 mon.a (mon.0) 3557 : cluster 0 osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[29284]: audit 2024-03-11T21:40:56.684974+0000 mon.c (mon.2) 1230 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[29284]: audit 2024-03-11T21:40:56.685348+0000 mon.a (mon.0) 3558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[21512]: cluster 2024-03-11T21:40:56.555123+0000 mon.a (mon.0) 3555 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[21512]: audit 2024-03-11T21:40:56.565237+0000 mon.a (mon.0) 3556 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]': finished 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[21512]: cluster 2024-03-11T21:40:56.574690+0000 mon.a (mon.0) 3557 : cluster 0 osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[21512]: audit 2024-03-11T21:40:56.684974+0000 mon.c (mon.2) 1230 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:57 smithi143 bash[21512]: audit 2024-03-11T21:40:56.685348+0000 mon.a (mon.0) 3558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:57.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:57.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:57 smithi203 bash[23654]: cluster 2024-03-11T21:40:56.555123+0000 mon.a (mon.0) 3555 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:40:57.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:57 smithi203 bash[23654]: audit 2024-03-11T21:40:56.565237+0000 mon.a (mon.0) 3556 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-126", "mode": "writeback"}]': finished 2024-03-11T21:40:57.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:57 smithi203 bash[23654]: cluster 2024-03-11T21:40:56.574690+0000 mon.a (mon.0) 3557 : cluster 0 osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:40:57.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:57 smithi203 bash[23654]: audit 2024-03-11T21:40:56.684974+0000 mon.c (mon.2) 1230 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:57 smithi203 bash[23654]: audit 2024-03-11T21:40:56.685348+0000 mon.a (mon.0) 3558 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:40:57.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.872 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:57.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:58.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:58.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:58.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:58.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:58.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.390 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:58.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.519 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:58.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:58.650 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:58.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[29284]: audit 2024-03-11T21:40:57.577621+0000 mon.a (mon.0) 3559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:58.692 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[29284]: cluster 2024-03-11T21:40:57.580867+0000 mgr.y (mgr.14550) 699 : cluster 0 pgmap v1173: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[29284]: audit 2024-03-11T21:40:57.586693+0000 mon.c (mon.2) 1231 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[29284]: cluster 2024-03-11T21:40:57.588842+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[29284]: audit 2024-03-11T21:40:57.594615+0000 mon.a (mon.0) 3561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[21512]: audit 2024-03-11T21:40:57.577621+0000 mon.a (mon.0) 3559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[21512]: cluster 2024-03-11T21:40:57.580867+0000 mgr.y (mgr.14550) 699 : cluster 0 pgmap v1173: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[21512]: audit 2024-03-11T21:40:57.586693+0000 mon.c (mon.2) 1231 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[21512]: cluster 2024-03-11T21:40:57.588842+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:40:58.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:58 smithi143 bash[21512]: audit 2024-03-11T21:40:57.594615+0000 mon.a (mon.0) 3561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:58.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:58.895 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[23654]: audit 2024-03-11T21:40:57.577621+0000 mon.a (mon.0) 3559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:40:58.895 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[23654]: cluster 2024-03-11T21:40:57.580867+0000 mgr.y (mgr.14550) 699 : cluster 0 pgmap v1173: 246 pgs: 26 creating+peering, 220 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:40:58.895 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[23654]: audit 2024-03-11T21:40:57.586693+0000 mon.c (mon.2) 1231 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[23654]: cluster 2024-03-11T21:40:57.588842+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:40:58.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[23654]: audit 2024-03-11T21:40:57.594615+0000 mon.a (mon.0) 3561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]: dispatch 2024-03-11T21:40:58.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:59.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.042 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:59.042 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:59.043 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:59.146 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.175 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:59.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:59.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:59.278 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:40:58 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:40:59.305 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:59.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:59.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:59.590 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9061 ms) 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6041 ms) 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4915 ms) 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25032 ms) 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.591 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14120 ms) 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11931 ms) 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000012 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11986 ms) 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5975 ms) 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (562881 ms total) 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-11T21:40:59.592 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1140 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10798 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18082 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (7 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30027 ms total) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6977 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8061 ms) 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrub... 2024-03-11T21:40:59.593 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24977 ms) 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10086 ms) 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7919 ms) 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7976 ms) 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10000 ms) 2024-03-11T21:40:59.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-11T21:40:59.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:59.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:59.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.813 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: cluster 2024-03-11T21:40:58.577875+0000 mon.a (mon.0) 3562 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:58.579659+0000 mon.a (mon.0) 3563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: cluster 2024-03-11T21:40:58.584073+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:58.978115+0000 mgr.y (mgr.14550) 700 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.162337+0000 mon.c (mon.2) 1232 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.162734+0000 mon.c (mon.2) 1233 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.163067+0000 mon.a (mon.0) 3565 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.163195+0000 mon.c (mon.2) 1234 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.163701+0000 mon.a (mon.0) 3566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:40:59 smithi203 bash[23654]: audit 2024-03-11T21:40:59.164119+0000 mon.a (mon.0) 3567 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:59.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:59.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: cluster 2024-03-11T21:40:58.577875+0000 mon.a (mon.0) 3562 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:59.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:58.579659+0000 mon.a (mon.0) 3563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:59.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: cluster 2024-03-11T21:40:58.584073+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:40:59.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:58.978115+0000 mgr.y (mgr.14550) 700 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.162337+0000 mon.c (mon.2) 1232 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.162734+0000 mon.c (mon.2) 1233 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.163067+0000 mon.a (mon.0) 3565 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.163195+0000 mon.c (mon.2) 1234 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.163701+0000 mon.a (mon.0) 3566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[21512]: audit 2024-03-11T21:40:59.164119+0000 mon.a (mon.0) 3567 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: cluster 2024-03-11T21:40:58.577875+0000 mon.a (mon.0) 3562 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:58.579659+0000 mon.a (mon.0) 3563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-126"}]': finished 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: cluster 2024-03-11T21:40:58.584073+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:58.978115+0000 mgr.y (mgr.14550) 700 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.162337+0000 mon.c (mon.2) 1232 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.162734+0000 mon.c (mon.2) 1233 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.163067+0000 mon.a (mon.0) 3565 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.163195+0000 mon.c (mon.2) 1234 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.163701+0000 mon.a (mon.0) 3566 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]: dispatch 2024-03-11T21:40:59.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:40:59 smithi143 bash[29284]: audit 2024-03-11T21:40:59.164119+0000 mon.a (mon.0) 3567 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]: dispatch 2024-03-11T21:40:59.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:00.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:00.064 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:00.083 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:00.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:00.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.340 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:00.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:00.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.603 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:00.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:00.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:59.582020+0000 mgr.y (mgr.14550) 701 : cluster 0 pgmap v1176: 246 pgs: 246 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 2 op/s 2024-03-11T21:41:00.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:59.590218+0000 mon.a (mon.0) 3568 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:00.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: audit 2024-03-11T21:40:59.600140+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-11T21:41:00.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: audit 2024-03-11T21:40:59.600365+0000 mon.a (mon.0) 3570 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]': finished 2024-03-11T21:41:00.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: audit 2024-03-11T21:40:59.600497+0000 mon.a (mon.0) 3571 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]': finished 2024-03-11T21:41:00.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:00 smithi203 bash[23654]: cluster 2024-03-11T21:40:59.608210+0000 mon.a (mon.0) 3572 : cluster 0 osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:41:00.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:00.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:00.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:59.582020+0000 mgr.y (mgr.14550) 701 : cluster 0 pgmap v1176: 246 pgs: 246 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 2 op/s 2024-03-11T21:41:00.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:59.590218+0000 mon.a (mon.0) 3568 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: audit 2024-03-11T21:40:59.600140+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: audit 2024-03-11T21:40:59.600365+0000 mon.a (mon.0) 3570 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: audit 2024-03-11T21:40:59.600497+0000 mon.a (mon.0) 3571 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[29284]: cluster 2024-03-11T21:40:59.608210+0000 mon.a (mon.0) 3572 : cluster 0 osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:59.582020+0000 mgr.y (mgr.14550) 701 : cluster 0 pgmap v1176: 246 pgs: 246 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 2 op/s 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:59.590218+0000 mon.a (mon.0) 3568 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: audit 2024-03-11T21:40:59.600140+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: audit 2024-03-11T21:40:59.600365+0000 mon.a (mon.0) 3570 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.12", "id": [2, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: audit 2024-03-11T21:40:59.600497+0000 mon.a (mon.0) 3571 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1b", "id": [5, 3]}]': finished 2024-03-11T21:41:00.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:00 smithi143 bash[21512]: cluster 2024-03-11T21:40:59.608210+0000 mon.a (mon.0) 3572 : cluster 0 osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:41:00.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:00.992 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:01.092 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:01.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:01.123 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:01.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:01.251 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:01.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:01.382 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:01.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:01.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:01.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:01.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: cluster 2024-03-11T21:41:00.619851+0000 mon.a (mon.0) 3573 : cluster 0 osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:41:01.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: audit 2024-03-11T21:41:00.621571+0000 mon.c (mon.2) 1235 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.639 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: audit 2024-03-11T21:41:00.624181+0000 mon.a (mon.0) 3574 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.640 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: audit 2024-03-11T21:41:01.514352+0000 mon.a (mon.0) 3575 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:01.640 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: cluster 2024-03-11T21:41:01.524069+0000 mon.a (mon.0) 3576 : cluster 0 osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:41:01.640 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: audit 2024-03-11T21:41:01.545619+0000 mon.c (mon.2) 1236 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:01.640 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:01 smithi203 bash[23654]: audit 2024-03-11T21:41:01.546233+0000 mon.a (mon.0) 3577 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:01.654 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:01.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: cluster 2024-03-11T21:41:00.619851+0000 mon.a (mon.0) 3573 : cluster 0 osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:41:01.751 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: audit 2024-03-11T21:41:00.621571+0000 mon.c (mon.2) 1235 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:01.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:01.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:01.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: audit 2024-03-11T21:41:00.624181+0000 mon.a (mon.0) 3574 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: audit 2024-03-11T21:41:01.514352+0000 mon.a (mon.0) 3575 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: cluster 2024-03-11T21:41:01.524069+0000 mon.a (mon.0) 3576 : cluster 0 osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:41:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: audit 2024-03-11T21:41:01.545619+0000 mon.c (mon.2) 1236 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:01.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[21512]: audit 2024-03-11T21:41:01.546233+0000 mon.a (mon.0) 3577 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: cluster 2024-03-11T21:41:00.619851+0000 mon.a (mon.0) 3573 : cluster 0 osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: audit 2024-03-11T21:41:00.621571+0000 mon.c (mon.2) 1235 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: audit 2024-03-11T21:41:00.624181+0000 mon.a (mon.0) 3574 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: audit 2024-03-11T21:41:01.514352+0000 mon.a (mon.0) 3575 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: cluster 2024-03-11T21:41:01.524069+0000 mon.a (mon.0) 3576 : cluster 0 osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: audit 2024-03-11T21:41:01.545619+0000 mon.c (mon.2) 1236 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:01.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:01 smithi143 bash[29284]: audit 2024-03-11T21:41:01.546233+0000 mon.a (mon.0) 3577 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:02.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:02.044 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:02.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:02.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:02.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:02.306 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:02.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:02.442 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:02.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:02.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:02.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:02.697 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:02.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:02 smithi203 bash[23654]: cluster 2024-03-11T21:41:01.582539+0000 mgr.y (mgr.14550) 702 : cluster 0 pgmap v1180: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:41:02.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:02 smithi203 bash[23654]: audit 2024-03-11T21:41:02.545962+0000 mon.a (mon.0) 3578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:02.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:02 smithi203 bash[23654]: cluster 2024-03-11T21:41:02.554291+0000 mon.a (mon.0) 3579 : cluster 0 osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:41:02.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:02 smithi203 bash[23654]: audit 2024-03-11T21:41:02.557976+0000 mon.c (mon.2) 1237 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:02 smithi203 bash[23654]: audit 2024-03-11T21:41:02.558990+0000 mon.a (mon.0) 3580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.800 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:02.828 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:02.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[29284]: cluster 2024-03-11T21:41:01.582539+0000 mgr.y (mgr.14550) 702 : cluster 0 pgmap v1180: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:41:02.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[29284]: audit 2024-03-11T21:41:02.545962+0000 mon.a (mon.0) 3578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:02.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[29284]: cluster 2024-03-11T21:41:02.554291+0000 mon.a (mon.0) 3579 : cluster 0 osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[29284]: audit 2024-03-11T21:41:02.557976+0000 mon.c (mon.2) 1237 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[29284]: audit 2024-03-11T21:41:02.558990+0000 mon.a (mon.0) 3580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21512]: cluster 2024-03-11T21:41:01.582539+0000 mgr.y (mgr.14550) 702 : cluster 0 pgmap v1180: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21512]: audit 2024-03-11T21:41:02.545962+0000 mon.a (mon.0) 3578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21512]: cluster 2024-03-11T21:41:02.554291+0000 mon.a (mon.0) 3579 : cluster 0 osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21512]: audit 2024-03-11T21:41:02.557976+0000 mon.c (mon.2) 1237 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:02 smithi143 bash[21512]: audit 2024-03-11T21:41:02.558990+0000 mon.a (mon.0) 3580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:02.958 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:03.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:03.088 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:03.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:03.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:03.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:03.350 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:03.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:03.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:03.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.612 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.692 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: cluster 2024-03-11T21:41:02.788192+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: cluster 2024-03-11T21:41:02.788572+0000 osd.6 (osd.6) 330 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: audit 2024-03-11T21:41:03.548928+0000 mon.a (mon.0) 3581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: audit 2024-03-11T21:41:03.559266+0000 mon.c (mon.2) 1238 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: cluster 2024-03-11T21:41:03.561287+0000 mon.a (mon.0) 3582 : cluster 0 osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[21512]: audit 2024-03-11T21:41:03.563357+0000 mon.a (mon.0) 3583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: cluster 2024-03-11T21:41:02.788192+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: cluster 2024-03-11T21:41:02.788572+0000 osd.6 (osd.6) 330 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: audit 2024-03-11T21:41:03.548928+0000 mon.a (mon.0) 3581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:03.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: audit 2024-03-11T21:41:03.559266+0000 mon.c (mon.2) 1238 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: cluster 2024-03-11T21:41:03.561287+0000 mon.a (mon.0) 3582 : cluster 0 osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:41:03.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:03 smithi143 bash[29284]: audit 2024-03-11T21:41:03.563357+0000 mon.a (mon.0) 3583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: cluster 2024-03-11T21:41:02.788192+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:03.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: cluster 2024-03-11T21:41:02.788572+0000 osd.6 (osd.6) 330 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:03.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: audit 2024-03-11T21:41:03.548928+0000 mon.a (mon.0) 3581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:03.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: audit 2024-03-11T21:41:03.559266+0000 mon.c (mon.2) 1238 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: cluster 2024-03-11T21:41:03.561287+0000 mon.a (mon.0) 3582 : cluster 0 osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:41:03.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:03 smithi203 bash[23654]: audit 2024-03-11T21:41:03.563357+0000 mon.a (mon.0) 3583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]: dispatch 2024-03-11T21:41:03.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:03.873 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:03.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:04.001 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:04.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:04.126 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:04.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:04.258 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:04.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:04.389 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:04.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:04.516 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:04.619 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:04.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:04.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:04.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[29284]: cluster 2024-03-11T21:41:03.583300+0000 mgr.y (mgr.14550) 703 : cluster 0 pgmap v1183: 246 pgs: 9 unknown, 237 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:41:04.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[29284]: cluster 2024-03-11T21:41:04.549758+0000 mon.a (mon.0) 3584 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:04.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[29284]: audit 2024-03-11T21:41:04.557182+0000 mon.a (mon.0) 3585 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]': finished 2024-03-11T21:41:04.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[29284]: cluster 2024-03-11T21:41:04.563785+0000 mon.a (mon.0) 3586 : cluster 0 osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:41:04.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[21512]: cluster 2024-03-11T21:41:03.583300+0000 mgr.y (mgr.14550) 703 : cluster 0 pgmap v1183: 246 pgs: 9 unknown, 237 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:41:04.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[21512]: cluster 2024-03-11T21:41:04.549758+0000 mon.a (mon.0) 3584 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:04.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[21512]: audit 2024-03-11T21:41:04.557182+0000 mon.a (mon.0) 3585 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]': finished 2024-03-11T21:41:04.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:04 smithi143 bash[21512]: cluster 2024-03-11T21:41:04.563785+0000 mon.a (mon.0) 3586 : cluster 0 osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:41:04.769 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:04.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:04.884 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:04 smithi203 bash[23654]: cluster 2024-03-11T21:41:03.583300+0000 mgr.y (mgr.14550) 703 : cluster 0 pgmap v1183: 246 pgs: 9 unknown, 237 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:41:04.884 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:04 smithi203 bash[23654]: cluster 2024-03-11T21:41:04.549758+0000 mon.a (mon.0) 3584 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:04.884 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:04 smithi203 bash[23654]: audit 2024-03-11T21:41:04.557182+0000 mon.a (mon.0) 3585 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-128", "mode": "writeback"}]': finished 2024-03-11T21:41:04.884 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:04 smithi203 bash[23654]: cluster 2024-03-11T21:41:04.563785+0000 mon.a (mon.0) 3586 : cluster 0 osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:41:04.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:05.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:05.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:05.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.157 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:05.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:05.292 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:05.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:05.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:05.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:05.554 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:05.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:05.668 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:05 smithi143 bash[21512]: cluster 2024-03-11T21:41:05.075840+0000 osd.7 (osd.7) 400 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:05.668 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:05 smithi143 bash[21512]: cluster 2024-03-11T21:41:05.076316+0000 osd.7 (osd.7) 401 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:05 smithi143 bash[29284]: cluster 2024-03-11T21:41:05.075840+0000 osd.7 (osd.7) 400 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:05.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:05 smithi143 bash[29284]: cluster 2024-03-11T21:41:05.076316+0000 osd.7 (osd.7) 401 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:05.684 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:05.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:05.816 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:05.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:05.930 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:05 smithi203 bash[23654]: cluster 2024-03-11T21:41:05.075840+0000 osd.7 (osd.7) 400 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:05.930 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:05 smithi203 bash[23654]: cluster 2024-03-11T21:41:05.076316+0000 osd.7 (osd.7) 401 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:05.943 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:06.046 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:06.073 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:06.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:06.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:06.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:06.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:06.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:06.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.595 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:06.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:06 smithi203 bash[23654]: cluster 2024-03-11T21:41:05.584446+0000 mgr.y (mgr.14550) 704 : cluster 0 pgmap v1185: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:06.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:06 smithi203 bash[23654]: cluster 2024-03-11T21:41:05.638147+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:41:06.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:06 smithi203 bash[23654]: cluster 2024-03-11T21:41:06.513615+0000 mon.a (mon.0) 3588 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:06.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:06.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:06.855 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:06.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[21512]: cluster 2024-03-11T21:41:05.584446+0000 mgr.y (mgr.14550) 704 : cluster 0 pgmap v1185: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:06.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[21512]: cluster 2024-03-11T21:41:05.638147+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:41:06.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[21512]: cluster 2024-03-11T21:41:06.513615+0000 mon.a (mon.0) 3588 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[29284]: cluster 2024-03-11T21:41:05.584446+0000 mgr.y (mgr.14550) 704 : cluster 0 pgmap v1185: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[29284]: cluster 2024-03-11T21:41:05.638147+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:41:06.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:06 smithi143 bash[29284]: cluster 2024-03-11T21:41:06.513615+0000 mon.a (mon.0) 3588 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:06.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:06.985 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:07.089 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:07.119 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:07.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:07.248 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:07.351 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:07.380 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:07.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:07.509 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:07.612 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:07.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:07.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[29284]: cluster 2024-03-11T21:41:06.873778+0000 osd.2 (osd.2) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[29284]: cluster 2024-03-11T21:41:06.874211+0000 osd.2 (osd.2) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[29284]: audit 2024-03-11T21:41:07.212713+0000 mon.a (mon.0) 3589 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[29284]: audit 2024-03-11T21:41:07.217251+0000 mon.c (mon.2) 1239 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[21512]: cluster 2024-03-11T21:41:06.873778+0000 osd.2 (osd.2) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[21512]: cluster 2024-03-11T21:41:06.874211+0000 osd.2 (osd.2) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[21512]: audit 2024-03-11T21:41:07.212713+0000 mon.a (mon.0) 3589 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:07.754 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:07 smithi143 bash[21512]: audit 2024-03-11T21:41:07.217251+0000 mon.c (mon.2) 1239 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:07.767 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:07.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:07.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:07 smithi203 bash[23654]: cluster 2024-03-11T21:41:06.873778+0000 osd.2 (osd.2) 355 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:07.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:07 smithi203 bash[23654]: cluster 2024-03-11T21:41:06.874211+0000 osd.2 (osd.2) 356 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:07.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:07 smithi203 bash[23654]: audit 2024-03-11T21:41:07.212713+0000 mon.a (mon.0) 3589 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:07.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:07 smithi203 bash[23654]: audit 2024-03-11T21:41:07.217251+0000 mon.c (mon.2) 1239 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:07.898 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:08.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:08.030 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:08.133 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:08.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:08.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:08.287 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:08.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:08.418 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:08.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:08.549 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:08.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:08.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:08.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:08.810 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:08.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:08.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:08 smithi203 bash[23654]: cluster 2024-03-11T21:41:07.585024+0000 mgr.y (mgr.14550) 705 : cluster 0 pgmap v1187: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-03-11T21:41:08.925 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:08 smithi203 bash[23654]: cluster 2024-03-11T21:41:07.646677+0000 mon.a (mon.0) 3590 : cluster 0 osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:41:08.942 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:08.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:08 smithi143 bash[29284]: cluster 2024-03-11T21:41:07.585024+0000 mgr.y (mgr.14550) 705 : cluster 0 pgmap v1187: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-03-11T21:41:08.942 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:08 smithi143 bash[29284]: cluster 2024-03-11T21:41:07.646677+0000 mon.a (mon.0) 3590 : cluster 0 osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:41:08.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:08 smithi143 bash[21512]: cluster 2024-03-11T21:41:07.585024+0000 mgr.y (mgr.14550) 705 : cluster 0 pgmap v1187: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-03-11T21:41:08.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:08 smithi143 bash[21512]: cluster 2024-03-11T21:41:07.646677+0000 mon.a (mon.0) 3590 : cluster 0 osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:41:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:09.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:09.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:08 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:09.202 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:09.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:09.332 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:09.435 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:09.465 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:09.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:09.592 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:09.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.721 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:09.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.851 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:09.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[21512]: cluster 2024-03-11T21:41:08.504537+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[21512]: cluster 2024-03-11T21:41:08.505082+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[21512]: audit 2024-03-11T21:41:08.985969+0000 mgr.y (mgr.14550) 706 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[21512]: cluster 2024-03-11T21:41:09.218621+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[29284]: cluster 2024-03-11T21:41:08.504537+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[29284]: cluster 2024-03-11T21:41:08.505082+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[29284]: audit 2024-03-11T21:41:08.985969+0000 mgr.y (mgr.14550) 706 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:09.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:09 smithi143 bash[29284]: cluster 2024-03-11T21:41:09.218621+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:41:09.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:09.981 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:10.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:09 smithi203 bash[23654]: cluster 2024-03-11T21:41:08.504537+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:10.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:09 smithi203 bash[23654]: cluster 2024-03-11T21:41:08.505082+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:10.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:09 smithi203 bash[23654]: audit 2024-03-11T21:41:08.985969+0000 mgr.y (mgr.14550) 706 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:10.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:09 smithi203 bash[23654]: cluster 2024-03-11T21:41:09.218621+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:41:10.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:10.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:10.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:10.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:10.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:10.372 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:10.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:10.502 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:10.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:10.632 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:10.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:10.762 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:10.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:10 smithi203 bash[23654]: cluster 2024-03-11T21:41:09.585659+0000 mgr.y (mgr.14550) 707 : cluster 0 pgmap v1190: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-03-11T21:41:10.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:10 smithi203 bash[23654]: audit 2024-03-11T21:41:09.687475+0000 mon.c (mon.2) 1240 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:10.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:10 smithi203 bash[23654]: audit 2024-03-11T21:41:09.688044+0000 mon.a (mon.0) 3592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:10.865 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:10.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:10.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:11.022 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:11.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[21512]: cluster 2024-03-11T21:41:09.585659+0000 mgr.y (mgr.14550) 707 : cluster 0 pgmap v1190: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-03-11T21:41:11.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[21512]: audit 2024-03-11T21:41:09.687475+0000 mon.c (mon.2) 1240 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:11.103 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[21512]: audit 2024-03-11T21:41:09.688044+0000 mon.a (mon.0) 3592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:11.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[29284]: cluster 2024-03-11T21:41:09.585659+0000 mgr.y (mgr.14550) 707 : cluster 0 pgmap v1190: 246 pgs: 246 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 407 B/s wr, 1 op/s 2024-03-11T21:41:11.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[29284]: audit 2024-03-11T21:41:09.687475+0000 mon.c (mon.2) 1240 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:11.104 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:10 smithi143 bash[29284]: audit 2024-03-11T21:41:09.688044+0000 mon.a (mon.0) 3592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:11.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:11.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:11.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:11.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:11.389 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.417 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:11.520 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.542 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:11.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:11.672 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:11.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.802 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:11.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:11.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: audit 2024-03-11T21:41:10.656158+0000 mon.a (mon.0) 3593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: cluster 2024-03-11T21:41:10.662500+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: audit 2024-03-11T21:41:10.668103+0000 mon.c (mon.2) 1241 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: audit 2024-03-11T21:41:10.670549+0000 mon.a (mon.0) 3595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: cluster 2024-03-11T21:41:11.515737+0000 mon.a (mon.0) 3596 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: audit 2024-03-11T21:41:11.519549+0000 mon.a (mon.0) 3597 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:12.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:11 smithi203 bash[23654]: cluster 2024-03-11T21:41:11.526626+0000 mon.a (mon.0) 3598 : cluster 0 osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:41:12.035 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: audit 2024-03-11T21:41:10.656158+0000 mon.a (mon.0) 3593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: cluster 2024-03-11T21:41:10.662500+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: audit 2024-03-11T21:41:10.668103+0000 mon.c (mon.2) 1241 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: audit 2024-03-11T21:41:10.670549+0000 mon.a (mon.0) 3595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: cluster 2024-03-11T21:41:11.515737+0000 mon.a (mon.0) 3596 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: audit 2024-03-11T21:41:11.519549+0000 mon.a (mon.0) 3597 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:12.046 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[21512]: cluster 2024-03-11T21:41:11.526626+0000 mon.a (mon.0) 3598 : cluster 0 osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: audit 2024-03-11T21:41:10.656158+0000 mon.a (mon.0) 3593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: cluster 2024-03-11T21:41:10.662500+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: audit 2024-03-11T21:41:10.668103+0000 mon.c (mon.2) 1241 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: audit 2024-03-11T21:41:10.670549+0000 mon.a (mon.0) 3595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]: dispatch 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: cluster 2024-03-11T21:41:11.515737+0000 mon.a (mon.0) 3596 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: audit 2024-03-11T21:41:11.519549+0000 mon.a (mon.0) 3597 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-128"}]': finished 2024-03-11T21:41:12.047 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:11 smithi143 bash[29284]: cluster 2024-03-11T21:41:11.526626+0000 mon.a (mon.0) 3598 : cluster 0 osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:41:12.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:12.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:12.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:12.304 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:12.318 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:12.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 -- 172.21.15.143:0/860643605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9006a4b0 msgr2=0x7fbe9010ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 --2- 172.21.15.143:0/860643605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9006a4b0 0x7fbe9010ecd0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009f90 tx=0x7fbe8800b330 comp rx=0 tx=0).stop 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 -- 172.21.15.143:0/860643605 shutdown_connections 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 --2- 172.21.15.143:0/860643605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90107e90 0x7fbe90108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 --2- 172.21.15.143:0/860643605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9006a4b0 0x7fbe9010ecd0 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 --2- 172.21.15.143:0/860643605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90069b10 0x7fbe90069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.453+0000 7fbe96740700 1 -- 172.21.15.143:0/860643605 >> 172.21.15.143:0/860643605 conn(0x7fbe90076d80 msgr2=0x7fbe90077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- 172.21.15.143:0/860643605 shutdown_connections 2024-03-11T21:41:12.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- 172.21.15.143:0/860643605 wait complete. 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 librados: starting msgr at 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 librados: starting objecter 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 Processor -- start 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- start start 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 librados: setting wanted keys 2024-03-11T21:41:12.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 librados: calling monclient init 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 0x7fbe90101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90107e90 0x7fbe90101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe901115f0 con 0x7fbe9006a4b0 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90111470 con 0x7fbe90107e90 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90111770 con 0x7fbe90069b10 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:45310/0 (socket says 172.21.15.143:45310) 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 -- 172.21.15.143:0/802211559 learned_addr learned my addr 172.21.15.143:0/802211559 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8f7fe700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 0x7fbe90101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe94cdd700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90107e90 0x7fbe90101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 -- 172.21.15.143:0/802211559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90107e90 msgr2=0x7fbe90101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90107e90 0x7fbe90101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 -- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 msgr2=0x7fbe90101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 0x7fbe90101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 -- 172.21.15.143:0/802211559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe901a5fc0 con 0x7fbe90069b10 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8f7fe700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 0x7fbe90101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8ffff700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000ec10 tx=0x7fbe8000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8d7fa700 10 client.?.objecter ms_handle_connect 0x7fbe90069b10 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8000cd40 con 0x7fbe90069b10 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fbe800044a0 con 0x7fbe90069b10 2024-03-11T21:41:12.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80010430 con 0x7fbe90069b10 2024-03-11T21:41:12.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe901a62b0 con 0x7fbe90069b10 2024-03-11T21:41:12.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 10 client.42005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:12.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe901a6770 con 0x7fbe90069b10 2024-03-11T21:41:12.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.457+0000 7fbe96740700 1 librados: init done 2024-03-11T21:41:12.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9004faa0 con 0x7fbe90069b10 2024-03-11T21:41:12.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbe80003fb0 con 0x7fbe90069b10 2024-03-11T21:41:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:12.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 0x7fbe78071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(770..770 src has 87..770) v4 ==== 10225+0+0 (secure 0 0 0) 0x7fbe8008f410 con 0x7fbe90069b10 2024-03-11T21:41:12.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 10 client.42005.objecter ms_dispatch 0x7fbe90001550 osd_map(770..770 src has 87..770) v4 2024-03-11T21:41:12.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 3 client.42005.objecter handle_osd_map got epochs [770,770] > 0 2024-03-11T21:41:12.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 3 client.42005.objecter handle_osd_map decoding full epoch 770 2024-03-11T21:41:12.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8f7fe700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 0x7fbe78071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 20 client.42005.objecter dump_active .. 0 homeless 2024-03-11T21:41:12.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8d7fa700 10 client.42005.objecter ms_handle_connect 0x7fbe7806f820 2024-03-11T21:41:12.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.461+0000 7fbe8f7fe700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 0x7fbe78071ce0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fbe90108500 tx=0x7fbe8400a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.465+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fbe8001d060 con 0x7fbe90069b10 2024-03-11T21:41:12.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:12.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:12.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.709+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fbe901a6a30 con 0x7fbe90069b10 2024-03-11T21:41:12.707 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:12.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- 172.21.15.143:0/3498207118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820101700 msgr2=0x7ff82010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 --2- 172.21.15.143:0/3498207118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820101700 0x7ff82010db70 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7ff81800b0a0 tx=0x7ff81800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- 172.21.15.143:0/3498207118 shutdown_connections 2024-03-11T21:41:12.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 --2- 172.21.15.143:0/3498207118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff820106dc0 0x7ff8201071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 --2- 172.21.15.143:0/3498207118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820101700 0x7ff82010db70 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 --2- 172.21.15.143:0/3498207118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 0x7ff8201011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- 172.21.15.143:0/3498207118 >> 172.21.15.143:0/3498207118 conn(0x7ff820076d80 msgr2=0x7ff820077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- 172.21.15.143:0/3498207118 shutdown_connections 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- 172.21.15.143:0/3498207118 wait complete. 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 librados: starting msgr at 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 librados: starting objecter 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 Processor -- start 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 -- start start 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 librados: setting wanted keys 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.773+0000 7ff825795700 1 librados: calling monclient init 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 0x7ff8201029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff820101700 0x7ff820102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820110540 con 0x7ff820101700 2024-03-11T21:41:12.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8201103c0 con 0x7ff820100d40 2024-03-11T21:41:12.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8201106c0 con 0x7ff820106dc0 2024-03-11T21:41:12.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:45324/0 (socket says 172.21.15.143:45324) 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 learned_addr learned my addr 172.21.15.143:0/455505687 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff820101700 0x7ff820102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 0x7ff8201029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 msgr2=0x7ff8201029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 0x7ff8201029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff820101700 msgr2=0x7ff820102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff820101700 0x7ff820102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8201a6060 con 0x7ff820106dc0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff820100d40 0x7ff8201029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7ff818006070 tx=0x7ff818004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 10 client.?.objecter ms_handle_connect 0x7ff820106dc0 2024-03-11T21:41:12.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818021030 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8201a62f0 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 10 client.42011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff818007a00 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8201a67f0 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 librados: init done 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818004470 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff80c005130 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff818024070 con 0x7ff820106dc0 2024-03-11T21:41:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 0x7ff808071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(771..771 src has 87..771) v4 ==== 9852+0+0 (secure 0 0 0) 0x7ff81800f020 con 0x7ff820106dc0 2024-03-11T21:41:12.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 osd_map(771..771 src has 87..771) v4 2024-03-11T21:41:12.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 3 client.42011.objecter handle_osd_map got epochs [771,771] > 0 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 3 client.42011.objecter handle_osd_map decoding full epoch 771 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 0x7ff808071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 20 client.42011.objecter dump_active .. 0 homeless 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.777+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v771) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff818090080 con 0x7ff820106dc0 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 10 client.42011.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 1 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff80c004760 0x7ff80c004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 1 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 0x7ff808071ce0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7ff81400bb90 tx=0x7ff814005d80 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 0 1 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 4 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 10 client.42011.objecter _send_command 1 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c008ba0 con 0x7ff80c004760 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80806f820 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff80c004760 0x7ff80c004b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c004760 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff80c004760 0x7ff80c004b80 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c008ba0 con 0x7ff80c004760 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 10 client.42011.objecter _finish_command 1 = system:0 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 4 2024-03-11T21:41:12.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.781+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 0 1 2024-03-11T21:41:12.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 10 client.42011.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 2 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 2 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c005410 osd=0 3 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 0 2 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 4 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 10 client.42011.objecter _send_command 2 2024-03-11T21:41:12.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.797+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c004210 con 0x7ff80c004760 2024-03-11T21:41:12.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c004210 con 0x7ff80c004760 2024-03-11T21:41:12.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 10 client.42011.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 4 2024-03-11T21:41:12.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 0 2 2024-03-11T21:41:12.785 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:41:12.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 10 client.42011.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 3 2024-03-11T21:41:12.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff80c002780 0x7ff80c002bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 1 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 4 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 10 client.42011.objecter _send_command 3 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c003100 con 0x7ff80c002780 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff80c002780 0x7ff80c002bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff80c002780 0x7ff80c002bc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c002780 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c003100 con 0x7ff80c002780 2024-03-11T21:41:12.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:41:12.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 10 client.42011.objecter _finish_command 3 = system:0 2024-03-11T21:41:12.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 4 2024-03-11T21:41:12.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.801+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 1 3 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 10 client.42011.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 4 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 4 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0081f0 osd=1 3 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 1 4 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 4 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 10 client.42011.objecter _send_command 4 2024-03-11T21:41:12.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.817+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c003b10 con 0x7ff80c002780 2024-03-11T21:41:12.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c003b10 con 0x7ff80c002780 2024-03-11T21:41:12.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 10 client.42011.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 4 2024-03-11T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 1 4 2024-03-11T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 10 client.42011.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 5 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff80c00b270 0x7ff80c00d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 5 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 2 5 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 4 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 10 client.42011.objecter _send_command 5 2024-03-11T21:41:12.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c0105e0 con 0x7ff80c00b270 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff80c00b270 0x7ff80c00d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c00b270 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff80c00b270 0x7ff80c00d660 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c0105e0 con 0x7ff80c00b270 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 10 client.42011.objecter _finish_command 5 = system:0 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 4 2024-03-11T21:41:12.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.821+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 2 5 2024-03-11T21:41:12.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 10 client.42011.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 6 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 6 2024-03-11T21:41:12.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c007ec0 osd=2 3 2024-03-11T21:41:12.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 2 6 2024-03-11T21:41:12.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 4 2024-03-11T21:41:12.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 10 client.42011.objecter _send_command 6 2024-03-11T21:41:12.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.837+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c0035f0 con 0x7ff80c00b270 2024-03-11T21:41:12.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c0035f0 con 0x7ff80c00b270 2024-03-11T21:41:12.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 10 client.42011.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 4 2024-03-11T21:41:12.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 2 6 2024-03-11T21:41:12.826 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:41:12.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 10 client.42011.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 7 2024-03-11T21:41:12.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff80c00ed20 0x7ff80c00f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 7 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 3 7 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 4 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 10 client.42011.objecter _send_command 7 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c00f6f0 con 0x7ff80c00ed20 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff80c00ed20 0x7ff80c00f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c00ed20 2024-03-11T21:41:12.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff80c00ed20 0x7ff80c00f1b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c00f6f0 con 0x7ff80c00ed20 2024-03-11T21:41:12.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:41:12.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 10 client.42011.objecter _finish_command 7 = system:0 2024-03-11T21:41:12.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 4 2024-03-11T21:41:12.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.841+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 3 7 2024-03-11T21:41:12.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 10 client.42011.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 8 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 8 2024-03-11T21:41:12.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c001b80 osd=3 3 2024-03-11T21:41:12.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 3 8 2024-03-11T21:41:12.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 4 2024-03-11T21:41:12.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 10 client.42011.objecter _send_command 8 2024-03-11T21:41:12.846 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c00f6f0 con 0x7ff80c00ed20 2024-03-11T21:41:12.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c00f6f0 con 0x7ff80c00ed20 2024-03-11T21:41:12.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff807fff700 10 client.42011.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 4 2024-03-11T21:41:12.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 3 8 2024-03-11T21:41:12.848 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 10 client.42011.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 9 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff80c00ff10 0x7ff80c0147b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 9 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 4 9 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 4 2024-03-11T21:41:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 10 client.42011.objecter _send_command 9 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.861+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c0176d0 con 0x7ff80c00ff10 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff80c00ff10 0x7ff80c0147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c00ff10 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff80c00ff10 0x7ff80c0147b0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c0176d0 con 0x7ff80c00ff10 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 10 client.42011.objecter _finish_command 9 = system:0 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 4 2024-03-11T21:41:12.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.865+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 4 9 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 10 client.42011.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 10 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 10 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0023f0 osd=4 3 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 4 10 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 4 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 10 client.42011.objecter _send_command 10 2024-03-11T21:41:12.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.881+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c0164c0 con 0x7ff80c00ff10 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c0164c0 con 0x7ff80c00ff10 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff807fff700 10 client.42011.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 4 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 4 10 2024-03-11T21:41:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 10 client.42011.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 11 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff80c016be0 0x7ff80c017020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 11 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 5 11 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 4 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 10 client.42011.objecter _send_command 11 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.885+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c01aac0 con 0x7ff80c016be0 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff80c016be0 0x7ff80c017020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c016be0 2024-03-11T21:41:12.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff81e7fc700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff80c016be0 0x7ff80c017020 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c01aac0 con 0x7ff80c016be0 2024-03-11T21:41:12.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:41:12.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 10 client.42011.objecter _finish_command 11 = system:0 2024-03-11T21:41:12.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 4 2024-03-11T21:41:12.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.889+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 5 11 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 10 client.42011.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 12 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 12 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c0164c0 osd=5 3 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 5 12 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 4 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 10 client.42011.objecter _send_command 12 2024-03-11T21:41:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.905+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c01aac0 con 0x7ff80c016be0 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c01aac0 con 0x7ff80c016be0 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff807fff700 10 client.42011.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 4 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 5 12 2024-03-11T21:41:12.895 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 10 client.42011.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 13 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff80c019e30 0x7ff80c01a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 13 2024-03-11T21:41:12.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 6 13 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 4 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 10 client.42011.objecter _send_command 13 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.909+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c01de30 con 0x7ff80c019e30 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff80c019e30 0x7ff80c01a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c019e30 2024-03-11T21:41:12.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff81f7fe700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff80c019e30 0x7ff80c01a2b0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c01de30 con 0x7ff80c019e30 2024-03-11T21:41:12.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:41:12.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 10 client.42011.objecter _finish_command 13 = system:0 2024-03-11T21:41:12.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 4 2024-03-11T21:41:12.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.913+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 6 13 2024-03-11T21:41:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 10 client.42011.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 14 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 14 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c01aac0 osd=6 3 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 6 14 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 4 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 10 client.42011.objecter _send_command 14 2024-03-11T21:41:12.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.929+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c001410 con 0x7ff80c019e30 2024-03-11T21:41:12.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.933+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c001410 con 0x7ff80c019e30 2024-03-11T21:41:12.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.933+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.933+0000 7ff807fff700 10 client.42011.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.933+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 4 2024-03-11T21:41:12.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.933+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 6 14 2024-03-11T21:41:12.920 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:41:12.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 10 client.42011.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:12.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 15 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff80c015e70 0x7ff80c00e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 15 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 7 15 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 4 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 10 client.42011.objecter _send_command 15 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff80c00e8b0 con 0x7ff80c015e70 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff80c015e70 0x7ff80c00e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 10 client.42011.objecter ms_handle_connect 0x7ff80c015e70 2024-03-11T21:41:12.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff81effd700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff80c015e70 0x7ff80c00e370 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:12.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff80c00e8b0 con 0x7ff80c015e70 2024-03-11T21:41:12.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:41:12.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 10 client.42011.objecter _finish_command 15 = system:0 2024-03-11T21:41:12.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 4 2024-03-11T21:41:12.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.937+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 7 15 2024-03-11T21:41:12.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 10 client.42011.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign -1 16 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 20 client.42011.objecter _get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 15 client.42011.objecter _session_command_op_remove -1 16 2024-03-11T21:41:12.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 20 client.42011.objecter get_session s=0x7ff80c001410 osd=7 3 2024-03-11T21:41:12.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 15 client.42011.objecter _session_command_op_assign 7 16 2024-03-11T21:41:12.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 4 2024-03-11T21:41:12.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 10 client.42011.objecter _send_command 16 2024-03-11T21:41:12.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.957+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff80c00e8b0 con 0x7ff80c015e70 2024-03-11T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:12 smithi143 bash[21512]: cluster 2024-03-11T21:41:11.586101+0000 mgr.y (mgr.14550) 708 : cluster 0 pgmap v1193: 246 pgs: 246 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:12 smithi143 bash[21512]: cluster 2024-03-11T21:41:12.530842+0000 mon.a (mon.0) 3599 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:12.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:12 smithi143 bash[29284]: cluster 2024-03-11T21:41:11.586101+0000 mgr.y (mgr.14550) 708 : cluster 0 pgmap v1193: 246 pgs: 246 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:41:12.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:12 smithi143 bash[29284]: cluster 2024-03-11T21:41:12.530842+0000 mon.a (mon.0) 3599 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:12.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff807fff700 1 -- 172.21.15.143:0/455505687 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff80c00e8b0 con 0x7ff80c015e70 2024-03-11T21:41:12.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff807fff700 10 client.42011.objecter ms_dispatch 0x7ff820001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:12.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff807fff700 10 client.42011.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:12.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff807fff700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 4 2024-03-11T21:41:12.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff807fff700 15 client.42011.objecter _session_command_op_remove 7 16 2024-03-11T21:41:12.944 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:41:12.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 librados: watch_flush enter 2024-03-11T21:41:12.945 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 librados: watch_flush exit 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.0 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff80c004760 msgr2=0x7ff80c004b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff80c004760 0x7ff80c004b80 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c005410 osd=0 1 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.1 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff80c002780 msgr2=0x7ff80c002bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff80c002780 0x7ff80c002bc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.946 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0081f0 osd=1 1 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.2 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff80c00b270 msgr2=0x7ff80c00d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff80c00b270 0x7ff80c00d660 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c007ec0 osd=2 1 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.3 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff80c00ed20 msgr2=0x7ff80c00f1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff80c00ed20 0x7ff80c00f1b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001b80 osd=3 1 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.4 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff80c00ff10 msgr2=0x7ff80c0147b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff80c00ff10 0x7ff80c0147b0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0023f0 osd=4 1 2024-03-11T21:41:12.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.5 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff80c016be0 msgr2=0x7ff80c017020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 reap_dead start 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff80c016be0 0x7ff80c017020 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 reap_dead start 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c0164c0 osd=5 1 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.6 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff80c019e30 msgr2=0x7ff80c01a2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff80c019e30 0x7ff80c01a2b0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c01aac0 osd=6 1 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter close_session for osd.7 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff80c015e70 msgr2=0x7ff80c00e370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff80c015e70 0x7ff80c00e370 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter put_session s=0x7ff80c001410 osd=7 1 2024-03-11T21:41:12.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 20 client.42011.objecter shutdown clearing up homeless session... 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 10 client.42011.objecter successfully canceled tick 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff81f7fe700 1 -- 172.21.15.143:0/455505687 reap_dead start 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 msgr2=0x7ff808071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 0x7ff808071ce0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7ff81400bb90 tx=0x7ff814005d80 comp rx=0 tx=0).stop 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 msgr2=0x7ff820103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7ff818006070 tx=0x7ff818004dc0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 shutdown_connections 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff820106dc0 0x7ff820103440 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 --2- 172.21.15.143:0/455505687 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff80806f820 0x7ff808071ce0 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 >> 172.21.15.143:0/455505687 conn(0x7ff820076d80 msgr2=0x7ff820069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 shutdown_connections 2024-03-11T21:41:12.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 -- 172.21.15.143:0/455505687 wait complete. 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:12.961+0000 7ff825795700 1 librados: shutdown 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:41:12.950 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:41:12.951 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:41:12.957 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:13.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:12 smithi203 bash[23654]: cluster 2024-03-11T21:41:11.586101+0000 mgr.y (mgr.14550) 708 : cluster 0 pgmap v1193: 246 pgs: 246 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:41:13.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:12 smithi203 bash[23654]: cluster 2024-03-11T21:41:12.530842+0000 mon.a (mon.0) 3599 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:13.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:13.088 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:13.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:13.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:13.258 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:41:13.258 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:41:13.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:13.330 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 -- 172.21.15.143:0/3654385729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34101ec0 msgr2=0x7fad34102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/3654385729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34101ec0 0x7fad34102340 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fad280098f0 tx=0x7fad28009c00 comp rx=0 tx=0).stop 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 -- 172.21.15.143:0/3654385729 shutdown_connections 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/3654385729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34102880 0x7fad3410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/3654385729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad34107f30 0x7fad34108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/3654385729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34101ec0 0x7fad34102340 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 -- 172.21.15.143:0/3654385729 >> 172.21.15.143:0/3654385729 conn(0x7fad34078eb0 msgr2=0x7fad340792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:13.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 -- 172.21.15.143:0/3654385729 shutdown_connections 2024-03-11T21:41:13.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.373+0000 7fad3a7bc700 1 -- 172.21.15.143:0/3654385729 wait complete. 2024-03-11T21:41:13.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 librados: starting msgr at 2024-03-11T21:41:13.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 librados: starting objecter 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 Processor -- start 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- start start 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 librados: setting wanted keys 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 librados: calling monclient init 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 0x7fad340740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 0x7fad34074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad34107f30 0x7fad34074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad341116a0 con 0x7fad34102880 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad34111520 con 0x7fad34107f30 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad34111820 con 0x7fad34101ec0 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad33fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 0x7fad340740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:13.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 0x7fad34074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad33fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 0x7fad340740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:35144/0 (socket says 172.21.15.143:35144) 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad33fff700 1 -- 172.21.15.143:0/1829433910 learned_addr learned my addr 172.21.15.143:0/1829433910 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 -- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 msgr2=0x7fad340740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 0x7fad340740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 -- 172.21.15.143:0/1829433910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad34107f30 msgr2=0x7fad34074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad34107f30 0x7fad34074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 -- 172.21.15.143:0/1829433910 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad341a6210 con 0x7fad34102880 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad337fe700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 0x7fad34074620 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fad28009230 tx=0x7fad2800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 10 client.?.objecter ms_handle_connect 0x7fad34102880 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800b660 con 0x7fad34102880 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad341a64a0 con 0x7fad34102880 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fad28005040 con 0x7fad34102880 2024-03-11T21:41:13.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad280038a0 con 0x7fad34102880 2024-03-11T21:41:13.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 10 client.30813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:13.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad341a6950 con 0x7fad34102880 2024-03-11T21:41:13.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 librados: init done 2024-03-11T21:41:13.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3404faa0 con 0x7fad34102880 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.377+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fad2800b800 con 0x7fad34102880 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 0x7fad1c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(771..771 src has 87..771) v4 ==== 9852+0+0 (secure 0 0 0) 0x7fad2808ffe0 con 0x7fad34102880 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 10 client.30813.objecter ms_dispatch 0x7fad34001550 osd_map(771..771 src has 87..771) v4 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 3 client.30813.objecter handle_osd_map got epochs [771,771] > 0 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 3 client.30813.objecter handle_osd_map decoding full epoch 771 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad33fff700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 0x7fad1c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 20 client.30813.objecter dump_active .. 0 homeless 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad317fa700 10 client.30813.objecter ms_handle_connect 0x7fad1c06f820 2024-03-11T21:41:13.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.381+0000 7fad33fff700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 0x7fad1c071ce0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fad24009ad0 tx=0x7fad24009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:13.368 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.385+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fad28013030 con 0x7fad34102880 2024-03-11T21:41:13.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:13.453 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:13.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:13.577 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:13.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.609+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7fad341a6c70 con 0x7fad34102880 2024-03-11T21:41:13.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.677+0000 7fad317fa700 1 -- 172.21.15.143:0/1829433910 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v772) v1 ==== 113+0+0 (secure 0 0 0) 0x7fad2805cbd0 con 0x7fad34102880 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.681+0000 7fbe8d7fa700 1 -- 172.21.15.143:0/802211559 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v772) v1 ==== 91+0+0 (secure 0 0 0) 0x7fbe8005be00 con 0x7fbe90069b10 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 10 librados: watch_flush enter 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 10 librados: watch_flush exit 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 20 client.42005.objecter shutdown clearing up homeless session... 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 10 client.42005.objecter successfully canceled tick 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 msgr2=0x7fbe78071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 0x7fbe78071ce0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fbe90108500 tx=0x7fbe8400a380 comp rx=0 tx=0).stop 2024-03-11T21:41:13.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 msgr2=0x7fbe90100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000ec10 tx=0x7fbe8000ef20 comp rx=0 tx=0).stop 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 shutdown_connections 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbe7806f820 0x7fbe78071ce0 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe90107e90 0x7fbe90101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe90069b10 0x7fbe90100e10 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 --2- 172.21.15.143:0/802211559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9006a4b0 0x7fbe90101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 >> 172.21.15.143:0/802211559 conn(0x7fbe90076d80 msgr2=0x7fbe90105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 shutdown_connections 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 -- 172.21.15.143:0/802211559 wait complete. 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.685+0000 7fbe96740700 1 librados: shutdown 2024-03-11T21:41:13.677 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.681+0000 7fad3a7bc700 10 librados: watch_flush enter 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.681+0000 7fad3a7bc700 10 librados: watch_flush exit 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.681+0000 7fad3a7bc700 20 client.30813.objecter shutdown clearing up homeless session... 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.681+0000 7fad3a7bc700 10 client.30813.objecter successfully canceled tick 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.693+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 msgr2=0x7fad1c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.693+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 0x7fad1c071ce0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fad24009ad0 tx=0x7fad24009290 comp rx=0 tx=0).stop 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.693+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 msgr2=0x7fad34074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:13.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.693+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 0x7fad34074620 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fad28009230 tx=0x7fad2800fa70 comp rx=0 tx=0).stop 2024-03-11T21:41:13.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 shutdown_connections 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fad1c06f820 0x7fad1c071ce0 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad34107f30 0x7fad34074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad34101ec0 0x7fad340740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 --2- 172.21.15.143:0/1829433910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad34102880 0x7fad34074620 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:13.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 >> 172.21.15.143:0/1829433910 conn(0x7fad34078eb0 msgr2=0x7fad34108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:13.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 shutdown_connections 2024-03-11T21:41:13.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 -- 172.21.15.143:0/1829433910 wait complete. 2024-03-11T21:41:13.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:13.701+0000 7fad3a7bc700 1 librados: shutdown 2024-03-11T21:41:13.687 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:41:13.701 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:13.734 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[29284]: cluster 2024-03-11T21:41:12.672896+0000 mon.a (mon.0) 3600 : cluster 0 osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[29284]: audit 2024-03-11T21:41:12.715991+0000 mon.c (mon.2) 1242 : audit 1 from='client.? 172.21.15.143:0/802211559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[29284]: audit 2024-03-11T21:41:12.782257+0000 mon.c (mon.2) 1243 : audit 0 from='client.? 172.21.15.143:0/455505687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[29284]: audit 2024-03-11T21:41:13.114629+0000 mon.a (mon.0) 3601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[29284]: audit 2024-03-11T21:41:13.612877+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[21512]: cluster 2024-03-11T21:41:12.672896+0000 mon.a (mon.0) 3600 : cluster 0 osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[21512]: audit 2024-03-11T21:41:12.715991+0000 mon.c (mon.2) 1242 : audit 1 from='client.? 172.21.15.143:0/802211559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[21512]: audit 2024-03-11T21:41:12.782257+0000 mon.c (mon.2) 1243 : audit 0 from='client.? 172.21.15.143:0/455505687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[21512]: audit 2024-03-11T21:41:13.114629+0000 mon.a (mon.0) 3601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:13.735 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:13 smithi143 bash[21512]: audit 2024-03-11T21:41:13.612877+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:41:13.804 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:13.826 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:13.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:14.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:13 smithi203 bash[23654]: cluster 2024-03-11T21:41:12.672896+0000 mon.a (mon.0) 3600 : cluster 0 osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:41:14.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:13 smithi203 bash[23654]: audit 2024-03-11T21:41:12.715991+0000 mon.c (mon.2) 1242 : audit 1 from='client.? 172.21.15.143:0/802211559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:14.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:13 smithi203 bash[23654]: audit 2024-03-11T21:41:12.782257+0000 mon.c (mon.2) 1243 : audit 0 from='client.? 172.21.15.143:0/455505687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:14.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:13 smithi203 bash[23654]: audit 2024-03-11T21:41:13.114629+0000 mon.a (mon.0) 3601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:41:14.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:13 smithi203 bash[23654]: audit 2024-03-11T21:41:13.612877+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:41:14.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:14.082 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:14.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:14.212 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:14.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:14.342 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:14.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:14.473 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:14.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:14.603 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:14.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:14.732 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:14.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:14.862 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:14.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: cluster 2024-03-11T21:41:13.587866+0000 mgr.y (mgr.14550) 709 : cluster 0 pgmap v1195: 214 pgs: 214 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-11T21:41:14.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: cluster 2024-03-11T21:41:13.671369+0000 mon.a (mon.0) 3603 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: audit 2024-03-11T21:41:13.680386+0000 mon.a (mon.0) 3604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: audit 2024-03-11T21:41:13.680599+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: cluster 2024-03-11T21:41:13.686091+0000 mon.a (mon.0) 3606 : cluster 0 osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: audit 2024-03-11T21:41:13.691339+0000 mon.c (mon.2) 1244 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: audit 2024-03-11T21:41:13.691902+0000 mon.a (mon.0) 3607 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: cluster 2024-03-11T21:41:14.199924+0000 osd.5 (osd.5) 421 : cluster 0 1.a scrub starts 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[21512]: cluster 2024-03-11T21:41:14.202608+0000 osd.5 (osd.5) 422 : cluster 0 1.a scrub ok 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: cluster 2024-03-11T21:41:13.587866+0000 mgr.y (mgr.14550) 709 : cluster 0 pgmap v1195: 214 pgs: 214 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: cluster 2024-03-11T21:41:13.671369+0000 mon.a (mon.0) 3603 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: audit 2024-03-11T21:41:13.680386+0000 mon.a (mon.0) 3604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: audit 2024-03-11T21:41:13.680599+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:41:14.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: cluster 2024-03-11T21:41:13.686091+0000 mon.a (mon.0) 3606 : cluster 0 osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:41:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: audit 2024-03-11T21:41:13.691339+0000 mon.c (mon.2) 1244 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: audit 2024-03-11T21:41:13.691902+0000 mon.a (mon.0) 3607 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: cluster 2024-03-11T21:41:14.199924+0000 osd.5 (osd.5) 421 : cluster 0 1.a scrub starts 2024-03-11T21:41:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:14 smithi143 bash[29284]: cluster 2024-03-11T21:41:14.202608+0000 osd.5 (osd.5) 422 : cluster 0 1.a scrub ok 2024-03-11T21:41:14.965 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:14.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: cluster 2024-03-11T21:41:13.587866+0000 mgr.y (mgr.14550) 709 : cluster 0 pgmap v1195: 214 pgs: 214 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-11T21:41:14.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: cluster 2024-03-11T21:41:13.671369+0000 mon.a (mon.0) 3603 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:14.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: audit 2024-03-11T21:41:13.680386+0000 mon.a (mon.0) 3604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:14.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: audit 2024-03-11T21:41:13.680599+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.143:0/1829433910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:41:14.976 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: cluster 2024-03-11T21:41:13.686091+0000 mon.a (mon.0) 3606 : cluster 0 osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:41:14.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: audit 2024-03-11T21:41:13.691339+0000 mon.c (mon.2) 1244 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: audit 2024-03-11T21:41:13.691902+0000 mon.a (mon.0) 3607 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:14.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: cluster 2024-03-11T21:41:14.199924+0000 osd.5 (osd.5) 421 : cluster 0 1.a scrub starts 2024-03-11T21:41:14.977 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:14 smithi203 bash[23654]: cluster 2024-03-11T21:41:14.202608+0000 osd.5 (osd.5) 422 : cluster 0 1.a scrub ok 2024-03-11T21:41:14.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:15.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:15.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:15.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.252 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:15.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:15.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.511 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:15.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:15.641 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:15.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:15.768 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:15.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.897 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:16.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: cluster 2024-03-11T21:41:14.641321+0000 osd.2 (osd.2) 357 : cluster 0 313.5s0 scrub starts 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: cluster 2024-03-11T21:41:14.644203+0000 osd.2 (osd.2) 358 : cluster 0 313.5 scrub ok 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: audit 2024-03-11T21:41:14.686192+0000 mon.a (mon.0) 3608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: cluster 2024-03-11T21:41:14.694945+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: audit 2024-03-11T21:41:14.710592+0000 mon.c (mon.2) 1245 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:15 smithi203 bash[23654]: audit 2024-03-11T21:41:14.712231+0000 mon.a (mon.0) 3610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:16.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: cluster 2024-03-11T21:41:14.641321+0000 osd.2 (osd.2) 357 : cluster 0 313.5s0 scrub starts 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: cluster 2024-03-11T21:41:14.644203+0000 osd.2 (osd.2) 358 : cluster 0 313.5 scrub ok 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: audit 2024-03-11T21:41:14.686192+0000 mon.a (mon.0) 3608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: cluster 2024-03-11T21:41:14.694945+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: audit 2024-03-11T21:41:14.710592+0000 mon.c (mon.2) 1245 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[21512]: audit 2024-03-11T21:41:14.712231+0000 mon.a (mon.0) 3610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: cluster 2024-03-11T21:41:14.641321+0000 osd.2 (osd.2) 357 : cluster 0 313.5s0 scrub starts 2024-03-11T21:41:16.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: cluster 2024-03-11T21:41:14.644203+0000 osd.2 (osd.2) 358 : cluster 0 313.5 scrub ok 2024-03-11T21:41:16.144 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: audit 2024-03-11T21:41:14.686192+0000 mon.a (mon.0) 3608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:16.144 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: cluster 2024-03-11T21:41:14.694945+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:41:16.144 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: audit 2024-03-11T21:41:14.710592+0000 mon.c (mon.2) 1245 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.144 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:15 smithi143 bash[29284]: audit 2024-03-11T21:41:14.712231+0000 mon.a (mon.0) 3610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:16.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:16.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:16.289 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:16.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:16.421 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:16.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:16.552 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:16.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:16.681 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:16.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:41:16.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.370198+0000 osd.3 (osd.3) 407 : cluster 0 9.1b scrub starts 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.372829+0000 osd.3 (osd.3) 408 : cluster 0 9.1b scrub ok 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.531492+0000 osd.4 (osd.4) 393 : cluster 0 3.14 scrub starts 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.535515+0000 osd.4 (osd.4) 394 : cluster 0 3.14 scrub ok 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.589004+0000 mgr.y (mgr.14550) 710 : cluster 0 pgmap v1198: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.649363+0000 osd.2 (osd.2) 359 : cluster 0 5.6 scrub starts 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.651116+0000 osd.2 (osd.2) 360 : cluster 0 5.6 scrub ok 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: audit 2024-03-11T21:41:15.694364+0000 mon.a (mon.0) 3611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: cluster 2024-03-11T21:41:15.700757+0000 mon.a (mon.0) 3612 : cluster 0 osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: audit 2024-03-11T21:41:15.701616+0000 mon.c (mon.2) 1246 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:16 smithi203 bash[23654]: audit 2024-03-11T21:41:15.701928+0000 mon.a (mon.0) 3613 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.813 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:16.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:16.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.370198+0000 osd.3 (osd.3) 407 : cluster 0 9.1b scrub starts 2024-03-11T21:41:16.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.372829+0000 osd.3 (osd.3) 408 : cluster 0 9.1b scrub ok 2024-03-11T21:41:16.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.531492+0000 osd.4 (osd.4) 393 : cluster 0 3.14 scrub starts 2024-03-11T21:41:16.942 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.535515+0000 osd.4 (osd.4) 394 : cluster 0 3.14 scrub ok 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.589004+0000 mgr.y (mgr.14550) 710 : cluster 0 pgmap v1198: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.649363+0000 osd.2 (osd.2) 359 : cluster 0 5.6 scrub starts 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.651116+0000 osd.2 (osd.2) 360 : cluster 0 5.6 scrub ok 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: audit 2024-03-11T21:41:15.694364+0000 mon.a (mon.0) 3611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: cluster 2024-03-11T21:41:15.700757+0000 mon.a (mon.0) 3612 : cluster 0 osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: audit 2024-03-11T21:41:15.701616+0000 mon.c (mon.2) 1246 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[21512]: audit 2024-03-11T21:41:15.701928+0000 mon.a (mon.0) 3613 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.370198+0000 osd.3 (osd.3) 407 : cluster 0 9.1b scrub starts 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.372829+0000 osd.3 (osd.3) 408 : cluster 0 9.1b scrub ok 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.531492+0000 osd.4 (osd.4) 393 : cluster 0 3.14 scrub starts 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.535515+0000 osd.4 (osd.4) 394 : cluster 0 3.14 scrub ok 2024-03-11T21:41:16.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.589004+0000 mgr.y (mgr.14550) 710 : cluster 0 pgmap v1198: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.649363+0000 osd.2 (osd.2) 359 : cluster 0 5.6 scrub starts 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.651116+0000 osd.2 (osd.2) 360 : cluster 0 5.6 scrub ok 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: audit 2024-03-11T21:41:15.694364+0000 mon.a (mon.0) 3611 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: cluster 2024-03-11T21:41:15.700757+0000 mon.a (mon.0) 3612 : cluster 0 osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: audit 2024-03-11T21:41:15.701616+0000 mon.c (mon.2) 1246 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:16 smithi143 bash[29284]: audit 2024-03-11T21:41:15.701928+0000 mon.a (mon.0) 3613 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.945 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:16.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:17.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:17.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:17.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:41:17.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:17.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:17.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.358 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:17.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.487 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:17.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.617 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:17.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:17.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:17.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:17.881 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:17.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:16.316116+0000 osd.0 (osd.0) 402 : cluster 0 313.0s0 scrub starts 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:16.318270+0000 osd.0 (osd.0) 403 : cluster 0 313.0 scrub ok 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:16.665269+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:16.667900+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: audit 2024-03-11T21:41:16.696693+0000 mon.a (mon.0) 3614 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:16.702085+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: audit 2024-03-11T21:41:16.706878+0000 mon.c (mon.2) 1247 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: audit 2024-03-11T21:41:16.713098+0000 mon.a (mon.0) 3616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.192033+0000 osd.5 (osd.5) 423 : cluster 0 5.7 scrub starts 2024-03-11T21:41:17.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:17 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.193611+0000 osd.5 (osd.5) 424 : cluster 0 5.7 scrub ok 2024-03-11T21:41:18.011 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:18.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.142 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:16.316116+0000 osd.0 (osd.0) 402 : cluster 0 313.0s0 scrub starts 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:16.318270+0000 osd.0 (osd.0) 403 : cluster 0 313.0 scrub ok 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:16.665269+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:16.667900+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: audit 2024-03-11T21:41:16.696693+0000 mon.a (mon.0) 3614 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:16.702085+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: audit 2024-03-11T21:41:16.706878+0000 mon.c (mon.2) 1247 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: audit 2024-03-11T21:41:16.713098+0000 mon.a (mon.0) 3616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:18.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.192033+0000 osd.5 (osd.5) 423 : cluster 0 5.7 scrub starts 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.193611+0000 osd.5 (osd.5) 424 : cluster 0 5.7 scrub ok 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:16.316116+0000 osd.0 (osd.0) 402 : cluster 0 313.0s0 scrub starts 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:16.318270+0000 osd.0 (osd.0) 403 : cluster 0 313.0 scrub ok 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:16.665269+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:16.667900+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: audit 2024-03-11T21:41:16.696693+0000 mon.a (mon.0) 3614 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:16.702085+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: audit 2024-03-11T21:41:16.706878+0000 mon.c (mon.2) 1247 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: audit 2024-03-11T21:41:16.713098+0000 mon.a (mon.0) 3616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]: dispatch 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.192033+0000 osd.5 (osd.5) 423 : cluster 0 5.7 scrub starts 2024-03-11T21:41:18.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:17 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.193611+0000 osd.5 (osd.5) 424 : cluster 0 5.7 scrub ok 2024-03-11T21:41:18.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:18.271 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:18.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.402 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:18.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:18.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:18.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.665 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:18.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:41:18.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:18.795 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:18.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.342541+0000 osd.1 (osd.1) 213 : cluster 0 5.1d scrub starts 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.344677+0000 osd.1 (osd.1) 214 : cluster 0 5.1d scrub ok 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.356803+0000 osd.0 (osd.0) 404 : cluster 0 39.4 scrub starts 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.359072+0000 osd.0 (osd.0) 405 : cluster 0 39.4 scrub ok 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.376240+0000 osd.3 (osd.3) 409 : cluster 0 6.6 scrub starts 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.378279+0000 osd.3 (osd.3) 410 : cluster 0 6.6 scrub ok 2024-03-11T21:41:18.909 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.476036+0000 osd.7 (osd.7) 402 : cluster 0 313.2s0 scrub starts 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.477905+0000 osd.7 (osd.7) 403 : cluster 0 313.2 scrub ok 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.589800+0000 mgr.y (mgr.14550) 711 : cluster 0 pgmap v1201: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.707935+0000 mon.a (mon.0) 3617 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: audit 2024-03-11T21:41:17.710239+0000 mon.a (mon.0) 3618 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]': finished 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.718677+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.882824+0000 osd.6 (osd.6) 331 : cluster 0 313.3s0 scrub starts 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:17.885807+0000 osd.6 (osd.6) 332 : cluster 0 313.3 scrub ok 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: audit 2024-03-11T21:41:17.905964+0000 mon.c (mon.2) 1248 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: audit 2024-03-11T21:41:17.906418+0000 mon.a (mon.0) 3620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.180563+0000 osd.5 (osd.5) 425 : cluster 0 1.1 scrub starts 2024-03-11T21:41:18.910 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.182505+0000 osd.5 (osd.5) 426 : cluster 0 1.1 scrub ok 2024-03-11T21:41:18.922 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:19.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:19.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:18 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:19.052 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:19.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.342541+0000 osd.1 (osd.1) 213 : cluster 0 5.1d scrub starts 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.344677+0000 osd.1 (osd.1) 214 : cluster 0 5.1d scrub ok 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.356803+0000 osd.0 (osd.0) 404 : cluster 0 39.4 scrub starts 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.359072+0000 osd.0 (osd.0) 405 : cluster 0 39.4 scrub ok 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.376240+0000 osd.3 (osd.3) 409 : cluster 0 6.6 scrub starts 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.378279+0000 osd.3 (osd.3) 410 : cluster 0 6.6 scrub ok 2024-03-11T21:41:19.149 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.476036+0000 osd.7 (osd.7) 402 : cluster 0 313.2s0 scrub starts 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.477905+0000 osd.7 (osd.7) 403 : cluster 0 313.2 scrub ok 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.589800+0000 mgr.y (mgr.14550) 711 : cluster 0 pgmap v1201: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.707935+0000 mon.a (mon.0) 3617 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: audit 2024-03-11T21:41:17.710239+0000 mon.a (mon.0) 3618 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]': finished 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.718677+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.882824+0000 osd.6 (osd.6) 331 : cluster 0 313.3s0 scrub starts 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:17.885807+0000 osd.6 (osd.6) 332 : cluster 0 313.3 scrub ok 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: audit 2024-03-11T21:41:17.905964+0000 mon.c (mon.2) 1248 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: audit 2024-03-11T21:41:17.906418+0000 mon.a (mon.0) 3620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.180563+0000 osd.5 (osd.5) 425 : cluster 0 1.1 scrub starts 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.182505+0000 osd.5 (osd.5) 426 : cluster 0 1.1 scrub ok 2024-03-11T21:41:19.150 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.342541+0000 osd.1 (osd.1) 213 : cluster 0 5.1d scrub starts 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.344677+0000 osd.1 (osd.1) 214 : cluster 0 5.1d scrub ok 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.356803+0000 osd.0 (osd.0) 404 : cluster 0 39.4 scrub starts 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.359072+0000 osd.0 (osd.0) 405 : cluster 0 39.4 scrub ok 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.376240+0000 osd.3 (osd.3) 409 : cluster 0 6.6 scrub starts 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.378279+0000 osd.3 (osd.3) 410 : cluster 0 6.6 scrub ok 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.476036+0000 osd.7 (osd.7) 402 : cluster 0 313.2s0 scrub starts 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.477905+0000 osd.7 (osd.7) 403 : cluster 0 313.2 scrub ok 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.589800+0000 mgr.y (mgr.14550) 711 : cluster 0 pgmap v1201: 246 pgs: 28 unknown, 218 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.707935+0000 mon.a (mon.0) 3617 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: audit 2024-03-11T21:41:17.710239+0000 mon.a (mon.0) 3618 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-130", "mode": "writeback"}]': finished 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.718677+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.882824+0000 osd.6 (osd.6) 331 : cluster 0 313.3s0 scrub starts 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:17.885807+0000 osd.6 (osd.6) 332 : cluster 0 313.3 scrub ok 2024-03-11T21:41:19.151 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: audit 2024-03-11T21:41:17.905964+0000 mon.c (mon.2) 1248 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: audit 2024-03-11T21:41:17.906418+0000 mon.a (mon.0) 3620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.180563+0000 osd.5 (osd.5) 425 : cluster 0 1.1 scrub starts 2024-03-11T21:41:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:18 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.182505+0000 osd.5 (osd.5) 426 : cluster 0 1.1 scrub ok 2024-03-11T21:41:19.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.181 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:19.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:19.307 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:19.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:19.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:19.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.558 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:19.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:19.689 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:19.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:19.820 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:19.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:19.948 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.341065+0000 osd.3 (osd.3) 411 : cluster 0 5.17 scrub starts 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.342520+0000 osd.3 (osd.3) 412 : cluster 0 5.17 scrub ok 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.462784+0000 osd.7 (osd.7) 404 : cluster 0 2.8 scrub starts 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.465407+0000 osd.7 (osd.7) 405 : cluster 0 2.8 scrub ok 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: audit 2024-03-11T21:41:18.726051+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: audit 2024-03-11T21:41:18.736443+0000 mon.c (mon.2) 1249 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.737093+0000 mon.a (mon.0) 3622 : cluster 0 osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: audit 2024-03-11T21:41:18.741633+0000 mon.a (mon.0) 3623 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.845959+0000 osd.6 (osd.6) 333 : cluster 0 39.8 scrub starts 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:18.848388+0000 osd.6 (osd.6) 334 : cluster 0 39.8 scrub ok 2024-03-11T21:41:20.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: audit 2024-03-11T21:41:18.993729+0000 mgr.y (mgr.14550) 712 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:20.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.726555+0000 mon.a (mon.0) 3624 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:20.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: audit 2024-03-11T21:41:19.728722+0000 mon.a (mon.0) 3625 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:20.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:19 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.735934+0000 mon.a (mon.0) 3626 : cluster 0 osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:41:20.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.079 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:20.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.341065+0000 osd.3 (osd.3) 411 : cluster 0 5.17 scrub starts 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.342520+0000 osd.3 (osd.3) 412 : cluster 0 5.17 scrub ok 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.462784+0000 osd.7 (osd.7) 404 : cluster 0 2.8 scrub starts 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.465407+0000 osd.7 (osd.7) 405 : cluster 0 2.8 scrub ok 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: audit 2024-03-11T21:41:18.726051+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: audit 2024-03-11T21:41:18.736443+0000 mon.c (mon.2) 1249 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.737093+0000 mon.a (mon.0) 3622 : cluster 0 osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: audit 2024-03-11T21:41:18.741633+0000 mon.a (mon.0) 3623 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.845959+0000 osd.6 (osd.6) 333 : cluster 0 39.8 scrub starts 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:18.848388+0000 osd.6 (osd.6) 334 : cluster 0 39.8 scrub ok 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: audit 2024-03-11T21:41:18.993729+0000 mgr.y (mgr.14550) 712 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.726555+0000 mon.a (mon.0) 3624 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: audit 2024-03-11T21:41:19.728722+0000 mon.a (mon.0) 3625 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.735934+0000 mon.a (mon.0) 3626 : cluster 0 osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.341065+0000 osd.3 (osd.3) 411 : cluster 0 5.17 scrub starts 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.342520+0000 osd.3 (osd.3) 412 : cluster 0 5.17 scrub ok 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.462784+0000 osd.7 (osd.7) 404 : cluster 0 2.8 scrub starts 2024-03-11T21:41:20.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.465407+0000 osd.7 (osd.7) 405 : cluster 0 2.8 scrub ok 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: audit 2024-03-11T21:41:18.726051+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: audit 2024-03-11T21:41:18.736443+0000 mon.c (mon.2) 1249 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.737093+0000 mon.a (mon.0) 3622 : cluster 0 osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: audit 2024-03-11T21:41:18.741633+0000 mon.a (mon.0) 3623 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]: dispatch 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.845959+0000 osd.6 (osd.6) 333 : cluster 0 39.8 scrub starts 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:18.848388+0000 osd.6 (osd.6) 334 : cluster 0 39.8 scrub ok 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: audit 2024-03-11T21:41:18.993729+0000 mgr.y (mgr.14550) 712 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.726555+0000 mon.a (mon.0) 3624 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: audit 2024-03-11T21:41:19.728722+0000 mon.a (mon.0) 3625 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-130"}]': finished 2024-03-11T21:41:20.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:19 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.735934+0000 mon.a (mon.0) 3626 : cluster 0 osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:41:20.210 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:20.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:20.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.469 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:20.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:20.599 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:20.705 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.733 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:20.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:20.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:20 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.435158+0000 osd.1 (osd.1) 215 : cluster 0 3.4 scrub starts 2024-03-11T21:41:20.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:20 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.438155+0000 osd.1 (osd.1) 216 : cluster 0 3.4 scrub ok 2024-03-11T21:41:20.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:20 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.558927+0000 osd.4 (osd.4) 395 : cluster 0 6.1e scrub starts 2024-03-11T21:41:20.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:20 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.561335+0000 osd.4 (osd.4) 396 : cluster 0 6.1e scrub ok 2024-03-11T21:41:20.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:20 smithi203 bash[23654]: cluster 2024-03-11T21:41:19.590446+0000 mgr.y (mgr.14550) 713 : cluster 0 pgmap v1204: 246 pgs: 246 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:41:20.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:20.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.977 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.435158+0000 osd.1 (osd.1) 215 : cluster 0 3.4 scrub starts 2024-03-11T21:41:20.977 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.438155+0000 osd.1 (osd.1) 216 : cluster 0 3.4 scrub ok 2024-03-11T21:41:20.977 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.558927+0000 osd.4 (osd.4) 395 : cluster 0 6.1e scrub starts 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.561335+0000 osd.4 (osd.4) 396 : cluster 0 6.1e scrub ok 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[21512]: cluster 2024-03-11T21:41:19.590446+0000 mgr.y (mgr.14550) 713 : cluster 0 pgmap v1204: 246 pgs: 246 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.435158+0000 osd.1 (osd.1) 215 : cluster 0 3.4 scrub starts 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.438155+0000 osd.1 (osd.1) 216 : cluster 0 3.4 scrub ok 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.558927+0000 osd.4 (osd.4) 395 : cluster 0 6.1e scrub starts 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.561335+0000 osd.4 (osd.4) 396 : cluster 0 6.1e scrub ok 2024-03-11T21:41:20.978 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:20 smithi143 bash[29284]: cluster 2024-03-11T21:41:19.590446+0000 mgr.y (mgr.14550) 713 : cluster 0 pgmap v1204: 246 pgs: 246 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:41:20.990 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:21.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:21.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:21.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:21.245 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:21.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:21.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:21.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:21.489 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:21.490 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:21.526 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:21.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:21.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:21.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:21.783 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:21.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:21.914 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.348212+0000 osd.3 (osd.3) 413 : cluster 0 4.0 scrub starts 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.351137+0000 osd.3 (osd.3) 414 : cluster 0 4.0 scrub ok 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.458583+0000 osd.1 (osd.1) 217 : cluster 0 39.17 scrub starts 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.464579+0000 osd.1 (osd.1) 218 : cluster 0 39.17 scrub ok 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.551620+0000 osd.4 (osd.4) 397 : cluster 0 6.2 scrub starts 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.553699+0000 osd.4 (osd.4) 398 : cluster 0 6.2 scrub ok 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.633591+0000 osd.2 (osd.2) 363 : cluster 0 6.f scrub starts 2024-03-11T21:41:21.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.635790+0000 osd.2 (osd.2) 364 : cluster 0 6.f scrub ok 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:20.770272+0000 mon.a (mon.0) 3627 : cluster 0 osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.518230+0000 mon.a (mon.0) 3628 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.528024+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: audit 2024-03-11T21:41:21.541264+0000 mon.c (mon.2) 1250 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[21512]: audit 2024-03-11T21:41:21.560628+0000 mon.a (mon.0) 3630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.348212+0000 osd.3 (osd.3) 413 : cluster 0 4.0 scrub starts 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.351137+0000 osd.3 (osd.3) 414 : cluster 0 4.0 scrub ok 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.458583+0000 osd.1 (osd.1) 217 : cluster 0 39.17 scrub starts 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.464579+0000 osd.1 (osd.1) 218 : cluster 0 39.17 scrub ok 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.551620+0000 osd.4 (osd.4) 397 : cluster 0 6.2 scrub starts 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.553699+0000 osd.4 (osd.4) 398 : cluster 0 6.2 scrub ok 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.633591+0000 osd.2 (osd.2) 363 : cluster 0 6.f scrub starts 2024-03-11T21:41:21.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.635790+0000 osd.2 (osd.2) 364 : cluster 0 6.f scrub ok 2024-03-11T21:41:21.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:20.770272+0000 mon.a (mon.0) 3627 : cluster 0 osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:41:21.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.518230+0000 mon.a (mon.0) 3628 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:21.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.528024+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:41:21.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: audit 2024-03-11T21:41:21.541264+0000 mon.c (mon.2) 1250 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:21.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:21 smithi143 bash[29284]: audit 2024-03-11T21:41:21.560628+0000 mon.a (mon.0) 3630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:22.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:22.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.348212+0000 osd.3 (osd.3) 413 : cluster 0 4.0 scrub starts 2024-03-11T21:41:22.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.351137+0000 osd.3 (osd.3) 414 : cluster 0 4.0 scrub ok 2024-03-11T21:41:22.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.458583+0000 osd.1 (osd.1) 217 : cluster 0 39.17 scrub starts 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.464579+0000 osd.1 (osd.1) 218 : cluster 0 39.17 scrub ok 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.551620+0000 osd.4 (osd.4) 397 : cluster 0 6.2 scrub starts 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.553699+0000 osd.4 (osd.4) 398 : cluster 0 6.2 scrub ok 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.633591+0000 osd.2 (osd.2) 363 : cluster 0 6.f scrub starts 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.635790+0000 osd.2 (osd.2) 364 : cluster 0 6.f scrub ok 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:20.770272+0000 mon.a (mon.0) 3627 : cluster 0 osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.518230+0000 mon.a (mon.0) 3628 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.528024+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: audit 2024-03-11T21:41:21.541264+0000 mon.c (mon.2) 1250 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:22.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:21 smithi203 bash[23654]: audit 2024-03-11T21:41:21.560628+0000 mon.a (mon.0) 3630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:22.040 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:22.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:22.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:22.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:22.285 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:22.302 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:22.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:22.432 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:22.514 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:22.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:22.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:22.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:22.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:22.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.337838+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.350943+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.481016+0000 osd.1 (osd.1) 219 : cluster 0 2.15 scrub starts 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.483161+0000 osd.7 (osd.7) 406 : cluster 0 5.9 scrub starts 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.483357+0000 osd.1 (osd.1) 220 : cluster 0 2.15 scrub ok 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.484753+0000 osd.7 (osd.7) 407 : cluster 0 5.9 scrub ok 2024-03-11T21:41:22.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.590967+0000 mgr.y (mgr.14550) 714 : cluster 0 pgmap v1208: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 264 B/s wr, 2 op/s 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.904122+0000 osd.6 (osd.6) 335 : cluster 0 313.7s0 scrub starts 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:21.907631+0000 osd.6 (osd.6) 336 : cluster 0 313.7 scrub ok 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.195178+0000 osd.5 (osd.5) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.197150+0000 osd.5 (osd.5) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: audit 2024-03-11T21:41:22.259449+0000 mon.a (mon.0) 3631 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: audit 2024-03-11T21:41:22.262725+0000 mon.c (mon.2) 1251 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: audit 2024-03-11T21:41:22.552805+0000 mon.a (mon.0) 3632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.559187+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: audit 2024-03-11T21:41:22.561392+0000 mon.c (mon.2) 1252 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:22 smithi203 bash[23654]: audit 2024-03-11T21:41:22.563797+0000 mon.a (mon.0) 3634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:22.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:22.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.337838+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-03-11T21:41:22.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.350943+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.481016+0000 osd.1 (osd.1) 219 : cluster 0 2.15 scrub starts 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.483161+0000 osd.7 (osd.7) 406 : cluster 0 5.9 scrub starts 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.483357+0000 osd.1 (osd.1) 220 : cluster 0 2.15 scrub ok 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.484753+0000 osd.7 (osd.7) 407 : cluster 0 5.9 scrub ok 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.590967+0000 mgr.y (mgr.14550) 714 : cluster 0 pgmap v1208: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 264 B/s wr, 2 op/s 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.904122+0000 osd.6 (osd.6) 335 : cluster 0 313.7s0 scrub starts 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:21.907631+0000 osd.6 (osd.6) 336 : cluster 0 313.7 scrub ok 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.195178+0000 osd.5 (osd.5) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.197150+0000 osd.5 (osd.5) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: audit 2024-03-11T21:41:22.259449+0000 mon.a (mon.0) 3631 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: audit 2024-03-11T21:41:22.262725+0000 mon.c (mon.2) 1251 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: audit 2024-03-11T21:41:22.552805+0000 mon.a (mon.0) 3632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.559187+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:41:22.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: audit 2024-03-11T21:41:22.561392+0000 mon.c (mon.2) 1252 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[21512]: audit 2024-03-11T21:41:22.563797+0000 mon.a (mon.0) 3634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.337838+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.350943+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.481016+0000 osd.1 (osd.1) 219 : cluster 0 2.15 scrub starts 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.483161+0000 osd.7 (osd.7) 406 : cluster 0 5.9 scrub starts 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.483357+0000 osd.1 (osd.1) 220 : cluster 0 2.15 scrub ok 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.484753+0000 osd.7 (osd.7) 407 : cluster 0 5.9 scrub ok 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.590967+0000 mgr.y (mgr.14550) 714 : cluster 0 pgmap v1208: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 264 B/s wr, 2 op/s 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.904122+0000 osd.6 (osd.6) 335 : cluster 0 313.7s0 scrub starts 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:21.907631+0000 osd.6 (osd.6) 336 : cluster 0 313.7 scrub ok 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.195178+0000 osd.5 (osd.5) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:41:22.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.197150+0000 osd.5 (osd.5) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: audit 2024-03-11T21:41:22.259449+0000 mon.a (mon.0) 3631 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: audit 2024-03-11T21:41:22.262725+0000 mon.c (mon.2) 1251 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: audit 2024-03-11T21:41:22.552805+0000 mon.a (mon.0) 3632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.559187+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: audit 2024-03-11T21:41:22.561392+0000 mon.c (mon.2) 1252 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:22 smithi143 bash[29284]: audit 2024-03-11T21:41:22.563797+0000 mon.a (mon.0) 3634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:22.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:23.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.094 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:23.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:23.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:23.354 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:23.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:23.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:23.538 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:23.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:23.609 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:23.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.738 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.600327+0000 osd.4 (osd.4) 399 : cluster 0 9.16 scrub starts 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.601686+0000 osd.4 (osd.4) 400 : cluster 0 9.16 scrub ok 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.894459+0000 osd.5 (osd.5) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: cluster 2024-03-11T21:41:22.894834+0000 osd.5 (osd.5) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: audit 2024-03-11T21:41:23.556133+0000 mon.a (mon.0) 3635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:23.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: audit 2024-03-11T21:41:23.563110+0000 mon.c (mon.2) 1253 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.567204+0000 mon.a (mon.0) 3636 : cluster 0 osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:41:23.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[29284]: audit 2024-03-11T21:41:23.568851+0000 mon.a (mon.0) 3637 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.853 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.600327+0000 osd.4 (osd.4) 399 : cluster 0 9.16 scrub starts 2024-03-11T21:41:23.853 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.601686+0000 osd.4 (osd.4) 400 : cluster 0 9.16 scrub ok 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.894459+0000 osd.5 (osd.5) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: cluster 2024-03-11T21:41:22.894834+0000 osd.5 (osd.5) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: audit 2024-03-11T21:41:23.556133+0000 mon.a (mon.0) 3635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: audit 2024-03-11T21:41:23.563110+0000 mon.c (mon.2) 1253 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.567204+0000 mon.a (mon.0) 3636 : cluster 0 osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:41:23.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:23 smithi143 bash[21512]: audit 2024-03-11T21:41:23.568851+0000 mon.a (mon.0) 3637 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.867 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:23.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.600327+0000 osd.4 (osd.4) 399 : cluster 0 9.16 scrub starts 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.601686+0000 osd.4 (osd.4) 400 : cluster 0 9.16 scrub ok 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.894459+0000 osd.5 (osd.5) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: cluster 2024-03-11T21:41:22.894834+0000 osd.5 (osd.5) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: audit 2024-03-11T21:41:23.556133+0000 mon.a (mon.0) 3635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: audit 2024-03-11T21:41:23.563110+0000 mon.c (mon.2) 1253 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.567204+0000 mon.a (mon.0) 3636 : cluster 0 osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:23 smithi203 bash[23654]: audit 2024-03-11T21:41:23.568851+0000 mon.a (mon.0) 3637 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:23.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:24.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:24.130 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:24.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:24.262 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:24.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:24.393 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:24.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:24.524 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:24.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:24.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:24.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:24.786 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:24.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:24.917 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:25.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.489348+0000 osd.7 (osd.7) 408 : cluster 0 9.11 scrub starts 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.490844+0000 osd.7 (osd.7) 409 : cluster 0 9.11 scrub ok 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.591421+0000 mgr.y (mgr.14550) 715 : cluster 0 pgmap v1211: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.911645+0000 osd.6 (osd.6) 337 : cluster 0 5.2 scrub starts 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:23.913957+0000 osd.6 (osd.6) 338 : cluster 0 5.2 scrub ok 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: audit 2024-03-11T21:41:24.564561+0000 mon.a (mon.0) 3638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: cluster 2024-03-11T21:41:24.569041+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: audit 2024-03-11T21:41:24.570421+0000 mon.c (mon.2) 1254 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:24 smithi203 bash[23654]: audit 2024-03-11T21:41:24.570992+0000 mon.a (mon.0) 3640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.489348+0000 osd.7 (osd.7) 408 : cluster 0 9.11 scrub starts 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.490844+0000 osd.7 (osd.7) 409 : cluster 0 9.11 scrub ok 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.591421+0000 mgr.y (mgr.14550) 715 : cluster 0 pgmap v1211: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.911645+0000 osd.6 (osd.6) 337 : cluster 0 5.2 scrub starts 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:23.913957+0000 osd.6 (osd.6) 338 : cluster 0 5.2 scrub ok 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: audit 2024-03-11T21:41:24.564561+0000 mon.a (mon.0) 3638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: cluster 2024-03-11T21:41:24.569041+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: audit 2024-03-11T21:41:24.570421+0000 mon.c (mon.2) 1254 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.032 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[29284]: audit 2024-03-11T21:41:24.570992+0000 mon.a (mon.0) 3640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.489348+0000 osd.7 (osd.7) 408 : cluster 0 9.11 scrub starts 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.490844+0000 osd.7 (osd.7) 409 : cluster 0 9.11 scrub ok 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.591421+0000 mgr.y (mgr.14550) 715 : cluster 0 pgmap v1211: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.911645+0000 osd.6 (osd.6) 337 : cluster 0 5.2 scrub starts 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:23.913957+0000 osd.6 (osd.6) 338 : cluster 0 5.2 scrub ok 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: audit 2024-03-11T21:41:24.564561+0000 mon.a (mon.0) 3638 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: cluster 2024-03-11T21:41:24.569041+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: audit 2024-03-11T21:41:24.570421+0000 mon.c (mon.2) 1254 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.033 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:24 smithi143 bash[21512]: audit 2024-03-11T21:41:24.570992+0000 mon.a (mon.0) 3640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]: dispatch 2024-03-11T21:41:25.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:25.153 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.178 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:25.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:25.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:25.412 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:25.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:25.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:25.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:25.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:25.809 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:25.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:25.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.967 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:26.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:25 smithi203 bash[23654]: cluster 2024-03-11T21:41:24.310087+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:26.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:25 smithi203 bash[23654]: cluster 2024-03-11T21:41:24.310504+0000 osd.3 (osd.3) 418 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:26.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:25 smithi203 bash[23654]: cluster 2024-03-11T21:41:25.565597+0000 mon.a (mon.0) 3641 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:26.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:25 smithi203 bash[23654]: audit 2024-03-11T21:41:25.568091+0000 mon.a (mon.0) 3642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]': finished 2024-03-11T21:41:26.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:25 smithi203 bash[23654]: cluster 2024-03-11T21:41:25.576347+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:41:26.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[21512]: cluster 2024-03-11T21:41:24.310087+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[21512]: cluster 2024-03-11T21:41:24.310504+0000 osd.3 (osd.3) 418 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[21512]: cluster 2024-03-11T21:41:25.565597+0000 mon.a (mon.0) 3641 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[21512]: audit 2024-03-11T21:41:25.568091+0000 mon.a (mon.0) 3642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]': finished 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[21512]: cluster 2024-03-11T21:41:25.576347+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[29284]: cluster 2024-03-11T21:41:24.310087+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[29284]: cluster 2024-03-11T21:41:24.310504+0000 osd.3 (osd.3) 418 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[29284]: cluster 2024-03-11T21:41:25.565597+0000 mon.a (mon.0) 3641 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[29284]: audit 2024-03-11T21:41:25.568091+0000 mon.a (mon.0) 3642 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-132", "mode": "writeback"}]': finished 2024-03-11T21:41:26.083 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:25 smithi143 bash[29284]: cluster 2024-03-11T21:41:25.576347+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:41:26.094 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:26.197 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.225 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:26.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:26.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:26.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:26.488 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:26.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:26.617 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:26.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:26.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:26.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:26.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:27.004 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:25.363152+0000 osd.3 (osd.3) 419 : cluster 0 9.2 scrub starts 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:25.364382+0000 osd.3 (osd.3) 420 : cluster 0 9.2 scrub ok 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:25.592258+0000 mgr.y (mgr.14550) 716 : cluster 0 pgmap v1214: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.299852+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.303554+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:41:27.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:26 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.573532+0000 mon.a (mon.0) 3644 : cluster 0 osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:41:27.080 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:25.363152+0000 osd.3 (osd.3) 419 : cluster 0 9.2 scrub starts 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:25.364382+0000 osd.3 (osd.3) 420 : cluster 0 9.2 scrub ok 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:25.592258+0000 mgr.y (mgr.14550) 716 : cluster 0 pgmap v1214: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.299852+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.303554+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.573532+0000 mon.a (mon.0) 3644 : cluster 0 osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:25.363152+0000 osd.3 (osd.3) 419 : cluster 0 9.2 scrub starts 2024-03-11T21:41:27.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:25.364382+0000 osd.3 (osd.3) 420 : cluster 0 9.2 scrub ok 2024-03-11T21:41:27.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:25.592258+0000 mgr.y (mgr.14550) 716 : cluster 0 pgmap v1214: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:27.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.299852+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:41:27.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.303554+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:41:27.082 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:26 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.573532+0000 mon.a (mon.0) 3644 : cluster 0 osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:41:27.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:27.135 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:27.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:27.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:27.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.647 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:27.903 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:27.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.325524+0000 osd.3 (osd.3) 421 : cluster 0 6.c scrub starts 2024-03-11T21:41:27.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.327411+0000 osd.3 (osd.3) 422 : cluster 0 6.c scrub ok 2024-03-11T21:41:27.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.481071+0000 osd.7 (osd.7) 410 : cluster 0 3.f scrub starts 2024-03-11T21:41:27.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.485632+0000 osd.7 (osd.7) 411 : cluster 0 3.f scrub ok 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.707140+0000 osd.2 (osd.2) 365 : cluster 0 5.d scrub starts 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.709260+0000 osd.2 (osd.2) 366 : cluster 0 5.d scrub ok 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.839982+0000 osd.1 (osd.1) 221 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:26.840423+0000 osd.1 (osd.1) 222 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.585865+0000 mon.a (mon.0) 3645 : cluster 0 osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: audit 2024-03-11T21:41:27.650282+0000 mon.c (mon.2) 1255 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: audit 2024-03-11T21:41:27.650710+0000 mon.a (mon.0) 3646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.665765+0000 osd.4 (osd.4) 401 : cluster 0 5.18 scrub starts 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.667831+0000 osd.4 (osd.4) 402 : cluster 0 5.18 scrub ok 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.325524+0000 osd.3 (osd.3) 421 : cluster 0 6.c scrub starts 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.327411+0000 osd.3 (osd.3) 422 : cluster 0 6.c scrub ok 2024-03-11T21:41:27.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.481071+0000 osd.7 (osd.7) 410 : cluster 0 3.f scrub starts 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.485632+0000 osd.7 (osd.7) 411 : cluster 0 3.f scrub ok 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.707140+0000 osd.2 (osd.2) 365 : cluster 0 5.d scrub starts 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.709260+0000 osd.2 (osd.2) 366 : cluster 0 5.d scrub ok 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.839982+0000 osd.1 (osd.1) 221 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:26.840423+0000 osd.1 (osd.1) 222 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.585865+0000 mon.a (mon.0) 3645 : cluster 0 osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: audit 2024-03-11T21:41:27.650282+0000 mon.c (mon.2) 1255 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: audit 2024-03-11T21:41:27.650710+0000 mon.a (mon.0) 3646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.665765+0000 osd.4 (osd.4) 401 : cluster 0 5.18 scrub starts 2024-03-11T21:41:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:27 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.667831+0000 osd.4 (osd.4) 402 : cluster 0 5.18 scrub ok 2024-03-11T21:41:28.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.325524+0000 osd.3 (osd.3) 421 : cluster 0 6.c scrub starts 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.327411+0000 osd.3 (osd.3) 422 : cluster 0 6.c scrub ok 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.481071+0000 osd.7 (osd.7) 410 : cluster 0 3.f scrub starts 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.485632+0000 osd.7 (osd.7) 411 : cluster 0 3.f scrub ok 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.707140+0000 osd.2 (osd.2) 365 : cluster 0 5.d scrub starts 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.709260+0000 osd.2 (osd.2) 366 : cluster 0 5.d scrub ok 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.839982+0000 osd.1 (osd.1) 221 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:26.840423+0000 osd.1 (osd.1) 222 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.585865+0000 mon.a (mon.0) 3645 : cluster 0 osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: audit 2024-03-11T21:41:27.650282+0000 mon.c (mon.2) 1255 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: audit 2024-03-11T21:41:27.650710+0000 mon.a (mon.0) 3646 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.665765+0000 osd.4 (osd.4) 401 : cluster 0 5.18 scrub starts 2024-03-11T21:41:28.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:27 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.667831+0000 osd.4 (osd.4) 402 : cluster 0 5.18 scrub ok 2024-03-11T21:41:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:28.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:28.159 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:28.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.289 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:28.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.418 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:28.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:28.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:28.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.677 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:28.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.187939+0000 osd.0 (osd.0) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.188313+0000 osd.0 (osd.0) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.461967+0000 osd.1 (osd.1) 223 : cluster 0 3.0 scrub starts 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.464985+0000 osd.1 (osd.1) 224 : cluster 0 3.0 scrub ok 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.592841+0000 mgr.y (mgr.14550) 717 : cluster 0 pgmap v1217: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.718076+0000 osd.2 (osd.2) 367 : cluster 0 5.a scrub starts 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.719610+0000 osd.2 (osd.2) 368 : cluster 0 5.a scrub ok 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.844909+0000 osd.6 (osd.6) 339 : cluster 0 1.6 scrub starts 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:27.847096+0000 osd.6 (osd.6) 340 : cluster 0 1.6 scrub ok 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: audit 2024-03-11T21:41:28.582881+0000 mon.a (mon.0) 3647 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:28.800 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:28.590023+0000 mon.a (mon.0) 3648 : cluster 0 osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:41:28.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: audit 2024-03-11T21:41:28.614142+0000 mon.c (mon.2) 1256 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:28.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: audit 2024-03-11T21:41:28.614466+0000 mon.a (mon.0) 3649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:28.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:28.703593+0000 osd.4 (osd.4) 403 : cluster 0 6.e scrub starts 2024-03-11T21:41:28.801 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[21512]: cluster 2024-03-11T21:41:28.707182+0000 osd.4 (osd.4) 404 : cluster 0 6.e scrub ok 2024-03-11T21:41:28.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:28.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.936 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:29.028 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.187939+0000 osd.0 (osd.0) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.188313+0000 osd.0 (osd.0) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.461967+0000 osd.1 (osd.1) 223 : cluster 0 3.0 scrub starts 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.464985+0000 osd.1 (osd.1) 224 : cluster 0 3.0 scrub ok 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.592841+0000 mgr.y (mgr.14550) 717 : cluster 0 pgmap v1217: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.718076+0000 osd.2 (osd.2) 367 : cluster 0 5.a scrub starts 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.719610+0000 osd.2 (osd.2) 368 : cluster 0 5.a scrub ok 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.844909+0000 osd.6 (osd.6) 339 : cluster 0 1.6 scrub starts 2024-03-11T21:41:29.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:27.847096+0000 osd.6 (osd.6) 340 : cluster 0 1.6 scrub ok 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: audit 2024-03-11T21:41:28.582881+0000 mon.a (mon.0) 3647 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:28.590023+0000 mon.a (mon.0) 3648 : cluster 0 osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: audit 2024-03-11T21:41:28.614142+0000 mon.c (mon.2) 1256 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: audit 2024-03-11T21:41:28.614466+0000 mon.a (mon.0) 3649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:28.703593+0000 osd.4 (osd.4) 403 : cluster 0 6.e scrub starts 2024-03-11T21:41:29.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:28 smithi203 bash[23654]: cluster 2024-03-11T21:41:28.707182+0000 osd.4 (osd.4) 404 : cluster 0 6.e scrub ok 2024-03-11T21:41:29.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:29.066 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.187939+0000 osd.0 (osd.0) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.188313+0000 osd.0 (osd.0) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.461967+0000 osd.1 (osd.1) 223 : cluster 0 3.0 scrub starts 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.464985+0000 osd.1 (osd.1) 224 : cluster 0 3.0 scrub ok 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.592841+0000 mgr.y (mgr.14550) 717 : cluster 0 pgmap v1217: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.718076+0000 osd.2 (osd.2) 367 : cluster 0 5.a scrub starts 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.719610+0000 osd.2 (osd.2) 368 : cluster 0 5.a scrub ok 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.844909+0000 osd.6 (osd.6) 339 : cluster 0 1.6 scrub starts 2024-03-11T21:41:29.076 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:27.847096+0000 osd.6 (osd.6) 340 : cluster 0 1.6 scrub ok 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: audit 2024-03-11T21:41:28.582881+0000 mon.a (mon.0) 3647 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:28.590023+0000 mon.a (mon.0) 3648 : cluster 0 osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: audit 2024-03-11T21:41:28.614142+0000 mon.c (mon.2) 1256 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: audit 2024-03-11T21:41:28.614466+0000 mon.a (mon.0) 3649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:28.703593+0000 osd.4 (osd.4) 403 : cluster 0 6.e scrub starts 2024-03-11T21:41:29.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:28 smithi143 bash[29284]: cluster 2024-03-11T21:41:28.707182+0000 osd.4 (osd.4) 404 : cluster 0 6.e scrub ok 2024-03-11T21:41:29.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.196 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:29.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.327 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:29.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:29.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:29.692 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:29.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:29.848 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:29.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:29.977 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:30.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:28.318303+0000 osd.0 (osd.0) 408 : cluster 0 6.0 scrub starts 2024-03-11T21:41:30.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:28.319665+0000 osd.0 (osd.0) 409 : cluster 0 6.0 scrub ok 2024-03-11T21:41:30.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: audit 2024-03-11T21:41:29.003055+0000 mgr.y (mgr.14550) 718 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:30.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.271544+0000 osd.5 (osd.5) 433 : cluster 0 5.f scrub starts 2024-03-11T21:41:30.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.273146+0000 osd.5 (osd.5) 434 : cluster 0 5.f scrub ok 2024-03-11T21:41:30.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: audit 2024-03-11T21:41:29.596654+0000 mon.a (mon.0) 3650 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:30.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.601948+0000 mon.a (mon.0) 3651 : cluster 0 osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:41:30.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.669299+0000 osd.4 (osd.4) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:41:30.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:29 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.674448+0000 osd.4 (osd.4) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:41:30.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:30.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:28.318303+0000 osd.0 (osd.0) 408 : cluster 0 6.0 scrub starts 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:28.319665+0000 osd.0 (osd.0) 409 : cluster 0 6.0 scrub ok 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: audit 2024-03-11T21:41:29.003055+0000 mgr.y (mgr.14550) 718 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.271544+0000 osd.5 (osd.5) 433 : cluster 0 5.f scrub starts 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.273146+0000 osd.5 (osd.5) 434 : cluster 0 5.f scrub ok 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: audit 2024-03-11T21:41:29.596654+0000 mon.a (mon.0) 3650 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:30.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.601948+0000 mon.a (mon.0) 3651 : cluster 0 osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.669299+0000 osd.4 (osd.4) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.674448+0000 osd.4 (osd.4) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:28.318303+0000 osd.0 (osd.0) 408 : cluster 0 6.0 scrub starts 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:28.319665+0000 osd.0 (osd.0) 409 : cluster 0 6.0 scrub ok 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: audit 2024-03-11T21:41:29.003055+0000 mgr.y (mgr.14550) 718 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.271544+0000 osd.5 (osd.5) 433 : cluster 0 5.f scrub starts 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.273146+0000 osd.5 (osd.5) 434 : cluster 0 5.f scrub ok 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: audit 2024-03-11T21:41:29.596654+0000 mon.a (mon.0) 3650 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.601948+0000 mon.a (mon.0) 3651 : cluster 0 osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.669299+0000 osd.4 (osd.4) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:41:30.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:29 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.674448+0000 osd.4 (osd.4) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:41:30.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:30.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:30.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:30.372 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:30.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:30.502 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:30.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:30.628 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:30.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:30.757 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:30.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:30.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:30.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:31.018 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:31.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:31.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.507924+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub starts 2024-03-11T21:41:31.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.509100+0000 osd.7 (osd.7) 413 : cluster 0 2.b scrub ok 2024-03-11T21:41:31.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.594008+0000 mgr.y (mgr.14550) 719 : cluster 0 pgmap v1219: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:31.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.848149+0000 osd.6 (osd.6) 341 : cluster 0 39.19 scrub starts 2024-03-11T21:41:31.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:29.852771+0000 osd.6 (osd.6) 342 : cluster 0 39.19 scrub ok 2024-03-11T21:41:31.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: cluster 2024-03-11T21:41:30.602812+0000 mon.a (mon.0) 3652 : cluster 0 osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:41:31.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: audit 2024-03-11T21:41:30.650090+0000 mon.c (mon.2) 1257 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:30 smithi203 bash[23654]: audit 2024-03-11T21:41:30.650495+0000 mon.a (mon.0) 3653 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.145 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.507924+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub starts 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.509100+0000 osd.7 (osd.7) 413 : cluster 0 2.b scrub ok 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.594008+0000 mgr.y (mgr.14550) 719 : cluster 0 pgmap v1219: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.848149+0000 osd.6 (osd.6) 341 : cluster 0 39.19 scrub starts 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:29.852771+0000 osd.6 (osd.6) 342 : cluster 0 39.19 scrub ok 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: cluster 2024-03-11T21:41:30.602812+0000 mon.a (mon.0) 3652 : cluster 0 osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: audit 2024-03-11T21:41:30.650090+0000 mon.c (mon.2) 1257 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[29284]: audit 2024-03-11T21:41:30.650495+0000 mon.a (mon.0) 3653 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.507924+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub starts 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.509100+0000 osd.7 (osd.7) 413 : cluster 0 2.b scrub ok 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.594008+0000 mgr.y (mgr.14550) 719 : cluster 0 pgmap v1219: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.848149+0000 osd.6 (osd.6) 341 : cluster 0 39.19 scrub starts 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:29.852771+0000 osd.6 (osd.6) 342 : cluster 0 39.19 scrub ok 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: cluster 2024-03-11T21:41:30.602812+0000 mon.a (mon.0) 3652 : cluster 0 osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: audit 2024-03-11T21:41:30.650090+0000 mon.c (mon.2) 1257 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:30 smithi143 bash[21512]: audit 2024-03-11T21:41:30.650495+0000 mon.a (mon.0) 3653 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:31.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:31.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:31.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:31.404 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:31.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:31.536 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:31.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.665 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:31.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:30.407895+0000 osd.1 (osd.1) 225 : cluster 0 39.7 scrub starts 2024-03-11T21:41:31.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:30.411626+0000 osd.1 (osd.1) 226 : cluster 0 39.7 scrub ok 2024-03-11T21:41:31.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:30.523104+0000 osd.7 (osd.7) 414 : cluster 0 1.9 scrub starts 2024-03-11T21:41:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:31.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:31.926 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:31.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:30.407895+0000 osd.1 (osd.1) 225 : cluster 0 39.7 scrub starts 2024-03-11T21:41:31.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:30.411626+0000 osd.1 (osd.1) 226 : cluster 0 39.7 scrub ok 2024-03-11T21:41:31.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:30.523104+0000 osd.7 (osd.7) 414 : cluster 0 1.9 scrub starts 2024-03-11T21:41:31.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:30.525001+0000 osd.7 (osd.7) 415 : cluster 0 1.9 scrub ok 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: audit 2024-03-11T21:41:31.525100+0000 mon.a (mon.0) 3654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.528728+0000 mon.a (mon.0) 3655 : cluster 0 osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: audit 2024-03-11T21:41:31.530230+0000 mon.c (mon.2) 1258 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: audit 2024-03-11T21:41:31.541166+0000 mon.a (mon.0) 3656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.688313+0000 osd.4 (osd.4) 407 : cluster 0 39.10 scrub starts 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.690546+0000 osd.4 (osd.4) 408 : cluster 0 39.10 scrub ok 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:30.407895+0000 osd.1 (osd.1) 225 : cluster 0 39.7 scrub starts 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:30.411626+0000 osd.1 (osd.1) 226 : cluster 0 39.7 scrub ok 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:30.523104+0000 osd.7 (osd.7) 414 : cluster 0 1.9 scrub starts 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:30.525001+0000 osd.7 (osd.7) 415 : cluster 0 1.9 scrub ok 2024-03-11T21:41:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: audit 2024-03-11T21:41:31.525100+0000 mon.a (mon.0) 3654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.528728+0000 mon.a (mon.0) 3655 : cluster 0 osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:41:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: audit 2024-03-11T21:41:31.530230+0000 mon.c (mon.2) 1258 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: audit 2024-03-11T21:41:31.541166+0000 mon.a (mon.0) 3656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.688313+0000 osd.4 (osd.4) 407 : cluster 0 39.10 scrub starts 2024-03-11T21:41:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:31 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.690546+0000 osd.4 (osd.4) 408 : cluster 0 39.10 scrub ok 2024-03-11T21:41:32.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:32.040 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:30.525001+0000 osd.7 (osd.7) 415 : cluster 0 1.9 scrub ok 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: audit 2024-03-11T21:41:31.525100+0000 mon.a (mon.0) 3654 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.528728+0000 mon.a (mon.0) 3655 : cluster 0 osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: audit 2024-03-11T21:41:31.530230+0000 mon.c (mon.2) 1258 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: audit 2024-03-11T21:41:31.541166+0000 mon.a (mon.0) 3656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]: dispatch 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.688313+0000 osd.4 (osd.4) 407 : cluster 0 39.10 scrub starts 2024-03-11T21:41:32.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:31 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.690546+0000 osd.4 (osd.4) 408 : cluster 0 39.10 scrub ok 2024-03-11T21:41:32.054 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:32.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.185 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:32.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:32.315 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:32.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:32.469 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:32.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:32.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:32.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:32.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:32.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.318852+0000 osd.0 (osd.0) 410 : cluster 0 2.13 scrub starts 2024-03-11T21:41:32.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.320680+0000 osd.0 (osd.0) 411 : cluster 0 2.13 scrub ok 2024-03-11T21:41:32.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.331267+0000 osd.3 (osd.3) 423 : cluster 0 5.13 scrub starts 2024-03-11T21:41:32.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.332294+0000 osd.3 (osd.3) 424 : cluster 0 5.13 scrub ok 2024-03-11T21:41:32.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.507303+0000 osd.7 (osd.7) 416 : cluster 0 9.17 scrub starts 2024-03-11T21:41:32.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.509811+0000 osd.7 (osd.7) 417 : cluster 0 9.17 scrub ok 2024-03-11T21:41:32.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:31.594524+0000 mgr.y (mgr.14550) 720 : cluster 0 pgmap v1223: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:32.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.535057+0000 mon.a (mon.0) 3657 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:32.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: audit 2024-03-11T21:41:32.542958+0000 mon.a (mon.0) 3658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:32.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:32 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.546702+0000 mon.a (mon.0) 3659 : cluster 0 osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:41:32.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:32.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:33.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.318852+0000 osd.0 (osd.0) 410 : cluster 0 2.13 scrub starts 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.320680+0000 osd.0 (osd.0) 411 : cluster 0 2.13 scrub ok 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.331267+0000 osd.3 (osd.3) 423 : cluster 0 5.13 scrub starts 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.332294+0000 osd.3 (osd.3) 424 : cluster 0 5.13 scrub ok 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.507303+0000 osd.7 (osd.7) 416 : cluster 0 9.17 scrub starts 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.509811+0000 osd.7 (osd.7) 417 : cluster 0 9.17 scrub ok 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:31.594524+0000 mgr.y (mgr.14550) 720 : cluster 0 pgmap v1223: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.535057+0000 mon.a (mon.0) 3657 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: audit 2024-03-11T21:41:32.542958+0000 mon.a (mon.0) 3658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.546702+0000 mon.a (mon.0) 3659 : cluster 0 osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.318852+0000 osd.0 (osd.0) 410 : cluster 0 2.13 scrub starts 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.320680+0000 osd.0 (osd.0) 411 : cluster 0 2.13 scrub ok 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.331267+0000 osd.3 (osd.3) 423 : cluster 0 5.13 scrub starts 2024-03-11T21:41:33.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.332294+0000 osd.3 (osd.3) 424 : cluster 0 5.13 scrub ok 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.507303+0000 osd.7 (osd.7) 416 : cluster 0 9.17 scrub starts 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.509811+0000 osd.7 (osd.7) 417 : cluster 0 9.17 scrub ok 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:31.594524+0000 mgr.y (mgr.14550) 720 : cluster 0 pgmap v1223: 246 pgs: 246 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.535057+0000 mon.a (mon.0) 3657 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: audit 2024-03-11T21:41:32.542958+0000 mon.a (mon.0) 3658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-132"}]': finished 2024-03-11T21:41:33.082 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:32 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.546702+0000 mon.a (mon.0) 3659 : cluster 0 osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:41:33.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:33.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:33.228 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:33.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:33.358 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:33.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.489 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:33.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:33.613 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:33.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:33.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:33.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:33.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:33.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.005 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:34.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.363698+0000 osd.0 (osd.0) 412 : cluster 0 6.10 scrub starts 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.365457+0000 osd.0 (osd.0) 413 : cluster 0 6.10 scrub ok 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.437729+0000 osd.1 (osd.1) 227 : cluster 0 39.c scrub starts 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.439994+0000 osd.1 (osd.1) 228 : cluster 0 39.c scrub ok 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.790456+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:32.795419+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.551096+0000 mon.a (mon.0) 3660 : cluster 0 osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:41:34.118 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.363698+0000 osd.0 (osd.0) 412 : cluster 0 6.10 scrub starts 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.365457+0000 osd.0 (osd.0) 413 : cluster 0 6.10 scrub ok 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.437729+0000 osd.1 (osd.1) 227 : cluster 0 39.c scrub starts 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.439994+0000 osd.1 (osd.1) 228 : cluster 0 39.c scrub ok 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.790456+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:32.795419+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-03-11T21:41:34.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:34 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.551096+0000 mon.a (mon.0) 3660 : cluster 0 osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:41:34.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:34.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:34.261 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:34.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.363698+0000 osd.0 (osd.0) 412 : cluster 0 6.10 scrub starts 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.365457+0000 osd.0 (osd.0) 413 : cluster 0 6.10 scrub ok 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.437729+0000 osd.1 (osd.1) 227 : cluster 0 39.c scrub starts 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.439994+0000 osd.1 (osd.1) 228 : cluster 0 39.c scrub ok 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.790456+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:32.795419+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-03-11T21:41:34.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:34 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.551096+0000 mon.a (mon.0) 3660 : cluster 0 osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:41:34.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.394 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:34.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.525 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:34.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:34.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.789 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:34.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:34.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:35.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.047 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:35.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.470637+0000 osd.1 (osd.1) 229 : cluster 0 6.4 scrub starts 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.472910+0000 osd.1 (osd.1) 230 : cluster 0 6.4 scrub ok 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.595186+0000 mgr.y (mgr.14550) 721 : cluster 0 pgmap v1226: 214 pgs: 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.721787+0000 osd.4 (osd.4) 409 : cluster 0 6.1a scrub starts 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.724988+0000 osd.4 (osd.4) 410 : cluster 0 6.1a scrub ok 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.741884+0000 osd.6 (osd.6) 345 : cluster 0 5.1f scrub starts 2024-03-11T21:41:35.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:33.751921+0000 osd.6 (osd.6) 346 : cluster 0 5.1f scrub ok 2024-03-11T21:41:35.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:34.229773+0000 osd.5 (osd.5) 435 : cluster 0 2.2 scrub starts 2024-03-11T21:41:35.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:34.231986+0000 osd.5 (osd.5) 436 : cluster 0 2.2 scrub ok 2024-03-11T21:41:35.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: cluster 2024-03-11T21:41:34.560008+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e793: 8 total, 8 up, 8 in 2024-03-11T21:41:35.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: audit 2024-03-11T21:41:34.565060+0000 mon.c (mon.2) 1259 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:35 smithi203 bash[23654]: audit 2024-03-11T21:41:34.565637+0000 mon.a (mon.0) 3662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.178 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:35.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:35.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:35.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.470637+0000 osd.1 (osd.1) 229 : cluster 0 6.4 scrub starts 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.472910+0000 osd.1 (osd.1) 230 : cluster 0 6.4 scrub ok 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.595186+0000 mgr.y (mgr.14550) 721 : cluster 0 pgmap v1226: 214 pgs: 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.721787+0000 osd.4 (osd.4) 409 : cluster 0 6.1a scrub starts 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.724988+0000 osd.4 (osd.4) 410 : cluster 0 6.1a scrub ok 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.741884+0000 osd.6 (osd.6) 345 : cluster 0 5.1f scrub starts 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:33.751921+0000 osd.6 (osd.6) 346 : cluster 0 5.1f scrub ok 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:34.229773+0000 osd.5 (osd.5) 435 : cluster 0 2.2 scrub starts 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:34.231986+0000 osd.5 (osd.5) 436 : cluster 0 2.2 scrub ok 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: cluster 2024-03-11T21:41:34.560008+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e793: 8 total, 8 up, 8 in 2024-03-11T21:41:35.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: audit 2024-03-11T21:41:34.565060+0000 mon.c (mon.2) 1259 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[21512]: audit 2024-03-11T21:41:34.565637+0000 mon.a (mon.0) 3662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.470637+0000 osd.1 (osd.1) 229 : cluster 0 6.4 scrub starts 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.472910+0000 osd.1 (osd.1) 230 : cluster 0 6.4 scrub ok 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.595186+0000 mgr.y (mgr.14550) 721 : cluster 0 pgmap v1226: 214 pgs: 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.721787+0000 osd.4 (osd.4) 409 : cluster 0 6.1a scrub starts 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.724988+0000 osd.4 (osd.4) 410 : cluster 0 6.1a scrub ok 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.741884+0000 osd.6 (osd.6) 345 : cluster 0 5.1f scrub starts 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:33.751921+0000 osd.6 (osd.6) 346 : cluster 0 5.1f scrub ok 2024-03-11T21:41:35.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:34.229773+0000 osd.5 (osd.5) 435 : cluster 0 2.2 scrub starts 2024-03-11T21:41:35.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:34.231986+0000 osd.5 (osd.5) 436 : cluster 0 2.2 scrub ok 2024-03-11T21:41:35.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: cluster 2024-03-11T21:41:34.560008+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e793: 8 total, 8 up, 8 in 2024-03-11T21:41:35.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: audit 2024-03-11T21:41:34.565060+0000 mon.c (mon.2) 1259 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:35 smithi143 bash[29284]: audit 2024-03-11T21:41:34.565637+0000 mon.a (mon.0) 3662 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:35.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:35.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:35.583 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:35.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.713 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:35.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:35.838 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:35.910 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:41:35.910 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:41:35.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:35.961 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:36.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:36.091 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:36.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.217 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- 172.21.15.143:0/323872639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 msgr2=0x7fc19c108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/323872639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c108310 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0098f0 tx=0x7fc18c009c00 comp rx=0 tx=0).stop 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- 172.21.15.143:0/323872639 shutdown_connections 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/323872639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c108310 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/323872639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/323872639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 0x7fc19c102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:36.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- 172.21.15.143:0/323872639 >> 172.21.15.143:0/323872639 conn(0x7fc19c078eb0 msgr2=0x7fc19c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- 172.21.15.143:0/323872639 shutdown_connections 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- 172.21.15.143:0/323872639 wait complete. 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 librados: starting msgr at 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 librados: starting objecter 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 Processor -- start 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- start start 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 librados: setting wanted keys 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 librados: calling monclient init 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 0x7fc19c0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c111620 con 0x7fc19c107f30 2024-03-11T21:41:36.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c1114a0 con 0x7fc19c101ec0 2024-03-11T21:41:36.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc1a0df6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c1117a0 con 0x7fc19c102880 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc199d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc199d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33096/0 (socket says 172.21.15.143:33096) 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc199d9b700 1 -- 172.21.15.143:0/1973282995 learned_addr learned my addr 172.21.15.143:0/1973282995 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19a59c700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 0x7fc19c0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19ad9d700 1 -- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 msgr2=0x7fc19c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19ad9d700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:36.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19ad9d700 1 -- 172.21.15.143:0/1973282995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 msgr2=0x7fc19c0740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.328+0000 7fc19ad9d700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 0x7fc19c0740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc19ad9d700 1 -- 172.21.15.143:0/1973282995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c1a61d0 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc199d9b700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc19ad9d700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c074b60 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fc194009cd0 tx=0x7fc19400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 10 client.?.objecter ms_handle_connect 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19400cdc0 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c1a6460 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc194003a00 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc1a0df6700 10 client.30819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940044f0 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1a69f0 con 0x7fc19c107f30 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc1a0df6700 1 librados: init done 2024-03-11T21:41:36.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c04faa0 con 0x7fc19c107f30 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc19401a020 con 0x7fc19c107f30 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 0x7fc184071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(794..794 src has 87..794) v4 ==== 10294+0+0 (secure 0 0 0) 0x7fc19408e690 con 0x7fc19c107f30 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 10 client.30819.objecter ms_dispatch 0x7fc19c001550 osd_map(794..794 src has 87..794) v4 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 3 client.30819.objecter handle_osd_map got epochs [794,794] > 0 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 3 client.30819.objecter handle_osd_map decoding full epoch 794 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc19a59c700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 0x7fc184071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 20 client.30819.objecter dump_active .. 0 homeless 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc18b7fe700 10 client.30819.objecter ms_handle_connect 0x7fc18406f820 2024-03-11T21:41:36.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.332+0000 7fc19a59c700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 0x7fc184071ce0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009fd0 tx=0x7fc18c019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:36.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.323 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.336+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc194010040 con 0x7fc19c107f30 2024-03-11T21:41:36.341 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:36.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:36.471 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:36.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:36.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc19c1a6d10 con 0x7fc19c107f30 2024-03-11T21:41:36.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.601 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:36.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.412087+0000 osd.3 (osd.3) 425 : cluster 0 3.2 scrub starts 2024-03-11T21:41:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.415621+0000 osd.3 (osd.3) 426 : cluster 0 3.2 scrub ok 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.424962+0000 osd.7 (osd.7) 418 : cluster 0 9.e scrub starts 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.427428+0000 osd.7 (osd.7) 419 : cluster 0 9.e scrub ok 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: audit 2024-03-11T21:41:35.557595+0000 mon.a (mon.0) 3663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.566495+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e794: 8 total, 8 up, 8 in 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: audit 2024-03-11T21:41:35.580366+0000 mon.c (mon.2) 1260 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: audit 2024-03-11T21:41:35.582192+0000 mon.a (mon.0) 3665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.595851+0000 mgr.y (mgr.14550) 722 : cluster 0 pgmap v1229: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.840403+0000 osd.2 (osd.2) 369 : cluster 0 39.1f scrub starts 2024-03-11T21:41:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:36 smithi203 bash[23654]: cluster 2024-03-11T21:41:35.844647+0000 osd.2 (osd.2) 370 : cluster 0 39.1f scrub ok 2024-03-11T21:41:36.731 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:36.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:36.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.412087+0000 osd.3 (osd.3) 425 : cluster 0 3.2 scrub starts 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.415621+0000 osd.3 (osd.3) 426 : cluster 0 3.2 scrub ok 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.424962+0000 osd.7 (osd.7) 418 : cluster 0 9.e scrub starts 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.427428+0000 osd.7 (osd.7) 419 : cluster 0 9.e scrub ok 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: audit 2024-03-11T21:41:35.557595+0000 mon.a (mon.0) 3663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.566495+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e794: 8 total, 8 up, 8 in 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: audit 2024-03-11T21:41:35.580366+0000 mon.c (mon.2) 1260 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: audit 2024-03-11T21:41:35.582192+0000 mon.a (mon.0) 3665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.595851+0000 mgr.y (mgr.14550) 722 : cluster 0 pgmap v1229: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.840403+0000 osd.2 (osd.2) 369 : cluster 0 39.1f scrub starts 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[21512]: cluster 2024-03-11T21:41:35.844647+0000 osd.2 (osd.2) 370 : cluster 0 39.1f scrub ok 2024-03-11T21:41:36.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.412087+0000 osd.3 (osd.3) 425 : cluster 0 3.2 scrub starts 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.415621+0000 osd.3 (osd.3) 426 : cluster 0 3.2 scrub ok 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.424962+0000 osd.7 (osd.7) 418 : cluster 0 9.e scrub starts 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.427428+0000 osd.7 (osd.7) 419 : cluster 0 9.e scrub ok 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: audit 2024-03-11T21:41:35.557595+0000 mon.a (mon.0) 3663 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.566495+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e794: 8 total, 8 up, 8 in 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: audit 2024-03-11T21:41:35.580366+0000 mon.c (mon.2) 1260 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: audit 2024-03-11T21:41:35.582192+0000 mon.a (mon.0) 3665 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.595851+0000 mgr.y (mgr.14550) 722 : cluster 0 pgmap v1229: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.840403+0000 osd.2 (osd.2) 369 : cluster 0 39.1f scrub starts 2024-03-11T21:41:36.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:36 smithi143 bash[29284]: cluster 2024-03-11T21:41:35.844647+0000 osd.2 (osd.2) 370 : cluster 0 39.1f scrub ok 2024-03-11T21:41:36.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.990 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:37.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:37.120 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:37.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:37.250 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:37.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.379 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:37.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:37.502 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:37.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.564+0000 7fc18b7fe700 1 -- 172.21.15.143:0/1973282995 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v796) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc19405b040 con 0x7fc19c107f30 2024-03-11T21:41:37.552 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 10 librados: watch_flush enter 2024-03-11T21:41:37.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 10 librados: watch_flush exit 2024-03-11T21:41:37.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 20 client.30819.objecter shutdown clearing up homeless session... 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 10 client.30819.objecter successfully canceled tick 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 msgr2=0x7fc184071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 0x7fc184071ce0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009fd0 tx=0x7fc18c019040 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 msgr2=0x7fc19c074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c074b60 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fc194009cd0 tx=0x7fc19400c620 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 shutdown_connections 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc18406f820 0x7fc184071ce0 unknown :-1 s=CLOSED pgs=3447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c107f30 0x7fc19c074b60 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19c101ec0 0x7fc19c0740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 --2- 172.21.15.143:0/1973282995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c102880 0x7fc19c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 >> 172.21.15.143:0/1973282995 conn(0x7fc19c078eb0 msgr2=0x7fc19c105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 shutdown_connections 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 -- 172.21.15.143:0/1973282995 wait complete. 2024-03-11T21:41:37.555 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.568+0000 7fc1a0df6700 1 librados: shutdown 2024-03-11T21:41:37.568 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:41:37.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- 172.21.15.143:0/542805619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0102050 msgr2=0x7f87b0105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- 172.21.15.143:0/542805619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0102050 0x7f87b0105b80 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f87a00099c0 tx=0x7f87a0009cd0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- 172.21.15.143:0/542805619 shutdown_connections 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- 172.21.15.143:0/542805619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b0106260 0x7f87b010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- 172.21.15.143:0/542805619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b01016a0 0x7f87b0101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- 172.21.15.143:0/542805619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0102050 0x7f87b0105b80 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- 172.21.15.143:0/542805619 >> 172.21.15.143:0/542805619 conn(0x7f87b00772e0 msgr2=0x7f87b00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- 172.21.15.143:0/542805619 shutdown_connections 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- 172.21.15.143:0/542805619 wait complete. 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 librados: starting msgr at 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 librados: starting objecter 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 Processor -- start 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- start start 2024-03-11T21:41:37.581 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 librados: setting wanted keys 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 librados: calling monclient init 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 0x7f87b019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 0x7f87b019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0113a20 con 0x7f87b01016a0 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01138a0 con 0x7f87b0102050 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b7a2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0113ba0 con 0x7f87b0106260 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b57c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b57c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:58502/0 (socket says 172.21.15.143:58502) 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b5fc7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 0x7f87b019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.592+0000 7f87b57c6700 1 -- 172.21.15.143:0/118744990 learned_addr learned my addr 172.21.15.143:0/118744990 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b4fc5700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 0x7f87b019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 -- 172.21.15.143:0/118744990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 msgr2=0x7f87b019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.582 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 0x7f87b019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 -- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 msgr2=0x7f87b019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 -- 172.21.15.143:0/118744990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b019f080 con 0x7f87b0106260 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b57c6700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b4fc5700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 0x7f87b019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b5fc7700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 0x7f87b019e940 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f87a800b670 tx=0x7f87a80076c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 10 client.?.objecter ms_handle_connect 0x7f87b0106260 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8009230 con 0x7f87b0106260 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f87a80093d0 con 0x7f87b0106260 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b7a2a700 1 -- 172.21.15.143:0/118744990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b019f370 con 0x7f87b0106260 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b7a2a700 10 client.42029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:37.583 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b7a2a700 1 -- 172.21.15.143:0/118744990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b01a6bf0 con 0x7f87b0106260 2024-03-11T21:41:37.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b7a2a700 1 librados: init done 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8017470 con 0x7f87b0106260 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8790005120 con 0x7f87b0106260 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f87a801e020 con 0x7f87b0106260 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 0x7f879c071c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(796..796 src has 87..796) v4 ==== 10289+0+0 (secure 0 0 0) 0x7f87a808ed70 con 0x7f87b0106260 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 10 client.42029.objecter ms_dispatch 0x7f87b0001550 osd_map(796..796 src has 87..796) v4 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 3 client.42029.objecter handle_osd_map got epochs [796,796] > 0 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87a67fc700 3 client.42029.objecter handle_osd_map decoding full epoch 796 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.596+0000 7f87b57c6700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 0x7f879c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.600+0000 7f87a67fc700 20 client.42029.objecter dump_active .. 0 homeless 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.600+0000 7f87a67fc700 10 client.42029.objecter ms_handle_connect 0x7f879c06f740 2024-03-11T21:41:37.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.600+0000 7f87b57c6700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 0x7f879c071c00 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005950 tx=0x7f87ac00b500 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.604+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f87a8016e90 con 0x7f87b0106260 2024-03-11T21:41:37.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:37.633 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: audit 2024-03-11T21:41:36.566084+0000 mon.a (mon.0) 3666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: audit 2024-03-11T21:41:36.575029+0000 mon.c (mon.2) 1261 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: cluster 2024-03-11T21:41:36.575753+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e795: 8 total, 8 up, 8 in 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: audit 2024-03-11T21:41:36.576280+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: audit 2024-03-11T21:41:36.577065+0000 mon.a (mon.0) 3669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: cluster 2024-03-11T21:41:36.713484+0000 osd.6 (osd.6) 347 : cluster 0 2.14 scrub starts 2024-03-11T21:41:37.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: cluster 2024-03-11T21:41:36.715486+0000 osd.6 (osd.6) 348 : cluster 0 2.14 scrub ok 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[21512]: audit 2024-03-11T21:41:37.297806+0000 mon.c (mon.2) 1262 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: audit 2024-03-11T21:41:36.566084+0000 mon.a (mon.0) 3666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: audit 2024-03-11T21:41:36.575029+0000 mon.c (mon.2) 1261 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: cluster 2024-03-11T21:41:36.575753+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e795: 8 total, 8 up, 8 in 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: audit 2024-03-11T21:41:36.576280+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: audit 2024-03-11T21:41:36.577065+0000 mon.a (mon.0) 3669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: cluster 2024-03-11T21:41:36.713484+0000 osd.6 (osd.6) 347 : cluster 0 2.14 scrub starts 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: cluster 2024-03-11T21:41:36.715486+0000 osd.6 (osd.6) 348 : cluster 0 2.14 scrub ok 2024-03-11T21:41:37.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:37 smithi143 bash[29284]: audit 2024-03-11T21:41:37.297806+0000 mon.c (mon.2) 1262 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- 172.21.15.143:0/4243241999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd806b050 msgr2=0x7f9fd8106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- 172.21.15.143:0/4243241999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd806b050 0x7f9fd8106b40 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009990 tx=0x7f9fc8009ca0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- 172.21.15.143:0/4243241999 shutdown_connections 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- 172.21.15.143:0/4243241999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9fd8107220 0x7f9fd810edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- 172.21.15.143:0/4243241999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fd806a6a0 0x7f9fd806aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- 172.21.15.143:0/4243241999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd806b050 0x7f9fd8106b40 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- 172.21.15.143:0/4243241999 >> 172.21.15.143:0/4243241999 conn(0x7f9fd8077de0 msgr2=0x7f9fd80781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- 172.21.15.143:0/4243241999 shutdown_connections 2024-03-11T21:41:37.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- 172.21.15.143:0/4243241999 wait complete. 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 librados: starting msgr at 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 librados: starting objecter 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 Processor -- start 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 -- start start 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 librados: setting wanted keys 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 librados: calling monclient init 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9fd806a6a0 0x7f9fd819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.720+0000 7f9fe009f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fd806b050 0x7f9fd819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 0x7f9fd8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8111690 con 0x7f9fd806a6a0 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8111510 con 0x7f9fd806b050 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8111810 con 0x7f9fd8107220 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 0x7f9fd8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 0x7f9fd8197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33116/0 (socket says 172.21.15.143:33116) 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fdde3b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9fd806a6a0 0x7f9fd819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 learned_addr learned my addr 172.21.15.143:0/1061357964 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fd806b050 msgr2=0x7f9fd819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fd806b050 0x7f9fd819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9fd806a6a0 msgr2=0x7f9fd819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9fd806a6a0 0x7f9fd819d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd81981f0 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 0x7f9fd8197a90 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c370 tx=0x7f9fd000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 10 client.?.objecter ms_handle_connect 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000ce20 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9fd00090d0 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd8198480 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd00046b0 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 10 client.42035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd8100cc0 con 0x7f9fd8107220 2024-03-11T21:41:37.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 librados: init done 2024-03-11T21:41:37.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9fd81035b0 con 0x7f9fd8107220 2024-03-11T21:41:37.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9fd001a020 con 0x7f9fd8107220 2024-03-11T21:41:37.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9fc406f820 0x7f9fc4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(796..796 src has 87..796) v4 ==== 10289+0+0 (secure 0 0 0) 0x7f9fd008e790 con 0x7f9fd8107220 2024-03-11T21:41:37.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 osd_map(796..796 src has 87..796) v4 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 3 client.42035.objecter handle_osd_map got epochs [796,796] > 0 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 3 client.42035.objecter handle_osd_map decoding full epoch 796 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9fc406f820 0x7f9fc4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 20 client.42035.objecter dump_active .. 0 homeless 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v796) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9fd0010040 con 0x7f9fd8107220 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 10 client.42035.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 1 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9fd80629f0 0x7f9fd8062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 1 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 0 1 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 4 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 10 client.42035.objecter _send_command 1 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd8198fc0 con 0x7f9fd80629f0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fc406f820 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9fc406f820 0x7f9fc4071ce0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8198bd0 tx=0x7f9fd400b3c0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.724+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9fd80629f0 0x7f9fd8062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd80629f0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9fd80629f0 0x7f9fd8062dd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd8198fc0 con 0x7f9fd80629f0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 10 client.42035.objecter _finish_command 1 = system:0 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 4 2024-03-11T21:41:37.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.728+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 0 1 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 10 client.42035.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 2 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 2 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd804faa0 osd=0 3 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 0 2 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 4 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 10 client.42035.objecter _send_command 2 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd8062470 con 0x7f9fd80629f0 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd8062470 con 0x7f9fd80629f0 2024-03-11T21:41:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fceffd700 10 client.42035.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 4 2024-03-11T21:41:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.744+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 0 2 2024-03-11T21:41:37.732 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:41:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 10 client.42035.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f9fd81099b0 0x7f9fd8100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 1 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 4 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 10 client.42035.objecter _send_command 3 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd8102cc0 con 0x7f9fd81099b0 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f9fd81099b0 0x7f9fd8100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd81099b0 2024-03-11T21:41:37.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f9fd81099b0 0x7f9fd8100fe0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd8102cc0 con 0x7f9fd81099b0 2024-03-11T21:41:37.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:41:37.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 10 client.42035.objecter _finish_command 3 = system:0 2024-03-11T21:41:37.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 4 2024-03-11T21:41:37.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.748+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 1 3 2024-03-11T21:41:37.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 10 client.42035.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 4 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 4 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd802d830 osd=1 3 2024-03-11T21:41:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 1 4 2024-03-11T21:41:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 4 2024-03-11T21:41:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 10 client.42035.objecter _send_command 4 2024-03-11T21:41:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.764+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd802dbf0 con 0x7f9fd81099b0 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd802dbf0 con 0x7f9fd81099b0 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 10 client.42035.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 4 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 1 4 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 10 client.42035.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 5 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9fd8109090 0x7f9fd802ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 5 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 2 5 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 4 2024-03-11T21:41:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 10 client.42035.objecter _send_command 5 2024-03-11T21:41:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd802d3c0 con 0x7f9fd8109090 2024-03-11T21:41:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9fd8109090 0x7f9fd802ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd8109090 2024-03-11T21:41:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9fd8109090 0x7f9fd802ce80 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd802d3c0 con 0x7f9fd8109090 2024-03-11T21:41:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:41:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 10 client.42035.objecter _finish_command 5 = system:0 2024-03-11T21:41:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 4 2024-03-11T21:41:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.768+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 2 5 2024-03-11T21:41:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 10 client.42035.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 6 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 6 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd8108f20 osd=2 3 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 2 6 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 4 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 10 client.42035.objecter _send_command 6 2024-03-11T21:41:37.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.784+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd802d3c0 con 0x7f9fd8109090 2024-03-11T21:41:37.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd802d3c0 con 0x7f9fd8109090 2024-03-11T21:41:37.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 10 client.42035.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 4 2024-03-11T21:41:37.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 2 6 2024-03-11T21:41:37.774 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:41:37.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 10 client.42035.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 7 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9fd8101cf0 0x7f9fd8102150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 7 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 3 7 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 4 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 10 client.42035.objecter _send_command 7 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd81b47d0 con 0x7f9fd8101cf0 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9fd8101cf0 0x7f9fd8102150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd8101cf0 2024-03-11T21:41:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9fd8101cf0 0x7f9fd8102150 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd81b47d0 con 0x7f9fd8101cf0 2024-03-11T21:41:37.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:41:37.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 10 client.42035.objecter _finish_command 7 = system:0 2024-03-11T21:41:37.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 4 2024-03-11T21:41:37.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.788+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 3 7 2024-03-11T21:41:37.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 10 client.42035.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 8 2024-03-11T21:41:37.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 8 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd802d3c0 osd=3 3 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 3 8 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 4 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 10 client.42035.objecter _send_command 8 2024-03-11T21:41:37.792 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.804+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd8101770 con 0x7f9fd8101cf0 2024-03-11T21:41:37.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd8101770 con 0x7f9fd8101cf0 2024-03-11T21:41:37.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 10 client.42035.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 4 2024-03-11T21:41:37.793 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 3 8 2024-03-11T21:41:37.794 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:41:37.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 10 client.42035.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 9 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9fd81af8c0 0x7f9fd81b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 9 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 4 9 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 4 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 10 client.42035.objecter _send_command 9 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd81b81c0 con 0x7f9fd81af8c0 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9fd81af8c0 0x7f9fd81b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd81af8c0 2024-03-11T21:41:37.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.808+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9fd81af8c0 0x7f9fd81b2f40 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.812+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd81b81c0 con 0x7f9fd81af8c0 2024-03-11T21:41:37.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.812+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:41:37.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.812+0000 7f9fceffd700 10 client.42035.objecter _finish_command 9 = system:0 2024-03-11T21:41:37.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.812+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 4 2024-03-11T21:41:37.796 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.812+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 4 9 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 10 client.42035.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 10 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 10 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd8101770 osd=4 3 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 4 10 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 4 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 10 client.42035.objecter _send_command 10 2024-03-11T21:41:37.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.828+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd81b3480 con 0x7f9fd81af8c0 2024-03-11T21:41:37.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd81b3480 con 0x7f9fd81af8c0 2024-03-11T21:41:37.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 10 client.42035.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 4 2024-03-11T21:41:37.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 4 10 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 10 client.42035.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 11 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9fd81b3ba0 0x7f9fd81b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 11 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 5 11 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 4 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 10 client.42035.objecter _send_command 11 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd81bb3a0 con 0x7f9fd81b3ba0 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9fd81b3ba0 0x7f9fd81b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd81b3ba0 2024-03-11T21:41:37.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fdd63a700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9fd81b3ba0 0x7f9fd81b3fe0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd81bb3a0 con 0x7f9fd81b3ba0 2024-03-11T21:41:37.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:41:37.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 10 client.42035.objecter _finish_command 11 = system:0 2024-03-11T21:41:37.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 4 2024-03-11T21:41:37.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.836+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 5 11 2024-03-11T21:41:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.840+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f8790005410 con 0x7f87b0106260 2024-03-11T21:41:37.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 10 client.42035.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 12 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 12 2024-03-11T21:41:37.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b3480 osd=5 3 2024-03-11T21:41:37.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 5 12 2024-03-11T21:41:37.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 4 2024-03-11T21:41:37.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 10 client.42035.objecter _send_command 12 2024-03-11T21:41:37.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.856+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd81b8a30 con 0x7f9fd81b3ba0 2024-03-11T21:41:37.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd81b8a30 con 0x7f9fd81b3ba0 2024-03-11T21:41:37.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 10 client.42035.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 4 2024-03-11T21:41:37.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 5 12 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 10 client.42035.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 13 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9fd81b90b0 0x7f9fd81b9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 13 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 6 13 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 4 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 10 client.42035.objecter _send_command 13 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd81be740 con 0x7f9fd81b90b0 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9fd81b90b0 0x7f9fd81b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fde63c700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9fd81b90b0 0x7f9fd81b9510 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.864+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd81b90b0 2024-03-11T21:41:37.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.868+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd81be740 con 0x7f9fd81b90b0 2024-03-11T21:41:37.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.868+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:41:37.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.868+0000 7f9fceffd700 10 client.42035.objecter _finish_command 13 = system:0 2024-03-11T21:41:37.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.868+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 4 2024-03-11T21:41:37.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.868+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 6 13 2024-03-11T21:41:37.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 10 client.42035.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 14 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 14 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b8a30 osd=6 3 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 6 14 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 4 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 10 client.42035.objecter _send_command 14 2024-03-11T21:41:37.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.884+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd81b9a50 con 0x7f9fd81b90b0 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd81b9a50 con 0x7f9fd81b90b0 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fceffd700 10 client.42035.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 4 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 6 14 2024-03-11T21:41:37.875 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 10 client.42035.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 15 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9fd81ba170 0x7f9fd81ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 15 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 7 15 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 4 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 10 client.42035.objecter _send_command 15 2024-03-11T21:41:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fd81bac40 con 0x7f9fd81ba170 2024-03-11T21:41:37.877 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.888+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9fd81ba170 0x7f9fd81ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:41:37.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 10 client.42035.objecter ms_handle_connect 0x7f9fd81ba170 2024-03-11T21:41:37.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fdde3b700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9fd81ba170 0x7f9fd81ba5b0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:41:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:37.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9fd81bac40 con 0x7f9fd81ba170 2024-03-11T21:41:37.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:41:37.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 10 client.42035.objecter _finish_command 15 = system:0 2024-03-11T21:41:37.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 4 2024-03-11T21:41:37.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.892+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 7 15 2024-03-11T21:41:37.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 10 client.42035.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign -1 16 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 20 client.42035.objecter _get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_remove -1 16 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 20 client.42035.objecter get_session s=0x7f9fd81b9a50 osd=7 3 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 15 client.42035.objecter _session_command_op_assign 7 16 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 4 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 10 client.42035.objecter _send_command 16 2024-03-11T21:41:37.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.912+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9fd81bac40 con 0x7f9fd81ba170 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fceffd700 1 -- 172.21.15.143:0/1061357964 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9fd81bac40 con 0x7f9fd81ba170 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fceffd700 10 client.42035.objecter ms_dispatch 0x7f9fd8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fceffd700 10 client.42035.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fceffd700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 4 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fceffd700 15 client.42035.objecter _session_command_op_remove 7 16 2024-03-11T21:41:37.901 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:41:37.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 librados: watch_flush enter 2024-03-11T21:41:37.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 librados: watch_flush exit 2024-03-11T21:41:37.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.0 2024-03-11T21:41:37.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9fd80629f0 msgr2=0x7f9fd8062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f9fd80629f0 0x7f9fd8062dd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd804faa0 osd=0 1 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.1 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f9fd81099b0 msgr2=0x7f9fd8100fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f9fd81099b0 0x7f9fd8100fe0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d830 osd=1 1 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.2 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9fd8109090 msgr2=0x7f9fd802ce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f9fd8109090 0x7f9fd802ce80 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8108f20 osd=2 1 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.3 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9fd8101cf0 msgr2=0x7f9fd8102150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f9fd8101cf0 0x7f9fd8102150 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd802d3c0 osd=3 1 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.4 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9fd81af8c0 msgr2=0x7f9fd81b2f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f9fd81af8c0 0x7f9fd81b2f40 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd8101770 osd=4 1 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.5 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9fd81b3ba0 msgr2=0x7f9fd81b3fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f9fd81b3ba0 0x7f9fd81b3fe0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b3480 osd=5 1 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.6 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9fd81b90b0 msgr2=0x7f9fd81b9510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f9fd81b90b0 0x7f9fd81b9510 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b8a30 osd=6 1 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter close_session for osd.7 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9fd81ba170 msgr2=0x7f9fd81ba5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f9fd81ba170 0x7f9fd81ba5b0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter put_session s=0x7f9fd81b9a50 osd=7 1 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 20 client.42035.objecter shutdown clearing up homeless session... 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 10 client.42035.objecter successfully canceled tick 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 reap_dead start 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 reap_dead start 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9fc406f820 msgr2=0x7f9fc4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9fc406f820 0x7f9fc4071ce0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8198bd0 tx=0x7f9fd400b3c0 comp rx=0 tx=0).stop 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 msgr2=0x7f9fd8197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 reap_dead start 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 --2- 172.21.15.143:0/1061357964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9fd8107220 0x7f9fd8197a90 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c370 tx=0x7f9fd000c680 comp rx=0 tx=0).stop 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 reap_dead start 2024-03-11T21:41:37.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fde63c700 1 -- 172.21.15.143:0/1061357964 reap_dead start 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 shutdown_connections 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 >> 172.21.15.143:0/1061357964 conn(0x7f9fd8077de0 msgr2=0x7f9fd80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 shutdown_connections 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 -- 172.21.15.143:0/1061357964 wait complete. 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:37.916+0000 7f9fe009f700 1 librados: shutdown 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:41:37.906 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:41:37.907 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:41:37.907 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:41:37.907 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:41:37.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: audit 2024-03-11T21:41:36.566084+0000 mon.a (mon.0) 3666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: audit 2024-03-11T21:41:36.575029+0000 mon.c (mon.2) 1261 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: cluster 2024-03-11T21:41:36.575753+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e795: 8 total, 8 up, 8 in 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: audit 2024-03-11T21:41:36.576280+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: audit 2024-03-11T21:41:36.577065+0000 mon.a (mon.0) 3669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:38.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: cluster 2024-03-11T21:41:36.713484+0000 osd.6 (osd.6) 347 : cluster 0 2.14 scrub starts 2024-03-11T21:41:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: cluster 2024-03-11T21:41:36.715486+0000 osd.6 (osd.6) 348 : cluster 0 2.14 scrub ok 2024-03-11T21:41:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:37 smithi203 bash[23654]: audit 2024-03-11T21:41:37.297806+0000 mon.c (mon.2) 1262 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:38.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.125 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:38.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:38.254 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:38.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:38.385 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:38.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.576+0000 7f87a67fc700 1 -- 172.21.15.143:0/118744990 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v797) v1 ==== 113+0+0 (secure 0 0 0) 0x7f87a805b820 con 0x7f87b0106260 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 10 librados: watch_flush enter 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 10 librados: watch_flush exit 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 20 client.42029.objecter shutdown clearing up homeless session... 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 10 client.42029.objecter successfully canceled tick 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 msgr2=0x7f879c071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 0x7f879c071c00 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005950 tx=0x7f87ac00b500 comp rx=0 tx=0).stop 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 msgr2=0x7f87b019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 0x7f87b019e940 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f87a800b670 tx=0x7f87a80076c0 comp rx=0 tx=0).stop 2024-03-11T21:41:38.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 shutdown_connections 2024-03-11T21:41:38.571 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f879c06f740 0x7f879c071c00 unknown :-1 s=CLOSED pgs=3448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b01016a0 0x7f87b019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b0106260 0x7f87b019e940 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 --2- 172.21.15.143:0/118744990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87b0102050 0x7f87b019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.580+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 >> 172.21.15.143:0/118744990 conn(0x7f87b00772e0 msgr2=0x7f87b0103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.584+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 shutdown_connections 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.584+0000 7f879bfff700 1 -- 172.21.15.143:0/118744990 wait complete. 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:41:38.584+0000 7f879bfff700 1 librados: shutdown 2024-03-11T21:41:38.573 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:41:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:38.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:38.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:38.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.399679+0000 osd.3 (osd.3) 427 : cluster 0 9.1a scrub starts 2024-03-11T21:41:38.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.401267+0000 osd.3 (osd.3) 428 : cluster 0 9.1a scrub ok 2024-03-11T21:41:38.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.413180+0000 osd.0 (osd.0) 414 : cluster 0 6.7 scrub starts 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.414910+0000 osd.0 (osd.0) 415 : cluster 0 6.7 scrub ok 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.566490+0000 mon.a (mon.0) 3670 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.568194+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.568274+0000 mon.a (mon.0) 3672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.572343+0000 mon.a (mon.0) 3673 : cluster 0 osdmap e796: 8 total, 8 up, 8 in 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.574900+0000 mon.c (mon.2) 1263 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.577626+0000 mon.a (mon.0) 3674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.596432+0000 mgr.y (mgr.14550) 723 : cluster 0 pgmap v1232: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.730700+0000 mon.c (mon.2) 1264 : audit 0 from='client.? 172.21.15.143:0/1061357964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.846187+0000 mon.c (mon.2) 1265 : audit 1 from='client.? 172.21.15.143:0/118744990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:37.846521+0000 mon.a (mon.0) 3675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.870285+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:37.871736+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: cluster 2024-03-11T21:41:38.569338+0000 mon.a (mon.0) 3676 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:38.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:38 smithi203 bash[23654]: audit 2024-03-11T21:41:38.571615+0000 mon.a (mon.0) 3677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]': finished 2024-03-11T21:41:38.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:38.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:38.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.399679+0000 osd.3 (osd.3) 427 : cluster 0 9.1a scrub starts 2024-03-11T21:41:38.943 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.401267+0000 osd.3 (osd.3) 428 : cluster 0 9.1a scrub ok 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.413180+0000 osd.0 (osd.0) 414 : cluster 0 6.7 scrub starts 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.414910+0000 osd.0 (osd.0) 415 : cluster 0 6.7 scrub ok 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.566490+0000 mon.a (mon.0) 3670 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.568194+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.568274+0000 mon.a (mon.0) 3672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.572343+0000 mon.a (mon.0) 3673 : cluster 0 osdmap e796: 8 total, 8 up, 8 in 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.574900+0000 mon.c (mon.2) 1263 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.577626+0000 mon.a (mon.0) 3674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.596432+0000 mgr.y (mgr.14550) 723 : cluster 0 pgmap v1232: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.730700+0000 mon.c (mon.2) 1264 : audit 0 from='client.? 172.21.15.143:0/1061357964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.846187+0000 mon.c (mon.2) 1265 : audit 1 from='client.? 172.21.15.143:0/118744990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:37.846521+0000 mon.a (mon.0) 3675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.870285+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-03-11T21:41:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:37.871736+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: cluster 2024-03-11T21:41:38.569338+0000 mon.a (mon.0) 3676 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[29284]: audit 2024-03-11T21:41:38.571615+0000 mon.a (mon.0) 3677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]': finished 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.399679+0000 osd.3 (osd.3) 427 : cluster 0 9.1a scrub starts 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.401267+0000 osd.3 (osd.3) 428 : cluster 0 9.1a scrub ok 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.413180+0000 osd.0 (osd.0) 414 : cluster 0 6.7 scrub starts 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.414910+0000 osd.0 (osd.0) 415 : cluster 0 6.7 scrub ok 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.566490+0000 mon.a (mon.0) 3670 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.568194+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.143:0/1973282995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.568274+0000 mon.a (mon.0) 3672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.572343+0000 mon.a (mon.0) 3673 : cluster 0 osdmap e796: 8 total, 8 up, 8 in 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.574900+0000 mon.c (mon.2) 1263 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.577626+0000 mon.a (mon.0) 3674 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]: dispatch 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.596432+0000 mgr.y (mgr.14550) 723 : cluster 0 pgmap v1232: 246 pgs: 7 creating+peering, 25 unknown, 214 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.730700+0000 mon.c (mon.2) 1264 : audit 0 from='client.? 172.21.15.143:0/1061357964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.846187+0000 mon.c (mon.2) 1265 : audit 1 from='client.? 172.21.15.143:0/118744990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:37.846521+0000 mon.a (mon.0) 3675 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.870285+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:37.871736+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: cluster 2024-03-11T21:41:38.569338+0000 mon.a (mon.0) 3676 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:38.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:38 smithi143 bash[21512]: audit 2024-03-11T21:41:38.571615+0000 mon.a (mon.0) 3677 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-134", "mode": "writeback"}]': finished 2024-03-11T21:41:38.961 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:39.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:39.076 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:39.091 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:39.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.220 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:39.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:39.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:39.479 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:39.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:39.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:39.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: cluster 2024-03-11T21:41:38.430941+0000 osd.7 (osd.7) 420 : cluster 0 1.2 scrub starts 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: cluster 2024-03-11T21:41:38.433370+0000 osd.7 (osd.7) 421 : cluster 0 1.2 scrub ok 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: audit 2024-03-11T21:41:38.571834+0000 mon.a (mon.0) 3678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: cluster 2024-03-11T21:41:38.584076+0000 mon.a (mon.0) 3679 : cluster 0 osdmap e797: 8 total, 8 up, 8 in 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: audit 2024-03-11T21:41:38.638473+0000 mon.c (mon.2) 1266 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: audit 2024-03-11T21:41:38.638879+0000 mon.a (mon.0) 3680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:39 smithi203 bash[23654]: audit 2024-03-11T21:41:39.005252+0000 mgr.y (mgr.14550) 724 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:39.742 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:39.845 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:39.857 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: cluster 2024-03-11T21:41:38.430941+0000 osd.7 (osd.7) 420 : cluster 0 1.2 scrub starts 2024-03-11T21:41:39.857 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: cluster 2024-03-11T21:41:38.433370+0000 osd.7 (osd.7) 421 : cluster 0 1.2 scrub ok 2024-03-11T21:41:39.857 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: audit 2024-03-11T21:41:38.571834+0000 mon.a (mon.0) 3678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: cluster 2024-03-11T21:41:38.584076+0000 mon.a (mon.0) 3679 : cluster 0 osdmap e797: 8 total, 8 up, 8 in 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: audit 2024-03-11T21:41:38.638473+0000 mon.c (mon.2) 1266 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: audit 2024-03-11T21:41:38.638879+0000 mon.a (mon.0) 3680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[21512]: audit 2024-03-11T21:41:39.005252+0000 mgr.y (mgr.14550) 724 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: cluster 2024-03-11T21:41:38.430941+0000 osd.7 (osd.7) 420 : cluster 0 1.2 scrub starts 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: cluster 2024-03-11T21:41:38.433370+0000 osd.7 (osd.7) 421 : cluster 0 1.2 scrub ok 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: audit 2024-03-11T21:41:38.571834+0000 mon.a (mon.0) 3678 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: cluster 2024-03-11T21:41:38.584076+0000 mon.a (mon.0) 3679 : cluster 0 osdmap e797: 8 total, 8 up, 8 in 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: audit 2024-03-11T21:41:38.638473+0000 mon.c (mon.2) 1266 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: audit 2024-03-11T21:41:38.638879+0000 mon.a (mon.0) 3680 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:39.858 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:39 smithi143 bash[29284]: audit 2024-03-11T21:41:39.005252+0000 mgr.y (mgr.14550) 724 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:39.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:39.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.005 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:40.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:40.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:40.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:40.268 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:40.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:40.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.527 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:40.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:40.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:40.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: cluster 2024-03-11T21:41:39.597656+0000 mgr.y (mgr.14550) 725 : cluster 0 pgmap v1234: 246 pgs: 7 creating+peering, 239 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:40.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: audit 2024-03-11T21:41:39.604301+0000 mon.a (mon.0) 3681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:40.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: cluster 2024-03-11T21:41:39.612037+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e798: 8 total, 8 up, 8 in 2024-03-11T21:41:40.693 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: audit 2024-03-11T21:41:39.616560+0000 mon.c (mon.2) 1267 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: audit 2024-03-11T21:41:39.617407+0000 mon.a (mon.0) 3683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: cluster 2024-03-11T21:41:39.827442+0000 osd.2 (osd.2) 373 : cluster 0 39.16 scrub starts 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[29284]: cluster 2024-03-11T21:41:39.829545+0000 osd.2 (osd.2) 374 : cluster 0 39.16 scrub ok 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: cluster 2024-03-11T21:41:39.597656+0000 mgr.y (mgr.14550) 725 : cluster 0 pgmap v1234: 246 pgs: 7 creating+peering, 239 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: audit 2024-03-11T21:41:39.604301+0000 mon.a (mon.0) 3681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: cluster 2024-03-11T21:41:39.612037+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e798: 8 total, 8 up, 8 in 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: audit 2024-03-11T21:41:39.616560+0000 mon.c (mon.2) 1267 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: audit 2024-03-11T21:41:39.617407+0000 mon.a (mon.0) 3683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: cluster 2024-03-11T21:41:39.827442+0000 osd.2 (osd.2) 373 : cluster 0 39.16 scrub starts 2024-03-11T21:41:40.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:40 smithi143 bash[21512]: cluster 2024-03-11T21:41:39.829545+0000 osd.2 (osd.2) 374 : cluster 0 39.16 scrub ok 2024-03-11T21:41:40.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:40.790 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:40.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:40.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: cluster 2024-03-11T21:41:39.597656+0000 mgr.y (mgr.14550) 725 : cluster 0 pgmap v1234: 246 pgs: 7 creating+peering, 239 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:40.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: audit 2024-03-11T21:41:39.604301+0000 mon.a (mon.0) 3681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:40.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: cluster 2024-03-11T21:41:39.612037+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e798: 8 total, 8 up, 8 in 2024-03-11T21:41:40.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: audit 2024-03-11T21:41:39.616560+0000 mon.c (mon.2) 1267 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: audit 2024-03-11T21:41:39.617407+0000 mon.a (mon.0) 3683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]: dispatch 2024-03-11T21:41:40.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: cluster 2024-03-11T21:41:39.827442+0000 osd.2 (osd.2) 373 : cluster 0 39.16 scrub starts 2024-03-11T21:41:40.904 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:40 smithi203 bash[23654]: cluster 2024-03-11T21:41:39.829545+0000 osd.2 (osd.2) 374 : cluster 0 39.16 scrub ok 2024-03-11T21:41:40.919 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:41.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:41.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:41.086 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:41:41.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:41:41.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:41.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:41.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:41.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.443 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:41.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:41.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:41.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.832 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:41.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.963 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:42.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:41:42.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:42.107 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:42.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:42.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:42.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.332370+0000 osd.0 (osd.0) 416 : cluster 0 5.3 scrub starts 2024-03-11T21:41:42.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.335165+0000 osd.0 (osd.0) 417 : cluster 0 5.3 scrub ok 2024-03-11T21:41:42.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.604666+0000 mon.a (mon.0) 3684 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:42.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: audit 2024-03-11T21:41:40.606786+0000 mon.a (mon.0) 3685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:42.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.614711+0000 mon.a (mon.0) 3686 : cluster 0 osdmap e799: 8 total, 8 up, 8 in 2024-03-11T21:41:42.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.660258+0000 osd.4 (osd.4) 411 : cluster 0 39.b scrub starts 2024-03-11T21:41:42.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:40.664670+0000 osd.4 (osd.4) 412 : cluster 0 39.b scrub ok 2024-03-11T21:41:42.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:42 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.541659+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e800: 8 total, 8 up, 8 in 2024-03-11T21:41:42.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:42.346 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.332370+0000 osd.0 (osd.0) 416 : cluster 0 5.3 scrub starts 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.335165+0000 osd.0 (osd.0) 417 : cluster 0 5.3 scrub ok 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.604666+0000 mon.a (mon.0) 3684 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: audit 2024-03-11T21:41:40.606786+0000 mon.a (mon.0) 3685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.614711+0000 mon.a (mon.0) 3686 : cluster 0 osdmap e799: 8 total, 8 up, 8 in 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.660258+0000 osd.4 (osd.4) 411 : cluster 0 39.b scrub starts 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:40.664670+0000 osd.4 (osd.4) 412 : cluster 0 39.b scrub ok 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.541659+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e800: 8 total, 8 up, 8 in 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.332370+0000 osd.0 (osd.0) 416 : cluster 0 5.3 scrub starts 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.335165+0000 osd.0 (osd.0) 417 : cluster 0 5.3 scrub ok 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.604666+0000 mon.a (mon.0) 3684 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:42.347 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: audit 2024-03-11T21:41:40.606786+0000 mon.a (mon.0) 3685 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-134"}]': finished 2024-03-11T21:41:42.348 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.614711+0000 mon.a (mon.0) 3686 : cluster 0 osdmap e799: 8 total, 8 up, 8 in 2024-03-11T21:41:42.348 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.660258+0000 osd.4 (osd.4) 411 : cluster 0 39.b scrub starts 2024-03-11T21:41:42.348 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:40.664670+0000 osd.4 (osd.4) 412 : cluster 0 39.b scrub ok 2024-03-11T21:41:42.348 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:42 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.541659+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e800: 8 total, 8 up, 8 in 2024-03-11T21:41:42.361 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:42.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:42.493 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:42.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:42.622 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:42.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:42.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:42.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:42.885 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:42.988 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:43.016 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:43.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:41:43.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:43.141 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:43.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:43.252 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.373673+0000 osd.0 (osd.0) 418 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.375158+0000 osd.0 (osd.0) 419 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.448647+0000 osd.7 (osd.7) 422 : cluster 0 1.0 scrub starts 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.450891+0000 osd.7 (osd.7) 423 : cluster 0 1.0 scrub ok 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.518945+0000 osd.3 (osd.3) 429 : cluster 0 39.11 scrub starts 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.521287+0000 osd.3 (osd.3) 430 : cluster 0 39.11 scrub ok 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.598481+0000 mgr.y (mgr.14550) 726 : cluster 0 pgmap v1238: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.820334+0000 osd.2 (osd.2) 375 : cluster 0 2.1 scrub starts 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:41.822589+0000 osd.2 (osd.2) 376 : cluster 0 2.1 scrub ok 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.236786+0000 osd.5 (osd.5) 437 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.239040+0000 osd.5 (osd.5) 438 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.552487+0000 mon.a (mon.0) 3688 : cluster 0 osdmap e801: 8 total, 8 up, 8 in 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: audit 2024-03-11T21:41:42.554006+0000 mon.c (mon.2) 1268 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: audit 2024-03-11T21:41:42.555291+0000 mon.a (mon.0) 3689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.628965+0000 osd.4 (osd.4) 413 : cluster 0 5.1 scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.630266+0000 osd.4 (osd.4) 414 : cluster 0 5.1 scrub ok 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.808349+0000 osd.2 (osd.2) 377 : cluster 0 2.e scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.810764+0000 osd.2 (osd.2) 378 : cluster 0 2.e scrub ok 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.373673+0000 osd.0 (osd.0) 418 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.375158+0000 osd.0 (osd.0) 419 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.448647+0000 osd.7 (osd.7) 422 : cluster 0 1.0 scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.450891+0000 osd.7 (osd.7) 423 : cluster 0 1.0 scrub ok 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.518945+0000 osd.3 (osd.3) 429 : cluster 0 39.11 scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.521287+0000 osd.3 (osd.3) 430 : cluster 0 39.11 scrub ok 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.598481+0000 mgr.y (mgr.14550) 726 : cluster 0 pgmap v1238: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.820334+0000 osd.2 (osd.2) 375 : cluster 0 2.1 scrub starts 2024-03-11T21:41:43.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:41.822589+0000 osd.2 (osd.2) 376 : cluster 0 2.1 scrub ok 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.236786+0000 osd.5 (osd.5) 437 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.239040+0000 osd.5 (osd.5) 438 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.552487+0000 mon.a (mon.0) 3688 : cluster 0 osdmap e801: 8 total, 8 up, 8 in 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: audit 2024-03-11T21:41:42.554006+0000 mon.c (mon.2) 1268 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: audit 2024-03-11T21:41:42.555291+0000 mon.a (mon.0) 3689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.628965+0000 osd.4 (osd.4) 413 : cluster 0 5.1 scrub starts 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.630266+0000 osd.4 (osd.4) 414 : cluster 0 5.1 scrub ok 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.808349+0000 osd.2 (osd.2) 377 : cluster 0 2.e scrub starts 2024-03-11T21:41:43.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:43 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.810764+0000 osd.2 (osd.2) 378 : cluster 0 2.e scrub ok 2024-03-11T21:41:43.263 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.373673+0000 osd.0 (osd.0) 418 : cluster 0 5.1e deep-scrub starts 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.375158+0000 osd.0 (osd.0) 419 : cluster 0 5.1e deep-scrub ok 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.448647+0000 osd.7 (osd.7) 422 : cluster 0 1.0 scrub starts 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.450891+0000 osd.7 (osd.7) 423 : cluster 0 1.0 scrub ok 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.518945+0000 osd.3 (osd.3) 429 : cluster 0 39.11 scrub starts 2024-03-11T21:41:43.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.521287+0000 osd.3 (osd.3) 430 : cluster 0 39.11 scrub ok 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.598481+0000 mgr.y (mgr.14550) 726 : cluster 0 pgmap v1238: 214 pgs: 214 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.820334+0000 osd.2 (osd.2) 375 : cluster 0 2.1 scrub starts 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:41.822589+0000 osd.2 (osd.2) 376 : cluster 0 2.1 scrub ok 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.236786+0000 osd.5 (osd.5) 437 : cluster 0 5.16 deep-scrub starts 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.239040+0000 osd.5 (osd.5) 438 : cluster 0 5.16 deep-scrub ok 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.552487+0000 mon.a (mon.0) 3688 : cluster 0 osdmap e801: 8 total, 8 up, 8 in 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: audit 2024-03-11T21:41:42.554006+0000 mon.c (mon.2) 1268 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: audit 2024-03-11T21:41:42.555291+0000 mon.a (mon.0) 3689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.628965+0000 osd.4 (osd.4) 413 : cluster 0 5.1 scrub starts 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.630266+0000 osd.4 (osd.4) 414 : cluster 0 5.1 scrub ok 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.808349+0000 osd.2 (osd.2) 377 : cluster 0 2.e scrub starts 2024-03-11T21:41:43.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:43 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.810764+0000 osd.2 (osd.2) 378 : cluster 0 2.e scrub ok 2024-03-11T21:41:43.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:43.395 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:43.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:43.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:43.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:43.666 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:43.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:43.796 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:43.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:43.927 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:44.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:44.058 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:44.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.426318+0000 osd.7 (osd.7) 424 : cluster 0 9.d scrub starts 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.428699+0000 osd.7 (osd.7) 425 : cluster 0 9.d scrub ok 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.680174+0000 osd.6 (osd.6) 349 : cluster 0 39.1e scrub starts 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:42.681615+0000 osd.6 (osd.6) 350 : cluster 0 39.1e scrub ok 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: audit 2024-03-11T21:41:43.542861+0000 mon.a (mon.0) 3690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:44.172 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.547757+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e802: 8 total, 8 up, 8 in 2024-03-11T21:41:44.173 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: audit 2024-03-11T21:41:43.557741+0000 mon.c (mon.2) 1269 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.173 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: audit 2024-03-11T21:41:43.558879+0000 mon.a (mon.0) 3692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.173 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.760768+0000 osd.2 (osd.2) 379 : cluster 0 2.10 scrub starts 2024-03-11T21:41:44.173 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:44 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.763268+0000 osd.2 (osd.2) 380 : cluster 0 2.10 scrub ok 2024-03-11T21:41:44.188 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:44.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.426318+0000 osd.7 (osd.7) 424 : cluster 0 9.d scrub starts 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.428699+0000 osd.7 (osd.7) 425 : cluster 0 9.d scrub ok 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.680174+0000 osd.6 (osd.6) 349 : cluster 0 39.1e scrub starts 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:42.681615+0000 osd.6 (osd.6) 350 : cluster 0 39.1e scrub ok 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: audit 2024-03-11T21:41:43.542861+0000 mon.a (mon.0) 3690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.547757+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e802: 8 total, 8 up, 8 in 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: audit 2024-03-11T21:41:43.557741+0000 mon.c (mon.2) 1269 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: audit 2024-03-11T21:41:43.558879+0000 mon.a (mon.0) 3692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.760768+0000 osd.2 (osd.2) 379 : cluster 0 2.10 scrub starts 2024-03-11T21:41:44.303 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.763268+0000 osd.2 (osd.2) 380 : cluster 0 2.10 scrub ok 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.426318+0000 osd.7 (osd.7) 424 : cluster 0 9.d scrub starts 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.428699+0000 osd.7 (osd.7) 425 : cluster 0 9.d scrub ok 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.680174+0000 osd.6 (osd.6) 349 : cluster 0 39.1e scrub starts 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:42.681615+0000 osd.6 (osd.6) 350 : cluster 0 39.1e scrub ok 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: audit 2024-03-11T21:41:43.542861+0000 mon.a (mon.0) 3690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.547757+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e802: 8 total, 8 up, 8 in 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: audit 2024-03-11T21:41:43.557741+0000 mon.c (mon.2) 1269 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: audit 2024-03-11T21:41:43.558879+0000 mon.a (mon.0) 3692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.760768+0000 osd.2 (osd.2) 379 : cluster 0 2.10 scrub starts 2024-03-11T21:41:44.304 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:44 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.763268+0000 osd.2 (osd.2) 380 : cluster 0 2.10 scrub ok 2024-03-11T21:41:44.316 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:44.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.447 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:44.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:44.608 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:44.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.708 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:44.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:44.839 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:44.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:44.968 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:45.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:45.097 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:45.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.454726+0000 osd.0 (osd.0) 420 : cluster 0 3.6 scrub starts 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.458975+0000 osd.0 (osd.0) 421 : cluster 0 3.6 scrub ok 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: cluster 2024-03-11T21:41:43.599096+0000 mgr.y (mgr.14550) 727 : cluster 0 pgmap v1241: 246 pgs: 24 creating+peering, 8 unknown, 214 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: audit 2024-03-11T21:41:44.556475+0000 mon.a (mon.0) 3693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: cluster 2024-03-11T21:41:44.564546+0000 mon.a (mon.0) 3694 : cluster 0 osdmap e803: 8 total, 8 up, 8 in 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: audit 2024-03-11T21:41:44.567104+0000 mon.c (mon.2) 1270 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[29284]: audit 2024-03-11T21:41:44.568652+0000 mon.a (mon.0) 3695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.454726+0000 osd.0 (osd.0) 420 : cluster 0 3.6 scrub starts 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.458975+0000 osd.0 (osd.0) 421 : cluster 0 3.6 scrub ok 2024-03-11T21:41:45.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: cluster 2024-03-11T21:41:43.599096+0000 mgr.y (mgr.14550) 727 : cluster 0 pgmap v1241: 246 pgs: 24 creating+peering, 8 unknown, 214 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-11T21:41:45.216 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: audit 2024-03-11T21:41:44.556475+0000 mon.a (mon.0) 3693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:45.216 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: cluster 2024-03-11T21:41:44.564546+0000 mon.a (mon.0) 3694 : cluster 0 osdmap e803: 8 total, 8 up, 8 in 2024-03-11T21:41:45.216 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: audit 2024-03-11T21:41:44.567104+0000 mon.c (mon.2) 1270 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.216 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:45 smithi143 bash[21512]: audit 2024-03-11T21:41:44.568652+0000 mon.a (mon.0) 3695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.230 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:45.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.454726+0000 osd.0 (osd.0) 420 : cluster 0 3.6 scrub starts 2024-03-11T21:41:45.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.458975+0000 osd.0 (osd.0) 421 : cluster 0 3.6 scrub ok 2024-03-11T21:41:45.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: cluster 2024-03-11T21:41:43.599096+0000 mgr.y (mgr.14550) 727 : cluster 0 pgmap v1241: 246 pgs: 24 creating+peering, 8 unknown, 214 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-11T21:41:45.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: audit 2024-03-11T21:41:44.556475+0000 mon.a (mon.0) 3693 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:45.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: cluster 2024-03-11T21:41:44.564546+0000 mon.a (mon.0) 3694 : cluster 0 osdmap e803: 8 total, 8 up, 8 in 2024-03-11T21:41:45.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: audit 2024-03-11T21:41:44.567104+0000 mon.c (mon.2) 1270 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:45 smithi203 bash[23654]: audit 2024-03-11T21:41:44.568652+0000 mon.a (mon.0) 3695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:45.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:45.359 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:45.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.490 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:45.594 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.621 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:45.625 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:45.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:45.750 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:45.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:45.880 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:45.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:46.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:46.142 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:46.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:44.437296+0000 osd.7 (osd.7) 426 : cluster 0 2.5 scrub starts 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:44.439835+0000 osd.7 (osd.7) 427 : cluster 0 2.5 scrub ok 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.249423+0000 osd.5 (osd.5) 439 : cluster 0 6.a scrub starts 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.251717+0000 osd.5 (osd.5) 440 : cluster 0 6.a scrub ok 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: audit 2024-03-11T21:41:45.558664+0000 mon.a (mon.0) 3696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:46.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.562887+0000 mon.a (mon.0) 3697 : cluster 0 osdmap e804: 8 total, 8 up, 8 in 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: audit 2024-03-11T21:41:45.568417+0000 mon.c (mon.2) 1271 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: audit 2024-03-11T21:41:45.569826+0000 mon.a (mon.0) 3698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.592067+0000 osd.4 (osd.4) 415 : cluster 0 39.15 scrub starts 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.595119+0000 osd.4 (osd.4) 416 : cluster 0 39.15 scrub ok 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.799489+0000 osd.2 (osd.2) 381 : cluster 0 5.8 scrub starts 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.803229+0000 osd.2 (osd.2) 382 : cluster 0 5.8 scrub ok 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:44.437296+0000 osd.7 (osd.7) 426 : cluster 0 2.5 scrub starts 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:44.439835+0000 osd.7 (osd.7) 427 : cluster 0 2.5 scrub ok 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.249423+0000 osd.5 (osd.5) 439 : cluster 0 6.a scrub starts 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.251717+0000 osd.5 (osd.5) 440 : cluster 0 6.a scrub ok 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: audit 2024-03-11T21:41:45.558664+0000 mon.a (mon.0) 3696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:46.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.562887+0000 mon.a (mon.0) 3697 : cluster 0 osdmap e804: 8 total, 8 up, 8 in 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: audit 2024-03-11T21:41:45.568417+0000 mon.c (mon.2) 1271 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: audit 2024-03-11T21:41:45.569826+0000 mon.a (mon.0) 3698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.592067+0000 osd.4 (osd.4) 415 : cluster 0 39.15 scrub starts 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.595119+0000 osd.4 (osd.4) 416 : cluster 0 39.15 scrub ok 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.799489+0000 osd.2 (osd.2) 381 : cluster 0 5.8 scrub starts 2024-03-11T21:41:46.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:46 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.803229+0000 osd.2 (osd.2) 382 : cluster 0 5.8 scrub ok 2024-03-11T21:41:46.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:44.437296+0000 osd.7 (osd.7) 426 : cluster 0 2.5 scrub starts 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:44.439835+0000 osd.7 (osd.7) 427 : cluster 0 2.5 scrub ok 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.249423+0000 osd.5 (osd.5) 439 : cluster 0 6.a scrub starts 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.251717+0000 osd.5 (osd.5) 440 : cluster 0 6.a scrub ok 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: audit 2024-03-11T21:41:45.558664+0000 mon.a (mon.0) 3696 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.562887+0000 mon.a (mon.0) 3697 : cluster 0 osdmap e804: 8 total, 8 up, 8 in 2024-03-11T21:41:46.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: audit 2024-03-11T21:41:45.568417+0000 mon.c (mon.2) 1271 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: audit 2024-03-11T21:41:45.569826+0000 mon.a (mon.0) 3698 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]: dispatch 2024-03-11T21:41:46.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.592067+0000 osd.4 (osd.4) 415 : cluster 0 39.15 scrub starts 2024-03-11T21:41:46.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.595119+0000 osd.4 (osd.4) 416 : cluster 0 39.15 scrub ok 2024-03-11T21:41:46.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.799489+0000 osd.2 (osd.2) 381 : cluster 0 5.8 scrub starts 2024-03-11T21:41:46.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:46 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.803229+0000 osd.2 (osd.2) 382 : cluster 0 5.8 scrub ok 2024-03-11T21:41:46.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:46.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:46.540 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:46.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.648 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:46.670 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:46.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:46.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:46.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:46.931 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:47.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:47.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:47.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:47.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:47.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: cluster 2024-03-11T21:41:45.599934+0000 mgr.y (mgr.14550) 728 : cluster 0 pgmap v1244: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:41:47.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.559766+0000 mon.a (mon.0) 3699 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:47.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: audit 2024-03-11T21:41:46.567176+0000 mon.a (mon.0) 3700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]': finished 2024-03-11T21:41:47.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.572120+0000 mon.a (mon.0) 3701 : cluster 0 osdmap e805: 8 total, 8 up, 8 in 2024-03-11T21:41:47.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.619542+0000 osd.4 (osd.4) 417 : cluster 0 5.e scrub starts 2024-03-11T21:41:47.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.623788+0000 osd.4 (osd.4) 418 : cluster 0 5.e scrub ok 2024-03-11T21:41:47.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: audit 2024-03-11T21:41:46.666953+0000 mon.c (mon.2) 1272 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:47 smithi203 bash[23654]: audit 2024-03-11T21:41:46.667411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:47.319 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: cluster 2024-03-11T21:41:45.599934+0000 mgr.y (mgr.14550) 728 : cluster 0 pgmap v1244: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.559766+0000 mon.a (mon.0) 3699 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: audit 2024-03-11T21:41:46.567176+0000 mon.a (mon.0) 3700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]': finished 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.572120+0000 mon.a (mon.0) 3701 : cluster 0 osdmap e805: 8 total, 8 up, 8 in 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.619542+0000 osd.4 (osd.4) 417 : cluster 0 5.e scrub starts 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.623788+0000 osd.4 (osd.4) 418 : cluster 0 5.e scrub ok 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: audit 2024-03-11T21:41:46.666953+0000 mon.c (mon.2) 1272 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[21512]: audit 2024-03-11T21:41:46.667411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.384 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: cluster 2024-03-11T21:41:45.599934+0000 mgr.y (mgr.14550) 728 : cluster 0 pgmap v1244: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.559766+0000 mon.a (mon.0) 3699 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: audit 2024-03-11T21:41:46.567176+0000 mon.a (mon.0) 3700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-136", "mode": "writeback"}]': finished 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.572120+0000 mon.a (mon.0) 3701 : cluster 0 osdmap e805: 8 total, 8 up, 8 in 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.619542+0000 osd.4 (osd.4) 417 : cluster 0 5.e scrub starts 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.623788+0000 osd.4 (osd.4) 418 : cluster 0 5.e scrub ok 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: audit 2024-03-11T21:41:46.666953+0000 mon.c (mon.2) 1272 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.385 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:47 smithi143 bash[29284]: audit 2024-03-11T21:41:46.667411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:47.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:47.450 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:47.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:47.585 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:47.666 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:47.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:47.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:47.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:47.846 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:47.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:48.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:48.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.241 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.294464+0000 osd.1 (osd.1) 231 : cluster 0 9.4 scrub starts 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.296684+0000 osd.1 (osd.1) 232 : cluster 0 9.4 scrub ok 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.399001+0000 osd.7 (osd.7) 428 : cluster 0 2.1c scrub starts 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:46.401198+0000 osd.7 (osd.7) 429 : cluster 0 2.1c scrub ok 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:47.224019+0000 osd.5 (osd.5) 441 : cluster 0 6.19 scrub starts 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:47.226154+0000 osd.5 (osd.5) 442 : cluster 0 6.19 scrub ok 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: audit 2024-03-11T21:41:47.570125+0000 mon.a (mon.0) 3703 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: cluster 2024-03-11T21:41:47.577457+0000 mon.a (mon.0) 3704 : cluster 0 osdmap e806: 8 total, 8 up, 8 in 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: audit 2024-03-11T21:41:47.578888+0000 mon.c (mon.2) 1273 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:48 smithi203 bash[23654]: audit 2024-03-11T21:41:47.579556+0000 mon.a (mon.0) 3705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.371 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.294464+0000 osd.1 (osd.1) 231 : cluster 0 9.4 scrub starts 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.296684+0000 osd.1 (osd.1) 232 : cluster 0 9.4 scrub ok 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.399001+0000 osd.7 (osd.7) 428 : cluster 0 2.1c scrub starts 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:46.401198+0000 osd.7 (osd.7) 429 : cluster 0 2.1c scrub ok 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:47.224019+0000 osd.5 (osd.5) 441 : cluster 0 6.19 scrub starts 2024-03-11T21:41:48.444 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:47.226154+0000 osd.5 (osd.5) 442 : cluster 0 6.19 scrub ok 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: audit 2024-03-11T21:41:47.570125+0000 mon.a (mon.0) 3703 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: cluster 2024-03-11T21:41:47.577457+0000 mon.a (mon.0) 3704 : cluster 0 osdmap e806: 8 total, 8 up, 8 in 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: audit 2024-03-11T21:41:47.578888+0000 mon.c (mon.2) 1273 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[21512]: audit 2024-03-11T21:41:47.579556+0000 mon.a (mon.0) 3705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.294464+0000 osd.1 (osd.1) 231 : cluster 0 9.4 scrub starts 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.296684+0000 osd.1 (osd.1) 232 : cluster 0 9.4 scrub ok 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.399001+0000 osd.7 (osd.7) 428 : cluster 0 2.1c scrub starts 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:46.401198+0000 osd.7 (osd.7) 429 : cluster 0 2.1c scrub ok 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:47.224019+0000 osd.5 (osd.5) 441 : cluster 0 6.19 scrub starts 2024-03-11T21:41:48.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:47.226154+0000 osd.5 (osd.5) 442 : cluster 0 6.19 scrub ok 2024-03-11T21:41:48.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: audit 2024-03-11T21:41:47.570125+0000 mon.a (mon.0) 3703 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:41:48.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: cluster 2024-03-11T21:41:47.577457+0000 mon.a (mon.0) 3704 : cluster 0 osdmap e806: 8 total, 8 up, 8 in 2024-03-11T21:41:48.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: audit 2024-03-11T21:41:47.578888+0000 mon.c (mon.2) 1273 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:48 smithi143 bash[29284]: audit 2024-03-11T21:41:47.579556+0000 mon.a (mon.0) 3705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]: dispatch 2024-03-11T21:41:48.474 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:48.501 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:48.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:48.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:48.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:48.794 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:48.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:48.918 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:48.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:49.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:49.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:49.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: cluster 2024-03-11T21:41:47.600492+0000 mgr.y (mgr.14550) 729 : cluster 0 pgmap v1247: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:49.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: cluster 2024-03-11T21:41:48.429393+0000 osd.0 (osd.0) 422 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:41:49.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: cluster 2024-03-11T21:41:48.432260+0000 osd.0 (osd.0) 423 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:41:49.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: cluster 2024-03-11T21:41:48.570990+0000 mon.a (mon.0) 3706 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:49.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: audit 2024-03-11T21:41:48.578147+0000 mon.a (mon.0) 3707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:49.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:49 smithi203 bash[23654]: cluster 2024-03-11T21:41:48.581793+0000 mon.a (mon.0) 3708 : cluster 0 osdmap e807: 8 total, 8 up, 8 in 2024-03-11T21:41:49.043 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:49.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:49.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:49.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: cluster 2024-03-11T21:41:47.600492+0000 mgr.y (mgr.14550) 729 : cluster 0 pgmap v1247: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:49.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: cluster 2024-03-11T21:41:48.429393+0000 osd.0 (osd.0) 422 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:41:49.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: cluster 2024-03-11T21:41:48.432260+0000 osd.0 (osd.0) 423 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:41:49.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: cluster 2024-03-11T21:41:48.570990+0000 mon.a (mon.0) 3706 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:49.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: audit 2024-03-11T21:41:48.578147+0000 mon.a (mon.0) 3707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[21512]: cluster 2024-03-11T21:41:48.581793+0000 mon.a (mon.0) 3708 : cluster 0 osdmap e807: 8 total, 8 up, 8 in 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: cluster 2024-03-11T21:41:47.600492+0000 mgr.y (mgr.14550) 729 : cluster 0 pgmap v1247: 246 pgs: 24 creating+peering, 222 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: cluster 2024-03-11T21:41:48.429393+0000 osd.0 (osd.0) 422 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: cluster 2024-03-11T21:41:48.432260+0000 osd.0 (osd.0) 423 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: cluster 2024-03-11T21:41:48.570990+0000 mon.a (mon.0) 3706 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: audit 2024-03-11T21:41:48.578147+0000 mon.a (mon.0) 3707 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-136"}]': finished 2024-03-11T21:41:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:49 smithi143 bash[29284]: cluster 2024-03-11T21:41:48.581793+0000 mon.a (mon.0) 3708 : cluster 0 osdmap e807: 8 total, 8 up, 8 in 2024-03-11T21:41:49.277 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:49.306 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:49.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:49.436 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:49.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:49.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:49.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:49.701 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:49.804 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:49.826 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:49.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:49.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:50.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:50.077 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:50.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[29284]: audit 2024-03-11T21:41:49.016627+0000 mgr.y (mgr.14550) 730 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.264522+0000 osd.5 (osd.5) 443 : cluster 0 5.15 scrub starts 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.267511+0000 osd.5 (osd.5) 444 : cluster 0 5.15 scrub ok 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.584957+0000 mon.a (mon.0) 3709 : cluster 0 osdmap e808: 8 total, 8 up, 8 in 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[21512]: audit 2024-03-11T21:41:49.016627+0000 mgr.y (mgr.14550) 730 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.264522+0000 osd.5 (osd.5) 443 : cluster 0 5.15 scrub starts 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.267511+0000 osd.5 (osd.5) 444 : cluster 0 5.15 scrub ok 2024-03-11T21:41:50.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:50 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.584957+0000 mon.a (mon.0) 3709 : cluster 0 osdmap e808: 8 total, 8 up, 8 in 2024-03-11T21:41:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:50.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:50.206 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:50.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:50 smithi203 bash[23654]: audit 2024-03-11T21:41:49.016627+0000 mgr.y (mgr.14550) 730 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:50.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:50 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.264522+0000 osd.5 (osd.5) 443 : cluster 0 5.15 scrub starts 2024-03-11T21:41:50.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:50 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.267511+0000 osd.5 (osd.5) 444 : cluster 0 5.15 scrub ok 2024-03-11T21:41:50.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:50 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.584957+0000 mon.a (mon.0) 3709 : cluster 0 osdmap e808: 8 total, 8 up, 8 in 2024-03-11T21:41:50.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:50.335 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:50.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:50.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:50.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:50.595 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:50.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:50.727 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:50.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:50.860 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:50.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:50.991 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:51.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.122 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:51.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.248 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.261129+0000 osd.1 (osd.1) 233 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:41:51.248 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.263403+0000 osd.1 (osd.1) 234 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:41:51.248 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.600942+0000 mgr.y (mgr.14550) 731 : cluster 0 pgmap v1250: 214 pgs: 214 active+clean; 4.4 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.602626+0000 osd.3 (osd.3) 431 : cluster 0 2.1e scrub starts 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.603806+0000 osd.3 (osd.3) 432 : cluster 0 2.1e scrub ok 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.690630+0000 osd.6 (osd.6) 351 : cluster 0 3.10 scrub starts 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:49.695239+0000 osd.6 (osd.6) 352 : cluster 0 3.10 scrub ok 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: cluster 2024-03-11T21:41:50.598677+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e809: 8 total, 8 up, 8 in 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: audit 2024-03-11T21:41:50.605948+0000 mon.c (mon.2) 1274 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[21512]: audit 2024-03-11T21:41:50.606577+0000 mon.a (mon.0) 3711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.261129+0000 osd.1 (osd.1) 233 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.263403+0000 osd.1 (osd.1) 234 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:41:51.249 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.600942+0000 mgr.y (mgr.14550) 731 : cluster 0 pgmap v1250: 214 pgs: 214 active+clean; 4.4 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.602626+0000 osd.3 (osd.3) 431 : cluster 0 2.1e scrub starts 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.603806+0000 osd.3 (osd.3) 432 : cluster 0 2.1e scrub ok 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.690630+0000 osd.6 (osd.6) 351 : cluster 0 3.10 scrub starts 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:49.695239+0000 osd.6 (osd.6) 352 : cluster 0 3.10 scrub ok 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: cluster 2024-03-11T21:41:50.598677+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e809: 8 total, 8 up, 8 in 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: audit 2024-03-11T21:41:50.605948+0000 mon.c (mon.2) 1274 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.250 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:51 smithi143 bash[29284]: audit 2024-03-11T21:41:50.606577+0000 mon.a (mon.0) 3711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.251 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.261129+0000 osd.1 (osd.1) 233 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.263403+0000 osd.1 (osd.1) 234 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.600942+0000 mgr.y (mgr.14550) 731 : cluster 0 pgmap v1250: 214 pgs: 214 active+clean; 4.4 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.602626+0000 osd.3 (osd.3) 431 : cluster 0 2.1e scrub starts 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.603806+0000 osd.3 (osd.3) 432 : cluster 0 2.1e scrub ok 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.690630+0000 osd.6 (osd.6) 351 : cluster 0 3.10 scrub starts 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:49.695239+0000 osd.6 (osd.6) 352 : cluster 0 3.10 scrub ok 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: cluster 2024-03-11T21:41:50.598677+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e809: 8 total, 8 up, 8 in 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: audit 2024-03-11T21:41:50.605948+0000 mon.c (mon.2) 1274 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.279 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:51 smithi203 bash[23654]: audit 2024-03-11T21:41:50.606577+0000 mon.a (mon.0) 3711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:41:51.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:51.378 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:51.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:51.509 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:51.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.634 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:51.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.765 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:51.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.895 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:51.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:52.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:52.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:52.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:52.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:52.389 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.401 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:41:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:52.417 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:52.520 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.288122+0000 osd.5 (osd.5) 445 : cluster 0 3.1c scrub starts 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.290320+0000 osd.5 (osd.5) 446 : cluster 0 3.1c scrub ok 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.384074+0000 osd.7 (osd.7) 430 : cluster 0 39.9 scrub starts 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.386891+0000 osd.7 (osd.7) 431 : cluster 0 39.9 scrub ok 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: audit 2024-03-11T21:41:51.533505+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.538254+0000 mon.a (mon.0) 3713 : cluster 0 osdmap e810: 8 total, 8 up, 8 in 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: audit 2024-03-11T21:41:51.554815+0000 mon.c (mon.2) 1275 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: audit 2024-03-11T21:41:51.556440+0000 mon.a (mon.0) 3714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.601574+0000 mgr.y (mgr.14550) 732 : cluster 0 pgmap v1253: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.647838+0000 osd.4 (osd.4) 419 : cluster 0 39.3 scrub starts 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.650715+0000 osd.4 (osd.4) 420 : cluster 0 39.3 scrub ok 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.715953+0000 osd.6 (osd.6) 353 : cluster 0 1.7 scrub starts 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.718258+0000 osd.6 (osd.6) 354 : cluster 0 1.7 scrub ok 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.787394+0000 osd.2 (osd.2) 383 : cluster 0 5.b scrub starts 2024-03-11T21:41:52.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: cluster 2024-03-11T21:41:51.789009+0000 osd.2 (osd.2) 384 : cluster 0 5.b scrub ok 2024-03-11T21:41:52.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: audit 2024-03-11T21:41:52.340615+0000 mon.a (mon.0) 3715 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:52.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:52 smithi203 bash[23654]: audit 2024-03-11T21:41:52.345162+0000 mon.c (mon.2) 1276 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:52.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:52.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:52.676 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:52.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.288122+0000 osd.5 (osd.5) 445 : cluster 0 3.1c scrub starts 2024-03-11T21:41:52.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.290320+0000 osd.5 (osd.5) 446 : cluster 0 3.1c scrub ok 2024-03-11T21:41:52.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.384074+0000 osd.7 (osd.7) 430 : cluster 0 39.9 scrub starts 2024-03-11T21:41:52.687 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.386891+0000 osd.7 (osd.7) 431 : cluster 0 39.9 scrub ok 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: audit 2024-03-11T21:41:51.533505+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.538254+0000 mon.a (mon.0) 3713 : cluster 0 osdmap e810: 8 total, 8 up, 8 in 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: audit 2024-03-11T21:41:51.554815+0000 mon.c (mon.2) 1275 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: audit 2024-03-11T21:41:51.556440+0000 mon.a (mon.0) 3714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.601574+0000 mgr.y (mgr.14550) 732 : cluster 0 pgmap v1253: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.647838+0000 osd.4 (osd.4) 419 : cluster 0 39.3 scrub starts 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.650715+0000 osd.4 (osd.4) 420 : cluster 0 39.3 scrub ok 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.715953+0000 osd.6 (osd.6) 353 : cluster 0 1.7 scrub starts 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.718258+0000 osd.6 (osd.6) 354 : cluster 0 1.7 scrub ok 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.787394+0000 osd.2 (osd.2) 383 : cluster 0 5.b scrub starts 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: cluster 2024-03-11T21:41:51.789009+0000 osd.2 (osd.2) 384 : cluster 0 5.b scrub ok 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: audit 2024-03-11T21:41:52.340615+0000 mon.a (mon.0) 3715 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:52.688 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[29284]: audit 2024-03-11T21:41:52.345162+0000 mon.c (mon.2) 1276 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.288122+0000 osd.5 (osd.5) 445 : cluster 0 3.1c scrub starts 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.290320+0000 osd.5 (osd.5) 446 : cluster 0 3.1c scrub ok 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.384074+0000 osd.7 (osd.7) 430 : cluster 0 39.9 scrub starts 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.386891+0000 osd.7 (osd.7) 431 : cluster 0 39.9 scrub ok 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: audit 2024-03-11T21:41:51.533505+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.538254+0000 mon.a (mon.0) 3713 : cluster 0 osdmap e810: 8 total, 8 up, 8 in 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: audit 2024-03-11T21:41:51.554815+0000 mon.c (mon.2) 1275 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: audit 2024-03-11T21:41:51.556440+0000 mon.a (mon.0) 3714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.601574+0000 mgr.y (mgr.14550) 732 : cluster 0 pgmap v1253: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.647838+0000 osd.4 (osd.4) 419 : cluster 0 39.3 scrub starts 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.650715+0000 osd.4 (osd.4) 420 : cluster 0 39.3 scrub ok 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.715953+0000 osd.6 (osd.6) 353 : cluster 0 1.7 scrub starts 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.718258+0000 osd.6 (osd.6) 354 : cluster 0 1.7 scrub ok 2024-03-11T21:41:52.689 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.787394+0000 osd.2 (osd.2) 383 : cluster 0 5.b scrub starts 2024-03-11T21:41:52.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: cluster 2024-03-11T21:41:51.789009+0000 osd.2 (osd.2) 384 : cluster 0 5.b scrub ok 2024-03-11T21:41:52.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: audit 2024-03-11T21:41:52.340615+0000 mon.a (mon.0) 3715 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:52.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:52 smithi143 bash[21512]: audit 2024-03-11T21:41:52.345162+0000 mon.c (mon.2) 1276 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:52.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:52.806 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:52.910 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:52.939 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:53.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.072 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:53.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:53.202 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:53.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:53.332 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:53.435 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:53.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:53.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.591 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:53.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: cluster 2024-03-11T21:41:52.457028+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: cluster 2024-03-11T21:41:52.458530+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: audit 2024-03-11T21:41:52.553487+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: cluster 2024-03-11T21:41:52.557919+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e811: 8 total, 8 up, 8 in 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: audit 2024-03-11T21:41:52.560179+0000 mon.c (mon.2) 1277 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: audit 2024-03-11T21:41:52.562317+0000 mon.a (mon.0) 3718 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.274461+0000 osd.5 (osd.5) 447 : cluster 0 5.1b scrub starts 2024-03-11T21:41:53.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.275829+0000 osd.5 (osd.5) 448 : cluster 0 5.1b scrub ok 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: cluster 2024-03-11T21:41:52.457028+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: cluster 2024-03-11T21:41:52.458530+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: audit 2024-03-11T21:41:52.553487+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: cluster 2024-03-11T21:41:52.557919+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e811: 8 total, 8 up, 8 in 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: audit 2024-03-11T21:41:52.560179+0000 mon.c (mon.2) 1277 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: audit 2024-03-11T21:41:52.562317+0000 mon.a (mon.0) 3718 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.274461+0000 osd.5 (osd.5) 447 : cluster 0 5.1b scrub starts 2024-03-11T21:41:53.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:53 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.275829+0000 osd.5 (osd.5) 448 : cluster 0 5.1b scrub ok 2024-03-11T21:41:53.719 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:53.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: cluster 2024-03-11T21:41:52.457028+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: cluster 2024-03-11T21:41:52.458530+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: audit 2024-03-11T21:41:52.553487+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: cluster 2024-03-11T21:41:52.557919+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e811: 8 total, 8 up, 8 in 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: audit 2024-03-11T21:41:52.560179+0000 mon.c (mon.2) 1277 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: audit 2024-03-11T21:41:52.562317+0000 mon.a (mon.0) 3718 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:41:53.835 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.274461+0000 osd.5 (osd.5) 447 : cluster 0 5.1b scrub starts 2024-03-11T21:41:53.836 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:53 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.275829+0000 osd.5 (osd.5) 448 : cluster 0 5.1b scrub ok 2024-03-11T21:41:53.850 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:53.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:53.980 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:54.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:54.110 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:54.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:54.237 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:54.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:54.362 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:54.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:54.494 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:54.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:54.625 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:54.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:54.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:54.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.401838+0000 osd.7 (osd.7) 432 : cluster 0 3.15 scrub starts 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.405767+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub ok 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: audit 2024-03-11T21:41:53.555943+0000 mon.a (mon.0) 3719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.560165+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e812: 8 total, 8 up, 8 in 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: audit 2024-03-11T21:41:53.561937+0000 mon.c (mon.2) 1278 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: audit 2024-03-11T21:41:53.569992+0000 mon.a (mon.0) 3721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.582164+0000 osd.3 (osd.3) 433 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.585649+0000 osd.3 (osd.3) 434 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.602386+0000 mgr.y (mgr.14550) 733 : cluster 0 pgmap v1256: 246 pgs: 15 unknown, 231 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.680587+0000 osd.6 (osd.6) 355 : cluster 0 2.1a scrub starts 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:53.683459+0000 osd.6 (osd.6) 356 : cluster 0 2.1a scrub ok 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.176539+0000 osd.1 (osd.1) 235 : cluster 0 9.14 scrub starts 2024-03-11T21:41:54.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:54 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.179002+0000 osd.1 (osd.1) 236 : cluster 0 9.14 scrub ok 2024-03-11T21:41:54.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.401838+0000 osd.7 (osd.7) 432 : cluster 0 3.15 scrub starts 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.405767+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub ok 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: audit 2024-03-11T21:41:53.555943+0000 mon.a (mon.0) 3719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.560165+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e812: 8 total, 8 up, 8 in 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: audit 2024-03-11T21:41:53.561937+0000 mon.c (mon.2) 1278 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: audit 2024-03-11T21:41:53.569992+0000 mon.a (mon.0) 3721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.582164+0000 osd.3 (osd.3) 433 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.585649+0000 osd.3 (osd.3) 434 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.602386+0000 mgr.y (mgr.14550) 733 : cluster 0 pgmap v1256: 246 pgs: 15 unknown, 231 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.680587+0000 osd.6 (osd.6) 355 : cluster 0 2.1a scrub starts 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:53.683459+0000 osd.6 (osd.6) 356 : cluster 0 2.1a scrub ok 2024-03-11T21:41:54.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.176539+0000 osd.1 (osd.1) 235 : cluster 0 9.14 scrub starts 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.179002+0000 osd.1 (osd.1) 236 : cluster 0 9.14 scrub ok 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.401838+0000 osd.7 (osd.7) 432 : cluster 0 3.15 scrub starts 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.405767+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub ok 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: audit 2024-03-11T21:41:53.555943+0000 mon.a (mon.0) 3719 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.560165+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e812: 8 total, 8 up, 8 in 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: audit 2024-03-11T21:41:53.561937+0000 mon.c (mon.2) 1278 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: audit 2024-03-11T21:41:53.569992+0000 mon.a (mon.0) 3721 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.582164+0000 osd.3 (osd.3) 433 : cluster 0 6.13 deep-scrub starts 2024-03-11T21:41:54.868 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.585649+0000 osd.3 (osd.3) 434 : cluster 0 6.13 deep-scrub ok 2024-03-11T21:41:54.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.602386+0000 mgr.y (mgr.14550) 733 : cluster 0 pgmap v1256: 246 pgs: 15 unknown, 231 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:54.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.680587+0000 osd.6 (osd.6) 355 : cluster 0 2.1a scrub starts 2024-03-11T21:41:54.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:53.683459+0000 osd.6 (osd.6) 356 : cluster 0 2.1a scrub ok 2024-03-11T21:41:54.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.176539+0000 osd.1 (osd.1) 235 : cluster 0 9.14 scrub starts 2024-03-11T21:41:54.869 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:54 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.179002+0000 osd.1 (osd.1) 236 : cluster 0 9.14 scrub ok 2024-03-11T21:41:54.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:54.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:55.009 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:55.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:55.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:55.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:55.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.398 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:55.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:55.528 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:55.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.492276+0000 osd.0 (osd.0) 426 : cluster 0 313.6s0 scrub starts 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.495829+0000 osd.0 (osd.0) 427 : cluster 0 313.6 scrub ok 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: audit 2024-03-11T21:41:54.576190+0000 mon.a (mon.0) 3722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.594022+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e813: 8 total, 8 up, 8 in 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: audit 2024-03-11T21:41:54.596340+0000 mon.c (mon.2) 1279 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: audit 2024-03-11T21:41:54.597270+0000 mon.a (mon.0) 3724 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.644 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.627811+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub starts 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.630375+0000 osd.3 (osd.3) 436 : cluster 0 6.1f scrub ok 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.739144+0000 osd.4 (osd.4) 421 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.741240+0000 osd.4 (osd.4) 422 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.859372+0000 osd.2 (osd.2) 385 : cluster 0 39.6 scrub starts 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:54.862654+0000 osd.2 (osd.2) 386 : cluster 0 39.6 scrub ok 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.272601+0000 osd.5 (osd.5) 449 : cluster 0 9.7 scrub starts 2024-03-11T21:41:55.645 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:55 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.275667+0000 osd.5 (osd.5) 450 : cluster 0 9.7 scrub ok 2024-03-11T21:41:55.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:55.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:55.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:55.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.492276+0000 osd.0 (osd.0) 426 : cluster 0 313.6s0 scrub starts 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.495829+0000 osd.0 (osd.0) 427 : cluster 0 313.6 scrub ok 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: audit 2024-03-11T21:41:54.576190+0000 mon.a (mon.0) 3722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.594022+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e813: 8 total, 8 up, 8 in 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: audit 2024-03-11T21:41:54.596340+0000 mon.c (mon.2) 1279 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: audit 2024-03-11T21:41:54.597270+0000 mon.a (mon.0) 3724 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.627811+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub starts 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.630375+0000 osd.3 (osd.3) 436 : cluster 0 6.1f scrub ok 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.739144+0000 osd.4 (osd.4) 421 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.741240+0000 osd.4 (osd.4) 422 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:41:55.917 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.859372+0000 osd.2 (osd.2) 385 : cluster 0 39.6 scrub starts 2024-03-11T21:41:55.918 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:54.862654+0000 osd.2 (osd.2) 386 : cluster 0 39.6 scrub ok 2024-03-11T21:41:55.918 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.272601+0000 osd.5 (osd.5) 449 : cluster 0 9.7 scrub starts 2024-03-11T21:41:55.918 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.275667+0000 osd.5 (osd.5) 450 : cluster 0 9.7 scrub ok 2024-03-11T21:41:55.918 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:55.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.492276+0000 osd.0 (osd.0) 426 : cluster 0 313.6s0 scrub starts 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.495829+0000 osd.0 (osd.0) 427 : cluster 0 313.6 scrub ok 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: audit 2024-03-11T21:41:54.576190+0000 mon.a (mon.0) 3722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.594022+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e813: 8 total, 8 up, 8 in 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: audit 2024-03-11T21:41:54.596340+0000 mon.c (mon.2) 1279 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: audit 2024-03-11T21:41:54.597270+0000 mon.a (mon.0) 3724 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.627811+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub starts 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.630375+0000 osd.3 (osd.3) 436 : cluster 0 6.1f scrub ok 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.739144+0000 osd.4 (osd.4) 421 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.741240+0000 osd.4 (osd.4) 422 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.859372+0000 osd.2 (osd.2) 385 : cluster 0 39.6 scrub starts 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:54.862654+0000 osd.2 (osd.2) 386 : cluster 0 39.6 scrub ok 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.272601+0000 osd.5 (osd.5) 449 : cluster 0 9.7 scrub starts 2024-03-11T21:41:55.919 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:55 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.275667+0000 osd.5 (osd.5) 450 : cluster 0 9.7 scrub ok 2024-03-11T21:41:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:56.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.046 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:56.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:56.173 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:56.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.297 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:56.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:56.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:56.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.217563+0000 osd.1 (osd.1) 237 : cluster 0 9.1f scrub starts 2024-03-11T21:41:56.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.220186+0000 osd.1 (osd.1) 238 : cluster 0 9.1f scrub ok 2024-03-11T21:41:56.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: audit 2024-03-11T21:41:55.580734+0000 mon.a (mon.0) 3725 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:41:56.666 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.588279+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e814: 8 total, 8 up, 8 in 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.603239+0000 mgr.y (mgr.14550) 734 : cluster 0 pgmap v1259: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.642447+0000 osd.6 (osd.6) 357 : cluster 0 2.7 scrub starts 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.649074+0000 osd.6 (osd.6) 358 : cluster 0 2.7 scrub ok 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.735556+0000 osd.4 (osd.4) 423 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:55.738270+0000 osd.4 (osd.4) 424 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: audit 2024-03-11T21:41:55.798190+0000 mon.c (mon.2) 1280 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.286614+0000 osd.5 (osd.5) 451 : cluster 0 9.12 scrub starts 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.287960+0000 osd.5 (osd.5) 452 : cluster 0 9.12 scrub ok 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: audit 2024-03-11T21:41:56.476927+0000 mon.c (mon.2) 1281 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: audit 2024-03-11T21:41:56.478224+0000 mon.c (mon.2) 1282 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:41:56.667 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[21512]: audit 2024-03-11T21:41:56.486689+0000 mon.a (mon.0) 3727 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.217563+0000 osd.1 (osd.1) 237 : cluster 0 9.1f scrub starts 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.220186+0000 osd.1 (osd.1) 238 : cluster 0 9.1f scrub ok 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: audit 2024-03-11T21:41:55.580734+0000 mon.a (mon.0) 3725 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.588279+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e814: 8 total, 8 up, 8 in 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.603239+0000 mgr.y (mgr.14550) 734 : cluster 0 pgmap v1259: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.642447+0000 osd.6 (osd.6) 357 : cluster 0 2.7 scrub starts 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.649074+0000 osd.6 (osd.6) 358 : cluster 0 2.7 scrub ok 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.735556+0000 osd.4 (osd.4) 423 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:55.738270+0000 osd.4 (osd.4) 424 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: audit 2024-03-11T21:41:55.798190+0000 mon.c (mon.2) 1280 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.286614+0000 osd.5 (osd.5) 451 : cluster 0 9.12 scrub starts 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.287960+0000 osd.5 (osd.5) 452 : cluster 0 9.12 scrub ok 2024-03-11T21:41:56.668 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: audit 2024-03-11T21:41:56.476927+0000 mon.c (mon.2) 1281 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:41:56.669 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: audit 2024-03-11T21:41:56.478224+0000 mon.c (mon.2) 1282 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:41:56.669 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:56 smithi143 bash[29284]: audit 2024-03-11T21:41:56.486689+0000 mon.a (mon.0) 3727 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:56.680 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:56.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.217563+0000 osd.1 (osd.1) 237 : cluster 0 9.1f scrub starts 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.220186+0000 osd.1 (osd.1) 238 : cluster 0 9.1f scrub ok 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: audit 2024-03-11T21:41:55.580734+0000 mon.a (mon.0) 3725 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.588279+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e814: 8 total, 8 up, 8 in 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.603239+0000 mgr.y (mgr.14550) 734 : cluster 0 pgmap v1259: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.642447+0000 osd.6 (osd.6) 357 : cluster 0 2.7 scrub starts 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.649074+0000 osd.6 (osd.6) 358 : cluster 0 2.7 scrub ok 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.735556+0000 osd.4 (osd.4) 423 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:55.738270+0000 osd.4 (osd.4) 424 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: audit 2024-03-11T21:41:55.798190+0000 mon.c (mon.2) 1280 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.286614+0000 osd.5 (osd.5) 451 : cluster 0 9.12 scrub starts 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.287960+0000 osd.5 (osd.5) 452 : cluster 0 9.12 scrub ok 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: audit 2024-03-11T21:41:56.476927+0000 mon.c (mon.2) 1281 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: audit 2024-03-11T21:41:56.478224+0000 mon.c (mon.2) 1282 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:41:56.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:56 smithi203 bash[23654]: audit 2024-03-11T21:41:56.486689+0000 mon.a (mon.0) 3727 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:41:56.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:56.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.943 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:57.046 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:57.073 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:57.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:57.204 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:57.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:57.334 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:57.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:57.462 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:57.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:57.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:57.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:57.723 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:57.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.249044+0000 osd.1 (osd.1) 239 : cluster 0 39.1c scrub starts 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.250739+0000 osd.1 (osd.1) 240 : cluster 0 39.1c scrub ok 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: audit 2024-03-11T21:41:56.682364+0000 mon.c (mon.2) 1283 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: audit 2024-03-11T21:41:56.682996+0000 mon.a (mon.0) 3728 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: audit 2024-03-11T21:41:56.684367+0000 mon.c (mon.2) 1284 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: audit 2024-03-11T21:41:56.684880+0000 mon.a (mon.0) 3729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.713643+0000 osd.4 (osd.4) 425 : cluster 0 1.5 scrub starts 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:56.715855+0000 osd.4 (osd.4) 426 : cluster 0 1.5 scrub ok 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.326992+0000 osd.5 (osd.5) 453 : cluster 0 39.1a scrub starts 2024-03-11T21:41:57.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:57 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.330027+0000 osd.5 (osd.5) 454 : cluster 0 39.1a scrub ok 2024-03-11T21:41:57.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:57.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.249044+0000 osd.1 (osd.1) 239 : cluster 0 39.1c scrub starts 2024-03-11T21:41:57.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.250739+0000 osd.1 (osd.1) 240 : cluster 0 39.1c scrub ok 2024-03-11T21:41:57.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: audit 2024-03-11T21:41:56.682364+0000 mon.c (mon.2) 1283 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: audit 2024-03-11T21:41:56.682996+0000 mon.a (mon.0) 3728 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.837 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: audit 2024-03-11T21:41:56.684367+0000 mon.c (mon.2) 1284 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: audit 2024-03-11T21:41:56.684880+0000 mon.a (mon.0) 3729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.713643+0000 osd.4 (osd.4) 425 : cluster 0 1.5 scrub starts 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:56.715855+0000 osd.4 (osd.4) 426 : cluster 0 1.5 scrub ok 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.326992+0000 osd.5 (osd.5) 453 : cluster 0 39.1a scrub starts 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.330027+0000 osd.5 (osd.5) 454 : cluster 0 39.1a scrub ok 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.249044+0000 osd.1 (osd.1) 239 : cluster 0 39.1c scrub starts 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.250739+0000 osd.1 (osd.1) 240 : cluster 0 39.1c scrub ok 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: audit 2024-03-11T21:41:56.682364+0000 mon.c (mon.2) 1283 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: audit 2024-03-11T21:41:56.682996+0000 mon.a (mon.0) 3728 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: audit 2024-03-11T21:41:56.684367+0000 mon.c (mon.2) 1284 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: audit 2024-03-11T21:41:56.684880+0000 mon.a (mon.0) 3729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]: dispatch 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.713643+0000 osd.4 (osd.4) 425 : cluster 0 1.5 scrub starts 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:56.715855+0000 osd.4 (osd.4) 426 : cluster 0 1.5 scrub ok 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.326992+0000 osd.5 (osd.5) 453 : cluster 0 39.1a scrub starts 2024-03-11T21:41:57.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:57 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.330027+0000 osd.5 (osd.5) 454 : cluster 0 39.1a scrub ok 2024-03-11T21:41:57.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:57.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:58.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:58.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.244 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:58.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:58.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.504 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:58.607 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.638 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.604247+0000 mgr.y (mgr.14550) 735 : cluster 0 pgmap v1260: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: audit 2024-03-11T21:41:57.613579+0000 mon.a (mon.0) 3730 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]': finished 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.621890+0000 mon.a (mon.0) 3731 : cluster 0 osdmap e815: 8 total, 8 up, 8 in 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.692666+0000 osd.3 (osd.3) 437 : cluster 0 1.4 scrub starts 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.696200+0000 osd.3 (osd.3) 438 : cluster 0 1.4 scrub ok 2024-03-11T21:41:58.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.722348+0000 osd.4 (osd.4) 427 : cluster 0 6.17 scrub starts 2024-03-11T21:41:58.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:57.724681+0000 osd.4 (osd.4) 428 : cluster 0 6.17 scrub ok 2024-03-11T21:41:58.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:58.363725+0000 osd.5 (osd.5) 455 : cluster 0 6.d scrub starts 2024-03-11T21:41:58.696 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[29284]: cluster 2024-03-11T21:41:58.366185+0000 osd.5 (osd.5) 456 : cluster 0 6.d scrub ok 2024-03-11T21:41:58.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:58.769 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:58.867 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.604247+0000 mgr.y (mgr.14550) 735 : cluster 0 pgmap v1260: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: audit 2024-03-11T21:41:57.613579+0000 mon.a (mon.0) 3730 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]': finished 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.621890+0000 mon.a (mon.0) 3731 : cluster 0 osdmap e815: 8 total, 8 up, 8 in 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.692666+0000 osd.3 (osd.3) 437 : cluster 0 1.4 scrub starts 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.696200+0000 osd.3 (osd.3) 438 : cluster 0 1.4 scrub ok 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.722348+0000 osd.4 (osd.4) 427 : cluster 0 6.17 scrub starts 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:57.724681+0000 osd.4 (osd.4) 428 : cluster 0 6.17 scrub ok 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:58.363725+0000 osd.5 (osd.5) 455 : cluster 0 6.d scrub starts 2024-03-11T21:41:58.868 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:58 smithi143 bash[21512]: cluster 2024-03-11T21:41:58.366185+0000 osd.5 (osd.5) 456 : cluster 0 6.d scrub ok 2024-03-11T21:41:58.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.604247+0000 mgr.y (mgr.14550) 735 : cluster 0 pgmap v1260: 246 pgs: 246 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: audit 2024-03-11T21:41:57.613579+0000 mon.a (mon.0) 3730 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-138"}]': finished 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.621890+0000 mon.a (mon.0) 3731 : cluster 0 osdmap e815: 8 total, 8 up, 8 in 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.692666+0000 osd.3 (osd.3) 437 : cluster 0 1.4 scrub starts 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.696200+0000 osd.3 (osd.3) 438 : cluster 0 1.4 scrub ok 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.722348+0000 osd.4 (osd.4) 427 : cluster 0 6.17 scrub starts 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:57.724681+0000 osd.4 (osd.4) 428 : cluster 0 6.17 scrub ok 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:58.363725+0000 osd.5 (osd.5) 455 : cluster 0 6.d scrub starts 2024-03-11T21:41:58.883 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:58 smithi203 bash[23654]: cluster 2024-03-11T21:41:58.366185+0000 osd.5 (osd.5) 456 : cluster 0 6.d scrub ok 2024-03-11T21:41:58.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:59.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:59.041 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:59.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:59.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:59.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:59.279 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:41:59.302 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:59.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:59.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:59.564 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:59.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:59.693 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:59.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:59.822 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:59.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: cluster 2024-03-11T21:41:58.611386+0000 osd.0 (osd.0) 428 : cluster 0 5.5 scrub starts 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: cluster 2024-03-11T21:41:58.613675+0000 osd.0 (osd.0) 429 : cluster 0 5.5 scrub ok 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: cluster 2024-03-11T21:41:58.627480+0000 mon.a (mon.0) 3732 : cluster 0 osdmap e816: 8 total, 8 up, 8 in 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: audit 2024-03-11T21:41:59.025798+0000 mgr.y (mgr.14550) 736 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.366068+0000 osd.5 (osd.5) 457 : cluster 0 313.1s0 scrub starts 2024-03-11T21:41:59.934 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.370497+0000 osd.5 (osd.5) 458 : cluster 0 313.1 scrub ok 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: cluster 2024-03-11T21:41:58.611386+0000 osd.0 (osd.0) 428 : cluster 0 5.5 scrub starts 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: cluster 2024-03-11T21:41:58.613675+0000 osd.0 (osd.0) 429 : cluster 0 5.5 scrub ok 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: cluster 2024-03-11T21:41:58.627480+0000 mon.a (mon.0) 3732 : cluster 0 osdmap e816: 8 total, 8 up, 8 in 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: audit 2024-03-11T21:41:59.025798+0000 mgr.y (mgr.14550) 736 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.366068+0000 osd.5 (osd.5) 457 : cluster 0 313.1s0 scrub starts 2024-03-11T21:41:59.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:41:59 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.370497+0000 osd.5 (osd.5) 458 : cluster 0 313.1 scrub ok 2024-03-11T21:41:59.945 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:59.992 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:41:59.992 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:41:59.992 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:41:59.993 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:42:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: cluster 2024-03-11T21:41:58.611386+0000 osd.0 (osd.0) 428 : cluster 0 5.5 scrub starts 2024-03-11T21:42:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: cluster 2024-03-11T21:41:58.613675+0000 osd.0 (osd.0) 429 : cluster 0 5.5 scrub ok 2024-03-11T21:42:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: cluster 2024-03-11T21:41:58.627480+0000 mon.a (mon.0) 3732 : cluster 0 osdmap e816: 8 total, 8 up, 8 in 2024-03-11T21:42:00.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: audit 2024-03-11T21:41:59.025798+0000 mgr.y (mgr.14550) 736 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:00.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.366068+0000 osd.5 (osd.5) 457 : cluster 0 313.1s0 scrub starts 2024-03-11T21:42:00.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:41:59 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.370497+0000 osd.5 (osd.5) 458 : cluster 0 313.1 scrub ok 2024-03-11T21:42:00.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:00.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:00.198 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:00.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.330 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:00.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:00.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:00.591 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:00.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:00.716 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:00.820 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.842 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.417865+0000 osd.7 (osd.7) 434 : cluster 0 39.13 scrub starts 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.422521+0000 osd.7 (osd.7) 435 : cluster 0 39.13 scrub ok 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.577384+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub starts 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.579080+0000 osd.6 (osd.6) 360 : cluster 0 2.17 scrub ok 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.605064+0000 mgr.y (mgr.14550) 737 : cluster 0 pgmap v1263: 214 pgs: 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: cluster 2024-03-11T21:41:59.646962+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e817: 8 total, 8 up, 8 in 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: audit 2024-03-11T21:41:59.649367+0000 mon.c (mon.2) 1285 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: audit 2024-03-11T21:41:59.658341+0000 mon.a (mon.0) 3734 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[21512]: audit 2024-03-11T21:42:00.638470+0000 mon.a (mon.0) 3735 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.417865+0000 osd.7 (osd.7) 434 : cluster 0 39.13 scrub starts 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.422521+0000 osd.7 (osd.7) 435 : cluster 0 39.13 scrub ok 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.577384+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub starts 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.579080+0000 osd.6 (osd.6) 360 : cluster 0 2.17 scrub ok 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.605064+0000 mgr.y (mgr.14550) 737 : cluster 0 pgmap v1263: 214 pgs: 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: cluster 2024-03-11T21:41:59.646962+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e817: 8 total, 8 up, 8 in 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: audit 2024-03-11T21:41:59.649367+0000 mon.c (mon.2) 1285 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: audit 2024-03-11T21:41:59.658341+0000 mon.a (mon.0) 3734 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:00 smithi143 bash[29284]: audit 2024-03-11T21:42:00.638470+0000 mon.a (mon.0) 3735 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:00.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:00.958 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.417865+0000 osd.7 (osd.7) 434 : cluster 0 39.13 scrub starts 2024-03-11T21:42:00.958 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.422521+0000 osd.7 (osd.7) 435 : cluster 0 39.13 scrub ok 2024-03-11T21:42:00.958 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.577384+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub starts 2024-03-11T21:42:00.958 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.579080+0000 osd.6 (osd.6) 360 : cluster 0 2.17 scrub ok 2024-03-11T21:42:00.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.605064+0000 mgr.y (mgr.14550) 737 : cluster 0 pgmap v1263: 214 pgs: 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:00.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: cluster 2024-03-11T21:41:59.646962+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e817: 8 total, 8 up, 8 in 2024-03-11T21:42:00.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: audit 2024-03-11T21:41:59.649367+0000 mon.c (mon.2) 1285 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: audit 2024-03-11T21:41:59.658341+0000 mon.a (mon.0) 3734 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:00.959 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:00 smithi203 bash[23654]: audit 2024-03-11T21:42:00.638470+0000 mon.a (mon.0) 3735 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:00.974 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:01.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:01.104 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:01.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:01.235 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:01.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:01.365 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:01.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:01.496 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:01.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:01.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:01.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:00.290112+0000 osd.1 (osd.1) 241 : cluster 0 5.c deep-scrub starts 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:00.291995+0000 osd.1 (osd.1) 242 : cluster 0 5.c deep-scrub ok 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:00.568067+0000 osd.0 (osd.0) 430 : cluster 0 3.1 scrub starts 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:00.572449+0000 osd.0 (osd.0) 431 : cluster 0 3.1 scrub ok 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:00.653335+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e818: 8 total, 8 up, 8 in 2024-03-11T21:42:01.746 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: audit 2024-03-11T21:42:00.654829+0000 mon.c (mon.2) 1286 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: audit 2024-03-11T21:42:00.661216+0000 mon.a (mon.0) 3737 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: audit 2024-03-11T21:42:01.640421+0000 mon.a (mon.0) 3738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:01.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.644987+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e819: 8 total, 8 up, 8 in 2024-03-11T21:42:01.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: audit 2024-03-11T21:42:01.645445+0000 mon.c (mon.2) 1287 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.747 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:01 smithi203 bash[23654]: audit 2024-03-11T21:42:01.645990+0000 mon.a (mon.0) 3740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.761 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:01.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:01.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:00.290112+0000 osd.1 (osd.1) 241 : cluster 0 5.c deep-scrub starts 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:00.291995+0000 osd.1 (osd.1) 242 : cluster 0 5.c deep-scrub ok 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:00.568067+0000 osd.0 (osd.0) 430 : cluster 0 3.1 scrub starts 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:00.572449+0000 osd.0 (osd.0) 431 : cluster 0 3.1 scrub ok 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:00.653335+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e818: 8 total, 8 up, 8 in 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: audit 2024-03-11T21:42:00.654829+0000 mon.c (mon.2) 1286 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: audit 2024-03-11T21:42:00.661216+0000 mon.a (mon.0) 3737 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: audit 2024-03-11T21:42:01.640421+0000 mon.a (mon.0) 3738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:01.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.644987+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e819: 8 total, 8 up, 8 in 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: audit 2024-03-11T21:42:01.645445+0000 mon.c (mon.2) 1287 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[21512]: audit 2024-03-11T21:42:01.645990+0000 mon.a (mon.0) 3740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:00.290112+0000 osd.1 (osd.1) 241 : cluster 0 5.c deep-scrub starts 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:00.291995+0000 osd.1 (osd.1) 242 : cluster 0 5.c deep-scrub ok 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:00.568067+0000 osd.0 (osd.0) 430 : cluster 0 3.1 scrub starts 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:00.572449+0000 osd.0 (osd.0) 431 : cluster 0 3.1 scrub ok 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:00.653335+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e818: 8 total, 8 up, 8 in 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: audit 2024-03-11T21:42:00.654829+0000 mon.c (mon.2) 1286 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: audit 2024-03-11T21:42:00.661216+0000 mon.a (mon.0) 3737 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: audit 2024-03-11T21:42:01.640421+0000 mon.a (mon.0) 3738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.644987+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e819: 8 total, 8 up, 8 in 2024-03-11T21:42:01.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: audit 2024-03-11T21:42:01.645445+0000 mon.c (mon.2) 1287 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:01 smithi143 bash[29284]: audit 2024-03-11T21:42:01.645990+0000 mon.a (mon.0) 3740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:42:01.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.020 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:02.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:02.151 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:02.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:02.281 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:02.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:02.412 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:02.443 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:02.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:02.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:02.668 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:02.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.548033+0000 osd.0 (osd.0) 432 : cluster 0 9.9 deep-scrub starts 2024-03-11T21:42:02.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.549345+0000 osd.0 (osd.0) 433 : cluster 0 9.9 deep-scrub ok 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.605779+0000 mgr.y (mgr.14550) 738 : cluster 0 pgmap v1266: 246 pgs: 6 creating+peering, 26 unknown, 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.658595+0000 osd.3 (osd.3) 439 : cluster 0 9.8 scrub starts 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:01.660103+0000 osd.3 (osd.3) 440 : cluster 0 9.8 scrub ok 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: audit 2024-03-11T21:42:02.643537+0000 mon.a (mon.0) 3741 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: cluster 2024-03-11T21:42:02.651425+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e820: 8 total, 8 up, 8 in 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: audit 2024-03-11T21:42:02.657365+0000 mon.c (mon.2) 1288 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:02.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:02 smithi203 bash[23654]: audit 2024-03-11T21:42:02.660715+0000 mon.a (mon.0) 3743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:02.799 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:02.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:02.931 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.548033+0000 osd.0 (osd.0) 432 : cluster 0 9.9 deep-scrub starts 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.549345+0000 osd.0 (osd.0) 433 : cluster 0 9.9 deep-scrub ok 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.605779+0000 mgr.y (mgr.14550) 738 : cluster 0 pgmap v1266: 246 pgs: 6 creating+peering, 26 unknown, 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.658595+0000 osd.3 (osd.3) 439 : cluster 0 9.8 scrub starts 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:01.660103+0000 osd.3 (osd.3) 440 : cluster 0 9.8 scrub ok 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: audit 2024-03-11T21:42:02.643537+0000 mon.a (mon.0) 3741 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: cluster 2024-03-11T21:42:02.651425+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e820: 8 total, 8 up, 8 in 2024-03-11T21:42:02.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: audit 2024-03-11T21:42:02.657365+0000 mon.c (mon.2) 1288 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[29284]: audit 2024-03-11T21:42:02.660715+0000 mon.a (mon.0) 3743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.548033+0000 osd.0 (osd.0) 432 : cluster 0 9.9 deep-scrub starts 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.549345+0000 osd.0 (osd.0) 433 : cluster 0 9.9 deep-scrub ok 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.605779+0000 mgr.y (mgr.14550) 738 : cluster 0 pgmap v1266: 246 pgs: 6 creating+peering, 26 unknown, 214 active+clean; 4.4 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.658595+0000 osd.3 (osd.3) 439 : cluster 0 9.8 scrub starts 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:01.660103+0000 osd.3 (osd.3) 440 : cluster 0 9.8 scrub ok 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: audit 2024-03-11T21:42:02.643537+0000 mon.a (mon.0) 3741 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: cluster 2024-03-11T21:42:02.651425+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e820: 8 total, 8 up, 8 in 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: audit 2024-03-11T21:42:02.657365+0000 mon.c (mon.2) 1288 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:02.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:02 smithi143 bash[21512]: audit 2024-03-11T21:42:02.660715+0000 mon.a (mon.0) 3743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:42:03.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:03.061 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:03.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:03.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:03.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:42:03.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:03.320 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:03.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:03.451 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:03.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2940286331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 msgr2=0x7f9cf01071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2940286331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf01071a0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009990 tx=0x7f9ce0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2940286331 shutdown_connections 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2940286331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2940286331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf01071a0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2940286331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 0x7f9cf01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2940286331 >> 172.21.15.143:0/2940286331 conn(0x7f9cf0076d80 msgr2=0x7f9cf0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2940286331 shutdown_connections 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2940286331 wait complete. 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 librados: starting msgr at 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 librados: starting objecter 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 Processor -- start 2024-03-11T21:42:03.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- start start 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 librados: setting wanted keys 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 librados: calling monclient init 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 0x7f9cf019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0110530 con 0x7f9cf0101700 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01103b0 con 0x7f9cf0100d40 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01106b0 con 0x7f9cf0106dc0 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44568/0 (socket says 172.21.15.143:44568) 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 -- 172.21.15.143:0/2987626409 learned_addr learned my addr 172.21.15.143:0/2987626409 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cee7fc700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9ceeffd700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 0x7f9cf019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 -- 172.21.15.143:0/2987626409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 msgr2=0x7f9cf019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 0x7f9cf019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 -- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 msgr2=0x7f9cf019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf019dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 -- 172.21.15.143:0/2987626409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0198180 con 0x7f9cf0106dc0 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cee7fc700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cef7fe700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800ef30 tx=0x7f9ce800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cd7fff700 10 client.?.objecter ms_handle_connect 0x7f9cf0106dc0 2024-03-11T21:42:03.490 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cd7fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:03.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80092e0 con 0x7f9cf0106dc0 2024-03-11T21:42:03.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf0198410 con 0x7f9cf0106dc0 2024-03-11T21:42:03.491 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9ce8004bb0 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8004020 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 10 client.42041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf01028f0 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.500+0000 7f9cf52a9700 1 librados: init done 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0102400 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9ce801b030 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 0x7f9cd8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(820..820 src has 87..820) v4 ==== 10261+0+0 (secure 0 0 0) 0x7f9ce808e7b0 con 0x7f9cf0106dc0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 10 client.42041.objecter ms_dispatch 0x7f9cf0001550 osd_map(820..820 src has 87..820) v4 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 3 client.42041.objecter handle_osd_map got epochs [820,820] > 0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 3 client.42041.objecter handle_osd_map decoding full epoch 820 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9ceeffd700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 0x7f9cd8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 20 client.42041.objecter dump_active .. 0 homeless 2024-03-11T21:42:03.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9cd7fff700 10 client.42041.objecter ms_handle_connect 0x7f9cd806f820 2024-03-11T21:42:03.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.504+0000 7f9ceeffd700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 0x7f9cd8071ce0 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000b2e0 tx=0x7f9ce001a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.508+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9ce8022020 con 0x7f9cf0106dc0 2024-03-11T21:42:03.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.575 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:03.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:03.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:03.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.752+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f9cf0198f00 con 0x7f9cf0106dc0 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/2213825109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c107220 msgr2=0x7f552c10edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- 172.21.15.143:0/2213825109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c107220 0x7f552c10edd0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f552400b0a0 tx=0x7f552400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/2213825109 shutdown_connections 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- 172.21.15.143:0/2213825109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c107220 0x7f552c10edd0 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- 172.21.15.143:0/2213825109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c06a6a0 0x7f552c06aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- 172.21.15.143:0/2213825109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 0x7f552c106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/2213825109 >> 172.21.15.143:0/2213825109 conn(0x7f552c077de0 msgr2=0x7f552c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:03.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/2213825109 shutdown_connections 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/2213825109 wait complete. 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 librados: starting msgr at 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 librados: starting objecter 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 Processor -- start 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- start start 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 librados: setting wanted keys 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 librados: calling monclient init 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 0x7f552c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c107220 0x7f552c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c1116a0 con 0x7f552c06b050 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c111520 con 0x7f552c107220 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c111820 con 0x7f552c06a6a0 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44586/0 (socket says 172.21.15.143:44586) 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 -- 172.21.15.143:0/1235199940 learned_addr learned my addr 172.21.15.143:0/1235199940 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552a7fc700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 0x7f552c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552b7fe700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c107220 0x7f552c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 -- 172.21.15.143:0/1235199940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c107220 msgr2=0x7f552c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c107220 0x7f552c101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 -- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 msgr2=0x7f552c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 0x7f552c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 -- 172.21.15.143:0/1235199940 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c1a6130 con 0x7f552c06a6a0 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f552affd700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f551c00b530 tx=0x7f551c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 10 client.?.objecter ms_handle_connect 0x7f552c06a6a0 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c015020 con 0x7f552c06a6a0 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f551c011cb0 con 0x7f552c06a6a0 2024-03-11T21:42:03.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c1a6420 con 0x7f552c06a6a0 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c010d90 con 0x7f552c06a6a0 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 10 client.42047.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c1a6960 con 0x7f552c06a6a0 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 librados: init done 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c04faa0 con 0x7f552c06a6a0 2024-03-11T21:42:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.776+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f551c018020 con 0x7f552c06a6a0 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 0x7f5514071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(821..821 src has 87..821) v4 ==== 10261+0+0 (secure 0 0 0) 0x7f551c08e610 con 0x7f552c06a6a0 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 10 client.42047.objecter ms_dispatch 0x7f552c001550 osd_map(821..821 src has 87..821) v4 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 3 client.42047.objecter handle_osd_map got epochs [821,821] > 0 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 3 client.42047.objecter handle_osd_map decoding full epoch 821 2024-03-11T21:42:03.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f552a7fc700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 0x7f5514071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 20 client.42047.objecter dump_active .. 0 homeless 2024-03-11T21:42:03.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f5513fff700 10 client.42047.objecter ms_handle_connect 0x7f551406f820 2024-03-11T21:42:03.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.780+0000 7f552a7fc700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 0x7f5514071ce0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f5520009680 tx=0x7f5520009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.784+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f551c01f020 con 0x7f552c06a6a0 2024-03-11T21:42:03.800 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.820 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: cluster 2024-03-11T21:42:02.511789+0000 osd.6 (osd.6) 361 : cluster 0 5.11 scrub starts 2024-03-11T21:42:03.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: cluster 2024-03-11T21:42:02.516940+0000 osd.6 (osd.6) 362 : cluster 0 5.11 scrub ok 2024-03-11T21:42:03.821 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: cluster 2024-03-11T21:42:02.542633+0000 osd.0 (osd.0) 434 : cluster 0 3.17 scrub starts 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: cluster 2024-03-11T21:42:02.546987+0000 osd.0 (osd.0) 435 : cluster 0 3.17 scrub ok 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: audit 2024-03-11T21:42:03.645097+0000 mon.a (mon.0) 3744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: cluster 2024-03-11T21:42:03.651202+0000 mon.a (mon.0) 3745 : cluster 0 osdmap e821: 8 total, 8 up, 8 in 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: audit 2024-03-11T21:42:03.651906+0000 mon.c (mon.2) 1289 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[29284]: audit 2024-03-11T21:42:03.655608+0000 mon.a (mon.0) 3746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: cluster 2024-03-11T21:42:02.511789+0000 osd.6 (osd.6) 361 : cluster 0 5.11 scrub starts 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: cluster 2024-03-11T21:42:02.516940+0000 osd.6 (osd.6) 362 : cluster 0 5.11 scrub ok 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: cluster 2024-03-11T21:42:02.542633+0000 osd.0 (osd.0) 434 : cluster 0 3.17 scrub starts 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: cluster 2024-03-11T21:42:02.546987+0000 osd.0 (osd.0) 435 : cluster 0 3.17 scrub ok 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: audit 2024-03-11T21:42:03.645097+0000 mon.a (mon.0) 3744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:42:03.822 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: cluster 2024-03-11T21:42:03.651202+0000 mon.a (mon.0) 3745 : cluster 0 osdmap e821: 8 total, 8 up, 8 in 2024-03-11T21:42:03.823 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: audit 2024-03-11T21:42:03.651906+0000 mon.c (mon.2) 1289 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:03.823 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:03 smithi143 bash[21512]: audit 2024-03-11T21:42:03.655608+0000 mon.a (mon.0) 3746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:03.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2966888935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20109370 msgr2=0x7f4c20110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2966888935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20109370 0x7f4c20110f40 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009fc0 tx=0x7f4c1800b330 comp rx=0 tx=0).stop 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2966888935 shutdown_connections 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2966888935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20109370 0x7f4c20110f40 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2966888935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c20104290 0x7f4c20104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2966888935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c20104c40 0x7f4c20108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2966888935 >> 172.21.15.143:0/2966888935 conn(0x7f4c200ffb40 msgr2=0x7f4c20101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2966888935 shutdown_connections 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2966888935 wait complete. 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 librados: starting msgr at 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 librados: starting objecter 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 Processor -- start 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- start start 2024-03-11T21:42:03.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 librados: setting wanted keys 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 librados: calling monclient init 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 0x7f4c20199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c20104c40 0x7f4c2019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c20109370 0x7f4c2019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20113870 con 0x7f4c20104290 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c201136f0 con 0x7f4c20109370 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c201139f0 con 0x7f4c20104c40 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c20104c40 0x7f4c2019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 0x7f4c20199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 0x7f4c20199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41168/0 (socket says 172.21.15.143:41168) 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 -- 172.21.15.143:0/2965112337 learned_addr learned my addr 172.21.15.143:0/2965112337 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c20109370 0x7f4c2019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c20104c40 msgr2=0x7f4c2019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c20104c40 0x7f4c2019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c20109370 msgr2=0x7f4c2019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:03.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c20109370 0x7f4c2019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:03.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2019edd0 con 0x7f4c20104290 2024-03-11T21:42:03.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 0x7f4c20199dc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c920 tx=0x7f4c1400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1dffb700 10 client.?.objecter ms_handle_connect 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1dffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14007b00 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2019f0c0 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4c14004cb0 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14005400 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 10 client.30849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.900+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c201a6940 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 1 librados: init done 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4c2004faa0 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4c14004750 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4c1006f820 0x7f4c10071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(821..821 src has 87..821) v4 ==== 10261+0+0 (secure 0 0 0) 0x7f4c1408e910 con 0x7f4c20104290 2024-03-11T21:42:03.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 osd_map(821..821 src has 87..821) v4 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 3 client.30849.objecter handle_osd_map got epochs [821,821] > 0 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 3 client.30849.objecter handle_osd_map decoding full epoch 821 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4c1006f820 0x7f4c10071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 20 client.30849.objecter dump_active .. 0 homeless 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v821) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4c14014030 con 0x7f4c20104290 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 10 client.30849.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 1 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f4c200629f0 0x7f4c20062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 1 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 0 1 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 4 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 10 client.30849.objecter _send_command 1 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201a9d60 con 0x7f4c200629f0 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c1006f820 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4c1006f820 0x7f4c10071ce0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009750 tx=0x7f4c0c009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f4c200629f0 0x7f4c20062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f4c200629f0 0x7f4c20062dd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c200629f0 2024-03-11T21:42:03.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201a9d60 con 0x7f4c200629f0 2024-03-11T21:42:03.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:42:03.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 1 = system:0 2024-03-11T21:42:03.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 4 2024-03-11T21:42:03.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.904+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 0 1 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 10 client.30849.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 2 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 2 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2019b120 osd=0 3 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 0 2 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 4 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 10 client.30849.objecter _send_command 2 2024-03-11T21:42:03.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c20062470 con 0x7f4c200629f0 2024-03-11T21:42:03.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c20062470 con 0x7f4c200629f0 2024-03-11T21:42:03.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:03.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:03.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 4 2024-03-11T21:42:03.911 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 0 2 2024-03-11T21:42:03.911 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 10 client.30849.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f4c2002c6b0 0x7f4c2002cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 1 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 4 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 10 client.30849.objecter _send_command 3 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c2010bc90 con 0x7f4c2002c6b0 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f4c2002c6b0 0x7f4c2002cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c2002c6b0 2024-03-11T21:42:03.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f4c2002c6b0 0x7f4c2002cab0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c2010bc90 con 0x7f4c2002c6b0 2024-03-11T21:42:03.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:42:03.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 3 = system:0 2024-03-11T21:42:03.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 4 2024-03-11T21:42:03.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.924+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 1 3 2024-03-11T21:42:03.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 10 client.30849.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:03.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 4 2024-03-11T21:42:03.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 4 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2010b610 osd=1 3 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 1 4 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 4 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 10 client.30849.objecter _send_command 4 2024-03-11T21:42:03.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.940+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c2010b090 con 0x7f4c2002c6b0 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c2010b090 con 0x7f4c2002c6b0 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 4 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 1 4 2024-03-11T21:42:03.930 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 10 client.30849.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 5 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f4c201abf20 0x7f4c201ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 5 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 2 5 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 4 2024-03-11T21:42:03.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 10 client.30849.objecter _send_command 5 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201b1170 con 0x7f4c201abf20 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f4c201abf20 0x7f4c201ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201abf20 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f4c201abf20 0x7f4c201ae310 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201b1170 con 0x7f4c201abf20 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 5 = system:0 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 4 2024-03-11T21:42:03.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.944+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 2 5 2024-03-11T21:42:03.942 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 10 client.30849.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 6 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 6 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c2010b490 osd=2 3 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 2 6 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 4 2024-03-11T21:42:03.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 10 client.30849.objecter _send_command 6 2024-03-11T21:42:03.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c2002d240 con 0x7f4c201abf20 2024-03-11T21:42:03.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c2002d240 con 0x7f4c201abf20 2024-03-11T21:42:03.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:03.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:03.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 4 2024-03-11T21:42:03.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 2 6 2024-03-11T21:42:03.951 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 10 client.30849.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 7 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f4c201af9d0 0x7f4c201afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 7 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 3 7 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 4 2024-03-11T21:42:03.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 10 client.30849.objecter _send_command 7 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201a9a80 con 0x7f4c201af9d0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f4c201af9d0 0x7f4c201afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201af9d0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f4c201af9d0 0x7f4c201afe30 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201a9a80 con 0x7f4c201af9d0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 7 = system:0 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 4 2024-03-11T21:42:03.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.964+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 3 7 2024-03-11T21:42:03.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 10 client.30849.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:03.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 8 2024-03-11T21:42:03.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 8 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201a93b0 osd=3 3 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 3 8 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 4 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 10 client.30849.objecter _send_command 8 2024-03-11T21:42:03.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c201a9a80 con 0x7f4c201af9d0 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c201a9a80 con 0x7f4c201af9d0 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 4 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 3 8 2024-03-11T21:42:03.971 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 10 client.30849.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 9 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f4c201b0b40 0x7f4c201b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 9 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 4 9 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 4 2024-03-11T21:42:03.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 10 client.30849.objecter _send_command 9 2024-03-11T21:42:03.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201b81a0 con 0x7f4c201b0b40 2024-03-11T21:42:03.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f4c201b0b40 0x7f4c201b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201b0b40 2024-03-11T21:42:03.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.984+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f4c201b0b40 0x7f4c201b5340 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:03.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.992+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201b81a0 con 0x7f4c201b0b40 2024-03-11T21:42:03.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.992+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:42:03.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.992+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 9 = system:0 2024-03-11T21:42:03.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.992+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 4 2024-03-11T21:42:03.977 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.992+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 4 9 2024-03-11T21:42:03.984 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:03.996+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f552c0ff240 con 0x7f551406f820 2024-03-11T21:42:03.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 10 client.30849.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:03.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 10 2024-03-11T21:42:03.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 10 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201a9a80 osd=4 3 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 4 10 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 4 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 10 client.30849.objecter _send_command 10 2024-03-11T21:42:03.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.008+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c201b5880 con 0x7f4c201b0b40 2024-03-11T21:42:03.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c201b5880 con 0x7f4c201b0b40 2024-03-11T21:42:03.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:03.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:03.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 4 2024-03-11T21:42:03.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 4 10 2024-03-11T21:42:03.998 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 10 client.30849.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 11 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f4c201b5fa0 0x7f4c201b6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 11 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 5 11 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 4 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 10 client.30849.objecter _send_command 11 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201bb540 con 0x7f4c201b5fa0 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f4c201b5fa0 0x7f4c201b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:03.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201b5fa0 2024-03-11T21:42:04.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.012+0000 7f4c1ffff700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f4c201b5fa0 0x7f4c201b6400 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:04.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.016+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201bb540 con 0x7f4c201b5fa0 2024-03-11T21:42:04.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.016+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:42:04.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.016+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 11 = system:0 2024-03-11T21:42:04.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.016+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 4 2024-03-11T21:42:04.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.016+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 5 11 2024-03-11T21:42:04.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.020+0000 7f5513fff700 1 -- 172.21.15.143:0/1235199940 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+633708 (secure 0 0 0) 0x7f552c0ff240 con 0x7f551406f820 2024-03-11T21:42:04.013 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 10 librados: watch_flush enter 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 10 librados: watch_flush exit 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 20 client.42047.objecter shutdown clearing up homeless session... 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 10 client.42047.objecter successfully canceled tick 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 msgr2=0x7f5514071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 0x7f5514071ce0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f5520009680 tx=0x7f5520009260 comp rx=0 tx=0).stop 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 msgr2=0x7f552c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f551c00b530 tx=0x7f551c00b840 comp rx=0 tx=0).stop 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 shutdown_connections 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f551406f820 0x7f5514071ce0 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f552c107220 0x7f552c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f552c06a6a0 0x7f552c100d90 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 --2- 172.21.15.143:0/1235199940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f552c06b050 0x7f552c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.024+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 >> 172.21.15.143:0/1235199940 conn(0x7f552c077de0 msgr2=0x7f552c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.028+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 shutdown_connections 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.028+0000 7f553144b700 1 -- 172.21.15.143:0/1235199940 wait complete. 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.028+0000 7f553144b700 1 librados: shutdown 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 10 client.30849.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 12 2024-03-11T21:42:04.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 12 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b5880 osd=5 3 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 5 12 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 4 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 10 client.30849.objecter _send_command 12 2024-03-11T21:42:04.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.036+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c201b6a50 con 0x7f4c201b5fa0 2024-03-11T21:42:04.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c201b6a50 con 0x7f4c201b5fa0 2024-03-11T21:42:04.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:04.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:04.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 4 2024-03-11T21:42:04.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 5 12 2024-03-11T21:42:04.026 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 10 client.30849.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 13 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4c201b70d0 0x7f4c201b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 13 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 6 13 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 4 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 10 client.30849.objecter _send_command 13 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201b7c30 con 0x7f4c201b70d0 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4c201b70d0 0x7f4c201b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201b70d0 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.040+0000 7f4c254b5700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4c201b70d0 0x7f4c201b7550 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.044+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201b7c30 con 0x7f4c201b70d0 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.044+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:42:04.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.044+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 13 = system:0 2024-03-11T21:42:04.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.044+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 4 2024-03-11T21:42:04.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.044+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 6 13 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: cluster 2024-03-11T21:42:02.511789+0000 osd.6 (osd.6) 361 : cluster 0 5.11 scrub starts 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: cluster 2024-03-11T21:42:02.516940+0000 osd.6 (osd.6) 362 : cluster 0 5.11 scrub ok 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: cluster 2024-03-11T21:42:02.542633+0000 osd.0 (osd.0) 434 : cluster 0 3.17 scrub starts 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: cluster 2024-03-11T21:42:02.546987+0000 osd.0 (osd.0) 435 : cluster 0 3.17 scrub ok 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: audit 2024-03-11T21:42:03.645097+0000 mon.a (mon.0) 3744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: cluster 2024-03-11T21:42:03.651202+0000 mon.a (mon.0) 3745 : cluster 0 osdmap e821: 8 total, 8 up, 8 in 2024-03-11T21:42:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: audit 2024-03-11T21:42:03.651906+0000 mon.c (mon.2) 1289 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:04.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:03 smithi203 bash[23654]: audit 2024-03-11T21:42:03.655608+0000 mon.a (mon.0) 3746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:04.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 10 client.30849.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 14 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 14 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b6a50 osd=6 3 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 6 14 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 4 2024-03-11T21:42:04.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 10 client.30849.objecter _send_command 14 2024-03-11T21:42:04.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.060+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c201b7c30 con 0x7f4c201b70d0 2024-03-11T21:42:04.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c201b7c30 con 0x7f4c201b70d0 2024-03-11T21:42:04.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:04.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:04.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 4 2024-03-11T21:42:04.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 6 14 2024-03-11T21:42:04.052 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 10 client.30849.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 15 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f4c201bc8e0 0x7f4c201becd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 15 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 7 15 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 4 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 10 client.30849.objecter _send_command 15 2024-03-11T21:42:04.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.064+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c201c1b30 con 0x7f4c201bc8e0 2024-03-11T21:42:04.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f4c201bc8e0 0x7f4c201becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:04.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 10 client.30849.objecter ms_handle_connect 0x7f4c201bc8e0 2024-03-11T21:42:04.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c24cb4700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f4c201bc8e0 0x7f4c201becd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:04.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c201c1b30 con 0x7f4c201bc8e0 2024-03-11T21:42:04.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:42:04.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 15 = system:0 2024-03-11T21:42:04.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 4 2024-03-11T21:42:04.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.068+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 7 15 2024-03-11T21:42:04.063 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 10 client.30849.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign -1 16 2024-03-11T21:42:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 20 client.30849.objecter _get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_remove -1 16 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 20 client.30849.objecter get_session s=0x7f4c201b7c30 osd=7 3 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 15 client.30849.objecter _session_command_op_assign 7 16 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 4 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 10 client.30849.objecter _send_command 16 2024-03-11T21:42:04.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.084+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4c201bf210 con 0x7f4c201bc8e0 2024-03-11T21:42:04.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c1dffb700 1 -- 172.21.15.143:0/2965112337 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4c201bf210 con 0x7f4c201bc8e0 2024-03-11T21:42:04.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c1dffb700 10 client.30849.objecter ms_dispatch 0x7f4c20001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:04.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c1dffb700 10 client.30849.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:04.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c1dffb700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 4 2024-03-11T21:42:04.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c1dffb700 15 client.30849.objecter _session_command_op_remove 7 16 2024-03-11T21:42:04.077 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 librados: watch_flush enter 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 librados: watch_flush exit 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.0 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f4c200629f0 msgr2=0x7f4c20062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f4c200629f0 0x7f4c20062dd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2019b120 osd=0 1 2024-03-11T21:42:04.078 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.1 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f4c2002c6b0 msgr2=0x7f4c2002cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f4c2002c6b0 0x7f4c2002cab0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b610 osd=1 1 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.2 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f4c201abf20 msgr2=0x7f4c201ae310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f4c201abf20 0x7f4c201ae310 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c2010b490 osd=2 1 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.3 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f4c201af9d0 msgr2=0x7f4c201afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f4c201af9d0 0x7f4c201afe30 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a93b0 osd=3 1 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.4 2024-03-11T21:42:04.079 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f4c201b0b40 msgr2=0x7f4c201b5340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f4c201b0b40 0x7f4c201b5340 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201a9a80 osd=4 1 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.5 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f4c201b5fa0 msgr2=0x7f4c201b6400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c254b5700 1 -- 172.21.15.143:0/2965112337 reap_dead start 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f4c201b5fa0 0x7f4c201b6400 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c254b5700 1 -- 172.21.15.143:0/2965112337 reap_dead start 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c254b5700 1 -- 172.21.15.143:0/2965112337 reap_dead start 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b5880 osd=5 1 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.6 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4c201b70d0 msgr2=0x7f4c201b7550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f4c201b70d0 0x7f4c201b7550 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b6a50 osd=6 1 2024-03-11T21:42:04.080 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter close_session for osd.7 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f4c201bc8e0 msgr2=0x7f4c201becd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f4c201bc8e0 0x7f4c201becd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter put_session s=0x7f4c201b7c30 osd=7 1 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 20 client.30849.objecter shutdown clearing up homeless session... 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 10 client.30849.objecter successfully canceled tick 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4c1006f820 msgr2=0x7f4c10071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4c1006f820 0x7f4c10071ce0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009750 tx=0x7f4c0c009310 comp rx=0 tx=0).stop 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 msgr2=0x7f4c20199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 --2- 172.21.15.143:0/2965112337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c20104290 0x7f4c20199dc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c920 tx=0x7f4c1400cc30 comp rx=0 tx=0).stop 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c254b5700 1 -- 172.21.15.143:0/2965112337 reap_dead start 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 shutdown_connections 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 >> 172.21.15.143:0/2965112337 conn(0x7f4c200ffb40 msgr2=0x7f4c20100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 shutdown_connections 2024-03-11T21:42:04.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 -- 172.21.15.143:0/2965112337 wait complete. 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.092+0000 7f4c26f18700 1 librados: shutdown 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:42:04.082 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:42:04.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:04.188 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:04.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.319 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:04.422 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:04.452 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:04.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:04.580 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:04.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.648+0000 7f9cd7fff700 1 -- 172.21.15.143:0/2987626409 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v822) v1 ==== 87+0+0 (secure 0 0 0) 0x7f9ce8010040 con 0x7f9cf0106dc0 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 10 librados: watch_flush enter 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 10 librados: watch_flush exit 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 20 client.42041.objecter shutdown clearing up homeless session... 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 10 client.42041.objecter successfully canceled tick 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 msgr2=0x7f9cd8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 0x7f9cd8071ce0 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000b2e0 tx=0x7f9ce001a040 comp rx=0 tx=0).stop 2024-03-11T21:42:04.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 msgr2=0x7f9cf0197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800ef30 tx=0x7f9ce800c3a0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 shutdown_connections 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9cd806f820 0x7f9cd8071ce0 unknown :-1 s=CLOSED pgs=3450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cf0106dc0 0x7f9cf0197a20 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf0100d40 0x7f9cf019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 --2- 172.21.15.143:0/2987626409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf0101700 0x7f9cf019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 >> 172.21.15.143:0/2987626409 conn(0x7f9cf0076d80 msgr2=0x7f9cf0104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 shutdown_connections 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 -- 172.21.15.143:0/2987626409 wait complete. 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:04.652+0000 7f9cf52a9700 1 librados: shutdown 2024-03-11T21:42:04.638 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:42:04.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:04.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:04.840 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: cluster 2024-03-11T21:42:03.606824+0000 mgr.y (mgr.14550) 739 : cluster 0 pgmap v1269: 246 pgs: 6 creating+peering, 1 unknown, 239 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: cluster 2024-03-11T21:42:03.750199+0000 osd.3 (osd.3) 441 : cluster 0 3.7 scrub starts 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: cluster 2024-03-11T21:42:03.752916+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub ok 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:03.757599+0000 mon.c (mon.2) 1290 : audit 1 from='client.? 172.21.15.143:0/2987626409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:03.758045+0000 mon.a (mon.0) 3747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:03.909673+0000 mon.a (mon.0) 3748 : audit 0 from='client.? 172.21.15.143:0/2965112337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:04.004090+0000 mgr.y (mgr.14550) 740 : audit 0 from='client.42047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: cluster 2024-03-11T21:42:04.645531+0000 mon.a (mon.0) 3749 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:04.647292+0000 mon.a (mon.0) 3750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:04.647360+0000 mon.a (mon.0) 3751 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: cluster 2024-03-11T21:42:04.652531+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e822: 8 total, 8 up, 8 in 2024-03-11T21:42:04.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:04.656296+0000 mon.c (mon.2) 1291 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[29284]: audit 2024-03-11T21:42:04.657294+0000 mon.a (mon.0) 3753 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: cluster 2024-03-11T21:42:03.606824+0000 mgr.y (mgr.14550) 739 : cluster 0 pgmap v1269: 246 pgs: 6 creating+peering, 1 unknown, 239 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: cluster 2024-03-11T21:42:03.750199+0000 osd.3 (osd.3) 441 : cluster 0 3.7 scrub starts 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: cluster 2024-03-11T21:42:03.752916+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub ok 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:03.757599+0000 mon.c (mon.2) 1290 : audit 1 from='client.? 172.21.15.143:0/2987626409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:03.758045+0000 mon.a (mon.0) 3747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:03.909673+0000 mon.a (mon.0) 3748 : audit 0 from='client.? 172.21.15.143:0/2965112337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:04.004090+0000 mgr.y (mgr.14550) 740 : audit 0 from='client.42047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: cluster 2024-03-11T21:42:04.645531+0000 mon.a (mon.0) 3749 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:04.647292+0000 mon.a (mon.0) 3750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:04.647360+0000 mon.a (mon.0) 3751 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: cluster 2024-03-11T21:42:04.652531+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e822: 8 total, 8 up, 8 in 2024-03-11T21:42:04.852 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:04.656296+0000 mon.c (mon.2) 1291 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.853 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:04 smithi143 bash[21512]: audit 2024-03-11T21:42:04.657294+0000 mon.a (mon.0) 3753 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.943 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.955 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: cluster 2024-03-11T21:42:03.606824+0000 mgr.y (mgr.14550) 739 : cluster 0 pgmap v1269: 246 pgs: 6 creating+peering, 1 unknown, 239 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: cluster 2024-03-11T21:42:03.750199+0000 osd.3 (osd.3) 441 : cluster 0 3.7 scrub starts 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: cluster 2024-03-11T21:42:03.752916+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub ok 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:03.757599+0000 mon.c (mon.2) 1290 : audit 1 from='client.? 172.21.15.143:0/2987626409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:03.758045+0000 mon.a (mon.0) 3747 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:03.909673+0000 mon.a (mon.0) 3748 : audit 0 from='client.? 172.21.15.143:0/2965112337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:04.004090+0000 mgr.y (mgr.14550) 740 : audit 0 from='client.42047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: cluster 2024-03-11T21:42:04.645531+0000 mon.a (mon.0) 3749 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:04.647292+0000 mon.a (mon.0) 3750 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:04.647360+0000 mon.a (mon.0) 3751 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: cluster 2024-03-11T21:42:04.652531+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e822: 8 total, 8 up, 8 in 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:04.656296+0000 mon.c (mon.2) 1291 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:04 smithi203 bash[23654]: audit 2024-03-11T21:42:04.657294+0000 mon.a (mon.0) 3753 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:42:04.971 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:05.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.102 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:05.205 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:05.230 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:05.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:05.362 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:05.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:05.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.621 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:05.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:05.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:05.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.879 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:05.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[29284]: cluster 2024-03-11T21:42:04.405209+0000 osd.7 (osd.7) 436 : cluster 0 2.0 scrub starts 2024-03-11T21:42:05.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[29284]: cluster 2024-03-11T21:42:04.407605+0000 osd.7 (osd.7) 437 : cluster 0 2.0 scrub ok 2024-03-11T21:42:05.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[29284]: audit 2024-03-11T21:42:05.649258+0000 mon.a (mon.0) 3754 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:42:05.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[29284]: cluster 2024-03-11T21:42:05.653640+0000 mon.a (mon.0) 3755 : cluster 0 osdmap e823: 8 total, 8 up, 8 in 2024-03-11T21:42:05.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[21512]: cluster 2024-03-11T21:42:04.405209+0000 osd.7 (osd.7) 436 : cluster 0 2.0 scrub starts 2024-03-11T21:42:05.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[21512]: cluster 2024-03-11T21:42:04.407605+0000 osd.7 (osd.7) 437 : cluster 0 2.0 scrub ok 2024-03-11T21:42:05.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[21512]: audit 2024-03-11T21:42:05.649258+0000 mon.a (mon.0) 3754 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:42:05.882 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:05 smithi143 bash[21512]: cluster 2024-03-11T21:42:05.653640+0000 mon.a (mon.0) 3755 : cluster 0 osdmap e823: 8 total, 8 up, 8 in 2024-03-11T21:42:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:05.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:05.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:05 smithi203 bash[23654]: cluster 2024-03-11T21:42:04.405209+0000 osd.7 (osd.7) 436 : cluster 0 2.0 scrub starts 2024-03-11T21:42:05.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:05 smithi203 bash[23654]: cluster 2024-03-11T21:42:04.407605+0000 osd.7 (osd.7) 437 : cluster 0 2.0 scrub ok 2024-03-11T21:42:05.995 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:05 smithi203 bash[23654]: audit 2024-03-11T21:42:05.649258+0000 mon.a (mon.0) 3754 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:42:05.996 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:05 smithi203 bash[23654]: cluster 2024-03-11T21:42:05.653640+0000 mon.a (mon.0) 3755 : cluster 0 osdmap e823: 8 total, 8 up, 8 in 2024-03-11T21:42:06.012 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:06.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:06.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.143 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:06.246 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:06.273 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:06.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.403 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:06.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:06.537 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:06.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:06.671 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:06.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[21512]: cluster 2024-03-11T21:42:05.509661+0000 osd.6 (osd.6) 363 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[21512]: cluster 2024-03-11T21:42:05.515027+0000 osd.6 (osd.6) 364 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[21512]: cluster 2024-03-11T21:42:05.608000+0000 mgr.y (mgr.14550) 741 : cluster 0 pgmap v1272: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[21512]: cluster 2024-03-11T21:42:06.545837+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e824: 8 total, 8 up, 8 in 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[29284]: cluster 2024-03-11T21:42:05.509661+0000 osd.6 (osd.6) 363 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[29284]: cluster 2024-03-11T21:42:05.515027+0000 osd.6 (osd.6) 364 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:42:06.796 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[29284]: cluster 2024-03-11T21:42:05.608000+0000 mgr.y (mgr.14550) 741 : cluster 0 pgmap v1272: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:06.797 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:06 smithi143 bash[29284]: cluster 2024-03-11T21:42:06.545837+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e824: 8 total, 8 up, 8 in 2024-03-11T21:42:06.798 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-03-11T21:42:03.649903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596486,"num_objects":216,"num_object_clones":1,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":2306,"num_read_kb":2046,"num_write":766,"num_write_kb":3991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2373,"ondisk_log_size":2373,"up":738,"acting":738,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":705,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":692288,"kb_used_data":114200,"kb_used_omap":38,"kb_used_meta":577817,"kb_avail":749268928,"statfs":{"total":767960285184,"available":767251382272,"internally_reserved":0,"allocated":116940800,"data_stored":76213144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":591685045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011339"},"pg_stats":[{"pgid":"328.1f","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.666118+0000","last_change":"2024-03-11T21:41:59.666118+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.666118+0000","last_undegraded":"2024-03-11T21:41:59.666118+0000","last_fullsized":"2024-03-11T21:41:59.666118+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.1e","version":"0'0","reported_seq":14,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703419+0000","last_change":"2024-03-11T21:42:00.669027+0000","last_active":"2024-03-11T21:42:02.703419+0000","last_peered":"2024-03-11T21:42:02.703419+0000","last_clean":"2024-03-11T21:42:02.703419+0000","last_became_active":"2024-03-11T21:42:00.668634+0000","last_became_peered":"2024-03-11T21:42:00.668634+0000","last_unstale":"2024-03-11T21:42:02.703419+0000","last_undegraded":"2024-03-11T21:42:02.703419+0000","last_fullsized":"2024-03-11T21:42:02.703419+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.388572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"328.1d","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999529+0000","last_change":"2024-03-11T21:42:00.682682+0000","last_active":"2024-03-11T21:42:02.999529+0000","last_peered":"2024-03-11T21:42:02.999529+0000","last_clean":"2024-03-11T21:42:02.999529+0000","last_became_active":"2024-03-11T21:42:00.682035+0000","last_became_peered":"2024-03-11T21:42:00.682035+0000","last_unstale":"2024-03-11T21:42:02.999529+0000","last_undegraded":"2024-03-11T21:42:02.999529+0000","last_fullsized":"2024-03-11T21:42:02.999529+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.713283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.1c","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644721+0000","last_change":"2024-03-11T21:42:00.680563+0000","last_active":"2024-03-11T21:42:02.644721+0000","last_peered":"2024-03-11T21:42:02.644721+0000","last_clean":"2024-03-11T21:42:02.644721+0000","last_became_active":"2024-03-11T21:42:00.679742+0000","last_became_peered":"2024-03-11T21:42:00.679742+0000","last_unstale":"2024-03-11T21:42:02.644721+0000","last_undegraded":"2024-03-11T21:42:02.644721+0000","last_fullsized":"2024-03-11T21:42:02.644721+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.690977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.1b","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999447+0000","last_change":"2024-03-11T21:42:00.677969+0000","last_active":"2024-03-11T21:42:02.999447+0000","last_peered":"2024-03-11T21:42:02.999447+0000","last_clean":"2024-03-11T21:42:02.999447+0000","last_became_active":"2024-03-11T21:42:00.677505+0000","last_became_peered":"2024-03-11T21:42:00.677505+0000","last_unstale":"2024-03-11T21:42:02.999447+0000","last_undegraded":"2024-03-11T21:42:02.999447+0000","last_fullsized":"2024-03-11T21:42:02.999447+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.408789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.1a","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.668661+0000","last_change":"2024-03-11T21:41:59.668661+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.668661+0000","last_undegraded":"2024-03-11T21:41:59.668661+0000","last_fullsized":"2024-03-11T21:41:59.668661+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.19","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644171+0000","last_change":"2024-03-11T21:42:00.679993+0000","last_active":"2024-03-11T21:42:02.644171+0000","last_peered":"2024-03-11T21:42:02.644171+0000","last_clean":"2024-03-11T21:42:02.644171+0000","last_became_active":"2024-03-11T21:42:00.679236+0000","last_became_peered":"2024-03-11T21:42:00.679236+0000","last_unstale":"2024-03-11T21:42:02.644171+0000","last_undegraded":"2024-03-11T21:42:02.644171+0000","last_fullsized":"2024-03-11T21:42:02.644171+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.161807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.18","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.642624+0000","last_change":"2024-03-11T21:41:59.642624+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.642624+0000","last_undegraded":"2024-03-11T21:41:59.642624+0000","last_fullsized":"2024-03-11T21:41:59.642624+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.17","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646313+0000","last_change":"2024-03-11T21:42:00.680042+0000","last_active":"2024-03-11T21:42:02.646313+0000","last_peered":"2024-03-11T21:42:02.646313+0000","last_clean":"2024-03-11T21:42:02.646313+0000","last_became_active":"2024-03-11T21:42:00.679472+0000","last_became_peered":"2024-03-11T21:42:00.679472+0000","last_unstale":"2024-03-11T21:42:02.646313+0000","last_undegraded":"2024-03-11T21:42:02.646313+0000","last_fullsized":"2024-03-11T21:42:02.646313+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:26.672525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.16","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.660239+0000","last_change":"2024-03-11T21:42:00.689660+0000","last_active":"2024-03-11T21:42:02.660239+0000","last_peered":"2024-03-11T21:42:02.660239+0000","last_clean":"2024-03-11T21:42:02.660239+0000","last_became_active":"2024-03-11T21:42:00.688674+0000","last_became_peered":"2024-03-11T21:42:00.688674+0000","last_unstale":"2024-03-11T21:42:02.660239+0000","last_undegraded":"2024-03-11T21:42:02.660239+0000","last_fullsized":"2024-03-11T21:42:02.660239+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.734131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"328.15","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999865+0000","last_change":"2024-03-11T21:42:00.666523+0000","last_active":"2024-03-11T21:42:02.999865+0000","last_peered":"2024-03-11T21:42:02.999865+0000","last_clean":"2024-03-11T21:42:02.999865+0000","last_became_active":"2024-03-11T21:42:00.666121+0000","last_became_peered":"2024-03-11T21:42:00.666121+0000","last_unstale":"2024-03-11T21:42:02.999865+0000","last_undegraded":"2024-03-11T21:42:02.999865+0000","last_fullsized":"2024-03-11T21:42:02.999865+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.633463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"328.14","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999238+0000","last_change":"2024-03-11T21:42:00.681818+0000","last_active":"2024-03-11T21:42:02.999238+0000","last_peered":"2024-03-11T21:42:02.999238+0000","last_clean":"2024-03-11T21:42:02.999238+0000","last_became_active":"2024-03-11T21:42:00.681282+0000","last_became_peered":"2024-03-11T21:42:00.681282+0000","last_unstale":"2024-03-11T21:42:02.999238+0000","last_undegraded":"2024-03-11T21:42:02.999238+0000","last_fullsized":"2024-03-11T21:42:02.999238+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:29.298371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.13","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.642928+0000","last_change":"2024-03-11T21:41:59.642928+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.642928+0000","last_undegraded":"2024-03-11T21:41:59.642928+0000","last_fullsized":"2024-03-11T21:41:59.642928+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.12","version":"0'0","reported_seq":14,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702201+0000","last_change":"2024-03-11T21:42:00.680397+0000","last_active":"2024-03-11T21:42:02.702201+0000","last_peered":"2024-03-11T21:42:02.702201+0000","last_clean":"2024-03-11T21:42:02.702201+0000","last_became_active":"2024-03-11T21:42:00.679566+0000","last_became_peered":"2024-03-11T21:42:00.679566+0000","last_unstale":"2024-03-11T21:42:02.702201+0000","last_undegraded":"2024-03-11T21:42:02.702201+0000","last_fullsized":"2024-03-11T21:42:02.702201+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.709674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"328.11","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997346+0000","last_change":"2024-03-11T21:42:00.668298+0000","last_active":"2024-03-11T21:42:02.997346+0000","last_peered":"2024-03-11T21:42:02.997346+0000","last_clean":"2024-03-11T21:42:02.997346+0000","last_became_active":"2024-03-11T21:42:00.667847+0000","last_became_peered":"2024-03-11T21:42:00.667847+0000","last_unstale":"2024-03-11T21:42:02.997346+0000","last_undegraded":"2024-03-11T21:42:02.997346+0000","last_fullsized":"2024-03-11T21:42:02.997346+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.341343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.10","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.645516+0000","last_change":"2024-03-11T21:41:59.645516+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.645516+0000","last_undegraded":"2024-03-11T21:41:59.645516+0000","last_fullsized":"2024-03-11T21:41:59.645516+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.f","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.647081+0000","last_change":"2024-03-11T21:42:00.675009+0000","last_active":"2024-03-11T21:42:02.647081+0000","last_peered":"2024-03-11T21:42:02.647081+0000","last_clean":"2024-03-11T21:42:02.647081+0000","last_became_active":"2024-03-11T21:42:00.674395+0000","last_became_peered":"2024-03-11T21:42:00.674395+0000","last_unstale":"2024-03-11T21:42:02.647081+0000","last_undegraded":"2024-03-11T21:42:02.647081+0000","last_fullsized":"2024-03-11T21:42:02.647081+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.678676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.e","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004789+0000","last_change":"2024-03-11T21:42:00.685331+0000","last_active":"2024-03-11T21:42:03.004789+0000","last_peered":"2024-03-11T21:42:03.004789+0000","last_clean":"2024-03-11T21:42:03.004789+0000","last_became_active":"2024-03-11T21:42:00.684938+0000","last_became_peered":"2024-03-11T21:42:00.684938+0000","last_unstale":"2024-03-11T21:42:03.004789+0000","last_undegraded":"2024-03-11T21:42:03.004789+0000","last_fullsized":"2024-03-11T21:42:03.004789+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.792457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.d","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645754+0000","last_change":"2024-03-11T21:42:00.664423+0000","last_active":"2024-03-11T21:42:02.645754+0000","last_peered":"2024-03-11T21:42:02.645754+0000","last_clean":"2024-03-11T21:42:02.645754+0000","last_became_active":"2024-03-11T21:42:00.663468+0000","last_became_peered":"2024-03-11T21:42:00.663468+0000","last_unstale":"2024-03-11T21:42:02.645754+0000","last_undegraded":"2024-03-11T21:42:02.645754+0000","last_fullsized":"2024-03-11T21:42:02.645754+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:16.268666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.c","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999086+0000","last_change":"2024-03-11T21:42:00.675551+0000","last_active":"2024-03-11T21:42:02.999086+0000","last_peered":"2024-03-11T21:42:02.999086+0000","last_clean":"2024-03-11T21:42:02.999086+0000","last_became_active":"2024-03-11T21:42:00.675124+0000","last_became_peered":"2024-03-11T21:42:00.675124+0000","last_unstale":"2024-03-11T21:42:02.999086+0000","last_undegraded":"2024-03-11T21:42:02.999086+0000","last_fullsized":"2024-03-11T21:42:02.999086+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.654328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"328.b","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999683+0000","last_change":"2024-03-11T21:42:00.677779+0000","last_active":"2024-03-11T21:42:02.999683+0000","last_peered":"2024-03-11T21:42:02.999683+0000","last_clean":"2024-03-11T21:42:02.999683+0000","last_became_active":"2024-03-11T21:42:00.677425+0000","last_became_peered":"2024-03-11T21:42:00.677425+0000","last_unstale":"2024-03-11T21:42:02.999683+0000","last_undegraded":"2024-03-11T21:42:02.999683+0000","last_fullsized":"2024-03-11T21:42:02.999683+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:59.880597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.a","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998528+0000","last_change":"2024-03-11T21:42:00.675386+0000","last_active":"2024-03-11T21:42:02.998528+0000","last_peered":"2024-03-11T21:42:02.998528+0000","last_clean":"2024-03-11T21:42:02.998528+0000","last_became_active":"2024-03-11T21:42:00.675121+0000","last_became_peered":"2024-03-11T21:42:00.675121+0000","last_unstale":"2024-03-11T21:42:02.998528+0000","last_undegraded":"2024-03-11T21:42:02.998528+0000","last_fullsized":"2024-03-11T21:42:02.998528+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.826057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.9","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999622+0000","last_change":"2024-03-11T21:42:00.666289+0000","last_active":"2024-03-11T21:42:02.999622+0000","last_peered":"2024-03-11T21:42:02.999622+0000","last_clean":"2024-03-11T21:42:02.999622+0000","last_became_active":"2024-03-11T21:42:00.665950+0000","last_became_peered":"2024-03-11T21:42:00.665950+0000","last_unstale":"2024-03-11T21:42:02.999622+0000","last_undegraded":"2024-03-11T21:42:02.999622+0000","last_fullsized":"2024-03-11T21:42:02.999622+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:18.646528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"328.8","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997950+0000","last_change":"2024-03-11T21:42:00.681388+0000","last_active":"2024-03-11T21:42:02.997950+0000","last_peered":"2024-03-11T21:42:02.997950+0000","last_clean":"2024-03-11T21:42:02.997950+0000","last_became_active":"2024-03-11T21:42:00.681036+0000","last_became_peered":"2024-03-11T21:42:00.681036+0000","last_unstale":"2024-03-11T21:42:02.997950+0000","last_undegraded":"2024-03-11T21:42:02.997950+0000","last_fullsized":"2024-03-11T21:42:02.997950+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.784555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.7","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003453+0000","last_change":"2024-03-11T21:42:00.669121+0000","last_active":"2024-03-11T21:42:03.003453+0000","last_peered":"2024-03-11T21:42:03.003453+0000","last_clean":"2024-03-11T21:42:03.003453+0000","last_became_active":"2024-03-11T21:42:00.668672+0000","last_became_peered":"2024-03-11T21:42:00.668672+0000","last_unstale":"2024-03-11T21:42:03.003453+0000","last_undegraded":"2024-03-11T21:42:03.003453+0000","last_fullsized":"2024-03-11T21:42:03.003453+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:27.872845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"328.6","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002098+0000","last_change":"2024-03-11T21:42:00.659676+0000","last_active":"2024-03-11T21:42:03.002098+0000","last_peered":"2024-03-11T21:42:03.002098+0000","last_clean":"2024-03-11T21:42:03.002098+0000","last_became_active":"2024-03-11T21:42:00.659320+0000","last_became_peered":"2024-03-11T21:42:00.659320+0000","last_unstale":"2024-03-11T21:42:03.002098+0000","last_undegraded":"2024-03-11T21:42:03.002098+0000","last_fullsized":"2024-03-11T21:42:03.002098+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.101587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"328.5","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000150+0000","last_change":"2024-03-11T21:42:00.682674+0000","last_active":"2024-03-11T21:42:03.000150+0000","last_peered":"2024-03-11T21:42:03.000150+0000","last_clean":"2024-03-11T21:42:03.000150+0000","last_became_active":"2024-03-11T21:42:00.681428+0000","last_became_peered":"2024-03-11T21:42:00.681428+0000","last_unstale":"2024-03-11T21:42:03.000150+0000","last_undegraded":"2024-03-11T21:42:03.000150+0000","last_fullsized":"2024-03-11T21:42:03.000150+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.732523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"328.4","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004257+0000","last_change":"2024-03-11T21:42:00.689800+0000","last_active":"2024-03-11T21:42:03.004257+0000","last_peered":"2024-03-11T21:42:03.004257+0000","last_clean":"2024-03-11T21:42:03.004257+0000","last_became_active":"2024-03-11T21:42:00.689473+0000","last_became_peered":"2024-03-11T21:42:00.689473+0000","last_unstale":"2024-03-11T21:42:03.004257+0000","last_undegraded":"2024-03-11T21:42:03.004257+0000","last_fullsized":"2024-03-11T21:42:03.004257+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.674188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"328.3","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644793+0000","last_change":"2024-03-11T21:42:00.680203+0000","last_active":"2024-03-11T21:42:02.644793+0000","last_peered":"2024-03-11T21:42:02.644793+0000","last_clean":"2024-03-11T21:42:02.644793+0000","last_became_active":"2024-03-11T21:42:00.679631+0000","last_became_peered":"2024-03-11T21:42:00.679631+0000","last_unstale":"2024-03-11T21:42:02.644793+0000","last_undegraded":"2024-03-11T21:42:02.644793+0000","last_fullsized":"2024-03-11T21:42:02.644793+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:28.628629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"328.2","version":"0'0","reported_seq":1,"reported_epoch":817,"state":"creating+peering","last_fresh":"2024-03-11T21:41:59.642756+0000","last_change":"2024-03-11T21:41:59.642756+0000","last_active":"2024-03-11T21:41:59.631418+0000","last_peered":"2024-03-11T21:41:59.631418+0000","last_clean":"2024-03-11T21:41:59.631418+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:41:59.642756+0000","last_undegraded":"2024-03-11T21:41:59.642756+0000","last_fullsized":"2024-03-11T21:41:59.642756+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"328.1","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.652391+0000","last_change":"2024-03-11T21:42:00.669983+0000","last_active":"2024-03-11T21:42:02.652391+0000","last_peered":"2024-03-11T21:42:02.652391+0000","last_clean":"2024-03-11T21:42:02.652391+0000","last_became_active":"2024-03-11T21:42:00.669170+0000","last_became_peered":"2024-03-11T21:42:00.669170+0000","last_unstale":"2024-03-11T21:42:02.652391+0000","last_undegraded":"2024-03-11T21:42:02.652391+0000","last_fullsized":"2024-03-11T21:42:02.652391+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.982322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"328.0","version":"0'0","reported_seq":16,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.651468+0000","last_change":"2024-03-11T21:42:00.689502+0000","last_active":"2024-03-11T21:42:02.651468+0000","last_peered":"2024-03-11T21:42:02.651468+0000","last_clean":"2024-03-11T21:42:02.651468+0000","last_became_active":"2024-03-11T21:42:00.689058+0000","last_became_peered":"2024-03-11T21:42:00.689058+0000","last_unstale":"2024-03-11T21:42:02.651468+0000","last_undegraded":"2024-03-11T21:42:02.651468+0000","last_fullsized":"2024-03-11T21:42:02.651468+0000","mapping_epoch":817,"log_start":"0'0","ondisk_log_start":"0'0","created":817,"last_epoch_clean":818,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:59.631418+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.631418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.611658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.7","version":"713'5","reported_seq":366,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000986+0000","last_change":"2024-03-11T21:41:30.605842+0000","last_active":"2024-03-11T21:42:03.000986+0000","last_peered":"2024-03-11T21:42:03.000986+0000","last_clean":"2024-03-11T21:42:03.000986+0000","last_became_active":"2024-03-11T21:39:36.667532+0000","last_became_peered":"2024-03-11T21:39:36.667532+0000","last_unstale":"2024-03-11T21:42:03.000986+0000","last_undegraded":"2024-03-11T21:42:03.000986+0000","last_fullsized":"2024-03-11T21:42:03.000986+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'5","last_scrub_stamp":"2024-03-11T21:41:21.907642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.907642+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.610875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039920429999999998,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.6","version":"723'5","reported_seq":362,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005331+0000","last_change":"2024-03-11T21:41:54.496117+0000","last_active":"2024-03-11T21:42:03.005331+0000","last_peered":"2024-03-11T21:42:03.005331+0000","last_clean":"2024-03-11T21:42:03.005331+0000","last_became_active":"2024-03-11T21:39:36.664544+0000","last_became_peered":"2024-03-11T21:39:36.664544+0000","last_unstale":"2024-03-11T21:42:03.005331+0000","last_undegraded":"2024-03-11T21:42:03.005331+0000","last_fullsized":"2024-03-11T21:42:03.005331+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'5","last_scrub_stamp":"2024-03-11T21:41:54.495843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.495843+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:59.005140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012127220000000001,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.5","version":"757'3","reported_seq":366,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.659006+0000","last_change":"2024-03-11T21:41:30.632295+0000","last_active":"2024-03-11T21:42:02.659006+0000","last_peered":"2024-03-11T21:42:02.659006+0000","last_clean":"2024-03-11T21:42:02.659006+0000","last_became_active":"2024-03-11T21:39:36.664433+0000","last_became_peered":"2024-03-11T21:39:36.664433+0000","last_unstale":"2024-03-11T21:42:02.659006+0000","last_undegraded":"2024-03-11T21:42:02.659006+0000","last_fullsized":"2024-03-11T21:42:02.659006+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'3","last_scrub_stamp":"2024-03-11T21:41:14.644215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.644215+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.785317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018642859999999999,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.4","version":"749'15","reported_seq":370,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702961+0000","last_change":"2024-03-11T21:41:30.618704+0000","last_active":"2024-03-11T21:42:02.702961+0000","last_peered":"2024-03-11T21:42:02.702961+0000","last_clean":"2024-03-11T21:42:02.702961+0000","last_became_active":"2024-03-11T21:39:36.675026+0000","last_became_peered":"2024-03-11T21:39:36.675026+0000","last_unstale":"2024-03-11T21:42:02.702961+0000","last_undegraded":"2024-03-11T21:42:02.702961+0000","last_fullsized":"2024-03-11T21:42:02.702961+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:39:35.496133+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031897900000000001,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.3","version":"731'7","reported_seq":369,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998391+0000","last_change":"2024-03-11T21:41:30.606913+0000","last_active":"2024-03-11T21:42:02.998391+0000","last_peered":"2024-03-11T21:42:02.998391+0000","last_clean":"2024-03-11T21:42:02.998391+0000","last_became_active":"2024-03-11T21:39:36.653164+0000","last_became_peered":"2024-03-11T21:39:36.653164+0000","last_unstale":"2024-03-11T21:42:02.998391+0000","last_undegraded":"2024-03-11T21:42:02.998391+0000","last_fullsized":"2024-03-11T21:42:02.998391+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7","last_scrub_stamp":"2024-03-11T21:41:17.885818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.885818+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.833251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0050295210000000003,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.2","version":"805'7","reported_seq":368,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998596+0000","last_change":"2024-03-11T21:41:30.609679+0000","last_active":"2024-03-11T21:42:02.998596+0000","last_peered":"2024-03-11T21:42:02.998596+0000","last_clean":"2024-03-11T21:42:02.998596+0000","last_became_active":"2024-03-11T21:39:36.673314+0000","last_became_peered":"2024-03-11T21:39:36.673314+0000","last_unstale":"2024-03-11T21:42:02.998596+0000","last_undegraded":"2024-03-11T21:42:02.998596+0000","last_fullsized":"2024-03-11T21:42:02.998596+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:17.477919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.477919+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:30.411638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015195479999999999,"stat_sum":{"num_bytes":4000017,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.1","version":"797'22","reported_seq":381,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997917+0000","last_change":"2024-03-11T21:41:59.370755+0000","last_active":"2024-03-11T21:42:02.997917+0000","last_peered":"2024-03-11T21:42:02.997917+0000","last_clean":"2024-03-11T21:42:02.997917+0000","last_became_active":"2024-03-11T21:39:36.651751+0000","last_became_peered":"2024-03-11T21:39:36.651751+0000","last_unstale":"2024-03-11T21:42:02.997917+0000","last_undegraded":"2024-03-11T21:42:02.997917+0000","last_fullsized":"2024-03-11T21:42:02.997917+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'22","last_scrub_stamp":"2024-03-11T21:41:59.370508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.370508+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.457296+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.006595956,"stat_sum":{"num_bytes":15,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"313.0","version":"749'6","reported_seq":362,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002781+0000","last_change":"2024-03-11T21:41:30.603475+0000","last_active":"2024-03-11T21:42:03.002781+0000","last_peered":"2024-03-11T21:42:03.002781+0000","last_clean":"2024-03-11T21:42:03.002781+0000","last_became_active":"2024-03-11T21:39:36.661261+0000","last_became_peered":"2024-03-11T21:39:36.661261+0000","last_unstale":"2024-03-11T21:42:03.002781+0000","last_undegraded":"2024-03-11T21:42:03.002781+0000","last_fullsized":"2024-03-11T21:42:03.002781+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:16.318277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.318277+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:20.810940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019677119999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":1387,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.701814+0000","last_change":"2024-03-11T21:38:58.856206+0000","last_active":"2024-03-11T21:42:02.701814+0000","last_peered":"2024-03-11T21:42:02.701814+0000","last_clean":"2024-03-11T21:42:02.701814+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:42:02.701814+0000","last_undegraded":"2024-03-11T21:42:02.701814+0000","last_fullsized":"2024-03-11T21:42:02.701814+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:58.856090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:58.856090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1407,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997255+0000","last_change":"2024-03-11T21:41:29.853017+0000","last_active":"2024-03-11T21:42:02.997255+0000","last_peered":"2024-03-11T21:42:02.997255+0000","last_clean":"2024-03-11T21:42:02.997255+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:42:02.997255+0000","last_undegraded":"2024-03-11T21:42:02.997255+0000","last_fullsized":"2024-03-11T21:42:02.997255+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.852782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.852782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:55.425301+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1391,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000619+0000","last_change":"2024-03-11T21:41:57.330251+0000","last_active":"2024-03-11T21:42:03.000619+0000","last_peered":"2024-03-11T21:42:03.000619+0000","last_clean":"2024-03-11T21:42:03.000619+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:42:03.000619+0000","last_undegraded":"2024-03-11T21:42:03.000619+0000","last_fullsized":"2024-03-11T21:42:03.000619+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.330037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.330037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.606411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":1378,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702614+0000","last_change":"2024-03-11T21:40:25.470377+0000","last_active":"2024-03-11T21:42:02.702614+0000","last_peered":"2024-03-11T21:42:02.702614+0000","last_clean":"2024-03-11T21:42:02.702614+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:42:02.702614+0000","last_undegraded":"2024-03-11T21:42:02.702614+0000","last_fullsized":"2024-03-11T21:42:02.702614+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.470230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.470230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":1300,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.676950+0000","last_change":"2024-03-11T21:41:56.250851+0000","last_active":"2024-03-11T21:41:59.676950+0000","last_peered":"2024-03-11T21:41:59.676950+0000","last_clean":"2024-03-11T21:41:59.676950+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:41:59.676950+0000","last_undegraded":"2024-03-11T21:41:59.676950+0000","last_fullsized":"2024-03-11T21:41:59.676950+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.250744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.250744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:01.111142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1407,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000500+0000","last_change":"2024-03-11T21:40:28.770797+0000","last_active":"2024-03-11T21:42:03.000500+0000","last_peered":"2024-03-11T21:42:03.000500+0000","last_clean":"2024-03-11T21:42:03.000500+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:42:03.000500+0000","last_undegraded":"2024-03-11T21:42:03.000500+0000","last_fullsized":"2024-03-11T21:42:03.000500+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.770668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1408,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001254+0000","last_change":"2024-03-11T21:41:42.681742+0000","last_active":"2024-03-11T21:42:03.001254+0000","last_peered":"2024-03-11T21:42:03.001254+0000","last_clean":"2024-03-11T21:42:03.001254+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:42:03.001254+0000","last_undegraded":"2024-03-11T21:42:03.001254+0000","last_fullsized":"2024-03-11T21:42:03.001254+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.681620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.645321+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.681620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.544804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1421,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.661285+0000","last_change":"2024-03-11T21:41:35.844756+0000","last_active":"2024-03-11T21:42:02.661285+0000","last_peered":"2024-03-11T21:42:02.661285+0000","last_clean":"2024-03-11T21:42:02.661285+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:42:02.661285+0000","last_undegraded":"2024-03-11T21:42:02.661285+0000","last_fullsized":"2024-03-11T21:42:02.661285+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.844652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.844652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.557477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1406,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644175+0000","last_change":"2024-03-11T21:41:31.690755+0000","last_active":"2024-03-11T21:42:02.644175+0000","last_peered":"2024-03-11T21:42:02.644175+0000","last_clean":"2024-03-11T21:42:02.644175+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:42:02.644175+0000","last_undegraded":"2024-03-11T21:42:02.644175+0000","last_fullsized":"2024-03-11T21:42:02.644175+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.690559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.690559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.787294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1386,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.701914+0000","last_change":"2024-03-11T21:41:41.521503+0000","last_active":"2024-03-11T21:42:02.701914+0000","last_peered":"2024-03-11T21:42:02.701914+0000","last_clean":"2024-03-11T21:42:02.701914+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:42:02.701914+0000","last_undegraded":"2024-03-11T21:42:02.701914+0000","last_fullsized":"2024-03-11T21:42:02.701914+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.521300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.521300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.506707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1388,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003794+0000","last_change":"2024-03-11T21:40:13.532111+0000","last_active":"2024-03-11T21:42:03.003794+0000","last_peered":"2024-03-11T21:42:03.003794+0000","last_clean":"2024-03-11T21:42:03.003794+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:42:03.003794+0000","last_undegraded":"2024-03-11T21:42:03.003794+0000","last_fullsized":"2024-03-11T21:42:03.003794+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.532000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":1379,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003340+0000","last_change":"2024-03-11T21:41:59.422739+0000","last_active":"2024-03-11T21:42:03.003340+0000","last_peered":"2024-03-11T21:42:03.003340+0000","last_clean":"2024-03-11T21:42:03.003340+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:42:03.003340+0000","last_undegraded":"2024-03-11T21:42:03.003340+0000","last_fullsized":"2024-03-11T21:42:03.003340+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.422534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.422534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.121392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1388,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004471+0000","last_change":"2024-03-11T21:40:27.533904+0000","last_active":"2024-03-11T21:42:03.004471+0000","last_peered":"2024-03-11T21:42:03.004471+0000","last_clean":"2024-03-11T21:42:03.004471+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:42:03.004471+0000","last_undegraded":"2024-03-11T21:42:03.004471+0000","last_fullsized":"2024-03-11T21:42:03.004471+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.533778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.533778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1406,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646963+0000","last_change":"2024-03-11T21:41:45.595342+0000","last_active":"2024-03-11T21:42:02.646963+0000","last_peered":"2024-03-11T21:42:02.646963+0000","last_clean":"2024-03-11T21:42:02.646963+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:42:02.646963+0000","last_undegraded":"2024-03-11T21:42:02.646963+0000","last_fullsized":"2024-03-11T21:42:02.646963+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.595129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.595129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.837904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1420,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.659227+0000","last_change":"2024-03-11T21:41:39.829836+0000","last_active":"2024-03-11T21:42:02.659227+0000","last_peered":"2024-03-11T21:42:02.659227+0000","last_clean":"2024-03-11T21:42:02.659227+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:42:02.659227+0000","last_undegraded":"2024-03-11T21:42:02.659227+0000","last_fullsized":"2024-03-11T21:42:02.659227+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:39.829557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:39.829557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.135654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":796,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.681050+0000","last_change":"2024-03-11T21:41:20.464811+0000","last_active":"2024-03-11T21:41:59.681050+0000","last_peered":"2024-03-11T21:41:59.681050+0000","last_clean":"2024-03-11T21:41:59.681050+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:41:59.681050+0000","last_undegraded":"2024-03-11T21:41:59.681050+0000","last_fullsized":"2024-03-11T21:41:59.681050+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.464589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.464589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.019825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1408,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997031+0000","last_change":"2024-03-11T21:41:18.848489+0000","last_active":"2024-03-11T21:42:02.997031+0000","last_peered":"2024-03-11T21:42:02.997031+0000","last_clean":"2024-03-11T21:42:02.997031+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:42:02.997031+0000","last_undegraded":"2024-03-11T21:42:02.997031+0000","last_fullsized":"2024-03-11T21:42:02.997031+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.848394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.848394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:19.547180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":1386,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997423+0000","last_change":"2024-03-11T21:41:51.387114+0000","last_active":"2024-03-11T21:42:02.997423+0000","last_peered":"2024-03-11T21:42:02.997423+0000","last_clean":"2024-03-11T21:42:02.997423+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:42:02.997423+0000","last_undegraded":"2024-03-11T21:42:02.997423+0000","last_fullsized":"2024-03-11T21:42:02.997423+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.386905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.386905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.204956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":1379,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702004+0000","last_change":"2024-03-11T21:40:19.428421+0000","last_active":"2024-03-11T21:42:02.702004+0000","last_peered":"2024-03-11T21:42:02.702004+0000","last_clean":"2024-03-11T21:42:02.702004+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:42:02.702004+0000","last_undegraded":"2024-03-11T21:42:02.702004+0000","last_fullsized":"2024-03-11T21:42:02.702004+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.428205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.428205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1406,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645655+0000","last_change":"2024-03-11T21:41:40.664787+0000","last_active":"2024-03-11T21:42:02.645655+0000","last_peered":"2024-03-11T21:42:02.645655+0000","last_clean":"2024-03-11T21:42:02.645655+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:42:02.645655+0000","last_undegraded":"2024-03-11T21:42:02.645655+0000","last_fullsized":"2024-03-11T21:42:02.645655+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:40.664675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.664675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.324510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":1268,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.677863+0000","last_change":"2024-03-11T21:41:32.440218+0000","last_active":"2024-03-11T21:41:59.677863+0000","last_peered":"2024-03-11T21:41:59.677863+0000","last_clean":"2024-03-11T21:41:59.677863+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:41:59.677863+0000","last_undegraded":"2024-03-11T21:41:59.677863+0000","last_fullsized":"2024-03-11T21:41:59.677863+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.440004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.440004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.246313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":1378,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703537+0000","last_change":"2024-03-11T21:38:45.918826+0000","last_active":"2024-03-11T21:42:02.703537+0000","last_peered":"2024-03-11T21:42:02.703537+0000","last_clean":"2024-03-11T21:42:02.703537+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:42:02.703537+0000","last_undegraded":"2024-03-11T21:42:02.703537+0000","last_fullsized":"2024-03-11T21:42:02.703537+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:45.918744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:45.918744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1386,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703786+0000","last_change":"2024-03-11T21:38:59.900843+0000","last_active":"2024-03-11T21:42:02.703786+0000","last_peered":"2024-03-11T21:42:02.703786+0000","last_clean":"2024-03-11T21:42:02.703786+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:42:02.703786+0000","last_undegraded":"2024-03-11T21:42:02.703786+0000","last_fullsized":"2024-03-11T21:42:02.703786+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:59.900747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:59.900747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1388,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.006620+0000","last_change":"2024-03-11T21:40:26.557225+0000","last_active":"2024-03-11T21:42:03.006620+0000","last_peered":"2024-03-11T21:42:03.006620+0000","last_clean":"2024-03-11T21:42:03.006620+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:42:03.006620+0000","last_undegraded":"2024-03-11T21:42:03.006620+0000","last_fullsized":"2024-03-11T21:42:03.006620+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.557127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.557127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1384,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997737+0000","last_change":"2024-03-11T21:40:02.769849+0000","last_active":"2024-03-11T21:42:02.997737+0000","last_peered":"2024-03-11T21:42:02.997737+0000","last_clean":"2024-03-11T21:42:02.997737+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:42:02.997737+0000","last_undegraded":"2024-03-11T21:42:02.997737+0000","last_fullsized":"2024-03-11T21:42:02.997737+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.769628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.769628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":1372,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997547+0000","last_change":"2024-03-11T21:39:52.904395+0000","last_active":"2024-03-11T21:42:02.997547+0000","last_peered":"2024-03-11T21:42:02.997547+0000","last_clean":"2024-03-11T21:42:02.997547+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:42:02.997547+0000","last_undegraded":"2024-03-11T21:42:02.997547+0000","last_fullsized":"2024-03-11T21:42:02.997547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:52.904287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:39:52.904287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":796,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.676587+0000","last_change":"2024-03-11T21:40:21.384992+0000","last_active":"2024-03-11T21:41:59.676587+0000","last_peered":"2024-03-11T21:41:59.676587+0000","last_clean":"2024-03-11T21:41:59.676587+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:41:59.676587+0000","last_undegraded":"2024-03-11T21:41:59.676587+0000","last_fullsized":"2024-03-11T21:41:59.676587+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.384763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1407,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645550+0000","last_change":"2024-03-11T21:41:51.650830+0000","last_active":"2024-03-11T21:42:02.645550+0000","last_peered":"2024-03-11T21:42:02.645550+0000","last_clean":"2024-03-11T21:42:02.645550+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:42:02.645550+0000","last_undegraded":"2024-03-11T21:42:02.645550+0000","last_fullsized":"2024-03-11T21:42:02.645550+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.650722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.650722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.155813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1389,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004575+0000","last_change":"2024-03-11T21:41:17.359280+0000","last_active":"2024-03-11T21:42:03.004575+0000","last_peered":"2024-03-11T21:42:03.004575+0000","last_clean":"2024-03-11T21:42:03.004575+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:42:03.004575+0000","last_undegraded":"2024-03-11T21:42:03.004575+0000","last_fullsized":"2024-03-11T21:42:03.004575+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.628243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":1372,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999946+0000","last_change":"2024-03-11T21:38:46.791275+0000","last_active":"2024-03-11T21:42:02.999946+0000","last_peered":"2024-03-11T21:42:02.999946+0000","last_clean":"2024-03-11T21:42:02.999946+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:42:02.999946+0000","last_undegraded":"2024-03-11T21:42:02.999946+0000","last_fullsized":"2024-03-11T21:42:02.999946+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:46.791177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:38:46.791177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1421,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.659583+0000","last_change":"2024-03-11T21:41:54.862852+0000","last_active":"2024-03-11T21:42:02.659583+0000","last_peered":"2024-03-11T21:42:02.659583+0000","last_clean":"2024-03-11T21:42:02.659583+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:42:02.659583+0000","last_undegraded":"2024-03-11T21:42:02.659583+0000","last_fullsized":"2024-03-11T21:42:02.659583+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.862666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.862666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.186701+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":1268,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.681653+0000","last_change":"2024-03-11T21:41:30.411743+0000","last_active":"2024-03-11T21:41:59.681653+0000","last_peered":"2024-03-11T21:41:59.681653+0000","last_clean":"2024-03-11T21:41:59.681653+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:41:59.681653+0000","last_undegraded":"2024-03-11T21:41:59.681653+0000","last_fullsized":"2024-03-11T21:41:59.681653+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:30.411631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.411631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:30.767399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1395,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005572+0000","last_change":"2024-03-11T21:40:09.512570+0000","last_active":"2024-03-11T21:42:03.005572+0000","last_peered":"2024-03-11T21:42:03.005572+0000","last_clean":"2024-03-11T21:42:03.005572+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:42:03.005572+0000","last_undegraded":"2024-03-11T21:42:03.005572+0000","last_fullsized":"2024-03-11T21:42:03.005572+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.512296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.512296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1769,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998284+0000","last_change":"2024-03-11T21:41:21.484873+0000","last_active":"2024-03-11T21:42:02.998284+0000","last_peered":"2024-03-11T21:42:02.998284+0000","last_clean":"2024-03-11T21:42:02.998284+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:42:02.998284+0000","last_undegraded":"2024-03-11T21:42:02.998284+0000","last_fullsized":"2024-03-11T21:42:02.998284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:21.484757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.484757+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.841600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1578,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002953+0000","last_change":"2024-03-11T21:41:26.485912+0000","last_active":"2024-03-11T21:42:03.002953+0000","last_peered":"2024-03-11T21:42:03.002953+0000","last_clean":"2024-03-11T21:42:03.002953+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:42:03.002953+0000","last_undegraded":"2024-03-11T21:42:03.002953+0000","last_fullsized":"2024-03-11T21:42:03.002953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:41:26.485646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.485646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.594156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1588,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.660243+0000","last_change":"2024-03-11T21:41:42.811010+0000","last_active":"2024-03-11T21:42:02.660243+0000","last_peered":"2024-03-11T21:42:02.660243+0000","last_clean":"2024-03-11T21:42:02.660243+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:42:02.660243+0000","last_undegraded":"2024-03-11T21:42:02.660243+0000","last_fullsized":"2024-03-11T21:42:02.660243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.810777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.810777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.272393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1540,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000867+0000","last_change":"2024-03-11T21:41:45.252015+0000","last_active":"2024-03-11T21:42:03.000867+0000","last_peered":"2024-03-11T21:42:03.000867+0000","last_clean":"2024-03-11T21:42:03.000867+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:42:03.000867+0000","last_undegraded":"2024-03-11T21:42:03.000867+0000","last_fullsized":"2024-03-11T21:42:03.000867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.251730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.251730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.958525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1399,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002671+0000","last_change":"2024-03-11T21:40:14.758296+0000","last_active":"2024-03-11T21:42:03.002671+0000","last_peered":"2024-03-11T21:42:03.002671+0000","last_clean":"2024-03-11T21:42:03.002671+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:42:03.002671+0000","last_undegraded":"2024-03-11T21:42:03.002671+0000","last_fullsized":"2024-03-11T21:42:03.002671+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:14.758086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.758086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1578,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001915+0000","last_change":"2024-03-11T21:41:26.303821+0000","last_active":"2024-03-11T21:42:03.001915+0000","last_peered":"2024-03-11T21:42:03.001915+0000","last_clean":"2024-03-11T21:42:03.001915+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:42:03.001915+0000","last_undegraded":"2024-03-11T21:42:03.001915+0000","last_fullsized":"2024-03-11T21:42:03.001915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:26.303567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.303567+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.068658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1450,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.678723+0000","last_change":"2024-03-11T21:40:24.347139+0000","last_active":"2024-03-11T21:41:59.678723+0000","last_peered":"2024-03-11T21:41:59.678723+0000","last_clean":"2024-03-11T21:41:59.678723+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:41:59.678723+0000","last_undegraded":"2024-03-11T21:41:59.678723+0000","last_fullsized":"2024-03-11T21:41:59.678723+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.347032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.347032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1580,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.654980+0000","last_change":"2024-03-11T21:41:27.719707+0000","last_active":"2024-03-11T21:42:02.654980+0000","last_peered":"2024-03-11T21:42:02.654980+0000","last_clean":"2024-03-11T21:42:02.654980+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:42:02.654980+0000","last_undegraded":"2024-03-11T21:42:02.654980+0000","last_fullsized":"2024-03-11T21:42:02.654980+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.719614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.719614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.914311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1545,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004032+0000","last_change":"2024-03-11T21:41:52.458683+0000","last_active":"2024-03-11T21:42:03.004032+0000","last_peered":"2024-03-11T21:42:03.004032+0000","last_clean":"2024-03-11T21:42:03.004032+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:42:03.004032+0000","last_undegraded":"2024-03-11T21:42:03.004032+0000","last_fullsized":"2024-03-11T21:42:03.004032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:52.458535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:41:52.458535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.765250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1398,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003528+0000","last_change":"2024-03-11T21:41:55.275905+0000","last_active":"2024-03-11T21:42:03.003528+0000","last_peered":"2024-03-11T21:42:03.003528+0000","last_clean":"2024-03-11T21:42:03.003528+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:42:03.003528+0000","last_undegraded":"2024-03-11T21:42:03.003528+0000","last_fullsized":"2024-03-11T21:42:03.003528+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.275680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.275680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.360123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1592,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002931+0000","last_change":"2024-03-11T21:38:59.769177+0000","last_active":"2024-03-11T21:42:03.002931+0000","last_peered":"2024-03-11T21:42:03.002931+0000","last_clean":"2024-03-11T21:42:03.002931+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:42:03.002931+0000","last_undegraded":"2024-03-11T21:42:03.002931+0000","last_fullsized":"2024-03-11T21:42:03.002931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_deep_scrub":"389'39","last_deep_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_clean_scrub_stamp":"2024-03-11T21:38:59.768919+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1588,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.658710+0000","last_change":"2024-03-11T21:41:37.871821+0000","last_active":"2024-03-11T21:42:02.658710+0000","last_peered":"2024-03-11T21:42:02.658710+0000","last_clean":"2024-03-11T21:42:02.658710+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:42:02.658710+0000","last_undegraded":"2024-03-11T21:42:02.658710+0000","last_fullsized":"2024-03-11T21:42:02.658710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.871740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.871740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.172436+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1580,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.657642+0000","last_change":"2024-03-11T21:41:51.789113+0000","last_active":"2024-03-11T21:42:02.657642+0000","last_peered":"2024-03-11T21:42:02.657642+0000","last_clean":"2024-03-11T21:42:02.657642+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:42:02.657642+0000","last_undegraded":"2024-03-11T21:42:02.657642+0000","last_fullsized":"2024-03-11T21:42:02.657642+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.789014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.789014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.998815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1527,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997150+0000","last_change":"2024-03-11T21:39:59.804040+0000","last_active":"2024-03-11T21:42:02.997150+0000","last_peered":"2024-03-11T21:42:02.997150+0000","last_clean":"2024-03-11T21:42:02.997150+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:42:02.997150+0000","last_undegraded":"2024-03-11T21:42:02.997150+0000","last_fullsized":"2024-03-11T21:42:02.997150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:59.803820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.803820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":776,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997470+0000","last_change":"2024-03-11T21:41:22.197315+0000","last_active":"2024-03-11T21:42:02.997470+0000","last_peered":"2024-03-11T21:42:02.997470+0000","last_clean":"2024-03-11T21:42:02.997470+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:42:02.997470+0000","last_undegraded":"2024-03-11T21:42:02.997470+0000","last_fullsized":"2024-03-11T21:42:02.997470+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:22.197156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.197156+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.113967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1270,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.671896+0000","last_change":"2024-03-11T21:40:05.414458+0000","last_active":"2024-03-11T21:41:59.671896+0000","last_peered":"2024-03-11T21:41:59.671896+0000","last_clean":"2024-03-11T21:41:59.671896+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:41:59.671896+0000","last_undegraded":"2024-03-11T21:41:59.671896+0000","last_fullsized":"2024-03-11T21:41:59.671896+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:05.414369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.414369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1644,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998460+0000","last_change":"2024-03-11T21:41:32.795704+0000","last_active":"2024-03-11T21:42:02.998460+0000","last_peered":"2024-03-11T21:42:02.998460+0000","last_clean":"2024-03-11T21:42:02.998460+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:42:02.998460+0000","last_undegraded":"2024-03-11T21:42:02.998460+0000","last_fullsized":"2024-03-11T21:42:02.998460+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:41:32.795430+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.795430+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.294775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1547,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002643+0000","last_change":"2024-03-11T21:41:29.509226+0000","last_active":"2024-03-11T21:42:03.002643+0000","last_peered":"2024-03-11T21:42:03.002643+0000","last_clean":"2024-03-11T21:42:03.002643+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:42:03.002643+0000","last_undegraded":"2024-03-11T21:42:03.002643+0000","last_fullsized":"2024-03-11T21:42:03.002643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.509105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.509105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:43.027891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1456,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:42:00.292106+0000","last_change":"2024-03-11T21:42:00.292135+0000","last_active":"2024-03-11T21:42:00.292106+0000","last_peered":"2024-03-11T21:42:00.292106+0000","last_clean":"2024-03-11T21:42:00.292106+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:42:00.292106+0000","last_undegraded":"2024-03-11T21:42:00.292106+0000","last_fullsized":"2024-03-11T21:42:00.292106+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.292001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.532304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.661735+0000","last_change":"2024-03-11T21:41:20.636042+0000","last_active":"2024-03-11T21:42:02.661735+0000","last_peered":"2024-03-11T21:42:02.661735+0000","last_clean":"2024-03-11T21:42:02.661735+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:42:02.661735+0000","last_undegraded":"2024-03-11T21:42:02.661735+0000","last_fullsized":"2024-03-11T21:42:02.661735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.635800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.635800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.846076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1828,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998705+0000","last_change":"2024-03-11T21:41:30.525150+0000","last_active":"2024-03-11T21:42:02.998705+0000","last_peered":"2024-03-11T21:42:02.998705+0000","last_clean":"2024-03-11T21:42:02.998705+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:42:02.998705+0000","last_undegraded":"2024-03-11T21:42:02.998705+0000","last_fullsized":"2024-03-11T21:42:02.998705+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:30.525009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.525009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.617899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1402,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702155+0000","last_change":"2024-03-11T21:40:08.203021+0000","last_active":"2024-03-11T21:42:02.702155+0000","last_peered":"2024-03-11T21:42:02.702155+0000","last_clean":"2024-03-11T21:42:02.702155+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:42:02.702155+0000","last_undegraded":"2024-03-11T21:42:02.702155+0000","last_fullsized":"2024-03-11T21:42:02.702155+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:40:08.202874+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:40:08.202874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1570,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703367+0000","last_change":"2024-03-11T21:38:42.809925+0000","last_active":"2024-03-11T21:42:02.703367+0000","last_peered":"2024-03-11T21:42:02.703367+0000","last_clean":"2024-03-11T21:42:02.703367+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:42:02.703367+0000","last_undegraded":"2024-03-11T21:42:02.703367+0000","last_fullsized":"2024-03-11T21:42:02.703367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:38:42.809678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:42.809678+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1416,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.676545+0000","last_change":"2024-03-11T21:40:09.399602+0000","last_active":"2024-03-11T21:41:59.676545+0000","last_peered":"2024-03-11T21:41:59.676545+0000","last_clean":"2024-03-11T21:41:59.676545+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:41:59.676545+0000","last_undegraded":"2024-03-11T21:41:59.676545+0000","last_fullsized":"2024-03-11T21:41:59.676545+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.399381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.399381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1819,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.659715+0000","last_change":"2024-03-11T21:41:26.709378+0000","last_active":"2024-03-11T21:42:02.659715+0000","last_peered":"2024-03-11T21:42:02.659715+0000","last_clean":"2024-03-11T21:42:02.659715+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:42:02.659715+0000","last_undegraded":"2024-03-11T21:42:02.659715+0000","last_fullsized":"2024-03-11T21:42:02.659715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:41:26.709267+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.709267+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.444646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1571,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646982+0000","last_change":"2024-03-11T21:41:28.709111+0000","last_active":"2024-03-11T21:42:02.646982+0000","last_peered":"2024-03-11T21:42:02.646982+0000","last_clean":"2024-03-11T21:42:02.646982+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:42:02.646982+0000","last_undegraded":"2024-03-11T21:42:02.646982+0000","last_fullsized":"2024-03-11T21:42:02.646982+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.707202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.707202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.875323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":776,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000407+0000","last_change":"2024-03-11T21:41:14.202929+0000","last_active":"2024-03-11T21:42:03.000407+0000","last_peered":"2024-03-11T21:42:03.000407+0000","last_clean":"2024-03-11T21:42:03.000407+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:42:03.000407+0000","last_undegraded":"2024-03-11T21:42:03.000407+0000","last_fullsized":"2024-03-11T21:42:03.000407+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:14.202622+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:37:53.724768+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.202622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.086810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1405,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702853+0000","last_change":"2024-03-11T21:41:25.364494+0000","last_active":"2024-03-11T21:42:02.702853+0000","last_peered":"2024-03-11T21:42:02.702853+0000","last_clean":"2024-03-11T21:42:02.702853+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:42:02.702853+0000","last_undegraded":"2024-03-11T21:42:02.702853+0000","last_fullsized":"2024-03-11T21:42:02.702853+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.364387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.364387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.701511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1597,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702209+0000","last_change":"2024-03-11T21:40:06.159993+0000","last_active":"2024-03-11T21:42:02.702209+0000","last_peered":"2024-03-11T21:42:02.702209+0000","last_clean":"2024-03-11T21:42:02.702209+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:42:02.702209+0000","last_undegraded":"2024-03-11T21:42:02.702209+0000","last_fullsized":"2024-03-11T21:42:02.702209+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:40:06.159823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.159823+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1444,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.675276+0000","last_change":"2024-03-11T21:40:08.421673+0000","last_active":"2024-03-11T21:41:59.675276+0000","last_peered":"2024-03-11T21:41:59.675276+0000","last_clean":"2024-03-11T21:41:59.675276+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:41:59.675276+0000","last_undegraded":"2024-03-11T21:41:59.675276+0000","last_fullsized":"2024-03-11T21:41:59.675276+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_clean_scrub_stamp":"2024-03-11T21:40:08.421516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1804,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646653+0000","last_change":"2024-03-11T21:41:46.623952+0000","last_active":"2024-03-11T21:42:02.646653+0000","last_peered":"2024-03-11T21:42:02.646653+0000","last_clean":"2024-03-11T21:42:02.646653+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:42:02.646653+0000","last_undegraded":"2024-03-11T21:42:02.646653+0000","last_fullsized":"2024-03-11T21:42:02.646653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:46.623794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.623794+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.014629+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1557,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001979+0000","last_change":"2024-03-11T21:41:58.366463+0000","last_active":"2024-03-11T21:42:03.001979+0000","last_peered":"2024-03-11T21:42:03.001979+0000","last_clean":"2024-03-11T21:42:03.001979+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:42:03.001979+0000","last_undegraded":"2024-03-11T21:42:03.001979+0000","last_fullsized":"2024-03-11T21:42:03.001979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.366196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.366196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.319086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1386,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999757+0000","last_change":"2024-03-11T21:38:41.784165+0000","last_active":"2024-03-11T21:42:02.999757+0000","last_peered":"2024-03-11T21:42:02.999757+0000","last_clean":"2024-03-11T21:42:02.999757+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:42:02.999757+0000","last_undegraded":"2024-03-11T21:42:02.999757+0000","last_fullsized":"2024-03-11T21:42:02.999757+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:41.783976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:41.783976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1599,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644946+0000","last_change":"2024-03-11T21:40:15.854712+0000","last_active":"2024-03-11T21:42:02.644946+0000","last_peered":"2024-03-11T21:42:02.644946+0000","last_clean":"2024-03-11T21:42:02.644946+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:42:02.644946+0000","last_undegraded":"2024-03-11T21:42:02.644946+0000","last_fullsized":"2024-03-11T21:42:02.644946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:40:15.854104+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.854104+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1556,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997484+0000","last_change":"2024-03-11T21:41:18.465526+0000","last_active":"2024-03-11T21:42:02.997484+0000","last_peered":"2024-03-11T21:42:02.997484+0000","last_clean":"2024-03-11T21:42:02.997484+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:42:02.997484+0000","last_undegraded":"2024-03-11T21:42:02.997484+0000","last_fullsized":"2024-03-11T21:42:02.997484+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.465412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.465412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.489077+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1544,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003910+0000","last_change":"2024-03-11T21:41:29.273257+0000","last_active":"2024-03-11T21:42:03.003910+0000","last_peered":"2024-03-11T21:42:03.003910+0000","last_clean":"2024-03-11T21:42:03.003910+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:42:03.003910+0000","last_undegraded":"2024-03-11T21:42:03.003910+0000","last_fullsized":"2024-03-11T21:42:03.003910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.273150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.273150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.658499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1537,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703584+0000","last_change":"2024-03-11T21:41:26.327523+0000","last_active":"2024-03-11T21:42:02.703584+0000","last_peered":"2024-03-11T21:42:02.703584+0000","last_clean":"2024-03-11T21:42:02.703584+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:42:02.703584+0000","last_undegraded":"2024-03-11T21:42:02.703584+0000","last_fullsized":"2024-03-11T21:42:02.703584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.327416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.327416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.740808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":780,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703858+0000","last_change":"2024-03-11T21:41:57.696381+0000","last_active":"2024-03-11T21:42:02.703858+0000","last_peered":"2024-03-11T21:42:02.703858+0000","last_clean":"2024-03-11T21:42:02.703858+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:42:02.703858+0000","last_undegraded":"2024-03-11T21:42:02.703858+0000","last_fullsized":"2024-03-11T21:42:02.703858+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:41:57.696205+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.696205+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:57.700510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1387,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.701596+0000","last_change":"2024-03-11T21:40:03.109654+0000","last_active":"2024-03-11T21:42:02.701596+0000","last_peered":"2024-03-11T21:42:02.701596+0000","last_clean":"2024-03-11T21:42:02.701596+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:42:02.701596+0000","last_undegraded":"2024-03-11T21:42:02.701596+0000","last_fullsized":"2024-03-11T21:42:02.701596+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.109518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.109518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1790,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702459+0000","last_change":"2024-03-11T21:39:59.035379+0000","last_active":"2024-03-11T21:42:02.702459+0000","last_peered":"2024-03-11T21:42:02.702459+0000","last_clean":"2024-03-11T21:42:02.702459+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:42:02.702459+0000","last_undegraded":"2024-03-11T21:42:02.702459+0000","last_fullsized":"2024-03-11T21:42:02.702459+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:39:59.035148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.035148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.647234+0000","last_change":"2024-03-11T21:40:17.919859+0000","last_active":"2024-03-11T21:42:02.647234+0000","last_peered":"2024-03-11T21:42:02.647234+0000","last_clean":"2024-03-11T21:42:02.647234+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:42:02.647234+0000","last_undegraded":"2024-03-11T21:42:02.647234+0000","last_fullsized":"2024-03-11T21:42:02.647234+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:17.919574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:40:17.919574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1598,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005885+0000","last_change":"2024-03-11T21:41:43.459254+0000","last_active":"2024-03-11T21:42:03.005885+0000","last_peered":"2024-03-11T21:42:03.005885+0000","last_clean":"2024-03-11T21:42:03.005885+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:42:03.005885+0000","last_undegraded":"2024-03-11T21:42:03.005885+0000","last_fullsized":"2024-03-11T21:42:03.005885+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:41:43.458986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.458986+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.423341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1592,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001484+0000","last_change":"2024-03-11T21:41:55.649415+0000","last_active":"2024-03-11T21:42:03.001484+0000","last_peered":"2024-03-11T21:42:03.001484+0000","last_clean":"2024-03-11T21:42:03.001484+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:42:03.001484+0000","last_undegraded":"2024-03-11T21:42:03.001484+0000","last_fullsized":"2024-03-11T21:42:03.001484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:41:55.649085+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.649085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.612818+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1538,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999915+0000","last_change":"2024-03-11T21:40:23.749973+0000","last_active":"2024-03-11T21:42:02.999915+0000","last_peered":"2024-03-11T21:42:02.999915+0000","last_clean":"2024-03-11T21:42:02.999915+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:42:02.999915+0000","last_undegraded":"2024-03-11T21:42:02.999915+0000","last_fullsized":"2024-03-11T21:42:02.999915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:40:23.749772+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.749772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":791,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645729+0000","last_change":"2024-03-11T21:41:56.716081+0000","last_active":"2024-03-11T21:42:02.645729+0000","last_peered":"2024-03-11T21:42:02.645729+0000","last_clean":"2024-03-11T21:42:02.645729+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:42:02.645729+0000","last_undegraded":"2024-03-11T21:42:02.645729+0000","last_fullsized":"2024-03-11T21:42:02.645729+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:56.715868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.715868+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:57.439090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1393,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000215+0000","last_change":"2024-03-11T21:41:42.428955+0000","last_active":"2024-03-11T21:42:03.000215+0000","last_peered":"2024-03-11T21:42:03.000215+0000","last_clean":"2024-03-11T21:42:03.000215+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:42:03.000215+0000","last_undegraded":"2024-03-11T21:42:03.000215+0000","last_fullsized":"2024-03-11T21:42:03.000215+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.428709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.428709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.809420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1575,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.644514+0000","last_change":"2024-03-11T21:41:42.630400+0000","last_active":"2024-03-11T21:42:02.644514+0000","last_peered":"2024-03-11T21:42:02.644514+0000","last_clean":"2024-03-11T21:42:02.644514+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:42:02.644514+0000","last_undegraded":"2024-03-11T21:42:02.644514+0000","last_fullsized":"2024-03-11T21:42:02.644514+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.630272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.630272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:43.927942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2672,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702199+0000","last_change":"2024-03-11T21:41:20.351387+0000","last_active":"2024-03-11T21:42:02.702199+0000","last_peered":"2024-03-11T21:42:02.702199+0000","last_clean":"2024-03-11T21:42:02.702199+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:42:02.702199+0000","last_undegraded":"2024-03-11T21:42:02.702199+0000","last_fullsized":"2024-03-11T21:42:02.702199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:41:20.351150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.351150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.077803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1118,"num_read_kb":1113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1570,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.705108+0000","last_change":"2024-03-11T21:38:30.798109+0000","last_active":"2024-03-11T21:42:02.705108+0000","last_peered":"2024-03-11T21:42:02.705108+0000","last_clean":"2024-03-11T21:42:02.705108+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:42:02.705108+0000","last_undegraded":"2024-03-11T21:42:02.705108+0000","last_fullsized":"2024-03-11T21:42:02.705108+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:38:30.797863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:30.797863+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1469,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.680793+0000","last_change":"2024-03-11T21:40:06.437710+0000","last_active":"2024-03-11T21:41:59.680793+0000","last_peered":"2024-03-11T21:41:59.680793+0000","last_clean":"2024-03-11T21:41:59.680793+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:41:59.680793+0000","last_undegraded":"2024-03-11T21:41:59.680793+0000","last_fullsized":"2024-03-11T21:41:59.680793+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:40:06.437601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.437601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1563,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646462+0000","last_change":"2024-03-11T21:41:20.553933+0000","last_active":"2024-03-11T21:42:02.646462+0000","last_peered":"2024-03-11T21:42:02.646462+0000","last_clean":"2024-03-11T21:42:02.646462+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:42:02.646462+0000","last_undegraded":"2024-03-11T21:42:02.646462+0000","last_fullsized":"2024-03-11T21:42:02.646462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.553712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.553712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.127880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":793,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000922+0000","last_change":"2024-03-11T21:41:27.847353+0000","last_active":"2024-03-11T21:42:03.000922+0000","last_peered":"2024-03-11T21:42:03.000922+0000","last_clean":"2024-03-11T21:42:03.000922+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:42:03.000922+0000","last_undegraded":"2024-03-11T21:42:03.000922+0000","last_fullsized":"2024-03-11T21:42:03.000922+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:27.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.847107+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.711792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1393,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004934+0000","last_change":"2024-03-11T21:41:35.427640+0000","last_active":"2024-03-11T21:42:03.004934+0000","last_peered":"2024-03-11T21:42:03.004934+0000","last_clean":"2024-03-11T21:42:03.004934+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:42:03.004934+0000","last_undegraded":"2024-03-11T21:42:03.004934+0000","last_fullsized":"2024-03-11T21:42:03.004934+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.427438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.427438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.404944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1560,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998346+0000","last_change":"2024-03-11T21:41:23.914266+0000","last_active":"2024-03-11T21:42:02.998346+0000","last_peered":"2024-03-11T21:42:02.998346+0000","last_clean":"2024-03-11T21:42:02.998346+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:42:02.998346+0000","last_undegraded":"2024-03-11T21:42:02.998346+0000","last_fullsized":"2024-03-11T21:42:02.998346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.913967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.913967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.777062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1734,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.680031+0000","last_change":"2024-03-11T21:41:19.438289+0000","last_active":"2024-03-11T21:41:59.680031+0000","last_peered":"2024-03-11T21:41:59.680031+0000","last_clean":"2024-03-11T21:41:59.680031+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:41:59.680031+0000","last_undegraded":"2024-03-11T21:41:59.680031+0000","last_fullsized":"2024-03-11T21:41:59.680031+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:41:19.438160+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.438160+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.848834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1548,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002923+0000","last_change":"2024-03-11T21:41:44.440095+0000","last_active":"2024-03-11T21:42:03.002923+0000","last_peered":"2024-03-11T21:42:03.002923+0000","last_clean":"2024-03-11T21:42:03.002923+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:42:03.002923+0000","last_undegraded":"2024-03-11T21:42:03.002923+0000","last_fullsized":"2024-03-11T21:42:03.002923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:44.439846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:44.439846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.010761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1409,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.674728+0000","last_change":"2024-03-11T21:40:17.365129+0000","last_active":"2024-03-11T21:41:59.674728+0000","last_peered":"2024-03-11T21:41:59.674728+0000","last_clean":"2024-03-11T21:41:59.674728+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:41:59.674728+0000","last_undegraded":"2024-03-11T21:41:59.674728+0000","last_fullsized":"2024-03-11T21:41:59.674728+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:40:17.364997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":778,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.674888+0000","last_change":"2024-03-11T21:40:27.432675+0000","last_active":"2024-03-11T21:41:59.674888+0000","last_peered":"2024-03-11T21:41:59.674888+0000","last_clean":"2024-03-11T21:41:59.674888+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:41:59.674888+0000","last_undegraded":"2024-03-11T21:41:59.674888+0000","last_fullsized":"2024-03-11T21:41:59.674888+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:40:27.432564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.432564+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1394,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005321+0000","last_change":"2024-03-11T21:40:18.629853+0000","last_active":"2024-03-11T21:42:03.005321+0000","last_peered":"2024-03-11T21:42:03.005321+0000","last_clean":"2024-03-11T21:42:03.005321+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:42:03.005321+0000","last_undegraded":"2024-03-11T21:42:03.005321+0000","last_fullsized":"2024-03-11T21:42:03.005321+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.629736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.629736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1553,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003844+0000","last_change":"2024-03-11T21:41:17.193786+0000","last_active":"2024-03-11T21:42:03.003844+0000","last_peered":"2024-03-11T21:42:03.003844+0000","last_clean":"2024-03-11T21:42:03.003844+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:42:03.003844+0000","last_undegraded":"2024-03-11T21:42:03.003844+0000","last_fullsized":"2024-03-11T21:42:03.003844+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.193618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.193618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.977433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1549,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997272+0000","last_change":"2024-03-11T21:38:48.793542+0000","last_active":"2024-03-11T21:42:02.997272+0000","last_peered":"2024-03-11T21:42:02.997272+0000","last_clean":"2024-03-11T21:42:02.997272+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:42:02.997272+0000","last_undegraded":"2024-03-11T21:42:02.997272+0000","last_fullsized":"2024-03-11T21:42:02.997272+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:48.793401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:38:48.793401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1596,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003674+0000","last_change":"2024-03-11T21:42:00.572712+0000","last_active":"2024-03-11T21:42:03.003674+0000","last_peered":"2024-03-11T21:42:03.003674+0000","last_clean":"2024-03-11T21:42:03.003674+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:42:03.003674+0000","last_undegraded":"2024-03-11T21:42:03.003674+0000","last_fullsized":"2024-03-11T21:42:03.003674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:00.572462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.572462+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.915234+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1419,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.678854+0000","last_change":"2024-03-11T21:41:33.473130+0000","last_active":"2024-03-11T21:41:59.678854+0000","last_peered":"2024-03-11T21:41:59.678854+0000","last_clean":"2024-03-11T21:41:59.678854+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:41:59.678854+0000","last_undegraded":"2024-03-11T21:41:59.678854+0000","last_fullsized":"2024-03-11T21:41:59.678854+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.472920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.472920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.414677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1396,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.701872+0000","last_change":"2024-03-11T21:42:01.660218+0000","last_active":"2024-03-11T21:42:02.701872+0000","last_peered":"2024-03-11T21:42:02.701872+0000","last_clean":"2024-03-11T21:42:02.701872+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:42:02.701872+0000","last_undegraded":"2024-03-11T21:42:02.701872+0000","last_fullsized":"2024-03-11T21:42:02.701872+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.660109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.660109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:31.505443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1532,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999692+0000","last_change":"2024-03-11T21:40:22.764357+0000","last_active":"2024-03-11T21:42:02.999692+0000","last_peered":"2024-03-11T21:42:02.999692+0000","last_clean":"2024-03-11T21:42:02.999692+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:42:02.999692+0000","last_undegraded":"2024-03-11T21:42:02.999692+0000","last_fullsized":"2024-03-11T21:42:02.999692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:22.764240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:22.764240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1821,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997371+0000","last_change":"2024-03-11T21:41:41.451026+0000","last_active":"2024-03-11T21:42:02.997371+0000","last_peered":"2024-03-11T21:42:02.997371+0000","last_clean":"2024-03-11T21:42:02.997371+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:42:02.997371+0000","last_undegraded":"2024-03-11T21:42:02.997371+0000","last_fullsized":"2024-03-11T21:42:02.997371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:41.450896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.450896+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.423644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1553,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001288+0000","last_change":"2024-03-11T21:40:10.827418+0000","last_active":"2024-03-11T21:42:03.001288+0000","last_peered":"2024-03-11T21:42:03.001288+0000","last_clean":"2024-03-11T21:42:03.001288+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:42:03.001288+0000","last_undegraded":"2024-03-11T21:42:03.001288+0000","last_fullsized":"2024-03-11T21:42:03.001288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:10.827185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:40:10.827185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702591+0000","last_change":"2024-03-11T21:41:35.415804+0000","last_active":"2024-03-11T21:42:02.702591+0000","last_peered":"2024-03-11T21:42:02.702591+0000","last_clean":"2024-03-11T21:42:02.702591+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:42:02.702591+0000","last_undegraded":"2024-03-11T21:42:02.702591+0000","last_fullsized":"2024-03-11T21:42:02.702591+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:35.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.415630+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.778929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":349,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.006613+0000","last_change":"2024-03-11T21:41:37.415103+0000","last_active":"2024-03-11T21:42:03.006613+0000","last_peered":"2024-03-11T21:42:03.006613+0000","last_clean":"2024-03-11T21:42:03.006613+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:42:03.006613+0000","last_undegraded":"2024-03-11T21:42:03.006613+0000","last_fullsized":"2024-03-11T21:42:03.006613+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.414919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.414919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.220108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1829,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999787+0000","last_change":"2024-03-11T21:41:38.433752+0000","last_active":"2024-03-11T21:42:02.999787+0000","last_peered":"2024-03-11T21:42:02.999787+0000","last_clean":"2024-03-11T21:42:02.999787+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:42:02.999787+0000","last_undegraded":"2024-03-11T21:42:02.999787+0000","last_fullsized":"2024-03-11T21:42:02.999787+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:38.433381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:38.433381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.908595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1399,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000724+0000","last_change":"2024-03-11T21:40:19.766314+0000","last_active":"2024-03-11T21:42:03.000724+0000","last_peered":"2024-03-11T21:42:03.000724+0000","last_clean":"2024-03-11T21:42:03.000724+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:42:03.000724+0000","last_undegraded":"2024-03-11T21:42:03.000724+0000","last_fullsized":"2024-03-11T21:42:03.000724+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.766007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.766007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1580,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.660667+0000","last_change":"2024-03-11T21:41:15.651241+0000","last_active":"2024-03-11T21:42:02.660667+0000","last_peered":"2024-03-11T21:42:02.660667+0000","last_clean":"2024-03-11T21:42:02.660667+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:42:02.660667+0000","last_undegraded":"2024-03-11T21:42:02.660667+0000","last_fullsized":"2024-03-11T21:42:02.660667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.651121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.651121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.721840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1589,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.653583+0000","last_change":"2024-03-11T21:41:41.822816+0000","last_active":"2024-03-11T21:42:02.653583+0000","last_peered":"2024-03-11T21:42:02.653583+0000","last_clean":"2024-03-11T21:42:02.653583+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:42:02.653583+0000","last_undegraded":"2024-03-11T21:42:02.653583+0000","last_fullsized":"2024-03-11T21:42:02.653583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.822599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.822599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.296947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":1501,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.672607+0000","last_change":"2024-03-11T21:41:27.465101+0000","last_active":"2024-03-11T21:41:59.672607+0000","last_peered":"2024-03-11T21:41:59.672607+0000","last_clean":"2024-03-11T21:41:59.672607+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:41:59.672607+0000","last_undegraded":"2024-03-11T21:41:59.672607+0000","last_fullsized":"2024-03-11T21:41:59.672607+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:41:27.464990+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.464990+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.787751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1528,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002657+0000","last_change":"2024-03-11T21:39:51.887003+0000","last_active":"2024-03-11T21:42:03.002657+0000","last_peered":"2024-03-11T21:42:03.002657+0000","last_clean":"2024-03-11T21:42:03.002657+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:42:03.002657+0000","last_undegraded":"2024-03-11T21:42:03.002657+0000","last_fullsized":"2024-03-11T21:42:03.002657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:51.886841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:51.886841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":777,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999128+0000","last_change":"2024-03-11T21:41:18.182642+0000","last_active":"2024-03-11T21:42:02.999128+0000","last_peered":"2024-03-11T21:42:02.999128+0000","last_clean":"2024-03-11T21:42:02.999128+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:42:02.999128+0000","last_undegraded":"2024-03-11T21:42:02.999128+0000","last_fullsized":"2024-03-11T21:42:02.999128+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:18.182513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.182513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.290004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1403,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003310+0000","last_change":"2024-03-11T21:42:01.549509+0000","last_active":"2024-03-11T21:42:03.003310+0000","last_peered":"2024-03-11T21:42:03.003310+0000","last_clean":"2024-03-11T21:42:03.003310+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:42:03.003310+0000","last_undegraded":"2024-03-11T21:42:03.003310+0000","last_fullsized":"2024-03-11T21:42:03.003310+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.549350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.861337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1559,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005497+0000","last_change":"2024-03-11T21:41:58.613981+0000","last_active":"2024-03-11T21:42:03.005497+0000","last_peered":"2024-03-11T21:42:03.005497+0000","last_clean":"2024-03-11T21:42:03.005497+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:42:03.005497+0000","last_undegraded":"2024-03-11T21:42:03.005497+0000","last_fullsized":"2024-03-11T21:42:03.005497+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.613685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.613685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:23.992971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1630,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645312+0000","last_change":"2024-03-11T21:41:29.674697+0000","last_active":"2024-03-11T21:42:02.645312+0000","last_peered":"2024-03-11T21:42:02.645312+0000","last_clean":"2024-03-11T21:42:02.645312+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:42:02.645312+0000","last_undegraded":"2024-03-11T21:42:02.645312+0000","last_fullsized":"2024-03-11T21:42:02.645312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:41:29.674458+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.674458+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.291740+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1562,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001039+0000","last_change":"2024-03-11T21:41:34.232210+0000","last_active":"2024-03-11T21:42:03.001039+0000","last_peered":"2024-03-11T21:42:03.001039+0000","last_clean":"2024-03-11T21:42:03.001039+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:42:03.001039+0000","last_undegraded":"2024-03-11T21:42:03.001039+0000","last_fullsized":"2024-03-11T21:42:03.001039+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:34.231996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:34.231996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.938253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1537,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.704083+0000","last_change":"2024-03-11T21:41:17.378484+0000","last_active":"2024-03-11T21:42:02.704083+0000","last_peered":"2024-03-11T21:42:02.704083+0000","last_clean":"2024-03-11T21:42:02.704083+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:42:02.704083+0000","last_undegraded":"2024-03-11T21:42:02.704083+0000","last_fullsized":"2024-03-11T21:42:02.704083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.378291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.378291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:27.116068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":793,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001110+0000","last_change":"2024-03-11T21:41:51.718486+0000","last_active":"2024-03-11T21:42:03.001110+0000","last_peered":"2024-03-11T21:42:03.001110+0000","last_clean":"2024-03-11T21:42:03.001110+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:42:03.001110+0000","last_undegraded":"2024-03-11T21:42:03.001110+0000","last_fullsized":"2024-03-11T21:42:03.001110+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:51.718268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.718268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.247208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1398,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003561+0000","last_change":"2024-03-11T21:40:21.837198+0000","last_active":"2024-03-11T21:42:03.003561+0000","last_peered":"2024-03-11T21:42:03.003561+0000","last_clean":"2024-03-11T21:42:03.003561+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:42:03.003561+0000","last_undegraded":"2024-03-11T21:42:03.003561+0000","last_fullsized":"2024-03-11T21:42:03.003561+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.836981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.836981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1792,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005306+0000","last_change":"2024-03-11T21:41:40.335413+0000","last_active":"2024-03-11T21:42:03.005306+0000","last_peered":"2024-03-11T21:42:03.005306+0000","last_clean":"2024-03-11T21:42:03.005306+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:42:03.005306+0000","last_undegraded":"2024-03-11T21:42:03.005306+0000","last_fullsized":"2024-03-11T21:42:03.005306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:41:40.335178+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.335178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.489495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1446,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.675883+0000","last_change":"2024-03-11T21:39:51.220612+0000","last_active":"2024-03-11T21:41:59.675883+0000","last_peered":"2024-03-11T21:41:59.675883+0000","last_clean":"2024-03-11T21:41:59.675883+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:41:59.675883+0000","last_undegraded":"2024-03-11T21:41:59.675883+0000","last_fullsized":"2024-03-11T21:41:59.675883+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:39:51.220503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:39:51.220503+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1817,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002036+0000","last_change":"2024-03-11T21:40:05.829359+0000","last_active":"2024-03-11T21:42:03.002036+0000","last_peered":"2024-03-11T21:42:03.002036+0000","last_clean":"2024-03-11T21:42:03.002036+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:42:03.002036+0000","last_undegraded":"2024-03-11T21:42:03.002036+0000","last_fullsized":"2024-03-11T21:42:03.002036+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:40:05.829098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.829098+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1417,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.678329+0000","last_change":"2024-03-11T21:40:03.406984+0000","last_active":"2024-03-11T21:41:59.678329+0000","last_peered":"2024-03-11T21:41:59.678329+0000","last_clean":"2024-03-11T21:41:59.678329+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:41:59.678329+0000","last_undegraded":"2024-03-11T21:41:59.678329+0000","last_fullsized":"2024-03-11T21:41:59.678329+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.406773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.406773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1545,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003115+0000","last_change":"2024-03-11T21:41:28.319779+0000","last_active":"2024-03-11T21:42:03.003115+0000","last_peered":"2024-03-11T21:42:03.003115+0000","last_clean":"2024-03-11T21:42:03.003115+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:42:03.003115+0000","last_undegraded":"2024-03-11T21:42:03.003115+0000","last_fullsized":"2024-03-11T21:42:03.003115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.319670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.319670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.589490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1278,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.679809+0000","last_change":"2024-03-11T21:41:46.296818+0000","last_active":"2024-03-11T21:41:59.679809+0000","last_peered":"2024-03-11T21:41:59.679809+0000","last_clean":"2024-03-11T21:41:59.679809+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:41:59.679809+0000","last_undegraded":"2024-03-11T21:41:59.679809+0000","last_fullsized":"2024-03-11T21:41:59.679809+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.296690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.296690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.764625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1589,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.652479+0000","last_change":"2024-03-11T21:41:45.803597+0000","last_active":"2024-03-11T21:42:02.652479+0000","last_peered":"2024-03-11T21:42:02.652479+0000","last_clean":"2024-03-11T21:42:02.652479+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:42:02.652479+0000","last_undegraded":"2024-03-11T21:42:02.652479+0000","last_fullsized":"2024-03-11T21:42:02.652479+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.803247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.803247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.321102+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1588,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005036+0000","last_change":"2024-03-11T21:39:58.844136+0000","last_active":"2024-03-11T21:42:03.005036+0000","last_peered":"2024-03-11T21:42:03.005036+0000","last_clean":"2024-03-11T21:42:03.005036+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:42:03.005036+0000","last_undegraded":"2024-03-11T21:42:03.005036+0000","last_fullsized":"2024-03-11T21:42:03.005036+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:39:58.843910+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.843910+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1592,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.647235+0000","last_change":"2024-03-11T21:40:20.836641+0000","last_active":"2024-03-11T21:42:02.647235+0000","last_peered":"2024-03-11T21:42:02.647235+0000","last_clean":"2024-03-11T21:42:02.647235+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:42:02.647235+0000","last_undegraded":"2024-03-11T21:42:02.647235+0000","last_fullsized":"2024-03-11T21:42:02.647235+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:40:20.836413+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.836413+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1546,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703987+0000","last_change":"2024-03-11T21:40:24.477381+0000","last_active":"2024-03-11T21:42:02.703987+0000","last_peered":"2024-03-11T21:42:02.703987+0000","last_clean":"2024-03-11T21:42:02.703987+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:42:02.703987+0000","last_undegraded":"2024-03-11T21:42:02.703987+0000","last_fullsized":"2024-03-11T21:42:02.703987+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.477246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.477246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1395,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.704853+0000","last_change":"2024-03-11T21:41:15.373061+0000","last_active":"2024-03-11T21:42:02.704853+0000","last_peered":"2024-03-11T21:42:02.704853+0000","last_clean":"2024-03-11T21:42:02.704853+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:42:02.704853+0000","last_undegraded":"2024-03-11T21:42:02.704853+0000","last_fullsized":"2024-03-11T21:42:02.704853+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.372842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.372842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.257601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1566,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997926+0000","last_change":"2024-03-11T21:39:50.852078+0000","last_active":"2024-03-11T21:42:02.997926+0000","last_peered":"2024-03-11T21:42:02.997926+0000","last_clean":"2024-03-11T21:42:02.997926+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:42:02.997926+0000","last_undegraded":"2024-03-11T21:42:02.997926+0000","last_fullsized":"2024-03-11T21:42:02.997926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:39:50.851951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:39:50.851951+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1597,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.652222+0000","last_change":"2024-03-11T21:41:43.763506+0000","last_active":"2024-03-11T21:42:02.652222+0000","last_peered":"2024-03-11T21:42:02.652222+0000","last_clean":"2024-03-11T21:42:02.652222+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:42:02.652222+0000","last_undegraded":"2024-03-11T21:42:02.652222+0000","last_fullsized":"2024-03-11T21:42:02.652222+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:43.763282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.763282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:01.513528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1549,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.705254+0000","last_change":"2024-03-11T21:41:18.342623+0000","last_active":"2024-03-11T21:42:02.705254+0000","last_peered":"2024-03-11T21:42:02.705254+0000","last_clean":"2024-03-11T21:42:02.705254+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:42:02.705254+0000","last_undegraded":"2024-03-11T21:42:02.705254+0000","last_fullsized":"2024-03-11T21:42:02.705254+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.342525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.342525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.897683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.659708+0000","last_change":"2024-03-11T21:41:16.668285+0000","last_active":"2024-03-11T21:42:02.659708+0000","last_peered":"2024-03-11T21:42:02.659708+0000","last_clean":"2024-03-11T21:42:02.659708+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:42:02.659708+0000","last_undegraded":"2024-03-11T21:42:02.659708+0000","last_fullsized":"2024-03-11T21:42:02.659708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.667920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.667920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.513028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1395,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.701602+0000","last_change":"2024-03-11T21:41:37.401402+0000","last_active":"2024-03-11T21:42:02.701602+0000","last_peered":"2024-03-11T21:42:02.701602+0000","last_clean":"2024-03-11T21:42:02.701602+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:42:02.701602+0000","last_undegraded":"2024-03-11T21:42:02.701602+0000","last_fullsized":"2024-03-11T21:42:02.701602+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.401275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.401275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.811147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1593,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997220+0000","last_change":"2024-03-11T21:41:49.695635+0000","last_active":"2024-03-11T21:42:02.997220+0000","last_peered":"2024-03-11T21:42:02.997220+0000","last_clean":"2024-03-11T21:42:02.997220+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:42:02.997220+0000","last_undegraded":"2024-03-11T21:42:02.997220+0000","last_fullsized":"2024-03-11T21:42:02.997220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:41:49.695261+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.695261+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.305229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997871+0000","last_change":"2024-03-11T21:40:13.608538+0000","last_active":"2024-03-11T21:42:02.997871+0000","last_peered":"2024-03-11T21:42:02.997871+0000","last_clean":"2024-03-11T21:42:02.997871+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:42:02.997871+0000","last_undegraded":"2024-03-11T21:42:02.997871+0000","last_fullsized":"2024-03-11T21:42:02.997871+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.608147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.608147+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.608147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1552,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003210+0000","last_change":"2024-03-11T21:41:42.239297+0000","last_active":"2024-03-11T21:42:03.003210+0000","last_peered":"2024-03-11T21:42:03.003210+0000","last_clean":"2024-03-11T21:42:03.003210+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:42:03.003210+0000","last_undegraded":"2024-03-11T21:42:03.003210+0000","last_fullsized":"2024-03-11T21:42:03.003210+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.239053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.921882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1526,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002556+0000","last_change":"2024-03-11T21:39:01.789461+0000","last_active":"2024-03-11T21:42:03.002556+0000","last_peered":"2024-03-11T21:42:03.002556+0000","last_clean":"2024-03-11T21:42:03.002556+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:42:03.002556+0000","last_undegraded":"2024-03-11T21:42:03.002556+0000","last_fullsized":"2024-03-11T21:42:03.002556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:01.789079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:01.789079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1387,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703915+0000","last_change":"2024-03-11T21:38:38.738989+0000","last_active":"2024-03-11T21:42:02.703915+0000","last_peered":"2024-03-11T21:42:02.703915+0000","last_clean":"2024-03-11T21:42:02.703915+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:42:02.703915+0000","last_undegraded":"2024-03-11T21:42:02.703915+0000","last_fullsized":"2024-03-11T21:42:02.703915+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.738774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.738774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1574,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002887+0000","last_change":"2024-03-11T21:40:26.728613+0000","last_active":"2024-03-11T21:42:03.002887+0000","last_peered":"2024-03-11T21:42:03.002887+0000","last_clean":"2024-03-11T21:42:03.002887+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:42:03.002887+0000","last_undegraded":"2024-03-11T21:42:03.002887+0000","last_fullsized":"2024-03-11T21:42:03.002887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:26.728512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.728512+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1560,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000983+0000","last_change":"2024-03-11T21:40:20.810639+0000","last_active":"2024-03-11T21:42:03.000983+0000","last_peered":"2024-03-11T21:42:03.000983+0000","last_clean":"2024-03-11T21:42:03.000983+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:42:03.000983+0000","last_undegraded":"2024-03-11T21:42:03.000983+0000","last_fullsized":"2024-03-11T21:42:03.000983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.810193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.810193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1791,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002213+0000","last_change":"2024-03-11T21:41:49.267753+0000","last_active":"2024-03-11T21:42:03.002213+0000","last_peered":"2024-03-11T21:42:03.002213+0000","last_clean":"2024-03-11T21:42:03.002213+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:42:03.002213+0000","last_undegraded":"2024-03-11T21:42:03.002213+0000","last_fullsized":"2024-03-11T21:42:03.002213+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:41:49.267522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.267522+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.956526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1544,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.006340+0000","last_change":"2024-03-11T21:40:23.588457+0000","last_active":"2024-03-11T21:42:03.006340+0000","last_peered":"2024-03-11T21:42:03.006340+0000","last_clean":"2024-03-11T21:42:03.006340+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:42:03.006340+0000","last_undegraded":"2024-03-11T21:42:03.006340+0000","last_fullsized":"2024-03-11T21:42:03.006340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:23.588113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.588113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1419,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997504+0000","last_change":"2024-03-11T21:40:26.722273+0000","last_active":"2024-03-11T21:42:02.997504+0000","last_peered":"2024-03-11T21:42:02.997504+0000","last_clean":"2024-03-11T21:42:02.997504+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:42:02.997504+0000","last_undegraded":"2024-03-11T21:42:02.997504+0000","last_fullsized":"2024-03-11T21:42:02.997504+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.722122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.722122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1585,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004193+0000","last_change":"2024-03-11T21:40:21.551903+0000","last_active":"2024-03-11T21:42:03.004193+0000","last_peered":"2024-03-11T21:42:03.004193+0000","last_clean":"2024-03-11T21:42:03.004193+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:42:03.004193+0000","last_undegraded":"2024-03-11T21:42:03.004193+0000","last_fullsized":"2024-03-11T21:42:03.004193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:21.551780+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.551780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1557,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004809+0000","last_change":"2024-03-11T21:41:31.320806+0000","last_active":"2024-03-11T21:42:03.004809+0000","last_peered":"2024-03-11T21:42:03.004809+0000","last_clean":"2024-03-11T21:42:03.004809+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:42:03.004809+0000","last_undegraded":"2024-03-11T21:42:03.004809+0000","last_fullsized":"2024-03-11T21:42:03.004809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.320685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.320685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:50.628945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1780,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703573+0000","last_change":"2024-03-11T21:40:18.396071+0000","last_active":"2024-03-11T21:42:02.703573+0000","last_peered":"2024-03-11T21:42:02.703573+0000","last_clean":"2024-03-11T21:42:02.703573+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:42:02.703573+0000","last_undegraded":"2024-03-11T21:42:02.703573+0000","last_fullsized":"2024-03-11T21:42:02.703573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:40:18.395966+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.395966+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1569,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.647944+0000","last_change":"2024-03-11T21:41:57.724943+0000","last_active":"2024-03-11T21:42:02.647944+0000","last_peered":"2024-03-11T21:42:02.647944+0000","last_clean":"2024-03-11T21:42:02.647944+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:42:02.647944+0000","last_undegraded":"2024-03-11T21:42:02.647944+0000","last_fullsized":"2024-03-11T21:42:02.647944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.724694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:26.895410+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.724694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:18.584812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1274,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.682065+0000","last_change":"2024-03-11T21:41:55.220405+0000","last_active":"2024-03-11T21:41:59.682065+0000","last_peered":"2024-03-11T21:41:59.682065+0000","last_clean":"2024-03-11T21:41:59.682065+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:41:59.682065+0000","last_undegraded":"2024-03-11T21:41:59.682065+0000","last_fullsized":"2024-03-11T21:41:59.682065+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.220196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.220196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.381564+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1571,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003066+0000","last_change":"2024-03-11T21:41:53.406126+0000","last_active":"2024-03-11T21:42:03.003066+0000","last_peered":"2024-03-11T21:42:03.003066+0000","last_clean":"2024-03-11T21:42:03.003066+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:42:03.003066+0000","last_undegraded":"2024-03-11T21:42:03.003066+0000","last_fullsized":"2024-03-11T21:42:03.003066+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:41:53.405780+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.405780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.864386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1583,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999584+0000","last_change":"2024-03-11T21:41:36.715615+0000","last_active":"2024-03-11T21:42:02.999584+0000","last_peered":"2024-03-11T21:42:02.999584+0000","last_clean":"2024-03-11T21:42:02.999584+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:42:02.999584+0000","last_undegraded":"2024-03-11T21:42:02.999584+0000","last_fullsized":"2024-03-11T21:42:02.999584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:36.715491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:41:36.715491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.194697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1550,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.703946+0000","last_change":"2024-03-11T21:41:31.332398+0000","last_active":"2024-03-11T21:42:02.703946+0000","last_peered":"2024-03-11T21:42:02.703946+0000","last_clean":"2024-03-11T21:42:02.703946+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:42:02.703946+0000","last_undegraded":"2024-03-11T21:42:02.703946+0000","last_fullsized":"2024-03-11T21:42:02.703946+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.332299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.332299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.360663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1553,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003111+0000","last_change":"2024-03-11T21:41:32.365712+0000","last_active":"2024-03-11T21:42:03.003111+0000","last_peered":"2024-03-11T21:42:03.003111+0000","last_clean":"2024-03-11T21:42:03.003111+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:42:03.003111+0000","last_undegraded":"2024-03-11T21:42:03.003111+0000","last_fullsized":"2024-03-11T21:42:03.003111+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.365462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.365462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.063953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1401,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005749+0000","last_change":"2024-03-11T21:39:58.428530+0000","last_active":"2024-03-11T21:42:03.005749+0000","last_peered":"2024-03-11T21:42:03.005749+0000","last_clean":"2024-03-11T21:42:03.005749+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:42:03.005749+0000","last_undegraded":"2024-03-11T21:42:03.005749+0000","last_fullsized":"2024-03-11T21:42:03.005749+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:58.428408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.428408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1608,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645244+0000","last_change":"2024-03-11T21:41:15.535768+0000","last_active":"2024-03-11T21:42:02.645244+0000","last_peered":"2024-03-11T21:42:02.645244+0000","last_clean":"2024-03-11T21:42:02.645244+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:42:02.645244+0000","last_undegraded":"2024-03-11T21:42:02.645244+0000","last_fullsized":"2024-03-11T21:42:02.645244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:15.535528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.535528+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.789676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1460,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.680025+0000","last_change":"2024-03-11T21:41:21.483590+0000","last_active":"2024-03-11T21:41:59.680025+0000","last_peered":"2024-03-11T21:41:59.680025+0000","last_clean":"2024-03-11T21:41:59.680025+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:41:59.680025+0000","last_undegraded":"2024-03-11T21:41:59.680025+0000","last_fullsized":"2024-03-11T21:41:59.680025+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.483367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.483367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.508889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1428,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.675673+0000","last_change":"2024-03-11T21:40:18.382831+0000","last_active":"2024-03-11T21:41:59.675673+0000","last_peered":"2024-03-11T21:41:59.675673+0000","last_clean":"2024-03-11T21:41:59.675673+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:41:59.675673+0000","last_undegraded":"2024-03-11T21:41:59.675673+0000","last_fullsized":"2024-03-11T21:41:59.675673+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.382717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1537,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702359+0000","last_change":"2024-03-11T21:41:21.351275+0000","last_active":"2024-03-11T21:42:02.702359+0000","last_peered":"2024-03-11T21:42:02.702359+0000","last_clean":"2024-03-11T21:42:02.702359+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:42:02.702359+0000","last_undegraded":"2024-03-11T21:42:02.702359+0000","last_fullsized":"2024-03-11T21:42:02.702359+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.350961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.350961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.348048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1400,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004787+0000","last_change":"2024-03-11T21:41:48.432501+0000","last_active":"2024-03-11T21:42:03.004787+0000","last_peered":"2024-03-11T21:42:03.004787+0000","last_clean":"2024-03-11T21:42:03.004787+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:42:03.004787+0000","last_undegraded":"2024-03-11T21:42:03.004787+0000","last_fullsized":"2024-03-11T21:42:03.004787+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_clean_scrub_stamp":"2024-03-11T21:41:48.432274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.542101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1578,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.006868+0000","last_change":"2024-03-11T21:42:02.547377+0000","last_active":"2024-03-11T21:42:03.006868+0000","last_peered":"2024-03-11T21:42:03.006868+0000","last_clean":"2024-03-11T21:42:03.006868+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:42:03.006868+0000","last_undegraded":"2024-03-11T21:42:03.006868+0000","last_fullsized":"2024-03-11T21:42:03.006868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:42:02.547005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.547005+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.798290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1551,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003466+0000","last_change":"2024-03-11T21:40:13.779995+0000","last_active":"2024-03-11T21:42:03.003466+0000","last_peered":"2024-03-11T21:42:03.003466+0000","last_clean":"2024-03-11T21:42:03.003466+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:42:03.003466+0000","last_undegraded":"2024-03-11T21:42:03.003466+0000","last_fullsized":"2024-03-11T21:42:03.003466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.779418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.779418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1575,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997801+0000","last_change":"2024-03-11T21:42:02.517280+0000","last_active":"2024-03-11T21:42:02.997801+0000","last_peered":"2024-03-11T21:42:02.997801+0000","last_clean":"2024-03-11T21:42:02.997801+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:42:02.997801+0000","last_undegraded":"2024-03-11T21:42:02.997801+0000","last_fullsized":"2024-03-11T21:42:02.997801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:02.516952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.516952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.145639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1527,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999545+0000","last_change":"2024-03-11T21:39:54.822173+0000","last_active":"2024-03-11T21:42:02.999545+0000","last_peered":"2024-03-11T21:42:02.999545+0000","last_clean":"2024-03-11T21:42:02.999545+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:42:02.999545+0000","last_undegraded":"2024-03-11T21:42:02.999545+0000","last_fullsized":"2024-03-11T21:42:02.999545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:54.822057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:39:54.822057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1403,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.005001+0000","last_change":"2024-03-11T21:40:19.599299+0000","last_active":"2024-03-11T21:42:03.005001+0000","last_peered":"2024-03-11T21:42:03.005001+0000","last_clean":"2024-03-11T21:42:03.005001+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:42:03.005001+0000","last_undegraded":"2024-03-11T21:42:03.005001+0000","last_fullsized":"2024-03-11T21:42:03.005001+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.599110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.599110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1581,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003208+0000","last_change":"2024-03-11T21:40:25.893678+0000","last_active":"2024-03-11T21:42:03.003208+0000","last_peered":"2024-03-11T21:42:03.003208+0000","last_clean":"2024-03-11T21:42:03.003208+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:42:03.003208+0000","last_undegraded":"2024-03-11T21:42:03.003208+0000","last_fullsized":"2024-03-11T21:42:03.003208+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:40:25.893452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.893452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1582,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001403+0000","last_change":"2024-03-11T21:41:59.579259+0000","last_active":"2024-03-11T21:42:03.001403+0000","last_peered":"2024-03-11T21:42:03.001403+0000","last_clean":"2024-03-11T21:42:03.001403+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:42:03.001403+0000","last_undegraded":"2024-03-11T21:42:03.001403+0000","last_fullsized":"2024-03-11T21:42:03.001403+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.579089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.579089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.224629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1531,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997668+0000","last_change":"2024-03-11T21:40:16.866813+0000","last_active":"2024-03-11T21:42:02.997668+0000","last_peered":"2024-03-11T21:42:02.997668+0000","last_clean":"2024-03-11T21:42:02.997668+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:42:02.997668+0000","last_undegraded":"2024-03-11T21:42:02.997668+0000","last_fullsized":"2024-03-11T21:42:02.997668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:16.866545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:16.866545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1537,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.704187+0000","last_change":"2024-03-11T21:41:53.585801+0000","last_active":"2024-03-11T21:42:02.704187+0000","last_peered":"2024-03-11T21:42:02.704187+0000","last_clean":"2024-03-11T21:42:02.704187+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:42:02.704187+0000","last_undegraded":"2024-03-11T21:42:02.704187+0000","last_fullsized":"2024-03-11T21:42:02.704187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.585657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:22.256998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1384,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000108+0000","last_change":"2024-03-11T21:40:18.842726+0000","last_active":"2024-03-11T21:42:03.000108+0000","last_peered":"2024-03-11T21:42:03.000108+0000","last_clean":"2024-03-11T21:42:03.000108+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:42:03.000108+0000","last_undegraded":"2024-03-11T21:42:03.000108+0000","last_fullsized":"2024-03-11T21:42:03.000108+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.842520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.842520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":1513,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.675008+0000","last_change":"2024-03-11T21:39:59.292770+0000","last_active":"2024-03-11T21:41:59.675008+0000","last_peered":"2024-03-11T21:41:59.675008+0000","last_clean":"2024-03-11T21:41:59.675008+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:41:59.675008+0000","last_undegraded":"2024-03-11T21:41:59.675008+0000","last_fullsized":"2024-03-11T21:41:59.675008+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:39:59.292520+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.292520+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1557,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.997217+0000","last_change":"2024-03-11T21:40:24.841665+0000","last_active":"2024-03-11T21:42:02.997217+0000","last_peered":"2024-03-11T21:42:02.997217+0000","last_clean":"2024-03-11T21:42:02.997217+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:42:02.997217+0000","last_undegraded":"2024-03-11T21:42:02.997217+0000","last_fullsized":"2024-03-11T21:42:02.997217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1817,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001515+0000","last_change":"2024-03-11T21:41:33.752080+0000","last_active":"2024-03-11T21:42:03.001515+0000","last_peered":"2024-03-11T21:42:03.001515+0000","last_clean":"2024-03-11T21:42:03.001515+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:42:03.001515+0000","last_undegraded":"2024-03-11T21:42:03.001515+0000","last_fullsized":"2024-03-11T21:42:03.001515+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:33.751930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.751930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:38.500379+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1528,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999922+0000","last_change":"2024-03-11T21:40:13.900758+0000","last_active":"2024-03-11T21:42:02.999922+0000","last_peered":"2024-03-11T21:42:02.999922+0000","last_clean":"2024-03-11T21:42:02.999922+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:42:02.999922+0000","last_undegraded":"2024-03-11T21:42:02.999922+0000","last_fullsized":"2024-03-11T21:42:02.999922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:40:13.900659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.900659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1406,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000730+0000","last_change":"2024-03-11T21:41:56.288055+0000","last_active":"2024-03-11T21:42:03.000730+0000","last_peered":"2024-03-11T21:42:03.000730+0000","last_clean":"2024-03-11T21:42:03.000730+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:42:03.000730+0000","last_undegraded":"2024-03-11T21:42:03.000730+0000","last_fullsized":"2024-03-11T21:42:03.000730+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.287964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.287964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.371511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1578,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702004+0000","last_change":"2024-03-11T21:38:49.931344+0000","last_active":"2024-03-11T21:42:02.702004+0000","last_peered":"2024-03-11T21:42:02.702004+0000","last_clean":"2024-03-11T21:42:02.702004+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:42:02.702004+0000","last_undegraded":"2024-03-11T21:42:02.702004+0000","last_fullsized":"2024-03-11T21:42:02.702004+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:38:49.930664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:49.930664+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1549,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702627+0000","last_change":"2024-03-11T21:39:58.005873+0000","last_active":"2024-03-11T21:42:02.702627+0000","last_peered":"2024-03-11T21:42:02.702627+0000","last_clean":"2024-03-11T21:42:02.702627+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:42:02.702627+0000","last_undegraded":"2024-03-11T21:42:02.702627+0000","last_fullsized":"2024-03-11T21:42:02.702627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:39:58.005747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.005747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1550,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.006200+0000","last_change":"2024-03-11T21:41:41.375255+0000","last_active":"2024-03-11T21:42:03.006200+0000","last_peered":"2024-03-11T21:42:03.006200+0000","last_clean":"2024-03-11T21:42:03.006200+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:42:03.006200+0000","last_undegraded":"2024-03-11T21:42:03.006200+0000","last_fullsized":"2024-03-11T21:42:03.006200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.375162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.174376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1419,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.679627+0000","last_change":"2024-03-11T21:41:49.263645+0000","last_active":"2024-03-11T21:41:59.679627+0000","last_peered":"2024-03-11T21:41:59.679627+0000","last_clean":"2024-03-11T21:41:59.679627+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:41:59.679627+0000","last_undegraded":"2024-03-11T21:41:59.679627+0000","last_fullsized":"2024-03-11T21:41:59.679627+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.263416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.236541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1384,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998067+0000","last_change":"2024-03-11T21:41:23.491021+0000","last_active":"2024-03-11T21:42:02.998067+0000","last_peered":"2024-03-11T21:42:02.998067+0000","last_clean":"2024-03-11T21:42:02.998067+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:42:02.998067+0000","last_undegraded":"2024-03-11T21:42:02.998067+0000","last_fullsized":"2024-03-11T21:42:02.998067+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.490850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.490850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.079493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1559,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.004996+0000","last_change":"2024-03-11T21:40:05.512627+0000","last_active":"2024-03-11T21:42:03.004996+0000","last_peered":"2024-03-11T21:42:03.004996+0000","last_clean":"2024-03-11T21:42:03.004996+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:42:03.004996+0000","last_undegraded":"2024-03-11T21:42:03.004996+0000","last_fullsized":"2024-03-11T21:42:03.004996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:40:05.512463+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.512463+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1582,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.998297+0000","last_change":"2024-03-11T21:41:53.683571+0000","last_active":"2024-03-11T21:42:02.998297+0000","last_peered":"2024-03-11T21:42:02.998297+0000","last_clean":"2024-03-11T21:42:02.998297+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:42:02.998297+0000","last_undegraded":"2024-03-11T21:42:02.998297+0000","last_fullsized":"2024-03-11T21:42:02.998297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.683465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.683465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.982276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1413,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.680690+0000","last_change":"2024-03-11T21:41:17.344885+0000","last_active":"2024-03-11T21:41:59.680690+0000","last_peered":"2024-03-11T21:41:59.680690+0000","last_clean":"2024-03-11T21:41:59.680690+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:41:59.680690+0000","last_undegraded":"2024-03-11T21:41:59.680690+0000","last_fullsized":"2024-03-11T21:41:59.680690+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.344687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.344687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.727912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1563,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.647015+0000","last_change":"2024-03-11T21:41:19.561625+0000","last_active":"2024-03-11T21:42:02.647015+0000","last_peered":"2024-03-11T21:42:02.647015+0000","last_clean":"2024-03-11T21:42:02.647015+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:42:02.647015+0000","last_undegraded":"2024-03-11T21:42:02.647015+0000","last_fullsized":"2024-03-11T21:42:02.647015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.561348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.561348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.036445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1417,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.643803+0000","last_change":"2024-03-11T21:41:55.738509+0000","last_active":"2024-03-11T21:42:02.643803+0000","last_peered":"2024-03-11T21:42:02.643803+0000","last_clean":"2024-03-11T21:42:02.643803+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:42:02.643803+0000","last_undegraded":"2024-03-11T21:42:02.643803+0000","last_fullsized":"2024-03-11T21:42:02.643803+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.738282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.398264+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"714'19","reported_seq":1607,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646756+0000","last_change":"2024-03-11T21:40:13.889468+0000","last_active":"2024-03-11T21:42:02.646756+0000","last_peered":"2024-03-11T21:42:02.646756+0000","last_clean":"2024-03-11T21:42:02.646756+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:42:02.646756+0000","last_undegraded":"2024-03-11T21:42:02.646756+0000","last_fullsized":"2024-03-11T21:42:02.646756+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'19","last_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_deep_scrub":"714'19","last_deep_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.889354+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1548,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702952+0000","last_change":"2024-03-11T21:39:48.014842+0000","last_active":"2024-03-11T21:42:02.702952+0000","last_peered":"2024-03-11T21:42:02.702952+0000","last_clean":"2024-03-11T21:42:02.702952+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:42:02.702952+0000","last_undegraded":"2024-03-11T21:42:02.702952+0000","last_fullsized":"2024-03-11T21:42:02.702952+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:48.014741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:39:48.014741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1565,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645495+0000","last_change":"2024-03-11T21:41:54.741470+0000","last_active":"2024-03-11T21:42:02.645495+0000","last_peered":"2024-03-11T21:42:02.645495+0000","last_clean":"2024-03-11T21:42:02.645495+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:42:02.645495+0000","last_undegraded":"2024-03-11T21:42:02.645495+0000","last_fullsized":"2024-03-11T21:42:02.645495+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.741251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.722799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1538,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.705108+0000","last_change":"2024-03-11T21:41:54.630605+0000","last_active":"2024-03-11T21:42:02.705108+0000","last_peered":"2024-03-11T21:42:02.705108+0000","last_clean":"2024-03-11T21:42:02.705108+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:42:02.705108+0000","last_undegraded":"2024-03-11T21:42:02.705108+0000","last_fullsized":"2024-03-11T21:42:02.705108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.630386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.630386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.496284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1393,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002792+0000","last_change":"2024-03-11T21:41:31.510028+0000","last_active":"2024-03-11T21:42:03.002792+0000","last_peered":"2024-03-11T21:42:03.002792+0000","last_clean":"2024-03-11T21:42:03.002792+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:42:03.002792+0000","last_undegraded":"2024-03-11T21:42:03.002792+0000","last_fullsized":"2024-03-11T21:42:03.002792+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.509821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.509821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.206142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1579,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002168+0000","last_change":"2024-03-11T21:40:28.925505+0000","last_active":"2024-03-11T21:42:03.002168+0000","last_peered":"2024-03-11T21:42:03.002168+0000","last_clean":"2024-03-11T21:42:03.002168+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:42:03.002168+0000","last_undegraded":"2024-03-11T21:42:03.002168+0000","last_fullsized":"2024-03-11T21:42:03.002168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:40:28.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.925267+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1546,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999406+0000","last_change":"2024-03-11T21:41:46.401421+0000","last_active":"2024-03-11T21:42:02.999406+0000","last_peered":"2024-03-11T21:42:02.999406+0000","last_clean":"2024-03-11T21:42:02.999406+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:42:02.999406+0000","last_undegraded":"2024-03-11T21:42:02.999406+0000","last_fullsized":"2024-03-11T21:42:02.999406+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.401211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.401211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.564792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1543,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.001177+0000","last_change":"2024-03-11T21:41:53.275977+0000","last_active":"2024-03-11T21:42:03.001177+0000","last_peered":"2024-03-11T21:42:03.001177+0000","last_clean":"2024-03-11T21:42:03.001177+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:42:03.001177+0000","last_undegraded":"2024-03-11T21:42:03.001177+0000","last_fullsized":"2024-03-11T21:42:03.001177+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.275834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.275834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1552,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003420+0000","last_change":"2024-03-11T21:40:25.590355+0000","last_active":"2024-03-11T21:42:03.003420+0000","last_peered":"2024-03-11T21:42:03.003420+0000","last_clean":"2024-03-11T21:42:03.003420+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:42:03.003420+0000","last_undegraded":"2024-03-11T21:42:03.003420+0000","last_fullsized":"2024-03-11T21:42:03.003420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.590246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.590246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1420,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.646307+0000","last_change":"2024-03-11T21:41:22.601767+0000","last_active":"2024-03-11T21:42:02.646307+0000","last_peered":"2024-03-11T21:42:02.646307+0000","last_clean":"2024-03-11T21:42:02.646307+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:42:02.646307+0000","last_undegraded":"2024-03-11T21:42:02.646307+0000","last_fullsized":"2024-03-11T21:42:02.646307+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.601690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.601690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.459593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1614,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.002074+0000","last_change":"2024-03-11T21:41:51.290471+0000","last_active":"2024-03-11T21:42:03.002074+0000","last_peered":"2024-03-11T21:42:03.002074+0000","last_clean":"2024-03-11T21:42:03.002074+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:42:03.002074+0000","last_undegraded":"2024-03-11T21:42:03.002074+0000","last_fullsized":"2024-03-11T21:42:03.002074+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:41:51.290324+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.290324+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.691718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1549,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.003058+0000","last_change":"2024-03-11T21:40:06.837164+0000","last_active":"2024-03-11T21:42:03.003058+0000","last_peered":"2024-03-11T21:42:03.003058+0000","last_clean":"2024-03-11T21:42:03.003058+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:42:03.003058+0000","last_undegraded":"2024-03-11T21:42:03.003058+0000","last_fullsized":"2024-03-11T21:42:03.003058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:06.836871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.836871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1539,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.999358+0000","last_change":"2024-03-11T21:40:04.809344+0000","last_active":"2024-03-11T21:42:02.999358+0000","last_peered":"2024-03-11T21:42:02.999358+0000","last_clean":"2024-03-11T21:42:02.999358+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:42:02.999358+0000","last_undegraded":"2024-03-11T21:42:02.999358+0000","last_fullsized":"2024-03-11T21:42:02.999358+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:04.809056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.809056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1548,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000008+0000","last_change":"2024-03-11T21:41:47.226377+0000","last_active":"2024-03-11T21:42:03.000008+0000","last_peered":"2024-03-11T21:42:03.000008+0000","last_clean":"2024-03-11T21:42:03.000008+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:42:03.000008+0000","last_undegraded":"2024-03-11T21:42:03.000008+0000","last_fullsized":"2024-03-11T21:42:03.000008+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:47.226166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:47.226166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.255553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1393,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.000103+0000","last_change":"2024-03-11T21:40:02.830965+0000","last_active":"2024-03-11T21:42:03.000103+0000","last_peered":"2024-03-11T21:42:03.000103+0000","last_clean":"2024-03-11T21:42:03.000103+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:42:03.000103+0000","last_undegraded":"2024-03-11T21:42:03.000103+0000","last_fullsized":"2024-03-11T21:42:03.000103+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.830743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.830743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1590,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.007016+0000","last_change":"2024-03-11T21:40:06.481218+0000","last_active":"2024-03-11T21:42:03.007016+0000","last_peered":"2024-03-11T21:42:03.007016+0000","last_clean":"2024-03-11T21:42:03.007016+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:42:03.007016+0000","last_undegraded":"2024-03-11T21:42:03.007016+0000","last_fullsized":"2024-03-11T21:42:03.007016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:40:06.481094+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.481094+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1550,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.704075+0000","last_change":"2024-03-11T21:41:49.603979+0000","last_active":"2024-03-11T21:42:02.704075+0000","last_peered":"2024-03-11T21:42:02.704075+0000","last_clean":"2024-03-11T21:42:02.704075+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:42:02.704075+0000","last_undegraded":"2024-03-11T21:42:02.704075+0000","last_fullsized":"2024-03-11T21:42:02.704075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.603815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.603815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:55.515634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1409,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.675519+0000","last_change":"2024-03-11T21:40:20.395801+0000","last_active":"2024-03-11T21:41:59.675519+0000","last_peered":"2024-03-11T21:41:59.675519+0000","last_clean":"2024-03-11T21:41:59.675519+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:41:59.675519+0000","last_undegraded":"2024-03-11T21:41:59.675519+0000","last_fullsized":"2024-03-11T21:41:59.675519+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.395704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1570,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.645911+0000","last_change":"2024-03-11T21:41:33.725278+0000","last_active":"2024-03-11T21:42:02.645911+0000","last_peered":"2024-03-11T21:42:02.645911+0000","last_clean":"2024-03-11T21:42:02.645911+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:42:02.645911+0000","last_undegraded":"2024-03-11T21:42:02.645911+0000","last_fullsized":"2024-03-11T21:42:02.645911+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.725010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.725010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.536659+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1267,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-03-11T21:41:59.679361+0000","last_change":"2024-03-11T21:41:54.179283+0000","last_active":"2024-03-11T21:41:59.679361+0000","last_peered":"2024-03-11T21:41:59.679361+0000","last_clean":"2024-03-11T21:41:59.679361+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:41:59.679361+0000","last_undegraded":"2024-03-11T21:41:59.679361+0000","last_fullsized":"2024-03-11T21:41:59.679361+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.179012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.179012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.476815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1576,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.704942+0000","last_change":"2024-03-11T21:40:15.377800+0000","last_active":"2024-03-11T21:42:02.704942+0000","last_peered":"2024-03-11T21:42:02.704942+0000","last_clean":"2024-03-11T21:42:02.704942+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:42:02.704942+0000","last_undegraded":"2024-03-11T21:42:02.704942+0000","last_fullsized":"2024-03-11T21:42:02.704942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:40:15.377635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.377635+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1559,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:03.007050+0000","last_change":"2024-03-11T21:40:04.478158+0000","last_active":"2024-03-11T21:42:03.007050+0000","last_peered":"2024-03-11T21:42:03.007050+0000","last_clean":"2024-03-11T21:42:03.007050+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:42:03.007050+0000","last_undegraded":"2024-03-11T21:42:03.007050+0000","last_fullsized":"2024-03-11T21:42:03.007050+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:40:04.477919+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.477919+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1574,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.643837+0000","last_change":"2024-03-11T21:41:27.668035+0000","last_active":"2024-03-11T21:42:02.643837+0000","last_peered":"2024-03-11T21:42:02.643837+0000","last_clean":"2024-03-11T21:42:02.643837+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:42:02.643837+0000","last_undegraded":"2024-03-11T21:42:02.643837+0000","last_fullsized":"2024-03-11T21:42:02.643837+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.667840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.667840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.798307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1528,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-03-11T21:42:02.702961+0000","last_change":"2024-03-11T21:38:39.757621+0000","last_active":"2024-03-11T21:42:02.702961+0000","last_peered":"2024-03-11T21:42:02.702961+0000","last_clean":"2024-03-11T21:42:02.702961+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:42:02.702961+0000","last_undegraded":"2024-03-11T21:42:02.702961+0000","last_fullsized":"2024-03-11T21:42:02.702961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:39.757313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:38:39.757313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":328,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":313,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":7},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1118,"num_read_kb":1113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841775,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105144,"kb_used_data":15640,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93640008,"statfs":{"total":95995035648,"available":95887368192,"internally_reserved":0,"allocated":16015360,"data_stored":10905617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168824,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88920,"kb_used_data":13292,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93656232,"statfs":{"total":95995035648,"available":95903981568,"internally_reserved":0,"allocated":13611008,"data_stored":8531813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352446,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53688,"kb_used_data":13272,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93691464,"statfs":{"total":95995035648,"available":95940059136,"internally_reserved":0,"allocated":13590528,"data_stored":8512628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82600,"kb_used_data":15316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93662552,"statfs":{"total":95995035648,"available":95910453248,"internally_reserved":0,"allocated":15683584,"data_stored":10586498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378914,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94768,"kb_used_data":13596,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93650384,"statfs":{"total":95995035648,"available":95897993216,"internally_reserved":0,"allocated":13922304,"data_stored":8829201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117270,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77004,"kb_used_data":13868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93668148,"statfs":{"total":95995035648,"available":95916183552,"internally_reserved":0,"allocated":14200832,"data_stored":9126215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855628,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100668,"kb_used_data":15336,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93644484,"statfs":{"total":95995035648,"available":95891951616,"internally_reserved":0,"allocated":15704064,"data_stored":10582692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495874,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89496,"kb_used_data":13880,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655656,"statfs":{"total":95995035648,"available":95903391744,"internally_reserved":0,"allocated":14213120,"data_stored":9138480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2039808,"data_stored":2039808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2035712,"data_stored":2035712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2052096,"data_stored":2052096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":328,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:42:06.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:42:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.811 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:06.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:06 smithi203 bash[23654]: cluster 2024-03-11T21:42:05.509661+0000 osd.6 (osd.6) 363 : cluster 0 2.11 deep-scrub starts 2024-03-11T21:42:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:06 smithi203 bash[23654]: cluster 2024-03-11T21:42:05.515027+0000 osd.6 (osd.6) 364 : cluster 0 2.11 deep-scrub ok 2024-03-11T21:42:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:06 smithi203 bash[23654]: cluster 2024-03-11T21:42:05.608000+0000 mgr.y (mgr.14550) 741 : cluster 0 pgmap v1272: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:06 smithi203 bash[23654]: cluster 2024-03-11T21:42:06.545837+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e824: 8 total, 8 up, 8 in 2024-03-11T21:42:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:07.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:07.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:07.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.195 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:07.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:07.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:07.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.459 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:07.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.591 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:07.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:07.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:07 smithi203 bash[23654]: cluster 2024-03-11T21:42:06.838198+0000 osd.6 (osd.6) 365 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:07.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:07 smithi203 bash[23654]: cluster 2024-03-11T21:42:06.838449+0000 osd.6 (osd.6) 366 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:07.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:07 smithi203 bash[23654]: audit 2024-03-11T21:42:07.373042+0000 mon.a (mon.0) 3757 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:07.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:07 smithi203 bash[23654]: audit 2024-03-11T21:42:07.375158+0000 mon.c (mon.2) 1292 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:07.722 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:07.825 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:07.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:07.943 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[21512]: cluster 2024-03-11T21:42:06.838198+0000 osd.6 (osd.6) 365 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[21512]: cluster 2024-03-11T21:42:06.838449+0000 osd.6 (osd.6) 366 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[21512]: audit 2024-03-11T21:42:07.373042+0000 mon.a (mon.0) 3757 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[21512]: audit 2024-03-11T21:42:07.375158+0000 mon.c (mon.2) 1292 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[29284]: cluster 2024-03-11T21:42:06.838198+0000 osd.6 (osd.6) 365 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[29284]: cluster 2024-03-11T21:42:06.838449+0000 osd.6 (osd.6) 366 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[29284]: audit 2024-03-11T21:42:07.373042+0000 mon.a (mon.0) 3757 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:07.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:07 smithi143 bash[29284]: audit 2024-03-11T21:42:07.375158+0000 mon.c (mon.2) 1292 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:07.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:08.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:08.116 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:08.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:08.246 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:08.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:08.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:08.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.506 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:08.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.637 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:08.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:08.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:08 smithi143 bash[21512]: cluster 2024-03-11T21:42:07.608831+0000 mgr.y (mgr.14550) 742 : cluster 0 pgmap v1275: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:08.753 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:08 smithi143 bash[21512]: cluster 2024-03-11T21:42:07.714589+0000 mon.a (mon.0) 3758 : cluster 0 osdmap e825: 8 total, 8 up, 8 in 2024-03-11T21:42:08.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:08 smithi143 bash[29284]: cluster 2024-03-11T21:42:07.608831+0000 mgr.y (mgr.14550) 742 : cluster 0 pgmap v1275: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:08.753 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:08 smithi143 bash[29284]: cluster 2024-03-11T21:42:07.714589+0000 mon.a (mon.0) 3758 : cluster 0 osdmap e825: 8 total, 8 up, 8 in 2024-03-11T21:42:08.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:08.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:42:08.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:42:08.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:08.897 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:09.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:08 smithi203 bash[23654]: cluster 2024-03-11T21:42:07.608831+0000 mgr.y (mgr.14550) 742 : cluster 0 pgmap v1275: 246 pgs: 246 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:09.012 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:08 smithi203 bash[23654]: cluster 2024-03-11T21:42:07.714589+0000 mon.a (mon.0) 3758 : cluster 0 osdmap e825: 8 total, 8 up, 8 in 2024-03-11T21:42:09.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.159 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:09.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:09.290 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.422 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.553 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:09.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:09.684 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:09.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.815 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.930 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:09 smithi143 bash[21512]: audit 2024-03-11T21:42:09.033760+0000 mgr.y (mgr.14550) 743 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:09.930 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:09 smithi143 bash[29284]: audit 2024-03-11T21:42:09.033760+0000 mgr.y (mgr.14550) 743 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:09.946 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:10.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:09 smithi203 bash[23654]: audit 2024-03-11T21:42:09.033760+0000 mgr.y (mgr.14550) 743 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:10.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.075 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:10.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.207 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:10.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:10.338 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:10.375 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:10.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:10.468 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:10.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:10.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.730 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:10.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.845 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:10 smithi143 bash[21512]: cluster 2024-03-11T21:42:09.610214+0000 mgr.y (mgr.14550) 744 : cluster 0 pgmap v1277: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-11T21:42:10.846 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:10 smithi143 bash[29284]: cluster 2024-03-11T21:42:09.610214+0000 mgr.y (mgr.14550) 744 : cluster 0 pgmap v1277: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-11T21:42:10.862 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:10.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.989 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:11.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:10 smithi203 bash[23654]: cluster 2024-03-11T21:42:09.610214+0000 mgr.y (mgr.14550) 744 : cluster 0 pgmap v1277: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-11T21:42:11.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:11.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:11.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:11.252 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:11.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:11.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:11.387 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:11.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:11.510 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:11.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:11.637 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:11.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:11.765 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:11.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:11.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.026 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:12.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:12.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:12.193 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:12.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:12.285 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:12.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:12.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:12.547 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:12.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:12.678 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:12.693 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[21512]: cluster 2024-03-11T21:42:11.546970+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e826: 8 total, 8 up, 8 in 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[21512]: cluster 2024-03-11T21:42:11.610698+0000 mgr.y (mgr.14550) 745 : cluster 0 pgmap v1279: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[21512]: cluster 2024-03-11T21:42:11.646372+0000 osd.2 (osd.2) 387 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[21512]: cluster 2024-03-11T21:42:11.646820+0000 osd.2 (osd.2) 388 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[29284]: cluster 2024-03-11T21:42:11.546970+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e826: 8 total, 8 up, 8 in 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[29284]: cluster 2024-03-11T21:42:11.610698+0000 mgr.y (mgr.14550) 745 : cluster 0 pgmap v1279: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[29284]: cluster 2024-03-11T21:42:11.646372+0000 osd.2 (osd.2) 387 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:12.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:12 smithi143 bash[29284]: cluster 2024-03-11T21:42:11.646820+0000 osd.2 (osd.2) 388 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:12.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:12 smithi203 bash[23654]: cluster 2024-03-11T21:42:11.546970+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e826: 8 total, 8 up, 8 in 2024-03-11T21:42:12.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:12 smithi203 bash[23654]: cluster 2024-03-11T21:42:11.610698+0000 mgr.y (mgr.14550) 745 : cluster 0 pgmap v1279: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-11T21:42:12.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:12 smithi203 bash[23654]: cluster 2024-03-11T21:42:11.646372+0000 osd.2 (osd.2) 387 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:12.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:12 smithi203 bash[23654]: cluster 2024-03-11T21:42:11.646820+0000 osd.2 (osd.2) 388 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:12.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.810 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:12.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:13.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:13.069 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:13.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:13.199 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:13.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:13.331 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:13.422 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:13.422 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:13.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:13.456 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:13.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:13.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:13 smithi203 bash[23654]: cluster 2024-03-11T21:42:12.567173+0000 mon.a (mon.0) 3760 : cluster 0 osdmap e827: 8 total, 8 up, 8 in 2024-03-11T21:42:13.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:13.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:13 smithi143 bash[21512]: cluster 2024-03-11T21:42:12.567173+0000 mon.a (mon.0) 3760 : cluster 0 osdmap e827: 8 total, 8 up, 8 in 2024-03-11T21:42:13.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:13 smithi143 bash[29284]: cluster 2024-03-11T21:42:12.567173+0000 mon.a (mon.0) 3760 : cluster 0 osdmap e827: 8 total, 8 up, 8 in 2024-03-11T21:42:13.718 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:13.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:13.848 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:13.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.980 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:14.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.109 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:14.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.240 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:14.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:14.370 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:14.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:14.501 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:14.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:14.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:14.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:14.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:14 smithi203 bash[23654]: cluster 2024-03-11T21:42:13.611938+0000 mgr.y (mgr.14550) 746 : cluster 0 pgmap v1281: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-03-11T21:42:14.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:14 smithi203 bash[23654]: cluster 2024-03-11T21:42:14.120720+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:14.745 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:14 smithi203 bash[23654]: cluster 2024-03-11T21:42:14.121072+0000 osd.7 (osd.7) 439 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:14.760 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:14.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[29284]: cluster 2024-03-11T21:42:13.611938+0000 mgr.y (mgr.14550) 746 : cluster 0 pgmap v1281: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[29284]: cluster 2024-03-11T21:42:14.120720+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[29284]: cluster 2024-03-11T21:42:14.121072+0000 osd.7 (osd.7) 439 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[21512]: cluster 2024-03-11T21:42:13.611938+0000 mgr.y (mgr.14550) 746 : cluster 0 pgmap v1281: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[21512]: cluster 2024-03-11T21:42:14.120720+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:14.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:14 smithi143 bash[21512]: cluster 2024-03-11T21:42:14.121072+0000 osd.7 (osd.7) 439 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:14.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:15.019 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:15.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:15.151 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:15.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:15.282 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:15.385 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:15.412 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:15.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:15.544 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:15.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:15.676 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:15.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:15.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:15 smithi203 bash[23654]: cluster 2024-03-11T21:42:14.585106+0000 mon.a (mon.0) 3761 : cluster 0 osdmap e828: 8 total, 8 up, 8 in 2024-03-11T21:42:15.807 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:15.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.922 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:15 smithi143 bash[21512]: cluster 2024-03-11T21:42:14.585106+0000 mon.a (mon.0) 3761 : cluster 0 osdmap e828: 8 total, 8 up, 8 in 2024-03-11T21:42:15.922 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:15 smithi143 bash[29284]: cluster 2024-03-11T21:42:14.585106+0000 mon.a (mon.0) 3761 : cluster 0 osdmap e828: 8 total, 8 up, 8 in 2024-03-11T21:42:15.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:16.071 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:16.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.202 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.306 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.332 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.435 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:16.592 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:16.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:16.724 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:16.731 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:16 smithi143 bash[29284]: cluster 2024-03-11T21:42:15.612727+0000 mgr.y (mgr.14550) 747 : cluster 0 pgmap v1283: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:16.732 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:16 smithi143 bash[21512]: cluster 2024-03-11T21:42:15.612727+0000 mgr.y (mgr.14550) 747 : cluster 0 pgmap v1283: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:16.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.855 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:16.984 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:17.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:16 smithi203 bash[23654]: cluster 2024-03-11T21:42:15.612727+0000 mgr.y (mgr.14550) 747 : cluster 0 pgmap v1283: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:17.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:17.115 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:17.218 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:17.246 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:17.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:17.375 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:17.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.503 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:17.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.632 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:17.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.763 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:17.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:17.889 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:17.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:18.016 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:18.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:18.146 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:18.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:18.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:18.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:18.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:18.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:18.535 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:18.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:18.664 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:18.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:18.778 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: cluster 2024-03-11T21:42:17.477090+0000 osd.4 (osd.4) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: cluster 2024-03-11T21:42:17.477456+0000 osd.4 (osd.4) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: cluster 2024-03-11T21:42:17.590978+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e829: 8 total, 8 up, 8 in 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: cluster 2024-03-11T21:42:17.613444+0000 mgr.y (mgr.14550) 748 : cluster 0 pgmap v1285: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: audit 2024-03-11T21:42:17.771317+0000 mon.c (mon.2) 1293 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: audit 2024-03-11T21:42:17.772237+0000 mon.a (mon.0) 3763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: audit 2024-03-11T21:42:17.773571+0000 mon.c (mon.2) 1294 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:18 smithi203 bash[23654]: audit 2024-03-11T21:42:17.774201+0000 mon.a (mon.0) 3764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.795 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:18.844 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: cluster 2024-03-11T21:42:17.477090+0000 osd.4 (osd.4) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: cluster 2024-03-11T21:42:17.477456+0000 osd.4 (osd.4) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: cluster 2024-03-11T21:42:17.590978+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e829: 8 total, 8 up, 8 in 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: cluster 2024-03-11T21:42:17.613444+0000 mgr.y (mgr.14550) 748 : cluster 0 pgmap v1285: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: audit 2024-03-11T21:42:17.771317+0000 mon.c (mon.2) 1293 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: audit 2024-03-11T21:42:17.772237+0000 mon.a (mon.0) 3763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: audit 2024-03-11T21:42:17.773571+0000 mon.c (mon.2) 1294 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.845 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[29284]: audit 2024-03-11T21:42:17.774201+0000 mon.a (mon.0) 3764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: cluster 2024-03-11T21:42:17.477090+0000 osd.4 (osd.4) 429 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: cluster 2024-03-11T21:42:17.477456+0000 osd.4 (osd.4) 430 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: cluster 2024-03-11T21:42:17.590978+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e829: 8 total, 8 up, 8 in 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: cluster 2024-03-11T21:42:17.613444+0000 mgr.y (mgr.14550) 748 : cluster 0 pgmap v1285: 246 pgs: 246 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 7.0 KiB/s wr, 1 op/s 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: audit 2024-03-11T21:42:17.771317+0000 mon.c (mon.2) 1293 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: audit 2024-03-11T21:42:17.772237+0000 mon.a (mon.0) 3763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:18.846 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: audit 2024-03-11T21:42:17.773571+0000 mon.c (mon.2) 1294 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.847 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:18 smithi143 bash[21512]: audit 2024-03-11T21:42:17.774201+0000 mon.a (mon.0) 3764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]: dispatch 2024-03-11T21:42:18.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:18.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:19.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:19.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:19.051 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:19.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:19.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:19.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:19.311 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:19.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:19.444 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:19.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:19.575 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:19.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:19.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:19.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:19.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:19 smithi203 bash[23654]: audit 2024-03-11T21:42:18.595184+0000 mon.a (mon.0) 3765 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]': finished 2024-03-11T21:42:19.819 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:19 smithi203 bash[23654]: cluster 2024-03-11T21:42:18.601218+0000 mon.a (mon.0) 3766 : cluster 0 osdmap e830: 8 total, 8 up, 8 in 2024-03-11T21:42:19.820 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:19 smithi203 bash[23654]: audit 2024-03-11T21:42:19.044755+0000 mgr.y (mgr.14550) 749 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:19.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:19.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:19.962 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:20.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:20.090 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:20.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:20.213 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:20.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:20.343 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:19 smithi143 bash[21512]: audit 2024-03-11T21:42:18.595184+0000 mon.a (mon.0) 3765 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]': finished 2024-03-11T21:42:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:19 smithi143 bash[21512]: cluster 2024-03-11T21:42:18.601218+0000 mon.a (mon.0) 3766 : cluster 0 osdmap e830: 8 total, 8 up, 8 in 2024-03-11T21:42:20.439 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:19 smithi143 bash[21512]: audit 2024-03-11T21:42:19.044755+0000 mgr.y (mgr.14550) 749 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:20.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.473 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:20.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:20.603 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:20.690 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[21512]: cluster 2024-03-11T21:42:19.611961+0000 mon.a (mon.0) 3767 : cluster 0 osdmap e831: 8 total, 8 up, 8 in 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[21512]: cluster 2024-03-11T21:42:19.616873+0000 mgr.y (mgr.14550) 750 : cluster 0 pgmap v1288: 214 pgs: 214 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[29284]: audit 2024-03-11T21:42:18.595184+0000 mon.a (mon.0) 3765 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-140"}]': finished 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[29284]: cluster 2024-03-11T21:42:18.601218+0000 mon.a (mon.0) 3766 : cluster 0 osdmap e830: 8 total, 8 up, 8 in 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[29284]: audit 2024-03-11T21:42:19.044755+0000 mgr.y (mgr.14550) 749 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[29284]: cluster 2024-03-11T21:42:19.611961+0000 mon.a (mon.0) 3767 : cluster 0 osdmap e831: 8 total, 8 up, 8 in 2024-03-11T21:42:20.691 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:20 smithi143 bash[29284]: cluster 2024-03-11T21:42:19.616873+0000 mgr.y (mgr.14550) 750 : cluster 0 pgmap v1288: 214 pgs: 214 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:42:20.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:20 smithi203 bash[23654]: cluster 2024-03-11T21:42:19.611961+0000 mon.a (mon.0) 3767 : cluster 0 osdmap e831: 8 total, 8 up, 8 in 2024-03-11T21:42:20.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:20 smithi203 bash[23654]: cluster 2024-03-11T21:42:19.616873+0000 mgr.y (mgr.14550) 750 : cluster 0 pgmap v1288: 214 pgs: 214 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:42:20.730 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:20.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:20.859 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:20.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:20.986 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:21.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:21.117 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:21.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:21.246 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:21.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:21.378 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:21.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:21.507 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:21.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:21.649 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:21.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:21.780 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:21.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[29284]: cluster 2024-03-11T21:42:20.627447+0000 mon.a (mon.0) 3768 : cluster 0 osdmap e832: 8 total, 8 up, 8 in 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[29284]: audit 2024-03-11T21:42:20.631850+0000 mon.c (mon.2) 1295 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[29284]: audit 2024-03-11T21:42:20.638023+0000 mon.a (mon.0) 3769 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[21512]: cluster 2024-03-11T21:42:20.627447+0000 mon.a (mon.0) 3768 : cluster 0 osdmap e832: 8 total, 8 up, 8 in 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[21512]: audit 2024-03-11T21:42:20.631850+0000 mon.c (mon.2) 1295 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:21.895 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:21 smithi143 bash[21512]: audit 2024-03-11T21:42:20.638023+0000 mon.a (mon.0) 3769 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:21.910 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:22.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:22.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:21 smithi203 bash[23654]: cluster 2024-03-11T21:42:20.627447+0000 mon.a (mon.0) 3768 : cluster 0 osdmap e832: 8 total, 8 up, 8 in 2024-03-11T21:42:22.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:21 smithi203 bash[23654]: audit 2024-03-11T21:42:20.631850+0000 mon.c (mon.2) 1295 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:22.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:21 smithi203 bash[23654]: audit 2024-03-11T21:42:20.638023+0000 mon.a (mon.0) 3769 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:22.039 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:22.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:22.170 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:22.194 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:22.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:22.301 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:22.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:22.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:22.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:22.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:22.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:22.699 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:22.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:22.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:22.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:22.959 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: cluster 2024-03-11T21:42:21.617807+0000 mgr.y (mgr.14550) 751 : cluster 0 pgmap v1290: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: audit 2024-03-11T21:42:21.624223+0000 mon.a (mon.0) 3770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: cluster 2024-03-11T21:42:21.633581+0000 mon.a (mon.0) 3771 : cluster 0 osdmap e833: 8 total, 8 up, 8 in 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: cluster 2024-03-11T21:42:21.670797+0000 mon.a (mon.0) 3772 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: audit 2024-03-11T21:42:21.712337+0000 mon.c (mon.2) 1296 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: audit 2024-03-11T21:42:21.712814+0000 mon.a (mon.0) 3773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[29284]: audit 2024-03-11T21:42:22.391599+0000 mon.c (mon.2) 1297 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:22.961 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: cluster 2024-03-11T21:42:21.617807+0000 mgr.y (mgr.14550) 751 : cluster 0 pgmap v1290: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: audit 2024-03-11T21:42:21.624223+0000 mon.a (mon.0) 3770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: cluster 2024-03-11T21:42:21.633581+0000 mon.a (mon.0) 3771 : cluster 0 osdmap e833: 8 total, 8 up, 8 in 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: cluster 2024-03-11T21:42:21.670797+0000 mon.a (mon.0) 3772 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: audit 2024-03-11T21:42:21.712337+0000 mon.c (mon.2) 1296 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: audit 2024-03-11T21:42:21.712814+0000 mon.a (mon.0) 3773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:22.962 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:22 smithi143 bash[21512]: audit 2024-03-11T21:42:22.391599+0000 mon.c (mon.2) 1297 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: cluster 2024-03-11T21:42:21.617807+0000 mgr.y (mgr.14550) 751 : cluster 0 pgmap v1290: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: audit 2024-03-11T21:42:21.624223+0000 mon.a (mon.0) 3770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: cluster 2024-03-11T21:42:21.633581+0000 mon.a (mon.0) 3771 : cluster 0 osdmap e833: 8 total, 8 up, 8 in 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: cluster 2024-03-11T21:42:21.670797+0000 mon.a (mon.0) 3772 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: audit 2024-03-11T21:42:21.712337+0000 mon.c (mon.2) 1296 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: audit 2024-03-11T21:42:21.712814+0000 mon.a (mon.0) 3773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:23.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:22 smithi203 bash[23654]: audit 2024-03-11T21:42:22.391599+0000 mon.c (mon.2) 1297 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:23.064 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:23.091 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:23.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:23.223 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:23.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:23.334 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:23.334 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:23.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:23.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.483 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:23.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:23.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:23.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.737 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:23.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:23.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:23.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[21512]: audit 2024-03-11T21:42:22.655383+0000 mon.a (mon.0) 3774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:23.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[21512]: cluster 2024-03-11T21:42:22.662965+0000 mon.a (mon.0) 3775 : cluster 0 osdmap e834: 8 total, 8 up, 8 in 2024-03-11T21:42:23.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[21512]: audit 2024-03-11T21:42:22.667261+0000 mon.c (mon.2) 1298 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[21512]: audit 2024-03-11T21:42:22.672529+0000 mon.a (mon.0) 3776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[29284]: audit 2024-03-11T21:42:22.655383+0000 mon.a (mon.0) 3774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:23.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[29284]: cluster 2024-03-11T21:42:22.662965+0000 mon.a (mon.0) 3775 : cluster 0 osdmap e834: 8 total, 8 up, 8 in 2024-03-11T21:42:23.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[29284]: audit 2024-03-11T21:42:22.667261+0000 mon.c (mon.2) 1298 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.881 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:23 smithi143 bash[29284]: audit 2024-03-11T21:42:22.672529+0000 mon.a (mon.0) 3776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:23.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:23 smithi203 bash[23654]: audit 2024-03-11T21:42:22.655383+0000 mon.a (mon.0) 3774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:23.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:23 smithi203 bash[23654]: cluster 2024-03-11T21:42:22.662965+0000 mon.a (mon.0) 3775 : cluster 0 osdmap e834: 8 total, 8 up, 8 in 2024-03-11T21:42:23.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:23 smithi203 bash[23654]: audit 2024-03-11T21:42:22.667261+0000 mon.c (mon.2) 1298 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:23 smithi203 bash[23654]: audit 2024-03-11T21:42:22.672529+0000 mon.a (mon.0) 3776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:23.996 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- 172.21.15.143:0/1321306087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f006a720 msgr2=0x7fe1f0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- 172.21.15.143:0/1321306087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f006a720 0x7fe1f0106a40 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7fe1e40099c0 tx=0x7fe1e4009cd0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- 172.21.15.143:0/1321306087 shutdown_connections 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- 172.21.15.143:0/1321306087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f0069d70 0x7fe1f006a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- 172.21.15.143:0/1321306087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 0x7fe1f010ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- 172.21.15.143:0/1321306087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f006a720 0x7fe1f0106a40 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- 172.21.15.143:0/1321306087 >> 172.21.15.143:0/1321306087 conn(0x7fe1f0076d80 msgr2=0x7fe1f0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- 172.21.15.143:0/1321306087 shutdown_connections 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- 172.21.15.143:0/1321306087 wait complete. 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 librados: starting msgr at 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 librados: starting objecter 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 Processor -- start 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- start start 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 librados: setting wanted keys 2024-03-11T21:42:24.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 librados: calling monclient init 2024-03-11T21:42:24.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 0x7fe1f0199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:24.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 0x7fe1f019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 0x7fe1f019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0111650 con 0x7fe1f006a720 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f01114d0 con 0x7fe1f0107120 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.055+0000 7fe1f8453700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f01117d0 con 0x7fe1f0069d70 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f61ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 0x7fe1f0199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f61ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 0x7fe1f0199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:58222/0 (socket says 172.21.15.143:58222) 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f61ef700 1 -- 172.21.15.143:0/540765965 learned_addr learned my addr 172.21.15.143:0/540765965 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 0x7fe1f019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f69f0700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 0x7fe1f019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 -- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 msgr2=0x7fe1f0199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 0x7fe1f0199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 -- 172.21.15.143:0/540765965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 msgr2=0x7fe1f019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 0x7fe1f019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 -- 172.21.15.143:0/540765965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f019ee90 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f59ee700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 0x7fe1f019a3c0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0069930 tx=0x7fe1e400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 10 client.?.objecter ms_handle_connect 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4009e60 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f019f120 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe1e400fce0 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e400bea0 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f8453700 10 client.30855.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f01a6a00 con 0x7fe1f006a720 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f8453700 1 librados: init done 2024-03-11T21:42:24.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f004faa0 con 0x7fe1f006a720 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe1e4003b20 con 0x7fe1f006a720 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 0x7fe1d8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(835..835 src has 87..835) v4 ==== 10261+0+0 (secure 0 0 0) 0x7fe1e408fc90 con 0x7fe1f006a720 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 10 client.30855.objecter ms_dispatch 0x7fe1f0001550 osd_map(835..835 src has 87..835) v4 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 3 client.30855.objecter handle_osd_map got epochs [835,835] > 0 2024-03-11T21:42:24.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 3 client.30855.objecter handle_osd_map decoding full epoch 835 2024-03-11T21:42:24.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f61ef700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 0x7fe1d8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:24.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 20 client.30855.objecter dump_active .. 0 homeless 2024-03-11T21:42:24.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1ef7fe700 10 client.30855.objecter ms_handle_connect 0x7fe1d806f820 2024-03-11T21:42:24.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.059+0000 7fe1f61ef700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 0x7fe1d8071ce0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009440 tx=0x7fe1e800b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:24.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.067+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe1e4013070 con 0x7fe1f006a720 2024-03-11T21:42:24.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.126 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:24.229 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:24.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.267+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1f010d760 con 0x7fe1d806f820 2024-03-11T21:42:24.254 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:24.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.291+0000 7fe1ef7fe700 1 -- 172.21.15.143:0/540765965 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+635812 (secure 0 0 0) 0x7fe1f010d760 con 0x7fe1d806f820 2024-03-11T21:42:24.280 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:42:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 10 librados: watch_flush enter 2024-03-11T21:42:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 10 librados: watch_flush exit 2024-03-11T21:42:24.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 20 client.30855.objecter shutdown clearing up homeless session... 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 10 client.30855.objecter successfully canceled tick 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 msgr2=0x7fe1d8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 0x7fe1d8071ce0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009440 tx=0x7fe1e800b040 comp rx=0 tx=0).stop 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 msgr2=0x7fe1f019a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 0x7fe1f019a3c0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0069930 tx=0x7fe1e400fa00 comp rx=0 tx=0).stop 2024-03-11T21:42:24.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 shutdown_connections 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe1d806f820 0x7fe1d8071ce0 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f0107120 0x7fe1f019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1f0069d70 0x7fe1f0199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 --2- 172.21.15.143:0/540765965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1f006a720 0x7fe1f019a3c0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 >> 172.21.15.143:0/540765965 conn(0x7fe1f0076d80 msgr2=0x7fe1f010d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:24.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 shutdown_connections 2024-03-11T21:42:24.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 -- 172.21.15.143:0/540765965 wait complete. 2024-03-11T21:42:24.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:24.295+0000 7fe1f8453700 1 librados: shutdown 2024-03-11T21:42:24.292 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:42:24.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:24.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.515 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:24.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:24.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:24 smithi203 bash[23654]: cluster 2024-03-11T21:42:23.619117+0000 mgr.y (mgr.14550) 752 : cluster 0 pgmap v1293: 246 pgs: 9 unknown, 237 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-11T21:42:24.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:24 smithi203 bash[23654]: audit 2024-03-11T21:42:23.663635+0000 mon.a (mon.0) 3777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:24.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:24 smithi203 bash[23654]: cluster 2024-03-11T21:42:23.669101+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e835: 8 total, 8 up, 8 in 2024-03-11T21:42:24.658 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:24 smithi203 bash[23654]: audit 2024-03-11T21:42:23.671294+0000 mon.c (mon.2) 1299 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.659 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:24 smithi203 bash[23654]: audit 2024-03-11T21:42:23.671814+0000 mon.a (mon.0) 3779 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:24.772 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:24.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:24.886 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[21512]: cluster 2024-03-11T21:42:23.619117+0000 mgr.y (mgr.14550) 752 : cluster 0 pgmap v1293: 246 pgs: 9 unknown, 237 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[21512]: audit 2024-03-11T21:42:23.663635+0000 mon.a (mon.0) 3777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[21512]: cluster 2024-03-11T21:42:23.669101+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e835: 8 total, 8 up, 8 in 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[21512]: audit 2024-03-11T21:42:23.671294+0000 mon.c (mon.2) 1299 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[21512]: audit 2024-03-11T21:42:23.671814+0000 mon.a (mon.0) 3779 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[29284]: cluster 2024-03-11T21:42:23.619117+0000 mgr.y (mgr.14550) 752 : cluster 0 pgmap v1293: 246 pgs: 9 unknown, 237 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[29284]: audit 2024-03-11T21:42:23.663635+0000 mon.a (mon.0) 3777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[29284]: cluster 2024-03-11T21:42:23.669101+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e835: 8 total, 8 up, 8 in 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[29284]: audit 2024-03-11T21:42:23.671294+0000 mon.c (mon.2) 1299 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.887 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:24 smithi143 bash[29284]: audit 2024-03-11T21:42:23.671814+0000 mon.a (mon.0) 3779 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]: dispatch 2024-03-11T21:42:24.904 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:25.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:25.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.166 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:25.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.300 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:25.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.430 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:25.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:25.563 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:25.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.694 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: audit 2024-03-11T21:42:24.272086+0000 mgr.y (mgr.14550) 753 : audit 0 from='client.30855 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:25.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: cluster 2024-03-11T21:42:24.664301+0000 mon.a (mon.0) 3780 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:25.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: audit 2024-03-11T21:42:24.665762+0000 mon.a (mon.0) 3781 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]': finished 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: cluster 2024-03-11T21:42:24.669276+0000 mon.a (mon.0) 3782 : cluster 0 osdmap e836: 8 total, 8 up, 8 in 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: audit 2024-03-11T21:42:24.671327+0000 mon.c (mon.2) 1300 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[29284]: audit 2024-03-11T21:42:24.678168+0000 mon.a (mon.0) 3783 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: audit 2024-03-11T21:42:24.272086+0000 mgr.y (mgr.14550) 753 : audit 0 from='client.30855 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: cluster 2024-03-11T21:42:24.664301+0000 mon.a (mon.0) 3780 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: audit 2024-03-11T21:42:24.665762+0000 mon.a (mon.0) 3781 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]': finished 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: cluster 2024-03-11T21:42:24.669276+0000 mon.a (mon.0) 3782 : cluster 0 osdmap e836: 8 total, 8 up, 8 in 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: audit 2024-03-11T21:42:24.671327+0000 mon.c (mon.2) 1300 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:25.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:25 smithi143 bash[21512]: audit 2024-03-11T21:42:24.678168+0000 mon.a (mon.0) 3783 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:25.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:25.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:25.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:25.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: audit 2024-03-11T21:42:24.272086+0000 mgr.y (mgr.14550) 753 : audit 0 from='client.30855 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: cluster 2024-03-11T21:42:24.664301+0000 mon.a (mon.0) 3780 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: audit 2024-03-11T21:42:24.665762+0000 mon.a (mon.0) 3781 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-142", "mode": "writeback"}]': finished 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: cluster 2024-03-11T21:42:24.669276+0000 mon.a (mon.0) 3782 : cluster 0 osdmap e836: 8 total, 8 up, 8 in 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: audit 2024-03-11T21:42:24.671327+0000 mon.c (mon.2) 1300 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:26.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:25 smithi203 bash[23654]: audit 2024-03-11T21:42:24.678168+0000 mon.a (mon.0) 3783 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:26.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:26.084 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:26.187 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.214 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:26.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:26.344 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:26.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:26.476 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:26.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:26.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:26.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:26.738 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:26.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.859 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: cluster 2024-03-11T21:42:25.619879+0000 mgr.y (mgr.14550) 754 : cluster 0 pgmap v1296: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:25.685103+0000 mon.a (mon.0) 3784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:25.692182+0000 mon.c (mon.2) 1301 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: cluster 2024-03-11T21:42:25.692353+0000 mon.a (mon.0) 3785 : cluster 0 osdmap e837: 8 total, 8 up, 8 in 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:25.693369+0000 mon.a (mon.0) 3786 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:26.548054+0000 mon.a (mon.0) 3787 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: cluster 2024-03-11T21:42:26.557281+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e838: 8 total, 8 up, 8 in 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:26.559462+0000 mon.c (mon.2) 1302 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[21512]: audit 2024-03-11T21:42:26.560857+0000 mon.a (mon.0) 3789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:26.860 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: cluster 2024-03-11T21:42:25.619879+0000 mgr.y (mgr.14550) 754 : cluster 0 pgmap v1296: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:25.685103+0000 mon.a (mon.0) 3784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:25.692182+0000 mon.c (mon.2) 1301 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: cluster 2024-03-11T21:42:25.692353+0000 mon.a (mon.0) 3785 : cluster 0 osdmap e837: 8 total, 8 up, 8 in 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:25.693369+0000 mon.a (mon.0) 3786 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:26.548054+0000 mon.a (mon.0) 3787 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: cluster 2024-03-11T21:42:26.557281+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e838: 8 total, 8 up, 8 in 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:26.559462+0000 mon.c (mon.2) 1302 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:26.861 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:26 smithi143 bash[29284]: audit 2024-03-11T21:42:26.560857+0000 mon.a (mon.0) 3789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:26.870 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:26.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:27.001 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: cluster 2024-03-11T21:42:25.619879+0000 mgr.y (mgr.14550) 754 : cluster 0 pgmap v1296: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:25.685103+0000 mon.a (mon.0) 3784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:25.692182+0000 mon.c (mon.2) 1301 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: cluster 2024-03-11T21:42:25.692353+0000 mon.a (mon.0) 3785 : cluster 0 osdmap e837: 8 total, 8 up, 8 in 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:25.693369+0000 mon.a (mon.0) 3786 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:26.548054+0000 mon.a (mon.0) 3787 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: cluster 2024-03-11T21:42:26.557281+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e838: 8 total, 8 up, 8 in 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:26.559462+0000 mon.c (mon.2) 1302 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:27.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:26 smithi203 bash[23654]: audit 2024-03-11T21:42:26.560857+0000 mon.a (mon.0) 3789 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:27.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:27.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:27.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:27.261 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:27.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:27.391 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:27.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:27.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:27.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:27.652 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:27.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:27.782 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:27.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:27.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:27 smithi203 bash[23654]: cluster 2024-03-11T21:42:27.548375+0000 mon.a (mon.0) 3790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:27.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:27 smithi203 bash[23654]: audit 2024-03-11T21:42:27.550087+0000 mon.a (mon.0) 3791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:27.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:27 smithi203 bash[23654]: cluster 2024-03-11T21:42:27.556178+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e839: 8 total, 8 up, 8 in 2024-03-11T21:42:27.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:27 smithi203 bash[23654]: audit 2024-03-11T21:42:27.557801+0000 mon.c (mon.2) 1303 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:27.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:27 smithi203 bash[23654]: audit 2024-03-11T21:42:27.558596+0000 mon.a (mon.0) 3793 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:27.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[21512]: cluster 2024-03-11T21:42:27.548375+0000 mon.a (mon.0) 3790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[21512]: audit 2024-03-11T21:42:27.550087+0000 mon.a (mon.0) 3791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[21512]: cluster 2024-03-11T21:42:27.556178+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e839: 8 total, 8 up, 8 in 2024-03-11T21:42:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[21512]: audit 2024-03-11T21:42:27.557801+0000 mon.c (mon.2) 1303 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:27.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[21512]: audit 2024-03-11T21:42:27.558596+0000 mon.a (mon.0) 3793 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[29284]: cluster 2024-03-11T21:42:27.548375+0000 mon.a (mon.0) 3790 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[29284]: audit 2024-03-11T21:42:27.550087+0000 mon.a (mon.0) 3791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:42:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[29284]: cluster 2024-03-11T21:42:27.556178+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e839: 8 total, 8 up, 8 in 2024-03-11T21:42:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[29284]: audit 2024-03-11T21:42:27.557801+0000 mon.c (mon.2) 1303 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:27.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:27 smithi143 bash[29284]: audit 2024-03-11T21:42:27.558596+0000 mon.a (mon.0) 3793 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:42:28.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.043 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:28.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.174 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:28.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:28.306 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:28.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.439 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:28.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:42:28.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:28.567 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:28.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:28.692 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:28.731 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1294,"stamp":"2024-03-11T21:42:23.665155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596646,"num_objects":236,"num_object_clones":1,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":2326,"num_read_kb":2066,"num_write":786,"num_write_kb":4011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2393,"ondisk_log_size":2393,"up":726,"acting":726,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":743,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":694440,"kb_used_data":116328,"kb_used_omap":38,"kb_used_meta":577817,"kb_avail":749266776,"statfs":{"total":767960285184,"available":767249178624,"internally_reserved":0,"allocated":119119872,"data_stored":77702710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":591685045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":160,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049027"},"pg_stats":[{"pgid":"329.1f","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712678+0000","last_change":"2024-03-11T21:42:21.666765+0000","last_active":"2024-03-11T21:42:22.712678+0000","last_peered":"2024-03-11T21:42:22.712678+0000","last_clean":"2024-03-11T21:42:22.712678+0000","last_became_active":"2024-03-11T21:42:21.666384+0000","last_became_peered":"2024-03-11T21:42:21.666384+0000","last_unstale":"2024-03-11T21:42:22.712678+0000","last_undegraded":"2024-03-11T21:42:22.712678+0000","last_fullsized":"2024-03-11T21:42:22.712678+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:27.538770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"329.1e","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713967+0000","last_change":"2024-03-11T21:42:21.658955+0000","last_active":"2024-03-11T21:42:22.713967+0000","last_peered":"2024-03-11T21:42:22.713967+0000","last_clean":"2024-03-11T21:42:22.713967+0000","last_became_active":"2024-03-11T21:42:21.658725+0000","last_became_peered":"2024-03-11T21:42:21.658725+0000","last_unstale":"2024-03-11T21:42:22.713967+0000","last_undegraded":"2024-03-11T21:42:22.713967+0000","last_fullsized":"2024-03-11T21:42:22.713967+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:22.331553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"329.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:42:20.606629+0000","last_change":"2024-03-11T21:42:20.606629+0000","last_active":"2024-03-11T21:42:20.606629+0000","last_peered":"2024-03-11T21:42:20.606629+0000","last_clean":"2024-03-11T21:42:20.606629+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:42:20.606629+0000","last_undegraded":"2024-03-11T21:42:20.606629+0000","last_fullsized":"2024-03-11T21:42:20.606629+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"329.1c","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712367+0000","last_change":"2024-03-11T21:42:21.650793+0000","last_active":"2024-03-11T21:42:22.712367+0000","last_peered":"2024-03-11T21:42:22.712367+0000","last_clean":"2024-03-11T21:42:22.712367+0000","last_became_active":"2024-03-11T21:42:21.650441+0000","last_became_peered":"2024-03-11T21:42:21.650441+0000","last_unstale":"2024-03-11T21:42:22.712367+0000","last_undegraded":"2024-03-11T21:42:22.712367+0000","last_fullsized":"2024-03-11T21:42:22.712367+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.602514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"329.1b","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713932+0000","last_change":"2024-03-11T21:42:21.666199+0000","last_active":"2024-03-11T21:42:22.713932+0000","last_peered":"2024-03-11T21:42:22.713932+0000","last_clean":"2024-03-11T21:42:22.713932+0000","last_became_active":"2024-03-11T21:42:21.665517+0000","last_became_peered":"2024-03-11T21:42:21.665517+0000","last_unstale":"2024-03-11T21:42:22.713932+0000","last_undegraded":"2024-03-11T21:42:22.713932+0000","last_fullsized":"2024-03-11T21:42:22.713932+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:50.589985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"329.1a","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.663490+0000","last_change":"2024-03-11T21:42:21.676354+0000","last_active":"2024-03-11T21:42:22.663490+0000","last_peered":"2024-03-11T21:42:22.663490+0000","last_clean":"2024-03-11T21:42:22.663490+0000","last_became_active":"2024-03-11T21:42:21.676039+0000","last_became_peered":"2024-03-11T21:42:21.676039+0000","last_unstale":"2024-03-11T21:42:22.663490+0000","last_undegraded":"2024-03-11T21:42:22.663490+0000","last_fullsized":"2024-03-11T21:42:22.663490+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:32.437612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"329.19","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708341+0000","last_change":"2024-03-11T21:42:21.666337+0000","last_active":"2024-03-11T21:42:22.708341+0000","last_peered":"2024-03-11T21:42:22.708341+0000","last_clean":"2024-03-11T21:42:22.708341+0000","last_became_active":"2024-03-11T21:42:21.665601+0000","last_became_peered":"2024-03-11T21:42:21.665601+0000","last_unstale":"2024-03-11T21:42:22.708341+0000","last_undegraded":"2024-03-11T21:42:22.708341+0000","last_fullsized":"2024-03-11T21:42:22.708341+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:38.585441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"329.18","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708169+0000","last_change":"2024-03-11T21:42:21.660939+0000","last_active":"2024-03-11T21:42:22.708169+0000","last_peered":"2024-03-11T21:42:22.708169+0000","last_clean":"2024-03-11T21:42:22.708169+0000","last_became_active":"2024-03-11T21:42:21.660472+0000","last_became_peered":"2024-03-11T21:42:21.660472+0000","last_unstale":"2024-03-11T21:42:22.708169+0000","last_undegraded":"2024-03-11T21:42:22.708169+0000","last_fullsized":"2024-03-11T21:42:22.708169+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:34.504300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"329.17","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705635+0000","last_change":"2024-03-11T21:42:21.676425+0000","last_active":"2024-03-11T21:42:22.705635+0000","last_peered":"2024-03-11T21:42:22.705635+0000","last_clean":"2024-03-11T21:42:22.705635+0000","last_became_active":"2024-03-11T21:42:21.669360+0000","last_became_peered":"2024-03-11T21:42:21.669360+0000","last_unstale":"2024-03-11T21:42:22.705635+0000","last_undegraded":"2024-03-11T21:42:22.705635+0000","last_fullsized":"2024-03-11T21:42:22.705635+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:50.396263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"329.16","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658663+0000","last_change":"2024-03-11T21:42:21.647705+0000","last_active":"2024-03-11T21:42:22.658663+0000","last_peered":"2024-03-11T21:42:22.658663+0000","last_clean":"2024-03-11T21:42:22.658663+0000","last_became_active":"2024-03-11T21:42:21.647381+0000","last_became_peered":"2024-03-11T21:42:21.647381+0000","last_unstale":"2024-03-11T21:42:22.658663+0000","last_undegraded":"2024-03-11T21:42:22.658663+0000","last_fullsized":"2024-03-11T21:42:22.658663+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:36.296962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"329.15","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705945+0000","last_change":"2024-03-11T21:42:21.676500+0000","last_active":"2024-03-11T21:42:22.705945+0000","last_peered":"2024-03-11T21:42:22.705945+0000","last_clean":"2024-03-11T21:42:22.705945+0000","last_became_active":"2024-03-11T21:42:21.675762+0000","last_became_peered":"2024-03-11T21:42:21.675762+0000","last_unstale":"2024-03-11T21:42:22.705945+0000","last_undegraded":"2024-03-11T21:42:22.705945+0000","last_fullsized":"2024-03-11T21:42:22.705945+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:34.601023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"329.14","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.672208+0000","last_change":"2024-03-11T21:42:21.681716+0000","last_active":"2024-03-11T21:42:22.672208+0000","last_peered":"2024-03-11T21:42:22.672208+0000","last_clean":"2024-03-11T21:42:22.672208+0000","last_became_active":"2024-03-11T21:42:21.681263+0000","last_became_peered":"2024-03-11T21:42:21.681263+0000","last_unstale":"2024-03-11T21:42:22.672208+0000","last_undegraded":"2024-03-11T21:42:22.672208+0000","last_fullsized":"2024-03-11T21:42:22.672208+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:34.863824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"329.13","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.668298+0000","last_change":"2024-03-11T21:42:21.677163+0000","last_active":"2024-03-11T21:42:22.668298+0000","last_peered":"2024-03-11T21:42:22.668298+0000","last_clean":"2024-03-11T21:42:22.668298+0000","last_became_active":"2024-03-11T21:42:21.676828+0000","last_became_peered":"2024-03-11T21:42:21.676828+0000","last_unstale":"2024-03-11T21:42:22.668298+0000","last_undegraded":"2024-03-11T21:42:22.668298+0000","last_fullsized":"2024-03-11T21:42:22.668298+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:39.006490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"329.12","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708496+0000","last_change":"2024-03-11T21:42:21.667790+0000","last_active":"2024-03-11T21:42:22.708496+0000","last_peered":"2024-03-11T21:42:22.708496+0000","last_clean":"2024-03-11T21:42:22.708496+0000","last_became_active":"2024-03-11T21:42:21.666900+0000","last_became_peered":"2024-03-11T21:42:21.666900+0000","last_unstale":"2024-03-11T21:42:22.708496+0000","last_undegraded":"2024-03-11T21:42:22.708496+0000","last_fullsized":"2024-03-11T21:42:22.708496+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.426278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"329.11","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708543+0000","last_change":"2024-03-11T21:42:21.655693+0000","last_active":"2024-03-11T21:42:22.708543+0000","last_peered":"2024-03-11T21:42:22.708543+0000","last_clean":"2024-03-11T21:42:22.708543+0000","last_became_active":"2024-03-11T21:42:21.655361+0000","last_became_peered":"2024-03-11T21:42:21.655361+0000","last_unstale":"2024-03-11T21:42:22.708543+0000","last_undegraded":"2024-03-11T21:42:22.708543+0000","last_fullsized":"2024-03-11T21:42:22.708543+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:31.839839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"329.10","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709557+0000","last_change":"2024-03-11T21:42:21.654830+0000","last_active":"2024-03-11T21:42:22.709557+0000","last_peered":"2024-03-11T21:42:22.709557+0000","last_clean":"2024-03-11T21:42:22.709557+0000","last_became_active":"2024-03-11T21:42:21.654498+0000","last_became_peered":"2024-03-11T21:42:21.654498+0000","last_unstale":"2024-03-11T21:42:22.709557+0000","last_undegraded":"2024-03-11T21:42:22.709557+0000","last_fullsized":"2024-03-11T21:42:22.709557+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.287400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"329.f","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706579+0000","last_change":"2024-03-11T21:42:21.668162+0000","last_active":"2024-03-11T21:42:22.706579+0000","last_peered":"2024-03-11T21:42:22.706579+0000","last_clean":"2024-03-11T21:42:22.706579+0000","last_became_active":"2024-03-11T21:42:21.666288+0000","last_became_peered":"2024-03-11T21:42:21.666288+0000","last_unstale":"2024-03-11T21:42:22.706579+0000","last_undegraded":"2024-03-11T21:42:22.706579+0000","last_fullsized":"2024-03-11T21:42:22.706579+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.165726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"329.e","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710596+0000","last_change":"2024-03-11T21:42:21.655700+0000","last_active":"2024-03-11T21:42:22.710596+0000","last_peered":"2024-03-11T21:42:22.710596+0000","last_clean":"2024-03-11T21:42:22.710596+0000","last_became_active":"2024-03-11T21:42:21.654617+0000","last_became_peered":"2024-03-11T21:42:21.654617+0000","last_unstale":"2024-03-11T21:42:22.710596+0000","last_undegraded":"2024-03-11T21:42:22.710596+0000","last_fullsized":"2024-03-11T21:42:22.710596+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:27.925177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"329.d","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707334+0000","last_change":"2024-03-11T21:42:21.677403+0000","last_active":"2024-03-11T21:42:22.707334+0000","last_peered":"2024-03-11T21:42:22.707334+0000","last_clean":"2024-03-11T21:42:22.707334+0000","last_became_active":"2024-03-11T21:42:21.676984+0000","last_became_peered":"2024-03-11T21:42:21.676984+0000","last_unstale":"2024-03-11T21:42:22.707334+0000","last_undegraded":"2024-03-11T21:42:22.707334+0000","last_fullsized":"2024-03-11T21:42:22.707334+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:32.438591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"329.c","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704419+0000","last_change":"2024-03-11T21:42:21.672035+0000","last_active":"2024-03-11T21:42:22.704419+0000","last_peered":"2024-03-11T21:42:22.704419+0000","last_clean":"2024-03-11T21:42:22.704419+0000","last_became_active":"2024-03-11T21:42:21.671588+0000","last_became_peered":"2024-03-11T21:42:21.671588+0000","last_unstale":"2024-03-11T21:42:22.704419+0000","last_undegraded":"2024-03-11T21:42:22.704419+0000","last_fullsized":"2024-03-11T21:42:22.704419+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:22.910782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"329.b","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706938+0000","last_change":"2024-03-11T21:42:21.673939+0000","last_active":"2024-03-11T21:42:22.706938+0000","last_peered":"2024-03-11T21:42:22.706938+0000","last_clean":"2024-03-11T21:42:22.706938+0000","last_became_active":"2024-03-11T21:42:21.673435+0000","last_became_peered":"2024-03-11T21:42:21.673435+0000","last_unstale":"2024-03-11T21:42:22.706938+0000","last_undegraded":"2024-03-11T21:42:22.706938+0000","last_fullsized":"2024-03-11T21:42:22.706938+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.306069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"329.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:42:20.606629+0000","last_change":"2024-03-11T21:42:20.606629+0000","last_active":"2024-03-11T21:42:20.606629+0000","last_peered":"2024-03-11T21:42:20.606629+0000","last_clean":"2024-03-11T21:42:20.606629+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:42:20.606629+0000","last_undegraded":"2024-03-11T21:42:20.606629+0000","last_fullsized":"2024-03-11T21:42:20.606629+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"329.9","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708885+0000","last_change":"2024-03-11T21:42:21.652305+0000","last_active":"2024-03-11T21:42:22.708885+0000","last_peered":"2024-03-11T21:42:22.708885+0000","last_clean":"2024-03-11T21:42:22.708885+0000","last_became_active":"2024-03-11T21:42:21.651822+0000","last_became_peered":"2024-03-11T21:42:21.651822+0000","last_unstale":"2024-03-11T21:42:22.708885+0000","last_undegraded":"2024-03-11T21:42:22.708885+0000","last_fullsized":"2024-03-11T21:42:22.708885+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.534615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"329.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:42:20.606629+0000","last_change":"2024-03-11T21:42:20.606629+0000","last_active":"2024-03-11T21:42:20.606629+0000","last_peered":"2024-03-11T21:42:20.606629+0000","last_clean":"2024-03-11T21:42:20.606629+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:42:20.606629+0000","last_undegraded":"2024-03-11T21:42:20.606629+0000","last_fullsized":"2024-03-11T21:42:20.606629+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"329.7","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671273+0000","last_change":"2024-03-11T21:42:21.665977+0000","last_active":"2024-03-11T21:42:22.671273+0000","last_peered":"2024-03-11T21:42:22.671273+0000","last_clean":"2024-03-11T21:42:22.671273+0000","last_became_active":"2024-03-11T21:42:21.665664+0000","last_became_peered":"2024-03-11T21:42:21.665664+0000","last_unstale":"2024-03-11T21:42:22.671273+0000","last_undegraded":"2024-03-11T21:42:22.671273+0000","last_fullsized":"2024-03-11T21:42:22.671273+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:32.379255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"329.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-11T21:42:20.606629+0000","last_change":"2024-03-11T21:42:20.606629+0000","last_active":"2024-03-11T21:42:20.606629+0000","last_peered":"2024-03-11T21:42:20.606629+0000","last_clean":"2024-03-11T21:42:20.606629+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-11T21:42:20.606629+0000","last_undegraded":"2024-03-11T21:42:20.606629+0000","last_fullsized":"2024-03-11T21:42:20.606629+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"329.5","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669898+0000","last_change":"2024-03-11T21:42:21.667995+0000","last_active":"2024-03-11T21:42:22.669898+0000","last_peered":"2024-03-11T21:42:22.669898+0000","last_clean":"2024-03-11T21:42:22.669898+0000","last_became_active":"2024-03-11T21:42:21.667611+0000","last_became_peered":"2024-03-11T21:42:21.667611+0000","last_unstale":"2024-03-11T21:42:22.669898+0000","last_undegraded":"2024-03-11T21:42:22.669898+0000","last_fullsized":"2024-03-11T21:42:22.669898+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:38.851824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"329.4","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709200+0000","last_change":"2024-03-11T21:42:21.667851+0000","last_active":"2024-03-11T21:42:22.709200+0000","last_peered":"2024-03-11T21:42:22.709200+0000","last_clean":"2024-03-11T21:42:22.709200+0000","last_became_active":"2024-03-11T21:42:21.667267+0000","last_became_peered":"2024-03-11T21:42:21.667267+0000","last_unstale":"2024-03-11T21:42:22.709200+0000","last_undegraded":"2024-03-11T21:42:22.709200+0000","last_fullsized":"2024-03-11T21:42:22.709200+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:26.672229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"329.3","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.657615+0000","last_change":"2024-03-11T21:42:21.658328+0000","last_active":"2024-03-11T21:42:22.657615+0000","last_peered":"2024-03-11T21:42:22.657615+0000","last_clean":"2024-03-11T21:42:22.657615+0000","last_became_active":"2024-03-11T21:42:21.657960+0000","last_became_peered":"2024-03-11T21:42:21.657960+0000","last_unstale":"2024-03-11T21:42:22.657615+0000","last_undegraded":"2024-03-11T21:42:22.657615+0000","last_fullsized":"2024-03-11T21:42:22.657615+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:50.074326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"329.2","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703729+0000","last_change":"2024-03-11T21:42:21.663084+0000","last_active":"2024-03-11T21:42:22.703729+0000","last_peered":"2024-03-11T21:42:22.703729+0000","last_clean":"2024-03-11T21:42:22.703729+0000","last_became_active":"2024-03-11T21:42:21.662598+0000","last_became_peered":"2024-03-11T21:42:21.662598+0000","last_unstale":"2024-03-11T21:42:22.703729+0000","last_undegraded":"2024-03-11T21:42:22.703729+0000","last_fullsized":"2024-03-11T21:42:22.703729+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:50.068487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"329.1","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703842+0000","last_change":"2024-03-11T21:42:21.675305+0000","last_active":"2024-03-11T21:42:22.703842+0000","last_peered":"2024-03-11T21:42:22.703842+0000","last_clean":"2024-03-11T21:42:22.703842+0000","last_became_active":"2024-03-11T21:42:21.674821+0000","last_became_peered":"2024-03-11T21:42:21.674821+0000","last_unstale":"2024-03-11T21:42:22.703842+0000","last_undegraded":"2024-03-11T21:42:22.703842+0000","last_fullsized":"2024-03-11T21:42:22.703842+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:44.100142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"329.0","version":"0'0","reported_seq":13,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709648+0000","last_change":"2024-03-11T21:42:21.662798+0000","last_active":"2024-03-11T21:42:22.709648+0000","last_peered":"2024-03-11T21:42:22.709648+0000","last_clean":"2024-03-11T21:42:22.709648+0000","last_became_active":"2024-03-11T21:42:21.662475+0000","last_became_peered":"2024-03-11T21:42:21.662475+0000","last_unstale":"2024-03-11T21:42:22.709648+0000","last_undegraded":"2024-03-11T21:42:22.709648+0000","last_fullsized":"2024-03-11T21:42:22.709648+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":832,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:20.606629+0000","last_clean_scrub_stamp":"2024-03-11T21:42:20.606629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:47.651510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.7","version":"833'7","reported_seq":397,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705804+0000","last_change":"2024-03-11T21:41:30.605842+0000","last_active":"2024-03-11T21:42:22.705804+0000","last_peered":"2024-03-11T21:42:22.705804+0000","last_clean":"2024-03-11T21:42:22.705804+0000","last_became_active":"2024-03-11T21:39:36.667532+0000","last_became_peered":"2024-03-11T21:39:36.667532+0000","last_unstale":"2024-03-11T21:42:22.705804+0000","last_undegraded":"2024-03-11T21:42:22.705804+0000","last_fullsized":"2024-03-11T21:42:22.705804+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'5","last_scrub_stamp":"2024-03-11T21:41:21.907642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.907642+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039920429999999998,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.6","version":"833'7","reported_seq":394,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715194+0000","last_change":"2024-03-11T21:41:54.496117+0000","last_active":"2024-03-11T21:42:22.715194+0000","last_peered":"2024-03-11T21:42:22.715194+0000","last_clean":"2024-03-11T21:42:22.715194+0000","last_became_active":"2024-03-11T21:39:36.664544+0000","last_became_peered":"2024-03-11T21:39:36.664544+0000","last_unstale":"2024-03-11T21:42:22.715194+0000","last_undegraded":"2024-03-11T21:42:22.715194+0000","last_fullsized":"2024-03-11T21:42:22.715194+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'5","last_scrub_stamp":"2024-03-11T21:41:54.495843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.495843+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.388867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012127220000000001,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.5","version":"833'5","reported_seq":397,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712894+0000","last_change":"2024-03-11T21:41:30.632295+0000","last_active":"2024-03-11T21:42:22.712894+0000","last_peered":"2024-03-11T21:42:22.712894+0000","last_clean":"2024-03-11T21:42:22.712894+0000","last_became_active":"2024-03-11T21:39:36.664433+0000","last_became_peered":"2024-03-11T21:39:36.664433+0000","last_unstale":"2024-03-11T21:42:22.712894+0000","last_undegraded":"2024-03-11T21:42:22.712894+0000","last_fullsized":"2024-03-11T21:42:22.712894+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'3","last_scrub_stamp":"2024-03-11T21:41:14.644215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.644215+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018642859999999999,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.4","version":"833'19","reported_seq":403,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669477+0000","last_change":"2024-03-11T21:41:30.618704+0000","last_active":"2024-03-11T21:42:22.669477+0000","last_peered":"2024-03-11T21:42:22.669477+0000","last_clean":"2024-03-11T21:42:22.669477+0000","last_became_active":"2024-03-11T21:39:36.675026+0000","last_became_peered":"2024-03-11T21:39:36.675026+0000","last_unstale":"2024-03-11T21:42:22.669477+0000","last_undegraded":"2024-03-11T21:42:22.669477+0000","last_fullsized":"2024-03-11T21:42:22.669477+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:39:35.496133+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031897900000000001,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.3","version":"833'13","reported_seq":404,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705478+0000","last_change":"2024-03-11T21:41:30.606913+0000","last_active":"2024-03-11T21:42:22.705478+0000","last_peered":"2024-03-11T21:42:22.705478+0000","last_clean":"2024-03-11T21:42:22.705478+0000","last_became_active":"2024-03-11T21:39:36.653164+0000","last_became_peered":"2024-03-11T21:39:36.653164+0000","last_unstale":"2024-03-11T21:42:22.705478+0000","last_undegraded":"2024-03-11T21:42:22.705478+0000","last_fullsized":"2024-03-11T21:42:22.705478+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7","last_scrub_stamp":"2024-03-11T21:41:17.885818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.885818+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.497480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0050295210000000003,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.2","version":"833'9","reported_seq":399,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703643+0000","last_change":"2024-03-11T21:41:30.609679+0000","last_active":"2024-03-11T21:42:22.703643+0000","last_peered":"2024-03-11T21:42:22.703643+0000","last_clean":"2024-03-11T21:42:22.703643+0000","last_became_active":"2024-03-11T21:39:36.673314+0000","last_became_peered":"2024-03-11T21:39:36.673314+0000","last_unstale":"2024-03-11T21:42:22.703643+0000","last_undegraded":"2024-03-11T21:42:22.703643+0000","last_fullsized":"2024-03-11T21:42:22.703643+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:17.477919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.477919+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.256319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015195479999999999,"stat_sum":{"num_bytes":4000033,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.1","version":"797'22","reported_seq":410,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708285+0000","last_change":"2024-03-11T21:41:59.370755+0000","last_active":"2024-03-11T21:42:22.708285+0000","last_peered":"2024-03-11T21:42:22.708285+0000","last_clean":"2024-03-11T21:42:22.708285+0000","last_became_active":"2024-03-11T21:39:36.651751+0000","last_became_peered":"2024-03-11T21:39:36.651751+0000","last_unstale":"2024-03-11T21:42:22.708285+0000","last_undegraded":"2024-03-11T21:42:22.708285+0000","last_fullsized":"2024-03-11T21:42:22.708285+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'22","last_scrub_stamp":"2024-03-11T21:41:59.370508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.370508+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.077709+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.006595956,"stat_sum":{"num_bytes":15,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"313.0","version":"833'8","reported_seq":394,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708193+0000","last_change":"2024-03-11T21:41:30.603475+0000","last_active":"2024-03-11T21:42:22.708193+0000","last_peered":"2024-03-11T21:42:22.708193+0000","last_clean":"2024-03-11T21:42:22.708193+0000","last_became_active":"2024-03-11T21:39:36.661261+0000","last_became_peered":"2024-03-11T21:39:36.661261+0000","last_unstale":"2024-03-11T21:42:22.708193+0000","last_undegraded":"2024-03-11T21:42:22.708193+0000","last_fullsized":"2024-03-11T21:42:22.708193+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:16.318277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.318277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.971390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019677119999999999,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":1414,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661054+0000","last_change":"2024-03-11T21:38:58.856206+0000","last_active":"2024-03-11T21:42:22.661054+0000","last_peered":"2024-03-11T21:42:22.661054+0000","last_clean":"2024-03-11T21:42:22.661054+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:42:22.661054+0000","last_undegraded":"2024-03-11T21:42:22.661054+0000","last_fullsized":"2024-03-11T21:42:22.661054+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:58.856090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:58.856090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1434,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.702732+0000","last_change":"2024-03-11T21:41:29.853017+0000","last_active":"2024-03-11T21:42:22.702732+0000","last_peered":"2024-03-11T21:42:22.702732+0000","last_clean":"2024-03-11T21:42:22.702732+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:42:22.702732+0000","last_undegraded":"2024-03-11T21:42:22.702732+0000","last_fullsized":"2024-03-11T21:42:22.702732+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.852782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.852782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:55.425301+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1418,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708714+0000","last_change":"2024-03-11T21:41:57.330251+0000","last_active":"2024-03-11T21:42:22.708714+0000","last_peered":"2024-03-11T21:42:22.708714+0000","last_clean":"2024-03-11T21:42:22.708714+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:42:22.708714+0000","last_undegraded":"2024-03-11T21:42:22.708714+0000","last_fullsized":"2024-03-11T21:42:22.708714+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.330037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.330037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.606411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":1405,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.668510+0000","last_change":"2024-03-11T21:40:25.470377+0000","last_active":"2024-03-11T21:42:22.668510+0000","last_peered":"2024-03-11T21:42:22.668510+0000","last_clean":"2024-03-11T21:42:22.668510+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:42:22.668510+0000","last_undegraded":"2024-03-11T21:42:22.668510+0000","last_fullsized":"2024-03-11T21:42:22.668510+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.470230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.470230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.720087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":1328,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616224+0000","last_change":"2024-03-11T21:41:56.250851+0000","last_active":"2024-03-11T21:42:19.616224+0000","last_peered":"2024-03-11T21:42:19.616224+0000","last_clean":"2024-03-11T21:42:19.616224+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:42:19.616224+0000","last_undegraded":"2024-03-11T21:42:19.616224+0000","last_fullsized":"2024-03-11T21:42:19.616224+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.250744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.250744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:01.111142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1434,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705269+0000","last_change":"2024-03-11T21:40:28.770797+0000","last_active":"2024-03-11T21:42:22.705269+0000","last_peered":"2024-03-11T21:42:22.705269+0000","last_clean":"2024-03-11T21:42:22.705269+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:42:22.705269+0000","last_undegraded":"2024-03-11T21:42:22.705269+0000","last_fullsized":"2024-03-11T21:42:22.705269+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.770668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.376452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1435,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705506+0000","last_change":"2024-03-11T21:41:42.681742+0000","last_active":"2024-03-11T21:42:22.705506+0000","last_peered":"2024-03-11T21:42:22.705506+0000","last_clean":"2024-03-11T21:42:22.705506+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:42:22.705506+0000","last_undegraded":"2024-03-11T21:42:22.705506+0000","last_fullsized":"2024-03-11T21:42:22.705506+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.681620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.645321+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.681620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.544804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1448,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714591+0000","last_change":"2024-03-11T21:41:35.844756+0000","last_active":"2024-03-11T21:42:22.714591+0000","last_peered":"2024-03-11T21:42:22.714591+0000","last_clean":"2024-03-11T21:42:22.714591+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:42:22.714591+0000","last_undegraded":"2024-03-11T21:42:22.714591+0000","last_fullsized":"2024-03-11T21:42:22.714591+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.844652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.844652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.557477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1434,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.656327+0000","last_change":"2024-03-11T21:41:31.690755+0000","last_active":"2024-03-11T21:42:22.656327+0000","last_peered":"2024-03-11T21:42:22.656327+0000","last_clean":"2024-03-11T21:42:22.656327+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:42:22.656327+0000","last_undegraded":"2024-03-11T21:42:22.656327+0000","last_fullsized":"2024-03-11T21:42:22.656327+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.690559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.690559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.787294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1413,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661044+0000","last_change":"2024-03-11T21:41:41.521503+0000","last_active":"2024-03-11T21:42:22.661044+0000","last_peered":"2024-03-11T21:42:22.661044+0000","last_clean":"2024-03-11T21:42:22.661044+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:42:22.661044+0000","last_undegraded":"2024-03-11T21:42:22.661044+0000","last_fullsized":"2024-03-11T21:42:22.661044+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.521300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.521300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.506707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1416,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712117+0000","last_change":"2024-03-11T21:40:13.532111+0000","last_active":"2024-03-11T21:42:22.712117+0000","last_peered":"2024-03-11T21:42:22.712117+0000","last_clean":"2024-03-11T21:42:22.712117+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:42:22.712117+0000","last_undegraded":"2024-03-11T21:42:22.712117+0000","last_fullsized":"2024-03-11T21:42:22.712117+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.532000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":1406,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705404+0000","last_change":"2024-03-11T21:41:59.422739+0000","last_active":"2024-03-11T21:42:22.705404+0000","last_peered":"2024-03-11T21:42:22.705404+0000","last_clean":"2024-03-11T21:42:22.705404+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:42:22.705404+0000","last_undegraded":"2024-03-11T21:42:22.705404+0000","last_fullsized":"2024-03-11T21:42:22.705404+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.422534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.422534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.121392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1416,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712609+0000","last_change":"2024-03-11T21:40:27.533904+0000","last_active":"2024-03-11T21:42:22.712609+0000","last_peered":"2024-03-11T21:42:22.712609+0000","last_clean":"2024-03-11T21:42:22.712609+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:42:22.712609+0000","last_undegraded":"2024-03-11T21:42:22.712609+0000","last_fullsized":"2024-03-11T21:42:22.712609+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.533778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.533778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1434,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659916+0000","last_change":"2024-03-11T21:41:45.595342+0000","last_active":"2024-03-11T21:42:22.659916+0000","last_peered":"2024-03-11T21:42:22.659916+0000","last_clean":"2024-03-11T21:42:22.659916+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:42:22.659916+0000","last_undegraded":"2024-03-11T21:42:22.659916+0000","last_fullsized":"2024-03-11T21:42:22.659916+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.595129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.595129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.837904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1447,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712915+0000","last_change":"2024-03-11T21:41:39.829836+0000","last_active":"2024-03-11T21:42:22.712915+0000","last_peered":"2024-03-11T21:42:22.712915+0000","last_clean":"2024-03-11T21:42:22.712915+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:42:22.712915+0000","last_undegraded":"2024-03-11T21:42:22.712915+0000","last_fullsized":"2024-03-11T21:42:22.712915+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:39.829557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:39.829557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.135654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":824,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.623127+0000","last_change":"2024-03-11T21:41:20.464811+0000","last_active":"2024-03-11T21:42:19.623127+0000","last_peered":"2024-03-11T21:42:19.623127+0000","last_clean":"2024-03-11T21:42:19.623127+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:42:19.623127+0000","last_undegraded":"2024-03-11T21:42:19.623127+0000","last_fullsized":"2024-03-11T21:42:19.623127+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.464589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.464589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.019825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1435,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.702660+0000","last_change":"2024-03-11T21:41:18.848489+0000","last_active":"2024-03-11T21:42:22.702660+0000","last_peered":"2024-03-11T21:42:22.702660+0000","last_clean":"2024-03-11T21:42:22.702660+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:42:22.702660+0000","last_undegraded":"2024-03-11T21:42:22.702660+0000","last_fullsized":"2024-03-11T21:42:22.702660+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.848394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.848394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.484135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":1413,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703310+0000","last_change":"2024-03-11T21:41:51.387114+0000","last_active":"2024-03-11T21:42:22.703310+0000","last_peered":"2024-03-11T21:42:22.703310+0000","last_clean":"2024-03-11T21:42:22.703310+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:42:22.703310+0000","last_undegraded":"2024-03-11T21:42:22.703310+0000","last_fullsized":"2024-03-11T21:42:22.703310+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.386905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.386905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.204956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":1406,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.662003+0000","last_change":"2024-03-11T21:40:19.428421+0000","last_active":"2024-03-11T21:42:22.662003+0000","last_peered":"2024-03-11T21:42:22.662003+0000","last_clean":"2024-03-11T21:42:22.662003+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:42:22.662003+0000","last_undegraded":"2024-03-11T21:42:22.662003+0000","last_fullsized":"2024-03-11T21:42:22.662003+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.428205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.428205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.720051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1435,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658635+0000","last_change":"2024-03-11T21:41:40.664787+0000","last_active":"2024-03-11T21:42:22.658635+0000","last_peered":"2024-03-11T21:42:22.658635+0000","last_clean":"2024-03-11T21:42:22.658635+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:42:22.658635+0000","last_undegraded":"2024-03-11T21:42:22.658635+0000","last_fullsized":"2024-03-11T21:42:22.658635+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:40.664675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.664675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.324510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":1296,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616513+0000","last_change":"2024-03-11T21:41:32.440218+0000","last_active":"2024-03-11T21:42:19.616513+0000","last_peered":"2024-03-11T21:42:19.616513+0000","last_clean":"2024-03-11T21:42:19.616513+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:42:19.616513+0000","last_undegraded":"2024-03-11T21:42:19.616513+0000","last_fullsized":"2024-03-11T21:42:19.616513+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.440004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.440004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.246313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":1406,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.670632+0000","last_change":"2024-03-11T21:38:45.918826+0000","last_active":"2024-03-11T21:42:22.670632+0000","last_peered":"2024-03-11T21:42:22.670632+0000","last_clean":"2024-03-11T21:42:22.670632+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:42:22.670632+0000","last_undegraded":"2024-03-11T21:42:22.670632+0000","last_fullsized":"2024-03-11T21:42:22.670632+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:45.918744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:45.918744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1414,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.670758+0000","last_change":"2024-03-11T21:38:59.900843+0000","last_active":"2024-03-11T21:42:22.670758+0000","last_peered":"2024-03-11T21:42:22.670758+0000","last_clean":"2024-03-11T21:42:22.670758+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:42:22.670758+0000","last_undegraded":"2024-03-11T21:42:22.670758+0000","last_fullsized":"2024-03-11T21:42:22.670758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:59.900747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:59.900747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1417,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716443+0000","last_change":"2024-03-11T21:40:26.557225+0000","last_active":"2024-03-11T21:42:22.716443+0000","last_peered":"2024-03-11T21:42:22.716443+0000","last_clean":"2024-03-11T21:42:22.716443+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:42:22.716443+0000","last_undegraded":"2024-03-11T21:42:22.716443+0000","last_fullsized":"2024-03-11T21:42:22.716443+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.557127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.557127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.536660+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1411,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708503+0000","last_change":"2024-03-11T21:40:02.769849+0000","last_active":"2024-03-11T21:42:22.708503+0000","last_peered":"2024-03-11T21:42:22.708503+0000","last_clean":"2024-03-11T21:42:22.708503+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:42:22.708503+0000","last_undegraded":"2024-03-11T21:42:22.708503+0000","last_fullsized":"2024-03-11T21:42:22.708503+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.769628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.769628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":1399,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703436+0000","last_change":"2024-03-11T21:39:52.904395+0000","last_active":"2024-03-11T21:42:22.703436+0000","last_peered":"2024-03-11T21:42:22.703436+0000","last_clean":"2024-03-11T21:42:22.703436+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:42:22.703436+0000","last_undegraded":"2024-03-11T21:42:22.703436+0000","last_fullsized":"2024-03-11T21:42:22.703436+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:52.904287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:39:52.904287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":825,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616883+0000","last_change":"2024-03-11T21:40:21.384992+0000","last_active":"2024-03-11T21:42:19.616883+0000","last_peered":"2024-03-11T21:42:19.616883+0000","last_clean":"2024-03-11T21:42:19.616883+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:42:19.616883+0000","last_undegraded":"2024-03-11T21:42:19.616883+0000","last_fullsized":"2024-03-11T21:42:19.616883+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.384763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1435,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658453+0000","last_change":"2024-03-11T21:41:51.650830+0000","last_active":"2024-03-11T21:42:22.658453+0000","last_peered":"2024-03-11T21:42:22.658453+0000","last_clean":"2024-03-11T21:42:22.658453+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:42:22.658453+0000","last_undegraded":"2024-03-11T21:42:22.658453+0000","last_fullsized":"2024-03-11T21:42:22.658453+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.650722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.650722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.155813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1417,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712932+0000","last_change":"2024-03-11T21:41:17.359280+0000","last_active":"2024-03-11T21:42:22.712932+0000","last_peered":"2024-03-11T21:42:22.712932+0000","last_clean":"2024-03-11T21:42:22.712932+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:42:22.712932+0000","last_undegraded":"2024-03-11T21:42:22.712932+0000","last_fullsized":"2024-03-11T21:42:22.712932+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.628243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":1399,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706110+0000","last_change":"2024-03-11T21:38:46.791275+0000","last_active":"2024-03-11T21:42:22.706110+0000","last_peered":"2024-03-11T21:42:22.706110+0000","last_clean":"2024-03-11T21:42:22.706110+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:42:22.706110+0000","last_undegraded":"2024-03-11T21:42:22.706110+0000","last_fullsized":"2024-03-11T21:42:22.706110+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:46.791177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:38:46.791177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1448,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713198+0000","last_change":"2024-03-11T21:41:54.862852+0000","last_active":"2024-03-11T21:42:22.713198+0000","last_peered":"2024-03-11T21:42:22.713198+0000","last_clean":"2024-03-11T21:42:22.713198+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:42:22.713198+0000","last_undegraded":"2024-03-11T21:42:22.713198+0000","last_fullsized":"2024-03-11T21:42:22.713198+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.862666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.862666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.186701+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":1296,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.625463+0000","last_change":"2024-03-11T21:41:30.411743+0000","last_active":"2024-03-11T21:42:19.625463+0000","last_peered":"2024-03-11T21:42:19.625463+0000","last_clean":"2024-03-11T21:42:19.625463+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:42:19.625463+0000","last_undegraded":"2024-03-11T21:42:19.625463+0000","last_fullsized":"2024-03-11T21:42:19.625463+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:30.411631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.411631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:30.767399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1423,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715419+0000","last_change":"2024-03-11T21:40:09.512570+0000","last_active":"2024-03-11T21:42:22.715419+0000","last_peered":"2024-03-11T21:42:22.715419+0000","last_clean":"2024-03-11T21:42:22.715419+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:42:22.715419+0000","last_undegraded":"2024-03-11T21:42:22.715419+0000","last_fullsized":"2024-03-11T21:42:22.715419+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.512296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.512296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1800,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707499+0000","last_change":"2024-03-11T21:41:21.484873+0000","last_active":"2024-03-11T21:42:22.707499+0000","last_peered":"2024-03-11T21:42:22.707499+0000","last_clean":"2024-03-11T21:42:22.707499+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:42:22.707499+0000","last_undegraded":"2024-03-11T21:42:22.707499+0000","last_fullsized":"2024-03-11T21:42:22.707499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:21.484757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.484757+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.841600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1605,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706927+0000","last_change":"2024-03-11T21:41:26.485912+0000","last_active":"2024-03-11T21:42:22.706927+0000","last_peered":"2024-03-11T21:42:22.706927+0000","last_clean":"2024-03-11T21:42:22.706927+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:42:22.706927+0000","last_undegraded":"2024-03-11T21:42:22.706927+0000","last_fullsized":"2024-03-11T21:42:22.706927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:41:26.485646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.485646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.594156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1615,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714145+0000","last_change":"2024-03-11T21:41:42.811010+0000","last_active":"2024-03-11T21:42:22.714145+0000","last_peered":"2024-03-11T21:42:22.714145+0000","last_clean":"2024-03-11T21:42:22.714145+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:42:22.714145+0000","last_undegraded":"2024-03-11T21:42:22.714145+0000","last_fullsized":"2024-03-11T21:42:22.714145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.810777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.810777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.272393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1567,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709008+0000","last_change":"2024-03-11T21:41:45.252015+0000","last_active":"2024-03-11T21:42:22.709008+0000","last_peered":"2024-03-11T21:42:22.709008+0000","last_clean":"2024-03-11T21:42:22.709008+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:42:22.709008+0000","last_undegraded":"2024-03-11T21:42:22.709008+0000","last_fullsized":"2024-03-11T21:42:22.709008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.251730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.251730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.958525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1426,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710080+0000","last_change":"2024-03-11T21:40:14.758296+0000","last_active":"2024-03-11T21:42:22.710080+0000","last_peered":"2024-03-11T21:42:22.710080+0000","last_clean":"2024-03-11T21:42:22.710080+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:42:22.710080+0000","last_undegraded":"2024-03-11T21:42:22.710080+0000","last_fullsized":"2024-03-11T21:42:22.710080+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:14.758086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.758086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1605,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709827+0000","last_change":"2024-03-11T21:41:26.303821+0000","last_active":"2024-03-11T21:42:22.709827+0000","last_peered":"2024-03-11T21:42:22.709827+0000","last_clean":"2024-03-11T21:42:22.709827+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:42:22.709827+0000","last_undegraded":"2024-03-11T21:42:22.709827+0000","last_fullsized":"2024-03-11T21:42:22.709827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:26.303567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.303567+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.068658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1478,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.617487+0000","last_change":"2024-03-11T21:40:24.347139+0000","last_active":"2024-03-11T21:42:19.617487+0000","last_peered":"2024-03-11T21:42:19.617487+0000","last_clean":"2024-03-11T21:42:19.617487+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:42:19.617487+0000","last_undegraded":"2024-03-11T21:42:19.617487+0000","last_fullsized":"2024-03-11T21:42:19.617487+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.347032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.347032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1607,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708905+0000","last_change":"2024-03-11T21:41:27.719707+0000","last_active":"2024-03-11T21:42:22.708905+0000","last_peered":"2024-03-11T21:42:22.708905+0000","last_clean":"2024-03-11T21:42:22.708905+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:42:22.708905+0000","last_undegraded":"2024-03-11T21:42:22.708905+0000","last_fullsized":"2024-03-11T21:42:22.708905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.719614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.719614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.914311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1573,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.711283+0000","last_change":"2024-03-11T21:41:52.458683+0000","last_active":"2024-03-11T21:42:22.711283+0000","last_peered":"2024-03-11T21:42:22.711283+0000","last_clean":"2024-03-11T21:42:22.711283+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:42:22.711283+0000","last_undegraded":"2024-03-11T21:42:22.711283+0000","last_fullsized":"2024-03-11T21:42:22.711283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:52.458535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:41:52.458535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.765250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1425,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710244+0000","last_change":"2024-03-11T21:41:55.275905+0000","last_active":"2024-03-11T21:42:22.710244+0000","last_peered":"2024-03-11T21:42:22.710244+0000","last_clean":"2024-03-11T21:42:22.710244+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:42:22.710244+0000","last_undegraded":"2024-03-11T21:42:22.710244+0000","last_fullsized":"2024-03-11T21:42:22.710244+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.275680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.275680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.360123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1619,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706881+0000","last_change":"2024-03-11T21:38:59.769177+0000","last_active":"2024-03-11T21:42:22.706881+0000","last_peered":"2024-03-11T21:42:22.706881+0000","last_clean":"2024-03-11T21:42:22.706881+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:42:22.706881+0000","last_undegraded":"2024-03-11T21:42:22.706881+0000","last_fullsized":"2024-03-11T21:42:22.706881+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_deep_scrub":"389'39","last_deep_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_clean_scrub_stamp":"2024-03-11T21:38:59.768919+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1615,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.711991+0000","last_change":"2024-03-11T21:41:37.871821+0000","last_active":"2024-03-11T21:42:22.711991+0000","last_peered":"2024-03-11T21:42:22.711991+0000","last_clean":"2024-03-11T21:42:22.711991+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:42:22.711991+0000","last_undegraded":"2024-03-11T21:42:22.711991+0000","last_fullsized":"2024-03-11T21:42:22.711991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.871740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.871740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.172436+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1607,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.711310+0000","last_change":"2024-03-11T21:41:51.789113+0000","last_active":"2024-03-11T21:42:22.711310+0000","last_peered":"2024-03-11T21:42:22.711310+0000","last_clean":"2024-03-11T21:42:22.711310+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:42:22.711310+0000","last_undegraded":"2024-03-11T21:42:22.711310+0000","last_fullsized":"2024-03-11T21:42:22.711310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.789014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.789014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.998815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1554,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703264+0000","last_change":"2024-03-11T21:39:59.804040+0000","last_active":"2024-03-11T21:42:22.703264+0000","last_peered":"2024-03-11T21:42:22.703264+0000","last_clean":"2024-03-11T21:42:22.703264+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:42:22.703264+0000","last_undegraded":"2024-03-11T21:42:22.703264+0000","last_fullsized":"2024-03-11T21:42:22.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:59.803820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.803820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":804,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708061+0000","last_change":"2024-03-11T21:41:22.197315+0000","last_active":"2024-03-11T21:42:22.708061+0000","last_peered":"2024-03-11T21:42:22.708061+0000","last_clean":"2024-03-11T21:42:22.708061+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:42:22.708061+0000","last_undegraded":"2024-03-11T21:42:22.708061+0000","last_fullsized":"2024-03-11T21:42:22.708061+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:22.197156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.197156+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.113967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1298,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.614242+0000","last_change":"2024-03-11T21:40:05.414458+0000","last_active":"2024-03-11T21:42:19.614242+0000","last_peered":"2024-03-11T21:42:19.614242+0000","last_clean":"2024-03-11T21:42:19.614242+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:42:19.614242+0000","last_undegraded":"2024-03-11T21:42:19.614242+0000","last_fullsized":"2024-03-11T21:42:19.614242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:05.414369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.414369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1671,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704820+0000","last_change":"2024-03-11T21:41:32.795704+0000","last_active":"2024-03-11T21:42:22.704820+0000","last_peered":"2024-03-11T21:42:22.704820+0000","last_clean":"2024-03-11T21:42:22.704820+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:42:22.704820+0000","last_undegraded":"2024-03-11T21:42:22.704820+0000","last_fullsized":"2024-03-11T21:42:22.704820+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:41:32.795430+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.795430+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.294775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1574,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704612+0000","last_change":"2024-03-11T21:41:29.509226+0000","last_active":"2024-03-11T21:42:22.704612+0000","last_peered":"2024-03-11T21:42:22.704612+0000","last_clean":"2024-03-11T21:42:22.704612+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:42:22.704612+0000","last_undegraded":"2024-03-11T21:42:22.704612+0000","last_fullsized":"2024-03-11T21:42:22.704612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.509105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.509105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:43.027891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1484,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616389+0000","last_change":"2024-03-11T21:42:00.292135+0000","last_active":"2024-03-11T21:42:19.616389+0000","last_peered":"2024-03-11T21:42:19.616389+0000","last_clean":"2024-03-11T21:42:19.616389+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:42:19.616389+0000","last_undegraded":"2024-03-11T21:42:19.616389+0000","last_fullsized":"2024-03-11T21:42:19.616389+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.292001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.532304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1603,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715256+0000","last_change":"2024-03-11T21:41:20.636042+0000","last_active":"2024-03-11T21:42:22.715256+0000","last_peered":"2024-03-11T21:42:22.715256+0000","last_clean":"2024-03-11T21:42:22.715256+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:42:22.715256+0000","last_undegraded":"2024-03-11T21:42:22.715256+0000","last_fullsized":"2024-03-11T21:42:22.715256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.635800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.635800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.846076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1856,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707708+0000","last_change":"2024-03-11T21:41:30.525150+0000","last_active":"2024-03-11T21:42:22.707708+0000","last_peered":"2024-03-11T21:42:22.707708+0000","last_clean":"2024-03-11T21:42:22.707708+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:42:22.707708+0000","last_undegraded":"2024-03-11T21:42:22.707708+0000","last_fullsized":"2024-03-11T21:42:22.707708+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:30.525009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.525009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.617899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1429,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661810+0000","last_change":"2024-03-11T21:40:08.203021+0000","last_active":"2024-03-11T21:42:22.661810+0000","last_peered":"2024-03-11T21:42:22.661810+0000","last_clean":"2024-03-11T21:42:22.661810+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:42:22.661810+0000","last_undegraded":"2024-03-11T21:42:22.661810+0000","last_fullsized":"2024-03-11T21:42:22.661810+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:40:08.202874+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:40:08.202874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1597,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.668965+0000","last_change":"2024-03-11T21:38:42.809925+0000","last_active":"2024-03-11T21:42:22.668965+0000","last_peered":"2024-03-11T21:42:22.668965+0000","last_clean":"2024-03-11T21:42:22.668965+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:42:22.668965+0000","last_undegraded":"2024-03-11T21:42:22.668965+0000","last_fullsized":"2024-03-11T21:42:22.668965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:38:42.809678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:42.809678+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1444,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616733+0000","last_change":"2024-03-11T21:40:09.399602+0000","last_active":"2024-03-11T21:42:19.616733+0000","last_peered":"2024-03-11T21:42:19.616733+0000","last_clean":"2024-03-11T21:42:19.616733+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:42:19.616733+0000","last_undegraded":"2024-03-11T21:42:19.616733+0000","last_fullsized":"2024-03-11T21:42:19.616733+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.399381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.399381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1850,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713503+0000","last_change":"2024-03-11T21:41:26.709378+0000","last_active":"2024-03-11T21:42:22.713503+0000","last_peered":"2024-03-11T21:42:22.713503+0000","last_clean":"2024-03-11T21:42:22.713503+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:42:22.713503+0000","last_undegraded":"2024-03-11T21:42:22.713503+0000","last_fullsized":"2024-03-11T21:42:22.713503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:41:26.709267+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.709267+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.444646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1599,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659749+0000","last_change":"2024-03-11T21:41:28.709111+0000","last_active":"2024-03-11T21:42:22.659749+0000","last_peered":"2024-03-11T21:42:22.659749+0000","last_clean":"2024-03-11T21:42:22.659749+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:42:22.659749+0000","last_undegraded":"2024-03-11T21:42:22.659749+0000","last_fullsized":"2024-03-11T21:42:22.659749+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.707202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.707202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.875323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":804,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708597+0000","last_change":"2024-03-11T21:41:14.202929+0000","last_active":"2024-03-11T21:42:22.708597+0000","last_peered":"2024-03-11T21:42:22.708597+0000","last_clean":"2024-03-11T21:42:22.708597+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:42:22.708597+0000","last_undegraded":"2024-03-11T21:42:22.708597+0000","last_fullsized":"2024-03-11T21:42:22.708597+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:14.202622+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:37:53.724768+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.202622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.086810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1432,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.664586+0000","last_change":"2024-03-11T21:41:25.364494+0000","last_active":"2024-03-11T21:42:22.664586+0000","last_peered":"2024-03-11T21:42:22.664586+0000","last_clean":"2024-03-11T21:42:22.664586+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:42:22.664586+0000","last_undegraded":"2024-03-11T21:42:22.664586+0000","last_fullsized":"2024-03-11T21:42:22.664586+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.364387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.364387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.701511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1624,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661860+0000","last_change":"2024-03-11T21:40:06.159993+0000","last_active":"2024-03-11T21:42:22.661860+0000","last_peered":"2024-03-11T21:42:22.661860+0000","last_clean":"2024-03-11T21:42:22.661860+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:42:22.661860+0000","last_undegraded":"2024-03-11T21:42:22.661860+0000","last_fullsized":"2024-03-11T21:42:22.661860+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:40:06.159823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.159823+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1472,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.613063+0000","last_change":"2024-03-11T21:40:08.421673+0000","last_active":"2024-03-11T21:42:19.613063+0000","last_peered":"2024-03-11T21:42:19.613063+0000","last_clean":"2024-03-11T21:42:19.613063+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:42:19.613063+0000","last_undegraded":"2024-03-11T21:42:19.613063+0000","last_fullsized":"2024-03-11T21:42:19.613063+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_clean_scrub_stamp":"2024-03-11T21:40:08.421516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1836,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659515+0000","last_change":"2024-03-11T21:41:46.623952+0000","last_active":"2024-03-11T21:42:22.659515+0000","last_peered":"2024-03-11T21:42:22.659515+0000","last_clean":"2024-03-11T21:42:22.659515+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:42:22.659515+0000","last_undegraded":"2024-03-11T21:42:22.659515+0000","last_fullsized":"2024-03-11T21:42:22.659515+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:46.623794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.623794+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.014629+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1584,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709847+0000","last_change":"2024-03-11T21:41:58.366463+0000","last_active":"2024-03-11T21:42:22.709847+0000","last_peered":"2024-03-11T21:42:22.709847+0000","last_clean":"2024-03-11T21:42:22.709847+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:42:22.709847+0000","last_undegraded":"2024-03-11T21:42:22.709847+0000","last_fullsized":"2024-03-11T21:42:22.709847+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.366196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.366196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.319086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1413,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704128+0000","last_change":"2024-03-11T21:38:41.784165+0000","last_active":"2024-03-11T21:42:22.704128+0000","last_peered":"2024-03-11T21:42:22.704128+0000","last_clean":"2024-03-11T21:42:22.704128+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:42:22.704128+0000","last_undegraded":"2024-03-11T21:42:22.704128+0000","last_fullsized":"2024-03-11T21:42:22.704128+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:41.783976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:41.783976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399281+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1627,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.656542+0000","last_change":"2024-03-11T21:40:15.854712+0000","last_active":"2024-03-11T21:42:22.656542+0000","last_peered":"2024-03-11T21:42:22.656542+0000","last_clean":"2024-03-11T21:42:22.656542+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:42:22.656542+0000","last_undegraded":"2024-03-11T21:42:22.656542+0000","last_fullsized":"2024-03-11T21:42:22.656542+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:40:15.854104+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.854104+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1583,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703752+0000","last_change":"2024-03-11T21:41:18.465526+0000","last_active":"2024-03-11T21:42:22.703752+0000","last_peered":"2024-03-11T21:42:22.703752+0000","last_clean":"2024-03-11T21:42:22.703752+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:42:22.703752+0000","last_undegraded":"2024-03-11T21:42:22.703752+0000","last_fullsized":"2024-03-11T21:42:22.703752+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.465412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.465412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.489077+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1571,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710809+0000","last_change":"2024-03-11T21:41:29.273257+0000","last_active":"2024-03-11T21:42:22.710809+0000","last_peered":"2024-03-11T21:42:22.710809+0000","last_clean":"2024-03-11T21:42:22.710809+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:42:22.710809+0000","last_undegraded":"2024-03-11T21:42:22.710809+0000","last_fullsized":"2024-03-11T21:42:22.710809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.273150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.273150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.658499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1564,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671221+0000","last_change":"2024-03-11T21:41:26.327523+0000","last_active":"2024-03-11T21:42:22.671221+0000","last_peered":"2024-03-11T21:42:22.671221+0000","last_clean":"2024-03-11T21:42:22.671221+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:42:22.671221+0000","last_undegraded":"2024-03-11T21:42:22.671221+0000","last_fullsized":"2024-03-11T21:42:22.671221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.327416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.327416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.740808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":807,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671676+0000","last_change":"2024-03-11T21:41:57.696381+0000","last_active":"2024-03-11T21:42:22.671676+0000","last_peered":"2024-03-11T21:42:22.671676+0000","last_clean":"2024-03-11T21:42:22.671676+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:42:22.671676+0000","last_undegraded":"2024-03-11T21:42:22.671676+0000","last_fullsized":"2024-03-11T21:42:22.671676+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:41:57.696205+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.696205+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:57.700510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1415,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671982+0000","last_change":"2024-03-11T21:40:03.109654+0000","last_active":"2024-03-11T21:42:22.671982+0000","last_peered":"2024-03-11T21:42:22.671982+0000","last_clean":"2024-03-11T21:42:22.671982+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:42:22.671982+0000","last_undegraded":"2024-03-11T21:42:22.671982+0000","last_fullsized":"2024-03-11T21:42:22.671982+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.109518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.109518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.720007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1821,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.662504+0000","last_change":"2024-03-11T21:39:59.035379+0000","last_active":"2024-03-11T21:42:22.662504+0000","last_peered":"2024-03-11T21:42:22.662504+0000","last_clean":"2024-03-11T21:42:22.662504+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:42:22.662504+0000","last_undegraded":"2024-03-11T21:42:22.662504+0000","last_fullsized":"2024-03-11T21:42:22.662504+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:39:59.035148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.035148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719901+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1604,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.656723+0000","last_change":"2024-03-11T21:40:17.919859+0000","last_active":"2024-03-11T21:42:22.656723+0000","last_peered":"2024-03-11T21:42:22.656723+0000","last_clean":"2024-03-11T21:42:22.656723+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:42:22.656723+0000","last_undegraded":"2024-03-11T21:42:22.656723+0000","last_fullsized":"2024-03-11T21:42:22.656723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:17.919574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:40:17.919574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1626,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715916+0000","last_change":"2024-03-11T21:41:43.459254+0000","last_active":"2024-03-11T21:42:22.715916+0000","last_peered":"2024-03-11T21:42:22.715916+0000","last_clean":"2024-03-11T21:42:22.715916+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:42:22.715916+0000","last_undegraded":"2024-03-11T21:42:22.715916+0000","last_fullsized":"2024-03-11T21:42:22.715916+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:41:43.458986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.458986+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.423341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1619,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706181+0000","last_change":"2024-03-11T21:41:55.649415+0000","last_active":"2024-03-11T21:42:22.706181+0000","last_peered":"2024-03-11T21:42:22.706181+0000","last_clean":"2024-03-11T21:42:22.706181+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:42:22.706181+0000","last_undegraded":"2024-03-11T21:42:22.706181+0000","last_fullsized":"2024-03-11T21:42:22.706181+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:41:55.649085+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.649085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.612818+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1565,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704337+0000","last_change":"2024-03-11T21:40:23.749973+0000","last_active":"2024-03-11T21:42:22.704337+0000","last_peered":"2024-03-11T21:42:22.704337+0000","last_clean":"2024-03-11T21:42:22.704337+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:42:22.704337+0000","last_undegraded":"2024-03-11T21:42:22.704337+0000","last_fullsized":"2024-03-11T21:42:22.704337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:40:23.749772+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.749772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399982+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":819,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658228+0000","last_change":"2024-03-11T21:41:56.716081+0000","last_active":"2024-03-11T21:42:22.658228+0000","last_peered":"2024-03-11T21:42:22.658228+0000","last_clean":"2024-03-11T21:42:22.658228+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:42:22.658228+0000","last_undegraded":"2024-03-11T21:42:22.658228+0000","last_fullsized":"2024-03-11T21:42:22.658228+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:56.715868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.715868+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:57.439090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1421,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706278+0000","last_change":"2024-03-11T21:41:42.428955+0000","last_active":"2024-03-11T21:42:22.706278+0000","last_peered":"2024-03-11T21:42:22.706278+0000","last_clean":"2024-03-11T21:42:22.706278+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:42:22.706278+0000","last_undegraded":"2024-03-11T21:42:22.706278+0000","last_fullsized":"2024-03-11T21:42:22.706278+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.428709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.428709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.809420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1603,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.656400+0000","last_change":"2024-03-11T21:41:42.630400+0000","last_active":"2024-03-11T21:42:22.656400+0000","last_peered":"2024-03-11T21:42:22.656400+0000","last_clean":"2024-03-11T21:42:22.656400+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:42:22.656400+0000","last_undegraded":"2024-03-11T21:42:22.656400+0000","last_fullsized":"2024-03-11T21:42:22.656400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.630272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.630272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:43.927942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2719,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.662210+0000","last_change":"2024-03-11T21:41:20.351387+0000","last_active":"2024-03-11T21:42:22.662210+0000","last_peered":"2024-03-11T21:42:22.662210+0000","last_clean":"2024-03-11T21:42:22.662210+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:42:22.662210+0000","last_undegraded":"2024-03-11T21:42:22.662210+0000","last_fullsized":"2024-03-11T21:42:22.662210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:41:20.351150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.351150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.077803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1138,"num_read_kb":1133,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1605,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.670804+0000","last_change":"2024-03-11T21:42:03.753084+0000","last_active":"2024-03-11T21:42:22.670804+0000","last_peered":"2024-03-11T21:42:22.670804+0000","last_clean":"2024-03-11T21:42:22.670804+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:42:22.670804+0000","last_undegraded":"2024-03-11T21:42:22.670804+0000","last_fullsized":"2024-03-11T21:42:22.670804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:42:03.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:03.752924+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:08.213946+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1497,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.619343+0000","last_change":"2024-03-11T21:40:06.437710+0000","last_active":"2024-03-11T21:42:19.619343+0000","last_peered":"2024-03-11T21:42:19.619343+0000","last_clean":"2024-03-11T21:42:19.619343+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:42:19.619343+0000","last_undegraded":"2024-03-11T21:42:19.619343+0000","last_fullsized":"2024-03-11T21:42:19.619343+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:40:06.437601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.437601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1591,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.657030+0000","last_change":"2024-03-11T21:41:20.553933+0000","last_active":"2024-03-11T21:42:22.657030+0000","last_peered":"2024-03-11T21:42:22.657030+0000","last_clean":"2024-03-11T21:42:22.657030+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:42:22.657030+0000","last_undegraded":"2024-03-11T21:42:22.657030+0000","last_fullsized":"2024-03-11T21:42:22.657030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.553712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.553712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.127880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":821,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705151+0000","last_change":"2024-03-11T21:41:27.847353+0000","last_active":"2024-03-11T21:42:22.705151+0000","last_peered":"2024-03-11T21:42:22.705151+0000","last_clean":"2024-03-11T21:42:22.705151+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:42:22.705151+0000","last_undegraded":"2024-03-11T21:42:22.705151+0000","last_fullsized":"2024-03-11T21:42:22.705151+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:27.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.847107+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.711792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1421,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706428+0000","last_change":"2024-03-11T21:41:35.427640+0000","last_active":"2024-03-11T21:42:22.706428+0000","last_peered":"2024-03-11T21:42:22.706428+0000","last_clean":"2024-03-11T21:42:22.706428+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:42:22.706428+0000","last_undegraded":"2024-03-11T21:42:22.706428+0000","last_fullsized":"2024-03-11T21:42:22.706428+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.427438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.427438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.404944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1587,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704711+0000","last_change":"2024-03-11T21:41:23.914266+0000","last_active":"2024-03-11T21:42:22.704711+0000","last_peered":"2024-03-11T21:42:22.704711+0000","last_clean":"2024-03-11T21:42:22.704711+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:42:22.704711+0000","last_undegraded":"2024-03-11T21:42:22.704711+0000","last_fullsized":"2024-03-11T21:42:22.704711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.913967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.913967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.777062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1766,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.624322+0000","last_change":"2024-03-11T21:41:19.438289+0000","last_active":"2024-03-11T21:42:19.624322+0000","last_peered":"2024-03-11T21:42:19.624322+0000","last_clean":"2024-03-11T21:42:19.624322+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:42:19.624322+0000","last_undegraded":"2024-03-11T21:42:19.624322+0000","last_fullsized":"2024-03-11T21:42:19.624322+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:41:19.438160+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.438160+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.848834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1575,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706756+0000","last_change":"2024-03-11T21:41:44.440095+0000","last_active":"2024-03-11T21:42:22.706756+0000","last_peered":"2024-03-11T21:42:22.706756+0000","last_clean":"2024-03-11T21:42:22.706756+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:42:22.706756+0000","last_undegraded":"2024-03-11T21:42:22.706756+0000","last_fullsized":"2024-03-11T21:42:22.706756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:44.439846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:44.439846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.010761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1437,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.612283+0000","last_change":"2024-03-11T21:40:17.365129+0000","last_active":"2024-03-11T21:42:19.612283+0000","last_peered":"2024-03-11T21:42:19.612283+0000","last_clean":"2024-03-11T21:42:19.612283+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:42:19.612283+0000","last_undegraded":"2024-03-11T21:42:19.612283+0000","last_fullsized":"2024-03-11T21:42:19.612283+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:40:17.364997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":807,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.617456+0000","last_change":"2024-03-11T21:40:27.432675+0000","last_active":"2024-03-11T21:42:19.617456+0000","last_peered":"2024-03-11T21:42:19.617456+0000","last_clean":"2024-03-11T21:42:19.617456+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:42:19.617456+0000","last_undegraded":"2024-03-11T21:42:19.617456+0000","last_fullsized":"2024-03-11T21:42:19.617456+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:40:27.432564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.432564+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1423,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715268+0000","last_change":"2024-03-11T21:40:18.629853+0000","last_active":"2024-03-11T21:42:22.715268+0000","last_peered":"2024-03-11T21:42:22.715268+0000","last_clean":"2024-03-11T21:42:22.715268+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:42:22.715268+0000","last_undegraded":"2024-03-11T21:42:22.715268+0000","last_fullsized":"2024-03-11T21:42:22.715268+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.629736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.629736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1580,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710713+0000","last_change":"2024-03-11T21:41:17.193786+0000","last_active":"2024-03-11T21:42:22.710713+0000","last_peered":"2024-03-11T21:42:22.710713+0000","last_clean":"2024-03-11T21:42:22.710713+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:42:22.710713+0000","last_undegraded":"2024-03-11T21:42:22.710713+0000","last_fullsized":"2024-03-11T21:42:22.710713+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.193618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.193618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.977433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1584,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703423+0000","last_change":"2024-03-11T21:42:04.407830+0000","last_active":"2024-03-11T21:42:22.703423+0000","last_peered":"2024-03-11T21:42:22.703423+0000","last_clean":"2024-03-11T21:42:22.703423+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:42:22.703423+0000","last_undegraded":"2024-03-11T21:42:22.703423+0000","last_fullsized":"2024-03-11T21:42:22.703423+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:04.407615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:42:04.407615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.937658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1624,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709960+0000","last_change":"2024-03-11T21:42:00.572712+0000","last_active":"2024-03-11T21:42:22.709960+0000","last_peered":"2024-03-11T21:42:22.709960+0000","last_clean":"2024-03-11T21:42:22.709960+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:42:22.709960+0000","last_undegraded":"2024-03-11T21:42:22.709960+0000","last_fullsized":"2024-03-11T21:42:22.709960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:00.572462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.572462+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.915234+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1447,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.618359+0000","last_change":"2024-03-11T21:41:33.473130+0000","last_active":"2024-03-11T21:42:19.618359+0000","last_peered":"2024-03-11T21:42:19.618359+0000","last_clean":"2024-03-11T21:42:19.618359+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:42:19.618359+0000","last_undegraded":"2024-03-11T21:42:19.618359+0000","last_fullsized":"2024-03-11T21:42:19.618359+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.472920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.472920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.414677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1423,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661155+0000","last_change":"2024-03-11T21:42:01.660218+0000","last_active":"2024-03-11T21:42:22.661155+0000","last_peered":"2024-03-11T21:42:22.661155+0000","last_clean":"2024-03-11T21:42:22.661155+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:42:22.661155+0000","last_undegraded":"2024-03-11T21:42:22.661155+0000","last_fullsized":"2024-03-11T21:42:22.661155+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.660109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.660109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:31.505443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1559,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705910+0000","last_change":"2024-03-11T21:40:22.764357+0000","last_active":"2024-03-11T21:42:22.705910+0000","last_peered":"2024-03-11T21:42:22.705910+0000","last_clean":"2024-03-11T21:42:22.705910+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:42:22.705910+0000","last_undegraded":"2024-03-11T21:42:22.705910+0000","last_fullsized":"2024-03-11T21:42:22.705910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:22.764240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:22.764240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.400067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1848,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703577+0000","last_change":"2024-03-11T21:41:41.451026+0000","last_active":"2024-03-11T21:42:22.703577+0000","last_peered":"2024-03-11T21:42:22.703577+0000","last_clean":"2024-03-11T21:42:22.703577+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:42:22.703577+0000","last_undegraded":"2024-03-11T21:42:22.703577+0000","last_fullsized":"2024-03-11T21:42:22.703577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:41.450896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.450896+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:00.423644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1580,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708980+0000","last_change":"2024-03-11T21:40:10.827418+0000","last_active":"2024-03-11T21:42:22.708980+0000","last_peered":"2024-03-11T21:42:22.708980+0000","last_clean":"2024-03-11T21:42:22.708980+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:42:22.708980+0000","last_undegraded":"2024-03-11T21:42:22.708980+0000","last_fullsized":"2024-03-11T21:42:22.708980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:10.827185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:40:10.827185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1603,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.664087+0000","last_change":"2024-03-11T21:41:35.415804+0000","last_active":"2024-03-11T21:42:22.664087+0000","last_peered":"2024-03-11T21:42:22.664087+0000","last_clean":"2024-03-11T21:42:22.664087+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:42:22.664087+0000","last_undegraded":"2024-03-11T21:42:22.664087+0000","last_fullsized":"2024-03-11T21:42:22.664087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:35.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.415630+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.778929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":377,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716207+0000","last_change":"2024-03-11T21:41:37.415103+0000","last_active":"2024-03-11T21:42:22.716207+0000","last_peered":"2024-03-11T21:42:22.716207+0000","last_clean":"2024-03-11T21:42:22.716207+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:42:22.716207+0000","last_undegraded":"2024-03-11T21:42:22.716207+0000","last_fullsized":"2024-03-11T21:42:22.716207+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.414919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.414919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.220108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1856,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704426+0000","last_change":"2024-03-11T21:41:38.433752+0000","last_active":"2024-03-11T21:42:22.704426+0000","last_peered":"2024-03-11T21:42:22.704426+0000","last_clean":"2024-03-11T21:42:22.704426+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:42:22.704426+0000","last_undegraded":"2024-03-11T21:42:22.704426+0000","last_fullsized":"2024-03-11T21:42:22.704426+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:38.433381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:38.433381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.908595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1426,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708757+0000","last_change":"2024-03-11T21:40:19.766314+0000","last_active":"2024-03-11T21:42:22.708757+0000","last_peered":"2024-03-11T21:42:22.708757+0000","last_clean":"2024-03-11T21:42:22.708757+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:42:22.708757+0000","last_undegraded":"2024-03-11T21:42:22.708757+0000","last_fullsized":"2024-03-11T21:42:22.708757+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.766007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.766007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1607,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714627+0000","last_change":"2024-03-11T21:41:15.651241+0000","last_active":"2024-03-11T21:42:22.714627+0000","last_peered":"2024-03-11T21:42:22.714627+0000","last_clean":"2024-03-11T21:42:22.714627+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:42:22.714627+0000","last_undegraded":"2024-03-11T21:42:22.714627+0000","last_fullsized":"2024-03-11T21:42:22.714627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.651121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.651121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.721840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1616,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709747+0000","last_change":"2024-03-11T21:41:41.822816+0000","last_active":"2024-03-11T21:42:22.709747+0000","last_peered":"2024-03-11T21:42:22.709747+0000","last_clean":"2024-03-11T21:42:22.709747+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:42:22.709747+0000","last_undegraded":"2024-03-11T21:42:22.709747+0000","last_fullsized":"2024-03-11T21:42:22.709747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.822599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.822599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:54.296947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":1529,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.615394+0000","last_change":"2024-03-11T21:41:27.465101+0000","last_active":"2024-03-11T21:42:19.615394+0000","last_peered":"2024-03-11T21:42:19.615394+0000","last_clean":"2024-03-11T21:42:19.615394+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:42:19.615394+0000","last_undegraded":"2024-03-11T21:42:19.615394+0000","last_fullsized":"2024-03-11T21:42:19.615394+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:41:27.464990+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.464990+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.787751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1555,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706598+0000","last_change":"2024-03-11T21:39:51.887003+0000","last_active":"2024-03-11T21:42:22.706598+0000","last_peered":"2024-03-11T21:42:22.706598+0000","last_clean":"2024-03-11T21:42:22.706598+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:42:22.706598+0000","last_undegraded":"2024-03-11T21:42:22.706598+0000","last_fullsized":"2024-03-11T21:42:22.706598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:51.886841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:51.886841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":804,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708525+0000","last_change":"2024-03-11T21:41:18.182642+0000","last_active":"2024-03-11T21:42:22.708525+0000","last_peered":"2024-03-11T21:42:22.708525+0000","last_clean":"2024-03-11T21:42:22.708525+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:42:22.708525+0000","last_undegraded":"2024-03-11T21:42:22.708525+0000","last_fullsized":"2024-03-11T21:42:22.708525+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:18.182513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.182513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.290004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1431,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709282+0000","last_change":"2024-03-11T21:42:01.549509+0000","last_active":"2024-03-11T21:42:22.709282+0000","last_peered":"2024-03-11T21:42:22.709282+0000","last_clean":"2024-03-11T21:42:22.709282+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:42:22.709282+0000","last_undegraded":"2024-03-11T21:42:22.709282+0000","last_fullsized":"2024-03-11T21:42:22.709282+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.549350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.861337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1587,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715275+0000","last_change":"2024-03-11T21:41:58.613981+0000","last_active":"2024-03-11T21:42:22.715275+0000","last_peered":"2024-03-11T21:42:22.715275+0000","last_clean":"2024-03-11T21:42:22.715275+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:42:22.715275+0000","last_undegraded":"2024-03-11T21:42:22.715275+0000","last_fullsized":"2024-03-11T21:42:22.715275+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.613685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.613685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:23.992971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1658,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658202+0000","last_change":"2024-03-11T21:41:29.674697+0000","last_active":"2024-03-11T21:42:22.658202+0000","last_peered":"2024-03-11T21:42:22.658202+0000","last_clean":"2024-03-11T21:42:22.658202+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:42:22.658202+0000","last_undegraded":"2024-03-11T21:42:22.658202+0000","last_fullsized":"2024-03-11T21:42:22.658202+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:41:29.674458+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.674458+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:53.291740+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1589,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709127+0000","last_change":"2024-03-11T21:41:34.232210+0000","last_active":"2024-03-11T21:42:22.709127+0000","last_peered":"2024-03-11T21:42:22.709127+0000","last_clean":"2024-03-11T21:42:22.709127+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:42:22.709127+0000","last_undegraded":"2024-03-11T21:42:22.709127+0000","last_fullsized":"2024-03-11T21:42:22.709127+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:34.231996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:34.231996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.938253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1564,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671342+0000","last_change":"2024-03-11T21:41:17.378484+0000","last_active":"2024-03-11T21:42:22.671342+0000","last_peered":"2024-03-11T21:42:22.671342+0000","last_clean":"2024-03-11T21:42:22.671342+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:42:22.671342+0000","last_undegraded":"2024-03-11T21:42:22.671342+0000","last_fullsized":"2024-03-11T21:42:22.671342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.378291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.378291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:27.116068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":821,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705983+0000","last_change":"2024-03-11T21:41:51.718486+0000","last_active":"2024-03-11T21:42:22.705983+0000","last_peered":"2024-03-11T21:42:22.705983+0000","last_clean":"2024-03-11T21:42:22.705983+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:42:22.705983+0000","last_undegraded":"2024-03-11T21:42:22.705983+0000","last_fullsized":"2024-03-11T21:42:22.705983+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:51.718268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.718268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.247208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1426,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710338+0000","last_change":"2024-03-11T21:40:21.837198+0000","last_active":"2024-03-11T21:42:22.710338+0000","last_peered":"2024-03-11T21:42:22.710338+0000","last_clean":"2024-03-11T21:42:22.710338+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:42:22.710338+0000","last_undegraded":"2024-03-11T21:42:22.710338+0000","last_fullsized":"2024-03-11T21:42:22.710338+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.836981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.836981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1824,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714959+0000","last_change":"2024-03-11T21:41:40.335413+0000","last_active":"2024-03-11T21:42:22.714959+0000","last_peered":"2024-03-11T21:42:22.714959+0000","last_clean":"2024-03-11T21:42:22.714959+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:42:22.714959+0000","last_undegraded":"2024-03-11T21:42:22.714959+0000","last_fullsized":"2024-03-11T21:42:22.714959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:41:40.335178+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.335178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.489495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1474,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.614741+0000","last_change":"2024-03-11T21:39:51.220612+0000","last_active":"2024-03-11T21:42:19.614741+0000","last_peered":"2024-03-11T21:42:19.614741+0000","last_clean":"2024-03-11T21:42:19.614741+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:42:19.614741+0000","last_undegraded":"2024-03-11T21:42:19.614741+0000","last_fullsized":"2024-03-11T21:42:19.614741+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:39:51.220503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:39:51.220503+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1848,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709993+0000","last_change":"2024-03-11T21:40:05.829359+0000","last_active":"2024-03-11T21:42:22.709993+0000","last_peered":"2024-03-11T21:42:22.709993+0000","last_clean":"2024-03-11T21:42:22.709993+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:42:22.709993+0000","last_undegraded":"2024-03-11T21:42:22.709993+0000","last_fullsized":"2024-03-11T21:42:22.709993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:40:05.829098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.829098+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1445,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.616874+0000","last_change":"2024-03-11T21:40:03.406984+0000","last_active":"2024-03-11T21:42:19.616874+0000","last_peered":"2024-03-11T21:42:19.616874+0000","last_clean":"2024-03-11T21:42:19.616874+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:42:19.616874+0000","last_undegraded":"2024-03-11T21:42:19.616874+0000","last_fullsized":"2024-03-11T21:42:19.616874+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.406773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.406773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1573,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709086+0000","last_change":"2024-03-11T21:41:28.319779+0000","last_active":"2024-03-11T21:42:22.709086+0000","last_peered":"2024-03-11T21:42:22.709086+0000","last_clean":"2024-03-11T21:42:22.709086+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:42:22.709086+0000","last_undegraded":"2024-03-11T21:42:22.709086+0000","last_fullsized":"2024-03-11T21:42:22.709086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.319670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.319670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.589490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1306,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.618103+0000","last_change":"2024-03-11T21:41:46.296818+0000","last_active":"2024-03-11T21:42:19.618103+0000","last_peered":"2024-03-11T21:42:19.618103+0000","last_clean":"2024-03-11T21:42:19.618103+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:42:19.618103+0000","last_undegraded":"2024-03-11T21:42:19.618103+0000","last_fullsized":"2024-03-11T21:42:19.618103+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.296690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.296690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.764625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1616,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710443+0000","last_change":"2024-03-11T21:41:45.803597+0000","last_active":"2024-03-11T21:42:22.710443+0000","last_peered":"2024-03-11T21:42:22.710443+0000","last_clean":"2024-03-11T21:42:22.710443+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:42:22.710443+0000","last_undegraded":"2024-03-11T21:42:22.710443+0000","last_fullsized":"2024-03-11T21:42:22.710443+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.803247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.803247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.321102+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1615,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706874+0000","last_change":"2024-03-11T21:39:58.844136+0000","last_active":"2024-03-11T21:42:22.706874+0000","last_peered":"2024-03-11T21:42:22.706874+0000","last_clean":"2024-03-11T21:42:22.706874+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:42:22.706874+0000","last_undegraded":"2024-03-11T21:42:22.706874+0000","last_fullsized":"2024-03-11T21:42:22.706874+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:39:58.843910+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.843910+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1620,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659368+0000","last_change":"2024-03-11T21:40:20.836641+0000","last_active":"2024-03-11T21:42:22.659368+0000","last_peered":"2024-03-11T21:42:22.659368+0000","last_clean":"2024-03-11T21:42:22.659368+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:42:22.659368+0000","last_undegraded":"2024-03-11T21:42:22.659368+0000","last_fullsized":"2024-03-11T21:42:22.659368+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:40:20.836413+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.836413+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1573,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669245+0000","last_change":"2024-03-11T21:40:24.477381+0000","last_active":"2024-03-11T21:42:22.669245+0000","last_peered":"2024-03-11T21:42:22.669245+0000","last_clean":"2024-03-11T21:42:22.669245+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:42:22.669245+0000","last_undegraded":"2024-03-11T21:42:22.669245+0000","last_fullsized":"2024-03-11T21:42:22.669245+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.477246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.477246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.720070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1422,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669523+0000","last_change":"2024-03-11T21:41:15.373061+0000","last_active":"2024-03-11T21:42:22.669523+0000","last_peered":"2024-03-11T21:42:22.669523+0000","last_clean":"2024-03-11T21:42:22.669523+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:42:22.669523+0000","last_undegraded":"2024-03-11T21:42:22.669523+0000","last_fullsized":"2024-03-11T21:42:22.669523+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.372842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.372842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.257601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1593,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707201+0000","last_change":"2024-03-11T21:39:50.852078+0000","last_active":"2024-03-11T21:42:22.707201+0000","last_peered":"2024-03-11T21:42:22.707201+0000","last_clean":"2024-03-11T21:42:22.707201+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:42:22.707201+0000","last_undegraded":"2024-03-11T21:42:22.707201+0000","last_fullsized":"2024-03-11T21:42:22.707201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:39:50.851951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:39:50.851951+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1624,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710187+0000","last_change":"2024-03-11T21:41:43.763506+0000","last_active":"2024-03-11T21:42:22.710187+0000","last_peered":"2024-03-11T21:42:22.710187+0000","last_clean":"2024-03-11T21:42:22.710187+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:42:22.710187+0000","last_undegraded":"2024-03-11T21:42:22.710187+0000","last_fullsized":"2024-03-11T21:42:22.710187+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:43.763282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.763282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:01.513528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1576,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671303+0000","last_change":"2024-03-11T21:41:18.342623+0000","last_active":"2024-03-11T21:42:22.671303+0000","last_peered":"2024-03-11T21:42:22.671303+0000","last_clean":"2024-03-11T21:42:22.671303+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:42:22.671303+0000","last_undegraded":"2024-03-11T21:42:22.671303+0000","last_fullsized":"2024-03-11T21:42:22.671303+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.342525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.342525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.897683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1603,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712585+0000","last_change":"2024-03-11T21:41:16.668285+0000","last_active":"2024-03-11T21:42:22.712585+0000","last_peered":"2024-03-11T21:42:22.712585+0000","last_clean":"2024-03-11T21:42:22.712585+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:42:22.712585+0000","last_undegraded":"2024-03-11T21:42:22.712585+0000","last_fullsized":"2024-03-11T21:42:22.712585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.667920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.667920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.513028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1422,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.664832+0000","last_change":"2024-03-11T21:41:37.401402+0000","last_active":"2024-03-11T21:42:22.664832+0000","last_peered":"2024-03-11T21:42:22.664832+0000","last_clean":"2024-03-11T21:42:22.664832+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:42:22.664832+0000","last_undegraded":"2024-03-11T21:42:22.664832+0000","last_fullsized":"2024-03-11T21:42:22.664832+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.401275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.401275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.811147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1620,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.702852+0000","last_change":"2024-03-11T21:41:49.695635+0000","last_active":"2024-03-11T21:42:22.702852+0000","last_peered":"2024-03-11T21:42:22.702852+0000","last_clean":"2024-03-11T21:42:22.702852+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:42:22.702852+0000","last_undegraded":"2024-03-11T21:42:22.702852+0000","last_fullsized":"2024-03-11T21:42:22.702852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:41:49.695261+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.695261+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.305229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1611,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703480+0000","last_change":"2024-03-11T21:42:05.515337+0000","last_active":"2024-03-11T21:42:22.703480+0000","last_peered":"2024-03-11T21:42:22.703480+0000","last_clean":"2024-03-11T21:42:22.703480+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:42:22.703480+0000","last_undegraded":"2024-03-11T21:42:22.703480+0000","last_fullsized":"2024-03-11T21:42:22.703480+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:42:05.515038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:33.818265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1579,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710207+0000","last_change":"2024-03-11T21:41:42.239297+0000","last_active":"2024-03-11T21:42:22.710207+0000","last_peered":"2024-03-11T21:42:22.710207+0000","last_clean":"2024-03-11T21:42:22.710207+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:42:22.710207+0000","last_undegraded":"2024-03-11T21:42:22.710207+0000","last_fullsized":"2024-03-11T21:42:22.710207+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.239053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.921882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1553,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707865+0000","last_change":"2024-03-11T21:39:01.789461+0000","last_active":"2024-03-11T21:42:22.707865+0000","last_peered":"2024-03-11T21:42:22.707865+0000","last_clean":"2024-03-11T21:42:22.707865+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:42:22.707865+0000","last_undegraded":"2024-03-11T21:42:22.707865+0000","last_fullsized":"2024-03-11T21:42:22.707865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:01.789079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:01.789079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1414,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.667444+0000","last_change":"2024-03-11T21:38:38.738989+0000","last_active":"2024-03-11T21:42:22.667444+0000","last_peered":"2024-03-11T21:42:22.667444+0000","last_clean":"2024-03-11T21:42:22.667444+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:42:22.667444+0000","last_undegraded":"2024-03-11T21:42:22.667444+0000","last_fullsized":"2024-03-11T21:42:22.667444+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:38.738774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:38:38.738774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1601,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704742+0000","last_change":"2024-03-11T21:40:26.728613+0000","last_active":"2024-03-11T21:42:22.704742+0000","last_peered":"2024-03-11T21:42:22.704742+0000","last_clean":"2024-03-11T21:42:22.704742+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:42:22.704742+0000","last_undegraded":"2024-03-11T21:42:22.704742+0000","last_fullsized":"2024-03-11T21:42:22.704742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:26.728512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.728512+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.400116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1587,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709016+0000","last_change":"2024-03-11T21:40:20.810639+0000","last_active":"2024-03-11T21:42:22.709016+0000","last_peered":"2024-03-11T21:42:22.709016+0000","last_clean":"2024-03-11T21:42:22.709016+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:42:22.709016+0000","last_undegraded":"2024-03-11T21:42:22.709016+0000","last_fullsized":"2024-03-11T21:42:22.709016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.810193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.810193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1822,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710337+0000","last_change":"2024-03-11T21:41:49.267753+0000","last_active":"2024-03-11T21:42:22.710337+0000","last_peered":"2024-03-11T21:42:22.710337+0000","last_clean":"2024-03-11T21:42:22.710337+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:42:22.710337+0000","last_undegraded":"2024-03-11T21:42:22.710337+0000","last_fullsized":"2024-03-11T21:42:22.710337+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:41:49.267522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.267522+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.956526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1572,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716486+0000","last_change":"2024-03-11T21:40:23.588457+0000","last_active":"2024-03-11T21:42:22.716486+0000","last_peered":"2024-03-11T21:42:22.716486+0000","last_clean":"2024-03-11T21:42:22.716486+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:42:22.716486+0000","last_undegraded":"2024-03-11T21:42:22.716486+0000","last_fullsized":"2024-03-11T21:42:22.716486+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:23.588113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.588113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1446,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703094+0000","last_change":"2024-03-11T21:40:26.722273+0000","last_active":"2024-03-11T21:42:22.703094+0000","last_peered":"2024-03-11T21:42:22.703094+0000","last_clean":"2024-03-11T21:42:22.703094+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:42:22.703094+0000","last_undegraded":"2024-03-11T21:42:22.703094+0000","last_fullsized":"2024-03-11T21:42:22.703094+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.722122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.722122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.376398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1613,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.712386+0000","last_change":"2024-03-11T21:40:21.551903+0000","last_active":"2024-03-11T21:42:22.712386+0000","last_peered":"2024-03-11T21:42:22.712386+0000","last_clean":"2024-03-11T21:42:22.712386+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:42:22.712386+0000","last_undegraded":"2024-03-11T21:42:22.712386+0000","last_fullsized":"2024-03-11T21:42:22.712386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:21.551780+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.551780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1585,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713927+0000","last_change":"2024-03-11T21:41:31.320806+0000","last_active":"2024-03-11T21:42:22.713927+0000","last_peered":"2024-03-11T21:42:22.713927+0000","last_clean":"2024-03-11T21:42:22.713927+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:42:22.713927+0000","last_undegraded":"2024-03-11T21:42:22.713927+0000","last_fullsized":"2024-03-11T21:42:22.713927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.320685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.320685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:50.628945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1811,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.670925+0000","last_change":"2024-03-11T21:40:18.396071+0000","last_active":"2024-03-11T21:42:22.670925+0000","last_peered":"2024-03-11T21:42:22.670925+0000","last_clean":"2024-03-11T21:42:22.670925+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:42:22.670925+0000","last_undegraded":"2024-03-11T21:42:22.670925+0000","last_fullsized":"2024-03-11T21:42:22.670925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:40:18.395966+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.395966+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1597,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659910+0000","last_change":"2024-03-11T21:41:57.724943+0000","last_active":"2024-03-11T21:42:22.659910+0000","last_peered":"2024-03-11T21:42:22.659910+0000","last_clean":"2024-03-11T21:42:22.659910+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:42:22.659910+0000","last_undegraded":"2024-03-11T21:42:22.659910+0000","last_fullsized":"2024-03-11T21:42:22.659910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.724694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:26.895410+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.724694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:18.584812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1302,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.626069+0000","last_change":"2024-03-11T21:41:55.220405+0000","last_active":"2024-03-11T21:42:19.626069+0000","last_peered":"2024-03-11T21:42:19.626069+0000","last_clean":"2024-03-11T21:42:19.626069+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:42:19.626069+0000","last_undegraded":"2024-03-11T21:42:19.626069+0000","last_fullsized":"2024-03-11T21:42:19.626069+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.220196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.220196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.381564+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1598,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707146+0000","last_change":"2024-03-11T21:41:53.406126+0000","last_active":"2024-03-11T21:42:22.707146+0000","last_peered":"2024-03-11T21:42:22.707146+0000","last_clean":"2024-03-11T21:42:22.707146+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:42:22.707146+0000","last_undegraded":"2024-03-11T21:42:22.707146+0000","last_fullsized":"2024-03-11T21:42:22.707146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:41:53.405780+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.405780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.864386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1610,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705398+0000","last_change":"2024-03-11T21:41:36.715615+0000","last_active":"2024-03-11T21:42:22.705398+0000","last_peered":"2024-03-11T21:42:22.705398+0000","last_clean":"2024-03-11T21:42:22.705398+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:42:22.705398+0000","last_undegraded":"2024-03-11T21:42:22.705398+0000","last_fullsized":"2024-03-11T21:42:22.705398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:36.715491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:41:36.715491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.194697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1577,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669039+0000","last_change":"2024-03-11T21:41:31.332398+0000","last_active":"2024-03-11T21:42:22.669039+0000","last_peered":"2024-03-11T21:42:22.669039+0000","last_clean":"2024-03-11T21:42:22.669039+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:42:22.669039+0000","last_undegraded":"2024-03-11T21:42:22.669039+0000","last_fullsized":"2024-03-11T21:42:22.669039+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.332299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.332299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.360663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1581,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708805+0000","last_change":"2024-03-11T21:41:32.365712+0000","last_active":"2024-03-11T21:42:22.708805+0000","last_peered":"2024-03-11T21:42:22.708805+0000","last_clean":"2024-03-11T21:42:22.708805+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:42:22.708805+0000","last_undegraded":"2024-03-11T21:42:22.708805+0000","last_fullsized":"2024-03-11T21:42:22.708805+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.365462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.365462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.063953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1429,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.715896+0000","last_change":"2024-03-11T21:39:58.428530+0000","last_active":"2024-03-11T21:42:22.715896+0000","last_peered":"2024-03-11T21:42:22.715896+0000","last_clean":"2024-03-11T21:42:22.715896+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:42:22.715896+0000","last_undegraded":"2024-03-11T21:42:22.715896+0000","last_fullsized":"2024-03-11T21:42:22.715896+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:58.428408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.428408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1636,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658404+0000","last_change":"2024-03-11T21:41:15.535768+0000","last_active":"2024-03-11T21:42:22.658404+0000","last_peered":"2024-03-11T21:42:22.658404+0000","last_clean":"2024-03-11T21:42:22.658404+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:42:22.658404+0000","last_undegraded":"2024-03-11T21:42:22.658404+0000","last_fullsized":"2024-03-11T21:42:22.658404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:15.535528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.535528+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.789676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1488,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.617926+0000","last_change":"2024-03-11T21:41:21.483590+0000","last_active":"2024-03-11T21:42:19.617926+0000","last_peered":"2024-03-11T21:42:19.617926+0000","last_clean":"2024-03-11T21:42:19.617926+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:42:19.617926+0000","last_undegraded":"2024-03-11T21:42:19.617926+0000","last_fullsized":"2024-03-11T21:42:19.617926+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.483367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.483367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.508889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1456,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.614577+0000","last_change":"2024-03-11T21:40:18.382831+0000","last_active":"2024-03-11T21:42:19.614577+0000","last_peered":"2024-03-11T21:42:19.614577+0000","last_clean":"2024-03-11T21:42:19.614577+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:42:19.614577+0000","last_undegraded":"2024-03-11T21:42:19.614577+0000","last_fullsized":"2024-03-11T21:42:19.614577+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.382717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1564,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.662397+0000","last_change":"2024-03-11T21:41:21.351275+0000","last_active":"2024-03-11T21:42:22.662397+0000","last_peered":"2024-03-11T21:42:22.662397+0000","last_clean":"2024-03-11T21:42:22.662397+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:42:22.662397+0000","last_undegraded":"2024-03-11T21:42:22.662397+0000","last_fullsized":"2024-03-11T21:42:22.662397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.350961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.350961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.348048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1428,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714609+0000","last_change":"2024-03-11T21:41:48.432501+0000","last_active":"2024-03-11T21:42:22.714609+0000","last_peered":"2024-03-11T21:42:22.714609+0000","last_clean":"2024-03-11T21:42:22.714609+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:42:22.714609+0000","last_undegraded":"2024-03-11T21:42:22.714609+0000","last_fullsized":"2024-03-11T21:42:22.714609+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_clean_scrub_stamp":"2024-03-11T21:41:48.432274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.542101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1606,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716649+0000","last_change":"2024-03-11T21:42:02.547377+0000","last_active":"2024-03-11T21:42:22.716649+0000","last_peered":"2024-03-11T21:42:22.716649+0000","last_clean":"2024-03-11T21:42:22.716649+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:42:22.716649+0000","last_undegraded":"2024-03-11T21:42:22.716649+0000","last_fullsized":"2024-03-11T21:42:22.716649+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:42:02.547005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.547005+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.798290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1579,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710693+0000","last_change":"2024-03-11T21:40:13.779995+0000","last_active":"2024-03-11T21:42:22.710693+0000","last_peered":"2024-03-11T21:42:22.710693+0000","last_clean":"2024-03-11T21:42:22.710693+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:42:22.710693+0000","last_undegraded":"2024-03-11T21:42:22.710693+0000","last_fullsized":"2024-03-11T21:42:22.710693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.779418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.779418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1602,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703357+0000","last_change":"2024-03-11T21:42:02.517280+0000","last_active":"2024-03-11T21:42:22.703357+0000","last_peered":"2024-03-11T21:42:22.703357+0000","last_clean":"2024-03-11T21:42:22.703357+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:42:22.703357+0000","last_undegraded":"2024-03-11T21:42:22.703357+0000","last_fullsized":"2024-03-11T21:42:22.703357+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:02.516952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.516952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.145639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1554,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704349+0000","last_change":"2024-03-11T21:39:54.822173+0000","last_active":"2024-03-11T21:42:22.704349+0000","last_peered":"2024-03-11T21:42:22.704349+0000","last_clean":"2024-03-11T21:42:22.704349+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:42:22.704349+0000","last_undegraded":"2024-03-11T21:42:22.704349+0000","last_fullsized":"2024-03-11T21:42:22.704349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:54.822057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:39:54.822057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399565+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1431,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.713880+0000","last_change":"2024-03-11T21:40:19.599299+0000","last_active":"2024-03-11T21:42:22.713880+0000","last_peered":"2024-03-11T21:42:22.713880+0000","last_clean":"2024-03-11T21:42:22.713880+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:42:22.713880+0000","last_undegraded":"2024-03-11T21:42:22.713880+0000","last_fullsized":"2024-03-11T21:42:22.713880+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.599110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.599110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1609,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710262+0000","last_change":"2024-03-11T21:40:25.893678+0000","last_active":"2024-03-11T21:42:22.710262+0000","last_peered":"2024-03-11T21:42:22.710262+0000","last_clean":"2024-03-11T21:42:22.710262+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:42:22.710262+0000","last_undegraded":"2024-03-11T21:42:22.710262+0000","last_fullsized":"2024-03-11T21:42:22.710262+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:40:25.893452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.893452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1609,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706158+0000","last_change":"2024-03-11T21:41:59.579259+0000","last_active":"2024-03-11T21:42:22.706158+0000","last_peered":"2024-03-11T21:42:22.706158+0000","last_clean":"2024-03-11T21:42:22.706158+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:42:22.706158+0000","last_undegraded":"2024-03-11T21:42:22.706158+0000","last_fullsized":"2024-03-11T21:42:22.706158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.579089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.579089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.224629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1558,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703963+0000","last_change":"2024-03-11T21:40:16.866813+0000","last_active":"2024-03-11T21:42:22.703963+0000","last_peered":"2024-03-11T21:42:22.703963+0000","last_clean":"2024-03-11T21:42:22.703963+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:42:22.703963+0000","last_undegraded":"2024-03-11T21:42:22.703963+0000","last_fullsized":"2024-03-11T21:42:22.703963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:16.866545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:16.866545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.400025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1564,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.669258+0000","last_change":"2024-03-11T21:41:53.585801+0000","last_active":"2024-03-11T21:42:22.669258+0000","last_peered":"2024-03-11T21:42:22.669258+0000","last_clean":"2024-03-11T21:42:22.669258+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:42:22.669258+0000","last_undegraded":"2024-03-11T21:42:22.669258+0000","last_fullsized":"2024-03-11T21:42:22.669258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.585657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:22.256998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1411,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704459+0000","last_change":"2024-03-11T21:40:18.842726+0000","last_active":"2024-03-11T21:42:22.704459+0000","last_peered":"2024-03-11T21:42:22.704459+0000","last_clean":"2024-03-11T21:42:22.704459+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:42:22.704459+0000","last_undegraded":"2024-03-11T21:42:22.704459+0000","last_fullsized":"2024-03-11T21:42:22.704459+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.842520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.842520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":1541,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.612764+0000","last_change":"2024-03-11T21:39:59.292770+0000","last_active":"2024-03-11T21:42:19.612764+0000","last_peered":"2024-03-11T21:42:19.612764+0000","last_clean":"2024-03-11T21:42:19.612764+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:42:19.612764+0000","last_undegraded":"2024-03-11T21:42:19.612764+0000","last_fullsized":"2024-03-11T21:42:19.612764+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:39:59.292520+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.292520+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1584,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708336+0000","last_change":"2024-03-11T21:40:24.841665+0000","last_active":"2024-03-11T21:42:22.708336+0000","last_peered":"2024-03-11T21:42:22.708336+0000","last_clean":"2024-03-11T21:42:22.708336+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:42:22.708336+0000","last_undegraded":"2024-03-11T21:42:22.708336+0000","last_fullsized":"2024-03-11T21:42:22.708336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1848,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706309+0000","last_change":"2024-03-11T21:41:33.752080+0000","last_active":"2024-03-11T21:42:22.706309+0000","last_peered":"2024-03-11T21:42:22.706309+0000","last_clean":"2024-03-11T21:42:22.706309+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:42:22.706309+0000","last_undegraded":"2024-03-11T21:42:22.706309+0000","last_fullsized":"2024-03-11T21:42:22.706309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:33.751930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.751930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:38.500379+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1555,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706155+0000","last_change":"2024-03-11T21:40:13.900758+0000","last_active":"2024-03-11T21:42:22.706155+0000","last_peered":"2024-03-11T21:42:22.706155+0000","last_clean":"2024-03-11T21:42:22.706155+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:42:22.706155+0000","last_undegraded":"2024-03-11T21:42:22.706155+0000","last_fullsized":"2024-03-11T21:42:22.706155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:40:13.900659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.900659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1433,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708864+0000","last_change":"2024-03-11T21:41:56.288055+0000","last_active":"2024-03-11T21:42:22.708864+0000","last_peered":"2024-03-11T21:42:22.708864+0000","last_clean":"2024-03-11T21:42:22.708864+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:42:22.708864+0000","last_undegraded":"2024-03-11T21:42:22.708864+0000","last_fullsized":"2024-03-11T21:42:22.708864+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.287964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.287964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.371511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1605,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.661412+0000","last_change":"2024-03-11T21:38:49.931344+0000","last_active":"2024-03-11T21:42:22.661412+0000","last_peered":"2024-03-11T21:42:22.661412+0000","last_clean":"2024-03-11T21:42:22.661412+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:42:22.661412+0000","last_undegraded":"2024-03-11T21:42:22.661412+0000","last_fullsized":"2024-03-11T21:42:22.661412+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:38:49.930664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:38:49.930664+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1577,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.663303+0000","last_change":"2024-03-11T21:39:58.005873+0000","last_active":"2024-03-11T21:42:22.663303+0000","last_peered":"2024-03-11T21:42:22.663303+0000","last_clean":"2024-03-11T21:42:22.663303+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:42:22.663303+0000","last_undegraded":"2024-03-11T21:42:22.663303+0000","last_fullsized":"2024-03-11T21:42:22.663303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:39:58.005747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.005747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1578,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716385+0000","last_change":"2024-03-11T21:41:41.375255+0000","last_active":"2024-03-11T21:42:22.716385+0000","last_peered":"2024-03-11T21:42:22.716385+0000","last_clean":"2024-03-11T21:42:22.716385+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:42:22.716385+0000","last_undegraded":"2024-03-11T21:42:22.716385+0000","last_fullsized":"2024-03-11T21:42:22.716385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.375162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.174376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1447,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.617755+0000","last_change":"2024-03-11T21:41:49.263645+0000","last_active":"2024-03-11T21:42:19.617755+0000","last_peered":"2024-03-11T21:42:19.617755+0000","last_clean":"2024-03-11T21:42:19.617755+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:42:19.617755+0000","last_undegraded":"2024-03-11T21:42:19.617755+0000","last_fullsized":"2024-03-11T21:42:19.617755+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.263416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.236541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1411,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.703673+0000","last_change":"2024-03-11T21:41:23.491021+0000","last_active":"2024-03-11T21:42:22.703673+0000","last_peered":"2024-03-11T21:42:22.703673+0000","last_clean":"2024-03-11T21:42:22.703673+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:42:22.703673+0000","last_undegraded":"2024-03-11T21:42:22.703673+0000","last_fullsized":"2024-03-11T21:42:22.703673+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.490850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.490850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.079493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1587,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.714261+0000","last_change":"2024-03-11T21:40:05.512627+0000","last_active":"2024-03-11T21:42:22.714261+0000","last_peered":"2024-03-11T21:42:22.714261+0000","last_clean":"2024-03-11T21:42:22.714261+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:42:22.714261+0000","last_undegraded":"2024-03-11T21:42:22.714261+0000","last_fullsized":"2024-03-11T21:42:22.714261+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:40:05.512463+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.512463+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1609,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704635+0000","last_change":"2024-03-11T21:41:53.683571+0000","last_active":"2024-03-11T21:42:22.704635+0000","last_peered":"2024-03-11T21:42:22.704635+0000","last_clean":"2024-03-11T21:42:22.704635+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:42:22.704635+0000","last_undegraded":"2024-03-11T21:42:22.704635+0000","last_fullsized":"2024-03-11T21:42:22.704635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.683465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.683465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.982276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1441,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.618162+0000","last_change":"2024-03-11T21:41:17.344885+0000","last_active":"2024-03-11T21:42:19.618162+0000","last_peered":"2024-03-11T21:42:19.618162+0000","last_clean":"2024-03-11T21:42:19.618162+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:42:19.618162+0000","last_undegraded":"2024-03-11T21:42:19.618162+0000","last_fullsized":"2024-03-11T21:42:19.618162+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.344687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.344687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.727912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1591,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659884+0000","last_change":"2024-03-11T21:41:19.561625+0000","last_active":"2024-03-11T21:42:22.659884+0000","last_peered":"2024-03-11T21:42:22.659884+0000","last_clean":"2024-03-11T21:42:22.659884+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:42:22.659884+0000","last_undegraded":"2024-03-11T21:42:22.659884+0000","last_fullsized":"2024-03-11T21:42:22.659884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.561348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.561348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:44.036445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1445,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.655906+0000","last_change":"2024-03-11T21:41:55.738509+0000","last_active":"2024-03-11T21:42:22.655906+0000","last_peered":"2024-03-11T21:42:22.655906+0000","last_clean":"2024-03-11T21:42:22.655906+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:42:22.655906+0000","last_undegraded":"2024-03-11T21:42:22.655906+0000","last_fullsized":"2024-03-11T21:42:22.655906+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.738282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.398264+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"714'19","reported_seq":1635,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.657169+0000","last_change":"2024-03-11T21:40:13.889468+0000","last_active":"2024-03-11T21:42:22.657169+0000","last_peered":"2024-03-11T21:42:22.657169+0000","last_clean":"2024-03-11T21:42:22.657169+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:42:22.657169+0000","last_undegraded":"2024-03-11T21:42:22.657169+0000","last_fullsized":"2024-03-11T21:42:22.657169+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'19","last_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_deep_scrub":"714'19","last_deep_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.889354+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1575,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.668839+0000","last_change":"2024-03-11T21:39:48.014842+0000","last_active":"2024-03-11T21:42:22.668839+0000","last_peered":"2024-03-11T21:42:22.668839+0000","last_clean":"2024-03-11T21:42:22.668839+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:42:22.668839+0000","last_undegraded":"2024-03-11T21:42:22.668839+0000","last_fullsized":"2024-03-11T21:42:22.668839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:48.014741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:39:48.014741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1593,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.658931+0000","last_change":"2024-03-11T21:41:54.741470+0000","last_active":"2024-03-11T21:42:22.658931+0000","last_peered":"2024-03-11T21:42:22.658931+0000","last_clean":"2024-03-11T21:42:22.658931+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:42:22.658931+0000","last_undegraded":"2024-03-11T21:42:22.658931+0000","last_fullsized":"2024-03-11T21:42:22.658931+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.741251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.722799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1565,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671014+0000","last_change":"2024-03-11T21:41:54.630605+0000","last_active":"2024-03-11T21:42:22.671014+0000","last_peered":"2024-03-11T21:42:22.671014+0000","last_clean":"2024-03-11T21:42:22.671014+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:42:22.671014+0000","last_undegraded":"2024-03-11T21:42:22.671014+0000","last_fullsized":"2024-03-11T21:42:22.671014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.630386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.630386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.496284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1420,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706746+0000","last_change":"2024-03-11T21:41:31.510028+0000","last_active":"2024-03-11T21:42:22.706746+0000","last_peered":"2024-03-11T21:42:22.706746+0000","last_clean":"2024-03-11T21:42:22.706746+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:42:22.706746+0000","last_undegraded":"2024-03-11T21:42:22.706746+0000","last_fullsized":"2024-03-11T21:42:22.706746+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.509821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.509821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.206142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1606,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.710191+0000","last_change":"2024-03-11T21:40:28.925505+0000","last_active":"2024-03-11T21:42:22.710191+0000","last_peered":"2024-03-11T21:42:22.710191+0000","last_clean":"2024-03-11T21:42:22.710191+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:42:22.710191+0000","last_undegraded":"2024-03-11T21:42:22.710191+0000","last_fullsized":"2024-03-11T21:42:22.710191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:40:28.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.925267+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1573,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.705606+0000","last_change":"2024-03-11T21:41:46.401421+0000","last_active":"2024-03-11T21:42:22.705606+0000","last_peered":"2024-03-11T21:42:22.705606+0000","last_clean":"2024-03-11T21:42:22.705606+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:42:22.705606+0000","last_undegraded":"2024-03-11T21:42:22.705606+0000","last_fullsized":"2024-03-11T21:42:22.705606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.401211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.401211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.564792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1570,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708898+0000","last_change":"2024-03-11T21:41:53.275977+0000","last_active":"2024-03-11T21:42:22.708898+0000","last_peered":"2024-03-11T21:42:22.708898+0000","last_clean":"2024-03-11T21:42:22.708898+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:42:22.708898+0000","last_undegraded":"2024-03-11T21:42:22.708898+0000","last_fullsized":"2024-03-11T21:42:22.708898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.275834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.275834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1580,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709405+0000","last_change":"2024-03-11T21:40:25.590355+0000","last_active":"2024-03-11T21:42:22.709405+0000","last_peered":"2024-03-11T21:42:22.709405+0000","last_clean":"2024-03-11T21:42:22.709405+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:42:22.709405+0000","last_undegraded":"2024-03-11T21:42:22.709405+0000","last_fullsized":"2024-03-11T21:42:22.709405+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.590246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.590246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1448,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.659058+0000","last_change":"2024-03-11T21:41:22.601767+0000","last_active":"2024-03-11T21:42:22.659058+0000","last_peered":"2024-03-11T21:42:22.659058+0000","last_clean":"2024-03-11T21:42:22.659058+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:42:22.659058+0000","last_undegraded":"2024-03-11T21:42:22.659058+0000","last_fullsized":"2024-03-11T21:42:22.659058+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.601690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.601690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.459593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1641,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.709995+0000","last_change":"2024-03-11T21:41:51.290471+0000","last_active":"2024-03-11T21:42:22.709995+0000","last_peered":"2024-03-11T21:42:22.709995+0000","last_clean":"2024-03-11T21:42:22.709995+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:42:22.709995+0000","last_undegraded":"2024-03-11T21:42:22.709995+0000","last_fullsized":"2024-03-11T21:42:22.709995+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:41:51.290324+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.290324+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.691718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1576,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.707013+0000","last_change":"2024-03-11T21:40:06.837164+0000","last_active":"2024-03-11T21:42:22.707013+0000","last_peered":"2024-03-11T21:42:22.707013+0000","last_clean":"2024-03-11T21:42:22.707013+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:42:22.707013+0000","last_undegraded":"2024-03-11T21:42:22.707013+0000","last_fullsized":"2024-03-11T21:42:22.707013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:06.836871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.836871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1566,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.704137+0000","last_change":"2024-03-11T21:40:04.809344+0000","last_active":"2024-03-11T21:42:22.704137+0000","last_peered":"2024-03-11T21:42:22.704137+0000","last_clean":"2024-03-11T21:42:22.704137+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:42:22.704137+0000","last_undegraded":"2024-03-11T21:42:22.704137+0000","last_fullsized":"2024-03-11T21:42:22.704137+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:04.809056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.809056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1575,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.708891+0000","last_change":"2024-03-11T21:41:47.226377+0000","last_active":"2024-03-11T21:42:22.708891+0000","last_peered":"2024-03-11T21:42:22.708891+0000","last_clean":"2024-03-11T21:42:22.708891+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:42:22.708891+0000","last_undegraded":"2024-03-11T21:42:22.708891+0000","last_fullsized":"2024-03-11T21:42:22.708891+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:47.226166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:47.226166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.255553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1420,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.706113+0000","last_change":"2024-03-11T21:40:02.830965+0000","last_active":"2024-03-11T21:42:22.706113+0000","last_peered":"2024-03-11T21:42:22.706113+0000","last_clean":"2024-03-11T21:42:22.706113+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:42:22.706113+0000","last_undegraded":"2024-03-11T21:42:22.706113+0000","last_fullsized":"2024-03-11T21:42:22.706113+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.830743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.830743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.399893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1618,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716816+0000","last_change":"2024-03-11T21:40:06.481218+0000","last_active":"2024-03-11T21:42:22.716816+0000","last_peered":"2024-03-11T21:42:22.716816+0000","last_clean":"2024-03-11T21:42:22.716816+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:42:22.716816+0000","last_undegraded":"2024-03-11T21:42:22.716816+0000","last_fullsized":"2024-03-11T21:42:22.716816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:40:06.481094+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.481094+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1577,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671071+0000","last_change":"2024-03-11T21:41:49.603979+0000","last_active":"2024-03-11T21:42:22.671071+0000","last_peered":"2024-03-11T21:42:22.671071+0000","last_clean":"2024-03-11T21:42:22.671071+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:42:22.671071+0000","last_undegraded":"2024-03-11T21:42:22.671071+0000","last_fullsized":"2024-03-11T21:42:22.671071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.603815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.603815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:55.515634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1437,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.613318+0000","last_change":"2024-03-11T21:40:20.395801+0000","last_active":"2024-03-11T21:42:19.613318+0000","last_peered":"2024-03-11T21:42:19.613318+0000","last_clean":"2024-03-11T21:42:19.613318+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:42:19.613318+0000","last_undegraded":"2024-03-11T21:42:19.613318+0000","last_fullsized":"2024-03-11T21:42:19.613318+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.395704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:22.261787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1598,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.656991+0000","last_change":"2024-03-11T21:41:33.725278+0000","last_active":"2024-03-11T21:42:22.656991+0000","last_peered":"2024-03-11T21:42:22.656991+0000","last_clean":"2024-03-11T21:42:22.656991+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:42:22.656991+0000","last_undegraded":"2024-03-11T21:42:22.656991+0000","last_fullsized":"2024-03-11T21:42:22.656991+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.725010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.725010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.536659+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1295,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-11T21:42:19.623665+0000","last_change":"2024-03-11T21:41:54.179283+0000","last_active":"2024-03-11T21:42:19.623665+0000","last_peered":"2024-03-11T21:42:19.623665+0000","last_clean":"2024-03-11T21:42:19.623665+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:42:19.623665+0000","last_undegraded":"2024-03-11T21:42:19.623665+0000","last_fullsized":"2024-03-11T21:42:19.623665+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.179012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.179012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.476815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1603,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.671628+0000","last_change":"2024-03-11T21:40:15.377800+0000","last_active":"2024-03-11T21:42:22.671628+0000","last_peered":"2024-03-11T21:42:22.671628+0000","last_clean":"2024-03-11T21:42:22.671628+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:42:22.671628+0000","last_undegraded":"2024-03-11T21:42:22.671628+0000","last_fullsized":"2024-03-11T21:42:22.671628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:40:15.377635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.377635+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.720031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1587,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.716828+0000","last_change":"2024-03-11T21:40:04.478158+0000","last_active":"2024-03-11T21:42:22.716828+0000","last_peered":"2024-03-11T21:42:22.716828+0000","last_clean":"2024-03-11T21:42:22.716828+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:42:22.716828+0000","last_undegraded":"2024-03-11T21:42:22.716828+0000","last_fullsized":"2024-03-11T21:42:22.716828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:40:04.477919+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.477919+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1602,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.655912+0000","last_change":"2024-03-11T21:41:27.668035+0000","last_active":"2024-03-11T21:42:22.655912+0000","last_peered":"2024-03-11T21:42:22.655912+0000","last_clean":"2024-03-11T21:42:22.655912+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:42:22.655912+0000","last_undegraded":"2024-03-11T21:42:22.655912+0000","last_fullsized":"2024-03-11T21:42:22.655912+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.667840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.667840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.798307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1555,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.668763+0000","last_change":"2024-03-11T21:38:39.757621+0000","last_active":"2024-03-11T21:42:22.668763+0000","last_peered":"2024-03-11T21:42:22.668763+0000","last_clean":"2024-03-11T21:42:22.668763+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:42:22.668763+0000","last_undegraded":"2024-03-11T21:42:22.668763+0000","last_fullsized":"2024-03-11T21:42:22.668763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:39.757313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:38:39.757313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.719715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":329,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":7},{"poolid":313,"num_pg":8,"stat_sum":{"num_bytes":4000249,"num_objects":37,"num_object_clones":1,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":57,"num_write_kb":3959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":24,"acting":24,"num_store_stats":7},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1138,"num_read_kb":1133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841779,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105416,"kb_used_data":15924,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93639736,"statfs":{"total":95995035648,"available":95887089664,"internally_reserved":0,"allocated":16306176,"data_stored":11107454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168828,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89184,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655968,"statfs":{"total":95995035648,"available":95903711232,"internally_reserved":0,"allocated":13893632,"data_stored":8725458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352450,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53904,"kb_used_data":13500,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93691248,"statfs":{"total":95995035648,"available":95939837952,"internally_reserved":0,"allocated":13824000,"data_stored":8662659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738683,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82888,"kb_used_data":15592,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93662264,"statfs":{"total":95995035648,"available":95910158336,"internally_reserved":0,"allocated":15966208,"data_stored":10785739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378918,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95040,"kb_used_data":13856,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93650112,"statfs":{"total":95995035648,"available":95897714688,"internally_reserved":0,"allocated":14188544,"data_stored":9017273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117274,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77292,"kb_used_data":14144,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93667860,"statfs":{"total":95995035648,"available":95915888640,"internally_reserved":0,"allocated":14483456,"data_stored":9314645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855632,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100968,"kb_used_data":15612,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93644184,"statfs":{"total":95995035648,"available":95891644416,"internally_reserved":0,"allocated":15986688,"data_stored":10781933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495878,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89748,"kb_used_data":14132,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93655404,"statfs":{"total":95995035648,"available":95903133696,"internally_reserved":0,"allocated":14471168,"data_stored":9307549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2088960,"data_stored":2088960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2084864,"data_stored":2084864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2093056,"data_stored":2093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":329,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:42:28.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:42:28.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[21512]: cluster 2024-03-11T21:42:27.620532+0000 mgr.y (mgr.14550) 755 : cluster 0 pgmap v1300: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[21512]: audit 2024-03-11T21:42:28.557931+0000 mon.a (mon.0) 3794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[21512]: cluster 2024-03-11T21:42:28.562131+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e840: 8 total, 8 up, 8 in 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[21512]: audit 2024-03-11T21:42:28.564116+0000 mon.c (mon.2) 1304 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[21512]: audit 2024-03-11T21:42:28.564706+0000 mon.a (mon.0) 3796 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[29284]: cluster 2024-03-11T21:42:27.620532+0000 mgr.y (mgr.14550) 755 : cluster 0 pgmap v1300: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[29284]: audit 2024-03-11T21:42:28.557931+0000 mon.a (mon.0) 3794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:42:28.824 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[29284]: cluster 2024-03-11T21:42:28.562131+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e840: 8 total, 8 up, 8 in 2024-03-11T21:42:28.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[29284]: audit 2024-03-11T21:42:28.564116+0000 mon.c (mon.2) 1304 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.825 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:28 smithi143 bash[29284]: audit 2024-03-11T21:42:28.564706+0000 mon.a (mon.0) 3796 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.835 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:28.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:28.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:28 smithi203 bash[23654]: cluster 2024-03-11T21:42:27.620532+0000 mgr.y (mgr.14550) 755 : cluster 0 pgmap v1300: 246 pgs: 246 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:28.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:28 smithi203 bash[23654]: audit 2024-03-11T21:42:28.557931+0000 mon.a (mon.0) 3794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:42:28.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:28 smithi203 bash[23654]: cluster 2024-03-11T21:42:28.562131+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e840: 8 total, 8 up, 8 in 2024-03-11T21:42:28.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:28 smithi203 bash[23654]: audit 2024-03-11T21:42:28.564116+0000 mon.c (mon.2) 1304 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:28 smithi203 bash[23654]: audit 2024-03-11T21:42:28.564706+0000 mon.a (mon.0) 3796 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:42:28.965 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:29.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:29.092 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:29.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:29.207 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:29.221 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:29.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- 172.21.15.143:0/60680358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348104290 msgr2=0x7f7348104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 --2- 172.21.15.143:0/60680358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348104290 0x7f7348104670 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f73440099c0 tx=0x7f7344009cd0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- 172.21.15.143:0/60680358 shutdown_connections 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 --2- 172.21.15.143:0/60680358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348104290 0x7f7348104670 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 --2- 172.21.15.143:0/60680358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348109370 0x7f7348110f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 --2- 172.21.15.143:0/60680358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 0x7f7348108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- 172.21.15.143:0/60680358 >> 172.21.15.143:0/60680358 conn(0x7f73480ffb40 msgr2=0x7f7348101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- 172.21.15.143:0/60680358 shutdown_connections 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- 172.21.15.143:0/60680358 wait complete. 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 librados: starting msgr at 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 librados: starting objecter 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 Processor -- start 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 -- start start 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 librados: setting wanted keys 2024-03-11T21:42:29.262 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.275+0000 7f734f6d4700 1 librados: calling monclient init 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348104290 0x7f7348199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 0x7f734819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348113810 con 0x7f7348109370 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348113690 con 0x7f7348104290 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7348113990 con 0x7f7348104c40 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57240/0 (socket says 172.21.15.143:57240) 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 -- 172.21.15.143:0/4120257785 learned_addr learned my addr 172.21.15.143:0/4120257785 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734cc6f700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 0x7f734819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734d470700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348104290 0x7f7348199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 -- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 msgr2=0x7f734819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 0x7f734819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 -- 172.21.15.143:0/4120257785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348104290 msgr2=0x7f7348199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348104290 0x7f7348199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 -- 172.21.15.143:0/4120257785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734819ee20 con 0x7f7348109370 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734dc71700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f7340007d70 tx=0x7f734000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 10 client.?.objecter ms_handle_connect 0x7f7348109370 2024-03-11T21:42:29.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:29.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340016020 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f734000ad40 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734819f110 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340012e60 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 10 client.30861.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73481a6990 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 librados: init done 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734804faa0 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7340019020 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 0x7f7334071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(840..840 src has 87..840) v4 ==== 10287+0+0 (secure 0 0 0) 0x7f734008ec40 con 0x7f7348109370 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 10 client.30861.objecter ms_dispatch 0x7f7348001550 osd_map(840..840 src has 87..840) v4 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 3 client.30861.objecter handle_osd_map got epochs [840,840] > 0 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 3 client.30861.objecter handle_osd_map decoding full epoch 840 2024-03-11T21:42:29.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734d470700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 0x7f7334071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 20 client.30861.objecter dump_active .. 0 homeless 2024-03-11T21:42:29.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f734d470700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 0x7f7334071ce0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f73440059d0 tx=0x7f7344005940 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.279+0000 7f733e7fc700 10 client.30861.objecter ms_handle_connect 0x7f733406f820 2024-03-11T21:42:29.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.287+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7340020020 con 0x7f7348109370 2024-03-11T21:42:29.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:29.346 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:29.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:29.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 -- 172.21.15.143:0/882522584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec100fc0 msgr2=0x7f3cec1013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/882522584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec100fc0 0x7f3cec1013a0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40099c0 tx=0x7f3ce4009cd0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 -- 172.21.15.143:0/882522584 shutdown_connections 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/882522584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec100fc0 0x7f3cec1013a0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/882522584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec1060a0 0x7f3cec10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/882522584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec101970 0x7f3cec1059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 -- 172.21.15.143:0/882522584 >> 172.21.15.143:0/882522584 conn(0x7f3cec077de0 msgr2=0x7f3cec0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 -- 172.21.15.143:0/882522584 shutdown_connections 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 -- 172.21.15.143:0/882522584 wait complete. 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 librados: starting msgr at 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 librados: starting objecter 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.483+0000 7f3cf24ed700 1 Processor -- start 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- start start 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 librados: setting wanted keys 2024-03-11T21:42:29.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 librados: calling monclient init 2024-03-11T21:42:29.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec100fc0 0x7f3cec19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec101970 0x7f3cec19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec110550 con 0x7f3cec1060a0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec1103d0 con 0x7f3cec100fc0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec1106d0 con 0x7f3cec101970 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec101970 0x7f3cec19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57264/0 (socket says 172.21.15.143:57264) 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 learned_addr learned my addr 172.21.15.143:0/4067508717 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec100fc0 0x7f3cec19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec101970 msgr2=0x7f3cec19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec101970 0x7f3cec19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec100fc0 msgr2=0x7f3cec19d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec100fc0 0x7f3cec19d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec198220 con 0x7f3cec1060a0 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cec100fc0 0x7f3cec19d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:42:29.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cec101970 0x7f3cec19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c370 tx=0x7f3cdc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 10 client.?.objecter ms_handle_connect 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00ce20 con 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3cdc0090d0 con 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec1984b0 con 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 10 client.30867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0046b0 con 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec102990 con 0x7f3cec1060a0 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 librados: init done 2024-03-11T21:42:29.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3cec1024a0 con 0x7f3cec1060a0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3cdc01a020 con 0x7f3cec1060a0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.487+0000 7f3ce97fa700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 0x7f3cd8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(840..840 src has 87..840) v4 ==== 10287+0+0 (secure 0 0 0) 0x7f3cdc08e740 con 0x7f3cec1060a0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 osd_map(840..840 src has 87..840) v4 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 3 client.30867.objecter handle_osd_map got epochs [840,840] > 0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 3 client.30867.objecter handle_osd_map decoding full epoch 840 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 0x7f3cd8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 20 client.30867.objecter dump_active .. 0 homeless 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v840) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3cdc021060 con 0x7f3cec1060a0 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 1 2024-03-11T21:42:29.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3cec0629f0 0x7f3cec062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 1 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 0 1 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 4 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 10 client.30867.objecter _send_command 1 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec04faa0 con 0x7f3cec0629f0 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3cec0629f0 0x7f3cec062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 0x7f3cd8071ce0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4000c00 tx=0x7f3ce4019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cd806f820 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3cec0629f0 0x7f3cec062dd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec0629f0 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec04faa0 con 0x7f3cec0629f0 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 1 = system:0 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 4 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.491+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 0 1 2024-03-11T21:42:29.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:29.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 2 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 2 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec198ff0 osd=0 3 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 0 2 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 4 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 10 client.30867.objecter _send_command 2 2024-03-11T21:42:29.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.507+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec062470 con 0x7f3cec0629f0 2024-03-11T21:42:29.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec062470 con 0x7f3cec0629f0 2024-03-11T21:42:29.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 4 2024-03-11T21:42:29.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 0 2 2024-03-11T21:42:29.495 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:42:29.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 3 2024-03-11T21:42:29.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3cec1098b0 0x7f3cec102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 1 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 4 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 10 client.30867.objecter _send_command 3 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec104920 con 0x7f3cec1098b0 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3cec1098b0 0x7f3cec102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec1098b0 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3cec1098b0 0x7f3cec102c40 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec104920 con 0x7f3cec1098b0 2024-03-11T21:42:29.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:42:29.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 3 = system:0 2024-03-11T21:42:29.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 4 2024-03-11T21:42:29.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.511+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 1 3 2024-03-11T21:42:29.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.515+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f73481a6c20 con 0x7f7348109370 2024-03-11T21:42:29.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 4 2024-03-11T21:42:29.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 4 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec02d830 osd=1 3 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 1 4 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 4 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 10 client.30867.objecter _send_command 4 2024-03-11T21:42:29.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec02dbf0 con 0x7f3cec1098b0 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec02dbf0 con 0x7f3cec1098b0 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 4 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 1 4 2024-03-11T21:42:29.516 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 5 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3cec108f90 0x7f3cec02ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 5 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 2 5 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 4 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 10 client.30867.objecter _send_command 5 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec02d3c0 con 0x7f3cec108f90 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3cec108f90 0x7f3cec02ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec108f90 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.531+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3cec108f90 0x7f3cec02ce80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.535+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec02d3c0 con 0x7f3cec108f90 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.535+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.535+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 5 = system:0 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.535+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 4 2024-03-11T21:42:29.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.535+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 2 5 2024-03-11T21:42:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 6 2024-03-11T21:42:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 6 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec108e20 osd=2 3 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 2 6 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 4 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 10 client.30867.objecter _send_command 6 2024-03-11T21:42:29.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec1aec10 con 0x7f3cec108f90 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec1aec10 con 0x7f3cec108f90 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 4 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 2 6 2024-03-11T21:42:29.536 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 7 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3cec02ca70 0x7f3cec1b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 7 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 3 7 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 4 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 10 client.30867.objecter _send_command 7 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec103fc0 con 0x7f3cec02ca70 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.551+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3cec02ca70 0x7f3cec1b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec02ca70 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3cec02ca70 0x7f3cec1b1f50 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec103fc0 con 0x7f3cec02ca70 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 7 = system:0 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 4 2024-03-11T21:42:29.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 3 7 2024-03-11T21:42:29.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.555+0000 7f733e7fc700 1 -- 172.21.15.143:0/4120257785 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v841) v1 ==== 97+0+0 (secure 0 0 0) 0x7f734005b680 con 0x7f7348109370 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.559+0000 7f734f6d4700 10 librados: watch_flush enter 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.559+0000 7f734f6d4700 10 librados: watch_flush exit 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.559+0000 7f734f6d4700 20 client.30861.objecter shutdown clearing up homeless session... 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.559+0000 7f734f6d4700 10 client.30861.objecter successfully canceled tick 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 msgr2=0x7f7334071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 0x7f7334071ce0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f73440059d0 tx=0x7f7344005940 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 msgr2=0x7f734819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f7340007d70 tx=0x7f734000a340 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 shutdown_connections 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f733406f820 0x7f7334071ce0 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7348109370 0x7f734819e6e0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348104290 0x7f7348199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 --2- 172.21.15.143:0/4120257785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7348104c40 0x7f734819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 >> 172.21.15.143:0/4120257785 conn(0x7f73480ffb40 msgr2=0x7f73481002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 shutdown_connections 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 -- 172.21.15.143:0/4120257785 wait complete. 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.563+0000 7f734f6d4700 1 librados: shutdown 2024-03-11T21:42:29.549 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 8 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 8 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec02c900 osd=3 3 2024-03-11T21:42:29.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 3 8 2024-03-11T21:42:29.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 4 2024-03-11T21:42:29.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 10 client.30867.objecter _send_command 8 2024-03-11T21:42:29.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec1033d0 con 0x7f3cec02ca70 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec1033d0 con 0x7f3cec02ca70 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 4 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 3 8 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:42:29.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 9 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3cec1afaa0 0x7f3cec1b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 9 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 4 9 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 4 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 10 client.30867.objecter _send_command 9 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec103330 con 0x7f3cec1afaa0 2024-03-11T21:42:29.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.579+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3cec1afaa0 0x7f3cec1b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3cec1afaa0 0x7f3cec1b5620 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec1afaa0 2024-03-11T21:42:29.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec103330 con 0x7f3cec1afaa0 2024-03-11T21:42:29.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:42:29.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 9 = system:0 2024-03-11T21:42:29.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 4 2024-03-11T21:42:29.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.583+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 4 9 2024-03-11T21:42:29.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 10 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 10 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec103180 osd=4 3 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 4 10 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 4 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 10 client.30867.objecter _send_command 10 2024-03-11T21:42:29.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.599+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec103330 con 0x7f3cec1afaa0 2024-03-11T21:42:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec103330 con 0x7f3cec1afaa0 2024-03-11T21:42:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 4 2024-03-11T21:42:29.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 4 10 2024-03-11T21:42:29.590 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:42:29.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 11 2024-03-11T21:42:29.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3cec1b7840 0x7f3cec1b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 11 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 5 11 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 4 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 10 client.30867.objecter _send_command 11 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec1bb690 con 0x7f3cec1b7840 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3cec1b7840 0x7f3cec1b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec1b7840 2024-03-11T21:42:29.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ceb7fe700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3cec1b7840 0x7f3cec1b7ca0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec1bb690 con 0x7f3cec1b7840 2024-03-11T21:42:29.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:42:29.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 11 = system:0 2024-03-11T21:42:29.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 4 2024-03-11T21:42:29.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.607+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 5 11 2024-03-11T21:42:29.600 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:29.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 12 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 12 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec103330 osd=5 3 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 5 12 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 4 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 10 client.30867.objecter _send_command 12 2024-03-11T21:42:29.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.631+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec1b5b60 con 0x7f3cec1b7840 2024-03-11T21:42:29.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec1b5b60 con 0x7f3cec1b7840 2024-03-11T21:42:29.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 4 2024-03-11T21:42:29.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 5 12 2024-03-11T21:42:29.619 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:42:29.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 13 2024-03-11T21:42:29.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3cec1b6280 0x7f3cec1b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 13 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 6 13 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 4 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 10 client.30867.objecter _send_command 13 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec1b6dc0 con 0x7f3cec1b6280 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3cec1b6280 0x7f3cec1b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec1b6280 2024-03-11T21:42:29.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.635+0000 7f3cf0a8a700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3cec1b6280 0x7f3cec1b6700 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.639+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec1b6dc0 con 0x7f3cec1b6280 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.639+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.639+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 13 = system:0 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.639+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 4 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.639+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 6 13 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 14 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 14 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec1b5b60 osd=6 3 2024-03-11T21:42:29.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 6 14 2024-03-11T21:42:29.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 4 2024-03-11T21:42:29.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 10 client.30867.objecter _send_command 14 2024-03-11T21:42:29.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.655+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec1b6dc0 con 0x7f3cec1b6280 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec1b6dc0 con 0x7f3cec1b6280 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 4 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 6 14 2024-03-11T21:42:29.644 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 15 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3cec1b6f30 0x7f3cec1beab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 15 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 7 15 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 4 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 10 client.30867.objecter _send_command 15 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cec1c1b30 con 0x7f3cec1b6f30 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.659+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3cec1b6f30 0x7f3cec1beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 10 client.30867.objecter ms_handle_connect 0x7f3cec1b6f30 2024-03-11T21:42:29.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3cebfff700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3cec1b6f30 0x7f3cec1beab0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:29.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3cec1c1b30 con 0x7f3cec1b6f30 2024-03-11T21:42:29.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:42:29.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 15 = system:0 2024-03-11T21:42:29.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 4 2024-03-11T21:42:29.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.663+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 7 15 2024-03-11T21:42:29.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 10 client.30867.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign -1 16 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 20 client.30867.objecter _get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_remove -1 16 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 20 client.30867.objecter get_session s=0x7f3cec1b6dc0 osd=7 3 2024-03-11T21:42:29.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 15 client.30867.objecter _session_command_op_assign 7 16 2024-03-11T21:42:29.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 4 2024-03-11T21:42:29.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 10 client.30867.objecter _send_command 16 2024-03-11T21:42:29.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.683+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3cec1c1b30 con 0x7f3cec1b6f30 2024-03-11T21:42:29.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.049989+0000 mgr.y (mgr.14550) 756 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.490431+0000 mon.a (mon.0) 3797 : audit 0 from='client.? 172.21.15.143:0/4067508717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.516541+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: cluster 2024-03-11T21:42:29.558100+0000 mon.a (mon.0) 3799 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.559464+0000 mon.a (mon.0) 3800 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.559531+0000 mon.a (mon.0) 3801 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: cluster 2024-03-11T21:42:29.563146+0000 mon.a (mon.0) 3802 : cluster 0 osdmap e841: 8 total, 8 up, 8 in 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.563720+0000 mon.c (mon.2) 1305 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[21512]: audit 2024-03-11T21:42:29.565127+0000 mon.a (mon.0) 3803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.049989+0000 mgr.y (mgr.14550) 756 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:29.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.490431+0000 mon.a (mon.0) 3797 : audit 0 from='client.? 172.21.15.143:0/4067508717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:29.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.516541+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:29.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: cluster 2024-03-11T21:42:29.558100+0000 mon.a (mon.0) 3799 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:29.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.559464+0000 mon.a (mon.0) 3800 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:42:29.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3ce97fa700 1 -- 172.21.15.143:0/4067508717 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3cec1c1b30 con 0x7f3cec1b6f30 2024-03-11T21:42:29.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3ce97fa700 10 client.30867.objecter ms_dispatch 0x7f3cec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:29.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3ce97fa700 10 client.30867.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:29.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3ce97fa700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 4 2024-03-11T21:42:29.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3ce97fa700 15 client.30867.objecter _session_command_op_remove 7 16 2024-03-11T21:42:29.716 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:42:29.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 librados: watch_flush enter 2024-03-11T21:42:29.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 librados: watch_flush exit 2024-03-11T21:42:29.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.0 2024-03-11T21:42:29.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3cec0629f0 msgr2=0x7f3cec062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3cec0629f0 0x7f3cec062dd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec198ff0 osd=0 1 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.1 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3cec1098b0 msgr2=0x7f3cec102c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3cec1098b0 0x7f3cec102c40 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02d830 osd=1 1 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.2 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3cec108f90 msgr2=0x7f3cec02ce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3cec108f90 0x7f3cec02ce80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec108e20 osd=2 1 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.3 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3cec02ca70 msgr2=0x7f3cec1b1f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3cec02ca70 0x7f3cec1b1f50 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec02c900 osd=3 1 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.4 2024-03-11T21:42:29.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 reap_dead start 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3cec1afaa0 msgr2=0x7f3cec1b5620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3cec1afaa0 0x7f3cec1b5620 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103180 osd=4 1 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.5 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3cec1b7840 msgr2=0x7f3cec1b7ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3cec1b7840 0x7f3cec1b7ca0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec103330 osd=5 1 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.6 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3cec1b6280 msgr2=0x7f3cec1b6700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3cec1b6280 0x7f3cec1b6700 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b5b60 osd=6 1 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter close_session for osd.7 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3cec1b6f30 msgr2=0x7f3cec1beab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3cec1b6f30 0x7f3cec1beab0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter put_session s=0x7f3cec1b6dc0 osd=7 1 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 20 client.30867.objecter shutdown clearing up homeless session... 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf0a8a700 1 -- 172.21.15.143:0/4067508717 reap_dead start 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.699+0000 7f3cf24ed700 10 client.30867.objecter successfully canceled tick 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 msgr2=0x7f3cd8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 0x7f3cd8071ce0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4000c00 tx=0x7f3ce4019040 comp rx=0 tx=0).stop 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 msgr2=0x7f3cec197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c370 tx=0x7f3cdc00c680 comp rx=0 tx=0).stop 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 shutdown_connections 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cec1060a0 0x7f3cec197ac0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 --2- 172.21.15.143:0/4067508717 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3cd806f820 0x7f3cd8071ce0 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 >> 172.21.15.143:0/4067508717 conn(0x7f3cec077de0 msgr2=0x7f3cec06a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 shutdown_connections 2024-03-11T21:42:29.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 -- 172.21.15.143:0/4067508717 wait complete. 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:29.703+0000 7f3cf24ed700 1 librados: shutdown 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:42:29.721 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:42:29.738 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.049989+0000 mgr.y (mgr.14550) 756 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.490431+0000 mon.a (mon.0) 3797 : audit 0 from='client.? 172.21.15.143:0/4067508717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.516541+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: cluster 2024-03-11T21:42:29.558100+0000 mon.a (mon.0) 3799 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.559464+0000 mon.a (mon.0) 3800 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:42:29.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.559531+0000 mon.a (mon.0) 3801 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:29.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: cluster 2024-03-11T21:42:29.563146+0000 mon.a (mon.0) 3802 : cluster 0 osdmap e841: 8 total, 8 up, 8 in 2024-03-11T21:42:29.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.563720+0000 mon.c (mon.2) 1305 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:29 smithi203 bash[23654]: audit 2024-03-11T21:42:29.565127+0000 mon.a (mon.0) 3803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:29.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.559531+0000 mon.a (mon.0) 3801 : audit 1 from='client.? 172.21.15.143:0/4120257785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:29.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: cluster 2024-03-11T21:42:29.563146+0000 mon.a (mon.0) 3802 : cluster 0 osdmap e841: 8 total, 8 up, 8 in 2024-03-11T21:42:29.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.563720+0000 mon.c (mon.2) 1305 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.851 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:29 smithi143 bash[29284]: audit 2024-03-11T21:42:29.565127+0000 mon.a (mon.0) 3803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:42:29.865 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:29.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:29.996 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:30.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:30.127 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:30.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:30.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:30.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:30.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:30.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:30.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:30.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:30.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:30.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:30.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:30 smithi203 bash[23654]: cluster 2024-03-11T21:42:29.620958+0000 mgr.y (mgr.14550) 757 : cluster 0 pgmap v1303: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:42:30.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:30 smithi203 bash[23654]: audit 2024-03-11T21:42:30.568679+0000 mon.a (mon.0) 3804 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:42:30.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:30 smithi203 bash[23654]: cluster 2024-03-11T21:42:30.576445+0000 mon.a (mon.0) 3805 : cluster 0 osdmap e842: 8 total, 8 up, 8 in 2024-03-11T21:42:30.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:30 smithi203 bash[23654]: audit 2024-03-11T21:42:30.674841+0000 mon.c (mon.2) 1306 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:30.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:30 smithi203 bash[23654]: audit 2024-03-11T21:42:30.675263+0000 mon.a (mon.0) 3806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:30.781 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:30.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:30.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[29284]: cluster 2024-03-11T21:42:29.620958+0000 mgr.y (mgr.14550) 757 : cluster 0 pgmap v1303: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:42:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[29284]: audit 2024-03-11T21:42:30.568679+0000 mon.a (mon.0) 3804 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:42:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[29284]: cluster 2024-03-11T21:42:30.576445+0000 mon.a (mon.0) 3805 : cluster 0 osdmap e842: 8 total, 8 up, 8 in 2024-03-11T21:42:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[29284]: audit 2024-03-11T21:42:30.674841+0000 mon.c (mon.2) 1306 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:30.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[29284]: audit 2024-03-11T21:42:30.675263+0000 mon.a (mon.0) 3806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:30.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[21512]: cluster 2024-03-11T21:42:29.620958+0000 mgr.y (mgr.14550) 757 : cluster 0 pgmap v1303: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:42:30.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[21512]: audit 2024-03-11T21:42:30.568679+0000 mon.a (mon.0) 3804 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:42:30.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[21512]: cluster 2024-03-11T21:42:30.576445+0000 mon.a (mon.0) 3805 : cluster 0 osdmap e842: 8 total, 8 up, 8 in 2024-03-11T21:42:30.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[21512]: audit 2024-03-11T21:42:30.674841+0000 mon.c (mon.2) 1306 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:30.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:30 smithi143 bash[21512]: audit 2024-03-11T21:42:30.675263+0000 mon.a (mon.0) 3806 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:31.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:31.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:31.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:31.170 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:31.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:31.302 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:31.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:31.432 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:31.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:31.565 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:31.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:31.696 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:31.799 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:31.821 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:31.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:31.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:31 smithi203 bash[23654]: cluster 2024-03-11T21:42:31.547286+0000 mon.a (mon.0) 3807 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:31.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:31 smithi203 bash[23654]: audit 2024-03-11T21:42:31.572012+0000 mon.a (mon.0) 3808 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:31.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:31 smithi203 bash[23654]: cluster 2024-03-11T21:42:31.580255+0000 mon.a (mon.0) 3809 : cluster 0 osdmap e843: 8 total, 8 up, 8 in 2024-03-11T21:42:31.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:31 smithi203 bash[23654]: audit 2024-03-11T21:42:31.583707+0000 mon.c (mon.2) 1307 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:31 smithi203 bash[23654]: audit 2024-03-11T21:42:31.584871+0000 mon.a (mon.0) 3810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[29284]: cluster 2024-03-11T21:42:31.547286+0000 mon.a (mon.0) 3807 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[29284]: audit 2024-03-11T21:42:31.572012+0000 mon.a (mon.0) 3808 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[29284]: cluster 2024-03-11T21:42:31.580255+0000 mon.a (mon.0) 3809 : cluster 0 osdmap e843: 8 total, 8 up, 8 in 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[29284]: audit 2024-03-11T21:42:31.583707+0000 mon.c (mon.2) 1307 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[29284]: audit 2024-03-11T21:42:31.584871+0000 mon.a (mon.0) 3810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[21512]: cluster 2024-03-11T21:42:31.547286+0000 mon.a (mon.0) 3807 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[21512]: audit 2024-03-11T21:42:31.572012+0000 mon.a (mon.0) 3808 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[21512]: cluster 2024-03-11T21:42:31.580255+0000 mon.a (mon.0) 3809 : cluster 0 osdmap e843: 8 total, 8 up, 8 in 2024-03-11T21:42:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[21512]: audit 2024-03-11T21:42:31.583707+0000 mon.c (mon.2) 1307 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:31 smithi143 bash[21512]: audit 2024-03-11T21:42:31.584871+0000 mon.a (mon.0) 3810 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:31.952 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:32.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:32.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:32.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.215 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:32.318 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.345 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.347 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:32.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:32.476 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:32.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:32.610 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:32.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:32.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: cluster 2024-03-11T21:42:31.621710+0000 mgr.y (mgr.14550) 758 : cluster 0 pgmap v1306: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:32.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: audit 2024-03-11T21:42:32.581237+0000 mon.a (mon.0) 3811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:32.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: cluster 2024-03-11T21:42:32.589002+0000 mon.a (mon.0) 3812 : cluster 0 osdmap e844: 8 total, 8 up, 8 in 2024-03-11T21:42:32.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: audit 2024-03-11T21:42:32.648947+0000 mon.c (mon.2) 1308 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: audit 2024-03-11T21:42:32.649684+0000 mon.a (mon.0) 3813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: audit 2024-03-11T21:42:32.650974+0000 mon.c (mon.2) 1309 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:32 smithi203 bash[23654]: audit 2024-03-11T21:42:32.651659+0000 mon.a (mon.0) 3814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:32.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:32.871 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:32.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: cluster 2024-03-11T21:42:31.621710+0000 mgr.y (mgr.14550) 758 : cluster 0 pgmap v1306: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:32.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: audit 2024-03-11T21:42:32.581237+0000 mon.a (mon.0) 3811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: cluster 2024-03-11T21:42:32.589002+0000 mon.a (mon.0) 3812 : cluster 0 osdmap e844: 8 total, 8 up, 8 in 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: audit 2024-03-11T21:42:32.648947+0000 mon.c (mon.2) 1308 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: audit 2024-03-11T21:42:32.649684+0000 mon.a (mon.0) 3813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: audit 2024-03-11T21:42:32.650974+0000 mon.c (mon.2) 1309 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[29284]: audit 2024-03-11T21:42:32.651659+0000 mon.a (mon.0) 3814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: cluster 2024-03-11T21:42:31.621710+0000 mgr.y (mgr.14550) 758 : cluster 0 pgmap v1306: 246 pgs: 246 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: audit 2024-03-11T21:42:32.581237+0000 mon.a (mon.0) 3811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]': finished 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: cluster 2024-03-11T21:42:32.589002+0000 mon.a (mon.0) 3812 : cluster 0 osdmap e844: 8 total, 8 up, 8 in 2024-03-11T21:42:32.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: audit 2024-03-11T21:42:32.648947+0000 mon.c (mon.2) 1308 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: audit 2024-03-11T21:42:32.649684+0000 mon.a (mon.0) 3813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:32.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: audit 2024-03-11T21:42:32.650974+0000 mon.c (mon.2) 1309 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:32 smithi143 bash[21512]: audit 2024-03-11T21:42:32.651659+0000 mon.a (mon.0) 3814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-142"}]: dispatch 2024-03-11T21:42:32.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:33.002 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:33.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:33.134 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:33.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:33.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:33.395 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:33.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:33.527 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:33.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:33.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:33.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:42:33.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:42:33.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:33.782 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:33.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:33.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:33 smithi203 bash[23654]: cluster 2024-03-11T21:42:32.858817+0000 osd.5 (osd.5) 459 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:33.897 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:33 smithi203 bash[23654]: cluster 2024-03-11T21:42:32.859093+0000 osd.5 (osd.5) 460 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:33.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:34.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:33 smithi143 bash[29284]: cluster 2024-03-11T21:42:32.858817+0000 osd.5 (osd.5) 459 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:33 smithi143 bash[29284]: cluster 2024-03-11T21:42:32.859093+0000 osd.5 (osd.5) 460 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:34.025 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:33 smithi143 bash[21512]: cluster 2024-03-11T21:42:32.858817+0000 osd.5 (osd.5) 459 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:34.026 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:33 smithi143 bash[21512]: cluster 2024-03-11T21:42:32.859093+0000 osd.5 (osd.5) 460 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:34.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:34.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.168 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:34.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.306 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:34.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:34.437 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:34.541 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:34.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:34.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:34.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:34.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:34 smithi203 bash[23654]: cluster 2024-03-11T21:42:33.623975+0000 mgr.y (mgr.14550) 759 : cluster 0 pgmap v1308: 246 pgs: 246 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:42:34.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:34 smithi203 bash[23654]: cluster 2024-03-11T21:42:33.723042+0000 mon.a (mon.0) 3815 : cluster 0 osdmap e845: 8 total, 8 up, 8 in 2024-03-11T21:42:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:34 smithi143 bash[21512]: cluster 2024-03-11T21:42:33.623975+0000 mgr.y (mgr.14550) 759 : cluster 0 pgmap v1308: 246 pgs: 246 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:42:34.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:34 smithi143 bash[21512]: cluster 2024-03-11T21:42:33.723042+0000 mon.a (mon.0) 3815 : cluster 0 osdmap e845: 8 total, 8 up, 8 in 2024-03-11T21:42:34.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:34 smithi143 bash[29284]: cluster 2024-03-11T21:42:33.623975+0000 mgr.y (mgr.14550) 759 : cluster 0 pgmap v1308: 246 pgs: 246 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:42:34.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:34 smithi143 bash[29284]: cluster 2024-03-11T21:42:33.723042+0000 mon.a (mon.0) 3815 : cluster 0 osdmap e845: 8 total, 8 up, 8 in 2024-03-11T21:42:34.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:35.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:35.090 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:35.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:35.220 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:35.266 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:35.266 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:35.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:35.350 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:35.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:35.480 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:35.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:35.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:35.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:35.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:35.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:35.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:35 smithi203 bash[23654]: cluster 2024-03-11T21:42:34.356077+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:36.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:35 smithi203 bash[23654]: cluster 2024-03-11T21:42:34.356502+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:36.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:35 smithi203 bash[23654]: cluster 2024-03-11T21:42:34.725450+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e846: 8 total, 8 up, 8 in 2024-03-11T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:35 smithi203 bash[23654]: audit 2024-03-11T21:42:34.732721+0000 mon.c (mon.2) 1310 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:35 smithi203 bash[23654]: audit 2024-03-11T21:42:34.733419+0000 mon.a (mon.0) 3817 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.017 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:36.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:36.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[21512]: cluster 2024-03-11T21:42:34.356077+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:36.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[21512]: cluster 2024-03-11T21:42:34.356502+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:36.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[21512]: cluster 2024-03-11T21:42:34.725450+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e846: 8 total, 8 up, 8 in 2024-03-11T21:42:36.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[21512]: audit 2024-03-11T21:42:34.732721+0000 mon.c (mon.2) 1310 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.132 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[21512]: audit 2024-03-11T21:42:34.733419+0000 mon.a (mon.0) 3817 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.133 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[29284]: cluster 2024-03-11T21:42:34.356077+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:36.133 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[29284]: cluster 2024-03-11T21:42:34.356502+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:36.133 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[29284]: cluster 2024-03-11T21:42:34.725450+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e846: 8 total, 8 up, 8 in 2024-03-11T21:42:36.133 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[29284]: audit 2024-03-11T21:42:34.732721+0000 mon.c (mon.2) 1310 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.133 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:35 smithi143 bash[29284]: audit 2024-03-11T21:42:34.733419+0000 mon.a (mon.0) 3817 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:36.147 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:36.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:36.278 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:36.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:36.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:36.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.541 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:36.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:36.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:36.779 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: cluster 2024-03-11T21:42:35.624636+0000 mgr.y (mgr.14550) 760 : cluster 0 pgmap v1311: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:35.725319+0000 mon.a (mon.0) 3818 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: cluster 2024-03-11T21:42:35.735399+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e847: 8 total, 8 up, 8 in 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:35.782805+0000 mon.c (mon.2) 1311 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:35.783404+0000 mon.a (mon.0) 3820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: cluster 2024-03-11T21:42:36.549598+0000 mon.a (mon.0) 3821 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:36.553657+0000 mon.a (mon.0) 3822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: cluster 2024-03-11T21:42:36.561823+0000 mon.a (mon.0) 3823 : cluster 0 osdmap e848: 8 total, 8 up, 8 in 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:36.563089+0000 mon.c (mon.2) 1312 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:36.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:36 smithi203 bash[23654]: audit 2024-03-11T21:42:36.567937+0000 mon.a (mon.0) 3824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:36.800 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:36.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:36.932 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:37.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:37.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: cluster 2024-03-11T21:42:35.624636+0000 mgr.y (mgr.14550) 760 : cluster 0 pgmap v1311: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:42:37.047 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:35.725319+0000 mon.a (mon.0) 3818 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: cluster 2024-03-11T21:42:35.735399+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e847: 8 total, 8 up, 8 in 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:35.782805+0000 mon.c (mon.2) 1311 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:35.783404+0000 mon.a (mon.0) 3820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: cluster 2024-03-11T21:42:36.549598+0000 mon.a (mon.0) 3821 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:36.553657+0000 mon.a (mon.0) 3822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: cluster 2024-03-11T21:42:36.561823+0000 mon.a (mon.0) 3823 : cluster 0 osdmap e848: 8 total, 8 up, 8 in 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:36.563089+0000 mon.c (mon.2) 1312 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[21512]: audit 2024-03-11T21:42:36.567937+0000 mon.a (mon.0) 3824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: cluster 2024-03-11T21:42:35.624636+0000 mgr.y (mgr.14550) 760 : cluster 0 pgmap v1311: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:42:37.048 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:35.725319+0000 mon.a (mon.0) 3818 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: cluster 2024-03-11T21:42:35.735399+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e847: 8 total, 8 up, 8 in 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:35.782805+0000 mon.c (mon.2) 1311 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:35.783404+0000 mon.a (mon.0) 3820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: cluster 2024-03-11T21:42:36.549598+0000 mon.a (mon.0) 3821 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:36.553657+0000 mon.a (mon.0) 3822 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: cluster 2024-03-11T21:42:36.561823+0000 mon.a (mon.0) 3823 : cluster 0 osdmap e848: 8 total, 8 up, 8 in 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:36.563089+0000 mon.c (mon.2) 1312 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:37.049 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:36 smithi143 bash[29284]: audit 2024-03-11T21:42:36.567937+0000 mon.a (mon.0) 3824 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:37.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:37.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:37.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:37.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:37.322 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:37.426 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:37.446 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:37.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.570 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:37.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:37.701 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:37.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:37.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: cluster 2024-03-11T21:42:36.966465+0000 osd.1 (osd.1) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: cluster 2024-03-11T21:42:36.966800+0000 osd.1 (osd.1) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: audit 2024-03-11T21:42:37.425391+0000 mon.c (mon.2) 1313 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: audit 2024-03-11T21:42:37.555862+0000 mon.a (mon.0) 3825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: cluster 2024-03-11T21:42:37.559016+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e849: 8 total, 8 up, 8 in 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: audit 2024-03-11T21:42:37.563690+0000 mon.c (mon.2) 1314 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[21512]: audit 2024-03-11T21:42:37.564140+0000 mon.a (mon.0) 3827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: cluster 2024-03-11T21:42:36.966465+0000 osd.1 (osd.1) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: cluster 2024-03-11T21:42:36.966800+0000 osd.1 (osd.1) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:37.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: audit 2024-03-11T21:42:37.425391+0000 mon.c (mon.2) 1313 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:37.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: audit 2024-03-11T21:42:37.555862+0000 mon.a (mon.0) 3825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:37.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: cluster 2024-03-11T21:42:37.559016+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e849: 8 total, 8 up, 8 in 2024-03-11T21:42:37.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: audit 2024-03-11T21:42:37.563690+0000 mon.c (mon.2) 1314 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:37.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:37 smithi143 bash[29284]: audit 2024-03-11T21:42:37.564140+0000 mon.a (mon.0) 3827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: cluster 2024-03-11T21:42:36.966465+0000 osd.1 (osd.1) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: cluster 2024-03-11T21:42:36.966800+0000 osd.1 (osd.1) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: audit 2024-03-11T21:42:37.425391+0000 mon.c (mon.2) 1313 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: audit 2024-03-11T21:42:37.555862+0000 mon.a (mon.0) 3825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: cluster 2024-03-11T21:42:37.559016+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e849: 8 total, 8 up, 8 in 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: audit 2024-03-11T21:42:37.563690+0000 mon.c (mon.2) 1314 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:38.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:37 smithi203 bash[23654]: audit 2024-03-11T21:42:37.564140+0000 mon.a (mon.0) 3827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:42:38.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:38.092 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:38.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:38.221 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:38.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:38.352 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:38.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:38.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:38.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:38.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:38.744 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:38.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: cluster 2024-03-11T21:42:37.365061+0000 osd.0 (osd.0) 436 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: cluster 2024-03-11T21:42:37.365234+0000 osd.0 (osd.0) 437 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: cluster 2024-03-11T21:42:37.625172+0000 mgr.y (mgr.14550) 761 : cluster 0 pgmap v1315: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: cluster 2024-03-11T21:42:38.557580+0000 mon.a (mon.0) 3828 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:38.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: audit 2024-03-11T21:42:38.568707+0000 mon.a (mon.0) 3829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]': finished 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[29284]: cluster 2024-03-11T21:42:38.576889+0000 mon.a (mon.0) 3830 : cluster 0 osdmap e850: 8 total, 8 up, 8 in 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: cluster 2024-03-11T21:42:37.365061+0000 osd.0 (osd.0) 436 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: cluster 2024-03-11T21:42:37.365234+0000 osd.0 (osd.0) 437 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: cluster 2024-03-11T21:42:37.625172+0000 mgr.y (mgr.14550) 761 : cluster 0 pgmap v1315: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: cluster 2024-03-11T21:42:38.557580+0000 mon.a (mon.0) 3828 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: audit 2024-03-11T21:42:38.568707+0000 mon.a (mon.0) 3829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]': finished 2024-03-11T21:42:38.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:38 smithi143 bash[21512]: cluster 2024-03-11T21:42:38.576889+0000 mon.a (mon.0) 3830 : cluster 0 osdmap e850: 8 total, 8 up, 8 in 2024-03-11T21:42:38.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:39.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: cluster 2024-03-11T21:42:37.365061+0000 osd.0 (osd.0) 436 : cluster 0 purged_snaps scrub starts 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: cluster 2024-03-11T21:42:37.365234+0000 osd.0 (osd.0) 437 : cluster 0 purged_snaps scrub ok 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: cluster 2024-03-11T21:42:37.625172+0000 mgr.y (mgr.14550) 761 : cluster 0 pgmap v1315: 246 pgs: 4 creating+peering, 28 unknown, 214 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: cluster 2024-03-11T21:42:38.557580+0000 mon.a (mon.0) 3828 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: audit 2024-03-11T21:42:38.568707+0000 mon.a (mon.0) 3829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-144", "mode": "readproxy"}]': finished 2024-03-11T21:42:39.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:38 smithi203 bash[23654]: cluster 2024-03-11T21:42:38.576889+0000 mon.a (mon.0) 3830 : cluster 0 osdmap e850: 8 total, 8 up, 8 in 2024-03-11T21:42:39.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.135 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:39.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:39.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:39.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.391 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:39.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:39.522 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:39.529 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:39.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:39.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:39.782 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:39.783 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:39.783 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:39.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:39.913 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:39.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:39 smithi143 bash[21512]: audit 2024-03-11T21:42:39.057861+0000 mgr.y (mgr.14550) 762 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:39.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:39 smithi143 bash[29284]: audit 2024-03-11T21:42:39.057861+0000 mgr.y (mgr.14550) 762 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:40.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:40.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:39 smithi203 bash[23654]: audit 2024-03-11T21:42:39.057861+0000 mgr.y (mgr.14550) 762 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:40.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:40.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:40.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:40.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:40.442 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:40.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:40.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:40.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:40.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:40.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:40.834 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:40.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:40.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:41.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:40 smithi203 bash[23654]: cluster 2024-03-11T21:42:39.626323+0000 mgr.y (mgr.14550) 763 : cluster 0 pgmap v1317: 246 pgs: 4 creating+peering, 242 active+clean; 4.4 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:42:41.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.080 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:40 smithi143 bash[21512]: cluster 2024-03-11T21:42:39.626323+0000 mgr.y (mgr.14550) 763 : cluster 0 pgmap v1317: 246 pgs: 4 creating+peering, 242 active+clean; 4.4 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:42:41.081 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:40 smithi143 bash[29284]: cluster 2024-03-11T21:42:39.626323+0000 mgr.y (mgr.14550) 763 : cluster 0 pgmap v1317: 246 pgs: 4 creating+peering, 242 active+clean; 4.4 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:42:41.093 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:41.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.222 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:41.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:41.353 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:41.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.480 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:41.587 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:41.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.739 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:41.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:41.865 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:41.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:41.981 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:41 smithi143 bash[29284]: cluster 2024-03-11T21:42:41.552992+0000 mon.a (mon.0) 3831 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:41.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:41 smithi143 bash[21512]: cluster 2024-03-11T21:42:41.552992+0000 mon.a (mon.0) 3831 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:41.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:42.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:41 smithi203 bash[23654]: cluster 2024-03-11T21:42:41.552992+0000 mon.a (mon.0) 3831 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:42.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:42.128 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:42.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.259 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:42.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.389 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:42.444 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:42.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:42.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.648 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.652 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.41882.0:5 310.6 e660/656 310:602f83fe:::foo:head v 660'4, mlcod=660'1)", 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:07.537732+0000", 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 215.130843484, 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.004262559, 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537732+0000", 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.653 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537732+0000", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537732+0000", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.0999999999999999e-07 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537739+0000", 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.4559999999999997e-06 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.654 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537749+0000", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.077e-05 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.537757+0000", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.9279999999999993e-06 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.538321+0000", 2024-03-11T21:42:42.655 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00056398699999999995 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.538355+0000", 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3995000000000002e-05 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.541857+0000", 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0035016359999999998 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.656 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:07.541994+0000", 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013727699999999999 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:11 314.4 314:20000000:%01:::head [pgnls start_epoch 692] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e692)", 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:39.564650+0000", 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 183.103924788, 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0037248670000000002, 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.657 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 11 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564650+0000", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564650+0000", 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.658 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564652+0000", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6249999999999999e-06 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564658+0000", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.229e-06 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564669+0000", 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0651e-05 2024-03-11T21:42:42.659 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564684+0000", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5498e-05 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564750+0000", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.5998000000000001e-05 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.564755+0000", 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.8559999999999998e-06 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.660 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.568210+0000", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0034547689999999999 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.568250+0000", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0367999999999997e-05 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:39.568375+0000", 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012487300000000001 2024-03-11T21:42:42.661 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:45 314.4 314:20000000::::head [pgnls start_epoch 694] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e694)", 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 181.163517246, 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.010032127, 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.662 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 45 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.663 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505061+0000", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9689999999999999e-06 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505074+0000", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3366e-05 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505105+0000", 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0725999999999997e-05 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.664 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505123+0000", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7966e-05 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505241+0000", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000117896 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505265+0000", 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4635e-05 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.665 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.514780+0000", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0095147430000000009 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.514928+0000", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014781500000000001 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.515090+0000", 2024-03-11T21:42:42.666 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016201099999999999 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:183 315.18 315:18000000::::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 173.09866171100001, 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056069010000000001, 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.667 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 183 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569916+0000", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5380000000000001e-06 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569926+0000", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7419999999999992e-06 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569946+0000", 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0543e-05 2024-03-11T21:42:42.669 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569956+0000", 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0026e-05 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.570029+0000", 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3238000000000006e-05 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.670 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.570036+0000", 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.781e-06 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575396+0000", 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0053596629999999998 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575419+0000", 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3620000000000001e-05 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.671 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575520+0000", 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00010075 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(osd.1.295:36 313.4s0 313:21c23a29:test-rados-api-smithi143-71773-117::foo:2 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e712)", 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:11.570212+0000", 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 151.09836305799999, 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0044339990000000001, 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.672 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570212+0000", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570212+0000", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.673 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570216+0000", 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6399999999999999e-06 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570231+0000", 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.575e-05 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570265+0000", 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4025e-05 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.674 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570293+0000", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7801999999999999e-05 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570405+0000", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000111938 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.570413+0000", 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5270000000000001e-06 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.675 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.573123+0000", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0027098809999999999 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.573250+0000", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000127312 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:11.574646+0000", 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0013961240000000001 2024-03-11T21:42:42.676 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1404 316.7 e713/704 316:e031f6fd:test-rados-api-smithi143-71773-117::bar:2 v 713'4, mlcod=712'1)", 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:12.518712+0000", 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 150.14986294900001, 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0045152860000000003, 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.677 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518712+0000", 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518712+0000", 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518712+0000", 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1899999999999998e-07 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518718+0000", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5770000000000001e-06 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518737+0000", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9120999999999999e-05 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.518749+0000", 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1541e-05 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.679 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.519306+0000", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00055702099999999999 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.519332+0000", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6444999999999998e-05 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.523168+0000", 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0038360780000000001 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.680 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:12.523227+0000", 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9184000000000001e-05 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:945 321.17 321:e8000000:test-rados-api-smithi143-71773-127:::head [pgnls start_epoch 757] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e757)", 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:56.628525+0000", 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 106.040050088, 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0038667799999999998, 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.681 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 945 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628525+0000", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628525+0000", 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.682 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628527+0000", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.012e-06 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628536+0000", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2439999999999999e-06 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628555+0000", 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8695e-05 2024-03-11T21:42:42.683 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628563+0000", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1929999999999993e-06 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628625+0000", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.1660000000000003e-05 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.628629+0000", 2024-03-11T21:42:42.684 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5789999999999997e-06 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.632222+0000", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.003592362 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.632246+0000", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3944e-05 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.632392+0000", 2024-03-11T21:42:42.685 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014609099999999999 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1064 321.17 321:e8000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 104.029119137, 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0059295169999999996, 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.686 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1064 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639460+0000", 2024-03-11T21:42:42.687 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8199999999999998e-06 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639477+0000", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7003e-05 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639515+0000", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8584999999999997e-05 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639541+0000", 2024-03-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5503e-05 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639658+0000", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011668899999999999 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639664+0000", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.998e-06 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645183+0000", 2024-03-11T21:42:42.689 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0055196500000000001 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645212+0000", 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8430999999999999e-05 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645385+0000", 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000173838 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.690 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1107 322.14 322:28000000:test-rados-api-smithi143-71773-129:::head [pgnls start_epoch 765] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e765)", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:04.588963+0000", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 98.079611955000004, 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.004001948, 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1107 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.588963+0000", 2024-03-11T21:42:42.691 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.588963+0000", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.588965+0000", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2409999999999998e-06 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.588977+0000", 2024-03-11T21:42:42.692 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1205e-05 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.589002+0000", 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4998e-05 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.589015+0000", 2024-03-11T21:42:42.693 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3499000000000001e-05 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.589115+0000", 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9895999999999994e-05 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.589122+0000", 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7669999999999998e-06 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.694 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.592855+0000", 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.003733157 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.592881+0000", 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6373e-05 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:04.592965+0000", 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3812000000000003e-05 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1233 322.14 322:28000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 91.031500574999995, 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007289383, 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1233 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.696 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637077+0000", 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1019999999999999e-06 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.697 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637086+0000", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.8800000000000003e-06 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637107+0000", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0761e-05 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637116+0000", 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6659999999999997e-06 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.698 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637229+0000", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011346899999999999 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637239+0000", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.5189999999999998e-06 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644264+0000", 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0070247779999999998 2024-03-11T21:42:42.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644287+0000", 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.376e-05 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644364+0000", 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6447999999999997e-05 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.700 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1731 325.18 325:18000000::::head [pgnls start_epoch 799] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e799)", 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:40.631490+0000", 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 62.037085535999999, 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0035153509999999999, 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1731 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.701 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631490+0000", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631490+0000", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631493+0000", 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7699999999999999e-06 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.702 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631513+0000", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9703000000000001e-05 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631554+0000", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1094999999999999e-05 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631583+0000", 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9062999999999999e-05 2024-03-11T21:42:42.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631740+0000", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000156985 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.631751+0000", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1117e-05 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.634795+0000", 2024-03-11T21:42:42.704 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0030439400000000002 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.634840+0000", 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5059999999999999e-05 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:40.635005+0000", 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000164618 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.705 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(9.4,QREQUEST,e:804,from:1,reservation_nonce:10)", 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 56.388213178999997, 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0060827479999999998, 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.706 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280364+0000", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0980000000000001e-06 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280376+0000", 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.186e-05 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.707 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280401+0000", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4698999999999998e-05 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280414+0000", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3743e-05 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280546+0000", 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013215699999999999 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.708 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280558+0000", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.187e-05 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286364+0000", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0058053419999999998 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286388+0000", 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3997000000000001e-05 2024-03-11T21:42:42.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286445+0000", 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6981999999999998e-05 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1431 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'1, mlcod=0'0)", 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 56.074511123000001, 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0053908690000000004, 2024-03-11T21:42:42.710 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.711 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0699999999999998e-07 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594071+0000", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0010000000000004e-06 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594096+0000", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4499e-05 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.712 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594107+0000", 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0920999999999999e-05 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594184+0000", 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6915000000000005e-05 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594209+0000", 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5340000000000001e-05 2024-03-11T21:42:42.713 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599371+0000", 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0051624430000000001 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599455+0000", 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3343000000000005e-05 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.714 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.41921.0:1794 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'2, mlcod=0'0)", 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 56.071488217000002, 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.015262123000000001, 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.715 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.716 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597088+0000", 2024-03-11T21:42:42.717 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0249999999999999e-06 2024-03-11T21:42:42.717 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.717 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.717 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.717 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604616+0000", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0075278020000000001 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604631+0000", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5469e-05 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604641+0000", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.4579999999999999e-06 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.718 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.605630+0000", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000989284 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.605661+0000", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0975000000000001e-05 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612289+0000", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0066284650000000001 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.719 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612349+0000", 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9645000000000002e-05 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1831 326.10 326:08000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 54.073631442, 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0045681610000000003, 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.720 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1831 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.721 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594947+0000", 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1089999999999998e-06 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594961+0000", 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3849000000000001e-05 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.722 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594987+0000", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6274999999999998e-05 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595000+0000", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3328e-05 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595193+0000", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019239100000000001 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.723 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595206+0000", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3543000000000001e-05 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599283+0000", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0040770390000000002 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599332+0000", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.8374000000000001e-05 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.724 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599512+0000", 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018025300000000001 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(5.1c,QREQUEST,e:813,from:4,reservation_nonce:19)", 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 47.933282900000002, 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007466922, 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.725 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735294+0000", 2024-03-11T21:42:42.726 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9800000000000001e-06 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735304+0000", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7429999999999997e-06 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735331+0000", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.743e-05 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735347+0000", 2024-03-11T21:42:42.727 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6052000000000001e-05 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735461+0000", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011319699999999999 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735472+0000", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1195999999999999e-05 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742601+0000", 2024-03-11T21:42:42.728 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007128962 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742642+0000", 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1245000000000002e-05 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742759+0000", 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000117117 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.729 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2151 329.14 329:28000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 12.069322542, 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0091880580000000007, 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2151 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.730 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599257+0000", 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0430000000000001e-06 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.731 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599274+0000", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7410000000000001e-05 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599311+0000", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6906000000000002e-05 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599329+0000", 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8246000000000001e-05 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.732 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599538+0000", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020909100000000001 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599550+0000", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1982e-05 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608239+0000", 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.008689235 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.733 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608293+0000", 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.3168999999999999e-05 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608441+0000", 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014797599999999999 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2216 329.14 329:28000000:%01:::head [pgnls start_epoch 844] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e844)", 2024-03-11T21:42:42.734 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 10.063997151000001, 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0089340849999999996, 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2216 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:42:42.735 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604581+0000", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1530000000000001e-06 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604598+0000", 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6719e-05 2024-03-11T21:42:42.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604635+0000", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7562999999999999e-05 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604654+0000", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8709999999999999e-05 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604798+0000", 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014399899999999999 2024-03-11T21:42:42.737 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604809+0000", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0482999999999999e-05 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.611926+0000", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0071172049999999997 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.612036+0000", 2024-03-11T21:42:42.738 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00010983400000000001 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.613512+0000", 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0014764190000000001 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2312 313.4s0 313:2473172f:test-rados-api-smithi143-71773-145::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e847)", 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 6.928829092, 2024-03-11T21:42:42.739 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.042486758999999999, 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2312 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.740 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739749+0000", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1429999999999998e-06 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.741 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739769+0000", 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9596e-05 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739801+0000", 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2595999999999998e-05 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739828+0000", 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.635e-05 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.742 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.743 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.740021+0000", 2024-03-11T21:42:42.743 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019352400000000001 2024-03-11T21:42:42.743 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.743 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.740044+0000", 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2379999999999999e-05 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.753843+0000", 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013799567 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.744 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.754197+0000", 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00035403000000000002 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.755533+0000", 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001335911 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.757815+0000", 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002282362 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.782133+0000", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.024317314 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.782233+0000", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9986000000000001e-05 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(osd.1.295:43 313.4s0 313:2473172f:test-rados-api-smithi143-71773-145::foo:head [read 0~1] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e850)", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:42:42.746 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 4.0856771529999998, 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012771879999999999, 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:42:42.747 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582901+0000", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.117e-06 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582916+0000", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4956e-05 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582957+0000", 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0802e-05 2024-03-11T21:42:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582984+0000", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.703e-05 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.583133+0000", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014922899999999999 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.583145+0000", 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1476e-05 2024-03-11T21:42:42.749 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.591781+0000", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0086360289999999999 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.592078+0000", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029759799999999999 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.595670+0000", 2024-03-11T21:42:42.750 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0035916429999999998 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:42:42.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.781 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:42.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:43.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.022 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:42 smithi143 bash[29284]: cluster 2024-03-11T21:42:41.627520+0000 mgr.y (mgr.14550) 764 : cluster 0 pgmap v1318: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-11T21:42:43.022 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:42 smithi143 bash[21512]: cluster 2024-03-11T21:42:41.627520+0000 mgr.y (mgr.14550) 764 : cluster 0 pgmap v1318: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-11T21:42:43.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:42 smithi203 bash[23654]: cluster 2024-03-11T21:42:41.627520+0000 mgr.y (mgr.14550) 764 : cluster 0 pgmap v1318: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-11T21:42:43.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:43.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:43.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/2468876445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0100d40 msgr2=0x7f44e01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- 172.21.15.143:0/2468876445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0100d40 0x7f44e01011c0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009990 tx=0x7f44d0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/2468876445 shutdown_connections 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- 172.21.15.143:0/2468876445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0106dc0 0x7f44e01071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- 172.21.15.143:0/2468876445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0101700 0x7f44e010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- 172.21.15.143:0/2468876445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0100d40 0x7f44e01011c0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/2468876445 >> 172.21.15.143:0/2468876445 conn(0x7f44e0076d80 msgr2=0x7f44e0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/2468876445 shutdown_connections 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/2468876445 wait complete. 2024-03-11T21:42:43.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 librados: starting msgr at 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 librados: starting objecter 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 Processor -- start 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- start start 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 librados: setting wanted keys 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 librados: calling monclient init 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 0x7f44e019d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0101700 0x7f44e019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e01105c0 con 0x7f44e0101700 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0110440 con 0x7f44e0100d40 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0110740 con 0x7f44e0106dc0 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:36290/0 (socket says 172.21.15.143:36290) 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 -- 172.21.15.143:0/370961892 learned_addr learned my addr 172.21.15.143:0/370961892 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:43.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e521f700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0101700 0x7f44e019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e5a20700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 0x7f44e019d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 -- 172.21.15.143:0/370961892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 msgr2=0x7f44e019d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 0x7f44e019d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 -- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0101700 msgr2=0x7f44e019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0101700 0x7f44e019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 -- 172.21.15.143:0/370961892 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0198200 con 0x7f44e0106dc0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e5a20700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 0x7f44e019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e6221700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b550 tx=0x7f44dc00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44d6ffd700 10 client.?.objecter ms_handle_connect 0x7f44e0106dc0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44d6ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc015020 con 0x7f44e0106dc0 2024-03-11T21:42:43.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f44dc011cb0 con 0x7f44e0106dc0 2024-03-11T21:42:43.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e01984f0 con 0x7f44e0106dc0 2024-03-11T21:42:43.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc010d20 con 0x7f44e0106dc0 2024-03-11T21:42:43.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 10 client.42077.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:43.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e01a4760 con 0x7f44e0106dc0 2024-03-11T21:42:43.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 librados: init done 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.206+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e004faa0 con 0x7f44e0106dc0 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f44dc018020 con 0x7f44e0106dc0 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 0x7f44cc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(850..850 src has 87..850) v4 ==== 10261+0+0 (secure 0 0 0) 0x7f44dc08e490 con 0x7f44e0106dc0 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 10 client.42077.objecter ms_dispatch 0x7f44e0001550 osd_map(850..850 src has 87..850) v4 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 3 client.42077.objecter handle_osd_map got epochs [850,850] > 0 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 3 client.42077.objecter handle_osd_map decoding full epoch 850 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44e5a20700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 0x7f44cc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 20 client.42077.objecter dump_active .. 0 homeless 2024-03-11T21:42:43.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44e5a20700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 0x7f44cc071ce0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f44e0106900 tx=0x7f44d800b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:43.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.210+0000 7f44d6ffd700 10 client.42077.objecter ms_handle_connect 0x7f44cc06f820 2024-03-11T21:42:43.199 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.214+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f44dc01f020 con 0x7f44e0106dc0 2024-03-11T21:42:43.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.294 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:43.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:43.426 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:43.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f44e01a4a80 con 0x7f44e0106dc0 2024-03-11T21:42:43.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44d6ffd700 1 -- 172.21.15.143:0/370961892 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2617 (secure 0 0 0) 0x7f44dc05ae80 con 0x7f44e0106dc0 2024-03-11T21:42:43.494 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:42:43.494 INFO:teuthology.orchestra.run.smithi143.stdout:{"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"CACHE_POOL_NO_HIT_SET":{"severity":"HEALTH_WARN","summary":{"message":"1 cache pools are missing hit_sets","count":1},"muted":false},"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"3 pool(s) do not have an application enabled","count":3},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1650,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":850,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192696,"num_in_osds":8,"osd_in_since":1710192115,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":246}],"num_pgs":246,"num_pools":10,"num_objects":237,"data_bytes":4596654,"bytes_used":762892288,"bytes_avail":767197392896,"bytes_total":767960285184,"read_bytes_sec":868,"write_bytes_sec":173,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.143:8443/","prometheus":"http://172.21.15.143:9283/"}},"servicemap":{"epoch":25,"modified":"2024-03-11T21:39:59.554469+0000","services":{"rgw":{"daemons":{"summary":"","14427":{"start_epoch":13,"start_stamp":"2024-03-11T21:23:10.349156+0000","gid":14427,"addr":"172.21.15.143:0/3842484601","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi143","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi143","id":"foo.a","kernel_description":"#110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024","kernel_version":"5.15.0-100-generic","mem_swap_kb":"2097148","mem_total_kb":"32746924","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"25063d57-28bf-41f2-b5d5-290103490a0a","zone_name":"default","zonegroup_id":"4aab72bc-9c1b-4496-a7a5-2f18f6205dcf","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 10 librados: watch_flush enter 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 10 librados: watch_flush exit 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 20 client.42077.objecter shutdown clearing up homeless session... 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 10 client.42077.objecter successfully canceled tick 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 msgr2=0x7f44cc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 0x7f44cc071ce0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f44e0106900 tx=0x7f44d800b040 comp rx=0 tx=0).stop 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 msgr2=0x7f44e0197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b550 tx=0x7f44dc00b860 comp rx=0 tx=0).stop 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 shutdown_connections 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f44cc06f820 0x7f44cc071ce0 unknown :-1 s=CLOSED pgs=3456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e0106dc0 0x7f44e0197aa0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44e0100d40 0x7f44e019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 --2- 172.21.15.143:0/370961892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e0101700 0x7f44e019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 >> 172.21.15.143:0/370961892 conn(0x7f44e0076d80 msgr2=0x7f44e00ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 shutdown_connections 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 -- 172.21.15.143:0/370961892 wait complete. 2024-03-11T21:42:43.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:43.510+0000 7f44e7c84700 1 librados: shutdown 2024-03-11T21:42:43.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.546 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:43.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:43.672 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:43.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:43.797 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:43.798 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:43 smithi143 bash[21512]: audit 2024-03-11T21:42:43.512252+0000 mon.c (mon.2) 1315 : audit 0 from='client.? 172.21.15.143:0/370961892' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:42:43.798 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:43 smithi143 bash[29284]: audit 2024-03-11T21:42:43.512252+0000 mon.c (mon.2) 1315 : audit 0 from='client.? 172.21.15.143:0/370961892' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:42:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:43.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:43.923 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:44.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:44.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:43 smithi203 bash[23654]: audit 2024-03-11T21:42:43.512252+0000 mon.c (mon.2) 1315 : audit 0 from='client.? 172.21.15.143:0/370961892' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:42:44.054 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:44.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:44.183 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:44.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.314 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:44.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:44.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:44.575 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:44.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:44.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:44.833 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:44.936 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:44.964 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:45.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:44 smithi203 bash[23654]: cluster 2024-03-11T21:42:43.628840+0000 mgr.y (mgr.14550) 765 : cluster 0 pgmap v1319: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 B/s wr, 1 op/s 2024-03-11T21:42:45.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:45.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:44 smithi143 bash[21512]: cluster 2024-03-11T21:42:43.628840+0000 mgr.y (mgr.14550) 765 : cluster 0 pgmap v1319: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 B/s wr, 1 op/s 2024-03-11T21:42:45.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:44 smithi143 bash[29284]: cluster 2024-03-11T21:42:43.628840+0000 mgr.y (mgr.14550) 765 : cluster 0 pgmap v1319: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 144 B/s wr, 1 op/s 2024-03-11T21:42:45.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:45.224 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:45.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:45.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:45.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.486 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:45.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.619 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:45.722 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:45.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:45.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:45.878 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:45.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:46.009 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:46.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:46.140 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:46.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:42:46.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:46.271 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:46.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:46.395 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:46.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:46.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:46.658 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:46.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:46.783 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:46.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:46.914 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:47.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:47.044 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:47.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:46 smithi203 bash[23654]: cluster 2024-03-11T21:42:45.629827+0000 mgr.y (mgr.14550) 766 : cluster 0 pgmap v1320: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 126 B/s wr, 1 op/s 2024-03-11T21:42:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:47.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:47.174 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:47.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:46 smithi143 bash[29284]: cluster 2024-03-11T21:42:45.629827+0000 mgr.y (mgr.14550) 766 : cluster 0 pgmap v1320: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 126 B/s wr, 1 op/s 2024-03-11T21:42:47.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:46 smithi143 bash[21512]: cluster 2024-03-11T21:42:45.629827+0000 mgr.y (mgr.14550) 766 : cluster 0 pgmap v1320: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 126 B/s wr, 1 op/s 2024-03-11T21:42:47.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:47.304 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:47.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:47.435 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:47.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:47.567 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:47.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:47.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:47.714 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-11T21:42:47.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:42:47.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:47.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:47.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:47.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:48.064 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:48.090 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:48.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.223 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:48.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:48.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:48.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.484 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:48.587 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:48.614 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:48.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:48.742 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:48.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:48.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:49.000 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:49.048 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:48 smithi203 bash[23654]: cluster 2024-03-11T21:42:47.630643+0000 mgr.y (mgr.14550) 767 : cluster 0 pgmap v1321: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:42:49.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:49.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:49.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:48 smithi143 bash[29284]: cluster 2024-03-11T21:42:47.630643+0000 mgr.y (mgr.14550) 767 : cluster 0 pgmap v1321: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:42:49.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:48 smithi143 bash[21512]: cluster 2024-03-11T21:42:47.630643+0000 mgr.y (mgr.14550) 767 : cluster 0 pgmap v1321: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:42:49.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.259 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:49.279 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:49.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:49.389 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:49.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.519 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:49.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.651 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:49.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/3515315356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0106ec0 msgr2=0x7ff0c01072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- 172.21.15.143:0/3515315356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0106ec0 0x7ff0c01072a0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00098f0 tx=0x7ff0b0009c00 comp rx=0 tx=0).stop 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/3515315356 shutdown_connections 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- 172.21.15.143:0/3515315356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0106ec0 0x7ff0c01072a0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- 172.21.15.143:0/3515315356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c0101160 0x7ff0c010ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- 172.21.15.143:0/3515315356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0c01007a0 0x7ff0c0100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/3515315356 >> 172.21.15.143:0/3515315356 conn(0x7ff0c0076000 msgr2=0x7ff0c0076410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/3515315356 shutdown_connections 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/3515315356 wait complete. 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 librados: starting msgr at 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 librados: starting objecter 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 Processor -- start 2024-03-11T21:42:49.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- start start 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 librados: setting wanted keys 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 librados: calling monclient init 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c01007a0 0x7ff0c019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 0x7ff0c019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0c0106ec0 0x7ff0c0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0111670 con 0x7ff0c0101160 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01114f0 con 0x7ff0c0106ec0 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01117f0 con 0x7ff0c01007a0 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 0x7ff0c019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 0x7ff0c019dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:60892/0 (socket says 172.21.15.143:60892) 2024-03-11T21:42:49.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 -- 172.21.15.143:0/4076548193 learned_addr learned my addr 172.21.15.143:0/4076548193 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c01007a0 0x7ff0c019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0c0106ec0 0x7ff0c0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c01007a0 msgr2=0x7ff0c019d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c01007a0 0x7ff0c019d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0c0106ec0 msgr2=0x7ff0c0197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0c0106ec0 0x7ff0c0197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c0198170 con 0x7ff0c0101160 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0c01007a0 0x7ff0c019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 0x7ff0c019dd90 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400c920 tx=0x7ff0b400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0a7fff700 10 client.?.objecter ms_handle_connect 0x7ff0c0101160 2024-03-11T21:42:49.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0a7fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4007b00 con 0x7ff0c0101160 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff0b4004cb0 con 0x7ff0c0101160 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c0198460 con 0x7ff0c0101160 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40054a0 con 0x7ff0c0101160 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 10 client.30879.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c01a4720 con 0x7ff0c0101160 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 librados: init done 2024-03-11T21:42:49.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.698+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff0c004faa0 con 0x7ff0c0101160 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff0b4004750 con 0x7ff0c0101160 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0a806f820 0x7ff0a8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(850..850 src has 87..850) v4 ==== 10261+0+0 (secure 0 0 0) 0x7ff0b408edc0 con 0x7ff0c0101160 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 osd_map(850..850 src has 87..850) v4 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 3 client.30879.objecter handle_osd_map got epochs [850,850] > 0 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 3 client.30879.objecter handle_osd_map decoding full epoch 850 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0a806f820 0x7ff0a8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 20 client.30879.objecter dump_active .. 0 homeless 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v850) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff0b4014030 con 0x7ff0c0101160 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 10 client.30879.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 1 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0a806f820 2024-03-11T21:42:49.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0a806f820 0x7ff0a8071ce0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00098c0 tx=0x7ff0b0019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0c0062c40 0x7ff0c0108ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 1 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 0 1 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 4 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 10 client.30879.objecter _send_command 1 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c010a260 con 0x7ff0c0062c40 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0c0062c40 0x7ff0c0108ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0c0062c40 0x7ff0c0108ba0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c0062c40 2024-03-11T21:42:49.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c010a260 con 0x7ff0c0062c40 2024-03-11T21:42:49.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:42:49.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 1 = system:0 2024-03-11T21:42:49.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 4 2024-03-11T21:42:49.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.702+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 0 1 2024-03-11T21:42:49.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 10 client.30879.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 2 2024-03-11T21:42:49.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 2 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0198f40 osd=0 3 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 0 2 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 4 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 10 client.30879.objecter _send_command 2 2024-03-11T21:42:49.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c01090e0 con 0x7ff0c0062c40 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c01090e0 con 0x7ff0c0062c40 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 4 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 0 2 2024-03-11T21:42:49.708 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:42:49.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 10 client.30879.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 3 2024-03-11T21:42:49.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0c0062830 0x7ff0c01a89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 1 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 4 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 10 client.30879.objecter _send_command 3 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c0109fd0 con 0x7ff0c0062830 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0c0062830 0x7ff0c01a89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c0062830 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.722+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0c0062830 0x7ff0c01a89d0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.726+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c0109fd0 con 0x7ff0c0062830 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.726+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.726+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 3 = system:0 2024-03-11T21:42:49.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.726+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 4 2024-03-11T21:42:49.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.726+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 1 3 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 10 client.30879.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 4 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 4 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c00626c0 osd=1 3 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 1 4 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 4 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 10 client.30879.objecter _send_command 4 2024-03-11T21:42:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.742+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c0109540 con 0x7ff0c0062830 2024-03-11T21:42:49.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c0109540 con 0x7ff0c0062830 2024-03-11T21:42:49.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 4 2024-03-11T21:42:49.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 1 4 2024-03-11T21:42:49.730 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 10 client.30879.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 5 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0c002d390 0x7ff0c01ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 5 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 2 5 2024-03-11T21:42:49.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 4 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 10 client.30879.objecter _send_command 5 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c01aef00 con 0x7ff0c002d390 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0c002d390 0x7ff0c01ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c002d390 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0c002d390 0x7ff0c01ac0a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c01aef00 con 0x7ff0c002d390 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 5 = system:0 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 4 2024-03-11T21:42:49.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.746+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 2 5 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 10 client.30879.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 6 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 6 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0109940 osd=2 3 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 2 6 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 4 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 10 client.30879.objecter _send_command 6 2024-03-11T21:42:49.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c002c900 con 0x7ff0c002d390 2024-03-11T21:42:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c002c900 con 0x7ff0c002d390 2024-03-11T21:42:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 4 2024-03-11T21:42:49.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 2 6 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 10 client.30879.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 7 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0c01ad760 0x7ff0c01adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 7 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 3 7 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 4 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 10 client.30879.objecter _send_command 7 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c01a9db0 con 0x7ff0c01ad760 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0c01ad760 0x7ff0c01adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c01ad760 2024-03-11T21:42:49.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.766+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0c01ad760 0x7ff0c01adba0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.770+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c01a9db0 con 0x7ff0c01ad760 2024-03-11T21:42:49.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.770+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:42:49.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.770+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 7 = system:0 2024-03-11T21:42:49.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.770+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 4 2024-03-11T21:42:49.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.770+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 3 7 2024-03-11T21:42:49.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 10 client.30879.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 8 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 8 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a96e0 osd=3 3 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 3 8 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 4 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 10 client.30879.objecter _send_command 8 2024-03-11T21:42:49.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.786+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c01a9160 con 0x7ff0c01ad760 2024-03-11T21:42:49.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c01a9160 con 0x7ff0c01ad760 2024-03-11T21:42:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 4 2024-03-11T21:42:49.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 3 8 2024-03-11T21:42:49.774 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 10 client.30879.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 9 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0c01018b0 0x7ff0c0101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 9 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 4 9 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 4 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 10 client.30879.objecter _send_command 9 2024-03-11T21:42:49.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c01a9080 con 0x7ff0c01018b0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0c01018b0 0x7ff0c0101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c01018b0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0c01018b0 0x7ff0c0101cf0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c01a9080 con 0x7ff0c01018b0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 9 = system:0 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 4 2024-03-11T21:42:49.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.790+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 4 9 2024-03-11T21:42:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 10 client.30879.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 10 2024-03-11T21:42:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 10 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a8f10 osd=4 3 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 4 10 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 4 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 10 client.30879.objecter _send_command 10 2024-03-11T21:42:49.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.814+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c01a9080 con 0x7ff0c01018b0 2024-03-11T21:42:49.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c01a9080 con 0x7ff0c01018b0 2024-03-11T21:42:49.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 4 2024-03-11T21:42:49.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 4 10 2024-03-11T21:42:49.802 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:42:49.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 10 client.30879.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 11 2024-03-11T21:42:49.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0c0102740 0x7ff0c0102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 11 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 5 11 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 4 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 10 client.30879.objecter _send_command 11 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c0103230 con 0x7ff0c0102740 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0c0102740 0x7ff0c0102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c0102740 2024-03-11T21:42:49.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0be59c700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0c0102740 0x7ff0c0102ba0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c0103230 con 0x7ff0c0102740 2024-03-11T21:42:49.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:42:49.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 11 = system:0 2024-03-11T21:42:49.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 4 2024-03-11T21:42:49.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.818+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 5 11 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 10 client.30879.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 12 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 12 2024-03-11T21:42:49.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01a9080 osd=5 3 2024-03-11T21:42:49.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 5 12 2024-03-11T21:42:49.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 4 2024-03-11T21:42:49.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 10 client.30879.objecter _send_command 12 2024-03-11T21:42:49.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.854+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c0103230 con 0x7ff0c0102740 2024-03-11T21:42:49.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c0103230 con 0x7ff0c0102740 2024-03-11T21:42:49.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 4 2024-03-11T21:42:49.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 5 12 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 10 client.30879.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 13 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0c01b94f0 0x7ff0c01bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 13 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 6 13 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 4 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 10 client.30879.objecter _send_command 13 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c01be830 con 0x7ff0c01b94f0 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0c01b94f0 0x7ff0c01bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c01b94f0 2024-03-11T21:42:49.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.858+0000 7ff0bf59e700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0c01b94f0 0x7ff0c01bb8e0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.862+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c01be830 con 0x7ff0c01b94f0 2024-03-11T21:42:49.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.862+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:42:49.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.862+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 13 = system:0 2024-03-11T21:42:49.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.862+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 4 2024-03-11T21:42:49.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.862+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 6 13 2024-03-11T21:42:49.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.894+0000 7ff0c5024700 10 client.30879.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 14 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 14 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c0103230 osd=6 3 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 6 14 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 4 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 10 client.30879.objecter _send_command 14 2024-03-11T21:42:49.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.898+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c01be850 con 0x7ff0c01b94f0 2024-03-11T21:42:49.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0b8025ba0 con 0x7ff0c01b94f0 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 4 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 6 14 2024-03-11T21:42:49.886 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 10 client.30879.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 15 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0c01bc570 0x7ff0c01bc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 15 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 7 15 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 4 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 10 client.30879.objecter _send_command 15 2024-03-11T21:42:49.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0c01be260 con 0x7ff0c01bc570 2024-03-11T21:42:49.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0c01bc570 0x7ff0c01bc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0bed9d700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0c01bc570 0x7ff0c01bc9b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.902+0000 7ff0a7fff700 10 client.30879.objecter ms_handle_connect 0x7ff0c01bc570 2024-03-11T21:42:49.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.906+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0c01be260 con 0x7ff0c01bc570 2024-03-11T21:42:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.906+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:42:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.906+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 15 = system:0 2024-03-11T21:42:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.906+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 4 2024-03-11T21:42:49.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.906+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 7 15 2024-03-11T21:42:49.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:49 smithi203 bash[23654]: audit 2024-03-11T21:42:48.832436+0000 mon.c (mon.2) 1316 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:49.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:49 smithi203 bash[23654]: audit 2024-03-11T21:42:48.833306+0000 mon.a (mon.0) 3832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:49.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:49 smithi203 bash[23654]: audit 2024-03-11T21:42:49.065867+0000 mgr.y (mgr.14550) 768 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:49.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:49 smithi203 bash[23654]: audit 2024-03-11T21:42:49.703997+0000 mon.a (mon.0) 3833 : audit 0 from='client.? 172.21.15.143:0/4076548193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:49.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:49.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:42:49.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign -1 16 2024-03-11T21:42:49.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter _get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_remove -1 16 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter get_session s=0x7ff0c01bbe60 osd=7 3 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 15 client.30879.objecter _session_command_op_assign 7 16 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 4 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter _send_command 16 2024-03-11T21:42:49.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0c01be260 con 0x7ff0c01bc570 2024-03-11T21:42:49.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0a7fff700 1 -- 172.21.15.143:0/4076548193 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0c01be260 con 0x7ff0c01bc570 2024-03-11T21:42:49.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0a7fff700 10 client.30879.objecter ms_dispatch 0x7ff0c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:42:49.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0a7fff700 10 client.30879.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:42:49.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0a7fff700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 4 2024-03-11T21:42:49.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0a7fff700 15 client.30879.objecter _session_command_op_remove 7 16 2024-03-11T21:42:49.928 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 librados: watch_flush enter 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 librados: watch_flush exit 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.0 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0c0062c40 msgr2=0x7ff0c0108ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7ff0c0062c40 0x7ff0c0108ba0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0198f40 osd=0 1 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.1 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0c0062830 msgr2=0x7ff0c01a89d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7ff0c0062830 0x7ff0c01a89d0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c00626c0 osd=1 1 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.2 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0c002d390 msgr2=0x7ff0c01ac0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7ff0c002d390 0x7ff0c01ac0a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0109940 osd=2 1 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.3 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0c01ad760 msgr2=0x7ff0c01adba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7ff0c01ad760 0x7ff0c01adba0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a96e0 osd=3 1 2024-03-11T21:42:49.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.4 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0c01018b0 msgr2=0x7ff0c0101cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0bf59e700 1 -- 172.21.15.143:0/4076548193 reap_dead start 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7ff0c01018b0 0x7ff0c0101cf0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0bf59e700 1 -- 172.21.15.143:0/4076548193 reap_dead start 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a8f10 osd=4 1 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.5 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0c0102740 msgr2=0x7ff0c0102ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7ff0c0102740 0x7ff0c0102ba0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01a9080 osd=5 1 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.6 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0c01b94f0 msgr2=0x7ff0c01bb8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7ff0c01b94f0 0x7ff0c01bb8e0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c0103230 osd=6 1 2024-03-11T21:42:49.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter close_session for osd.7 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0c01bc570 msgr2=0x7ff0c01bc9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7ff0c01bc570 0x7ff0c01bc9b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter put_session s=0x7ff0c01bbe60 osd=7 1 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 20 client.30879.objecter shutdown clearing up homeless session... 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 10 client.30879.objecter successfully canceled tick 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0a806f820 msgr2=0x7ff0a8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff0a806f820 0x7ff0a8071ce0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00098c0 tx=0x7ff0b0019040 comp rx=0 tx=0).stop 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 msgr2=0x7ff0c019dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 --2- 172.21.15.143:0/4076548193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0c0101160 0x7ff0c019dd90 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400c920 tx=0x7ff0b400cc30 comp rx=0 tx=0).stop 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0bf59e700 1 -- 172.21.15.143:0/4076548193 reap_dead start 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 shutdown_connections 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 >> 172.21.15.143:0/4076548193 conn(0x7ff0c0076000 msgr2=0x7ff0c00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 shutdown_connections 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 -- 172.21.15.143:0/4076548193 wait complete. 2024-03-11T21:42:49.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.942+0000 7ff0c5024700 1 librados: shutdown 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:42:49.933 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/2821823241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 msgr2=0x7ff5201071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- 172.21.15.143:0/2821823241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff5201071e0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0099c0 tx=0x7ff51c009cd0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/2821823241 shutdown_connections 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- 172.21.15.143:0/2821823241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff5201071e0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- 172.21.15.143:0/2821823241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100cf0 0x7ff52010ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- 172.21.15.143:0/2821823241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100330 0x7ff5201007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/2821823241 >> 172.21.15.143:0/2821823241 conn(0x7ff5200772e0 msgr2=0x7ff5200fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/2821823241 shutdown_connections 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/2821823241 wait complete. 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 librados: starting msgr at 2024-03-11T21:42:49.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 librados: starting objecter 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 Processor -- start 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- start start 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 librados: setting wanted keys 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 librados: calling monclient init 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100cf0 0x7ff52019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff520197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201116a0 con 0x7ff520106e00 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520111520 con 0x7ff520100cf0 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff520111820 con 0x7ff520100330 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37104/0 (socket says 172.21.15.143:37104) 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 -- 172.21.15.143:0/3612126026 learned_addr learned my addr 172.21.15.143:0/3612126026 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff5264ac700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff520197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff5254aa700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100cf0 0x7ff52019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 -- 172.21.15.143:0/3612126026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100cf0 msgr2=0x7ff52019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100cf0 0x7ff52019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 -- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 msgr2=0x7ff520197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff520197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 -- 172.21.15.143:0/3612126026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520198200 con 0x7ff520100330 2024-03-11T21:42:49.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff5264ac700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff520197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff525cab700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff516ffd700 10 client.?.objecter ms_handle_connect 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff516ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c009e60 con 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff51c00f460 con 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff520198490 con 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c017650 con 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 10 client.42089.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5201a47b0 con 0x7ff520100330 2024-03-11T21:42:49.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.954+0000 7ff527f0f700 1 librados: init done 2024-03-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff51c00f960 con 0x7ff520100330 2024-03-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52004faa0 con 0x7ff520100330 2024-03-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(851..851 src has 87..851) v4 ==== 10261+0+0 (secure 0 0 0) 0x7ff51c08fa50 con 0x7ff520100330 2024-03-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 10 client.42089.objecter ms_dispatch 0x7ff520001550 osd_map(851..851 src has 87..851) v4 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 3 client.42089.objecter handle_osd_map got epochs [851,851] > 0 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 3 client.42089.objecter handle_osd_map decoding full epoch 851 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff5254aa700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 20 client.42089.objecter dump_active .. 0 homeless 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff5254aa700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7ff5200ffbd0 tx=0x7ff51000a300 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:42:49.944 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.958+0000 7ff516ffd700 10 client.42089.objecter ms_handle_connect 0x7ff50c06f820 2024-03-11T21:42:49.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:49.962+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff51c013070 con 0x7ff520100330 2024-03-11T21:42:50.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:50.032 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:50.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:50.135 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[29284]: audit 2024-03-11T21:42:48.832436+0000 mon.c (mon.2) 1316 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:50.135 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[29284]: audit 2024-03-11T21:42:48.833306+0000 mon.a (mon.0) 3832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[29284]: audit 2024-03-11T21:42:49.065867+0000 mgr.y (mgr.14550) 768 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[29284]: audit 2024-03-11T21:42:49.703997+0000 mon.a (mon.0) 3833 : audit 0 from='client.? 172.21.15.143:0/4076548193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[21512]: audit 2024-03-11T21:42:48.832436+0000 mon.c (mon.2) 1316 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[21512]: audit 2024-03-11T21:42:48.833306+0000 mon.a (mon.0) 3832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[21512]: audit 2024-03-11T21:42:49.065867+0000 mgr.y (mgr.14550) 768 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:50.136 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:49 smithi143 bash[21512]: audit 2024-03-11T21:42:49.703997+0000 mon.a (mon.0) 3833 : audit 0 from='client.? 172.21.15.143:0/4076548193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:50.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:50.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.186+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7ff5201a4980 con 0x7ff520100330 2024-03-11T21:42:50.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:50.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:50.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:50.427 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:50.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:50.559 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:50.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:50.689 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:50.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.818+0000 7ff516ffd700 1 -- 172.21.15.143:0/3612126026 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v852) v1 ==== 91+0+0 (secure 0 0 0) 0x7ff51c05c4a0 con 0x7ff520100330 2024-03-11T21:42:50.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 10 librados: watch_flush enter 2024-03-11T21:42:50.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 10 librados: watch_flush exit 2024-03-11T21:42:50.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 20 client.42089.objecter shutdown clearing up homeless session... 2024-03-11T21:42:50.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 10 client.42089.objecter successfully canceled tick 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 msgr2=0x7ff50c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7ff5200ffbd0 tx=0x7ff51000a300 comp rx=0 tx=0).stop 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 msgr2=0x7ff52019d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.822+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c003da0 comp rx=0 tx=0).stop 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 shutdown_connections 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff520106e00 0x7ff520197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff520100330 0x7ff52019d8e0 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:50.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 --2- 172.21.15.143:0/3612126026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff520100cf0 0x7ff52019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:42:50.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 >> 172.21.15.143:0/3612126026 conn(0x7ff5200772e0 msgr2=0x7ff5201048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:42:50.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 shutdown_connections 2024-03-11T21:42:50.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 -- 172.21.15.143:0/3612126026 wait complete. 2024-03-11T21:42:50.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:42:50.826+0000 7ff527f0f700 1 librados: shutdown 2024-03-11T21:42:50.813 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:42:50.822 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:50.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.944 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:50.983 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: cluster 2024-03-11T21:42:49.631769+0000 mgr.y (mgr.14550) 769 : cluster 0 pgmap v1322: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: audit 2024-03-11T21:42:49.808949+0000 mon.a (mon.0) 3834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: cluster 2024-03-11T21:42:49.813530+0000 mon.a (mon.0) 3835 : cluster 0 osdmap e851: 8 total, 8 up, 8 in 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: audit 2024-03-11T21:42:49.816779+0000 mon.c (mon.2) 1317 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: audit 2024-03-11T21:42:49.819339+0000 mon.a (mon.0) 3836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: audit 2024-03-11T21:42:50.189474+0000 mon.c (mon.2) 1318 : audit 1 from='client.? 172.21.15.143:0/3612126026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:50 smithi203 bash[23654]: audit 2024-03-11T21:42:50.190042+0000 mon.a (mon.0) 3837 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:51.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: cluster 2024-03-11T21:42:49.631769+0000 mgr.y (mgr.14550) 769 : cluster 0 pgmap v1322: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: audit 2024-03-11T21:42:49.808949+0000 mon.a (mon.0) 3834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: cluster 2024-03-11T21:42:49.813530+0000 mon.a (mon.0) 3835 : cluster 0 osdmap e851: 8 total, 8 up, 8 in 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: audit 2024-03-11T21:42:49.816779+0000 mon.c (mon.2) 1317 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: audit 2024-03-11T21:42:49.819339+0000 mon.a (mon.0) 3836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: audit 2024-03-11T21:42:50.189474+0000 mon.c (mon.2) 1318 : audit 1 from='client.? 172.21.15.143:0/3612126026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.103 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[29284]: audit 2024-03-11T21:42:50.190042+0000 mon.a (mon.0) 3837 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: cluster 2024-03-11T21:42:49.631769+0000 mgr.y (mgr.14550) 769 : cluster 0 pgmap v1322: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: audit 2024-03-11T21:42:49.808949+0000 mon.a (mon.0) 3834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: cluster 2024-03-11T21:42:49.813530+0000 mon.a (mon.0) 3835 : cluster 0 osdmap e851: 8 total, 8 up, 8 in 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: audit 2024-03-11T21:42:49.816779+0000 mon.c (mon.2) 1317 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: audit 2024-03-11T21:42:49.819339+0000 mon.a (mon.0) 3836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: audit 2024-03-11T21:42:50.189474+0000 mon.c (mon.2) 1318 : audit 1 from='client.? 172.21.15.143:0/3612126026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:50 smithi143 bash[21512]: audit 2024-03-11T21:42:50.190042+0000 mon.a (mon.0) 3837 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:51.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.204 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:51.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:51.336 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:51.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:51.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:51.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:51.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:51.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: cluster 2024-03-11T21:42:50.809666+0000 mon.a (mon.0) 3838 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: cluster 2024-03-11T21:42:50.809758+0000 mon.a (mon.0) 3839 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.812141+0000 mon.a (mon.0) 3840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.812244+0000 mon.a (mon.0) 3841 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: cluster 2024-03-11T21:42:50.817950+0000 mon.a (mon.0) 3842 : cluster 0 osdmap e852: 8 total, 8 up, 8 in 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.873844+0000 mon.c (mon.2) 1319 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.874269+0000 mon.a (mon.0) 3843 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.875067+0000 mon.c (mon.2) 1320 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: audit 2024-03-11T21:42:50.875373+0000 mon.a (mon.0) 3844 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.840 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.737090+0000 osd.4 (osd.4) 431 : cluster 0 3.1a scrub starts 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.740463+0000 osd.4 (osd.4) 432 : cluster 0 3.1a scrub ok 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: cluster 2024-03-11T21:42:50.809666+0000 mon.a (mon.0) 3838 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: cluster 2024-03-11T21:42:50.809758+0000 mon.a (mon.0) 3839 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.812141+0000 mon.a (mon.0) 3840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.812244+0000 mon.a (mon.0) 3841 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: cluster 2024-03-11T21:42:50.817950+0000 mon.a (mon.0) 3842 : cluster 0 osdmap e852: 8 total, 8 up, 8 in 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.873844+0000 mon.c (mon.2) 1319 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.874269+0000 mon.a (mon.0) 3843 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.875067+0000 mon.c (mon.2) 1320 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: audit 2024-03-11T21:42:50.875373+0000 mon.a (mon.0) 3844 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.737090+0000 osd.4 (osd.4) 431 : cluster 0 3.1a scrub starts 2024-03-11T21:42:51.841 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:51 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.740463+0000 osd.4 (osd.4) 432 : cluster 0 3.1a scrub ok 2024-03-11T21:42:51.851 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:51.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:51.983 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:52.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:52.097 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: cluster 2024-03-11T21:42:50.809666+0000 mon.a (mon.0) 3838 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:52.097 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: cluster 2024-03-11T21:42:50.809758+0000 mon.a (mon.0) 3839 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.812141+0000 mon.a (mon.0) 3840 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]': finished 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.812244+0000 mon.a (mon.0) 3841 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: cluster 2024-03-11T21:42:50.817950+0000 mon.a (mon.0) 3842 : cluster 0 osdmap e852: 8 total, 8 up, 8 in 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.873844+0000 mon.c (mon.2) 1319 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.874269+0000 mon.a (mon.0) 3843 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.875067+0000 mon.c (mon.2) 1320 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: audit 2024-03-11T21:42:50.875373+0000 mon.a (mon.0) 3844 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-144"}]: dispatch 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.737090+0000 osd.4 (osd.4) 431 : cluster 0 3.1a scrub starts 2024-03-11T21:42:52.098 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:51 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.740463+0000 osd.4 (osd.4) 432 : cluster 0 3.1a scrub ok 2024-03-11T21:42:52.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:52.194 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:42:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:52.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:52.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:52.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:52.363 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:52.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:52.489 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:52.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:52.722 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.750 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:52.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:52.863 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.632268+0000 mgr.y (mgr.14550) 770 : cluster 0 pgmap v1325: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.833080+0000 mon.a (mon.0) 3845 : cluster 0 osdmap e853: 8 total, 8 up, 8 in 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.847914+0000 osd.3 (osd.3) 445 : cluster 0 313.4s0 scrub starts 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:51.855833+0000 osd.3 (osd.3) 446 : cluster 0 313.4 scrub ok 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.248984+0000 osd.6 (osd.6) 367 : cluster 0 313.7s0 scrub starts 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.251093+0000 osd.6 (osd.6) 368 : cluster 0 313.7 scrub ok 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: audit 2024-03-11T21:42:52.440822+0000 mon.c (mon.2) 1321 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:52.864 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.566163+0000 mon.a (mon.0) 3846 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.632268+0000 mgr.y (mgr.14550) 770 : cluster 0 pgmap v1325: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.833080+0000 mon.a (mon.0) 3845 : cluster 0 osdmap e853: 8 total, 8 up, 8 in 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.847914+0000 osd.3 (osd.3) 445 : cluster 0 313.4s0 scrub starts 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:51.855833+0000 osd.3 (osd.3) 446 : cluster 0 313.4 scrub ok 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.248984+0000 osd.6 (osd.6) 367 : cluster 0 313.7s0 scrub starts 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.251093+0000 osd.6 (osd.6) 368 : cluster 0 313.7 scrub ok 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: audit 2024-03-11T21:42:52.440822+0000 mon.c (mon.2) 1321 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:52.865 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:52 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.566163+0000 mon.a (mon.0) 3846 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:52.875 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:52.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.004 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:53.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:42:53.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:53.131 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:53.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.632268+0000 mgr.y (mgr.14550) 770 : cluster 0 pgmap v1325: 246 pgs: 246 active+clean; 4.4 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.833080+0000 mon.a (mon.0) 3845 : cluster 0 osdmap e853: 8 total, 8 up, 8 in 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.847914+0000 osd.3 (osd.3) 445 : cluster 0 313.4s0 scrub starts 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:51.855833+0000 osd.3 (osd.3) 446 : cluster 0 313.4 scrub ok 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.248984+0000 osd.6 (osd.6) 367 : cluster 0 313.7s0 scrub starts 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.251093+0000 osd.6 (osd.6) 368 : cluster 0 313.7 scrub ok 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: audit 2024-03-11T21:42:52.440822+0000 mon.c (mon.2) 1321 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:53.245 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:52 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.566163+0000 mon.a (mon.0) 3846 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:42:53.260 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:53.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.390 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:53.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:53.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:53.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:53.648 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:53.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.779 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:42:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:53.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.911 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:54.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.389334+0000 osd.7 (osd.7) 440 : cluster 0 9.3 scrub starts 2024-03-11T21:42:54.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.391278+0000 osd.7 (osd.7) 441 : cluster 0 9.3 scrub ok 2024-03-11T21:42:54.025 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.489573+0000 osd.2 (osd.2) 389 : cluster 0 6.14 scrub starts 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.491701+0000 osd.2 (osd.2) 390 : cluster 0 6.14 scrub ok 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.532778+0000 osd.0 (osd.0) 438 : cluster 0 9.5 scrub starts 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.535074+0000 osd.0 (osd.0) 439 : cluster 0 9.5 scrub ok 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.848250+0000 osd.3 (osd.3) 447 : cluster 0 3.18 scrub starts 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.850328+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub ok 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: cluster 2024-03-11T21:42:52.855927+0000 mon.a (mon.0) 3847 : cluster 0 osdmap e854: 8 total, 8 up, 8 in 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: audit 2024-03-11T21:42:52.870241+0000 mon.c (mon.2) 1322 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.026 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:53 smithi203 bash[23654]: audit 2024-03-11T21:42:52.871168+0000 mon.a (mon.0) 3848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:54.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:54.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:54.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.389334+0000 osd.7 (osd.7) 440 : cluster 0 9.3 scrub starts 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.391278+0000 osd.7 (osd.7) 441 : cluster 0 9.3 scrub ok 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.489573+0000 osd.2 (osd.2) 389 : cluster 0 6.14 scrub starts 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.491701+0000 osd.2 (osd.2) 390 : cluster 0 6.14 scrub ok 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.532778+0000 osd.0 (osd.0) 438 : cluster 0 9.5 scrub starts 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.535074+0000 osd.0 (osd.0) 439 : cluster 0 9.5 scrub ok 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.848250+0000 osd.3 (osd.3) 447 : cluster 0 3.18 scrub starts 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.850328+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub ok 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: cluster 2024-03-11T21:42:52.855927+0000 mon.a (mon.0) 3847 : cluster 0 osdmap e854: 8 total, 8 up, 8 in 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: audit 2024-03-11T21:42:52.870241+0000 mon.c (mon.2) 1322 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[29284]: audit 2024-03-11T21:42:52.871168+0000 mon.a (mon.0) 3848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.389334+0000 osd.7 (osd.7) 440 : cluster 0 9.3 scrub starts 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.391278+0000 osd.7 (osd.7) 441 : cluster 0 9.3 scrub ok 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.489573+0000 osd.2 (osd.2) 389 : cluster 0 6.14 scrub starts 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.491701+0000 osd.2 (osd.2) 390 : cluster 0 6.14 scrub ok 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.532778+0000 osd.0 (osd.0) 438 : cluster 0 9.5 scrub starts 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.535074+0000 osd.0 (osd.0) 439 : cluster 0 9.5 scrub ok 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.848250+0000 osd.3 (osd.3) 447 : cluster 0 3.18 scrub starts 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.850328+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub ok 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: cluster 2024-03-11T21:42:52.855927+0000 mon.a (mon.0) 3847 : cluster 0 osdmap e854: 8 total, 8 up, 8 in 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: audit 2024-03-11T21:42:52.870241+0000 mon.c (mon.2) 1322 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:53 smithi143 bash[21512]: audit 2024-03-11T21:42:52.871168+0000 mon.a (mon.0) 3848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:42:54.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:54.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.419 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:54.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:54.550 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:54.566 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:54.566 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:54.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:54.674 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:54.762 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:54.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:54.799 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:54.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:54.927 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:55.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.385753+0000 osd.7 (osd.7) 442 : cluster 0 3.d scrub starts 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.390853+0000 osd.7 (osd.7) 443 : cluster 0 3.d scrub ok 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.497994+0000 osd.2 (osd.2) 391 : cluster 0 313.5s0 scrub starts 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.501814+0000 osd.2 (osd.2) 392 : cluster 0 313.5 scrub ok 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.577452+0000 osd.5 (osd.5) 461 : cluster 0 1.8 scrub starts 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.579662+0000 osd.5 (osd.5) 462 : cluster 0 1.8 scrub ok 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.634388+0000 mgr.y (mgr.14550) 771 : cluster 0 pgmap v1328: 246 pgs: 29 creating+peering, 3 unknown, 214 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: audit 2024-03-11T21:42:53.863768+0000 mon.a (mon.0) 3849 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:53.875123+0000 mon.a (mon.0) 3850 : cluster 0 osdmap e855: 8 total, 8 up, 8 in 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: audit 2024-03-11T21:42:53.933763+0000 mon.c (mon.2) 1323 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: audit 2024-03-11T21:42:53.934308+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.059427+0000 osd.1 (osd.1) 245 : cluster 0 9.4 scrub starts 2024-03-11T21:42:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:54 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.061125+0000 osd.1 (osd.1) 246 : cluster 0 9.4 scrub ok 2024-03-11T21:42:55.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.060 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:55.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:55.190 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:55.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.385753+0000 osd.7 (osd.7) 442 : cluster 0 3.d scrub starts 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.390853+0000 osd.7 (osd.7) 443 : cluster 0 3.d scrub ok 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.497994+0000 osd.2 (osd.2) 391 : cluster 0 313.5s0 scrub starts 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.501814+0000 osd.2 (osd.2) 392 : cluster 0 313.5 scrub ok 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.577452+0000 osd.5 (osd.5) 461 : cluster 0 1.8 scrub starts 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.579662+0000 osd.5 (osd.5) 462 : cluster 0 1.8 scrub ok 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.634388+0000 mgr.y (mgr.14550) 771 : cluster 0 pgmap v1328: 246 pgs: 29 creating+peering, 3 unknown, 214 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: audit 2024-03-11T21:42:53.863768+0000 mon.a (mon.0) 3849 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:53.875123+0000 mon.a (mon.0) 3850 : cluster 0 osdmap e855: 8 total, 8 up, 8 in 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: audit 2024-03-11T21:42:53.933763+0000 mon.c (mon.2) 1323 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: audit 2024-03-11T21:42:53.934308+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.059427+0000 osd.1 (osd.1) 245 : cluster 0 9.4 scrub starts 2024-03-11T21:42:55.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.061125+0000 osd.1 (osd.1) 246 : cluster 0 9.4 scrub ok 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.385753+0000 osd.7 (osd.7) 442 : cluster 0 3.d scrub starts 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.390853+0000 osd.7 (osd.7) 443 : cluster 0 3.d scrub ok 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.497994+0000 osd.2 (osd.2) 391 : cluster 0 313.5s0 scrub starts 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.501814+0000 osd.2 (osd.2) 392 : cluster 0 313.5 scrub ok 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.577452+0000 osd.5 (osd.5) 461 : cluster 0 1.8 scrub starts 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.579662+0000 osd.5 (osd.5) 462 : cluster 0 1.8 scrub ok 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.634388+0000 mgr.y (mgr.14550) 771 : cluster 0 pgmap v1328: 246 pgs: 29 creating+peering, 3 unknown, 214 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: audit 2024-03-11T21:42:53.863768+0000 mon.a (mon.0) 3849 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:53.875123+0000 mon.a (mon.0) 3850 : cluster 0 osdmap e855: 8 total, 8 up, 8 in 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: audit 2024-03-11T21:42:53.933763+0000 mon.c (mon.2) 1323 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: audit 2024-03-11T21:42:53.934308+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.059427+0000 osd.1 (osd.1) 245 : cluster 0 9.4 scrub starts 2024-03-11T21:42:55.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:54 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.061125+0000 osd.1 (osd.1) 246 : cluster 0 9.4 scrub ok 2024-03-11T21:42:55.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:55.320 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:55.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:42:55.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:55.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:55.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:55.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:55.706 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:55.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:55.836 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:55.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:55.944 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.458337+0000 osd.0 (osd.0) 440 : cluster 0 9.b scrub starts 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.460832+0000 osd.0 (osd.0) 441 : cluster 0 9.b scrub ok 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.752212+0000 osd.3 (osd.3) 449 : cluster 0 9.19 scrub starts 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.754511+0000 osd.3 (osd.3) 450 : cluster 0 9.19 scrub ok 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: audit 2024-03-11T21:42:54.866225+0000 mon.a (mon.0) 3852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: audit 2024-03-11T21:42:54.870851+0000 mon.c (mon.2) 1324 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:54.874845+0000 mon.a (mon.0) 3853 : cluster 0 osdmap e856: 8 total, 8 up, 8 in 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: audit 2024-03-11T21:42:54.877028+0000 mon.a (mon.0) 3854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.083456+0000 osd.1 (osd.1) 247 : cluster 0 2.9 scrub starts 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.085810+0000 osd.1 (osd.1) 248 : cluster 0 2.9 scrub ok 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: audit 2024-03-11T21:42:55.867991+0000 mon.a (mon.0) 3855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.877355+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e857: 8 total, 8 up, 8 in 2024-03-11T21:42:55.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.458337+0000 osd.0 (osd.0) 440 : cluster 0 9.b scrub starts 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.460832+0000 osd.0 (osd.0) 441 : cluster 0 9.b scrub ok 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.752212+0000 osd.3 (osd.3) 449 : cluster 0 9.19 scrub starts 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.754511+0000 osd.3 (osd.3) 450 : cluster 0 9.19 scrub ok 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: audit 2024-03-11T21:42:54.866225+0000 mon.a (mon.0) 3852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: audit 2024-03-11T21:42:54.870851+0000 mon.c (mon.2) 1324 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:54.874845+0000 mon.a (mon.0) 3853 : cluster 0 osdmap e856: 8 total, 8 up, 8 in 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: audit 2024-03-11T21:42:54.877028+0000 mon.a (mon.0) 3854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.083456+0000 osd.1 (osd.1) 247 : cluster 0 2.9 scrub starts 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.085810+0000 osd.1 (osd.1) 248 : cluster 0 2.9 scrub ok 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: audit 2024-03-11T21:42:55.867991+0000 mon.a (mon.0) 3855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:42:55.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:55 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.877355+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e857: 8 total, 8 up, 8 in 2024-03-11T21:42:55.963 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:56.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.093 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:56.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:56.208 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.458337+0000 osd.0 (osd.0) 440 : cluster 0 9.b scrub starts 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.460832+0000 osd.0 (osd.0) 441 : cluster 0 9.b scrub ok 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.752212+0000 osd.3 (osd.3) 449 : cluster 0 9.19 scrub starts 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.754511+0000 osd.3 (osd.3) 450 : cluster 0 9.19 scrub ok 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: audit 2024-03-11T21:42:54.866225+0000 mon.a (mon.0) 3852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: audit 2024-03-11T21:42:54.870851+0000 mon.c (mon.2) 1324 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:54.874845+0000 mon.a (mon.0) 3853 : cluster 0 osdmap e856: 8 total, 8 up, 8 in 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: audit 2024-03-11T21:42:54.877028+0000 mon.a (mon.0) 3854 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.083456+0000 osd.1 (osd.1) 247 : cluster 0 2.9 scrub starts 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.085810+0000 osd.1 (osd.1) 248 : cluster 0 2.9 scrub ok 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: audit 2024-03-11T21:42:55.867991+0000 mon.a (mon.0) 3855 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi143-71773-111", "overlaypool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:42:56.209 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:55 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.877355+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e857: 8 total, 8 up, 8 in 2024-03-11T21:42:56.225 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:56.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:56.356 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:56.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.487 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:56.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.612 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:56.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:56.734 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:56.808 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:56.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.855 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:56.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.977 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:57.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.504805+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub starts 2024-03-11T21:42:57.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.507582+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub starts 2024-03-11T21:42:57.035 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.509509+0000 osd.0 (osd.0) 443 : cluster 0 3.1f scrub ok 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.510362+0000 osd.2 (osd.2) 394 : cluster 0 6.f scrub ok 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.543785+0000 osd.5 (osd.5) 463 : cluster 0 39.0 scrub starts 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.545185+0000 osd.5 (osd.5) 464 : cluster 0 39.0 scrub ok 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.635526+0000 mgr.y (mgr.14550) 772 : cluster 0 pgmap v1331: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.705977+0000 osd.3 (osd.3) 451 : cluster 0 39.e scrub starts 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:55.707917+0000 osd.3 (osd.3) 452 : cluster 0 39.e scrub ok 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:55.874290+0000 mon.c (mon.2) 1325 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:55.880914+0000 mon.a (mon.0) 3857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:56.530056+0000 mon.c (mon.2) 1326 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.557726+0000 mon.a (mon.0) 3858 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:56.559565+0000 mon.a (mon.0) 3859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]': finished 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.563632+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e858: 8 total, 8 up, 8 in 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:56.627394+0000 mon.c (mon.2) 1327 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: audit 2024-03-11T21:42:56.628142+0000 mon.a (mon.0) 3861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.036 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.715258+0000 osd.4 (osd.4) 433 : cluster 0 4.1 scrub starts 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.717317+0000 osd.4 (osd.4) 434 : cluster 0 4.1 scrub ok 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.504805+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub starts 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.507582+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub starts 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.509509+0000 osd.0 (osd.0) 443 : cluster 0 3.1f scrub ok 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.510362+0000 osd.2 (osd.2) 394 : cluster 0 6.f scrub ok 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.543785+0000 osd.5 (osd.5) 463 : cluster 0 39.0 scrub starts 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.545185+0000 osd.5 (osd.5) 464 : cluster 0 39.0 scrub ok 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.635526+0000 mgr.y (mgr.14550) 772 : cluster 0 pgmap v1331: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.705977+0000 osd.3 (osd.3) 451 : cluster 0 39.e scrub starts 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:55.707917+0000 osd.3 (osd.3) 452 : cluster 0 39.e scrub ok 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:55.874290+0000 mon.c (mon.2) 1325 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:55.880914+0000 mon.a (mon.0) 3857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.037 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:56.530056+0000 mon.c (mon.2) 1326 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.557726+0000 mon.a (mon.0) 3858 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:56.559565+0000 mon.a (mon.0) 3859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]': finished 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.563632+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e858: 8 total, 8 up, 8 in 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:56.627394+0000 mon.c (mon.2) 1327 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: audit 2024-03-11T21:42:56.628142+0000 mon.a (mon.0) 3861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.715258+0000 osd.4 (osd.4) 433 : cluster 0 4.1 scrub starts 2024-03-11T21:42:57.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:56 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.717317+0000 osd.4 (osd.4) 434 : cluster 0 4.1 scrub ok 2024-03-11T21:42:57.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:57.102 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:57.205 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:57.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.504805+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub starts 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.507582+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub starts 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.509509+0000 osd.0 (osd.0) 443 : cluster 0 3.1f scrub ok 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.510362+0000 osd.2 (osd.2) 394 : cluster 0 6.f scrub ok 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.543785+0000 osd.5 (osd.5) 463 : cluster 0 39.0 scrub starts 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.545185+0000 osd.5 (osd.5) 464 : cluster 0 39.0 scrub ok 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.635526+0000 mgr.y (mgr.14550) 772 : cluster 0 pgmap v1331: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.705977+0000 osd.3 (osd.3) 451 : cluster 0 39.e scrub starts 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:55.707917+0000 osd.3 (osd.3) 452 : cluster 0 39.e scrub ok 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:55.874290+0000 mon.c (mon.2) 1325 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:55.880914+0000 mon.a (mon.0) 3857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]: dispatch 2024-03-11T21:42:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:56.530056+0000 mon.c (mon.2) 1326 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.557726+0000 mon.a (mon.0) 3858 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:56.559565+0000 mon.a (mon.0) 3859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi143-71773-146", "mode": "writeback"}]': finished 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.563632+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e858: 8 total, 8 up, 8 in 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:56.627394+0000 mon.c (mon.2) 1327 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: audit 2024-03-11T21:42:56.628142+0000 mon.a (mon.0) 3861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.715258+0000 osd.4 (osd.4) 433 : cluster 0 4.1 scrub starts 2024-03-11T21:42:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:56 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.717317+0000 osd.4 (osd.4) 434 : cluster 0 4.1 scrub ok 2024-03-11T21:42:57.330 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.351 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:57.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.480 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:57.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.609 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:57.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.739 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:57.842 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:57.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:58.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:58.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.347065+0000 osd.7 (osd.7) 444 : cluster 0 39.5 scrub starts 2024-03-11T21:42:58.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.351524+0000 osd.7 (osd.7) 445 : cluster 0 39.5 scrub ok 2024-03-11T21:42:58.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.550823+0000 osd.5 (osd.5) 465 : cluster 0 3.5 scrub starts 2024-03-11T21:42:58.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.554314+0000 osd.5 (osd.5) 466 : cluster 0 3.5 scrub ok 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.736602+0000 osd.3 (osd.3) 453 : cluster 0 6.1b scrub starts 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:56.738373+0000 osd.3 (osd.3) 454 : cluster 0 6.1b scrub ok 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.214540+0000 mon.c (mon.2) 1328 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.216149+0000 mon.c (mon.2) 1329 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.226362+0000 mon.a (mon.0) 3862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.295577+0000 osd.6 (osd.6) 369 : cluster 0 9.18 scrub starts 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.297080+0000 osd.6 (osd.6) 370 : cluster 0 9.18 scrub ok 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.562553+0000 mon.a (mon.0) 3863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.569713+0000 mon.a (mon.0) 3864 : cluster 0 osdmap e859: 8 total, 8 up, 8 in 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.572847+0000 mon.c (mon.2) 1330 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: audit 2024-03-11T21:42:57.573722+0000 mon.a (mon.0) 3865 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.709518+0000 osd.4 (osd.4) 435 : cluster 0 2.f scrub starts 2024-03-11T21:42:58.115 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:57 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.711938+0000 osd.4 (osd.4) 436 : cluster 0 2.f scrub ok 2024-03-11T21:42:58.127 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.347065+0000 osd.7 (osd.7) 444 : cluster 0 39.5 scrub starts 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.351524+0000 osd.7 (osd.7) 445 : cluster 0 39.5 scrub ok 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.550823+0000 osd.5 (osd.5) 465 : cluster 0 3.5 scrub starts 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.554314+0000 osd.5 (osd.5) 466 : cluster 0 3.5 scrub ok 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.736602+0000 osd.3 (osd.3) 453 : cluster 0 6.1b scrub starts 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:56.738373+0000 osd.3 (osd.3) 454 : cluster 0 6.1b scrub ok 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.214540+0000 mon.c (mon.2) 1328 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.216149+0000 mon.c (mon.2) 1329 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.226362+0000 mon.a (mon.0) 3862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.295577+0000 osd.6 (osd.6) 369 : cluster 0 9.18 scrub starts 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.297080+0000 osd.6 (osd.6) 370 : cluster 0 9.18 scrub ok 2024-03-11T21:42:58.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.562553+0000 mon.a (mon.0) 3863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.569713+0000 mon.a (mon.0) 3864 : cluster 0 osdmap e859: 8 total, 8 up, 8 in 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.572847+0000 mon.c (mon.2) 1330 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: audit 2024-03-11T21:42:57.573722+0000 mon.a (mon.0) 3865 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.709518+0000 osd.4 (osd.4) 435 : cluster 0 2.f scrub starts 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.711938+0000 osd.4 (osd.4) 436 : cluster 0 2.f scrub ok 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.347065+0000 osd.7 (osd.7) 444 : cluster 0 39.5 scrub starts 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.351524+0000 osd.7 (osd.7) 445 : cluster 0 39.5 scrub ok 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.550823+0000 osd.5 (osd.5) 465 : cluster 0 3.5 scrub starts 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.554314+0000 osd.5 (osd.5) 466 : cluster 0 3.5 scrub ok 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.736602+0000 osd.3 (osd.3) 453 : cluster 0 6.1b scrub starts 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:56.738373+0000 osd.3 (osd.3) 454 : cluster 0 6.1b scrub ok 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.214540+0000 mon.c (mon.2) 1328 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:42:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.216149+0000 mon.c (mon.2) 1329 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.226362+0000 mon.a (mon.0) 3862 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.295577+0000 osd.6 (osd.6) 369 : cluster 0 9.18 scrub starts 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.297080+0000 osd.6 (osd.6) 370 : cluster 0 9.18 scrub ok 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.562553+0000 mon.a (mon.0) 3863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.569713+0000 mon.a (mon.0) 3864 : cluster 0 osdmap e859: 8 total, 8 up, 8 in 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.572847+0000 mon.c (mon.2) 1330 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: audit 2024-03-11T21:42:57.573722+0000 mon.a (mon.0) 3865 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.709518+0000 osd.4 (osd.4) 435 : cluster 0 2.f scrub starts 2024-03-11T21:42:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:57 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.711938+0000 osd.4 (osd.4) 436 : cluster 0 2.f scrub ok 2024-03-11T21:42:58.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:58.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:58.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.388 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:58.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.519 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:58.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:58.650 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:58.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:58.781 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:58.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:58.910 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:58.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[21512]: cluster 2024-03-11T21:42:57.636413+0000 mgr.y (mgr.14550) 773 : cluster 0 pgmap v1335: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:42:58.944 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[21512]: audit 2024-03-11T21:42:58.576078+0000 mon.a (mon.0) 3866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.583234+0000 mon.a (mon.0) 3867 : cluster 0 osdmap e860: 8 total, 8 up, 8 in 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[21512]: audit 2024-03-11T21:42:58.588640+0000 mon.c (mon.2) 1331 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[21512]: audit 2024-03-11T21:42:58.589640+0000 mon.a (mon.0) 3868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[29284]: cluster 2024-03-11T21:42:57.636413+0000 mgr.y (mgr.14550) 773 : cluster 0 pgmap v1335: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[29284]: audit 2024-03-11T21:42:58.576078+0000 mon.a (mon.0) 3866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.583234+0000 mon.a (mon.0) 3867 : cluster 0 osdmap e860: 8 total, 8 up, 8 in 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[29284]: audit 2024-03-11T21:42:58.588640+0000 mon.c (mon.2) 1331 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:58.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:58 smithi143 bash[29284]: audit 2024-03-11T21:42:58.589640+0000 mon.a (mon.0) 3868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:59.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:58 smithi203 bash[23654]: cluster 2024-03-11T21:42:57.636413+0000 mgr.y (mgr.14550) 773 : cluster 0 pgmap v1335: 246 pgs: 29 creating+peering, 217 active+clean; 4.4 MiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:42:59.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:58 smithi203 bash[23654]: audit 2024-03-11T21:42:58.576078+0000 mon.a (mon.0) 3866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:42:59.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:58 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.583234+0000 mon.a (mon.0) 3867 : cluster 0 osdmap e860: 8 total, 8 up, 8 in 2024-03-11T21:42:59.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:58 smithi203 bash[23654]: audit 2024-03-11T21:42:58.588640+0000 mon.c (mon.2) 1331 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:59.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:58 smithi203 bash[23654]: audit 2024-03-11T21:42:58.589640+0000 mon.a (mon.0) 3868 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:42:59.042 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:59.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:42:59.171 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:59.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:59.275 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:59.296 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:59.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:59.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:59.553 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:59.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:59.684 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:59.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:59.815 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:59.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:59.943 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.337718+0000 osd.6 (osd.6) 371 : cluster 0 39.1d scrub starts 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.341419+0000 osd.6 (osd.6) 372 : cluster 0 39.1d scrub ok 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.420523+0000 osd.7 (osd.7) 446 : cluster 0 6.15 scrub starts 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.421792+0000 osd.7 (osd.7) 447 : cluster 0 6.15 scrub ok 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.425106+0000 osd.0 (osd.0) 444 : cluster 0 9.1e scrub starts 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.427432+0000 osd.0 (osd.0) 445 : cluster 0 9.1e scrub ok 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.544197+0000 osd.2 (osd.2) 395 : cluster 0 5.d scrub starts 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:58.546625+0000 osd.2 (osd.2) 396 : cluster 0 5.d scrub ok 2024-03-11T21:43:00.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: audit 2024-03-11T21:42:59.077224+0000 mgr.y (mgr.14550) 774 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:00.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.576303+0000 mon.a (mon.0) 3869 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:43:00.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: audit 2024-03-11T21:42:59.578114+0000 mon.a (mon.0) 3870 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:43:00.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.582637+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e861: 8 total, 8 up, 8 in 2024-03-11T21:43:00.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: audit 2024-03-11T21:42:59.584509+0000 mon.c (mon.2) 1332 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:42:59 smithi203 bash[23654]: audit 2024-03-11T21:42:59.584993+0000 mon.a (mon.0) 3872 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:00.066 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.337718+0000 osd.6 (osd.6) 371 : cluster 0 39.1d scrub starts 2024-03-11T21:43:00.066 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.341419+0000 osd.6 (osd.6) 372 : cluster 0 39.1d scrub ok 2024-03-11T21:43:00.066 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.420523+0000 osd.7 (osd.7) 446 : cluster 0 6.15 scrub starts 2024-03-11T21:43:00.066 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.421792+0000 osd.7 (osd.7) 447 : cluster 0 6.15 scrub ok 2024-03-11T21:43:00.066 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.425106+0000 osd.0 (osd.0) 444 : cluster 0 9.1e scrub starts 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.427432+0000 osd.0 (osd.0) 445 : cluster 0 9.1e scrub ok 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.544197+0000 osd.2 (osd.2) 395 : cluster 0 5.d scrub starts 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:58.546625+0000 osd.2 (osd.2) 396 : cluster 0 5.d scrub ok 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: audit 2024-03-11T21:42:59.077224+0000 mgr.y (mgr.14550) 774 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.576303+0000 mon.a (mon.0) 3869 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: audit 2024-03-11T21:42:59.578114+0000 mon.a (mon.0) 3870 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.582637+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e861: 8 total, 8 up, 8 in 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: audit 2024-03-11T21:42:59.584509+0000 mon.c (mon.2) 1332 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[21512]: audit 2024-03-11T21:42:59.584993+0000 mon.a (mon.0) 3872 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.337718+0000 osd.6 (osd.6) 371 : cluster 0 39.1d scrub starts 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.341419+0000 osd.6 (osd.6) 372 : cluster 0 39.1d scrub ok 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.420523+0000 osd.7 (osd.7) 446 : cluster 0 6.15 scrub starts 2024-03-11T21:43:00.067 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.421792+0000 osd.7 (osd.7) 447 : cluster 0 6.15 scrub ok 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.425106+0000 osd.0 (osd.0) 444 : cluster 0 9.1e scrub starts 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.427432+0000 osd.0 (osd.0) 445 : cluster 0 9.1e scrub ok 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.544197+0000 osd.2 (osd.2) 395 : cluster 0 5.d scrub starts 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:58.546625+0000 osd.2 (osd.2) 396 : cluster 0 5.d scrub ok 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: audit 2024-03-11T21:42:59.077224+0000 mgr.y (mgr.14550) 774 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.576303+0000 mon.a (mon.0) 3869 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: audit 2024-03-11T21:42:59.578114+0000 mon.a (mon.0) 3870 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.582637+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e861: 8 total, 8 up, 8 in 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: audit 2024-03-11T21:42:59.584509+0000 mon.c (mon.2) 1332 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.068 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:42:59 smithi143 bash[29284]: audit 2024-03-11T21:42:59.584993+0000 mon.a (mon.0) 3872 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:43:00.074 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:00.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:00.203 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:00.222 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:00.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:00.331 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:00.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:00.462 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:00.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:00.594 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:00.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:00.719 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:00.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:00.851 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.413326+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.415638+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.431386+0000 osd.7 (osd.7) 448 : cluster 0 3.11 scrub starts 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.435544+0000 osd.7 (osd.7) 449 : cluster 0 3.11 scrub ok 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.637311+0000 mgr.y (mgr.14550) 775 : cluster 0 pgmap v1338: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.783409+0000 osd.3 (osd.3) 455 : cluster 0 39.18 scrub starts 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.786369+0000 osd.3 (osd.3) 456 : cluster 0 39.18 scrub ok 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.945852+0000 osd.1 (osd.1) 249 : cluster 0 4.2 scrub starts 2024-03-11T21:43:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:42:59.949150+0000 osd.1 (osd.1) 250 : cluster 0 4.2 scrub ok 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: audit 2024-03-11T21:43:00.586974+0000 mon.a (mon.0) 3873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: cluster 2024-03-11T21:43:00.594201+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e862: 8 total, 8 up, 8 in 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: audit 2024-03-11T21:43:00.596525+0000 mon.c (mon.2) 1333 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[21512]: audit 2024-03-11T21:43:00.597048+0000 mon.a (mon.0) 3875 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.413326+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.415638+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.431386+0000 osd.7 (osd.7) 448 : cluster 0 3.11 scrub starts 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.435544+0000 osd.7 (osd.7) 449 : cluster 0 3.11 scrub ok 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.637311+0000 mgr.y (mgr.14550) 775 : cluster 0 pgmap v1338: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.783409+0000 osd.3 (osd.3) 455 : cluster 0 39.18 scrub starts 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.786369+0000 osd.3 (osd.3) 456 : cluster 0 39.18 scrub ok 2024-03-11T21:43:00.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.945852+0000 osd.1 (osd.1) 249 : cluster 0 4.2 scrub starts 2024-03-11T21:43:00.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:42:59.949150+0000 osd.1 (osd.1) 250 : cluster 0 4.2 scrub ok 2024-03-11T21:43:00.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: audit 2024-03-11T21:43:00.586974+0000 mon.a (mon.0) 3873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:43:00.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: cluster 2024-03-11T21:43:00.594201+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e862: 8 total, 8 up, 8 in 2024-03-11T21:43:00.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: audit 2024-03-11T21:43:00.596525+0000 mon.c (mon.2) 1333 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:00.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:00 smithi143 bash[29284]: audit 2024-03-11T21:43:00.597048+0000 mon.a (mon.0) 3875 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:00.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:00.978 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:01.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:01.092 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.413326+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-03-11T21:43:01.092 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.415638+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-03-11T21:43:01.092 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.431386+0000 osd.7 (osd.7) 448 : cluster 0 3.11 scrub starts 2024-03-11T21:43:01.092 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.435544+0000 osd.7 (osd.7) 449 : cluster 0 3.11 scrub ok 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.637311+0000 mgr.y (mgr.14550) 775 : cluster 0 pgmap v1338: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.783409+0000 osd.3 (osd.3) 455 : cluster 0 39.18 scrub starts 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.786369+0000 osd.3 (osd.3) 456 : cluster 0 39.18 scrub ok 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.945852+0000 osd.1 (osd.1) 249 : cluster 0 4.2 scrub starts 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:42:59.949150+0000 osd.1 (osd.1) 250 : cluster 0 4.2 scrub ok 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: audit 2024-03-11T21:43:00.586974+0000 mon.a (mon.0) 3873 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: cluster 2024-03-11T21:43:00.594201+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e862: 8 total, 8 up, 8 in 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: audit 2024-03-11T21:43:00.596525+0000 mon.c (mon.2) 1333 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:01.093 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:00 smithi203 bash[23654]: audit 2024-03-11T21:43:00.597048+0000 mon.a (mon.0) 3875 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:43:01.107 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:01.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:01.237 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:01.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.366 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:01.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:01.495 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:01.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:01.621 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:01.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.752 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:01.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:01.877 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:01.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:01 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.558661+0000 mon.a (mon.0) 3876 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:01.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:01 smithi203 bash[23654]: audit 2024-03-11T21:43:01.589063+0000 mon.a (mon.0) 3877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:43:01.968 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:01 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.602795+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e863: 8 total, 8 up, 8 in 2024-03-11T21:43:01.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.008 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:02.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.115 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.558661+0000 mon.a (mon.0) 3876 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:02.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[21512]: audit 2024-03-11T21:43:01.589063+0000 mon.a (mon.0) 3877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:43:02.116 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.602795+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e863: 8 total, 8 up, 8 in 2024-03-11T21:43:02.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.558661+0000 mon.a (mon.0) 3876 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:02.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[29284]: audit 2024-03-11T21:43:01.589063+0000 mon.a (mon.0) 3877 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:43:02.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:01 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.602795+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e863: 8 total, 8 up, 8 in 2024-03-11T21:43:02.139 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.140 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:02.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.271 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:02.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.402 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:02.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.533 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:02.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.663 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:02.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:02.788 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:02.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.410702+0000 osd.0 (osd.0) 448 : cluster 0 39.f scrub starts 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.415611+0000 osd.0 (osd.0) 449 : cluster 0 39.f scrub ok 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.433921+0000 osd.7 (osd.7) 450 : cluster 0 6.5 scrub starts 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.435962+0000 osd.7 (osd.7) 451 : cluster 0 6.5 scrub ok 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.638040+0000 mgr.y (mgr.14550) 776 : cluster 0 pgmap v1341: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.809991+0000 osd.3 (osd.3) 457 : cluster 0 3.b scrub starts 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:01.813925+0000 osd.3 (osd.3) 458 : cluster 0 3.b scrub ok 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.317081+0000 osd.6 (osd.6) 373 : cluster 0 39.8 scrub starts 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.319020+0000 osd.6 (osd.6) 374 : cluster 0 39.8 scrub ok 2024-03-11T21:43:02.903 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.491963+0000 osd.5 (osd.5) 467 : cluster 0 2.18 scrub starts 2024-03-11T21:43:02.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.494106+0000 osd.5 (osd.5) 468 : cluster 0 2.18 scrub ok 2024-03-11T21:43:02.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.589167+0000 mon.a (mon.0) 3879 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:43:02.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.621269+0000 osd.4 (osd.4) 437 : cluster 0 3.9 scrub starts 2024-03-11T21:43:02.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.625413+0000 osd.4 (osd.4) 438 : cluster 0 3.9 scrub ok 2024-03-11T21:43:02.904 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.410702+0000 osd.0 (osd.0) 448 : cluster 0 39.f scrub starts 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.415611+0000 osd.0 (osd.0) 449 : cluster 0 39.f scrub ok 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.433921+0000 osd.7 (osd.7) 450 : cluster 0 6.5 scrub starts 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.435962+0000 osd.7 (osd.7) 451 : cluster 0 6.5 scrub ok 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.638040+0000 mgr.y (mgr.14550) 776 : cluster 0 pgmap v1341: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.809991+0000 osd.3 (osd.3) 457 : cluster 0 3.b scrub starts 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:01.813925+0000 osd.3 (osd.3) 458 : cluster 0 3.b scrub ok 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.317081+0000 osd.6 (osd.6) 373 : cluster 0 39.8 scrub starts 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.319020+0000 osd.6 (osd.6) 374 : cluster 0 39.8 scrub ok 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.491963+0000 osd.5 (osd.5) 467 : cluster 0 2.18 scrub starts 2024-03-11T21:43:02.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.494106+0000 osd.5 (osd.5) 468 : cluster 0 2.18 scrub ok 2024-03-11T21:43:02.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.589167+0000 mon.a (mon.0) 3879 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:43:02.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.621269+0000 osd.4 (osd.4) 437 : cluster 0 3.9 scrub starts 2024-03-11T21:43:02.906 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:02 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.625413+0000 osd.4 (osd.4) 438 : cluster 0 3.9 scrub ok 2024-03-11T21:43:02.917 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:03.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:03.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.410702+0000 osd.0 (osd.0) 448 : cluster 0 39.f scrub starts 2024-03-11T21:43:03.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.415611+0000 osd.0 (osd.0) 449 : cluster 0 39.f scrub ok 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.433921+0000 osd.7 (osd.7) 450 : cluster 0 6.5 scrub starts 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.435962+0000 osd.7 (osd.7) 451 : cluster 0 6.5 scrub ok 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.638040+0000 mgr.y (mgr.14550) 776 : cluster 0 pgmap v1341: 246 pgs: 246 active+clean; 4.4 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.809991+0000 osd.3 (osd.3) 457 : cluster 0 3.b scrub starts 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:01.813925+0000 osd.3 (osd.3) 458 : cluster 0 3.b scrub ok 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.317081+0000 osd.6 (osd.6) 373 : cluster 0 39.8 scrub starts 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.319020+0000 osd.6 (osd.6) 374 : cluster 0 39.8 scrub ok 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.491963+0000 osd.5 (osd.5) 467 : cluster 0 2.18 scrub starts 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.494106+0000 osd.5 (osd.5) 468 : cluster 0 2.18 scrub ok 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.589167+0000 mon.a (mon.0) 3879 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.621269+0000 osd.4 (osd.4) 437 : cluster 0 3.9 scrub starts 2024-03-11T21:43:03.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:02 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.625413+0000 osd.4 (osd.4) 438 : cluster 0 3.9 scrub ok 2024-03-11T21:43:03.048 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:03.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:03.180 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:03.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:03.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.440 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:03.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:03.572 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:03.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:03.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:03.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:03.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.958 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:04.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:03 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.434002+0000 osd.0 (osd.0) 450 : cluster 0 39.12 scrub starts 2024-03-11T21:43:04.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:03 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.436837+0000 osd.0 (osd.0) 451 : cluster 0 39.12 scrub ok 2024-03-11T21:43:04.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:03 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.845793+0000 osd.1 (osd.1) 251 : cluster 0 6.1 scrub starts 2024-03-11T21:43:04.015 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:03 smithi143 bash[21512]: cluster 2024-03-11T21:43:02.848083+0000 osd.1 (osd.1) 252 : cluster 0 6.1 scrub ok 2024-03-11T21:43:04.061 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.434002+0000 osd.0 (osd.0) 450 : cluster 0 39.12 scrub starts 2024-03-11T21:43:04.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.436837+0000 osd.0 (osd.0) 451 : cluster 0 39.12 scrub ok 2024-03-11T21:43:04.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.845793+0000 osd.1 (osd.1) 251 : cluster 0 6.1 scrub starts 2024-03-11T21:43:04.073 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:02.848083+0000 osd.1 (osd.1) 252 : cluster 0 6.1 scrub ok 2024-03-11T21:43:04.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:04.131 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:04.131 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:04.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:03 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.434002+0000 osd.0 (osd.0) 450 : cluster 0 39.12 scrub starts 2024-03-11T21:43:04.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:03 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.436837+0000 osd.0 (osd.0) 451 : cluster 0 39.12 scrub ok 2024-03-11T21:43:04.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:03 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.845793+0000 osd.1 (osd.1) 251 : cluster 0 6.1 scrub starts 2024-03-11T21:43:04.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:03 smithi203 bash[23654]: cluster 2024-03-11T21:43:02.848083+0000 osd.1 (osd.1) 252 : cluster 0 6.1 scrub ok 2024-03-11T21:43:04.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:04.219 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:04.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:04.349 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:04.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:04.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:04.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:04.735 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:04.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 -- 172.21.15.143:0/1328462781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 msgr2=0x7fb3341071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:04.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 --2- 172.21.15.143:0/1328462781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341071b0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7fb324009990 tx=0x7fb324009ca0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 -- 172.21.15.143:0/1328462781 shutdown_connections 2024-03-11T21:43:04.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 --2- 172.21.15.143:0/1328462781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3341016e0 0x7fb33410db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 --2- 172.21.15.143:0/1328462781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341071b0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 --2- 172.21.15.143:0/1328462781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb334100d20 0x7fb3341011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 -- 172.21.15.143:0/1328462781 >> 172.21.15.143:0/1328462781 conn(0x7fb334077c80 msgr2=0x7fb334078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 -- 172.21.15.143:0/1328462781 shutdown_connections 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.770+0000 7fb3388bf700 1 -- 172.21.15.143:0/1328462781 wait complete. 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 librados: starting msgr at 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 librados: starting objecter 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 Processor -- start 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- start start 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 librados: setting wanted keys 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 librados: calling monclient init 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:04.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3341016e0 0x7fb33419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410f470 con 0x7fb334100d20 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410f2f0 con 0x7fb3341016e0 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410f5f0 con 0x7fb334106dd0 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48280/0 (socket says 172.21.15.143:48280) 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb332d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 -- 172.21.15.143:0/2374175560 learned_addr learned my addr 172.21.15.143:0/2374175560 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb331d9b700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3341016e0 0x7fb33419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 -- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 msgr2=0x7fb3341957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 -- 172.21.15.143:0/2374175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3341016e0 msgr2=0x7fb33419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3341016e0 0x7fb33419bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:04.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 -- 172.21.15.143:0/2374175560 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334195fd0 con 0x7fb334100d20 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb33259c700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb3240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 10 client.?.objecter ms_handle_connect 0x7fb334100d20 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324027070 con 0x7fb334100d20 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb334196260 con 0x7fb334100d20 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb324022470 con 0x7fb334100d20 2024-03-11T21:43:04.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324017650 con 0x7fb334100d20 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 10 client.30891.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341a24d0 con 0x7fb334100d20 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 librados: init done 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334102410 con 0x7fb334100d20 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb32402a070 con 0x7fb334100d20 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 0x7fb31c073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(863..863 src has 87..863) v4 ==== 10287+0+0 (secure 0 0 0) 0x7fb32408fce0 con 0x7fb334100d20 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 10 client.30891.objecter ms_dispatch 0x7fb334001550 osd_map(863..863 src has 87..863) v4 2024-03-11T21:43:04.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 3 client.30891.objecter handle_osd_map got epochs [863,863] > 0 2024-03-11T21:43:04.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb3237fe700 3 client.30891.objecter handle_osd_map decoding full epoch 863 2024-03-11T21:43:04.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.774+0000 7fb331d9b700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 0x7fb31c073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:04.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.778+0000 7fb3237fe700 20 client.30891.objecter dump_active .. 0 homeless 2024-03-11T21:43:04.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.778+0000 7fb3237fe700 10 client.30891.objecter ms_handle_connect 0x7fb31c071a30 2024-03-11T21:43:04.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.778+0000 7fb331d9b700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 0x7fb31c073ef0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fb328006fd0 tx=0x7fb328008040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:04.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.782+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb324013070 con 0x7fb334100d20 2024-03-11T21:43:04.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:04.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:04.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:04.994+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3341081d0 con 0x7fb31c071a30 2024-03-11T21:43:04.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:43:04.986 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:05.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.014+0000 7fb3237fe700 1 -- 172.21.15.143:0/2374175560 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+632335 (secure 0 0 0) 0x7fb3341081d0 con 0x7fb31c071a30 2024-03-11T21:43:05.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:43:05.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 10 librados: watch_flush enter 2024-03-11T21:43:05.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 10 librados: watch_flush exit 2024-03-11T21:43:05.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 20 client.30891.objecter shutdown clearing up homeless session... 2024-03-11T21:43:05.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 10 client.30891.objecter successfully canceled tick 2024-03-11T21:43:05.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 msgr2=0x7fb31c073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:05.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 0x7fb31c073ef0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fb328006fd0 tx=0x7fb328008040 comp rx=0 tx=0).stop 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 msgr2=0x7fb33419b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb3240047b0 comp rx=0 tx=0).stop 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 shutdown_connections 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb31c071a30 0x7fb31c073ef0 unknown :-1 s=CLOSED pgs=3459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334100d20 0x7fb33419b670 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:05.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334106dd0 0x7fb3341957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 --2- 172.21.15.143:0/2374175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3341016e0 0x7fb33419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 >> 172.21.15.143:0/2374175560 conn(0x7fb334077c80 msgr2=0x7fb334107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 shutdown_connections 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 -- 172.21.15.143:0/2374175560 wait complete. 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:05.018+0000 7fb3388bf700 1 librados: shutdown 2024-03-11T21:43:05.017 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:43:05.077 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[21512]: cluster 2024-03-11T21:43:03.640346+0000 mgr.y (mgr.14550) 777 : cluster 0 pgmap v1342: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:43:05.078 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[21512]: cluster 2024-03-11T21:43:03.767723+0000 osd.3 (osd.3) 459 : cluster 0 39.d scrub starts 2024-03-11T21:43:05.078 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[21512]: cluster 2024-03-11T21:43:03.770558+0000 osd.3 (osd.3) 460 : cluster 0 39.d scrub ok 2024-03-11T21:43:05.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:03.640346+0000 mgr.y (mgr.14550) 777 : cluster 0 pgmap v1342: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:43:05.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:03.767723+0000 osd.3 (osd.3) 459 : cluster 0 39.d scrub starts 2024-03-11T21:43:05.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:04 smithi143 bash[29284]: cluster 2024-03-11T21:43:03.770558+0000 osd.3 (osd.3) 460 : cluster 0 39.d scrub ok 2024-03-11T21:43:05.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:05.099 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:04 smithi203 bash[23654]: cluster 2024-03-11T21:43:03.640346+0000 mgr.y (mgr.14550) 777 : cluster 0 pgmap v1342: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 202 B/s wr, 1 op/s 2024-03-11T21:43:05.099 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:04 smithi203 bash[23654]: cluster 2024-03-11T21:43:03.767723+0000 osd.3 (osd.3) 459 : cluster 0 39.d scrub starts 2024-03-11T21:43:05.099 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:04 smithi203 bash[23654]: cluster 2024-03-11T21:43:03.770558+0000 osd.3 (osd.3) 460 : cluster 0 39.d scrub ok 2024-03-11T21:43:05.112 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:05.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:05.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:05.370 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:05.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:05.501 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:05.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.631 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:05.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:05.761 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:05.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:05.891 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:05.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.384683+0000 osd.0 (osd.0) 452 : cluster 0 3.1b scrub starts 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.387710+0000 osd.0 (osd.0) 453 : cluster 0 3.1b scrub ok 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.643521+0000 osd.2 (osd.2) 397 : cluster 0 5.6 scrub starts 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.644702+0000 osd.2 (osd.2) 398 : cluster 0 5.6 scrub ok 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.873981+0000 osd.1 (osd.1) 253 : cluster 0 3.19 scrub starts 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: cluster 2024-03-11T21:43:04.876377+0000 osd.1 (osd.1) 254 : cluster 0 3.19 scrub ok 2024-03-11T21:43:06.006 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:05 smithi203 bash[23654]: audit 2024-03-11T21:43:04.997052+0000 mgr.y (mgr.14550) 778 : audit 0 from='client.30891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:06.023 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:06.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:06.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.384683+0000 osd.0 (osd.0) 452 : cluster 0 3.1b scrub starts 2024-03-11T21:43:06.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.387710+0000 osd.0 (osd.0) 453 : cluster 0 3.1b scrub ok 2024-03-11T21:43:06.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.643521+0000 osd.2 (osd.2) 397 : cluster 0 5.6 scrub starts 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.644702+0000 osd.2 (osd.2) 398 : cluster 0 5.6 scrub ok 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.873981+0000 osd.1 (osd.1) 253 : cluster 0 3.19 scrub starts 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:04.876377+0000 osd.1 (osd.1) 254 : cluster 0 3.19 scrub ok 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: audit 2024-03-11T21:43:04.997052+0000 mgr.y (mgr.14550) 778 : audit 0 from='client.30891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.384683+0000 osd.0 (osd.0) 452 : cluster 0 3.1b scrub starts 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.387710+0000 osd.0 (osd.0) 453 : cluster 0 3.1b scrub ok 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.643521+0000 osd.2 (osd.2) 397 : cluster 0 5.6 scrub starts 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.644702+0000 osd.2 (osd.2) 398 : cluster 0 5.6 scrub ok 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.873981+0000 osd.1 (osd.1) 253 : cluster 0 3.19 scrub starts 2024-03-11T21:43:06.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:04.876377+0000 osd.1 (osd.1) 254 : cluster 0 3.19 scrub ok 2024-03-11T21:43:06.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: audit 2024-03-11T21:43:04.997052+0000 mgr.y (mgr.14550) 778 : audit 0 from='client.30891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:06.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:06.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:06.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:06.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:06.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:06.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:06.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:06.680 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:06.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.808 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:06.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:05.599478+0000 osd.2 (osd.2) 399 : cluster 0 39.16 scrub starts 2024-03-11T21:43:06.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:05.601324+0000 osd.2 (osd.2) 400 : cluster 0 39.16 scrub ok 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:05.641058+0000 mgr.y (mgr.14550) 779 : cluster 0 pgmap v1343: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1013 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:05.856777+0000 osd.1 (osd.1) 255 : cluster 0 9.0 scrub starts 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:05.859244+0000 osd.1 (osd.1) 256 : cluster 0 9.0 scrub ok 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.571402+0000 mon.a (mon.0) 3880 : cluster 0 osdmap e864: 8 total, 8 up, 8 in 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.678435+0000 osd.4 (osd.4) 439 : cluster 0 3.14 scrub starts 2024-03-11T21:43:06.924 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:06 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.683445+0000 osd.4 (osd.4) 440 : cluster 0 3.14 scrub ok 2024-03-11T21:43:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:07.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.066 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:05.599478+0000 osd.2 (osd.2) 399 : cluster 0 39.16 scrub starts 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:05.601324+0000 osd.2 (osd.2) 400 : cluster 0 39.16 scrub ok 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:05.641058+0000 mgr.y (mgr.14550) 779 : cluster 0 pgmap v1343: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1013 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:05.856777+0000 osd.1 (osd.1) 255 : cluster 0 9.0 scrub starts 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:05.859244+0000 osd.1 (osd.1) 256 : cluster 0 9.0 scrub ok 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.571402+0000 mon.a (mon.0) 3880 : cluster 0 osdmap e864: 8 total, 8 up, 8 in 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.678435+0000 osd.4 (osd.4) 439 : cluster 0 3.14 scrub starts 2024-03-11T21:43:07.181 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.683445+0000 osd.4 (osd.4) 440 : cluster 0 3.14 scrub ok 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:05.599478+0000 osd.2 (osd.2) 399 : cluster 0 39.16 scrub starts 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:05.601324+0000 osd.2 (osd.2) 400 : cluster 0 39.16 scrub ok 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:05.641058+0000 mgr.y (mgr.14550) 779 : cluster 0 pgmap v1343: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1013 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:05.856777+0000 osd.1 (osd.1) 255 : cluster 0 9.0 scrub starts 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:05.859244+0000 osd.1 (osd.1) 256 : cluster 0 9.0 scrub ok 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.571402+0000 mon.a (mon.0) 3880 : cluster 0 osdmap e864: 8 total, 8 up, 8 in 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.678435+0000 osd.4 (osd.4) 439 : cluster 0 3.14 scrub starts 2024-03-11T21:43:07.182 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:06 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.683445+0000 osd.4 (osd.4) 440 : cluster 0 3.14 scrub ok 2024-03-11T21:43:07.196 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:07.328 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:07.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:07.461 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:07.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:07.591 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:07.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.723 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:07.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:07.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:07.979 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:08.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.601661+0000 osd.2 (osd.2) 401 : cluster 0 5.a scrub starts 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.604040+0000 osd.2 (osd.2) 402 : cluster 0 5.a scrub ok 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.771453+0000 osd.3 (osd.3) 461 : cluster 0 2.19 scrub starts 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:06.774468+0000 osd.3 (osd.3) 462 : cluster 0 2.19 scrub ok 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.460010+0000 osd.5 (osd.5) 469 : cluster 0 9.f scrub starts 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.462854+0000 osd.5 (osd.5) 470 : cluster 0 9.f scrub ok 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: audit 2024-03-11T21:43:07.478444+0000 mon.a (mon.0) 3881 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:08.094 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:07 smithi203 bash[23654]: audit 2024-03-11T21:43:07.480144+0000 mon.c (mon.2) 1334 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:08.109 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:08.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.601661+0000 osd.2 (osd.2) 401 : cluster 0 5.a scrub starts 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.604040+0000 osd.2 (osd.2) 402 : cluster 0 5.a scrub ok 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.771453+0000 osd.3 (osd.3) 461 : cluster 0 2.19 scrub starts 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:06.774468+0000 osd.3 (osd.3) 462 : cluster 0 2.19 scrub ok 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.460010+0000 osd.5 (osd.5) 469 : cluster 0 9.f scrub starts 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.462854+0000 osd.5 (osd.5) 470 : cluster 0 9.f scrub ok 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: audit 2024-03-11T21:43:07.478444+0000 mon.a (mon.0) 3881 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[29284]: audit 2024-03-11T21:43:07.480144+0000 mon.c (mon.2) 1334 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.601661+0000 osd.2 (osd.2) 401 : cluster 0 5.a scrub starts 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.604040+0000 osd.2 (osd.2) 402 : cluster 0 5.a scrub ok 2024-03-11T21:43:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.771453+0000 osd.3 (osd.3) 461 : cluster 0 2.19 scrub starts 2024-03-11T21:43:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:06.774468+0000 osd.3 (osd.3) 462 : cluster 0 2.19 scrub ok 2024-03-11T21:43:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.460010+0000 osd.5 (osd.5) 469 : cluster 0 9.f scrub starts 2024-03-11T21:43:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.462854+0000 osd.5 (osd.5) 470 : cluster 0 9.f scrub ok 2024-03-11T21:43:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: audit 2024-03-11T21:43:07.478444+0000 mon.a (mon.0) 3881 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:08.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:07 smithi143 bash[21512]: audit 2024-03-11T21:43:07.480144+0000 mon.c (mon.2) 1334 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:08.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:08.239 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:08.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.371 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:08.415 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1342,"stamp":"2024-03-11T21:43:03.638274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596710,"num_objects":244,"num_object_clones":1,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":2378,"num_read_kb":2113,"num_write":794,"num_write_kb":4016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2405,"ondisk_log_size":2405,"up":738,"acting":738,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":738,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":788660,"kb_used_data":120272,"kb_used_omap":38,"kb_used_meta":668057,"kb_avail":749172556,"statfs":{"total":767960285184,"available":767152697344,"internally_reserved":0,"allocated":123158528,"data_stored":80351951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":684090805},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.060475"},"pg_stats":[{"pgid":"331.1f","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814016+0000","last_change":"2024-03-11T21:42:53.911750+0000","last_active":"2024-03-11T21:43:01.814016+0000","last_peered":"2024-03-11T21:43:01.814016+0000","last_clean":"2024-03-11T21:43:01.814016+0000","last_became_active":"2024-03-11T21:42:53.911364+0000","last_became_peered":"2024-03-11T21:42:53.911364+0000","last_unstale":"2024-03-11T21:43:01.814016+0000","last_undegraded":"2024-03-11T21:43:01.814016+0000","last_fullsized":"2024-03-11T21:43:01.814016+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:57.911858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.1e","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813523+0000","last_change":"2024-03-11T21:42:53.907615+0000","last_active":"2024-03-11T21:43:01.813523+0000","last_peered":"2024-03-11T21:43:01.813523+0000","last_clean":"2024-03-11T21:43:01.813523+0000","last_became_active":"2024-03-11T21:42:53.906626+0000","last_became_peered":"2024-03-11T21:42:53.906626+0000","last_unstale":"2024-03-11T21:43:01.813523+0000","last_undegraded":"2024-03-11T21:43:01.813523+0000","last_fullsized":"2024-03-11T21:43:01.813523+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.450054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.1d","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428488+0000","last_change":"2024-03-11T21:42:53.895058+0000","last_active":"2024-03-11T21:43:02.428488+0000","last_peered":"2024-03-11T21:43:02.428488+0000","last_clean":"2024-03-11T21:43:02.428488+0000","last_became_active":"2024-03-11T21:42:53.894071+0000","last_became_peered":"2024-03-11T21:42:53.894071+0000","last_unstale":"2024-03-11T21:43:02.428488+0000","last_undegraded":"2024-03-11T21:43:02.428488+0000","last_fullsized":"2024-03-11T21:43:02.428488+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.790840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"331.1c","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617458+0000","last_change":"2024-03-11T21:42:53.915393+0000","last_active":"2024-03-11T21:43:01.617458+0000","last_peered":"2024-03-11T21:43:01.617458+0000","last_clean":"2024-03-11T21:43:01.617458+0000","last_became_active":"2024-03-11T21:42:53.914733+0000","last_became_peered":"2024-03-11T21:42:53.914733+0000","last_unstale":"2024-03-11T21:43:01.617458+0000","last_undegraded":"2024-03-11T21:43:01.617458+0000","last_fullsized":"2024-03-11T21:43:01.617458+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.1b","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715628+0000","last_change":"2024-03-11T21:42:53.914069+0000","last_active":"2024-03-11T21:43:01.715628+0000","last_peered":"2024-03-11T21:43:01.715628+0000","last_clean":"2024-03-11T21:43:01.715628+0000","last_became_active":"2024-03-11T21:42:53.913697+0000","last_became_peered":"2024-03-11T21:42:53.913697+0000","last_unstale":"2024-03-11T21:43:01.715628+0000","last_undegraded":"2024-03-11T21:43:01.715628+0000","last_fullsized":"2024-03-11T21:43:01.715628+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:07.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"331.1a","version":"0'0","reported_seq":32,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639122+0000","last_change":"2024-03-11T21:42:53.905076+0000","last_active":"2024-03-11T21:43:01.639122+0000","last_peered":"2024-03-11T21:43:01.639122+0000","last_clean":"2024-03-11T21:43:01.639122+0000","last_became_active":"2024-03-11T21:42:53.904735+0000","last_became_peered":"2024-03-11T21:42:53.904735+0000","last_unstale":"2024-03-11T21:43:01.639122+0000","last_undegraded":"2024-03-11T21:43:01.639122+0000","last_fullsized":"2024-03-11T21:43:01.639122+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:55.299133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"331.19","version":"863'2","reported_seq":37,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811257+0000","last_change":"2024-03-11T21:42:53.907547+0000","last_active":"2024-03-11T21:43:01.811257+0000","last_peered":"2024-03-11T21:43:01.811257+0000","last_clean":"2024-03-11T21:43:01.811257+0000","last_became_active":"2024-03-11T21:42:53.906281+0000","last_became_peered":"2024-03-11T21:42:53.906281+0000","last_unstale":"2024-03-11T21:43:01.811257+0000","last_undegraded":"2024-03-11T21:43:01.811257+0000","last_fullsized":"2024-03-11T21:43:01.811257+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.902615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.18","version":"858'1","reported_seq":30,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.580820+0000","last_change":"2024-03-11T21:42:53.926054+0000","last_active":"2024-03-11T21:42:59.580820+0000","last_peered":"2024-03-11T21:42:59.580820+0000","last_clean":"2024-03-11T21:42:59.580820+0000","last_became_active":"2024-03-11T21:42:53.925657+0000","last_became_peered":"2024-03-11T21:42:53.925657+0000","last_unstale":"2024-03-11T21:42:59.580820+0000","last_undegraded":"2024-03-11T21:42:59.580820+0000","last_fullsized":"2024-03-11T21:42:59.580820+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.031284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"331.17","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814368+0000","last_change":"2024-03-11T21:42:53.926434+0000","last_active":"2024-03-11T21:43:01.814368+0000","last_peered":"2024-03-11T21:43:01.814368+0000","last_clean":"2024-03-11T21:43:01.814368+0000","last_became_active":"2024-03-11T21:42:53.926105+0000","last_became_peered":"2024-03-11T21:42:53.926105+0000","last_unstale":"2024-03-11T21:43:01.814368+0000","last_undegraded":"2024-03-11T21:43:01.814368+0000","last_fullsized":"2024-03-11T21:43:01.814368+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.171741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.16","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593624+0000","last_change":"2024-03-11T21:42:53.917766+0000","last_active":"2024-03-11T21:43:01.593624+0000","last_peered":"2024-03-11T21:43:01.593624+0000","last_clean":"2024-03-11T21:43:01.593624+0000","last_became_active":"2024-03-11T21:42:53.917407+0000","last_became_peered":"2024-03-11T21:42:53.917407+0000","last_unstale":"2024-03-11T21:43:01.593624+0000","last_undegraded":"2024-03-11T21:43:01.593624+0000","last_fullsized":"2024-03-11T21:43:01.593624+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.676686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"331.15","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617828+0000","last_change":"2024-03-11T21:42:53.911540+0000","last_active":"2024-03-11T21:43:01.617828+0000","last_peered":"2024-03-11T21:43:01.617828+0000","last_clean":"2024-03-11T21:43:01.617828+0000","last_became_active":"2024-03-11T21:42:53.910683+0000","last_became_peered":"2024-03-11T21:42:53.910683+0000","last_unstale":"2024-03-11T21:43:01.617828+0000","last_undegraded":"2024-03-11T21:43:01.617828+0000","last_fullsized":"2024-03-11T21:43:01.617828+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.345395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.14","version":"0'0","reported_seq":32,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640118+0000","last_change":"2024-03-11T21:42:53.906251+0000","last_active":"2024-03-11T21:43:01.640118+0000","last_peered":"2024-03-11T21:43:01.640118+0000","last_clean":"2024-03-11T21:43:01.640118+0000","last_became_active":"2024-03-11T21:42:53.905919+0000","last_became_peered":"2024-03-11T21:42:53.905919+0000","last_unstale":"2024-03-11T21:43:01.640118+0000","last_undegraded":"2024-03-11T21:43:01.640118+0000","last_fullsized":"2024-03-11T21:43:01.640118+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.040978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"331.13","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812710+0000","last_change":"2024-03-11T21:42:53.905721+0000","last_active":"2024-03-11T21:43:01.812710+0000","last_peered":"2024-03-11T21:43:01.812710+0000","last_clean":"2024-03-11T21:43:01.812710+0000","last_became_active":"2024-03-11T21:42:53.904966+0000","last_became_peered":"2024-03-11T21:42:53.904966+0000","last_unstale":"2024-03-11T21:43:01.812710+0000","last_undegraded":"2024-03-11T21:43:01.812710+0000","last_fullsized":"2024-03-11T21:43:01.812710+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:58.599235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.12","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617187+0000","last_change":"2024-03-11T21:42:53.907646+0000","last_active":"2024-03-11T21:43:01.617187+0000","last_peered":"2024-03-11T21:43:01.617187+0000","last_clean":"2024-03-11T21:43:01.617187+0000","last_became_active":"2024-03-11T21:42:53.907312+0000","last_became_peered":"2024-03-11T21:42:53.907312+0000","last_unstale":"2024-03-11T21:43:01.617187+0000","last_undegraded":"2024-03-11T21:43:01.617187+0000","last_fullsized":"2024-03-11T21:43:01.617187+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:53.887196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.11","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.616978+0000","last_change":"2024-03-11T21:42:53.915879+0000","last_active":"2024-03-11T21:43:01.616978+0000","last_peered":"2024-03-11T21:43:01.616978+0000","last_clean":"2024-03-11T21:43:01.616978+0000","last_became_active":"2024-03-11T21:42:53.915039+0000","last_became_peered":"2024-03-11T21:42:53.915039+0000","last_unstale":"2024-03-11T21:43:01.616978+0000","last_undegraded":"2024-03-11T21:43:01.616978+0000","last_fullsized":"2024-03-11T21:43:01.616978+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.457725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.10","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811163+0000","last_change":"2024-03-11T21:42:53.926718+0000","last_active":"2024-03-11T21:43:01.811163+0000","last_peered":"2024-03-11T21:43:01.811163+0000","last_clean":"2024-03-11T21:43:01.811163+0000","last_became_active":"2024-03-11T21:42:53.926193+0000","last_became_peered":"2024-03-11T21:42:53.926193+0000","last_unstale":"2024-03-11T21:43:01.811163+0000","last_undegraded":"2024-03-11T21:43:01.811163+0000","last_fullsized":"2024-03-11T21:43:01.811163+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:58.701524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.f","version":"0'0","reported_seq":28,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.583128+0000","last_change":"2024-03-11T21:42:53.901754+0000","last_active":"2024-03-11T21:42:59.583128+0000","last_peered":"2024-03-11T21:42:59.583128+0000","last_clean":"2024-03-11T21:42:59.583128+0000","last_became_active":"2024-03-11T21:42:53.900930+0000","last_became_peered":"2024-03-11T21:42:53.900930+0000","last_unstale":"2024-03-11T21:42:59.583128+0000","last_undegraded":"2024-03-11T21:42:59.583128+0000","last_fullsized":"2024-03-11T21:42:59.583128+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.919832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"331.e","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.427064+0000","last_change":"2024-03-11T21:42:53.921368+0000","last_active":"2024-03-11T21:43:02.427064+0000","last_peered":"2024-03-11T21:43:02.427064+0000","last_clean":"2024-03-11T21:43:02.427064+0000","last_became_active":"2024-03-11T21:42:53.921068+0000","last_became_peered":"2024-03-11T21:42:53.921068+0000","last_unstale":"2024-03-11T21:43:02.427064+0000","last_undegraded":"2024-03-11T21:43:02.427064+0000","last_fullsized":"2024-03-11T21:43:02.427064+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:55.790656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"331.d","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617352+0000","last_change":"2024-03-11T21:42:53.929593+0000","last_active":"2024-03-11T21:43:01.617352+0000","last_peered":"2024-03-11T21:43:01.617352+0000","last_clean":"2024-03-11T21:43:01.617352+0000","last_became_active":"2024-03-11T21:42:53.929260+0000","last_became_peered":"2024-03-11T21:42:53.929260+0000","last_unstale":"2024-03-11T21:43:01.617352+0000","last_undegraded":"2024-03-11T21:43:01.617352+0000","last_fullsized":"2024-03-11T21:43:01.617352+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:21.031479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.c","version":"0'0","reported_seq":28,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582489+0000","last_change":"2024-03-11T21:42:53.923944+0000","last_active":"2024-03-11T21:42:59.582489+0000","last_peered":"2024-03-11T21:42:59.582489+0000","last_clean":"2024-03-11T21:42:59.582489+0000","last_became_active":"2024-03-11T21:42:53.923157+0000","last_became_peered":"2024-03-11T21:42:53.923157+0000","last_unstale":"2024-03-11T21:42:59.582489+0000","last_undegraded":"2024-03-11T21:42:59.582489+0000","last_fullsized":"2024-03-11T21:42:59.582489+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:21.020286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"331.b","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.426966+0000","last_change":"2024-03-11T21:42:53.901277+0000","last_active":"2024-03-11T21:43:02.426966+0000","last_peered":"2024-03-11T21:43:02.426966+0000","last_clean":"2024-03-11T21:43:02.426966+0000","last_became_active":"2024-03-11T21:42:53.896385+0000","last_became_peered":"2024-03-11T21:42:53.896385+0000","last_unstale":"2024-03-11T21:43:02.426966+0000","last_undegraded":"2024-03-11T21:43:02.426966+0000","last_fullsized":"2024-03-11T21:43:02.426966+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:00.990671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"331.a","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715451+0000","last_change":"2024-03-11T21:42:53.908468+0000","last_active":"2024-03-11T21:43:01.715451+0000","last_peered":"2024-03-11T21:43:01.715451+0000","last_clean":"2024-03-11T21:43:01.715451+0000","last_became_active":"2024-03-11T21:42:53.907772+0000","last_became_peered":"2024-03-11T21:42:53.907772+0000","last_unstale":"2024-03-11T21:43:01.715451+0000","last_undegraded":"2024-03-11T21:43:01.715451+0000","last_fullsized":"2024-03-11T21:43:01.715451+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.539786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"331.9","version":"858'2","reported_seq":37,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617103+0000","last_change":"2024-03-11T21:42:53.903444+0000","last_active":"2024-03-11T21:43:01.617103+0000","last_peered":"2024-03-11T21:43:01.617103+0000","last_clean":"2024-03-11T21:43:01.617103+0000","last_became_active":"2024-03-11T21:42:53.903116+0000","last_became_peered":"2024-03-11T21:42:53.903116+0000","last_unstale":"2024-03-11T21:43:01.617103+0000","last_undegraded":"2024-03-11T21:43:01.617103+0000","last_fullsized":"2024-03-11T21:43:01.617103+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:03.323390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.8","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.616481+0000","last_change":"2024-03-11T21:42:53.927901+0000","last_active":"2024-03-11T21:43:01.616481+0000","last_peered":"2024-03-11T21:43:01.616481+0000","last_clean":"2024-03-11T21:43:01.616481+0000","last_became_active":"2024-03-11T21:42:53.927535+0000","last_became_peered":"2024-03-11T21:42:53.927535+0000","last_unstale":"2024-03-11T21:43:01.616481+0000","last_undegraded":"2024-03-11T21:43:01.616481+0000","last_fullsized":"2024-03-11T21:43:01.616481+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.262952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.7","version":"858'2","reported_seq":37,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814147+0000","last_change":"2024-03-11T21:42:53.909941+0000","last_active":"2024-03-11T21:43:01.814147+0000","last_peered":"2024-03-11T21:43:01.814147+0000","last_clean":"2024-03-11T21:43:01.814147+0000","last_became_active":"2024-03-11T21:42:53.909493+0000","last_became_peered":"2024-03-11T21:42:53.909493+0000","last_unstale":"2024-03-11T21:43:01.814147+0000","last_undegraded":"2024-03-11T21:43:01.814147+0000","last_fullsized":"2024-03-11T21:43:01.814147+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:02.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.6","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618343+0000","last_change":"2024-03-11T21:42:53.926875+0000","last_active":"2024-03-11T21:43:01.618343+0000","last_peered":"2024-03-11T21:43:01.618343+0000","last_clean":"2024-03-11T21:43:01.618343+0000","last_became_active":"2024-03-11T21:42:53.926435+0000","last_became_peered":"2024-03-11T21:42:53.926435+0000","last_unstale":"2024-03-11T21:43:01.618343+0000","last_undegraded":"2024-03-11T21:43:01.618343+0000","last_fullsized":"2024-03-11T21:43:01.618343+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:06.595140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"331.5","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.427622+0000","last_change":"2024-03-11T21:42:53.900428+0000","last_active":"2024-03-11T21:43:02.427622+0000","last_peered":"2024-03-11T21:43:02.427622+0000","last_clean":"2024-03-11T21:43:02.427622+0000","last_became_active":"2024-03-11T21:42:53.898519+0000","last_became_peered":"2024-03-11T21:42:53.898519+0000","last_unstale":"2024-03-11T21:43:02.427622+0000","last_undegraded":"2024-03-11T21:43:02.427622+0000","last_fullsized":"2024-03-11T21:43:02.427622+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.471594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"331.4","version":"0'0","reported_seq":32,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639821+0000","last_change":"2024-03-11T21:42:53.904065+0000","last_active":"2024-03-11T21:43:01.639821+0000","last_peered":"2024-03-11T21:43:01.639821+0000","last_clean":"2024-03-11T21:43:01.639821+0000","last_became_active":"2024-03-11T21:42:53.899637+0000","last_became_peered":"2024-03-11T21:42:53.899637+0000","last_unstale":"2024-03-11T21:43:01.639821+0000","last_undegraded":"2024-03-11T21:43:01.639821+0000","last_fullsized":"2024-03-11T21:43:01.639821+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.639152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"331.3","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813144+0000","last_change":"2024-03-11T21:42:53.908605+0000","last_active":"2024-03-11T21:43:01.813144+0000","last_peered":"2024-03-11T21:43:01.813144+0000","last_clean":"2024-03-11T21:43:01.813144+0000","last_became_active":"2024-03-11T21:42:53.905987+0000","last_became_peered":"2024-03-11T21:42:53.905987+0000","last_unstale":"2024-03-11T21:43:01.813144+0000","last_undegraded":"2024-03-11T21:43:01.813144+0000","last_fullsized":"2024-03-11T21:43:01.813144+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.547145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"331.2","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.590712+0000","last_change":"2024-03-11T21:42:53.906650+0000","last_active":"2024-03-11T21:43:01.590712+0000","last_peered":"2024-03-11T21:43:01.590712+0000","last_clean":"2024-03-11T21:43:01.590712+0000","last_became_active":"2024-03-11T21:42:53.906285+0000","last_became_peered":"2024-03-11T21:42:53.906285+0000","last_unstale":"2024-03-11T21:43:01.590712+0000","last_undegraded":"2024-03-11T21:43:01.590712+0000","last_fullsized":"2024-03-11T21:43:01.590712+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:03.220898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"331.1","version":"0'0","reported_seq":34,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.590733+0000","last_change":"2024-03-11T21:42:53.898216+0000","last_active":"2024-03-11T21:43:01.590733+0000","last_peered":"2024-03-11T21:43:01.590733+0000","last_clean":"2024-03-11T21:43:01.590733+0000","last_became_active":"2024-03-11T21:42:53.897344+0000","last_became_peered":"2024-03-11T21:42:53.897344+0000","last_unstale":"2024-03-11T21:43:01.590733+0000","last_undegraded":"2024-03-11T21:43:01.590733+0000","last_fullsized":"2024-03-11T21:43:01.590733+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:00.234115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"331.0","version":"0'0","reported_seq":32,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.638925+0000","last_change":"2024-03-11T21:42:53.902943+0000","last_active":"2024-03-11T21:43:01.638925+0000","last_peered":"2024-03-11T21:43:01.638925+0000","last_clean":"2024-03-11T21:43:01.638925+0000","last_became_active":"2024-03-11T21:42:53.902573+0000","last_became_peered":"2024-03-11T21:42:53.902573+0000","last_unstale":"2024-03-11T21:43:01.638925+0000","last_undegraded":"2024-03-11T21:43:01.638925+0000","last_fullsized":"2024-03-11T21:43:01.638925+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":854,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:52.841217+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.841217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.436434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.7","version":"855'8","reported_seq":474,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640270+0000","last_change":"2024-03-11T21:42:52.251220+0000","last_active":"2024-03-11T21:43:01.640270+0000","last_peered":"2024-03-11T21:43:01.640270+0000","last_clean":"2024-03-11T21:43:01.640270+0000","last_became_active":"2024-03-11T21:39:36.667532+0000","last_became_peered":"2024-03-11T21:39:36.667532+0000","last_unstale":"2024-03-11T21:43:01.640270+0000","last_undegraded":"2024-03-11T21:43:01.640270+0000","last_fullsized":"2024-03-11T21:43:01.640270+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'7","last_scrub_stamp":"2024-03-11T21:42:52.251098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.251098+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.164217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039920429999999998,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.6","version":"833'7","reported_seq":461,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623661+0000","last_change":"2024-03-11T21:41:54.496117+0000","last_active":"2024-03-11T21:43:01.623661+0000","last_peered":"2024-03-11T21:43:01.623661+0000","last_clean":"2024-03-11T21:43:01.623661+0000","last_became_active":"2024-03-11T21:39:36.664544+0000","last_became_peered":"2024-03-11T21:39:36.664544+0000","last_unstale":"2024-03-11T21:43:01.623661+0000","last_undegraded":"2024-03-11T21:43:01.623661+0000","last_fullsized":"2024-03-11T21:43:01.623661+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'5","last_scrub_stamp":"2024-03-11T21:41:54.495843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.495843+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:18.482664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012127220000000001,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.5","version":"833'5","reported_seq":471,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617376+0000","last_change":"2024-03-11T21:42:53.502167+0000","last_active":"2024-03-11T21:43:01.617376+0000","last_peered":"2024-03-11T21:43:01.617376+0000","last_clean":"2024-03-11T21:43:01.617376+0000","last_became_active":"2024-03-11T21:39:36.664433+0000","last_became_peered":"2024-03-11T21:39:36.664433+0000","last_unstale":"2024-03-11T21:43:01.617376+0000","last_undegraded":"2024-03-11T21:43:01.617376+0000","last_fullsized":"2024-03-11T21:43:01.617376+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'5","last_scrub_stamp":"2024-03-11T21:42:53.501825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.501825+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:03.491732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018642859999999999,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.4","version":"847'20","reported_seq":479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812628+0000","last_change":"2024-03-11T21:42:51.856040+0000","last_active":"2024-03-11T21:43:01.812628+0000","last_peered":"2024-03-11T21:43:01.812628+0000","last_clean":"2024-03-11T21:43:01.812628+0000","last_became_active":"2024-03-11T21:39:36.675026+0000","last_became_peered":"2024-03-11T21:39:36.675026+0000","last_unstale":"2024-03-11T21:43:01.812628+0000","last_undegraded":"2024-03-11T21:43:01.812628+0000","last_fullsized":"2024-03-11T21:43:01.812628+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'20","last_scrub_stamp":"2024-03-11T21:42:51.855842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:42:51.855842+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:57.374933+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0031897900000000001,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.3","version":"833'13","reported_seq":470,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639256+0000","last_change":"2024-03-11T21:41:30.606913+0000","last_active":"2024-03-11T21:43:01.639256+0000","last_peered":"2024-03-11T21:43:01.639256+0000","last_clean":"2024-03-11T21:43:01.639256+0000","last_became_active":"2024-03-11T21:39:36.653164+0000","last_became_peered":"2024-03-11T21:39:36.653164+0000","last_unstale":"2024-03-11T21:43:01.639256+0000","last_undegraded":"2024-03-11T21:43:01.639256+0000","last_fullsized":"2024-03-11T21:43:01.639256+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7","last_scrub_stamp":"2024-03-11T21:41:17.885818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.885818+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0050295210000000003,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.2","version":"833'9","reported_seq":466,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715278+0000","last_change":"2024-03-11T21:41:30.609679+0000","last_active":"2024-03-11T21:43:01.715278+0000","last_peered":"2024-03-11T21:43:01.715278+0000","last_clean":"2024-03-11T21:43:01.715278+0000","last_became_active":"2024-03-11T21:39:36.673314+0000","last_became_peered":"2024-03-11T21:39:36.673314+0000","last_unstale":"2024-03-11T21:43:01.715278+0000","last_undegraded":"2024-03-11T21:43:01.715278+0000","last_fullsized":"2024-03-11T21:43:01.715278+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:17.477919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.477919+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015195479999999999,"stat_sum":{"num_bytes":4000033,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.1","version":"855'24","reported_seq":483,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428234+0000","last_change":"2024-03-11T21:41:59.370755+0000","last_active":"2024-03-11T21:43:02.428234+0000","last_peered":"2024-03-11T21:43:02.428234+0000","last_clean":"2024-03-11T21:43:02.428234+0000","last_became_active":"2024-03-11T21:39:36.651751+0000","last_became_peered":"2024-03-11T21:39:36.651751+0000","last_unstale":"2024-03-11T21:43:02.428234+0000","last_undegraded":"2024-03-11T21:43:02.428234+0000","last_fullsized":"2024-03-11T21:43:02.428234+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'22","last_scrub_stamp":"2024-03-11T21:41:59.370508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.370508+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.982762+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.006595956,"stat_sum":{"num_bytes":31,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"313.0","version":"855'9","reported_seq":468,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622410+0000","last_change":"2024-03-11T21:41:30.603475+0000","last_active":"2024-03-11T21:43:01.622410+0000","last_peered":"2024-03-11T21:43:01.622410+0000","last_clean":"2024-03-11T21:43:01.622410+0000","last_became_active":"2024-03-11T21:39:36.661261+0000","last_became_peered":"2024-03-11T21:39:36.661261+0000","last_unstale":"2024-03-11T21:43:01.622410+0000","last_undegraded":"2024-03-11T21:43:01.622410+0000","last_fullsized":"2024-03-11T21:43:01.622410+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":688,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'6","last_scrub_stamp":"2024-03-11T21:41:16.318277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:39:35.496133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.318277+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019677119999999999,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811489+0000","last_change":"2024-03-11T21:42:59.786704+0000","last_active":"2024-03-11T21:43:01.811489+0000","last_peered":"2024-03-11T21:43:01.811489+0000","last_clean":"2024-03-11T21:43:01.811489+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:43:01.811489+0000","last_undegraded":"2024-03-11T21:43:01.811489+0000","last_fullsized":"2024-03-11T21:43:01.811489+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.786379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.786379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.633663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1491,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.638793+0000","last_change":"2024-03-11T21:41:29.853017+0000","last_active":"2024-03-11T21:43:01.638793+0000","last_peered":"2024-03-11T21:43:01.638793+0000","last_clean":"2024-03-11T21:43:01.638793+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:43:01.638793+0000","last_undegraded":"2024-03-11T21:43:01.638793+0000","last_fullsized":"2024-03-11T21:43:01.638793+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.852782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.852782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1476,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429086+0000","last_change":"2024-03-11T21:41:57.330251+0000","last_active":"2024-03-11T21:43:02.429086+0000","last_peered":"2024-03-11T21:43:02.429086+0000","last_clean":"2024-03-11T21:43:02.429086+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:43:02.429086+0000","last_undegraded":"2024-03-11T21:43:02.429086+0000","last_fullsized":"2024-03-11T21:43:02.429086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.330037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.330037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.606411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":1462,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813160+0000","last_change":"2024-03-11T21:40:25.470377+0000","last_active":"2024-03-11T21:43:01.813160+0000","last_peered":"2024-03-11T21:43:01.813160+0000","last_clean":"2024-03-11T21:43:01.813160+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:43:01.813160+0000","last_undegraded":"2024-03-11T21:43:01.813160+0000","last_fullsized":"2024-03-11T21:43:01.813160+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.470230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.470230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":1388,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.580940+0000","last_change":"2024-03-11T21:41:56.250851+0000","last_active":"2024-03-11T21:42:59.580940+0000","last_peered":"2024-03-11T21:42:59.580940+0000","last_clean":"2024-03-11T21:42:59.580940+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:42:59.580940+0000","last_undegraded":"2024-03-11T21:42:59.580940+0000","last_fullsized":"2024-03-11T21:42:59.580940+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.250744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.250744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:01.111142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1499,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640164+0000","last_change":"2024-03-11T21:42:58.341631+0000","last_active":"2024-03-11T21:43:01.640164+0000","last_peered":"2024-03-11T21:43:01.640164+0000","last_clean":"2024-03-11T21:43:01.640164+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:43:01.640164+0000","last_undegraded":"2024-03-11T21:43:01.640164+0000","last_fullsized":"2024-03-11T21:43:01.640164+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.341432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.341432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.778503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1493,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640178+0000","last_change":"2024-03-11T21:41:42.681742+0000","last_active":"2024-03-11T21:43:01.640178+0000","last_peered":"2024-03-11T21:43:01.640178+0000","last_clean":"2024-03-11T21:43:01.640178+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:43:01.640178+0000","last_undegraded":"2024-03-11T21:43:01.640178+0000","last_fullsized":"2024-03-11T21:43:01.640178+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.681620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.645321+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.681620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1505,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618246+0000","last_change":"2024-03-11T21:41:35.844756+0000","last_active":"2024-03-11T21:43:01.618246+0000","last_peered":"2024-03-11T21:43:01.618246+0000","last_clean":"2024-03-11T21:43:01.618246+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:43:01.618246+0000","last_undegraded":"2024-03-11T21:43:01.618246+0000","last_fullsized":"2024-03-11T21:43:01.618246+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.844652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.844652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:02.557477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1493,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.591086+0000","last_change":"2024-03-11T21:41:31.690755+0000","last_active":"2024-03-11T21:43:01.591086+0000","last_peered":"2024-03-11T21:43:01.591086+0000","last_clean":"2024-03-11T21:43:01.591086+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:43:01.591086+0000","last_undegraded":"2024-03-11T21:43:01.591086+0000","last_fullsized":"2024-03-11T21:43:01.591086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.690559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.690559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1471,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811321+0000","last_change":"2024-03-11T21:41:41.521503+0000","last_active":"2024-03-11T21:43:01.811321+0000","last_peered":"2024-03-11T21:43:01.811321+0000","last_clean":"2024-03-11T21:43:01.811321+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:43:01.811321+0000","last_undegraded":"2024-03-11T21:43:01.811321+0000","last_fullsized":"2024-03-11T21:43:01.811321+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.521300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.521300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1483,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.436930+0000","last_change":"2024-03-11T21:43:02.436956+0000","last_active":"2024-03-11T21:43:02.436930+0000","last_peered":"2024-03-11T21:43:02.436930+0000","last_clean":"2024-03-11T21:43:02.436930+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:43:02.436930+0000","last_undegraded":"2024-03-11T21:43:02.436930+0000","last_fullsized":"2024-03-11T21:43:02.436930+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.436843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.436843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.695057+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":1465,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716992+0000","last_change":"2024-03-11T21:41:59.422739+0000","last_active":"2024-03-11T21:43:01.716992+0000","last_peered":"2024-03-11T21:43:01.716992+0000","last_clean":"2024-03-11T21:43:01.716992+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:43:01.716992+0000","last_undegraded":"2024-03-11T21:43:01.716992+0000","last_fullsized":"2024-03-11T21:43:01.716992+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.422534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.422534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.121392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1475,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622904+0000","last_change":"2024-03-11T21:40:27.533904+0000","last_active":"2024-03-11T21:43:01.622904+0000","last_peered":"2024-03-11T21:43:01.622904+0000","last_clean":"2024-03-11T21:43:01.622904+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:43:01.622904+0000","last_undegraded":"2024-03-11T21:43:01.622904+0000","last_fullsized":"2024-03-11T21:43:01.622904+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.533778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.533778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1492,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.594086+0000","last_change":"2024-03-11T21:41:45.595342+0000","last_active":"2024-03-11T21:43:01.594086+0000","last_peered":"2024-03-11T21:43:01.594086+0000","last_clean":"2024-03-11T21:43:01.594086+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:43:01.594086+0000","last_undegraded":"2024-03-11T21:43:01.594086+0000","last_fullsized":"2024-03-11T21:43:01.594086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.595129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.595129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1504,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618663+0000","last_change":"2024-03-11T21:41:39.829836+0000","last_active":"2024-03-11T21:43:01.618663+0000","last_peered":"2024-03-11T21:43:01.618663+0000","last_clean":"2024-03-11T21:43:01.618663+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:43:01.618663+0000","last_undegraded":"2024-03-11T21:43:01.618663+0000","last_fullsized":"2024-03-11T21:43:01.618663+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:39.829557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:39.829557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":884,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582877+0000","last_change":"2024-03-11T21:41:20.464811+0000","last_active":"2024-03-11T21:42:59.582877+0000","last_peered":"2024-03-11T21:42:59.582877+0000","last_clean":"2024-03-11T21:42:59.582877+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:42:59.582877+0000","last_undegraded":"2024-03-11T21:42:59.582877+0000","last_fullsized":"2024-03-11T21:42:59.582877+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.464589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.464589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1500,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.319180+0000","last_change":"2024-03-11T21:43:02.319235+0000","last_active":"2024-03-11T21:43:02.319180+0000","last_peered":"2024-03-11T21:43:02.319180+0000","last_clean":"2024-03-11T21:43:02.319180+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:43:02.319180+0000","last_undegraded":"2024-03-11T21:43:02.319180+0000","last_fullsized":"2024-03-11T21:43:02.319180+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.319030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.319030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.439681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":1471,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715252+0000","last_change":"2024-03-11T21:41:51.387114+0000","last_active":"2024-03-11T21:43:01.715252+0000","last_peered":"2024-03-11T21:43:01.715252+0000","last_clean":"2024-03-11T21:43:01.715252+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:43:01.715252+0000","last_undegraded":"2024-03-11T21:43:01.715252+0000","last_fullsized":"2024-03-11T21:43:01.715252+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.386905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.386905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.204956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":1463,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811173+0000","last_change":"2024-03-11T21:40:19.428421+0000","last_active":"2024-03-11T21:43:01.811173+0000","last_peered":"2024-03-11T21:43:01.811173+0000","last_clean":"2024-03-11T21:43:01.811173+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:43:01.811173+0000","last_undegraded":"2024-03-11T21:43:01.811173+0000","last_fullsized":"2024-03-11T21:43:01.811173+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.428205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.428205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1493,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.592004+0000","last_change":"2024-03-11T21:41:40.664787+0000","last_active":"2024-03-11T21:43:01.592004+0000","last_peered":"2024-03-11T21:43:01.592004+0000","last_clean":"2024-03-11T21:43:01.592004+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:43:01.592004+0000","last_undegraded":"2024-03-11T21:43:01.592004+0000","last_fullsized":"2024-03-11T21:43:01.592004+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:40.664675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.664675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":1357,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581685+0000","last_change":"2024-03-11T21:41:32.440218+0000","last_active":"2024-03-11T21:42:59.581685+0000","last_peered":"2024-03-11T21:42:59.581685+0000","last_clean":"2024-03-11T21:42:59.581685+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:42:59.581685+0000","last_undegraded":"2024-03-11T21:42:59.581685+0000","last_fullsized":"2024-03-11T21:42:59.581685+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.440004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.440004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":1463,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813189+0000","last_change":"2024-03-11T21:38:45.918826+0000","last_active":"2024-03-11T21:43:01.813189+0000","last_peered":"2024-03-11T21:43:01.813189+0000","last_clean":"2024-03-11T21:43:01.813189+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:43:01.813189+0000","last_undegraded":"2024-03-11T21:43:01.813189+0000","last_fullsized":"2024-03-11T21:43:01.813189+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:45.918744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:38:45.918744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813832+0000","last_change":"2024-03-11T21:42:55.708191+0000","last_active":"2024-03-11T21:43:01.813832+0000","last_peered":"2024-03-11T21:43:01.813832+0000","last_clean":"2024-03-11T21:43:01.813832+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:43:01.813832+0000","last_undegraded":"2024-03-11T21:43:01.813832+0000","last_fullsized":"2024-03-11T21:43:01.813832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.707930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.707930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.053621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1483,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623833+0000","last_change":"2024-03-11T21:43:01.415713+0000","last_active":"2024-03-11T21:43:01.623833+0000","last_peered":"2024-03-11T21:43:01.623833+0000","last_clean":"2024-03-11T21:43:01.623833+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:43:01.623833+0000","last_undegraded":"2024-03-11T21:43:01.623833+0000","last_fullsized":"2024-03-11T21:43:01.623833+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.415616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.415616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.957916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1478,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428197+0000","last_change":"2024-03-11T21:42:55.545333+0000","last_active":"2024-03-11T21:43:02.428197+0000","last_peered":"2024-03-11T21:43:02.428197+0000","last_clean":"2024-03-11T21:43:02.428197+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:43:02.428197+0000","last_undegraded":"2024-03-11T21:43:02.428197+0000","last_fullsized":"2024-03-11T21:43:02.428197+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.640588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":1457,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715263+0000","last_change":"2024-03-11T21:39:52.904395+0000","last_active":"2024-03-11T21:43:01.715263+0000","last_peered":"2024-03-11T21:43:01.715263+0000","last_clean":"2024-03-11T21:43:01.715263+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:43:01.715263+0000","last_undegraded":"2024-03-11T21:43:01.715263+0000","last_fullsized":"2024-03-11T21:43:01.715263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:52.904287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:39:52.904287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":885,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582320+0000","last_change":"2024-03-11T21:40:21.384992+0000","last_active":"2024-03-11T21:42:59.582320+0000","last_peered":"2024-03-11T21:42:59.582320+0000","last_clean":"2024-03-11T21:42:59.582320+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:42:59.582320+0000","last_undegraded":"2024-03-11T21:42:59.582320+0000","last_fullsized":"2024-03-11T21:42:59.582320+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.384763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1493,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.592222+0000","last_change":"2024-03-11T21:41:51.650830+0000","last_active":"2024-03-11T21:43:01.592222+0000","last_peered":"2024-03-11T21:43:01.592222+0000","last_clean":"2024-03-11T21:43:01.592222+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:43:01.592222+0000","last_undegraded":"2024-03-11T21:43:01.592222+0000","last_fullsized":"2024-03-11T21:43:01.592222+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.650722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.650722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.155813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1475,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623328+0000","last_change":"2024-03-11T21:41:17.359280+0000","last_active":"2024-03-11T21:43:01.623328+0000","last_peered":"2024-03-11T21:43:01.623328+0000","last_clean":"2024-03-11T21:43:01.623328+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:43:01.623328+0000","last_undegraded":"2024-03-11T21:43:01.623328+0000","last_fullsized":"2024-03-11T21:43:01.623328+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":1465,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716160+0000","last_change":"2024-03-11T21:42:56.351908+0000","last_active":"2024-03-11T21:43:01.716160+0000","last_peered":"2024-03-11T21:43:01.716160+0000","last_clean":"2024-03-11T21:43:01.716160+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:43:01.716160+0000","last_undegraded":"2024-03-11T21:43:01.716160+0000","last_fullsized":"2024-03-11T21:43:01.716160+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.351542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.351542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:06.855022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1505,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618682+0000","last_change":"2024-03-11T21:41:54.862852+0000","last_active":"2024-03-11T21:43:01.618682+0000","last_peered":"2024-03-11T21:43:01.618682+0000","last_clean":"2024-03-11T21:43:01.618682+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:43:01.618682+0000","last_undegraded":"2024-03-11T21:43:01.618682+0000","last_fullsized":"2024-03-11T21:43:01.618682+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.862666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.862666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.186701+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":1357,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.583054+0000","last_change":"2024-03-11T21:41:30.411743+0000","last_active":"2024-03-11T21:42:59.583054+0000","last_peered":"2024-03-11T21:42:59.583054+0000","last_clean":"2024-03-11T21:42:59.583054+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:42:59.583054+0000","last_undegraded":"2024-03-11T21:42:59.583054+0000","last_fullsized":"2024-03-11T21:42:59.583054+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:30.411631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.411631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1489,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623621+0000","last_change":"2024-03-11T21:42:52.535387+0000","last_active":"2024-03-11T21:43:01.623621+0000","last_peered":"2024-03-11T21:43:01.623621+0000","last_clean":"2024-03-11T21:43:01.623621+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:43:01.623621+0000","last_undegraded":"2024-03-11T21:43:01.623621+0000","last_fullsized":"2024-03-11T21:43:01.623621+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.535087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.535087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.542418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1867,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715815+0000","last_change":"2024-03-11T21:41:21.484873+0000","last_active":"2024-03-11T21:43:01.715815+0000","last_peered":"2024-03-11T21:43:01.715815+0000","last_clean":"2024-03-11T21:43:01.715815+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:43:01.715815+0000","last_undegraded":"2024-03-11T21:43:01.715815+0000","last_fullsized":"2024-03-11T21:43:01.715815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:21.484757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.484757+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"389'27","reported_seq":1663,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.717134+0000","last_change":"2024-03-11T21:41:26.485912+0000","last_active":"2024-03-11T21:43:01.717134+0000","last_peered":"2024-03-11T21:43:01.717134+0000","last_clean":"2024-03-11T21:43:01.717134+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:43:01.717134+0000","last_undegraded":"2024-03-11T21:43:01.717134+0000","last_fullsized":"2024-03-11T21:43:01.717134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:41:26.485646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.485646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1672,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618849+0000","last_change":"2024-03-11T21:41:42.811010+0000","last_active":"2024-03-11T21:43:01.618849+0000","last_peered":"2024-03-11T21:43:01.618849+0000","last_clean":"2024-03-11T21:43:01.618849+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:43:01.618849+0000","last_undegraded":"2024-03-11T21:43:01.618849+0000","last_fullsized":"2024-03-11T21:43:01.618849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.810777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.810777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.272393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1626,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429687+0000","last_change":"2024-03-11T21:41:45.252015+0000","last_active":"2024-03-11T21:43:02.429687+0000","last_peered":"2024-03-11T21:43:02.429687+0000","last_clean":"2024-03-11T21:43:02.429687+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:43:02.429687+0000","last_undegraded":"2024-03-11T21:43:02.429687+0000","last_fullsized":"2024-03-11T21:43:02.429687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.251730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.251730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1484,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432247+0000","last_change":"2024-03-11T21:40:14.758296+0000","last_active":"2024-03-11T21:43:02.432247+0000","last_peered":"2024-03-11T21:43:02.432247+0000","last_clean":"2024-03-11T21:43:02.432247+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:43:02.432247+0000","last_undegraded":"2024-03-11T21:43:02.432247+0000","last_fullsized":"2024-03-11T21:43:02.432247+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:14.758086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.758086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"389'20","reported_seq":1663,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430626+0000","last_change":"2024-03-11T21:41:26.303821+0000","last_active":"2024-03-11T21:43:02.430626+0000","last_peered":"2024-03-11T21:43:02.430626+0000","last_clean":"2024-03-11T21:43:02.430626+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:43:02.430626+0000","last_undegraded":"2024-03-11T21:43:02.430626+0000","last_fullsized":"2024-03-11T21:43:02.430626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:26.303567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.303567+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1539,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581966+0000","last_change":"2024-03-11T21:40:24.347139+0000","last_active":"2024-03-11T21:42:59.581966+0000","last_peered":"2024-03-11T21:42:59.581966+0000","last_clean":"2024-03-11T21:42:59.581966+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:42:59.581966+0000","last_undegraded":"2024-03-11T21:42:59.581966+0000","last_fullsized":"2024-03-11T21:42:59.581966+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.347032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.347032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1665,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617071+0000","last_change":"2024-03-11T21:41:27.719707+0000","last_active":"2024-03-11T21:43:01.617071+0000","last_peered":"2024-03-11T21:43:01.617071+0000","last_clean":"2024-03-11T21:43:01.617071+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:43:01.617071+0000","last_undegraded":"2024-03-11T21:43:01.617071+0000","last_fullsized":"2024-03-11T21:43:01.617071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.719614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.719614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1632,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622895+0000","last_change":"2024-03-11T21:41:52.458683+0000","last_active":"2024-03-11T21:43:01.622895+0000","last_peered":"2024-03-11T21:43:01.622895+0000","last_clean":"2024-03-11T21:43:01.622895+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:43:01.622895+0000","last_undegraded":"2024-03-11T21:43:01.622895+0000","last_fullsized":"2024-03-11T21:43:01.622895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:52.458535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:41:52.458535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.765250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1483,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432309+0000","last_change":"2024-03-11T21:41:55.275905+0000","last_active":"2024-03-11T21:43:02.432309+0000","last_peered":"2024-03-11T21:43:02.432309+0000","last_clean":"2024-03-11T21:43:02.432309+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:43:02.432309+0000","last_undegraded":"2024-03-11T21:43:02.432309+0000","last_fullsized":"2024-03-11T21:43:02.432309+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.275680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.275680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:24.360123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"389'39","reported_seq":1685,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716630+0000","last_change":"2024-03-11T21:42:53.391146+0000","last_active":"2024-03-11T21:43:01.716630+0000","last_peered":"2024-03-11T21:43:01.716630+0000","last_clean":"2024-03-11T21:43:01.716630+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:43:01.716630+0000","last_undegraded":"2024-03-11T21:43:01.716630+0000","last_fullsized":"2024-03-11T21:43:01.716630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:42:53.390867+0000","last_deep_scrub":"389'39","last_deep_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.390867+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.412714+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1672,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617660+0000","last_change":"2024-03-11T21:41:37.871821+0000","last_active":"2024-03-11T21:43:01.617660+0000","last_peered":"2024-03-11T21:43:01.617660+0000","last_clean":"2024-03-11T21:43:01.617660+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:43:01.617660+0000","last_undegraded":"2024-03-11T21:43:01.617660+0000","last_fullsized":"2024-03-11T21:43:01.617660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.871740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.871740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1664,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617478+0000","last_change":"2024-03-11T21:41:51.789113+0000","last_active":"2024-03-11T21:43:01.617478+0000","last_peered":"2024-03-11T21:43:01.617478+0000","last_clean":"2024-03-11T21:43:01.617478+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:43:01.617478+0000","last_undegraded":"2024-03-11T21:43:01.617478+0000","last_fullsized":"2024-03-11T21:43:01.617478+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.789014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.789014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:15.998815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1613,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715124+0000","last_change":"2024-03-11T21:39:59.804040+0000","last_active":"2024-03-11T21:43:01.715124+0000","last_peered":"2024-03-11T21:43:01.715124+0000","last_clean":"2024-03-11T21:43:01.715124+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:43:01.715124+0000","last_undegraded":"2024-03-11T21:43:01.715124+0000","last_fullsized":"2024-03-11T21:43:01.715124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:59.803820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.803820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":870,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.427413+0000","last_change":"2024-03-11T21:42:53.579836+0000","last_active":"2024-03-11T21:43:02.427413+0000","last_peered":"2024-03-11T21:43:02.427413+0000","last_clean":"2024-03-11T21:43:02.427413+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:43:02.427413+0000","last_undegraded":"2024-03-11T21:43:02.427413+0000","last_fullsized":"2024-03-11T21:43:02.427413+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:42:53.579671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.579671+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.307026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1358,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581674+0000","last_change":"2024-03-11T21:40:05.414458+0000","last_active":"2024-03-11T21:42:59.581674+0000","last_peered":"2024-03-11T21:42:59.581674+0000","last_clean":"2024-03-11T21:42:59.581674+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:42:59.581674+0000","last_undegraded":"2024-03-11T21:42:59.581674+0000","last_fullsized":"2024-03-11T21:42:59.581674+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:05.414369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.414369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"389'44","reported_seq":1729,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639317+0000","last_change":"2024-03-11T21:41:32.795704+0000","last_active":"2024-03-11T21:43:01.639317+0000","last_peered":"2024-03-11T21:43:01.639317+0000","last_clean":"2024-03-11T21:43:01.639317+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:43:01.639317+0000","last_undegraded":"2024-03-11T21:43:01.639317+0000","last_fullsized":"2024-03-11T21:43:01.639317+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'44","last_scrub_stamp":"2024-03-11T21:41:32.795430+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.795430+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1632,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716422+0000","last_change":"2024-03-11T21:41:29.509226+0000","last_active":"2024-03-11T21:43:01.716422+0000","last_peered":"2024-03-11T21:43:01.716422+0000","last_clean":"2024-03-11T21:43:01.716422+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:43:01.716422+0000","last_undegraded":"2024-03-11T21:43:01.716422+0000","last_fullsized":"2024-03-11T21:43:01.716422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.509105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.509105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1545,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581574+0000","last_change":"2024-03-11T21:42:00.292135+0000","last_active":"2024-03-11T21:42:59.581574+0000","last_peered":"2024-03-11T21:42:59.581574+0000","last_clean":"2024-03-11T21:42:59.581574+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:42:59.581574+0000","last_undegraded":"2024-03-11T21:42:59.581574+0000","last_fullsized":"2024-03-11T21:42:59.581574+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.292001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.532304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1668,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618482+0000","last_change":"2024-03-11T21:42:55.510765+0000","last_active":"2024-03-11T21:43:01.618482+0000","last_peered":"2024-03-11T21:43:01.618482+0000","last_clean":"2024-03-11T21:43:01.618482+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:43:01.618482+0000","last_undegraded":"2024-03-11T21:43:01.618482+0000","last_fullsized":"2024-03-11T21:43:01.618482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.510373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.510373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.380386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1914,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715974+0000","last_change":"2024-03-11T21:41:30.525150+0000","last_active":"2024-03-11T21:43:01.715974+0000","last_peered":"2024-03-11T21:43:01.715974+0000","last_clean":"2024-03-11T21:43:01.715974+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:43:01.715974+0000","last_undegraded":"2024-03-11T21:43:01.715974+0000","last_fullsized":"2024-03-11T21:43:01.715974+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:30.525009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.525009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1486,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811742+0000","last_change":"2024-03-11T21:40:08.203021+0000","last_active":"2024-03-11T21:43:01.811742+0000","last_peered":"2024-03-11T21:43:01.811742+0000","last_clean":"2024-03-11T21:43:01.811742+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:43:01.811742+0000","last_undegraded":"2024-03-11T21:43:01.811742+0000","last_fullsized":"2024-03-11T21:43:01.811742+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:40:08.202874+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:40:08.202874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"389'15","reported_seq":1662,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814023+0000","last_change":"2024-03-11T21:43:01.814079+0000","last_active":"2024-03-11T21:43:01.814023+0000","last_peered":"2024-03-11T21:43:01.814023+0000","last_clean":"2024-03-11T21:43:01.814023+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:43:01.814023+0000","last_undegraded":"2024-03-11T21:43:01.814023+0000","last_fullsized":"2024-03-11T21:43:01.814023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:43:01.813932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.813932+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.924935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1505,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581858+0000","last_change":"2024-03-11T21:40:09.399602+0000","last_active":"2024-03-11T21:42:59.581858+0000","last_peered":"2024-03-11T21:42:59.581858+0000","last_clean":"2024-03-11T21:42:59.581858+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:42:59.581858+0000","last_undegraded":"2024-03-11T21:42:59.581858+0000","last_fullsized":"2024-03-11T21:42:59.581858+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.399381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.399381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1923,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.618133+0000","last_change":"2024-03-11T21:42:58.546849+0000","last_active":"2024-03-11T21:43:01.618133+0000","last_peered":"2024-03-11T21:43:01.618133+0000","last_clean":"2024-03-11T21:43:01.618133+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:43:01.618133+0000","last_undegraded":"2024-03-11T21:43:01.618133+0000","last_fullsized":"2024-03-11T21:43:01.618133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:42:58.546638+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.546638+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.143964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1657,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.594293+0000","last_change":"2024-03-11T21:41:28.709111+0000","last_active":"2024-03-11T21:43:01.594293+0000","last_peered":"2024-03-11T21:43:01.594293+0000","last_clean":"2024-03-11T21:43:01.594293+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:43:01.594293+0000","last_undegraded":"2024-03-11T21:43:01.594293+0000","last_fullsized":"2024-03-11T21:43:01.594293+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.707202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.707202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":862,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428679+0000","last_change":"2024-03-11T21:41:14.202929+0000","last_active":"2024-03-11T21:43:02.428679+0000","last_peered":"2024-03-11T21:43:02.428679+0000","last_clean":"2024-03-11T21:43:02.428679+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:43:02.428679+0000","last_undegraded":"2024-03-11T21:43:02.428679+0000","last_fullsized":"2024-03-11T21:43:02.428679+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:14.202622+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:37:53.724768+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.202622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1489,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811835+0000","last_change":"2024-03-11T21:41:25.364494+0000","last_active":"2024-03-11T21:43:01.811835+0000","last_peered":"2024-03-11T21:43:01.811835+0000","last_clean":"2024-03-11T21:43:01.811835+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:43:01.811835+0000","last_undegraded":"2024-03-11T21:43:01.811835+0000","last_fullsized":"2024-03-11T21:43:01.811835+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.364387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.364387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"389'27","reported_seq":1682,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811833+0000","last_change":"2024-03-11T21:40:06.159993+0000","last_active":"2024-03-11T21:43:01.811833+0000","last_peered":"2024-03-11T21:43:01.811833+0000","last_clean":"2024-03-11T21:43:01.811833+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:43:01.811833+0000","last_undegraded":"2024-03-11T21:43:01.811833+0000","last_fullsized":"2024-03-11T21:43:01.811833+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:40:06.159823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.159823+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1541,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581200+0000","last_change":"2024-03-11T21:42:55.086019+0000","last_active":"2024-03-11T21:42:59.581200+0000","last_peered":"2024-03-11T21:42:59.581200+0000","last_clean":"2024-03-11T21:42:59.581200+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:42:59.581200+0000","last_undegraded":"2024-03-11T21:42:59.581200+0000","last_fullsized":"2024-03-11T21:42:59.581200+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.085823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.085823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.857865+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1902,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593835+0000","last_change":"2024-03-11T21:41:46.623952+0000","last_active":"2024-03-11T21:43:01.593835+0000","last_peered":"2024-03-11T21:43:01.593835+0000","last_clean":"2024-03-11T21:43:01.593835+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:43:01.593835+0000","last_undegraded":"2024-03-11T21:43:01.593835+0000","last_fullsized":"2024-03-11T21:43:01.593835+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:46.623794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.623794+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1642,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430570+0000","last_change":"2024-03-11T21:41:58.366463+0000","last_active":"2024-03-11T21:43:02.430570+0000","last_peered":"2024-03-11T21:43:02.430570+0000","last_clean":"2024-03-11T21:43:02.430570+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:43:02.430570+0000","last_undegraded":"2024-03-11T21:43:02.430570+0000","last_fullsized":"2024-03-11T21:43:02.430570+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.366196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.366196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.319086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715572+0000","last_change":"2024-03-11T21:42:52.391444+0000","last_active":"2024-03-11T21:43:01.715572+0000","last_peered":"2024-03-11T21:43:01.715572+0000","last_clean":"2024-03-11T21:43:01.715572+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:43:01.715572+0000","last_undegraded":"2024-03-11T21:43:01.715572+0000","last_fullsized":"2024-03-11T21:43:01.715572+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.391288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.391288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.913321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"389'23","reported_seq":1694,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.625629+0000","last_change":"2024-03-11T21:43:02.625794+0000","last_active":"2024-03-11T21:43:02.625629+0000","last_peered":"2024-03-11T21:43:02.625629+0000","last_clean":"2024-03-11T21:43:02.625629+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:43:02.625629+0000","last_undegraded":"2024-03-11T21:43:02.625629+0000","last_fullsized":"2024-03-11T21:43:02.625629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:43:02.625424+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.625424+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.526217+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1642,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715278+0000","last_change":"2024-03-11T21:41:18.465526+0000","last_active":"2024-03-11T21:43:01.715278+0000","last_peered":"2024-03-11T21:43:01.715278+0000","last_clean":"2024-03-11T21:43:01.715278+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:43:01.715278+0000","last_undegraded":"2024-03-11T21:43:01.715278+0000","last_fullsized":"2024-03-11T21:43:01.715278+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.465412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.465412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1629,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.433238+0000","last_change":"2024-03-11T21:41:29.273257+0000","last_active":"2024-03-11T21:43:02.433238+0000","last_peered":"2024-03-11T21:43:02.433238+0000","last_clean":"2024-03-11T21:43:02.433238+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:43:02.433238+0000","last_undegraded":"2024-03-11T21:43:02.433238+0000","last_fullsized":"2024-03-11T21:43:02.433238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.273150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.273150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1621,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813005+0000","last_change":"2024-03-11T21:41:26.327523+0000","last_active":"2024-03-11T21:43:01.813005+0000","last_peered":"2024-03-11T21:43:01.813005+0000","last_clean":"2024-03-11T21:43:01.813005+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:43:01.813005+0000","last_undegraded":"2024-03-11T21:43:01.813005+0000","last_fullsized":"2024-03-11T21:43:01.813005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.327416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.327416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":864,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813146+0000","last_change":"2024-03-11T21:41:57.696381+0000","last_active":"2024-03-11T21:43:01.813146+0000","last_peered":"2024-03-11T21:43:01.813146+0000","last_clean":"2024-03-11T21:43:01.813146+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:43:01.813146+0000","last_undegraded":"2024-03-11T21:43:01.813146+0000","last_fullsized":"2024-03-11T21:43:01.813146+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:41:57.696205+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.696205+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1472,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813250+0000","last_change":"2024-03-11T21:40:03.109654+0000","last_active":"2024-03-11T21:43:01.813250+0000","last_peered":"2024-03-11T21:43:01.813250+0000","last_clean":"2024-03-11T21:43:01.813250+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:43:01.813250+0000","last_undegraded":"2024-03-11T21:43:01.813250+0000","last_fullsized":"2024-03-11T21:43:01.813250+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.109518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.109518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1886,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812092+0000","last_change":"2024-03-11T21:39:59.035379+0000","last_active":"2024-03-11T21:43:01.812092+0000","last_peered":"2024-03-11T21:43:01.812092+0000","last_clean":"2024-03-11T21:43:01.812092+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:43:01.812092+0000","last_undegraded":"2024-03-11T21:43:01.812092+0000","last_fullsized":"2024-03-11T21:43:01.812092+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:39:59.035148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.035148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1670,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.591060+0000","last_change":"2024-03-11T21:42:56.717568+0000","last_active":"2024-03-11T21:43:01.591060+0000","last_peered":"2024-03-11T21:43:01.591060+0000","last_clean":"2024-03-11T21:43:01.591060+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:43:01.591060+0000","last_undegraded":"2024-03-11T21:43:01.591060+0000","last_fullsized":"2024-03-11T21:43:01.591060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.717329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.717329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.987508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"389'24","reported_seq":1685,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623904+0000","last_change":"2024-03-11T21:41:43.459254+0000","last_active":"2024-03-11T21:43:01.623904+0000","last_peered":"2024-03-11T21:43:01.623904+0000","last_clean":"2024-03-11T21:43:01.623904+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:43:01.623904+0000","last_undegraded":"2024-03-11T21:43:01.623904+0000","last_fullsized":"2024-03-11T21:43:01.623904+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:41:43.458986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.458986+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1677,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640533+0000","last_change":"2024-03-11T21:41:55.649415+0000","last_active":"2024-03-11T21:43:01.640533+0000","last_peered":"2024-03-11T21:43:01.640533+0000","last_clean":"2024-03-11T21:43:01.640533+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:43:01.640533+0000","last_undegraded":"2024-03-11T21:43:01.640533+0000","last_fullsized":"2024-03-11T21:43:01.640533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:41:55.649085+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.649085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.612818+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1623,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716264+0000","last_change":"2024-03-11T21:40:23.749973+0000","last_active":"2024-03-11T21:43:01.716264+0000","last_peered":"2024-03-11T21:43:01.716264+0000","last_clean":"2024-03-11T21:43:01.716264+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:43:01.716264+0000","last_undegraded":"2024-03-11T21:43:01.716264+0000","last_fullsized":"2024-03-11T21:43:01.716264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:40:23.749772+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.749772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":877,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.592137+0000","last_change":"2024-03-11T21:41:56.716081+0000","last_active":"2024-03-11T21:43:01.592137+0000","last_peered":"2024-03-11T21:43:01.592137+0000","last_clean":"2024-03-11T21:43:01.592137+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:43:01.592137+0000","last_undegraded":"2024-03-11T21:43:01.592137+0000","last_fullsized":"2024-03-11T21:43:01.592137+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:56.715868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.715868+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716285+0000","last_change":"2024-03-11T21:41:42.428955+0000","last_active":"2024-03-11T21:43:01.716285+0000","last_peered":"2024-03-11T21:43:01.716285+0000","last_clean":"2024-03-11T21:43:01.716285+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:43:01.716285+0000","last_undegraded":"2024-03-11T21:43:01.716285+0000","last_fullsized":"2024-03-11T21:43:01.716285+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.428709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.428709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1661,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.590677+0000","last_change":"2024-03-11T21:41:42.630400+0000","last_active":"2024-03-11T21:43:01.590677+0000","last_peered":"2024-03-11T21:43:01.590677+0000","last_clean":"2024-03-11T21:43:01.590677+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:43:01.590677+0000","last_undegraded":"2024-03-11T21:43:01.590677+0000","last_fullsized":"2024-03-11T21:43:01.590677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.630272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.630272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2816,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812024+0000","last_change":"2024-03-11T21:41:20.351387+0000","last_active":"2024-03-11T21:43:01.812024+0000","last_peered":"2024-03-11T21:43:01.812024+0000","last_clean":"2024-03-11T21:43:01.812024+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:43:01.812024+0000","last_undegraded":"2024-03-11T21:43:01.812024+0000","last_fullsized":"2024-03-11T21:43:01.812024+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:41:20.351150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.351150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1178,"num_read_kb":1173,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"389'17","reported_seq":1663,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814716+0000","last_change":"2024-03-11T21:42:03.753084+0000","last_active":"2024-03-11T21:43:01.814716+0000","last_peered":"2024-03-11T21:43:01.814716+0000","last_clean":"2024-03-11T21:43:01.814716+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:43:01.814716+0000","last_undegraded":"2024-03-11T21:43:01.814716+0000","last_fullsized":"2024-03-11T21:43:01.814716+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:42:03.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:03.752924+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:08.213946+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1558,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582512+0000","last_change":"2024-03-11T21:40:06.437710+0000","last_active":"2024-03-11T21:42:59.582512+0000","last_peered":"2024-03-11T21:42:59.582512+0000","last_clean":"2024-03-11T21:42:59.582512+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:42:59.582512+0000","last_undegraded":"2024-03-11T21:42:59.582512+0000","last_fullsized":"2024-03-11T21:42:59.582512+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:40:06.437601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.437601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1649,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.592100+0000","last_change":"2024-03-11T21:41:20.553933+0000","last_active":"2024-03-11T21:43:01.592100+0000","last_peered":"2024-03-11T21:43:01.592100+0000","last_clean":"2024-03-11T21:43:01.592100+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:43:01.592100+0000","last_undegraded":"2024-03-11T21:43:01.592100+0000","last_fullsized":"2024-03-11T21:43:01.592100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.553712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.553712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":878,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640002+0000","last_change":"2024-03-11T21:41:27.847353+0000","last_active":"2024-03-11T21:43:01.640002+0000","last_peered":"2024-03-11T21:43:01.640002+0000","last_clean":"2024-03-11T21:43:01.640002+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:43:01.640002+0000","last_undegraded":"2024-03-11T21:43:01.640002+0000","last_fullsized":"2024-03-11T21:43:01.640002+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:27.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.847107+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716508+0000","last_change":"2024-03-11T21:41:35.427640+0000","last_active":"2024-03-11T21:43:01.716508+0000","last_peered":"2024-03-11T21:43:01.716508+0000","last_clean":"2024-03-11T21:43:01.716508+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:43:01.716508+0000","last_undegraded":"2024-03-11T21:43:01.716508+0000","last_fullsized":"2024-03-11T21:43:01.716508+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.427438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.427438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1644,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639314+0000","last_change":"2024-03-11T21:41:23.914266+0000","last_active":"2024-03-11T21:43:01.639314+0000","last_peered":"2024-03-11T21:43:01.639314+0000","last_clean":"2024-03-11T21:43:01.639314+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:43:01.639314+0000","last_undegraded":"2024-03-11T21:43:01.639314+0000","last_fullsized":"2024-03-11T21:43:01.639314+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.913967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.913967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"481'54","reported_seq":1834,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582586+0000","last_change":"2024-03-11T21:41:19.438289+0000","last_active":"2024-03-11T21:42:59.582586+0000","last_peered":"2024-03-11T21:42:59.582586+0000","last_clean":"2024-03-11T21:42:59.582586+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:42:59.582586+0000","last_undegraded":"2024-03-11T21:42:59.582586+0000","last_fullsized":"2024-03-11T21:42:59.582586+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'54","last_scrub_stamp":"2024-03-11T21:41:19.438160+0000","last_deep_scrub":"481'54","last_deep_scrub_stamp":"2024-03-11T21:34:55.143614+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.438160+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1633,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716496+0000","last_change":"2024-03-11T21:41:44.440095+0000","last_active":"2024-03-11T21:43:01.716496+0000","last_peered":"2024-03-11T21:43:01.716496+0000","last_clean":"2024-03-11T21:43:01.716496+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:43:01.716496+0000","last_undegraded":"2024-03-11T21:43:01.716496+0000","last_fullsized":"2024-03-11T21:43:01.716496+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:44.439846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:44.439846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1497,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.580945+0000","last_change":"2024-03-11T21:40:17.365129+0000","last_active":"2024-03-11T21:42:59.580945+0000","last_peered":"2024-03-11T21:42:59.580945+0000","last_clean":"2024-03-11T21:42:59.580945+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:42:59.580945+0000","last_undegraded":"2024-03-11T21:42:59.580945+0000","last_fullsized":"2024-03-11T21:42:59.580945+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:40:17.364997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":867,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581055+0000","last_change":"2024-03-11T21:40:27.432675+0000","last_active":"2024-03-11T21:42:59.581055+0000","last_peered":"2024-03-11T21:42:59.581055+0000","last_clean":"2024-03-11T21:42:59.581055+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:42:59.581055+0000","last_undegraded":"2024-03-11T21:42:59.581055+0000","last_fullsized":"2024-03-11T21:42:59.581055+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:40:27.432564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.432564+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1489,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623363+0000","last_change":"2024-03-11T21:42:54.461064+0000","last_active":"2024-03-11T21:43:01.623363+0000","last_peered":"2024-03-11T21:43:01.623363+0000","last_clean":"2024-03-11T21:43:01.623363+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:43:01.623363+0000","last_undegraded":"2024-03-11T21:43:01.623363+0000","last_fullsized":"2024-03-11T21:43:01.623363+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.460846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.460846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.439479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1639,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.433232+0000","last_change":"2024-03-11T21:41:17.193786+0000","last_active":"2024-03-11T21:43:02.433232+0000","last_peered":"2024-03-11T21:43:02.433232+0000","last_clean":"2024-03-11T21:43:02.433232+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:43:02.433232+0000","last_undegraded":"2024-03-11T21:43:02.433232+0000","last_fullsized":"2024-03-11T21:43:02.433232+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.193618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.193618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1643,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715198+0000","last_change":"2024-03-11T21:42:04.407830+0000","last_active":"2024-03-11T21:43:01.715198+0000","last_peered":"2024-03-11T21:43:01.715198+0000","last_clean":"2024-03-11T21:43:01.715198+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:43:01.715198+0000","last_undegraded":"2024-03-11T21:43:01.715198+0000","last_fullsized":"2024-03-11T21:43:01.715198+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:04.407615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:42:04.407615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.937658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"389'28","reported_seq":1682,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622747+0000","last_change":"2024-03-11T21:42:00.572712+0000","last_active":"2024-03-11T21:43:01.622747+0000","last_peered":"2024-03-11T21:43:01.622747+0000","last_clean":"2024-03-11T21:43:01.622747+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:43:01.622747+0000","last_undegraded":"2024-03-11T21:43:01.622747+0000","last_fullsized":"2024-03-11T21:43:01.622747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:00.572462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.572462+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.915234+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1507,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582175+0000","last_change":"2024-03-11T21:41:33.473130+0000","last_active":"2024-03-11T21:42:59.582175+0000","last_peered":"2024-03-11T21:42:59.582175+0000","last_clean":"2024-03-11T21:42:59.582175+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:42:59.582175+0000","last_undegraded":"2024-03-11T21:42:59.582175+0000","last_fullsized":"2024-03-11T21:42:59.582175+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.472920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.472920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1480,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811618+0000","last_change":"2024-03-11T21:42:01.660218+0000","last_active":"2024-03-11T21:43:01.811618+0000","last_peered":"2024-03-11T21:43:01.811618+0000","last_clean":"2024-03-11T21:43:01.811618+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:43:01.811618+0000","last_undegraded":"2024-03-11T21:43:01.811618+0000","last_fullsized":"2024-03-11T21:43:01.811618+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.660109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.660109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:31.505443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1617,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716197+0000","last_change":"2024-03-11T21:40:22.764357+0000","last_active":"2024-03-11T21:43:01.716197+0000","last_peered":"2024-03-11T21:43:01.716197+0000","last_clean":"2024-03-11T21:43:01.716197+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:43:01.716197+0000","last_undegraded":"2024-03-11T21:43:01.716197+0000","last_fullsized":"2024-03-11T21:43:01.716197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:22.764240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:22.764240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1907,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715273+0000","last_change":"2024-03-11T21:41:41.451026+0000","last_active":"2024-03-11T21:43:01.715273+0000","last_peered":"2024-03-11T21:43:01.715273+0000","last_clean":"2024-03-11T21:43:01.715273+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:43:01.715273+0000","last_undegraded":"2024-03-11T21:43:01.715273+0000","last_fullsized":"2024-03-11T21:43:01.715273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:41.450896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.450896+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1638,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429376+0000","last_change":"2024-03-11T21:40:10.827418+0000","last_active":"2024-03-11T21:43:02.429376+0000","last_peered":"2024-03-11T21:43:02.429376+0000","last_clean":"2024-03-11T21:43:02.429376+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:43:02.429376+0000","last_undegraded":"2024-03-11T21:43:02.429376+0000","last_fullsized":"2024-03-11T21:43:02.429376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:10.827185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:40:10.827185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"389'20","reported_seq":1660,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811530+0000","last_change":"2024-03-11T21:41:35.415804+0000","last_active":"2024-03-11T21:43:01.811530+0000","last_peered":"2024-03-11T21:43:01.811530+0000","last_clean":"2024-03-11T21:43:01.811530+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:43:01.811530+0000","last_undegraded":"2024-03-11T21:43:01.811530+0000","last_fullsized":"2024-03-11T21:43:01.811530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:35.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.415630+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":435,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623744+0000","last_change":"2024-03-11T21:41:37.415103+0000","last_active":"2024-03-11T21:43:01.623744+0000","last_peered":"2024-03-11T21:43:01.623744+0000","last_clean":"2024-03-11T21:43:01.623744+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:43:01.623744+0000","last_undegraded":"2024-03-11T21:43:01.623744+0000","last_fullsized":"2024-03-11T21:43:01.623744+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.414919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.414919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1914,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716130+0000","last_change":"2024-03-11T21:41:38.433752+0000","last_active":"2024-03-11T21:43:01.716130+0000","last_peered":"2024-03-11T21:43:01.716130+0000","last_clean":"2024-03-11T21:43:01.716130+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:43:01.716130+0000","last_undegraded":"2024-03-11T21:43:01.716130+0000","last_fullsized":"2024-03-11T21:43:01.716130+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:38.433381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:38.433381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.908595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1484,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429364+0000","last_change":"2024-03-11T21:40:19.766314+0000","last_active":"2024-03-11T21:43:02.429364+0000","last_peered":"2024-03-11T21:43:02.429364+0000","last_clean":"2024-03-11T21:43:02.429364+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:43:02.429364+0000","last_undegraded":"2024-03-11T21:43:02.429364+0000","last_fullsized":"2024-03-11T21:43:02.429364+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.766007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.766007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1665,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.619055+0000","last_change":"2024-03-11T21:41:15.651241+0000","last_active":"2024-03-11T21:43:01.619055+0000","last_peered":"2024-03-11T21:43:01.619055+0000","last_clean":"2024-03-11T21:43:01.619055+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:43:01.619055+0000","last_undegraded":"2024-03-11T21:43:01.619055+0000","last_fullsized":"2024-03-11T21:43:01.619055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.651121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.651121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1673,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617458+0000","last_change":"2024-03-11T21:41:41.822816+0000","last_active":"2024-03-11T21:43:01.617458+0000","last_peered":"2024-03-11T21:43:01.617458+0000","last_clean":"2024-03-11T21:43:01.617458+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:43:01.617458+0000","last_undegraded":"2024-03-11T21:43:01.617458+0000","last_fullsized":"2024-03-11T21:43:01.617458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.822599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.822599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"389'38","reported_seq":1589,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582130+0000","last_change":"2024-03-11T21:41:27.465101+0000","last_active":"2024-03-11T21:42:59.582130+0000","last_peered":"2024-03-11T21:42:59.582130+0000","last_clean":"2024-03-11T21:42:59.582130+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:42:59.582130+0000","last_undegraded":"2024-03-11T21:42:59.582130+0000","last_fullsized":"2024-03-11T21:42:59.582130+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:41:27.464990+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.464990+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1621,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716425+0000","last_change":"2024-03-11T21:43:01.436177+0000","last_active":"2024-03-11T21:43:01.716425+0000","last_peered":"2024-03-11T21:43:01.716425+0000","last_clean":"2024-03-11T21:43:01.716425+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:43:01.716425+0000","last_undegraded":"2024-03-11T21:43:01.716425+0000","last_fullsized":"2024-03-11T21:43:01.716425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.435975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.435975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.229611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":862,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428541+0000","last_change":"2024-03-11T21:41:18.182642+0000","last_active":"2024-03-11T21:43:02.428541+0000","last_peered":"2024-03-11T21:43:02.428541+0000","last_clean":"2024-03-11T21:43:02.428541+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:43:02.428541+0000","last_undegraded":"2024-03-11T21:43:02.428541+0000","last_fullsized":"2024-03-11T21:43:02.428541+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:18.182513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.182513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1489,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622608+0000","last_change":"2024-03-11T21:42:01.549509+0000","last_active":"2024-03-11T21:43:01.622608+0000","last_peered":"2024-03-11T21:43:01.622608+0000","last_clean":"2024-03-11T21:43:01.622608+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:43:01.622608+0000","last_undegraded":"2024-03-11T21:43:01.622608+0000","last_fullsized":"2024-03-11T21:43:01.622608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.549350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:07.861337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1645,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623564+0000","last_change":"2024-03-11T21:41:58.613981+0000","last_active":"2024-03-11T21:43:01.623564+0000","last_peered":"2024-03-11T21:43:01.623564+0000","last_clean":"2024-03-11T21:43:01.623564+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:43:01.623564+0000","last_undegraded":"2024-03-11T21:43:01.623564+0000","last_fullsized":"2024-03-11T21:43:01.623564+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.613685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.613685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:23.992971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"389'37","reported_seq":1716,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.591678+0000","last_change":"2024-03-11T21:41:29.674697+0000","last_active":"2024-03-11T21:43:01.591678+0000","last_peered":"2024-03-11T21:43:01.591678+0000","last_clean":"2024-03-11T21:43:01.591678+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:43:01.591678+0000","last_undegraded":"2024-03-11T21:43:01.591678+0000","last_fullsized":"2024-03-11T21:43:01.591678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:41:29.674458+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.674458+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1647,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429997+0000","last_change":"2024-03-11T21:41:34.232210+0000","last_active":"2024-03-11T21:43:02.429997+0000","last_peered":"2024-03-11T21:43:02.429997+0000","last_clean":"2024-03-11T21:43:02.429997+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:43:02.429997+0000","last_undegraded":"2024-03-11T21:43:02.429997+0000","last_fullsized":"2024-03-11T21:43:02.429997+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:34.231996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:34.231996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1622,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814044+0000","last_change":"2024-03-11T21:41:17.378484+0000","last_active":"2024-03-11T21:43:01.814044+0000","last_peered":"2024-03-11T21:43:01.814044+0000","last_clean":"2024-03-11T21:43:01.814044+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:43:01.814044+0000","last_undegraded":"2024-03-11T21:43:01.814044+0000","last_fullsized":"2024-03-11T21:43:01.814044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.378291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.378291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":878,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640374+0000","last_change":"2024-03-11T21:41:51.718486+0000","last_active":"2024-03-11T21:43:01.640374+0000","last_peered":"2024-03-11T21:43:01.640374+0000","last_clean":"2024-03-11T21:43:01.640374+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:43:01.640374+0000","last_undegraded":"2024-03-11T21:43:01.640374+0000","last_fullsized":"2024-03-11T21:43:01.640374+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:51.718268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.718268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1484,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432583+0000","last_change":"2024-03-11T21:40:21.837198+0000","last_active":"2024-03-11T21:43:02.432583+0000","last_peered":"2024-03-11T21:43:02.432583+0000","last_clean":"2024-03-11T21:43:02.432583+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:43:02.432583+0000","last_undegraded":"2024-03-11T21:43:02.432583+0000","last_fullsized":"2024-03-11T21:43:02.432583+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.836981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.836981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1890,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623342+0000","last_change":"2024-03-11T21:41:40.335413+0000","last_active":"2024-03-11T21:43:01.623342+0000","last_peered":"2024-03-11T21:43:01.623342+0000","last_clean":"2024-03-11T21:43:01.623342+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:43:01.623342+0000","last_undegraded":"2024-03-11T21:43:01.623342+0000","last_fullsized":"2024-03-11T21:43:01.623342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:41:40.335178+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.335178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.489495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1542,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.949354+0000","last_change":"2024-03-11T21:42:59.949425+0000","last_active":"2024-03-11T21:42:59.949354+0000","last_peered":"2024-03-11T21:42:59.949354+0000","last_clean":"2024-03-11T21:42:59.949354+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:42:59.949354+0000","last_undegraded":"2024-03-11T21:42:59.949354+0000","last_fullsized":"2024-03-11T21:42:59.949354+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:42:59.949164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.949164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.748601+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"389'28","reported_seq":1922,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430334+0000","last_change":"2024-03-11T21:42:56.554589+0000","last_active":"2024-03-11T21:43:02.430334+0000","last_peered":"2024-03-11T21:43:02.430334+0000","last_clean":"2024-03-11T21:43:02.430334+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:43:02.430334+0000","last_undegraded":"2024-03-11T21:43:02.430334+0000","last_fullsized":"2024-03-11T21:43:02.430334+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:56.554326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.554326+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.354495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1505,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581806+0000","last_change":"2024-03-11T21:40:03.406984+0000","last_active":"2024-03-11T21:42:59.581806+0000","last_peered":"2024-03-11T21:42:59.581806+0000","last_clean":"2024-03-11T21:42:59.581806+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:42:59.581806+0000","last_undegraded":"2024-03-11T21:42:59.581806+0000","last_fullsized":"2024-03-11T21:42:59.581806+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.406773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.406773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1631,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622855+0000","last_change":"2024-03-11T21:41:28.319779+0000","last_active":"2024-03-11T21:43:01.622855+0000","last_peered":"2024-03-11T21:43:01.622855+0000","last_clean":"2024-03-11T21:43:01.622855+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:43:01.622855+0000","last_undegraded":"2024-03-11T21:43:01.622855+0000","last_fullsized":"2024-03-11T21:43:01.622855+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:28.319670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.319670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1374,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581946+0000","last_change":"2024-03-11T21:42:54.061349+0000","last_active":"2024-03-11T21:42:59.581946+0000","last_peered":"2024-03-11T21:42:59.581946+0000","last_clean":"2024-03-11T21:42:59.581946+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:42:59.581946+0000","last_undegraded":"2024-03-11T21:42:59.581946+0000","last_fullsized":"2024-03-11T21:42:59.581946+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.061137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.061137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.609578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1674,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.616604+0000","last_change":"2024-03-11T21:41:45.803597+0000","last_active":"2024-03-11T21:43:01.616604+0000","last_peered":"2024-03-11T21:43:01.616604+0000","last_clean":"2024-03-11T21:43:01.616604+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:43:01.616604+0000","last_undegraded":"2024-03-11T21:43:01.616604+0000","last_fullsized":"2024-03-11T21:43:01.616604+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.803247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.803247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"389'24","reported_seq":1673,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716702+0000","last_change":"2024-03-11T21:39:58.844136+0000","last_active":"2024-03-11T21:43:01.716702+0000","last_peered":"2024-03-11T21:43:01.716702+0000","last_clean":"2024-03-11T21:43:01.716702+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:43:01.716702+0000","last_undegraded":"2024-03-11T21:43:01.716702+0000","last_fullsized":"2024-03-11T21:43:01.716702+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:39:58.843910+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.843910+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1686,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593414+0000","last_change":"2024-03-11T21:42:57.712087+0000","last_active":"2024-03-11T21:43:01.593414+0000","last_peered":"2024-03-11T21:43:01.593414+0000","last_clean":"2024-03-11T21:43:01.593414+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:43:01.593414+0000","last_undegraded":"2024-03-11T21:43:01.593414+0000","last_fullsized":"2024-03-11T21:43:01.593414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:42:57.711946+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.711946+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.971914+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1630,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813652+0000","last_change":"2024-03-11T21:40:24.477381+0000","last_active":"2024-03-11T21:43:01.813652+0000","last_peered":"2024-03-11T21:43:01.813652+0000","last_clean":"2024-03-11T21:43:01.813652+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:43:01.813652+0000","last_undegraded":"2024-03-11T21:43:01.813652+0000","last_fullsized":"2024-03-11T21:43:01.813652+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.477246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.477246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813907+0000","last_change":"2024-03-11T21:41:15.373061+0000","last_active":"2024-03-11T21:43:01.813907+0000","last_peered":"2024-03-11T21:43:01.813907+0000","last_clean":"2024-03-11T21:43:01.813907+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:43:01.813907+0000","last_undegraded":"2024-03-11T21:43:01.813907+0000","last_fullsized":"2024-03-11T21:43:01.813907+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.372842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.372842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"389'19","reported_seq":1659,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715664+0000","last_change":"2024-03-11T21:42:59.435802+0000","last_active":"2024-03-11T21:43:01.715664+0000","last_peered":"2024-03-11T21:43:01.715664+0000","last_clean":"2024-03-11T21:43:01.715664+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:43:01.715664+0000","last_undegraded":"2024-03-11T21:43:01.715664+0000","last_fullsized":"2024-03-11T21:43:01.715664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:42:59.435557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.435557+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:08.758869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1681,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.616593+0000","last_change":"2024-03-11T21:41:43.763506+0000","last_active":"2024-03-11T21:43:01.616593+0000","last_peered":"2024-03-11T21:43:01.616593+0000","last_clean":"2024-03-11T21:43:01.616593+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:43:01.616593+0000","last_undegraded":"2024-03-11T21:43:01.616593+0000","last_fullsized":"2024-03-11T21:43:01.616593+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:43.763282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:41:43.763282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1634,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814896+0000","last_change":"2024-03-11T21:41:18.342623+0000","last_active":"2024-03-11T21:43:01.814896+0000","last_peered":"2024-03-11T21:43:01.814896+0000","last_clean":"2024-03-11T21:43:01.814896+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:43:01.814896+0000","last_undegraded":"2024-03-11T21:43:01.814896+0000","last_fullsized":"2024-03-11T21:43:01.814896+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.342525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.342525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1668,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.617931+0000","last_change":"2024-03-11T21:42:52.491927+0000","last_active":"2024-03-11T21:43:01.617931+0000","last_peered":"2024-03-11T21:43:01.617931+0000","last_clean":"2024-03-11T21:43:01.617931+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:43:01.617931+0000","last_undegraded":"2024-03-11T21:43:01.617931+0000","last_fullsized":"2024-03-11T21:43:01.617931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.491711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.491711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.191914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812115+0000","last_change":"2024-03-11T21:41:37.401402+0000","last_active":"2024-03-11T21:43:01.812115+0000","last_peered":"2024-03-11T21:43:01.812115+0000","last_clean":"2024-03-11T21:43:01.812115+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:43:01.812115+0000","last_undegraded":"2024-03-11T21:43:01.812115+0000","last_fullsized":"2024-03-11T21:43:01.812115+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.401275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.401275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.811147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"389'10","reported_seq":1677,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.638979+0000","last_change":"2024-03-11T21:41:49.695635+0000","last_active":"2024-03-11T21:43:01.638979+0000","last_peered":"2024-03-11T21:43:01.638979+0000","last_clean":"2024-03-11T21:43:01.638979+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:43:01.638979+0000","last_undegraded":"2024-03-11T21:43:01.638979+0000","last_fullsized":"2024-03-11T21:43:01.638979+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:41:49.695261+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.695261+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:19.305229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1668,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639498+0000","last_change":"2024-03-11T21:42:05.515337+0000","last_active":"2024-03-11T21:43:01.639498+0000","last_peered":"2024-03-11T21:43:01.639498+0000","last_clean":"2024-03-11T21:43:01.639498+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:43:01.639498+0000","last_undegraded":"2024-03-11T21:43:01.639498+0000","last_fullsized":"2024-03-11T21:43:01.639498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:42:05.515038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:33.818265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1638,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432507+0000","last_change":"2024-03-11T21:41:42.239297+0000","last_active":"2024-03-11T21:43:02.432507+0000","last_peered":"2024-03-11T21:43:02.432507+0000","last_clean":"2024-03-11T21:43:02.432507+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:43:02.432507+0000","last_undegraded":"2024-03-11T21:43:02.432507+0000","last_fullsized":"2024-03-11T21:43:02.432507+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.239053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1620,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716371+0000","last_change":"2024-03-11T21:42:58.421950+0000","last_active":"2024-03-11T21:43:01.716371+0000","last_peered":"2024-03-11T21:43:01.716371+0000","last_clean":"2024-03-11T21:43:01.716371+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:43:01.716371+0000","last_undegraded":"2024-03-11T21:43:01.716371+0000","last_fullsized":"2024-03-11T21:43:01.716371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.421800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.421800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.437625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1479,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812212+0000","last_change":"2024-03-11T21:42:54.754828+0000","last_active":"2024-03-11T21:43:01.812212+0000","last_peered":"2024-03-11T21:43:01.812212+0000","last_clean":"2024-03-11T21:43:01.812212+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:43:01.812212+0000","last_undegraded":"2024-03-11T21:43:01.812212+0000","last_fullsized":"2024-03-11T21:43:01.812212+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.754521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.754521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.619900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"389'19","reported_seq":1659,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716473+0000","last_change":"2024-03-11T21:40:26.728613+0000","last_active":"2024-03-11T21:43:01.716473+0000","last_peered":"2024-03-11T21:43:01.716473+0000","last_clean":"2024-03-11T21:43:01.716473+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:43:01.716473+0000","last_undegraded":"2024-03-11T21:43:01.716473+0000","last_fullsized":"2024-03-11T21:43:01.716473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:26.728512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.728512+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1645,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429847+0000","last_change":"2024-03-11T21:40:20.810639+0000","last_active":"2024-03-11T21:43:02.429847+0000","last_peered":"2024-03-11T21:43:02.429847+0000","last_clean":"2024-03-11T21:43:02.429847+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:43:02.429847+0000","last_undegraded":"2024-03-11T21:43:02.429847+0000","last_fullsized":"2024-03-11T21:43:02.429847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.810193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.810193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1889,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430881+0000","last_change":"2024-03-11T21:41:49.267753+0000","last_active":"2024-03-11T21:43:02.430881+0000","last_peered":"2024-03-11T21:43:02.430881+0000","last_clean":"2024-03-11T21:43:02.430881+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:43:02.430881+0000","last_undegraded":"2024-03-11T21:43:02.430881+0000","last_fullsized":"2024-03-11T21:43:02.430881+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:41:49.267522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.267522+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:14.956526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1631,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.624089+0000","last_change":"2024-03-11T21:40:23.588457+0000","last_active":"2024-03-11T21:43:01.624089+0000","last_peered":"2024-03-11T21:43:01.624089+0000","last_clean":"2024-03-11T21:43:01.624089+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:43:01.624089+0000","last_undegraded":"2024-03-11T21:43:01.624089+0000","last_fullsized":"2024-03-11T21:43:01.624089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:23.588113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.588113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1511,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639245+0000","last_change":"2024-03-11T21:42:57.297254+0000","last_active":"2024-03-11T21:43:01.639245+0000","last_peered":"2024-03-11T21:43:01.639245+0000","last_clean":"2024-03-11T21:43:01.639245+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:43:01.639245+0000","last_undegraded":"2024-03-11T21:43:01.639245+0000","last_fullsized":"2024-03-11T21:43:01.639245+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:57.297086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.297086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.243930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"389'19","reported_seq":1671,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623181+0000","last_change":"2024-03-11T21:40:21.551903+0000","last_active":"2024-03-11T21:43:01.623181+0000","last_peered":"2024-03-11T21:43:01.623181+0000","last_clean":"2024-03-11T21:43:01.623181+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:43:01.623181+0000","last_undegraded":"2024-03-11T21:43:01.623181+0000","last_fullsized":"2024-03-11T21:43:01.623181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:21.551780+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.551780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1643,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623164+0000","last_change":"2024-03-11T21:41:31.320806+0000","last_active":"2024-03-11T21:43:01.623164+0000","last_peered":"2024-03-11T21:43:01.623164+0000","last_clean":"2024-03-11T21:43:01.623164+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:43:01.623164+0000","last_undegraded":"2024-03-11T21:43:01.623164+0000","last_fullsized":"2024-03-11T21:43:01.623164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.320685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.320685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1876,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.812982+0000","last_change":"2024-03-11T21:40:18.396071+0000","last_active":"2024-03-11T21:43:01.812982+0000","last_peered":"2024-03-11T21:43:01.812982+0000","last_clean":"2024-03-11T21:43:01.812982+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:43:01.812982+0000","last_undegraded":"2024-03-11T21:43:01.812982+0000","last_fullsized":"2024-03-11T21:43:01.812982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:40:18.395966+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.395966+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1656,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.594421+0000","last_change":"2024-03-11T21:41:57.724943+0000","last_active":"2024-03-11T21:43:01.594421+0000","last_peered":"2024-03-11T21:43:01.594421+0000","last_clean":"2024-03-11T21:43:01.594421+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:43:01.594421+0000","last_undegraded":"2024-03-11T21:43:01.594421+0000","last_fullsized":"2024-03-11T21:43:01.594421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.724694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:26.895410+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.724694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:18.584812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1362,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.583300+0000","last_change":"2024-03-11T21:41:55.220405+0000","last_active":"2024-03-11T21:42:59.583300+0000","last_peered":"2024-03-11T21:42:59.583300+0000","last_clean":"2024-03-11T21:42:59.583300+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:42:59.583300+0000","last_undegraded":"2024-03-11T21:42:59.583300+0000","last_fullsized":"2024-03-11T21:42:59.583300+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.220196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.220196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.381564+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"389'19","reported_seq":1657,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716787+0000","last_change":"2024-03-11T21:41:53.406126+0000","last_active":"2024-03-11T21:43:01.716787+0000","last_peered":"2024-03-11T21:43:01.716787+0000","last_clean":"2024-03-11T21:43:01.716787+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:43:01.716787+0000","last_undegraded":"2024-03-11T21:43:01.716787+0000","last_fullsized":"2024-03-11T21:43:01.716787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:41:53.405780+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.405780+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.864386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1667,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640335+0000","last_change":"2024-03-11T21:41:36.715615+0000","last_active":"2024-03-11T21:43:01.640335+0000","last_peered":"2024-03-11T21:43:01.640335+0000","last_clean":"2024-03-11T21:43:01.640335+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:43:01.640335+0000","last_undegraded":"2024-03-11T21:43:01.640335+0000","last_fullsized":"2024-03-11T21:43:01.640335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:36.715491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:54.928741+0000","last_clean_scrub_stamp":"2024-03-11T21:41:36.715491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1634,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813670+0000","last_change":"2024-03-11T21:41:31.332398+0000","last_active":"2024-03-11T21:43:01.813670+0000","last_peered":"2024-03-11T21:43:01.813670+0000","last_clean":"2024-03-11T21:43:01.813670+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:43:01.813670+0000","last_undegraded":"2024-03-11T21:43:01.813670+0000","last_fullsized":"2024-03-11T21:43:01.813670+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.332299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.332299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1639,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.622708+0000","last_change":"2024-03-11T21:41:32.365712+0000","last_active":"2024-03-11T21:43:01.622708+0000","last_peered":"2024-03-11T21:43:01.622708+0000","last_clean":"2024-03-11T21:43:01.622708+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:43:01.622708+0000","last_undegraded":"2024-03-11T21:43:01.622708+0000","last_fullsized":"2024-03-11T21:43:01.622708+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.365462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:07.660940+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.365462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1496,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623842+0000","last_change":"2024-03-11T21:42:58.427672+0000","last_active":"2024-03-11T21:43:01.623842+0000","last_peered":"2024-03-11T21:43:01.623842+0000","last_clean":"2024-03-11T21:43:01.623842+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:43:01.623842+0000","last_undegraded":"2024-03-11T21:43:01.623842+0000","last_fullsized":"2024-03-11T21:43:01.623842+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.427446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.427446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.731909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"389'20","reported_seq":1694,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.592733+0000","last_change":"2024-03-11T21:41:15.535768+0000","last_active":"2024-03-11T21:43:01.592733+0000","last_peered":"2024-03-11T21:43:01.592733+0000","last_clean":"2024-03-11T21:43:01.592733+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:43:01.592733+0000","last_undegraded":"2024-03-11T21:43:01.592733+0000","last_fullsized":"2024-03-11T21:43:01.592733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:15.535528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.535528+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1548,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582298+0000","last_change":"2024-03-11T21:41:21.483590+0000","last_active":"2024-03-11T21:42:59.582298+0000","last_peered":"2024-03-11T21:42:59.582298+0000","last_clean":"2024-03-11T21:42:59.582298+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:42:59.582298+0000","last_undegraded":"2024-03-11T21:42:59.582298+0000","last_fullsized":"2024-03-11T21:42:59.582298+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.483367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.483367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1516,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581559+0000","last_change":"2024-03-11T21:40:18.382831+0000","last_active":"2024-03-11T21:42:59.581559+0000","last_peered":"2024-03-11T21:42:59.581559+0000","last_clean":"2024-03-11T21:42:59.581559+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:42:59.581559+0000","last_undegraded":"2024-03-11T21:42:59.581559+0000","last_fullsized":"2024-03-11T21:42:59.581559+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.382717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1621,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811719+0000","last_change":"2024-03-11T21:41:21.351275+0000","last_active":"2024-03-11T21:43:01.811719+0000","last_peered":"2024-03-11T21:43:01.811719+0000","last_clean":"2024-03-11T21:43:01.811719+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:43:01.811719+0000","last_undegraded":"2024-03-11T21:43:01.811719+0000","last_fullsized":"2024-03-11T21:43:01.811719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.350961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.350961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1486,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623156+0000","last_change":"2024-03-11T21:41:48.432501+0000","last_active":"2024-03-11T21:43:01.623156+0000","last_peered":"2024-03-11T21:43:01.623156+0000","last_clean":"2024-03-11T21:43:01.623156+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:43:01.623156+0000","last_undegraded":"2024-03-11T21:43:01.623156+0000","last_fullsized":"2024-03-11T21:43:01.623156+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_clean_scrub_stamp":"2024-03-11T21:41:48.432274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:12.542101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"389'12","reported_seq":1665,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623988+0000","last_change":"2024-03-11T21:42:02.547377+0000","last_active":"2024-03-11T21:43:01.623988+0000","last_peered":"2024-03-11T21:43:01.623988+0000","last_clean":"2024-03-11T21:43:01.623988+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:43:01.623988+0000","last_undegraded":"2024-03-11T21:43:01.623988+0000","last_fullsized":"2024-03-11T21:43:01.623988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:42:02.547005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.547005+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.798290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1637,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432800+0000","last_change":"2024-03-11T21:40:13.779995+0000","last_active":"2024-03-11T21:43:02.432800+0000","last_peered":"2024-03-11T21:43:02.432800+0000","last_clean":"2024-03-11T21:43:02.432800+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:43:02.432800+0000","last_undegraded":"2024-03-11T21:43:02.432800+0000","last_fullsized":"2024-03-11T21:43:02.432800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:13.779418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.779418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1659,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639295+0000","last_change":"2024-03-11T21:42:02.517280+0000","last_active":"2024-03-11T21:43:01.639295+0000","last_peered":"2024-03-11T21:43:01.639295+0000","last_clean":"2024-03-11T21:43:01.639295+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:43:01.639295+0000","last_undegraded":"2024-03-11T21:43:01.639295+0000","last_fullsized":"2024-03-11T21:43:01.639295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:02.516952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.516952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:21.145639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1612,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716018+0000","last_change":"2024-03-11T21:39:54.822173+0000","last_active":"2024-03-11T21:43:01.716018+0000","last_peered":"2024-03-11T21:43:01.716018+0000","last_clean":"2024-03-11T21:43:01.716018+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:43:01.716018+0000","last_undegraded":"2024-03-11T21:43:01.716018+0000","last_fullsized":"2024-03-11T21:43:01.716018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:54.822057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:39:54.822057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1489,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623311+0000","last_change":"2024-03-11T21:40:19.599299+0000","last_active":"2024-03-11T21:43:01.623311+0000","last_peered":"2024-03-11T21:43:01.623311+0000","last_clean":"2024-03-11T21:43:01.623311+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:43:01.623311+0000","last_undegraded":"2024-03-11T21:43:01.623311+0000","last_fullsized":"2024-03-11T21:43:01.623311+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.599110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.599110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"389'15","reported_seq":1667,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.432419+0000","last_change":"2024-03-11T21:40:25.893678+0000","last_active":"2024-03-11T21:43:02.432419+0000","last_peered":"2024-03-11T21:43:02.432419+0000","last_clean":"2024-03-11T21:43:02.432419+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:43:02.432419+0000","last_undegraded":"2024-03-11T21:43:02.432419+0000","last_fullsized":"2024-03-11T21:43:02.432419+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:40:25.893452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.893452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1667,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640503+0000","last_change":"2024-03-11T21:41:59.579259+0000","last_active":"2024-03-11T21:43:01.640503+0000","last_peered":"2024-03-11T21:43:01.640503+0000","last_clean":"2024-03-11T21:43:01.640503+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:43:01.640503+0000","last_undegraded":"2024-03-11T21:43:01.640503+0000","last_fullsized":"2024-03-11T21:43:01.640503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.579089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.579089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:11.224629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1616,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715429+0000","last_change":"2024-03-11T21:40:16.866813+0000","last_active":"2024-03-11T21:43:01.715429+0000","last_peered":"2024-03-11T21:43:01.715429+0000","last_clean":"2024-03-11T21:43:01.715429+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:43:01.715429+0000","last_undegraded":"2024-03-11T21:43:01.715429+0000","last_fullsized":"2024-03-11T21:43:01.715429+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:16.866545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:16.866545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1621,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813752+0000","last_change":"2024-03-11T21:41:53.585801+0000","last_active":"2024-03-11T21:43:01.813752+0000","last_peered":"2024-03-11T21:43:01.813752+0000","last_clean":"2024-03-11T21:43:01.813752+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:43:01.813752+0000","last_undegraded":"2024-03-11T21:43:01.813752+0000","last_fullsized":"2024-03-11T21:43:01.813752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.585657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:22.256998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1470,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716342+0000","last_change":"2024-03-11T21:40:18.842726+0000","last_active":"2024-03-11T21:43:01.716342+0000","last_peered":"2024-03-11T21:43:01.716342+0000","last_clean":"2024-03-11T21:43:01.716342+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:43:01.716342+0000","last_undegraded":"2024-03-11T21:43:01.716342+0000","last_fullsized":"2024-03-11T21:43:01.716342+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.842520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.842520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"538'34","reported_seq":1601,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581113+0000","last_change":"2024-03-11T21:39:59.292770+0000","last_active":"2024-03-11T21:42:59.581113+0000","last_peered":"2024-03-11T21:42:59.581113+0000","last_clean":"2024-03-11T21:42:59.581113+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:42:59.581113+0000","last_undegraded":"2024-03-11T21:42:59.581113+0000","last_fullsized":"2024-03-11T21:42:59.581113+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:39:59.292520+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:39:59.292520+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1651,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.494272+0000","last_change":"2024-03-11T21:43:02.494327+0000","last_active":"2024-03-11T21:43:02.494272+0000","last_peered":"2024-03-11T21:43:02.494272+0000","last_clean":"2024-03-11T21:43:02.494272+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:43:02.494272+0000","last_undegraded":"2024-03-11T21:43:02.494272+0000","last_fullsized":"2024-03-11T21:43:02.494272+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.494116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.494116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.677295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1913,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.640598+0000","last_change":"2024-03-11T21:41:33.752080+0000","last_active":"2024-03-11T21:43:01.640598+0000","last_peered":"2024-03-11T21:43:01.640598+0000","last_clean":"2024-03-11T21:43:01.640598+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:43:01.640598+0000","last_undegraded":"2024-03-11T21:43:01.640598+0000","last_fullsized":"2024-03-11T21:43:01.640598+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:33.751930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.751930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1613,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716307+0000","last_change":"2024-03-11T21:40:13.900758+0000","last_active":"2024-03-11T21:43:01.716307+0000","last_peered":"2024-03-11T21:43:01.716307+0000","last_clean":"2024-03-11T21:43:01.716307+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:43:01.716307+0000","last_undegraded":"2024-03-11T21:43:01.716307+0000","last_fullsized":"2024-03-11T21:43:01.716307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:40:13.900659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:13.900659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1492,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429369+0000","last_change":"2024-03-11T21:41:56.288055+0000","last_active":"2024-03-11T21:43:02.429369+0000","last_peered":"2024-03-11T21:43:02.429369+0000","last_clean":"2024-03-11T21:43:02.429369+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:43:02.429369+0000","last_undegraded":"2024-03-11T21:43:02.429369+0000","last_fullsized":"2024-03-11T21:43:02.429369+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.287964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.287964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"389'15","reported_seq":1671,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811701+0000","last_change":"2024-03-11T21:42:52.850471+0000","last_active":"2024-03-11T21:43:01.811701+0000","last_peered":"2024-03-11T21:43:01.811701+0000","last_clean":"2024-03-11T21:43:01.811701+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:43:01.811701+0000","last_undegraded":"2024-03-11T21:43:01.811701+0000","last_fullsized":"2024-03-11T21:43:01.811701+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:42:52.850333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.850333+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.477895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1634,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.811854+0000","last_change":"2024-03-11T21:39:58.005873+0000","last_active":"2024-03-11T21:43:01.811854+0000","last_peered":"2024-03-11T21:43:01.811854+0000","last_clean":"2024-03-11T21:43:01.811854+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:43:01.811854+0000","last_undegraded":"2024-03-11T21:43:01.811854+0000","last_fullsized":"2024-03-11T21:43:01.811854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:39:58.005747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:39:58.005747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1636,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.624051+0000","last_change":"2024-03-11T21:41:41.375255+0000","last_active":"2024-03-11T21:43:01.624051+0000","last_peered":"2024-03-11T21:43:01.624051+0000","last_clean":"2024-03-11T21:43:01.624051+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:43:01.624051+0000","last_undegraded":"2024-03-11T21:43:01.624051+0000","last_fullsized":"2024-03-11T21:43:01.624051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.375162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:09.174376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1507,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582162+0000","last_change":"2024-03-11T21:41:49.263645+0000","last_active":"2024-03-11T21:42:59.582162+0000","last_peered":"2024-03-11T21:42:59.582162+0000","last_clean":"2024-03-11T21:42:59.582162+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:42:59.582162+0000","last_undegraded":"2024-03-11T21:42:59.582162+0000","last_fullsized":"2024-03-11T21:42:59.582162+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.263416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:06.236541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1470,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715444+0000","last_change":"2024-03-11T21:41:23.491021+0000","last_active":"2024-03-11T21:43:01.715444+0000","last_peered":"2024-03-11T21:43:01.715444+0000","last_clean":"2024-03-11T21:43:01.715444+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:43:01.715444+0000","last_undegraded":"2024-03-11T21:43:01.715444+0000","last_fullsized":"2024-03-11T21:43:01.715444+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.490850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.490850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"389'9","reported_seq":1645,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623205+0000","last_change":"2024-03-11T21:40:05.512627+0000","last_active":"2024-03-11T21:43:01.623205+0000","last_peered":"2024-03-11T21:43:01.623205+0000","last_clean":"2024-03-11T21:43:01.623205+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:43:01.623205+0000","last_undegraded":"2024-03-11T21:43:01.623205+0000","last_fullsized":"2024-03-11T21:43:01.623205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:40:05.512463+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:40:05.512463+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1667,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.639227+0000","last_change":"2024-03-11T21:41:53.683571+0000","last_active":"2024-03-11T21:43:01.639227+0000","last_peered":"2024-03-11T21:43:01.639227+0000","last_clean":"2024-03-11T21:43:01.639227+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:43:01.639227+0000","last_undegraded":"2024-03-11T21:43:01.639227+0000","last_fullsized":"2024-03-11T21:43:01.639227+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.683465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.683465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.982276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1501,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582446+0000","last_change":"2024-03-11T21:41:17.344885+0000","last_active":"2024-03-11T21:42:59.582446+0000","last_peered":"2024-03-11T21:42:59.582446+0000","last_clean":"2024-03-11T21:42:59.582446+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:42:59.582446+0000","last_undegraded":"2024-03-11T21:42:59.582446+0000","last_fullsized":"2024-03-11T21:42:59.582446+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.344687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.344687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1649,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.594478+0000","last_change":"2024-03-11T21:41:19.561625+0000","last_active":"2024-03-11T21:43:01.594478+0000","last_peered":"2024-03-11T21:43:01.594478+0000","last_clean":"2024-03-11T21:43:01.594478+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:43:01.594478+0000","last_undegraded":"2024-03-11T21:43:01.594478+0000","last_fullsized":"2024-03-11T21:43:01.594478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.561348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.561348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1504,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.590046+0000","last_change":"2024-03-11T21:41:55.738509+0000","last_active":"2024-03-11T21:43:01.590046+0000","last_peered":"2024-03-11T21:43:01.590046+0000","last_clean":"2024-03-11T21:43:01.590046+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:43:01.590046+0000","last_undegraded":"2024-03-11T21:43:01.590046+0000","last_fullsized":"2024-03-11T21:43:01.590046+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.738282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.398264+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"714'19","reported_seq":1702,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593125+0000","last_change":"2024-03-11T21:42:51.740712+0000","last_active":"2024-03-11T21:43:01.593125+0000","last_peered":"2024-03-11T21:43:01.593125+0000","last_clean":"2024-03-11T21:43:01.593125+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:43:01.593125+0000","last_undegraded":"2024-03-11T21:43:01.593125+0000","last_fullsized":"2024-03-11T21:43:01.593125+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'19","last_scrub_stamp":"2024-03-11T21:42:51.740477+0000","last_deep_scrub":"714'19","last_deep_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_clean_scrub_stamp":"2024-03-11T21:42:51.740477+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.872799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1633,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813568+0000","last_change":"2024-03-11T21:39:48.014842+0000","last_active":"2024-03-11T21:43:01.813568+0000","last_peered":"2024-03-11T21:43:01.813568+0000","last_clean":"2024-03-11T21:43:01.813568+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:43:01.813568+0000","last_undegraded":"2024-03-11T21:43:01.813568+0000","last_fullsized":"2024-03-11T21:43:01.813568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:48.014741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:39:48.014741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1651,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593934+0000","last_change":"2024-03-11T21:41:54.741470+0000","last_active":"2024-03-11T21:43:01.593934+0000","last_peered":"2024-03-11T21:43:01.593934+0000","last_clean":"2024-03-11T21:43:01.593934+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:43:01.593934+0000","last_undegraded":"2024-03-11T21:43:01.593934+0000","last_fullsized":"2024-03-11T21:43:01.593934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.741251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:20.722799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1622,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814740+0000","last_change":"2024-03-11T21:41:54.630605+0000","last_active":"2024-03-11T21:43:01.814740+0000","last_peered":"2024-03-11T21:43:01.814740+0000","last_clean":"2024-03-11T21:43:01.814740+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:43:01.814740+0000","last_undegraded":"2024-03-11T21:43:01.814740+0000","last_fullsized":"2024-03-11T21:43:01.814740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.630386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.630386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:17.496284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1478,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.717026+0000","last_change":"2024-03-11T21:41:31.510028+0000","last_active":"2024-03-11T21:43:01.717026+0000","last_peered":"2024-03-11T21:43:01.717026+0000","last_clean":"2024-03-11T21:43:01.717026+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:43:01.717026+0000","last_undegraded":"2024-03-11T21:43:01.717026+0000","last_fullsized":"2024-03-11T21:43:01.717026+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.509821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.509821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"389'18","reported_seq":1664,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430865+0000","last_change":"2024-03-11T21:40:28.925505+0000","last_active":"2024-03-11T21:43:02.430865+0000","last_peered":"2024-03-11T21:43:02.430865+0000","last_clean":"2024-03-11T21:43:02.430865+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:43:02.430865+0000","last_undegraded":"2024-03-11T21:43:02.430865+0000","last_fullsized":"2024-03-11T21:43:02.430865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'18","last_scrub_stamp":"2024-03-11T21:40:28.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.925267+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1632,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715995+0000","last_change":"2024-03-11T21:41:46.401421+0000","last_active":"2024-03-11T21:43:01.715995+0000","last_peered":"2024-03-11T21:43:01.715995+0000","last_clean":"2024-03-11T21:43:01.715995+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:43:01.715995+0000","last_undegraded":"2024-03-11T21:43:01.715995+0000","last_fullsized":"2024-03-11T21:43:01.715995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.401211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.401211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1628,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.429331+0000","last_change":"2024-03-11T21:41:53.275977+0000","last_active":"2024-03-11T21:43:02.429331+0000","last_peered":"2024-03-11T21:43:02.429331+0000","last_clean":"2024-03-11T21:43:02.429331+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:43:02.429331+0000","last_undegraded":"2024-03-11T21:43:02.429331+0000","last_fullsized":"2024-03-11T21:43:02.429331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.275834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.275834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:05.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1647,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.623027+0000","last_change":"2024-03-11T21:42:59.415876+0000","last_active":"2024-03-11T21:43:01.623027+0000","last_peered":"2024-03-11T21:43:01.623027+0000","last_clean":"2024-03-11T21:43:01.623027+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:43:01.623027+0000","last_undegraded":"2024-03-11T21:43:01.623027+0000","last_fullsized":"2024-03-11T21:43:01.623027+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.415650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.415650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.486892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1506,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.593092+0000","last_change":"2024-03-11T21:41:22.601767+0000","last_active":"2024-03-11T21:43:01.593092+0000","last_peered":"2024-03-11T21:43:01.593092+0000","last_clean":"2024-03-11T21:43:01.593092+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:43:01.593092+0000","last_undegraded":"2024-03-11T21:43:01.593092+0000","last_fullsized":"2024-03-11T21:43:01.593092+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.601690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.601690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"389'34","reported_seq":1699,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.430911+0000","last_change":"2024-03-11T21:41:51.290471+0000","last_active":"2024-03-11T21:43:02.430911+0000","last_peered":"2024-03-11T21:43:02.430911+0000","last_clean":"2024-03-11T21:43:02.430911+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:43:02.430911+0000","last_undegraded":"2024-03-11T21:43:02.430911+0000","last_fullsized":"2024-03-11T21:43:02.430911+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:41:51.290324+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.290324+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.691718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1635,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716842+0000","last_change":"2024-03-11T21:40:06.837164+0000","last_active":"2024-03-11T21:43:01.716842+0000","last_peered":"2024-03-11T21:43:01.716842+0000","last_clean":"2024-03-11T21:43:01.716842+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:43:01.716842+0000","last_undegraded":"2024-03-11T21:43:01.716842+0000","last_fullsized":"2024-03-11T21:43:01.716842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:06.836871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.836871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1624,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.715709+0000","last_change":"2024-03-11T21:40:04.809344+0000","last_active":"2024-03-11T21:43:01.715709+0000","last_peered":"2024-03-11T21:43:01.715709+0000","last_clean":"2024-03-11T21:43:01.715709+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:43:01.715709+0000","last_undegraded":"2024-03-11T21:43:01.715709+0000","last_fullsized":"2024-03-11T21:43:01.715709+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:04.809056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.809056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1634,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:02.428947+0000","last_change":"2024-03-11T21:41:47.226377+0000","last_active":"2024-03-11T21:43:02.428947+0000","last_peered":"2024-03-11T21:43:02.428947+0000","last_clean":"2024-03-11T21:43:02.428947+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:43:02.428947+0000","last_undegraded":"2024-03-11T21:43:02.428947+0000","last_fullsized":"2024-03-11T21:43:02.428947+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:47.226166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:47.226166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:13.255553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1478,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.716227+0000","last_change":"2024-03-11T21:40:02.830965+0000","last_active":"2024-03-11T21:43:01.716227+0000","last_peered":"2024-03-11T21:43:01.716227+0000","last_clean":"2024-03-11T21:43:01.716227+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:43:01.716227+0000","last_undegraded":"2024-03-11T21:43:01.716227+0000","last_fullsized":"2024-03-11T21:43:01.716227+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.830743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.830743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"389'24","reported_seq":1684,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.624109+0000","last_change":"2024-03-11T21:42:55.509855+0000","last_active":"2024-03-11T21:43:01.624109+0000","last_peered":"2024-03-11T21:43:01.624109+0000","last_clean":"2024-03-11T21:43:01.624109+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:43:01.624109+0000","last_undegraded":"2024-03-11T21:43:01.624109+0000","last_fullsized":"2024-03-11T21:43:01.624109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:42:55.509532+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.509532+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:58.493575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1635,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813950+0000","last_change":"2024-03-11T21:41:49.603979+0000","last_active":"2024-03-11T21:43:01.813950+0000","last_peered":"2024-03-11T21:43:01.813950+0000","last_clean":"2024-03-11T21:43:01.813950+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:43:01.813950+0000","last_undegraded":"2024-03-11T21:43:01.813950+0000","last_fullsized":"2024-03-11T21:43:01.813950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.603815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.603815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1497,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.581270+0000","last_change":"2024-03-11T21:40:20.395801+0000","last_active":"2024-03-11T21:42:59.581270+0000","last_peered":"2024-03-11T21:42:59.581270+0000","last_clean":"2024-03-11T21:42:59.581270+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:42:59.581270+0000","last_undegraded":"2024-03-11T21:42:59.581270+0000","last_fullsized":"2024-03-11T21:42:59.581270+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.395704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1656,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.591820+0000","last_change":"2024-03-11T21:41:33.725278+0000","last_active":"2024-03-11T21:43:01.591820+0000","last_peered":"2024-03-11T21:43:01.591820+0000","last_clean":"2024-03-11T21:43:01.591820+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:43:01.591820+0000","last_undegraded":"2024-03-11T21:43:01.591820+0000","last_fullsized":"2024-03-11T21:43:01.591820+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.725010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.725010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1355,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-03-11T21:42:59.582472+0000","last_change":"2024-03-11T21:41:54.179283+0000","last_active":"2024-03-11T21:42:59.582472+0000","last_peered":"2024-03-11T21:42:59.582472+0000","last_clean":"2024-03-11T21:42:59.582472+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:42:59.582472+0000","last_undegraded":"2024-03-11T21:42:59.582472+0000","last_fullsized":"2024-03-11T21:42:59.582472+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.179012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.179012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:03.476815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"389'20","reported_seq":1660,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.814231+0000","last_change":"2024-03-11T21:40:15.377800+0000","last_active":"2024-03-11T21:43:01.814231+0000","last_peered":"2024-03-11T21:43:01.814231+0000","last_clean":"2024-03-11T21:43:01.814231+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:43:01.814231+0000","last_undegraded":"2024-03-11T21:43:01.814231+0000","last_fullsized":"2024-03-11T21:43:01.814231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:40:15.377635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.377635+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1645,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.624116+0000","last_change":"2024-03-11T21:40:04.478158+0000","last_active":"2024-03-11T21:43:01.624116+0000","last_peered":"2024-03-11T21:43:01.624116+0000","last_clean":"2024-03-11T21:43:01.624116+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:43:01.624116+0000","last_undegraded":"2024-03-11T21:43:01.624116+0000","last_fullsized":"2024-03-11T21:43:01.624116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:40:04.477919+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.477919+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1661,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.590325+0000","last_change":"2024-03-11T21:41:27.668035+0000","last_active":"2024-03-11T21:43:01.590325+0000","last_peered":"2024-03-11T21:43:01.590325+0000","last_clean":"2024-03-11T21:43:01.590325+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:43:01.590325+0000","last_undegraded":"2024-03-11T21:43:01.590325+0000","last_fullsized":"2024-03-11T21:43:01.590325+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.667840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.667840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1620,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-03-11T21:43:01.813326+0000","last_change":"2024-03-11T21:42:56.738473+0000","last_active":"2024-03-11T21:43:01.813326+0000","last_peered":"2024-03-11T21:43:01.813326+0000","last_clean":"2024-03-11T21:43:01.813326+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:43:01.813326+0000","last_undegraded":"2024-03-11T21:43:01.813326+0000","last_fullsized":"2024-03-11T21:43:01.813326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.738379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.738379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.984522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":331,"num_pg":32,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":313,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":42,"num_object_clones":1,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":24,"num_read_kb":7,"num_write":62,"num_write_kb":3964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6500352,"data_stored":6500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":24,"acting":24,"num_store_stats":7},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1129,"num_read_kb":886,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1178,"num_read_kb":1173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841787,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123992,"kb_used_data":16428,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93621160,"statfs":{"total":95995035648,"available":95868067840,"internally_reserved":0,"allocated":16822272,"data_stored":11447402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168836,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107736,"kb_used_data":14060,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93637416,"statfs":{"total":95995035648,"available":95884713984,"internally_reserved":0,"allocated":14397440,"data_stored":9057206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352458,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54408,"kb_used_data":13992,"kb_used_omap":27,"kb_used_meta":40356,"kb_avail":93690744,"statfs":{"total":95995035648,"available":95939321856,"internally_reserved":0,"allocated":14327808,"data_stored":8990355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":41324873},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105580,"kb_used_data":16076,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93639572,"statfs":{"total":95995035648,"available":95886921728,"internally_reserved":0,"allocated":16461824,"data_stored":11114510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378926,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99712,"kb_used_data":14356,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93645440,"statfs":{"total":95995035648,"available":95892930560,"internally_reserved":0,"allocated":14700544,"data_stored":9353125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117282,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95820,"kb_used_data":14624,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93649332,"statfs":{"total":95995035648,"available":95896915968,"internally_reserved":0,"allocated":14974976,"data_stored":9635232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855640,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123656,"kb_used_data":16104,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93621496,"statfs":{"total":95995035648,"available":95868411904,"internally_reserved":0,"allocated":16490496,"data_stored":11110720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495886,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77756,"kb_used_data":14632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93667396,"statfs":{"total":95995035648,"available":95915413504,"internally_reserved":0,"allocated":14983168,"data_stored":9643401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2101248,"data_stored":2101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2105344,"data_stored":2105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":331,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:43:08.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.501 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:08.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:08.625 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:08.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- 172.21.15.143:0/4039511334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 msgr2=0x7f22f01022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 --2- 172.21.15.143:0/4039511334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f01022f0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009990 tx=0x7f22e4009ca0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- 172.21.15.143:0/4039511334 shutdown_connections 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 --2- 172.21.15.143:0/4039511334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f010ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 --2- 172.21.15.143:0/4039511334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0107ea0 0x7f22f0108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 --2- 172.21.15.143:0/4039511334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f01022f0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- 172.21.15.143:0/4039511334 >> 172.21.15.143:0/4039511334 conn(0x7f22f0078eb0 msgr2=0x7f22f00792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- 172.21.15.143:0/4039511334 shutdown_connections 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- 172.21.15.143:0/4039511334 wait complete. 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 librados: starting msgr at 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 librados: starting objecter 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 Processor -- start 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 -- start start 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 librados: setting wanted keys 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.726+0000 7f22f8563700 1 librados: calling monclient init 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f00740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0107ea0 0x7f22f0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0111620 con 0x7f22f0102830 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f01114a0 con 0x7f22f0107ea0 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f01117a0 con 0x7f22f0101e90 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f00740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f0074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48292/0 (socket says 172.21.15.143:48292) 2024-03-11T21:43:08.715 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 -- 172.21.15.143:0/4279324816 learned_addr learned my addr 172.21.15.143:0/4279324816 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0107ea0 msgr2=0x7f22f0074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0107ea0 0x7f22f0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 msgr2=0x7f22f0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f01a6110 con 0x7f22f0101e90 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f0102830 0x7f22f0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f00740e0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ada0 tx=0x7f22e800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 10 client.?.objecter ms_handle_connect 0x7f22f0101e90 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8017020 con 0x7f22f0101e90 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f22e800ce00 con 0x7f22f0101e90 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01a6400 con 0x7f22f0101e90 2024-03-11T21:43:08.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8005280 con 0x7f22f0101e90 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 10 client.42101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f01a6990 con 0x7f22f0101e90 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f8563700 1 librados: init done 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f22d0005100 con 0x7f22f0101e90 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f22e800c980 con 0x7f22f0101e90 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f22d806f820 0x7f22d8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(864..864 src has 87..864) v4 ==== 10287+0+0 (secure 0 0 0) 0x7f22e808ee10 con 0x7f22f0101e90 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 osd_map(864..864 src has 87..864) v4 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 3 client.42101.objecter handle_osd_map got epochs [864,864] > 0 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 3 client.42101.objecter handle_osd_map decoding full epoch 864 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f22d806f820 0x7f22d8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 20 client.42101.objecter dump_active .. 0 homeless 2024-03-11T21:43:08.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v864) v1 ==== 54+0+16 (secure 0 0 0) 0x7f22e8021020 con 0x7f22f0101e90 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 1 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f22d0004730 0x7f22d0004b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 1 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 0 1 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 4 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 10 client.42101.objecter _send_command 1 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d0008b70 con 0x7f22d0004730 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f22d806f820 0x7f22d8071ce0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f22f00792d0 tx=0x7f22e401a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.730+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d806f820 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f22d0004730 0x7f22d0004b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0004730 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f22d0004730 0x7f22d0004b50 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d0008b70 con 0x7f22d0004730 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:43:08.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 1 = system:0 2024-03-11T21:43:08.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 4 2024-03-11T21:43:08.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.734+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 0 1 2024-03-11T21:43:08.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 2 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 2 2024-03-11T21:43:08.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d00053e0 osd=0 3 2024-03-11T21:43:08.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 0 2 2024-03-11T21:43:08.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 4 2024-03-11T21:43:08.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 10 client.42101.objecter _send_command 2 2024-03-11T21:43:08.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.750+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d00041e0 con 0x7f22d0004730 2024-03-11T21:43:08.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d00041e0 con 0x7f22d0004730 2024-03-11T21:43:08.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 4 2024-03-11T21:43:08.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 0 2 2024-03-11T21:43:08.739 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:43:08.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 3 2024-03-11T21:43:08.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f22d0002750 0x7f22d0002b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 1 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 4 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 10 client.42101.objecter _send_command 3 2024-03-11T21:43:08.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d00030d0 con 0x7f22d0002750 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f22d0002750 0x7f22d0002b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0002750 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f22d0002750 0x7f22d0002b90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d00030d0 con 0x7f22d0002750 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 3 = system:0 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 4 2024-03-11T21:43:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.754+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 1 3 2024-03-11T21:43:08.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:08.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.770+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 4 2024-03-11T21:43:08.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 4 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d00081c0 osd=1 3 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 1 4 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 4 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 10 client.42101.objecter _send_command 4 2024-03-11T21:43:08.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d0003ae0 con 0x7f22d0002750 2024-03-11T21:43:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:08.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d0003ae0 con 0x7f22d0002750 2024-03-11T21:43:08.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 4 2024-03-11T21:43:08.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 1 4 2024-03-11T21:43:08.760 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 5 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f22d000b240 0x7f22d000d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 5 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 2 5 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 4 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 10 client.42101.objecter _send_command 5 2024-03-11T21:43:08.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d0010490 con 0x7f22d000b240 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f22d000b240 0x7f22d000d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d000b240 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f22d000b240 0x7f22d000d630 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d0010490 con 0x7f22d000b240 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 5 = system:0 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 4 2024-03-11T21:43:08.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.774+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 2 5 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 6 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 6 2024-03-11T21:43:08.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0007c40 osd=2 3 2024-03-11T21:43:08.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 2 6 2024-03-11T21:43:08.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 4 2024-03-11T21:43:08.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 10 client.42101.objecter _send_command 6 2024-03-11T21:43:08.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d0003630 con 0x7f22d000b240 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d0003630 con 0x7f22d000b240 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 4 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 2 6 2024-03-11T21:43:08.780 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 7 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f22d0002060 0x7f22d000ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 7 2024-03-11T21:43:08.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 3 7 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 4 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 10 client.42101.objecter _send_command 7 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d000f3c0 con 0x7f22d0002060 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f22d0002060 0x7f22d000ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0002060 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f22d0002060 0x7f22d000ecf0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d000f3c0 con 0x7f22d0002060 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:43:08.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 7 = system:0 2024-03-11T21:43:08.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 4 2024-03-11T21:43:08.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.794+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 3 7 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 8 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 8 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0003880 osd=3 3 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 3 8 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 4 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 10 client.42101.objecter _send_command 8 2024-03-11T21:43:08.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d00015d0 con 0x7f22d0002060 2024-03-11T21:43:08.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d00015d0 con 0x7f22d0002060 2024-03-11T21:43:08.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 4 2024-03-11T21:43:08.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.814+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 3 8 2024-03-11T21:43:08.802 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:43:08.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 9 2024-03-11T21:43:08.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f22d0012520 0x7f22d0014910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 9 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 4 9 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 4 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 10 client.42101.objecter _send_command 9 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d0017770 con 0x7f22d0012520 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f22d0012520 0x7f22d0014910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0012520 2024-03-11T21:43:08.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f22d0012520 0x7f22d0014910 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d0017770 con 0x7f22d0012520 2024-03-11T21:43:08.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:43:08.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 9 = system:0 2024-03-11T21:43:08.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 4 2024-03-11T21:43:08.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.818+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 4 9 2024-03-11T21:43:08.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 10 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 10 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0001820 osd=4 3 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 4 10 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 4 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 10 client.42101.objecter _send_command 10 2024-03-11T21:43:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.834+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d0014e50 con 0x7f22d0012520 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d0014e50 con 0x7f22d0012520 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 4 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 4 10 2024-03-11T21:43:08.823 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 11 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f22d0015570 0x7f22d0017960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 11 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 5 11 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 4 2024-03-11T21:43:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 10 client.42101.objecter _send_command 11 2024-03-11T21:43:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d001a960 con 0x7f22d0015570 2024-03-11T21:43:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f22d0015570 0x7f22d0017960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0015570 2024-03-11T21:43:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.838+0000 7f22f5afe700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f22d0015570 0x7f22d0017960 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.842+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d001a960 con 0x7f22d0015570 2024-03-11T21:43:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.842+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:43:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.842+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 11 = system:0 2024-03-11T21:43:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.842+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 4 2024-03-11T21:43:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.842+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 5 11 2024-03-11T21:43:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 12 2024-03-11T21:43:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 12 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0014e50 osd=5 3 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 5 12 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 4 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 10 client.42101.objecter _send_command 12 2024-03-11T21:43:08.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.858+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d001a920 con 0x7f22d0015570 2024-03-11T21:43:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d001a920 con 0x7f22d0015570 2024-03-11T21:43:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 4 2024-03-11T21:43:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 5 12 2024-03-11T21:43:08.848 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:43:08.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 13 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f22d0019c20 0x7f22d001a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 13 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 6 13 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 4 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 10 client.42101.objecter _send_command 13 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d001dd00 con 0x7f22d0019c20 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f22d0019c20 0x7f22d001a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0019c20 2024-03-11T21:43:08.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.862+0000 7f22f6b00700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f22d0019c20 0x7f22d001a060 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.866+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d001dd00 con 0x7f22d0019c20 2024-03-11T21:43:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.866+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:43:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.866+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 13 = system:0 2024-03-11T21:43:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.866+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 4 2024-03-11T21:43:08.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.866+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 6 13 2024-03-11T21:43:08.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 14 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 14 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d001a920 osd=6 3 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 6 14 2024-03-11T21:43:08.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 4 2024-03-11T21:43:08.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 10 client.42101.objecter _send_command 14 2024-03-11T21:43:08.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.882+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d0017f40 con 0x7f22d0019c20 2024-03-11T21:43:08.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d0017f40 con 0x7f22d0019c20 2024-03-11T21:43:08.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 4 2024-03-11T21:43:08.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 6 14 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 15 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f22d0018660 0x7f22d0018ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 15 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 7 15 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 4 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 10 client.42101.objecter _send_command 15 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d0019320 con 0x7f22d0018660 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.886+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f22d0018660 0x7f22d0018ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22f62ff700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f22d0018660 0x7f22d0018ac0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:08.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 10 client.42101.objecter ms_handle_connect 0x7f22d0018660 2024-03-11T21:43:08.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22d0019320 con 0x7f22d0018660 2024-03-11T21:43:08.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:43:08.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 15 = system:0 2024-03-11T21:43:08.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 4 2024-03-11T21:43:08.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.890+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 7 15 2024-03-11T21:43:08.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 10 client.42101.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign -1 16 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 20 client.42101.objecter _get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_remove -1 16 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 20 client.42101.objecter get_session s=0x7f22d0017f40 osd=7 3 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 15 client.42101.objecter _session_command_op_assign 7 16 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 4 2024-03-11T21:43:08.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 10 client.42101.objecter _send_command 16 2024-03-11T21:43:08.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.906+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f22d0019320 con 0x7f22d0018660 2024-03-11T21:43:08.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ef7fe700 1 -- 172.21.15.143:0/4279324816 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f22d0019320 con 0x7f22d0018660 2024-03-11T21:43:08.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ef7fe700 10 client.42101.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:08.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ef7fe700 10 client.42101.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:08.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ef7fe700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 4 2024-03-11T21:43:08.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ef7fe700 15 client.42101.objecter _session_command_op_remove 7 16 2024-03-11T21:43:08.895 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 librados: watch_flush enter 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 librados: watch_flush exit 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.0 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f22d0004730 msgr2=0x7f22d0004b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f22d0004730 0x7f22d0004b50 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00053e0 osd=0 1 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.1 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f22d0002750 msgr2=0x7f22d0002b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f22d0002750 0x7f22d0002b90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d00081c0 osd=1 1 2024-03-11T21:43:08.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.2 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f22d000b240 msgr2=0x7f22d000d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f22d000b240 0x7f22d000d630 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0007c40 osd=2 1 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.3 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f22d0002060 msgr2=0x7f22d000ecf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f22d0002060 0x7f22d000ecf0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0003880 osd=3 1 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.4 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f22d0012520 msgr2=0x7f22d0014910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f22d0012520 0x7f22d0014910 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0001820 osd=4 1 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.5 2024-03-11T21:43:08.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f22d0015570 msgr2=0x7f22d0017960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f22d0015570 0x7f22d0017960 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0014e50 osd=5 1 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.6 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f22d0019c20 msgr2=0x7f22d001a060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f22d0019c20 0x7f22d001a060 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d001a920 osd=6 1 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter close_session for osd.7 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f22d0018660 msgr2=0x7f22d0018ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f22d0018660 0x7f22d0018ac0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter put_session s=0x7f22d0017f40 osd=7 1 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 20 client.42101.objecter shutdown clearing up homeless session... 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 10 client.42101.objecter successfully canceled tick 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22f6b00700 1 -- 172.21.15.143:0/4279324816 reap_dead start 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22f6b00700 1 -- 172.21.15.143:0/4279324816 reap_dead start 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f22d806f820 msgr2=0x7f22d8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f22d806f820 0x7f22d8071ce0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f22f00792d0 tx=0x7f22e401a040 comp rx=0 tx=0).stop 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22f6b00700 1 -- 172.21.15.143:0/4279324816 reap_dead start 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 msgr2=0x7f22f00740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 --2- 172.21.15.143:0/4279324816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22f0101e90 0x7f22f00740e0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ada0 tx=0x7f22e800c330 comp rx=0 tx=0).stop 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22f6b00700 1 -- 172.21.15.143:0/4279324816 reap_dead start 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22f6b00700 1 -- 172.21.15.143:0/4279324816 reap_dead start 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 shutdown_connections 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 >> 172.21.15.143:0/4279324816 conn(0x7f22f0078eb0 msgr2=0x7f22f01006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 shutdown_connections 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 -- 172.21.15.143:0/4279324816 wait complete. 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:08.910+0000 7f22ed7fa700 1 librados: shutdown 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:08.900 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:08.901 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:08.901 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:08.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.000 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.641660+0000 mgr.y (mgr.14550) 780 : cluster 0 pgmap v1345: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:43:09.000 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.657691+0000 osd.7 (osd.7) 452 : cluster 0 6.12 scrub starts 2024-03-11T21:43:09.000 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.660082+0000 osd.7 (osd.7) 453 : cluster 0 6.12 scrub ok 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.787048+0000 osd.3 (osd.3) 463 : cluster 0 2.1b scrub starts 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.788490+0000 osd.3 (osd.3) 464 : cluster 0 2.1b scrub ok 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.906888+0000 osd.1 (osd.1) 257 : cluster 0 2.6 scrub starts 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:07.910013+0000 osd.1 (osd.1) 258 : cluster 0 2.6 scrub ok 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.618352+0000 osd.4 (osd.4) 441 : cluster 0 5.18 scrub starts 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.620434+0000 osd.4 (osd.4) 442 : cluster 0 5.18 scrub ok 2024-03-11T21:43:09.001 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:08 smithi203 bash[23654]: audit 2024-03-11T21:43:08.734078+0000 mon.c (mon.2) 1335 : audit 0 from='client.? 172.21.15.143:0/4279324816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:09.014 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:09.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:09.145 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.641660+0000 mgr.y (mgr.14550) 780 : cluster 0 pgmap v1345: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.657691+0000 osd.7 (osd.7) 452 : cluster 0 6.12 scrub starts 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.660082+0000 osd.7 (osd.7) 453 : cluster 0 6.12 scrub ok 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.787048+0000 osd.3 (osd.3) 463 : cluster 0 2.1b scrub starts 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.788490+0000 osd.3 (osd.3) 464 : cluster 0 2.1b scrub ok 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.906888+0000 osd.1 (osd.1) 257 : cluster 0 2.6 scrub starts 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:07.910013+0000 osd.1 (osd.1) 258 : cluster 0 2.6 scrub ok 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.618352+0000 osd.4 (osd.4) 441 : cluster 0 5.18 scrub starts 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.620434+0000 osd.4 (osd.4) 442 : cluster 0 5.18 scrub ok 2024-03-11T21:43:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[21512]: audit 2024-03-11T21:43:08.734078+0000 mon.c (mon.2) 1335 : audit 0 from='client.? 172.21.15.143:0/4279324816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.641660+0000 mgr.y (mgr.14550) 780 : cluster 0 pgmap v1345: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.657691+0000 osd.7 (osd.7) 452 : cluster 0 6.12 scrub starts 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.660082+0000 osd.7 (osd.7) 453 : cluster 0 6.12 scrub ok 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.787048+0000 osd.3 (osd.3) 463 : cluster 0 2.1b scrub starts 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.788490+0000 osd.3 (osd.3) 464 : cluster 0 2.1b scrub ok 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.906888+0000 osd.1 (osd.1) 257 : cluster 0 2.6 scrub starts 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:07.910013+0000 osd.1 (osd.1) 258 : cluster 0 2.6 scrub ok 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.618352+0000 osd.4 (osd.4) 441 : cluster 0 5.18 scrub starts 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.620434+0000 osd.4 (osd.4) 442 : cluster 0 5.18 scrub ok 2024-03-11T21:43:09.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:08 smithi143 bash[29284]: audit 2024-03-11T21:43:08.734078+0000 mon.c (mon.2) 1335 : audit 0 from='client.? 172.21.15.143:0/4279324816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:09.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:09.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:09.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.402 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:09.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:09.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:09.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:09.661 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:09.753 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:09.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.791 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:09.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:09.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:10.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.337499+0000 osd.0 (osd.0) 454 : cluster 0 2.1f scrub starts 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.339932+0000 osd.0 (osd.0) 455 : cluster 0 2.1f scrub ok 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.953830+0000 osd.1 (osd.1) 259 : cluster 0 1.3 scrub starts 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:08.955925+0000 osd.1 (osd.1) 260 : cluster 0 1.3 scrub ok 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: audit 2024-03-11T21:43:09.085778+0000 mgr.y (mgr.14550) 781 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.459858+0000 osd.5 (osd.5) 471 : cluster 0 3.16 scrub starts 2024-03-11T21:43:10.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:09 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.463695+0000 osd.5 (osd.5) 472 : cluster 0 3.16 scrub ok 2024-03-11T21:43:10.052 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:10.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.337499+0000 osd.0 (osd.0) 454 : cluster 0 2.1f scrub starts 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.339932+0000 osd.0 (osd.0) 455 : cluster 0 2.1f scrub ok 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.953830+0000 osd.1 (osd.1) 259 : cluster 0 1.3 scrub starts 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:08.955925+0000 osd.1 (osd.1) 260 : cluster 0 1.3 scrub ok 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: audit 2024-03-11T21:43:09.085778+0000 mgr.y (mgr.14550) 781 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.459858+0000 osd.5 (osd.5) 471 : cluster 0 3.16 scrub starts 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.463695+0000 osd.5 (osd.5) 472 : cluster 0 3.16 scrub ok 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.337499+0000 osd.0 (osd.0) 454 : cluster 0 2.1f scrub starts 2024-03-11T21:43:10.167 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.339932+0000 osd.0 (osd.0) 455 : cluster 0 2.1f scrub ok 2024-03-11T21:43:10.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.953830+0000 osd.1 (osd.1) 259 : cluster 0 1.3 scrub starts 2024-03-11T21:43:10.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:08.955925+0000 osd.1 (osd.1) 260 : cluster 0 1.3 scrub ok 2024-03-11T21:43:10.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: audit 2024-03-11T21:43:09.085778+0000 mgr.y (mgr.14550) 781 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:10.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.459858+0000 osd.5 (osd.5) 471 : cluster 0 3.16 scrub starts 2024-03-11T21:43:10.168 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.463695+0000 osd.5 (osd.5) 472 : cluster 0 3.16 scrub ok 2024-03-11T21:43:10.182 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:10.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:10.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.440 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:10.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:10.570 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:10.673 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:10.700 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:10.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:10.831 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:10.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.307562+0000 osd.0 (osd.0) 456 : cluster 0 39.14 scrub starts 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.309024+0000 osd.0 (osd.0) 457 : cluster 0 39.14 scrub ok 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.363216+0000 osd.6 (osd.6) 375 : cluster 0 1.6 scrub starts 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.365326+0000 osd.6 (osd.6) 376 : cluster 0 1.6 scrub ok 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.644189+0000 mgr.y (mgr.14550) 782 : cluster 0 pgmap v1346: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.962341+0000 osd.1 (osd.1) 261 : cluster 0 2.a scrub starts 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:09.964023+0000 osd.1 (osd.1) 262 : cluster 0 2.a scrub ok 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.491667+0000 osd.5 (osd.5) 473 : cluster 0 2.3 scrub starts 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.493868+0000 osd.5 (osd.5) 474 : cluster 0 2.3 scrub ok 2024-03-11T21:43:10.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.307562+0000 osd.0 (osd.0) 456 : cluster 0 39.14 scrub starts 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.309024+0000 osd.0 (osd.0) 457 : cluster 0 39.14 scrub ok 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.363216+0000 osd.6 (osd.6) 375 : cluster 0 1.6 scrub starts 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.365326+0000 osd.6 (osd.6) 376 : cluster 0 1.6 scrub ok 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.644189+0000 mgr.y (mgr.14550) 782 : cluster 0 pgmap v1346: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.962341+0000 osd.1 (osd.1) 261 : cluster 0 2.a scrub starts 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:09.964023+0000 osd.1 (osd.1) 262 : cluster 0 2.a scrub ok 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.491667+0000 osd.5 (osd.5) 473 : cluster 0 2.3 scrub starts 2024-03-11T21:43:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:10 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.493868+0000 osd.5 (osd.5) 474 : cluster 0 2.3 scrub ok 2024-03-11T21:43:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:11.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.307562+0000 osd.0 (osd.0) 456 : cluster 0 39.14 scrub starts 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.309024+0000 osd.0 (osd.0) 457 : cluster 0 39.14 scrub ok 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.363216+0000 osd.6 (osd.6) 375 : cluster 0 1.6 scrub starts 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.365326+0000 osd.6 (osd.6) 376 : cluster 0 1.6 scrub ok 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.644189+0000 mgr.y (mgr.14550) 782 : cluster 0 pgmap v1346: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:11.072 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.962341+0000 osd.1 (osd.1) 261 : cluster 0 2.a scrub starts 2024-03-11T21:43:11.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:09.964023+0000 osd.1 (osd.1) 262 : cluster 0 2.a scrub ok 2024-03-11T21:43:11.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.491667+0000 osd.5 (osd.5) 473 : cluster 0 2.3 scrub starts 2024-03-11T21:43:11.073 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:10 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.493868+0000 osd.5 (osd.5) 474 : cluster 0 2.3 scrub ok 2024-03-11T21:43:11.087 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:11.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:11.219 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:11.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:11.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:11.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:43:11.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:11.475 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:11.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:11.598 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:11.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:11.722 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- 172.21.15.143:0/2599177778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8104c20 msgr2=0x7f2ec8108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/2599177778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8104c20 0x7f2ec8108c70 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0099c0 tx=0x7f2ebc009cd0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- 172.21.15.143:0/2599177778 shutdown_connections 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/2599177778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8109350 0x7f2ec8110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/2599177778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/2599177778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8104c20 0x7f2ec8108c70 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- 172.21.15.143:0/2599177778 >> 172.21.15.143:0/2599177778 conn(0x7f2ec80ffb00 msgr2=0x7f2ec8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:11.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- 172.21.15.143:0/2599177778 shutdown_connections 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- 172.21.15.143:0/2599177778 wait complete. 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 librados: starting msgr at 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 librados: starting objecter 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 Processor -- start 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 -- start start 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 librados: setting wanted keys 2024-03-11T21:43:11.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.798+0000 7f2ecf49f700 1 librados: calling monclient init 2024-03-11T21:43:11.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8104c20 0x7f2ec8076830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8109350 0x7f2ec8076d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec81138b0 con 0x7f2ec8109350 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec8113730 con 0x7f2ec8104c20 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec8113a30 con 0x7f2ec8104270 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecda3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8109350 0x7f2ec8076d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:52554/0 (socket says 172.21.15.143:52554) 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 -- 172.21.15.143:0/4150716118 learned_addr learned my addr 172.21.15.143:0/4150716118 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecca3a700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8104c20 0x7f2ec8076830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 -- 172.21.15.143:0/4150716118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8104c20 msgr2=0x7f2ec8076830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8104c20 0x7f2ec8076830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 -- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8109350 msgr2=0x7f2ec8076d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8109350 0x7f2ec8076d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 -- 172.21.15.143:0/4150716118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec81aa540 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecd23b700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000c3a0 tx=0x7f2ec000c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 10 client.?.objecter ms_handle_connect 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00090d0 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2ec0009270 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec81aa7d0 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0004690 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 10 client.42107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec81aace0 con 0x7f2ec8104270 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 librados: init done 2024-03-11T21:43:11.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec8105750 con 0x7f2ec8104270 2024-03-11T21:43:11.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2ec001a020 con 0x7f2ec8104270 2024-03-11T21:43:11.790 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 0x7f2eb4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(864..864 src has 87..864) v4 ==== 10287+0+0 (secure 0 0 0) 0x7f2ec008e730 con 0x7f2ec8104270 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 10 client.42107.objecter ms_dispatch 0x7f2ec8001550 osd_map(864..864 src has 87..864) v4 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 3 client.42107.objecter handle_osd_map got epochs [864,864] > 0 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 3 client.42107.objecter handle_osd_map decoding full epoch 864 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2ecca3a700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 0x7f2eb4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.802+0000 7f2eba7fc700 20 client.42107.objecter dump_active .. 0 homeless 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.806+0000 7f2eba7fc700 10 client.42107.objecter ms_handle_connect 0x7f2eb406f820 2024-03-11T21:43:11.791 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.806+0000 7f2ecca3a700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 0x7f2eb4071ce0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc005940 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:11.794 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:11.810+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2ec0010040 con 0x7f2ec8104270 2024-03-11T21:43:11.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:11.849 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:11.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:11.979 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:12.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.022+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2ec81aaf10 con 0x7f2ec8104270 2024-03-11T21:43:12.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.108 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:12.194 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.329351+0000 osd.0 (osd.0) 458 : cluster 0 6.16 scrub starts 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.331119+0000 osd.0 (osd.0) 459 : cluster 0 6.16 scrub ok 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.966136+0000 osd.1 (osd.1) 263 : cluster 0 5.19 scrub starts 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:10.968593+0000 osd.1 (osd.1) 264 : cluster 0 5.19 scrub ok 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.523085+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.524708+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: audit 2024-03-11T21:43:11.637019+0000 mon.c (mon.2) 1336 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: audit 2024-03-11T21:43:11.637535+0000 mon.a (mon.0) 3882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.643134+0000 osd.4 (osd.4) 443 : cluster 0 6.e scrub starts 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.645874+0000 osd.4 (osd.4) 444 : cluster 0 6.e scrub ok 2024-03-11T21:43:12.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.329351+0000 osd.0 (osd.0) 458 : cluster 0 6.16 scrub starts 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.331119+0000 osd.0 (osd.0) 459 : cluster 0 6.16 scrub ok 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.966136+0000 osd.1 (osd.1) 263 : cluster 0 5.19 scrub starts 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:10.968593+0000 osd.1 (osd.1) 264 : cluster 0 5.19 scrub ok 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.523085+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.524708+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: audit 2024-03-11T21:43:11.637019+0000 mon.c (mon.2) 1336 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: audit 2024-03-11T21:43:11.637535+0000 mon.a (mon.0) 3882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.643134+0000 osd.4 (osd.4) 443 : cluster 0 6.e scrub starts 2024-03-11T21:43:12.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:11 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.645874+0000 osd.4 (osd.4) 444 : cluster 0 6.e scrub ok 2024-03-11T21:43:12.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.329351+0000 osd.0 (osd.0) 458 : cluster 0 6.16 scrub starts 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.331119+0000 osd.0 (osd.0) 459 : cluster 0 6.16 scrub ok 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.966136+0000 osd.1 (osd.1) 263 : cluster 0 5.19 scrub starts 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:10.968593+0000 osd.1 (osd.1) 264 : cluster 0 5.19 scrub ok 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.523085+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.524708+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-03-11T21:43:12.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: audit 2024-03-11T21:43:11.637019+0000 mon.c (mon.2) 1336 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: audit 2024-03-11T21:43:11.637535+0000 mon.a (mon.0) 3882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:12.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.643134+0000 osd.4 (osd.4) 443 : cluster 0 6.e scrub starts 2024-03-11T21:43:12.224 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:11 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.645874+0000 osd.4 (osd.4) 444 : cluster 0 6.e scrub ok 2024-03-11T21:43:12.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:12.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.370 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:12.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:12.500 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:12.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.598+0000 7f2eba7fc700 1 -- 172.21.15.143:0/4150716118 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v865) v1 ==== 101+0+0 (secure 0 0 0) 0x7f2ec005b0f0 con 0x7f2ec8104270 2024-03-11T21:43:12.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 10 librados: watch_flush enter 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 10 librados: watch_flush exit 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 20 client.42107.objecter shutdown clearing up homeless session... 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 10 client.42107.objecter successfully canceled tick 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 msgr2=0x7f2eb4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 0x7f2eb4071ce0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc005940 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 msgr2=0x7f2ec80762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000c3a0 tx=0x7f2ec000c6b0 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 shutdown_connections 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2eb406f820 0x7f2eb4071ce0 unknown :-1 s=CLOSED pgs=3461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec8109350 0x7f2ec8076d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec8104270 0x7f2ec80762f0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 --2- 172.21.15.143:0/4150716118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec8104c20 0x7f2ec8076830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 >> 172.21.15.143:0/4150716118 conn(0x7f2ec80ffb00 msgr2=0x7f2ec8100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:12.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 shutdown_connections 2024-03-11T21:43:12.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 -- 172.21.15.143:0/4150716118 wait complete. 2024-03-11T21:43:12.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:12.606+0000 7f2ecf49f700 1 librados: shutdown 2024-03-11T21:43:12.592 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:43:12.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:12.629 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:12.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:12.753 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:12.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:12.885 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:12.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:13.015 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:13.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.367712+0000 osd.6 (osd.6) 377 : cluster 0 313.3s0 scrub starts 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.370304+0000 osd.6 (osd.6) 378 : cluster 0 313.3 scrub ok 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.645020+0000 mgr.y (mgr.14550) 783 : cluster 0 pgmap v1347: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.700271+0000 osd.3 (osd.3) 465 : cluster 0 3.8 scrub starts 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:11.702294+0000 osd.3 (osd.3) 466 : cluster 0 3.8 scrub ok 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.026172+0000 mon.c (mon.2) 1337 : audit 1 from='client.? 172.21.15.143:0/4150716118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.130 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.026710+0000 mon.a (mon.0) 3883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.489263+0000 osd.2 (osd.2) 405 : cluster 0 2.c scrub starts 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.491037+0000 osd.2 (osd.2) 406 : cluster 0 2.c scrub ok 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.581852+0000 mon.a (mon.0) 3884 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.592044+0000 mon.a (mon.0) 3885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.592301+0000 mon.a (mon.0) 3886 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.601645+0000 mon.a (mon.0) 3887 : cluster 0 osdmap e865: 8 total, 8 up, 8 in 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.605862+0000 mon.c (mon.2) 1338 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: audit 2024-03-11T21:43:12.607521+0000 mon.a (mon.0) 3888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.637249+0000 osd.4 (osd.4) 445 : cluster 0 6.1a scrub starts 2024-03-11T21:43:13.131 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:12 smithi203 bash[23654]: cluster 2024-03-11T21:43:12.639788+0000 osd.4 (osd.4) 446 : cluster 0 6.1a scrub ok 2024-03-11T21:43:13.145 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.367712+0000 osd.6 (osd.6) 377 : cluster 0 313.3s0 scrub starts 2024-03-11T21:43:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.370304+0000 osd.6 (osd.6) 378 : cluster 0 313.3 scrub ok 2024-03-11T21:43:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.645020+0000 mgr.y (mgr.14550) 783 : cluster 0 pgmap v1347: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.700271+0000 osd.3 (osd.3) 465 : cluster 0 3.8 scrub starts 2024-03-11T21:43:13.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:11.702294+0000 osd.3 (osd.3) 466 : cluster 0 3.8 scrub ok 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.026172+0000 mon.c (mon.2) 1337 : audit 1 from='client.? 172.21.15.143:0/4150716118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.026710+0000 mon.a (mon.0) 3883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.489263+0000 osd.2 (osd.2) 405 : cluster 0 2.c scrub starts 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.491037+0000 osd.2 (osd.2) 406 : cluster 0 2.c scrub ok 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.581852+0000 mon.a (mon.0) 3884 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.592044+0000 mon.a (mon.0) 3885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.592301+0000 mon.a (mon.0) 3886 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.601645+0000 mon.a (mon.0) 3887 : cluster 0 osdmap e865: 8 total, 8 up, 8 in 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.605862+0000 mon.c (mon.2) 1338 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: audit 2024-03-11T21:43:12.607521+0000 mon.a (mon.0) 3888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.637249+0000 osd.4 (osd.4) 445 : cluster 0 6.1a scrub starts 2024-03-11T21:43:13.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[21512]: cluster 2024-03-11T21:43:12.639788+0000 osd.4 (osd.4) 446 : cluster 0 6.1a scrub ok 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.367712+0000 osd.6 (osd.6) 377 : cluster 0 313.3s0 scrub starts 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.370304+0000 osd.6 (osd.6) 378 : cluster 0 313.3 scrub ok 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.645020+0000 mgr.y (mgr.14550) 783 : cluster 0 pgmap v1347: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.700271+0000 osd.3 (osd.3) 465 : cluster 0 3.8 scrub starts 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:11.702294+0000 osd.3 (osd.3) 466 : cluster 0 3.8 scrub ok 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.026172+0000 mon.c (mon.2) 1337 : audit 1 from='client.? 172.21.15.143:0/4150716118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.026710+0000 mon.a (mon.0) 3883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.489263+0000 osd.2 (osd.2) 405 : cluster 0 2.c scrub starts 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.491037+0000 osd.2 (osd.2) 406 : cluster 0 2.c scrub ok 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.581852+0000 mon.a (mon.0) 3884 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.592044+0000 mon.a (mon.0) 3885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:13.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.592301+0000 mon.a (mon.0) 3886 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:13.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.601645+0000 mon.a (mon.0) 3887 : cluster 0 osdmap e865: 8 total, 8 up, 8 in 2024-03-11T21:43:13.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.605862+0000 mon.c (mon.2) 1338 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: audit 2024-03-11T21:43:12.607521+0000 mon.a (mon.0) 3888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:13.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.637249+0000 osd.4 (osd.4) 445 : cluster 0 6.1a scrub starts 2024-03-11T21:43:13.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:12 smithi143 bash[29284]: cluster 2024-03-11T21:43:12.639788+0000 osd.4 (osd.4) 446 : cluster 0 6.1a scrub ok 2024-03-11T21:43:13.247 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:13.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:13.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:13.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:13.532 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:13.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:13.656 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:13.658 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:43:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:13.744 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:15 314.2 314:40000000:%01:::head [pgnls start_epoch 692] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e692)", 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 214.18781742100001, 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086210539999999995, 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.745 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 15 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567520+0000", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8760000000000001e-06 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567534+0000", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3526999999999999e-05 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567557+0000", 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3102e-05 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567574+0000", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7799e-05 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567706+0000", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013165599999999999 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567717+0000", 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.119e-05 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.575810+0000", 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0080927120000000002 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.575903+0000", 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.289e-05 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.576138+0000", 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000235302 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.0s2 697/688 ECSubWrite(tid=2186, reqid=client.41921.0:108, at_version=697'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 209.24640553099999, 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019466730000000002, 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:43:13.750 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508931+0000", 2024-03-11T21:43:13.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.917e-06 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509226+0000", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029490400000000002 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509351+0000", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000124769 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509386+0000", 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5311999999999997e-05 2024-03-11T21:43:13.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509542+0000", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015589099999999999 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509566+0000", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4437e-05 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.528307+0000", 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018741203000000001 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.528396+0000", 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8296999999999998e-05 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:258 316.7 316:e031f6fd:test-rados-api-smithi143-71773-117::bar:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e712)", 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 182.181642407, 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021393550000000001, 2024-03-11T21:43:13.754 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 258 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:43:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573695+0000", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1329999999999999e-06 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573710+0000", 2024-03-11T21:43:13.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5079000000000001e-05 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573745+0000", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4885999999999998e-05 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573773+0000", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7919000000000002e-05 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573933+0000", 2024-03-11T21:43:13.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015959500000000001 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573949+0000", 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6443000000000002e-05 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.576634+0000", 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026843209999999999 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577229+0000", 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00059544800000000005 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.590928+0000", 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013698988 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.591095+0000", 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016727300000000001 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594119+0000", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0030232950000000001 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594197+0000", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7943999999999998e-05 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594438+0000", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024140500000000001 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594500+0000", 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1722000000000003e-05 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.595086+0000", 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00058609899999999997 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.6s1 718/688 ECSubWrite(tid=2189, reqid=client.41921.0:368, at_version=718'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 176.185858177, 2024-03-11T21:43:13.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014535064, 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569478+0000", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569541+0000", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3212000000000004e-05 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569570+0000", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8343e-05 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569594+0000", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4252000000000001e-05 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569754+0000", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000160512 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569777+0000", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3034999999999999e-05 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.583691+0000", 2024-03-11T21:43:13.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013913576 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.584011+0000", 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032058400000000001 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(osd.6.0:797 317.1e e723/717 317:7b8f8022:test-rados-api-smithi143-71773-119::foo:head v 723'1, mlcod=0'0)", 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.21545229899999, 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073935069999999997, 2024-03-11T21:43:13.765 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539883+0000", 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7700000000000003e-07 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539895+0000", 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1646e-05 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539925+0000", 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9952000000000001e-05 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539945+0000", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9786999999999999e-05 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.540089+0000", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014464499999999999 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.540144+0000", 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4528999999999999e-05 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.547153+0000", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007008841 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.547276+0000", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012323 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.41921.0:805 320.19 e748/743 320:98ec0ffc:test-rados-api-smithi143-71773-125::foo:4 v 748'3, mlcod=748'1)", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:43:13.769 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 146.10897086400001, 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010667311000000001, 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2699999999999999e-07 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646371+0000", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6209999999999998e-06 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646382+0000", 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1248e-05 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646390+0000", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1749999999999999e-06 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646452+0000", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2311999999999996e-05 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646483+0000", 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1106999999999999e-05 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656894+0000", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010410815 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.657031+0000", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013730599999999999 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:808 320.2 320:423c9130:test-rados-api-smithi143-71773-125::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e748)", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 146.096518769, 2024-03-11T21:43:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019173852000000002, 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 808 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658818+0000", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.951e-06 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658826+0000", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8599999999999993e-06 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658843+0000", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7813999999999999e-05 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658852+0000", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8019999999999998e-06 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658919+0000", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6619999999999996e-05 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.659154+0000", 2024-03-11T21:43:13.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000234749 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.663498+0000", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043444169999999997 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.663669+0000", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017117799999999999 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.674712+0000", 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011042545000000001 2024-03-11T21:43:13.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.674871+0000", 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015880199999999999 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.677990+0000", 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0031191140000000001 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.778 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:809 320.2 320:423c9130:test-rados-api-smithi143-71773-125::bar:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e748)", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 146.07706981199999, 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007111717, 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 809 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678266+0000", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.717e-06 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678276+0000", 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7189999999999996e-06 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678306+0000", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0035999999999998e-05 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678322+0000", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5599000000000001e-05 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678503+0000", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018142899999999999 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678853+0000", 2024-03-11T21:43:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034932300000000002 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.681568+0000", 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002714958 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.681729+0000", 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016142899999999999 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.682222+0000", 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049319800000000003 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.682385+0000", 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016276600000000001 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.685376+0000", 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029915430000000002 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1041 321.c 321:30000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 135.15451785499999, 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0074456820000000003, 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1041 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600820+0000", 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.18e-06 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600860+0000", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0034999999999999e-05 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600898+0000", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8108999999999997e-05 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600917+0000", 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8692999999999999e-05 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.601061+0000", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014444300000000001 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.601072+0000", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1232e-05 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608051+0000", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069788489999999996 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608109+0000", 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8149000000000002e-05 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608262+0000", 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015299199999999999 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1232 322.4 322:20000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 122.13124945600001, 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007980282, 2024-03-11T21:43:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1232 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624089+0000", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9500000000000003e-06 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624110+0000", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0948999999999999e-05 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624148+0000", 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7795e-05 2024-03-11T21:43:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624168+0000", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0400000000000001e-05 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624313+0000", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000145036 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624324+0000", 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0706e-05 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.631717+0000", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073927020000000001 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.631813+0000", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5853999999999994e-05 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.632065+0000", 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025289000000000003 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1682 325.11 325:8f82c992:test-rados-api-smithi143-71773-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e797)", 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 95.173150704999998, 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010991031, 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1682 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582186+0000", 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4870000000000001e-06 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582201+0000", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4521e-05 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582222+0000", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1540999999999999e-05 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582234+0000", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1651000000000001e-05 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582335+0000", 2024-03-11T21:43:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100454 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582578+0000", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024389600000000001 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587764+0000", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051852089999999997 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587864+0000", 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100305 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587997+0000", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013257399999999999 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.588663+0000", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00066632400000000004 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.590973+0000", 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023103009999999998 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593057+0000", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020840500000000001 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593124+0000", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6678000000000001e-05 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593175+0000", 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1039999999999999e-05 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1684 325.11 325:8f82c992:test-rados-api-smithi143-71773-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e797)", 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:38.593677+0000", 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 95.161657109000004, 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006716044, 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1684 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593677+0000", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593677+0000", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593678+0000", 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3399999999999998e-07 2024-03-11T21:43:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593683+0000", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2569999999999998e-06 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593692+0000", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7860000000000002e-06 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593697+0000", 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0030000000000002e-06 2024-03-11T21:43:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593953+0000", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025588199999999997 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593980+0000", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6696e-05 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.600247+0000", 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062676479999999998 2024-03-11T21:43:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.600313+0000", 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5213000000000001e-05 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.600394+0000", 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0824999999999999e-05 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.4,QREQUEST,e:804,from:1,reservation_nonce:11)", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.472756318999998, 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0068787520000000001, 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282581+0000", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1070000000000001e-06 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282596+0000", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5043e-05 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282633+0000", 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6748999999999997e-05 2024-03-11T21:43:13.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282651+0000", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8122999999999999e-05 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282813+0000", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016193199999999999 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282825+0000", 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1823e-05 2024-03-11T21:43:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.288552+0000", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057266460000000002 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.288592+0000", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0519000000000003e-05 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.289457+0000", 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00086481000000000004 2024-03-11T21:43:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1794 326.12 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.181573008000001, 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.037717492999999998, 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1794 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573765+0000", 2024-03-11T21:43:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3270000000000001e-06 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579409+0000", 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056436849999999998 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579451+0000", 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2525999999999999e-05 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579473+0000", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1656000000000001e-05 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579659+0000", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018667299999999999 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580270+0000", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00061076700000000002 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590053+0000", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0097825080000000005 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590326+0000", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027315 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590671+0000", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034527100000000002 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.608122+0000", 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017450514 2024-03-11T21:43:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.608268+0000", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014647799999999999 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611190+0000", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029218650000000001 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611353+0000", 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000162979 2024-03-11T21:43:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611479+0000", 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012609399999999999 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1795 326.12 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.143613289000001, 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026799703000000001, 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1795 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611724+0000", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.807e-06 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611739+0000", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4620000000000001e-05 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611763+0000", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4411999999999999e-05 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611779+0000", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5866999999999999e-05 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611918+0000", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013868600000000001 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612104+0000", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000186338 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638186+0000", 2024-03-11T21:43:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026081984999999999 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638383+0000", 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000196868 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638521+0000", 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013812000000000001 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1830 326.0 326:00000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 85.172426995999999, 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071741649999999997, 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1830 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582911+0000", 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.783e-06 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582931+0000", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9525999999999999e-05 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582972+0000", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1118999999999997e-05 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582993+0000", 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0740000000000001e-05 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.583163+0000", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016989800000000001 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.583181+0000", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8077e-05 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.589650+0000", 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064692509999999996 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.589754+0000", 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010387699999999999 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.590082+0000", 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032789400000000002 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2159 329.16 329:68000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:43:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:30.616650+0000", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 43.138684345999998, 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066874409999999997, 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2159 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616650+0000", 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616650+0000", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616654+0000", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.596e-06 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616671+0000", 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7295999999999998e-05 2024-03-11T21:43:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616710+0000", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8463000000000001e-05 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616728+0000", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8736000000000002e-05 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616810+0000", 2024-03-11T21:43:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1527999999999996e-05 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.616815+0000", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7369999999999997e-06 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.622912+0000", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060976959999999997 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.623001+0000", 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8268000000000003e-05 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.623338+0000", 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00033712100000000001 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.4s2 847/688 ECSubWrite(tid=912, reqid=client.41921.0:2312, at_version=847'20, trim_to=0'0, roll_forward_to=833'19))", 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 38.003559907000003, 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016333995, 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7300000000000001e-07 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751820+0000", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4733e-05 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751846+0000", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5893000000000001e-05 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751870+0000", 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4414999999999999e-05 2024-03-11T21:43:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.753501+0000", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016312239999999999 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.753526+0000", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4986000000000001e-05 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.768033+0000", 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01450655 2024-03-11T21:43:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.768109+0000", 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5721000000000002e-05 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14427.0:4141 3.13 e864/78 3:cc909b6b:::obj_delete_at_hint.0000000106:head v 864'27, mlcod=864'25)", 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 2.1643298, 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011996013999999999, 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591006+0000", 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2799999999999995e-07 2024-03-11T21:43:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591011+0000", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.801e-06 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591023+0000", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1243999999999999e-05 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591029+0000", 2024-03-11T21:43:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591091+0000", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2040999999999998e-05 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591110+0000", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.931e-05 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.602922+0000", 2024-03-11T21:43:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011811991000000001 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603001+0000", 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8938999999999998e-05 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4150 3.3 3:c35966f0:::obj_delete_at_hint.0000000109:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 2.121781328, 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011028938, 2024-03-11T21:43:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4150 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:43:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633554+0000", 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2799999999999995e-07 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633559+0000", 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6580000000000001e-06 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633567+0000", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4989999999999994e-06 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633572+0000", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6349999999999997e-06 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633607+0000", 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5110000000000001e-05 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633657+0000", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0251999999999999e-05 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633765+0000", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108041 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.634713+0000", 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00094732400000000004 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644210+0000", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094969240000000003 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644482+0000", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000272272 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644543+0000", 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1472999999999999e-05 2024-03-11T21:43:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644582+0000", 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8921999999999998e-05 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:43:13.842 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:43:13.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.862 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:13.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.987 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:14.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:14.118 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:14.194 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: cluster 2024-03-11T21:43:13.520000+0000 osd.5 (osd.5) 475 : cluster 0 2.16 scrub starts 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: cluster 2024-03-11T21:43:13.521472+0000 osd.5 (osd.5) 476 : cluster 0 2.16 scrub ok 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: audit 2024-03-11T21:43:13.594793+0000 mon.a (mon.0) 3889 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: cluster 2024-03-11T21:43:13.604156+0000 mon.a (mon.0) 3890 : cluster 0 osdmap e866: 8 total, 8 up, 8 in 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: audit 2024-03-11T21:43:13.654221+0000 mon.c (mon.2) 1339 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: audit 2024-03-11T21:43:13.654949+0000 mon.a (mon.0) 3891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: audit 2024-03-11T21:43:13.656110+0000 mon.c (mon.2) 1340 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[21512]: audit 2024-03-11T21:43:13.656731+0000 mon.a (mon.0) 3892 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: cluster 2024-03-11T21:43:13.520000+0000 osd.5 (osd.5) 475 : cluster 0 2.16 scrub starts 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: cluster 2024-03-11T21:43:13.521472+0000 osd.5 (osd.5) 476 : cluster 0 2.16 scrub ok 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: audit 2024-03-11T21:43:13.594793+0000 mon.a (mon.0) 3889 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:43:14.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: cluster 2024-03-11T21:43:13.604156+0000 mon.a (mon.0) 3890 : cluster 0 osdmap e866: 8 total, 8 up, 8 in 2024-03-11T21:43:14.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: audit 2024-03-11T21:43:13.654221+0000 mon.c (mon.2) 1339 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: audit 2024-03-11T21:43:13.654949+0000 mon.a (mon.0) 3891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: audit 2024-03-11T21:43:13.656110+0000 mon.c (mon.2) 1340 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:13 smithi143 bash[29284]: audit 2024-03-11T21:43:13.656731+0000 mon.a (mon.0) 3892 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: cluster 2024-03-11T21:43:13.520000+0000 osd.5 (osd.5) 475 : cluster 0 2.16 scrub starts 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: cluster 2024-03-11T21:43:13.521472+0000 osd.5 (osd.5) 476 : cluster 0 2.16 scrub ok 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: audit 2024-03-11T21:43:13.594793+0000 mon.a (mon.0) 3889 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]': finished 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: cluster 2024-03-11T21:43:13.604156+0000 mon.a (mon.0) 3890 : cluster 0 osdmap e866: 8 total, 8 up, 8 in 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: audit 2024-03-11T21:43:13.654221+0000 mon.c (mon.2) 1339 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: audit 2024-03-11T21:43:13.654949+0000 mon.a (mon.0) 3891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:14.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: audit 2024-03-11T21:43:13.656110+0000 mon.c (mon.2) 1340 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:13 smithi203 bash[23654]: audit 2024-03-11T21:43:13.656731+0000 mon.a (mon.0) 3892 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-146"}]: dispatch 2024-03-11T21:43:14.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:14.380 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:14.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:14.513 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:14.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:14.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:14.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:43:14.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:14.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:14.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:15.021 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:15.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.152 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:15.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:15.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:13.646136+0000 mgr.y (mgr.14550) 784 : cluster 0 pgmap v1350: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-11T21:43:15.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.058178+0000 osd.1 (osd.1) 265 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:43:15.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.059668+0000 osd.1 (osd.1) 266 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:43:15.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.594715+0000 mon.a (mon.0) 3893 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:15.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.594783+0000 mon.a (mon.0) 3894 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:43:15.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:15 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.611687+0000 mon.a (mon.0) 3895 : cluster 0 osdmap e867: 8 total, 8 up, 8 in 2024-03-11T21:43:15.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:15.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.415 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:15.444 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:13.646136+0000 mgr.y (mgr.14550) 784 : cluster 0 pgmap v1350: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.058178+0000 osd.1 (osd.1) 265 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.059668+0000 osd.1 (osd.1) 266 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.594715+0000 mon.a (mon.0) 3893 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.594783+0000 mon.a (mon.0) 3894 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.611687+0000 mon.a (mon.0) 3895 : cluster 0 osdmap e867: 8 total, 8 up, 8 in 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:13.646136+0000 mgr.y (mgr.14550) 784 : cluster 0 pgmap v1350: 246 pgs: 246 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.058178+0000 osd.1 (osd.1) 265 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.059668+0000 osd.1 (osd.1) 266 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.594715+0000 mon.a (mon.0) 3893 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.594783+0000 mon.a (mon.0) 3894 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:43:15.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:15 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.611687+0000 mon.a (mon.0) 3895 : cluster 0 osdmap e867: 8 total, 8 up, 8 in 2024-03-11T21:43:15.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:15.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:15.651 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:15.675 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:15.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:15.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:15.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:43:15.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:15.966 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:16.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:16.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.515556+0000 osd.7 (osd.7) 454 : cluster 0 6.8 scrub starts 2024-03-11T21:43:16.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.517342+0000 osd.7 (osd.7) 455 : cluster 0 6.8 scrub ok 2024-03-11T21:43:16.080 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.524797+0000 osd.2 (osd.2) 407 : cluster 0 5.8 scrub starts 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.526965+0000 osd.2 (osd.2) 408 : cluster 0 5.8 scrub ok 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.612459+0000 osd.6 (osd.6) 379 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.613073+0000 osd.6 (osd.6) 380 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.815443+0000 osd.4 (osd.4) 447 : cluster 0 39.10 scrub starts 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:14.816949+0000 osd.4 (osd.4) 448 : cluster 0 39.10 scrub ok 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.536095+0000 osd.5 (osd.5) 477 : cluster 0 2.12 scrub starts 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.538352+0000 osd.5 (osd.5) 478 : cluster 0 2.12 scrub ok 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.568354+0000 osd.2 (osd.2) 409 : cluster 0 2.10 scrub starts 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.569556+0000 osd.2 (osd.2) 410 : cluster 0 2.10 scrub ok 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.615817+0000 mon.a (mon.0) 3896 : cluster 0 osdmap e868: 8 total, 8 up, 8 in 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: audit 2024-03-11T21:43:15.618846+0000 mon.c (mon.2) 1341 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.081 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:16 smithi203 bash[23654]: audit 2024-03-11T21:43:15.622478+0000 mon.a (mon.0) 3897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.515556+0000 osd.7 (osd.7) 454 : cluster 0 6.8 scrub starts 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.517342+0000 osd.7 (osd.7) 455 : cluster 0 6.8 scrub ok 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.524797+0000 osd.2 (osd.2) 407 : cluster 0 5.8 scrub starts 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.526965+0000 osd.2 (osd.2) 408 : cluster 0 5.8 scrub ok 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.612459+0000 osd.6 (osd.6) 379 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:16.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.613073+0000 osd.6 (osd.6) 380 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.815443+0000 osd.4 (osd.4) 447 : cluster 0 39.10 scrub starts 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:14.816949+0000 osd.4 (osd.4) 448 : cluster 0 39.10 scrub ok 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.536095+0000 osd.5 (osd.5) 477 : cluster 0 2.12 scrub starts 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.538352+0000 osd.5 (osd.5) 478 : cluster 0 2.12 scrub ok 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.568354+0000 osd.2 (osd.2) 409 : cluster 0 2.10 scrub starts 2024-03-11T21:43:16.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.569556+0000 osd.2 (osd.2) 410 : cluster 0 2.10 scrub ok 2024-03-11T21:43:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.615817+0000 mon.a (mon.0) 3896 : cluster 0 osdmap e868: 8 total, 8 up, 8 in 2024-03-11T21:43:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: audit 2024-03-11T21:43:15.618846+0000 mon.c (mon.2) 1341 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[21512]: audit 2024-03-11T21:43:15.622478+0000 mon.a (mon.0) 3897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.515556+0000 osd.7 (osd.7) 454 : cluster 0 6.8 scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.517342+0000 osd.7 (osd.7) 455 : cluster 0 6.8 scrub ok 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.524797+0000 osd.2 (osd.2) 407 : cluster 0 5.8 scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.526965+0000 osd.2 (osd.2) 408 : cluster 0 5.8 scrub ok 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.612459+0000 osd.6 (osd.6) 379 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.613073+0000 osd.6 (osd.6) 380 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.815443+0000 osd.4 (osd.4) 447 : cluster 0 39.10 scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:14.816949+0000 osd.4 (osd.4) 448 : cluster 0 39.10 scrub ok 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.536095+0000 osd.5 (osd.5) 477 : cluster 0 2.12 scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.538352+0000 osd.5 (osd.5) 478 : cluster 0 2.12 scrub ok 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.568354+0000 osd.2 (osd.2) 409 : cluster 0 2.10 scrub starts 2024-03-11T21:43:16.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.569556+0000 osd.2 (osd.2) 410 : cluster 0 2.10 scrub ok 2024-03-11T21:43:16.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.615817+0000 mon.a (mon.0) 3896 : cluster 0 osdmap e868: 8 total, 8 up, 8 in 2024-03-11T21:43:16.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: audit 2024-03-11T21:43:15.618846+0000 mon.c (mon.2) 1341 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:16 smithi143 bash[29284]: audit 2024-03-11T21:43:15.622478+0000 mon.a (mon.0) 3897 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:16.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:16.225 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:16.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:16.357 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:16.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:16.486 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:16.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:16.630 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:16.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:16.759 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:16.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:16.890 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:16.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:17.018 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:17.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:17.148 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:17.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.409988+0000 osd.6 (osd.6) 381 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.412057+0000 osd.6 (osd.6) 382 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:15.646991+0000 mgr.y (mgr.14550) 785 : cluster 0 pgmap v1353: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 176 op/s 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.064347+0000 osd.1 (osd.1) 267 : cluster 0 5.12 scrub starts 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.065812+0000 osd.1 (osd.1) 268 : cluster 0 5.12 scrub ok 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.246144+0000 osd.0 (osd.0) 460 : cluster 0 9.1c scrub starts 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.248527+0000 osd.0 (osd.0) 461 : cluster 0 9.1c scrub ok 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: audit 2024-03-11T21:43:16.616743+0000 mon.a (mon.0) 3898 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.623291+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e869: 8 total, 8 up, 8 in 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: audit 2024-03-11T21:43:16.678488+0000 mon.c (mon.2) 1342 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.263 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[21512]: audit 2024-03-11T21:43:16.678832+0000 mon.a (mon.0) 3900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.409988+0000 osd.6 (osd.6) 381 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.412057+0000 osd.6 (osd.6) 382 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:15.646991+0000 mgr.y (mgr.14550) 785 : cluster 0 pgmap v1353: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 176 op/s 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.064347+0000 osd.1 (osd.1) 267 : cluster 0 5.12 scrub starts 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.065812+0000 osd.1 (osd.1) 268 : cluster 0 5.12 scrub ok 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.246144+0000 osd.0 (osd.0) 460 : cluster 0 9.1c scrub starts 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.248527+0000 osd.0 (osd.0) 461 : cluster 0 9.1c scrub ok 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: audit 2024-03-11T21:43:16.616743+0000 mon.a (mon.0) 3898 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.623291+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e869: 8 total, 8 up, 8 in 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: audit 2024-03-11T21:43:16.678488+0000 mon.c (mon.2) 1342 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:17 smithi143 bash[29284]: audit 2024-03-11T21:43:16.678832+0000 mon.a (mon.0) 3900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.278 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:17.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.409988+0000 osd.6 (osd.6) 381 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.412057+0000 osd.6 (osd.6) 382 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:15.646991+0000 mgr.y (mgr.14550) 785 : cluster 0 pgmap v1353: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 176 op/s 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.064347+0000 osd.1 (osd.1) 267 : cluster 0 5.12 scrub starts 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.065812+0000 osd.1 (osd.1) 268 : cluster 0 5.12 scrub ok 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.246144+0000 osd.0 (osd.0) 460 : cluster 0 9.1c scrub starts 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.248527+0000 osd.0 (osd.0) 461 : cluster 0 9.1c scrub ok 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: audit 2024-03-11T21:43:16.616743+0000 mon.a (mon.0) 3898 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.623291+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e869: 8 total, 8 up, 8 in 2024-03-11T21:43:17.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: audit 2024-03-11T21:43:16.678488+0000 mon.c (mon.2) 1342 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:17 smithi203 bash[23654]: audit 2024-03-11T21:43:16.678832+0000 mon.a (mon.0) 3900 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:43:17.401 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:17.406 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:17.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:17.534 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:17.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:17.663 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:17.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:17.794 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:17.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:17.925 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:18.028 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:18.055 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:18.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.364943+0000 osd.6 (osd.6) 383 : cluster 0 5.1f scrub starts 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.366870+0000 osd.6 (osd.6) 384 : cluster 0 5.1f scrub ok 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.571189+0000 osd.5 (osd.5) 479 : cluster 0 9.6 scrub starts 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:16.573334+0000 osd.5 (osd.5) 480 : cluster 0 9.6 scrub ok 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.254781+0000 osd.0 (osd.0) 462 : cluster 0 3.12 scrub starts 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.258860+0000 osd.0 (osd.0) 463 : cluster 0 3.12 scrub ok 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.505404+0000 osd.2 (osd.2) 411 : cluster 0 39.1f scrub starts 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.510277+0000 osd.2 (osd.2) 412 : cluster 0 39.1f scrub ok 2024-03-11T21:43:18.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: audit 2024-03-11T21:43:17.619612+0000 mon.a (mon.0) 3901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:43:18.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.627887+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e870: 8 total, 8 up, 8 in 2024-03-11T21:43:18.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: audit 2024-03-11T21:43:17.650919+0000 mon.c (mon.2) 1343 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.171 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:18 smithi203 bash[23654]: audit 2024-03-11T21:43:17.651563+0000 mon.a (mon.0) 3903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.185 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:18.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:18.315 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:18.419 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:18.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.364943+0000 osd.6 (osd.6) 383 : cluster 0 5.1f scrub starts 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.366870+0000 osd.6 (osd.6) 384 : cluster 0 5.1f scrub ok 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.571189+0000 osd.5 (osd.5) 479 : cluster 0 9.6 scrub starts 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:16.573334+0000 osd.5 (osd.5) 480 : cluster 0 9.6 scrub ok 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.254781+0000 osd.0 (osd.0) 462 : cluster 0 3.12 scrub starts 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.258860+0000 osd.0 (osd.0) 463 : cluster 0 3.12 scrub ok 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.505404+0000 osd.2 (osd.2) 411 : cluster 0 39.1f scrub starts 2024-03-11T21:43:18.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.510277+0000 osd.2 (osd.2) 412 : cluster 0 39.1f scrub ok 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: audit 2024-03-11T21:43:17.619612+0000 mon.a (mon.0) 3901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.627887+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e870: 8 total, 8 up, 8 in 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: audit 2024-03-11T21:43:17.650919+0000 mon.c (mon.2) 1343 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[29284]: audit 2024-03-11T21:43:17.651563+0000 mon.a (mon.0) 3903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.364943+0000 osd.6 (osd.6) 383 : cluster 0 5.1f scrub starts 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.366870+0000 osd.6 (osd.6) 384 : cluster 0 5.1f scrub ok 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.571189+0000 osd.5 (osd.5) 479 : cluster 0 9.6 scrub starts 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:16.573334+0000 osd.5 (osd.5) 480 : cluster 0 9.6 scrub ok 2024-03-11T21:43:18.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.254781+0000 osd.0 (osd.0) 462 : cluster 0 3.12 scrub starts 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.258860+0000 osd.0 (osd.0) 463 : cluster 0 3.12 scrub ok 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.505404+0000 osd.2 (osd.2) 411 : cluster 0 39.1f scrub starts 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.510277+0000 osd.2 (osd.2) 412 : cluster 0 39.1f scrub ok 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: audit 2024-03-11T21:43:17.619612+0000 mon.a (mon.0) 3901 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.627887+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e870: 8 total, 8 up, 8 in 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: audit 2024-03-11T21:43:17.650919+0000 mon.c (mon.2) 1343 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:18 smithi143 bash[21512]: audit 2024-03-11T21:43:17.651563+0000 mon.a (mon.0) 3903 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:18.443 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:18.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:18.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:18.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:18.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:18.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:18.835 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:18.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:18.966 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:17.648143+0000 mgr.y (mgr.14550) 786 : cluster 0 pgmap v1356: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.210655+0000 osd.0 (osd.0) 464 : cluster 0 313.0s0 scrub starts 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.214924+0000 osd.0 (osd.0) 465 : cluster 0 313.0 scrub ok 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.505860+0000 osd.2 (osd.2) 413 : cluster 0 2.e scrub starts 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.507508+0000 osd.2 (osd.2) 414 : cluster 0 2.e scrub ok 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: audit 2024-03-11T21:43:18.627726+0000 mon.a (mon.0) 3904 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]': finished 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.633493+0000 mon.a (mon.0) 3905 : cluster 0 osdmap e871: 8 total, 8 up, 8 in 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: audit 2024-03-11T21:43:18.695481+0000 mon.c (mon.2) 1344 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: audit 2024-03-11T21:43:18.696219+0000 mon.a (mon.0) 3906 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.077 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: audit 2024-03-11T21:43:18.697435+0000 mon.c (mon.2) 1345 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.078 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: audit 2024-03-11T21:43:18.698165+0000 mon.a (mon.0) 3907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.078 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.718981+0000 osd.4 (osd.4) 449 : cluster 0 6.2 scrub starts 2024-03-11T21:43:19.078 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[23654]: cluster 2024-03-11T21:43:18.720915+0000 osd.4 (osd.4) 450 : cluster 0 6.2 scrub ok 2024-03-11T21:43:19.098 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:19.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:19.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:19.330 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:17.648143+0000 mgr.y (mgr.14550) 786 : cluster 0 pgmap v1356: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.210655+0000 osd.0 (osd.0) 464 : cluster 0 313.0s0 scrub starts 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.214924+0000 osd.0 (osd.0) 465 : cluster 0 313.0 scrub ok 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.505860+0000 osd.2 (osd.2) 413 : cluster 0 2.e scrub starts 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.507508+0000 osd.2 (osd.2) 414 : cluster 0 2.e scrub ok 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: audit 2024-03-11T21:43:18.627726+0000 mon.a (mon.0) 3904 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]': finished 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.633493+0000 mon.a (mon.0) 3905 : cluster 0 osdmap e871: 8 total, 8 up, 8 in 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: audit 2024-03-11T21:43:18.695481+0000 mon.c (mon.2) 1344 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: audit 2024-03-11T21:43:18.696219+0000 mon.a (mon.0) 3906 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: audit 2024-03-11T21:43:18.697435+0000 mon.c (mon.2) 1345 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: audit 2024-03-11T21:43:18.698165+0000 mon.a (mon.0) 3907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.718981+0000 osd.4 (osd.4) 449 : cluster 0 6.2 scrub starts 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[21512]: cluster 2024-03-11T21:43:18.720915+0000 osd.4 (osd.4) 450 : cluster 0 6.2 scrub ok 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:17.648143+0000 mgr.y (mgr.14550) 786 : cluster 0 pgmap v1356: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.210655+0000 osd.0 (osd.0) 464 : cluster 0 313.0s0 scrub starts 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.214924+0000 osd.0 (osd.0) 465 : cluster 0 313.0 scrub ok 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.505860+0000 osd.2 (osd.2) 413 : cluster 0 2.e scrub starts 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.507508+0000 osd.2 (osd.2) 414 : cluster 0 2.e scrub ok 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: audit 2024-03-11T21:43:18.627726+0000 mon.a (mon.0) 3904 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]': finished 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.633493+0000 mon.a (mon.0) 3905 : cluster 0 osdmap e871: 8 total, 8 up, 8 in 2024-03-11T21:43:19.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: audit 2024-03-11T21:43:18.695481+0000 mon.c (mon.2) 1344 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: audit 2024-03-11T21:43:18.696219+0000 mon.a (mon.0) 3906 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:19.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: audit 2024-03-11T21:43:18.697435+0000 mon.c (mon.2) 1345 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: audit 2024-03-11T21:43:18.698165+0000 mon.a (mon.0) 3907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-148"}]: dispatch 2024-03-11T21:43:19.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.718981+0000 osd.4 (osd.4) 449 : cluster 0 6.2 scrub starts 2024-03-11T21:43:19.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:19 smithi143 bash[29284]: cluster 2024-03-11T21:43:18.720915+0000 osd.4 (osd.4) 450 : cluster 0 6.2 scrub ok 2024-03-11T21:43:19.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.502 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:19.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:19.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:19.622 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:19.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:19.748 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:19.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:19.887 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:19.991 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:20.015 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:20.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:20.143 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:20.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:20.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: audit 2024-03-11T21:43:19.093657+0000 mgr.y (mgr.14550) 787 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:20.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.496969+0000 osd.2 (osd.2) 415 : cluster 0 39.6 scrub starts 2024-03-11T21:43:20.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.498996+0000 osd.2 (osd.2) 416 : cluster 0 39.6 scrub ok 2024-03-11T21:43:20.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.520757+0000 osd.5 (osd.5) 481 : cluster 0 9.a scrub starts 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.522417+0000 osd.5 (osd.5) 482 : cluster 0 9.a scrub ok 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.638206+0000 mon.a (mon.0) 3908 : cluster 0 osdmap e872: 8 total, 8 up, 8 in 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: audit 2024-03-11T21:43:19.093657+0000 mgr.y (mgr.14550) 787 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.496969+0000 osd.2 (osd.2) 415 : cluster 0 39.6 scrub starts 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.498996+0000 osd.2 (osd.2) 416 : cluster 0 39.6 scrub ok 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.520757+0000 osd.5 (osd.5) 481 : cluster 0 9.a scrub starts 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.522417+0000 osd.5 (osd.5) 482 : cluster 0 9.a scrub ok 2024-03-11T21:43:20.258 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:20 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.638206+0000 mon.a (mon.0) 3908 : cluster 0 osdmap e872: 8 total, 8 up, 8 in 2024-03-11T21:43:20.273 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: audit 2024-03-11T21:43:19.093657+0000 mgr.y (mgr.14550) 787 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.496969+0000 osd.2 (osd.2) 415 : cluster 0 39.6 scrub starts 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.498996+0000 osd.2 (osd.2) 416 : cluster 0 39.6 scrub ok 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.520757+0000 osd.5 (osd.5) 481 : cluster 0 9.a scrub starts 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.522417+0000 osd.5 (osd.5) 482 : cluster 0 9.a scrub ok 2024-03-11T21:43:20.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:20 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.638206+0000 mon.a (mon.0) 3908 : cluster 0 osdmap e872: 8 total, 8 up, 8 in 2024-03-11T21:43:20.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:20.403 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:20.460 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:20.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:20.534 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:20.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:20.662 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:20.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:20.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:20.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:20.919 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:21.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:21.050 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:21.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:21.176 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:21.265 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.371229+0000 osd.6 (osd.6) 385 : cluster 0 3.a scrub starts 2024-03-11T21:43:21.265 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.376089+0000 osd.6 (osd.6) 386 : cluster 0 3.a scrub ok 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:19.649212+0000 mgr.y (mgr.14550) 788 : cluster 0 pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.115754+0000 osd.1 (osd.1) 269 : cluster 0 39.2 scrub starts 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.116858+0000 osd.1 (osd.1) 270 : cluster 0 39.2 scrub ok 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.297980+0000 osd.0 (osd.0) 466 : cluster 0 6.0 scrub starts 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.299049+0000 osd.0 (osd.0) 467 : cluster 0 6.0 scrub ok 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.512524+0000 osd.2 (osd.2) 417 : cluster 0 5.b scrub starts 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.513925+0000 osd.2 (osd.2) 418 : cluster 0 5.b scrub ok 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.539231+0000 osd.5 (osd.5) 483 : cluster 0 3.1d scrub starts 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.541455+0000 osd.5 (osd.5) 484 : cluster 0 3.1d scrub ok 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: cluster 2024-03-11T21:43:20.647710+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e873: 8 total, 8 up, 8 in 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: audit 2024-03-11T21:43:20.666316+0000 mon.c (mon.2) 1346 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:21 smithi203 bash[23654]: audit 2024-03-11T21:43:20.668203+0000 mon.a (mon.0) 3910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:21.306 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:21.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:21.436 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.371229+0000 osd.6 (osd.6) 385 : cluster 0 3.a scrub starts 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.376089+0000 osd.6 (osd.6) 386 : cluster 0 3.a scrub ok 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:19.649212+0000 mgr.y (mgr.14550) 788 : cluster 0 pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.115754+0000 osd.1 (osd.1) 269 : cluster 0 39.2 scrub starts 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.116858+0000 osd.1 (osd.1) 270 : cluster 0 39.2 scrub ok 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.297980+0000 osd.0 (osd.0) 466 : cluster 0 6.0 scrub starts 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.299049+0000 osd.0 (osd.0) 467 : cluster 0 6.0 scrub ok 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.512524+0000 osd.2 (osd.2) 417 : cluster 0 5.b scrub starts 2024-03-11T21:43:21.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.513925+0000 osd.2 (osd.2) 418 : cluster 0 5.b scrub ok 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.539231+0000 osd.5 (osd.5) 483 : cluster 0 3.1d scrub starts 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.541455+0000 osd.5 (osd.5) 484 : cluster 0 3.1d scrub ok 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: cluster 2024-03-11T21:43:20.647710+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e873: 8 total, 8 up, 8 in 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: audit 2024-03-11T21:43:20.666316+0000 mon.c (mon.2) 1346 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[21512]: audit 2024-03-11T21:43:20.668203+0000 mon.a (mon.0) 3910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.371229+0000 osd.6 (osd.6) 385 : cluster 0 3.a scrub starts 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.376089+0000 osd.6 (osd.6) 386 : cluster 0 3.a scrub ok 2024-03-11T21:43:21.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:19.649212+0000 mgr.y (mgr.14550) 788 : cluster 0 pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.115754+0000 osd.1 (osd.1) 269 : cluster 0 39.2 scrub starts 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.116858+0000 osd.1 (osd.1) 270 : cluster 0 39.2 scrub ok 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.297980+0000 osd.0 (osd.0) 466 : cluster 0 6.0 scrub starts 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.299049+0000 osd.0 (osd.0) 467 : cluster 0 6.0 scrub ok 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.512524+0000 osd.2 (osd.2) 417 : cluster 0 5.b scrub starts 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.513925+0000 osd.2 (osd.2) 418 : cluster 0 5.b scrub ok 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.539231+0000 osd.5 (osd.5) 483 : cluster 0 3.1d scrub starts 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.541455+0000 osd.5 (osd.5) 484 : cluster 0 3.1d scrub ok 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: cluster 2024-03-11T21:43:20.647710+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e873: 8 total, 8 up, 8 in 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: audit 2024-03-11T21:43:20.666316+0000 mon.c (mon.2) 1346 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:21 smithi143 bash[29284]: audit 2024-03-11T21:43:20.668203+0000 mon.a (mon.0) 3910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:21.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:21.574 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:21.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:21.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:21.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:21.838 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:21.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:22.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:22.100 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:22.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.212 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.085043+0000 osd.7 (osd.7) 456 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.085272+0000 osd.7 (osd.7) 457 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.565750+0000 mon.a (mon.0) 3911 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: audit 2024-03-11T21:43:21.568646+0000 mon.a (mon.0) 3912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.573602+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e874: 8 total, 8 up, 8 in 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: audit 2024-03-11T21:43:21.784009+0000 mon.c (mon.2) 1347 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: audit 2024-03-11T21:43:21.785268+0000 mon.a (mon.0) 3914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: audit 2024-03-11T21:43:21.786486+0000 mon.c (mon.2) 1348 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[21512]: audit 2024-03-11T21:43:21.787092+0000 mon.a (mon.0) 3915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.085043+0000 osd.7 (osd.7) 456 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.085272+0000 osd.7 (osd.7) 457 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.565750+0000 mon.a (mon.0) 3911 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: audit 2024-03-11T21:43:21.568646+0000 mon.a (mon.0) 3912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.573602+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e874: 8 total, 8 up, 8 in 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: audit 2024-03-11T21:43:21.784009+0000 mon.c (mon.2) 1347 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: audit 2024-03-11T21:43:21.785268+0000 mon.a (mon.0) 3914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: audit 2024-03-11T21:43:21.786486+0000 mon.c (mon.2) 1348 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:22 smithi143 bash[29284]: audit 2024-03-11T21:43:21.787092+0000 mon.a (mon.0) 3915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.226 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:22.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.085043+0000 osd.7 (osd.7) 456 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.085272+0000 osd.7 (osd.7) 457 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.565750+0000 mon.a (mon.0) 3911 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: audit 2024-03-11T21:43:21.568646+0000 mon.a (mon.0) 3912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.573602+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e874: 8 total, 8 up, 8 in 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: audit 2024-03-11T21:43:21.784009+0000 mon.c (mon.2) 1347 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: audit 2024-03-11T21:43:21.785268+0000 mon.a (mon.0) 3914 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: audit 2024-03-11T21:43:21.786486+0000 mon.c (mon.2) 1348 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:22 smithi203 bash[23654]: audit 2024-03-11T21:43:21.787092+0000 mon.a (mon.0) 3915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-150"}]: dispatch 2024-03-11T21:43:22.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.354 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:22.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.485 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:22.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:22.613 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:22.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.744 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:22.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:22.875 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:22.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:23.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:23.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:23.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:23.236 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:43:23.236 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:43:23.236 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:43:23.236 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.481098+0000 osd.7 (osd.7) 458 : cluster 0 39.1 scrub starts 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.483708+0000 osd.7 (osd.7) 459 : cluster 0 39.1 scrub ok 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.551928+0000 osd.5 (osd.5) 485 : cluster 0 5.7 scrub starts 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.554327+0000 osd.5 (osd.5) 486 : cluster 0 5.7 scrub ok 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:21.649978+0000 mgr.y (mgr.14550) 789 : cluster 0 pgmap v1362: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:22.267492+0000 osd.0 (osd.0) 468 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:22.269022+0000 osd.0 (osd.0) 469 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: audit 2024-03-11T21:43:22.502148+0000 mon.a (mon.0) 3916 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: audit 2024-03-11T21:43:22.505266+0000 mon.c (mon.2) 1349 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.248 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:23 smithi203 bash[23654]: cluster 2024-03-11T21:43:22.598754+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e875: 8 total, 8 up, 8 in 2024-03-11T21:43:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:23.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.370 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.481098+0000 osd.7 (osd.7) 458 : cluster 0 39.1 scrub starts 2024-03-11T21:43:23.370 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.483708+0000 osd.7 (osd.7) 459 : cluster 0 39.1 scrub ok 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.551928+0000 osd.5 (osd.5) 485 : cluster 0 5.7 scrub starts 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.554327+0000 osd.5 (osd.5) 486 : cluster 0 5.7 scrub ok 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:21.649978+0000 mgr.y (mgr.14550) 789 : cluster 0 pgmap v1362: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:22.267492+0000 osd.0 (osd.0) 468 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:22.269022+0000 osd.0 (osd.0) 469 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: audit 2024-03-11T21:43:22.502148+0000 mon.a (mon.0) 3916 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: audit 2024-03-11T21:43:22.505266+0000 mon.c (mon.2) 1349 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[21512]: cluster 2024-03-11T21:43:22.598754+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e875: 8 total, 8 up, 8 in 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.481098+0000 osd.7 (osd.7) 458 : cluster 0 39.1 scrub starts 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.483708+0000 osd.7 (osd.7) 459 : cluster 0 39.1 scrub ok 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.551928+0000 osd.5 (osd.5) 485 : cluster 0 5.7 scrub starts 2024-03-11T21:43:23.371 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.554327+0000 osd.5 (osd.5) 486 : cluster 0 5.7 scrub ok 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:21.649978+0000 mgr.y (mgr.14550) 789 : cluster 0 pgmap v1362: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:22.267492+0000 osd.0 (osd.0) 468 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:22.269022+0000 osd.0 (osd.0) 469 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: audit 2024-03-11T21:43:22.502148+0000 mon.a (mon.0) 3916 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: audit 2024-03-11T21:43:22.505266+0000 mon.c (mon.2) 1349 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:23 smithi143 bash[29284]: cluster 2024-03-11T21:43:22.598754+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e875: 8 total, 8 up, 8 in 2024-03-11T21:43:23.384 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:23.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.515 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:23.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:23.642 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:23.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:23.766 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:23.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:23.896 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:23.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:24.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:24.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: cluster 2024-03-11T21:43:22.457502+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub starts 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: cluster 2024-03-11T21:43:22.461609+0000 osd.7 (osd.7) 461 : cluster 0 3.e scrub ok 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.294711+0000 osd.0 (osd.0) 470 : cluster 0 39.4 scrub starts 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.296504+0000 osd.0 (osd.0) 471 : cluster 0 39.4 scrub ok 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.602061+0000 mon.a (mon.0) 3918 : cluster 0 osdmap e876: 8 total, 8 up, 8 in 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: audit 2024-03-11T21:43:23.604886+0000 mon.c (mon.2) 1350 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.153 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:24 smithi203 bash[23654]: audit 2024-03-11T21:43:23.615221+0000 mon.a (mon.0) 3919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:43:24.158 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:24.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:24.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: cluster 2024-03-11T21:43:22.457502+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub starts 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: cluster 2024-03-11T21:43:22.461609+0000 osd.7 (osd.7) 461 : cluster 0 3.e scrub ok 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.294711+0000 osd.0 (osd.0) 470 : cluster 0 39.4 scrub starts 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.296504+0000 osd.0 (osd.0) 471 : cluster 0 39.4 scrub ok 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.602061+0000 mon.a (mon.0) 3918 : cluster 0 osdmap e876: 8 total, 8 up, 8 in 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: audit 2024-03-11T21:43:23.604886+0000 mon.c (mon.2) 1350 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[29284]: audit 2024-03-11T21:43:23.615221+0000 mon.a (mon.0) 3919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: cluster 2024-03-11T21:43:22.457502+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub starts 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: cluster 2024-03-11T21:43:22.461609+0000 osd.7 (osd.7) 461 : cluster 0 3.e scrub ok 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.294711+0000 osd.0 (osd.0) 470 : cluster 0 39.4 scrub starts 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.296504+0000 osd.0 (osd.0) 471 : cluster 0 39.4 scrub ok 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.602061+0000 mon.a (mon.0) 3918 : cluster 0 osdmap e876: 8 total, 8 up, 8 in 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: audit 2024-03-11T21:43:23.604886+0000 mon.c (mon.2) 1350 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:24 smithi143 bash[21512]: audit 2024-03-11T21:43:23.615221+0000 mon.a (mon.0) 3919 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:24.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:24.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.414 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:24.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.544 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:24.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.672 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:24.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:24.800 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:24.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:24.930 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:25.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:25.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:25.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.409246+0000 osd.7 (osd.7) 462 : cluster 0 9.13 scrub starts 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.412296+0000 osd.7 (osd.7) 463 : cluster 0 9.13 scrub ok 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:23.650772+0000 mgr.y (mgr.14550) 790 : cluster 0 pgmap v1365: 246 pgs: 2 creating+peering, 30 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.382391+0000 osd.4 (osd.4) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.382660+0000 osd.4 (osd.4) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: audit 2024-03-11T21:43:24.602188+0000 mon.a (mon.0) 3920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.610633+0000 mon.a (mon.0) 3921 : cluster 0 osdmap e877: 8 total, 8 up, 8 in 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: audit 2024-03-11T21:43:24.707787+0000 mon.c (mon.2) 1351 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: audit 2024-03-11T21:43:24.708465+0000 mon.a (mon.0) 3922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: audit 2024-03-11T21:43:24.709442+0000 mon.c (mon.2) 1352 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: audit 2024-03-11T21:43:24.709934+0000 mon.a (mon.0) 3923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.732507+0000 osd.3 (osd.3) 467 : cluster 0 9.1 scrub starts 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.734452+0000 osd.3 (osd.3) 468 : cluster 0 9.1 scrub ok 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.760246+0000 osd.4 (osd.4) 453 : cluster 0 9.16 scrub starts 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.762552+0000 osd.4 (osd.4) 454 : cluster 0 9.16 scrub ok 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.409246+0000 osd.7 (osd.7) 462 : cluster 0 9.13 scrub starts 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.412296+0000 osd.7 (osd.7) 463 : cluster 0 9.13 scrub ok 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:23.650772+0000 mgr.y (mgr.14550) 790 : cluster 0 pgmap v1365: 246 pgs: 2 creating+peering, 30 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.382391+0000 osd.4 (osd.4) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.382660+0000 osd.4 (osd.4) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: audit 2024-03-11T21:43:24.602188+0000 mon.a (mon.0) 3920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:25.179 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.610633+0000 mon.a (mon.0) 3921 : cluster 0 osdmap e877: 8 total, 8 up, 8 in 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: audit 2024-03-11T21:43:24.707787+0000 mon.c (mon.2) 1351 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: audit 2024-03-11T21:43:24.708465+0000 mon.a (mon.0) 3922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: audit 2024-03-11T21:43:24.709442+0000 mon.c (mon.2) 1352 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: audit 2024-03-11T21:43:24.709934+0000 mon.a (mon.0) 3923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.732507+0000 osd.3 (osd.3) 467 : cluster 0 9.1 scrub starts 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.734452+0000 osd.3 (osd.3) 468 : cluster 0 9.1 scrub ok 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.760246+0000 osd.4 (osd.4) 453 : cluster 0 9.16 scrub starts 2024-03-11T21:43:25.180 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:25 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.762552+0000 osd.4 (osd.4) 454 : cluster 0 9.16 scrub ok 2024-03-11T21:43:25.193 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:25.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:25.324 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:25.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.439 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.409246+0000 osd.7 (osd.7) 462 : cluster 0 9.13 scrub starts 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.412296+0000 osd.7 (osd.7) 463 : cluster 0 9.13 scrub ok 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:23.650772+0000 mgr.y (mgr.14550) 790 : cluster 0 pgmap v1365: 246 pgs: 2 creating+peering, 30 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.382391+0000 osd.4 (osd.4) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.382660+0000 osd.4 (osd.4) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: audit 2024-03-11T21:43:24.602188+0000 mon.a (mon.0) 3920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.610633+0000 mon.a (mon.0) 3921 : cluster 0 osdmap e877: 8 total, 8 up, 8 in 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: audit 2024-03-11T21:43:24.707787+0000 mon.c (mon.2) 1351 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: audit 2024-03-11T21:43:24.708465+0000 mon.a (mon.0) 3922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: audit 2024-03-11T21:43:24.709442+0000 mon.c (mon.2) 1352 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: audit 2024-03-11T21:43:24.709934+0000 mon.a (mon.0) 3923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-152"}]: dispatch 2024-03-11T21:43:25.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.732507+0000 osd.3 (osd.3) 467 : cluster 0 9.1 scrub starts 2024-03-11T21:43:25.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.734452+0000 osd.3 (osd.3) 468 : cluster 0 9.1 scrub ok 2024-03-11T21:43:25.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.760246+0000 osd.4 (osd.4) 453 : cluster 0 9.16 scrub starts 2024-03-11T21:43:25.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:25 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.762552+0000 osd.4 (osd.4) 454 : cluster 0 9.16 scrub ok 2024-03-11T21:43:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:25.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.585 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:25.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:25.711 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:25.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:25.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:25.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:25.960 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:26.064 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:26.088 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:26.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.335053+0000 osd.6 (osd.6) 387 : cluster 0 5.2 scrub starts 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.336777+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub ok 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.427417+0000 osd.7 (osd.7) 464 : cluster 0 6.1c scrub starts 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.429150+0000 osd.7 (osd.7) 465 : cluster 0 6.1c scrub ok 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.631326+0000 osd.5 (osd.5) 487 : cluster 0 1.a deep-scrub starts 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:24.638111+0000 osd.5 (osd.5) 488 : cluster 0 1.a deep-scrub ok 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.030506+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:43:26.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.032092+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.281196+0000 osd.0 (osd.0) 472 : cluster 0 3.6 scrub starts 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.283986+0000 osd.0 (osd.0) 473 : cluster 0 3.6 scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.620399+0000 mon.a (mon.0) 3924 : cluster 0 osdmap e878: 8 total, 8 up, 8 in 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.335053+0000 osd.6 (osd.6) 387 : cluster 0 5.2 scrub starts 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.336777+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.427417+0000 osd.7 (osd.7) 464 : cluster 0 6.1c scrub starts 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.429150+0000 osd.7 (osd.7) 465 : cluster 0 6.1c scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.631326+0000 osd.5 (osd.5) 487 : cluster 0 1.a deep-scrub starts 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:24.638111+0000 osd.5 (osd.5) 488 : cluster 0 1.a deep-scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.030506+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.032092+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:43:26.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.281196+0000 osd.0 (osd.0) 472 : cluster 0 3.6 scrub starts 2024-03-11T21:43:26.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.283986+0000 osd.0 (osd.0) 473 : cluster 0 3.6 scrub ok 2024-03-11T21:43:26.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:26 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.620399+0000 mon.a (mon.0) 3924 : cluster 0 osdmap e878: 8 total, 8 up, 8 in 2024-03-11T21:43:26.216 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:26.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:26.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:26.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:26.480 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:26.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.335053+0000 osd.6 (osd.6) 387 : cluster 0 5.2 scrub starts 2024-03-11T21:43:26.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.336777+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub ok 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.427417+0000 osd.7 (osd.7) 464 : cluster 0 6.1c scrub starts 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.429150+0000 osd.7 (osd.7) 465 : cluster 0 6.1c scrub ok 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.631326+0000 osd.5 (osd.5) 487 : cluster 0 1.a deep-scrub starts 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:24.638111+0000 osd.5 (osd.5) 488 : cluster 0 1.a deep-scrub ok 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.030506+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.032092+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.281196+0000 osd.0 (osd.0) 472 : cluster 0 3.6 scrub starts 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.283986+0000 osd.0 (osd.0) 473 : cluster 0 3.6 scrub ok 2024-03-11T21:43:26.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:26 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.620399+0000 mon.a (mon.0) 3924 : cluster 0 osdmap e878: 8 total, 8 up, 8 in 2024-03-11T21:43:26.583 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:26.617 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:26.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:26.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:26.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:26.878 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:26.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:27.008 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:27.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:27.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:27.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:27.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.372103+0000 osd.6 (osd.6) 389 : cluster 0 1.7 scrub starts 2024-03-11T21:43:27.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.374035+0000 osd.6 (osd.6) 390 : cluster 0 1.7 scrub ok 2024-03-11T21:43:27.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:25.651546+0000 mgr.y (mgr.14550) 791 : cluster 0 pgmap v1368: 214 pgs: 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.026555+0000 osd.1 (osd.1) 273 : cluster 0 5.1d scrub starts 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.028188+0000 osd.1 (osd.1) 274 : cluster 0 5.1d scrub ok 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.267228+0000 osd.0 (osd.0) 474 : cluster 0 2.13 scrub starts 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.268953+0000 osd.0 (osd.0) 475 : cluster 0 2.13 scrub ok 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.568035+0000 mon.a (mon.0) 3925 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.624236+0000 mon.a (mon.0) 3926 : cluster 0 osdmap e879: 8 total, 8 up, 8 in 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: audit 2024-03-11T21:43:26.632488+0000 mon.c (mon.2) 1353 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: audit 2024-03-11T21:43:26.640856+0000 mon.a (mon.0) 3927 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.742348+0000 osd.3 (osd.3) 469 : cluster 0 5.0 scrub starts 2024-03-11T21:43:27.253 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:27 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.744802+0000 osd.3 (osd.3) 470 : cluster 0 5.0 scrub ok 2024-03-11T21:43:27.268 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:27.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:27.423 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.372103+0000 osd.6 (osd.6) 389 : cluster 0 1.7 scrub starts 2024-03-11T21:43:27.423 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.374035+0000 osd.6 (osd.6) 390 : cluster 0 1.7 scrub ok 2024-03-11T21:43:27.423 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:25.651546+0000 mgr.y (mgr.14550) 791 : cluster 0 pgmap v1368: 214 pgs: 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.026555+0000 osd.1 (osd.1) 273 : cluster 0 5.1d scrub starts 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.028188+0000 osd.1 (osd.1) 274 : cluster 0 5.1d scrub ok 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.267228+0000 osd.0 (osd.0) 474 : cluster 0 2.13 scrub starts 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.268953+0000 osd.0 (osd.0) 475 : cluster 0 2.13 scrub ok 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.568035+0000 mon.a (mon.0) 3925 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.624236+0000 mon.a (mon.0) 3926 : cluster 0 osdmap e879: 8 total, 8 up, 8 in 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: audit 2024-03-11T21:43:26.632488+0000 mon.c (mon.2) 1353 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: audit 2024-03-11T21:43:26.640856+0000 mon.a (mon.0) 3927 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.742348+0000 osd.3 (osd.3) 469 : cluster 0 5.0 scrub starts 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.744802+0000 osd.3 (osd.3) 470 : cluster 0 5.0 scrub ok 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.372103+0000 osd.6 (osd.6) 389 : cluster 0 1.7 scrub starts 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.374035+0000 osd.6 (osd.6) 390 : cluster 0 1.7 scrub ok 2024-03-11T21:43:27.424 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:25.651546+0000 mgr.y (mgr.14550) 791 : cluster 0 pgmap v1368: 214 pgs: 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.026555+0000 osd.1 (osd.1) 273 : cluster 0 5.1d scrub starts 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.028188+0000 osd.1 (osd.1) 274 : cluster 0 5.1d scrub ok 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.267228+0000 osd.0 (osd.0) 474 : cluster 0 2.13 scrub starts 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.268953+0000 osd.0 (osd.0) 475 : cluster 0 2.13 scrub ok 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.568035+0000 mon.a (mon.0) 3925 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.624236+0000 mon.a (mon.0) 3926 : cluster 0 osdmap e879: 8 total, 8 up, 8 in 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: audit 2024-03-11T21:43:26.632488+0000 mon.c (mon.2) 1353 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: audit 2024-03-11T21:43:26.640856+0000 mon.a (mon.0) 3927 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.742348+0000 osd.3 (osd.3) 469 : cluster 0 5.0 scrub starts 2024-03-11T21:43:27.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:27 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.744802+0000 osd.3 (osd.3) 470 : cluster 0 5.0 scrub ok 2024-03-11T21:43:27.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:27.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:27.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:27.666 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:27.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:27.794 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:27.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.921 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:28.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.051 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:28.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:28.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:28.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:28.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:28.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.355897+0000 osd.6 (osd.6) 391 : cluster 0 39.19 scrub starts 2024-03-11T21:43:28.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.358592+0000 osd.6 (osd.6) 392 : cluster 0 39.19 scrub ok 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.555128+0000 osd.5 (osd.5) 489 : cluster 0 5.f scrub starts 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:26.557590+0000 osd.5 (osd.5) 490 : cluster 0 5.f scrub ok 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.074777+0000 osd.1 (osd.1) 275 : cluster 0 2.15 scrub starts 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.076137+0000 osd.1 (osd.1) 276 : cluster 0 2.15 scrub ok 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.318229+0000 osd.6 (osd.6) 393 : cluster 0 39.1e scrub starts 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.319753+0000 osd.6 (osd.6) 394 : cluster 0 39.1e scrub ok 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.620730+0000 mon.a (mon.0) 3928 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.628768+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e880: 8 total, 8 up, 8 in 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.640325+0000 mon.c (mon.2) 1354 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.641470+0000 mon.a (mon.0) 3930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.695460+0000 mon.c (mon.2) 1355 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.695851+0000 mon.a (mon.0) 3931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.696399+0000 mon.c (mon.2) 1356 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[29284]: audit 2024-03-11T21:43:27.696683+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.355897+0000 osd.6 (osd.6) 391 : cluster 0 39.19 scrub starts 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.358592+0000 osd.6 (osd.6) 392 : cluster 0 39.19 scrub ok 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.555128+0000 osd.5 (osd.5) 489 : cluster 0 5.f scrub starts 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:26.557590+0000 osd.5 (osd.5) 490 : cluster 0 5.f scrub ok 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.074777+0000 osd.1 (osd.1) 275 : cluster 0 2.15 scrub starts 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.076137+0000 osd.1 (osd.1) 276 : cluster 0 2.15 scrub ok 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.318229+0000 osd.6 (osd.6) 393 : cluster 0 39.1e scrub starts 2024-03-11T21:43:28.427 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.319753+0000 osd.6 (osd.6) 394 : cluster 0 39.1e scrub ok 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.620730+0000 mon.a (mon.0) 3928 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.628768+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e880: 8 total, 8 up, 8 in 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.640325+0000 mon.c (mon.2) 1354 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.641470+0000 mon.a (mon.0) 3930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.695460+0000 mon.c (mon.2) 1355 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.695851+0000 mon.a (mon.0) 3931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.696399+0000 mon.c (mon.2) 1356 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.428 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:28 smithi143 bash[21512]: audit 2024-03-11T21:43:27.696683+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.355897+0000 osd.6 (osd.6) 391 : cluster 0 39.19 scrub starts 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.358592+0000 osd.6 (osd.6) 392 : cluster 0 39.19 scrub ok 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.555128+0000 osd.5 (osd.5) 489 : cluster 0 5.f scrub starts 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:26.557590+0000 osd.5 (osd.5) 490 : cluster 0 5.f scrub ok 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.074777+0000 osd.1 (osd.1) 275 : cluster 0 2.15 scrub starts 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.076137+0000 osd.1 (osd.1) 276 : cluster 0 2.15 scrub ok 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.318229+0000 osd.6 (osd.6) 393 : cluster 0 39.1e scrub starts 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.319753+0000 osd.6 (osd.6) 394 : cluster 0 39.1e scrub ok 2024-03-11T21:43:28.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.620730+0000 mon.a (mon.0) 3928 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-71773-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.628768+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e880: 8 total, 8 up, 8 in 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.640325+0000 mon.c (mon.2) 1354 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.641470+0000 mon.a (mon.0) 3930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi143-71773-111","var": "dedup_tier","val": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.695460+0000 mon.c (mon.2) 1355 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.695851+0000 mon.a (mon.0) 3931 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.696399+0000 mon.c (mon.2) 1356 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:28 smithi203 bash[23654]: audit 2024-03-11T21:43:27.696683+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi143-71773-111", "tierpool": "test-rados-api-smithi143-71773-154"}]: dispatch 2024-03-11T21:43:28.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.571 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:28.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:28.701 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:28.804 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:28.930 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:28.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:28.959 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:29.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:29.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:29.219 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:29.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:29.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.554376+0000 osd.5 (osd.5) 491 : cluster 0 3.c deep-scrub starts 2024-03-11T21:43:29.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.559029+0000 osd.5 (osd.5) 492 : cluster 0 3.c deep-scrub ok 2024-03-11T21:43:29.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:27.652628+0000 mgr.y (mgr.14550) 792 : cluster 0 pgmap v1371: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:29.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:28.628336+0000 mon.a (mon.0) 3933 : cluster 0 osdmap e881: 8 total, 8 up, 8 in 2024-03-11T21:43:29.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:28.723709+0000 osd.3 (osd.3) 471 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:43:29.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:29 smithi203 bash[23654]: cluster 2024-03-11T21:43:28.728775+0000 osd.3 (osd.3) 472 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:43:29.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:29.348 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:29.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:29.466 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.554376+0000 osd.5 (osd.5) 491 : cluster 0 3.c deep-scrub starts 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.559029+0000 osd.5 (osd.5) 492 : cluster 0 3.c deep-scrub ok 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:27.652628+0000 mgr.y (mgr.14550) 792 : cluster 0 pgmap v1371: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:28.628336+0000 mon.a (mon.0) 3933 : cluster 0 osdmap e881: 8 total, 8 up, 8 in 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:28.723709+0000 osd.3 (osd.3) 471 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[21512]: cluster 2024-03-11T21:43:28.728775+0000 osd.3 (osd.3) 472 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.554376+0000 osd.5 (osd.5) 491 : cluster 0 3.c deep-scrub starts 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.559029+0000 osd.5 (osd.5) 492 : cluster 0 3.c deep-scrub ok 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:27.652628+0000 mgr.y (mgr.14550) 792 : cluster 0 pgmap v1371: 246 pgs: 32 unknown, 214 active+clean; 4.4 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:28.628336+0000 mon.a (mon.0) 3933 : cluster 0 osdmap e881: 8 total, 8 up, 8 in 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:28.723709+0000 osd.3 (osd.3) 471 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:43:29.467 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:29 smithi143 bash[29284]: cluster 2024-03-11T21:43:28.728775+0000 osd.3 (osd.3) 472 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:43:29.476 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:29.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:29.605 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:29.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:29.735 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:29.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:29.868 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:29.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:30.000 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:30.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:30.129 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:30.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:30.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:30.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:30.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:28.333429+0000 osd.6 (osd.6) 395 : cluster 0 2.17 scrub starts 2024-03-11T21:43:30.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:28.335016+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub ok 2024-03-11T21:43:30.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: audit 2024-03-11T21:43:29.104986+0000 mgr.y (mgr.14550) 793 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.111053+0000 osd.1 (osd.1) 277 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.115404+0000 osd.1 (osd.1) 278 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.251447+0000 osd.7 (osd.7) 466 : cluster 0 2.1d scrub starts 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.253863+0000 osd.7 (osd.7) 467 : cluster 0 2.1d scrub ok 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.641520+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e882: 8 total, 8 up, 8 in 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: audit 2024-03-11T21:43:29.645092+0000 mon.c (mon.2) 1357 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: audit 2024-03-11T21:43:29.645482+0000 mon.a (mon.0) 3935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.716151+0000 osd.3 (osd.3) 473 : cluster 0 9.c scrub starts 2024-03-11T21:43:30.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:30 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.719861+0000 osd.3 (osd.3) 474 : cluster 0 9.c scrub ok 2024-03-11T21:43:30.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:28.333429+0000 osd.6 (osd.6) 395 : cluster 0 2.17 scrub starts 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:28.335016+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub ok 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: audit 2024-03-11T21:43:29.104986+0000 mgr.y (mgr.14550) 793 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.111053+0000 osd.1 (osd.1) 277 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.115404+0000 osd.1 (osd.1) 278 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.251447+0000 osd.7 (osd.7) 466 : cluster 0 2.1d scrub starts 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.253863+0000 osd.7 (osd.7) 467 : cluster 0 2.1d scrub ok 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.641520+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e882: 8 total, 8 up, 8 in 2024-03-11T21:43:30.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: audit 2024-03-11T21:43:29.645092+0000 mon.c (mon.2) 1357 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: audit 2024-03-11T21:43:29.645482+0000 mon.a (mon.0) 3935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.716151+0000 osd.3 (osd.3) 473 : cluster 0 9.c scrub starts 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.719861+0000 osd.3 (osd.3) 474 : cluster 0 9.c scrub ok 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:28.333429+0000 osd.6 (osd.6) 395 : cluster 0 2.17 scrub starts 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:28.335016+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub ok 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: audit 2024-03-11T21:43:29.104986+0000 mgr.y (mgr.14550) 793 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.111053+0000 osd.1 (osd.1) 277 : cluster 0 3.4 deep-scrub starts 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.115404+0000 osd.1 (osd.1) 278 : cluster 0 3.4 deep-scrub ok 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.251447+0000 osd.7 (osd.7) 466 : cluster 0 2.1d scrub starts 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.253863+0000 osd.7 (osd.7) 467 : cluster 0 2.1d scrub ok 2024-03-11T21:43:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.641520+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e882: 8 total, 8 up, 8 in 2024-03-11T21:43:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: audit 2024-03-11T21:43:29.645092+0000 mon.c (mon.2) 1357 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: audit 2024-03-11T21:43:29.645482+0000 mon.a (mon.0) 3935 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.716151+0000 osd.3 (osd.3) 473 : cluster 0 9.c scrub starts 2024-03-11T21:43:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:30 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.719861+0000 osd.3 (osd.3) 474 : cluster 0 9.c scrub ok 2024-03-11T21:43:30.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.515 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:30.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.644 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:30.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:30.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/807329424 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c10e4a0 msgr2=0x7f681c10e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:30.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- 172.21.15.143:0/807329424 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c10e4a0 0x7f681c10e880 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f6810009990 tx=0x7f6810009ca0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/807329424 shutdown_connections 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- 172.21.15.143:0/807329424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c108dd0 0x7f681c1152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- 172.21.15.143:0/807329424 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c10e4a0 0x7f681c10e880 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- 172.21.15.143:0/807329424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/807329424 >> 172.21.15.143:0/807329424 conn(0x7f681c0755b0 msgr2=0x7f681c0779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/807329424 shutdown_connections 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/807329424 wait complete. 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 librados: starting msgr at 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 librados: starting objecter 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 Processor -- start 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- start start 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 librados: setting wanted keys 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 librados: calling monclient init 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 0x7f681c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:30.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c117bd0 con 0x7f681c10e4a0 2024-03-11T21:43:30.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c117a50 con 0x7f681c108410 2024-03-11T21:43:30.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c117d50 con 0x7f681c108dd0 2024-03-11T21:43:30.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:30.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 0x7f681c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:30.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:53672/0 (socket says 172.21.15.143:53672) 2024-03-11T21:43:30.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681ad9d700 1 -- 172.21.15.143:0/3389514015 learned_addr learned my addr 172.21.15.143:0/3389514015 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681a59c700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 -- 172.21.15.143:0/3389514015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 msgr2=0x7f681c19e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c19e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 -- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 msgr2=0x7f681c1a2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 -- 172.21.15.143:0/3389514015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c1a31d0 con 0x7f681c108dd0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681a59c700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f681ad9d700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6819d9b700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 0x7f681c19e700 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c970 tx=0x7f680c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f680b7fe700 10 client.?.objecter ms_handle_connect 0x7f681c108dd0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f680b7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c00ce60 con 0x7f681c108dd0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f680c004830 con 0x7f681c108dd0 2024-03-11T21:43:30.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0053a0 con 0x7f681c108dd0 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c1a34c0 con 0x7f681c108dd0 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 10 client.42113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1aad40 con 0x7f681c108dd0 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.773+0000 7f6820ac4700 1 librados: init done 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c04faa0 con 0x7f681c108dd0 2024-03-11T21:43:30.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f680c005540 con 0x7f681c108dd0 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 0x7f6804071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(883..883 src has 587..883) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f680c08e780 con 0x7f681c108dd0 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 10 client.42113.objecter ms_dispatch 0x7f681c001550 osd_map(883..883 src has 587..883) v4 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 3 client.42113.objecter handle_osd_map got epochs [883,883] > 0 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 3 client.42113.objecter handle_osd_map decoding full epoch 883 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f681a59c700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 0x7f6804071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 20 client.42113.objecter dump_active .. 0 homeless 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f680b7fe700 10 client.42113.objecter ms_handle_connect 0x7f680406f820 2024-03-11T21:43:30.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.777+0000 7f681a59c700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 0x7f6804071ce0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f681000b2e0 tx=0x7f6810005e80 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:30.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.781+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f680c014030 con 0x7f681c108dd0 2024-03-11T21:43:30.773 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:30.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.901 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:30.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:30.985+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f681c0760a0 con 0x7f680406f820 2024-03-11T21:43:31.001 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.013+0000 7f680b7fe700 1 -- 172.21.15.143:0/3389514015 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+530621 (secure 0 0 0) 0x7f681c0760a0 con 0x7f680406f820 2024-03-11T21:43:31.008 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 10 librados: watch_flush enter 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 10 librados: watch_flush exit 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 20 client.42113.objecter shutdown clearing up homeless session... 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 10 client.42113.objecter successfully canceled tick 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 msgr2=0x7f6804071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 0x7f6804071ce0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f681000b2e0 tx=0x7f6810005e80 comp rx=0 tx=0).stop 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 msgr2=0x7f681c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.017+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 0x7f681c19e700 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c970 tx=0x7f680c00cc80 comp rx=0 tx=0).stop 2024-03-11T21:43:31.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 shutdown_connections 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f680406f820 0x7f6804071ce0 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c10e4a0 0x7f681c1a2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c108410 0x7f681c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 --2- 172.21.15.143:0/3389514015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c108dd0 0x7f681c19e700 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 >> 172.21.15.143:0/3389514015 conn(0x7f681c0755b0 msgr2=0x7f681c075990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 shutdown_connections 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 -- 172.21.15.143:0/3389514015 wait complete. 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:31.021+0000 7f6820ac4700 1 librados: shutdown 2024-03-11T21:43:31.013 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:43:31.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:31.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:31.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:31.151 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: cluster 2024-03-11T21:43:29.653176+0000 mgr.y (mgr.14550) 794 : cluster 0 pgmap v1374: 206 pgs: 206 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:31.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: audit 2024-03-11T21:43:30.643800+0000 mon.a (mon.0) 3936 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: cluster 2024-03-11T21:43:30.649446+0000 mon.a (mon.0) 3937 : cluster 0 osdmap e883: 8 total, 8 up, 8 in 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: audit 2024-03-11T21:43:30.659292+0000 mon.c (mon.2) 1358 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: audit 2024-03-11T21:43:30.659751+0000 mon.a (mon.0) 3938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: cluster 2024-03-11T21:43:30.679770+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub starts 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[29284]: cluster 2024-03-11T21:43:30.682014+0000 osd.3 (osd.3) 476 : cluster 0 3.1e scrub ok 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: cluster 2024-03-11T21:43:29.653176+0000 mgr.y (mgr.14550) 794 : cluster 0 pgmap v1374: 206 pgs: 206 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: audit 2024-03-11T21:43:30.643800+0000 mon.a (mon.0) 3936 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: cluster 2024-03-11T21:43:30.649446+0000 mon.a (mon.0) 3937 : cluster 0 osdmap e883: 8 total, 8 up, 8 in 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: audit 2024-03-11T21:43:30.659292+0000 mon.c (mon.2) 1358 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: audit 2024-03-11T21:43:30.659751+0000 mon.a (mon.0) 3938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: cluster 2024-03-11T21:43:30.679770+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub starts 2024-03-11T21:43:31.154 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:31 smithi143 bash[21512]: cluster 2024-03-11T21:43:30.682014+0000 osd.3 (osd.3) 476 : cluster 0 3.1e scrub ok 2024-03-11T21:43:31.162 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:31.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:31.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: cluster 2024-03-11T21:43:29.653176+0000 mgr.y (mgr.14550) 794 : cluster 0 pgmap v1374: 206 pgs: 206 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:31.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: audit 2024-03-11T21:43:30.643800+0000 mon.a (mon.0) 3936 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:31.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: cluster 2024-03-11T21:43:30.649446+0000 mon.a (mon.0) 3937 : cluster 0 osdmap e883: 8 total, 8 up, 8 in 2024-03-11T21:43:31.277 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: audit 2024-03-11T21:43:30.659292+0000 mon.c (mon.2) 1358 : audit 1 from='client.? 172.21.15.143:0/1726746027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: audit 2024-03-11T21:43:30.659751+0000 mon.a (mon.0) 3938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]: dispatch 2024-03-11T21:43:31.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: cluster 2024-03-11T21:43:30.679770+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub starts 2024-03-11T21:43:31.278 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:31 smithi203 bash[23654]: cluster 2024-03-11T21:43:30.682014+0000 osd.3 (osd.3) 476 : cluster 0 3.1e scrub ok 2024-03-11T21:43:31.293 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:31.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.419 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:31.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8095 ms) 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13063 ms) 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8102 ms) 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12778 ms) 2024-03-11T21:43:31.602 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7992 ms) 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8042 ms) 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: ok, hit_set contains 327:602f83fe:::foo:head 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9042 ms) 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,0 2024-03-11T21:43:31.603 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first is 1710193326 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,1710193334,1710193335,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,1710193334,1710193335,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193326,1710193328,1710193329,1710193331,1710193332,1710193334,1710193335,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1710193329,1710193331,1710193332,1710193334,1710193335,1710193337,1710193338,0 2024-03-11T21:43:31.604 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first now 1710193329, trimmed 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20994 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: foo0 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14105 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18107 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22778 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5031 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (2958 ms) 2024-03-11T21:43:31.605 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3021 ms) 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3006 ms) 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (233110 ms total) 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (895939 ms total) 2024-03-11T21:43:31.606 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-11T21:43:31.611 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71623 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71623 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71816 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71816 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72076 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72076 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71945 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71945 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.612 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71850 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71850 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71702 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71702 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71924 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71924 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72197 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72197 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72234 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72234 2024-03-11T21:43:31.613 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71753 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71753 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72087 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72087 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72047 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72047 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71607 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71607 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.614 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71639 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71639 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72059 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72059 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71597 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71597 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71615 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71615 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72343 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72343 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.615 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71824 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71824 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72379 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72379 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71888 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71888 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72178 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72178 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72402 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72402 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.616 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71832 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71832 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72065 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72065 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72309 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72309 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72109 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72109 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72349 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72349 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.617 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72033 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72033 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71783 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71783 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72388 2024-03-11T21:43:31.618 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72388 2024-03-11T21:43:31.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.679 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:31.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:31.808 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:31.911 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:31.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:32.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.069 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:32.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:32.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:32.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:32.326 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:32.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- 172.21.15.143:0/1057865966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290102830 msgr2=0x7f729010ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- 172.21.15.143:0/1057865966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290102830 0x7f729010ece0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f728800b0a0 tx=0x7f728800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- 172.21.15.143:0/1057865966 shutdown_connections 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- 172.21.15.143:0/1057865966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290102830 0x7f729010ece0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- 172.21.15.143:0/1057865966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7290107ea0 0x7f7290108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- 172.21.15.143:0/1057865966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290101e90 0x7f72901022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- 172.21.15.143:0/1057865966 >> 172.21.15.143:0/1057865966 conn(0x7f7290078eb0 msgr2=0x7f72900792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:32.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- 172.21.15.143:0/1057865966 shutdown_connections 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- 172.21.15.143:0/1057865966 wait complete. 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 librados: starting msgr at 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 librados: starting objecter 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 Processor -- start 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- start start 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 librados: setting wanted keys 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 librados: calling monclient init 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7290101e90 0x7f729019d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290102830 0x7f729019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 0x7f7290197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290111600 con 0x7f7290107ea0 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290111480 con 0x7f7290101e90 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7296889700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7290111780 con 0x7f7290102830 2024-03-11T21:43:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f728f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290102830 0x7f729019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f728f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290102830 0x7f729019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:34536/0 (socket says 172.21.15.143:34536) 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f728f7fe700 1 -- 172.21.15.143:0/1073989988 learned_addr learned my addr 172.21.15.143:0/1073989988 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 0x7f7290197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7290101e90 0x7f729019d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290102830 msgr2=0x7f729019dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7290102830 0x7f729019dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7290101e90 msgr2=0x7f729019d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7290101e90 0x7f729019d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.341+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72901980d0 con 0x7f7290107ea0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 0x7f7290197970 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f728800b070 tx=0x7f7288004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 10 client.?.objecter ms_handle_connect 0x7f7290107ea0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288022070 con 0x7f7290107ea0 2024-03-11T21:43:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7290198360 con 0x7f7290107ea0 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7288007c40 con 0x7f7290107ea0 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288005280 con 0x7f7290107ea0 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 10 client.30915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7290074010 con 0x7f7290107ea0 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 librados: init done 2024-03-11T21:43:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7290103550 con 0x7f7290107ea0 2024-03-11T21:43:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f728802d020 con 0x7f7290107ea0 2024-03-11T21:43:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f727806f820 0x7f7278071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(884..884 src has 587..884) v4 ==== 9021+0+0 (secure 0 0 0) 0x7f728800f020 con 0x7f7290107ea0 2024-03-11T21:43:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 osd_map(884..884 src has 587..884) v4 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 3 client.30915.objecter handle_osd_map got epochs [884,884] > 0 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 3 client.30915.objecter handle_osd_map decoding full epoch 884 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f727806f820 0x7f7278071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 20 client.30915.objecter dump_active .. 0 homeless 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v884) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7288034070 con 0x7f7290107ea0 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 10 client.30915.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 1 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f72900629f0 0x7f7290062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 1 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 0 1 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 4 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 10 client.30915.objecter _send_command 1 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7290198ea0 con 0x7f72900629f0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f72900629f0 0x7f7290062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f727806f820 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f727806f820 0x7f7278071ce0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f7280005fd0 tx=0x7f7280005db0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f72900629f0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.345+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f72900629f0 0x7f7290062dd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.349+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7290198ea0 con 0x7f72900629f0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.349+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.349+0000 7f728d7fa700 10 client.30915.objecter _finish_command 1 = system:0 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.349+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 4 2024-03-11T21:43:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.349+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 0 1 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 10 client.30915.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 2 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 2 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729004faa0 osd=0 3 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 0 2 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 4 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 10 client.30915.objecter _send_command 2 2024-03-11T21:43:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.365+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7290062470 con 0x7f72900629f0 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7290062470 con 0x7f72900629f0 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 10 client.30915.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 4 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 0 2 2024-03-11T21:43:32.354 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 10 client.30915.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 3 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f729002d9a0 0x7f72900742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 3 2024-03-11T21:43:32.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 1 3 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 4 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 10 client.30915.objecter _send_command 3 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7290075fb0 con 0x7f729002d9a0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f729002d9a0 0x7f72900742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f729002d9a0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f729002d9a0 0x7f72900742d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7290075fb0 con 0x7f729002d9a0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 10 client.30915.objecter _finish_command 3 = system:0 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 4 2024-03-11T21:43:32.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.369+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 1 3 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 10 client.30915.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 4 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 4 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729002d830 osd=1 3 2024-03-11T21:43:32.373 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 1 4 2024-03-11T21:43:32.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 4 2024-03-11T21:43:32.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 10 client.30915.objecter _send_command 4 2024-03-11T21:43:32.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.385+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7290109a10 con 0x7f729002d9a0 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7290109a10 con 0x7f729002d9a0 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 10 client.30915.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 4 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 1 4 2024-03-11T21:43:32.375 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 10 client.30915.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 5 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7290108ea0 0x7f729002ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 5 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 2 5 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 4 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 10 client.30915.objecter _send_command 5 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f729002d3c0 con 0x7f7290108ea0 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7290108ea0 0x7f729002ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f7290108ea0 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7290108ea0 0x7f729002ce80 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f729002d3c0 con 0x7f7290108ea0 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 10 client.30915.objecter _finish_command 5 = system:0 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 4 2024-03-11T21:43:32.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.389+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 2 5 2024-03-11T21:43:32.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 10 client.30915.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 6 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 6 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f7290108d30 osd=2 3 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 2 6 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 4 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 10 client.30915.objecter _send_command 6 2024-03-11T21:43:32.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f72901aa6b0 con 0x7f7290108ea0 2024-03-11T21:43:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72901aa6b0 con 0x7f7290108ea0 2024-03-11T21:43:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 10 client.30915.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 4 2024-03-11T21:43:32.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 2 6 2024-03-11T21:43:32.396 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:43:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 10 client.30915.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 7 2024-03-11T21:43:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f729002ca70 0x7f72901aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 7 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 3 7 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 4 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 10 client.30915.objecter _send_command 7 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7290075670 con 0x7f729002ca70 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f729002ca70 0x7f72901aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f729002ca70 0x7f72901aba40 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f729002ca70 2024-03-11T21:43:32.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7290075670 con 0x7f729002ca70 2024-03-11T21:43:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:43:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 10 client.30915.objecter _finish_command 7 = system:0 2024-03-11T21:43:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 4 2024-03-11T21:43:32.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.409+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 3 7 2024-03-11T21:43:32.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 10 client.30915.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 8 2024-03-11T21:43:32.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 8 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f729002c900 osd=3 3 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 3 8 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 4 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 10 client.30915.objecter _send_command 8 2024-03-11T21:43:32.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7290074a60 con 0x7f729002ca70 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7290074a60 con 0x7f729002ca70 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 10 client.30915.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 4 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 3 8 2024-03-11T21:43:32.416 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 10 client.30915.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 9 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f72901ab5a0 0x7f72901b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 9 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 4 9 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 4 2024-03-11T21:43:32.417 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 10 client.30915.objecter _send_command 9 2024-03-11T21:43:32.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72901b4100 con 0x7f72901ab5a0 2024-03-11T21:43:32.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f72901ab5a0 0x7f72901b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f72901ab5a0 0x7f72901b1250 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.418 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.429+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f72901ab5a0 2024-03-11T21:43:32.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.433+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72901b4100 con 0x7f72901ab5a0 2024-03-11T21:43:32.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.433+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:43:32.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.433+0000 7f728d7fa700 10 client.30915.objecter _finish_command 9 = system:0 2024-03-11T21:43:32.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.433+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 4 2024-03-11T21:43:32.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.433+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 4 9 2024-03-11T21:43:32.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:32.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 10 client.30915.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 10 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 10 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f7290074a60 osd=4 3 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 4 10 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 4 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 10 client.30915.objecter _send_command 10 2024-03-11T21:43:32.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.449+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f72901aaab0 con 0x7f72901ab5a0 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: audit 2024-03-11T21:43:30.992307+0000 mgr.y (mgr.14550) 795 : audit 0 from='client.42113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.086268+0000 osd.1 (osd.1) 279 : cluster 0 39.17 scrub starts 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.090195+0000 osd.1 (osd.1) 280 : cluster 0 39.17 scrub ok 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.577429+0000 mon.a (mon.0) 3939 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: audit 2024-03-11T21:43:31.581865+0000 mon.a (mon.0) 3940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:32.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:32 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.590475+0000 mon.a (mon.0) 3941 : cluster 0 osdmap e884: 8 total, 8 up, 8 in 2024-03-11T21:43:32.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72901aaab0 con 0x7f72901ab5a0 2024-03-11T21:43:32.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f728d7fa700 10 client.30915.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 4 2024-03-11T21:43:32.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 4 10 2024-03-11T21:43:32.442 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 10 client.30915.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 11 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f72901aa860 0x7f72901b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 11 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 5 11 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 4 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 10 client.30915.objecter _send_command 11 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.453+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72901acd70 con 0x7f72901aa860 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f72901aa860 0x7f72901b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f72901aa860 2024-03-11T21:43:32.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728f7fe700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f72901aa860 0x7f72901b2910 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: audit 2024-03-11T21:43:30.992307+0000 mgr.y (mgr.14550) 795 : audit 0 from='client.42113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.086268+0000 osd.1 (osd.1) 279 : cluster 0 39.17 scrub starts 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.090195+0000 osd.1 (osd.1) 280 : cluster 0 39.17 scrub ok 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.577429+0000 mon.a (mon.0) 3939 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: audit 2024-03-11T21:43:31.581865+0000 mon.a (mon.0) 3940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.590475+0000 mon.a (mon.0) 3941 : cluster 0 osdmap e884: 8 total, 8 up, 8 in 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:32.445 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: audit 2024-03-11T21:43:30.992307+0000 mgr.y (mgr.14550) 795 : audit 0 from='client.42113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.086268+0000 osd.1 (osd.1) 279 : cluster 0 39.17 scrub starts 2024-03-11T21:43:32.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.090195+0000 osd.1 (osd.1) 280 : cluster 0 39.17 scrub ok 2024-03-11T21:43:32.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.577429+0000 mon.a (mon.0) 3939 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:43:32.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: audit 2024-03-11T21:43:31.581865+0000 mon.a (mon.0) 3940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi143-71773-111"}]': finished 2024-03-11T21:43:32.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:32 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.590475+0000 mon.a (mon.0) 3941 : cluster 0 osdmap e884: 8 total, 8 up, 8 in 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72901acd70 con 0x7f72901aa860 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 10 client.30915.objecter _finish_command 11 = system:0 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 4 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.457+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 5 11 2024-03-11T21:43:32.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 10 client.30915.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 12 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 12 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901aaf10 osd=5 3 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 5 12 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 4 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 10 client.30915.objecter _send_command 12 2024-03-11T21:43:32.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f72901acd70 con 0x7f72901aa860 2024-03-11T21:43:32.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72901acd70 con 0x7f72901aa860 2024-03-11T21:43:32.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 10 client.30915.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 4 2024-03-11T21:43:32.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 5 12 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 10 client.30915.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 13 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f72901b3360 0x7f72901b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 13 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 6 13 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 4 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 10 client.30915.objecter _send_command 13 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72901b3d50 con 0x7f72901b3360 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f72901b3360 0x7f72901b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f72901b3360 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.477+0000 7f7294e26700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f72901b3360 0x7f72901b37c0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.481+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72901b3d50 con 0x7f72901b3360 2024-03-11T21:43:32.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.481+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:43:32.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.481+0000 7f728d7fa700 10 client.30915.objecter _finish_command 13 = system:0 2024-03-11T21:43:32.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.481+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 4 2024-03-11T21:43:32.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.481+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 6 13 2024-03-11T21:43:32.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 10 client.30915.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 14 2024-03-11T21:43:32.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 14 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901aacc0 osd=6 3 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 6 14 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 4 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 10 client.30915.objecter _send_command 14 2024-03-11T21:43:32.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.497+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f72901b3d50 con 0x7f72901b3360 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72901b3d50 con 0x7f72901b3360 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 10 client.30915.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 4 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 6 14 2024-03-11T21:43:32.486 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 10 client.30915.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 15 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f72901b8630 0x7f72901baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 15 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 7 15 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 4 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 10 client.30915.objecter _send_command 15 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72901bdab0 con 0x7f72901b8630 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f72901b8630 0x7f72901baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 10 client.30915.objecter ms_handle_connect 0x7f72901b8630 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728ffff700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f72901b8630 0x7f72901baa60 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:32.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72901bdab0 con 0x7f72901b8630 2024-03-11T21:43:32.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:43:32.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 10 client.30915.objecter _finish_command 15 = system:0 2024-03-11T21:43:32.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 4 2024-03-11T21:43:32.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.501+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 7 15 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 10 client.30915.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign -1 16 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 20 client.30915.objecter _get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 15 client.30915.objecter _session_command_op_remove -1 16 2024-03-11T21:43:32.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 20 client.30915.objecter get_session s=0x7f72901acd70 osd=7 3 2024-03-11T21:43:32.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 15 client.30915.objecter _session_command_op_assign 7 16 2024-03-11T21:43:32.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 4 2024-03-11T21:43:32.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 10 client.30915.objecter _send_command 16 2024-03-11T21:43:32.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.521+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f72901bc880 con 0x7f72901b8630 2024-03-11T21:43:32.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f728d7fa700 1 -- 172.21.15.143:0/1073989988 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72901bc880 con 0x7f72901b8630 2024-03-11T21:43:32.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f728d7fa700 10 client.30915.objecter ms_dispatch 0x7f7290001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:32.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f728d7fa700 10 client.30915.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:32.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f728d7fa700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 4 2024-03-11T21:43:32.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f728d7fa700 15 client.30915.objecter _session_command_op_remove 7 16 2024-03-11T21:43:32.511 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:43:32.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 librados: watch_flush enter 2024-03-11T21:43:32.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 librados: watch_flush exit 2024-03-11T21:43:32.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.0 2024-03-11T21:43:32.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f72900629f0 msgr2=0x7f7290062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f72900629f0 0x7f7290062dd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729004faa0 osd=0 1 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.1 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f729002d9a0 msgr2=0x7f72900742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f729002d9a0 0x7f72900742d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002d830 osd=1 1 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.2 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7290108ea0 msgr2=0x7f729002ce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f7290108ea0 0x7f729002ce80 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290108d30 osd=2 1 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.3 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f729002ca70 msgr2=0x7f72901aba40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f729002ca70 0x7f72901aba40 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f729002c900 osd=3 1 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.4 2024-03-11T21:43:32.512 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f72901ab5a0 msgr2=0x7f72901b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f72901ab5a0 0x7f72901b1250 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f7290074a60 osd=4 1 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.5 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f72901aa860 msgr2=0x7f72901b2910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f72901aa860 0x7f72901b2910 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aaf10 osd=5 1 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.6 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f72901b3360 msgr2=0x7f72901b37c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f72901b3360 0x7f72901b37c0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901aacc0 osd=6 1 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter close_session for osd.7 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f72901b8630 msgr2=0x7f72901baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f72901b8630 0x7f72901baa60 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter put_session s=0x7f72901acd70 osd=7 1 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 20 client.30915.objecter shutdown clearing up homeless session... 2024-03-11T21:43:32.513 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 10 client.30915.objecter successfully canceled tick 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 reap_dead start 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 reap_dead start 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 reap_dead start 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 reap_dead start 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f727806f820 msgr2=0x7f7278071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f727806f820 0x7f7278071ce0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f7280005fd0 tx=0x7f7280005db0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7294e26700 1 -- 172.21.15.143:0/1073989988 reap_dead start 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 msgr2=0x7f7290197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 0x7f7290197970 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f728800b070 tx=0x7f7288004ce0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 shutdown_connections 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 --2- 172.21.15.143:0/1073989988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7290107ea0 0x7f7290197970 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 >> 172.21.15.143:0/1073989988 conn(0x7f7290078eb0 msgr2=0x7f7290100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 shutdown_connections 2024-03-11T21:43:32.514 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 -- 172.21.15.143:0/1073989988 wait complete. 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:32.525+0000 7f7296889700 1 librados: shutdown 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:32.515 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:32.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:32.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:32.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:32.717 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:32.718 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:32.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:32.826 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:32.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:32.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:33.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.087 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:33.101 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1375,"stamp":"2024-03-11T21:43:30.649474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2804,"num_read_kb":2556,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2589,"ondisk_log_size":2589,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":686,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":795544,"kb_used_data":119768,"kb_used_omap":38,"kb_used_meta":673625,"kb_avail":749165672,"statfs":{"total":767960285184,"available":767145648128,"internally_reserved":0,"allocated":122642432,"data_stored":80016425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":689792437},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000289,"num_objects":-43,"num_object_clones":-1,"num_object_copies":-129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":-22,"num_read_kb":-4,"num_write":-67,"num_write_kb":-3967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030368"},"pg_stats":[{"pgid":"39.18","version":"0'0","reported_seq":1514,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.636203+0000","last_change":"2024-03-11T21:42:59.786704+0000","last_active":"2024-03-11T21:43:27.636203+0000","last_peered":"2024-03-11T21:43:27.636203+0000","last_clean":"2024-03-11T21:43:27.636203+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:43:27.636203+0000","last_undegraded":"2024-03-11T21:43:27.636203+0000","last_fullsized":"2024-03-11T21:43:27.636203+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.786379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.786379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.633663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1534,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.636727+0000","last_change":"2024-03-11T21:43:26.358694+0000","last_active":"2024-03-11T21:43:27.636727+0000","last_peered":"2024-03-11T21:43:27.636727+0000","last_clean":"2024-03-11T21:43:27.636727+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:43:27.636727+0000","last_undegraded":"2024-03-11T21:43:27.636727+0000","last_fullsized":"2024-03-11T21:43:27.636727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.358597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.358597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:40.493651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1510,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638267+0000","last_change":"2024-03-11T21:41:57.330251+0000","last_active":"2024-03-11T21:43:27.638267+0000","last_peered":"2024-03-11T21:43:27.638267+0000","last_clean":"2024-03-11T21:43:27.638267+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:43:27.638267+0000","last_undegraded":"2024-03-11T21:43:27.638267+0000","last_fullsized":"2024-03-11T21:43:27.638267+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.330037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.330037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":1496,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646515+0000","last_change":"2024-03-11T21:40:25.470377+0000","last_active":"2024-03-11T21:43:27.646515+0000","last_peered":"2024-03-11T21:43:27.646515+0000","last_clean":"2024-03-11T21:43:27.646515+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:43:27.646515+0000","last_undegraded":"2024-03-11T21:43:27.646515+0000","last_fullsized":"2024-03-11T21:43:27.646515+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.470230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.470230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":1430,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643039+0000","last_change":"2024-03-11T21:41:56.250851+0000","last_active":"2024-03-11T21:43:29.643039+0000","last_peered":"2024-03-11T21:43:29.643039+0000","last_clean":"2024-03-11T21:43:29.643039+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:43:29.643039+0000","last_undegraded":"2024-03-11T21:43:29.643039+0000","last_fullsized":"2024-03-11T21:43:29.643039+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.250744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.250744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1533,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647680+0000","last_change":"2024-03-11T21:42:58.341631+0000","last_active":"2024-03-11T21:43:27.647680+0000","last_peered":"2024-03-11T21:43:27.647680+0000","last_clean":"2024-03-11T21:43:27.647680+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:43:27.647680+0000","last_undegraded":"2024-03-11T21:43:27.647680+0000","last_fullsized":"2024-03-11T21:43:27.647680+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.341432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.341432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.778503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1535,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.648560+0000","last_change":"2024-03-11T21:43:27.319867+0000","last_active":"2024-03-11T21:43:27.648560+0000","last_peered":"2024-03-11T21:43:27.648560+0000","last_clean":"2024-03-11T21:43:27.648560+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:43:27.648560+0000","last_undegraded":"2024-03-11T21:43:27.648560+0000","last_fullsized":"2024-03-11T21:43:27.648560+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.319760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.645321+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.319760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:39.518500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1548,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.627291+0000","last_change":"2024-03-11T21:43:17.510480+0000","last_active":"2024-03-11T21:43:27.627291+0000","last_peered":"2024-03-11T21:43:27.627291+0000","last_clean":"2024-03-11T21:43:27.627291+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:43:27.627291+0000","last_undegraded":"2024-03-11T21:43:27.627291+0000","last_fullsized":"2024-03-11T21:43:27.627291+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:17.510286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:17.510286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.842823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1535,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.623478+0000","last_change":"2024-03-11T21:43:14.817104+0000","last_active":"2024-03-11T21:43:27.623478+0000","last_peered":"2024-03-11T21:43:27.623478+0000","last_clean":"2024-03-11T21:43:27.623478+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:43:27.623478+0000","last_undegraded":"2024-03-11T21:43:27.623478+0000","last_fullsized":"2024-03-11T21:43:27.623478+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.816959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.816959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.630107+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1505,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638621+0000","last_change":"2024-03-11T21:41:41.521503+0000","last_active":"2024-03-11T21:43:27.638621+0000","last_peered":"2024-03-11T21:43:27.638621+0000","last_clean":"2024-03-11T21:43:27.638621+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:43:27.638621+0000","last_undegraded":"2024-03-11T21:43:27.638621+0000","last_fullsized":"2024-03-11T21:43:27.638621+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.521300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.521300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1517,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646644+0000","last_change":"2024-03-11T21:43:02.436956+0000","last_active":"2024-03-11T21:43:27.646644+0000","last_peered":"2024-03-11T21:43:27.646644+0000","last_clean":"2024-03-11T21:43:27.646644+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:43:27.646644+0000","last_undegraded":"2024-03-11T21:43:27.646644+0000","last_fullsized":"2024-03-11T21:43:27.646644+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.436843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.436843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.695057+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":1499,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658261+0000","last_change":"2024-03-11T21:41:59.422739+0000","last_active":"2024-03-11T21:43:27.658261+0000","last_peered":"2024-03-11T21:43:27.658261+0000","last_clean":"2024-03-11T21:43:27.658261+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:43:27.658261+0000","last_undegraded":"2024-03-11T21:43:27.658261+0000","last_fullsized":"2024-03-11T21:43:27.658261+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.422534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.422534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1517,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650135+0000","last_change":"2024-03-11T21:43:09.309127+0000","last_active":"2024-03-11T21:43:27.650135+0000","last_peered":"2024-03-11T21:43:27.650135+0000","last_clean":"2024-03-11T21:43:27.650135+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:43:27.650135+0000","last_undegraded":"2024-03-11T21:43:27.650135+0000","last_fullsized":"2024-03-11T21:43:27.650135+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:09.309029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.309029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.575434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1527,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.627051+0000","last_change":"2024-03-11T21:41:45.595342+0000","last_active":"2024-03-11T21:43:27.627051+0000","last_peered":"2024-03-11T21:43:27.627051+0000","last_clean":"2024-03-11T21:43:27.627051+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:43:27.627051+0000","last_undegraded":"2024-03-11T21:43:27.627051+0000","last_fullsized":"2024-03-11T21:43:27.627051+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.595129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.595129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1547,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628827+0000","last_change":"2024-03-11T21:43:05.601474+0000","last_active":"2024-03-11T21:43:27.628827+0000","last_peered":"2024-03-11T21:43:27.628827+0000","last_clean":"2024-03-11T21:43:27.628827+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:43:27.628827+0000","last_undegraded":"2024-03-11T21:43:27.628827+0000","last_fullsized":"2024-03-11T21:43:27.628827+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:05.601329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:05.601329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:21.104290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":926,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643869+0000","last_change":"2024-03-11T21:41:20.464811+0000","last_active":"2024-03-11T21:43:29.643869+0000","last_peered":"2024-03-11T21:43:29.643869+0000","last_clean":"2024-03-11T21:43:29.643869+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:43:29.643869+0000","last_undegraded":"2024-03-11T21:43:29.643869+0000","last_fullsized":"2024-03-11T21:43:29.643869+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.464589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.464589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1534,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637163+0000","last_change":"2024-03-11T21:43:02.319235+0000","last_active":"2024-03-11T21:43:27.637163+0000","last_peered":"2024-03-11T21:43:27.637163+0000","last_clean":"2024-03-11T21:43:27.637163+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:43:27.637163+0000","last_undegraded":"2024-03-11T21:43:27.637163+0000","last_fullsized":"2024-03-11T21:43:27.637163+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.319030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.319030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.439681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":1505,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650852+0000","last_change":"2024-03-11T21:41:51.387114+0000","last_active":"2024-03-11T21:43:27.650852+0000","last_peered":"2024-03-11T21:43:27.650852+0000","last_clean":"2024-03-11T21:43:27.650852+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:43:27.650852+0000","last_undegraded":"2024-03-11T21:43:27.650852+0000","last_fullsized":"2024-03-11T21:43:27.650852+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.386905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.386905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":1497,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.642244+0000","last_change":"2024-03-11T21:40:19.428421+0000","last_active":"2024-03-11T21:43:27.642244+0000","last_peered":"2024-03-11T21:43:27.642244+0000","last_clean":"2024-03-11T21:43:27.642244+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:43:27.642244+0000","last_undegraded":"2024-03-11T21:43:27.642244+0000","last_fullsized":"2024-03-11T21:43:27.642244+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.428205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.428205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1527,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625227+0000","last_change":"2024-03-11T21:41:40.664787+0000","last_active":"2024-03-11T21:43:27.625227+0000","last_peered":"2024-03-11T21:43:27.625227+0000","last_clean":"2024-03-11T21:43:27.625227+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:43:27.625227+0000","last_undegraded":"2024-03-11T21:43:27.625227+0000","last_fullsized":"2024-03-11T21:43:27.625227+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:40.664675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.664675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":1399,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644130+0000","last_change":"2024-03-11T21:41:32.440218+0000","last_active":"2024-03-11T21:43:29.644130+0000","last_peered":"2024-03-11T21:43:29.644130+0000","last_clean":"2024-03-11T21:43:29.644130+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:43:29.644130+0000","last_undegraded":"2024-03-11T21:43:29.644130+0000","last_fullsized":"2024-03-11T21:43:29.644130+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.440004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.440004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":1505,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.648966+0000","last_change":"2024-03-11T21:43:03.770710+0000","last_active":"2024-03-11T21:43:27.648966+0000","last_peered":"2024-03-11T21:43:27.648966+0000","last_clean":"2024-03-11T21:43:27.648966+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:43:27.648966+0000","last_undegraded":"2024-03-11T21:43:27.648966+0000","last_fullsized":"2024-03-11T21:43:27.648966+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:03.770563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:03.770563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.680813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649151+0000","last_change":"2024-03-11T21:42:55.708191+0000","last_active":"2024-03-11T21:43:27.649151+0000","last_peered":"2024-03-11T21:43:27.649151+0000","last_clean":"2024-03-11T21:43:27.649151+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:43:27.649151+0000","last_undegraded":"2024-03-11T21:43:27.649151+0000","last_fullsized":"2024-03-11T21:43:27.649151+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.707930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.707930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.053621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1517,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658393+0000","last_change":"2024-03-11T21:43:01.415713+0000","last_active":"2024-03-11T21:43:27.658393+0000","last_peered":"2024-03-11T21:43:27.658393+0000","last_clean":"2024-03-11T21:43:27.658393+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:43:27.658393+0000","last_undegraded":"2024-03-11T21:43:27.658393+0000","last_fullsized":"2024-03-11T21:43:27.658393+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.415616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.415616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.957916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1512,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638304+0000","last_change":"2024-03-11T21:42:55.545333+0000","last_active":"2024-03-11T21:43:27.638304+0000","last_peered":"2024-03-11T21:43:27.638304+0000","last_clean":"2024-03-11T21:43:27.638304+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:43:27.638304+0000","last_undegraded":"2024-03-11T21:43:27.638304+0000","last_fullsized":"2024-03-11T21:43:27.638304+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.640588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":1500,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.651188+0000","last_change":"2024-03-11T21:43:21.483812+0000","last_active":"2024-03-11T21:43:27.651188+0000","last_peered":"2024-03-11T21:43:27.651188+0000","last_clean":"2024-03-11T21:43:27.651188+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:43:27.651188+0000","last_undegraded":"2024-03-11T21:43:27.651188+0000","last_fullsized":"2024-03-11T21:43:27.651188+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:21.483712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:21.483712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.108340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":935,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643417+0000","last_change":"2024-03-11T21:43:20.116935+0000","last_active":"2024-03-11T21:43:29.643417+0000","last_peered":"2024-03-11T21:43:29.643417+0000","last_clean":"2024-03-11T21:43:29.643417+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:43:29.643417+0000","last_undegraded":"2024-03-11T21:43:29.643417+0000","last_fullsized":"2024-03-11T21:43:29.643417+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.116862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.116862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:48.214569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1528,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625355+0000","last_change":"2024-03-11T21:41:51.650830+0000","last_active":"2024-03-11T21:43:27.625355+0000","last_peered":"2024-03-11T21:43:27.625355+0000","last_clean":"2024-03-11T21:43:27.625355+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:43:27.625355+0000","last_undegraded":"2024-03-11T21:43:27.625355+0000","last_fullsized":"2024-03-11T21:43:27.625355+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.650722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.650722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1518,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650245+0000","last_change":"2024-03-11T21:43:23.296720+0000","last_active":"2024-03-11T21:43:27.650245+0000","last_peered":"2024-03-11T21:43:27.650245+0000","last_clean":"2024-03-11T21:43:27.650245+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:43:27.650245+0000","last_undegraded":"2024-03-11T21:43:27.650245+0000","last_fullsized":"2024-03-11T21:43:27.650245+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:23.296514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.296514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.542454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":1500,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.659941+0000","last_change":"2024-03-11T21:42:56.351908+0000","last_active":"2024-03-11T21:43:27.659941+0000","last_peered":"2024-03-11T21:43:27.659941+0000","last_clean":"2024-03-11T21:43:27.659941+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:43:27.659941+0000","last_undegraded":"2024-03-11T21:43:27.659941+0000","last_fullsized":"2024-03-11T21:43:27.659941+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.351542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.351542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:06.855022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1547,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628658+0000","last_change":"2024-03-11T21:43:19.499150+0000","last_active":"2024-03-11T21:43:27.628658+0000","last_peered":"2024-03-11T21:43:27.628658+0000","last_clean":"2024-03-11T21:43:27.628658+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:43:27.628658+0000","last_undegraded":"2024-03-11T21:43:27.628658+0000","last_fullsized":"2024-03-11T21:43:27.628658+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:19.499002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.499002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.961116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":1399,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643960+0000","last_change":"2024-03-11T21:41:30.411743+0000","last_active":"2024-03-11T21:43:29.643960+0000","last_peered":"2024-03-11T21:43:29.643960+0000","last_clean":"2024-03-11T21:43:29.643960+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:43:29.643960+0000","last_undegraded":"2024-03-11T21:43:29.643960+0000","last_fullsized":"2024-03-11T21:43:29.643960+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:30.411631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.411631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1524,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652714+0000","last_change":"2024-03-11T21:42:52.535387+0000","last_active":"2024-03-11T21:43:27.652714+0000","last_peered":"2024-03-11T21:43:27.652714+0000","last_clean":"2024-03-11T21:43:27.652714+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:43:27.652714+0000","last_undegraded":"2024-03-11T21:43:27.652714+0000","last_fullsized":"2024-03-11T21:43:27.652714+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.535087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.535087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.542418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1906,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652940+0000","last_change":"2024-03-11T21:41:21.484873+0000","last_active":"2024-03-11T21:43:27.652940+0000","last_peered":"2024-03-11T21:43:27.652940+0000","last_clean":"2024-03-11T21:43:27.652940+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:43:27.652940+0000","last_undegraded":"2024-03-11T21:43:27.652940+0000","last_fullsized":"2024-03-11T21:43:27.652940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:21.484757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.484757+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"864'39","reported_seq":1716,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.662160+0000","last_change":"2024-03-11T21:41:26.485912+0000","last_active":"2024-03-11T21:43:27.662160+0000","last_peered":"2024-03-11T21:43:27.662160+0000","last_clean":"2024-03-11T21:43:27.662160+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:43:27.662160+0000","last_undegraded":"2024-03-11T21:43:27.662160+0000","last_fullsized":"2024-03-11T21:43:27.662160+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:41:26.485646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.485646+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1715,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.629023+0000","last_change":"2024-03-11T21:43:18.507612+0000","last_active":"2024-03-11T21:43:27.629023+0000","last_peered":"2024-03-11T21:43:27.629023+0000","last_clean":"2024-03-11T21:43:27.629023+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:43:27.629023+0000","last_undegraded":"2024-03-11T21:43:27.629023+0000","last_fullsized":"2024-03-11T21:43:27.629023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:18.507512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:18.507512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:39.158678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1660,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.639228+0000","last_change":"2024-03-11T21:41:45.252015+0000","last_active":"2024-03-11T21:43:27.639228+0000","last_peered":"2024-03-11T21:43:27.639228+0000","last_clean":"2024-03-11T21:43:27.639228+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:43:27.639228+0000","last_undegraded":"2024-03-11T21:43:27.639228+0000","last_fullsized":"2024-03-11T21:43:27.639228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.251730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.251730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1526,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655204+0000","last_change":"2024-03-11T21:43:16.573624+0000","last_active":"2024-03-11T21:43:27.655204+0000","last_peered":"2024-03-11T21:43:27.655204+0000","last_clean":"2024-03-11T21:43:27.655204+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:43:27.655204+0000","last_undegraded":"2024-03-11T21:43:27.655204+0000","last_fullsized":"2024-03-11T21:43:27.655204+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.573343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.573343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.314480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"864'30","reported_seq":1721,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.643009+0000","last_change":"2024-03-11T21:43:27.559460+0000","last_active":"2024-03-11T21:43:27.643009+0000","last_peered":"2024-03-11T21:43:27.643009+0000","last_clean":"2024-03-11T21:43:27.643009+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:43:27.643009+0000","last_undegraded":"2024-03-11T21:43:27.643009+0000","last_fullsized":"2024-03-11T21:43:27.643009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_deep_scrub":"864'30","last_deep_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.559051+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.066844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1589,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642977+0000","last_change":"2024-03-11T21:43:25.032236+0000","last_active":"2024-03-11T21:43:29.642977+0000","last_peered":"2024-03-11T21:43:29.642977+0000","last_clean":"2024-03-11T21:43:29.642977+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:43:29.642977+0000","last_undegraded":"2024-03-11T21:43:29.642977+0000","last_fullsized":"2024-03-11T21:43:29.642977+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.032097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.032097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:52.373268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1707,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625831+0000","last_change":"2024-03-11T21:43:06.604303+0000","last_active":"2024-03-11T21:43:27.625831+0000","last_peered":"2024-03-11T21:43:27.625831+0000","last_clean":"2024-03-11T21:43:27.625831+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:43:27.625831+0000","last_undegraded":"2024-03-11T21:43:27.625831+0000","last_fullsized":"2024-03-11T21:43:27.625831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:06.604059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.604059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.007219+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1666,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646654+0000","last_change":"2024-03-11T21:41:52.458683+0000","last_active":"2024-03-11T21:43:27.646654+0000","last_peered":"2024-03-11T21:43:27.646654+0000","last_clean":"2024-03-11T21:43:27.646654+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:43:27.646654+0000","last_undegraded":"2024-03-11T21:43:27.646654+0000","last_fullsized":"2024-03-11T21:43:27.646654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:52.458535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:41:52.458535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1517,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658602+0000","last_change":"2024-03-11T21:41:55.275905+0000","last_active":"2024-03-11T21:43:27.658602+0000","last_peered":"2024-03-11T21:43:27.658602+0000","last_clean":"2024-03-11T21:43:27.658602+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:43:27.658602+0000","last_undegraded":"2024-03-11T21:43:27.658602+0000","last_fullsized":"2024-03-11T21:43:27.658602+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.275680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.275680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"864'53","reported_seq":1741,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.661093+0000","last_change":"2024-03-11T21:42:53.391146+0000","last_active":"2024-03-11T21:43:27.661093+0000","last_peered":"2024-03-11T21:43:27.661093+0000","last_clean":"2024-03-11T21:43:27.661093+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:43:27.661093+0000","last_undegraded":"2024-03-11T21:43:27.661093+0000","last_fullsized":"2024-03-11T21:43:27.661093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:42:53.390867+0000","last_deep_scrub":"389'39","last_deep_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.390867+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.412714+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1715,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628091+0000","last_change":"2024-03-11T21:43:12.491271+0000","last_active":"2024-03-11T21:43:27.628091+0000","last_peered":"2024-03-11T21:43:27.628091+0000","last_clean":"2024-03-11T21:43:27.628091+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:43:27.628091+0000","last_undegraded":"2024-03-11T21:43:27.628091+0000","last_fullsized":"2024-03-11T21:43:27.628091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:12.491049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:43:12.491049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.211389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1707,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.626706+0000","last_change":"2024-03-11T21:43:20.514020+0000","last_active":"2024-03-11T21:43:27.626706+0000","last_peered":"2024-03-11T21:43:27.626706+0000","last_clean":"2024-03-11T21:43:27.626706+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:43:27.626706+0000","last_undegraded":"2024-03-11T21:43:27.626706+0000","last_fullsized":"2024-03-11T21:43:27.626706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.513928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.513928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.755239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1655,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650354+0000","last_change":"2024-03-11T21:43:14.517541+0000","last_active":"2024-03-11T21:43:27.650354+0000","last_peered":"2024-03-11T21:43:27.650354+0000","last_clean":"2024-03-11T21:43:27.650354+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:43:27.650354+0000","last_undegraded":"2024-03-11T21:43:27.650354+0000","last_fullsized":"2024-03-11T21:43:27.650354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.517349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.517349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.900019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":904,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637455+0000","last_change":"2024-03-11T21:42:53.579836+0000","last_active":"2024-03-11T21:43:27.637455+0000","last_peered":"2024-03-11T21:43:27.637455+0000","last_clean":"2024-03-11T21:43:27.637455+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:43:27.637455+0000","last_undegraded":"2024-03-11T21:43:27.637455+0000","last_fullsized":"2024-03-11T21:43:27.637455+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:42:53.579671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.579671+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.307026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1408,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643327+0000","last_change":"2024-03-11T21:43:05.859471+0000","last_active":"2024-03-11T21:43:29.643327+0000","last_peered":"2024-03-11T21:43:29.643327+0000","last_clean":"2024-03-11T21:43:29.643327+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:43:29.643327+0000","last_undegraded":"2024-03-11T21:43:29.643327+0000","last_fullsized":"2024-03-11T21:43:29.643327+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:05.859254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:43:05.859254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.817814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"864'60","reported_seq":1795,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652718+0000","last_change":"2024-03-11T21:43:19.376362+0000","last_active":"2024-03-11T21:43:27.652718+0000","last_peered":"2024-03-11T21:43:27.652718+0000","last_clean":"2024-03-11T21:43:27.652718+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:43:27.652718+0000","last_undegraded":"2024-03-11T21:43:27.652718+0000","last_fullsized":"2024-03-11T21:43:27.652718+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'60","last_scrub_stamp":"2024-03-11T21:43:19.376099+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.376099+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.890107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1667,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656696+0000","last_change":"2024-03-11T21:41:29.509226+0000","last_active":"2024-03-11T21:43:27.656696+0000","last_peered":"2024-03-11T21:43:27.656696+0000","last_clean":"2024-03-11T21:43:27.656696+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:43:27.656696+0000","last_undegraded":"2024-03-11T21:43:27.656696+0000","last_fullsized":"2024-03-11T21:43:27.656696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.509105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.509105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1587,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643194+0000","last_change":"2024-03-11T21:42:00.292135+0000","last_active":"2024-03-11T21:43:29.643194+0000","last_peered":"2024-03-11T21:43:29.643194+0000","last_clean":"2024-03-11T21:43:29.643194+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:43:29.643194+0000","last_undegraded":"2024-03-11T21:43:29.643194+0000","last_fullsized":"2024-03-11T21:43:29.643194+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.292001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1703,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628069+0000","last_change":"2024-03-11T21:42:55.510765+0000","last_active":"2024-03-11T21:43:27.628069+0000","last_peered":"2024-03-11T21:43:27.628069+0000","last_clean":"2024-03-11T21:43:27.628069+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:43:27.628069+0000","last_undegraded":"2024-03-11T21:43:27.628069+0000","last_fullsized":"2024-03-11T21:43:27.628069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.510373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.510373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.380386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1948,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.653242+0000","last_change":"2024-03-11T21:41:30.525150+0000","last_active":"2024-03-11T21:43:27.653242+0000","last_peered":"2024-03-11T21:43:27.653242+0000","last_clean":"2024-03-11T21:43:27.653242+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:43:27.653242+0000","last_undegraded":"2024-03-11T21:43:27.653242+0000","last_fullsized":"2024-03-11T21:43:27.653242+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:30.525009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.525009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1529,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.639221+0000","last_change":"2024-03-11T21:43:24.734683+0000","last_active":"2024-03-11T21:43:27.639221+0000","last_peered":"2024-03-11T21:43:27.639221+0000","last_clean":"2024-03-11T21:43:27.639221+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:43:27.639221+0000","last_undegraded":"2024-03-11T21:43:27.639221+0000","last_fullsized":"2024-03-11T21:43:27.639221+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:43:24.734459+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.734459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.277343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"864'21","reported_seq":1706,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647612+0000","last_change":"2024-03-11T21:43:01.814079+0000","last_active":"2024-03-11T21:43:27.647612+0000","last_peered":"2024-03-11T21:43:27.647612+0000","last_clean":"2024-03-11T21:43:27.647612+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:43:27.647612+0000","last_undegraded":"2024-03-11T21:43:27.647612+0000","last_fullsized":"2024-03-11T21:43:27.647612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:43:01.813932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.813932+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.924935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1555,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643318+0000","last_change":"2024-03-11T21:43:09.964115+0000","last_active":"2024-03-11T21:43:29.643318+0000","last_peered":"2024-03-11T21:43:29.643318+0000","last_clean":"2024-03-11T21:43:29.643318+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:43:29.643318+0000","last_undegraded":"2024-03-11T21:43:29.643318+0000","last_fullsized":"2024-03-11T21:43:29.643318+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:09.964027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.964027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.470467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1963,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.626818+0000","last_change":"2024-03-11T21:42:58.546849+0000","last_active":"2024-03-11T21:43:27.626818+0000","last_peered":"2024-03-11T21:43:27.626818+0000","last_clean":"2024-03-11T21:43:27.626818+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:43:27.626818+0000","last_undegraded":"2024-03-11T21:43:27.626818+0000","last_fullsized":"2024-03-11T21:43:27.626818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:42:58.546638+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.546638+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.143964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1700,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.626975+0000","last_change":"2024-03-11T21:43:11.646001+0000","last_active":"2024-03-11T21:43:27.626975+0000","last_peered":"2024-03-11T21:43:27.626975+0000","last_clean":"2024-03-11T21:43:27.626975+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:43:27.626975+0000","last_undegraded":"2024-03-11T21:43:27.626975+0000","last_fullsized":"2024-03-11T21:43:27.626975+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:11.645880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.645880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:40.786412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":904,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637765+0000","last_change":"2024-03-11T21:43:24.638284+0000","last_active":"2024-03-11T21:43:27.637765+0000","last_peered":"2024-03-11T21:43:27.637765+0000","last_clean":"2024-03-11T21:43:27.637765+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:43:27.637765+0000","last_undegraded":"2024-03-11T21:43:27.637765+0000","last_fullsized":"2024-03-11T21:43:27.637765+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:24.638122+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:43:24.638122+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.638122+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.581584+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1524,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.645458+0000","last_change":"2024-03-11T21:41:25.364494+0000","last_active":"2024-03-11T21:43:27.645458+0000","last_peered":"2024-03-11T21:43:27.645458+0000","last_clean":"2024-03-11T21:43:27.645458+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:43:27.645458+0000","last_undegraded":"2024-03-11T21:43:27.645458+0000","last_fullsized":"2024-03-11T21:43:27.645458+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.364387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.364387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"864'39","reported_seq":1742,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637259+0000","last_change":"2024-03-11T21:43:11.702556+0000","last_active":"2024-03-11T21:43:27.637259+0000","last_peered":"2024-03-11T21:43:27.637259+0000","last_clean":"2024-03-11T21:43:27.637259+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:43:27.637259+0000","last_undegraded":"2024-03-11T21:43:27.637259+0000","last_fullsized":"2024-03-11T21:43:27.637259+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:43:11.702298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.702298+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.406412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1583,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642902+0000","last_change":"2024-03-11T21:42:55.086019+0000","last_active":"2024-03-11T21:43:29.642902+0000","last_peered":"2024-03-11T21:43:29.642902+0000","last_clean":"2024-03-11T21:43:29.642902+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:43:29.642902+0000","last_undegraded":"2024-03-11T21:43:29.642902+0000","last_fullsized":"2024-03-11T21:43:29.642902+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.085823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.085823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.857865+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1942,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.626623+0000","last_change":"2024-03-11T21:41:46.623952+0000","last_active":"2024-03-11T21:43:27.626623+0000","last_peered":"2024-03-11T21:43:27.626623+0000","last_clean":"2024-03-11T21:43:27.626623+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:43:27.626623+0000","last_undegraded":"2024-03-11T21:43:27.626623+0000","last_fullsized":"2024-03-11T21:43:27.626623+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:46.623794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.623794+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1677,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.643838+0000","last_change":"2024-03-11T21:41:58.366463+0000","last_active":"2024-03-11T21:43:27.643838+0000","last_peered":"2024-03-11T21:43:27.643838+0000","last_clean":"2024-03-11T21:43:27.643838+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:43:27.643838+0000","last_undegraded":"2024-03-11T21:43:27.643838+0000","last_fullsized":"2024-03-11T21:43:27.643838+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.366196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.366196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1514,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655416+0000","last_change":"2024-03-11T21:42:52.391444+0000","last_active":"2024-03-11T21:43:27.655416+0000","last_peered":"2024-03-11T21:43:27.655416+0000","last_clean":"2024-03-11T21:43:27.655416+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:43:27.655416+0000","last_undegraded":"2024-03-11T21:43:27.655416+0000","last_fullsized":"2024-03-11T21:43:27.655416+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.391288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.391288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.913321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"864'29","reported_seq":1737,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622098+0000","last_change":"2024-03-11T21:43:02.625794+0000","last_active":"2024-03-11T21:43:27.622098+0000","last_peered":"2024-03-11T21:43:27.622098+0000","last_clean":"2024-03-11T21:43:27.622098+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:43:27.622098+0000","last_undegraded":"2024-03-11T21:43:27.622098+0000","last_fullsized":"2024-03-11T21:43:27.622098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:43:02.625424+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.625424+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.526217+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1676,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650853+0000","last_change":"2024-03-11T21:41:18.465526+0000","last_active":"2024-03-11T21:43:27.650853+0000","last_peered":"2024-03-11T21:43:27.650853+0000","last_clean":"2024-03-11T21:43:27.650853+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:43:27.650853+0000","last_undegraded":"2024-03-11T21:43:27.650853+0000","last_fullsized":"2024-03-11T21:43:27.650853+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.465412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.465412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1672,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.661364+0000","last_change":"2024-03-11T21:43:26.557894+0000","last_active":"2024-03-11T21:43:27.661364+0000","last_peered":"2024-03-11T21:43:27.661364+0000","last_clean":"2024-03-11T21:43:27.661364+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:43:27.661364+0000","last_undegraded":"2024-03-11T21:43:27.661364+0000","last_fullsized":"2024-03-11T21:43:27.661364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.557601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.557601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:43.491296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1656,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649500+0000","last_change":"2024-03-11T21:41:26.327523+0000","last_active":"2024-03-11T21:43:27.649500+0000","last_peered":"2024-03-11T21:43:27.649500+0000","last_clean":"2024-03-11T21:43:27.649500+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:43:27.649500+0000","last_undegraded":"2024-03-11T21:43:27.649500+0000","last_fullsized":"2024-03-11T21:43:27.649500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.327416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.327416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":898,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649600+0000","last_change":"2024-03-11T21:41:57.696381+0000","last_active":"2024-03-11T21:43:27.649600+0000","last_peered":"2024-03-11T21:43:27.649600+0000","last_clean":"2024-03-11T21:43:27.649600+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:43:27.649600+0000","last_undegraded":"2024-03-11T21:43:27.649600+0000","last_fullsized":"2024-03-11T21:43:27.649600+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:41:57.696205+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.696205+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1506,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649699+0000","last_change":"2024-03-11T21:40:03.109654+0000","last_active":"2024-03-11T21:43:27.649699+0000","last_peered":"2024-03-11T21:43:27.649699+0000","last_clean":"2024-03-11T21:43:27.649699+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:43:27.649699+0000","last_undegraded":"2024-03-11T21:43:27.649699+0000","last_fullsized":"2024-03-11T21:43:27.649699+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.109518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.109518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1934,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637892+0000","last_change":"2024-03-11T21:43:26.745028+0000","last_active":"2024-03-11T21:43:27.637892+0000","last_peered":"2024-03-11T21:43:27.637892+0000","last_clean":"2024-03-11T21:43:27.637892+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:43:27.637892+0000","last_undegraded":"2024-03-11T21:43:27.637892+0000","last_fullsized":"2024-03-11T21:43:27.637892+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:43:26.744813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.744813+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.970870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1704,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622443+0000","last_change":"2024-03-11T21:42:56.717568+0000","last_active":"2024-03-11T21:43:27.622443+0000","last_peered":"2024-03-11T21:43:27.622443+0000","last_clean":"2024-03-11T21:43:27.622443+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:43:27.622443+0000","last_undegraded":"2024-03-11T21:43:27.622443+0000","last_fullsized":"2024-03-11T21:43:27.622443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.717329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.717329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.987508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"864'36","reported_seq":1745,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655569+0000","last_change":"2024-03-11T21:43:25.284142+0000","last_active":"2024-03-11T21:43:27.655569+0000","last_peered":"2024-03-11T21:43:27.655569+0000","last_clean":"2024-03-11T21:43:27.655569+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:43:27.655569+0000","last_undegraded":"2024-03-11T21:43:27.655569+0000","last_fullsized":"2024-03-11T21:43:27.655569+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'36","last_scrub_stamp":"2024-03-11T21:43:25.283990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.283990+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:47.098678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1711,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652822+0000","last_change":"2024-03-11T21:41:55.649415+0000","last_active":"2024-03-11T21:43:27.652822+0000","last_peered":"2024-03-11T21:43:27.652822+0000","last_clean":"2024-03-11T21:43:27.652822+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:43:27.652822+0000","last_undegraded":"2024-03-11T21:43:27.652822+0000","last_fullsized":"2024-03-11T21:43:27.652822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:41:55.649085+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.649085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1657,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655891+0000","last_change":"2024-03-11T21:40:23.749973+0000","last_active":"2024-03-11T21:43:27.655891+0000","last_peered":"2024-03-11T21:43:27.655891+0000","last_clean":"2024-03-11T21:43:27.655891+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:43:27.655891+0000","last_undegraded":"2024-03-11T21:43:27.655891+0000","last_fullsized":"2024-03-11T21:43:27.655891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:40:23.749772+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.749772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":911,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625349+0000","last_change":"2024-03-11T21:41:56.716081+0000","last_active":"2024-03-11T21:43:27.625349+0000","last_peered":"2024-03-11T21:43:27.625349+0000","last_clean":"2024-03-11T21:43:27.625349+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:43:27.625349+0000","last_undegraded":"2024-03-11T21:43:27.625349+0000","last_fullsized":"2024-03-11T21:43:27.625349+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:56.715868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.715868+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.664350+0000","last_change":"2024-03-11T21:41:42.428955+0000","last_active":"2024-03-11T21:43:27.664350+0000","last_peered":"2024-03-11T21:43:27.664350+0000","last_clean":"2024-03-11T21:43:27.664350+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:43:27.664350+0000","last_undegraded":"2024-03-11T21:43:27.664350+0000","last_fullsized":"2024-03-11T21:43:27.664350+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.428709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.428709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1695,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622105+0000","last_change":"2024-03-11T21:41:42.630400+0000","last_active":"2024-03-11T21:43:27.622105+0000","last_peered":"2024-03-11T21:43:27.622105+0000","last_clean":"2024-03-11T21:43:27.622105+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:43:27.622105+0000","last_undegraded":"2024-03-11T21:43:27.622105+0000","last_fullsized":"2024-03-11T21:43:27.622105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.630272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.630272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2876,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637542+0000","last_change":"2024-03-11T21:41:20.351387+0000","last_active":"2024-03-11T21:43:27.637542+0000","last_peered":"2024-03-11T21:43:27.637542+0000","last_clean":"2024-03-11T21:43:27.637542+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:43:27.637542+0000","last_undegraded":"2024-03-11T21:43:27.637542+0000","last_fullsized":"2024-03-11T21:43:27.637542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:41:20.351150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.351150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1203,"num_read_kb":1198,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"864'21","reported_seq":1703,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656140+0000","last_change":"2024-03-11T21:42:03.753084+0000","last_active":"2024-03-11T21:43:27.656140+0000","last_peered":"2024-03-11T21:43:27.656140+0000","last_clean":"2024-03-11T21:43:27.656140+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:43:27.656140+0000","last_undegraded":"2024-03-11T21:43:27.656140+0000","last_fullsized":"2024-03-11T21:43:27.656140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:42:03.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:03.752924+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1608,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643133+0000","last_change":"2024-03-11T21:43:07.910395+0000","last_active":"2024-03-11T21:43:29.643133+0000","last_peered":"2024-03-11T21:43:29.643133+0000","last_clean":"2024-03-11T21:43:29.643133+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:43:29.643133+0000","last_undegraded":"2024-03-11T21:43:29.643133+0000","last_fullsized":"2024-03-11T21:43:29.643133+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:43:07.910034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.910034+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:36.399679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1691,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.623050+0000","last_change":"2024-03-11T21:43:18.721177+0000","last_active":"2024-03-11T21:43:27.623050+0000","last_peered":"2024-03-11T21:43:27.623050+0000","last_clean":"2024-03-11T21:43:27.623050+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:43:27.623050+0000","last_undegraded":"2024-03-11T21:43:27.623050+0000","last_fullsized":"2024-03-11T21:43:27.623050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:18.720924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:43:18.720924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.027183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":920,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647803+0000","last_change":"2024-03-11T21:43:09.365565+0000","last_active":"2024-03-11T21:43:27.647803+0000","last_peered":"2024-03-11T21:43:27.647803+0000","last_clean":"2024-03-11T21:43:27.647803+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:43:27.647803+0000","last_undegraded":"2024-03-11T21:43:27.647803+0000","last_fullsized":"2024-03-11T21:43:27.647803+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:09.365340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.365340+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.184435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660393+0000","last_change":"2024-03-11T21:41:35.427640+0000","last_active":"2024-03-11T21:43:27.660393+0000","last_peered":"2024-03-11T21:43:27.660393+0000","last_clean":"2024-03-11T21:43:27.660393+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:43:27.660393+0000","last_undegraded":"2024-03-11T21:43:27.660393+0000","last_fullsized":"2024-03-11T21:43:27.660393+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.427438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.427438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1686,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652444+0000","last_change":"2024-03-11T21:43:24.336932+0000","last_active":"2024-03-11T21:43:27.652444+0000","last_peered":"2024-03-11T21:43:27.652444+0000","last_clean":"2024-03-11T21:43:27.652444+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:43:27.652444+0000","last_undegraded":"2024-03-11T21:43:27.652444+0000","last_fullsized":"2024-03-11T21:43:27.652444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.336783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.336783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.429760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"864'66","reported_seq":1908,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644565+0000","last_change":"2024-03-11T21:43:29.115547+0000","last_active":"2024-03-11T21:43:29.644565+0000","last_peered":"2024-03-11T21:43:29.644565+0000","last_clean":"2024-03-11T21:43:29.644565+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:43:29.644565+0000","last_undegraded":"2024-03-11T21:43:29.644565+0000","last_fullsized":"2024-03-11T21:43:29.644565+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'66","last_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_deep_scrub":"864'66","last_deep_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.115409+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:58.235006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1667,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660773+0000","last_change":"2024-03-11T21:41:44.440095+0000","last_active":"2024-03-11T21:43:27.660773+0000","last_peered":"2024-03-11T21:43:27.660773+0000","last_clean":"2024-03-11T21:43:27.660773+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:43:27.660773+0000","last_undegraded":"2024-03-11T21:43:27.660773+0000","last_fullsized":"2024-03-11T21:43:27.660773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:44.439846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:44.439846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1547,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642420+0000","last_change":"2024-03-11T21:43:02.848311+0000","last_active":"2024-03-11T21:43:29.642420+0000","last_peered":"2024-03-11T21:43:29.642420+0000","last_clean":"2024-03-11T21:43:29.642420+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:43:29.642420+0000","last_undegraded":"2024-03-11T21:43:29.642420+0000","last_fullsized":"2024-03-11T21:43:29.642420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.848093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.848093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.818328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":917,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642512+0000","last_change":"2024-03-11T21:43:08.956187+0000","last_active":"2024-03-11T21:43:29.642512+0000","last_peered":"2024-03-11T21:43:29.642512+0000","last_clean":"2024-03-11T21:43:29.642512+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:43:29.642512+0000","last_undegraded":"2024-03-11T21:43:29.642512+0000","last_fullsized":"2024-03-11T21:43:29.642512+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:08.955941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.955941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:24.648893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1523,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650041+0000","last_change":"2024-03-11T21:42:54.461064+0000","last_active":"2024-03-11T21:43:27.650041+0000","last_peered":"2024-03-11T21:43:27.650041+0000","last_clean":"2024-03-11T21:43:27.650041+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:43:27.650041+0000","last_undegraded":"2024-03-11T21:43:27.650041+0000","last_fullsized":"2024-03-11T21:43:27.650041+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.460846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.460846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.439479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1681,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660007+0000","last_change":"2024-03-11T21:43:21.554709+0000","last_active":"2024-03-11T21:43:27.660007+0000","last_peered":"2024-03-11T21:43:27.660007+0000","last_clean":"2024-03-11T21:43:27.660007+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:43:27.660007+0000","last_undegraded":"2024-03-11T21:43:27.660007+0000","last_fullsized":"2024-03-11T21:43:27.660007+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:21.554345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:21.554345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:50.053679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1677,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650546+0000","last_change":"2024-03-11T21:42:04.407830+0000","last_active":"2024-03-11T21:43:27.650546+0000","last_peered":"2024-03-11T21:43:27.650546+0000","last_clean":"2024-03-11T21:43:27.650546+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:43:27.650546+0000","last_undegraded":"2024-03-11T21:43:27.650546+0000","last_fullsized":"2024-03-11T21:43:27.650546+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:04.407615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:42:04.407615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"864'42","reported_seq":1737,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646166+0000","last_change":"2024-03-11T21:42:00.572712+0000","last_active":"2024-03-11T21:43:27.646166+0000","last_peered":"2024-03-11T21:43:27.646166+0000","last_clean":"2024-03-11T21:43:27.646166+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:43:27.646166+0000","last_undegraded":"2024-03-11T21:43:27.646166+0000","last_fullsized":"2024-03-11T21:43:27.646166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:00.572462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.572462+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1549,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644398+0000","last_change":"2024-03-11T21:41:33.473130+0000","last_active":"2024-03-11T21:43:29.644398+0000","last_peered":"2024-03-11T21:43:29.644398+0000","last_clean":"2024-03-11T21:43:29.644398+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:43:29.644398+0000","last_undegraded":"2024-03-11T21:43:29.644398+0000","last_fullsized":"2024-03-11T21:43:29.644398+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.472920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.472920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1514,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.636464+0000","last_change":"2024-03-11T21:42:01.660218+0000","last_active":"2024-03-11T21:43:27.636464+0000","last_peered":"2024-03-11T21:43:27.636464+0000","last_clean":"2024-03-11T21:43:27.636464+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:43:27.636464+0000","last_undegraded":"2024-03-11T21:43:27.636464+0000","last_fullsized":"2024-03-11T21:43:27.636464+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.660109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.660109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:31.505443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1651,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.659617+0000","last_change":"2024-03-11T21:40:22.764357+0000","last_active":"2024-03-11T21:43:27.659617+0000","last_peered":"2024-03-11T21:43:27.659617+0000","last_clean":"2024-03-11T21:43:27.659617+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:43:27.659617+0000","last_undegraded":"2024-03-11T21:43:27.659617+0000","last_fullsized":"2024-03-11T21:43:27.659617+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:22.764240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:22.764240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1941,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650655+0000","last_change":"2024-03-11T21:41:41.451026+0000","last_active":"2024-03-11T21:43:27.650655+0000","last_peered":"2024-03-11T21:43:27.650655+0000","last_clean":"2024-03-11T21:43:27.650655+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:43:27.650655+0000","last_undegraded":"2024-03-11T21:43:27.650655+0000","last_fullsized":"2024-03-11T21:43:27.650655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:41.450896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.450896+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1680,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638953+0000","last_change":"2024-03-11T21:43:10.494102+0000","last_active":"2024-03-11T21:43:27.638953+0000","last_peered":"2024-03-11T21:43:27.638953+0000","last_clean":"2024-03-11T21:43:27.638953+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:43:27.638953+0000","last_undegraded":"2024-03-11T21:43:27.638953+0000","last_fullsized":"2024-03-11T21:43:27.638953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.493878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.493878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.695374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"864'30","reported_seq":1709,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.644150+0000","last_change":"2024-03-11T21:41:35.415804+0000","last_active":"2024-03-11T21:43:27.644150+0000","last_peered":"2024-03-11T21:43:27.644150+0000","last_clean":"2024-03-11T21:43:27.644150+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:43:27.644150+0000","last_undegraded":"2024-03-11T21:43:27.644150+0000","last_fullsized":"2024-03-11T21:43:27.644150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:35.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.415630+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":470,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658160+0000","last_change":"2024-03-11T21:41:37.415103+0000","last_active":"2024-03-11T21:43:27.658160+0000","last_peered":"2024-03-11T21:43:27.658160+0000","last_clean":"2024-03-11T21:43:27.658160+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:43:27.658160+0000","last_undegraded":"2024-03-11T21:43:27.658160+0000","last_fullsized":"2024-03-11T21:43:27.658160+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.414919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.414919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1948,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655148+0000","last_change":"2024-03-11T21:41:38.433752+0000","last_active":"2024-03-11T21:43:27.655148+0000","last_peered":"2024-03-11T21:43:27.655148+0000","last_clean":"2024-03-11T21:43:27.655148+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:43:27.655148+0000","last_undegraded":"2024-03-11T21:43:27.655148+0000","last_fullsized":"2024-03-11T21:43:27.655148+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:38.433381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:38.433381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1526,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638535+0000","last_change":"2024-03-11T21:43:19.522646+0000","last_active":"2024-03-11T21:43:27.638535+0000","last_peered":"2024-03-11T21:43:27.638535+0000","last_clean":"2024-03-11T21:43:27.638535+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:43:27.638535+0000","last_undegraded":"2024-03-11T21:43:27.638535+0000","last_fullsized":"2024-03-11T21:43:27.638535+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:19.522430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.522430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.951836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1707,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.629479+0000","last_change":"2024-03-11T21:43:04.644845+0000","last_active":"2024-03-11T21:43:27.629479+0000","last_peered":"2024-03-11T21:43:27.629479+0000","last_clean":"2024-03-11T21:43:27.629479+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:43:27.629479+0000","last_undegraded":"2024-03-11T21:43:27.629479+0000","last_fullsized":"2024-03-11T21:43:27.629479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:04.644707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.644707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.280834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1715,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625690+0000","last_change":"2024-03-11T21:43:11.524842+0000","last_active":"2024-03-11T21:43:27.625690+0000","last_peered":"2024-03-11T21:43:27.625690+0000","last_clean":"2024-03-11T21:43:27.625690+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:43:27.625690+0000","last_undegraded":"2024-03-11T21:43:27.625690+0000","last_fullsized":"2024-03-11T21:43:27.625690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:11.524712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.524712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.769096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"864'50","reported_seq":1649,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643893+0000","last_change":"2024-03-11T21:41:27.465101+0000","last_active":"2024-03-11T21:43:29.643893+0000","last_peered":"2024-03-11T21:43:29.643893+0000","last_clean":"2024-03-11T21:43:29.643893+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:43:29.643893+0000","last_undegraded":"2024-03-11T21:43:29.643893+0000","last_fullsized":"2024-03-11T21:43:29.643893+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'38","last_scrub_stamp":"2024-03-11T21:41:27.464990+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.464990+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1655,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660505+0000","last_change":"2024-03-11T21:43:01.436177+0000","last_active":"2024-03-11T21:43:27.660505+0000","last_peered":"2024-03-11T21:43:27.660505+0000","last_clean":"2024-03-11T21:43:27.660505+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:43:27.660505+0000","last_undegraded":"2024-03-11T21:43:27.660505+0000","last_fullsized":"2024-03-11T21:43:27.660505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.435975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.435975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.229611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":896,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.643976+0000","last_change":"2024-03-11T21:41:18.182642+0000","last_active":"2024-03-11T21:43:27.643976+0000","last_peered":"2024-03-11T21:43:27.643976+0000","last_clean":"2024-03-11T21:43:27.643976+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:43:27.643976+0000","last_undegraded":"2024-03-11T21:43:27.643976+0000","last_fullsized":"2024-03-11T21:43:27.643976+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:18.182513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.182513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1523,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.645382+0000","last_change":"2024-03-11T21:42:01.549509+0000","last_active":"2024-03-11T21:43:27.645382+0000","last_peered":"2024-03-11T21:43:27.645382+0000","last_clean":"2024-03-11T21:43:27.645382+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:43:27.645382+0000","last_undegraded":"2024-03-11T21:43:27.645382+0000","last_fullsized":"2024-03-11T21:43:27.645382+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.549350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1679,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652422+0000","last_change":"2024-03-11T21:41:58.613981+0000","last_active":"2024-03-11T21:43:27.652422+0000","last_peered":"2024-03-11T21:43:27.652422+0000","last_clean":"2024-03-11T21:43:27.652422+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:43:27.652422+0000","last_undegraded":"2024-03-11T21:43:27.652422+0000","last_fullsized":"2024-03-11T21:43:27.652422+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.613685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.613685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"864'47","reported_seq":1765,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.624206+0000","last_change":"2024-03-11T21:41:29.674697+0000","last_active":"2024-03-11T21:43:27.624206+0000","last_peered":"2024-03-11T21:43:27.624206+0000","last_clean":"2024-03-11T21:43:27.624206+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:43:27.624206+0000","last_undegraded":"2024-03-11T21:43:27.624206+0000","last_fullsized":"2024-03-11T21:43:27.624206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:41:29.674458+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.674458+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1681,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.639549+0000","last_change":"2024-03-11T21:41:34.232210+0000","last_active":"2024-03-11T21:43:27.639549+0000","last_peered":"2024-03-11T21:43:27.639549+0000","last_clean":"2024-03-11T21:43:27.639549+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:43:27.639549+0000","last_undegraded":"2024-03-11T21:43:27.639549+0000","last_fullsized":"2024-03-11T21:43:27.639549+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:34.231996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:34.231996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1656,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.654941+0000","last_change":"2024-03-11T21:41:17.378484+0000","last_active":"2024-03-11T21:43:27.654941+0000","last_peered":"2024-03-11T21:43:27.654941+0000","last_clean":"2024-03-11T21:43:27.654941+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:43:27.654941+0000","last_undegraded":"2024-03-11T21:43:27.654941+0000","last_fullsized":"2024-03-11T21:43:27.654941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.378291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.378291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":920,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.653765+0000","last_change":"2024-03-11T21:43:25.374247+0000","last_active":"2024-03-11T21:43:27.653765+0000","last_peered":"2024-03-11T21:43:27.653765+0000","last_clean":"2024-03-11T21:43:27.653765+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:43:27.653765+0000","last_undegraded":"2024-03-11T21:43:27.653765+0000","last_fullsized":"2024-03-11T21:43:27.653765+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:25.374049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.374049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:48.553744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1526,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658667+0000","last_change":"2024-03-11T21:43:07.463108+0000","last_active":"2024-03-11T21:43:27.658667+0000","last_peered":"2024-03-11T21:43:27.658667+0000","last_clean":"2024-03-11T21:43:27.658667+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:43:27.658667+0000","last_undegraded":"2024-03-11T21:43:27.658667+0000","last_fullsized":"2024-03-11T21:43:27.658667+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.462866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.462866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.089106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1929,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649877+0000","last_change":"2024-03-11T21:41:40.335413+0000","last_active":"2024-03-11T21:43:27.649877+0000","last_peered":"2024-03-11T21:43:27.649877+0000","last_clean":"2024-03-11T21:43:27.649877+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:43:27.649877+0000","last_undegraded":"2024-03-11T21:43:27.649877+0000","last_fullsized":"2024-03-11T21:43:27.649877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:41:40.335178+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.335178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1584,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642836+0000","last_change":"2024-03-11T21:42:59.949425+0000","last_active":"2024-03-11T21:43:29.642836+0000","last_peered":"2024-03-11T21:43:29.642836+0000","last_clean":"2024-03-11T21:43:29.642836+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:43:29.642836+0000","last_undegraded":"2024-03-11T21:43:29.642836+0000","last_fullsized":"2024-03-11T21:43:29.642836+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:42:59.949164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.949164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.748601+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"864'40","reported_seq":1979,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.644000+0000","last_change":"2024-03-11T21:42:56.554589+0000","last_active":"2024-03-11T21:43:27.644000+0000","last_peered":"2024-03-11T21:43:27.644000+0000","last_clean":"2024-03-11T21:43:27.644000+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:43:27.644000+0000","last_undegraded":"2024-03-11T21:43:27.644000+0000","last_fullsized":"2024-03-11T21:43:27.644000+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:56.554326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.554326+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.354495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1555,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644229+0000","last_change":"2024-03-11T21:43:14.059791+0000","last_active":"2024-03-11T21:43:29.644229+0000","last_peered":"2024-03-11T21:43:29.644229+0000","last_clean":"2024-03-11T21:43:29.644229+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:43:29.644229+0000","last_undegraded":"2024-03-11T21:43:29.644229+0000","last_fullsized":"2024-03-11T21:43:29.644229+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.059672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:14.059672+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.059672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.082658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1674,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647090+0000","last_change":"2024-03-11T21:43:20.299173+0000","last_active":"2024-03-11T21:43:27.647090+0000","last_peered":"2024-03-11T21:43:27.647090+0000","last_clean":"2024-03-11T21:43:27.647090+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:43:27.647090+0000","last_undegraded":"2024-03-11T21:43:27.647090+0000","last_fullsized":"2024-03-11T21:43:27.647090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.432022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1416,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644319+0000","last_change":"2024-03-11T21:42:54.061349+0000","last_active":"2024-03-11T21:43:29.644319+0000","last_peered":"2024-03-11T21:43:29.644319+0000","last_clean":"2024-03-11T21:43:29.644319+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:43:29.644319+0000","last_undegraded":"2024-03-11T21:43:29.644319+0000","last_fullsized":"2024-03-11T21:43:29.644319+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.061137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.061137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.609578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1716,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625617+0000","last_change":"2024-03-11T21:43:14.527193+0000","last_active":"2024-03-11T21:43:27.625617+0000","last_peered":"2024-03-11T21:43:27.625617+0000","last_clean":"2024-03-11T21:43:27.625617+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:43:27.625617+0000","last_undegraded":"2024-03-11T21:43:27.625617+0000","last_fullsized":"2024-03-11T21:43:27.625617+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.526975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.526975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.358244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"864'32","reported_seq":1728,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660872+0000","last_change":"2024-03-11T21:43:22.461842+0000","last_active":"2024-03-11T21:43:27.660872+0000","last_peered":"2024-03-11T21:43:27.660872+0000","last_clean":"2024-03-11T21:43:27.660872+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:43:27.660872+0000","last_undegraded":"2024-03-11T21:43:27.660872+0000","last_fullsized":"2024-03-11T21:43:27.660872+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:43:22.461621+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.461621+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:46.244505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1721,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.627316+0000","last_change":"2024-03-11T21:42:57.712087+0000","last_active":"2024-03-11T21:43:27.627316+0000","last_peered":"2024-03-11T21:43:27.627316+0000","last_clean":"2024-03-11T21:43:27.627316+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:43:27.627316+0000","last_undegraded":"2024-03-11T21:43:27.627316+0000","last_fullsized":"2024-03-11T21:43:27.627316+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:42:57.711946+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.711946+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.971914+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1665,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.648241+0000","last_change":"2024-03-11T21:40:24.477381+0000","last_active":"2024-03-11T21:43:27.648241+0000","last_peered":"2024-03-11T21:43:27.648241+0000","last_clean":"2024-03-11T21:43:27.648241+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:43:27.648241+0000","last_undegraded":"2024-03-11T21:43:27.648241+0000","last_fullsized":"2024-03-11T21:43:27.648241+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.477246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.477246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655302+0000","last_change":"2024-03-11T21:41:15.373061+0000","last_active":"2024-03-11T21:43:27.655302+0000","last_peered":"2024-03-11T21:43:27.655302+0000","last_clean":"2024-03-11T21:43:27.655302+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:43:27.655302+0000","last_undegraded":"2024-03-11T21:43:27.655302+0000","last_fullsized":"2024-03-11T21:43:27.655302+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.372842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.372842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"864'27","reported_seq":1705,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652196+0000","last_change":"2024-03-11T21:42:59.435802+0000","last_active":"2024-03-11T21:43:27.652196+0000","last_peered":"2024-03-11T21:43:27.652196+0000","last_clean":"2024-03-11T21:43:27.652196+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:43:27.652196+0000","last_undegraded":"2024-03-11T21:43:27.652196+0000","last_fullsized":"2024-03-11T21:43:27.652196+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:42:59.435557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.435557+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:08.758869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1723,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625612+0000","last_change":"2024-03-11T21:43:15.569698+0000","last_active":"2024-03-11T21:43:27.625612+0000","last_peered":"2024-03-11T21:43:27.625612+0000","last_clean":"2024-03-11T21:43:27.625612+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:43:27.625612+0000","last_undegraded":"2024-03-11T21:43:27.625612+0000","last_fullsized":"2024-03-11T21:43:27.625612+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.569560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.569560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.367926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1668,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656474+0000","last_change":"2024-03-11T21:41:18.342623+0000","last_active":"2024-03-11T21:43:27.656474+0000","last_peered":"2024-03-11T21:43:27.656474+0000","last_clean":"2024-03-11T21:43:27.656474+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:43:27.656474+0000","last_undegraded":"2024-03-11T21:43:27.656474+0000","last_fullsized":"2024-03-11T21:43:27.656474+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.342525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.342525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1702,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628994+0000","last_change":"2024-03-11T21:42:52.491927+0000","last_active":"2024-03-11T21:43:27.628994+0000","last_peered":"2024-03-11T21:43:27.628994+0000","last_clean":"2024-03-11T21:43:27.628994+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:43:27.628994+0000","last_undegraded":"2024-03-11T21:43:27.628994+0000","last_fullsized":"2024-03-11T21:43:27.628994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.491711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.491711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.191914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.645742+0000","last_change":"2024-03-11T21:41:37.401402+0000","last_active":"2024-03-11T21:43:27.645742+0000","last_peered":"2024-03-11T21:43:27.645742+0000","last_clean":"2024-03-11T21:43:27.645742+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:43:27.645742+0000","last_undegraded":"2024-03-11T21:43:27.645742+0000","last_fullsized":"2024-03-11T21:43:27.645742+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.401275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.401275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"864'14","reported_seq":1717,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637594+0000","last_change":"2024-03-11T21:41:49.695635+0000","last_active":"2024-03-11T21:43:27.637594+0000","last_peered":"2024-03-11T21:43:27.637594+0000","last_clean":"2024-03-11T21:43:27.637594+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:43:27.637594+0000","last_undegraded":"2024-03-11T21:43:27.637594+0000","last_fullsized":"2024-03-11T21:43:27.637594+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:41:49.695261+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.695261+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1702,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.640630+0000","last_change":"2024-03-11T21:42:05.515337+0000","last_active":"2024-03-11T21:43:27.640630+0000","last_peered":"2024-03-11T21:43:27.640630+0000","last_clean":"2024-03-11T21:43:27.640630+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:43:27.640630+0000","last_undegraded":"2024-03-11T21:43:27.640630+0000","last_fullsized":"2024-03-11T21:43:27.640630+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:42:05.515038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:33.818265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1672,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646807+0000","last_change":"2024-03-11T21:41:42.239297+0000","last_active":"2024-03-11T21:43:27.646807+0000","last_peered":"2024-03-11T21:43:27.646807+0000","last_clean":"2024-03-11T21:43:27.646807+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:43:27.646807+0000","last_undegraded":"2024-03-11T21:43:27.646807+0000","last_fullsized":"2024-03-11T21:43:27.646807+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.239053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1654,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660201+0000","last_change":"2024-03-11T21:42:58.421950+0000","last_active":"2024-03-11T21:43:27.660201+0000","last_peered":"2024-03-11T21:43:27.660201+0000","last_clean":"2024-03-11T21:43:27.660201+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:43:27.660201+0000","last_undegraded":"2024-03-11T21:43:27.660201+0000","last_fullsized":"2024-03-11T21:43:27.660201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.421800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.421800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.437625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1514,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.641022+0000","last_change":"2024-03-11T21:42:54.754828+0000","last_active":"2024-03-11T21:43:27.641022+0000","last_peered":"2024-03-11T21:43:27.641022+0000","last_clean":"2024-03-11T21:43:27.641022+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:43:27.641022+0000","last_undegraded":"2024-03-11T21:43:27.641022+0000","last_fullsized":"2024-03-11T21:43:27.641022+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.754521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.754521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.619900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"864'27","reported_seq":1705,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.657105+0000","last_change":"2024-03-11T21:40:26.728613+0000","last_active":"2024-03-11T21:43:27.657105+0000","last_peered":"2024-03-11T21:43:27.657105+0000","last_clean":"2024-03-11T21:43:27.657105+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:43:27.657105+0000","last_undegraded":"2024-03-11T21:43:27.657105+0000","last_fullsized":"2024-03-11T21:43:27.657105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:26.728512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.728512+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1687,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.641936+0000","last_change":"2024-03-11T21:43:15.538577+0000","last_active":"2024-03-11T21:43:27.641936+0000","last_peered":"2024-03-11T21:43:27.641936+0000","last_clean":"2024-03-11T21:43:27.641936+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:43:27.641936+0000","last_undegraded":"2024-03-11T21:43:27.641936+0000","last_fullsized":"2024-03-11T21:43:27.641936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.538363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.538363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:22.713162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1928,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.644749+0000","last_change":"2024-03-11T21:41:49.267753+0000","last_active":"2024-03-11T21:43:27.644749+0000","last_peered":"2024-03-11T21:43:27.644749+0000","last_clean":"2024-03-11T21:43:27.644749+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:43:27.644749+0000","last_undegraded":"2024-03-11T21:43:27.644749+0000","last_fullsized":"2024-03-11T21:43:27.644749+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:41:49.267522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.267522+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1673,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660085+0000","last_change":"2024-03-11T21:43:10.331236+0000","last_active":"2024-03-11T21:43:27.660085+0000","last_peered":"2024-03-11T21:43:27.660085+0000","last_clean":"2024-03-11T21:43:27.660085+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:43:27.660085+0000","last_undegraded":"2024-03-11T21:43:27.660085+0000","last_fullsized":"2024-03-11T21:43:27.660085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.331124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.331124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:11.391846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1546,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638383+0000","last_change":"2024-03-11T21:42:57.297254+0000","last_active":"2024-03-11T21:43:27.638383+0000","last_peered":"2024-03-11T21:43:27.638383+0000","last_clean":"2024-03-11T21:43:27.638383+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:43:27.638383+0000","last_undegraded":"2024-03-11T21:43:27.638383+0000","last_fullsized":"2024-03-11T21:43:27.638383+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:57.297086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.297086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.243930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"864'25","reported_seq":1722,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646945+0000","last_change":"2024-03-11T21:43:17.259134+0000","last_active":"2024-03-11T21:43:27.646945+0000","last_peered":"2024-03-11T21:43:27.646945+0000","last_clean":"2024-03-11T21:43:27.646945+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:43:27.646945+0000","last_undegraded":"2024-03-11T21:43:27.646945+0000","last_fullsized":"2024-03-11T21:43:27.646945+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:43:17.258875+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:43:17.258875+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:24.144182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1685,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649616+0000","last_change":"2024-03-11T21:43:26.269090+0000","last_active":"2024-03-11T21:43:27.649616+0000","last_peered":"2024-03-11T21:43:27.649616+0000","last_clean":"2024-03-11T21:43:27.649616+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:43:27.649616+0000","last_undegraded":"2024-03-11T21:43:27.649616+0000","last_fullsized":"2024-03-11T21:43:27.649616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.268959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.268959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.544045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1915,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649165+0000","last_change":"2024-03-11T21:40:18.396071+0000","last_active":"2024-03-11T21:43:27.649165+0000","last_peered":"2024-03-11T21:43:27.649165+0000","last_clean":"2024-03-11T21:43:27.649165+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:43:27.649165+0000","last_undegraded":"2024-03-11T21:43:27.649165+0000","last_fullsized":"2024-03-11T21:43:27.649165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:40:18.395966+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:26:56.252414+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.395966+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1690,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.628012+0000","last_change":"2024-03-11T21:41:57.724943+0000","last_active":"2024-03-11T21:43:27.628012+0000","last_peered":"2024-03-11T21:43:27.628012+0000","last_clean":"2024-03-11T21:43:27.628012+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:43:27.628012+0000","last_undegraded":"2024-03-11T21:43:27.628012+0000","last_fullsized":"2024-03-11T21:43:27.628012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.724694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:26.895410+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.724694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1404,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644223+0000","last_change":"2024-03-11T21:41:55.220405+0000","last_active":"2024-03-11T21:43:29.644223+0000","last_peered":"2024-03-11T21:43:29.644223+0000","last_clean":"2024-03-11T21:43:29.644223+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:43:29.644223+0000","last_undegraded":"2024-03-11T21:43:29.644223+0000","last_fullsized":"2024-03-11T21:43:29.644223+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.220196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.220196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"864'25","reported_seq":1700,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.661647+0000","last_change":"2024-03-11T21:41:53.406126+0000","last_active":"2024-03-11T21:43:27.661647+0000","last_peered":"2024-03-11T21:43:27.661647+0000","last_clean":"2024-03-11T21:43:27.661647+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:43:27.661647+0000","last_undegraded":"2024-03-11T21:43:27.661647+0000","last_fullsized":"2024-03-11T21:43:27.661647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:41:53.405780+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.405780+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1709,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.645002+0000","last_change":"2024-03-11T21:43:15.412289+0000","last_active":"2024-03-11T21:43:27.645002+0000","last_peered":"2024-03-11T21:43:27.645002+0000","last_clean":"2024-03-11T21:43:27.645002+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:43:27.645002+0000","last_undegraded":"2024-03-11T21:43:27.645002+0000","last_fullsized":"2024-03-11T21:43:27.645002+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.412070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.526362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1668,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647893+0000","last_change":"2024-03-11T21:41:31.332398+0000","last_active":"2024-03-11T21:43:27.647893+0000","last_peered":"2024-03-11T21:43:27.647893+0000","last_clean":"2024-03-11T21:43:27.647893+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:43:27.647893+0000","last_undegraded":"2024-03-11T21:43:27.647893+0000","last_fullsized":"2024-03-11T21:43:27.647893+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.332299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.332299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1681,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646446+0000","last_change":"2024-03-11T21:43:22.269136+0000","last_active":"2024-03-11T21:43:27.646446+0000","last_peered":"2024-03-11T21:43:27.646446+0000","last_clean":"2024-03-11T21:43:27.646446+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:43:27.646446+0000","last_undegraded":"2024-03-11T21:43:27.646446+0000","last_fullsized":"2024-03-11T21:43:27.646446+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:22.269027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:22.269027+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.269027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.701754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1530,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652258+0000","last_change":"2024-03-11T21:42:58.427672+0000","last_active":"2024-03-11T21:43:27.652258+0000","last_peered":"2024-03-11T21:43:27.652258+0000","last_clean":"2024-03-11T21:43:27.652258+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:43:27.652258+0000","last_undegraded":"2024-03-11T21:43:27.652258+0000","last_fullsized":"2024-03-11T21:43:27.652258+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.427446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.427446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.731909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"864'30","reported_seq":1751,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625215+0000","last_change":"2024-03-11T21:43:06.683699+0000","last_active":"2024-03-11T21:43:27.625215+0000","last_peered":"2024-03-11T21:43:27.625215+0000","last_clean":"2024-03-11T21:43:27.625215+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:43:27.625215+0000","last_undegraded":"2024-03-11T21:43:27.625215+0000","last_fullsized":"2024-03-11T21:43:27.625215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:43:06.683455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.683455+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:22.624977+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1598,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643323+0000","last_change":"2024-03-11T21:43:27.076256+0000","last_active":"2024-03-11T21:43:29.643323+0000","last_peered":"2024-03-11T21:43:29.643323+0000","last_clean":"2024-03-11T21:43:29.643323+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:43:29.643323+0000","last_undegraded":"2024-03-11T21:43:29.643323+0000","last_fullsized":"2024-03-11T21:43:29.643323+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.357081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1567,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642559+0000","last_change":"2024-03-11T21:43:16.065946+0000","last_active":"2024-03-11T21:43:29.642559+0000","last_peered":"2024-03-11T21:43:29.642559+0000","last_clean":"2024-03-11T21:43:29.642559+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:43:29.642559+0000","last_undegraded":"2024-03-11T21:43:29.642559+0000","last_fullsized":"2024-03-11T21:43:29.642559+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.065817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.065817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.278869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1655,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.639661+0000","last_change":"2024-03-11T21:41:21.351275+0000","last_active":"2024-03-11T21:43:27.639661+0000","last_peered":"2024-03-11T21:43:27.639661+0000","last_clean":"2024-03-11T21:43:27.639661+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:43:27.639661+0000","last_undegraded":"2024-03-11T21:43:27.639661+0000","last_fullsized":"2024-03-11T21:43:27.639661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.350961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.350961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1520,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.651067+0000","last_change":"2024-03-11T21:41:48.432501+0000","last_active":"2024-03-11T21:43:27.651067+0000","last_peered":"2024-03-11T21:43:27.651067+0000","last_clean":"2024-03-11T21:43:27.651067+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:43:27.651067+0000","last_undegraded":"2024-03-11T21:43:27.651067+0000","last_fullsized":"2024-03-11T21:43:27.651067+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_clean_scrub_stamp":"2024-03-11T21:41:48.432274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"864'18","reported_seq":1708,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658921+0000","last_change":"2024-03-11T21:42:02.547377+0000","last_active":"2024-03-11T21:43:27.658921+0000","last_peered":"2024-03-11T21:43:27.658921+0000","last_clean":"2024-03-11T21:43:27.658921+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:43:27.658921+0000","last_undegraded":"2024-03-11T21:43:27.658921+0000","last_fullsized":"2024-03-11T21:43:27.658921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:42:02.547005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.547005+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1679,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.654938+0000","last_change":"2024-03-11T21:43:13.521572+0000","last_active":"2024-03-11T21:43:27.654938+0000","last_peered":"2024-03-11T21:43:27.654938+0000","last_clean":"2024-03-11T21:43:27.654938+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:43:27.654938+0000","last_undegraded":"2024-03-11T21:43:27.654938+0000","last_fullsized":"2024-03-11T21:43:27.654938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:13.521476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:13.521476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.205030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1693,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.640219+0000","last_change":"2024-03-11T21:42:02.517280+0000","last_active":"2024-03-11T21:43:27.640219+0000","last_peered":"2024-03-11T21:43:27.640219+0000","last_clean":"2024-03-11T21:43:27.640219+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:43:27.640219+0000","last_undegraded":"2024-03-11T21:43:27.640219+0000","last_fullsized":"2024-03-11T21:43:27.640219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:02.516952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.516952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1654,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.654315+0000","last_change":"2024-03-11T21:43:07.660322+0000","last_active":"2024-03-11T21:43:27.654315+0000","last_peered":"2024-03-11T21:43:27.654315+0000","last_clean":"2024-03-11T21:43:27.654315+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:43:27.654315+0000","last_undegraded":"2024-03-11T21:43:27.654315+0000","last_fullsized":"2024-03-11T21:43:27.654315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.660094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.660094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.424741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1531,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.650654+0000","last_change":"2024-03-11T21:43:16.248764+0000","last_active":"2024-03-11T21:43:27.650654+0000","last_peered":"2024-03-11T21:43:27.650654+0000","last_clean":"2024-03-11T21:43:27.650654+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:43:27.650654+0000","last_undegraded":"2024-03-11T21:43:27.650654+0000","last_fullsized":"2024-03-11T21:43:27.650654+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.248539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.248539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.321015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"864'21","reported_seq":1718,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646516+0000","last_change":"2024-03-11T21:43:09.463949+0000","last_active":"2024-03-11T21:43:27.646516+0000","last_peered":"2024-03-11T21:43:27.646516+0000","last_clean":"2024-03-11T21:43:27.646516+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:43:27.646516+0000","last_undegraded":"2024-03-11T21:43:27.646516+0000","last_fullsized":"2024-03-11T21:43:27.646516+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:43:09.463706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.463706+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.459582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1701,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652537+0000","last_change":"2024-03-11T21:41:59.579259+0000","last_active":"2024-03-11T21:43:27.652537+0000","last_peered":"2024-03-11T21:43:27.652537+0000","last_clean":"2024-03-11T21:43:27.652537+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:43:27.652537+0000","last_undegraded":"2024-03-11T21:43:27.652537+0000","last_fullsized":"2024-03-11T21:43:27.652537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.579089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.579089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1650,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.651308+0000","last_change":"2024-03-11T21:40:16.866813+0000","last_active":"2024-03-11T21:43:27.651308+0000","last_peered":"2024-03-11T21:43:27.651308+0000","last_clean":"2024-03-11T21:43:27.651308+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:43:27.651308+0000","last_undegraded":"2024-03-11T21:43:27.651308+0000","last_fullsized":"2024-03-11T21:43:27.651308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:16.866545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:16.866545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1655,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.648346+0000","last_change":"2024-03-11T21:41:53.585801+0000","last_active":"2024-03-11T21:43:27.648346+0000","last_peered":"2024-03-11T21:43:27.648346+0000","last_clean":"2024-03-11T21:43:27.648346+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:43:27.648346+0000","last_undegraded":"2024-03-11T21:43:27.648346+0000","last_fullsized":"2024-03-11T21:43:27.648346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.585657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1512,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656308+0000","last_change":"2024-03-11T21:43:23.412537+0000","last_active":"2024-03-11T21:43:27.656308+0000","last_peered":"2024-03-11T21:43:27.656308+0000","last_clean":"2024-03-11T21:43:27.656308+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:43:27.656308+0000","last_undegraded":"2024-03-11T21:43:27.656308+0000","last_fullsized":"2024-03-11T21:43:27.656308+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:23.412309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.412309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.639277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"864'46","reported_seq":1669,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642616+0000","last_change":"2024-03-11T21:43:04.876480+0000","last_active":"2024-03-11T21:43:29.642616+0000","last_peered":"2024-03-11T21:43:29.642616+0000","last_clean":"2024-03-11T21:43:29.642616+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:43:29.642616+0000","last_undegraded":"2024-03-11T21:43:29.642616+0000","last_fullsized":"2024-03-11T21:43:29.642616+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:43:04.876381+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.876381+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:12.934737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1685,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.637819+0000","last_change":"2024-03-11T21:43:02.494327+0000","last_active":"2024-03-11T21:43:27.637819+0000","last_peered":"2024-03-11T21:43:27.637819+0000","last_clean":"2024-03-11T21:43:27.637819+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:43:27.637819+0000","last_undegraded":"2024-03-11T21:43:27.637819+0000","last_fullsized":"2024-03-11T21:43:27.637819+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.494116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.494116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.677295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1960,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.653141+0000","last_change":"2024-03-11T21:43:16.367046+0000","last_active":"2024-03-11T21:43:27.653141+0000","last_peered":"2024-03-11T21:43:27.653141+0000","last_clean":"2024-03-11T21:43:27.653141+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:43:27.653141+0000","last_undegraded":"2024-03-11T21:43:27.653141+0000","last_fullsized":"2024-03-11T21:43:27.653141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:43:16.366877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.366877+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:45.000288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1656,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.660221+0000","last_change":"2024-03-11T21:43:24.429298+0000","last_active":"2024-03-11T21:43:27.660221+0000","last_peered":"2024-03-11T21:43:27.660221+0000","last_clean":"2024-03-11T21:43:27.660221+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:43:27.660221+0000","last_undegraded":"2024-03-11T21:43:27.660221+0000","last_fullsized":"2024-03-11T21:43:27.660221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:43:24.429154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.429154+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.701671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1526,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638805+0000","last_change":"2024-03-11T21:41:56.288055+0000","last_active":"2024-03-11T21:43:27.638805+0000","last_peered":"2024-03-11T21:43:27.638805+0000","last_clean":"2024-03-11T21:43:27.638805+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:43:27.638805+0000","last_undegraded":"2024-03-11T21:43:27.638805+0000","last_fullsized":"2024-03-11T21:43:27.638805+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.287964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.287964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"864'21","reported_seq":1714,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.636802+0000","last_change":"2024-03-11T21:42:52.850471+0000","last_active":"2024-03-11T21:43:27.636802+0000","last_peered":"2024-03-11T21:43:27.636802+0000","last_clean":"2024-03-11T21:43:27.636802+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:43:27.636802+0000","last_undegraded":"2024-03-11T21:43:27.636802+0000","last_fullsized":"2024-03-11T21:43:27.636802+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:42:52.850333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.850333+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.477895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1676,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.639926+0000","last_change":"2024-03-11T21:43:06.774760+0000","last_active":"2024-03-11T21:43:27.639926+0000","last_peered":"2024-03-11T21:43:27.639926+0000","last_clean":"2024-03-11T21:43:27.639926+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:43:27.639926+0000","last_undegraded":"2024-03-11T21:43:27.639926+0000","last_fullsized":"2024-03-11T21:43:27.639926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:43:06.774478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.774478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.859854+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1671,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656096+0000","last_change":"2024-03-11T21:41:41.375255+0000","last_active":"2024-03-11T21:43:27.656096+0000","last_peered":"2024-03-11T21:43:27.656096+0000","last_clean":"2024-03-11T21:43:27.656096+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:43:27.656096+0000","last_undegraded":"2024-03-11T21:43:27.656096+0000","last_fullsized":"2024-03-11T21:43:27.656096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.375162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1549,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.643153+0000","last_change":"2024-03-11T21:41:49.263645+0000","last_active":"2024-03-11T21:43:29.643153+0000","last_peered":"2024-03-11T21:43:29.643153+0000","last_clean":"2024-03-11T21:43:29.643153+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:43:29.643153+0000","last_undegraded":"2024-03-11T21:43:29.643153+0000","last_fullsized":"2024-03-11T21:43:29.643153+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.263416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1504,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.651501+0000","last_change":"2024-03-11T21:41:23.491021+0000","last_active":"2024-03-11T21:43:27.651501+0000","last_peered":"2024-03-11T21:43:27.651501+0000","last_clean":"2024-03-11T21:43:27.651501+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:43:27.651501+0000","last_undegraded":"2024-03-11T21:43:27.651501+0000","last_fullsized":"2024-03-11T21:43:27.651501+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.490850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.490850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"864'11","reported_seq":1691,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649698+0000","last_change":"2024-03-11T21:43:04.387983+0000","last_active":"2024-03-11T21:43:27.649698+0000","last_peered":"2024-03-11T21:43:27.649698+0000","last_clean":"2024-03-11T21:43:27.649698+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:43:27.649698+0000","last_undegraded":"2024-03-11T21:43:27.649698+0000","last_fullsized":"2024-03-11T21:43:27.649698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:43:04.387720+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.387720+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:07.836763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1701,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.652179+0000","last_change":"2024-03-11T21:41:53.683571+0000","last_active":"2024-03-11T21:43:27.652179+0000","last_peered":"2024-03-11T21:43:27.652179+0000","last_clean":"2024-03-11T21:43:27.652179+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:43:27.652179+0000","last_undegraded":"2024-03-11T21:43:27.652179+0000","last_fullsized":"2024-03-11T21:43:27.652179+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.683465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.683465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1551,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644122+0000","last_change":"2024-03-11T21:43:26.028281+0000","last_active":"2024-03-11T21:43:29.644122+0000","last_peered":"2024-03-11T21:43:29.644122+0000","last_clean":"2024-03-11T21:43:29.644122+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:43:29.644122+0000","last_undegraded":"2024-03-11T21:43:29.644122+0000","last_fullsized":"2024-03-11T21:43:29.644122+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.028192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.028192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:44.924494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1684,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.627008+0000","last_change":"2024-03-11T21:41:19.561625+0000","last_active":"2024-03-11T21:43:27.627008+0000","last_peered":"2024-03-11T21:43:27.627008+0000","last_clean":"2024-03-11T21:43:27.627008+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:43:27.627008+0000","last_undegraded":"2024-03-11T21:43:27.627008+0000","last_fullsized":"2024-03-11T21:43:27.627008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.561348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.561348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1538,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622881+0000","last_change":"2024-03-11T21:41:55.738509+0000","last_active":"2024-03-11T21:43:27.622881+0000","last_peered":"2024-03-11T21:43:27.622881+0000","last_clean":"2024-03-11T21:43:27.622881+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:43:27.622881+0000","last_undegraded":"2024-03-11T21:43:27.622881+0000","last_fullsized":"2024-03-11T21:43:27.622881+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.738282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"864'25","reported_seq":1745,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.623183+0000","last_change":"2024-03-11T21:42:51.740712+0000","last_active":"2024-03-11T21:43:27.623183+0000","last_peered":"2024-03-11T21:43:27.623183+0000","last_clean":"2024-03-11T21:43:27.623183+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:43:27.623183+0000","last_undegraded":"2024-03-11T21:43:27.623183+0000","last_fullsized":"2024-03-11T21:43:27.623183+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'19","last_scrub_stamp":"2024-03-11T21:42:51.740477+0000","last_deep_scrub":"714'19","last_deep_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_clean_scrub_stamp":"2024-03-11T21:42:51.740477+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.872799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1675,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.647310+0000","last_change":"2024-03-11T21:43:07.788606+0000","last_active":"2024-03-11T21:43:27.647310+0000","last_peered":"2024-03-11T21:43:27.647310+0000","last_clean":"2024-03-11T21:43:27.647310+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:43:27.647310+0000","last_undegraded":"2024-03-11T21:43:27.647310+0000","last_fullsized":"2024-03-11T21:43:27.647310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.788495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.788495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.174149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1686,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.625496+0000","last_change":"2024-03-11T21:41:54.741470+0000","last_active":"2024-03-11T21:43:27.625496+0000","last_peered":"2024-03-11T21:43:27.625496+0000","last_clean":"2024-03-11T21:43:27.625496+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:43:27.625496+0000","last_undegraded":"2024-03-11T21:43:27.625496+0000","last_fullsized":"2024-03-11T21:43:27.625496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.741251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1656,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.656260+0000","last_change":"2024-03-11T21:41:54.630605+0000","last_active":"2024-03-11T21:43:27.656260+0000","last_peered":"2024-03-11T21:43:27.656260+0000","last_clean":"2024-03-11T21:43:27.656260+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:43:27.656260+0000","last_undegraded":"2024-03-11T21:43:27.656260+0000","last_fullsized":"2024-03-11T21:43:27.656260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.630386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.630386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.661854+0000","last_change":"2024-03-11T21:41:31.510028+0000","last_active":"2024-03-11T21:43:27.661854+0000","last_peered":"2024-03-11T21:43:27.661854+0000","last_clean":"2024-03-11T21:43:27.661854+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:43:27.661854+0000","last_undegraded":"2024-03-11T21:43:27.661854+0000","last_fullsized":"2024-03-11T21:43:27.661854+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.509821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.509821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"864'24","reported_seq":1716,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.644461+0000","last_change":"2024-03-11T21:43:20.541592+0000","last_active":"2024-03-11T21:43:27.644461+0000","last_peered":"2024-03-11T21:43:27.644461+0000","last_clean":"2024-03-11T21:43:27.644461+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:43:27.644461+0000","last_undegraded":"2024-03-11T21:43:27.644461+0000","last_fullsized":"2024-03-11T21:43:27.644461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'24","last_scrub_stamp":"2024-03-11T21:43:20.541460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.541460+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.402842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1666,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.658522+0000","last_change":"2024-03-11T21:41:46.401421+0000","last_active":"2024-03-11T21:43:27.658522+0000","last_peered":"2024-03-11T21:43:27.658522+0000","last_clean":"2024-03-11T21:43:27.658522+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:43:27.658522+0000","last_undegraded":"2024-03-11T21:43:27.658522+0000","last_fullsized":"2024-03-11T21:43:27.658522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.401211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.401211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1663,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.638676+0000","last_change":"2024-03-11T21:41:53.275977+0000","last_active":"2024-03-11T21:43:27.638676+0000","last_peered":"2024-03-11T21:43:27.638676+0000","last_clean":"2024-03-11T21:43:27.638676+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:43:27.638676+0000","last_undegraded":"2024-03-11T21:43:27.638676+0000","last_fullsized":"2024-03-11T21:43:27.638676+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.275834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.275834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1681,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646786+0000","last_change":"2024-03-11T21:42:59.415876+0000","last_active":"2024-03-11T21:43:27.646786+0000","last_peered":"2024-03-11T21:43:27.646786+0000","last_clean":"2024-03-11T21:43:27.646786+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:43:27.646786+0000","last_undegraded":"2024-03-11T21:43:27.646786+0000","last_fullsized":"2024-03-11T21:43:27.646786+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.415650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.415650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.486892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1549,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.626019+0000","last_change":"2024-03-11T21:43:24.762700+0000","last_active":"2024-03-11T21:43:27.626019+0000","last_peered":"2024-03-11T21:43:27.626019+0000","last_clean":"2024-03-11T21:43:27.626019+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:43:27.626019+0000","last_undegraded":"2024-03-11T21:43:27.626019+0000","last_fullsized":"2024-03-11T21:43:27.626019+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.762559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.762559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.495720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"864'46","reported_seq":1752,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.643542+0000","last_change":"2024-03-11T21:41:51.290471+0000","last_active":"2024-03-11T21:43:27.643542+0000","last_peered":"2024-03-11T21:43:27.643542+0000","last_clean":"2024-03-11T21:43:27.643542+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:43:27.643542+0000","last_undegraded":"2024-03-11T21:43:27.643542+0000","last_fullsized":"2024-03-11T21:43:27.643542+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:41:51.290324+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.290324+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1669,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.661388+0000","last_change":"2024-03-11T21:40:06.837164+0000","last_active":"2024-03-11T21:43:27.661388+0000","last_peered":"2024-03-11T21:43:27.661388+0000","last_clean":"2024-03-11T21:43:27.661388+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:43:27.661388+0000","last_undegraded":"2024-03-11T21:43:27.661388+0000","last_fullsized":"2024-03-11T21:43:27.661388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:06.836871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:40:06.836871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1659,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.653728+0000","last_change":"2024-03-11T21:40:04.809344+0000","last_active":"2024-03-11T21:43:27.653728+0000","last_peered":"2024-03-11T21:43:27.653728+0000","last_clean":"2024-03-11T21:43:27.653728+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:43:27.653728+0000","last_undegraded":"2024-03-11T21:43:27.653728+0000","last_fullsized":"2024-03-11T21:43:27.653728+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:04.809056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.809056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1668,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.644900+0000","last_change":"2024-03-11T21:41:47.226377+0000","last_active":"2024-03-11T21:43:27.644900+0000","last_peered":"2024-03-11T21:43:27.644900+0000","last_clean":"2024-03-11T21:43:27.644900+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:43:27.644900+0000","last_undegraded":"2024-03-11T21:43:27.644900+0000","last_fullsized":"2024-03-11T21:43:27.644900+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:47.226166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:47.226166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1513,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.659805+0000","last_change":"2024-03-11T21:40:02.830965+0000","last_active":"2024-03-11T21:43:27.659805+0000","last_peered":"2024-03-11T21:43:27.659805+0000","last_clean":"2024-03-11T21:43:27.659805+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:43:27.659805+0000","last_undegraded":"2024-03-11T21:43:27.659805+0000","last_fullsized":"2024-03-11T21:43:27.659805+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:02.830743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.830743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"864'32","reported_seq":1730,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.659303+0000","last_change":"2024-03-11T21:42:55.509855+0000","last_active":"2024-03-11T21:43:27.659303+0000","last_peered":"2024-03-11T21:43:27.659303+0000","last_clean":"2024-03-11T21:43:27.659303+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:43:27.659303+0000","last_undegraded":"2024-03-11T21:43:27.659303+0000","last_fullsized":"2024-03-11T21:43:27.659303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:42:55.509532+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.509532+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:58.493575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1669,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.649527+0000","last_change":"2024-03-11T21:41:49.603979+0000","last_active":"2024-03-11T21:43:27.649527+0000","last_peered":"2024-03-11T21:43:27.649527+0000","last_clean":"2024-03-11T21:43:27.649527+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:43:27.649527+0000","last_undegraded":"2024-03-11T21:43:27.649527+0000","last_fullsized":"2024-03-11T21:43:27.649527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.603815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.603815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1547,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.642812+0000","last_change":"2024-03-11T21:43:10.968913+0000","last_active":"2024-03-11T21:43:29.642812+0000","last_peered":"2024-03-11T21:43:29.642812+0000","last_clean":"2024-03-11T21:43:29.642812+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:43:29.642812+0000","last_undegraded":"2024-03-11T21:43:29.642812+0000","last_fullsized":"2024-03-11T21:43:29.642812+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.968603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.968603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.483700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1699,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622867+0000","last_change":"2024-03-11T21:43:12.640536+0000","last_active":"2024-03-11T21:43:27.622867+0000","last_peered":"2024-03-11T21:43:27.622867+0000","last_clean":"2024-03-11T21:43:27.622867+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:43:27.622867+0000","last_undegraded":"2024-03-11T21:43:27.622867+0000","last_fullsized":"2024-03-11T21:43:27.622867+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:12.639805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:12.639805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.329943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1398,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-11T21:43:29.644556+0000","last_change":"2024-03-11T21:41:54.179283+0000","last_active":"2024-03-11T21:43:29.644556+0000","last_peered":"2024-03-11T21:43:29.644556+0000","last_clean":"2024-03-11T21:43:29.644556+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:43:29.644556+0000","last_undegraded":"2024-03-11T21:43:29.644556+0000","last_fullsized":"2024-03-11T21:43:29.644556+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.179012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.179012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"864'30","reported_seq":1710,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.655303+0000","last_change":"2024-03-11T21:40:15.377800+0000","last_active":"2024-03-11T21:43:27.655303+0000","last_peered":"2024-03-11T21:43:27.655303+0000","last_clean":"2024-03-11T21:43:27.655303+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:43:27.655303+0000","last_undegraded":"2024-03-11T21:43:27.655303+0000","last_fullsized":"2024-03-11T21:43:27.655303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:40:15.377635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.377635+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1687,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.659536+0000","last_change":"2024-03-11T21:43:08.340131+0000","last_active":"2024-03-11T21:43:27.659536+0000","last_peered":"2024-03-11T21:43:27.659536+0000","last_clean":"2024-03-11T21:43:27.659536+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:43:27.659536+0000","last_undegraded":"2024-03-11T21:43:27.659536+0000","last_fullsized":"2024-03-11T21:43:27.659536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:43:08.339938+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.339938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.620284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1703,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.622898+0000","last_change":"2024-03-11T21:43:08.620663+0000","last_active":"2024-03-11T21:43:27.622898+0000","last_peered":"2024-03-11T21:43:27.622898+0000","last_clean":"2024-03-11T21:43:27.622898+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:43:27.622898+0000","last_undegraded":"2024-03-11T21:43:27.622898+0000","last_fullsized":"2024-03-11T21:43:27.622898+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:08.620444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.620444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.349008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1655,"reported_epoch":880,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.646919+0000","last_change":"2024-03-11T21:42:56.738473+0000","last_active":"2024-03-11T21:43:27.646919+0000","last_peered":"2024-03-11T21:43:27.646919+0000","last_clean":"2024-03-11T21:43:27.646919+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:43:27.646919+0000","last_undegraded":"2024-03-11T21:43:27.646919+0000","last_fullsized":"2024-03-11T21:43:27.646919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.738379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.738379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.984522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1558,"num_read_kb":1315,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1203,"num_read_kb":1198,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841792,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124076,"kb_used_data":16356,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93621076,"statfs":{"total":95995035648,"available":95867981824,"internally_reserved":0,"allocated":16748544,"data_stored":11407932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168841,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107996,"kb_used_data":14152,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93637156,"statfs":{"total":95995035648,"available":95884447744,"internally_reserved":0,"allocated":14491648,"data_stored":9117941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352464,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59972,"kb_used_data":13512,"kb_used_omap":27,"kb_used_meta":45924,"kb_avail":93685180,"statfs":{"total":95995035648,"available":95933624320,"internally_reserved":0,"allocated":13836288,"data_stored":8634616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":47026505},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105680,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93639472,"statfs":{"total":95995035648,"available":95886819328,"internally_reserved":0,"allocated":16392192,"data_stored":11075048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378931,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99956,"kb_used_data":14444,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93645196,"statfs":{"total":95995035648,"available":95892680704,"internally_reserved":0,"allocated":14790656,"data_stored":9413852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117287,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96092,"kb_used_data":14548,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93649060,"statfs":{"total":95995035648,"available":95896637440,"internally_reserved":0,"allocated":14897152,"data_stored":9591666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855645,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123748,"kb_used_data":16028,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93621404,"statfs":{"total":95995035648,"available":95868317696,"internally_reserved":0,"allocated":16412672,"data_stored":11071242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495891,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78024,"kb_used_data":14720,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93667128,"statfs":{"total":95995035648,"available":95915139072,"internally_reserved":0,"allocated":15073280,"data_stored":9704128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:43:33.109 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:43:33.109 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:43:33.109 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T21:43:33.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:43:33.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:33.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.653971+0000 mgr.y (mgr.14550) 796 : cluster 0 pgmap v1377: 206 pgs: 206 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.910082+0000 osd.4 (osd.4) 455 : cluster 0 5.1 scrub starts 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: cluster 2024-03-11T21:43:31.912298+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub ok 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: audit 2024-03-11T21:43:32.349665+0000 mon.a (mon.0) 3942 : audit 0 from='client.? 172.21.15.143:0/1073989988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: cluster 2024-03-11T21:43:32.841970+0000 osd.2 (osd.2) 419 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[29284]: cluster 2024-03-11T21:43:32.842413+0000 osd.2 (osd.2) 420 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.653971+0000 mgr.y (mgr.14550) 796 : cluster 0 pgmap v1377: 206 pgs: 206 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.910082+0000 osd.4 (osd.4) 455 : cluster 0 5.1 scrub starts 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: cluster 2024-03-11T21:43:31.912298+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub ok 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: audit 2024-03-11T21:43:32.349665+0000 mon.a (mon.0) 3942 : audit 0 from='client.? 172.21.15.143:0/1073989988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: cluster 2024-03-11T21:43:32.841970+0000 osd.2 (osd.2) 419 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:33.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:33 smithi143 bash[21512]: cluster 2024-03-11T21:43:32.842413+0000 osd.2 (osd.2) 420 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:33.216 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:33.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.653971+0000 mgr.y (mgr.14550) 796 : cluster 0 pgmap v1377: 206 pgs: 206 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:33.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.910082+0000 osd.4 (osd.4) 455 : cluster 0 5.1 scrub starts 2024-03-11T21:43:33.280 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: cluster 2024-03-11T21:43:31.912298+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub ok 2024-03-11T21:43:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: audit 2024-03-11T21:43:32.349665+0000 mon.a (mon.0) 3942 : audit 0 from='client.? 172.21.15.143:0/1073989988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: cluster 2024-03-11T21:43:32.841970+0000 osd.2 (osd.2) 419 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:33 smithi203 bash[23654]: cluster 2024-03-11T21:43:32.842413+0000 osd.2 (osd.2) 420 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:33.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:33.346 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:33.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:33.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:43:33.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:33.624 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:33.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:33.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:33.876 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:33.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.007 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:34.110 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.108950+0000 osd.1 (osd.1) 281 : cluster 0 39.7 scrub starts 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.114495+0000 osd.1 (osd.1) 282 : cluster 0 39.7 scrub ok 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.478714+0000 osd.5 (osd.5) 493 : cluster 0 1.1 scrub starts 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.481020+0000 osd.5 (osd.5) 494 : cluster 0 1.1 scrub ok 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.606971+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e885: 8 total, 8 up, 8 in 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.108950+0000 osd.1 (osd.1) 281 : cluster 0 39.7 scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.114495+0000 osd.1 (osd.1) 282 : cluster 0 39.7 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.478714+0000 osd.5 (osd.5) 493 : cluster 0 1.1 scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.481020+0000 osd.5 (osd.5) 494 : cluster 0 1.1 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:34 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.606971+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e885: 8 total, 8 up, 8 in 2024-03-11T21:43:34.132 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:34.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 -- 172.21.15.143:0/2421316753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4109350 msgr2=0x7f7cb4110f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:34.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/2421316753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4109350 0x7f7cb4110f20 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40099c0 tx=0x7f7ca4009cd0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 -- 172.21.15.143:0/2421316753 shutdown_connections 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/2421316753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4104270 0x7f7cb4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/2421316753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4109350 0x7f7cb4110f20 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/2421316753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104c20 0x7f7cb4108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 -- 172.21.15.143:0/2421316753 >> 172.21.15.143:0/2421316753 conn(0x7f7cb40ffb00 msgr2=0x7f7cb4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 -- 172.21.15.143:0/2421316753 shutdown_connections 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 -- 172.21.15.143:0/2421316753 wait complete. 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 librados: starting msgr at 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.213+0000 7f7cbabe3700 1 librados: starting objecter 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 Processor -- start 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- start start 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 librados: setting wanted keys 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 librados: calling monclient init 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104270 0x7f7cb4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:34.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41138b0 con 0x7f7cb4109350 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4113730 con 0x7f7cb4104270 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4113a30 con 0x7f7cb4104c20 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb9180700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:33574/0 (socket says 172.21.15.143:33574) 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb9180700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51532/0 (socket says 172.21.15.143:51532) 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 -- 172.21.15.143:0/4193427518 learned_addr learned my addr 172.21.15.143:0/4193427518 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb897f700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104270 0x7f7cb4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 -- 172.21.15.143:0/4193427518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104270 msgr2=0x7f7cb4199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104270 0x7f7cb4199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 -- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 msgr2=0x7f7cb419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:34.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:34.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 -- 172.21.15.143:0/4193427518 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb419edc0 con 0x7f7cb4104c20 2024-03-11T21:43:34.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb9180700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:34.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb3fff700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b500 tx=0x7f7c9c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 10 client.?.objecter ms_handle_connect 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c015020 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb419f0b0 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f7c9c011cb0 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c010c90 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 10 client.42125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb41a6930 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 librados: init done 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb404faa0 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f7c9c018020 con 0x7f7cb4104c20 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 0x7f7ca0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:34.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(885..885 src has 587..885) v4 ==== 9021+0+0 (secure 0 0 0) 0x7f7c9c08dff0 con 0x7f7cb4104c20 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 10 client.42125.objecter ms_dispatch 0x7f7cb4001550 osd_map(885..885 src has 587..885) v4 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 3 client.42125.objecter handle_osd_map got epochs [885,885] > 0 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 3 client.42125.objecter handle_osd_map decoding full epoch 885 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb897f700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 0x7f7ca0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.217+0000 7f7cb1ffb700 20 client.42125.objecter dump_active .. 0 homeless 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.221+0000 7f7cb1ffb700 10 client.42125.objecter ms_handle_connect 0x7f7ca006f820 2024-03-11T21:43:34.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.221+0000 7f7cb897f700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 0x7f7ca0071ce0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009380 tx=0x7f7cac00b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:34.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.225+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7c9c01f020 con 0x7f7cb4104c20 2024-03-11T21:43:34.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:34.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:34.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:34.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:34.453+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f7cb41a6b60 con 0x7f7cb4104c20 2024-03-11T21:43:34.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:34.513 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:34.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:34 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.108950+0000 osd.1 (osd.1) 281 : cluster 0 39.7 scrub starts 2024-03-11T21:43:34.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:34 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.114495+0000 osd.1 (osd.1) 282 : cluster 0 39.7 scrub ok 2024-03-11T21:43:34.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:34 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.478714+0000 osd.5 (osd.5) 493 : cluster 0 1.1 scrub starts 2024-03-11T21:43:34.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:34 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.481020+0000 osd.5 (osd.5) 494 : cluster 0 1.1 scrub ok 2024-03-11T21:43:34.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:34 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.606971+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e885: 8 total, 8 up, 8 in 2024-03-11T21:43:34.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:34.644 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:34.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:34.774 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:34.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:34.903 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:35.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:35.034 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:35.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.145+0000 7f7cb1ffb700 1 -- 172.21.15.143:0/4193427518 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v886) v1 ==== 87+0+0 (secure 0 0 0) 0x7f7c9c05e8f0 con 0x7f7cb4104c20 2024-03-11T21:43:35.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 10 librados: watch_flush enter 2024-03-11T21:43:35.135 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 10 librados: watch_flush exit 2024-03-11T21:43:35.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 20 client.42125.objecter shutdown clearing up homeless session... 2024-03-11T21:43:35.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 10 client.42125.objecter successfully canceled tick 2024-03-11T21:43:35.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 msgr2=0x7f7ca0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 0x7f7ca0071ce0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009380 tx=0x7f7cac00b040 comp rx=0 tx=0).stop 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 msgr2=0x7f7cb419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b500 tx=0x7f7c9c00b810 comp rx=0 tx=0).stop 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 shutdown_connections 2024-03-11T21:43:35.138 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f7ca006f820 0x7f7ca0071ce0 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb4109350 0x7f7cb419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4104270 0x7f7cb4199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 --2- 172.21.15.143:0/4193427518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb4104c20 0x7f7cb419a2f0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.149+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 >> 172.21.15.143:0/4193427518 conn(0x7f7cb40ffb00 msgr2=0x7f7cb4100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.153+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 shutdown_connections 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.153+0000 7f7cbabe3700 1 -- 172.21.15.143:0/4193427518 wait complete. 2024-03-11T21:43:35.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:35.153+0000 7f7cbabe3700 1 librados: shutdown 2024-03-11T21:43:35.140 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:43:35.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: cluster 2024-03-11T21:43:33.655020+0000 mgr.y (mgr.14550) 797 : cluster 0 pgmap v1379: 206 pgs: 206 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:35.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.143235+0000 osd.1 (osd.1) 283 : cluster 0 3.0 scrub starts 2024-03-11T21:43:35.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.146442+0000 osd.1 (osd.1) 284 : cluster 0 3.0 scrub ok 2024-03-11T21:43:35.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.327163+0000 osd.7 (osd.7) 468 : cluster 0 9.15 scrub starts 2024-03-11T21:43:35.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.329423+0000 osd.7 (osd.7) 469 : cluster 0 9.15 scrub ok 2024-03-11T21:43:35.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: audit 2024-03-11T21:43:34.459566+0000 mon.c (mon.2) 1359 : audit 1 from='client.? 172.21.15.143:0/4193427518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:35 smithi203 bash[23654]: audit 2024-03-11T21:43:34.459969+0000 mon.a (mon.0) 3944 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.167 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:35.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:35.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:35.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:35.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: cluster 2024-03-11T21:43:33.655020+0000 mgr.y (mgr.14550) 797 : cluster 0 pgmap v1379: 206 pgs: 206 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:35.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.143235+0000 osd.1 (osd.1) 283 : cluster 0 3.0 scrub starts 2024-03-11T21:43:35.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.146442+0000 osd.1 (osd.1) 284 : cluster 0 3.0 scrub ok 2024-03-11T21:43:35.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.327163+0000 osd.7 (osd.7) 468 : cluster 0 9.15 scrub starts 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.329423+0000 osd.7 (osd.7) 469 : cluster 0 9.15 scrub ok 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: audit 2024-03-11T21:43:34.459566+0000 mon.c (mon.2) 1359 : audit 1 from='client.? 172.21.15.143:0/4193427518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[21512]: audit 2024-03-11T21:43:34.459969+0000 mon.a (mon.0) 3944 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: cluster 2024-03-11T21:43:33.655020+0000 mgr.y (mgr.14550) 797 : cluster 0 pgmap v1379: 206 pgs: 206 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.143235+0000 osd.1 (osd.1) 283 : cluster 0 3.0 scrub starts 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.146442+0000 osd.1 (osd.1) 284 : cluster 0 3.0 scrub ok 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.327163+0000 osd.7 (osd.7) 468 : cluster 0 9.15 scrub starts 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.329423+0000 osd.7 (osd.7) 469 : cluster 0 9.15 scrub ok 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: audit 2024-03-11T21:43:34.459566+0000 mon.c (mon.2) 1359 : audit 1 from='client.? 172.21.15.143:0/4193427518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:35 smithi143 bash[29284]: audit 2024-03-11T21:43:34.459969+0000 mon.a (mon.0) 3944 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:35.426 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:35.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:35.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:35.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:35.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:35.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:35.824 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:35.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:35.955 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:36.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:36.085 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:36.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:36.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:36.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.483889+0000 osd.6 (osd.6) 397 : cluster 0 2.7 scrub starts 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:34.486683+0000 osd.6 (osd.6) 398 : cluster 0 2.7 scrub ok 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:35.132764+0000 mon.a (mon.0) 3945 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: audit 2024-03-11T21:43:35.136904+0000 mon.a (mon.0) 3946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:35.144028+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e886: 8 total, 8 up, 8 in 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:35.398431+0000 osd.5 (osd.5) 495 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:36.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:36 smithi203 bash[23654]: cluster 2024-03-11T21:43:35.398844+0000 osd.5 (osd.5) 496 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:36.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:36.406 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:43:36.417 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.483889+0000 osd.6 (osd.6) 397 : cluster 0 2.7 scrub starts 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:34.486683+0000 osd.6 (osd.6) 398 : cluster 0 2.7 scrub ok 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:35.132764+0000 mon.a (mon.0) 3945 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: audit 2024-03-11T21:43:35.136904+0000 mon.a (mon.0) 3946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:35.144028+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e886: 8 total, 8 up, 8 in 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:35.398431+0000 osd.5 (osd.5) 495 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[21512]: cluster 2024-03-11T21:43:35.398844+0000 osd.5 (osd.5) 496 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.483889+0000 osd.6 (osd.6) 397 : cluster 0 2.7 scrub starts 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:34.486683+0000 osd.6 (osd.6) 398 : cluster 0 2.7 scrub ok 2024-03-11T21:43:36.418 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:35.132764+0000 mon.a (mon.0) 3945 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:36.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: audit 2024-03-11T21:43:35.136904+0000 mon.a (mon.0) 3946 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:36.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:35.144028+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e886: 8 total, 8 up, 8 in 2024-03-11T21:43:36.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:35.398431+0000 osd.5 (osd.5) 495 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:36.419 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:36 smithi143 bash[29284]: cluster 2024-03-11T21:43:35.398844+0000 osd.5 (osd.5) 496 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:36.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:36.473 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:36.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:36.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:36.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:36.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:36.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:36.861 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:36.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:36.992 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:37.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.124 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:37.227 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:37.254 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:37.357 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[21512]: cluster 2024-03-11T21:43:35.656023+0000 mgr.y (mgr.14550) 798 : cluster 0 pgmap v1381: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[21512]: cluster 2024-03-11T21:43:36.172999+0000 mon.a (mon.0) 3948 : cluster 0 osdmap e887: 8 total, 8 up, 8 in 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[21512]: cluster 2024-03-11T21:43:36.587787+0000 mon.a (mon.0) 3949 : cluster 0 osdmap e888: 8 total, 8 up, 8 in 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[21512]: cluster 2024-03-11T21:43:36.642921+0000 osd.3 (osd.3) 477 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[21512]: cluster 2024-03-11T21:43:36.643155+0000 osd.3 (osd.3) 478 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[29284]: cluster 2024-03-11T21:43:35.656023+0000 mgr.y (mgr.14550) 798 : cluster 0 pgmap v1381: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[29284]: cluster 2024-03-11T21:43:36.172999+0000 mon.a (mon.0) 3948 : cluster 0 osdmap e887: 8 total, 8 up, 8 in 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[29284]: cluster 2024-03-11T21:43:36.587787+0000 mon.a (mon.0) 3949 : cluster 0 osdmap e888: 8 total, 8 up, 8 in 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[29284]: cluster 2024-03-11T21:43:36.642921+0000 osd.3 (osd.3) 477 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:37.369 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:37 smithi143 bash[29284]: cluster 2024-03-11T21:43:36.643155+0000 osd.3 (osd.3) 478 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:37.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:37.514 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:37 smithi203 bash[23654]: cluster 2024-03-11T21:43:35.656023+0000 mgr.y (mgr.14550) 798 : cluster 0 pgmap v1381: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:43:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:37 smithi203 bash[23654]: cluster 2024-03-11T21:43:36.172999+0000 mon.a (mon.0) 3948 : cluster 0 osdmap e887: 8 total, 8 up, 8 in 2024-03-11T21:43:37.530 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:37 smithi203 bash[23654]: cluster 2024-03-11T21:43:36.587787+0000 mon.a (mon.0) 3949 : cluster 0 osdmap e888: 8 total, 8 up, 8 in 2024-03-11T21:43:37.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:37 smithi203 bash[23654]: cluster 2024-03-11T21:43:36.642921+0000 osd.3 (osd.3) 477 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:37.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:37 smithi203 bash[23654]: cluster 2024-03-11T21:43:36.643155+0000 osd.3 (osd.3) 478 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:37.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:37.644 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:37.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.775 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:37.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:37.907 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:38.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:38.035 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:38.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:38.167 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:38.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:43:38.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:38.295 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:38.342 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:38.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:38.418 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:38.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:38.547 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:38.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:38.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:38 smithi203 bash[23654]: audit 2024-03-11T21:43:37.526591+0000 mon.a (mon.0) 3950 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:38.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:38 smithi203 bash[23654]: audit 2024-03-11T21:43:37.528219+0000 mon.c (mon.2) 1360 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:38 smithi203 bash[23654]: cluster 2024-03-11T21:43:37.590571+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e889: 8 total, 8 up, 8 in 2024-03-11T21:43:38.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:38 smithi203 bash[23654]: cluster 2024-03-11T21:43:37.656756+0000 mgr.y (mgr.14550) 799 : cluster 0 pgmap v1385: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:38.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[29284]: audit 2024-03-11T21:43:37.526591+0000 mon.a (mon.0) 3950 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[29284]: audit 2024-03-11T21:43:37.528219+0000 mon.c (mon.2) 1360 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[29284]: cluster 2024-03-11T21:43:37.590571+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e889: 8 total, 8 up, 8 in 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[29284]: cluster 2024-03-11T21:43:37.656756+0000 mgr.y (mgr.14550) 799 : cluster 0 pgmap v1385: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[21512]: audit 2024-03-11T21:43:37.526591+0000 mon.a (mon.0) 3950 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[21512]: audit 2024-03-11T21:43:37.528219+0000 mon.c (mon.2) 1360 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[21512]: cluster 2024-03-11T21:43:37.590571+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e889: 8 total, 8 up, 8 in 2024-03-11T21:43:38.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:38 smithi143 bash[21512]: cluster 2024-03-11T21:43:37.656756+0000 mgr.y (mgr.14550) 799 : cluster 0 pgmap v1385: 206 pgs: 206 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:43:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:38.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:38.807 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:38.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:38.938 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:39.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.070 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:39.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:39.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:39.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:39.200 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:39.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:39.332 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:39.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:39.463 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:39.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:39.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:39.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:39.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:39 smithi203 bash[23654]: cluster 2024-03-11T21:43:38.981476+0000 osd.1 (osd.1) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:39.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:39 smithi203 bash[23654]: cluster 2024-03-11T21:43:38.982077+0000 osd.1 (osd.1) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:39.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:39 smithi203 bash[23654]: audit 2024-03-11T21:43:39.115900+0000 mgr.y (mgr.14550) 800 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:39.726 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:39.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:39.853 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[21512]: cluster 2024-03-11T21:43:38.981476+0000 osd.1 (osd.1) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[21512]: cluster 2024-03-11T21:43:38.982077+0000 osd.1 (osd.1) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[21512]: audit 2024-03-11T21:43:39.115900+0000 mgr.y (mgr.14550) 800 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[29284]: cluster 2024-03-11T21:43:38.981476+0000 osd.1 (osd.1) 285 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[29284]: cluster 2024-03-11T21:43:38.982077+0000 osd.1 (osd.1) 286 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:39.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:39 smithi143 bash[29284]: audit 2024-03-11T21:43:39.115900+0000 mgr.y (mgr.14550) 800 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:39.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:39.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:40.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:40.106 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:40.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:40.234 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:40.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:40.363 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:40.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:40.498 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:40.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.621 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:40.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:40 smithi143 bash[21512]: cluster 2024-03-11T21:43:39.606010+0000 mon.a (mon.0) 3952 : cluster 0 osdmap e890: 8 total, 8 up, 8 in 2024-03-11T21:43:40.695 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:40 smithi143 bash[21512]: cluster 2024-03-11T21:43:39.657294+0000 mgr.y (mgr.14550) 801 : cluster 0 pgmap v1387: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:40.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:40 smithi143 bash[29284]: cluster 2024-03-11T21:43:39.606010+0000 mon.a (mon.0) 3952 : cluster 0 osdmap e890: 8 total, 8 up, 8 in 2024-03-11T21:43:40.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:40 smithi143 bash[29284]: cluster 2024-03-11T21:43:39.657294+0000 mgr.y (mgr.14550) 801 : cluster 0 pgmap v1387: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:40.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:40.751 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:40 smithi203 bash[23654]: cluster 2024-03-11T21:43:39.606010+0000 mon.a (mon.0) 3952 : cluster 0 osdmap e890: 8 total, 8 up, 8 in 2024-03-11T21:43:40.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:40 smithi203 bash[23654]: cluster 2024-03-11T21:43:39.657294+0000 mgr.y (mgr.14550) 801 : cluster 0 pgmap v1387: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:40.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.880 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:40.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:41.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:41.140 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:41.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.272 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:41.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:41.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:41.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:41.536 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:41.558 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:41.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:41.664 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:41.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.789 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:41.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:41.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[21512]: cluster 2024-03-11T21:43:40.061036+0000 osd.0 (osd.0) 476 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:41.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[21512]: cluster 2024-03-11T21:43:40.061248+0000 osd.0 (osd.0) 477 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[21512]: cluster 2024-03-11T21:43:40.615676+0000 mon.a (mon.0) 3953 : cluster 0 osdmap e891: 8 total, 8 up, 8 in 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[21512]: cluster 2024-03-11T21:43:41.588696+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e892: 8 total, 8 up, 8 in 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[29284]: cluster 2024-03-11T21:43:40.061036+0000 osd.0 (osd.0) 476 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[29284]: cluster 2024-03-11T21:43:40.061248+0000 osd.0 (osd.0) 477 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[29284]: cluster 2024-03-11T21:43:40.615676+0000 mon.a (mon.0) 3953 : cluster 0 osdmap e891: 8 total, 8 up, 8 in 2024-03-11T21:43:41.905 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:41 smithi143 bash[29284]: cluster 2024-03-11T21:43:41.588696+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e892: 8 total, 8 up, 8 in 2024-03-11T21:43:41.911 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:41.912 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:41.920 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:41.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:41 smithi203 bash[23654]: cluster 2024-03-11T21:43:40.061036+0000 osd.0 (osd.0) 476 : cluster 0 purged_snaps scrub starts 2024-03-11T21:43:41.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:41 smithi203 bash[23654]: cluster 2024-03-11T21:43:40.061248+0000 osd.0 (osd.0) 477 : cluster 0 purged_snaps scrub ok 2024-03-11T21:43:41.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:41 smithi203 bash[23654]: cluster 2024-03-11T21:43:40.615676+0000 mon.a (mon.0) 3953 : cluster 0 osdmap e891: 8 total, 8 up, 8 in 2024-03-11T21:43:41.923 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:41 smithi203 bash[23654]: cluster 2024-03-11T21:43:41.588696+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e892: 8 total, 8 up, 8 in 2024-03-11T21:43:42.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:42.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:42.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:42.158 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:42.185 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:42.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.316 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:42.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:42.444 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:42.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:42.576 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:42.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:42 smithi203 bash[23654]: cluster 2024-03-11T21:43:41.657953+0000 mgr.y (mgr.14550) 802 : cluster 0 pgmap v1390: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:42.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:42.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:43:42.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.863 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:42.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:42.993 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:42.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:42 smithi143 bash[29284]: cluster 2024-03-11T21:43:41.657953+0000 mgr.y (mgr.14550) 802 : cluster 0 pgmap v1390: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:42.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:42 smithi143 bash[21512]: cluster 2024-03-11T21:43:41.657953+0000 mgr.y (mgr.14550) 802 : cluster 0 pgmap v1390: 206 pgs: 206 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:43.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:43.123 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:43.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:43.253 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:43.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:43.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:43.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:43.512 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:43.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- 172.21.15.143:0/2148290550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008109370 msgr2=0x7f5008110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:43.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- 172.21.15.143:0/2148290550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008109370 0x7f5008110f40 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f5000009fc0 tx=0x7f500000b330 comp rx=0 tx=0).stop 2024-03-11T21:43:43.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- 172.21.15.143:0/2148290550 shutdown_connections 2024-03-11T21:43:43.561 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- 172.21.15.143:0/2148290550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008109370 0x7f5008110f40 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- 172.21.15.143:0/2148290550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008104290 0x7f5008104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- 172.21.15.143:0/2148290550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f5008108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- 172.21.15.143:0/2148290550 >> 172.21.15.143:0/2148290550 conn(0x7f50080ffb40 msgr2=0x7f5008101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- 172.21.15.143:0/2148290550 shutdown_connections 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- 172.21.15.143:0/2148290550 wait complete. 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 librados: starting msgr at 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 librados: starting objecter 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 Processor -- start 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- start start 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 librados: setting wanted keys 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 librados: calling monclient init 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 0x7f5008199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:43.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008109370 0x7f500819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5008113870 con 0x7f5008104c40 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081136f0 con 0x7f5008109370 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500d10b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50081139f0 con 0x7f5008104290 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f5006d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 0x7f5008199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47456/0 (socket says 172.21.15.143:47456) 2024-03-11T21:43:43.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.573+0000 7f500659c700 1 -- 172.21.15.143:0/613248684 learned_addr learned my addr 172.21.15.143:0/613248684 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500759e700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008109370 0x7f500819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 -- 172.21.15.143:0/613248684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008109370 msgr2=0x7f500819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008109370 0x7f500819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 -- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 msgr2=0x7f500819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 -- 172.21.15.143:0/613248684 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500819ed70 con 0x7f5008104290 2024-03-11T21:43:43.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500659c700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f5006d9d700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 0x7f5008199d60 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c970 tx=0x7f4ff800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 10 client.?.objecter ms_handle_connect 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff800ce60 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f4ff8004830 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500819f060 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80053a0 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500d10b700 10 client.42131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50081a68e0 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500d10b700 1 librados: init done 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500804faa0 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4ff8005540 con 0x7f5008104290 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 0x7f4ff4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:43.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(892..892 src has 587..892) v4 ==== 9021+0+0 (secure 0 0 0) 0x7f4ff808e710 con 0x7f5008104290 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 10 client.42131.objecter ms_dispatch 0x7f5008001550 osd_map(892..892 src has 587..892) v4 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 3 client.42131.objecter handle_osd_map got epochs [892,892] > 0 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 3 client.42131.objecter handle_osd_map decoding full epoch 892 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500659c700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 0x7f4ff4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 20 client.42131.objecter dump_active .. 0 homeless 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f4ffffff700 10 client.42131.objecter ms_handle_connect 0x7f4ff406f820 2024-03-11T21:43:43.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.577+0000 7f500659c700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 0x7f4ff4071ce0 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7f5008101dc0 tx=0x7f4ff0009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:43.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.581+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4ff8014030 con 0x7f5008104290 2024-03-11T21:43:43.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:43.635 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:43.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:43.757 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:43.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.801+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5008100880 con 0x7f4ff406f820 2024-03-11T21:43:43.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.817+0000 7f4ffffff700 1 -- 172.21.15.143:0/613248684 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+533125 (secure 0 0 0) 0x7f5008100880 con 0x7f4ff406f820 2024-03-11T21:43:43.810 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 10 librados: watch_flush enter 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 10 librados: watch_flush exit 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 20 client.42131.objecter shutdown clearing up homeless session... 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 10 client.42131.objecter successfully canceled tick 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 msgr2=0x7f4ff4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:43.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 0x7f4ff4071ce0 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7f5008101dc0 tx=0x7f4ff0009310 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 msgr2=0x7f5008199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 0x7f5008199d60 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c970 tx=0x7f4ff800cc80 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 shutdown_connections 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f4ff406f820 0x7f4ff4071ce0 unknown :-1 s=CLOSED pgs=3466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5008109370 0x7f500819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5008104290 0x7f5008199d60 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 --2- 172.21.15.143:0/613248684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5008104c40 0x7f500819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 >> 172.21.15.143:0/613248684 conn(0x7f50080ffb40 msgr2=0x7f5008100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 shutdown_connections 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 -- 172.21.15.143:0/613248684 wait complete. 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:43.821+0000 7f500d10b700 1 librados: shutdown 2024-03-11T21:43:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:43:43.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:43.888 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:43.991 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.018 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:44.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.149 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:44.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:44.279 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:44.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:44.408 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:44.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:44.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:44.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:44.669 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:44.772 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:44.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:44 smithi203 bash[23654]: cluster 2024-03-11T21:43:43.658861+0000 mgr.y (mgr.14550) 803 : cluster 0 pgmap v1391: 206 pgs: 206 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:44.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:44 smithi203 bash[23654]: audit 2024-03-11T21:43:43.804263+0000 mgr.y (mgr.14550) 804 : audit 0 from='client.42131 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:44.797 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:44.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:44 smithi143 bash[21512]: cluster 2024-03-11T21:43:43.658861+0000 mgr.y (mgr.14550) 803 : cluster 0 pgmap v1391: 206 pgs: 206 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:44.912 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:44 smithi143 bash[21512]: audit 2024-03-11T21:43:43.804263+0000 mgr.y (mgr.14550) 804 : audit 0 from='client.42131 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:44.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:44 smithi143 bash[29284]: cluster 2024-03-11T21:43:43.658861+0000 mgr.y (mgr.14550) 803 : cluster 0 pgmap v1391: 206 pgs: 206 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:44.912 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:44 smithi143 bash[29284]: audit 2024-03-11T21:43:43.804263+0000 mgr.y (mgr.14550) 804 : audit 0 from='client.42131 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:44.928 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:45.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:45.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.192 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:45.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.322 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:45.418 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1391,"stamp":"2024-03-11T21:43:43.658117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2819,"num_read_kb":2571,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2589,"ondisk_log_size":2589,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":618,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":790196,"kb_used_data":89796,"kb_used_omap":38,"kb_used_meta":695833,"kb_avail":749171020,"statfs":{"total":767960285184,"available":767151124480,"internally_reserved":0,"allocated":91951104,"data_stored":54664193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":712533429},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001853"},"pg_stats":[{"pgid":"39.18","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634717+0000","last_change":"2024-03-11T21:42:59.786704+0000","last_active":"2024-03-11T21:43:41.634717+0000","last_peered":"2024-03-11T21:43:41.634717+0000","last_clean":"2024-03-11T21:43:41.634717+0000","last_became_active":"2024-03-11T21:31:37.390082+0000","last_became_peered":"2024-03-11T21:31:37.390082+0000","last_unstale":"2024-03-11T21:43:41.634717+0000","last_undegraded":"2024-03-11T21:43:41.634717+0000","last_fullsized":"2024-03-11T21:43:41.634717+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.786379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.786379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.633663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":1558,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588110+0000","last_change":"2024-03-11T21:43:26.358694+0000","last_active":"2024-03-11T21:43:41.588110+0000","last_peered":"2024-03-11T21:43:41.588110+0000","last_clean":"2024-03-11T21:43:41.588110+0000","last_became_active":"2024-03-11T21:28:54.772898+0000","last_became_peered":"2024-03-11T21:28:54.772898+0000","last_unstale":"2024-03-11T21:43:41.588110+0000","last_undegraded":"2024-03-11T21:43:41.588110+0000","last_fullsized":"2024-03-11T21:43:41.588110+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.358597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:33:28.577678+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.358597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:40.493651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":1534,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633702+0000","last_change":"2024-03-11T21:41:57.330251+0000","last_active":"2024-03-11T21:43:41.633702+0000","last_peered":"2024-03-11T21:43:41.633702+0000","last_clean":"2024-03-11T21:43:41.633702+0000","last_became_active":"2024-03-11T21:28:54.807416+0000","last_became_peered":"2024-03-11T21:28:54.807416+0000","last_unstale":"2024-03-11T21:43:41.633702+0000","last_undegraded":"2024-03-11T21:43:41.633702+0000","last_fullsized":"2024-03-11T21:43:41.633702+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.330037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.330037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":1519,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636887+0000","last_change":"2024-03-11T21:40:25.470377+0000","last_active":"2024-03-11T21:43:41.636887+0000","last_peered":"2024-03-11T21:43:41.636887+0000","last_clean":"2024-03-11T21:43:41.636887+0000","last_became_active":"2024-03-11T21:28:57.517211+0000","last_became_peered":"2024-03-11T21:28:57.517211+0000","last_unstale":"2024-03-11T21:43:41.636887+0000","last_undegraded":"2024-03-11T21:43:41.636887+0000","last_fullsized":"2024-03-11T21:43:41.636887+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.470230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.470230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":1446,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604958+0000","last_change":"2024-03-11T21:41:56.250851+0000","last_active":"2024-03-11T21:43:39.604958+0000","last_peered":"2024-03-11T21:43:39.604958+0000","last_clean":"2024-03-11T21:43:39.604958+0000","last_became_active":"2024-03-11T21:31:37.402113+0000","last_became_peered":"2024-03-11T21:31:37.402113+0000","last_unstale":"2024-03-11T21:43:39.604958+0000","last_undegraded":"2024-03-11T21:43:39.604958+0000","last_fullsized":"2024-03-11T21:43:39.604958+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.250744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.250744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":1557,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590314+0000","last_change":"2024-03-11T21:42:58.341631+0000","last_active":"2024-03-11T21:43:41.590314+0000","last_peered":"2024-03-11T21:43:41.590314+0000","last_clean":"2024-03-11T21:43:41.590314+0000","last_became_active":"2024-03-11T21:28:54.833641+0000","last_became_peered":"2024-03-11T21:28:54.833641+0000","last_unstale":"2024-03-11T21:43:41.590314+0000","last_undegraded":"2024-03-11T21:43:41.590314+0000","last_fullsized":"2024-03-11T21:43:41.590314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.341432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:28.770668+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.341432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.778503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":1559,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590263+0000","last_change":"2024-03-11T21:43:27.319867+0000","last_active":"2024-03-11T21:43:41.590263+0000","last_peered":"2024-03-11T21:43:41.590263+0000","last_clean":"2024-03-11T21:43:41.590263+0000","last_became_active":"2024-03-11T21:28:54.810656+0000","last_became_peered":"2024-03-11T21:28:54.810656+0000","last_unstale":"2024-03-11T21:43:41.590263+0000","last_undegraded":"2024-03-11T21:43:41.590263+0000","last_fullsized":"2024-03-11T21:43:41.590263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.319760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.645321+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.319760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:39.518500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":1572,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590202+0000","last_change":"2024-03-11T21:43:17.510480+0000","last_active":"2024-03-11T21:43:41.590202+0000","last_peered":"2024-03-11T21:43:41.590202+0000","last_clean":"2024-03-11T21:43:41.590202+0000","last_became_active":"2024-03-11T21:28:54.853190+0000","last_became_peered":"2024-03-11T21:28:54.853190+0000","last_unstale":"2024-03-11T21:43:41.590202+0000","last_undegraded":"2024-03-11T21:43:41.590202+0000","last_fullsized":"2024-03-11T21:43:41.590202+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:17.510286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:17.510286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.842823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":1559,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633502+0000","last_change":"2024-03-11T21:43:14.817104+0000","last_active":"2024-03-11T21:43:41.633502+0000","last_peered":"2024-03-11T21:43:41.633502+0000","last_clean":"2024-03-11T21:43:41.633502+0000","last_became_active":"2024-03-11T21:28:53.811049+0000","last_became_peered":"2024-03-11T21:28:53.811049+0000","last_unstale":"2024-03-11T21:43:41.633502+0000","last_undegraded":"2024-03-11T21:43:41.633502+0000","last_fullsized":"2024-03-11T21:43:41.633502+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.816959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.816959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.630107+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":1528,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635409+0000","last_change":"2024-03-11T21:41:41.521503+0000","last_active":"2024-03-11T21:43:41.635409+0000","last_peered":"2024-03-11T21:43:41.635409+0000","last_clean":"2024-03-11T21:43:41.635409+0000","last_became_active":"2024-03-11T21:28:57.524480+0000","last_became_peered":"2024-03-11T21:28:57.524480+0000","last_unstale":"2024-03-11T21:43:41.635409+0000","last_undegraded":"2024-03-11T21:43:41.635409+0000","last_fullsized":"2024-03-11T21:43:41.635409+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.521300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.521300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":1541,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635186+0000","last_change":"2024-03-11T21:43:02.436956+0000","last_active":"2024-03-11T21:43:41.635186+0000","last_peered":"2024-03-11T21:43:41.635186+0000","last_clean":"2024-03-11T21:43:41.635186+0000","last_became_active":"2024-03-11T21:28:54.872503+0000","last_became_peered":"2024-03-11T21:28:54.872503+0000","last_unstale":"2024-03-11T21:43:41.635186+0000","last_undegraded":"2024-03-11T21:43:41.635186+0000","last_fullsized":"2024-03-11T21:43:41.635186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.436843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:13.532000+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.436843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.695057+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":1523,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.641503+0000","last_change":"2024-03-11T21:41:59.422739+0000","last_active":"2024-03-11T21:43:41.641503+0000","last_peered":"2024-03-11T21:43:41.641503+0000","last_clean":"2024-03-11T21:43:41.641503+0000","last_became_active":"2024-03-11T21:28:53.856333+0000","last_became_peered":"2024-03-11T21:28:53.856333+0000","last_unstale":"2024-03-11T21:43:41.641503+0000","last_undegraded":"2024-03-11T21:43:41.641503+0000","last_fullsized":"2024-03-11T21:43:41.641503+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:59.422534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:59.422534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":1541,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635623+0000","last_change":"2024-03-11T21:43:09.309127+0000","last_active":"2024-03-11T21:43:41.635623+0000","last_peered":"2024-03-11T21:43:41.635623+0000","last_clean":"2024-03-11T21:43:41.635623+0000","last_became_active":"2024-03-11T21:28:54.867012+0000","last_became_peered":"2024-03-11T21:28:54.867012+0000","last_unstale":"2024-03-11T21:43:41.635623+0000","last_undegraded":"2024-03-11T21:43:41.635623+0000","last_fullsized":"2024-03-11T21:43:41.635623+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:09.309029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.309029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.575434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":1551,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639095+0000","last_change":"2024-03-11T21:41:45.595342+0000","last_active":"2024-03-11T21:43:41.639095+0000","last_peered":"2024-03-11T21:43:41.639095+0000","last_clean":"2024-03-11T21:43:41.639095+0000","last_became_active":"2024-03-11T21:28:53.826192+0000","last_became_peered":"2024-03-11T21:28:53.826192+0000","last_unstale":"2024-03-11T21:43:41.639095+0000","last_undegraded":"2024-03-11T21:43:41.639095+0000","last_fullsized":"2024-03-11T21:43:41.639095+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.595129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:05.211634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.595129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":1571,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590003+0000","last_change":"2024-03-11T21:43:05.601474+0000","last_active":"2024-03-11T21:43:41.590003+0000","last_peered":"2024-03-11T21:43:41.590003+0000","last_clean":"2024-03-11T21:43:41.590003+0000","last_became_active":"2024-03-11T21:28:54.853989+0000","last_became_peered":"2024-03-11T21:28:54.853989+0000","last_unstale":"2024-03-11T21:43:41.590003+0000","last_undegraded":"2024-03-11T21:43:41.590003+0000","last_fullsized":"2024-03-11T21:43:41.590003+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:05.601329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:05.601329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:21.104290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":951,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.606372+0000","last_change":"2024-03-11T21:43:31.090288+0000","last_active":"2024-03-11T21:43:39.606372+0000","last_peered":"2024-03-11T21:43:39.606372+0000","last_clean":"2024-03-11T21:43:39.606372+0000","last_became_active":"2024-03-11T21:33:59.393763+0000","last_became_peered":"2024-03-11T21:33:59.393763+0000","last_unstale":"2024-03-11T21:43:39.606372+0000","last_undegraded":"2024-03-11T21:43:39.606372+0000","last_fullsized":"2024-03-11T21:43:39.606372+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:31.090201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:31.090201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:45.025687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":1558,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.587970+0000","last_change":"2024-03-11T21:43:02.319235+0000","last_active":"2024-03-11T21:43:41.587970+0000","last_peered":"2024-03-11T21:43:41.587970+0000","last_clean":"2024-03-11T21:43:41.587970+0000","last_became_active":"2024-03-11T21:28:54.833594+0000","last_became_peered":"2024-03-11T21:28:54.833594+0000","last_unstale":"2024-03-11T21:43:41.587970+0000","last_undegraded":"2024-03-11T21:43:41.587970+0000","last_fullsized":"2024-03-11T21:43:41.587970+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.319030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.319030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.439681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":1529,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633443+0000","last_change":"2024-03-11T21:41:51.387114+0000","last_active":"2024-03-11T21:43:41.633443+0000","last_peered":"2024-03-11T21:43:41.633443+0000","last_clean":"2024-03-11T21:43:41.633443+0000","last_became_active":"2024-03-11T21:33:59.483008+0000","last_became_peered":"2024-03-11T21:33:59.483008+0000","last_unstale":"2024-03-11T21:43:41.633443+0000","last_undegraded":"2024-03-11T21:43:41.633443+0000","last_fullsized":"2024-03-11T21:43:41.633443+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.386905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.386905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":1520,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636510+0000","last_change":"2024-03-11T21:40:19.428421+0000","last_active":"2024-03-11T21:43:41.636510+0000","last_peered":"2024-03-11T21:43:41.636510+0000","last_clean":"2024-03-11T21:43:41.636510+0000","last_became_active":"2024-03-11T21:28:57.494831+0000","last_became_peered":"2024-03-11T21:28:57.494831+0000","last_unstale":"2024-03-11T21:43:41.636510+0000","last_undegraded":"2024-03-11T21:43:41.636510+0000","last_fullsized":"2024-03-11T21:43:41.636510+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:19.428205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:40:19.428205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":1551,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635146+0000","last_change":"2024-03-11T21:41:40.664787+0000","last_active":"2024-03-11T21:43:41.635146+0000","last_peered":"2024-03-11T21:43:41.635146+0000","last_clean":"2024-03-11T21:43:41.635146+0000","last_became_active":"2024-03-11T21:28:53.856419+0000","last_became_peered":"2024-03-11T21:28:53.856419+0000","last_unstale":"2024-03-11T21:43:41.635146+0000","last_undegraded":"2024-03-11T21:43:41.635146+0000","last_fullsized":"2024-03-11T21:43:41.635146+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:40.664675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.664675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":1415,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605284+0000","last_change":"2024-03-11T21:41:32.440218+0000","last_active":"2024-03-11T21:43:39.605284+0000","last_peered":"2024-03-11T21:43:39.605284+0000","last_clean":"2024-03-11T21:43:39.605284+0000","last_became_active":"2024-03-11T21:31:37.416197+0000","last_became_peered":"2024-03-11T21:31:37.416197+0000","last_unstale":"2024-03-11T21:43:39.605284+0000","last_undegraded":"2024-03-11T21:43:39.605284+0000","last_fullsized":"2024-03-11T21:43:39.605284+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:32.440004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.440004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":1528,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637149+0000","last_change":"2024-03-11T21:43:03.770710+0000","last_active":"2024-03-11T21:43:41.637149+0000","last_peered":"2024-03-11T21:43:41.637149+0000","last_clean":"2024-03-11T21:43:41.637149+0000","last_became_active":"2024-03-11T21:28:57.568855+0000","last_became_peered":"2024-03-11T21:28:57.568855+0000","last_unstale":"2024-03-11T21:43:41.637149+0000","last_undegraded":"2024-03-11T21:43:41.637149+0000","last_fullsized":"2024-03-11T21:43:41.637149+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:03.770563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:03.770563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.680813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":1536,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638998+0000","last_change":"2024-03-11T21:42:55.708191+0000","last_active":"2024-03-11T21:43:41.638998+0000","last_peered":"2024-03-11T21:43:41.638998+0000","last_clean":"2024-03-11T21:43:41.638998+0000","last_became_active":"2024-03-11T21:28:57.546794+0000","last_became_peered":"2024-03-11T21:28:57.546794+0000","last_unstale":"2024-03-11T21:43:41.638998+0000","last_undegraded":"2024-03-11T21:43:41.638998+0000","last_fullsized":"2024-03-11T21:43:41.638998+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.707930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.707930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.053621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":1541,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636191+0000","last_change":"2024-03-11T21:43:01.415713+0000","last_active":"2024-03-11T21:43:41.636191+0000","last_peered":"2024-03-11T21:43:41.636191+0000","last_clean":"2024-03-11T21:43:41.636191+0000","last_became_active":"2024-03-11T21:28:54.850326+0000","last_became_peered":"2024-03-11T21:28:54.850326+0000","last_unstale":"2024-03-11T21:43:41.636191+0000","last_undegraded":"2024-03-11T21:43:41.636191+0000","last_fullsized":"2024-03-11T21:43:41.636191+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.415616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.415616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.957916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":1536,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631985+0000","last_change":"2024-03-11T21:42:55.545333+0000","last_active":"2024-03-11T21:43:41.631985+0000","last_peered":"2024-03-11T21:43:41.631985+0000","last_clean":"2024-03-11T21:43:41.631985+0000","last_became_active":"2024-03-11T21:28:54.822174+0000","last_became_peered":"2024-03-11T21:28:54.822174+0000","last_unstale":"2024-03-11T21:43:41.631985+0000","last_undegraded":"2024-03-11T21:43:41.631985+0000","last_fullsized":"2024-03-11T21:43:41.631985+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.640588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":1524,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633980+0000","last_change":"2024-03-11T21:43:21.483812+0000","last_active":"2024-03-11T21:43:41.633980+0000","last_peered":"2024-03-11T21:43:41.633980+0000","last_clean":"2024-03-11T21:43:41.633980+0000","last_became_active":"2024-03-11T21:28:53.852084+0000","last_became_peered":"2024-03-11T21:28:53.852084+0000","last_unstale":"2024-03-11T21:43:41.633980+0000","last_undegraded":"2024-03-11T21:43:41.633980+0000","last_fullsized":"2024-03-11T21:43:41.633980+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:21.483712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:21.483712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.108340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":951,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.606601+0000","last_change":"2024-03-11T21:43:20.116935+0000","last_active":"2024-03-11T21:43:39.606601+0000","last_peered":"2024-03-11T21:43:39.606601+0000","last_clean":"2024-03-11T21:43:39.606601+0000","last_became_active":"2024-03-11T21:33:59.394798+0000","last_became_peered":"2024-03-11T21:33:59.394798+0000","last_unstale":"2024-03-11T21:43:39.606601+0000","last_undegraded":"2024-03-11T21:43:39.606601+0000","last_fullsized":"2024-03-11T21:43:39.606601+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.116862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.384763+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.116862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:48.214569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":1552,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635343+0000","last_change":"2024-03-11T21:41:51.650830+0000","last_active":"2024-03-11T21:43:41.635343+0000","last_peered":"2024-03-11T21:43:41.635343+0000","last_clean":"2024-03-11T21:43:41.635343+0000","last_became_active":"2024-03-11T21:28:53.827310+0000","last_became_peered":"2024-03-11T21:28:53.827310+0000","last_unstale":"2024-03-11T21:43:41.635343+0000","last_undegraded":"2024-03-11T21:43:41.635343+0000","last_fullsized":"2024-03-11T21:43:41.635343+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:51.650722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.650722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":1542,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635090+0000","last_change":"2024-03-11T21:43:23.296720+0000","last_active":"2024-03-11T21:43:41.635090+0000","last_peered":"2024-03-11T21:43:41.635090+0000","last_clean":"2024-03-11T21:43:41.635090+0000","last_became_active":"2024-03-11T21:28:54.861661+0000","last_became_peered":"2024-03-11T21:28:54.861661+0000","last_unstale":"2024-03-11T21:43:41.635090+0000","last_undegraded":"2024-03-11T21:43:41.635090+0000","last_fullsized":"2024-03-11T21:43:41.635090+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:23.296514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.296514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.542454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":1524,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.641949+0000","last_change":"2024-03-11T21:42:56.351908+0000","last_active":"2024-03-11T21:43:41.641949+0000","last_peered":"2024-03-11T21:43:41.641949+0000","last_clean":"2024-03-11T21:43:41.641949+0000","last_became_active":"2024-03-11T21:28:53.852106+0000","last_became_peered":"2024-03-11T21:28:53.852106+0000","last_unstale":"2024-03-11T21:43:41.641949+0000","last_undegraded":"2024-03-11T21:43:41.641949+0000","last_fullsized":"2024-03-11T21:43:41.641949+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.351542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:19.435397+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.351542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:06.855022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":1571,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590194+0000","last_change":"2024-03-11T21:43:19.499150+0000","last_active":"2024-03-11T21:43:41.590194+0000","last_peered":"2024-03-11T21:43:41.590194+0000","last_clean":"2024-03-11T21:43:41.590194+0000","last_became_active":"2024-03-11T21:28:54.869533+0000","last_became_peered":"2024-03-11T21:28:54.869533+0000","last_unstale":"2024-03-11T21:43:41.590194+0000","last_undegraded":"2024-03-11T21:43:41.590194+0000","last_fullsized":"2024-03-11T21:43:41.590194+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:19.499002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:35.782796+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.499002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.961116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":1423,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.606750+0000","last_change":"2024-03-11T21:43:33.114611+0000","last_active":"2024-03-11T21:43:39.606750+0000","last_peered":"2024-03-11T21:43:39.606750+0000","last_clean":"2024-03-11T21:43:39.606750+0000","last_became_active":"2024-03-11T21:31:37.418177+0000","last_became_peered":"2024-03-11T21:31:37.418177+0000","last_unstale":"2024-03-11T21:43:39.606750+0000","last_undegraded":"2024-03-11T21:43:39.606750+0000","last_fullsized":"2024-03-11T21:43:39.606750+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:33.114501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:49.756221+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.114501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:46.658048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1548,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635849+0000","last_change":"2024-03-11T21:42:52.535387+0000","last_active":"2024-03-11T21:43:41.635849+0000","last_peered":"2024-03-11T21:43:41.635849+0000","last_clean":"2024-03-11T21:43:41.635849+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:43:41.635849+0000","last_undegraded":"2024-03-11T21:43:41.635849+0000","last_fullsized":"2024-03-11T21:43:41.635849+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.535087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.535087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.542418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":1933,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636059+0000","last_change":"2024-03-11T21:41:21.484873+0000","last_active":"2024-03-11T21:43:41.636059+0000","last_peered":"2024-03-11T21:43:41.636059+0000","last_clean":"2024-03-11T21:43:41.636059+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:43:41.636059+0000","last_undegraded":"2024-03-11T21:43:41.636059+0000","last_fullsized":"2024-03-11T21:43:41.636059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:21.484757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.484757+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"864'39","reported_seq":1740,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.644624+0000","last_change":"2024-03-11T21:41:26.485912+0000","last_active":"2024-03-11T21:43:41.644624+0000","last_peered":"2024-03-11T21:43:41.644624+0000","last_clean":"2024-03-11T21:43:41.644624+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:43:41.644624+0000","last_undegraded":"2024-03-11T21:43:41.644624+0000","last_fullsized":"2024-03-11T21:43:41.644624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'27","last_scrub_stamp":"2024-03-11T21:41:26.485646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.485646+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1739,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590423+0000","last_change":"2024-03-11T21:43:18.507612+0000","last_active":"2024-03-11T21:43:41.590423+0000","last_peered":"2024-03-11T21:43:41.590423+0000","last_clean":"2024-03-11T21:43:41.590423+0000","last_became_active":"2024-03-11T21:23:02.924661+0000","last_became_peered":"2024-03-11T21:23:02.924661+0000","last_unstale":"2024-03-11T21:43:41.590423+0000","last_undegraded":"2024-03-11T21:43:41.590423+0000","last_fullsized":"2024-03-11T21:43:41.590423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:18.507512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:18.507512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:39.158678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1684,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634442+0000","last_change":"2024-03-11T21:41:45.252015+0000","last_active":"2024-03-11T21:43:41.634442+0000","last_peered":"2024-03-11T21:43:41.634442+0000","last_clean":"2024-03-11T21:43:41.634442+0000","last_became_active":"2024-03-11T21:23:09.139858+0000","last_became_peered":"2024-03-11T21:23:09.139858+0000","last_unstale":"2024-03-11T21:43:41.634442+0000","last_undegraded":"2024-03-11T21:43:41.634442+0000","last_fullsized":"2024-03-11T21:43:41.634442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:45.251730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:45.251730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.412820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1550,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638629+0000","last_change":"2024-03-11T21:43:16.573624+0000","last_active":"2024-03-11T21:43:41.638629+0000","last_peered":"2024-03-11T21:43:41.638629+0000","last_clean":"2024-03-11T21:43:41.638629+0000","last_became_active":"2024-03-11T21:31:37.386712+0000","last_became_peered":"2024-03-11T21:31:37.386712+0000","last_unstale":"2024-03-11T21:43:41.638629+0000","last_undegraded":"2024-03-11T21:43:41.638629+0000","last_fullsized":"2024-03-11T21:43:41.638629+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.573343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.573343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.314480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"864'30","reported_seq":1745,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638843+0000","last_change":"2024-03-11T21:43:27.559460+0000","last_active":"2024-03-11T21:43:41.638843+0000","last_peered":"2024-03-11T21:43:41.638843+0000","last_clean":"2024-03-11T21:43:41.638843+0000","last_became_active":"2024-03-11T21:23:04.932052+0000","last_became_peered":"2024-03-11T21:23:04.932052+0000","last_unstale":"2024-03-11T21:43:41.638843+0000","last_undegraded":"2024-03-11T21:43:41.638843+0000","last_fullsized":"2024-03-11T21:43:41.638843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_deep_scrub":"864'30","last_deep_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.559051+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.066844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1605,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605054+0000","last_change":"2024-03-11T21:43:25.032236+0000","last_active":"2024-03-11T21:43:39.605054+0000","last_peered":"2024-03-11T21:43:39.605054+0000","last_clean":"2024-03-11T21:43:39.605054+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:43:39.605054+0000","last_undegraded":"2024-03-11T21:43:39.605054+0000","last_fullsized":"2024-03-11T21:43:39.605054+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.032097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.032097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:52.373268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1731,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589513+0000","last_change":"2024-03-11T21:43:06.604303+0000","last_active":"2024-03-11T21:43:41.589513+0000","last_peered":"2024-03-11T21:43:41.589513+0000","last_clean":"2024-03-11T21:43:41.589513+0000","last_became_active":"2024-03-11T21:23:07.066782+0000","last_became_peered":"2024-03-11T21:23:07.066782+0000","last_unstale":"2024-03-11T21:43:41.589513+0000","last_undegraded":"2024-03-11T21:43:41.589513+0000","last_fullsized":"2024-03-11T21:43:41.589513+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:06.604059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.604059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.007219+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1690,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635148+0000","last_change":"2024-03-11T21:41:52.458683+0000","last_active":"2024-03-11T21:43:41.635148+0000","last_peered":"2024-03-11T21:43:41.635148+0000","last_clean":"2024-03-11T21:43:41.635148+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:43:41.635148+0000","last_undegraded":"2024-03-11T21:43:41.635148+0000","last_fullsized":"2024-03-11T21:43:41.635148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:52.458535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:41:52.458535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1541,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638927+0000","last_change":"2024-03-11T21:41:55.275905+0000","last_active":"2024-03-11T21:43:41.638927+0000","last_peered":"2024-03-11T21:43:41.638927+0000","last_clean":"2024-03-11T21:43:41.638927+0000","last_became_active":"2024-03-11T21:28:51.378896+0000","last_became_peered":"2024-03-11T21:28:51.378896+0000","last_unstale":"2024-03-11T21:43:41.638927+0000","last_undegraded":"2024-03-11T21:43:41.638927+0000","last_fullsized":"2024-03-11T21:43:41.638927+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.275680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.275680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"864'53","reported_seq":1765,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642586+0000","last_change":"2024-03-11T21:42:53.391146+0000","last_active":"2024-03-11T21:43:41.642586+0000","last_peered":"2024-03-11T21:43:41.642586+0000","last_clean":"2024-03-11T21:43:41.642586+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:43:41.642586+0000","last_undegraded":"2024-03-11T21:43:41.642586+0000","last_fullsized":"2024-03-11T21:43:41.642586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'39","last_scrub_stamp":"2024-03-11T21:42:53.390867+0000","last_deep_scrub":"389'39","last_deep_scrub_stamp":"2024-03-11T21:38:59.768919+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.390867+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:04.412714+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1739,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589932+0000","last_change":"2024-03-11T21:43:12.491271+0000","last_active":"2024-03-11T21:43:41.589932+0000","last_peered":"2024-03-11T21:43:41.589932+0000","last_clean":"2024-03-11T21:43:41.589932+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:43:41.589932+0000","last_undegraded":"2024-03-11T21:43:41.589932+0000","last_fullsized":"2024-03-11T21:43:41.589932+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:12.491049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:43:12.491049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.211389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1731,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589896+0000","last_change":"2024-03-11T21:43:20.514020+0000","last_active":"2024-03-11T21:43:41.589896+0000","last_peered":"2024-03-11T21:43:41.589896+0000","last_clean":"2024-03-11T21:43:41.589896+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:43:41.589896+0000","last_undegraded":"2024-03-11T21:43:41.589896+0000","last_fullsized":"2024-03-11T21:43:41.589896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.513928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:30.130050+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.513928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.755239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1679,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631720+0000","last_change":"2024-03-11T21:43:14.517541+0000","last_active":"2024-03-11T21:43:41.631720+0000","last_peered":"2024-03-11T21:43:41.631720+0000","last_clean":"2024-03-11T21:43:41.631720+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:43:41.631720+0000","last_undegraded":"2024-03-11T21:43:41.631720+0000","last_fullsized":"2024-03-11T21:43:41.631720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.517349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.517349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.900019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":928,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631589+0000","last_change":"2024-03-11T21:42:53.579836+0000","last_active":"2024-03-11T21:43:41.631589+0000","last_peered":"2024-03-11T21:43:41.631589+0000","last_clean":"2024-03-11T21:43:41.631589+0000","last_became_active":"2024-03-11T21:37:52.451599+0000","last_became_peered":"2024-03-11T21:37:52.451599+0000","last_unstale":"2024-03-11T21:43:41.631589+0000","last_undegraded":"2024-03-11T21:43:41.631589+0000","last_fullsized":"2024-03-11T21:43:41.631589+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:42:53.579671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:42:53.579671+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.307026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1425,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605125+0000","last_change":"2024-03-11T21:43:05.859471+0000","last_active":"2024-03-11T21:43:39.605125+0000","last_peered":"2024-03-11T21:43:39.605125+0000","last_clean":"2024-03-11T21:43:39.605125+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:43:39.605125+0000","last_undegraded":"2024-03-11T21:43:39.605125+0000","last_fullsized":"2024-03-11T21:43:39.605125+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:05.859254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.054820+0000","last_clean_scrub_stamp":"2024-03-11T21:43:05.859254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.817814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"864'60","reported_seq":1819,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589342+0000","last_change":"2024-03-11T21:43:19.376362+0000","last_active":"2024-03-11T21:43:41.589342+0000","last_peered":"2024-03-11T21:43:41.589342+0000","last_clean":"2024-03-11T21:43:41.589342+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:43:41.589342+0000","last_undegraded":"2024-03-11T21:43:41.589342+0000","last_fullsized":"2024-03-11T21:43:41.589342+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'60","last_scrub_stamp":"2024-03-11T21:43:19.376099+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.376099+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.890107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1691,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639898+0000","last_change":"2024-03-11T21:41:29.509226+0000","last_active":"2024-03-11T21:43:41.639898+0000","last_peered":"2024-03-11T21:43:41.639898+0000","last_clean":"2024-03-11T21:43:41.639898+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:43:41.639898+0000","last_undegraded":"2024-03-11T21:43:41.639898+0000","last_fullsized":"2024-03-11T21:43:41.639898+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.509105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.509105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1603,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605275+0000","last_change":"2024-03-11T21:42:00.292135+0000","last_active":"2024-03-11T21:43:39.605275+0000","last_peered":"2024-03-11T21:43:39.605275+0000","last_clean":"2024-03-11T21:43:39.605275+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:43:39.605275+0000","last_undegraded":"2024-03-11T21:43:39.605275+0000","last_fullsized":"2024-03-11T21:43:39.605275+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.292001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1727,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590349+0000","last_change":"2024-03-11T21:42:55.510765+0000","last_active":"2024-03-11T21:43:41.590349+0000","last_peered":"2024-03-11T21:43:41.590349+0000","last_clean":"2024-03-11T21:43:41.590349+0000","last_became_active":"2024-03-11T21:23:09.153532+0000","last_became_peered":"2024-03-11T21:23:09.153532+0000","last_unstale":"2024-03-11T21:43:41.590349+0000","last_undegraded":"2024-03-11T21:43:41.590349+0000","last_fullsized":"2024-03-11T21:43:41.590349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.510373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.510373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.380386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":1972,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636394+0000","last_change":"2024-03-11T21:41:30.525150+0000","last_active":"2024-03-11T21:43:41.636394+0000","last_peered":"2024-03-11T21:43:41.636394+0000","last_clean":"2024-03-11T21:43:41.636394+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:43:41.636394+0000","last_undegraded":"2024-03-11T21:43:41.636394+0000","last_fullsized":"2024-03-11T21:43:41.636394+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:30.525009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.525009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1552,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636189+0000","last_change":"2024-03-11T21:43:24.734683+0000","last_active":"2024-03-11T21:43:41.636189+0000","last_peered":"2024-03-11T21:43:41.636189+0000","last_clean":"2024-03-11T21:43:41.636189+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:43:41.636189+0000","last_undegraded":"2024-03-11T21:43:41.636189+0000","last_fullsized":"2024-03-11T21:43:41.636189+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:43:24.734459+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.734459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.277343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"864'21","reported_seq":1729,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637456+0000","last_change":"2024-03-11T21:43:01.814079+0000","last_active":"2024-03-11T21:43:41.637456+0000","last_peered":"2024-03-11T21:43:41.637456+0000","last_clean":"2024-03-11T21:43:41.637456+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:43:41.637456+0000","last_undegraded":"2024-03-11T21:43:41.637456+0000","last_fullsized":"2024-03-11T21:43:41.637456+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:43:01.813932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.813932+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.924935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1571,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.606394+0000","last_change":"2024-03-11T21:43:09.964115+0000","last_active":"2024-03-11T21:43:39.606394+0000","last_peered":"2024-03-11T21:43:39.606394+0000","last_clean":"2024-03-11T21:43:39.606394+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:43:39.606394+0000","last_undegraded":"2024-03-11T21:43:39.606394+0000","last_fullsized":"2024-03-11T21:43:39.606394+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:09.964027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.964027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.470467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":1990,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590078+0000","last_change":"2024-03-11T21:42:58.546849+0000","last_active":"2024-03-11T21:43:41.590078+0000","last_peered":"2024-03-11T21:43:41.590078+0000","last_clean":"2024-03-11T21:43:41.590078+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:43:41.590078+0000","last_undegraded":"2024-03-11T21:43:41.590078+0000","last_fullsized":"2024-03-11T21:43:41.590078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:42:58.546638+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:24:11.167229+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.546638+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.143964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1724,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642799+0000","last_change":"2024-03-11T21:43:11.646001+0000","last_active":"2024-03-11T21:43:41.642799+0000","last_peered":"2024-03-11T21:43:41.642799+0000","last_clean":"2024-03-11T21:43:41.642799+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:43:41.642799+0000","last_undegraded":"2024-03-11T21:43:41.642799+0000","last_fullsized":"2024-03-11T21:43:41.642799+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:11.645880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.645880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:40.786412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":928,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633216+0000","last_change":"2024-03-11T21:43:24.638284+0000","last_active":"2024-03-11T21:43:41.633216+0000","last_peered":"2024-03-11T21:43:41.633216+0000","last_clean":"2024-03-11T21:43:41.633216+0000","last_became_active":"2024-03-11T21:37:52.459900+0000","last_became_peered":"2024-03-11T21:37:52.459900+0000","last_unstale":"2024-03-11T21:43:41.633216+0000","last_undegraded":"2024-03-11T21:43:41.633216+0000","last_fullsized":"2024-03-11T21:43:41.633216+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:24.638122+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:43:24.638122+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.638122+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.581584+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1547,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637284+0000","last_change":"2024-03-11T21:41:25.364494+0000","last_active":"2024-03-11T21:43:41.637284+0000","last_peered":"2024-03-11T21:43:41.637284+0000","last_clean":"2024-03-11T21:43:41.637284+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:43:41.637284+0000","last_undegraded":"2024-03-11T21:43:41.637284+0000","last_fullsized":"2024-03-11T21:43:41.637284+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.364387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.364387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"864'39","reported_seq":1765,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635855+0000","last_change":"2024-03-11T21:43:11.702556+0000","last_active":"2024-03-11T21:43:41.635855+0000","last_peered":"2024-03-11T21:43:41.635855+0000","last_clean":"2024-03-11T21:43:41.635855+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:43:41.635855+0000","last_undegraded":"2024-03-11T21:43:41.635855+0000","last_fullsized":"2024-03-11T21:43:41.635855+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:43:11.702298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.702298+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.406412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1599,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604402+0000","last_change":"2024-03-11T21:42:55.086019+0000","last_active":"2024-03-11T21:43:39.604402+0000","last_peered":"2024-03-11T21:43:39.604402+0000","last_clean":"2024-03-11T21:43:39.604402+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:43:39.604402+0000","last_undegraded":"2024-03-11T21:43:39.604402+0000","last_fullsized":"2024-03-11T21:43:39.604402+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:55.085823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:08.421516+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.085823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.857865+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"84'11","reported_seq":1969,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.641217+0000","last_change":"2024-03-11T21:41:46.623952+0000","last_active":"2024-03-11T21:43:41.641217+0000","last_peered":"2024-03-11T21:43:41.641217+0000","last_clean":"2024-03-11T21:43:41.641217+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:43:41.641217+0000","last_undegraded":"2024-03-11T21:43:41.641217+0000","last_fullsized":"2024-03-11T21:43:41.641217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:41:46.623794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.623794+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007502+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1701,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637552+0000","last_change":"2024-03-11T21:41:58.366463+0000","last_active":"2024-03-11T21:43:41.637552+0000","last_peered":"2024-03-11T21:43:41.637552+0000","last_clean":"2024-03-11T21:43:41.637552+0000","last_became_active":"2024-03-11T21:31:37.387649+0000","last_became_peered":"2024-03-11T21:31:37.387649+0000","last_unstale":"2024-03-11T21:43:41.637552+0000","last_undegraded":"2024-03-11T21:43:41.637552+0000","last_fullsized":"2024-03-11T21:43:41.637552+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.366196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.366196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1538,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637128+0000","last_change":"2024-03-11T21:42:52.391444+0000","last_active":"2024-03-11T21:43:41.637128+0000","last_peered":"2024-03-11T21:43:41.637128+0000","last_clean":"2024-03-11T21:43:41.637128+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:43:41.637128+0000","last_undegraded":"2024-03-11T21:43:41.637128+0000","last_fullsized":"2024-03-11T21:43:41.637128+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.391288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.391288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:20.913321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"864'29","reported_seq":1761,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632804+0000","last_change":"2024-03-11T21:43:02.625794+0000","last_active":"2024-03-11T21:43:41.632804+0000","last_peered":"2024-03-11T21:43:41.632804+0000","last_clean":"2024-03-11T21:43:41.632804+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:43:41.632804+0000","last_undegraded":"2024-03-11T21:43:41.632804+0000","last_fullsized":"2024-03-11T21:43:41.632804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'23","last_scrub_stamp":"2024-03-11T21:43:02.625424+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.625424+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:17.526217+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1700,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632276+0000","last_change":"2024-03-11T21:41:18.465526+0000","last_active":"2024-03-11T21:43:41.632276+0000","last_peered":"2024-03-11T21:43:41.632276+0000","last_clean":"2024-03-11T21:43:41.632276+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:43:41.632276+0000","last_undegraded":"2024-03-11T21:43:41.632276+0000","last_fullsized":"2024-03-11T21:43:41.632276+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.465412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.465412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1696,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.640753+0000","last_change":"2024-03-11T21:43:26.557894+0000","last_active":"2024-03-11T21:43:41.640753+0000","last_peered":"2024-03-11T21:43:41.640753+0000","last_clean":"2024-03-11T21:43:41.640753+0000","last_became_active":"2024-03-11T21:23:07.096752+0000","last_became_peered":"2024-03-11T21:23:07.096752+0000","last_unstale":"2024-03-11T21:43:41.640753+0000","last_undegraded":"2024-03-11T21:43:41.640753+0000","last_fullsized":"2024-03-11T21:43:41.640753+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.557601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.557601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:43.491296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1679,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637354+0000","last_change":"2024-03-11T21:41:26.327523+0000","last_active":"2024-03-11T21:43:41.637354+0000","last_peered":"2024-03-11T21:43:41.637354+0000","last_clean":"2024-03-11T21:43:41.637354+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:43:41.637354+0000","last_undegraded":"2024-03-11T21:43:41.637354+0000","last_fullsized":"2024-03-11T21:43:41.637354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.327416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.327416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":921,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637578+0000","last_change":"2024-03-11T21:41:57.696381+0000","last_active":"2024-03-11T21:43:41.637578+0000","last_peered":"2024-03-11T21:43:41.637578+0000","last_clean":"2024-03-11T21:43:41.637578+0000","last_became_active":"2024-03-11T21:37:52.461823+0000","last_became_peered":"2024-03-11T21:37:52.461823+0000","last_unstale":"2024-03-11T21:43:41.637578+0000","last_undegraded":"2024-03-11T21:43:41.637578+0000","last_fullsized":"2024-03-11T21:43:41.637578+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:41:57.696205+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:37:52.699591+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.696205+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790743+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637529+0000","last_change":"2024-03-11T21:43:29.720004+0000","last_active":"2024-03-11T21:43:41.637529+0000","last_peered":"2024-03-11T21:43:41.637529+0000","last_clean":"2024-03-11T21:43:41.637529+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:43:41.637529+0000","last_undegraded":"2024-03-11T21:43:41.637529+0000","last_fullsized":"2024-03-11T21:43:41.637529+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:29.719865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.719865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:54.789094+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"297'13","reported_seq":1960,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636389+0000","last_change":"2024-03-11T21:43:26.745028+0000","last_active":"2024-03-11T21:43:41.636389+0000","last_peered":"2024-03-11T21:43:41.636389+0000","last_clean":"2024-03-11T21:43:41.636389+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:43:41.636389+0000","last_undegraded":"2024-03-11T21:43:41.636389+0000","last_fullsized":"2024-03-11T21:43:41.636389+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:43:26.744813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.744813+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.970870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1728,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632811+0000","last_change":"2024-03-11T21:42:56.717568+0000","last_active":"2024-03-11T21:43:41.632811+0000","last_peered":"2024-03-11T21:43:41.632811+0000","last_clean":"2024-03-11T21:43:41.632811+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:43:41.632811+0000","last_undegraded":"2024-03-11T21:43:41.632811+0000","last_fullsized":"2024-03-11T21:43:41.632811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.717329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:52.898357+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.717329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:56.987508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"864'36","reported_seq":1769,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635979+0000","last_change":"2024-03-11T21:43:25.284142+0000","last_active":"2024-03-11T21:43:41.635979+0000","last_peered":"2024-03-11T21:43:41.635979+0000","last_clean":"2024-03-11T21:43:41.635979+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:43:41.635979+0000","last_undegraded":"2024-03-11T21:43:41.635979+0000","last_fullsized":"2024-03-11T21:43:41.635979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'36","last_scrub_stamp":"2024-03-11T21:43:25.283990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.283990+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:47.098678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":1743,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590123+0000","last_change":"2024-03-11T21:43:34.486923+0000","last_active":"2024-03-11T21:43:41.590123+0000","last_peered":"2024-03-11T21:43:41.590123+0000","last_clean":"2024-03-11T21:43:41.590123+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:43:41.590123+0000","last_undegraded":"2024-03-11T21:43:41.590123+0000","last_fullsized":"2024-03-11T21:43:41.590123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:43:34.486694+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:35:00.408810+0000","last_clean_scrub_stamp":"2024-03-11T21:43:34.486694+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:57.173042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1681,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638206+0000","last_change":"2024-03-11T21:40:23.749973+0000","last_active":"2024-03-11T21:43:41.638206+0000","last_peered":"2024-03-11T21:43:41.638206+0000","last_clean":"2024-03-11T21:43:41.638206+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:43:41.638206+0000","last_undegraded":"2024-03-11T21:43:41.638206+0000","last_fullsized":"2024-03-11T21:43:41.638206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:40:23.749772+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:40:23.749772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":935,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636924+0000","last_change":"2024-03-11T21:41:56.716081+0000","last_active":"2024-03-11T21:43:41.636924+0000","last_peered":"2024-03-11T21:43:41.636924+0000","last_clean":"2024-03-11T21:43:41.636924+0000","last_became_active":"2024-03-11T21:37:52.440743+0000","last_became_peered":"2024-03-11T21:37:52.440743+0000","last_unstale":"2024-03-11T21:43:41.636924+0000","last_undegraded":"2024-03-11T21:43:41.636924+0000","last_fullsized":"2024-03-11T21:43:41.636924+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:56.715868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.715868+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642164+0000","last_change":"2024-03-11T21:41:42.428955+0000","last_active":"2024-03-11T21:43:41.642164+0000","last_peered":"2024-03-11T21:43:41.642164+0000","last_clean":"2024-03-11T21:43:41.642164+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:43:41.642164+0000","last_undegraded":"2024-03-11T21:43:41.642164+0000","last_fullsized":"2024-03-11T21:43:41.642164+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.428709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.428709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1727,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632651+0000","last_change":"2024-03-11T21:43:31.912543+0000","last_active":"2024-03-11T21:43:41.632651+0000","last_peered":"2024-03-11T21:43:41.632651+0000","last_clean":"2024-03-11T21:43:41.632651+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:43:41.632651+0000","last_undegraded":"2024-03-11T21:43:41.632651+0000","last_fullsized":"2024-03-11T21:43:41.632651+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:31.912310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:31.912310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:45.500693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2914,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636032+0000","last_change":"2024-03-11T21:41:20.351387+0000","last_active":"2024-03-11T21:43:41.636032+0000","last_peered":"2024-03-11T21:43:41.636032+0000","last_clean":"2024-03-11T21:43:41.636032+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:43:41.636032+0000","last_undegraded":"2024-03-11T21:43:41.636032+0000","last_fullsized":"2024-03-11T21:43:41.636032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:41:20.351150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.351150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1218,"num_read_kb":1213,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"864'21","reported_seq":1726,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639259+0000","last_change":"2024-03-11T21:42:03.753084+0000","last_active":"2024-03-11T21:43:41.639259+0000","last_peered":"2024-03-11T21:43:41.639259+0000","last_clean":"2024-03-11T21:43:41.639259+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:43:41.639259+0000","last_undegraded":"2024-03-11T21:43:41.639259+0000","last_fullsized":"2024-03-11T21:43:41.639259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'17","last_scrub_stamp":"2024-03-11T21:42:03.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:03.752924+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":1624,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605813+0000","last_change":"2024-03-11T21:43:07.910395+0000","last_active":"2024-03-11T21:43:39.605813+0000","last_peered":"2024-03-11T21:43:39.605813+0000","last_clean":"2024-03-11T21:43:39.605813+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:43:39.605813+0000","last_undegraded":"2024-03-11T21:43:39.605813+0000","last_fullsized":"2024-03-11T21:43:39.605813+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:43:07.910034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.910034+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:36.399679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1715,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633859+0000","last_change":"2024-03-11T21:43:18.721177+0000","last_active":"2024-03-11T21:43:41.633859+0000","last_peered":"2024-03-11T21:43:41.633859+0000","last_clean":"2024-03-11T21:43:41.633859+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:43:41.633859+0000","last_undegraded":"2024-03-11T21:43:41.633859+0000","last_fullsized":"2024-03-11T21:43:41.633859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:18.720924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:58.912133+0000","last_clean_scrub_stamp":"2024-03-11T21:43:18.720924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.027183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":944,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589720+0000","last_change":"2024-03-11T21:43:09.365565+0000","last_active":"2024-03-11T21:43:41.589720+0000","last_peered":"2024-03-11T21:43:41.589720+0000","last_clean":"2024-03-11T21:43:41.589720+0000","last_became_active":"2024-03-11T21:37:52.449191+0000","last_became_peered":"2024-03-11T21:37:52.449191+0000","last_unstale":"2024-03-11T21:43:41.589720+0000","last_undegraded":"2024-03-11T21:43:41.589720+0000","last_fullsized":"2024-03-11T21:43:41.589720+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:09.365340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.365340+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.184435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642710+0000","last_change":"2024-03-11T21:41:35.427640+0000","last_active":"2024-03-11T21:43:41.642710+0000","last_peered":"2024-03-11T21:43:41.642710+0000","last_clean":"2024-03-11T21:43:41.642710+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:43:41.642710+0000","last_undegraded":"2024-03-11T21:43:41.642710+0000","last_fullsized":"2024-03-11T21:43:41.642710+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:35.427438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.427438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1710,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588868+0000","last_change":"2024-03-11T21:43:24.336932+0000","last_active":"2024-03-11T21:43:41.588868+0000","last_peered":"2024-03-11T21:43:41.588868+0000","last_clean":"2024-03-11T21:43:41.588868+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:43:41.588868+0000","last_undegraded":"2024-03-11T21:43:41.588868+0000","last_fullsized":"2024-03-11T21:43:41.588868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.336783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.336783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.429760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"864'66","reported_seq":1926,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.607835+0000","last_change":"2024-03-11T21:43:29.115547+0000","last_active":"2024-03-11T21:43:39.607835+0000","last_peered":"2024-03-11T21:43:39.607835+0000","last_clean":"2024-03-11T21:43:39.607835+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:43:39.607835+0000","last_undegraded":"2024-03-11T21:43:39.607835+0000","last_fullsized":"2024-03-11T21:43:39.607835+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'66","last_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_deep_scrub":"864'66","last_deep_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.115409+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:58.235006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1691,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642446+0000","last_change":"2024-03-11T21:41:44.440095+0000","last_active":"2024-03-11T21:43:41.642446+0000","last_peered":"2024-03-11T21:43:41.642446+0000","last_clean":"2024-03-11T21:43:41.642446+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:43:41.642446+0000","last_undegraded":"2024-03-11T21:43:41.642446+0000","last_fullsized":"2024-03-11T21:43:41.642446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:44.439846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:44.439846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1563,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604032+0000","last_change":"2024-03-11T21:43:02.848311+0000","last_active":"2024-03-11T21:43:39.604032+0000","last_peered":"2024-03-11T21:43:39.604032+0000","last_clean":"2024-03-11T21:43:39.604032+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:43:39.604032+0000","last_undegraded":"2024-03-11T21:43:39.604032+0000","last_fullsized":"2024-03-11T21:43:39.604032+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.848093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.848093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.818328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":933,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604414+0000","last_change":"2024-03-11T21:43:08.956187+0000","last_active":"2024-03-11T21:43:39.604414+0000","last_peered":"2024-03-11T21:43:39.604414+0000","last_clean":"2024-03-11T21:43:39.604414+0000","last_became_active":"2024-03-11T21:37:52.444124+0000","last_became_peered":"2024-03-11T21:37:52.444124+0000","last_unstale":"2024-03-11T21:43:39.604414+0000","last_undegraded":"2024-03-11T21:43:39.604414+0000","last_fullsized":"2024-03-11T21:43:39.604414+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:08.955941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.955941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:24.648893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1547,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635626+0000","last_change":"2024-03-11T21:42:54.461064+0000","last_active":"2024-03-11T21:43:41.635626+0000","last_peered":"2024-03-11T21:43:41.635626+0000","last_clean":"2024-03-11T21:43:41.635626+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:43:41.635626+0000","last_undegraded":"2024-03-11T21:43:41.635626+0000","last_fullsized":"2024-03-11T21:43:41.635626+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.460846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.460846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:09.439479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1705,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.640580+0000","last_change":"2024-03-11T21:43:21.554709+0000","last_active":"2024-03-11T21:43:41.640580+0000","last_peered":"2024-03-11T21:43:41.640580+0000","last_clean":"2024-03-11T21:43:41.640580+0000","last_became_active":"2024-03-11T21:31:37.390793+0000","last_became_peered":"2024-03-11T21:31:37.390793+0000","last_unstale":"2024-03-11T21:43:41.640580+0000","last_undegraded":"2024-03-11T21:43:41.640580+0000","last_fullsized":"2024-03-11T21:43:41.640580+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:21.554345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:21.554345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:50.053679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1701,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631968+0000","last_change":"2024-03-11T21:42:04.407830+0000","last_active":"2024-03-11T21:43:41.631968+0000","last_peered":"2024-03-11T21:43:41.631968+0000","last_clean":"2024-03-11T21:43:41.631968+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:43:41.631968+0000","last_undegraded":"2024-03-11T21:43:41.631968+0000","last_fullsized":"2024-03-11T21:43:41.631968+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:04.407615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:42:04.407615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"864'42","reported_seq":1761,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635035+0000","last_change":"2024-03-11T21:42:00.572712+0000","last_active":"2024-03-11T21:43:41.635035+0000","last_peered":"2024-03-11T21:43:41.635035+0000","last_clean":"2024-03-11T21:43:41.635035+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:43:41.635035+0000","last_undegraded":"2024-03-11T21:43:41.635035+0000","last_fullsized":"2024-03-11T21:43:41.635035+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:00.572462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:00.572462+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1565,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.607173+0000","last_change":"2024-03-11T21:41:33.473130+0000","last_active":"2024-03-11T21:43:39.607173+0000","last_peered":"2024-03-11T21:43:39.607173+0000","last_clean":"2024-03-11T21:43:39.607173+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:43:39.607173+0000","last_undegraded":"2024-03-11T21:43:39.607173+0000","last_fullsized":"2024-03-11T21:43:39.607173+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:33.472920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:33.472920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634887+0000","last_change":"2024-03-11T21:42:01.660218+0000","last_active":"2024-03-11T21:43:41.634887+0000","last_peered":"2024-03-11T21:43:41.634887+0000","last_clean":"2024-03-11T21:43:41.634887+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:43:41.634887+0000","last_undegraded":"2024-03-11T21:43:41.634887+0000","last_fullsized":"2024-03-11T21:43:41.634887+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.660109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.660109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1675,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.641945+0000","last_change":"2024-03-11T21:40:22.764357+0000","last_active":"2024-03-11T21:43:41.641945+0000","last_peered":"2024-03-11T21:43:41.641945+0000","last_clean":"2024-03-11T21:43:41.641945+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:43:41.641945+0000","last_undegraded":"2024-03-11T21:43:41.641945+0000","last_fullsized":"2024-03-11T21:43:41.641945+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:22.764240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:22.764240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":1965,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631983+0000","last_change":"2024-03-11T21:41:41.451026+0000","last_active":"2024-03-11T21:43:41.631983+0000","last_peered":"2024-03-11T21:43:41.631983+0000","last_clean":"2024-03-11T21:43:41.631983+0000","last_became_active":"2024-03-11T21:37:46.440092+0000","last_became_peered":"2024-03-11T21:37:46.440092+0000","last_unstale":"2024-03-11T21:43:41.631983+0000","last_undegraded":"2024-03-11T21:43:41.631983+0000","last_fullsized":"2024-03-11T21:43:41.631983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:41.450896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.450896+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1704,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635633+0000","last_change":"2024-03-11T21:43:10.494102+0000","last_active":"2024-03-11T21:43:41.635633+0000","last_peered":"2024-03-11T21:43:41.635633+0000","last_clean":"2024-03-11T21:43:41.635633+0000","last_became_active":"2024-03-11T21:23:02.938437+0000","last_became_peered":"2024-03-11T21:23:02.938437+0000","last_unstale":"2024-03-11T21:43:41.635633+0000","last_undegraded":"2024-03-11T21:43:41.635633+0000","last_fullsized":"2024-03-11T21:43:41.635633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.493878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:32.395316+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.493878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.695374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"864'30","reported_seq":1732,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637014+0000","last_change":"2024-03-11T21:41:35.415804+0000","last_active":"2024-03-11T21:43:41.637014+0000","last_peered":"2024-03-11T21:43:41.637014+0000","last_clean":"2024-03-11T21:43:41.637014+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:43:41.637014+0000","last_undegraded":"2024-03-11T21:43:41.637014+0000","last_fullsized":"2024-03-11T21:43:41.637014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:41:35.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:35.415630+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":494,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636044+0000","last_change":"2024-03-11T21:41:37.415103+0000","last_active":"2024-03-11T21:43:41.636044+0000","last_peered":"2024-03-11T21:43:41.636044+0000","last_clean":"2024-03-11T21:43:41.636044+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:43:41.636044+0000","last_undegraded":"2024-03-11T21:43:41.636044+0000","last_fullsized":"2024-03-11T21:43:41.636044+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.414919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.414919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":1972,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636418+0000","last_change":"2024-03-11T21:41:38.433752+0000","last_active":"2024-03-11T21:43:41.636418+0000","last_peered":"2024-03-11T21:43:41.636418+0000","last_clean":"2024-03-11T21:43:41.636418+0000","last_became_active":"2024-03-11T21:37:52.463122+0000","last_became_peered":"2024-03-11T21:37:52.463122+0000","last_unstale":"2024-03-11T21:43:41.636418+0000","last_undegraded":"2024-03-11T21:43:41.636418+0000","last_fullsized":"2024-03-11T21:43:41.636418+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:41:38.433381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:41:38.433381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1550,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634083+0000","last_change":"2024-03-11T21:43:19.522646+0000","last_active":"2024-03-11T21:43:41.634083+0000","last_peered":"2024-03-11T21:43:41.634083+0000","last_clean":"2024-03-11T21:43:41.634083+0000","last_became_active":"2024-03-11T21:31:37.387138+0000","last_became_peered":"2024-03-11T21:31:37.387138+0000","last_unstale":"2024-03-11T21:43:41.634083+0000","last_undegraded":"2024-03-11T21:43:41.634083+0000","last_fullsized":"2024-03-11T21:43:41.634083+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:19.522430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:19.522430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.951836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1731,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590555+0000","last_change":"2024-03-11T21:43:04.644845+0000","last_active":"2024-03-11T21:43:41.590555+0000","last_peered":"2024-03-11T21:43:41.590555+0000","last_clean":"2024-03-11T21:43:41.590555+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:43:41.590555+0000","last_undegraded":"2024-03-11T21:43:41.590555+0000","last_fullsized":"2024-03-11T21:43:41.590555+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:04.644707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.644707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.280834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1739,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589524+0000","last_change":"2024-03-11T21:43:11.524842+0000","last_active":"2024-03-11T21:43:41.589524+0000","last_peered":"2024-03-11T21:43:41.589524+0000","last_clean":"2024-03-11T21:43:41.589524+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:43:41.589524+0000","last_undegraded":"2024-03-11T21:43:41.589524+0000","last_fullsized":"2024-03-11T21:43:41.589524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:11.524712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:11.524712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.769096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"864'50","reported_seq":1674,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605360+0000","last_change":"2024-03-11T21:43:34.146577+0000","last_active":"2024-03-11T21:43:39.605360+0000","last_peered":"2024-03-11T21:43:39.605360+0000","last_clean":"2024-03-11T21:43:39.605360+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:43:39.605360+0000","last_undegraded":"2024-03-11T21:43:39.605360+0000","last_fullsized":"2024-03-11T21:43:39.605360+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'50","last_scrub_stamp":"2024-03-11T21:43:34.146449+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:43:34.146449+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:45.787704+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1679,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642379+0000","last_change":"2024-03-11T21:43:01.436177+0000","last_active":"2024-03-11T21:43:41.642379+0000","last_peered":"2024-03-11T21:43:41.642379+0000","last_clean":"2024-03-11T21:43:41.642379+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:43:41.642379+0000","last_undegraded":"2024-03-11T21:43:41.642379+0000","last_fullsized":"2024-03-11T21:43:41.642379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:01.435975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:01.435975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:28.229611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":928,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633014+0000","last_change":"2024-03-11T21:43:33.481151+0000","last_active":"2024-03-11T21:43:41.633014+0000","last_peered":"2024-03-11T21:43:41.633014+0000","last_clean":"2024-03-11T21:43:41.633014+0000","last_became_active":"2024-03-11T21:37:52.456487+0000","last_became_peered":"2024-03-11T21:37:52.456487+0000","last_unstale":"2024-03-11T21:43:41.633014+0000","last_undegraded":"2024-03-11T21:43:41.633014+0000","last_fullsized":"2024-03-11T21:43:41.633014+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:33.481025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.481025+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:50.181977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1547,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634801+0000","last_change":"2024-03-11T21:42:01.549509+0000","last_active":"2024-03-11T21:43:41.634801+0000","last_peered":"2024-03-11T21:43:41.634801+0000","last_clean":"2024-03-11T21:43:41.634801+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:43:41.634801+0000","last_undegraded":"2024-03-11T21:43:41.634801+0000","last_fullsized":"2024-03-11T21:43:41.634801+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:42:01.549350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1703,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635788+0000","last_change":"2024-03-11T21:41:58.613981+0000","last_active":"2024-03-11T21:43:41.635788+0000","last_peered":"2024-03-11T21:43:41.635788+0000","last_clean":"2024-03-11T21:43:41.635788+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:43:41.635788+0000","last_undegraded":"2024-03-11T21:43:41.635788+0000","last_fullsized":"2024-03-11T21:43:41.635788+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:58.613685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:58.613685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"864'47","reported_seq":1789,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634881+0000","last_change":"2024-03-11T21:41:29.674697+0000","last_active":"2024-03-11T21:43:41.634881+0000","last_peered":"2024-03-11T21:43:41.634881+0000","last_clean":"2024-03-11T21:43:41.634881+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:43:41.634881+0000","last_undegraded":"2024-03-11T21:43:41.634881+0000","last_fullsized":"2024-03-11T21:43:41.634881+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'37","last_scrub_stamp":"2024-03-11T21:41:29.674458+0000","last_deep_scrub":"113'27","last_deep_scrub_stamp":"2024-03-11T21:25:30.974544+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.674458+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007426+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1705,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634728+0000","last_change":"2024-03-11T21:41:34.232210+0000","last_active":"2024-03-11T21:43:41.634728+0000","last_peered":"2024-03-11T21:43:41.634728+0000","last_clean":"2024-03-11T21:43:41.634728+0000","last_became_active":"2024-03-11T21:31:37.388378+0000","last_became_peered":"2024-03-11T21:31:37.388378+0000","last_unstale":"2024-03-11T21:43:41.634728+0000","last_undegraded":"2024-03-11T21:43:41.634728+0000","last_fullsized":"2024-03-11T21:43:41.634728+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:34.231996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:34.231996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.412765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1679,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634632+0000","last_change":"2024-03-11T21:41:17.378484+0000","last_active":"2024-03-11T21:43:41.634632+0000","last_peered":"2024-03-11T21:43:41.634632+0000","last_clean":"2024-03-11T21:43:41.634632+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:43:41.634632+0000","last_undegraded":"2024-03-11T21:43:41.634632+0000","last_fullsized":"2024-03-11T21:43:41.634632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.378291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.378291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":944,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589641+0000","last_change":"2024-03-11T21:43:25.374247+0000","last_active":"2024-03-11T21:43:41.589641+0000","last_peered":"2024-03-11T21:43:41.589641+0000","last_clean":"2024-03-11T21:43:41.589641+0000","last_became_active":"2024-03-11T21:37:52.462184+0000","last_became_peered":"2024-03-11T21:37:52.462184+0000","last_unstale":"2024-03-11T21:43:41.589641+0000","last_undegraded":"2024-03-11T21:43:41.589641+0000","last_fullsized":"2024-03-11T21:43:41.589641+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:43:25.374049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.374049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:48.553744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1550,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639734+0000","last_change":"2024-03-11T21:43:07.463108+0000","last_active":"2024-03-11T21:43:41.639734+0000","last_peered":"2024-03-11T21:43:41.639734+0000","last_clean":"2024-03-11T21:43:41.639734+0000","last_became_active":"2024-03-11T21:31:37.387032+0000","last_became_peered":"2024-03-11T21:31:37.387032+0000","last_unstale":"2024-03-11T21:43:41.639734+0000","last_undegraded":"2024-03-11T21:43:41.639734+0000","last_fullsized":"2024-03-11T21:43:41.639734+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.462866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.462866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.089106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":1956,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635626+0000","last_change":"2024-03-11T21:41:40.335413+0000","last_active":"2024-03-11T21:43:41.635626+0000","last_peered":"2024-03-11T21:43:41.635626+0000","last_clean":"2024-03-11T21:43:41.635626+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:43:41.635626+0000","last_undegraded":"2024-03-11T21:43:41.635626+0000","last_fullsized":"2024-03-11T21:43:41.635626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:41:40.335178+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:27:12.664354+0000","last_clean_scrub_stamp":"2024-03-11T21:41:40.335178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"87'2","reported_seq":1600,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605181+0000","last_change":"2024-03-11T21:42:59.949425+0000","last_active":"2024-03-11T21:43:39.605181+0000","last_peered":"2024-03-11T21:43:39.605181+0000","last_clean":"2024-03-11T21:43:39.605181+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:43:39.605181+0000","last_undegraded":"2024-03-11T21:43:39.605181+0000","last_fullsized":"2024-03-11T21:43:39.605181+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T21:42:59.949164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.949164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:13.748601+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"864'40","reported_seq":2006,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637670+0000","last_change":"2024-03-11T21:42:56.554589+0000","last_active":"2024-03-11T21:43:41.637670+0000","last_peered":"2024-03-11T21:43:41.637670+0000","last_clean":"2024-03-11T21:43:41.637670+0000","last_became_active":"2024-03-11T21:23:05.019560+0000","last_became_peered":"2024-03-11T21:23:05.019560+0000","last_unstale":"2024-03-11T21:43:41.637670+0000","last_undegraded":"2024-03-11T21:43:41.637670+0000","last_fullsized":"2024-03-11T21:43:41.637670+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'28","last_scrub_stamp":"2024-03-11T21:42:56.554326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.554326+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.354495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1571,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605575+0000","last_change":"2024-03-11T21:43:14.059791+0000","last_active":"2024-03-11T21:43:39.605575+0000","last_peered":"2024-03-11T21:43:39.605575+0000","last_clean":"2024-03-11T21:43:39.605575+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:43:39.605575+0000","last_undegraded":"2024-03-11T21:43:39.605575+0000","last_fullsized":"2024-03-11T21:43:39.605575+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.059672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:14.059672+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.059672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.082658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1698,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634518+0000","last_change":"2024-03-11T21:43:20.299173+0000","last_active":"2024-03-11T21:43:41.634518+0000","last_peered":"2024-03-11T21:43:41.634518+0000","last_clean":"2024-03-11T21:43:41.634518+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:43:41.634518+0000","last_undegraded":"2024-03-11T21:43:41.634518+0000","last_fullsized":"2024-03-11T21:43:41.634518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:20.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.432022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1432,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605858+0000","last_change":"2024-03-11T21:42:54.061349+0000","last_active":"2024-03-11T21:43:39.605858+0000","last_peered":"2024-03-11T21:43:39.605858+0000","last_clean":"2024-03-11T21:43:39.605858+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:43:39.605858+0000","last_undegraded":"2024-03-11T21:43:39.605858+0000","last_fullsized":"2024-03-11T21:43:39.605858+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.061137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.061137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.609578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1740,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.587246+0000","last_change":"2024-03-11T21:43:14.527193+0000","last_active":"2024-03-11T21:43:41.587246+0000","last_peered":"2024-03-11T21:43:41.587246+0000","last_clean":"2024-03-11T21:43:41.587246+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:43:41.587246+0000","last_undegraded":"2024-03-11T21:43:41.587246+0000","last_fullsized":"2024-03-11T21:43:41.587246+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:14.526975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:02.944044+0000","last_clean_scrub_stamp":"2024-03-11T21:43:14.526975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.358244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"864'32","reported_seq":1752,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.643849+0000","last_change":"2024-03-11T21:43:22.461842+0000","last_active":"2024-03-11T21:43:41.643849+0000","last_peered":"2024-03-11T21:43:41.643849+0000","last_clean":"2024-03-11T21:43:41.643849+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:43:41.643849+0000","last_undegraded":"2024-03-11T21:43:41.643849+0000","last_fullsized":"2024-03-11T21:43:41.643849+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:43:22.461621+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.461621+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:46.244505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":1745,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.643373+0000","last_change":"2024-03-11T21:42:57.712087+0000","last_active":"2024-03-11T21:43:41.643373+0000","last_peered":"2024-03-11T21:43:41.643373+0000","last_clean":"2024-03-11T21:43:41.643373+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:43:41.643373+0000","last_undegraded":"2024-03-11T21:43:41.643373+0000","last_fullsized":"2024-03-11T21:43:41.643373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:42:57.711946+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:26:49.041020+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.711946+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.971914+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1688,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637687+0000","last_change":"2024-03-11T21:40:24.477381+0000","last_active":"2024-03-11T21:43:41.637687+0000","last_peered":"2024-03-11T21:43:41.637687+0000","last_clean":"2024-03-11T21:43:41.637687+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:43:41.637687+0000","last_undegraded":"2024-03-11T21:43:41.637687+0000","last_fullsized":"2024-03-11T21:43:41.637687+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:24.477246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:40:24.477246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1536,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637766+0000","last_change":"2024-03-11T21:41:15.373061+0000","last_active":"2024-03-11T21:43:41.637766+0000","last_peered":"2024-03-11T21:43:41.637766+0000","last_clean":"2024-03-11T21:43:41.637766+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:43:41.637766+0000","last_undegraded":"2024-03-11T21:43:41.637766+0000","last_fullsized":"2024-03-11T21:43:41.637766+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.372842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.372842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"864'27","reported_seq":1729,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635709+0000","last_change":"2024-03-11T21:42:59.435802+0000","last_active":"2024-03-11T21:43:41.635709+0000","last_peered":"2024-03-11T21:43:41.635709+0000","last_clean":"2024-03-11T21:43:41.635709+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:43:41.635709+0000","last_undegraded":"2024-03-11T21:43:41.635709+0000","last_fullsized":"2024-03-11T21:43:41.635709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:42:59.435557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.435557+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:08.758869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1747,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.587133+0000","last_change":"2024-03-11T21:43:15.569698+0000","last_active":"2024-03-11T21:43:41.587133+0000","last_peered":"2024-03-11T21:43:41.587133+0000","last_clean":"2024-03-11T21:43:41.587133+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:43:41.587133+0000","last_undegraded":"2024-03-11T21:43:41.587133+0000","last_fullsized":"2024-03-11T21:43:41.587133+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.569560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.569560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.367926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1691,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639411+0000","last_change":"2024-03-11T21:41:18.342623+0000","last_active":"2024-03-11T21:43:41.639411+0000","last_peered":"2024-03-11T21:43:41.639411+0000","last_clean":"2024-03-11T21:43:41.639411+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:43:41.639411+0000","last_undegraded":"2024-03-11T21:43:41.639411+0000","last_fullsized":"2024-03-11T21:43:41.639411+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:18.342525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.342525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1726,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590238+0000","last_change":"2024-03-11T21:42:52.491927+0000","last_active":"2024-03-11T21:43:41.590238+0000","last_peered":"2024-03-11T21:43:41.590238+0000","last_clean":"2024-03-11T21:43:41.590238+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:43:41.590238+0000","last_undegraded":"2024-03-11T21:43:41.590238+0000","last_fullsized":"2024-03-11T21:43:41.590238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:52.491711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.491711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:01.191914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1536,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637344+0000","last_change":"2024-03-11T21:41:37.401402+0000","last_active":"2024-03-11T21:43:41.637344+0000","last_peered":"2024-03-11T21:43:41.637344+0000","last_clean":"2024-03-11T21:43:41.637344+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:43:41.637344+0000","last_undegraded":"2024-03-11T21:43:41.637344+0000","last_fullsized":"2024-03-11T21:43:41.637344+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:37.401275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:37.401275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"864'14","reported_seq":1741,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588318+0000","last_change":"2024-03-11T21:41:49.695635+0000","last_active":"2024-03-11T21:43:41.588318+0000","last_peered":"2024-03-11T21:43:41.588318+0000","last_clean":"2024-03-11T21:43:41.588318+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:43:41.588318+0000","last_undegraded":"2024-03-11T21:43:41.588318+0000","last_fullsized":"2024-03-11T21:43:41.588318+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-03-11T21:41:49.695261+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.695261+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.530249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1726,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588466+0000","last_change":"2024-03-11T21:42:05.515337+0000","last_active":"2024-03-11T21:43:41.588466+0000","last_peered":"2024-03-11T21:43:41.588466+0000","last_clean":"2024-03-11T21:43:41.588466+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:43:41.588466+0000","last_undegraded":"2024-03-11T21:43:41.588466+0000","last_fullsized":"2024-03-11T21:43:41.588466+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:42:05.515038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.530336+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1696,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639515+0000","last_change":"2024-03-11T21:41:42.239297+0000","last_active":"2024-03-11T21:43:41.639515+0000","last_peered":"2024-03-11T21:43:41.639515+0000","last_clean":"2024-03-11T21:43:41.639515+0000","last_became_active":"2024-03-11T21:31:37.386739+0000","last_became_peered":"2024-03-11T21:31:37.386739+0000","last_unstale":"2024-03-11T21:43:41.639515+0000","last_undegraded":"2024-03-11T21:43:41.639515+0000","last_fullsized":"2024-03-11T21:43:41.639515+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:42.239053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.412884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1678,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642313+0000","last_change":"2024-03-11T21:42:58.421950+0000","last_active":"2024-03-11T21:43:41.642313+0000","last_peered":"2024-03-11T21:43:41.642313+0000","last_clean":"2024-03-11T21:43:41.642313+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:43:41.642313+0000","last_undegraded":"2024-03-11T21:43:41.642313+0000","last_fullsized":"2024-03-11T21:43:41.642313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.421800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.421800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:14.437625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636780+0000","last_change":"2024-03-11T21:42:54.754828+0000","last_active":"2024-03-11T21:43:41.636780+0000","last_peered":"2024-03-11T21:43:41.636780+0000","last_clean":"2024-03-11T21:43:41.636780+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:43:41.636780+0000","last_undegraded":"2024-03-11T21:43:41.636780+0000","last_fullsized":"2024-03-11T21:43:41.636780+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:54.754521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:54.754521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.619900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"864'27","reported_seq":1729,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.640285+0000","last_change":"2024-03-11T21:40:26.728613+0000","last_active":"2024-03-11T21:43:41.640285+0000","last_peered":"2024-03-11T21:43:41.640285+0000","last_clean":"2024-03-11T21:43:41.640285+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:43:41.640285+0000","last_undegraded":"2024-03-11T21:43:41.640285+0000","last_fullsized":"2024-03-11T21:43:41.640285+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:40:26.728512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.728512+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1711,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634638+0000","last_change":"2024-03-11T21:43:15.538577+0000","last_active":"2024-03-11T21:43:41.634638+0000","last_peered":"2024-03-11T21:43:41.634638+0000","last_clean":"2024-03-11T21:43:41.634638+0000","last_became_active":"2024-03-11T21:23:02.940161+0000","last_became_peered":"2024-03-11T21:23:02.940161+0000","last_unstale":"2024-03-11T21:43:41.634638+0000","last_undegraded":"2024-03-11T21:43:41.634638+0000","last_fullsized":"2024-03-11T21:43:41.634638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.538363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.538363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:22.713162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"297'13","reported_seq":1955,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.638006+0000","last_change":"2024-03-11T21:41:49.267753+0000","last_active":"2024-03-11T21:43:41.638006+0000","last_peered":"2024-03-11T21:43:41.638006+0000","last_clean":"2024-03-11T21:43:41.638006+0000","last_became_active":"2024-03-11T21:31:37.388174+0000","last_became_peered":"2024-03-11T21:31:37.388174+0000","last_unstale":"2024-03-11T21:43:41.638006+0000","last_undegraded":"2024-03-11T21:43:41.638006+0000","last_fullsized":"2024-03-11T21:43:41.638006+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'13","last_scrub_stamp":"2024-03-11T21:41:49.267522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.267522+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1697,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636125+0000","last_change":"2024-03-11T21:43:10.331236+0000","last_active":"2024-03-11T21:43:41.636125+0000","last_peered":"2024-03-11T21:43:41.636125+0000","last_clean":"2024-03-11T21:43:41.636125+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:43:41.636125+0000","last_undegraded":"2024-03-11T21:43:41.636125+0000","last_fullsized":"2024-03-11T21:43:41.636125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.331124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.331124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:11.391846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1570,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588905+0000","last_change":"2024-03-11T21:42:57.297254+0000","last_active":"2024-03-11T21:43:41.588905+0000","last_peered":"2024-03-11T21:43:41.588905+0000","last_clean":"2024-03-11T21:43:41.588905+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:43:41.588905+0000","last_undegraded":"2024-03-11T21:43:41.588905+0000","last_fullsized":"2024-03-11T21:43:41.588905+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:57.297086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:57.297086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.243930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"864'25","reported_seq":1746,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635434+0000","last_change":"2024-03-11T21:43:17.259134+0000","last_active":"2024-03-11T21:43:41.635434+0000","last_peered":"2024-03-11T21:43:41.635434+0000","last_clean":"2024-03-11T21:43:41.635434+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:43:41.635434+0000","last_undegraded":"2024-03-11T21:43:41.635434+0000","last_fullsized":"2024-03-11T21:43:41.635434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:43:17.258875+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:43:17.258875+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:24.144182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1709,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635417+0000","last_change":"2024-03-11T21:43:26.269090+0000","last_active":"2024-03-11T21:43:41.635417+0000","last_peered":"2024-03-11T21:43:41.635417+0000","last_clean":"2024-03-11T21:43:41.635417+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:43:41.635417+0000","last_undegraded":"2024-03-11T21:43:41.635417+0000","last_fullsized":"2024-03-11T21:43:41.635417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.268959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:54.342588+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.268959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.544045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":1949,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637217+0000","last_change":"2024-03-11T21:43:28.729150+0000","last_active":"2024-03-11T21:43:41.637217+0000","last_peered":"2024-03-11T21:43:41.637217+0000","last_clean":"2024-03-11T21:43:41.637217+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:43:41.637217+0000","last_undegraded":"2024-03-11T21:43:41.637217+0000","last_fullsized":"2024-03-11T21:43:41.637217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:43:28.728794+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:43:28.728794+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.728794+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:58.307267+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1714,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.643701+0000","last_change":"2024-03-11T21:41:57.724943+0000","last_active":"2024-03-11T21:43:41.643701+0000","last_peered":"2024-03-11T21:43:41.643701+0000","last_clean":"2024-03-11T21:43:41.643701+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:43:41.643701+0000","last_undegraded":"2024-03-11T21:43:41.643701+0000","last_fullsized":"2024-03-11T21:43:41.643701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:57.724694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:26.895410+0000","last_clean_scrub_stamp":"2024-03-11T21:41:57.724694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1420,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.607084+0000","last_change":"2024-03-11T21:41:55.220405+0000","last_active":"2024-03-11T21:43:39.607084+0000","last_peered":"2024-03-11T21:43:39.607084+0000","last_clean":"2024-03-11T21:43:39.607084+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:43:39.607084+0000","last_undegraded":"2024-03-11T21:43:39.607084+0000","last_fullsized":"2024-03-11T21:43:39.607084+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.220196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.220196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134966+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"864'25","reported_seq":1724,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642871+0000","last_change":"2024-03-11T21:41:53.406126+0000","last_active":"2024-03-11T21:43:41.642871+0000","last_peered":"2024-03-11T21:43:41.642871+0000","last_clean":"2024-03-11T21:43:41.642871+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:43:41.642871+0000","last_undegraded":"2024-03-11T21:43:41.642871+0000","last_fullsized":"2024-03-11T21:43:41.642871+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'19","last_scrub_stamp":"2024-03-11T21:41:53.405780+0000","last_deep_scrub":"100'13","last_deep_scrub_stamp":"2024-03-11T21:24:19.148248+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.405780+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1733,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589498+0000","last_change":"2024-03-11T21:43:15.412289+0000","last_active":"2024-03-11T21:43:41.589498+0000","last_peered":"2024-03-11T21:43:41.589498+0000","last_clean":"2024-03-11T21:43:41.589498+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:43:41.589498+0000","last_undegraded":"2024-03-11T21:43:41.589498+0000","last_fullsized":"2024-03-11T21:43:41.589498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_clean_scrub_stamp":"2024-03-11T21:43:15.412070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.526362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1691,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637596+0000","last_change":"2024-03-11T21:41:31.332398+0000","last_active":"2024-03-11T21:43:41.637596+0000","last_peered":"2024-03-11T21:43:41.637596+0000","last_clean":"2024-03-11T21:43:41.637596+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:43:41.637596+0000","last_undegraded":"2024-03-11T21:43:41.637596+0000","last_fullsized":"2024-03-11T21:43:41.637596+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.332299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.332299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1705,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634472+0000","last_change":"2024-03-11T21:43:22.269136+0000","last_active":"2024-03-11T21:43:41.634472+0000","last_peered":"2024-03-11T21:43:41.634472+0000","last_clean":"2024-03-11T21:43:41.634472+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:43:41.634472+0000","last_undegraded":"2024-03-11T21:43:41.634472+0000","last_fullsized":"2024-03-11T21:43:41.634472+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:22.269027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:22.269027+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.269027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.701754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1554,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635832+0000","last_change":"2024-03-11T21:42:58.427672+0000","last_active":"2024-03-11T21:43:41.635832+0000","last_peered":"2024-03-11T21:43:41.635832+0000","last_clean":"2024-03-11T21:43:41.635832+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:43:41.635832+0000","last_undegraded":"2024-03-11T21:43:41.635832+0000","last_fullsized":"2024-03-11T21:43:41.635832+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:58.427446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:42:58.427446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.731909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"864'30","reported_seq":1775,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636751+0000","last_change":"2024-03-11T21:43:06.683699+0000","last_active":"2024-03-11T21:43:41.636751+0000","last_peered":"2024-03-11T21:43:41.636751+0000","last_clean":"2024-03-11T21:43:41.636751+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:43:41.636751+0000","last_undegraded":"2024-03-11T21:43:41.636751+0000","last_fullsized":"2024-03-11T21:43:41.636751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-03-11T21:43:06.683455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.683455+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:22.624977+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1615,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605610+0000","last_change":"2024-03-11T21:43:27.076256+0000","last_active":"2024-03-11T21:43:39.605610+0000","last_peered":"2024-03-11T21:43:39.605610+0000","last_clean":"2024-03-11T21:43:39.605610+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:43:39.605610+0000","last_undegraded":"2024-03-11T21:43:39.605610+0000","last_fullsized":"2024-03-11T21:43:39.605610+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.357081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1583,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605007+0000","last_change":"2024-03-11T21:43:16.065946+0000","last_active":"2024-03-11T21:43:39.605007+0000","last_peered":"2024-03-11T21:43:39.605007+0000","last_clean":"2024-03-11T21:43:39.605007+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:43:39.605007+0000","last_undegraded":"2024-03-11T21:43:39.605007+0000","last_fullsized":"2024-03-11T21:43:39.605007+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.065817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:18.382717+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.065817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.278869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1678,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634387+0000","last_change":"2024-03-11T21:41:21.351275+0000","last_active":"2024-03-11T21:43:41.634387+0000","last_peered":"2024-03-11T21:43:41.634387+0000","last_clean":"2024-03-11T21:43:41.634387+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:43:41.634387+0000","last_undegraded":"2024-03-11T21:43:41.634387+0000","last_fullsized":"2024-03-11T21:43:41.634387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.350961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.215038+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.350961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790536+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1544,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635301+0000","last_change":"2024-03-11T21:41:48.432501+0000","last_active":"2024-03-11T21:43:41.635301+0000","last_peered":"2024-03-11T21:43:41.635301+0000","last_clean":"2024-03-11T21:43:41.635301+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:43:41.635301+0000","last_undegraded":"2024-03-11T21:43:41.635301+0000","last_fullsized":"2024-03-11T21:43:41.635301+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:48.432274+0000","last_clean_scrub_stamp":"2024-03-11T21:41:48.432274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"864'18","reported_seq":1732,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636469+0000","last_change":"2024-03-11T21:42:02.547377+0000","last_active":"2024-03-11T21:43:41.636469+0000","last_peered":"2024-03-11T21:43:41.636469+0000","last_clean":"2024-03-11T21:43:41.636469+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:43:41.636469+0000","last_undegraded":"2024-03-11T21:43:41.636469+0000","last_fullsized":"2024-03-11T21:43:41.636469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'12","last_scrub_stamp":"2024-03-11T21:42:02.547005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.547005+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1703,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.640561+0000","last_change":"2024-03-11T21:43:13.521572+0000","last_active":"2024-03-11T21:43:41.640561+0000","last_peered":"2024-03-11T21:43:41.640561+0000","last_clean":"2024-03-11T21:43:41.640561+0000","last_became_active":"2024-03-11T21:23:02.928397+0000","last_became_peered":"2024-03-11T21:23:02.928397+0000","last_unstale":"2024-03-11T21:43:41.640561+0000","last_undegraded":"2024-03-11T21:43:41.640561+0000","last_fullsized":"2024-03-11T21:43:41.640561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:13.521476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:13.521476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.205030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1717,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.588406+0000","last_change":"2024-03-11T21:42:02.517280+0000","last_active":"2024-03-11T21:43:41.588406+0000","last_peered":"2024-03-11T21:43:41.588406+0000","last_clean":"2024-03-11T21:43:41.588406+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:43:41.588406+0000","last_undegraded":"2024-03-11T21:43:41.588406+0000","last_fullsized":"2024-03-11T21:43:41.588406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:02.516952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:29.972683+0000","last_clean_scrub_stamp":"2024-03-11T21:42:02.516952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.530292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1678,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635843+0000","last_change":"2024-03-11T21:43:07.660322+0000","last_active":"2024-03-11T21:43:41.635843+0000","last_peered":"2024-03-11T21:43:41.635843+0000","last_clean":"2024-03-11T21:43:41.635843+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:43:41.635843+0000","last_undegraded":"2024-03-11T21:43:41.635843+0000","last_fullsized":"2024-03-11T21:43:41.635843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.660094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.660094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:19.424741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1555,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635426+0000","last_change":"2024-03-11T21:43:16.248764+0000","last_active":"2024-03-11T21:43:41.635426+0000","last_peered":"2024-03-11T21:43:41.635426+0000","last_clean":"2024-03-11T21:43:41.635426+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:43:41.635426+0000","last_undegraded":"2024-03-11T21:43:41.635426+0000","last_fullsized":"2024-03-11T21:43:41.635426+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:16.248539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.248539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.321015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"864'21","reported_seq":1742,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639225+0000","last_change":"2024-03-11T21:43:09.463949+0000","last_active":"2024-03-11T21:43:41.639225+0000","last_peered":"2024-03-11T21:43:41.639225+0000","last_clean":"2024-03-11T21:43:41.639225+0000","last_became_active":"2024-03-11T21:31:37.387283+0000","last_became_peered":"2024-03-11T21:31:37.387283+0000","last_unstale":"2024-03-11T21:43:41.639225+0000","last_undegraded":"2024-03-11T21:43:41.639225+0000","last_fullsized":"2024-03-11T21:43:41.639225+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:43:09.463706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:09.463706+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.459582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1733,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589972+0000","last_change":"2024-03-11T21:43:28.335119+0000","last_active":"2024-03-11T21:43:41.589972+0000","last_peered":"2024-03-11T21:43:41.589972+0000","last_clean":"2024-03-11T21:43:41.589972+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:43:41.589972+0000","last_undegraded":"2024-03-11T21:43:41.589972+0000","last_fullsized":"2024-03-11T21:43:41.589972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:28.335020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:41.700576+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.335020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:55.290802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1674,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632736+0000","last_change":"2024-03-11T21:40:16.866813+0000","last_active":"2024-03-11T21:43:41.632736+0000","last_peered":"2024-03-11T21:43:41.632736+0000","last_clean":"2024-03-11T21:43:41.632736+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:43:41.632736+0000","last_undegraded":"2024-03-11T21:43:41.632736+0000","last_fullsized":"2024-03-11T21:43:41.632736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:16.866545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:16.866545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1678,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634426+0000","last_change":"2024-03-11T21:41:53.585801+0000","last_active":"2024-03-11T21:43:41.634426+0000","last_peered":"2024-03-11T21:43:41.634426+0000","last_clean":"2024-03-11T21:43:41.634426+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:43:41.634426+0000","last_undegraded":"2024-03-11T21:43:41.634426+0000","last_fullsized":"2024-03-11T21:43:41.634426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.585657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1536,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639290+0000","last_change":"2024-03-11T21:43:23.412537+0000","last_active":"2024-03-11T21:43:41.639290+0000","last_peered":"2024-03-11T21:43:41.639290+0000","last_clean":"2024-03-11T21:43:41.639290+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:43:41.639290+0000","last_undegraded":"2024-03-11T21:43:41.639290+0000","last_fullsized":"2024-03-11T21:43:41.639290+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:23.412309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.412309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.639277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"864'46","reported_seq":1686,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604337+0000","last_change":"2024-03-11T21:43:04.876480+0000","last_active":"2024-03-11T21:43:39.604337+0000","last_peered":"2024-03-11T21:43:39.604337+0000","last_clean":"2024-03-11T21:43:39.604337+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:43:39.604337+0000","last_undegraded":"2024-03-11T21:43:39.604337+0000","last_fullsized":"2024-03-11T21:43:39.604337+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2024-03-11T21:43:04.876381+0000","last_deep_scrub":"538'34","last_deep_scrub_stamp":"2024-03-11T21:36:39.151333+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.876381+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:12.934737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1709,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.631805+0000","last_change":"2024-03-11T21:43:02.494327+0000","last_active":"2024-03-11T21:43:41.631805+0000","last_peered":"2024-03-11T21:43:41.631805+0000","last_clean":"2024-03-11T21:43:41.631805+0000","last_became_active":"2024-03-11T21:24:58.849342+0000","last_became_peered":"2024-03-11T21:24:58.849342+0000","last_unstale":"2024-03-11T21:43:41.631805+0000","last_undegraded":"2024-03-11T21:43:41.631805+0000","last_fullsized":"2024-03-11T21:43:41.631805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:02.494116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:02.494116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:16.677295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"84'11","reported_seq":1987,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.590134+0000","last_change":"2024-03-11T21:43:16.367046+0000","last_active":"2024-03-11T21:43:41.590134+0000","last_peered":"2024-03-11T21:43:41.590134+0000","last_clean":"2024-03-11T21:43:41.590134+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:43:41.590134+0000","last_undegraded":"2024-03-11T21:43:41.590134+0000","last_fullsized":"2024-03-11T21:43:41.590134+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:43:16.366877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:16.366877+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:45.000288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"84'1","reported_seq":1680,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642082+0000","last_change":"2024-03-11T21:43:24.429298+0000","last_active":"2024-03-11T21:43:41.642082+0000","last_peered":"2024-03-11T21:43:41.642082+0000","last_clean":"2024-03-11T21:43:41.642082+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:43:41.642082+0000","last_undegraded":"2024-03-11T21:43:41.642082+0000","last_fullsized":"2024-03-11T21:43:41.642082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T21:43:24.429154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.429154+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.701671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1550,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634100+0000","last_change":"2024-03-11T21:41:56.288055+0000","last_active":"2024-03-11T21:43:41.634100+0000","last_peered":"2024-03-11T21:43:41.634100+0000","last_clean":"2024-03-11T21:43:41.634100+0000","last_became_active":"2024-03-11T21:31:37.386933+0000","last_became_peered":"2024-03-11T21:31:37.386933+0000","last_unstale":"2024-03-11T21:43:41.634100+0000","last_undegraded":"2024-03-11T21:43:41.634100+0000","last_fullsized":"2024-03-11T21:43:41.634100+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:56.287964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:56.287964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.412931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"864'21","reported_seq":1737,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635716+0000","last_change":"2024-03-11T21:42:52.850471+0000","last_active":"2024-03-11T21:43:41.635716+0000","last_peered":"2024-03-11T21:43:41.635716+0000","last_clean":"2024-03-11T21:43:41.635716+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:43:41.635716+0000","last_undegraded":"2024-03-11T21:43:41.635716+0000","last_fullsized":"2024-03-11T21:43:41.635716+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15","last_scrub_stamp":"2024-03-11T21:42:52.850333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:42:52.850333+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:05.477895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1699,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636449+0000","last_change":"2024-03-11T21:43:06.774760+0000","last_active":"2024-03-11T21:43:41.636449+0000","last_peered":"2024-03-11T21:43:41.636449+0000","last_clean":"2024-03-11T21:43:41.636449+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:43:41.636449+0000","last_undegraded":"2024-03-11T21:43:41.636449+0000","last_fullsized":"2024-03-11T21:43:41.636449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:43:06.774478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:06.774478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:33.859854+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1695,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636093+0000","last_change":"2024-03-11T21:41:41.375255+0000","last_active":"2024-03-11T21:43:41.636093+0000","last_peered":"2024-03-11T21:43:41.636093+0000","last_clean":"2024-03-11T21:43:41.636093+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:43:41.636093+0000","last_undegraded":"2024-03-11T21:43:41.636093+0000","last_fullsized":"2024-03-11T21:43:41.636093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:41:41.375162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1565,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.605349+0000","last_change":"2024-03-11T21:41:49.263645+0000","last_active":"2024-03-11T21:43:39.605349+0000","last_peered":"2024-03-11T21:43:39.605349+0000","last_clean":"2024-03-11T21:43:39.605349+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:43:39.605349+0000","last_undegraded":"2024-03-11T21:43:39.605349+0000","last_fullsized":"2024-03-11T21:43:39.605349+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:49.263416+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.263416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1528,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635426+0000","last_change":"2024-03-11T21:41:23.491021+0000","last_active":"2024-03-11T21:43:41.635426+0000","last_peered":"2024-03-11T21:43:41.635426+0000","last_clean":"2024-03-11T21:43:41.635426+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:43:41.635426+0000","last_undegraded":"2024-03-11T21:43:41.635426+0000","last_fullsized":"2024-03-11T21:43:41.635426+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.490850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.490850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"864'11","reported_seq":1715,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635498+0000","last_change":"2024-03-11T21:43:04.387983+0000","last_active":"2024-03-11T21:43:41.635498+0000","last_peered":"2024-03-11T21:43:41.635498+0000","last_clean":"2024-03-11T21:43:41.635498+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:43:41.635498+0000","last_undegraded":"2024-03-11T21:43:41.635498+0000","last_fullsized":"2024-03-11T21:43:41.635498+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'9","last_scrub_stamp":"2024-03-11T21:43:04.387720+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:43:04.387720+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:07.836763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1725,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.589048+0000","last_change":"2024-03-11T21:41:53.683571+0000","last_active":"2024-03-11T21:43:41.589048+0000","last_peered":"2024-03-11T21:43:41.589048+0000","last_clean":"2024-03-11T21:43:41.589048+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:43:41.589048+0000","last_undegraded":"2024-03-11T21:43:41.589048+0000","last_fullsized":"2024-03-11T21:43:41.589048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.683465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.683465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:46.530189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1567,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.606111+0000","last_change":"2024-03-11T21:43:26.028281+0000","last_active":"2024-03-11T21:43:39.606111+0000","last_peered":"2024-03-11T21:43:39.606111+0000","last_clean":"2024-03-11T21:43:39.606111+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:43:39.606111+0000","last_undegraded":"2024-03-11T21:43:39.606111+0000","last_fullsized":"2024-03-11T21:43:39.606111+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.028192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:21.433272+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.028192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:44.924494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1708,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642894+0000","last_change":"2024-03-11T21:41:19.561625+0000","last_active":"2024-03-11T21:43:41.642894+0000","last_peered":"2024-03-11T21:43:41.642894+0000","last_clean":"2024-03-11T21:43:41.642894+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:43:41.642894+0000","last_undegraded":"2024-03-11T21:43:41.642894+0000","last_fullsized":"2024-03-11T21:43:41.642894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.561348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:59.988522+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.561348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1562,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632497+0000","last_change":"2024-03-11T21:41:55.738509+0000","last_active":"2024-03-11T21:43:41.632497+0000","last_peered":"2024-03-11T21:43:41.632497+0000","last_clean":"2024-03-11T21:43:41.632497+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:43:41.632497+0000","last_undegraded":"2024-03-11T21:43:41.632497+0000","last_fullsized":"2024-03-11T21:43:41.632497+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:55.738282+0000","last_clean_scrub_stamp":"2024-03-11T21:41:55.738282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007572+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"864'25","reported_seq":1769,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634224+0000","last_change":"2024-03-11T21:42:51.740712+0000","last_active":"2024-03-11T21:43:41.634224+0000","last_peered":"2024-03-11T21:43:41.634224+0000","last_clean":"2024-03-11T21:43:41.634224+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:43:41.634224+0000","last_undegraded":"2024-03-11T21:43:41.634224+0000","last_fullsized":"2024-03-11T21:43:41.634224+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'19","last_scrub_stamp":"2024-03-11T21:42:51.740477+0000","last_deep_scrub":"714'19","last_deep_scrub_stamp":"2024-03-11T21:40:13.889354+0000","last_clean_scrub_stamp":"2024-03-11T21:42:51.740477+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:18.872799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1698,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637427+0000","last_change":"2024-03-11T21:43:07.788606+0000","last_active":"2024-03-11T21:43:41.637427+0000","last_peered":"2024-03-11T21:43:41.637427+0000","last_clean":"2024-03-11T21:43:41.637427+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:43:41.637427+0000","last_undegraded":"2024-03-11T21:43:41.637427+0000","last_fullsized":"2024-03-11T21:43:41.637427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:07.788495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:43:07.788495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:15.174149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1710,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636821+0000","last_change":"2024-03-11T21:41:54.741470+0000","last_active":"2024-03-11T21:43:41.636821+0000","last_peered":"2024-03-11T21:43:41.636821+0000","last_clean":"2024-03-11T21:43:41.636821+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:43:41.636821+0000","last_undegraded":"2024-03-11T21:43:41.636821+0000","last_fullsized":"2024-03-11T21:43:41.636821+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:54.741251+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.741251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.007596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1679,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639317+0000","last_change":"2024-03-11T21:41:54.630605+0000","last_active":"2024-03-11T21:43:41.639317+0000","last_peered":"2024-03-11T21:43:41.639317+0000","last_clean":"2024-03-11T21:43:41.639317+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:43:41.639317+0000","last_undegraded":"2024-03-11T21:43:41.639317+0000","last_fullsized":"2024-03-11T21:43:41.639317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.630386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.630386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1537,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.644474+0000","last_change":"2024-03-11T21:41:31.510028+0000","last_active":"2024-03-11T21:43:41.644474+0000","last_peered":"2024-03-11T21:43:41.644474+0000","last_clean":"2024-03-11T21:43:41.644474+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:43:41.644474+0000","last_undegraded":"2024-03-11T21:43:41.644474+0000","last_fullsized":"2024-03-11T21:43:41.644474+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.509821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.509821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"864'24","reported_seq":1740,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637818+0000","last_change":"2024-03-11T21:43:20.541592+0000","last_active":"2024-03-11T21:43:41.637818+0000","last_peered":"2024-03-11T21:43:41.637818+0000","last_clean":"2024-03-11T21:43:41.637818+0000","last_became_active":"2024-03-11T21:23:04.935991+0000","last_became_peered":"2024-03-11T21:23:04.935991+0000","last_unstale":"2024-03-11T21:43:41.637818+0000","last_undegraded":"2024-03-11T21:43:41.637818+0000","last_fullsized":"2024-03-11T21:43:41.637818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'24","last_scrub_stamp":"2024-03-11T21:43:20.541460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.541460+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:23.402842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1690,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.641542+0000","last_change":"2024-03-11T21:41:46.401421+0000","last_active":"2024-03-11T21:43:41.641542+0000","last_peered":"2024-03-11T21:43:41.641542+0000","last_clean":"2024-03-11T21:43:41.641542+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:43:41.641542+0000","last_undegraded":"2024-03-11T21:43:41.641542+0000","last_fullsized":"2024-03-11T21:43:41.641542+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:46.401211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:41:46.401211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1687,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635601+0000","last_change":"2024-03-11T21:41:53.275977+0000","last_active":"2024-03-11T21:43:41.635601+0000","last_peered":"2024-03-11T21:43:41.635601+0000","last_clean":"2024-03-11T21:43:41.635601+0000","last_became_active":"2024-03-11T21:23:07.083628+0000","last_became_peered":"2024-03-11T21:23:07.083628+0000","last_unstale":"2024-03-11T21:43:41.635601+0000","last_undegraded":"2024-03-11T21:43:41.635601+0000","last_fullsized":"2024-03-11T21:43:41.635601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:53.275834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:41:53.275834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1705,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634589+0000","last_change":"2024-03-11T21:42:59.415876+0000","last_active":"2024-03-11T21:43:41.634589+0000","last_peered":"2024-03-11T21:43:41.634589+0000","last_clean":"2024-03-11T21:43:41.634589+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:43:41.634589+0000","last_undegraded":"2024-03-11T21:43:41.634589+0000","last_fullsized":"2024-03-11T21:43:41.634589+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:59.415650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:59.415650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:27.486892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1573,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639680+0000","last_change":"2024-03-11T21:43:24.762700+0000","last_active":"2024-03-11T21:43:41.639680+0000","last_peered":"2024-03-11T21:43:41.639680+0000","last_clean":"2024-03-11T21:43:41.639680+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:43:41.639680+0000","last_undegraded":"2024-03-11T21:43:41.639680+0000","last_fullsized":"2024-03-11T21:43:41.639680+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.762559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.762559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.495720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"864'46","reported_seq":1776,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639166+0000","last_change":"2024-03-11T21:41:51.290471+0000","last_active":"2024-03-11T21:43:41.639166+0000","last_peered":"2024-03-11T21:43:41.639166+0000","last_clean":"2024-03-11T21:43:41.639166+0000","last_became_active":"2024-03-11T21:31:37.387243+0000","last_became_peered":"2024-03-11T21:31:37.387243+0000","last_unstale":"2024-03-11T21:43:41.639166+0000","last_undegraded":"2024-03-11T21:43:41.639166+0000","last_fullsized":"2024-03-11T21:43:41.639166+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'34","last_scrub_stamp":"2024-03-11T21:41:51.290324+0000","last_deep_scrub":"120'22","last_deep_scrub_stamp":"2024-03-11T21:26:31.763327+0000","last_clean_scrub_stamp":"2024-03-11T21:41:51.290324+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.412981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1701,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642732+0000","last_change":"2024-03-11T21:43:29.254214+0000","last_active":"2024-03-11T21:43:41.642732+0000","last_peered":"2024-03-11T21:43:41.642732+0000","last_clean":"2024-03-11T21:43:41.642732+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:43:41.642732+0000","last_undegraded":"2024-03-11T21:43:41.642732+0000","last_fullsized":"2024-03-11T21:43:41.642732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:29.253887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.253887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:44.093330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1683,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.635496+0000","last_change":"2024-03-11T21:40:04.809344+0000","last_active":"2024-03-11T21:43:41.635496+0000","last_peered":"2024-03-11T21:43:41.635496+0000","last_clean":"2024-03-11T21:43:41.635496+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:43:41.635496+0000","last_undegraded":"2024-03-11T21:43:41.635496+0000","last_fullsized":"2024-03-11T21:43:41.635496+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:04.809056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:40:04.809056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1692,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.633431+0000","last_change":"2024-03-11T21:41:47.226377+0000","last_active":"2024-03-11T21:43:41.633431+0000","last_peered":"2024-03-11T21:43:41.633431+0000","last_clean":"2024-03-11T21:43:41.633431+0000","last_became_active":"2024-03-11T21:31:37.386729+0000","last_became_peered":"2024-03-11T21:31:37.386729+0000","last_unstale":"2024-03-11T21:43:41.633431+0000","last_undegraded":"2024-03-11T21:43:41.633431+0000","last_fullsized":"2024-03-11T21:43:41.633431+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:47.226166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:41:47.226166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.413069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1545,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.642097+0000","last_change":"2024-03-11T21:43:34.329646+0000","last_active":"2024-03-11T21:43:41.642097+0000","last_peered":"2024-03-11T21:43:41.642097+0000","last_clean":"2024-03-11T21:43:41.642097+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:43:41.642097+0000","last_undegraded":"2024-03-11T21:43:41.642097+0000","last_fullsized":"2024-03-11T21:43:41.642097+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:34.329432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:43:34.329432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:43.983740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"864'32","reported_seq":1754,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636704+0000","last_change":"2024-03-11T21:42:55.509855+0000","last_active":"2024-03-11T21:43:41.636704+0000","last_peered":"2024-03-11T21:43:41.636704+0000","last_clean":"2024-03-11T21:43:41.636704+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:43:41.636704+0000","last_undegraded":"2024-03-11T21:43:41.636704+0000","last_fullsized":"2024-03-11T21:43:41.636704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'24","last_scrub_stamp":"2024-03-11T21:42:55.509532+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-03-11T21:26:51.608221+0000","last_clean_scrub_stamp":"2024-03-11T21:42:55.509532+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:58.493575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1692,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639081+0000","last_change":"2024-03-11T21:41:49.603979+0000","last_active":"2024-03-11T21:43:41.639081+0000","last_peered":"2024-03-11T21:43:41.639081+0000","last_clean":"2024-03-11T21:43:41.639081+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:43:41.639081+0000","last_undegraded":"2024-03-11T21:43:41.639081+0000","last_fullsized":"2024-03-11T21:43:41.639081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:49.603815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:41:49.603815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.790713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1564,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.604548+0000","last_change":"2024-03-11T21:43:10.968913+0000","last_active":"2024-03-11T21:43:39.604548+0000","last_peered":"2024-03-11T21:43:39.604548+0000","last_clean":"2024-03-11T21:43:39.604548+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:43:39.604548+0000","last_undegraded":"2024-03-11T21:43:39.604548+0000","last_fullsized":"2024-03-11T21:43:39.604548+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:10.968603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:20.395704+0000","last_clean_scrub_stamp":"2024-03-11T21:43:10.968603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.483700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1723,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.634063+0000","last_change":"2024-03-11T21:43:12.640536+0000","last_active":"2024-03-11T21:43:41.634063+0000","last_peered":"2024-03-11T21:43:41.634063+0000","last_clean":"2024-03-11T21:43:41.634063+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:43:41.634063+0000","last_undegraded":"2024-03-11T21:43:41.634063+0000","last_fullsized":"2024-03-11T21:43:41.634063+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:12.639805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:43:12.639805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:37.329943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1414,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-11T21:43:39.607626+0000","last_change":"2024-03-11T21:41:54.179283+0000","last_active":"2024-03-11T21:43:39.607626+0000","last_peered":"2024-03-11T21:43:39.607626+0000","last_clean":"2024-03-11T21:43:39.607626+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:43:39.607626+0000","last_undegraded":"2024-03-11T21:43:39.607626+0000","last_fullsized":"2024-03-11T21:43:39.607626+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:54.179012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:41:54.179012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:41.134835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"864'30","reported_seq":1741,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.639197+0000","last_change":"2024-03-11T21:43:30.682139+0000","last_active":"2024-03-11T21:43:41.639197+0000","last_peered":"2024-03-11T21:43:41.639197+0000","last_clean":"2024-03-11T21:43:41.639197+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:43:41.639197+0000","last_undegraded":"2024-03-11T21:43:41.639197+0000","last_fullsized":"2024-03-11T21:43:41.639197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:43:30.682020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.682020+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:47.002828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1711,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.636816+0000","last_change":"2024-03-11T21:43:08.340131+0000","last_active":"2024-03-11T21:43:41.636816+0000","last_peered":"2024-03-11T21:43:41.636816+0000","last_clean":"2024-03-11T21:43:41.636816+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:43:41.636816+0000","last_undegraded":"2024-03-11T21:43:41.636816+0000","last_fullsized":"2024-03-11T21:43:41.636816+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:43:08.339938+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.339938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.620284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1727,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.632637+0000","last_change":"2024-03-11T21:43:08.620663+0000","last_active":"2024-03-11T21:43:41.632637+0000","last_peered":"2024-03-11T21:43:41.632637+0000","last_clean":"2024-03-11T21:43:41.632637+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:43:41.632637+0000","last_undegraded":"2024-03-11T21:43:41.632637+0000","last_fullsized":"2024-03-11T21:43:41.632637+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:08.620444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:43:08.620444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.349008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1678,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-11T21:43:41.637263+0000","last_change":"2024-03-11T21:42:56.738473+0000","last_active":"2024-03-11T21:43:41.637263+0000","last_peered":"2024-03-11T21:43:41.637263+0000","last_clean":"2024-03-11T21:43:41.637263+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:43:41.637263+0000","last_undegraded":"2024-03-11T21:43:41.637263+0000","last_fullsized":"2024-03-11T21:43:41.637263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:42:56.738379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:42:56.738379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:10.984522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1558,"num_read_kb":1315,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1218,"num_read_kb":1213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841795,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119536,"kb_used_data":11264,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93625616,"statfs":{"total":95995035648,"available":95872630784,"internally_reserved":0,"allocated":11534336,"data_stored":6872507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168844,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105048,"kb_used_data":10912,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93640104,"statfs":{"total":95995035648,"available":95887466496,"internally_reserved":0,"allocated":11173888,"data_stored":6542607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352466,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57828,"kb_used_data":11372,"kb_used_omap":27,"kb_used_meta":45924,"kb_avail":93687324,"statfs":{"total":95995035648,"available":95935819776,"internally_reserved":0,"allocated":11644928,"data_stored":6897180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":47026505},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101144,"kb_used_data":10920,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93644008,"statfs":{"total":95995035648,"available":95891464192,"internally_reserved":0,"allocated":11182080,"data_stored":6543719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378934,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115036,"kb_used_data":11172,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93630116,"statfs":{"total":95995035648,"available":95877238784,"internally_reserved":0,"allocated":11440128,"data_stored":6805750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117290,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97296,"kb_used_data":11728,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93647856,"statfs":{"total":95995035648,"available":95895404544,"internally_reserved":0,"allocated":12009472,"data_stored":7325531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855648,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119216,"kb_used_data":10944,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93625936,"statfs":{"total":95995035648,"available":95872958464,"internally_reserved":0,"allocated":11206656,"data_stored":6544009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495894,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75092,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670060,"statfs":{"total":95995035648,"available":95918141440,"internally_reserved":0,"allocated":11759616,"data_stored":7132890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:43:45.426 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:43:45.426 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-11T21:43:45.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:45.453 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:45.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:45.577 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:45.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:45.708 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:45.811 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:45.836 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:45.887 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:45.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:45.968 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:46.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.099 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:46.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:46.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:46.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:46.359 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:46.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:46.490 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:46.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:46.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:46.788 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:46.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.920 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:46.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:46 smithi143 bash[21512]: cluster 2024-03-11T21:43:45.659551+0000 mgr.y (mgr.14550) 805 : cluster 0 pgmap v1392: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-11T21:43:46.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:46 smithi143 bash[29284]: cluster 2024-03-11T21:43:45.659551+0000 mgr.y (mgr.14550) 805 : cluster 0 pgmap v1392: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-11T21:43:47.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:46 smithi203 bash[23654]: cluster 2024-03-11T21:43:45.659551+0000 mgr.y (mgr.14550) 805 : cluster 0 pgmap v1392: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-11T21:43:47.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:47.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:47.181 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:47.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.313 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:47.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:47.442 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:47.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:47.582 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:47.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.703 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:47.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/685511046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80104c30 msgr2=0x7f3c80108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- 172.21.15.143:0/685511046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80104c30 0x7f3c80108c80 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f3c680099c0 tx=0x7f3c68009cd0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/685511046 shutdown_connections 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- 172.21.15.143:0/685511046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104280 0x7f3c80104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- 172.21.15.143:0/685511046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c80109360 0x7f3c80110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- 172.21.15.143:0/685511046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80104c30 0x7f3c80108c80 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/685511046 >> 172.21.15.143:0/685511046 conn(0x7f3c800ffb10 msgr2=0x7f3c80101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/685511046 shutdown_connections 2024-03-11T21:43:47.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/685511046 wait complete. 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 librados: starting msgr at 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 librados: starting objecter 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 Processor -- start 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- start start 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 librados: setting wanted keys 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 librados: calling monclient init 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c80104280 0x7f3c80199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104c30 0x7f3c8019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80113850 con 0x7f3c80104c30 2024-03-11T21:43:47.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801136d0 con 0x7f3c80104280 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801139d0 con 0x7f3c80109360 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c767fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104c30 0x7f3c8019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51218/0 (socket says 172.21.15.143:51218) 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 learned_addr learned my addr 172.21.15.143:0/2727032799 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c80104280 0x7f3c80199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c80104280 msgr2=0x7f3c80199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c80104280 0x7f3c80199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104c30 msgr2=0x7f3c8019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104c30 0x7f3c8019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8019ede0 con 0x7f3c80109360 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c80104c30 0x7f3c8019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:47.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800ef30 tx=0x7f3c7800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7cff9700 10 client.?.objecter ms_handle_connect 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7cff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c780092e0 con 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3c78004bb0 con 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78004020 con 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8019f070 con 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 10 client.42137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c801a6950 con 0x7f3c80109360 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.825+0000 7f3c852f4700 1 librados: init done 2024-03-11T21:43:47.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3c8004faa0 con 0x7f3c80109360 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3c7801b030 con 0x7f3c80109360 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3c6c06f820 0x7f3c6c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(892..892 src has 842..892) v4 ==== 9021+0+0 (secure 0 0 0) 0x7f3c7808ec50 con 0x7f3c80109360 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 osd_map(892..892 src has 842..892) v4 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 3 client.42137.objecter handle_osd_map got epochs [892,892] > 0 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 3 client.42137.objecter handle_osd_map decoding full epoch 892 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3c6c06f820 0x7f3c6c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 20 client.42137.objecter dump_active .. 0 homeless 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v892) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3c78022020 con 0x7f3c80109360 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 10 client.42137.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 1 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3c800629f0 0x7f3c80062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 1 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 0 1 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 4 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 10 client.42137.objecter _send_command 1 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801a9d70 con 0x7f3c800629f0 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c6c06f820 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3c6c06f820 0x7f3c6c071ce0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009680 tx=0x7f3c70009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3c800629f0 0x7f3c80062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3c800629f0 0x7f3c80062dd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c800629f0 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c801a9d70 con 0x7f3c800629f0 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 1 = system:0 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 4 2024-03-11T21:43:47.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.829+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 0 1 2024-03-11T21:43:47.828 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:47.836 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 10 client.42137.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.836 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 2 2024-03-11T21:43:47.836 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 2 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8019b130 osd=0 3 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 0 2 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 4 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 10 client.42137.objecter _send_command 2 2024-03-11T21:43:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.849+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c80062470 con 0x7f3c800629f0 2024-03-11T21:43:47.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c80062470 con 0x7f3c800629f0 2024-03-11T21:43:47.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 4 2024-03-11T21:43:47.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 0 2 2024-03-11T21:43:47.839 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:43:47.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 10 client.42137.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 3 2024-03-11T21:43:47.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3c8002c6b0 0x7f3c8002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 1 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 4 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 10 client.42137.objecter _send_command 3 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c8010bcd0 con 0x7f3c8002c6b0 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3c8002c6b0 0x7f3c8002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c8002c6b0 2024-03-11T21:43:47.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3c8002c6b0 0x7f3c8002ca90 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c8010bcd0 con 0x7f3c8002c6b0 2024-03-11T21:43:47.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:43:47.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 3 = system:0 2024-03-11T21:43:47.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 4 2024-03-11T21:43:47.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.853+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 1 3 2024-03-11T21:43:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 10 client.42137.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 4 2024-03-11T21:43:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 4 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8010b650 osd=1 3 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 1 4 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 4 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 10 client.42137.objecter _send_command 4 2024-03-11T21:43:47.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.869+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c8010b0d0 con 0x7f3c8002c6b0 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c8010b0d0 con 0x7f3c8002c6b0 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 4 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 1 4 2024-03-11T21:43:47.859 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:43:47.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 10 client.42137.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 5 2024-03-11T21:43:47.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3c801abf30 0x7f3c801ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 5 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 2 5 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 4 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 10 client.42137.objecter _send_command 5 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c8002dcb0 con 0x7f3c801abf30 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3c801abf30 0x7f3c801ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801abf30 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3c801abf30 0x7f3c801ae320 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c8002dcb0 con 0x7f3c801abf30 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:43:47.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 5 = system:0 2024-03-11T21:43:47.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 4 2024-03-11T21:43:47.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.873+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 2 5 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 10 client.42137.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 6 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 6 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8010b4d0 osd=2 3 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 2 6 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 4 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 10 client.42137.objecter _send_command 6 2024-03-11T21:43:47.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.889+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c8002d220 con 0x7f3c801abf30 2024-03-11T21:43:47.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c8002d220 con 0x7f3c801abf30 2024-03-11T21:43:47.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 4 2024-03-11T21:43:47.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 2 6 2024-03-11T21:43:47.880 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:43:47.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 10 client.42137.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 7 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3c801a98d0 0x7f3c801b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 7 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 3 7 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 4 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 10 client.42137.objecter _send_command 7 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801b4980 con 0x7f3c801a98d0 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3c801a98d0 0x7f3c801b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801a98d0 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3c801a98d0 0x7f3c801b19f0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c801b4980 con 0x7f3c801a98d0 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 7 = system:0 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 4 2024-03-11T21:43:47.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.893+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 3 7 2024-03-11T21:43:47.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 10 client.42137.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 8 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 8 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c8002d620 osd=3 3 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 3 8 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 4 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 10 client.42137.objecter _send_command 8 2024-03-11T21:43:47.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c801a8e40 con 0x7f3c801a98d0 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c801a8e40 con 0x7f3c801a98d0 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 4 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 3 8 2024-03-11T21:43:47.901 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 10 client.42137.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 9 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3c801af540 0x7f3c801b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 9 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 4 9 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 4 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 10 client.42137.objecter _send_command 9 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801a8db0 con 0x7f3c801af540 2024-03-11T21:43:47.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.913+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3c801af540 0x7f3c801b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801af540 2024-03-11T21:43:47.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3c801af540 0x7f3c801b1fd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c801a8db0 con 0x7f3c801af540 2024-03-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 9 = system:0 2024-03-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 4 2024-03-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.917+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 4 9 2024-03-11T21:43:47.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 10 client.42137.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 10 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 10 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801a8bf0 osd=4 3 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 4 10 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 4 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 10 client.42137.objecter _send_command 10 2024-03-11T21:43:47.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.937+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c801a8db0 con 0x7f3c801af540 2024-03-11T21:43:47.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c801a8db0 con 0x7f3c801af540 2024-03-11T21:43:47.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 4 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 4 10 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 10 client.42137.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 11 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3c801b2ad0 0x7f3c801b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 11 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 5 11 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 4 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 10 client.42137.objecter _send_command 11 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801bb500 con 0x7f3c801b2ad0 2024-03-11T21:43:47.928 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3c801b2ad0 0x7f3c801b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801b2ad0 2024-03-11T21:43:47.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.941+0000 7f3c767fc700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3c801b2ad0 0x7f3c801b2f10 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.945+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c68036a50 con 0x7f3c801b2ad0 2024-03-11T21:43:47.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.945+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:43:47.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.945+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 11 = system:0 2024-03-11T21:43:47.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.945+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 4 2024-03-11T21:43:47.930 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.945+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 5 11 2024-03-11T21:43:47.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 10 client.42137.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 12 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 12 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801a8db0 osd=5 3 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 5 12 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 4 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 10 client.42137.objecter _send_command 12 2024-03-11T21:43:47.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.965+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c801b8be0 con 0x7f3c801b2ad0 2024-03-11T21:43:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.952 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:47.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:47.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c801b8be0 con 0x7f3c801b2ad0 2024-03-11T21:43:47.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 4 2024-03-11T21:43:47.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 5 12 2024-03-11T21:43:47.955 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:43:47.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 10 client.42137.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 13 2024-03-11T21:43:47.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3c801b9300 0x7f3c801b9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 13 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 6 13 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 4 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 10 client.42137.objecter _send_command 13 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801bafe0 con 0x7f3c801b9300 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3c801b9300 0x7f3c801b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801b9300 2024-03-11T21:43:47.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7f7fe700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3c801b9300 0x7f3c801b9760 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c801bafe0 con 0x7f3c801b9300 2024-03-11T21:43:47.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:43:47.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 13 = system:0 2024-03-11T21:43:47.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 4 2024-03-11T21:43:47.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.969+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 6 13 2024-03-11T21:43:47.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 10 client.42137.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 14 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 14 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801b8be0 osd=6 3 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 6 14 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 4 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 10 client.42137.objecter _send_command 14 2024-03-11T21:43:47.975 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.989+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c801bafe0 con 0x7f3c801b9300 2024-03-11T21:43:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c801bafe0 con 0x7f3c801b9300 2024-03-11T21:43:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:47.978 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 4 2024-03-11T21:43:47.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 6 14 2024-03-11T21:43:47.979 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:43:47.979 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 10 client.42137.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 15 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3c801ba2c0 0x7f3c801ba740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 15 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 7 15 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 4 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 10 client.42137.objecter _send_command 15 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c801bac80 con 0x7f3c801ba2c0 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3c801ba2c0 0x7f3c801ba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 10 client.42137.objecter ms_handle_connect 0x7f3c801ba2c0 2024-03-11T21:43:47.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7effd700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3c801ba2c0 0x7f3c801ba740 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:47.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c801bac80 con 0x7f3c801ba2c0 2024-03-11T21:43:47.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:43:47.981 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 15 = system:0 2024-03-11T21:43:47.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 4 2024-03-11T21:43:47.982 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:47.993+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 7 15 2024-03-11T21:43:47.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 10 client.42137.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign -1 16 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 20 client.42137.objecter _get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_remove -1 16 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 20 client.42137.objecter get_session s=0x7f3c801bafe0 osd=7 3 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 15 client.42137.objecter _session_command_op_assign 7 16 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 4 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 10 client.42137.objecter _send_command 16 2024-03-11T21:43:48.000 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.013+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3c801bac80 con 0x7f3c801ba2c0 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7cff9700 1 -- 172.21.15.143:0/2727032799 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3c801bac80 con 0x7f3c801ba2c0 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7cff9700 10 client.42137.objecter ms_dispatch 0x7f3c80001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7cff9700 10 client.42137.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7cff9700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 4 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7cff9700 15 client.42137.objecter _session_command_op_remove 7 16 2024-03-11T21:43:48.004 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:43:48.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 librados: watch_flush enter 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 librados: watch_flush exit 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.0 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3c800629f0 msgr2=0x7f3c80062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3c800629f0 0x7f3c80062dd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8019b130 osd=0 1 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.1 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3c8002c6b0 msgr2=0x7f3c8002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3c8002c6b0 0x7f3c8002ca90 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b650 osd=1 1 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.2 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3c801abf30 msgr2=0x7f3c801ae320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3c801abf30 0x7f3c801ae320 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8010b4d0 osd=2 1 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.3 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3c801a98d0 msgr2=0x7f3c801b19f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3c801a98d0 0x7f3c801b19f0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c8002d620 osd=3 1 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.4 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 reap_dead start 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3c801af540 msgr2=0x7f3c801b1fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3c801af540 0x7f3c801b1fd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 reap_dead start 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8bf0 osd=4 1 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.5 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3c801b2ad0 msgr2=0x7f3c801b2f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3c801b2ad0 0x7f3c801b2f10 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801a8db0 osd=5 1 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.6 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3c801b9300 msgr2=0x7f3c801b9760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3c801b9300 0x7f3c801b9760 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801b8be0 osd=6 1 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter close_session for osd.7 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3c801ba2c0 msgr2=0x7f3c801ba740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3c801ba2c0 0x7f3c801ba740 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter put_session s=0x7f3c801bafe0 osd=7 1 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 20 client.42137.objecter shutdown clearing up homeless session... 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 10 client.42137.objecter successfully canceled tick 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3c6c06f820 msgr2=0x7f3c6c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3c6c06f820 0x7f3c6c071ce0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009680 tx=0x7f3c70009260 comp rx=0 tx=0).stop 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 msgr2=0x7f3c8019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:48.008 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c7f7fe700 1 -- 172.21.15.143:0/2727032799 reap_dead start 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800ef30 tx=0x7f3c7800c3a0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 shutdown_connections 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 --2- 172.21.15.143:0/2727032799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c80109360 0x7f3c8019e6a0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 >> 172.21.15.143:0/2727032799 conn(0x7f3c800ffb10 msgr2=0x7f3c801001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 shutdown_connections 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 -- 172.21.15.143:0/2727032799 wait complete. 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:48.017+0000 7f3c852f4700 1 librados: shutdown 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:48.009 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:48.010 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:48.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:48.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:48.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:48.198 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:48.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.329 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:48.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:48.460 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:48.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.590 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:48.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:48.718 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:48.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.830 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:48 smithi203 bash[23654]: cluster 2024-03-11T21:43:47.660222+0000 mgr.y (mgr.14550) 806 : cluster 0 pgmap v1393: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:48.830 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:48 smithi203 bash[23654]: audit 2024-03-11T21:43:47.832703+0000 mon.c (mon.2) 1361 : audit 0 from='client.? 172.21.15.143:0/2727032799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:48.844 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:48.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:48 smithi143 bash[21512]: cluster 2024-03-11T21:43:47.660222+0000 mgr.y (mgr.14550) 806 : cluster 0 pgmap v1393: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:48.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:48 smithi143 bash[21512]: audit 2024-03-11T21:43:47.832703+0000 mon.c (mon.2) 1361 : audit 0 from='client.? 172.21.15.143:0/2727032799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:48.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:48 smithi143 bash[29284]: cluster 2024-03-11T21:43:47.660222+0000 mgr.y (mgr.14550) 806 : cluster 0 pgmap v1393: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:48.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:48 smithi143 bash[29284]: audit 2024-03-11T21:43:47.832703+0000 mon.c (mon.2) 1361 : audit 0 from='client.? 172.21.15.143:0/2727032799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:48.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:48.971 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:49.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:49.101 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:49.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:49.232 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:49.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:49.363 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:49.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:49.494 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:49.530 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:49.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:49.625 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:49.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:49.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:49.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:49.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:49 smithi203 bash[23654]: audit 2024-03-11T21:43:49.119230+0000 mgr.y (mgr.14550) 807 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:49.879 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:49.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:49 smithi143 bash[29284]: audit 2024-03-11T21:43:49.119230+0000 mgr.y (mgr.14550) 807 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:49.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:49 smithi143 bash[21512]: audit 2024-03-11T21:43:49.119230+0000 mgr.y (mgr.14550) 807 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:49.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:50.014 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:50.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:50.143 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:50.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:50.275 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:50.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:50.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:50.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:50.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:50.669 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:50.773 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:50.780 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:50 smithi203 bash[23654]: cluster 2024-03-11T21:43:49.662476+0000 mgr.y (mgr.14550) 808 : cluster 0 pgmap v1394: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:43:50.800 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:50.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.923 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:50.994 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:43:50.995 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:43:50.995 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:43:50.995 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:43:51.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.041+0000 7fe260913700 1 -- 172.21.15.143:0/1148340071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c074010 msgr2=0x7fe25c0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.041+0000 7fe260913700 1 --2- 172.21.15.143:0/1148340071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c074010 0x7fe25c0743f0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b0a0 tx=0x7fe24c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/1148340071 shutdown_connections 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- 172.21.15.143:0/1148340071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c10e870 0x7fe25c110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- 172.21.15.143:0/1148340071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c074010 0x7fe25c0743f0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- 172.21.15.143:0/1148340071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074930 0x7fe25c10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/1148340071 >> 172.21.15.143:0/1148340071 conn(0x7fe25c078eb0 msgr2=0x7fe25c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/1148340071 shutdown_connections 2024-03-11T21:43:51.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/1148340071 wait complete. 2024-03-11T21:43:51.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 librados: starting msgr at 2024-03-11T21:43:51.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 librados: starting objecter 2024-03-11T21:43:51.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 Processor -- start 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- start start 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 librados: setting wanted keys 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 librados: calling monclient init 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 0x7fe25c199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 0x7fe25c19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c113800 con 0x7fe25c074010 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c113680 con 0x7fe25c074930 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c113980 con 0x7fe25c10e870 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51240/0 (socket says 172.21.15.143:51240) 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 -- 172.21.15.143:0/2154661623 learned_addr learned my addr 172.21.15.143:0/2154661623 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25a59c700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 0x7fe25c199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:51.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe259d9b700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 0x7fe25c19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:51.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 -- 172.21.15.143:0/2154661623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 msgr2=0x7fe25c19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:51.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 0x7fe25c19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 -- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 msgr2=0x7fe25c199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:51.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 0x7fe25c199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 -- 172.21.15.143:0/2154661623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c19ede0 con 0x7fe25c10e870 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe259d9b700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 0x7fe25c19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25a59c700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 0x7fe25c199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe25ad9d700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e970 tx=0x7fe25400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 10 client.?.objecter ms_handle_connect 0x7fe25c10e870 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25400cbd0 con 0x7fe25c10e870 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe25400ee60 con 0x7fe25c10e870 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c19f0d0 con 0x7fe25c10e870 2024-03-11T21:43:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254005470 con 0x7fe25c10e870 2024-03-11T21:43:51.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 10 client.42143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:51.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c1a6950 con 0x7fe25c10e870 2024-03-11T21:43:51.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 librados: init done 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c04faa0 con 0x7fe25c10e870 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.045+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe254004750 con 0x7fe25c10e870 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 0x7fe248071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(892..892 src has 842..892) v4 ==== 9021+0+0 (secure 0 0 0) 0x7fe25408ea40 con 0x7fe25c10e870 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 10 client.42143.objecter ms_dispatch 0x7fe25c001550 osd_map(892..892 src has 842..892) v4 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 3 client.42143.objecter handle_osd_map got epochs [892,892] > 0 2024-03-11T21:43:51.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 3 client.42143.objecter handle_osd_map decoding full epoch 892 2024-03-11T21:43:51.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe25a59c700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 0x7fe248071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:51.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 20 client.42143.objecter dump_active .. 0 homeless 2024-03-11T21:43:51.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe25a59c700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 0x7fe248071ce0 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009e60 tx=0x7fe24c00b700 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:51.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.049+0000 7fe2537fe700 10 client.42143.objecter ms_handle_connect 0x7fe24806f820 2024-03-11T21:43:51.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.053+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe25405b910 con 0x7fe25c10e870 2024-03-11T21:43:51.046 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:51.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:51.174 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:51.176 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:51.193 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:50 smithi143 bash[21512]: cluster 2024-03-11T21:43:49.662476+0000 mgr.y (mgr.14550) 808 : cluster 0 pgmap v1394: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:43:51.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:50 smithi143 bash[29284]: cluster 2024-03-11T21:43:49.662476+0000 mgr.y (mgr.14550) 808 : cluster 0 pgmap v1394: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:43:51.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.277+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe25c19b130 con 0x7fe25c10e870 2024-03-11T21:43:51.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.436 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.569 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:51.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:43:51.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:51.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.745+0000 7fe2537fe700 1 -- 172.21.15.143:0/2154661623 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v893) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe25405f360 con 0x7fe25c10e870 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 10 librados: watch_flush enter 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 10 librados: watch_flush exit 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 20 client.42143.objecter shutdown clearing up homeless session... 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 10 client.42143.objecter successfully canceled tick 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 msgr2=0x7fe248071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 0x7fe248071ce0 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009e60 tx=0x7fe24c00b700 comp rx=0 tx=0).stop 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 msgr2=0x7fe25c19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e970 tx=0x7fe25400ec80 comp rx=0 tx=0).stop 2024-03-11T21:43:51.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 shutdown_connections 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe24806f820 0x7fe248071ce0 unknown :-1 s=CLOSED pgs=3468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe25c074010 0x7fe25c199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe25c10e870 0x7fe25c19e6a0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 --2- 172.21.15.143:0/2154661623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe25c074930 0x7fe25c19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 >> 172.21.15.143:0/2154661623 conn(0x7fe25c078eb0 msgr2=0x7fe25c1062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 shutdown_connections 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 -- 172.21.15.143:0/2154661623 wait complete. 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:51.749+0000 7fe260913700 1 librados: shutdown 2024-03-11T21:43:51.738 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:43:51.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:43:51.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.869 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:51.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:51 smithi143 bash[21512]: audit 2024-03-11T21:43:51.282898+0000 mon.c (mon.2) 1362 : audit 1 from='client.? 172.21.15.143:0/2154661623' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:51.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:51 smithi143 bash[21512]: audit 2024-03-11T21:43:51.283286+0000 mon.a (mon.0) 3955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:51.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:51 smithi143 bash[29284]: audit 2024-03-11T21:43:51.282898+0000 mon.c (mon.2) 1362 : audit 1 from='client.? 172.21.15.143:0/2154661623' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:51.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:51 smithi143 bash[29284]: audit 2024-03-11T21:43:51.283286+0000 mon.a (mon.0) 3955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:51.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:51.998 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:52.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:51 smithi203 bash[23654]: audit 2024-03-11T21:43:51.282898+0000 mon.c (mon.2) 1362 : audit 1 from='client.? 172.21.15.143:0/2154661623' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:52.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:51 smithi203 bash[23654]: audit 2024-03-11T21:43:51.283286+0000 mon.a (mon.0) 3955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:52.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:52.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:52.242 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:43:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:52.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:52.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:52.386 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:52.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:52.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:52.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:52.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:52.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:52.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:52 smithi203 bash[23654]: cluster 2024-03-11T21:43:51.663202+0000 mgr.y (mgr.14550) 809 : cluster 0 pgmap v1395: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:43:52.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:52 smithi203 bash[23654]: cluster 2024-03-11T21:43:51.733320+0000 mon.a (mon.0) 3956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:52.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:52 smithi203 bash[23654]: audit 2024-03-11T21:43:51.742257+0000 mon.a (mon.0) 3957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:52.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:52 smithi203 bash[23654]: cluster 2024-03-11T21:43:51.750871+0000 mon.a (mon.0) 3958 : cluster 0 osdmap e893: 8 total, 8 up, 8 in 2024-03-11T21:43:52.764 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:52 smithi203 bash[23654]: audit 2024-03-11T21:43:52.544699+0000 mon.c (mon.2) 1363 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:52.781 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:52.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:52.912 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:53.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[29284]: cluster 2024-03-11T21:43:51.663202+0000 mgr.y (mgr.14550) 809 : cluster 0 pgmap v1395: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[29284]: cluster 2024-03-11T21:43:51.733320+0000 mon.a (mon.0) 3956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[29284]: audit 2024-03-11T21:43:51.742257+0000 mon.a (mon.0) 3957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[29284]: cluster 2024-03-11T21:43:51.750871+0000 mon.a (mon.0) 3958 : cluster 0 osdmap e893: 8 total, 8 up, 8 in 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[29284]: audit 2024-03-11T21:43:52.544699+0000 mon.c (mon.2) 1363 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21512]: cluster 2024-03-11T21:43:51.663202+0000 mgr.y (mgr.14550) 809 : cluster 0 pgmap v1395: 206 pgs: 206 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21512]: cluster 2024-03-11T21:43:51.733320+0000 mon.a (mon.0) 3956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:53.027 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21512]: audit 2024-03-11T21:43:51.742257+0000 mon.a (mon.0) 3957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:53.028 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21512]: cluster 2024-03-11T21:43:51.750871+0000 mon.a (mon.0) 3958 : cluster 0 osdmap e893: 8 total, 8 up, 8 in 2024-03-11T21:43:53.028 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:52 smithi143 bash[21512]: audit 2024-03-11T21:43:52.544699+0000 mon.c (mon.2) 1363 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:53.043 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:53.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:53.173 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:53.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:53.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:53.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:53.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.568 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:53.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:53.700 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:53.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:53.828 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:53.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.957 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:54.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:54.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:54.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:54.220 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:54.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:54.352 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:54.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:54.483 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:54.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:54.608 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:54.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:54.723 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:54 smithi143 bash[21512]: cluster 2024-03-11T21:43:53.665158+0000 mgr.y (mgr.14550) 810 : cluster 0 pgmap v1397: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:54.738 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:54.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:54.852 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:54 smithi203 bash[23654]: cluster 2024-03-11T21:43:53.665158+0000 mgr.y (mgr.14550) 810 : cluster 0 pgmap v1397: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:54.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:54.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:54.996 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:55.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:54 smithi143 bash[29284]: cluster 2024-03-11T21:43:53.665158+0000 mgr.y (mgr.14550) 810 : cluster 0 pgmap v1397: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:55.099 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:55.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:55.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:55.258 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:55.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:55.388 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:55.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:55.516 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:55.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:55.645 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:55.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:55.776 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:55.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:43:55.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:55.900 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:56.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:56.029 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:56.132 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:56.159 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:56.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:56.292 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:56.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:56.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- 172.21.15.143:0/2412861219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406b050 msgr2=0x7f2b74106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- 172.21.15.143:0/2412861219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406b050 0x7f2b74106b40 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f2b68009990 tx=0x7f2b68009ca0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- 172.21.15.143:0/2412861219 shutdown_connections 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- 172.21.15.143:0/2412861219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b74107220 0x7f2b7410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- 172.21.15.143:0/2412861219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7406a6a0 0x7f2b7406aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- 172.21.15.143:0/2412861219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406b050 0x7f2b74106b40 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- 172.21.15.143:0/2412861219 >> 172.21.15.143:0/2412861219 conn(0x7f2b74077de0 msgr2=0x7f2b740781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- 172.21.15.143:0/2412861219 shutdown_connections 2024-03-11T21:43:56.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- 172.21.15.143:0/2412861219 wait complete. 2024-03-11T21:43:56.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 librados: starting msgr at 2024-03-11T21:43:56.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 librados: starting objecter 2024-03-11T21:43:56.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 Processor -- start 2024-03-11T21:43:56.411 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 -- start start 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 librados: setting wanted keys 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 librados: calling monclient init 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 0x7f2b741012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.420+0000 7f2b7a782700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b74107220 0x7f2b74101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741116d0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74111550 con 0x7f2b74107220 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74111850 con 0x7f2b7406b050 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 0x7f2b741012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46472/0 (socket says 172.21.15.143:46472) 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 -- 172.21.15.143:0/3184153652 learned_addr learned my addr 172.21.15.143:0/3184153652 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b78d1f700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b74107220 0x7f2b74101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:56.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 -- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 msgr2=0x7f2b741012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 0x7f2b741012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 -- 172.21.15.143:0/3184153652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b74107220 msgr2=0x7f2b74101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b74107220 0x7f2b74101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 -- 172.21.15.143:0/3184153652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b741a61f0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b737fe700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 0x7f2b741012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b73fff700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b530 tx=0x7f2b6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 10 client.?.objecter ms_handle_connect 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64015020 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b741a64e0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f2b64011cb0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64010d90 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 10 client.30945.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741a6a40 con 0x7f2b7406a6a0 2024-03-11T21:43:56.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 librados: init done 2024-03-11T21:43:56.414 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7404faa0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f2b64018020 con 0x7f2b7406a6a0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.424+0000 7f2b717fa700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 0x7f2b5c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(893..893 src has 842..893) v4 ==== 9021+0+0 (secure 0 0 0) 0x7f2b6408e0c0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 10 client.30945.objecter ms_dispatch 0x7f2b74001550 osd_map(893..893 src has 842..893) v4 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 3 client.30945.objecter handle_osd_map got epochs [893,893] > 0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 3 client.30945.objecter handle_osd_map decoding full epoch 893 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b737fe700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 0x7f2b5c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 20 client.30945.objecter dump_active .. 0 homeless 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b737fe700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 0x7f2b5c071ce0 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f2b74102820 tx=0x7f2b6801a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:43:56.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.428+0000 7f2b717fa700 10 client.30945.objecter ms_handle_connect 0x7f2b5c06f820 2024-03-11T21:43:56.419 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:56.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.432+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2b6401f020 con 0x7f2b7406a6a0 2024-03-11T21:43:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:56.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:56.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:56.560 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:56.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.644+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} v 0) v1 -- 0x7f2b741a6ca0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:56.676 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:56.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.736+0000 7f2b717fa700 1 -- 172.21.15.143:0/3184153652 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]=0 set pool 1 pg_num to 21 v894) v1 ==== 129+0+0 (secure 0 0 0) 0x7f2b6405aff0 con 0x7f2b7406a6a0 2024-03-11T21:43:56.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 10 librados: watch_flush enter 2024-03-11T21:43:56.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 10 librados: watch_flush exit 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 20 client.30945.objecter shutdown clearing up homeless session... 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 10 client.30945.objecter successfully canceled tick 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 msgr2=0x7f2b5c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 0x7f2b5c071ce0 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f2b74102820 tx=0x7f2b6801a040 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 msgr2=0x7f2b74100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.740+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b530 tx=0x7f2b6400b840 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 shutdown_connections 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f2b5c06f820 0x7f2b5c071ce0 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b7406a6a0 0x7f2b74100d90 secure :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b530 tx=0x7f2b6400b840 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b74107220 0x7f2b74101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 --2- 172.21.15.143:0/3184153652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b7406b050 0x7f2b741012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:43:56.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 >> 172.21.15.143:0/3184153652 conn(0x7f2b74077de0 msgr2=0x7f2b740feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:43:56.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 shutdown_connections 2024-03-11T21:43:56.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 -- 172.21.15.143:0/3184153652 wait complete. 2024-03-11T21:43:56.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:43:56.744+0000 7f2b7a782700 1 librados: shutdown 2024-03-11T21:43:56.732 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pg_num to 21 2024-03-11T21:43:56.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:56.790 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:56 smithi203 bash[23654]: cluster 2024-03-11T21:43:55.666217+0000 mgr.y (mgr.14550) 811 : cluster 0 pgmap v1398: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:56.790 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:56 smithi203 bash[23654]: audit 2024-03-11T21:43:56.649212+0000 mon.a (mon.0) 3959 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:43:56.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:56.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:56.934 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:56.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:56 smithi143 bash[29284]: cluster 2024-03-11T21:43:55.666217+0000 mgr.y (mgr.14550) 811 : cluster 0 pgmap v1398: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:56.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:56 smithi143 bash[29284]: audit 2024-03-11T21:43:56.649212+0000 mon.a (mon.0) 3959 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:43:56.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:56 smithi143 bash[21512]: cluster 2024-03-11T21:43:55.666217+0000 mgr.y (mgr.14550) 811 : cluster 0 pgmap v1398: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:56.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:56 smithi143 bash[21512]: audit 2024-03-11T21:43:56.649212+0000 mon.a (mon.0) 3959 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:43:57.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:57.065 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:57.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:57.197 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:57.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:57.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:57.428 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:57.454 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:57.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.581 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:57.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.709 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:57.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:57.834 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:57.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[29284]: audit 2024-03-11T21:43:56.743163+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[29284]: cluster 2024-03-11T21:43:56.748484+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e894: 8 total, 8 up, 8 in 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[29284]: audit 2024-03-11T21:43:57.286338+0000 mon.c (mon.2) 1364 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[29284]: audit 2024-03-11T21:43:57.667498+0000 mon.c (mon.2) 1365 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[29284]: audit 2024-03-11T21:43:57.668475+0000 mon.a (mon.0) 3962 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[21512]: audit 2024-03-11T21:43:56.743163+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:43:57.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[21512]: cluster 2024-03-11T21:43:56.748484+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e894: 8 total, 8 up, 8 in 2024-03-11T21:43:57.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[21512]: audit 2024-03-11T21:43:57.286338+0000 mon.c (mon.2) 1364 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:43:57.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[21512]: audit 2024-03-11T21:43:57.667498+0000 mon.c (mon.2) 1365 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:57.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:57 smithi143 bash[21512]: audit 2024-03-11T21:43:57.668475+0000 mon.a (mon.0) 3962 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:57.960 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:58.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:57 smithi203 bash[23654]: audit 2024-03-11T21:43:56.743163+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.143:0/3184153652' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:43:58.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:57 smithi203 bash[23654]: cluster 2024-03-11T21:43:56.748484+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e894: 8 total, 8 up, 8 in 2024-03-11T21:43:58.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:57 smithi203 bash[23654]: audit 2024-03-11T21:43:57.286338+0000 mon.c (mon.2) 1364 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:43:58.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:57 smithi203 bash[23654]: audit 2024-03-11T21:43:57.667498+0000 mon.c (mon.2) 1365 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:58.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:57 smithi203 bash[23654]: audit 2024-03-11T21:43:57.668475+0000 mon.a (mon.0) 3962 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:58.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:58.087 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:58.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:58.216 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:58.222 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:58.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:58.347 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:58.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:58.478 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:58.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:58.611 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:58.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:58.749 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:58.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:58.871 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: cluster 2024-03-11T21:43:57.666984+0000 mgr.y (mgr.14550) 812 : cluster 0 pgmap v1400: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:58.872 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: audit 2024-03-11T21:43:57.753766+0000 mon.a (mon.0) 3963 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:43:58.872 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: cluster 2024-03-11T21:43:57.756517+0000 mon.a (mon.0) 3964 : cluster 0 osdmap e895: 8 total, 8 up, 8 in 2024-03-11T21:43:58.872 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: audit 2024-03-11T21:43:58.026313+0000 mon.c (mon.2) 1366 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:58.872 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: audit 2024-03-11T21:43:58.027571+0000 mon.c (mon.2) 1367 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:58.872 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:58 smithi203 bash[23654]: audit 2024-03-11T21:43:58.033393+0000 mon.a (mon.0) 3965 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:58.886 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:58.989 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:58.999 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: cluster 2024-03-11T21:43:57.666984+0000 mgr.y (mgr.14550) 812 : cluster 0 pgmap v1400: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:58.999 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: audit 2024-03-11T21:43:57.753766+0000 mon.a (mon.0) 3963 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: cluster 2024-03-11T21:43:57.756517+0000 mon.a (mon.0) 3964 : cluster 0 osdmap e895: 8 total, 8 up, 8 in 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: audit 2024-03-11T21:43:58.026313+0000 mon.c (mon.2) 1366 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: audit 2024-03-11T21:43:58.027571+0000 mon.c (mon.2) 1367 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[21512]: audit 2024-03-11T21:43:58.033393+0000 mon.a (mon.0) 3965 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: cluster 2024-03-11T21:43:57.666984+0000 mgr.y (mgr.14550) 812 : cluster 0 pgmap v1400: 206 pgs: 206 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: audit 2024-03-11T21:43:57.753766+0000 mon.a (mon.0) 3963 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:43:59.000 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: cluster 2024-03-11T21:43:57.756517+0000 mon.a (mon.0) 3964 : cluster 0 osdmap e895: 8 total, 8 up, 8 in 2024-03-11T21:43:59.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: audit 2024-03-11T21:43:58.026313+0000 mon.c (mon.2) 1366 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:59.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: audit 2024-03-11T21:43:58.027571+0000 mon.c (mon.2) 1367 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:59.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:58 smithi143 bash[29284]: audit 2024-03-11T21:43:58.033393+0000 mon.a (mon.0) 3965 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:43:59.014 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:59.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:59.145 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:59.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:59.274 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:59.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:43:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:43:59.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:59.405 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:59.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:59.530 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:59.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:59.656 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:59.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:59.794 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:59.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:59.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[21512]: cluster 2024-03-11T21:43:58.762773+0000 mon.a (mon.0) 3966 : cluster 0 osdmap e896: 8 total, 8 up, 8 in 2024-03-11T21:43:59.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[21512]: audit 2024-03-11T21:43:59.125150+0000 mgr.y (mgr.14550) 813 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:59.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[21512]: audit 2024-03-11T21:43:59.668136+0000 mon.c (mon.2) 1368 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:59.901 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[21512]: audit 2024-03-11T21:43:59.669620+0000 mon.a (mon.0) 3967 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:59.901 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[29284]: cluster 2024-03-11T21:43:58.762773+0000 mon.a (mon.0) 3966 : cluster 0 osdmap e896: 8 total, 8 up, 8 in 2024-03-11T21:43:59.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[29284]: audit 2024-03-11T21:43:59.125150+0000 mgr.y (mgr.14550) 813 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:59.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[29284]: audit 2024-03-11T21:43:59.668136+0000 mon.c (mon.2) 1368 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:59.902 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:43:59 smithi143 bash[29284]: audit 2024-03-11T21:43:59.669620+0000 mon.a (mon.0) 3967 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:43:59.924 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:59.989 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:43:59.989 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:43:59.989 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:43:59.989 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:43:59.989 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:44:00.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:59 smithi203 bash[23654]: cluster 2024-03-11T21:43:58.762773+0000 mon.a (mon.0) 3966 : cluster 0 osdmap e896: 8 total, 8 up, 8 in 2024-03-11T21:44:00.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:59 smithi203 bash[23654]: audit 2024-03-11T21:43:59.125150+0000 mgr.y (mgr.14550) 813 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:00.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:59 smithi203 bash[23654]: audit 2024-03-11T21:43:59.668136+0000 mon.c (mon.2) 1368 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:44:00.028 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:43:59 smithi203 bash[23654]: audit 2024-03-11T21:43:59.669620+0000 mon.a (mon.0) 3967 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:44:00.052 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:00.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:00.183 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:00.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:00.313 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:00.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.445 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:00.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:00.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:00.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.815 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:00 smithi203 bash[23654]: cluster 2024-03-11T21:43:59.667747+0000 mgr.y (mgr.14550) 814 : cluster 0 pgmap v1403: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:44:00.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:00 smithi203 bash[23654]: audit 2024-03-11T21:43:59.779247+0000 mon.a (mon.0) 3968 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:44:00.816 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:00 smithi203 bash[23654]: cluster 2024-03-11T21:43:59.785083+0000 mon.a (mon.0) 3969 : cluster 0 osdmap e897: 8 total, 8 up, 8 in 2024-03-11T21:44:00.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:44:00.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:00.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[21512]: cluster 2024-03-11T21:43:59.667747+0000 mgr.y (mgr.14550) 814 : cluster 0 pgmap v1403: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[21512]: audit 2024-03-11T21:43:59.779247+0000 mon.a (mon.0) 3968 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[21512]: cluster 2024-03-11T21:43:59.785083+0000 mon.a (mon.0) 3969 : cluster 0 osdmap e897: 8 total, 8 up, 8 in 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[29284]: cluster 2024-03-11T21:43:59.667747+0000 mgr.y (mgr.14550) 814 : cluster 0 pgmap v1403: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[29284]: audit 2024-03-11T21:43:59.779247+0000 mon.a (mon.0) 3968 : audit 1 from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:44:00.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:00 smithi143 bash[29284]: cluster 2024-03-11T21:43:59.785083+0000 mon.a (mon.0) 3969 : cluster 0 osdmap e897: 8 total, 8 up, 8 in 2024-03-11T21:44:00.960 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:01.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.116 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:01.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:01.219 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:01.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:01.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:01.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:01.482 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:01.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:01.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:01.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:01.741 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:01.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:01.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:01.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:01 smithi143 bash[29284]: cluster 2024-03-11T21:44:00.794571+0000 mon.a (mon.0) 3970 : cluster 0 osdmap e898: 8 total, 8 up, 8 in 2024-03-11T21:44:01.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:01 smithi143 bash[21512]: cluster 2024-03-11T21:44:00.794571+0000 mon.a (mon.0) 3970 : cluster 0 osdmap e898: 8 total, 8 up, 8 in 2024-03-11T21:44:01.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:01.982 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:01 smithi203 bash[23654]: cluster 2024-03-11T21:44:00.794571+0000 mon.a (mon.0) 3970 : cluster 0 osdmap e898: 8 total, 8 up, 8 in 2024-03-11T21:44:01.998 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:02.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:02.129 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:02.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.260 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:02.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.392 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:02.417 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:02.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:02.523 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:02.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:02.653 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:02.741 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:44:02.742 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:02.742 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:44:02.742 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-03-11T21:44:02.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 0 2024-03-11T21:44:02.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:02.783 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:02.789 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:02 smithi143 bash[21512]: cluster 2024-03-11T21:44:01.668479+0000 mgr.y (mgr.14550) 815 : cluster 0 pgmap v1406: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:02.789 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:02 smithi143 bash[21512]: cluster 2024-03-11T21:44:01.809415+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e899: 8 total, 8 up, 8 in 2024-03-11T21:44:02.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:02.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:03.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:03.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:02 smithi203 bash[23654]: cluster 2024-03-11T21:44:01.668479+0000 mgr.y (mgr.14550) 815 : cluster 0 pgmap v1406: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:03.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:02 smithi203 bash[23654]: cluster 2024-03-11T21:44:01.809415+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e899: 8 total, 8 up, 8 in 2024-03-11T21:44:03.035 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:03.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1641657558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850100fc0 msgr2=0x7fa850101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- 172.21.15.143:0/1641657558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850100fc0 0x7fa850101440 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fa840009990 tx=0x7fa840009ca0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1641657558 shutdown_connections 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- 172.21.15.143:0/1641657558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850106e30 0x7fa850107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- 172.21.15.143:0/1641657558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 0x7fa85010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- 172.21.15.143:0/1641657558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850100fc0 0x7fa850101440 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1641657558 >> 172.21.15.143:0/1641657558 conn(0x7fa850077bd0 msgr2=0x7fa850077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1641657558 shutdown_connections 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1641657558 wait complete. 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 librados: starting msgr at 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 librados: starting objecter 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 Processor -- start 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- start start 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 librados: setting wanted keys 2024-03-11T21:44:03.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 librados: calling monclient init 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 0x7fa85019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 0x7fa85019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850106e30 0x7fa8501979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501105c0 con 0x7fa850100fc0 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa850110440 con 0x7fa850101980 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa850110740 con 0x7fa850106e30 2024-03-11T21:44:03.150 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 0x7fa85019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 0x7fa85019d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:46496/0 (socket says 172.21.15.143:46496) 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 -- 172.21.15.143:0/1529913781 learned_addr learned my addr 172.21.15.143:0/1529913781 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850106e30 0x7fa8501979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 0x7fa85019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850106e30 msgr2=0x7fa8501979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa850106e30 0x7fa8501979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 msgr2=0x7fa85019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 0x7fa85019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850198140 con 0x7fa850100fc0 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa850101980 0x7fa85019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 0x7fa85019d820 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fa84800c370 tx=0x7fa84800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8467fc700 10 client.?.objecter ms_handle_connect 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8467fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84800cea0 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8501983d0 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa8480090d0 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa848004660 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 10 client.30951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8501a46f0 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 librados: init done 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.160+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa85004faa0 con 0x7fa850100fc0 2024-03-11T21:44:03.152 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa84801a020 con 0x7fa850100fc0 2024-03-11T21:44:03.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa83c06f820 0x7fa83c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(900..900 src has 842..900) v4 ==== 9021+0+0 (secure 0 0 0) 0x7fa84808e280 con 0x7fa850100fc0 2024-03-11T21:44:03.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 osd_map(900..900 src has 842..900) v4 2024-03-11T21:44:03.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa83c06f820 0x7fa83c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 3 client.30951.objecter handle_osd_map got epochs [900,900] > 0 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 3 client.30951.objecter handle_osd_map decoding full epoch 900 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 20 client.30951.objecter dump_active .. 0 homeless 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v900) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa848010040 con 0x7fa850100fc0 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 10 client.30951.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 1 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa85010a290 0x7fa85010a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 1 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 0 1 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 4 2024-03-11T21:44:03.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 10 client.30951.objecter _send_command 1 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa850104950 con 0x7fa85010a290 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa83c06f820 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa83c06f820 0x7fa83c071ce0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7fa84000b2e0 tx=0x7fa84001a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa85010a290 0x7fa85010a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa85010a290 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa85010a290 0x7fa85010a670 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa850104950 con 0x7fa85010a290 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 10 client.30951.objecter _finish_command 1 = system:0 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 4 2024-03-11T21:44:03.155 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.164+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 0 1 2024-03-11T21:44:03.165 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 10 client.30951.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 2 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 2 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa850198f10 osd=0 3 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 0 2 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 4 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 10 client.30951.objecter _send_command 2 2024-03-11T21:44:03.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa850109d10 con 0x7fa85010a290 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa850109d10 con 0x7fa85010a290 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 10 client.30951.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 4 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 0 2 2024-03-11T21:44:03.172 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 10 client.30951.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 3 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa850062f00 0x7fa8501aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 3 2024-03-11T21:44:03.173 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 1 3 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 4 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 10 client.30951.objecter _send_command 3 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa85002d9a0 con 0x7fa850062f00 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa850062f00 0x7fa8501aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa850062f00 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa850062f00 0x7fa8501aace0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa85002d9a0 con 0x7fa850062f00 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 10 client.30951.objecter _finish_command 3 = system:0 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 4 2024-03-11T21:44:03.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.184+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 1 3 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 10 client.30951.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 4 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 4 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002d830 osd=1 3 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 1 4 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 4 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 10 client.30951.objecter _send_command 4 2024-03-11T21:44:03.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501ab220 con 0x7fa850062f00 2024-03-11T21:44:03.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501ab220 con 0x7fa850062f00 2024-03-11T21:44:03.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 10 client.30951.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 4 2024-03-11T21:44:03.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 1 4 2024-03-11T21:44:03.193 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 10 client.30951.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 5 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa8500625e0 0x7fa8501ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 5 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 2 5 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 4 2024-03-11T21:44:03.194 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 10 client.30951.objecter _send_command 5 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa85002d390 con 0x7fa8500625e0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa8500625e0 0x7fa8501ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8500625e0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.204+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa8500625e0 0x7fa8501ae560 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.208+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa85002d390 con 0x7fa8500625e0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.208+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.208+0000 7fa8467fc700 10 client.30951.objecter _finish_command 5 = system:0 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.208+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 4 2024-03-11T21:44:03.195 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.208+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 2 5 2024-03-11T21:44:03.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:02 smithi143 bash[29284]: cluster 2024-03-11T21:44:01.668479+0000 mgr.y (mgr.14550) 815 : cluster 0 pgmap v1406: 216 pgs: 7 peering, 10 unknown, 199 active+clean; 582 KiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:03.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:02 smithi143 bash[29284]: cluster 2024-03-11T21:44:01.809415+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e899: 8 total, 8 up, 8 in 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 10 client.30951.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 6 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 6 2024-03-11T21:44:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa850062470 osd=2 3 2024-03-11T21:44:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 2 6 2024-03-11T21:44:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 4 2024-03-11T21:44:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 10 client.30951.objecter _send_command 6 2024-03-11T21:44:03.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.224+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501aeaa0 con 0x7fa8500625e0 2024-03-11T21:44:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501aeaa0 con 0x7fa8500625e0 2024-03-11T21:44:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 10 client.30951.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 4 2024-03-11T21:44:03.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 2 6 2024-03-11T21:44:03.215 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 10 client.30951.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 7 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa8501ac0b0 0x7fa8501aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 7 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 3 7 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 4 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 10 client.30951.objecter _send_command 7 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa85002cbd0 con 0x7fa8501ac0b0 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa8501ac0b0 0x7fa8501aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.216 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8501ac0b0 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa8501ac0b0 0x7fa8501aecb0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa85002cbd0 con 0x7fa8501ac0b0 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 10 client.30951.objecter _finish_command 7 = system:0 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 4 2024-03-11T21:44:03.217 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.228+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 3 7 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 10 client.30951.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 8 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 8 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002c900 osd=3 3 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 3 8 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 4 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 10 client.30951.objecter _send_command 8 2024-03-11T21:44:03.236 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501ab620 con 0x7fa8501ac0b0 2024-03-11T21:44:03.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501ab620 con 0x7fa8501ac0b0 2024-03-11T21:44:03.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa8467fc700 10 client.30951.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 4 2024-03-11T21:44:03.237 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 3 8 2024-03-11T21:44:03.238 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:44:03.238 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 10 client.30951.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 9 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa8501afed0 0x7fa8501b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 9 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 4 9 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 4 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 10 client.30951.objecter _send_command 9 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.248+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8501b84d0 con 0x7fa8501afed0 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa8501afed0 0x7fa8501b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa8501afed0 0x7fa8501b5670 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.239 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8501afed0 2024-03-11T21:44:03.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa8501b84d0 con 0x7fa8501afed0 2024-03-11T21:44:03.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:44:03.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 10 client.30951.objecter _finish_command 9 = system:0 2024-03-11T21:44:03.240 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 4 2024-03-11T21:44:03.241 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.252+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 4 9 2024-03-11T21:44:03.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 10 client.30951.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 10 2024-03-11T21:44:03.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 10 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa85002cc30 osd=4 3 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 4 10 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 4 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 10 client.30951.objecter _send_command 10 2024-03-11T21:44:03.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.272+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501b5bb0 con 0x7fa8501afed0 2024-03-11T21:44:03.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501b5bb0 con 0x7fa8501afed0 2024-03-11T21:44:03.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 10 client.30951.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 4 2024-03-11T21:44:03.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 4 10 2024-03-11T21:44:03.264 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:44:03.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 10 client.30951.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 11 2024-03-11T21:44:03.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa8501b62d0 0x7fa8501b6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 11 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 5 11 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 4 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 10 client.30951.objecter _send_command 11 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8501bb860 con 0x7fa8501b62d0 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa8501b62d0 0x7fa8501b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8501b62d0 2024-03-11T21:44:03.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.276+0000 7fa854bc1700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa8501b62d0 0x7fa8501b6710 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.280+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa8501bb860 con 0x7fa8501b62d0 2024-03-11T21:44:03.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.280+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:44:03.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.280+0000 7fa8467fc700 10 client.30951.objecter _finish_command 11 = system:0 2024-03-11T21:44:03.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.280+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 4 2024-03-11T21:44:03.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.280+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 5 11 2024-03-11T21:44:03.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:03.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 10 client.30951.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 12 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 12 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501b5bb0 osd=5 3 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 5 12 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 4 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 10 client.30951.objecter _send_command 12 2024-03-11T21:44:03.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.296+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501ba690 con 0x7fa8501b62d0 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501ba690 con 0x7fa8501b62d0 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 10 client.30951.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 4 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 5 12 2024-03-11T21:44:03.287 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 10 client.30951.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 13 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa8501bad10 0x7fa8501bb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 13 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 6 13 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 4 2024-03-11T21:44:03.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 10 client.30951.objecter _send_command 13 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8501bec00 con 0x7fa8501bad10 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa8501bad10 0x7fa8501bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8501bad10 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa855bc3700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa8501bad10 0x7fa8501bb130 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa8501bec00 con 0x7fa8501bad10 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 10 client.30951.objecter _finish_command 13 = system:0 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 4 2024-03-11T21:44:03.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.300+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 6 13 2024-03-11T21:44:03.294 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:03.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 10 client.30951.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 14 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 14 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501ba690 osd=6 3 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 6 14 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 4 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 10 client.30951.objecter _send_command 14 2024-03-11T21:44:03.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.316+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501b6ca0 con 0x7fa8501bad10 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501b6ca0 con 0x7fa8501bad10 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 10 client.30951.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 4 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 6 14 2024-03-11T21:44:03.308 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 10 client.30951.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 15 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa8501b7370 0x7fa8501b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 15 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 7 15 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 4 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 10 client.30951.objecter _send_command 15 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8501b7ed0 con 0x7fa8501b7370 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa8501b7370 0x7fa8501b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8467fc700 10 client.30951.objecter ms_handle_connect 0x7fa8501b7370 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.320+0000 7fa8553c2700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa8501b7370 0x7fa8501b77d0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.324+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa8501b7ed0 con 0x7fa8501b7370 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.324+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:44:03.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.324+0000 7fa8467fc700 10 client.30951.objecter _finish_command 15 = system:0 2024-03-11T21:44:03.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.324+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 4 2024-03-11T21:44:03.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.324+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 7 15 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 10 client.30951.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign -1 16 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 20 client.30951.objecter _get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 15 client.30951.objecter _session_command_op_remove -1 16 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 20 client.30951.objecter get_session s=0x7fa8501b6ca0 osd=7 3 2024-03-11T21:44:03.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 15 client.30951.objecter _session_command_op_assign 7 16 2024-03-11T21:44:03.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 4 2024-03-11T21:44:03.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 10 client.30951.objecter _send_command 16 2024-03-11T21:44:03.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.340+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa8501b7ed0 con 0x7fa8501b7370 2024-03-11T21:44:03.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa8467fc700 1 -- 172.21.15.143:0/1529913781 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa8501b7ed0 con 0x7fa8501b7370 2024-03-11T21:44:03.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa8467fc700 10 client.30951.objecter ms_dispatch 0x7fa850001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:03.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa8467fc700 10 client.30951.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:03.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa8467fc700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 4 2024-03-11T21:44:03.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa8467fc700 15 client.30951.objecter _session_command_op_remove 7 16 2024-03-11T21:44:03.331 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 librados: watch_flush enter 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 librados: watch_flush exit 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.0 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa85010a290 msgr2=0x7fa85010a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fa85010a290 0x7fa85010a670 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850198f10 osd=0 1 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.1 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa850062f00 msgr2=0x7fa8501aace0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fa850062f00 0x7fa8501aace0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002d830 osd=1 1 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.2 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa8500625e0 msgr2=0x7fa8501ae560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fa8500625e0 0x7fa8501ae560 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa850062470 osd=2 1 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.3 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa8501ac0b0 msgr2=0x7fa8501aecb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fa8501ac0b0 0x7fa8501aecb0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa855bc3700 1 -- 172.21.15.143:0/1529913781 reap_dead start 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002c900 osd=3 1 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.4 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa8501afed0 msgr2=0x7fa8501b5670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fa8501afed0 0x7fa8501b5670 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa85002cc30 osd=4 1 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.5 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa855bc3700 1 -- 172.21.15.143:0/1529913781 reap_dead start 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa8501b62d0 msgr2=0x7fa8501b6710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fa8501b62d0 0x7fa8501b6710 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b5bb0 osd=5 1 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.6 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa8501bad10 msgr2=0x7fa8501bb130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fa8501bad10 0x7fa8501bb130 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501ba690 osd=6 1 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter close_session for osd.7 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa8501b7370 msgr2=0x7fa8501b77d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fa8501b7370 0x7fa8501b77d0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter put_session s=0x7fa8501b6ca0 osd=7 1 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 20 client.30951.objecter shutdown clearing up homeless session... 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 10 client.30951.objecter successfully canceled tick 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa83c06f820 msgr2=0x7fa83c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa83c06f820 0x7fa83c071ce0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7fa84000b2e0 tx=0x7fa84001a040 comp rx=0 tx=0).stop 2024-03-11T21:44:03.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 msgr2=0x7fa85019d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 --2- 172.21.15.143:0/1529913781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa850100fc0 0x7fa85019d820 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fa84800c370 tx=0x7fa84800c680 comp rx=0 tx=0).stop 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa855bc3700 1 -- 172.21.15.143:0/1529913781 reap_dead start 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 shutdown_connections 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 >> 172.21.15.143:0/1529913781 conn(0x7fa850077bd0 msgr2=0x7fa85006a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 shutdown_connections 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 -- 172.21.15.143:0/1529913781 wait complete. 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:03.344+0000 7fa857626700 1 librados: shutdown 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:03.335 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:03.336 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:03.336 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:03.336 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:03.336 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:03.336 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:03.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:03.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:03.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:03.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:03.687 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:03.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:03.821 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:03.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:03.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:03 smithi143 bash[21512]: cluster 2024-03-11T21:44:02.822394+0000 mon.a (mon.0) 3972 : cluster 0 osdmap e900: 8 total, 8 up, 8 in 2024-03-11T21:44:03.937 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:03 smithi143 bash[21512]: audit 2024-03-11T21:44:03.168203+0000 mon.a (mon.0) 3973 : audit 0 from='client.? 172.21.15.143:0/1529913781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:03.937 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:03 smithi143 bash[29284]: cluster 2024-03-11T21:44:02.822394+0000 mon.a (mon.0) 3972 : cluster 0 osdmap e900: 8 total, 8 up, 8 in 2024-03-11T21:44:03.938 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:03 smithi143 bash[29284]: audit 2024-03-11T21:44:03.168203+0000 mon.a (mon.0) 3973 : audit 0 from='client.? 172.21.15.143:0/1529913781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:03.954 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:04.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:03 smithi203 bash[23654]: cluster 2024-03-11T21:44:02.822394+0000 mon.a (mon.0) 3972 : cluster 0 osdmap e900: 8 total, 8 up, 8 in 2024-03-11T21:44:04.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:03 smithi203 bash[23654]: audit 2024-03-11T21:44:03.168203+0000 mon.a (mon.0) 3973 : audit 0 from='client.? 172.21.15.143:0/1529913781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:04.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.085 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:04.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.215 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:04.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.343 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:04.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.473 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:04.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:04.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:04.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:04.736 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:04.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:04.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:04 smithi203 bash[23654]: cluster 2024-03-11T21:44:03.670516+0000 mgr.y (mgr.14550) 816 : cluster 0 pgmap v1409: 216 pgs: 8 peering, 2 unknown, 206 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 137 KiB/s wr, 22 op/s; 178 KiB/s, 1 objects/s recovering 2024-03-11T21:44:04.850 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:04 smithi203 bash[23654]: cluster 2024-03-11T21:44:03.827577+0000 mon.a (mon.0) 3974 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-03-11T21:44:04.866 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:04.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:04.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:04 smithi143 bash[29284]: cluster 2024-03-11T21:44:03.670516+0000 mgr.y (mgr.14550) 816 : cluster 0 pgmap v1409: 216 pgs: 8 peering, 2 unknown, 206 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 137 KiB/s wr, 22 op/s; 178 KiB/s, 1 objects/s recovering 2024-03-11T21:44:04.980 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:04 smithi143 bash[29284]: cluster 2024-03-11T21:44:03.827577+0000 mon.a (mon.0) 3974 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-03-11T21:44:04.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:04 smithi143 bash[21512]: cluster 2024-03-11T21:44:03.670516+0000 mgr.y (mgr.14550) 816 : cluster 0 pgmap v1409: 216 pgs: 8 peering, 2 unknown, 206 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 137 KiB/s wr, 22 op/s; 178 KiB/s, 1 objects/s recovering 2024-03-11T21:44:04.981 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:04 smithi143 bash[21512]: cluster 2024-03-11T21:44:03.827577+0000 mon.a (mon.0) 3974 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-03-11T21:44:04.996 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:05.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:05.125 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:05.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:05.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:05.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:05.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:05.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:05.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:05.600 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:44:05.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:05.642 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:05.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:05.773 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:05.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:05.904 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:06.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:06.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:06.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:06.166 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:06.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:06.295 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:06.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:06.425 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:06.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 -- 172.21.15.143:0/178803368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 msgr2=0x7fce74104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:06.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 --2- 172.21.15.143:0/178803368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74104650 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7fce640099c0 tx=0x7fce64009cd0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 -- 172.21.15.143:0/178803368 shutdown_connections 2024-03-11T21:44:06.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 --2- 172.21.15.143:0/178803368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 0x7fce74110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 --2- 172.21.15.143:0/178803368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74104650 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 --2- 172.21.15.143:0/178803368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce74108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 -- 172.21.15.143:0/178803368 >> 172.21.15.143:0/178803368 conn(0x7fce740ffb00 msgr2=0x7fce74101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 -- 172.21.15.143:0/178803368 shutdown_connections 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.528+0000 7fce79ca5700 1 -- 172.21.15.143:0/178803368 wait complete. 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 librados: starting msgr at 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 librados: starting objecter 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 Processor -- start 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- start start 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 librados: setting wanted keys 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 librados: calling monclient init 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 0x7fce7419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:06.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74113870 con 0x7fce74104c20 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce741136f0 con 0x7fce74109350 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce741139f0 con 0x7fce74104270 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33956/0 (socket says 172.21.15.143:33956) 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 -- 172.21.15.143:0/4147099893 learned_addr learned my addr 172.21.15.143:0/4147099893 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce737fe700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce73fff700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 0x7fce7419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 -- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 msgr2=0x7fce74199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 -- 172.21.15.143:0/4147099893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 msgr2=0x7fce7419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 0x7fce7419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 -- 172.21.15.143:0/4147099893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7419ee50 con 0x7fce74104c20 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce72ffd700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00e970 tx=0x7fce5c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 10 client.?.objecter ms_handle_connect 0x7fce74104c20 2024-03-11T21:44:06.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00cbd0 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7419f140 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 10 client.30957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce741a69c0 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 librados: init done 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fce5c00cd70 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c005590 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce54005120 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fce5c004750 con 0x7fce74104c20 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.532+0000 7fce70ff9700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 0x7fce60071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:06.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(901..901 src has 842..901) v4 ==== 9021+0+0 (secure 0 0 0) 0x7fce5c08ebe0 con 0x7fce74104c20 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 10 client.30957.objecter ms_dispatch 0x7fce74001550 osd_map(901..901 src has 842..901) v4 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 3 client.30957.objecter handle_osd_map got epochs [901,901] > 0 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 3 client.30957.objecter handle_osd_map decoding full epoch 901 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce737fe700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 0x7fce60071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 20 client.30957.objecter dump_active .. 0 homeless 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce70ff9700 10 client.30957.objecter ms_handle_connect 0x7fce6006f820 2024-03-11T21:44:06.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.536+0000 7fce737fe700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 0x7fce60071ce0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fce640037a0 tx=0x7fce64019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:06.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.540+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fce5c05ba90 con 0x7fce74104c20 2024-03-11T21:44:06.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:06.557 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:06.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.752+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fce54005410 con 0x7fce74104c20 2024-03-11T21:44:06.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:06.815 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:06.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.852+0000 7fce70ff9700 1 -- 172.21.15.143:0/4147099893 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v902) v1 ==== 91+0+0 (secure 0 0 0) 0x7fce5c05f4e0 con 0x7fce74104c20 2024-03-11T21:44:06.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.852+0000 7fce79ca5700 10 librados: watch_flush enter 2024-03-11T21:44:06.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 10 librados: watch_flush exit 2024-03-11T21:44:06.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 20 client.30957.objecter shutdown clearing up homeless session... 2024-03-11T21:44:06.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 10 client.30957.objecter successfully canceled tick 2024-03-11T21:44:06.842 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 msgr2=0x7fce60071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:06.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 0x7fce60071ce0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fce640037a0 tx=0x7fce64019040 comp rx=0 tx=0).stop 2024-03-11T21:44:06.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 msgr2=0x7fce7419a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:06.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.856+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00e970 tx=0x7fce5c00ec80 comp rx=0 tx=0).stop 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 shutdown_connections 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fce6006f820 0x7fce60071ce0 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce74109350 0x7fce7419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74104270 0x7fce74199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 --2- 172.21.15.143:0/4147099893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74104c20 0x7fce7419a380 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 >> 172.21.15.143:0/4147099893 conn(0x7fce740ffb00 msgr2=0x7fce74100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 shutdown_connections 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 -- 172.21.15.143:0/4147099893 wait complete. 2024-03-11T21:44:06.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:06.860+0000 7fce79ca5700 1 librados: shutdown 2024-03-11T21:44:06.851 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:44:06.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:06.936 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:06.988 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:06 smithi203 bash[23654]: cluster 2024-03-11T21:44:05.671632+0000 mgr.y (mgr.14550) 817 : cluster 0 pgmap v1411: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 113 KiB/s wr, 18 op/s; 146 KiB/s, 1 objects/s recovering 2024-03-11T21:44:06.988 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:06 smithi203 bash[23654]: audit 2024-03-11T21:44:06.759591+0000 mon.a (mon.0) 3975 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:07.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:07.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:44:07.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:07.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:06 smithi143 bash[21512]: cluster 2024-03-11T21:44:05.671632+0000 mgr.y (mgr.14550) 817 : cluster 0 pgmap v1411: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 113 KiB/s wr, 18 op/s; 146 KiB/s, 1 objects/s recovering 2024-03-11T21:44:07.062 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:06 smithi143 bash[21512]: audit 2024-03-11T21:44:06.759591+0000 mon.a (mon.0) 3975 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:07.063 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:06 smithi143 bash[29284]: cluster 2024-03-11T21:44:05.671632+0000 mgr.y (mgr.14550) 817 : cluster 0 pgmap v1411: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 113 KiB/s wr, 18 op/s; 146 KiB/s, 1 objects/s recovering 2024-03-11T21:44:07.063 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:06 smithi143 bash[29284]: audit 2024-03-11T21:44:06.759591+0000 mon.a (mon.0) 3975 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:07.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:07.186 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:07.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:07.313 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:07.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:07.442 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:07.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:07.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:07.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:07.705 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:07.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:07.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.956 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:08.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:08.087 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:08.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[29284]: cluster 2024-03-11T21:44:06.854693+0000 mon.a (mon.0) 3976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[29284]: audit 2024-03-11T21:44:06.856277+0000 mon.a (mon.0) 3977 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[29284]: cluster 2024-03-11T21:44:06.860324+0000 mon.a (mon.0) 3978 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[29284]: audit 2024-03-11T21:44:07.577955+0000 mon.c (mon.2) 1369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[21512]: cluster 2024-03-11T21:44:06.854693+0000 mon.a (mon.0) 3976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[21512]: audit 2024-03-11T21:44:06.856277+0000 mon.a (mon.0) 3977 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[21512]: cluster 2024-03-11T21:44:06.860324+0000 mon.a (mon.0) 3978 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-03-11T21:44:08.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:07 smithi143 bash[21512]: audit 2024-03-11T21:44:07.577955+0000 mon.c (mon.2) 1369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.219 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:08.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:07 smithi203 bash[23654]: cluster 2024-03-11T21:44:06.854693+0000 mon.a (mon.0) 3976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:08.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:07 smithi203 bash[23654]: audit 2024-03-11T21:44:06.856277+0000 mon.a (mon.0) 3977 : audit 1 from='client.? 172.21.15.143:0/4147099893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:08.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:07 smithi203 bash[23654]: cluster 2024-03-11T21:44:06.860324+0000 mon.a (mon.0) 3978 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-03-11T21:44:08.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:07 smithi203 bash[23654]: audit 2024-03-11T21:44:07.577955+0000 mon.c (mon.2) 1369 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:08.349 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:08.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.479 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:08.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.611 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:08.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:08.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:08.868 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:08.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:08.994 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:09.051 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:09.051 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:09.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:09.116 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:09.116 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.356567+0000 osd.3 (osd.3) 479 : cluster 0 39.e scrub starts 2024-03-11T21:44:09.116 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.358049+0000 osd.3 (osd.3) 480 : cluster 0 39.e scrub ok 2024-03-11T21:44:09.116 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.556552+0000 osd.5 (osd.5) 497 : cluster 0 39.0 scrub starts 2024-03-11T21:44:09.117 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.559053+0000 osd.5 (osd.5) 498 : cluster 0 39.0 scrub ok 2024-03-11T21:44:09.117 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.672465+0000 mgr.y (mgr.14550) 818 : cluster 0 pgmap v1413: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 94 KiB/s wr, 15 op/s; 122 KiB/s, 0 objects/s recovering 2024-03-11T21:44:09.117 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.928710+0000 osd.4 (osd.4) 457 : cluster 0 6.1e scrub starts 2024-03-11T21:44:09.117 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:08 smithi203 bash[23654]: cluster 2024-03-11T21:44:07.929899+0000 osd.4 (osd.4) 458 : cluster 0 6.1e scrub ok 2024-03-11T21:44:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:09.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.356567+0000 osd.3 (osd.3) 479 : cluster 0 39.e scrub starts 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.358049+0000 osd.3 (osd.3) 480 : cluster 0 39.e scrub ok 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.556552+0000 osd.5 (osd.5) 497 : cluster 0 39.0 scrub starts 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.559053+0000 osd.5 (osd.5) 498 : cluster 0 39.0 scrub ok 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.672465+0000 mgr.y (mgr.14550) 818 : cluster 0 pgmap v1413: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 94 KiB/s wr, 15 op/s; 122 KiB/s, 0 objects/s recovering 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.928710+0000 osd.4 (osd.4) 457 : cluster 0 6.1e scrub starts 2024-03-11T21:44:09.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[21512]: cluster 2024-03-11T21:44:07.929899+0000 osd.4 (osd.4) 458 : cluster 0 6.1e scrub ok 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.356567+0000 osd.3 (osd.3) 479 : cluster 0 39.e scrub starts 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.358049+0000 osd.3 (osd.3) 480 : cluster 0 39.e scrub ok 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.556552+0000 osd.5 (osd.5) 497 : cluster 0 39.0 scrub starts 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.559053+0000 osd.5 (osd.5) 498 : cluster 0 39.0 scrub ok 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.672465+0000 mgr.y (mgr.14550) 818 : cluster 0 pgmap v1413: 216 pgs: 8 peering, 208 active+clean; 713 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 94 KiB/s wr, 15 op/s; 122 KiB/s, 0 objects/s recovering 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.928710+0000 osd.4 (osd.4) 457 : cluster 0 6.1e scrub starts 2024-03-11T21:44:09.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:08 smithi143 bash[29284]: cluster 2024-03-11T21:44:07.929899+0000 osd.4 (osd.4) 458 : cluster 0 6.1e scrub ok 2024-03-11T21:44:09.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:09.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:09.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:09.375 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:09.463 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:09.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:09.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:09.506 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:09.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:09.636 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:09.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:09.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:09.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:09.880 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:09 smithi143 bash[29284]: audit 2024-03-11T21:44:09.133694+0000 mgr.y (mgr.14550) 819 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:09.880 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:09 smithi143 bash[21512]: audit 2024-03-11T21:44:09.133694+0000 mgr.y (mgr.14550) 819 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:09.894 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:09.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:44:09.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:44:09.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:10.032 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:10.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:10.143 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:09 smithi203 bash[23654]: audit 2024-03-11T21:44:09.133694+0000 mgr.y (mgr.14550) 819 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:10.154 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:10.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:10.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:10.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:10.413 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:10.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:10.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:10.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:10.678 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:10.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:10.807 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:10.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:10.939 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.247843+0000 osd.1 (osd.1) 287 : cluster 0 1.b scrub starts 2024-03-11T21:44:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.250424+0000 osd.1 (osd.1) 288 : cluster 0 1.b scrub ok 2024-03-11T21:44:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.384261+0000 osd.0 (osd.0) 478 : cluster 0 1.11 scrub starts 2024-03-11T21:44:10.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.386203+0000 osd.0 (osd.0) 479 : cluster 0 1.11 scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.504270+0000 osd.7 (osd.7) 470 : cluster 0 1.d scrub starts 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.506824+0000 osd.7 (osd.7) 471 : cluster 0 1.d scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.589433+0000 osd.6 (osd.6) 399 : cluster 0 1.f scrub starts 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.590902+0000 osd.6 (osd.6) 400 : cluster 0 1.f scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.619203+0000 osd.5 (osd.5) 499 : cluster 0 1.10 scrub starts 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.621651+0000 osd.5 (osd.5) 500 : cluster 0 1.10 scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.673704+0000 mgr.y (mgr.14550) 820 : cluster 0 pgmap v1414: 216 pgs: 216 active+clean; 713 KiB data, 752 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 14 op/s; 104 KiB/s, 0 objects/s recovering 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.951533+0000 osd.4 (osd.4) 459 : cluster 0 1.e scrub starts 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:09.953472+0000 osd.4 (osd.4) 460 : cluster 0 1.e scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.204418+0000 osd.1 (osd.1) 289 : cluster 0 1.13 scrub starts 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.205708+0000 osd.1 (osd.1) 290 : cluster 0 1.13 scrub ok 2024-03-11T21:44:10.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.247843+0000 osd.1 (osd.1) 287 : cluster 0 1.b scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.250424+0000 osd.1 (osd.1) 288 : cluster 0 1.b scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.384261+0000 osd.0 (osd.0) 478 : cluster 0 1.11 scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.386203+0000 osd.0 (osd.0) 479 : cluster 0 1.11 scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.504270+0000 osd.7 (osd.7) 470 : cluster 0 1.d scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.506824+0000 osd.7 (osd.7) 471 : cluster 0 1.d scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.589433+0000 osd.6 (osd.6) 399 : cluster 0 1.f scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.590902+0000 osd.6 (osd.6) 400 : cluster 0 1.f scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.619203+0000 osd.5 (osd.5) 499 : cluster 0 1.10 scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.621651+0000 osd.5 (osd.5) 500 : cluster 0 1.10 scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.673704+0000 mgr.y (mgr.14550) 820 : cluster 0 pgmap v1414: 216 pgs: 216 active+clean; 713 KiB data, 752 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 14 op/s; 104 KiB/s, 0 objects/s recovering 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.951533+0000 osd.4 (osd.4) 459 : cluster 0 1.e scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:09.953472+0000 osd.4 (osd.4) 460 : cluster 0 1.e scrub ok 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.204418+0000 osd.1 (osd.1) 289 : cluster 0 1.13 scrub starts 2024-03-11T21:44:10.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:10 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.205708+0000 osd.1 (osd.1) 290 : cluster 0 1.13 scrub ok 2024-03-11T21:44:11.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:11.168 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:11.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:11.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.247843+0000 osd.1 (osd.1) 287 : cluster 0 1.b scrub starts 2024-03-11T21:44:11.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.250424+0000 osd.1 (osd.1) 288 : cluster 0 1.b scrub ok 2024-03-11T21:44:11.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.384261+0000 osd.0 (osd.0) 478 : cluster 0 1.11 scrub starts 2024-03-11T21:44:11.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.386203+0000 osd.0 (osd.0) 479 : cluster 0 1.11 scrub ok 2024-03-11T21:44:11.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.504270+0000 osd.7 (osd.7) 470 : cluster 0 1.d scrub starts 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.506824+0000 osd.7 (osd.7) 471 : cluster 0 1.d scrub ok 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.589433+0000 osd.6 (osd.6) 399 : cluster 0 1.f scrub starts 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.590902+0000 osd.6 (osd.6) 400 : cluster 0 1.f scrub ok 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.619203+0000 osd.5 (osd.5) 499 : cluster 0 1.10 scrub starts 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.621651+0000 osd.5 (osd.5) 500 : cluster 0 1.10 scrub ok 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.673704+0000 mgr.y (mgr.14550) 820 : cluster 0 pgmap v1414: 216 pgs: 216 active+clean; 713 KiB data, 752 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 14 op/s; 104 KiB/s, 0 objects/s recovering 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.951533+0000 osd.4 (osd.4) 459 : cluster 0 1.e scrub starts 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:09.953472+0000 osd.4 (osd.4) 460 : cluster 0 1.e scrub ok 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.204418+0000 osd.1 (osd.1) 289 : cluster 0 1.13 scrub starts 2024-03-11T21:44:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:10 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.205708+0000 osd.1 (osd.1) 290 : cluster 0 1.13 scrub ok 2024-03-11T21:44:11.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:11.320 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:11.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:11.449 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:11.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:11.580 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:11.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:11.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:11.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.840 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:11.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:11.972 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:12.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.222907+0000 osd.2 (osd.2) 421 : cluster 0 1.14 scrub starts 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.224359+0000 osd.2 (osd.2) 422 : cluster 0 1.14 scrub ok 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.298233+0000 osd.3 (osd.3) 481 : cluster 0 1.c scrub starts 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.300783+0000 osd.3 (osd.3) 482 : cluster 0 1.c scrub ok 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.556435+0000 osd.6 (osd.6) 401 : cluster 0 1.6 scrub starts 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.557711+0000 osd.6 (osd.6) 402 : cluster 0 1.6 scrub ok 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.614802+0000 osd.5 (osd.5) 501 : cluster 0 1.12 scrub starts 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.617248+0000 osd.5 (osd.5) 502 : cluster 0 1.12 scrub ok 2024-03-11T21:44:12.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.965785+0000 osd.4 (osd.4) 461 : cluster 0 39.15 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:10.968431+0000 osd.4 (osd.4) 462 : cluster 0 39.15 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.217121+0000 osd.1 (osd.1) 291 : cluster 0 1.3 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.219383+0000 osd.1 (osd.1) 292 : cluster 0 1.3 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.222907+0000 osd.2 (osd.2) 421 : cluster 0 1.14 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.224359+0000 osd.2 (osd.2) 422 : cluster 0 1.14 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.298233+0000 osd.3 (osd.3) 481 : cluster 0 1.c scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.300783+0000 osd.3 (osd.3) 482 : cluster 0 1.c scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.556435+0000 osd.6 (osd.6) 401 : cluster 0 1.6 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.557711+0000 osd.6 (osd.6) 402 : cluster 0 1.6 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.614802+0000 osd.5 (osd.5) 501 : cluster 0 1.12 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.617248+0000 osd.5 (osd.5) 502 : cluster 0 1.12 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.965785+0000 osd.4 (osd.4) 461 : cluster 0 39.15 scrub starts 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:10.968431+0000 osd.4 (osd.4) 462 : cluster 0 39.15 scrub ok 2024-03-11T21:44:12.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.217121+0000 osd.1 (osd.1) 291 : cluster 0 1.3 scrub starts 2024-03-11T21:44:12.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:11 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.219383+0000 osd.1 (osd.1) 292 : cluster 0 1.3 scrub ok 2024-03-11T21:44:12.103 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:12.195 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:12.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.229 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.222907+0000 osd.2 (osd.2) 421 : cluster 0 1.14 scrub starts 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.224359+0000 osd.2 (osd.2) 422 : cluster 0 1.14 scrub ok 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.298233+0000 osd.3 (osd.3) 481 : cluster 0 1.c scrub starts 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.300783+0000 osd.3 (osd.3) 482 : cluster 0 1.c scrub ok 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.556435+0000 osd.6 (osd.6) 401 : cluster 0 1.6 scrub starts 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.557711+0000 osd.6 (osd.6) 402 : cluster 0 1.6 scrub ok 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.614802+0000 osd.5 (osd.5) 501 : cluster 0 1.12 scrub starts 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.617248+0000 osd.5 (osd.5) 502 : cluster 0 1.12 scrub ok 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.965785+0000 osd.4 (osd.4) 461 : cluster 0 39.15 scrub starts 2024-03-11T21:44:12.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:10.968431+0000 osd.4 (osd.4) 462 : cluster 0 39.15 scrub ok 2024-03-11T21:44:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.217121+0000 osd.1 (osd.1) 291 : cluster 0 1.3 scrub starts 2024-03-11T21:44:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:11 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.219383+0000 osd.1 (osd.1) 292 : cluster 0 1.3 scrub ok 2024-03-11T21:44:12.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.356 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:12.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:12.489 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:12.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:12.618 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:12.704 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:12.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:12.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:12.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.878 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:12.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.284898+0000 osd.3 (osd.3) 483 : cluster 0 39.a scrub starts 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.286367+0000 osd.3 (osd.3) 484 : cluster 0 39.a scrub ok 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.573708+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.575762+0000 osd.7 (osd.7) 472 : cluster 0 1.0 scrub starts 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.576009+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.580463+0000 osd.7 (osd.7) 473 : cluster 0 1.0 scrub ok 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.636181+0000 osd.5 (osd.5) 503 : cluster 0 1.1 scrub starts 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.638009+0000 osd.5 (osd.5) 504 : cluster 0 1.1 scrub ok 2024-03-11T21:44:12.993 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.674616+0000 mgr.y (mgr.14550) 821 : cluster 0 pgmap v1415: 216 pgs: 216 active+clean; 713 KiB data, 751 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:12.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.929180+0000 osd.4 (osd.4) 463 : cluster 0 1.5 scrub starts 2024-03-11T21:44:12.994 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:12 smithi203 bash[23654]: cluster 2024-03-11T21:44:11.931563+0000 osd.4 (osd.4) 464 : cluster 0 1.5 scrub ok 2024-03-11T21:44:13.009 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:13.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:13.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.284898+0000 osd.3 (osd.3) 483 : cluster 0 39.a scrub starts 2024-03-11T21:44:13.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.286367+0000 osd.3 (osd.3) 484 : cluster 0 39.a scrub ok 2024-03-11T21:44:13.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.573708+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:44:13.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.575762+0000 osd.7 (osd.7) 472 : cluster 0 1.0 scrub starts 2024-03-11T21:44:13.123 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.576009+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.580463+0000 osd.7 (osd.7) 473 : cluster 0 1.0 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.636181+0000 osd.5 (osd.5) 503 : cluster 0 1.1 scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.638009+0000 osd.5 (osd.5) 504 : cluster 0 1.1 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.674616+0000 mgr.y (mgr.14550) 821 : cluster 0 pgmap v1415: 216 pgs: 216 active+clean; 713 KiB data, 751 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.929180+0000 osd.4 (osd.4) 463 : cluster 0 1.5 scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[21512]: cluster 2024-03-11T21:44:11.931563+0000 osd.4 (osd.4) 464 : cluster 0 1.5 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.284898+0000 osd.3 (osd.3) 483 : cluster 0 39.a scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.286367+0000 osd.3 (osd.3) 484 : cluster 0 39.a scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.573708+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.575762+0000 osd.7 (osd.7) 472 : cluster 0 1.0 scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.576009+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.580463+0000 osd.7 (osd.7) 473 : cluster 0 1.0 scrub ok 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.636181+0000 osd.5 (osd.5) 503 : cluster 0 1.1 scrub starts 2024-03-11T21:44:13.124 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.638009+0000 osd.5 (osd.5) 504 : cluster 0 1.1 scrub ok 2024-03-11T21:44:13.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.674616+0000 mgr.y (mgr.14550) 821 : cluster 0 pgmap v1415: 216 pgs: 216 active+clean; 713 KiB data, 751 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:13.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.929180+0000 osd.4 (osd.4) 463 : cluster 0 1.5 scrub starts 2024-03-11T21:44:13.125 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:12 smithi143 bash[29284]: cluster 2024-03-11T21:44:11.931563+0000 osd.4 (osd.4) 464 : cluster 0 1.5 scrub ok 2024-03-11T21:44:13.140 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:13.243 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.268 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:13.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.400 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:13.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.529 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:13.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:13.660 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:13.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:13.792 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:13.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:13.906 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:12.321944+0000 osd.3 (osd.3) 485 : cluster 0 1.4 scrub starts 2024-03-11T21:44:13.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:12.323796+0000 osd.3 (osd.3) 486 : cluster 0 1.4 scrub ok 2024-03-11T21:44:13.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:12.546560+0000 osd.7 (osd.7) 474 : cluster 0 1.2 scrub starts 2024-03-11T21:44:13.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:12.548166+0000 osd.7 (osd.7) 475 : cluster 0 1.2 scrub ok 2024-03-11T21:44:13.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.194162+0000 osd.1 (osd.1) 293 : cluster 0 39.c scrub starts 2024-03-11T21:44:13.907 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:13 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.195629+0000 osd.1 (osd.1) 294 : cluster 0 39.c scrub ok 2024-03-11T21:44:13.920 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:14.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:12.321944+0000 osd.3 (osd.3) 485 : cluster 0 1.4 scrub starts 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:12.323796+0000 osd.3 (osd.3) 486 : cluster 0 1.4 scrub ok 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:12.546560+0000 osd.7 (osd.7) 474 : cluster 0 1.2 scrub starts 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:12.548166+0000 osd.7 (osd.7) 475 : cluster 0 1.2 scrub ok 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.194162+0000 osd.1 (osd.1) 293 : cluster 0 39.c scrub starts 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.195629+0000 osd.1 (osd.1) 294 : cluster 0 39.c scrub ok 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:12.321944+0000 osd.3 (osd.3) 485 : cluster 0 1.4 scrub starts 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:12.323796+0000 osd.3 (osd.3) 486 : cluster 0 1.4 scrub ok 2024-03-11T21:44:14.038 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:12.546560+0000 osd.7 (osd.7) 474 : cluster 0 1.2 scrub starts 2024-03-11T21:44:14.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:12.548166+0000 osd.7 (osd.7) 475 : cluster 0 1.2 scrub ok 2024-03-11T21:44:14.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.194162+0000 osd.1 (osd.1) 293 : cluster 0 39.c scrub starts 2024-03-11T21:44:14.039 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:13 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.195629+0000 osd.1 (osd.1) 294 : cluster 0 39.c scrub ok 2024-03-11T21:44:14.055 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:14.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.184 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:14.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:14.314 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:14.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:14.445 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:14.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.553 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:14.574 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:14.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:14.704 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:14.720 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:44:14.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.830 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:14.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:14.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.501651+0000 osd.7 (osd.7) 476 : cluster 0 5.1a scrub starts 2024-03-11T21:44:14.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.504450+0000 osd.7 (osd.7) 477 : cluster 0 5.1a scrub ok 2024-03-11T21:44:14.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.540572+0000 osd.6 (osd.6) 405 : cluster 0 2.1a scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.542869+0000 osd.6 (osd.6) 406 : cluster 0 2.1a scrub ok 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.597105+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.599187+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:13.676634+0000 mgr.y (mgr.14550) 822 : cluster 0 pgmap v1416: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.209285+0000 osd.1 (osd.1) 295 : cluster 0 6.4 scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.211502+0000 osd.1 (osd.1) 296 : cluster 0 6.4 scrub ok 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.501651+0000 osd.7 (osd.7) 476 : cluster 0 5.1a scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.504450+0000 osd.7 (osd.7) 477 : cluster 0 5.1a scrub ok 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.540572+0000 osd.6 (osd.6) 405 : cluster 0 2.1a scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.542869+0000 osd.6 (osd.6) 406 : cluster 0 2.1a scrub ok 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.597105+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:44:14.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.599187+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:44:14.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:13.676634+0000 mgr.y (mgr.14550) 822 : cluster 0 pgmap v1416: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:14.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.209285+0000 osd.1 (osd.1) 295 : cluster 0 6.4 scrub starts 2024-03-11T21:44:14.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:14 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.211502+0000 osd.1 (osd.1) 296 : cluster 0 6.4 scrub ok 2024-03-11T21:44:14.960 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:15.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.090 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:15.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.501651+0000 osd.7 (osd.7) 476 : cluster 0 5.1a scrub starts 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.504450+0000 osd.7 (osd.7) 477 : cluster 0 5.1a scrub ok 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.540572+0000 osd.6 (osd.6) 405 : cluster 0 2.1a scrub starts 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.542869+0000 osd.6 (osd.6) 406 : cluster 0 2.1a scrub ok 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.597105+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.599187+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:13.676634+0000 mgr.y (mgr.14550) 822 : cluster 0 pgmap v1416: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.209285+0000 osd.1 (osd.1) 295 : cluster 0 6.4 scrub starts 2024-03-11T21:44:15.114 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:14 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.211502+0000 osd.1 (osd.1) 296 : cluster 0 6.4 scrub ok 2024-03-11T21:44:15.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.221 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:15.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:15.353 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:15.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.484 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:15.587 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:15.610 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:15.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.720+0000 7fd11b236700 1 -- 172.21.15.143:0/1706882224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114104270 msgr2=0x7fd114104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.720+0000 7fd11b236700 1 --2- 172.21.15.143:0/1706882224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114104270 0x7fd114104650 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fd1080099c0 tx=0x7fd108009cd0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/1706882224 shutdown_connections 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- 172.21.15.143:0/1706882224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114109350 0x7fd114110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- 172.21.15.143:0/1706882224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114104270 0x7fd114104650 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- 172.21.15.143:0/1706882224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 0x7fd114108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/1706882224 >> 172.21.15.143:0/1706882224 conn(0x7fd1140ffac0 msgr2=0x7fd114101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/1706882224 shutdown_connections 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/1706882224 wait complete. 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 librados: starting msgr at 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 librados: starting objecter 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 Processor -- start 2024-03-11T21:44:15.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- start start 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 librados: setting wanted keys 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 librados: calling monclient init 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 0x7fd11419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 0x7fd11419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd114113870 con 0x7fd114104270 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1141136f0 con 0x7fd114104c20 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1141139f0 con 0x7fd114109350 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd118fd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd118fd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35198/0 (socket says 172.21.15.143:35198) 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd118fd2700 1 -- 172.21.15.143:0/2225798662 learned_addr learned my addr 172.21.15.143:0/2225798662 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 0x7fd11419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:15.713 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd113fff700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 0x7fd11419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 -- 172.21.15.143:0/2225798662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 msgr2=0x7fd11419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 0x7fd11419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 -- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 msgr2=0x7fd114199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 -- 172.21.15.143:0/2225798662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11419ee20 con 0x7fd114109350 2024-03-11T21:44:15.714 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd118fd2700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:44:15.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:15.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd1197d3700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 0x7fd11419e6e0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00efb0 tx=0x7fd10c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:15.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 10 client.?.objecter ms_handle_connect 0x7fd114109350 2024-03-11T21:44:15.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:15.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c00ca50 con 0x7fd114109350 2024-03-11T21:44:15.716 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11419f110 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd10c00cbf0 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c004740 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 10 client.42167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1141a6990 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 librados: init done 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11404faa0 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.724+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd10c012050 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 0x7fd104071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(902..902 src has 842..902) v4 ==== 9021+0+0 (secure 0 0 0) 0x7fd10c08e250 con 0x7fd114109350 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 10 client.42167.objecter ms_dispatch 0x7fd114001550 osd_map(902..902 src has 842..902) v4 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 3 client.42167.objecter handle_osd_map got epochs [902,902] > 0 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 3 client.42167.objecter handle_osd_map decoding full epoch 902 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd118fd2700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 0x7fd104071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 20 client.42167.objecter dump_active .. 0 homeless 2024-03-11T21:44:15.717 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd111ffb700 10 client.42167.objecter ms_handle_connect 0x7fd10406f820 2024-03-11T21:44:15.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.728+0000 7fd118fd2700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 0x7fd104071ce0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7fd1080037a0 tx=0x7fd108019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:15.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.732+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd10c022060 con 0x7fd114109350 2024-03-11T21:44:15.736 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:15.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:15.864 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:15.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:15.964+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} v 0) v1 -- 0x7fd1141a6b60 con 0x7fd114109350 2024-03-11T21:44:15.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:15.992 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:16.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.259768+0000 osd.3 (osd.3) 487 : cluster 0 6.b scrub starts 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.261990+0000 osd.3 (osd.3) 488 : cluster 0 6.b scrub ok 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.574728+0000 osd.5 (osd.5) 507 : cluster 0 6.a scrub starts 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:14.576228+0000 osd.5 (osd.5) 508 : cluster 0 6.a scrub ok 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.170261+0000 osd.1 (osd.1) 297 : cluster 0 39.1c scrub starts 2024-03-11T21:44:16.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:15 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.172862+0000 osd.1 (osd.1) 298 : cluster 0 39.1c scrub ok 2024-03-11T21:44:16.114 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:16.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.259768+0000 osd.3 (osd.3) 487 : cluster 0 6.b scrub starts 2024-03-11T21:44:16.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.261990+0000 osd.3 (osd.3) 488 : cluster 0 6.b scrub ok 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.574728+0000 osd.5 (osd.5) 507 : cluster 0 6.a scrub starts 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:14.576228+0000 osd.5 (osd.5) 508 : cluster 0 6.a scrub ok 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.170261+0000 osd.1 (osd.1) 297 : cluster 0 39.1c scrub starts 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.172862+0000 osd.1 (osd.1) 298 : cluster 0 39.1c scrub ok 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.259768+0000 osd.3 (osd.3) 487 : cluster 0 6.b scrub starts 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.261990+0000 osd.3 (osd.3) 488 : cluster 0 6.b scrub ok 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.574728+0000 osd.5 (osd.5) 507 : cluster 0 6.a scrub starts 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:14.576228+0000 osd.5 (osd.5) 508 : cluster 0 6.a scrub ok 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.170261+0000 osd.1 (osd.1) 297 : cluster 0 39.1c scrub starts 2024-03-11T21:44:16.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:15 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.172862+0000 osd.1 (osd.1) 298 : cluster 0 39.1c scrub ok 2024-03-11T21:44:16.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:16.237 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:16.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:16.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:16.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:16.501 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:16.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:16.632 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:16.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:16.758 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:16.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:16.887 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:16.912 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.924+0000 7fd111ffb700 1 -- 172.21.15.143:0/2225798662 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]=0 set osd.5 primary-affinity to 0 (802) v903) v1 ==== 129+0+0 (secure 0 0 0) 0x7fd10c010040 con 0x7fd114109350 2024-03-11T21:44:16.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 10 librados: watch_flush enter 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 10 librados: watch_flush exit 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 20 client.42167.objecter shutdown clearing up homeless session... 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 10 client.42167.objecter successfully canceled tick 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 msgr2=0x7fd104071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 0x7fd104071ce0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7fd1080037a0 tx=0x7fd108019040 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 msgr2=0x7fd11419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 0x7fd11419e6e0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00efb0 tx=0x7fd10c00c3a0 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 shutdown_connections 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd10406f820 0x7fd104071ce0 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd114104270 0x7fd114199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd114109350 0x7fd11419e6e0 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 --2- 172.21.15.143:0/2225798662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd114104c20 0x7fd11419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:16.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.928+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 >> 172.21.15.143:0/2225798662 conn(0x7fd1140ffac0 msgr2=0x7fd1141001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.932+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 shutdown_connections 2024-03-11T21:44:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.932+0000 7fd11b236700 1 -- 172.21.15.143:0/2225798662 wait complete. 2024-03-11T21:44:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:16.932+0000 7fd11b236700 1 librados: shutdown 2024-03-11T21:44:16.919 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.5 primary-affinity to 0 (802) 2024-03-11T21:44:16.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.011 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.260064+0000 osd.3 (osd.3) 489 : cluster 0 39.1b scrub starts 2024-03-11T21:44:17.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.265549+0000 osd.3 (osd.3) 490 : cluster 0 39.1b scrub ok 2024-03-11T21:44:17.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.603444+0000 osd.5 (osd.5) 509 : cluster 0 5.16 scrub starts 2024-03-11T21:44:17.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.605550+0000 osd.5 (osd.5) 510 : cluster 0 5.16 scrub ok 2024-03-11T21:44:17.012 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.677272+0000 mgr.y (mgr.14550) 823 : cluster 0 pgmap v1417: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.967835+0000 osd.4 (osd.4) 465 : cluster 0 3.3 scrub starts 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: audit 2024-03-11T21:44:15.968720+0000 mon.c (mon.2) 1370 : audit 1 from='client.? 172.21.15.143:0/2225798662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: audit 2024-03-11T21:44:15.969158+0000 mon.a (mon.0) 3979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:15.970628+0000 osd.4 (osd.4) 466 : cluster 0 3.3 scrub ok 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:16.209513+0000 osd.1 (osd.1) 299 : cluster 0 9.14 scrub starts 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[29284]: cluster 2024-03-11T21:44:16.211850+0000 osd.1 (osd.1) 300 : cluster 0 9.14 scrub ok 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.260064+0000 osd.3 (osd.3) 489 : cluster 0 39.1b scrub starts 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.265549+0000 osd.3 (osd.3) 490 : cluster 0 39.1b scrub ok 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.603444+0000 osd.5 (osd.5) 509 : cluster 0 5.16 scrub starts 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.605550+0000 osd.5 (osd.5) 510 : cluster 0 5.16 scrub ok 2024-03-11T21:44:17.013 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.677272+0000 mgr.y (mgr.14550) 823 : cluster 0 pgmap v1417: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.967835+0000 osd.4 (osd.4) 465 : cluster 0 3.3 scrub starts 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: audit 2024-03-11T21:44:15.968720+0000 mon.c (mon.2) 1370 : audit 1 from='client.? 172.21.15.143:0/2225798662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: audit 2024-03-11T21:44:15.969158+0000 mon.a (mon.0) 3979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:15.970628+0000 osd.4 (osd.4) 466 : cluster 0 3.3 scrub ok 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:16.209513+0000 osd.1 (osd.1) 299 : cluster 0 9.14 scrub starts 2024-03-11T21:44:17.014 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:16 smithi143 bash[21512]: cluster 2024-03-11T21:44:16.211850+0000 osd.1 (osd.1) 300 : cluster 0 9.14 scrub ok 2024-03-11T21:44:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.260064+0000 osd.3 (osd.3) 489 : cluster 0 39.1b scrub starts 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.265549+0000 osd.3 (osd.3) 490 : cluster 0 39.1b scrub ok 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.603444+0000 osd.5 (osd.5) 509 : cluster 0 5.16 scrub starts 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.605550+0000 osd.5 (osd.5) 510 : cluster 0 5.16 scrub ok 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.677272+0000 mgr.y (mgr.14550) 823 : cluster 0 pgmap v1417: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.967835+0000 osd.4 (osd.4) 465 : cluster 0 3.3 scrub starts 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: audit 2024-03-11T21:44:15.968720+0000 mon.c (mon.2) 1370 : audit 1 from='client.? 172.21.15.143:0/2225798662' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: audit 2024-03-11T21:44:15.969158+0000 mon.a (mon.0) 3979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:15.970628+0000 osd.4 (osd.4) 466 : cluster 0 3.3 scrub ok 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:16.209513+0000 osd.1 (osd.1) 299 : cluster 0 9.14 scrub starts 2024-03-11T21:44:17.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:16 smithi203 bash[23654]: cluster 2024-03-11T21:44:16.211850+0000 osd.1 (osd.1) 300 : cluster 0 9.14 scrub ok 2024-03-11T21:44:17.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.138 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:17.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.270 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:17.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.399 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:17.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:17.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.658 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:17.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.787 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:17.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.919 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:18.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:18.047 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:18.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.162 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: audit 2024-03-11T21:44:16.919646+0000 mon.a (mon.0) 3980 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-03-11T21:44:18.162 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:16.923660+0000 mon.a (mon.0) 3981 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-03-11T21:44:18.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: audit 2024-03-11T21:44:16.919646+0000 mon.a (mon.0) 3980 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-03-11T21:44:18.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:16.923660+0000 mon.a (mon.0) 3981 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-03-11T21:44:18.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:18.260 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:18.260 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:17 smithi203 bash[23654]: audit 2024-03-11T21:44:16.919646+0000 mon.a (mon.0) 3980 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-03-11T21:44:18.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:17 smithi203 bash[23654]: cluster 2024-03-11T21:44:16.923660+0000 mon.a (mon.0) 3981 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-03-11T21:44:18.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:18.307 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:18.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:18.437 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:18.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:18.568 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:18.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.699 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:18.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:18.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:18.932 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:18.959 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:19.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:19.092 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:19.125 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.177475+0000 osd.2 (osd.2) 423 : cluster 0 6.14 scrub starts 2024-03-11T21:44:19.125 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.179535+0000 osd.2 (osd.2) 424 : cluster 0 6.14 scrub ok 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.312490+0000 osd.3 (osd.3) 491 : cluster 0 5.17 scrub starts 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.313862+0000 osd.3 (osd.3) 492 : cluster 0 5.17 scrub ok 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.495300+0000 osd.6 (osd.6) 407 : cluster 0 3.10 scrub starts 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.497742+0000 osd.6 (osd.6) 408 : cluster 0 3.10 scrub ok 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.678050+0000 mgr.y (mgr.14550) 824 : cluster 0 pgmap v1419: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:19.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:18 smithi203 bash[23654]: cluster 2024-03-11T21:44:17.934105+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-03-11T21:44:19.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:44:19.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.177475+0000 osd.2 (osd.2) 423 : cluster 0 6.14 scrub starts 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.179535+0000 osd.2 (osd.2) 424 : cluster 0 6.14 scrub ok 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.312490+0000 osd.3 (osd.3) 491 : cluster 0 5.17 scrub starts 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.313862+0000 osd.3 (osd.3) 492 : cluster 0 5.17 scrub ok 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.495300+0000 osd.6 (osd.6) 407 : cluster 0 3.10 scrub starts 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.497742+0000 osd.6 (osd.6) 408 : cluster 0 3.10 scrub ok 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.678050+0000 mgr.y (mgr.14550) 824 : cluster 0 pgmap v1419: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[29284]: cluster 2024-03-11T21:44:17.934105+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.177475+0000 osd.2 (osd.2) 423 : cluster 0 6.14 scrub starts 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.179535+0000 osd.2 (osd.2) 424 : cluster 0 6.14 scrub ok 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.312490+0000 osd.3 (osd.3) 491 : cluster 0 5.17 scrub starts 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.313862+0000 osd.3 (osd.3) 492 : cluster 0 5.17 scrub ok 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.495300+0000 osd.6 (osd.6) 407 : cluster 0 3.10 scrub starts 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.497742+0000 osd.6 (osd.6) 408 : cluster 0 3.10 scrub ok 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.678050+0000 mgr.y (mgr.14550) 824 : cluster 0 pgmap v1419: 216 pgs: 216 active+clean; 582 KiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:18 smithi143 bash[21512]: cluster 2024-03-11T21:44:17.934105+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-03-11T21:44:19.218 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:19.281 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:19.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.342 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:19.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:19.469 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:19.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:19.600 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:19.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:19.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:19.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.864 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:19.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:19.991 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:20.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.156190+0000 osd.2 (osd.2) 425 : cluster 0 1.a scrub starts 2024-03-11T21:44:20.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.158218+0000 osd.2 (osd.2) 426 : cluster 0 1.a scrub ok 2024-03-11T21:44:20.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.209679+0000 osd.0 (osd.0) 480 : cluster 0 1.1 scrub starts 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.210884+0000 osd.0 (osd.0) 481 : cluster 0 1.1 scrub ok 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.232983+0000 osd.1 (osd.1) 301 : cluster 0 1.12 scrub starts 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.234947+0000 osd.1 (osd.1) 302 : cluster 0 1.12 scrub ok 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.292363+0000 osd.3 (osd.3) 493 : cluster 0 6.6 scrub starts 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.293919+0000 osd.3 (osd.3) 494 : cluster 0 6.6 scrub ok 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.378055+0000 osd.7 (osd.7) 478 : cluster 0 1.10 scrub starts 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.379113+0000 osd.7 (osd.7) 479 : cluster 0 1.10 scrub ok 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:18.945210+0000 mon.a (mon.0) 3983 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: audit 2024-03-11T21:44:19.141681+0000 mgr.y (mgr.14550) 825 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.211915+0000 osd.1 (osd.1) 303 : cluster 0 5.15 scrub starts 2024-03-11T21:44:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:19 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.214220+0000 osd.1 (osd.1) 304 : cluster 0 5.15 scrub ok 2024-03-11T21:44:20.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.121 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.156190+0000 osd.2 (osd.2) 425 : cluster 0 1.a scrub starts 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.158218+0000 osd.2 (osd.2) 426 : cluster 0 1.a scrub ok 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.209679+0000 osd.0 (osd.0) 480 : cluster 0 1.1 scrub starts 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.210884+0000 osd.0 (osd.0) 481 : cluster 0 1.1 scrub ok 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.232983+0000 osd.1 (osd.1) 301 : cluster 0 1.12 scrub starts 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.234947+0000 osd.1 (osd.1) 302 : cluster 0 1.12 scrub ok 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.292363+0000 osd.3 (osd.3) 493 : cluster 0 6.6 scrub starts 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.293919+0000 osd.3 (osd.3) 494 : cluster 0 6.6 scrub ok 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.378055+0000 osd.7 (osd.7) 478 : cluster 0 1.10 scrub starts 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.379113+0000 osd.7 (osd.7) 479 : cluster 0 1.10 scrub ok 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:18.945210+0000 mon.a (mon.0) 3983 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: audit 2024-03-11T21:44:19.141681+0000 mgr.y (mgr.14550) 825 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:20.163 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.211915+0000 osd.1 (osd.1) 303 : cluster 0 5.15 scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.214220+0000 osd.1 (osd.1) 304 : cluster 0 5.15 scrub ok 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.156190+0000 osd.2 (osd.2) 425 : cluster 0 1.a scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.158218+0000 osd.2 (osd.2) 426 : cluster 0 1.a scrub ok 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.209679+0000 osd.0 (osd.0) 480 : cluster 0 1.1 scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.210884+0000 osd.0 (osd.0) 481 : cluster 0 1.1 scrub ok 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.232983+0000 osd.1 (osd.1) 301 : cluster 0 1.12 scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.234947+0000 osd.1 (osd.1) 302 : cluster 0 1.12 scrub ok 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.292363+0000 osd.3 (osd.3) 493 : cluster 0 6.6 scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.293919+0000 osd.3 (osd.3) 494 : cluster 0 6.6 scrub ok 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.378055+0000 osd.7 (osd.7) 478 : cluster 0 1.10 scrub starts 2024-03-11T21:44:20.164 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.379113+0000 osd.7 (osd.7) 479 : cluster 0 1.10 scrub ok 2024-03-11T21:44:20.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:18.945210+0000 mon.a (mon.0) 3983 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-03-11T21:44:20.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: audit 2024-03-11T21:44:19.141681+0000 mgr.y (mgr.14550) 825 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:20.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.211915+0000 osd.1 (osd.1) 303 : cluster 0 5.15 scrub starts 2024-03-11T21:44:20.165 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:19 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.214220+0000 osd.1 (osd.1) 304 : cluster 0 5.15 scrub ok 2024-03-11T21:44:20.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:20.256 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:20.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:20.387 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:20.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.517 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:20.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:20.648 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:20.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:20.777 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:20.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:20.910 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:21.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.142 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:21.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.120566+0000 osd.2 (osd.2) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.122721+0000 osd.2 (osd.2) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.678905+0000 mgr.y (mgr.14550) 826 : cluster 0 pgmap v1422: 216 pgs: 216 active+clean; 378 KiB data, 757 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:19.950617+0000 mon.a (mon.0) 3984 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:20.291918+0000 osd.7 (osd.7) 480 : cluster 0 6.3 scrub starts 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[21512]: cluster 2024-03-11T21:44:20.293970+0000 osd.7 (osd.7) 481 : cluster 0 6.3 scrub ok 2024-03-11T21:44:21.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.120566+0000 osd.2 (osd.2) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:44:21.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.122721+0000 osd.2 (osd.2) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:44:21.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.678905+0000 mgr.y (mgr.14550) 826 : cluster 0 pgmap v1422: 216 pgs: 216 active+clean; 378 KiB data, 757 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:44:21.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:19.950617+0000 mon.a (mon.0) 3984 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-03-11T21:44:21.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:20.291918+0000 osd.7 (osd.7) 480 : cluster 0 6.3 scrub starts 2024-03-11T21:44:21.154 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:20 smithi143 bash[29284]: cluster 2024-03-11T21:44:20.293970+0000 osd.7 (osd.7) 481 : cluster 0 6.3 scrub ok 2024-03-11T21:44:21.166 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:21.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:21.284 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.120566+0000 osd.2 (osd.2) 427 : cluster 0 1.8 scrub starts 2024-03-11T21:44:21.285 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.122721+0000 osd.2 (osd.2) 428 : cluster 0 1.8 scrub ok 2024-03-11T21:44:21.285 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.678905+0000 mgr.y (mgr.14550) 826 : cluster 0 pgmap v1422: 216 pgs: 216 active+clean; 378 KiB data, 757 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:44:21.285 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:19.950617+0000 mon.a (mon.0) 3984 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-03-11T21:44:21.285 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:20.291918+0000 osd.7 (osd.7) 480 : cluster 0 6.3 scrub starts 2024-03-11T21:44:21.285 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:20 smithi203 bash[23654]: cluster 2024-03-11T21:44:20.293970+0000 osd.7 (osd.7) 481 : cluster 0 6.3 scrub ok 2024-03-11T21:44:21.289 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:21.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.428 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.558 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:21.688 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:21.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:21.818 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:21.921 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:22.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:22.080 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:22.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.212 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:21 smithi203 bash[23654]: cluster 2024-03-11T21:44:20.961230+0000 mon.a (mon.0) 3985 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-03-11T21:44:22.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:21 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.336718+0000 osd.7 (osd.7) 482 : cluster 0 5.10 scrub starts 2024-03-11T21:44:22.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:21 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.616920+0000 mon.a (mon.0) 3986 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-03-11T21:44:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:22.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.341 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:22.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.445 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:20.961230+0000 mon.a (mon.0) 3985 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.336718+0000 osd.7 (osd.7) 482 : cluster 0 5.10 scrub starts 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.616920+0000 mon.a (mon.0) 3986 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:20.961230+0000 mon.a (mon.0) 3985 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.336718+0000 osd.7 (osd.7) 482 : cluster 0 5.10 scrub starts 2024-03-11T21:44:22.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.616920+0000 mon.a (mon.0) 3986 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-03-11T21:44:22.472 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:22.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:22.612 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:22.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- 172.21.15.143:0/2437252449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c10e810 msgr2=0x7fc44c110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- 172.21.15.143:0/2437252449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c10e810 0x7fc44c110c00 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7fc44400b0a0 tx=0x7fc44400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- 172.21.15.143:0/2437252449 shutdown_connections 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- 172.21.15.143:0/2437252449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c10e810 0x7fc44c110c00 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- 172.21.15.143:0/2437252449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c074010 0x7fc44c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- 172.21.15.143:0/2437252449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074930 0x7fc44c10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- 172.21.15.143:0/2437252449 >> 172.21.15.143:0/2437252449 conn(0x7fc44c078f00 msgr2=0x7fc44c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- 172.21.15.143:0/2437252449 shutdown_connections 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- 172.21.15.143:0/2437252449 wait complete. 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 librados: starting msgr at 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 librados: starting objecter 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 Processor -- start 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- start start 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 librados: setting wanted keys 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 librados: calling monclient init 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 0x7fc44c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c074930 0x7fc44c19ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c10e810 0x7fc44c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c1137c0 con 0x7fc44c074010 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c113640 con 0x7fc44c10e810 2024-03-11T21:44:22.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc452bdc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c113940 con 0x7fc44c074930 2024-03-11T21:44:22.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc450978700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 0x7fc44c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc450978700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 0x7fc44c19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35246/0 (socket says 172.21.15.143:35246) 2024-03-11T21:44:22.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc450978700 1 -- 172.21.15.143:0/1424208914 learned_addr learned my addr 172.21.15.143:0/1424208914 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:22.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.748+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c074930 0x7fc44c19ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c074930 msgr2=0x7fc44c19ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc44c074930 0x7fc44c19ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c10e810 0x7fc44c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c10e810 msgr2=0x7fc44c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c10e810 0x7fc44c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c19a410 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc44c10e810 0x7fc44c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 0x7fc44c19fa60 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fc44000d950 tx=0x7fc44000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 10 client.?.objecter ms_handle_connect 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440016070 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c19a700 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc440009dc0 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000f3c0 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 10 client.30975.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c1a69e0 con 0x7fc44c074010 2024-03-11T21:44:22.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 librados: init done 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc44c075700 con 0x7fc44c074010 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc440019070 con 0x7fc44c074010 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc43c06f820 0x7fc43c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(909..909 src has 842..909) v4 ==== 8964+0+0 (secure 0 0 0) 0x7fc44008e950 con 0x7fc44c074010 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 osd_map(909..909 src has 842..909) v4 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 3 client.30975.objecter handle_osd_map got epochs [909,909] > 0 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 3 client.30975.objecter handle_osd_map decoding full epoch 909 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc43c06f820 0x7fc43c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 20 client.30975.objecter dump_active .. 0 homeless 2024-03-11T21:44:22.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v909) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc44008f080 con 0x7fc44c074010 2024-03-11T21:44:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 10 client.30975.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 1 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc44c062c40 0x7fc44c107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 1 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 0 1 2024-03-11T21:44:22.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 4 2024-03-11T21:44:22.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 10 client.30975.objecter _send_command 1 2024-03-11T21:44:22.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c19aff0 con 0x7fc44c062c40 2024-03-11T21:44:22.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc43c06f820 0x7fc43c071ce0 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fc438009680 tx=0x7fc438009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc43c06f820 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.752+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc44c062c40 0x7fc44c107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c062c40 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc44c062c40 0x7fc44c107990 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c19aff0 con 0x7fc44c062c40 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 10 client.30975.objecter _finish_command 1 = system:0 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 4 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.756+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 0 1 2024-03-11T21:44:22.747 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 10 client.30975.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 2 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 2 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c04faa0 osd=0 3 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 0 2 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 4 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 10 client.30975.objecter _send_command 2 2024-03-11T21:44:22.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.772+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c107ed0 con 0x7fc44c062c40 2024-03-11T21:44:22.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c107ed0 con 0x7fc44c062c40 2024-03-11T21:44:22.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 10 client.30975.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 4 2024-03-11T21:44:22.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 0 2 2024-03-11T21:44:22.763 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 10 client.30975.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc44c02ce80 0x7fc44c02d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 1 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 4 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 10 client.30975.objecter _send_command 3 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1ad4f0 con 0x7fc44c02ce80 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc44c02ce80 0x7fc44c02d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c02ce80 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc44c02ce80 0x7fc44c02d280 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1ad4f0 con 0x7fc44c02ce80 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 10 client.30975.objecter _finish_command 3 = system:0 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 4 2024-03-11T21:44:22.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.776+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 1 3 2024-03-11T21:44:22.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 10 client.30975.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 4 2024-03-11T21:44:22.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 4 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c02dd40 osd=1 3 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 1 4 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 4 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 10 client.30975.objecter _send_command 4 2024-03-11T21:44:22.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.792+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c02c900 con 0x7fc44c02ce80 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c02c900 con 0x7fc44c02ce80 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 10 client.30975.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 4 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 1 4 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 10 client.30975.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 5 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc44c1abd50 0x7fc44c1ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 5 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.784 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 2 5 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 4 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 10 client.30975.objecter _send_command 5 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c108fc0 con 0x7fc44c1abd50 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc44c1abd50 0x7fc44c1ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1abd50 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc44c1abd50 0x7fc44c1ac170 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c108fc0 con 0x7fc44c1abd50 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 10 client.30975.objecter _finish_command 5 = system:0 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 4 2024-03-11T21:44:22.785 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.796+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 2 5 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 10 client.30975.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 6 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 6 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1088b0 osd=2 3 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 2 6 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 4 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 10 client.30975.objecter _send_command 6 2024-03-11T21:44:22.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.812+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c108330 con 0x7fc44c1abd50 2024-03-11T21:44:22.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c108330 con 0x7fc44c1abd50 2024-03-11T21:44:22.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 10 client.30975.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 4 2024-03-11T21:44:22.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 2 6 2024-03-11T21:44:22.804 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:44:22.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 10 client.30975.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 7 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc44c1af6b0 0x7fc44c1b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 7 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 3 7 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 4 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 10 client.30975.objecter _send_command 7 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1b4a70 con 0x7fc44c1af6b0 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc44c1af6b0 0x7fc44c1b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1af6b0 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc44c1af6b0 0x7fc44c1b1aa0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1b4a70 con 0x7fc44c1af6b0 2024-03-11T21:44:22.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:44:22.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 10 client.30975.objecter _finish_command 7 = system:0 2024-03-11T21:44:22.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 4 2024-03-11T21:44:22.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.816+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 3 7 2024-03-11T21:44:22.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 10 client.30975.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 8 2024-03-11T21:44:22.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 8 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c108730 osd=3 3 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 3 8 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 4 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 10 client.30975.objecter _send_command 8 2024-03-11T21:44:22.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c1ac900 con 0x7fc44c1af6b0 2024-03-11T21:44:22.824 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:44:22.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c1ac900 con 0x7fc44c1af6b0 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 10 client.30975.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 4 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 3 8 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 10 client.30975.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 9 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc44c1ab8b0 0x7fc44c1b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 9 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 4 9 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 4 2024-03-11T21:44:22.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 10 client.30975.objecter _send_command 9 2024-03-11T21:44:22.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1ac820 con 0x7fc44c1ab8b0 2024-03-11T21:44:22.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc44c1ab8b0 0x7fc44c1b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1ab8b0 2024-03-11T21:44:22.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.836+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc44c1ab8b0 0x7fc44c1b2030 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.840+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1ac820 con 0x7fc44c1ab8b0 2024-03-11T21:44:22.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.840+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:44:22.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.840+0000 7fc449ffb700 10 client.30975.objecter _finish_command 9 = system:0 2024-03-11T21:44:22.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.840+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 4 2024-03-11T21:44:22.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.840+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 4 9 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 10 client.30975.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 10 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 10 2024-03-11T21:44:22.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1ac6b0 osd=4 3 2024-03-11T21:44:22.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 4 10 2024-03-11T21:44:22.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 4 2024-03-11T21:44:22.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 10 client.30975.objecter _send_command 10 2024-03-11T21:44:22.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.856+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c1ac820 con 0x7fc44c1ab8b0 2024-03-11T21:44:22.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c1ac820 con 0x7fc44c1ab8b0 2024-03-11T21:44:22.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 10 client.30975.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 4 2024-03-11T21:44:22.848 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 4 10 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 10 client.30975.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 11 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc44c1b2a80 0x7fc44c1b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.849 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 11 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 5 11 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 4 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 10 client.30975.objecter _send_command 11 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1b3620 con 0x7fc44c1b2a80 2024-03-11T21:44:22.850 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc44c1b2a80 0x7fc44c1b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1b2a80 2024-03-11T21:44:22.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.860+0000 7fc44bfff700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc44c1b2a80 0x7fc44c1b2f00 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.864+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1b3620 con 0x7fc44c1b2a80 2024-03-11T21:44:22.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.864+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:44:22.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.864+0000 7fc449ffb700 10 client.30975.objecter _finish_command 11 = system:0 2024-03-11T21:44:22.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.864+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 4 2024-03-11T21:44:22.852 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.864+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 5 11 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 10 client.30975.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 12 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 12 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1ac820 osd=5 3 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 5 12 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 4 2024-03-11T21:44:22.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 10 client.30975.objecter _send_command 12 2024-03-11T21:44:22.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.880+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c1b8cd0 con 0x7fc44c1b2a80 2024-03-11T21:44:22.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c1b8cd0 con 0x7fc44c1b2a80 2024-03-11T21:44:22.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc449ffb700 10 client.30975.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 4 2024-03-11T21:44:22.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 5 12 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 10 client.30975.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 13 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc44c1b9350 0x7fc44c1b97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 13 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 6 13 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 4 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 10 client.30975.objecter _send_command 13 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1be7f0 con 0x7fc44c1b9350 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.884+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc44c1b9350 0x7fc44c1b97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc451179700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc44c1b9350 0x7fc44c1b97b0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1b9350 2024-03-11T21:44:22.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1be7f0 con 0x7fc44c1b9350 2024-03-11T21:44:22.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:44:22.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 10 client.30975.objecter _finish_command 13 = system:0 2024-03-11T21:44:22.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 4 2024-03-11T21:44:22.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.888+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 6 13 2024-03-11T21:44:22.877 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:22.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 10 client.30975.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 14 2024-03-11T21:44:22.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 14 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1b8cd0 osd=6 3 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 6 14 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 4 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 10 client.30975.objecter _send_command 14 2024-03-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.904+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c1bd630 con 0x7fc44c1b9350 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c1bd630 con 0x7fc44c1b9350 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 10 client.30975.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 4 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 6 14 2024-03-11T21:44:22.896 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:44:22.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 10 client.30975.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:22.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 15 2024-03-11T21:44:22.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc44c1bdcb0 0x7fc44c1be0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:22.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.897 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 15 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 7 15 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 4 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 10 client.30975.objecter _send_command 15 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c1c1b60 con 0x7fc44c1bdcb0 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc44c1bdcb0 0x7fc44c1be0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 10 client.30975.objecter ms_handle_connect 0x7fc44c1bdcb0 2024-03-11T21:44:22.898 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc450978700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc44c1bdcb0 0x7fc44c1be0f0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:22.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc44c1c1b60 con 0x7fc44c1bdcb0 2024-03-11T21:44:22.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:44:22.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 10 client.30975.objecter _finish_command 15 = system:0 2024-03-11T21:44:22.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 4 2024-03-11T21:44:22.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.908+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 7 15 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign -1 16 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter _get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_remove -1 16 2024-03-11T21:44:22.915 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter get_session s=0x7fc44c1bd630 osd=7 3 2024-03-11T21:44:22.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 15 client.30975.objecter _session_command_op_assign 7 16 2024-03-11T21:44:22.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 4 2024-03-11T21:44:22.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter _send_command 16 2024-03-11T21:44:22.916 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc44c1b9d40 con 0x7fc44c1bdcb0 2024-03-11T21:44:22.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc449ffb700 1 -- 172.21.15.143:0/1424208914 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc44c1b9d40 con 0x7fc44c1bdcb0 2024-03-11T21:44:22.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc449ffb700 10 client.30975.objecter ms_dispatch 0x7fc44c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:22.917 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc449ffb700 10 client.30975.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:22.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc449ffb700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 4 2024-03-11T21:44:22.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc449ffb700 15 client.30975.objecter _session_command_op_remove 7 16 2024-03-11T21:44:22.918 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:44:22.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 librados: watch_flush enter 2024-03-11T21:44:22.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 librados: watch_flush exit 2024-03-11T21:44:22.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.0 2024-03-11T21:44:22.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc44c062c40 msgr2=0x7fc44c107990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fc44c062c40 0x7fc44c107990 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c04faa0 osd=0 1 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.1 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc44c02ce80 msgr2=0x7fc44c02d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fc44c02ce80 0x7fc44c02d280 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c02dd40 osd=1 1 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.2 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc44c1abd50 msgr2=0x7fc44c1ac170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fc44c1abd50 0x7fc44c1ac170 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1088b0 osd=2 1 2024-03-11T21:44:22.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.3 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc44c1af6b0 msgr2=0x7fc44c1b1aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fc44c1af6b0 0x7fc44c1b1aa0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c108730 osd=3 1 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.4 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc451179700 1 -- 172.21.15.143:0/1424208914 reap_dead start 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc44c1ab8b0 msgr2=0x7fc44c1b2030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fc44c1ab8b0 0x7fc44c1b2030 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc451179700 1 -- 172.21.15.143:0/1424208914 reap_dead start 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac6b0 osd=4 1 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.5 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc44c1b2a80 msgr2=0x7fc44c1b2f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fc44c1b2a80 0x7fc44c1b2f00 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1ac820 osd=5 1 2024-03-11T21:44:22.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.6 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc44c1b9350 msgr2=0x7fc44c1b97b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fc44c1b9350 0x7fc44c1b97b0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1b8cd0 osd=6 1 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter close_session for osd.7 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc44c1bdcb0 msgr2=0x7fc44c1be0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fc44c1bdcb0 0x7fc44c1be0f0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:22.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter put_session s=0x7fc44c1bd630 osd=7 1 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 20 client.30975.objecter shutdown clearing up homeless session... 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 10 client.30975.objecter successfully canceled tick 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc43c06f820 msgr2=0x7fc43c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc43c06f820 0x7fc43c071ce0 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fc438009680 tx=0x7fc438009260 comp rx=0 tx=0).stop 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 msgr2=0x7fc44c19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.928+0000 7fc452bdc700 1 --2- 172.21.15.143:0/1424208914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c074010 0x7fc44c19fa60 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fc44000d950 tx=0x7fc44000dc60 comp rx=0 tx=0).stop 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc451179700 1 -- 172.21.15.143:0/1424208914 reap_dead start 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 shutdown_connections 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 >> 172.21.15.143:0/1424208914 conn(0x7fc44c078f00 msgr2=0x7fc44c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 shutdown_connections 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc452bdc700 1 -- 172.21.15.143:0/1424208914 wait complete. 2024-03-11T21:44:22.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:22.932+0000 7fc452bdc700 1 librados: shutdown 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:22.928 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:22.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.003 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.136 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.289752+0000 osd.0 (osd.0) 482 : cluster 0 6.7 scrub starts 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.291780+0000 osd.0 (osd.0) 483 : cluster 0 6.7 scrub ok 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.302381+0000 osd.3 (osd.3) 495 : cluster 0 6.c scrub starts 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.335724+0000 osd.3 (osd.3) 496 : cluster 0 6.c scrub ok 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.338886+0000 osd.7 (osd.7) 483 : cluster 0 5.10 scrub ok 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:21.679706+0000 mgr.y (mgr.14550) 827 : cluster 0 pgmap v1426: 216 pgs: 32 unknown, 184 active+clean; 378 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 7.2 KiB/s rd, 58 KiB/s wr, 9 op/s; 42 KiB/s, 0 objects/s recovering 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:22.328303+0000 osd.7 (osd.7) 484 : cluster 0 5.4 scrub starts 2024-03-11T21:44:23.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:22.330712+0000 osd.7 (osd.7) 485 : cluster 0 5.4 scrub ok 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: audit 2024-03-11T21:44:22.611746+0000 mon.c (mon.2) 1371 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: cluster 2024-03-11T21:44:22.617028+0000 mon.a (mon.0) 3987 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[21512]: audit 2024-03-11T21:44:22.758056+0000 mon.a (mon.0) 3988 : audit 0 from='client.? 172.21.15.143:0/1424208914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.289752+0000 osd.0 (osd.0) 482 : cluster 0 6.7 scrub starts 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.291780+0000 osd.0 (osd.0) 483 : cluster 0 6.7 scrub ok 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.302381+0000 osd.3 (osd.3) 495 : cluster 0 6.c scrub starts 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.335724+0000 osd.3 (osd.3) 496 : cluster 0 6.c scrub ok 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.338886+0000 osd.7 (osd.7) 483 : cluster 0 5.10 scrub ok 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:21.679706+0000 mgr.y (mgr.14550) 827 : cluster 0 pgmap v1426: 216 pgs: 32 unknown, 184 active+clean; 378 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 7.2 KiB/s rd, 58 KiB/s wr, 9 op/s; 42 KiB/s, 0 objects/s recovering 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:22.328303+0000 osd.7 (osd.7) 484 : cluster 0 5.4 scrub starts 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:22.330712+0000 osd.7 (osd.7) 485 : cluster 0 5.4 scrub ok 2024-03-11T21:44:23.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: audit 2024-03-11T21:44:22.611746+0000 mon.c (mon.2) 1371 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: cluster 2024-03-11T21:44:22.617028+0000 mon.a (mon.0) 3987 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-03-11T21:44:23.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:22 smithi143 bash[29284]: audit 2024-03-11T21:44:22.758056+0000 mon.a (mon.0) 3988 : audit 0 from='client.? 172.21.15.143:0/1424208914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:23.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.264 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.289752+0000 osd.0 (osd.0) 482 : cluster 0 6.7 scrub starts 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.291780+0000 osd.0 (osd.0) 483 : cluster 0 6.7 scrub ok 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.302381+0000 osd.3 (osd.3) 495 : cluster 0 6.c scrub starts 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.335724+0000 osd.3 (osd.3) 496 : cluster 0 6.c scrub ok 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.338886+0000 osd.7 (osd.7) 483 : cluster 0 5.10 scrub ok 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:21.679706+0000 mgr.y (mgr.14550) 827 : cluster 0 pgmap v1426: 216 pgs: 32 unknown, 184 active+clean; 378 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 7.2 KiB/s rd, 58 KiB/s wr, 9 op/s; 42 KiB/s, 0 objects/s recovering 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:22.328303+0000 osd.7 (osd.7) 484 : cluster 0 5.4 scrub starts 2024-03-11T21:44:23.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:22.330712+0000 osd.7 (osd.7) 485 : cluster 0 5.4 scrub ok 2024-03-11T21:44:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: audit 2024-03-11T21:44:22.611746+0000 mon.c (mon.2) 1371 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: cluster 2024-03-11T21:44:22.617028+0000 mon.a (mon.0) 3987 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-03-11T21:44:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:22 smithi203 bash[23654]: audit 2024-03-11T21:44:22.758056+0000 mon.a (mon.0) 3988 : audit 0 from='client.? 172.21.15.143:0/1424208914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:23.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:23.394 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:23.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:23.523 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:23.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:23.652 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:23.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:23.783 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:23.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:23.891 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:44:23.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:24.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:24.044 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:24.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:24.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:24.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:24.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:23 smithi203 bash[23654]: cluster 2024-03-11T21:44:22.284910+0000 osd.0 (osd.0) 484 : cluster 0 5.1b scrub starts 2024-03-11T21:44:24.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:23 smithi203 bash[23654]: cluster 2024-03-11T21:44:22.287299+0000 osd.0 (osd.0) 485 : cluster 0 5.1b scrub ok 2024-03-11T21:44:24.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:23 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.620295+0000 mon.a (mon.0) 3989 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[29284]: cluster 2024-03-11T21:44:22.284910+0000 osd.0 (osd.0) 484 : cluster 0 5.1b scrub starts 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[29284]: cluster 2024-03-11T21:44:22.287299+0000 osd.0 (osd.0) 485 : cluster 0 5.1b scrub ok 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.620295+0000 mon.a (mon.0) 3989 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[21512]: cluster 2024-03-11T21:44:22.284910+0000 osd.0 (osd.0) 484 : cluster 0 5.1b scrub starts 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[21512]: cluster 2024-03-11T21:44:22.287299+0000 osd.0 (osd.0) 485 : cluster 0 5.1b scrub ok 2024-03-11T21:44:24.287 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:23 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.620295+0000 mon.a (mon.0) 3989 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-03-11T21:44:24.306 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:24.384 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:44:24.385 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:24.385 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-11T21:44:24.385 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:44:24.385 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:44:24.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-11T21:44:24.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.436 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:24.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:24.563 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:24.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.685 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:24.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- 172.21.15.143:0/692263555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710109370 msgr2=0x7fb710110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- 172.21.15.143:0/692263555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710109370 0x7fb710110f40 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fb708009fc0 tx=0x7fb70800b330 comp rx=0 tx=0).stop 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- 172.21.15.143:0/692263555 shutdown_connections 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- 172.21.15.143:0/692263555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710104290 0x7fb710104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- 172.21.15.143:0/692263555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710109370 0x7fb710110f40 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- 172.21.15.143:0/692263555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb710104c40 0x7fb710108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- 172.21.15.143:0/692263555 >> 172.21.15.143:0/692263555 conn(0x7fb7100ffb40 msgr2=0x7fb710101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- 172.21.15.143:0/692263555 shutdown_connections 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- 172.21.15.143:0/692263555 wait complete. 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 librados: starting msgr at 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 librados: starting objecter 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 Processor -- start 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- start start 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 librados: setting wanted keys 2024-03-11T21:44:24.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 librados: calling monclient init 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb710104c40 0x7fb71019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 0x7fb71019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb710113870 con 0x7fb710109370 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101136f0 con 0x7fb710104c40 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb717ce9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101139f0 con 0x7fb710104290 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:53304/0 (socket says 172.21.15.143:53304) 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 -- 172.21.15.143:0/2117466497 learned_addr learned my addr 172.21.15.143:0/2117466497 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb716286700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 0x7fb71019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 -- 172.21.15.143:0/2117466497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb710104c40 msgr2=0x7fb71019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:24.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb710104c40 0x7fb71019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:24.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 -- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 msgr2=0x7fb71019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:24.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.764+0000 7fb715a85700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 0x7fb71019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:24.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb715a85700 1 -- 172.21.15.143:0/2117466497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71019ed90 con 0x7fb710104290 2024-03-11T21:44:24.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb716286700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 0x7fb71019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb715a85700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00ec10 tx=0x7fb70c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 10 client.?.objecter ms_handle_connect 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c00cd40 con 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71019f080 con 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb70c0044a0 con 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c010430 con 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb717ce9700 10 client.42185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7101a6900 con 0x7fb710104290 2024-03-11T21:44:24.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb717ce9700 1 librados: init done 2024-03-11T21:44:24.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71004faa0 con 0x7fb710104290 2024-03-11T21:44:24.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb70c004000 con 0x7fb710104290 2024-03-11T21:44:24.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 0x7fb6fc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(911..911 src has 842..911) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb70c08eb90 con 0x7fb710104290 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 10 client.42185.objecter ms_dispatch 0x7fb710001550 osd_map(911..911 src has 842..911) v4 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 3 client.42185.objecter handle_osd_map got epochs [911,911] > 0 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 3 client.42185.objecter handle_osd_map decoding full epoch 911 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb715284700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 0x7fb6fc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 20 client.42185.objecter dump_active .. 0 homeless 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb706ffd700 10 client.42185.objecter ms_handle_connect 0x7fb6fc06f820 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.768+0000 7fb715284700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 0x7fb6fc071ce0 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7fb710101de0 tx=0x7fb700005bf0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:24.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.772+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb70c05bac0 con 0x7fb710104290 2024-03-11T21:44:24.788 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.810 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:24.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:24.936 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:24.984 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:24.996+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb7101a6b90 con 0x7fb710104290 2024-03-11T21:44:25.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:25.061 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:25.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:25.193 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:25.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.161438+0000 osd.2 (osd.2) 429 : cluster 0 6.f scrub starts 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.163122+0000 osd.2 (osd.2) 430 : cluster 0 6.f scrub ok 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.256790+0000 osd.0 (osd.0) 486 : cluster 0 5.3 scrub starts 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.259027+0000 osd.0 (osd.0) 487 : cluster 0 5.3 scrub ok 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:23.680637+0000 mgr.y (mgr.14550) 828 : cluster 0 pgmap v1429: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:24.009876+0000 osd.4 (osd.4) 467 : cluster 0 5.e scrub starts 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:24.011978+0000 osd.4 (osd.4) 468 : cluster 0 5.e scrub ok 2024-03-11T21:44:25.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[21512]: cluster 2024-03-11T21:44:24.626756+0000 mon.a (mon.0) 3990 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.161438+0000 osd.2 (osd.2) 429 : cluster 0 6.f scrub starts 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.163122+0000 osd.2 (osd.2) 430 : cluster 0 6.f scrub ok 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.256790+0000 osd.0 (osd.0) 486 : cluster 0 5.3 scrub starts 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.259027+0000 osd.0 (osd.0) 487 : cluster 0 5.3 scrub ok 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:23.680637+0000 mgr.y (mgr.14550) 828 : cluster 0 pgmap v1429: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:24.009876+0000 osd.4 (osd.4) 467 : cluster 0 5.e scrub starts 2024-03-11T21:44:25.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:24.011978+0000 osd.4 (osd.4) 468 : cluster 0 5.e scrub ok 2024-03-11T21:44:25.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:24 smithi143 bash[29284]: cluster 2024-03-11T21:44:24.626756+0000 mon.a (mon.0) 3990 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-03-11T21:44:25.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.161438+0000 osd.2 (osd.2) 429 : cluster 0 6.f scrub starts 2024-03-11T21:44:25.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.163122+0000 osd.2 (osd.2) 430 : cluster 0 6.f scrub ok 2024-03-11T21:44:25.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.256790+0000 osd.0 (osd.0) 486 : cluster 0 5.3 scrub starts 2024-03-11T21:44:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.259027+0000 osd.0 (osd.0) 487 : cluster 0 5.3 scrub ok 2024-03-11T21:44:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:23.680637+0000 mgr.y (mgr.14550) 828 : cluster 0 pgmap v1429: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:24.009876+0000 osd.4 (osd.4) 467 : cluster 0 5.e scrub starts 2024-03-11T21:44:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:24.011978+0000 osd.4 (osd.4) 468 : cluster 0 5.e scrub ok 2024-03-11T21:44:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:24 smithi203 bash[23654]: cluster 2024-03-11T21:44:24.626756+0000 mon.a (mon.0) 3990 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-03-11T21:44:25.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.322 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:25.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:25.448 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:25.552 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.581 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:25.613 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.624+0000 7fb706ffd700 1 -- 172.21.15.143:0/2117466497 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v912) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb70c05f510 con 0x7fb710104290 2024-03-11T21:44:25.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.628+0000 7fb717ce9700 10 librados: watch_flush enter 2024-03-11T21:44:25.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.628+0000 7fb717ce9700 10 librados: watch_flush exit 2024-03-11T21:44:25.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.628+0000 7fb717ce9700 20 client.42185.objecter shutdown clearing up homeless session... 2024-03-11T21:44:25.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.628+0000 7fb717ce9700 10 client.42185.objecter successfully canceled tick 2024-03-11T21:44:25.629 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 msgr2=0x7fb6fc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 0x7fb6fc071ce0 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7fb710101de0 tx=0x7fb700005bf0 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 msgr2=0x7fb710199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00ec10 tx=0x7fb70c00ef20 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 shutdown_connections 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb6fc06f820 0x7fb6fc071ce0 unknown :-1 s=CLOSED pgs=3475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb710109370 0x7fb71019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb710104290 0x7fb710199d80 secure :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00ec10 tx=0x7fb70c00ef20 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 --2- 172.21.15.143:0/2117466497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb710104c40 0x7fb71019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.640+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 >> 172.21.15.143:0/2117466497 conn(0x7fb7100ffb40 msgr2=0x7fb7101001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:25.630 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.644+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 shutdown_connections 2024-03-11T21:44:25.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.644+0000 7fb717ce9700 1 -- 172.21.15.143:0/2117466497 wait complete. 2024-03-11T21:44:25.631 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:25.644+0000 7fb717ce9700 1 librados: shutdown 2024-03-11T21:44:25.633 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:44:25.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:25.703 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:25.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.830 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:25.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:25.961 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:26.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:26.092 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:26.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:26.221 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: cluster 2024-03-11T21:44:24.405008+0000 osd.7 (osd.7) 486 : cluster 0 3.13 scrub starts 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: cluster 2024-03-11T21:44:24.408187+0000 osd.7 (osd.7) 487 : cluster 0 3.13 scrub ok 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: audit 2024-03-11T21:44:25.001674+0000 mon.c (mon.2) 1372 : audit 1 from='client.? 172.21.15.143:0/2117466497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: audit 2024-03-11T21:44:25.001966+0000 mon.a (mon.0) 3991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.618009+0000 mon.a (mon.0) 3992 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: audit 2024-03-11T21:44:25.621270+0000 mon.a (mon.0) 3993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:26.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:26 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.627651+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-03-11T21:44:26.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:26.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: cluster 2024-03-11T21:44:24.405008+0000 osd.7 (osd.7) 486 : cluster 0 3.13 scrub starts 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: cluster 2024-03-11T21:44:24.408187+0000 osd.7 (osd.7) 487 : cluster 0 3.13 scrub ok 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: audit 2024-03-11T21:44:25.001674+0000 mon.c (mon.2) 1372 : audit 1 from='client.? 172.21.15.143:0/2117466497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: audit 2024-03-11T21:44:25.001966+0000 mon.a (mon.0) 3991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.618009+0000 mon.a (mon.0) 3992 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: audit 2024-03-11T21:44:25.621270+0000 mon.a (mon.0) 3993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.627651+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: cluster 2024-03-11T21:44:24.405008+0000 osd.7 (osd.7) 486 : cluster 0 3.13 scrub starts 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: cluster 2024-03-11T21:44:24.408187+0000 osd.7 (osd.7) 487 : cluster 0 3.13 scrub ok 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: audit 2024-03-11T21:44:25.001674+0000 mon.c (mon.2) 1372 : audit 1 from='client.? 172.21.15.143:0/2117466497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: audit 2024-03-11T21:44:25.001966+0000 mon.a (mon.0) 3991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:26.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.618009+0000 mon.a (mon.0) 3992 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:26.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: audit 2024-03-11T21:44:25.621270+0000 mon.a (mon.0) 3993 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:26.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:26 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.627651+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-03-11T21:44:26.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:26.475 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:26.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:26.616 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:26.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:44:26.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:26.746 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:26.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:26.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:26.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:27.007 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:27.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.231231+0000 osd.2 (osd.2) 431 : cluster 0 5.d scrub starts 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.233827+0000 osd.2 (osd.2) 432 : cluster 0 5.d scrub ok 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.237975+0000 osd.1 (osd.1) 305 : cluster 0 9.12 scrub starts 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.240044+0000 osd.1 (osd.1) 306 : cluster 0 9.12 scrub ok 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:25.681064+0000 mgr.y (mgr.14550) 829 : cluster 0 pgmap v1432: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.218133+0000 osd.2 (osd.2) 433 : cluster 0 2.1 scrub starts 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.219860+0000 osd.2 (osd.2) 434 : cluster 0 2.1 scrub ok 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.311044+0000 osd.3 (osd.3) 497 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.312852+0000 osd.3 (osd.3) 498 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.617593+0000 mon.a (mon.0) 3995 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-03-11T21:44:27.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:27 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.621300+0000 mon.a (mon.0) 3996 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:27.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:27.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:27.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.231231+0000 osd.2 (osd.2) 431 : cluster 0 5.d scrub starts 2024-03-11T21:44:27.253 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.233827+0000 osd.2 (osd.2) 432 : cluster 0 5.d scrub ok 2024-03-11T21:44:27.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.237975+0000 osd.1 (osd.1) 305 : cluster 0 9.12 scrub starts 2024-03-11T21:44:27.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.240044+0000 osd.1 (osd.1) 306 : cluster 0 9.12 scrub ok 2024-03-11T21:44:27.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:25.681064+0000 mgr.y (mgr.14550) 829 : cluster 0 pgmap v1432: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:27.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.218133+0000 osd.2 (osd.2) 433 : cluster 0 2.1 scrub starts 2024-03-11T21:44:27.254 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.219860+0000 osd.2 (osd.2) 434 : cluster 0 2.1 scrub ok 2024-03-11T21:44:27.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.311044+0000 osd.3 (osd.3) 497 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:44:27.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.312852+0000 osd.3 (osd.3) 498 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:44:27.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.617593+0000 mon.a (mon.0) 3995 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-03-11T21:44:27.255 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.621300+0000 mon.a (mon.0) 3996 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:27.255 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.231231+0000 osd.2 (osd.2) 431 : cluster 0 5.d scrub starts 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.233827+0000 osd.2 (osd.2) 432 : cluster 0 5.d scrub ok 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.237975+0000 osd.1 (osd.1) 305 : cluster 0 9.12 scrub starts 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.240044+0000 osd.1 (osd.1) 306 : cluster 0 9.12 scrub ok 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:25.681064+0000 mgr.y (mgr.14550) 829 : cluster 0 pgmap v1432: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.218133+0000 osd.2 (osd.2) 433 : cluster 0 2.1 scrub starts 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.219860+0000 osd.2 (osd.2) 434 : cluster 0 2.1 scrub ok 2024-03-11T21:44:27.256 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.311044+0000 osd.3 (osd.3) 497 : cluster 0 6.11 deep-scrub starts 2024-03-11T21:44:27.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.312852+0000 osd.3 (osd.3) 498 : cluster 0 6.11 deep-scrub ok 2024-03-11T21:44:27.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.617593+0000 mon.a (mon.0) 3995 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-03-11T21:44:27.257 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:27 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.621300+0000 mon.a (mon.0) 3996 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:27.267 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:27.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:27.396 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:27.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:27.528 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:27.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:27.657 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:27.683 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:44:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:44:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:44:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:44:27.683 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:44:27.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:27.787 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:27.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:27.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:28.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.049 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:28.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.266060+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub starts 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.267723+0000 osd.1 (osd.1) 308 : cluster 0 6.1d scrub ok 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.438252+0000 osd.7 (osd.7) 488 : cluster 0 1.9 scrub starts 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.440007+0000 osd.7 (osd.7) 489 : cluster 0 1.9 scrub ok 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.843744+0000 osd.6 (osd.6) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:28.164 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.844153+0000 osd.6 (osd.6) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.977092+0000 osd.4 (osd.4) 469 : cluster 0 3.1c scrub starts 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:26.980186+0000 osd.4 (osd.4) 470 : cluster 0 3.1c scrub ok 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.180011+0000 osd.2 (osd.2) 435 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.182255+0000 osd.2 (osd.2) 436 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.248770+0000 osd.0 (osd.0) 488 : cluster 0 9.9 scrub starts 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.251562+0000 osd.0 (osd.0) 489 : cluster 0 9.9 scrub ok 2024-03-11T21:44:28.165 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:28 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.622193+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-03-11T21:44:28.182 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:28.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.312 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:28.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:28.440 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:28.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.266060+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub starts 2024-03-11T21:44:28.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.267723+0000 osd.1 (osd.1) 308 : cluster 0 6.1d scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.438252+0000 osd.7 (osd.7) 488 : cluster 0 1.9 scrub starts 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.440007+0000 osd.7 (osd.7) 489 : cluster 0 1.9 scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.843744+0000 osd.6 (osd.6) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.844153+0000 osd.6 (osd.6) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.977092+0000 osd.4 (osd.4) 469 : cluster 0 3.1c scrub starts 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:26.980186+0000 osd.4 (osd.4) 470 : cluster 0 3.1c scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.180011+0000 osd.2 (osd.2) 435 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.182255+0000 osd.2 (osd.2) 436 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.248770+0000 osd.0 (osd.0) 488 : cluster 0 9.9 scrub starts 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.251562+0000 osd.0 (osd.0) 489 : cluster 0 9.9 scrub ok 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.622193+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-03-11T21:44:28.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.266060+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.267723+0000 osd.1 (osd.1) 308 : cluster 0 6.1d scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.438252+0000 osd.7 (osd.7) 488 : cluster 0 1.9 scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.440007+0000 osd.7 (osd.7) 489 : cluster 0 1.9 scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.843744+0000 osd.6 (osd.6) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.844153+0000 osd.6 (osd.6) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.977092+0000 osd.4 (osd.4) 469 : cluster 0 3.1c scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:26.980186+0000 osd.4 (osd.4) 470 : cluster 0 3.1c scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.180011+0000 osd.2 (osd.2) 435 : cluster 0 2.3 deep-scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.182255+0000 osd.2 (osd.2) 436 : cluster 0 2.3 deep-scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.248770+0000 osd.0 (osd.0) 488 : cluster 0 9.9 scrub starts 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.251562+0000 osd.0 (osd.0) 489 : cluster 0 9.9 scrub ok 2024-03-11T21:44:28.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:28 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.622193+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-03-11T21:44:28.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:44:28.570 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:28.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.692 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:28.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:44:28.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:28.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:28.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:28.979 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:29.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:29.108 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:29.156 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:44:29.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.240 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:29.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:29.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:27.681634+0000 mgr.y (mgr.14550) 830 : cluster 0 pgmap v1435: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.177482+0000 osd.2 (osd.2) 437 : cluster 0 2.10 scrub starts 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.179946+0000 osd.2 (osd.2) 438 : cluster 0 2.10 scrub ok 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.287461+0000 osd.3 (osd.3) 499 : cluster 0 9.1b scrub starts 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.292590+0000 osd.3 (osd.3) 500 : cluster 0 9.1b scrub ok 2024-03-11T21:44:29.355 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:29 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.629501+0000 mon.a (mon.0) 3998 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-03-11T21:44:29.369 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:29.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:27.681634+0000 mgr.y (mgr.14550) 830 : cluster 0 pgmap v1435: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:29.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.177482+0000 osd.2 (osd.2) 437 : cluster 0 2.10 scrub starts 2024-03-11T21:44:29.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.179946+0000 osd.2 (osd.2) 438 : cluster 0 2.10 scrub ok 2024-03-11T21:44:29.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.287461+0000 osd.3 (osd.3) 499 : cluster 0 9.1b scrub starts 2024-03-11T21:44:29.445 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.292590+0000 osd.3 (osd.3) 500 : cluster 0 9.1b scrub ok 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.629501+0000 mon.a (mon.0) 3998 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:27.681634+0000 mgr.y (mgr.14550) 830 : cluster 0 pgmap v1435: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.177482+0000 osd.2 (osd.2) 437 : cluster 0 2.10 scrub starts 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.179946+0000 osd.2 (osd.2) 438 : cluster 0 2.10 scrub ok 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.287461+0000 osd.3 (osd.3) 499 : cluster 0 9.1b scrub starts 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.292590+0000 osd.3 (osd.3) 500 : cluster 0 9.1b scrub ok 2024-03-11T21:44:29.446 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:29 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.629501+0000 mon.a (mon.0) 3998 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-03-11T21:44:29.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:29.500 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:29.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.635 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:29.739 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:29.766 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:29.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:29.894 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:29.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:30.027 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:30.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:30.155 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:30.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.254615+0000 osd.1 (osd.1) 309 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.257243+0000 osd.1 (osd.1) 310 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.921556+0000 osd.7 (osd.7) 490 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.922011+0000 osd.7 (osd.7) 491 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: audit 2024-03-11T21:44:29.149771+0000 mgr.y (mgr.14550) 831 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:30.270 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:30 smithi203 bash[23654]: cluster 2024-03-11T21:44:29.633303+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-03-11T21:44:30.286 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:30.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.410 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.254615+0000 osd.1 (osd.1) 309 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:44:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.257243+0000 osd.1 (osd.1) 310 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:44:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.921556+0000 osd.7 (osd.7) 490 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:30.446 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.922011+0000 osd.7 (osd.7) 491 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: audit 2024-03-11T21:44:29.149771+0000 mgr.y (mgr.14550) 831 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:30.447 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[21512]: cluster 2024-03-11T21:44:29.633303+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-03-11T21:44:30.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.254615+0000 osd.1 (osd.1) 309 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:44:30.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.257243+0000 osd.1 (osd.1) 310 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:44:30.447 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.921556+0000 osd.7 (osd.7) 490 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:30.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.922011+0000 osd.7 (osd.7) 491 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:30.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: audit 2024-03-11T21:44:29.149771+0000 mgr.y (mgr.14550) 831 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:30.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:30 smithi143 bash[29284]: cluster 2024-03-11T21:44:29.633303+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-03-11T21:44:30.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:30.539 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:30.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:30.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:30.798 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:30.901 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:30.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:31.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:31.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:31.185 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:31.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.311 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:31.391 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:31.410 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.969935+0000 osd.4 (osd.4) 471 : cluster 0 6.17 scrub starts 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[21512]: cluster 2024-03-11T21:44:28.972856+0000 osd.4 (osd.4) 472 : cluster 0 6.17 scrub ok 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[21512]: cluster 2024-03-11T21:44:29.682047+0000 mgr.y (mgr.14550) 832 : cluster 0 pgmap v1438: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[21512]: cluster 2024-03-11T21:44:30.642844+0000 mon.a (mon.0) 4000 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.969935+0000 osd.4 (osd.4) 471 : cluster 0 6.17 scrub starts 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[29284]: cluster 2024-03-11T21:44:28.972856+0000 osd.4 (osd.4) 472 : cluster 0 6.17 scrub ok 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[29284]: cluster 2024-03-11T21:44:29.682047+0000 mgr.y (mgr.14550) 832 : cluster 0 pgmap v1438: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:31.411 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:31 smithi143 bash[29284]: cluster 2024-03-11T21:44:30.642844+0000 mon.a (mon.0) 4000 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-03-11T21:44:31.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:31.433 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:31.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:31 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.969935+0000 osd.4 (osd.4) 471 : cluster 0 6.17 scrub starts 2024-03-11T21:44:31.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:31 smithi203 bash[23654]: cluster 2024-03-11T21:44:28.972856+0000 osd.4 (osd.4) 472 : cluster 0 6.17 scrub ok 2024-03-11T21:44:31.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:31 smithi203 bash[23654]: cluster 2024-03-11T21:44:29.682047+0000 mgr.y (mgr.14550) 832 : cluster 0 pgmap v1438: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:31.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:31 smithi203 bash[23654]: cluster 2024-03-11T21:44:30.642844+0000 mon.a (mon.0) 4000 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-03-11T21:44:31.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.561 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:31.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:31.698 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:31.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.824 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:31.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.953 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.082 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.210 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:30.672035+0000 osd.6 (osd.6) 411 : cluster 0 5.11 scrub starts 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:30.676908+0000 osd.6 (osd.6) 412 : cluster 0 5.11 scrub ok 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.125920+0000 osd.2 (osd.2) 439 : cluster 0 5.a deep-scrub starts 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.127469+0000 osd.2 (osd.2) 440 : cluster 0 5.a deep-scrub ok 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.280058+0000 osd.3 (osd.3) 501 : cluster 0 3.2 scrub starts 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.283460+0000 osd.3 (osd.3) 502 : cluster 0 3.2 scrub ok 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.300827+0000 osd.0 (osd.0) 490 : cluster 0 5.1e scrub starts 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.303381+0000 osd.0 (osd.0) 491 : cluster 0 5.1e scrub ok 2024-03-11T21:44:32.212 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.670106+0000 mon.a (mon.0) 4001 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:30.672035+0000 osd.6 (osd.6) 411 : cluster 0 5.11 scrub starts 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:30.676908+0000 osd.6 (osd.6) 412 : cluster 0 5.11 scrub ok 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.125920+0000 osd.2 (osd.2) 439 : cluster 0 5.a deep-scrub starts 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.127469+0000 osd.2 (osd.2) 440 : cluster 0 5.a deep-scrub ok 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.280058+0000 osd.3 (osd.3) 501 : cluster 0 3.2 scrub starts 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.283460+0000 osd.3 (osd.3) 502 : cluster 0 3.2 scrub ok 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.300827+0000 osd.0 (osd.0) 490 : cluster 0 5.1e scrub starts 2024-03-11T21:44:32.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.303381+0000 osd.0 (osd.0) 491 : cluster 0 5.1e scrub ok 2024-03-11T21:44:32.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:32 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.670106+0000 mon.a (mon.0) 4001 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-03-11T21:44:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:30.672035+0000 osd.6 (osd.6) 411 : cluster 0 5.11 scrub starts 2024-03-11T21:44:32.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:30.676908+0000 osd.6 (osd.6) 412 : cluster 0 5.11 scrub ok 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.125920+0000 osd.2 (osd.2) 439 : cluster 0 5.a deep-scrub starts 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.127469+0000 osd.2 (osd.2) 440 : cluster 0 5.a deep-scrub ok 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.280058+0000 osd.3 (osd.3) 501 : cluster 0 3.2 scrub starts 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.283460+0000 osd.3 (osd.3) 502 : cluster 0 3.2 scrub ok 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.300827+0000 osd.0 (osd.0) 490 : cluster 0 5.1e scrub starts 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.303381+0000 osd.0 (osd.0) 491 : cluster 0 5.1e scrub ok 2024-03-11T21:44:32.357 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:32 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.670106+0000 mon.a (mon.0) 4001 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-03-11T21:44:32.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:32.465 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:32.528 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:44:32.528 INFO:teuthology.orchestra.run.smithi203.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:44:32.528 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:44:32.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.587 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:32.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.713 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:32.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:32.843 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:32.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.974 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:33.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:33.103 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:33.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:33.232 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.683018+0000 mgr.y (mgr.14550) 833 : cluster 0 pgmap v1441: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.698769+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:31.701609+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.227358+0000 osd.4 (osd.4) 473 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.227851+0000 osd.4 (osd.4) 474 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:33.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.230704+0000 osd.3 (osd.3) 503 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:44:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.231991+0000 osd.3 (osd.3) 504 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:44:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.641092+0000 mon.a (mon.0) 4002 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:33 smithi203 bash[23654]: cluster 2024-03-11T21:44:32.654703+0000 mon.a (mon.0) 4003 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-03-11T21:44:33.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.683018+0000 mgr.y (mgr.14550) 833 : cluster 0 pgmap v1441: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.698769+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:31.701609+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.227358+0000 osd.4 (osd.4) 473 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.227851+0000 osd.4 (osd.4) 474 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.230704+0000 osd.3 (osd.3) 503 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.231991+0000 osd.3 (osd.3) 504 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.641092+0000 mon.a (mon.0) 4002 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:33.354 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[21512]: cluster 2024-03-11T21:44:32.654703+0000 mon.a (mon.0) 4003 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.683018+0000 mgr.y (mgr.14550) 833 : cluster 0 pgmap v1441: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.698769+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:31.701609+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.227358+0000 osd.4 (osd.4) 473 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.227851+0000 osd.4 (osd.4) 474 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.230704+0000 osd.3 (osd.3) 503 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.231991+0000 osd.3 (osd.3) 504 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.641092+0000 mon.a (mon.0) 4002 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:33.355 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:33 smithi143 bash[29284]: cluster 2024-03-11T21:44:32.654703+0000 mon.a (mon.0) 4003 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-03-11T21:44:33.365 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:33.430 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:33.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:33.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:33.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:33.628 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:33.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:33.756 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:33.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:33.884 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:33.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.017 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:34.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.148 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:34.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:34.280 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:34.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:34.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:34.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.540 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:34.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.675 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:34.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.013202+0000 osd.4 (osd.4) 475 : cluster 0 9.10 scrub starts 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.015574+0000 osd.4 (osd.4) 476 : cluster 0 9.10 scrub ok 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.292049+0000 osd.1 (osd.1) 311 : cluster 0 6.d scrub starts 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.293541+0000 osd.1 (osd.1) 312 : cluster 0 6.d scrub ok 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.401958+0000 osd.7 (osd.7) 492 : cluster 0 9.17 scrub starts 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.405429+0000 osd.7 (osd.7) 493 : cluster 0 9.17 scrub ok 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.670145+0000 mon.a (mon.0) 4004 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-03-11T21:44:34.789 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[29284]: cluster 2024-03-11T21:44:33.684114+0000 mgr.y (mgr.14550) 834 : cluster 0 pgmap v1444: 216 pgs: 5 unknown, 211 active+clean; 582 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.013202+0000 osd.4 (osd.4) 475 : cluster 0 9.10 scrub starts 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.015574+0000 osd.4 (osd.4) 476 : cluster 0 9.10 scrub ok 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.292049+0000 osd.1 (osd.1) 311 : cluster 0 6.d scrub starts 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.293541+0000 osd.1 (osd.1) 312 : cluster 0 6.d scrub ok 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.401958+0000 osd.7 (osd.7) 492 : cluster 0 9.17 scrub starts 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.405429+0000 osd.7 (osd.7) 493 : cluster 0 9.17 scrub ok 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.670145+0000 mon.a (mon.0) 4004 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-03-11T21:44:34.790 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:34 smithi143 bash[21512]: cluster 2024-03-11T21:44:33.684114+0000 mgr.y (mgr.14550) 834 : cluster 0 pgmap v1444: 216 pgs: 5 unknown, 211 active+clean; 582 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:34.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:34.846 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:44:34.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.013202+0000 osd.4 (osd.4) 475 : cluster 0 9.10 scrub starts 2024-03-11T21:44:34.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.015574+0000 osd.4 (osd.4) 476 : cluster 0 9.10 scrub ok 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.292049+0000 osd.1 (osd.1) 311 : cluster 0 6.d scrub starts 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.293541+0000 osd.1 (osd.1) 312 : cluster 0 6.d scrub ok 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.401958+0000 osd.7 (osd.7) 492 : cluster 0 9.17 scrub starts 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.405429+0000 osd.7 (osd.7) 493 : cluster 0 9.17 scrub ok 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.670145+0000 mon.a (mon.0) 4004 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-03-11T21:44:34.917 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:34 smithi203 bash[23654]: cluster 2024-03-11T21:44:33.684114+0000 mgr.y (mgr.14550) 834 : cluster 0 pgmap v1444: 216 pgs: 5 unknown, 211 active+clean; 582 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:34.931 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:35.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.059 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:35.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:35.185 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:35.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:35.316 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:35.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:35.446 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:35.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:35.577 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:35.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:35.705 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:35.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:35.837 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:35.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:35.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[29284]: cluster 2024-03-11T21:44:34.210210+0000 osd.3 (osd.3) 505 : cluster 0 4.0 scrub starts 2024-03-11T21:44:35.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[29284]: cluster 2024-03-11T21:44:34.211940+0000 osd.3 (osd.3) 506 : cluster 0 4.0 scrub ok 2024-03-11T21:44:35.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[29284]: cluster 2024-03-11T21:44:34.682007+0000 mon.a (mon.0) 4005 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-03-11T21:44:35.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[21512]: cluster 2024-03-11T21:44:34.210210+0000 osd.3 (osd.3) 505 : cluster 0 4.0 scrub starts 2024-03-11T21:44:35.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[21512]: cluster 2024-03-11T21:44:34.211940+0000 osd.3 (osd.3) 506 : cluster 0 4.0 scrub ok 2024-03-11T21:44:35.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:35 smithi143 bash[21512]: cluster 2024-03-11T21:44:34.682007+0000 mon.a (mon.0) 4005 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-03-11T21:44:35.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:35 smithi203 bash[23654]: cluster 2024-03-11T21:44:34.210210+0000 osd.3 (osd.3) 505 : cluster 0 4.0 scrub starts 2024-03-11T21:44:35.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:35 smithi203 bash[23654]: cluster 2024-03-11T21:44:34.211940+0000 osd.3 (osd.3) 506 : cluster 0 4.0 scrub ok 2024-03-11T21:44:35.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:35 smithi203 bash[23654]: cluster 2024-03-11T21:44:34.682007+0000 mon.a (mon.0) 4005 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-03-11T21:44:35.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:36.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.096 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:36.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.231 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:36.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.360 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:36.463 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:36.491 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:36.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:36.631 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:36.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:36.761 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:36.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:36.876 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:36 smithi203 bash[23654]: cluster 2024-03-11T21:44:34.968918+0000 osd.4 (osd.4) 477 : cluster 0 5.1c scrub starts 2024-03-11T21:44:36.876 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:36 smithi203 bash[23654]: cluster 2024-03-11T21:44:34.970503+0000 osd.4 (osd.4) 478 : cluster 0 5.1c scrub ok 2024-03-11T21:44:36.876 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:36 smithi203 bash[23654]: cluster 2024-03-11T21:44:35.686073+0000 mgr.y (mgr.14550) 835 : cluster 0 pgmap v1447: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:36.876 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:36 smithi203 bash[23654]: cluster 2024-03-11T21:44:35.691416+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-03-11T21:44:36.876 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:36 smithi203 bash[23654]: cluster 2024-03-11T21:44:36.617553+0000 mon.a (mon.0) 4007 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-03-11T21:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:36.908 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[29284]: cluster 2024-03-11T21:44:34.968918+0000 osd.4 (osd.4) 477 : cluster 0 5.1c scrub starts 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[29284]: cluster 2024-03-11T21:44:34.970503+0000 osd.4 (osd.4) 478 : cluster 0 5.1c scrub ok 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[29284]: cluster 2024-03-11T21:44:35.686073+0000 mgr.y (mgr.14550) 835 : cluster 0 pgmap v1447: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[29284]: cluster 2024-03-11T21:44:35.691416+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[29284]: cluster 2024-03-11T21:44:36.617553+0000 mon.a (mon.0) 4007 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[21512]: cluster 2024-03-11T21:44:34.968918+0000 osd.4 (osd.4) 477 : cluster 0 5.1c scrub starts 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[21512]: cluster 2024-03-11T21:44:34.970503+0000 osd.4 (osd.4) 478 : cluster 0 5.1c scrub ok 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[21512]: cluster 2024-03-11T21:44:35.686073+0000 mgr.y (mgr.14550) 835 : cluster 0 pgmap v1447: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[21512]: cluster 2024-03-11T21:44:35.691416+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-03-11T21:44:36.909 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:36 smithi143 bash[21512]: cluster 2024-03-11T21:44:36.617553+0000 mon.a (mon.0) 4007 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-03-11T21:44:36.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:37.025 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:37.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:37.149 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:37.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:37.279 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:37.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:37.409 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:37.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:37.538 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:37.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:37.668 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: cluster 2024-03-11T21:44:36.191573+0000 osd.3 (osd.3) 507 : cluster 0 9.2 scrub starts 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: cluster 2024-03-11T21:44:36.193570+0000 osd.3 (osd.3) 508 : cluster 0 9.2 scrub ok 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: cluster 2024-03-11T21:44:36.474792+0000 osd.7 (osd.7) 494 : cluster 0 2.8 scrub starts 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: cluster 2024-03-11T21:44:36.477227+0000 osd.7 (osd.7) 495 : cluster 0 2.8 scrub ok 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.624596+0000 mon.a (mon.0) 4008 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: audit 2024-03-11T21:44:37.651575+0000 mon.a (mon.0) 4009 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:37.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:37 smithi203 bash[23654]: audit 2024-03-11T21:44:37.654550+0000 mon.c (mon.2) 1373 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:37.800 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: cluster 2024-03-11T21:44:36.191573+0000 osd.3 (osd.3) 507 : cluster 0 9.2 scrub starts 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: cluster 2024-03-11T21:44:36.193570+0000 osd.3 (osd.3) 508 : cluster 0 9.2 scrub ok 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: cluster 2024-03-11T21:44:36.474792+0000 osd.7 (osd.7) 494 : cluster 0 2.8 scrub starts 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: cluster 2024-03-11T21:44:36.477227+0000 osd.7 (osd.7) 495 : cluster 0 2.8 scrub ok 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.624596+0000 mon.a (mon.0) 4008 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: audit 2024-03-11T21:44:37.651575+0000 mon.a (mon.0) 4009 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[21512]: audit 2024-03-11T21:44:37.654550+0000 mon.c (mon.2) 1373 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: cluster 2024-03-11T21:44:36.191573+0000 osd.3 (osd.3) 507 : cluster 0 9.2 scrub starts 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: cluster 2024-03-11T21:44:36.193570+0000 osd.3 (osd.3) 508 : cluster 0 9.2 scrub ok 2024-03-11T21:44:37.812 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: cluster 2024-03-11T21:44:36.474792+0000 osd.7 (osd.7) 494 : cluster 0 2.8 scrub starts 2024-03-11T21:44:37.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: cluster 2024-03-11T21:44:36.477227+0000 osd.7 (osd.7) 495 : cluster 0 2.8 scrub ok 2024-03-11T21:44:37.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.624596+0000 mon.a (mon.0) 4008 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-03-11T21:44:37.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: audit 2024-03-11T21:44:37.651575+0000 mon.a (mon.0) 4009 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:37.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:37 smithi143 bash[29284]: audit 2024-03-11T21:44:37.654550+0000 mon.c (mon.2) 1373 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:37.828 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:37.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:37.956 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:38.083 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:38.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:38.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.339 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:44:38.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:38.364 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T21:44:38.364 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:44:38.364 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:44:38.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T21:44:38.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.465 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2 313.2s0 313:4de65fb6:test-rados-api-smithi143-71773-113::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 300.91535522300001, 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071012540000000004, 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.478 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.481 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:44:38.481 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.481 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570425+0000", 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8229999999999999e-06 2024-03-11T21:44:38.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570437+0000", 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2719e-05 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570471+0000", 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3231000000000001e-05 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570493+0000", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2235e-05 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570634+0000", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014130800000000001 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570744+0000", 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109374 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.571127+0000", 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000383345 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.572914+0000", 2024-03-11T21:44:38.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017873469999999999 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.577431+0000", 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045166659999999999 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.577523+0000", 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2206000000000005e-05 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.486 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(39.16,QREQUEST,e:716,from:2,reservation_nonce:13)", 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 262.894130464, 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012178527999999999, 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.487 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591649+0000", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.57e-06 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591663+0000", 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3713999999999999e-05 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591709+0000", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5725e-05 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591733+0000", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4019000000000002e-05 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591931+0000", 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019819400000000001 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591950+0000", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8732999999999998e-05 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603664+0000", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011714354999999999 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603709+0000", 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5252e-05 2024-03-11T21:44:38.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603825+0000", 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011596599999999999 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(2.e,QREQUEST,e:718,from:2,reservation_nonce:21)", 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 260.91921853500003, 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014035486999999999, 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.491 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566560+0000", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8339999999999999e-06 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566573+0000", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2283000000000001e-05 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566635+0000", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2661000000000002e-05 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566660+0000", 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5126e-05 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566834+0000", 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000173535 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566851+0000", 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7192000000000001e-05 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580427+0000", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013575436999999999 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580473+0000", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6029000000000003e-05 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580594+0000", 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012139 2024-03-11T21:44:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.41921.0:805 320.19 e748/743 320:98ec0ffc:test-rados-api-smithi143-71773-125::foo:4 v 748'3, mlcod=748'1)", 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 230.83937926499999, 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01057773, 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:44:38.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646399+0000", 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4700000000000001e-07 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646408+0000", 2024-03-11T21:44:38.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4849999999999998e-06 2024-03-11T21:44:38.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.498 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.501 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646424+0000", 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6155e-05 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646435+0000", 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1148999999999999e-05 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646522+0000", 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6561e-05 2024-03-11T21:44:38.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646549+0000", 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7432000000000001e-05 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656880+0000", 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010331138 2024-03-11T21:44:38.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656976+0000", 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5062999999999994e-05 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:859 320.a 320:50000000::::head [pgnls start_epoch 751] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e751)", 2024-03-11T21:44:38.504 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 227.83093876800001, 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0090076349999999999, 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 859 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:44:38.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654842+0000", 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4400000000000001e-06 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.506 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654858+0000", 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6022999999999999e-05 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654889+0000", 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1619000000000003e-05 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654914+0000", 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5058999999999998e-05 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663527+0000", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086127559999999992 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663583+0000", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5402000000000003e-05 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663846+0000", 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026333599999999997 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:863 320.e 320:70000000::::head [pgnls start_epoch 751] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e751)", 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 227.81594889799999, 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076691629999999997, 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 863 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669831+0000", 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.163e-06 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669848+0000", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6575999999999999e-05 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671377+0000", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015295059999999999 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671406+0000", 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8396000000000001e-05 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671570+0000", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016444499999999999 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671587+0000", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6405999999999998e-05 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677156+0000", 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00556889 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677245+0000", 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9530999999999994e-05 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677497+0000", 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025224999999999998 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.513 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1036 321.10 321:08000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 219.896005696, 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071837230000000004, 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1036 2024-03-11T21:44:38.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.515 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589776+0000", 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.25e-06 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589797+0000", 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1222000000000001e-05 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589842+0000", 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5491999999999997e-05 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589870+0000", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7206e-05 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.590058+0000", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018823200000000001 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.590068+0000", 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9799999999999993e-06 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596575+0000", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065068879999999997 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596677+0000", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101894 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596955+0000", 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000278559 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.518 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1112 322.a 322:50000000:test-rados-api-smithi143-71773-129:::head [pgnls start_epoch 765] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e765)", 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 213.89469205399999, 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094745179999999995, 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1112 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591088+0000", 2024-03-11T21:44:38.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8899999999999999e-06 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591104+0000", 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6532999999999999e-05 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591133+0000", 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.889e-05 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591150+0000", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6262000000000001e-05 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591333+0000", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018294200000000001 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591351+0000", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8209000000000001e-05 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.598498+0000", 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071470290000000001 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.598589+0000", 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0953999999999997e-05 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.600560+0000", 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001970809 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1238 322.a 322:50000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.83689396400001, 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064194079999999997, 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1238 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648887+0000", 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5899999999999999e-06 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648904+0000", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7172000000000001e-05 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648942+0000", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7709000000000001e-05 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648964+0000", 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2237000000000001e-05 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.649112+0000", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000147999 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.649122+0000", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9459999999999993e-06 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.654984+0000", 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0058623640000000001 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.527 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.655072+0000", 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7698e-05 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.655303+0000", 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000230693 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1730 325.8 325:10000000::::head [pgnls start_epoch 799] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e799)", 2024-03-11T21:44:38.528 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 177.869775369, 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01054078, 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1730 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616005+0000", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7299999999999999e-06 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616023+0000", 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8026e-05 2024-03-11T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616066+0000", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2592000000000001e-05 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616086+0000", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9596999999999999e-05 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616229+0000", 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014340199999999999 2024-03-11T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616241+0000", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.148e-05 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626197+0000", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0099566089999999999 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626293+0000", 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6193999999999994e-05 2024-03-11T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626542+0000", 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024915000000000001 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:187 313.2s0 313:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.90529862700001, 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069664389999999996, 2024-03-11T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580480+0000", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6130000000000001e-06 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580491+0000", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.084e-05 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580511+0000", 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9763e-05 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580527+0000", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6532e-05 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580704+0000", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000176973 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting_for_map not empty", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580721+0000", 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7108999999999998e-05 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.587172+0000", 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064507699999999998 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.587445+0000", 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027283899999999998 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:188 313.2s0 313:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.873441942, 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.024990762999999999, 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612338+0000", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9170000000000002e-06 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612353+0000", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5247e-05 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612392+0000", 2024-03-11T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8232000000000003e-05 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612410+0000", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8916e-05 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612584+0000", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017358200000000001 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612980+0000", 2024-03-11T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039607899999999997 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.630531+0000", 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017550530000000002 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.634443+0000", 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039128399999999999 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.637259+0000", 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028156259999999999 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.637326+0000", 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6793999999999998e-05 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1832 326.8 326:10000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 169.89036113399999, 2024-03-11T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073047190000000003, 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1832 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595420+0000", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0470000000000004e-06 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595440+0000", 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0435999999999999e-05 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595486+0000", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5158999999999999e-05 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595513+0000", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7656000000000001e-05 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595709+0000", 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019585499999999999 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595726+0000", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6708999999999999e-05 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602402+0000", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066760949999999999 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602494+0000", 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2264999999999998e-05 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602721+0000", 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022649700000000001 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(3.7,QREQUEST,e:821,from:3,reservation_nonce:15)", 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:03.739302+0000", 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 154.746474984, 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006104333, 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739302+0000", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739302+0000", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739304+0000", 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2630000000000002e-06 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739319+0000", 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4552000000000001e-05 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739346+0000", 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7217e-05 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739367+0000", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0911999999999999e-05 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739615+0000", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024817399999999999 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.739635+0000", 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9378999999999999e-05 2024-03-11T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.745267+0000", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056326429999999997 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.745318+0000", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0528999999999997e-05 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:03.745406+0000", 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8664000000000004e-05 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.551 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2152 329.c 329:30000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 127.88108307100001, 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066286260000000003, 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2152 2024-03-11T21:44:38.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604698+0000", 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5599999999999998e-06 2024-03-11T21:44:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604715+0000", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.736e-05 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604754+0000", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8729000000000001e-05 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604773+0000", 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8879e-05 2024-03-11T21:44:38.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604956+0000", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018369600000000001 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604970+0000", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.375e-05 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611112+0000", 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061424670000000004 2024-03-11T21:44:38.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611171+0000", 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8239000000000003e-05 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611323+0000", 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000151946 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4141 3.13 3:cc909b6b:::obj_delete_at_hint.0000000106:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.895237995000002, 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012831966, 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4141 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590540+0000", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7000000000000004e-07 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590544+0000", 2024-03-11T21:44:38.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7190000000000001e-06 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590554+0000", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8390000000000002e-06 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590560+0000", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3990000000000003e-06 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590623+0000", 2024-03-11T21:44:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2971000000000004e-05 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590718+0000", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5625999999999996e-05 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590906+0000", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000187868 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591951+0000", 2024-03-11T21:44:38.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010447189999999999 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.602960+0000", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011009443000000001 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603241+0000", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028086300000000001 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603312+0000", 2024-03-11T21:44:38.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0821999999999994e-05 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603371+0000", 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9027000000000003e-05 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2933 332.9 332:90000000:%01:::head [pgnls start_epoch 871] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e871)", 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 79.819899644000003, 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069345129999999998, 2024-03-11T21:44:38.562 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2933 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665881+0000", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5870000000000001e-06 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665898+0000", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6739999999999999e-05 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665936+0000", 2024-03-11T21:44:38.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7932000000000002e-05 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665961+0000", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4845999999999999e-05 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.666161+0000", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020025899999999999 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.666172+0000", 2024-03-11T21:44:38.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.093e-05 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672490+0000", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006318532 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672585+0000", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5128000000000001e-05 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672812+0000", 2024-03-11T21:44:38.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022655900000000001 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.c,QREQUEST,e:882,from:3,reservation_nonce:9)", 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 68.783014913000002, 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083183479999999997, 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702764+0000", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1059999999999998e-06 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702774+0000", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0217000000000001e-05 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702813+0000", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8065000000000002e-05 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702827+0000", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4823e-05 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702974+0000", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014656399999999999 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702990+0000", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6473e-05 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.710927+0000", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0079361369999999994 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.710974+0000", 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7905999999999998e-05 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.711080+0000", 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106057 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4366 5.9 5:93e5b521:::notify.7:head [watch ping cookie 93859898882688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e910)", 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:28.984174+0000", 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 9.5016028210000005, 2024-03-11T21:44:38.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012046610000000001, 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4366 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984174+0000", 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984174+0000", 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984177+0000", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1300000000000001e-06 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984192+0000", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4317e-05 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984227+0000", 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5029e-05 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984248+0000", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1676e-05 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.984730+0000", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000481896 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.985024+0000", 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029386400000000002 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.985379+0000", 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035474900000000001 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4376 5.9 5:93e5b521:::notify.7:head [watch ping cookie 93859898882688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:33.988495+0000", 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 4.497282491, 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091102399999999997, 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:44:38.576 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4376 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988495+0000", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988495+0000", 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:44:38.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988498+0000", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2559999999999998e-06 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988512+0000", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.402e-05 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988575+0000", 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3032000000000002e-05 2024-03-11T21:44:38.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.988601+0000", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6066000000000001e-05 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.989041+0000", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00044023100000000002 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.989212+0000", 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017048000000000001 2024-03-11T21:44:38.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.989406+0000", 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019393899999999999 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:44:38.580 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:44:38.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.605 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- 172.21.15.143:0/2280684001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc1071b0 msgr2=0x7fdddc10ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- 172.21.15.143:0/2280684001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc1071b0 0x7fdddc10ed50 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7fddd400b0d0 tx=0x7fddd400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- 172.21.15.143:0/2280684001 shutdown_connections 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- 172.21.15.143:0/2280684001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdddc06a6a0 0x7fdddc06aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- 172.21.15.143:0/2280684001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc1071b0 0x7fdddc10ed50 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- 172.21.15.143:0/2280684001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdddc06b050 0x7fdddc106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- 172.21.15.143:0/2280684001 >> 172.21.15.143:0/2280684001 conn(0x7fdddc077a40 msgr2=0x7fdddc077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- 172.21.15.143:0/2280684001 shutdown_connections 2024-03-11T21:44:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- 172.21.15.143:0/2280684001 wait complete. 2024-03-11T21:44:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 librados: starting msgr at 2024-03-11T21:44:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 librados: starting objecter 2024-03-11T21:44:38.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 Processor -- start 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- start start 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 librados: setting wanted keys 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 librados: calling monclient init 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdddc06a6a0 0x7fdddc19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdddc1071b0 0x7fdddc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc111680 con 0x7fdddc06a6a0 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc111500 con 0x7fdddc1071b0 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde2549700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc111800 con 0x7fdddc06b050 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdddb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdddbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdddc06a6a0 0x7fdddc19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdddb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37284/0 (socket says 172.21.15.143:37284) 2024-03-11T21:44:38.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdddb7fe700 1 -- 172.21.15.143:0/1910083281 learned_addr learned my addr 172.21.15.143:0/1910083281 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.683+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdddc1071b0 0x7fdddc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdddc1071b0 msgr2=0x7fdddc1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdddc1071b0 0x7fdddc1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdddc06a6a0 msgr2=0x7fdddc19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdddc06a6a0 0x7fdddc19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc198130 con 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fddd000c8c0 tx=0x7fddd000cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 10 client.?.objecter ms_handle_connect 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0004220 con 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde2549700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc198420 con 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fddd00043c0 con 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00056c0 con 0x7fdddc06b050 2024-03-11T21:44:38.673 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde2549700 10 client.42209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:38.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde2549700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc1a46e0 con 0x7fdddc06b050 2024-03-11T21:44:38.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde2549700 1 librados: init done 2024-03-11T21:44:38.674 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fddbc005100 con 0x7fdddc06b050 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fddd0004750 con 0x7fdddc06b050 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 0x7fddc4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(925..925 src has 842..925) v4 ==== 9108+0+0 (secure 0 0 0) 0x7fddd008e300 con 0x7fdddc06b050 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 osd_map(925..925 src has 842..925) v4 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 3 client.42209.objecter handle_osd_map got epochs [925,925] > 0 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 3 client.42209.objecter handle_osd_map decoding full epoch 925 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 0x7fddc4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 20 client.42209.objecter dump_active .. 0 homeless 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v925) v1 ==== 54+0+16 (secure 0 0 0) 0x7fddd0014030 con 0x7fdddc06b050 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 1 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddc406f820 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 0x7fddc4071ce0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7fdddc198b10 tx=0x7fddcc009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fddbc004730 0x7fddbc004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 1 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 0 1 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 4 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 10 client.42209.objecter _send_command 1 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc008b70 con 0x7fddbc004730 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fddbc004730 0x7fddbc004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fddbc004730 0x7fddbc004b70 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.687+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc004730 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.691+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc008b70 con 0x7fddbc004730 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.691+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.691+0000 7fddd97fa700 10 client.42209.objecter _finish_command 1 = system:0 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.691+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 4 2024-03-11T21:44:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.691+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 0 1 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 2 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 2 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc0053e0 osd=0 3 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 0 2 2024-03-11T21:44:38.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 4 2024-03-11T21:44:38.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 10 client.42209.objecter _send_command 2 2024-03-11T21:44:38.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.707+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc0042a0 con 0x7fddbc004730 2024-03-11T21:44:38.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc0042a0 con 0x7fddbc004730 2024-03-11T21:44:38.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 10 client.42209.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 4 2024-03-11T21:44:38.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 0 2 2024-03-11T21:44:38.696 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:44:38.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fddbc002750 0x7fddbc002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 1 3 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 4 2024-03-11T21:44:38.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 10 client.42209.objecter _send_command 3 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc003110 con 0x7fddbc002750 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fddbc002750 0x7fddbc002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc002750 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fddbc002750 0x7fddbc002bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc003110 con 0x7fddbc002750 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 10 client.42209.objecter _finish_command 3 = system:0 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 4 2024-03-11T21:44:38.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.711+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 1 3 2024-03-11T21:44:38.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.713 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.417798+0000 osd.0 (osd.0) 492 : cluster 0 3.17 scrub starts 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.420155+0000 osd.0 (osd.0) 493 : cluster 0 3.17 scrub ok 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.455594+0000 osd.7 (osd.7) 496 : cluster 0 9.e scrub starts 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.460017+0000 osd.7 (osd.7) 497 : cluster 0 9.e scrub ok 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.562432+0000 osd.3 (osd.3) 509 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.562850+0000 osd.3 (osd.3) 510 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:37.686847+0000 mgr.y (mgr.14550) 836 : cluster 0 pgmap v1450: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: cluster 2024-03-11T21:44:38.624244+0000 mon.a (mon.0) 4010 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[21512]: audit 2024-03-11T21:44:38.690344+0000 mon.c (mon.2) 1374 : audit 0 from='client.? 172.21.15.143:0/1910083281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.417798+0000 osd.0 (osd.0) 492 : cluster 0 3.17 scrub starts 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.420155+0000 osd.0 (osd.0) 493 : cluster 0 3.17 scrub ok 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.455594+0000 osd.7 (osd.7) 496 : cluster 0 9.e scrub starts 2024-03-11T21:44:38.714 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.460017+0000 osd.7 (osd.7) 497 : cluster 0 9.e scrub ok 2024-03-11T21:44:38.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.562432+0000 osd.3 (osd.3) 509 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:38.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.562850+0000 osd.3 (osd.3) 510 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:38.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:37.686847+0000 mgr.y (mgr.14550) 836 : cluster 0 pgmap v1450: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:38.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: cluster 2024-03-11T21:44:38.624244+0000 mon.a (mon.0) 4010 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-03-11T21:44:38.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:38 smithi143 bash[29284]: audit 2024-03-11T21:44:38.690344+0000 mon.c (mon.2) 1374 : audit 0 from='client.? 172.21.15.143:0/1910083281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 4 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 4 2024-03-11T21:44:38.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc004530 osd=1 3 2024-03-11T21:44:38.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 1 4 2024-03-11T21:44:38.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 4 2024-03-11T21:44:38.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 10 client.42209.objecter _send_command 4 2024-03-11T21:44:38.719 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.731+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc003af0 con 0x7fddbc002750 2024-03-11T21:44:38.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc003af0 con 0x7fddbc002750 2024-03-11T21:44:38.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 10 client.42209.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 4 2024-03-11T21:44:38.720 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 1 4 2024-03-11T21:44:38.721 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:44:38.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 5 2024-03-11T21:44:38.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fddbc00b240 0x7fddbc00d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 5 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 2 5 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 4 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 10 client.42209.objecter _send_command 5 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc010520 con 0x7fddbc00b240 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fddbc00b240 0x7fddbc00d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc00b240 2024-03-11T21:44:38.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fddbc00b240 0x7fddbc00d630 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc010520 con 0x7fddbc00b240 2024-03-11T21:44:38.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:44:38.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 10 client.42209.objecter _finish_command 5 = system:0 2024-03-11T21:44:38.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 4 2024-03-11T21:44:38.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.735+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 2 5 2024-03-11T21:44:38.733 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 6 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 6 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc008140 osd=2 3 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 2 6 2024-03-11T21:44:38.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 4 2024-03-11T21:44:38.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 10 client.42209.objecter _send_command 6 2024-03-11T21:44:38.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc003570 con 0x7fddbc00b240 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc003570 con 0x7fddbc00b240 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 10 client.42209.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 4 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 2 6 2024-03-11T21:44:38.741 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 7 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fddbc002090 0x7fddbc00ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 7 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 3 7 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 4 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 10 client.42209.objecter _send_command 7 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc013e70 con 0x7fddbc002090 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fddbc002090 0x7fddbc00ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc002090 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.755+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fddbc002090 0x7fddbc00ecf0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.759+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc013e70 con 0x7fddbc002090 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.759+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.759+0000 7fddd97fa700 10 client.42209.objecter _finish_command 7 = system:0 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.759+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 4 2024-03-11T21:44:38.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.759+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 3 7 2024-03-11T21:44:38.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 8 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 8 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc0037c0 osd=3 3 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 3 8 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 4 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 10 client.42209.objecter _send_command 8 2024-03-11T21:44:38.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc0016c0 con 0x7fddbc002090 2024-03-11T21:44:38.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.775+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc0016c0 con 0x7fddbc002090 2024-03-11T21:44:38.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 10 client.42209.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 4 2024-03-11T21:44:38.762 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 3 8 2024-03-11T21:44:38.763 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:44:38.763 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 9 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fddbc00e850 0x7fddbc0146e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 9 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 4 9 2024-03-11T21:44:38.764 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 4 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 10 client.42209.objecter _send_command 9 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc017540 con 0x7fddbc00e850 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fddbc00e850 0x7fddbc0146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc00e850 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fddbc00e850 0x7fddbc0146e0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc017540 con 0x7fddbc00e850 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 10 client.42209.objecter _finish_command 9 = system:0 2024-03-11T21:44:38.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 4 2024-03-11T21:44:38.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.779+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 4 9 2024-03-11T21:44:38.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 10 2024-03-11T21:44:38.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 10 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc001470 osd=4 3 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 4 10 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 4 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 10 client.42209.objecter _send_command 10 2024-03-11T21:44:38.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.795+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc00ddc0 con 0x7fddbc00e850 2024-03-11T21:44:38.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc00ddc0 con 0x7fddbc00e850 2024-03-11T21:44:38.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 10 client.42209.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 4 2024-03-11T21:44:38.787 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 4 10 2024-03-11T21:44:38.787 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 11 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fddbc00ff10 0x7fddbc017db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 11 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 5 11 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 4 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 10 client.42209.objecter _send_command 11 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc00dd30 con 0x7fddbc00ff10 2024-03-11T21:44:38.788 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fddbc00ff10 0x7fddbc017db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fdddb7fe700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fddbc00ff10 0x7fddbc017db0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc00ff10 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc00dd30 con 0x7fddbc00ff10 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 10 client.42209.objecter _finish_command 11 = system:0 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 4 2024-03-11T21:44:38.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.803+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 5 11 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 12 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 12 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc00db70 osd=5 3 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 5 12 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 4 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 10 client.42209.objecter _send_command 12 2024-03-11T21:44:38.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.819+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc00dd30 con 0x7fddbc00ff10 2024-03-11T21:44:38.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc00dd30 con 0x7fddbc00ff10 2024-03-11T21:44:38.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 10 client.42209.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 4 2024-03-11T21:44:38.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 5 12 2024-03-11T21:44:38.811 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:44:38.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 13 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fddbc018800 0x7fddbc018c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 13 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 6 13 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 4 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 10 client.42209.objecter _send_command 13 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc01ddf0 con 0x7fddbc018800 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fddbc018800 0x7fddbc018c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fdde0ae6700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fddbc018800 0x7fddbc018c40 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc018800 2024-03-11T21:44:38.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc01ddf0 con 0x7fddbc018800 2024-03-11T21:44:38.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:44:38.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 10 client.42209.objecter _finish_command 13 = system:0 2024-03-11T21:44:38.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 4 2024-03-11T21:44:38.814 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.827+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 6 13 2024-03-11T21:44:38.831 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 14 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 14 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc00dd30 osd=6 3 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 6 14 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 4 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 10 client.42209.objecter _send_command 14 2024-03-11T21:44:38.832 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.847+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc019290 con 0x7fddbc018800 2024-03-11T21:44:38.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:38.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.851+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc019290 con 0x7fddbc018800 2024-03-11T21:44:38.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.851+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.851+0000 7fddd97fa700 10 client.42209.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.851+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 4 2024-03-11T21:44:38.838 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.851+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 6 14 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 15 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fddbc019980 0x7fddbc019de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.839 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 15 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 7 15 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 4 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 10 client.42209.objecter _send_command 15 2024-03-11T21:44:38.840 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddbc01a4e0 con 0x7fddbc019980 2024-03-11T21:44:38.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fddbc019980 0x7fddbc019de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:38.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fddd97fa700 10 client.42209.objecter ms_handle_connect 0x7fddbc019980 2024-03-11T21:44:38.841 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.855+0000 7fdddbfff700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fddbc019980 0x7fddbc019de0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:38.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.859+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddbc01a4e0 con 0x7fddbc019980 2024-03-11T21:44:38.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.859+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:44:38.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.859+0000 7fddd97fa700 10 client.42209.objecter _finish_command 15 = system:0 2024-03-11T21:44:38.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.859+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 4 2024-03-11T21:44:38.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.859+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 7 15 2024-03-11T21:44:38.847 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.417798+0000 osd.0 (osd.0) 492 : cluster 0 3.17 scrub starts 2024-03-11T21:44:38.847 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.420155+0000 osd.0 (osd.0) 493 : cluster 0 3.17 scrub ok 2024-03-11T21:44:38.847 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.455594+0000 osd.7 (osd.7) 496 : cluster 0 9.e scrub starts 2024-03-11T21:44:38.847 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.460017+0000 osd.7 (osd.7) 497 : cluster 0 9.e scrub ok 2024-03-11T21:44:38.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.562432+0000 osd.3 (osd.3) 509 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:38.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.562850+0000 osd.3 (osd.3) 510 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:38.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:37.686847+0000 mgr.y (mgr.14550) 836 : cluster 0 pgmap v1450: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:38.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: cluster 2024-03-11T21:44:38.624244+0000 mon.a (mon.0) 4010 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-03-11T21:44:38.848 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:38 smithi203 bash[23654]: audit 2024-03-11T21:44:38.690344+0000 mon.c (mon.2) 1374 : audit 0 from='client.? 172.21.15.143:0/1910083281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 10 client.42209.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign -1 16 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 20 client.42209.objecter _get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_remove -1 16 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 20 client.42209.objecter get_session s=0x7fddbc019290 osd=7 3 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 15 client.42209.objecter _session_command_op_assign 7 16 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 4 2024-03-11T21:44:38.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 10 client.42209.objecter _send_command 16 2024-03-11T21:44:38.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.875+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fddbc01a4e0 con 0x7fddbc019980 2024-03-11T21:44:38.862 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddd97fa700 1 -- 172.21.15.143:0/1910083281 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fddbc01a4e0 con 0x7fddbc019980 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddd97fa700 10 client.42209.objecter ms_dispatch 0x7fdddc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddd97fa700 10 client.42209.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddd97fa700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 4 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddd97fa700 15 client.42209.objecter _session_command_op_remove 7 16 2024-03-11T21:44:38.864 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 librados: watch_flush enter 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 librados: watch_flush exit 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.0 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fddbc004730 msgr2=0x7fddbc004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fddbc004730 0x7fddbc004b70 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0053e0 osd=0 1 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.1 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fddbc002750 msgr2=0x7fddbc002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fddbc002750 0x7fddbc002bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc004530 osd=1 1 2024-03-11T21:44:38.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.2 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fddbc00b240 msgr2=0x7fddbc00d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fddbc00b240 0x7fddbc00d630 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc008140 osd=2 1 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.3 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fddbc002090 msgr2=0x7fddbc00ecf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fddbc002090 0x7fddbc00ecf0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc0037c0 osd=3 1 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.4 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fddbc00e850 msgr2=0x7fddbc0146e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fddbc00e850 0x7fddbc0146e0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc001470 osd=4 1 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.5 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fddbc00ff10 msgr2=0x7fddbc017db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fddbc00ff10 0x7fddbc017db0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00db70 osd=5 1 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fdde0ae6700 1 -- 172.21.15.143:0/1910083281 reap_dead start 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.6 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fddbc018800 msgr2=0x7fddbc018c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fddbc018800 0x7fddbc018c40 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fdde0ae6700 1 -- 172.21.15.143:0/1910083281 reap_dead start 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc00dd30 osd=6 1 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter close_session for osd.7 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fddbc019980 msgr2=0x7fddbc019de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fddbc019980 0x7fddbc019de0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter put_session s=0x7fddbc019290 osd=7 1 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 20 client.42209.objecter shutdown clearing up homeless session... 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 10 client.42209.objecter successfully canceled tick 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fdde0ae6700 1 -- 172.21.15.143:0/1910083281 reap_dead start 2024-03-11T21:44:38.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 msgr2=0x7fddc4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 0x7fddc4071ce0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7fdddc198b10 tx=0x7fddcc009260 comp rx=0 tx=0).stop 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 msgr2=0x7fdddc19dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fddd000c8c0 tx=0x7fddd000cbd0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 shutdown_connections 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdddc06b050 0x7fdddc19dd50 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 --2- 172.21.15.143:0/1910083281 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc406f820 0x7fddc4071ce0 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 >> 172.21.15.143:0/1910083281 conn(0x7fdddc077a40 msgr2=0x7fdddc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 shutdown_connections 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 -- 172.21.15.143:0/1910083281 wait complete. 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:38.879+0000 7fddc2ffd700 1 librados: shutdown 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:38.869 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:38.870 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:38.870 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:38.870 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:38.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.985 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:39.114 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:39.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.245 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:39.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:39.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:39.377 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:39.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:44:39.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:39.503 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:39.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.636 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:39.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: cluster 2024-03-11T21:44:38.158019+0000 osd.3 (osd.3) 511 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: cluster 2024-03-11T21:44:38.164097+0000 osd.3 (osd.3) 512 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: cluster 2024-03-11T21:44:38.710719+0000 osd.6 (osd.6) 415 : cluster 0 9.18 scrub starts 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: cluster 2024-03-11T21:44:38.714438+0000 osd.6 (osd.6) 416 : cluster 0 9.18 scrub ok 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: audit 2024-03-11T21:44:39.157755+0000 mgr.y (mgr.14550) 837 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:39.751 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.624859+0000 mon.a (mon.0) 4011 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: cluster 2024-03-11T21:44:38.158019+0000 osd.3 (osd.3) 511 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: cluster 2024-03-11T21:44:38.164097+0000 osd.3 (osd.3) 512 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: cluster 2024-03-11T21:44:38.710719+0000 osd.6 (osd.6) 415 : cluster 0 9.18 scrub starts 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: cluster 2024-03-11T21:44:38.714438+0000 osd.6 (osd.6) 416 : cluster 0 9.18 scrub ok 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: audit 2024-03-11T21:44:39.157755+0000 mgr.y (mgr.14550) 837 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:39.752 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:39 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.624859+0000 mon.a (mon.0) 4011 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-03-11T21:44:39.763 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: cluster 2024-03-11T21:44:38.158019+0000 osd.3 (osd.3) 511 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: cluster 2024-03-11T21:44:38.164097+0000 osd.3 (osd.3) 512 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: cluster 2024-03-11T21:44:38.710719+0000 osd.6 (osd.6) 415 : cluster 0 9.18 scrub starts 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: cluster 2024-03-11T21:44:38.714438+0000 osd.6 (osd.6) 416 : cluster 0 9.18 scrub ok 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: audit 2024-03-11T21:44:39.157755+0000 mgr.y (mgr.14550) 837 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:39.878 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:39 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.624859+0000 mon.a (mon.0) 4011 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-03-11T21:44:39.893 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:39.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.021 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:40.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:40.152 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:40.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:40.282 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:40.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:40.414 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:40.517 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:40.544 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:40.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.674 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:40.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.805 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:40.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.938 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:40.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.413833+0000 osd.0 (osd.0) 494 : cluster 0 9.1d scrub starts 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.418925+0000 osd.0 (osd.0) 495 : cluster 0 9.1d scrub ok 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.687373+0000 mgr.y (mgr.14550) 838 : cluster 0 pgmap v1453: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.025367+0000 osd.1 (osd.1) 313 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.025970+0000 osd.1 (osd.1) 314 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.633567+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.413833+0000 osd.0 (osd.0) 494 : cluster 0 9.1d scrub starts 2024-03-11T21:44:40.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.418925+0000 osd.0 (osd.0) 495 : cluster 0 9.1d scrub ok 2024-03-11T21:44:40.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.687373+0000 mgr.y (mgr.14550) 838 : cluster 0 pgmap v1453: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:44:40.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.025367+0000 osd.1 (osd.1) 313 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:40.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.025970+0000 osd.1 (osd.1) 314 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:40.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:40 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.633567+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-03-11T21:44:41.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.413833+0000 osd.0 (osd.0) 494 : cluster 0 9.1d scrub starts 2024-03-11T21:44:41.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.418925+0000 osd.0 (osd.0) 495 : cluster 0 9.1d scrub ok 2024-03-11T21:44:41.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.687373+0000 mgr.y (mgr.14550) 838 : cluster 0 pgmap v1453: 216 pgs: 6 unknown, 210 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:44:41.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.025367+0000 osd.1 (osd.1) 313 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:41.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.025970+0000 osd.1 (osd.1) 314 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:41.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:40 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.633567+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-03-11T21:44:41.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:41.066 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:41.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:41.197 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:41.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.329 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:41.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 -- 172.21.15.143:0/4009554495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac069b10 msgr2=0x7f70ac069f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 --2- 172.21.15.143:0/4009554495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac069b10 0x7f70ac069f70 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f709c009990 tx=0x7f709c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 -- 172.21.15.143:0/4009554495 shutdown_connections 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 --2- 172.21.15.143:0/4009554495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 0x7f70ac108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 --2- 172.21.15.143:0/4009554495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac06a4b0 0x7f70ac10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 --2- 172.21.15.143:0/4009554495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac069b10 0x7f70ac069f70 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 -- 172.21.15.143:0/4009554495 >> 172.21.15.143:0/4009554495 conn(0x7f70ac076d80 msgr2=0x7f70ac077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:41.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 -- 172.21.15.143:0/4009554495 shutdown_connections 2024-03-11T21:44:41.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 -- 172.21.15.143:0/4009554495 wait complete. 2024-03-11T21:44:41.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 librados: starting msgr at 2024-03-11T21:44:41.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.463+0000 7f70b3943700 1 librados: starting objecter 2024-03-11T21:44:41.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 Processor -- start 2024-03-11T21:44:41.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- start start 2024-03-11T21:44:41.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 librados: setting wanted keys 2024-03-11T21:44:41.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 librados: calling monclient init 2024-03-11T21:44:41.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac069b10 0x7f70ac199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 0x7f70ac19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac111610 con 0x7f70ac107e90 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac111490 con 0x7f70ac069b10 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac111790 con 0x7f70ac06a4b0 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37296/0 (socket says 172.21.15.143:37296) 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 -- 172.21.15.143:0/3835841255 learned_addr learned my addr 172.21.15.143:0/3835841255 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b1ee0700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 0x7f70ac19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b16df700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac069b10 0x7f70ac199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 -- 172.21.15.143:0/3835841255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac069b10 msgr2=0x7f70ac199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac069b10 0x7f70ac199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 -- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 msgr2=0x7f70ac19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:41.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 0x7f70ac19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 -- 172.21.15.143:0/3835841255 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac19ee20 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b0ede700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 10 client.?.objecter ms_handle_connect 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c027070 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f709c022470 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac19f0b0 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c017650 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 10 client.42215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac1a6990 con 0x7f70ac06a4b0 2024-03-11T21:44:41.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 librados: init done 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac04faa0 con 0x7f70ac06a4b0 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f709c02a070 con 0x7f70ac06a4b0 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.467+0000 7f70a27fc700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 0x7f7098071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(927..927 src has 842..927) v4 ==== 9082+0+0 (secure 0 0 0) 0x7f709c08f790 con 0x7f70ac06a4b0 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 10 client.42215.objecter ms_dispatch 0x7f70ac001550 osd_map(927..927 src has 842..927) v4 2024-03-11T21:44:41.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 3 client.42215.objecter handle_osd_map got epochs [927,927] > 0 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 3 client.42215.objecter handle_osd_map decoding full epoch 927 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70b16df700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 0x7f7098071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 20 client.42215.objecter dump_active .. 0 homeless 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70b16df700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 0x7f7098071ce0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009680 tx=0x7f70a8009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:44:41.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.471+0000 7f70a27fc700 10 client.42215.objecter ms_handle_connect 0x7f709806f820 2024-03-11T21:44:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:41.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.475+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f709c013070 con 0x7f70ac06a4b0 2024-03-11T21:44:41.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:41.580 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:41.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:41.723+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f70ac1a6b60 con 0x7f70ac06a4b0 2024-03-11T21:44:41.712 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:41.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:41.848 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:41.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:42.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.941292+0000 osd.4 (osd.4) 479 : cluster 0 4.1 scrub starts 2024-03-11T21:44:42.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:39.942494+0000 osd.4 (osd.4) 480 : cluster 0 4.1 scrub ok 2024-03-11T21:44:42.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.196048+0000 osd.2 (osd.2) 441 : cluster 0 2.c scrub starts 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.197679+0000 osd.2 (osd.2) 442 : cluster 0 2.c scrub ok 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.409669+0000 osd.0 (osd.0) 496 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:40.422488+0000 osd.0 (osd.0) 497 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.637857+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.941292+0000 osd.4 (osd.4) 479 : cluster 0 4.1 scrub starts 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:39.942494+0000 osd.4 (osd.4) 480 : cluster 0 4.1 scrub ok 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.196048+0000 osd.2 (osd.2) 441 : cluster 0 2.c scrub starts 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.197679+0000 osd.2 (osd.2) 442 : cluster 0 2.c scrub ok 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.409669+0000 osd.0 (osd.0) 496 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:44:42.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:40.422488+0000 osd.0 (osd.0) 497 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:44:42.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.637857+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-03-11T21:44:42.108 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:42.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.941292+0000 osd.4 (osd.4) 479 : cluster 0 4.1 scrub starts 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:39.942494+0000 osd.4 (osd.4) 480 : cluster 0 4.1 scrub ok 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.196048+0000 osd.2 (osd.2) 441 : cluster 0 2.c scrub starts 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.197679+0000 osd.2 (osd.2) 442 : cluster 0 2.c scrub ok 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.409669+0000 osd.0 (osd.0) 496 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:40.422488+0000 osd.0 (osd.0) 497 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:44:42.223 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:42 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.637857+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-03-11T21:44:42.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:42.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:42.352 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:42.367 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:42.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:42.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:42.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:42.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:42.629 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:42.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.651+0000 7f70a27fc700 1 -- 172.21.15.143:0/3835841255 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v929) v1 ==== 87+0+0 (secure 0 0 0) 0x7f709c05c680 con 0x7f70ac06a4b0 2024-03-11T21:44:42.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.655+0000 7f70b3943700 10 librados: watch_flush enter 2024-03-11T21:44:42.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.655+0000 7f70b3943700 10 librados: watch_flush exit 2024-03-11T21:44:42.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.655+0000 7f70b3943700 20 client.42215.objecter shutdown clearing up homeless session... 2024-03-11T21:44:42.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.655+0000 7f70b3943700 10 client.42215.objecter successfully canceled tick 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 msgr2=0x7f7098071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 0x7f7098071ce0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009680 tx=0x7f70a8009260 comp rx=0 tx=0).stop 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 msgr2=0x7f70ac19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c0047b0 comp rx=0 tx=0).stop 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 shutdown_connections 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f709806f820 0x7f7098071ce0 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ac107e90 0x7f70ac19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:42.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70ac069b10 0x7f70ac199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:42.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 --2- 172.21.15.143:0/3835841255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ac06a4b0 0x7f70ac19a350 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:44:42.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 >> 172.21.15.143:0/3835841255 conn(0x7f70ac076d80 msgr2=0x7f70ac10d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:44:42.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 shutdown_connections 2024-03-11T21:44:42.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 -- 172.21.15.143:0/3835841255 wait complete. 2024-03-11T21:44:42.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:44:42.659+0000 7f70b3943700 1 librados: shutdown 2024-03-11T21:44:42.649 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is set 2024-03-11T21:44:42.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.765 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:42.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:42.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:42.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:43.027 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:43.077 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:44:43.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:43.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.375004+0000 osd.0 (osd.0) 498 : cluster 0 6.9 scrub starts 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.376313+0000 osd.0 (osd.0) 499 : cluster 0 6.9 scrub ok 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.687821+0000 mgr.y (mgr.14550) 839 : cluster 0 pgmap v1456: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.712492+0000 osd.6 (osd.6) 417 : cluster 0 5.2 scrub starts 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:41.716428+0000 osd.6 (osd.6) 418 : cluster 0 5.2 scrub ok 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: audit 2024-03-11T21:44:41.727772+0000 mon.c (mon.2) 1375 : audit 1 from='client.? 172.21.15.143:0/3835841255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: audit 2024-03-11T21:44:42.094381+0000 mon.a (mon.0) 4014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:42.147740+0000 osd.2 (osd.2) 443 : cluster 0 5.b scrub starts 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:42.149862+0000 osd.2 (osd.2) 444 : cluster 0 5.b scrub ok 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:42.631438+0000 mon.a (mon.0) 4015 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:42.633603+0000 mon.a (mon.0) 4016 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: audit 2024-03-11T21:44:42.641441+0000 mon.a (mon.0) 4017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[21512]: cluster 2024-03-11T21:44:42.649112+0000 mon.a (mon.0) 4018 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.375004+0000 osd.0 (osd.0) 498 : cluster 0 6.9 scrub starts 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.376313+0000 osd.0 (osd.0) 499 : cluster 0 6.9 scrub ok 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.687821+0000 mgr.y (mgr.14550) 839 : cluster 0 pgmap v1456: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.712492+0000 osd.6 (osd.6) 417 : cluster 0 5.2 scrub starts 2024-03-11T21:44:43.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:41.716428+0000 osd.6 (osd.6) 418 : cluster 0 5.2 scrub ok 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: audit 2024-03-11T21:44:41.727772+0000 mon.c (mon.2) 1375 : audit 1 from='client.? 172.21.15.143:0/3835841255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: audit 2024-03-11T21:44:42.094381+0000 mon.a (mon.0) 4014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:42.147740+0000 osd.2 (osd.2) 443 : cluster 0 5.b scrub starts 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:42.149862+0000 osd.2 (osd.2) 444 : cluster 0 5.b scrub ok 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:42.631438+0000 mon.a (mon.0) 4015 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:42.633603+0000 mon.a (mon.0) 4016 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: audit 2024-03-11T21:44:42.641441+0000 mon.a (mon.0) 4017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:43 smithi143 bash[29284]: cluster 2024-03-11T21:44:42.649112+0000 mon.a (mon.0) 4018 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-03-11T21:44:43.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:43.285 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:43.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:43.400 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.375004+0000 osd.0 (osd.0) 498 : cluster 0 6.9 scrub starts 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.376313+0000 osd.0 (osd.0) 499 : cluster 0 6.9 scrub ok 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.687821+0000 mgr.y (mgr.14550) 839 : cluster 0 pgmap v1456: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.712492+0000 osd.6 (osd.6) 417 : cluster 0 5.2 scrub starts 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:41.716428+0000 osd.6 (osd.6) 418 : cluster 0 5.2 scrub ok 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: audit 2024-03-11T21:44:41.727772+0000 mon.c (mon.2) 1375 : audit 1 from='client.? 172.21.15.143:0/3835841255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: audit 2024-03-11T21:44:42.094381+0000 mon.a (mon.0) 4014 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:42.147740+0000 osd.2 (osd.2) 443 : cluster 0 5.b scrub starts 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:42.149862+0000 osd.2 (osd.2) 444 : cluster 0 5.b scrub ok 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:42.631438+0000 mon.a (mon.0) 4015 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:42.633603+0000 mon.a (mon.0) 4016 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: audit 2024-03-11T21:44:42.641441+0000 mon.a (mon.0) 4017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:43.401 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:43 smithi203 bash[23654]: cluster 2024-03-11T21:44:42.649112+0000 mon.a (mon.0) 4018 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-03-11T21:44:43.417 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:43.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:43.548 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:43.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:43.682 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:43.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:43.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:43.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:43.944 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:44.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:44.077 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:44.181 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:44.208 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:44.312 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:44.340 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:44.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:44.464 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:44.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:44.594 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:44.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.708 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:44 smithi143 bash[29284]: cluster 2024-03-11T21:44:43.652762+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-03-11T21:44:44.708 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:44 smithi143 bash[29284]: cluster 2024-03-11T21:44:43.688623+0000 mgr.y (mgr.14550) 840 : cluster 0 pgmap v1459: 216 pgs: 29 creating+peering, 3 unknown, 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:44.708 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:44 smithi143 bash[21512]: cluster 2024-03-11T21:44:43.652762+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-03-11T21:44:44.708 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:44 smithi143 bash[21512]: cluster 2024-03-11T21:44:43.688623+0000 mgr.y (mgr.14550) 840 : cluster 0 pgmap v1459: 216 pgs: 29 creating+peering, 3 unknown, 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:44.724 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:44.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.855 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:44.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.984 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:45.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:44 smithi203 bash[23654]: cluster 2024-03-11T21:44:43.652762+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-03-11T21:44:45.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:44 smithi203 bash[23654]: cluster 2024-03-11T21:44:43.688623+0000 mgr.y (mgr.14550) 840 : cluster 0 pgmap v1459: 216 pgs: 29 creating+peering, 3 unknown, 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:45.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:45.114 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:45.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:45.244 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:45.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:45.374 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:45.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:45.505 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:45.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:45.636 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:45.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:45.766 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:45.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:45.899 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:45.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:45 smithi143 bash[21512]: cluster 2024-03-11T21:44:44.663268+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-03-11T21:44:45.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:45 smithi143 bash[29284]: cluster 2024-03-11T21:44:44.663268+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-03-11T21:44:45.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:44:46.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.014 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:45 smithi203 bash[23654]: cluster 2024-03-11T21:44:44.663268+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-03-11T21:44:46.029 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:46.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.160 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:46.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.289 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:46.393 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:46.421 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:46.468 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:46.469 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:46.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:46.545 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:46.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:46.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[21512]: cluster 2024-03-11T21:44:45.663833+0000 mon.a (mon.0) 4021 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-03-11T21:44:46.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[21512]: cluster 2024-03-11T21:44:45.689234+0000 mgr.y (mgr.14550) 841 : cluster 0 pgmap v1462: 216 pgs: 29 creating+peering, 187 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:46.660 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[21512]: cluster 2024-03-11T21:44:46.621060+0000 mon.a (mon.0) 4022 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-03-11T21:44:46.675 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:46.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:46.804 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:46.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:46.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[29284]: cluster 2024-03-11T21:44:45.663833+0000 mon.a (mon.0) 4021 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-03-11T21:44:46.917 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[29284]: cluster 2024-03-11T21:44:45.689234+0000 mgr.y (mgr.14550) 841 : cluster 0 pgmap v1462: 216 pgs: 29 creating+peering, 187 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:46.918 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:46 smithi143 bash[29284]: cluster 2024-03-11T21:44:46.621060+0000 mon.a (mon.0) 4022 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-03-11T21:44:46.931 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:47.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:46 smithi203 bash[23654]: cluster 2024-03-11T21:44:45.663833+0000 mon.a (mon.0) 4021 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-03-11T21:44:47.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:46 smithi203 bash[23654]: cluster 2024-03-11T21:44:45.689234+0000 mgr.y (mgr.14550) 841 : cluster 0 pgmap v1462: 216 pgs: 29 creating+peering, 187 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:47.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:46 smithi203 bash[23654]: cluster 2024-03-11T21:44:46.621060+0000 mon.a (mon.0) 4022 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-03-11T21:44:47.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:47.062 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:47.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.190 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.319 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.449 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:47.578 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:47.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:47.707 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:47.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:47.969 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:48.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:48.099 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:48.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.229 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:48.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:48.358 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:48.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.488 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:48.494 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:48.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.616 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:48.733 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:48 smithi203 bash[23654]: cluster 2024-03-11T21:44:47.690436+0000 mgr.y (mgr.14550) 842 : cluster 0 pgmap v1465: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:48.733 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:48 smithi203 bash[23654]: cluster 2024-03-11T21:44:47.694931+0000 mon.a (mon.0) 4023 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-03-11T21:44:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:48.848 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:48.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:48 smithi143 bash[29284]: cluster 2024-03-11T21:44:47.690436+0000 mgr.y (mgr.14550) 842 : cluster 0 pgmap v1465: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:48.859 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:48 smithi143 bash[29284]: cluster 2024-03-11T21:44:47.694931+0000 mon.a (mon.0) 4023 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-03-11T21:44:48.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:48 smithi143 bash[21512]: cluster 2024-03-11T21:44:47.690436+0000 mgr.y (mgr.14550) 842 : cluster 0 pgmap v1465: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:48.860 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:48 smithi143 bash[21512]: cluster 2024-03-11T21:44:47.694931+0000 mon.a (mon.0) 4023 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-03-11T21:44:48.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:48.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:49.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:49.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:49.137 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:49.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:49.269 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:49.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:49.401 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:49.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:49.525 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:49.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:49.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:49.655 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:49.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:49.779 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:49.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:49.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:49.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[29284]: cluster 2024-03-11T21:44:48.696222+0000 osd.5 (osd.5) 511 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:49.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[29284]: cluster 2024-03-11T21:44:48.696496+0000 osd.5 (osd.5) 512 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[29284]: cluster 2024-03-11T21:44:48.704584+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[29284]: audit 2024-03-11T21:44:49.165647+0000 mgr.y (mgr.14550) 843 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[21512]: cluster 2024-03-11T21:44:48.696222+0000 osd.5 (osd.5) 511 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[21512]: cluster 2024-03-11T21:44:48.696496+0000 osd.5 (osd.5) 512 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[21512]: cluster 2024-03-11T21:44:48.704584+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-03-11T21:44:49.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:49 smithi143 bash[21512]: audit 2024-03-11T21:44:49.165647+0000 mgr.y (mgr.14550) 843 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:50.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:50.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:49 smithi203 bash[23654]: cluster 2024-03-11T21:44:48.696222+0000 osd.5 (osd.5) 511 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:50.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:49 smithi203 bash[23654]: cluster 2024-03-11T21:44:48.696496+0000 osd.5 (osd.5) 512 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:50.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:49 smithi203 bash[23654]: cluster 2024-03-11T21:44:48.704584+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-03-11T21:44:50.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:49 smithi203 bash[23654]: audit 2024-03-11T21:44:49.165647+0000 mgr.y (mgr.14550) 843 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:50.036 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:50.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:50.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:50.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:50.422 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:50.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.551 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:50.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.683 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:50.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:50.814 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:50.917 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.929 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:50 smithi143 bash[21512]: cluster 2024-03-11T21:44:49.691009+0000 mgr.y (mgr.14550) 844 : cluster 0 pgmap v1467: 216 pgs: 27 creating+peering, 5 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:50.929 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:50 smithi143 bash[21512]: cluster 2024-03-11T21:44:49.710294+0000 mon.a (mon.0) 4025 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-03-11T21:44:50.929 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:50 smithi143 bash[29284]: cluster 2024-03-11T21:44:49.691009+0000 mgr.y (mgr.14550) 844 : cluster 0 pgmap v1467: 216 pgs: 27 creating+peering, 5 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:50.929 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:50 smithi143 bash[29284]: cluster 2024-03-11T21:44:49.710294+0000 mon.a (mon.0) 4025 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-03-11T21:44:50.945 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:50 smithi203 bash[23654]: cluster 2024-03-11T21:44:49.691009+0000 mgr.y (mgr.14550) 844 : cluster 0 pgmap v1467: 216 pgs: 27 creating+peering, 5 unknown, 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:50 smithi203 bash[23654]: cluster 2024-03-11T21:44:49.710294+0000 mon.a (mon.0) 4025 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-03-11T21:44:51.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:51.074 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:51.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:51.207 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:51.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:51.467 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:51.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:51.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:51.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.729 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:51.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:51 smithi143 bash[29284]: cluster 2024-03-11T21:44:50.707073+0000 mon.a (mon.0) 4026 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-03-11T21:44:51.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:51 smithi143 bash[29284]: cluster 2024-03-11T21:44:51.615850+0000 mon.a (mon.0) 4027 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:51.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:51 smithi143 bash[21512]: cluster 2024-03-11T21:44:50.707073+0000 mon.a (mon.0) 4026 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-03-11T21:44:51.844 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:51 smithi143 bash[21512]: cluster 2024-03-11T21:44:51.615850+0000 mon.a (mon.0) 4027 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:51.856 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:51.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:51.988 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:52.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:51 smithi203 bash[23654]: cluster 2024-03-11T21:44:50.707073+0000 mon.a (mon.0) 4026 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-03-11T21:44:52.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:51 smithi203 bash[23654]: cluster 2024-03-11T21:44:51.615850+0000 mon.a (mon.0) 4027 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:52.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:52.118 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:52.195 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:44:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:52.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:52.250 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:52.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:52.381 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:52.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:52.512 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:52.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:52.642 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:52.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:52.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[29284]: cluster 2024-03-11T21:44:51.692316+0000 mgr.y (mgr.14550) 845 : cluster 0 pgmap v1470: 216 pgs: 27 creating+peering, 189 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:44:52.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[29284]: cluster 2024-03-11T21:44:51.734959+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-03-11T21:44:52.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[29284]: audit 2024-03-11T21:44:52.696318+0000 mon.a (mon.0) 4029 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:52.755 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[29284]: audit 2024-03-11T21:44:52.701228+0000 mon.c (mon.2) 1376 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:52.755 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[21512]: cluster 2024-03-11T21:44:51.692316+0000 mgr.y (mgr.14550) 845 : cluster 0 pgmap v1470: 216 pgs: 27 creating+peering, 189 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:44:52.756 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[21512]: cluster 2024-03-11T21:44:51.734959+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-03-11T21:44:52.756 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[21512]: audit 2024-03-11T21:44:52.696318+0000 mon.a (mon.0) 4029 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:52.756 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:52 smithi143 bash[21512]: audit 2024-03-11T21:44:52.701228+0000 mon.c (mon.2) 1376 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:52.767 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:52.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:52.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:52 smithi203 bash[23654]: cluster 2024-03-11T21:44:51.692316+0000 mgr.y (mgr.14550) 845 : cluster 0 pgmap v1470: 216 pgs: 27 creating+peering, 189 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:44:52.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:52 smithi203 bash[23654]: cluster 2024-03-11T21:44:51.734959+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-03-11T21:44:52.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:52 smithi203 bash[23654]: audit 2024-03-11T21:44:52.696318+0000 mon.a (mon.0) 4029 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:52.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:52 smithi203 bash[23654]: audit 2024-03-11T21:44:52.701228+0000 mon.c (mon.2) 1376 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:52.897 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:53.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:53.028 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:53.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.155 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:53.259 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:53.284 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:53.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.416 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:53.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:53.549 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:53.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:53.673 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:53.777 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:53.798 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:53.885 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:53 smithi143 bash[29284]: cluster 2024-03-11T21:44:52.738330+0000 mon.a (mon.0) 4030 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-03-11T21:44:53.885 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:53 smithi143 bash[21512]: cluster 2024-03-11T21:44:52.738330+0000 mon.a (mon.0) 4030 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-03-11T21:44:53.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:53.926 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:54.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:54.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:53 smithi203 bash[23654]: cluster 2024-03-11T21:44:52.738330+0000 mon.a (mon.0) 4030 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-03-11T21:44:54.056 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:54.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:54.187 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:54.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:54.317 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:54.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:54.449 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:54.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:54.580 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:54.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:54.710 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:54.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:54.835 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:54.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:54.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.105112+0000 osd.0 (osd.0) 500 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:54.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.105605+0000 osd.0 (osd.0) 501 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:54.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.513775+0000 osd.2 (osd.2) 445 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.514213+0000 osd.2 (osd.2) 446 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.693394+0000 mgr.y (mgr.14550) 846 : cluster 0 pgmap v1473: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[21512]: cluster 2024-03-11T21:44:53.770674+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.105112+0000 osd.0 (osd.0) 500 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.105605+0000 osd.0 (osd.0) 501 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.513775+0000 osd.2 (osd.2) 445 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.514213+0000 osd.2 (osd.2) 446 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.693394+0000 mgr.y (mgr.14550) 846 : cluster 0 pgmap v1473: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:54.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:54 smithi143 bash[29284]: cluster 2024-03-11T21:44:53.770674+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-03-11T21:44:54.965 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.105112+0000 osd.0 (osd.0) 500 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.105605+0000 osd.0 (osd.0) 501 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.513775+0000 osd.2 (osd.2) 445 : cluster 0 purged_snaps scrub starts 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.514213+0000 osd.2 (osd.2) 446 : cluster 0 purged_snaps scrub ok 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.693394+0000 mgr.y (mgr.14550) 846 : cluster 0 pgmap v1473: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:44:55.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:54 smithi203 bash[23654]: cluster 2024-03-11T21:44:53.770674+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-03-11T21:44:55.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:55.097 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:55.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:55.227 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:55.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.357 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:55.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:55.487 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:55.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:55.619 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:55.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.748 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:55.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:55.876 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:55.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:56.006 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:56.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:55 smithi203 bash[23654]: cluster 2024-03-11T21:44:54.776329+0000 mon.a (mon.0) 4032 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-03-11T21:44:56.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:56.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:55 smithi143 bash[29284]: cluster 2024-03-11T21:44:54.776329+0000 mon.a (mon.0) 4032 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-03-11T21:44:56.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:55 smithi143 bash[21512]: cluster 2024-03-11T21:44:54.776329+0000 mon.a (mon.0) 4032 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-03-11T21:44:56.133 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:56.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.257 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:56.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:56.388 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:56.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:56.518 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:56.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:56.649 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:56.660 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:56.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:56.882 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.909 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:57.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.023 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:56 smithi203 bash[23654]: cluster 2024-03-11T21:44:55.694510+0000 mgr.y (mgr.14550) 847 : cluster 0 pgmap v1476: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:56 smithi203 bash[23654]: cluster 2024-03-11T21:44:55.794767+0000 mon.a (mon.0) 4033 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-03-11T21:44:57.024 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:56 smithi203 bash[23654]: cluster 2024-03-11T21:44:56.629464+0000 mon.a (mon.0) 4034 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-03-11T21:44:57.040 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:57.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[29284]: cluster 2024-03-11T21:44:55.694510+0000 mgr.y (mgr.14550) 847 : cluster 0 pgmap v1476: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[29284]: cluster 2024-03-11T21:44:55.794767+0000 mon.a (mon.0) 4033 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[29284]: cluster 2024-03-11T21:44:56.629464+0000 mon.a (mon.0) 4034 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[21512]: cluster 2024-03-11T21:44:55.694510+0000 mgr.y (mgr.14550) 847 : cluster 0 pgmap v1476: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[21512]: cluster 2024-03-11T21:44:55.794767+0000 mon.a (mon.0) 4033 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-03-11T21:44:57.155 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:56 smithi143 bash[21512]: cluster 2024-03-11T21:44:56.629464+0000 mon.a (mon.0) 4034 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-03-11T21:44:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:57.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:57.301 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:57.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:57.432 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:57.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:57.564 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:57.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:57.697 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:57.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:57.829 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:57.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:58.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:58.095 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:58.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:58.221 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:58.251 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:44:58.251 INFO:teuthology.orchestra.run.smithi143.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:44:58.251 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:44:58.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:58.349 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:58.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.471 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:58.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.597 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:58.658 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:58.659 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[29284]: cluster 2024-03-11T21:44:57.648551+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[29284]: cluster 2024-03-11T21:44:57.695274+0000 mgr.y (mgr.14550) 848 : cluster 0 pgmap v1480: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[29284]: audit 2024-03-11T21:44:58.077004+0000 mon.c (mon.2) 1377 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[29284]: audit 2024-03-11T21:44:58.608323+0000 mon.a (mon.0) 4036 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[29284]: audit 2024-03-11T21:44:58.618101+0000 mon.a (mon.0) 4037 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[21512]: cluster 2024-03-11T21:44:57.648551+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[21512]: cluster 2024-03-11T21:44:57.695274+0000 mgr.y (mgr.14550) 848 : cluster 0 pgmap v1480: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:58.681 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[21512]: audit 2024-03-11T21:44:58.077004+0000 mon.c (mon.2) 1377 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:58.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[21512]: audit 2024-03-11T21:44:58.608323+0000 mon.a (mon.0) 4036 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.682 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:58 smithi143 bash[21512]: audit 2024-03-11T21:44:58.618101+0000 mon.a (mon.0) 4037 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.721 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:58.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:58.852 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:58.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:58.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:58 smithi203 bash[23654]: cluster 2024-03-11T21:44:57.648551+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-03-11T21:44:58.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:58 smithi203 bash[23654]: cluster 2024-03-11T21:44:57.695274+0000 mgr.y (mgr.14550) 848 : cluster 0 pgmap v1480: 216 pgs: 28 unknown, 188 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:44:58.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:58 smithi203 bash[23654]: audit 2024-03-11T21:44:58.077004+0000 mon.c (mon.2) 1377 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:58.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:58 smithi203 bash[23654]: audit 2024-03-11T21:44:58.608323+0000 mon.a (mon.0) 4036 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.967 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:58 smithi203 bash[23654]: audit 2024-03-11T21:44:58.618101+0000 mon.a (mon.0) 4037 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:58.984 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:59.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.113 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:59.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.244 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:59.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.370 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:59.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.500 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:59.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:44:59.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:59.630 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:59.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.760 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:59.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.874 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[23654]: cluster 2024-03-11T21:44:58.651161+0000 mon.a (mon.0) 4038 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-03-11T21:44:59.874 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[23654]: audit 2024-03-11T21:44:59.173632+0000 mgr.y (mgr.14550) 849 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:59.874 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[23654]: audit 2024-03-11T21:44:59.216971+0000 mon.c (mon.2) 1378 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:59.875 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[23654]: audit 2024-03-11T21:44:59.218369+0000 mon.c (mon.2) 1379 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:59.875 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:44:59 smithi203 bash[23654]: audit 2024-03-11T21:44:59.226039+0000 mon.a (mon.0) 4039 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:59.889 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[21512]: cluster 2024-03-11T21:44:58.651161+0000 mon.a (mon.0) 4038 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[21512]: audit 2024-03-11T21:44:59.173632+0000 mgr.y (mgr.14550) 849 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[21512]: audit 2024-03-11T21:44:59.216971+0000 mon.c (mon.2) 1378 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[21512]: audit 2024-03-11T21:44:59.218369+0000 mon.c (mon.2) 1379 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[21512]: audit 2024-03-11T21:44:59.226039+0000 mon.a (mon.0) 4039 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:59.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[29284]: cluster 2024-03-11T21:44:58.651161+0000 mon.a (mon.0) 4038 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-03-11T21:44:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[29284]: audit 2024-03-11T21:44:59.173632+0000 mgr.y (mgr.14550) 849 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[29284]: audit 2024-03-11T21:44:59.216971+0000 mon.c (mon.2) 1378 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[29284]: audit 2024-03-11T21:44:59.218369+0000 mon.c (mon.2) 1379 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:59.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:44:59 smithi143 bash[29284]: audit 2024-03-11T21:44:59.226039+0000 mon.a (mon.0) 4039 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:44:59.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:00.020 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:00.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.151 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:00.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:00.280 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:00.383 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:00.410 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:00.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.532 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1798323932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 msgr2=0x7f3bc410dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1798323932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc410dc70 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b0d0 tx=0x7f3bbc00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1798323932 shutdown_connections 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1798323932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc410dc70 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1798323932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc4100fc0 0x7f3bc41013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1798323932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc41059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1798323932 >> 172.21.15.143:0/1798323932 conn(0x7f3bc4077de0 msgr2=0x7f3bc40781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1798323932 shutdown_connections 2024-03-11T21:45:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1798323932 wait complete. 2024-03-11T21:45:00.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 librados: starting msgr at 2024-03-11T21:45:00.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 librados: starting objecter 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 Processor -- start 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- start start 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 librados: setting wanted keys 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 librados: calling monclient init 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc4100fc0 0x7f3bc4102a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc4102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc41034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc41105c0 con 0x7f3bc4101970 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc4110440 con 0x7f3bc4100fc0 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc4110740 con 0x7f3bc41060a0 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc41034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc4102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc41034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49674/0 (socket says 172.21.15.143:49674) 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 learned_addr learned my addr 172.21.15.143:0/1060525798 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc4102fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40962/0 (socket says 172.21.15.143:40962) 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc4100fc0 msgr2=0x7f3bc4102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc4100fc0 0x7f3bc4102a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 msgr2=0x7f3bc4102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc4102fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc41a6230 con 0x7f3bc41060a0 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bc4101970 0x7f3bc4102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc41034e0 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc006070 tx=0x7f3bbc007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc0ff9700 10 client.?.objecter ms_handle_connect 0x7f3bc41060a0 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc0ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc021020 con 0x7f3bc41060a0 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3bbc004c50 con 0x7f3bc41060a0 2024-03-11T21:45:00.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc41a64c0 con 0x7f3bc41060a0 2024-03-11T21:45:00.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc005050 con 0x7f3bc41060a0 2024-03-11T21:45:00.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 10 client.42239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:00.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc41a69c0 con 0x7f3bc41060a0 2024-03-11T21:45:00.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.607+0000 7f3bc955f700 1 librados: init done 2024-03-11T21:45:00.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3bc404faa0 con 0x7f3bc41060a0 2024-03-11T21:45:00.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3bbc024020 con 0x7f3bc41060a0 2024-03-11T21:45:00.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3bb006f820 0x7f3bb0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3bb006f820 0x7f3bb0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.598 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(946..946 src has 842..946) v4 ==== 9161+0+0 (secure 0 0 0) 0x7f3bbc00f020 con 0x7f3bc41060a0 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 osd_map(946..946 src has 842..946) v4 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 3 client.42239.objecter handle_osd_map got epochs [946,946] > 0 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 3 client.42239.objecter handle_osd_map decoding full epoch 946 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 20 client.42239.objecter dump_active .. 0 homeless 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v946) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3bbc090080 con 0x7f3bc41060a0 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 10 client.42239.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 1 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3bc40629f0 0x7f3bc4062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 1 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 0 1 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 4 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 10 client.42239.objecter _send_command 1 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41a9db0 con 0x7f3bc40629f0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bb006f820 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3bb006f820 0x7f3bb0071ce0 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40097b0 tx=0x7f3bb4006d20 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3bc40629f0 0x7f3bc4062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc40629f0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3bc40629f0 0x7f3bc4062dd0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41a9db0 con 0x7f3bc40629f0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 1 = system:0 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 4 2024-03-11T21:45:00.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.611+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 0 1 2024-03-11T21:45:00.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 10 client.42239.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 2 2024-03-11T21:45:00.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 2 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4103f90 osd=0 3 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 0 2 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 4 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 10 client.42239.objecter _send_command 2 2024-03-11T21:45:00.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc4062470 con 0x7f3bc40629f0 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc4062470 con 0x7f3bc40629f0 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 4 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 0 2 2024-03-11T21:45:00.617 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:45:00.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 10 client.42239.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 3 2024-03-11T21:45:00.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3bc402c6b0 0x7f3bc402cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 1 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 4 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 10 client.42239.objecter _send_command 3 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc4109a80 con 0x7f3bc402c6b0 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3bc402c6b0 0x7f3bc402cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3bc402c6b0 0x7f3bc402cab0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.631+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc402c6b0 2024-03-11T21:45:00.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.635+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc4109a80 con 0x7f3bc402c6b0 2024-03-11T21:45:00.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.635+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:45:00.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.635+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 3 = system:0 2024-03-11T21:45:00.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.635+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 4 2024-03-11T21:45:00.620 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.635+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 1 3 2024-03-11T21:45:00.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 10 client.42239.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 4 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 4 2024-03-11T21:45:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4109400 osd=1 3 2024-03-11T21:45:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 1 4 2024-03-11T21:45:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 4 2024-03-11T21:45:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 10 client.42239.objecter _send_command 4 2024-03-11T21:45:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.655+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc4108e80 con 0x7f3bc402c6b0 2024-03-11T21:45:00.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.663+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc4108e80 con 0x7f3bc402c6b0 2024-03-11T21:45:00.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.663+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.663+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.663+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 4 2024-03-11T21:45:00.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.663+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 1 4 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 10 client.42239.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 5 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3bc41abf70 0x7f3bc41ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 5 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3bc41abf70 0x7f3bc41ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 2 5 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 4 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 10 client.42239.objecter _send_command 5 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc402dcd0 con 0x7f3bc41abf70 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41abf70 2024-03-11T21:45:00.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.667+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3bc41abf70 0x7f3bc41ae360 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.671+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc402dcd0 con 0x7f3bc41abf70 2024-03-11T21:45:00.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.671+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:45:00.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.671+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 5 = system:0 2024-03-11T21:45:00.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.671+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 4 2024-03-11T21:45:00.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.671+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 2 5 2024-03-11T21:45:00.661 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 10 client.42239.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 6 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 6 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc4109280 osd=2 3 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 2 6 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 4 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 10 client.42239.objecter _send_command 6 2024-03-11T21:45:00.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc402d240 con 0x7f3bc41abf70 2024-03-11T21:45:00.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc402d240 con 0x7f3bc41abf70 2024-03-11T21:45:00.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.680 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 4 2024-03-11T21:45:00.681 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 2 6 2024-03-11T21:45:00.681 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 10 client.42239.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 7 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3bc41a9910 0x7f3bc41b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 7 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 3 7 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 4 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 10 client.42239.objecter _send_command 7 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41b4a10 con 0x7f3bc41a9910 2024-03-11T21:45:00.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3bc41a9910 0x7f3bc41b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41a9910 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3bc41a9910 0x7f3bc41b1a30 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41b4a10 con 0x7f3bc41a9910 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 7 = system:0 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 4 2024-03-11T21:45:00.683 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.695+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 3 7 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 10 client.42239.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 8 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 8 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc402d640 osd=3 3 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 3 8 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 4 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 10 client.42239.objecter _send_command 8 2024-03-11T21:45:00.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc41a8e80 con 0x7f3bc41a9910 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc41a8e80 con 0x7f3bc41a9910 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 4 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 3 8 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 10 client.42239.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 9 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3bc41af580 0x7f3bc41b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 9 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 4 9 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 4 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 10 client.42239.objecter _send_command 9 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41a8da0 con 0x7f3bc41af580 2024-03-11T21:45:00.703 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.715+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3bc41af580 0x7f3bc41b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41af580 2024-03-11T21:45:00.704 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3bc41af580 0x7f3bc41b2010 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41a8da0 con 0x7f3bc41af580 2024-03-11T21:45:00.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:45:00.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 9 = system:0 2024-03-11T21:45:00.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 4 2024-03-11T21:45:00.705 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.719+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 4 9 2024-03-11T21:45:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 10 client.42239.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 10 2024-03-11T21:45:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 10 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41a8c30 osd=4 3 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 4 10 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 4 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 10 client.42239.objecter _send_command 10 2024-03-11T21:45:00.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.735+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc41a8da0 con 0x7f3bc41af580 2024-03-11T21:45:00.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.739+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc41a8da0 con 0x7f3bc41af580 2024-03-11T21:45:00.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.739+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.739+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.739+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 4 2024-03-11T21:45:00.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.739+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 4 10 2024-03-11T21:45:00.727 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:45:00.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 10 client.42239.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 11 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3bc41b2a60 0x7f3bc41b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 11 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 5 11 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 4 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 10 client.42239.objecter _send_command 11 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41bb5c0 con 0x7f3bc41b2a60 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3bc41b2a60 0x7f3bc41b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bbbfff700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3bc41b2a60 0x7f3bc41b2e80 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41b2a60 2024-03-11T21:45:00.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41bb5c0 con 0x7f3bc41b2a60 2024-03-11T21:45:00.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:45:00.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 11 = system:0 2024-03-11T21:45:00.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 4 2024-03-11T21:45:00.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.743+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 5 11 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 10 client.42239.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 12 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 12 2024-03-11T21:45:00.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41a8da0 osd=5 3 2024-03-11T21:45:00.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 5 12 2024-03-11T21:45:00.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 4 2024-03-11T21:45:00.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 10 client.42239.objecter _send_command 12 2024-03-11T21:45:00.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.763+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc41b6c60 con 0x7f3bc41b2a60 2024-03-11T21:45:00.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc41b6c60 con 0x7f3bc41b2a60 2024-03-11T21:45:00.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 4 2024-03-11T21:45:00.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 5 12 2024-03-11T21:45:00.752 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 10 client.42239.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 13 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3bc41b72e0 0x7f3bc41b7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 13 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 6 13 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 4 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 10 client.42239.objecter _send_command 13 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41b7e10 con 0x7f3bc41b72e0 2024-03-11T21:45:00.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3bc41b72e0 0x7f3bc41b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc37fe700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3bc41b72e0 0x7f3bc41b7720 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41b72e0 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41b7e10 con 0x7f3bc41b72e0 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 13 = system:0 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 4 2024-03-11T21:45:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.767+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 6 13 2024-03-11T21:45:00.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:00.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 10 client.42239.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 14 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 14 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41b6c60 osd=6 3 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 6 14 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 4 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 10 client.42239.objecter _send_command 14 2024-03-11T21:45:00.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.787+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc41b7e10 con 0x7f3bc41b72e0 2024-03-11T21:45:00.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc41b7e10 con 0x7f3bc41b72e0 2024-03-11T21:45:00.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 4 2024-03-11T21:45:00.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 6 14 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 10 client.42239.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 15 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3bc41b7f80 0x7f3bc41be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 15 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 7 15 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 4 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 10 client.42239.objecter _send_command 15 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bc41c19a0 con 0x7f3bc41b7f80 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3bc41b7f80 0x7f3bc41be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc0ff9700 10 client.42239.objecter ms_handle_connect 0x7f3bc41b7f80 2024-03-11T21:45:00.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.791+0000 7f3bc2ffd700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3bc41b7f80 0x7f3bc41be970 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:00.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.795+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bc41c19a0 con 0x7f3bc41b7f80 2024-03-11T21:45:00.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.795+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:45:00.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.795+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 15 = system:0 2024-03-11T21:45:00.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.795+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 4 2024-03-11T21:45:00.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.795+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 7 15 2024-03-11T21:45:00.785 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[29284]: cluster 2024-03-11T21:44:59.658654+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[29284]: cluster 2024-03-11T21:44:59.696035+0000 mgr.y (mgr.14550) 850 : cluster 0 pgmap v1483: 216 pgs: 216 active+clean; 584 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[29284]: audit 2024-03-11T21:45:00.612426+0000 mon.c (mon.2) 1380 : audit 0 from='client.? 172.21.15.143:0/1060525798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:00.656087+0000 mon.a (mon.0) 4041 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[21512]: cluster 2024-03-11T21:44:59.658654+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[21512]: cluster 2024-03-11T21:44:59.696035+0000 mgr.y (mgr.14550) 850 : cluster 0 pgmap v1483: 216 pgs: 216 active+clean; 584 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[21512]: audit 2024-03-11T21:45:00.612426+0000 mon.c (mon.2) 1380 : audit 0 from='client.? 172.21.15.143:0/1060525798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:00.795 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:00.656087+0000 mon.a (mon.0) 4041 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 10 client.42239.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign -1 16 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 20 client.42239.objecter _get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_remove -1 16 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 20 client.42239.objecter get_session s=0x7f3bc41b7e10 osd=7 3 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 15 client.42239.objecter _session_command_op_assign 7 16 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 4 2024-03-11T21:45:00.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 10 client.42239.objecter _send_command 16 2024-03-11T21:45:00.799 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.811+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3bc41c0770 con 0x7f3bc41b7f80 2024-03-11T21:45:00.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.815+0000 7f3bc0ff9700 1 -- 172.21.15.143:0/1060525798 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bc41c0770 con 0x7f3bc41b7f80 2024-03-11T21:45:00.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.815+0000 7f3bc0ff9700 10 client.42239.objecter ms_dispatch 0x7f3bc4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:00.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.815+0000 7f3bc0ff9700 10 client.42239.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:00.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.815+0000 7f3bc0ff9700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 4 2024-03-11T21:45:00.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.815+0000 7f3bc0ff9700 15 client.42239.objecter _session_command_op_remove 7 16 2024-03-11T21:45:00.803 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:45:00.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 librados: watch_flush enter 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 librados: watch_flush exit 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.0 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3bc40629f0 msgr2=0x7f3bc4062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f3bc40629f0 0x7f3bc4062dd0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4103f90 osd=0 1 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.1 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3bc402c6b0 msgr2=0x7f3bc402cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f3bc402c6b0 0x7f3bc402cab0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.804 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109400 osd=1 1 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.2 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3bc41abf70 msgr2=0x7f3bc41ae360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f3bc41abf70 0x7f3bc41ae360 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc4109280 osd=2 1 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.3 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3bc41a9910 msgr2=0x7f3bc41b1a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f3bc41a9910 0x7f3bc41b1a30 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc402d640 osd=3 1 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.4 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3bc41af580 msgr2=0x7f3bc41b2010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f3bc41af580 0x7f3bc41b2010 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8c30 osd=4 1 2024-03-11T21:45:00.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.5 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3bc41b2a60 msgr2=0x7f3bc41b2e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f3bc41b2a60 0x7f3bc41b2e80 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41a8da0 osd=5 1 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.6 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3bc41b72e0 msgr2=0x7f3bc41b7720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f3bc41b72e0 0x7f3bc41b7720 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b6c60 osd=6 1 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter close_session for osd.7 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3bc41b7f80 msgr2=0x7f3bc41be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f3bc41b7f80 0x7f3bc41be970 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter put_session s=0x7f3bc41b7e10 osd=7 1 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 20 client.42239.objecter shutdown clearing up homeless session... 2024-03-11T21:45:00.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 10 client.42239.objecter successfully canceled tick 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 reap_dead start 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 reap_dead start 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3bb006f820 msgr2=0x7f3bb0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3bb006f820 0x7f3bb0071ce0 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40097b0 tx=0x7f3bb4006d20 comp rx=0 tx=0).stop 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 msgr2=0x7f3bc41034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 --2- 172.21.15.143:0/1060525798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bc41060a0 0x7f3bc41034e0 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc006070 tx=0x7f3bbc007a20 comp rx=0 tx=0).stop 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 reap_dead start 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 reap_dead start 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc37fe700 1 -- 172.21.15.143:0/1060525798 reap_dead start 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 shutdown_connections 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 >> 172.21.15.143:0/1060525798 conn(0x7f3bc4077de0 msgr2=0x7f3bc406a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 shutdown_connections 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 -- 172.21.15.143:0/1060525798 wait complete. 2024-03-11T21:45:00.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:00.819+0000 7f3bc955f700 1 librados: shutdown 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:00.808 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:00.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:00.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:00.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:00 smithi203 bash[23654]: cluster 2024-03-11T21:44:59.658654+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-03-11T21:45:00.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:00 smithi203 bash[23654]: cluster 2024-03-11T21:44:59.696035+0000 mgr.y (mgr.14550) 850 : cluster 0 pgmap v1483: 216 pgs: 216 active+clean; 584 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-03-11T21:45:00.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:00 smithi203 bash[23654]: audit 2024-03-11T21:45:00.612426+0000 mon.c (mon.2) 1380 : audit 0 from='client.? 172.21.15.143:0/1060525798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:00.918 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:00.656087+0000 mon.a (mon.0) 4041 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-03-11T21:45:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:01.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:01.038 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:01.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:01.163 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:01.266 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:01.293 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:01.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:01.424 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:01.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:01.554 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:01.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:01.693 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:01.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:01.823 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:01.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:01.951 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:02.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:02.079 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:02.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:02.208 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:02.243 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:02.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.340 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:02.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:45:02.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.482 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:02.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:02.614 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:02.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:02.739 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:02.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:02 smithi143 bash[29284]: cluster 2024-03-11T21:45:01.697825+0000 mgr.y (mgr.14550) 851 : cluster 0 pgmap v1486: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:02.853 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:02 smithi143 bash[29284]: cluster 2024-03-11T21:45:01.699197+0000 mon.a (mon.0) 4042 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-03-11T21:45:02.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:02 smithi143 bash[21512]: cluster 2024-03-11T21:45:01.697825+0000 mgr.y (mgr.14550) 851 : cluster 0 pgmap v1486: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:02.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:02 smithi143 bash[21512]: cluster 2024-03-11T21:45:01.699197+0000 mon.a (mon.0) 4042 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-03-11T21:45:02.868 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:02.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:02.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:02 smithi203 bash[23654]: cluster 2024-03-11T21:45:01.697825+0000 mgr.y (mgr.14550) 851 : cluster 0 pgmap v1486: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:02.983 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:02 smithi203 bash[23654]: cluster 2024-03-11T21:45:01.699197+0000 mon.a (mon.0) 4042 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-03-11T21:45:03.001 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:03.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:03.132 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:03.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:03.261 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:03.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:03.393 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:03.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:03.525 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:03.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:03.655 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:03.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:03.769 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:03 smithi143 bash[29284]: cluster 2024-03-11T21:45:02.691852+0000 mon.a (mon.0) 4043 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-03-11T21:45:03.769 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:03 smithi143 bash[29284]: cluster 2024-03-11T21:45:03.689842+0000 mon.a (mon.0) 4044 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-03-11T21:45:03.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:03 smithi143 bash[21512]: cluster 2024-03-11T21:45:02.691852+0000 mon.a (mon.0) 4043 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-03-11T21:45:03.770 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:03 smithi143 bash[21512]: cluster 2024-03-11T21:45:03.689842+0000 mon.a (mon.0) 4044 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-03-11T21:45:03.784 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:03.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:03.916 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:04.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:04.029 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:03 smithi203 bash[23654]: cluster 2024-03-11T21:45:02.691852+0000 mon.a (mon.0) 4043 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-03-11T21:45:04.030 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:03 smithi203 bash[23654]: cluster 2024-03-11T21:45:03.689842+0000 mon.a (mon.0) 4044 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-03-11T21:45:04.045 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:04.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:04.176 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:04.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:04.305 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:04.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:04.436 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:04.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.568 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:04.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:04.701 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:04.804 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.832 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:04.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:04.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:04 smithi143 bash[21512]: cluster 2024-03-11T21:45:03.698420+0000 mgr.y (mgr.14550) 852 : cluster 0 pgmap v1489: 216 pgs: 5 unknown, 211 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:45:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:04 smithi143 bash[21512]: cluster 2024-03-11T21:45:04.700773+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-03-11T21:45:04.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:04 smithi143 bash[29284]: cluster 2024-03-11T21:45:03.698420+0000 mgr.y (mgr.14550) 852 : cluster 0 pgmap v1489: 216 pgs: 5 unknown, 211 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:45:04.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:04 smithi143 bash[29284]: cluster 2024-03-11T21:45:04.700773+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-03-11T21:45:04.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:04 smithi203 bash[23654]: cluster 2024-03-11T21:45:03.698420+0000 mgr.y (mgr.14550) 852 : cluster 0 pgmap v1489: 216 pgs: 5 unknown, 211 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:45:04.947 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:04 smithi203 bash[23654]: cluster 2024-03-11T21:45:04.700773+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-03-11T21:45:04.964 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:05.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:05.092 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:05.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:05.220 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:05.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:05.352 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:05.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:05.477 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:05.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:05.606 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/3433066927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 msgr2=0x7f912c10db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- 172.21.15.143:0/3433066927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c10db70 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f912400b0a0 tx=0x7f912400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/3433066927 shutdown_connections 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- 172.21.15.143:0/3433066927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c106dc0 0x7f912c1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- 172.21.15.143:0/3433066927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c10db70 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- 172.21.15.143:0/3433066927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c100d40 0x7f912c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/3433066927 >> 172.21.15.143:0/3433066927 conn(0x7f912c076d80 msgr2=0x7f912c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:05.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/3433066927 shutdown_connections 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/3433066927 wait complete. 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 librados: starting msgr at 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 librados: starting objecter 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 Processor -- start 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- start start 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 librados: setting wanted keys 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 librados: calling monclient init 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 0x7f912c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c110570 con 0x7f912c100d40 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1103f0 con 0x7f912c106dc0 2024-03-11T21:45:05.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1106f0 con 0x7f912c101700 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51716/0 (socket says 172.21.15.143:51716) 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57516/0 (socket says 172.21.15.143:57516) 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 -- 172.21.15.143:0/2584948318 learned_addr learned my addr 172.21.15.143:0/2584948318 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912bfff700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 0x7f912c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 -- 172.21.15.143:0/2584948318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 msgr2=0x7f912c199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 0x7f912c199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 -- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 msgr2=0x7f912c19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 -- 172.21.15.143:0/2584948318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c19a240 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912bfff700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 0x7f912c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912b7fe700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f912affd700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f912000d950 tx=0x7f912000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9128ff9700 10 client.?.objecter ms_handle_connect 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9128ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120016070 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9120009dc0 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c19a530 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 10 client.42251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c1a67f0 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 librados: init done 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912000f3c0 con 0x7f912c101700 2024-03-11T21:45:05.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.659+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9118005150 con 0x7f912c101700 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9120019070 con 0x7f912c101700 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 0x7f9114071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(951..951 src has 842..951) v4 ==== 9186+0+0 (secure 0 0 0) 0x7f912008e710 con 0x7f912c101700 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 10 client.42251.objecter ms_dispatch 0x7f912c001550 osd_map(951..951 src has 842..951) v4 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 3 client.42251.objecter handle_osd_map got epochs [951,951] > 0 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 3 client.42251.objecter handle_osd_map decoding full epoch 951 2024-03-11T21:45:05.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f912b7fe700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 0x7f9114071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:05.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 20 client.42251.objecter dump_active .. 0 homeless 2024-03-11T21:45:05.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f9128ff9700 10 client.42251.objecter ms_handle_connect 0x7f911406f820 2024-03-11T21:45:05.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.663+0000 7f912b7fe700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 0x7f9114071ce0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f911c009680 tx=0x7f911c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:05.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.667+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f912005b590 con 0x7f912c101700 2024-03-11T21:45:05.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:05.737 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:05.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:05.864 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:05.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:05.899+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9118005440 con 0x7f912c101700 2024-03-11T21:45:05.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:05.995 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:06.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:45:06.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:06.125 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:06.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:06.255 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:06.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:06.384 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:06.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.514 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:06.617 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:06.627 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.639+0000 7f9128ff9700 1 -- 172.21.15.143:0/2584948318 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v953) v1 ==== 97+0+0 (secure 0 0 0) 0x7f912005efe0 con 0x7f912c101700 2024-03-11T21:45:06.632 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 10 librados: watch_flush enter 2024-03-11T21:45:06.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 10 librados: watch_flush exit 2024-03-11T21:45:06.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 20 client.42251.objecter shutdown clearing up homeless session... 2024-03-11T21:45:06.633 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 10 client.42251.objecter successfully canceled tick 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 msgr2=0x7f9114071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 0x7f9114071ce0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f911c009680 tx=0x7f911c009260 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 msgr2=0x7f912c19fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f912000d950 tx=0x7f912000dc60 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 shutdown_connections 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f911406f820 0x7f9114071ce0 secure :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f911c009680 tx=0x7f911c009260 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f912c100d40 0x7f912c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f912c106dc0 0x7f912c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 --2- 172.21.15.143:0/2584948318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f912c101700 0x7f912c19fe60 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:06.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.647+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 >> 172.21.15.143:0/2584948318 conn(0x7f912c076d80 msgr2=0x7f912c069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:06.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.651+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 shutdown_connections 2024-03-11T21:45:06.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.651+0000 7f9131d0e700 1 -- 172.21.15.143:0/2584948318 wait complete. 2024-03-11T21:45:06.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:06.651+0000 7f9131d0e700 1 librados: shutdown 2024-03-11T21:45:06.637 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is set 2024-03-11T21:45:06.654 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:06.720 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:45:06.720 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:45:06.721 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:45:06.721 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.653804 2024-03-11T21:45:06.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 0.6538037510086266 2024-03-11T21:45:06.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:06.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.912 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:07.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:07.037 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:07.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.159 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:07.160 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:07.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:07.290 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:07.393 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: cluster 2024-03-11T21:45:05.699136+0000 mgr.y (mgr.14550) 853 : cluster 0 pgmap v1492: 216 pgs: 216 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: cluster 2024-03-11T21:45:05.701278+0000 mon.a (mon.0) 4046 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: audit 2024-03-11T21:45:05.900461+0000 mon.c (mon.2) 1381 : audit 1 from='client.? 172.21.15.143:0/2584948318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: audit 2024-03-11T21:45:05.900824+0000 mon.a (mon.0) 4047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: cluster 2024-03-11T21:45:06.625373+0000 mon.a (mon.0) 4048 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: audit 2024-03-11T21:45:06.627860+0000 mon.a (mon.0) 4049 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:07.405 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[21512]: cluster 2024-03-11T21:45:06.636879+0000 mon.a (mon.0) 4050 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: cluster 2024-03-11T21:45:05.699136+0000 mgr.y (mgr.14550) 853 : cluster 0 pgmap v1492: 216 pgs: 216 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: cluster 2024-03-11T21:45:05.701278+0000 mon.a (mon.0) 4046 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: audit 2024-03-11T21:45:05.900461+0000 mon.c (mon.2) 1381 : audit 1 from='client.? 172.21.15.143:0/2584948318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: audit 2024-03-11T21:45:05.900824+0000 mon.a (mon.0) 4047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: cluster 2024-03-11T21:45:06.625373+0000 mon.a (mon.0) 4048 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: audit 2024-03-11T21:45:06.627860+0000 mon.a (mon.0) 4049 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:07.406 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:07 smithi143 bash[29284]: cluster 2024-03-11T21:45:06.636879+0000 mon.a (mon.0) 4050 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-03-11T21:45:07.419 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:07.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: cluster 2024-03-11T21:45:05.699136+0000 mgr.y (mgr.14550) 853 : cluster 0 pgmap v1492: 216 pgs: 216 active+clean; 1.6 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: cluster 2024-03-11T21:45:05.701278+0000 mon.a (mon.0) 4046 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: audit 2024-03-11T21:45:05.900461+0000 mon.c (mon.2) 1381 : audit 1 from='client.? 172.21.15.143:0/2584948318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: audit 2024-03-11T21:45:05.900824+0000 mon.a (mon.0) 4047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: cluster 2024-03-11T21:45:06.625373+0000 mon.a (mon.0) 4048 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:07.531 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: audit 2024-03-11T21:45:06.627860+0000 mon.a (mon.0) 4049 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:07.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:07 smithi203 bash[23654]: cluster 2024-03-11T21:45:06.636879+0000 mon.a (mon.0) 4050 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-03-11T21:45:07.548 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:07.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:07.677 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:07.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:07.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:07.937 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:08.068 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:08.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.198 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.302 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:08.328 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:08.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.456 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:08.586 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:08.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:08.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:08 smithi203 bash[23654]: cluster 2024-03-11T21:45:07.643810+0000 mon.a (mon.0) 4051 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-03-11T21:45:08.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:08 smithi203 bash[23654]: cluster 2024-03-11T21:45:07.699597+0000 mgr.y (mgr.14550) 854 : cluster 0 pgmap v1495: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:08.701 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:08 smithi203 bash[23654]: audit 2024-03-11T21:45:07.736322+0000 mon.c (mon.2) 1382 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:08.716 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:08.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:08.847 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:08.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[29284]: cluster 2024-03-11T21:45:07.643810+0000 mon.a (mon.0) 4051 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-03-11T21:45:08.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[29284]: cluster 2024-03-11T21:45:07.699597+0000 mgr.y (mgr.14550) 854 : cluster 0 pgmap v1495: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:08.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[29284]: audit 2024-03-11T21:45:07.736322+0000 mon.c (mon.2) 1382 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:08.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[21512]: cluster 2024-03-11T21:45:07.643810+0000 mon.a (mon.0) 4051 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-03-11T21:45:08.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[21512]: cluster 2024-03-11T21:45:07.699597+0000 mgr.y (mgr.14550) 854 : cluster 0 pgmap v1495: 216 pgs: 32 unknown, 184 active+clean; 582 KiB data, 767 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:08.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:08 smithi143 bash[21512]: audit 2024-03-11T21:45:07.736322+0000 mon.c (mon.2) 1382 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:08.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.978 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:09.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.107 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:09.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:45:09.237 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:09.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.366 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:09.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:09.497 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:09.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.629 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:09.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.743 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:09 smithi203 bash[23654]: cluster 2024-03-11T21:45:08.650420+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-03-11T21:45:09.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:09 smithi203 bash[23654]: audit 2024-03-11T21:45:09.181687+0000 mgr.y (mgr.14550) 855 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:09.760 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:09.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:09.888 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:09.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:09 smithi143 bash[29284]: cluster 2024-03-11T21:45:08.650420+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-03-11T21:45:09.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:09 smithi143 bash[29284]: audit 2024-03-11T21:45:09.181687+0000 mgr.y (mgr.14550) 855 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:09.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:09 smithi143 bash[21512]: cluster 2024-03-11T21:45:08.650420+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-03-11T21:45:09.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:09 smithi143 bash[21512]: audit 2024-03-11T21:45:09.181687+0000 mgr.y (mgr.14550) 855 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:09.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:10.018 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:10.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:10.148 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:10.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.278 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:10.381 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:10.407 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:10.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.536 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:10.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:45:10.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:10.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:10 smithi143 bash[21512]: cluster 2024-03-11T21:45:09.651829+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-03-11T21:45:10.674 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:10 smithi143 bash[21512]: cluster 2024-03-11T21:45:09.700437+0000 mgr.y (mgr.14550) 856 : cluster 0 pgmap v1498: 216 pgs: 26 creating+peering, 6 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:10.675 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:10 smithi143 bash[29284]: cluster 2024-03-11T21:45:09.651829+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-03-11T21:45:10.675 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:10 smithi143 bash[29284]: cluster 2024-03-11T21:45:09.700437+0000 mgr.y (mgr.14550) 856 : cluster 0 pgmap v1498: 216 pgs: 26 creating+peering, 6 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:10.689 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:10.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:10 smithi203 bash[23654]: cluster 2024-03-11T21:45:09.651829+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-03-11T21:45:10.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:10 smithi203 bash[23654]: cluster 2024-03-11T21:45:09.700437+0000 mgr.y (mgr.14550) 856 : cluster 0 pgmap v1498: 216 pgs: 26 creating+peering, 6 unknown, 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:10.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.819 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:10.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.951 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:11.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:11.079 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:11.132 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:11.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.209 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:11.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:11.339 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:11.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:11.464 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:11.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:11.596 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:11.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.726 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:11.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:11.857 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:11.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:11 smithi143 bash[21512]: cluster 2024-03-11T21:45:10.675611+0000 mon.a (mon.0) 4054 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-03-11T21:45:11.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:11 smithi143 bash[29284]: cluster 2024-03-11T21:45:10.675611+0000 mon.a (mon.0) 4054 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-03-11T21:45:11.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.971 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:11 smithi203 bash[23654]: cluster 2024-03-11T21:45:10.675611+0000 mon.a (mon.0) 4054 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-03-11T21:45:11.988 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:12.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:12.118 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:12.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:12.249 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:12.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:12.380 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:12.388 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:12.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:12.510 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:12.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:12.639 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:12.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:12.764 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:12.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:12 smithi203 bash[23654]: cluster 2024-03-11T21:45:11.700902+0000 mgr.y (mgr.14550) 857 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:12.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:12 smithi203 bash[23654]: cluster 2024-03-11T21:45:11.703161+0000 mon.a (mon.0) 4055 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-03-11T21:45:12.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:12.892 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:12.945 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:12 smithi143 bash[29284]: cluster 2024-03-11T21:45:11.700902+0000 mgr.y (mgr.14550) 857 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:12.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:12 smithi143 bash[29284]: cluster 2024-03-11T21:45:11.703161+0000 mon.a (mon.0) 4055 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-03-11T21:45:12.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:12 smithi143 bash[21512]: cluster 2024-03-11T21:45:11.700902+0000 mgr.y (mgr.14550) 857 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:12.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:12 smithi143 bash[21512]: cluster 2024-03-11T21:45:11.703161+0000 mon.a (mon.0) 4055 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-03-11T21:45:12.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:13.024 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:13.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:13.158 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:13.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:13.277 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:13.377 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:45:13.377 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:45:13.377 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:45:13.377 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:45:13.377 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:45:13.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:13.409 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:13.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:13.538 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:13.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:13.671 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:13.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:13.785 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:13 smithi143 bash[21512]: cluster 2024-03-11T21:45:12.695071+0000 mon.a (mon.0) 4056 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:13.785 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:13 smithi143 bash[21512]: cluster 2024-03-11T21:45:12.703932+0000 mon.a (mon.0) 4057 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-03-11T21:45:13.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:13 smithi143 bash[29284]: cluster 2024-03-11T21:45:12.695071+0000 mon.a (mon.0) 4056 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:13.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:13 smithi143 bash[29284]: cluster 2024-03-11T21:45:12.703932+0000 mon.a (mon.0) 4057 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-03-11T21:45:13.800 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:13.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:13.928 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:14.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:13 smithi203 bash[23654]: cluster 2024-03-11T21:45:12.695071+0000 mon.a (mon.0) 4056 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:13 smithi203 bash[23654]: cluster 2024-03-11T21:45:12.703932+0000 mon.a (mon.0) 4057 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-03-11T21:45:14.056 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:14.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.186 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:14.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.317 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:14.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:14.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:14.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:14.568 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:14.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.698 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:14.801 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:14.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:14 smithi143 bash[29284]: cluster 2024-03-11T21:45:13.702142+0000 mgr.y (mgr.14550) 858 : cluster 0 pgmap v1503: 216 pgs: 3 unknown, 213 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:45:14.813 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:14 smithi143 bash[29284]: cluster 2024-03-11T21:45:13.787429+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-03-11T21:45:14.813 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:14 smithi143 bash[21512]: cluster 2024-03-11T21:45:13.702142+0000 mgr.y (mgr.14550) 858 : cluster 0 pgmap v1503: 216 pgs: 3 unknown, 213 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:45:14.813 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:14 smithi143 bash[21512]: cluster 2024-03-11T21:45:13.787429+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-03-11T21:45:14.827 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:14.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:14.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:14 smithi203 bash[23654]: cluster 2024-03-11T21:45:13.702142+0000 mgr.y (mgr.14550) 858 : cluster 0 pgmap v1503: 216 pgs: 3 unknown, 213 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 252 B/s wr, 1 op/s 2024-03-11T21:45:14.942 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:14 smithi203 bash[23654]: cluster 2024-03-11T21:45:13.787429+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-03-11T21:45:14.956 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:15.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:15.086 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:15.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.216 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:15.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.346 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:15.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:15.477 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:15.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:15.604 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:15.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:15.735 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:15.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:15.864 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:15.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.980 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:15 smithi203 bash[23654]: cluster 2024-03-11T21:45:14.723542+0000 mon.a (mon.0) 4059 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-03-11T21:45:15.995 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:16.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.109 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:15 smithi143 bash[21512]: cluster 2024-03-11T21:45:14.723542+0000 mon.a (mon.0) 4059 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-03-11T21:45:16.110 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:15 smithi143 bash[29284]: cluster 2024-03-11T21:45:14.723542+0000 mon.a (mon.0) 4059 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-03-11T21:45:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:16.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.255 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:16.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.382 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:16.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:16.509 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:16.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.640 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:16.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.769 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:16.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:16 smithi203 bash[23654]: cluster 2024-03-11T21:45:15.703214+0000 mgr.y (mgr.14550) 859 : cluster 0 pgmap v1506: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:45:16.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.883 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:16 smithi143 bash[21512]: cluster 2024-03-11T21:45:15.703214+0000 mgr.y (mgr.14550) 859 : cluster 0 pgmap v1506: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:45:16.884 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:16 smithi143 bash[29284]: cluster 2024-03-11T21:45:15.703214+0000 mgr.y (mgr.14550) 859 : cluster 0 pgmap v1506: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:45:16.900 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:16.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:45:17.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.033 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:17.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.162 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:17.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.288 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:17.420 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:17.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.550 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:17.558 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:17.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:17.681 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:17.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.812 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:17.941 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:18.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:18.072 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:18.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:18.203 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:18.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.334 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:18.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:18.459 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:18.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.590 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:18.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:18.720 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:18.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:18.849 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:18.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.965 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:18 smithi203 bash[23654]: cluster 2024-03-11T21:45:17.704130+0000 mgr.y (mgr.14550) 860 : cluster 0 pgmap v1507: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:45:18.982 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:19.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:19.111 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:19.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:18 smithi143 bash[21512]: cluster 2024-03-11T21:45:17.704130+0000 mgr.y (mgr.14550) 860 : cluster 0 pgmap v1507: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:45:19.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:18 smithi143 bash[29284]: cluster 2024-03-11T21:45:17.704130+0000 mgr.y (mgr.14550) 860 : cluster 0 pgmap v1507: 216 pgs: 216 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:45:19.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:19.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:45:19.240 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:19.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:19.371 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:19.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:19.501 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:19.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:19.633 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:19.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:19.764 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:19.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:19.896 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:19.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:20.025 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:20.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:20.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:20.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.283 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:20.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:20.414 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:20.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:20.545 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:20.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:20.676 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:20.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.803 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:20.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:20.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:20 smithi143 bash[29284]: audit 2024-03-11T21:45:19.193008+0000 mgr.y (mgr.14550) 861 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:20.926 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:20 smithi143 bash[29284]: cluster 2024-03-11T21:45:19.706341+0000 mgr.y (mgr.14550) 862 : cluster 0 pgmap v1508: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 146 B/s wr, 1 op/s 2024-03-11T21:45:20.926 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:20 smithi143 bash[21512]: audit 2024-03-11T21:45:19.193008+0000 mgr.y (mgr.14550) 861 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:20.926 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:20 smithi143 bash[21512]: cluster 2024-03-11T21:45:19.706341+0000 mgr.y (mgr.14550) 862 : cluster 0 pgmap v1508: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 146 B/s wr, 1 op/s 2024-03-11T21:45:20.933 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:21.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:20 smithi203 bash[23654]: audit 2024-03-11T21:45:19.193008+0000 mgr.y (mgr.14550) 861 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:21.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:20 smithi203 bash[23654]: cluster 2024-03-11T21:45:19.706341+0000 mgr.y (mgr.14550) 862 : cluster 0 pgmap v1508: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 146 B/s wr, 1 op/s 2024-03-11T21:45:21.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:21.064 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:21.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:21.194 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:21.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:21.325 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:21.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.454 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:21.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.585 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:21.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.714 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:21.817 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:21.844 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:21.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:21.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:21 smithi203 bash[23654]: cluster 2024-03-11T21:45:21.630025+0000 mon.a (mon.0) 4060 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:21.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:21 smithi203 bash[23654]: cluster 2024-03-11T21:45:21.639576+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-03-11T21:45:21.977 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:22.081 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:22.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:21 smithi143 bash[21512]: cluster 2024-03-11T21:45:21.630025+0000 mon.a (mon.0) 4060 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:22.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:21 smithi143 bash[21512]: cluster 2024-03-11T21:45:21.639576+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-03-11T21:45:22.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:21 smithi143 bash[29284]: cluster 2024-03-11T21:45:21.630025+0000 mon.a (mon.0) 4060 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:22.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:21 smithi143 bash[29284]: cluster 2024-03-11T21:45:21.639576+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-03-11T21:45:22.107 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:22.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:22.238 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:22.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.362 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:22.411 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:22.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- 172.21.15.143:0/985515893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370107f30 msgr2=0x7f9370108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- 172.21.15.143:0/985515893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370107f30 0x7f9370108310 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7f9360009990 tx=0x7f9360009ca0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- 172.21.15.143:0/985515893 shutdown_connections 2024-03-11T21:45:22.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- 172.21.15.143:0/985515893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370102880 0x7f937010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.450 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- 172.21.15.143:0/985515893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370107f30 0x7f9370108310 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- 172.21.15.143:0/985515893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9370101ec0 0x7f9370102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- 172.21.15.143:0/985515893 >> 172.21.15.143:0/985515893 conn(0x7f9370078eb0 msgr2=0x7f93700792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- 172.21.15.143:0/985515893 shutdown_connections 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- 172.21.15.143:0/985515893 wait complete. 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 librados: starting msgr at 2024-03-11T21:45:22.451 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 librados: starting objecter 2024-03-11T21:45:22.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 Processor -- start 2024-03-11T21:45:22.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- start start 2024-03-11T21:45:22.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 librados: setting wanted keys 2024-03-11T21:45:22.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 librados: calling monclient init 2024-03-11T21:45:22.452 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9370102880 0x7f9370074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370107f30 0x7f9370074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93701116e0 con 0x7f9370107f30 2024-03-11T21:45:22.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370111560 con 0x7f9370102880 2024-03-11T21:45:22.453 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9375236700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9370111860 con 0x7f9370101ec0 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38818/0 (socket says 172.21.15.143:38818) 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 -- 172.21.15.143:0/124861137 learned_addr learned my addr 172.21.15.143:0/124861137 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370107f30 0x7f9370074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9370102880 0x7f9370074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.454 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9370102880 msgr2=0x7f9370074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9370102880 0x7f9370074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370107f30 msgr2=0x7f9370074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370107f30 0x7f9370074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93701a6240 con 0x7f9370101ec0 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370107f30 0x7f9370074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f9360000c00 tx=0x7f9360005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.455 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9357fff700 10 client.?.objecter ms_handle_connect 0x7f9370101ec0 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9357fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.458+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360027070 con 0x7f9370101ec0 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93701a64d0 con 0x7f9370101ec0 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9360022470 con 0x7f9370101ec0 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360017650 con 0x7f9370101ec0 2024-03-11T21:45:22.456 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 10 client.42269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93701a6920 con 0x7f9370101ec0 2024-03-11T21:45:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 librados: init done 2024-03-11T21:45:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f937004faa0 con 0x7f9370101ec0 2024-03-11T21:45:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f936002a070 con 0x7f9370101ec0 2024-03-11T21:45:22.457 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f935806f820 0x7f9358071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(962..962 src has 842..962) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f936008fd20 con 0x7f9370101ec0 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 osd_map(962..962 src has 842..962) v4 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 3 client.42269.objecter handle_osd_map got epochs [962,962] > 0 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 3 client.42269.objecter handle_osd_map decoding full epoch 962 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f935806f820 0x7f9358071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.458 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 20 client.42269.objecter dump_active .. 0 homeless 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v962) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9360031020 con 0x7f9370101ec0 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 10 client.42269.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 1 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f93700629f0 0x7f9370062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 1 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 0 1 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 4 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 10 client.42269.objecter _send_command 1 2024-03-11T21:45:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701a9dd0 con 0x7f93700629f0 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f935806f820 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f935806f820 0x7f9358071ce0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f9364006fd0 tx=0x7f9364008040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f93700629f0 0x7f9370062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93700629f0 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f93700629f0 0x7f9370062dd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701a9dd0 con 0x7f93700629f0 2024-03-11T21:45:22.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:45:22.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 10 client.42269.objecter _finish_command 1 = system:0 2024-03-11T21:45:22.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 4 2024-03-11T21:45:22.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.462+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 0 1 2024-03-11T21:45:22.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 10 client.42269.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 2 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 2 2024-03-11T21:45:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f9370075610 osd=0 3 2024-03-11T21:45:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 0 2 2024-03-11T21:45:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 4 2024-03-11T21:45:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 10 client.42269.objecter _send_command 2 2024-03-11T21:45:22.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9370062470 con 0x7f93700629f0 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9370062470 con 0x7f93700629f0 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9357fff700 10 client.42269.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 4 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 0 2 2024-03-11T21:45:22.470 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 10 client.42269.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 3 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f937002c6b0 0x7f937002cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 3 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 1 3 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 4 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 10 client.42269.objecter _send_command 3 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f937010aaa0 con 0x7f937002c6b0 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.482+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f937002c6b0 0x7f937002cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f937002c6b0 0x7f937002cab0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f937002c6b0 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f937010aaa0 con 0x7f937002c6b0 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:45:22.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 10 client.42269.objecter _finish_command 3 = system:0 2024-03-11T21:45:22.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 4 2024-03-11T21:45:22.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.486+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 1 3 2024-03-11T21:45:22.488 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 10 client.42269.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 4 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 4 2024-03-11T21:45:22.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937010a420 osd=1 3 2024-03-11T21:45:22.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 1 4 2024-03-11T21:45:22.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 4 2024-03-11T21:45:22.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 10 client.42269.objecter _send_command 4 2024-03-11T21:45:22.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9370109ea0 con 0x7f937002c6b0 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9370109ea0 con 0x7f937002c6b0 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 10 client.42269.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 4 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 1 4 2024-03-11T21:45:22.497 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:45:22.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 10 client.42269.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.498 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 5 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f93701abf90 0x7f93701ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 5 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 2 5 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 4 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 10 client.42269.objecter _send_command 5 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f937002dcd0 con 0x7f93701abf90 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f93701abf90 0x7f93701ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701abf90 2024-03-11T21:45:22.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f93701abf90 0x7f93701ae380 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f937002dcd0 con 0x7f93701abf90 2024-03-11T21:45:22.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:45:22.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 10 client.42269.objecter _finish_command 5 = system:0 2024-03-11T21:45:22.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 4 2024-03-11T21:45:22.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.510+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 2 5 2024-03-11T21:45:22.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 10 client.42269.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 6 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 6 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937010ac50 osd=2 3 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 2 6 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 4 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 10 client.42269.objecter _send_command 6 2024-03-11T21:45:22.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f937002d240 con 0x7f93701abf90 2024-03-11T21:45:22.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f937002d240 con 0x7f93701abf90 2024-03-11T21:45:22.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9357fff700 10 client.42269.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 4 2024-03-11T21:45:22.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.530+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 2 6 2024-03-11T21:45:22.519 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 10 client.42269.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 7 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f93701a9930 0x7f93701b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 7 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 3 7 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 4 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 10 client.42269.objecter _send_command 7 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701b4960 con 0x7f93701a9930 2024-03-11T21:45:22.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f93701a9930 0x7f93701b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701a9930 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f93701a9930 0x7f93701b1a50 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701b4960 con 0x7f93701a9930 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 10 client.42269.objecter _finish_command 7 = system:0 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 4 2024-03-11T21:45:22.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.534+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 3 7 2024-03-11T21:45:22.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 10 client.42269.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 8 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 8 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f937002d640 osd=3 3 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 3 8 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 4 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 10 client.42269.objecter _send_command 8 2024-03-11T21:45:22.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.550+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f93701a8fc0 con 0x7f93701a9930 2024-03-11T21:45:22.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f93701a8fc0 con 0x7f93701a9930 2024-03-11T21:45:22.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 10 client.42269.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 4 2024-03-11T21:45:22.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 3 8 2024-03-11T21:45:22.540 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:45:22.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 10 client.42269.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 9 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f93701af5a0 0x7f93701b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 9 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 4 9 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 4 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 10 client.42269.objecter _send_command 9 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701b8350 con 0x7f93701af5a0 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f93701af5a0 0x7f93701b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701af5a0 2024-03-11T21:45:22.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.554+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f93701af5a0 0x7f93701b3110 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.558+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701b8350 con 0x7f93701af5a0 2024-03-11T21:45:22.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.558+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:45:22.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.558+0000 7f9357fff700 10 client.42269.objecter _finish_command 9 = system:0 2024-03-11T21:45:22.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.558+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 4 2024-03-11T21:45:22.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.558+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 4 9 2024-03-11T21:45:22.559 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 10 client.42269.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 10 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 10 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701a8fc0 osd=4 3 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 4 10 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 4 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 10 client.42269.objecter _send_command 10 2024-03-11T21:45:22.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.574+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f93701b3650 con 0x7f93701af5a0 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f93701b3650 con 0x7f93701af5a0 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 10 client.42269.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 4 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 4 10 2024-03-11T21:45:22.564 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 10 client.42269.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 11 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f93701b3dc0 0x7f93701b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 11 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 5 11 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 4 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 10 client.42269.objecter _send_command 11 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701bb530 con 0x7f93701b3dc0 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f93701b3dc0 0x7f93701b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f936e59c700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f93701b3dc0 0x7f93701b4220 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.578+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701b3dc0 2024-03-11T21:45:22.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.582+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701bb530 con 0x7f93701b3dc0 2024-03-11T21:45:22.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.582+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:45:22.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.582+0000 7f9357fff700 10 client.42269.objecter _finish_command 11 = system:0 2024-03-11T21:45:22.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.582+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 4 2024-03-11T21:45:22.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.582+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 5 11 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 10 client.42269.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 12 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 12 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701b3650 osd=5 3 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 5 12 2024-03-11T21:45:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 4 2024-03-11T21:45:22.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 10 client.42269.objecter _send_command 12 2024-03-11T21:45:22.585 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.598+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f93701bb530 con 0x7f93701b3dc0 2024-03-11T21:45:22.588 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f93701bb530 con 0x7f93701b3dc0 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 10 client.42269.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 4 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 5 12 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 10 client.42269.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 13 2024-03-11T21:45:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f93701b90d0 0x7f93701b9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 13 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 6 13 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 4 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 10 client.42269.objecter _send_command 13 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701be920 con 0x7f93701b90d0 2024-03-11T21:45:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f93701b90d0 0x7f93701b9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:22.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f936f59e700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f93701b90d0 0x7f93701b9530 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.591 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.602+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701b90d0 2024-03-11T21:45:22.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.606+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701be920 con 0x7f93701b90d0 2024-03-11T21:45:22.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.606+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:45:22.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.606+0000 7f9357fff700 10 client.42269.objecter _finish_command 13 = system:0 2024-03-11T21:45:22.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.606+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 4 2024-03-11T21:45:22.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.606+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 6 13 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 10 client.42269.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 14 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 14 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701bb530 osd=6 3 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 6 14 2024-03-11T21:45:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 4 2024-03-11T21:45:22.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 10 client.42269.objecter _send_command 14 2024-03-11T21:45:22.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.622+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f93701b9b80 con 0x7f93701b90d0 2024-03-11T21:45:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f93701b9b80 con 0x7f93701b90d0 2024-03-11T21:45:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9357fff700 10 client.42269.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 4 2024-03-11T21:45:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 6 14 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 10 client.42269.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 15 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f93701ba2a0 0x7f93701ba700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.615 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 15 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 7 15 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 4 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 10 client.42269.objecter _send_command 15 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.626+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93701bada0 con 0x7f93701ba2a0 2024-03-11T21:45:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f93701ba2a0 0x7f93701ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 10 client.42269.objecter ms_handle_connect 0x7f93701ba2a0 2024-03-11T21:45:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f936ed9d700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f93701ba2a0 0x7f93701ba700 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:22.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93701bada0 con 0x7f93701ba2a0 2024-03-11T21:45:22.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:45:22.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 10 client.42269.objecter _finish_command 15 = system:0 2024-03-11T21:45:22.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 4 2024-03-11T21:45:22.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.630+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 7 15 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 10 client.42269.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign -1 16 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 20 client.42269.objecter _get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 15 client.42269.objecter _session_command_op_remove -1 16 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 20 client.42269.objecter get_session s=0x7f93701b9b80 osd=7 3 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 15 client.42269.objecter _session_command_op_assign 7 16 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 4 2024-03-11T21:45:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 10 client.42269.objecter _send_command 16 2024-03-11T21:45:22.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.650+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f93701bada0 con 0x7f93701ba2a0 2024-03-11T21:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9357fff700 1 -- 172.21.15.143:0/124861137 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f93701bada0 con 0x7f93701ba2a0 2024-03-11T21:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9357fff700 10 client.42269.objecter ms_dispatch 0x7f9370001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9357fff700 10 client.42269.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9357fff700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 4 2024-03-11T21:45:22.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9357fff700 15 client.42269.objecter _session_command_op_remove 7 16 2024-03-11T21:45:22.641 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 librados: watch_flush enter 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 librados: watch_flush exit 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.0 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f93700629f0 msgr2=0x7f9370062dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f93700629f0 0x7f9370062dd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f9370075610 osd=0 1 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.1 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f937002c6b0 msgr2=0x7f937002cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7f937002c6b0 0x7f937002cab0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010a420 osd=1 1 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.2 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f93701abf90 msgr2=0x7f93701ae380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7f93701abf90 0x7f93701ae380 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937010ac50 osd=2 1 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.3 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f93701a9930 msgr2=0x7f93701b1a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f93701a9930 0x7f93701b1a50 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f937002d640 osd=3 1 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.4 2024-03-11T21:45:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f93701af5a0 msgr2=0x7f93701b3110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7f93701af5a0 0x7f93701b3110 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701a8fc0 osd=4 1 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.5 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f93701b3dc0 msgr2=0x7f93701b4220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7f93701b3dc0 0x7f93701b4220 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b3650 osd=5 1 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.6 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f93701b90d0 msgr2=0x7f93701b9530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7f93701b90d0 0x7f93701b9530 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701bb530 osd=6 1 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter close_session for osd.7 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f93701ba2a0 msgr2=0x7f93701ba700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f93701ba2a0 0x7f93701ba700 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter put_session s=0x7f93701b9b80 osd=7 1 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 20 client.42269.objecter shutdown clearing up homeless session... 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 10 client.42269.objecter successfully canceled tick 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f936f59e700 1 -- 172.21.15.143:0/124861137 reap_dead start 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f936f59e700 1 -- 172.21.15.143:0/124861137 reap_dead start 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f936f59e700 1 -- 172.21.15.143:0/124861137 reap_dead start 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f936f59e700 1 -- 172.21.15.143:0/124861137 reap_dead start 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f935806f820 msgr2=0x7f9358071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f935806f820 0x7f9358071ce0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f9364006fd0 tx=0x7f9364008040 comp rx=0 tx=0).stop 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f936f59e700 1 -- 172.21.15.143:0/124861137 reap_dead start 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 msgr2=0x7f93700740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f9360000c00 tx=0x7f9360005e80 comp rx=0 tx=0).stop 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 shutdown_connections 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 --2- 172.21.15.143:0/124861137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9370101ec0 0x7f93700740e0 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 >> 172.21.15.143:0/124861137 conn(0x7f9370078eb0 msgr2=0x7f9370108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 shutdown_connections 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 -- 172.21.15.143:0/124861137 wait complete. 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:22.654+0000 7f9375236700 1 librados: shutdown 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:22.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.741 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:22.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:22.873 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:22.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:22.989 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:22 smithi203 bash[23654]: cluster 2024-03-11T21:45:21.707159+0000 mgr.y (mgr.14550) 863 : cluster 0 pgmap v1510: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-03-11T21:45:22.989 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:22 smithi203 bash[23654]: audit 2024-03-11T21:45:22.465079+0000 mon.c (mon.2) 1383 : audit 0 from='client.? 172.21.15.143:0/124861137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:23.005 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:23.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:23.119 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:22 smithi143 bash[21512]: cluster 2024-03-11T21:45:21.707159+0000 mgr.y (mgr.14550) 863 : cluster 0 pgmap v1510: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-03-11T21:45:23.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:22 smithi143 bash[21512]: audit 2024-03-11T21:45:22.465079+0000 mon.c (mon.2) 1383 : audit 0 from='client.? 172.21.15.143:0/124861137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:23.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:22 smithi143 bash[29284]: cluster 2024-03-11T21:45:21.707159+0000 mgr.y (mgr.14550) 863 : cluster 0 pgmap v1510: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-03-11T21:45:23.120 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:22 smithi143 bash[29284]: audit 2024-03-11T21:45:22.465079+0000 mon.c (mon.2) 1383 : audit 0 from='client.? 172.21.15.143:0/124861137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:23.133 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:23.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:23.265 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:23.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:23.395 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:23.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:23.525 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:23.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:23.657 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:23.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:23.788 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:23.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:23.903 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:23 smithi203 bash[23654]: audit 2024-03-11T21:45:22.759547+0000 mon.c (mon.2) 1384 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:23.918 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:23.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:23 smithi143 bash[21512]: audit 2024-03-11T21:45:22.759547+0000 mon.c (mon.2) 1384 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:23.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:23 smithi143 bash[29284]: audit 2024-03-11T21:45:22.759547+0000 mon.c (mon.2) 1384 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:24.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:24.048 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:24.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:24.178 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:24.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:24.310 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:24.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.441 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:24.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:24.573 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:24.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:24.706 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:24.809 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:24.837 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:24.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.968 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:25.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:24 smithi203 bash[23654]: cluster 2024-03-11T21:45:23.709282+0000 mgr.y (mgr.14550) 864 : cluster 0 pgmap v1511: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:25.060 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:24 smithi143 bash[21512]: cluster 2024-03-11T21:45:23.709282+0000 mgr.y (mgr.14550) 864 : cluster 0 pgmap v1511: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:25.060 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:24 smithi143 bash[29284]: cluster 2024-03-11T21:45:23.709282+0000 mgr.y (mgr.14550) 864 : cluster 0 pgmap v1511: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:25.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.094 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:25.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.220 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/3947693368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 msgr2=0x7fd3e010e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- 172.21.15.143:0/3947693368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e330 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009990 tx=0x7fd3d0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/3947693368 shutdown_connections 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- 172.21.15.143:0/3947693368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e010e870 0x7fd3e0110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- 172.21.15.143:0/3947693368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e006a440 0x7fd3e006a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- 172.21.15.143:0/3947693368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e330 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/3947693368 >> 172.21.15.143:0/3947693368 conn(0x7fd3e0077de0 msgr2=0x7fd3e00781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/3947693368 shutdown_connections 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/3947693368 wait complete. 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 librados: starting msgr at 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 librados: starting objecter 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 Processor -- start 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- start start 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 librados: setting wanted keys 2024-03-11T21:45:25.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 librados: calling monclient init 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 0x7fd3e0107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e006be80 con 0x7fd3e006adf0 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e006bd00 con 0x7fd3e010e870 2024-03-11T21:45:25.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e006c000 con 0x7fd3e006a440 2024-03-11T21:45:25.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60136/0 (socket says 172.21.15.143:60136) 2024-03-11T21:45:25.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 -- 172.21.15.143:0/2502142117 learned_addr learned my addr 172.21.15.143:0/2502142117 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:25.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e5cc6700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:25.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 -- 172.21.15.143:0/2502142117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 msgr2=0x7fd3e0107f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:25.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e6cc8700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 0x7fd3e0107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 0x7fd3e0107f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 -- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 msgr2=0x7fd3e010e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 -- 172.21.15.143:0/2502142117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0108850 con 0x7fd3e006a440 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e6cc8700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 0x7fd3e0107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e64c7700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80060b0 tx=0x7fd3d800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3d77fe700 10 client.?.objecter ms_handle_connect 0x7fd3e006a440 2024-03-11T21:45:25.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3d77fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d801a070 con 0x7fd3e006a440 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e0108b40 con 0x7fd3e006a440 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd3d800ddd0 con 0x7fd3e006a440 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 10 client.42275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e01af110 con 0x7fd3e006a440 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3e872b700 1 librados: init done 2024-03-11T21:45:25.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.270+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80145c0 con 0x7fd3e006a440 2024-03-11T21:45:25.265 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e01013a0 con 0x7fd3e006a440 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd3d801d070 con 0x7fd3e006a440 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 0x7fd3cc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(962..962 src has 842..962) v4 ==== 9194+0+0 (secure 0 0 0) 0x7fd3d808e170 con 0x7fd3e006a440 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 10 client.42275.objecter ms_dispatch 0x7fd3e0001550 osd_map(962..962 src has 842..962) v4 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 3 client.42275.objecter handle_osd_map got epochs [962,962] > 0 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 3 client.42275.objecter handle_osd_map decoding full epoch 962 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 20 client.42275.objecter dump_active .. 0 homeless 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3e5cc6700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 0x7fd3cc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3e5cc6700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 0x7fd3cc071ce0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00098e0 tx=0x7fd3d0005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.274+0000 7fd3d77fe700 10 client.42275.objecter ms_handle_connect 0x7fd3cc06f820 2024-03-11T21:45:25.266 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.278+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd3d8010070 con 0x7fd3e006a440 2024-03-11T21:45:25.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.342 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:25.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.470 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:25.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.498+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266} v 0) v1 -- 0x7fd3e01092f0 con 0x7fd3e006a440 2024-03-11T21:45:25.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.598 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:25.697 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:25.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:25.731 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:25.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.766+0000 7fd3d77fe700 1 -- 172.21.15.143:0/2502142117 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]=0 set osd.2 primary-affinity to 0.653804 (8428472) v963) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd3d805e9c0 con 0x7fd3e006a440 2024-03-11T21:45:25.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 10 librados: watch_flush enter 2024-03-11T21:45:25.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 10 librados: watch_flush exit 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 20 client.42275.objecter shutdown clearing up homeless session... 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 10 client.42275.objecter successfully canceled tick 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 msgr2=0x7fd3cc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 0x7fd3cc071ce0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00098e0 tx=0x7fd3d0005960 comp rx=0 tx=0).stop 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 msgr2=0x7fd3e010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:25.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80060b0 tx=0x7fd3d800d3b0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 shutdown_connections 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3cc06f820 0x7fd3cc071ce0 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3e010e870 0x7fd3e0107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e006a440 0x7fd3e010dd40 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.770+0000 7fd3e872b700 1 --2- 172.21.15.143:0/2502142117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3e006adf0 0x7fd3e010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.774+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 >> 172.21.15.143:0/2502142117 conn(0x7fd3e0077de0 msgr2=0x7fd3e00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:25.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.774+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 shutdown_connections 2024-03-11T21:45:25.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.774+0000 7fd3e872b700 1 -- 172.21.15.143:0/2502142117 wait complete. 2024-03-11T21:45:25.760 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:25.774+0000 7fd3e872b700 1 librados: shutdown 2024-03-11T21:45:25.766 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.2 primary-affinity to 0.653804 (8428472) 2024-03-11T21:45:25.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:25 smithi143 bash[29284]: audit 2024-03-11T21:45:25.503560+0000 mon.c (mon.2) 1385 : audit 1 from='client.? 172.21.15.143:0/2502142117' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.843 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:25 smithi143 bash[29284]: audit 2024-03-11T21:45:25.503895+0000 mon.a (mon.0) 4062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.843 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:25 smithi143 bash[21512]: audit 2024-03-11T21:45:25.503560+0000 mon.c (mon.2) 1385 : audit 1 from='client.? 172.21.15.143:0/2502142117' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.843 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:25 smithi143 bash[21512]: audit 2024-03-11T21:45:25.503895+0000 mon.a (mon.0) 4062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.854 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:25.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:25 smithi203 bash[23654]: audit 2024-03-11T21:45:25.503560+0000 mon.c (mon.2) 1385 : audit 1 from='client.? 172.21.15.143:0/2502142117' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:25 smithi203 bash[23654]: audit 2024-03-11T21:45:25.503895+0000 mon.a (mon.0) 4062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]: dispatch 2024-03-11T21:45:25.986 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:26.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:26.116 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:26.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:26.246 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:26.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:26.377 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:26.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:26.506 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:26.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:26.638 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:26.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:26.777 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:26.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:26.903 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:27.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.016 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[21512]: cluster 2024-03-11T21:45:25.710563+0000 mgr.y (mgr.14550) 865 : cluster 0 pgmap v1512: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:27.016 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[21512]: audit 2024-03-11T21:45:25.765697+0000 mon.a (mon.0) 4063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]': finished 2024-03-11T21:45:27.016 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[21512]: cluster 2024-03-11T21:45:25.775401+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-03-11T21:45:27.017 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[21512]: cluster 2024-03-11T21:45:26.777544+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-03-11T21:45:27.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[29284]: cluster 2024-03-11T21:45:25.710563+0000 mgr.y (mgr.14550) 865 : cluster 0 pgmap v1512: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:27.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[29284]: audit 2024-03-11T21:45:25.765697+0000 mon.a (mon.0) 4063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]': finished 2024-03-11T21:45:27.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[29284]: cluster 2024-03-11T21:45:25.775401+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-03-11T21:45:27.017 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:26 smithi143 bash[29284]: cluster 2024-03-11T21:45:26.777544+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-03-11T21:45:27.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:26 smithi203 bash[23654]: cluster 2024-03-11T21:45:25.710563+0000 mgr.y (mgr.14550) 865 : cluster 0 pgmap v1512: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:27.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:26 smithi203 bash[23654]: audit 2024-03-11T21:45:25.765697+0000 mon.a (mon.0) 4063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6538037510086266}]': finished 2024-03-11T21:45:27.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:26 smithi203 bash[23654]: cluster 2024-03-11T21:45:25.775401+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-03-11T21:45:27.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:26 smithi203 bash[23654]: cluster 2024-03-11T21:45:26.777544+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-03-11T21:45:27.032 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:27.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:27.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:45:27.264 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.308 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:27.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.443 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:27.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:27.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:27.572 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:27.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:27.702 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:27.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:27.833 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:27.936 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.964 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:28.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.094 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:28.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:28.225 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:28.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:28.355 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:28.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.486 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:28.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:28.618 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:28.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.749 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:28.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:28.854 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:28 smithi143 bash[21512]: cluster 2024-03-11T21:45:27.711580+0000 mgr.y (mgr.14550) 866 : cluster 0 pgmap v1515: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:45:28.854 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:28 smithi143 bash[29284]: cluster 2024-03-11T21:45:27.711580+0000 mgr.y (mgr.14550) 866 : cluster 0 pgmap v1515: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:45:28.874 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:29.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 -- 172.21.15.143:0/3884311921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 msgr2=0x7fcc38108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/3884311921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38108280 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009990 tx=0x7fcc28009ca0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 -- 172.21.15.143:0/3884311921 shutdown_connections 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/3884311921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/3884311921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38108280 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/3884311921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 0x7fcc381022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 -- 172.21.15.143:0/3884311921 >> 172.21.15.143:0/3884311921 conn(0x7fcc38078eb0 msgr2=0x7fcc380792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 -- 172.21.15.143:0/3884311921 shutdown_connections 2024-03-11T21:45:29.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.038+0000 7fcc3d32a700 1 -- 172.21.15.143:0/3884311921 wait complete. 2024-03-11T21:45:29.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 librados: starting msgr at 2024-03-11T21:45:29.028 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 librados: starting objecter 2024-03-11T21:45:29.028 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 Processor -- start 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- start start 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 librados: setting wanted keys 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 librados: calling monclient init 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 0x7fcc3819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3819dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:29.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc381115a0 con 0x7fcc38102830 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38111420 con 0x7fcc38101e90 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38111720 con 0x7fcc38107ea0 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc367fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3819dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60156/0 (socket says 172.21.15.143:60156) 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 -- 172.21.15.143:0/513535540 learned_addr learned my addr 172.21.15.143:0/513535540 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc36ffd700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 0x7fcc3819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 -- 172.21.15.143:0/513535540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 msgr2=0x7fcc3819d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 0x7fcc3819d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 -- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 msgr2=0x7fcc3819dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:29.031 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3819dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 -- 172.21.15.143:0/513535540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38198160 con 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc367fc700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3819dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc377fe700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c370 tx=0x7fcc3000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 10 client.?.objecter ms_handle_connect 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000ce20 con 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fcc300090d0 con 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc381983f0 con 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 10 client.42281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc381a4710 con 0x7fcc38107ea0 2024-03-11T21:45:29.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 librados: init done 2024-03-11T21:45:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:29.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:28 smithi203 bash[23654]: cluster 2024-03-11T21:45:27.711580+0000 mgr.y (mgr.14550) 866 : cluster 0 pgmap v1515: 216 pgs: 216 active+clean; 583 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300046b0 con 0x7fcc38107ea0 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc24005120 con 0x7fcc38107ea0 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.042+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fcc3001a020 con 0x7fcc38107ea0 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 0x7fcc20071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(964..964 src has 842..964) v4 ==== 9166+0+0 (secure 0 0 0) 0x7fcc3008e1c0 con 0x7fcc38107ea0 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 10 client.42281.objecter ms_dispatch 0x7fcc38001550 osd_map(964..964 src has 842..964) v4 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 3 client.42281.objecter handle_osd_map got epochs [964,964] > 0 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 3 client.42281.objecter handle_osd_map decoding full epoch 964 2024-03-11T21:45:29.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc36ffd700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 0x7fcc20071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:29.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 20 client.42281.objecter dump_active .. 0 homeless 2024-03-11T21:45:29.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc36ffd700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 0x7fcc20071ce0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b2e0 tx=0x7fcc2801a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:29.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.046+0000 7fcc1ffff700 10 client.42281.objecter ms_handle_connect 0x7fcc2006f820 2024-03-11T21:45:29.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.050+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fcc30010040 con 0x7fcc38107ea0 2024-03-11T21:45:29.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:29.154 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:29.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:29.267 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.282+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fcc24005410 con 0x7fcc38107ea0 2024-03-11T21:45:29.283 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:29.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:29.411 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:29.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:29.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:45:29.541 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:29.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:29.673 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:29.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:29 smithi203 bash[23654]: audit 2024-03-11T21:45:29.284775+0000 mon.c (mon.2) 1386 : audit 1 from='client.? 172.21.15.143:0/513535540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:29 smithi203 bash[23654]: audit 2024-03-11T21:45:29.285350+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.802+0000 7fcc1ffff700 1 -- 172.21.15.143:0/513535540 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v965) v1 ==== 91+0+0 (secure 0 0 0) 0x7fcc3005afe0 con 0x7fcc38107ea0 2024-03-11T21:45:29.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 10 librados: watch_flush enter 2024-03-11T21:45:29.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 10 librados: watch_flush exit 2024-03-11T21:45:29.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 20 client.42281.objecter shutdown clearing up homeless session... 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 10 client.42281.objecter successfully canceled tick 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 msgr2=0x7fcc20071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 0x7fcc20071ce0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b2e0 tx=0x7fcc2801a040 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 msgr2=0x7fcc38197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c370 tx=0x7fcc3000c680 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 shutdown_connections 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fcc2006f820 0x7fcc20071ce0 unknown :-1 s=CLOSED pgs=3491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc38107ea0 0x7fcc38197a00 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc38101e90 0x7fcc3819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 --2- 172.21.15.143:0/513535540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc38102830 0x7fcc3819dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:29.798 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.810+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 >> 172.21.15.143:0/513535540 conn(0x7fcc38078eb0 msgr2=0x7fcc38105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:29.799 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: Running main() from gmock_main.cc 2024-03-11T21:45:29.799 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] Global test environment set-up. 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (945030 ms) 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4664 ms) 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4014 ms) 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5039 ms) 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-11T21:45:29.800 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6954 ms) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (965701 ms total) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4982 ms) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6120 ms) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7916 ms) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5992 ms) 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-11T21:45:29.801 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4027 ms) 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi143-72399-11. 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: Waiting for snaps to purge. 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19130 ms) 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48167 ms total) 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [----------] Global test environment tear-down 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (1013868 ms total) 2024-03-11T21:45:29.802 INFO:tasks.workunit.client.0.smithi143.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-11T21:45:29.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.814+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 shutdown_connections 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71963 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71963 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71589 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71589 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71631 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71631 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71933 2024-03-11T21:45:29.804 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71933 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71647 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71647 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=71992 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 71992 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72000 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72000 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72282 2024-03-11T21:45:29.805 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72282 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=72432 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 72432 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ exit 0 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ cleanup 2024-03-11T21:45:29.806 INFO:tasks.workunit.client.0.smithi143.stderr:+ pkill -P 71577 2024-03-11T21:45:29.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.814+0000 7fcc3d32a700 1 -- 172.21.15.143:0/513535540 wait complete. 2024-03-11T21:45:29.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:29.814+0000 7fcc3d32a700 1 librados: shutdown 2024-03-11T21:45:29.807 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:45:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.808 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:29.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:29.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:29 smithi143 bash[29284]: audit 2024-03-11T21:45:29.284775+0000 mon.c (mon.2) 1386 : audit 1 from='client.? 172.21.15.143:0/513535540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.831 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:29 smithi143 bash[29284]: audit 2024-03-11T21:45:29.285350+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:29 smithi143 bash[21512]: audit 2024-03-11T21:45:29.284775+0000 mon.c (mon.2) 1386 : audit 1 from='client.? 172.21.15.143:0/513535540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.832 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:29 smithi143 bash[21512]: audit 2024-03-11T21:45:29.285350+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:29.890 INFO:tasks.workunit.client.0.smithi143.stderr:+ true 2024-03-11T21:45:29.890 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-11T21:45:29.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T21:45:29.903 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-03-11T21:45:29.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T21:45:29.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:29.970 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:30.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:30.097 INFO:teuthology.orchestra.run.smithi143.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi143 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:30.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:30.226 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:30.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:30.335 DEBUG:teuthology.parallel:result is None 2024-03-11T21:45:30.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:45:30.346 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:45:30.346 DEBUG:teuthology.orchestra.run.smithi143:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-11T21:45:30.355 INFO:teuthology.orchestra.run.smithi203.stderr:Error response from daemon: Cannot kill container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:30.433 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:45:30.433 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-11T21:45:30.442 INFO:tasks.thrashosds:joining thrashosds 2024-03-11T21:45:31.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:30 smithi203 bash[23654]: audit 2024-03-11T21:45:29.204311+0000 mgr.y (mgr.14550) 867 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:30 smithi203 bash[23654]: cluster 2024-03-11T21:45:29.713680+0000 mgr.y (mgr.14550) 868 : cluster 0 pgmap v1516: 216 pgs: 216 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T21:45:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:30 smithi203 bash[23654]: cluster 2024-03-11T21:45:29.785699+0000 mon.a (mon.0) 4067 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:30 smithi203 bash[23654]: audit 2024-03-11T21:45:29.796993+0000 mon.a (mon.0) 4068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:30 smithi203 bash[23654]: cluster 2024-03-11T21:45:29.805536+0000 mon.a (mon.0) 4069 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-03-11T21:45:31.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[21512]: audit 2024-03-11T21:45:29.204311+0000 mgr.y (mgr.14550) 867 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:31.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[21512]: cluster 2024-03-11T21:45:29.713680+0000 mgr.y (mgr.14550) 868 : cluster 0 pgmap v1516: 216 pgs: 216 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T21:45:31.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[21512]: cluster 2024-03-11T21:45:29.785699+0000 mon.a (mon.0) 4067 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:31.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[21512]: audit 2024-03-11T21:45:29.796993+0000 mon.a (mon.0) 4068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:31.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[21512]: cluster 2024-03-11T21:45:29.805536+0000 mon.a (mon.0) 4069 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-03-11T21:45:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[29284]: audit 2024-03-11T21:45:29.204311+0000 mgr.y (mgr.14550) 867 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[29284]: cluster 2024-03-11T21:45:29.713680+0000 mgr.y (mgr.14550) 868 : cluster 0 pgmap v1516: 216 pgs: 216 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T21:45:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[29284]: cluster 2024-03-11T21:45:29.785699+0000 mon.a (mon.0) 4067 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[29284]: audit 2024-03-11T21:45:29.796993+0000 mon.a (mon.0) 4068 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:30 smithi143 bash[29284]: cluster 2024-03-11T21:45:29.805536+0000 mon.a (mon.0) 4069 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-03-11T21:45:32.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:30.431786+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:45:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:30.433103+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:45:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:30.994850+0000 osd.1 (osd.1) 315 : cluster 0 5.c scrub starts 2024-03-11T21:45:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:30.997302+0000 osd.1 (osd.1) 316 : cluster 0 5.c scrub ok 2024-03-11T21:45:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:31.228946+0000 osd.7 (osd.7) 498 : cluster 0 2.b scrub starts 2024-03-11T21:45:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:31 smithi203 bash[23654]: cluster 2024-03-11T21:45:31.231351+0000 osd.7 (osd.7) 499 : cluster 0 2.b scrub ok 2024-03-11T21:45:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:30.431786+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:45:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:30.433103+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:45:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:30.994850+0000 osd.1 (osd.1) 315 : cluster 0 5.c scrub starts 2024-03-11T21:45:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:30.997302+0000 osd.1 (osd.1) 316 : cluster 0 5.c scrub ok 2024-03-11T21:45:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:31.228946+0000 osd.7 (osd.7) 498 : cluster 0 2.b scrub starts 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[29284]: cluster 2024-03-11T21:45:31.231351+0000 osd.7 (osd.7) 499 : cluster 0 2.b scrub ok 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:30.431786+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:30.433103+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:30.994850+0000 osd.1 (osd.1) 315 : cluster 0 5.c scrub starts 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:30.997302+0000 osd.1 (osd.1) 316 : cluster 0 5.c scrub ok 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:31.228946+0000 osd.7 (osd.7) 498 : cluster 0 2.b scrub starts 2024-03-11T21:45:32.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:31 smithi143 bash[21512]: cluster 2024-03-11T21:45:31.231351+0000 osd.7 (osd.7) 499 : cluster 0 2.b scrub ok 2024-03-11T21:45:32.445 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:32.635 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:32.635 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:33.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:32 smithi203 bash[23654]: cluster 2024-03-11T21:45:31.256160+0000 osd.2 (osd.2) 447 : cluster 0 2.10 scrub starts 2024-03-11T21:45:33.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:32 smithi203 bash[23654]: cluster 2024-03-11T21:45:31.258583+0000 osd.2 (osd.2) 448 : cluster 0 2.10 scrub ok 2024-03-11T21:45:33.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:32 smithi203 bash[23654]: cluster 2024-03-11T21:45:31.714822+0000 mgr.y (mgr.14550) 869 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:45:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:32 smithi203 bash[23654]: cluster 2024-03-11T21:45:32.637104+0000 mon.a (mon.0) 4070 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:33.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[29284]: cluster 2024-03-11T21:45:31.256160+0000 osd.2 (osd.2) 447 : cluster 0 2.10 scrub starts 2024-03-11T21:45:33.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[29284]: cluster 2024-03-11T21:45:31.258583+0000 osd.2 (osd.2) 448 : cluster 0 2.10 scrub ok 2024-03-11T21:45:33.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[29284]: cluster 2024-03-11T21:45:31.714822+0000 mgr.y (mgr.14550) 869 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:45:33.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[29284]: cluster 2024-03-11T21:45:32.637104+0000 mon.a (mon.0) 4070 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:33.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[21512]: cluster 2024-03-11T21:45:31.256160+0000 osd.2 (osd.2) 447 : cluster 0 2.10 scrub starts 2024-03-11T21:45:33.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[21512]: cluster 2024-03-11T21:45:31.258583+0000 osd.2 (osd.2) 448 : cluster 0 2.10 scrub ok 2024-03-11T21:45:33.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[21512]: cluster 2024-03-11T21:45:31.714822+0000 mgr.y (mgr.14550) 869 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 378 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-11T21:45:33.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:32 smithi143 bash[21512]: cluster 2024-03-11T21:45:32.637104+0000 mon.a (mon.0) 4070 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:45:33.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:45:33.760 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:45:33.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-03-11T21:45:33.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[21512]: cluster 2024-03-11T21:45:32.442444+0000 osd.0 (osd.0) 504 : cluster 0 5.3 scrub starts 2024-03-11T21:45:33.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[21512]: cluster 2024-03-11T21:45:32.444889+0000 osd.0 (osd.0) 505 : cluster 0 5.3 scrub ok 2024-03-11T21:45:33.945 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[21512]: cluster 2024-03-11T21:45:32.694301+0000 osd.4 (osd.4) 481 : cluster 0 2.18 scrub starts 2024-03-11T21:45:33.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[21512]: cluster 2024-03-11T21:45:32.696638+0000 osd.4 (osd.4) 482 : cluster 0 2.18 scrub ok 2024-03-11T21:45:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[29284]: cluster 2024-03-11T21:45:32.442444+0000 osd.0 (osd.0) 504 : cluster 0 5.3 scrub starts 2024-03-11T21:45:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[29284]: cluster 2024-03-11T21:45:32.444889+0000 osd.0 (osd.0) 505 : cluster 0 5.3 scrub ok 2024-03-11T21:45:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[29284]: cluster 2024-03-11T21:45:32.694301+0000 osd.4 (osd.4) 481 : cluster 0 2.18 scrub starts 2024-03-11T21:45:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:33 smithi143 bash[29284]: cluster 2024-03-11T21:45:32.696638+0000 osd.4 (osd.4) 482 : cluster 0 2.18 scrub ok 2024-03-11T21:45:34.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:45:34.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:33 smithi203 bash[23654]: cluster 2024-03-11T21:45:32.442444+0000 osd.0 (osd.0) 504 : cluster 0 5.3 scrub starts 2024-03-11T21:45:34.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:33 smithi203 bash[23654]: cluster 2024-03-11T21:45:32.444889+0000 osd.0 (osd.0) 505 : cluster 0 5.3 scrub ok 2024-03-11T21:45:34.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:33 smithi203 bash[23654]: cluster 2024-03-11T21:45:32.694301+0000 osd.4 (osd.4) 481 : cluster 0 2.18 scrub starts 2024-03-11T21:45:34.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:33 smithi203 bash[23654]: cluster 2024-03-11T21:45:32.696638+0000 osd.4 (osd.4) 482 : cluster 0 2.18 scrub ok 2024-03-11T21:45:35.088 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:33.268460+0000 osd.3 (osd.3) 513 : cluster 0 1.8 scrub starts 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:33.271344+0000 osd.3 (osd.3) 514 : cluster 0 1.8 scrub ok 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:33.715864+0000 mgr.y (mgr.14550) 870 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:33.743469+0000 osd.4 (osd.4) 483 : cluster 0 3.9 scrub starts 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:33.747557+0000 osd.4 (osd.4) 484 : cluster 0 3.9 scrub ok 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.238359+0000 osd.7 (osd.7) 500 : cluster 0 9.11 scrub starts 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.241192+0000 osd.7 (osd.7) 501 : cluster 0 9.11 scrub ok 2024-03-11T21:45:35.110 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:33.268460+0000 osd.3 (osd.3) 513 : cluster 0 1.8 scrub starts 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:33.271344+0000 osd.3 (osd.3) 514 : cluster 0 1.8 scrub ok 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:33.715864+0000 mgr.y (mgr.14550) 870 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:33.743469+0000 osd.4 (osd.4) 483 : cluster 0 3.9 scrub starts 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:33.747557+0000 osd.4 (osd.4) 484 : cluster 0 3.9 scrub ok 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.238359+0000 osd.7 (osd.7) 500 : cluster 0 9.11 scrub starts 2024-03-11T21:45:35.111 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:34 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.241192+0000 osd.7 (osd.7) 501 : cluster 0 9.11 scrub ok 2024-03-11T21:45:35.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:33.268460+0000 osd.3 (osd.3) 513 : cluster 0 1.8 scrub starts 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:33.271344+0000 osd.3 (osd.3) 514 : cluster 0 1.8 scrub ok 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:33.715864+0000 mgr.y (mgr.14550) 870 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:33.743469+0000 osd.4 (osd.4) 483 : cluster 0 3.9 scrub starts 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:33.747557+0000 osd.4 (osd.4) 484 : cluster 0 3.9 scrub ok 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.238359+0000 osd.7 (osd.7) 500 : cluster 0 9.11 scrub starts 2024-03-11T21:45:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:34 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.241192+0000 osd.7 (osd.7) 501 : cluster 0 9.11 scrub ok 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.262424+0000 osd.2 (osd.2) 449 : cluster 0 6.14 scrub starts 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.264698+0000 osd.2 (osd.2) 450 : cluster 0 6.14 scrub ok 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.290358+0000 osd.3 (osd.3) 515 : cluster 0 9.1a scrub starts 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.292792+0000 osd.3 (osd.3) 516 : cluster 0 9.1a scrub ok 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.724099+0000 osd.4 (osd.4) 485 : cluster 0 3.1a scrub starts 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:34.728009+0000 osd.4 (osd.4) 486 : cluster 0 3.1a scrub ok 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:35.268706+0000 osd.7 (osd.7) 502 : cluster 0 2.1c scrub starts 2024-03-11T21:45:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[21512]: cluster 2024-03-11T21:45:35.270991+0000 osd.7 (osd.7) 503 : cluster 0 2.1c scrub ok 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.262424+0000 osd.2 (osd.2) 449 : cluster 0 6.14 scrub starts 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.264698+0000 osd.2 (osd.2) 450 : cluster 0 6.14 scrub ok 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.290358+0000 osd.3 (osd.3) 515 : cluster 0 9.1a scrub starts 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.292792+0000 osd.3 (osd.3) 516 : cluster 0 9.1a scrub ok 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.724099+0000 osd.4 (osd.4) 485 : cluster 0 3.1a scrub starts 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:34.728009+0000 osd.4 (osd.4) 486 : cluster 0 3.1a scrub ok 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:35.268706+0000 osd.7 (osd.7) 502 : cluster 0 2.1c scrub starts 2024-03-11T21:45:36.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:35 smithi143 bash[29284]: cluster 2024-03-11T21:45:35.270991+0000 osd.7 (osd.7) 503 : cluster 0 2.1c scrub ok 2024-03-11T21:45:36.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.262424+0000 osd.2 (osd.2) 449 : cluster 0 6.14 scrub starts 2024-03-11T21:45:36.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.264698+0000 osd.2 (osd.2) 450 : cluster 0 6.14 scrub ok 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.290358+0000 osd.3 (osd.3) 515 : cluster 0 9.1a scrub starts 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.292792+0000 osd.3 (osd.3) 516 : cluster 0 9.1a scrub ok 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.724099+0000 osd.4 (osd.4) 485 : cluster 0 3.1a scrub starts 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:34.728009+0000 osd.4 (osd.4) 486 : cluster 0 3.1a scrub ok 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:35.268706+0000 osd.7 (osd.7) 502 : cluster 0 2.1c scrub starts 2024-03-11T21:45:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:35 smithi203 bash[23654]: cluster 2024-03-11T21:45:35.270991+0000 osd.7 (osd.7) 503 : cluster 0 2.1c scrub ok 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[21512]: cluster 2024-03-11T21:45:35.407468+0000 osd.0 (osd.0) 506 : cluster 0 5.5 scrub starts 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[21512]: cluster 2024-03-11T21:45:35.409766+0000 osd.0 (osd.0) 507 : cluster 0 5.5 scrub ok 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[21512]: cluster 2024-03-11T21:45:35.716927+0000 mgr.y (mgr.14550) 871 : cluster 0 pgmap v1520: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[29284]: cluster 2024-03-11T21:45:35.407468+0000 osd.0 (osd.0) 506 : cluster 0 5.5 scrub starts 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[29284]: cluster 2024-03-11T21:45:35.409766+0000 osd.0 (osd.0) 507 : cluster 0 5.5 scrub ok 2024-03-11T21:45:36.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:36 smithi143 bash[29284]: cluster 2024-03-11T21:45:35.716927+0000 mgr.y (mgr.14550) 871 : cluster 0 pgmap v1520: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:37.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:36 smithi203 bash[23654]: cluster 2024-03-11T21:45:35.407468+0000 osd.0 (osd.0) 506 : cluster 0 5.5 scrub starts 2024-03-11T21:45:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:36 smithi203 bash[23654]: cluster 2024-03-11T21:45:35.409766+0000 osd.0 (osd.0) 507 : cluster 0 5.5 scrub ok 2024-03-11T21:45:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:36 smithi203 bash[23654]: cluster 2024-03-11T21:45:35.716927+0000 mgr.y (mgr.14550) 871 : cluster 0 pgmap v1520: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:45:38.154 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:38.155 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:38.155 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:38.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:36.259114+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-03-11T21:45:38.195 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:36.261565+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:36.519871+0000 osd.6 (osd.6) 419 : cluster 0 2.16 scrub starts 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:36.522237+0000 osd.6 (osd.6) 420 : cluster 0 2.16 scrub ok 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.754239+0000 osd.4 (osd.4) 487 : cluster 0 2.f scrub starts 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.757451+0000 osd.4 (osd.4) 488 : cluster 0 2.f scrub ok 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[21512]: audit 2024-03-11T21:45:37.778591+0000 mon.c (mon.2) 1387 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:36.259114+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-03-11T21:45:38.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:36.261565+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-03-11T21:45:38.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:36.519871+0000 osd.6 (osd.6) 419 : cluster 0 2.16 scrub starts 2024-03-11T21:45:38.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:36.522237+0000 osd.6 (osd.6) 420 : cluster 0 2.16 scrub ok 2024-03-11T21:45:38.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.754239+0000 osd.4 (osd.4) 487 : cluster 0 2.f scrub starts 2024-03-11T21:45:38.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.757451+0000 osd.4 (osd.4) 488 : cluster 0 2.f scrub ok 2024-03-11T21:45:38.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:37 smithi143 bash[29284]: audit 2024-03-11T21:45:37.778591+0000 mon.c (mon.2) 1387 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:38.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:36.259114+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-03-11T21:45:38.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:36.261565+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-03-11T21:45:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:36.519871+0000 osd.6 (osd.6) 419 : cluster 0 2.16 scrub starts 2024-03-11T21:45:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:36.522237+0000 osd.6 (osd.6) 420 : cluster 0 2.16 scrub ok 2024-03-11T21:45:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.754239+0000 osd.4 (osd.4) 487 : cluster 0 2.f scrub starts 2024-03-11T21:45:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.757451+0000 osd.4 (osd.4) 488 : cluster 0 2.f scrub ok 2024-03-11T21:45:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:37 smithi203 bash[23654]: audit 2024-03-11T21:45:37.778591+0000 mon.c (mon.2) 1387 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 -- 172.21.15.143:0/2226205193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 msgr2=0x7fd410108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 --2- 172.21.15.143:0/2226205193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd410108890 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7fd400009990 tx=0x7fd400009ca0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 -- 172.21.15.143:0/2226205193 shutdown_connections 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 --2- 172.21.15.143:0/2226205193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd410108dd0 0x7fd4101152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 --2- 172.21.15.143:0/2226205193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd41010e4a0 0x7fd41010e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 --2- 172.21.15.143:0/2226205193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd410108890 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 -- 172.21.15.143:0/2226205193 >> 172.21.15.143:0/2226205193 conn(0x7fd4100755b0 msgr2=0x7fd4100779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 -- 172.21.15.143:0/2226205193 shutdown_connections 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 -- 172.21.15.143:0/2226205193 wait complete. 2024-03-11T21:45:38.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 librados: starting msgr at 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 librados: starting objecter 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.674+0000 7fd41544e700 1 Processor -- start 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- start start 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 librados: setting wanted keys 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 librados: calling monclient init 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd41019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd410108dd0 0x7fd41019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd41010e4a0 0x7fd4101a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410117c30 con 0x7fd41010e4a0 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410117ab0 con 0x7fd410108dd0 2024-03-11T21:45:38.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd410117db0 con 0x7fd410108410 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd41019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd41010e4a0 0x7fd4101a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd41019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46974/0 (socket says 172.21.15.143:46974) 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 -- 172.21.15.143:0/3999055141 learned_addr learned my addr 172.21.15.143:0/3999055141 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd410108dd0 0x7fd41019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd410108dd0 msgr2=0x7fd41019e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd410108dd0 0x7fd41019e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd41010e4a0 msgr2=0x7fd4101a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd41010e4a0 0x7fd4101a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4101a3270 con 0x7fd410108410 2024-03-11T21:45:38.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd41010e4a0 0x7fd4101a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd41019e260 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c970 tx=0x7fd3f800cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 10 client.?.objecter ms_handle_connect 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007aa0 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd3f8004cb0 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8005450 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4101a3560 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 10 client.42287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4101aade0 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 librados: init done 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd41004faa0 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd3f8007c40 con 0x7fd410108410 2024-03-11T21:45:38.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3fc06f820 0x7fd3fc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(965..965 src has 842..965) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fd3f808eae0 con 0x7fd410108410 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 osd_map(965..965 src has 842..965) v4 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 3 client.42287.objecter handle_osd_map got epochs [965,965] > 0 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 3 client.42287.objecter handle_osd_map decoding full epoch 965 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3fc06f820 0x7fd3fc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 20 client.42287.objecter dump_active .. 0 homeless 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v965) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd3f8025020 con 0x7fd410108410 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 10 client.42287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 1 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd41010fa60 0x7fd41010fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 1 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 0 1 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 4 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 10 client.42287.objecter _send_command 1 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.678+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101ae140 con 0x7fd41010fa60 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3fc06f820 0x7fd3fc071ce0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b2e0 tx=0x7fd400005e80 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd3fc06f820 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd41010fa60 0x7fd41010fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd41010fa60 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd41010fa60 0x7fd41010fe40 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.670 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101ae140 con 0x7fd41010fa60 2024-03-11T21:45:38.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:45:38.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 10 client.42287.objecter _finish_command 1 = system:0 2024-03-11T21:45:38.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 4 2024-03-11T21:45:38.671 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.682+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 0 1 2024-03-11T21:45:38.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 10 client.42287.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 2 2024-03-11T21:45:38.685 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 2 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd41019f5c0 osd=0 3 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 0 2 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 4 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 10 client.42287.objecter _send_command 2 2024-03-11T21:45:38.686 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.698+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd41010f4e0 con 0x7fd41010fa60 2024-03-11T21:45:38.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd41010f4e0 con 0x7fd41010fa60 2024-03-11T21:45:38.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 10 client.42287.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 4 2024-03-11T21:45:38.687 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 0 2 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stdout:osd.0: {} 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 10 client.42287.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 3 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd41002c6b0 0x7fd41002ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 1 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 4 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 10 client.42287.objecter _send_command 3 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd410063070 con 0x7fd41002c6b0 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd41002c6b0 0x7fd41002ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd41002c6b0 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd41002c6b0 0x7fd41002ca90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd410063070 con 0x7fd41002c6b0 2024-03-11T21:45:38.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 3: 0 ) v1 2024-03-11T21:45:38.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 10 client.42287.objecter _finish_command 3 = system:0 2024-03-11T21:45:38.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 4 2024-03-11T21:45:38.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.702+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 1 3 2024-03-11T21:45:38.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 10 client.42287.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 4 2024-03-11T21:45:38.706 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 4 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4100629f0 osd=1 3 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 1 4 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 4 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 10 client.42287.objecter _send_command 4 2024-03-11T21:45:38.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.718+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd410062470 con 0x7fd41002c6b0 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd410062470 con 0x7fd41002c6b0 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 10 client.42287.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 4 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 1 4 2024-03-11T21:45:38.708 INFO:teuthology.orchestra.run.smithi143.stdout:osd.1: {} 2024-03-11T21:45:38.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 10 client.42287.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 5 2024-03-11T21:45:38.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd4101b0300 0x7fd4101b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 5 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 2 5 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 4 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 10 client.42287.objecter _send_command 5 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101b5550 con 0x7fd4101b0300 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd4101b0300 0x7fd4101b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101b0300 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd4101b0300 0x7fd4101b26f0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101b5550 con 0x7fd4101b0300 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 5: 0 ) v1 2024-03-11T21:45:38.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 10 client.42287.objecter _finish_command 5 = system:0 2024-03-11T21:45:38.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 4 2024-03-11T21:45:38.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.722+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 2 5 2024-03-11T21:45:38.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 10 client.42287.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 6 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 6 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd410062870 osd=2 3 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 2 6 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 4 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 10 client.42287.objecter _send_command 6 2024-03-11T21:45:38.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.738+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd41002d220 con 0x7fd4101b0300 2024-03-11T21:45:38.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd41002d220 con 0x7fd4101b0300 2024-03-11T21:45:38.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 10 client.42287.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 4 2024-03-11T21:45:38.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 2 6 2024-03-11T21:45:38.729 INFO:teuthology.orchestra.run.smithi143.stdout:osd.2: {} 2024-03-11T21:45:38.729 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 10 client.42287.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 7 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd4101adca0 0x7fd4101b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 7 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 3 7 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 4 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 10 client.42287.objecter _send_command 7 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101b8eb0 con 0x7fd4101adca0 2024-03-11T21:45:38.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd4101adca0 0x7fd4101b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd4101adca0 0x7fd4101b3db0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101adca0 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101b8eb0 con 0x7fd4101adca0 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 7: 0 ) v1 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 10 client.42287.objecter _finish_command 7 = system:0 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 4 2024-03-11T21:45:38.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.742+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 3 7 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 10 client.42287.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 8 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 8 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd41002d620 osd=3 3 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 3 8 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 4 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 10 client.42287.objecter _send_command 8 2024-03-11T21:45:38.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4101ad2d0 con 0x7fd4101adca0 2024-03-11T21:45:38.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd4101ad2d0 con 0x7fd4101adca0 2024-03-11T21:45:38.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd407fff700 10 client.42287.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 4 2024-03-11T21:45:38.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 3 8 2024-03-11T21:45:38.750 INFO:teuthology.orchestra.run.smithi143.stdout:osd.3: {} 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 10 client.42287.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 9 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd4101b3910 0x7fd4101b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 9 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 4 9 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 4 2024-03-11T21:45:38.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 10 client.42287.objecter _send_command 9 2024-03-11T21:45:38.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101ad1f0 con 0x7fd4101b3910 2024-03-11T21:45:38.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.762+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd4101b3910 0x7fd4101b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101b3910 2024-03-11T21:45:38.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd4101b3910 0x7fd4101b7710 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101ad1f0 con 0x7fd4101b3910 2024-03-11T21:45:38.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 9: 0 ) v1 2024-03-11T21:45:38.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 10 client.42287.objecter _finish_command 9 = system:0 2024-03-11T21:45:38.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 4 2024-03-11T21:45:38.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.766+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 4 9 2024-03-11T21:45:38.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 10 client.42287.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 10 2024-03-11T21:45:38.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 10 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101ad080 osd=4 3 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 4 10 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 4 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 10 client.42287.objecter _send_command 10 2024-03-11T21:45:38.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.782+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4101ad1f0 con 0x7fd4101b3910 2024-03-11T21:45:38.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd4101ad1f0 con 0x7fd4101b3910 2024-03-11T21:45:38.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd407fff700 10 client.42287.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 4 2024-03-11T21:45:38.774 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 4 10 2024-03-11T21:45:38.775 INFO:teuthology.orchestra.run.smithi143.stdout:osd.4: {} 2024-03-11T21:45:38.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 10 client.42287.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.775 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 11 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd4101b8160 0x7fd4101b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 11 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 5 11 2024-03-11T21:45:38.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 4 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 10 client.42287.objecter _send_command 11 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.786+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101b8c80 con 0x7fd4101b8160 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd4101b8160 0x7fd4101b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd40e7fc700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd4101b8160 0x7fd4101b85c0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101b8160 2024-03-11T21:45:38.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101b8c80 con 0x7fd4101b8160 2024-03-11T21:45:38.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 11: 0 ) v1 2024-03-11T21:45:38.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 10 client.42287.objecter _finish_command 11 = system:0 2024-03-11T21:45:38.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 4 2024-03-11T21:45:38.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.790+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 5 11 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 10 client.42287.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 12 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 12 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101ad1f0 osd=5 3 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 5 12 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 4 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 10 client.42287.objecter _send_command 12 2024-03-11T21:45:38.795 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.810+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4101bd0e0 con 0x7fd4101b8160 2024-03-11T21:45:38.799 INFO:teuthology.orchestra.run.smithi143.stdout:osd.5: {} 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd4101bd0e0 con 0x7fd4101b8160 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 10 client.42287.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 4 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 5 12 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 10 client.42287.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 13 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd4101bd760 0x7fd4101bfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.800 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 13 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 6 13 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 4 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 10 client.42287.objecter _send_command 13 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101c2ab0 con 0x7fd4101bd760 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd4101bd760 0x7fd4101bfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd40f7fe700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd4101bd760 0x7fd4101bfb50 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101bd760 2024-03-11T21:45:38.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101c2ab0 con 0x7fd4101bd760 2024-03-11T21:45:38.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 13: 0 ) v1 2024-03-11T21:45:38.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 10 client.42287.objecter _finish_command 13 = system:0 2024-03-11T21:45:38.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 4 2024-03-11T21:45:38.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.814+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 6 13 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 10 client.42287.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 14 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 14 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101bd0e0 osd=6 3 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 6 14 2024-03-11T21:45:38.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 4 2024-03-11T21:45:38.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 10 client.42287.objecter _send_command 14 2024-03-11T21:45:38.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.834+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4101c2ab0 con 0x7fd4101bd760 2024-03-11T21:45:38.826 INFO:teuthology.orchestra.run.smithi143.stdout:osd.6: {} 2024-03-11T21:45:38.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd4101c2ab0 con 0x7fd4101bd760 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd407fff700 10 client.42287.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 4 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 6 14 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 10 client.42287.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 15 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd4101c0650 0x7fd4101c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 15 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 7 15 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 4 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 10 client.42287.objecter _send_command 15 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.838+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4101c5e50 con 0x7fd4101c0650 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd4101c0650 0x7fd4101c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 10 client.42287.objecter ms_handle_connect 0x7fd4101c0650 2024-03-11T21:45:38.828 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd40effd700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd4101c0650 0x7fd4101c0ab0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:38.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4101c5e50 con 0x7fd4101c0650 2024-03-11T21:45:38.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 15: 0 ) v1 2024-03-11T21:45:38.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 10 client.42287.objecter _finish_command 15 = system:0 2024-03-11T21:45:38.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 4 2024-03-11T21:45:38.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.842+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 7 15 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 10 client.42287.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign -1 16 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 20 client.42287.objecter _get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 15 client.42287.objecter _session_command_op_remove -1 16 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 20 client.42287.objecter get_session s=0x7fd4101c2ab0 osd=7 3 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 15 client.42287.objecter _session_command_op_assign 7 16 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 4 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 10 client.42287.objecter _send_command 16 2024-03-11T21:45:38.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.862+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4101c4cd0 con 0x7fd4101c0650 2024-03-11T21:45:38.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd407fff700 1 -- 172.21.15.143:0/3999055141 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd4101c4cd0 con 0x7fd4101c0650 2024-03-11T21:45:38.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd407fff700 10 client.42287.objecter ms_dispatch 0x7fd410001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-11T21:45:38.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd407fff700 10 client.42287.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-11T21:45:38.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd407fff700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 4 2024-03-11T21:45:38.851 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd407fff700 15 client.42287.objecter _session_command_op_remove 7 16 2024-03-11T21:45:38.852 INFO:teuthology.orchestra.run.smithi143.stdout:osd.7: {} 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 librados: watch_flush enter 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 librados: watch_flush exit 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.0 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd41010fa60 msgr2=0x7fd41010fe40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7fd41010fa60 0x7fd41010fe40 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41019f5c0 osd=0 1 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.1 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd41002c6b0 msgr2=0x7fd41002ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fd41002c6b0 0x7fd41002ca90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.853 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4100629f0 osd=1 1 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.2 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd4101b0300 msgr2=0x7fd4101b26f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fd4101b0300 0x7fd4101b26f0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd410062870 osd=2 1 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.3 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd4101adca0 msgr2=0x7fd4101b3db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7fd4101adca0 0x7fd4101b3db0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd41002d620 osd=3 1 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.4 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd4101b3910 msgr2=0x7fd4101b7710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fd4101b3910 0x7fd4101b7710 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad080 osd=4 1 2024-03-11T21:45:38.854 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.5 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd4101b8160 msgr2=0x7fd4101b85c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7fd4101b8160 0x7fd4101b85c0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101ad1f0 osd=5 1 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.6 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd4101bd760 msgr2=0x7fd4101bfb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7fd4101bd760 0x7fd4101bfb50 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101bd0e0 osd=6 1 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter close_session for osd.7 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd4101c0650 msgr2=0x7fd4101c0ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7fd4101c0650 0x7fd4101c0ab0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter put_session s=0x7fd4101c2ab0 osd=7 1 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 20 client.42287.objecter shutdown clearing up homeless session... 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 10 client.42287.objecter successfully canceled tick 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3fc06f820 msgr2=0x7fd3fc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd3fc06f820 0x7fd3fc071ce0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b2e0 tx=0x7fd400005e80 comp rx=0 tx=0).stop 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 msgr2=0x7fd41019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 --2- 172.21.15.143:0/3999055141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd410108410 0x7fd41019e260 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c970 tx=0x7fd3f800cd30 comp rx=0 tx=0).stop 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd40f7fe700 1 -- 172.21.15.143:0/3999055141 reap_dead start 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 shutdown_connections 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 >> 172.21.15.143:0/3999055141 conn(0x7fd4100755b0 msgr2=0x7fd410076ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 shutdown_connections 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 -- 172.21.15.143:0/3999055141 wait complete. 2024-03-11T21:45:38.856 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:38.866+0000 7fd41544e700 1 librados: shutdown 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:38.857 INFO:teuthology.orchestra.run.smithi143.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:39.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.302668+0000 osd.2 (osd.2) 453 : cluster 0 5.8 scrub starts 2024-03-11T21:45:39.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.303946+0000 osd.2 (osd.2) 454 : cluster 0 5.8 scrub ok 2024-03-11T21:45:39.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.415088+0000 osd.0 (osd.0) 508 : cluster 0 3.1f scrub starts 2024-03-11T21:45:39.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.419260+0000 osd.0 (osd.0) 509 : cluster 0 3.1f scrub ok 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:37.717747+0000 mgr.y (mgr.14550) 872 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: audit 2024-03-11T21:45:38.682301+0000 mon.c (mon.2) 1388 : audit 0 from='client.? 172.21.15.143:0/3999055141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.778029+0000 osd.4 (osd.4) 489 : cluster 0 3.14 scrub starts 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.781945+0000 osd.4 (osd.4) 490 : cluster 0 3.14 scrub ok 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.302668+0000 osd.2 (osd.2) 453 : cluster 0 5.8 scrub starts 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.303946+0000 osd.2 (osd.2) 454 : cluster 0 5.8 scrub ok 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.415088+0000 osd.0 (osd.0) 508 : cluster 0 3.1f scrub starts 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.419260+0000 osd.0 (osd.0) 509 : cluster 0 3.1f scrub ok 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:37.717747+0000 mgr.y (mgr.14550) 872 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: audit 2024-03-11T21:45:38.682301+0000 mon.c (mon.2) 1388 : audit 0 from='client.? 172.21.15.143:0/3999055141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.778029+0000 osd.4 (osd.4) 489 : cluster 0 3.14 scrub starts 2024-03-11T21:45:39.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:38 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.781945+0000 osd.4 (osd.4) 490 : cluster 0 3.14 scrub ok 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.302668+0000 osd.2 (osd.2) 453 : cluster 0 5.8 scrub starts 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.303946+0000 osd.2 (osd.2) 454 : cluster 0 5.8 scrub ok 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.415088+0000 osd.0 (osd.0) 508 : cluster 0 3.1f scrub starts 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.419260+0000 osd.0 (osd.0) 509 : cluster 0 3.1f scrub ok 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:37.717747+0000 mgr.y (mgr.14550) 872 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:39.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: audit 2024-03-11T21:45:38.682301+0000 mon.c (mon.2) 1388 : audit 0 from='client.? 172.21.15.143:0/3999055141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:39.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.778029+0000 osd.4 (osd.4) 489 : cluster 0 3.14 scrub starts 2024-03-11T21:45:39.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:38 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.781945+0000 osd.4 (osd.4) 490 : cluster 0 3.14 scrub ok 2024-03-11T21:45:39.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.262182+0000 osd.2 (osd.2) 455 : cluster 0 1.a scrub starts 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.263997+0000 osd.2 (osd.2) 456 : cluster 0 1.a scrub ok 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.925325+0000 osd.1 (osd.1) 317 : cluster 0 9.1f scrub starts 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[29284]: cluster 2024-03-11T21:45:38.930036+0000 osd.1 (osd.1) 318 : cluster 0 9.1f scrub ok 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.262182+0000 osd.2 (osd.2) 455 : cluster 0 1.a scrub starts 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.263997+0000 osd.2 (osd.2) 456 : cluster 0 1.a scrub ok 2024-03-11T21:45:40.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.925325+0000 osd.1 (osd.1) 317 : cluster 0 9.1f scrub starts 2024-03-11T21:45:40.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:39 smithi143 bash[21512]: cluster 2024-03-11T21:45:38.930036+0000 osd.1 (osd.1) 318 : cluster 0 9.1f scrub ok 2024-03-11T21:45:40.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:39 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.262182+0000 osd.2 (osd.2) 455 : cluster 0 1.a scrub starts 2024-03-11T21:45:40.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:39 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.263997+0000 osd.2 (osd.2) 456 : cluster 0 1.a scrub ok 2024-03-11T21:45:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:39 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.925325+0000 osd.1 (osd.1) 317 : cluster 0 9.1f scrub starts 2024-03-11T21:45:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:39 smithi203 bash[23654]: cluster 2024-03-11T21:45:38.930036+0000 osd.1 (osd.1) 318 : cluster 0 9.1f scrub ok 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: audit 2024-03-11T21:45:39.215592+0000 mgr.y (mgr.14550) 873 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.311205+0000 osd.2 (osd.2) 457 : cluster 0 1.14 scrub starts 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.314141+0000 osd.2 (osd.2) 458 : cluster 0 1.14 scrub ok 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.355961+0000 osd.7 (osd.7) 504 : cluster 0 3.f scrub starts 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.360734+0000 osd.7 (osd.7) 505 : cluster 0 3.f scrub ok 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.637733+0000 osd.6 (osd.6) 421 : cluster 0 3.a scrub starts 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.644957+0000 osd.6 (osd.6) 422 : cluster 0 3.a scrub ok 2024-03-11T21:45:41.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[29284]: cluster 2024-03-11T21:45:39.719552+0000 mgr.y (mgr.14550) 874 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: audit 2024-03-11T21:45:39.215592+0000 mgr.y (mgr.14550) 873 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.311205+0000 osd.2 (osd.2) 457 : cluster 0 1.14 scrub starts 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.314141+0000 osd.2 (osd.2) 458 : cluster 0 1.14 scrub ok 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.355961+0000 osd.7 (osd.7) 504 : cluster 0 3.f scrub starts 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.360734+0000 osd.7 (osd.7) 505 : cluster 0 3.f scrub ok 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.637733+0000 osd.6 (osd.6) 421 : cluster 0 3.a scrub starts 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.644957+0000 osd.6 (osd.6) 422 : cluster 0 3.a scrub ok 2024-03-11T21:45:41.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:40 smithi143 bash[21512]: cluster 2024-03-11T21:45:39.719552+0000 mgr.y (mgr.14550) 874 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:41.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: audit 2024-03-11T21:45:39.215592+0000 mgr.y (mgr.14550) 873 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.311205+0000 osd.2 (osd.2) 457 : cluster 0 1.14 scrub starts 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.314141+0000 osd.2 (osd.2) 458 : cluster 0 1.14 scrub ok 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.355961+0000 osd.7 (osd.7) 504 : cluster 0 3.f scrub starts 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.360734+0000 osd.7 (osd.7) 505 : cluster 0 3.f scrub ok 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.637733+0000 osd.6 (osd.6) 421 : cluster 0 3.a scrub starts 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.644957+0000 osd.6 (osd.6) 422 : cluster 0 3.a scrub ok 2024-03-11T21:45:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:40 smithi203 bash[23654]: cluster 2024-03-11T21:45:39.719552+0000 mgr.y (mgr.14550) 874 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:42.446 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:42.531 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[51372]: logger=cleanup t=2024-03-11T21:45:42.061198715Z level=info msg="Completed cleanup jobs" duration=57.640512ms 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.381227+0000 osd.0 (osd.0) 510 : cluster 0 3.1b scrub starts 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.383830+0000 osd.0 (osd.0) 511 : cluster 0 3.1b scrub ok 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.476650+0000 osd.6 (osd.6) 423 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.477088+0000 osd.6 (osd.6) 424 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.720347+0000 mgr.y (mgr.14550) 875 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:43.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[29284]: cluster 2024-03-11T21:45:41.852073+0000 mon.a (mon.0) 4071 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.381227+0000 osd.0 (osd.0) 510 : cluster 0 3.1b scrub starts 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.383830+0000 osd.0 (osd.0) 511 : cluster 0 3.1b scrub ok 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.476650+0000 osd.6 (osd.6) 423 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.477088+0000 osd.6 (osd.6) 424 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.720347+0000 mgr.y (mgr.14550) 875 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:42 smithi143 bash[21512]: cluster 2024-03-11T21:45:41.852073+0000 mon.a (mon.0) 4071 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-03-11T21:45:43.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.381227+0000 osd.0 (osd.0) 510 : cluster 0 3.1b scrub starts 2024-03-11T21:45:43.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.383830+0000 osd.0 (osd.0) 511 : cluster 0 3.1b scrub ok 2024-03-11T21:45:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.476650+0000 osd.6 (osd.6) 423 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.477088+0000 osd.6 (osd.6) 424 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.720347+0000 mgr.y (mgr.14550) 875 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:45:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:42 smithi203 bash[23654]: cluster 2024-03-11T21:45:41.852073+0000 mon.a (mon.0) 4071 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-03-11T21:45:44.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:42.290005+0000 osd.3 (osd.3) 517 : cluster 0 3.7 scrub starts 2024-03-11T21:45:44.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:42.294252+0000 osd.3 (osd.3) 518 : cluster 0 3.7 scrub ok 2024-03-11T21:45:44.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:42.595649+0000 osd.6 (osd.6) 425 : cluster 0 2.14 scrub starts 2024-03-11T21:45:44.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:42.600790+0000 osd.6 (osd.6) 426 : cluster 0 2.14 scrub ok 2024-03-11T21:45:44.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.110155+0000 osd.4 (osd.4) 491 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.110644+0000 osd.4 (osd.4) 492 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:42.290005+0000 osd.3 (osd.3) 517 : cluster 0 3.7 scrub starts 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:42.294252+0000 osd.3 (osd.3) 518 : cluster 0 3.7 scrub ok 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:42.595649+0000 osd.6 (osd.6) 425 : cluster 0 2.14 scrub starts 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:42.600790+0000 osd.6 (osd.6) 426 : cluster 0 2.14 scrub ok 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.110155+0000 osd.4 (osd.4) 491 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:44.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:43 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.110644+0000 osd.4 (osd.4) 492 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:44.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:42.290005+0000 osd.3 (osd.3) 517 : cluster 0 3.7 scrub starts 2024-03-11T21:45:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:42.294252+0000 osd.3 (osd.3) 518 : cluster 0 3.7 scrub ok 2024-03-11T21:45:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:42.595649+0000 osd.6 (osd.6) 425 : cluster 0 2.14 scrub starts 2024-03-11T21:45:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:42.600790+0000 osd.6 (osd.6) 426 : cluster 0 2.14 scrub ok 2024-03-11T21:45:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.110155+0000 osd.4 (osd.4) 491 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:43 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.110644+0000 osd.4 (osd.4) 492 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.333315+0000 osd.0 (osd.0) 512 : cluster 0 9.b scrub starts 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.341125+0000 osd.0 (osd.0) 513 : cluster 0 9.b scrub ok 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.637513+0000 osd.6 (osd.6) 427 : cluster 0 5.1f scrub starts 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.640963+0000 osd.6 (osd.6) 428 : cluster 0 5.1f scrub ok 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.722041+0000 mgr.y (mgr.14550) 876 : cluster 0 pgmap v1525: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.855714+0000 mon.a (mon.0) 4072 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.873092+0000 osd.1 (osd.1) 319 : cluster 0 9.4 scrub starts 2024-03-11T21:45:45.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[21512]: cluster 2024-03-11T21:45:43.875733+0000 osd.1 (osd.1) 320 : cluster 0 9.4 scrub ok 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.333315+0000 osd.0 (osd.0) 512 : cluster 0 9.b scrub starts 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.341125+0000 osd.0 (osd.0) 513 : cluster 0 9.b scrub ok 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.637513+0000 osd.6 (osd.6) 427 : cluster 0 5.1f scrub starts 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.640963+0000 osd.6 (osd.6) 428 : cluster 0 5.1f scrub ok 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.722041+0000 mgr.y (mgr.14550) 876 : cluster 0 pgmap v1525: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.855714+0000 mon.a (mon.0) 4072 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.873092+0000 osd.1 (osd.1) 319 : cluster 0 9.4 scrub starts 2024-03-11T21:45:45.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:44 smithi143 bash[29284]: cluster 2024-03-11T21:45:43.875733+0000 osd.1 (osd.1) 320 : cluster 0 9.4 scrub ok 2024-03-11T21:45:45.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.333315+0000 osd.0 (osd.0) 512 : cluster 0 9.b scrub starts 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.341125+0000 osd.0 (osd.0) 513 : cluster 0 9.b scrub ok 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.637513+0000 osd.6 (osd.6) 427 : cluster 0 5.1f scrub starts 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.640963+0000 osd.6 (osd.6) 428 : cluster 0 5.1f scrub ok 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.722041+0000 mgr.y (mgr.14550) 876 : cluster 0 pgmap v1525: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.855714+0000 mon.a (mon.0) 4072 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.873092+0000 osd.1 (osd.1) 319 : cluster 0 9.4 scrub starts 2024-03-11T21:45:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:44 smithi203 bash[23654]: cluster 2024-03-11T21:45:43.875733+0000 osd.1 (osd.1) 320 : cluster 0 9.4 scrub ok 2024-03-11T21:45:46.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:44.437821+0000 osd.2 (osd.2) 459 : cluster 0 2.1 scrub starts 2024-03-11T21:45:46.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:44.440090+0000 osd.2 (osd.2) 460 : cluster 0 2.1 scrub ok 2024-03-11T21:45:46.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:44.481628+0000 osd.7 (osd.7) 506 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:46.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:44.482088+0000 osd.7 (osd.7) 507 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:46.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:44.870443+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.305481+0000 osd.7 (osd.7) 508 : cluster 0 5.9 scrub starts 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.309223+0000 osd.7 (osd.7) 509 : cluster 0 5.9 scrub ok 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:44.437821+0000 osd.2 (osd.2) 459 : cluster 0 2.1 scrub starts 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:44.440090+0000 osd.2 (osd.2) 460 : cluster 0 2.1 scrub ok 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:44.481628+0000 osd.7 (osd.7) 506 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:44.482088+0000 osd.7 (osd.7) 507 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:44.870443+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.305481+0000 osd.7 (osd.7) 508 : cluster 0 5.9 scrub starts 2024-03-11T21:45:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:45 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.309223+0000 osd.7 (osd.7) 509 : cluster 0 5.9 scrub ok 2024-03-11T21:45:46.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:44.437821+0000 osd.2 (osd.2) 459 : cluster 0 2.1 scrub starts 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:44.440090+0000 osd.2 (osd.2) 460 : cluster 0 2.1 scrub ok 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:44.481628+0000 osd.7 (osd.7) 506 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:44.482088+0000 osd.7 (osd.7) 507 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:44.870443+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.305481+0000 osd.7 (osd.7) 508 : cluster 0 5.9 scrub starts 2024-03-11T21:45:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:45 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.309223+0000 osd.7 (osd.7) 509 : cluster 0 5.9 scrub ok 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/805524413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 msgr2=0x7fdc74107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/805524413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74107160 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009990 tx=0x7fdc64009ca0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/805524413 shutdown_connections 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/805524413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74107160 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/805524413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc741016d0 0x7fdc7410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/805524413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc74100d10 0x7fdc74101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/805524413 >> 172.21.15.143:0/805524413 conn(0x7fdc74077cb0 msgr2=0x7fdc740780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:46.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/805524413 shutdown_connections 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/805524413 wait complete. 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 librados: starting msgr at 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 librados: starting objecter 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 Processor -- start 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- start start 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 librados: setting wanted keys 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 librados: calling monclient init 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 0x7fdc7419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc74110550 con 0x7fdc74106d80 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741103d0 con 0x7fdc741016d0 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.778+0000 7fdc7a4eb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741106d0 con 0x7fdc74100d10 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:46.768 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48522/0 (socket says 172.21.15.143:48522) 2024-03-11T21:45:46.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 -- 172.21.15.143:0/3957390175 learned_addr learned my addr 172.21.15.143:0/3957390175 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:46.769 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc78a88700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc737fe700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 0x7fdc7419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 -- 172.21.15.143:0/3957390175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 msgr2=0x7fdc7419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 0x7fdc7419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 -- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 msgr2=0x7fdc74197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:46.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 -- 172.21.15.143:0/3957390175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc741981c0 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc737fe700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 0x7fdc7419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc73fff700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fdc640057f0 tx=0x7fdc6400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 10 client.?.objecter ms_handle_connect 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64026070 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fdc6400fbc0 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc640173f0 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc74198450 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc7a4eb700 10 client.42293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc741a4770 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc7a4eb700 1 librados: init done 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc74102430 con 0x7fdc74100d10 2024-03-11T21:45:46.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fdc64009d10 con 0x7fdc74100d10 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.782+0000 7fdc717fa700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 0x7fdc60071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(969..969 src has 842..969) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fdc6408f0e0 con 0x7fdc74100d10 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 10 client.42293.objecter ms_dispatch 0x7fdc74001550 osd_map(969..969 src has 842..969) v4 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 3 client.42293.objecter handle_osd_map got epochs [969,969] > 0 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 3 client.42293.objecter handle_osd_map decoding full epoch 969 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc737fe700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 0x7fdc60071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:46.772 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 20 client.42293.objecter dump_active .. 0 homeless 2024-03-11T21:45:46.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc717fa700 10 client.42293.objecter ms_handle_connect 0x7fdc6006f820 2024-03-11T21:45:46.773 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.786+0000 7fdc737fe700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 0x7fdc60071ce0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0094f0 tx=0x7fdc5c009040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:46.776 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:46.790+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fdc64013070 con 0x7fdc74100d10 2024-03-11T21:45:47.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.022+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fdc74198f90 con 0x7fdc74100d10 2024-03-11T21:45:47.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/3170530771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0720107220 msgr2=0x7f072010edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- 172.21.15.143:0/3170530771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0720107220 0x7f072010edd0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f071800b0a0 tx=0x7f071800b3b0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/3170530771 shutdown_connections 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- 172.21.15.143:0/3170530771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f072006a6a0 0x7f072006aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- 172.21.15.143:0/3170530771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0720107220 0x7f072010edd0 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- 172.21.15.143:0/3170530771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006b050 0x7f0720106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/3170530771 >> 172.21.15.143:0/3170530771 conn(0x7f0720077de0 msgr2=0x7f07200781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/3170530771 shutdown_connections 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/3170530771 wait complete. 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 librados: starting msgr at 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 librados: starting objecter 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 Processor -- start 2024-03-11T21:45:47.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- start start 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 librados: setting wanted keys 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 librados: calling monclient init 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006a6a0 0x7f0720199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0720107220 0x7f072019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07201116e0 con 0x7f0720107220 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0720111560 con 0x7f072006a6a0 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0720111860 con 0x7f072006b050 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48548/0 (socket says 172.21.15.143:48548) 2024-03-11T21:45:47.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 -- 172.21.15.143:0/1467715407 learned_addr learned my addr 172.21.15.143:0/1467715407 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0725d2e700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0720107220 0x7f072019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f072552d700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006a6a0 0x7f0720199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 -- 172.21.15.143:0/1467715407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006a6a0 msgr2=0x7f0720199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006a6a0 0x7f0720199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 -- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0720107220 msgr2=0x7f072019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0720107220 0x7f072019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 -- 172.21.15.143:0/1467715407 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072019ee80 con 0x7f072006b050 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0724d2c700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f071000b500 tx=0x7f071000b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f07167fc700 10 client.?.objecter ms_handle_connect 0x7f072006b050 2024-03-11T21:45:47.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f07167fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710015020 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0710011cb0 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072019f170 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710010c90 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 10 client.42299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07201a69f0 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.082+0000 7f0727791700 1 librados: init done 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072004faa0 con 0x7f072006b050 2024-03-11T21:45:47.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0710018020 con 0x7f072006b050 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 0x7f070c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(969..969 src has 842..969) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f071008de10 con 0x7f072006b050 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 10 client.42299.objecter ms_dispatch 0x7f0720001550 osd_map(969..969 src has 842..969) v4 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 3 client.42299.objecter handle_osd_map got epochs [969,969] > 0 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 3 client.42299.objecter handle_osd_map decoding full epoch 969 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 20 client.42299.objecter dump_active .. 0 homeless 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f072552d700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 0x7f070c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:45:47.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f07167fc700 10 client.42299.objecter ms_handle_connect 0x7f070c06f820 2024-03-11T21:45:47.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.086+0000 7f072552d700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 0x7f070c071ce0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f071c009680 tx=0x7f071c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:45:47.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.090+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f071001f020 con 0x7f072006b050 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.350592+0000 osd.3 (osd.3) 519 : cluster 0 6.1f scrub starts 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.352849+0000 osd.3 (osd.3) 520 : cluster 0 6.1f scrub ok 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.576863+0000 osd.6 (osd.6) 429 : cluster 0 2.17 scrub starts 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.579288+0000 osd.6 (osd.6) 430 : cluster 0 2.17 scrub ok 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.723062+0000 mgr.y (mgr.14550) 877 : cluster 0 pgmap v1528: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.788191+0000 osd.1 (osd.1) 321 : cluster 0 3.19 scrub starts 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:45.792824+0000 osd.1 (osd.1) 322 : cluster 0 3.19 scrub ok 2024-03-11T21:45:47.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[29284]: cluster 2024-03-11T21:45:46.645989+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.350592+0000 osd.3 (osd.3) 519 : cluster 0 6.1f scrub starts 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.352849+0000 osd.3 (osd.3) 520 : cluster 0 6.1f scrub ok 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.576863+0000 osd.6 (osd.6) 429 : cluster 0 2.17 scrub starts 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.579288+0000 osd.6 (osd.6) 430 : cluster 0 2.17 scrub ok 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.723062+0000 mgr.y (mgr.14550) 877 : cluster 0 pgmap v1528: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.788191+0000 osd.1 (osd.1) 321 : cluster 0 3.19 scrub starts 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:45.792824+0000 osd.1 (osd.1) 322 : cluster 0 3.19 scrub ok 2024-03-11T21:45:47.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:46 smithi143 bash[21512]: cluster 2024-03-11T21:45:46.645989+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-03-11T21:45:47.228 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:45:47.229 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:45:47.229 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:45:47.229 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.350592+0000 osd.3 (osd.3) 519 : cluster 0 6.1f scrub starts 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.352849+0000 osd.3 (osd.3) 520 : cluster 0 6.1f scrub ok 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.576863+0000 osd.6 (osd.6) 429 : cluster 0 2.17 scrub starts 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.579288+0000 osd.6 (osd.6) 430 : cluster 0 2.17 scrub ok 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.723062+0000 mgr.y (mgr.14550) 877 : cluster 0 pgmap v1528: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.788191+0000 osd.1 (osd.1) 321 : cluster 0 3.19 scrub starts 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:45.792824+0000 osd.1 (osd.1) 322 : cluster 0 3.19 scrub ok 2024-03-11T21:45:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:46 smithi203 bash[23654]: cluster 2024-03-11T21:45:46.645989+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-03-11T21:45:47.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.302+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} v 0) v1 -- 0x7f07201a6c20 con 0x7f072006b050 2024-03-11T21:45:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.902+0000 7fdc717fa700 1 -- 172.21.15.143:0/3957390175 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v970) v1 ==== 91+0+0 (secure 0 0 0) 0x7fdc640095a0 con 0x7fdc74100d10 2024-03-11T21:45:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.902+0000 7f07167fc700 1 -- 172.21.15.143:0/1467715407 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]=0 set pool 1 pgp_num to 21 v970) v1 ==== 131+0+0 (secure 0 0 0) 0x7f071005ae20 con 0x7f072006b050 2024-03-11T21:45:47.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.902+0000 7fdc7a4eb700 10 librados: watch_flush enter 2024-03-11T21:45:47.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 10 librados: watch_flush exit 2024-03-11T21:45:47.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 20 client.42293.objecter shutdown clearing up homeless session... 2024-03-11T21:45:47.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 10 client.42293.objecter successfully canceled tick 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 msgr2=0x7fdc60071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 0x7fdc60071ce0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0094f0 tx=0x7fdc5c009040 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 msgr2=0x7fdc7419d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fdc640057f0 tx=0x7fdc6400bfa0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 shutdown_connections 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdc6006f820 0x7fdc60071ce0 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc74106d80 0x7fdc74197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc74100d10 0x7fdc7419d8a0 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 --2- 172.21.15.143:0/3957390175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc741016d0 0x7fdc7419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 >> 172.21.15.143:0/3957390175 conn(0x7fdc74077cb0 msgr2=0x7fdc7406a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:47.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 shutdown_connections 2024-03-11T21:45:47.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 -- 172.21.15.143:0/3957390175 wait complete. 2024-03-11T21:45:47.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7fdc7a4eb700 1 librados: shutdown 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 10 librados: watch_flush enter 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 10 librados: watch_flush exit 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 20 client.42299.objecter shutdown clearing up homeless session... 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 10 client.42299.objecter successfully canceled tick 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 msgr2=0x7f070c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 0x7f070c071ce0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f071c009680 tx=0x7f071c009260 comp rx=0 tx=0).stop 2024-03-11T21:45:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 msgr2=0x7f072019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f071000b500 tx=0x7f071000b810 comp rx=0 tx=0).stop 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 shutdown_connections 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f070c06f820 0x7f070c071ce0 secure :-1 s=CLOSED pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f071c009680 tx=0x7f071c009260 comp rx=0 tx=0).stop 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0720107220 0x7f072019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f072006a6a0 0x7f0720199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 --2- 172.21.15.143:0/1467715407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f072006b050 0x7f072019a400 secure :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f071000b500 tx=0x7f071000b810 comp rx=0 tx=0).stop 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 >> 172.21.15.143:0/1467715407 conn(0x7f0720077de0 msgr2=0x7f072010d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:45:47.895 INFO:teuthology.orchestra.run.smithi143.stderr:noscrub is unset 2024-03-11T21:45:47.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 shutdown_connections 2024-03-11T21:45:47.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 -- 172.21.15.143:0/1467715407 wait complete. 2024-03-11T21:45:47.896 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:45:47.906+0000 7f0727791700 1 librados: shutdown 2024-03-11T21:45:47.896 INFO:teuthology.orchestra.run.smithi143.stderr:set pool 1 pgp_num to 21 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: cluster 2024-03-11T21:45:46.876215+0000 osd.4 (osd.4) 493 : cluster 0 6.2 scrub starts 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: cluster 2024-03-11T21:45:46.878342+0000 osd.4 (osd.4) 494 : cluster 0 6.2 scrub ok 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: audit 2024-03-11T21:45:47.027263+0000 mon.c (mon.2) 1389 : audit 1 from='client.? 172.21.15.143:0/3957390175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: audit 2024-03-11T21:45:47.027603+0000 mon.a (mon.0) 4075 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: audit 2024-03-11T21:45:47.306386+0000 mon.c (mon.2) 1390 : audit 1 from='client.? 172.21.15.143:0/1467715407' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[21512]: audit 2024-03-11T21:45:47.306717+0000 mon.a (mon.0) 4076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: cluster 2024-03-11T21:45:46.876215+0000 osd.4 (osd.4) 493 : cluster 0 6.2 scrub starts 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: cluster 2024-03-11T21:45:46.878342+0000 osd.4 (osd.4) 494 : cluster 0 6.2 scrub ok 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: audit 2024-03-11T21:45:47.027263+0000 mon.c (mon.2) 1389 : audit 1 from='client.? 172.21.15.143:0/3957390175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: audit 2024-03-11T21:45:47.027603+0000 mon.a (mon.0) 4075 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: audit 2024-03-11T21:45:47.306386+0000 mon.c (mon.2) 1390 : audit 1 from='client.? 172.21.15.143:0/1467715407' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:47 smithi143 bash[29284]: audit 2024-03-11T21:45:47.306717+0000 mon.a (mon.0) 4076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:45:48.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: cluster 2024-03-11T21:45:46.876215+0000 osd.4 (osd.4) 493 : cluster 0 6.2 scrub starts 2024-03-11T21:45:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: cluster 2024-03-11T21:45:46.878342+0000 osd.4 (osd.4) 494 : cluster 0 6.2 scrub ok 2024-03-11T21:45:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: audit 2024-03-11T21:45:47.027263+0000 mon.c (mon.2) 1389 : audit 1 from='client.? 172.21.15.143:0/3957390175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: audit 2024-03-11T21:45:47.027603+0000 mon.a (mon.0) 4075 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: audit 2024-03-11T21:45:47.306386+0000 mon.c (mon.2) 1390 : audit 1 from='client.? 172.21.15.143:0/1467715407' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:47 smithi203 bash[23654]: audit 2024-03-11T21:45:47.306717+0000 mon.a (mon.0) 4076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:45:48.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.665753+0000 osd.6 (osd.6) 431 : cluster 0 2.7 scrub starts 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.669723+0000 osd.6 (osd.6) 432 : cluster 0 2.7 scrub ok 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.723847+0000 mgr.y (mgr.14550) 878 : cluster 0 pgmap v1530: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.738689+0000 osd.1 (osd.1) 323 : cluster 0 2.a scrub starts 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.741260+0000 osd.1 (osd.1) 324 : cluster 0 2.a scrub ok 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.889536+0000 osd.4 (osd.4) 495 : cluster 0 9.16 scrub starts 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: audit 2024-03-11T21:45:47.893631+0000 mon.a (mon.0) 4077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: audit 2024-03-11T21:45:47.893937+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.898282+0000 osd.4 (osd.4) 496 : cluster 0 9.16 scrub ok 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[21512]: cluster 2024-03-11T21:45:47.904639+0000 mon.a (mon.0) 4079 : cluster 0 osdmap e970: 8 total, 8 up, 8 in 2024-03-11T21:45:48.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.665753+0000 osd.6 (osd.6) 431 : cluster 0 2.7 scrub starts 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.669723+0000 osd.6 (osd.6) 432 : cluster 0 2.7 scrub ok 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.723847+0000 mgr.y (mgr.14550) 878 : cluster 0 pgmap v1530: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.738689+0000 osd.1 (osd.1) 323 : cluster 0 2.a scrub starts 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.741260+0000 osd.1 (osd.1) 324 : cluster 0 2.a scrub ok 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.889536+0000 osd.4 (osd.4) 495 : cluster 0 9.16 scrub starts 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: audit 2024-03-11T21:45:47.893631+0000 mon.a (mon.0) 4077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: audit 2024-03-11T21:45:47.893937+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.898282+0000 osd.4 (osd.4) 496 : cluster 0 9.16 scrub ok 2024-03-11T21:45:48.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:48 smithi143 bash[29284]: cluster 2024-03-11T21:45:47.904639+0000 mon.a (mon.0) 4079 : cluster 0 osdmap e970: 8 total, 8 up, 8 in 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.665753+0000 osd.6 (osd.6) 431 : cluster 0 2.7 scrub starts 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.669723+0000 osd.6 (osd.6) 432 : cluster 0 2.7 scrub ok 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.723847+0000 mgr.y (mgr.14550) 878 : cluster 0 pgmap v1530: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.738689+0000 osd.1 (osd.1) 323 : cluster 0 2.a scrub starts 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.741260+0000 osd.1 (osd.1) 324 : cluster 0 2.a scrub ok 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.889536+0000 osd.4 (osd.4) 495 : cluster 0 9.16 scrub starts 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: audit 2024-03-11T21:45:47.893631+0000 mon.a (mon.0) 4077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: audit 2024-03-11T21:45:47.893937+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.898282+0000 osd.4 (osd.4) 496 : cluster 0 9.16 scrub ok 2024-03-11T21:45:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:48 smithi203 bash[23654]: cluster 2024-03-11T21:45:47.904639+0000 mon.a (mon.0) 4079 : cluster 0 osdmap e970: 8 total, 8 up, 8 in 2024-03-11T21:45:49.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:45:49.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:45:49.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-11T21:45:50.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[21512]: cluster 2024-03-11T21:45:48.244883+0000 osd.0 (osd.0) 514 : cluster 0 9.5 scrub starts 2024-03-11T21:45:50.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[21512]: cluster 2024-03-11T21:45:48.247647+0000 osd.0 (osd.0) 515 : cluster 0 9.5 scrub ok 2024-03-11T21:45:50.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[21512]: cluster 2024-03-11T21:45:48.341979+0000 osd.3 (osd.3) 521 : cluster 0 6.13 scrub starts 2024-03-11T21:45:50.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[21512]: cluster 2024-03-11T21:45:48.344177+0000 osd.3 (osd.3) 522 : cluster 0 6.13 scrub ok 2024-03-11T21:45:50.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[29284]: cluster 2024-03-11T21:45:48.244883+0000 osd.0 (osd.0) 514 : cluster 0 9.5 scrub starts 2024-03-11T21:45:50.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[29284]: cluster 2024-03-11T21:45:48.247647+0000 osd.0 (osd.0) 515 : cluster 0 9.5 scrub ok 2024-03-11T21:45:50.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[29284]: cluster 2024-03-11T21:45:48.341979+0000 osd.3 (osd.3) 521 : cluster 0 6.13 scrub starts 2024-03-11T21:45:50.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:49 smithi143 bash[29284]: cluster 2024-03-11T21:45:48.344177+0000 osd.3 (osd.3) 522 : cluster 0 6.13 scrub ok 2024-03-11T21:45:50.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:49 smithi203 bash[23654]: cluster 2024-03-11T21:45:48.244883+0000 osd.0 (osd.0) 514 : cluster 0 9.5 scrub starts 2024-03-11T21:45:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:49 smithi203 bash[23654]: cluster 2024-03-11T21:45:48.247647+0000 osd.0 (osd.0) 515 : cluster 0 9.5 scrub ok 2024-03-11T21:45:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:49 smithi203 bash[23654]: cluster 2024-03-11T21:45:48.341979+0000 osd.3 (osd.3) 521 : cluster 0 6.13 scrub starts 2024-03-11T21:45:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:49 smithi203 bash[23654]: cluster 2024-03-11T21:45:48.344177+0000 osd.3 (osd.3) 522 : cluster 0 6.13 scrub ok 2024-03-11T21:45:51.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.193329+0000 osd.0 (osd.0) 516 : cluster 0 2.1f scrub starts 2024-03-11T21:45:51.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.196539+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub ok 2024-03-11T21:45:51.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: audit 2024-03-11T21:45:49.225425+0000 mgr.y (mgr.14550) 879 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:51.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.441038+0000 osd.2 (osd.2) 461 : cluster 0 2.c scrub starts 2024-03-11T21:45:51.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.443304+0000 osd.2 (osd.2) 462 : cluster 0 2.c scrub ok 2024-03-11T21:45:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.725582+0000 mgr.y (mgr.14550) 880 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-11T21:45:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.909733+0000 osd.4 (osd.4) 497 : cluster 0 5.18 scrub starts 2024-03-11T21:45:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[21512]: cluster 2024-03-11T21:45:49.911868+0000 osd.4 (osd.4) 498 : cluster 0 5.18 scrub ok 2024-03-11T21:45:51.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.193329+0000 osd.0 (osd.0) 516 : cluster 0 2.1f scrub starts 2024-03-11T21:45:51.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.196539+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub ok 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: audit 2024-03-11T21:45:49.225425+0000 mgr.y (mgr.14550) 879 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.441038+0000 osd.2 (osd.2) 461 : cluster 0 2.c scrub starts 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.443304+0000 osd.2 (osd.2) 462 : cluster 0 2.c scrub ok 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.725582+0000 mgr.y (mgr.14550) 880 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.909733+0000 osd.4 (osd.4) 497 : cluster 0 5.18 scrub starts 2024-03-11T21:45:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:50 smithi143 bash[29284]: cluster 2024-03-11T21:45:49.911868+0000 osd.4 (osd.4) 498 : cluster 0 5.18 scrub ok 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.193329+0000 osd.0 (osd.0) 516 : cluster 0 2.1f scrub starts 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.196539+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub ok 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: audit 2024-03-11T21:45:49.225425+0000 mgr.y (mgr.14550) 879 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.441038+0000 osd.2 (osd.2) 461 : cluster 0 2.c scrub starts 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.443304+0000 osd.2 (osd.2) 462 : cluster 0 2.c scrub ok 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.725582+0000 mgr.y (mgr.14550) 880 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.909733+0000 osd.4 (osd.4) 497 : cluster 0 5.18 scrub starts 2024-03-11T21:45:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:50 smithi203 bash[23654]: cluster 2024-03-11T21:45:49.911868+0000 osd.4 (osd.4) 498 : cluster 0 5.18 scrub ok 2024-03-11T21:45:52.196 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:45:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:52.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.688440+0000 osd.6 (osd.6) 433 : cluster 0 1.f scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.691419+0000 osd.6 (osd.6) 434 : cluster 0 1.f scrub ok 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.833036+0000 osd.1 (osd.1) 325 : cluster 0 4.2 scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.836459+0000 osd.1 (osd.1) 326 : cluster 0 4.2 scrub ok 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.918664+0000 osd.4 (osd.4) 499 : cluster 0 6.1a scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:50.920996+0000 osd.4 (osd.4) 500 : cluster 0 6.1a scrub ok 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[29284]: cluster 2024-03-11T21:45:51.652604+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e971: 8 total, 8 up, 8 in 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.688440+0000 osd.6 (osd.6) 433 : cluster 0 1.f scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.691419+0000 osd.6 (osd.6) 434 : cluster 0 1.f scrub ok 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.833036+0000 osd.1 (osd.1) 325 : cluster 0 4.2 scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.836459+0000 osd.1 (osd.1) 326 : cluster 0 4.2 scrub ok 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.918664+0000 osd.4 (osd.4) 499 : cluster 0 6.1a scrub starts 2024-03-11T21:45:52.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:50.920996+0000 osd.4 (osd.4) 500 : cluster 0 6.1a scrub ok 2024-03-11T21:45:52.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:51 smithi143 bash[21512]: cluster 2024-03-11T21:45:51.652604+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e971: 8 total, 8 up, 8 in 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.688440+0000 osd.6 (osd.6) 433 : cluster 0 1.f scrub starts 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.691419+0000 osd.6 (osd.6) 434 : cluster 0 1.f scrub ok 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.833036+0000 osd.1 (osd.1) 325 : cluster 0 4.2 scrub starts 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.836459+0000 osd.1 (osd.1) 326 : cluster 0 4.2 scrub ok 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.918664+0000 osd.4 (osd.4) 499 : cluster 0 6.1a scrub starts 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:50.920996+0000 osd.4 (osd.4) 500 : cluster 0 6.1a scrub ok 2024-03-11T21:45:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:51 smithi203 bash[23654]: cluster 2024-03-11T21:45:51.652604+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e971: 8 total, 8 up, 8 in 2024-03-11T21:45:53.059 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:53.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[21512]: cluster 2024-03-11T21:45:51.174776+0000 osd.0 (osd.0) 518 : cluster 0 6.16 scrub starts 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[21512]: cluster 2024-03-11T21:45:51.177034+0000 osd.0 (osd.0) 519 : cluster 0 6.16 scrub ok 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[21512]: cluster 2024-03-11T21:45:51.726368+0000 mgr.y (mgr.14550) 881 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[21512]: audit 2024-03-11T21:45:52.812542+0000 mon.c (mon.2) 1391 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[29284]: cluster 2024-03-11T21:45:51.174776+0000 osd.0 (osd.0) 518 : cluster 0 6.16 scrub starts 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[29284]: cluster 2024-03-11T21:45:51.177034+0000 osd.0 (osd.0) 519 : cluster 0 6.16 scrub ok 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[29284]: cluster 2024-03-11T21:45:51.726368+0000 mgr.y (mgr.14550) 881 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:53.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:52 smithi143 bash[29284]: audit 2024-03-11T21:45:52.812542+0000 mon.c (mon.2) 1391 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:53.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:52 smithi203 bash[23654]: cluster 2024-03-11T21:45:51.174776+0000 osd.0 (osd.0) 518 : cluster 0 6.16 scrub starts 2024-03-11T21:45:53.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:52 smithi203 bash[23654]: cluster 2024-03-11T21:45:51.177034+0000 osd.0 (osd.0) 519 : cluster 0 6.16 scrub ok 2024-03-11T21:45:53.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:52 smithi203 bash[23654]: cluster 2024-03-11T21:45:51.726368+0000 mgr.y (mgr.14550) 881 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:45:53.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:52 smithi203 bash[23654]: audit 2024-03-11T21:45:52.812542+0000 mon.c (mon.2) 1391 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:54.078 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:54.079 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:54.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.305447+0000 osd.3 (osd.3) 523 : cluster 0 9.8 scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.311049+0000 osd.3 (osd.3) 524 : cluster 0 9.8 scrub ok 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.669124+0000 osd.6 (osd.6) 435 : cluster 0 2.1a scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.671664+0000 osd.6 (osd.6) 436 : cluster 0 2.1a scrub ok 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.959864+0000 osd.4 (osd.4) 501 : cluster 0 5.f scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[21512]: cluster 2024-03-11T21:45:52.962296+0000 osd.4 (osd.4) 502 : cluster 0 5.f scrub ok 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.305447+0000 osd.3 (osd.3) 523 : cluster 0 9.8 scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.311049+0000 osd.3 (osd.3) 524 : cluster 0 9.8 scrub ok 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.669124+0000 osd.6 (osd.6) 435 : cluster 0 2.1a scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.671664+0000 osd.6 (osd.6) 436 : cluster 0 2.1a scrub ok 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.959864+0000 osd.4 (osd.4) 501 : cluster 0 5.f scrub starts 2024-03-11T21:45:54.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:53 smithi143 bash[29284]: cluster 2024-03-11T21:45:52.962296+0000 osd.4 (osd.4) 502 : cluster 0 5.f scrub ok 2024-03-11T21:45:54.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.305447+0000 osd.3 (osd.3) 523 : cluster 0 9.8 scrub starts 2024-03-11T21:45:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.311049+0000 osd.3 (osd.3) 524 : cluster 0 9.8 scrub ok 2024-03-11T21:45:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.669124+0000 osd.6 (osd.6) 435 : cluster 0 2.1a scrub starts 2024-03-11T21:45:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.671664+0000 osd.6 (osd.6) 436 : cluster 0 2.1a scrub ok 2024-03-11T21:45:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.959864+0000 osd.4 (osd.4) 501 : cluster 0 5.f scrub starts 2024-03-11T21:45:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:53 smithi203 bash[23654]: cluster 2024-03-11T21:45:52.962296+0000 osd.4 (osd.4) 502 : cluster 0 5.f scrub ok 2024-03-11T21:45:55.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[29284]: cluster 2024-03-11T21:45:53.242870+0000 osd.0 (osd.0) 520 : cluster 0 3.12 scrub starts 2024-03-11T21:45:55.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[29284]: cluster 2024-03-11T21:45:53.246961+0000 osd.0 (osd.0) 521 : cluster 0 3.12 scrub ok 2024-03-11T21:45:55.161 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[29284]: cluster 2024-03-11T21:45:53.728278+0000 mgr.y (mgr.14550) 882 : cluster 0 pgmap v1535: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:45:55.161 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[21512]: cluster 2024-03-11T21:45:53.242870+0000 osd.0 (osd.0) 520 : cluster 0 3.12 scrub starts 2024-03-11T21:45:55.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[21512]: cluster 2024-03-11T21:45:53.246961+0000 osd.0 (osd.0) 521 : cluster 0 3.12 scrub ok 2024-03-11T21:45:55.162 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:54 smithi143 bash[21512]: cluster 2024-03-11T21:45:53.728278+0000 mgr.y (mgr.14550) 882 : cluster 0 pgmap v1535: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:45:55.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:54 smithi203 bash[23654]: cluster 2024-03-11T21:45:53.242870+0000 osd.0 (osd.0) 520 : cluster 0 3.12 scrub starts 2024-03-11T21:45:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:54 smithi203 bash[23654]: cluster 2024-03-11T21:45:53.246961+0000 osd.0 (osd.0) 521 : cluster 0 3.12 scrub ok 2024-03-11T21:45:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:54 smithi203 bash[23654]: cluster 2024-03-11T21:45:53.728278+0000 mgr.y (mgr.14550) 882 : cluster 0 pgmap v1535: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:45:56.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:55 smithi143 bash[29284]: cluster 2024-03-11T21:45:54.773139+0000 osd.1 (osd.1) 327 : cluster 0 9.6 scrub starts 2024-03-11T21:45:56.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:55 smithi143 bash[29284]: cluster 2024-03-11T21:45:54.775703+0000 osd.1 (osd.1) 328 : cluster 0 9.6 scrub ok 2024-03-11T21:45:56.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:55 smithi143 bash[21512]: cluster 2024-03-11T21:45:54.773139+0000 osd.1 (osd.1) 327 : cluster 0 9.6 scrub starts 2024-03-11T21:45:56.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:55 smithi143 bash[21512]: cluster 2024-03-11T21:45:54.775703+0000 osd.1 (osd.1) 328 : cluster 0 9.6 scrub ok 2024-03-11T21:45:56.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:55 smithi203 bash[23654]: cluster 2024-03-11T21:45:54.773139+0000 osd.1 (osd.1) 327 : cluster 0 9.6 scrub starts 2024-03-11T21:45:56.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:55 smithi203 bash[23654]: cluster 2024-03-11T21:45:54.775703+0000 osd.1 (osd.1) 328 : cluster 0 9.6 scrub ok 2024-03-11T21:45:56.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:56 smithi143 bash[21512]: cluster 2024-03-11T21:45:55.682442+0000 osd.6 (osd.6) 437 : cluster 0 1.7 scrub starts 2024-03-11T21:45:56.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:56 smithi143 bash[21512]: cluster 2024-03-11T21:45:55.684851+0000 osd.6 (osd.6) 438 : cluster 0 1.7 scrub ok 2024-03-11T21:45:56.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:56 smithi143 bash[21512]: cluster 2024-03-11T21:45:55.729266+0000 mgr.y (mgr.14550) 883 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:56.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:56 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.265053+0000 osd.7 (osd.7) 510 : cluster 0 2.5 scrub starts 2024-03-11T21:45:56.986 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:56 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.267386+0000 osd.7 (osd.7) 511 : cluster 0 2.5 scrub ok 2024-03-11T21:45:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:56 smithi203 bash[23654]: cluster 2024-03-11T21:45:55.682442+0000 osd.6 (osd.6) 437 : cluster 0 1.7 scrub starts 2024-03-11T21:45:57.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:56 smithi203 bash[23654]: cluster 2024-03-11T21:45:55.684851+0000 osd.6 (osd.6) 438 : cluster 0 1.7 scrub ok 2024-03-11T21:45:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:56 smithi203 bash[23654]: cluster 2024-03-11T21:45:55.729266+0000 mgr.y (mgr.14550) 883 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:56 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.265053+0000 osd.7 (osd.7) 510 : cluster 0 2.5 scrub starts 2024-03-11T21:45:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:56 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.267386+0000 osd.7 (osd.7) 511 : cluster 0 2.5 scrub ok 2024-03-11T21:45:57.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:55.682442+0000 osd.6 (osd.6) 437 : cluster 0 1.7 scrub starts 2024-03-11T21:45:57.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:55.684851+0000 osd.6 (osd.6) 438 : cluster 0 1.7 scrub ok 2024-03-11T21:45:57.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:55.729266+0000 mgr.y (mgr.14550) 883 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:57.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.265053+0000 osd.7 (osd.7) 510 : cluster 0 2.5 scrub starts 2024-03-11T21:45:57.336 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.267386+0000 osd.7 (osd.7) 511 : cluster 0 2.5 scrub ok 2024-03-11T21:45:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.294228+0000 osd.3 (osd.3) 525 : cluster 0 3.18 scrub starts 2024-03-11T21:45:58.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.297238+0000 osd.3 (osd.3) 526 : cluster 0 3.18 scrub ok 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.679059+0000 osd.6 (osd.6) 439 : cluster 0 1.6 scrub starts 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.681778+0000 osd.6 (osd.6) 440 : cluster 0 1.6 scrub ok 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.943770+0000 osd.4 (osd.4) 503 : cluster 0 6.e scrub starts 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[29284]: cluster 2024-03-11T21:45:56.946053+0000 osd.4 (osd.4) 504 : cluster 0 6.e scrub ok 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.294228+0000 osd.3 (osd.3) 525 : cluster 0 3.18 scrub starts 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.297238+0000 osd.3 (osd.3) 526 : cluster 0 3.18 scrub ok 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.679059+0000 osd.6 (osd.6) 439 : cluster 0 1.6 scrub starts 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.681778+0000 osd.6 (osd.6) 440 : cluster 0 1.6 scrub ok 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.943770+0000 osd.4 (osd.4) 503 : cluster 0 6.e scrub starts 2024-03-11T21:45:58.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:57 smithi143 bash[21512]: cluster 2024-03-11T21:45:56.946053+0000 osd.4 (osd.4) 504 : cluster 0 6.e scrub ok 2024-03-11T21:45:58.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.294228+0000 osd.3 (osd.3) 525 : cluster 0 3.18 scrub starts 2024-03-11T21:45:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.297238+0000 osd.3 (osd.3) 526 : cluster 0 3.18 scrub ok 2024-03-11T21:45:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.679059+0000 osd.6 (osd.6) 439 : cluster 0 1.6 scrub starts 2024-03-11T21:45:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.681778+0000 osd.6 (osd.6) 440 : cluster 0 1.6 scrub ok 2024-03-11T21:45:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.943770+0000 osd.4 (osd.4) 503 : cluster 0 6.e scrub starts 2024-03-11T21:45:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:57 smithi203 bash[23654]: cluster 2024-03-11T21:45:56.946053+0000 osd.4 (osd.4) 504 : cluster 0 6.e scrub ok 2024-03-11T21:45:59.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.521296+0000 osd.3 (osd.3) 527 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:59.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.521756+0000 osd.3 (osd.3) 528 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:59.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.635462+0000 osd.6 (osd.6) 441 : cluster 0 2.11 scrub starts 2024-03-11T21:45:59.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.637835+0000 osd.6 (osd.6) 442 : cluster 0 2.11 scrub ok 2024-03-11T21:45:59.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.729985+0000 mgr.y (mgr.14550) 884 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:58 smithi203 bash[23654]: cluster 2024-03-11T21:45:57.951445+0000 mon.a (mon.0) 4081 : cluster 0 osdmap e972: 8 total, 8 up, 8 in 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.521296+0000 osd.3 (osd.3) 527 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.521756+0000 osd.3 (osd.3) 528 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.635462+0000 osd.6 (osd.6) 441 : cluster 0 2.11 scrub starts 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.637835+0000 osd.6 (osd.6) 442 : cluster 0 2.11 scrub ok 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.729985+0000 mgr.y (mgr.14550) 884 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:45:59.215 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[21512]: cluster 2024-03-11T21:45:57.951445+0000 mon.a (mon.0) 4081 : cluster 0 osdmap e972: 8 total, 8 up, 8 in 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.521296+0000 osd.3 (osd.3) 527 : cluster 0 purged_snaps scrub starts 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.521756+0000 osd.3 (osd.3) 528 : cluster 0 purged_snaps scrub ok 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.635462+0000 osd.6 (osd.6) 441 : cluster 0 2.11 scrub starts 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.637835+0000 osd.6 (osd.6) 442 : cluster 0 2.11 scrub ok 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.729985+0000 mgr.y (mgr.14550) 884 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:45:59.216 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:58 smithi143 bash[29284]: cluster 2024-03-11T21:45:57.951445+0000 mon.a (mon.0) 4081 : cluster 0 osdmap e972: 8 total, 8 up, 8 in 2024-03-11T21:45:59.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:00.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.343803+0000 osd.3 (osd.3) 529 : cluster 0 9.19 scrub starts 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.347372+0000 osd.3 (osd.3) 530 : cluster 0 9.19 scrub ok 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.682634+0000 osd.6 (osd.6) 443 : cluster 0 6.a scrub starts 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.684985+0000 osd.6 (osd.6) 444 : cluster 0 6.a scrub ok 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.747529+0000 osd.1 (osd.1) 329 : cluster 0 2.9 scrub starts 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: cluster 2024-03-11T21:45:58.749953+0000 osd.1 (osd.1) 330 : cluster 0 2.9 scrub ok 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: audit 2024-03-11T21:45:59.314171+0000 mon.c (mon.2) 1392 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: audit 2024-03-11T21:45:59.840172+0000 mon.a (mon.0) 4082 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[29284]: audit 2024-03-11T21:45:59.847458+0000 mon.a (mon.0) 4083 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.343803+0000 osd.3 (osd.3) 529 : cluster 0 9.19 scrub starts 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.347372+0000 osd.3 (osd.3) 530 : cluster 0 9.19 scrub ok 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.682634+0000 osd.6 (osd.6) 443 : cluster 0 6.a scrub starts 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.684985+0000 osd.6 (osd.6) 444 : cluster 0 6.a scrub ok 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.747529+0000 osd.1 (osd.1) 329 : cluster 0 2.9 scrub starts 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: cluster 2024-03-11T21:45:58.749953+0000 osd.1 (osd.1) 330 : cluster 0 2.9 scrub ok 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: audit 2024-03-11T21:45:59.314171+0000 mon.c (mon.2) 1392 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: audit 2024-03-11T21:45:59.840172+0000 mon.a (mon.0) 4082 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:45:59 smithi143 bash[21512]: audit 2024-03-11T21:45:59.847458+0000 mon.a (mon.0) 4083 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.343803+0000 osd.3 (osd.3) 529 : cluster 0 9.19 scrub starts 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.347372+0000 osd.3 (osd.3) 530 : cluster 0 9.19 scrub ok 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.682634+0000 osd.6 (osd.6) 443 : cluster 0 6.a scrub starts 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.684985+0000 osd.6 (osd.6) 444 : cluster 0 6.a scrub ok 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.747529+0000 osd.1 (osd.1) 329 : cluster 0 2.9 scrub starts 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: cluster 2024-03-11T21:45:58.749953+0000 osd.1 (osd.1) 330 : cluster 0 2.9 scrub ok 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: audit 2024-03-11T21:45:59.314171+0000 mon.c (mon.2) 1392 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: audit 2024-03-11T21:45:59.840172+0000 mon.a (mon.0) 4082 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:45:59 smithi203 bash[23654]: audit 2024-03-11T21:45:59.847458+0000 mon.a (mon.0) 4083 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:00.533 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:00.533 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:46:00.533 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:46:00.533 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:46:00.533 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: audit 2024-03-11T21:45:59.229059+0000 mgr.y (mgr.14550) 885 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.246671+0000 osd.0 (osd.0) 522 : cluster 0 9.1e scrub starts 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.250588+0000 osd.0 (osd.0) 523 : cluster 0 9.1e scrub ok 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.413529+0000 osd.2 (osd.2) 463 : cluster 0 5.b scrub starts 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.415196+0000 osd.2 (osd.2) 464 : cluster 0 5.b scrub ok 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.730913+0000 mgr.y (mgr.14550) 886 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.949071+0000 osd.4 (osd.4) 505 : cluster 0 5.1 scrub starts 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: cluster 2024-03-11T21:45:59.951538+0000 osd.4 (osd.4) 506 : cluster 0 5.1 scrub ok 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: audit 2024-03-11T21:46:00.526965+0000 mon.c (mon.2) 1393 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:01.221 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: audit 2024-03-11T21:46:00.527821+0000 mon.c (mon.2) 1394 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[29284]: audit 2024-03-11T21:46:00.534518+0000 mon.a (mon.0) 4084 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: audit 2024-03-11T21:45:59.229059+0000 mgr.y (mgr.14550) 885 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.246671+0000 osd.0 (osd.0) 522 : cluster 0 9.1e scrub starts 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.250588+0000 osd.0 (osd.0) 523 : cluster 0 9.1e scrub ok 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.413529+0000 osd.2 (osd.2) 463 : cluster 0 5.b scrub starts 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.415196+0000 osd.2 (osd.2) 464 : cluster 0 5.b scrub ok 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.730913+0000 mgr.y (mgr.14550) 886 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.949071+0000 osd.4 (osd.4) 505 : cluster 0 5.1 scrub starts 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: cluster 2024-03-11T21:45:59.951538+0000 osd.4 (osd.4) 506 : cluster 0 5.1 scrub ok 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: audit 2024-03-11T21:46:00.526965+0000 mon.c (mon.2) 1393 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:01.222 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: audit 2024-03-11T21:46:00.527821+0000 mon.c (mon.2) 1394 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:01.223 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:00 smithi143 bash[21512]: audit 2024-03-11T21:46:00.534518+0000 mon.a (mon.0) 4084 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: audit 2024-03-11T21:45:59.229059+0000 mgr.y (mgr.14550) 885 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.246671+0000 osd.0 (osd.0) 522 : cluster 0 9.1e scrub starts 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.250588+0000 osd.0 (osd.0) 523 : cluster 0 9.1e scrub ok 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.413529+0000 osd.2 (osd.2) 463 : cluster 0 5.b scrub starts 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.415196+0000 osd.2 (osd.2) 464 : cluster 0 5.b scrub ok 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.730913+0000 mgr.y (mgr.14550) 886 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.949071+0000 osd.4 (osd.4) 505 : cluster 0 5.1 scrub starts 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: cluster 2024-03-11T21:45:59.951538+0000 osd.4 (osd.4) 506 : cluster 0 5.1 scrub ok 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: audit 2024-03-11T21:46:00.526965+0000 mon.c (mon.2) 1393 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: audit 2024-03-11T21:46:00.527821+0000 mon.c (mon.2) 1394 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:00 smithi203 bash[23654]: audit 2024-03-11T21:46:00.534518+0000 mon.a (mon.0) 4084 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:01.961 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 -- 172.21.15.143:0/1355421542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a4e0 msgr2=0x7f9df406a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 --2- 172.21.15.143:0/1355421542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a4e0 0x7f9df406a940 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009990 tx=0x7f9ddc009ca0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 -- 172.21.15.143:0/1355421542 shutdown_connections 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 --2- 172.21.15.143:0/1355421542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406ae80 0x7f9df410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 --2- 172.21.15.143:0/1355421542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 0x7f9df4108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 --2- 172.21.15.143:0/1355421542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406a4e0 0x7f9df406a940 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 -- 172.21.15.143:0/1355421542 >> 172.21.15.143:0/1355421542 conn(0x7f9df4077c80 msgr2=0x7f9df4078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:02.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 -- 172.21.15.143:0/1355421542 shutdown_connections 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.333+0000 7f9df9506700 1 -- 172.21.15.143:0/1355421542 wait complete. 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 librados: starting msgr at 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 librados: starting objecter 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 Processor -- start 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- start start 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 librados: setting wanted keys 2024-03-11T21:46:02.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 librados: calling monclient init 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406ae80 0x7f9df419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 0x7f9df419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4111610 con 0x7f9df406a4e0 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4111490 con 0x7f9df4107e90 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4111790 con 0x7f9df406ae80 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57824/0 (socket says 172.21.15.143:57824) 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 -- 172.21.15.143:0/3476601939 learned_addr learned my addr 172.21.15.143:0/3476601939 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df27fc700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406ae80 0x7f9df419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df37fe700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 0x7f9df419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 -- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406ae80 msgr2=0x7f9df419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406ae80 0x7f9df419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 -- 172.21.15.143:0/3476601939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 msgr2=0x7f9df419e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 0x7f9df419e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 -- 172.21.15.143:0/3476601939 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df419ee70 con 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df2ffd700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ba50 tx=0x7f9de400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9debfff700 10 client.?.objecter ms_handle_connect 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9debfff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4019070 con 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df419f160 con 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f9de400ce30 con 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4007370 con 0x7f9df406a4e0 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 10 client.31101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:02.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df41a69e0 con 0x7f9df406a4e0 2024-03-11T21:46:02.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 librados: init done 2024-03-11T21:46:02.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.337+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df404faa0 con 0x7f9df406a4e0 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9de401c070 con 0x7f9df406a4e0 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(973..973 src has 842..973) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f9de408de10 con 0x7f9df406a4e0 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 10 client.31101.objecter ms_dispatch 0x7f9df4001550 osd_map(973..973 src has 842..973) v4 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 3 client.31101.objecter handle_osd_map got epochs [973,973] > 0 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 3 client.31101.objecter handle_osd_map decoding full epoch 973 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9df27fc700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 20 client.31101.objecter dump_active .. 0 homeless 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9debfff700 10 client.31101.objecter ms_handle_connect 0x7f9de006f820 2024-03-11T21:46:02.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.341+0000 7f9df27fc700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00b2e0 tx=0x7f9ddc005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.330 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:02.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.345+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9de405ada0 con 0x7f9df406a4e0 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- 172.21.15.143:0/1344508464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700106dc0 msgr2=0x7f87001071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- 172.21.15.143:0/1344508464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700106dc0 0x7f87001071a0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009990 tx=0x7f86f0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- 172.21.15.143:0/1344508464 shutdown_connections 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- 172.21.15.143:0/1344508464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700106dc0 0x7f87001071a0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- 172.21.15.143:0/1344508464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8700101700 0x7f870010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- 172.21.15.143:0/1344508464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8700100d40 0x7f87001011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- 172.21.15.143:0/1344508464 >> 172.21.15.143:0/1344508464 conn(0x7f8700076d80 msgr2=0x7f8700077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- 172.21.15.143:0/1344508464 shutdown_connections 2024-03-11T21:46:02.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- 172.21.15.143:0/1344508464 wait complete. 2024-03-11T21:46:02.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 librados: starting msgr at 2024-03-11T21:46:02.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 librados: starting objecter 2024-03-11T21:46:02.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 Processor -- start 2024-03-11T21:46:02.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- start start 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 librados: setting wanted keys 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 librados: calling monclient init 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 0x7f870019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8700101700 0x7f870019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8700106dc0 0x7f8700197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8700110520 con 0x7f8700100d40 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87001103a0 con 0x7f8700101700 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f8704b41700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87001106a0 con 0x7f8700106dc0 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 0x7f870019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 0x7f870019d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57844/0 (socket says 172.21.15.143:57844) 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 -- 172.21.15.143:0/2817069150 learned_addr learned my addr 172.21.15.143:0/2817069150 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8700106dc0 0x7f8700197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fdd9b700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8700101700 0x7f870019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8700106dc0 msgr2=0x7f8700197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8700106dc0 0x7f8700197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8700101700 msgr2=0x7f870019dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8700101700 0x7f870019dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.473+0000 7f86fe59c700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87001980a0 con 0x7f8700100d40 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fe59c700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 0x7f870019d780 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f86f00057c0 tx=0x7f86f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.?.objecter ms_handle_connect 0x7f8700100d40 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:02.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0027070 con 0x7f8700100d40 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8700198330 con 0x7f8700100d40 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 10 client.31107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f86f0022470 con 0x7f8700100d40 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0017650 con 0x7f8700100d40 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87001028f0 con 0x7f8700100d40 2024-03-11T21:46:02.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 librados: init done 2024-03-11T21:46:02.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f870004faa0 con 0x7f8700100d40 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f86f002a070 con 0x7f8700100d40 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f86e806f820 0x7f86e8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(973..973 src has 842..973) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f86f008f7a0 con 0x7f8700100d40 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.31107.objecter ms_dispatch 0x7f8700001550 osd_map(973..973 src has 842..973) v4 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 3 client.31107.objecter handle_osd_map got epochs [973,973] > 0 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 3 client.31107.objecter handle_osd_map decoding full epoch 973 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fdd9b700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f86e806f820 0x7f86e8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 20 client.31107.objecter dump_active .. 0 homeless 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f86f0013070 con 0x7f8700100d40 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 0x7f8700109e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8700198e70 con 0x7f8700109a70 2024-03-11T21:46:02.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f86e806f820 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fdd9b700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f86e806f820 0x7f86e8071ce0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f86f4005ea0 tx=0x7f86f4005e30 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 0x7f8700109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f870010a390 con 0x7f8700109a70 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 0x7f8700109e50 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f800f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f8700109a70 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.477+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8019020 con 0x7f8700109a70 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f86f8009110 con 0x7f8700109a70 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 msgr2=0x7f8700109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 0x7f8700109e50 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f800f7a0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 msgr2=0x7f8700109e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.481+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700109a70 0x7f8700109e50 secure :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f800f7a0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.485+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 0x7f870002c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.485+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870010a980 con 0x7f870010a5a0 2024-03-11T21:46:02.475 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 0x7f870002c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f87001a8670 con 0x7f870010a5a0 2024-03-11T21:46:02.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 0x7f870002c6b0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f870010a5a0 2024-03-11T21:46:02.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8019030 con 0x7f870010a5a0 2024-03-11T21:46:02.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.0 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f86f8004db0 con 0x7f870010a5a0 2024-03-11T21:46:02.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 msgr2=0x7f870002c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 0x7f870002c6b0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f8004b40 comp rx=0 tx=0).stop 2024-03-11T21:46:02.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 msgr2=0x7f870002c6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.478 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f870010a5a0 0x7f870002c6b0 secure :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f3e0 tx=0x7f86f8004b40 comp rx=0 tx=0).stop 2024-03-11T21:46:02.478 INFO:teuthology.orchestra.run.smithi143.stdout:mon.a: {} 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 0x7f870002d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8700104340 con 0x7f870002cbf0 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 0x7f870002d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8700104540 con 0x7f870002cbf0 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 0x7f870002d000 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f7e0 tx=0x7f86f8004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f870002cbf0 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.489+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8036020 con 0x7f870002cbf0 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f86f8007700 con 0x7f870002cbf0 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 msgr2=0x7f870002d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 0x7f870002d000 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f7e0 tx=0x7f86f8004ad0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 msgr2=0x7f870002d000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870002cbf0 0x7f870002d000 secure :-1 s=CLOSED pgs=4229 cs=0 l=1 rev1=1 crypto rx=0x7f86f800f7e0 tx=0x7f86f8004ad0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.480 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.493+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 reap_dead start 2024-03-11T21:46:02.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 0x7f8700197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8700104750 con 0x7f870019dcc0 2024-03-11T21:46:02.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 0x7f8700197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f87001048d0 con 0x7f870019dcc0 2024-03-11T21:46:02.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 0x7f8700197940 secure :-1 s=READY pgs=4230 cs=0 l=1 rev1=1 crypto rx=0x7f86f800b010 tx=0x7f86f802a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f870019dcc0 2024-03-11T21:46:02.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.501+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8036040 con 0x7f870019dcc0 2024-03-11T21:46:02.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f86f803d020 con 0x7f870019dcc0 2024-03-11T21:46:02.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 msgr2=0x7f8700197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 0x7f8700197940 secure :-1 s=READY pgs=4230 cs=0 l=1 rev1=1 crypto rx=0x7f86f800b010 tx=0x7f86f802a000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.492 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 msgr2=0x7f8700197940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.493 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f870019dcc0 0x7f8700197940 secure :-1 s=CLOSED pgs=4230 cs=0 l=1 rev1=1 crypto rx=0x7f86f800b010 tx=0x7f86f802a000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.493 INFO:teuthology.orchestra.run.smithi143.stdout:mon.b: {} 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 0x7f8700102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870002d920 con 0x7f870002d540 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 0x7f8700102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f870002dad0 con 0x7f870002d540 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f870002d540 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 0x7f8700102ba0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f86f8011a60 tx=0x7f86f8026000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8036050 con 0x7f870002d540 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.2 v2:172.21.15.143:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f86f8004030 con 0x7f870002d540 2024-03-11T21:46:02.494 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 msgr2=0x7f8700102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 0x7f8700102ba0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f86f8011a60 tx=0x7f86f8026000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 msgr2=0x7f8700102ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.495 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.505+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f870002d540 0x7f8700102ba0 secure :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f86f8011a60 tx=0x7f86f8026000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 0x7f87001034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:02.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8700103a00 con 0x7f87001030e0 2024-03-11T21:46:02.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 0x7f87001034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:02.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f8700103bb0 con 0x7f87001030e0 2024-03-11T21:46:02.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 10 client.31107.objecter ms_handle_connect 0x7f87001030e0 2024-03-11T21:46:02.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 10 client.31107.objecter resend_mon_ops 2024-03-11T21:46:02.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86fed9d700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 0x7f87001034c0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f870002daa0 tx=0x7f86f8045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:02.502 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8019070 con 0x7f87001030e0 2024-03-11T21:46:02.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 <== mon.2 v2:172.21.15.143:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f86f8055780 con 0x7f87001030e0 2024-03-11T21:46:02.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 msgr2=0x7f87001034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 0x7f87001034c0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f870002daa0 tx=0x7f86f8045000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 msgr2=0x7f87001034c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-11T21:46:02.503 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.513+0000 7f86ef7fe700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87001030e0 0x7f87001034c0 secure :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f870002daa0 tx=0x7f86f8045000 comp rx=0 tx=0).stop 2024-03-11T21:46:02.504 INFO:teuthology.orchestra.run.smithi143.stdout:mon.c: {} 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 10 librados: watch_flush enter 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 10 librados: watch_flush exit 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 20 client.31107.objecter shutdown clearing up homeless session... 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 10 client.31107.objecter successfully canceled tick 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f86e806f820 msgr2=0x7f86e8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f86e806f820 0x7f86e8071ce0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f86f4005ea0 tx=0x7f86f4005e30 comp rx=0 tx=0).stop 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 msgr2=0x7f870019d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 --2- 172.21.15.143:0/2817069150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8700100d40 0x7f870019d780 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f86f00057c0 tx=0x7f86f00057f0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f86fed9d700 1 -- 172.21.15.143:0/2817069150 reap_dead start 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 shutdown_connections 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 >> 172.21.15.143:0/2817069150 conn(0x7f8700076d80 msgr2=0x7f8700069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 shutdown_connections 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 -- 172.21.15.143:0/2817069150 wait complete. 2024-03-11T21:46:02.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.517+0000 7f8704b41700 1 librados: shutdown 2024-03-11T21:46:02.506 INFO:teuthology.orchestra.run.smithi143.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:46:02.506 INFO:teuthology.orchestra.run.smithi143.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:46:02.506 INFO:teuthology.orchestra.run.smithi143.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:46:02.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.581+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9df41a6c10 con 0x7f9df406a4e0 2024-03-11T21:46:02.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.661+0000 7f9debfff700 1 -- 172.21.15.143:0/3476601939 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v974) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9de405e7f0 con 0x7f9df406a4e0 2024-03-11T21:46:02.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 10 librados: watch_flush enter 2024-03-11T21:46:02.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 10 librados: watch_flush exit 2024-03-11T21:46:02.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 20 client.31101.objecter shutdown clearing up homeless session... 2024-03-11T21:46:02.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 10 client.31101.objecter successfully canceled tick 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 msgr2=0x7f9de0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00b2e0 tx=0x7f9ddc005960 comp rx=0 tx=0).stop 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 msgr2=0x7f9df4199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ba50 tx=0x7f9de400bd60 comp rx=0 tx=0).stop 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 shutdown_connections 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f9de006f820 0x7f9de0071ce0 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df406a4e0 0x7f9df4199e60 secure :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ba50 tx=0x7f9de400bd60 comp rx=0 tx=0).stop 2024-03-11T21:46:02.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9df4107e90 0x7f9df419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 --2- 172.21.15.143:0/3476601939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9df406ae80 0x7f9df419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:02.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 >> 172.21.15.143:0/3476601939 conn(0x7f9df4077c80 msgr2=0x7f9df410d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:02.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.665+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 shutdown_connections 2024-03-11T21:46:02.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.669+0000 7f9df9506700 1 -- 172.21.15.143:0/3476601939 wait complete. 2024-03-11T21:46:02.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:02.669+0000 7f9df9506700 1 librados: shutdown 2024-03-11T21:46:02.657 INFO:teuthology.orchestra.run.smithi143.stderr:nodeep-scrub is unset 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:00.945417+0000 osd.4 (osd.4) 507 : cluster 0 3.1d scrub starts 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:00.949385+0000 osd.4 (osd.4) 508 : cluster 0 3.1d scrub ok 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:01.659184+0000 mon.a (mon.0) 4085 : cluster 0 osdmap e973: 8 total, 8 up, 8 in 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:01.731647+0000 mgr.y (mgr.14550) 887 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:01.734645+0000 osd.6 (osd.6) 445 : cluster 0 5.11 scrub starts 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: cluster 2024-03-11T21:46:01.739578+0000 osd.6 (osd.6) 446 : cluster 0 5.11 scrub ok 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: audit 2024-03-11T21:46:02.482364+0000 mon.a (mon.0) 4086 : audit 0 from='client.? 172.21.15.143:0/2817069150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:46:02.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[21512]: audit 2024-03-11T21:46:02.585320+0000 mon.a (mon.0) 4087 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:00.945417+0000 osd.4 (osd.4) 507 : cluster 0 3.1d scrub starts 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:00.949385+0000 osd.4 (osd.4) 508 : cluster 0 3.1d scrub ok 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:01.659184+0000 mon.a (mon.0) 4085 : cluster 0 osdmap e973: 8 total, 8 up, 8 in 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:01.731647+0000 mgr.y (mgr.14550) 887 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:01.734645+0000 osd.6 (osd.6) 445 : cluster 0 5.11 scrub starts 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: cluster 2024-03-11T21:46:01.739578+0000 osd.6 (osd.6) 446 : cluster 0 5.11 scrub ok 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: audit 2024-03-11T21:46:02.482364+0000 mon.a (mon.0) 4086 : audit 0 from='client.? 172.21.15.143:0/2817069150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:46:02.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:02 smithi143 bash[29284]: audit 2024-03-11T21:46:02.585320+0000 mon.a (mon.0) 4087 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:00.945417+0000 osd.4 (osd.4) 507 : cluster 0 3.1d scrub starts 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:00.949385+0000 osd.4 (osd.4) 508 : cluster 0 3.1d scrub ok 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:01.659184+0000 mon.a (mon.0) 4085 : cluster 0 osdmap e973: 8 total, 8 up, 8 in 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:01.731647+0000 mgr.y (mgr.14550) 887 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:01.734645+0000 osd.6 (osd.6) 445 : cluster 0 5.11 scrub starts 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: cluster 2024-03-11T21:46:01.739578+0000 osd.6 (osd.6) 446 : cluster 0 5.11 scrub ok 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: audit 2024-03-11T21:46:02.482364+0000 mon.a (mon.0) 4086 : audit 0 from='client.? 172.21.15.143:0/2817069150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:46:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:02 smithi203 bash[23654]: audit 2024-03-11T21:46:02.585320+0000 mon.a (mon.0) 4087 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:03.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.264709+0000 osd.3 (osd.3) 531 : cluster 0 6.1b scrub starts 2024-03-11T21:46:03.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.266305+0000 osd.3 (osd.3) 532 : cluster 0 6.1b scrub ok 2024-03-11T21:46:03.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.663941+0000 mon.a (mon.0) 4088 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:03.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: audit 2024-03-11T21:46:02.665765+0000 mon.a (mon.0) 4089 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.670250+0000 mon.a (mon.0) 4090 : cluster 0 osdmap e974: 8 total, 8 up, 8 in 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.836097+0000 osd.1 (osd.1) 331 : cluster 0 9.0 scrub starts 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[29284]: cluster 2024-03-11T21:46:02.838197+0000 osd.1 (osd.1) 332 : cluster 0 9.0 scrub ok 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.264709+0000 osd.3 (osd.3) 531 : cluster 0 6.1b scrub starts 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.266305+0000 osd.3 (osd.3) 532 : cluster 0 6.1b scrub ok 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.663941+0000 mon.a (mon.0) 4088 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: audit 2024-03-11T21:46:02.665765+0000 mon.a (mon.0) 4089 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.670250+0000 mon.a (mon.0) 4090 : cluster 0 osdmap e974: 8 total, 8 up, 8 in 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.836097+0000 osd.1 (osd.1) 331 : cluster 0 9.0 scrub starts 2024-03-11T21:46:03.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:03 smithi143 bash[21512]: cluster 2024-03-11T21:46:02.838197+0000 osd.1 (osd.1) 332 : cluster 0 9.0 scrub ok 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.264709+0000 osd.3 (osd.3) 531 : cluster 0 6.1b scrub starts 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.266305+0000 osd.3 (osd.3) 532 : cluster 0 6.1b scrub ok 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.663941+0000 mon.a (mon.0) 4088 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: audit 2024-03-11T21:46:02.665765+0000 mon.a (mon.0) 4089 : audit 1 from='client.? 172.21.15.143:0/3476601939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.670250+0000 mon.a (mon.0) 4090 : cluster 0 osdmap e974: 8 total, 8 up, 8 in 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.836097+0000 osd.1 (osd.1) 331 : cluster 0 9.0 scrub starts 2024-03-11T21:46:04.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:03 smithi203 bash[23654]: cluster 2024-03-11T21:46:02.838197+0000 osd.1 (osd.1) 332 : cluster 0 9.0 scrub ok 2024-03-11T21:46:04.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:46:04.946 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.275839+0000 osd.3 (osd.3) 533 : cluster 0 2.1b scrub starts 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.278100+0000 osd.3 (osd.3) 534 : cluster 0 2.1b scrub ok 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.668697+0000 osd.6 (osd.6) 447 : cluster 0 3.10 scrub starts 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.671944+0000 osd.6 (osd.6) 448 : cluster 0 3.10 scrub ok 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.733472+0000 mgr.y (mgr.14550) 888 : cluster 0 pgmap v1543: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.275839+0000 osd.3 (osd.3) 533 : cluster 0 2.1b scrub starts 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.278100+0000 osd.3 (osd.3) 534 : cluster 0 2.1b scrub ok 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.668697+0000 osd.6 (osd.6) 447 : cluster 0 3.10 scrub starts 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.671944+0000 osd.6 (osd.6) 448 : cluster 0 3.10 scrub ok 2024-03-11T21:46:04.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:04 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.733472+0000 mgr.y (mgr.14550) 888 : cluster 0 pgmap v1543: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:05.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:04 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.275839+0000 osd.3 (osd.3) 533 : cluster 0 2.1b scrub starts 2024-03-11T21:46:05.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:04 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.278100+0000 osd.3 (osd.3) 534 : cluster 0 2.1b scrub ok 2024-03-11T21:46:05.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:04 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.668697+0000 osd.6 (osd.6) 447 : cluster 0 3.10 scrub starts 2024-03-11T21:46:05.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:04 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.671944+0000 osd.6 (osd.6) 448 : cluster 0 3.10 scrub ok 2024-03-11T21:46:05.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:04 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.733472+0000 mgr.y (mgr.14550) 888 : cluster 0 pgmap v1543: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.906446+0000 osd.4 (osd.4) 509 : cluster 0 1.e scrub starts 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[29284]: cluster 2024-03-11T21:46:03.908774+0000 osd.4 (osd.4) 510 : cluster 0 1.e scrub ok 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[29284]: cluster 2024-03-11T21:46:04.317122+0000 osd.3 (osd.3) 535 : cluster 0 3.b scrub starts 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[29284]: cluster 2024-03-11T21:46:04.320007+0000 osd.3 (osd.3) 536 : cluster 0 3.b scrub ok 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.906446+0000 osd.4 (osd.4) 509 : cluster 0 1.e scrub starts 2024-03-11T21:46:05.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[21512]: cluster 2024-03-11T21:46:03.908774+0000 osd.4 (osd.4) 510 : cluster 0 1.e scrub ok 2024-03-11T21:46:05.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[21512]: cluster 2024-03-11T21:46:04.317122+0000 osd.3 (osd.3) 535 : cluster 0 3.b scrub starts 2024-03-11T21:46:05.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:05 smithi143 bash[21512]: cluster 2024-03-11T21:46:04.320007+0000 osd.3 (osd.3) 536 : cluster 0 3.b scrub ok 2024-03-11T21:46:06.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:05 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.906446+0000 osd.4 (osd.4) 509 : cluster 0 1.e scrub starts 2024-03-11T21:46:06.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:05 smithi203 bash[23654]: cluster 2024-03-11T21:46:03.908774+0000 osd.4 (osd.4) 510 : cluster 0 1.e scrub ok 2024-03-11T21:46:06.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:05 smithi203 bash[23654]: cluster 2024-03-11T21:46:04.317122+0000 osd.3 (osd.3) 535 : cluster 0 3.b scrub starts 2024-03-11T21:46:06.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:05 smithi203 bash[23654]: cluster 2024-03-11T21:46:04.320007+0000 osd.3 (osd.3) 536 : cluster 0 3.b scrub ok 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[21512]: cluster 2024-03-11T21:46:05.404716+0000 osd.2 (osd.2) 465 : cluster 0 5.d scrub starts 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[21512]: cluster 2024-03-11T21:46:05.408155+0000 osd.2 (osd.2) 466 : cluster 0 5.d scrub ok 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[21512]: cluster 2024-03-11T21:46:05.734497+0000 mgr.y (mgr.14550) 889 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.659317+0000 mon.a (mon.0) 4091 : cluster 0 osdmap e975: 8 total, 8 up, 8 in 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[29284]: cluster 2024-03-11T21:46:05.404716+0000 osd.2 (osd.2) 465 : cluster 0 5.d scrub starts 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[29284]: cluster 2024-03-11T21:46:05.408155+0000 osd.2 (osd.2) 466 : cluster 0 5.d scrub ok 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[29284]: cluster 2024-03-11T21:46:05.734497+0000 mgr.y (mgr.14550) 889 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:46:06.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:06 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.659317+0000 mon.a (mon.0) 4091 : cluster 0 osdmap e975: 8 total, 8 up, 8 in 2024-03-11T21:46:07.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:06 smithi203 bash[23654]: cluster 2024-03-11T21:46:05.404716+0000 osd.2 (osd.2) 465 : cluster 0 5.d scrub starts 2024-03-11T21:46:07.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:06 smithi203 bash[23654]: cluster 2024-03-11T21:46:05.408155+0000 osd.2 (osd.2) 466 : cluster 0 5.d scrub ok 2024-03-11T21:46:07.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:06 smithi203 bash[23654]: cluster 2024-03-11T21:46:05.734497+0000 mgr.y (mgr.14550) 889 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:46:07.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:06 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.659317+0000 mon.a (mon.0) 4091 : cluster 0 osdmap e975: 8 total, 8 up, 8 in 2024-03-11T21:46:07.675 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.476774+0000 osd.2 (osd.2) 467 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.477226+0000 osd.2 (osd.2) 468 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.664201+0000 osd.6 (osd.6) 449 : cluster 0 9.18 scrub starts 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.667072+0000 osd.6 (osd.6) 450 : cluster 0 9.18 scrub ok 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.788896+0000 osd.1 (osd.1) 333 : cluster 0 9.f scrub starts 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.796208+0000 osd.1 (osd.1) 334 : cluster 0 9.f scrub ok 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.476774+0000 osd.2 (osd.2) 467 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:07.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.477226+0000 osd.2 (osd.2) 468 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.664201+0000 osd.6 (osd.6) 449 : cluster 0 9.18 scrub starts 2024-03-11T21:46:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.667072+0000 osd.6 (osd.6) 450 : cluster 0 9.18 scrub ok 2024-03-11T21:46:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.788896+0000 osd.1 (osd.1) 333 : cluster 0 9.f scrub starts 2024-03-11T21:46:07.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:07 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.796208+0000 osd.1 (osd.1) 334 : cluster 0 9.f scrub ok 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.476774+0000 osd.2 (osd.2) 467 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.477226+0000 osd.2 (osd.2) 468 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.664201+0000 osd.6 (osd.6) 449 : cluster 0 9.18 scrub starts 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.667072+0000 osd.6 (osd.6) 450 : cluster 0 9.18 scrub ok 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.788896+0000 osd.1 (osd.1) 333 : cluster 0 9.f scrub starts 2024-03-11T21:46:08.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:07 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.796208+0000 osd.1 (osd.1) 334 : cluster 0 9.f scrub ok 2024-03-11T21:46:08.697 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:08.993 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[29284]: cluster 2024-03-11T21:46:06.997784+0000 osd.4 (osd.4) 511 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[29284]: cluster 2024-03-11T21:46:07.000321+0000 osd.4 (osd.4) 512 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[29284]: cluster 2024-03-11T21:46:07.735176+0000 mgr.y (mgr.14550) 890 : cluster 0 pgmap v1546: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[29284]: audit 2024-03-11T21:46:07.853197+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[29284]: audit 2024-03-11T21:46:07.856088+0000 mon.c (mon.2) 1395 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[21512]: cluster 2024-03-11T21:46:06.997784+0000 osd.4 (osd.4) 511 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[21512]: cluster 2024-03-11T21:46:07.000321+0000 osd.4 (osd.4) 512 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[21512]: cluster 2024-03-11T21:46:07.735176+0000 mgr.y (mgr.14550) 890 : cluster 0 pgmap v1546: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[21512]: audit 2024-03-11T21:46:07.853197+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:08.994 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:08 smithi143 bash[21512]: audit 2024-03-11T21:46:07.856088+0000 mon.c (mon.2) 1395 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:09.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:08 smithi203 bash[23654]: cluster 2024-03-11T21:46:06.997784+0000 osd.4 (osd.4) 511 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:46:09.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:08 smithi203 bash[23654]: cluster 2024-03-11T21:46:07.000321+0000 osd.4 (osd.4) 512 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:46:09.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:08 smithi203 bash[23654]: cluster 2024-03-11T21:46:07.735176+0000 mgr.y (mgr.14550) 890 : cluster 0 pgmap v1546: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:09.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:08 smithi203 bash[23654]: audit 2024-03-11T21:46:07.853197+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:46:09.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:08 smithi203 bash[23654]: audit 2024-03-11T21:46:07.856088+0000 mon.c (mon.2) 1395 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:09.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[21512]: cluster 2024-03-11T21:46:07.951272+0000 osd.4 (osd.4) 513 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[21512]: cluster 2024-03-11T21:46:07.956372+0000 osd.4 (osd.4) 514 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[21512]: cluster 2024-03-11T21:46:08.936809+0000 osd.1 (osd.1) 335 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[21512]: cluster 2024-03-11T21:46:08.937403+0000 osd.1 (osd.1) 336 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[29284]: cluster 2024-03-11T21:46:07.951272+0000 osd.4 (osd.4) 513 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[29284]: cluster 2024-03-11T21:46:07.956372+0000 osd.4 (osd.4) 514 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[29284]: cluster 2024-03-11T21:46:08.936809+0000 osd.1 (osd.1) 335 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:09.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:09 smithi143 bash[29284]: cluster 2024-03-11T21:46:08.937403+0000 osd.1 (osd.1) 336 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:10.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:09 smithi203 bash[23654]: cluster 2024-03-11T21:46:07.951272+0000 osd.4 (osd.4) 513 : cluster 0 3.1c deep-scrub starts 2024-03-11T21:46:10.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:09 smithi203 bash[23654]: cluster 2024-03-11T21:46:07.956372+0000 osd.4 (osd.4) 514 : cluster 0 3.1c deep-scrub ok 2024-03-11T21:46:10.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:09 smithi203 bash[23654]: cluster 2024-03-11T21:46:08.936809+0000 osd.1 (osd.1) 335 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:10.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:09 smithi203 bash[23654]: cluster 2024-03-11T21:46:08.937403+0000 osd.1 (osd.1) 336 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[29284]: audit 2024-03-11T21:46:09.240254+0000 mgr.y (mgr.14550) 891 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.251162+0000 osd.0 (osd.0) 524 : cluster 0 6.18 scrub starts 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.253278+0000 osd.0 (osd.0) 525 : cluster 0 6.18 scrub ok 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.701127+0000 mon.a (mon.0) 4093 : cluster 0 osdmap e976: 8 total, 8 up, 8 in 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.735691+0000 mgr.y (mgr.14550) 892 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[21512]: audit 2024-03-11T21:46:09.240254+0000 mgr.y (mgr.14550) 891 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.251162+0000 osd.0 (osd.0) 524 : cluster 0 6.18 scrub starts 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.253278+0000 osd.0 (osd.0) 525 : cluster 0 6.18 scrub ok 2024-03-11T21:46:11.001 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.701127+0000 mon.a (mon.0) 4093 : cluster 0 osdmap e976: 8 total, 8 up, 8 in 2024-03-11T21:46:11.002 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:10 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.735691+0000 mgr.y (mgr.14550) 892 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:11.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:10 smithi203 bash[23654]: audit 2024-03-11T21:46:09.240254+0000 mgr.y (mgr.14550) 891 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:11.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:10 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.251162+0000 osd.0 (osd.0) 524 : cluster 0 6.18 scrub starts 2024-03-11T21:46:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:10 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.253278+0000 osd.0 (osd.0) 525 : cluster 0 6.18 scrub ok 2024-03-11T21:46:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:10 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.701127+0000 mon.a (mon.0) 4093 : cluster 0 osdmap e976: 8 total, 8 up, 8 in 2024-03-11T21:46:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:10 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.735691+0000 mgr.y (mgr.14550) 892 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.994095+0000 osd.4 (osd.4) 515 : cluster 0 3.3 scrub starts 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:09.998927+0000 osd.4 (osd.4) 516 : cluster 0 3.3 scrub ok 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.244639+0000 osd.0 (osd.0) 526 : cluster 0 6.10 scrub starts 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.247326+0000 osd.0 (osd.0) 527 : cluster 0 6.10 scrub ok 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.385766+0000 osd.2 (osd.2) 469 : cluster 0 2.3 scrub starts 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.388595+0000 osd.2 (osd.2) 470 : cluster 0 2.3 scrub ok 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.683506+0000 osd.6 (osd.6) 451 : cluster 0 5.2 scrub starts 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:10.685744+0000 osd.6 (osd.6) 452 : cluster 0 5.2 scrub ok 2024-03-11T21:46:12.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:11 smithi203 bash[23654]: cluster 2024-03-11T21:46:11.664346+0000 mon.a (mon.0) 4094 : cluster 0 osdmap e977: 8 total, 8 up, 8 in 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.994095+0000 osd.4 (osd.4) 515 : cluster 0 3.3 scrub starts 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:09.998927+0000 osd.4 (osd.4) 516 : cluster 0 3.3 scrub ok 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.244639+0000 osd.0 (osd.0) 526 : cluster 0 6.10 scrub starts 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.247326+0000 osd.0 (osd.0) 527 : cluster 0 6.10 scrub ok 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.385766+0000 osd.2 (osd.2) 469 : cluster 0 2.3 scrub starts 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.388595+0000 osd.2 (osd.2) 470 : cluster 0 2.3 scrub ok 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.683506+0000 osd.6 (osd.6) 451 : cluster 0 5.2 scrub starts 2024-03-11T21:46:12.064 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:10.685744+0000 osd.6 (osd.6) 452 : cluster 0 5.2 scrub ok 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[29284]: cluster 2024-03-11T21:46:11.664346+0000 mon.a (mon.0) 4094 : cluster 0 osdmap e977: 8 total, 8 up, 8 in 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.994095+0000 osd.4 (osd.4) 515 : cluster 0 3.3 scrub starts 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:09.998927+0000 osd.4 (osd.4) 516 : cluster 0 3.3 scrub ok 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.244639+0000 osd.0 (osd.0) 526 : cluster 0 6.10 scrub starts 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.247326+0000 osd.0 (osd.0) 527 : cluster 0 6.10 scrub ok 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.385766+0000 osd.2 (osd.2) 469 : cluster 0 2.3 scrub starts 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.388595+0000 osd.2 (osd.2) 470 : cluster 0 2.3 scrub ok 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.683506+0000 osd.6 (osd.6) 451 : cluster 0 5.2 scrub starts 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:10.685744+0000 osd.6 (osd.6) 452 : cluster 0 5.2 scrub ok 2024-03-11T21:46:12.065 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:11 smithi143 bash[21512]: cluster 2024-03-11T21:46:11.664346+0000 mon.a (mon.0) 4094 : cluster 0 osdmap e977: 8 total, 8 up, 8 in 2024-03-11T21:46:12.446 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[21512]: cluster 2024-03-11T21:46:11.238939+0000 osd.3 (osd.3) 537 : cluster 0 3.5 scrub starts 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[21512]: cluster 2024-03-11T21:46:11.243479+0000 osd.3 (osd.3) 538 : cluster 0 3.5 scrub ok 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[21512]: cluster 2024-03-11T21:46:11.736265+0000 mgr.y (mgr.14550) 893 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[29284]: cluster 2024-03-11T21:46:11.238939+0000 osd.3 (osd.3) 537 : cluster 0 3.5 scrub starts 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[29284]: cluster 2024-03-11T21:46:11.243479+0000 osd.3 (osd.3) 538 : cluster 0 3.5 scrub ok 2024-03-11T21:46:12.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:12 smithi143 bash[29284]: cluster 2024-03-11T21:46:11.736265+0000 mgr.y (mgr.14550) 893 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:46:13.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:12 smithi203 bash[23654]: cluster 2024-03-11T21:46:11.238939+0000 osd.3 (osd.3) 537 : cluster 0 3.5 scrub starts 2024-03-11T21:46:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:12 smithi203 bash[23654]: cluster 2024-03-11T21:46:11.243479+0000 osd.3 (osd.3) 538 : cluster 0 3.5 scrub ok 2024-03-11T21:46:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:12 smithi203 bash[23654]: cluster 2024-03-11T21:46:11.736265+0000 mgr.y (mgr.14550) 893 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.30666.0:1 308.7s0 308:e7e50e30:test-rados-api-smithi143-71773-105::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e652)", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:38:59.531093+0000", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 434.26499314500001, 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.128094935, 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.30666", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/631890419", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531093+0000", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531093+0000", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531095+0000", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0090000000000002e-06 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531103+0000", 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8569999999999995e-06 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531122+0000", 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9780999999999999e-05 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531137+0000", 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4676e-05 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531218+0000", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1111000000000003e-05 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:38:59.531223+0000", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2140000000000004e-06 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.652423+0000", 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.121199088 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.652830+0000", 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00040713300000000001 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:00.659188+0000", 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0063580659999999999 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:816 320.10 320:08000000:%01:::head [pgnls start_epoch 749] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e749)", 2024-03-11T21:46:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:48.628253+0000", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 325.16783303199998, 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.047407206, 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 816 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.628253+0000", 2024-03-11T21:46:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.628253+0000", 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.628255+0000", 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0250000000000001e-06 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.628265+0000", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9809999999999998e-06 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.674238+0000", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.045973061000000003 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.674261+0000", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2843000000000001e-05 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.674725+0000", 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00046442199999999998 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.674776+0000", 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.1106000000000002e-05 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:48.675660+0000", 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00088376800000000005 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.792 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1228 322.0 322:00000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:11.528192+0000", 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 302.26789407899997, 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.097662613999999995, 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1228 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.793 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.528192+0000", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.528192+0000", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.528195+0000", 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9809999999999999e-06 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.794 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.528210+0000", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5366000000000002e-05 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.625212+0000", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.097001511999999998 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.625267+0000", 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5250000000000001e-05 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.795 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.625603+0000", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00033575899999999998 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.625649+0000", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.5991000000000001e-05 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.625854+0000", 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020575500000000001 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.796 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2313 330.4 330:2473172f:test-rados-api-smithi143-71773-145::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e850)", 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:38.581125+0000", 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 215.214960894, 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014854697, 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2313 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.797 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.581125+0000", 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.581125+0000", 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.581128+0000", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0970000000000002e-06 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.581145+0000", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7243000000000001e-05 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582216+0000", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0010705739999999999 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582244+0000", 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7685999999999999e-05 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582394+0000", 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015022599999999999 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.595980+0000", 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013585870999999999 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3803 3.0 3:02683654:reshard::reshard.0000000010:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:10.492982+0000", 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 183.30310351700001, 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001972115, 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3803 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.801 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.492982+0000", 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.492982+0000", 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.802 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.492984+0000", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.61e-06 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.492991+0000", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8870000000000003e-06 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493005+0000", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4306e-05 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493012+0000", 2024-03-11T21:46:13.803 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0049999999999998e-06 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493076+0000", 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3621999999999995e-05 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493151+0000", 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4833999999999995e-05 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:46:13.804 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493286+0000", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000135492 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.493661+0000", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00037475699999999998 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.494171+0000", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00050992500000000005 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.494859+0000", 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00068824000000000001 2024-03-11T21:46:13.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.494920+0000", 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.0874e-05 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:10.494955+0000", 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4563000000000001e-05 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.806 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3833 3.19 3:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.178751+0000", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.61733502000001, 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0020419869999999999, 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3833 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.807 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178751+0000", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178751+0000", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178754+0000", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.734e-06 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178784+0000", 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0799000000000001e-05 2024-03-11T21:46:13.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178810+0000", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5143e-05 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178825+0000", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5182e-05 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.178938+0000", 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000113584 2024-03-11T21:46:13.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.179080+0000", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014186400000000001 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.179345+0000", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000264536 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.180044+0000", 2024-03-11T21:46:13.810 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000698809 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.180540+0000", 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00049636700000000005 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.180647+0000", 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000106613 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.811 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.180728+0000", 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1731999999999996e-05 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.180793+0000", 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.4623999999999997e-05 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3835 3.19 3:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.181773+0000", 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.614312778, 2024-03-11T21:46:13.812 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002512139, 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3835 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181773+0000", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181773+0000", 2024-03-11T21:46:13.813 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181775+0000", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4640000000000001e-06 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181782+0000", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8460000000000008e-06 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181799+0000", 2024-03-11T21:46:13.814 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6429000000000001e-05 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181810+0000", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1177e-05 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.181932+0000", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000122067 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.182095+0000", 2024-03-11T21:46:13.815 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000162929 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.182459+0000", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00036347400000000002 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.183533+0000", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0010740649999999999 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.183791+0000", 2024-03-11T21:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025861999999999999 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184130+0000", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00033921500000000002 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184219+0000", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.8571000000000003e-05 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184285+0000", 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6282e-05 2024-03-11T21:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3836 3.4 3:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.184707+0000", 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.611378997, 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0022184420000000002, 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.818 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3836 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184707+0000", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184707+0000", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184709+0000", 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8700000000000001e-06 2024-03-11T21:46:13.819 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184724+0000", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5201999999999999e-05 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184750+0000", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5789999999999999e-05 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184772+0000", 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1795999999999999e-05 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.820 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.184892+0000", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000120384 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.185047+0000", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015454600000000001 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.185334+0000", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00028705199999999999 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.186095+0000", 2024-03-11T21:46:13.821 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00076108100000000004 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.186637+0000", 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000542582 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.186763+0000", 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012528300000000001 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.822 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.186853+0000", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.0100999999999997e-05 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.186925+0000", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2755000000000004e-05 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3838 3.4 3:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.188363+0000", 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.607723155, 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002570929, 2024-03-11T21:46:13.823 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3838 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188363+0000", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188363+0000", 2024-03-11T21:46:13.824 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188365+0000", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3199999999999998e-06 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188378+0000", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2506e-05 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188408+0000", 2024-03-11T21:46:13.825 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0639000000000003e-05 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188424+0000", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6163000000000001e-05 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188553+0000", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012829700000000001 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.188810+0000", 2024-03-11T21:46:13.826 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025699000000000002 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.189328+0000", 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00051789999999999996 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.190182+0000", 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00085441100000000002 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.827 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.190669+0000", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00048673 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.190786+0000", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011750400000000001 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.190871+0000", 2024-03-11T21:46:13.828 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.5011000000000003e-05 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.190934+0000", 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2458000000000004e-05 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3854 3.4 3:264a8785:::obj_delete_at_hint.0000000011:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.216986+0000", 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.57909946500001, 2024-03-11T21:46:13.829 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0020707379999999999, 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3854 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.216986+0000", 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.830 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.216986+0000", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.216988+0000", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.699e-06 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.216997+0000", 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.7569999999999993e-06 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.831 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.217014+0000", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6671e-05 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.217022+0000", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.7900000000000005e-06 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.217126+0000", 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000103848 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.832 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.217286+0000", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016016199999999999 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.217583+0000", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000296249 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.218311+0000", 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00072847799999999996 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.833 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.218862+0000", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00055109799999999997 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.218958+0000", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.5319e-05 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.219002+0000", 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4567000000000002e-05 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.834 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.219057+0000", 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5099999999999998e-05 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:3856 3.4 3:264a8785:::obj_delete_at_hint.0000000011:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.220061+0000", 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.57602482999999, 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0021469150000000001, 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.835 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3856 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220061+0000", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220061+0000", 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.836 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220063+0000", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.421e-06 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220073+0000", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0866999999999999e-05 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220088+0000", 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4878999999999999e-05 2024-03-11T21:46:13.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220102+0000", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3856000000000001e-05 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220175+0000", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2781000000000007e-05 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220305+0000", 2024-03-11T21:46:13.838 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013028000000000001 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.220584+0000", 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027846300000000001 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.221325+0000", 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00074155100000000002 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.839 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.221783+0000", 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00045814500000000003 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.221900+0000", 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000116543 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.222075+0000", 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000175439 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.840 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.222208+0000", 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013269000000000001 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4121 3.19 3:98c4c580:::obj_delete_at_hint.0000000100:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.565608+0000", 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.230478296, 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0027173089999999998, 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.841 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4121 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565608+0000", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565608+0000", 2024-03-11T21:46:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565611+0000", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0639999999999998e-06 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565628+0000", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6918e-05 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565666+0000", 2024-03-11T21:46:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8615000000000002e-05 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565686+0000", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9715e-05 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.565828+0000", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014249099999999999 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.566037+0000", 2024-03-11T21:46:13.844 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020830600000000001 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.566437+0000", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00039992499999999998 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.567437+0000", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00099993400000000007 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.568193+0000", 2024-03-11T21:46:13.845 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00075661100000000002 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.568210+0000", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6753999999999999e-05 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.568267+0000", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6827000000000003e-05 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.568325+0000", 2024-03-11T21:46:13.846 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.8149000000000002e-05 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4133 3.0 3:0255b7ba:::obj_delete_at_hint.0000000104:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.584601+0000", 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.21148448299999, 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0018475449999999999, 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.847 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4133 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584601+0000", 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584601+0000", 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.848 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584604+0000", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.407e-06 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584620+0000", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5857000000000001e-05 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584647+0000", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7498999999999999e-05 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.849 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584664+0000", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6324000000000002e-05 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584772+0000", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000108336 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.584908+0000", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000136095 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:46:13.850 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.585142+0000", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00023361500000000001 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.585764+0000", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00062199599999999999 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.586135+0000", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000371216 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.586335+0000", 2024-03-11T21:46:13.851 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019981900000000001 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.586394+0000", 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9799999999999997e-05 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.586449+0000", 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.4580999999999998e-05 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.852 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4160 3.4 3:258fd845:::obj_delete_at_hint.0000000113:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.672900+0000", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.123185511, 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001896968, 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4160 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.853 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672900+0000", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672900+0000", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672903+0000", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2079999999999999e-06 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.854 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672915+0000", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1898e-05 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672935+0000", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0411000000000002e-05 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672946+0000", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1412e-05 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.673026+0000", 2024-03-11T21:46:13.855 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.0049999999999994e-05 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.673163+0000", 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013646900000000001 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.673402+0000", 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000238769 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.856 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.674192+0000", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000790626 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.674292+0000", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9701e-05 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.674719+0000", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00042709900000000001 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.857 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.674756+0000", 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6602e-05 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.674797+0000", 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1723e-05 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4162 3.4 3:258fd845:::obj_delete_at_hint.0000000113:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.675473+0000", 2024-03-11T21:46:13.858 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 182.12061285999999, 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013425447, 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4162 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675473+0000", 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.859 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675473+0000", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675474+0000", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.5199999999999995e-07 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675489+0000", 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5332000000000002e-05 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675504+0000", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4521999999999999e-05 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675512+0000", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.5040000000000002e-06 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675570+0000", 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7912999999999999e-05 2024-03-11T21:46:13.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675670+0000", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9753000000000005e-05 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675816+0000", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014641400000000001 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.676652+0000", 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00083525500000000003 2024-03-11T21:46:13.862 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688447+0000", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011795129 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688773+0000", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00032598799999999998 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688841+0000", 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.8287000000000005e-05 2024-03-11T21:46:13.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688899+0000", 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7398000000000002e-05 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:3122 334.14 334:283cf3a8:test-rados-api-smithi143-71773-153::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:24.617936+0000", 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 169.178149529, 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.04553923, 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.864 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3122 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.617936+0000", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.617936+0000", 2024-03-11T21:46:13.865 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.617940+0000", 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3459999999999998e-06 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.617956+0000", 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.592e-05 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.866 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.617992+0000", 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.5889000000000001e-05 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.618017+0000", 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5063e-05 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.644338+0000", 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.026321520000000001 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.867 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.646421+0000", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00208304 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.646701+0000", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027968699999999998 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.648886+0000", 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0021847020000000002 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.868 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.649218+0000", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00033273999999999999 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.663345+0000", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014126420000000001 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.663420+0000", 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4795999999999993e-05 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.869 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.663476+0000", 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6107000000000003e-05 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:3126 334.14 334:283cf3a8:test-rados-api-smithi143-71773-153::foo:head [set-redirect] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:24.670823+0000", 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 169.12526257499999, 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0018680669999999999, 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.870 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3126 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670823+0000", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670823+0000", 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670827+0000", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4659999999999999e-06 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670841+0000", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3828e-05 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670875+0000", 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.4702000000000001e-05 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.872 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670893+0000", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7238e-05 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.670992+0000", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9554000000000003e-05 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.671117+0000", 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012520899999999999 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.873 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.671311+0000", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000193353 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.671893+0000", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00058218099999999997 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.672239+0000", 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00034596799999999999 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.874 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.672591+0000", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00035200900000000002 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.672644+0000", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.3569999999999997e-05 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.672691+0000", 2024-03-11T21:46:13.875 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.6989e-05 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:3128 334.14 334:283cf3a8:test-rados-api-smithi143-71773-153::foo:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:24.676587+0000", 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 169.11949847899999, 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0043615670000000002, 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:13.876 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3128 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676587+0000", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676587+0000", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.877 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676589+0000", 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5200000000000001e-06 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676597+0000", 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.1300000000000001e-06 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676620+0000", 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2532999999999999e-05 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.878 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676632+0000", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2656e-05 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676707+0000", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.4259999999999997e-05 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.676793+0000", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6378000000000005e-05 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.879 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.680949+0000", 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0041560900000000003 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(2.11,QREQUEST,e:918,from:6,reservation_nonce:14)", 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:44:31.688044+0000", 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 102.108042154, 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0027774620000000001, 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.880 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688044+0000", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688044+0000", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688046+0000", 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7540000000000001e-06 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.881 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688056+0000", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0219e-05 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688082+0000", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6642999999999999e-05 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.688111+0000", 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8501999999999999e-05 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.882 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.690657+0000", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002546024 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.690701+0000", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4499000000000002e-05 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.690821+0000", 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000119821 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.883 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.42197.0:2 338.0 e918/917 338:0384ab8e:::oid:head v 918'1, mlcod=0'0)", 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:44:31.700268+0000", 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 102.095817747, 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0059928239999999999, 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700268+0000", 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.884 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700268+0000", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700268+0000", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5199999999999998e-07 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700278+0000", 2024-03-11T21:46:13.885 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6229999999999992e-06 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700286+0000", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.593e-06 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700294+0000", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4600000000000003e-06 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700372+0000", 2024-03-11T21:46:13.886 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.8033000000000003e-05 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.700394+0000", 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1393e-05 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.706227+0000", 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0058337190000000002 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:13.887 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.706261+0000", 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3751000000000003e-05 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:13.888 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:12.020107+0000 osd.4 (osd.4) 517 : cluster 0 1.5 scrub starts 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:12.022509+0000 osd.4 (osd.4) 518 : cluster 0 1.5 scrub ok 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:12.247226+0000 osd.0 (osd.0) 528 : cluster 0 9.1c scrub starts 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:12.248612+0000 osd.0 (osd.0) 529 : cluster 0 9.1c scrub ok 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:13.322961+0000 osd.7 (osd.7) 512 : cluster 0 9.d scrub starts 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[29284]: cluster 2024-03-11T21:46:13.326816+0000 osd.7 (osd.7) 513 : cluster 0 9.d scrub ok 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:12.020107+0000 osd.4 (osd.4) 517 : cluster 0 1.5 scrub starts 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:12.022509+0000 osd.4 (osd.4) 518 : cluster 0 1.5 scrub ok 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:12.247226+0000 osd.0 (osd.0) 528 : cluster 0 9.1c scrub starts 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:12.248612+0000 osd.0 (osd.0) 529 : cluster 0 9.1c scrub ok 2024-03-11T21:46:13.889 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:13.322961+0000 osd.7 (osd.7) 512 : cluster 0 9.d scrub starts 2024-03-11T21:46:13.890 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:13 smithi143 bash[21512]: cluster 2024-03-11T21:46:13.326816+0000 osd.7 (osd.7) 513 : cluster 0 9.d scrub ok 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:12.020107+0000 osd.4 (osd.4) 517 : cluster 0 1.5 scrub starts 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:12.022509+0000 osd.4 (osd.4) 518 : cluster 0 1.5 scrub ok 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:12.247226+0000 osd.0 (osd.0) 528 : cluster 0 9.1c scrub starts 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:12.248612+0000 osd.0 (osd.0) 529 : cluster 0 9.1c scrub ok 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:13.322961+0000 osd.7 (osd.7) 512 : cluster 0 9.d scrub starts 2024-03-11T21:46:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:13 smithi203 bash[23654]: cluster 2024-03-11T21:46:13.326816+0000 osd.7 (osd.7) 513 : cluster 0 9.d scrub ok 2024-03-11T21:46:14.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/716242288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a40107e90 msgr2=0x7f5a40108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:14.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/716242288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a40107e90 0x7f5a40108270 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009990 tx=0x7f5a30009ca0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/716242288 shutdown_connections 2024-03-11T21:46:14.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/716242288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4006ae80 0x7f5a4010ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/716242288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a40107e90 0x7f5a40108270 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/716242288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006a4e0 0x7f5a4006a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/716242288 >> 172.21.15.143:0/716242288 conn(0x7f5a40077c80 msgr2=0x7f5a40078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/716242288 shutdown_connections 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/716242288 wait complete. 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 librados: starting msgr at 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 librados: starting objecter 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 Processor -- start 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- start start 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 librados: setting wanted keys 2024-03-11T21:46:14.466 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 librados: calling monclient init 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 0x7f5a4019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 0x7f5a40197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40111610 con 0x7f5a40107e90 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40111490 con 0x7f5a4006ae80 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40111790 con 0x7f5a4006a4e0 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 0x7f5a40197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:14.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38534/0 (socket says 172.21.15.143:38534) 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3f7fe700 1 -- 172.21.15.143:0/927058162 learned_addr learned my addr 172.21.15.143:0/927058162 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3effd700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 0x7f5a4019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 -- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 msgr2=0x7f5a4019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 -- 172.21.15.143:0/927058162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 msgr2=0x7f5a4019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 0x7f5a4019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 -- 172.21.15.143:0/927058162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40198400 con 0x7f5a40107e90 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3f7fe700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3effd700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 0x7f5a4019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3ffff700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 0x7f5a40197b20 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800efb0 tx=0x7f5a3800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3cff9700 10 client.?.objecter ms_handle_connect 0x7f5a40107e90 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3cff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800ca50 con 0x7f5a40107e90 2024-03-11T21:46:14.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5a3800cbf0 con 0x7f5a40107e90 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a401986f0 con 0x7f5a40107e90 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38004740 con 0x7f5a40107e90 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 10 client.31119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a401a49d0 con 0x7f5a40107e90 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.476+0000 7f5a45e4e700 1 librados: init done 2024-03-11T21:46:14.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40103540 con 0x7f5a40107e90 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5a38012050 con 0x7f5a40107e90 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 0x7f5a2c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(977..977 src has 842..977) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f5a3808e200 con 0x7f5a40107e90 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 10 client.31119.objecter ms_dispatch 0x7f5a40001550 osd_map(977..977 src has 842..977) v4 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 3 client.31119.objecter handle_osd_map got epochs [977,977] > 0 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 3 client.31119.objecter handle_osd_map decoding full epoch 977 2024-03-11T21:46:14.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3f7fe700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 0x7f5a2c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:14.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 20 client.31119.objecter dump_active .. 0 homeless 2024-03-11T21:46:14.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3f7fe700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 0x7f5a2c071ce0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000b2e0 tx=0x7f5a3001a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:14.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.480+0000 7f5a3cff9700 10 client.31119.objecter ms_handle_connect 0x7f5a2c06f820 2024-03-11T21:46:14.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.484+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5a38022060 con 0x7f5a40107e90 2024-03-11T21:46:14.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.704+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f5a40198fe0 con 0x7f5a40107e90 2024-03-11T21:46:14.721 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a3cff9700 1 -- 172.21.15.143:0/927058162 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v978) v1 ==== 113+0+0 (secure 0 0 0) 0x7f5a38010040 con 0x7f5a40107e90 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 10 librados: watch_flush enter 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 10 librados: watch_flush exit 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 20 client.31119.objecter shutdown clearing up homeless session... 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 10 client.31119.objecter successfully canceled tick 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 msgr2=0x7f5a2c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 0x7f5a2c071ce0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000b2e0 tx=0x7f5a3001a040 comp rx=0 tx=0).stop 2024-03-11T21:46:14.723 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 msgr2=0x7f5a40197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:14.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 0x7f5a40197b20 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800efb0 tx=0x7f5a3800c3a0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 shutdown_connections 2024-03-11T21:46:14.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5a2c06f820 0x7f5a2c071ce0 unknown :-1 s=CLOSED pgs=3497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a40107e90 0x7f5a40197b20 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.724 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a4006a4e0 0x7f5a4019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 --2- 172.21.15.143:0/927058162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a4006ae80 0x7f5a4019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.732+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 >> 172.21.15.143:0/927058162 conn(0x7f5a40077c80 msgr2=0x7f5a400ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.736+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 shutdown_connections 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.736+0000 7f5a45e4e700 1 -- 172.21.15.143:0/927058162 wait complete. 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:14.736+0000 7f5a45e4e700 1 librados: shutdown 2024-03-11T21:46:14.725 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:46:15.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: cluster 2024-03-11T21:46:13.272390+0000 osd.0 (osd.0) 530 : cluster 0 6.0 scrub starts 2024-03-11T21:46:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: cluster 2024-03-11T21:46:13.274479+0000 osd.0 (osd.0) 531 : cluster 0 6.0 scrub ok 2024-03-11T21:46:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: cluster 2024-03-11T21:46:13.737104+0000 mgr.y (mgr.14550) 894 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.333420+0000 osd.7 (osd.7) 514 : cluster 0 9.7 scrub starts 2024-03-11T21:46:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.339039+0000 osd.7 (osd.7) 515 : cluster 0 9.7 scrub ok 2024-03-11T21:46:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:14 smithi203 bash[23654]: audit 2024-03-11T21:46:14.708332+0000 mon.a (mon.0) 4095 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: cluster 2024-03-11T21:46:13.272390+0000 osd.0 (osd.0) 530 : cluster 0 6.0 scrub starts 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: cluster 2024-03-11T21:46:13.274479+0000 osd.0 (osd.0) 531 : cluster 0 6.0 scrub ok 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: cluster 2024-03-11T21:46:13.737104+0000 mgr.y (mgr.14550) 894 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.333420+0000 osd.7 (osd.7) 514 : cluster 0 9.7 scrub starts 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.339039+0000 osd.7 (osd.7) 515 : cluster 0 9.7 scrub ok 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[29284]: audit 2024-03-11T21:46:14.708332+0000 mon.a (mon.0) 4095 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:15.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: cluster 2024-03-11T21:46:13.272390+0000 osd.0 (osd.0) 530 : cluster 0 6.0 scrub starts 2024-03-11T21:46:15.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: cluster 2024-03-11T21:46:13.274479+0000 osd.0 (osd.0) 531 : cluster 0 6.0 scrub ok 2024-03-11T21:46:15.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: cluster 2024-03-11T21:46:13.737104+0000 mgr.y (mgr.14550) 894 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:46:15.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.333420+0000 osd.7 (osd.7) 514 : cluster 0 9.7 scrub starts 2024-03-11T21:46:15.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.339039+0000 osd.7 (osd.7) 515 : cluster 0 9.7 scrub ok 2024-03-11T21:46:15.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:14 smithi143 bash[21512]: audit 2024-03-11T21:46:14.708332+0000 mon.a (mon.0) 4095 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:15.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:46:15.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.050682+0000 osd.4 (osd.4) 519 : cluster 0 9.10 scrub starts 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.052649+0000 osd.4 (osd.4) 520 : cluster 0 9.10 scrub ok 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.149028+0000 osd.3 (osd.3) 539 : cluster 0 3.8 scrub starts 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.151736+0000 osd.3 (osd.3) 540 : cluster 0 3.8 scrub ok 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: audit 2024-03-11T21:46:14.736730+0000 mon.a (mon.0) 4096 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:14.745175+0000 mon.a (mon.0) 4097 : cluster 0 osdmap e978: 8 total, 8 up, 8 in 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.347136+0000 osd.7 (osd.7) 516 : cluster 0 2.0 scrub starts 2024-03-11T21:46:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:15 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.349874+0000 osd.7 (osd.7) 517 : cluster 0 2.0 scrub ok 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.050682+0000 osd.4 (osd.4) 519 : cluster 0 9.10 scrub starts 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.052649+0000 osd.4 (osd.4) 520 : cluster 0 9.10 scrub ok 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.149028+0000 osd.3 (osd.3) 539 : cluster 0 3.8 scrub starts 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.151736+0000 osd.3 (osd.3) 540 : cluster 0 3.8 scrub ok 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: audit 2024-03-11T21:46:14.736730+0000 mon.a (mon.0) 4096 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:14.745175+0000 mon.a (mon.0) 4097 : cluster 0 osdmap e978: 8 total, 8 up, 8 in 2024-03-11T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.347136+0000 osd.7 (osd.7) 516 : cluster 0 2.0 scrub starts 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.349874+0000 osd.7 (osd.7) 517 : cluster 0 2.0 scrub ok 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.050682+0000 osd.4 (osd.4) 519 : cluster 0 9.10 scrub starts 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.052649+0000 osd.4 (osd.4) 520 : cluster 0 9.10 scrub ok 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.149028+0000 osd.3 (osd.3) 539 : cluster 0 3.8 scrub starts 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.151736+0000 osd.3 (osd.3) 540 : cluster 0 3.8 scrub ok 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: audit 2024-03-11T21:46:14.736730+0000 mon.a (mon.0) 4096 : audit 1 from='client.? 172.21.15.143:0/927058162' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:14.745175+0000 mon.a (mon.0) 4097 : cluster 0 osdmap e978: 8 total, 8 up, 8 in 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.347136+0000 osd.7 (osd.7) 516 : cluster 0 2.0 scrub starts 2024-03-11T21:46:16.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:15 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.349874+0000 osd.7 (osd.7) 517 : cluster 0 2.0 scrub ok 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.020888+0000 osd.4 (osd.4) 521 : cluster 0 5.e scrub starts 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.024164+0000 osd.4 (osd.4) 522 : cluster 0 5.e scrub ok 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.295690+0000 osd.0 (osd.0) 532 : cluster 0 2.13 scrub starts 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.297102+0000 osd.0 (osd.0) 533 : cluster 0 2.13 scrub ok 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.738230+0000 mgr.y (mgr.14550) 895 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.782594+0000 osd.1 (osd.1) 337 : cluster 0 5.7 scrub starts 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:15.784941+0000 osd.1 (osd.1) 338 : cluster 0 5.7 scrub ok 2024-03-11T21:46:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:16 smithi203 bash[23654]: cluster 2024-03-11T21:46:16.666439+0000 mon.a (mon.0) 4098 : cluster 0 osdmap e979: 8 total, 8 up, 8 in 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.020888+0000 osd.4 (osd.4) 521 : cluster 0 5.e scrub starts 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.024164+0000 osd.4 (osd.4) 522 : cluster 0 5.e scrub ok 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.295690+0000 osd.0 (osd.0) 532 : cluster 0 2.13 scrub starts 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.297102+0000 osd.0 (osd.0) 533 : cluster 0 2.13 scrub ok 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.738230+0000 mgr.y (mgr.14550) 895 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:46:17.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.782594+0000 osd.1 (osd.1) 337 : cluster 0 5.7 scrub starts 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:15.784941+0000 osd.1 (osd.1) 338 : cluster 0 5.7 scrub ok 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[21512]: cluster 2024-03-11T21:46:16.666439+0000 mon.a (mon.0) 4098 : cluster 0 osdmap e979: 8 total, 8 up, 8 in 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.020888+0000 osd.4 (osd.4) 521 : cluster 0 5.e scrub starts 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.024164+0000 osd.4 (osd.4) 522 : cluster 0 5.e scrub ok 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.295690+0000 osd.0 (osd.0) 532 : cluster 0 2.13 scrub starts 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.297102+0000 osd.0 (osd.0) 533 : cluster 0 2.13 scrub ok 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.738230+0000 mgr.y (mgr.14550) 895 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.782594+0000 osd.1 (osd.1) 337 : cluster 0 5.7 scrub starts 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:15.784941+0000 osd.1 (osd.1) 338 : cluster 0 5.7 scrub ok 2024-03-11T21:46:17.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:16 smithi143 bash[29284]: cluster 2024-03-11T21:46:16.666439+0000 mon.a (mon.0) 4098 : cluster 0 osdmap e979: 8 total, 8 up, 8 in 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:16.268997+0000 osd.0 (osd.0) 534 : cluster 0 3.6 scrub starts 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:16.273586+0000 osd.0 (osd.0) 535 : cluster 0 3.6 scrub ok 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:16.388284+0000 osd.7 (osd.7) 518 : cluster 0 3.15 scrub starts 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:16.391989+0000 osd.7 (osd.7) 519 : cluster 0 3.15 scrub ok 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.507787+0000 osd.5 (osd.5) 513 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:18.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:17 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.508631+0000 osd.5 (osd.5) 514 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:16.268997+0000 osd.0 (osd.0) 534 : cluster 0 3.6 scrub starts 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:16.273586+0000 osd.0 (osd.0) 535 : cluster 0 3.6 scrub ok 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:16.388284+0000 osd.7 (osd.7) 518 : cluster 0 3.15 scrub starts 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:16.391989+0000 osd.7 (osd.7) 519 : cluster 0 3.15 scrub ok 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.507787+0000 osd.5 (osd.5) 513 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:18.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.508631+0000 osd.5 (osd.5) 514 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:16.268997+0000 osd.0 (osd.0) 534 : cluster 0 3.6 scrub starts 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:16.273586+0000 osd.0 (osd.0) 535 : cluster 0 3.6 scrub ok 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:16.388284+0000 osd.7 (osd.7) 518 : cluster 0 3.15 scrub starts 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:16.391989+0000 osd.7 (osd.7) 519 : cluster 0 3.15 scrub ok 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.507787+0000 osd.5 (osd.5) 513 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:18.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:17 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.508631+0000 osd.5 (osd.5) 514 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.159038+0000 osd.3 (osd.3) 541 : cluster 0 2.e deep-scrub starts 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.161565+0000 osd.3 (osd.3) 542 : cluster 0 2.e deep-scrub ok 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.265153+0000 osd.0 (osd.0) 536 : cluster 0 6.7 scrub starts 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.267443+0000 osd.0 (osd.0) 537 : cluster 0 6.7 scrub ok 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.739028+0000 mgr.y (mgr.14550) 896 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.771037+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e980: 8 total, 8 up, 8 in 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.823612+0000 osd.1 (osd.1) 339 : cluster 0 2.15 scrub starts 2024-03-11T21:46:19.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:18 smithi203 bash[23654]: cluster 2024-03-11T21:46:17.825991+0000 osd.1 (osd.1) 340 : cluster 0 2.15 scrub ok 2024-03-11T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.159038+0000 osd.3 (osd.3) 541 : cluster 0 2.e deep-scrub starts 2024-03-11T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.161565+0000 osd.3 (osd.3) 542 : cluster 0 2.e deep-scrub ok 2024-03-11T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.265153+0000 osd.0 (osd.0) 536 : cluster 0 6.7 scrub starts 2024-03-11T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.267443+0000 osd.0 (osd.0) 537 : cluster 0 6.7 scrub ok 2024-03-11T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.739028+0000 mgr.y (mgr.14550) 896 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.771037+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e980: 8 total, 8 up, 8 in 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.823612+0000 osd.1 (osd.1) 339 : cluster 0 2.15 scrub starts 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[21512]: cluster 2024-03-11T21:46:17.825991+0000 osd.1 (osd.1) 340 : cluster 0 2.15 scrub ok 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.159038+0000 osd.3 (osd.3) 541 : cluster 0 2.e deep-scrub starts 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.161565+0000 osd.3 (osd.3) 542 : cluster 0 2.e deep-scrub ok 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.265153+0000 osd.0 (osd.0) 536 : cluster 0 6.7 scrub starts 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.267443+0000 osd.0 (osd.0) 537 : cluster 0 6.7 scrub ok 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.739028+0000 mgr.y (mgr.14550) 896 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.771037+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e980: 8 total, 8 up, 8 in 2024-03-11T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.823612+0000 osd.1 (osd.1) 339 : cluster 0 2.15 scrub starts 2024-03-11T21:46:19.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:18 smithi143 bash[29284]: cluster 2024-03-11T21:46:17.825991+0000 osd.1 (osd.1) 340 : cluster 0 2.15 scrub ok 2024-03-11T21:46:19.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:19.874 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:19.874 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:20.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:19 smithi203 bash[23654]: cluster 2024-03-11T21:46:18.144157+0000 osd.3 (osd.3) 543 : cluster 0 9.1 scrub starts 2024-03-11T21:46:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:19 smithi203 bash[23654]: cluster 2024-03-11T21:46:18.147663+0000 osd.3 (osd.3) 544 : cluster 0 9.1 scrub ok 2024-03-11T21:46:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:19 smithi203 bash[23654]: cluster 2024-03-11T21:46:18.240578+0000 osd.0 (osd.0) 538 : cluster 0 5.1b scrub starts 2024-03-11T21:46:20.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:19 smithi203 bash[23654]: cluster 2024-03-11T21:46:18.243186+0000 osd.0 (osd.0) 539 : cluster 0 5.1b scrub ok 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[21512]: cluster 2024-03-11T21:46:18.144157+0000 osd.3 (osd.3) 543 : cluster 0 9.1 scrub starts 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[21512]: cluster 2024-03-11T21:46:18.147663+0000 osd.3 (osd.3) 544 : cluster 0 9.1 scrub ok 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[21512]: cluster 2024-03-11T21:46:18.240578+0000 osd.0 (osd.0) 538 : cluster 0 5.1b scrub starts 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[21512]: cluster 2024-03-11T21:46:18.243186+0000 osd.0 (osd.0) 539 : cluster 0 5.1b scrub ok 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[29284]: cluster 2024-03-11T21:46:18.144157+0000 osd.3 (osd.3) 543 : cluster 0 9.1 scrub starts 2024-03-11T21:46:20.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[29284]: cluster 2024-03-11T21:46:18.147663+0000 osd.3 (osd.3) 544 : cluster 0 9.1 scrub ok 2024-03-11T21:46:20.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[29284]: cluster 2024-03-11T21:46:18.240578+0000 osd.0 (osd.0) 538 : cluster 0 5.1b scrub starts 2024-03-11T21:46:20.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:19 smithi143 bash[29284]: cluster 2024-03-11T21:46:18.243186+0000 osd.0 (osd.0) 539 : cluster 0 5.1b scrub ok 2024-03-11T21:46:21.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:20 smithi203 bash[23654]: cluster 2024-03-11T21:46:19.186376+0000 osd.3 (osd.3) 545 : cluster 0 2.19 scrub starts 2024-03-11T21:46:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:20 smithi203 bash[23654]: cluster 2024-03-11T21:46:19.189419+0000 osd.3 (osd.3) 546 : cluster 0 2.19 scrub ok 2024-03-11T21:46:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:20 smithi203 bash[23654]: audit 2024-03-11T21:46:19.248783+0000 mgr.y (mgr.14550) 897 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:20 smithi203 bash[23654]: cluster 2024-03-11T21:46:19.740898+0000 mgr.y (mgr.14550) 898 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:21.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[29284]: cluster 2024-03-11T21:46:19.186376+0000 osd.3 (osd.3) 545 : cluster 0 2.19 scrub starts 2024-03-11T21:46:21.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[29284]: cluster 2024-03-11T21:46:19.189419+0000 osd.3 (osd.3) 546 : cluster 0 2.19 scrub ok 2024-03-11T21:46:21.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[29284]: audit 2024-03-11T21:46:19.248783+0000 mgr.y (mgr.14550) 897 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:21.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[29284]: cluster 2024-03-11T21:46:19.740898+0000 mgr.y (mgr.14550) 898 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:21.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[21512]: cluster 2024-03-11T21:46:19.186376+0000 osd.3 (osd.3) 545 : cluster 0 2.19 scrub starts 2024-03-11T21:46:21.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[21512]: cluster 2024-03-11T21:46:19.189419+0000 osd.3 (osd.3) 546 : cluster 0 2.19 scrub ok 2024-03-11T21:46:21.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[21512]: audit 2024-03-11T21:46:19.248783+0000 mgr.y (mgr.14550) 897 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:21.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:20 smithi143 bash[21512]: cluster 2024-03-11T21:46:19.740898+0000 mgr.y (mgr.14550) 898 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:22.447 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:22.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:22 smithi143 bash[21512]: cluster 2024-03-11T21:46:21.665553+0000 mon.a (mon.0) 4100 : cluster 0 osdmap e981: 8 total, 8 up, 8 in 2024-03-11T21:46:22.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:22 smithi143 bash[21512]: cluster 2024-03-11T21:46:21.741594+0000 mgr.y (mgr.14550) 899 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:22.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:22 smithi143 bash[29284]: cluster 2024-03-11T21:46:21.665553+0000 mon.a (mon.0) 4100 : cluster 0 osdmap e981: 8 total, 8 up, 8 in 2024-03-11T21:46:22.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:22 smithi143 bash[29284]: cluster 2024-03-11T21:46:21.741594+0000 mgr.y (mgr.14550) 899 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:23.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:22 smithi203 bash[23654]: cluster 2024-03-11T21:46:21.665553+0000 mon.a (mon.0) 4100 : cluster 0 osdmap e981: 8 total, 8 up, 8 in 2024-03-11T21:46:23.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:22 smithi203 bash[23654]: cluster 2024-03-11T21:46:21.741594+0000 mgr.y (mgr.14550) 899 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.062236+0000 osd.0 (osd.0) 540 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.062625+0000 osd.0 (osd.0) 541 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.202482+0000 osd.3 (osd.3) 547 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.207661+0000 osd.3 (osd.3) 548 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.685846+0000 mon.a (mon.0) 4101 : cluster 0 osdmap e982: 8 total, 8 up, 8 in 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.732658+0000 osd.1 (osd.1) 341 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: cluster 2024-03-11T21:46:22.734959+0000 osd.1 (osd.1) 342 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:46:23.947 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[21512]: audit 2024-03-11T21:46:22.887827+0000 mon.c (mon.2) 1396 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.062236+0000 osd.0 (osd.0) 540 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.062625+0000 osd.0 (osd.0) 541 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.202482+0000 osd.3 (osd.3) 547 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.207661+0000 osd.3 (osd.3) 548 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.685846+0000 mon.a (mon.0) 4101 : cluster 0 osdmap e982: 8 total, 8 up, 8 in 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.732658+0000 osd.1 (osd.1) 341 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: cluster 2024-03-11T21:46:22.734959+0000 osd.1 (osd.1) 342 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:46:23.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:23 smithi143 bash[29284]: audit 2024-03-11T21:46:22.887827+0000 mon.c (mon.2) 1396 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.062236+0000 osd.0 (osd.0) 540 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.062625+0000 osd.0 (osd.0) 541 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.202482+0000 osd.3 (osd.3) 547 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.207661+0000 osd.3 (osd.3) 548 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.685846+0000 mon.a (mon.0) 4101 : cluster 0 osdmap e982: 8 total, 8 up, 8 in 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.732658+0000 osd.1 (osd.1) 341 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: cluster 2024-03-11T21:46:22.734959+0000 osd.1 (osd.1) 342 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:46:24.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:23 smithi203 bash[23654]: audit 2024-03-11T21:46:22.887827+0000 mon.c (mon.2) 1396 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- 172.21.15.143:0/4050674086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc100d40 msgr2=0x7f76fc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 --2- 172.21.15.143:0/4050674086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc100d40 0x7f76fc1011c0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009990 tx=0x7f76ec009ca0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- 172.21.15.143:0/4050674086 shutdown_connections 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 --2- 172.21.15.143:0/4050674086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc101700 0x7f76fc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 --2- 172.21.15.143:0/4050674086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc106dc0 0x7f76fc1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 --2- 172.21.15.143:0/4050674086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc100d40 0x7f76fc1011c0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- 172.21.15.143:0/4050674086 >> 172.21.15.143:0/4050674086 conn(0x7f76fc076d80 msgr2=0x7f76fc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- 172.21.15.143:0/4050674086 shutdown_connections 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- 172.21.15.143:0/4050674086 wait complete. 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 librados: starting msgr at 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 librados: starting objecter 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 Processor -- start 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 -- start start 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 librados: setting wanted keys 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.476+0000 7f7700bb6700 1 librados: calling monclient init 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc100d40 0x7f76fc19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc110570 con 0x7f76fc106dc0 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc1103f0 con 0x7f76fc100d40 2024-03-11T21:46:24.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc1106f0 con 0x7f76fc101700 2024-03-11T21:46:24.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:24.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76f9d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:24.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35232/0 (socket says 172.21.15.143:35232) 2024-03-11T21:46:24.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76f9d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:57306/0 (socket says 172.21.15.143:57306) 2024-03-11T21:46:24.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 -- 172.21.15.143:0/2984463447 learned_addr learned my addr 172.21.15.143:0/2984463447 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:24.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fa59c700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc100d40 0x7f76fc19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:24.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 -- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 msgr2=0x7f76fc19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:24.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 -- 172.21.15.143:0/2984463447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc100d40 msgr2=0x7f76fc19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc100d40 0x7f76fc19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 -- 172.21.15.143:0/2984463447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc197070 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76f9d9b700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fad9d700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f76f400b970 tx=0x7f76f400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 10 client.?.objecter ms_handle_connect 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4019070 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f76f400cdb0 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc197360 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4014e40 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 10 client.31125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc1a36a0 con 0x7f76fc106dc0 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 librados: init done 2024-03-11T21:46:24.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc04faa0 con 0x7f76fc106dc0 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f76f401c070 con 0x7f76fc106dc0 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 0x7f76e4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(982..982 src has 842..982) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f76f408daa0 con 0x7f76fc106dc0 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 10 client.31125.objecter ms_dispatch 0x7f76fc001550 osd_map(982..982 src has 842..982) v4 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76fa59c700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 0x7f76e4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:24.473 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 3 client.31125.objecter handle_osd_map got epochs [982,982] > 0 2024-03-11T21:46:24.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 3 client.31125.objecter handle_osd_map decoding full epoch 982 2024-03-11T21:46:24.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.480+0000 7f76eb7fe700 20 client.31125.objecter dump_active .. 0 homeless 2024-03-11T21:46:24.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.484+0000 7f76eb7fe700 10 client.31125.objecter ms_handle_connect 0x7f76e406f820 2024-03-11T21:46:24.474 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.484+0000 7f76fa59c700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 0x7f76e4071ce0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f76fc197a50 tx=0x7f76f0009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:24.476 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.488+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f76f405e500 con 0x7f76fc106dc0 2024-03-11T21:46:24.556 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:24.557 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:46:24.557 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:46:24.557 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:24.712 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:24.724+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f76fc069b10 con 0x7f76fc106dc0 2024-03-11T21:46:24.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:24 smithi143 bash[29284]: cluster 2024-03-11T21:46:23.743385+0000 mgr.y (mgr.14550) 900 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:24.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:24 smithi143 bash[21512]: cluster 2024-03-11T21:46:23.743385+0000 mgr.y (mgr.14550) 900 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:25.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:24 smithi203 bash[23654]: cluster 2024-03-11T21:46:23.743385+0000 mgr.y (mgr.14550) 900 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:46:25.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:46:25.682 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.692+0000 7f76eb7fe700 1 -- 172.21.15.143:0/2984463447 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v983) v1 ==== 133+0+0 (secure 0 0 0) 0x7f76f4062ac0 con 0x7f76fc106dc0 2024-03-11T21:46:25.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 10 librados: watch_flush enter 2024-03-11T21:46:25.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 10 librados: watch_flush exit 2024-03-11T21:46:25.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 20 client.31125.objecter shutdown clearing up homeless session... 2024-03-11T21:46:25.684 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 10 client.31125.objecter successfully canceled tick 2024-03-11T21:46:25.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 msgr2=0x7f76e4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:25.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 0x7f76e4071ce0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f76fc197a50 tx=0x7f76f0009260 comp rx=0 tx=0).stop 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 msgr2=0x7f76fc196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f76f400b970 tx=0x7f76f400bc80 comp rx=0 tx=0).stop 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 shutdown_connections 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f76e406f820 0x7f76e4071ce0 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76fc106dc0 0x7f76fc196910 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:25.693 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76fc100d40 0x7f76fc19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 --2- 172.21.15.143:0/2984463447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76fc101700 0x7f76fc19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 >> 172.21.15.143:0/2984463447 conn(0x7f76fc076d80 msgr2=0x7f76fc107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 shutdown_connections 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 -- 172.21.15.143:0/2984463447 wait complete. 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:25.696+0000 7f7700bb6700 1 librados: shutdown 2024-03-11T21:46:25.694 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T21:46:25.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[29284]: cluster 2024-03-11T21:46:24.372637+0000 osd.7 (osd.7) 520 : cluster 0 3.d scrub starts 2024-03-11T21:46:25.732 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[29284]: cluster 2024-03-11T21:46:24.377502+0000 osd.7 (osd.7) 521 : cluster 0 3.d scrub ok 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[29284]: audit 2024-03-11T21:46:24.728589+0000 mon.a (mon.0) 4102 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[29284]: cluster 2024-03-11T21:46:24.745690+0000 osd.1 (osd.1) 343 : cluster 0 6.1 scrub starts 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[29284]: cluster 2024-03-11T21:46:24.748007+0000 osd.1 (osd.1) 344 : cluster 0 6.1 scrub ok 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[21512]: cluster 2024-03-11T21:46:24.372637+0000 osd.7 (osd.7) 520 : cluster 0 3.d scrub starts 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[21512]: cluster 2024-03-11T21:46:24.377502+0000 osd.7 (osd.7) 521 : cluster 0 3.d scrub ok 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[21512]: audit 2024-03-11T21:46:24.728589+0000 mon.a (mon.0) 4102 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[21512]: cluster 2024-03-11T21:46:24.745690+0000 osd.1 (osd.1) 343 : cluster 0 6.1 scrub starts 2024-03-11T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:25 smithi143 bash[21512]: cluster 2024-03-11T21:46:24.748007+0000 osd.1 (osd.1) 344 : cluster 0 6.1 scrub ok 2024-03-11T21:46:26.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:25 smithi203 bash[23654]: cluster 2024-03-11T21:46:24.372637+0000 osd.7 (osd.7) 520 : cluster 0 3.d scrub starts 2024-03-11T21:46:26.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:25 smithi203 bash[23654]: cluster 2024-03-11T21:46:24.377502+0000 osd.7 (osd.7) 521 : cluster 0 3.d scrub ok 2024-03-11T21:46:26.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:25 smithi203 bash[23654]: audit 2024-03-11T21:46:24.728589+0000 mon.a (mon.0) 4102 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:46:26.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:25 smithi203 bash[23654]: cluster 2024-03-11T21:46:24.745690+0000 osd.1 (osd.1) 343 : cluster 0 6.1 scrub starts 2024-03-11T21:46:26.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:25 smithi203 bash[23654]: cluster 2024-03-11T21:46:24.748007+0000 osd.1 (osd.1) 344 : cluster 0 6.1 scrub ok 2024-03-11T21:46:26.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:46:26.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.098554+0000 osd.4 (osd.4) 523 : cluster 0 5.1c scrub starts 2024-03-11T21:46:26.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.101358+0000 osd.4 (osd.4) 524 : cluster 0 5.1c scrub ok 2024-03-11T21:46:26.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.218709+0000 osd.3 (osd.3) 549 : cluster 0 3.c scrub starts 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.223044+0000 osd.3 (osd.3) 550 : cluster 0 3.c scrub ok 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.381200+0000 osd.7 (osd.7) 522 : cluster 0 3.11 scrub starts 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.385598+0000 osd.7 (osd.7) 523 : cluster 0 3.11 scrub ok 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: audit 2024-03-11T21:46:25.697873+0000 mon.a (mon.0) 4103 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.703616+0000 mon.a (mon.0) 4104 : cluster 0 osdmap e983: 8 total, 8 up, 8 in 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[29284]: cluster 2024-03-11T21:46:25.744040+0000 mgr.y (mgr.14550) 901 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.098554+0000 osd.4 (osd.4) 523 : cluster 0 5.1c scrub starts 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.101358+0000 osd.4 (osd.4) 524 : cluster 0 5.1c scrub ok 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.218709+0000 osd.3 (osd.3) 549 : cluster 0 3.c scrub starts 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.223044+0000 osd.3 (osd.3) 550 : cluster 0 3.c scrub ok 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.381200+0000 osd.7 (osd.7) 522 : cluster 0 3.11 scrub starts 2024-03-11T21:46:26.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.385598+0000 osd.7 (osd.7) 523 : cluster 0 3.11 scrub ok 2024-03-11T21:46:26.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: audit 2024-03-11T21:46:25.697873+0000 mon.a (mon.0) 4103 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:26.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.703616+0000 mon.a (mon.0) 4104 : cluster 0 osdmap e983: 8 total, 8 up, 8 in 2024-03-11T21:46:26.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:26 smithi143 bash[21512]: cluster 2024-03-11T21:46:25.744040+0000 mgr.y (mgr.14550) 901 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.098554+0000 osd.4 (osd.4) 523 : cluster 0 5.1c scrub starts 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.101358+0000 osd.4 (osd.4) 524 : cluster 0 5.1c scrub ok 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.218709+0000 osd.3 (osd.3) 549 : cluster 0 3.c scrub starts 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.223044+0000 osd.3 (osd.3) 550 : cluster 0 3.c scrub ok 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.381200+0000 osd.7 (osd.7) 522 : cluster 0 3.11 scrub starts 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.385598+0000 osd.7 (osd.7) 523 : cluster 0 3.11 scrub ok 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: audit 2024-03-11T21:46:25.697873+0000 mon.a (mon.0) 4103 : audit 1 from='client.? 172.21.15.143:0/2984463447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.703616+0000 mon.a (mon.0) 4104 : cluster 0 osdmap e983: 8 total, 8 up, 8 in 2024-03-11T21:46:27.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:26 smithi203 bash[23654]: cluster 2024-03-11T21:46:25.744040+0000 mgr.y (mgr.14550) 901 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.104464+0000 osd.4 (osd.4) 525 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.108330+0000 osd.4 (osd.4) 526 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.189403+0000 osd.3 (osd.3) 551 : cluster 0 2.12 scrub starts 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.193240+0000 osd.3 (osd.3) 552 : cluster 0 2.12 scrub ok 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.229967+0000 osd.0 (osd.0) 542 : cluster 0 1.1 scrub starts 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.234594+0000 osd.0 (osd.0) 543 : cluster 0 1.1 scrub ok 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.403678+0000 osd.7 (osd.7) 524 : cluster 0 6.15 scrub starts 2024-03-11T21:46:28.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:27 smithi203 bash[23654]: cluster 2024-03-11T21:46:26.405917+0000 osd.7 (osd.7) 525 : cluster 0 6.15 scrub ok 2024-03-11T21:46:28.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.104464+0000 osd.4 (osd.4) 525 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:46:28.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.108330+0000 osd.4 (osd.4) 526 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:46:28.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.189403+0000 osd.3 (osd.3) 551 : cluster 0 2.12 scrub starts 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.193240+0000 osd.3 (osd.3) 552 : cluster 0 2.12 scrub ok 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.229967+0000 osd.0 (osd.0) 542 : cluster 0 1.1 scrub starts 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.234594+0000 osd.0 (osd.0) 543 : cluster 0 1.1 scrub ok 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.403678+0000 osd.7 (osd.7) 524 : cluster 0 6.15 scrub starts 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[29284]: cluster 2024-03-11T21:46:26.405917+0000 osd.7 (osd.7) 525 : cluster 0 6.15 scrub ok 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.104464+0000 osd.4 (osd.4) 525 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.108330+0000 osd.4 (osd.4) 526 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.189403+0000 osd.3 (osd.3) 551 : cluster 0 2.12 scrub starts 2024-03-11T21:46:28.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.193240+0000 osd.3 (osd.3) 552 : cluster 0 2.12 scrub ok 2024-03-11T21:46:28.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.229967+0000 osd.0 (osd.0) 542 : cluster 0 1.1 scrub starts 2024-03-11T21:46:28.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.234594+0000 osd.0 (osd.0) 543 : cluster 0 1.1 scrub ok 2024-03-11T21:46:28.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.403678+0000 osd.7 (osd.7) 524 : cluster 0 6.15 scrub starts 2024-03-11T21:46:28.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:27 smithi143 bash[21512]: cluster 2024-03-11T21:46:26.405917+0000 osd.7 (osd.7) 525 : cluster 0 6.15 scrub ok 2024-03-11T21:46:29.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:28 smithi203 bash[23654]: cluster 2024-03-11T21:46:27.226422+0000 osd.3 (osd.3) 553 : cluster 0 3.1e deep-scrub starts 2024-03-11T21:46:29.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:28 smithi203 bash[23654]: cluster 2024-03-11T21:46:27.236599+0000 osd.3 (osd.3) 554 : cluster 0 3.1e deep-scrub ok 2024-03-11T21:46:29.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:28 smithi203 bash[23654]: cluster 2024-03-11T21:46:27.437144+0000 osd.7 (osd.7) 526 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:46:29.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:28 smithi203 bash[23654]: cluster 2024-03-11T21:46:27.439705+0000 osd.7 (osd.7) 527 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:46:29.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:28 smithi203 bash[23654]: cluster 2024-03-11T21:46:27.744603+0000 mgr.y (mgr.14550) 902 : cluster 0 pgmap v1564: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[29284]: cluster 2024-03-11T21:46:27.226422+0000 osd.3 (osd.3) 553 : cluster 0 3.1e deep-scrub starts 2024-03-11T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[29284]: cluster 2024-03-11T21:46:27.236599+0000 osd.3 (osd.3) 554 : cluster 0 3.1e deep-scrub ok 2024-03-11T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[29284]: cluster 2024-03-11T21:46:27.437144+0000 osd.7 (osd.7) 526 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[29284]: cluster 2024-03-11T21:46:27.439705+0000 osd.7 (osd.7) 527 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[29284]: cluster 2024-03-11T21:46:27.744603+0000 mgr.y (mgr.14550) 902 : cluster 0 pgmap v1564: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[21512]: cluster 2024-03-11T21:46:27.226422+0000 osd.3 (osd.3) 553 : cluster 0 3.1e deep-scrub starts 2024-03-11T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[21512]: cluster 2024-03-11T21:46:27.236599+0000 osd.3 (osd.3) 554 : cluster 0 3.1e deep-scrub ok 2024-03-11T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[21512]: cluster 2024-03-11T21:46:27.437144+0000 osd.7 (osd.7) 526 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[21512]: cluster 2024-03-11T21:46:27.439705+0000 osd.7 (osd.7) 527 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:28 smithi143 bash[21512]: cluster 2024-03-11T21:46:27.744603+0000 mgr.y (mgr.14550) 902 : cluster 0 pgmap v1564: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:46:29.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:30.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:29 smithi203 bash[23654]: cluster 2024-03-11T21:46:28.308324+0000 osd.0 (osd.0) 544 : cluster 0 9.9 scrub starts 2024-03-11T21:46:30.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:29 smithi203 bash[23654]: cluster 2024-03-11T21:46:28.311385+0000 osd.0 (osd.0) 545 : cluster 0 9.9 scrub ok 2024-03-11T21:46:30.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:29 smithi143 bash[29284]: cluster 2024-03-11T21:46:28.308324+0000 osd.0 (osd.0) 544 : cluster 0 9.9 scrub starts 2024-03-11T21:46:30.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:29 smithi143 bash[29284]: cluster 2024-03-11T21:46:28.311385+0000 osd.0 (osd.0) 545 : cluster 0 9.9 scrub ok 2024-03-11T21:46:30.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:29 smithi143 bash[21512]: cluster 2024-03-11T21:46:28.308324+0000 osd.0 (osd.0) 544 : cluster 0 9.9 scrub starts 2024-03-11T21:46:30.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:29 smithi143 bash[21512]: cluster 2024-03-11T21:46:28.311385+0000 osd.0 (osd.0) 545 : cluster 0 9.9 scrub ok 2024-03-11T21:46:30.245 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: cluster 2024-03-11T21:46:29.229105+0000 osd.3 (osd.3) 555 : cluster 0 9.c deep-scrub starts 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: cluster 2024-03-11T21:46:29.232282+0000 osd.3 (osd.3) 556 : cluster 0 9.c deep-scrub ok 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: audit 2024-03-11T21:46:29.256371+0000 mgr.y (mgr.14550) 903 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: cluster 2024-03-11T21:46:29.305334+0000 osd.0 (osd.0) 546 : cluster 0 3.17 scrub starts 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: cluster 2024-03-11T21:46:29.307562+0000 osd.0 (osd.0) 547 : cluster 0 3.17 scrub ok 2024-03-11T21:46:31.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:30 smithi203 bash[23654]: cluster 2024-03-11T21:46:29.746633+0000 mgr.y (mgr.14550) 904 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: cluster 2024-03-11T21:46:29.229105+0000 osd.3 (osd.3) 555 : cluster 0 9.c deep-scrub starts 2024-03-11T21:46:31.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: cluster 2024-03-11T21:46:29.232282+0000 osd.3 (osd.3) 556 : cluster 0 9.c deep-scrub ok 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: audit 2024-03-11T21:46:29.256371+0000 mgr.y (mgr.14550) 903 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: cluster 2024-03-11T21:46:29.305334+0000 osd.0 (osd.0) 546 : cluster 0 3.17 scrub starts 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: cluster 2024-03-11T21:46:29.307562+0000 osd.0 (osd.0) 547 : cluster 0 3.17 scrub ok 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[29284]: cluster 2024-03-11T21:46:29.746633+0000 mgr.y (mgr.14550) 904 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: cluster 2024-03-11T21:46:29.229105+0000 osd.3 (osd.3) 555 : cluster 0 9.c deep-scrub starts 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: cluster 2024-03-11T21:46:29.232282+0000 osd.3 (osd.3) 556 : cluster 0 9.c deep-scrub ok 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: audit 2024-03-11T21:46:29.256371+0000 mgr.y (mgr.14550) 903 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: cluster 2024-03-11T21:46:29.305334+0000 osd.0 (osd.0) 546 : cluster 0 3.17 scrub starts 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: cluster 2024-03-11T21:46:29.307562+0000 osd.0 (osd.0) 547 : cluster 0 3.17 scrub ok 2024-03-11T21:46:31.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:30 smithi143 bash[21512]: cluster 2024-03-11T21:46:29.746633+0000 mgr.y (mgr.14550) 904 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:31.266 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.015422+0000 osd.4 (osd.4) 527 : cluster 0 5.a deep-scrub starts 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.018117+0000 osd.4 (osd.4) 528 : cluster 0 5.a deep-scrub ok 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.250304+0000 osd.3 (osd.3) 557 : cluster 0 5.14 scrub starts 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.253358+0000 osd.3 (osd.3) 558 : cluster 0 5.14 scrub ok 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.753945+0000 osd.1 (osd.1) 345 : cluster 0 5.19 scrub starts 2024-03-11T21:46:32.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:31 smithi203 bash[23654]: cluster 2024-03-11T21:46:30.755827+0000 osd.1 (osd.1) 346 : cluster 0 5.19 scrub ok 2024-03-11T21:46:32.090 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.015422+0000 osd.4 (osd.4) 527 : cluster 0 5.a deep-scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.018117+0000 osd.4 (osd.4) 528 : cluster 0 5.a deep-scrub ok 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.250304+0000 osd.3 (osd.3) 557 : cluster 0 5.14 scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.253358+0000 osd.3 (osd.3) 558 : cluster 0 5.14 scrub ok 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.753945+0000 osd.1 (osd.1) 345 : cluster 0 5.19 scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[29284]: cluster 2024-03-11T21:46:30.755827+0000 osd.1 (osd.1) 346 : cluster 0 5.19 scrub ok 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.015422+0000 osd.4 (osd.4) 527 : cluster 0 5.a deep-scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.018117+0000 osd.4 (osd.4) 528 : cluster 0 5.a deep-scrub ok 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.250304+0000 osd.3 (osd.3) 557 : cluster 0 5.14 scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.253358+0000 osd.3 (osd.3) 558 : cluster 0 5.14 scrub ok 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.753945+0000 osd.1 (osd.1) 345 : cluster 0 5.19 scrub starts 2024-03-11T21:46:32.091 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:31 smithi143 bash[21512]: cluster 2024-03-11T21:46:30.755827+0000 osd.1 (osd.1) 346 : cluster 0 5.19 scrub ok 2024-03-11T21:46:32.405 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:32.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[29284]: cluster 2024-03-11T21:46:31.304575+0000 osd.0 (osd.0) 548 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:46:32.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[29284]: cluster 2024-03-11T21:46:31.308108+0000 osd.0 (osd.0) 549 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:46:32.947 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[29284]: cluster 2024-03-11T21:46:31.486637+0000 osd.7 (osd.7) 528 : cluster 0 6.12 scrub starts 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[29284]: cluster 2024-03-11T21:46:31.488995+0000 osd.7 (osd.7) 529 : cluster 0 6.12 scrub ok 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[29284]: cluster 2024-03-11T21:46:31.747920+0000 mgr.y (mgr.14550) 905 : cluster 0 pgmap v1566: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21512]: cluster 2024-03-11T21:46:31.304575+0000 osd.0 (osd.0) 548 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21512]: cluster 2024-03-11T21:46:31.308108+0000 osd.0 (osd.0) 549 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21512]: cluster 2024-03-11T21:46:31.486637+0000 osd.7 (osd.7) 528 : cluster 0 6.12 scrub starts 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21512]: cluster 2024-03-11T21:46:31.488995+0000 osd.7 (osd.7) 529 : cluster 0 6.12 scrub ok 2024-03-11T21:46:32.948 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:32 smithi143 bash[21512]: cluster 2024-03-11T21:46:31.747920+0000 mgr.y (mgr.14550) 905 : cluster 0 pgmap v1566: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:46:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:32 smithi203 bash[23654]: cluster 2024-03-11T21:46:31.304575+0000 osd.0 (osd.0) 548 : cluster 0 3.1 deep-scrub starts 2024-03-11T21:46:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:32 smithi203 bash[23654]: cluster 2024-03-11T21:46:31.308108+0000 osd.0 (osd.0) 549 : cluster 0 3.1 deep-scrub ok 2024-03-11T21:46:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:32 smithi203 bash[23654]: cluster 2024-03-11T21:46:31.486637+0000 osd.7 (osd.7) 528 : cluster 0 6.12 scrub starts 2024-03-11T21:46:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:32 smithi203 bash[23654]: cluster 2024-03-11T21:46:31.488995+0000 osd.7 (osd.7) 529 : cluster 0 6.12 scrub ok 2024-03-11T21:46:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:32 smithi203 bash[23654]: cluster 2024-03-11T21:46:31.747920+0000 mgr.y (mgr.14550) 905 : cluster 0 pgmap v1566: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:46:34.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:33 smithi203 bash[23654]: cluster 2024-03-11T21:46:32.036957+0000 osd.4 (osd.4) 529 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:46:34.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:33 smithi203 bash[23654]: cluster 2024-03-11T21:46:32.039201+0000 osd.4 (osd.4) 530 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:46:34.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:33 smithi143 bash[21512]: cluster 2024-03-11T21:46:32.036957+0000 osd.4 (osd.4) 529 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:46:34.079 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:33 smithi143 bash[21512]: cluster 2024-03-11T21:46:32.039201+0000 osd.4 (osd.4) 530 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:46:34.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:33 smithi143 bash[29284]: cluster 2024-03-11T21:46:32.036957+0000 osd.4 (osd.4) 529 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:46:34.079 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:33 smithi143 bash[29284]: cluster 2024-03-11T21:46:32.039201+0000 osd.4 (osd.4) 530 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.191835+0000 osd.3 (osd.3) 559 : cluster 0 1.4 scrub starts 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.194859+0000 osd.3 (osd.3) 560 : cluster 0 1.4 scrub ok 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.265452+0000 osd.0 (osd.0) 550 : cluster 0 5.1e scrub starts 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.267701+0000 osd.0 (osd.0) 551 : cluster 0 5.1e scrub ok 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.749796+0000 mgr.y (mgr.14550) 906 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.829113+0000 osd.1 (osd.1) 347 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:46:35.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[29284]: cluster 2024-03-11T21:46:33.835387+0000 osd.1 (osd.1) 348 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.191835+0000 osd.3 (osd.3) 559 : cluster 0 1.4 scrub starts 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.194859+0000 osd.3 (osd.3) 560 : cluster 0 1.4 scrub ok 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.265452+0000 osd.0 (osd.0) 550 : cluster 0 5.1e scrub starts 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.267701+0000 osd.0 (osd.0) 551 : cluster 0 5.1e scrub ok 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.749796+0000 mgr.y (mgr.14550) 906 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.829113+0000 osd.1 (osd.1) 347 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:46:35.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:34 smithi143 bash[21512]: cluster 2024-03-11T21:46:33.835387+0000 osd.1 (osd.1) 348 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.191835+0000 osd.3 (osd.3) 559 : cluster 0 1.4 scrub starts 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.194859+0000 osd.3 (osd.3) 560 : cluster 0 1.4 scrub ok 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.265452+0000 osd.0 (osd.0) 550 : cluster 0 5.1e scrub starts 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.267701+0000 osd.0 (osd.0) 551 : cluster 0 5.1e scrub ok 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.749796+0000 mgr.y (mgr.14550) 906 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.829113+0000 osd.1 (osd.1) 347 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:46:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:34 smithi203 bash[23654]: cluster 2024-03-11T21:46:33.835387+0000 osd.1 (osd.1) 348 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:46:36.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[29284]: cluster 2024-03-11T21:46:34.465639+0000 osd.7 (osd.7) 530 : cluster 0 3.16 scrub starts 2024-03-11T21:46:36.195 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[29284]: cluster 2024-03-11T21:46:34.469644+0000 osd.7 (osd.7) 531 : cluster 0 3.16 scrub ok 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[29284]: cluster 2024-03-11T21:46:34.866208+0000 osd.1 (osd.1) 349 : cluster 0 5.12 scrub starts 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[29284]: cluster 2024-03-11T21:46:34.867878+0000 osd.1 (osd.1) 350 : cluster 0 5.12 scrub ok 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[21512]: cluster 2024-03-11T21:46:34.465639+0000 osd.7 (osd.7) 530 : cluster 0 3.16 scrub starts 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[21512]: cluster 2024-03-11T21:46:34.469644+0000 osd.7 (osd.7) 531 : cluster 0 3.16 scrub ok 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[21512]: cluster 2024-03-11T21:46:34.866208+0000 osd.1 (osd.1) 349 : cluster 0 5.12 scrub starts 2024-03-11T21:46:36.196 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:35 smithi143 bash[21512]: cluster 2024-03-11T21:46:34.867878+0000 osd.1 (osd.1) 350 : cluster 0 5.12 scrub ok 2024-03-11T21:46:36.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:35 smithi203 bash[23654]: cluster 2024-03-11T21:46:34.465639+0000 osd.7 (osd.7) 530 : cluster 0 3.16 scrub starts 2024-03-11T21:46:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:35 smithi203 bash[23654]: cluster 2024-03-11T21:46:34.469644+0000 osd.7 (osd.7) 531 : cluster 0 3.16 scrub ok 2024-03-11T21:46:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:35 smithi203 bash[23654]: cluster 2024-03-11T21:46:34.866208+0000 osd.1 (osd.1) 349 : cluster 0 5.12 scrub starts 2024-03-11T21:46:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:35 smithi203 bash[23654]: cluster 2024-03-11T21:46:34.867878+0000 osd.1 (osd.1) 350 : cluster 0 5.12 scrub ok 2024-03-11T21:46:36.394 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:36.394 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:46:36.394 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:46:36.394 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:46:36.395 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:36.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/1647685883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 msgr2=0x7fe6a806a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:36.777 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- 172.21.15.143:0/1647685883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a806a9a0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009990 tx=0x7fe69c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/1647685883 shutdown_connections 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- 172.21.15.143:0/1647685883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a806aee0 0x7fe6a810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- 172.21.15.143:0/1647685883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a8103df0 0x7fe6a81041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- 172.21.15.143:0/1647685883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a806a9a0 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/1647685883 >> 172.21.15.143:0/1647685883 conn(0x7fe6a8076d80 msgr2=0x7fe6a8077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/1647685883 shutdown_connections 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/1647685883 wait complete. 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 librados: starting msgr at 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 librados: starting objecter 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 Processor -- start 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- start start 2024-03-11T21:46:36.778 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 librados: setting wanted keys 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 librados: calling monclient init 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 0x7fe6a81013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 0x7fe6a8101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8111800 con 0x7fe6a8103df0 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8111680 con 0x7fe6a806aee0 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8111980 con 0x7fe6a806a520 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 0x7fe6a8101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:46598/0 (socket says 172.21.15.143:46598) 2024-03-11T21:46:36.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a659c700 1 -- 172.21.15.143:0/2601001199 learned_addr learned my addr 172.21.15.143:0/2601001199 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:36.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a5d9b700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 0x7fe6a81013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:36.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 -- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 msgr2=0x7fe6a8100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:36.781 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 -- 172.21.15.143:0/2601001199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 msgr2=0x7fe6a81013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 0x7fe6a81013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 -- 172.21.15.143:0/2601001199 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a81a6270 con 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a659c700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a5d9b700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 0x7fe6a81013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6a6d9d700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 0x7fe6a8101930 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000efc0 tx=0x7fe6a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 10 client.?.objecter ms_handle_connect 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00092e0 con 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a81a6500 con 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe6a0004bb0 con 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0004020 con 0x7fe6a8103df0 2024-03-11T21:46:36.782 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 10 client.31131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a81a6a40 con 0x7fe6a8103df0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 librados: init done 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a804faa0 con 0x7fe6a8103df0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.791+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe6a001b030 con 0x7fe6a8103df0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 0x7fe690071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(983..983 src has 933..983) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fe6a008e3c0 con 0x7fe6a8103df0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 10 client.31131.objecter ms_dispatch 0x7fe6a8001550 osd_map(983..983 src has 933..983) v4 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 3 client.31131.objecter handle_osd_map got epochs [983,983] > 0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 3 client.31131.objecter handle_osd_map decoding full epoch 983 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6a659c700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 0x7fe690071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 20 client.31131.objecter dump_active .. 0 homeless 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6a659c700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 0x7fe690071ce0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fe698009ea0 tx=0x7fe698009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:36.783 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.795+0000 7fe6977fe700 10 client.31131.objecter ms_handle_connect 0x7fe69006f820 2024-03-11T21:46:36.786 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:36.799+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe6a0022020 con 0x7fe6a8103df0 2024-03-11T21:46:37.009 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.023+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fe6a81a6ca0 con 0x7fe6a8103df0 2024-03-11T21:46:37.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[21512]: cluster 2024-03-11T21:46:35.240585+0000 osd.3 (osd.3) 561 : cluster 0 1.c scrub starts 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[21512]: cluster 2024-03-11T21:46:35.243415+0000 osd.3 (osd.3) 562 : cluster 0 1.c scrub ok 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[21512]: cluster 2024-03-11T21:46:35.750903+0000 mgr.y (mgr.14550) 907 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[21512]: cluster 2024-03-11T21:46:35.863217+0000 osd.1 (osd.1) 351 : cluster 0 9.a scrub starts 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[21512]: cluster 2024-03-11T21:46:35.865904+0000 osd.1 (osd.1) 352 : cluster 0 9.a scrub ok 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[29284]: cluster 2024-03-11T21:46:35.240585+0000 osd.3 (osd.3) 561 : cluster 0 1.c scrub starts 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[29284]: cluster 2024-03-11T21:46:35.243415+0000 osd.3 (osd.3) 562 : cluster 0 1.c scrub ok 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[29284]: cluster 2024-03-11T21:46:35.750903+0000 mgr.y (mgr.14550) 907 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[29284]: cluster 2024-03-11T21:46:35.863217+0000 osd.1 (osd.1) 351 : cluster 0 9.a scrub starts 2024-03-11T21:46:37.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:36 smithi143 bash[29284]: cluster 2024-03-11T21:46:35.865904+0000 osd.1 (osd.1) 352 : cluster 0 9.a scrub ok 2024-03-11T21:46:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:36 smithi203 bash[23654]: cluster 2024-03-11T21:46:35.240585+0000 osd.3 (osd.3) 561 : cluster 0 1.c scrub starts 2024-03-11T21:46:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:36 smithi203 bash[23654]: cluster 2024-03-11T21:46:35.243415+0000 osd.3 (osd.3) 562 : cluster 0 1.c scrub ok 2024-03-11T21:46:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:36 smithi203 bash[23654]: cluster 2024-03-11T21:46:35.750903+0000 mgr.y (mgr.14550) 907 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:46:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:36 smithi203 bash[23654]: cluster 2024-03-11T21:46:35.863217+0000 osd.1 (osd.1) 351 : cluster 0 9.a scrub starts 2024-03-11T21:46:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:36 smithi203 bash[23654]: cluster 2024-03-11T21:46:35.865904+0000 osd.1 (osd.1) 352 : cluster 0 9.a scrub ok 2024-03-11T21:46:37.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:46:37.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.831+0000 7fe6977fe700 1 -- 172.21.15.143:0/2601001199 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v984) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe6a0010040 con 0x7fe6a8103df0 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 10 librados: watch_flush enter 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 10 librados: watch_flush exit 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 20 client.31131.objecter shutdown clearing up homeless session... 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 10 client.31131.objecter successfully canceled tick 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 msgr2=0x7fe690071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 0x7fe690071ce0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fe698009ea0 tx=0x7fe698009290 comp rx=0 tx=0).stop 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 msgr2=0x7fe6a8101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 0x7fe6a8101930 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000efc0 tx=0x7fe6a000c3a0 comp rx=0 tx=0).stop 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 shutdown_connections 2024-03-11T21:46:37.829 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe69006f820 0x7fe690071ce0 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6a8103df0 0x7fe6a8101930 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6a806a520 0x7fe6a8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 --2- 172.21.15.143:0/2601001199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a806aee0 0x7fe6a81013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 >> 172.21.15.143:0/2601001199 conn(0x7fe6a8076d80 msgr2=0x7fe6a8106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 shutdown_connections 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.835+0000 7fe6acadf700 1 -- 172.21.15.143:0/2601001199 wait complete. 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:37.839+0000 7fe6acadf700 1 librados: shutdown 2024-03-11T21:46:37.830 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[29284]: cluster 2024-03-11T21:46:36.328034+0000 osd.0 (osd.0) 552 : cluster 0 6.9 scrub starts 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[29284]: cluster 2024-03-11T21:46:36.329521+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub ok 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[29284]: cluster 2024-03-11T21:46:36.896767+0000 osd.1 (osd.1) 353 : cluster 0 5.1d scrub starts 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[29284]: cluster 2024-03-11T21:46:36.898811+0000 osd.1 (osd.1) 354 : cluster 0 5.1d scrub ok 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[29284]: audit 2024-03-11T21:46:37.025130+0000 mon.a (mon.0) 4105 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:38.143 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[21512]: cluster 2024-03-11T21:46:36.328034+0000 osd.0 (osd.0) 552 : cluster 0 6.9 scrub starts 2024-03-11T21:46:38.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[21512]: cluster 2024-03-11T21:46:36.329521+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub ok 2024-03-11T21:46:38.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[21512]: cluster 2024-03-11T21:46:36.896767+0000 osd.1 (osd.1) 353 : cluster 0 5.1d scrub starts 2024-03-11T21:46:38.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[21512]: cluster 2024-03-11T21:46:36.898811+0000 osd.1 (osd.1) 354 : cluster 0 5.1d scrub ok 2024-03-11T21:46:38.144 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:37 smithi143 bash[21512]: audit 2024-03-11T21:46:37.025130+0000 mon.a (mon.0) 4105 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:38.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:37 smithi203 bash[23654]: cluster 2024-03-11T21:46:36.328034+0000 osd.0 (osd.0) 552 : cluster 0 6.9 scrub starts 2024-03-11T21:46:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:37 smithi203 bash[23654]: cluster 2024-03-11T21:46:36.329521+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub ok 2024-03-11T21:46:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:37 smithi203 bash[23654]: cluster 2024-03-11T21:46:36.896767+0000 osd.1 (osd.1) 353 : cluster 0 5.1d scrub starts 2024-03-11T21:46:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:37 smithi203 bash[23654]: cluster 2024-03-11T21:46:36.898811+0000 osd.1 (osd.1) 354 : cluster 0 5.1d scrub ok 2024-03-11T21:46:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:37 smithi203 bash[23654]: audit 2024-03-11T21:46:37.025130+0000 mon.a (mon.0) 4105 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:38.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.407419+0000 osd.7 (osd.7) 532 : cluster 0 9.3 scrub starts 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.412355+0000 osd.7 (osd.7) 533 : cluster 0 9.3 scrub ok 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.751681+0000 mgr.y (mgr.14550) 908 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.802222+0000 osd.6 (osd.6) 453 : cluster 0 2.16 scrub starts 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.804412+0000 osd.6 (osd.6) 454 : cluster 0 2.16 scrub ok 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: audit 2024-03-11T21:46:37.834108+0000 mon.a (mon.0) 4106 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.842277+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e984: 8 total, 8 up, 8 in 2024-03-11T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.876735+0000 osd.1 (osd.1) 355 : cluster 0 3.0 scrub starts 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: cluster 2024-03-11T21:46:37.880259+0000 osd.1 (osd.1) 356 : cluster 0 3.0 scrub ok 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[21512]: audit 2024-03-11T21:46:37.902087+0000 mon.c (mon.2) 1397 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.407419+0000 osd.7 (osd.7) 532 : cluster 0 9.3 scrub starts 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.412355+0000 osd.7 (osd.7) 533 : cluster 0 9.3 scrub ok 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.751681+0000 mgr.y (mgr.14550) 908 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.802222+0000 osd.6 (osd.6) 453 : cluster 0 2.16 scrub starts 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.804412+0000 osd.6 (osd.6) 454 : cluster 0 2.16 scrub ok 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: audit 2024-03-11T21:46:37.834108+0000 mon.a (mon.0) 4106 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.842277+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e984: 8 total, 8 up, 8 in 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.876735+0000 osd.1 (osd.1) 355 : cluster 0 3.0 scrub starts 2024-03-11T21:46:39.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: cluster 2024-03-11T21:46:37.880259+0000 osd.1 (osd.1) 356 : cluster 0 3.0 scrub ok 2024-03-11T21:46:39.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:38 smithi143 bash[29284]: audit 2024-03-11T21:46:37.902087+0000 mon.c (mon.2) 1397 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.251 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.407419+0000 osd.7 (osd.7) 532 : cluster 0 9.3 scrub starts 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.412355+0000 osd.7 (osd.7) 533 : cluster 0 9.3 scrub ok 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.751681+0000 mgr.y (mgr.14550) 908 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.802222+0000 osd.6 (osd.6) 453 : cluster 0 2.16 scrub starts 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.804412+0000 osd.6 (osd.6) 454 : cluster 0 2.16 scrub ok 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: audit 2024-03-11T21:46:37.834108+0000 mon.a (mon.0) 4106 : audit 1 from='client.? 172.21.15.143:0/2601001199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.842277+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e984: 8 total, 8 up, 8 in 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.876735+0000 osd.1 (osd.1) 355 : cluster 0 3.0 scrub starts 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: cluster 2024-03-11T21:46:37.880259+0000 osd.1 (osd.1) 356 : cluster 0 3.0 scrub ok 2024-03-11T21:46:39.252 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:38 smithi203 bash[23654]: audit 2024-03-11T21:46:37.902087+0000 mon.c (mon.2) 1397 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:40.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[21512]: cluster 2024-03-11T21:46:38.408904+0000 osd.0 (osd.0) 554 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[21512]: cluster 2024-03-11T21:46:38.411224+0000 osd.7 (osd.7) 534 : cluster 0 9.13 scrub starts 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[21512]: cluster 2024-03-11T21:46:38.414625+0000 osd.0 (osd.0) 555 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[21512]: cluster 2024-03-11T21:46:38.419132+0000 osd.7 (osd.7) 535 : cluster 0 9.13 scrub ok 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[29284]: cluster 2024-03-11T21:46:38.408904+0000 osd.0 (osd.0) 554 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[29284]: cluster 2024-03-11T21:46:38.411224+0000 osd.7 (osd.7) 534 : cluster 0 9.13 scrub starts 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[29284]: cluster 2024-03-11T21:46:38.414625+0000 osd.0 (osd.0) 555 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:46:40.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:39 smithi143 bash[29284]: cluster 2024-03-11T21:46:38.419132+0000 osd.7 (osd.7) 535 : cluster 0 9.13 scrub ok 2024-03-11T21:46:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:39 smithi203 bash[23654]: cluster 2024-03-11T21:46:38.408904+0000 osd.0 (osd.0) 554 : cluster 0 9.1d deep-scrub starts 2024-03-11T21:46:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:39 smithi203 bash[23654]: cluster 2024-03-11T21:46:38.411224+0000 osd.7 (osd.7) 534 : cluster 0 9.13 scrub starts 2024-03-11T21:46:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:39 smithi203 bash[23654]: cluster 2024-03-11T21:46:38.414625+0000 osd.0 (osd.0) 555 : cluster 0 9.1d deep-scrub ok 2024-03-11T21:46:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:39 smithi203 bash[23654]: cluster 2024-03-11T21:46:38.419132+0000 osd.7 (osd.7) 535 : cluster 0 9.13 scrub ok 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[21512]: audit 2024-03-11T21:46:39.265458+0000 mgr.y (mgr.14550) 909 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[21512]: cluster 2024-03-11T21:46:39.413293+0000 osd.7 (osd.7) 536 : cluster 0 6.5 scrub starts 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[21512]: cluster 2024-03-11T21:46:39.415731+0000 osd.7 (osd.7) 537 : cluster 0 6.5 scrub ok 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[21512]: cluster 2024-03-11T21:46:39.753548+0000 mgr.y (mgr.14550) 910 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[29284]: audit 2024-03-11T21:46:39.265458+0000 mgr.y (mgr.14550) 909 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[29284]: cluster 2024-03-11T21:46:39.413293+0000 osd.7 (osd.7) 536 : cluster 0 6.5 scrub starts 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[29284]: cluster 2024-03-11T21:46:39.415731+0000 osd.7 (osd.7) 537 : cluster 0 6.5 scrub ok 2024-03-11T21:46:41.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:40 smithi143 bash[29284]: cluster 2024-03-11T21:46:39.753548+0000 mgr.y (mgr.14550) 910 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:40 smithi203 bash[23654]: audit 2024-03-11T21:46:39.265458+0000 mgr.y (mgr.14550) 909 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:40 smithi203 bash[23654]: cluster 2024-03-11T21:46:39.413293+0000 osd.7 (osd.7) 536 : cluster 0 6.5 scrub starts 2024-03-11T21:46:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:40 smithi203 bash[23654]: cluster 2024-03-11T21:46:39.415731+0000 osd.7 (osd.7) 537 : cluster 0 6.5 scrub ok 2024-03-11T21:46:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:40 smithi203 bash[23654]: cluster 2024-03-11T21:46:39.753548+0000 mgr.y (mgr.14550) 910 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:42.197 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.371680+0000 osd.2 (osd.2) 471 : cluster 0 1.a deep-scrub starts 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.373241+0000 osd.2 (osd.2) 472 : cluster 0 1.a deep-scrub ok 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.463636+0000 osd.7 (osd.7) 538 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.469147+0000 osd.7 (osd.7) 539 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.948436+0000 osd.1 (osd.1) 357 : cluster 0 2.d scrub starts 2024-03-11T21:46:42.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[29284]: cluster 2024-03-11T21:46:40.950993+0000 osd.1 (osd.1) 358 : cluster 0 2.d scrub ok 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.371680+0000 osd.2 (osd.2) 471 : cluster 0 1.a deep-scrub starts 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.373241+0000 osd.2 (osd.2) 472 : cluster 0 1.a deep-scrub ok 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.463636+0000 osd.7 (osd.7) 538 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.469147+0000 osd.7 (osd.7) 539 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.948436+0000 osd.1 (osd.1) 357 : cluster 0 2.d scrub starts 2024-03-11T21:46:42.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:41 smithi143 bash[21512]: cluster 2024-03-11T21:46:40.950993+0000 osd.1 (osd.1) 358 : cluster 0 2.d scrub ok 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.371680+0000 osd.2 (osd.2) 471 : cluster 0 1.a deep-scrub starts 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.373241+0000 osd.2 (osd.2) 472 : cluster 0 1.a deep-scrub ok 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.463636+0000 osd.7 (osd.7) 538 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.469147+0000 osd.7 (osd.7) 539 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.948436+0000 osd.1 (osd.1) 357 : cluster 0 2.d scrub starts 2024-03-11T21:46:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:41 smithi203 bash[23654]: cluster 2024-03-11T21:46:40.950993+0000 osd.1 (osd.1) 358 : cluster 0 2.d scrub ok 2024-03-11T21:46:42.365 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:43.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[21512]: cluster 2024-03-11T21:46:41.446338+0000 osd.7 (osd.7) 540 : cluster 0 9.15 scrub starts 2024-03-11T21:46:43.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[21512]: cluster 2024-03-11T21:46:41.450456+0000 osd.7 (osd.7) 541 : cluster 0 9.15 scrub ok 2024-03-11T21:46:43.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[21512]: cluster 2024-03-11T21:46:41.754613+0000 mgr.y (mgr.14550) 911 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[29284]: cluster 2024-03-11T21:46:41.446338+0000 osd.7 (osd.7) 540 : cluster 0 9.15 scrub starts 2024-03-11T21:46:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[29284]: cluster 2024-03-11T21:46:41.450456+0000 osd.7 (osd.7) 541 : cluster 0 9.15 scrub ok 2024-03-11T21:46:43.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:42 smithi143 bash[29284]: cluster 2024-03-11T21:46:41.754613+0000 mgr.y (mgr.14550) 911 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:42 smithi203 bash[23654]: cluster 2024-03-11T21:46:41.446338+0000 osd.7 (osd.7) 540 : cluster 0 9.15 scrub starts 2024-03-11T21:46:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:42 smithi203 bash[23654]: cluster 2024-03-11T21:46:41.450456+0000 osd.7 (osd.7) 541 : cluster 0 9.15 scrub ok 2024-03-11T21:46:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:42 smithi203 bash[23654]: cluster 2024-03-11T21:46:41.754613+0000 mgr.y (mgr.14550) 911 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:43.385 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[21512]: cluster 2024-03-11T21:46:42.226808+0000 osd.3 (osd.3) 563 : cluster 0 6.11 scrub starts 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[21512]: cluster 2024-03-11T21:46:42.229109+0000 osd.3 (osd.3) 564 : cluster 0 6.11 scrub ok 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[21512]: cluster 2024-03-11T21:46:42.432803+0000 osd.7 (osd.7) 542 : cluster 0 2.1d scrub starts 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[21512]: cluster 2024-03-11T21:46:42.435113+0000 osd.7 (osd.7) 543 : cluster 0 2.1d scrub ok 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[29284]: cluster 2024-03-11T21:46:42.226808+0000 osd.3 (osd.3) 563 : cluster 0 6.11 scrub starts 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[29284]: cluster 2024-03-11T21:46:42.229109+0000 osd.3 (osd.3) 564 : cluster 0 6.11 scrub ok 2024-03-11T21:46:44.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[29284]: cluster 2024-03-11T21:46:42.432803+0000 osd.7 (osd.7) 542 : cluster 0 2.1d scrub starts 2024-03-11T21:46:44.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:43 smithi143 bash[29284]: cluster 2024-03-11T21:46:42.435113+0000 osd.7 (osd.7) 543 : cluster 0 2.1d scrub ok 2024-03-11T21:46:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:43 smithi203 bash[23654]: cluster 2024-03-11T21:46:42.226808+0000 osd.3 (osd.3) 563 : cluster 0 6.11 scrub starts 2024-03-11T21:46:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:43 smithi203 bash[23654]: cluster 2024-03-11T21:46:42.229109+0000 osd.3 (osd.3) 564 : cluster 0 6.11 scrub ok 2024-03-11T21:46:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:43 smithi203 bash[23654]: cluster 2024-03-11T21:46:42.432803+0000 osd.7 (osd.7) 542 : cluster 0 2.1d scrub starts 2024-03-11T21:46:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:43 smithi203 bash[23654]: cluster 2024-03-11T21:46:42.435113+0000 osd.7 (osd.7) 543 : cluster 0 2.1d scrub ok 2024-03-11T21:46:45.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[29284]: cluster 2024-03-11T21:46:43.427260+0000 osd.0 (osd.0) 556 : cluster 0 1.11 scrub starts 2024-03-11T21:46:45.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[29284]: cluster 2024-03-11T21:46:43.429096+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub ok 2024-03-11T21:46:45.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[29284]: cluster 2024-03-11T21:46:43.756436+0000 mgr.y (mgr.14550) 912 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:45.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[21512]: cluster 2024-03-11T21:46:43.427260+0000 osd.0 (osd.0) 556 : cluster 0 1.11 scrub starts 2024-03-11T21:46:45.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[21512]: cluster 2024-03-11T21:46:43.429096+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub ok 2024-03-11T21:46:45.279 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:44 smithi143 bash[21512]: cluster 2024-03-11T21:46:43.756436+0000 mgr.y (mgr.14550) 912 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:45.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:44 smithi203 bash[23654]: cluster 2024-03-11T21:46:43.427260+0000 osd.0 (osd.0) 556 : cluster 0 1.11 scrub starts 2024-03-11T21:46:45.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:44 smithi203 bash[23654]: cluster 2024-03-11T21:46:43.429096+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub ok 2024-03-11T21:46:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:44 smithi203 bash[23654]: cluster 2024-03-11T21:46:43.756436+0000 mgr.y (mgr.14550) 912 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:46.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:45 smithi143 bash[29284]: cluster 2024-03-11T21:46:44.470057+0000 osd.0 (osd.0) 558 : cluster 0 3.1f scrub starts 2024-03-11T21:46:46.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:45 smithi143 bash[29284]: cluster 2024-03-11T21:46:44.472532+0000 osd.0 (osd.0) 559 : cluster 0 3.1f scrub ok 2024-03-11T21:46:46.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:45 smithi143 bash[21512]: cluster 2024-03-11T21:46:44.470057+0000 osd.0 (osd.0) 558 : cluster 0 3.1f scrub starts 2024-03-11T21:46:46.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:45 smithi143 bash[21512]: cluster 2024-03-11T21:46:44.472532+0000 osd.0 (osd.0) 559 : cluster 0 3.1f scrub ok 2024-03-11T21:46:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:45 smithi203 bash[23654]: cluster 2024-03-11T21:46:44.470057+0000 osd.0 (osd.0) 558 : cluster 0 3.1f scrub starts 2024-03-11T21:46:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:45 smithi203 bash[23654]: cluster 2024-03-11T21:46:44.472532+0000 osd.0 (osd.0) 559 : cluster 0 3.1f scrub ok 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.288086+0000 osd.3 (osd.3) 565 : cluster 0 3.2 scrub starts 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.293087+0000 osd.3 (osd.3) 566 : cluster 0 3.2 scrub ok 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.432925+0000 osd.7 (osd.7) 544 : cluster 0 3.e scrub starts 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.437066+0000 osd.7 (osd.7) 545 : cluster 0 3.e scrub ok 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.757553+0000 mgr.y (mgr.14550) 913 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:47.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.878257+0000 osd.1 (osd.1) 359 : cluster 0 3.4 scrub starts 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[21512]: cluster 2024-03-11T21:46:45.884841+0000 osd.1 (osd.1) 360 : cluster 0 3.4 scrub ok 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.288086+0000 osd.3 (osd.3) 565 : cluster 0 3.2 scrub starts 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.293087+0000 osd.3 (osd.3) 566 : cluster 0 3.2 scrub ok 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.432925+0000 osd.7 (osd.7) 544 : cluster 0 3.e scrub starts 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.437066+0000 osd.7 (osd.7) 545 : cluster 0 3.e scrub ok 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.757553+0000 mgr.y (mgr.14550) 913 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.878257+0000 osd.1 (osd.1) 359 : cluster 0 3.4 scrub starts 2024-03-11T21:46:47.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:46 smithi143 bash[29284]: cluster 2024-03-11T21:46:45.884841+0000 osd.1 (osd.1) 360 : cluster 0 3.4 scrub ok 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.288086+0000 osd.3 (osd.3) 565 : cluster 0 3.2 scrub starts 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.293087+0000 osd.3 (osd.3) 566 : cluster 0 3.2 scrub ok 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.432925+0000 osd.7 (osd.7) 544 : cluster 0 3.e scrub starts 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.437066+0000 osd.7 (osd.7) 545 : cluster 0 3.e scrub ok 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.757553+0000 mgr.y (mgr.14550) 913 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.878257+0000 osd.1 (osd.1) 359 : cluster 0 3.4 scrub starts 2024-03-11T21:46:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:46 smithi203 bash[23654]: cluster 2024-03-11T21:46:45.884841+0000 osd.1 (osd.1) 360 : cluster 0 3.4 scrub ok 2024-03-11T21:46:48.197 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[29284]: cluster 2024-03-11T21:46:46.250417+0000 osd.3 (osd.3) 567 : cluster 0 6.b scrub starts 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[29284]: cluster 2024-03-11T21:46:46.252878+0000 osd.3 (osd.3) 568 : cluster 0 6.b scrub ok 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[29284]: cluster 2024-03-11T21:46:46.999161+0000 osd.6 (osd.6) 455 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[29284]: cluster 2024-03-11T21:46:46.999541+0000 osd.6 (osd.6) 456 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[21512]: cluster 2024-03-11T21:46:46.250417+0000 osd.3 (osd.3) 567 : cluster 0 6.b scrub starts 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[21512]: cluster 2024-03-11T21:46:46.252878+0000 osd.3 (osd.3) 568 : cluster 0 6.b scrub ok 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[21512]: cluster 2024-03-11T21:46:46.999161+0000 osd.6 (osd.6) 455 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:48.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:47 smithi143 bash[21512]: cluster 2024-03-11T21:46:46.999541+0000 osd.6 (osd.6) 456 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:109 315.0 315:02cb0792:test-rados-api-smithi143-71773-115::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e700)", 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:47.544948+0000", 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 420.74925409700001, 2024-03-11T21:46:48.282 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.014225295000000001, 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 109 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.544948+0000", 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.283 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.544948+0000", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.544951+0000", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0850000000000001e-06 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.544965+0000", 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4599e-05 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.284 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.544998+0000", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2279e-05 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.545013+0000", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5685e-05 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.545160+0000", 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014710100000000001 2024-03-11T21:46:48.285 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.545167+0000", 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7449999999999998e-06 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.548783+0000", 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0036160010000000002 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:46:48.286 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.549063+0000", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027951400000000002 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.556143+0000", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0070805060000000003 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.556284+0000", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000140364 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.287 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.556792+0000", 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00050806099999999997 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.556876+0000", 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4699999999999999e-05 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:47.559173+0000", 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0022966549999999999 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.288 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDECSubOpWrite(313.3s1 728/688 ECSubWrite(tid=800, reqid=client.41921.0:474, at_version=728'6, trim_to=0'0, roll_forward_to=713'5))", 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:27.522305+0000", 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 380.77189688999999, 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.010331388, 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522305+0000", 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.289 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522305+0000", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522305+0000", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0699999999999999e-07 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522342+0000", 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7264000000000001e-05 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.290 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522351+0000", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3820000000000004e-06 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522361+0000", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.005e-05 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.522612+0000", 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00025083800000000003 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.291 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.531841+0000", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0092293849999999997 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.532607+0000", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00076617099999999995 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:27.532636+0000", 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9091e-05 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.292 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:854 320.14 320:28000000::::head [pgnls start_epoch 751] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e751)", 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:50.648691+0000", 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 357.64551064900002, 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0070109639999999997, 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 854 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.293 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648691+0000", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648691+0000", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648694+0000", 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1329999999999999e-06 2024-03-11T21:46:48.294 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648709+0000", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4324999999999999e-05 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648740+0000", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1529000000000003e-05 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648756+0000", 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5957e-05 2024-03-11T21:46:48.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648842+0000", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6139999999999999e-05 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.648848+0000", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.3469999999999998e-06 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.655471+0000", 2024-03-11T21:46:48.296 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.006623004 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.655523+0000", 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2757999999999999e-05 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:50.655702+0000", 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00017877100000000001 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.297 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:915 321.d 321:b3b32f0f:test-rados-api-smithi143-71773-127::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e757)", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:56.577326+0000", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 351.716876098, 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0079440680000000003, 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 915 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.298 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577326+0000", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577326+0000", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577328+0000", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1880000000000001e-06 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.299 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577342+0000", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4265e-05 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577361+0000", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8635e-05 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577373+0000", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1949e-05 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.300 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577453+0000", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.0618000000000006e-05 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.577464+0000", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0752e-05 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.582227+0000", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.004762629 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for blocked object", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.582410+0000", 2024-03-11T21:46:48.301 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018331200000000001 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.583364+0000", 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00095390999999999998 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.583449+0000", 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4680999999999998e-05 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:46:48.302 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.583607+0000", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000158455 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.584173+0000", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00056623700000000003 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.584703+0000", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00052985499999999997 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.303 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.585148+0000", 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00044472899999999999 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.585220+0000", 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1934999999999995e-05 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:56.585270+0000", 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.9917999999999999e-05 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.304 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1045 321.a 321:50000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:58.615634+0000", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 349.67856742100003, 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0043341899999999999, 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1045 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.305 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615634+0000", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615634+0000", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615637+0000", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2740000000000002e-06 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.306 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615648+0000", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1545999999999999e-05 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615674+0000", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5486e-05 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615684+0000", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0652999999999999e-05 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.307 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615762+0000", 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7353000000000003e-05 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.615767+0000", 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5210000000000002e-06 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.619792+0000", 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0040245699999999999 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.308 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.619835+0000", 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2827999999999997e-05 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.619969+0000", 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013395900000000001 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1431 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'1, mlcod=0'0)", 2024-03-11T21:46:48.309 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.593905+0000", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 301.70029627899999, 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0055299600000000004, 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593905+0000", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593905+0000", 2024-03-11T21:46:48.310 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593906+0000", 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.91e-07 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593912+0000", 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.93e-06 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.311 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593929+0000", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6968000000000001e-05 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.593938+0000", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.3850000000000008e-06 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594037+0000", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.8950000000000006e-05 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.312 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594072+0000", 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.5479999999999999e-05 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599369+0000", 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0052965599999999996 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599435+0000", 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6495999999999995e-05 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.313 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.41921.0:1794 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'2, mlcod=0'0)", 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.595333+0000", 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 301.69886923299998, 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.016874855000000001, 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.595333+0000", 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.314 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.595333+0000", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.595333+0000", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6599999999999995e-07 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.603107+0000", 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0077737090000000002 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.315 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.603126+0000", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8766e-05 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.603137+0000", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.15e-05 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.603222+0000", 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.4505000000000005e-05 2024-03-11T21:46:48.316 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.603248+0000", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6339999999999999e-05 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612111+0000", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0088633240000000005 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612207+0000", 2024-03-11T21:46:48.317 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.5945000000000004e-05 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1861 326.1f 326:f8000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:48.642865+0000", 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 299.65133698099999, 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0050264840000000003, 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.318 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1861 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642865+0000", 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642865+0000", 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.319 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642867+0000", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1670000000000002e-06 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642878+0000", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1474999999999999e-05 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642903+0000", 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4219e-05 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.320 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642913+0000", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0423e-05 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642987+0000", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3657000000000003e-05 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.642992+0000", 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.1680000000000003e-06 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.647679+0000", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0046871270000000001 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.647709+0000", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0001999999999998e-05 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.647891+0000", 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000182246 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.322 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1896 327.6 327:602f83fe:::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e814)", 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:55.598936+0000", 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 292.69526588899998, 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.052443180999999998, 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1896 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.598936+0000", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.598936+0000", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.598940+0000", 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0600000000000001e-06 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.324 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.598961+0000", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0962999999999999e-05 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.643912+0000", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.044951415000000002 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.643946+0000", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3528999999999997e-05 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.644157+0000", 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000211358 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.644175+0000", 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7506000000000001e-05 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.651103+0000", 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0069281450000000001 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:55.651379+0000", 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027620500000000002 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1944 327.6 327:60000000::::head [pgnls start_epoch 815] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e815)", 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:57.650875+0000", 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 290.64332718700001, 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0076876039999999998, 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.327 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1944 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650875+0000", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650875+0000", 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650878+0000", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.63e-06 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650894+0000", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5892999999999998e-05 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650933+0000", 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8782000000000002e-05 2024-03-11T21:46:48.329 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.650951+0000", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8410000000000002e-05 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.651115+0000", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016350499999999999 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.651125+0000", 2024-03-11T21:46:48.330 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0482999999999999e-05 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.658256+0000", 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0071302730000000003 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.658342+0000", 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6378000000000005e-05 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:57.658562+0000", 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00022024999999999999 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2145 329.10 329:0cafdcee:test-rados-api-smithi143-71773-143::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:30.583080+0000", 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 257.71112196399997, 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011287785999999999, 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "reached pg", 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.332 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2145 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583080+0000", 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583080+0000", 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583084+0000", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.9559999999999999e-06 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583105+0000", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0823e-05 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583143+0000", 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8878999999999998e-05 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.334 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583171+0000", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7373e-05 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.583322+0000", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015127200000000001 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.594368+0000", 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011045483 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.335 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2212 329.10 329:08000000:%01:::head [pgnls start_epoch 844] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e844)", 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:32.592759+0000", 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 255.70144301799999, 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0075767489999999998, 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2212 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.336 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592759+0000", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592759+0000", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592763+0000", 2024-03-11T21:46:48.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.3030000000000001e-06 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592789+0000", 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6347e-05 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592853+0000", 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.3768000000000003e-05 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.338 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.592881+0000", 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7504999999999998e-05 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.593429+0000", 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00054798499999999997 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.339 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.593448+0000", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9639999999999998e-05 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.599937+0000", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0064883559999999998 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.600060+0000", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012388000000000001 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.600335+0000", 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00027496500000000001 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2309 329.10 329:0cafdcee:test-rados-api-smithi143-71773-143::foo0:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected+supports_pool_eio e844)", 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:32.700772+0000", 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 255.59342960399999, 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0036011789999999999, 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.341 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2309 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700772+0000", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700772+0000", 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700775+0000", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9069999999999999e-06 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700788+0000", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3293e-05 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700839+0000", 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.0355000000000002e-05 2024-03-11T21:46:48.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.700910+0000", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0845000000000004e-05 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.701171+0000", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00026190600000000001 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.701408+0000", 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00023686000000000001 2024-03-11T21:46:48.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.701715+0000", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00030715499999999998 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.703075+0000", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0013597259999999999 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.704086+0000", 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0010106189999999999 2024-03-11T21:46:48.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.704183+0000", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6798000000000004e-05 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.704273+0000", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.0264000000000002e-05 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.704373+0000", 2024-03-11T21:46:48.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000100451 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.14427.0:4162 3.4 e864/295 3:258fd845:::obj_delete_at_hint.0000000113:head v 864'66, mlcod=864'64)", 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.675901+0000", 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 216.61830062499999, 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012595683, 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675901+0000", 2024-03-11T21:46:48.347 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675901+0000", 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675902+0000", 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5600000000000005e-07 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675909+0000", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.5140000000000004e-06 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675927+0000", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7912999999999999e-05 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.675936+0000", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.7239999999999998e-06 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.676010+0000", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3742e-05 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.676038+0000", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8039999999999999e-05 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688426+0000", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012387921 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.688497+0000", 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1073000000000002e-05 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2929 332.e 332:70000000:%01:::head [pgnls start_epoch 871] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e871)", 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:18.661435+0000", 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 209.632766266, 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.006031135, 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2929 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661435+0000", 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661435+0000", 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.352 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661437+0000", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8530000000000001e-06 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661448+0000", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.075e-05 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661468+0000", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9944000000000001e-05 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661477+0000", 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.9439999999999994e-06 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661611+0000", 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013436200000000001 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.661622+0000", 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1059e-05 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.667227+0000", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056044930000000003 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.667280+0000", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2985000000000002e-05 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:18.667467+0000", 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018674499999999999 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:3130 334.9 334:9470e5b5:test-rados-api-smithi143-71773-153::foo-chunk:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:24.682601+0000", 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 203.61160112900001, 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0035894999999999998, 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3130 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.356 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682601+0000", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682601+0000", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682602+0000", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.401e-06 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682610+0000", 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6480000000000003e-06 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682629+0000", 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9176000000000001e-05 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682640+0000", 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1155000000000001e-05 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.358 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682737+0000", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6588000000000004e-05 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.682838+0000", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00010152700000000001 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.686190+0000", 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0033520049999999999 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(3.1e,QREQUEST,e:883,from:3,reservation_nonce:16)", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:30.668835+0000", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 197.62536681399999, 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0036666390000000002, 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668835+0000", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668835+0000", 2024-03-11T21:46:48.360 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668837+0000", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.956e-06 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668848+0000", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1192999999999999e-05 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668867+0000", 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8885e-05 2024-03-11T21:46:48.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668877+0000", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.047e-05 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668958+0000", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.0920000000000005e-05 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.668965+0000", 2024-03-11T21:46:48.362 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.246e-06 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.672425+0000", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.003460566 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.672446+0000", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1005e-05 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:30.672502+0000", 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.5398000000000001e-05 2024-03-11T21:46:48.363 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(6.17,QREQUEST,e:915,from:4,reservation_nonce:22)", 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:44:28.960116+0000", 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 139.33408569700001, 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0061064379999999996, 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960116+0000", 2024-03-11T21:46:48.364 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960116+0000", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960118+0000", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3369999999999998e-06 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960130+0000", 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1654999999999999e-05 2024-03-11T21:46:48.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960156+0000", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5871999999999999e-05 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960169+0000", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3237e-05 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960369+0000", 2024-03-11T21:46:48.366 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019947399999999999 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.960382+0000", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3393e-05 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.966115+0000", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0057329620000000003 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.966145+0000", 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0199999999999999e-05 2024-03-11T21:46:48.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:28.966222+0000", 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.7307999999999999e-05 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.42197.0:2 338.0 e918/917 338:0384ab8e:::oid:head v 918'1, mlcod=0'0)", 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:44:31.698728+0000", 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 136.59547356900001, 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0071051279999999996, 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.368 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698728+0000", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698728+0000", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698729+0000", 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9500000000000002e-07 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698736+0000", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6990000000000003e-06 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698746+0000", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.6229999999999992e-06 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698758+0000", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1875000000000001e-05 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698816+0000", 2024-03-11T21:46:48.373 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.7572999999999999e-05 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.698845+0000", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9745999999999999e-05 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.705781+0000", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0069359859999999999 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:31.705833+0000", 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.2030999999999997e-05 2024-03-11T21:46:48.374 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.42263.0:3 345.0 345:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e959)", 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:45:12.708905+0000", 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 95.585296933999999, 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.042853523999999997, 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.42263", 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3317538040", 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 3 2024-03-11T21:46:48.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708905+0000", 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708905+0000", 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:46:48.376 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708907+0000", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1849999999999998e-06 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708915+0000", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.2009999999999999e-06 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708930+0000", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4640999999999999e-05 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:46:48.377 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.708943+0000", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2974e-05 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.709026+0000", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3428000000000002e-05 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.709037+0000", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1255e-05 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:46:48.378 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.728980+0000", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.019942683999999999 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.729147+0000", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016647999999999999 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.729346+0000", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019907200000000001 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.730281+0000", 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00093541999999999996 2024-03-11T21:46:48.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.731017+0000", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00073548799999999996 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.751673+0000", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.020655956999999999 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.751722+0000", 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.9462999999999997e-05 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:46:48.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:12.751758+0000", 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6276000000000003e-05 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:46:48.381 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:48.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:47 smithi203 bash[23654]: cluster 2024-03-11T21:46:46.250417+0000 osd.3 (osd.3) 567 : cluster 0 6.b scrub starts 2024-03-11T21:46:48.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:47 smithi203 bash[23654]: cluster 2024-03-11T21:46:46.252878+0000 osd.3 (osd.3) 568 : cluster 0 6.b scrub ok 2024-03-11T21:46:48.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:47 smithi203 bash[23654]: cluster 2024-03-11T21:46:46.999161+0000 osd.6 (osd.6) 455 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:48.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:47 smithi203 bash[23654]: cluster 2024-03-11T21:46:46.999541+0000 osd.6 (osd.6) 456 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:48.562 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- 172.21.15.143:0/331480923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0106f40 msgr2=0x7fa9d01018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- 172.21.15.143:0/331480923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0106f40 0x7fa9d01018f0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800b0d0 tx=0x7fa9c800b3e0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- 172.21.15.143:0/331480923 shutdown_connections 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- 172.21.15.143:0/331480923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0106f40 0x7fa9d01018f0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- 172.21.15.143:0/331480923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0104ab0 0x7fa9d0104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- 172.21.15.143:0/331480923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0101e30 0x7fa9d01022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- 172.21.15.143:0/331480923 >> 172.21.15.143:0/331480923 conn(0x7fa9d0076d80 msgr2=0x7fa9d0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- 172.21.15.143:0/331480923 shutdown_connections 2024-03-11T21:46:48.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- 172.21.15.143:0/331480923 wait complete. 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 librados: starting msgr at 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 librados: starting objecter 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 Processor -- start 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- start start 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 librados: setting wanted keys 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 librados: calling monclient init 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 0x7fa9d019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0106f40 0x7fa9d01a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0117e10 con 0x7fa9d0104ab0 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0117c90 con 0x7fa9d0106f40 2024-03-11T21:46:48.565 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d725b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0117f90 con 0x7fa9d0101e30 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:47664/0 (socket says 172.21.15.143:47664) 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 -- 172.21.15.143:0/1818694013 learned_addr learned my addr 172.21.15.143:0/1818694013 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9cffff700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 0x7fa9d019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 -- 172.21.15.143:0/1818694013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0106f40 msgr2=0x7fa9d01a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d57f8700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0106f40 0x7fa9d01a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0106f40 0x7fa9d01a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 -- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 msgr2=0x7fa9d019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 0x7fa9d019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 -- 172.21.15.143:0/1818694013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d01a3330 con 0x7fa9d0101e30 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9cffff700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 0x7fa9d019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:46:48.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9d4ff7700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8007a20 tx=0x7fa9c8007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9cdffb700 10 client.?.objecter ms_handle_connect 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.575+0000 7fa9cdffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8021020 con 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d01a35c0 con 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa9c8004e40 con 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c80052b0 con 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9d725b700 10 client.42341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d01aaea0 con 0x7fa9d0101e30 2024-03-11T21:46:48.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9d725b700 1 librados: init done 2024-03-11T21:46:48.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d004faa0 con 0x7fa9d0101e30 2024-03-11T21:46:48.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa9c8024020 con 0x7fa9d0101e30 2024-03-11T21:46:48.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 0x7fa9b8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:48.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(985..985 src has 933..985) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fa9c800f020 con 0x7fa9d0101e30 2024-03-11T21:46:48.568 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 10 client.42341.objecter ms_dispatch 0x7fa9d0001550 osd_map(985..985 src has 933..985) v4 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 3 client.42341.objecter handle_osd_map got epochs [985,985] > 0 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 3 client.42341.objecter handle_osd_map decoding full epoch 985 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cffff700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 0x7fa9b8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 20 client.42341.objecter dump_active .. 0 homeless 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cdffb700 10 client.42341.objecter ms_handle_connect 0x7fa9b806f820 2024-03-11T21:46:48.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.579+0000 7fa9cffff700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 0x7fa9b8071ce0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0101bf0 tx=0x7fa9c400a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:48.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.583+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa9c8011e50 con 0x7fa9d0101e30 2024-03-11T21:46:48.789 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.803+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fa9d01ab070 con 0x7fa9d0101e30 2024-03-11T21:46:48.899 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.911+0000 7fa9cdffb700 1 -- 172.21.15.143:0/1818694013 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v986) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa9c8009cb0 con 0x7fa9d0101e30 2024-03-11T21:46:48.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 10 librados: watch_flush enter 2024-03-11T21:46:48.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 10 librados: watch_flush exit 2024-03-11T21:46:48.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 20 client.42341.objecter shutdown clearing up homeless session... 2024-03-11T21:46:48.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 10 client.42341.objecter successfully canceled tick 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 msgr2=0x7fa9b8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 0x7fa9b8071ce0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0101bf0 tx=0x7fa9c400a380 comp rx=0 tx=0).stop 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 msgr2=0x7fa9d019e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8007a20 tx=0x7fa9c8007a50 comp rx=0 tx=0).stop 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 shutdown_connections 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa9b806f820 0x7fa9b8071ce0 unknown :-1 s=CLOSED pgs=3500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9d0106f40 0x7fa9d01a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9d0101e30 0x7fa9d019e320 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 --2- 172.21.15.143:0/1818694013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d0104ab0 0x7fa9d019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:48.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 >> 172.21.15.143:0/1818694013 conn(0x7fa9d0076d80 msgr2=0x7fa9d00ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:48.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 shutdown_connections 2024-03-11T21:46:48.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 -- 172.21.15.143:0/1818694013 wait complete. 2024-03-11T21:46:48.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:48.919+0000 7fa9d725b700 1 librados: shutdown 2024-03-11T21:46:48.908 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.418241+0000 osd.7 (osd.7) 546 : cluster 0 5.4 scrub starts 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.420747+0000 osd.7 (osd.7) 547 : cluster 0 5.4 scrub ok 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.758348+0000 mgr.y (mgr.14550) 914 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.908659+0000 mon.a (mon.0) 4108 : cluster 0 osdmap e985: 8 total, 8 up, 8 in 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.966186+0000 osd.4 (osd.4) 531 : cluster 0 3.14 scrub starts 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:47.970781+0000 osd.4 (osd.4) 532 : cluster 0 3.14 scrub ok 2024-03-11T21:46:49.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.277511+0000 osd.4 (osd.4) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.278023+0000 osd.4 (osd.4) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: audit 2024-03-11T21:46:48.804503+0000 mon.c (mon.2) 1398 : audit 1 from='client.? 172.21.15.143:0/1818694013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[29284]: audit 2024-03-11T21:46:48.804919+0000 mon.a (mon.0) 4109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.418241+0000 osd.7 (osd.7) 546 : cluster 0 5.4 scrub starts 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.420747+0000 osd.7 (osd.7) 547 : cluster 0 5.4 scrub ok 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.758348+0000 mgr.y (mgr.14550) 914 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.908659+0000 mon.a (mon.0) 4108 : cluster 0 osdmap e985: 8 total, 8 up, 8 in 2024-03-11T21:46:49.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.966186+0000 osd.4 (osd.4) 531 : cluster 0 3.14 scrub starts 2024-03-11T21:46:49.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:47.970781+0000 osd.4 (osd.4) 532 : cluster 0 3.14 scrub ok 2024-03-11T21:46:49.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.277511+0000 osd.4 (osd.4) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:49.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.278023+0000 osd.4 (osd.4) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:49.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: audit 2024-03-11T21:46:48.804503+0000 mon.c (mon.2) 1398 : audit 1 from='client.? 172.21.15.143:0/1818694013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:48 smithi143 bash[21512]: audit 2024-03-11T21:46:48.804919+0000 mon.a (mon.0) 4109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.418241+0000 osd.7 (osd.7) 546 : cluster 0 5.4 scrub starts 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.420747+0000 osd.7 (osd.7) 547 : cluster 0 5.4 scrub ok 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.758348+0000 mgr.y (mgr.14550) 914 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.908659+0000 mon.a (mon.0) 4108 : cluster 0 osdmap e985: 8 total, 8 up, 8 in 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.966186+0000 osd.4 (osd.4) 531 : cluster 0 3.14 scrub starts 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:47.970781+0000 osd.4 (osd.4) 532 : cluster 0 3.14 scrub ok 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.277511+0000 osd.4 (osd.4) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.278023+0000 osd.4 (osd.4) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: audit 2024-03-11T21:46:48.804503+0000 mon.c (mon.2) 1398 : audit 1 from='client.? 172.21.15.143:0/1818694013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.263 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:48 smithi203 bash[23654]: audit 2024-03-11T21:46:48.804919+0000 mon.a (mon.0) 4109 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:46:49.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:46:49.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:49.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:46:50.197 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.366703+0000 osd.0 (osd.0) 560 : cluster 0 3.1b scrub starts 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.369012+0000 osd.0 (osd.0) 561 : cluster 0 3.1b scrub ok 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: audit 2024-03-11T21:46:48.903851+0000 mon.a (mon.0) 4110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.912492+0000 mon.a (mon.0) 4111 : cluster 0 osdmap e986: 8 total, 8 up, 8 in 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.941002+0000 osd.1 (osd.1) 361 : cluster 0 9.14 scrub starts 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[21512]: cluster 2024-03-11T21:46:48.943314+0000 osd.1 (osd.1) 362 : cluster 0 9.14 scrub ok 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.366703+0000 osd.0 (osd.0) 560 : cluster 0 3.1b scrub starts 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.369012+0000 osd.0 (osd.0) 561 : cluster 0 3.1b scrub ok 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: audit 2024-03-11T21:46:48.903851+0000 mon.a (mon.0) 4110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.912492+0000 mon.a (mon.0) 4111 : cluster 0 osdmap e986: 8 total, 8 up, 8 in 2024-03-11T21:46:50.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.941002+0000 osd.1 (osd.1) 361 : cluster 0 9.14 scrub starts 2024-03-11T21:46:50.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:49 smithi143 bash[29284]: cluster 2024-03-11T21:46:48.943314+0000 osd.1 (osd.1) 362 : cluster 0 9.14 scrub ok 2024-03-11T21:46:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.366703+0000 osd.0 (osd.0) 560 : cluster 0 3.1b scrub starts 2024-03-11T21:46:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.369012+0000 osd.0 (osd.0) 561 : cluster 0 3.1b scrub ok 2024-03-11T21:46:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: audit 2024-03-11T21:46:48.903851+0000 mon.a (mon.0) 4110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:46:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.912492+0000 mon.a (mon.0) 4111 : cluster 0 osdmap e986: 8 total, 8 up, 8 in 2024-03-11T21:46:50.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.941002+0000 osd.1 (osd.1) 361 : cluster 0 9.14 scrub starts 2024-03-11T21:46:50.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:49 smithi203 bash[23654]: cluster 2024-03-11T21:46:48.943314+0000 osd.1 (osd.1) 362 : cluster 0 9.14 scrub ok 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: audit 2024-03-11T21:46:49.276065+0000 mgr.y (mgr.14550) 915 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: cluster 2024-03-11T21:46:49.287084+0000 osd.3 (osd.3) 569 : cluster 0 6.f scrub starts 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: cluster 2024-03-11T21:46:49.289392+0000 osd.3 (osd.3) 570 : cluster 0 6.f scrub ok 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: cluster 2024-03-11T21:46:49.426000+0000 osd.7 (osd.7) 548 : cluster 0 1.d scrub starts 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: cluster 2024-03-11T21:46:49.428585+0000 osd.7 (osd.7) 549 : cluster 0 1.d scrub ok 2024-03-11T21:46:51.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[21512]: cluster 2024-03-11T21:46:49.758963+0000 mgr.y (mgr.14550) 916 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: audit 2024-03-11T21:46:49.276065+0000 mgr.y (mgr.14550) 915 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: cluster 2024-03-11T21:46:49.287084+0000 osd.3 (osd.3) 569 : cluster 0 6.f scrub starts 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: cluster 2024-03-11T21:46:49.289392+0000 osd.3 (osd.3) 570 : cluster 0 6.f scrub ok 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: cluster 2024-03-11T21:46:49.426000+0000 osd.7 (osd.7) 548 : cluster 0 1.d scrub starts 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: cluster 2024-03-11T21:46:49.428585+0000 osd.7 (osd.7) 549 : cluster 0 1.d scrub ok 2024-03-11T21:46:51.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:50 smithi143 bash[29284]: cluster 2024-03-11T21:46:49.758963+0000 mgr.y (mgr.14550) 916 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: audit 2024-03-11T21:46:49.276065+0000 mgr.y (mgr.14550) 915 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: cluster 2024-03-11T21:46:49.287084+0000 osd.3 (osd.3) 569 : cluster 0 6.f scrub starts 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: cluster 2024-03-11T21:46:49.289392+0000 osd.3 (osd.3) 570 : cluster 0 6.f scrub ok 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: cluster 2024-03-11T21:46:49.426000+0000 osd.7 (osd.7) 548 : cluster 0 1.d scrub starts 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: cluster 2024-03-11T21:46:49.428585+0000 osd.7 (osd.7) 549 : cluster 0 1.d scrub ok 2024-03-11T21:46:51.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:50 smithi203 bash[23654]: cluster 2024-03-11T21:46:49.758963+0000 mgr.y (mgr.14550) 916 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:52.197 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:46:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:46:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:52.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:51 smithi143 bash[29284]: cluster 2024-03-11T21:46:50.302223+0000 osd.3 (osd.3) 571 : cluster 0 9.1b deep-scrub starts 2024-03-11T21:46:52.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:51 smithi143 bash[29284]: cluster 2024-03-11T21:46:50.308117+0000 osd.3 (osd.3) 572 : cluster 0 9.1b deep-scrub ok 2024-03-11T21:46:52.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:51 smithi143 bash[21512]: cluster 2024-03-11T21:46:50.302223+0000 osd.3 (osd.3) 571 : cluster 0 9.1b deep-scrub starts 2024-03-11T21:46:52.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:51 smithi143 bash[21512]: cluster 2024-03-11T21:46:50.308117+0000 osd.3 (osd.3) 572 : cluster 0 9.1b deep-scrub ok 2024-03-11T21:46:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:51 smithi203 bash[23654]: cluster 2024-03-11T21:46:50.302223+0000 osd.3 (osd.3) 571 : cluster 0 9.1b deep-scrub starts 2024-03-11T21:46:52.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:51 smithi203 bash[23654]: cluster 2024-03-11T21:46:50.308117+0000 osd.3 (osd.3) 572 : cluster 0 9.1b deep-scrub ok 2024-03-11T21:46:53.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:52 smithi143 bash[21512]: cluster 2024-03-11T21:46:51.760076+0000 mgr.y (mgr.14550) 917 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:53.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:52 smithi143 bash[21512]: audit 2024-03-11T21:46:52.916506+0000 mon.c (mon.2) 1399 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:53.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:52 smithi143 bash[29284]: cluster 2024-03-11T21:46:51.760076+0000 mgr.y (mgr.14550) 917 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:53.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:52 smithi143 bash[29284]: audit 2024-03-11T21:46:52.916506+0000 mon.c (mon.2) 1399 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:53.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:52 smithi203 bash[23654]: cluster 2024-03-11T21:46:51.760076+0000 mgr.y (mgr.14550) 917 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:46:53.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:52 smithi203 bash[23654]: audit 2024-03-11T21:46:52.916506+0000 mon.c (mon.2) 1399 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.250797+0000 osd.3 (osd.3) 573 : cluster 0 5.16 scrub starts 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.252257+0000 osd.3 (osd.3) 574 : cluster 0 5.16 scrub ok 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.362934+0000 osd.0 (osd.0) 562 : cluster 0 5.5 scrub starts 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.365414+0000 osd.0 (osd.0) 563 : cluster 0 5.5 scrub ok 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.480892+0000 osd.7 (osd.7) 550 : cluster 0 1.10 scrub starts 2024-03-11T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[21512]: cluster 2024-03-11T21:46:52.483399+0000 osd.7 (osd.7) 551 : cluster 0 1.10 scrub ok 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.250797+0000 osd.3 (osd.3) 573 : cluster 0 5.16 scrub starts 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.252257+0000 osd.3 (osd.3) 574 : cluster 0 5.16 scrub ok 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.362934+0000 osd.0 (osd.0) 562 : cluster 0 5.5 scrub starts 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.365414+0000 osd.0 (osd.0) 563 : cluster 0 5.5 scrub ok 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.480892+0000 osd.7 (osd.7) 550 : cluster 0 1.10 scrub starts 2024-03-11T21:46:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:53 smithi143 bash[29284]: cluster 2024-03-11T21:46:52.483399+0000 osd.7 (osd.7) 551 : cluster 0 1.10 scrub ok 2024-03-11T21:46:54.243 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:54.244 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.250797+0000 osd.3 (osd.3) 573 : cluster 0 5.16 scrub starts 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.252257+0000 osd.3 (osd.3) 574 : cluster 0 5.16 scrub ok 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.362934+0000 osd.0 (osd.0) 562 : cluster 0 5.5 scrub starts 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.365414+0000 osd.0 (osd.0) 563 : cluster 0 5.5 scrub ok 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.480892+0000 osd.7 (osd.7) 550 : cluster 0 1.10 scrub starts 2024-03-11T21:46:54.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:53 smithi203 bash[23654]: cluster 2024-03-11T21:46:52.483399+0000 osd.7 (osd.7) 551 : cluster 0 1.10 scrub ok 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.473436+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub starts 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.476511+0000 osd.7 (osd.7) 553 : cluster 0 6.3 scrub ok 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.761521+0000 mgr.y (mgr.14550) 918 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.942309+0000 osd.4 (osd.4) 535 : cluster 0 3.1a scrub starts 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.948030+0000 osd.4 (osd.4) 536 : cluster 0 3.1a scrub ok 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.982867+0000 osd.1 (osd.1) 363 : cluster 0 1.b scrub starts 2024-03-11T21:46:55.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[21512]: cluster 2024-03-11T21:46:53.985435+0000 osd.1 (osd.1) 364 : cluster 0 1.b scrub ok 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.473436+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub starts 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.476511+0000 osd.7 (osd.7) 553 : cluster 0 6.3 scrub ok 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.761521+0000 mgr.y (mgr.14550) 918 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.942309+0000 osd.4 (osd.4) 535 : cluster 0 3.1a scrub starts 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.948030+0000 osd.4 (osd.4) 536 : cluster 0 3.1a scrub ok 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.982867+0000 osd.1 (osd.1) 363 : cluster 0 1.b scrub starts 2024-03-11T21:46:55.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:54 smithi143 bash[29284]: cluster 2024-03-11T21:46:53.985435+0000 osd.1 (osd.1) 364 : cluster 0 1.b scrub ok 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.473436+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub starts 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.476511+0000 osd.7 (osd.7) 553 : cluster 0 6.3 scrub ok 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.761521+0000 mgr.y (mgr.14550) 918 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.942309+0000 osd.4 (osd.4) 535 : cluster 0 3.1a scrub starts 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.948030+0000 osd.4 (osd.4) 536 : cluster 0 3.1a scrub ok 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.982867+0000 osd.1 (osd.1) 363 : cluster 0 1.b scrub starts 2024-03-11T21:46:55.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:54 smithi203 bash[23654]: cluster 2024-03-11T21:46:53.985435+0000 osd.1 (osd.1) 364 : cluster 0 1.b scrub ok 2024-03-11T21:46:56.042 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:55 smithi143 bash[29284]: cluster 2024-03-11T21:46:54.518812+0000 osd.7 (osd.7) 554 : cluster 0 5.10 scrub starts 2024-03-11T21:46:56.043 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:55 smithi143 bash[29284]: cluster 2024-03-11T21:46:54.521023+0000 osd.7 (osd.7) 555 : cluster 0 5.10 scrub ok 2024-03-11T21:46:56.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:55 smithi143 bash[21512]: cluster 2024-03-11T21:46:54.518812+0000 osd.7 (osd.7) 554 : cluster 0 5.10 scrub starts 2024-03-11T21:46:56.043 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:55 smithi143 bash[21512]: cluster 2024-03-11T21:46:54.521023+0000 osd.7 (osd.7) 555 : cluster 0 5.10 scrub ok 2024-03-11T21:46:56.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:55 smithi203 bash[23654]: cluster 2024-03-11T21:46:54.518812+0000 osd.7 (osd.7) 554 : cluster 0 5.10 scrub starts 2024-03-11T21:46:56.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:55 smithi203 bash[23654]: cluster 2024-03-11T21:46:54.521023+0000 osd.7 (osd.7) 555 : cluster 0 5.10 scrub ok 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.508020+0000 osd.2 (osd.2) 473 : cluster 0 2.c scrub starts 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.510426+0000 osd.2 (osd.2) 474 : cluster 0 2.c scrub ok 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.726116+0000 osd.6 (osd.6) 457 : cluster 0 2.17 scrub starts 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.728352+0000 osd.6 (osd.6) 458 : cluster 0 2.17 scrub ok 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.762555+0000 mgr.y (mgr.14550) 919 : cluster 0 pgmap v1581: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:57.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.983076+0000 osd.4 (osd.4) 537 : cluster 0 3.9 scrub starts 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[29284]: cluster 2024-03-11T21:46:55.986958+0000 osd.4 (osd.4) 538 : cluster 0 3.9 scrub ok 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.508020+0000 osd.2 (osd.2) 473 : cluster 0 2.c scrub starts 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.510426+0000 osd.2 (osd.2) 474 : cluster 0 2.c scrub ok 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.726116+0000 osd.6 (osd.6) 457 : cluster 0 2.17 scrub starts 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.728352+0000 osd.6 (osd.6) 458 : cluster 0 2.17 scrub ok 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.762555+0000 mgr.y (mgr.14550) 919 : cluster 0 pgmap v1581: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.983076+0000 osd.4 (osd.4) 537 : cluster 0 3.9 scrub starts 2024-03-11T21:46:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:56 smithi143 bash[21512]: cluster 2024-03-11T21:46:55.986958+0000 osd.4 (osd.4) 538 : cluster 0 3.9 scrub ok 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.508020+0000 osd.2 (osd.2) 473 : cluster 0 2.c scrub starts 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.510426+0000 osd.2 (osd.2) 474 : cluster 0 2.c scrub ok 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.726116+0000 osd.6 (osd.6) 457 : cluster 0 2.17 scrub starts 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.728352+0000 osd.6 (osd.6) 458 : cluster 0 2.17 scrub ok 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.762555+0000 mgr.y (mgr.14550) 919 : cluster 0 pgmap v1581: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.983076+0000 osd.4 (osd.4) 537 : cluster 0 3.9 scrub starts 2024-03-11T21:46:57.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:56 smithi203 bash[23654]: cluster 2024-03-11T21:46:55.986958+0000 osd.4 (osd.4) 538 : cluster 0 3.9 scrub ok 2024-03-11T21:46:58.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.267569+0000 osd.3 (osd.3) 575 : cluster 0 4.0 scrub starts 2024-03-11T21:46:58.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.270785+0000 osd.3 (osd.3) 576 : cluster 0 4.0 scrub ok 2024-03-11T21:46:58.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.485485+0000 osd.7 (osd.7) 556 : cluster 0 5.1a scrub starts 2024-03-11T21:46:58.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.487765+0000 osd.7 (osd.7) 557 : cluster 0 5.1a scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.694498+0000 osd.6 (osd.6) 459 : cluster 0 2.7 scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.697788+0000 osd.6 (osd.6) 460 : cluster 0 2.7 scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.724374+0000 osd.7 (osd.7) 558 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:56.724892+0000 osd.7 (osd.7) 559 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[29284]: cluster 2024-03-11T21:46:57.683924+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e987: 8 total, 8 up, 8 in 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.267569+0000 osd.3 (osd.3) 575 : cluster 0 4.0 scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.270785+0000 osd.3 (osd.3) 576 : cluster 0 4.0 scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.485485+0000 osd.7 (osd.7) 556 : cluster 0 5.1a scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.487765+0000 osd.7 (osd.7) 557 : cluster 0 5.1a scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.694498+0000 osd.6 (osd.6) 459 : cluster 0 2.7 scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.697788+0000 osd.6 (osd.6) 460 : cluster 0 2.7 scrub ok 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.724374+0000 osd.7 (osd.7) 558 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:56.724892+0000 osd.7 (osd.7) 559 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:57 smithi143 bash[21512]: cluster 2024-03-11T21:46:57.683924+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e987: 8 total, 8 up, 8 in 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.267569+0000 osd.3 (osd.3) 575 : cluster 0 4.0 scrub starts 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.270785+0000 osd.3 (osd.3) 576 : cluster 0 4.0 scrub ok 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.485485+0000 osd.7 (osd.7) 556 : cluster 0 5.1a scrub starts 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.487765+0000 osd.7 (osd.7) 557 : cluster 0 5.1a scrub ok 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.694498+0000 osd.6 (osd.6) 459 : cluster 0 2.7 scrub starts 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.697788+0000 osd.6 (osd.6) 460 : cluster 0 2.7 scrub ok 2024-03-11T21:46:58.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.724374+0000 osd.7 (osd.7) 558 : cluster 0 purged_snaps scrub starts 2024-03-11T21:46:58.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:56.724892+0000 osd.7 (osd.7) 559 : cluster 0 purged_snaps scrub ok 2024-03-11T21:46:58.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:57 smithi203 bash[23654]: cluster 2024-03-11T21:46:57.683924+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e987: 8 total, 8 up, 8 in 2024-03-11T21:46:59.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:58 smithi143 bash[21512]: cluster 2024-03-11T21:46:57.763381+0000 mgr.y (mgr.14550) 920 : cluster 0 pgmap v1583: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-11T21:46:59.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:58 smithi143 bash[29284]: cluster 2024-03-11T21:46:57.763381+0000 mgr.y (mgr.14550) 920 : cluster 0 pgmap v1583: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-11T21:46:59.273 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:58 smithi203 bash[23654]: cluster 2024-03-11T21:46:57.763381+0000 mgr.y (mgr.14550) 920 : cluster 0 pgmap v1583: 184 pgs: 184 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-11T21:46:59.357 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:46:59.357 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:46:59.358 INFO:teuthology.orchestra.run.smithi143.stdout: "num_ops": 0 2024-03-11T21:46:59.358 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 -- 172.21.15.143:0/966747644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644102880 msgr2=0x7f864410ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 --2- 172.21.15.143:0/966747644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644102880 0x7f864410ec70 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f863c00b0a0 tx=0x7f863c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 -- 172.21.15.143:0/966747644 shutdown_connections 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 --2- 172.21.15.143:0/966747644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f8644108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 --2- 172.21.15.143:0/966747644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644102880 0x7f864410ec70 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 --2- 172.21.15.143:0/966747644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8644101ec0 0x7f8644102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 -- 172.21.15.143:0/966747644 >> 172.21.15.143:0/966747644 conn(0x7f8644078eb0 msgr2=0x7f86440792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 -- 172.21.15.143:0/966747644 shutdown_connections 2024-03-11T21:46:59.389 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.398+0000 7f8649ac4700 1 -- 172.21.15.143:0/966747644 wait complete. 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 librados: starting msgr at 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 librados: starting objecter 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 Processor -- start 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- start start 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 librados: setting wanted keys 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 librados: calling monclient init 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644101ec0 0x7f8644199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8644102880 0x7f864419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86441116f0 con 0x7f8644107f30 2024-03-11T21:46:59.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644111570 con 0x7f8644102880 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8644111870 con 0x7f8644101ec0 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:37010/0 (socket says 172.21.15.143:37010) 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f86437fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644101ec0 0x7f8644199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 -- 172.21.15.143:0/1009008213 learned_addr learned my addr 172.21.15.143:0/1009008213 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 -- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644101ec0 msgr2=0x7f8644199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644101ec0 0x7f8644199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 -- 172.21.15.143:0/1009008213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8644102880 msgr2=0x7f864419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8644102880 0x7f864419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 -- 172.21.15.143:0/1009008213 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864419ee90 con 0x7f8644107f30 2024-03-11T21:46:59.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8643fff700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f863c00b070 tx=0x7f863c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:59.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 10 client.?.objecter ms_handle_connect 0x7f8644107f30 2024-03-11T21:46:59.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:46:59.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c021020 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f863c004c50 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864419f120 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c0052b0 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 10 client.31143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86441a6a00 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 librados: init done 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644103550 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f863c024020 con 0x7f8644107f30 2024-03-11T21:46:59.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 0x7f862c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(987..987 src has 933..987) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f863c08fab0 con 0x7f8644107f30 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 10 client.31143.objecter ms_dispatch 0x7f8644001550 osd_map(987..987 src has 933..987) v4 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 3 client.31143.objecter handle_osd_map got epochs [987,987] > 0 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f8640ff9700 3 client.31143.objecter handle_osd_map decoding full epoch 987 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.402+0000 7f86437fe700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 0x7f862c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.406+0000 7f8640ff9700 20 client.31143.objecter dump_active .. 0 homeless 2024-03-11T21:46:59.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.406+0000 7f86437fe700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 0x7f862c071ce0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f8634009680 tx=0x7f8634009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:46:59.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.406+0000 7f8640ff9700 10 client.31143.objecter ms_handle_connect 0x7f862c06f820 2024-03-11T21:46:59.398 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.410+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f863c02b020 con 0x7f8644107f30 2024-03-11T21:46:59.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:46:59.618 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.630+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f864419b1e0 con 0x7f8644107f30 2024-03-11T21:46:59.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.962+0000 7f8640ff9700 1 -- 172.21.15.143:0/1009008213 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v988) v1 ==== 113+0+0 (secure 0 0 0) 0x7f863c05ca40 con 0x7f8644107f30 2024-03-11T21:46:59.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.966+0000 7f8649ac4700 10 librados: watch_flush enter 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 10 librados: watch_flush exit 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 20 client.31143.objecter shutdown clearing up homeless session... 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 10 client.31143.objecter successfully canceled tick 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 msgr2=0x7f862c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 0x7f862c071ce0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f8634009680 tx=0x7f8634009260 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 msgr2=0x7f864419e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f863c00b070 tx=0x7f863c007a20 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 shutdown_connections 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f862c06f820 0x7f862c071ce0 unknown :-1 s=CLOSED pgs=3501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8644107f30 0x7f864419e750 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8644101ec0 0x7f8644199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 --2- 172.21.15.143:0/1009008213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8644102880 0x7f864419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:46:59.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 >> 172.21.15.143:0/1009008213 conn(0x7f8644078eb0 msgr2=0x7f864410cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:46:59.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 shutdown_connections 2024-03-11T21:46:59.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 -- 172.21.15.143:0/1009008213 wait complete. 2024-03-11T21:46:59.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:46:59.970+0000 7f8649ac4700 1 librados: shutdown 2024-03-11T21:46:59.961 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:58.426707+0000 osd.0 (osd.0) 564 : cluster 0 9.5 scrub starts 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:58.429902+0000 osd.0 (osd.0) 565 : cluster 0 9.5 scrub ok 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:58.701614+0000 osd.6 (osd.6) 461 : cluster 0 3.a scrub starts 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:58.706479+0000 osd.6 (osd.6) 462 : cluster 0 3.a scrub ok 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:58.998826+0000 osd.4 (osd.4) 539 : cluster 0 2.f scrub starts 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.001866+0000 osd.4 (osd.4) 540 : cluster 0 2.f scrub ok 2024-03-11T21:47:00.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:46:59 smithi203 bash[23654]: audit 2024-03-11T21:46:59.633796+0000 mon.a (mon.0) 4113 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:58.426707+0000 osd.0 (osd.0) 564 : cluster 0 9.5 scrub starts 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:58.429902+0000 osd.0 (osd.0) 565 : cluster 0 9.5 scrub ok 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:58.701614+0000 osd.6 (osd.6) 461 : cluster 0 3.a scrub starts 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:58.706479+0000 osd.6 (osd.6) 462 : cluster 0 3.a scrub ok 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:58.998826+0000 osd.4 (osd.4) 539 : cluster 0 2.f scrub starts 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.001866+0000 osd.4 (osd.4) 540 : cluster 0 2.f scrub ok 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[21512]: audit 2024-03-11T21:46:59.633796+0000 mon.a (mon.0) 4113 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:00.432 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:58.426707+0000 osd.0 (osd.0) 564 : cluster 0 9.5 scrub starts 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:58.429902+0000 osd.0 (osd.0) 565 : cluster 0 9.5 scrub ok 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:58.701614+0000 osd.6 (osd.6) 461 : cluster 0 3.a scrub starts 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:58.706479+0000 osd.6 (osd.6) 462 : cluster 0 3.a scrub ok 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:58.998826+0000 osd.4 (osd.4) 539 : cluster 0 2.f scrub starts 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.001866+0000 osd.4 (osd.4) 540 : cluster 0 2.f scrub ok 2024-03-11T21:47:00.433 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:46:59 smithi143 bash[29284]: audit 2024-03-11T21:46:59.633796+0000 mon.a (mon.0) 4113 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:00.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:47:01.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.173647+0000 osd.3 (osd.3) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.174014+0000 osd.3 (osd.3) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: audit 2024-03-11T21:46:59.287043+0000 mgr.y (mgr.14550) 921 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.475727+0000 osd.0 (osd.0) 566 : cluster 0 9.b deep-scrub starts 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.482448+0000 osd.0 (osd.0) 567 : cluster 0 9.b deep-scrub ok 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.765161+0000 mgr.y (mgr.14550) 922 : cluster 0 pgmap v1584: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: audit 2024-03-11T21:46:59.963597+0000 mon.a (mon.0) 4114 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:01.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:46:59.969640+0000 mon.a (mon.0) 4115 : cluster 0 osdmap e988: 8 total, 8 up, 8 in 2024-03-11T21:47:01.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:47:00.048925+0000 osd.4 (osd.4) 541 : cluster 0 9.16 scrub starts 2024-03-11T21:47:01.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: cluster 2024-03-11T21:47:00.053606+0000 osd.4 (osd.4) 542 : cluster 0 9.16 scrub ok 2024-03-11T21:47:01.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:00 smithi203 bash[23654]: audit 2024-03-11T21:47:00.593914+0000 mon.c (mon.2) 1400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:01.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.173647+0000 osd.3 (osd.3) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.174014+0000 osd.3 (osd.3) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: audit 2024-03-11T21:46:59.287043+0000 mgr.y (mgr.14550) 921 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.475727+0000 osd.0 (osd.0) 566 : cluster 0 9.b deep-scrub starts 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.482448+0000 osd.0 (osd.0) 567 : cluster 0 9.b deep-scrub ok 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.765161+0000 mgr.y (mgr.14550) 922 : cluster 0 pgmap v1584: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: audit 2024-03-11T21:46:59.963597+0000 mon.a (mon.0) 4114 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:46:59.969640+0000 mon.a (mon.0) 4115 : cluster 0 osdmap e988: 8 total, 8 up, 8 in 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:47:00.048925+0000 osd.4 (osd.4) 541 : cluster 0 9.16 scrub starts 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: cluster 2024-03-11T21:47:00.053606+0000 osd.4 (osd.4) 542 : cluster 0 9.16 scrub ok 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[21512]: audit 2024-03-11T21:47:00.593914+0000 mon.c (mon.2) 1400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.173647+0000 osd.3 (osd.3) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:01.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.174014+0000 osd.3 (osd.3) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: audit 2024-03-11T21:46:59.287043+0000 mgr.y (mgr.14550) 921 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.475727+0000 osd.0 (osd.0) 566 : cluster 0 9.b deep-scrub starts 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.482448+0000 osd.0 (osd.0) 567 : cluster 0 9.b deep-scrub ok 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.765161+0000 mgr.y (mgr.14550) 922 : cluster 0 pgmap v1584: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: audit 2024-03-11T21:46:59.963597+0000 mon.a (mon.0) 4114 : audit 1 from='client.? 172.21.15.143:0/1009008213' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:46:59.969640+0000 mon.a (mon.0) 4115 : cluster 0 osdmap e988: 8 total, 8 up, 8 in 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:47:00.048925+0000 osd.4 (osd.4) 541 : cluster 0 9.16 scrub starts 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: cluster 2024-03-11T21:47:00.053606+0000 osd.4 (osd.4) 542 : cluster 0 9.16 scrub ok 2024-03-11T21:47:01.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:00 smithi143 bash[29284]: audit 2024-03-11T21:47:00.593914+0000 mon.c (mon.2) 1400 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:02.448 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:03.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:02 smithi203 bash[23654]: cluster 2024-03-11T21:47:01.745230+0000 osd.6 (osd.6) 463 : cluster 0 1.7 scrub starts 2024-03-11T21:47:03.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:02 smithi203 bash[23654]: cluster 2024-03-11T21:47:01.747584+0000 osd.6 (osd.6) 464 : cluster 0 1.7 scrub ok 2024-03-11T21:47:03.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:02 smithi203 bash[23654]: cluster 2024-03-11T21:47:01.766261+0000 mgr.y (mgr.14550) 923 : cluster 0 pgmap v1586: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:03.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:02 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.061153+0000 osd.4 (osd.4) 543 : cluster 0 2.18 scrub starts 2024-03-11T21:47:03.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:02 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.063626+0000 osd.4 (osd.4) 544 : cluster 0 2.18 scrub ok 2024-03-11T21:47:03.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[29284]: cluster 2024-03-11T21:47:01.745230+0000 osd.6 (osd.6) 463 : cluster 0 1.7 scrub starts 2024-03-11T21:47:03.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[29284]: cluster 2024-03-11T21:47:01.747584+0000 osd.6 (osd.6) 464 : cluster 0 1.7 scrub ok 2024-03-11T21:47:03.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[29284]: cluster 2024-03-11T21:47:01.766261+0000 mgr.y (mgr.14550) 923 : cluster 0 pgmap v1586: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.061153+0000 osd.4 (osd.4) 543 : cluster 0 2.18 scrub starts 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.063626+0000 osd.4 (osd.4) 544 : cluster 0 2.18 scrub ok 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21512]: cluster 2024-03-11T21:47:01.745230+0000 osd.6 (osd.6) 463 : cluster 0 1.7 scrub starts 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21512]: cluster 2024-03-11T21:47:01.747584+0000 osd.6 (osd.6) 464 : cluster 0 1.7 scrub ok 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21512]: cluster 2024-03-11T21:47:01.766261+0000 mgr.y (mgr.14550) 923 : cluster 0 pgmap v1586: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.061153+0000 osd.4 (osd.4) 543 : cluster 0 2.18 scrub starts 2024-03-11T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:02 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.063626+0000 osd.4 (osd.4) 544 : cluster 0 2.18 scrub ok 2024-03-11T21:47:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:03 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.472219+0000 osd.7 (osd.7) 560 : cluster 0 1.0 scrub starts 2024-03-11T21:47:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:03 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.476049+0000 osd.7 (osd.7) 561 : cluster 0 1.0 scrub ok 2024-03-11T21:47:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:03 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.950419+0000 osd.1 (osd.1) 365 : cluster 0 6.4 scrub starts 2024-03-11T21:47:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:03 smithi203 bash[23654]: cluster 2024-03-11T21:47:02.953146+0000 osd.1 (osd.1) 366 : cluster 0 6.4 scrub ok 2024-03-11T21:47:04.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.472219+0000 osd.7 (osd.7) 560 : cluster 0 1.0 scrub starts 2024-03-11T21:47:04.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.476049+0000 osd.7 (osd.7) 561 : cluster 0 1.0 scrub ok 2024-03-11T21:47:04.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.950419+0000 osd.1 (osd.1) 365 : cluster 0 6.4 scrub starts 2024-03-11T21:47:04.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[21512]: cluster 2024-03-11T21:47:02.953146+0000 osd.1 (osd.1) 366 : cluster 0 6.4 scrub ok 2024-03-11T21:47:04.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.472219+0000 osd.7 (osd.7) 560 : cluster 0 1.0 scrub starts 2024-03-11T21:47:04.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.476049+0000 osd.7 (osd.7) 561 : cluster 0 1.0 scrub ok 2024-03-11T21:47:04.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.950419+0000 osd.1 (osd.1) 365 : cluster 0 6.4 scrub starts 2024-03-11T21:47:04.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:03 smithi143 bash[29284]: cluster 2024-03-11T21:47:02.953146+0000 osd.1 (osd.1) 366 : cluster 0 6.4 scrub ok 2024-03-11T21:47:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:04 smithi203 bash[23654]: cluster 2024-03-11T21:47:03.569781+0000 osd.2 (osd.2) 475 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:47:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:04 smithi203 bash[23654]: cluster 2024-03-11T21:47:03.572444+0000 osd.2 (osd.2) 476 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:47:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:04 smithi203 bash[23654]: cluster 2024-03-11T21:47:03.768163+0000 mgr.y (mgr.14550) 924 : cluster 0 pgmap v1587: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:05.288 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:05.288 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:05.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[29284]: cluster 2024-03-11T21:47:03.569781+0000 osd.2 (osd.2) 475 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:47:05.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[29284]: cluster 2024-03-11T21:47:03.572444+0000 osd.2 (osd.2) 476 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:47:05.309 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[29284]: cluster 2024-03-11T21:47:03.768163+0000 mgr.y (mgr.14550) 924 : cluster 0 pgmap v1587: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:05.310 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[21512]: cluster 2024-03-11T21:47:03.569781+0000 osd.2 (osd.2) 475 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:47:05.310 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[21512]: cluster 2024-03-11T21:47:03.572444+0000 osd.2 (osd.2) 476 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:47:05.310 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:04 smithi143 bash[21512]: cluster 2024-03-11T21:47:03.768163+0000 mgr.y (mgr.14550) 924 : cluster 0 pgmap v1587: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:06.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:05 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.465979+0000 osd.7 (osd.7) 562 : cluster 0 1.2 scrub starts 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:05 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.468522+0000 osd.7 (osd.7) 563 : cluster 0 1.2 scrub ok 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:05 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.482687+0000 osd.0 (osd.0) 568 : cluster 0 6.16 scrub starts 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:05 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.484918+0000 osd.0 (osd.0) 569 : cluster 0 6.16 scrub ok 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:06 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.586214+0000 osd.2 (osd.2) 477 : cluster 0 6.14 deep-scrub starts 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:06 smithi203 bash[23654]: cluster 2024-03-11T21:47:04.588559+0000 osd.2 (osd.2) 478 : cluster 0 6.14 deep-scrub ok 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:06 smithi203 bash[23654]: audit 2024-03-11T21:47:05.884703+0000 mon.a (mon.0) 4116 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:06 smithi203 bash[23654]: audit 2024-03-11T21:47:05.892106+0000 mon.a (mon.0) 4117 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.465979+0000 osd.7 (osd.7) 562 : cluster 0 1.2 scrub starts 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.468522+0000 osd.7 (osd.7) 563 : cluster 0 1.2 scrub ok 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.482687+0000 osd.0 (osd.0) 568 : cluster 0 6.16 scrub starts 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.484918+0000 osd.0 (osd.0) 569 : cluster 0 6.16 scrub ok 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.586214+0000 osd.2 (osd.2) 477 : cluster 0 6.14 deep-scrub starts 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: cluster 2024-03-11T21:47:04.588559+0000 osd.2 (osd.2) 478 : cluster 0 6.14 deep-scrub ok 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: audit 2024-03-11T21:47:05.884703+0000 mon.a (mon.0) 4116 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:06.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[21512]: audit 2024-03-11T21:47:05.892106+0000 mon.a (mon.0) 4117 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.465979+0000 osd.7 (osd.7) 562 : cluster 0 1.2 scrub starts 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.468522+0000 osd.7 (osd.7) 563 : cluster 0 1.2 scrub ok 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.482687+0000 osd.0 (osd.0) 568 : cluster 0 6.16 scrub starts 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.484918+0000 osd.0 (osd.0) 569 : cluster 0 6.16 scrub ok 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.586214+0000 osd.2 (osd.2) 477 : cluster 0 6.14 deep-scrub starts 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: cluster 2024-03-11T21:47:04.588559+0000 osd.2 (osd.2) 478 : cluster 0 6.14 deep-scrub ok 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: audit 2024-03-11T21:47:05.884703+0000 mon.a (mon.0) 4116 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:06.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:06 smithi143 bash[29284]: audit 2024-03-11T21:47:05.892106+0000 mon.a (mon.0) 4117 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[21512]: cluster 2024-03-11T21:47:05.768807+0000 mgr.y (mgr.14550) 925 : cluster 0 pgmap v1588: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:07.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[21512]: audit 2024-03-11T21:47:06.600717+0000 mon.a (mon.0) 4118 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[21512]: audit 2024-03-11T21:47:06.616652+0000 mon.a (mon.0) 4119 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[29284]: cluster 2024-03-11T21:47:05.768807+0000 mgr.y (mgr.14550) 925 : cluster 0 pgmap v1588: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:07.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[29284]: audit 2024-03-11T21:47:06.600717+0000 mon.a (mon.0) 4118 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:07 smithi143 bash[29284]: audit 2024-03-11T21:47:06.616652+0000 mon.a (mon.0) 4119 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:07 smithi203 bash[23654]: cluster 2024-03-11T21:47:05.768807+0000 mgr.y (mgr.14550) 925 : cluster 0 pgmap v1588: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:07 smithi203 bash[23654]: audit 2024-03-11T21:47:06.600717+0000 mon.a (mon.0) 4118 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:07 smithi203 bash[23654]: audit 2024-03-11T21:47:06.616652+0000 mon.a (mon.0) 4119 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:08.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: cluster 2024-03-11T21:47:06.126938+0000 osd.3 (osd.3) 579 : cluster 0 6.c scrub starts 2024-03-11T21:47:08.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: cluster 2024-03-11T21:47:06.129243+0000 osd.3 (osd.3) 580 : cluster 0 6.c scrub ok 2024-03-11T21:47:08.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: cluster 2024-03-11T21:47:06.844194+0000 osd.6 (osd.6) 465 : cluster 0 1.f scrub starts 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: cluster 2024-03-11T21:47:06.846578+0000 osd.6 (osd.6) 466 : cluster 0 1.f scrub ok 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: audit 2024-03-11T21:47:07.244352+0000 mon.c (mon.2) 1401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: audit 2024-03-11T21:47:07.246372+0000 mon.c (mon.2) 1402 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: audit 2024-03-11T21:47:07.257594+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[21512]: audit 2024-03-11T21:47:07.951704+0000 mon.c (mon.2) 1403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: cluster 2024-03-11T21:47:06.126938+0000 osd.3 (osd.3) 579 : cluster 0 6.c scrub starts 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: cluster 2024-03-11T21:47:06.129243+0000 osd.3 (osd.3) 580 : cluster 0 6.c scrub ok 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: cluster 2024-03-11T21:47:06.844194+0000 osd.6 (osd.6) 465 : cluster 0 1.f scrub starts 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: cluster 2024-03-11T21:47:06.846578+0000 osd.6 (osd.6) 466 : cluster 0 1.f scrub ok 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: audit 2024-03-11T21:47:07.244352+0000 mon.c (mon.2) 1401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:08.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: audit 2024-03-11T21:47:07.246372+0000 mon.c (mon.2) 1402 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:08.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: audit 2024-03-11T21:47:07.257594+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:08.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:08 smithi143 bash[29284]: audit 2024-03-11T21:47:07.951704+0000 mon.c (mon.2) 1403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: cluster 2024-03-11T21:47:06.126938+0000 osd.3 (osd.3) 579 : cluster 0 6.c scrub starts 2024-03-11T21:47:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: cluster 2024-03-11T21:47:06.129243+0000 osd.3 (osd.3) 580 : cluster 0 6.c scrub ok 2024-03-11T21:47:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: cluster 2024-03-11T21:47:06.844194+0000 osd.6 (osd.6) 465 : cluster 0 1.f scrub starts 2024-03-11T21:47:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: cluster 2024-03-11T21:47:06.846578+0000 osd.6 (osd.6) 466 : cluster 0 1.f scrub ok 2024-03-11T21:47:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: audit 2024-03-11T21:47:07.244352+0000 mon.c (mon.2) 1401 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:08.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: audit 2024-03-11T21:47:07.246372+0000 mon.c (mon.2) 1402 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:08.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: audit 2024-03-11T21:47:07.257594+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:47:08.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:08 smithi203 bash[23654]: audit 2024-03-11T21:47:07.951704+0000 mon.c (mon.2) 1403 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:09 smithi203 bash[23654]: cluster 2024-03-11T21:47:07.434388+0000 osd.0 (osd.0) 570 : cluster 0 5.3 scrub starts 2024-03-11T21:47:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:09 smithi203 bash[23654]: cluster 2024-03-11T21:47:07.437457+0000 osd.0 (osd.0) 571 : cluster 0 5.3 scrub ok 2024-03-11T21:47:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:09 smithi203 bash[23654]: cluster 2024-03-11T21:47:07.769491+0000 mgr.y (mgr.14550) 926 : cluster 0 pgmap v1589: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:09.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[21512]: cluster 2024-03-11T21:47:07.434388+0000 osd.0 (osd.0) 570 : cluster 0 5.3 scrub starts 2024-03-11T21:47:09.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[21512]: cluster 2024-03-11T21:47:07.437457+0000 osd.0 (osd.0) 571 : cluster 0 5.3 scrub ok 2024-03-11T21:47:09.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[21512]: cluster 2024-03-11T21:47:07.769491+0000 mgr.y (mgr.14550) 926 : cluster 0 pgmap v1589: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:09.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[29284]: cluster 2024-03-11T21:47:07.434388+0000 osd.0 (osd.0) 570 : cluster 0 5.3 scrub starts 2024-03-11T21:47:09.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[29284]: cluster 2024-03-11T21:47:07.437457+0000 osd.0 (osd.0) 571 : cluster 0 5.3 scrub ok 2024-03-11T21:47:09.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:09 smithi143 bash[29284]: cluster 2024-03-11T21:47:07.769491+0000 mgr.y (mgr.14550) 926 : cluster 0 pgmap v1589: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:09.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:47:10.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/3799527204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168107290 msgr2=0x7fd16810edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/3799527204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168107290 0x7fd16810edf0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fd16000b0d0 tx=0x7fd16000b3e0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/3799527204 shutdown_connections 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/3799527204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168107290 0x7fd16810edf0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/3799527204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168100670 0x7fd168100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/3799527204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168101020 0x7fd168106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/3799527204 >> 172.21.15.143:0/3799527204 conn(0x7fd168077380 msgr2=0x7fd1680fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/3799527204 shutdown_connections 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/3799527204 wait complete. 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 librados: starting msgr at 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 librados: starting objecter 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 Processor -- start 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- start start 2024-03-11T21:47:10.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 librados: setting wanted keys 2024-03-11T21:47:10.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 librados: calling monclient init 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168100670 0x7fd1681015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 0x7fd168102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168111650 con 0x7fd168101020 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1681114d0 con 0x7fd168100670 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1681117d0 con 0x7fd168107290 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35236/0 (socket says 172.21.15.143:35236) 2024-03-11T21:47:10.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 -- 172.21.15.143:0/908788705 learned_addr learned my addr 172.21.15.143:0/908788705 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16e942700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 0x7fd168102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16e141700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168100670 0x7fd1681015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 -- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 msgr2=0x7fd168102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 0x7fd168102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 -- 172.21.15.143:0/908788705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168100670 msgr2=0x7fd1681015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168100670 0x7fd1681015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 -- 172.21.15.143:0/908788705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1681a5fe0 con 0x7fd168101020 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16e942700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 0x7fd168102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd16d940700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fd16400c9b0 tx=0x7fd16400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 10 client.?.objecter ms_handle_connect 0x7fd168101020 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164007b00 con 0x7fd168101020 2024-03-11T21:47:10.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1681a62d0 con 0x7fd168101020 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fd164007ca0 con 0x7fd168101020 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164018660 con 0x7fd168101020 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 10 client.31149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1681a6810 con 0x7fd168101020 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 librados: init done 2024-03-11T21:47:10.026 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16804faa0 con 0x7fd168101020 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.034+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fd16400f450 con 0x7fd168101020 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 0x7fd154071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(988..988 src has 933..988) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fd16408ea30 con 0x7fd168101020 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 10 client.31149.objecter ms_dispatch 0x7fd168001550 osd_map(988..988 src has 933..988) v4 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 3 client.31149.objecter handle_osd_map got epochs [988,988] > 0 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 3 client.31149.objecter handle_osd_map decoding full epoch 988 2024-03-11T21:47:10.027 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.038+0000 7fd15f7fe700 20 client.31149.objecter dump_active .. 0 homeless 2024-03-11T21:47:10.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.042+0000 7fd16e141700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 0x7fd154071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:10.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.042+0000 7fd16e141700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 0x7fd154071ce0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fd158009680 tx=0x7fd158009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:10.029 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.042+0000 7fd15f7fe700 10 client.31149.objecter ms_handle_connect 0x7fd15406f820 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [], 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stdout: "complaint_time": 30, 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stdout: "num_blocked_ops": 0 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:47:10.030 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.042+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd164014030 con 0x7fd168101020 2024-03-11T21:47:10.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.144113+0000 osd.3 (osd.3) 581 : cluster 0 5.13 scrub starts 2024-03-11T21:47:10.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.146146+0000 osd.3 (osd.3) 582 : cluster 0 5.13 scrub ok 2024-03-11T21:47:10.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.426488+0000 osd.0 (osd.0) 572 : cluster 0 3.12 scrub starts 2024-03-11T21:47:10.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.428674+0000 osd.0 (osd.0) 573 : cluster 0 3.12 scrub ok 2024-03-11T21:47:10.198 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.955990+0000 osd.1 (osd.1) 367 : cluster 0 9.12 deep-scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:08.958325+0000 osd.1 (osd.1) 368 : cluster 0 9.12 deep-scrub ok 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.022466+0000 osd.4 (osd.4) 545 : cluster 0 6.1a scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.024822+0000 osd.4 (osd.4) 546 : cluster 0 6.1a scrub ok 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.144113+0000 osd.3 (osd.3) 581 : cluster 0 5.13 scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.146146+0000 osd.3 (osd.3) 582 : cluster 0 5.13 scrub ok 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.426488+0000 osd.0 (osd.0) 572 : cluster 0 3.12 scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.428674+0000 osd.0 (osd.0) 573 : cluster 0 3.12 scrub ok 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.955990+0000 osd.1 (osd.1) 367 : cluster 0 9.12 deep-scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:08.958325+0000 osd.1 (osd.1) 368 : cluster 0 9.12 deep-scrub ok 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.022466+0000 osd.4 (osd.4) 545 : cluster 0 6.1a scrub starts 2024-03-11T21:47:10.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:10 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.024822+0000 osd.4 (osd.4) 546 : cluster 0 6.1a scrub ok 2024-03-11T21:47:10.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:10.270+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fd1681a6ad0 con 0x7fd168101020 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.144113+0000 osd.3 (osd.3) 581 : cluster 0 5.13 scrub starts 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.146146+0000 osd.3 (osd.3) 582 : cluster 0 5.13 scrub ok 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.426488+0000 osd.0 (osd.0) 572 : cluster 0 3.12 scrub starts 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.428674+0000 osd.0 (osd.0) 573 : cluster 0 3.12 scrub ok 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.955990+0000 osd.1 (osd.1) 367 : cluster 0 9.12 deep-scrub starts 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:08.958325+0000 osd.1 (osd.1) 368 : cluster 0 9.12 deep-scrub ok 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.022466+0000 osd.4 (osd.4) 545 : cluster 0 6.1a scrub starts 2024-03-11T21:47:10.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:10 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.024822+0000 osd.4 (osd.4) 546 : cluster 0 6.1a scrub ok 2024-03-11T21:47:10.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:47:11.012 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.022+0000 7fd15f7fe700 1 -- 172.21.15.143:0/908788705 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v989) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd16405bac0 con 0x7fd168101020 2024-03-11T21:47:11.015 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.026+0000 7fd1703a5700 10 librados: watch_flush enter 2024-03-11T21:47:11.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.026+0000 7fd1703a5700 10 librados: watch_flush exit 2024-03-11T21:47:11.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.026+0000 7fd1703a5700 20 client.31149.objecter shutdown clearing up homeless session... 2024-03-11T21:47:11.016 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.026+0000 7fd1703a5700 10 client.31149.objecter successfully canceled tick 2024-03-11T21:47:11.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 msgr2=0x7fd154071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:11.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 0x7fd154071ce0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fd158009680 tx=0x7fd158009260 comp rx=0 tx=0).stop 2024-03-11T21:47:11.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 msgr2=0x7fd168101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:11.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fd16400c9b0 tx=0x7fd16400ccc0 comp rx=0 tx=0).stop 2024-03-11T21:47:11.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 shutdown_connections 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fd15406f820 0x7fd154071ce0 unknown :-1 s=CLOSED pgs=3502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd168107290 0x7fd168102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd168100670 0x7fd1681015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 --2- 172.21.15.143:0/908788705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd168101020 0x7fd168101b30 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 >> 172.21.15.143:0/908788705 conn(0x7fd168077380 msgr2=0x7fd168104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 shutdown_connections 2024-03-11T21:47:11.019 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 -- 172.21.15.143:0/908788705 wait complete. 2024-03-11T21:47:11.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:11.030+0000 7fd1703a5700 1 librados: shutdown 2024-03-11T21:47:11.025 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.171026+0000 osd.3 (osd.3) 583 : cluster 0 2.1e scrub starts 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.173249+0000 osd.3 (osd.3) 584 : cluster 0 2.1e scrub ok 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: audit 2024-03-11T21:47:09.295738+0000 mgr.y (mgr.14550) 927 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.383804+0000 osd.7 (osd.7) 564 : cluster 0 1.9 scrub starts 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.386450+0000 osd.7 (osd.7) 565 : cluster 0 1.9 scrub ok 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.631195+0000 osd.2 (osd.2) 479 : cluster 0 2.10 scrub starts 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.633650+0000 osd.2 (osd.2) 480 : cluster 0 2.10 scrub ok 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.771349+0000 mgr.y (mgr.14550) 928 : cluster 0 pgmap v1590: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.997343+0000 osd.4 (osd.4) 547 : cluster 0 6.2 scrub starts 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:09.999542+0000 osd.4 (osd.4) 548 : cluster 0 6.2 scrub ok 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: audit 2024-03-11T21:47:10.273087+0000 mon.a (mon.0) 4121 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:11.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.338623+0000 osd.7 (osd.7) 566 : cluster 0 3.13 scrub starts 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.342628+0000 osd.7 (osd.7) 567 : cluster 0 3.13 scrub ok 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.171026+0000 osd.3 (osd.3) 583 : cluster 0 2.1e scrub starts 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.173249+0000 osd.3 (osd.3) 584 : cluster 0 2.1e scrub ok 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: audit 2024-03-11T21:47:09.295738+0000 mgr.y (mgr.14550) 927 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.383804+0000 osd.7 (osd.7) 564 : cluster 0 1.9 scrub starts 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.386450+0000 osd.7 (osd.7) 565 : cluster 0 1.9 scrub ok 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.631195+0000 osd.2 (osd.2) 479 : cluster 0 2.10 scrub starts 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.633650+0000 osd.2 (osd.2) 480 : cluster 0 2.10 scrub ok 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.771349+0000 mgr.y (mgr.14550) 928 : cluster 0 pgmap v1590: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.997343+0000 osd.4 (osd.4) 547 : cluster 0 6.2 scrub starts 2024-03-11T21:47:11.088 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:09.999542+0000 osd.4 (osd.4) 548 : cluster 0 6.2 scrub ok 2024-03-11T21:47:11.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: audit 2024-03-11T21:47:10.273087+0000 mon.a (mon.0) 4121 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:11.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.338623+0000 osd.7 (osd.7) 566 : cluster 0 3.13 scrub starts 2024-03-11T21:47:11.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:11 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.342628+0000 osd.7 (osd.7) 567 : cluster 0 3.13 scrub ok 2024-03-11T21:47:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.171026+0000 osd.3 (osd.3) 583 : cluster 0 2.1e scrub starts 2024-03-11T21:47:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.173249+0000 osd.3 (osd.3) 584 : cluster 0 2.1e scrub ok 2024-03-11T21:47:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: audit 2024-03-11T21:47:09.295738+0000 mgr.y (mgr.14550) 927 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.383804+0000 osd.7 (osd.7) 564 : cluster 0 1.9 scrub starts 2024-03-11T21:47:11.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.386450+0000 osd.7 (osd.7) 565 : cluster 0 1.9 scrub ok 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.631195+0000 osd.2 (osd.2) 479 : cluster 0 2.10 scrub starts 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.633650+0000 osd.2 (osd.2) 480 : cluster 0 2.10 scrub ok 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.771349+0000 mgr.y (mgr.14550) 928 : cluster 0 pgmap v1590: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.997343+0000 osd.4 (osd.4) 547 : cluster 0 6.2 scrub starts 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:09.999542+0000 osd.4 (osd.4) 548 : cluster 0 6.2 scrub ok 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: audit 2024-03-11T21:47:10.273087+0000 mon.a (mon.0) 4121 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.338623+0000 osd.7 (osd.7) 566 : cluster 0 3.13 scrub starts 2024-03-11T21:47:11.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:11 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.342628+0000 osd.7 (osd.7) 567 : cluster 0 3.13 scrub ok 2024-03-11T21:47:11.961 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:47:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.161112+0000 osd.3 (osd.3) 585 : cluster 0 5.17 deep-scrub starts 2024-03-11T21:47:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.163062+0000 osd.3 (osd.3) 586 : cluster 0 5.17 deep-scrub ok 2024-03-11T21:47:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.603016+0000 osd.2 (osd.2) 481 : cluster 0 5.6 scrub starts 2024-03-11T21:47:12.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.605283+0000 osd.2 (osd.2) 482 : cluster 0 5.6 scrub ok 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.892194+0000 osd.6 (osd.6) 467 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:10.897144+0000 osd.6 (osd.6) 468 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.008394+0000 osd.1 (osd.1) 369 : cluster 0 1.13 scrub starts 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.011265+0000 osd.1 (osd.1) 370 : cluster 0 1.13 scrub ok 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: audit 2024-03-11T21:47:11.025868+0000 mon.a (mon.0) 4122 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.029947+0000 mon.a (mon.0) 4123 : cluster 0 osdmap e989: 8 total, 8 up, 8 in 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.290569+0000 osd.7 (osd.7) 568 : cluster 0 9.17 scrub starts 2024-03-11T21:47:12.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:12 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.297348+0000 osd.7 (osd.7) 569 : cluster 0 9.17 scrub ok 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.161112+0000 osd.3 (osd.3) 585 : cluster 0 5.17 deep-scrub starts 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.163062+0000 osd.3 (osd.3) 586 : cluster 0 5.17 deep-scrub ok 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.603016+0000 osd.2 (osd.2) 481 : cluster 0 5.6 scrub starts 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.605283+0000 osd.2 (osd.2) 482 : cluster 0 5.6 scrub ok 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.892194+0000 osd.6 (osd.6) 467 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:47:12.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:10.897144+0000 osd.6 (osd.6) 468 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.008394+0000 osd.1 (osd.1) 369 : cluster 0 1.13 scrub starts 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.011265+0000 osd.1 (osd.1) 370 : cluster 0 1.13 scrub ok 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: audit 2024-03-11T21:47:11.025868+0000 mon.a (mon.0) 4122 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.029947+0000 mon.a (mon.0) 4123 : cluster 0 osdmap e989: 8 total, 8 up, 8 in 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.290569+0000 osd.7 (osd.7) 568 : cluster 0 9.17 scrub starts 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.297348+0000 osd.7 (osd.7) 569 : cluster 0 9.17 scrub ok 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.161112+0000 osd.3 (osd.3) 585 : cluster 0 5.17 deep-scrub starts 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.163062+0000 osd.3 (osd.3) 586 : cluster 0 5.17 deep-scrub ok 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.603016+0000 osd.2 (osd.2) 481 : cluster 0 5.6 scrub starts 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.605283+0000 osd.2 (osd.2) 482 : cluster 0 5.6 scrub ok 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.892194+0000 osd.6 (osd.6) 467 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:47:12.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:10.897144+0000 osd.6 (osd.6) 468 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.008394+0000 osd.1 (osd.1) 369 : cluster 0 1.13 scrub starts 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.011265+0000 osd.1 (osd.1) 370 : cluster 0 1.13 scrub ok 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: audit 2024-03-11T21:47:11.025868+0000 mon.a (mon.0) 4122 : audit 1 from='client.? 172.21.15.143:0/908788705' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.029947+0000 mon.a (mon.0) 4123 : cluster 0 osdmap e989: 8 total, 8 up, 8 in 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.290569+0000 osd.7 (osd.7) 568 : cluster 0 9.17 scrub starts 2024-03-11T21:47:12.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:12 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.297348+0000 osd.7 (osd.7) 569 : cluster 0 9.17 scrub ok 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.771900+0000 mgr.y (mgr.14550) 929 : cluster 0 pgmap v1592: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.861653+0000 osd.6 (osd.6) 469 : cluster 0 6.a scrub starts 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:11.863722+0000 osd.6 (osd.6) 470 : cluster 0 6.a scrub ok 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.036426+0000 mon.a (mon.0) 4124 : cluster 0 osdmap e990: 8 total, 8 up, 8 in 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.289190+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub starts 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.292984+0000 osd.7 (osd.7) 571 : cluster 0 2.8 scrub ok 2024-03-11T21:47:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.614668+0000 osd.2 (osd.2) 483 : cluster 0 1.8 scrub starts 2024-03-11T21:47:13.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:13 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.617644+0000 osd.2 (osd.2) 484 : cluster 0 1.8 scrub ok 2024-03-11T21:47:13.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.771900+0000 mgr.y (mgr.14550) 929 : cluster 0 pgmap v1592: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.861653+0000 osd.6 (osd.6) 469 : cluster 0 6.a scrub starts 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:11.863722+0000 osd.6 (osd.6) 470 : cluster 0 6.a scrub ok 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.036426+0000 mon.a (mon.0) 4124 : cluster 0 osdmap e990: 8 total, 8 up, 8 in 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.289190+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub starts 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.292984+0000 osd.7 (osd.7) 571 : cluster 0 2.8 scrub ok 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.614668+0000 osd.2 (osd.2) 483 : cluster 0 1.8 scrub starts 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.617644+0000 osd.2 (osd.2) 484 : cluster 0 1.8 scrub ok 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.771900+0000 mgr.y (mgr.14550) 929 : cluster 0 pgmap v1592: 184 pgs: 184 active+clean; 582 KiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.861653+0000 osd.6 (osd.6) 469 : cluster 0 6.a scrub starts 2024-03-11T21:47:13.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:11.863722+0000 osd.6 (osd.6) 470 : cluster 0 6.a scrub ok 2024-03-11T21:47:13.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.036426+0000 mon.a (mon.0) 4124 : cluster 0 osdmap e990: 8 total, 8 up, 8 in 2024-03-11T21:47:13.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.289190+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub starts 2024-03-11T21:47:13.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.292984+0000 osd.7 (osd.7) 571 : cluster 0 2.8 scrub ok 2024-03-11T21:47:13.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.614668+0000 osd.2 (osd.2) 483 : cluster 0 1.8 scrub starts 2024-03-11T21:47:13.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:13 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.617644+0000 osd.2 (osd.2) 484 : cluster 0 1.8 scrub ok 2024-03-11T21:47:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:14 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.534484+0000 osd.0 (osd.0) 574 : cluster 0 2.1f scrub starts 2024-03-11T21:47:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:14 smithi203 bash[23654]: cluster 2024-03-11T21:47:12.537553+0000 osd.0 (osd.0) 575 : cluster 0 2.1f scrub ok 2024-03-11T21:47:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:14 smithi203 bash[23654]: cluster 2024-03-11T21:47:13.257388+0000 osd.7 (osd.7) 572 : cluster 0 9.e scrub starts 2024-03-11T21:47:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:14 smithi203 bash[23654]: cluster 2024-03-11T21:47:13.262116+0000 osd.7 (osd.7) 573 : cluster 0 9.e scrub ok 2024-03-11T21:47:14.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.534484+0000 osd.0 (osd.0) 574 : cluster 0 2.1f scrub starts 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[21512]: cluster 2024-03-11T21:47:12.537553+0000 osd.0 (osd.0) 575 : cluster 0 2.1f scrub ok 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[21512]: cluster 2024-03-11T21:47:13.257388+0000 osd.7 (osd.7) 572 : cluster 0 9.e scrub starts 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[21512]: cluster 2024-03-11T21:47:13.262116+0000 osd.7 (osd.7) 573 : cluster 0 9.e scrub ok 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.534484+0000 osd.0 (osd.0) 574 : cluster 0 2.1f scrub starts 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[29284]: cluster 2024-03-11T21:47:12.537553+0000 osd.0 (osd.0) 575 : cluster 0 2.1f scrub ok 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[29284]: cluster 2024-03-11T21:47:13.257388+0000 osd.7 (osd.7) 572 : cluster 0 9.e scrub starts 2024-03-11T21:47:14.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:14 smithi143 bash[29284]: cluster 2024-03-11T21:47:13.262116+0000 osd.7 (osd.7) 573 : cluster 0 9.e scrub ok 2024-03-11T21:47:15.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:13.773726+0000 mgr.y (mgr.14550) 930 : cluster 0 pgmap v1594: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.053539+0000 osd.4 (osd.4) 549 : cluster 0 5.f scrub starts 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.055710+0000 osd.4 (osd.4) 550 : cluster 0 5.f scrub ok 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.232320+0000 osd.7 (osd.7) 574 : cluster 0 2.1c scrub starts 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.235165+0000 osd.7 (osd.7) 575 : cluster 0 2.1c scrub ok 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.609781+0000 osd.2 (osd.2) 485 : cluster 0 5.8 scrub starts 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[29284]: cluster 2024-03-11T21:47:14.612247+0000 osd.2 (osd.2) 486 : cluster 0 5.8 scrub ok 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:13.773726+0000 mgr.y (mgr.14550) 930 : cluster 0 pgmap v1594: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.053539+0000 osd.4 (osd.4) 549 : cluster 0 5.f scrub starts 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.055710+0000 osd.4 (osd.4) 550 : cluster 0 5.f scrub ok 2024-03-11T21:47:15.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.232320+0000 osd.7 (osd.7) 574 : cluster 0 2.1c scrub starts 2024-03-11T21:47:15.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.235165+0000 osd.7 (osd.7) 575 : cluster 0 2.1c scrub ok 2024-03-11T21:47:15.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.609781+0000 osd.2 (osd.2) 485 : cluster 0 5.8 scrub starts 2024-03-11T21:47:15.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:15 smithi143 bash[21512]: cluster 2024-03-11T21:47:14.612247+0000 osd.2 (osd.2) 486 : cluster 0 5.8 scrub ok 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:13.773726+0000 mgr.y (mgr.14550) 930 : cluster 0 pgmap v1594: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.053539+0000 osd.4 (osd.4) 549 : cluster 0 5.f scrub starts 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.055710+0000 osd.4 (osd.4) 550 : cluster 0 5.f scrub ok 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.232320+0000 osd.7 (osd.7) 574 : cluster 0 2.1c scrub starts 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.235165+0000 osd.7 (osd.7) 575 : cluster 0 2.1c scrub ok 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.609781+0000 osd.2 (osd.2) 485 : cluster 0 5.8 scrub starts 2024-03-11T21:47:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:15 smithi203 bash[23654]: cluster 2024-03-11T21:47:14.612247+0000 osd.2 (osd.2) 486 : cluster 0 5.8 scrub ok 2024-03-11T21:47:15.627 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:16.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[29284]: cluster 2024-03-11T21:47:15.014392+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub starts 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[29284]: cluster 2024-03-11T21:47:15.017021+0000 osd.1 (osd.1) 372 : cluster 0 1.3 scrub ok 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[29284]: cluster 2024-03-11T21:47:15.092313+0000 osd.4 (osd.4) 551 : cluster 0 6.e scrub starts 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[29284]: cluster 2024-03-11T21:47:15.095003+0000 osd.4 (osd.4) 552 : cluster 0 6.e scrub ok 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[21512]: cluster 2024-03-11T21:47:15.014392+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub starts 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[21512]: cluster 2024-03-11T21:47:15.017021+0000 osd.1 (osd.1) 372 : cluster 0 1.3 scrub ok 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[21512]: cluster 2024-03-11T21:47:15.092313+0000 osd.4 (osd.4) 551 : cluster 0 6.e scrub starts 2024-03-11T21:47:16.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:16 smithi143 bash[21512]: cluster 2024-03-11T21:47:15.095003+0000 osd.4 (osd.4) 552 : cluster 0 6.e scrub ok 2024-03-11T21:47:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:16 smithi203 bash[23654]: cluster 2024-03-11T21:47:15.014392+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub starts 2024-03-11T21:47:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:16 smithi203 bash[23654]: cluster 2024-03-11T21:47:15.017021+0000 osd.1 (osd.1) 372 : cluster 0 1.3 scrub ok 2024-03-11T21:47:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:16 smithi203 bash[23654]: cluster 2024-03-11T21:47:15.092313+0000 osd.4 (osd.4) 551 : cluster 0 6.e scrub starts 2024-03-11T21:47:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:16 smithi203 bash[23654]: cluster 2024-03-11T21:47:15.095003+0000 osd.4 (osd.4) 552 : cluster 0 6.e scrub ok 2024-03-11T21:47:16.648 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:17.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:17 smithi143 bash[21512]: cluster 2024-03-11T21:47:15.774915+0000 mgr.y (mgr.14550) 931 : cluster 0 pgmap v1595: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:17.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:17 smithi143 bash[29284]: cluster 2024-03-11T21:47:15.774915+0000 mgr.y (mgr.14550) 931 : cluster 0 pgmap v1595: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:17.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:17 smithi203 bash[23654]: cluster 2024-03-11T21:47:15.774915+0000 mgr.y (mgr.14550) 931 : cluster 0 pgmap v1595: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:19.198 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[29284]: cluster 2024-03-11T21:47:17.775860+0000 mgr.y (mgr.14550) 932 : cluster 0 pgmap v1596: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[29284]: cluster 2024-03-11T21:47:17.988013+0000 osd.1 (osd.1) 373 : cluster 0 2.2 scrub starts 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[29284]: cluster 2024-03-11T21:47:17.989692+0000 osd.1 (osd.1) 374 : cluster 0 2.2 scrub ok 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[29284]: cluster 2024-03-11T21:47:18.195531+0000 osd.7 (osd.7) 576 : cluster 0 9.11 scrub starts 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[29284]: cluster 2024-03-11T21:47:18.198551+0000 osd.7 (osd.7) 577 : cluster 0 9.11 scrub ok 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[21512]: cluster 2024-03-11T21:47:17.775860+0000 mgr.y (mgr.14550) 932 : cluster 0 pgmap v1596: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[21512]: cluster 2024-03-11T21:47:17.988013+0000 osd.1 (osd.1) 373 : cluster 0 2.2 scrub starts 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[21512]: cluster 2024-03-11T21:47:17.989692+0000 osd.1 (osd.1) 374 : cluster 0 2.2 scrub ok 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[21512]: cluster 2024-03-11T21:47:18.195531+0000 osd.7 (osd.7) 576 : cluster 0 9.11 scrub starts 2024-03-11T21:47:19.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:19 smithi143 bash[21512]: cluster 2024-03-11T21:47:18.198551+0000 osd.7 (osd.7) 577 : cluster 0 9.11 scrub ok 2024-03-11T21:47:19.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:47:19.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[23654]: cluster 2024-03-11T21:47:17.775860+0000 mgr.y (mgr.14550) 932 : cluster 0 pgmap v1596: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:19.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[23654]: cluster 2024-03-11T21:47:17.988013+0000 osd.1 (osd.1) 373 : cluster 0 2.2 scrub starts 2024-03-11T21:47:19.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[23654]: cluster 2024-03-11T21:47:17.989692+0000 osd.1 (osd.1) 374 : cluster 0 2.2 scrub ok 2024-03-11T21:47:19.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[23654]: cluster 2024-03-11T21:47:18.195531+0000 osd.7 (osd.7) 576 : cluster 0 9.11 scrub starts 2024-03-11T21:47:19.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:19 smithi203 bash[23654]: cluster 2024-03-11T21:47:18.198551+0000 osd.7 (osd.7) 577 : cluster 0 9.11 scrub ok 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:18.528922+0000 osd.0 (osd.0) 576 : cluster 0 9.1e scrub starts 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:18.533931+0000 osd.0 (osd.0) 577 : cluster 0 9.1e scrub ok 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.120520+0000 osd.4 (osd.4) 553 : cluster 0 5.18 deep-scrub starts 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.122832+0000 osd.4 (osd.4) 554 : cluster 0 5.18 deep-scrub ok 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.147464+0000 osd.7 (osd.7) 578 : cluster 0 2.b scrub starts 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.149653+0000 osd.7 (osd.7) 579 : cluster 0 2.b scrub ok 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.345246+0000 osd.2 (osd.2) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.345681+0000 osd.2 (osd.2) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:18.528922+0000 osd.0 (osd.0) 576 : cluster 0 9.1e scrub starts 2024-03-11T21:47:20.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:18.533931+0000 osd.0 (osd.0) 577 : cluster 0 9.1e scrub ok 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.120520+0000 osd.4 (osd.4) 553 : cluster 0 5.18 deep-scrub starts 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.122832+0000 osd.4 (osd.4) 554 : cluster 0 5.18 deep-scrub ok 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.147464+0000 osd.7 (osd.7) 578 : cluster 0 2.b scrub starts 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.149653+0000 osd.7 (osd.7) 579 : cluster 0 2.b scrub ok 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.345246+0000 osd.2 (osd.2) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:20.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:20 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.345681+0000 osd.2 (osd.2) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:20.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:18.528922+0000 osd.0 (osd.0) 576 : cluster 0 9.1e scrub starts 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:18.533931+0000 osd.0 (osd.0) 577 : cluster 0 9.1e scrub ok 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.120520+0000 osd.4 (osd.4) 553 : cluster 0 5.18 deep-scrub starts 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.122832+0000 osd.4 (osd.4) 554 : cluster 0 5.18 deep-scrub ok 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.147464+0000 osd.7 (osd.7) 578 : cluster 0 2.b scrub starts 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.149653+0000 osd.7 (osd.7) 579 : cluster 0 2.b scrub ok 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.345246+0000 osd.2 (osd.2) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:20 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.345681+0000 osd.2 (osd.2) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:45 314.4 314:20000000::::head [pgnls start_epoch 694] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e694)", 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 459.823956092, 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.010032127, 2024-03-11T21:47:21.319 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 45 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.320 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505058+0000", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505061+0000", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.9689999999999999e-06 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505074+0000", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3366e-05 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.321 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505105+0000", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0725999999999997e-05 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505123+0000", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7966e-05 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505241+0000", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000117896 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.322 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.505265+0000", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4635e-05 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.514780+0000", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0095147430000000009 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.514928+0000", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014781500000000001 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:41.515090+0000", 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00016201099999999999 2024-03-11T21:47:21.323 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:183 315.18 315:18000000::::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 451.75910055700001, 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0056069010000000001, 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.324 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 183 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569913+0000", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569916+0000", 2024-03-11T21:47:21.325 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5380000000000001e-06 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569926+0000", 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7419999999999992e-06 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569946+0000", 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0543e-05 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.326 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.569956+0000", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0026e-05 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.570029+0000", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.3238000000000006e-05 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.570036+0000", 2024-03-11T21:47:21.327 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.781e-06 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575396+0000", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0053596629999999998 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575419+0000", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3620000000000001e-05 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:39:49.575520+0000", 2024-03-11T21:47:21.328 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00010075 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1064 321.17 321:e8000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 382.68955798299999, 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0059295169999999996, 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.329 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1064 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639456+0000", 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.330 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639460+0000", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8199999999999998e-06 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639477+0000", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7003e-05 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639515+0000", 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.8584999999999997e-05 2024-03-11T21:47:21.331 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639541+0000", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5503e-05 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639658+0000", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011668899999999999 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.639664+0000", 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.998e-06 2024-03-11T21:47:21.332 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645183+0000", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0055196500000000001 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645212+0000", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8430999999999999e-05 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:40:58.645385+0000", 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000173838 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1233 322.14 322:28000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 369.69193942099997, 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007289383, 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1233 2024-03-11T21:47:21.334 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637074+0000", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637077+0000", 2024-03-11T21:47:21.335 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.1019999999999999e-06 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637086+0000", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.8800000000000003e-06 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637107+0000", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0761e-05 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637116+0000", 2024-03-11T21:47:21.336 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.6659999999999997e-06 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637229+0000", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011346899999999999 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.637239+0000", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.5189999999999998e-06 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644264+0000", 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0070247779999999998 2024-03-11T21:47:21.337 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644287+0000", 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.376e-05 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:11.644364+0000", 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6447999999999997e-05 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.338 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(9.4,QREQUEST,e:804,from:1,reservation_nonce:10)", 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 335.04865202500002, 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0060827479999999998, 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.339 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280362+0000", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280364+0000", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.0980000000000001e-06 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280376+0000", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.186e-05 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.340 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280401+0000", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4698999999999998e-05 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280414+0000", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3743e-05 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280546+0000", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013215699999999999 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.280558+0000", 2024-03-11T21:47:21.341 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.187e-05 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286364+0000", 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0058053419999999998 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286388+0000", 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.3997000000000001e-05 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.342 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.286445+0000", 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.6981999999999998e-05 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(osd.4.0:1431 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'1, mlcod=0'0)", 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 334.73494996900001, 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0053908690000000004, 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594064+0000", 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0699999999999998e-07 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.344 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594071+0000", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0010000000000004e-06 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594096+0000", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.4499e-05 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594107+0000", 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0920999999999999e-05 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.345 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594184+0000", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.6915000000000005e-05 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.594209+0000", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5340000000000001e-05 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599371+0000", 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0051624430000000001 2024-03-11T21:47:21.346 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.599455+0000", 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.3343000000000005e-05 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.41921.0:1794 326.12 e805/801 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head v 805'2, mlcod=0'0)", 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 334.731927063, 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.015262123000000001, 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.347 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597087+0000", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.597088+0000", 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0249999999999999e-06 2024-03-11T21:47:21.348 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604616+0000", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0075278020000000001 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604631+0000", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5469e-05 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.604641+0000", 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.4579999999999999e-06 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.349 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.605630+0000", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000989284 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.605661+0000", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.0975000000000001e-05 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612289+0000", 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0066284650000000001 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.350 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:46.612349+0000", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.9645000000000002e-05 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:1831 326.10 326:08000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 332.734070288, 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0045681610000000003, 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.351 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 1831 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594944+0000", 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.352 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594947+0000", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1089999999999998e-06 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594961+0000", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3849000000000001e-05 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.594987+0000", 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.6274999999999998e-05 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.353 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595000+0000", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3328e-05 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595193+0000", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019239100000000001 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.595206+0000", 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3543000000000001e-05 2024-03-11T21:47:21.354 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599283+0000", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0040770390000000002 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599332+0000", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.8374000000000001e-05 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:48.599512+0000", 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00018025300000000001 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(5.1c,QREQUEST,e:813,from:4,reservation_nonce:19)", 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 326.59372174600003, 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007466922, 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.356 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735292+0000", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735294+0000", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9800000000000001e-06 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.357 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735304+0000", 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.7429999999999997e-06 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735331+0000", 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.743e-05 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735347+0000", 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6052000000000001e-05 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.358 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735461+0000", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00011319699999999999 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.735472+0000", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1195999999999999e-05 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742601+0000", 2024-03-11T21:47:21.359 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.007128962 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742642+0000", 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.1245000000000002e-05 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:41:54.742759+0000", 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000117117 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.360 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2151 329.14 329:28000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 290.72976138799999, 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0091880580000000007, 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.361 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2151 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599253+0000", 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.362 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599257+0000", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0430000000000001e-06 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599274+0000", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.7410000000000001e-05 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599311+0000", 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6906000000000002e-05 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.363 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599329+0000", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8246000000000001e-05 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599538+0000", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00020909100000000001 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.599550+0000", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1982e-05 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.364 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608239+0000", 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.008689235 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608293+0000", 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.3168999999999999e-05 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:30.608441+0000", 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014797599999999999 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.365 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2216 329.14 329:28000000:%01:::head [pgnls start_epoch 844] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e844)", 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 288.724435997, 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0089340849999999996, 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2216 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.366 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604578+0000", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604581+0000", 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1530000000000001e-06 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.367 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604598+0000", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6719e-05 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604635+0000", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7562999999999999e-05 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604654+0000", 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8709999999999999e-05 2024-03-11T21:47:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604798+0000", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014399899999999999 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.604809+0000", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0482999999999999e-05 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.611926+0000", 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0071172049999999997 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.369 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.612036+0000", 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00010983400000000001 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:32.613512+0000", 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0014764190000000001 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2312 313.4s0 313:2473172f:test-rados-api-smithi143-71773-145::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e847)", 2024-03-11T21:47:21.370 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 285.58926793799998, 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.042486758999999999, 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2312 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739746+0000", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739749+0000", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.1429999999999998e-06 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739769+0000", 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9596e-05 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.372 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739801+0000", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.2595999999999998e-05 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.739828+0000", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.635e-05 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.740021+0000", 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00019352400000000001 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.373 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.740044+0000", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2379999999999999e-05 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.753843+0000", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.013799567 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.754197+0000", 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00035403000000000002 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.374 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.755533+0000", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001335911 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_committed", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.757815+0000", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.002282362 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.782133+0000", 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.024317314 2024-03-11T21:47:21.375 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:35.782233+0000", 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9986000000000001e-05 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(osd.1.295:43 313.4s0 313:2473172f:test-rados-api-smithi143-71773-145::foo:head [read 0~1] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e850)", 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:47:21.376 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 282.74611599899998, 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012771879999999999, 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.377 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582898+0000", 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582901+0000", 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.117e-06 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.378 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582916+0000", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4956e-05 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582957+0000", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.0802e-05 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.582984+0000", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.703e-05 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.583133+0000", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00014922899999999999 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.583145+0000", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1476e-05 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.591781+0000", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0086360289999999999 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.592078+0000", 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00029759799999999999 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:42:38.595670+0000", 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0035916429999999998 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4144 3.b 3:d4f18cce:::obj_delete_at_hint.0000000107:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.612143+0000", 2024-03-11T21:47:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 249.71687127199999, 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.020062612000000001, 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4144 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612143+0000", 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612143+0000", 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612143+0000", 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.54e-07 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612148+0000", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.7720000000000002e-06 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612159+0000", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0987e-05 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612166+0000", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.421e-06 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612208+0000", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.2827000000000002e-05 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612275+0000", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.6846000000000003e-05 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.612429+0000", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00015332900000000001 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.613292+0000", 2024-03-11T21:47:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00086304399999999999 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.622916+0000", 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0096247820000000001 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.632075+0000", 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0091585999999999994 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.632143+0000", 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.7774000000000005e-05 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.632205+0000", 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2376e-05 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.14427.0:4159 3.2 3:47403372:::obj_delete_at_hint.0000000112:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:11.659628+0000", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 249.66938620900001, 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.012945497, 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.14427", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4159 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659628+0000", 2024-03-11T21:47:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659628+0000", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659628+0000", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0100000000000004e-07 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659633+0000", 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4469999999999999e-06 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659643+0000", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9240000000000002e-06 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659657+0000", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3771000000000001e-05 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659719+0000", 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2915000000000003e-05 2024-03-11T21:47:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.659819+0000", 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9819000000000001e-05 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.660051+0000", 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00023200400000000001 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.660687+0000", 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000635349 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.660831+0000", 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000144604 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672428+0000", 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011596746 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.392 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672510+0000", 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 8.2260000000000002e-05 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:11.672573+0000", 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.2957e-05 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.41921.0:2808 331.10 331:08000000:%01:::head [pgnls start_epoch 866] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e866)", 2024-03-11T21:47:21.393 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:13.613984+0000", 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 247.715030396, 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.005763331, 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.41921", 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2808 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.394 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.613984+0000", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.613984+0000", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.613987+0000", 2024-03-11T21:47:21.395 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3299999999999999e-06 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.614003+0000", 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.6062e-05 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.614037+0000", 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.3673000000000001e-05 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.396 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.614064+0000", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7858999999999999e-05 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.614200+0000", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00013519699999999999 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.614210+0000", 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.022e-05 2024-03-11T21:47:21.397 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.619400+0000", 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0051896690000000001 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.619492+0000", 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.2904000000000004e-05 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.398 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:13.619747+0000", 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000254417 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_repop(client.41921.0:3122 334.14 e877/876 334:283cf3a8:test-rados-api-smithi143-71773-153::foo:head v 877'1, mlcod=0'0)", 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:43:24.651495+0000", 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 236.67751856699999, 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.011618702, 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.399 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651495+0000", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651495+0000", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651496+0000", 2024-03-11T21:47:21.400 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.9099999999999991e-07 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651508+0000", 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1666e-05 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651527+0000", 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.9225e-05 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.401 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.651543+0000", 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.5319e-05 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.653588+0000", 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0020456150000000002 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.653626+0000", 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.7787000000000003e-05 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.402 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.663022+0000", 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0093965479999999994 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:43:24.663114+0000", 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1551000000000005e-05 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.403 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "osd_op(client.42191.0:2 337.0 337:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e913)", 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:44:26.621902+0000", 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 174.70711150599999, 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0348605449999999, 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.42191", 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.143:0/3302230881", 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 2 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.404 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621902+0000", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621902+0000", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621904+0000", 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.8500000000000001e-06 2024-03-11T21:47:21.405 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621911+0000", 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.0779999999999998e-06 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621923+0000", 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1324e-05 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.406 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.621935+0000", 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.2131e-05 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.623192+0000", 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.001257083 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered", 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:26.623202+0000", 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.048e-05 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.407 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.648555+0000", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0253530120000001 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.648679+0000", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00012331000000000001 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.648861+0000", 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000181929 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.408 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op_commit", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.649775+0000", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.00091402400000000004 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.656581+0000", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0068060639999999997 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.656631+0000", 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 5.0093999999999998e-05 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.409 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent", 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.656703+0000", 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.1940999999999994e-05 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:44:27.656763+0000", 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.0225e-05 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.410 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(6.2,QREQUEST,e:969,from:4,reservation_nonce:23)", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:45:46.874670+0000", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 94.454344301000006, 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0049008289999999998, 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874670+0000", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874670+0000", 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874672+0000", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.2500000000000001e-06 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874685+0000", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.3403e-05 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874722+0000", 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 3.6788000000000001e-05 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.412 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874747+0000", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.5395999999999999e-05 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874839+0000", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 9.1135000000000001e-05 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.874846+0000", 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 7.2529999999999998e-06 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.413 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.879498+0000", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.004651784 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.879526+0000", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8589999999999999e-05 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:45:46.879570+0000", 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 4.4230000000000002e-05 2024-03-11T21:47:21.414 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "description": "MOSDScrubReserve(9.1c,QREQUEST,e:991,from:0,reservation_nonce:15)", 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "initiated_at": "2024-03-11T21:47:20.564889+0000", 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "age": 0.76412471900000001, 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.0058322519999999996, 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "started", 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ 2024-03-11T21:47:21.415 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564889+0000", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "header_read", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564889+0000", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564891+0000", 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.807e-06 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.416 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564901+0000", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.0293e-05 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564929+0000", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.7869e-05 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.564944+0000", 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.4348999999999999e-05 2024-03-11T21:47:21.417 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.565063+0000", 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.000118999 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "op must wait for map", 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.565074+0000", 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 1.1429e-05 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg", 2024-03-11T21:47:21.418 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.570624+0000", 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.005549773 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started", 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.570652+0000", 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 2.8085999999999998e-05 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: { 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done", 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2024-03-11T21:47:20.570721+0000", 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 6.9647000000000006e-05 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-11T21:47:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-11T21:47:21.420 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: audit 2024-03-11T21:47:19.307121+0000 mgr.y (mgr.14550) 933 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:19.776914+0000 mgr.y (mgr.14550) 934 : cluster 0 pgmap v1597: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.073406+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e991: 8 total, 8 up, 8 in 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.085628+0000 osd.4 (osd.4) 555 : cluster 0 3.1c scrub starts 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.090599+0000 osd.4 (osd.4) 556 : cluster 0 3.1c scrub ok 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.170171+0000 osd.7 (osd.7) 580 : cluster 0 3.f scrub starts 2024-03-11T21:47:21.420 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.177715+0000 osd.7 (osd.7) 581 : cluster 0 3.f scrub ok 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.571148+0000 osd.0 (osd.0) 578 : cluster 0 9.1c scrub starts 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.573699+0000 osd.0 (osd.0) 579 : cluster 0 9.1c scrub ok 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: audit 2024-03-11T21:47:19.307121+0000 mgr.y (mgr.14550) 933 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:19.776914+0000 mgr.y (mgr.14550) 934 : cluster 0 pgmap v1597: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.073406+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e991: 8 total, 8 up, 8 in 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.085628+0000 osd.4 (osd.4) 555 : cluster 0 3.1c scrub starts 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.090599+0000 osd.4 (osd.4) 556 : cluster 0 3.1c scrub ok 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.170171+0000 osd.7 (osd.7) 580 : cluster 0 3.f scrub starts 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.177715+0000 osd.7 (osd.7) 581 : cluster 0 3.f scrub ok 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.571148+0000 osd.0 (osd.0) 578 : cluster 0 9.1c scrub starts 2024-03-11T21:47:21.421 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:21 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.573699+0000 osd.0 (osd.0) 579 : cluster 0 9.1c scrub ok 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: audit 2024-03-11T21:47:19.307121+0000 mgr.y (mgr.14550) 933 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:19.776914+0000 mgr.y (mgr.14550) 934 : cluster 0 pgmap v1597: 184 pgs: 184 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.073406+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e991: 8 total, 8 up, 8 in 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.085628+0000 osd.4 (osd.4) 555 : cluster 0 3.1c scrub starts 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.090599+0000 osd.4 (osd.4) 556 : cluster 0 3.1c scrub ok 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.170171+0000 osd.7 (osd.7) 580 : cluster 0 3.f scrub starts 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.177715+0000 osd.7 (osd.7) 581 : cluster 0 3.f scrub ok 2024-03-11T21:47:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.571148+0000 osd.0 (osd.0) 578 : cluster 0 9.1c scrub starts 2024-03-11T21:47:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:21 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.573699+0000 osd.0 (osd.0) 579 : cluster 0 9.1c scrub ok 2024-03-11T21:47:21.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3125689530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18104290 msgr2=0x7f0f18104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:21.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3125689530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18104290 0x7f0f18104670 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f0f080099c0 tx=0x7f0f08009cd0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3125689530 shutdown_connections 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3125689530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18109370 0x7f0f18110f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3125689530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18104290 0x7f0f18104670 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3125689530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104c40 0x7f0f18108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3125689530 >> 172.21.15.143:0/3125689530 conn(0x7f0f180ffb40 msgr2=0x7f0f18101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3125689530 shutdown_connections 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3125689530 wait complete. 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 librados: starting msgr at 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 librados: starting objecter 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 Processor -- start 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- start start 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 librados: setting wanted keys 2024-03-11T21:47:21.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 librados: calling monclient init 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 0x7f0f18199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18113830 con 0x7f0f18104c40 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f181136b0 con 0x7f0f18104290 2024-03-11T21:47:21.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f181139b0 con 0x7f0f18109370 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f177fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54518/0 (socket says 172.21.15.143:54518) 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f177fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:42958/0 (socket says 172.21.15.143:42958) 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 -- 172.21.15.143:0/3954030127 learned_addr learned my addr 172.21.15.143:0/3954030127 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f16ffd700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 0x7f0f18199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 -- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 msgr2=0x7f0f1819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 -- 172.21.15.143:0/3954030127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 msgr2=0x7f0f18199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 0x7f0f18199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 -- 172.21.15.143:0/3954030127 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1819ee60 con 0x7f0f18104c40 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f16ffd700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 0x7f0f18199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f177fe700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:47:21.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f167fc700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c970 tx=0x7f0f0c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0efffff700 10 client.?.objecter ms_handle_connect 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0efffff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00ce60 con 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0f0c004830 con 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1819f150 con 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 10 client.31155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181a69d0 con 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.649+0000 7f0f1d302700 1 librados: init done 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005450 con 0x7f0f18104c40 2024-03-11T21:47:21.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1804faa0 con 0x7f0f18104c40 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0f0c007b00 con 0x7f0f18104c40 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 0x7f0f00071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(992..992 src has 933..992) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f0f0c08e3e0 con 0x7f0f18104c40 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 10 client.31155.objecter ms_dispatch 0x7f0f18001550 osd_map(992..992 src has 933..992) v4 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 3 client.31155.objecter handle_osd_map got epochs [992,992] > 0 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 3 client.31155.objecter handle_osd_map decoding full epoch 992 2024-03-11T21:47:21.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0f16ffd700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 0x7f0f00071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:21.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 20 client.31155.objecter dump_active .. 0 homeless 2024-03-11T21:47:21.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0efffff700 10 client.31155.objecter ms_handle_connect 0x7f0f0006f820 2024-03-11T21:47:21.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.653+0000 7f0f16ffd700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 0x7f0f00071ce0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f0f080037a0 tx=0x7f0f08005ea0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:21.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.657+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0f0c014030 con 0x7f0f18104c40 2024-03-11T21:47:21.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:21.897+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f0f181a6c00 con 0x7f0f18104c40 2024-03-11T21:47:22.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.077+0000 7f0efffff700 1 -- 172.21.15.143:0/3954030127 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v993) v1 ==== 113+0+0 (secure 0 0 0) 0x7f0f0c05b370 con 0x7f0f18104c40 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 10 librados: watch_flush enter 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 10 librados: watch_flush exit 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 20 client.31155.objecter shutdown clearing up homeless session... 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 10 client.31155.objecter successfully canceled tick 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 msgr2=0x7f0f00071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 0x7f0f00071ce0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f0f080037a0 tx=0x7f0f08005ea0 comp rx=0 tx=0).stop 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 msgr2=0x7f0f1819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c970 tx=0x7f0f0c00cc80 comp rx=0 tx=0).stop 2024-03-11T21:47:22.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 shutdown_connections 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f0f0006f820 0x7f0f00071ce0 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f18109370 0x7f0f1819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f18104290 0x7f0f18199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 --2- 172.21.15.143:0/3954030127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f18104c40 0x7f0f1819a390 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 >> 172.21.15.143:0/3954030127 conn(0x7f0f180ffb40 msgr2=0x7f0f18100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 shutdown_connections 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 -- 172.21.15.143:0/3954030127 wait complete. 2024-03-11T21:47:22.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:22.081+0000 7f0f1d302700 1 librados: shutdown 2024-03-11T21:47:22.079 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:47:22.411 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:22.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.732394+0000 osd.5 (osd.5) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:22.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[21512]: cluster 2024-03-11T21:47:20.732754+0000 osd.5 (osd.5) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:22.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[21512]: cluster 2024-03-11T21:47:21.092204+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e992: 8 total, 8 up, 8 in 2024-03-11T21:47:22.411 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[21512]: audit 2024-03-11T21:47:21.900477+0000 mon.a (mon.0) 4127 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:22.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.732394+0000 osd.5 (osd.5) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:22.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[29284]: cluster 2024-03-11T21:47:20.732754+0000 osd.5 (osd.5) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:22.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[29284]: cluster 2024-03-11T21:47:21.092204+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e992: 8 total, 8 up, 8 in 2024-03-11T21:47:22.412 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:22 smithi143 bash[29284]: audit 2024-03-11T21:47:21.900477+0000 mon.a (mon.0) 4127 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:22.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:47:22.424 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:22 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.732394+0000 osd.5 (osd.5) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:22.424 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:22 smithi203 bash[23654]: cluster 2024-03-11T21:47:20.732754+0000 osd.5 (osd.5) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:22.424 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:22 smithi203 bash[23654]: cluster 2024-03-11T21:47:21.092204+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e992: 8 total, 8 up, 8 in 2024-03-11T21:47:22.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:22 smithi203 bash[23654]: audit 2024-03-11T21:47:21.900477+0000 mon.a (mon.0) 4127 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:23.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-11T21:47:23.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:21.716403+0000 osd.6 (osd.6) 471 : cluster 0 3.10 scrub starts 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:21.721812+0000 osd.6 (osd.6) 472 : cluster 0 3.10 scrub ok 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:21.777617+0000 mgr.y (mgr.14550) 935 : cluster 0 pgmap v1600: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: audit 2024-03-11T21:47:22.082872+0000 mon.a (mon.0) 4128 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.089955+0000 mon.a (mon.0) 4129 : cluster 0 osdmap e993: 8 total, 8 up, 8 in 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.539764+0000 osd.0 (osd.0) 580 : cluster 0 5.1b deep-scrub starts 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.542097+0000 osd.0 (osd.0) 581 : cluster 0 5.1b deep-scrub ok 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.717371+0000 osd.2 (osd.2) 489 : cluster 0 9.6 scrub starts 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.722012+0000 osd.2 (osd.2) 490 : cluster 0 9.6 scrub ok 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[21512]: audit 2024-03-11T21:47:22.987196+0000 mon.c (mon.2) 1404 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:21.716403+0000 osd.6 (osd.6) 471 : cluster 0 3.10 scrub starts 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:21.721812+0000 osd.6 (osd.6) 472 : cluster 0 3.10 scrub ok 2024-03-11T21:47:23.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:21.777617+0000 mgr.y (mgr.14550) 935 : cluster 0 pgmap v1600: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: audit 2024-03-11T21:47:22.082872+0000 mon.a (mon.0) 4128 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.089955+0000 mon.a (mon.0) 4129 : cluster 0 osdmap e993: 8 total, 8 up, 8 in 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.539764+0000 osd.0 (osd.0) 580 : cluster 0 5.1b deep-scrub starts 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.542097+0000 osd.0 (osd.0) 581 : cluster 0 5.1b deep-scrub ok 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.717371+0000 osd.2 (osd.2) 489 : cluster 0 9.6 scrub starts 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.722012+0000 osd.2 (osd.2) 490 : cluster 0 9.6 scrub ok 2024-03-11T21:47:23.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:23 smithi143 bash[29284]: audit 2024-03-11T21:47:22.987196+0000 mon.c (mon.2) 1404 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:21.716403+0000 osd.6 (osd.6) 471 : cluster 0 3.10 scrub starts 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:21.721812+0000 osd.6 (osd.6) 472 : cluster 0 3.10 scrub ok 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:21.777617+0000 mgr.y (mgr.14550) 935 : cluster 0 pgmap v1600: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: audit 2024-03-11T21:47:22.082872+0000 mon.a (mon.0) 4128 : audit 1 from='client.? 172.21.15.143:0/3954030127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.089955+0000 mon.a (mon.0) 4129 : cluster 0 osdmap e993: 8 total, 8 up, 8 in 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.539764+0000 osd.0 (osd.0) 580 : cluster 0 5.1b deep-scrub starts 2024-03-11T21:47:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.542097+0000 osd.0 (osd.0) 581 : cluster 0 5.1b deep-scrub ok 2024-03-11T21:47:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.717371+0000 osd.2 (osd.2) 489 : cluster 0 9.6 scrub starts 2024-03-11T21:47:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.722012+0000 osd.2 (osd.2) 490 : cluster 0 9.6 scrub ok 2024-03-11T21:47:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:23 smithi203 bash[23654]: audit 2024-03-11T21:47:22.987196+0000 mon.c (mon.2) 1404 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.759248+0000 osd.6 (osd.6) 473 : cluster 0 2.14 scrub starts 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:22.763361+0000 osd.6 (osd.6) 474 : cluster 0 2.14 scrub ok 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.047777+0000 osd.4 (osd.4) 557 : cluster 0 5.1 scrub starts 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.050574+0000 osd.4 (osd.4) 558 : cluster 0 5.1 scrub ok 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.104310+0000 osd.7 (osd.7) 582 : cluster 0 5.9 scrub starts 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.107347+0000 osd.7 (osd.7) 583 : cluster 0 5.9 scrub ok 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.171360+0000 osd.3 (osd.3) 587 : cluster 0 6.6 scrub starts 2024-03-11T21:47:24.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.174040+0000 osd.3 (osd.3) 588 : cluster 0 6.6 scrub ok 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.759248+0000 osd.6 (osd.6) 473 : cluster 0 2.14 scrub starts 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:22.763361+0000 osd.6 (osd.6) 474 : cluster 0 2.14 scrub ok 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.047777+0000 osd.4 (osd.4) 557 : cluster 0 5.1 scrub starts 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.050574+0000 osd.4 (osd.4) 558 : cluster 0 5.1 scrub ok 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.104310+0000 osd.7 (osd.7) 582 : cluster 0 5.9 scrub starts 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.107347+0000 osd.7 (osd.7) 583 : cluster 0 5.9 scrub ok 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.171360+0000 osd.3 (osd.3) 587 : cluster 0 6.6 scrub starts 2024-03-11T21:47:24.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:24 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.174040+0000 osd.3 (osd.3) 588 : cluster 0 6.6 scrub ok 2024-03-11T21:47:24.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.759248+0000 osd.6 (osd.6) 473 : cluster 0 2.14 scrub starts 2024-03-11T21:47:24.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:22.763361+0000 osd.6 (osd.6) 474 : cluster 0 2.14 scrub ok 2024-03-11T21:47:24.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.047777+0000 osd.4 (osd.4) 557 : cluster 0 5.1 scrub starts 2024-03-11T21:47:24.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.050574+0000 osd.4 (osd.4) 558 : cluster 0 5.1 scrub ok 2024-03-11T21:47:24.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.104310+0000 osd.7 (osd.7) 582 : cluster 0 5.9 scrub starts 2024-03-11T21:47:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.107347+0000 osd.7 (osd.7) 583 : cluster 0 5.9 scrub ok 2024-03-11T21:47:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.171360+0000 osd.3 (osd.3) 587 : cluster 0 6.6 scrub starts 2024-03-11T21:47:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:24 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.174040+0000 osd.3 (osd.3) 588 : cluster 0 6.6 scrub ok 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:23.779539+0000 mgr.y (mgr.14550) 936 : cluster 0 pgmap v1602: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.034301+0000 osd.4 (osd.4) 559 : cluster 0 6.1e scrub starts 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.036596+0000 osd.4 (osd.4) 560 : cluster 0 6.1e scrub ok 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.126618+0000 osd.3 (osd.3) 589 : cluster 0 9.2 scrub starts 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.128913+0000 osd.3 (osd.3) 590 : cluster 0 9.2 scrub ok 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.584298+0000 osd.0 (osd.0) 582 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.584695+0000 osd.0 (osd.0) 583 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.718411+0000 osd.2 (osd.2) 491 : cluster 0 5.b scrub starts 2024-03-11T21:47:25.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[21512]: cluster 2024-03-11T21:47:24.720699+0000 osd.2 (osd.2) 492 : cluster 0 5.b scrub ok 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:23.779539+0000 mgr.y (mgr.14550) 936 : cluster 0 pgmap v1602: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.034301+0000 osd.4 (osd.4) 559 : cluster 0 6.1e scrub starts 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.036596+0000 osd.4 (osd.4) 560 : cluster 0 6.1e scrub ok 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.126618+0000 osd.3 (osd.3) 589 : cluster 0 9.2 scrub starts 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.128913+0000 osd.3 (osd.3) 590 : cluster 0 9.2 scrub ok 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.584298+0000 osd.0 (osd.0) 582 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.584695+0000 osd.0 (osd.0) 583 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.718411+0000 osd.2 (osd.2) 491 : cluster 0 5.b scrub starts 2024-03-11T21:47:25.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:25 smithi143 bash[29284]: cluster 2024-03-11T21:47:24.720699+0000 osd.2 (osd.2) 492 : cluster 0 5.b scrub ok 2024-03-11T21:47:25.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:23.779539+0000 mgr.y (mgr.14550) 936 : cluster 0 pgmap v1602: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:47:25.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.034301+0000 osd.4 (osd.4) 559 : cluster 0 6.1e scrub starts 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.036596+0000 osd.4 (osd.4) 560 : cluster 0 6.1e scrub ok 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.126618+0000 osd.3 (osd.3) 589 : cluster 0 9.2 scrub starts 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.128913+0000 osd.3 (osd.3) 590 : cluster 0 9.2 scrub ok 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.584298+0000 osd.0 (osd.0) 582 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.584695+0000 osd.0 (osd.0) 583 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.718411+0000 osd.2 (osd.2) 491 : cluster 0 5.b scrub starts 2024-03-11T21:47:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:25 smithi203 bash[23654]: cluster 2024-03-11T21:47:24.720699+0000 osd.2 (osd.2) 492 : cluster 0 5.b scrub ok 2024-03-11T21:47:26.448 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:26 smithi143 bash[21512]: cluster 2024-03-11T21:47:25.132641+0000 mon.a (mon.0) 4130 : cluster 0 osdmap e994: 8 total, 8 up, 8 in 2024-03-11T21:47:26.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:26 smithi143 bash[29284]: cluster 2024-03-11T21:47:25.132641+0000 mon.a (mon.0) 4130 : cluster 0 osdmap e994: 8 total, 8 up, 8 in 2024-03-11T21:47:26.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:26 smithi203 bash[23654]: cluster 2024-03-11T21:47:25.132641+0000 mon.a (mon.0) 4130 : cluster 0 osdmap e994: 8 total, 8 up, 8 in 2024-03-11T21:47:27.221 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:25.780650+0000 mgr.y (mgr.14550) 937 : cluster 0 pgmap v1604: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.154426+0000 osd.7 (osd.7) 584 : cluster 0 2.5 scrub starts 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.157014+0000 osd.7 (osd.7) 585 : cluster 0 2.5 scrub ok 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.167639+0000 osd.3 (osd.3) 591 : cluster 0 6.13 scrub starts 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.170216+0000 osd.3 (osd.3) 592 : cluster 0 6.13 scrub ok 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.645344+0000 osd.2 (osd.2) 493 : cluster 0 2.1 scrub starts 2024-03-11T21:47:27.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.647760+0000 osd.2 (osd.2) 494 : cluster 0 2.1 scrub ok 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:25.780650+0000 mgr.y (mgr.14550) 937 : cluster 0 pgmap v1604: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.154426+0000 osd.7 (osd.7) 584 : cluster 0 2.5 scrub starts 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.157014+0000 osd.7 (osd.7) 585 : cluster 0 2.5 scrub ok 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.167639+0000 osd.3 (osd.3) 591 : cluster 0 6.13 scrub starts 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.170216+0000 osd.3 (osd.3) 592 : cluster 0 6.13 scrub ok 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.645344+0000 osd.2 (osd.2) 493 : cluster 0 2.1 scrub starts 2024-03-11T21:47:27.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:27 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.647760+0000 osd.2 (osd.2) 494 : cluster 0 2.1 scrub ok 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:25.780650+0000 mgr.y (mgr.14550) 937 : cluster 0 pgmap v1604: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.154426+0000 osd.7 (osd.7) 584 : cluster 0 2.5 scrub starts 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.157014+0000 osd.7 (osd.7) 585 : cluster 0 2.5 scrub ok 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.167639+0000 osd.3 (osd.3) 591 : cluster 0 6.13 scrub starts 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.170216+0000 osd.3 (osd.3) 592 : cluster 0 6.13 scrub ok 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.645344+0000 osd.2 (osd.2) 493 : cluster 0 2.1 scrub starts 2024-03-11T21:47:27.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:27 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.647760+0000 osd.2 (osd.2) 494 : cluster 0 2.1 scrub ok 2024-03-11T21:47:28.034 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.964201+0000 osd.1 (osd.1) 375 : cluster 0 6.d scrub starts 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:26.966592+0000 osd.1 (osd.1) 376 : cluster 0 6.d scrub ok 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.071922+0000 osd.4 (osd.4) 561 : cluster 0 1.5 scrub starts 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.074336+0000 osd.4 (osd.4) 562 : cluster 0 1.5 scrub ok 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.617859+0000 osd.0 (osd.0) 584 : cluster 0 6.7 scrub starts 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.620266+0000 osd.0 (osd.0) 585 : cluster 0 6.7 scrub ok 2024-03-11T21:47:28.425 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.656161+0000 osd.2 (osd.2) 495 : cluster 0 2.3 scrub starts 2024-03-11T21:47:28.426 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:28 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.658607+0000 osd.2 (osd.2) 496 : cluster 0 2.3 scrub ok 2024-03-11T21:47:28.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.964201+0000 osd.1 (osd.1) 375 : cluster 0 6.d scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:26.966592+0000 osd.1 (osd.1) 376 : cluster 0 6.d scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.071922+0000 osd.4 (osd.4) 561 : cluster 0 1.5 scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.074336+0000 osd.4 (osd.4) 562 : cluster 0 1.5 scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.617859+0000 osd.0 (osd.0) 584 : cluster 0 6.7 scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.620266+0000 osd.0 (osd.0) 585 : cluster 0 6.7 scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.656161+0000 osd.2 (osd.2) 495 : cluster 0 2.3 scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.658607+0000 osd.2 (osd.2) 496 : cluster 0 2.3 scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.964201+0000 osd.1 (osd.1) 375 : cluster 0 6.d scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:26.966592+0000 osd.1 (osd.1) 376 : cluster 0 6.d scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.071922+0000 osd.4 (osd.4) 561 : cluster 0 1.5 scrub starts 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.074336+0000 osd.4 (osd.4) 562 : cluster 0 1.5 scrub ok 2024-03-11T21:47:28.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.617859+0000 osd.0 (osd.0) 584 : cluster 0 6.7 scrub starts 2024-03-11T21:47:28.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.620266+0000 osd.0 (osd.0) 585 : cluster 0 6.7 scrub ok 2024-03-11T21:47:28.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.656161+0000 osd.2 (osd.2) 495 : cluster 0 2.3 scrub starts 2024-03-11T21:47:28.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:28 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.658607+0000 osd.2 (osd.2) 496 : cluster 0 2.3 scrub ok 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[29284]: cluster 2024-03-11T21:47:27.781448+0000 mgr.y (mgr.14550) 938 : cluster 0 pgmap v1605: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[29284]: cluster 2024-03-11T21:47:28.145942+0000 osd.3 (osd.3) 593 : cluster 0 6.1f scrub starts 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[29284]: cluster 2024-03-11T21:47:28.148347+0000 osd.3 (osd.3) 594 : cluster 0 6.1f scrub ok 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[21512]: cluster 2024-03-11T21:47:27.781448+0000 mgr.y (mgr.14550) 938 : cluster 0 pgmap v1605: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[21512]: cluster 2024-03-11T21:47:28.145942+0000 osd.3 (osd.3) 593 : cluster 0 6.1f scrub starts 2024-03-11T21:47:29.403 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:29 smithi143 bash[21512]: cluster 2024-03-11T21:47:28.148347+0000 osd.3 (osd.3) 594 : cluster 0 6.1f scrub ok 2024-03-11T21:47:29.534 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:47:29.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:29 smithi203 bash[23654]: cluster 2024-03-11T21:47:27.781448+0000 mgr.y (mgr.14550) 938 : cluster 0 pgmap v1605: 184 pgs: 184 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-03-11T21:47:29.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:29 smithi203 bash[23654]: cluster 2024-03-11T21:47:28.145942+0000 osd.3 (osd.3) 593 : cluster 0 6.1f scrub starts 2024-03-11T21:47:29.535 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:29 smithi203 bash[23654]: cluster 2024-03-11T21:47:28.148347+0000 osd.3 (osd.3) 594 : cluster 0 6.1f scrub ok 2024-03-11T21:47:30.311 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:47:30.311 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:47:30.312 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:47:30.312 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:47:30.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:30 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.721236+0000 osd.2 (osd.2) 497 : cluster 0 5.d deep-scrub starts 2024-03-11T21:47:30.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:30 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.724487+0000 osd.2 (osd.2) 498 : cluster 0 5.d deep-scrub ok 2024-03-11T21:47:30.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:30 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.721236+0000 osd.2 (osd.2) 497 : cluster 0 5.d deep-scrub starts 2024-03-11T21:47:30.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:30 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.724487+0000 osd.2 (osd.2) 498 : cluster 0 5.d deep-scrub ok 2024-03-11T21:47:30.503 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:30 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.721236+0000 osd.2 (osd.2) 497 : cluster 0 5.d deep-scrub starts 2024-03-11T21:47:30.503 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:30 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.724487+0000 osd.2 (osd.2) 498 : cluster 0 5.d deep-scrub ok 2024-03-11T21:47:31.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: audit 2024-03-11T21:47:29.315283+0000 mgr.y (mgr.14550) 939 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.769539+0000 osd.6 (osd.6) 475 : cluster 0 9.18 scrub starts 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.771752+0000 osd.6 (osd.6) 476 : cluster 0 9.18 scrub ok 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.783309+0000 mgr.y (mgr.14550) 940 : cluster 0 pgmap v1606: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.900169+0000 osd.1 (osd.1) 377 : cluster 0 6.1d scrub starts 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:29.902474+0000 osd.1 (osd.1) 378 : cluster 0 6.1d scrub ok 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.105194+0000 osd.7 (osd.7) 586 : cluster 0 9.d scrub starts 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.109118+0000 osd.7 (osd.7) 587 : cluster 0 9.d scrub ok 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.745343+0000 osd.2 (osd.2) 499 : cluster 0 2.e scrub starts 2024-03-11T21:47:31.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.746950+0000 osd.2 (osd.2) 500 : cluster 0 2.e scrub ok 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: audit 2024-03-11T21:47:29.315283+0000 mgr.y (mgr.14550) 939 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.769539+0000 osd.6 (osd.6) 475 : cluster 0 9.18 scrub starts 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.771752+0000 osd.6 (osd.6) 476 : cluster 0 9.18 scrub ok 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.783309+0000 mgr.y (mgr.14550) 940 : cluster 0 pgmap v1606: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.900169+0000 osd.1 (osd.1) 377 : cluster 0 6.1d scrub starts 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:29.902474+0000 osd.1 (osd.1) 378 : cluster 0 6.1d scrub ok 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.105194+0000 osd.7 (osd.7) 586 : cluster 0 9.d scrub starts 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.109118+0000 osd.7 (osd.7) 587 : cluster 0 9.d scrub ok 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.745343+0000 osd.2 (osd.2) 499 : cluster 0 2.e scrub starts 2024-03-11T21:47:31.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:31 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.746950+0000 osd.2 (osd.2) 500 : cluster 0 2.e scrub ok 2024-03-11T21:47:31.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: audit 2024-03-11T21:47:29.315283+0000 mgr.y (mgr.14550) 939 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.769539+0000 osd.6 (osd.6) 475 : cluster 0 9.18 scrub starts 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.771752+0000 osd.6 (osd.6) 476 : cluster 0 9.18 scrub ok 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.783309+0000 mgr.y (mgr.14550) 940 : cluster 0 pgmap v1606: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.900169+0000 osd.1 (osd.1) 377 : cluster 0 6.1d scrub starts 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:29.902474+0000 osd.1 (osd.1) 378 : cluster 0 6.1d scrub ok 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.105194+0000 osd.7 (osd.7) 586 : cluster 0 9.d scrub starts 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.109118+0000 osd.7 (osd.7) 587 : cluster 0 9.d scrub ok 2024-03-11T21:47:31.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.745343+0000 osd.2 (osd.2) 499 : cluster 0 2.e scrub starts 2024-03-11T21:47:31.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:31 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.746950+0000 osd.2 (osd.2) 500 : cluster 0 2.e scrub ok 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- 172.21.15.143:0/2259109245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde006a720 msgr2=0x7fdde0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- 172.21.15.143:0/2259109245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde006a720 0x7fdde0106a90 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009920 tx=0x7fddd4009c30 comp rx=0 tx=0).stop 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- 172.21.15.143:0/2259109245 shutdown_connections 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- 172.21.15.143:0/2259109245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde01071c0 0x7fdde010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- 172.21.15.143:0/2259109245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde0069d70 0x7fdde006a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- 172.21.15.143:0/2259109245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde006a720 0x7fdde0106a90 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:31.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- 172.21.15.143:0/2259109245 >> 172.21.15.143:0/2259109245 conn(0x7fdde0076d80 msgr2=0x7fdde0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:31.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- 172.21.15.143:0/2259109245 shutdown_connections 2024-03-11T21:47:31.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- 172.21.15.143:0/2259109245 wait complete. 2024-03-11T21:47:31.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 librados: starting msgr at 2024-03-11T21:47:31.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 librados: starting objecter 2024-03-11T21:47:31.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 Processor -- start 2024-03-11T21:47:31.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- start start 2024-03-11T21:47:31.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 librados: setting wanted keys 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 librados: calling monclient init 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde006a720 0x7fdde0101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 0x7fdde0101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0111680 con 0x7fdde01071c0 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0111500 con 0x7fdde006a720 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdde5977700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0111800 con 0x7fdde0069d70 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdddeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdddf7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 0x7fdde0101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdddeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:37664/0 (socket says 172.21.15.143:37664) 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fdddeffd700 1 -- 172.21.15.143:0/1353654968 learned_addr learned my addr 172.21.15.143:0/1353654968 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.697+0000 7fddde7fc700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde006a720 0x7fdde0101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:31.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 -- 172.21.15.143:0/1353654968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde006a720 msgr2=0x7fdde0101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde006a720 0x7fdde0101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 -- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 msgr2=0x7fdde0101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 0x7fdde0101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 -- 172.21.15.143:0/1353654968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde01a6260 con 0x7fdde0069d70 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddf7fe700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 0x7fdde0101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdddeffd700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fddd000d950 tx=0x7fddd000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 10 client.?.objecter ms_handle_connect 0x7fdde0069d70 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 10 client.?.objecter resend_mon_ops 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0016070 con 0x7fdde0069d70 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fddd000fd10 con 0x7fdde0069d70 2024-03-11T21:47:31.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde01a6550 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde5977700 10 client.42365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd000e3b0 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde01a69f0 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde5977700 1 librados: init done 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fddd0019070 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 0x7fddc8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde0103570 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fddde7fc700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 0x7fddc8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(994..994 src has 944..994) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fddd0057fb0 con 0x7fdde0069d70 2024-03-11T21:47:31.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 10 client.42365.objecter ms_dispatch 0x7fdde0001550 osd_map(994..994 src has 944..994) v4 2024-03-11T21:47:31.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 3 client.42365.objecter handle_osd_map got epochs [994,994] > 0 2024-03-11T21:47:31.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 3 client.42365.objecter handle_osd_map decoding full epoch 994 2024-03-11T21:47:31.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.701+0000 7fdde4975700 20 client.42365.objecter dump_active .. 0 homeless 2024-03-11T21:47:31.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.705+0000 7fdde4975700 10 client.42365.objecter ms_handle_connect 0x7fddc806f6f0 2024-03-11T21:47:31.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.705+0000 7fddde7fc700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 0x7fddc8071bb0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005e90 tx=0x7fddd4019040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:31.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.709+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fddd005a610 con 0x7fdde0069d70 2024-03-11T21:47:31.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:31.941+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fdde01022f0 con 0x7fdde0069d70 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.731916+0000 osd.6 (osd.6) 477 : cluster 0 2.11 scrub starts 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[29284]: cluster 2024-03-11T21:47:30.734324+0000 osd.6 (osd.6) 478 : cluster 0 2.11 scrub ok 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[29284]: audit 2024-03-11T21:47:31.945854+0000 mon.c (mon.2) 1405 : audit 1 from='client.? 172.21.15.143:0/1353654968' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[29284]: audit 2024-03-11T21:47:31.946595+0000 mon.a (mon.0) 4131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.731916+0000 osd.6 (osd.6) 477 : cluster 0 2.11 scrub starts 2024-03-11T21:47:32.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[21512]: cluster 2024-03-11T21:47:30.734324+0000 osd.6 (osd.6) 478 : cluster 0 2.11 scrub ok 2024-03-11T21:47:32.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[21512]: audit 2024-03-11T21:47:31.945854+0000 mon.c (mon.2) 1405 : audit 1 from='client.? 172.21.15.143:0/1353654968' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:32 smithi143 bash[21512]: audit 2024-03-11T21:47:31.946595+0000 mon.a (mon.0) 4131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:32 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.731916+0000 osd.6 (osd.6) 477 : cluster 0 2.11 scrub starts 2024-03-11T21:47:32.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:32 smithi203 bash[23654]: cluster 2024-03-11T21:47:30.734324+0000 osd.6 (osd.6) 478 : cluster 0 2.11 scrub ok 2024-03-11T21:47:32.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:32 smithi203 bash[23654]: audit 2024-03-11T21:47:31.945854+0000 mon.c (mon.2) 1405 : audit 1 from='client.? 172.21.15.143:0/1353654968' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:32 smithi203 bash[23654]: audit 2024-03-11T21:47:31.946595+0000 mon.a (mon.0) 4131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:47:32.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.701+0000 7fdde4975700 1 -- 172.21.15.143:0/1353654968 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v995) v1 ==== 133+0+0 (secure 0 0 0) 0x7fddd0010460 con 0x7fdde0069d70 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 10 librados: watch_flush enter 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 10 librados: watch_flush exit 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 20 client.42365.objecter shutdown clearing up homeless session... 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 10 client.42365.objecter successfully canceled tick 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 msgr2=0x7fddc8071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 0x7fddc8071bb0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005e90 tx=0x7fddd4019040 comp rx=0 tx=0).stop 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 msgr2=0x7fdde0100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fddd000d950 tx=0x7fddd000dc60 comp rx=0 tx=0).stop 2024-03-11T21:47:32.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 shutdown_connections 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fddc806f6f0 0x7fddc8071bb0 unknown :-1 s=CLOSED pgs=3504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde01071c0 0x7fdde0101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde0069d70 0x7fdde0100e10 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 --2- 172.21.15.143:0/1353654968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde006a720 0x7fdde0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 >> 172.21.15.143:0/1353654968 conn(0x7fdde0076d80 msgr2=0x7fdde00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 shutdown_connections 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 -- 172.21.15.143:0/1353654968 wait complete. 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:32.705+0000 7fdde5977700 1 librados: shutdown 2024-03-11T21:47:32.696 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:31.687504+0000 osd.6 (osd.6) 479 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:31.689872+0000 osd.6 (osd.6) 480 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:31.784426+0000 mgr.y (mgr.14550) 941 : cluster 0 pgmap v1607: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:31.893037+0000 osd.1 (osd.1) 379 : cluster 0 5.15 scrub starts 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:31.895770+0000 osd.1 (osd.1) 380 : cluster 0 5.15 scrub ok 2024-03-11T21:47:33.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.114475+0000 osd.3 (osd.3) 595 : cluster 0 3.7 scrub starts 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.118441+0000 osd.3 (osd.3) 596 : cluster 0 3.7 scrub ok 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: audit 2024-03-11T21:47:32.694222+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.701723+0000 mon.a (mon.0) 4133 : cluster 0 osdmap e995: 8 total, 8 up, 8 in 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:31.687504+0000 osd.6 (osd.6) 479 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:31.689872+0000 osd.6 (osd.6) 480 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:31.784426+0000 mgr.y (mgr.14550) 941 : cluster 0 pgmap v1607: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:31.893037+0000 osd.1 (osd.1) 379 : cluster 0 5.15 scrub starts 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:31.895770+0000 osd.1 (osd.1) 380 : cluster 0 5.15 scrub ok 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.114475+0000 osd.3 (osd.3) 595 : cluster 0 3.7 scrub starts 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.118441+0000 osd.3 (osd.3) 596 : cluster 0 3.7 scrub ok 2024-03-11T21:47:33.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: audit 2024-03-11T21:47:32.694222+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:33 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.701723+0000 mon.a (mon.0) 4133 : cluster 0 osdmap e995: 8 total, 8 up, 8 in 2024-03-11T21:47:33.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:31.687504+0000 osd.6 (osd.6) 479 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:31.689872+0000 osd.6 (osd.6) 480 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:31.784426+0000 mgr.y (mgr.14550) 941 : cluster 0 pgmap v1607: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:31.893037+0000 osd.1 (osd.1) 379 : cluster 0 5.15 scrub starts 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:31.895770+0000 osd.1 (osd.1) 380 : cluster 0 5.15 scrub ok 2024-03-11T21:47:33.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.114475+0000 osd.3 (osd.3) 595 : cluster 0 3.7 scrub starts 2024-03-11T21:47:33.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.118441+0000 osd.3 (osd.3) 596 : cluster 0 3.7 scrub ok 2024-03-11T21:47:33.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: audit 2024-03-11T21:47:32.694222+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:47:33.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:33 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.701723+0000 mon.a (mon.0) 4133 : cluster 0 osdmap e995: 8 total, 8 up, 8 in 2024-03-11T21:47:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.682628+0000 osd.6 (osd.6) 481 : cluster 0 5.11 scrub starts 2024-03-11T21:47:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.685169+0000 osd.6 (osd.6) 482 : cluster 0 5.11 scrub ok 2024-03-11T21:47:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.945937+0000 osd.1 (osd.1) 381 : cluster 0 1.12 scrub starts 2024-03-11T21:47:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:32.949250+0000 osd.1 (osd.1) 382 : cluster 0 1.12 scrub ok 2024-03-11T21:47:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:33.088664+0000 osd.4 (osd.4) 563 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:33.091348+0000 osd.4 (osd.4) 564 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:33.655775+0000 osd.0 (osd.0) 586 : cluster 0 2.13 scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[21512]: cluster 2024-03-11T21:47:33.658168+0000 osd.0 (osd.0) 587 : cluster 0 2.13 scrub ok 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.682628+0000 osd.6 (osd.6) 481 : cluster 0 5.11 scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.685169+0000 osd.6 (osd.6) 482 : cluster 0 5.11 scrub ok 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.945937+0000 osd.1 (osd.1) 381 : cluster 0 1.12 scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:32.949250+0000 osd.1 (osd.1) 382 : cluster 0 1.12 scrub ok 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:33.088664+0000 osd.4 (osd.4) 563 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:33.091348+0000 osd.4 (osd.4) 564 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:33.655775+0000 osd.0 (osd.0) 586 : cluster 0 2.13 scrub starts 2024-03-11T21:47:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:34 smithi143 bash[29284]: cluster 2024-03-11T21:47:33.658168+0000 osd.0 (osd.0) 587 : cluster 0 2.13 scrub ok 2024-03-11T21:47:34.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.682628+0000 osd.6 (osd.6) 481 : cluster 0 5.11 scrub starts 2024-03-11T21:47:34.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.685169+0000 osd.6 (osd.6) 482 : cluster 0 5.11 scrub ok 2024-03-11T21:47:34.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.945937+0000 osd.1 (osd.1) 381 : cluster 0 1.12 scrub starts 2024-03-11T21:47:34.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:32.949250+0000 osd.1 (osd.1) 382 : cluster 0 1.12 scrub ok 2024-03-11T21:47:34.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:33.088664+0000 osd.4 (osd.4) 563 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:47:34.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:33.091348+0000 osd.4 (osd.4) 564 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:47:34.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:33.655775+0000 osd.0 (osd.0) 586 : cluster 0 2.13 scrub starts 2024-03-11T21:47:34.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:34 smithi203 bash[23654]: cluster 2024-03-11T21:47:33.658168+0000 osd.0 (osd.0) 587 : cluster 0 2.13 scrub ok 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[29284]: cluster 2024-03-11T21:47:33.785614+0000 mgr.y (mgr.14550) 942 : cluster 0 pgmap v1609: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[29284]: cluster 2024-03-11T21:47:34.122184+0000 osd.4 (osd.4) 565 : cluster 0 1.e scrub starts 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[29284]: cluster 2024-03-11T21:47:34.124525+0000 osd.4 (osd.4) 566 : cluster 0 1.e scrub ok 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[29284]: cluster 2024-03-11T21:47:34.607476+0000 osd.0 (osd.0) 588 : cluster 0 6.10 scrub starts 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[29284]: cluster 2024-03-11T21:47:34.609816+0000 osd.0 (osd.0) 589 : cluster 0 6.10 scrub ok 2024-03-11T21:47:35.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[21512]: cluster 2024-03-11T21:47:33.785614+0000 mgr.y (mgr.14550) 942 : cluster 0 pgmap v1609: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[21512]: cluster 2024-03-11T21:47:34.122184+0000 osd.4 (osd.4) 565 : cluster 0 1.e scrub starts 2024-03-11T21:47:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[21512]: cluster 2024-03-11T21:47:34.124525+0000 osd.4 (osd.4) 566 : cluster 0 1.e scrub ok 2024-03-11T21:47:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[21512]: cluster 2024-03-11T21:47:34.607476+0000 osd.0 (osd.0) 588 : cluster 0 6.10 scrub starts 2024-03-11T21:47:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:35 smithi143 bash[21512]: cluster 2024-03-11T21:47:34.609816+0000 osd.0 (osd.0) 589 : cluster 0 6.10 scrub ok 2024-03-11T21:47:35.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:35 smithi203 bash[23654]: cluster 2024-03-11T21:47:33.785614+0000 mgr.y (mgr.14550) 942 : cluster 0 pgmap v1609: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:35.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:35 smithi203 bash[23654]: cluster 2024-03-11T21:47:34.122184+0000 osd.4 (osd.4) 565 : cluster 0 1.e scrub starts 2024-03-11T21:47:35.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:35 smithi203 bash[23654]: cluster 2024-03-11T21:47:34.124525+0000 osd.4 (osd.4) 566 : cluster 0 1.e scrub ok 2024-03-11T21:47:35.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:35 smithi203 bash[23654]: cluster 2024-03-11T21:47:34.607476+0000 osd.0 (osd.0) 588 : cluster 0 6.10 scrub starts 2024-03-11T21:47:35.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:35 smithi203 bash[23654]: cluster 2024-03-11T21:47:34.609816+0000 osd.0 (osd.0) 589 : cluster 0 6.10 scrub ok 2024-03-11T21:47:35.871 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.097089+0000 osd.1 (osd.1) 383 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.097694+0000 osd.1 (osd.1) 384 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.195568+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e996: 8 total, 8 up, 8 in 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.097089+0000 osd.1 (osd.1) 383 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.097694+0000 osd.1 (osd.1) 384 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:36.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:36 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.195568+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e996: 8 total, 8 up, 8 in 2024-03-11T21:47:36.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:36 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.097089+0000 osd.1 (osd.1) 383 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:36.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:36 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.097694+0000 osd.1 (osd.1) 384 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:36.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:36 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.195568+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e996: 8 total, 8 up, 8 in 2024-03-11T21:47:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:37 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.728346+0000 osd.6 (osd.6) 483 : cluster 0 1.6 scrub starts 2024-03-11T21:47:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:37 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.731036+0000 osd.6 (osd.6) 484 : cluster 0 1.6 scrub ok 2024-03-11T21:47:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:37 smithi203 bash[23654]: cluster 2024-03-11T21:47:35.786534+0000 mgr.y (mgr.14550) 943 : cluster 0 pgmap v1611: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:37.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.728346+0000 osd.6 (osd.6) 483 : cluster 0 1.6 scrub starts 2024-03-11T21:47:37.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.731036+0000 osd.6 (osd.6) 484 : cluster 0 1.6 scrub ok 2024-03-11T21:47:37.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[21512]: cluster 2024-03-11T21:47:35.786534+0000 mgr.y (mgr.14550) 943 : cluster 0 pgmap v1611: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:37.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.728346+0000 osd.6 (osd.6) 483 : cluster 0 1.6 scrub starts 2024-03-11T21:47:37.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.731036+0000 osd.6 (osd.6) 484 : cluster 0 1.6 scrub ok 2024-03-11T21:47:37.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:37 smithi143 bash[29284]: cluster 2024-03-11T21:47:35.786534+0000 mgr.y (mgr.14550) 943 : cluster 0 pgmap v1611: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:38.281 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:38.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[29284]: cluster 2024-03-11T21:47:36.748669+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-03-11T21:47:38.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[29284]: cluster 2024-03-11T21:47:36.750219+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-03-11T21:47:38.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[29284]: cluster 2024-03-11T21:47:37.678111+0000 osd.0 (osd.0) 590 : cluster 0 6.18 scrub starts 2024-03-11T21:47:38.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[29284]: cluster 2024-03-11T21:47:37.680538+0000 osd.0 (osd.0) 591 : cluster 0 6.18 scrub ok 2024-03-11T21:47:38.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[29284]: audit 2024-03-11T21:47:38.022578+0000 mon.c (mon.2) 1406 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:38.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[21512]: cluster 2024-03-11T21:47:36.748669+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-03-11T21:47:38.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[21512]: cluster 2024-03-11T21:47:36.750219+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-03-11T21:47:38.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[21512]: cluster 2024-03-11T21:47:37.678111+0000 osd.0 (osd.0) 590 : cluster 0 6.18 scrub starts 2024-03-11T21:47:38.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[21512]: cluster 2024-03-11T21:47:37.680538+0000 osd.0 (osd.0) 591 : cluster 0 6.18 scrub ok 2024-03-11T21:47:38.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:38 smithi143 bash[21512]: audit 2024-03-11T21:47:38.022578+0000 mon.c (mon.2) 1406 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:38.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:38 smithi203 bash[23654]: cluster 2024-03-11T21:47:36.748669+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-03-11T21:47:38.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:38 smithi203 bash[23654]: cluster 2024-03-11T21:47:36.750219+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-03-11T21:47:38.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:38 smithi203 bash[23654]: cluster 2024-03-11T21:47:37.678111+0000 osd.0 (osd.0) 590 : cluster 0 6.18 scrub starts 2024-03-11T21:47:38.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:38 smithi203 bash[23654]: cluster 2024-03-11T21:47:37.680538+0000 osd.0 (osd.0) 591 : cluster 0 6.18 scrub ok 2024-03-11T21:47:38.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:38 smithi203 bash[23654]: audit 2024-03-11T21:47:38.022578+0000 mon.c (mon.2) 1406 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:38.829 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:47:38.829 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:47:38.829 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:47:38.829 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:47:38.829 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[21512]: cluster 2024-03-11T21:47:37.787054+0000 mgr.y (mgr.14550) 944 : cluster 0 pgmap v1612: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[21512]: cluster 2024-03-11T21:47:38.123680+0000 osd.4 (osd.4) 567 : cluster 0 3.1d deep-scrub starts 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[21512]: cluster 2024-03-11T21:47:38.135309+0000 osd.4 (osd.4) 568 : cluster 0 3.1d deep-scrub ok 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[29284]: cluster 2024-03-11T21:47:37.787054+0000 mgr.y (mgr.14550) 944 : cluster 0 pgmap v1612: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[29284]: cluster 2024-03-11T21:47:38.123680+0000 osd.4 (osd.4) 567 : cluster 0 3.1d deep-scrub starts 2024-03-11T21:47:39.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:39 smithi143 bash[29284]: cluster 2024-03-11T21:47:38.135309+0000 osd.4 (osd.4) 568 : cluster 0 3.1d deep-scrub ok 2024-03-11T21:47:39.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:47:39.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:39 smithi203 bash[23654]: cluster 2024-03-11T21:47:37.787054+0000 mgr.y (mgr.14550) 944 : cluster 0 pgmap v1612: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:39.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:39 smithi203 bash[23654]: cluster 2024-03-11T21:47:38.123680+0000 osd.4 (osd.4) 567 : cluster 0 3.1d deep-scrub starts 2024-03-11T21:47:39.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:39 smithi203 bash[23654]: cluster 2024-03-11T21:47:38.135309+0000 osd.4 (osd.4) 568 : cluster 0 3.1d deep-scrub ok 2024-03-11T21:47:39.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:47:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:40 smithi203 bash[23654]: cluster 2024-03-11T21:47:38.836456+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-03-11T21:47:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:40 smithi203 bash[23654]: cluster 2024-03-11T21:47:38.838831+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-03-11T21:47:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:40 smithi203 bash[23654]: cluster 2024-03-11T21:47:39.164072+0000 osd.4 (osd.4) 569 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:47:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:40 smithi203 bash[23654]: cluster 2024-03-11T21:47:39.166346+0000 osd.4 (osd.4) 570 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[21512]: cluster 2024-03-11T21:47:38.836456+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[21512]: cluster 2024-03-11T21:47:38.838831+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[21512]: cluster 2024-03-11T21:47:39.164072+0000 osd.4 (osd.4) 569 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[21512]: cluster 2024-03-11T21:47:39.166346+0000 osd.4 (osd.4) 570 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[29284]: cluster 2024-03-11T21:47:38.836456+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[29284]: cluster 2024-03-11T21:47:38.838831+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[29284]: cluster 2024-03-11T21:47:39.164072+0000 osd.4 (osd.4) 569 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:47:40.580 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:40 smithi143 bash[29284]: cluster 2024-03-11T21:47:39.166346+0000 osd.4 (osd.4) 570 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: audit 2024-03-11T21:47:39.323167+0000 mgr.y (mgr.14550) 945 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: cluster 2024-03-11T21:47:39.788237+0000 mgr.y (mgr.14550) 946 : cluster 0 pgmap v1613: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: cluster 2024-03-11T21:47:39.850957+0000 osd.1 (osd.1) 387 : cluster 0 9.1f scrub starts 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: cluster 2024-03-11T21:47:39.856917+0000 osd.1 (osd.1) 388 : cluster 0 9.1f scrub ok 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: cluster 2024-03-11T21:47:40.139254+0000 osd.4 (osd.4) 571 : cluster 0 6.17 scrub starts 2024-03-11T21:47:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:41 smithi203 bash[23654]: cluster 2024-03-11T21:47:40.141723+0000 osd.4 (osd.4) 572 : cluster 0 6.17 scrub ok 2024-03-11T21:47:41.628 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: audit 2024-03-11T21:47:39.323167+0000 mgr.y (mgr.14550) 945 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:41.628 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: cluster 2024-03-11T21:47:39.788237+0000 mgr.y (mgr.14550) 946 : cluster 0 pgmap v1613: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:41.628 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: cluster 2024-03-11T21:47:39.850957+0000 osd.1 (osd.1) 387 : cluster 0 9.1f scrub starts 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: cluster 2024-03-11T21:47:39.856917+0000 osd.1 (osd.1) 388 : cluster 0 9.1f scrub ok 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: cluster 2024-03-11T21:47:40.139254+0000 osd.4 (osd.4) 571 : cluster 0 6.17 scrub starts 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[29284]: cluster 2024-03-11T21:47:40.141723+0000 osd.4 (osd.4) 572 : cluster 0 6.17 scrub ok 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: audit 2024-03-11T21:47:39.323167+0000 mgr.y (mgr.14550) 945 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: cluster 2024-03-11T21:47:39.788237+0000 mgr.y (mgr.14550) 946 : cluster 0 pgmap v1613: 184 pgs: 184 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: cluster 2024-03-11T21:47:39.850957+0000 osd.1 (osd.1) 387 : cluster 0 9.1f scrub starts 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: cluster 2024-03-11T21:47:39.856917+0000 osd.1 (osd.1) 388 : cluster 0 9.1f scrub ok 2024-03-11T21:47:41.629 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: cluster 2024-03-11T21:47:40.139254+0000 osd.4 (osd.4) 571 : cluster 0 6.17 scrub starts 2024-03-11T21:47:41.630 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:41 smithi143 bash[21512]: cluster 2024-03-11T21:47:40.141723+0000 osd.4 (osd.4) 572 : cluster 0 6.17 scrub ok 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/69909813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 msgr2=0x7fe32410ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- 172.21.15.143:0/69909813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32410ed40 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00b0d0 tx=0x7fe31c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/69909813 shutdown_connections 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- 172.21.15.143:0/69909813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32410ed40 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- 172.21.15.143:0/69909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324069d70 0x7fe32406a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- 172.21.15.143:0/69909813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32406a720 0x7fe324106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/69909813 >> 172.21.15.143:0/69909813 conn(0x7fe324076d80 msgr2=0x7fe324077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/69909813 shutdown_connections 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/69909813 wait complete. 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 librados: starting msgr at 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 librados: starting objecter 2024-03-11T21:47:41.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 Processor -- start 2024-03-11T21:47:41.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- start start 2024-03-11T21:47:41.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 librados: setting wanted keys 2024-03-11T21:47:41.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 librados: calling monclient init 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 0x7fe324199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe32406a720 0x7fe32419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324111680 con 0x7fe324069d70 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324111500 con 0x7fe32406a720 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe324111800 con 0x7fe3241071c0 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:42772/0 (socket says 172.21.15.143:42772) 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 -- 172.21.15.143:0/4243452649 learned_addr learned my addr 172.21.15.143:0/4243452649 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:47:41.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329404700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 0x7fe324199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe328c03700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe32406a720 0x7fe32419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 -- 172.21.15.143:0/4243452649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe32406a720 msgr2=0x7fe32419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe32406a720 0x7fe32419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 -- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 msgr2=0x7fe324199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 0x7fe324199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 -- 172.21.15.143:0/4243452649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32419ed40 con 0x7fe3241071c0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329404700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 0x7fe324199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe329c05700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006070 tx=0x7fe31c007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe31a7fc700 10 client.?.objecter ms_handle_connect 0x7fe3241071c0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe31a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c021020 con 0x7fe3241071c0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32419efd0 con 0x7fe3241071c0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe31c004c50 con 0x7fe3241071c0 2024-03-11T21:47:41.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c005050 con 0x7fe3241071c0 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 10 client.42371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3241a68b0 con 0x7fe3241071c0 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.817+0000 7fe32b668700 1 librados: init done 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32404faa0 con 0x7fe3241071c0 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe31c024020 con 0x7fe3241071c0 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 0x7fe310071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:41.812 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(996..996 src has 944..996) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fe31c00f020 con 0x7fe3241071c0 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 10 client.42371.objecter ms_dispatch 0x7fe324001550 osd_map(996..996 src has 944..996) v4 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 3 client.42371.objecter handle_osd_map got epochs [996,996] > 0 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 3 client.42371.objecter handle_osd_map decoding full epoch 996 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe329404700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 0x7fe310071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 20 client.42371.objecter dump_active .. 0 homeless 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe31a7fc700 10 client.42371.objecter ms_handle_connect 0x7fe31006f820 2024-03-11T21:47:41.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.821+0000 7fe329404700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 0x7fe310071ce0 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fe32000afd0 tx=0x7fe32000a380 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:41.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:41.825+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe31c05c850 con 0x7fe3241071c0 2024-03-11T21:47:42.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.049+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fe3241a6a80 con 0x7fe3241071c0 2024-03-11T21:47:42.226 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.233+0000 7fe31a7fc700 1 -- 172.21.15.143:0/4243452649 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v997) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe31c0602a0 con 0x7fe3241071c0 2024-03-11T21:47:42.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.237+0000 7fe32b668700 10 librados: watch_flush enter 2024-03-11T21:47:42.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.237+0000 7fe32b668700 10 librados: watch_flush exit 2024-03-11T21:47:42.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.237+0000 7fe32b668700 20 client.42371.objecter shutdown clearing up homeless session... 2024-03-11T21:47:42.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.237+0000 7fe32b668700 10 client.42371.objecter successfully canceled tick 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 msgr2=0x7fe310071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 0x7fe310071ce0 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fe32000afd0 tx=0x7fe32000a380 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 msgr2=0x7fe32419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006070 tx=0x7fe31c007a20 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 shutdown_connections 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe31006f820 0x7fe310071ce0 unknown :-1 s=CLOSED pgs=3505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe324069d70 0x7fe324199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3241071c0 0x7fe32419e600 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 --2- 172.21.15.143:0/4243452649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe32406a720 0x7fe32419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:42.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.241+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 >> 172.21.15.143:0/4243452649 conn(0x7fe324076d80 msgr2=0x7fe32410d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:42.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.245+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 shutdown_connections 2024-03-11T21:47:42.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.245+0000 7fe32b668700 1 -- 172.21.15.143:0/4243452649 wait complete. 2024-03-11T21:47:42.234 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:42.245+0000 7fe32b668700 1 librados: shutdown 2024-03-11T21:47:42.235 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:40.807827+0000 osd.1 (osd.1) 389 : cluster 0 3.19 scrub starts 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:40.814504+0000 osd.1 (osd.1) 390 : cluster 0 3.19 scrub ok 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.164350+0000 osd.4 (osd.4) 573 : cluster 0 5.e deep-scrub starts 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.169456+0000 osd.4 (osd.4) 574 : cluster 0 5.e deep-scrub ok 2024-03-11T21:47:42.449 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.176161+0000 osd.3 (osd.3) 597 : cluster 0 9.1a scrub starts 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.179010+0000 osd.3 (osd.3) 598 : cluster 0 9.1a scrub ok 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.180373+0000 osd.7 (osd.7) 588 : cluster 0 3.d scrub starts 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.185447+0000 osd.7 (osd.7) 589 : cluster 0 3.d scrub ok 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: cluster 2024-03-11T21:47:41.789216+0000 mgr.y (mgr.14550) 947 : cluster 0 pgmap v1614: 184 pgs: 184 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: audit 2024-03-11T21:47:42.054012+0000 mon.c (mon.2) 1407 : audit 1 from='client.? 172.21.15.143:0/4243452649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[29284]: audit 2024-03-11T21:47:42.054392+0000 mon.a (mon.0) 4135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:40.807827+0000 osd.1 (osd.1) 389 : cluster 0 3.19 scrub starts 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:40.814504+0000 osd.1 (osd.1) 390 : cluster 0 3.19 scrub ok 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.164350+0000 osd.4 (osd.4) 573 : cluster 0 5.e deep-scrub starts 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.169456+0000 osd.4 (osd.4) 574 : cluster 0 5.e deep-scrub ok 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.176161+0000 osd.3 (osd.3) 597 : cluster 0 9.1a scrub starts 2024-03-11T21:47:42.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.179010+0000 osd.3 (osd.3) 598 : cluster 0 9.1a scrub ok 2024-03-11T21:47:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.180373+0000 osd.7 (osd.7) 588 : cluster 0 3.d scrub starts 2024-03-11T21:47:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.185447+0000 osd.7 (osd.7) 589 : cluster 0 3.d scrub ok 2024-03-11T21:47:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: cluster 2024-03-11T21:47:41.789216+0000 mgr.y (mgr.14550) 947 : cluster 0 pgmap v1614: 184 pgs: 184 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: audit 2024-03-11T21:47:42.054012+0000 mon.c (mon.2) 1407 : audit 1 from='client.? 172.21.15.143:0/4243452649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:42 smithi143 bash[21512]: audit 2024-03-11T21:47:42.054392+0000 mon.a (mon.0) 4135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:40.807827+0000 osd.1 (osd.1) 389 : cluster 0 3.19 scrub starts 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:40.814504+0000 osd.1 (osd.1) 390 : cluster 0 3.19 scrub ok 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.164350+0000 osd.4 (osd.4) 573 : cluster 0 5.e deep-scrub starts 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.169456+0000 osd.4 (osd.4) 574 : cluster 0 5.e deep-scrub ok 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.176161+0000 osd.3 (osd.3) 597 : cluster 0 9.1a scrub starts 2024-03-11T21:47:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.179010+0000 osd.3 (osd.3) 598 : cluster 0 9.1a scrub ok 2024-03-11T21:47:42.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.180373+0000 osd.7 (osd.7) 588 : cluster 0 3.d scrub starts 2024-03-11T21:47:42.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.185447+0000 osd.7 (osd.7) 589 : cluster 0 3.d scrub ok 2024-03-11T21:47:42.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: cluster 2024-03-11T21:47:41.789216+0000 mgr.y (mgr.14550) 947 : cluster 0 pgmap v1614: 184 pgs: 184 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:47:42.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: audit 2024-03-11T21:47:42.054012+0000 mon.c (mon.2) 1407 : audit 1 from='client.? 172.21.15.143:0/4243452649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:42.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:42 smithi203 bash[23654]: audit 2024-03-11T21:47:42.054392+0000 mon.a (mon.0) 4135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:43.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.198842+0000 osd.4 (osd.4) 575 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.204524+0000 osd.4 (osd.4) 576 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: audit 2024-03-11T21:47:42.229367+0000 mon.a (mon.0) 4136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.238055+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e997: 8 total, 8 up, 8 in 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.636377+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub starts 2024-03-11T21:47:43.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:43 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.639781+0000 osd.0 (osd.0) 593 : cluster 0 3.17 scrub ok 2024-03-11T21:47:43.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.198842+0000 osd.4 (osd.4) 575 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.204524+0000 osd.4 (osd.4) 576 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: audit 2024-03-11T21:47:42.229367+0000 mon.a (mon.0) 4136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.238055+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e997: 8 total, 8 up, 8 in 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.636377+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub starts 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.639781+0000 osd.0 (osd.0) 593 : cluster 0 3.17 scrub ok 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.198842+0000 osd.4 (osd.4) 575 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.204524+0000 osd.4 (osd.4) 576 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: audit 2024-03-11T21:47:42.229367+0000 mon.a (mon.0) 4136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.238055+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e997: 8 total, 8 up, 8 in 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.636377+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub starts 2024-03-11T21:47:43.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:43 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.639781+0000 osd.0 (osd.0) 593 : cluster 0 3.17 scrub ok 2024-03-11T21:47:44.391 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:44 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.862512+0000 osd.1 (osd.1) 391 : cluster 0 5.c scrub starts 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:44 smithi203 bash[23654]: cluster 2024-03-11T21:47:42.864823+0000 osd.1 (osd.1) 392 : cluster 0 5.c scrub ok 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:44 smithi203 bash[23654]: cluster 2024-03-11T21:47:43.641772+0000 osd.0 (osd.0) 594 : cluster 0 3.6 scrub starts 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:44 smithi203 bash[23654]: cluster 2024-03-11T21:47:43.645771+0000 osd.0 (osd.0) 595 : cluster 0 3.6 scrub ok 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:44 smithi203 bash[23654]: cluster 2024-03-11T21:47:43.791070+0000 mgr.y (mgr.14550) 948 : cluster 0 pgmap v1616: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:44.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.862512+0000 osd.1 (osd.1) 391 : cluster 0 5.c scrub starts 2024-03-11T21:47:44.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[21512]: cluster 2024-03-11T21:47:42.864823+0000 osd.1 (osd.1) 392 : cluster 0 5.c scrub ok 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[21512]: cluster 2024-03-11T21:47:43.641772+0000 osd.0 (osd.0) 594 : cluster 0 3.6 scrub starts 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[21512]: cluster 2024-03-11T21:47:43.645771+0000 osd.0 (osd.0) 595 : cluster 0 3.6 scrub ok 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[21512]: cluster 2024-03-11T21:47:43.791070+0000 mgr.y (mgr.14550) 948 : cluster 0 pgmap v1616: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.862512+0000 osd.1 (osd.1) 391 : cluster 0 5.c scrub starts 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[29284]: cluster 2024-03-11T21:47:42.864823+0000 osd.1 (osd.1) 392 : cluster 0 5.c scrub ok 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[29284]: cluster 2024-03-11T21:47:43.641772+0000 osd.0 (osd.0) 594 : cluster 0 3.6 scrub starts 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[29284]: cluster 2024-03-11T21:47:43.645771+0000 osd.0 (osd.0) 595 : cluster 0 3.6 scrub ok 2024-03-11T21:47:44.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:44 smithi143 bash[29284]: cluster 2024-03-11T21:47:43.791070+0000 mgr.y (mgr.14550) 948 : cluster 0 pgmap v1616: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:45.528 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:45 smithi203 bash[23654]: cluster 2024-03-11T21:47:44.127802+0000 osd.7 (osd.7) 590 : cluster 0 9.7 scrub starts 2024-03-11T21:47:45.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:45 smithi203 bash[23654]: cluster 2024-03-11T21:47:44.133685+0000 osd.7 (osd.7) 591 : cluster 0 9.7 scrub ok 2024-03-11T21:47:45.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:45 smithi203 bash[23654]: cluster 2024-03-11T21:47:44.684999+0000 osd.2 (osd.2) 501 : cluster 0 5.a scrub starts 2024-03-11T21:47:45.529 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:45 smithi203 bash[23654]: cluster 2024-03-11T21:47:44.687335+0000 osd.2 (osd.2) 502 : cluster 0 5.a scrub ok 2024-03-11T21:47:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[21512]: cluster 2024-03-11T21:47:44.127802+0000 osd.7 (osd.7) 590 : cluster 0 9.7 scrub starts 2024-03-11T21:47:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[21512]: cluster 2024-03-11T21:47:44.133685+0000 osd.7 (osd.7) 591 : cluster 0 9.7 scrub ok 2024-03-11T21:47:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[21512]: cluster 2024-03-11T21:47:44.684999+0000 osd.2 (osd.2) 501 : cluster 0 5.a scrub starts 2024-03-11T21:47:45.699 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[21512]: cluster 2024-03-11T21:47:44.687335+0000 osd.2 (osd.2) 502 : cluster 0 5.a scrub ok 2024-03-11T21:47:45.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[29284]: cluster 2024-03-11T21:47:44.127802+0000 osd.7 (osd.7) 590 : cluster 0 9.7 scrub starts 2024-03-11T21:47:45.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[29284]: cluster 2024-03-11T21:47:44.133685+0000 osd.7 (osd.7) 591 : cluster 0 9.7 scrub ok 2024-03-11T21:47:45.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[29284]: cluster 2024-03-11T21:47:44.684999+0000 osd.2 (osd.2) 501 : cluster 0 5.a scrub starts 2024-03-11T21:47:45.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:45 smithi143 bash[29284]: cluster 2024-03-11T21:47:44.687335+0000 osd.2 (osd.2) 502 : cluster 0 5.a scrub ok 2024-03-11T21:47:46.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:46 smithi203 bash[23654]: cluster 2024-03-11T21:47:45.153236+0000 osd.7 (osd.7) 592 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:47:46.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:46 smithi203 bash[23654]: cluster 2024-03-11T21:47:45.155577+0000 osd.7 (osd.7) 593 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:47:46.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:46 smithi203 bash[23654]: cluster 2024-03-11T21:47:45.666199+0000 osd.0 (osd.0) 596 : cluster 0 9.9 scrub starts 2024-03-11T21:47:46.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:46 smithi203 bash[23654]: cluster 2024-03-11T21:47:45.669034+0000 osd.0 (osd.0) 597 : cluster 0 9.9 scrub ok 2024-03-11T21:47:46.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:46 smithi203 bash[23654]: cluster 2024-03-11T21:47:45.792161+0000 mgr.y (mgr.14550) 949 : cluster 0 pgmap v1617: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:46.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[29284]: cluster 2024-03-11T21:47:45.153236+0000 osd.7 (osd.7) 592 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:47:46.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[29284]: cluster 2024-03-11T21:47:45.155577+0000 osd.7 (osd.7) 593 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:47:46.699 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[29284]: cluster 2024-03-11T21:47:45.666199+0000 osd.0 (osd.0) 596 : cluster 0 9.9 scrub starts 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[29284]: cluster 2024-03-11T21:47:45.669034+0000 osd.0 (osd.0) 597 : cluster 0 9.9 scrub ok 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[29284]: cluster 2024-03-11T21:47:45.792161+0000 mgr.y (mgr.14550) 949 : cluster 0 pgmap v1617: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[21512]: cluster 2024-03-11T21:47:45.153236+0000 osd.7 (osd.7) 592 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[21512]: cluster 2024-03-11T21:47:45.155577+0000 osd.7 (osd.7) 593 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[21512]: cluster 2024-03-11T21:47:45.666199+0000 osd.0 (osd.0) 596 : cluster 0 9.9 scrub starts 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[21512]: cluster 2024-03-11T21:47:45.669034+0000 osd.0 (osd.0) 597 : cluster 0 9.9 scrub ok 2024-03-11T21:47:46.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:46 smithi143 bash[21512]: cluster 2024-03-11T21:47:45.792161+0000 mgr.y (mgr.14550) 949 : cluster 0 pgmap v1617: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:47.158 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:47:47.167 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:47:47.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:47:47.167 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:47:47.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:15 314.2 314:40000000:%01:::head [pgnls start_epoch 692] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e692)", 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 487.60029714199999, 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086210539999999995, 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 15 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567517+0000", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567520+0000", 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8760000000000001e-06 2024-03-11T21:47:47.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567534+0000", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3526999999999999e-05 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567557+0000", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3102e-05 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567574+0000", 2024-03-11T21:47:47.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7799e-05 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567706+0000", 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013165599999999999 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.567717+0000", 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.119e-05 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.575810+0000", 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0080927120000000002 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.575903+0000", 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.289e-05 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:39.576138+0000", 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000235302 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.172 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.0s2 697/688 ECSubWrite(tid=2186, reqid=client.41921.0:108, at_version=697'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 482.658885252, 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019466730000000002, 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508929+0000", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.508931+0000", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.917e-06 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509226+0000", 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029490400000000002 2024-03-11T21:47:47.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509351+0000", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000124769 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509386+0000", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5311999999999997e-05 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509542+0000", 2024-03-11T21:47:47.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015589099999999999 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.509566+0000", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4437e-05 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.528307+0000", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018741203000000001 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:44.528396+0000", 2024-03-11T21:47:47.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8296999999999998e-05 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:258 316.7 316:e031f6fd:test-rados-api-smithi143-71773-117::bar:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e712)", 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 455.59412212799998, 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021393550000000001, 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 258 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573692+0000", 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573695+0000", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1329999999999999e-06 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573710+0000", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5079000000000001e-05 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573745+0000", 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4885999999999998e-05 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573773+0000", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7919000000000002e-05 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573933+0000", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015959500000000001 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.573949+0000", 2024-03-11T21:47:47.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6443000000000002e-05 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.576634+0000", 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026843209999999999 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577229+0000", 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00059544800000000005 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.590928+0000", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013698988 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.591095+0000", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016727300000000001 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594119+0000", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0030232950000000001 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:47:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594197+0000", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7943999999999998e-05 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594438+0000", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024140500000000001 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.594500+0000", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1722000000000003e-05 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.595086+0000", 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00058609899999999997 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.6s1 718/688 ECSubWrite(tid=2189, reqid=client.41921.0:368, at_version=718'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 449.59833789800001, 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014535064, 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569476+0000", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569478+0000", 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569541+0000", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3212000000000004e-05 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569570+0000", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8343e-05 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569594+0000", 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4252000000000001e-05 2024-03-11T21:47:47.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569754+0000", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000160512 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.569777+0000", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3034999999999999e-05 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.583691+0000", 2024-03-11T21:47:47.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013913576 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.584011+0000", 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032058400000000001 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(osd.6.0:797 317.1e e723/717 317:7b8f8022:test-rados-api-smithi143-71773-119::foo:head v 723'1, mlcod=0'0)", 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 444.62793202, 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073935069999999997, 2024-03-11T21:47:47.188 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539882+0000", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539883+0000", 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7700000000000003e-07 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539895+0000", 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1646e-05 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539925+0000", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9952000000000001e-05 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539945+0000", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9786999999999999e-05 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.540089+0000", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014464499999999999 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.540144+0000", 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4528999999999999e-05 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.547153+0000", 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007008841 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.547276+0000", 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012323 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.41921.0:805 320.19 e748/743 320:98ec0ffc:test-rados-api-smithi143-71773-125::foo:4 v 748'3, mlcod=748'1)", 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 419.52145058500003, 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010667311000000001, 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646364+0000", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2699999999999999e-07 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646371+0000", 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6209999999999998e-06 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646382+0000", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1248e-05 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646390+0000", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1749999999999999e-06 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646452+0000", 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2311999999999996e-05 2024-03-11T21:47:47.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646483+0000", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1106999999999999e-05 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656894+0000", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010410815 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.657031+0000", 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013730599999999999 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:808 320.2 320:423c9130:test-rados-api-smithi143-71773-125::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e748)", 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 419.50899849000001, 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019173852000000002, 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.197 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 808 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658816+0000", 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658818+0000", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.951e-06 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658826+0000", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8599999999999993e-06 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658843+0000", 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7813999999999999e-05 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658852+0000", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8019999999999998e-06 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.658919+0000", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6619999999999996e-05 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.659154+0000", 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000234749 2024-03-11T21:47:47.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.663498+0000", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043444169999999997 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.663669+0000", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017117799999999999 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.674712+0000", 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011042545000000001 2024-03-11T21:47:47.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.674871+0000", 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015880199999999999 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.677990+0000", 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0031191140000000001 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:809 320.2 320:423c9130:test-rados-api-smithi143-71773-125::bar:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e748)", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 419.489549533, 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007111717, 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 809 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678265+0000", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678266+0000", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.717e-06 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678276+0000", 2024-03-11T21:47:47.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7189999999999996e-06 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678306+0000", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0035999999999998e-05 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678322+0000", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5599000000000001e-05 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678503+0000", 2024-03-11T21:47:47.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018142899999999999 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.678853+0000", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034932300000000002 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.681568+0000", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002714958 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.681729+0000", 2024-03-11T21:47:47.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016142899999999999 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.682222+0000", 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049319800000000003 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.682385+0000", 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016276600000000001 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.685376+0000", 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029915430000000002 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1041 321.c 321:30000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 408.56699757600001, 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0074456820000000003, 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1041 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600817+0000", 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600820+0000", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.18e-06 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600860+0000", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0034999999999999e-05 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600898+0000", 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8108999999999997e-05 2024-03-11T21:47:47.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.600917+0000", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8692999999999999e-05 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.601061+0000", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014444300000000001 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.601072+0000", 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1232e-05 2024-03-11T21:47:47.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608051+0000", 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069788489999999996 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608109+0000", 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8149000000000002e-05 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.608262+0000", 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015299199999999999 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1232 322.4 322:20000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 395.54372917699999, 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007980282, 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.213 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1232 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624085+0000", 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624089+0000", 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9500000000000003e-06 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624110+0000", 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0948999999999999e-05 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624148+0000", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7795e-05 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624168+0000", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0400000000000001e-05 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624313+0000", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000145036 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.624324+0000", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0706e-05 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.631717+0000", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073927020000000001 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.631813+0000", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5853999999999994e-05 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.632065+0000", 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025289000000000003 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1682 325.11 325:8f82c992:test-rados-api-smithi143-71773-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e797)", 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 368.58563042600002, 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010991031, 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.218 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1682 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582184+0000", 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582186+0000", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4870000000000001e-06 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582201+0000", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4521e-05 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582222+0000", 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1540999999999999e-05 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582234+0000", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1651000000000001e-05 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582335+0000", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100454 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.582578+0000", 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024389600000000001 2024-03-11T21:47:47.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587764+0000", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051852089999999997 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587864+0000", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100305 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.587997+0000", 2024-03-11T21:47:47.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013257399999999999 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.588663+0000", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00066632400000000004 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.590973+0000", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023103009999999998 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593057+0000", 2024-03-11T21:47:47.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020840500000000001 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593124+0000", 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6678000000000001e-05 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:38.593175+0000", 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1039999999999999e-05 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.4,QREQUEST,e:804,from:1,reservation_nonce:11)", 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 360.88523604, 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0068787520000000001, 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282578+0000", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282581+0000", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1070000000000001e-06 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282596+0000", 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5043e-05 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282633+0000", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6748999999999997e-05 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282651+0000", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8122999999999999e-05 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282813+0000", 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016193199999999999 2024-03-11T21:47:47.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.282825+0000", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1823e-05 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.288552+0000", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057266460000000002 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.288592+0000", 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0519000000000003e-05 2024-03-11T21:47:47.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.289457+0000", 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00086481000000000004 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1794 326.12 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 360.594052729, 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.037717492999999998, 2024-03-11T21:47:47.229 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1794 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573762+0000", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.573765+0000", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3270000000000001e-06 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579409+0000", 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056436849999999998 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579451+0000", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2525999999999999e-05 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579473+0000", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1656000000000001e-05 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.579659+0000", 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018667299999999999 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580270+0000", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00061076700000000002 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590053+0000", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0097825080000000005 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590326+0000", 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027315 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.590671+0000", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034527100000000002 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.608122+0000", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017450514 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.608268+0000", 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014647799999999999 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611190+0000", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029218650000000001 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611353+0000", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000162979 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611479+0000", 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012609399999999999 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.235 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1795 326.12 326:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 360.55609300999998, 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026799703000000001, 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1795 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.236 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611721+0000", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611724+0000", 2024-03-11T21:47:47.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.807e-06 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611739+0000", 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4620000000000001e-05 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611763+0000", 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4411999999999999e-05 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611779+0000", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5866999999999999e-05 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.611918+0000", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013868600000000001 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612104+0000", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000186338 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638186+0000", 2024-03-11T21:47:47.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026081984999999999 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638383+0000", 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000196868 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.638521+0000", 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013812000000000001 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1830 326.0 326:00000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 358.58490671700002, 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071741649999999997, 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1830 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582908+0000", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582911+0000", 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.783e-06 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582931+0000", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9525999999999999e-05 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582972+0000", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1118999999999997e-05 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.582993+0000", 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0740000000000001e-05 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.583163+0000", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016989800000000001 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.583181+0000", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8077e-05 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.589650+0000", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064692509999999996 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.589754+0000", 2024-03-11T21:47:47.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010387699999999999 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.590082+0000", 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032789400000000002 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDECSubOpWrite(313.4s2 847/688 ECSubWrite(tid=912, reqid=client.41921.0:2312, at_version=847'20, trim_to=0'0, roll_forward_to=833'19))", 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 311.41603962800002, 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016333995, 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.245 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751775+0000", 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7300000000000001e-07 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751820+0000", 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4733e-05 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751846+0000", 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5893000000000001e-05 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.751870+0000", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4414999999999999e-05 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.753501+0000", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016312239999999999 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.753526+0000", 2024-03-11T21:47:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4986000000000001e-05 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.768033+0000", 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01450655 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:35.768109+0000", 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5721000000000002e-05 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14427.0:4141 3.13 e864/78 3:cc909b6b:::obj_delete_at_hint.0000000106:head v 864'27, mlcod=864'25)", 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 275.57680952099997, 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011996013999999999, 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591005+0000", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591006+0000", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2799999999999995e-07 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591011+0000", 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.801e-06 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591023+0000", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1243999999999999e-05 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591029+0000", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591091+0000", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2040999999999998e-05 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591110+0000", 2024-03-11T21:47:47.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.931e-05 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.602922+0000", 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011811991000000001 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603001+0000", 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8938999999999998e-05 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4150 3.3 3:c35966f0:::obj_delete_at_hint.0000000109:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 275.53426104900001, 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011028938, 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4150 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:47:47.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633553+0000", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633554+0000", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2799999999999995e-07 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633559+0000", 2024-03-11T21:47:47.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6580000000000001e-06 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633567+0000", 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4989999999999994e-06 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633572+0000", 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6349999999999997e-06 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633607+0000", 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5110000000000001e-05 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633657+0000", 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0251999999999999e-05 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633765+0000", 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108041 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.634713+0000", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00094732400000000004 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644210+0000", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094969240000000003 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644482+0000", 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000272272 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644543+0000", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1472999999999999e-05 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644582+0000", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8921999999999998e-05 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(5.11,QREQUEST,e:916,from:6,reservation_nonce:24)", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:30.641374+0000", 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 196.526440057, 2024-03-11T21:47:47.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.029698929999999998, 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641374+0000", 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641374+0000", 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641375+0000", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0240000000000001e-06 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641382+0000", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2230000000000004e-06 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641397+0000", 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5542000000000001e-05 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641410+0000", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2799999999999999e-05 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641490+0000", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0431000000000003e-05 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.641498+0000", 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9899999999999997e-06 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.656448+0000", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014949572 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.656491+0000", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3399000000000002e-05 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.671073+0000", 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014581949 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.263 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(5.c,QREQUEST,e:997,from:1,reservation_nonce:13)", 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:47:42.852801+0000", 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 4.3150132389999998, 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073600239999999997, 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852801+0000", 2024-03-11T21:47:47.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852801+0000", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852803+0000", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2460000000000002e-06 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852815+0000", 2024-03-11T21:47:47.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.189e-05 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852877+0000", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1367999999999999e-05 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.852899+0000", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2010000000000001e-05 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.853040+0000", 2024-03-11T21:47:47.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014141100000000001 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.853057+0000", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6909e-05 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.859995+0000", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069379840000000003 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.860038+0000", 2024-03-11T21:47:47.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2839000000000001e-05 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:42.860161+0000", 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012336700000000001 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:47:47.268 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:47:47.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:47:47.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:47 smithi143 bash[21512]: cluster 2024-03-11T21:47:46.716200+0000 osd.0 (osd.0) 598 : cluster 0 1.1 scrub starts 2024-03-11T21:47:47.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:47 smithi143 bash[21512]: cluster 2024-03-11T21:47:46.718858+0000 osd.0 (osd.0) 599 : cluster 0 1.1 scrub ok 2024-03-11T21:47:47.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:47 smithi143 bash[29284]: cluster 2024-03-11T21:47:46.716200+0000 osd.0 (osd.0) 598 : cluster 0 1.1 scrub starts 2024-03-11T21:47:47.949 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:47 smithi143 bash[29284]: cluster 2024-03-11T21:47:46.718858+0000 osd.0 (osd.0) 599 : cluster 0 1.1 scrub ok 2024-03-11T21:47:47.999 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:48.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:47 smithi203 bash[23654]: cluster 2024-03-11T21:47:46.716200+0000 osd.0 (osd.0) 598 : cluster 0 1.1 scrub starts 2024-03-11T21:47:48.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:47 smithi203 bash[23654]: cluster 2024-03-11T21:47:46.718858+0000 osd.0 (osd.0) 599 : cluster 0 1.1 scrub ok 2024-03-11T21:47:48.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[21512]: cluster 2024-03-11T21:47:47.253903+0000 osd.3 (osd.3) 599 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:47:48.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[21512]: cluster 2024-03-11T21:47:47.256265+0000 osd.3 (osd.3) 600 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:47:48.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[21512]: cluster 2024-03-11T21:47:47.684076+0000 osd.0 (osd.0) 600 : cluster 0 6.0 scrub starts 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[21512]: cluster 2024-03-11T21:47:47.686477+0000 osd.0 (osd.0) 601 : cluster 0 6.0 scrub ok 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[21512]: cluster 2024-03-11T21:47:47.792904+0000 mgr.y (mgr.14550) 950 : cluster 0 pgmap v1618: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[29284]: cluster 2024-03-11T21:47:47.253903+0000 osd.3 (osd.3) 599 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[29284]: cluster 2024-03-11T21:47:47.256265+0000 osd.3 (osd.3) 600 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[29284]: cluster 2024-03-11T21:47:47.684076+0000 osd.0 (osd.0) 600 : cluster 0 6.0 scrub starts 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[29284]: cluster 2024-03-11T21:47:47.686477+0000 osd.0 (osd.0) 601 : cluster 0 6.0 scrub ok 2024-03-11T21:47:48.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:48 smithi143 bash[29284]: cluster 2024-03-11T21:47:47.792904+0000 mgr.y (mgr.14550) 950 : cluster 0 pgmap v1618: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:49.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:48 smithi203 bash[23654]: cluster 2024-03-11T21:47:47.253903+0000 osd.3 (osd.3) 599 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:47:49.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:48 smithi203 bash[23654]: cluster 2024-03-11T21:47:47.256265+0000 osd.3 (osd.3) 600 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:47:49.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:48 smithi203 bash[23654]: cluster 2024-03-11T21:47:47.684076+0000 osd.0 (osd.0) 600 : cluster 0 6.0 scrub starts 2024-03-11T21:47:49.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:48 smithi203 bash[23654]: cluster 2024-03-11T21:47:47.686477+0000 osd.0 (osd.0) 601 : cluster 0 6.0 scrub ok 2024-03-11T21:47:49.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:48 smithi203 bash[23654]: cluster 2024-03-11T21:47:47.792904+0000 mgr.y (mgr.14550) 950 : cluster 0 pgmap v1618: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:47:49.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:47:51.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: audit 2024-03-11T21:47:49.331068+0000 mgr.y (mgr.14550) 951 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:51.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.626547+0000 osd.6 (osd.6) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:51.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.626924+0000 osd.6 (osd.6) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.714966+0000 mon.a (mon.0) 4138 : cluster 0 osdmap e998: 8 total, 8 up, 8 in 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.765242+0000 osd.0 (osd.0) 602 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.771849+0000 osd.0 (osd.0) 603 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.793751+0000 mgr.y (mgr.14550) 952 : cluster 0 pgmap v1620: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.860264+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:49.863605+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:50.221341+0000 osd.7 (osd.7) 594 : cluster 0 3.11 scrub starts 2024-03-11T21:47:51.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:50 smithi203 bash[23654]: cluster 2024-03-11T21:47:50.225575+0000 osd.7 (osd.7) 595 : cluster 0 3.11 scrub ok 2024-03-11T21:47:51.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: audit 2024-03-11T21:47:49.331068+0000 mgr.y (mgr.14550) 951 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.626547+0000 osd.6 (osd.6) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.626924+0000 osd.6 (osd.6) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.714966+0000 mon.a (mon.0) 4138 : cluster 0 osdmap e998: 8 total, 8 up, 8 in 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.765242+0000 osd.0 (osd.0) 602 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.771849+0000 osd.0 (osd.0) 603 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.793751+0000 mgr.y (mgr.14550) 952 : cluster 0 pgmap v1620: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.860264+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:49.863605+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:50.221341+0000 osd.7 (osd.7) 594 : cluster 0 3.11 scrub starts 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[21512]: cluster 2024-03-11T21:47:50.225575+0000 osd.7 (osd.7) 595 : cluster 0 3.11 scrub ok 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: audit 2024-03-11T21:47:49.331068+0000 mgr.y (mgr.14550) 951 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:51.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.626547+0000 osd.6 (osd.6) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.626924+0000 osd.6 (osd.6) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.714966+0000 mon.a (mon.0) 4138 : cluster 0 osdmap e998: 8 total, 8 up, 8 in 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.765242+0000 osd.0 (osd.0) 602 : cluster 0 3.1f deep-scrub starts 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.771849+0000 osd.0 (osd.0) 603 : cluster 0 3.1f deep-scrub ok 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.793751+0000 mgr.y (mgr.14550) 952 : cluster 0 pgmap v1620: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.860264+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:49.863605+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:50.221341+0000 osd.7 (osd.7) 594 : cluster 0 3.11 scrub starts 2024-03-11T21:47:51.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:50 smithi143 bash[29284]: cluster 2024-03-11T21:47:50.225575+0000 osd.7 (osd.7) 595 : cluster 0 3.11 scrub ok 2024-03-11T21:47:51.390 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/936531144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc10e920 msgr2=0x7fb4dc110d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/936531144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc10e920 0x7fb4dc110d10 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400b0a0 tx=0x7fb4d400b3b0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/936531144 shutdown_connections 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/936531144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc10e920 0x7fb4dc110d10 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/936531144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc06a4a0 0x7fb4dc06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/936531144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 0x7fb4dc10e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/936531144 >> 172.21.15.143:0/936531144 conn(0x7fb4dc076d30 msgr2=0x7fb4dc077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/936531144 shutdown_connections 2024-03-11T21:47:51.391 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/936531144 wait complete. 2024-03-11T21:47:51.392 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 librados: starting msgr at 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 librados: starting objecter 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 Processor -- start 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- start start 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 librados: setting wanted keys 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 librados: calling monclient init 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 0x7fb4dc19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc10e920 0x7fb4dc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc1137f0 con 0x7fb4dc06adc0 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc113670 con 0x7fb4dc10e920 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc113970 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:51.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49844/0 (socket says 172.21.15.143:49844) 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 -- 172.21.15.143:0/2784411015 learned_addr learned my addr 172.21.15.143:0/2784411015 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4dbfff700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 0x7fb4dc19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e1572700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc10e920 0x7fb4dc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 -- 172.21.15.143:0/2784411015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc10e920 msgr2=0x7fb4dc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc10e920 0x7fb4dc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 -- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 msgr2=0x7fb4dc19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 0x7fb4dc19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 -- 172.21.15.143:0/2784411015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc19ed70 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e0d71700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00d950 tx=0x7fb4cc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4d9ffb700 10 client.?.objecter ms_handle_connect 0x7fb4dc06a4a0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4d9ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc016070 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb4cc009dc0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc00f3c0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc19f060 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 10 client.42377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:47:51.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc1a68e0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 librados: init done 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.400+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc04faa0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb4cc019070 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 0x7fb4c8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4dbfff700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 0x7fb4c8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(998..998 src has 944..998) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fb4cc08e350 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 10 client.42377.objecter ms_dispatch 0x7fb4dc001550 osd_map(998..998 src has 944..998) v4 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 3 client.42377.objecter handle_osd_map got epochs [998,998] > 0 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 3 client.42377.objecter handle_osd_map decoding full epoch 998 2024-03-11T21:47:51.396 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 20 client.42377.objecter dump_active .. 0 homeless 2024-03-11T21:47:51.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4dbfff700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 0x7fb4c8071ce0 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009680 tx=0x7fb4c4009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:47:51.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.404+0000 7fb4d9ffb700 10 client.42377.objecter ms_handle_connect 0x7fb4c806f820 2024-03-11T21:47:51.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.408+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb4cc05b2e0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.619 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.628+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fb4dc1a6ab0 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.718 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.724+0000 7fb4d9ffb700 1 -- 172.21.15.143:0/2784411015 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v999) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb4cc05ed30 con 0x7fb4dc06a4a0 2024-03-11T21:47:51.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 10 librados: watch_flush enter 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 10 librados: watch_flush exit 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 20 client.42377.objecter shutdown clearing up homeless session... 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 10 client.42377.objecter successfully canceled tick 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 msgr2=0x7fb4c8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 0x7fb4c8071ce0 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009680 tx=0x7fb4c4009260 comp rx=0 tx=0).stop 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 msgr2=0x7fb4dc199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00d950 tx=0x7fb4cc00dc60 comp rx=0 tx=0).stop 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 shutdown_connections 2024-03-11T21:47:51.727 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb4c806f820 0x7fb4c8071ce0 unknown :-1 s=CLOSED pgs=3506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc10e920 0x7fb4dc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4dc06a4a0 0x7fb4dc199d60 secure :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00d950 tx=0x7fb4cc00dc60 comp rx=0 tx=0).stop 2024-03-11T21:47:51.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 --2- 172.21.15.143:0/2784411015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4dc06adc0 0x7fb4dc19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:47:51.728 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.736+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 >> 172.21.15.143:0/2784411015 conn(0x7fb4dc076d30 msgr2=0x7fb4dc0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:47:51.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.740+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 shutdown_connections 2024-03-11T21:47:51.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.740+0000 7fb4e2fd5700 1 -- 172.21.15.143:0/2784411015 wait complete. 2024-03-11T21:47:51.730 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:47:51.740+0000 7fb4e2fd5700 1 librados: shutdown 2024-03-11T21:47:51.732 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-11T21:47:52.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:51 smithi203 bash[23654]: cluster 2024-03-11T21:47:50.724577+0000 osd.0 (osd.0) 604 : cluster 0 9.1d scrub starts 2024-03-11T21:47:52.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:51 smithi203 bash[23654]: cluster 2024-03-11T21:47:50.729030+0000 osd.0 (osd.0) 605 : cluster 0 9.1d scrub ok 2024-03-11T21:47:52.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:51 smithi203 bash[23654]: audit 2024-03-11T21:47:51.633599+0000 mon.c (mon.2) 1408 : audit 1 from='client.? 172.21.15.143:0/2784411015' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:51 smithi203 bash[23654]: audit 2024-03-11T21:47:51.633944+0000 mon.a (mon.0) 4139 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[29284]: cluster 2024-03-11T21:47:50.724577+0000 osd.0 (osd.0) 604 : cluster 0 9.1d scrub starts 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[29284]: cluster 2024-03-11T21:47:50.729030+0000 osd.0 (osd.0) 605 : cluster 0 9.1d scrub ok 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[29284]: audit 2024-03-11T21:47:51.633599+0000 mon.c (mon.2) 1408 : audit 1 from='client.? 172.21.15.143:0/2784411015' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[29284]: audit 2024-03-11T21:47:51.633944+0000 mon.a (mon.0) 4139 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[21512]: cluster 2024-03-11T21:47:50.724577+0000 osd.0 (osd.0) 604 : cluster 0 9.1d scrub starts 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[21512]: cluster 2024-03-11T21:47:50.729030+0000 osd.0 (osd.0) 605 : cluster 0 9.1d scrub ok 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[21512]: audit 2024-03-11T21:47:51.633599+0000 mon.c (mon.2) 1408 : audit 1 from='client.? 172.21.15.143:0/2784411015' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.092 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:51 smithi143 bash[21512]: audit 2024-03-11T21:47:51.633944+0000 mon.a (mon.0) 4139 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:47:52.449 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:47:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:52.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:47:52.635 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.172759+0000 osd.4 (osd.4) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.173332+0000 osd.4 (osd.4) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: audit 2024-03-11T21:47:51.720683+0000 mon.a (mon.0) 4140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.731990+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e999: 8 total, 8 up, 8 in 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.781145+0000 osd.2 (osd.2) 503 : cluster 0 6.f scrub starts 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.782595+0000 osd.2 (osd.2) 504 : cluster 0 6.f scrub ok 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.791797+0000 osd.1 (osd.1) 395 : cluster 0 4.2 scrub starts 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.793760+0000 osd.1 (osd.1) 396 : cluster 0 4.2 scrub ok 2024-03-11T21:47:52.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:51.794254+0000 mgr.y (mgr.14550) 953 : cluster 0 pgmap v1622: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.219638+0000 osd.7 (osd.7) 596 : cluster 0 2.0 scrub starts 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.225842+0000 osd.7 (osd.7) 597 : cluster 0 2.0 scrub ok 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.172759+0000 osd.4 (osd.4) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.173332+0000 osd.4 (osd.4) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: audit 2024-03-11T21:47:51.720683+0000 mon.a (mon.0) 4140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.731990+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e999: 8 total, 8 up, 8 in 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.781145+0000 osd.2 (osd.2) 503 : cluster 0 6.f scrub starts 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.782595+0000 osd.2 (osd.2) 504 : cluster 0 6.f scrub ok 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.791797+0000 osd.1 (osd.1) 395 : cluster 0 4.2 scrub starts 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.793760+0000 osd.1 (osd.1) 396 : cluster 0 4.2 scrub ok 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:51.794254+0000 mgr.y (mgr.14550) 953 : cluster 0 pgmap v1622: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:52.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.219638+0000 osd.7 (osd.7) 596 : cluster 0 2.0 scrub starts 2024-03-11T21:47:52.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:52 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.225842+0000 osd.7 (osd.7) 597 : cluster 0 2.0 scrub ok 2024-03-11T21:47:53.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.172759+0000 osd.4 (osd.4) 577 : cluster 0 purged_snaps scrub starts 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.173332+0000 osd.4 (osd.4) 578 : cluster 0 purged_snaps scrub ok 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: audit 2024-03-11T21:47:51.720683+0000 mon.a (mon.0) 4140 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.731990+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e999: 8 total, 8 up, 8 in 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.781145+0000 osd.2 (osd.2) 503 : cluster 0 6.f scrub starts 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.782595+0000 osd.2 (osd.2) 504 : cluster 0 6.f scrub ok 2024-03-11T21:47:53.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.791797+0000 osd.1 (osd.1) 395 : cluster 0 4.2 scrub starts 2024-03-11T21:47:53.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.793760+0000 osd.1 (osd.1) 396 : cluster 0 4.2 scrub ok 2024-03-11T21:47:53.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:51.794254+0000 mgr.y (mgr.14550) 953 : cluster 0 pgmap v1622: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:47:53.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.219638+0000 osd.7 (osd.7) 596 : cluster 0 2.0 scrub starts 2024-03-11T21:47:53.034 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:52 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.225842+0000 osd.7 (osd.7) 597 : cluster 0 2.0 scrub ok 2024-03-11T21:47:54.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:53 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.735615+0000 osd.0 (osd.0) 606 : cluster 0 3.1 scrub starts 2024-03-11T21:47:54.007 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:53 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.738846+0000 osd.0 (osd.0) 607 : cluster 0 3.1 scrub ok 2024-03-11T21:47:54.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:53 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.837078+0000 osd.1 (osd.1) 397 : cluster 0 2.9 scrub starts 2024-03-11T21:47:54.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:53 smithi203 bash[23654]: cluster 2024-03-11T21:47:52.839496+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub ok 2024-03-11T21:47:54.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:53 smithi203 bash[23654]: audit 2024-03-11T21:47:53.059521+0000 mon.c (mon.2) 1409 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.735615+0000 osd.0 (osd.0) 606 : cluster 0 3.1 scrub starts 2024-03-11T21:47:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.738846+0000 osd.0 (osd.0) 607 : cluster 0 3.1 scrub ok 2024-03-11T21:47:54.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.837078+0000 osd.1 (osd.1) 397 : cluster 0 2.9 scrub starts 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[29284]: cluster 2024-03-11T21:47:52.839496+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub ok 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[29284]: audit 2024-03-11T21:47:53.059521+0000 mon.c (mon.2) 1409 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.735615+0000 osd.0 (osd.0) 606 : cluster 0 3.1 scrub starts 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.738846+0000 osd.0 (osd.0) 607 : cluster 0 3.1 scrub ok 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.837078+0000 osd.1 (osd.1) 397 : cluster 0 2.9 scrub starts 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[21512]: cluster 2024-03-11T21:47:52.839496+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub ok 2024-03-11T21:47:54.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:53 smithi143 bash[21512]: audit 2024-03-11T21:47:53.059521+0000 mon.c (mon.2) 1409 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:55.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:54 smithi203 bash[23654]: cluster 2024-03-11T21:47:53.311290+0000 osd.3 (osd.3) 601 : cluster 0 9.19 scrub starts 2024-03-11T21:47:55.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:54 smithi203 bash[23654]: cluster 2024-03-11T21:47:53.315433+0000 osd.3 (osd.3) 602 : cluster 0 9.19 scrub ok 2024-03-11T21:47:55.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:54 smithi203 bash[23654]: cluster 2024-03-11T21:47:53.796095+0000 mgr.y (mgr.14550) 954 : cluster 0 pgmap v1623: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:55.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:54 smithi203 bash[23654]: cluster 2024-03-11T21:47:54.191647+0000 osd.7 (osd.7) 598 : cluster 0 3.15 scrub starts 2024-03-11T21:47:55.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:54 smithi203 bash[23654]: cluster 2024-03-11T21:47:54.195751+0000 osd.7 (osd.7) 599 : cluster 0 3.15 scrub ok 2024-03-11T21:47:55.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[21512]: cluster 2024-03-11T21:47:53.311290+0000 osd.3 (osd.3) 601 : cluster 0 9.19 scrub starts 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[21512]: cluster 2024-03-11T21:47:53.315433+0000 osd.3 (osd.3) 602 : cluster 0 9.19 scrub ok 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[21512]: cluster 2024-03-11T21:47:53.796095+0000 mgr.y (mgr.14550) 954 : cluster 0 pgmap v1623: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[21512]: cluster 2024-03-11T21:47:54.191647+0000 osd.7 (osd.7) 598 : cluster 0 3.15 scrub starts 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[21512]: cluster 2024-03-11T21:47:54.195751+0000 osd.7 (osd.7) 599 : cluster 0 3.15 scrub ok 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[29284]: cluster 2024-03-11T21:47:53.311290+0000 osd.3 (osd.3) 601 : cluster 0 9.19 scrub starts 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[29284]: cluster 2024-03-11T21:47:53.315433+0000 osd.3 (osd.3) 602 : cluster 0 9.19 scrub ok 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[29284]: cluster 2024-03-11T21:47:53.796095+0000 mgr.y (mgr.14550) 954 : cluster 0 pgmap v1623: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[29284]: cluster 2024-03-11T21:47:54.191647+0000 osd.7 (osd.7) 598 : cluster 0 3.15 scrub starts 2024-03-11T21:47:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:54 smithi143 bash[29284]: cluster 2024-03-11T21:47:54.195751+0000 osd.7 (osd.7) 599 : cluster 0 3.15 scrub ok 2024-03-11T21:47:55.465 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:47:55.466 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:47:55.466 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:47:55.466 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:47:56.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:55 smithi203 bash[23654]: cluster 2024-03-11T21:47:54.300210+0000 osd.3 (osd.3) 603 : cluster 0 2.1b scrub starts 2024-03-11T21:47:56.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:55 smithi203 bash[23654]: cluster 2024-03-11T21:47:54.302729+0000 osd.3 (osd.3) 604 : cluster 0 2.1b scrub ok 2024-03-11T21:47:56.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:55 smithi143 bash[29284]: cluster 2024-03-11T21:47:54.300210+0000 osd.3 (osd.3) 603 : cluster 0 2.1b scrub starts 2024-03-11T21:47:56.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:55 smithi143 bash[29284]: cluster 2024-03-11T21:47:54.302729+0000 osd.3 (osd.3) 604 : cluster 0 2.1b scrub ok 2024-03-11T21:47:56.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:55 smithi143 bash[21512]: cluster 2024-03-11T21:47:54.300210+0000 osd.3 (osd.3) 603 : cluster 0 2.1b scrub starts 2024-03-11T21:47:56.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:55 smithi143 bash[21512]: cluster 2024-03-11T21:47:54.302729+0000 osd.3 (osd.3) 604 : cluster 0 2.1b scrub ok 2024-03-11T21:47:56.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:47:57.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:56 smithi203 bash[23654]: cluster 2024-03-11T21:47:55.797214+0000 mgr.y (mgr.14550) 955 : cluster 0 pgmap v1624: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:57.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:56 smithi143 bash[29284]: cluster 2024-03-11T21:47:55.797214+0000 mgr.y (mgr.14550) 955 : cluster 0 pgmap v1624: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:57.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:56 smithi143 bash[21512]: cluster 2024-03-11T21:47:55.797214+0000 mgr.y (mgr.14550) 955 : cluster 0 pgmap v1624: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:47:57.326 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:58.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.309698+0000 osd.3 (osd.3) 605 : cluster 0 3.18 scrub starts 2024-03-11T21:47:58.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.313494+0000 osd.3 (osd.3) 606 : cluster 0 3.18 scrub ok 2024-03-11T21:47:58.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.727108+0000 osd.2 (osd.2) 505 : cluster 0 1.a deep-scrub starts 2024-03-11T21:47:58.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.729659+0000 osd.0 (osd.0) 608 : cluster 0 3.1b scrub starts 2024-03-11T21:47:58.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.730105+0000 osd.2 (osd.2) 506 : cluster 0 1.a deep-scrub ok 2024-03-11T21:47:58.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.733384+0000 osd.0 (osd.0) 609 : cluster 0 3.1b scrub ok 2024-03-11T21:47:58.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.818610+0000 osd.1 (osd.1) 399 : cluster 0 9.4 scrub starts 2024-03-11T21:47:58.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:57 smithi203 bash[23654]: cluster 2024-03-11T21:47:56.821836+0000 osd.1 (osd.1) 400 : cluster 0 9.4 scrub ok 2024-03-11T21:47:58.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.309698+0000 osd.3 (osd.3) 605 : cluster 0 3.18 scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.313494+0000 osd.3 (osd.3) 606 : cluster 0 3.18 scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.727108+0000 osd.2 (osd.2) 505 : cluster 0 1.a deep-scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.729659+0000 osd.0 (osd.0) 608 : cluster 0 3.1b scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.730105+0000 osd.2 (osd.2) 506 : cluster 0 1.a deep-scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.733384+0000 osd.0 (osd.0) 609 : cluster 0 3.1b scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.818610+0000 osd.1 (osd.1) 399 : cluster 0 9.4 scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[21512]: cluster 2024-03-11T21:47:56.821836+0000 osd.1 (osd.1) 400 : cluster 0 9.4 scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.309698+0000 osd.3 (osd.3) 605 : cluster 0 3.18 scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.313494+0000 osd.3 (osd.3) 606 : cluster 0 3.18 scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.727108+0000 osd.2 (osd.2) 505 : cluster 0 1.a deep-scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.729659+0000 osd.0 (osd.0) 608 : cluster 0 3.1b scrub starts 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.730105+0000 osd.2 (osd.2) 506 : cluster 0 1.a deep-scrub ok 2024-03-11T21:47:58.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.733384+0000 osd.0 (osd.0) 609 : cluster 0 3.1b scrub ok 2024-03-11T21:47:58.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.818610+0000 osd.1 (osd.1) 399 : cluster 0 9.4 scrub starts 2024-03-11T21:47:58.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:57 smithi143 bash[29284]: cluster 2024-03-11T21:47:56.821836+0000 osd.1 (osd.1) 400 : cluster 0 9.4 scrub ok 2024-03-11T21:47:59.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:58 smithi203 bash[23654]: cluster 2024-03-11T21:47:57.798020+0000 mgr.y (mgr.14550) 956 : cluster 0 pgmap v1625: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-11T21:47:59.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:58 smithi203 bash[23654]: cluster 2024-03-11T21:47:57.860278+0000 osd.1 (osd.1) 401 : cluster 0 5.7 scrub starts 2024-03-11T21:47:59.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:58 smithi203 bash[23654]: cluster 2024-03-11T21:47:57.863263+0000 osd.1 (osd.1) 402 : cluster 0 5.7 scrub ok 2024-03-11T21:47:59.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:58 smithi203 bash[23654]: cluster 2024-03-11T21:47:58.187468+0000 osd.7 (osd.7) 600 : cluster 0 6.12 scrub starts 2024-03-11T21:47:59.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:58 smithi203 bash[23654]: cluster 2024-03-11T21:47:58.189719+0000 osd.7 (osd.7) 601 : cluster 0 6.12 scrub ok 2024-03-11T21:47:59.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[21512]: cluster 2024-03-11T21:47:57.798020+0000 mgr.y (mgr.14550) 956 : cluster 0 pgmap v1625: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-11T21:47:59.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[21512]: cluster 2024-03-11T21:47:57.860278+0000 osd.1 (osd.1) 401 : cluster 0 5.7 scrub starts 2024-03-11T21:47:59.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[21512]: cluster 2024-03-11T21:47:57.863263+0000 osd.1 (osd.1) 402 : cluster 0 5.7 scrub ok 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[21512]: cluster 2024-03-11T21:47:58.187468+0000 osd.7 (osd.7) 600 : cluster 0 6.12 scrub starts 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[21512]: cluster 2024-03-11T21:47:58.189719+0000 osd.7 (osd.7) 601 : cluster 0 6.12 scrub ok 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[29284]: cluster 2024-03-11T21:47:57.798020+0000 mgr.y (mgr.14550) 956 : cluster 0 pgmap v1625: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[29284]: cluster 2024-03-11T21:47:57.860278+0000 osd.1 (osd.1) 401 : cluster 0 5.7 scrub starts 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[29284]: cluster 2024-03-11T21:47:57.863263+0000 osd.1 (osd.1) 402 : cluster 0 5.7 scrub ok 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[29284]: cluster 2024-03-11T21:47:58.187468+0000 osd.7 (osd.7) 600 : cluster 0 6.12 scrub starts 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:58 smithi143 bash[29284]: cluster 2024-03-11T21:47:58.189719+0000 osd.7 (osd.7) 601 : cluster 0 6.12 scrub ok 2024-03-11T21:47:59.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:47:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:00.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:59 smithi203 bash[23654]: cluster 2024-03-11T21:47:58.870857+0000 osd.1 (osd.1) 403 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:48:00.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:47:59 smithi203 bash[23654]: cluster 2024-03-11T21:47:58.872657+0000 osd.1 (osd.1) 404 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:48:00.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:59 smithi143 bash[21512]: cluster 2024-03-11T21:47:58.870857+0000 osd.1 (osd.1) 403 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:48:00.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:47:59 smithi143 bash[21512]: cluster 2024-03-11T21:47:58.872657+0000 osd.1 (osd.1) 404 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:48:00.199 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:59 smithi143 bash[29284]: cluster 2024-03-11T21:47:58.870857+0000 osd.1 (osd.1) 403 : cluster 0 9.0 deep-scrub starts 2024-03-11T21:48:00.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:47:59 smithi143 bash[29284]: cluster 2024-03-11T21:47:58.872657+0000 osd.1 (osd.1) 404 : cluster 0 9.0 deep-scrub ok 2024-03-11T21:48:00.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/27976904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 msgr2=0x7fed1c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:00.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- 172.21.15.143:0/27976904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c102340 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b0a0 tx=0x7fed0c00b3b0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/27976904 shutdown_connections 2024-03-11T21:48:00.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- 172.21.15.143:0/27976904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- 172.21.15.143:0/27976904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 0x7fed1c108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- 172.21.15.143:0/27976904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c102340 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/27976904 >> 172.21.15.143:0/27976904 conn(0x7fed1c078eb0 msgr2=0x7fed1c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/27976904 shutdown_connections 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/27976904 wait complete. 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 librados: starting msgr at 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 librados: starting objecter 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 Processor -- start 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- start start 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 librados: setting wanted keys 2024-03-11T21:48:00.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 librados: calling monclient init 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 0x7fed1c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c1116a0 con 0x7fed1c102880 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c111520 con 0x7fed1c107f30 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c111820 con 0x7fed1c101ec0 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2111b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2111b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:34780/0 (socket says 172.21.15.143:34780) 2024-03-11T21:48:00.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2111b700 1 -- 172.21.15.143:0/1729442018 learned_addr learned my addr 172.21.15.143:0/1729442018 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2191c700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 0x7fed1c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 -- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 msgr2=0x7fed1c19d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 -- 172.21.15.143:0/1729442018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 msgr2=0x7fed1c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 0x7fed1c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 -- 172.21.15.143:0/1729442018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c198220 con 0x7fed1c102880 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2111b700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:48:00.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2091a700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c19de40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b070 tx=0x7fed0c0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed1a7fc700 10 client.?.objecter ms_handle_connect 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed1a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c021020 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fed0c004c50 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c1984b0 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.644+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c005050 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2337f700 10 client.31179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c1a47d0 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2337f700 1 librados: init done 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c04faa0 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fed0c024020 con 0x7fed1c102880 2024-03-11T21:48:00.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 0x7fed08071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(999..999 src has 944..999) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fed0c00f020 con 0x7fed1c102880 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2111b700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 0x7fed08071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 10 client.31179.objecter ms_dispatch 0x7fed1c001550 osd_map(999..999 src has 944..999) v4 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 3 client.31179.objecter handle_osd_map got epochs [999,999] > 0 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 3 client.31179.objecter handle_osd_map decoding full epoch 999 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 20 client.31179.objecter dump_active .. 0 homeless 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed1a7fc700 10 client.31179.objecter ms_handle_connect 0x7fed0806f820 2024-03-11T21:48:00.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.648+0000 7fed2111b700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 0x7fed08071ce0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7fed1c198c00 tx=0x7fed04009290 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:00.645 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.652+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fed0c05c7d0 con 0x7fed1c102880 2024-03-11T21:48:00.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:00.872+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fed1c1a49a0 con 0x7fed1c102880 2024-03-11T21:48:01.010 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:01.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:00 smithi143 bash[21512]: audit 2024-03-11T21:47:59.339190+0000 mgr.y (mgr.14550) 957 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:01.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:00 smithi143 bash[21512]: cluster 2024-03-11T21:47:59.799791+0000 mgr.y (mgr.14550) 958 : cluster 0 pgmap v1626: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:01.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:00 smithi143 bash[29284]: audit 2024-03-11T21:47:59.339190+0000 mgr.y (mgr.14550) 957 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:01.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:00 smithi143 bash[29284]: cluster 2024-03-11T21:47:59.799791+0000 mgr.y (mgr.14550) 958 : cluster 0 pgmap v1626: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:00 smithi203 bash[23654]: audit 2024-03-11T21:47:59.339190+0000 mgr.y (mgr.14550) 957 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:01.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:00 smithi203 bash[23654]: cluster 2024-03-11T21:47:59.799791+0000 mgr.y (mgr.14550) 958 : cluster 0 pgmap v1626: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:01.797 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.804+0000 7fed1a7fc700 1 -- 172.21.15.143:0/1729442018 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v1000) v1 ==== 113+0+0 (secure 0 0 0) 0x7fed0c060220 con 0x7fed1c102880 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 10 librados: watch_flush enter 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 10 librados: watch_flush exit 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 20 client.31179.objecter shutdown clearing up homeless session... 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 10 client.31179.objecter successfully canceled tick 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 msgr2=0x7fed08071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 0x7fed08071ce0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7fed1c198c00 tx=0x7fed04009290 comp rx=0 tx=0).stop 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 msgr2=0x7fed1c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c19de40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b070 tx=0x7fed0c0079a0 comp rx=0 tx=0).stop 2024-03-11T21:48:01.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 shutdown_connections 2024-03-11T21:48:01.802 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fed0806f820 0x7fed08071ce0 unknown :-1 s=CLOSED pgs=3507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c107f30 0x7fed1c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed1c101ec0 0x7fed1c19d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 --2- 172.21.15.143:0/1729442018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed1c102880 0x7fed1c19de40 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 >> 172.21.15.143:0/1729442018 conn(0x7fed1c078eb0 msgr2=0x7fed1c105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 shutdown_connections 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 -- 172.21.15.143:0/1729442018 wait complete. 2024-03-11T21:48:01.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:01.808+0000 7fed2337f700 1 librados: shutdown 2024-03-11T21:48:01.804 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:48:02.041 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:01 smithi203 bash[23654]: audit 2024-03-11T21:48:00.876524+0000 mon.a (mon.0) 4142 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:02.054 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:01 smithi143 bash[21512]: audit 2024-03-11T21:48:00.876524+0000 mon.a (mon.0) 4142 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:02.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:01 smithi143 bash[29284]: audit 2024-03-11T21:48:00.876524+0000 mon.a (mon.0) 4142 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:02.449 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:02.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-11T21:48:02.949 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:01.757685+0000 osd.0 (osd.0) 610 : cluster 0 5.1e scrub starts 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:01.759396+0000 osd.0 (osd.0) 611 : cluster 0 5.1e scrub ok 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:01.801118+0000 mgr.y (mgr.14550) 959 : cluster 0 pgmap v1627: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: audit 2024-03-11T21:48:01.809839+0000 mon.a (mon.0) 4143 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:01.817008+0000 mon.a (mon.0) 4144 : cluster 0 osdmap e1000: 8 total, 8 up, 8 in 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:02.145234+0000 osd.7 (osd.7) 602 : cluster 0 3.16 scrub starts 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[21512]: cluster 2024-03-11T21:48:02.147905+0000 osd.7 (osd.7) 603 : cluster 0 3.16 scrub ok 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:01.757685+0000 osd.0 (osd.0) 610 : cluster 0 5.1e scrub starts 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:01.759396+0000 osd.0 (osd.0) 611 : cluster 0 5.1e scrub ok 2024-03-11T21:48:02.950 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:01.801118+0000 mgr.y (mgr.14550) 959 : cluster 0 pgmap v1627: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:48:02.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: audit 2024-03-11T21:48:01.809839+0000 mon.a (mon.0) 4143 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:02.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:01.817008+0000 mon.a (mon.0) 4144 : cluster 0 osdmap e1000: 8 total, 8 up, 8 in 2024-03-11T21:48:02.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:02.145234+0000 osd.7 (osd.7) 602 : cluster 0 3.16 scrub starts 2024-03-11T21:48:02.951 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:02 smithi143 bash[29284]: cluster 2024-03-11T21:48:02.147905+0000 osd.7 (osd.7) 603 : cluster 0 3.16 scrub ok 2024-03-11T21:48:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:01.757685+0000 osd.0 (osd.0) 610 : cluster 0 5.1e scrub starts 2024-03-11T21:48:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:01.759396+0000 osd.0 (osd.0) 611 : cluster 0 5.1e scrub ok 2024-03-11T21:48:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:01.801118+0000 mgr.y (mgr.14550) 959 : cluster 0 pgmap v1627: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-03-11T21:48:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: audit 2024-03-11T21:48:01.809839+0000 mon.a (mon.0) 4143 : audit 1 from='client.? 172.21.15.143:0/1729442018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:03.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:01.817008+0000 mon.a (mon.0) 4144 : cluster 0 osdmap e1000: 8 total, 8 up, 8 in 2024-03-11T21:48:03.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:02.145234+0000 osd.7 (osd.7) 602 : cluster 0 3.16 scrub starts 2024-03-11T21:48:03.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:02 smithi203 bash[23654]: cluster 2024-03-11T21:48:02.147905+0000 osd.7 (osd.7) 603 : cluster 0 3.16 scrub ok 2024-03-11T21:48:03.754 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:03.754 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:48:03.754 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:48:03.754 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:48:03.754 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:04.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:02.521441+0000 osd.3 (osd.3) 607 : cluster 0 9.8 scrub starts 2024-03-11T21:48:04.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:02.523677+0000 osd.3 (osd.3) 608 : cluster 0 9.8 scrub ok 2024-03-11T21:48:04.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.112435+0000 osd.7 (osd.7) 604 : cluster 0 6.15 scrub starts 2024-03-11T21:48:04.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.114768+0000 osd.7 (osd.7) 605 : cluster 0 6.15 scrub ok 2024-03-11T21:48:04.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.211731+0000 osd.7 (osd.7) 606 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:04.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.212202+0000 osd.7 (osd.7) 607 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:04.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:02.521441+0000 osd.3 (osd.3) 607 : cluster 0 9.8 scrub starts 2024-03-11T21:48:04.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:02.523677+0000 osd.3 (osd.3) 608 : cluster 0 9.8 scrub ok 2024-03-11T21:48:04.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.112435+0000 osd.7 (osd.7) 604 : cluster 0 6.15 scrub starts 2024-03-11T21:48:04.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.114768+0000 osd.7 (osd.7) 605 : cluster 0 6.15 scrub ok 2024-03-11T21:48:04.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.211731+0000 osd.7 (osd.7) 606 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:04.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:03 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.212202+0000 osd.7 (osd.7) 607 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:02.521441+0000 osd.3 (osd.3) 607 : cluster 0 9.8 scrub starts 2024-03-11T21:48:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:02.523677+0000 osd.3 (osd.3) 608 : cluster 0 9.8 scrub ok 2024-03-11T21:48:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.112435+0000 osd.7 (osd.7) 604 : cluster 0 6.15 scrub starts 2024-03-11T21:48:04.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.114768+0000 osd.7 (osd.7) 605 : cluster 0 6.15 scrub ok 2024-03-11T21:48:04.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.211731+0000 osd.7 (osd.7) 606 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:04.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:03 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.212202+0000 osd.7 (osd.7) 607 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:04.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:48:05.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.494089+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.499225+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.797687+0000 osd.0 (osd.0) 612 : cluster 0 1.11 scrub starts 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.800012+0000 osd.0 (osd.0) 613 : cluster 0 1.11 scrub ok 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.802882+0000 mgr.y (mgr.14550) 960 : cluster 0 pgmap v1629: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:03.840979+0000 mon.a (mon.0) 4145 : cluster 0 osdmap e1001: 8 total, 8 up, 8 in 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.122264+0000 osd.7 (osd.7) 608 : cluster 0 9.13 scrub starts 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.127532+0000 osd.7 (osd.7) 609 : cluster 0 9.13 scrub ok 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.494089+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.499225+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-03-11T21:48:05.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.797687+0000 osd.0 (osd.0) 612 : cluster 0 1.11 scrub starts 2024-03-11T21:48:05.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.800012+0000 osd.0 (osd.0) 613 : cluster 0 1.11 scrub ok 2024-03-11T21:48:05.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.802882+0000 mgr.y (mgr.14550) 960 : cluster 0 pgmap v1629: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:05.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:03.840979+0000 mon.a (mon.0) 4145 : cluster 0 osdmap e1001: 8 total, 8 up, 8 in 2024-03-11T21:48:05.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.122264+0000 osd.7 (osd.7) 608 : cluster 0 9.13 scrub starts 2024-03-11T21:48:05.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:04 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.127532+0000 osd.7 (osd.7) 609 : cluster 0 9.13 scrub ok 2024-03-11T21:48:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.494089+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-03-11T21:48:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.499225+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-03-11T21:48:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.797687+0000 osd.0 (osd.0) 612 : cluster 0 1.11 scrub starts 2024-03-11T21:48:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.800012+0000 osd.0 (osd.0) 613 : cluster 0 1.11 scrub ok 2024-03-11T21:48:05.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.802882+0000 mgr.y (mgr.14550) 960 : cluster 0 pgmap v1629: 184 pgs: 184 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:05.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:03.840979+0000 mon.a (mon.0) 4145 : cluster 0 osdmap e1001: 8 total, 8 up, 8 in 2024-03-11T21:48:05.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.122264+0000 osd.7 (osd.7) 608 : cluster 0 9.13 scrub starts 2024-03-11T21:48:05.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:04 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.127532+0000 osd.7 (osd.7) 609 : cluster 0 9.13 scrub ok 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.207125+0000 osd.4 (osd.4) 579 : cluster 0 3.9 scrub starts 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.211305+0000 osd.4 (osd.4) 580 : cluster 0 3.9 scrub ok 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.767818+0000 osd.0 (osd.0) 614 : cluster 0 6.9 scrub starts 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:04.771992+0000 osd.0 (osd.0) 615 : cluster 0 6.9 scrub ok 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:05.099346+0000 osd.7 (osd.7) 610 : cluster 0 3.e scrub starts 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[21512]: cluster 2024-03-11T21:48:05.103515+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub ok 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.207125+0000 osd.4 (osd.4) 579 : cluster 0 3.9 scrub starts 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.211305+0000 osd.4 (osd.4) 580 : cluster 0 3.9 scrub ok 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.767818+0000 osd.0 (osd.0) 614 : cluster 0 6.9 scrub starts 2024-03-11T21:48:06.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:04.771992+0000 osd.0 (osd.0) 615 : cluster 0 6.9 scrub ok 2024-03-11T21:48:06.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:05.099346+0000 osd.7 (osd.7) 610 : cluster 0 3.e scrub starts 2024-03-11T21:48:06.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:05 smithi143 bash[29284]: cluster 2024-03-11T21:48:05.103515+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub ok 2024-03-11T21:48:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.207125+0000 osd.4 (osd.4) 579 : cluster 0 3.9 scrub starts 2024-03-11T21:48:06.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.211305+0000 osd.4 (osd.4) 580 : cluster 0 3.9 scrub ok 2024-03-11T21:48:06.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.767818+0000 osd.0 (osd.0) 614 : cluster 0 6.9 scrub starts 2024-03-11T21:48:06.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:04.771992+0000 osd.0 (osd.0) 615 : cluster 0 6.9 scrub ok 2024-03-11T21:48:06.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:05.099346+0000 osd.7 (osd.7) 610 : cluster 0 3.e scrub starts 2024-03-11T21:48:06.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:05 smithi203 bash[23654]: cluster 2024-03-11T21:48:05.103515+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub ok 2024-03-11T21:48:07.199 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[21512]: cluster 2024-03-11T21:48:05.803902+0000 mgr.y (mgr.14550) 961 : cluster 0 pgmap v1631: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[21512]: cluster 2024-03-11T21:48:06.060662+0000 osd.7 (osd.7) 612 : cluster 0 2.1d scrub starts 2024-03-11T21:48:07.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[21512]: cluster 2024-03-11T21:48:06.062988+0000 osd.7 (osd.7) 613 : cluster 0 2.1d scrub ok 2024-03-11T21:48:07.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[29284]: cluster 2024-03-11T21:48:05.803902+0000 mgr.y (mgr.14550) 961 : cluster 0 pgmap v1631: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:07.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[29284]: cluster 2024-03-11T21:48:06.060662+0000 osd.7 (osd.7) 612 : cluster 0 2.1d scrub starts 2024-03-11T21:48:07.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:06 smithi143 bash[29284]: cluster 2024-03-11T21:48:06.062988+0000 osd.7 (osd.7) 613 : cluster 0 2.1d scrub ok 2024-03-11T21:48:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:06 smithi203 bash[23654]: cluster 2024-03-11T21:48:05.803902+0000 mgr.y (mgr.14550) 961 : cluster 0 pgmap v1631: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:06 smithi203 bash[23654]: cluster 2024-03-11T21:48:06.060662+0000 osd.7 (osd.7) 612 : cluster 0 2.1d scrub starts 2024-03-11T21:48:07.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:06 smithi203 bash[23654]: cluster 2024-03-11T21:48:06.062988+0000 osd.7 (osd.7) 613 : cluster 0 2.1d scrub ok 2024-03-11T21:48:07.477 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:08.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:07 smithi143 bash[21512]: audit 2024-03-11T21:48:07.350510+0000 mon.c (mon.2) 1410 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:08.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:07 smithi143 bash[29284]: audit 2024-03-11T21:48:07.350510+0000 mon.c (mon.2) 1410 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:08.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:07 smithi203 bash[23654]: audit 2024-03-11T21:48:07.350510+0000 mon.c (mon.2) 1410 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: cluster 2024-03-11T21:48:07.804601+0000 mgr.y (mgr.14550) 962 : cluster 0 pgmap v1632: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: cluster 2024-03-11T21:48:07.811840+0000 osd.6 (osd.6) 489 : cluster 0 3.a scrub starts 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: cluster 2024-03-11T21:48:07.814035+0000 osd.6 (osd.6) 490 : cluster 0 3.a scrub ok 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: audit 2024-03-11T21:48:08.094100+0000 mon.c (mon.2) 1411 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: audit 2024-03-11T21:48:08.094741+0000 mon.c (mon.2) 1412 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: audit 2024-03-11T21:48:08.097170+0000 mon.c (mon.2) 1413 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: audit 2024-03-11T21:48:08.120794+0000 mon.a (mon.0) 4146 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: cluster 2024-03-11T21:48:08.251126+0000 osd.4 (osd.4) 581 : cluster 0 4.1 scrub starts 2024-03-11T21:48:09.146 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[29284]: cluster 2024-03-11T21:48:08.253293+0000 osd.4 (osd.4) 582 : cluster 0 4.1 scrub ok 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: cluster 2024-03-11T21:48:07.804601+0000 mgr.y (mgr.14550) 962 : cluster 0 pgmap v1632: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: cluster 2024-03-11T21:48:07.811840+0000 osd.6 (osd.6) 489 : cluster 0 3.a scrub starts 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: cluster 2024-03-11T21:48:07.814035+0000 osd.6 (osd.6) 490 : cluster 0 3.a scrub ok 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: audit 2024-03-11T21:48:08.094100+0000 mon.c (mon.2) 1411 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: audit 2024-03-11T21:48:08.094741+0000 mon.c (mon.2) 1412 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: audit 2024-03-11T21:48:08.097170+0000 mon.c (mon.2) 1413 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: audit 2024-03-11T21:48:08.120794+0000 mon.a (mon.0) 4146 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: cluster 2024-03-11T21:48:08.251126+0000 osd.4 (osd.4) 581 : cluster 0 4.1 scrub starts 2024-03-11T21:48:09.147 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:08 smithi143 bash[21512]: cluster 2024-03-11T21:48:08.253293+0000 osd.4 (osd.4) 582 : cluster 0 4.1 scrub ok 2024-03-11T21:48:09.281 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: cluster 2024-03-11T21:48:07.804601+0000 mgr.y (mgr.14550) 962 : cluster 0 pgmap v1632: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: cluster 2024-03-11T21:48:07.811840+0000 osd.6 (osd.6) 489 : cluster 0 3.a scrub starts 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: cluster 2024-03-11T21:48:07.814035+0000 osd.6 (osd.6) 490 : cluster 0 3.a scrub ok 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: audit 2024-03-11T21:48:08.094100+0000 mon.c (mon.2) 1411 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: audit 2024-03-11T21:48:08.094741+0000 mon.c (mon.2) 1412 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: audit 2024-03-11T21:48:08.097170+0000 mon.c (mon.2) 1413 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: audit 2024-03-11T21:48:08.120794+0000 mon.a (mon.0) 4146 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: cluster 2024-03-11T21:48:08.251126+0000 osd.4 (osd.4) 581 : cluster 0 4.1 scrub starts 2024-03-11T21:48:09.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:08 smithi203 bash[23654]: cluster 2024-03-11T21:48:08.253293+0000 osd.4 (osd.4) 582 : cluster 0 4.1 scrub ok 2024-03-11T21:48:09.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:10.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/3184584318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c101ec0 msgr2=0x7f359c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- 172.21.15.143:0/3184584318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c101ec0 0x7f359c102340 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f3584009990 tx=0x7f3584009ca0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/3184584318 shutdown_connections 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- 172.21.15.143:0/3184584318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c107f30 0x7f359c108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- 172.21.15.143:0/3184584318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c102880 0x7f359c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- 172.21.15.143:0/3184584318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c101ec0 0x7f359c102340 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/3184584318 >> 172.21.15.143:0/3184584318 conn(0x7f359c078eb0 msgr2=0x7f359c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/3184584318 shutdown_connections 2024-03-11T21:48:10.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/3184584318 wait complete. 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 librados: starting msgr at 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 librados: starting objecter 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 Processor -- start 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- start start 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 librados: setting wanted keys 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 librados: calling monclient init 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c102880 0x7f359c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c107f30 0x7f359c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c111690 con 0x7f359c101ec0 2024-03-11T21:48:10.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c111510 con 0x7f359c107f30 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c111810 con 0x7f359c102880 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33778/0 (socket says 172.21.15.143:33778) 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 -- 172.21.15.143:0/795434172 learned_addr learned my addr 172.21.15.143:0/795434172 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359b7fe700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c102880 0x7f359c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 -- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c102880 msgr2=0x7f359c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a08f8700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c107f30 0x7f359c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c102880 0x7f359c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 -- 172.21.15.143:0/795434172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c107f30 msgr2=0x7f359c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c107f30 0x7f359c074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 -- 172.21.15.143:0/795434172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c1a6140 con 0x7f359c101ec0 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f359bfff700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f358c00e9a0 tx=0x7f358c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:10.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35997fa700 10 client.?.objecter ms_handle_connect 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35997fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00cbd0 con 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f358c00cd70 con 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c1a6430 con 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c010e60 con 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 10 client.31185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.931+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1a6940 con 0x7f359c101ec0 2024-03-11T21:48:10.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35a235b700 1 librados: init done 2024-03-11T21:48:10.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c04faa0 con 0x7f359c101ec0 2024-03-11T21:48:10.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f358c010430 con 0x7f359c101ec0 2024-03-11T21:48:10.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 0x7f3588071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:10.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1001..1001 src has 944..1001) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f358c08e970 con 0x7f359c101ec0 2024-03-11T21:48:10.926 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 10 client.31185.objecter ms_dispatch 0x7f359c001550 osd_map(1001..1001 src has 944..1001) v4 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 3 client.31185.objecter handle_osd_map got epochs [1001,1001] > 0 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 3 client.31185.objecter handle_osd_map decoding full epoch 1001 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f359b7fe700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 0x7f3588071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 20 client.31185.objecter dump_active .. 0 homeless 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f35997fa700 10 client.31185.objecter ms_handle_connect 0x7f358806f820 2024-03-11T21:48:10.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.935+0000 7f359b7fe700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 0x7f3588071ce0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f358400b2e0 tx=0x7f3584005960 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:10.929 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:10.939+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f358c05b900 con 0x7f359c101ec0 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: audit 2024-03-11T21:48:09.350047+0000 mgr.y (mgr.14550) 963 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: cluster 2024-03-11T21:48:09.545843+0000 osd.3 (osd.3) 611 : cluster 0 2.19 scrub starts 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: cluster 2024-03-11T21:48:09.547984+0000 osd.3 (osd.3) 612 : cluster 0 2.19 scrub ok 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: cluster 2024-03-11T21:48:09.806478+0000 mgr.y (mgr.14550) 964 : cluster 0 pgmap v1633: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: cluster 2024-03-11T21:48:10.189719+0000 osd.4 (osd.4) 583 : cluster 0 9.16 scrub starts 2024-03-11T21:48:11.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:10 smithi203 bash[23654]: cluster 2024-03-11T21:48:10.196230+0000 osd.4 (osd.4) 584 : cluster 0 9.16 scrub ok 2024-03-11T21:48:11.151 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.163+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f359c1a6bc0 con 0x7f359c101ec0 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: audit 2024-03-11T21:48:09.350047+0000 mgr.y (mgr.14550) 963 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: cluster 2024-03-11T21:48:09.545843+0000 osd.3 (osd.3) 611 : cluster 0 2.19 scrub starts 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: cluster 2024-03-11T21:48:09.547984+0000 osd.3 (osd.3) 612 : cluster 0 2.19 scrub ok 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: cluster 2024-03-11T21:48:09.806478+0000 mgr.y (mgr.14550) 964 : cluster 0 pgmap v1633: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: cluster 2024-03-11T21:48:10.189719+0000 osd.4 (osd.4) 583 : cluster 0 9.16 scrub starts 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[21512]: cluster 2024-03-11T21:48:10.196230+0000 osd.4 (osd.4) 584 : cluster 0 9.16 scrub ok 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: audit 2024-03-11T21:48:09.350047+0000 mgr.y (mgr.14550) 963 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:11.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: cluster 2024-03-11T21:48:09.545843+0000 osd.3 (osd.3) 611 : cluster 0 2.19 scrub starts 2024-03-11T21:48:11.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: cluster 2024-03-11T21:48:09.547984+0000 osd.3 (osd.3) 612 : cluster 0 2.19 scrub ok 2024-03-11T21:48:11.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: cluster 2024-03-11T21:48:09.806478+0000 mgr.y (mgr.14550) 964 : cluster 0 pgmap v1633: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:11.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: cluster 2024-03-11T21:48:10.189719+0000 osd.4 (osd.4) 583 : cluster 0 9.16 scrub starts 2024-03-11T21:48:11.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:10 smithi143 bash[29284]: cluster 2024-03-11T21:48:10.196230+0000 osd.4 (osd.4) 584 : cluster 0 9.16 scrub ok 2024-03-11T21:48:11.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.867+0000 7f35997fa700 1 -- 172.21.15.143:0/795434172 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v1002) v1 ==== 133+0+0 (secure 0 0 0) 0x7f358c07b0f0 con 0x7f359c101ec0 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 10 librados: watch_flush enter 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 10 librados: watch_flush exit 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 20 client.31185.objecter shutdown clearing up homeless session... 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 10 client.31185.objecter successfully canceled tick 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 msgr2=0x7f3588071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 0x7f3588071ce0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f358400b2e0 tx=0x7f3584005960 comp rx=0 tx=0).stop 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 msgr2=0x7f359c0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f358c00e9a0 tx=0x7f358c00ecb0 comp rx=0 tx=0).stop 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 shutdown_connections 2024-03-11T21:48:11.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f358806f820 0x7f3588071ce0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f359c101ec0 0x7f359c0740e0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c107f30 0x7f359c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 --2- 172.21.15.143:0/795434172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f359c102880 0x7f359c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 >> 172.21.15.143:0/795434172 conn(0x7f359c078eb0 msgr2=0x7f359c108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 shutdown_connections 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 -- 172.21.15.143:0/795434172 wait complete. 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:11.871+0000 7f35a235b700 1 librados: shutdown 2024-03-11T21:48:11.862 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[21512]: audit 2024-03-11T21:48:11.165018+0000 mon.a (mon.0) 4147 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.185036+0000 osd.4 (osd.4) 585 : cluster 0 2.f scrub starts 2024-03-11T21:48:12.121 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.187546+0000 osd.4 (osd.4) 586 : cluster 0 2.f scrub ok 2024-03-11T21:48:12.121 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[29284]: audit 2024-03-11T21:48:11.165018+0000 mon.a (mon.0) 4147 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:12.121 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.185036+0000 osd.4 (osd.4) 585 : cluster 0 2.f scrub starts 2024-03-11T21:48:12.121 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:11 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.187546+0000 osd.4 (osd.4) 586 : cluster 0 2.f scrub ok 2024-03-11T21:48:12.209 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:12.209 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:48:12.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:269 316.14 316:28000000:%01:::head [pgnls start_epoch 713] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e713)", 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:12.543835+0000", 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 479.67308294999998, 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098804559999999993, 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 269 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.210 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543835+0000", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543835+0000", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543838+0000", 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4790000000000001e-06 2024-03-11T21:48:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543855+0000", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7084000000000001e-05 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543922+0000", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6871000000000004e-05 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.543951+0000", 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9036000000000001e-05 2024-03-11T21:48:12.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.544263+0000", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031158999999999999 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.544280+0000", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7085999999999999e-05 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.553352+0000", 2024-03-11T21:48:12.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0090720409999999994 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.553457+0000", 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105614 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.553715+0000", 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025765500000000003 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.b,QREQUEST,e:719,from:0,reservation_nonce:8)", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:18.615419+0000", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 473.60149855200001, 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059829080000000003, 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615419+0000", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615419+0000", 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615421+0000", 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3269999999999999e-06 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615434+0000", 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3066e-05 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615468+0000", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3458999999999999e-05 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615484+0000", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6252999999999998e-05 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615646+0000", 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000161502 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.615665+0000", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8919e-05 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.621203+0000", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0055383180000000004 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.621247+0000", 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3952000000000002e-05 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:18.621402+0000", 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000155112 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:414 317.c 317:30000000::::head [pgnls start_epoch 725] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e725)", 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:24.621961+0000", 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 467.59495691500001, 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0092719550000000001, 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.219 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 414 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.621961+0000", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.621961+0000", 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.621964+0000", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3699999999999999e-06 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.621982+0000", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7518999999999999e-05 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.622020+0000", 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8798999999999999e-05 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.622040+0000", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9830000000000002e-05 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.622230+0000", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018993800000000001 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.622251+0000", 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1265000000000001e-05 2024-03-11T21:48:12.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.630916+0000", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086650640000000001 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.631009+0000", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2983000000000001e-05 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.631233+0000", 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022318699999999999 2024-03-11T21:48:12.223 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.1.295:41 313.1s0 313:98ec0ffc:test-rados-api-smithi143-71773-125::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e748)", 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.620122+0000", 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 444.596796047, 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062010989999999998, 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620122+0000", 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620122+0000", 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620123+0000", 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.850000000000001e-07 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620130+0000", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6299999999999998e-06 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620144+0000", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3772e-05 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620155+0000", 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0961000000000001e-05 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620345+0000", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018980099999999999 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting_for_map not empty", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.620349+0000", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.391e-06 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.624905+0000", 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045554009999999997 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.625031+0000", 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000126002 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.626323+0000", 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001292156 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:828 320.16 320:68000000:%01:::head [pgnls start_epoch 749] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e749)", 2024-03-11T21:48:12.228 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:48.694174+0000", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 443.52274370999999, 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010131485000000001, 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 828 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694174+0000", 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694174+0000", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694177+0000", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.303e-06 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694193+0000", 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5869999999999999e-05 2024-03-11T21:48:12.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694234+0000", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0905000000000002e-05 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694260+0000", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5982999999999998e-05 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694489+0000", 2024-03-11T21:48:12.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022914300000000001 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.694506+0000", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6719999999999999e-05 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.703979+0000", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094727700000000001 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.704073+0000", 2024-03-11T21:48:12.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4067000000000006e-05 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.704305+0000", 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023272399999999999 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:916 321.0 321:00000000:test-rados-api-smithi143-71773-127:::head [pgnls start_epoch 757] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e757)", 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:56.582831+0000", 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.63408619099999, 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0077930910000000003, 2024-03-11T21:48:12.233 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 916 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582831+0000", 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582831+0000", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582835+0000", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5999999999999998e-06 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582858+0000", 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2952e-05 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582908+0000", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9985000000000003e-05 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.582938+0000", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0136000000000001e-05 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.583111+0000", 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000172695 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.583127+0000", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6606e-05 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.590123+0000", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069959949999999996 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.590227+0000", 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010351 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.590625+0000", 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039761199999999999 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.6.56:74 313.1s0 313:8e7c84a9:test-rados-api-smithi143-71773-129::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e765)", 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:04.570573+0000", 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 427.64634431600001, 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057627210000000002, 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.238 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570573+0000", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570573+0000", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570578+0000", 2024-03-11T21:48:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3669999999999999e-06 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570596+0000", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8573e-05 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570646+0000", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0062000000000003e-05 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570677+0000", 2024-03-11T21:48:12.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1118999999999998e-05 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570923+0000", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024603 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.570946+0000", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2558999999999999e-05 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.576025+0000", 2024-03-11T21:48:12.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0050785700000000001 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.576336+0000", 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031144099999999997 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1735 325.1c 325:38000000::::head [pgnls start_epoch 799] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e799)", 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:40.633892+0000", 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.58302596099998, 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083313659999999998, 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1735 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633892+0000", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633892+0000", 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633895+0000", 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4259999999999999e-06 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633912+0000", 2024-03-11T21:48:12.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6767000000000001e-05 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633949+0000", 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7526e-05 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.633969+0000", 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9749e-05 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.634174+0000", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020484499999999999 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.634185+0000", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1286e-05 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.641904+0000", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0077191689999999997 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.641999+0000", 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4591000000000003e-05 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.642223+0000", 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000224007 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1834 326.4 326:20000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:48.604348+0000", 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 383.612569565, 2024-03-11T21:48:12.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011376669000000001, 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1834 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604348+0000", 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604348+0000", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604351+0000", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1269999999999999e-06 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604368+0000", 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6393e-05 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604404+0000", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6922000000000003e-05 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604424+0000", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9156000000000001e-05 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604599+0000", 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017526900000000001 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.604616+0000", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6623e-05 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.615395+0000", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010779514 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.615491+0000", 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5748000000000006e-05 2024-03-11T21:48:12.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.615725+0000", 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023391699999999999 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(39.6,QREQUEST,e:813,from:2,reservation_nonce:15)", 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:54.847361+0000", 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 377.36955661299999, 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069463060000000002, 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.252 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847361+0000", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847361+0000", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847363+0000", 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1399999999999998e-06 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847375+0000", 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2e-05 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847408+0000", 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3204999999999998e-05 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847424+0000", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5756999999999998e-05 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847610+0000", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000185725 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.847622+0000", 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1846e-05 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.854111+0000", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064889819999999999 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.854139+0000", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8561999999999999e-05 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:54.854307+0000", 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016808900000000001 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.256 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2325 330.1a 330:58000000:test-rados-api-smithi143-71773-145:::head [pgnls start_epoch 850] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e850)", 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:38.607950+0000", 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.60896796600002, 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070081620000000001, 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2325 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.257 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.607950+0000", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.607950+0000", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.607953+0000", 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1070000000000001e-06 2024-03-11T21:48:12.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.607970+0000", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6833e-05 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.608004+0000", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4869000000000001e-05 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.608022+0000", 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7842e-05 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.608191+0000", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016902599999999999 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.608202+0000", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0447e-05 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.614622+0000", 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064204459999999998 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.614721+0000", 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8890999999999999e-05 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.614958+0000", 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000236701 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2912 332.3 332:c7e8819c:test-rados-api-smithi143-71773-149::bar:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-03-11T21:48:12.261 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:16.654300+0000", 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 295.56261787599999, 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019273306, 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:12.262 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2912 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654300+0000", 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654300+0000", 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654302+0000", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5799999999999999e-06 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654315+0000", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2585999999999999e-05 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654342+0000", 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6865000000000001e-05 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654366+0000", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4570999999999999e-05 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654943+0000", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00057711999999999998 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.654966+0000", 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.262e-05 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.668861+0000", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013894462999999999 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.669112+0000", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0002519 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.669431+0000", 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000318759 2024-03-11T21:48:12.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.671249+0000", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001817733 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.671617+0000", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00036780899999999997 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.673352+0000", 2024-03-11T21:48:12.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001735543 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.673478+0000", 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012603399999999999 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:16.673573+0000", 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4722999999999994e-05 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(5.e,QREQUEST,e:910,from:4,reservation_nonce:21)", 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:24.003150+0000", 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 228.21376809399999, 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061944649999999997, 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003150+0000", 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003150+0000", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003152+0000", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6740000000000001e-06 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003165+0000", 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.274e-05 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003197+0000", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2091999999999997e-05 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003213+0000", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6016999999999999e-05 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003320+0000", 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010732 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.003331+0000", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0891e-05 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.009290+0000", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059587549999999996 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.009304+0000", 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3653e-05 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:24.009344+0000", 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0323e-05 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(6.17,QREQUEST,e:915,from:4,reservation_nonce:23)", 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:28.962803+0000", 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 223.25411485000001, 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065597190000000003, 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.273 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962803+0000", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962803+0000", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962805+0000", 2024-03-11T21:48:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0669999999999999e-06 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962817+0000", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2218999999999999e-05 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962851+0000", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3946000000000003e-05 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.962873+0000", 2024-03-11T21:48:12.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2180000000000001e-05 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.963004+0000", 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130667 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.963015+0000", 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1137e-05 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.969255+0000", 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062396279999999997 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.969288+0000", 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2963000000000003e-05 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:28.969362+0000", 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4912000000000004e-05 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(6.d,QREQUEST,e:919,from:1,reservation_nonce:3)", 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:33.281203+0000", 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 218.93571482999999, 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061098059999999997, 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281203+0000", 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281203+0000", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281205+0000", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.21e-06 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281218+0000", 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3413e-05 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281251+0000", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3009000000000002e-05 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281269+0000", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7604e-05 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281456+0000", 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000186689 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.281468+0000", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.255e-05 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.287114+0000", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.005645855 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.287155+0000", 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0881999999999999e-05 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:33.287313+0000", 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000157594 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(3.17,QREQUEST,e:923,from:0,reservation_nonce:21)", 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:37.397344+0000", 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 214.819573475, 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014982822, 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.282 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397344+0000", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397344+0000", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397346+0000", 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.889e-06 2024-03-11T21:48:12.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397357+0000", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1357e-05 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397386+0000", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8623e-05 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397403+0000", 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6688999999999999e-05 2024-03-11T21:48:12.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397572+0000", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016945 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.397590+0000", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7683e-05 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.411953+0000", 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014363406 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.412003+0000", 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0002e-05 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:37.412327+0000", 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032372299999999999 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.42263.0:3 345.0 e959/958 345:0384ab8e:::oid:head v 959'1, mlcod=0'0)", 2024-03-11T21:48:12.286 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:12.731262+0000", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 179.485655861, 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0084483570000000001, 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731262+0000", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731262+0000", 2024-03-11T21:48:12.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731262+0000", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4000000000000001e-07 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731274+0000", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.19e-05 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731297+0000", 2024-03-11T21:48:12.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2523000000000001e-05 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.731319+0000", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2391999999999998e-05 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.732261+0000", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00094129100000000002 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.732311+0000", 2024-03-11T21:48:12.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0315999999999997e-05 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.739618+0000", 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073066210000000001 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:12.739710+0000", 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2573999999999998e-05 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.290 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(6.1f,QREQUEST,e:968,from:3,reservation_nonce:16)", 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:45.338636+0000", 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 146.87828162, 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064524530000000004, 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338636+0000", 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338636+0000", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338638+0000", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.137e-06 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338650+0000", 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1440000000000001e-05 2024-03-11T21:48:12.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338683+0000", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3132000000000001e-05 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338712+0000", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8895999999999999e-05 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338880+0000", 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016840400000000001 2024-03-11T21:48:12.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.338898+0000", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8033000000000002e-05 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.344987+0000", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060891529999999999 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.345016+0000", 2024-03-11T21:48:12.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8319000000000001e-05 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:45.345088+0000", 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2939000000000001e-05 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.1d,QREQUEST,e:984,from:0,reservation_nonce:11)", 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:46:38.399259+0000", 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 93.817658668999997, 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061531700000000003, 2024-03-11T21:48:12.295 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399259+0000", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399259+0000", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399261+0000", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.503e-06 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399274+0000", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2994e-05 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399304+0000", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9845e-05 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399335+0000", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0889999999999997e-05 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399518+0000", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018261300000000001 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.399529+0000", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1633e-05 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.405311+0000", 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057818039999999998 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.405340+0000", 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8992e-05 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:46:38.405412+0000", 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1896000000000004e-05 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(3.1c,QREQUEST,e:991,from:4,reservation_nonce:7)", 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:47:20.078791+0000", 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 52.138126782999997, 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0063675340000000002, 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078791+0000", 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078791+0000", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078793+0000", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.764e-06 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078803+0000", 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0879e-05 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078832+0000", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8456999999999999e-05 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078849+0000", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6612e-05 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.078997+0000", 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000148047 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.079018+0000", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1543e-05 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.084988+0000", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059698850000000003 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.085035+0000", 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6632999999999999e-05 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:12.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:47:20.085158+0000", 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000123714 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:12.304 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:12.305 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:11 smithi203 bash[23654]: audit 2024-03-11T21:48:11.165018+0000 mon.a (mon.0) 4147 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:48:12.305 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:11 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.185036+0000 osd.4 (osd.4) 585 : cluster 0 2.f scrub starts 2024-03-11T21:48:12.305 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:11 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.187546+0000 osd.4 (osd.4) 586 : cluster 0 2.f scrub ok 2024-03-11T21:48:12.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:48:12.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.535398+0000 osd.3 (osd.3) 613 : cluster 0 3.b scrub starts 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.539138+0000 osd.3 (osd.3) 614 : cluster 0 3.b scrub ok 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.807589+0000 mgr.y (mgr.14550) 965 : cluster 0 pgmap v1634: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21512]: audit 2024-03-11T21:48:11.869517+0000 mon.a (mon.0) 4148 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[21512]: cluster 2024-03-11T21:48:11.877782+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e1002: 8 total, 8 up, 8 in 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.535398+0000 osd.3 (osd.3) 613 : cluster 0 3.b scrub starts 2024-03-11T21:48:13.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.539138+0000 osd.3 (osd.3) 614 : cluster 0 3.b scrub ok 2024-03-11T21:48:13.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.807589+0000 mgr.y (mgr.14550) 965 : cluster 0 pgmap v1634: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T21:48:13.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[29284]: audit 2024-03-11T21:48:11.869517+0000 mon.a (mon.0) 4148 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:13.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:12 smithi143 bash[29284]: cluster 2024-03-11T21:48:11.877782+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e1002: 8 total, 8 up, 8 in 2024-03-11T21:48:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:12 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.535398+0000 osd.3 (osd.3) 613 : cluster 0 3.b scrub starts 2024-03-11T21:48:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:12 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.539138+0000 osd.3 (osd.3) 614 : cluster 0 3.b scrub ok 2024-03-11T21:48:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:12 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.807589+0000 mgr.y (mgr.14550) 965 : cluster 0 pgmap v1634: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T21:48:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:12 smithi203 bash[23654]: audit 2024-03-11T21:48:11.869517+0000 mon.a (mon.0) 4148 : audit 1 from='client.? 172.21.15.143:0/795434172' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:48:13.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:12 smithi203 bash[23654]: cluster 2024-03-11T21:48:11.877782+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e1002: 8 total, 8 up, 8 in 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[21512]: cluster 2024-03-11T21:48:12.694848+0000 osd.6 (osd.6) 491 : cluster 0 1.7 scrub starts 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[21512]: cluster 2024-03-11T21:48:12.697463+0000 osd.6 (osd.6) 492 : cluster 0 1.7 scrub ok 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[21512]: cluster 2024-03-11T21:48:12.889702+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e1003: 8 total, 8 up, 8 in 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[21512]: cluster 2024-03-11T21:48:13.254818+0000 osd.5 (osd.5) 517 : cluster 0 9.f deep-scrub starts 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[21512]: cluster 2024-03-11T21:48:13.261636+0000 osd.5 (osd.5) 518 : cluster 0 9.f deep-scrub ok 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[29284]: cluster 2024-03-11T21:48:12.694848+0000 osd.6 (osd.6) 491 : cluster 0 1.7 scrub starts 2024-03-11T21:48:14.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[29284]: cluster 2024-03-11T21:48:12.697463+0000 osd.6 (osd.6) 492 : cluster 0 1.7 scrub ok 2024-03-11T21:48:14.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[29284]: cluster 2024-03-11T21:48:12.889702+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e1003: 8 total, 8 up, 8 in 2024-03-11T21:48:14.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[29284]: cluster 2024-03-11T21:48:13.254818+0000 osd.5 (osd.5) 517 : cluster 0 9.f deep-scrub starts 2024-03-11T21:48:14.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:13 smithi143 bash[29284]: cluster 2024-03-11T21:48:13.261636+0000 osd.5 (osd.5) 518 : cluster 0 9.f deep-scrub ok 2024-03-11T21:48:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:13 smithi203 bash[23654]: cluster 2024-03-11T21:48:12.694848+0000 osd.6 (osd.6) 491 : cluster 0 1.7 scrub starts 2024-03-11T21:48:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:13 smithi203 bash[23654]: cluster 2024-03-11T21:48:12.697463+0000 osd.6 (osd.6) 492 : cluster 0 1.7 scrub ok 2024-03-11T21:48:14.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:13 smithi203 bash[23654]: cluster 2024-03-11T21:48:12.889702+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e1003: 8 total, 8 up, 8 in 2024-03-11T21:48:14.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:13 smithi203 bash[23654]: cluster 2024-03-11T21:48:13.254818+0000 osd.5 (osd.5) 517 : cluster 0 9.f deep-scrub starts 2024-03-11T21:48:14.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:13 smithi203 bash[23654]: cluster 2024-03-11T21:48:13.261636+0000 osd.5 (osd.5) 518 : cluster 0 9.f deep-scrub ok 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:13.661600+0000 osd.6 (osd.6) 493 : cluster 0 2.7 scrub starts 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:13.665035+0000 osd.6 (osd.6) 494 : cluster 0 2.7 scrub ok 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:13.809443+0000 mgr.y (mgr.14550) 966 : cluster 0 pgmap v1637: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.053647+0000 osd.1 (osd.1) 405 : cluster 0 6.1 scrub starts 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.055996+0000 osd.1 (osd.1) 406 : cluster 0 6.1 scrub ok 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.075142+0000 osd.7 (osd.7) 614 : cluster 0 5.4 scrub starts 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.077399+0000 osd.7 (osd.7) 615 : cluster 0 5.4 scrub ok 2024-03-11T21:48:15.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:13.661600+0000 osd.6 (osd.6) 493 : cluster 0 2.7 scrub starts 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:13.665035+0000 osd.6 (osd.6) 494 : cluster 0 2.7 scrub ok 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:13.809443+0000 mgr.y (mgr.14550) 966 : cluster 0 pgmap v1637: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.053647+0000 osd.1 (osd.1) 405 : cluster 0 6.1 scrub starts 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.055996+0000 osd.1 (osd.1) 406 : cluster 0 6.1 scrub ok 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.075142+0000 osd.7 (osd.7) 614 : cluster 0 5.4 scrub starts 2024-03-11T21:48:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:14 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.077399+0000 osd.7 (osd.7) 615 : cluster 0 5.4 scrub ok 2024-03-11T21:48:15.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:13.661600+0000 osd.6 (osd.6) 493 : cluster 0 2.7 scrub starts 2024-03-11T21:48:15.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:13.665035+0000 osd.6 (osd.6) 494 : cluster 0 2.7 scrub ok 2024-03-11T21:48:15.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:13.809443+0000 mgr.y (mgr.14550) 966 : cluster 0 pgmap v1637: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:15.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.053647+0000 osd.1 (osd.1) 405 : cluster 0 6.1 scrub starts 2024-03-11T21:48:15.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.055996+0000 osd.1 (osd.1) 406 : cluster 0 6.1 scrub ok 2024-03-11T21:48:15.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.075142+0000 osd.7 (osd.7) 614 : cluster 0 5.4 scrub starts 2024-03-11T21:48:15.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:14 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.077399+0000 osd.7 (osd.7) 615 : cluster 0 5.4 scrub ok 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.561670+0000 osd.3 (osd.3) 615 : cluster 0 3.8 scrub starts 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:14.566035+0000 osd.3 (osd.3) 616 : cluster 0 3.8 scrub ok 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.065993+0000 osd.1 (osd.1) 407 : cluster 0 2.15 scrub starts 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.068467+0000 osd.1 (osd.1) 408 : cluster 0 2.15 scrub ok 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.324496+0000 osd.5 (osd.5) 519 : cluster 0 2.12 scrub starts 2024-03-11T21:48:16.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.326829+0000 osd.5 (osd.5) 520 : cluster 0 2.12 scrub ok 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.561670+0000 osd.3 (osd.3) 615 : cluster 0 3.8 scrub starts 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:14.566035+0000 osd.3 (osd.3) 616 : cluster 0 3.8 scrub ok 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.065993+0000 osd.1 (osd.1) 407 : cluster 0 2.15 scrub starts 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.068467+0000 osd.1 (osd.1) 408 : cluster 0 2.15 scrub ok 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.324496+0000 osd.5 (osd.5) 519 : cluster 0 2.12 scrub starts 2024-03-11T21:48:16.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:15 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.326829+0000 osd.5 (osd.5) 520 : cluster 0 2.12 scrub ok 2024-03-11T21:48:16.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.561670+0000 osd.3 (osd.3) 615 : cluster 0 3.8 scrub starts 2024-03-11T21:48:16.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:14.566035+0000 osd.3 (osd.3) 616 : cluster 0 3.8 scrub ok 2024-03-11T21:48:16.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.065993+0000 osd.1 (osd.1) 407 : cluster 0 2.15 scrub starts 2024-03-11T21:48:16.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.068467+0000 osd.1 (osd.1) 408 : cluster 0 2.15 scrub ok 2024-03-11T21:48:16.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.324496+0000 osd.5 (osd.5) 519 : cluster 0 2.12 scrub starts 2024-03-11T21:48:16.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:15 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.326829+0000 osd.5 (osd.5) 520 : cluster 0 2.12 scrub ok 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.545538+0000 osd.3 (osd.3) 617 : cluster 0 9.c scrub starts 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.548004+0000 osd.3 (osd.3) 618 : cluster 0 9.c scrub ok 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.572824+0000 osd.2 (osd.2) 507 : cluster 0 2.c scrub starts 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.574347+0000 osd.2 (osd.2) 508 : cluster 0 2.c scrub ok 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.604354+0000 osd.6 (osd.6) 495 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.606614+0000 osd.6 (osd.6) 496 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:15.810437+0000 mgr.y (mgr.14550) 967 : cluster 0 pgmap v1638: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:17.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.023407+0000 osd.1 (osd.1) 409 : cluster 0 2.4 scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.025875+0000 osd.1 (osd.1) 410 : cluster 0 2.4 scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.059304+0000 osd.7 (osd.7) 616 : cluster 0 5.10 scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.061737+0000 osd.7 (osd.7) 617 : cluster 0 5.10 scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.545538+0000 osd.3 (osd.3) 617 : cluster 0 9.c scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.548004+0000 osd.3 (osd.3) 618 : cluster 0 9.c scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.572824+0000 osd.2 (osd.2) 507 : cluster 0 2.c scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.574347+0000 osd.2 (osd.2) 508 : cluster 0 2.c scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.604354+0000 osd.6 (osd.6) 495 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.606614+0000 osd.6 (osd.6) 496 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:15.810437+0000 mgr.y (mgr.14550) 967 : cluster 0 pgmap v1638: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.023407+0000 osd.1 (osd.1) 409 : cluster 0 2.4 scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.025875+0000 osd.1 (osd.1) 410 : cluster 0 2.4 scrub ok 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.059304+0000 osd.7 (osd.7) 616 : cluster 0 5.10 scrub starts 2024-03-11T21:48:17.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:16 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.061737+0000 osd.7 (osd.7) 617 : cluster 0 5.10 scrub ok 2024-03-11T21:48:17.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.545538+0000 osd.3 (osd.3) 617 : cluster 0 9.c scrub starts 2024-03-11T21:48:17.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.548004+0000 osd.3 (osd.3) 618 : cluster 0 9.c scrub ok 2024-03-11T21:48:17.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.572824+0000 osd.2 (osd.2) 507 : cluster 0 2.c scrub starts 2024-03-11T21:48:17.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.574347+0000 osd.2 (osd.2) 508 : cluster 0 2.c scrub ok 2024-03-11T21:48:17.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.604354+0000 osd.6 (osd.6) 495 : cluster 0 2.17 deep-scrub starts 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.606614+0000 osd.6 (osd.6) 496 : cluster 0 2.17 deep-scrub ok 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:15.810437+0000 mgr.y (mgr.14550) 967 : cluster 0 pgmap v1638: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.023407+0000 osd.1 (osd.1) 409 : cluster 0 2.4 scrub starts 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.025875+0000 osd.1 (osd.1) 410 : cluster 0 2.4 scrub ok 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.059304+0000 osd.7 (osd.7) 616 : cluster 0 5.10 scrub starts 2024-03-11T21:48:17.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:16 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.061737+0000 osd.7 (osd.7) 617 : cluster 0 5.10 scrub ok 2024-03-11T21:48:17.528 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:17.777 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.893795+0000 osd.0 (osd.0) 616 : cluster 0 5.5 scrub starts 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:16.896124+0000 osd.0 (osd.0) 617 : cluster 0 5.5 scrub ok 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.063347+0000 osd.7 (osd.7) 618 : cluster 0 9.3 scrub starts 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.067893+0000 osd.7 (osd.7) 619 : cluster 0 9.3 scrub ok 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.350842+0000 osd.5 (osd.5) 521 : cluster 0 3.c scrub starts 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.354047+0000 osd.5 (osd.5) 522 : cluster 0 3.c scrub ok 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.893795+0000 osd.0 (osd.0) 616 : cluster 0 5.5 scrub starts 2024-03-11T21:48:18.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:16.896124+0000 osd.0 (osd.0) 617 : cluster 0 5.5 scrub ok 2024-03-11T21:48:18.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.063347+0000 osd.7 (osd.7) 618 : cluster 0 9.3 scrub starts 2024-03-11T21:48:18.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.067893+0000 osd.7 (osd.7) 619 : cluster 0 9.3 scrub ok 2024-03-11T21:48:18.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.350842+0000 osd.5 (osd.5) 521 : cluster 0 3.c scrub starts 2024-03-11T21:48:18.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:17 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.354047+0000 osd.5 (osd.5) 522 : cluster 0 3.c scrub ok 2024-03-11T21:48:18.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.893795+0000 osd.0 (osd.0) 616 : cluster 0 5.5 scrub starts 2024-03-11T21:48:18.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:16.896124+0000 osd.0 (osd.0) 617 : cluster 0 5.5 scrub ok 2024-03-11T21:48:18.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.063347+0000 osd.7 (osd.7) 618 : cluster 0 9.3 scrub starts 2024-03-11T21:48:18.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.067893+0000 osd.7 (osd.7) 619 : cluster 0 9.3 scrub ok 2024-03-11T21:48:18.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.350842+0000 osd.5 (osd.5) 521 : cluster 0 3.c scrub starts 2024-03-11T21:48:18.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:17 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.354047+0000 osd.5 (osd.5) 522 : cluster 0 3.c scrub ok 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.811184+0000 mgr.y (mgr.14550) 968 : cluster 0 pgmap v1639: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.869358+0000 osd.0 (osd.0) 618 : cluster 0 5.3 scrub starts 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:17.872500+0000 osd.0 (osd.0) 619 : cluster 0 5.3 scrub ok 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.043728+0000 osd.1 (osd.1) 411 : cluster 0 5.12 scrub starts 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.045420+0000 osd.1 (osd.1) 412 : cluster 0 5.12 scrub ok 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.258787+0000 osd.4 (osd.4) 587 : cluster 0 3.14 scrub starts 2024-03-11T21:48:19.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.262996+0000 osd.4 (osd.4) 588 : cluster 0 3.14 scrub ok 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.811184+0000 mgr.y (mgr.14550) 968 : cluster 0 pgmap v1639: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.869358+0000 osd.0 (osd.0) 618 : cluster 0 5.3 scrub starts 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:17.872500+0000 osd.0 (osd.0) 619 : cluster 0 5.3 scrub ok 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.043728+0000 osd.1 (osd.1) 411 : cluster 0 5.12 scrub starts 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.045420+0000 osd.1 (osd.1) 412 : cluster 0 5.12 scrub ok 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.258787+0000 osd.4 (osd.4) 587 : cluster 0 3.14 scrub starts 2024-03-11T21:48:19.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:18 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.262996+0000 osd.4 (osd.4) 588 : cluster 0 3.14 scrub ok 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.811184+0000 mgr.y (mgr.14550) 968 : cluster 0 pgmap v1639: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.869358+0000 osd.0 (osd.0) 618 : cluster 0 5.3 scrub starts 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:17.872500+0000 osd.0 (osd.0) 619 : cluster 0 5.3 scrub ok 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.043728+0000 osd.1 (osd.1) 411 : cluster 0 5.12 scrub starts 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.045420+0000 osd.1 (osd.1) 412 : cluster 0 5.12 scrub ok 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.258787+0000 osd.4 (osd.4) 587 : cluster 0 3.14 scrub starts 2024-03-11T21:48:19.220 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:18 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.262996+0000 osd.4 (osd.4) 588 : cluster 0 3.14 scrub ok 2024-03-11T21:48:19.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.911721+0000 osd.0 (osd.0) 620 : cluster 0 6.16 scrub starts 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:18.914142+0000 osd.0 (osd.0) 621 : cluster 0 6.16 scrub ok 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.015182+0000 osd.1 (osd.1) 413 : cluster 0 5.19 scrub starts 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.016472+0000 osd.1 (osd.1) 414 : cluster 0 5.19 scrub ok 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.084449+0000 osd.7 (osd.7) 620 : cluster 0 1.d scrub starts 2024-03-11T21:48:20.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:19 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.085661+0000 osd.7 (osd.7) 621 : cluster 0 1.d scrub ok 2024-03-11T21:48:20.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.911721+0000 osd.0 (osd.0) 620 : cluster 0 6.16 scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:18.914142+0000 osd.0 (osd.0) 621 : cluster 0 6.16 scrub ok 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.015182+0000 osd.1 (osd.1) 413 : cluster 0 5.19 scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.016472+0000 osd.1 (osd.1) 414 : cluster 0 5.19 scrub ok 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.084449+0000 osd.7 (osd.7) 620 : cluster 0 1.d scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.085661+0000 osd.7 (osd.7) 621 : cluster 0 1.d scrub ok 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.911721+0000 osd.0 (osd.0) 620 : cluster 0 6.16 scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:18.914142+0000 osd.0 (osd.0) 621 : cluster 0 6.16 scrub ok 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.015182+0000 osd.1 (osd.1) 413 : cluster 0 5.19 scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.016472+0000 osd.1 (osd.1) 414 : cluster 0 5.19 scrub ok 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.084449+0000 osd.7 (osd.7) 620 : cluster 0 1.d scrub starts 2024-03-11T21:48:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:19 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.085661+0000 osd.7 (osd.7) 621 : cluster 0 1.d scrub ok 2024-03-11T21:48:20.888 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:20.889 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:48:20.889 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:48:20.889 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:20 smithi203 bash[23654]: audit 2024-03-11T21:48:19.358440+0000 mgr.y (mgr.14550) 969 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:20 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.813050+0000 mgr.y (mgr.14550) 970 : cluster 0 pgmap v1640: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:20 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.935135+0000 osd.0 (osd.0) 622 : cluster 0 9.5 deep-scrub starts 2024-03-11T21:48:21.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:20 smithi203 bash[23654]: cluster 2024-03-11T21:48:19.937238+0000 osd.0 (osd.0) 623 : cluster 0 9.5 deep-scrub ok 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.147+0000 7fe4d2586700 1 -- 172.21.15.143:0/3559631170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc101930 msgr2=0x7fe4cc105970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.147+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3559631170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc101930 0x7fe4cc105970 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009920 tx=0x7fe4bc009c30 comp rx=0 tx=0).stop 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3559631170 shutdown_connections 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3559631170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc106050 0x7fe4cc10dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3559631170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc100f80 0x7fe4cc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3559631170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc101930 0x7fe4cc105970 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.139 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3559631170 >> 172.21.15.143:0/3559631170 conn(0x7fe4cc077d30 msgr2=0x7fe4cc078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3559631170 shutdown_connections 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3559631170 wait complete. 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 librados: starting msgr at 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 librados: starting objecter 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 Processor -- start 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- start start 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 librados: setting wanted keys 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 librados: calling monclient init 2024-03-11T21:48:21.141 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc101930 0x7fe4cc19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1105b0 con 0x7fe4cc106050 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc110430 con 0x7fe4cc101930 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc110730 con 0x7fe4cc100f80 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4cbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4cbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:56482/0 (socket says 172.21.15.143:56482) 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4cbfff700 1 -- 172.21.15.143:0/3847591806 learned_addr learned my addr 172.21.15.143:0/3847591806 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39884/0 (socket says 172.21.15.143:39884) 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4cb7fe700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc101930 0x7fe4cc19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 -- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 msgr2=0x7fe4cc19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:21.142 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 -- 172.21.15.143:0/3847591806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc101930 msgr2=0x7fe4cc19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc101930 0x7fe4cc19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 -- 172.21.15.143:0/3847591806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc1981a0 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4cbfff700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d0b23700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009cd0 tx=0x7fe4c400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 10 client.?.objecter ms_handle_connect 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400cdc0 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fe4c4003a00 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc198430 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40044f0 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 10 client.31191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc1a4750 con 0x7fe4cc106050 2024-03-11T21:48:21.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 librados: init done 2024-03-11T21:48:21.144 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc04faa0 con 0x7fe4cc106050 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.151+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe4c401a020 con 0x7fe4cc106050 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 0x7fe4b8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1003..1003 src has 944..1003) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fe4c408e030 con 0x7fe4cc106050 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 10 client.31191.objecter ms_dispatch 0x7fe4cc001550 osd_map(1003..1003 src has 944..1003) v4 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 3 client.31191.objecter handle_osd_map got epochs [1003,1003] > 0 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 3 client.31191.objecter handle_osd_map decoding full epoch 1003 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4cbfff700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 0x7fe4b8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 20 client.31191.objecter dump_active .. 0 homeless 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4c97fa700 10 client.31191.objecter ms_handle_connect 0x7fe4b806f820 2024-03-11T21:48:21.145 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.155+0000 7fe4cbfff700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 0x7fe4b8071ce0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc198b80 tx=0x7fe4b4005f90 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:21.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.159+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe4c4010040 con 0x7fe4cc106050 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[21512]: audit 2024-03-11T21:48:19.358440+0000 mgr.y (mgr.14550) 969 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.813050+0000 mgr.y (mgr.14550) 970 : cluster 0 pgmap v1640: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.935135+0000 osd.0 (osd.0) 622 : cluster 0 9.5 deep-scrub starts 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[21512]: cluster 2024-03-11T21:48:19.937238+0000 osd.0 (osd.0) 623 : cluster 0 9.5 deep-scrub ok 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[29284]: audit 2024-03-11T21:48:19.358440+0000 mgr.y (mgr.14550) 969 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.813050+0000 mgr.y (mgr.14550) 970 : cluster 0 pgmap v1640: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:48:21.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.935135+0000 osd.0 (osd.0) 622 : cluster 0 9.5 deep-scrub starts 2024-03-11T21:48:21.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:20 smithi143 bash[29284]: cluster 2024-03-11T21:48:19.937238+0000 osd.0 (osd.0) 623 : cluster 0 9.5 deep-scrub ok 2024-03-11T21:48:21.371 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.383+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7fe4cc02d830 con 0x7fe4cc106050 2024-03-11T21:48:21.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:48:21.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.939+0000 7fe4c97fa700 1 -- 172.21.15.143:0/3847591806 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v1004) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe4c405b040 con 0x7fe4cc106050 2024-03-11T21:48:21.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 10 librados: watch_flush enter 2024-03-11T21:48:21.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 10 librados: watch_flush exit 2024-03-11T21:48:21.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 20 client.31191.objecter shutdown clearing up homeless session... 2024-03-11T21:48:21.931 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 10 client.31191.objecter successfully canceled tick 2024-03-11T21:48:21.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 msgr2=0x7fe4b8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:21.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 0x7fe4b8071ce0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc198b80 tx=0x7fe4b4005f90 comp rx=0 tx=0).stop 2024-03-11T21:48:21.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 msgr2=0x7fe4cc197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:21.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009cd0 tx=0x7fe4c400c620 comp rx=0 tx=0).stop 2024-03-11T21:48:21.932 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 shutdown_connections 2024-03-11T21:48:21.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fe4b806f820 0x7fe4b8071ce0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4cc106050 0x7fe4cc197a40 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4cc100f80 0x7fe4cc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 --2- 172.21.15.143:0/3847591806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4cc101930 0x7fe4cc19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:21.933 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.943+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 >> 172.21.15.143:0/3847591806 conn(0x7fe4cc077d30 msgr2=0x7fe4cc06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.947+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 shutdown_connections 2024-03-11T21:48:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.947+0000 7fe4d2586700 1 -- 172.21.15.143:0/3847591806 wait complete. 2024-03-11T21:48:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:21.947+0000 7fe4d2586700 1 librados: shutdown 2024-03-11T21:48:21.938 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:48:22.184 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:20.497671+0000 osd.3 (osd.3) 619 : cluster 0 3.1e scrub starts 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:20.502200+0000 osd.3 (osd.3) 620 : cluster 0 3.1e scrub ok 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:20.543922+0000 osd.2 (osd.2) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:20.546483+0000 osd.2 (osd.2) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.140432+0000 osd.7 (osd.7) 622 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.143807+0000 osd.7 (osd.7) 623 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:48:22.185 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[21512]: audit 2024-03-11T21:48:21.384825+0000 mon.a (mon.0) 4151 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:20.497671+0000 osd.3 (osd.3) 619 : cluster 0 3.1e scrub starts 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:20.502200+0000 osd.3 (osd.3) 620 : cluster 0 3.1e scrub ok 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:20.543922+0000 osd.2 (osd.2) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:20.546483+0000 osd.2 (osd.2) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.140432+0000 osd.7 (osd.7) 622 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.143807+0000 osd.7 (osd.7) 623 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:48:22.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:21 smithi143 bash[29284]: audit 2024-03-11T21:48:21.384825+0000 mon.a (mon.0) 4151 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:20.497671+0000 osd.3 (osd.3) 619 : cluster 0 3.1e scrub starts 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:20.502200+0000 osd.3 (osd.3) 620 : cluster 0 3.1e scrub ok 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:20.543922+0000 osd.2 (osd.2) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:20.546483+0000 osd.2 (osd.2) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.140432+0000 osd.7 (osd.7) 622 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.143807+0000 osd.7 (osd.7) 623 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:48:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:21 smithi203 bash[23654]: audit 2024-03-11T21:48:21.384825+0000 mon.a (mon.0) 4151 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:22.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-11T21:48:23.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.161233+0000 osd.3 (osd.3) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:23.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.161449+0000 osd.3 (osd.3) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:23.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.813985+0000 mgr.y (mgr.14550) 971 : cluster 0 pgmap v1641: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:48:23.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: audit 2024-03-11T21:48:21.939928+0000 mon.a (mon.0) 4152 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:21.948470+0000 mon.a (mon.0) 4153 : cluster 0 osdmap e1004: 8 total, 8 up, 8 in 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.174823+0000 osd.7 (osd.7) 624 : cluster 0 9.15 deep-scrub starts 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.179779+0000 osd.7 (osd.7) 625 : cluster 0 9.15 deep-scrub ok 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.294859+0000 osd.4 (osd.4) 589 : cluster 0 3.1a scrub starts 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.298720+0000 osd.4 (osd.4) 590 : cluster 0 3.1a scrub ok 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.466698+0000 osd.5 (osd.5) 523 : cluster 0 9.a scrub starts 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[29284]: cluster 2024-03-11T21:48:22.468819+0000 osd.5 (osd.5) 524 : cluster 0 9.a scrub ok 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.161233+0000 osd.3 (osd.3) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.161449+0000 osd.3 (osd.3) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.813985+0000 mgr.y (mgr.14550) 971 : cluster 0 pgmap v1641: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: audit 2024-03-11T21:48:21.939928+0000 mon.a (mon.0) 4152 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:21.948470+0000 mon.a (mon.0) 4153 : cluster 0 osdmap e1004: 8 total, 8 up, 8 in 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.174823+0000 osd.7 (osd.7) 624 : cluster 0 9.15 deep-scrub starts 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.179779+0000 osd.7 (osd.7) 625 : cluster 0 9.15 deep-scrub ok 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.294859+0000 osd.4 (osd.4) 589 : cluster 0 3.1a scrub starts 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.298720+0000 osd.4 (osd.4) 590 : cluster 0 3.1a scrub ok 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.466698+0000 osd.5 (osd.5) 523 : cluster 0 9.a scrub starts 2024-03-11T21:48:23.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:22 smithi143 bash[21512]: cluster 2024-03-11T21:48:22.468819+0000 osd.5 (osd.5) 524 : cluster 0 9.a scrub ok 2024-03-11T21:48:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.161233+0000 osd.3 (osd.3) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.161449+0000 osd.3 (osd.3) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.813985+0000 mgr.y (mgr.14550) 971 : cluster 0 pgmap v1641: 184 pgs: 184 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: audit 2024-03-11T21:48:21.939928+0000 mon.a (mon.0) 4152 : audit 1 from='client.? 172.21.15.143:0/3847591806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:21.948470+0000 mon.a (mon.0) 4153 : cluster 0 osdmap e1004: 8 total, 8 up, 8 in 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.174823+0000 osd.7 (osd.7) 624 : cluster 0 9.15 deep-scrub starts 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.179779+0000 osd.7 (osd.7) 625 : cluster 0 9.15 deep-scrub ok 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.294859+0000 osd.4 (osd.4) 589 : cluster 0 3.1a scrub starts 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.298720+0000 osd.4 (osd.4) 590 : cluster 0 3.1a scrub ok 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.466698+0000 osd.5 (osd.5) 523 : cluster 0 9.a scrub starts 2024-03-11T21:48:23.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:22 smithi203 bash[23654]: cluster 2024-03-11T21:48:22.468819+0000 osd.5 (osd.5) 524 : cluster 0 9.a scrub ok 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[21512]: audit 2024-03-11T21:48:23.140002+0000 mon.c (mon.2) 1414 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.142006+0000 osd.7 (osd.7) 626 : cluster 0 6.3 scrub starts 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.145148+0000 osd.7 (osd.7) 627 : cluster 0 6.3 scrub ok 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.180686+0000 osd.1 (osd.1) 415 : cluster 0 2.d scrub starts 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.183049+0000 osd.1 (osd.1) 416 : cluster 0 2.d scrub ok 2024-03-11T21:48:24.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[29284]: audit 2024-03-11T21:48:23.140002+0000 mon.c (mon.2) 1414 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.142006+0000 osd.7 (osd.7) 626 : cluster 0 6.3 scrub starts 2024-03-11T21:48:24.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.145148+0000 osd.7 (osd.7) 627 : cluster 0 6.3 scrub ok 2024-03-11T21:48:24.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.180686+0000 osd.1 (osd.1) 415 : cluster 0 2.d scrub starts 2024-03-11T21:48:24.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:23 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.183049+0000 osd.1 (osd.1) 416 : cluster 0 2.d scrub ok 2024-03-11T21:48:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:23 smithi203 bash[23654]: audit 2024-03-11T21:48:23.140002+0000 mon.c (mon.2) 1414 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:23 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.142006+0000 osd.7 (osd.7) 626 : cluster 0 6.3 scrub starts 2024-03-11T21:48:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:23 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.145148+0000 osd.7 (osd.7) 627 : cluster 0 6.3 scrub ok 2024-03-11T21:48:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:23 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.180686+0000 osd.1 (osd.1) 415 : cluster 0 2.d scrub starts 2024-03-11T21:48:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:23 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.183049+0000 osd.1 (osd.1) 416 : cluster 0 2.d scrub ok 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.815808+0000 mgr.y (mgr.14550) 972 : cluster 0 pgmap v1643: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.974213+0000 osd.0 (osd.0) 624 : cluster 0 3.12 scrub starts 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[21512]: cluster 2024-03-11T21:48:23.978648+0000 osd.0 (osd.0) 625 : cluster 0 3.12 scrub ok 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[21512]: cluster 2024-03-11T21:48:24.145032+0000 osd.1 (osd.1) 417 : cluster 0 3.4 scrub starts 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[21512]: cluster 2024-03-11T21:48:24.150724+0000 osd.1 (osd.1) 418 : cluster 0 3.4 scrub ok 2024-03-11T21:48:25.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.815808+0000 mgr.y (mgr.14550) 972 : cluster 0 pgmap v1643: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.974213+0000 osd.0 (osd.0) 624 : cluster 0 3.12 scrub starts 2024-03-11T21:48:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[29284]: cluster 2024-03-11T21:48:23.978648+0000 osd.0 (osd.0) 625 : cluster 0 3.12 scrub ok 2024-03-11T21:48:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[29284]: cluster 2024-03-11T21:48:24.145032+0000 osd.1 (osd.1) 417 : cluster 0 3.4 scrub starts 2024-03-11T21:48:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:24 smithi143 bash[29284]: cluster 2024-03-11T21:48:24.150724+0000 osd.1 (osd.1) 418 : cluster 0 3.4 scrub ok 2024-03-11T21:48:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:24 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.815808+0000 mgr.y (mgr.14550) 972 : cluster 0 pgmap v1643: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:24 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.974213+0000 osd.0 (osd.0) 624 : cluster 0 3.12 scrub starts 2024-03-11T21:48:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:24 smithi203 bash[23654]: cluster 2024-03-11T21:48:23.978648+0000 osd.0 (osd.0) 625 : cluster 0 3.12 scrub ok 2024-03-11T21:48:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:24 smithi203 bash[23654]: cluster 2024-03-11T21:48:24.145032+0000 osd.1 (osd.1) 417 : cluster 0 3.4 scrub starts 2024-03-11T21:48:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:24 smithi203 bash[23654]: cluster 2024-03-11T21:48:24.150724+0000 osd.1 (osd.1) 418 : cluster 0 3.4 scrub ok 2024-03-11T21:48:26.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:25 smithi203 bash[23654]: cluster 2024-03-11T21:48:25.100423+0000 osd.1 (osd.1) 419 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:48:26.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:25 smithi203 bash[23654]: cluster 2024-03-11T21:48:25.102944+0000 osd.1 (osd.1) 420 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:48:26.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:25 smithi203 bash[23654]: cluster 2024-03-11T21:48:25.387408+0000 osd.5 (osd.5) 525 : cluster 0 2.16 scrub starts 2024-03-11T21:48:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:25 smithi203 bash[23654]: cluster 2024-03-11T21:48:25.389709+0000 osd.5 (osd.5) 526 : cluster 0 2.16 scrub ok 2024-03-11T21:48:26.402 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:26.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[21512]: cluster 2024-03-11T21:48:25.100423+0000 osd.1 (osd.1) 419 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:48:26.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[21512]: cluster 2024-03-11T21:48:25.102944+0000 osd.1 (osd.1) 420 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:48:26.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[21512]: cluster 2024-03-11T21:48:25.387408+0000 osd.5 (osd.5) 525 : cluster 0 2.16 scrub starts 2024-03-11T21:48:26.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[21512]: cluster 2024-03-11T21:48:25.389709+0000 osd.5 (osd.5) 526 : cluster 0 2.16 scrub ok 2024-03-11T21:48:26.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[29284]: cluster 2024-03-11T21:48:25.100423+0000 osd.1 (osd.1) 419 : cluster 0 5.1d deep-scrub starts 2024-03-11T21:48:26.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[29284]: cluster 2024-03-11T21:48:25.102944+0000 osd.1 (osd.1) 420 : cluster 0 5.1d deep-scrub ok 2024-03-11T21:48:26.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[29284]: cluster 2024-03-11T21:48:25.387408+0000 osd.5 (osd.5) 525 : cluster 0 2.16 scrub starts 2024-03-11T21:48:26.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:25 smithi143 bash[29284]: cluster 2024-03-11T21:48:25.389709+0000 osd.5 (osd.5) 526 : cluster 0 2.16 scrub ok 2024-03-11T21:48:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:26 smithi203 bash[23654]: cluster 2024-03-11T21:48:25.816802+0000 mgr.y (mgr.14550) 973 : cluster 0 pgmap v1644: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:26 smithi203 bash[23654]: cluster 2024-03-11T21:48:26.174438+0000 osd.7 (osd.7) 628 : cluster 0 6.5 scrub starts 2024-03-11T21:48:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:26 smithi203 bash[23654]: cluster 2024-03-11T21:48:26.176909+0000 osd.7 (osd.7) 629 : cluster 0 6.5 scrub ok 2024-03-11T21:48:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:26 smithi203 bash[23654]: cluster 2024-03-11T21:48:26.416209+0000 osd.5 (osd.5) 527 : cluster 0 1.10 scrub starts 2024-03-11T21:48:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:26 smithi203 bash[23654]: cluster 2024-03-11T21:48:26.418472+0000 osd.5 (osd.5) 528 : cluster 0 1.10 scrub ok 2024-03-11T21:48:27.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[21512]: cluster 2024-03-11T21:48:25.816802+0000 mgr.y (mgr.14550) 973 : cluster 0 pgmap v1644: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:27.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[21512]: cluster 2024-03-11T21:48:26.174438+0000 osd.7 (osd.7) 628 : cluster 0 6.5 scrub starts 2024-03-11T21:48:27.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[21512]: cluster 2024-03-11T21:48:26.176909+0000 osd.7 (osd.7) 629 : cluster 0 6.5 scrub ok 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[21512]: cluster 2024-03-11T21:48:26.416209+0000 osd.5 (osd.5) 527 : cluster 0 1.10 scrub starts 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[21512]: cluster 2024-03-11T21:48:26.418472+0000 osd.5 (osd.5) 528 : cluster 0 1.10 scrub ok 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[29284]: cluster 2024-03-11T21:48:25.816802+0000 mgr.y (mgr.14550) 973 : cluster 0 pgmap v1644: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[29284]: cluster 2024-03-11T21:48:26.174438+0000 osd.7 (osd.7) 628 : cluster 0 6.5 scrub starts 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[29284]: cluster 2024-03-11T21:48:26.176909+0000 osd.7 (osd.7) 629 : cluster 0 6.5 scrub ok 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[29284]: cluster 2024-03-11T21:48:26.416209+0000 osd.5 (osd.5) 527 : cluster 0 1.10 scrub starts 2024-03-11T21:48:27.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:26 smithi143 bash[29284]: cluster 2024-03-11T21:48:26.418472+0000 osd.5 (osd.5) 528 : cluster 0 1.10 scrub ok 2024-03-11T21:48:27.589 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:28.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:27 smithi203 bash[23654]: cluster 2024-03-11T21:48:27.126408+0000 osd.1 (osd.1) 421 : cluster 0 3.0 scrub starts 2024-03-11T21:48:28.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:27 smithi203 bash[23654]: cluster 2024-03-11T21:48:27.130349+0000 osd.1 (osd.1) 422 : cluster 0 3.0 scrub ok 2024-03-11T21:48:28.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:27 smithi143 bash[29284]: cluster 2024-03-11T21:48:27.126408+0000 osd.1 (osd.1) 421 : cluster 0 3.0 scrub starts 2024-03-11T21:48:28.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:27 smithi143 bash[29284]: cluster 2024-03-11T21:48:27.130349+0000 osd.1 (osd.1) 422 : cluster 0 3.0 scrub ok 2024-03-11T21:48:28.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:27 smithi143 bash[21512]: cluster 2024-03-11T21:48:27.126408+0000 osd.1 (osd.1) 421 : cluster 0 3.0 scrub starts 2024-03-11T21:48:28.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:27 smithi143 bash[21512]: cluster 2024-03-11T21:48:27.130349+0000 osd.1 (osd.1) 422 : cluster 0 3.0 scrub ok 2024-03-11T21:48:29.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[29284]: cluster 2024-03-11T21:48:27.817598+0000 mgr.y (mgr.14550) 974 : cluster 0 pgmap v1645: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:29.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[29284]: cluster 2024-03-11T21:48:28.100538+0000 osd.7 (osd.7) 630 : cluster 0 3.13 scrub starts 2024-03-11T21:48:29.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[29284]: cluster 2024-03-11T21:48:28.104651+0000 osd.7 (osd.7) 631 : cluster 0 3.13 scrub ok 2024-03-11T21:48:29.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[21512]: cluster 2024-03-11T21:48:27.817598+0000 mgr.y (mgr.14550) 974 : cluster 0 pgmap v1645: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:29.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[21512]: cluster 2024-03-11T21:48:28.100538+0000 osd.7 (osd.7) 630 : cluster 0 3.13 scrub starts 2024-03-11T21:48:29.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:28 smithi143 bash[21512]: cluster 2024-03-11T21:48:28.104651+0000 osd.7 (osd.7) 631 : cluster 0 3.13 scrub ok 2024-03-11T21:48:29.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:28 smithi203 bash[23654]: cluster 2024-03-11T21:48:27.817598+0000 mgr.y (mgr.14550) 974 : cluster 0 pgmap v1645: 184 pgs: 184 active+clean; 582 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:29.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:28 smithi203 bash[23654]: cluster 2024-03-11T21:48:28.100538+0000 osd.7 (osd.7) 630 : cluster 0 3.13 scrub starts 2024-03-11T21:48:29.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:28 smithi203 bash[23654]: cluster 2024-03-11T21:48:28.104651+0000 osd.7 (osd.7) 631 : cluster 0 3.13 scrub ok 2024-03-11T21:48:29.338 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:29.338 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:48:29.338 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:48:29.338 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:48:29.338 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:29.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:30.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:48:30.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:28.512188+0000 osd.3 (osd.3) 623 : cluster 0 1.c deep-scrub starts 2024-03-11T21:48:30.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:28.516552+0000 osd.3 (osd.3) 624 : cluster 0 1.c deep-scrub ok 2024-03-11T21:48:30.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.056257+0000 osd.7 (osd.7) 632 : cluster 0 2.8 scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.058440+0000 osd.7 (osd.7) 633 : cluster 0 2.8 scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.204139+0000 osd.4 (osd.4) 591 : cluster 0 6.e scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.205878+0000 osd.4 (osd.4) 592 : cluster 0 6.e scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.364595+0000 osd.5 (osd.5) 529 : cluster 0 5.16 scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.366415+0000 osd.5 (osd.5) 530 : cluster 0 5.16 scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:28.512188+0000 osd.3 (osd.3) 623 : cluster 0 1.c deep-scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:28.516552+0000 osd.3 (osd.3) 624 : cluster 0 1.c deep-scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.056257+0000 osd.7 (osd.7) 632 : cluster 0 2.8 scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.058440+0000 osd.7 (osd.7) 633 : cluster 0 2.8 scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.204139+0000 osd.4 (osd.4) 591 : cluster 0 6.e scrub starts 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.205878+0000 osd.4 (osd.4) 592 : cluster 0 6.e scrub ok 2024-03-11T21:48:30.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.364595+0000 osd.5 (osd.5) 529 : cluster 0 5.16 scrub starts 2024-03-11T21:48:30.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:30 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.366415+0000 osd.5 (osd.5) 530 : cluster 0 5.16 scrub ok 2024-03-11T21:48:30.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:28.512188+0000 osd.3 (osd.3) 623 : cluster 0 1.c deep-scrub starts 2024-03-11T21:48:30.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:28.516552+0000 osd.3 (osd.3) 624 : cluster 0 1.c deep-scrub ok 2024-03-11T21:48:30.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.056257+0000 osd.7 (osd.7) 632 : cluster 0 2.8 scrub starts 2024-03-11T21:48:30.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.058440+0000 osd.7 (osd.7) 633 : cluster 0 2.8 scrub ok 2024-03-11T21:48:30.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.204139+0000 osd.4 (osd.4) 591 : cluster 0 6.e scrub starts 2024-03-11T21:48:30.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.205878+0000 osd.4 (osd.4) 592 : cluster 0 6.e scrub ok 2024-03-11T21:48:30.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.364595+0000 osd.5 (osd.5) 529 : cluster 0 5.16 scrub starts 2024-03-11T21:48:30.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:30 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.366415+0000 osd.5 (osd.5) 530 : cluster 0 5.16 scrub ok 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/3503604431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 msgr2=0x7f0028114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- 172.21.15.143:0/3503604431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f0028114ef0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f002000b0a0 tx=0x7f002000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/3503604431 shutdown_connections 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- 172.21.15.143:0/3503604431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f0028114ef0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- 172.21.15.143:0/3503604431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 0x7f0028074010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- 172.21.15.143:0/3503604431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f00280749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/3503604431 >> 172.21.15.143:0/3503604431 conn(0x7f0028078eb0 msgr2=0x7f00280792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/3503604431 shutdown_connections 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/3503604431 wait complete. 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 librados: starting msgr at 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 librados: starting objecter 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 Processor -- start 2024-03-11T21:48:31.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- start start 2024-03-11T21:48:31.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 librados: setting wanted keys 2024-03-11T21:48:31.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 librados: calling monclient init 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 0x7f002819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f00281a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028117c40 con 0x7f0028112b00 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028117ac0 con 0x7f0028075df0 2024-03-11T21:48:31.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0028117dc0 con 0x7f0028074550 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51540/0 (socket says 172.21.15.143:51540) 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 -- 172.21.15.143:0/2901736934 learned_addr learned my addr 172.21.15.143:0/2901736934 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002d475700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f00281a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0027fff700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 0x7f002819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 -- 172.21.15.143:0/2901736934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 msgr2=0x7f002819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 0x7f002819e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 -- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 msgr2=0x7f00281a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f00281a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 -- 172.21.15.143:0/2901736934 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00281a3230 con 0x7f0028074550 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002d475700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f00281a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002cc74700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f001000b530 tx=0x7f001000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0025ffb700 10 client.?.objecter ms_handle_connect 0x7f0028074550 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0025ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:31.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010015020 con 0x7f0028074550 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00281a3520 con 0x7f0028074550 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 10 client.42401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00281aada0 con 0x7f0028074550 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f0010011cb0 con 0x7f0028074550 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f002eed8700 1 librados: init done 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.095+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010010d90 con 0x7f0028074550 2024-03-11T21:48:31.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002804faa0 con 0x7f0028074550 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f0010018020 con 0x7f0028074550 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 0x7f0014071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1004..1004 src has 954..1004) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f001008dfe0 con 0x7f0028074550 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 10 client.42401.objecter ms_dispatch 0x7f0028001550 osd_map(1004..1004 src has 954..1004) v4 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 3 client.42401.objecter handle_osd_map got epochs [1004,1004] > 0 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 3 client.42401.objecter handle_osd_map decoding full epoch 1004 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0027fff700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 0x7f0014071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 20 client.42401.objecter dump_active .. 0 homeless 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0025ffb700 10 client.42401.objecter ms_handle_connect 0x7f001406f820 2024-03-11T21:48:31.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.099+0000 7f0027fff700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 0x7f0014071ce0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f0018009680 tx=0x7f0018009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:31.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.103+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f001001f020 con 0x7f0028074550 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[29284]: audit 2024-03-11T21:48:29.366242+0000 mgr.y (mgr.14550) 975 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[29284]: cluster 2024-03-11T21:48:29.819425+0000 mgr.y (mgr.14550) 976 : cluster 0 pgmap v1646: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[29284]: cluster 2024-03-11T21:48:30.328694+0000 osd.5 (osd.5) 531 : cluster 0 1.8 scrub starts 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[29284]: cluster 2024-03-11T21:48:30.330930+0000 osd.5 (osd.5) 532 : cluster 0 1.8 scrub ok 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[21512]: audit 2024-03-11T21:48:29.366242+0000 mgr.y (mgr.14550) 975 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[21512]: cluster 2024-03-11T21:48:29.819425+0000 mgr.y (mgr.14550) 976 : cluster 0 pgmap v1646: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:31.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[21512]: cluster 2024-03-11T21:48:30.328694+0000 osd.5 (osd.5) 531 : cluster 0 1.8 scrub starts 2024-03-11T21:48:31.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:31 smithi143 bash[21512]: cluster 2024-03-11T21:48:30.330930+0000 osd.5 (osd.5) 532 : cluster 0 1.8 scrub ok 2024-03-11T21:48:31.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:31 smithi203 bash[23654]: audit 2024-03-11T21:48:29.366242+0000 mgr.y (mgr.14550) 975 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:31.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:31 smithi203 bash[23654]: cluster 2024-03-11T21:48:29.819425+0000 mgr.y (mgr.14550) 976 : cluster 0 pgmap v1646: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:31.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:31 smithi203 bash[23654]: cluster 2024-03-11T21:48:30.328694+0000 osd.5 (osd.5) 531 : cluster 0 1.8 scrub starts 2024-03-11T21:48:31.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:31 smithi203 bash[23654]: cluster 2024-03-11T21:48:30.330930+0000 osd.5 (osd.5) 532 : cluster 0 1.8 scrub ok 2024-03-11T21:48:31.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:31.327+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f00281ab000 con 0x7f0028074550 2024-03-11T21:48:32.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.011+0000 7f0025ffb700 1 -- 172.21.15.143:0/2901736934 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v1005) v1 ==== 133+0+0 (secure 0 0 0) 0x7f001005aff0 con 0x7f0028074550 2024-03-11T21:48:32.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 10 librados: watch_flush enter 2024-03-11T21:48:32.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 10 librados: watch_flush exit 2024-03-11T21:48:32.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 20 client.42401.objecter shutdown clearing up homeless session... 2024-03-11T21:48:32.005 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 10 client.42401.objecter successfully canceled tick 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 msgr2=0x7f0014071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 0x7f0014071ce0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f0018009680 tx=0x7f0018009260 comp rx=0 tx=0).stop 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 msgr2=0x7f002819e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f001000b530 tx=0x7f001000b840 comp rx=0 tx=0).stop 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 shutdown_connections 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f001406f820 0x7f0014071ce0 unknown :-1 s=CLOSED pgs=3510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:32.006 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0028112b00 0x7f00281a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0028074550 0x7f002819e220 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 --2- 172.21.15.143:0/2901736934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0028075df0 0x7f002819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 >> 172.21.15.143:0/2901736934 conn(0x7f0028078eb0 msgr2=0x7f0028111440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.015+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 shutdown_connections 2024-03-11T21:48:32.007 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.019+0000 7f002eed8700 1 -- 172.21.15.143:0/2901736934 wait complete. 2024-03-11T21:48:32.010 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:32.019+0000 7f002eed8700 1 librados: shutdown 2024-03-11T21:48:32.011 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-11T21:48:32.258 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:32.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[29284]: cluster 2024-03-11T21:48:31.124970+0000 osd.1 (osd.1) 423 : cluster 0 2.6 scrub starts 2024-03-11T21:48:32.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[29284]: cluster 2024-03-11T21:48:31.127201+0000 osd.1 (osd.1) 424 : cluster 0 2.6 scrub ok 2024-03-11T21:48:32.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[29284]: audit 2024-03-11T21:48:31.330216+0000 mon.c (mon.2) 1415 : audit 1 from='client.? 172.21.15.143:0/2901736934' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.258 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[29284]: audit 2024-03-11T21:48:31.330685+0000 mon.a (mon.0) 4154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[21512]: cluster 2024-03-11T21:48:31.124970+0000 osd.1 (osd.1) 423 : cluster 0 2.6 scrub starts 2024-03-11T21:48:32.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[21512]: cluster 2024-03-11T21:48:31.127201+0000 osd.1 (osd.1) 424 : cluster 0 2.6 scrub ok 2024-03-11T21:48:32.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[21512]: audit 2024-03-11T21:48:31.330216+0000 mon.c (mon.2) 1415 : audit 1 from='client.? 172.21.15.143:0/2901736934' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.259 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:32 smithi143 bash[21512]: audit 2024-03-11T21:48:31.330685+0000 mon.a (mon.0) 4154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:32 smithi203 bash[23654]: cluster 2024-03-11T21:48:31.124970+0000 osd.1 (osd.1) 423 : cluster 0 2.6 scrub starts 2024-03-11T21:48:32.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:32 smithi203 bash[23654]: cluster 2024-03-11T21:48:31.127201+0000 osd.1 (osd.1) 424 : cluster 0 2.6 scrub ok 2024-03-11T21:48:32.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:32 smithi203 bash[23654]: audit 2024-03-11T21:48:31.330216+0000 mon.c (mon.2) 1415 : audit 1 from='client.? 172.21.15.143:0/2901736934' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:32 smithi203 bash[23654]: audit 2024-03-11T21:48:31.330685+0000 mon.a (mon.0) 4154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:32.953 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:48:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:31.820454+0000 mgr.y (mgr.14550) 977 : cluster 0 pgmap v1647: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: audit 2024-03-11T21:48:32.003683+0000 mon.a (mon.0) 4155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.012978+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e1005: 8 total, 8 up, 8 in 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.101812+0000 osd.7 (osd.7) 634 : cluster 0 9.e scrub starts 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.106462+0000 osd.7 (osd.7) 635 : cluster 0 9.e scrub ok 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.256097+0000 osd.4 (osd.4) 593 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:48:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:33 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.258382+0000 osd.4 (osd.4) 594 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:48:33.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:31.820454+0000 mgr.y (mgr.14550) 977 : cluster 0 pgmap v1647: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:33.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: audit 2024-03-11T21:48:32.003683+0000 mon.a (mon.0) 4155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.012978+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e1005: 8 total, 8 up, 8 in 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.101812+0000 osd.7 (osd.7) 634 : cluster 0 9.e scrub starts 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.106462+0000 osd.7 (osd.7) 635 : cluster 0 9.e scrub ok 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.256097+0000 osd.4 (osd.4) 593 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.258382+0000 osd.4 (osd.4) 594 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:31.820454+0000 mgr.y (mgr.14550) 977 : cluster 0 pgmap v1647: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: audit 2024-03-11T21:48:32.003683+0000 mon.a (mon.0) 4155 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.012978+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e1005: 8 total, 8 up, 8 in 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.101812+0000 osd.7 (osd.7) 634 : cluster 0 9.e scrub starts 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.106462+0000 osd.7 (osd.7) 635 : cluster 0 9.e scrub ok 2024-03-11T21:48:33.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.256097+0000 osd.4 (osd.4) 593 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:48:33.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:33 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.258382+0000 osd.4 (osd.4) 594 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:48:34.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.655341+0000 osd.2 (osd.2) 511 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:48:34.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:32.657590+0000 osd.2 (osd.2) 512 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:48:34.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.106350+0000 osd.7 (osd.7) 636 : cluster 0 5.1a scrub starts 2024-03-11T21:48:34.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.108882+0000 osd.7 (osd.7) 637 : cluster 0 5.1a scrub ok 2024-03-11T21:48:34.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.305379+0000 osd.4 (osd.4) 595 : cluster 0 5.18 scrub starts 2024-03-11T21:48:34.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:34 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.307698+0000 osd.4 (osd.4) 596 : cluster 0 5.18 scrub ok 2024-03-11T21:48:34.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.655341+0000 osd.2 (osd.2) 511 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:32.657590+0000 osd.2 (osd.2) 512 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.106350+0000 osd.7 (osd.7) 636 : cluster 0 5.1a scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.108882+0000 osd.7 (osd.7) 637 : cluster 0 5.1a scrub ok 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.305379+0000 osd.4 (osd.4) 595 : cluster 0 5.18 scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.307698+0000 osd.4 (osd.4) 596 : cluster 0 5.18 scrub ok 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.655341+0000 osd.2 (osd.2) 511 : cluster 0 5.8 deep-scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:32.657590+0000 osd.2 (osd.2) 512 : cluster 0 5.8 deep-scrub ok 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.106350+0000 osd.7 (osd.7) 636 : cluster 0 5.1a scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.108882+0000 osd.7 (osd.7) 637 : cluster 0 5.1a scrub ok 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.305379+0000 osd.4 (osd.4) 595 : cluster 0 5.18 scrub starts 2024-03-11T21:48:34.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:34 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.307698+0000 osd.4 (osd.4) 596 : cluster 0 5.18 scrub ok 2024-03-11T21:48:34.933 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:35 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.615092+0000 osd.2 (osd.2) 513 : cluster 0 1.14 scrub starts 2024-03-11T21:48:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:35 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.617701+0000 osd.2 (osd.2) 514 : cluster 0 1.14 scrub ok 2024-03-11T21:48:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:35 smithi203 bash[23654]: cluster 2024-03-11T21:48:33.822304+0000 mgr.y (mgr.14550) 978 : cluster 0 pgmap v1649: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:35 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.008477+0000 osd.0 (osd.0) 626 : cluster 0 9.b scrub starts 2024-03-11T21:48:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:35 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.014465+0000 osd.0 (osd.0) 627 : cluster 0 9.b scrub ok 2024-03-11T21:48:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.615092+0000 osd.2 (osd.2) 513 : cluster 0 1.14 scrub starts 2024-03-11T21:48:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.617701+0000 osd.2 (osd.2) 514 : cluster 0 1.14 scrub ok 2024-03-11T21:48:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[21512]: cluster 2024-03-11T21:48:33.822304+0000 mgr.y (mgr.14550) 978 : cluster 0 pgmap v1649: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.008477+0000 osd.0 (osd.0) 626 : cluster 0 9.b scrub starts 2024-03-11T21:48:35.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.014465+0000 osd.0 (osd.0) 627 : cluster 0 9.b scrub ok 2024-03-11T21:48:35.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.615092+0000 osd.2 (osd.2) 513 : cluster 0 1.14 scrub starts 2024-03-11T21:48:35.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.617701+0000 osd.2 (osd.2) 514 : cluster 0 1.14 scrub ok 2024-03-11T21:48:35.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[29284]: cluster 2024-03-11T21:48:33.822304+0000 mgr.y (mgr.14550) 978 : cluster 0 pgmap v1649: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:35.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.008477+0000 osd.0 (osd.0) 626 : cluster 0 9.b scrub starts 2024-03-11T21:48:35.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:35 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.014465+0000 osd.0 (osd.0) 627 : cluster 0 9.b scrub ok 2024-03-11T21:48:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:36 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.393521+0000 osd.2 (osd.2) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:36.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:36 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.394003+0000 osd.2 (osd.2) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:36.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:36 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.661277+0000 osd.2 (osd.2) 517 : cluster 0 5.b deep-scrub starts 2024-03-11T21:48:36.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:36 smithi203 bash[23654]: cluster 2024-03-11T21:48:34.663666+0000 osd.2 (osd.2) 518 : cluster 0 5.b deep-scrub ok 2024-03-11T21:48:36.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:36 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.051718+0000 mon.a (mon.0) 4157 : cluster 0 osdmap e1006: 8 total, 8 up, 8 in 2024-03-11T21:48:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.393521+0000 osd.2 (osd.2) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:36.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.394003+0000 osd.2 (osd.2) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.661277+0000 osd.2 (osd.2) 517 : cluster 0 5.b deep-scrub starts 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[21512]: cluster 2024-03-11T21:48:34.663666+0000 osd.2 (osd.2) 518 : cluster 0 5.b deep-scrub ok 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.051718+0000 mon.a (mon.0) 4157 : cluster 0 osdmap e1006: 8 total, 8 up, 8 in 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.393521+0000 osd.2 (osd.2) 515 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.394003+0000 osd.2 (osd.2) 516 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.661277+0000 osd.2 (osd.2) 517 : cluster 0 5.b deep-scrub starts 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[29284]: cluster 2024-03-11T21:48:34.663666+0000 osd.2 (osd.2) 518 : cluster 0 5.b deep-scrub ok 2024-03-11T21:48:36.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:36 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.051718+0000 mon.a (mon.0) 4157 : cluster 0 osdmap e1006: 8 total, 8 up, 8 in 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.482450+0000 osd.6 (osd.6) 497 : cluster 0 2.14 scrub starts 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.484600+0000 osd.6 (osd.6) 498 : cluster 0 2.14 scrub ok 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.497781+0000 osd.3 (osd.3) 625 : cluster 0 9.1 scrub starts 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.501528+0000 osd.3 (osd.3) 626 : cluster 0 9.1 scrub ok 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.688882+0000 osd.2 (osd.2) 519 : cluster 0 2.1 scrub starts 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.691042+0000 osd.2 (osd.2) 520 : cluster 0 2.1 scrub ok 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:35.823362+0000 mgr.y (mgr.14550) 979 : cluster 0 pgmap v1651: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:37.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:36.355577+0000 osd.5 (osd.5) 533 : cluster 0 6.a scrub starts 2024-03-11T21:48:37.284 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:37 smithi203 bash[23654]: cluster 2024-03-11T21:48:36.357821+0000 osd.5 (osd.5) 534 : cluster 0 6.a scrub ok 2024-03-11T21:48:37.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.482450+0000 osd.6 (osd.6) 497 : cluster 0 2.14 scrub starts 2024-03-11T21:48:37.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.484600+0000 osd.6 (osd.6) 498 : cluster 0 2.14 scrub ok 2024-03-11T21:48:37.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.497781+0000 osd.3 (osd.3) 625 : cluster 0 9.1 scrub starts 2024-03-11T21:48:37.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.501528+0000 osd.3 (osd.3) 626 : cluster 0 9.1 scrub ok 2024-03-11T21:48:37.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.688882+0000 osd.2 (osd.2) 519 : cluster 0 2.1 scrub starts 2024-03-11T21:48:37.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.691042+0000 osd.2 (osd.2) 520 : cluster 0 2.1 scrub ok 2024-03-11T21:48:37.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:35.823362+0000 mgr.y (mgr.14550) 979 : cluster 0 pgmap v1651: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:37.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:36.355577+0000 osd.5 (osd.5) 533 : cluster 0 6.a scrub starts 2024-03-11T21:48:37.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[21512]: cluster 2024-03-11T21:48:36.357821+0000 osd.5 (osd.5) 534 : cluster 0 6.a scrub ok 2024-03-11T21:48:37.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.482450+0000 osd.6 (osd.6) 497 : cluster 0 2.14 scrub starts 2024-03-11T21:48:37.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.484600+0000 osd.6 (osd.6) 498 : cluster 0 2.14 scrub ok 2024-03-11T21:48:37.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.497781+0000 osd.3 (osd.3) 625 : cluster 0 9.1 scrub starts 2024-03-11T21:48:37.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.501528+0000 osd.3 (osd.3) 626 : cluster 0 9.1 scrub ok 2024-03-11T21:48:37.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.688882+0000 osd.2 (osd.2) 519 : cluster 0 2.1 scrub starts 2024-03-11T21:48:37.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.691042+0000 osd.2 (osd.2) 520 : cluster 0 2.1 scrub ok 2024-03-11T21:48:37.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:35.823362+0000 mgr.y (mgr.14550) 979 : cluster 0 pgmap v1651: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:37.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:36.355577+0000 osd.5 (osd.5) 533 : cluster 0 6.a scrub starts 2024-03-11T21:48:37.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:37 smithi143 bash[29284]: cluster 2024-03-11T21:48:36.357821+0000 osd.5 (osd.5) 534 : cluster 0 6.a scrub ok 2024-03-11T21:48:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:110 315.5 315:a1d3b400:test-rados-api-smithi143-71773-115::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e700)", 2024-03-11T21:48:37.936 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:47.554929+0000", 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 530.38720671800002, 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0095245980000000004, 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.937 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 110 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.554929+0000", 2024-03-11T21:48:37.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.554929+0000", 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.554932+0000", 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2260000000000001e-06 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.554947+0000", 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5509999999999999e-05 2024-03-11T21:48:37.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.554986+0000", 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8501999999999998e-05 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.555006+0000", 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.978e-05 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.555096+0000", 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0147000000000003e-05 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.555101+0000", 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.429e-06 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.561239+0000", 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006137923 2024-03-11T21:48:37.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.561739+0000", 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049994700000000004 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.563792+0000", 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020531830000000001 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for rw locks", 2024-03-11T21:48:37.946 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.563927+0000", 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013507300000000001 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.564288+0000", 2024-03-11T21:48:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000360381 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:47.564453+0000", 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016549699999999999 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.948 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:167 313.7s0 313:e031f6fd:test-rados-api-smithi143-71773-117::bar:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e712)", 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:11.577336+0000", 2024-03-11T21:48:37.949 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 506.36479903600002, 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0097125719999999992, 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577336+0000", 2024-03-11T21:48:37.950 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577336+0000", 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577340+0000", 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6069999999999999e-06 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.951 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577356+0000", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6336000000000001e-05 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577403+0000", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6387999999999997e-05 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577433+0000", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0735999999999998e-05 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.952 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577551+0000", 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011793799999999999 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577559+0000", 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1780000000000003e-06 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.586532+0000", 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089726609999999998 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.953 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.586792+0000", 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025956200000000002 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.587049+0000", 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025716600000000002 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.954 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:168 313.7s0 313:e031f6fd:test-rados-api-smithi143-71773-117::bar:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e712)", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:11.577491+0000", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 506.36464472799997, 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011747518, 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577491+0000", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.955 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577491+0000", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577492+0000", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.697e-06 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577503+0000", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0637e-05 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577527+0000", 2024-03-11T21:48:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3832000000000002e-05 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577547+0000", 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0173000000000002e-05 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577639+0000", 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1735000000000002e-05 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting_for_map not empty", 2024-03-11T21:48:37.957 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.577642+0000", 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5209999999999998e-06 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.587124+0000", 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094822629999999995 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.587334+0000", 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020922700000000001 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.958 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:11.589238+0000", 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019044330000000001 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:266 316.8 316:10000000:%01:::head [pgnls start_epoch 713] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e713)", 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:12.525265+0000", 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 505.41687025800002, 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0072003550000000003, 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.959 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 266 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525265+0000", 2024-03-11T21:48:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525265+0000", 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525320+0000", 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084000000000003e-05 2024-03-11T21:48:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525347+0000", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7229999999999998e-05 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525393+0000", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6022000000000001e-05 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525423+0000", 2024-03-11T21:48:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9822e-05 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525603+0000", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000179605 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.525614+0000", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1096e-05 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.532096+0000", 2024-03-11T21:48:37.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064823670000000002 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.532199+0000", 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102431 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:12.532465+0000", 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000266698 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(39.16,QREQUEST,e:716,from:2,reservation_nonce:12)", 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:15.581725+0000", 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 502.36040993699999, 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089567239999999992, 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581725+0000", 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.965 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581725+0000", 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581728+0000", 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7369999999999998e-06 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581742+0000", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3686000000000001e-05 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581777+0000", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5079000000000001e-05 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581794+0000", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7173e-05 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581930+0000", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013572000000000001 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.581936+0000", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0889999999999996e-06 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.590517+0000", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0085812419999999993 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.590561+0000", 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3741e-05 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.590682+0000", 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012125699999999999 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:372 317.1e 317:7b8f8022:test-rados-api-smithi143-71773-119::foo:3 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e723)", 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:22.518954+0000", 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.42318077499999, 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027157189000000002, 2024-03-11T21:48:37.969 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 372 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.518954+0000", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.518954+0000", 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.518959+0000", 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4800000000000003e-06 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.971 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.518978+0000", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9190000000000001e-05 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.519029+0000", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1088000000000003e-05 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.519059+0000", 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9553e-05 2024-03-11T21:48:37.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.519251+0000", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019268099999999999 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.519269+0000", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7122999999999999e-05 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.532318+0000", 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013049622 2024-03-11T21:48:37.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.532700+0000", 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000381889 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.539625+0000", 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069250350000000004 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.540227+0000", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060168500000000002 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.545127+0000", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048996689999999997 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:22.546112+0000", 2024-03-11T21:48:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0009851739999999999 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:409 317.10 317:08000000::::head [pgnls start_epoch 725] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e725)", 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:24.612885+0000", 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 493.32925011399999, 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006394187, 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.976 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 409 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612885+0000", 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612885+0000", 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.977 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612890+0000", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0599999999999998e-06 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612907+0000", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6823000000000001e-05 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612958+0000", 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0667999999999997e-05 2024-03-11T21:48:37.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.612986+0000", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8476e-05 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.613162+0000", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000175448 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.613173+0000", 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1636e-05 2024-03-11T21:48:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.618788+0000", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056144080000000004 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.618951+0000", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163492 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:24.619279+0000", 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032817600000000001 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.980 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:474 313.3s0 313:d7fbd44d:test-rados-api-smithi143-71773-121::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:27.501628+0000", 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 490.44050678899998, 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.026704421999999998, 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 474 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.981 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501628+0000", 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501628+0000", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501631+0000", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7379999999999999e-06 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501643+0000", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1405e-05 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501666+0000", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3020999999999999e-05 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501680+0000", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4151000000000001e-05 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501865+0000", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000184964 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.501874+0000", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.85e-06 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.513071+0000", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011197386 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.513263+0000", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019199000000000001 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:48:37.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.513747+0000", 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00048430400000000002 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.515008+0000", 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012606309999999999 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.528272+0000", 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013264442 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:27.528333+0000", 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.054e-05 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:835 320.5 320:a0000000:%01:::head [pgnls start_epoch 749] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e749)", 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:48.709471+0000", 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 469.23266461600002, 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076372059999999997, 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.987 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 835 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709471+0000", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709471+0000", 2024-03-11T21:48:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709474+0000", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9440000000000001e-06 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709490+0000", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6812000000000001e-05 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709528+0000", 2024-03-11T21:48:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7539000000000001e-05 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709546+0000", 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8275999999999999e-05 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709684+0000", 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013769300000000001 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.709695+0000", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1026000000000001e-05 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.716662+0000", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006966659 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.716754+0000", 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2268000000000005e-05 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:48.717108+0000", 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035398899999999997 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:932 321.1 321:80000000:test-rados-api-smithi143-71773-127:::head [pgnls start_epoch 757] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e757)", 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:56.609074+0000", 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 461.333061493, 2024-03-11T21:48:37.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071950989999999999, 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 932 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609074+0000", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609074+0000", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609077+0000", 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.58e-06 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609096+0000", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8206000000000002e-05 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609134+0000", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8068999999999998e-05 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609153+0000", 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9623000000000002e-05 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609297+0000", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014372500000000001 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.609309+0000", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1790000000000001e-05 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.615997+0000", 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066882770000000003 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.616085+0000", 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8300000000000005e-05 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:56.616269+0000", 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018352900000000001 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:37.997 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1101 322.11 322:8e7c84a9:test-rados-api-smithi143-71773-129::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e765)", 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:04.563605+0000", 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 453.378530082, 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018265064000000001, 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:37.998 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1101 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563605+0000", 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563605+0000", 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563609+0000", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8999999999999999e-06 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563630+0000", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0462000000000002e-05 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563676+0000", 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6765999999999999e-05 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563706+0000", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0139000000000001e-05 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563906+0000", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019960099999999999 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.563925+0000", 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9199999999999999e-05 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.569788+0000", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0058628279999999996 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for blocked object", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.571042+0000", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012538619999999999 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.578334+0000", 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0072917049999999999 2024-03-11T21:48:38.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.578606+0000", 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000272269 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.578964+0000", 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035779699999999998 2024-03-11T21:48:38.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.580169+0000", 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0012047959999999999 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.581202+0000", 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010335279999999999 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.581703+0000", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00050077200000000002 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.581796+0000", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3331000000000005e-05 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.581870+0000", 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4108000000000003e-05 2024-03-11T21:48:38.005 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1242 322.e 322:70000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:11.658956+0000", 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 446.28317900899998, 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017666408000000001, 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1242 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.658956+0000", 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.658956+0000", 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.658960+0000", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0090000000000001e-06 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.658979+0000", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8355999999999998e-05 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.659019+0000", 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0343000000000003e-05 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.659038+0000", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9168999999999999e-05 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.659215+0000", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017638500000000001 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.659232+0000", 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7456e-05 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.676429+0000", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017196554999999999 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.676479+0000", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0294000000000003e-05 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.676623+0000", 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014384099999999999 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.010 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2162 329.1 329:80000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:30.625423+0000", 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 367.31671268600002, 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007084468, 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2162 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.011 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625423+0000", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625423+0000", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625425+0000", 2024-03-11T21:48:38.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.272e-06 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625436+0000", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1455e-05 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625457+0000", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1092000000000001e-05 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625467+0000", 2024-03-11T21:48:38.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8169999999999994e-06 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625550+0000", 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2310000000000003e-05 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.625555+0000", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0860000000000001e-06 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.632397+0000", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0068425370000000001 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.632422+0000", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5233e-05 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.632507+0000", 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4666000000000006e-05 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2326 330.6 330:60000000:test-rados-api-smithi143-71773-145:::head [pgnls start_epoch 850] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e850)", 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:38.616111+0000", 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 359.32602396800002, 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083677019999999994, 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:48:38.016 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2326 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616111+0000", 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616111+0000", 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616115+0000", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4470000000000002e-06 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616131+0000", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5926e-05 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616167+0000", 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6229000000000002e-05 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616185+0000", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8170000000000001e-05 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616360+0000", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174482 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.616378+0000", 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8833000000000001e-05 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.624170+0000", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0077911530000000003 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.624259+0000", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8987999999999996e-05 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:38.624479+0000", 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022047399999999999 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.020 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14427.0:4150 3.3 e864/78 3:c35966f0:::obj_delete_at_hint.0000000109:head v 864'47, mlcod=864'45)", 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.633852+0000", 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.308283141, 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010368281, 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633852+0000", 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633852+0000", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633853+0000", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5199999999999996e-07 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633858+0000", 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4720000000000003e-06 2024-03-11T21:48:38.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633869+0000", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0385e-05 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633876+0000", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6809999999999998e-06 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633927+0000", 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0871000000000001e-05 2024-03-11T21:48:38.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.633950+0000", 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3176999999999998e-05 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644099+0000", 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010149096 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.644220+0000", 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012094700000000001 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.025 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(1.a,QREQUEST,e:877,from:5,reservation_nonce:7)", 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:24.610772+0000", 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 313.33136356300002, 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019461638, 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610772+0000", 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610772+0000", 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610774+0000", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8479999999999999e-06 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610783+0000", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3929999999999997e-06 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610805+0000", 2024-03-11T21:48:38.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1871000000000001e-05 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610819+0000", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3867e-05 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610937+0000", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118218 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.610946+0000", 2024-03-11T21:48:38.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3729999999999999e-06 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.628878+0000", 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017931493999999999 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.628900+0000", 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2688999999999999e-05 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:24.630233+0000", 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001332885 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(3.1e,QREQUEST,e:883,from:3,reservation_nonce:17)", 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:30.668683+0000", 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 307.273451868, 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064422990000000003, 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668683+0000", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668683+0000", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668687+0000", 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.596e-06 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668703+0000", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6359999999999999e-05 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668740+0000", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6705000000000002e-05 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668759+0000", 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8770000000000002e-05 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668942+0000", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018306600000000001 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.668959+0000", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6739e-05 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.675014+0000", 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006055665 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.675046+0000", 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1848999999999999e-05 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:30.675126+0000", 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9548999999999993e-05 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.42197.0:2 338.0 338:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-03-11T21:48:38.035 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:44:30.644246+0000", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 247.29788893599999, 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.05822942, 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.42197", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1515746225", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644246+0000", 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644246+0000", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644249+0000", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3609999999999999e-06 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644257+0000", 2024-03-11T21:48:38.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4479999999999994e-06 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644272+0000", 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4919999999999999e-05 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.644284+0000", 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1504e-05 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.645997+0000", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001713462 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:30.646008+0000", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0502999999999999e-05 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.692972+0000", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.046964647 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.693307+0000", 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00033499300000000001 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.693642+0000", 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000334469 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.694559+0000", 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091774700000000001 2024-03-11T21:48:38.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.701832+0000", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0072721950000000004 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.702278+0000", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00044658600000000001 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.702387+0000", 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010846200000000001 2024-03-11T21:48:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:44:31.702476+0000", 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9122999999999994e-05 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.42245.0:3 343.0 343:0384ab8e:::oid:head [write 0~1048576] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e949)", 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:02.700060+0000", 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 215.24207524100001, 2024-03-11T21:48:38.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15142731200000001, 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.42245", 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3932584904", 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.700060+0000", 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.700060+0000", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.700062+0000", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2560000000000001e-06 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.701344+0000", 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001281945 2024-03-11T21:48:38.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.701360+0000", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5688e-05 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.701373+0000", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2609000000000001e-05 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.701440+0000", 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7809000000000001e-05 2024-03-11T21:48:38.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.701450+0000", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6719999999999999e-06 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.838598+0000", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13714807300000001 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.838921+0000", 2024-03-11T21:48:38.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00032335600000000001 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,7", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.839260+0000", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00033887000000000001 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.844605+0000", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053448840000000003 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848772+0000", 2024-03-11T21:48:38.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0041665269999999997 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.851301+0000", 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025292729999999999 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.851409+0000", 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107791 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.851487+0000", 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8559000000000005e-05 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.42245.0:5 343.0 343:0384ab8e:::oid:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:03.687533+0000", 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 214.25460189099999, 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0097909020000000006, 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.42245", 2024-03-11T21:48:38.049 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3932584904", 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 5 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687533+0000", 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:48:38.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687533+0000", 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687537+0000", 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1099999999999996e-06 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687555+0000", 2024-03-11T21:48:38.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7504e-05 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687602+0000", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6519e-05 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687628+0000", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6872e-05 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687786+0000", 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015715 2024-03-11T21:48:38.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.687808+0000", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2442999999999998e-05 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.691653+0000", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038448699999999998 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.691907+0000", 2024-03-11T21:48:38.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025374099999999998 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,7", 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.693143+0000", 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001236584 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.694948+0000", 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001805309 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.696744+0000", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001795023 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.697041+0000", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029768999999999998 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.697203+0000", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016177300000000001 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:48:38.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:03.697324+0000", 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121314 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:48:38.056 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:38 smithi203 bash[23654]: cluster 2024-03-11T21:48:36.478645+0000 osd.6 (osd.6) 499 : cluster 0 3.10 scrub starts 2024-03-11T21:48:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:38 smithi203 bash[23654]: cluster 2024-03-11T21:48:36.481766+0000 osd.6 (osd.6) 500 : cluster 0 3.10 scrub ok 2024-03-11T21:48:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:38 smithi203 bash[23654]: cluster 2024-03-11T21:48:37.023296+0000 osd.0 (osd.0) 628 : cluster 0 2.1f scrub starts 2024-03-11T21:48:38.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:38 smithi203 bash[23654]: cluster 2024-03-11T21:48:37.026862+0000 osd.0 (osd.0) 629 : cluster 0 2.1f scrub ok 2024-03-11T21:48:38.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[29284]: cluster 2024-03-11T21:48:36.478645+0000 osd.6 (osd.6) 499 : cluster 0 3.10 scrub starts 2024-03-11T21:48:38.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[29284]: cluster 2024-03-11T21:48:36.481766+0000 osd.6 (osd.6) 500 : cluster 0 3.10 scrub ok 2024-03-11T21:48:38.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[29284]: cluster 2024-03-11T21:48:37.023296+0000 osd.0 (osd.0) 628 : cluster 0 2.1f scrub starts 2024-03-11T21:48:38.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[29284]: cluster 2024-03-11T21:48:37.026862+0000 osd.0 (osd.0) 629 : cluster 0 2.1f scrub ok 2024-03-11T21:48:38.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[21512]: cluster 2024-03-11T21:48:36.478645+0000 osd.6 (osd.6) 499 : cluster 0 3.10 scrub starts 2024-03-11T21:48:38.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[21512]: cluster 2024-03-11T21:48:36.481766+0000 osd.6 (osd.6) 500 : cluster 0 3.10 scrub ok 2024-03-11T21:48:38.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[21512]: cluster 2024-03-11T21:48:37.023296+0000 osd.0 (osd.0) 628 : cluster 0 2.1f scrub starts 2024-03-11T21:48:38.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:38 smithi143 bash[21512]: cluster 2024-03-11T21:48:37.026862+0000 osd.0 (osd.0) 629 : cluster 0 2.1f scrub ok 2024-03-11T21:48:38.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:37.823895+0000 mgr.y (mgr.14550) 980 : cluster 0 pgmap v1652: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.067122+0000 osd.1 (osd.1) 425 : cluster 0 9.14 scrub starts 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.069816+0000 osd.1 (osd.1) 426 : cluster 0 9.14 scrub ok 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: audit 2024-03-11T21:48:38.174437+0000 mon.c (mon.2) 1416 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.333615+0000 osd.4 (osd.4) 597 : cluster 0 6.1a scrub starts 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.336144+0000 osd.4 (osd.4) 598 : cluster 0 6.1a scrub ok 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.403491+0000 osd.5 (osd.5) 535 : cluster 0 2.18 scrub starts 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.405673+0000 osd.5 (osd.5) 536 : cluster 0 2.18 scrub ok 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.705001+0000 osd.2 (osd.2) 521 : cluster 0 5.6 scrub starts 2024-03-11T21:48:39.366 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.707107+0000 osd.2 (osd.2) 522 : cluster 0 5.6 scrub ok 2024-03-11T21:48:39.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:37.823895+0000 mgr.y (mgr.14550) 980 : cluster 0 pgmap v1652: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.067122+0000 osd.1 (osd.1) 425 : cluster 0 9.14 scrub starts 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.069816+0000 osd.1 (osd.1) 426 : cluster 0 9.14 scrub ok 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: audit 2024-03-11T21:48:38.174437+0000 mon.c (mon.2) 1416 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.333615+0000 osd.4 (osd.4) 597 : cluster 0 6.1a scrub starts 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.336144+0000 osd.4 (osd.4) 598 : cluster 0 6.1a scrub ok 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.403491+0000 osd.5 (osd.5) 535 : cluster 0 2.18 scrub starts 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.405673+0000 osd.5 (osd.5) 536 : cluster 0 2.18 scrub ok 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.705001+0000 osd.2 (osd.2) 521 : cluster 0 5.6 scrub starts 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.707107+0000 osd.2 (osd.2) 522 : cluster 0 5.6 scrub ok 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:37.823895+0000 mgr.y (mgr.14550) 980 : cluster 0 pgmap v1652: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.067122+0000 osd.1 (osd.1) 425 : cluster 0 9.14 scrub starts 2024-03-11T21:48:39.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.069816+0000 osd.1 (osd.1) 426 : cluster 0 9.14 scrub ok 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: audit 2024-03-11T21:48:38.174437+0000 mon.c (mon.2) 1416 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.333615+0000 osd.4 (osd.4) 597 : cluster 0 6.1a scrub starts 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.336144+0000 osd.4 (osd.4) 598 : cluster 0 6.1a scrub ok 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.403491+0000 osd.5 (osd.5) 535 : cluster 0 2.18 scrub starts 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.405673+0000 osd.5 (osd.5) 536 : cluster 0 2.18 scrub ok 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.705001+0000 osd.2 (osd.2) 521 : cluster 0 5.6 scrub starts 2024-03-11T21:48:39.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:39 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.707107+0000 osd.2 (osd.2) 522 : cluster 0 5.6 scrub ok 2024-03-11T21:48:39.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:40.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:40 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.450974+0000 osd.6 (osd.6) 501 : cluster 0 1.f deep-scrub starts 2024-03-11T21:48:40.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:40 smithi143 bash[21512]: cluster 2024-03-11T21:48:38.453628+0000 osd.6 (osd.6) 502 : cluster 0 1.f deep-scrub ok 2024-03-11T21:48:40.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:40 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.450974+0000 osd.6 (osd.6) 501 : cluster 0 1.f deep-scrub starts 2024-03-11T21:48:40.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:40 smithi143 bash[29284]: cluster 2024-03-11T21:48:38.453628+0000 osd.6 (osd.6) 502 : cluster 0 1.f deep-scrub ok 2024-03-11T21:48:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:40 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.450974+0000 osd.6 (osd.6) 501 : cluster 0 1.f deep-scrub starts 2024-03-11T21:48:40.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:40 smithi203 bash[23654]: cluster 2024-03-11T21:48:38.453628+0000 osd.6 (osd.6) 502 : cluster 0 1.f deep-scrub ok 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/3518952082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898104260 msgr2=0x7f4898104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- 172.21.15.143:0/3518952082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898104260 0x7f4898104640 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f4888009fc0 tx=0x7f488800b330 comp rx=0 tx=0).stop 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/3518952082 shutdown_connections 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- 172.21.15.143:0/3518952082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898109340 0x7f4898110f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- 172.21.15.143:0/3518952082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898104260 0x7f4898104640 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- 172.21.15.143:0/3518952082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104c10 0x7f4898108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/3518952082 >> 172.21.15.143:0/3518952082 conn(0x7f48980ffaf0 msgr2=0x7f4898101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/3518952082 shutdown_connections 2024-03-11T21:48:41.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/3518952082 wait complete. 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 librados: starting msgr at 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 librados: starting objecter 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 Processor -- start 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- start start 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 librados: setting wanted keys 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 librados: calling monclient init 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 0x7f4898199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898104c10 0x7f489819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 0x7f489819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4898113830 con 0x7f4898104260 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48981136b0 con 0x7f4898104c10 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48981139b0 con 0x7f4898109340 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4897fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 0x7f489819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4897fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 0x7f489819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:50014/0 (socket says 172.21.15.143:50014) 2024-03-11T21:48:41.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4897fff700 1 -- 172.21.15.143:0/2999679126 learned_addr learned my addr 172.21.15.143:0/2999679126 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:41.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 0x7f4898199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:41.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4896ffd700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898104c10 0x7f489819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:41.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 -- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 msgr2=0x7f489819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:41.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 0x7f489819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 -- 172.21.15.143:0/2999679126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898104c10 msgr2=0x7f489819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898104c10 0x7f489819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 -- 172.21.15.143:0/2999679126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489819ee40 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f48977fe700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 0x7f4898199e30 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f488800b6c0 tx=0x7f488800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4894ff9700 10 client.?.objecter ms_handle_connect 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4894ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888021070 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f488801d450 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880052a0 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489819f0d0 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 10 client.31203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48981a69b0 con 0x7f4898104260 2024-03-11T21:48:41.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.386+0000 7f489dda8700 1 librados: init done 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4888024070 con 0x7f4898104260 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489804faa0 con 0x7f4898104260 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 0x7f4884071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1006..1006 src has 954..1006) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f488800f020 con 0x7f4898104260 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 10 client.31203.objecter ms_dispatch 0x7f4898001550 osd_map(1006..1006 src has 954..1006) v4 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 3 client.31203.objecter handle_osd_map got epochs [1006,1006] > 0 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 3 client.31203.objecter handle_osd_map decoding full epoch 1006 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4896ffd700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 0x7f4884071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 20 client.31203.objecter dump_active .. 0 homeless 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4896ffd700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 0x7f4884071ce0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f4880009750 tx=0x7f4880009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:41.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.390+0000 7f4894ff9700 10 client.31203.objecter ms_handle_connect 0x7f488406f820 2024-03-11T21:48:41.386 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.394+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f488802b020 con 0x7f4898104260 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:39.092200+0000 osd.0 (osd.0) 630 : cluster 0 9.1e scrub starts 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:39.097084+0000 osd.0 (osd.0) 631 : cluster 0 9.1e scrub ok 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: audit 2024-03-11T21:48:39.377200+0000 mgr.y (mgr.14550) 981 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:39.497183+0000 osd.6 (osd.6) 503 : cluster 0 5.1f scrub starts 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:39.500046+0000 osd.6 (osd.6) 504 : cluster 0 5.1f scrub ok 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:39.825712+0000 mgr.y (mgr.14550) 982 : cluster 0 pgmap v1653: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:41.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.092697+0000 osd.1 (osd.1) 427 : cluster 0 1.b deep-scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.094792+0000 osd.1 (osd.1) 428 : cluster 0 1.b deep-scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.291502+0000 osd.4 (osd.4) 599 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.294132+0000 osd.4 (osd.4) 600 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.361412+0000 osd.5 (osd.5) 537 : cluster 0 3.1c scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.366343+0000 osd.5 (osd.5) 538 : cluster 0 3.1c scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:39.092200+0000 osd.0 (osd.0) 630 : cluster 0 9.1e scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:39.097084+0000 osd.0 (osd.0) 631 : cluster 0 9.1e scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: audit 2024-03-11T21:48:39.377200+0000 mgr.y (mgr.14550) 981 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:39.497183+0000 osd.6 (osd.6) 503 : cluster 0 5.1f scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:39.500046+0000 osd.6 (osd.6) 504 : cluster 0 5.1f scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:39.825712+0000 mgr.y (mgr.14550) 982 : cluster 0 pgmap v1653: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.092697+0000 osd.1 (osd.1) 427 : cluster 0 1.b deep-scrub starts 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.094792+0000 osd.1 (osd.1) 428 : cluster 0 1.b deep-scrub ok 2024-03-11T21:48:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.291502+0000 osd.4 (osd.4) 599 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:48:41.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.294132+0000 osd.4 (osd.4) 600 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:48:41.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.361412+0000 osd.5 (osd.5) 537 : cluster 0 3.1c scrub starts 2024-03-11T21:48:41.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:41 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.366343+0000 osd.5 (osd.5) 538 : cluster 0 3.1c scrub ok 2024-03-11T21:48:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:39.092200+0000 osd.0 (osd.0) 630 : cluster 0 9.1e scrub starts 2024-03-11T21:48:41.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:39.097084+0000 osd.0 (osd.0) 631 : cluster 0 9.1e scrub ok 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: audit 2024-03-11T21:48:39.377200+0000 mgr.y (mgr.14550) 981 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:39.497183+0000 osd.6 (osd.6) 503 : cluster 0 5.1f scrub starts 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:39.500046+0000 osd.6 (osd.6) 504 : cluster 0 5.1f scrub ok 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:39.825712+0000 mgr.y (mgr.14550) 982 : cluster 0 pgmap v1653: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.092697+0000 osd.1 (osd.1) 427 : cluster 0 1.b deep-scrub starts 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.094792+0000 osd.1 (osd.1) 428 : cluster 0 1.b deep-scrub ok 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.291502+0000 osd.4 (osd.4) 599 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.294132+0000 osd.4 (osd.4) 600 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.361412+0000 osd.5 (osd.5) 537 : cluster 0 3.1c scrub starts 2024-03-11T21:48:41.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:41 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.366343+0000 osd.5 (osd.5) 538 : cluster 0 3.1c scrub ok 2024-03-11T21:48:41.616 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:41.626+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f48981a6b80 con 0x7f4898104260 2024-03-11T21:48:42.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.070+0000 7f4894ff9700 1 -- 172.21.15.143:0/2999679126 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v1007) v1 ==== 113+0+0 (secure 0 0 0) 0x7f488805ca90 con 0x7f4898104260 2024-03-11T21:48:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 10 librados: watch_flush enter 2024-03-11T21:48:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 10 librados: watch_flush exit 2024-03-11T21:48:42.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 20 client.31203.objecter shutdown clearing up homeless session... 2024-03-11T21:48:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 10 client.31203.objecter successfully canceled tick 2024-03-11T21:48:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 msgr2=0x7f4884071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 0x7f4884071ce0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f4880009750 tx=0x7f4880009310 comp rx=0 tx=0).stop 2024-03-11T21:48:42.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 msgr2=0x7f4898199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 0x7f4898199e30 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f488800b6c0 tx=0x7f488800ccb0 comp rx=0 tx=0).stop 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 shutdown_connections 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f488406f820 0x7f4884071ce0 unknown :-1 s=CLOSED pgs=3511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898104260 0x7f4898199e30 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4898109340 0x7f489819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 --2- 172.21.15.143:0/2999679126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4898104c10 0x7f489819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 >> 172.21.15.143:0/2999679126 conn(0x7f48980ffaf0 msgr2=0x7f4898100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 shutdown_connections 2024-03-11T21:48:42.069 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 -- 172.21.15.143:0/2999679126 wait complete. 2024-03-11T21:48:42.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:42.078+0000 7f489dda8700 1 librados: shutdown 2024-03-11T21:48:42.072 INFO:teuthology.orchestra.run.smithi143.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.457055+0000 osd.3 (osd.3) 627 : cluster 0 5.0 scrub starts 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21512]: cluster 2024-03-11T21:48:40.459883+0000 osd.3 (osd.3) 628 : cluster 0 5.0 scrub ok 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21512]: audit 2024-03-11T21:48:41.629768+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21512]: cluster 2024-03-11T21:48:41.662845+0000 osd.2 (osd.2) 523 : cluster 0 2.10 scrub starts 2024-03-11T21:48:42.323 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[21512]: cluster 2024-03-11T21:48:41.665021+0000 osd.2 (osd.2) 524 : cluster 0 2.10 scrub ok 2024-03-11T21:48:42.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.457055+0000 osd.3 (osd.3) 627 : cluster 0 5.0 scrub starts 2024-03-11T21:48:42.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[29284]: cluster 2024-03-11T21:48:40.459883+0000 osd.3 (osd.3) 628 : cluster 0 5.0 scrub ok 2024-03-11T21:48:42.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[29284]: audit 2024-03-11T21:48:41.629768+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:42.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[29284]: cluster 2024-03-11T21:48:41.662845+0000 osd.2 (osd.2) 523 : cluster 0 2.10 scrub starts 2024-03-11T21:48:42.324 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:42 smithi143 bash[29284]: cluster 2024-03-11T21:48:41.665021+0000 osd.2 (osd.2) 524 : cluster 0 2.10 scrub ok 2024-03-11T21:48:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:42 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.457055+0000 osd.3 (osd.3) 627 : cluster 0 5.0 scrub starts 2024-03-11T21:48:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:42 smithi203 bash[23654]: cluster 2024-03-11T21:48:40.459883+0000 osd.3 (osd.3) 628 : cluster 0 5.0 scrub ok 2024-03-11T21:48:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:42 smithi203 bash[23654]: audit 2024-03-11T21:48:41.629768+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:42 smithi203 bash[23654]: cluster 2024-03-11T21:48:41.662845+0000 osd.2 (osd.2) 523 : cluster 0 2.10 scrub starts 2024-03-11T21:48:42.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:42 smithi203 bash[23654]: cluster 2024-03-11T21:48:41.665021+0000 osd.2 (osd.2) 524 : cluster 0 2.10 scrub ok 2024-03-11T21:48:42.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:41.826813+0000 mgr.y (mgr.14550) 983 : cluster 0 pgmap v1654: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.028734+0000 osd.0 (osd.0) 632 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.031069+0000 osd.0 (osd.0) 633 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: audit 2024-03-11T21:48:42.073045+0000 mon.a (mon.0) 4159 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.080149+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e1007: 8 total, 8 up, 8 in 2024-03-11T21:48:43.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.146764+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub starts 2024-03-11T21:48:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.148339+0000 osd.1 (osd.1) 430 : cluster 0 6.4 scrub ok 2024-03-11T21:48:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.209179+0000 osd.7 (osd.7) 638 : cluster 0 9.11 scrub starts 2024-03-11T21:48:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.214499+0000 osd.7 (osd.7) 639 : cluster 0 9.11 scrub ok 2024-03-11T21:48:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.711718+0000 osd.2 (osd.2) 525 : cluster 0 2.e scrub starts 2024-03-11T21:48:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.714067+0000 osd.2 (osd.2) 526 : cluster 0 2.e scrub ok 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:41.826813+0000 mgr.y (mgr.14550) 983 : cluster 0 pgmap v1654: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.028734+0000 osd.0 (osd.0) 632 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.031069+0000 osd.0 (osd.0) 633 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: audit 2024-03-11T21:48:42.073045+0000 mon.a (mon.0) 4159 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.080149+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e1007: 8 total, 8 up, 8 in 2024-03-11T21:48:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.146764+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub starts 2024-03-11T21:48:43.204 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.148339+0000 osd.1 (osd.1) 430 : cluster 0 6.4 scrub ok 2024-03-11T21:48:43.204 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.209179+0000 osd.7 (osd.7) 638 : cluster 0 9.11 scrub starts 2024-03-11T21:48:43.204 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.214499+0000 osd.7 (osd.7) 639 : cluster 0 9.11 scrub ok 2024-03-11T21:48:43.204 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.711718+0000 osd.2 (osd.2) 525 : cluster 0 2.e scrub starts 2024-03-11T21:48:43.204 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:43 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.714067+0000 osd.2 (osd.2) 526 : cluster 0 2.e scrub ok 2024-03-11T21:48:43.409 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:41.826813+0000 mgr.y (mgr.14550) 983 : cluster 0 pgmap v1654: 184 pgs: 1 active+clean+scrubbing, 183 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.028734+0000 osd.0 (osd.0) 632 : cluster 0 6.10 deep-scrub starts 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.031069+0000 osd.0 (osd.0) 633 : cluster 0 6.10 deep-scrub ok 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: audit 2024-03-11T21:48:42.073045+0000 mon.a (mon.0) 4159 : audit 1 from='client.? 172.21.15.143:0/2999679126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.080149+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e1007: 8 total, 8 up, 8 in 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.146764+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub starts 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.148339+0000 osd.1 (osd.1) 430 : cluster 0 6.4 scrub ok 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.209179+0000 osd.7 (osd.7) 638 : cluster 0 9.11 scrub starts 2024-03-11T21:48:43.434 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.214499+0000 osd.7 (osd.7) 639 : cluster 0 9.11 scrub ok 2024-03-11T21:48:43.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.711718+0000 osd.2 (osd.2) 525 : cluster 0 2.e scrub starts 2024-03-11T21:48:43.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:43 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.714067+0000 osd.2 (osd.2) 526 : cluster 0 2.e scrub ok 2024-03-11T21:48:44.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:44 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.466299+0000 osd.3 (osd.3) 629 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:48:44.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:44 smithi203 bash[23654]: cluster 2024-03-11T21:48:42.476169+0000 osd.3 (osd.3) 630 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:48:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:44 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.466299+0000 osd.3 (osd.3) 629 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:48:44.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:44 smithi143 bash[21512]: cluster 2024-03-11T21:48:42.476169+0000 osd.3 (osd.3) 630 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:48:44.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:44 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.466299+0000 osd.3 (osd.3) 629 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:48:44.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:44 smithi143 bash[29284]: cluster 2024-03-11T21:48:42.476169+0000 osd.3 (osd.3) 630 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:48:45.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:43.828726+0000 mgr.y (mgr.14550) 984 : cluster 0 pgmap v1656: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.165026+0000 osd.7 (osd.7) 640 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.166868+0000 osd.7 (osd.7) 641 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.262569+0000 osd.4 (osd.4) 601 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.265046+0000 osd.4 (osd.4) 602 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.403567+0000 osd.5 (osd.5) 539 : cluster 0 9.6 scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.407180+0000 osd.5 (osd.5) 540 : cluster 0 9.6 scrub ok 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:43.828726+0000 mgr.y (mgr.14550) 984 : cluster 0 pgmap v1656: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.165026+0000 osd.7 (osd.7) 640 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.166868+0000 osd.7 (osd.7) 641 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.262569+0000 osd.4 (osd.4) 601 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.265046+0000 osd.4 (osd.4) 602 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.403567+0000 osd.5 (osd.5) 539 : cluster 0 9.6 scrub starts 2024-03-11T21:48:45.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:45 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.407180+0000 osd.5 (osd.5) 540 : cluster 0 9.6 scrub ok 2024-03-11T21:48:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:43.828726+0000 mgr.y (mgr.14550) 984 : cluster 0 pgmap v1656: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.165026+0000 osd.7 (osd.7) 640 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:48:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.166868+0000 osd.7 (osd.7) 641 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:48:45.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.262569+0000 osd.4 (osd.4) 601 : cluster 0 9.10 deep-scrub starts 2024-03-11T21:48:45.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.265046+0000 osd.4 (osd.4) 602 : cluster 0 9.10 deep-scrub ok 2024-03-11T21:48:45.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.403567+0000 osd.5 (osd.5) 539 : cluster 0 9.6 scrub starts 2024-03-11T21:48:45.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:45 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.407180+0000 osd.5 (osd.5) 540 : cluster 0 9.6 scrub ok 2024-03-11T21:48:46.293 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:46.293 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:48:46.293 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-03-11T21:48:46.293 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:46.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:46 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.455450+0000 osd.6 (osd.6) 505 : cluster 0 2.1a scrub starts 2024-03-11T21:48:46.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:46 smithi203 bash[23654]: cluster 2024-03-11T21:48:44.456808+0000 osd.6 (osd.6) 506 : cluster 0 2.1a scrub ok 2024-03-11T21:48:46.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:46 smithi203 bash[23654]: cluster 2024-03-11T21:48:45.165822+0000 osd.1 (osd.1) 431 : cluster 0 1.3 scrub starts 2024-03-11T21:48:46.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:46 smithi203 bash[23654]: cluster 2024-03-11T21:48:45.168415+0000 osd.1 (osd.1) 432 : cluster 0 1.3 scrub ok 2024-03-11T21:48:46.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.455450+0000 osd.6 (osd.6) 505 : cluster 0 2.1a scrub starts 2024-03-11T21:48:46.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[21512]: cluster 2024-03-11T21:48:44.456808+0000 osd.6 (osd.6) 506 : cluster 0 2.1a scrub ok 2024-03-11T21:48:46.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[21512]: cluster 2024-03-11T21:48:45.165822+0000 osd.1 (osd.1) 431 : cluster 0 1.3 scrub starts 2024-03-11T21:48:46.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[21512]: cluster 2024-03-11T21:48:45.168415+0000 osd.1 (osd.1) 432 : cluster 0 1.3 scrub ok 2024-03-11T21:48:46.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.455450+0000 osd.6 (osd.6) 505 : cluster 0 2.1a scrub starts 2024-03-11T21:48:46.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[29284]: cluster 2024-03-11T21:48:44.456808+0000 osd.6 (osd.6) 506 : cluster 0 2.1a scrub ok 2024-03-11T21:48:46.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[29284]: cluster 2024-03-11T21:48:45.165822+0000 osd.1 (osd.1) 431 : cluster 0 1.3 scrub starts 2024-03-11T21:48:46.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:46 smithi143 bash[29284]: cluster 2024-03-11T21:48:45.168415+0000 osd.1 (osd.1) 432 : cluster 0 1.3 scrub ok 2024-03-11T21:48:47.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:48:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:47 smithi203 bash[23654]: cluster 2024-03-11T21:48:45.829763+0000 mgr.y (mgr.14550) 985 : cluster 0 pgmap v1657: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:47 smithi203 bash[23654]: cluster 2024-03-11T21:48:46.152901+0000 osd.1 (osd.1) 433 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:48:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:47 smithi203 bash[23654]: cluster 2024-03-11T21:48:46.155212+0000 osd.1 (osd.1) 434 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:48:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:47 smithi203 bash[23654]: cluster 2024-03-11T21:48:46.162253+0000 osd.7 (osd.7) 642 : cluster 0 2.1c scrub starts 2024-03-11T21:48:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:47 smithi203 bash[23654]: cluster 2024-03-11T21:48:46.163947+0000 osd.7 (osd.7) 643 : cluster 0 2.1c scrub ok 2024-03-11T21:48:47.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[29284]: cluster 2024-03-11T21:48:45.829763+0000 mgr.y (mgr.14550) 985 : cluster 0 pgmap v1657: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:47.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[29284]: cluster 2024-03-11T21:48:46.152901+0000 osd.1 (osd.1) 433 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[29284]: cluster 2024-03-11T21:48:46.155212+0000 osd.1 (osd.1) 434 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[29284]: cluster 2024-03-11T21:48:46.162253+0000 osd.7 (osd.7) 642 : cluster 0 2.1c scrub starts 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[29284]: cluster 2024-03-11T21:48:46.163947+0000 osd.7 (osd.7) 643 : cluster 0 2.1c scrub ok 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[21512]: cluster 2024-03-11T21:48:45.829763+0000 mgr.y (mgr.14550) 985 : cluster 0 pgmap v1657: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[21512]: cluster 2024-03-11T21:48:46.152901+0000 osd.1 (osd.1) 433 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[21512]: cluster 2024-03-11T21:48:46.155212+0000 osd.1 (osd.1) 434 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[21512]: cluster 2024-03-11T21:48:46.162253+0000 osd.7 (osd.7) 642 : cluster 0 2.1c scrub starts 2024-03-11T21:48:47.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:47 smithi143 bash[21512]: cluster 2024-03-11T21:48:46.163947+0000 osd.7 (osd.7) 643 : cluster 0 2.1c scrub ok 2024-03-11T21:48:47.724 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:48.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:48 smithi143 bash[29284]: cluster 2024-03-11T21:48:47.197308+0000 osd.7 (osd.7) 644 : cluster 0 1.0 scrub starts 2024-03-11T21:48:48.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:48 smithi143 bash[29284]: cluster 2024-03-11T21:48:47.198979+0000 osd.7 (osd.7) 645 : cluster 0 1.0 scrub ok 2024-03-11T21:48:48.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:48 smithi143 bash[21512]: cluster 2024-03-11T21:48:47.197308+0000 osd.7 (osd.7) 644 : cluster 0 1.0 scrub starts 2024-03-11T21:48:48.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:48 smithi143 bash[21512]: cluster 2024-03-11T21:48:47.198979+0000 osd.7 (osd.7) 645 : cluster 0 1.0 scrub ok 2024-03-11T21:48:48.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:48 smithi203 bash[23654]: cluster 2024-03-11T21:48:47.197308+0000 osd.7 (osd.7) 644 : cluster 0 1.0 scrub starts 2024-03-11T21:48:48.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:48 smithi203 bash[23654]: cluster 2024-03-11T21:48:47.198979+0000 osd.7 (osd.7) 645 : cluster 0 1.0 scrub ok 2024-03-11T21:48:49.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[23654]: cluster 2024-03-11T21:48:47.830502+0000 mgr.y (mgr.14550) 986 : cluster 0 pgmap v1658: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:49.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[23654]: cluster 2024-03-11T21:48:48.372592+0000 osd.6 (osd.6) 507 : cluster 0 5.2 scrub starts 2024-03-11T21:48:49.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[23654]: cluster 2024-03-11T21:48:48.374799+0000 osd.6 (osd.6) 508 : cluster 0 5.2 scrub ok 2024-03-11T21:48:49.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[23654]: cluster 2024-03-11T21:48:48.414382+0000 osd.3 (osd.3) 631 : cluster 0 1.4 scrub starts 2024-03-11T21:48:49.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[23654]: cluster 2024-03-11T21:48:48.417152+0000 osd.3 (osd.3) 632 : cluster 0 1.4 scrub ok 2024-03-11T21:48:49.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[29284]: cluster 2024-03-11T21:48:47.830502+0000 mgr.y (mgr.14550) 986 : cluster 0 pgmap v1658: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:49.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[29284]: cluster 2024-03-11T21:48:48.372592+0000 osd.6 (osd.6) 507 : cluster 0 5.2 scrub starts 2024-03-11T21:48:49.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[29284]: cluster 2024-03-11T21:48:48.374799+0000 osd.6 (osd.6) 508 : cluster 0 5.2 scrub ok 2024-03-11T21:48:49.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[29284]: cluster 2024-03-11T21:48:48.414382+0000 osd.3 (osd.3) 631 : cluster 0 1.4 scrub starts 2024-03-11T21:48:49.405 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[29284]: cluster 2024-03-11T21:48:48.417152+0000 osd.3 (osd.3) 632 : cluster 0 1.4 scrub ok 2024-03-11T21:48:49.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[21512]: cluster 2024-03-11T21:48:47.830502+0000 mgr.y (mgr.14550) 986 : cluster 0 pgmap v1658: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:49.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[21512]: cluster 2024-03-11T21:48:48.372592+0000 osd.6 (osd.6) 507 : cluster 0 5.2 scrub starts 2024-03-11T21:48:49.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[21512]: cluster 2024-03-11T21:48:48.374799+0000 osd.6 (osd.6) 508 : cluster 0 5.2 scrub ok 2024-03-11T21:48:49.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[21512]: cluster 2024-03-11T21:48:48.414382+0000 osd.3 (osd.3) 631 : cluster 0 1.4 scrub starts 2024-03-11T21:48:49.406 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:49 smithi143 bash[21512]: cluster 2024-03-11T21:48:48.417152+0000 osd.3 (osd.3) 632 : cluster 0 1.4 scrub ok 2024-03-11T21:48:49.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:50.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[21512]: cluster 2024-03-11T21:48:49.265543+0000 osd.4 (osd.4) 603 : cluster 0 5.1 scrub starts 2024-03-11T21:48:50.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[21512]: cluster 2024-03-11T21:48:49.267695+0000 osd.4 (osd.4) 604 : cluster 0 5.1 scrub ok 2024-03-11T21:48:50.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[21512]: cluster 2024-03-11T21:48:49.301475+0000 osd.5 (osd.5) 541 : cluster 0 2.2 scrub starts 2024-03-11T21:48:50.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[21512]: cluster 2024-03-11T21:48:49.303643+0000 osd.5 (osd.5) 542 : cluster 0 2.2 scrub ok 2024-03-11T21:48:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[29284]: cluster 2024-03-11T21:48:49.265543+0000 osd.4 (osd.4) 603 : cluster 0 5.1 scrub starts 2024-03-11T21:48:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[29284]: cluster 2024-03-11T21:48:49.267695+0000 osd.4 (osd.4) 604 : cluster 0 5.1 scrub ok 2024-03-11T21:48:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[29284]: cluster 2024-03-11T21:48:49.301475+0000 osd.5 (osd.5) 541 : cluster 0 2.2 scrub starts 2024-03-11T21:48:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:50 smithi143 bash[29284]: cluster 2024-03-11T21:48:49.303643+0000 osd.5 (osd.5) 542 : cluster 0 2.2 scrub ok 2024-03-11T21:48:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:50 smithi203 bash[23654]: cluster 2024-03-11T21:48:49.265543+0000 osd.4 (osd.4) 603 : cluster 0 5.1 scrub starts 2024-03-11T21:48:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:50 smithi203 bash[23654]: cluster 2024-03-11T21:48:49.267695+0000 osd.4 (osd.4) 604 : cluster 0 5.1 scrub ok 2024-03-11T21:48:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:50 smithi203 bash[23654]: cluster 2024-03-11T21:48:49.301475+0000 osd.5 (osd.5) 541 : cluster 0 2.2 scrub starts 2024-03-11T21:48:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:50 smithi203 bash[23654]: cluster 2024-03-11T21:48:49.303643+0000 osd.5 (osd.5) 542 : cluster 0 2.2 scrub ok 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 -- 172.21.15.143:0/2956882981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc109350 msgr2=0x7f32cc110f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 --2- 172.21.15.143:0/2956882981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc109350 0x7f32cc110f20 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f32c400b0d0 tx=0x7f32c400b3e0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 -- 172.21.15.143:0/2956882981 shutdown_connections 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 --2- 172.21.15.143:0/2956882981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc104270 0x7f32cc104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 --2- 172.21.15.143:0/2956882981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc109350 0x7f32cc110f20 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 --2- 172.21.15.143:0/2956882981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 0x7f32cc108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 -- 172.21.15.143:0/2956882981 >> 172.21.15.143:0/2956882981 conn(0x7f32cc0ffb00 msgr2=0x7f32cc101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 -- 172.21.15.143:0/2956882981 shutdown_connections 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.274+0000 7f32d44f2700 1 -- 172.21.15.143:0/2956882981 wait complete. 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 librados: starting msgr at 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 librados: starting objecter 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 Processor -- start 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- start start 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 librados: setting wanted keys 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 librados: calling monclient init 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc104270 0x7f32cc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 0x7f32cc19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113850 con 0x7f32cc109350 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1136d0 con 0x7f32cc104c20 2024-03-11T21:48:51.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1139d0 con 0x7f32cc104270 2024-03-11T21:48:51.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:51.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:51668/0 (socket says 172.21.15.143:51668) 2024-03-11T21:48:51.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 -- 172.21.15.143:0/1159915918 learned_addr learned my addr 172.21.15.143:0/1159915918 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:48:51.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d228e700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc104270 0x7f32cc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:51.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 -- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc104270 msgr2=0x7f32cc199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:51.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc104270 0x7f32cc199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 -- 172.21.15.143:0/1159915918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 msgr2=0x7f32cc19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d1a8d700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 0x7f32cc19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 0x7f32cc19a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 -- 172.21.15.143:0/1159915918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc19ed70 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d2a8f700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006070 tx=0x7f32c4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 10 client.?.objecter ms_handle_connect 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4021020 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f32c4004c50 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc19f000 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 10 client.31209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc1a68e0 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 librados: init done 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4005050 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc04faa0 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f32c4024020 con 0x7f32cc109350 2024-03-11T21:48:51.272 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.278+0000 7f32c37fe700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 0x7f32b8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1007..1007 src has 954..1007) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f32c400f020 con 0x7f32cc109350 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 10 client.31209.objecter ms_dispatch 0x7f32cc001550 osd_map(1007..1007 src has 954..1007) v4 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 3 client.31209.objecter handle_osd_map got epochs [1007,1007] > 0 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 3 client.31209.objecter handle_osd_map decoding full epoch 1007 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32d228e700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 0x7f32b8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 20 client.31209.objecter dump_active .. 0 homeless 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32c37fe700 10 client.31209.objecter ms_handle_connect 0x7f32b806f820 2024-03-11T21:48:51.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.282+0000 7f32d228e700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 0x7f32b8071ce0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009680 tx=0x7f32bc009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:48:51.276 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.286+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f32c405c870 con 0x7f32cc109350 2024-03-11T21:48:51.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: audit 2024-03-11T21:48:49.386028+0000 mgr.y (mgr.14550) 987 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:51.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: cluster 2024-03-11T21:48:49.832454+0000 mgr.y (mgr.14550) 988 : cluster 0 pgmap v1659: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:51.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: cluster 2024-03-11T21:48:50.082056+0000 osd.0 (osd.0) 634 : cluster 0 9.1c scrub starts 2024-03-11T21:48:51.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: cluster 2024-03-11T21:48:50.083440+0000 osd.0 (osd.0) 635 : cluster 0 9.1c scrub ok 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: cluster 2024-03-11T21:48:50.109625+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[21512]: cluster 2024-03-11T21:48:50.111275+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: audit 2024-03-11T21:48:49.386028+0000 mgr.y (mgr.14550) 987 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: cluster 2024-03-11T21:48:49.832454+0000 mgr.y (mgr.14550) 988 : cluster 0 pgmap v1659: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: cluster 2024-03-11T21:48:50.082056+0000 osd.0 (osd.0) 634 : cluster 0 9.1c scrub starts 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: cluster 2024-03-11T21:48:50.083440+0000 osd.0 (osd.0) 635 : cluster 0 9.1c scrub ok 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: cluster 2024-03-11T21:48:50.109625+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-03-11T21:48:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:51 smithi143 bash[29284]: cluster 2024-03-11T21:48:50.111275+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-03-11T21:48:51.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:51.510+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f32cc1a6ab0 con 0x7f32cc109350 2024-03-11T21:48:51.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: audit 2024-03-11T21:48:49.386028+0000 mgr.y (mgr.14550) 987 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:51.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: cluster 2024-03-11T21:48:49.832454+0000 mgr.y (mgr.14550) 988 : cluster 0 pgmap v1659: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:51.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: cluster 2024-03-11T21:48:50.082056+0000 osd.0 (osd.0) 634 : cluster 0 9.1c scrub starts 2024-03-11T21:48:51.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: cluster 2024-03-11T21:48:50.083440+0000 osd.0 (osd.0) 635 : cluster 0 9.1c scrub ok 2024-03-11T21:48:51.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: cluster 2024-03-11T21:48:50.109625+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-03-11T21:48:51.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:51 smithi203 bash[23654]: cluster 2024-03-11T21:48:50.111275+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-03-11T21:48:51.809 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:52.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.130+0000 7f32c37fe700 1 -- 172.21.15.143:0/1159915918 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v1008) v1 ==== 133+0+0 (secure 0 0 0) 0x7f32c40602c0 con 0x7f32cc109350 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 10 librados: watch_flush enter 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 10 librados: watch_flush exit 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 20 client.31209.objecter shutdown clearing up homeless session... 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 10 client.31209.objecter successfully canceled tick 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 msgr2=0x7f32b8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 0x7f32b8071ce0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009680 tx=0x7f32bc009260 comp rx=0 tx=0).stop 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 msgr2=0x7f32cc19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:48:52.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006070 tx=0x7f32c4007a20 comp rx=0 tx=0).stop 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 shutdown_connections 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f32b806f820 0x7f32b8071ce0 unknown :-1 s=CLOSED pgs=3512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc109350 0x7f32cc19e630 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32cc104270 0x7f32cc199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 --2- 172.21.15.143:0/1159915918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32cc104c20 0x7f32cc19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.134+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 >> 172.21.15.143:0/1159915918 conn(0x7f32cc0ffb00 msgr2=0x7f32cc100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:48:52.127 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.138+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 shutdown_connections 2024-03-11T21:48:52.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.138+0000 7f32d44f2700 1 -- 172.21.15.143:0/1159915918 wait complete. 2024-03-11T21:48:52.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:48:52.138+0000 7f32d44f2700 1 librados: shutdown 2024-03-11T21:48:52.128 INFO:teuthology.orchestra.run.smithi143.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-11T21:48:52.371 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:48:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[29284]: audit 2024-03-11T21:48:51.513032+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[29284]: cluster 2024-03-11T21:48:51.590962+0000 osd.5 (osd.5) 543 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[29284]: cluster 2024-03-11T21:48:51.591394+0000 osd.5 (osd.5) 544 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[21512]: audit 2024-03-11T21:48:51.513032+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[21512]: cluster 2024-03-11T21:48:51.590962+0000 osd.5 (osd.5) 543 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:52.372 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:52 smithi143 bash[21512]: cluster 2024-03-11T21:48:51.591394+0000 osd.5 (osd.5) 544 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:52.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:52 smithi203 bash[23654]: audit 2024-03-11T21:48:51.513032+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:48:52.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:52 smithi203 bash[23654]: cluster 2024-03-11T21:48:51.590962+0000 osd.5 (osd.5) 543 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:52.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:52 smithi203 bash[23654]: cluster 2024-03-11T21:48:51.591394+0000 osd.5 (osd.5) 544 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:52.965 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-11T21:48:52.965 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-11T21:48:52.965 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:51.833551+0000 mgr.y (mgr.14550) 989 : cluster 0 pgmap v1660: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.065292+0000 osd.1 (osd.1) 437 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.065889+0000 osd.1 (osd.1) 438 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: audit 2024-03-11T21:48:52.131727+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.139728+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e1008: 8 total, 8 up, 8 in 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.305120+0000 osd.5 (osd.5) 545 : cluster 0 5.f scrub starts 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.307996+0000 osd.5 (osd.5) 546 : cluster 0 5.f scrub ok 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.442806+0000 osd.3 (osd.3) 633 : cluster 0 6.11 scrub starts 2024-03-11T21:48:53.199 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.444719+0000 osd.3 (osd.3) 634 : cluster 0 6.11 scrub ok 2024-03-11T21:48:53.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.890881+0000 osd.2 (osd.2) 527 : cluster 0 5.d scrub starts 2024-03-11T21:48:53.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:53 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.893301+0000 osd.2 (osd.2) 528 : cluster 0 5.d scrub ok 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:51.833551+0000 mgr.y (mgr.14550) 989 : cluster 0 pgmap v1660: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.065292+0000 osd.1 (osd.1) 437 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.065889+0000 osd.1 (osd.1) 438 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: audit 2024-03-11T21:48:52.131727+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.139728+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e1008: 8 total, 8 up, 8 in 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.305120+0000 osd.5 (osd.5) 545 : cluster 0 5.f scrub starts 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.307996+0000 osd.5 (osd.5) 546 : cluster 0 5.f scrub ok 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.442806+0000 osd.3 (osd.3) 633 : cluster 0 6.11 scrub starts 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.444719+0000 osd.3 (osd.3) 634 : cluster 0 6.11 scrub ok 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.890881+0000 osd.2 (osd.2) 527 : cluster 0 5.d scrub starts 2024-03-11T21:48:53.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.893301+0000 osd.2 (osd.2) 528 : cluster 0 5.d scrub ok 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:51.833551+0000 mgr.y (mgr.14550) 989 : cluster 0 pgmap v1660: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.065292+0000 osd.1 (osd.1) 437 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.065889+0000 osd.1 (osd.1) 438 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: audit 2024-03-11T21:48:52.131727+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.143:0/1159915918' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.139728+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e1008: 8 total, 8 up, 8 in 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.305120+0000 osd.5 (osd.5) 545 : cluster 0 5.f scrub starts 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.307996+0000 osd.5 (osd.5) 546 : cluster 0 5.f scrub ok 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.442806+0000 osd.3 (osd.3) 633 : cluster 0 6.11 scrub starts 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.444719+0000 osd.3 (osd.3) 634 : cluster 0 6.11 scrub ok 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.890881+0000 osd.2 (osd.2) 527 : cluster 0 5.d scrub starts 2024-03-11T21:48:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:53 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.893301+0000 osd.2 (osd.2) 528 : cluster 0 5.d scrub ok 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.457230+0000 osd.6 (osd.6) 509 : cluster 0 9.18 scrub starts 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:52.459365+0000 osd.6 (osd.6) 510 : cluster 0 9.18 scrub ok 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.117762+0000 osd.0 (osd.0) 636 : cluster 0 6.7 scrub starts 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.120497+0000 osd.0 (osd.0) 637 : cluster 0 6.7 scrub ok 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: audit 2024-03-11T21:48:53.206826+0000 mon.c (mon.2) 1417 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.221964+0000 osd.7 (osd.7) 646 : cluster 0 1.2 scrub starts 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.224288+0000 osd.7 (osd.7) 647 : cluster 0 1.2 scrub ok 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.293084+0000 osd.5 (osd.5) 547 : cluster 0 9.12 scrub starts 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.295862+0000 osd.5 (osd.5) 548 : cluster 0 9.12 scrub ok 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.432557+0000 osd.4 (osd.4) 605 : cluster 0 6.1e scrub starts 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.434953+0000 osd.4 (osd.4) 606 : cluster 0 6.1e scrub ok 2024-03-11T21:48:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.457230+0000 osd.6 (osd.6) 509 : cluster 0 9.18 scrub starts 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:52.459365+0000 osd.6 (osd.6) 510 : cluster 0 9.18 scrub ok 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.117762+0000 osd.0 (osd.0) 636 : cluster 0 6.7 scrub starts 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.120497+0000 osd.0 (osd.0) 637 : cluster 0 6.7 scrub ok 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: audit 2024-03-11T21:48:53.206826+0000 mon.c (mon.2) 1417 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.221964+0000 osd.7 (osd.7) 646 : cluster 0 1.2 scrub starts 2024-03-11T21:48:54.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.224288+0000 osd.7 (osd.7) 647 : cluster 0 1.2 scrub ok 2024-03-11T21:48:54.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.293084+0000 osd.5 (osd.5) 547 : cluster 0 9.12 scrub starts 2024-03-11T21:48:54.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.295862+0000 osd.5 (osd.5) 548 : cluster 0 9.12 scrub ok 2024-03-11T21:48:54.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.432557+0000 osd.4 (osd.4) 605 : cluster 0 6.1e scrub starts 2024-03-11T21:48:54.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:54 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.434953+0000 osd.4 (osd.4) 606 : cluster 0 6.1e scrub ok 2024-03-11T21:48:54.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.457230+0000 osd.6 (osd.6) 509 : cluster 0 9.18 scrub starts 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:52.459365+0000 osd.6 (osd.6) 510 : cluster 0 9.18 scrub ok 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.117762+0000 osd.0 (osd.0) 636 : cluster 0 6.7 scrub starts 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.120497+0000 osd.0 (osd.0) 637 : cluster 0 6.7 scrub ok 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: audit 2024-03-11T21:48:53.206826+0000 mon.c (mon.2) 1417 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.221964+0000 osd.7 (osd.7) 646 : cluster 0 1.2 scrub starts 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.224288+0000 osd.7 (osd.7) 647 : cluster 0 1.2 scrub ok 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.293084+0000 osd.5 (osd.5) 547 : cluster 0 9.12 scrub starts 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.295862+0000 osd.5 (osd.5) 548 : cluster 0 9.12 scrub ok 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.432557+0000 osd.4 (osd.4) 605 : cluster 0 6.1e scrub starts 2024-03-11T21:48:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:54 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.434953+0000 osd.4 (osd.4) 606 : cluster 0 6.1e scrub ok 2024-03-11T21:48:54.749 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:48:54.749 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-03-11T21:48:54.749 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-03-11T21:48:54.749 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-03-11T21:48:54.750 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:48:55.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:55 smithi203 bash[23654]: cluster 2024-03-11T21:48:53.835412+0000 mgr.y (mgr.14550) 990 : cluster 0 pgmap v1662: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:55.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:55 smithi203 bash[23654]: cluster 2024-03-11T21:48:54.289566+0000 osd.5 (osd.5) 549 : cluster 0 2.3 scrub starts 2024-03-11T21:48:55.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:55 smithi203 bash[23654]: cluster 2024-03-11T21:48:54.291940+0000 osd.5 (osd.5) 550 : cluster 0 2.3 scrub ok 2024-03-11T21:48:55.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:55 smithi203 bash[23654]: cluster 2024-03-11T21:48:54.295377+0000 osd.0 (osd.0) 638 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:55.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:55 smithi203 bash[23654]: cluster 2024-03-11T21:48:54.295792+0000 osd.0 (osd.0) 639 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:55.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[29284]: cluster 2024-03-11T21:48:53.835412+0000 mgr.y (mgr.14550) 990 : cluster 0 pgmap v1662: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:55.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[29284]: cluster 2024-03-11T21:48:54.289566+0000 osd.5 (osd.5) 549 : cluster 0 2.3 scrub starts 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[29284]: cluster 2024-03-11T21:48:54.291940+0000 osd.5 (osd.5) 550 : cluster 0 2.3 scrub ok 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[29284]: cluster 2024-03-11T21:48:54.295377+0000 osd.0 (osd.0) 638 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[29284]: cluster 2024-03-11T21:48:54.295792+0000 osd.0 (osd.0) 639 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[21512]: cluster 2024-03-11T21:48:53.835412+0000 mgr.y (mgr.14550) 990 : cluster 0 pgmap v1662: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[21512]: cluster 2024-03-11T21:48:54.289566+0000 osd.5 (osd.5) 549 : cluster 0 2.3 scrub starts 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[21512]: cluster 2024-03-11T21:48:54.291940+0000 osd.5 (osd.5) 550 : cluster 0 2.3 scrub ok 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[21512]: cluster 2024-03-11T21:48:54.295377+0000 osd.0 (osd.0) 638 : cluster 0 purged_snaps scrub starts 2024-03-11T21:48:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:55 smithi143 bash[21512]: cluster 2024-03-11T21:48:54.295792+0000 osd.0 (osd.0) 639 : cluster 0 purged_snaps scrub ok 2024-03-11T21:48:55.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.130080+0000 osd.0 (osd.0) 640 : cluster 0 6.18 scrub starts 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.132526+0000 osd.0 (osd.0) 641 : cluster 0 6.18 scrub ok 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.138609+0000 osd.1 (osd.1) 439 : cluster 0 2.9 scrub starts 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.140896+0000 osd.1 (osd.1) 440 : cluster 0 2.9 scrub ok 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.161856+0000 mon.a (mon.0) 4164 : cluster 0 osdmap e1009: 8 total, 8 up, 8 in 2024-03-11T21:48:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.181232+0000 osd.7 (osd.7) 648 : cluster 0 5.9 scrub starts 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.188197+0000 osd.7 (osd.7) 649 : cluster 0 5.9 scrub ok 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.453700+0000 osd.4 (osd.4) 607 : cluster 0 1.e scrub starts 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.456107+0000 osd.4 (osd.4) 608 : cluster 0 1.e scrub ok 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.130080+0000 osd.0 (osd.0) 640 : cluster 0 6.18 scrub starts 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.132526+0000 osd.0 (osd.0) 641 : cluster 0 6.18 scrub ok 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.138609+0000 osd.1 (osd.1) 439 : cluster 0 2.9 scrub starts 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.140896+0000 osd.1 (osd.1) 440 : cluster 0 2.9 scrub ok 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.161856+0000 mon.a (mon.0) 4164 : cluster 0 osdmap e1009: 8 total, 8 up, 8 in 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.181232+0000 osd.7 (osd.7) 648 : cluster 0 5.9 scrub starts 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.188197+0000 osd.7 (osd.7) 649 : cluster 0 5.9 scrub ok 2024-03-11T21:48:56.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.453700+0000 osd.4 (osd.4) 607 : cluster 0 1.e scrub starts 2024-03-11T21:48:56.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:56 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.456107+0000 osd.4 (osd.4) 608 : cluster 0 1.e scrub ok 2024-03-11T21:48:56.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.130080+0000 osd.0 (osd.0) 640 : cluster 0 6.18 scrub starts 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.132526+0000 osd.0 (osd.0) 641 : cluster 0 6.18 scrub ok 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.138609+0000 osd.1 (osd.1) 439 : cluster 0 2.9 scrub starts 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.140896+0000 osd.1 (osd.1) 440 : cluster 0 2.9 scrub ok 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.161856+0000 mon.a (mon.0) 4164 : cluster 0 osdmap e1009: 8 total, 8 up, 8 in 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.181232+0000 osd.7 (osd.7) 648 : cluster 0 5.9 scrub starts 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.188197+0000 osd.7 (osd.7) 649 : cluster 0 5.9 scrub ok 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.453700+0000 osd.4 (osd.4) 607 : cluster 0 1.e scrub starts 2024-03-11T21:48:56.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:56 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.456107+0000 osd.4 (osd.4) 608 : cluster 0 1.e scrub ok 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.471949+0000 osd.6 (osd.6) 511 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.474473+0000 osd.6 (osd.6) 512 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:55.836417+0000 mgr.y (mgr.14550) 991 : cluster 0 pgmap v1664: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:56.271835+0000 osd.5 (osd.5) 551 : cluster 0 5.15 scrub starts 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:56.275183+0000 osd.5 (osd.5) 552 : cluster 0 5.15 scrub ok 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:56.455302+0000 osd.4 (osd.4) 609 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[21512]: cluster 2024-03-11T21:48:56.457598+0000 osd.4 (osd.4) 610 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.471949+0000 osd.6 (osd.6) 511 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.474473+0000 osd.6 (osd.6) 512 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:48:57.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:55.836417+0000 mgr.y (mgr.14550) 991 : cluster 0 pgmap v1664: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:57.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:56.271835+0000 osd.5 (osd.5) 551 : cluster 0 5.15 scrub starts 2024-03-11T21:48:57.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:56.275183+0000 osd.5 (osd.5) 552 : cluster 0 5.15 scrub ok 2024-03-11T21:48:57.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:56.455302+0000 osd.4 (osd.4) 609 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:48:57.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:57 smithi143 bash[29284]: cluster 2024-03-11T21:48:56.457598+0000 osd.4 (osd.4) 610 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.471949+0000 osd.6 (osd.6) 511 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.474473+0000 osd.6 (osd.6) 512 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:55.836417+0000 mgr.y (mgr.14550) 991 : cluster 0 pgmap v1664: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:56.271835+0000 osd.5 (osd.5) 551 : cluster 0 5.15 scrub starts 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:56.275183+0000 osd.5 (osd.5) 552 : cluster 0 5.15 scrub ok 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:56.455302+0000 osd.4 (osd.4) 609 : cluster 0 5.1c deep-scrub starts 2024-03-11T21:48:57.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:57 smithi203 bash[23654]: cluster 2024-03-11T21:48:56.457598+0000 osd.4 (osd.4) 610 : cluster 0 5.1c deep-scrub ok 2024-03-11T21:48:58.450 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[29284]: cluster 2024-03-11T21:48:57.193159+0000 osd.7 (osd.7) 650 : cluster 0 9.17 scrub starts 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[29284]: cluster 2024-03-11T21:48:57.200109+0000 osd.7 (osd.7) 651 : cluster 0 9.17 scrub ok 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[29284]: cluster 2024-03-11T21:48:57.292690+0000 osd.5 (osd.5) 553 : cluster 0 1.12 scrub starts 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[29284]: cluster 2024-03-11T21:48:57.295192+0000 osd.5 (osd.5) 554 : cluster 0 1.12 scrub ok 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[21512]: cluster 2024-03-11T21:48:57.193159+0000 osd.7 (osd.7) 650 : cluster 0 9.17 scrub starts 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[21512]: cluster 2024-03-11T21:48:57.200109+0000 osd.7 (osd.7) 651 : cluster 0 9.17 scrub ok 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[21512]: cluster 2024-03-11T21:48:57.292690+0000 osd.5 (osd.5) 553 : cluster 0 1.12 scrub starts 2024-03-11T21:48:58.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:58 smithi143 bash[21512]: cluster 2024-03-11T21:48:57.295192+0000 osd.5 (osd.5) 554 : cluster 0 1.12 scrub ok 2024-03-11T21:48:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:58 smithi203 bash[23654]: cluster 2024-03-11T21:48:57.193159+0000 osd.7 (osd.7) 650 : cluster 0 9.17 scrub starts 2024-03-11T21:48:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:58 smithi203 bash[23654]: cluster 2024-03-11T21:48:57.200109+0000 osd.7 (osd.7) 651 : cluster 0 9.17 scrub ok 2024-03-11T21:48:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:58 smithi203 bash[23654]: cluster 2024-03-11T21:48:57.292690+0000 osd.5 (osd.5) 553 : cluster 0 1.12 scrub starts 2024-03-11T21:48:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:58 smithi203 bash[23654]: cluster 2024-03-11T21:48:57.295192+0000 osd.5 (osd.5) 554 : cluster 0 1.12 scrub ok 2024-03-11T21:48:59.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[21512]: cluster 2024-03-11T21:48:57.837217+0000 mgr.y (mgr.14550) 992 : cluster 0 pgmap v1665: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.241834+0000 osd.7 (osd.7) 652 : cluster 0 3.f scrub starts 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.246217+0000 osd.7 (osd.7) 653 : cluster 0 3.f scrub ok 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.276603+0000 osd.5 (osd.5) 555 : cluster 0 5.1b scrub starts 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.278917+0000 osd.5 (osd.5) 556 : cluster 0 5.1b scrub ok 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[29284]: cluster 2024-03-11T21:48:57.837217+0000 mgr.y (mgr.14550) 992 : cluster 0 pgmap v1665: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.241834+0000 osd.7 (osd.7) 652 : cluster 0 3.f scrub starts 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.246217+0000 osd.7 (osd.7) 653 : cluster 0 3.f scrub ok 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.276603+0000 osd.5 (osd.5) 555 : cluster 0 5.1b scrub starts 2024-03-11T21:48:59.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:48:59 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.278917+0000 osd.5 (osd.5) 556 : cluster 0 5.1b scrub ok 2024-03-11T21:48:59.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:48:59.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[23654]: cluster 2024-03-11T21:48:57.837217+0000 mgr.y (mgr.14550) 992 : cluster 0 pgmap v1665: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:59.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.241834+0000 osd.7 (osd.7) 652 : cluster 0 3.f scrub starts 2024-03-11T21:48:59.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.246217+0000 osd.7 (osd.7) 653 : cluster 0 3.f scrub ok 2024-03-11T21:48:59.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.276603+0000 osd.5 (osd.5) 555 : cluster 0 5.1b scrub starts 2024-03-11T21:48:59.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:48:59 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.278917+0000 osd.5 (osd.5) 556 : cluster 0 5.1b scrub ok 2024-03-11T21:49:00.319 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:00.450 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:00 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.489053+0000 osd.6 (osd.6) 513 : cluster 0 2.11 scrub starts 2024-03-11T21:49:00.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:00 smithi143 bash[21512]: cluster 2024-03-11T21:48:58.491464+0000 osd.6 (osd.6) 514 : cluster 0 2.11 scrub ok 2024-03-11T21:49:00.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:00 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.489053+0000 osd.6 (osd.6) 513 : cluster 0 2.11 scrub starts 2024-03-11T21:49:00.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:00 smithi143 bash[29284]: cluster 2024-03-11T21:48:58.491464+0000 osd.6 (osd.6) 514 : cluster 0 2.11 scrub ok 2024-03-11T21:49:00.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:00 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.489053+0000 osd.6 (osd.6) 513 : cluster 0 2.11 scrub starts 2024-03-11T21:49:00.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:00 smithi203 bash[23654]: cluster 2024-03-11T21:48:58.491464+0000 osd.6 (osd.6) 514 : cluster 0 2.11 scrub ok 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: audit 2024-03-11T21:48:59.396301+0000 mgr.y (mgr.14550) 993 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:48:59.839096+0000 mgr.y (mgr.14550) 994 : cluster 0 pgmap v1666: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.171724+0000 osd.7 (osd.7) 654 : cluster 0 2.b deep-scrub starts 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.174068+0000 osd.7 (osd.7) 655 : cluster 0 2.b deep-scrub ok 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.263111+0000 osd.5 (osd.5) 557 : cluster 0 3.1d scrub starts 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.267530+0000 osd.5 (osd.5) 558 : cluster 0 3.1d scrub ok 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.552675+0000 osd.3 (osd.3) 635 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:00.555807+0000 osd.3 (osd.3) 636 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: audit 2024-03-11T21:48:59.396301+0000 mgr.y (mgr.14550) 993 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:01.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:48:59.839096+0000 mgr.y (mgr.14550) 994 : cluster 0 pgmap v1666: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.171724+0000 osd.7 (osd.7) 654 : cluster 0 2.b deep-scrub starts 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.174068+0000 osd.7 (osd.7) 655 : cluster 0 2.b deep-scrub ok 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.263111+0000 osd.5 (osd.5) 557 : cluster 0 3.1d scrub starts 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.267530+0000 osd.5 (osd.5) 558 : cluster 0 3.1d scrub ok 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.552675+0000 osd.3 (osd.3) 635 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:49:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:00.555807+0000 osd.3 (osd.3) 636 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:49:01.465 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: audit 2024-03-11T21:48:59.396301+0000 mgr.y (mgr.14550) 993 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:01.465 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:48:59.839096+0000 mgr.y (mgr.14550) 994 : cluster 0 pgmap v1666: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.171724+0000 osd.7 (osd.7) 654 : cluster 0 2.b deep-scrub starts 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.174068+0000 osd.7 (osd.7) 655 : cluster 0 2.b deep-scrub ok 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.263111+0000 osd.5 (osd.5) 557 : cluster 0 3.1d scrub starts 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.267530+0000 osd.5 (osd.5) 558 : cluster 0 3.1d scrub ok 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.552675+0000 osd.3 (osd.3) 635 : cluster 0 5.14 deep-scrub starts 2024-03-11T21:49:01.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:00.555807+0000 osd.3 (osd.3) 636 : cluster 0 5.14 deep-scrub ok 2024-03-11T21:49:02.450 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.175218+0000 osd.7 (osd.7) 656 : cluster 0 2.5 deep-scrub starts 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.178336+0000 osd.7 (osd.7) 657 : cluster 0 2.5 deep-scrub ok 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.308815+0000 osd.5 (osd.5) 559 : cluster 0 6.d scrub starts 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.311525+0000 osd.5 (osd.5) 560 : cluster 0 6.d scrub ok 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.451539+0000 osd.4 (osd.4) 611 : cluster 0 5.e scrub starts 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.453448+0000 osd.4 (osd.4) 612 : cluster 0 5.e scrub ok 2024-03-11T21:49:02.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.175218+0000 osd.7 (osd.7) 656 : cluster 0 2.5 deep-scrub starts 2024-03-11T21:49:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.178336+0000 osd.7 (osd.7) 657 : cluster 0 2.5 deep-scrub ok 2024-03-11T21:49:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.308815+0000 osd.5 (osd.5) 559 : cluster 0 6.d scrub starts 2024-03-11T21:49:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.311525+0000 osd.5 (osd.5) 560 : cluster 0 6.d scrub ok 2024-03-11T21:49:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.451539+0000 osd.4 (osd.4) 611 : cluster 0 5.e scrub starts 2024-03-11T21:49:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:02 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.453448+0000 osd.4 (osd.4) 612 : cluster 0 5.e scrub ok 2024-03-11T21:49:02.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.175218+0000 osd.7 (osd.7) 656 : cluster 0 2.5 deep-scrub starts 2024-03-11T21:49:02.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.178336+0000 osd.7 (osd.7) 657 : cluster 0 2.5 deep-scrub ok 2024-03-11T21:49:02.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.308815+0000 osd.5 (osd.5) 559 : cluster 0 6.d scrub starts 2024-03-11T21:49:02.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.311525+0000 osd.5 (osd.5) 560 : cluster 0 6.d scrub ok 2024-03-11T21:49:02.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.451539+0000 osd.4 (osd.4) 611 : cluster 0 5.e scrub starts 2024-03-11T21:49:02.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:02 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.453448+0000 osd.4 (osd.4) 612 : cluster 0 5.e scrub ok 2024-03-11T21:49:03.266 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2 313.2s0 313:4de65fb6:test-rados-api-smithi143-71773-113::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 565.70369107600004, 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071012540000000004, 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.267 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570422+0000", 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570425+0000", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8229999999999999e-06 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570437+0000", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2719e-05 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570471+0000", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3231000000000001e-05 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570493+0000", 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2235e-05 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570634+0000", 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014130800000000001 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.570744+0000", 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109374 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.571127+0000", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000383345 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.572914+0000", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017873469999999999 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.577431+0000", 2024-03-11T21:49:03.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045166659999999999 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:39:37.577523+0000", 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2206000000000005e-05 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(39.16,QREQUEST,e:716,from:2,reservation_nonce:13)", 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 527.68246631700003, 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012178527999999999, 2024-03-11T21:49:03.272 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591647+0000", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591649+0000", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.57e-06 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591663+0000", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3713999999999999e-05 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591709+0000", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5725e-05 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591733+0000", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4019000000000002e-05 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591931+0000", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019819400000000001 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.591950+0000", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8732999999999998e-05 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603664+0000", 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011714354999999999 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603709+0000", 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5252e-05 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:15.603825+0000", 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011596599999999999 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(2.e,QREQUEST,e:718,from:2,reservation_nonce:21)", 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 525.70755438799995, 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014035486999999999, 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566559+0000", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566560+0000", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8339999999999999e-06 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566573+0000", 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2283000000000001e-05 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566635+0000", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2661000000000002e-05 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566660+0000", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5126e-05 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566834+0000", 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000173535 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.566851+0000", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7192000000000001e-05 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580427+0000", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013575436999999999 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580473+0000", 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6029000000000003e-05 2024-03-11T21:49:03.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:17.580594+0000", 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012139 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.41921.0:805 320.19 e748/743 320:98ec0ffc:test-rados-api-smithi143-71773-125::foo:4 v 748'3, mlcod=748'1)", 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.62771511800003, 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01057773, 2024-03-11T21:49:03.281 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646398+0000", 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646399+0000", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4700000000000001e-07 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646408+0000", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4849999999999998e-06 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646424+0000", 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6155e-05 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646435+0000", 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1148999999999999e-05 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646522+0000", 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6561e-05 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.646549+0000", 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7432000000000001e-05 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656880+0000", 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010331138 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:47.656976+0000", 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5062999999999994e-05 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.285 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:859 320.a 320:50000000::::head [pgnls start_epoch 751] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e751)", 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 492.61927462099999, 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0090076349999999999, 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 859 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.286 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654838+0000", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654842+0000", 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4400000000000001e-06 2024-03-11T21:49:03.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654858+0000", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6022999999999999e-05 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654889+0000", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1619000000000003e-05 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.654914+0000", 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5058999999999998e-05 2024-03-11T21:49:03.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663527+0000", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0086127559999999992 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663583+0000", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5402000000000003e-05 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.663846+0000", 2024-03-11T21:49:03.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026333599999999997 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:863 320.e 320:70000000::::head [pgnls start_epoch 751] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e751)", 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 492.60428475100002, 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076691629999999997, 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.290 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 863 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669828+0000", 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669831+0000", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.163e-06 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.669848+0000", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6575999999999999e-05 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671377+0000", 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015295059999999999 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671406+0000", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8396000000000001e-05 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671570+0000", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016444499999999999 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.671587+0000", 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6405999999999998e-05 2024-03-11T21:49:03.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677156+0000", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00556889 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677245+0000", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9530999999999994e-05 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:50.677497+0000", 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025224999999999998 2024-03-11T21:49:03.294 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1036 321.10 321:08000000::::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e759)", 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 484.68434154900001, 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071837230000000004, 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1036 2024-03-11T21:49:03.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589771+0000", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589776+0000", 2024-03-11T21:49:03.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.25e-06 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589797+0000", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1222000000000001e-05 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589842+0000", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5491999999999997e-05 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.589870+0000", 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7206e-05 2024-03-11T21:49:03.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.590058+0000", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018823200000000001 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.590068+0000", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9799999999999993e-06 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596575+0000", 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065068879999999997 2024-03-11T21:49:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596677+0000", 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101894 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:40:58.596955+0000", 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000278559 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.299 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1112 322.a 322:50000000:test-rados-api-smithi143-71773-129:::head [pgnls start_epoch 765] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e765)", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 478.683027907, 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094745179999999995, 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1112 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591085+0000", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591088+0000", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8899999999999999e-06 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591104+0000", 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6532999999999999e-05 2024-03-11T21:49:03.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591133+0000", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.889e-05 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591150+0000", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6262000000000001e-05 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591333+0000", 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018294200000000001 2024-03-11T21:49:03.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.591351+0000", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8209000000000001e-05 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.598498+0000", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071470290000000001 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.598589+0000", 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0953999999999997e-05 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:04.600560+0000", 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001970809 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1238 322.a 322:50000000::::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 471.62522981699999, 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064194079999999997, 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.304 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1238 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648883+0000", 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648887+0000", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5899999999999999e-06 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648904+0000", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7172000000000001e-05 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648942+0000", 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7709000000000001e-05 2024-03-11T21:49:03.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.648964+0000", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2237000000000001e-05 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.649112+0000", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000147999 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.649122+0000", 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9459999999999993e-06 2024-03-11T21:49:03.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.654984+0000", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0058623640000000001 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.655072+0000", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7698e-05 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:11.655303+0000", 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000230693 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.308 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1730 325.8 325:10000000::::head [pgnls start_epoch 799] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e799)", 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 442.658111222, 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01054078, 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1730 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.309 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616002+0000", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616005+0000", 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7299999999999999e-06 2024-03-11T21:49:03.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616023+0000", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8026e-05 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616066+0000", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2592000000000001e-05 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616086+0000", 2024-03-11T21:49:03.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9596999999999999e-05 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616229+0000", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014340199999999999 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.616241+0000", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.148e-05 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626197+0000", 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0099566089999999999 2024-03-11T21:49:03.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626293+0000", 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6193999999999994e-05 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:40.626542+0000", 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024915000000000001 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.313 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:187 313.2s0 313:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 436.69363448000001, 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069664389999999996, 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580478+0000", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580480+0000", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6130000000000001e-06 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580491+0000", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.084e-05 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580511+0000", 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9763e-05 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580527+0000", 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6532e-05 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580704+0000", 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000176973 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting_for_map not empty", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.580721+0000", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7108999999999998e-05 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.587172+0000", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064507699999999998 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.587445+0000", 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027283899999999998 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(osd.4.35:188 313.2s0 313:4db57ebb:test-rados-api-smithi143-71773-137::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e805)", 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 436.66177779499998, 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.024990762999999999, 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:49:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612335+0000", 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612338+0000", 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9170000000000002e-06 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612353+0000", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5247e-05 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612392+0000", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8232000000000003e-05 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612410+0000", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8916e-05 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612584+0000", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017358200000000001 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.612980+0000", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039607899999999997 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_started", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.630531+0000", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017550530000000002 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_committed", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.634443+0000", 2024-03-11T21:49:03.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039128399999999999 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.637259+0000", 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028156259999999999 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:46.637326+0000", 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6793999999999998e-05 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:1832 326.8 326:10000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 434.67869698700002, 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073047190000000003, 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 1832 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595416+0000", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595420+0000", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0470000000000004e-06 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595440+0000", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0435999999999999e-05 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595486+0000", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5158999999999999e-05 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595513+0000", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7656000000000001e-05 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595709+0000", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019585499999999999 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.595726+0000", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6708999999999999e-05 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602402+0000", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066760949999999999 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602494+0000", 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2264999999999998e-05 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:41:48.602721+0000", 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022649700000000001 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2152 329.c 329:30000000:test-rados-api-smithi143-71773-143:::head [pgnls start_epoch 842] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e842)", 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 392.66941892400001, 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066286260000000003, 2024-03-11T21:49:03.327 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2152 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604694+0000", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604698+0000", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5599999999999998e-06 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604715+0000", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.736e-05 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604754+0000", 2024-03-11T21:49:03.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8729000000000001e-05 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604773+0000", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8879e-05 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604956+0000", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018369600000000001 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.604970+0000", 2024-03-11T21:49:03.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.375e-05 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611112+0000", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061424670000000004 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611171+0000", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8239000000000003e-05 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:42:30.611323+0000", 2024-03-11T21:49:03.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000151946 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14427.0:4141 3.13 3:cc909b6b:::obj_delete_at_hint.0000000106:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 351.68357384799998, 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.012831966, 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14427", 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/3842484601", 2024-03-11T21:49:03.332 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 4141 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590539+0000", 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590540+0000", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7000000000000004e-07 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590544+0000", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7190000000000001e-06 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590554+0000", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8390000000000002e-06 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590560+0000", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3990000000000003e-06 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590623+0000", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2971000000000004e-05 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590718+0000", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5625999999999996e-05 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.590906+0000", 2024-03-11T21:49:03.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000187868 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.591951+0000", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010447189999999999 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.602960+0000", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011009443000000001 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603241+0000", 2024-03-11T21:49:03.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028086300000000001 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603312+0000", 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0821999999999994e-05 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:11.603371+0000", 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9027000000000003e-05 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.337 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.41921.0:2933 332.9 332:90000000:%01:::head [pgnls start_epoch 871] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e871)", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 344.60823549700001, 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069345129999999998, 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.41921", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/1726746027", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 2933 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:49:03.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665877+0000", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665881+0000", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5870000000000001e-06 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665898+0000", 2024-03-11T21:49:03.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6739999999999999e-05 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665936+0000", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7932000000000002e-05 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.665961+0000", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4845999999999999e-05 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.666161+0000", 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020025899999999999 2024-03-11T21:49:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.666172+0000", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.093e-05 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672490+0000", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.006318532 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672585+0000", 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5128000000000001e-05 2024-03-11T21:49:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:18.672812+0000", 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022655900000000001 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDScrubReserve(9.c,QREQUEST,e:882,from:3,reservation_nonce:9)", 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.57135076600002, 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083183479999999997, 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.342 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702762+0000", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702764+0000", 2024-03-11T21:49:03.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1059999999999998e-06 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702774+0000", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0217000000000001e-05 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702813+0000", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8065000000000002e-05 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702827+0000", 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4823e-05 2024-03-11T21:49:03.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702974+0000", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014656399999999999 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.702990+0000", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6473e-05 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.710927+0000", 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0079361369999999994 2024-03-11T21:49:03.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.710974+0000", 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7905999999999998e-05 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:43:29.711080+0000", 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106057 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.42245.0:3 343.0 e949/948 343:0384ab8e:::oid:head v 949'1, mlcod=0'0)", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:02.842194+0000", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 240.43191864100001, 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0088916199999999994, 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.842194+0000", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.842194+0000", 2024-03-11T21:49:03.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.842195+0000", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.217e-06 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848210+0000", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060146540000000004 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848230+0000", 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9706000000000001e-05 2024-03-11T21:49:03.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848244+0000", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4583000000000001e-05 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848378+0000", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013335300000000001 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.848431+0000", 2024-03-11T21:49:03.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3176e-05 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.850963+0000", 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002532374 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:02.851086+0000", 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000122557 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.42257.0:3 344.0 344:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e955)", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:08.650209+0000", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 234.62390367200001, 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.092273321000000005, 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.42257", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2401876849", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650209+0000", 2024-03-11T21:49:03.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650209+0000", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650212+0000", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6000000000000001e-06 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650227+0000", 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5064e-05 2024-03-11T21:49:03.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650255+0000", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8209e-05 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.650282+0000", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6945999999999999e-05 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.660490+0000", 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010208204 2024-03-11T21:49:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.660504+0000", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3533000000000001e-05 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.736755+0000", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.076251359000000005 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.736927+0000", 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017137200000000001 2024-03-11T21:49:03.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.737315+0000", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00038852199999999998 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.740495+0000", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0031799760000000002 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.740743+0000", 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024823899999999998 2024-03-11T21:49:03.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.742394+0000", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001650372 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.742434+0000", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0040999999999998e-05 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:08.742483+0000", 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8884000000000002e-05 2024-03-11T21:49:03.356 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.42257.0:5 344.0 344:0384ab8e:::oid:head [write 0~128] snapc 3=[3,2] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e956)", 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-03-11T21:45:09.652901+0000", 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 233.62121219400001, 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011668556, 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.42257", 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "172.21.15.143:0/2401876849", 2024-03-11T21:49:03.357 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 5 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652901+0000", 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652901+0000", 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-03-11T21:49:03.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652903+0000", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7219999999999999e-06 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652917+0000", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3611000000000001e-05 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652943+0000", 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5711000000000002e-05 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.652964+0000", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1039999999999998e-05 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.653056+0000", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2392999999999994e-05 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op must wait for map", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.653065+0000", 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7110000000000001e-06 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.657749+0000", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046842170000000001 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.657867+0000", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117904 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.658065+0000", 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019761400000000001 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.658913+0000", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00084789299999999995 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.660532+0000", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001619592 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.664488+0000", 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039559510000000001 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.664534+0000", 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6090000000000001e-05 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-03-11T21:45:09.664569+0000", 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5107000000000001e-05 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-11T21:49:03.363 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-11T21:49:03.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:03 smithi203 bash[23654]: cluster 2024-03-11T21:49:01.840134+0000 mgr.y (mgr.14550) 995 : cluster 0 pgmap v1667: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:03.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:03 smithi203 bash[23654]: cluster 2024-03-11T21:49:02.190956+0000 osd.7 (osd.7) 658 : cluster 0 9.d scrub starts 2024-03-11T21:49:03.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:03 smithi203 bash[23654]: cluster 2024-03-11T21:49:02.194202+0000 osd.7 (osd.7) 659 : cluster 0 9.d scrub ok 2024-03-11T21:49:03.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:03 smithi203 bash[23654]: cluster 2024-03-11T21:49:02.845221+0000 osd.2 (osd.2) 529 : cluster 0 6.f deep-scrub starts 2024-03-11T21:49:03.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:03 smithi203 bash[23654]: cluster 2024-03-11T21:49:02.847697+0000 osd.2 (osd.2) 530 : cluster 0 6.f deep-scrub ok 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[21512]: cluster 2024-03-11T21:49:01.840134+0000 mgr.y (mgr.14550) 995 : cluster 0 pgmap v1667: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[21512]: cluster 2024-03-11T21:49:02.190956+0000 osd.7 (osd.7) 658 : cluster 0 9.d scrub starts 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[21512]: cluster 2024-03-11T21:49:02.194202+0000 osd.7 (osd.7) 659 : cluster 0 9.d scrub ok 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[21512]: cluster 2024-03-11T21:49:02.845221+0000 osd.2 (osd.2) 529 : cluster 0 6.f deep-scrub starts 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[21512]: cluster 2024-03-11T21:49:02.847697+0000 osd.2 (osd.2) 530 : cluster 0 6.f deep-scrub ok 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[29284]: cluster 2024-03-11T21:49:01.840134+0000 mgr.y (mgr.14550) 995 : cluster 0 pgmap v1667: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[29284]: cluster 2024-03-11T21:49:02.190956+0000 osd.7 (osd.7) 658 : cluster 0 9.d scrub starts 2024-03-11T21:49:03.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[29284]: cluster 2024-03-11T21:49:02.194202+0000 osd.7 (osd.7) 659 : cluster 0 9.d scrub ok 2024-03-11T21:49:03.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[29284]: cluster 2024-03-11T21:49:02.845221+0000 osd.2 (osd.2) 529 : cluster 0 6.f deep-scrub starts 2024-03-11T21:49:03.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:03 smithi143 bash[29284]: cluster 2024-03-11T21:49:02.847697+0000 osd.2 (osd.2) 530 : cluster 0 6.f deep-scrub ok 2024-03-11T21:49:04.042 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-11T21:49:04.042 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:49:04.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.185566+0000 osd.7 (osd.7) 660 : cluster 0 2.0 scrub starts 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.187352+0000 osd.7 (osd.7) 661 : cluster 0 2.0 scrub ok 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.595829+0000 osd.3 (osd.3) 637 : cluster 0 6.b scrub starts 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.598244+0000 osd.3 (osd.3) 638 : cluster 0 6.b scrub ok 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.185566+0000 osd.7 (osd.7) 660 : cluster 0 2.0 scrub starts 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.187352+0000 osd.7 (osd.7) 661 : cluster 0 2.0 scrub ok 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.595829+0000 osd.3 (osd.3) 637 : cluster 0 6.b scrub starts 2024-03-11T21:49:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:04 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.598244+0000 osd.3 (osd.3) 638 : cluster 0 6.b scrub ok 2024-03-11T21:49:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:04 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.185566+0000 osd.7 (osd.7) 660 : cluster 0 2.0 scrub starts 2024-03-11T21:49:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:04 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.187352+0000 osd.7 (osd.7) 661 : cluster 0 2.0 scrub ok 2024-03-11T21:49:04.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:04 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.595829+0000 osd.3 (osd.3) 637 : cluster 0 6.b scrub starts 2024-03-11T21:49:04.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:04 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.598244+0000 osd.3 (osd.3) 638 : cluster 0 6.b scrub ok 2024-03-11T21:49:05.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.527425+0000 osd.6 (osd.6) 515 : cluster 0 1.6 scrub starts 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.530240+0000 osd.6 (osd.6) 516 : cluster 0 1.6 scrub ok 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:03.841987+0000 mgr.y (mgr.14550) 996 : cluster 0 pgmap v1668: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:04.234463+0000 osd.0 (osd.0) 642 : cluster 0 9.9 scrub starts 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:04.236538+0000 osd.0 (osd.0) 643 : cluster 0 9.9 scrub ok 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:04.576707+0000 osd.3 (osd.3) 639 : cluster 0 9.1b scrub starts 2024-03-11T21:49:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:05 smithi203 bash[23654]: cluster 2024-03-11T21:49:04.582658+0000 osd.3 (osd.3) 640 : cluster 0 9.1b scrub ok 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.527425+0000 osd.6 (osd.6) 515 : cluster 0 1.6 scrub starts 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.530240+0000 osd.6 (osd.6) 516 : cluster 0 1.6 scrub ok 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:03.841987+0000 mgr.y (mgr.14550) 996 : cluster 0 pgmap v1668: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:04.234463+0000 osd.0 (osd.0) 642 : cluster 0 9.9 scrub starts 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:04.236538+0000 osd.0 (osd.0) 643 : cluster 0 9.9 scrub ok 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:04.576707+0000 osd.3 (osd.3) 639 : cluster 0 9.1b scrub starts 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[21512]: cluster 2024-03-11T21:49:04.582658+0000 osd.3 (osd.3) 640 : cluster 0 9.1b scrub ok 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.527425+0000 osd.6 (osd.6) 515 : cluster 0 1.6 scrub starts 2024-03-11T21:49:05.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.530240+0000 osd.6 (osd.6) 516 : cluster 0 1.6 scrub ok 2024-03-11T21:49:05.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:03.841987+0000 mgr.y (mgr.14550) 996 : cluster 0 pgmap v1668: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:05.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:04.234463+0000 osd.0 (osd.0) 642 : cluster 0 9.9 scrub starts 2024-03-11T21:49:05.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:04.236538+0000 osd.0 (osd.0) 643 : cluster 0 9.9 scrub ok 2024-03-11T21:49:05.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:04.576707+0000 osd.3 (osd.3) 639 : cluster 0 9.1b scrub starts 2024-03-11T21:49:05.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:05 smithi143 bash[29284]: cluster 2024-03-11T21:49:04.582658+0000 osd.3 (osd.3) 640 : cluster 0 9.1b scrub ok 2024-03-11T21:49:06.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:06 smithi203 bash[23654]: cluster 2024-03-11T21:49:05.204339+0000 osd.0 (osd.0) 644 : cluster 0 9.1d scrub starts 2024-03-11T21:49:06.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:06 smithi203 bash[23654]: cluster 2024-03-11T21:49:05.210036+0000 osd.0 (osd.0) 645 : cluster 0 9.1d scrub ok 2024-03-11T21:49:06.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:06 smithi203 bash[23654]: cluster 2024-03-11T21:49:05.439382+0000 osd.4 (osd.4) 613 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:49:06.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:06 smithi203 bash[23654]: cluster 2024-03-11T21:49:05.442115+0000 osd.4 (osd.4) 614 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:49:06.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[29284]: cluster 2024-03-11T21:49:05.204339+0000 osd.0 (osd.0) 644 : cluster 0 9.1d scrub starts 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[29284]: cluster 2024-03-11T21:49:05.210036+0000 osd.0 (osd.0) 645 : cluster 0 9.1d scrub ok 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[29284]: cluster 2024-03-11T21:49:05.439382+0000 osd.4 (osd.4) 613 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[29284]: cluster 2024-03-11T21:49:05.442115+0000 osd.4 (osd.4) 614 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[21512]: cluster 2024-03-11T21:49:05.204339+0000 osd.0 (osd.0) 644 : cluster 0 9.1d scrub starts 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[21512]: cluster 2024-03-11T21:49:05.210036+0000 osd.0 (osd.0) 645 : cluster 0 9.1d scrub ok 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[21512]: cluster 2024-03-11T21:49:05.439382+0000 osd.4 (osd.4) 613 : cluster 0 6.17 deep-scrub starts 2024-03-11T21:49:06.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:06 smithi143 bash[21512]: cluster 2024-03-11T21:49:05.442115+0000 osd.4 (osd.4) 614 : cluster 0 6.17 deep-scrub ok 2024-03-11T21:49:07.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:07 smithi203 bash[23654]: cluster 2024-03-11T21:49:05.843097+0000 mgr.y (mgr.14550) 997 : cluster 0 pgmap v1669: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T21:49:07.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:07 smithi203 bash[23654]: cluster 2024-03-11T21:49:06.173391+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-03-11T21:49:07.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:07 smithi203 bash[23654]: cluster 2024-03-11T21:49:06.177068+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-03-11T21:49:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:07 smithi203 bash[23654]: cluster 2024-03-11T21:49:06.410900+0000 osd.4 (osd.4) 615 : cluster 0 3.3 scrub starts 2024-03-11T21:49:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:07 smithi203 bash[23654]: cluster 2024-03-11T21:49:06.415757+0000 osd.4 (osd.4) 616 : cluster 0 3.3 scrub ok 2024-03-11T21:49:07.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[29284]: cluster 2024-03-11T21:49:05.843097+0000 mgr.y (mgr.14550) 997 : cluster 0 pgmap v1669: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[29284]: cluster 2024-03-11T21:49:06.173391+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[29284]: cluster 2024-03-11T21:49:06.177068+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[29284]: cluster 2024-03-11T21:49:06.410900+0000 osd.4 (osd.4) 615 : cluster 0 3.3 scrub starts 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[29284]: cluster 2024-03-11T21:49:06.415757+0000 osd.4 (osd.4) 616 : cluster 0 3.3 scrub ok 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[21512]: cluster 2024-03-11T21:49:05.843097+0000 mgr.y (mgr.14550) 997 : cluster 0 pgmap v1669: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[21512]: cluster 2024-03-11T21:49:06.173391+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[21512]: cluster 2024-03-11T21:49:06.177068+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[21512]: cluster 2024-03-11T21:49:06.410900+0000 osd.4 (osd.4) 615 : cluster 0 3.3 scrub starts 2024-03-11T21:49:07.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:07 smithi143 bash[21512]: cluster 2024-03-11T21:49:06.415757+0000 osd.4 (osd.4) 616 : cluster 0 3.3 scrub ok 2024-03-11T21:49:08.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:08 smithi203 bash[23654]: cluster 2024-03-11T21:49:07.271720+0000 osd.5 (osd.5) 561 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:49:08.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:08 smithi203 bash[23654]: cluster 2024-03-11T21:49:07.274254+0000 osd.5 (osd.5) 562 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:49:08.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:08 smithi203 bash[23654]: audit 2024-03-11T21:49:08.219022+0000 mon.c (mon.2) 1418 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[21512]: cluster 2024-03-11T21:49:07.271720+0000 osd.5 (osd.5) 561 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:49:08.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[21512]: cluster 2024-03-11T21:49:07.274254+0000 osd.5 (osd.5) 562 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:49:08.700 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[21512]: audit 2024-03-11T21:49:08.219022+0000 mon.c (mon.2) 1418 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[29284]: cluster 2024-03-11T21:49:07.271720+0000 osd.5 (osd.5) 561 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:49:08.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[29284]: cluster 2024-03-11T21:49:07.274254+0000 osd.5 (osd.5) 562 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:49:08.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:08 smithi143 bash[29284]: audit 2024-03-11T21:49:08.219022+0000 mon.c (mon.2) 1418 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.827 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:09.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:49:09.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: cluster 2024-03-11T21:49:07.843906+0000 mgr.y (mgr.14550) 998 : cluster 0 pgmap v1670: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:09.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: cluster 2024-03-11T21:49:08.046318+0000 osd.1 (osd.1) 441 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:49:09.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: cluster 2024-03-11T21:49:08.052446+0000 osd.1 (osd.1) 442 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: audit 2024-03-11T21:49:08.247911+0000 mon.c (mon.2) 1419 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: cluster 2024-03-11T21:49:08.509708+0000 osd.3 (osd.3) 641 : cluster 0 6.c scrub starts 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: cluster 2024-03-11T21:49:08.511830+0000 osd.3 (osd.3) 642 : cluster 0 6.c scrub ok 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: audit 2024-03-11T21:49:08.963378+0000 mon.c (mon.2) 1420 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: audit 2024-03-11T21:49:08.965294+0000 mon.c (mon.2) 1421 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:09.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:09 smithi203 bash[23654]: audit 2024-03-11T21:49:08.975968+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: cluster 2024-03-11T21:49:07.843906+0000 mgr.y (mgr.14550) 998 : cluster 0 pgmap v1670: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: cluster 2024-03-11T21:49:08.046318+0000 osd.1 (osd.1) 441 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: cluster 2024-03-11T21:49:08.052446+0000 osd.1 (osd.1) 442 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: audit 2024-03-11T21:49:08.247911+0000 mon.c (mon.2) 1419 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: cluster 2024-03-11T21:49:08.509708+0000 osd.3 (osd.3) 641 : cluster 0 6.c scrub starts 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: cluster 2024-03-11T21:49:08.511830+0000 osd.3 (osd.3) 642 : cluster 0 6.c scrub ok 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: audit 2024-03-11T21:49:08.963378+0000 mon.c (mon.2) 1420 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: audit 2024-03-11T21:49:08.965294+0000 mon.c (mon.2) 1421 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:09.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[29284]: audit 2024-03-11T21:49:08.975968+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: cluster 2024-03-11T21:49:07.843906+0000 mgr.y (mgr.14550) 998 : cluster 0 pgmap v1670: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: cluster 2024-03-11T21:49:08.046318+0000 osd.1 (osd.1) 441 : cluster 0 3.19 deep-scrub starts 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: cluster 2024-03-11T21:49:08.052446+0000 osd.1 (osd.1) 442 : cluster 0 3.19 deep-scrub ok 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: audit 2024-03-11T21:49:08.247911+0000 mon.c (mon.2) 1419 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: cluster 2024-03-11T21:49:08.509708+0000 osd.3 (osd.3) 641 : cluster 0 6.c scrub starts 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: cluster 2024-03-11T21:49:08.511830+0000 osd.3 (osd.3) 642 : cluster 0 6.c scrub ok 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: audit 2024-03-11T21:49:08.963378+0000 mon.c (mon.2) 1420 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: audit 2024-03-11T21:49:08.965294+0000 mon.c (mon.2) 1421 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:09.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:09 smithi143 bash[21512]: audit 2024-03-11T21:49:08.975968+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:49:10.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:10 smithi203 bash[23654]: cluster 2024-03-11T21:49:09.048995+0000 osd.1 (osd.1) 443 : cluster 0 2.a scrub starts 2024-03-11T21:49:10.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:10 smithi203 bash[23654]: cluster 2024-03-11T21:49:09.050554+0000 osd.1 (osd.1) 444 : cluster 0 2.a scrub ok 2024-03-11T21:49:10.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:10 smithi203 bash[23654]: audit 2024-03-11T21:49:09.407619+0000 mgr.y (mgr.14550) 999 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:10.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:10 smithi203 bash[23654]: cluster 2024-03-11T21:49:09.845746+0000 mgr.y (mgr.14550) 1000 : cluster 0 pgmap v1671: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:10.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[29284]: cluster 2024-03-11T21:49:09.048995+0000 osd.1 (osd.1) 443 : cluster 0 2.a scrub starts 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[29284]: cluster 2024-03-11T21:49:09.050554+0000 osd.1 (osd.1) 444 : cluster 0 2.a scrub ok 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[29284]: audit 2024-03-11T21:49:09.407619+0000 mgr.y (mgr.14550) 999 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[29284]: cluster 2024-03-11T21:49:09.845746+0000 mgr.y (mgr.14550) 1000 : cluster 0 pgmap v1671: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[21512]: cluster 2024-03-11T21:49:09.048995+0000 osd.1 (osd.1) 443 : cluster 0 2.a scrub starts 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[21512]: cluster 2024-03-11T21:49:09.050554+0000 osd.1 (osd.1) 444 : cluster 0 2.a scrub ok 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[21512]: audit 2024-03-11T21:49:09.407619+0000 mgr.y (mgr.14550) 999 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:10.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:10 smithi143 bash[21512]: cluster 2024-03-11T21:49:09.845746+0000 mgr.y (mgr.14550) 1000 : cluster 0 pgmap v1671: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:11.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:11 smithi203 bash[23654]: cluster 2024-03-11T21:49:10.163805+0000 osd.0 (osd.0) 648 : cluster 0 3.6 scrub starts 2024-03-11T21:49:11.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:11 smithi203 bash[23654]: cluster 2024-03-11T21:49:10.167624+0000 osd.0 (osd.0) 649 : cluster 0 3.6 scrub ok 2024-03-11T21:49:11.700 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:11 smithi143 bash[29284]: cluster 2024-03-11T21:49:10.163805+0000 osd.0 (osd.0) 648 : cluster 0 3.6 scrub starts 2024-03-11T21:49:11.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:11 smithi143 bash[29284]: cluster 2024-03-11T21:49:10.167624+0000 osd.0 (osd.0) 649 : cluster 0 3.6 scrub ok 2024-03-11T21:49:11.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:11 smithi143 bash[21512]: cluster 2024-03-11T21:49:10.163805+0000 osd.0 (osd.0) 648 : cluster 0 3.6 scrub starts 2024-03-11T21:49:11.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:11 smithi143 bash[21512]: cluster 2024-03-11T21:49:10.167624+0000 osd.0 (osd.0) 649 : cluster 0 3.6 scrub ok 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/2961172448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060101ec0 msgr2=0x7f7060102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- 172.21.15.143:0/2961172448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060101ec0 0x7f7060102340 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f7050009990 tx=0x7f7050009ca0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/2961172448 shutdown_connections 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- 172.21.15.143:0/2961172448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060107f30 0x7f7060108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- 172.21.15.143:0/2961172448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060102880 0x7f706010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- 172.21.15.143:0/2961172448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060101ec0 0x7f7060102340 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.993 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/2961172448 >> 172.21.15.143:0/2961172448 conn(0x7f7060078eb0 msgr2=0x7f70600792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/2961172448 shutdown_connections 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/2961172448 wait complete. 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 librados: starting msgr at 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 librados: starting objecter 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 Processor -- start 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- start start 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 librados: setting wanted keys 2024-03-11T21:49:11.994 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 librados: calling monclient init 2024-03-11T21:49:11.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:11.995 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060102880 0x7f7060074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70601116f0 con 0x7f7060101ec0 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7060111570 con 0x7f7060102880 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7060111870 con 0x7f7060107f30 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7065ecb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60240/0 (socket says 172.21.15.143:60240) 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7065ecb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41394/0 (socket says 172.21.15.143:41394) 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 -- 172.21.15.143:0/718197423 learned_addr learned my addr 172.21.15.143:0/718197423 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 -- 172.21.15.143:0/718197423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060102880 msgr2=0x7f7060074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060102880 0x7f7060074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 -- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 msgr2=0x7f70600740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:11.996 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 -- 172.21.15.143:0/718197423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70601a6180 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7065ecb700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f70666cc700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f705800b970 tx=0x7f705800bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7056ffd700 10 client.?.objecter ms_handle_connect 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7056ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058019070 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f705800cdb0 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70601a6470 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058014e40 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 10 client.42419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70601a69b0 con 0x7f7060107f30 2024-03-11T21:49:11.997 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.001+0000 7f706812f700 1 librados: init done 2024-03-11T21:49:11.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706004faa0 con 0x7f7060107f30 2024-03-11T21:49:11.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f705801c070 con 0x7f7060107f30 2024-03-11T21:49:11.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 0x7f704c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:11.998 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1009..1009 src has 954..1009) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f705808dae0 con 0x7f7060107f30 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 10 client.42419.objecter ms_dispatch 0x7f7060001550 osd_map(1009..1009 src has 954..1009) v4 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 3 client.42419.objecter handle_osd_map got epochs [1009,1009] > 0 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 3 client.42419.objecter handle_osd_map decoding full epoch 1009 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7065ecb700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 0x7f704c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 20 client.42419.objecter dump_active .. 0 homeless 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7065ecb700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 0x7f704c071ce0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f705c009680 tx=0x7f705c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:11.999 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.005+0000 7f7056ffd700 10 client.42419.objecter ms_handle_connect 0x7f704c06f820 2024-03-11T21:49:12.002 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.009+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f705805e540 con 0x7f7060107f30 2024-03-11T21:49:12.200 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:12.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.221+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70601a6cd0 con 0x7f7060107f30 2024-03-11T21:49:12.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f7056ffd700 1 -- 172.21.15.143:0/718197423 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1009) v1 ==== 74+0+26463 (secure 0 0 0) 0x7f7058062b00 con 0x7f7060107f30 2024-03-11T21:49:12.220 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:49:12.221 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1009,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","created":"2024-03-11T21:12:47.384719+0000","modified":"2024-03-11T21:48:55.144865+0000","last_up_change":"2024-03-11T21:31:36.320182+0000","last_in_change":"2024-03-11T21:21:55.456354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":117,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:08.036690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"970","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"895","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:23:01.432928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:23:03.441476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-03-11T21:23:04.441432+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:23:05.388543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:23:07.642696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-11T21:28:36.978109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6803","nonce":2903432023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6805","nonce":2903432023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6809","nonce":2903432023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6807","nonce":2903432023}]},"public_addr":"172.21.15.143:6803/2903432023","cluster_addr":"172.21.15.143:6805/2903432023","heartbeat_back_addr":"172.21.15.143:6809/2903432023","heartbeat_front_addr":"172.21.15.143:6807/2903432023","state":["exists","up"]},{"osd":1,"uuid":"2948dc44-68d3-4b3d-92bf-2b0c46374f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":246,"up_from":295,"up_thru":963,"down_at":247,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6811","nonce":4235040845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6813","nonce":4235040845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6817","nonce":4235040845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6815","nonce":4235040845}]},"public_addr":"172.21.15.143:6811/4235040845","cluster_addr":"172.21.15.143:6813/4235040845","heartbeat_back_addr":"172.21.15.143:6817/4235040845","heartbeat_front_addr":"172.21.15.143:6815/4235040845","state":["exists","up"]},{"osd":2,"uuid":"7217589a-5835-407b-91a7-f5c055a48f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6819","nonce":2592131625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6821","nonce":2592131625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6825","nonce":2592131625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6823","nonce":2592131625}]},"public_addr":"172.21.15.143:6819/2592131625","cluster_addr":"172.21.15.143:6821/2592131625","heartbeat_back_addr":"172.21.15.143:6825/2592131625","heartbeat_front_addr":"172.21.15.143:6823/2592131625","state":["exists","up"]},{"osd":3,"uuid":"a3e7177f-a5a0-44ad-a65d-9aeac3a680cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6827","nonce":2146571980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6829","nonce":2146571980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6833","nonce":2146571980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6831","nonce":2146571980}]},"public_addr":"172.21.15.143:6827/2146571980","cluster_addr":"172.21.15.143:6829/2146571980","heartbeat_back_addr":"172.21.15.143:6833/2146571980","heartbeat_front_addr":"172.21.15.143:6831/2146571980","state":["exists","up"]},{"osd":4,"uuid":"d7b9e088-d377-4335-8863-11d581828f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6801","nonce":2090835691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6803","nonce":2090835691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6807","nonce":2090835691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6805","nonce":2090835691}]},"public_addr":"172.21.15.203:6801/2090835691","cluster_addr":"172.21.15.203:6803/2090835691","heartbeat_back_addr":"172.21.15.203:6807/2090835691","heartbeat_front_addr":"172.21.15.203:6805/2090835691","state":["exists","up"]},{"osd":5,"uuid":"b3b2a75c-68a5-4e00-8068-622e1ebbaa01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1002,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6809","nonce":2432038345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6811","nonce":2432038345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6815","nonce":2432038345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6813","nonce":2432038345}]},"public_addr":"172.21.15.203:6809/2432038345","cluster_addr":"172.21.15.203:6811/2432038345","heartbeat_back_addr":"172.21.15.203:6815/2432038345","heartbeat_front_addr":"172.21.15.203:6813/2432038345","state":["exists","up"]},{"osd":6,"uuid":"ceffb234-db0c-4ffa-a142-9b201df9b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6817","nonce":513748834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6819","nonce":513748834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6823","nonce":513748834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6821","nonce":513748834}]},"public_addr":"172.21.15.203:6817/513748834","cluster_addr":"172.21.15.203:6819/513748834","heartbeat_back_addr":"172.21.15.203:6823/513748834","heartbeat_front_addr":"172.21.15.203:6821/513748834","state":["exists","up"]},{"osd":7,"uuid":"88b53171-69d2-4922-a787-e63d44d01c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6825","nonce":1251497952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6827","nonce":1251497952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6831","nonce":1251497952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6829","nonce":1251497952}]},"public_addr":"172.21.15.203:6825/1251497952","cluster_addr":"172.21.15.203:6827/1251497952","heartbeat_back_addr":"172.21.15.203:6831/1251497952","heartbeat_front_addr":"172.21.15.203:6829/1251497952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:54.295800+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:52.065895+0000","dead_epoch":247},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:34.394010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:21.161452+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:47:51.173339+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:51.591400+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:47:49.626930+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:03.212207+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.7","osds":[0,4,2]}],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"9.10","mappings":[{"from":7,"to":2}]},{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"9.1d","mappings":[{"from":7,"to":2}]},{"pgid":"9.1e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6801/3638739149":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4274199357":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/3217951344":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/3912416566":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/906834629":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2791822486":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2190104670":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4171894415":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2845155916":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2913593434":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/143435387":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2930358289":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6800/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3638739149":"2024-03-12T21:23:57.022184+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":28,"snaps":[{"begin":5,"length":2}]},{"pool":35,"snaps":[{"begin":1,"length":4}]},{"pool":39,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":137,"snaps":[{"begin":2,"length":3}]},{"pool":156,"snaps":[{"begin":2,"length":1}]},{"pool":165,"snaps":[{"begin":2,"length":4}]},{"pool":173,"snaps":[{"begin":3,"length":4}]},{"pool":178,"snaps":[{"begin":2,"length":1}]},{"pool":185,"snaps":[{"begin":3,"length":2}]},{"pool":198,"snaps":[{"begin":4,"length":4}]},{"pool":210,"snaps":[{"begin":7,"length":1}]},{"pool":230,"snaps":[{"begin":8,"length":1}]},{"pool":241,"snaps":[{"begin":10,"length":1}]},{"pool":244,"snaps":[{"begin":11,"length":2}]},{"pool":256,"snaps":[{"begin":14,"length":1}]},{"pool":316,"snaps":[{"begin":2,"length":1}]},{"pool":317,"snaps":[{"begin":3,"length":1}]},{"pool":320,"snaps":[{"begin":4,"length":1}]},{"pool":324,"snaps":[{"begin":6,"length":1}]},{"pool":336,"snaps":[{"begin":1,"length":1}]},{"pool":338,"snaps":[{"begin":1,"length":1}]},{"pool":339,"snaps":[{"begin":1,"length":2}]},{"pool":340,"snaps":[{"begin":3,"length":1}]},{"pool":341,"snaps":[{"begin":2,"length":2}]},{"pool":342,"snaps":[{"begin":2,"length":3}]},{"pool":343,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:49:12.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 10 librados: watch_flush enter 2024-03-11T21:49:12.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 10 librados: watch_flush exit 2024-03-11T21:49:12.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 20 client.42419.objecter shutdown clearing up homeless session... 2024-03-11T21:49:12.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 10 client.42419.objecter successfully canceled tick 2024-03-11T21:49:12.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 msgr2=0x7f704c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:12.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 0x7f704c071ce0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f705c009680 tx=0x7f705c009260 comp rx=0 tx=0).stop 2024-03-11T21:49:12.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 msgr2=0x7f7060074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:12.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f705800b970 tx=0x7f705800bc80 comp rx=0 tx=0).stop 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 shutdown_connections 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f704c06f820 0x7f704c071ce0 unknown :-1 s=CLOSED pgs=3513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7060101ec0 0x7f70600740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7060107f30 0x7f7060074b60 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 --2- 172.21.15.143:0/718197423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060102880 0x7f7060074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 >> 172.21.15.143:0/718197423 conn(0x7f7060078eb0 msgr2=0x7f7060108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 shutdown_connections 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 -- 172.21.15.143:0/718197423 wait complete. 2024-03-11T21:49:12.224 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:12.229+0000 7f706812f700 1 librados: shutdown 2024-03-11T21:49:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:11.575223+0000 osd.6 (osd.6) 517 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:11.575600+0000 osd.6 (osd.6) 518 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:13.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:11.794770+0000 osd.2 (osd.2) 531 : cluster 0 5.a scrub starts 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:11.796827+0000 osd.2 (osd.2) 532 : cluster 0 5.a scrub ok 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:11.846780+0000 mgr.y (mgr.14550) 1001 : cluster 0 pgmap v1672: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:12.185128+0000 osd.5 (osd.5) 563 : cluster 0 1.1 scrub starts 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:12.187452+0000 osd.5 (osd.5) 564 : cluster 0 1.1 scrub ok 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: audit 2024-03-11T21:49:12.227478+0000 mon.c (mon.2) 1422 : audit 0 from='client.? 172.21.15.143:0/718197423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:12 smithi203 bash[23654]: cluster 2024-03-11T21:49:12.261807+0000 mon.a (mon.0) 4166 : cluster 0 osdmap e1010: 8 total, 8 up, 8 in 2024-03-11T21:49:13.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:11.575223+0000 osd.6 (osd.6) 517 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:13.089 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:11.575600+0000 osd.6 (osd.6) 518 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:11.794770+0000 osd.2 (osd.2) 531 : cluster 0 5.a scrub starts 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:11.796827+0000 osd.2 (osd.2) 532 : cluster 0 5.a scrub ok 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:11.846780+0000 mgr.y (mgr.14550) 1001 : cluster 0 pgmap v1672: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:12.185128+0000 osd.5 (osd.5) 563 : cluster 0 1.1 scrub starts 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:12.187452+0000 osd.5 (osd.5) 564 : cluster 0 1.1 scrub ok 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: audit 2024-03-11T21:49:12.227478+0000 mon.c (mon.2) 1422 : audit 0 from='client.? 172.21.15.143:0/718197423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.090 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[21512]: cluster 2024-03-11T21:49:12.261807+0000 mon.a (mon.0) 4166 : cluster 0 osdmap e1010: 8 total, 8 up, 8 in 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:11.575223+0000 osd.6 (osd.6) 517 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:11.575600+0000 osd.6 (osd.6) 518 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:11.794770+0000 osd.2 (osd.2) 531 : cluster 0 5.a scrub starts 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:11.796827+0000 osd.2 (osd.2) 532 : cluster 0 5.a scrub ok 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:11.846780+0000 mgr.y (mgr.14550) 1001 : cluster 0 pgmap v1672: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:13.091 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:12.185128+0000 osd.5 (osd.5) 563 : cluster 0 1.1 scrub starts 2024-03-11T21:49:13.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:12.187452+0000 osd.5 (osd.5) 564 : cluster 0 1.1 scrub ok 2024-03-11T21:49:13.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: audit 2024-03-11T21:49:12.227478+0000 mon.c (mon.2) 1422 : audit 0 from='client.? 172.21.15.143:0/718197423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.092 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:12 smithi143 bash[29284]: cluster 2024-03-11T21:49:12.261807+0000 mon.a (mon.0) 4166 : cluster 0 osdmap e1010: 8 total, 8 up, 8 in 2024-03-11T21:49:13.094 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:49:13.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:49:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:13 smithi203 bash[23654]: cluster 2024-03-11T21:49:12.152835+0000 osd.0 (osd.0) 650 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:49:14.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:13 smithi203 bash[23654]: cluster 2024-03-11T21:49:12.155426+0000 osd.0 (osd.0) 651 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:49:14.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:13 smithi143 bash[29284]: cluster 2024-03-11T21:49:12.152835+0000 osd.0 (osd.0) 650 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:49:14.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:13 smithi143 bash[29284]: cluster 2024-03-11T21:49:12.155426+0000 osd.0 (osd.0) 651 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:49:14.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:13 smithi143 bash[21512]: cluster 2024-03-11T21:49:12.152835+0000 osd.0 (osd.0) 650 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:49:14.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:13 smithi143 bash[21512]: cluster 2024-03-11T21:49:12.155426+0000 osd.0 (osd.0) 651 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:49:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.526026+0000 osd.3 (osd.3) 643 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:49:15.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.528303+0000 osd.3 (osd.3) 644 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.559355+0000 osd.4 (osd.4) 617 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.559845+0000 osd.4 (osd.4) 618 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.741938+0000 mon.a (mon.0) 4167 : cluster 0 osdmap e1011: 8 total, 8 up, 8 in 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:13.847589+0000 mgr.y (mgr.14550) 1002 : cluster 0 pgmap v1675: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:14.016785+0000 osd.1 (osd.1) 445 : cluster 0 9.1f deep-scrub starts 2024-03-11T21:49:15.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:14 smithi203 bash[23654]: cluster 2024-03-11T21:49:14.028503+0000 osd.1 (osd.1) 446 : cluster 0 9.1f deep-scrub ok 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.526026+0000 osd.3 (osd.3) 643 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.528303+0000 osd.3 (osd.3) 644 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.559355+0000 osd.4 (osd.4) 617 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.559845+0000 osd.4 (osd.4) 618 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.741938+0000 mon.a (mon.0) 4167 : cluster 0 osdmap e1011: 8 total, 8 up, 8 in 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:13.847589+0000 mgr.y (mgr.14550) 1002 : cluster 0 pgmap v1675: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:14.016785+0000 osd.1 (osd.1) 445 : cluster 0 9.1f deep-scrub starts 2024-03-11T21:49:15.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[21512]: cluster 2024-03-11T21:49:14.028503+0000 osd.1 (osd.1) 446 : cluster 0 9.1f deep-scrub ok 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.526026+0000 osd.3 (osd.3) 643 : cluster 0 2.1e deep-scrub starts 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.528303+0000 osd.3 (osd.3) 644 : cluster 0 2.1e deep-scrub ok 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.559355+0000 osd.4 (osd.4) 617 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.559845+0000 osd.4 (osd.4) 618 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.741938+0000 mon.a (mon.0) 4167 : cluster 0 osdmap e1011: 8 total, 8 up, 8 in 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:13.847589+0000 mgr.y (mgr.14550) 1002 : cluster 0 pgmap v1675: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:14.016785+0000 osd.1 (osd.1) 445 : cluster 0 9.1f deep-scrub starts 2024-03-11T21:49:15.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:14 smithi143 bash[29284]: cluster 2024-03-11T21:49:14.028503+0000 osd.1 (osd.1) 446 : cluster 0 9.1f deep-scrub ok 2024-03-11T21:49:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:15 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.009045+0000 osd.1 (osd.1) 447 : cluster 0 5.c scrub starts 2024-03-11T21:49:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:15 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.011460+0000 osd.1 (osd.1) 448 : cluster 0 5.c scrub ok 2024-03-11T21:49:16.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:15 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.144073+0000 osd.5 (osd.5) 565 : cluster 0 9.7 scrub starts 2024-03-11T21:49:16.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:15 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.152953+0000 osd.5 (osd.5) 566 : cluster 0 9.7 scrub ok 2024-03-11T21:49:16.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.009045+0000 osd.1 (osd.1) 447 : cluster 0 5.c scrub starts 2024-03-11T21:49:16.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.011460+0000 osd.1 (osd.1) 448 : cluster 0 5.c scrub ok 2024-03-11T21:49:16.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.144073+0000 osd.5 (osd.5) 565 : cluster 0 9.7 scrub starts 2024-03-11T21:49:16.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.152953+0000 osd.5 (osd.5) 566 : cluster 0 9.7 scrub ok 2024-03-11T21:49:16.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.009045+0000 osd.1 (osd.1) 447 : cluster 0 5.c scrub starts 2024-03-11T21:49:16.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.011460+0000 osd.1 (osd.1) 448 : cluster 0 5.c scrub ok 2024-03-11T21:49:16.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.144073+0000 osd.5 (osd.5) 565 : cluster 0 9.7 scrub starts 2024-03-11T21:49:16.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:15 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.152953+0000 osd.5 (osd.5) 566 : cluster 0 9.7 scrub ok 2024-03-11T21:49:17.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.172013+0000 osd.0 (osd.0) 652 : cluster 0 6.0 scrub starts 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.174263+0000 osd.0 (osd.0) 653 : cluster 0 6.0 scrub ok 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.410667+0000 osd.7 (osd.7) 662 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.411135+0000 osd.7 (osd.7) 663 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.774343+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e1012: 8 total, 8 up, 8 in 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:15.848354+0000 mgr.y (mgr.14550) 1003 : cluster 0 pgmap v1677: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:16.173542+0000 osd.7 (osd.7) 664 : cluster 0 3.d deep-scrub starts 2024-03-11T21:49:17.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:16 smithi203 bash[23654]: cluster 2024-03-11T21:49:16.179055+0000 osd.7 (osd.7) 665 : cluster 0 3.d deep-scrub ok 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.172013+0000 osd.0 (osd.0) 652 : cluster 0 6.0 scrub starts 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.174263+0000 osd.0 (osd.0) 653 : cluster 0 6.0 scrub ok 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.410667+0000 osd.7 (osd.7) 662 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.411135+0000 osd.7 (osd.7) 663 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.774343+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e1012: 8 total, 8 up, 8 in 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:15.848354+0000 mgr.y (mgr.14550) 1003 : cluster 0 pgmap v1677: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:16.173542+0000 osd.7 (osd.7) 664 : cluster 0 3.d deep-scrub starts 2024-03-11T21:49:17.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[21512]: cluster 2024-03-11T21:49:16.179055+0000 osd.7 (osd.7) 665 : cluster 0 3.d deep-scrub ok 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.172013+0000 osd.0 (osd.0) 652 : cluster 0 6.0 scrub starts 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.174263+0000 osd.0 (osd.0) 653 : cluster 0 6.0 scrub ok 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.410667+0000 osd.7 (osd.7) 662 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.411135+0000 osd.7 (osd.7) 663 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.774343+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e1012: 8 total, 8 up, 8 in 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:15.848354+0000 mgr.y (mgr.14550) 1003 : cluster 0 pgmap v1677: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:16.173542+0000 osd.7 (osd.7) 664 : cluster 0 3.d deep-scrub starts 2024-03-11T21:49:17.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:16 smithi143 bash[29284]: cluster 2024-03-11T21:49:16.179055+0000 osd.7 (osd.7) 665 : cluster 0 3.d deep-scrub ok 2024-03-11T21:49:17.875 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[21512]: cluster 2024-03-11T21:49:17.849137+0000 mgr.y (mgr.14550) 1004 : cluster 0 pgmap v1678: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[21512]: cluster 2024-03-11T21:49:17.951499+0000 osd.1 (osd.1) 449 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:49:19.152 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[21512]: cluster 2024-03-11T21:49:17.955685+0000 osd.1 (osd.1) 450 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:49:19.152 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[29284]: cluster 2024-03-11T21:49:17.849137+0000 mgr.y (mgr.14550) 1004 : cluster 0 pgmap v1678: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:19.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[29284]: cluster 2024-03-11T21:49:17.951499+0000 osd.1 (osd.1) 449 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:49:19.153 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:18 smithi143 bash[29284]: cluster 2024-03-11T21:49:17.955685+0000 osd.1 (osd.1) 450 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:49:19.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:18 smithi203 bash[23654]: cluster 2024-03-11T21:49:17.849137+0000 mgr.y (mgr.14550) 1004 : cluster 0 pgmap v1678: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:19.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:18 smithi203 bash[23654]: cluster 2024-03-11T21:49:17.951499+0000 osd.1 (osd.1) 449 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:49:19.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:18 smithi203 bash[23654]: cluster 2024-03-11T21:49:17.955685+0000 osd.1 (osd.1) 450 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:49:19.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:49:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:19 smithi143 bash[29284]: cluster 2024-03-11T21:49:18.184246+0000 osd.0 (osd.0) 654 : cluster 0 3.1b scrub starts 2024-03-11T21:49:20.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:19 smithi143 bash[29284]: cluster 2024-03-11T21:49:18.187410+0000 osd.0 (osd.0) 655 : cluster 0 3.1b scrub ok 2024-03-11T21:49:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:19 smithi143 bash[21512]: cluster 2024-03-11T21:49:18.184246+0000 osd.0 (osd.0) 654 : cluster 0 3.1b scrub starts 2024-03-11T21:49:20.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:19 smithi143 bash[21512]: cluster 2024-03-11T21:49:18.187410+0000 osd.0 (osd.0) 655 : cluster 0 3.1b scrub ok 2024-03-11T21:49:20.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:19 smithi203 bash[23654]: cluster 2024-03-11T21:49:18.184246+0000 osd.0 (osd.0) 654 : cluster 0 3.1b scrub starts 2024-03-11T21:49:20.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:19 smithi203 bash[23654]: cluster 2024-03-11T21:49:18.187410+0000 osd.0 (osd.0) 655 : cluster 0 3.1b scrub ok 2024-03-11T21:49:21.200 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: audit 2024-03-11T21:49:19.417736+0000 mgr.y (mgr.14550) 1005 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: cluster 2024-03-11T21:49:19.621957+0000 osd.3 (osd.3) 645 : cluster 0 5.13 scrub starts 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: cluster 2024-03-11T21:49:19.624574+0000 osd.3 (osd.3) 646 : cluster 0 5.13 scrub ok 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: cluster 2024-03-11T21:49:19.851011+0000 mgr.y (mgr.14550) 1006 : cluster 0 pgmap v1679: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.214804+0000 osd.7 (osd.7) 666 : cluster 0 6.8 scrub starts 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.217569+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub ok 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: audit 2024-03-11T21:49:19.417736+0000 mgr.y (mgr.14550) 1005 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: cluster 2024-03-11T21:49:19.621957+0000 osd.3 (osd.3) 645 : cluster 0 5.13 scrub starts 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: cluster 2024-03-11T21:49:19.624574+0000 osd.3 (osd.3) 646 : cluster 0 5.13 scrub ok 2024-03-11T21:49:21.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: cluster 2024-03-11T21:49:19.851011+0000 mgr.y (mgr.14550) 1006 : cluster 0 pgmap v1679: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:21.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.214804+0000 osd.7 (osd.7) 666 : cluster 0 6.8 scrub starts 2024-03-11T21:49:21.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:20 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.217569+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub ok 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2614873126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f4074010 msgr2=0x7f33f40743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2614873126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f4074010 0x7f33f40743f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f33e80098f0 tx=0x7f33e8009c00 comp rx=0 tx=0).stop 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2614873126 shutdown_connections 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2614873126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f4074010 0x7f33f40743f0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2614873126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f410e870 0x7f33f4110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2614873126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074930 0x7f33f410e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2614873126 >> 172.21.15.143:0/2614873126 conn(0x7f33f4078eb0 msgr2=0x7f33f40792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:21.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2614873126 shutdown_connections 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2614873126 wait complete. 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 librados: starting msgr at 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 librados: starting objecter 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 Processor -- start 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- start start 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 librados: setting wanted keys 2024-03-11T21:49:21.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 librados: calling monclient init 2024-03-11T21:49:21.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f4074930 0x7f33f419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f41137c0 con 0x7f33f410e870 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4113640 con 0x7f33f4074930 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4113940 con 0x7f33f4074010 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40396/0 (socket says 172.21.15.143:40396) 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 -- 172.21.15.143:0/2987738452 learned_addr learned my addr 172.21.15.143:0/2987738452 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36434/0 (socket says 172.21.15.143:36434) 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f1d9b700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f4074930 0x7f33f419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 -- 172.21.15.143:0/2987738452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f4074930 msgr2=0x7f33f419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:21.257 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f4074930 0x7f33f419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 -- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 msgr2=0x7f33f419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 -- 172.21.15.143:0/2987738452 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f419ee30 con 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f2d9d700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f259c700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f33e80059d0 tx=0x7f33e800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33e37fe700 10 client.?.objecter ms_handle_connect 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33e37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8027070 con 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f419f0c0 con 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f33e800bb60 con 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8004930 con 0x7f33f4074010 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 10 client.42425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:21.258 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f41a69a0 con 0x7f33f4074010 2024-03-11T21:49:21.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.261+0000 7f33f8da0700 1 librados: init done 2024-03-11T21:49:21.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f404faa0 con 0x7f33f4074010 2024-03-11T21:49:21.259 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f33e802a070 con 0x7f33f4074010 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 0x7f33dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33f1d9b700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 0x7f33dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1012..1012 src has 954..1012) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f33e808f340 con 0x7f33f4074010 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 10 client.42425.objecter ms_dispatch 0x7f33f4001550 osd_map(1012..1012 src has 954..1012) v4 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 3 client.42425.objecter handle_osd_map got epochs [1012,1012] > 0 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 3 client.42425.objecter handle_osd_map decoding full epoch 1012 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 20 client.42425.objecter dump_active .. 0 homeless 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33f1d9b700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 0x7f33dc071ce0 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009750 tx=0x7f33e4009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:21.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.265+0000 7f33e37fe700 10 client.42425.objecter ms_handle_connect 0x7f33dc06f820 2024-03-11T21:49:21.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.269+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f33e8013030 con 0x7f33f4074010 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: audit 2024-03-11T21:49:19.417736+0000 mgr.y (mgr.14550) 1005 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: cluster 2024-03-11T21:49:19.621957+0000 osd.3 (osd.3) 645 : cluster 0 5.13 scrub starts 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: cluster 2024-03-11T21:49:19.624574+0000 osd.3 (osd.3) 646 : cluster 0 5.13 scrub ok 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: cluster 2024-03-11T21:49:19.851011+0000 mgr.y (mgr.14550) 1006 : cluster 0 pgmap v1679: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.214804+0000 osd.7 (osd.7) 666 : cluster 0 6.8 scrub starts 2024-03-11T21:49:21.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:20 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.217569+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub ok 2024-03-11T21:49:21.477 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.485+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33f402d830 con 0x7f33f4074010 2024-03-11T21:49:21.479 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.485+0000 7f33e37fe700 1 -- 172.21.15.143:0/2987738452 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1012) v1 ==== 74+0+26463 (secure 0 0 0) 0x7f33e805c2d0 con 0x7f33f4074010 2024-03-11T21:49:21.480 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:49:21.480 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1012,"fsid":"eebe2e36-dfeb-11ee-95c8-87774f69a715","created":"2024-03-11T21:12:47.384719+0000","modified":"2024-03-11T21:49:15.754743+0000","last_up_change":"2024-03-11T21:31:36.320182+0000","last_in_change":"2024-03-11T21:21:55.456354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":117,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:08.036690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"970","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"895","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:23:01.432928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:23:03.441476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-03-11T21:23:04.441432+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:23:05.388543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:23:07.642696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-11T21:28:36.978109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62c096ed-1ea0-4ca6-ab1c-9f5e186ba52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6803","nonce":2903432023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6805","nonce":2903432023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6809","nonce":2903432023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2903432023},{"type":"v1","addr":"172.21.15.143:6807","nonce":2903432023}]},"public_addr":"172.21.15.143:6803/2903432023","cluster_addr":"172.21.15.143:6805/2903432023","heartbeat_back_addr":"172.21.15.143:6809/2903432023","heartbeat_front_addr":"172.21.15.143:6807/2903432023","state":["exists","up"]},{"osd":1,"uuid":"2948dc44-68d3-4b3d-92bf-2b0c46374f02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":246,"up_from":295,"up_thru":963,"down_at":247,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6811","nonce":4235040845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6813","nonce":4235040845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6817","nonce":4235040845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4235040845},{"type":"v1","addr":"172.21.15.143:6815","nonce":4235040845}]},"public_addr":"172.21.15.143:6811/4235040845","cluster_addr":"172.21.15.143:6813/4235040845","heartbeat_back_addr":"172.21.15.143:6817/4235040845","heartbeat_front_addr":"172.21.15.143:6815/4235040845","state":["exists","up"]},{"osd":2,"uuid":"7217589a-5835-407b-91a7-f5c055a48f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6819","nonce":2592131625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6821","nonce":2592131625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6825","nonce":2592131625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2592131625},{"type":"v1","addr":"172.21.15.143:6823","nonce":2592131625}]},"public_addr":"172.21.15.143:6819/2592131625","cluster_addr":"172.21.15.143:6821/2592131625","heartbeat_back_addr":"172.21.15.143:6825/2592131625","heartbeat_front_addr":"172.21.15.143:6823/2592131625","state":["exists","up"]},{"osd":3,"uuid":"a3e7177f-a5a0-44ad-a65d-9aeac3a680cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6827","nonce":2146571980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6829","nonce":2146571980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6833","nonce":2146571980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2146571980},{"type":"v1","addr":"172.21.15.143:6831","nonce":2146571980}]},"public_addr":"172.21.15.143:6827/2146571980","cluster_addr":"172.21.15.143:6829/2146571980","heartbeat_back_addr":"172.21.15.143:6833/2146571980","heartbeat_front_addr":"172.21.15.143:6831/2146571980","state":["exists","up"]},{"osd":4,"uuid":"d7b9e088-d377-4335-8863-11d581828f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6801","nonce":2090835691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6803","nonce":2090835691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6807","nonce":2090835691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2090835691},{"type":"v1","addr":"172.21.15.203:6805","nonce":2090835691}]},"public_addr":"172.21.15.203:6801/2090835691","cluster_addr":"172.21.15.203:6803/2090835691","heartbeat_back_addr":"172.21.15.203:6807/2090835691","heartbeat_front_addr":"172.21.15.203:6805/2090835691","state":["exists","up"]},{"osd":5,"uuid":"b3b2a75c-68a5-4e00-8068-622e1ebbaa01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1002,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6809","nonce":2432038345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6811","nonce":2432038345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6815","nonce":2432038345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2432038345},{"type":"v1","addr":"172.21.15.203:6813","nonce":2432038345}]},"public_addr":"172.21.15.203:6809/2432038345","cluster_addr":"172.21.15.203:6811/2432038345","heartbeat_back_addr":"172.21.15.203:6815/2432038345","heartbeat_front_addr":"172.21.15.203:6813/2432038345","state":["exists","up"]},{"osd":6,"uuid":"ceffb234-db0c-4ffa-a142-9b201df9b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6817","nonce":513748834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6819","nonce":513748834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6823","nonce":513748834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":513748834},{"type":"v1","addr":"172.21.15.203:6821","nonce":513748834}]},"public_addr":"172.21.15.203:6817/513748834","cluster_addr":"172.21.15.203:6819/513748834","heartbeat_back_addr":"172.21.15.203:6823/513748834","heartbeat_front_addr":"172.21.15.203:6821/513748834","state":["exists","up"]},{"osd":7,"uuid":"88b53171-69d2-4922-a787-e63d44d01c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6825","nonce":1251497952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6827","nonce":1251497952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6831","nonce":1251497952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1251497952},{"type":"v1","addr":"172.21.15.203:6829","nonce":1251497952}]},"public_addr":"172.21.15.203:6825/1251497952","cluster_addr":"172.21.15.203:6827/1251497952","heartbeat_back_addr":"172.21.15.203:6831/1251497952","heartbeat_front_addr":"172.21.15.203:6829/1251497952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:54.295800+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:52.065895+0000","dead_epoch":247},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:34.394010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:21.161452+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:13.559850+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:51.591400+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:11.575609+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:15.411141+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.7","osds":[0,4,2]}],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"9.10","mappings":[{"from":7,"to":2}]},{"pgid":"9.18","mappings":[{"from":7,"to":2}]},{"pgid":"9.1d","mappings":[{"from":7,"to":2}]},{"pgid":"9.1e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6801/3638739149":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4274199357":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/3217951344":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/3912416566":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/906834629":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2791822486":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2190104670":"2024-03-12T21:23:57.022184+0000","172.21.15.143:0/4171894415":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/3787195463":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2845155916":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6801/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:0/2913593434":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/143435387":"2024-03-12T21:13:53.401579+0000","172.21.15.143:0/2930358289":"2024-03-12T21:23:57.022184+0000","172.21.15.143:6800/1839244747":"2024-03-12T21:13:19.169870+0000","172.21.15.143:6800/3638739149":"2024-03-12T21:23:57.022184+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":28,"snaps":[{"begin":5,"length":2}]},{"pool":35,"snaps":[{"begin":1,"length":4}]},{"pool":39,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":137,"snaps":[{"begin":2,"length":3}]},{"pool":156,"snaps":[{"begin":2,"length":1}]},{"pool":165,"snaps":[{"begin":2,"length":4}]},{"pool":173,"snaps":[{"begin":3,"length":4}]},{"pool":178,"snaps":[{"begin":2,"length":1}]},{"pool":185,"snaps":[{"begin":3,"length":2}]},{"pool":198,"snaps":[{"begin":4,"length":4}]},{"pool":210,"snaps":[{"begin":7,"length":1}]},{"pool":230,"snaps":[{"begin":8,"length":1}]},{"pool":241,"snaps":[{"begin":10,"length":1}]},{"pool":244,"snaps":[{"begin":11,"length":2}]},{"pool":256,"snaps":[{"begin":14,"length":1}]},{"pool":316,"snaps":[{"begin":2,"length":1}]},{"pool":317,"snaps":[{"begin":3,"length":1}]},{"pool":320,"snaps":[{"begin":4,"length":1}]},{"pool":324,"snaps":[{"begin":6,"length":1}]},{"pool":336,"snaps":[{"begin":1,"length":1}]},{"pool":338,"snaps":[{"begin":1,"length":1}]},{"pool":339,"snaps":[{"begin":1,"length":2}]},{"pool":340,"snaps":[{"begin":3,"length":1}]},{"pool":341,"snaps":[{"begin":2,"length":2}]},{"pool":342,"snaps":[{"begin":2,"length":3}]},{"pool":343,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 10 librados: watch_flush enter 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 10 librados: watch_flush exit 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 20 client.42425.objecter shutdown clearing up homeless session... 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 10 client.42425.objecter successfully canceled tick 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 msgr2=0x7f33dc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 0x7f33dc071ce0 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009750 tx=0x7f33e4009310 comp rx=0 tx=0).stop 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 msgr2=0x7f33f4199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f33e80059d0 tx=0x7f33e800fa70 comp rx=0 tx=0).stop 2024-03-11T21:49:21.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 shutdown_connections 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f33dc06f820 0x7f33dc071ce0 unknown :-1 s=CLOSED pgs=3514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33f410e870 0x7f33f419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33f4074010 0x7f33f4199e20 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 --2- 172.21.15.143:0/2987738452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33f4074930 0x7f33f419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 >> 172.21.15.143:0/2987738452 conn(0x7f33f4078eb0 msgr2=0x7f33f41062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 shutdown_connections 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 -- 172.21.15.143:0/2987738452 wait complete. 2024-03-11T21:49:21.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:21.489+0000 7f33f8da0700 1 librados: shutdown 2024-03-11T21:49:22.200 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.658468+0000 osd.3 (osd.3) 647 : cluster 0 4.0 scrub starts 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.661780+0000 osd.3 (osd.3) 648 : cluster 0 4.0 scrub ok 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.984735+0000 osd.1 (osd.1) 451 : cluster 0 9.0 scrub starts 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[21512]: cluster 2024-03-11T21:49:20.987211+0000 osd.1 (osd.1) 452 : cluster 0 9.0 scrub ok 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[21512]: audit 2024-03-11T21:49:21.489519+0000 mon.c (mon.2) 1423 : audit 0 from='client.? 172.21.15.143:0/2987738452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.658468+0000 osd.3 (osd.3) 647 : cluster 0 4.0 scrub starts 2024-03-11T21:49:22.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.661780+0000 osd.3 (osd.3) 648 : cluster 0 4.0 scrub ok 2024-03-11T21:49:22.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.984735+0000 osd.1 (osd.1) 451 : cluster 0 9.0 scrub starts 2024-03-11T21:49:22.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[29284]: cluster 2024-03-11T21:49:20.987211+0000 osd.1 (osd.1) 452 : cluster 0 9.0 scrub ok 2024-03-11T21:49:22.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:21 smithi143 bash[29284]: audit 2024-03-11T21:49:21.489519+0000 mon.c (mon.2) 1423 : audit 0 from='client.? 172.21.15.143:0/2987738452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:22.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:21 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.658468+0000 osd.3 (osd.3) 647 : cluster 0 4.0 scrub starts 2024-03-11T21:49:22.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:21 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.661780+0000 osd.3 (osd.3) 648 : cluster 0 4.0 scrub ok 2024-03-11T21:49:22.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:21 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.984735+0000 osd.1 (osd.1) 451 : cluster 0 9.0 scrub starts 2024-03-11T21:49:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:21 smithi203 bash[23654]: cluster 2024-03-11T21:49:20.987211+0000 osd.1 (osd.1) 452 : cluster 0 9.0 scrub ok 2024-03-11T21:49:22.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:21 smithi203 bash[23654]: audit 2024-03-11T21:49:21.489519+0000 mon.c (mon.2) 1423 : audit 0 from='client.? 172.21.15.143:0/2987738452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:22.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:49:22.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:49:22.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:49:22.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:49:22.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:49:22.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:49:22.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:49:22.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[21512]: cluster 2024-03-11T21:49:21.670041+0000 osd.3 (osd.3) 649 : cluster 0 6.13 scrub starts 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[21512]: cluster 2024-03-11T21:49:21.672321+0000 osd.3 (osd.3) 650 : cluster 0 6.13 scrub ok 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[21512]: cluster 2024-03-11T21:49:21.851992+0000 mgr.y (mgr.14550) 1007 : cluster 0 pgmap v1680: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[29284]: cluster 2024-03-11T21:49:21.670041+0000 osd.3 (osd.3) 649 : cluster 0 6.13 scrub starts 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[29284]: cluster 2024-03-11T21:49:21.672321+0000 osd.3 (osd.3) 650 : cluster 0 6.13 scrub ok 2024-03-11T21:49:23.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:22 smithi143 bash[29284]: cluster 2024-03-11T21:49:21.851992+0000 mgr.y (mgr.14550) 1007 : cluster 0 pgmap v1680: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-03-11T21:49:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:22 smithi203 bash[23654]: cluster 2024-03-11T21:49:21.670041+0000 osd.3 (osd.3) 649 : cluster 0 6.13 scrub starts 2024-03-11T21:49:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:22 smithi203 bash[23654]: cluster 2024-03-11T21:49:21.672321+0000 osd.3 (osd.3) 650 : cluster 0 6.13 scrub ok 2024-03-11T21:49:23.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:22 smithi203 bash[23654]: cluster 2024-03-11T21:49:21.851992+0000 mgr.y (mgr.14550) 1007 : cluster 0 pgmap v1680: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[29284]: cluster 2024-03-11T21:49:22.219826+0000 osd.0 (osd.0) 656 : cluster 0 5.1e scrub starts 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[29284]: cluster 2024-03-11T21:49:22.222255+0000 osd.0 (osd.0) 657 : cluster 0 5.1e scrub ok 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[29284]: cluster 2024-03-11T21:49:22.925863+0000 osd.1 (osd.1) 453 : cluster 0 9.4 scrub starts 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[29284]: cluster 2024-03-11T21:49:22.929420+0000 osd.1 (osd.1) 454 : cluster 0 9.4 scrub ok 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[29284]: audit 2024-03-11T21:49:23.281742+0000 mon.c (mon.2) 1424 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:24.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[21512]: cluster 2024-03-11T21:49:22.219826+0000 osd.0 (osd.0) 656 : cluster 0 5.1e scrub starts 2024-03-11T21:49:24.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[21512]: cluster 2024-03-11T21:49:22.222255+0000 osd.0 (osd.0) 657 : cluster 0 5.1e scrub ok 2024-03-11T21:49:24.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[21512]: cluster 2024-03-11T21:49:22.925863+0000 osd.1 (osd.1) 453 : cluster 0 9.4 scrub starts 2024-03-11T21:49:24.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[21512]: cluster 2024-03-11T21:49:22.929420+0000 osd.1 (osd.1) 454 : cluster 0 9.4 scrub ok 2024-03-11T21:49:24.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:23 smithi143 bash[21512]: audit 2024-03-11T21:49:23.281742+0000 mon.c (mon.2) 1424 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:24.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:23 smithi203 bash[23654]: cluster 2024-03-11T21:49:22.219826+0000 osd.0 (osd.0) 656 : cluster 0 5.1e scrub starts 2024-03-11T21:49:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:23 smithi203 bash[23654]: cluster 2024-03-11T21:49:22.222255+0000 osd.0 (osd.0) 657 : cluster 0 5.1e scrub ok 2024-03-11T21:49:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:23 smithi203 bash[23654]: cluster 2024-03-11T21:49:22.925863+0000 osd.1 (osd.1) 453 : cluster 0 9.4 scrub starts 2024-03-11T21:49:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:23 smithi203 bash[23654]: cluster 2024-03-11T21:49:22.929420+0000 osd.1 (osd.1) 454 : cluster 0 9.4 scrub ok 2024-03-11T21:49:24.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:23 smithi203 bash[23654]: audit 2024-03-11T21:49:23.281742+0000 mon.c (mon.2) 1424 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:23.686052+0000 osd.3 (osd.3) 651 : cluster 0 5.17 scrub starts 2024-03-11T21:49:25.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:23.688341+0000 osd.3 (osd.3) 652 : cluster 0 5.17 scrub ok 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:23.853768+0000 mgr.y (mgr.14550) 1008 : cluster 0 pgmap v1681: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:23.961211+0000 osd.1 (osd.1) 455 : cluster 0 6.1 scrub starts 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:23.963621+0000 osd.1 (osd.1) 456 : cluster 0 6.1 scrub ok 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.075132+0000 osd.5 (osd.5) 567 : cluster 0 5.7 scrub starts 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.077590+0000 osd.5 (osd.5) 568 : cluster 0 5.7 scrub ok 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.175267+0000 osd.7 (osd.7) 668 : cluster 0 6.12 scrub starts 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.177568+0000 osd.7 (osd.7) 669 : cluster 0 6.12 scrub ok 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:23.686052+0000 osd.3 (osd.3) 651 : cluster 0 5.17 scrub starts 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:23.688341+0000 osd.3 (osd.3) 652 : cluster 0 5.17 scrub ok 2024-03-11T21:49:25.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:23.853768+0000 mgr.y (mgr.14550) 1008 : cluster 0 pgmap v1681: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:23.961211+0000 osd.1 (osd.1) 455 : cluster 0 6.1 scrub starts 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:23.963621+0000 osd.1 (osd.1) 456 : cluster 0 6.1 scrub ok 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.075132+0000 osd.5 (osd.5) 567 : cluster 0 5.7 scrub starts 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.077590+0000 osd.5 (osd.5) 568 : cluster 0 5.7 scrub ok 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.175267+0000 osd.7 (osd.7) 668 : cluster 0 6.12 scrub starts 2024-03-11T21:49:25.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:24 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.177568+0000 osd.7 (osd.7) 669 : cluster 0 6.12 scrub ok 2024-03-11T21:49:25.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:23.686052+0000 osd.3 (osd.3) 651 : cluster 0 5.17 scrub starts 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:23.688341+0000 osd.3 (osd.3) 652 : cluster 0 5.17 scrub ok 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:23.853768+0000 mgr.y (mgr.14550) 1008 : cluster 0 pgmap v1681: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:23.961211+0000 osd.1 (osd.1) 455 : cluster 0 6.1 scrub starts 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:23.963621+0000 osd.1 (osd.1) 456 : cluster 0 6.1 scrub ok 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.075132+0000 osd.5 (osd.5) 567 : cluster 0 5.7 scrub starts 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.077590+0000 osd.5 (osd.5) 568 : cluster 0 5.7 scrub ok 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.175267+0000 osd.7 (osd.7) 668 : cluster 0 6.12 scrub starts 2024-03-11T21:49:25.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:24 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.177568+0000 osd.7 (osd.7) 669 : cluster 0 6.12 scrub ok 2024-03-11T21:49:26.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.229616+0000 osd.0 (osd.0) 658 : cluster 0 1.11 scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.232108+0000 osd.0 (osd.0) 659 : cluster 0 1.11 scrub ok 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.601977+0000 osd.6 (osd.6) 519 : cluster 0 3.a scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.607118+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub ok 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.690390+0000 osd.3 (osd.3) 653 : cluster 0 3.7 scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:24.694131+0000 osd.3 (osd.3) 654 : cluster 0 3.7 scrub ok 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.096927+0000 osd.5 (osd.5) 569 : cluster 0 9.f scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.103680+0000 osd.5 (osd.5) 570 : cluster 0 9.f scrub ok 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.138731+0000 osd.7 (osd.7) 670 : cluster 0 9.13 scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.143964+0000 osd.7 (osd.7) 671 : cluster 0 9.13 scrub ok 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.229616+0000 osd.0 (osd.0) 658 : cluster 0 1.11 scrub starts 2024-03-11T21:49:26.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.232108+0000 osd.0 (osd.0) 659 : cluster 0 1.11 scrub ok 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.601977+0000 osd.6 (osd.6) 519 : cluster 0 3.a scrub starts 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.607118+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub ok 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.690390+0000 osd.3 (osd.3) 653 : cluster 0 3.7 scrub starts 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:24.694131+0000 osd.3 (osd.3) 654 : cluster 0 3.7 scrub ok 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.096927+0000 osd.5 (osd.5) 569 : cluster 0 9.f scrub starts 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.103680+0000 osd.5 (osd.5) 570 : cluster 0 9.f scrub ok 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.138731+0000 osd.7 (osd.7) 670 : cluster 0 9.13 scrub starts 2024-03-11T21:49:26.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:25 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.143964+0000 osd.7 (osd.7) 671 : cluster 0 9.13 scrub ok 2024-03-11T21:49:26.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.229616+0000 osd.0 (osd.0) 658 : cluster 0 1.11 scrub starts 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.232108+0000 osd.0 (osd.0) 659 : cluster 0 1.11 scrub ok 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.601977+0000 osd.6 (osd.6) 519 : cluster 0 3.a scrub starts 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.607118+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub ok 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.690390+0000 osd.3 (osd.3) 653 : cluster 0 3.7 scrub starts 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:24.694131+0000 osd.3 (osd.3) 654 : cluster 0 3.7 scrub ok 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.096927+0000 osd.5 (osd.5) 569 : cluster 0 9.f scrub starts 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.103680+0000 osd.5 (osd.5) 570 : cluster 0 9.f scrub ok 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.138731+0000 osd.7 (osd.7) 670 : cluster 0 9.13 scrub starts 2024-03-11T21:49:26.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:25 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.143964+0000 osd.7 (osd.7) 671 : cluster 0 9.13 scrub ok 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.712805+0000 osd.3 (osd.3) 655 : cluster 0 6.6 scrub starts 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.715469+0000 osd.3 (osd.3) 656 : cluster 0 6.6 scrub ok 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:25.854877+0000 mgr.y (mgr.14550) 1009 : cluster 0 pgmap v1682: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:26.065873+0000 osd.5 (osd.5) 571 : cluster 0 3.c scrub starts 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:26.069921+0000 osd.5 (osd.5) 572 : cluster 0 3.c scrub ok 2024-03-11T21:49:27.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:26.242081+0000 osd.4 (osd.4) 619 : cluster 0 9.16 scrub starts 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[21512]: cluster 2024-03-11T21:49:26.248668+0000 osd.4 (osd.4) 620 : cluster 0 9.16 scrub ok 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.712805+0000 osd.3 (osd.3) 655 : cluster 0 6.6 scrub starts 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.715469+0000 osd.3 (osd.3) 656 : cluster 0 6.6 scrub ok 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:25.854877+0000 mgr.y (mgr.14550) 1009 : cluster 0 pgmap v1682: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:26.065873+0000 osd.5 (osd.5) 571 : cluster 0 3.c scrub starts 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:26.069921+0000 osd.5 (osd.5) 572 : cluster 0 3.c scrub ok 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:26.242081+0000 osd.4 (osd.4) 619 : cluster 0 9.16 scrub starts 2024-03-11T21:49:27.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:26 smithi143 bash[29284]: cluster 2024-03-11T21:49:26.248668+0000 osd.4 (osd.4) 620 : cluster 0 9.16 scrub ok 2024-03-11T21:49:27.254 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.255 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.255 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.255 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.258 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.258 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.258 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.264 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.712805+0000 osd.3 (osd.3) 655 : cluster 0 6.6 scrub starts 2024-03-11T21:49:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.715469+0000 osd.3 (osd.3) 656 : cluster 0 6.6 scrub ok 2024-03-11T21:49:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:25.854877+0000 mgr.y (mgr.14550) 1009 : cluster 0 pgmap v1682: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-03-11T21:49:27.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:26.065873+0000 osd.5 (osd.5) 571 : cluster 0 3.c scrub starts 2024-03-11T21:49:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:26.069921+0000 osd.5 (osd.5) 572 : cluster 0 3.c scrub ok 2024-03-11T21:49:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:26.242081+0000 osd.4 (osd.4) 619 : cluster 0 9.16 scrub starts 2024-03-11T21:49:27.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:26 smithi203 bash[23654]: cluster 2024-03-11T21:49:26.248668+0000 osd.4 (osd.4) 620 : cluster 0 9.16 scrub ok 2024-03-11T21:49:28.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:27 smithi143 bash[29284]: cluster 2024-03-11T21:49:27.007372+0000 osd.1 (osd.1) 457 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:49:28.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:27 smithi143 bash[29284]: cluster 2024-03-11T21:49:27.009680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:49:28.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:27 smithi143 bash[21512]: cluster 2024-03-11T21:49:27.007372+0000 osd.1 (osd.1) 457 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:49:28.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:27 smithi143 bash[21512]: cluster 2024-03-11T21:49:27.009680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:49:28.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:27 smithi203 bash[23654]: cluster 2024-03-11T21:49:27.007372+0000 osd.1 (osd.1) 457 : cluster 0 5.12 deep-scrub starts 2024-03-11T21:49:28.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:27 smithi203 bash[23654]: cluster 2024-03-11T21:49:27.009680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 deep-scrub ok 2024-03-11T21:49:29.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[21512]: cluster 2024-03-11T21:49:27.247373+0000 osd.0 (osd.0) 660 : cluster 0 3.1f scrub starts 2024-03-11T21:49:29.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[21512]: cluster 2024-03-11T21:49:27.251235+0000 osd.0 (osd.0) 661 : cluster 0 3.1f scrub ok 2024-03-11T21:49:29.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[21512]: cluster 2024-03-11T21:49:27.855712+0000 mgr.y (mgr.14550) 1010 : cluster 0 pgmap v1683: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:29.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[21512]: cluster 2024-03-11T21:49:28.160575+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub starts 2024-03-11T21:49:29.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[21512]: cluster 2024-03-11T21:49:28.164714+0000 osd.7 (osd.7) 673 : cluster 0 3.11 scrub ok 2024-03-11T21:49:29.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[29284]: cluster 2024-03-11T21:49:27.247373+0000 osd.0 (osd.0) 660 : cluster 0 3.1f scrub starts 2024-03-11T21:49:29.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[29284]: cluster 2024-03-11T21:49:27.251235+0000 osd.0 (osd.0) 661 : cluster 0 3.1f scrub ok 2024-03-11T21:49:29.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[29284]: cluster 2024-03-11T21:49:27.855712+0000 mgr.y (mgr.14550) 1010 : cluster 0 pgmap v1683: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:29.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[29284]: cluster 2024-03-11T21:49:28.160575+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub starts 2024-03-11T21:49:29.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:28 smithi143 bash[29284]: cluster 2024-03-11T21:49:28.164714+0000 osd.7 (osd.7) 673 : cluster 0 3.11 scrub ok 2024-03-11T21:49:29.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:28 smithi203 bash[23654]: cluster 2024-03-11T21:49:27.247373+0000 osd.0 (osd.0) 660 : cluster 0 3.1f scrub starts 2024-03-11T21:49:29.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:28 smithi203 bash[23654]: cluster 2024-03-11T21:49:27.251235+0000 osd.0 (osd.0) 661 : cluster 0 3.1f scrub ok 2024-03-11T21:49:29.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:28 smithi203 bash[23654]: cluster 2024-03-11T21:49:27.855712+0000 mgr.y (mgr.14550) 1010 : cluster 0 pgmap v1683: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:29.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:28 smithi203 bash[23654]: cluster 2024-03-11T21:49:28.160575+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub starts 2024-03-11T21:49:29.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:28 smithi203 bash[23654]: cluster 2024-03-11T21:49:28.164714+0000 osd.7 (osd.7) 673 : cluster 0 3.11 scrub ok 2024-03-11T21:49:29.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:49:30.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:29 smithi143 bash[29284]: cluster 2024-03-11T21:49:28.212848+0000 osd.0 (osd.0) 662 : cluster 0 3.1 scrub starts 2024-03-11T21:49:30.217 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:29 smithi143 bash[29284]: cluster 2024-03-11T21:49:28.217310+0000 osd.0 (osd.0) 663 : cluster 0 3.1 scrub ok 2024-03-11T21:49:30.217 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:29 smithi143 bash[21512]: cluster 2024-03-11T21:49:28.212848+0000 osd.0 (osd.0) 662 : cluster 0 3.1 scrub starts 2024-03-11T21:49:30.218 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:29 smithi143 bash[21512]: cluster 2024-03-11T21:49:28.217310+0000 osd.0 (osd.0) 663 : cluster 0 3.1 scrub ok 2024-03-11T21:49:30.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:29 smithi203 bash[23654]: cluster 2024-03-11T21:49:28.212848+0000 osd.0 (osd.0) 662 : cluster 0 3.1 scrub starts 2024-03-11T21:49:30.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:29 smithi203 bash[23654]: cluster 2024-03-11T21:49:28.217310+0000 osd.0 (osd.0) 663 : cluster 0 3.1 scrub ok 2024-03-11T21:49:31.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: audit 2024-03-11T21:49:29.426953+0000 mgr.y (mgr.14550) 1011 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:31.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: cluster 2024-03-11T21:49:29.698180+0000 osd.6 (osd.6) 521 : cluster 0 2.17 scrub starts 2024-03-11T21:49:31.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: cluster 2024-03-11T21:49:29.700248+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub ok 2024-03-11T21:49:31.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: cluster 2024-03-11T21:49:29.857666+0000 mgr.y (mgr.14550) 1012 : cluster 0 pgmap v1684: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:31.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: cluster 2024-03-11T21:49:30.087014+0000 osd.5 (osd.5) 573 : cluster 0 3.16 deep-scrub starts 2024-03-11T21:49:31.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:30 smithi203 bash[23654]: cluster 2024-03-11T21:49:30.095440+0000 osd.5 (osd.5) 574 : cluster 0 3.16 deep-scrub ok 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: audit 2024-03-11T21:49:29.426953+0000 mgr.y (mgr.14550) 1011 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: cluster 2024-03-11T21:49:29.698180+0000 osd.6 (osd.6) 521 : cluster 0 2.17 scrub starts 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: cluster 2024-03-11T21:49:29.700248+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub ok 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: cluster 2024-03-11T21:49:29.857666+0000 mgr.y (mgr.14550) 1012 : cluster 0 pgmap v1684: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: cluster 2024-03-11T21:49:30.087014+0000 osd.5 (osd.5) 573 : cluster 0 3.16 deep-scrub starts 2024-03-11T21:49:31.346 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[29284]: cluster 2024-03-11T21:49:30.095440+0000 osd.5 (osd.5) 574 : cluster 0 3.16 deep-scrub ok 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: audit 2024-03-11T21:49:29.426953+0000 mgr.y (mgr.14550) 1011 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: cluster 2024-03-11T21:49:29.698180+0000 osd.6 (osd.6) 521 : cluster 0 2.17 scrub starts 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: cluster 2024-03-11T21:49:29.700248+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub ok 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: cluster 2024-03-11T21:49:29.857666+0000 mgr.y (mgr.14550) 1012 : cluster 0 pgmap v1684: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: cluster 2024-03-11T21:49:30.087014+0000 osd.5 (osd.5) 573 : cluster 0 3.16 deep-scrub starts 2024-03-11T21:49:31.347 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:30 smithi143 bash[21512]: cluster 2024-03-11T21:49:30.095440+0000 osd.5 (osd.5) 574 : cluster 0 3.16 deep-scrub ok 2024-03-11T21:49:32.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:31 smithi203 bash[23654]: cluster 2024-03-11T21:49:31.089145+0000 osd.4 (osd.4) 621 : cluster 0 3.9 scrub starts 2024-03-11T21:49:32.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:31 smithi203 bash[23654]: cluster 2024-03-11T21:49:31.093500+0000 osd.4 (osd.4) 622 : cluster 0 3.9 scrub ok 2024-03-11T21:49:32.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:31 smithi203 bash[23654]: cluster 2024-03-11T21:49:31.167807+0000 osd.7 (osd.7) 674 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:49:32.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:31 smithi203 bash[23654]: cluster 2024-03-11T21:49:31.172191+0000 osd.7 (osd.7) 675 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[29284]: cluster 2024-03-11T21:49:31.089145+0000 osd.4 (osd.4) 621 : cluster 0 3.9 scrub starts 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[29284]: cluster 2024-03-11T21:49:31.093500+0000 osd.4 (osd.4) 622 : cluster 0 3.9 scrub ok 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[29284]: cluster 2024-03-11T21:49:31.167807+0000 osd.7 (osd.7) 674 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[29284]: cluster 2024-03-11T21:49:31.172191+0000 osd.7 (osd.7) 675 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:49:32.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[21512]: cluster 2024-03-11T21:49:31.089145+0000 osd.4 (osd.4) 621 : cluster 0 3.9 scrub starts 2024-03-11T21:49:32.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[21512]: cluster 2024-03-11T21:49:31.093500+0000 osd.4 (osd.4) 622 : cluster 0 3.9 scrub ok 2024-03-11T21:49:32.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[21512]: cluster 2024-03-11T21:49:31.167807+0000 osd.7 (osd.7) 674 : cluster 0 3.15 deep-scrub starts 2024-03-11T21:49:32.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:31 smithi143 bash[21512]: cluster 2024-03-11T21:49:31.172191+0000 osd.7 (osd.7) 675 : cluster 0 3.15 deep-scrub ok 2024-03-11T21:49:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:32 smithi203 bash[23654]: cluster 2024-03-11T21:49:31.858660+0000 mgr.y (mgr.14550) 1013 : cluster 0 pgmap v1685: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:33.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:32 smithi203 bash[23654]: cluster 2024-03-11T21:49:32.004092+0000 osd.1 (osd.1) 459 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:49:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:32 smithi203 bash[23654]: cluster 2024-03-11T21:49:32.006157+0000 osd.1 (osd.1) 460 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:49:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:32 smithi203 bash[23654]: cluster 2024-03-11T21:49:32.019584+0000 osd.5 (osd.5) 575 : cluster 0 1.a scrub starts 2024-03-11T21:49:33.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:32 smithi203 bash[23654]: cluster 2024-03-11T21:49:32.021862+0000 osd.5 (osd.5) 576 : cluster 0 1.a scrub ok 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[29284]: cluster 2024-03-11T21:49:31.858660+0000 mgr.y (mgr.14550) 1013 : cluster 0 pgmap v1685: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[29284]: cluster 2024-03-11T21:49:32.004092+0000 osd.1 (osd.1) 459 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[29284]: cluster 2024-03-11T21:49:32.006157+0000 osd.1 (osd.1) 460 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[29284]: cluster 2024-03-11T21:49:32.019584+0000 osd.5 (osd.5) 575 : cluster 0 1.a scrub starts 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[29284]: cluster 2024-03-11T21:49:32.021862+0000 osd.5 (osd.5) 576 : cluster 0 1.a scrub ok 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21512]: cluster 2024-03-11T21:49:31.858660+0000 mgr.y (mgr.14550) 1013 : cluster 0 pgmap v1685: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21512]: cluster 2024-03-11T21:49:32.004092+0000 osd.1 (osd.1) 459 : cluster 0 5.19 deep-scrub starts 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21512]: cluster 2024-03-11T21:49:32.006157+0000 osd.1 (osd.1) 460 : cluster 0 5.19 deep-scrub ok 2024-03-11T21:49:33.285 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21512]: cluster 2024-03-11T21:49:32.019584+0000 osd.5 (osd.5) 575 : cluster 0 1.a scrub starts 2024-03-11T21:49:33.286 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:32 smithi143 bash[21512]: cluster 2024-03-11T21:49:32.021862+0000 osd.5 (osd.5) 576 : cluster 0 1.a scrub ok 2024-03-11T21:49:35.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[21512]: cluster 2024-03-11T21:49:33.169686+0000 osd.0 (osd.0) 664 : cluster 0 6.9 scrub starts 2024-03-11T21:49:35.104 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[21512]: cluster 2024-03-11T21:49:33.172038+0000 osd.0 (osd.0) 665 : cluster 0 6.9 scrub ok 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[21512]: cluster 2024-03-11T21:49:33.767070+0000 osd.3 (osd.3) 657 : cluster 0 9.2 scrub starts 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[21512]: cluster 2024-03-11T21:49:33.769528+0000 osd.3 (osd.3) 658 : cluster 0 9.2 scrub ok 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[21512]: cluster 2024-03-11T21:49:33.860575+0000 mgr.y (mgr.14550) 1014 : cluster 0 pgmap v1686: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[29284]: cluster 2024-03-11T21:49:33.169686+0000 osd.0 (osd.0) 664 : cluster 0 6.9 scrub starts 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[29284]: cluster 2024-03-11T21:49:33.172038+0000 osd.0 (osd.0) 665 : cluster 0 6.9 scrub ok 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[29284]: cluster 2024-03-11T21:49:33.767070+0000 osd.3 (osd.3) 657 : cluster 0 9.2 scrub starts 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[29284]: cluster 2024-03-11T21:49:33.769528+0000 osd.3 (osd.3) 658 : cluster 0 9.2 scrub ok 2024-03-11T21:49:35.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:34 smithi143 bash[29284]: cluster 2024-03-11T21:49:33.860575+0000 mgr.y (mgr.14550) 1014 : cluster 0 pgmap v1686: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:35.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:34 smithi203 bash[23654]: cluster 2024-03-11T21:49:33.169686+0000 osd.0 (osd.0) 664 : cluster 0 6.9 scrub starts 2024-03-11T21:49:35.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:34 smithi203 bash[23654]: cluster 2024-03-11T21:49:33.172038+0000 osd.0 (osd.0) 665 : cluster 0 6.9 scrub ok 2024-03-11T21:49:35.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:34 smithi203 bash[23654]: cluster 2024-03-11T21:49:33.767070+0000 osd.3 (osd.3) 657 : cluster 0 9.2 scrub starts 2024-03-11T21:49:35.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:34 smithi203 bash[23654]: cluster 2024-03-11T21:49:33.769528+0000 osd.3 (osd.3) 658 : cluster 0 9.2 scrub ok 2024-03-11T21:49:35.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:34 smithi203 bash[23654]: cluster 2024-03-11T21:49:33.860575+0000 mgr.y (mgr.14550) 1014 : cluster 0 pgmap v1686: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:36.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:35 smithi143 bash[29284]: cluster 2024-03-11T21:49:35.031010+0000 osd.1 (osd.1) 461 : cluster 0 3.4 scrub starts 2024-03-11T21:49:36.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:35 smithi143 bash[29284]: cluster 2024-03-11T21:49:35.036276+0000 osd.1 (osd.1) 462 : cluster 0 3.4 scrub ok 2024-03-11T21:49:36.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:35 smithi143 bash[21512]: cluster 2024-03-11T21:49:35.031010+0000 osd.1 (osd.1) 461 : cluster 0 3.4 scrub starts 2024-03-11T21:49:36.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:35 smithi143 bash[21512]: cluster 2024-03-11T21:49:35.036276+0000 osd.1 (osd.1) 462 : cluster 0 3.4 scrub ok 2024-03-11T21:49:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:35 smithi203 bash[23654]: cluster 2024-03-11T21:49:35.031010+0000 osd.1 (osd.1) 461 : cluster 0 3.4 scrub starts 2024-03-11T21:49:36.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:35 smithi203 bash[23654]: cluster 2024-03-11T21:49:35.036276+0000 osd.1 (osd.1) 462 : cluster 0 3.4 scrub ok 2024-03-11T21:49:37.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[29284]: cluster 2024-03-11T21:49:35.641083+0000 osd.6 (osd.6) 523 : cluster 0 1.7 scrub starts 2024-03-11T21:49:37.280 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[29284]: cluster 2024-03-11T21:49:35.643542+0000 osd.6 (osd.6) 524 : cluster 0 1.7 scrub ok 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[29284]: cluster 2024-03-11T21:49:35.861599+0000 mgr.y (mgr.14550) 1015 : cluster 0 pgmap v1687: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[29284]: cluster 2024-03-11T21:49:36.130158+0000 osd.7 (osd.7) 676 : cluster 0 6.15 scrub starts 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[29284]: cluster 2024-03-11T21:49:36.132405+0000 osd.7 (osd.7) 677 : cluster 0 6.15 scrub ok 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[21512]: cluster 2024-03-11T21:49:35.641083+0000 osd.6 (osd.6) 523 : cluster 0 1.7 scrub starts 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[21512]: cluster 2024-03-11T21:49:35.643542+0000 osd.6 (osd.6) 524 : cluster 0 1.7 scrub ok 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[21512]: cluster 2024-03-11T21:49:35.861599+0000 mgr.y (mgr.14550) 1015 : cluster 0 pgmap v1687: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[21512]: cluster 2024-03-11T21:49:36.130158+0000 osd.7 (osd.7) 676 : cluster 0 6.15 scrub starts 2024-03-11T21:49:37.281 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:36 smithi143 bash[21512]: cluster 2024-03-11T21:49:36.132405+0000 osd.7 (osd.7) 677 : cluster 0 6.15 scrub ok 2024-03-11T21:49:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:36 smithi203 bash[23654]: cluster 2024-03-11T21:49:35.641083+0000 osd.6 (osd.6) 523 : cluster 0 1.7 scrub starts 2024-03-11T21:49:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:36 smithi203 bash[23654]: cluster 2024-03-11T21:49:35.643542+0000 osd.6 (osd.6) 524 : cluster 0 1.7 scrub ok 2024-03-11T21:49:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:36 smithi203 bash[23654]: cluster 2024-03-11T21:49:35.861599+0000 mgr.y (mgr.14550) 1015 : cluster 0 pgmap v1687: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:36 smithi203 bash[23654]: cluster 2024-03-11T21:49:36.130158+0000 osd.7 (osd.7) 676 : cluster 0 6.15 scrub starts 2024-03-11T21:49:37.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:36 smithi203 bash[23654]: cluster 2024-03-11T21:49:36.132405+0000 osd.7 (osd.7) 677 : cluster 0 6.15 scrub ok 2024-03-11T21:49:38.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:37 smithi203 bash[23654]: cluster 2024-03-11T21:49:37.007506+0000 osd.5 (osd.5) 577 : cluster 0 9.a scrub starts 2024-03-11T21:49:38.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:37 smithi203 bash[23654]: cluster 2024-03-11T21:49:37.010978+0000 osd.5 (osd.5) 578 : cluster 0 9.a scrub ok 2024-03-11T21:49:38.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:37 smithi203 bash[23654]: cluster 2024-03-11T21:49:37.070609+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-03-11T21:49:38.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:37 smithi203 bash[23654]: cluster 2024-03-11T21:49:37.072148+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-03-11T21:49:38.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[29284]: cluster 2024-03-11T21:49:37.007506+0000 osd.5 (osd.5) 577 : cluster 0 9.a scrub starts 2024-03-11T21:49:38.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[29284]: cluster 2024-03-11T21:49:37.010978+0000 osd.5 (osd.5) 578 : cluster 0 9.a scrub ok 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[29284]: cluster 2024-03-11T21:49:37.070609+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[29284]: cluster 2024-03-11T21:49:37.072148+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[21512]: cluster 2024-03-11T21:49:37.007506+0000 osd.5 (osd.5) 577 : cluster 0 9.a scrub starts 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[21512]: cluster 2024-03-11T21:49:37.010978+0000 osd.5 (osd.5) 578 : cluster 0 9.a scrub ok 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[21512]: cluster 2024-03-11T21:49:37.070609+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-03-11T21:49:38.455 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:37 smithi143 bash[21512]: cluster 2024-03-11T21:49:37.072148+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-03-11T21:49:39.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:38 smithi203 bash[23654]: cluster 2024-03-11T21:49:37.862319+0000 mgr.y (mgr.14550) 1016 : cluster 0 pgmap v1688: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:38 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.161472+0000 osd.4 (osd.4) 623 : cluster 0 3.14 scrub starts 2024-03-11T21:49:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:38 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.165529+0000 osd.4 (osd.4) 624 : cluster 0 3.14 scrub ok 2024-03-11T21:49:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:38 smithi203 bash[23654]: audit 2024-03-11T21:49:38.314370+0000 mon.c (mon.2) 1425 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.301 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[21512]: cluster 2024-03-11T21:49:37.862319+0000 mgr.y (mgr.14550) 1016 : cluster 0 pgmap v1688: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:39.301 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.161472+0000 osd.4 (osd.4) 623 : cluster 0 3.14 scrub starts 2024-03-11T21:49:39.301 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.165529+0000 osd.4 (osd.4) 624 : cluster 0 3.14 scrub ok 2024-03-11T21:49:39.301 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[21512]: audit 2024-03-11T21:49:38.314370+0000 mon.c (mon.2) 1425 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[29284]: cluster 2024-03-11T21:49:37.862319+0000 mgr.y (mgr.14550) 1016 : cluster 0 pgmap v1688: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:39.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.161472+0000 osd.4 (osd.4) 623 : cluster 0 3.14 scrub starts 2024-03-11T21:49:39.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.165529+0000 osd.4 (osd.4) 624 : cluster 0 3.14 scrub ok 2024-03-11T21:49:39.302 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:38 smithi143 bash[29284]: audit 2024-03-11T21:49:38.314370+0000 mon.c (mon.2) 1425 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:49:39.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- 172.21.15.143:0/4099455110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c106e30 msgr2=0x7fab0c107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 --2- 172.21.15.143:0/4099455110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c106e30 0x7fab0c107210 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fab040098f0 tx=0x7fab04009c00 comp rx=0 tx=0).stop 2024-03-11T21:49:40.032 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- 172.21.15.143:0/4099455110 shutdown_connections 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 --2- 172.21.15.143:0/4099455110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c106e30 0x7fab0c107210 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 --2- 172.21.15.143:0/4099455110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c101980 0x7fab0c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 --2- 172.21.15.143:0/4099455110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c100fc0 0x7fab0c101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- 172.21.15.143:0/4099455110 >> 172.21.15.143:0/4099455110 conn(0x7fab0c077bd0 msgr2=0x7fab0c077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- 172.21.15.143:0/4099455110 shutdown_connections 2024-03-11T21:49:40.033 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- 172.21.15.143:0/4099455110 wait complete. 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 librados: starting msgr at 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 librados: starting objecter 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 Processor -- start 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 -- start start 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 librados: setting wanted keys 2024-03-11T21:49:40.034 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.888+0000 7fab1118c700 1 librados: calling monclient init 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c100fc0 0x7fab0c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 0x7fab0c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c106e30 0x7fab0c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c110630 con 0x7fab0c101980 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c1104b0 con 0x7fab0c100fc0 2024-03-11T21:49:40.035 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c1107b0 con 0x7fab0c106e30 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c106e30 0x7fab0c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 0x7fab0c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 0x7fab0c19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:44394/0 (socket says 172.21.15.143:44394) 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c106e30 0x7fab0c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44044/0 (socket says 172.21.15.143:44044) 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 -- 172.21.15.143:0/2514236933 learned_addr learned my addr 172.21.15.143:0/2514236933 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:40.036 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0ad9d700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c100fc0 0x7fab0c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 -- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c106e30 msgr2=0x7fab0c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab0c106e30 0x7fab0c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 -- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c100fc0 msgr2=0x7fab0c19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c100fc0 0x7fab0c19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c198240 con 0x7fab0c101980 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0ad9d700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab0c100fc0 0x7fab0c19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:49:40.037 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0a59c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 0x7fab0c19de60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7faaf400ec10 tx=0x7faaf400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 10 client.?.objecter ms_handle_connect 0x7fab0c101980 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400cd40 con 0x7fab0c101980 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c198530 con 0x7fab0c101980 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7faaf40044a0 con 0x7fab0c101980 2024-03-11T21:49:40.038 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4010430 con 0x7fab0c101980 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 10 client.31227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c1a47f0 con 0x7fab0c101980 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 librados: init done 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 10 client.31227.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 15 client.31227.objecter _session_command_op_assign -1 1 2024-03-11T21:49:40.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 20 client.31227.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 10 client.31227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab0c102450 con 0x7fab0c101980 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7faaf4003fb0 con 0x7fab0c101980 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faaf806f820 0x7faaf8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1012..1012 src has 954..1012) v4 ==== 8794+0+0 (secure 0 0 0) 0x7faaf408ee70 con 0x7fab0c101980 2024-03-11T21:49:40.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 10 client.31227.objecter ms_dispatch 0x7fab0c001550 osd_map(1012..1012 src has 954..1012) v4 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 3 client.31227.objecter handle_osd_map got epochs [1012,1012] > 0 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab03fff700 3 client.31227.objecter handle_osd_map decoding full epoch 1012 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.892+0000 7fab0ad9d700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faaf806f820 0x7faaf8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter checking command 1 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faaf807b9d0 0x7faaf807de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter _get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 15 client.31227.objecter _session_command_op_remove -1 1 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter _get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 15 client.31227.objecter _session_command_op_assign 6 1 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 4 2024-03-11T21:49:40.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter _send_command 1 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faaf4023e50 con 0x7faaf807b9d0 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter dump_active .. 0 homeless 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=1012) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaf408f2c0 con 0x7fab0c101980 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter ms_handle_connect 0x7faaf806f820 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab0ad9d700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faaf806f820 0x7faaf8071ce0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fab04009fd0 tx=0x7fab0400f600 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:40.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab0b59e700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faaf807b9d0 0x7faaf807de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:40.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter ms_handle_connect 0x7faaf807b9d0 2024-03-11T21:49:40.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab0b59e700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faaf807b9d0 0x7faaf807de70 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:40.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== osd.6 v2:172.21.15.203:6816/513748834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faaf4023e50 con 0x7faaf807b9d0 2024-03-11T21:49:40.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:40.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 10 client.31227.objecter _finish_command 1 = system:0 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 4 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.896+0000 7fab03fff700 15 client.31227.objecter _session_command_op_remove 6 1 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 client.31227.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 15 client.31227.objecter _session_command_op_assign -1 2 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter _get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter _get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 15 client.31227.objecter _session_command_op_remove -1 2 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter get_session s=0x7faaf807b300 osd=6 3 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 15 client.31227.objecter _session_command_op_assign 6 2 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 4 2024-03-11T21:49:40.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 client.31227.objecter _send_command 2 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 --> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab0c04faa0 con 0x7faaf807b9d0 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab03fff700 1 -- 172.21.15.143:0/2514236933 <== osd.6 v2:172.21.15.203:6816/513748834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab0c04faa0 con 0x7faaf807b9d0 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab03fff700 10 client.31227.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab03fff700 10 client.31227.objecter _finish_command 2 = system:0 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab03fff700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 4 2024-03-11T21:49:40.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab03fff700 15 client.31227.objecter _session_command_op_remove 6 2 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 librados: watch_flush enter 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 librados: watch_flush exit 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 client.31227.objecter close_session for osd.6 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faaf807b9d0 msgr2=0x7faaf807de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.203:6816/513748834,v1:172.21.15.203:6817/513748834] conn(0x7faaf807b9d0 0x7faaf807de70 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter put_session s=0x7faaf807b300 osd=6 1 2024-03-11T21:49:40.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 20 client.31227.objecter shutdown clearing up homeless session... 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 10 client.31227.objecter successfully canceled tick 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faaf806f820 msgr2=0x7faaf8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7faaf806f820 0x7faaf8071ce0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fab04009fd0 tx=0x7fab0400f600 comp rx=0 tx=0).stop 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 msgr2=0x7fab0c19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 --2- 172.21.15.143:0/2514236933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab0c101980 0x7fab0c19de60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7faaf400ec10 tx=0x7faaf400ef20 comp rx=0 tx=0).stop 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab0b59e700 1 -- 172.21.15.143:0/2514236933 reap_dead start 2024-03-11T21:49:40.052 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 shutdown_connections 2024-03-11T21:49:40.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 >> 172.21.15.143:0/2514236933 conn(0x7fab0c077bd0 msgr2=0x7fab0c06a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:40.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 shutdown_connections 2024-03-11T21:49:40.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 -- 172.21.15.143:0/2514236933 wait complete. 2024-03-11T21:49:40.053 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:39.916+0000 7fab1118c700 1 librados: shutdown 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.516554+0000 osd.2 (osd.2) 533 : cluster 0 6.14 scrub starts 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.519071+0000 osd.2 (osd.2) 534 : cluster 0 6.14 scrub ok 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.767939+0000 osd.3 (osd.3) 659 : cluster 0 6.1f scrub starts 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.770733+0000 osd.3 (osd.3) 660 : cluster 0 6.1f scrub ok 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:38.997291+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-03-11T21:49:40.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[29284]: cluster 2024-03-11T21:49:39.000443+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.516554+0000 osd.2 (osd.2) 533 : cluster 0 6.14 scrub starts 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.519071+0000 osd.2 (osd.2) 534 : cluster 0 6.14 scrub ok 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.767939+0000 osd.3 (osd.3) 659 : cluster 0 6.1f scrub starts 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.770733+0000 osd.3 (osd.3) 660 : cluster 0 6.1f scrub ok 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:38.997291+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-03-11T21:49:40.273 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:40 smithi143 bash[21512]: cluster 2024-03-11T21:49:39.000443+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-03-11T21:49:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.516554+0000 osd.2 (osd.2) 533 : cluster 0 6.14 scrub starts 2024-03-11T21:49:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.519071+0000 osd.2 (osd.2) 534 : cluster 0 6.14 scrub ok 2024-03-11T21:49:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.767939+0000 osd.3 (osd.3) 659 : cluster 0 6.1f scrub starts 2024-03-11T21:49:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.770733+0000 osd.3 (osd.3) 660 : cluster 0 6.1f scrub ok 2024-03-11T21:49:40.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:38.997291+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-03-11T21:49:40.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:40 smithi203 bash[23654]: cluster 2024-03-11T21:49:39.000443+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-03-11T21:49:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:41 smithi203 bash[23654]: audit 2024-03-11T21:49:39.438177+0000 mgr.y (mgr.14550) 1017 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:41 smithi203 bash[23654]: cluster 2024-03-11T21:49:39.549500+0000 osd.6 (osd.6) 525 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:49:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:41 smithi203 bash[23654]: cluster 2024-03-11T21:49:39.552894+0000 osd.6 (osd.6) 526 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:49:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:41 smithi203 bash[23654]: cluster 2024-03-11T21:49:39.864103+0000 mgr.y (mgr.14550) 1018 : cluster 0 pgmap v1689: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[29284]: audit 2024-03-11T21:49:39.438177+0000 mgr.y (mgr.14550) 1017 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:41.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[29284]: cluster 2024-03-11T21:49:39.549500+0000 osd.6 (osd.6) 525 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:49:41.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[29284]: cluster 2024-03-11T21:49:39.552894+0000 osd.6 (osd.6) 526 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:49:41.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[29284]: cluster 2024-03-11T21:49:39.864103+0000 mgr.y (mgr.14550) 1018 : cluster 0 pgmap v1689: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:41.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[21512]: audit 2024-03-11T21:49:39.438177+0000 mgr.y (mgr.14550) 1017 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:41.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[21512]: cluster 2024-03-11T21:49:39.549500+0000 osd.6 (osd.6) 525 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:49:41.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[21512]: cluster 2024-03-11T21:49:39.552894+0000 osd.6 (osd.6) 526 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:49:41.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:41 smithi143 bash[21512]: cluster 2024-03-11T21:49:39.864103+0000 mgr.y (mgr.14550) 1018 : cluster 0 pgmap v1689: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:40.176313+0000 osd.0 (osd.0) 666 : cluster 0 5.5 scrub starts 2024-03-11T21:49:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:40.178745+0000 osd.0 (osd.0) 667 : cluster 0 5.5 scrub ok 2024-03-11T21:49:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.104643+0000 osd.5 (osd.5) 579 : cluster 0 3.5 scrub starts 2024-03-11T21:49:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.109575+0000 osd.5 (osd.5) 580 : cluster 0 3.5 scrub ok 2024-03-11T21:49:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.167747+0000 osd.4 (osd.4) 625 : cluster 0 4.1 scrub starts 2024-03-11T21:49:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:42 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.170163+0000 osd.4 (osd.4) 626 : cluster 0 4.1 scrub ok 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:40.176313+0000 osd.0 (osd.0) 666 : cluster 0 5.5 scrub starts 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:40.178745+0000 osd.0 (osd.0) 667 : cluster 0 5.5 scrub ok 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.104643+0000 osd.5 (osd.5) 579 : cluster 0 3.5 scrub starts 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.109575+0000 osd.5 (osd.5) 580 : cluster 0 3.5 scrub ok 2024-03-11T21:49:42.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.167747+0000 osd.4 (osd.4) 625 : cluster 0 4.1 scrub starts 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.170163+0000 osd.4 (osd.4) 626 : cluster 0 4.1 scrub ok 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:40.176313+0000 osd.0 (osd.0) 666 : cluster 0 5.5 scrub starts 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:40.178745+0000 osd.0 (osd.0) 667 : cluster 0 5.5 scrub ok 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.104643+0000 osd.5 (osd.5) 579 : cluster 0 3.5 scrub starts 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.109575+0000 osd.5 (osd.5) 580 : cluster 0 3.5 scrub ok 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.167747+0000 osd.4 (osd.4) 625 : cluster 0 4.1 scrub starts 2024-03-11T21:49:42.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:42 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.170163+0000 osd.4 (osd.4) 626 : cluster 0 4.1 scrub ok 2024-03-11T21:49:42.509 INFO:teuthology.orchestra.run.smithi143.stdout:240518168916 2024-03-11T21:49:42.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:49:42.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/3534966247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 msgr2=0x7fdb1c1059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- 172.21.15.143:0/3534966247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c1059c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009920 tx=0x7fdb0c009c30 comp rx=0 tx=0).stop 2024-03-11T21:49:42.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/3534966247 shutdown_connections 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- 172.21.15.143:0/3534966247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c1060a0 0x7fdb1c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- 172.21.15.143:0/3534966247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c100fc0 0x7fdb1c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- 172.21.15.143:0/3534966247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c1059c0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/3534966247 >> 172.21.15.143:0/3534966247 conn(0x7fdb1c077de0 msgr2=0x7fdb1c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/3534966247 shutdown_connections 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/3534966247 wait complete. 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 librados: starting msgr at 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 librados: starting objecter 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 Processor -- start 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- start start 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 librados: setting wanted keys 2024-03-11T21:49:42.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 librados: calling monclient init 2024-03-11T21:49:42.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c100fc0 0x7fdb1c19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c1060a0 0x7fdb1c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c110560 con 0x7fdb1c101970 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c1103e0 con 0x7fdb1c100fc0 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c1106e0 con 0x7fdb1c1060a0 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c19aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:44420/0 (socket says 172.21.15.143:44420) 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 -- 172.21.15.143:0/2030958070 learned_addr learned my addr 172.21.15.143:0/2030958070 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb221d3700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c1060a0 0x7fdb1c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb219d2700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c100fc0 0x7fdb1c19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 -- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c1060a0 msgr2=0x7fdb1c19e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c1060a0 0x7fdb1c19e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 -- 172.21.15.143:0/2030958070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c100fc0 msgr2=0x7fdb1c19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c100fc0 0x7fdb1c19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c19ee80 con 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb221d3700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb1c1060a0 0x7fdb1c19e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb219d2700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb1c100fc0 0x7fdb1c19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb211d1700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c19aa40 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c005e90 tx=0x7fdb0c00fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb12ffd700 10 client.?.objecter ms_handle_connect 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb12ffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c027070 con 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fdb0c005040 con 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c0048d0 con 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.872+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c19f110 con 0x7fdb1c101970 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 10 client.31233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:42.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c19f5a0 con 0x7fdb1c101970 2024-03-11T21:49:42.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 1 librados: init done 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 10 client.31233.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 15 client.31233.objecter _session_command_op_assign -1 1 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 20 client.31233.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 10 client.31233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb1c1024a0 con 0x7fdb1c101970 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fdb0c02a070 con 0x7fdb1c101970 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdb0806f820 0x7fdb08071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:42.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1012..1012 src has 954..1012) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fdb0c08f1d0 con 0x7fdb1c101970 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter ms_dispatch 0x7fdb1c001550 osd_map(1012..1012 src has 954..1012) v4 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 3 client.31233.objecter handle_osd_map got epochs [1012,1012] > 0 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 3 client.31233.objecter handle_osd_map decoding full epoch 1012 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb219d2700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdb0806f820 0x7fdb08071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter checking command 1 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdb0807b820 0x7fdb0807dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter _get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 15 client.31233.objecter _session_command_op_remove -1 1 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter _get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 15 client.31233.objecter _session_command_op_assign 2 1 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 4 2024-03-11T21:49:42.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter _send_command 1 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb0c024ec0 con 0x7fdb0807b820 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter dump_active .. 0 homeless 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=1012) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb0c096050 con 0x7fdb1c101970 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb221d3700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdb0807b820 0x7fdb0807dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter ms_handle_connect 0x7fdb0806f820 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb219d2700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdb0806f820 0x7fdb08071ce0 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fdb18005fd0 tx=0x7fdb18009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb221d3700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdb0807b820 0x7fdb0807dcc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter ms_handle_connect 0x7fdb0807b820 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== osd.2 v2:172.21.15.143:6818/2592131625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb0c024ec0 con 0x7fdb0807b820 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter ms_dispatch 0x7fdb1c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 10 client.31233.objecter _finish_command 1 = system:0 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 4 2024-03-11T21:49:42.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.876+0000 7fdb12ffd700 15 client.31233.objecter _session_command_op_remove 2 1 2024-03-11T21:49:42.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 10 client.31233.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:42.891 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 15 client.31233.objecter _session_command_op_assign -1 2 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 20 client.31233.objecter _get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 20 client.31233.objecter _get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 15 client.31233.objecter _session_command_op_remove -1 2 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 20 client.31233.objecter get_session s=0x7fdb0807b150 osd=2 3 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 15 client.31233.objecter _session_command_op_assign 2 2 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 4 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 10 client.31233.objecter _send_command 2 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 --> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb1c04faa0 con 0x7fdb0807b820 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb12ffd700 1 -- 172.21.15.143:0/2030958070 <== osd.2 v2:172.21.15.143:6818/2592131625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdb1c04faa0 con 0x7fdb0807b820 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb12ffd700 10 client.31233.objecter ms_dispatch 0x7fdb1c001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb12ffd700 10 client.31233.objecter _finish_command 2 = system:0 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb12ffd700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 4 2024-03-11T21:49:42.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.896+0000 7fdb12ffd700 15 client.31233.objecter _session_command_op_remove 2 2 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 10 librados: watch_flush enter 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 10 librados: watch_flush exit 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 10 client.31233.objecter close_session for osd.2 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdb0807b820 msgr2=0x7fdb0807dcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6818/2592131625,v1:172.21.15.143:6819/2592131625] conn(0x7fdb0807b820 0x7fdb0807dcc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 20 client.31233.objecter put_session s=0x7fdb0807b150 osd=2 1 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 20 client.31233.objecter shutdown clearing up homeless session... 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 10 client.31233.objecter successfully canceled tick 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdb0806f820 msgr2=0x7fdb08071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fdb0806f820 0x7fdb08071ce0 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fdb18005fd0 tx=0x7fdb18009310 comp rx=0 tx=0).stop 2024-03-11T21:49:42.894 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 msgr2=0x7fdb1c19aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 --2- 172.21.15.143:0/2030958070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb1c101970 0x7fdb1c19aa40 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c005e90 tx=0x7fdb0c00fb40 comp rx=0 tx=0).stop 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb221d3700 1 -- 172.21.15.143:0/2030958070 reap_dead start 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 shutdown_connections 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 >> 172.21.15.143:0/2030958070 conn(0x7fdb1c077de0 msgr2=0x7fdb1c1049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 shutdown_connections 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 -- 172.21.15.143:0/2030958070 wait complete. 2024-03-11T21:49:42.895 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:42.900+0000 7fdb23c36700 1 librados: shutdown 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 -- 172.21.15.143:0/754793546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678106550 msgr2=0x7f9678106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 --2- 172.21.15.143:0/754793546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678106550 0x7f9678106930 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f96680099c0 tx=0x7f9668009cd0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 -- 172.21.15.143:0/754793546 shutdown_connections 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 --2- 172.21.15.143:0/754793546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678106550 0x7f9678106930 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 --2- 172.21.15.143:0/754793546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9678100e60 0x7f967810d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 --2- 172.21.15.143:0/754793546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 -- 172.21.15.143:0/754793546 >> 172.21.15.143:0/754793546 conn(0x7f96780772e0 msgr2=0x7f96780fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.020+0000 7f967e675700 1 -- 172.21.15.143:0/754793546 shutdown_connections 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- 172.21.15.143:0/754793546 wait complete. 2024-03-11T21:49:43.017 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 librados: starting msgr at 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 librados: starting objecter 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 Processor -- start 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- start start 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 librados: setting wanted keys 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 librados: calling monclient init 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678100e60 0x7f9678198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9678106550 0x7f96781a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96781105b0 con 0x7f9678100e60 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678110430 con 0x7f9678106550 2024-03-11T21:49:43.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9678110730 con 0x7f96781004a0 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96777fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678100e60 0x7f9678198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:44100/0 (socket says 172.21.15.143:44100) 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 -- 172.21.15.143:0/262748159 learned_addr learned my addr 172.21.15.143:0/262748159 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967cc12700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9678106550 0x7f96781a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 -- 172.21.15.143:0/262748159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9678106550 msgr2=0x7f96781a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9678106550 0x7f96781a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 -- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678100e60 msgr2=0x7f9678198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678100e60 0x7f9678198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.020 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96781a3f50 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96777fe700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9678100e60 0x7f9678198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f9677fff700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678198990 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f9668005910 tx=0x7f9668009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96757fa700 10 client.?.objecter ms_handle_connect 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96757fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668009e60 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f966800fe10 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668029820 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9678196780 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 10 client.42443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9678196be0 con 0x7f96781004a0 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 librados: init done 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 10 client.42443.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 15 client.42443.objecter _session_command_op_assign -1 1 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 20 client.42443.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:43.021 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 10 client.42443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:43.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.024+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9678101bc0 con 0x7f96781004a0 2024-03-11T21:49:43.022 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f966801e4d0 con 0x7f96781004a0 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f966006f820 0x7f9660071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1012..1012 src has 954..1012) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f96680933f0 con 0x7f96781004a0 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter ms_dispatch 0x7f9678001550 osd_map(1012..1012 src has 954..1012) v4 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 3 client.42443.objecter handle_osd_map got epochs [1012,1012] > 0 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 3 client.42443.objecter handle_osd_map decoding full epoch 1012 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96777fe700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f966006f820 0x7f9660071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter checking command 1 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f966007b9d0 0x7f966007de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter _get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 15 client.42443.objecter _session_command_op_remove -1 1 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter _get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.023 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 15 client.42443.objecter _session_command_op_assign 0 1 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 4 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter _send_command 1 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f966801fe50 con 0x7f966007b9d0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter dump_active .. 0 homeless 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=1012) v2 ==== 24+0+0 (secure 0 0 0) 0x7f966809a050 con 0x7f96781004a0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f967cc12700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f966007b9d0 0x7f966007de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter ms_handle_connect 0x7f966006f820 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96777fe700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f966006f820 0x7f9660071ce0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f966c009680 tx=0x7f966c009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter ms_handle_connect 0x7f966007b9d0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f967cc12700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f966007b9d0 0x7f966007de70 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== osd.0 v2:172.21.15.143:6802/2903432023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f966801fe50 con 0x7f966007b9d0 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter ms_dispatch 0x7f9678001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:43.024 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 10 client.42443.objecter _finish_command 1 = system:0 2024-03-11T21:49:43.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 4 2024-03-11T21:49:43.025 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.028+0000 7f96757fa700 15 client.42443.objecter _session_command_op_remove 0 1 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 client.42443.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 15 client.42443.objecter _session_command_op_assign -1 2 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter _get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter _get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 15 client.42443.objecter _session_command_op_remove -1 2 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter get_session s=0x7f966007b300 osd=0 3 2024-03-11T21:49:43.039 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 15 client.42443.objecter _session_command_op_assign 0 2 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 4 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 client.42443.objecter _send_command 2 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 --> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f967804faa0 con 0x7f966007b9d0 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f96757fa700 1 -- 172.21.15.143:0/262748159 <== osd.0 v2:172.21.15.143:6802/2903432023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f967804faa0 con 0x7f966007b9d0 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f96757fa700 10 client.42443.objecter ms_dispatch 0x7f9678001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f96757fa700 10 client.42443.objecter _finish_command 2 = system:0 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f96757fa700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 4 2024-03-11T21:49:43.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f96757fa700 15 client.42443.objecter _session_command_op_remove 0 2 2024-03-11T21:49:43.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 librados: watch_flush enter 2024-03-11T21:49:43.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 librados: watch_flush exit 2024-03-11T21:49:43.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 client.42443.objecter close_session for osd.0 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f966007b9d0 msgr2=0x7f966007de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6802/2903432023,v1:172.21.15.143:6803/2903432023] conn(0x7f966007b9d0 0x7f966007de70 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter put_session s=0x7f966007b300 osd=0 1 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 20 client.42443.objecter shutdown clearing up homeless session... 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.044+0000 7f967e675700 10 client.42443.objecter successfully canceled tick 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f966006f820 msgr2=0x7f9660071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f966006f820 0x7f9660071ce0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f966c009680 tx=0x7f966c009260 comp rx=0 tx=0).stop 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 msgr2=0x7f9678198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 --2- 172.21.15.143:0/262748159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96781004a0 0x7f9678198990 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f9668005910 tx=0x7f9668009570 comp rx=0 tx=0).stop 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967cc12700 1 -- 172.21.15.143:0/262748159 reap_dead start 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 shutdown_connections 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 >> 172.21.15.143:0/262748159 conn(0x7f96780772e0 msgr2=0x7f96780fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 shutdown_connections 2024-03-11T21:49:43.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 -- 172.21.15.143:0/262748159 wait complete. 2024-03-11T21:49:43.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:43.048+0000 7f967e675700 1 librados: shutdown 2024-03-11T21:49:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:43 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.864857+0000 mgr.y (mgr.14550) 1019 : cluster 0 pgmap v1690: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:43 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.980781+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-03-11T21:49:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:43 smithi203 bash[23654]: cluster 2024-03-11T21:49:41.983501+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-03-11T21:49:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:43 smithi203 bash[23654]: cluster 2024-03-11T21:49:42.143897+0000 osd.4 (osd.4) 627 : cluster 0 2.f deep-scrub starts 2024-03-11T21:49:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:43 smithi203 bash[23654]: cluster 2024-03-11T21:49:42.147694+0000 osd.4 (osd.4) 628 : cluster 0 2.f deep-scrub ok 2024-03-11T21:49:43.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.864857+0000 mgr.y (mgr.14550) 1019 : cluster 0 pgmap v1690: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:43.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.980781+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-03-11T21:49:43.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[29284]: cluster 2024-03-11T21:49:41.983501+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-03-11T21:49:43.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[29284]: cluster 2024-03-11T21:49:42.143897+0000 osd.4 (osd.4) 627 : cluster 0 2.f deep-scrub starts 2024-03-11T21:49:43.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[29284]: cluster 2024-03-11T21:49:42.147694+0000 osd.4 (osd.4) 628 : cluster 0 2.f deep-scrub ok 2024-03-11T21:49:43.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.864857+0000 mgr.y (mgr.14550) 1019 : cluster 0 pgmap v1690: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:49:43.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.980781+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-03-11T21:49:43.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[21512]: cluster 2024-03-11T21:49:41.983501+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-03-11T21:49:43.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[21512]: cluster 2024-03-11T21:49:42.143897+0000 osd.4 (osd.4) 627 : cluster 0 2.f deep-scrub starts 2024-03-11T21:49:43.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:43 smithi143 bash[21512]: cluster 2024-03-11T21:49:42.147694+0000 osd.4 (osd.4) 628 : cluster 0 2.f deep-scrub ok 2024-03-11T21:49:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:44 smithi203 bash[23654]: cluster 2024-03-11T21:49:42.644595+0000 osd.3 (osd.3) 661 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:49:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:44 smithi203 bash[23654]: cluster 2024-03-11T21:49:42.646997+0000 osd.3 (osd.3) 662 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:49:44.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:44 smithi203 bash[23654]: cluster 2024-03-11T21:49:43.137735+0000 osd.7 (osd.7) 678 : cluster 0 5.4 scrub starts 2024-03-11T21:49:44.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:44 smithi203 bash[23654]: cluster 2024-03-11T21:49:43.140041+0000 osd.7 (osd.7) 679 : cluster 0 5.4 scrub ok 2024-03-11T21:49:44.358 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[29284]: cluster 2024-03-11T21:49:42.644595+0000 osd.3 (osd.3) 661 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:49:44.358 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[29284]: cluster 2024-03-11T21:49:42.646997+0000 osd.3 (osd.3) 662 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:49:44.358 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[29284]: cluster 2024-03-11T21:49:43.137735+0000 osd.7 (osd.7) 678 : cluster 0 5.4 scrub starts 2024-03-11T21:49:44.358 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[29284]: cluster 2024-03-11T21:49:43.140041+0000 osd.7 (osd.7) 679 : cluster 0 5.4 scrub ok 2024-03-11T21:49:44.359 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[21512]: cluster 2024-03-11T21:49:42.644595+0000 osd.3 (osd.3) 661 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:49:44.359 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[21512]: cluster 2024-03-11T21:49:42.646997+0000 osd.3 (osd.3) 662 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:49:44.359 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[21512]: cluster 2024-03-11T21:49:43.137735+0000 osd.7 (osd.7) 678 : cluster 0 5.4 scrub starts 2024-03-11T21:49:44.359 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:44 smithi143 bash[21512]: cluster 2024-03-11T21:49:43.140041+0000 osd.7 (osd.7) 679 : cluster 0 5.4 scrub ok 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 -- 172.21.15.143:0/478113061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 msgr2=0x7fee24108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 --2- 172.21.15.143:0/478113061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee24108c90 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fee14009920 tx=0x7fee14009c30 comp rx=0 tx=0).stop 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 -- 172.21.15.143:0/478113061 shutdown_connections 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 --2- 172.21.15.143:0/478113061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee24110f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 --2- 172.21.15.143:0/478113061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee24104290 0x7fee24104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 --2- 172.21.15.143:0/478113061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee24108c90 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 -- 172.21.15.143:0/478113061 >> 172.21.15.143:0/478113061 conn(0x7fee240ffb40 msgr2=0x7fee24101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 -- 172.21.15.143:0/478113061 shutdown_connections 2024-03-11T21:49:44.516 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 -- 172.21.15.143:0/478113061 wait complete. 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 librados: starting msgr at 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.520+0000 7fee2bf95700 1 librados: starting objecter 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 Processor -- start 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- start start 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 librados: setting wanted keys 2024-03-11T21:49:44.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 librados: calling monclient init 2024-03-11T21:49:44.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee24104290 0x7fee24199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:44.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee2419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee2419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24113830 con 0x7fee24104c40 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241136b0 con 0x7fee24104290 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee241139b0 con 0x7fee24109370 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee2419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee29530700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee2419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee2419e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40780/0 (socket says 172.21.15.143:40780) 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 -- 172.21.15.143:0/1984455423 learned_addr learned my addr 172.21.15.143:0/1984455423 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 -- 172.21.15.143:0/1984455423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee24104290 msgr2=0x7fee24199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee24104290 0x7fee24199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 -- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 msgr2=0x7fee2419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee2419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2419eed0 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee29530700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee24104c40 0x7fee2419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2a532700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee2419e790 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00e3c0 tx=0x7fee1c00e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 10 client.?.objecter ms_handle_connect 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0092d0 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2419f1c0 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fee1c00ee70 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c007d40 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 10 client.42449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee241a6a40 con 0x7fee24109370 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 librados: init done 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 10 client.42449.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 15 client.42449.objecter _session_command_op_assign -1 1 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 20 client.42449.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:44.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 10 client.42449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:44.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee24105770 con 0x7fee24109370 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fee1c012050 con 0x7fee24109370 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fee1006f820 0x7fee10071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1012..1012 src has 954..1012) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fee1c08e320 con 0x7fee24109370 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 10 client.42449.objecter ms_dispatch 0x7fee24001550 osd_map(1012..1012 src has 954..1012) v4 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 3 client.42449.objecter handle_osd_map got epochs [1012,1012] > 0 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 3 client.42449.objecter handle_osd_map decoding full epoch 1012 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee29d31700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fee1006f820 0x7fee10071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 10 client.42449.objecter checking command 1 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fee1007b950 0x7fee1007ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter _get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 15 client.42449.objecter _session_command_op_remove -1 1 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter _get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 15 client.42449.objecter _session_command_op_assign 1 1 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 4 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 10 client.42449.objecter _send_command 1 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee1007e4c0 con 0x7fee1007b950 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 20 client.42449.objecter dump_active .. 0 homeless 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=1012) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee1c095050 con 0x7fee24109370 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.524+0000 7fee29530700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fee1007b950 0x7fee1007ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 10 client.42449.objecter ms_handle_connect 0x7fee1006f820 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee29d31700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fee1006f820 0x7fee10071ce0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7fee2000a990 tx=0x7fee20005c80 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 10 client.42449.objecter ms_handle_connect 0x7fee1007b950 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee29530700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fee1007b950 0x7fee1007ddf0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== osd.1 v2:172.21.15.143:6810/4235040845 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fee1007e4c0 con 0x7fee1007b950 2024-03-11T21:49:44.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 10 client.42449.objecter ms_dispatch 0x7fee24001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:44.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 10 client.42449.objecter _finish_command 1 = system:0 2024-03-11T21:49:44.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 4 2024-03-11T21:49:44.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.528+0000 7fee1affd700 15 client.42449.objecter _session_command_op_remove 1 1 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 client.42449.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 15 client.42449.objecter _session_command_op_assign -1 2 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter _get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter _get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 15 client.42449.objecter _session_command_op_remove -1 2 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter get_session s=0x7fee1007b280 osd=1 3 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 15 client.42449.objecter _session_command_op_assign 1 2 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 4 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 client.42449.objecter _send_command 2 2024-03-11T21:49:44.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 --> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fee2404faa0 con 0x7fee1007b950 2024-03-11T21:49:44.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee1affd700 1 -- 172.21.15.143:0/1984455423 <== osd.1 v2:172.21.15.143:6810/4235040845 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+13 (crc 0 0 0) 0x7fee2404faa0 con 0x7fee1007b950 2024-03-11T21:49:44.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee1affd700 10 client.42449.objecter ms_dispatch 0x7fee24001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:44.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee1affd700 10 client.42449.objecter _finish_command 2 = system:0 2024-03-11T21:49:44.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee1affd700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 4 2024-03-11T21:49:44.539 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee1affd700 15 client.42449.objecter _session_command_op_remove 1 2 2024-03-11T21:49:44.541 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 librados: watch_flush enter 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 librados: watch_flush exit 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 client.42449.objecter close_session for osd.1 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fee1007b950 msgr2=0x7fee1007ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6810/4235040845,v1:172.21.15.143:6811/4235040845] conn(0x7fee1007b950 0x7fee1007ddf0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter put_session s=0x7fee1007b280 osd=1 1 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 20 client.42449.objecter shutdown clearing up homeless session... 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 10 client.42449.objecter successfully canceled tick 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fee1006f820 msgr2=0x7fee10071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fee1006f820 0x7fee10071ce0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7fee2000a990 tx=0x7fee20005c80 comp rx=0 tx=0).stop 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 msgr2=0x7fee2419e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 --2- 172.21.15.143:0/1984455423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee24109370 0x7fee2419e790 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00e3c0 tx=0x7fee1c00e6d0 comp rx=0 tx=0).stop 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2a532700 1 -- 172.21.15.143:0/1984455423 reap_dead start 2024-03-11T21:49:44.542 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 shutdown_connections 2024-03-11T21:49:44.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 >> 172.21.15.143:0/1984455423 conn(0x7fee240ffb40 msgr2=0x7fee241002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:44.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 shutdown_connections 2024-03-11T21:49:44.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 -- 172.21.15.143:0/1984455423 wait complete. 2024-03-11T21:49:44.543 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:44.544+0000 7fee2bf95700 1 librados: shutdown 2024-03-11T21:49:45.240 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[21512]: cluster 2024-03-11T21:49:43.452280+0000 osd.2 (osd.2) 535 : cluster 0 1.14 scrub starts 2024-03-11T21:49:45.240 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[21512]: cluster 2024-03-11T21:49:43.454739+0000 osd.2 (osd.2) 536 : cluster 0 1.14 scrub ok 2024-03-11T21:49:45.240 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[21512]: cluster 2024-03-11T21:49:43.866671+0000 mgr.y (mgr.14550) 1020 : cluster 0 pgmap v1691: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:45 smithi203 bash[23654]: cluster 2024-03-11T21:49:43.452280+0000 osd.2 (osd.2) 535 : cluster 0 1.14 scrub starts 2024-03-11T21:49:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:45 smithi203 bash[23654]: cluster 2024-03-11T21:49:43.454739+0000 osd.2 (osd.2) 536 : cluster 0 1.14 scrub ok 2024-03-11T21:49:45.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:45 smithi203 bash[23654]: cluster 2024-03-11T21:49:43.866671+0000 mgr.y (mgr.14550) 1020 : cluster 0 pgmap v1691: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:45.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[29284]: cluster 2024-03-11T21:49:43.452280+0000 osd.2 (osd.2) 535 : cluster 0 1.14 scrub starts 2024-03-11T21:49:45.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[29284]: cluster 2024-03-11T21:49:43.454739+0000 osd.2 (osd.2) 536 : cluster 0 1.14 scrub ok 2024-03-11T21:49:45.701 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:45 smithi143 bash[29284]: cluster 2024-03-11T21:49:43.866671+0000 mgr.y (mgr.14550) 1020 : cluster 0 pgmap v1691: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:46.157 INFO:teuthology.orchestra.run.smithi143.stdout:81604379006 2024-03-11T21:49:46.158 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:49:46.174 INFO:teuthology.orchestra.run.smithi143.stdout:34359738771 2024-03-11T21:49:46.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:49:46.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 -- 172.21.15.143:0/718768736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4101700 msgr2=0x7fedd410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 --2- 172.21.15.143:0/718768736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4101700 0x7fedd410db70 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009f90 tx=0x7fedcc00b330 comp rx=0 tx=0).stop 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 -- 172.21.15.143:0/718768736 shutdown_connections 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 --2- 172.21.15.143:0/718768736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4101700 0x7fedd410db70 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 --2- 172.21.15.143:0/718768736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4106dc0 0x7fedd41071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 --2- 172.21.15.143:0/718768736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedd4100d40 0x7fedd41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 -- 172.21.15.143:0/718768736 >> 172.21.15.143:0/718768736 conn(0x7fedd4076d80 msgr2=0x7fedd4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 -- 172.21.15.143:0/718768736 shutdown_connections 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 -- 172.21.15.143:0/718768736 wait complete. 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 librados: starting msgr at 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.324+0000 7fedd8832700 1 librados: starting objecter 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 Processor -- start 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- start start 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 librados: setting wanted keys 2024-03-11T21:49:46.321 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 librados: calling monclient init 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 0x7fedd419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4101700 0x7fedd419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedd4106dc0 0x7fedd419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4110520 con 0x7fedd4100d40 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd41103a0 con 0x7fedd4101700 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd41106a0 con 0x7fedd4106dc0 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 0x7fedd419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 0x7fedd419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43750/0 (socket says 172.21.15.143:43750) 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedd4106dc0 0x7fedd419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.322 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 -- 172.21.15.143:0/1702833322 learned_addr learned my addr 172.21.15.143:0/1702833322 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd1d9b700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4101700 0x7fedd419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 -- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedd4106dc0 msgr2=0x7fedd419e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedd4106dc0 0x7fedd419e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 -- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4101700 msgr2=0x7fedd419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4101700 0x7fedd419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd419ec90 con 0x7fedd4100d40 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd1d9b700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd4101700 0x7fedd419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd259c700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 0x7fedd419a480 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c9b0 tx=0x7fedc400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedc37fe700 10 client.?.objecter ms_handle_connect 0x7fedd4100d40 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedc37fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4007b00 con 0x7fedd4100d40 2024-03-11T21:49:46.324 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fedc4007ca0 con 0x7fedd4100d40 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4018660 con 0x7fedd4100d40 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd419ef80 con 0x7fedd4100d40 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 10 client.31251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd41a6840 con 0x7fedd4100d40 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 librados: init done 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 10 client.31251.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 15 client.31251.objecter _session_command_op_assign -1 1 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 20 client.31251.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 10 client.31251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.325 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.328+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedd4102400 con 0x7fedd4100d40 2024-03-11T21:49:46.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fedc400f450 con 0x7fedd4100d40 2024-03-11T21:49:46.326 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fedbc06f820 0x7fedbc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1013..1013 src has 954..1013) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fedc408ec80 con 0x7fedd4100d40 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 10 client.31251.objecter ms_dispatch 0x7fedd4001550 osd_map(1013..1013 src has 954..1013) v4 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 3 client.31251.objecter handle_osd_map got epochs [1013,1013] > 0 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 3 client.31251.objecter handle_osd_map decoding full epoch 1013 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 10 client.31251.objecter checking command 1 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fedbc07b950 0x7fedbc07ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter _get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 15 client.31251.objecter _session_command_op_remove -1 1 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter _get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 15 client.31251.objecter _session_command_op_assign 4 1 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 4 2024-03-11T21:49:46.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 10 client.31251.objecter _send_command 1 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fedc4020e50 con 0x7fedbc07b950 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 20 client.31251.objecter dump_active .. 0 homeless 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=1013) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedc408f030 con 0x7fedd4100d40 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedd1d9b700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fedbc06f820 0x7fedbc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedd2d9d700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fedbc07b950 0x7fedbc07ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 10 client.31251.objecter ms_handle_connect 0x7fedbc06f820 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedd1d9b700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fedbc06f820 0x7fedbc071ce0 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7fedd4077530 tx=0x7fedc8009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedc37fe700 10 client.31251.objecter ms_handle_connect 0x7fedbc07b950 2024-03-11T21:49:46.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.332+0000 7fedd2d9d700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fedbc07b950 0x7fedbc07ddf0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.336+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== osd.4 v2:172.21.15.203:6800/2090835691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fedc4020e50 con 0x7fedbc07b950 2024-03-11T21:49:46.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.336+0000 7fedc37fe700 10 client.31251.objecter ms_dispatch 0x7fedd4001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:46.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.336+0000 7fedc37fe700 10 client.31251.objecter _finish_command 1 = system:0 2024-03-11T21:49:46.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.336+0000 7fedc37fe700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 4 2024-03-11T21:49:46.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.336+0000 7fedc37fe700 15 client.31251.objecter _session_command_op_remove 4 1 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 client.31251.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 15 client.31251.objecter _session_command_op_assign -1 2 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter _get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter _get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 15 client.31251.objecter _session_command_op_remove -1 2 2024-03-11T21:49:46.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter get_session s=0x7fedbc07b280 osd=4 3 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 15 client.31251.objecter _session_command_op_assign 4 2 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 4 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 client.31251.objecter _send_command 2 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 --> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedd404faa0 con 0x7fedbc07b950 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedc37fe700 1 -- 172.21.15.143:0/1702833322 <== osd.4 v2:172.21.15.203:6800/2090835691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fedd404faa0 con 0x7fedbc07b950 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedc37fe700 10 client.31251.objecter ms_dispatch 0x7fedd4001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedc37fe700 10 client.31251.objecter _finish_command 2 = system:0 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedc37fe700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 4 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedc37fe700 15 client.31251.objecter _session_command_op_remove 4 2 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 librados: watch_flush enter 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 librados: watch_flush exit 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 client.31251.objecter close_session for osd.4 2024-03-11T21:49:46.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fedbc07b950 msgr2=0x7fedbc07ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.203:6800/2090835691,v1:172.21.15.203:6801/2090835691] conn(0x7fedbc07b950 0x7fedbc07ddf0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter put_session s=0x7fedbc07b280 osd=4 1 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 20 client.31251.objecter shutdown clearing up homeless session... 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 10 client.31251.objecter successfully canceled tick 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fedbc06f820 msgr2=0x7fedbc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fedbc06f820 0x7fedbc071ce0 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7fedd4077530 tx=0x7fedc8009310 comp rx=0 tx=0).stop 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 msgr2=0x7fedd419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 --2- 172.21.15.143:0/1702833322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedd4100d40 0x7fedd419a480 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c9b0 tx=0x7fedc400ccc0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd2d9d700 1 -- 172.21.15.143:0/1702833322 reap_dead start 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 shutdown_connections 2024-03-11T21:49:46.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.352+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 >> 172.21.15.143:0/1702833322 conn(0x7fedd4076d80 msgr2=0x7fedd40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.356+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 shutdown_connections 2024-03-11T21:49:46.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.356+0000 7fedd8832700 1 -- 172.21.15.143:0/1702833322 wait complete. 2024-03-11T21:49:46.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.356+0000 7fedd8832700 1 librados: shutdown 2024-03-11T21:49:46.386 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:44.672836+0000 osd.3 (osd.3) 663 : cluster 0 2.1b scrub starts 2024-03-11T21:49:46.386 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:44.674700+0000 osd.3 (osd.3) 664 : cluster 0 2.1b scrub ok 2024-03-11T21:49:46.386 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:44.969935+0000 osd.2 (osd.2) 537 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:44.970463+0000 osd.2 (osd.2) 538 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.038999+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e1013: 8 total, 8 up, 8 in 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.137951+0000 osd.4 (osd.4) 629 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.142089+0000 osd.4 (osd.4) 630 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:44.672836+0000 osd.3 (osd.3) 663 : cluster 0 2.1b scrub starts 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:44.674700+0000 osd.3 (osd.3) 664 : cluster 0 2.1b scrub ok 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:44.969935+0000 osd.2 (osd.2) 537 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:44.970463+0000 osd.2 (osd.2) 538 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.038999+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e1013: 8 total, 8 up, 8 in 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.137951+0000 osd.4 (osd.4) 629 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:49:46.387 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:46 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.142089+0000 osd.4 (osd.4) 630 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- 172.21.15.143:0/1795902080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64100d40 msgr2=0x7f8e641011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- 172.21.15.143:0/1795902080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64100d40 0x7f8e641011c0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009990 tx=0x7f8e54009ca0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- 172.21.15.143:0/1795902080 shutdown_connections 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- 172.21.15.143:0/1795902080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64101700 0x7f8e6410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- 172.21.15.143:0/1795902080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e64106dc0 0x7f8e641071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- 172.21.15.143:0/1795902080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64100d40 0x7f8e641011c0 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- 172.21.15.143:0/1795902080 >> 172.21.15.143:0/1795902080 conn(0x7f8e64076d80 msgr2=0x7f8e64077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- 172.21.15.143:0/1795902080 shutdown_connections 2024-03-11T21:49:46.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- 172.21.15.143:0/1795902080 wait complete. 2024-03-11T21:49:46.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 librados: starting msgr at 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 librados: starting objecter 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 Processor -- start 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- start start 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 librados: setting wanted keys 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 librados: calling monclient init 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 0x7f8e64196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 0x7f8e64196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e64106dc0 0x7f8e641972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64110570 con 0x7f8e64100d40 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641103f0 con 0x7f8e64106dc0 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e6b55e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641106f0 con 0x7f8e64101700 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e692fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 0x7f8e64196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 0x7f8e64196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 0x7f8e64196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40818/0 (socket says 172.21.15.143:40818) 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e692fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 0x7f8e64196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43772/0 (socket says 172.21.15.143:43772) 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e692fa700 1 -- 172.21.15.143:0/2599863608 learned_addr learned my addr 172.21.15.143:0/2599863608 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e69afb700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e64106dc0 0x7f8e641972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 -- 172.21.15.143:0/2599863608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e64106dc0 msgr2=0x7f8e641972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e64106dc0 0x7f8e641972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 -- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 msgr2=0x7f8e64196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 0x7f8e64196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e68af9700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e641a6050 con 0x7f8e64101700 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.528+0000 7f8e692fa700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e64100d40 0x7f8e64196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e68af9700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 0x7f8e64196d80 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f8e540059d0 tx=0x7f8e540057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.?.objecter ms_handle_connect 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54027070 con 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e6b55e700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e641a62e0 con 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f8e54022470 con 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54017650 con 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e6b55e700 10 client.42461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e6b55e700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e641a6790 con 0x7f8e64101700 2024-03-11T21:49:46.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e6b55e700 1 librados: init done 2024-03-11T21:49:46.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e4ffff700 10 client.42461.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e4ffff700 15 client.42461.objecter _session_command_op_assign -1 1 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e4ffff700 20 client.42461.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e4ffff700 10 client.42461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8e44002920 con 0x7f8e64101700 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8e5402a070 con 0x7f8e64101700 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8e5006f820 0x7f8e50071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e692fa700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8e5006f820 0x7f8e50071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1013..1013 src has 954..1013) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f8e5408f4f0 con 0x7f8e64101700 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.42461.objecter ms_dispatch 0x7f8e64001550 osd_map(1013..1013 src has 954..1013) v4 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 3 client.42461.objecter handle_osd_map got epochs [1013,1013] > 0 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 3 client.42461.objecter handle_osd_map decoding full epoch 1013 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e692fa700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8e5006f820 0x7f8e50071ce0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f8e60005950 tx=0x7f8e6000a3b0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.42461.objecter checking command 1 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8e5007b9d0 0x7f8e5007de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter _get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 15 client.42461.objecter _session_command_op_remove -1 1 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter _get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 15 client.42461.objecter _session_command_op_assign 3 1 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 4 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.42461.objecter _send_command 1 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e5007e540 con 0x7f8e5007b9d0 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 20 client.42461.objecter dump_active .. 0 homeless 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 10 client.42461.objecter ms_handle_connect 0x7f8e5006f820 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=1013) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e5408f8a0 con 0x7f8e64101700 2024-03-11T21:49:46.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.532+0000 7f8e69afb700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8e5007b9d0 0x7f8e5007de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 10 client.42461.objecter ms_handle_connect 0x7f8e5007b9d0 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e69afb700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8e5007b9d0 0x7f8e5007de70 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== osd.3 v2:172.21.15.143:6826/2146571980 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8e5007e540 con 0x7f8e5007b9d0 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 10 client.42461.objecter ms_dispatch 0x7f8e64001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 10 client.42461.objecter _finish_command 1 = system:0 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 4 2024-03-11T21:49:46.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.536+0000 7f8e5a7fc700 15 client.42461.objecter _session_command_op_remove 3 1 2024-03-11T21:49:46.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:44.672836+0000 osd.3 (osd.3) 663 : cluster 0 2.1b scrub starts 2024-03-11T21:49:46.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:44.674700+0000 osd.3 (osd.3) 664 : cluster 0 2.1b scrub ok 2024-03-11T21:49:46.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:44.969935+0000 osd.2 (osd.2) 537 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:46.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:44.970463+0000 osd.2 (osd.2) 538 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:46.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.038999+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e1013: 8 total, 8 up, 8 in 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.137951+0000 osd.4 (osd.4) 629 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:46 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.142089+0000 osd.4 (osd.4) 630 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 client.42461.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 15 client.42461.objecter _session_command_op_assign -1 2 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter _get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter _get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 15 client.42461.objecter _session_command_op_remove -1 2 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter get_session s=0x7f8e5007b300 osd=3 3 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 15 client.42461.objecter _session_command_op_assign 3 2 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 4 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 client.42461.objecter _send_command 2 2024-03-11T21:49:46.547 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 --> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8e44001240 con 0x7f8e5007b9d0 2024-03-11T21:49:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e5a7fc700 1 -- 172.21.15.143:0/2599863608 <== osd.3 v2:172.21.15.143:6826/2146571980 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8e44001240 con 0x7f8e5007b9d0 2024-03-11T21:49:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e5a7fc700 10 client.42461.objecter ms_dispatch 0x7f8e64001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e5a7fc700 10 client.42461.objecter _finish_command 2 = system:0 2024-03-11T21:49:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e5a7fc700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 4 2024-03-11T21:49:46.548 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e5a7fc700 15 client.42461.objecter _session_command_op_remove 3 2 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 librados: watch_flush enter 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 librados: watch_flush exit 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 client.42461.objecter close_session for osd.3 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8e5007b9d0 msgr2=0x7f8e5007de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6826/2146571980,v1:172.21.15.143:6827/2146571980] conn(0x7f8e5007b9d0 0x7f8e5007de70 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.549 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter put_session s=0x7f8e5007b300 osd=3 1 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 20 client.42461.objecter shutdown clearing up homeless session... 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 10 client.42461.objecter successfully canceled tick 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8e5006f820 msgr2=0x7f8e50071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f8e5006f820 0x7f8e50071ce0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f8e60005950 tx=0x7f8e6000a3b0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 msgr2=0x7f8e64196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 --2- 172.21.15.143:0/2599863608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e64101700 0x7f8e64196d80 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f8e540059d0 tx=0x7f8e540057c0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e69afb700 1 -- 172.21.15.143:0/2599863608 reap_dead start 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 shutdown_connections 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 >> 172.21.15.143:0/2599863608 conn(0x7f8e64076d80 msgr2=0x7f8e640ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 shutdown_connections 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.552+0000 7f8e4ffff700 1 -- 172.21.15.143:0/2599863608 wait complete. 2024-03-11T21:49:46.550 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.556+0000 7f8e4ffff700 1 librados: shutdown 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/3092812749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104270 msgr2=0x7f5e8c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/3092812749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104270 0x7f5e8c104650 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b0d0 tx=0x7f5e7c00b3e0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/3092812749 shutdown_connections 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/3092812749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e8c109350 0x7f5e8c110f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/3092812749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104270 0x7f5e8c104650 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/3092812749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104c20 0x7f5e8c108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/3092812749 >> 172.21.15.143:0/3092812749 conn(0x7f5e8c0ffac0 msgr2=0x7f5e8c101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/3092812749 shutdown_connections 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/3092812749 wait complete. 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 librados: starting msgr at 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 librados: starting objecter 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 Processor -- start 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- start start 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 librados: setting wanted keys 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 librados: calling monclient init 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 0x7f5e8c199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104c20 0x7f5e8c19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e8c109350 0x7f5e8c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c113800 con 0x7f5e8c104270 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c113680 con 0x7f5e8c109350 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c113980 con 0x7f5e8c104c20 2024-03-11T21:49:46.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 0x7f5e8c199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104c20 0x7f5e8c19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 0x7f5e8c199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43778/0 (socket says 172.21.15.143:43778) 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 -- 172.21.15.143:0/875774284 learned_addr learned my addr 172.21.15.143:0/875774284 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 -- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104c20 msgr2=0x7f5e8c19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e8c104c20 0x7f5e8c19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 -- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e8c109350 msgr2=0x7f5e8c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e8c109350 0x7f5e8c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c19ede0 con 0x7f5e8c104270 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e8b7fe700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 0x7f5e8c199dd0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b770 tx=0x7f5e7c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e88ff9700 10 client.?.objecter ms_handle_connect 0x7f5e8c104270 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e88ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c01f070 con 0x7f5e8c104270 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f5e7c003c00 con 0x7f5e8c104270 2024-03-11T21:49:46.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c19f070 con 0x7f5e8c104270 2024-03-11T21:49:46.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c01b6a0 con 0x7f5e8c104270 2024-03-11T21:49:46.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 10 client.31263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c1a6950 con 0x7f5e8c104270 2024-03-11T21:49:46.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 librados: init done 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 10 client.31263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 15 client.31263.objecter _session_command_op_assign -1 1 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 20 client.31263.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 10 client.31263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.660+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e8c105750 con 0x7f5e8c104270 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5e7c022070 con 0x7f5e8c104270 2024-03-11T21:49:46.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e7806f820 0x7f5e78071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1013..1013 src has 954..1013) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f5e7c00f020 con 0x7f5e8c104270 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter ms_dispatch 0x7f5e8c001550 osd_map(1013..1013 src has 954..1013) v4 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e8affd700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e7806f820 0x7f5e78071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 3 client.31263.objecter handle_osd_map got epochs [1013,1013] > 0 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 3 client.31263.objecter handle_osd_map decoding full epoch 1013 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter checking command 1 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5e7807b950 0x7f5e7807ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter _get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 15 client.31263.objecter _session_command_op_remove -1 1 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter _get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 15 client.31263.objecter _session_command_op_assign 7 1 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 4 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter _send_command 1 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e7807e4c0 con 0x7f5e7807b950 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter dump_active .. 0 homeless 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=1013) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e7c090010 con 0x7f5e8c104270 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter ms_handle_connect 0x7f5e7806f820 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e8affd700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e7806f820 0x7f5e78071ce0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009680 tx=0x7f5e74009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e8bfff700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5e7807b950 0x7f5e7807ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter ms_handle_connect 0x7f5e7807b950 2024-03-11T21:49:46.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e8bfff700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5e7807b950 0x7f5e7807ddf0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== osd.7 v2:172.21.15.203:6824/1251497952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5e7807e4c0 con 0x7f5e7807b950 2024-03-11T21:49:46.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter ms_dispatch 0x7f5e8c001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:46.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 10 client.31263.objecter _finish_command 1 = system:0 2024-03-11T21:49:46.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 4 2024-03-11T21:49:46.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.664+0000 7f5e88ff9700 15 client.31263.objecter _session_command_op_remove 7 1 2024-03-11T21:49:46.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 10 client.31263.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 15 client.31263.objecter _session_command_op_assign -1 2 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 20 client.31263.objecter _get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 20 client.31263.objecter _get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 15 client.31263.objecter _session_command_op_remove -1 2 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 20 client.31263.objecter get_session s=0x7f5e7807b280 osd=7 3 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 15 client.31263.objecter _session_command_op_assign 7 2 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 4 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 10 client.31263.objecter _send_command 2 2024-03-11T21:49:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.680+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 --> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e8c04faa0 con 0x7f5e7807b950 2024-03-11T21:49:46.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e88ff9700 1 -- 172.21.15.143:0/875774284 <== osd.7 v2:172.21.15.203:6824/1251497952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e8c04faa0 con 0x7f5e7807b950 2024-03-11T21:49:46.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e88ff9700 10 client.31263.objecter ms_dispatch 0x7f5e8c001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:46.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e88ff9700 10 client.31263.objecter _finish_command 2 = system:0 2024-03-11T21:49:46.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e88ff9700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 4 2024-03-11T21:49:46.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e88ff9700 15 client.31263.objecter _session_command_op_remove 7 2 2024-03-11T21:49:46.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 10 librados: watch_flush enter 2024-03-11T21:49:46.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 10 librados: watch_flush exit 2024-03-11T21:49:46.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 10 client.31263.objecter close_session for osd.7 2024-03-11T21:49:46.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5e7807b950 msgr2=0x7f5e7807ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.203:6824/1251497952,v1:172.21.15.203:6825/1251497952] conn(0x7f5e7807b950 0x7f5e7807ddf0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 20 client.31263.objecter put_session s=0x7f5e7807b280 osd=7 1 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 20 client.31263.objecter shutdown clearing up homeless session... 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 10 client.31263.objecter successfully canceled tick 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e7806f820 msgr2=0x7f5e78071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f5e7806f820 0x7f5e78071ce0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009680 tx=0x7f5e74009260 comp rx=0 tx=0).stop 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 msgr2=0x7f5e8c199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 --2- 172.21.15.143:0/875774284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e8c104270 0x7f5e8c199dd0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b770 tx=0x7f5e7c007a20 comp rx=0 tx=0).stop 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e8bfff700 1 -- 172.21.15.143:0/875774284 reap_dead start 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 shutdown_connections 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 >> 172.21.15.143:0/875774284 conn(0x7f5e8c0ffac0 msgr2=0x7f5e8c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 shutdown_connections 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 -- 172.21.15.143:0/875774284 wait complete. 2024-03-11T21:49:46.679 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.684+0000 7f5e91b3a700 1 librados: shutdown 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- 172.21.15.143:0/3413903615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac406ae80 msgr2=0x7feac410e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- 172.21.15.143:0/3413903615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac406ae80 0x7feac410e2f0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7feab40098f0 tx=0x7feab4009c00 comp rx=0 tx=0).stop 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- 172.21.15.143:0/3413903615 shutdown_connections 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- 172.21.15.143:0/3413903615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac410e830 0x7feac4110c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- 172.21.15.143:0/3413903615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 0x7feac406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- 172.21.15.143:0/3413903615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac406ae80 0x7feac410e2f0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- 172.21.15.143:0/3413903615 >> 172.21.15.143:0/3413903615 conn(0x7feac4077d30 msgr2=0x7feac4078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- 172.21.15.143:0/3413903615 shutdown_connections 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- 172.21.15.143:0/3413903615 wait complete. 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 librados: starting msgr at 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 librados: starting objecter 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 Processor -- start 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- start start 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 librados: setting wanted keys 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 librados: calling monclient init 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 0x7feac4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac406ae80 0x7feac419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 0x7feac419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac4113850 con 0x7feac406ae80 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41136d0 con 0x7feac406a560 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.884+0000 7feacb5e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac41139d0 con 0x7feac410e830 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 0x7feac419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac8b83700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac406ae80 0x7feac419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 0x7feac419e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:40856/0 (socket says 172.21.15.143:40856) 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 -- 172.21.15.143:0/1199078257 learned_addr learned my addr 172.21.15.143:0/1199078257 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 -- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 msgr2=0x7feac4199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9384700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 0x7feac4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 0x7feac4199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 -- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac406ae80 msgr2=0x7feac419a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac406ae80 0x7feac419a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac419ef20 con 0x7feac410e830 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac8b83700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac406ae80 0x7feac419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9384700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac406a560 0x7feac4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:49:46.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9b85700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 0x7feac419e7e0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c3d0 tx=0x7feabc00c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.?.objecter ms_handle_connect 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.?.objecter resend_mon_ops 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc019070 con 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac419f210 con 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7feabc009310 con 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0046d0 con 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 10 client.42473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac41a6a90 con 0x7feac410e830 2024-03-11T21:49:46.884 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 1 librados: init done 2024-03-11T21:49:46.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 10 client.42473.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-11T21:49:46.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 15 client.42473.objecter _session_command_op_assign -1 1 2024-03-11T21:49:46.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 20 client.42473.objecter _get_session osd=-1 returning homeless 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 10 client.42473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feac4101320 con 0x7feac410e830 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7feabc01c070 con 0x7feac410e830 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feab006f820 0x7feab0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1013..1013 src has 954..1013) v4 ==== 8794+0+0 (secure 0 0 0) 0x7feabc08e3e0 con 0x7feac410e830 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.42473.objecter ms_dispatch 0x7feac4001550 osd_map(1013..1013 src has 954..1013) v4 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 3 client.42473.objecter handle_osd_map got epochs [1013,1013] > 0 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 3 client.42473.objecter handle_osd_map decoding full epoch 1013 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9384700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feab006f820 0x7feab0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.42473.objecter checking command 1 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7feab007b950 0x7feab007ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter _get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 15 client.42473.objecter _session_command_op_remove -1 1 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter _get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 15 client.42473.objecter _session_command_op_assign 5 1 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 4 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.42473.objecter _send_command 1 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feabc008ec0 con 0x7feab007b950 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 20 client.42473.objecter dump_active .. 0 homeless 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=1013) v2 ==== 24+0+0 (secure 0 0 0) 0x7feabc095050 con 0x7feac410e830 2024-03-11T21:49:46.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac9384700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feab006f820 0x7feab0071ce0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7feac0007900 tx=0x7feac0008040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feaba7fc700 10 client.42473.objecter ms_handle_connect 0x7feab006f820 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.888+0000 7feac8b83700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7feab007b950 0x7feab007ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 10 client.42473.objecter ms_handle_connect 0x7feab007b950 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feac8b83700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7feab007b950 0x7feab007ddf0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== osd.5 v2:172.21.15.203:6808/2432038345 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feabc008ec0 con 0x7feab007b950 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 10 client.42473.objecter ms_dispatch 0x7feac4001550 command_reply(tid 1: 0 ) v1 2024-03-11T21:49:46.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 10 client.42473.objecter _finish_command 1 = system:0 2024-03-11T21:49:46.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 4 2024-03-11T21:49:46.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.892+0000 7feaba7fc700 15 client.42473.objecter _session_command_op_remove 5 1 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 client.42473.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 15 client.42473.objecter _session_command_op_assign -1 2 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter _get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter _get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 15 client.42473.objecter _session_command_op_remove -1 2 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter get_session s=0x7feab007b280 osd=5 3 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 15 client.42473.objecter _session_command_op_assign 5 2 2024-03-11T21:49:46.902 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 4 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 client.42473.objecter _send_command 2 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 --> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feac404faa0 con 0x7feab007b950 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feaba7fc700 1 -- 172.21.15.143:0/1199078257 <== osd.5 v2:172.21.15.203:6808/2432038345 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feac404faa0 con 0x7feab007b950 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feaba7fc700 10 client.42473.objecter ms_dispatch 0x7feac4001550 command_reply(tid 2: 0 ) v1 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feaba7fc700 10 client.42473.objecter _finish_command 2 = system:0 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feaba7fc700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 4 2024-03-11T21:49:46.903 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feaba7fc700 15 client.42473.objecter _session_command_op_remove 5 2 2024-03-11T21:49:46.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 librados: watch_flush enter 2024-03-11T21:49:46.904 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 librados: watch_flush exit 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 client.42473.objecter close_session for osd.5 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7feab007b950 msgr2=0x7feab007ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.203:6808/2432038345,v1:172.21.15.203:6809/2432038345] conn(0x7feab007b950 0x7feab007ddf0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter put_session s=0x7feab007b280 osd=5 1 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 20 client.42473.objecter shutdown clearing up homeless session... 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 10 client.42473.objecter successfully canceled tick 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feab006f820 msgr2=0x7feab0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7feab006f820 0x7feab0071ce0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7feac0007900 tx=0x7feac0008040 comp rx=0 tx=0).stop 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 msgr2=0x7feac419e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 --2- 172.21.15.143:0/1199078257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac410e830 0x7feac419e7e0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c3d0 tx=0x7feabc00c6e0 comp rx=0 tx=0).stop 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feac9b85700 1 -- 172.21.15.143:0/1199078257 reap_dead start 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 shutdown_connections 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 >> 172.21.15.143:0/1199078257 conn(0x7feac4077d30 msgr2=0x7feac410faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 shutdown_connections 2024-03-11T21:49:46.905 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 -- 172.21.15.143:0/1199078257 wait complete. 2024-03-11T21:49:46.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:49:46.908+0000 7feacb5e8700 1 librados: shutdown 2024-03-11T21:49:47.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.198388+0000 osd.0 (osd.0) 668 : cluster 0 9.5 scrub starts 2024-03-11T21:49:47.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.202270+0000 osd.0 (osd.0) 669 : cluster 0 9.5 scrub ok 2024-03-11T21:49:47.201 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[21512]: cluster 2024-03-11T21:49:45.867603+0000 mgr.y (mgr.14550) 1021 : cluster 0 pgmap v1693: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:47.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.198388+0000 osd.0 (osd.0) 668 : cluster 0 9.5 scrub starts 2024-03-11T21:49:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.202270+0000 osd.0 (osd.0) 669 : cluster 0 9.5 scrub ok 2024-03-11T21:49:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:47 smithi143 bash[29284]: cluster 2024-03-11T21:49:45.867603+0000 mgr.y (mgr.14550) 1021 : cluster 0 pgmap v1693: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:47.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:47 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.198388+0000 osd.0 (osd.0) 668 : cluster 0 9.5 scrub starts 2024-03-11T21:49:47.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:47 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.202270+0000 osd.0 (osd.0) 669 : cluster 0 9.5 scrub ok 2024-03-11T21:49:47.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:47 smithi203 bash[23654]: cluster 2024-03-11T21:49:45.867603+0000 mgr.y (mgr.14550) 1021 : cluster 0 pgmap v1693: 184 pgs: 184 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:48.121 INFO:teuthology.orchestra.run.smithi143.stdout:1267015352539 2024-03-11T21:49:48.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:49:48.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.088812+0000 osd.7 (osd.7) 680 : cluster 0 5.10 scrub starts 2024-03-11T21:49:48.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.091138+0000 osd.7 (osd.7) 681 : cluster 0 5.10 scrub ok 2024-03-11T21:49:48.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.153728+0000 osd.5 (osd.5) 581 : cluster 0 5.16 scrub starts 2024-03-11T21:49:48.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.156133+0000 osd.5 (osd.5) 582 : cluster 0 5.16 scrub ok 2024-03-11T21:49:48.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.088812+0000 osd.7 (osd.7) 680 : cluster 0 5.10 scrub starts 2024-03-11T21:49:48.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.091138+0000 osd.7 (osd.7) 681 : cluster 0 5.10 scrub ok 2024-03-11T21:49:48.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.153728+0000 osd.5 (osd.5) 581 : cluster 0 5.16 scrub starts 2024-03-11T21:49:48.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:48 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.156133+0000 osd.5 (osd.5) 582 : cluster 0 5.16 scrub ok 2024-03-11T21:49:48.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:48 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.088812+0000 osd.7 (osd.7) 680 : cluster 0 5.10 scrub starts 2024-03-11T21:49:48.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:48 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.091138+0000 osd.7 (osd.7) 681 : cluster 0 5.10 scrub ok 2024-03-11T21:49:48.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:48 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.153728+0000 osd.5 (osd.5) 581 : cluster 0 5.16 scrub starts 2024-03-11T21:49:48.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:48 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.156133+0000 osd.5 (osd.5) 582 : cluster 0 5.16 scrub ok 2024-03-11T21:49:49.439 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:49 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.709374+0000 osd.3 (osd.3) 665 : cluster 0 9.19 scrub starts 2024-03-11T21:49:49.439 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:49 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.713358+0000 osd.3 (osd.3) 666 : cluster 0 9.19 scrub ok 2024-03-11T21:49:49.439 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:49 smithi203 bash[23654]: cluster 2024-03-11T21:49:47.869618+0000 mgr.y (mgr.14550) 1022 : cluster 0 pgmap v1694: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:49.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.709374+0000 osd.3 (osd.3) 665 : cluster 0 9.19 scrub starts 2024-03-11T21:49:49.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.713358+0000 osd.3 (osd.3) 666 : cluster 0 9.19 scrub ok 2024-03-11T21:49:49.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[29284]: cluster 2024-03-11T21:49:47.869618+0000 mgr.y (mgr.14550) 1022 : cluster 0 pgmap v1694: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:49.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.709374+0000 osd.3 (osd.3) 665 : cluster 0 9.19 scrub starts 2024-03-11T21:49:49.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.713358+0000 osd.3 (osd.3) 666 : cluster 0 9.19 scrub ok 2024-03-11T21:49:49.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:49 smithi143 bash[21512]: cluster 2024-03-11T21:49:47.869618+0000 mgr.y (mgr.14550) 1022 : cluster 0 pgmap v1694: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:49.535 INFO:teuthology.orchestra.run.smithi143.stdout:150323855721 2024-03-11T21:49:49.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:49:49.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:49:50.097 INFO:teuthology.orchestra.run.smithi143.stdout:283467841868 2024-03-11T21:49:50.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:49:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:50 smithi143 bash[29284]: cluster 2024-03-11T21:49:49.134917+0000 osd.5 (osd.5) 583 : cluster 0 2.12 scrub starts 2024-03-11T21:49:50.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:50 smithi143 bash[29284]: cluster 2024-03-11T21:49:49.137368+0000 osd.5 (osd.5) 584 : cluster 0 2.12 scrub ok 2024-03-11T21:49:50.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:50 smithi143 bash[21512]: cluster 2024-03-11T21:49:49.134917+0000 osd.5 (osd.5) 583 : cluster 0 2.12 scrub starts 2024-03-11T21:49:50.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:50 smithi143 bash[21512]: cluster 2024-03-11T21:49:49.137368+0000 osd.5 (osd.5) 584 : cluster 0 2.12 scrub ok 2024-03-11T21:49:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:50 smithi203 bash[23654]: cluster 2024-03-11T21:49:49.134917+0000 osd.5 (osd.5) 583 : cluster 0 2.12 scrub starts 2024-03-11T21:49:50.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:50 smithi203 bash[23654]: cluster 2024-03-11T21:49:49.137368+0000 osd.5 (osd.5) 584 : cluster 0 2.12 scrub ok 2024-03-11T21:49:50.681 INFO:teuthology.orchestra.run.smithi143.stdout:197568495967 2024-03-11T21:49:50.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:49:51.114 INFO:teuthology.orchestra.run.smithi143.stdout:115964117363 2024-03-11T21:49:51.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:49:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[29284]: audit 2024-03-11T21:49:49.449328+0000 mgr.y (mgr.14550) 1023 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[29284]: cluster 2024-03-11T21:49:49.685694+0000 osd.3 (osd.3) 667 : cluster 0 9.8 scrub starts 2024-03-11T21:49:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[29284]: cluster 2024-03-11T21:49:49.688115+0000 osd.3 (osd.3) 668 : cluster 0 9.8 scrub ok 2024-03-11T21:49:51.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[29284]: cluster 2024-03-11T21:49:49.871483+0000 mgr.y (mgr.14550) 1024 : cluster 0 pgmap v1695: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:51.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[21512]: audit 2024-03-11T21:49:49.449328+0000 mgr.y (mgr.14550) 1023 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:51.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[21512]: cluster 2024-03-11T21:49:49.685694+0000 osd.3 (osd.3) 667 : cluster 0 9.8 scrub starts 2024-03-11T21:49:51.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[21512]: cluster 2024-03-11T21:49:49.688115+0000 osd.3 (osd.3) 668 : cluster 0 9.8 scrub ok 2024-03-11T21:49:51.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:51 smithi143 bash[21512]: cluster 2024-03-11T21:49:49.871483+0000 mgr.y (mgr.14550) 1024 : cluster 0 pgmap v1695: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:51.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:51 smithi203 bash[23654]: audit 2024-03-11T21:49:49.449328+0000 mgr.y (mgr.14550) 1023 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:51.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:51 smithi203 bash[23654]: cluster 2024-03-11T21:49:49.685694+0000 osd.3 (osd.3) 667 : cluster 0 9.8 scrub starts 2024-03-11T21:49:51.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:51 smithi203 bash[23654]: cluster 2024-03-11T21:49:49.688115+0000 osd.3 (osd.3) 668 : cluster 0 9.8 scrub ok 2024-03-11T21:49:51.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:51 smithi203 bash[23654]: cluster 2024-03-11T21:49:49.871483+0000 mgr.y (mgr.14550) 1024 : cluster 0 pgmap v1695: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:52.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:49:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:52.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.285487+0000 osd.0 (osd.0) 670 : cluster 0 9.b scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.290395+0000 osd.0 (osd.0) 671 : cluster 0 9.b scrub ok 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.555727+0000 osd.6 (osd.6) 527 : cluster 0 5.2 scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.557888+0000 osd.6 (osd.6) 528 : cluster 0 5.2 scrub ok 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.728550+0000 osd.3 (osd.3) 669 : cluster 0 9.1a deep-scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.730250+0000 osd.3 (osd.3) 670 : cluster 0 9.1a deep-scrub ok 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.801640+0000 osd.3 (osd.3) 671 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:50.802046+0000 osd.3 (osd.3) 672 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.095184+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e1014: 8 total, 8 up, 8 in 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.204274+0000 osd.5 (osd.5) 585 : cluster 0 3.1c scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.210956+0000 osd.5 (osd.5) 586 : cluster 0 3.1c scrub ok 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.285487+0000 osd.0 (osd.0) 670 : cluster 0 9.b scrub starts 2024-03-11T21:49:52.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.290395+0000 osd.0 (osd.0) 671 : cluster 0 9.b scrub ok 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.555727+0000 osd.6 (osd.6) 527 : cluster 0 5.2 scrub starts 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.557888+0000 osd.6 (osd.6) 528 : cluster 0 5.2 scrub ok 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.728550+0000 osd.3 (osd.3) 669 : cluster 0 9.1a deep-scrub starts 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.730250+0000 osd.3 (osd.3) 670 : cluster 0 9.1a deep-scrub ok 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.801640+0000 osd.3 (osd.3) 671 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:50.802046+0000 osd.3 (osd.3) 672 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.095184+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e1014: 8 total, 8 up, 8 in 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.204274+0000 osd.5 (osd.5) 585 : cluster 0 3.1c scrub starts 2024-03-11T21:49:52.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:52 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.210956+0000 osd.5 (osd.5) 586 : cluster 0 3.1c scrub ok 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.285487+0000 osd.0 (osd.0) 670 : cluster 0 9.b scrub starts 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.290395+0000 osd.0 (osd.0) 671 : cluster 0 9.b scrub ok 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.555727+0000 osd.6 (osd.6) 527 : cluster 0 5.2 scrub starts 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.557888+0000 osd.6 (osd.6) 528 : cluster 0 5.2 scrub ok 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.728550+0000 osd.3 (osd.3) 669 : cluster 0 9.1a deep-scrub starts 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.730250+0000 osd.3 (osd.3) 670 : cluster 0 9.1a deep-scrub ok 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.801640+0000 osd.3 (osd.3) 671 : cluster 0 purged_snaps scrub starts 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:50.802046+0000 osd.3 (osd.3) 672 : cluster 0 purged_snaps scrub ok 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.095184+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e1014: 8 total, 8 up, 8 in 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.204274+0000 osd.5 (osd.5) 585 : cluster 0 3.1c scrub starts 2024-03-11T21:49:52.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:52 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.210956+0000 osd.5 (osd.5) 586 : cluster 0 3.1c scrub ok 2024-03-11T21:49:53.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.756364+0000 osd.3 (osd.3) 673 : cluster 0 3.18 scrub starts 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.760040+0000 osd.3 (osd.3) 674 : cluster 0 3.18 scrub ok 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.840577+0000 osd.1 (osd.1) 469 : cluster 0 2.6 scrub starts 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.842963+0000 osd.1 (osd.1) 470 : cluster 0 2.6 scrub ok 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:51.872197+0000 mgr.y (mgr.14550) 1025 : cluster 0 pgmap v1697: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.043110+0000 osd.7 (osd.7) 682 : cluster 0 3.e scrub starts 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.047259+0000 osd.7 (osd.7) 683 : cluster 0 3.e scrub ok 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.090559+0000 osd.4 (osd.4) 631 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.092935+0000 osd.4 (osd.4) 632 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.424867+0000 osd.2 (osd.2) 539 : cluster 0 2.c scrub starts 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.427271+0000 osd.2 (osd.2) 540 : cluster 0 2.c scrub ok 2024-03-11T21:49:53.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.756364+0000 osd.3 (osd.3) 673 : cluster 0 3.18 scrub starts 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.760040+0000 osd.3 (osd.3) 674 : cluster 0 3.18 scrub ok 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.840577+0000 osd.1 (osd.1) 469 : cluster 0 2.6 scrub starts 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.842963+0000 osd.1 (osd.1) 470 : cluster 0 2.6 scrub ok 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:51.872197+0000 mgr.y (mgr.14550) 1025 : cluster 0 pgmap v1697: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.043110+0000 osd.7 (osd.7) 682 : cluster 0 3.e scrub starts 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.047259+0000 osd.7 (osd.7) 683 : cluster 0 3.e scrub ok 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.090559+0000 osd.4 (osd.4) 631 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.092935+0000 osd.4 (osd.4) 632 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.424867+0000 osd.2 (osd.2) 539 : cluster 0 2.c scrub starts 2024-03-11T21:49:53.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:53 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.427271+0000 osd.2 (osd.2) 540 : cluster 0 2.c scrub ok 2024-03-11T21:49:53.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.756364+0000 osd.3 (osd.3) 673 : cluster 0 3.18 scrub starts 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.760040+0000 osd.3 (osd.3) 674 : cluster 0 3.18 scrub ok 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.840577+0000 osd.1 (osd.1) 469 : cluster 0 2.6 scrub starts 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.842963+0000 osd.1 (osd.1) 470 : cluster 0 2.6 scrub ok 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:51.872197+0000 mgr.y (mgr.14550) 1025 : cluster 0 pgmap v1697: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.043110+0000 osd.7 (osd.7) 682 : cluster 0 3.e scrub starts 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.047259+0000 osd.7 (osd.7) 683 : cluster 0 3.e scrub ok 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.090559+0000 osd.4 (osd.4) 631 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.092935+0000 osd.4 (osd.4) 632 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.424867+0000 osd.2 (osd.2) 539 : cluster 0 2.c scrub starts 2024-03-11T21:49:53.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:53 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.427271+0000 osd.2 (osd.2) 540 : cluster 0 2.c scrub ok 2024-03-11T21:49:53.773 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:53.773 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:53.773 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:53.774 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:53.774 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.282588+0000 osd.0 (osd.0) 672 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:52.285319+0000 osd.0 (osd.0) 673 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.073281+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.075991+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.200171+0000 osd.5 (osd.5) 587 : cluster 0 2.16 scrub starts 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.202480+0000 osd.5 (osd.5) 588 : cluster 0 2.16 scrub ok 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: audit 2024-03-11T21:49:53.347078+0000 mon.c (mon.2) 1426 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.414323+0000 osd.2 (osd.2) 541 : cluster 0 5.6 scrub starts 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.416746+0000 osd.2 (osd.2) 542 : cluster 0 5.6 scrub ok 2024-03-11T21:49:54.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.282588+0000 osd.0 (osd.0) 672 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:52.285319+0000 osd.0 (osd.0) 673 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.073281+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.075991+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.200171+0000 osd.5 (osd.5) 587 : cluster 0 2.16 scrub starts 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.202480+0000 osd.5 (osd.5) 588 : cluster 0 2.16 scrub ok 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: audit 2024-03-11T21:49:53.347078+0000 mon.c (mon.2) 1426 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.414323+0000 osd.2 (osd.2) 541 : cluster 0 5.6 scrub starts 2024-03-11T21:49:54.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:54 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.416746+0000 osd.2 (osd.2) 542 : cluster 0 5.6 scrub ok 2024-03-11T21:49:54.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.282588+0000 osd.0 (osd.0) 672 : cluster 0 5.3 deep-scrub starts 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:52.285319+0000 osd.0 (osd.0) 673 : cluster 0 5.3 deep-scrub ok 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.073281+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.075991+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.200171+0000 osd.5 (osd.5) 587 : cluster 0 2.16 scrub starts 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.202480+0000 osd.5 (osd.5) 588 : cluster 0 2.16 scrub ok 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: audit 2024-03-11T21:49:53.347078+0000 mon.c (mon.2) 1426 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.414323+0000 osd.2 (osd.2) 541 : cluster 0 5.6 scrub starts 2024-03-11T21:49:54.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:54 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.416746+0000 osd.2 (osd.2) 542 : cluster 0 5.6 scrub ok 2024-03-11T21:49:54.791 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:54.791 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:55.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.802381+0000 osd.1 (osd.1) 471 : cluster 0 9.14 deep-scrub starts 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.805348+0000 osd.1 (osd.1) 472 : cluster 0 9.14 deep-scrub ok 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[21512]: cluster 2024-03-11T21:49:53.873342+0000 mgr.y (mgr.14550) 1026 : cluster 0 pgmap v1698: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[21512]: cluster 2024-03-11T21:49:54.423902+0000 osd.2 (osd.2) 543 : cluster 0 5.b scrub starts 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[21512]: cluster 2024-03-11T21:49:54.426435+0000 osd.2 (osd.2) 544 : cluster 0 5.b scrub ok 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.802381+0000 osd.1 (osd.1) 471 : cluster 0 9.14 deep-scrub starts 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.805348+0000 osd.1 (osd.1) 472 : cluster 0 9.14 deep-scrub ok 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[29284]: cluster 2024-03-11T21:49:53.873342+0000 mgr.y (mgr.14550) 1026 : cluster 0 pgmap v1698: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[29284]: cluster 2024-03-11T21:49:54.423902+0000 osd.2 (osd.2) 543 : cluster 0 5.b scrub starts 2024-03-11T21:49:55.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:55 smithi143 bash[29284]: cluster 2024-03-11T21:49:54.426435+0000 osd.2 (osd.2) 544 : cluster 0 5.b scrub ok 2024-03-11T21:49:55.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:55 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.802381+0000 osd.1 (osd.1) 471 : cluster 0 9.14 deep-scrub starts 2024-03-11T21:49:55.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:55 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.805348+0000 osd.1 (osd.1) 472 : cluster 0 9.14 deep-scrub ok 2024-03-11T21:49:55.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:55 smithi203 bash[23654]: cluster 2024-03-11T21:49:53.873342+0000 mgr.y (mgr.14550) 1026 : cluster 0 pgmap v1698: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:55.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:55 smithi203 bash[23654]: cluster 2024-03-11T21:49:54.423902+0000 osd.2 (osd.2) 543 : cluster 0 5.b scrub starts 2024-03-11T21:49:55.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:55 smithi203 bash[23654]: cluster 2024-03-11T21:49:54.426435+0000 osd.2 (osd.2) 544 : cluster 0 5.b scrub ok 2024-03-11T21:49:55.802 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:56.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:56 smithi143 bash[29284]: cluster 2024-03-11T21:49:54.519048+0000 osd.6 (osd.6) 529 : cluster 0 2.14 scrub starts 2024-03-11T21:49:56.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:56 smithi143 bash[29284]: cluster 2024-03-11T21:49:54.521359+0000 osd.6 (osd.6) 530 : cluster 0 2.14 scrub ok 2024-03-11T21:49:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:56 smithi143 bash[21512]: cluster 2024-03-11T21:49:54.519048+0000 osd.6 (osd.6) 529 : cluster 0 2.14 scrub starts 2024-03-11T21:49:56.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:56 smithi143 bash[21512]: cluster 2024-03-11T21:49:54.521359+0000 osd.6 (osd.6) 530 : cluster 0 2.14 scrub ok 2024-03-11T21:49:56.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:56 smithi203 bash[23654]: cluster 2024-03-11T21:49:54.519048+0000 osd.6 (osd.6) 529 : cluster 0 2.14 scrub starts 2024-03-11T21:49:56.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:56 smithi203 bash[23654]: cluster 2024-03-11T21:49:54.521359+0000 osd.6 (osd.6) 530 : cluster 0 2.14 scrub ok 2024-03-11T21:49:57.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:57 smithi143 bash[21512]: cluster 2024-03-11T21:49:55.873946+0000 mgr.y (mgr.14550) 1027 : cluster 0 pgmap v1699: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:57.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:57 smithi143 bash[29284]: cluster 2024-03-11T21:49:55.873946+0000 mgr.y (mgr.14550) 1027 : cluster 0 pgmap v1699: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:57.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:57 smithi203 bash[23654]: cluster 2024-03-11T21:49:55.873946+0000 mgr.y (mgr.14550) 1027 : cluster 0 pgmap v1699: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:49:58.396 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[21512]: cluster 2024-03-11T21:49:56.828845+0000 osd.1 (osd.1) 473 : cluster 0 5.1d scrub starts 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[21512]: cluster 2024-03-11T21:49:56.831164+0000 osd.1 (osd.1) 474 : cluster 0 5.1d scrub ok 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.102172+0000 osd.4 (osd.4) 635 : cluster 0 1.5 scrub starts 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.105021+0000 osd.4 (osd.4) 636 : cluster 0 1.5 scrub ok 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[29284]: cluster 2024-03-11T21:49:56.828845+0000 osd.1 (osd.1) 473 : cluster 0 5.1d scrub starts 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[29284]: cluster 2024-03-11T21:49:56.831164+0000 osd.1 (osd.1) 474 : cluster 0 5.1d scrub ok 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.102172+0000 osd.4 (osd.4) 635 : cluster 0 1.5 scrub starts 2024-03-11T21:49:58.397 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:58 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.105021+0000 osd.4 (osd.4) 636 : cluster 0 1.5 scrub ok 2024-03-11T21:49:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:58 smithi203 bash[23654]: cluster 2024-03-11T21:49:56.828845+0000 osd.1 (osd.1) 473 : cluster 0 5.1d scrub starts 2024-03-11T21:49:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:58 smithi203 bash[23654]: cluster 2024-03-11T21:49:56.831164+0000 osd.1 (osd.1) 474 : cluster 0 5.1d scrub ok 2024-03-11T21:49:58.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:58 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.102172+0000 osd.4 (osd.4) 635 : cluster 0 1.5 scrub starts 2024-03-11T21:49:58.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:58 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.105021+0000 osd.4 (osd.4) 636 : cluster 0 1.5 scrub ok 2024-03-11T21:49:59.378 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.842042+0000 osd.1 (osd.1) 475 : cluster 0 2.d scrub starts 2024-03-11T21:49:59.378 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.843879+0000 osd.1 (osd.1) 476 : cluster 0 2.d scrub ok 2024-03-11T21:49:59.378 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.874663+0000 mgr.y (mgr.14550) 1028 : cluster 0 pgmap v1700: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:49:59.378 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.938719+0000 osd.7 (osd.7) 684 : cluster 0 1.d scrub starts 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:57.941288+0000 osd.7 (osd.7) 685 : cluster 0 1.d scrub ok 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:58.751281+0000 osd.3 (osd.3) 675 : cluster 0 3.b scrub starts 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[29284]: cluster 2024-03-11T21:49:58.753530+0000 osd.3 (osd.3) 676 : cluster 0 3.b scrub ok 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.842042+0000 osd.1 (osd.1) 475 : cluster 0 2.d scrub starts 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.843879+0000 osd.1 (osd.1) 476 : cluster 0 2.d scrub ok 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.874663+0000 mgr.y (mgr.14550) 1028 : cluster 0 pgmap v1700: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.938719+0000 osd.7 (osd.7) 684 : cluster 0 1.d scrub starts 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:57.941288+0000 osd.7 (osd.7) 685 : cluster 0 1.d scrub ok 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:58.751281+0000 osd.3 (osd.3) 675 : cluster 0 3.b scrub starts 2024-03-11T21:49:59.379 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:49:59 smithi143 bash[21512]: cluster 2024-03-11T21:49:58.753530+0000 osd.3 (osd.3) 676 : cluster 0 3.b scrub ok 2024-03-11T21:49:59.450 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.842042+0000 osd.1 (osd.1) 475 : cluster 0 2.d scrub starts 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.843879+0000 osd.1 (osd.1) 476 : cluster 0 2.d scrub ok 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.874663+0000 mgr.y (mgr.14550) 1028 : cluster 0 pgmap v1700: 184 pgs: 184 active+clean; 582 KiB data, 779 MiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.938719+0000 osd.7 (osd.7) 684 : cluster 0 1.d scrub starts 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:57.941288+0000 osd.7 (osd.7) 685 : cluster 0 1.d scrub ok 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:58.751281+0000 osd.3 (osd.3) 675 : cluster 0 3.b scrub starts 2024-03-11T21:49:59.451 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[23654]: cluster 2024-03-11T21:49:58.753530+0000 osd.3 (osd.3) 676 : cluster 0 3.b scrub ok 2024-03-11T21:49:59.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:49:59 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:00.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.169139+0000 osd.5 (osd.5) 589 : cluster 0 6.a scrub starts 2024-03-11T21:50:00.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.171376+0000 osd.5 (osd.5) 590 : cluster 0 6.a scrub ok 2024-03-11T21:50:00.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.328225+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-03-11T21:50:00.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.331138+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.000131+0000 mon.a (mon.0) 4171 : cluster 3 Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.000177+0000 mon.a (mon.0) 4172 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.000197+0000 mon.a (mon.0) 4173 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.000220+0000 mon.a (mon.0) 4174 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.169139+0000 osd.5 (osd.5) 589 : cluster 0 6.a scrub starts 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.171376+0000 osd.5 (osd.5) 590 : cluster 0 6.a scrub ok 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.328225+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.331138+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.000131+0000 mon.a (mon.0) 4171 : cluster 3 Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.000177+0000 mon.a (mon.0) 4172 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.000197+0000 mon.a (mon.0) 4173 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:50:00.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:00 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.000220+0000 mon.a (mon.0) 4174 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:50:00.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.169139+0000 osd.5 (osd.5) 589 : cluster 0 6.a scrub starts 2024-03-11T21:50:00.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.171376+0000 osd.5 (osd.5) 590 : cluster 0 6.a scrub ok 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.328225+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.331138+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.000131+0000 mon.a (mon.0) 4171 : cluster 3 Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.000177+0000 mon.a (mon.0) 4172 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.000197+0000 mon.a (mon.0) 4173 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:50:00.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:00 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.000220+0000 mon.a (mon.0) 4174 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:50:01.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: audit 2024-03-11T21:49:59.460613+0000 mgr.y (mgr.14550) 1029 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.839459+0000 osd.1 (osd.1) 477 : cluster 0 1.3 scrub starts 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.842072+0000 osd.1 (osd.1) 478 : cluster 0 1.3 scrub ok 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: cluster 2024-03-11T21:49:59.876627+0000 mgr.y (mgr.14550) 1030 : cluster 0 pgmap v1701: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.102715+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[29284]: cluster 2024-03-11T21:50:00.105394+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: audit 2024-03-11T21:49:59.460613+0000 mgr.y (mgr.14550) 1029 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.839459+0000 osd.1 (osd.1) 477 : cluster 0 1.3 scrub starts 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.842072+0000 osd.1 (osd.1) 478 : cluster 0 1.3 scrub ok 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: cluster 2024-03-11T21:49:59.876627+0000 mgr.y (mgr.14550) 1030 : cluster 0 pgmap v1701: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:01.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.102715+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-03-11T21:50:01.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:01 smithi143 bash[21512]: cluster 2024-03-11T21:50:00.105394+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-03-11T21:50:01.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: audit 2024-03-11T21:49:59.460613+0000 mgr.y (mgr.14550) 1029 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:01.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.839459+0000 osd.1 (osd.1) 477 : cluster 0 1.3 scrub starts 2024-03-11T21:50:01.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.842072+0000 osd.1 (osd.1) 478 : cluster 0 1.3 scrub ok 2024-03-11T21:50:01.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: cluster 2024-03-11T21:49:59.876627+0000 mgr.y (mgr.14550) 1030 : cluster 0 pgmap v1701: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:01.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.102715+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-03-11T21:50:01.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:01 smithi203 bash[23654]: cluster 2024-03-11T21:50:00.105394+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-03-11T21:50:02.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:02 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:02.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:02 smithi143 bash[21512]: cluster 2024-03-11T21:50:01.222605+0000 osd.5 (osd.5) 591 : cluster 0 2.3 scrub starts 2024-03-11T21:50:02.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:02 smithi143 bash[21512]: cluster 2024-03-11T21:50:01.225002+0000 osd.5 (osd.5) 592 : cluster 0 2.3 scrub ok 2024-03-11T21:50:02.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:02 smithi143 bash[29284]: cluster 2024-03-11T21:50:01.222605+0000 osd.5 (osd.5) 591 : cluster 0 2.3 scrub starts 2024-03-11T21:50:02.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:02 smithi143 bash[29284]: cluster 2024-03-11T21:50:01.225002+0000 osd.5 (osd.5) 592 : cluster 0 2.3 scrub ok 2024-03-11T21:50:02.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:02 smithi203 bash[23654]: cluster 2024-03-11T21:50:01.222605+0000 osd.5 (osd.5) 591 : cluster 0 2.3 scrub starts 2024-03-11T21:50:02.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:02 smithi203 bash[23654]: cluster 2024-03-11T21:50:01.225002+0000 osd.5 (osd.5) 592 : cluster 0 2.3 scrub ok 2024-03-11T21:50:03.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:01.516687+0000 osd.6 (osd.6) 531 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:01.519015+0000 osd.6 (osd.6) 532 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:01.877426+0000 mgr.y (mgr.14550) 1031 : cluster 0 pgmap v1702: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.061088+0000 osd.4 (osd.4) 639 : cluster 0 1.e scrub starts 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.063428+0000 osd.4 (osd.4) 640 : cluster 0 1.e scrub ok 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.807842+0000 osd.3 (osd.3) 677 : cluster 0 3.1e scrub starts 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.812277+0000 osd.3 (osd.3) 678 : cluster 0 3.1e scrub ok 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:01.516687+0000 osd.6 (osd.6) 531 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:01.519015+0000 osd.6 (osd.6) 532 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:50:03.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:01.877426+0000 mgr.y (mgr.14550) 1031 : cluster 0 pgmap v1702: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-11T21:50:03.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.061088+0000 osd.4 (osd.4) 639 : cluster 0 1.e scrub starts 2024-03-11T21:50:03.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.063428+0000 osd.4 (osd.4) 640 : cluster 0 1.e scrub ok 2024-03-11T21:50:03.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.807842+0000 osd.3 (osd.3) 677 : cluster 0 3.1e scrub starts 2024-03-11T21:50:03.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:03 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.812277+0000 osd.3 (osd.3) 678 : cluster 0 3.1e scrub ok 2024-03-11T21:50:03.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:01.516687+0000 osd.6 (osd.6) 531 : cluster 0 5.11 deep-scrub starts 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:01.519015+0000 osd.6 (osd.6) 532 : cluster 0 5.11 deep-scrub ok 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:01.877426+0000 mgr.y (mgr.14550) 1031 : cluster 0 pgmap v1702: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.061088+0000 osd.4 (osd.4) 639 : cluster 0 1.e scrub starts 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.063428+0000 osd.4 (osd.4) 640 : cluster 0 1.e scrub ok 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.807842+0000 osd.3 (osd.3) 677 : cluster 0 3.1e scrub starts 2024-03-11T21:50:03.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:03 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.812277+0000 osd.3 (osd.3) 678 : cluster 0 3.1e scrub ok 2024-03-11T21:50:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:04 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.954018+0000 osd.7 (osd.7) 686 : cluster 0 2.1d scrub starts 2024-03-11T21:50:04.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:04 smithi143 bash[29284]: cluster 2024-03-11T21:50:02.956321+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub ok 2024-03-11T21:50:04.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:04 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.954018+0000 osd.7 (osd.7) 686 : cluster 0 2.1d scrub starts 2024-03-11T21:50:04.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:04 smithi143 bash[21512]: cluster 2024-03-11T21:50:02.956321+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub ok 2024-03-11T21:50:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:04 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.954018+0000 osd.7 (osd.7) 686 : cluster 0 2.1d scrub starts 2024-03-11T21:50:04.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:04 smithi203 bash[23654]: cluster 2024-03-11T21:50:02.956321+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub ok 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:03.868641+0000 osd.1 (osd.1) 479 : cluster 0 6.4 scrub starts 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:03.870870+0000 osd.1 (osd.1) 480 : cluster 0 6.4 scrub ok 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:03.878891+0000 mgr.y (mgr.14550) 1032 : cluster 0 pgmap v1703: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:03.971855+0000 osd.7 (osd.7) 688 : cluster 0 9.15 scrub starts 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:03.975810+0000 osd.7 (osd.7) 689 : cluster 0 9.15 scrub ok 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.163876+0000 osd.5 (osd.5) 593 : cluster 0 1.10 scrub starts 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.167201+0000 osd.5 (osd.5) 594 : cluster 0 1.10 scrub ok 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.446810+0000 osd.0 (osd.0) 676 : cluster 0 6.16 scrub starts 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.449082+0000 osd.0 (osd.0) 677 : cluster 0 6.16 scrub ok 2024-03-11T21:50:05.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.819677+0000 osd.3 (osd.3) 679 : cluster 0 1.c scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[21512]: cluster 2024-03-11T21:50:04.822664+0000 osd.3 (osd.3) 680 : cluster 0 1.c scrub ok 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:03.868641+0000 osd.1 (osd.1) 479 : cluster 0 6.4 scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:03.870870+0000 osd.1 (osd.1) 480 : cluster 0 6.4 scrub ok 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:03.878891+0000 mgr.y (mgr.14550) 1032 : cluster 0 pgmap v1703: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:03.971855+0000 osd.7 (osd.7) 688 : cluster 0 9.15 scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:03.975810+0000 osd.7 (osd.7) 689 : cluster 0 9.15 scrub ok 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.163876+0000 osd.5 (osd.5) 593 : cluster 0 1.10 scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.167201+0000 osd.5 (osd.5) 594 : cluster 0 1.10 scrub ok 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.446810+0000 osd.0 (osd.0) 676 : cluster 0 6.16 scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.449082+0000 osd.0 (osd.0) 677 : cluster 0 6.16 scrub ok 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.819677+0000 osd.3 (osd.3) 679 : cluster 0 1.c scrub starts 2024-03-11T21:50:05.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:05 smithi143 bash[29284]: cluster 2024-03-11T21:50:04.822664+0000 osd.3 (osd.3) 680 : cluster 0 1.c scrub ok 2024-03-11T21:50:05.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:03.868641+0000 osd.1 (osd.1) 479 : cluster 0 6.4 scrub starts 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:03.870870+0000 osd.1 (osd.1) 480 : cluster 0 6.4 scrub ok 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:03.878891+0000 mgr.y (mgr.14550) 1032 : cluster 0 pgmap v1703: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:03.971855+0000 osd.7 (osd.7) 688 : cluster 0 9.15 scrub starts 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:03.975810+0000 osd.7 (osd.7) 689 : cluster 0 9.15 scrub ok 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.163876+0000 osd.5 (osd.5) 593 : cluster 0 1.10 scrub starts 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.167201+0000 osd.5 (osd.5) 594 : cluster 0 1.10 scrub ok 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.446810+0000 osd.0 (osd.0) 676 : cluster 0 6.16 scrub starts 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.449082+0000 osd.0 (osd.0) 677 : cluster 0 6.16 scrub ok 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.819677+0000 osd.3 (osd.3) 679 : cluster 0 1.c scrub starts 2024-03-11T21:50:05.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:05 smithi203 bash[23654]: cluster 2024-03-11T21:50:04.822664+0000 osd.3 (osd.3) 680 : cluster 0 1.c scrub ok 2024-03-11T21:50:06.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:06 smithi143 bash[21512]: cluster 2024-03-11T21:50:05.447270+0000 osd.0 (osd.0) 678 : cluster 0 9.1e scrub starts 2024-03-11T21:50:06.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:06 smithi143 bash[21512]: cluster 2024-03-11T21:50:05.452146+0000 osd.0 (osd.0) 679 : cluster 0 9.1e scrub ok 2024-03-11T21:50:06.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:06 smithi143 bash[29284]: cluster 2024-03-11T21:50:05.447270+0000 osd.0 (osd.0) 678 : cluster 0 9.1e scrub starts 2024-03-11T21:50:06.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:06 smithi143 bash[29284]: cluster 2024-03-11T21:50:05.452146+0000 osd.0 (osd.0) 679 : cluster 0 9.1e scrub ok 2024-03-11T21:50:06.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:06 smithi203 bash[23654]: cluster 2024-03-11T21:50:05.447270+0000 osd.0 (osd.0) 678 : cluster 0 9.1e scrub starts 2024-03-11T21:50:06.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:06 smithi203 bash[23654]: cluster 2024-03-11T21:50:05.452146+0000 osd.0 (osd.0) 679 : cluster 0 9.1e scrub ok 2024-03-11T21:50:07.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[21512]: cluster 2024-03-11T21:50:05.879816+0000 mgr.y (mgr.14550) 1033 : cluster 0 pgmap v1704: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:07.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[21512]: cluster 2024-03-11T21:50:05.997360+0000 osd.7 (osd.7) 690 : cluster 0 6.5 scrub starts 2024-03-11T21:50:07.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[21512]: cluster 2024-03-11T21:50:05.999699+0000 osd.7 (osd.7) 691 : cluster 0 6.5 scrub ok 2024-03-11T21:50:07.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[21512]: cluster 2024-03-11T21:50:06.069433+0000 osd.4 (osd.4) 641 : cluster 0 5.18 scrub starts 2024-03-11T21:50:07.434 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[21512]: cluster 2024-03-11T21:50:06.071767+0000 osd.4 (osd.4) 642 : cluster 0 5.18 scrub ok 2024-03-11T21:50:07.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[29284]: cluster 2024-03-11T21:50:05.879816+0000 mgr.y (mgr.14550) 1033 : cluster 0 pgmap v1704: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:07.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[29284]: cluster 2024-03-11T21:50:05.997360+0000 osd.7 (osd.7) 690 : cluster 0 6.5 scrub starts 2024-03-11T21:50:07.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[29284]: cluster 2024-03-11T21:50:05.999699+0000 osd.7 (osd.7) 691 : cluster 0 6.5 scrub ok 2024-03-11T21:50:07.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[29284]: cluster 2024-03-11T21:50:06.069433+0000 osd.4 (osd.4) 641 : cluster 0 5.18 scrub starts 2024-03-11T21:50:07.435 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:07 smithi143 bash[29284]: cluster 2024-03-11T21:50:06.071767+0000 osd.4 (osd.4) 642 : cluster 0 5.18 scrub ok 2024-03-11T21:50:07.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:07 smithi203 bash[23654]: cluster 2024-03-11T21:50:05.879816+0000 mgr.y (mgr.14550) 1033 : cluster 0 pgmap v1704: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:07 smithi203 bash[23654]: cluster 2024-03-11T21:50:05.997360+0000 osd.7 (osd.7) 690 : cluster 0 6.5 scrub starts 2024-03-11T21:50:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:07 smithi203 bash[23654]: cluster 2024-03-11T21:50:05.999699+0000 osd.7 (osd.7) 691 : cluster 0 6.5 scrub ok 2024-03-11T21:50:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:07 smithi203 bash[23654]: cluster 2024-03-11T21:50:06.069433+0000 osd.4 (osd.4) 641 : cluster 0 5.18 scrub starts 2024-03-11T21:50:07.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:07 smithi203 bash[23654]: cluster 2024-03-11T21:50:06.071767+0000 osd.4 (osd.4) 642 : cluster 0 5.18 scrub ok 2024-03-11T21:50:07.588 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- 172.21.15.143:0/3474082518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 msgr2=0x7fa22c110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 --2- 172.21.15.143:0/3474082518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c110f30 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7fa2240099c0 tx=0x7fa224009cd0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- 172.21.15.143:0/3474082518 shutdown_connections 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 --2- 172.21.15.143:0/3474082518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c110f30 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 --2- 172.21.15.143:0/3474082518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104280 0x7fa22c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 --2- 172.21.15.143:0/3474082518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104c30 0x7fa22c108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- 172.21.15.143:0/3474082518 >> 172.21.15.143:0/3474082518 conn(0x7fa22c0ffb10 msgr2=0x7fa22c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- 172.21.15.143:0/3474082518 shutdown_connections 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- 172.21.15.143:0/3474082518 wait complete. 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 librados: starting msgr at 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 librados: starting objecter 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 Processor -- start 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 -- start start 2024-03-11T21:50:07.589 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 librados: setting wanted keys 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.595+0000 7fa23298a700 1 librados: calling monclient init 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 0x7fa22c199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 0x7fa22c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c113880 con 0x7fa22c104280 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c113700 con 0x7fa22c104c30 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c113a00 con 0x7fa22c109360 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa22bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 0x7fa22c199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38748/0 (socket says 172.21.15.143:38748) 2024-03-11T21:50:07.590 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 -- 172.21.15.143:0/1991580612 learned_addr learned my addr 172.21.15.143:0/1991580612 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:07.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa22b7fe700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 0x7fa22c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:07.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 -- 172.21.15.143:0/1991580612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 msgr2=0x7fa22c19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:07.592 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 0x7fa22c19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 -- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 msgr2=0x7fa22c199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 0x7fa22c199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 -- 172.21.15.143:0/1991580612 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c19ee10 con 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa22bfff700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 0x7fa22c199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa22b7fe700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 0x7fa22c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa230f27700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fa224005910 tx=0x7fa224003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 10 client.?.objecter ms_handle_connect 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224009e60 con 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fa22400f460 con 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224017650 con 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22c19f0a0 con 0x7fa22c109360 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 10 client.42479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:07.593 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22c1a6980 con 0x7fa22c109360 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 librados: init done 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22c04faa0 con 0x7fa22c109360 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa22400f980 con 0x7fa22c109360 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 0x7fa218071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1014..1014 src has 964..1014) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fa22408f550 con 0x7fa22c109360 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 10 client.42479.objecter ms_dispatch 0x7fa22c001550 osd_map(1014..1014 src has 964..1014) v4 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 3 client.42479.objecter handle_osd_map got epochs [1014,1014] > 0 2024-03-11T21:50:07.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 3 client.42479.objecter handle_osd_map decoding full epoch 1014 2024-03-11T21:50:07.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa22bfff700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 0x7fa218071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:07.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.599+0000 7fa2297fa700 20 client.42479.objecter dump_active .. 0 homeless 2024-03-11T21:50:07.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.603+0000 7fa22bfff700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 0x7fa218071ce0 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00a7b0 tx=0x7fa21c00d040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:07.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.603+0000 7fa2297fa700 10 client.42479.objecter ms_handle_connect 0x7fa21806f820 2024-03-11T21:50:07.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.607+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa224013070 con 0x7fa22c109360 2024-03-11T21:50:07.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.811+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa22c1a6b50 con 0x7fa22c109360 2024-03-11T21:50:07.803 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.811+0000 7fa2297fa700 1 -- 172.21.15.143:0/1991580612 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa22405c560 con 0x7fa22c109360 2024-03-11T21:50:07.803 INFO:teuthology.orchestra.run.smithi143.stdout:240518168921 2024-03-11T21:50:07.805 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 10 librados: watch_flush enter 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 10 librados: watch_flush exit 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 20 client.42479.objecter shutdown clearing up homeless session... 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 10 client.42479.objecter successfully canceled tick 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 msgr2=0x7fa218071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 0x7fa218071ce0 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00a7b0 tx=0x7fa21c00d040 comp rx=0 tx=0).stop 2024-03-11T21:50:07.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 msgr2=0x7fa22c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fa224005910 tx=0x7fa224003da0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 shutdown_connections 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fa21806f820 0x7fa218071ce0 unknown :-1 s=CLOSED pgs=3523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa22c104280 0x7fa22c199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa22c109360 0x7fa22c19e6d0 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 --2- 172.21.15.143:0/1991580612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa22c104c30 0x7fa22c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 >> 172.21.15.143:0/1991580612 conn(0x7fa22c0ffb10 msgr2=0x7fa22c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 shutdown_connections 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 -- 172.21.15.143:0/1991580612 wait complete. 2024-03-11T21:50:07.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:07.815+0000 7fa23298a700 1 librados: shutdown 2024-03-11T21:50:08.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 -- 172.21.15.143:0/700071931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff028108dd0 msgr2=0x7ff0281152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 --2- 172.21.15.143:0/700071931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff028108dd0 0x7ff0281152e0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7ff02000b0a0 tx=0x7ff02000b3b0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 -- 172.21.15.143:0/700071931 shutdown_connections 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 --2- 172.21.15.143:0/700071931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff028108dd0 0x7ff0281152e0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 --2- 172.21.15.143:0/700071931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff02810e4a0 0x7ff02810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 --2- 172.21.15.143:0/700071931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108410 0x7ff028108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 -- 172.21.15.143:0/700071931 >> 172.21.15.143:0/700071931 conn(0x7ff0280755b0 msgr2=0x7ff0280779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 -- 172.21.15.143:0/700071931 shutdown_connections 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.439+0000 7ff03041b700 1 -- 172.21.15.143:0/700071931 wait complete. 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 librados: starting msgr at 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 librados: starting objecter 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 Processor -- start 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- start start 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 librados: setting wanted keys 2024-03-11T21:50:08.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 librados: calling monclient init 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 0x7ff02819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108dd0 0x7ff02819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff028117bf0 con 0x7ff02810e4a0 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff028117a70 con 0x7ff028108dd0 2024-03-11T21:50:08.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff028117d70 con 0x7ff028108410 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e9b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 0x7ff02819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e9b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:37958/0 (socket says 172.21.15.143:37958) 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e9b8700 1 -- 172.21.15.143:0/2157870092 learned_addr learned my addr 172.21.15.143:0/2157870092 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02d9b6700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108dd0 0x7ff02819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 -- 172.21.15.143:0/2157870092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108dd0 msgr2=0x7ff02819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108dd0 0x7ff02819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 -- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 msgr2=0x7ff0281a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 -- 172.21.15.143:0/2157870092 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0281a3150 con 0x7ff028108410 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e9b8700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff02e1b7700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 0x7ff02819e140 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7ff02400e9f0 tx=0x7ff02400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 10 client.?.objecter ms_handle_connect 0x7ff028108410 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02400cb70 con 0x7ff028108410 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0281a3440 con 0x7ff028108410 2024-03-11T21:50:08.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff0240044a0 con 0x7ff028108410 2024-03-11T21:50:08.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024003cb0 con 0x7ff028108410 2024-03-11T21:50:08.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 10 client.42485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:08.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0281aacc0 con 0x7ff028108410 2024-03-11T21:50:08.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 librados: init done 2024-03-11T21:50:08.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff02804faa0 con 0x7ff028108410 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.443+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff024003e70 con 0x7ff028108410 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 0x7ff014071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7ff02408e970 con 0x7ff028108410 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 10 client.42485.objecter ms_dispatch 0x7ff028001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 3 client.42485.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff02d9b6700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 0x7ff014071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 3 client.42485.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:08.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 20 client.42485.objecter dump_active .. 0 homeless 2024-03-11T21:50:08.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff02d9b6700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 0x7ff014071ce0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7ff028110370 tx=0x7ff018005cf0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:08.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.447+0000 7ff01f7fe700 10 client.42485.objecter ms_handle_connect 0x7ff01406f820 2024-03-11T21:50:08.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.451+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff02405b900 con 0x7ff028108410 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[21512]: cluster 2024-03-11T21:50:06.962001+0000 osd.7 (osd.7) 692 : cluster 0 6.3 scrub starts 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[21512]: cluster 2024-03-11T21:50:06.965284+0000 osd.7 (osd.7) 693 : cluster 0 6.3 scrub ok 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.355631+0000 osd.0 (osd.0) 680 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.356069+0000 osd.0 (osd.0) 681 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[21512]: audit 2024-03-11T21:50:07.814526+0000 mon.c (mon.2) 1427 : audit 0 from='client.? 172.21.15.143:0/1991580612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:08.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[29284]: cluster 2024-03-11T21:50:06.962001+0000 osd.7 (osd.7) 692 : cluster 0 6.3 scrub starts 2024-03-11T21:50:08.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[29284]: cluster 2024-03-11T21:50:06.965284+0000 osd.7 (osd.7) 693 : cluster 0 6.3 scrub ok 2024-03-11T21:50:08.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.355631+0000 osd.0 (osd.0) 680 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:08.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.356069+0000 osd.0 (osd.0) 681 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:08.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:08 smithi143 bash[29284]: audit 2024-03-11T21:50:07.814526+0000 mon.c (mon.2) 1427 : audit 0 from='client.? 172.21.15.143:0/1991580612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:08.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:08 smithi203 bash[23654]: cluster 2024-03-11T21:50:06.962001+0000 osd.7 (osd.7) 692 : cluster 0 6.3 scrub starts 2024-03-11T21:50:08.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:08 smithi203 bash[23654]: cluster 2024-03-11T21:50:06.965284+0000 osd.7 (osd.7) 693 : cluster 0 6.3 scrub ok 2024-03-11T21:50:08.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:08 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.355631+0000 osd.0 (osd.0) 680 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:08.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:08 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.356069+0000 osd.0 (osd.0) 681 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:08.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:08 smithi203 bash[23654]: audit 2024-03-11T21:50:07.814526+0000 mon.c (mon.2) 1427 : audit 0 from='client.? 172.21.15.143:0/1991580612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:08.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.651+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff02819f4a0 con 0x7ff028108410 2024-03-11T21:50:08.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.651+0000 7ff01f7fe700 1 -- 172.21.15.143:0/2157870092 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff02405f350 con 0x7ff028108410 2024-03-11T21:50:08.644 INFO:teuthology.orchestra.run.smithi143.stdout:34359738776 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 10 librados: watch_flush enter 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 10 librados: watch_flush exit 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 20 client.42485.objecter shutdown clearing up homeless session... 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 10 client.42485.objecter successfully canceled tick 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 msgr2=0x7ff014071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 0x7ff014071ce0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7ff028110370 tx=0x7ff018005cf0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 msgr2=0x7ff02819e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 0x7ff02819e140 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7ff02400e9f0 tx=0x7ff02400ed00 comp rx=0 tx=0).stop 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 shutdown_connections 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff01406f820 0x7ff014071ce0 unknown :-1 s=CLOSED pgs=3524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff02810e4a0 0x7ff0281a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff028108410 0x7ff02819e140 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 --2- 172.21.15.143:0/2157870092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff028108dd0 0x7ff02819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 >> 172.21.15.143:0/2157870092 conn(0x7ff0280755b0 msgr2=0x7ff028075990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 shutdown_connections 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 -- 172.21.15.143:0/2157870092 wait complete. 2024-03-11T21:50:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.655+0000 7ff03041b700 1 librados: shutdown 2024-03-11T21:50:08.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- 172.21.15.143:0/1249545460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c102880 msgr2=0x7ff51c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- 172.21.15.143:0/1249545460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c102880 0x7ff51c10ec70 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7ff514009f90 tx=0x7ff51400b330 comp rx=0 tx=0).stop 2024-03-11T21:50:08.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- 172.21.15.143:0/1249545460 shutdown_connections 2024-03-11T21:50:08.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- 172.21.15.143:0/1249545460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c102880 0x7ff51c10ec70 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- 172.21.15.143:0/1249545460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- 172.21.15.143:0/1249545460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c101ec0 0x7ff51c102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- 172.21.15.143:0/1249545460 >> 172.21.15.143:0/1249545460 conn(0x7ff51c078eb0 msgr2=0x7ff51c0792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- 172.21.15.143:0/1249545460 shutdown_connections 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- 172.21.15.143:0/1249545460 wait complete. 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 librados: starting msgr at 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 librados: starting objecter 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 Processor -- start 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- start start 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 librados: setting wanted keys 2024-03-11T21:50:08.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 librados: calling monclient init 2024-03-11T21:50:08.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 0x7ff51c199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c102880 0x7ff51c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c111660 con 0x7ff51c101ec0 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1114e0 con 0x7ff51c102880 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff522d1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1117e0 con 0x7ff51c107f30 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38792/0 (socket says 172.21.15.143:38792) 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff520ab6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 0x7ff51c199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 -- 172.21.15.143:0/3412313863 learned_addr learned my addr 172.21.15.143:0/3412313863 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff51bfff700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c102880 0x7ff51c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 -- 172.21.15.143:0/3412313863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c102880 msgr2=0x7ff51c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c102880 0x7ff51c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 -- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 msgr2=0x7ff51c199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 0x7ff51c199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:08.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff5212b7700 1 -- 172.21.15.143:0/3412313863 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c19ee20 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.971+0000 7ff520ab6700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 0x7ff51c199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff5212b7700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7ff514009f60 tx=0x7ff51400cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 10 client.?.objecter ms_handle_connect 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514021070 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff514007b40 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514005040 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff522d1a700 1 -- 172.21.15.143:0/3412313863 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c19f0b0 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff522d1a700 10 client.42491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff522d1a700 1 -- 172.21.15.143:0/3412313863 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c1a6990 con 0x7ff51c107f30 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff522d1a700 1 librados: init done 2024-03-11T21:50:08.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4fc005120 con 0x7ff51c107f30 2024-03-11T21:50:08.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff514024070 con 0x7ff51c107f30 2024-03-11T21:50:08.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:08.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7ff51400f020 con 0x7ff51c107f30 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 10 client.42491.objecter ms_dispatch 0x7ff51c001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 3 client.42491.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 3 client.42491.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff520ab6700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 20 client.42491.objecter dump_active .. 0 homeless 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff520ab6700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7ff510005fd0 tx=0x7ff510009310 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:08.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.975+0000 7ff519ffb700 10 client.42491.objecter ms_handle_connect 0x7ff50c06f820 2024-03-11T21:50:08.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:08.979+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff51402b020 con 0x7ff51c107f30 2024-03-11T21:50:09.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.195+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff4fc005410 con 0x7ff51c107f30 2024-03-11T21:50:09.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.195+0000 7ff519ffb700 1 -- 172.21.15.143:0/3412313863 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff514024360 con 0x7ff51c107f30 2024-03-11T21:50:09.188 INFO:teuthology.orchestra.run.smithi143.stdout:283467841872 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 10 librados: watch_flush enter 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 10 librados: watch_flush exit 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 20 client.42491.objecter shutdown clearing up homeless session... 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 10 client.42491.objecter successfully canceled tick 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 msgr2=0x7ff50c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7ff510005fd0 tx=0x7ff510009310 comp rx=0 tx=0).stop 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 msgr2=0x7ff51c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7ff514009f60 tx=0x7ff51400cdb0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 shutdown_connections 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff50c06f820 0x7ff50c071ce0 unknown :-1 s=CLOSED pgs=3525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c101ec0 0x7ff51c199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c107f30 0x7ff51c19e6e0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 --2- 172.21.15.143:0/3412313863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff51c102880 0x7ff51c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 >> 172.21.15.143:0/3412313863 conn(0x7ff51c078eb0 msgr2=0x7ff51c10cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 shutdown_connections 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 -- 172.21.15.143:0/3412313863 wait complete. 2024-03-11T21:50:09.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.199+0000 7ff5077fe700 1 librados: shutdown 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.315+0000 7f23f8451700 1 -- 172.21.15.143:0/2949282141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 msgr2=0x7f23f0108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.315+0000 7f23f8451700 1 --2- 172.21.15.143:0/2949282141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f0108270 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009990 tx=0x7f23ec009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/2949282141 shutdown_connections 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- 172.21.15.143:0/2949282141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f0108270 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- 172.21.15.143:0/2949282141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f006a4b0 0x7f23f010ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- 172.21.15.143:0/2949282141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f0069b10 0x7f23f0069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/2949282141 >> 172.21.15.143:0/2949282141 conn(0x7f23f0076d80 msgr2=0x7f23f0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/2949282141 shutdown_connections 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/2949282141 wait complete. 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 librados: starting msgr at 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 librados: starting objecter 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 Processor -- start 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- start start 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 librados: setting wanted keys 2024-03-11T21:50:09.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 librados: calling monclient init 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f006a4b0 0x7f23f019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0111590 con 0x7f23f0107e90 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0111410 con 0x7f23f006a4b0 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0111710 con 0x7f23f0069b10 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f61ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f61ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38804/0 (socket says 172.21.15.143:38804) 2024-03-11T21:50:09.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f61ed700 1 -- 172.21.15.143:0/1896553989 learned_addr learned my addr 172.21.15.143:0/1896553989 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f59ec700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f006a4b0 0x7f23f019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 -- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 msgr2=0x7f23f0199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 -- 172.21.15.143:0/1896553989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f006a4b0 msgr2=0x7f23f019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f006a4b0 0x7f23f019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 -- 172.21.15.143:0/1896553989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f019ed90 con 0x7f23f0107e90 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f61ed700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f69ee700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f019e650 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f23e800efb0 tx=0x7f23e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 10 client.?.objecter ms_handle_connect 0x7f23f0107e90 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8003a00 con 0x7f23f0107e90 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f23e800ce20 con 0x7f23f0107e90 2024-03-11T21:50:09.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f019f080 con 0x7f23f0107e90 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8004670 con 0x7f23f0107e90 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 10 client.31293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f01a6900 con 0x7f23f0107e90 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 librados: init done 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f004faa0 con 0x7f23f0107e90 2024-03-11T21:50:09.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.319+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f23e8012050 con 0x7f23f0107e90 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 0x7f23dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f23e808e0d0 con 0x7f23f0107e90 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 10 client.31293.objecter ms_dispatch 0x7f23f0001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 3 client.31293.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 3 client.31293.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23f61ed700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 0x7f23dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:09.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 20 client.31293.objecter dump_active .. 0 homeless 2024-03-11T21:50:09.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23e77fe700 10 client.31293.objecter ms_handle_connect 0x7f23dc06f820 2024-03-11T21:50:09.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.323+0000 7f23f61ed700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 0x7f23dc071ce0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec00b630 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:09.317 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.327+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f23e8022060 con 0x7f23f0107e90 2024-03-11T21:50:09.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.572298+0000 osd.6 (osd.6) 533 : cluster 0 3.10 scrub starts 2024-03-11T21:50:09.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.575602+0000 osd.6 (osd.6) 534 : cluster 0 3.10 scrub ok 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.880612+0000 mgr.y (mgr.14550) 1034 : cluster 0 pgmap v1705: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.946962+0000 osd.7 (osd.7) 694 : cluster 0 2.8 scrub starts 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:07.949499+0000 osd.7 (osd.7) 695 : cluster 0 2.8 scrub ok 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:08.197196+0000 mon.a (mon.0) 4175 : cluster 0 osdmap e1015: 8 total, 8 up, 8 in 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: audit 2024-03-11T21:50:08.378690+0000 mon.c (mon.2) 1428 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:08.396004+0000 osd.2 (osd.2) 545 : cluster 0 5.d scrub starts 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: cluster 2024-03-11T21:50:08.398443+0000 osd.2 (osd.2) 546 : cluster 0 5.d scrub ok 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: audit 2024-03-11T21:50:08.654800+0000 mon.c (mon.2) 1429 : audit 0 from='client.? 172.21.15.143:0/2157870092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[29284]: audit 2024-03-11T21:50:09.067847+0000 mon.c (mon.2) 1430 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.572298+0000 osd.6 (osd.6) 533 : cluster 0 3.10 scrub starts 2024-03-11T21:50:09.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.575602+0000 osd.6 (osd.6) 534 : cluster 0 3.10 scrub ok 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.880612+0000 mgr.y (mgr.14550) 1034 : cluster 0 pgmap v1705: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.946962+0000 osd.7 (osd.7) 694 : cluster 0 2.8 scrub starts 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:07.949499+0000 osd.7 (osd.7) 695 : cluster 0 2.8 scrub ok 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:08.197196+0000 mon.a (mon.0) 4175 : cluster 0 osdmap e1015: 8 total, 8 up, 8 in 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: audit 2024-03-11T21:50:08.378690+0000 mon.c (mon.2) 1428 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:08.396004+0000 osd.2 (osd.2) 545 : cluster 0 5.d scrub starts 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: cluster 2024-03-11T21:50:08.398443+0000 osd.2 (osd.2) 546 : cluster 0 5.d scrub ok 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: audit 2024-03-11T21:50:08.654800+0000 mon.c (mon.2) 1429 : audit 0 from='client.? 172.21.15.143:0/2157870092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:09.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:09 smithi143 bash[21512]: audit 2024-03-11T21:50:09.067847+0000 mon.c (mon.2) 1430 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:09.458 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.572298+0000 osd.6 (osd.6) 533 : cluster 0 3.10 scrub starts 2024-03-11T21:50:09.458 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.575602+0000 osd.6 (osd.6) 534 : cluster 0 3.10 scrub ok 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.880612+0000 mgr.y (mgr.14550) 1034 : cluster 0 pgmap v1705: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.946962+0000 osd.7 (osd.7) 694 : cluster 0 2.8 scrub starts 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:07.949499+0000 osd.7 (osd.7) 695 : cluster 0 2.8 scrub ok 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:08.197196+0000 mon.a (mon.0) 4175 : cluster 0 osdmap e1015: 8 total, 8 up, 8 in 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: audit 2024-03-11T21:50:08.378690+0000 mon.c (mon.2) 1428 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:08.396004+0000 osd.2 (osd.2) 545 : cluster 0 5.d scrub starts 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: cluster 2024-03-11T21:50:08.398443+0000 osd.2 (osd.2) 546 : cluster 0 5.d scrub ok 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: audit 2024-03-11T21:50:08.654800+0000 mon.c (mon.2) 1429 : audit 0 from='client.? 172.21.15.143:0/2157870092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:09.459 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[23654]: audit 2024-03-11T21:50:09.067847+0000 mon.c (mon.2) 1430 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:09.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f23f01a6ad0 con 0x7f23f0107e90 2024-03-11T21:50:09.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23e77fe700 1 -- 172.21.15.143:0/1896553989 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+14 (secure 0 0 0) 0x7f23e8010040 con 0x7f23f0107e90 2024-03-11T21:50:09.534 INFO:teuthology.orchestra.run.smithi143.stdout:1267015352544 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 10 librados: watch_flush enter 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 10 librados: watch_flush exit 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 20 client.31293.objecter shutdown clearing up homeless session... 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 10 client.31293.objecter successfully canceled tick 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 msgr2=0x7f23dc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 0x7f23dc071ce0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec00b630 comp rx=0 tx=0).stop 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 msgr2=0x7f23f019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f019e650 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f23e800efb0 tx=0x7f23e800c3a0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 shutdown_connections 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f23dc06f820 0x7f23dc071ce0 unknown :-1 s=CLOSED pgs=3526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23f0107e90 0x7f23f019e650 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23f0069b10 0x7f23f0199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 --2- 172.21.15.143:0/1896553989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f006a4b0 0x7f23f019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 >> 172.21.15.143:0/1896553989 conn(0x7f23f0076d80 msgr2=0x7f23f010d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 shutdown_connections 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 -- 172.21.15.143:0/1896553989 wait complete. 2024-03-11T21:50:09.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:09.543+0000 7f23f8451700 1 librados: shutdown 2024-03-11T21:50:09.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:50:09 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[21512]: cluster 2024-03-11T21:50:08.538099+0000 osd.6 (osd.6) 535 : cluster 0 1.f scrub starts 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[21512]: cluster 2024-03-11T21:50:08.542752+0000 osd.6 (osd.6) 536 : cluster 0 1.f scrub ok 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[21512]: audit 2024-03-11T21:50:09.199382+0000 mon.c (mon.2) 1431 : audit 0 from='client.? 172.21.15.143:0/3412313863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[21512]: audit 2024-03-11T21:50:09.544928+0000 mon.a (mon.0) 4176 : audit 0 from='client.? 172.21.15.143:0/1896553989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[29284]: cluster 2024-03-11T21:50:08.538099+0000 osd.6 (osd.6) 535 : cluster 0 1.f scrub starts 2024-03-11T21:50:10.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[29284]: cluster 2024-03-11T21:50:08.542752+0000 osd.6 (osd.6) 536 : cluster 0 1.f scrub ok 2024-03-11T21:50:10.459 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[29284]: audit 2024-03-11T21:50:09.199382+0000 mon.c (mon.2) 1431 : audit 0 from='client.? 172.21.15.143:0/3412313863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:50:10.459 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:10 smithi143 bash[29284]: audit 2024-03-11T21:50:09.544928+0000 mon.a (mon.0) 4176 : audit 0 from='client.? 172.21.15.143:0/1896553989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:10.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:10 smithi203 bash[23654]: cluster 2024-03-11T21:50:08.538099+0000 osd.6 (osd.6) 535 : cluster 0 1.f scrub starts 2024-03-11T21:50:10.568 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:10 smithi203 bash[23654]: cluster 2024-03-11T21:50:08.542752+0000 osd.6 (osd.6) 536 : cluster 0 1.f scrub ok 2024-03-11T21:50:10.568 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:10 smithi203 bash[23654]: audit 2024-03-11T21:50:09.199382+0000 mon.c (mon.2) 1431 : audit 0 from='client.? 172.21.15.143:0/3412313863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:50:10.568 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:10 smithi203 bash[23654]: audit 2024-03-11T21:50:09.544928+0000 mon.a (mon.0) 4176 : audit 0 from='client.? 172.21.15.143:0/1896553989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:10.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168916 got 240518168921 for osd.6 2024-03-11T21:50:10.569 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: audit 2024-03-11T21:50:09.469006+0000 mgr.y (mgr.14550) 1035 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:09.493906+0000 osd.6 (osd.6) 537 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:09.496735+0000 osd.6 (osd.6) 538 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:09.882513+0000 mgr.y (mgr.14550) 1036 : cluster 0 pgmap v1707: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:09.969301+0000 osd.7 (osd.7) 696 : cluster 0 9.3 scrub starts 2024-03-11T21:50:11.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:09.973723+0000 osd.7 (osd.7) 697 : cluster 0 9.3 scrub ok 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.173034+0000 osd.5 (osd.5) 595 : cluster 0 9.6 scrub starts 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.176177+0000 osd.5 (osd.5) 596 : cluster 0 9.6 scrub ok 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.413241+0000 osd.0 (osd.0) 682 : cluster 0 3.12 scrub starts 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.417679+0000 osd.0 (osd.0) 683 : cluster 0 3.12 scrub ok 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.431938+0000 osd.2 (osd.2) 547 : cluster 0 5.8 scrub starts 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.433790+0000 osd.2 (osd.2) 548 : cluster 0 5.8 scrub ok 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: audit 2024-03-11T21:50:09.469006+0000 mgr.y (mgr.14550) 1035 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:09.493906+0000 osd.6 (osd.6) 537 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:09.496735+0000 osd.6 (osd.6) 538 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:09.882513+0000 mgr.y (mgr.14550) 1036 : cluster 0 pgmap v1707: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:11.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:09.969301+0000 osd.7 (osd.7) 696 : cluster 0 9.3 scrub starts 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:09.973723+0000 osd.7 (osd.7) 697 : cluster 0 9.3 scrub ok 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.173034+0000 osd.5 (osd.5) 595 : cluster 0 9.6 scrub starts 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.176177+0000 osd.5 (osd.5) 596 : cluster 0 9.6 scrub ok 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.413241+0000 osd.0 (osd.0) 682 : cluster 0 3.12 scrub starts 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.417679+0000 osd.0 (osd.0) 683 : cluster 0 3.12 scrub ok 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.431938+0000 osd.2 (osd.2) 547 : cluster 0 5.8 scrub starts 2024-03-11T21:50:11.454 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:11 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.433790+0000 osd.2 (osd.2) 548 : cluster 0 5.8 scrub ok 2024-03-11T21:50:11.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: audit 2024-03-11T21:50:09.469006+0000 mgr.y (mgr.14550) 1035 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:09.493906+0000 osd.6 (osd.6) 537 : cluster 0 5.1f deep-scrub starts 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:09.496735+0000 osd.6 (osd.6) 538 : cluster 0 5.1f deep-scrub ok 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:09.882513+0000 mgr.y (mgr.14550) 1036 : cluster 0 pgmap v1707: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:09.969301+0000 osd.7 (osd.7) 696 : cluster 0 9.3 scrub starts 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:09.973723+0000 osd.7 (osd.7) 697 : cluster 0 9.3 scrub ok 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.173034+0000 osd.5 (osd.5) 595 : cluster 0 9.6 scrub starts 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.176177+0000 osd.5 (osd.5) 596 : cluster 0 9.6 scrub ok 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.413241+0000 osd.0 (osd.0) 682 : cluster 0 3.12 scrub starts 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.417679+0000 osd.0 (osd.0) 683 : cluster 0 3.12 scrub ok 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.431938+0000 osd.2 (osd.2) 547 : cluster 0 5.8 scrub starts 2024-03-11T21:50:11.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:11 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.433790+0000 osd.2 (osd.2) 548 : cluster 0 5.8 scrub ok 2024-03-11T21:50:11.898 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738771 got 34359738776 for osd.0 2024-03-11T21:50:11.898 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:12.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.919907+0000 osd.1 (osd.1) 481 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[21512]: cluster 2024-03-11T21:50:10.922464+0000 osd.1 (osd.1) 482 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[21512]: cluster 2024-03-11T21:50:11.464892+0000 osd.2 (osd.2) 549 : cluster 0 2.1 scrub starts 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[21512]: cluster 2024-03-11T21:50:11.467416+0000 osd.2 (osd.2) 550 : cluster 0 2.1 scrub ok 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.919907+0000 osd.1 (osd.1) 481 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[29284]: cluster 2024-03-11T21:50:10.922464+0000 osd.1 (osd.1) 482 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[29284]: cluster 2024-03-11T21:50:11.464892+0000 osd.2 (osd.2) 549 : cluster 0 2.1 scrub starts 2024-03-11T21:50:12.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:12 smithi143 bash[29284]: cluster 2024-03-11T21:50:11.467416+0000 osd.2 (osd.2) 550 : cluster 0 2.1 scrub ok 2024-03-11T21:50:12.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:12 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.919907+0000 osd.1 (osd.1) 481 : cluster 0 6.1d deep-scrub starts 2024-03-11T21:50:12.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:12 smithi203 bash[23654]: cluster 2024-03-11T21:50:10.922464+0000 osd.1 (osd.1) 482 : cluster 0 6.1d deep-scrub ok 2024-03-11T21:50:12.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:12 smithi203 bash[23654]: cluster 2024-03-11T21:50:11.464892+0000 osd.2 (osd.2) 549 : cluster 0 2.1 scrub starts 2024-03-11T21:50:12.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:12 smithi203 bash[23654]: cluster 2024-03-11T21:50:11.467416+0000 osd.2 (osd.2) 550 : cluster 0 2.1 scrub ok 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 -- 172.21.15.143:0/4284854987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc106550 msgr2=0x7fc8dc106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 --2- 172.21.15.143:0/4284854987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc106550 0x7fc8dc106930 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009990 tx=0x7fc8cc009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 -- 172.21.15.143:0/4284854987 shutdown_connections 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 --2- 172.21.15.143:0/4284854987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc100e60 0x7fc8dc10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 --2- 172.21.15.143:0/4284854987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc106550 0x7fc8dc106930 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 --2- 172.21.15.143:0/4284854987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc1004a0 0x7fc8dc100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 -- 172.21.15.143:0/4284854987 >> 172.21.15.143:0/4284854987 conn(0x7fc8dc0772e0 msgr2=0x7fc8dc0fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 -- 172.21.15.143:0/4284854987 shutdown_connections 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.863+0000 7fc8e2659700 1 -- 172.21.15.143:0/4284854987 wait complete. 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 librados: starting msgr at 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 librados: starting objecter 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 Processor -- start 2024-03-11T21:50:12.857 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- start start 2024-03-11T21:50:12.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 librados: setting wanted keys 2024-03-11T21:50:12.858 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 librados: calling monclient init 2024-03-11T21:50:12.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:12.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc100e60 0x7fc8dc0fe010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:12.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 0x7fc8dc0fe550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:12.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10fc60 con 0x7fc8dc106550 2024-03-11T21:50:12.859 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10fae0 con 0x7fc8dc100e60 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10fde0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:38822/0 (socket says 172.21.15.143:38822) 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e0bf6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 0x7fc8dc0fe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 -- 172.21.15.143:0/1453953979 learned_addr learned my addr 172.21.15.143:0/1453953979 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8db7fe700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc100e60 0x7fc8dc0fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 -- 172.21.15.143:0/1453953979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc100e60 msgr2=0x7fc8dc0fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc100e60 0x7fc8dc0fe010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 -- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 msgr2=0x7fc8dc0fe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 0x7fc8dc0fe550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 -- 172.21.15.143:0/1453953979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc1a60e0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e0bf6700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 0x7fc8dc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:12.860 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8dbfff700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0057c0 tx=0x7fc8cc00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 10 client.?.objecter ms_handle_connect 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc026070 con 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fc8cc00fbf0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0173f0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc1a6370 con 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 10 client.42503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1a67d0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 librados: init done 2024-03-11T21:50:12.861 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc04faa0 con 0x7fc8dc1004a0 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.867+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc8cc009d10 con 0x7fc8dc1004a0 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 0x7fc8c4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fc8cc08f070 con 0x7fc8dc1004a0 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8db7fe700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 0x7fc8c4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 10 client.42503.objecter ms_dispatch 0x7fc8dc001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 3 client.42503.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 3 client.42503.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:12.862 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 20 client.42503.objecter dump_active .. 0 homeless 2024-03-11T21:50:12.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8d97fa700 10 client.42503.objecter ms_handle_connect 0x7fc8c406f820 2024-03-11T21:50:12.863 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.871+0000 7fc8db7fe700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 0x7fc8c4071ce0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc106dc0 tx=0x7fc8d0005cf0 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:12.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:12.875+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc8cc013070 con 0x7fc8dc1004a0 2024-03-11T21:50:13.081 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc8dc02d830 con 0x7fc8dc1004a0 2024-03-11T21:50:13.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8d97fa700 1 -- 172.21.15.143:0/1453953979 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc8cc0095a0 con 0x7fc8dc1004a0 2024-03-11T21:50:13.082 INFO:teuthology.orchestra.run.smithi143.stdout:150323855726 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 10 librados: watch_flush enter 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 10 librados: watch_flush exit 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 20 client.42503.objecter shutdown clearing up homeless session... 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 10 client.42503.objecter successfully canceled tick 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 msgr2=0x7fc8c4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 0x7fc8c4071ce0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc106dc0 tx=0x7fc8d0005cf0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 msgr2=0x7fc8dc0fdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0057c0 tx=0x7fc8cc00bfd0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 shutdown_connections 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fc8c406f820 0x7fc8c4071ce0 unknown :-1 s=CLOSED pgs=3527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc106550 0x7fc8dc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.091+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8dc1004a0 0x7fc8dc0fdad0 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.095+0000 7fc8e2659700 1 --2- 172.21.15.143:0/1453953979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc100e60 0x7fc8dc0fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:13.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.095+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 >> 172.21.15.143:0/1453953979 conn(0x7fc8dc0772e0 msgr2=0x7fc8dc1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:13.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.095+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 shutdown_connections 2024-03-11T21:50:13.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.095+0000 7fc8e2659700 1 -- 172.21.15.143:0/1453953979 wait complete. 2024-03-11T21:50:13.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:13.095+0000 7fc8e2659700 1 librados: shutdown 2024-03-11T21:50:13.428 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:11.489521+0000 osd.6 (osd.6) 539 : cluster 0 2.1a scrub starts 2024-03-11T21:50:13.428 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:11.491831+0000 osd.6 (osd.6) 540 : cluster 0 2.1a scrub ok 2024-03-11T21:50:13.428 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:11.883797+0000 mgr.y (mgr.14550) 1037 : cluster 0 pgmap v1708: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:13.428 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.196755+0000 osd.5 (osd.5) 597 : cluster 0 5.f deep-scrub starts 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.199124+0000 osd.5 (osd.5) 598 : cluster 0 5.f deep-scrub ok 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.413024+0000 osd.0 (osd.0) 684 : cluster 0 6.10 scrub starts 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.415397+0000 osd.0 (osd.0) 685 : cluster 0 6.10 scrub ok 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.497438+0000 osd.2 (osd.2) 551 : cluster 0 2.10 scrub starts 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.499598+0000 osd.2 (osd.2) 552 : cluster 0 2.10 scrub ok 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.933155+0000 osd.3 (osd.3) 681 : cluster 0 2.19 scrub starts 2024-03-11T21:50:13.429 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: cluster 2024-03-11T21:50:12.934794+0000 osd.3 (osd.3) 682 : cluster 0 2.19 scrub ok 2024-03-11T21:50:13.430 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[29284]: audit 2024-03-11T21:50:13.092893+0000 mon.c (mon.2) 1432 : audit 0 from='client.? 172.21.15.143:0/1453953979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:50:13.430 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:11.489521+0000 osd.6 (osd.6) 539 : cluster 0 2.1a scrub starts 2024-03-11T21:50:13.430 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:11.491831+0000 osd.6 (osd.6) 540 : cluster 0 2.1a scrub ok 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:11.883797+0000 mgr.y (mgr.14550) 1037 : cluster 0 pgmap v1708: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.196755+0000 osd.5 (osd.5) 597 : cluster 0 5.f deep-scrub starts 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.199124+0000 osd.5 (osd.5) 598 : cluster 0 5.f deep-scrub ok 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.413024+0000 osd.0 (osd.0) 684 : cluster 0 6.10 scrub starts 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.415397+0000 osd.0 (osd.0) 685 : cluster 0 6.10 scrub ok 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.497438+0000 osd.2 (osd.2) 551 : cluster 0 2.10 scrub starts 2024-03-11T21:50:13.431 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.499598+0000 osd.2 (osd.2) 552 : cluster 0 2.10 scrub ok 2024-03-11T21:50:13.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.933155+0000 osd.3 (osd.3) 681 : cluster 0 2.19 scrub starts 2024-03-11T21:50:13.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: cluster 2024-03-11T21:50:12.934794+0000 osd.3 (osd.3) 682 : cluster 0 2.19 scrub ok 2024-03-11T21:50:13.432 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:13 smithi143 bash[21512]: audit 2024-03-11T21:50:13.092893+0000 mon.c (mon.2) 1432 : audit 0 from='client.? 172.21.15.143:0/1453953979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:50:13.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:11.489521+0000 osd.6 (osd.6) 539 : cluster 0 2.1a scrub starts 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:11.491831+0000 osd.6 (osd.6) 540 : cluster 0 2.1a scrub ok 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:11.883797+0000 mgr.y (mgr.14550) 1037 : cluster 0 pgmap v1708: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.196755+0000 osd.5 (osd.5) 597 : cluster 0 5.f deep-scrub starts 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.199124+0000 osd.5 (osd.5) 598 : cluster 0 5.f deep-scrub ok 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.413024+0000 osd.0 (osd.0) 684 : cluster 0 6.10 scrub starts 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.415397+0000 osd.0 (osd.0) 685 : cluster 0 6.10 scrub ok 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.497438+0000 osd.2 (osd.2) 551 : cluster 0 2.10 scrub starts 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.499598+0000 osd.2 (osd.2) 552 : cluster 0 2.10 scrub ok 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.933155+0000 osd.3 (osd.3) 681 : cluster 0 2.19 scrub starts 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: cluster 2024-03-11T21:50:12.934794+0000 osd.3 (osd.3) 682 : cluster 0 2.19 scrub ok 2024-03-11T21:50:13.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:13 smithi203 bash[23654]: audit 2024-03-11T21:50:13.092893+0000 mon.c (mon.2) 1432 : audit 0 from='client.? 172.21.15.143:0/1453953979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:50:13.848 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841868 got 283467841872 for osd.7 2024-03-11T21:50:13.848 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:14.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1267015352539 got 1267015352544 for osd.1 2024-03-11T21:50:14.082 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:14.269 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.275+0000 7f3fcf177700 1 -- 172.21.15.143:0/2994451294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 msgr2=0x7f3fc81011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.275+0000 7f3fcf177700 1 --2- 172.21.15.143:0/2994451294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc81011c0 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009990 tx=0x7f3fb0009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.275+0000 7f3fcf177700 1 -- 172.21.15.143:0/2994451294 shutdown_connections 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.275+0000 7f3fcf177700 1 --2- 172.21.15.143:0/2994451294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 --2- 172.21.15.143:0/2994451294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 --2- 172.21.15.143:0/2994451294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc81011c0 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/2994451294 >> 172.21.15.143:0/2994451294 conn(0x7f3fc8076d80 msgr2=0x7f3fc8077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/2994451294 shutdown_connections 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/2994451294 wait complete. 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 librados: starting msgr at 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 librados: starting objecter 2024-03-11T21:50:14.270 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 Processor -- start 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- start start 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 librados: setting wanted keys 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 librados: calling monclient init 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc819e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8117bd0 con 0x7f3fc8106dc0 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8117a50 con 0x7f3fc8101700 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8117d50 con 0x7f3fc8100d40 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fccf13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc819e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39306/0 (socket says 172.21.15.143:39306) 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 -- 172.21.15.143:0/476160821 learned_addr learned my addr 172.21.15.143:0/476160821 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:14.271 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc7fff700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 -- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 msgr2=0x7f3fc819e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc819e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 -- 172.21.15.143:0/476160821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 msgr2=0x7f3fc819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.273 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 -- 172.21.15.143:0/476160821 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc81a31c0 con 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fccf13700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc819e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc7fff700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcd714700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c970 tx=0x7f3fb800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 10 client.?.objecter ms_handle_connect 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800ce60 con 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc81a34b0 con 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f3fb8004830 con 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 10 client.31305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc81aad30 con 0x7f3fc8106dc0 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 librados: init done 2024-03-11T21:50:14.274 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80053c0 con 0x7f3fc8106dc0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa8005120 con 0x7f3fc8106dc0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.279+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3fb800f450 con 0x7f3fc8106dc0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 0x7f3fb4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f3fb808e180 con 0x7f3fc8106dc0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 10 client.31305.objecter ms_dispatch 0x7f3fc8001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 3 client.31305.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 3 client.31305.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fccf13700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 0x7f3fb4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 20 client.31305.objecter dump_active .. 0 homeless 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fc5ffb700 10 client.31305.objecter ms_handle_connect 0x7f3fb406f820 2024-03-11T21:50:14.275 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.283+0000 7f3fccf13700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 0x7f3fb4071ce0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009480 tx=0x7f3fc000b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.287+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3fb8014030 con 0x7f3fc8106dc0 2024-03-11T21:50:14.327 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- 172.21.15.143:0/2746923345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac101700 msgr2=0x7fbdac10db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- 172.21.15.143:0/2746923345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac101700 0x7fbdac10db70 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0098f0 tx=0x7fbd9c009c00 comp rx=0 tx=0).stop 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- 172.21.15.143:0/2746923345 shutdown_connections 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- 172.21.15.143:0/2746923345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac101700 0x7fbdac10db70 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- 172.21.15.143:0/2746923345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac1071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- 172.21.15.143:0/2746923345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac100d40 0x7fbdac1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- 172.21.15.143:0/2746923345 >> 172.21.15.143:0/2746923345 conn(0x7fbdac076d80 msgr2=0x7fbdac077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- 172.21.15.143:0/2746923345 shutdown_connections 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- 172.21.15.143:0/2746923345 wait complete. 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 librados: starting msgr at 2024-03-11T21:50:14.328 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 librados: starting objecter 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 Processor -- start 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- start start 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 librados: setting wanted keys 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 librados: calling monclient init 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac101700 0x7fbdac102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.335+0000 7fbdb3076700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac110570 con 0x7fbdac100d40 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1103f0 con 0x7fbdac101700 2024-03-11T21:50:14.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1106f0 con 0x7fbdac106dc0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39322/0 (socket says 172.21.15.143:39322) 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 -- 172.21.15.143:0/1738825212 learned_addr learned my addr 172.21.15.143:0/1738825212 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 -- 172.21.15.143:0/1738825212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac101700 msgr2=0x7fbdac102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac101700 0x7fbdac102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 -- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 msgr2=0x7fbdac1029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 -- 172.21.15.143:0/1738825212 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac1a6110 con 0x7fbdac106dc0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb1613700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac103440 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009230 tx=0x7fbd9c00fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 10 client.?.objecter ms_handle_connect 0x7fbdac106dc0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c00b790 con 0x7fbdac106dc0 2024-03-11T21:50:14.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac1a63a0 con 0x7fbdac106dc0 2024-03-11T21:50:14.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fbd9c0054c0 con 0x7fbdac106dc0 2024-03-11T21:50:14.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0039d0 con 0x7fbdac106dc0 2024-03-11T21:50:14.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 10 client.42515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:14.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac1a6850 con 0x7fbdac106dc0 2024-03-11T21:50:14.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 librados: init done 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdac04faa0 con 0x7fbdac106dc0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbd9c005040 con 0x7fbdac106dc0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 0x7fbd94071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 0x7fbd94071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fbd9c08f6e0 con 0x7fbdac106dc0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 10 client.42515.objecter ms_dispatch 0x7fbdac001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 3 client.42515.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 3 client.42515.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 20 client.42515.objecter dump_active .. 0 homeless 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbdb0e12700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 0x7fbd94071ce0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009e60 tx=0x7fbda400b040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.339+0000 7fbda9ffb700 10 client.42515.objecter ms_handle_connect 0x7fbd9406f820 2024-03-11T21:50:14.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.347+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fbd9c013030 con 0x7fbdac106dc0 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.090385+0000 osd.4 (osd.4) 643 : cluster 0 5.e scrub starts 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.093253+0000 osd.4 (osd.4) 644 : cluster 0 5.e scrub ok 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.955717+0000 osd.3 (osd.3) 683 : cluster 0 9.1 scrub starts 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.959153+0000 osd.3 (osd.3) 684 : cluster 0 9.1 scrub ok 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.090385+0000 osd.4 (osd.4) 643 : cluster 0 5.e scrub starts 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.093253+0000 osd.4 (osd.4) 644 : cluster 0 5.e scrub ok 2024-03-11T21:50:14.361 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.955717+0000 osd.3 (osd.3) 683 : cluster 0 9.1 scrub starts 2024-03-11T21:50:14.362 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:14 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.959153+0000 osd.3 (osd.3) 684 : cluster 0 9.1 scrub ok 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/3109849861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 msgr2=0x7f875c06aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- 172.21.15.143:0/3109849861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c06aa80 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f874c009990 tx=0x7f874c009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/3109849861 shutdown_connections 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- 172.21.15.143:0/3109849861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c06aa80 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- 172.21.15.143:0/3109849861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 0x7f875c10edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- 172.21.15.143:0/3109849861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 0x7f875c106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/3109849861 >> 172.21.15.143:0/3109849861 conn(0x7f875c077de0 msgr2=0x7f875c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.482 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/3109849861 shutdown_connections 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/3109849861 wait complete. 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 librados: starting msgr at 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 librados: starting objecter 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 Processor -- start 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- start start 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 librados: setting wanted keys 2024-03-11T21:50:14.483 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 librados: calling monclient init 2024-03-11T21:50:14.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 0x7f875c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 0x7f875c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.485 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c1116a0 con 0x7f875c06a6a0 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c111520 con 0x7f875c06b050 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875c111820 con 0x7f875c107220 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39336/0 (socket says 172.21.15.143:39336) 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 -- 172.21.15.143:0/1380898719 learned_addr learned my addr 172.21.15.143:0/1380898719 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875bfff700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 0x7f875c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875affd700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 0x7f875c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 -- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 msgr2=0x7f875c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 0x7f875c101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 -- 172.21.15.143:0/1380898719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 msgr2=0x7f875c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 0x7f875c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.486 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 -- 172.21.15.143:0/1380898719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c1a6180 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f875b7fe700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f874c000c00 tx=0x7f874c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8758ff9700 10 client.?.objecter ms_handle_connect 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8758ff9700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c027070 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f874c022470 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c1a6410 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 10 client.31317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c1a6860 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8761f2b700 1 librados: init done 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.491+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c017650 con 0x7f875c06a6a0 2024-03-11T21:50:14.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c04faa0 con 0x7f875c06a6a0 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f874c02a070 con 0x7f875c06a6a0 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 0x7f8744071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1015..1015 src has 964..1015) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f874c08f710 con 0x7f875c06a6a0 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 10 client.31317.objecter ms_dispatch 0x7f875c001550 osd_map(1015..1015 src has 964..1015) v4 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 3 client.31317.objecter handle_osd_map got epochs [1015,1015] > 0 2024-03-11T21:50:14.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 3 client.31317.objecter handle_osd_map decoding full epoch 1015 2024-03-11T21:50:14.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f875affd700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 0x7f8744071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:14.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 20 client.31317.objecter dump_active .. 0 homeless 2024-03-11T21:50:14.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f8758ff9700 10 client.31317.objecter ms_handle_connect 0x7f874406f820 2024-03-11T21:50:14.489 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.495+0000 7f875affd700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 0x7f8744071ce0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750008040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:14.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.503+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f874c013070 con 0x7f875c06a6a0 2024-03-11T21:50:14.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.515+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3fa8005410 con 0x7f3fc8106dc0 2024-03-11T21:50:14.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.515+0000 7f3fc5ffb700 1 -- 172.21.15.143:0/476160821 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3fb805b190 con 0x7f3fc8106dc0 2024-03-11T21:50:14.508 INFO:teuthology.orchestra.run.smithi143.stdout:115964117369 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 10 librados: watch_flush enter 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 10 librados: watch_flush exit 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 20 client.31305.objecter shutdown clearing up homeless session... 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 10 client.31305.objecter successfully canceled tick 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 msgr2=0x7f3fb4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 0x7f3fb4071ce0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009480 tx=0x7f3fc000b040 comp rx=0 tx=0).stop 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 msgr2=0x7f3fc81a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.510 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c970 tx=0x7f3fb800cc80 comp rx=0 tx=0).stop 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 shutdown_connections 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f3fb406f820 0x7f3fb4071ce0 unknown :-1 s=CLOSED pgs=3528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fc8106dc0 0x7f3fc81a2a80 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fc8100d40 0x7f3fc819e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 --2- 172.21.15.143:0/476160821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fc8101700 0x7f3fc819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 >> 172.21.15.143:0/476160821 conn(0x7f3fc8076d80 msgr2=0x7f3fc8107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 shutdown_connections 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 -- 172.21.15.143:0/476160821 wait complete. 2024-03-11T21:50:14.511 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.519+0000 7f3fcf177700 1 librados: shutdown 2024-03-11T21:50:14.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:14 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.090385+0000 osd.4 (osd.4) 643 : cluster 0 5.e scrub starts 2024-03-11T21:50:14.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:14 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.093253+0000 osd.4 (osd.4) 644 : cluster 0 5.e scrub ok 2024-03-11T21:50:14.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:14 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.955717+0000 osd.3 (osd.3) 683 : cluster 0 9.1 scrub starts 2024-03-11T21:50:14.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:14 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.959153+0000 osd.3 (osd.3) 684 : cluster 0 9.1 scrub ok 2024-03-11T21:50:14.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.563+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbdac1a6ab0 con 0x7fbdac106dc0 2024-03-11T21:50:14.554 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.563+0000 7fbda9ffb700 1 -- 172.21.15.143:0/1738825212 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd9c05c6f0 con 0x7fbdac106dc0 2024-03-11T21:50:14.554 INFO:teuthology.orchestra.run.smithi143.stdout:81604379013 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 10 librados: watch_flush enter 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 10 librados: watch_flush exit 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 20 client.42515.objecter shutdown clearing up homeless session... 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 10 client.42515.objecter successfully canceled tick 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 msgr2=0x7fbd94071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 0x7fbd94071ce0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009e60 tx=0x7fbda400b040 comp rx=0 tx=0).stop 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 msgr2=0x7fbdac103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.557 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac103440 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009230 tx=0x7fbd9c00fb40 comp rx=0 tx=0).stop 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 shutdown_connections 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fbd9406f820 0x7fbd94071ce0 unknown :-1 s=CLOSED pgs=3529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdac100d40 0x7fbdac1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdac106dc0 0x7fbdac103440 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 --2- 172.21.15.143:0/1738825212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac101700 0x7fbdac102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 >> 172.21.15.143:0/1738825212 conn(0x7fbdac076d80 msgr2=0x7fbdac069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 shutdown_connections 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 -- 172.21.15.143:0/1738825212 wait complete. 2024-03-11T21:50:14.558 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.567+0000 7fbdb3076700 1 librados: shutdown 2024-03-11T21:50:14.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.715+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f875c1a6a30 con 0x7f875c06a6a0 2024-03-11T21:50:14.707 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.715+0000 7f8758ff9700 1 -- 172.21.15.143:0/1380898719 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f874c05c6a0 con 0x7f875c06a6a0 2024-03-11T21:50:14.708 INFO:teuthology.orchestra.run.smithi143.stdout:197568495973 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 10 librados: watch_flush enter 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 10 librados: watch_flush exit 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 20 client.31317.objecter shutdown clearing up homeless session... 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 10 client.31317.objecter successfully canceled tick 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 msgr2=0x7f8744071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 0x7f8744071ce0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750008040 comp rx=0 tx=0).stop 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 msgr2=0x7f875c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:14.710 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f874c000c00 tx=0x7f874c0047b0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 shutdown_connections 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f874406f820 0x7f8744071ce0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875c06a6a0 0x7f875c100d90 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f875c107220 0x7f875c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 --2- 172.21.15.143:0/1380898719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875c06b050 0x7f875c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 >> 172.21.15.143:0/1380898719 conn(0x7f875c077de0 msgr2=0x7f875c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 shutdown_connections 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 -- 172.21.15.143:0/1380898719 wait complete. 2024-03-11T21:50:14.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:14.719+0000 7f8761f2b700 1 librados: shutdown 2024-03-11T21:50:15.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.468073+0000 osd.6 (osd.6) 541 : cluster 0 9.18 scrub starts 2024-03-11T21:50:15.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.470688+0000 osd.6 (osd.6) 542 : cluster 0 9.18 scrub ok 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.885077+0000 mgr.y (mgr.14550) 1038 : cluster 0 pgmap v1709: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.892692+0000 osd.1 (osd.1) 483 : cluster 0 1.b scrub starts 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:13.895425+0000 osd.1 (osd.1) 484 : cluster 0 1.b scrub ok 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:14.211981+0000 osd.5 (osd.5) 599 : cluster 0 1.8 scrub starts 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: cluster 2024-03-11T21:50:14.215391+0000 osd.5 (osd.5) 600 : cluster 0 1.8 scrub ok 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: audit 2024-03-11T21:50:14.518877+0000 mon.a (mon.0) 4177 : audit 0 from='client.? 172.21.15.143:0/476160821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: audit 2024-03-11T21:50:14.565777+0000 mon.c (mon.2) 1433 : audit 0 from='client.? 172.21.15.143:0/1738825212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[29284]: audit 2024-03-11T21:50:14.718893+0000 mon.a (mon.0) 4178 : audit 0 from='client.? 172.21.15.143:0/1380898719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.468073+0000 osd.6 (osd.6) 541 : cluster 0 9.18 scrub starts 2024-03-11T21:50:15.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.470688+0000 osd.6 (osd.6) 542 : cluster 0 9.18 scrub ok 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.885077+0000 mgr.y (mgr.14550) 1038 : cluster 0 pgmap v1709: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.892692+0000 osd.1 (osd.1) 483 : cluster 0 1.b scrub starts 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:13.895425+0000 osd.1 (osd.1) 484 : cluster 0 1.b scrub ok 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:14.211981+0000 osd.5 (osd.5) 599 : cluster 0 1.8 scrub starts 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: cluster 2024-03-11T21:50:14.215391+0000 osd.5 (osd.5) 600 : cluster 0 1.8 scrub ok 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: audit 2024-03-11T21:50:14.518877+0000 mon.a (mon.0) 4177 : audit 0 from='client.? 172.21.15.143:0/476160821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: audit 2024-03-11T21:50:14.565777+0000 mon.c (mon.2) 1433 : audit 0 from='client.? 172.21.15.143:0/1738825212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:15.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:15 smithi143 bash[21512]: audit 2024-03-11T21:50:14.718893+0000 mon.a (mon.0) 4178 : audit 0 from='client.? 172.21.15.143:0/1380898719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:50:15.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.468073+0000 osd.6 (osd.6) 541 : cluster 0 9.18 scrub starts 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.470688+0000 osd.6 (osd.6) 542 : cluster 0 9.18 scrub ok 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.885077+0000 mgr.y (mgr.14550) 1038 : cluster 0 pgmap v1709: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.892692+0000 osd.1 (osd.1) 483 : cluster 0 1.b scrub starts 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:13.895425+0000 osd.1 (osd.1) 484 : cluster 0 1.b scrub ok 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:14.211981+0000 osd.5 (osd.5) 599 : cluster 0 1.8 scrub starts 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: cluster 2024-03-11T21:50:14.215391+0000 osd.5 (osd.5) 600 : cluster 0 1.8 scrub ok 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: audit 2024-03-11T21:50:14.518877+0000 mon.a (mon.0) 4177 : audit 0 from='client.? 172.21.15.143:0/476160821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: audit 2024-03-11T21:50:14.565777+0000 mon.c (mon.2) 1433 : audit 0 from='client.? 172.21.15.143:0/1738825212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:15.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:15 smithi203 bash[23654]: audit 2024-03-11T21:50:14.718893+0000 mon.a (mon.0) 4178 : audit 0 from='client.? 172.21.15.143:0/1380898719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:50:16.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855721 got 150323855726 for osd.4 2024-03-11T21:50:16.199 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:16.451 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.238214+0000 osd.5 (osd.5) 601 : cluster 0 5.1b scrub starts 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.240511+0000 osd.5 (osd.5) 602 : cluster 0 5.1b scrub ok 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.481546+0000 osd.2 (osd.2) 553 : cluster 0 6.f scrub starts 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.483986+0000 osd.2 (osd.2) 554 : cluster 0 6.f scrub ok 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.942339+0000 osd.3 (osd.3) 685 : cluster 0 9.c scrub starts 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.945712+0000 osd.3 (osd.3) 686 : cluster 0 9.c scrub ok 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.238214+0000 osd.5 (osd.5) 601 : cluster 0 5.1b scrub starts 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.240511+0000 osd.5 (osd.5) 602 : cluster 0 5.1b scrub ok 2024-03-11T21:50:16.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.481546+0000 osd.2 (osd.2) 553 : cluster 0 6.f scrub starts 2024-03-11T21:50:16.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.483986+0000 osd.2 (osd.2) 554 : cluster 0 6.f scrub ok 2024-03-11T21:50:16.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.942339+0000 osd.3 (osd.3) 685 : cluster 0 9.c scrub starts 2024-03-11T21:50:16.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:16 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.945712+0000 osd.3 (osd.3) 686 : cluster 0 9.c scrub ok 2024-03-11T21:50:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.238214+0000 osd.5 (osd.5) 601 : cluster 0 5.1b scrub starts 2024-03-11T21:50:16.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.240511+0000 osd.5 (osd.5) 602 : cluster 0 5.1b scrub ok 2024-03-11T21:50:16.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.481546+0000 osd.2 (osd.2) 553 : cluster 0 6.f scrub starts 2024-03-11T21:50:16.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.483986+0000 osd.2 (osd.2) 554 : cluster 0 6.f scrub ok 2024-03-11T21:50:16.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.942339+0000 osd.3 (osd.3) 685 : cluster 0 9.c scrub starts 2024-03-11T21:50:16.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:16 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.945712+0000 osd.3 (osd.3) 686 : cluster 0 9.c scrub ok 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[21512]: cluster 2024-03-11T21:50:15.886041+0000 mgr.y (mgr.14550) 1039 : cluster 0 pgmap v1710: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[21512]: cluster 2024-03-11T21:50:16.283944+0000 osd.5 (osd.5) 603 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[21512]: cluster 2024-03-11T21:50:16.286304+0000 osd.5 (osd.5) 604 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[21512]: cluster 2024-03-11T21:50:16.494456+0000 osd.2 (osd.2) 555 : cluster 0 2.e deep-scrub starts 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[21512]: cluster 2024-03-11T21:50:16.497218+0000 osd.2 (osd.2) 556 : cluster 0 2.e deep-scrub ok 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[29284]: cluster 2024-03-11T21:50:15.886041+0000 mgr.y (mgr.14550) 1039 : cluster 0 pgmap v1710: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:17.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[29284]: cluster 2024-03-11T21:50:16.283944+0000 osd.5 (osd.5) 603 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:50:17.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[29284]: cluster 2024-03-11T21:50:16.286304+0000 osd.5 (osd.5) 604 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:50:17.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[29284]: cluster 2024-03-11T21:50:16.494456+0000 osd.2 (osd.2) 555 : cluster 0 2.e deep-scrub starts 2024-03-11T21:50:17.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:17 smithi143 bash[29284]: cluster 2024-03-11T21:50:16.497218+0000 osd.2 (osd.2) 556 : cluster 0 2.e deep-scrub ok 2024-03-11T21:50:17.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:17 smithi203 bash[23654]: cluster 2024-03-11T21:50:15.886041+0000 mgr.y (mgr.14550) 1039 : cluster 0 pgmap v1710: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:17.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:17 smithi203 bash[23654]: cluster 2024-03-11T21:50:16.283944+0000 osd.5 (osd.5) 603 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:50:17.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:17 smithi203 bash[23654]: cluster 2024-03-11T21:50:16.286304+0000 osd.5 (osd.5) 604 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:50:17.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:17 smithi203 bash[23654]: cluster 2024-03-11T21:50:16.494456+0000 osd.2 (osd.2) 555 : cluster 0 2.e deep-scrub starts 2024-03-11T21:50:17.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:17 smithi203 bash[23654]: cluster 2024-03-11T21:50:16.497218+0000 osd.2 (osd.2) 556 : cluster 0 2.e deep-scrub ok 2024-03-11T21:50:17.689 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117363 got 115964117369 for osd.3 2024-03-11T21:50:17.689 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:18.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495967 got 197568495973 for osd.5 2024-03-11T21:50:18.000 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:18.039 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604379006 got 81604379013 for osd.2 2024-03-11T21:50:18.040 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:18.040 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:50:18.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:50:19.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:17.886959+0000 mgr.y (mgr.14550) 1040 : cluster 0 pgmap v1711: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:17.939962+0000 osd.7 (osd.7) 698 : cluster 0 6.1c scrub starts 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:17.942897+0000 osd.7 (osd.7) 699 : cluster 0 6.1c scrub ok 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.362760+0000 osd.5 (osd.5) 605 : cluster 0 2.18 scrub starts 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.365101+0000 osd.5 (osd.5) 606 : cluster 0 2.18 scrub ok 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.960429+0000 osd.3 (osd.3) 687 : cluster 0 3.8 scrub starts 2024-03-11T21:50:19.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:19 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.965025+0000 osd.3 (osd.3) 688 : cluster 0 3.8 scrub ok 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:17.886959+0000 mgr.y (mgr.14550) 1040 : cluster 0 pgmap v1711: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:17.939962+0000 osd.7 (osd.7) 698 : cluster 0 6.1c scrub starts 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:17.942897+0000 osd.7 (osd.7) 699 : cluster 0 6.1c scrub ok 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.362760+0000 osd.5 (osd.5) 605 : cluster 0 2.18 scrub starts 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.365101+0000 osd.5 (osd.5) 606 : cluster 0 2.18 scrub ok 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.960429+0000 osd.3 (osd.3) 687 : cluster 0 3.8 scrub starts 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.965025+0000 osd.3 (osd.3) 688 : cluster 0 3.8 scrub ok 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:17.886959+0000 mgr.y (mgr.14550) 1040 : cluster 0 pgmap v1711: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:19.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:17.939962+0000 osd.7 (osd.7) 698 : cluster 0 6.1c scrub starts 2024-03-11T21:50:19.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:17.942897+0000 osd.7 (osd.7) 699 : cluster 0 6.1c scrub ok 2024-03-11T21:50:19.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.362760+0000 osd.5 (osd.5) 605 : cluster 0 2.18 scrub starts 2024-03-11T21:50:19.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.365101+0000 osd.5 (osd.5) 606 : cluster 0 2.18 scrub ok 2024-03-11T21:50:19.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.960429+0000 osd.3 (osd.3) 687 : cluster 0 3.8 scrub starts 2024-03-11T21:50:19.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:19 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.965025+0000 osd.3 (osd.3) 688 : cluster 0 3.8 scrub ok 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.305755+0000 osd.1 (osd.1) 485 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.306151+0000 osd.1 (osd.1) 486 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.942636+0000 osd.7 (osd.7) 700 : cluster 0 9.11 scrub starts 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:18.945632+0000 osd.7 (osd.7) 701 : cluster 0 9.11 scrub ok 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.196429+0000 osd.4 (osd.4) 645 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.199025+0000 osd.4 (osd.4) 646 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.238212+0000 mon.a (mon.0) 4179 : cluster 0 osdmap e1016: 8 total, 8 up, 8 in 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.458515+0000 osd.0 (osd.0) 686 : cluster 0 9.1c scrub starts 2024-03-11T21:50:20.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:20 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.460920+0000 osd.0 (osd.0) 687 : cluster 0 9.1c scrub ok 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.305755+0000 osd.1 (osd.1) 485 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.306151+0000 osd.1 (osd.1) 486 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.942636+0000 osd.7 (osd.7) 700 : cluster 0 9.11 scrub starts 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:18.945632+0000 osd.7 (osd.7) 701 : cluster 0 9.11 scrub ok 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.196429+0000 osd.4 (osd.4) 645 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.199025+0000 osd.4 (osd.4) 646 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.238212+0000 mon.a (mon.0) 4179 : cluster 0 osdmap e1016: 8 total, 8 up, 8 in 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.458515+0000 osd.0 (osd.0) 686 : cluster 0 9.1c scrub starts 2024-03-11T21:50:20.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.460920+0000 osd.0 (osd.0) 687 : cluster 0 9.1c scrub ok 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.305755+0000 osd.1 (osd.1) 485 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.306151+0000 osd.1 (osd.1) 486 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.942636+0000 osd.7 (osd.7) 700 : cluster 0 9.11 scrub starts 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:18.945632+0000 osd.7 (osd.7) 701 : cluster 0 9.11 scrub ok 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.196429+0000 osd.4 (osd.4) 645 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.199025+0000 osd.4 (osd.4) 646 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.238212+0000 mon.a (mon.0) 4179 : cluster 0 osdmap e1016: 8 total, 8 up, 8 in 2024-03-11T21:50:20.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.458515+0000 osd.0 (osd.0) 686 : cluster 0 9.1c scrub starts 2024-03-11T21:50:20.704 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:20 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.460920+0000 osd.0 (osd.0) 687 : cluster 0 9.1c scrub ok 2024-03-11T21:50:21.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: audit 2024-03-11T21:50:19.476951+0000 mgr.y (mgr.14550) 1041 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:19.887792+0000 mgr.y (mgr.14550) 1042 : cluster 0 pgmap v1713: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.171324+0000 osd.5 (osd.5) 607 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.171685+0000 osd.5 (osd.5) 608 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.268388+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e1017: 8 total, 8 up, 8 in 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.408803+0000 osd.5 (osd.5) 609 : cluster 0 6.d scrub starts 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.411730+0000 osd.5 (osd.5) 610 : cluster 0 6.d scrub ok 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.986471+0000 osd.3 (osd.3) 689 : cluster 0 5.0 scrub starts 2024-03-11T21:50:21.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:21 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.989324+0000 osd.3 (osd.3) 690 : cluster 0 5.0 scrub ok 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: audit 2024-03-11T21:50:19.476951+0000 mgr.y (mgr.14550) 1041 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:19.887792+0000 mgr.y (mgr.14550) 1042 : cluster 0 pgmap v1713: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.171324+0000 osd.5 (osd.5) 607 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.171685+0000 osd.5 (osd.5) 608 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.268388+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e1017: 8 total, 8 up, 8 in 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.408803+0000 osd.5 (osd.5) 609 : cluster 0 6.d scrub starts 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.411730+0000 osd.5 (osd.5) 610 : cluster 0 6.d scrub ok 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.986471+0000 osd.3 (osd.3) 689 : cluster 0 5.0 scrub starts 2024-03-11T21:50:21.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.989324+0000 osd.3 (osd.3) 690 : cluster 0 5.0 scrub ok 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: audit 2024-03-11T21:50:19.476951+0000 mgr.y (mgr.14550) 1041 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:19.887792+0000 mgr.y (mgr.14550) 1042 : cluster 0 pgmap v1713: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.171324+0000 osd.5 (osd.5) 607 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.171685+0000 osd.5 (osd.5) 608 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.268388+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e1017: 8 total, 8 up, 8 in 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.408803+0000 osd.5 (osd.5) 609 : cluster 0 6.d scrub starts 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.411730+0000 osd.5 (osd.5) 610 : cluster 0 6.d scrub ok 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.986471+0000 osd.3 (osd.3) 689 : cluster 0 5.0 scrub starts 2024-03-11T21:50:21.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:21 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.989324+0000 osd.3 (osd.3) 690 : cluster 0 5.0 scrub ok 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.877919+0000 osd.1 (osd.1) 487 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.880184+0000 osd.1 (osd.1) 488 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.943594+0000 osd.7 (osd.7) 702 : cluster 0 3.13 scrub starts 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:20.947595+0000 osd.7 (osd.7) 703 : cluster 0 3.13 scrub ok 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:21.447439+0000 osd.5 (osd.5) 611 : cluster 0 1.12 deep-scrub starts 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:21.449835+0000 osd.5 (osd.5) 612 : cluster 0 1.12 deep-scrub ok 2024-03-11T21:50:22.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:21.477724+0000 osd.0 (osd.0) 688 : cluster 0 6.18 deep-scrub starts 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:21.480105+0000 osd.0 (osd.0) 689 : cluster 0 6.18 deep-scrub ok 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[29284]: cluster 2024-03-11T21:50:21.888863+0000 mgr.y (mgr.14550) 1043 : cluster 0 pgmap v1715: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.877919+0000 osd.1 (osd.1) 487 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.880184+0000 osd.1 (osd.1) 488 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.943594+0000 osd.7 (osd.7) 702 : cluster 0 3.13 scrub starts 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:20.947595+0000 osd.7 (osd.7) 703 : cluster 0 3.13 scrub ok 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:21.447439+0000 osd.5 (osd.5) 611 : cluster 0 1.12 deep-scrub starts 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:21.449835+0000 osd.5 (osd.5) 612 : cluster 0 1.12 deep-scrub ok 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:21.477724+0000 osd.0 (osd.0) 688 : cluster 0 6.18 deep-scrub starts 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:21.480105+0000 osd.0 (osd.0) 689 : cluster 0 6.18 deep-scrub ok 2024-03-11T21:50:22.453 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:22 smithi143 bash[21512]: cluster 2024-03-11T21:50:21.888863+0000 mgr.y (mgr.14550) 1043 : cluster 0 pgmap v1715: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.877919+0000 osd.1 (osd.1) 487 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.880184+0000 osd.1 (osd.1) 488 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.943594+0000 osd.7 (osd.7) 702 : cluster 0 3.13 scrub starts 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:20.947595+0000 osd.7 (osd.7) 703 : cluster 0 3.13 scrub ok 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:21.447439+0000 osd.5 (osd.5) 611 : cluster 0 1.12 deep-scrub starts 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:21.449835+0000 osd.5 (osd.5) 612 : cluster 0 1.12 deep-scrub ok 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:21.477724+0000 osd.0 (osd.0) 688 : cluster 0 6.18 deep-scrub starts 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:21.480105+0000 osd.0 (osd.0) 689 : cluster 0 6.18 deep-scrub ok 2024-03-11T21:50:22.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:22 smithi203 bash[23654]: cluster 2024-03-11T21:50:21.888863+0000 mgr.y (mgr.14550) 1043 : cluster 0 pgmap v1715: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.157625+0000 osd.4 (osd.4) 647 : cluster 0 9.10 scrub starts 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.160102+0000 osd.4 (osd.4) 648 : cluster 0 9.10 scrub ok 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.485227+0000 osd.0 (osd.0) 690 : cluster 0 6.7 scrub starts 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.487529+0000 osd.0 (osd.0) 691 : cluster 0 6.7 scrub ok 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.934269+0000 osd.3 (osd.3) 691 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.936993+0000 osd.3 (osd.3) 692 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.157625+0000 osd.4 (osd.4) 647 : cluster 0 9.10 scrub starts 2024-03-11T21:50:23.452 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.160102+0000 osd.4 (osd.4) 648 : cluster 0 9.10 scrub ok 2024-03-11T21:50:23.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.485227+0000 osd.0 (osd.0) 690 : cluster 0 6.7 scrub starts 2024-03-11T21:50:23.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.487529+0000 osd.0 (osd.0) 691 : cluster 0 6.7 scrub ok 2024-03-11T21:50:23.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.934269+0000 osd.3 (osd.3) 691 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:50:23.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:23 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.936993+0000 osd.3 (osd.3) 692 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:50:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.157625+0000 osd.4 (osd.4) 647 : cluster 0 9.10 scrub starts 2024-03-11T21:50:23.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.160102+0000 osd.4 (osd.4) 648 : cluster 0 9.10 scrub ok 2024-03-11T21:50:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.485227+0000 osd.0 (osd.0) 690 : cluster 0 6.7 scrub starts 2024-03-11T21:50:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.487529+0000 osd.0 (osd.0) 691 : cluster 0 6.7 scrub ok 2024-03-11T21:50:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.934269+0000 osd.3 (osd.3) 691 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:50:23.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:23 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.936993+0000 osd.3 (osd.3) 692 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:50:23.616 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.837206+0000 osd.1 (osd.1) 489 : cluster 0 3.19 scrub starts 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.841882+0000 osd.1 (osd.1) 490 : cluster 0 3.19 scrub ok 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.870574+0000 osd.7 (osd.7) 704 : cluster 0 9.e scrub starts 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:22.875184+0000 osd.7 (osd.7) 705 : cluster 0 9.e scrub ok 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:23.160503+0000 osd.4 (osd.4) 649 : cluster 0 6.1e scrub starts 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:23.162884+0000 osd.4 (osd.4) 650 : cluster 0 6.1e scrub ok 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: audit 2024-03-11T21:50:23.392982+0000 mon.c (mon.2) 1434 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:23.890941+0000 mgr.y (mgr.14550) 1044 : cluster 0 pgmap v1716: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:23.961438+0000 osd.3 (osd.3) 693 : cluster 0 6.11 scrub starts 2024-03-11T21:50:24.452 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[21512]: cluster 2024-03-11T21:50:23.963599+0000 osd.3 (osd.3) 694 : cluster 0 6.11 scrub ok 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.837206+0000 osd.1 (osd.1) 489 : cluster 0 3.19 scrub starts 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.841882+0000 osd.1 (osd.1) 490 : cluster 0 3.19 scrub ok 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.870574+0000 osd.7 (osd.7) 704 : cluster 0 9.e scrub starts 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:22.875184+0000 osd.7 (osd.7) 705 : cluster 0 9.e scrub ok 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:23.160503+0000 osd.4 (osd.4) 649 : cluster 0 6.1e scrub starts 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:23.162884+0000 osd.4 (osd.4) 650 : cluster 0 6.1e scrub ok 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: audit 2024-03-11T21:50:23.392982+0000 mon.c (mon.2) 1434 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:23.890941+0000 mgr.y (mgr.14550) 1044 : cluster 0 pgmap v1716: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:23.961438+0000 osd.3 (osd.3) 693 : cluster 0 6.11 scrub starts 2024-03-11T21:50:24.453 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:24 smithi143 bash[29284]: cluster 2024-03-11T21:50:23.963599+0000 osd.3 (osd.3) 694 : cluster 0 6.11 scrub ok 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.837206+0000 osd.1 (osd.1) 489 : cluster 0 3.19 scrub starts 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.841882+0000 osd.1 (osd.1) 490 : cluster 0 3.19 scrub ok 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.870574+0000 osd.7 (osd.7) 704 : cluster 0 9.e scrub starts 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:22.875184+0000 osd.7 (osd.7) 705 : cluster 0 9.e scrub ok 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:23.160503+0000 osd.4 (osd.4) 649 : cluster 0 6.1e scrub starts 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:23.162884+0000 osd.4 (osd.4) 650 : cluster 0 6.1e scrub ok 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: audit 2024-03-11T21:50:23.392982+0000 mon.c (mon.2) 1434 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:23.890941+0000 mgr.y (mgr.14550) 1044 : cluster 0 pgmap v1716: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:23.961438+0000 osd.3 (osd.3) 693 : cluster 0 6.11 scrub starts 2024-03-11T21:50:24.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:24 smithi203 bash[23654]: cluster 2024-03-11T21:50:23.963599+0000 osd.3 (osd.3) 694 : cluster 0 6.11 scrub ok 2024-03-11T21:50:25.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:25 smithi203 bash[23654]: cluster 2024-03-11T21:50:24.394117+0000 osd.5 (osd.5) 613 : cluster 0 1.1 scrub starts 2024-03-11T21:50:25.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:25 smithi203 bash[23654]: cluster 2024-03-11T21:50:24.396994+0000 osd.5 (osd.5) 614 : cluster 0 1.1 scrub ok 2024-03-11T21:50:25.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:25 smithi143 bash[21512]: cluster 2024-03-11T21:50:24.394117+0000 osd.5 (osd.5) 613 : cluster 0 1.1 scrub starts 2024-03-11T21:50:25.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:25 smithi143 bash[21512]: cluster 2024-03-11T21:50:24.396994+0000 osd.5 (osd.5) 614 : cluster 0 1.1 scrub ok 2024-03-11T21:50:25.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:25 smithi143 bash[29284]: cluster 2024-03-11T21:50:24.394117+0000 osd.5 (osd.5) 613 : cluster 0 1.1 scrub starts 2024-03-11T21:50:25.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:25 smithi143 bash[29284]: cluster 2024-03-11T21:50:24.396994+0000 osd.5 (osd.5) 614 : cluster 0 1.1 scrub ok 2024-03-11T21:50:26.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:26 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.361748+0000 osd.5 (osd.5) 615 : cluster 0 9.12 scrub starts 2024-03-11T21:50:26.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:26 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.364330+0000 osd.5 (osd.5) 616 : cluster 0 9.12 scrub ok 2024-03-11T21:50:26.532 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:26 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.891716+0000 mgr.y (mgr.14550) 1045 : cluster 0 pgmap v1717: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:26.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:26 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.996490+0000 osd.3 (osd.3) 695 : cluster 0 3.2 scrub starts 2024-03-11T21:50:26.533 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:26 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.999025+0000 osd.3 (osd.3) 696 : cluster 0 3.2 scrub ok 2024-03-11T21:50:26.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.361748+0000 osd.5 (osd.5) 615 : cluster 0 9.12 scrub starts 2024-03-11T21:50:26.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.364330+0000 osd.5 (osd.5) 616 : cluster 0 9.12 scrub ok 2024-03-11T21:50:26.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.891716+0000 mgr.y (mgr.14550) 1045 : cluster 0 pgmap v1717: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:26.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.996490+0000 osd.3 (osd.3) 695 : cluster 0 3.2 scrub starts 2024-03-11T21:50:26.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.999025+0000 osd.3 (osd.3) 696 : cluster 0 3.2 scrub ok 2024-03-11T21:50:26.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.361748+0000 osd.5 (osd.5) 615 : cluster 0 9.12 scrub starts 2024-03-11T21:50:26.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.364330+0000 osd.5 (osd.5) 616 : cluster 0 9.12 scrub ok 2024-03-11T21:50:26.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.891716+0000 mgr.y (mgr.14550) 1045 : cluster 0 pgmap v1717: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:26.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.996490+0000 osd.3 (osd.3) 695 : cluster 0 3.2 scrub starts 2024-03-11T21:50:26.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:26 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.999025+0000 osd.3 (osd.3) 696 : cluster 0 3.2 scrub ok 2024-03-11T21:50:27.701 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.645329+0000 osd.6 (osd.6) 543 : cluster 0 2.11 scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:25.647420+0000 osd.6 (osd.6) 544 : cluster 0 2.11 scrub ok 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.525719+0000 osd.0 (osd.0) 692 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.528341+0000 osd.0 (osd.0) 693 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.968478+0000 osd.3 (osd.3) 697 : cluster 0 6.c scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.970940+0000 osd.3 (osd.3) 698 : cluster 0 6.c scrub ok 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.645329+0000 osd.6 (osd.6) 543 : cluster 0 2.11 scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:25.647420+0000 osd.6 (osd.6) 544 : cluster 0 2.11 scrub ok 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.525719+0000 osd.0 (osd.0) 692 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.528341+0000 osd.0 (osd.0) 693 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.968478+0000 osd.3 (osd.3) 697 : cluster 0 6.c scrub starts 2024-03-11T21:50:27.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:27 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.970940+0000 osd.3 (osd.3) 698 : cluster 0 6.c scrub ok 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.645329+0000 osd.6 (osd.6) 543 : cluster 0 2.11 scrub starts 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:25.647420+0000 osd.6 (osd.6) 544 : cluster 0 2.11 scrub ok 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.525719+0000 osd.0 (osd.0) 692 : cluster 0 6.0 deep-scrub starts 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.528341+0000 osd.0 (osd.0) 693 : cluster 0 6.0 deep-scrub ok 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.968478+0000 osd.3 (osd.3) 697 : cluster 0 6.c scrub starts 2024-03-11T21:50:27.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:27 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.970940+0000 osd.3 (osd.3) 698 : cluster 0 6.c scrub ok 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- 172.21.15.143:0/2663565249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e01044c0 msgr2=0x7f95e010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- 172.21.15.143:0/2663565249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e01044c0 0x7f95e010ee70 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009990 tx=0x7f95d8009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- 172.21.15.143:0/2663565249 shutdown_connections 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- 172.21.15.143:0/2663565249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e01044c0 0x7f95e010ee70 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- 172.21.15.143:0/2663565249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e006abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.040 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- 172.21.15.143:0/2663565249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95e0103b00 0x7f95e0103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- 172.21.15.143:0/2663565249 >> 172.21.15.143:0/2663565249 conn(0x7f95e0076d80 msgr2=0x7f95e0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- 172.21.15.143:0/2663565249 shutdown_connections 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- 172.21.15.143:0/2663565249 wait complete. 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 librados: starting msgr at 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 librados: starting objecter 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 Processor -- start 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- start start 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 librados: setting wanted keys 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 librados: calling monclient init 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e0103b00 0x7f95e019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95e01044c0 0x7f95e0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e01117a0 con 0x7f95e0103b00 2024-03-11T21:50:28.041 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e0111620 con 0x7f95e01044c0 2024-03-11T21:50:28.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95e5723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e0111920 con 0x7f95e006a7f0 2024-03-11T21:50:28.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95deffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:28.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95deffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:52554/0 (socket says 172.21.15.143:52554) 2024-03-11T21:50:28.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95deffd700 1 -- 172.21.15.143:0/1914101444 learned_addr learned my addr 172.21.15.143:0/1914101444 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:28.042 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.047+0000 7f95de7fc700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e0103b00 0x7f95e019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:28.043 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 -- 172.21.15.143:0/1914101444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95e01044c0 msgr2=0x7f95e0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95e01044c0 0x7f95e0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 -- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e0103b00 msgr2=0x7f95e019dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e0103b00 0x7f95e019dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 -- 172.21.15.143:0/1914101444 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0198410 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95deffd700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f95d000ef10 tx=0x7f95d000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 10 client.?.objecter ms_handle_connect 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d000ca70 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7f95d0019070 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e01986a0 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0014480 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95e5723700 10 client.42527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e0100fb0 con 0x7f95e006a7f0 2024-03-11T21:50:28.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95e5723700 1 librados: init done 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e0198a10 con 0x7f95e006a7f0 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f95d0012050 con 0x7f95e006a7f0 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 0x7f95c8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1017..1017 src has 964..1017) v4 ==== 8794+0+0 (secure 0 0 0) 0x7f95d008e810 con 0x7f95e006a7f0 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 10 client.42527.objecter ms_dispatch 0x7f95e0001550 osd_map(1017..1017 src has 964..1017) v4 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 3 client.42527.objecter handle_osd_map got epochs [1017,1017] > 0 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 3 client.42527.objecter handle_osd_map decoding full epoch 1017 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95de7fc700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 0x7f95c8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 20 client.42527.objecter dump_active .. 0 homeless 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95c7fff700 10 client.42527.objecter ms_handle_connect 0x7f95c806f820 2024-03-11T21:50:28.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.051+0000 7f95de7fc700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 0x7f95c8071ce0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f95e0199280 tx=0x7f95d400a300 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:28.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.055+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f95e0198a10 con 0x7f95e006a7f0 2024-03-11T21:50:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.267+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95e004faa0 con 0x7f95c806f820 2024-03-11T21:50:28.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.319+0000 7f95c7fff700 1 -- 172.21.15.143:0/1914101444 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+477550 (secure 0 0 0) 0x7f95e004faa0 con 0x7f95c806f820 2024-03-11T21:50:28.314 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:50:28.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 10 librados: watch_flush enter 2024-03-11T21:50:28.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 10 librados: watch_flush exit 2024-03-11T21:50:28.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 20 client.42527.objecter shutdown clearing up homeless session... 2024-03-11T21:50:28.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 10 client.42527.objecter successfully canceled tick 2024-03-11T21:50:28.319 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 msgr2=0x7f95c8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 0x7f95c8071ce0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f95e0199280 tx=0x7f95d400a300 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 msgr2=0x7f95e019da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f95d000ef10 tx=0x7f95d000c3a0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 shutdown_connections 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7f95c806f820 0x7f95c8071ce0 unknown :-1 s=CLOSED pgs=3531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95e01044c0 0x7f95e0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95e006a7f0 0x7f95e019da60 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 --2- 172.21.15.143:0/1914101444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95e0103b00 0x7f95e019dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 >> 172.21.15.143:0/1914101444 conn(0x7f95e0076d80 msgr2=0x7f95e0106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 shutdown_connections 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 -- 172.21.15.143:0/1914101444 wait complete. 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:28.323+0000 7f95e5723700 1 librados: shutdown 2024-03-11T21:50:28.320 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.773284+0000 osd.7 (osd.7) 706 : cluster 0 1.9 scrub starts 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:26.775859+0000 osd.7 (osd.7) 707 : cluster 0 1.9 scrub ok 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.171355+0000 osd.4 (osd.4) 651 : cluster 0 5.1c scrub starts 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.173643+0000 osd.4 (osd.4) 652 : cluster 0 5.1c scrub ok 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.447760+0000 osd.5 (osd.5) 617 : cluster 0 3.1d scrub starts 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.451945+0000 osd.5 (osd.5) 618 : cluster 0 3.1d scrub ok 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: audit 2024-03-11T21:50:27.706138+0000 mon.a (mon.0) 4181 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: audit 2024-03-11T21:50:27.723861+0000 mon.a (mon.0) 4182 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.583 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: audit 2024-03-11T21:50:27.728819+0000 mon.c (mon.2) 1435 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: audit 2024-03-11T21:50:27.730307+0000 mon.c (mon.2) 1436 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: audit 2024-03-11T21:50:27.741633+0000 mon.a (mon.0) 4183 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.855395+0000 osd.1 (osd.1) 491 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.861401+0000 osd.1 (osd.1) 492 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.892193+0000 mgr.y (mgr.14550) 1046 : cluster 0 pgmap v1718: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.941687+0000 osd.3 (osd.3) 699 : cluster 0 5.14 scrub starts 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[21512]: cluster 2024-03-11T21:50:27.943473+0000 osd.3 (osd.3) 700 : cluster 0 5.14 scrub ok 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.773284+0000 osd.7 (osd.7) 706 : cluster 0 1.9 scrub starts 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:26.775859+0000 osd.7 (osd.7) 707 : cluster 0 1.9 scrub ok 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.171355+0000 osd.4 (osd.4) 651 : cluster 0 5.1c scrub starts 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.173643+0000 osd.4 (osd.4) 652 : cluster 0 5.1c scrub ok 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.447760+0000 osd.5 (osd.5) 617 : cluster 0 3.1d scrub starts 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.451945+0000 osd.5 (osd.5) 618 : cluster 0 3.1d scrub ok 2024-03-11T21:50:28.584 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: audit 2024-03-11T21:50:27.706138+0000 mon.a (mon.0) 4181 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: audit 2024-03-11T21:50:27.723861+0000 mon.a (mon.0) 4182 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: audit 2024-03-11T21:50:27.728819+0000 mon.c (mon.2) 1435 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: audit 2024-03-11T21:50:27.730307+0000 mon.c (mon.2) 1436 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: audit 2024-03-11T21:50:27.741633+0000 mon.a (mon.0) 4183 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.855395+0000 osd.1 (osd.1) 491 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.861401+0000 osd.1 (osd.1) 492 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.892193+0000 mgr.y (mgr.14550) 1046 : cluster 0 pgmap v1718: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.941687+0000 osd.3 (osd.3) 699 : cluster 0 5.14 scrub starts 2024-03-11T21:50:28.585 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:28 smithi143 bash[29284]: cluster 2024-03-11T21:50:27.943473+0000 osd.3 (osd.3) 700 : cluster 0 5.14 scrub ok 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.773284+0000 osd.7 (osd.7) 706 : cluster 0 1.9 scrub starts 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:26.775859+0000 osd.7 (osd.7) 707 : cluster 0 1.9 scrub ok 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.171355+0000 osd.4 (osd.4) 651 : cluster 0 5.1c scrub starts 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.173643+0000 osd.4 (osd.4) 652 : cluster 0 5.1c scrub ok 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.447760+0000 osd.5 (osd.5) 617 : cluster 0 3.1d scrub starts 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.451945+0000 osd.5 (osd.5) 618 : cluster 0 3.1d scrub ok 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: audit 2024-03-11T21:50:27.706138+0000 mon.a (mon.0) 4181 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: audit 2024-03-11T21:50:27.723861+0000 mon.a (mon.0) 4182 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: audit 2024-03-11T21:50:27.728819+0000 mon.c (mon.2) 1435 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: audit 2024-03-11T21:50:27.730307+0000 mon.c (mon.2) 1436 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:28.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: audit 2024-03-11T21:50:27.741633+0000 mon.a (mon.0) 4183 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:28.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.855395+0000 osd.1 (osd.1) 491 : cluster 0 1.13 deep-scrub starts 2024-03-11T21:50:28.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.861401+0000 osd.1 (osd.1) 492 : cluster 0 1.13 deep-scrub ok 2024-03-11T21:50:28.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.892193+0000 mgr.y (mgr.14550) 1046 : cluster 0 pgmap v1718: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-11T21:50:28.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.941687+0000 osd.3 (osd.3) 699 : cluster 0 5.14 scrub starts 2024-03-11T21:50:28.784 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:28 smithi203 bash[23654]: cluster 2024-03-11T21:50:27.943473+0000 osd.3 (osd.3) 700 : cluster 0 5.14 scrub ok 2024-03-11T21:50:29.328 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2024-03-11T21:50:27.891906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3217,"num_read_kb":2969,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3907,"ondisk_log_size":3907,"up":552,"acting":552,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":552,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":804884,"kb_used_data":20072,"kb_used_omap":38,"kb_used_meta":784665,"kb_avail":749156332,"statfs":{"total":767960285184,"available":767136083968,"internally_reserved":0,"allocated":20553728,"data_stored":6840599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":803497397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.659925"},"pg_stats":[{"pgid":"1.d","version":"98'128","reported_seq":594,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.879720+0000","last_change":"2024-03-11T21:49:57.941524+0000","last_active":"2024-03-11T21:50:20.879720+0000","last_peered":"2024-03-11T21:50:20.879720+0000","last_clean":"2024-03-11T21:50:20.879720+0000","last_became_active":"2024-03-11T21:44:03.844629+0000","last_became_peered":"2024-03-11T21:44:03.844629+0000","last_unstale":"2024-03-11T21:50:20.879720+0000","last_undegraded":"2024-03-11T21:50:20.879720+0000","last_fullsized":"2024-03-11T21:50:20.879720+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:57.941299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.941299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.824265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1832,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114117+0000","last_change":"2024-03-11T21:49:45.202481+0000","last_active":"2024-03-11T21:50:21.114117+0000","last_peered":"2024-03-11T21:50:21.114117+0000","last_clean":"2024-03-11T21:50:21.114117+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:50:21.114117+0000","last_undegraded":"2024-03-11T21:50:21.114117+0000","last_fullsized":"2024-03-11T21:50:21.114117+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:45.202279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:19.937242+0000","last_clean_scrub_stamp":"2024-03-11T21:49:45.202279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.007736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":2288,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877223+0000","last_change":"2024-03-11T21:48:55.188592+0000","last_active":"2024-03-11T21:50:20.877223+0000","last_peered":"2024-03-11T21:50:20.877223+0000","last_clean":"2024-03-11T21:50:20.877223+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:50:20.877223+0000","last_undegraded":"2024-03-11T21:50:20.877223+0000","last_fullsized":"2024-03-11T21:50:20.877223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:48:55.188208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.188208+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"864'39","reported_seq":2015,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.879616+0000","last_change":"2024-03-11T21:48:58.246525+0000","last_active":"2024-03-11T21:50:20.879616+0000","last_peered":"2024-03-11T21:50:20.879616+0000","last_clean":"2024-03-11T21:50:20.879616+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:50:20.879616+0000","last_undegraded":"2024-03-11T21:50:20.879616+0000","last_fullsized":"2024-03-11T21:50:20.879616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:48:58.246227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.246227+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1969,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.266171+0000","last_change":"2024-03-11T21:50:16.497449+0000","last_active":"2024-03-11T21:50:20.266171+0000","last_peered":"2024-03-11T21:50:20.266171+0000","last_clean":"2024-03-11T21:50:20.266171+0000","last_became_active":"2024-03-11T21:47:12.294681+0000","last_became_peered":"2024-03-11T21:47:12.294681+0000","last_unstale":"2024-03-11T21:50:20.266171+0000","last_undegraded":"2024-03-11T21:50:20.266171+0000","last_fullsized":"2024-03-11T21:50:20.266171+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.497231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.497231+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.497231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.271210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1768,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411762+0000","last_change":"2024-03-11T21:49:59.171605+0000","last_active":"2024-03-11T21:50:20.411762+0000","last_peered":"2024-03-11T21:50:20.411762+0000","last_clean":"2024-03-11T21:50:20.411762+0000","last_became_active":"2024-03-11T21:48:12.921751+0000","last_became_peered":"2024-03-11T21:48:12.921751+0000","last_unstale":"2024-03-11T21:50:20.411762+0000","last_undegraded":"2024-03-11T21:50:20.411762+0000","last_fullsized":"2024-03-11T21:50:20.411762+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:59.171386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.171386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.556200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"98'128","reported_seq":601,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261252+0000","last_change":"2024-03-11T21:50:02.063678+0000","last_active":"2024-03-11T21:50:20.261252+0000","last_peered":"2024-03-11T21:50:20.261252+0000","last_clean":"2024-03-11T21:50:20.261252+0000","last_became_active":"2024-03-11T21:44:03.835751+0000","last_became_peered":"2024-03-11T21:44:03.835751+0000","last_unstale":"2024-03-11T21:50:20.261252+0000","last_undegraded":"2024-03-11T21:50:20.261252+0000","last_fullsized":"2024-03-11T21:50:20.261252+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:02.063438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.063438+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.261457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1627,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412487+0000","last_change":"2024-03-11T21:50:10.176409+0000","last_active":"2024-03-11T21:50:20.412487+0000","last_peered":"2024-03-11T21:50:20.412487+0000","last_clean":"2024-03-11T21:50:20.412487+0000","last_became_active":"2024-03-11T21:48:12.933645+0000","last_became_peered":"2024-03-11T21:48:12.933645+0000","last_unstale":"2024-03-11T21:50:20.412487+0000","last_undegraded":"2024-03-11T21:50:20.412487+0000","last_fullsized":"2024-03-11T21:50:20.412487+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.176190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.176190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.935103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"864'30","reported_seq":1820,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413170+0000","last_change":"2024-03-11T21:49:26.070288+0000","last_active":"2024-03-11T21:50:20.413170+0000","last_peered":"2024-03-11T21:50:20.413170+0000","last_clean":"2024-03-11T21:50:20.413170+0000","last_became_active":"2024-03-11T21:48:12.917408+0000","last_became_peered":"2024-03-11T21:48:12.917408+0000","last_unstale":"2024-03-11T21:50:20.413170+0000","last_undegraded":"2024-03-11T21:50:20.413170+0000","last_fullsized":"2024-03-11T21:50:20.413170+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:49:26.069936+0000","last_deep_scrub":"864'30","last_deep_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_clean_scrub_stamp":"2024-03-11T21:49:26.069936+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:29.473340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1886,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264360+0000","last_change":"2024-03-11T21:49:57.844013+0000","last_active":"2024-03-11T21:50:20.264360+0000","last_peered":"2024-03-11T21:50:20.264360+0000","last_clean":"2024-03-11T21:50:20.264360+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:50:20.264360+0000","last_undegraded":"2024-03-11T21:50:20.264360+0000","last_fullsized":"2024-03-11T21:50:20.264360+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:57.843884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.843884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:01.534629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1962,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262607+0000","last_change":"2024-03-11T21:49:11.797005+0000","last_active":"2024-03-11T21:50:20.262607+0000","last_peered":"2024-03-11T21:50:20.262607+0000","last_clean":"2024-03-11T21:50:20.262607+0000","last_became_active":"2024-03-11T21:47:12.046031+0000","last_became_peered":"2024-03-11T21:47:12.046031+0000","last_unstale":"2024-03-11T21:50:20.262607+0000","last_undegraded":"2024-03-11T21:50:20.262607+0000","last_fullsized":"2024-03-11T21:50:20.262607+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:11.796833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:30.018128+0000","last_clean_scrub_stamp":"2024-03-11T21:49:11.796833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.545380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1974,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112528+0000","last_change":"2024-03-11T21:49:33.172302+0000","last_active":"2024-03-11T21:50:21.112528+0000","last_peered":"2024-03-11T21:50:21.112528+0000","last_clean":"2024-03-11T21:50:21.112528+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:50:21.112528+0000","last_undegraded":"2024-03-11T21:50:21.112528+0000","last_fullsized":"2024-03-11T21:50:21.112528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:33.172048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:49:33.172048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:45.866974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"98'128","reported_seq":1252,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114118+0000","last_change":"2024-03-11T21:50:08.542983+0000","last_active":"2024-03-11T21:50:21.114118+0000","last_peered":"2024-03-11T21:50:21.114118+0000","last_clean":"2024-03-11T21:50:21.114118+0000","last_became_active":"2024-03-11T21:44:03.836601+0000","last_became_peered":"2024-03-11T21:44:03.836601+0000","last_unstale":"2024-03-11T21:50:21.114118+0000","last_undegraded":"2024-03-11T21:50:21.114118+0000","last_fullsized":"2024-03-11T21:50:21.114118+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:08.542763+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:38.453642+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.542763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.187479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1610,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412561+0000","last_change":"2024-03-11T21:49:15.153209+0000","last_active":"2024-03-11T21:50:20.412561+0000","last_peered":"2024-03-11T21:50:20.412561+0000","last_clean":"2024-03-11T21:50:20.412561+0000","last_became_active":"2024-03-11T21:48:12.918994+0000","last_became_peered":"2024-03-11T21:48:12.918994+0000","last_unstale":"2024-03-11T21:50:20.412561+0000","last_undegraded":"2024-03-11T21:50:20.412561+0000","last_fullsized":"2024-03-11T21:50:20.412561+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:15.152963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:15.152963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.250809+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"864'53","reported_seq":2040,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878893+0000","last_change":"2024-03-11T21:49:16.179412+0000","last_active":"2024-03-11T21:50:20.878893+0000","last_peered":"2024-03-11T21:50:20.878893+0000","last_clean":"2024-03-11T21:50:20.878893+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:50:20.878893+0000","last_undegraded":"2024-03-11T21:50:20.878893+0000","last_fullsized":"2024-03-11T21:50:20.878893+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'53","last_scrub_stamp":"2024-03-11T21:49:16.179066+0000","last_deep_scrub":"864'53","last_deep_scrub_stamp":"2024-03-11T21:49:16.179066+0000","last_clean_scrub_stamp":"2024-03-11T21:49:16.179066+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:30.656025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2031,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264289+0000","last_change":"2024-03-11T21:49:52.427491+0000","last_active":"2024-03-11T21:50:20.264289+0000","last_peered":"2024-03-11T21:50:20.264289+0000","last_clean":"2024-03-11T21:50:20.264289+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:50:20.264289+0000","last_undegraded":"2024-03-11T21:50:20.264289+0000","last_fullsized":"2024-03-11T21:50:20.264289+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:52.427283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.427283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.324907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":2023,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262163+0000","last_change":"2024-03-11T21:49:54.426722+0000","last_active":"2024-03-11T21:50:20.262163+0000","last_peered":"2024-03-11T21:50:20.262163+0000","last_clean":"2024-03-11T21:50:20.262163+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:50:20.262163+0000","last_undegraded":"2024-03-11T21:50:20.262163+0000","last_fullsized":"2024-03-11T21:50:20.262163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.426445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:34.663677+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.426445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.164043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1954,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876063+0000","last_change":"2024-03-11T21:49:20.217882+0000","last_active":"2024-03-11T21:50:20.876063+0000","last_peered":"2024-03-11T21:50:20.876063+0000","last_clean":"2024-03-11T21:50:20.876063+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:50:20.876063+0000","last_undegraded":"2024-03-11T21:50:20.876063+0000","last_fullsized":"2024-03-11T21:50:20.876063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:20.217579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:45.155587+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.217579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.255994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":1007,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.410662+0000","last_change":"2024-03-11T21:50:14.215639+0000","last_active":"2024-03-11T21:50:20.410662+0000","last_peered":"2024-03-11T21:50:20.410662+0000","last_clean":"2024-03-11T21:50:20.410662+0000","last_became_active":"2024-03-11T21:48:12.901167+0000","last_became_peered":"2024-03-11T21:48:12.901167+0000","last_unstale":"2024-03-11T21:50:20.410662+0000","last_undegraded":"2024-03-11T21:50:20.410662+0000","last_fullsized":"2024-03-11T21:50:20.410662+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:14.215402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.215402+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:37.269457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1706,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.263669+0000","last_change":"2024-03-11T21:49:20.987439+0000","last_active":"2024-03-11T21:50:20.263669+0000","last_peered":"2024-03-11T21:50:20.263669+0000","last_clean":"2024-03-11T21:50:20.263669+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:50:20.263669+0000","last_undegraded":"2024-03-11T21:50:20.263669+0000","last_fullsized":"2024-03-11T21:50:20.263669+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:20.987221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:58.872662+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.987221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:39.254446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"864'60","reported_seq":2103,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113024+0000","last_change":"2024-03-11T21:49:24.607372+0000","last_active":"2024-03-11T21:50:21.113024+0000","last_peered":"2024-03-11T21:50:21.113024+0000","last_clean":"2024-03-11T21:50:21.113024+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:50:21.113024+0000","last_undegraded":"2024-03-11T21:50:21.113024+0000","last_fullsized":"2024-03-11T21:50:21.113024+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'60","last_scrub_stamp":"2024-03-11T21:49:24.607132+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.607132+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:46.992895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1966,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877314+0000","last_change":"2024-03-11T21:49:00.174305+0000","last_active":"2024-03-11T21:50:20.877314+0000","last_peered":"2024-03-11T21:50:20.877314+0000","last_clean":"2024-03-11T21:50:20.877314+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:50:20.877314+0000","last_undegraded":"2024-03-11T21:50:20.877314+0000","last_fullsized":"2024-03-11T21:50:20.877314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:00.174079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:00.174079+0000","last_clean_scrub_stamp":"2024-03-11T21:49:00.174079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:25.932355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1884,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.263260+0000","last_change":"2024-03-11T21:49:15.011676+0000","last_active":"2024-03-11T21:50:20.263260+0000","last_peered":"2024-03-11T21:50:20.263260+0000","last_clean":"2024-03-11T21:50:20.263260+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:50:20.263260+0000","last_undegraded":"2024-03-11T21:50:20.263260+0000","last_fullsized":"2024-03-11T21:50:20.263260+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:15.011473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:00.292001+0000","last_clean_scrub_stamp":"2024-03-11T21:49:15.011473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:20.352086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1965,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.265359+0000","last_change":"2024-03-11T21:50:15.484221+0000","last_active":"2024-03-11T21:50:20.265359+0000","last_peered":"2024-03-11T21:50:20.265359+0000","last_clean":"2024-03-11T21:50:20.265359+0000","last_became_active":"2024-03-11T21:47:12.046816+0000","last_became_peered":"2024-03-11T21:47:12.046816+0000","last_unstale":"2024-03-11T21:50:20.265359+0000","last_undegraded":"2024-03-11T21:50:20.265359+0000","last_fullsized":"2024-03-11T21:50:20.265359+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.484002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.847708+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.484002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.211380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":2251,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877747+0000","last_change":"2024-03-11T21:48:44.167031+0000","last_active":"2024-03-11T21:50:20.877747+0000","last_peered":"2024-03-11T21:50:20.877747+0000","last_clean":"2024-03-11T21:50:20.877747+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:50:20.877747+0000","last_undegraded":"2024-03-11T21:50:20.877747+0000","last_fullsized":"2024-03-11T21:50:20.877747+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:44.166875+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:44.166875+0000","last_clean_scrub_stamp":"2024-03-11T21:48:44.166875+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1828,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871418+0000","last_change":"2024-03-11T21:50:13.959402+0000","last_active":"2024-03-11T21:50:20.871418+0000","last_peered":"2024-03-11T21:50:20.871418+0000","last_clean":"2024-03-11T21:50:20.871418+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:50:20.871418+0000","last_undegraded":"2024-03-11T21:50:20.871418+0000","last_fullsized":"2024-03-11T21:50:20.871418+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:50:13.959166+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.959166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.656040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"864'21","reported_seq":2005,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871919+0000","last_change":"2024-03-11T21:49:58.753679+0000","last_active":"2024-03-11T21:50:20.871919+0000","last_peered":"2024-03-11T21:50:20.871919+0000","last_clean":"2024-03-11T21:50:20.871919+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:50:20.871919+0000","last_undegraded":"2024-03-11T21:50:20.871919+0000","last_fullsized":"2024-03-11T21:50:20.871919+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:58.753535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.753535+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:04.686060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1852,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.267609+0000","last_change":"2024-03-11T21:49:09.050688+0000","last_active":"2024-03-11T21:50:20.267609+0000","last_peered":"2024-03-11T21:50:20.267609+0000","last_clean":"2024-03-11T21:50:20.267609+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:50:20.267609+0000","last_undegraded":"2024-03-11T21:50:20.267609+0000","last_fullsized":"2024-03-11T21:50:20.267609+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:09.050558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:49:09.050558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:35.993285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":2362,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.265728+0000","last_change":"2024-03-11T21:50:08.398539+0000","last_active":"2024-03-11T21:50:20.265728+0000","last_peered":"2024-03-11T21:50:20.265728+0000","last_clean":"2024-03-11T21:50:20.265728+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:50:20.265728+0000","last_undegraded":"2024-03-11T21:50:20.265728+0000","last_fullsized":"2024-03-11T21:50:20.265728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:50:08.398448+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:47:29.724498+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.398448+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.217831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2008,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262796+0000","last_change":"2024-03-11T21:49:53.076319+0000","last_active":"2024-03-11T21:50:20.262796+0000","last_peered":"2024-03-11T21:50:20.262796+0000","last_clean":"2024-03-11T21:50:20.262796+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:50:20.262796+0000","last_undegraded":"2024-03-11T21:50:20.262796+0000","last_fullsized":"2024-03-11T21:50:20.262796+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.076002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.076002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.096336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":999,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411215+0000","last_change":"2024-03-11T21:49:32.022090+0000","last_active":"2024-03-11T21:50:20.411215+0000","last_peered":"2024-03-11T21:50:20.411215+0000","last_clean":"2024-03-11T21:50:20.411215+0000","last_became_active":"2024-03-11T21:48:12.927820+0000","last_became_peered":"2024-03-11T21:48:12.927820+0000","last_unstale":"2024-03-11T21:50:20.411215+0000","last_undegraded":"2024-03-11T21:50:20.411215+0000","last_fullsized":"2024-03-11T21:50:20.411215+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:32.021876+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:47:56.730128+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.021876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.676707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1823,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871858+0000","last_change":"2024-03-11T21:49:33.769769+0000","last_active":"2024-03-11T21:50:20.871858+0000","last_peered":"2024-03-11T21:50:20.871858+0000","last_clean":"2024-03-11T21:50:20.871858+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:50:20.871858+0000","last_undegraded":"2024-03-11T21:50:20.871858+0000","last_fullsized":"2024-03-11T21:50:20.871858+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:33.769541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:33.769541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.713101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"864'39","reported_seq":2041,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871416+0000","last_change":"2024-03-11T21:50:18.965374+0000","last_active":"2024-03-11T21:50:20.871416+0000","last_peered":"2024-03-11T21:50:20.871416+0000","last_clean":"2024-03-11T21:50:20.871416+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:50:20.871416+0000","last_undegraded":"2024-03-11T21:50:20.871416+0000","last_fullsized":"2024-03-11T21:50:20.871416+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:50:18.965036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.965036+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.703476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1888,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.880295+0000","last_change":"2024-03-11T21:50:20.880324+0000","last_active":"2024-03-11T21:50:20.880295+0000","last_peered":"2024-03-11T21:50:20.880295+0000","last_clean":"2024-03-11T21:50:20.880295+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:50:20.880295+0000","last_undegraded":"2024-03-11T21:50:20.880295+0000","last_fullsized":"2024-03-11T21:50:20.880295+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.880191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:20.880191+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.880191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.831112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"996'12","reported_seq":2342,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262576+0000","last_change":"2024-03-11T21:50:13.093620+0000","last_active":"2024-03-11T21:50:20.262576+0000","last_peered":"2024-03-11T21:50:20.262576+0000","last_clean":"2024-03-11T21:50:20.262576+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:50:20.262576+0000","last_undegraded":"2024-03-11T21:50:20.262576+0000","last_fullsized":"2024-03-11T21:50:20.262576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'12","last_scrub_stamp":"2024-03-11T21:50:13.093274+0000","last_deep_scrub":"996'12","last_deep_scrub_stamp":"2024-03-11T21:47:41.169471+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.093274+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.341385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1776,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412881+0000","last_change":"2024-03-11T21:50:20.412097+0000","last_active":"2024-03-11T21:50:20.412881+0000","last_peered":"2024-03-11T21:50:20.412881+0000","last_clean":"2024-03-11T21:50:20.412881+0000","last_became_active":"2024-03-11T21:48:12.931965+0000","last_became_peered":"2024-03-11T21:48:12.931965+0000","last_unstale":"2024-03-11T21:50:20.412881+0000","last_undegraded":"2024-03-11T21:50:20.412881+0000","last_fullsized":"2024-03-11T21:50:20.412881+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.411748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.411748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.330409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"98'128","reported_seq":1238,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.268467+0000","last_change":"2024-03-11T21:50:13.895680+0000","last_active":"2024-03-11T21:50:20.268467+0000","last_peered":"2024-03-11T21:50:20.268467+0000","last_clean":"2024-03-11T21:50:20.268467+0000","last_became_active":"2024-03-11T21:44:03.849412+0000","last_became_peered":"2024-03-11T21:44:03.849412+0000","last_unstale":"2024-03-11T21:50:20.268467+0000","last_undegraded":"2024-03-11T21:50:20.268467+0000","last_fullsized":"2024-03-11T21:50:20.268467+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:13.895439+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:40.094801+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.895439+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.462926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1813,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876585+0000","last_change":"2024-03-11T21:50:09.973954+0000","last_active":"2024-03-11T21:50:20.876585+0000","last_peered":"2024-03-11T21:50:20.876585+0000","last_clean":"2024-03-11T21:50:20.876585+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:50:20.876585+0000","last_undegraded":"2024-03-11T21:50:20.876585+0000","last_fullsized":"2024-03-11T21:50:20.876585+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:09.973736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.973736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.170519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"864'29","reported_seq":2045,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259678+0000","last_change":"2024-03-11T21:49:31.093759+0000","last_active":"2024-03-11T21:50:20.259678+0000","last_peered":"2024-03-11T21:50:20.259678+0000","last_clean":"2024-03-11T21:50:20.259678+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:50:20.259678+0000","last_undegraded":"2024-03-11T21:50:20.259678+0000","last_fullsized":"2024-03-11T21:50:20.259678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'29","last_scrub_stamp":"2024-03-11T21:49:31.093510+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:49:31.093510+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.763626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1983,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876385+0000","last_change":"2024-03-11T21:50:07.949808+0000","last_active":"2024-03-11T21:50:20.876385+0000","last_peered":"2024-03-11T21:50:20.876385+0000","last_clean":"2024-03-11T21:50:20.876385+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:50:20.876385+0000","last_undegraded":"2024-03-11T21:50:20.876385+0000","last_fullsized":"2024-03-11T21:50:20.876385+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.949512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.949512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.169783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1771,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413524+0000","last_change":"2024-03-11T21:50:12.199359+0000","last_active":"2024-03-11T21:50:20.413524+0000","last_peered":"2024-03-11T21:50:20.413524+0000","last_clean":"2024-03-11T21:50:20.413524+0000","last_became_active":"2024-03-11T21:48:12.919876+0000","last_became_peered":"2024-03-11T21:48:12.919876+0000","last_unstale":"2024-03-11T21:50:20.413524+0000","last_undegraded":"2024-03-11T21:50:20.413524+0000","last_fullsized":"2024-03-11T21:50:20.413524+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.199135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:12.199135+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.199135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.917003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1955,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872409+0000","last_change":"2024-03-11T21:49:08.512083+0000","last_active":"2024-03-11T21:50:20.872409+0000","last_peered":"2024-03-11T21:50:20.872409+0000","last_clean":"2024-03-11T21:50:20.872409+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:50:20.872409+0000","last_undegraded":"2024-03-11T21:50:20.872409+0000","last_fullsized":"2024-03-11T21:50:20.872409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:08.511840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:08.511840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":1215,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:22.937258+0000","last_change":"2024-03-11T21:50:22.937258+0000","last_active":"2024-03-11T21:50:22.937258+0000","last_peered":"2024-03-11T21:50:22.937258+0000","last_clean":"2024-03-11T21:50:22.937258+0000","last_became_active":"2024-03-11T21:43:58.779902+0000","last_became_peered":"2024-03-11T21:43:58.779902+0000","last_unstale":"2024-03-11T21:50:22.937258+0000","last_undegraded":"2024-03-11T21:50:22.937258+0000","last_fullsized":"2024-03-11T21:50:22.937258+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:50:22.937007+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:50:22.937007+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.937007+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:38.277966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1813,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872604+0000","last_change":"2024-03-11T21:50:15.945937+0000","last_active":"2024-03-11T21:50:20.872604+0000","last_peered":"2024-03-11T21:50:20.872604+0000","last_clean":"2024-03-11T21:50:20.872604+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:50:20.872604+0000","last_undegraded":"2024-03-11T21:50:20.872604+0000","last_fullsized":"2024-03-11T21:50:20.872604+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.945726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:29.232293+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.945726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.917087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"981'14","reported_seq":2317,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.989541+0000","last_change":"2024-03-11T21:50:20.989655+0000","last_active":"2024-03-11T21:50:20.989541+0000","last_peered":"2024-03-11T21:50:20.989541+0000","last_clean":"2024-03-11T21:50:20.989541+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:50:20.989541+0000","last_undegraded":"2024-03-11T21:50:20.989541+0000","last_fullsized":"2024-03-11T21:50:20.989541+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"981'14","last_scrub_stamp":"2024-03-11T21:50:20.989335+0000","last_deep_scrub":"981'14","last_deep_scrub_stamp":"2024-03-11T21:46:22.207673+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.989335+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.477578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":2013,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259943+0000","last_change":"2024-03-11T21:49:41.170382+0000","last_active":"2024-03-11T21:50:20.259943+0000","last_peered":"2024-03-11T21:50:20.259943+0000","last_clean":"2024-03-11T21:50:20.259943+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:50:20.259943+0000","last_undegraded":"2024-03-11T21:50:20.259943+0000","last_fullsized":"2024-03-11T21:50:20.259943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:41.170173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:32.039214+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.170173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.445835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"864'36","reported_seq":2045,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113420+0000","last_change":"2024-03-11T21:49:10.167764+0000","last_active":"2024-03-11T21:50:21.113420+0000","last_peered":"2024-03-11T21:50:21.113420+0000","last_clean":"2024-03-11T21:50:21.113420+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:50:21.113420+0000","last_undegraded":"2024-03-11T21:50:21.113420+0000","last_fullsized":"2024-03-11T21:50:21.113420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'36","last_scrub_stamp":"2024-03-11T21:49:10.167630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:10.167630+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:39.022647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":2027,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114621+0000","last_change":"2024-03-11T21:49:39.553141+0000","last_active":"2024-03-11T21:50:21.114621+0000","last_peered":"2024-03-11T21:50:21.114621+0000","last_clean":"2024-03-11T21:50:21.114621+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:50:21.114621+0000","last_undegraded":"2024-03-11T21:50:21.114621+0000","last_fullsized":"2024-03-11T21:50:21.114621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:49:39.552909+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:49:39.552909+0000","last_clean_scrub_stamp":"2024-03-11T21:49:39.552909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.259353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1965,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876928+0000","last_change":"2024-03-11T21:50:06.965521+0000","last_active":"2024-03-11T21:50:20.876928+0000","last_peered":"2024-03-11T21:50:20.876928+0000","last_clean":"2024-03-11T21:50:20.876928+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:50:20.876928+0000","last_undegraded":"2024-03-11T21:50:20.876928+0000","last_fullsized":"2024-03-11T21:50:20.876928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:50:06.965294+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.965294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.332677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":1237,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261542+0000","last_change":"2024-03-11T21:49:57.105257+0000","last_active":"2024-03-11T21:50:20.261542+0000","last_peered":"2024-03-11T21:50:20.261542+0000","last_clean":"2024-03-11T21:50:20.261542+0000","last_became_active":"2024-03-11T21:43:58.766374+0000","last_became_peered":"2024-03-11T21:43:58.766374+0000","last_unstale":"2024-03-11T21:50:20.261542+0000","last_undegraded":"2024-03-11T21:50:20.261542+0000","last_fullsized":"2024-03-11T21:50:20.261542+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:57.105035+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:32.258396+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.105035+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:04.561243+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1812,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877539+0000","last_change":"2024-03-11T21:49:02.194344+0000","last_active":"2024-03-11T21:50:20.877539+0000","last_peered":"2024-03-11T21:50:20.877539+0000","last_clean":"2024-03-11T21:50:20.877539+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:50:20.877539+0000","last_undegraded":"2024-03-11T21:50:20.877539+0000","last_fullsized":"2024-03-11T21:50:20.877539+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.194207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.194207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":2012,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259405+0000","last_change":"2024-03-11T21:49:52.093172+0000","last_active":"2024-03-11T21:50:20.259405+0000","last_peered":"2024-03-11T21:50:20.259405+0000","last_clean":"2024-03-11T21:50:20.259405+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:50:20.259405+0000","last_undegraded":"2024-03-11T21:50:20.259405+0000","last_fullsized":"2024-03-11T21:50:20.259405+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:52.092945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:52.092945+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.092945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.183187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":3589,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.477963+0000","last_change":"2024-03-11T21:49:20.662025+0000","last_active":"2024-03-11T21:50:21.477963+0000","last_peered":"2024-03-11T21:50:21.477963+0000","last_clean":"2024-03-11T21:50:21.477963+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:50:21.477963+0000","last_undegraded":"2024-03-11T21:50:21.477963+0000","last_fullsized":"2024-03-11T21:50:21.477963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:49:20.661792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.661792+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:29.945840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1617,"num_read_kb":1612,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"864'21","reported_seq":2002,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872778+0000","last_change":"2024-03-11T21:49:24.694334+0000","last_active":"2024-03-11T21:50:20.872778+0000","last_peered":"2024-03-11T21:50:20.872778+0000","last_clean":"2024-03-11T21:50:20.872778+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:50:20.872778+0000","last_undegraded":"2024-03-11T21:50:20.872778+0000","last_fullsized":"2024-03-11T21:50:20.872778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:24.694140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.694140+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.661510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"983'2","reported_seq":1906,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.266163+0000","last_change":"2024-03-11T21:49:51.843063+0000","last_active":"2024-03-11T21:50:20.266163+0000","last_peered":"2024-03-11T21:50:20.266163+0000","last_clean":"2024-03-11T21:50:20.266163+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:50:20.266163+0000","last_undegraded":"2024-03-11T21:50:20.266163+0000","last_fullsized":"2024-03-11T21:50:20.266163+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"983'2","last_scrub_stamp":"2024-03-11T21:49:51.842969+0000","last_deep_scrub":"983'2","last_deep_scrub_stamp":"2024-03-11T21:46:33.835412+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.842969+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.504075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2000,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259747+0000","last_change":"2024-03-11T21:50:00.105619+0000","last_active":"2024-03-11T21:50:20.259747+0000","last_peered":"2024-03-11T21:50:20.259747+0000","last_clean":"2024-03-11T21:50:20.259747+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:50:20.259747+0000","last_undegraded":"2024-03-11T21:50:20.259747+0000","last_fullsized":"2024-03-11T21:50:20.259747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:00.105404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:40.294143+0000","last_clean_scrub_stamp":"2024-03-11T21:50:00.105404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:27.224071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":1237,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114225+0000","last_change":"2024-03-11T21:49:03.530476+0000","last_active":"2024-03-11T21:50:21.114225+0000","last_peered":"2024-03-11T21:50:21.114225+0000","last_clean":"2024-03-11T21:50:21.114225+0000","last_became_active":"2024-03-11T21:43:58.775104+0000","last_became_peered":"2024-03-11T21:43:58.775104+0000","last_unstale":"2024-03-11T21:50:21.114225+0000","last_undegraded":"2024-03-11T21:50:21.114225+0000","last_fullsized":"2024-03-11T21:50:21.114225+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:03.530251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.530251+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:33.150287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1820,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:22.875375+0000","last_change":"2024-03-11T21:50:22.875436+0000","last_active":"2024-03-11T21:50:22.875375+0000","last_peered":"2024-03-11T21:50:22.875375+0000","last_clean":"2024-03-11T21:50:22.875375+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:50:22.875375+0000","last_undegraded":"2024-03-11T21:50:22.875375+0000","last_fullsized":"2024-03-11T21:50:22.875375+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.875194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.875194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.808679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2003,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112783+0000","last_change":"2024-03-11T21:49:50.558097+0000","last_active":"2024-03-11T21:50:21.112783+0000","last_peered":"2024-03-11T21:50:21.112783+0000","last_clean":"2024-03-11T21:50:21.112783+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:50:21.112783+0000","last_undegraded":"2024-03-11T21:50:21.112783+0000","last_fullsized":"2024-03-11T21:50:21.112783+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.557897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:31.689883+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.557897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:05.848781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"864'66","reported_seq":2289,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.607750+0000","last_change":"2024-03-11T21:49:35.036669+0000","last_active":"2024-03-11T21:50:20.607750+0000","last_peered":"2024-03-11T21:50:20.607750+0000","last_clean":"2024-03-11T21:50:20.607750+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:50:20.607750+0000","last_undegraded":"2024-03-11T21:50:20.607750+0000","last_fullsized":"2024-03-11T21:50:20.607750+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'66","last_scrub_stamp":"2024-03-11T21:49:35.036287+0000","last_deep_scrub":"864'66","last_deep_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_clean_scrub_stamp":"2024-03-11T21:49:35.036287+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:38.694468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1967,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878892+0000","last_change":"2024-03-11T21:49:01.178730+0000","last_active":"2024-03-11T21:50:20.878892+0000","last_peered":"2024-03-11T21:50:20.878892+0000","last_clean":"2024-03-11T21:50:20.878892+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:50:20.878892+0000","last_undegraded":"2024-03-11T21:50:20.878892+0000","last_fullsized":"2024-03-11T21:50:20.878892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:01.178360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:01.178360+0000","last_clean_scrub_stamp":"2024-03-11T21:49:01.178360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:23.060738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1845,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.263598+0000","last_change":"2024-03-11T21:49:23.963933+0000","last_active":"2024-03-11T21:50:20.263598+0000","last_peered":"2024-03-11T21:50:20.263598+0000","last_clean":"2024-03-11T21:50:20.263598+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:50:20.263598+0000","last_undegraded":"2024-03-11T21:50:20.263598+0000","last_fullsized":"2024-03-11T21:50:20.263598+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:23.963634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:49:23.963634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.821931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":1232,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.268202+0000","last_change":"2024-03-11T21:49:59.842316+0000","last_active":"2024-03-11T21:50:20.268202+0000","last_peered":"2024-03-11T21:50:20.268202+0000","last_clean":"2024-03-11T21:50:20.268202+0000","last_became_active":"2024-03-11T21:43:58.770127+0000","last_became_peered":"2024-03-11T21:43:58.770127+0000","last_unstale":"2024-03-11T21:50:20.268202+0000","last_undegraded":"2024-03-11T21:50:20.268202+0000","last_fullsized":"2024-03-11T21:50:20.268202+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:59.842082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.842082+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.761687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1831,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113616+0000","last_change":"2024-03-11T21:49:50.290565+0000","last_active":"2024-03-11T21:50:21.113616+0000","last_peered":"2024-03-11T21:50:21.113616+0000","last_clean":"2024-03-11T21:50:21.113616+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:50:21.113616+0000","last_undegraded":"2024-03-11T21:50:21.113616+0000","last_fullsized":"2024-03-11T21:50:21.113616+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.290404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:59.482460+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.290404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:03.873876+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1773,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413561+0000","last_change":"2024-03-11T21:49:24.077836+0000","last_active":"2024-03-11T21:50:20.413561+0000","last_peered":"2024-03-11T21:50:20.413561+0000","last_clean":"2024-03-11T21:50:20.413561+0000","last_became_active":"2024-03-11T21:48:12.932562+0000","last_became_peered":"2024-03-11T21:48:12.932562+0000","last_unstale":"2024-03-11T21:50:20.413561+0000","last_undegraded":"2024-03-11T21:50:20.413561+0000","last_fullsized":"2024-03-11T21:50:20.413561+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:24.077600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.077600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:48.998017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1976,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876097+0000","last_change":"2024-03-11T21:49:03.187461+0000","last_active":"2024-03-11T21:50:20.876097+0000","last_peered":"2024-03-11T21:50:20.876097+0000","last_clean":"2024-03-11T21:50:20.876097+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:50:20.876097+0000","last_undegraded":"2024-03-11T21:50:20.876097+0000","last_fullsized":"2024-03-11T21:50:20.876097+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:03.187356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.187356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"926'49","reported_seq":2053,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112356+0000","last_change":"2024-03-11T21:49:28.217569+0000","last_active":"2024-03-11T21:50:21.112356+0000","last_peered":"2024-03-11T21:50:21.112356+0000","last_clean":"2024-03-11T21:50:21.112356+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:50:21.112356+0000","last_undegraded":"2024-03-11T21:50:21.112356+0000","last_fullsized":"2024-03-11T21:50:21.112356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'49","last_scrub_stamp":"2024-03-11T21:49:28.217321+0000","last_deep_scrub":"926'49","last_deep_scrub_stamp":"2024-03-11T21:46:31.308112+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.217321+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:40.738379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1855,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264286+0000","last_change":"2024-03-11T21:50:03.871116+0000","last_active":"2024-03-11T21:50:20.264286+0000","last_peered":"2024-03-11T21:50:20.264286+0000","last_clean":"2024-03-11T21:50:20.264286+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:50:20.264286+0000","last_undegraded":"2024-03-11T21:50:20.264286+0000","last_fullsized":"2024-03-11T21:50:20.264286+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.870883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.870883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.896740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1813,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871233+0000","last_change":"2024-03-11T21:49:49.688339+0000","last_active":"2024-03-11T21:50:20.871233+0000","last_peered":"2024-03-11T21:50:20.871233+0000","last_clean":"2024-03-11T21:50:20.871233+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:50:20.871233+0000","last_undegraded":"2024-03-11T21:50:20.871233+0000","last_fullsized":"2024-03-11T21:50:20.871233+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:49.688127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:49.688127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.278794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1959,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878209+0000","last_change":"2024-03-11T21:49:43.140282+0000","last_active":"2024-03-11T21:50:20.878209+0000","last_peered":"2024-03-11T21:50:20.878209+0000","last_clean":"2024-03-11T21:50:20.878209+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:50:20.878209+0000","last_undegraded":"2024-03-11T21:50:20.878209+0000","last_fullsized":"2024-03-11T21:50:20.878209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:43.140051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.140051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:51.757216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":2249,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876317+0000","last_change":"2024-03-11T21:48:47.199220+0000","last_active":"2024-03-11T21:50:20.876317+0000","last_peered":"2024-03-11T21:50:20.876317+0000","last_clean":"2024-03-11T21:50:20.876317+0000","last_became_active":"2024-03-11T21:43:58.769784+0000","last_became_peered":"2024-03-11T21:43:58.769784+0000","last_unstale":"2024-03-11T21:50:20.876317+0000","last_undegraded":"2024-03-11T21:50:20.876317+0000","last_fullsized":"2024-03-11T21:50:20.876317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:47.198992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:48:47.198992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1781,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412058+0000","last_change":"2024-03-11T21:50:01.225236+0000","last_active":"2024-03-11T21:50:20.412058+0000","last_peered":"2024-03-11T21:50:20.412058+0000","last_clean":"2024-03-11T21:50:20.412058+0000","last_became_active":"2024-03-11T21:48:12.927553+0000","last_became_peered":"2024-03-11T21:48:12.927553+0000","last_unstale":"2024-03-11T21:50:20.412058+0000","last_undegraded":"2024-03-11T21:50:20.412058+0000","last_fullsized":"2024-03-11T21:50:20.412058+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:01.225015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.182269+0000","last_clean_scrub_stamp":"2024-03-11T21:50:01.225015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.013233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"1007'35","reported_seq":2014,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871615+0000","last_change":"2024-03-11T21:48:42.476536+0000","last_active":"2024-03-11T21:50:20.871615+0000","last_peered":"2024-03-11T21:50:20.871615+0000","last_clean":"2024-03-11T21:50:20.871615+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:50:20.871615+0000","last_undegraded":"2024-03-11T21:50:20.871615+0000","last_fullsized":"2024-03-11T21:50:20.871615+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"1007'35","last_scrub_stamp":"2024-03-11T21:48:42.476181+0000","last_deep_scrub":"1007'35","last_deep_scrub_stamp":"2024-03-11T21:48:42.476181+0000","last_clean_scrub_stamp":"2024-03-11T21:48:42.476181+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":786,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:22.487701+0000","last_change":"2024-03-11T21:50:22.487763+0000","last_active":"2024-03-11T21:50:22.487701+0000","last_peered":"2024-03-11T21:50:22.487701+0000","last_clean":"2024-03-11T21:50:22.487701+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:50:22.487701+0000","last_undegraded":"2024-03-11T21:50:22.487701+0000","last_fullsized":"2024-03-11T21:50:22.487701+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.487542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.487542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.594656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":2257,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877308+0000","last_change":"2024-03-11T21:48:53.224541+0000","last_active":"2024-03-11T21:50:20.877308+0000","last_peered":"2024-03-11T21:50:20.877308+0000","last_clean":"2024-03-11T21:50:20.877308+0000","last_became_active":"2024-03-11T21:43:58.771356+0000","last_became_peered":"2024-03-11T21:43:58.771356+0000","last_unstale":"2024-03-11T21:50:20.877308+0000","last_undegraded":"2024-03-11T21:50:20.877308+0000","last_fullsized":"2024-03-11T21:50:20.877308+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:53.224299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.224299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1627,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411383+0000","last_change":"2024-03-11T21:49:37.011325+0000","last_active":"2024-03-11T21:50:20.411383+0000","last_peered":"2024-03-11T21:50:20.411383+0000","last_clean":"2024-03-11T21:50:20.411383+0000","last_became_active":"2024-03-11T21:48:12.934652+0000","last_became_peered":"2024-03-11T21:48:12.934652+0000","last_unstale":"2024-03-11T21:50:20.411383+0000","last_undegraded":"2024-03-11T21:50:20.411383+0000","last_fullsized":"2024-03-11T21:50:20.411383+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:37.010995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:37.010995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.787561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":2015,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.266579+0000","last_change":"2024-03-11T21:49:53.416963+0000","last_active":"2024-03-11T21:50:20.266579+0000","last_peered":"2024-03-11T21:50:20.266579+0000","last_clean":"2024-03-11T21:50:20.266579+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:50:20.266579+0000","last_undegraded":"2024-03-11T21:50:20.266579+0000","last_fullsized":"2024-03-11T21:50:20.266579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.416759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.416759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.187572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2032,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262199+0000","last_change":"2024-03-11T21:50:11.467696+0000","last_active":"2024-03-11T21:50:20.262199+0000","last_peered":"2024-03-11T21:50:20.262199+0000","last_clean":"2024-03-11T21:50:20.262199+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:50:20.262199+0000","last_undegraded":"2024-03-11T21:50:20.262199+0000","last_fullsized":"2024-03-11T21:50:20.262199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.467429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.467429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.076952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"864'50","reported_seq":1955,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264349+0000","last_change":"2024-03-11T21:49:41.983620+0000","last_active":"2024-03-11T21:50:20.264349+0000","last_peered":"2024-03-11T21:50:20.264349+0000","last_clean":"2024-03-11T21:50:20.264349+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:50:20.264349+0000","last_undegraded":"2024-03-11T21:50:20.264349+0000","last_fullsized":"2024-03-11T21:50:20.264349+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'50","last_scrub_stamp":"2024-03-11T21:49:41.983506+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.983506+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.654046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1955,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878205+0000","last_change":"2024-03-11T21:50:06.000020+0000","last_active":"2024-03-11T21:50:20.878205+0000","last_peered":"2024-03-11T21:50:20.878205+0000","last_clean":"2024-03-11T21:50:20.878205+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:50:20.878205+0000","last_undegraded":"2024-03-11T21:50:20.878205+0000","last_fullsized":"2024-03-11T21:50:20.878205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:05.999714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:05.999714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.385337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":1013,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.410562+0000","last_change":"2024-03-11T21:49:12.187692+0000","last_active":"2024-03-11T21:50:20.410562+0000","last_peered":"2024-03-11T21:50:20.410562+0000","last_clean":"2024-03-11T21:50:20.410562+0000","last_became_active":"2024-03-11T21:48:12.927578+0000","last_became_peered":"2024-03-11T21:48:12.927578+0000","last_unstale":"2024-03-11T21:50:20.410562+0000","last_undegraded":"2024-03-11T21:50:20.410562+0000","last_fullsized":"2024-03-11T21:50:20.410562+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:12.187466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:12.187466+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1831,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.111837+0000","last_change":"2024-03-11T21:49:04.236699+0000","last_active":"2024-03-11T21:50:21.111837+0000","last_peered":"2024-03-11T21:50:21.111837+0000","last_clean":"2024-03-11T21:50:21.111837+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:50:21.111837+0000","last_undegraded":"2024-03-11T21:50:21.111837+0000","last_fullsized":"2024-03-11T21:50:21.111837+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:04.236547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.236547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:32.808872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1988,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114002+0000","last_change":"2024-03-11T21:49:40.179039+0000","last_active":"2024-03-11T21:50:21.114002+0000","last_peered":"2024-03-11T21:50:21.114002+0000","last_clean":"2024-03-11T21:50:21.114002+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:50:21.114002+0000","last_undegraded":"2024-03-11T21:50:21.114002+0000","last_fullsized":"2024-03-11T21:50:21.114002+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:40.178759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:40.178759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:47.369656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"864'47","reported_seq":2074,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.260685+0000","last_change":"2024-03-11T21:49:06.416101+0000","last_active":"2024-03-11T21:50:20.260685+0000","last_peered":"2024-03-11T21:50:20.260685+0000","last_clean":"2024-03-11T21:50:20.260685+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:50:20.260685+0000","last_undegraded":"2024-03-11T21:50:20.260685+0000","last_fullsized":"2024-03-11T21:50:20.260685+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'47","last_scrub_stamp":"2024-03-11T21:49:06.415771+0000","last_deep_scrub":"864'47","last_deep_scrub_stamp":"2024-03-11T21:47:42.204535+0000","last_clean_scrub_stamp":"2024-03-11T21:49:06.415771+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:26.851644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1790,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411915+0000","last_change":"2024-03-11T21:50:16.286535+0000","last_active":"2024-03-11T21:50:20.411915+0000","last_peered":"2024-03-11T21:50:20.411915+0000","last_clean":"2024-03-11T21:50:20.411915+0000","last_became_active":"2024-03-11T21:48:12.932548+0000","last_became_peered":"2024-03-11T21:48:12.932548+0000","last_unstale":"2024-03-11T21:50:20.411915+0000","last_undegraded":"2024-03-11T21:50:20.411915+0000","last_fullsized":"2024-03-11T21:50:20.411915+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.286316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.286316+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.286316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.316124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1955,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872753+0000","last_change":"2024-03-11T21:49:25.715777+0000","last_active":"2024-03-11T21:50:20.872753+0000","last_peered":"2024-03-11T21:50:20.872753+0000","last_clean":"2024-03-11T21:50:20.872753+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:50:20.872753+0000","last_undegraded":"2024-03-11T21:50:20.872753+0000","last_fullsized":"2024-03-11T21:50:20.872753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.715480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.715480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:32.761912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":1245,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114274+0000","last_change":"2024-03-11T21:49:35.643788+0000","last_active":"2024-03-11T21:50:21.114274+0000","last_peered":"2024-03-11T21:50:21.114274+0000","last_clean":"2024-03-11T21:50:21.114274+0000","last_became_active":"2024-03-11T21:43:58.778559+0000","last_became_peered":"2024-03-11T21:43:58.778559+0000","last_unstale":"2024-03-11T21:50:21.114274+0000","last_undegraded":"2024-03-11T21:50:21.114274+0000","last_fullsized":"2024-03-11T21:50:21.114274+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:35.643555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:35.643555+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.984275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1625,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412722+0000","last_change":"2024-03-11T21:49:25.103902+0000","last_active":"2024-03-11T21:50:20.412722+0000","last_peered":"2024-03-11T21:50:20.412722+0000","last_clean":"2024-03-11T21:50:20.412722+0000","last_became_active":"2024-03-11T21:48:12.931225+0000","last_became_peered":"2024-03-11T21:48:12.931225+0000","last_unstale":"2024-03-11T21:50:20.412722+0000","last_undegraded":"2024-03-11T21:50:20.412722+0000","last_fullsized":"2024-03-11T21:50:20.412722+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.103690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:13.261641+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.103690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:50.436849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":2329,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113590+0000","last_change":"2024-03-11T21:49:52.285519+0000","last_active":"2024-03-11T21:50:21.113590+0000","last_peered":"2024-03-11T21:50:21.113590+0000","last_clean":"2024-03-11T21:50:21.113590+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:50:21.113590+0000","last_undegraded":"2024-03-11T21:50:21.113590+0000","last_fullsized":"2024-03-11T21:50:21.113590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:49:52.285328+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:49:52.285328+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.285328+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.928958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1012'3","reported_seq":1883,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.266768+0000","last_change":"2024-03-11T21:49:17.955794+0000","last_active":"2024-03-11T21:50:20.266768+0000","last_peered":"2024-03-11T21:50:20.266768+0000","last_clean":"2024-03-11T21:50:20.266768+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:50:20.266768+0000","last_undegraded":"2024-03-11T21:50:20.266768+0000","last_fullsized":"2024-03-11T21:50:20.266768+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"1012'3","last_scrub_stamp":"2024-03-11T21:49:17.955691+0000","last_deep_scrub":"1012'3","last_deep_scrub_stamp":"2024-03-11T21:49:17.955691+0000","last_clean_scrub_stamp":"2024-03-11T21:49:17.955691+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.546696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"1003'42","reported_seq":2109,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413203+0000","last_change":"2024-03-11T21:49:41.109831+0000","last_active":"2024-03-11T21:50:20.413203+0000","last_peered":"2024-03-11T21:50:20.413203+0000","last_clean":"2024-03-11T21:50:20.413203+0000","last_became_active":"2024-03-11T21:48:12.928283+0000","last_became_peered":"2024-03-11T21:48:12.928283+0000","last_unstale":"2024-03-11T21:50:20.413203+0000","last_undegraded":"2024-03-11T21:50:20.413203+0000","last_fullsized":"2024-03-11T21:50:20.413203+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'42","last_scrub_stamp":"2024-03-11T21:49:41.109586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.109586+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.385002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1853,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.263661+0000","last_change":"2024-03-11T21:49:39.000874+0000","last_active":"2024-03-11T21:50:20.263661+0000","last_peered":"2024-03-11T21:50:20.263661+0000","last_clean":"2024-03-11T21:50:20.263661+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:50:20.263661+0000","last_undegraded":"2024-03-11T21:50:20.263661+0000","last_fullsized":"2024-03-11T21:50:20.263661+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:39.000460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:22.734970+0000","last_clean_scrub_stamp":"2024-03-11T21:49:39.000460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.139821+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1974,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.111594+0000","last_change":"2024-03-11T21:49:15.174558+0000","last_active":"2024-03-11T21:50:21.111594+0000","last_peered":"2024-03-11T21:50:21.111594+0000","last_clean":"2024-03-11T21:50:21.111594+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:50:21.111594+0000","last_undegraded":"2024-03-11T21:50:21.111594+0000","last_fullsized":"2024-03-11T21:50:21.111594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:15.174275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:48.263515+0000","last_clean_scrub_stamp":"2024-03-11T21:49:15.174275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:26.234433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"600'129","reported_seq":1227,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872053+0000","last_change":"2024-03-11T21:50:04.822903+0000","last_active":"2024-03-11T21:50:20.872053+0000","last_peered":"2024-03-11T21:50:20.872053+0000","last_clean":"2024-03-11T21:50:20.872053+0000","last_became_active":"2024-03-11T21:44:03.855000+0000","last_became_peered":"2024-03-11T21:44:03.855000+0000","last_unstale":"2024-03-11T21:50:20.872053+0000","last_undegraded":"2024-03-11T21:50:20.872053+0000","last_fullsized":"2024-03-11T21:50:20.872053+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:50:04.822675+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:48:28.516563+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.822675+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.182142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1714,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264020+0000","last_change":"2024-03-11T21:49:22.929666+0000","last_active":"2024-03-11T21:50:20.264020+0000","last_peered":"2024-03-11T21:50:20.264020+0000","last_clean":"2024-03-11T21:50:20.264020+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:50:20.264020+0000","last_undegraded":"2024-03-11T21:50:20.264020+0000","last_fullsized":"2024-03-11T21:50:20.264020+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:22.929444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:22.929444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:51.054329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2024,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261894+0000","last_change":"2024-03-11T21:50:10.433905+0000","last_active":"2024-03-11T21:50:20.261894+0000","last_peered":"2024-03-11T21:50:20.261894+0000","last_clean":"2024-03-11T21:50:20.261894+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:50:20.261894+0000","last_undegraded":"2024-03-11T21:50:20.261894+0000","last_fullsized":"2024-03-11T21:50:20.261894+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.433796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:32.657602+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.433796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.306140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"864'32","reported_seq":2027,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878567+0000","last_change":"2024-03-11T21:49:52.047520+0000","last_active":"2024-03-11T21:50:20.878567+0000","last_peered":"2024-03-11T21:50:20.878567+0000","last_clean":"2024-03-11T21:50:20.878567+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:50:20.878567+0000","last_undegraded":"2024-03-11T21:50:20.878567+0000","last_fullsized":"2024-03-11T21:50:20.878567+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:49:52.047270+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.047270+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.978245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":2029,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.262791+0000","last_change":"2024-03-11T21:49:42.148026+0000","last_active":"2024-03-11T21:50:20.262791+0000","last_peered":"2024-03-11T21:50:20.262791+0000","last_clean":"2024-03-11T21:50:20.262791+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:50:20.262791+0000","last_undegraded":"2024-03-11T21:50:20.262791+0000","last_fullsized":"2024-03-11T21:50:20.262791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:49:42.147709+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:49:42.147709+0000","last_clean_scrub_stamp":"2024-03-11T21:49:42.147709+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:01.011940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1964,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872096+0000","last_change":"2024-03-11T21:49:03.598478+0000","last_active":"2024-03-11T21:50:20.872096+0000","last_peered":"2024-03-11T21:50:20.872096+0000","last_clean":"2024-03-11T21:50:20.872096+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:50:20.872096+0000","last_undegraded":"2024-03-11T21:50:20.872096+0000","last_fullsized":"2024-03-11T21:50:20.872096+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:03.598257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.598257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"98'128","reported_seq":1230,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.268119+0000","last_change":"2024-03-11T21:48:46.155459+0000","last_active":"2024-03-11T21:50:20.268119+0000","last_peered":"2024-03-11T21:50:20.268119+0000","last_clean":"2024-03-11T21:50:20.268119+0000","last_became_active":"2024-03-11T21:44:03.854936+0000","last_became_peered":"2024-03-11T21:44:03.854936+0000","last_unstale":"2024-03-11T21:50:20.268119+0000","last_undegraded":"2024-03-11T21:50:20.268119+0000","last_fullsized":"2024-03-11T21:50:20.268119+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:46.155226+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:46.155226+0000","last_clean_scrub_stamp":"2024-03-11T21:48:46.155226+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1812,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872252+0000","last_change":"2024-03-11T21:49:04.582875+0000","last_active":"2024-03-11T21:50:20.872252+0000","last_peered":"2024-03-11T21:50:20.872252+0000","last_clean":"2024-03-11T21:50:20.872252+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:50:20.872252+0000","last_undegraded":"2024-03-11T21:50:20.872252+0000","last_fullsized":"2024-03-11T21:50:20.872252+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:04.582668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:50.308128+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.582668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:25.285605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"864'27","reported_seq":2004,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876540+0000","last_change":"2024-03-11T21:49:28.165004+0000","last_active":"2024-03-11T21:50:20.876540+0000","last_peered":"2024-03-11T21:50:20.876540+0000","last_clean":"2024-03-11T21:50:20.876540+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:50:20.876540+0000","last_undegraded":"2024-03-11T21:50:20.876540+0000","last_fullsized":"2024-03-11T21:50:20.876540+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'27","last_scrub_stamp":"2024-03-11T21:49:28.164728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.164728+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:42.889100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2039,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261603+0000","last_change":"2024-03-11T21:50:12.499821+0000","last_active":"2024-03-11T21:50:20.261603+0000","last_peered":"2024-03-11T21:50:20.261603+0000","last_clean":"2024-03-11T21:50:20.261603+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:50:20.261603+0000","last_undegraded":"2024-03-11T21:50:20.261603+0000","last_fullsized":"2024-03-11T21:50:20.261603+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.499608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.499608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.592309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1967,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872934+0000","last_change":"2024-03-11T21:49:23.688584+0000","last_active":"2024-03-11T21:50:20.872934+0000","last_peered":"2024-03-11T21:50:20.872934+0000","last_clean":"2024-03-11T21:50:20.872934+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:50:20.872934+0000","last_undegraded":"2024-03-11T21:50:20.872934+0000","last_fullsized":"2024-03-11T21:50:20.872934+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:23.688354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:10.163069+0000","last_clean_scrub_stamp":"2024-03-11T21:49:23.688354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:28.008708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2018,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.265147+0000","last_change":"2024-03-11T21:49:38.519296+0000","last_active":"2024-03-11T21:50:20.265147+0000","last_peered":"2024-03-11T21:50:20.265147+0000","last_clean":"2024-03-11T21:50:20.265147+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:50:20.265147+0000","last_undegraded":"2024-03-11T21:50:20.265147+0000","last_fullsized":"2024-03-11T21:50:20.265147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:38.519082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:04.588570+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.519082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.967686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"98'128","reported_seq":392,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.450015+0000","last_change":"2024-03-11T21:50:21.450081+0000","last_active":"2024-03-11T21:50:21.450015+0000","last_peered":"2024-03-11T21:50:21.450015+0000","last_clean":"2024-03-11T21:50:21.450015+0000","last_became_active":"2024-03-11T21:48:12.931083+0000","last_became_peered":"2024-03-11T21:48:12.931083+0000","last_unstale":"2024-03-11T21:50:21.450015+0000","last_undegraded":"2024-03-11T21:50:21.450015+0000","last_fullsized":"2024-03-11T21:50:21.450015+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:21.449847+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:50:21.449847+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.449847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.883373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1812,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872051+0000","last_change":"2024-03-11T21:49:50.730394+0000","last_active":"2024-03-11T21:50:20.872051+0000","last_peered":"2024-03-11T21:50:20.872051+0000","last_clean":"2024-03-11T21:50:20.872051+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:50:20.872051+0000","last_undegraded":"2024-03-11T21:50:20.872051+0000","last_fullsized":"2024-03-11T21:50:20.872051+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.730256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:50.730256+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.730256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.092968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"864'14","reported_seq":2033,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.111434+0000","last_change":"2024-03-11T21:50:07.575851+0000","last_active":"2024-03-11T21:50:21.111434+0000","last_peered":"2024-03-11T21:50:21.111434+0000","last_clean":"2024-03-11T21:50:21.111434+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:50:21.111434+0000","last_undegraded":"2024-03-11T21:50:21.111434+0000","last_fullsized":"2024-03-11T21:50:21.111434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'14","last_scrub_stamp":"2024-03-11T21:50:07.575613+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.575613+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:37.545349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":2010,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112635+0000","last_change":"2024-03-11T21:48:58.491688+0000","last_active":"2024-03-11T21:50:21.112635+0000","last_peered":"2024-03-11T21:50:21.112635+0000","last_clean":"2024-03-11T21:50:21.112635+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:50:21.112635+0000","last_undegraded":"2024-03-11T21:50:21.112635+0000","last_fullsized":"2024-03-11T21:50:21.112635+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.491478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.491478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:25.520954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1779,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412919+0000","last_change":"2024-03-11T21:49:47.156418+0000","last_active":"2024-03-11T21:50:20.412919+0000","last_peered":"2024-03-11T21:50:20.412919+0000","last_clean":"2024-03-11T21:50:20.412919+0000","last_became_active":"2024-03-11T21:48:12.934954+0000","last_became_peered":"2024-03-11T21:48:12.934954+0000","last_unstale":"2024-03-11T21:50:20.412919+0000","last_undegraded":"2024-03-11T21:50:20.412919+0000","last_fullsized":"2024-03-11T21:50:20.412919+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.156143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.156143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.148519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1953,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877899+0000","last_change":"2024-03-11T21:49:36.132619+0000","last_active":"2024-03-11T21:50:20.877899+0000","last_peered":"2024-03-11T21:50:20.877899+0000","last_clean":"2024-03-11T21:50:20.877899+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:50:20.877899+0000","last_undegraded":"2024-03-11T21:50:20.877899+0000","last_fullsized":"2024-03-11T21:50:20.877899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:36.132415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:36.132415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.867203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"98'128","reported_seq":615,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.111796+0000","last_change":"2024-03-11T21:49:24.232372+0000","last_active":"2024-03-11T21:50:21.111796+0000","last_peered":"2024-03-11T21:50:21.111796+0000","last_clean":"2024-03-11T21:50:21.111796+0000","last_became_active":"2024-03-11T21:44:03.859125+0000","last_became_peered":"2024-03-11T21:44:03.859125+0000","last_unstale":"2024-03-11T21:50:21.111796+0000","last_undegraded":"2024-03-11T21:50:21.111796+0000","last_fullsized":"2024-03-11T21:50:21.111796+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:24.232118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.232118+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:28.982513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1812,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872047+0000","last_change":"2024-03-11T21:49:47.713717+0000","last_active":"2024-03-11T21:50:20.872047+0000","last_peered":"2024-03-11T21:50:20.872047+0000","last_clean":"2024-03-11T21:50:20.872047+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:50:20.872047+0000","last_undegraded":"2024-03-11T21:50:20.872047+0000","last_fullsized":"2024-03-11T21:50:20.872047+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.713373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.713373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.603247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"864'27","reported_seq":2012,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.947768+0000","last_change":"2024-03-11T21:50:20.947860+0000","last_active":"2024-03-11T21:50:20.947768+0000","last_peered":"2024-03-11T21:50:20.947768+0000","last_clean":"2024-03-11T21:50:20.947768+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:50:20.947768+0000","last_undegraded":"2024-03-11T21:50:20.947768+0000","last_fullsized":"2024-03-11T21:50:20.947768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'27","last_scrub_stamp":"2024-03-11T21:50:20.947606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.947606+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:41.307317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1787,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411831+0000","last_change":"2024-03-11T21:49:49.137593+0000","last_active":"2024-03-11T21:50:20.411831+0000","last_peered":"2024-03-11T21:50:20.411831+0000","last_clean":"2024-03-11T21:50:20.411831+0000","last_became_active":"2024-03-11T21:48:12.919525+0000","last_became_peered":"2024-03-11T21:48:12.919525+0000","last_unstale":"2024-03-11T21:50:20.411831+0000","last_undegraded":"2024-03-11T21:50:20.411831+0000","last_fullsized":"2024-03-11T21:50:20.411831+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:49.137381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:49.137381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:50.130222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1003'15","reported_seq":2056,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413730+0000","last_change":"2024-03-11T21:48:56.275432+0000","last_active":"2024-03-11T21:50:20.413730+0000","last_peered":"2024-03-11T21:50:20.413730+0000","last_clean":"2024-03-11T21:50:20.413730+0000","last_became_active":"2024-03-11T21:48:12.931718+0000","last_became_peered":"2024-03-11T21:48:12.931718+0000","last_unstale":"2024-03-11T21:50:20.413730+0000","last_undegraded":"2024-03-11T21:50:20.413730+0000","last_fullsized":"2024-03-11T21:50:20.413730+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'15","last_scrub_stamp":"2024-03-11T21:48:56.275196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.275196+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:25.373689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1981,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113865+0000","last_change":"2024-03-11T21:50:04.449328+0000","last_active":"2024-03-11T21:50:21.113865+0000","last_peered":"2024-03-11T21:50:21.113865+0000","last_clean":"2024-03-11T21:50:21.113865+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:50:21.113865+0000","last_undegraded":"2024-03-11T21:50:21.113865+0000","last_fullsized":"2024-03-11T21:50:21.113865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:04.449094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.449094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.520679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"98'128","reported_seq":392,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411227+0000","last_change":"2024-03-11T21:50:04.167480+0000","last_active":"2024-03-11T21:50:20.411227+0000","last_peered":"2024-03-11T21:50:20.411227+0000","last_clean":"2024-03-11T21:50:20.411227+0000","last_became_active":"2024-03-11T21:48:12.918001+0000","last_became_peered":"2024-03-11T21:48:12.918001+0000","last_unstale":"2024-03-11T21:50:20.411227+0000","last_undegraded":"2024-03-11T21:50:20.411227+0000","last_fullsized":"2024-03-11T21:50:20.411227+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:04.167215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.167215+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.453687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1861,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112021+0000","last_change":"2024-03-11T21:50:13.470906+0000","last_active":"2024-03-11T21:50:21.112021+0000","last_peered":"2024-03-11T21:50:21.112021+0000","last_clean":"2024-03-11T21:50:21.112021+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:50:21.112021+0000","last_undegraded":"2024-03-11T21:50:21.112021+0000","last_fullsized":"2024-03-11T21:50:21.112021+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.470698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.470698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.566505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"864'25","reported_seq":2030,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112209+0000","last_change":"2024-03-11T21:50:10.417964+0000","last_active":"2024-03-11T21:50:21.112209+0000","last_peered":"2024-03-11T21:50:21.112209+0000","last_clean":"2024-03-11T21:50:21.112209+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:50:21.112209+0000","last_undegraded":"2024-03-11T21:50:21.112209+0000","last_fullsized":"2024-03-11T21:50:21.112209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:50:10.417693+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.417693+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.973121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1985,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113099+0000","last_change":"2024-03-11T21:49:12.155651+0000","last_active":"2024-03-11T21:50:21.113099+0000","last_peered":"2024-03-11T21:50:21.113099+0000","last_clean":"2024-03-11T21:50:21.113099+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:50:21.113099+0000","last_undegraded":"2024-03-11T21:50:21.113099+0000","last_fullsized":"2024-03-11T21:50:21.113099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:12.155438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:12.155438+0000","last_clean_scrub_stamp":"2024-03-11T21:49:12.155438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.870413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":2297,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872339+0000","last_change":"2024-03-11T21:49:00.556140+0000","last_active":"2024-03-11T21:50:20.872339+0000","last_peered":"2024-03-11T21:50:20.872339+0000","last_clean":"2024-03-11T21:50:20.872339+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:50:20.872339+0000","last_undegraded":"2024-03-11T21:50:20.872339+0000","last_fullsized":"2024-03-11T21:50:20.872339+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:49:00.555818+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:49:00.555818+0000","last_clean_scrub_stamp":"2024-03-11T21:49:00.555818+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1998,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.263292+0000","last_change":"2024-03-11T21:49:05.442785+0000","last_active":"2024-03-11T21:50:20.263292+0000","last_peered":"2024-03-11T21:50:20.263292+0000","last_clean":"2024-03-11T21:50:20.263292+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:50:20.263292+0000","last_undegraded":"2024-03-11T21:50:20.263292+0000","last_fullsized":"2024-03-11T21:50:20.263292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:05.442133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:05.442133+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.442133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.452987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1702,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.268425+0000","last_change":"2024-03-11T21:49:14.028737+0000","last_active":"2024-03-11T21:50:20.268425+0000","last_peered":"2024-03-11T21:50:20.268425+0000","last_clean":"2024-03-11T21:50:20.268425+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:50:20.268425+0000","last_undegraded":"2024-03-11T21:50:20.268425+0000","last_fullsized":"2024-03-11T21:50:20.268425+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:14.028514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:14.028514+0000","last_clean_scrub_stamp":"2024-03-11T21:49:14.028514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.796956+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"864'25","reported_seq":1999,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.879506+0000","last_change":"2024-03-11T21:49:31.172448+0000","last_active":"2024-03-11T21:50:20.879506+0000","last_peered":"2024-03-11T21:50:20.879506+0000","last_clean":"2024-03-11T21:50:20.879506+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:50:20.879506+0000","last_undegraded":"2024-03-11T21:50:20.879506+0000","last_fullsized":"2024-03-11T21:50:20.879506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:49:31.172202+0000","last_deep_scrub":"864'25","last_deep_scrub_stamp":"2024-03-11T21:49:31.172202+0000","last_clean_scrub_stamp":"2024-03-11T21:49:31.172202+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.148714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2017,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113701+0000","last_change":"2024-03-11T21:49:54.521581+0000","last_active":"2024-03-11T21:50:21.113701+0000","last_peered":"2024-03-11T21:50:21.113701+0000","last_clean":"2024-03-11T21:50:21.113701+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:50:21.113701+0000","last_undegraded":"2024-03-11T21:50:21.113701+0000","last_fullsized":"2024-03-11T21:50:21.113701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.521369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.521369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:58.972862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1967,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871919+0000","last_change":"2024-03-11T21:49:19.624801+0000","last_active":"2024-03-11T21:50:20.871919+0000","last_peered":"2024-03-11T21:50:20.871919+0000","last_clean":"2024-03-11T21:50:20.871919+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:50:20.871919+0000","last_undegraded":"2024-03-11T21:50:20.871919+0000","last_fullsized":"2024-03-11T21:50:20.871919+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:19.624584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:32.231996+0000","last_clean_scrub_stamp":"2024-03-11T21:49:19.624584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:28.312093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1989,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.111504+0000","last_change":"2024-03-11T21:50:12.415639+0000","last_active":"2024-03-11T21:50:21.111504+0000","last_peered":"2024-03-11T21:50:21.111504+0000","last_clean":"2024-03-11T21:50:21.111504+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:50:21.111504+0000","last_undegraded":"2024-03-11T21:50:21.111504+0000","last_fullsized":"2024-03-11T21:50:21.111504+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.415410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:42.031080+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.415410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.843940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1838,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113245+0000","last_change":"2024-03-11T21:50:05.452386+0000","last_active":"2024-03-11T21:50:21.113245+0000","last_peered":"2024-03-11T21:50:21.113245+0000","last_clean":"2024-03-11T21:50:21.113245+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:50:21.113245+0000","last_undegraded":"2024-03-11T21:50:21.113245+0000","last_fullsized":"2024-03-11T21:50:21.113245+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:05.452156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:05.452156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.486527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"864'30","reported_seq":2059,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.260425+0000","last_change":"2024-03-11T21:49:38.165791+0000","last_active":"2024-03-11T21:50:20.260425+0000","last_peered":"2024-03-11T21:50:20.260425+0000","last_clean":"2024-03-11T21:50:20.260425+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:50:20.260425+0000","last_undegraded":"2024-03-11T21:50:20.260425+0000","last_fullsized":"2024-03-11T21:50:20.260425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:49:38.165541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.165541+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.030299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1895,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.265351+0000","last_change":"2024-03-11T21:49:37.072247+0000","last_active":"2024-03-11T21:50:20.265351+0000","last_peered":"2024-03-11T21:50:20.265351+0000","last_clean":"2024-03-11T21:50:20.265351+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:50:20.265351+0000","last_undegraded":"2024-03-11T21:50:20.265351+0000","last_fullsized":"2024-03-11T21:50:20.265351+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:37.072152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:49:37.072152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:00.353230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1864,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.266371+0000","last_change":"2024-03-11T21:49:27.009907+0000","last_active":"2024-03-11T21:50:20.266371+0000","last_peered":"2024-03-11T21:50:20.266371+0000","last_clean":"2024-03-11T21:50:20.266371+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:50:20.266371+0000","last_undegraded":"2024-03-11T21:50:20.266371+0000","last_fullsized":"2024-03-11T21:50:20.266371+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:27.009691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:27.009691+0000","last_clean_scrub_stamp":"2024-03-11T21:49:27.009691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.867332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1954,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871541+0000","last_change":"2024-03-11T21:48:52.444873+0000","last_active":"2024-03-11T21:50:20.871541+0000","last_peered":"2024-03-11T21:50:20.871541+0000","last_clean":"2024-03-11T21:50:20.871541+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:50:20.871541+0000","last_undegraded":"2024-03-11T21:50:20.871541+0000","last_fullsized":"2024-03-11T21:50:20.871541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:52.444724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:26.312862+0000","last_clean_scrub_stamp":"2024-03-11T21:48:52.444724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1828,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112936+0000","last_change":"2024-03-11T21:49:05.210295+0000","last_active":"2024-03-11T21:50:21.112936+0000","last_peered":"2024-03-11T21:50:21.112936+0000","last_clean":"2024-03-11T21:50:21.112936+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:50:21.112936+0000","last_undegraded":"2024-03-11T21:50:21.112936+0000","last_fullsized":"2024-03-11T21:50:21.112936+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:05.210046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:38.414635+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.210046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:27.919621+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"864'18","reported_seq":2016,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114016+0000","last_change":"2024-03-11T21:49:06.177306+0000","last_active":"2024-03-11T21:50:21.114016+0000","last_peered":"2024-03-11T21:50:21.114016+0000","last_clean":"2024-03-11T21:50:21.114016+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:50:21.114016+0000","last_undegraded":"2024-03-11T21:50:21.114016+0000","last_fullsized":"2024-03-11T21:50:21.114016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'18","last_scrub_stamp":"2024-03-11T21:49:06.177079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:06.177079+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.723017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1778,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413378+0000","last_change":"2024-03-11T21:49:53.202771+0000","last_active":"2024-03-11T21:50:20.413378+0000","last_peered":"2024-03-11T21:50:20.413378+0000","last_clean":"2024-03-11T21:50:20.413378+0000","last_became_active":"2024-03-11T21:48:12.935533+0000","last_became_peered":"2024-03-11T21:48:12.935533+0000","last_unstale":"2024-03-11T21:50:20.413378+0000","last_undegraded":"2024-03-11T21:50:20.413378+0000","last_fullsized":"2024-03-11T21:50:20.413378+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.202493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.202493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.004431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2009,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112463+0000","last_change":"2024-03-11T21:50:01.519326+0000","last_active":"2024-03-11T21:50:21.112463+0000","last_peered":"2024-03-11T21:50:21.112463+0000","last_clean":"2024-03-11T21:50:21.112463+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:50:21.112463+0000","last_undegraded":"2024-03-11T21:50:21.112463+0000","last_fullsized":"2024-03-11T21:50:21.112463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:01.519026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:01.519026+0000","last_clean_scrub_stamp":"2024-03-11T21:50:01.519026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.326962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1953,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876950+0000","last_change":"2024-03-11T21:49:24.177802+0000","last_active":"2024-03-11T21:50:20.876950+0000","last_peered":"2024-03-11T21:50:20.876950+0000","last_clean":"2024-03-11T21:50:20.876950+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:50:20.876950+0000","last_undegraded":"2024-03-11T21:50:20.876950+0000","last_fullsized":"2024-03-11T21:50:20.876950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:24.177578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.177578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:25.277199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"600'129","reported_seq":1224,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261829+0000","last_change":"2024-03-11T21:49:43.454964+0000","last_active":"2024-03-11T21:50:20.261829+0000","last_peered":"2024-03-11T21:50:20.261829+0000","last_clean":"2024-03-11T21:50:20.261829+0000","last_became_active":"2024-03-11T21:44:03.841587+0000","last_became_peered":"2024-03-11T21:44:03.841587+0000","last_unstale":"2024-03-11T21:50:20.261829+0000","last_undegraded":"2024-03-11T21:50:20.261829+0000","last_fullsized":"2024-03-11T21:50:20.261829+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:49:43.454749+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:47:03.572458+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.454749+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.687423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1839,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113162+0000","last_change":"2024-03-11T21:50:19.461141+0000","last_active":"2024-03-11T21:50:21.113162+0000","last_peered":"2024-03-11T21:50:21.113162+0000","last_clean":"2024-03-11T21:50:21.113162+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:50:21.113162+0000","last_undegraded":"2024-03-11T21:50:21.113162+0000","last_fullsized":"2024-03-11T21:50:21.113162+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.460934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.460934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:28.244404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"1012'25","reported_seq":1813,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412822+0000","last_change":"2024-03-11T21:49:30.095706+0000","last_active":"2024-03-11T21:50:20.412822+0000","last_peered":"2024-03-11T21:50:20.412822+0000","last_clean":"2024-03-11T21:50:20.412822+0000","last_became_active":"2024-03-11T21:48:12.933804+0000","last_became_peered":"2024-03-11T21:48:12.933804+0000","last_unstale":"2024-03-11T21:50:20.412822+0000","last_undegraded":"2024-03-11T21:50:20.412822+0000","last_fullsized":"2024-03-11T21:50:20.412822+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1012'25","last_scrub_stamp":"2024-03-11T21:49:30.095454+0000","last_deep_scrub":"1012'25","last_deep_scrub_stamp":"2024-03-11T21:49:30.095454+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.095454+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:30.777865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2017,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114460+0000","last_change":"2024-03-11T21:49:29.700554+0000","last_active":"2024-03-11T21:50:21.114460+0000","last_peered":"2024-03-11T21:50:21.114460+0000","last_clean":"2024-03-11T21:50:21.114460+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:50:21.114460+0000","last_undegraded":"2024-03-11T21:50:21.114460+0000","last_fullsized":"2024-03-11T21:50:21.114460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:29.700259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:15.606625+0000","last_clean_scrub_stamp":"2024-03-11T21:49:29.700259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.122700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1957,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876701+0000","last_change":"2024-03-11T21:49:47.091364+0000","last_active":"2024-03-11T21:50:20.876701+0000","last_peered":"2024-03-11T21:50:20.876701+0000","last_clean":"2024-03-11T21:50:20.876701+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:50:20.876701+0000","last_undegraded":"2024-03-11T21:50:20.876701+0000","last_fullsized":"2024-03-11T21:50:20.876701+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.091151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.091151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:48.902126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1954,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872164+0000","last_change":"2024-03-11T21:49:21.672598+0000","last_active":"2024-03-11T21:50:20.872164+0000","last_peered":"2024-03-11T21:50:20.872164+0000","last_clean":"2024-03-11T21:50:20.872164+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:50:20.872164+0000","last_undegraded":"2024-03-11T21:50:20.872164+0000","last_fullsized":"2024-03-11T21:50:20.872164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:21.672335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:49:21.672335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:45.666488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1811,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877087+0000","last_change":"2024-03-11T21:49:25.144198+0000","last_active":"2024-03-11T21:50:20.877087+0000","last_peered":"2024-03-11T21:50:20.877087+0000","last_clean":"2024-03-11T21:50:20.877087+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:50:20.877087+0000","last_undegraded":"2024-03-11T21:50:20.877087+0000","last_fullsized":"2024-03-11T21:50:20.877087+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.143977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.143977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:49.482180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"864'46","reported_seq":1974,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:22.842094+0000","last_change":"2024-03-11T21:50:22.842183+0000","last_active":"2024-03-11T21:50:22.842094+0000","last_peered":"2024-03-11T21:50:22.842094+0000","last_clean":"2024-03-11T21:50:22.842094+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:50:22.842094+0000","last_undegraded":"2024-03-11T21:50:22.842094+0000","last_fullsized":"2024-03-11T21:50:22.842094+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'46","last_scrub_stamp":"2024-03-11T21:50:22.841896+0000","last_deep_scrub":"864'46","last_deep_scrub_stamp":"2024-03-11T21:49:08.052460+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.841896+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:38.474707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1784,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.410762+0000","last_change":"2024-03-11T21:50:18.365330+0000","last_active":"2024-03-11T21:50:20.410762+0000","last_peered":"2024-03-11T21:50:20.410762+0000","last_clean":"2024-03-11T21:50:20.410762+0000","last_became_active":"2024-03-11T21:48:12.900431+0000","last_became_peered":"2024-03-11T21:48:12.900431+0000","last_unstale":"2024-03-11T21:50:20.410762+0000","last_undegraded":"2024-03-11T21:50:20.410762+0000","last_fullsized":"2024-03-11T21:50:20.410762+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:18.365111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.365111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.139089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"988'12","reported_seq":2353,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114633+0000","last_change":"2024-03-11T21:50:09.497019+0000","last_active":"2024-03-11T21:50:21.114633+0000","last_peered":"2024-03-11T21:50:21.114633+0000","last_clean":"2024-03-11T21:50:21.114633+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:50:21.114633+0000","last_undegraded":"2024-03-11T21:50:21.114633+0000","last_fullsized":"2024-03-11T21:50:21.114633+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"988'12","last_scrub_stamp":"2024-03-11T21:50:09.496750+0000","last_deep_scrub":"988'12","last_deep_scrub_stamp":"2024-03-11T21:50:09.496750+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.496750+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.858411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"984'2","reported_seq":1955,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.878494+0000","last_change":"2024-03-11T21:50:17.943138+0000","last_active":"2024-03-11T21:50:20.878494+0000","last_peered":"2024-03-11T21:50:20.878494+0000","last_clean":"2024-03-11T21:50:20.878494+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:50:20.878494+0000","last_undegraded":"2024-03-11T21:50:20.878494+0000","last_fullsized":"2024-03-11T21:50:20.878494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"984'2","last_scrub_stamp":"2024-03-11T21:50:17.942909+0000","last_deep_scrub":"984'2","last_deep_scrub_stamp":"2024-03-11T21:48:21.143821+0000","last_clean_scrub_stamp":"2024-03-11T21:50:17.942909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.361236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1617,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411394+0000","last_change":"2024-03-11T21:48:53.296084+0000","last_active":"2024-03-11T21:50:20.411394+0000","last_peered":"2024-03-11T21:50:20.411394+0000","last_clean":"2024-03-11T21:50:20.411394+0000","last_became_active":"2024-03-11T21:48:12.931163+0000","last_became_peered":"2024-03-11T21:48:12.931163+0000","last_unstale":"2024-03-11T21:50:20.411394+0000","last_undegraded":"2024-03-11T21:50:20.411394+0000","last_fullsized":"2024-03-11T21:50:20.411394+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:53.295872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:08.958336+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.295872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"864'21","reported_seq":2013,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871284+0000","last_change":"2024-03-11T21:49:51.760336+0000","last_active":"2024-03-11T21:50:20.871284+0000","last_peered":"2024-03-11T21:50:20.871284+0000","last_clean":"2024-03-11T21:50:20.871284+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:50:20.871284+0000","last_undegraded":"2024-03-11T21:50:20.871284+0000","last_fullsized":"2024-03-11T21:50:20.871284+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:51.760049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.760049+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.173839+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1975,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871621+0000","last_change":"2024-03-11T21:50:12.934889+0000","last_active":"2024-03-11T21:50:20.871621+0000","last_peered":"2024-03-11T21:50:20.871621+0000","last_clean":"2024-03-11T21:50:20.871621+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:50:20.871621+0000","last_undegraded":"2024-03-11T21:50:20.871621+0000","last_fullsized":"2024-03-11T21:50:20.871621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:50:12.934798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.934798+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.758894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1979,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113747+0000","last_change":"2024-03-11T21:49:22.222504+0000","last_active":"2024-03-11T21:50:21.113747+0000","last_peered":"2024-03-11T21:50:21.113747+0000","last_clean":"2024-03-11T21:50:21.113747+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:50:21.113747+0000","last_undegraded":"2024-03-11T21:50:21.113747+0000","last_fullsized":"2024-03-11T21:50:21.113747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:22.222268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:49:22.222268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:33.507658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1854,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264955+0000","last_change":"2024-03-11T21:50:10.922696+0000","last_active":"2024-03-11T21:50:20.264955+0000","last_peered":"2024-03-11T21:50:20.264955+0000","last_clean":"2024-03-11T21:50:20.264955+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:50:20.264955+0000","last_undegraded":"2024-03-11T21:50:20.264955+0000","last_fullsized":"2024-03-11T21:50:20.264955+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.922475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:10.922475+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.922475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.052341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1811,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876138+0000","last_change":"2024-03-11T21:50:18.945863+0000","last_active":"2024-03-11T21:50:20.876138+0000","last_peered":"2024-03-11T21:50:20.876138+0000","last_clean":"2024-03-11T21:50:20.876138+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:50:20.876138+0000","last_undegraded":"2024-03-11T21:50:20.876138+0000","last_fullsized":"2024-03-11T21:50:20.876138+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:18.945645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.945645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.552804+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"864'11","reported_seq":1998,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.113163+0000","last_change":"2024-03-11T21:49:18.187721+0000","last_active":"2024-03-11T21:50:21.113163+0000","last_peered":"2024-03-11T21:50:21.113163+0000","last_clean":"2024-03-11T21:50:21.113163+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:50:21.113163+0000","last_undegraded":"2024-03-11T21:50:21.113163+0000","last_fullsized":"2024-03-11T21:50:21.113163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'11","last_scrub_stamp":"2024-03-11T21:49:18.187423+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:49:18.187423+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:35.358056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2016,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.112577+0000","last_change":"2024-03-11T21:50:11.492045+0000","last_active":"2024-03-11T21:50:21.112577+0000","last_peered":"2024-03-11T21:50:21.112577+0000","last_clean":"2024-03-11T21:50:21.112577+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:50:21.112577+0000","last_undegraded":"2024-03-11T21:50:21.112577+0000","last_fullsized":"2024-03-11T21:50:21.112577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.491841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.491841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.017126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1848,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.265822+0000","last_change":"2024-03-11T21:49:56.831339+0000","last_active":"2024-03-11T21:50:20.265822+0000","last_peered":"2024-03-11T21:50:20.265822+0000","last_clean":"2024-03-11T21:50:20.265822+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:50:20.265822+0000","last_undegraded":"2024-03-11T21:50:20.265822+0000","last_fullsized":"2024-03-11T21:50:20.265822+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:56.831173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:25.102956+0000","last_clean_scrub_stamp":"2024-03-11T21:49:56.831173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.886668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2000,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:23.163050+0000","last_change":"2024-03-11T21:50:23.163107+0000","last_active":"2024-03-11T21:50:23.163050+0000","last_peered":"2024-03-11T21:50:23.163050+0000","last_clean":"2024-03-11T21:50:23.163050+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:50:23.163050+0000","last_undegraded":"2024-03-11T21:50:23.163050+0000","last_fullsized":"2024-03-11T21:50:23.163050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:23.162896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:07.000334+0000","last_clean_scrub_stamp":"2024-03-11T21:50:23.162896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.117681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1854,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:22.160300+0000","last_change":"2024-03-11T21:50:22.160399+0000","last_active":"2024-03-11T21:50:22.160300+0000","last_peered":"2024-03-11T21:50:22.160300+0000","last_clean":"2024-03-11T21:50:22.160300+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:50:22.160300+0000","last_undegraded":"2024-03-11T21:50:22.160300+0000","last_fullsized":"2024-03-11T21:50:22.160300+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.160112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:44.265056+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.160112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:46.333624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"864'25","reported_seq":2052,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259906+0000","last_change":"2024-03-11T21:49:45.142338+0000","last_active":"2024-03-11T21:50:20.259906+0000","last_peered":"2024-03-11T21:50:20.259906+0000","last_clean":"2024-03-11T21:50:20.259906+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:50:20.259906+0000","last_undegraded":"2024-03-11T21:50:20.259906+0000","last_fullsized":"2024-03-11T21:50:20.259906+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:49:45.142100+0000","last_deep_scrub":"864'25","last_deep_scrub_stamp":"2024-03-11T21:49:45.142100+0000","last_clean_scrub_stamp":"2024-03-11T21:49:45.142100+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.214752+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1973,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871833+0000","last_change":"2024-03-11T21:49:44.674801+0000","last_active":"2024-03-11T21:50:20.871833+0000","last_peered":"2024-03-11T21:50:20.871833+0000","last_clean":"2024-03-11T21:50:20.871833+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:50:20.871833+0000","last_undegraded":"2024-03-11T21:50:20.871833+0000","last_fullsized":"2024-03-11T21:50:20.871833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:44.674705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:49:44.674705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:59.441784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1993,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.260983+0000","last_change":"2024-03-11T21:48:56.457826+0000","last_active":"2024-03-11T21:50:20.260983+0000","last_peered":"2024-03-11T21:50:20.260983+0000","last_clean":"2024-03-11T21:50:20.260983+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:50:20.260983+0000","last_undegraded":"2024-03-11T21:50:20.260983+0000","last_fullsized":"2024-03-11T21:50:20.260983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:56.457609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:56.457609+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.457609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1956,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872862+0000","last_change":"2024-03-11T21:49:38.770943+0000","last_active":"2024-03-11T21:50:20.872862+0000","last_peered":"2024-03-11T21:50:20.872862+0000","last_clean":"2024-03-11T21:50:20.872862+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:50:20.872862+0000","last_undegraded":"2024-03-11T21:50:20.872862+0000","last_fullsized":"2024-03-11T21:50:20.872862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:38.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:42.048814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1811,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.879317+0000","last_change":"2024-03-11T21:48:57.200335+0000","last_active":"2024-03-11T21:50:20.879317+0000","last_peered":"2024-03-11T21:50:20.879317+0000","last_clean":"2024-03-11T21:50:20.879317+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:50:20.879317+0000","last_undegraded":"2024-03-11T21:50:20.879317+0000","last_fullsized":"2024-03-11T21:50:20.879317+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.200119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.200119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"996'29","reported_seq":1929,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.413585+0000","last_change":"2024-03-11T21:49:00.267808+0000","last_active":"2024-03-11T21:50:20.413585+0000","last_peered":"2024-03-11T21:50:20.413585+0000","last_clean":"2024-03-11T21:50:20.413585+0000","last_became_active":"2024-03-11T21:48:12.920561+0000","last_became_peered":"2024-03-11T21:48:12.920561+0000","last_unstale":"2024-03-11T21:50:20.413585+0000","last_undegraded":"2024-03-11T21:50:20.413585+0000","last_fullsized":"2024-03-11T21:50:20.413585+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'29","last_scrub_stamp":"2024-03-11T21:49:00.267541+0000","last_deep_scrub":"996'29","last_deep_scrub_stamp":"2024-03-11T21:47:38.135324+0000","last_clean_scrub_stamp":"2024-03-11T21:49:00.267541+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:22.257314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1964,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877338+0000","last_change":"2024-03-11T21:48:46.164103+0000","last_active":"2024-03-11T21:50:20.877338+0000","last_peered":"2024-03-11T21:50:20.877338+0000","last_clean":"2024-03-11T21:50:20.877338+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:50:20.877338+0000","last_undegraded":"2024-03-11T21:50:20.877338+0000","last_fullsized":"2024-03-11T21:50:20.877338+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:46.163955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:48:46.163955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1762,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411765+0000","last_change":"2024-03-11T21:50:15.240821+0000","last_active":"2024-03-11T21:50:20.411765+0000","last_peered":"2024-03-11T21:50:20.411765+0000","last_clean":"2024-03-11T21:50:20.411765+0000","last_became_active":"2024-03-11T21:48:12.918435+0000","last_became_peered":"2024-03-11T21:48:12.918435+0000","last_unstale":"2024-03-11T21:50:20.411765+0000","last_undegraded":"2024-03-11T21:50:20.411765+0000","last_fullsized":"2024-03-11T21:50:20.411765+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.240520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:22.542107+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.240520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.552905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1989,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.480279+0000","last_change":"2024-03-11T21:50:21.480347+0000","last_active":"2024-03-11T21:50:21.480279+0000","last_peered":"2024-03-11T21:50:21.480279+0000","last_clean":"2024-03-11T21:50:21.480279+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:50:21.480279+0000","last_undegraded":"2024-03-11T21:50:21.480279+0000","last_fullsized":"2024-03-11T21:50:21.480279+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.480119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.480119+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.480119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.475906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1856,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.261477+0000","last_change":"2024-03-11T21:49:26.248908+0000","last_active":"2024-03-11T21:50:20.261477+0000","last_peered":"2024-03-11T21:50:20.261477+0000","last_clean":"2024-03-11T21:50:20.261477+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:50:20.261477+0000","last_undegraded":"2024-03-11T21:50:20.261477+0000","last_fullsized":"2024-03-11T21:50:20.261477+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:26.248678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:26.248678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.888306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"864'46","reported_seq":1851,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.412996+0000","last_change":"2024-03-11T21:49:51.211205+0000","last_active":"2024-03-11T21:50:20.412996+0000","last_peered":"2024-03-11T21:50:20.412996+0000","last_clean":"2024-03-11T21:50:20.412996+0000","last_became_active":"2024-03-11T21:48:12.924933+0000","last_became_peered":"2024-03-11T21:48:12.924933+0000","last_unstale":"2024-03-11T21:50:20.412996+0000","last_undegraded":"2024-03-11T21:50:20.412996+0000","last_fullsized":"2024-03-11T21:50:20.412996+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'46","last_scrub_stamp":"2024-03-11T21:49:51.210966+0000","last_deep_scrub":"864'46","last_deep_scrub_stamp":"2024-03-11T21:46:07.956383+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.210966+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:08.301652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1975,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.879428+0000","last_change":"2024-03-11T21:50:02.956633+0000","last_active":"2024-03-11T21:50:20.879428+0000","last_peered":"2024-03-11T21:50:20.879428+0000","last_clean":"2024-03-11T21:50:20.879428+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:50:20.879428+0000","last_undegraded":"2024-03-11T21:50:20.879428+0000","last_fullsized":"2024-03-11T21:50:20.879428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:02.956333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.956333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.561903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1957,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.876626+0000","last_change":"2024-03-11T21:48:33.109111+0000","last_active":"2024-03-11T21:50:20.876626+0000","last_peered":"2024-03-11T21:50:20.876626+0000","last_clean":"2024-03-11T21:50:20.876626+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:50:20.876626+0000","last_undegraded":"2024-03-11T21:50:20.876626+0000","last_fullsized":"2024-03-11T21:50:20.876626+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:33.108893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:48:33.108893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1759,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.411315+0000","last_change":"2024-03-11T21:49:07.274484+0000","last_active":"2024-03-11T21:50:20.411315+0000","last_peered":"2024-03-11T21:50:20.411315+0000","last_clean":"2024-03-11T21:50:20.411315+0000","last_became_active":"2024-03-11T21:48:12.924446+0000","last_became_peered":"2024-03-11T21:48:12.924446+0000","last_unstale":"2024-03-11T21:50:20.411315+0000","last_undegraded":"2024-03-11T21:50:20.411315+0000","last_fullsized":"2024-03-11T21:50:20.411315+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:07.274264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:07.274264+0000","last_clean_scrub_stamp":"2024-03-11T21:49:07.274264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.945725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1819,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.877458+0000","last_change":"2024-03-11T21:50:03.976125+0000","last_active":"2024-03-11T21:50:20.877458+0000","last_peered":"2024-03-11T21:50:20.877458+0000","last_clean":"2024-03-11T21:50:20.877458+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:50:20.877458+0000","last_undegraded":"2024-03-11T21:50:20.877458+0000","last_fullsized":"2024-03-11T21:50:20.877458+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.975820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:22.179791+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.975820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.994378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"864'32","reported_seq":2039,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114263+0000","last_change":"2024-03-11T21:49:27.251633+0000","last_active":"2024-03-11T21:50:21.114263+0000","last_peered":"2024-03-11T21:50:21.114263+0000","last_clean":"2024-03-11T21:50:21.114263+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:50:21.114263+0000","last_undegraded":"2024-03-11T21:50:21.114263+0000","last_fullsized":"2024-03-11T21:50:21.114263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:49:27.251246+0000","last_deep_scrub":"864'32","last_deep_scrub_stamp":"2024-03-11T21:47:49.771860+0000","last_clean_scrub_stamp":"2024-03-11T21:49:27.251246+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:53.373735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1968,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872721+0000","last_change":"2024-03-11T21:49:13.528524+0000","last_active":"2024-03-11T21:50:20.872721+0000","last_peered":"2024-03-11T21:50:20.872721+0000","last_clean":"2024-03-11T21:50:20.872721+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:50:20.872721+0000","last_undegraded":"2024-03-11T21:50:20.872721+0000","last_fullsized":"2024-03-11T21:50:20.872721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:13.528313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:13.528313+0000","last_clean_scrub_stamp":"2024-03-11T21:49:13.528313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:29.843580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1844,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.267758+0000","last_change":"2024-03-11T21:49:32.006309+0000","last_active":"2024-03-11T21:50:20.267758+0000","last_peered":"2024-03-11T21:50:20.267758+0000","last_clean":"2024-03-11T21:50:20.267758+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:50:20.267758+0000","last_undegraded":"2024-03-11T21:50:20.267758+0000","last_fullsized":"2024-03-11T21:50:20.267758+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.006162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.006162+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.006162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:49.239412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2007,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.259554+0000","last_change":"2024-03-11T21:50:19.199264+0000","last_active":"2024-03-11T21:50:20.259554+0000","last_peered":"2024-03-11T21:50:20.259554+0000","last_clean":"2024-03-11T21:50:20.259554+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:50:20.259554+0000","last_undegraded":"2024-03-11T21:50:20.259554+0000","last_fullsized":"2024-03-11T21:50:20.259554+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.199036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:19.199036+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.199036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.922229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1703,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.264730+0000","last_change":"2024-03-11T21:49:53.805510+0000","last_active":"2024-03-11T21:50:20.264730+0000","last_peered":"2024-03-11T21:50:20.264730+0000","last_clean":"2024-03-11T21:50:20.264730+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:50:20.264730+0000","last_undegraded":"2024-03-11T21:50:20.264730+0000","last_fullsized":"2024-03-11T21:50:20.264730+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.805355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:53.805355+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.805355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.623421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"983'35","reported_seq":2022,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.872901+0000","last_change":"2024-03-11T21:50:02.812689+0000","last_active":"2024-03-11T21:50:20.872901+0000","last_peered":"2024-03-11T21:50:20.872901+0000","last_clean":"2024-03-11T21:50:20.872901+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:50:20.872901+0000","last_undegraded":"2024-03-11T21:50:20.872901+0000","last_fullsized":"2024-03-11T21:50:20.872901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"983'35","last_scrub_stamp":"2024-03-11T21:50:02.812297+0000","last_deep_scrub":"983'35","last_deep_scrub_stamp":"2024-03-11T21:46:27.236614+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.812297+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.029970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":1996,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:21.114414+0000","last_change":"2024-03-11T21:49:59.331379+0000","last_active":"2024-03-11T21:50:21.114414+0000","last_peered":"2024-03-11T21:50:21.114414+0000","last_clean":"2024-03-11T21:50:21.114414+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:50:21.114414+0000","last_undegraded":"2024-03-11T21:50:21.114414+0000","last_fullsized":"2024-03-11T21:50:21.114414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:49:59.331148+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.331148+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:13.934714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":2011,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.258927+0000","last_change":"2024-03-11T21:50:06.071989+0000","last_active":"2024-03-11T21:50:20.258927+0000","last_peered":"2024-03-11T21:50:20.258927+0000","last_clean":"2024-03-11T21:50:20.258927+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:50:20.258927+0000","last_undegraded":"2024-03-11T21:50:20.258927+0000","last_fullsized":"2024-03-11T21:50:20.258927+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.071780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:19.122842+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.071780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.649775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1953,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-03-11T21:50:20.871745+0000","last_change":"2024-03-11T21:49:42.647288+0000","last_active":"2024-03-11T21:50:20.871745+0000","last_peered":"2024-03-11T21:50:20.871745+0000","last_clean":"2024-03-11T21:50:20.871745+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:50:20.871745+0000","last_undegraded":"2024-03-11T21:50:20.871745+0000","last_fullsized":"2024-03-11T21:50:20.871745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:42.647007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:42.647007+0000","last_clean_scrub_stamp":"2024-03-11T21:49:42.647007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:54.683532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2691,"ondisk_log_size":2691,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1557,"num_read_kb":1314,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1085,"ondisk_log_size":1085,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1617,"num_read_kb":1612,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841876,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101532,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643620,"statfs":{"total":95995035648,"available":95891066880,"internally_reserved":0,"allocated":2371584,"data_stored":635292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168925,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87932,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93657220,"statfs":{"total":95995035648,"available":95904993280,"internally_reserved":0,"allocated":2666496,"data_stored":960752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352548,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52412,"kb_used_data":2288,"kb_used_omap":27,"kb_used_meta":50084,"kb_avail":93692740,"statfs":{"total":95995035648,"available":95941365760,"internally_reserved":0,"allocated":2342912,"data_stored":632949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":51286345},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738780,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114300,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630852,"statfs":{"total":95995035648,"available":95877992448,"internally_reserved":0,"allocated":2674688,"data_stored":961864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379015,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110080,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635072,"statfs":{"total":95995035648,"available":95882313728,"internally_reserved":0,"allocated":2605056,"data_stored":896215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117371,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110392,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93634760,"statfs":{"total":95995035648,"available":95881994240,"internally_reserved":0,"allocated":2924544,"data_stored":1223378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855729,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118164,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93626988,"statfs":{"total":95995035648,"available":95874035712,"internally_reserved":0,"allocated":2371584,"data_stored":634474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495975,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110072,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635080,"statfs":{"total":95995035648,"available":95882321920,"internally_reserved":0,"allocated":2596864,"data_stored":895675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:50:29.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.121096+0000 osd.6 (osd.6) 545 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.121476+0000 osd.6 (osd.6) 546 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.180559+0000 osd.4 (osd.4) 653 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.186899+0000 osd.4 (osd.4) 654 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: audit 2024-03-11T21:50:28.272245+0000 mgr.y (mgr.14550) 1047 : audit 0 from='client.42527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.569921+0000 osd.0 (osd.0) 694 : cluster 0 9.1d scrub starts 2024-03-11T21:50:29.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.575395+0000 osd.0 (osd.0) 695 : cluster 0 9.1d scrub ok 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.754534+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e1018: 8 total, 8 up, 8 in 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.856776+0000 osd.1 (osd.1) 493 : cluster 0 5.c deep-scrub starts 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.859088+0000 osd.1 (osd.1) 494 : cluster 0 5.c deep-scrub ok 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.121096+0000 osd.6 (osd.6) 545 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.121476+0000 osd.6 (osd.6) 546 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.180559+0000 osd.4 (osd.4) 653 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.186899+0000 osd.4 (osd.4) 654 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: audit 2024-03-11T21:50:28.272245+0000 mgr.y (mgr.14550) 1047 : audit 0 from='client.42527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.569921+0000 osd.0 (osd.0) 694 : cluster 0 9.1d scrub starts 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.575395+0000 osd.0 (osd.0) 695 : cluster 0 9.1d scrub ok 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.754534+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e1018: 8 total, 8 up, 8 in 2024-03-11T21:50:29.703 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.856776+0000 osd.1 (osd.1) 493 : cluster 0 5.c deep-scrub starts 2024-03-11T21:50:29.704 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:29 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.859088+0000 osd.1 (osd.1) 494 : cluster 0 5.c deep-scrub ok 2024-03-11T21:50:29.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:29.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.121096+0000 osd.6 (osd.6) 545 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.121476+0000 osd.6 (osd.6) 546 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.180559+0000 osd.4 (osd.4) 653 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.186899+0000 osd.4 (osd.4) 654 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: audit 2024-03-11T21:50:28.272245+0000 mgr.y (mgr.14550) 1047 : audit 0 from='client.42527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.569921+0000 osd.0 (osd.0) 694 : cluster 0 9.1d scrub starts 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.575395+0000 osd.0 (osd.0) 695 : cluster 0 9.1d scrub ok 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.754534+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e1018: 8 total, 8 up, 8 in 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.856776+0000 osd.1 (osd.1) 493 : cluster 0 5.c deep-scrub starts 2024-03-11T21:50:29.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:29 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.859088+0000 osd.1 (osd.1) 494 : cluster 0 5.c deep-scrub ok 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.768343+0000 osd.7 (osd.7) 708 : cluster 0 5.1a scrub starts 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:28.773990+0000 osd.7 (osd.7) 709 : cluster 0 5.1a scrub ok 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:29.401106+0000 osd.5 (osd.5) 619 : cluster 0 5.15 scrub starts 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:29.403770+0000 osd.5 (osd.5) 620 : cluster 0 5.15 scrub ok 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: audit 2024-03-11T21:50:29.488221+0000 mgr.y (mgr.14550) 1048 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:29.689987+0000 osd.4 (osd.4) 655 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:29.690473+0000 osd.4 (osd.4) 656 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:30.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[21512]: cluster 2024-03-11T21:50:29.892926+0000 mgr.y (mgr.14550) 1049 : cluster 0 pgmap v1720: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.768343+0000 osd.7 (osd.7) 708 : cluster 0 5.1a scrub starts 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:28.773990+0000 osd.7 (osd.7) 709 : cluster 0 5.1a scrub ok 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:29.401106+0000 osd.5 (osd.5) 619 : cluster 0 5.15 scrub starts 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:29.403770+0000 osd.5 (osd.5) 620 : cluster 0 5.15 scrub ok 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: audit 2024-03-11T21:50:29.488221+0000 mgr.y (mgr.14550) 1048 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:29.689987+0000 osd.4 (osd.4) 655 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:29.690473+0000 osd.4 (osd.4) 656 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:30.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:30 smithi143 bash[29284]: cluster 2024-03-11T21:50:29.892926+0000 mgr.y (mgr.14550) 1049 : cluster 0 pgmap v1720: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:50:30.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.768343+0000 osd.7 (osd.7) 708 : cluster 0 5.1a scrub starts 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:28.773990+0000 osd.7 (osd.7) 709 : cluster 0 5.1a scrub ok 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:29.401106+0000 osd.5 (osd.5) 619 : cluster 0 5.15 scrub starts 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:29.403770+0000 osd.5 (osd.5) 620 : cluster 0 5.15 scrub ok 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: audit 2024-03-11T21:50:29.488221+0000 mgr.y (mgr.14550) 1048 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:29.689987+0000 osd.4 (osd.4) 655 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:29.690473+0000 osd.4 (osd.4) 656 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:30.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:30 smithi203 bash[23654]: cluster 2024-03-11T21:50:29.892926+0000 mgr.y (mgr.14550) 1049 : cluster 0 pgmap v1720: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[21512]: cluster 2024-03-11T21:50:30.315754+0000 mon.a (mon.0) 4185 : cluster 0 osdmap e1019: 8 total, 8 up, 8 in 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[21512]: cluster 2024-03-11T21:50:30.445242+0000 osd.5 (osd.5) 621 : cluster 0 3.c scrub starts 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[21512]: cluster 2024-03-11T21:50:30.449071+0000 osd.5 (osd.5) 622 : cluster 0 3.c scrub ok 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[21512]: cluster 2024-03-11T21:50:30.864021+0000 osd.3 (osd.3) 701 : cluster 0 6.b scrub starts 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[21512]: cluster 2024-03-11T21:50:30.866310+0000 osd.3 (osd.3) 702 : cluster 0 6.b scrub ok 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[29284]: cluster 2024-03-11T21:50:30.315754+0000 mon.a (mon.0) 4185 : cluster 0 osdmap e1019: 8 total, 8 up, 8 in 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[29284]: cluster 2024-03-11T21:50:30.445242+0000 osd.5 (osd.5) 621 : cluster 0 3.c scrub starts 2024-03-11T21:50:31.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[29284]: cluster 2024-03-11T21:50:30.449071+0000 osd.5 (osd.5) 622 : cluster 0 3.c scrub ok 2024-03-11T21:50:31.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[29284]: cluster 2024-03-11T21:50:30.864021+0000 osd.3 (osd.3) 701 : cluster 0 6.b scrub starts 2024-03-11T21:50:31.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:31 smithi143 bash[29284]: cluster 2024-03-11T21:50:30.866310+0000 osd.3 (osd.3) 702 : cluster 0 6.b scrub ok 2024-03-11T21:50:31.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:31 smithi203 bash[23654]: cluster 2024-03-11T21:50:30.315754+0000 mon.a (mon.0) 4185 : cluster 0 osdmap e1019: 8 total, 8 up, 8 in 2024-03-11T21:50:31.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:31 smithi203 bash[23654]: cluster 2024-03-11T21:50:30.445242+0000 osd.5 (osd.5) 621 : cluster 0 3.c scrub starts 2024-03-11T21:50:31.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:31 smithi203 bash[23654]: cluster 2024-03-11T21:50:30.449071+0000 osd.5 (osd.5) 622 : cluster 0 3.c scrub ok 2024-03-11T21:50:31.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:31 smithi203 bash[23654]: cluster 2024-03-11T21:50:30.864021+0000 osd.3 (osd.3) 701 : cluster 0 6.b scrub starts 2024-03-11T21:50:31.783 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:31 smithi203 bash[23654]: cluster 2024-03-11T21:50:30.866310+0000 osd.3 (osd.3) 702 : cluster 0 6.b scrub ok 2024-03-11T21:50:32.451 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:32 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:33.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:32 smithi203 bash[23654]: cluster 2024-03-11T21:50:31.893809+0000 mgr.y (mgr.14550) 1050 : cluster 0 pgmap v1722: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:33.201 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:32 smithi143 bash[29284]: cluster 2024-03-11T21:50:31.893809+0000 mgr.y (mgr.14550) 1050 : cluster 0 pgmap v1722: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:33.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:32 smithi143 bash[21512]: cluster 2024-03-11T21:50:31.893809+0000 mgr.y (mgr.14550) 1050 : cluster 0 pgmap v1722: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:34.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.563096+0000 osd.0 (osd.0) 696 : cluster 0 1.11 scrub starts 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.565798+0000 osd.0 (osd.0) 697 : cluster 0 1.11 scrub ok 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.700427+0000 osd.7 (osd.7) 710 : cluster 0 9.d scrub starts 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.704427+0000 osd.7 (osd.7) 711 : cluster 0 9.d scrub ok 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.797300+0000 osd.3 (osd.3) 703 : cluster 0 9.1b scrub starts 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:32.804038+0000 osd.3 (osd.3) 704 : cluster 0 9.1b scrub ok 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:33.467848+0000 osd.5 (osd.5) 623 : cluster 0 3.16 scrub starts 2024-03-11T21:50:34.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:33 smithi203 bash[23654]: cluster 2024-03-11T21:50:33.471549+0000 osd.5 (osd.5) 624 : cluster 0 3.16 scrub ok 2024-03-11T21:50:34.117 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:34.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.563096+0000 osd.0 (osd.0) 696 : cluster 0 1.11 scrub starts 2024-03-11T21:50:34.139 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.565798+0000 osd.0 (osd.0) 697 : cluster 0 1.11 scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.700427+0000 osd.7 (osd.7) 710 : cluster 0 9.d scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.704427+0000 osd.7 (osd.7) 711 : cluster 0 9.d scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.797300+0000 osd.3 (osd.3) 703 : cluster 0 9.1b scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:32.804038+0000 osd.3 (osd.3) 704 : cluster 0 9.1b scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:33.467848+0000 osd.5 (osd.5) 623 : cluster 0 3.16 scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[21512]: cluster 2024-03-11T21:50:33.471549+0000 osd.5 (osd.5) 624 : cluster 0 3.16 scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.563096+0000 osd.0 (osd.0) 696 : cluster 0 1.11 scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.565798+0000 osd.0 (osd.0) 697 : cluster 0 1.11 scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.700427+0000 osd.7 (osd.7) 710 : cluster 0 9.d scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.704427+0000 osd.7 (osd.7) 711 : cluster 0 9.d scrub ok 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.797300+0000 osd.3 (osd.3) 703 : cluster 0 9.1b scrub starts 2024-03-11T21:50:34.140 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:32.804038+0000 osd.3 (osd.3) 704 : cluster 0 9.1b scrub ok 2024-03-11T21:50:34.141 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:33.467848+0000 osd.5 (osd.5) 623 : cluster 0 3.16 scrub starts 2024-03-11T21:50:34.141 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:33 smithi143 bash[29284]: cluster 2024-03-11T21:50:33.471549+0000 osd.5 (osd.5) 624 : cluster 0 3.16 scrub ok 2024-03-11T21:50:35.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:34 smithi203 bash[23654]: cluster 2024-03-11T21:50:33.200229+0000 osd.4 (osd.4) 657 : cluster 0 6.17 scrub starts 2024-03-11T21:50:35.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:34 smithi203 bash[23654]: cluster 2024-03-11T21:50:33.202463+0000 osd.4 (osd.4) 658 : cluster 0 6.17 scrub ok 2024-03-11T21:50:35.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:34 smithi203 bash[23654]: cluster 2024-03-11T21:50:33.895592+0000 mgr.y (mgr.14550) 1051 : cluster 0 pgmap v1723: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:35.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:34 smithi203 bash[23654]: cluster 2024-03-11T21:50:34.499963+0000 osd.5 (osd.5) 625 : cluster 0 9.7 scrub starts 2024-03-11T21:50:35.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:34 smithi203 bash[23654]: cluster 2024-03-11T21:50:34.506534+0000 osd.5 (osd.5) 626 : cluster 0 9.7 scrub ok 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[29284]: cluster 2024-03-11T21:50:33.200229+0000 osd.4 (osd.4) 657 : cluster 0 6.17 scrub starts 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[29284]: cluster 2024-03-11T21:50:33.202463+0000 osd.4 (osd.4) 658 : cluster 0 6.17 scrub ok 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[29284]: cluster 2024-03-11T21:50:33.895592+0000 mgr.y (mgr.14550) 1051 : cluster 0 pgmap v1723: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[29284]: cluster 2024-03-11T21:50:34.499963+0000 osd.5 (osd.5) 625 : cluster 0 9.7 scrub starts 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[29284]: cluster 2024-03-11T21:50:34.506534+0000 osd.5 (osd.5) 626 : cluster 0 9.7 scrub ok 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[21512]: cluster 2024-03-11T21:50:33.200229+0000 osd.4 (osd.4) 657 : cluster 0 6.17 scrub starts 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[21512]: cluster 2024-03-11T21:50:33.202463+0000 osd.4 (osd.4) 658 : cluster 0 6.17 scrub ok 2024-03-11T21:50:35.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[21512]: cluster 2024-03-11T21:50:33.895592+0000 mgr.y (mgr.14550) 1051 : cluster 0 pgmap v1723: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:35.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[21512]: cluster 2024-03-11T21:50:34.499963+0000 osd.5 (osd.5) 625 : cluster 0 9.7 scrub starts 2024-03-11T21:50:35.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:34 smithi143 bash[21512]: cluster 2024-03-11T21:50:34.506534+0000 osd.5 (osd.5) 626 : cluster 0 9.7 scrub ok 2024-03-11T21:50:35.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:35 smithi143 bash[21512]: cluster 2024-03-11T21:50:34.639139+0000 osd.6 (osd.6) 547 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:50:35.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:35 smithi143 bash[21512]: cluster 2024-03-11T21:50:34.641712+0000 osd.6 (osd.6) 548 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:50:35.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:35 smithi143 bash[29284]: cluster 2024-03-11T21:50:34.639139+0000 osd.6 (osd.6) 547 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:50:35.952 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:35 smithi143 bash[29284]: cluster 2024-03-11T21:50:34.641712+0000 osd.6 (osd.6) 548 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:50:36.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:35 smithi203 bash[23654]: cluster 2024-03-11T21:50:34.639139+0000 osd.6 (osd.6) 547 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:50:36.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:35 smithi203 bash[23654]: cluster 2024-03-11T21:50:34.641712+0000 osd.6 (osd.6) 548 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:50:37.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:36 smithi203 bash[23654]: cluster 2024-03-11T21:50:35.717210+0000 osd.7 (osd.7) 712 : cluster 0 3.f scrub starts 2024-03-11T21:50:37.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:36 smithi203 bash[23654]: cluster 2024-03-11T21:50:35.722470+0000 osd.7 (osd.7) 713 : cluster 0 3.f scrub ok 2024-03-11T21:50:37.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:36 smithi203 bash[23654]: cluster 2024-03-11T21:50:35.896581+0000 mgr.y (mgr.14550) 1052 : cluster 0 pgmap v1724: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:37.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:36 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.555941+0000 osd.5 (osd.5) 627 : cluster 0 1.a scrub starts 2024-03-11T21:50:37.033 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:36 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.558431+0000 osd.5 (osd.5) 628 : cluster 0 1.a scrub ok 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[21512]: cluster 2024-03-11T21:50:35.717210+0000 osd.7 (osd.7) 712 : cluster 0 3.f scrub starts 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[21512]: cluster 2024-03-11T21:50:35.722470+0000 osd.7 (osd.7) 713 : cluster 0 3.f scrub ok 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[21512]: cluster 2024-03-11T21:50:35.896581+0000 mgr.y (mgr.14550) 1052 : cluster 0 pgmap v1724: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.555941+0000 osd.5 (osd.5) 627 : cluster 0 1.a scrub starts 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.558431+0000 osd.5 (osd.5) 628 : cluster 0 1.a scrub ok 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[29284]: cluster 2024-03-11T21:50:35.717210+0000 osd.7 (osd.7) 712 : cluster 0 3.f scrub starts 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[29284]: cluster 2024-03-11T21:50:35.722470+0000 osd.7 (osd.7) 713 : cluster 0 3.f scrub ok 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[29284]: cluster 2024-03-11T21:50:35.896581+0000 mgr.y (mgr.14550) 1052 : cluster 0 pgmap v1724: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.555941+0000 osd.5 (osd.5) 627 : cluster 0 1.a scrub starts 2024-03-11T21:50:37.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:36 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.558431+0000 osd.5 (osd.5) 628 : cluster 0 1.a scrub ok 2024-03-11T21:50:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- 172.21.15.143:0/278837913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074930 msgr2=0x7fb2b410e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/278837913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074930 0x7fb2b410e330 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009990 tx=0x7fb2a4009ca0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- 172.21.15.143:0/278837913 shutdown_connections 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/278837913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074010 0x7fb2b40743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/278837913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 0x7fb2b4110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/278837913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074930 0x7fb2b410e330 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- 172.21.15.143:0/278837913 >> 172.21.15.143:0/278837913 conn(0x7fb2b4078eb0 msgr2=0x7fb2b40792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- 172.21.15.143:0/278837913 shutdown_connections 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- 172.21.15.143:0/278837913 wait complete. 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 librados: starting msgr at 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 librados: starting objecter 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 Processor -- start 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- start start 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 librados: setting wanted keys 2024-03-11T21:50:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 librados: calling monclient init 2024-03-11T21:50:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 0x7fb2b419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 0x7fb2b419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b4113870 con 0x7fb2b4074930 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41136f0 con 0x7fb2b410e870 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2bbb13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41139f0 con 0x7fb2b4074010 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:49142/0 (socket says 172.21.15.143:49142) 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b90ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 0x7fb2b419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 -- 172.21.15.143:0/2439634761 learned_addr learned my addr 172.21.15.143:0/2439634761 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2ba0b0700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 0x7fb2b419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 -- 172.21.15.143:0/2439634761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 msgr2=0x7fb2b419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 0x7fb2b419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 -- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 msgr2=0x7fb2b419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 0x7fb2b419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b98af700 1 -- 172.21.15.143:0/2439634761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b419ee40 con 0x7fb2b4074010 2024-03-11T21:50:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.538+0000 7fb2b90ae700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 0x7fb2b419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2b98af700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000eb30 tx=0x7fb2b000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 10 client.?.objecter ms_handle_connect 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b000cc90 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7fb2b000ce30 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b419f130 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0003df0 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2bbb13700 10 client.42533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b41a69b0 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2bbb13700 1 librados: init done 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b404faa0 con 0x7fb2b4074010 2024-03-11T21:50:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb2b0003f90 con 0x7fb2b4074010 2024-03-11T21:50:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 0x7fb2a0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1019..1019 src has 964..1019) v4 ==== 8794+0+0 (secure 0 0 0) 0x7fb2b008ea80 con 0x7fb2b4074010 2024-03-11T21:50:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 10 client.42533.objecter ms_dispatch 0x7fb2b4001550 osd_map(1019..1019 src has 964..1019) v4 2024-03-11T21:50:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 3 client.42533.objecter handle_osd_map got epochs [1019,1019] > 0 2024-03-11T21:50:37.537 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 3 client.42533.objecter handle_osd_map decoding full epoch 1019 2024-03-11T21:50:37.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2b90ae700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 0x7fb2a0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:37.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 20 client.42533.objecter dump_active .. 0 homeless 2024-03-11T21:50:37.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2b90ae700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 0x7fb2a0071ce0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4101c70 tx=0x7fb2a401a040 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:37.538 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.542+0000 7fb2aaffd700 10 client.42533.objecter ms_handle_connect 0x7fb2a006f820 2024-03-11T21:50:37.540 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.546+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb2b0016330 con 0x7fb2b4074010 2024-03-11T21:50:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.746+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 --> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2b419b190 con 0x7fb2a006f820 2024-03-11T21:50:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.762+0000 7fb2aaffd700 1 -- 172.21.15.143:0/2439634761 <== mgr.14550 v2:172.21.15.143:6800/584821010 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+477720 (secure 0 0 0) 0x7fb2b419b190 con 0x7fb2a006f820 2024-03-11T21:50:37.760 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-11T21:50:37.765 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 10 librados: watch_flush enter 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 10 librados: watch_flush exit 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 20 client.42533.objecter shutdown clearing up homeless session... 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 10 client.42533.objecter successfully canceled tick 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 msgr2=0x7fb2a0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 0x7fb2a0071ce0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4101c70 tx=0x7fb2a401a040 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 msgr2=0x7fb2b4199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000eb30 tx=0x7fb2b000ee40 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 shutdown_connections 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7fb2a006f820 0x7fb2a0071ce0 unknown :-1 s=CLOSED pgs=3532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b410e870 0x7fb2b419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2b4074010 0x7fb2b4199e80 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 --2- 172.21.15.143:0/2439634761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2b4074930 0x7fb2b419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:37.766 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 >> 172.21.15.143:0/2439634761 conn(0x7fb2b4078eb0 msgr2=0x7fb2b41062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:37.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.766+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 shutdown_connections 2024-03-11T21:50:37.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.770+0000 7fb2bbb13700 1 -- 172.21.15.143:0/2439634761 wait complete. 2024-03-11T21:50:37.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:37.770+0000 7fb2bbb13700 1 librados: shutdown 2024-03-11T21:50:37.767 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-03-11T21:50:38.030 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.539245+0000 osd.0 (osd.0) 698 : cluster 0 3.17 scrub starts 2024-03-11T21:50:38.030 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.542826+0000 osd.0 (osd.0) 699 : cluster 0 3.17 scrub ok 2024-03-11T21:50:38.030 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.713185+0000 osd.7 (osd.7) 714 : cluster 0 2.1c scrub starts 2024-03-11T21:50:38.030 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[21512]: cluster 2024-03-11T21:50:36.715558+0000 osd.7 (osd.7) 715 : cluster 0 2.1c scrub ok 2024-03-11T21:50:38.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.539245+0000 osd.0 (osd.0) 698 : cluster 0 3.17 scrub starts 2024-03-11T21:50:38.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.542826+0000 osd.0 (osd.0) 699 : cluster 0 3.17 scrub ok 2024-03-11T21:50:38.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.713185+0000 osd.7 (osd.7) 714 : cluster 0 2.1c scrub starts 2024-03-11T21:50:38.031 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:37 smithi143 bash[29284]: cluster 2024-03-11T21:50:36.715558+0000 osd.7 (osd.7) 715 : cluster 0 2.1c scrub ok 2024-03-11T21:50:38.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:37 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.539245+0000 osd.0 (osd.0) 698 : cluster 0 3.17 scrub starts 2024-03-11T21:50:38.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:37 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.542826+0000 osd.0 (osd.0) 699 : cluster 0 3.17 scrub ok 2024-03-11T21:50:38.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:37 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.713185+0000 osd.7 (osd.7) 714 : cluster 0 2.1c scrub starts 2024-03-11T21:50:38.032 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:37 smithi203 bash[23654]: cluster 2024-03-11T21:50:36.715558+0000 osd.7 (osd.7) 715 : cluster 0 2.1c scrub ok 2024-03-11T21:50:38.718 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1724,"stamp":"2024-03-11T21:50:35.895839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596397,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3227,"num_read_kb":2979,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3907,"ondisk_log_size":3907,"up":552,"acting":552,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":552,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":805124,"kb_used_data":20328,"kb_used_omap":38,"kb_used_meta":784665,"kb_avail":749156092,"statfs":{"total":767960285184,"available":767135838208,"internally_reserved":0,"allocated":20815872,"data_stored":6985303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39499,"internal_metadata":803497397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003933"},"pg_stats":[{"pgid":"1.d","version":"98'128","reported_seq":598,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.865018+0000","last_change":"2024-03-11T21:49:57.941524+0000","last_active":"2024-03-11T21:50:30.865018+0000","last_peered":"2024-03-11T21:50:30.865018+0000","last_clean":"2024-03-11T21:50:30.865018+0000","last_became_active":"2024-03-11T21:44:03.844629+0000","last_became_peered":"2024-03-11T21:44:03.844629+0000","last_unstale":"2024-03-11T21:50:30.865018+0000","last_undegraded":"2024-03-11T21:50:30.865018+0000","last_fullsized":"2024-03-11T21:50:30.865018+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:57.941299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.941299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.824265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1836,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314505+0000","last_change":"2024-03-11T21:49:45.202481+0000","last_active":"2024-03-11T21:50:30.314505+0000","last_peered":"2024-03-11T21:50:30.314505+0000","last_clean":"2024-03-11T21:50:30.314505+0000","last_became_active":"2024-03-11T21:28:51.387647+0000","last_became_peered":"2024-03-11T21:28:51.387647+0000","last_unstale":"2024-03-11T21:50:30.314505+0000","last_undegraded":"2024-03-11T21:50:30.314505+0000","last_fullsized":"2024-03-11T21:50:30.314505+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:45.202279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:19.937242+0000","last_clean_scrub_stamp":"2024-03-11T21:49:45.202279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.007736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"84'11","reported_seq":2294,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862237+0000","last_change":"2024-03-11T21:48:55.188592+0000","last_active":"2024-03-11T21:50:30.862237+0000","last_peered":"2024-03-11T21:50:30.862237+0000","last_clean":"2024-03-11T21:50:30.862237+0000","last_became_active":"2024-03-11T21:23:07.086704+0000","last_became_peered":"2024-03-11T21:23:07.086704+0000","last_unstale":"2024-03-11T21:50:30.862237+0000","last_undegraded":"2024-03-11T21:50:30.862237+0000","last_fullsized":"2024-03-11T21:50:30.862237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:48:55.188208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.188208+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"864'39","reported_seq":2019,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.864973+0000","last_change":"2024-03-11T21:48:58.246525+0000","last_active":"2024-03-11T21:50:30.864973+0000","last_peered":"2024-03-11T21:50:30.864973+0000","last_clean":"2024-03-11T21:50:30.864973+0000","last_became_active":"2024-03-11T21:23:04.936993+0000","last_became_peered":"2024-03-11T21:23:04.936993+0000","last_unstale":"2024-03-11T21:50:30.864973+0000","last_undegraded":"2024-03-11T21:50:30.864973+0000","last_fullsized":"2024-03-11T21:50:30.864973+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:48:58.246227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.246227+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1973,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.430829+0000","last_change":"2024-03-11T21:50:16.497449+0000","last_active":"2024-03-11T21:50:31.430829+0000","last_peered":"2024-03-11T21:50:31.430829+0000","last_clean":"2024-03-11T21:50:31.430829+0000","last_became_active":"2024-03-11T21:47:12.294681+0000","last_became_peered":"2024-03-11T21:47:12.294681+0000","last_unstale":"2024-03-11T21:50:31.430829+0000","last_undegraded":"2024-03-11T21:50:31.430829+0000","last_fullsized":"2024-03-11T21:50:31.430829+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.497231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.497231+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.497231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.271210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1772,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314504+0000","last_change":"2024-03-11T21:49:59.171605+0000","last_active":"2024-03-11T21:50:30.314504+0000","last_peered":"2024-03-11T21:50:30.314504+0000","last_clean":"2024-03-11T21:50:30.314504+0000","last_became_active":"2024-03-11T21:48:12.921751+0000","last_became_peered":"2024-03-11T21:48:12.921751+0000","last_unstale":"2024-03-11T21:50:30.314504+0000","last_undegraded":"2024-03-11T21:50:30.314504+0000","last_fullsized":"2024-03-11T21:50:30.314504+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:59.171386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.171386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.556200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"98'128","reported_seq":605,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.152331+0000","last_change":"2024-03-11T21:50:02.063678+0000","last_active":"2024-03-11T21:50:31.152331+0000","last_peered":"2024-03-11T21:50:31.152331+0000","last_clean":"2024-03-11T21:50:31.152331+0000","last_became_active":"2024-03-11T21:44:03.835751+0000","last_became_peered":"2024-03-11T21:44:03.835751+0000","last_unstale":"2024-03-11T21:50:31.152331+0000","last_undegraded":"2024-03-11T21:50:31.152331+0000","last_fullsized":"2024-03-11T21:50:31.152331+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:02.063438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.063438+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.261457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1631,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314733+0000","last_change":"2024-03-11T21:50:10.176409+0000","last_active":"2024-03-11T21:50:30.314733+0000","last_peered":"2024-03-11T21:50:30.314733+0000","last_clean":"2024-03-11T21:50:30.314733+0000","last_became_active":"2024-03-11T21:48:12.933645+0000","last_became_peered":"2024-03-11T21:48:12.933645+0000","last_unstale":"2024-03-11T21:50:30.314733+0000","last_undegraded":"2024-03-11T21:50:30.314733+0000","last_fullsized":"2024-03-11T21:50:30.314733+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.176190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.176190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.935103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"864'30","reported_seq":1833,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.449292+0000","last_change":"2024-03-11T21:50:30.449420+0000","last_active":"2024-03-11T21:50:30.449292+0000","last_peered":"2024-03-11T21:50:30.449292+0000","last_clean":"2024-03-11T21:50:30.449292+0000","last_became_active":"2024-03-11T21:48:12.917408+0000","last_became_peered":"2024-03-11T21:48:12.917408+0000","last_unstale":"2024-03-11T21:50:30.449292+0000","last_undegraded":"2024-03-11T21:50:30.449292+0000","last_fullsized":"2024-03-11T21:50:30.449292+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:50:30.449082+0000","last_deep_scrub":"864'30","last_deep_scrub_stamp":"2024-03-11T21:43:27.559051+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.449082+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:52.653731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1890,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.318202+0000","last_change":"2024-03-11T21:49:57.844013+0000","last_active":"2024-03-11T21:50:30.318202+0000","last_peered":"2024-03-11T21:50:30.318202+0000","last_clean":"2024-03-11T21:50:30.318202+0000","last_became_active":"2024-03-11T21:31:37.388226+0000","last_became_peered":"2024-03-11T21:31:37.388226+0000","last_unstale":"2024-03-11T21:50:30.318202+0000","last_undegraded":"2024-03-11T21:50:30.318202+0000","last_fullsized":"2024-03-11T21:50:30.318202+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:57.843884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:04.246471+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.843884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:01.534629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1966,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.430149+0000","last_change":"2024-03-11T21:49:11.797005+0000","last_active":"2024-03-11T21:50:31.430149+0000","last_peered":"2024-03-11T21:50:31.430149+0000","last_clean":"2024-03-11T21:50:31.430149+0000","last_became_active":"2024-03-11T21:47:12.046031+0000","last_became_peered":"2024-03-11T21:47:12.046031+0000","last_unstale":"2024-03-11T21:50:31.430149+0000","last_undegraded":"2024-03-11T21:50:31.430149+0000","last_fullsized":"2024-03-11T21:50:31.430149+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:11.796833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:30.018128+0000","last_clean_scrub_stamp":"2024-03-11T21:49:11.796833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.545380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1978,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313109+0000","last_change":"2024-03-11T21:49:33.172302+0000","last_active":"2024-03-11T21:50:30.313109+0000","last_peered":"2024-03-11T21:50:30.313109+0000","last_clean":"2024-03-11T21:50:30.313109+0000","last_became_active":"2024-03-11T21:23:09.149550+0000","last_became_peered":"2024-03-11T21:23:09.149550+0000","last_unstale":"2024-03-11T21:50:30.313109+0000","last_undegraded":"2024-03-11T21:50:30.313109+0000","last_fullsized":"2024-03-11T21:50:30.313109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:33.172048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:58.518630+0000","last_clean_scrub_stamp":"2024-03-11T21:49:33.172048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:45.866974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"98'128","reported_seq":1256,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444131+0000","last_change":"2024-03-11T21:50:08.542983+0000","last_active":"2024-03-11T21:50:30.444131+0000","last_peered":"2024-03-11T21:50:30.444131+0000","last_clean":"2024-03-11T21:50:30.444131+0000","last_became_active":"2024-03-11T21:44:03.836601+0000","last_became_peered":"2024-03-11T21:44:03.836601+0000","last_unstale":"2024-03-11T21:50:30.444131+0000","last_undegraded":"2024-03-11T21:50:30.444131+0000","last_fullsized":"2024-03-11T21:50:30.444131+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:08.542763+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:38.453642+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.542763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.187479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1614,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314758+0000","last_change":"2024-03-11T21:49:15.153209+0000","last_active":"2024-03-11T21:50:30.314758+0000","last_peered":"2024-03-11T21:50:30.314758+0000","last_clean":"2024-03-11T21:50:30.314758+0000","last_became_active":"2024-03-11T21:48:12.918994+0000","last_became_peered":"2024-03-11T21:48:12.918994+0000","last_unstale":"2024-03-11T21:50:30.314758+0000","last_undegraded":"2024-03-11T21:50:30.314758+0000","last_fullsized":"2024-03-11T21:50:30.314758+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:15.152963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:15.152963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.250809+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"864'53","reported_seq":2044,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863985+0000","last_change":"2024-03-11T21:49:16.179412+0000","last_active":"2024-03-11T21:50:30.863985+0000","last_peered":"2024-03-11T21:50:30.863985+0000","last_clean":"2024-03-11T21:50:30.863985+0000","last_became_active":"2024-03-11T21:23:04.938522+0000","last_became_peered":"2024-03-11T21:23:04.938522+0000","last_unstale":"2024-03-11T21:50:30.863985+0000","last_undegraded":"2024-03-11T21:50:30.863985+0000","last_fullsized":"2024-03-11T21:50:30.863985+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'53","last_scrub_stamp":"2024-03-11T21:49:16.179066+0000","last_deep_scrub":"864'53","last_deep_scrub_stamp":"2024-03-11T21:49:16.179066+0000","last_clean_scrub_stamp":"2024-03-11T21:49:16.179066+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2035,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.429262+0000","last_change":"2024-03-11T21:49:52.427491+0000","last_active":"2024-03-11T21:50:31.429262+0000","last_peered":"2024-03-11T21:50:31.429262+0000","last_clean":"2024-03-11T21:50:31.429262+0000","last_became_active":"2024-03-11T21:23:02.941750+0000","last_became_peered":"2024-03-11T21:23:02.941750+0000","last_unstale":"2024-03-11T21:50:31.429262+0000","last_undegraded":"2024-03-11T21:50:31.429262+0000","last_fullsized":"2024-03-11T21:50:31.429262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:52.427283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:50.697855+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.427283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.324907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":2027,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.430515+0000","last_change":"2024-03-11T21:49:54.426722+0000","last_active":"2024-03-11T21:50:31.430515+0000","last_peered":"2024-03-11T21:50:31.430515+0000","last_clean":"2024-03-11T21:50:31.430515+0000","last_became_active":"2024-03-11T21:23:07.076173+0000","last_became_peered":"2024-03-11T21:23:07.076173+0000","last_unstale":"2024-03-11T21:50:31.430515+0000","last_undegraded":"2024-03-11T21:50:31.430515+0000","last_fullsized":"2024-03-11T21:50:31.430515+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.426445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:34.663677+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.426445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.164043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1958,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861667+0000","last_change":"2024-03-11T21:49:20.217882+0000","last_active":"2024-03-11T21:50:30.861667+0000","last_peered":"2024-03-11T21:50:30.861667+0000","last_clean":"2024-03-11T21:50:30.861667+0000","last_became_active":"2024-03-11T21:23:09.156343+0000","last_became_peered":"2024-03-11T21:23:09.156343+0000","last_unstale":"2024-03-11T21:50:30.861667+0000","last_undegraded":"2024-03-11T21:50:30.861667+0000","last_fullsized":"2024-03-11T21:50:30.861667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:20.217579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:45.155587+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.217579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.255994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"98'128","reported_seq":1011,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312481+0000","last_change":"2024-03-11T21:50:14.215639+0000","last_active":"2024-03-11T21:50:30.312481+0000","last_peered":"2024-03-11T21:50:30.312481+0000","last_clean":"2024-03-11T21:50:30.312481+0000","last_became_active":"2024-03-11T21:48:12.901167+0000","last_became_peered":"2024-03-11T21:48:12.901167+0000","last_unstale":"2024-03-11T21:50:30.312481+0000","last_undegraded":"2024-03-11T21:50:30.312481+0000","last_fullsized":"2024-03-11T21:50:30.312481+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:14.215402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.215402+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:37.269457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1710,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315995+0000","last_change":"2024-03-11T21:49:20.987439+0000","last_active":"2024-03-11T21:50:30.315995+0000","last_peered":"2024-03-11T21:50:30.315995+0000","last_clean":"2024-03-11T21:50:30.315995+0000","last_became_active":"2024-03-11T21:31:37.404102+0000","last_became_peered":"2024-03-11T21:31:37.404102+0000","last_unstale":"2024-03-11T21:50:30.315995+0000","last_undegraded":"2024-03-11T21:50:30.315995+0000","last_fullsized":"2024-03-11T21:50:30.315995+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:20.987221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:58.872662+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.987221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:39.254446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"864'60","reported_seq":2107,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443856+0000","last_change":"2024-03-11T21:49:24.607372+0000","last_active":"2024-03-11T21:50:30.443856+0000","last_peered":"2024-03-11T21:50:30.443856+0000","last_clean":"2024-03-11T21:50:30.443856+0000","last_became_active":"2024-03-11T21:31:37.416152+0000","last_became_peered":"2024-03-11T21:31:37.416152+0000","last_unstale":"2024-03-11T21:50:30.443856+0000","last_undegraded":"2024-03-11T21:50:30.443856+0000","last_fullsized":"2024-03-11T21:50:30.443856+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'60","last_scrub_stamp":"2024-03-11T21:49:24.607132+0000","last_deep_scrub":"104'28","last_deep_scrub_stamp":"2024-03-11T21:24:30.484231+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.607132+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:46.992895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1970,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863188+0000","last_change":"2024-03-11T21:49:00.174305+0000","last_active":"2024-03-11T21:50:30.863188+0000","last_peered":"2024-03-11T21:50:30.863188+0000","last_clean":"2024-03-11T21:50:30.863188+0000","last_became_active":"2024-03-11T21:23:02.946111+0000","last_became_peered":"2024-03-11T21:23:02.946111+0000","last_unstale":"2024-03-11T21:50:30.863188+0000","last_undegraded":"2024-03-11T21:50:30.863188+0000","last_fullsized":"2024-03-11T21:50:30.863188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:00.174079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:00.174079+0000","last_clean_scrub_stamp":"2024-03-11T21:49:00.174079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1896,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.319769+0000","last_change":"2024-03-11T21:50:28.859311+0000","last_active":"2024-03-11T21:50:30.319769+0000","last_peered":"2024-03-11T21:50:30.319769+0000","last_clean":"2024-03-11T21:50:30.319769+0000","last_became_active":"2024-03-11T21:31:37.397866+0000","last_became_peered":"2024-03-11T21:31:37.397866+0000","last_unstale":"2024-03-11T21:50:30.319769+0000","last_undegraded":"2024-03-11T21:50:30.319769+0000","last_fullsized":"2024-03-11T21:50:30.319769+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:28.859098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:28.859098+0000","last_clean_scrub_stamp":"2024-03-11T21:50:28.859098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:50.365471+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1969,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.431512+0000","last_change":"2024-03-11T21:50:15.484221+0000","last_active":"2024-03-11T21:50:31.431512+0000","last_peered":"2024-03-11T21:50:31.431512+0000","last_clean":"2024-03-11T21:50:31.431512+0000","last_became_active":"2024-03-11T21:47:12.046816+0000","last_became_peered":"2024-03-11T21:47:12.046816+0000","last_unstale":"2024-03-11T21:50:31.431512+0000","last_undegraded":"2024-03-11T21:50:31.431512+0000","last_fullsized":"2024-03-11T21:50:31.431512+0000","mapping_epoch":989,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.484002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.847708+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.484002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.211380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"98'128","reported_seq":2263,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862573+0000","last_change":"2024-03-11T21:50:26.776096+0000","last_active":"2024-03-11T21:50:30.862573+0000","last_peered":"2024-03-11T21:50:30.862573+0000","last_clean":"2024-03-11T21:50:30.862573+0000","last_became_active":"2024-03-11T21:37:52.453294+0000","last_became_peered":"2024-03-11T21:37:52.453294+0000","last_unstale":"2024-03-11T21:50:30.862573+0000","last_undegraded":"2024-03-11T21:50:30.862573+0000","last_fullsized":"2024-03-11T21:50:30.862573+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:26.775870+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:44.166875+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.775870+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.443250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8","reported_seq":1832,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411585+0000","last_change":"2024-03-11T21:50:13.959402+0000","last_active":"2024-03-11T21:50:30.411585+0000","last_peered":"2024-03-11T21:50:30.411585+0000","last_clean":"2024-03-11T21:50:30.411585+0000","last_became_active":"2024-03-11T21:28:49.936465+0000","last_became_peered":"2024-03-11T21:28:49.936465+0000","last_unstale":"2024-03-11T21:50:30.411585+0000","last_undegraded":"2024-03-11T21:50:30.411585+0000","last_fullsized":"2024-03-11T21:50:30.411585+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'8","last_scrub_stamp":"2024-03-11T21:50:13.959166+0000","last_deep_scrub":"333'8","last_deep_scrub_stamp":"2024-03-11T21:32:14.697115+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.959166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.656040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"864'21","reported_seq":2009,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412521+0000","last_change":"2024-03-11T21:49:58.753679+0000","last_active":"2024-03-11T21:50:30.412521+0000","last_peered":"2024-03-11T21:50:30.412521+0000","last_clean":"2024-03-11T21:50:30.412521+0000","last_became_active":"2024-03-11T21:23:04.944859+0000","last_became_peered":"2024-03-11T21:23:04.944859+0000","last_unstale":"2024-03-11T21:50:30.412521+0000","last_undegraded":"2024-03-11T21:50:30.412521+0000","last_fullsized":"2024-03-11T21:50:30.412521+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:58.753535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.753535+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:04.686060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1856,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.317524+0000","last_change":"2024-03-11T21:49:09.050688+0000","last_active":"2024-03-11T21:50:30.317524+0000","last_peered":"2024-03-11T21:50:30.317524+0000","last_clean":"2024-03-11T21:50:30.317524+0000","last_became_active":"2024-03-11T21:31:37.393692+0000","last_became_peered":"2024-03-11T21:31:37.393692+0000","last_unstale":"2024-03-11T21:50:30.317524+0000","last_undegraded":"2024-03-11T21:50:30.317524+0000","last_fullsized":"2024-03-11T21:50:30.317524+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:09.050558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:23.473313+0000","last_clean_scrub_stamp":"2024-03-11T21:49:09.050558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:35.993285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"100'12","reported_seq":2368,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.431056+0000","last_change":"2024-03-11T21:50:08.398539+0000","last_active":"2024-03-11T21:50:31.431056+0000","last_peered":"2024-03-11T21:50:31.431056+0000","last_clean":"2024-03-11T21:50:31.431056+0000","last_became_active":"2024-03-11T21:23:07.076267+0000","last_became_peered":"2024-03-11T21:23:07.076267+0000","last_unstale":"2024-03-11T21:50:31.431056+0000","last_undegraded":"2024-03-11T21:50:31.431056+0000","last_fullsized":"2024-03-11T21:50:31.431056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-03-11T21:50:08.398448+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-03-11T21:47:29.724498+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.398448+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.217831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2012,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.153477+0000","last_change":"2024-03-11T21:49:53.076319+0000","last_active":"2024-03-11T21:50:31.153477+0000","last_peered":"2024-03-11T21:50:31.153477+0000","last_clean":"2024-03-11T21:50:31.153477+0000","last_became_active":"2024-03-11T21:31:37.374797+0000","last_became_peered":"2024-03-11T21:31:37.374797+0000","last_unstale":"2024-03-11T21:50:31.153477+0000","last_undegraded":"2024-03-11T21:50:31.153477+0000","last_fullsized":"2024-03-11T21:50:31.153477+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.076002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:45.224251+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.076002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.096336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"98'128","reported_seq":1003,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313625+0000","last_change":"2024-03-11T21:49:32.022090+0000","last_active":"2024-03-11T21:50:30.313625+0000","last_peered":"2024-03-11T21:50:30.313625+0000","last_clean":"2024-03-11T21:50:30.313625+0000","last_became_active":"2024-03-11T21:48:12.927820+0000","last_became_peered":"2024-03-11T21:48:12.927820+0000","last_unstale":"2024-03-11T21:50:30.313625+0000","last_undegraded":"2024-03-11T21:50:30.313625+0000","last_fullsized":"2024-03-11T21:50:30.313625+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:32.021876+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:47:56.730128+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.021876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.676707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1827,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412784+0000","last_change":"2024-03-11T21:49:33.769769+0000","last_active":"2024-03-11T21:50:30.412784+0000","last_peered":"2024-03-11T21:50:30.412784+0000","last_clean":"2024-03-11T21:50:30.412784+0000","last_became_active":"2024-03-11T21:31:37.389881+0000","last_became_peered":"2024-03-11T21:31:37.389881+0000","last_unstale":"2024-03-11T21:50:30.412784+0000","last_undegraded":"2024-03-11T21:50:30.412784+0000","last_fullsized":"2024-03-11T21:50:30.412784+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:33.769541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:33.769541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.713101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"864'39","reported_seq":2045,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411666+0000","last_change":"2024-03-11T21:50:18.965374+0000","last_active":"2024-03-11T21:50:30.411666+0000","last_peered":"2024-03-11T21:50:30.411666+0000","last_clean":"2024-03-11T21:50:30.411666+0000","last_became_active":"2024-03-11T21:31:37.389684+0000","last_became_peered":"2024-03-11T21:31:37.389684+0000","last_unstale":"2024-03-11T21:50:30.411666+0000","last_undegraded":"2024-03-11T21:50:30.411666+0000","last_fullsized":"2024-03-11T21:50:30.411666+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'39","last_scrub_stamp":"2024-03-11T21:50:18.965036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.965036+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.703476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1892,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.317756+0000","last_change":"2024-03-11T21:50:20.880324+0000","last_active":"2024-03-11T21:50:30.317756+0000","last_peered":"2024-03-11T21:50:30.317756+0000","last_clean":"2024-03-11T21:50:30.317756+0000","last_became_active":"2024-03-11T21:31:37.387350+0000","last_became_peered":"2024-03-11T21:31:37.387350+0000","last_unstale":"2024-03-11T21:50:30.317756+0000","last_undegraded":"2024-03-11T21:50:30.317756+0000","last_fullsized":"2024-03-11T21:50:30.317756+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.880191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:20.880191+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.880191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.831112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"996'12","reported_seq":2348,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.153254+0000","last_change":"2024-03-11T21:50:13.093620+0000","last_active":"2024-03-11T21:50:31.153254+0000","last_peered":"2024-03-11T21:50:31.153254+0000","last_clean":"2024-03-11T21:50:31.153254+0000","last_became_active":"2024-03-11T21:23:07.073379+0000","last_became_peered":"2024-03-11T21:23:07.073379+0000","last_unstale":"2024-03-11T21:50:31.153254+0000","last_undegraded":"2024-03-11T21:50:31.153254+0000","last_fullsized":"2024-03-11T21:50:31.153254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'12","last_scrub_stamp":"2024-03-11T21:50:13.093274+0000","last_deep_scrub":"996'12","last_deep_scrub_stamp":"2024-03-11T21:47:41.169471+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.093274+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.341385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1781,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314236+0000","last_change":"2024-03-11T21:50:20.412097+0000","last_active":"2024-03-11T21:50:30.314236+0000","last_peered":"2024-03-11T21:50:30.314236+0000","last_clean":"2024-03-11T21:50:30.314236+0000","last_became_active":"2024-03-11T21:48:12.931965+0000","last_became_peered":"2024-03-11T21:48:12.931965+0000","last_unstale":"2024-03-11T21:50:30.314236+0000","last_undegraded":"2024-03-11T21:50:30.314236+0000","last_fullsized":"2024-03-11T21:50:30.314236+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.411748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.358679+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.411748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.330409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"98'128","reported_seq":1242,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316476+0000","last_change":"2024-03-11T21:50:13.895680+0000","last_active":"2024-03-11T21:50:30.316476+0000","last_peered":"2024-03-11T21:50:30.316476+0000","last_clean":"2024-03-11T21:50:30.316476+0000","last_became_active":"2024-03-11T21:44:03.849412+0000","last_became_peered":"2024-03-11T21:44:03.849412+0000","last_unstale":"2024-03-11T21:50:30.316476+0000","last_undegraded":"2024-03-11T21:50:30.316476+0000","last_fullsized":"2024-03-11T21:50:30.316476+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:13.895439+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:40.094801+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.895439+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.462926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1817,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862472+0000","last_change":"2024-03-11T21:50:09.973954+0000","last_active":"2024-03-11T21:50:30.862472+0000","last_peered":"2024-03-11T21:50:30.862472+0000","last_clean":"2024-03-11T21:50:30.862472+0000","last_became_active":"2024-03-11T21:31:37.403745+0000","last_became_peered":"2024-03-11T21:31:37.403745+0000","last_unstale":"2024-03-11T21:50:30.862472+0000","last_undegraded":"2024-03-11T21:50:30.862472+0000","last_fullsized":"2024-03-11T21:50:30.862472+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:09.973736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.973736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.170519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"864'29","reported_seq":2049,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.150937+0000","last_change":"2024-03-11T21:49:31.093759+0000","last_active":"2024-03-11T21:50:31.150937+0000","last_peered":"2024-03-11T21:50:31.150937+0000","last_clean":"2024-03-11T21:50:31.150937+0000","last_became_active":"2024-03-11T21:23:05.020725+0000","last_became_peered":"2024-03-11T21:23:05.020725+0000","last_unstale":"2024-03-11T21:50:31.150937+0000","last_undegraded":"2024-03-11T21:50:31.150937+0000","last_fullsized":"2024-03-11T21:50:31.150937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'29","last_scrub_stamp":"2024-03-11T21:49:31.093510+0000","last_deep_scrub":"389'23","last_deep_scrub_stamp":"2024-03-11T21:33:35.635256+0000","last_clean_scrub_stamp":"2024-03-11T21:49:31.093510+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.763626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1987,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862214+0000","last_change":"2024-03-11T21:50:07.949808+0000","last_active":"2024-03-11T21:50:30.862214+0000","last_peered":"2024-03-11T21:50:30.862214+0000","last_clean":"2024-03-11T21:50:30.862214+0000","last_became_active":"2024-03-11T21:31:37.401208+0000","last_became_peered":"2024-03-11T21:31:37.401208+0000","last_unstale":"2024-03-11T21:50:30.862214+0000","last_undegraded":"2024-03-11T21:50:30.862214+0000","last_fullsized":"2024-03-11T21:50:30.862214+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.949512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.633909+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.949512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.169783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1776,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315977+0000","last_change":"2024-03-11T21:50:12.199359+0000","last_active":"2024-03-11T21:50:30.315977+0000","last_peered":"2024-03-11T21:50:30.315977+0000","last_clean":"2024-03-11T21:50:30.315977+0000","last_became_active":"2024-03-11T21:48:12.919876+0000","last_became_peered":"2024-03-11T21:48:12.919876+0000","last_unstale":"2024-03-11T21:50:30.315977+0000","last_undegraded":"2024-03-11T21:50:30.315977+0000","last_fullsized":"2024-03-11T21:50:30.315977+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.199135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:12.199135+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.199135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.917003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1967,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413503+0000","last_change":"2024-03-11T21:50:26.971161+0000","last_active":"2024-03-11T21:50:30.413503+0000","last_peered":"2024-03-11T21:50:30.413503+0000","last_clean":"2024-03-11T21:50:30.413503+0000","last_became_active":"2024-03-11T21:23:09.151065+0000","last_became_peered":"2024-03-11T21:23:09.151065+0000","last_unstale":"2024-03-11T21:50:30.413503+0000","last_undegraded":"2024-03-11T21:50:30.413503+0000","last_fullsized":"2024-03-11T21:50:30.413503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.970950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.970950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:46.127417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"600'129","reported_seq":1219,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413786+0000","last_change":"2024-03-11T21:50:22.937258+0000","last_active":"2024-03-11T21:50:30.413786+0000","last_peered":"2024-03-11T21:50:30.413786+0000","last_clean":"2024-03-11T21:50:30.413786+0000","last_became_active":"2024-03-11T21:43:58.779902+0000","last_became_peered":"2024-03-11T21:43:58.779902+0000","last_unstale":"2024-03-11T21:50:30.413786+0000","last_undegraded":"2024-03-11T21:50:30.413786+0000","last_fullsized":"2024-03-11T21:50:30.413786+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:50:22.937007+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:50:22.937007+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.937007+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:38.277966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1817,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413941+0000","last_change":"2024-03-11T21:50:15.945937+0000","last_active":"2024-03-11T21:50:30.413941+0000","last_peered":"2024-03-11T21:50:30.413941+0000","last_clean":"2024-03-11T21:50:30.413941+0000","last_became_active":"2024-03-11T21:28:49.936657+0000","last_became_peered":"2024-03-11T21:28:49.936657+0000","last_unstale":"2024-03-11T21:50:30.413941+0000","last_undegraded":"2024-03-11T21:50:30.413941+0000","last_fullsized":"2024-03-11T21:50:30.413941+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.945726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:29.232293+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.945726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.917087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"981'14","reported_seq":2323,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412079+0000","last_change":"2024-03-11T21:50:20.989655+0000","last_active":"2024-03-11T21:50:30.412079+0000","last_peered":"2024-03-11T21:50:30.412079+0000","last_clean":"2024-03-11T21:50:30.412079+0000","last_became_active":"2024-03-11T21:31:37.390147+0000","last_became_peered":"2024-03-11T21:31:37.390147+0000","last_unstale":"2024-03-11T21:50:30.412079+0000","last_undegraded":"2024-03-11T21:50:30.412079+0000","last_fullsized":"2024-03-11T21:50:30.412079+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"981'14","last_scrub_stamp":"2024-03-11T21:50:20.989335+0000","last_deep_scrub":"981'14","last_deep_scrub_stamp":"2024-03-11T21:46:22.207673+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.989335+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.477578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":2017,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.151063+0000","last_change":"2024-03-11T21:49:41.170382+0000","last_active":"2024-03-11T21:50:31.151063+0000","last_peered":"2024-03-11T21:50:31.151063+0000","last_clean":"2024-03-11T21:50:31.151063+0000","last_became_active":"2024-03-11T21:23:06.063019+0000","last_became_peered":"2024-03-11T21:23:06.063019+0000","last_unstale":"2024-03-11T21:50:31.151063+0000","last_undegraded":"2024-03-11T21:50:31.151063+0000","last_fullsized":"2024-03-11T21:50:31.151063+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:41.170173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:32.039214+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.170173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.445835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001303785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"864'36","reported_seq":2049,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314353+0000","last_change":"2024-03-11T21:49:10.167764+0000","last_active":"2024-03-11T21:50:30.314353+0000","last_peered":"2024-03-11T21:50:30.314353+0000","last_clean":"2024-03-11T21:50:30.314353+0000","last_became_active":"2024-03-11T21:31:37.424432+0000","last_became_peered":"2024-03-11T21:31:37.424432+0000","last_unstale":"2024-03-11T21:50:30.314353+0000","last_undegraded":"2024-03-11T21:50:30.314353+0000","last_fullsized":"2024-03-11T21:50:30.314353+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'36","last_scrub_stamp":"2024-03-11T21:49:10.167630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:10.167630+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:39.022647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"124'2","reported_seq":2031,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444771+0000","last_change":"2024-03-11T21:49:39.553141+0000","last_active":"2024-03-11T21:50:30.444771+0000","last_peered":"2024-03-11T21:50:30.444771+0000","last_clean":"2024-03-11T21:50:30.444771+0000","last_became_active":"2024-03-11T21:23:02.932870+0000","last_became_peered":"2024-03-11T21:23:02.932870+0000","last_unstale":"2024-03-11T21:50:30.444771+0000","last_undegraded":"2024-03-11T21:50:30.444771+0000","last_fullsized":"2024-03-11T21:50:30.444771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:49:39.552909+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:49:39.552909+0000","last_clean_scrub_stamp":"2024-03-11T21:49:39.552909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.259353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"113'2","reported_seq":1969,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862862+0000","last_change":"2024-03-11T21:50:06.965521+0000","last_active":"2024-03-11T21:50:30.862862+0000","last_peered":"2024-03-11T21:50:30.862862+0000","last_clean":"2024-03-11T21:50:30.862862+0000","last_became_active":"2024-03-11T21:23:09.152953+0000","last_became_peered":"2024-03-11T21:23:09.152953+0000","last_unstale":"2024-03-11T21:50:30.862862+0000","last_undegraded":"2024-03-11T21:50:30.862862+0000","last_fullsized":"2024-03-11T21:50:30.862862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'2","last_scrub_stamp":"2024-03-11T21:50:06.965294+0000","last_deep_scrub":"113'2","last_deep_scrub_stamp":"2024-03-11T21:25:34.058815+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.965294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.332677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"98'128","reported_seq":1241,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.149805+0000","last_change":"2024-03-11T21:49:57.105257+0000","last_active":"2024-03-11T21:50:31.149805+0000","last_peered":"2024-03-11T21:50:31.149805+0000","last_clean":"2024-03-11T21:50:31.149805+0000","last_became_active":"2024-03-11T21:43:58.766374+0000","last_became_peered":"2024-03-11T21:43:58.766374+0000","last_unstale":"2024-03-11T21:50:31.149805+0000","last_undegraded":"2024-03-11T21:50:31.149805+0000","last_fullsized":"2024-03-11T21:50:31.149805+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:57.105035+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:48:32.258396+0000","last_clean_scrub_stamp":"2024-03-11T21:49:57.105035+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:04.561243+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1824,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:32.704622+0000","last_change":"2024-03-11T21:50:32.704679+0000","last_active":"2024-03-11T21:50:32.704622+0000","last_peered":"2024-03-11T21:50:32.704622+0000","last_clean":"2024-03-11T21:50:32.704622+0000","last_became_active":"2024-03-11T21:31:37.403613+0000","last_became_peered":"2024-03-11T21:31:37.403613+0000","last_unstale":"2024-03-11T21:50:32.704622+0000","last_undegraded":"2024-03-11T21:50:32.704622+0000","last_fullsized":"2024-03-11T21:50:32.704622+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:32.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:32.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.496485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":2016,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.150893+0000","last_change":"2024-03-11T21:49:52.093172+0000","last_active":"2024-03-11T21:50:31.150893+0000","last_peered":"2024-03-11T21:50:31.150893+0000","last_clean":"2024-03-11T21:50:31.150893+0000","last_became_active":"2024-03-11T21:23:07.069849+0000","last_became_peered":"2024-03-11T21:23:07.069849+0000","last_unstale":"2024-03-11T21:50:31.150893+0000","last_undegraded":"2024-03-11T21:50:31.150893+0000","last_fullsized":"2024-03-11T21:50:31.150893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:52.092945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:52.092945+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.092945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.183187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":3603,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.501327+0000","last_change":"2024-03-11T21:49:20.662025+0000","last_active":"2024-03-11T21:50:31.501327+0000","last_peered":"2024-03-11T21:50:31.501327+0000","last_clean":"2024-03-11T21:50:31.501327+0000","last_became_active":"2024-03-11T21:23:06.059486+0000","last_became_peered":"2024-03-11T21:23:06.059486+0000","last_unstale":"2024-03-11T21:50:31.501327+0000","last_undegraded":"2024-03-11T21:50:31.501327+0000","last_fullsized":"2024-03-11T21:50:31.501327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-03-11T21:49:20.661792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:04.912535+0000","last_clean_scrub_stamp":"2024-03-11T21:49:20.661792+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011081559999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1627,"num_read_kb":1622,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"864'21","reported_seq":2006,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.414100+0000","last_change":"2024-03-11T21:49:24.694334+0000","last_active":"2024-03-11T21:50:30.414100+0000","last_peered":"2024-03-11T21:50:30.414100+0000","last_clean":"2024-03-11T21:50:30.414100+0000","last_became_active":"2024-03-11T21:23:04.942738+0000","last_became_peered":"2024-03-11T21:23:04.942738+0000","last_unstale":"2024-03-11T21:50:30.414100+0000","last_undegraded":"2024-03-11T21:50:30.414100+0000","last_fullsized":"2024-03-11T21:50:30.414100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:24.694140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.694140+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.661510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"983'2","reported_seq":1910,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315685+0000","last_change":"2024-03-11T21:49:51.843063+0000","last_active":"2024-03-11T21:50:30.315685+0000","last_peered":"2024-03-11T21:50:30.315685+0000","last_clean":"2024-03-11T21:50:30.315685+0000","last_became_active":"2024-03-11T21:31:37.408111+0000","last_became_peered":"2024-03-11T21:31:37.408111+0000","last_unstale":"2024-03-11T21:50:30.315685+0000","last_undegraded":"2024-03-11T21:50:30.315685+0000","last_fullsized":"2024-03-11T21:50:30.315685+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"983'2","last_scrub_stamp":"2024-03-11T21:49:51.842969+0000","last_deep_scrub":"983'2","last_deep_scrub_stamp":"2024-03-11T21:46:33.835412+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.842969+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.504075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2004,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.151108+0000","last_change":"2024-03-11T21:50:00.105619+0000","last_active":"2024-03-11T21:50:31.151108+0000","last_peered":"2024-03-11T21:50:31.151108+0000","last_clean":"2024-03-11T21:50:31.151108+0000","last_became_active":"2024-03-11T21:23:09.158834+0000","last_became_peered":"2024-03-11T21:23:09.158834+0000","last_unstale":"2024-03-11T21:50:31.151108+0000","last_undegraded":"2024-03-11T21:50:31.151108+0000","last_fullsized":"2024-03-11T21:50:31.151108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:00.105404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:40.294143+0000","last_clean_scrub_stamp":"2024-03-11T21:50:00.105404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:27.224071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"98'128","reported_seq":1241,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443819+0000","last_change":"2024-03-11T21:49:03.530476+0000","last_active":"2024-03-11T21:50:30.443819+0000","last_peered":"2024-03-11T21:50:30.443819+0000","last_clean":"2024-03-11T21:50:30.443819+0000","last_became_active":"2024-03-11T21:43:58.775104+0000","last_became_peered":"2024-03-11T21:43:58.775104+0000","last_unstale":"2024-03-11T21:50:30.443819+0000","last_undegraded":"2024-03-11T21:50:30.443819+0000","last_fullsized":"2024-03-11T21:50:30.443819+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:03.530251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.530251+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:33.150287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1824,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863701+0000","last_change":"2024-03-11T21:50:22.875436+0000","last_active":"2024-03-11T21:50:30.863701+0000","last_peered":"2024-03-11T21:50:30.863701+0000","last_clean":"2024-03-11T21:50:30.863701+0000","last_became_active":"2024-03-11T21:31:37.404538+0000","last_became_peered":"2024-03-11T21:31:37.404538+0000","last_unstale":"2024-03-11T21:50:30.863701+0000","last_undegraded":"2024-03-11T21:50:30.863701+0000","last_fullsized":"2024-03-11T21:50:30.863701+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.875194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.875194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.808679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2007,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443559+0000","last_change":"2024-03-11T21:49:50.558097+0000","last_active":"2024-03-11T21:50:30.443559+0000","last_peered":"2024-03-11T21:50:30.443559+0000","last_clean":"2024-03-11T21:50:30.443559+0000","last_became_active":"2024-03-11T21:23:07.077070+0000","last_became_peered":"2024-03-11T21:23:07.077070+0000","last_unstale":"2024-03-11T21:50:30.443559+0000","last_undegraded":"2024-03-11T21:50:30.443559+0000","last_fullsized":"2024-03-11T21:50:30.443559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.557897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:31.689883+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.557897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:05.848781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"864'66","reported_seq":2295,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.320730+0000","last_change":"2024-03-11T21:49:35.036669+0000","last_active":"2024-03-11T21:50:30.320730+0000","last_peered":"2024-03-11T21:50:30.320730+0000","last_clean":"2024-03-11T21:50:30.320730+0000","last_became_active":"2024-03-11T21:31:37.416645+0000","last_became_peered":"2024-03-11T21:31:37.416645+0000","last_unstale":"2024-03-11T21:50:30.320730+0000","last_undegraded":"2024-03-11T21:50:30.320730+0000","last_fullsized":"2024-03-11T21:50:30.320730+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'66","last_scrub_stamp":"2024-03-11T21:49:35.036287+0000","last_deep_scrub":"864'66","last_deep_scrub_stamp":"2024-03-11T21:43:29.115409+0000","last_clean_scrub_stamp":"2024-03-11T21:49:35.036287+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:38.694468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1971,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863969+0000","last_change":"2024-03-11T21:49:01.178730+0000","last_active":"2024-03-11T21:50:30.863969+0000","last_peered":"2024-03-11T21:50:30.863969+0000","last_clean":"2024-03-11T21:50:30.863969+0000","last_became_active":"2024-03-11T21:23:02.939343+0000","last_became_peered":"2024-03-11T21:23:02.939343+0000","last_unstale":"2024-03-11T21:50:30.863969+0000","last_undegraded":"2024-03-11T21:50:30.863969+0000","last_fullsized":"2024-03-11T21:50:30.863969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:01.178360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:01.178360+0000","last_clean_scrub_stamp":"2024-03-11T21:49:01.178360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1849,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.317231+0000","last_change":"2024-03-11T21:49:23.963933+0000","last_active":"2024-03-11T21:50:30.317231+0000","last_peered":"2024-03-11T21:50:30.317231+0000","last_clean":"2024-03-11T21:50:30.317231+0000","last_became_active":"2024-03-11T21:31:37.411697+0000","last_became_peered":"2024-03-11T21:31:37.411697+0000","last_unstale":"2024-03-11T21:50:30.317231+0000","last_undegraded":"2024-03-11T21:50:30.317231+0000","last_fullsized":"2024-03-11T21:50:30.317231+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:23.963634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:17.364997+0000","last_clean_scrub_stamp":"2024-03-11T21:49:23.963634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.821931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"98'128","reported_seq":1236,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316025+0000","last_change":"2024-03-11T21:49:59.842316+0000","last_active":"2024-03-11T21:50:30.316025+0000","last_peered":"2024-03-11T21:50:30.316025+0000","last_clean":"2024-03-11T21:50:30.316025+0000","last_became_active":"2024-03-11T21:43:58.770127+0000","last_became_peered":"2024-03-11T21:43:58.770127+0000","last_unstale":"2024-03-11T21:50:30.316025+0000","last_undegraded":"2024-03-11T21:50:30.316025+0000","last_fullsized":"2024-03-11T21:50:30.316025+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:59.842082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.842082+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.761687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1835,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314175+0000","last_change":"2024-03-11T21:49:50.290565+0000","last_active":"2024-03-11T21:50:30.314175+0000","last_peered":"2024-03-11T21:50:30.314175+0000","last_clean":"2024-03-11T21:50:30.314175+0000","last_became_active":"2024-03-11T21:28:51.438664+0000","last_became_peered":"2024-03-11T21:28:51.438664+0000","last_unstale":"2024-03-11T21:50:30.314175+0000","last_undegraded":"2024-03-11T21:50:30.314175+0000","last_fullsized":"2024-03-11T21:50:30.314175+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.290404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:59.482460+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.290404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:03.873876+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1777,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315923+0000","last_change":"2024-03-11T21:49:24.077836+0000","last_active":"2024-03-11T21:50:30.315923+0000","last_peered":"2024-03-11T21:50:30.315923+0000","last_clean":"2024-03-11T21:50:30.315923+0000","last_became_active":"2024-03-11T21:48:12.932562+0000","last_became_peered":"2024-03-11T21:48:12.932562+0000","last_unstale":"2024-03-11T21:50:30.315923+0000","last_undegraded":"2024-03-11T21:50:30.315923+0000","last_fullsized":"2024-03-11T21:50:30.315923+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:24.077600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.077600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:48.998017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1980,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861860+0000","last_change":"2024-03-11T21:49:03.187461+0000","last_active":"2024-03-11T21:50:30.861860+0000","last_peered":"2024-03-11T21:50:30.861860+0000","last_clean":"2024-03-11T21:50:30.861860+0000","last_became_active":"2024-03-11T21:31:37.401157+0000","last_became_peered":"2024-03-11T21:31:37.401157+0000","last_unstale":"2024-03-11T21:50:30.861860+0000","last_undegraded":"2024-03-11T21:50:30.861860+0000","last_fullsized":"2024-03-11T21:50:30.861860+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:03.187356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.187356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"926'49","reported_seq":2057,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312626+0000","last_change":"2024-03-11T21:49:28.217569+0000","last_active":"2024-03-11T21:50:30.312626+0000","last_peered":"2024-03-11T21:50:30.312626+0000","last_clean":"2024-03-11T21:50:30.312626+0000","last_became_active":"2024-03-11T21:23:04.938663+0000","last_became_peered":"2024-03-11T21:23:04.938663+0000","last_unstale":"2024-03-11T21:50:30.312626+0000","last_undegraded":"2024-03-11T21:50:30.312626+0000","last_fullsized":"2024-03-11T21:50:30.312626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'49","last_scrub_stamp":"2024-03-11T21:49:28.217321+0000","last_deep_scrub":"926'49","last_deep_scrub_stamp":"2024-03-11T21:46:31.308112+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.217321+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:40.738379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1859,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.320131+0000","last_change":"2024-03-11T21:50:03.871116+0000","last_active":"2024-03-11T21:50:30.320131+0000","last_peered":"2024-03-11T21:50:30.320131+0000","last_clean":"2024-03-11T21:50:30.320131+0000","last_became_active":"2024-03-11T21:31:37.395110+0000","last_became_peered":"2024-03-11T21:31:37.395110+0000","last_unstale":"2024-03-11T21:50:30.320131+0000","last_undegraded":"2024-03-11T21:50:30.320131+0000","last_fullsized":"2024-03-11T21:50:30.320131+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.870883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.870883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.896740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1818,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411195+0000","last_change":"2024-03-11T21:49:49.688339+0000","last_active":"2024-03-11T21:50:30.411195+0000","last_peered":"2024-03-11T21:50:30.411195+0000","last_clean":"2024-03-11T21:50:30.411195+0000","last_became_active":"2024-03-11T21:28:49.938942+0000","last_became_peered":"2024-03-11T21:28:49.938942+0000","last_unstale":"2024-03-11T21:50:30.411195+0000","last_undegraded":"2024-03-11T21:50:30.411195+0000","last_fullsized":"2024-03-11T21:50:30.411195+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:49.688127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:49.688127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.278794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1963,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863576+0000","last_change":"2024-03-11T21:49:43.140282+0000","last_active":"2024-03-11T21:50:30.863576+0000","last_peered":"2024-03-11T21:50:30.863576+0000","last_clean":"2024-03-11T21:50:30.863576+0000","last_became_active":"2024-03-11T21:23:07.086925+0000","last_became_peered":"2024-03-11T21:23:07.086925+0000","last_unstale":"2024-03-11T21:50:30.863576+0000","last_undegraded":"2024-03-11T21:50:30.863576+0000","last_fullsized":"2024-03-11T21:50:30.863576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:43.140051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.140051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:51.757216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'128","reported_seq":2254,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861054+0000","last_change":"2024-03-11T21:48:47.199220+0000","last_active":"2024-03-11T21:50:30.861054+0000","last_peered":"2024-03-11T21:50:30.861054+0000","last_clean":"2024-03-11T21:50:30.861054+0000","last_became_active":"2024-03-11T21:43:58.769784+0000","last_became_peered":"2024-03-11T21:43:58.769784+0000","last_unstale":"2024-03-11T21:50:30.861054+0000","last_undegraded":"2024-03-11T21:50:30.861054+0000","last_fullsized":"2024-03-11T21:50:30.861054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:47.198992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:48:47.198992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1785,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313743+0000","last_change":"2024-03-11T21:50:01.225236+0000","last_active":"2024-03-11T21:50:30.313743+0000","last_peered":"2024-03-11T21:50:30.313743+0000","last_clean":"2024-03-11T21:50:30.313743+0000","last_became_active":"2024-03-11T21:48:12.927553+0000","last_became_peered":"2024-03-11T21:48:12.927553+0000","last_unstale":"2024-03-11T21:50:30.313743+0000","last_undegraded":"2024-03-11T21:50:30.313743+0000","last_fullsized":"2024-03-11T21:50:30.313743+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:01.225015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.182269+0000","last_clean_scrub_stamp":"2024-03-11T21:50:01.225015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.013233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"1007'35","reported_seq":2026,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412312+0000","last_change":"2024-03-11T21:50:25.999140+0000","last_active":"2024-03-11T21:50:30.412312+0000","last_peered":"2024-03-11T21:50:30.412312+0000","last_clean":"2024-03-11T21:50:30.412312+0000","last_became_active":"2024-03-11T21:23:04.937792+0000","last_became_peered":"2024-03-11T21:23:04.937792+0000","last_unstale":"2024-03-11T21:50:30.412312+0000","last_undegraded":"2024-03-11T21:50:30.412312+0000","last_fullsized":"2024-03-11T21:50:30.412312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"1007'35","last_scrub_stamp":"2024-03-11T21:50:25.999030+0000","last_deep_scrub":"1007'35","last_deep_scrub_stamp":"2024-03-11T21:48:42.476181+0000","last_clean_scrub_stamp":"2024-03-11T21:50:25.999030+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:40.727929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":790,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314344+0000","last_change":"2024-03-11T21:50:22.487763+0000","last_active":"2024-03-11T21:50:30.314344+0000","last_peered":"2024-03-11T21:50:30.314344+0000","last_clean":"2024-03-11T21:50:30.314344+0000","last_became_active":"2024-03-11T21:38:59.323409+0000","last_became_peered":"2024-03-11T21:38:59.323409+0000","last_unstale":"2024-03-11T21:50:30.314344+0000","last_undegraded":"2024-03-11T21:50:30.314344+0000","last_fullsized":"2024-03-11T21:50:30.314344+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.487542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:57.751652+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.487542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.594656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"98'128","reported_seq":2261,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861117+0000","last_change":"2024-03-11T21:48:53.224541+0000","last_active":"2024-03-11T21:50:30.861117+0000","last_peered":"2024-03-11T21:50:30.861117+0000","last_clean":"2024-03-11T21:50:30.861117+0000","last_became_active":"2024-03-11T21:43:58.771356+0000","last_became_peered":"2024-03-11T21:43:58.771356+0000","last_unstale":"2024-03-11T21:50:30.861117+0000","last_undegraded":"2024-03-11T21:50:30.861117+0000","last_fullsized":"2024-03-11T21:50:30.861117+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:48:53.224299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.224299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1631,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314357+0000","last_change":"2024-03-11T21:49:37.011325+0000","last_active":"2024-03-11T21:50:30.314357+0000","last_peered":"2024-03-11T21:50:30.314357+0000","last_clean":"2024-03-11T21:50:30.314357+0000","last_became_active":"2024-03-11T21:48:12.934652+0000","last_became_peered":"2024-03-11T21:48:12.934652+0000","last_unstale":"2024-03-11T21:50:30.314357+0000","last_undegraded":"2024-03-11T21:50:30.314357+0000","last_fullsized":"2024-03-11T21:50:30.314357+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:37.010995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:37.010995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.787561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":2019,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.431380+0000","last_change":"2024-03-11T21:49:53.416963+0000","last_active":"2024-03-11T21:50:31.431380+0000","last_peered":"2024-03-11T21:50:31.431380+0000","last_clean":"2024-03-11T21:50:31.431380+0000","last_became_active":"2024-03-11T21:23:07.077842+0000","last_became_peered":"2024-03-11T21:23:07.077842+0000","last_unstale":"2024-03-11T21:50:31.431380+0000","last_undegraded":"2024-03-11T21:50:31.431380+0000","last_fullsized":"2024-03-11T21:50:31.431380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.416759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.416759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.187572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2036,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.429256+0000","last_change":"2024-03-11T21:50:11.467696+0000","last_active":"2024-03-11T21:50:31.429256+0000","last_peered":"2024-03-11T21:50:31.429256+0000","last_clean":"2024-03-11T21:50:31.429256+0000","last_became_active":"2024-03-11T21:23:02.931522+0000","last_became_peered":"2024-03-11T21:23:02.931522+0000","last_unstale":"2024-03-11T21:50:31.429256+0000","last_undegraded":"2024-03-11T21:50:31.429256+0000","last_fullsized":"2024-03-11T21:50:31.429256+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.467429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.467429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.076952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"864'50","reported_seq":1959,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316686+0000","last_change":"2024-03-11T21:49:41.983620+0000","last_active":"2024-03-11T21:50:30.316686+0000","last_peered":"2024-03-11T21:50:30.316686+0000","last_clean":"2024-03-11T21:50:30.316686+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:50:30.316686+0000","last_undegraded":"2024-03-11T21:50:30.316686+0000","last_fullsized":"2024-03-11T21:50:30.316686+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'50","last_scrub_stamp":"2024-03-11T21:49:41.983506+0000","last_deep_scrub":"149'26","last_deep_scrub_stamp":"2024-03-11T21:29:02.205088+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.983506+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.654046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1959,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863646+0000","last_change":"2024-03-11T21:50:06.000020+0000","last_active":"2024-03-11T21:50:30.863646+0000","last_peered":"2024-03-11T21:50:30.863646+0000","last_clean":"2024-03-11T21:50:30.863646+0000","last_became_active":"2024-03-11T21:23:09.156135+0000","last_became_peered":"2024-03-11T21:23:09.156135+0000","last_unstale":"2024-03-11T21:50:30.863646+0000","last_undegraded":"2024-03-11T21:50:30.863646+0000","last_fullsized":"2024-03-11T21:50:30.863646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:05.999714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:05.999714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.385337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"98'128","reported_seq":1025,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312593+0000","last_change":"2024-03-11T21:50:24.397240+0000","last_active":"2024-03-11T21:50:30.312593+0000","last_peered":"2024-03-11T21:50:30.312593+0000","last_clean":"2024-03-11T21:50:30.312593+0000","last_became_active":"2024-03-11T21:48:12.927578+0000","last_became_peered":"2024-03-11T21:48:12.927578+0000","last_unstale":"2024-03-11T21:50:30.312593+0000","last_undegraded":"2024-03-11T21:50:30.312593+0000","last_fullsized":"2024-03-11T21:50:30.312593+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":4,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:24.397008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:24.397008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.483635+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1836,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312162+0000","last_change":"2024-03-11T21:49:04.236699+0000","last_active":"2024-03-11T21:50:30.312162+0000","last_peered":"2024-03-11T21:50:30.312162+0000","last_clean":"2024-03-11T21:50:30.312162+0000","last_became_active":"2024-03-11T21:28:51.395824+0000","last_became_peered":"2024-03-11T21:28:51.395824+0000","last_unstale":"2024-03-11T21:50:30.312162+0000","last_undegraded":"2024-03-11T21:50:30.312162+0000","last_fullsized":"2024-03-11T21:50:30.312162+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:04.236547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:01.549350+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.236547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:32.808872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1992,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314367+0000","last_change":"2024-03-11T21:49:40.179039+0000","last_active":"2024-03-11T21:50:30.314367+0000","last_peered":"2024-03-11T21:50:30.314367+0000","last_clean":"2024-03-11T21:50:30.314367+0000","last_became_active":"2024-03-11T21:31:37.429164+0000","last_became_peered":"2024-03-11T21:31:37.429164+0000","last_unstale":"2024-03-11T21:50:30.314367+0000","last_undegraded":"2024-03-11T21:50:30.314367+0000","last_fullsized":"2024-03-11T21:50:30.314367+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:40.178759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:40.178759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:47.369656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"864'47","reported_seq":2086,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.151753+0000","last_change":"2024-03-11T21:50:28.187253+0000","last_active":"2024-03-11T21:50:31.151753+0000","last_peered":"2024-03-11T21:50:31.151753+0000","last_clean":"2024-03-11T21:50:31.151753+0000","last_became_active":"2024-03-11T21:23:04.936976+0000","last_became_peered":"2024-03-11T21:23:04.936976+0000","last_unstale":"2024-03-11T21:50:31.151753+0000","last_undegraded":"2024-03-11T21:50:31.151753+0000","last_fullsized":"2024-03-11T21:50:31.151753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'47","last_scrub_stamp":"2024-03-11T21:50:28.186911+0000","last_deep_scrub":"864'47","last_deep_scrub_stamp":"2024-03-11T21:50:28.186911+0000","last_clean_scrub_stamp":"2024-03-11T21:50:28.186911+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:38.782451+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1794,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314787+0000","last_change":"2024-03-11T21:50:16.286535+0000","last_active":"2024-03-11T21:50:30.314787+0000","last_peered":"2024-03-11T21:50:30.314787+0000","last_clean":"2024-03-11T21:50:30.314787+0000","last_became_active":"2024-03-11T21:48:12.932548+0000","last_became_peered":"2024-03-11T21:48:12.932548+0000","last_unstale":"2024-03-11T21:50:30.314787+0000","last_undegraded":"2024-03-11T21:50:30.314787+0000","last_fullsized":"2024-03-11T21:50:30.314787+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.286316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.286316+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.286316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.316124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1959,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413745+0000","last_change":"2024-03-11T21:49:25.715777+0000","last_active":"2024-03-11T21:50:30.413745+0000","last_peered":"2024-03-11T21:50:30.413745+0000","last_clean":"2024-03-11T21:50:30.413745+0000","last_became_active":"2024-03-11T21:23:09.149604+0000","last_became_peered":"2024-03-11T21:23:09.149604+0000","last_unstale":"2024-03-11T21:50:30.413745+0000","last_undegraded":"2024-03-11T21:50:30.413745+0000","last_fullsized":"2024-03-11T21:50:30.413745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.715480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.715480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:32.761912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"98'128","reported_seq":1249,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444444+0000","last_change":"2024-03-11T21:49:35.643788+0000","last_active":"2024-03-11T21:50:30.444444+0000","last_peered":"2024-03-11T21:50:30.444444+0000","last_clean":"2024-03-11T21:50:30.444444+0000","last_became_active":"2024-03-11T21:43:58.778559+0000","last_became_peered":"2024-03-11T21:43:58.778559+0000","last_unstale":"2024-03-11T21:50:30.444444+0000","last_undegraded":"2024-03-11T21:50:30.444444+0000","last_fullsized":"2024-03-11T21:50:30.444444+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":593,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":3,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:49:35.643555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:49:35.643555+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:57.984275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1630,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315120+0000","last_change":"2024-03-11T21:49:25.103902+0000","last_active":"2024-03-11T21:50:30.315120+0000","last_peered":"2024-03-11T21:50:30.315120+0000","last_clean":"2024-03-11T21:50:30.315120+0000","last_became_active":"2024-03-11T21:48:12.931225+0000","last_became_peered":"2024-03-11T21:48:12.931225+0000","last_unstale":"2024-03-11T21:50:30.315120+0000","last_undegraded":"2024-03-11T21:50:30.315120+0000","last_fullsized":"2024-03-11T21:50:30.315120+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.103690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:13.261641+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.103690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:50.436849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"124'12","reported_seq":2335,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314603+0000","last_change":"2024-03-11T21:49:52.285519+0000","last_active":"2024-03-11T21:50:30.314603+0000","last_peered":"2024-03-11T21:50:30.314603+0000","last_clean":"2024-03-11T21:50:30.314603+0000","last_became_active":"2024-03-11T21:23:07.090419+0000","last_became_peered":"2024-03-11T21:23:07.090419+0000","last_unstale":"2024-03-11T21:50:30.314603+0000","last_undegraded":"2024-03-11T21:50:30.314603+0000","last_fullsized":"2024-03-11T21:50:30.314603+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'12","last_scrub_stamp":"2024-03-11T21:49:52.285328+0000","last_deep_scrub":"124'12","last_deep_scrub_stamp":"2024-03-11T21:49:52.285328+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.285328+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.928958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1012'3","reported_seq":1887,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316526+0000","last_change":"2024-03-11T21:49:17.955794+0000","last_active":"2024-03-11T21:50:30.316526+0000","last_peered":"2024-03-11T21:50:30.316526+0000","last_clean":"2024-03-11T21:50:30.316526+0000","last_became_active":"2024-03-11T21:31:37.394475+0000","last_became_peered":"2024-03-11T21:31:37.394475+0000","last_unstale":"2024-03-11T21:50:30.316526+0000","last_undegraded":"2024-03-11T21:50:30.316526+0000","last_fullsized":"2024-03-11T21:50:30.316526+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"1012'3","last_scrub_stamp":"2024-03-11T21:49:17.955691+0000","last_deep_scrub":"1012'3","last_deep_scrub_stamp":"2024-03-11T21:49:17.955691+0000","last_clean_scrub_stamp":"2024-03-11T21:49:17.955691+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.546696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00107047,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"1003'42","reported_seq":2115,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314530+0000","last_change":"2024-03-11T21:49:41.109831+0000","last_active":"2024-03-11T21:50:30.314530+0000","last_peered":"2024-03-11T21:50:30.314530+0000","last_clean":"2024-03-11T21:50:30.314530+0000","last_became_active":"2024-03-11T21:48:12.928283+0000","last_became_peered":"2024-03-11T21:48:12.928283+0000","last_unstale":"2024-03-11T21:50:30.314530+0000","last_undegraded":"2024-03-11T21:50:30.314530+0000","last_fullsized":"2024-03-11T21:50:30.314530+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'42","last_scrub_stamp":"2024-03-11T21:49:41.109586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.109586+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.385002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1857,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.319783+0000","last_change":"2024-03-11T21:49:39.000874+0000","last_active":"2024-03-11T21:50:30.319783+0000","last_peered":"2024-03-11T21:50:30.319783+0000","last_clean":"2024-03-11T21:50:30.319783+0000","last_became_active":"2024-03-11T21:31:37.397868+0000","last_became_peered":"2024-03-11T21:31:37.397868+0000","last_unstale":"2024-03-11T21:50:30.319783+0000","last_undegraded":"2024-03-11T21:50:30.319783+0000","last_fullsized":"2024-03-11T21:50:30.319783+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:39.000460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:22.734970+0000","last_clean_scrub_stamp":"2024-03-11T21:49:39.000460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.139821+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1986,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.311689+0000","last_change":"2024-03-11T21:50:26.528576+0000","last_active":"2024-03-11T21:50:30.311689+0000","last_peered":"2024-03-11T21:50:30.311689+0000","last_clean":"2024-03-11T21:50:30.311689+0000","last_became_active":"2024-03-11T21:23:09.161785+0000","last_became_peered":"2024-03-11T21:23:09.161785+0000","last_unstale":"2024-03-11T21:50:30.311689+0000","last_undegraded":"2024-03-11T21:50:30.311689+0000","last_fullsized":"2024-03-11T21:50:30.311689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.528352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:26.528352+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.528352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.412095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"600'129","reported_seq":1231,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412517+0000","last_change":"2024-03-11T21:50:04.822903+0000","last_active":"2024-03-11T21:50:30.412517+0000","last_peered":"2024-03-11T21:50:30.412517+0000","last_clean":"2024-03-11T21:50:30.412517+0000","last_became_active":"2024-03-11T21:44:03.855000+0000","last_became_peered":"2024-03-11T21:44:03.855000+0000","last_unstale":"2024-03-11T21:50:30.412517+0000","last_undegraded":"2024-03-11T21:50:30.412517+0000","last_fullsized":"2024-03-11T21:50:30.412517+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":4,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:50:04.822675+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:48:28.516563+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.822675+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.182142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1718,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.320123+0000","last_change":"2024-03-11T21:49:22.929666+0000","last_active":"2024-03-11T21:50:30.320123+0000","last_peered":"2024-03-11T21:50:30.320123+0000","last_clean":"2024-03-11T21:50:30.320123+0000","last_became_active":"2024-03-11T21:31:37.388510+0000","last_became_peered":"2024-03-11T21:31:37.388510+0000","last_unstale":"2024-03-11T21:50:30.320123+0000","last_undegraded":"2024-03-11T21:50:30.320123+0000","last_fullsized":"2024-03-11T21:50:30.320123+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:22.929444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:22.929444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:51.054329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2028,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.429845+0000","last_change":"2024-03-11T21:50:10.433905+0000","last_active":"2024-03-11T21:50:31.429845+0000","last_peered":"2024-03-11T21:50:31.429845+0000","last_clean":"2024-03-11T21:50:31.429845+0000","last_became_active":"2024-03-11T21:31:37.404786+0000","last_became_peered":"2024-03-11T21:31:37.404786+0000","last_unstale":"2024-03-11T21:50:31.429845+0000","last_undegraded":"2024-03-11T21:50:31.429845+0000","last_fullsized":"2024-03-11T21:50:31.429845+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.433796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:32.657602+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.433796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.306140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"864'32","reported_seq":2031,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.864448+0000","last_change":"2024-03-11T21:49:52.047520+0000","last_active":"2024-03-11T21:50:30.864448+0000","last_peered":"2024-03-11T21:50:30.864448+0000","last_clean":"2024-03-11T21:50:30.864448+0000","last_became_active":"2024-03-11T21:31:37.404980+0000","last_became_peered":"2024-03-11T21:31:37.404980+0000","last_unstale":"2024-03-11T21:50:30.864448+0000","last_undegraded":"2024-03-11T21:50:30.864448+0000","last_fullsized":"2024-03-11T21:50:30.864448+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:49:52.047270+0000","last_deep_scrub":"389'24","last_deep_scrub_stamp":"2024-03-11T21:33:31.683216+0000","last_clean_scrub_stamp":"2024-03-11T21:49:52.047270+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.978245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"121'4","reported_seq":2033,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.149854+0000","last_change":"2024-03-11T21:49:42.148026+0000","last_active":"2024-03-11T21:50:31.149854+0000","last_peered":"2024-03-11T21:50:31.149854+0000","last_clean":"2024-03-11T21:50:31.149854+0000","last_became_active":"2024-03-11T21:23:02.939859+0000","last_became_peered":"2024-03-11T21:23:02.939859+0000","last_unstale":"2024-03-11T21:50:31.149854+0000","last_undegraded":"2024-03-11T21:50:31.149854+0000","last_fullsized":"2024-03-11T21:50:31.149854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'4","last_scrub_stamp":"2024-03-11T21:49:42.147709+0000","last_deep_scrub":"121'4","last_deep_scrub_stamp":"2024-03-11T21:49:42.147709+0000","last_clean_scrub_stamp":"2024-03-11T21:49:42.147709+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:01.011940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1976,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.866498+0000","last_change":"2024-03-11T21:50:30.866621+0000","last_active":"2024-03-11T21:50:30.866498+0000","last_peered":"2024-03-11T21:50:30.866498+0000","last_clean":"2024-03-11T21:50:30.866498+0000","last_became_active":"2024-03-11T21:31:37.405786+0000","last_became_peered":"2024-03-11T21:31:37.405786+0000","last_unstale":"2024-03-11T21:50:30.866498+0000","last_undegraded":"2024-03-11T21:50:30.866498+0000","last_fullsized":"2024-03-11T21:50:30.866498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:30.866321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.866321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:40.250852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"98'128","reported_seq":1242,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315689+0000","last_change":"2024-03-11T21:50:27.861827+0000","last_active":"2024-03-11T21:50:30.315689+0000","last_peered":"2024-03-11T21:50:30.315689+0000","last_clean":"2024-03-11T21:50:30.315689+0000","last_became_active":"2024-03-11T21:44:03.854936+0000","last_became_peered":"2024-03-11T21:44:03.854936+0000","last_unstale":"2024-03-11T21:50:30.315689+0000","last_undegraded":"2024-03-11T21:50:30.315689+0000","last_fullsized":"2024-03-11T21:50:30.315689+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:27.861423+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:50:27.861423+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.861423+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.953952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1824,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:32.804204+0000","last_change":"2024-03-11T21:50:32.804263+0000","last_active":"2024-03-11T21:50:32.804204+0000","last_peered":"2024-03-11T21:50:32.804204+0000","last_clean":"2024-03-11T21:50:32.804204+0000","last_became_active":"2024-03-11T21:28:49.948174+0000","last_became_peered":"2024-03-11T21:28:49.948174+0000","last_unstale":"2024-03-11T21:50:32.804204+0000","last_undegraded":"2024-03-11T21:50:32.804204+0000","last_fullsized":"2024-03-11T21:50:32.804204+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:32.804048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:50.308128+0000","last_clean_scrub_stamp":"2024-03-11T21:50:32.804048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:49.548046+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"864'27","reported_seq":2008,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861943+0000","last_change":"2024-03-11T21:49:28.165004+0000","last_active":"2024-03-11T21:50:30.861943+0000","last_peered":"2024-03-11T21:50:30.861943+0000","last_clean":"2024-03-11T21:50:30.861943+0000","last_became_active":"2024-03-11T21:23:04.935196+0000","last_became_peered":"2024-03-11T21:23:04.935196+0000","last_unstale":"2024-03-11T21:50:30.861943+0000","last_undegraded":"2024-03-11T21:50:30.861943+0000","last_fullsized":"2024-03-11T21:50:30.861943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'27","last_scrub_stamp":"2024-03-11T21:49:28.164728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.164728+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:42.889100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2043,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.429739+0000","last_change":"2024-03-11T21:50:12.499821+0000","last_active":"2024-03-11T21:50:31.429739+0000","last_peered":"2024-03-11T21:50:31.429739+0000","last_clean":"2024-03-11T21:50:31.429739+0000","last_became_active":"2024-03-11T21:31:37.404254+0000","last_became_peered":"2024-03-11T21:31:37.404254+0000","last_unstale":"2024-03-11T21:50:31.429739+0000","last_undegraded":"2024-03-11T21:50:31.429739+0000","last_fullsized":"2024-03-11T21:50:31.429739+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.499608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:34.793536+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.499608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.592309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1971,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.414558+0000","last_change":"2024-03-11T21:49:23.688584+0000","last_active":"2024-03-11T21:50:30.414558+0000","last_peered":"2024-03-11T21:50:30.414558+0000","last_clean":"2024-03-11T21:50:30.414558+0000","last_became_active":"2024-03-11T21:31:37.406613+0000","last_became_peered":"2024-03-11T21:31:37.406613+0000","last_unstale":"2024-03-11T21:50:30.414558+0000","last_undegraded":"2024-03-11T21:50:30.414558+0000","last_fullsized":"2024-03-11T21:50:30.414558+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:23.688354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:10.163069+0000","last_clean_scrub_stamp":"2024-03-11T21:49:23.688354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2022,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.431655+0000","last_change":"2024-03-11T21:49:38.519296+0000","last_active":"2024-03-11T21:50:31.431655+0000","last_peered":"2024-03-11T21:50:31.431655+0000","last_clean":"2024-03-11T21:50:31.431655+0000","last_became_active":"2024-03-11T21:23:09.151526+0000","last_became_peered":"2024-03-11T21:23:09.151526+0000","last_unstale":"2024-03-11T21:50:31.431655+0000","last_undegraded":"2024-03-11T21:50:31.431655+0000","last_fullsized":"2024-03-11T21:50:31.431655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:38.519082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:04.588570+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.519082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.967686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"98'128","reported_seq":397,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313096+0000","last_change":"2024-03-11T21:50:21.450081+0000","last_active":"2024-03-11T21:50:30.313096+0000","last_peered":"2024-03-11T21:50:30.313096+0000","last_clean":"2024-03-11T21:50:30.313096+0000","last_became_active":"2024-03-11T21:48:12.931083+0000","last_became_peered":"2024-03-11T21:48:12.931083+0000","last_unstale":"2024-03-11T21:50:30.313096+0000","last_undegraded":"2024-03-11T21:50:30.313096+0000","last_fullsized":"2024-03-11T21:50:30.313096+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:21.449847+0000","last_deep_scrub":"98'128","last_deep_scrub_stamp":"2024-03-11T21:50:21.449847+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.449847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.883373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1816,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412850+0000","last_change":"2024-03-11T21:49:50.730394+0000","last_active":"2024-03-11T21:50:30.412850+0000","last_peered":"2024-03-11T21:50:30.412850+0000","last_clean":"2024-03-11T21:50:30.412850+0000","last_became_active":"2024-03-11T21:28:49.955717+0000","last_became_peered":"2024-03-11T21:28:49.955717+0000","last_unstale":"2024-03-11T21:50:30.412850+0000","last_undegraded":"2024-03-11T21:50:30.412850+0000","last_fullsized":"2024-03-11T21:50:30.412850+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:50.730256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:50.730256+0000","last_clean_scrub_stamp":"2024-03-11T21:49:50.730256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.092968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"864'14","reported_seq":2037,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.442672+0000","last_change":"2024-03-11T21:50:07.575851+0000","last_active":"2024-03-11T21:50:30.442672+0000","last_peered":"2024-03-11T21:50:30.442672+0000","last_clean":"2024-03-11T21:50:30.442672+0000","last_became_active":"2024-03-11T21:23:04.936937+0000","last_became_peered":"2024-03-11T21:23:04.936937+0000","last_unstale":"2024-03-11T21:50:30.442672+0000","last_undegraded":"2024-03-11T21:50:30.442672+0000","last_fullsized":"2024-03-11T21:50:30.442672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'14","last_scrub_stamp":"2024-03-11T21:50:07.575613+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-03-11T21:25:46.921404+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.575613+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:37.545349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":2022,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443297+0000","last_change":"2024-03-11T21:50:25.647633+0000","last_active":"2024-03-11T21:50:30.443297+0000","last_peered":"2024-03-11T21:50:30.443297+0000","last_clean":"2024-03-11T21:50:30.443297+0000","last_became_active":"2024-03-11T21:31:37.401594+0000","last_became_peered":"2024-03-11T21:31:37.401594+0000","last_unstale":"2024-03-11T21:50:30.443297+0000","last_undegraded":"2024-03-11T21:50:30.443297+0000","last_fullsized":"2024-03-11T21:50:30.443297+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:25.647430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:42:05.515038+0000","last_clean_scrub_stamp":"2024-03-11T21:50:25.647430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.476252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1783,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315538+0000","last_change":"2024-03-11T21:49:47.156418+0000","last_active":"2024-03-11T21:50:30.315538+0000","last_peered":"2024-03-11T21:50:30.315538+0000","last_clean":"2024-03-11T21:50:30.315538+0000","last_became_active":"2024-03-11T21:48:12.934954+0000","last_became_peered":"2024-03-11T21:48:12.934954+0000","last_unstale":"2024-03-11T21:50:30.315538+0000","last_undegraded":"2024-03-11T21:50:30.315538+0000","last_fullsized":"2024-03-11T21:50:30.315538+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.156143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:42.239053+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.156143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.148519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1957,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863635+0000","last_change":"2024-03-11T21:49:36.132619+0000","last_active":"2024-03-11T21:50:30.863635+0000","last_peered":"2024-03-11T21:50:30.863635+0000","last_clean":"2024-03-11T21:50:30.863635+0000","last_became_active":"2024-03-11T21:23:09.142481+0000","last_became_peered":"2024-03-11T21:23:09.142481+0000","last_unstale":"2024-03-11T21:50:30.863635+0000","last_undegraded":"2024-03-11T21:50:30.863635+0000","last_fullsized":"2024-03-11T21:50:30.863635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:36.132415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:08.094891+0000","last_clean_scrub_stamp":"2024-03-11T21:49:36.132415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.867203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"98'128","reported_seq":627,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:32.566031+0000","last_change":"2024-03-11T21:50:32.566102+0000","last_active":"2024-03-11T21:50:32.566031+0000","last_peered":"2024-03-11T21:50:32.566031+0000","last_clean":"2024-03-11T21:50:32.566031+0000","last_became_active":"2024-03-11T21:44:03.859125+0000","last_became_peered":"2024-03-11T21:44:03.859125+0000","last_unstale":"2024-03-11T21:50:32.566031+0000","last_undegraded":"2024-03-11T21:50:32.566031+0000","last_fullsized":"2024-03-11T21:50:32.566031+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:32.565812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:32.565812+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:01.196103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1817,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413257+0000","last_change":"2024-03-11T21:49:47.713717+0000","last_active":"2024-03-11T21:50:30.413257+0000","last_peered":"2024-03-11T21:50:30.413257+0000","last_clean":"2024-03-11T21:50:30.413257+0000","last_became_active":"2024-03-11T21:28:49.938018+0000","last_became_peered":"2024-03-11T21:28:49.938018+0000","last_unstale":"2024-03-11T21:50:30.413257+0000","last_undegraded":"2024-03-11T21:50:30.413257+0000","last_fullsized":"2024-03-11T21:50:30.413257+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.713373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.713373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.603247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"864'27","reported_seq":2016,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863199+0000","last_change":"2024-03-11T21:50:20.947860+0000","last_active":"2024-03-11T21:50:30.863199+0000","last_peered":"2024-03-11T21:50:30.863199+0000","last_clean":"2024-03-11T21:50:30.863199+0000","last_became_active":"2024-03-11T21:23:05.019554+0000","last_became_peered":"2024-03-11T21:23:05.019554+0000","last_unstale":"2024-03-11T21:50:30.863199+0000","last_undegraded":"2024-03-11T21:50:30.863199+0000","last_fullsized":"2024-03-11T21:50:30.863199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'27","last_scrub_stamp":"2024-03-11T21:50:20.947606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.947606+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:41.307317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1792,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314726+0000","last_change":"2024-03-11T21:49:49.137593+0000","last_active":"2024-03-11T21:50:30.314726+0000","last_peered":"2024-03-11T21:50:30.314726+0000","last_clean":"2024-03-11T21:50:30.314726+0000","last_became_active":"2024-03-11T21:48:12.919525+0000","last_became_peered":"2024-03-11T21:48:12.919525+0000","last_unstale":"2024-03-11T21:50:30.314726+0000","last_undegraded":"2024-03-11T21:50:30.314726+0000","last_fullsized":"2024-03-11T21:50:30.314726+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:49.137381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:49.137381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:50.130222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1003'15","reported_seq":2070,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315110+0000","last_change":"2024-03-11T21:50:29.404096+0000","last_active":"2024-03-11T21:50:30.315110+0000","last_peered":"2024-03-11T21:50:30.315110+0000","last_clean":"2024-03-11T21:50:30.315110+0000","last_became_active":"2024-03-11T21:48:12.931718+0000","last_became_peered":"2024-03-11T21:48:12.931718+0000","last_unstale":"2024-03-11T21:50:30.315110+0000","last_undegraded":"2024-03-11T21:50:30.315110+0000","last_fullsized":"2024-03-11T21:50:30.315110+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'15","last_scrub_stamp":"2024-03-11T21:50:29.403779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:50:29.403779+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:57.271791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1985,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315221+0000","last_change":"2024-03-11T21:50:04.449328+0000","last_active":"2024-03-11T21:50:30.315221+0000","last_peered":"2024-03-11T21:50:30.315221+0000","last_clean":"2024-03-11T21:50:30.315221+0000","last_became_active":"2024-03-11T21:23:09.150999+0000","last_became_peered":"2024-03-11T21:23:09.150999+0000","last_unstale":"2024-03-11T21:50:30.315221+0000","last_undegraded":"2024-03-11T21:50:30.315221+0000","last_fullsized":"2024-03-11T21:50:30.315221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:04.449094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:49.769830+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.449094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.520679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"98'128","reported_seq":397,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313852+0000","last_change":"2024-03-11T21:50:04.167480+0000","last_active":"2024-03-11T21:50:30.313852+0000","last_peered":"2024-03-11T21:50:30.313852+0000","last_clean":"2024-03-11T21:50:30.313852+0000","last_became_active":"2024-03-11T21:48:12.918001+0000","last_became_peered":"2024-03-11T21:48:12.918001+0000","last_unstale":"2024-03-11T21:50:30.313852+0000","last_undegraded":"2024-03-11T21:50:30.313852+0000","last_fullsized":"2024-03-11T21:50:30.313852+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'128","last_scrub_stamp":"2024-03-11T21:50:04.167215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:08.345892+0000","last_clean_scrub_stamp":"2024-03-11T21:50:04.167215+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.453687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1866,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443225+0000","last_change":"2024-03-11T21:50:13.470906+0000","last_active":"2024-03-11T21:50:30.443225+0000","last_peered":"2024-03-11T21:50:30.443225+0000","last_clean":"2024-03-11T21:50:30.443225+0000","last_became_active":"2024-03-11T21:33:59.482002+0000","last_became_peered":"2024-03-11T21:33:59.482002+0000","last_unstale":"2024-03-11T21:50:30.443225+0000","last_undegraded":"2024-03-11T21:50:30.443225+0000","last_fullsized":"2024-03-11T21:50:30.443225+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.470698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.470698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.566505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"864'25","reported_seq":2034,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312649+0000","last_change":"2024-03-11T21:50:10.417964+0000","last_active":"2024-03-11T21:50:30.312649+0000","last_peered":"2024-03-11T21:50:30.312649+0000","last_clean":"2024-03-11T21:50:30.312649+0000","last_became_active":"2024-03-11T21:23:04.946050+0000","last_became_peered":"2024-03-11T21:23:04.946050+0000","last_unstale":"2024-03-11T21:50:30.312649+0000","last_undegraded":"2024-03-11T21:50:30.312649+0000","last_fullsized":"2024-03-11T21:50:30.312649+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:50:10.417693+0000","last_deep_scrub":"116'13","last_deep_scrub_stamp":"2024-03-11T21:29:00.537639+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.417693+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.973121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1989,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313664+0000","last_change":"2024-03-11T21:49:12.155651+0000","last_active":"2024-03-11T21:50:30.313664+0000","last_peered":"2024-03-11T21:50:30.313664+0000","last_clean":"2024-03-11T21:50:30.313664+0000","last_became_active":"2024-03-11T21:23:02.943984+0000","last_became_peered":"2024-03-11T21:23:02.943984+0000","last_unstale":"2024-03-11T21:50:30.313664+0000","last_undegraded":"2024-03-11T21:50:30.313664+0000","last_fullsized":"2024-03-11T21:50:30.313664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:12.155438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:12.155438+0000","last_clean_scrub_stamp":"2024-03-11T21:49:12.155438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.870413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"122'12","reported_seq":2311,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413432+0000","last_change":"2024-03-11T21:50:27.943606+0000","last_active":"2024-03-11T21:50:30.413432+0000","last_peered":"2024-03-11T21:50:30.413432+0000","last_clean":"2024-03-11T21:50:30.413432+0000","last_became_active":"2024-03-11T21:23:07.088865+0000","last_became_peered":"2024-03-11T21:23:07.088865+0000","last_unstale":"2024-03-11T21:50:30.413432+0000","last_undegraded":"2024-03-11T21:50:30.413432+0000","last_fullsized":"2024-03-11T21:50:30.413432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'12","last_scrub_stamp":"2024-03-11T21:50:27.943479+0000","last_deep_scrub":"122'12","last_deep_scrub_stamp":"2024-03-11T21:49:00.555818+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.943479+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:53.110337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2010,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:33.202654+0000","last_change":"2024-03-11T21:50:33.202712+0000","last_active":"2024-03-11T21:50:33.202654+0000","last_peered":"2024-03-11T21:50:33.202654+0000","last_clean":"2024-03-11T21:50:33.202654+0000","last_became_active":"2024-03-11T21:23:09.151224+0000","last_became_peered":"2024-03-11T21:23:09.151224+0000","last_unstale":"2024-03-11T21:50:33.202654+0000","last_undegraded":"2024-03-11T21:50:33.202654+0000","last_fullsized":"2024-03-11T21:50:33.202654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:33.202474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:05.442133+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.202474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.899109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1706,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316601+0000","last_change":"2024-03-11T21:49:14.028737+0000","last_active":"2024-03-11T21:50:30.316601+0000","last_peered":"2024-03-11T21:50:30.316601+0000","last_clean":"2024-03-11T21:50:30.316601+0000","last_became_active":"2024-03-11T21:31:37.401227+0000","last_became_peered":"2024-03-11T21:31:37.401227+0000","last_unstale":"2024-03-11T21:50:30.316601+0000","last_undegraded":"2024-03-11T21:50:30.316601+0000","last_fullsized":"2024-03-11T21:50:30.316601+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:14.028514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:14.028514+0000","last_clean_scrub_stamp":"2024-03-11T21:49:14.028514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:34.796956+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"864'25","reported_seq":2003,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.864665+0000","last_change":"2024-03-11T21:49:31.172448+0000","last_active":"2024-03-11T21:50:30.864665+0000","last_peered":"2024-03-11T21:50:30.864665+0000","last_clean":"2024-03-11T21:50:30.864665+0000","last_became_active":"2024-03-11T21:23:04.941339+0000","last_became_peered":"2024-03-11T21:23:04.941339+0000","last_unstale":"2024-03-11T21:50:30.864665+0000","last_undegraded":"2024-03-11T21:50:30.864665+0000","last_fullsized":"2024-03-11T21:50:30.864665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:49:31.172202+0000","last_deep_scrub":"864'25","last_deep_scrub_stamp":"2024-03-11T21:49:31.172202+0000","last_clean_scrub_stamp":"2024-03-11T21:49:31.172202+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:52.148714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2021,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444188+0000","last_change":"2024-03-11T21:49:54.521581+0000","last_active":"2024-03-11T21:50:30.444188+0000","last_peered":"2024-03-11T21:50:30.444188+0000","last_clean":"2024-03-11T21:50:30.444188+0000","last_became_active":"2024-03-11T21:23:02.942975+0000","last_became_peered":"2024-03-11T21:23:02.942975+0000","last_unstale":"2024-03-11T21:50:30.444188+0000","last_undegraded":"2024-03-11T21:50:30.444188+0000","last_fullsized":"2024-03-11T21:50:30.444188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.521369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:15.412070+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.521369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:58.972862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1971,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412748+0000","last_change":"2024-03-11T21:49:19.624801+0000","last_active":"2024-03-11T21:50:30.412748+0000","last_peered":"2024-03-11T21:50:30.412748+0000","last_clean":"2024-03-11T21:50:30.412748+0000","last_became_active":"2024-03-11T21:31:37.402564+0000","last_became_peered":"2024-03-11T21:31:37.402564+0000","last_unstale":"2024-03-11T21:50:30.412748+0000","last_undegraded":"2024-03-11T21:50:30.412748+0000","last_fullsized":"2024-03-11T21:50:30.412748+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:19.624584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:32.231996+0000","last_clean_scrub_stamp":"2024-03-11T21:49:19.624584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1993,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.311590+0000","last_change":"2024-03-11T21:50:12.415639+0000","last_active":"2024-03-11T21:50:30.311590+0000","last_peered":"2024-03-11T21:50:30.311590+0000","last_clean":"2024-03-11T21:50:30.311590+0000","last_became_active":"2024-03-11T21:31:37.408591+0000","last_became_peered":"2024-03-11T21:31:37.408591+0000","last_unstale":"2024-03-11T21:50:30.311590+0000","last_undegraded":"2024-03-11T21:50:30.311590+0000","last_fullsized":"2024-03-11T21:50:30.311590+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:12.415410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:42.031080+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.415410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.843940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1842,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312905+0000","last_change":"2024-03-11T21:50:05.452386+0000","last_active":"2024-03-11T21:50:30.312905+0000","last_peered":"2024-03-11T21:50:30.312905+0000","last_clean":"2024-03-11T21:50:30.312905+0000","last_became_active":"2024-03-11T21:33:59.485694+0000","last_became_peered":"2024-03-11T21:33:59.485694+0000","last_unstale":"2024-03-11T21:50:30.312905+0000","last_undegraded":"2024-03-11T21:50:30.312905+0000","last_fullsized":"2024-03-11T21:50:30.312905+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:05.452156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:05.452156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:07.486527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"864'30","reported_seq":2063,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.152021+0000","last_change":"2024-03-11T21:49:38.165791+0000","last_active":"2024-03-11T21:50:31.152021+0000","last_peered":"2024-03-11T21:50:31.152021+0000","last_clean":"2024-03-11T21:50:31.152021+0000","last_became_active":"2024-03-11T21:23:04.931576+0000","last_became_peered":"2024-03-11T21:23:04.931576+0000","last_unstale":"2024-03-11T21:50:31.152021+0000","last_undegraded":"2024-03-11T21:50:31.152021+0000","last_fullsized":"2024-03-11T21:50:31.152021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'30","last_scrub_stamp":"2024-03-11T21:49:38.165541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.165541+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.030299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1899,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.319819+0000","last_change":"2024-03-11T21:49:37.072247+0000","last_active":"2024-03-11T21:50:30.319819+0000","last_peered":"2024-03-11T21:50:30.319819+0000","last_clean":"2024-03-11T21:50:30.319819+0000","last_became_active":"2024-03-11T21:31:37.409930+0000","last_became_peered":"2024-03-11T21:31:37.409930+0000","last_unstale":"2024-03-11T21:50:30.319819+0000","last_undegraded":"2024-03-11T21:50:30.319819+0000","last_fullsized":"2024-03-11T21:50:30.319819+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:37.072152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:53.706230+0000","last_clean_scrub_stamp":"2024-03-11T21:49:37.072152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:00.353230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1868,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316239+0000","last_change":"2024-03-11T21:49:27.009907+0000","last_active":"2024-03-11T21:50:30.316239+0000","last_peered":"2024-03-11T21:50:30.316239+0000","last_clean":"2024-03-11T21:50:30.316239+0000","last_became_active":"2024-03-11T21:31:37.397297+0000","last_became_peered":"2024-03-11T21:31:37.397297+0000","last_unstale":"2024-03-11T21:50:30.316239+0000","last_undegraded":"2024-03-11T21:50:30.316239+0000","last_fullsized":"2024-03-11T21:50:30.316239+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:27.009691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:27.009691+0000","last_clean_scrub_stamp":"2024-03-11T21:49:27.009691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.867332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1966,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411755+0000","last_change":"2024-03-11T21:50:23.963826+0000","last_active":"2024-03-11T21:50:30.411755+0000","last_peered":"2024-03-11T21:50:30.411755+0000","last_clean":"2024-03-11T21:50:30.411755+0000","last_became_active":"2024-03-11T21:23:09.151481+0000","last_became_peered":"2024-03-11T21:23:09.151481+0000","last_unstale":"2024-03-11T21:50:30.411755+0000","last_undegraded":"2024-03-11T21:50:30.411755+0000","last_fullsized":"2024-03-11T21:50:30.411755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:23.963609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:26.312862+0000","last_clean_scrub_stamp":"2024-03-11T21:50:23.963609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:28.946086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1840,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313763+0000","last_change":"2024-03-11T21:50:28.575621+0000","last_active":"2024-03-11T21:50:30.313763+0000","last_peered":"2024-03-11T21:50:30.313763+0000","last_clean":"2024-03-11T21:50:30.313763+0000","last_became_active":"2024-03-11T21:33:59.485589+0000","last_became_peered":"2024-03-11T21:33:59.485589+0000","last_unstale":"2024-03-11T21:50:30.313763+0000","last_undegraded":"2024-03-11T21:50:30.313763+0000","last_fullsized":"2024-03-11T21:50:30.313763+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:28.575406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:38.414635+0000","last_clean_scrub_stamp":"2024-03-11T21:50:28.575406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.391435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"864'18","reported_seq":2020,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315427+0000","last_change":"2024-03-11T21:49:06.177306+0000","last_active":"2024-03-11T21:50:30.315427+0000","last_peered":"2024-03-11T21:50:30.315427+0000","last_clean":"2024-03-11T21:50:30.315427+0000","last_became_active":"2024-03-11T21:23:04.946704+0000","last_became_peered":"2024-03-11T21:23:04.946704+0000","last_unstale":"2024-03-11T21:50:30.315427+0000","last_undegraded":"2024-03-11T21:50:30.315427+0000","last_fullsized":"2024-03-11T21:50:30.315427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'18","last_scrub_stamp":"2024-03-11T21:49:06.177079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:06.177079+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.723017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1782,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315975+0000","last_change":"2024-03-11T21:49:53.202771+0000","last_active":"2024-03-11T21:50:30.315975+0000","last_peered":"2024-03-11T21:50:30.315975+0000","last_clean":"2024-03-11T21:50:30.315975+0000","last_became_active":"2024-03-11T21:48:12.935533+0000","last_became_peered":"2024-03-11T21:48:12.935533+0000","last_unstale":"2024-03-11T21:50:30.315975+0000","last_undegraded":"2024-03-11T21:50:30.315975+0000","last_fullsized":"2024-03-11T21:50:30.315975+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.202493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.202493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.004431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2013,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443154+0000","last_change":"2024-03-11T21:50:01.519326+0000","last_active":"2024-03-11T21:50:30.443154+0000","last_peered":"2024-03-11T21:50:30.443154+0000","last_clean":"2024-03-11T21:50:30.443154+0000","last_became_active":"2024-03-11T21:23:07.082116+0000","last_became_peered":"2024-03-11T21:23:07.082116+0000","last_unstale":"2024-03-11T21:50:30.443154+0000","last_undegraded":"2024-03-11T21:50:30.443154+0000","last_fullsized":"2024-03-11T21:50:30.443154+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:01.519026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:01.519026+0000","last_clean_scrub_stamp":"2024-03-11T21:50:01.519026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.326962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1957,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862656+0000","last_change":"2024-03-11T21:49:24.177802+0000","last_active":"2024-03-11T21:50:30.862656+0000","last_peered":"2024-03-11T21:50:30.862656+0000","last_clean":"2024-03-11T21:50:30.862656+0000","last_became_active":"2024-03-11T21:23:09.153550+0000","last_became_peered":"2024-03-11T21:23:09.153550+0000","last_unstale":"2024-03-11T21:50:30.862656+0000","last_undegraded":"2024-03-11T21:50:30.862656+0000","last_fullsized":"2024-03-11T21:50:30.862656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:24.177578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:24:24.073667+0000","last_clean_scrub_stamp":"2024-03-11T21:49:24.177578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"600'129","reported_seq":1228,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.430866+0000","last_change":"2024-03-11T21:49:43.454964+0000","last_active":"2024-03-11T21:50:31.430866+0000","last_peered":"2024-03-11T21:50:31.430866+0000","last_clean":"2024-03-11T21:50:31.430866+0000","last_became_active":"2024-03-11T21:44:03.841587+0000","last_became_peered":"2024-03-11T21:44:03.841587+0000","last_unstale":"2024-03-11T21:50:31.430866+0000","last_undegraded":"2024-03-11T21:50:31.430866+0000","last_fullsized":"2024-03-11T21:50:31.430866+0000","mapping_epoch":900,"log_start":"0'0","ondisk_log_start":"0'0","created":895,"last_epoch_clean":901,"parent":"0.0","parent_split_bits":5,"last_scrub":"600'129","last_scrub_stamp":"2024-03-11T21:49:43.454749+0000","last_deep_scrub":"600'129","last_deep_scrub_stamp":"2024-03-11T21:47:03.572458+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.454749+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.687423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1843,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313429+0000","last_change":"2024-03-11T21:50:19.461141+0000","last_active":"2024-03-11T21:50:30.313429+0000","last_peered":"2024-03-11T21:50:30.313429+0000","last_clean":"2024-03-11T21:50:30.313429+0000","last_became_active":"2024-03-11T21:31:37.414127+0000","last_became_peered":"2024-03-11T21:31:37.414127+0000","last_unstale":"2024-03-11T21:50:30.313429+0000","last_undegraded":"2024-03-11T21:50:30.313429+0000","last_fullsized":"2024-03-11T21:50:30.313429+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.460934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.460934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:28.244404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"1012'25","reported_seq":1817,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315219+0000","last_change":"2024-03-11T21:49:30.095706+0000","last_active":"2024-03-11T21:50:30.315219+0000","last_peered":"2024-03-11T21:50:30.315219+0000","last_clean":"2024-03-11T21:50:30.315219+0000","last_became_active":"2024-03-11T21:48:12.933804+0000","last_became_peered":"2024-03-11T21:48:12.933804+0000","last_unstale":"2024-03-11T21:50:30.315219+0000","last_undegraded":"2024-03-11T21:50:30.315219+0000","last_fullsized":"2024-03-11T21:50:30.315219+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"1012'25","last_scrub_stamp":"2024-03-11T21:49:30.095454+0000","last_deep_scrub":"1012'25","last_deep_scrub_stamp":"2024-03-11T21:49:30.095454+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.095454+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:30.777865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2021,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444713+0000","last_change":"2024-03-11T21:49:29.700554+0000","last_active":"2024-03-11T21:50:30.444713+0000","last_peered":"2024-03-11T21:50:30.444713+0000","last_clean":"2024-03-11T21:50:30.444713+0000","last_became_active":"2024-03-11T21:23:02.938498+0000","last_became_peered":"2024-03-11T21:23:02.938498+0000","last_unstale":"2024-03-11T21:50:30.444713+0000","last_undegraded":"2024-03-11T21:50:30.444713+0000","last_fullsized":"2024-03-11T21:50:30.444713+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:29.700259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:15.606625+0000","last_clean_scrub_stamp":"2024-03-11T21:49:29.700259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:55.122700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1961,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862428+0000","last_change":"2024-03-11T21:49:47.091364+0000","last_active":"2024-03-11T21:50:30.862428+0000","last_peered":"2024-03-11T21:50:30.862428+0000","last_clean":"2024-03-11T21:50:30.862428+0000","last_became_active":"2024-03-11T21:23:07.086617+0000","last_became_peered":"2024-03-11T21:23:07.086617+0000","last_unstale":"2024-03-11T21:50:30.862428+0000","last_undegraded":"2024-03-11T21:50:30.862428+0000","last_fullsized":"2024-03-11T21:50:30.862428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:47.091151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:49:47.091151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:48.902126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1958,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413458+0000","last_change":"2024-03-11T21:49:21.672598+0000","last_active":"2024-03-11T21:50:30.413458+0000","last_peered":"2024-03-11T21:50:30.413458+0000","last_clean":"2024-03-11T21:50:30.413458+0000","last_became_active":"2024-03-11T21:23:09.162410+0000","last_became_peered":"2024-03-11T21:23:09.162410+0000","last_unstale":"2024-03-11T21:50:30.413458+0000","last_undegraded":"2024-03-11T21:50:30.413458+0000","last_fullsized":"2024-03-11T21:50:30.413458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:21.672335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:53.585657+0000","last_clean_scrub_stamp":"2024-03-11T21:49:21.672335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:45.666488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1815,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862866+0000","last_change":"2024-03-11T21:49:25.144198+0000","last_active":"2024-03-11T21:50:30.862866+0000","last_peered":"2024-03-11T21:50:30.862866+0000","last_clean":"2024-03-11T21:50:30.862866+0000","last_became_active":"2024-03-11T21:28:51.421928+0000","last_became_peered":"2024-03-11T21:28:51.421928+0000","last_unstale":"2024-03-11T21:50:30.862866+0000","last_undegraded":"2024-03-11T21:50:30.862866+0000","last_fullsized":"2024-03-11T21:50:30.862866+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:25.143977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:46.798825+0000","last_clean_scrub_stamp":"2024-03-11T21:49:25.143977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:49.482180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"864'46","reported_seq":1978,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.317633+0000","last_change":"2024-03-11T21:50:22.842183+0000","last_active":"2024-03-11T21:50:30.317633+0000","last_peered":"2024-03-11T21:50:30.317633+0000","last_clean":"2024-03-11T21:50:30.317633+0000","last_became_active":"2024-03-11T21:31:37.386330+0000","last_became_peered":"2024-03-11T21:31:37.386330+0000","last_unstale":"2024-03-11T21:50:30.317633+0000","last_undegraded":"2024-03-11T21:50:30.317633+0000","last_fullsized":"2024-03-11T21:50:30.317633+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'46","last_scrub_stamp":"2024-03-11T21:50:22.841896+0000","last_deep_scrub":"864'46","last_deep_scrub_stamp":"2024-03-11T21:49:08.052460+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.841896+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:38.474707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1788,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.312708+0000","last_change":"2024-03-11T21:50:18.365330+0000","last_active":"2024-03-11T21:50:30.312708+0000","last_peered":"2024-03-11T21:50:30.312708+0000","last_clean":"2024-03-11T21:50:30.312708+0000","last_became_active":"2024-03-11T21:48:12.900431+0000","last_became_peered":"2024-03-11T21:48:12.900431+0000","last_unstale":"2024-03-11T21:50:30.312708+0000","last_undegraded":"2024-03-11T21:50:30.312708+0000","last_fullsized":"2024-03-11T21:50:30.312708+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:18.365111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.365111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:42.139089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"988'12","reported_seq":2359,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.444932+0000","last_change":"2024-03-11T21:50:09.497019+0000","last_active":"2024-03-11T21:50:30.444932+0000","last_peered":"2024-03-11T21:50:30.444932+0000","last_clean":"2024-03-11T21:50:30.444932+0000","last_became_active":"2024-03-11T21:23:07.091848+0000","last_became_peered":"2024-03-11T21:23:07.091848+0000","last_unstale":"2024-03-11T21:50:30.444932+0000","last_undegraded":"2024-03-11T21:50:30.444932+0000","last_fullsized":"2024-03-11T21:50:30.444932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"988'12","last_scrub_stamp":"2024-03-11T21:50:09.496750+0000","last_deep_scrub":"988'12","last_deep_scrub_stamp":"2024-03-11T21:50:09.496750+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.496750+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.858411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"984'2","reported_seq":1959,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863983+0000","last_change":"2024-03-11T21:50:17.943138+0000","last_active":"2024-03-11T21:50:30.863983+0000","last_peered":"2024-03-11T21:50:30.863983+0000","last_clean":"2024-03-11T21:50:30.863983+0000","last_became_active":"2024-03-11T21:23:09.155949+0000","last_became_peered":"2024-03-11T21:23:09.155949+0000","last_unstale":"2024-03-11T21:50:30.863983+0000","last_undegraded":"2024-03-11T21:50:30.863983+0000","last_fullsized":"2024-03-11T21:50:30.863983+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"984'2","last_scrub_stamp":"2024-03-11T21:50:17.942909+0000","last_deep_scrub":"984'2","last_deep_scrub_stamp":"2024-03-11T21:48:21.143821+0000","last_clean_scrub_stamp":"2024-03-11T21:50:17.942909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.361236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1630,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314172+0000","last_change":"2024-03-11T21:50:25.364558+0000","last_active":"2024-03-11T21:50:30.314172+0000","last_peered":"2024-03-11T21:50:30.314172+0000","last_clean":"2024-03-11T21:50:30.314172+0000","last_became_active":"2024-03-11T21:48:12.931163+0000","last_became_peered":"2024-03-11T21:48:12.931163+0000","last_unstale":"2024-03-11T21:50:30.314172+0000","last_undegraded":"2024-03-11T21:50:30.314172+0000","last_fullsized":"2024-03-11T21:50:30.314172+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:25.364340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:08.958336+0000","last_clean_scrub_stamp":"2024-03-11T21:50:25.364340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.287227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"864'21","reported_seq":2017,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411323+0000","last_change":"2024-03-11T21:49:51.760336+0000","last_active":"2024-03-11T21:50:30.411323+0000","last_peered":"2024-03-11T21:50:30.411323+0000","last_clean":"2024-03-11T21:50:30.411323+0000","last_became_active":"2024-03-11T21:31:37.396485+0000","last_became_peered":"2024-03-11T21:31:37.396485+0000","last_unstale":"2024-03-11T21:50:30.411323+0000","last_undegraded":"2024-03-11T21:50:30.411323+0000","last_fullsized":"2024-03-11T21:50:30.411323+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'21","last_scrub_stamp":"2024-03-11T21:49:51.760049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:03.909395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.760049+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:15.173839+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":1979,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.411915+0000","last_change":"2024-03-11T21:50:12.934889+0000","last_active":"2024-03-11T21:50:30.411915+0000","last_peered":"2024-03-11T21:50:30.411915+0000","last_clean":"2024-03-11T21:50:30.411915+0000","last_became_active":"2024-03-11T21:23:02.931323+0000","last_became_peered":"2024-03-11T21:23:02.931323+0000","last_unstale":"2024-03-11T21:50:30.411915+0000","last_undegraded":"2024-03-11T21:50:30.411915+0000","last_fullsized":"2024-03-11T21:50:30.411915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:50:12.934798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.934798+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.758894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1983,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.314974+0000","last_change":"2024-03-11T21:49:22.222504+0000","last_active":"2024-03-11T21:50:30.314974+0000","last_peered":"2024-03-11T21:50:30.314974+0000","last_clean":"2024-03-11T21:50:30.314974+0000","last_became_active":"2024-03-11T21:23:07.084140+0000","last_became_peered":"2024-03-11T21:23:07.084140+0000","last_unstale":"2024-03-11T21:50:30.314974+0000","last_undegraded":"2024-03-11T21:50:30.314974+0000","last_fullsized":"2024-03-11T21:50:30.314974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:22.222268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:41.375162+0000","last_clean_scrub_stamp":"2024-03-11T21:49:22.222268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:33.507658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1858,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.318643+0000","last_change":"2024-03-11T21:50:10.922696+0000","last_active":"2024-03-11T21:50:30.318643+0000","last_peered":"2024-03-11T21:50:30.318643+0000","last_clean":"2024-03-11T21:50:30.318643+0000","last_became_active":"2024-03-11T21:31:37.389754+0000","last_became_peered":"2024-03-11T21:31:37.389754+0000","last_unstale":"2024-03-11T21:50:30.318643+0000","last_undegraded":"2024-03-11T21:50:30.318643+0000","last_fullsized":"2024-03-11T21:50:30.318643+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.922475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:10.922475+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.922475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.052341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1815,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.861594+0000","last_change":"2024-03-11T21:50:18.945863+0000","last_active":"2024-03-11T21:50:30.861594+0000","last_peered":"2024-03-11T21:50:30.861594+0000","last_clean":"2024-03-11T21:50:30.861594+0000","last_became_active":"2024-03-11T21:28:51.407874+0000","last_became_peered":"2024-03-11T21:28:51.407874+0000","last_unstale":"2024-03-11T21:50:30.861594+0000","last_undegraded":"2024-03-11T21:50:30.861594+0000","last_fullsized":"2024-03-11T21:50:30.861594+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:18.945645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:50:18.945645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.552804+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"864'11","reported_seq":2002,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313835+0000","last_change":"2024-03-11T21:49:18.187721+0000","last_active":"2024-03-11T21:50:30.313835+0000","last_peered":"2024-03-11T21:50:30.313835+0000","last_clean":"2024-03-11T21:50:30.313835+0000","last_became_active":"2024-03-11T21:23:04.941461+0000","last_became_peered":"2024-03-11T21:23:04.941461+0000","last_unstale":"2024-03-11T21:50:30.313835+0000","last_undegraded":"2024-03-11T21:50:30.313835+0000","last_fullsized":"2024-03-11T21:50:30.313835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'11","last_scrub_stamp":"2024-03-11T21:49:18.187423+0000","last_deep_scrub":"118'7","last_deep_scrub_stamp":"2024-03-11T21:26:20.454747+0000","last_clean_scrub_stamp":"2024-03-11T21:49:18.187423+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:35.358056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2021,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.443405+0000","last_change":"2024-03-11T21:50:11.492045+0000","last_active":"2024-03-11T21:50:30.443405+0000","last_peered":"2024-03-11T21:50:30.443405+0000","last_clean":"2024-03-11T21:50:30.443405+0000","last_became_active":"2024-03-11T21:23:02.930889+0000","last_became_peered":"2024-03-11T21:23:02.930889+0000","last_unstale":"2024-03-11T21:50:30.443405+0000","last_undegraded":"2024-03-11T21:50:30.443405+0000","last_fullsized":"2024-03-11T21:50:30.443405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.491841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:40.710906+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.491841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.017126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1852,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.320180+0000","last_change":"2024-03-11T21:49:56.831339+0000","last_active":"2024-03-11T21:50:30.320180+0000","last_peered":"2024-03-11T21:50:30.320180+0000","last_clean":"2024-03-11T21:50:30.320180+0000","last_became_active":"2024-03-11T21:31:37.406425+0000","last_became_peered":"2024-03-11T21:31:37.406425+0000","last_unstale":"2024-03-11T21:50:30.320180+0000","last_undegraded":"2024-03-11T21:50:30.320180+0000","last_fullsized":"2024-03-11T21:50:30.320180+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:56.831173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:25.102956+0000","last_clean_scrub_stamp":"2024-03-11T21:49:56.831173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.886668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2004,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.153544+0000","last_change":"2024-03-11T21:50:23.163107+0000","last_active":"2024-03-11T21:50:31.153544+0000","last_peered":"2024-03-11T21:50:31.153544+0000","last_clean":"2024-03-11T21:50:31.153544+0000","last_became_active":"2024-03-11T21:23:09.146902+0000","last_became_peered":"2024-03-11T21:23:09.146902+0000","last_unstale":"2024-03-11T21:50:31.153544+0000","last_undegraded":"2024-03-11T21:50:31.153544+0000","last_fullsized":"2024-03-11T21:50:31.153544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:23.162896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:07.000334+0000","last_clean_scrub_stamp":"2024-03-11T21:50:23.162896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.117681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1858,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.150227+0000","last_change":"2024-03-11T21:50:22.160399+0000","last_active":"2024-03-11T21:50:31.150227+0000","last_peered":"2024-03-11T21:50:31.150227+0000","last_clean":"2024-03-11T21:50:31.150227+0000","last_became_active":"2024-03-11T21:33:59.390421+0000","last_became_peered":"2024-03-11T21:33:59.390421+0000","last_unstale":"2024-03-11T21:50:31.150227+0000","last_undegraded":"2024-03-11T21:50:31.150227+0000","last_fullsized":"2024-03-11T21:50:31.150227+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.160112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:44.265056+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.160112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:46.333624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"864'25","reported_seq":2057,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.151426+0000","last_change":"2024-03-11T21:49:45.142338+0000","last_active":"2024-03-11T21:50:31.151426+0000","last_peered":"2024-03-11T21:50:31.151426+0000","last_clean":"2024-03-11T21:50:31.151426+0000","last_became_active":"2024-03-11T21:31:37.378357+0000","last_became_peered":"2024-03-11T21:31:37.378357+0000","last_unstale":"2024-03-11T21:50:31.151426+0000","last_undegraded":"2024-03-11T21:50:31.151426+0000","last_fullsized":"2024-03-11T21:50:31.151426+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'25","last_scrub_stamp":"2024-03-11T21:49:45.142100+0000","last_deep_scrub":"864'25","last_deep_scrub_stamp":"2024-03-11T21:49:45.142100+0000","last_clean_scrub_stamp":"2024-03-11T21:49:45.142100+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:56.214752+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1978,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412591+0000","last_change":"2024-03-11T21:49:44.674801+0000","last_active":"2024-03-11T21:50:30.412591+0000","last_peered":"2024-03-11T21:50:30.412591+0000","last_clean":"2024-03-11T21:50:30.412591+0000","last_became_active":"2024-03-11T21:23:02.926376+0000","last_became_peered":"2024-03-11T21:23:02.926376+0000","last_unstale":"2024-03-11T21:50:30.412591+0000","last_undegraded":"2024-03-11T21:50:30.412591+0000","last_fullsized":"2024-03-11T21:50:30.412591+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:44.674705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:39.218970+0000","last_clean_scrub_stamp":"2024-03-11T21:49:44.674705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:59.441784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":2005,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.152217+0000","last_change":"2024-03-11T21:50:27.173901+0000","last_active":"2024-03-11T21:50:31.152217+0000","last_peered":"2024-03-11T21:50:31.152217+0000","last_clean":"2024-03-11T21:50:31.152217+0000","last_became_active":"2024-03-11T21:23:07.072742+0000","last_became_peered":"2024-03-11T21:23:07.072742+0000","last_unstale":"2024-03-11T21:50:31.152217+0000","last_undegraded":"2024-03-11T21:50:31.152217+0000","last_fullsized":"2024-03-11T21:50:31.152217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:27.173653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:56.457609+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.173653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:51.149291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1960,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.414232+0000","last_change":"2024-03-11T21:49:38.770943+0000","last_active":"2024-03-11T21:50:30.414232+0000","last_peered":"2024-03-11T21:50:30.414232+0000","last_clean":"2024-03-11T21:50:30.414232+0000","last_became_active":"2024-03-11T21:23:09.157886+0000","last_became_peered":"2024-03-11T21:23:09.157886+0000","last_unstale":"2024-03-11T21:50:30.414232+0000","last_undegraded":"2024-03-11T21:50:30.414232+0000","last_fullsized":"2024-03-11T21:50:30.414232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:38.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:37:57.570887+0000","last_clean_scrub_stamp":"2024-03-11T21:49:38.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:42.048814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1816,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.864614+0000","last_change":"2024-03-11T21:48:57.200335+0000","last_active":"2024-03-11T21:50:30.864614+0000","last_peered":"2024-03-11T21:50:30.864614+0000","last_clean":"2024-03-11T21:50:30.864614+0000","last_became_active":"2024-03-11T21:31:37.403942+0000","last_became_peered":"2024-03-11T21:31:37.403942+0000","last_unstale":"2024-03-11T21:50:30.864614+0000","last_undegraded":"2024-03-11T21:50:30.864614+0000","last_fullsized":"2024-03-11T21:50:30.864614+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.200119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.200119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"996'29","reported_seq":1941,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315034+0000","last_change":"2024-03-11T21:50:27.452321+0000","last_active":"2024-03-11T21:50:30.315034+0000","last_peered":"2024-03-11T21:50:30.315034+0000","last_clean":"2024-03-11T21:50:30.315034+0000","last_became_active":"2024-03-11T21:48:12.920561+0000","last_became_peered":"2024-03-11T21:48:12.920561+0000","last_unstale":"2024-03-11T21:50:30.315034+0000","last_undegraded":"2024-03-11T21:50:30.315034+0000","last_fullsized":"2024-03-11T21:50:30.315034+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'29","last_scrub_stamp":"2024-03-11T21:50:27.451964+0000","last_deep_scrub":"996'29","last_deep_scrub_stamp":"2024-03-11T21:47:38.135324+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.451964+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.031370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1969,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863247+0000","last_change":"2024-03-11T21:48:46.164103+0000","last_active":"2024-03-11T21:50:30.863247+0000","last_peered":"2024-03-11T21:50:30.863247+0000","last_clean":"2024-03-11T21:50:30.863247+0000","last_became_active":"2024-03-11T21:23:02.944655+0000","last_became_peered":"2024-03-11T21:23:02.944655+0000","last_unstale":"2024-03-11T21:50:30.863247+0000","last_undegraded":"2024-03-11T21:50:30.863247+0000","last_fullsized":"2024-03-11T21:50:30.863247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:46.163955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:13.461151+0000","last_clean_scrub_stamp":"2024-03-11T21:48:46.163955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1766,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313687+0000","last_change":"2024-03-11T21:50:15.240821+0000","last_active":"2024-03-11T21:50:30.313687+0000","last_peered":"2024-03-11T21:50:30.313687+0000","last_clean":"2024-03-11T21:50:30.313687+0000","last_became_active":"2024-03-11T21:48:12.918435+0000","last_became_peered":"2024-03-11T21:48:12.918435+0000","last_unstale":"2024-03-11T21:50:30.313687+0000","last_undegraded":"2024-03-11T21:50:30.313687+0000","last_fullsized":"2024-03-11T21:50:30.313687+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.240520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:22.542107+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.240520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.552905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1993,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.311929+0000","last_change":"2024-03-11T21:50:21.480347+0000","last_active":"2024-03-11T21:50:30.311929+0000","last_peered":"2024-03-11T21:50:30.311929+0000","last_clean":"2024-03-11T21:50:30.311929+0000","last_became_active":"2024-03-11T21:31:37.409125+0000","last_became_peered":"2024-03-11T21:31:37.409125+0000","last_unstale":"2024-03-11T21:50:30.311929+0000","last_undegraded":"2024-03-11T21:50:30.311929+0000","last_fullsized":"2024-03-11T21:50:30.311929+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.480119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.480119+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.480119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.475906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1861,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.153023+0000","last_change":"2024-03-11T21:49:26.248908+0000","last_active":"2024-03-11T21:50:31.153023+0000","last_peered":"2024-03-11T21:50:31.153023+0000","last_clean":"2024-03-11T21:50:31.153023+0000","last_became_active":"2024-03-11T21:31:37.374518+0000","last_became_peered":"2024-03-11T21:31:37.374518+0000","last_unstale":"2024-03-11T21:50:31.153023+0000","last_undegraded":"2024-03-11T21:50:31.153023+0000","last_fullsized":"2024-03-11T21:50:31.153023+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:26.248678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:41.604986+0000","last_clean_scrub_stamp":"2024-03-11T21:49:26.248678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:44.888306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"864'46","reported_seq":1855,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315094+0000","last_change":"2024-03-11T21:49:51.211205+0000","last_active":"2024-03-11T21:50:30.315094+0000","last_peered":"2024-03-11T21:50:30.315094+0000","last_clean":"2024-03-11T21:50:30.315094+0000","last_became_active":"2024-03-11T21:48:12.924933+0000","last_became_peered":"2024-03-11T21:48:12.924933+0000","last_unstale":"2024-03-11T21:50:30.315094+0000","last_undegraded":"2024-03-11T21:50:30.315094+0000","last_fullsized":"2024-03-11T21:50:30.315094+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'46","last_scrub_stamp":"2024-03-11T21:49:51.210966+0000","last_deep_scrub":"864'46","last_deep_scrub_stamp":"2024-03-11T21:46:07.956383+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.210966+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:08.301652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1980,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.864525+0000","last_change":"2024-03-11T21:50:02.956633+0000","last_active":"2024-03-11T21:50:30.864525+0000","last_peered":"2024-03-11T21:50:30.864525+0000","last_clean":"2024-03-11T21:50:30.864525+0000","last_became_active":"2024-03-11T21:23:02.935427+0000","last_became_peered":"2024-03-11T21:23:02.935427+0000","last_unstale":"2024-03-11T21:50:30.864525+0000","last_undegraded":"2024-03-11T21:50:30.864525+0000","last_fullsized":"2024-03-11T21:50:30.864525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:02.956333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:01.894398+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.956333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.561903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1969,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.862580+0000","last_change":"2024-03-11T21:50:28.774364+0000","last_active":"2024-03-11T21:50:30.862580+0000","last_peered":"2024-03-11T21:50:30.862580+0000","last_clean":"2024-03-11T21:50:30.862580+0000","last_became_active":"2024-03-11T21:31:37.401330+0000","last_became_peered":"2024-03-11T21:31:37.401330+0000","last_unstale":"2024-03-11T21:50:30.862580+0000","last_undegraded":"2024-03-11T21:50:30.862580+0000","last_fullsized":"2024-03-11T21:50:30.862580+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:28.774007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:06.041722+0000","last_clean_scrub_stamp":"2024-03-11T21:50:28.774007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.125524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1763,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.313258+0000","last_change":"2024-03-11T21:49:07.274484+0000","last_active":"2024-03-11T21:50:30.313258+0000","last_peered":"2024-03-11T21:50:30.313258+0000","last_clean":"2024-03-11T21:50:30.313258+0000","last_became_active":"2024-03-11T21:48:12.924446+0000","last_became_peered":"2024-03-11T21:48:12.924446+0000","last_unstale":"2024-03-11T21:50:30.313258+0000","last_undegraded":"2024-03-11T21:50:30.313258+0000","last_fullsized":"2024-03-11T21:50:30.313258+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:07.274264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:07.274264+0000","last_clean_scrub_stamp":"2024-03-11T21:49:07.274264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:36.945725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1824,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.863280+0000","last_change":"2024-03-11T21:50:03.976125+0000","last_active":"2024-03-11T21:50:30.863280+0000","last_peered":"2024-03-11T21:50:30.863280+0000","last_clean":"2024-03-11T21:50:30.863280+0000","last_became_active":"2024-03-11T21:31:37.416235+0000","last_became_peered":"2024-03-11T21:31:37.416235+0000","last_unstale":"2024-03-11T21:50:30.863280+0000","last_undegraded":"2024-03-11T21:50:30.863280+0000","last_fullsized":"2024-03-11T21:50:30.863280+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.975820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:22.179791+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.975820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.994378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"864'32","reported_seq":2043,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.315844+0000","last_change":"2024-03-11T21:49:27.251633+0000","last_active":"2024-03-11T21:50:30.315844+0000","last_peered":"2024-03-11T21:50:30.315844+0000","last_clean":"2024-03-11T21:50:30.315844+0000","last_became_active":"2024-03-11T21:23:05.023830+0000","last_became_peered":"2024-03-11T21:23:05.023830+0000","last_unstale":"2024-03-11T21:50:30.315844+0000","last_undegraded":"2024-03-11T21:50:30.315844+0000","last_fullsized":"2024-03-11T21:50:30.315844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'32","last_scrub_stamp":"2024-03-11T21:49:27.251246+0000","last_deep_scrub":"864'32","last_deep_scrub_stamp":"2024-03-11T21:47:49.771860+0000","last_clean_scrub_stamp":"2024-03-11T21:49:27.251246+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:53.373735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1972,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413429+0000","last_change":"2024-03-11T21:49:13.528524+0000","last_active":"2024-03-11T21:50:30.413429+0000","last_peered":"2024-03-11T21:50:30.413429+0000","last_clean":"2024-03-11T21:50:30.413429+0000","last_became_active":"2024-03-11T21:23:02.942678+0000","last_became_peered":"2024-03-11T21:23:02.942678+0000","last_unstale":"2024-03-11T21:50:30.413429+0000","last_undegraded":"2024-03-11T21:50:30.413429+0000","last_fullsized":"2024-03-11T21:50:30.413429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:13.528313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:13.528313+0000","last_clean_scrub_stamp":"2024-03-11T21:49:13.528313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1848,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.317879+0000","last_change":"2024-03-11T21:49:32.006309+0000","last_active":"2024-03-11T21:50:30.317879+0000","last_peered":"2024-03-11T21:50:30.317879+0000","last_clean":"2024-03-11T21:50:30.317879+0000","last_became_active":"2024-03-11T21:31:37.389613+0000","last_became_peered":"2024-03-11T21:31:37.389613+0000","last_unstale":"2024-03-11T21:50:30.317879+0000","last_undegraded":"2024-03-11T21:50:30.317879+0000","last_fullsized":"2024-03-11T21:50:30.317879+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.006162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.006162+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.006162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:49.239412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2011,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.150979+0000","last_change":"2024-03-11T21:50:19.199264+0000","last_active":"2024-03-11T21:50:31.150979+0000","last_peered":"2024-03-11T21:50:31.150979+0000","last_clean":"2024-03-11T21:50:31.150979+0000","last_became_active":"2024-03-11T21:31:37.377885+0000","last_became_peered":"2024-03-11T21:31:37.377885+0000","last_unstale":"2024-03-11T21:50:31.150979+0000","last_undegraded":"2024-03-11T21:50:31.150979+0000","last_fullsized":"2024-03-11T21:50:31.150979+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.199036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:19.199036+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.199036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.922229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1707,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.320591+0000","last_change":"2024-03-11T21:49:53.805510+0000","last_active":"2024-03-11T21:50:30.320591+0000","last_peered":"2024-03-11T21:50:30.320591+0000","last_clean":"2024-03-11T21:50:30.320591+0000","last_became_active":"2024-03-11T21:31:37.394309+0000","last_became_peered":"2024-03-11T21:31:37.394309+0000","last_unstale":"2024-03-11T21:50:30.320591+0000","last_undegraded":"2024-03-11T21:50:30.320591+0000","last_fullsized":"2024-03-11T21:50:30.320591+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.805355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:53.805355+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.805355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:06.623421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"983'35","reported_seq":2026,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.413796+0000","last_change":"2024-03-11T21:50:02.812689+0000","last_active":"2024-03-11T21:50:30.413796+0000","last_peered":"2024-03-11T21:50:30.413796+0000","last_clean":"2024-03-11T21:50:30.413796+0000","last_became_active":"2024-03-11T21:23:05.025773+0000","last_became_peered":"2024-03-11T21:23:05.025773+0000","last_unstale":"2024-03-11T21:50:30.413796+0000","last_undegraded":"2024-03-11T21:50:30.413796+0000","last_fullsized":"2024-03-11T21:50:30.413796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"983'35","last_scrub_stamp":"2024-03-11T21:50:02.812297+0000","last_deep_scrub":"983'35","last_deep_scrub_stamp":"2024-03-11T21:46:27.236614+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.812297+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.029970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"104'2","reported_seq":2000,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.316157+0000","last_change":"2024-03-11T21:49:59.331379+0000","last_active":"2024-03-11T21:50:30.316157+0000","last_peered":"2024-03-11T21:50:30.316157+0000","last_clean":"2024-03-11T21:50:30.316157+0000","last_became_active":"2024-03-11T21:23:02.944099+0000","last_became_peered":"2024-03-11T21:23:02.944099+0000","last_unstale":"2024-03-11T21:50:30.316157+0000","last_undegraded":"2024-03-11T21:50:30.316157+0000","last_fullsized":"2024-03-11T21:50:30.316157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-03-11T21:49:59.331148+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-03-11T21:24:32.552332+0000","last_clean_scrub_stamp":"2024-03-11T21:49:59.331148+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:13.934714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":2015,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:31.150316+0000","last_change":"2024-03-11T21:50:06.071989+0000","last_active":"2024-03-11T21:50:31.150316+0000","last_peered":"2024-03-11T21:50:31.150316+0000","last_clean":"2024-03-11T21:50:31.150316+0000","last_became_active":"2024-03-11T21:31:37.373723+0000","last_became_peered":"2024-03-11T21:31:37.373723+0000","last_unstale":"2024-03-11T21:50:31.150316+0000","last_undegraded":"2024-03-11T21:50:31.150316+0000","last_fullsized":"2024-03-11T21:50:31.150316+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.071780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:47:19.122842+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.071780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.649775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1957,"reported_epoch":1019,"state":"active+clean","last_fresh":"2024-03-11T21:50:30.412103+0000","last_change":"2024-03-11T21:49:42.647288+0000","last_active":"2024-03-11T21:50:30.412103+0000","last_peered":"2024-03-11T21:50:30.412103+0000","last_clean":"2024-03-11T21:50:30.412103+0000","last_became_active":"2024-03-11T21:23:09.139122+0000","last_became_peered":"2024-03-11T21:23:09.139122+0000","last_unstale":"2024-03-11T21:50:30.412103+0000","last_undegraded":"2024-03-11T21:50:30.412103+0000","last_fullsized":"2024-03-11T21:50:30.412103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:42.647007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:42.647007+0000","last_clean_scrub_stamp":"2024-03-11T21:49:42.647007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:54.683532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2691,"ondisk_log_size":2691,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1557,"num_read_kb":1314,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1085,"ondisk_log_size":1085,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1627,"num_read_kb":1622,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":66,"seq":283467841878,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101564,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643588,"statfs":{"total":95995035648,"available":95891034112,"internally_reserved":0,"allocated":2404352,"data_stored":653380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":56,"seq":240518168927,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87960,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93657192,"statfs":{"total":95995035648,"available":95904964608,"internally_reserved":0,"allocated":2699264,"data_stored":978840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":295,"seq":1267015352550,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52440,"kb_used_data":2320,"kb_used_omap":27,"kb_used_meta":50084,"kb_avail":93692712,"statfs":{"total":95995035648,"available":95941337088,"internally_reserved":0,"allocated":2375680,"data_stored":651037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28343,"internal_metadata":51286345},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738782,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114332,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630820,"statfs":{"total":95995035648,"available":95877959680,"internally_reserved":0,"allocated":2707456,"data_stored":979952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379017,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110108,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635044,"statfs":{"total":95995035648,"available":95882285056,"internally_reserved":0,"allocated":2637824,"data_stored":914303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117373,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110416,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93634736,"statfs":{"total":95995035648,"available":95881969664,"internally_reserved":0,"allocated":2957312,"data_stored":1241466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855731,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118200,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93626952,"statfs":{"total":95995035648,"available":95873998848,"internally_reserved":0,"allocated":2404352,"data_stored":652562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495977,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110104,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635048,"statfs":{"total":95995035648,"available":95882289152,"internally_reserved":0,"allocated":2629632,"data_stored":913763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:50:38.726 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:50:38.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: cluster 2024-03-11T21:50:37.452053+0000 osd.2 (osd.2) 557 : cluster 0 5.a scrub starts 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: cluster 2024-03-11T21:50:37.453475+0000 osd.2 (osd.2) 558 : cluster 0 5.a scrub ok 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: audit 2024-03-11T21:50:37.752320+0000 mgr.y (mgr.14550) 1053 : audit 0 from='client.42533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: cluster 2024-03-11T21:50:37.831715+0000 osd.1 (osd.1) 495 : cluster 0 9.1f scrub starts 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: cluster 2024-03-11T21:50:37.836600+0000 osd.1 (osd.1) 496 : cluster 0 9.1f scrub ok 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: cluster 2024-03-11T21:50:37.897341+0000 mgr.y (mgr.14550) 1054 : cluster 0 pgmap v1725: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:50:39.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[29284]: audit 2024-03-11T21:50:38.426619+0000 mon.c (mon.2) 1437 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: cluster 2024-03-11T21:50:37.452053+0000 osd.2 (osd.2) 557 : cluster 0 5.a scrub starts 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: cluster 2024-03-11T21:50:37.453475+0000 osd.2 (osd.2) 558 : cluster 0 5.a scrub ok 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: audit 2024-03-11T21:50:37.752320+0000 mgr.y (mgr.14550) 1053 : audit 0 from='client.42533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: cluster 2024-03-11T21:50:37.831715+0000 osd.1 (osd.1) 495 : cluster 0 9.1f scrub starts 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: cluster 2024-03-11T21:50:37.836600+0000 osd.1 (osd.1) 496 : cluster 0 9.1f scrub ok 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: cluster 2024-03-11T21:50:37.897341+0000 mgr.y (mgr.14550) 1054 : cluster 0 pgmap v1725: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:50:39.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:38 smithi143 bash[21512]: audit 2024-03-11T21:50:38.426619+0000 mon.c (mon.2) 1437 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: cluster 2024-03-11T21:50:37.452053+0000 osd.2 (osd.2) 557 : cluster 0 5.a scrub starts 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: cluster 2024-03-11T21:50:37.453475+0000 osd.2 (osd.2) 558 : cluster 0 5.a scrub ok 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: audit 2024-03-11T21:50:37.752320+0000 mgr.y (mgr.14550) 1053 : audit 0 from='client.42533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: cluster 2024-03-11T21:50:37.831715+0000 osd.1 (osd.1) 495 : cluster 0 9.1f scrub starts 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: cluster 2024-03-11T21:50:37.836600+0000 osd.1 (osd.1) 496 : cluster 0 9.1f scrub ok 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: cluster 2024-03-11T21:50:37.897341+0000 mgr.y (mgr.14550) 1054 : cluster 0 pgmap v1725: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:50:39.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:38 smithi203 bash[23654]: audit 2024-03-11T21:50:38.426619+0000 mon.c (mon.2) 1437 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:39.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:50:39 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:40.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[29284]: cluster 2024-03-11T21:50:38.858776+0000 osd.1 (osd.1) 497 : cluster 0 2.a scrub starts 2024-03-11T21:50:40.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[29284]: cluster 2024-03-11T21:50:38.860981+0000 osd.1 (osd.1) 498 : cluster 0 2.a scrub ok 2024-03-11T21:50:40.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[29284]: cluster 2024-03-11T21:50:38.900731+0000 osd.3 (osd.3) 705 : cluster 0 5.17 scrub starts 2024-03-11T21:50:40.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[29284]: cluster 2024-03-11T21:50:38.902935+0000 osd.3 (osd.3) 706 : cluster 0 5.17 scrub ok 2024-03-11T21:50:40.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[21512]: cluster 2024-03-11T21:50:38.858776+0000 osd.1 (osd.1) 497 : cluster 0 2.a scrub starts 2024-03-11T21:50:40.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[21512]: cluster 2024-03-11T21:50:38.860981+0000 osd.1 (osd.1) 498 : cluster 0 2.a scrub ok 2024-03-11T21:50:40.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[21512]: cluster 2024-03-11T21:50:38.900731+0000 osd.3 (osd.3) 705 : cluster 0 5.17 scrub starts 2024-03-11T21:50:40.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:39 smithi143 bash[21512]: cluster 2024-03-11T21:50:38.902935+0000 osd.3 (osd.3) 706 : cluster 0 5.17 scrub ok 2024-03-11T21:50:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:39 smithi203 bash[23654]: cluster 2024-03-11T21:50:38.858776+0000 osd.1 (osd.1) 497 : cluster 0 2.a scrub starts 2024-03-11T21:50:40.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:39 smithi203 bash[23654]: cluster 2024-03-11T21:50:38.860981+0000 osd.1 (osd.1) 498 : cluster 0 2.a scrub ok 2024-03-11T21:50:40.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:39 smithi203 bash[23654]: cluster 2024-03-11T21:50:38.900731+0000 osd.3 (osd.3) 705 : cluster 0 5.17 scrub starts 2024-03-11T21:50:40.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:39 smithi203 bash[23654]: cluster 2024-03-11T21:50:38.902935+0000 osd.3 (osd.3) 706 : cluster 0 5.17 scrub ok 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[21512]: audit 2024-03-11T21:50:39.496438+0000 mgr.y (mgr.14550) 1055 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[21512]: cluster 2024-03-11T21:50:39.693621+0000 osd.7 (osd.7) 716 : cluster 0 1.2 scrub starts 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[21512]: cluster 2024-03-11T21:50:39.695966+0000 osd.7 (osd.7) 717 : cluster 0 1.2 scrub ok 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[21512]: cluster 2024-03-11T21:50:39.899248+0000 mgr.y (mgr.14550) 1056 : cluster 0 pgmap v1726: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[29284]: audit 2024-03-11T21:50:39.496438+0000 mgr.y (mgr.14550) 1055 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:41.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[29284]: cluster 2024-03-11T21:50:39.693621+0000 osd.7 (osd.7) 716 : cluster 0 1.2 scrub starts 2024-03-11T21:50:41.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[29284]: cluster 2024-03-11T21:50:39.695966+0000 osd.7 (osd.7) 717 : cluster 0 1.2 scrub ok 2024-03-11T21:50:41.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:40 smithi143 bash[29284]: cluster 2024-03-11T21:50:39.899248+0000 mgr.y (mgr.14550) 1056 : cluster 0 pgmap v1726: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:41.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:40 smithi203 bash[23654]: audit 2024-03-11T21:50:39.496438+0000 mgr.y (mgr.14550) 1055 : audit 0 from='client.24400 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:40 smithi203 bash[23654]: cluster 2024-03-11T21:50:39.693621+0000 osd.7 (osd.7) 716 : cluster 0 1.2 scrub starts 2024-03-11T21:50:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:40 smithi203 bash[23654]: cluster 2024-03-11T21:50:39.695966+0000 osd.7 (osd.7) 717 : cluster 0 1.2 scrub ok 2024-03-11T21:50:41.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:40 smithi203 bash[23654]: cluster 2024-03-11T21:50:39.899248+0000 mgr.y (mgr.14550) 1056 : cluster 0 pgmap v1726: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.623639+0000 osd.5 (osd.5) 629 : cluster 0 6.19 scrub starts 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.626670+0000 osd.5 (osd.5) 630 : cluster 0 6.19 scrub ok 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.763122+0000 osd.7 (osd.7) 718 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.763593+0000 osd.7 (osd.7) 719 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.822215+0000 mon.a (mon.0) 4186 : cluster 0 osdmap e1020: 8 total, 8 up, 8 in 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.831572+0000 osd.1 (osd.1) 499 : cluster 0 6.1 scrub starts 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[21512]: cluster 2024-03-11T21:50:40.833924+0000 osd.1 (osd.1) 500 : cluster 0 6.1 scrub ok 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.623639+0000 osd.5 (osd.5) 629 : cluster 0 6.19 scrub starts 2024-03-11T21:50:42.093 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.626670+0000 osd.5 (osd.5) 630 : cluster 0 6.19 scrub ok 2024-03-11T21:50:42.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.763122+0000 osd.7 (osd.7) 718 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:42.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.763593+0000 osd.7 (osd.7) 719 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:42.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.822215+0000 mon.a (mon.0) 4186 : cluster 0 osdmap e1020: 8 total, 8 up, 8 in 2024-03-11T21:50:42.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.831572+0000 osd.1 (osd.1) 499 : cluster 0 6.1 scrub starts 2024-03-11T21:50:42.094 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:41 smithi143 bash[29284]: cluster 2024-03-11T21:50:40.833924+0000 osd.1 (osd.1) 500 : cluster 0 6.1 scrub ok 2024-03-11T21:50:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.623639+0000 osd.5 (osd.5) 629 : cluster 0 6.19 scrub starts 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.626670+0000 osd.5 (osd.5) 630 : cluster 0 6.19 scrub ok 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.763122+0000 osd.7 (osd.7) 718 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.763593+0000 osd.7 (osd.7) 719 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.822215+0000 mon.a (mon.0) 4186 : cluster 0 osdmap e1020: 8 total, 8 up, 8 in 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.831572+0000 osd.1 (osd.1) 499 : cluster 0 6.1 scrub starts 2024-03-11T21:50:42.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:41 smithi203 bash[23654]: cluster 2024-03-11T21:50:40.833924+0000 osd.1 (osd.1) 500 : cluster 0 6.1 scrub ok 2024-03-11T21:50:42.452 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21512]: cluster 2024-03-11T21:50:41.678837+0000 osd.7 (osd.7) 720 : cluster 0 9.17 scrub starts 2024-03-11T21:50:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21512]: cluster 2024-03-11T21:50:41.687877+0000 osd.7 (osd.7) 721 : cluster 0 9.17 scrub ok 2024-03-11T21:50:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21512]: cluster 2024-03-11T21:50:41.852553+0000 osd.1 (osd.1) 501 : cluster 0 3.4 scrub starts 2024-03-11T21:50:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21512]: cluster 2024-03-11T21:50:41.858527+0000 osd.1 (osd.1) 502 : cluster 0 3.4 scrub ok 2024-03-11T21:50:43.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[21512]: cluster 2024-03-11T21:50:41.900153+0000 mgr.y (mgr.14550) 1057 : cluster 0 pgmap v1728: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[29284]: cluster 2024-03-11T21:50:41.678837+0000 osd.7 (osd.7) 720 : cluster 0 9.17 scrub starts 2024-03-11T21:50:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[29284]: cluster 2024-03-11T21:50:41.687877+0000 osd.7 (osd.7) 721 : cluster 0 9.17 scrub ok 2024-03-11T21:50:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[29284]: cluster 2024-03-11T21:50:41.852553+0000 osd.1 (osd.1) 501 : cluster 0 3.4 scrub starts 2024-03-11T21:50:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[29284]: cluster 2024-03-11T21:50:41.858527+0000 osd.1 (osd.1) 502 : cluster 0 3.4 scrub ok 2024-03-11T21:50:43.203 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:42 smithi143 bash[29284]: cluster 2024-03-11T21:50:41.900153+0000 mgr.y (mgr.14550) 1057 : cluster 0 pgmap v1728: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:43.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:42 smithi203 bash[23654]: cluster 2024-03-11T21:50:41.678837+0000 osd.7 (osd.7) 720 : cluster 0 9.17 scrub starts 2024-03-11T21:50:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:42 smithi203 bash[23654]: cluster 2024-03-11T21:50:41.687877+0000 osd.7 (osd.7) 721 : cluster 0 9.17 scrub ok 2024-03-11T21:50:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:42 smithi203 bash[23654]: cluster 2024-03-11T21:50:41.852553+0000 osd.1 (osd.1) 501 : cluster 0 3.4 scrub starts 2024-03-11T21:50:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:42 smithi203 bash[23654]: cluster 2024-03-11T21:50:41.858527+0000 osd.1 (osd.1) 502 : cluster 0 3.4 scrub ok 2024-03-11T21:50:43.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:42 smithi203 bash[23654]: cluster 2024-03-11T21:50:41.900153+0000 mgr.y (mgr.14550) 1057 : cluster 0 pgmap v1728: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:43.521 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.656222+0000 osd.7 (osd.7) 722 : cluster 0 1.0 scrub starts 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.658645+0000 osd.7 (osd.7) 723 : cluster 0 1.0 scrub ok 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.823124+0000 osd.1 (osd.1) 503 : cluster 0 9.0 scrub starts 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.825652+0000 osd.1 (osd.1) 504 : cluster 0 9.0 scrub ok 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.972176+0000 osd.3 (osd.3) 707 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:50:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[29284]: cluster 2024-03-11T21:50:42.974665+0000 osd.3 (osd.3) 708 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.656222+0000 osd.7 (osd.7) 722 : cluster 0 1.0 scrub starts 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.658645+0000 osd.7 (osd.7) 723 : cluster 0 1.0 scrub ok 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.823124+0000 osd.1 (osd.1) 503 : cluster 0 9.0 scrub starts 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.825652+0000 osd.1 (osd.1) 504 : cluster 0 9.0 scrub ok 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.972176+0000 osd.3 (osd.3) 707 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:50:44.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:43 smithi143 bash[21512]: cluster 2024-03-11T21:50:42.974665+0000 osd.3 (osd.3) 708 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:50:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.656222+0000 osd.7 (osd.7) 722 : cluster 0 1.0 scrub starts 2024-03-11T21:50:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.658645+0000 osd.7 (osd.7) 723 : cluster 0 1.0 scrub ok 2024-03-11T21:50:44.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.823124+0000 osd.1 (osd.1) 503 : cluster 0 9.0 scrub starts 2024-03-11T21:50:44.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.825652+0000 osd.1 (osd.1) 504 : cluster 0 9.0 scrub ok 2024-03-11T21:50:44.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.972176+0000 osd.3 (osd.3) 707 : cluster 0 5.13 deep-scrub starts 2024-03-11T21:50:44.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:43 smithi203 bash[23654]: cluster 2024-03-11T21:50:42.974665+0000 osd.3 (osd.3) 708 : cluster 0 5.13 deep-scrub ok 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[21512]: cluster 2024-03-11T21:50:43.902098+0000 mgr.y (mgr.14550) 1058 : cluster 0 pgmap v1729: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[21512]: cluster 2024-03-11T21:50:43.942888+0000 osd.3 (osd.3) 709 : cluster 0 2.1e scrub starts 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[21512]: cluster 2024-03-11T21:50:43.944699+0000 osd.3 (osd.3) 710 : cluster 0 2.1e scrub ok 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[29284]: cluster 2024-03-11T21:50:43.902098+0000 mgr.y (mgr.14550) 1058 : cluster 0 pgmap v1729: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[29284]: cluster 2024-03-11T21:50:43.942888+0000 osd.3 (osd.3) 709 : cluster 0 2.1e scrub starts 2024-03-11T21:50:45.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:44 smithi143 bash[29284]: cluster 2024-03-11T21:50:43.944699+0000 osd.3 (osd.3) 710 : cluster 0 2.1e scrub ok 2024-03-11T21:50:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:44 smithi203 bash[23654]: cluster 2024-03-11T21:50:43.902098+0000 mgr.y (mgr.14550) 1058 : cluster 0 pgmap v1729: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:45.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:44 smithi203 bash[23654]: cluster 2024-03-11T21:50:43.942888+0000 osd.3 (osd.3) 709 : cluster 0 2.1e scrub starts 2024-03-11T21:50:45.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:44 smithi203 bash[23654]: cluster 2024-03-11T21:50:43.944699+0000 osd.3 (osd.3) 710 : cluster 0 2.1e scrub ok 2024-03-11T21:50:46.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:45 smithi143 bash[21512]: cluster 2024-03-11T21:50:44.687946+0000 osd.7 (osd.7) 724 : cluster 0 2.0 scrub starts 2024-03-11T21:50:46.202 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:45 smithi143 bash[21512]: cluster 2024-03-11T21:50:44.690391+0000 osd.7 (osd.7) 725 : cluster 0 2.0 scrub ok 2024-03-11T21:50:46.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:45 smithi143 bash[29284]: cluster 2024-03-11T21:50:44.687946+0000 osd.7 (osd.7) 724 : cluster 0 2.0 scrub starts 2024-03-11T21:50:46.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:45 smithi143 bash[29284]: cluster 2024-03-11T21:50:44.690391+0000 osd.7 (osd.7) 725 : cluster 0 2.0 scrub ok 2024-03-11T21:50:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:45 smithi203 bash[23654]: cluster 2024-03-11T21:50:44.687946+0000 osd.7 (osd.7) 724 : cluster 0 2.0 scrub starts 2024-03-11T21:50:46.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:45 smithi203 bash[23654]: cluster 2024-03-11T21:50:44.690391+0000 osd.7 (osd.7) 725 : cluster 0 2.0 scrub ok 2024-03-11T21:50:47.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/1561125180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630101930 msgr2=0x7ff630105970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:47.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- 172.21.15.143:0/1561125180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630101930 0x7ff630105970 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0099c0 tx=0x7ff61c009cd0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/1561125180 shutdown_connections 2024-03-11T21:50:47.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- 172.21.15.143:0/1561125180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 0x7ff63010dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- 172.21.15.143:0/1561125180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630100f80 0x7ff630101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- 172.21.15.143:0/1561125180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630101930 0x7ff630105970 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/1561125180 >> 172.21.15.143:0/1561125180 conn(0x7ff630077d30 msgr2=0x7ff630078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/1561125180 shutdown_connections 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/1561125180 wait complete. 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 librados: starting msgr at 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 librados: starting objecter 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 Processor -- start 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- start start 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 librados: setting wanted keys 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 librados: calling monclient init 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 0x7ff63019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:47.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 0x7ff6301979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630110560 con 0x7ff630100f80 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6301103e0 con 0x7ff630106050 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6301106e0 con 0x7ff630101930 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff634f61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 0x7ff63019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:60608/0 (socket says 172.21.15.143:60608) 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 -- 172.21.15.143:0/2907399246 learned_addr learned my addr 172.21.15.143:0/2907399246 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff634f61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 0x7ff63019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36570/0 (socket says 172.21.15.143:36570) 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff635762700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 0x7ff6301979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 -- 172.21.15.143:0/2907399246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 msgr2=0x7ff6301979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:47.125 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 0x7ff6301979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 -- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 msgr2=0x7ff63019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 0x7ff63019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 -- 172.21.15.143:0/2907399246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630198110 con 0x7ff630101930 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62ffff700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00fa30 tx=0x7ff61c00fa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62dffb700 10 client.?.objecter ms_handle_connect 0x7ff630101930 2024-03-11T21:50:47.126 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62dffb700 10 client.?.objecter resend_mon_ops 2024-03-11T21:50:47.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c009e60 con 0x7ff630101930 2024-03-11T21:50:47.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(25 keys) v1 ==== 1055+0+0 (secure 0 0 0) 0x7ff61c00fd70 con 0x7ff630101930 2024-03-11T21:50:47.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6301983a0 con 0x7ff630101930 2024-03-11T21:50:47.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c00bea0 con 0x7ff630101930 2024-03-11T21:50:47.128 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 10 client.42539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6301a46c0 con 0x7ff630101930 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.130+0000 7ff6371c5700 1 librados: init done 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63004faa0 con 0x7ff630101930 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 21) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ff61c003bb0 con 0x7ff630101930 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 0x7ff620071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(1021..1021 src has 964..1021) v4 ==== 8794+0+0 (secure 0 0 0) 0x7ff61c08f740 con 0x7ff630101930 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 10 client.42539.objecter ms_dispatch 0x7ff630001550 osd_map(1021..1021 src has 964..1021) v4 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 3 client.42539.objecter handle_osd_map got epochs [1021,1021] > 0 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 3 client.42539.objecter handle_osd_map decoding full epoch 1021 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff634f61700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 0x7ff620071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 20 client.42539.objecter dump_active .. 0 homeless 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff62dffb700 10 client.42539.objecter ms_handle_connect 0x7ff62006f820 2024-03-11T21:50:47.129 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.134+0000 7ff634f61700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 0x7ff620071ce0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7ff630198af0 tx=0x7ff624009260 comp rx=0 tx=0).ready entity=mgr.14550 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:50:47.131 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.138+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff61c013070 con 0x7ff630101930 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:45.682169+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub starts 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:45.685143+0000 osd.7 (osd.7) 727 : cluster 0 5.9 scrub ok 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:45.886215+0000 osd.1 (osd.1) 505 : cluster 0 4.2 scrub starts 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:45.888480+0000 osd.1 (osd.1) 506 : cluster 0 4.2 scrub ok 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:45.902917+0000 mgr.y (mgr.14550) 1059 : cluster 0 pgmap v1730: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.052749+0000 osd.2 (osd.2) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:47.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.053197+0000 osd.2 (osd.2) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:45.682169+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub starts 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:45.685143+0000 osd.7 (osd.7) 727 : cluster 0 5.9 scrub ok 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:45.886215+0000 osd.1 (osd.1) 505 : cluster 0 4.2 scrub starts 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:45.888480+0000 osd.1 (osd.1) 506 : cluster 0 4.2 scrub ok 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:45.902917+0000 mgr.y (mgr.14550) 1059 : cluster 0 pgmap v1730: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.052749+0000 osd.2 (osd.2) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:47.203 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:46 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.053197+0000 osd.2 (osd.2) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:47.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:45.682169+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub starts 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:45.685143+0000 osd.7 (osd.7) 727 : cluster 0 5.9 scrub ok 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:45.886215+0000 osd.1 (osd.1) 505 : cluster 0 4.2 scrub starts 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:45.888480+0000 osd.1 (osd.1) 506 : cluster 0 4.2 scrub ok 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:45.902917+0000 mgr.y (mgr.14550) 1059 : cluster 0 pgmap v1730: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.052749+0000 osd.2 (osd.2) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:50:47.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:46 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.053197+0000 osd.2 (osd.2) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:50:47.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.350+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7ff6301a4890 con 0x7ff630101930 2024-03-11T21:50:47.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.358+0000 7ff62dffb700 1 -- 172.21.15.143:0/2907399246 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff61c021370 con 0x7ff630101930 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 10 librados: watch_flush enter 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 10 librados: watch_flush exit 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 20 client.42539.objecter shutdown clearing up homeless session... 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 10 client.42539.objecter successfully canceled tick 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 msgr2=0x7ff620071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 0x7ff620071ce0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7ff630198af0 tx=0x7ff624009260 comp rx=0 tx=0).stop 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 msgr2=0x7ff63019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:50:47.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00fa30 tx=0x7ff61c00fa60 comp rx=0 tx=0).stop 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 shutdown_connections 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:6800/584821010,v1:172.21.15.143:6801/584821010] conn(0x7ff62006f820 0x7ff620071ce0 unknown :-1 s=CLOSED pgs=3533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff630100f80 0x7ff63019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff630106050 0x7ff6301979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 --2- 172.21.15.143:0/2907399246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff630101930 0x7ff63019dd30 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 >> 172.21.15.143:0/2907399246 conn(0x7ff630077d30 msgr2=0x7ff6300ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:50:47.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 shutdown_connections 2024-03-11T21:50:47.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 -- 172.21.15.143:0/2907399246 wait complete. 2024-03-11T21:50:47.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-11T21:50:47.362+0000 7ff6371c5700 1 librados: shutdown 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE Bus STOPPING 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE Bus STOPPED 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE Bus STARTING 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE Serving on http://:::9283 2024-03-11T21:50:47.638 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21768]: [11/Mar/2024:21:50:47] ENGINE Bus STARTED 2024-03-11T21:50:47.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.057312+0000 osd.4 (osd.4) 659 : cluster 0 9.16 scrub starts 2024-03-11T21:50:47.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.064074+0000 osd.4 (osd.4) 660 : cluster 0 9.16 scrub ok 2024-03-11T21:50:47.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.839940+0000 osd.1 (osd.1) 507 : cluster 0 5.12 scrub starts 2024-03-11T21:50:47.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.841519+0000 osd.1 (osd.1) 508 : cluster 0 5.12 scrub ok 2024-03-11T21:50:47.952 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: cluster 2024-03-11T21:50:46.867559+0000 mon.a (mon.0) 4187 : cluster 0 osdmap e1021: 8 total, 8 up, 8 in 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.355112+0000 mon.c (mon.2) 1438 : audit 1 from='client.? 172.21.15.143:0/2907399246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.355652+0000 mon.a (mon.0) 4188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.360438+0000 mon.a (mon.0) 4189 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.392105+0000 mon.c (mon.2) 1439 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.393754+0000 mon.c (mon.2) 1440 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.394469+0000 mon.c (mon.2) 1441 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[21512]: audit 2024-03-11T21:50:47.401537+0000 mon.a (mon.0) 4190 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.057312+0000 osd.4 (osd.4) 659 : cluster 0 9.16 scrub starts 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.064074+0000 osd.4 (osd.4) 660 : cluster 0 9.16 scrub ok 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.839940+0000 osd.1 (osd.1) 507 : cluster 0 5.12 scrub starts 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.841519+0000 osd.1 (osd.1) 508 : cluster 0 5.12 scrub ok 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: cluster 2024-03-11T21:50:46.867559+0000 mon.a (mon.0) 4187 : cluster 0 osdmap e1021: 8 total, 8 up, 8 in 2024-03-11T21:50:47.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.355112+0000 mon.c (mon.2) 1438 : audit 1 from='client.? 172.21.15.143:0/2907399246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.355652+0000 mon.a (mon.0) 4188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.360438+0000 mon.a (mon.0) 4189 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.392105+0000 mon.c (mon.2) 1439 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.393754+0000 mon.c (mon.2) 1440 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.394469+0000 mon.c (mon.2) 1441 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:47.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:47 smithi143 bash[29284]: audit 2024-03-11T21:50:47.401537+0000 mon.a (mon.0) 4190 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:48.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.057312+0000 osd.4 (osd.4) 659 : cluster 0 9.16 scrub starts 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.064074+0000 osd.4 (osd.4) 660 : cluster 0 9.16 scrub ok 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.839940+0000 osd.1 (osd.1) 507 : cluster 0 5.12 scrub starts 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.841519+0000 osd.1 (osd.1) 508 : cluster 0 5.12 scrub ok 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: cluster 2024-03-11T21:50:46.867559+0000 mon.a (mon.0) 4187 : cluster 0 osdmap e1021: 8 total, 8 up, 8 in 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.355112+0000 mon.c (mon.2) 1438 : audit 1 from='client.? 172.21.15.143:0/2907399246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.355652+0000 mon.a (mon.0) 4188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.360438+0000 mon.a (mon.0) 4189 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.392105+0000 mon.c (mon.2) 1439 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.393754+0000 mon.c (mon.2) 1440 : audit 0 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.394469+0000 mon.c (mon.2) 1441 : audit 1 from='mgr.14550 172.21.15.143:0/622939970' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:48.283 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:47 smithi203 bash[23654]: audit 2024-03-11T21:50:47.401537+0000 mon.a (mon.0) 4190 : audit 1 from='mgr.14550 ' entity='mgr.y' 2024-03-11T21:50:48.328 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:50:48.337 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:50:48.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:48.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:48.365 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:50:48.366 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:50:48.402 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:50:48.413 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:50:48.413 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:50:48.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:50:48.596 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:48 smithi143 systemd[1]: Stopping Ceph mon.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:48.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:48 smithi143 bash[21512]: debug 2024-03-11T21:50:48.602+0000 7f6fba1ab700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:50:48.904 INFO:journalctl@ceph.mon.a.smithi143.stdout:Mar 11 21:50:48 smithi143 bash[21512]: debug 2024-03-11T21:50:48.602+0000 7f6fba1ab700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:50:49.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:50:49 smithi203 bash[48389]: debug there is no tcmu-runner data available 2024-03-11T21:50:49.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:50:49.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:49.965 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:50:49.965 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T21:50:49.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:50:50.201 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE Bus STOPPING 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE Bus STOPPED 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE Bus STARTING 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:50 smithi143 systemd[1]: Stopping Ceph mon.c for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[29284]: debug 2024-03-11T21:50:50.158+0000 7f947a064700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:50:50.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[29284]: debug 2024-03-11T21:50:50.158+0000 7f947a064700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:50:50.545 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE Serving on http://:::9283 2024-03-11T21:50:50.545 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:50 smithi143 bash[21768]: [11/Mar/2024:21:50:50] ENGINE Bus STARTED 2024-03-11T21:50:51.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:50:51.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:51 smithi143 bash[229448]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:50:51.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:51 smithi143 bash[229520]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:50:51.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:51 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T21:50:51.372 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 11 21:50:51 smithi143 systemd[1]: Stopped Ceph mon.c for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:50:51.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:51.391 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T21:50:51.391 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T21:50:51.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:50:51.684 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:51 smithi203 systemd[1]: Stopping Ceph mon.b for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:51.684 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:51 smithi203 bash[23654]: debug 2024-03-11T21:50:51.540+0000 7f8aba036700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:50:51.684 INFO:journalctl@ceph.mon.b.smithi203.stdout:Mar 11 21:50:51 smithi203 bash[23654]: debug 2024-03-11T21:50:51.540+0000 7f8aba036700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:50:52.452 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:52 smithi143 bash[21768]: ::ffff:172.21.15.203 - - [11/Mar/2024:21:50:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:52.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:50:52.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:52.669 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T21:50:52.669 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:50:52.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:50:52.952 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:52 smithi143 systemd[1]: Stopping Ceph mgr.y for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:52.952 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 11 21:50:52 smithi143 bash[21768]: debug 2024-03-11T21:50:52.802+0000 7f3bcf000700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:50:53.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:50:53.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:53.881 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:50:53.881 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:50:53.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:50:54.280 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Mar 11 21:50:53 smithi203 systemd[1]: Stopping Ceph mgr.x for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:54.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:50:54.966 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:54.966 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:50:54.966 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:50:54.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:50:55.452 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:50:54 smithi143 systemd[1]: Stopping Ceph osd.0 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:50:55.452 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:50:55 smithi143 bash[32288]: debug 2024-03-11T21:50:55.114+0000 7f7992301700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:50:55.452 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:50:55 smithi143 bash[32288]: debug 2024-03-11T21:50:55.114+0000 7f7992301700 -1 osd.0 1021 *** Got signal Terminated *** 2024-03-11T21:50:55.452 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:50:55 smithi143 bash[32288]: debug 2024-03-11T21:50:55.114+0000 7f7992301700 -1 osd.0 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:01.533 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:51:01 smithi143 bash[229658]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:51:01.533 INFO:journalctl@ceph.osd.0.smithi143.stdout:Mar 11 21:51:01 smithi143 bash[229721]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:51:02.282 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:51:01 smithi203 bash[52126]: ts=2024-03-11T21:51:01.902Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:51:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:51:01 smithi203 bash[52126]: ts=2024-03-11T21:51:01.905Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:51:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:51:01 smithi203 bash[52126]: ts=2024-03-11T21:51:01.907Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:51:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:51:01 smithi203 bash[52126]: ts=2024-03-11T21:51:01.907Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:51:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:51:01 smithi203 bash[52126]: ts=2024-03-11T21:51:01.910Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:51:03.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:51:03.742 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:03.742 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:51:03.743 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:51:03.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:51:04.202 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:03 smithi143 systemd[1]: Stopping Ceph osd.1 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:04.202 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:03 smithi143 bash[150181]: debug 2024-03-11T21:51:03.933+0000 7f3a4edca700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:04.202 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:03 smithi143 bash[150181]: debug 2024-03-11T21:51:03.933+0000 7f3a4edca700 -1 osd.1 1021 *** Got signal Terminated *** 2024-03-11T21:51:04.202 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:03 smithi143 bash[150181]: debug 2024-03-11T21:51:03.933+0000 7f3a4edca700 -1 osd.1 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:10.452 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:10 smithi143 bash[229856]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:51:10.452 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 11 21:51:10 smithi143 bash[229919]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:51:12.373 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:51:12.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:12.402 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:51:12.402 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:51:12.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:51:12.702 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:12 smithi143 systemd[1]: Stopping Ceph osd.2 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:12.702 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:12 smithi143 bash[43348]: debug 2024-03-11T21:51:12.573+0000 7f54ca84c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:12.702 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:12 smithi143 bash[43348]: debug 2024-03-11T21:51:12.573+0000 7f54ca84c700 -1 osd.2 1021 *** Got signal Terminated *** 2024-03-11T21:51:12.702 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:12 smithi143 bash[43348]: debug 2024-03-11T21:51:12.573+0000 7f54ca84c700 -1 osd.2 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:19.202 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:18 smithi143 bash[230050]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:51:19.202 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 11 21:51:18 smithi143 bash[230114]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:51:20.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:51:20.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:20.925 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:51:20.925 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:51:20.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:51:21.202 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:20 smithi143 systemd[1]: Stopping Ceph osd.3 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:21.202 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:21 smithi143 bash[49102]: debug 2024-03-11T21:51:21.121+0000 7f558d7ef700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:21.202 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:21 smithi143 bash[49102]: debug 2024-03-11T21:51:21.121+0000 7f558d7ef700 -1 osd.3 1021 *** Got signal Terminated *** 2024-03-11T21:51:21.202 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:21 smithi143 bash[49102]: debug 2024-03-11T21:51:21.121+0000 7f558d7ef700 -1 osd.3 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:21.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:21 smithi203 bash[27213]: debug 2024-03-11T21:51:21.176+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:22.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:21 smithi203 bash[43837]: debug 2024-03-11T21:51:21.720+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:22.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:22 smithi203 bash[27213]: debug 2024-03-11T21:51:22.136+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:22.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:22 smithi203 bash[38255]: debug 2024-03-11T21:51:22.044+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:22.988 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:22 smithi203 bash[43837]: debug 2024-03-11T21:51:22.728+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:23.282 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:23 smithi203 bash[27213]: debug 2024-03-11T21:51:23.124+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:23.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:23 smithi203 bash[38255]: debug 2024-03-11T21:51:22.996+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:23.452 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:23 smithi143 bash[49102]: debug 2024-03-11T21:51:23.041+0000 7f5589de3700 -1 osd.3 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.015106+0000 front 2024-03-11T21:50:59.015503+0000 (oldest deadline 2024-03-11T21:51:22.515201+0000) 2024-03-11T21:51:24.026 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:23 smithi203 bash[43837]: debug 2024-03-11T21:51:23.680+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:24.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:24 smithi203 bash[38255]: debug 2024-03-11T21:51:24.036+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:24.282 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:24 smithi203 bash[27213]: debug 2024-03-11T21:51:24.140+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:24.452 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:24 smithi143 bash[49102]: debug 2024-03-11T21:51:24.045+0000 7f5589de3700 -1 osd.3 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.015106+0000 front 2024-03-11T21:50:59.015503+0000 (oldest deadline 2024-03-11T21:51:22.515201+0000) 2024-03-11T21:51:25.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:24 smithi203 bash[43837]: debug 2024-03-11T21:51:24.684+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:25.452 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:25 smithi143 bash[49102]: debug 2024-03-11T21:51:25.033+0000 7f5589de3700 -1 osd.3 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.015106+0000 front 2024-03-11T21:50:59.015503+0000 (oldest deadline 2024-03-11T21:51:22.515201+0000) 2024-03-11T21:51:25.520 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:25 smithi203 bash[27213]: debug 2024-03-11T21:51:25.156+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:25.521 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:25 smithi203 bash[38255]: debug 2024-03-11T21:51:25.072+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:25.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:25 smithi203 bash[43837]: debug 2024-03-11T21:51:25.664+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:25.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:25 smithi203 bash[32509]: debug 2024-03-11T21:51:25.528+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:26.452 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:26 smithi143 bash[49102]: debug 2024-03-11T21:51:26.073+0000 7f5589de3700 -1 osd.3 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.015106+0000 front 2024-03-11T21:50:59.015503+0000 (oldest deadline 2024-03-11T21:51:22.515201+0000) 2024-03-11T21:51:26.514 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:26 smithi203 bash[38255]: debug 2024-03-11T21:51:26.080+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:26.514 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:26 smithi203 bash[27213]: debug 2024-03-11T21:51:26.156+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:26.782 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:26 smithi203 bash[32509]: debug 2024-03-11T21:51:26.524+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:26.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:26 smithi203 bash[43837]: debug 2024-03-11T21:51:26.628+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:27.529 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:27 smithi203 bash[27213]: debug 2024-03-11T21:51:27.172+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:27.529 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:27 smithi203 bash[38255]: debug 2024-03-11T21:51:27.072+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:27.702 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:27 smithi143 bash[230250]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:51:27.702 INFO:journalctl@ceph.osd.3.smithi143.stdout:Mar 11 21:51:27 smithi143 bash[230314]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:51:27.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:27 smithi203 bash[43837]: debug 2024-03-11T21:51:27.648+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:27.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:27 smithi203 bash[32509]: debug 2024-03-11T21:51:27.536+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:28.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:28 smithi203 bash[27213]: debug 2024-03-11T21:51:28.152+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:28.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:28 smithi203 bash[38255]: debug 2024-03-11T21:51:28.112+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:29.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:28 smithi203 bash[43837]: debug 2024-03-11T21:51:28.636+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:29.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:28 smithi203 bash[32509]: debug 2024-03-11T21:51:28.576+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:29.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:51:29.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:29.175 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:51:29.175 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T21:51:29.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:51:29.412 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[38255]: debug 2024-03-11T21:51:29.104+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:29.412 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[27213]: debug 2024-03-11T21:51:29.168+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:29.413 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:29 smithi203 systemd[1]: Stopping Ceph osd.4 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:29.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[27213]: debug 2024-03-11T21:51:29.420+0000 7f32a12a3700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:29.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[27213]: debug 2024-03-11T21:51:29.420+0000 7f32a12a3700 -1 osd.4 1021 *** Got signal Terminated *** 2024-03-11T21:51:29.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[27213]: debug 2024-03-11T21:51:29.420+0000 7f32a12a3700 -1 osd.4 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:29.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[43837]: debug 2024-03-11T21:51:29.608+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:29.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[32509]: debug 2024-03-11T21:51:29.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:29.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:29 smithi203 bash[32509]: debug 2024-03-11T21:51:29.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:30.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[38255]: debug 2024-03-11T21:51:30.072+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:30.533 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[27213]: debug 2024-03-11T21:51:30.196+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:30.533 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[27213]: debug 2024-03-11T21:51:30.196+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.780564+0000 front 2024-03-11T21:51:05.780593+0000 (oldest deadline 2024-03-11T21:51:29.880428+0000) 2024-03-11T21:51:31.025 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[43837]: debug 2024-03-11T21:51:30.648+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:31.025 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[43837]: debug 2024-03-11T21:51:30.648+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:31.025 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[32509]: debug 2024-03-11T21:51:30.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:31.025 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:30 smithi203 bash[32509]: debug 2024-03-11T21:51:30.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:31.282 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[27213]: debug 2024-03-11T21:51:31.204+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:31.283 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[27213]: debug 2024-03-11T21:51:31.204+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.780564+0000 front 2024-03-11T21:51:05.780593+0000 (oldest deadline 2024-03-11T21:51:29.880428+0000) 2024-03-11T21:51:31.283 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[38255]: debug 2024-03-11T21:51:31.032+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:31.283 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[38255]: debug 2024-03-11T21:51:31.032+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:32.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[32509]: debug 2024-03-11T21:51:31.592+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:32.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[32509]: debug 2024-03-11T21:51:31.592+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:32.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[43837]: debug 2024-03-11T21:51:31.652+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:32.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:31 smithi203 bash[43837]: debug 2024-03-11T21:51:31.652+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:32.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[27213]: debug 2024-03-11T21:51:32.204+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:32.533 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[27213]: debug 2024-03-11T21:51:32.204+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.780564+0000 front 2024-03-11T21:51:05.780593+0000 (oldest deadline 2024-03-11T21:51:29.880428+0000) 2024-03-11T21:51:32.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[38255]: debug 2024-03-11T21:51:32.080+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:32.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[38255]: debug 2024-03-11T21:51:32.080+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:33.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[32509]: debug 2024-03-11T21:51:32.608+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:33.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[32509]: debug 2024-03-11T21:51:32.608+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:33.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[43837]: debug 2024-03-11T21:51:32.684+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:33.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:32 smithi203 bash[43837]: debug 2024-03-11T21:51:32.684+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:33.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[27213]: debug 2024-03-11T21:51:33.232+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:33.533 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[27213]: debug 2024-03-11T21:51:33.232+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.780564+0000 front 2024-03-11T21:51:05.780593+0000 (oldest deadline 2024-03-11T21:51:29.880428+0000) 2024-03-11T21:51:33.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[38255]: debug 2024-03-11T21:51:33.120+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:33.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[38255]: debug 2024-03-11T21:51:33.120+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:34.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[32509]: debug 2024-03-11T21:51:33.604+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:34.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[32509]: debug 2024-03-11T21:51:33.604+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:34.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[43837]: debug 2024-03-11T21:51:33.724+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:34.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:33 smithi203 bash[43837]: debug 2024-03-11T21:51:33.724+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:34.532 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[27213]: debug 2024-03-11T21:51:34.188+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.978905+0000 front 2024-03-11T21:50:56.979105+0000 (oldest deadline 2024-03-11T21:51:21.078958+0000) 2024-03-11T21:51:34.533 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[27213]: debug 2024-03-11T21:51:34.188+0000 7f329d897700 -1 osd.4 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.780564+0000 front 2024-03-11T21:51:05.780593+0000 (oldest deadline 2024-03-11T21:51:29.880428+0000) 2024-03-11T21:51:34.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[38255]: debug 2024-03-11T21:51:34.112+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:34.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[38255]: debug 2024-03-11T21:51:34.112+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:34.982 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[32509]: debug 2024-03-11T21:51:34.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:34.982 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[32509]: debug 2024-03-11T21:51:34.628+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:34.983 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[43837]: debug 2024-03-11T21:51:34.700+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:34.983 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:34 smithi203 bash[43837]: debug 2024-03-11T21:51:34.700+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:35.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[38255]: debug 2024-03-11T21:51:35.156+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:35.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[38255]: debug 2024-03-11T21:51:35.156+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:35.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[43837]: debug 2024-03-11T21:51:35.720+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:35.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[43837]: debug 2024-03-11T21:51:35.720+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:35.783 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[104436]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:51:35.783 INFO:journalctl@ceph.osd.4.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[104501]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:51:35.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[32509]: debug 2024-03-11T21:51:35.596+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:35.783 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:35 smithi203 bash[32509]: debug 2024-03-11T21:51:35.596+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:36.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[38255]: debug 2024-03-11T21:51:36.128+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:36.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[38255]: debug 2024-03-11T21:51:36.128+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:36.990 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[32509]: debug 2024-03-11T21:51:36.616+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:36.993 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[32509]: debug 2024-03-11T21:51:36.616+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:36.994 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[43837]: debug 2024-03-11T21:51:36.716+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:36.994 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:36 smithi203 bash[43837]: debug 2024-03-11T21:51:36.716+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:37.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[38255]: debug 2024-03-11T21:51:37.092+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:37.283 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[38255]: debug 2024-03-11T21:51:37.092+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:37.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T21:51:37.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:37.722 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T21:51:37.723 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T21:51:37.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.5 2024-03-11T21:51:37.936 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[43837]: debug 2024-03-11T21:51:37.680+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:37.936 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[43837]: debug 2024-03-11T21:51:37.680+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:37.936 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[32509]: debug 2024-03-11T21:51:37.604+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:37.936 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[32509]: debug 2024-03-11T21:51:37.604+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:37.936 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 systemd[1]: Stopping Ceph osd.5 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:38.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[32509]: debug 2024-03-11T21:51:37.944+0000 7fcbc3b10700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:38.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[32509]: debug 2024-03-11T21:51:37.944+0000 7fcbc3b10700 -1 osd.5 1021 *** Got signal Terminated *** 2024-03-11T21:51:38.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:37 smithi203 bash[32509]: debug 2024-03-11T21:51:37.944+0000 7fcbc3b10700 -1 osd.5 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:38.282 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[38255]: debug 2024-03-11T21:51:38.104+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:38.283 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[38255]: debug 2024-03-11T21:51:38.104+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:39.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[32509]: debug 2024-03-11T21:51:38.607+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:39.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[32509]: debug 2024-03-11T21:51:38.607+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:39.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[43837]: debug 2024-03-11T21:51:38.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:39.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[43837]: debug 2024-03-11T21:51:38.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:39.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:38 smithi203 bash[43837]: debug 2024-03-11T21:51:38.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:39.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[38255]: debug 2024-03-11T21:51:39.107+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:39.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[38255]: debug 2024-03-11T21:51:39.107+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:40.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[32509]: debug 2024-03-11T21:51:39.635+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[32509]: debug 2024-03-11T21:51:39.635+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[32509]: debug 2024-03-11T21:51:39.635+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:12.909445+0000 front 2024-03-11T21:51:12.909098+0000 (oldest deadline 2024-03-11T21:51:38.709868+0000) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[32509]: debug 2024-03-11T21:51:39.635+0000 7fcbbc0fc700 -1 osd.5 1021 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14427.0:5179 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93859898876928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1021) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[43837]: debug 2024-03-11T21:51:39.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[43837]: debug 2024-03-11T21:51:39.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:40.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:39 smithi203 bash[43837]: debug 2024-03-11T21:51:39.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:40.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[38255]: debug 2024-03-11T21:51:40.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:40.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[38255]: debug 2024-03-11T21:51:40.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:40.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[38255]: debug 2024-03-11T21:51:40.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:41.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[32509]: debug 2024-03-11T21:51:40.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:41.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[32509]: debug 2024-03-11T21:51:40.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:41.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[32509]: debug 2024-03-11T21:51:40.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:12.909445+0000 front 2024-03-11T21:51:12.909098+0000 (oldest deadline 2024-03-11T21:51:38.709868+0000) 2024-03-11T21:51:41.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[32509]: debug 2024-03-11T21:51:40.647+0000 7fcbbc0fc700 -1 osd.5 1021 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14427.0:5179 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93859898876928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1021) 2024-03-11T21:51:41.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[43837]: debug 2024-03-11T21:51:40.587+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:41.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[43837]: debug 2024-03-11T21:51:40.587+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:41.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:40 smithi203 bash[43837]: debug 2024-03-11T21:51:40.587+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:41.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[38255]: debug 2024-03-11T21:51:41.063+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:41.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[38255]: debug 2024-03-11T21:51:41.063+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:41.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[38255]: debug 2024-03-11T21:51:41.063+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:42.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[32509]: debug 2024-03-11T21:51:41.683+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:42.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[32509]: debug 2024-03-11T21:51:41.683+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:42.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[32509]: debug 2024-03-11T21:51:41.683+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:12.909445+0000 front 2024-03-11T21:51:12.909098+0000 (oldest deadline 2024-03-11T21:51:38.709868+0000) 2024-03-11T21:51:42.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[32509]: debug 2024-03-11T21:51:41.683+0000 7fcbbc0fc700 -1 osd.5 1021 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14427.0:5179 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93859898876928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1021) 2024-03-11T21:51:42.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[43837]: debug 2024-03-11T21:51:41.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:42.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[43837]: debug 2024-03-11T21:51:41.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:42.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:41 smithi203 bash[43837]: debug 2024-03-11T21:51:41.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:42.535 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[38255]: debug 2024-03-11T21:51:42.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:42.535 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[38255]: debug 2024-03-11T21:51:42.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:42.535 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[38255]: debug 2024-03-11T21:51:42.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:43.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[32509]: debug 2024-03-11T21:51:42.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:59.104949+0000 front 2024-03-11T21:50:59.105026+0000 (oldest deadline 2024-03-11T21:51:25.004543+0000) 2024-03-11T21:51:43.032 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[32509]: debug 2024-03-11T21:51:42.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:07.306102+0000 front 2024-03-11T21:51:07.306377+0000 (oldest deadline 2024-03-11T21:51:29.506969+0000) 2024-03-11T21:51:43.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[32509]: debug 2024-03-11T21:51:42.647+0000 7fcbbc0fc700 -1 osd.5 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:12.909445+0000 front 2024-03-11T21:51:12.909098+0000 (oldest deadline 2024-03-11T21:51:38.709868+0000) 2024-03-11T21:51:43.033 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[32509]: debug 2024-03-11T21:51:42.647+0000 7fcbbc0fc700 -1 osd.5 1021 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14427.0:5179 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93859898876928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1021) 2024-03-11T21:51:43.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[43837]: debug 2024-03-11T21:51:42.583+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:43.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[43837]: debug 2024-03-11T21:51:42.583+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:43.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:42 smithi203 bash[43837]: debug 2024-03-11T21:51:42.583+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:43.367 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[38255]: debug 2024-03-11T21:51:43.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:43.367 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[38255]: debug 2024-03-11T21:51:43.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:43.367 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[38255]: debug 2024-03-11T21:51:43.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:43.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[43837]: debug 2024-03-11T21:51:43.591+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:43.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[43837]: debug 2024-03-11T21:51:43.591+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:43.782 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[43837]: debug 2024-03-11T21:51:43.591+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:44.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:43 smithi203 bash[104635]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:51:44.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[104699]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:51:44.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[38255]: debug 2024-03-11T21:51:44.051+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:44.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[38255]: debug 2024-03-11T21:51:44.051+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:44.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[38255]: debug 2024-03-11T21:51:44.051+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:45.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[43837]: debug 2024-03-11T21:51:44.567+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:45.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[43837]: debug 2024-03-11T21:51:44.567+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:45.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:44 smithi203 bash[43837]: debug 2024-03-11T21:51:44.567+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:45.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[38255]: debug 2024-03-11T21:51:45.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:45.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[38255]: debug 2024-03-11T21:51:45.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:45.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[38255]: debug 2024-03-11T21:51:45.043+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:46.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[43837]: debug 2024-03-11T21:51:45.559+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:46.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[43837]: debug 2024-03-11T21:51:45.559+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:46.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:45 smithi203 bash[43837]: debug 2024-03-11T21:51:45.559+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:46.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T21:51:46.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:46.251 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T21:51:46.251 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T21:51:46.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.6 2024-03-11T21:51:46.301 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.039+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:46.301 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.039+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:46.301 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.039+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:46.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 systemd[1]: Stopping Ceph osd.6 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:46.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.439+0000 7f029456e700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:46.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.439+0000 7f029456e700 -1 osd.6 1021 *** Got signal Terminated *** 2024-03-11T21:51:46.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[38255]: debug 2024-03-11T21:51:46.439+0000 7f029456e700 -1 osd.6 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:47.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[43837]: debug 2024-03-11T21:51:46.603+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:47.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[43837]: debug 2024-03-11T21:51:46.603+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:47.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:46 smithi203 bash[43837]: debug 2024-03-11T21:51:46.603+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:47.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[38255]: debug 2024-03-11T21:51:47.079+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:47.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[38255]: debug 2024-03-11T21:51:47.079+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:47.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[38255]: debug 2024-03-11T21:51:47.079+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:47.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[38255]: debug 2024-03-11T21:51:47.079+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.739212+0000 front 2024-03-11T21:51:23.739166+0000 (oldest deadline 2024-03-11T21:51:46.639129+0000) 2024-03-11T21:51:48.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[43837]: debug 2024-03-11T21:51:47.615+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:48.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[43837]: debug 2024-03-11T21:51:47.615+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:48.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[43837]: debug 2024-03-11T21:51:47.615+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:48.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:47 smithi203 bash[43837]: debug 2024-03-11T21:51:47.615+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:48.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[38255]: debug 2024-03-11T21:51:48.103+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:48.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[38255]: debug 2024-03-11T21:51:48.103+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:48.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[38255]: debug 2024-03-11T21:51:48.103+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:48.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[38255]: debug 2024-03-11T21:51:48.103+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.739212+0000 front 2024-03-11T21:51:23.739166+0000 (oldest deadline 2024-03-11T21:51:46.639129+0000) 2024-03-11T21:51:49.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[43837]: debug 2024-03-11T21:51:48.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:49.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[43837]: debug 2024-03-11T21:51:48.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:49.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[43837]: debug 2024-03-11T21:51:48.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:49.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:48 smithi203 bash[43837]: debug 2024-03-11T21:51:48.643+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:49.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[38255]: debug 2024-03-11T21:51:49.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:49.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[38255]: debug 2024-03-11T21:51:49.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:49.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[38255]: debug 2024-03-11T21:51:49.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:49.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[38255]: debug 2024-03-11T21:51:49.091+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.739212+0000 front 2024-03-11T21:51:23.739166+0000 (oldest deadline 2024-03-11T21:51:46.639129+0000) 2024-03-11T21:51:50.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[43837]: debug 2024-03-11T21:51:49.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:50.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[43837]: debug 2024-03-11T21:51:49.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:50.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[43837]: debug 2024-03-11T21:51:49.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:50.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:49 smithi203 bash[43837]: debug 2024-03-11T21:51:49.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:50.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[38255]: debug 2024-03-11T21:51:50.135+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:50.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[38255]: debug 2024-03-11T21:51:50.135+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:50.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[38255]: debug 2024-03-11T21:51:50.135+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:50.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[38255]: debug 2024-03-11T21:51:50.135+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.739212+0000 front 2024-03-11T21:51:23.739166+0000 (oldest deadline 2024-03-11T21:51:46.639129+0000) 2024-03-11T21:51:51.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[43837]: debug 2024-03-11T21:51:50.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:51.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[43837]: debug 2024-03-11T21:51:50.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:51.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[43837]: debug 2024-03-11T21:51:50.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:51.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:50 smithi203 bash[43837]: debug 2024-03-11T21:51:50.623+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:51.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[38255]: debug 2024-03-11T21:51:51.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:56.834059+0000 front 2024-03-11T21:50:56.834540+0000 (oldest deadline 2024-03-11T21:51:21.534049+0000) 2024-03-11T21:51:51.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[38255]: debug 2024-03-11T21:51:51.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:05.635744+0000 front 2024-03-11T21:51:05.635834+0000 (oldest deadline 2024-03-11T21:51:30.335538+0000) 2024-03-11T21:51:51.532 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[38255]: debug 2024-03-11T21:51:51.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:14.437344+0000 front 2024-03-11T21:51:14.437463+0000 (oldest deadline 2024-03-11T21:51:39.736980+0000) 2024-03-11T21:51:51.533 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[38255]: debug 2024-03-11T21:51:51.087+0000 7f0290b62700 -1 osd.6 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.739212+0000 front 2024-03-11T21:51:23.739166+0000 (oldest deadline 2024-03-11T21:51:46.639129+0000) 2024-03-11T21:51:52.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[43837]: debug 2024-03-11T21:51:51.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:52.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[43837]: debug 2024-03-11T21:51:51.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:52.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[43837]: debug 2024-03-11T21:51:51.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:52.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:51 smithi203 bash[43837]: debug 2024-03-11T21:51:51.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:52.731 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[104835]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:51:52.731 INFO:journalctl@ceph.osd.6.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[104898]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:51:52.732 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[43837]: debug 2024-03-11T21:51:52.683+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:52.732 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[43837]: debug 2024-03-11T21:51:52.683+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:52.732 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[43837]: debug 2024-03-11T21:51:52.683+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:52.732 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:52 smithi203 bash[43837]: debug 2024-03-11T21:51:52.683+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:53.992 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:53 smithi203 bash[43837]: debug 2024-03-11T21:51:53.695+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:53.992 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:53 smithi203 bash[43837]: debug 2024-03-11T21:51:53.695+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:53.992 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:53 smithi203 bash[43837]: debug 2024-03-11T21:51:53.695+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:53.992 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:53 smithi203 bash[43837]: debug 2024-03-11T21:51:53.695+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:54.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T21:51:54.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:54.756 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T21:51:54.756 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T21:51:54.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.7 2024-03-11T21:51:54.912 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.659+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:54.913 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.659+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:54.913 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.659+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:54.913 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.659+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:54.913 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 systemd[1]: Stopping Ceph osd.7 for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:51:55.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.919+0000 7feb1bb68700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:51:55.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.919+0000 7feb1bb68700 -1 osd.7 1021 *** Got signal Terminated *** 2024-03-11T21:51:55.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:54 smithi203 bash[43837]: debug 2024-03-11T21:51:54.919+0000 7feb1bb68700 -1 osd.7 1021 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:56.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:55 smithi203 bash[43837]: debug 2024-03-11T21:51:55.679+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:56.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:55 smithi203 bash[43837]: debug 2024-03-11T21:51:55.679+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:56.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:55 smithi203 bash[43837]: debug 2024-03-11T21:51:55.679+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:56.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:55 smithi203 bash[43837]: debug 2024-03-11T21:51:55.679+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:57.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:56 smithi203 bash[43837]: debug 2024-03-11T21:51:56.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:57.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:56 smithi203 bash[43837]: debug 2024-03-11T21:51:56.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:57.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:56 smithi203 bash[43837]: debug 2024-03-11T21:51:56.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:57.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:56 smithi203 bash[43837]: debug 2024-03-11T21:51:56.635+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:58.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:57 smithi203 bash[43837]: debug 2024-03-11T21:51:57.611+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:58.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:57 smithi203 bash[43837]: debug 2024-03-11T21:51:57.611+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:58.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:57 smithi203 bash[43837]: debug 2024-03-11T21:51:57.611+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:58.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:57 smithi203 bash[43837]: debug 2024-03-11T21:51:57.611+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:51:59.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:58 smithi203 bash[43837]: debug 2024-03-11T21:51:58.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:51:59.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:58 smithi203 bash[43837]: debug 2024-03-11T21:51:58.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:51:59.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:58 smithi203 bash[43837]: debug 2024-03-11T21:51:58.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:51:59.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:58 smithi203 bash[43837]: debug 2024-03-11T21:51:58.619+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:52:00.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:59 smithi203 bash[43837]: debug 2024-03-11T21:51:59.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6806 osd.0 since back 2024-03-11T21:50:57.518640+0000 front 2024-03-11T21:50:57.518305+0000 (oldest deadline 2024-03-11T21:51:21.018321+0000) 2024-03-11T21:52:00.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:59 smithi203 bash[43837]: debug 2024-03-11T21:51:59.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6814 osd.1 since back 2024-03-11T21:51:06.820600+0000 front 2024-03-11T21:51:06.820564+0000 (oldest deadline 2024-03-11T21:51:29.720200+0000) 2024-03-11T21:52:00.032 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:59 smithi203 bash[43837]: debug 2024-03-11T21:51:59.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6822 osd.2 since back 2024-03-11T21:51:16.122403+0000 front 2024-03-11T21:51:16.122865+0000 (oldest deadline 2024-03-11T21:51:38.422547+0000) 2024-03-11T21:52:00.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:59 smithi203 bash[43837]: debug 2024-03-11T21:51:59.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.143:6830 osd.3 since back 2024-03-11T21:51:23.624774+0000 front 2024-03-11T21:51:23.624663+0000 (oldest deadline 2024-03-11T21:51:47.124461+0000) 2024-03-11T21:52:00.033 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:51:59 smithi203 bash[43837]: debug 2024-03-11T21:51:59.663+0000 7feb14154700 -1 osd.7 1021 heartbeat_check: no reply from 172.21.15.203:6804 osd.4 since back 2024-03-11T21:51:34.126427+0000 front 2024-03-11T21:51:34.126487+0000 (oldest deadline 2024-03-11T21:51:59.326824+0000) 2024-03-11T21:52:01.532 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[105032]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:52:01.533 INFO:journalctl@ceph.osd.7.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[105097]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:52:02.282 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[52126]: ts=2024-03-11T21:52:01.902Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:52:02.282 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[52126]: ts=2024-03-11T21:52:01.905Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:52:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[52126]: ts=2024-03-11T21:52:01.908Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:52:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[52126]: ts=2024-03-11T21:52:01.908Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:52:02.283 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:01 smithi203 bash[52126]: ts=2024-03-11T21:52:01.909Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.143:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.143:8765: connect: connection refused" 2024-03-11T21:52:03.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T21:52:03.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:03.244 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T21:52:03.245 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:52:03.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:52:03.703 INFO:journalctl@ceph.rgw.foo.a.smithi143.stdout:Mar 11 21:52:03 smithi143 systemd[1]: Stopping Ceph rgw.foo.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:03.703 INFO:journalctl@ceph.rgw.foo.a.smithi143.stdout:Mar 11 21:52:03 smithi143 bash[53496]: debug 2024-03-11T21:52:03.392+0000 7f7348a98700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:52:03.703 INFO:journalctl@ceph.rgw.foo.a.smithi143.stdout:Mar 11 21:52:03 smithi143 bash[53496]: debug 2024-03-11T21:52:03.392+0000 7f7357310ac0 -1 shutting down 2024-03-11T21:52:14.258 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:52:14.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:14.282 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:52:14.282 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:52:14.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:52:14.660 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 systemd[1]: Stopping Ceph prometheus.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:14.660 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.442Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:52:14.660 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:52:14.660 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:52:14.660 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.443Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.444Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.444Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:52:14.661 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Mar 11 21:52:14 smithi203 bash[52126]: ts=2024-03-11T21:52:14.444Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:52:15.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:52:15.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:15.271 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:52:15.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:52:15.438 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:52:20.652 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:20.652 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:20.652 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:20.652 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:20.906 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:20.906 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.202 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.202 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.203 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:20 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.203 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.471 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.471 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.751 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:21.751 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:22.093 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:22.093 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: Stopping Ceph node-exporter.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:22.094 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:21 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:23.059 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:22 smithi143 bash[230913]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:52:23.059 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:22 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:52:23.059 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:22 smithi143 bash[230975]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:52:23.059 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:22 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:52:23.059 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:22 smithi143 systemd[1]: Stopped Ceph node-exporter.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:52:23.318 INFO:journalctl@ceph.node-exporter.a.smithi143.stdout:Mar 11 21:52:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:23.319 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:23 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:23.319 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:23 smithi143 systemd[1]: Stopping Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:23.319 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:23 smithi143 bash[63388]: ts=2024-03-11T21:52:23.285Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:52:24.202 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:23 smithi143 bash[231043]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:52:24.202 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:24 smithi143 bash[231104]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:52:24.203 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:24 smithi143 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:52:24.203 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:24 smithi143 systemd[1]: Stopped Ceph alertmanager.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:52:24.508 INFO:journalctl@ceph.alertmanager.a.smithi143.stdout:Mar 11 21:52:24 smithi143 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:24.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:52:24.926 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:52:29.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:29.783 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:29.783 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.063 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.063 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:29 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.315 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.315 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.671 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.671 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.962 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.962 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.963 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:30.963 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:30 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:31.282 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:31 smithi203 systemd[1]: Stopping Ceph iscsi.iscsi.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:31.282 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:31 smithi203 bash[48389]: debug Shutdown received 2024-03-11T21:52:41.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:41 smithi203 bash[105569]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:52:42.119 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:42 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:42.120 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:42 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:42.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:41 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:52:42.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:41 smithi203 bash[105638]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:52:42.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:41 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T21:52:42.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:41 smithi203 systemd[1]: Stopped Ceph iscsi.iscsi.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:52:42.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Mar 11 21:52:42 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:43.282 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:43 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:43.282 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:43 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:43.593 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:43 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:43.593 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:43 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:43.593 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:43 smithi203 systemd[1]: Stopping Ceph node-exporter.b for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:44.404 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 bash[105843]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:52:44.405 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:52:44.405 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 bash[105903]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:52:44.405 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T21:52:44.405 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: Stopped Ceph node-exporter.b for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:52:44.673 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:44.673 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:44.673 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:44 smithi203 systemd[1]: Stopping Ceph grafana.a for eebe2e36-dfeb-11ee-95c8-87774f69a715... 2024-03-11T21:52:45.443 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 bash[105973]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:52:45.443 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:52:45.443 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 bash[106034]: Error response from daemon: No such container: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:52:45.443 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 systemd[1]: ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:52:45.443 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 systemd[1]: Stopped Ceph grafana.a for eebe2e36-dfeb-11ee-95c8-87774f69a715. 2024-03-11T21:52:45.702 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Mar 11 21:52:45 smithi203 systemd[1]: /etc/systemd/system/ceph-eebe2e36-dfeb-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:52:45.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:52:45.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:52:45.968 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:52:45.970 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi143/crash 2024-03-11T21:52:45.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:52:46.009 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:52:46.009 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:52:46.011 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi203/crash 2024-03-11T21:52:46.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:52:46.024 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:52:46.024 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:52:46.025 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:52:46.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-11T21:52:46.068 INFO:teuthology.orchestra.run.smithi143.stdout:2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:52:46.069 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:52:46.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[SEC\]' /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-11T21:52:46.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[ERR\]' /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-11T21:52:46.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[WRN\]' /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-11T21:52:46.234 INFO:teuthology.orchestra.run.smithi143.stdout:2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-11T21:52:46.235 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:52:46.235 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:52:46.279 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:52:46.287 INFO:teuthology.orchestra.run.smithi143.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-03-11T21:52:46.292 INFO:teuthology.orchestra.run.smithi203.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-03-11T21:52:46.300 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:52:46.301 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T21:52:46.301 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T21:52:46.305 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr: 81.2% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/tcmu-runner.log: 68.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:52:46.306 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:52:46.307 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:52:46.307 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.3.log: 88.8% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:52:46.308 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:52:46.310 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:52:46.310 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T21:52:46.318 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.1.log: /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log: 93.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:52:46.318 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:52:46.320 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.b.log: 91.0% 92.6% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:52:46.320 INFO:teuthology.orchestra.run.smithi203.stderr: -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:52:46.320 INFO:teuthology.orchestra.run.smithi203.stderr: 92.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:52:46.322 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.4.log: 85.3% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:52:46.332 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log: 91.7% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:52:46.332 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:52:46.334 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:52:46.335 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:52:46.359 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: 95.1% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:52:46.636 INFO:teuthology.orchestra.run.smithi143.stderr: 93.6% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:52:46.857 INFO:teuthology.orchestra.run.smithi143.stderr: 90.9% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:52:47.761 INFO:teuthology.orchestra.run.smithi143.stderr: 92.1% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:52:48.239 INFO:teuthology.orchestra.run.smithi203.stderr: 92.7% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:52:49.372 INFO:teuthology.orchestra.run.smithi143.stderr: 91.8% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:52:54.245 INFO:teuthology.orchestra.run.smithi203.stderr: 93.8% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T21:52:54.322 INFO:teuthology.orchestra.run.smithi143.stderr: 93.8% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:52:54.793 INFO:teuthology.orchestra.run.smithi143.stderr: 94.1% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:52:55.145 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T21:52:55.618 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T21:52:55.925 INFO:teuthology.orchestra.run.smithi203.stderr: 94.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T21:52:55.927 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-11T21:52:55.927 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m9.642s 2024-03-11T21:52:55.927 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m36.293s 2024-03-11T21:52:55.927 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m1.438s 2024-03-11T21:52:56.284 INFO:teuthology.orchestra.run.smithi143.stderr: 94.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:52:56.377 INFO:teuthology.orchestra.run.smithi143.stderr: 94.0% -- replaced with /var/log/ceph/eebe2e36-dfeb-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T21:52:56.377 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-11T21:52:56.377 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m10.096s 2024-03-11T21:52:56.378 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m40.513s 2024-03-11T21:52:56.378 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m1.474s 2024-03-11T21:52:56.379 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:52:56.379 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi143/log 2024-03-11T21:52:56.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:53:01.029 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi203/log 2024-03-11T21:53:01.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:53:05.262 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:53:05.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 --force 2024-03-11T21:53:05.432 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:53:06.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eebe2e36-dfeb-11ee-95c8-87774f69a715 --force 2024-03-11T21:53:06.986 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: eebe2e36-dfeb-11ee-95c8-87774f69a715 2024-03-11T21:53:08.377 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:53:08.377 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:53:08.383 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:53:08.389 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:53:08.389 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T21:53:08.399 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T21:53:08.399 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:53:08.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:53:08.496 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T21:53:08.505 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T21:53:08.505 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:53:08.508 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:53:08.521 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:53:08.521 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-03-11T21:53:08.521 INFO:teuthology.orchestra.run.smithi143.stdout:*hv01.front.sepi 44.4.53.2 2 u 56 128 377 0.122 -9.083 1.629 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi143.stdout:+hv02.front.sepi 63.231.80.2 3 u 120 128 377 0.071 -2.266 1.524 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi143.stdout:+hv03.front.sepi 74.6.168.72 3 u 72 128 377 0.093 -5.026 2.108 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout:*hv01.front.sepi 44.4.53.2 2 u 56 128 377 0.092 -6.484 0.775 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout:+hv02.front.sepi 63.231.80.2 3 u 57 128 377 0.109 +1.013 1.004 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout:+hv03.front.sepi 74.6.168.72 3 u 124 128 377 0.105 -0.682 0.766 2024-03-11T21:53:08.522 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-11T21:53:08.522 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T21:53:08.531 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T21:53:08.532 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T21:53:08.541 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T21:53:08.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T21:53:08.558 INFO:teuthology.task.internal:Duration was 3041.591525 seconds 2024-03-11T21:53:08.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T21:53:08.566 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T21:53:08.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:53:08.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:53:08.605 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T21:53:08.605 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-03-11T21:53:08.605 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:53:08.655 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-11T21:53:08.655 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:53:08.665 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T21:53:08.665 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:53:08.699 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:53:08.827 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T21:53:08.827 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:53:08.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:53:09.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T21:53:09.300 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T21:53:09.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:53:09.312 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:53:09.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T21:53:09.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:53:09.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:53:09.364 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-03-11T21:53:09.380 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-11T21:53:09.415 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:53:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:53:09.422 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:53:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:53:09.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T21:53:09.472 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T21:53:09.472 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi143 2024-03-11T21:53:09.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:53:09.537 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/remote/smithi203 2024-03-11T21:53:09.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:53:09.570 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T21:53:09.570 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:53:09.573 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:53:09.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T21:53:09.626 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T21:53:09.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T21:53:09.636 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T21:53:09.636 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:53:09.639 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:53:09.642 INFO:teuthology.orchestra.run.smithi143.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:53 /home/ubuntu/cephtest 2024-03-11T21:53:09.661 INFO:teuthology.orchestra.run.smithi203.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:53 /home/ubuntu/cephtest 2024-03-11T21:53:09.663 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T21:53:09.674 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T21:53:09.695 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T21:53:09.721 DEBUG:teuthology.nuke:shortname: smithi143 2024-03-11T21:53:09.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:53:09.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:52:51.095129', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEL3NFitolg0GcBmA1aZq6ZUy9YNJX5q4shU5wJozfPBMjoHBuRLq5bz0GvpfWeyo2O18kiF7TALmAS+k76myWY='} 2024-03-11T21:53:09.749 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-11T21:53:09.749 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:53:09.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:52:51.096233', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBo4QJ51si+hcxMlkyJvTAtHEimko/01zzJxKlVxhkgPF+6wxegdI8U6w+5hqjNiOLxDGSAQOtD0rmEnMba0cAU='} 2024-03-11T21:53:09.799 INFO:teuthology.orchestra.console.smithi143:Power off 2024-03-11T21:53:09.799 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:53:09.822 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-03-11T21:53:09.822 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:09.836 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-11T21:53:09.837 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:53:09.846 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-11T21:53:09.859 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-11T21:53:09.859 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:09.882 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-11T21:53:13.848 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:13.883 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:13.958 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-11T21:53:13.991 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-11T21:53:17.962 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:17.992 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:18.072 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-03-11T21:53:18.101 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-11T21:53:18.173 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-03-11T21:53:22.102 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:53:22.212 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-11T21:53:22.313 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-11T21:53:22.416 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} duration: 3041.5915248394012 failure_reason: '"2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T21:53:22.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:53:22.511 INFO:teuthology.run:FAIL